Overview

It's Tuesday, 3 AM. Your procurement AI system suddenly starts approving invoices at 10x normal speed without human review. Your escalation alert threshold is misconfigured so monitoring doesn't catch it. Your team doesn't notice until 8 AM when they see the damage: $12M in invoices auto-approved that should have been reviewed, including several containing duplicate line items and inflated quantities. The AI didn't fail completely. It kept running exactly as designed, but a configuration issue removed the human safeguard. Now you have two hours before the invoice batch is transmitted to accounts payable and becomes immutable. Can you stop it? Can you reverse it? Can you validate which invoices are fraudulent? Do you have a runbook for this exact scenario? Do you know who to call at 3 AM? Do your engineers know the fallback process if they can't restore the system? Do you have a pre-written template for communicating a $12M procurement failure to your CFO? Most operations leaders don't. And that's the problem.

This chapter teaches you to prepare for AI failures before they happen. Not because you're pessimistic, but because you're professional. You prepare incident response playbooks for IT outages, supply chain disruptions, and cyber attacks. You run tabletop exercises to test whether your team knows what to do in a crisis. You maintain fallback processes so that if one system fails, you keep running. AI failures deserve the same level of professional preparation. The best incident response isn't reactive scrambling in a crisis with people panicking and making mistakes. It's proactive planning, clear procedures, well-trained teams, and the confidence that comes from rehearsing. By the end of this lesson, you'll have an incident response framework you can implement immediately, severity classification criteria, a detailed playbook template, and processes for learning from incidents. When something goes wrong, and with AI systems, something eventually will. You'll be the calm voice saying "we have a playbook for this, here's what we do."

AI Failure Modes: What Can Go Wrong

Before you can respond to failures, you need to understand what failures are possible. AI systems fail differently from traditional systems. Most failures are silent, not loud.

Accuracy degradation (silent failure): The model's accuracy gradually declines as it sees data it wasn't trained on. No alert fires. No user complaint. But decisions slowly get worse. You might not notice for weeks or months. By then, you've made thousands of poor decisions.

Data quality issues: The data feeding the model becomes incomplete, stale, or incorrect. Example: Supplier data isn't updated, so the model recommends suppliers that no longer exist. Or transaction data has a formatting error that breaks the model.

Bias or disparate impact (silent failure): The model systematically disadvantages a group you care about (minority-owned suppliers, female-owned vendors, etc.). No error message. No alert. You only discover it through an audit or lawsuit.

System availability failure (loud failure): The system is down, unreachable, or not responding. This one is obvious and gets immediate attention.

Model instability: The model starts producing wildly inconsistent recommendations. Same input produces different outputs. Users lose confidence and stop using the system.

Escalation queue backup: The number of decisions escalating to humans exceeds your capacity. Escalations sit in queue for days. Decisions wait. Users get frustrated.

Security breach: Someone gains unauthorized access to training data, model weights, or predictions. Your competitive advantages or sensitive customer data is compromised.

Incident Classification Framework: Matching Response to Severity

Create a classification system that determines urgency and response level. Not all failures deserve the same response. Proportionate response prevents alert fatigue and ensures resources go where they're needed.

Severity 1 (Critical): Immediate response required

The system has failed catastrophically or is making dangerously wrong decisions. Operational impact is immediate and material. Examples:

• System is down or completely inaccessible (can't process transactions at all)
  - Accuracy drops below 70% (model is guessing worse than random)
  - Major bias discovered affecting >5% of transactions with material impact
  - Security breach with confirmed unauthorized access
  - System making decisions counter to business logic (e.g., approving all invoices when it should approve none)
  - Escalation volume >75% (your team can't keep up)

Response time: Alert within 5 minutes. Incident commander assigned within 15 minutes. Executive escalation within 30 minutes. Target resolution:
- Accuracy drops to 75-85% (degraded but not catastrophic)
- System latency increases 3-5x (users experiencing noticeable delays)
- Escalation volume 50-75% (elevated but still manageable)
- Minor bias detected affecting

Response time: Alert within 15 minutes. Investigation started within 1 hour. Operations council escalation within 2 hours. Target resolution:
- Accuracy drops to 85-90% (slightly below normal but acceptable)
- Minor fairness concerns (not material disparate impact)
- Adoption slightly declining but not concerning
- Specific feature not working but rest of system operates
- Monitoring alerts not firing but metrics trending wrong

Response time: Alert within 24 hours. Investigation started within 48 hours. No executive escalation unless pattern emerges. Target resolution:
- Minor UI bugs or display issues
- Documentation gaps or typos
- Monitoring alert threshold needs tuning
- Performance improvement opportunity

Response time: Within 1-2 weeks. No escalation required.

Important: Classification discipline prevents two problems: (1) Over-escalation that causes alert fatigue and makes executives ignore real crises, (2) Under-escalation that causes slow response to serious problems. Be honest about severity. If you classify everything as Severity 1, you have no credibility when a real crisis happens.

The Four Phases of Incident Response

When an incident is detected, follow this protocol. The protocol is the same for all severities, but the speed and escalation level differ.

Phase 1: Activation (First 15-30 minutes for Severity 1, within 2 hours for Severity 2)

What happens: Incident is detected. Alert fires or someone reports a problem. System owner is notified (phone call, not Slack, Slack is for after you're on the phone). Incident commander is designated and takes control. Initial triage: What is the scope? How many transactions affected? Can we operate without this system? If the system is in critical failure, immediately activate fallback process and switch to manual/backup operation.

Key principles: Speed matters. Every minute the system is broken, you're hemorrhaging. Get the incident commander on a call immediately. Brief, action-oriented response. No long meetings. Just fix it.

Severity 1 example: Procurement AI is down. 2000 purchase orders are waiting to be processed. System owner is called at 9:03 AM. By 9:15 AM, incident commander is on call with engineering and procurement teams. By 9:25 AM, procurement has switched to manual processing for new orders. Engineering is investigating why the system went down.

Phase 2: Containment and Communication (Ongoing during incident)

What happens: You've addressed the immediate emergency (either fixed the system or switched to fallback). Now you communicate status regularly and work on fixing root cause.

Communication cadence: Depends on severity:

• Severity 1: Update exec sponsor every 15 minutes. One-line status: "Still investigating," "Found the issue, fixing now," "Issue fixed, validating."
  - Severity 2: Update operations leader every 30-60 minutes. Brief status: What we know, what we're doing, ETA to fix.
  - Severity 3: Daily update. What happened, what we're doing, timeline.

Each update includes: Status (investigating/working on fix/testing/resolved), current impact (how many transactions affected, business impact), action being taken, estimated time to resolution, next update time.

After incident resolves: Send all-hands communication (email, Slack, town hall) explaining: what happened, when, for how long, business impact, root cause, how we fixed it, what we're doing to prevent it. This builds trust and shows you're in control.

Phase 3: Investigation and Root Cause Analysis (24-48 hours post-incident)

What happens: Once you've fixed the immediate problem, investigate WHY it happened. Don't stop at surface causes.

Root cause investigation questions:

• What was the root cause? Not "accuracy dropped" but "why did accuracy drop?" Was it data drift? A recent model update? Change in supplier behavior? Market condition?
  - Why didn't monitoring catch this sooner? Was the alert threshold too high? Was monitoring not configured for this metric? Did we have a monitoring gap?
  - What should we have caught in testing that we missed? Did we test edge cases? Did we test failure modes?
  - What will we do differently in the future to prevent this?

Key principle: Blameless investigation. The goal is learning, not blame. You want people reporting problems early, not hiding them. "What went wrong?" not "Who made the mistake?"

Phase 4: Remediation and Learning (Days 2-7 post-incident)

What happens: You document changes to prevent recurrence. This is how you get smarter with each incident.

Changes might include:

• Technical: Retraining the model, fixing data pipeline, improving data quality monitoring, adding new alerts
  - Process: Faster detection (lower alert thresholds), faster response (better runbooks), manual fallback practice
  - Governance: Should we have caught this in approval? Should we have a different control?

Retrospective meeting: Within one week of incident, hold a blameless retrospective. Attendees: incident responders, system owner, exec sponsor (Severity 1 only), risk/compliance rep. Agenda: What happened? What was the root cause? What are we changing? By when? Who owns each change? Goal: Everyone understands what happened and leaves knowing how it's being prevented.

Documentation: Update incident playbook with lessons learned. If this failure mode is relevant to other systems, share it. Update monitoring thresholds. Update testing procedures. These changes accumulate and make your systems more resilient.

The AI Incident Response Playbook: Your Manual

Create a detailed runbook for each of your major AI systems. This becomes your team's bible during a crisis. When panic sets in, the playbook tells them exactly what to do.

Playbook sections:

• System overview: What does it do? How critical is it? How many transactions per day? What's the business impact if it's down?
  - Monitoring and alerting: What metrics indicate problems? What are the alert thresholds? Where do alerts go (who gets paged)?
  - Key contacts: System owner (name, phone, email), primary engineer, backup engineer, escalation contact (operations leader), executive sponsor, vendor support (if applicable). Include how to reach them at 2 AM on Sunday.
  - Common failure modes and responses: For each likely failure, spell out the response:

If accuracy degrades: Check data freshness and quality. Review recent changes to the model or data. Check for data drift. Retrain if needed. Monitor accuracy recovery.
- If system is down: Check infrastructure (is it accessible?). Activate fallback process. Call infrastructure team. Check vendor status (if cloud-hosted). Estimate recovery time based on typical issues.
- If bias is discovered: Pause autonomous execution immediately (switch to manual review if possible). Assess the scope and business impact. Notify legal and compliance. Don't keep making the biased decisions while investigating.
- If escalation queue backs up: Bring in additional reviewers if possible. Slow down transaction input if possible. Communicate delays to users. Escalations sitting in queue for days is a crisis.
- If vendor is unreachable: Follow your contingency plan. Do you have a backup vendor? Can you revert to an older version? What's the timeline?

• Escalation paths: For Severity 1: notify COO immediately. For Severity 2: notify operations leader. For Severity 3: notify system owner.
  - Decision trees: Flowcharts for common scenarios: "System is down. Is it a data problem or infrastructure problem?" "Accuracy dropped. Is it data quality or model degradation?"
  - Fallback procedures: How do you operate without the AI system? Step-by-step procedure. Who does what?
  - Communication templates: (see next section)
  - Recovery procedures: How do you validate that the system is working correctly after you've fixed it? Checklist of things to verify.
  - Post-incident process: When to hold retrospective (within one week). Who attends. How to document lessons learned.

Maintenance: Review and update quarterly. After each incident, update the playbook with what you learned. "We didn't have a procedure for this scenario, add one." "The person we called wasn't available at 2 AM, update contacts."

Tip: Run tabletop exercises quarterly. Gather your incident response team (system owner, engineer, operations leader, exec sponsor) and walk through a hypothetical scenario using your playbook. "It's Tuesday at 3 AM. System just started making wild decisions. You're on call. What's the first thing you do?" Run through the playbook. This surfaces gaps before you face a real crisis with real business impact.

Fallback Processes and Business Continuity

Every AI system that handles critical decisions needs a documented fallback process. This is your insurance policy against system failure.

Good fallback example: Procurement AI

Normal operation: Requisition → AI recommendation → Buyer reviews and approves → PO issued

Fallback operation: Requisition → Buyer uses sourcing rules from procedure document to identify supplier → Buyer approves supplier → PO issued

The fallback is slower (manual process takes 2x as long) but keeps business moving if AI is unavailable.

Bad fallback example: "When system is down, all requisitions are on hold until system is restored." This isn't a fallback. It's operational paralysis. You've made the business completely dependent on the AI system. This is a governance failure.

Why fallback processes matter: When the AI system fails and your team panics, the fallback process tells them: "Switch to this, investigate the problem on the side, keep the business running." It prevents a technical problem from becoming an operational crisis.

How to maintain fallback readiness: Practice the fallback monthly. Have your team execute the manual process once a month even though the system is working fine. This keeps their skills sharp. If your team hasn't executed the manual process in 6 months, they've forgotten how. When the real emergency happens, they'll be fumbling in the dark instead of executing smoothly.

Pre-Written Communication Templates

During a crisis, writing clear communication is hard. You're stressed, moving fast, trying to fix the problem. Pre-written templates ensure your communication is accurate and professional.

Internal team communication (Severity 1, to team working around the incident):

"The [System Name] is currently unavailable due to [brief description of problem]. We've activated our fallback process. Please use [fallback process] for all new [transactions]. Check #[slack-channel] for status updates every 15 minutes. Thank you for your patience."

Executive communication (Severity 1, to COO/steering committee):

"At [time], [System Name] went offline due to [root cause]. Current impact: approximately [number] transactions unable to be processed. We've activated our fallback process and are processing manually. Engineering team is investigating. ETA to restoration: [timeframe]. Next update: [when]. Our team has this under control."

Post-incident communication (all hands, after system is restored):

"On [date] at [time], the [System Name] experienced [brief description]. This resulted in [impact]. Our team quickly activated our fallback process and restored the system by [time]. Root cause: [cause]. To prevent this from happening again, we are implementing [changes]. We apologize for any disruption and appreciate your patience."

These templates take 30 seconds to fill in and ensure your communication is clear, accurate, and professional instead of panicked and vague.

What to Do Monday Morning

• Identify your highest-risk AI systems. List all AI systems you're using in operations. For each, assess: "If this failed, what's the business impact?" Rank them. The top 3-5 are your priority for incident response planning.
  - Define severity classification thresholds for your context. Use the framework provided but adjust for your business. What accuracy level is unacceptable for your system? What escalation volume exceeds your capacity? Document these criteria so classifications are consistent.
  - Document fallback processes for critical systems right now. For each critical AI system, write down: "If this system is unavailable, how do we operate?" Be specific: steps, who does what, timeline. Make it repeatable and testable.
  - Create incident response playbooks for your top 3 highest-risk systems. Use the template provided in this lesson. Include system overview, monitoring/alerting, key contacts (get real phone numbers and test them), common failure modes with specific responses, escalation paths, decision trees, fallback procedures, communication templates, and recovery checklist.
  - Pre-write communication templates for your organization's standard incident scenarios. Create templates for: internal team updates, executive escalation, post-incident all-hands communication. Make them easy to customize quickly.
  - Identify your incident response team members. Who's on call for Severity 1 incidents? Who has authority to activate fallback processes? Who communicates to executives? Get explicit commitments and contact information. Test that you can actually reach people at 2 AM.
  - Schedule your first quarterly tabletop exercise. Gather incident response team (system owner, engineer, ops leader, executive sponsor). Pick a scenario. Walk through it using your playbook. This surfaces gaps before a real incident.
  - Plan monthly fallback process testing. Pick one critical AI system each month and have your team execute the manual fallback process for real, even though the AI system is working. This keeps skills sharp.
  - Create an incident logging system. Every incident (Severity 1-4) gets logged. What happened? When? Impact? Root cause? What changed? After 6 months, analyze the log. Patterns will emerge. "We always have data quality issues on Mondays", fix the root cause.

Key Takeaways

• Prepare for AI failures before they happen, not during the crisis when people are panicking.
  - Classify incidents by severity so your response level matches the problem (avoid alert fatigue and resource misallocation).
  - Identify common failure modes before they happen so your team knows what could go wrong.
  - Build detailed playbooks for critical AI systems with system overview, monitoring, contacts, failure responses, escalation, communication, and recovery.
  - Document fallback processes for every critical system and test them monthly to keep skills sharp.
  - Pre-write communication templates so during a crisis you're communicating clearly, not improvising.
  - Follow the four-phase incident response model: activation, containment and communication, investigation and root cause analysis, remediation and learning.
  - Conduct blameless post-incident retrospectives within one week, focused on learning and improvement.
  - Run tabletop exercises quarterly to keep your incident response team sharp and ready.
  - Log every incident and analyze patterns over time; patterns reveal systemic issues worth fixing.