Hook

At 3:47 AM on a Tuesday, your anomaly detection system starts flagging everything in your production infrastructure as suspicious. The system is working exactly as designed. It's just detecting a distribution it's never seen before. Your incident response playbook doesn't have a step for "the AI is broken but we don't know how broken." Your on-call engineer tries to turn off the alerting, but the system is auto-remediating based on its flags, every alert triggers automated rollback of changes that weren't actually faulty. By 5:15 AM, you've artificially broken your own infrastructure trying to contain a system designed to prevent exactly that. This is the gap between incident response that assumes human-understandable failure modes and incident response in the age of AI.

Purpose

Traditional incident response is built on a simple model: a system either works or it fails. A database is up or down. An application is responding or not. A security vulnerability is either triggered or it isn't. This model breaks down with AI systems. They don't fail. They degrade. They don't fail all-at-once. They fail asymmetrically, working perfectly for 99% of inputs and catastrophically for edge cases. They don't fail for reasons your on-call engineer can understand in five minutes, the failure mode might be "the production data distribution shifted from training data" and the fix might be "retrain the model," which takes hours.

You need incident response protocols that account for how AI systems actually fail: detection methods that catch subtle degradation before customers notice, containment strategies that can isolate a failing AI system without breaking dependent processes, investigation frameworks that identify whether the problem is data, model, or decision logic, and recovery processes that can revert to pre-AI operations when necessary.

This lesson builds an AI incident response playbook that integrates with your existing IRP, establishes clear thresholds for when AI systems need human intervention, and trains your team to investigate AI failures systematically rather than reactively.

Why This Matters

For Your On-Call Engineers: They need to know: Is this an AI system failure or an infrastructure failure? If the anomaly detection system is wrong, how do I turn it off without breaking monitoring? If the chatbot is hallucinating, do I escalate or contain? An AI-aware incident playbook answers these questions in the moment, under pressure, at 3 AM.

For Your SLAs and Compliance: Your customers expect a certain incident response time. If an AI system fails and it takes your team two hours to even diagnose the failure (is it the model, the data pipeline, the infrastructure?), your SLA suffers. More critically, if an AI system violates a compliance requirement and you don't detect it quickly, you have longer exposure. Regulators ask: how fast did you detect the violation, and how quickly did you contain it?

For Your Board: AI incidents become news quickly. An AI system that makes biased decisions affecting customers, a chatbot that leaks sensitive information, an automated system that executes harmful actions without human override. These are reputation and legal events, not just operational incidents. Your board wants to know you can detect and contain these fast.

For Your Vendor Relationships: If your incident is caused by a third-party AI system (a SaaS platform, a pretrained model, an API), you need to know quickly so you can escalate to the vendor and understand your liability. Your playbook should include vendor escalation procedures.

Core Concepts

Key Insight 1: AI Failures Have Unique Detection Requirements

Traditional system failures are easy to detect: latency spikes, error rates exceed threshold, disk space fills, database replication lags. You've built monitoring for these. But AI systems fail in ways your current monitoring may not catch:

Silent degradation: A model's accuracy drifts from 95% to 88% over three months. The system is still processing requests and returning results. There are no error logs. Your latency is normal. Your infrastructure metrics are healthy. But the quality of decisions is declining. This goes undetected until a customer complains or an audit reveals it.

Asymmetric failure: A model trained on balanced data suddenly sees imbalanced inputs. It handles the majority class fine but fails catastrophically on the minority class. Your aggregate accuracy metrics look okay (because the majority class dominates), but your fairness metrics are terrible.

Slow cascade: Your demand forecasting model starts hallucinating demand in a particular segment. Supply chain teams trust the model, so they order inventory based on false forecasts. The overstock takes weeks to surface in financial reports. By then, the model has contaminated decision-making across the supply chain.

Unexpected failure modes: A chatbot trained to be helpful starts being helpful in unexpected ways, bypassing security controls because the training data had examples of admins bypassing controls and asking the system to do something "helpful." You didn't build tests for this because you didn't anticipate this failure mode.

Traditional monitoring (uptime, latency, error rates) won't catch any of these. You need AI-specific monitoring:

• Model accuracy degradation: Compare current model predictions against holdout test set or recent ground truth. Alert if accuracy falls below threshold.
  - Data distribution shift: Monitor statistical properties of input data. Alert if mean, variance, or distribution shape diverges from baseline.
  - Prediction variance: Monitor the range and distribution of model outputs. Alert if the model suddenly starts making predictions outside its normal range.
  - Fairness metric degradation: Monitor accuracy or error rates across demographic groups. Alert if fairness metrics diverge.
  - Confidence interval shrinkage: If your model provides confidence scores, monitor whether confidence is correlated with accuracy. Alert if the model is confident but wrong.
  - Explainability audit: For high-stakes decisions, sample and manually audit the model's reasoning. Alert if the reasoning becomes incoherent or unexplainable.

Key Insight 2: Incident Detection Must Be Automated and Continuous

You cannot rely on customers reporting AI failures. By the time a customer reports an issue, the system has potentially made hundreds of faulty decisions. You need continuous, automated detection that alerts within hours (or minutes for high-stakes systems) of failure.

Example detection thresholds:

• Demand Forecasting Model: Daily accuracy validation against holdout test set. Alert if accuracy drops below 85% or shows negative trend over 5 days.
• Anomaly Detection System: Daily comparison of model-flagged anomalies against actual incidents (precision and recall). Alert if precision drops below 70% or recall drops below 80%.
• Chatbot System: Random sampling of 100 conversations per day; human review for hallucinations, security violations, or out-of-distribution responses. Alert if >5% show problems.
• Bias-Sensitive System (hiring, lending): Daily fairness audit on protected characteristics (if legal). Alert if false positive or false negative rates diverge by >5% between groups.

These thresholds require:

1. Automated data collection: Your system logs enough data to compute these metrics without manual effort
2. Automated comparison: Daily or continuous comparison against baselines
3. Alert routing: When a threshold is breached, alert the appropriate team (data science, ops, compliance)
4. Runnable alerts: The alert must trigger a response, not sit in a dashboard

Key Insight 3: Incident Containment for AI Systems Requires Different Strategies

When a traditional system fails, containment is usually clear: take the system offline. Stop processing requests until you fix it. But AI systems are often deeply integrated into workflows. Taking the system offline means taking dependent processes offline too.

Your containment strategies need to be more nuanced:

Degrade to Previous AI Model or Version

If the new model is degrading, switch back to the previous model version. This is fast (minutes) and low-risk (the previous version was in production and working). Requires: model versioning infrastructure, automated rollback, validation that previous version still works.

Fallback to Rules-Based or Non-AI Process

If the AI system is failing, switch to a pre-AI process. For demand forecasting, this might be "use last quarter's forecast as the baseline." For incident detection, it might be "alert on predefined rule-based thresholds." This is slower (hours to implement) but keeps operations moving. Requires: pre-AI process is documented and can be implemented quickly, fallback process is acceptable to business (even if lower quality).

Containment at the Decision Level

If the AI system makes recommendations but humans make decisions, contain at the human level. Stop automatically approving high-confidence recommendations. Require human review for all recommendations during the incident. This is lower-risk (humans catch bad decisions) but increases latency and workload. Requires: human review infrastructure exists and has capacity.

Partial Containment

If the model fails only for certain types of inputs or customers, contain the blast radius. Route requests to subpopulations that the model still handles well to the AI system. Route problematic requests to the fallback process. This requires you can detect which inputs are problematic (hard but necessary).

Selective Shutdown

If shutting down the entire system is too disruptive, shut down the system for problematic use cases only. Turn off auto-remediation for the anomaly detection system, but keep alerting on. Disable recommendations for the problematic customer segments, but keep them for others.

Your incident playbook should list these strategies in order of preference for each AI system. When an incident is detected, the on-call engineer chooses the strategy that minimizes blast radius while keeping operations moving.

Key Insight 4: Investigation Requires Isolating the Three Failure Categories

When an AI system fails, the failure is always in one of three places:

Data Failure: The model is broken (not its problem). The data changed, and the model wasn't designed for the new data. Examples: upstream data format change, data distribution shift, data quality degradation, missing values, corruption in the data pipeline.

Investigation steps:

1. Check data freshness: is the training data stale? (If model was trained on 2024 data and it's now 2025, yes.)
2. Check data quality: does a sample of recent data look normal compared to training data? Statistical tests: mean, variance, distribution shape, missing value rates compared to baseline.
3. Check data lineage: did any upstream data sources change recently? Did a supplier change data format? Did a system start sending different values?
4. If you find data changes: does the model need retraining? Or is the data change expected and the model just hasn't seen it yet?

Model Failure: The data is fine. The model has a bug or wasn't designed for production conditions. Examples: model training had a bug, model hyperparameters are suboptimal, model was fit to a different data distribution than production, adversarial inputs are fooling the model.

Investigation steps:

1. Check model version and training: when was the model last trained? On what data? With what hyperparameters?
2. Check model architecture: is the model's assumption about the data still valid? Is the model designed for this type of data?
3. Check for adversarial inputs: are there specific types of inputs that cause the failure? (Plot accuracy by feature value to see if there's a failure mode.)
4. If you find a model issue: can you retrain the model with current data? Or do you need to change the model architecture or approach?

Decision Failure: The model is working as designed. The problem is how the output is being used. Examples: model's confidence score is being ignored, model recommendations are being followed without human review, model output is being applied to a use case it wasn't trained for.

Investigation steps:

1. Check decision logic: how is the model output being used? Is there a decision rule (e.g., "approve if confidence > 90%") that's too aggressive?
2. Check human override: are humans reviewing model recommendations? Are they overriding them when needed?
3. Check use case drift: is the model being applied to a different use case than it was trained for? (Model trained on US hiring data now being used for international hiring?)
4. If you find a decision issue: change the decision logic, require human review, or restrict use case.

Your investigation playbook should guide the on-call engineer through these questions systematically. Diagnosis should take 30-60 minutes, not days.

Key Insight 5: Recovery Requires Prepared Fallbacks and Documented Processes

Recovery from an AI system failure means one of two things:

Short-term recovery (hours): Get the system operational again by switching to an older model version, activating a fallback process, or increasing human review. The system is working again, but not at full capacity or quality. The goal is to prevent further damage while you investigate.

Long-term recovery (days to weeks): Understand the root cause, fix it, validate the fix, and gradually restore the system to full operation.

Your recovery plan should include:

• Model rollback procedure: How quickly can you revert to a previous model version? (Should be minutes. If it's hours, your deployment infrastructure is too slow.)
  - Fallback process activation: What's the non-AI process that replaces the system if needed? Is it documented? Can it be activated in 15 minutes?
  - Data repair procedure: If data was corrupted, how do you clean it? How do you retrain the model with clean data?
  - Model retraining procedure: How long does retraining take? Do you have a hold-out test set to validate the retrained model?
  - Gradual restoration: Once recovered, how do you gradually restore the system to production? Do you run the new and old systems in parallel? Do you route a percentage of traffic to the new system while monitoring?

Your playbook should include time estimates for each step so the incident commander can set realistic expectations.

Key Insight 6: Integration with Existing IRP

Your organization already has an incident response process. It probably includes:

• Detection and Alert: Someone detects something is wrong and raises an alarm
  - Triage and Severity: What's the severity? (Critical, High, Medium, Low)
  - Incident Commander Assigned: One person is in charge
  - Containment: Stop the problem from getting worse
  - Investigation: Root cause analysis
  - Resolution: Fix the problem
  - Communication: Update stakeholders
  - Post-Incident: Document what happened, prevent recurrence

Your AI incident response playbook should plug into this existing process at each stage:

• Detection: AI-specific monitoring alerts (model degradation, data drift) trigger the standard alert process
  - Triage: Incident commander uses AI-specific severity criteria (scope of AI impact, customer-facing vs. internal, reversibility)
  - Containment: Incident commander chooses from AI-specific containment strategies (rollback, fallback, human review)
  - Investigation: On-call team uses AI-specific investigation framework (data, model, decision failure)
  - Resolution: Implement recovery procedure (model retraining, data repair, decision logic change)
  - Post-Incident: Review lessons learned, improve monitoring, update playbook

The AI playbook doesn't replace your existing IRP. It extends it with AI-specific details.

Practical Use Cases

Use Case 1: Demand Forecasting Model Fails in Production, Detection and Containment

The Scenario: Your demand forecasting model is used by supply chain teams for inventory planning. The model makes one forecast per day, 100+ SKUs, and the forecast is published by 6 AM for the day's planning.

At 4:47 AM, your automated model monitoring detects that the model's accuracy (compared against holdout test set) has dropped from 92% to 76% overnight. This is a 16-point drop, way below the 85% threshold.

Detection Workflow:

1. Automated alert triggers: "Demand Forecasting Model Accuracy Alert: Current 76%, Threshold 85%"
2. Alert is routed to on-call data scientist and supply chain lead
3. On-call data scientist is paged (4:52 AM)
4. Data scientist checks:

• Model version: Updated 2 days ago (normal)
• Recent input data: Checking data quality...
• (Finding) Recent data has 15% missing values in a key supplier field (normal 0.5%)
• (Finding) A supplier changed their data format two days ago; the parsing is missing values

Triage (5:07 AM):

• Severity: High (forecast must be published by 6 AM; all-day impact if missed)
• Impact: Supply chain planning will use incorrect forecasts for 24+ hours if not fixed
• Reversibility: Partial (supply chain can use previous day's forecast as fallback)
• Incident Commander assigned (supply chain director, on call)

Containment Decision (5:12 AM):

• Option 1: Fix the data parsing, retrain the model (2+ hours, risky, won't make 6 AM deadline)
• Option 2: Use yesterday's forecast as today's forecast (5 min, lower quality, but safe)
• Option 3: Use previous two weeks' average as today's forecast (10 min, lower quality, but accounts for trend)
• Decision: Implement Option 3. Publish average-based forecast at 5:45 AM. Mark forecast as "degraded, using average due to upstream data issue."
• Long-term: Fix data parsing, retrain model, validate accuracy, restore to production by end of day

Recovery (5:45 AM):

• Forecast published to supply chain team with note: "Forecasts prepared using average of recent data due to upstream data quality issue. Expect revised forecasts by end of day."
• Data engineering team investigates supplier data format change
• By 10 AM: Data parsing fix deployed, model retrained on fixed data
• By 11 AM: Model accuracy validated (restored to 91%), retrained model deployed
• By 12 PM: New forecast published with high-quality model output
• By 2 PM: Incident closed, post-incident review scheduled

Key lesson: The incident went from detection (4:47) to containment (5:45) in less than an hour because the team had a documented fallback process and clear thresholds for when to activate it. Supply chain planning was never completely blind. They had a degraded but functional forecast.

Use Case 2: Anomaly Detection System Fails, False Alerts and Auto-Remediation Loop

The Scenario: Your NOC uses an anomaly detection system to flag unusual patterns in infrastructure. The system triggers alerts and, for non-critical services, auto-remediates by rolling back recent changes.

At 2:15 AM, the system starts flagging everything as anomalous. The infrastructure is fine, the model has just never seen this pattern before (a new cloud region came online with different baseline metrics).

The Problem:

• The model starts flagging normal behavior in the new region as anomalies
• Each anomaly triggers an auto-remediation workflow
• Auto-remediation rolls back recent changes
• Rolling back changes causes different patterns
• The model flags the rollback as another anomaly
• Each rollback causes more anomalies
• The system enters a loop: anomaly → rollback → new anomaly → new rollback

By 3:45 AM, you've artificially broken infrastructure that was working fine.

Detection (2:15 AM):

• Alert storm: 200+ anomaly alerts in 5 minutes (normal: 3-5 per day)
• Pattern: All alerts in the new region
• Automated detection: "Anomaly Detection Alert Storm, Region us-east-2"

Triage (2:22 AM):

• Severity: Critical (auto-remediation is degrading infrastructure)
• Decision: Disable auto-remediation immediately, keep alerting on

Containment (2:25 AM):

• Disable auto-remediation for anomaly detection system
• Change system to "alert only", humans review before taking action
• This stops the remediation loop immediately
• Infrastructure stabilizes

Investigation (2:30 AM onwards):

1. Data Science team reviews anomaly alerts: all from us-east-2
2. Compare infrastructure metrics in us-east-2 vs. us-west-1:

• CPU: us-east-2 averages 45%, us-west-1 averages 35% (different workload mix)
• Memory: us-east-2 has high variance, us-west-1 is stable (bursty workload)
• Disk I/O: completely different profiles

1. Finding: The model was trained on us-west-1 and us-center-1. It has never seen the metric profile from us-east-2. Everything us-east-2 does is "anomalous" by definition.
2. Root cause: Model training data didn't include all regions

Recovery (by 4:00 AM):

• Retrain model on baseline data from all regions
• Validate accuracy on holdout test set including all regions
• Deploy retrained model
• Gradually restore auto-remediation: first for low-risk actions, monitor for accuracy, then restore fully
• Post-incident: add new regions to model training pipeline before they go to production

Key lesson: The system was designed well (it had an "alert only" mode, human override was possible), but the on-call team didn't know to flip that switch immediately. After this incident, the runbook is: "If you see alert storm (>50 alerts/5 min), immediately switch system to 'alert only' mode. Ask questions later." This prevents feedback loops.

Use Case 3: Bias in Hiring Recommendation System, Slow Detection, Regulatory Exposure

The Scenario: Your hiring team uses an AI system to recommend qualified candidates. The system works by learning from your historical hiring data (who you've hired and promoted).

For six months, no one notices that the system is recommending women at lower rates than your applicant pool would suggest, because:

1. The system's aggregate accuracy is good (78%)
2. Your aggregate hiring numbers look balanced
3. But the system has learned that your company historically promoted men faster
4. It's making decisions consistent with your past bias, not with the present data

The issue surfaces only when an employment attorney asks for your model's decision log as part of discovery in a lawsuit.

Detection (Delayed, 6 months):

• Legal team receives EEOC complaint alleging AI hiring discrimination
• Legal asks for model decision logs and fairness audit
• Your fairness audit (never done before) reveals: women recommended at 42% rate vs. 52% in applicant pool
• System has been making discriminatory recommendations for 6 months

Why Detection Was Delayed:

• No automated fairness monitoring (was running, but no one was checking it)
• No regular audits (should be monthly for bias-sensitive systems)
• No detection threshold (should have alerted when fairness metrics diverged)

Triage (Incident Now):

• Severity: Critical (legal exposure, regulatory exposure, reputational exposure)
• Scope: 6 months of potentially discriminatory hiring decisions
• Reversibility: Impossible (hiring decisions are made, people are hired/not hired, offers cannot be unwound)

Containment (Immediate):

• Disable AI hiring recommendations immediately
• Switch to manual resume review for all open positions
• Audit all hiring recommendations from past 6 months
• Flag candidates who were not recommended but should have been for reconsideration

Investigation:

1. Data Scientists review training data: your historical hiring data reflects past bias
2. Model learned patterns from biased historical data
3. Root cause: You trained the system on your historical hiring data without debiasing the data or monitoring fairness
4. Secondary root cause: No one was monitoring fairness metrics
5. Tertiary root cause: No one knew bias monitoring was a requirement

Recovery (Long-term):

• Retrain model on debiased historical data (remove or reweight biased patterns)
• Implement monthly fairness audit across gender and race
• Alert if fairness metric diverges >5%
• Require human review of all hiring recommendations (system becomes decision support, not decision maker)
• Audit all hiring from past 6 months; offer interviews to candidates discriminated against
• Settlement negotiations with EEOC

Post-Incident:

• Legal liability (settlement, potential fines)
• Reputational damage (story in media)
• Process change: Monthly bias audits for all systems making people-related decisions
• Policy change: AI systems are decision support only, not decision makers, for hiring, lending, criminal justice

Key lesson: The failure wasn't the model. It was the lack of detection. You were running fairness metrics, but no one was checking them. And you were unaware that fairness monitoring is a compliance requirement. After this incident, the playbook is: "Every AI system making decisions affecting protected classes must have automated weekly fairness audits. Alert if any metric diverges. Investigate within 24 hours."

Examples

Example 1: AI Incident Response Playbook Template

System: Demand Forecasting Model

Owner: Supply Chain Analytics

On-Call: Data Scientist (primary), Demand Planning Manager (secondary)

Step
Detection
Action
Owner
Time

Detect
Daily 6 AM accuracy validation triggers alert if accuracy < 85% or negative trend over 5 days
Page on-call data scientist
Automated
Immediate

Triage
Data scientist checks: model version, data freshness, upstream data quality
Classify as High/Medium/Low based on: scope (SKUs affected), impact (supply chain can't plan?), reversibility (can use fallback?)
Data Scientist
10-15 min

Contain (Option A: Revert Model)
If model recently updated and accuracy degraded: revert to previous version
Roll back model deployment in production; validate accuracy restored
Data Scientist
15-20 min

Contain (Option B: Use Fallback)
If data issue detected: activate fallback process
Use previous day's forecast or 7-day average; notify supply chain team; mark forecast as "degraded"
Supply Chain Manager
5 min

Contain (Option C: Increase Review)
If issue is unclear
Require manual review of all forecasts before publication; increase forecast buffer
Supply Chain Manager + Data Scientist
20 min

Investigate
Determine root cause: data issue vs. model issue vs. decision logic
Check data quality, model training recency, upstream source changes, decision thresholds
Data Scientist
30-60 min

Resolve
Fix root cause
Retrain model, fix data pipeline, change decision logic, update thresholds
Data Scientist + Data Engineering
2-8 hours

Validate
Confirm fix worked
Run holdout test accuracy, validate on recent data, compare to baseline
Data Scientist
30-60 min

Restore
Gradually restore to normal operation
Deploy fixed model; run new model parallel with old for 24 hours; monitor accuracy; switch fully to new model
Data Scientist
24+ hours

Communicate
Keep stakeholders informed
Initial alert (detection), 15-min update (triage), 30-min update (initial containment), hourly updates (investigation), final resolution update
Incident Commander
Ongoing

Post-Incident
Prevent recurrence
Review what was missed in monitoring, improve alert thresholds, update playbook, train team
Data Science Lead
1-2 weeks

Example 2: AI Incident Severity Matrix

Severity
Definition
Example
Response Time
Escalation

Critical
AI system failure affecting customers, revenue, or compliance; irreversible decisions; system must be contained immediately
Bias in hiring system discovered; model making security violations; forecast system producing negative demand
Triage: 15 min, Contain: 1 hour
CIO, Chief Risk Officer, Legal

High
AI system degradation affecting operations or decision quality; reversible with fallback process; can be contained within hours
Model accuracy drops 15%; data quality degradation; fairness metrics diverge; false positive storm in anomaly detection
Triage: 30 min, Contain: 4 hours
VP Engineering, Product Lead

Medium
AI system showing warning signs; monitoring alerts; no customer impact yet; can be investigated during business hours
Model accuracy drifting slowly (3% over month); new data distribution emerging; fairness metrics showing early divergence
Investigate: business hours, Fix: 1-2 weeks
Team Lead, Product Owner

Low
Expected behavior or monitoring artifact; non-critical system; can be resolved opportunistically
Model confidence scores shifting but accuracy stable; non-critical alert spam; historical data retraining
Investigate: scheduling basis
Team (no escalation)

Example 3: Incident Communication Template

Time: 4:52 AM - Initial Alert

"ALERT: Demand Forecasting Model Accuracy has dropped to 76% (threshold 85%). On-call data scientist paged. Estimated impact: forecast publication delayed or degraded."

Time: 5:12 AM - Triage Update

"TRIAGE COMPLETE: Incident classified as High severity. Root cause identified: supplier data format change on 2025-04-08 causing missing values. Recommendation: use previous day's forecast as fallback (lower quality, but available on time). Alternative: delay forecast by 3 hours for model retraining. Supply chain director deciding now."

Time: 5:45 AM - Containment Update

"CONTAINED: Forecast published using average-based fallback. Supply chain team notified of degraded forecast quality. Data engineering investigating supplier data format change. Estimated resolution time: 6 hours. Next update: 11 AM."

Time: 12:30 PM - Resolution Update

"RESOLVED: Supplier data parsing fixed, model retrained and validated, new high-quality forecast published at 12:15 PM. Model accuracy restored to 91%. Incident closed. Post-incident review scheduled for Friday."

Example 4: AI Failure Investigation Decision Tree

AI System is Failing?
├─ Accuracy is degraded?
│ ├─ YES → Is data fresh? (When was model last trained?)
│ │ ├─ Data is old (>30 days) → Retrain model (DATA ROOT CAUSE)
│ │ ├─ Data is fresh → Check data quality
│ │ ├─ Data quality is bad (missing values, distribution shift) → Fix data, retrain (DATA ROOT CAUSE)
│ │ ├─ Data quality is good → Check model for overfitting or bugs (MODEL ROOT CAUSE)
│ └─ NO → Continue below
├─ False alert storm?
│ ├─ YES → Is system seeing unexpected inputs? (New data distribution, new region, new customer type)
│ │ ├─ YES → Model not trained on this distribution → Retrain on broader data (MODEL ROOT CAUSE)
│ │ ├─ NO → Check for adversarial inputs or model bugs (MODEL ROOT CAUSE)
├─ System making wrong decisions despite good predictions?
│ ├─ YES → Check decision logic and thresholds (DECISION ROOT CAUSE)
│ ├─ Confidence thresholds too low? → Increase thresholds (DECISION ROOT CAUSE)
│ ├─ Human override not working? → Check override workflow (DECISION ROOT CAUSE)
└─ If no root cause found after 60 minutes → Escalate, activate fallback, and investigate longer-term

Anti-Patterns

Anti-Pattern 1: "We'll Handle AI Incidents Like Regular Incidents"

Why This Fails: Traditional incident response assumes you can diagnose the problem in minutes. An on-call engineer can usually figure out "the database is down" or "the service is throwing errors" quickly. But "the model is degrading" or "the data distribution shifted" requires investigation, statistical analysis, data quality checks, model profiling. The diagnosis takes hours, not minutes. If you use your standard incident response timeline (diagnose in 5 min, contain in 15 min), you won't have time to even understand what happened.

What to Do Instead: Extend your incident response timeline for AI systems. Triage happens within 30 minutes (not 5). Containment happens within 4 hours (not 15 min). Investigation can take hours or days. Your playbook should acknowledge this extended timeline and give on-call engineers permission to activate fallback processes while investigation is ongoing, rather than waiting for a diagnosis.

Anti-Pattern 2: "We'll Detect AI Failures When Customers Complain"

Why This Fails: By the time a customer reports an issue, the damage is done. An AI system has been making bad decisions for days or weeks. You have no early warning. You have no ability to contain before impact. You're reactive, not proactive.

What to Do Instead: Implement automated, continuous detection for AI-specific failure modes. Daily accuracy monitoring. Weekly fairness audits. Continuous data quality checks. Automated alerts when metrics diverge from baseline. This is not optional. It's a core operational capability, like disk space monitoring or database replication lag monitoring. If you don't monitor it, you won't know when it fails.

Anti-Pattern 3: "The Data Science Team Can Fix Any AI Incident"

Why This Fails: AI incidents often require cross-functional response. A data quality issue requires data engineering (to fix the pipeline). A model degradation might require data science (to retrain) or business stakeholders (to adjust thresholds). A fairness violation requires legal and compliance. A chatbot hallucination requires product, security, and data science. If you assume the data science team can handle everything, you'll be slow to resolve and you'll miss critical expertise.

What to Do Instead: Establish cross-functional incident response for AI systems. On-call rotations should include data engineering, data science, and product. The incident commander role is critical. This person ensures all needed expertise is mobilized. Your playbook should specify who gets called for each type of AI failure.

Anti-Pattern 4: "We Don't Have a Fallback. We'll Just Fix the AI System"

Why This Fails: If your AI system is the only way your business operates, and the system fails, your business stops. Fixing the system might take hours or days. You have no ability to continue operations while fixing.

What to Do Instead: For critical AI systems, establish fallback processes. Not as a permanent alternative (the AI system is better), but as an emergency process to keep operations moving while you fix the AI system. A non-AI forecasting process, a rule-based alert system, manual review workflows. These fallbacks reduce the cost of AI failures from "business stops" to "business operates at reduced quality for a few hours."

Anti-Pattern 5: "Our Playbook is Perfect; We Don't Need to Update It"

Why This Fails: Your first few AI incidents will reveal gaps in your playbook. You'll discover detection mechanisms you didn't anticipate needing. You'll find containment strategies that don't work for your systems. You'll uncover communication failures. A playbook that never changes is a playbook that's not being tested against reality.

What to Do Instead: Treat post-incident reviews as opportunities to improve your playbook. After every AI incident, review: Did we detect this fast enough? Could we have detected faster? Did our containment strategy work? What should we have done differently? Update the playbook. Test the playbook. In six months, you should have learned enough to make significant improvements.

Human Judgment Checkpoints

Checkpoint 1: When to Activate Fallback vs. When to Investigate

Your on-call engineer detects an AI system failure. Should they immediately activate the fallback process, or should they investigate first?

The answer depends on: (1) how long can the system stay degraded before business impact is severe? (2) how certain are you that the fallback is safe? (3) can you investigate while the fallback is active?

If the answer to (1) is "minutes" (a real-time trading system, a security detection system), activate fallback immediately and investigate in parallel. If the answer to (1) is "hours" or "days" (a batch forecasting system, a recommendation engine), you might investigate first and only activate fallback if diagnosis takes too long.

The key human judgment: Don't wait for perfect information. If you're uncertain after 15-30 minutes of investigation, activate the fallback. Better to operate at reduced quality than to keep digging while the system fails.

Checkpoint 2: When an AI Incident Is Also a Compliance Incident

Not all AI failures are equal. Some are operational (a model is degrading, affecting forecast quality). Some are compliance violations (a bias in a hiring system, a security vulnerability in a chatbot).

When you detect a compliance violation, escalate immediately to legal and compliance, regardless of how early your diagnosis is. Don't wait for investigation to complete. Don't try to fix it internally first. Compliance violations have regulatory implications and statute-of-limitations implications. Early detection and reporting is legally important.

Your on-call process should have a question: "Could this incident be a compliance violation?" If yes, page your Chief Risk Officer or General Counsel immediately.

Checkpoint 3: Speed of Recovery vs. Quality of Fix

You're recovering from an AI failure. You have two options:

1. Quick fix (1 hour): Deploy a temporary fix that gets the system working but may not be durable (e.g., disable auto-remediation but require manual review of every decision)
2. Proper fix (8 hours): Investigate root cause thoroughly, fix it completely, validate, deploy with confidence

The human judgment: Do you need the quick fix immediately? Or can you operate on the temporary fix for a day or two while you do the proper fix?

Most of the time, the answer is "do the quick fix now, do the proper fix later." This reduces customer impact and gives you time to do thorough investigation. But sometimes (a security vulnerability, a compliance violation), you need to fix it properly before operating at all.

Make this decision with your incident commander and business stakeholder together. Don't let engineers decide to live with a sloppy fix for weeks because it was "good enough for now."

Key Takeaways

• 
  Establish AI-specific detection mechanisms: Standard infrastructure monitoring won't catch model degradation, data drift, or bias. Implement automated accuracy monitoring, data quality checks, and fairness audits.
• 
  Detection must be continuous and automated: You cannot rely on customers reporting AI failures. Catch failures within hours of occurrence, not days. Use daily validation and continuous monitoring for critical systems.
• 
  Have containment strategies ready before deployment: Understand how you'll contain each AI system if it fails. Can you revert to a previous model? Can you fall back to a non-AI process? Can you increase human review? Document these strategies and test them.
• 
  Use a systematic investigation framework: When an AI system fails, systematically check: is this a data issue, a model issue, or a decision logic issue? This structure helps diagnose faster than general troubleshooting.
• 
  Integrate AI incident response into existing IRP: Don't create a separate process. Extend your existing incident response process with AI-specific details (detection, containment, investigation, recovery).
• 
  Establish cross-functional incident response teams: AI incidents often require data engineering, data science, product, and compliance expertise. Make sure your incident response can mobilize the right expertise quickly.
• 
  Test your playbook regularly: Run incident simulations (tabletop exercises) at least annually. When you have a real incident, use the post-incident review to improve the playbook.
• 
  Make fallback processes first-class operational systems: Your fallback process isn't a backup. It's a critical operational system that keeps your business running if AI fails. Maintain it, test it, and document it as carefully as you maintain the AI system itself.