Learning Objectives

After completing this L3 workshop, federal CAIOs, CISOs, and senior AI program managers will be able to: first, classify AI incidents into the taxonomy used by NIST AI Safety Institute, OMB M-24-10, and the agency's own AI policy memo, distinguishing security incidents (e.g., model theft, prompt injection, data exfiltration), performance incidents (e.g., accuracy drift, fairness degradation), and safety incidents (e.g., harmful outputs, physical harm through AI-directed action); second, design an AI-specific incident response plan that integrates with the agency's existing FISMA-required incident response capability and CISA coordination obligations under the Cybersecurity Information Sharing Act; third, execute the five phases of AI incident response: preparation, detection and analysis, containment and eradication, recovery, and post-incident learning, aligned with NIST SP 800-61; fourth, coordinate with external stakeholders including CISA, the DOJ Civil Rights Division for rights-impacting incidents, the Office of Inspector General, Congressional committees, the press, and affected citizens, with appropriate documentation and timing; fifth, produce the after-action artifacts, including timeline reconstruction, root cause analysis, impact assessment, remediation plan, and public report, that mature agencies use to learn from incidents and demonstrate accountability; and sixth, integrate AI incident response with broader agency risk management, so that incidents feed into the NIST AI RMF MANAGE function and the enterprise risk register.

Key Topics Covered

The workshop covers nine topics. First, the AI incident taxonomy, including security, performance, and safety categories with worked federal examples. Second, the legal and policy frame: FISMA incident reporting, CISA coordination, OMB M-24-10 AI-specific obligations, agency-specific AI policies, and sector-specific rules (HIPAA breach notification, IRS tax information disclosure rules, etc.). Third, the NIST SP 800-61 five-phase incident response lifecycle adapted for AI. Fourth, detection and triage, including the monitoring stack that surfaces AI incidents and the triage criteria that distinguish severity. Fifth, containment strategies, including model rollback, traffic diversion, feature disablement, and in extreme cases full system takedown. Sixth, external coordination, including CISA notification within statutory windows, DOJ Civil Rights Division engagement for rights-impacting incidents, and congressional and IG communication. Seventh, communication with affected citizens and the public, including when and how to issue public notices. Eighth, post-incident learning through timeline, root cause, impact, remediation, and public report artifacts. Ninth, the integration of incident response with continuous improvement of the AI system and the agency's AI governance program.

Why This Matters for Government

AI incidents will happen in your agency. That is a certainty, not a probability, for any agency operating production AI at scale. The question is not whether you will experience an incident but whether your agency can respond fast, honestly, and effectively when it does. Federal incident response is where the work of the AI program is tested in public. An agency that has prepared can contain the damage, preserve its credibility, and learn. An agency that has not prepared produces exactly the kind of crisis that makes it into GAO reports, Inspector General hearings, congressional inquiries, and class action lawsuits.

The legal and policy frame is well-established and increasingly specific for AI. FISMA and OMB M-17-25 require agencies to maintain an incident response capability and report to CISA and the US-CERT. OMB M-24-10 layered AI-specific obligations on top, including timely disclosure of rights-impacting incidents and coordination with the agency CAIO. CISA coordinates federal cyber incident response under its authorities from the Cybersecurity Information Sharing Act of 2015 and the Cyber Incident Reporting for Critical Infrastructure Act. For rights-impacting AI incidents, the DOJ Civil Rights Division may engage, and affected beneficiaries have specific due process and equal protection rights that shape how the agency must respond. Each of these authorities sets timing, scope, and documentation expectations that an incident response plan must satisfy from the start.

The AI-specific twist is that many incidents do not look like traditional cyber incidents. A model that starts producing biased outputs after a data drift is not a network intrusion, but it is an incident under OMB M-24-10 if it is a rights-impacting use. A model that correctly processes data but produces harmful recommendations due to training data flaws is not a technical failure in the FISMA sense, but it is still an incident affecting citizens. A prompt injection attack that causes a chatbot to leak privileged information is both a security incident and a data breach. Your agency's incident response plan must be broad enough to handle all three, with clear escalation paths for each and the coordinating offices that need to be involved.

Mature agencies treat AI incidents as learning opportunities, not reputational threats to be minimized. The Partnership for Public Service's research on high-performing federal programs consistently shows that candor about incidents produces greater trust than denial, and accelerated remediation produces stronger long-run outcomes than bureaucratic delay. The leadership practice is simple: when something goes wrong, say so, fix it fast, learn from it, and tell the public what you learned. Agencies that do this build durable trust. Agencies that do not, lose it over time even if no single incident is catastrophic.

The AI Incident Taxonomy

A defensible AI incident taxonomy has three top-level categories and a dozen subcategories. Security incidents include: model theft (attackers exfiltrate the model weights or architecture), adversarial examples (crafted inputs that cause wrong outputs), prompt injection (crafted prompts that override instructions), model inversion (attackers extract training data from model outputs), data poisoning (attackers corrupt training data), and supply chain compromise (attackers compromise a model or dataset used downstream). Performance incidents include: accuracy drift (performance degrades below committed levels), fairness degradation (disaggregated metrics show new disparities), calibration failure (confidence estimates are no longer reliable), and data pipeline failure (input data has become corrupt or stale). Safety incidents include: harmful outputs (e.g., defamatory, dangerous, or unlawful content), physical harm (AI-directed actions cause real-world harm), and rights violations (due process, equal protection, or statutory rights are infringed).

For each subcategory, the agency's incident response plan should document detection signals, initial triage criteria, coordinating offices, external notification requirements, containment options, and typical recovery sequence. Not every incident will fit neatly into a single subcategory, and some will span multiple: a prompt injection attack that leaks PII is both a security incident and a privacy breach, and the response must activate both the security and privacy playbooks simultaneously.

Severity classification is separate from category. A Severity 1 incident affects mission-critical operations or produces significant citizen harm; a Severity 2 affects important operations or produces moderate harm; a Severity 3 is localized or low-impact. Severity determines response tempo and executive engagement: Severity 1 incidents page the CAIO and CISO within an hour and brief the Secretary within a day; Severity 3 incidents are handled by the program team with routine status reports. Mature agencies publish the severity classification framework so staff and vendors know the response they will get. Ambiguous severity produces slow response, which makes incidents worse.

The Five-Phase Incident Response Lifecycle for AI

NIST SP 800-61 defines a five-phase incident response lifecycle: preparation; detection and analysis; containment, eradication, and recovery; post-incident activity. Federal AI incident response adapts this lifecycle with AI-specific practices. Preparation includes standing up the AI incident response team (CAIO, CISO, SAOP, General Counsel, program owner, communications), defining runbooks for each incident subcategory, exercising the plan through tabletop and live exercises, and maintaining the evidence infrastructure that will be needed during an incident (lineage, logs, model registry, audit trails).

Detection and analysis brings the monitoring stack described in the data infrastructure lesson to bear on incident discovery. Alerts from drift detection, fairness monitoring, security monitoring, and user reports all feed into a single on-call queue that the AI incident response team reviews. Triage determines whether an alert is a true incident, which subcategory it falls into, and initial severity. Analysis expands the scope: what systems, users, and data are affected; what timeline did the incident follow; what controls failed or succeeded. Analysis produces the facts that subsequent phases rely on.

Containment, eradication, and recovery is the operational heart of response. Containment stops the bleeding, typically through model rollback to a known-good version, traffic diversion to a fallback system, disablement of specific features, or in extreme cases full system takedown. Eradication removes the root cause: patches the vulnerability, retrains on clean data, retires the compromised artifact. Recovery restores service to normal, verifies the recovery worked, and monitors closely for recurrence. For rights-impacting incidents, recovery also includes remediation for affected beneficiaries, which may involve re-adjudication of their cases.

Post-incident activity is where the agency turns the incident into institutional learning. The incident commander produces a timeline, a root cause analysis, an impact assessment, a remediation plan, and a public report if appropriate. These artifacts feed the agency's AI governance program, the CAIO's annual report, and the NIST AI RMF MANAGE function. Agencies that skip the post-incident phase relearn the same lessons repeatedly. Agencies that institutionalize it develop compounding capability over years.

External Coordination and Citizen Communication

AI incidents rarely stay inside one agency. The coordination tree starts with CISA, which is the federal civilian executive branch's coordinator for cyber incidents and increasingly for AI-specific incidents. FISMA reportable incidents must be reported to CISA within the statutorily-defined windows; AI-specific incidents often follow the same timing because AI incidents typically involve some security dimension. CISA's coordination includes information sharing with peer agencies, alerting to broader campaigns, and technical support to the affected agency. Treat CISA as a partner, not a burden; the intelligence they provide is worth the notification overhead.

For rights-impacting incidents, the DOJ Civil Rights Division should be looped in early. If your AI has produced disparate impact across protected classes, DOJ is a natural interlocutor, and engaging them early produces a better outcome than engaging them after a class action is filed. The agency's Inspector General should also be informed early for significant incidents; the CIGIE protocols and IG Act as amended provide the framework. Early IG engagement converts what could be an adversarial audit into a partnership on remediation, and IGs consistently report that early disclosure improves their ability to help the agency.

Citizen and public communication is a leadership discipline, not a PR exercise. The principle, grounded in the social contract, is that affected citizens deserve to know what happened, what the agency is doing about it, and how they can get their specific situation addressed. For rights-impacting incidents, this often includes individual notification to affected people, a public statement, a path for beneficiaries to request re-adjudication, and updates as the remediation progresses. The FTC's breach notification model and HIPAA breach rules provide templates. Timing matters: within 72 hours of confirmation is emerging as a norm for significant incidents, and waiting longer typically worsens the communications outcome. Agencies that communicate clearly and quickly build trust; agencies that hide, minimize, or delay lose trust and often face worse legal exposure.

Post-Incident Learning and Continuous Improvement

The post-incident phase is where agencies that take AI seriously separate themselves from those that do not. Five artifacts anchor the phase. The timeline is a minute-by-minute reconstruction of the incident from first signal through full recovery, including all detection, decision, and communication events. The root cause analysis identifies the underlying contributors, typically at multiple levels: the immediate technical cause, the governance gap that allowed the immediate cause, and the institutional condition that allowed the governance gap. A rigorous RCA names contributing factors without scapegoating; the agency that punishes people for surfacing root causes produces a culture that hides them next time. The impact assessment quantifies the harm: how many citizens were affected, how severely, with what duration, and how reversible. The remediation plan names specific actions, owners, and deadlines, with the AI Governance Board as accountability body. The public report, for significant incidents, presents the above in a citizen-accessible form.

These artifacts feed multiple downstream processes. The timeline supports any subsequent IG or GAO audit. The RCA drives changes in the AI system and often the agency's governance program. The impact assessment supports remediation budgeting and potentially litigation exposure analysis. The remediation plan is tracked to closure in the same rigor as any other program commitment. The public report builds trust when done candidly and damages it when done defensively.

Continuous improvement extends post-incident learning across incidents. The agency maintains an incident register that allows patterns to be identified over time. A single incident is a data point; three similar incidents is a signal; ten is a structural issue. CAIOs who review the incident register quarterly identify structural issues their teams may be too close to see. Trends in the register feed the annual CAIO report, the OMB M-24-10 annual inventory update, and the agency's AI governance roadmap. Agencies that do this consistently build a capability for institutional learning that competitors cannot match. Agencies that do not, repeat incidents because they never see the pattern until a GAO auditor points it out.

Exercises, Readiness, and Cultural Practice

No incident response plan is real until it has been exercised. Federal agencies should conduct at least one tabletop exercise and one live exercise per year for their AI incident response, with scenarios drawn from the incident taxonomy. A tabletop exercise walks a team through a realistic scenario without actually touching production systems; a live exercise simulates detection and response in a controlled environment, sometimes using a red team. Exercises surface gaps in runbooks, escalation paths, tools, and relationships that paper-only plans cannot find. The Department of Homeland Security's Cyber Storm exercises offer a federal template, and many agencies now include AI-specific scenarios.

Readiness also requires standing infrastructure. An on-call rotation staffed 24x7 for Severity 1, response runbooks kept current, a war-room with pre-provisioned communication channels, named relationships with CISA, DOJ, IG, and communications, and executive briefing templates ready to customize. Agencies that develop this infrastructure in the quiet periods can activate it in minutes when an incident happens. Agencies that build it during an incident lose critical time and often mismanage the response.

Cultural practice is the foundation. Agencies with strong AI incident response cultures exhibit three traits: dissent is safe, so early signals get surfaced; learning is prioritized over blame, so root causes get named; and transparency is the default, so citizens and oversight bodies know the agency is being straight with them. Agencies that punish early warning, scapegoat individuals, or hide incidents produce weak incident response regardless of how polished their runbooks are. The CAIO and CISO have a particular responsibility to model these cultural traits in their own handling of incidents, because the culture is established at the top and felt throughout. Running a serious incident response culture is, in this sense, as much a leadership practice as a technical one.

Related Lectures

L3 3.4.1 Enterprise AI Risk Management. L3 3.4.3 Cybersecurity for AI Systems. L3 3.4.4 Privacy Engineering for AI. L4 4.5 AI Incident Documentation and Response.