Skill.re
When Government AI Goes Wrong
Learning Objectives
After completing this lecture, participants will be able to identify the recurring patterns in publicly documented government AI failures, diagnose the governance and engineering choices that produced them, and apply the resulting lessons to prevent similar failures in their own agencies. The lecture is specifically organized around cases that have produced formal findings by Inspectors General, the Government Accountability Office, legislative bodies, courts, or independent commissions, so the analysis is grounded in published evidence rather than conjecture.
Specifically, you will be able to:
- Describe the top ten publicly documented government AI failures of the last decade, including the Michigan MiDAS unemployment fraud-detection system, Australia's Robodebt automated debt-recovery program, the Netherlands' SyRI algorithmic welfare-fraud system, the COMPAS recidivism risk-assessment controversy, the U.S. Internal Revenue Service audit-selection disparity, the Department of Veterans Affairs paused predictive analytics deployments, U.S. Customs and Border Protection/Transportation Security Administration facial recognition demographic performance, the United Kingdom's A-Level algorithmic grading exam, the New York City automated-decision Local Law 144 enforcement gaps, and the State of Arkansas Medicaid algorithm litigation.
- Identify the recurring systemic causes: inadequate pre-deployment evaluation (particularly subgroup and adversarial evaluation); missing or cosmetic human review; no opt-out or appeal mechanism; missing notice to affected individuals; inadequate data governance and consent; missing monitoring and drift detection; inadequate procurement scrutiny including absent SBOM and no exit rights; absent governance body; and missing accountable official.
- Map each failure pattern to the OMB M-24-10 minimum practice that would have prevented it (pre-deployment testing, ongoing monitoring, human review, notice, opt-out, consultation for rights-impacting; independent evaluation, contingency plans, real-world testing, human decision-making for safety-impacting) and to the NIST AI Risk Management Framework function (Govern, Map, Measure, Manage) that the failure implicates.
- Identify how the failure was surfaced: media reporting, advocacy litigation, Inspector General referral, GAO engagement, congressional inquiry, judicial review, or internal whistleblower. Understand the typical timeline from deployment to detection to response to resolution.
- Design a pre-deployment checklist and a post-incident readiness checklist that together would materially reduce the probability and impact of the failure patterns observed, in a way that aligns with agency strategy, AI Governance Board authority, and Chief AI Officer responsibilities.
- Communicate lessons from these cases to leadership in a way that produces action rather than abstract concern, with explicit references to the cases, the findings, and the policy levers that remain available.
Key Topics Covered
This lecture covers publicly documented government AI failures and the lessons they produce. Topics include:
Ten core cases. Each presented with (a) what the system did and who deployed it, (b) how it failed, (c) who detected the failure, (d) what the formal finding said (OIG, GAO, court, commission, parliament), (e) what was remediated, and (f) the M-24-10/NIST AI RMF lessons.
Michigan MiDAS. The Michigan Integrated Data Automated System was deployed to detect unemployment insurance fraud between 2013 and 2015. It issued tens of thousands of false fraud accusations with quadruple-damages penalties. Class action settled for millions of dollars. Lessons: missing human review, missing opt-out, missing notice, missing appeal, missing monitoring, missing Board oversight.
Australia Robodebt. The Online Compliance Intervention (OCI) / Robodebt system calculated debts using income-averaging, contrary to law, from 2016 to 2019. Australian Royal Commission found serial institutional failures. Government paid back 1.8 billion AUD. Lessons: model mismatch to legal framework, missing independent review, missing effective appeal, missing human-in-the-loop for consequential decisions.
Netherlands SyRI. SyRI was a risk-profiling system for welfare fraud. District Court of The Hague in 2020 found it violated Article 8 of the European Convention on Human Rights for lack of transparency and proportionality. Lessons: missing notice, missing transparency, proportionality failure, consultation failure.
COMPAS. A commercial recidivism risk-assessment tool used in several U.S. state systems. ProPublica 2016 analysis raised disparate-impact concerns. Academic and court debate ongoing. Lessons: missing independent evaluation, missing transparency of vendor scoring, contested fairness metrics.
U.S. IRS audit-selection. IRS automated audit-selection models produced disparate audit rates. National Taxpayer Advocate and GAO reports flagged concerns. IRS paused and remediated. Lessons: missing disparity monitoring, missing documented governance, missing connection to Taxpayer Bill of Rights.
VA predictive analytics pause. In 2023 VA paused several predictive analytics deployments after Office of Inspector General review identified governance and documentation gaps. Lessons: missing documentation, missing risk-tier classification, missing minimum-practice implementation.
CBP/TSA facial recognition. NIST Face Recognition Vendor Test documented demographic performance variation. GAO engagements drove program adjustments. Lessons: missing independent evaluation, missing demographic performance reporting.
United Kingdom A-Level algorithmic grading. During the COVID-19 pandemic, an algorithm downgraded exam results disproportionately affecting disadvantaged students. The government withdrew the algorithm. Lessons: missing subgroup evaluation, missing consultation, missing appeal, missing notice.
NYC Local Law 144 enforcement. Automated decision tools for employment bias audits revealed compliance gaps. Lessons: weak enforcement mechanism, narrow scope of audit requirement.
Arkansas Medicaid algorithm. Algorithm reduced home-care hours; litigation and federal court findings highlighted due-process failures. Lessons: missing reasoned explanation, missing effective appeal, missing consultation.
Systemic patterns. What unites the ten cases. Missing governance body. Missing accountable official. Missing minimum practices. Missing monitoring. Missing appeal. Missing notice.
The OMB M-24-10 and NIST AI RMF lens. Each lesson is mapped to a minimum practice or RMF function.
Practitioner checklists. Pre-deployment checklist; post-incident readiness checklist; leadership talking points that produce action.
Why This Matters for Government
Government AI failures are not hypothetical. They are documented in Inspector General reports, Government Accountability Office studies, court opinions, parliamentary commissions, and independent reviews. The cases that make the news are the cases that had enough harm to generate external investigation. For every case that reaches a Royal Commission or a federal court, many more are contained inside agencies through internal review. The visible cases are the tip of the iceberg. They are the cases where the governance system worked well enough that the problem became known and was addressed, even if imperfectly. They are also the cases that now inform federal policy.
The OMB Memorandum M-24-10 minimum practices for rights-impacting and safety-impacting AI are not abstract. They are the engineered response to the pattern visible across these cases. Pre-deployment testing is M-24-10's response to the evaluation-gap cases. Ongoing monitoring is M-24-10's response to the drift-gap and disparity-gap cases. Human review is M-24-10's response to the missing-human-review cases of MiDAS and Robodebt. Notice is M-24-10's response to SyRI and similar European cases. Opt-out is M-24-10's response to cases where affected individuals had no recourse. Consultation is M-24-10's response to cases where affected communities were never engaged. The NIST AI Risk Management Framework similarly crystallizes the four functions of Govern, Map, Measure, and Manage in response to the same pattern. Each function is a response to a failure mode documented across the case record.
The Government Accountability Office's AI Accountability Framework (GAO-21-519SP) organizes federal oversight around Governance, Data, Performance, and Monitoring for the same reasons. Every one of the ten core cases in this lecture has at least one failure in one of those four areas, and most have failures across multiple areas.
An L4 practitioner's job is to internalize the pattern. Not to memorize every case, but to recognize the signatures. When a program office proposes a new AI use without an evaluation plan, the MiDAS signature is present. When a program proposes a deployment without notice or appeal, the SyRI signature is present. When a program proposes deployment without independent evaluation, the COMPAS and facial-recognition signatures are present. When a program proposes ongoing operation without disparity monitoring, the IRS and A-Level signatures are present. The cases are memory aids for pattern recognition.
This lecture, therefore, is the library. The next lecture (Accountability Frameworks) is the architecture. Together, they give the practitioner the conceptual tools to prevent the recurrence of documented failures in their own agency, and to respond well when a novel failure occurs despite prevention.
The stakes are concrete. In Michigan, tens of thousands of false fraud accusations produced financial and personal harm and settled for millions. In Australia, Robodebt cost the government 1.8 billion AUD and reputational harm. In the Netherlands, SyRI was shut down by a court for human-rights violations. In the United Kingdom, A-Level grading required emergency rescission with political fallout. In Arkansas, a Medicaid algorithm was found to have deprived vulnerable individuals of necessary care. Each was avoidable. Each leaves documented evidence of what should have been done differently. Federal AI practitioners who do not know these cases will repeat them. Federal AI practitioners who do know them can produce different outcomes.
WHY THIS MATTERS FOR GOVERNMENT
The recurring patterns across documented government AI failures are small in number and easy to recognize once you know them. The rest of this lecture walks the ten cases, then summarizes the pattern catalog.
PATTERN 1: MISSING HUMAN REVIEW FOR CONSEQUENTIAL DECISIONS.
MiDAS issued fraud determinations with quadruple-damages penalties without human review. Robodebt calculated debts using income-averaging without human adjudication. Each case shows what happens when a model's output becomes the decision rather than an input to a decision.
M-24-10 response: human review is a minimum practice for rights-impacting AI.
PATTERN 2: MISSING APPEAL OR OPT-OUT.
SyRI offered no meaningful appeal to affected individuals. Arkansas Medicaid reductions were imposed without effective appeal. A-Level students had no meaningful appeal after downgrading.
M-24-10 response: opt-out where applicable, notice, and appeal built into the deployment.
PATTERN 3: MISSING INDEPENDENT EVALUATION.
COMPAS was deployed across state systems with vendor-controlled scoring and limited independent access for evaluation. Facial-recognition programs relied on vendor claims absent NIST FRVT analysis.
M-24-10 response: independent evaluation for safety-impacting; for rights-impacting, rigorous pre-deployment testing with subgroup analysis.
PATTERN 4: MISSING DISPARITY MONITORING.
IRS audit-selection models produced disparate audit rates without agency-level monitoring that would have detected the pattern.
M-24-10 response: ongoing monitoring including demographic performance.
PATTERN 5: MISSING NOTICE AND TRANSPARENCY.
SyRI and many facial-recognition programs operated without notice to affected individuals.
M-24-10 response: notice as a minimum practice.
PATTERN 6: MISSING GOVERNANCE BODY.
VA OIG findings identified governance documentation gaps. MiDAS had no equivalent to an AI Governance Board reviewing use cases.
M-24-10 response: the AI Governance Board is mandatory.
PATTERN 7: MISSING ACCOUNTABLE OFFICIAL.
Many cases lacked a named SES-level owner of the AI use. The owner of record was the contractor.
M-24-10 response: Senior Accountable Official per use.
PATTERN 8: MISSING CONSULTATION WITH AFFECTED COMMUNITIES.
SyRI, A-Level, Arkansas, and others did not substantively consult affected communities.
M-24-10 response: consultation is a minimum practice for rights-impacting AI.
PATTERN 9: MODEL-LEGAL MISMATCH.
Robodebt's income-averaging approach was not consistent with the legal framework for debt calculation.
M-24-10 response: authority-to-use analysis in Phase 1 of the lifecycle.
PATTERN 10: MISSING EXIT RIGHTS AND SUPPLY-CHAIN SCRUTINY.
COMPAS and similar cases were difficult to reform because of vendor lock-in and lack of exit rights.
M-24-10/EO 14028 response: SBOM, SSDF attestation, exit rights, and contract clauses that preserve agency leverage.
The ten patterns map to the ten failures and to the M-24-10 minimum practices. The rest of this lecture walks each case in more detail with the findings and the remediation actually implemented, followed by the practitioner checklists.
Related Lectures
L4 4.3.2 -- Accountability Frameworks for AI Failures
L4 4.3.3 -- Transparency: Citizens' Right to Know
L3 3.3.4 -- Incident Response for AI Systems
L2 2.2.5 -- Human Oversight of AI
L2 2.2.4 -- Equity and Fairness in Government AI
L3 3.2.1 -- Establishing an AI Governance Board
L2 2.1.5 -- The AI System Lifecycle
L4 4.1.x -- Civil Rights Compliance for Federal AI
L5 5.1.x -- Policy Analysis for AI Governance
L5 5.2.x -- Legislative and Regulatory Strategy