Learning Objectives

After completing this lecture, you will be able to:

• Understand the key concepts of when government ai goes wrong in a government context

• Analyze real-world case studies from government agencies

• Connect when government ai goes wrong to your agency's AI initiatives

• Identify next steps for applying these concepts in your role

Key Topics Covered

• Michigan unemployment fraud detection
• COMPAS recidivism scoring
• Child welfare algorithms
• Lessons learned from real failures

Why This Matters for Government

Government agencies face unique challenges when it comes to AI adoption. This lecture addresses these challenges head-on by providing all government employees with the knowledge and frameworks needed to navigate AI in the public sector responsibly and effectively.

As part of the L1 (AI Aware) curriculum, this lecture builds on the foundational principle that every AI system in government ultimately serves citizens. Whether you are working with AI tools daily or setting strategy for your agency, understanding when government ai goes wrong is essential for responsible, effective government AI adoption.

Lecture URL: https://skill.re/learn/govt/when-government-ai-goes-wrong.php

======================================================================

TRANSCRIPT: When Government AI Goes Wrong

======================================================================

What you will learn: Real case studies—Michigan unemployment fraud detection, COMPAS recidivism scoring, child welfare algorithms. What went wrong and why.

Theory is important. But sometimes, you learn more from failure than from principle.

In this lecture, we're going to examine three real cases where government AI systems went seriously wrong. We'll look at what happened, why it happened, and what lessons we can draw.

These are not hypotheticals. These are real people harmed by real AI systems in real government agencies.

WHY THIS MATTERS FOR GOVERNMENT

Learning from failures prevents you from repeating them. Understanding what went wrong in other agencies helps you avoid the same mistakes in yours.

CASE STUDY 1

What Happened:

During COVID-19 pandemic, Michigan received increased unemployment claims. The state implemented an AI system to detect fraudulent claims.

The system was trained on historical fraud cases. It flagged claims that didn't match historical patterns of "legitimate" claims.

The Problem:

The system was biased. It systematically over-flagged claims from certain demographic groups.

People whose claims were flagged had their benefits stopped. They had to appeal or reapply. Months of waiting with no income.

Some lost their homes. Some went hungry. Some had lawsuits against the state.

Why It Went Wrong:

• Training Data Bias: Historical fraud cases reflected investigator biases and policing biases, not actual fraud rates. The system learned those biases.

• Inadequate Testing: The system wasn't tested for demographic disparities before deployment.

• Scale: The system was deployed statewide instantly, affecting hundreds of thousands of people.

• Opacity: People didn't understand why their claims were flagged. There was no way to contest or appeal the AI decision.

• Urgency Excuse: Because of the pandemic crisis, proper due diligence was skipped.

Lessons:

• Crisis is not an excuse for skipping fairness testing

• High-stakes decisions require human review

• Transparency and appeals are essential

• Demographic testing must happen before deployment

CASE STUDY 2

What Happened:

COMPAS is an AI system used in U.S. courts to predict whether someone, if released, is likely to reoffend.

Judges use COMPAS scores in sentencing and parole decisions.

Researchers audited COMPAS and found it was biased. It systematically overpredicted recidivism for Black defendants and underpredicted it for white defendants.

The Consequences:

Black defendants were given longer sentences based on AI predictions that were systematically wrong.

Why It Went Wrong:

• Training Data Bias: COMPAS was trained on historical arrest and conviction data. But arrests and convictions are influenced by policing biases, not actual crime rates.

• Feedback Loop: The system was trained on arrests -> system predicted higher risk for groups that were over-arrested -> those groups got more police attention -> more arrests for those groups -> system's bias was reinforced.

• Lack of Transparency: For years, the system was used without public understanding of its biases.

• Assumption of Objectivity: The assumption was that an algorithm would be more objective than human judges. But the algorithm learned human biases from its training data.

• High Stakes: The system affected fundamental rights (freedom, sentence length). Yet it wasn't audited for fairness before deployment.

Lessons:

• Historical data reflects historical biases

• Feedback loops amplify bias

• Transparency is essential for high-stakes systems

• Algorithms don't eliminate human bias; they can scale it

• High-stakes systems require extensive fairness testing

CASE STUDY 3: CHILD WELFARE AI SYSTEMS (2016-Present)

What Happened:

Multiple cities and states developed AI systems to predict which children are at risk of abuse or neglect, to prioritize limited investigative resources.

Some systems were found to be biased against certain communities.

The Consequences:

Children in certain neighborhoods were investigated more frequently, even after controlling for actual risk.

Families had investigations in their records even if no abuse was found.

Why It Went Wrong:

• Training Data: Systems trained on investigation history, which reflects investigation bias (which communities get investigated more).

• Proxy Variables: Systems used variables correlated with race (poverty, neighborhood) as predictors.

• No Ground Truth: "Risk" was measured by investigations, not by actual abuse. So systems learned to predict who would be investigated, not who was actually at risk.

• Vulnerability: Child welfare is already high-stakes. Biased decisions have serious consequences.

• Lack of Accountability: For years, the biases weren't publicly acknowledged.

Lessons:

• Be careful about what constitutes "success." If you measure it by historical investigation patterns, you're measuring investigation bias, not actual risk.

• Proxy variables are problematic in high-stakes contexts

• Vulnerable populations need extra protection, not less

• Accountability and transparency matter from day one

CROSS-CUTTING LESSONS

From these three cases, what can we learn?

• Crisis doesn't justify skipping due diligence. If anything, high-stakes situations demand more caution.

• Historical data encodes historical biases. You must be very careful about what you're training on.

• Transparency and explanation are essential. People affected by AI decisions have a right to understand why.

• Human review is necessary for high-stakes decisions. No algorithm is good enough to remove humans from the loop.

• Testing for bias before deployment is non-negotiable. All three cases could have been prevented with adequate pre-deployment testing.

• Feedback loops amplify bias. What starts as biased training data becomes more biased as the system influences what happens next.

ANTI-PATTERNS / MISUSE RISKS

Anti-Pattern 1: "It's Better Than What It Replaces"

An agency implements a biased AI system but argues, "It's still better than the old manual process, which was also biased."

Risk: Bias is scaled and legitimized. The fact that humans were also biased doesn't justify deploying a biased algorithm.

Anti-Pattern 2: "The Bias Isn't That Bad"

An agency finds bias in their AI system but argues, "The overall accuracy is still good. The bias isn't that large."

Risk: Any systematic bias that disadvantages vulnerable populations is unacceptable.

Anti-Pattern 3: "We'll Fix It Later"

An agency deploys an AI system with known fairness concerns and promises to fix it after deployment.

Risk: "Later" never comes. The system continues causing harm.

Anti-Pattern 4: "There's No Better Alternative"

An agency keeps using a biased system because alternatives haven't been fully developed.

Risk: Using a biased system is worse than using a fair (even if less efficient) system.

PRACTICE / REFLECTION PROMPTS

• Does your agency use AI systems similar to the ones in these case studies? If so, have they been tested for bias?

• What would it mean if one of these systems was biased against your demographic group? How would you feel?

• If you discovered an AI system in your agency had significant bias, what would you do?

KEY TAKEAWAYS

• Real systems cause real harm. These aren't theoretical problems.

• Training data bias is the most common cause of algorithmic bias. Watch for it.

• Transparency and explanation are essential safeguards. Don't deploy opaque systems.

• Human review is necessary for high-stakes decisions. No AI system should be final on its own.

• Testing for bias before deployment prevents harm. Do the work upfront.

• Scale magnifies problems. A biased decision made by one person is harmful. A biased decision made by an algorithm at scale is a tragedy.

TERMS / GLOSSARY ITEMS

Bias: Systematic error or unfairness in decision-making.

Feedback Loop: When predictions influence outcomes, which become data for future predictions.

Ground Truth: What actually happened (as opposed to what was recorded or predicted).

Proxy Variable: A feature that indirectly represents something else (neighborhood as a proxy for race).

Imagine you're reviewing an AI system your agency proposed:

"We want to implement an AI system to identify which community members are at highest need for social services. We'll train the system on historical service utilization data. The system will flag people most likely to benefit from services."

Red flags (from these cases):

• Training on service utilization data: This measures who received services, not who needs services. Vulnerable groups might have received less services due to barriers, so they'd be flagged as lower-need.

• Bias risk: Historical service data reflects service patterns influenced by outreach, access barriers, etc. The system will learn those patterns.

• High stakes: This determines who gets offered services. Biasing it against vulnerable populations is exactly wrong.

What you'd recommend instead:

• Train on needs assessment data (direct evaluation of need), not on service utilization

• Test the system for bias before deployment, disaggregating by demographic group

• Build in human review so social workers can override the system's assessment based on their knowledge

• Make the system's reasoning transparent to clients

• Provide appeal process

10 minutes.

For each of the three case studies, write down:

• What went wrong?

• How could it have been prevented?

• Could this happen in your agency?

These case studies are cautionary tales. They're also evidence that the problems we've discussed in this course are real and have real consequences.

As you move forward in your work with government AI, let these lessons guide you. Test for bias. Ensure transparency. Keep humans in the loop. Don't deploy systems without adequate due diligence.

The work is hard. But the alternative—harming citizens through biased AI—is unacceptable.

Government AI CLUB Certification Program

Level 1: AI Aware | Algorithmic Fairness in Government | Lecture 5.2

A GOVT.CLUB initiative.

<- 1.5.3 The Human in the Loop 1.5.5 Your Role as an AI Steward ->

Start Your CLUB Certification

This lecture is part of L1: AI Aware—8 hours of comprehensive government AI training.

Explore CLUB Certification

Related Lectures

L1 1.5.1—Algorithmic Fairness in Government 15 min - Video + Cases

L1 1.5.2—Transparency: Citizens' Right to Know 10 min - Video + Reading

L1 1.5.3—The Human in the Loop 10 min - Video + Scenarios

Frequently Asked Questions

What will I learn in When Government AI Goes Wrong?

In this 15 min case studies lecture, you will Michigan unemployment fraud detection. COMPAS recidivism scoring. Child welfare algorithms. Lessons learned from real failures

What level is When Government AI Goes Wrong?

This is a Level 1 (AI Aware) lecture, part of Chapter 1.5 \u2014 Ethics and Citizen Impact. It is designed for all government employees.

How long is lecture 1.5.4?

Lecture 1.5.4 (When Government AI Goes Wrong) takes 15 min. It is delivered as a case studies format.

Do I need prerequisites for When Government AI Goes Wrong?

This lecture is part of L1 (AI Aware). Prerequisites: None.

What is the CLUB Certification?

CLUB (Community Leading Unified Benchmarks) is a maturity-based AI certification for government professionals with 5 levels (L1-L5), 215 lectures, and 25 chapters aligned with NIST AI RMF, OMB, and GAO frameworks.