Learning Objectives

After completing this lecture, you will be able to:

• Understand the key concepts of risk classification: safety-impacting vs. rights-impacting in a government context
• Connect risk classification: safety-impacting vs. rights-impacting to your agency's AI initiatives
• Identify next steps for applying these concepts in your role

Key Topics Covered

-
How to classify your AI systems

-
Case-by-case analysis exercises

-
Government context for risk classification: safety-impacting vs. rights-impacting

-
Practical applications and next steps

Why This Matters for Government

Government agencies face unique challenges when it comes to AI adoption. This lecture addresses these challenges head-on by providing analysts, project leads, team supervisors with the knowledge and frameworks needed to navigate AI in the public sector responsibly and effectively.

As part of the L2 (AI Practitioner) curriculum, this lecture builds on the foundational principle that every AI system in government ultimately serves citizens. Whether you are working with AI tools daily or setting strategy for your agency, understanding risk classification: safety-impacting vs. rights-impacting is essential for responsible, effective government AI adoption.

======================================================================

TRANSCRIPT: Risk Classification: Safety-Impacting vs. Rights-Impacting

======================================================================

Chapter: 3 -- AI Governance and Risk Management

What you will learn:

• How to classify AI systems by their impact category
• The distinction between safety-impacting and rights-impacting systems
• Why this classification matters for governance and oversight
• How to apply this framework to real government AI use cases
• Documentation and escalation requirements for each category
• How to reassess classifications as systems evolve

One of the most important skills in government AI governance is being able to look at an AI system and quickly assess its risk profile. Not all AI systems pose the same kinds of risks. An AI system that flags potential benefits fraud for manual review poses different governance challenges than an AI system making final eligibility determinations. A chatbot providing general information about visa requirements presents different concerns than a system used in border screening.

This is where risk classification comes in. Risk classification provides a common language for talking about AI system impact within government. It helps you determine what oversight level is appropriate, what validation is needed, and how to allocate governance resources.

In this lecture, we're focusing on one critical distinction: safety-impacting versus rights-impacting systems. This distinction shapes everything about how you govern an AI system, who needs to review it, and how often you need to audit it.

PURPOSE STATEMENT

Risk classification is the foundational practice in AI governance. Before you can build an appropriate oversight framework, set validation requirements, or allocate review resources, you need to understand what kind of risk the system actually poses. This lecture teaches you the safety-impacting and rights-impacting framework--one of the most widely-used classification systems in government AI policy.

WHY THIS MATTERS FOR GOVERNMENT

Federal agencies operate under Constitutional constraints, statutory requirements, and public accountability principles that require you to manage AI risks deliberately and carefully. The Office of Management and Budget (OMB) Memorandum M-24-10 on advancing governance, risk management, and responsible AI use emphasizes that agencies must classify AI systems by impact, document their risk profile, and implement appropriate oversight.

The distinction between safety-impacting and rights-impacting systems determines how you answer critical questions:

• How often should we audit this system? (Weekly? Monthly? Quarterly?)
• Who needs to sign off on this deployment? (Technical team? Legal? Executive?)
• What level of human oversight is required? (Sampling? Full review? Random spot-checks?)
• What testing and validation must happen before deployment? (Fairness analysis? Edge case testing? Domain expert review?)
• What happens if the system fails? (Can we just turn it off? Do we need a contingency plan?)

Getting this classification right affects your entire governance structure. Misclassify a rights-impacting system as low-impact and you miss critical oversight opportunities. Classify a system too conservatively and you over-allocate resources to low-risk systems.

SAFETY-IMPACTING SYSTEMS

A safety-impacting AI system is one whose failure or malfunction could reasonably result in significant injury or severe harm to human health or safety, or could reasonably result in loss of life.

Examples of safety-impacting systems in government:

• An AI system that assists in aviation safety decisions at the FAA
• A system that helps identify critical infrastructure vulnerability points
• AI used in emergency dispatch systems to prioritize emergency response
• A system that assists in nuclear facility operations monitoring
• AI that predicts bridge infrastructure failure patterns for maintenance prioritization

What makes these safety-impacting: Their failure mode is human injury or death. The chain of causation is direct and relatively short. If the system makes a wrong recommendation and that recommendation is followed, people could be harmed.

Key characteristics of safety-impacting systems:

• Physical consequences of failure (injury, death, critical infrastructure damage)
• Often operate in time-sensitive environments where humans cannot easily second-guess recommendations
• Failure consequences are often irreversible or extremely costly
• May operate with partial automation where human oversight exists but is limited by time or expertise constraints

Governance implications for safety-impacting systems:

• Require robust testing including edge cases, adversarial testing, and failure mode analysis
• Need formal risk assessment and safety validation before deployment
• Typically require continuous monitoring with automated alerts for performance degradation
• Require clear escalation procedures and human override capabilities
• May need independent expert review before deployment and periodically thereafter
• Demand detailed documentation of system design choices and failure modes

RIGHTS-IMPACTING SYSTEMS

A rights-impacting AI system is one whose failure, malfunction, or misuse could meaningfully impact civil rights, civil liberties, or privacy. This includes systems that could discriminate against individuals or groups, violate privacy, or deprive someone of a fundamental right.

Examples of rights-impacting systems in government:

• An AI system that assists in criminal sentencing recommendations
• A system that predicts recidivism to inform parole decisions
• AI used to screen job applicants for government positions (disparate impact risk)
• A system that identifies potentially fraudulent benefit applications (affects eligibility to public services)
• AI that flags individuals for enhanced screening in security or immigration contexts
• A system that determines loan eligibility for small business administration programs
• AI used to assess creditworthiness for federal credit programs
• A system that categorizes vulnerability levels for social services eligibility

What makes these rights-impacting: Their failure or misuse affects someone's access to fundamental services, opportunities, or liberty. The impact falls disproportionately on individuals or groups who lack alternative channels to remedy the harms.

Key characteristics of rights-impacting systems:

• Impact on access to government services or opportunities
• Potential for disparate impact on protected classes or vulnerable populations
• May involve sensitive personal information (race, gender, disability status, etc.)
• Failure mode includes discrimination or unfair treatment, not just accuracy failure
• Effects are often felt over time or across multiple decisions
• Individual harms may accumulate across the population

Governance implications for rights-impacting systems:

• Require bias detection and fairness analysis before deployment
• Need meaningful human review for high-stakes decisions (not just sampling)
• Demand transparency about how the system works and what factors drive decisions
• Require appeal or reconsideration mechanisms for individuals affected by adverse decisions
• Need demographic performance analysis and ongoing monitoring for bias drift
• Demand clear documentation of validation methodology and fairness metrics
• Often require legal review for compliance with civil rights laws and Equal Protection principles

THE DISTINCTION BETWEEN SAFETY AND RIGHTS IMPACTS

Safety-impacting and rights-impacting are distinct categories with different risk profiles. Here's why the distinction matters:

A safety-impacting system primarily poses a risk of physical harm. An AI system that fails in a safety-impacting domain has typically failed to prevent a dangerous outcome. The fundamental problem is accuracy and robustness under challenging conditions. The primary governance challenge is preventing failure through rigorous testing and monitoring.

A rights-impacting system primarily poses a risk of unfair treatment or discrimination. An AI system fails in a rights-impacting domain when it produces different outcomes for people in similar situations, especially along demographic lines. The system could be highly accurate in aggregate and still discriminate. The primary governance challenge is ensuring equitable treatment across populations.

This distinction shapes oversight approach:

• Safety-impacting: Focus on preventing failure and detecting degradation. "Is this system working?"
• Rights-impacting: Focus on equitable outcomes and potential discrimination. "Is this system fair?"

Note that a single system can be both safety-impacting AND rights-impacting. For example, an AI system used in emergency medical dispatch could affect both safety (if it fails to alert responders to critical situations) and rights (if it systematically under-allocates resources to certain neighborhoods). When a system poses both types of risk, governance requirements layer--you need both robust failure testing AND fairness analysis.

CLASSIFICATION METHODOLOGY

How do you actually classify a system? Start by asking these questions:

• What is the failure mode? Ask: "If this system gives the wrong answer, what happens?"
• If the answer is "someone could be injured or die," you're likely looking at safety-impacting.
• If the answer is "someone might not get access to a service they deserve" or "someone could face discrimination," you're likely looking at rights-impacting.
• Who bears the consequences of failure? Ask: "Who experiences harm if the system fails?"
• Safety failures typically harm the individuals directly affected by the decision
• Rights failures may harm individuals, but also affect broader population equity and public trust in government
• How much human review is feasible? Ask: "Can a human realistically review every decision?"
• If no: The system may require higher governance standards because humans can't catch problems
• If yes: You may be able to mitigate some risks through human oversight
• What is the sensitivity of the domain? Ask: "Does this decision affect access to fundamental government services or constitutional rights?"
• Higher sensitivity (voting, criminal justice, benefits) = higher rights impact concern
• Lower sensitivity (informational assistance) = potentially lower rights impact concern
• Does the system use sensitive information? Ask: "Does the system incorporate demographic data or other sensitive information?"
• If yes: Higher risk of disparate impact, rights-impacting classification more likely

PRACTICAL CASE EXAMPLES

Let's work through several government AI systems and practice classification.

CASE EXAMPLE 1: IRS Tax Fraud Detection System

An AI system that flags suspicious tax filings for enhanced review by IRS auditors.

Classification analysis:

• Failure mode: System flags non-suspicious filings as suspicious, burdening taxpayers with audits, OR system misses fraudulent filings, reducing tax compliance
• Rights impact: Yes. Individuals incorrectly flagged for audit face compliance burden and intrusive review. If the system systematically over-flags returns from certain demographic groups, it creates disparate impact.
• Safety impact: Probably not directly. The system doesn't cause physical harm if it fails.
• Classification: Rights-impacting. This system requires fairness analysis, demographic performance testing, and human review procedures that protect taxpayers from unfair targeting.

CASE EXAMPLE 2: FAA Runway Inspection AI

An AI system that analyzes runway surface imagery to detect cracks and damage requiring maintenance.

Classification analysis:

• Failure mode: System misses critical damage that could cause aircraft accidents
• Safety impact: Yes. System failure could directly contribute to loss of life.
• Rights impact: Probably not. A system that inspects runways doesn't make individual-level decisions affecting people's rights or services.
• Classification: Safety-impacting. This system requires rigorous edge-case testing, robustness testing under various weather conditions, and continuous monitoring for performance degradation.

CASE EXAMPLE 3: Benefits Eligibility Screening System

An AI system that performs initial screening of applications for social services benefits, flagging applications for full human review.

Classification analysis:

• Failure mode: System incorrectly denies eligible individuals or misses eligibility factors
• Rights impact: Yes. Citizens are denied benefits they're eligible for. If disparities exist across demographic groups, it creates discriminatory impact.
• Safety impact: Possibly indirect. Benefits denial could contribute to housing insecurity or food insecurity for vulnerable individuals, but this is an indirect safety concern.
• Classification: Strongly rights-impacting, possibly also safety-impacting. Requires fairness analysis, demographic testing, human review procedures, and appeal mechanisms.

REASSESSMENT AND SYSTEM EVOLUTION

Overview

AI systems don't remain static. As systems evolve, their classification may need to change.

Scenarios requiring reclassification:

• System scope expands: A system initially used in one office expands to multiple regions
• Decision authority increases: A system initially flagging cases for human review becomes fully automated
• Population affected changes: A system originally serving one demographic now serves broader population
• Use case expands: A system built for recommendations now used for final decisions
• Performance changes: A system that was performing well shows degradation in certain subpopulations

Reassessment should happen:

• Before major scope changes or deployment expansions
• When system performance monitoring reveals new issues
• When new protected classes or vulnerable populations are affected
• When deployment context changes significantly
• At least annually as part of system governance review

ANTI-PATTERNS

ANTI-PATTERN 1: Classification Creep

The system is initially classified as low-impact, but over time it accumulates more decision authority, more users, and more population impact--without reclassification. Teams forget the original assessment context and treat the system as lower-risk than it actually is.

How to avoid it: Establish regular reassessment schedules. When system scope changes, trigger formal reclassification. Document assumptions about system use and alert the team if those assumptions change.

ANTI-PATTERN 2: Conflating Accuracy with Fairness

A team assumes that if their system is accurate (high precision/recall), it's also fair and rights-compliant. They focus exclusively on accuracy metrics and skip fairness analysis. They discover only later that their accurate system systematically disadvantages certain demographic groups.

How to avoid it: Understand that accuracy and fairness are separate concerns. A system can be 95% accurate and still be unfair. Require both accuracy testing AND demographic parity analysis for rights-impacting systems.

ANTI-PATTERN 3: Over-Classification for Governance Avoidance

Leadership feels threatened by governance requirements and pushes back on classification, arguing the system is lower-impact than the evidence suggests. Pressure builds to classify the system conservatively to avoid oversight burden.

How to avoid it: Establish clear, objective classification criteria. Document the evidence supporting classification. Make clear that governance requirements are not punitive--they exist to reduce real risks. Classify systems based on actual impact, not on governance appetite.

PRACTICE PROMPTS

EXERCISE 1: Classify a System You Know

Think of an AI system in your agency or one you're familiar with. Apply the five classification questions:

• What is the failure mode?
• Who bears the consequences?
• How much human review is feasible?
• What is the domain sensitivity?
• Does it use sensitive information?

Based on your answers, classify the system as safety-impacting, rights-impacting, both, or neither. Document your reasoning. Are there aspects of the classification you're uncertain about?

EXERCISE 2: Case Study Analysis

For each of these systems, classify and document your reasoning:

A. An AI system that predicts maintenance needs for federal buildings

B. An AI system that routes social security disability claims to appropriate specialists

C. An AI system that optimizes schedules for security checkpoint staffing

D. An AI system that detects anomalies in federal financial transactions for investigation

EXERCISE 3: Governance Requirement Design

Choose one of the systems from Exercise 2. Based on your classification, design the governance approach:

• What validation testing is required before deployment?
• How frequently should the system be audited?
• What human review processes are needed?
• Who should approve this system before deployment?
• What monitoring and alerting should be in place?

KEY TAKEAWAYS

• Risk classification is foundational to AI governance. You cannot design appropriate oversight without understanding what kind of risk a system poses.
• Safety-impacting systems require robust failure prevention. These systems demand rigorous testing, continuous monitoring, and failure mode analysis because their failure can cause direct harm.
• Rights-impacting systems require fairness assurance. These systems demand demographic parity analysis, fairness testing, human review procedures, and appeal mechanisms because their failure can cause discrimination.
• Safety and rights impacts are distinct but can coexist. A system can pose only safety risk, only rights risk, both, or neither. Governance requirements layer when systems pose multiple types of risk.
• Classification methodology should be objective and documented. Use consistent criteria for classification. Document your reasoning. Be prepared to explain your classification to oversight bodies and affected communities.
• Classification should be reviewed and updated periodically. As systems evolve, as deployment context changes, and as performance data emerges, reassess whether original classification remains appropriate.
• Classification drives resource allocation. Agencies have finite governance resources. Accurate classification ensures high-risk systems receive appropriate oversight while avoiding over-governance of lower-risk systems.

GLOSSARY

Disparate Impact: The effect of ostensibly neutral practices that have a disproportionately negative impact on members of a protected class, even if discriminatory intent is absent.

Risk Classification: The systematic categorization of AI systems based on the types and magnitude of impacts they could have if they fail or are misused.

Safety-Impacting System: An AI system whose failure could reasonably result in significant injury, severe harm to human health or safety, or loss of life.

Rights-Impacting System: An AI system whose failure, malfunction, or misuse could meaningfully impact civil rights, civil liberties, or privacy, including the potential for discrimination.

Governance Oversight: The systematic practices, reviews, and controls that ensure an AI system operates as intended, remains fair and safe, and complies with applicable policies and laws.

The goal of risk classification is to match governance intensity to actual risk. You're not classifying systems to create bureaucratic burden--you're classifying them to prevent the right kinds of failures through the appropriate oversight mechanisms.

In practice, you'll find that different organizations apply classification frameworks slightly differently. Some agencies use additional categories (like "efficiency-impacting" systems), and others add nuance to these core categories. What's consistent across federal AI governance is the recognition that not all systems require the same oversight, and that the intensity and focus of governance should depend on what kind of failure the system could experience.

As you move forward in your agency's AI journey, you'll apply this classification framework repeatedly. Every new system will go through classification. Every major system change will trigger reclassification. The practice becomes easier with repetition, and having clear, objective criteria makes governance conversations more productive.

Take 2-3 minutes to reflect on this question: "What governance requirements would I want applied to an AI system that affects my own access to government services?" Consider how your answer might vary depending on what kind of service is at stake and how much the system's decision affects you. Use your reflection to inform your thinking about what governance is appropriate for systems in your agency.

Risk classification is your foundation for everything that comes next in this module. The chapters ahead address specific governance requirements for different types of AI systems--validation approaches, monitoring frameworks, human oversight design, and incident response procedures. All of those practices are informed by and layered onto the risk classification you establish here.

In the next lecture, we'll move from classification to project management. We'll explore how AI projects differ from traditional IT projects and what that means for planning and resource estimation. Understanding these differences will help you avoid common pitfalls as you execute AI projects in your agency.

Thank you for engaging with this critical foundational material. See you in the next lecture.

RESOURCES AND FURTHER READING

• OMB Memorandum M-24-10: "Advancing Governance, Risk Management, and Responsible AI Use in Government"
• NIST AI Risk Management Framework (RMF): Guidance on impact assessment and system characterization
• Executive Order 14110: "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence"

Government AI CLUB Certification Program

Level 2: AI Ready | Risk Classification: Safety-Impacting vs. Rights-Impacting | Lecture 2.3.3

A GOVT.CLUB initiative

Visit: https://govt.club/learn/lectures/l2/233-risk-classification.html

======================================================================

<- 2.3.4 AI Use Case Inventory and Documentation (OMB M-24-10)
2.3.6 Minimum Risk Management Practices ->

Start Your CLUB Certification

This lecture is part of L2: AI Practitioner -- 40 hours of comprehensive government AI training.

Explore CLUB Certification

Related Lectures

L2
2.3.1 -- NIST AI RMF: The GOVERN Function
60 min - Video + Workshop

L2
2.3.2 -- NIST AI RMF: MAP, MEASURE, MANAGE
60 min - Video + Workshop

L2
2.3.3 -- Your Agency's AI Governance Structure
60 min - Reading + Discussion