Learning Objectives

After completing this lecture, you will be able to:

• Understand the key concepts of testing and validating ai systems in a government context
• Apply knowledge of accuracy, fairness, robustness, edge cases
• Apply knowledge of building test plans
• Complete hands-on exercises that reinforce practical skills

Key Topics Covered

-
Test strategies for AI: accuracy, fairness, robustness, edge cases

-
Building test plans

-
Acceptance criteria for AI

Why This Matters for Government

Government agencies face unique challenges when it comes to AI adoption. This lecture addresses these challenges head-on by providing analysts, project leads, team supervisors with the knowledge and frameworks needed to navigate AI in the public sector responsibly and effectively.

As part of the L2 (AI Practitioner) curriculum, this lecture builds on the foundational principle that every AI system in government ultimately serves citizens. Whether you are working with AI tools daily or setting strategy for your agency, understanding testing and validating ai systems is essential for responsible, effective government AI adoption.

======================================================================

TRANSCRIPT: Testing and Validating AI Systems

======================================================================

Chapter: 4 -- AI Project Management

What you will learn:

• Why testing AI systems is fundamentally different from testing software
• Multiple dimensions of testing (accuracy, fairness, robustness, edge cases)
• How to design test data and test sets
• What constitutes adequate testing before government deployment
• How to report and document test results
• Continuous testing and monitoring in production
• Common testing pitfalls and how to avoid them

Testing traditional software is straightforward: You define inputs, specify expected outputs, run the test, verify you get those outputs. Same test, same result, every time.

Testing AI systems is fundamentally different. You can't just verify that "given input X, the system produces output Y." AI systems produce probabilistic outputs that vary. The same input might produce a slightly different output depending on how the model was trained and what data it's learning from.

This means testing AI systems requires a different approach. Instead of binary "test passes or fails," you're evaluating a system across multiple dimensions: Is the accuracy acceptable? Is it fair across demographic groups? Does it fail gracefully on edge cases? How does it perform on data it's never seen before?

This lecture teaches you the testing strategy for government AI systems. We'll walk through the dimensions of testing, what adequate validation looks like, and how to document that a system is ready for government deployment.

PURPOSE STATEMENT

Testing and validation are your primary controls for ensuring that deployed AI systems are accurate, fair, and robust. For government AI systems, validation isn't optional--it's a fundamental governance requirement. This lecture teaches you how to design validation approaches that provide evidence that a system is ready for deployment and safe to use.

WHY THIS MATTERS FOR GOVERNMENT

In government, deploying an untested AI system poses serious risks. An unvalidated benefits eligibility system might deny eligible citizens benefits. An unvalidated hiring system might discriminate against protected classes. An unvalidated fraud detection system might harass innocent citizens.

Government agencies have legal obligations to ensure systems are fair, accurate, and safe. The Administrative Procedure Act, civil rights laws, and agency-specific regulations all require evidence that systems work as intended. Testing and validation provide that evidence.

Additionally, government must be accountable to the public. Citizens deserve to know that systems affecting their access to services are validated and fair. Testing documentation is part of that accountability.

THE FOUR DIMENSIONS OF AI TESTING

Overview

Testing AI systems isn't just about accuracy. You need to test across four dimensions:

DIMENSION 1

Does the system produce correct outputs? With what frequency? On what types of cases?

Accuracy testing answers questions like:

• What's the overall accuracy? (percentage of correct decisions)
• What's the precision? (of cases the system flags as positive, how many really are?)
• What's the recall? (of all actual positive cases, how many does the system catch?)
• What's the false positive rate? (incorrectly flagged as positive)
• What's the false negative rate? (incorrectly flagged as negative)
• How does accuracy vary by case type or complexity?

For government: Choose accuracy metrics that match your business problem.

• Benefits eligibility: You probably care about recall (catch all eligible cases) more than precision (some false positives are okay if they go to manual review)
• Fraud detection: You care about precision (don't falsely accuse people) and recall (catch actual fraud)
• Maintenance prediction: You probably care about recall (catch what needs fixing) more than precision (false alarms are low-cost compared to missed maintenance)

DIMENSION 2

Does the system treat all demographic groups equitably or does it discriminate?

Fairness testing answers questions like:

• Does the system have equal accuracy across demographic groups?
• Are approval/denial rates equal across groups?
• Do false positive rates vary by demographic group?
• Would the system face disparate impact liability in civil rights terms?
• Are there proxy variables in the system that encode protected characteristics indirectly?

Fairness concepts:

• Demographic parity: Equal approval rates across groups (50% approval for group A, 50% approval for group B)
• Equalized odds: Equal true positive rates across groups (system catches 85% of eligible cases in both groups)
• Calibration: Predictions are equally reliable across groups (when system says "80% confidence," that actually means 80% in group A and group B)
• Predictive parity: False positive rates equal across groups

For government: No single fairness metric is perfect. Work with legal and civil rights teams to define what fairness means for your specific system.

DIMENSION 3

How does the system perform on edge cases and adversarial inputs?

Robustness testing answers questions like:

• How does the system handle missing data?
• How does it handle inputs outside its training distribution?
• How does it handle intentionally malicious inputs?
• What happens when data changes dramatically (pandemic, economic shift, policy change)?
• How does the system fail? Gracefully or catastrophically?

Edge cases and adversarial examples are critical for government systems. Citizens encounter edge cases. Bad actors might intentionally craft malicious inputs. The system needs to handle both.

Examples of edge case testing:

• An eligibility system trained on population of 1 million suddenly receives 10 million applications. How does it perform?
• A hiring system trained on applications from January-March might see completely different patterns in September. How does it generalize?
• A fraud detection system might encounter a type of fraud it's never seen. How does it respond?

DIMENSION 4

Is the system ready for operational deployment with appropriate governance?

Readiness testing answers questions like:

• Can the system be monitored? (are we collecting the right data?)
• Are human override mechanisms in place?
• Can the system be audited? (is the decision trail clear?)
• Are incident response procedures documented?
• Is rollback possible if the system fails?
• Are success metrics being tracked?
• Is there a process for updating the model as conditions change?

DESIGNING TEST DATA AND TEST SETS

Your test data determines what you can actually know about your system. Bad test data = unreliable test results.

Key principles for test data:

• Test data must be separate from training data
• Test data must be representative of real-world usage
• Test data must include sufficient examples of edge cases and underrepresented groups
• Test data must be clean and accurate (if your test data is mislabeled, your test results are meaningless)

DESIGNING REPRESENTATIVE TEST SETS

Population the system will serve: All citizens applying for benefits

Test set should include: Demographics matching the actual application population

The error: Building a test set that only includes "typical" cases. Missing edge cases and minority populations.

How to avoid it: Get demographic data on your actual population. Build test set with proportional representation and deliberate oversampling of underrepresented groups. (Oversample in test set to ensure you have enough examples to measure performance on small groups accurately.)

STRATIFIED TESTING

Test your system separately on different population segments:

• Overall accuracy: 82%
• Accuracy for Group A: 80%
• Accuracy for Group B: 84%
• Accuracy for older applicants: 78%
• Accuracy for applicants from rural areas: 81%

This reveals where the system performs worse and where fairness concerns exist.

TEST SET SIZE AND COMPOSITION

Rule of thumb: You need at least 100 examples of each important subgroup to get reliable accuracy estimates.

• Test set size: At least 1000 cases for low-stakes systems, 5000+ for high-stakes systems
• Subgroup representation: If you're testing fairness, ensure each demographic subgroup has sufficient representation

VALIDATION FRAMEWORKS FOR DIFFERENT SYSTEM TYPES

Different system types need different validation approaches.

SYSTEMS THAT MAKE FINAL DECISIONS (HIGH GOVERNANCE BURDEN)

Examples: Benefit eligibility determination, loan approval, hiring decisions

Validation requirements:

• Accuracy testing on representative population: 95%+ confidence in results
• Fairness analysis across all protected classes: Document disparities if they exist
• Edge case testing: Demonstrate robustness to unusual inputs
• Human review testing: Show that human reviewers can effectively override system
• Post-deployment monitoring: Plan for continuous fairness and accuracy monitoring

Typical validation duration: 2-4 months, rigorous documentation

SYSTEMS THAT FLAG CASES FOR HUMAN REVIEW (MEDIUM GOVERNANCE BURDEN)

Examples: Fraud detection, anomaly flagging, prioritization systems

Validation requirements:

• Accuracy of flagging: What percentage of true cases are caught?
• False positive rate: How many false alarms?
• Fairness analysis: Do flagging rates vary appropriately by demographics? (might not need perfect parity if certain populations legitimately have different risk profiles)
• Operational validation: Do reviewers find the flags useful? What percentage of flags result in action?

Typical validation duration: 4-8 weeks

SYSTEMS THAT PROVIDE RECOMMENDATIONS (LOWER GOVERNANCE BURDEN)

Examples: Informational chatbots, decision support tools

Validation requirements:

• Accuracy on representative questions
• Fairness analysis (don't want recommendations that are biased)
• Basic robustness (what happens on off-topic queries?)

Typical validation duration: 2-4 weeks

TESTING FOR FAIRNESS AND BIAS

Overview

Fairness testing is increasingly critical for government AI. It's not optional.

FAIRNESS TESTING METHODOLOGY

Step 1: Define what fairness means for your system

• Work with civil rights/compliance teams
• Consider legal obligations (civil rights laws, Equal Protection principles)
• Define what disparities you can tolerate

Step 2: Identify protected characteristics and proxy variables

• Obvious protected characteristics: Race, gender, age, disability status
• Proxy variables: Zip code (often correlates with race), name (often indicates gender/ethnicity), hiring school (might correlate with socioeconomic status)
• The system might not explicitly use protected characteristics but encode them indirectly

Step 3: Measure accuracy and rates by demographic group

• Calculate accuracy separately for each group
• Calculate approval/denial rates by group
• Calculate false positive and false negative rates by group
• Compare rates across groups

Step 4: Assess disparities

• Are differences statistically significant or noise?
• Are differences within acceptable tolerances?
• Can differences be explained by legitimate factors (e.g., different legitimate fraud risk profiles) or indicate bias?

Step 5: Document findings and remediation if needed

• If no disparities: Document that analysis was done and results
• If disparities exist but acceptable: Document the disparity and justification
• If unacceptable disparities: Plan remediation before deployment

EXAMPLE: Benefits Eligibility System Fairness Testing

Overall accuracy: 82%

Breakdown by race:

• White applicants: 84% accuracy
• Black applicants: 79% accuracy
• Hispanic applicants: 81% accuracy

Disparity of 5 percentage points between White and Black applicants. Questions:

• Is this statistically significant? (with 5000 test cases, probably yes)
• Is it within acceptable tolerance? (probably not--that's a meaningful gap)
• What's causing it? (need to investigate)
• Can it be fixed? (maybe--could be training data bias, could be legitimate performance differences if populations have different documentation rates)

DOCUMENTING TEST RESULTS FOR GOVERNMENT DEPLOYMENT

Overview

Test documentation serves two purposes: internal (is the system ready?) and external (accountability to the public and oversight bodies).

MINIMUM DOCUMENTATION FOR GOVERNMENT AI SYSTEMS

• EXECUTIVE SUMMARY
• What was tested?
• Key findings?
• Ready for deployment? (yes/no/conditional)
• TEST METHODOLOGY
• What test set was used?
• How large? What was its composition?
• Who performed the testing?
• ACCURACY RESULTS
• Overall accuracy and confidence intervals
• Accuracy by demographic group
• Accuracy by case type
• Comparison to baseline/human performance
• FAIRNESS ANALYSIS
• Demographic performance analysis
• Identified disparities
• Assessment of legal risk
• Mitigation strategies if disparities exist
• ROBUSTNESS TESTING
• Edge cases tested
• Adversarial examples tested
• Results and system behavior
• LIMITATIONS AND CAVEATS
• What the system is NOT expected to do
• Populations or scenarios where accuracy is lower
• Known failure modes
• Data staleness issues
• GOVERNANCE READINESS
• Monitoring mechanisms in place? (yes/no)
• Human override procedures? (documented? tested?)
• Incident response procedures? (yes/no)
• Audit trail mechanisms? (yes/no)

ANTI-PATTERNS

ANTI-PATTERN 1: Testing Only on Best-Case Data

Testing is done on clean, representative data. System launches, encounters real data that's messier, includes edge cases, and accuracy drops 15%. System is now in crisis.

How to avoid it: Test on real data from production environment if possible. Include realistic data quality issues in your test set.

ANTI-PATTERN 2: Fairness Testing as Afterthought

Accuracy testing is thorough. Fairness testing is done in the last week before deployment as a checkbox exercise. Results show disparities that should have been addressed months ago.

How to avoid it: Do fairness testing in parallel with accuracy testing, from the beginning. Make fairness part of success criteria.

ANTI-PATTERN 3: Test Results Ignored

Testing reveals that the system has fairness problems or accuracy is below threshold. Team acknowledges findings but deploysanyway "we can fix it in production."

How to avoid it: Make test results binding. Don't deploy until results meet agreed-upon criteria. If you choose to deploy despite findings, document that explicitly with stakeholder sign-off.

ANTI-PATTERN 4: Vanishing Test Data

Testing is complete. System is deployed. Test data is deleted. Six months later, someone asks "what accuracy did we validate?" and there's no record.

How to avoid it: Archive test data and all test results. You'll need them for audits, incident investigations, and future maintenance.

ANTI-PATTERN 5: Over-Optimization on Test Set

Team runs test, finds underperformance on a specific demographic group, tweaks the model to optimize for that group on the test set. When deployed, the tweak doesn't generalize.

How to avoid it: Don't over-optimize to individual test results. If accuracy is significantly different for a demographic group, figure out why and address root causes, not symptoms.

PRACTICE PROMPTS

EXERCISE 1: Test Plan Design

Choose a government AI system (real or hypothetical). Design a comprehensive test plan:

• What are the four dimensions you'll test?
• What test set will you use? (size, composition, representativeness)
• What accuracy metrics matter most?
• What fairness metrics will you measure?
• What edge cases must you test?
• What would pass/fail criteria be?

EXERCISE 2: Fairness Analysis

Take a hypothetical system with accuracy results:

Overall: 85%

Group A: 88%

Group B: 82%

Group C: 83%

Analyze:

• Is there a fairness concern?
• What's the magnitude of disparity?
• What questions would you ask to understand root cause?
• Would you deploy with these results? (why/why not)

EXERCISE 3: Test Documentation

Write executive summary and limitations section for a benefits eligibility AI that:

• Achieved 81% accuracy overall
• Achieved 79% accuracy for historically disadvantaged groups
• Showed edge case issues with cases over 10 years old
• Achieved 85% accuracy on straight-forward eligibility determinations

KEY TAKEAWAYS

• Testing AI systems requires multiple dimensions: accuracy, fairness, robustness, and operational readiness. Accuracy alone is insufficient.
• Test data quality determines test result reliability. Invest in creating representative test sets that include edge cases and demographic diversity.
• Fairness testing is mandatory for government systems. Test for disparate impact. Document demographic performance.
• Stratified testing reveals where systems perform well and poorly. Test on population subgroups, not just overall.
• Document test methodology and results thoroughly. This documentation serves governance, accountability, and future audits.
• Test results should be binding--don't deploy if results don't meet criteria. Testing is a governance control, not a checkbox.
• Edge case and robustness testing is essential for government systems. Citizens encounter edge cases. The system must handle them.
• Testing doesn't end at deployment. Continuous monitoring and testing in production catches issues that lab testing missed.

GLOSSARY

Test Set: A collection of data examples used to evaluate the performance of an AI system, separate from the training data used to build the system.

Stratified Testing: Testing a system separately on different population subgroups to identify whether performance varies by demographic group or case type.

Fairness Metric: Quantitative measure of whether an AI system treats different demographic groups equitably (e.g., demographic parity, equalized odds, calibration).

Disparate Impact: A pattern of outcomes where a facially neutral practice (like an AI system) has a disproportionately negative impact on members of a protected class.

Edge Case: An unusual or extreme input that falls outside the system's normal operating range and tests the limits of system robustness.

Testing and validation are not quality control activities--they're governance mechanisms. They provide evidence that systems are ready for deployment and safe to use. They create accountability by documenting how systems work and what results they produce.

In practice, this means:

• Testing starts early (sprint 2, not sprint 8)
• Testing results are transparent to stakeholders
• Fairness testing is as important as accuracy testing
• Results are binding--systems shouldn't deploy without meeting criteria
• Testing results are preserved for audits and future reference

The teams and agencies that get testing and validation right deploy systems with confidence. They can point to documentation showing the system was validated. They can defend decisions if questioned. They catch problems before they affect the public.

Reflect on a time when you deployed something (software, process, policy) that turned out to have unforeseen problems. What would advance validation have caught? What testing would have revealed the problem before deployment? Use this reflection to develop intuition for comprehensive testing approaches.

You've learned the framework for comprehensive testing of government AI systems. The next lecture focuses on working with external vendors and contractors--how to manage them so they deliver tested, validated systems that meet your standards. Good vendor management ensures good test results.

See you in the next lecture.

Government AI CLUB Certification Program

Level 2: AI Ready | Testing and Validating AI Systems | Lecture 2.4.4

A GOVT.CLUB initiative

Visit: https://govt.club/learn/lectures/l2/244-testing-validating-systems.html

======================================================================

<- 2.4.3 Working with AI Vendors and Contractors
2.4.5 Change Management for AI Adoption ->

Start Your CLUB Certification

This lecture is part of L2: AI Practitioner -- 40 hours of comprehensive government AI training.

Explore CLUB Certification

Related Lectures

L2
2.4.1 -- How AI Projects Differ from Traditional IT
60 min - Video + Comparison

L2
2.4.2 -- Requirements Gathering for AI
60 min - Workshop

L2
2.4.3 -- Working with AI Vendors and Contractors
60 min - Video + Checklist