Learning Objectives

After completing this capstone lab, you will be able to:

• Classify a concrete federal AI use case (permit processing at a hypothetical USDA Rural Development grants office) as rights-impacting, safety-impacting, or neither, under the OMB M-24-10 framework and NIST AI RMF categories MAP-1.1 through MAP-5.2.
  - Produce a complete AI Requirements Document covering business context, problem statement with quantified KPIs, success criteria, data requirements, validation approach, human-oversight design, and post-deployment monitoring plan aligned to FISMA Moderate and FedRAMP baselines.
  - Draft a realistic 26-week project plan covering discovery, MVP, pilot, scale-up, and steady-state operations with named roles (CAIO, program manager, vendor lead, privacy officer, OIG liaison, GSA contracting officer, 18F support).
  - Specify a testing strategy covering accuracy, fairness, edge cases, adversarial robustness, and pre-deployment evaluation consistent with the NIST AI RMF Playbook MEASURE subcategories and lessons from the IRS ID.me, Michigan MIDAS, and Idaho Medicaid cases.
  - Build a 15-minute executive briefing that communicates risk, return, and operational readiness to agency leadership, inspired by the patterns used by USDS and 18F in interagency reviews.
  - Integrate validation, QA, bias detection, monitoring dashboards, human-in-the-loop, and incident response into a coherent steady-state operating model aligned with CISA cybersecurity practices.

Key Topics Covered

• Risk classification under OMB M-24-10 rights-impacting and safety-impacting definitions, with worked examples derived from Michigan MIDAS (rights-impacting), CMS eligibility prediction (rights-impacting), HHS vaccine-appointment scheduling (limited-risk), and CBP Traveler Verification (rights- and safety-impacting).
  - AI Requirements Document structure: business context, measurable KPIs, data provenance, validation plan, human-oversight pattern, monitoring, stakeholders.
  - 26-week project timeline: discovery (weeks 1-4), MVP build (weeks 5-10), pilot (weeks 11-16), scale-up (weeks 17-22), steady-state transition (weeks 23-26).
  - Testing and measurement: accuracy on representative data, fairness across protected classes, edge-case testing, adversarial robustness, drift monitoring, incident response.
  - Change management: stakeholder engagement, training plans, union coordination, public-notice rulemaking where required by 5 USC 552 (FOIA) and 5 USC 552a (Privacy Act).
  - Vendor management: FedRAMP authorization, ISO/IEC 42001 alignment, contract performance clauses, exit strategy.
  - Executive briefing structure: problem, solution, expected outcomes, investment and ROI, risks and mitigation, decision requested.

Why This Matters for Government

The capstone lab is where all Level 2 knowledge becomes operational. You have studied NIST AI RMF, OMB M-24-10, workflow analysis, use-case development, prompt engineering, vendor management, testing, change management, and measurement. The capstone integrates those into a single 26-week federal pilot that a Chief AI Officer or program manager could realistically run.

The scenario. You are a program manager at a USDA Rural Development grants office processing 5,000 permit applications per month. Current processing time averages 60 days. The success rate (approved without rework) is 65 percent. Three GS-13 analysts spend the equivalent of 1.8 FTE processing, at an annual cost of roughly $500,000. The problem statement: inconsistent rule application delays qualified applicants and denies claims that should succeed. Your task is to plan an AI-assisted system that reduces processing time to under 20 days, raises first-pass success rate to 85 percent, and produces legally defensible decisions.

Step one: risk classification. Under OMB M-24-10, rights-impacting AI is AI whose output materially affects a person's access to, or loss of, opportunities, rights, or benefits. Federal grant decisions affecting individuals or small businesses are rights-impacting. You must document this classification in your agency's AI use-case inventory with the identifier required by M-24-10 Section 5. The classification triggers minimum practices under Section 5(c), including pre-deployment testing, documentation of intended purpose and reasonably foreseeable misuse, human review of consequential outcomes, consultation with affected communities, and publication of the intended use. This is not bureaucracy; it is the lesson Michigan learned the expensive way with MIDAS, where unemployment fraud allegations were issued by an automated system without adequate human review and resulted in a $20 million settlement and restored benefits to thousands of claimants.

Step two: requirements. The AI Requirements Document should include business context (why USDA is doing this, what citizens it serves, why now), quantified problem statement (60-day average, 65 percent first-pass, 5,000 applications/month, $500,000 annual cost), success criteria (under 20 days, 85 percent first-pass, equal or better fairness across demographic groups, demonstrable reduction in arbitrary decisions), data requirements (application fields, supporting documents, historical disposition data, legal rules as features), system functionality (document extraction, rule application, confidence scoring, human handoff), validation approach (pre-deployment evaluation on held-out data, fairness tests, adversarial tests, user acceptance), human oversight design (every denial reviewed by a human; all edge cases routed to human; human can override approvals), monitoring plan (drift, fairness, accuracy, user feedback), and stakeholders with their requirements (applicants, USDA program staff, USDA OIG, USDA privacy officer, GAO, OMB, potentially Congress).

Step three: project plan. Weeks 1-4 are discovery: stakeholder interviews, privacy impact assessment (42 USC 4321 NEPA analog for data systems), FISMA categorization and security categorization, data-sharing agreements where needed, confirmation of M-24-10 inventory entry, and initial documentation. Weeks 5-10 are MVP: develop or procure a model, build the human-review interface, instrument logging, run an internal dry-run on 200 historical applications, compare to ground truth. Weeks 11-16 are pilot: 20 percent of incoming applications routed through the AI-assisted flow, strict human review on every denial, weekly fairness and accuracy reports to the CAIO. Weeks 17-22 are scale-up: gradually expand to 100 percent of applications, preserve human review for denials and edge cases, monitor drift. Weeks 23-26 are steady-state transition: finalize operating procedures, update training, publish results, document lessons learned, close out pilot budget.

Step four: testing strategy. Accuracy testing on a representative sample of historical applications, including legally difficult cases. Fairness testing across applicant demographics where collected, with attention to disparate-impact patterns. Edge-case testing on applications with unusual field combinations, missing documentation, or foreign-language support. Adversarial testing to probe robustness to manipulation. Drift monitoring to catch silent degradation. The testing strategy should cite NIST AI RMF MEASURE-2.1 through MEASURE-2.11 and, where vendor-supplied, the EU AI Act Article 15 robustness obligations. Lessons from cases like COMPAS (recidivism risk scores with disparate fairness), Houston HISD (EVAAS teacher-evaluation overturned on due-process grounds), and SF DA tool (race-redaction experiment with valuable lessons) should inform your fairness-testing choices.

Step five: measurement framework. KPIs: processing time (target under 20 days), first-pass success rate (target 85 percent), human-review rate (target: 100 percent of denials, 100 percent of edge cases, sample of approvals), applicant-satisfaction scores, appeal rates, cost per processed application. Baseline: pre-deployment measurement of current process. Attribution: quasi-experimental design comparing pilot cohort to control cohort where feasible, with difference-in-differences analysis. Post-deployment measurement: monthly dashboard to CAIO, quarterly report to OMB, annual report public-facing.

Step six: executive briefing. A 15-minute structured briefing covers the problem, the proposed solution, expected outcomes (quantified), investment and ROI, risks and mitigation, and the decision requested. For a USDA Under Secretary or a congressional staffer, this is not a technical presentation; it is a decision brief that answers 'why should we do this, what could go wrong, and how do we know when we are done?' Patterns from USDS and 18F: show a short live demonstration if at all possible; present one worst-case scenario and its mitigation; name the decision requested explicitly.

Step seven: validation plan. This includes the QA checklist applied to every deployment release, confidence calibration (how model confidence maps to human review thresholds), bias detection pipelines, monitoring dashboards, human-in-the-loop patterns, incident response (who is paged, within what time, with what playbook), and the quality culture that sustains this over time. The Idaho Medicaid case is instructive: an opaque algorithm reduced benefit allocations without explainable justification, leading to successful litigation and a system roll-back. The lesson: validation and explainability are not nice-to-haves; they are prerequisites.

Evaluation criteria for the capstone: completeness, appropriateness, specificity, integration, feasibility, and governance. Specificity matters most; generic plans fail in practice. A good capstone reads like something a reviewer at the Office of Management and Budget or GAO could hold in their hands and understand. Connect to your real agency context where possible. Document your assumptions. Show your decision-making. Integrate concepts across lectures. This is what separates AI-ready practitioners from AI-curious observers.

Common pitfalls: underestimating privacy and civil-rights review time; skipping stakeholder engagement with union representatives where employees are affected; failing to design for appeal and contestation; treating vendor tools as black boxes without evidence packages; conflating pilot-scale success with scale-up success; neglecting drift monitoring; and producing generic documentation that does not engage with the specific statute (USDA Rural Development is governed by specific authorities) and the specific population served.

Related Lectures

L2
2.2.1 -- Workflow Analysis: Finding AI Opportunities
60 min - Workshop

L2
2.2.2 -- Building an AI Use Case: From Idea to Business Case
60 min - Workshop

L2
2.2.3 -- Prompt Engineering Mastery: Structured Prompts
60 min - Hands-On Lab