Learning Objectives

By the end of this lecture you will: (1) articulate at least seven structural differences between traditional IT projects and AI projects, ranging from determinism to data dependency to validation strategy; (2) map each difference to a specific governance artifact required by OMB Memorandum M-24-10, including impact assessments, AI use case inventories, and minimum risk management practices for rights-impacting and safety-impacting AI; (3) rewrite a sample Statement of Work for a government IT project so that it accommodates iterative AI discovery phases without violating Federal Acquisition Regulation constraints; (4) identify at least three historical government AI failures, such as the Michigan MIDAS unemployment fraud system (which falsely accused more than 40,000 claimants), the IRS ID.me facial recognition rollout, and the Dutch childcare benefits scandal (toeslagenaffaire), and explain which traditional-IT assumptions led to each failure; (5) propose a modified risk management approach aligned with the NIST AI Risk Management Framework 1.0 Map-Measure-Manage-Govern functions that replaces the classic waterfall risk register; (6) explain why fixed-price, fixed-scope contracting under FAR Part 15 is often unsuitable for model development phases and when Other Transaction Authority (OTA) or time-and-materials vehicles are appropriate; and (7) calibrate stakeholder expectations so that probabilistic outputs are not interpreted as deterministic guarantees during deployment.

Why This Matters for Government

Government agencies have spent four decades refining frameworks for traditional IT: the Clinger-Cohen Act, FITARA, TechFAR, the Federal Information Security Modernization Act (FISMA), OMB Circular A-130, and the Federal Risk and Authorization Management Program (FedRAMP). These frameworks assume that a system, once defined and authorized to operate, behaves predictably. Patch it, monitor it, reauthorize every three years, done. AI breaks most of these assumptions. A large language model that passes an Authority to Operate on Monday may produce different outputs on Tuesday because of prompt injection, context drift, or a silent vendor update. When the Internal Revenue Service deployed ID.me facial-recognition identity proofing in 2021 and 2022, it was treated as a classic vendor integration, with the result that millions of taxpayers lost access to their accounts and the agency was forced into an emergency policy reversal after civil-society complaints and congressional pressure. When Michigan deployed the MIDAS automated unemployment fraud detection system between 2013 and 2015, the agency ran it under traditional IT acceptance testing and a 93 percent false positive rate went undetected for two years, generating more than $20 million in wrongful penalties that the state later had to refund. The Netherlands equivalent, the SyRI and toeslagenaffaire childcare benefits scandal, resulted in the collapse of Prime Minister Mark Rutte's cabinet in 2021. These are not engineering failures in the traditional sense; they are governance failures caused by applying IT-era assumptions to AI-era systems. OMB Memorandum M-24-10, issued March 28, 2024 by Director Shalanda Young, is the first federal instrument that treats AI as categorically different, requiring each CFO Act agency to designate a Chief AI Officer, publish an annual use case inventory, and implement minimum practices for rights-impacting and safety-impacting AI before December 1 of each year. Executive Order 14110, signed October 30, 2023, set the broader direction. The EU AI Act, which entered force August 1, 2024, creates parallel obligations for high-risk systems and extraterritorial exposure for U.S. vendors selling into Europe. ISO/IEC 42001:2023 provides a certifiable AI management system standard. Understanding why AI projects differ from traditional IT is therefore not a philosophical exercise; it is the prerequisite for satisfying a concrete, enforceable, and rapidly expanding compliance regime. Program managers who cannot articulate these differences cannot negotiate realistic contracts, cannot staff teams with the right skill mix, cannot brief inspectors general credibly, and cannot respond to Government Accountability Office inquiries under the framework GAO published in June 2021.

Determinism vs Probabilistic Behavior

Traditional IT systems are deterministic at the functional level. A correctly built payroll system pays the same net amount for the same inputs every pay period. An SQL query against a stable table returns identical rows. These behaviors can be unit-tested with fixed assertions, regression-tested against a golden dataset, and accepted against a binary pass/fail criterion. AI systems, particularly those built on machine learning, produce probabilistic outputs whose correctness is defined in aggregate rather than per instance. A benefits-eligibility classifier operating at 92 percent accuracy will be wrong eight times out of a hundred, and which eight is not predictable. A large language model generating summaries of Freedom of Information Act requests may vary wording, emphasis, or factual grounding between calls even with identical prompts, because of sampling temperature, context window reshuffling, or model provider A/B testing. This has four concrete implications for government. First, acceptance criteria must be stated as distributions, not points: not 'the model shall classify correctly' but 'the model shall maintain at least 90 percent precision and 85 percent recall on the validation set, with no demographic subgroup dropping more than five percentage points below aggregate'. Second, monitoring must continue after go-live, because performance can drift even when the code has not changed. NIST AI RMF 1.0 Measure function explicitly addresses this. Third, the concept of an Authority to Operate must be extended with conditions for re-evaluation, which OMB M-24-10 requires for rights-impacting AI. Fourth, incident response must be designed for statistical failures, not just outages; the relevant question is not 'is it up?' but 'is the error distribution still within tolerance?'.

Data Dependency and Data Debt

Traditional IT depends on code; AI depends on data. This sounds glib until you consider the operational consequences. A code bug is usually reproducible, localizable, and fixable by a developer with access to a repository. A data bug manifests as degraded model output for a subset of cases, and tracing it back to a mislabeled training example, a shift in upstream data collection, or an entire population that was never represented in the training set requires tooling that most agencies lack: data lineage, data quality monitoring, data drift detection, and model explainability. The Epic Sepsis Model, deployed by Epic Systems to hundreds of hospitals, was found in 2021 by University of Michigan researchers (Wong et al., JAMA Internal Medicine) to perform far below vendor claims, with sensitivity of only 33 percent versus the advertised 76 percent, in part because it was trained on data that did not generalize. IBM Watson for Oncology, deployed at MD Anderson Cancer Center starting in 2013, was quietly shelved in 2017 after training on hypothetical cases rather than real patient data produced unsafe recommendations. Optum's chronic care algorithm, analyzed by Obermeyer et al. in Science 2019, encoded racial bias because it used healthcare spending as a proxy for medical need, systematically understating need for Black patients who received less spending. In each case, the failure was not in the code; it was in the data. Government agencies face the same risk with the added complication that their data was typically collected for a different purpose, governed by Privacy Act System of Records Notices, and subject to the E-Government Act Privacy Impact Assessment regime. Using pre-existing administrative records to train AI models often requires a fresh PIA, updated SORNs, and sometimes new Paperwork Reduction Act clearance. Data debt, the accumulated cost of cleaning, documenting, and governing data that was collected without AI use in mind, commonly dwarfs the cost of model development itself, and traditional IT project budgets almost never account for it.

Iterative, Hypothesis-Driven Development

Traditional IT projects follow a Systems Development Life Cycle with relatively predictable phases: initiation, requirements, design, construction, integration, operations, disposition. Even agile variants, now dominant in federal digital service work, still assume that each sprint produces demonstrable, accepted functionality. AI projects instead operate in a research-development loop closer to applied science: hypothesis, dataset, baseline, experiment, evaluate, iterate. Ninety percent of experiments fail or underperform a baseline. A team may spend six weeks demonstrating that an approach does not work, which is a valuable negative result in science but difficult to justify under classical earned-value management. Jennifer Pahlka, former U.S. Deputy CTO and author of Recoding America, argues that the mismatch between probabilistic discovery and waterfall governance is the single largest obstacle to useful AI in the federal government. Concrete implications include: contracting vehicles must allow for exploratory phases where the deliverable is a validated learning, not a built feature; program reviews must be structured around Go/No-Go gates at experiment boundaries, not Gantt-chart milestones; and budgetary rules must tolerate an unknown number of experiments before production investment. The Defense Innovation Unit and the General Services Administration Technology Transformation Services' 18F have piloted modular contracting and agile BPA vehicles that accommodate this pattern, but most civilian agency contracting officers still default to fixed-price, fixed-scope structures that are hostile to AI discovery work.

Ambiguous Requirements and Measurable Outcomes

Traditional IT requirements are expressed in terms of features and data flows: 'the system shall allow the user to upload a PDF', 'the system shall process 10,000 transactions per hour'. These are verifiable by direct observation. AI requirements, by contrast, are frequently aspirational: 'the system shall help claims adjusters identify fraud'. What does 'help' mean? At what precision and recall tradeoff? What is the baseline to be helped beyond? Translating these aspirations into measurable, contestable specifications requires a discovery phase that traditional IT methodologies treat as already complete before contract award. OMB M-24-10 implicitly acknowledges this by requiring agencies to document the intended use, expected benefits, risks, and mitigations for each AI use case before deployment, which forces the specification work upfront, but does not remove the need for it to evolve as experiments reveal what is actually possible. Good program managers plan explicitly for two phases: a discovery or pilot phase whose acceptance criterion is a well-specified production requirement, and a production phase whose acceptance criterion is the operationalization of that requirement. Trying to collapse these into a single contract invariably produces either an unsuccessful product or a painful contract modification.

Validation, Testing, and Red Teaming

Traditional IT validation relies on unit tests, integration tests, system tests, user acceptance tests, and operational readiness reviews. These are well-understood in the Capability Maturity Model Integration, the Systems Engineering Body of Knowledge, and DoD Instruction 5000.87 for software acquisition. AI validation requires additional techniques foreign to most government QA teams: statistical performance evaluation across stratified subgroups, bias and fairness testing against protected classes under the Civil Rights Act and the Equal Credit Opportunity Act, robustness testing against adversarial inputs, red-teaming for prompt injection and jailbreaks, data-leakage testing for membership inference, and continuous monitoring for concept drift. Executive Order 14110 Section 4.1 directs NIST to publish guidelines for AI red-teaming, which became NIST AI 600-1 Generative AI Profile in July 2024 and is being extended through the new U.S. AI Safety Institute. The Department of Defense has its own framework, the Responsible AI Strategy and Implementation Pathway (RAI S&I). Procurement-wise, Requests for Proposals must specify which validation techniques are required, how evidence will be delivered, and who bears the cost of red-teaming. Historically, this has been a major point of contract dispute, with vendors arguing that extensive red-teaming was not in scope and agencies arguing that it is implicit in the duty of safe operation.

Governance, Workforce, and Operating Model

Traditional IT governance is organized around CIOs, information system owners, system security officers, and privacy officers, with responsibilities codified in FISMA and the Privacy Act. AI adds a new set of roles: Chief AI Officer (mandated by OMB M-24-10 for each CFO Act agency), AI Governance Board, AI model owner, data steward, algorithmic accountability officer, and human oversight officer for high-risk systems. Agencies such as GSA, DHS, VA, HHS, CMS, FDA, NSA, DOD, State, and the IRS have stood up governance bodies with varying maturity, and CISA has published AI security guidance leveraging the MITRE ATLAS adversarial ML taxonomy. Workforce implications are equally profound. Federal IT workforces have been structured around Occupational Series 2210 information technology specialists; AI work requires additional competencies in statistics, machine learning, data engineering, MLOps, behavioral science, and human-computer interaction, often spread across Series 1515 (operations research), 1530 (statistics), and 0110 (economist), among others. OPM has begun updating qualification standards but the pace lags demand. Culturally, AI teams function better with researcher-like autonomy and tolerance for failure, while traditional IT shops reward predictability; forcing one model on the other is a common cause of attrition among scarce AI talent. A final operational difference: AI systems require live feedback loops from end users to keep models aligned with reality, creating a dependency on product management disciplines that are weakly represented in many agencies.

Procurement and Contracting Implications

Under FAR Part 15 negotiated procurements and Part 16 contract types, the default for defined-scope work is fixed-price, which minimizes agency risk but presumes that scope is knowable. AI discovery work is rarely scoped precisely enough for fixed-price; time-and-materials or cost-reimbursement contracts fit better for the exploratory phase, transitioning to firm-fixed-price for production operations once requirements stabilize. Other Transaction Authority under 10 U.S.C. 4022 (DoD) or 42 U.S.C. 7256b (DOE) allows more flexibility and has been used extensively by the Defense Innovation Unit, DARPA, and the Advanced Research Projects Agency for Health. The GSA Multiple Award Schedule now includes AI-specific SINs, and agencies can leverage the 18F Agile Blanket Purchase Agreement model. Contracting officers and their technical representatives must understand AI-specific acceptance criteria, or they risk either accepting systems that do not work or rejecting systems that work perfectly but were measured against the wrong benchmark. Intellectual property clauses need special attention: who owns fine-tuned model weights trained on government data? Under DFARS 252.227 and FAR 52.227, rights in technical data frameworks predate modern AI and are being reinterpreted case by case. Data use and re-use clauses must address whether vendor models can be trained on agency data, whether agency data can leave the authorization boundary, and whether the vendor's base model itself was trained on data the government considers sensitive.

Case Studies and Anti-Patterns

Michigan MIDAS (2013-2015): The Michigan Integrated Data Automated System flagged more than 40,000 unemployment claimants as fraudulent with a false positive rate later measured at roughly 93 percent, because acceptance tests treated the automated decisions as deterministic and no human-in-the-loop review caught the statistical error pattern. The state paid more than $20 million in settlements and was still in litigation as of 2023. IRS ID.me (2022): The agency contracted facial-recognition identity proofing from ID.me on a compressed timeline without a Privacy Impact Assessment update adequate to the scale of deployment; after civil society and Senator Ron Wyden objections, the Treasury reversed course and pursued Login.gov as an alternative. Dutch toeslagenaffaire (2005-2021): The Belastingdienst used an opaque risk-scoring algorithm that flagged immigrant families for fraud investigation at disproportionate rates; tens of thousands of families had childcare benefits clawed back unjustly, and the scandal toppled the Rutte III cabinet in January 2021. COMPAS (ProPublica 2016): Northpointe's recidivism scoring tool, used across state court systems, was shown by ProPublica to have disparate error rates by race, with Black defendants more likely to be falsely labeled high risk; the academic debate that followed produced important clarifications about fairness metrics. Epic Sepsis Model (2021): Widely deployed across hospitals, found by Wong et al. in JAMA Internal Medicine to perform far below vendor-reported accuracy, demonstrating the danger of accepting vendor marketing claims without independent validation. IBM Watson for Oncology (2013-2017): Trained on hypothetical rather than real cases at MD Anderson, produced recommendations described internally as 'unsafe and incorrect', project terminated. Optum chronic-care algorithm (Obermeyer et al. 2019): Encoded racial bias by using cost as a proxy for medical need, affecting more than 200 million Americans. In every case, the common failure mode was applying traditional-IT acceptance thinking to a probabilistic system and skipping the fairness, drift, and stratified-performance analyses that would have surfaced the problem earlier.

Applying the NIST AI Risk Management Framework

The NIST AI RMF 1.0, published January 26, 2023, provides four functions that replace the traditional IT risk register approach. GOVERN establishes policies, accountability, and culture; this is where Chief AI Officer authorities and AI Governance Boards live. MAP contextualizes AI risks to the specific use case, stakeholder community, and lifecycle stage; this is where historical failures like MIDAS and Optum should be introduced as analog cases. MEASURE quantifies trustworthiness characteristics: validity and reliability, safety, security and resilience, accountability and transparency, explainability and interpretability, privacy, and managed bias. MANAGE prioritizes responses, allocates resources, and makes go/no-go decisions. Crosswalks between RMF and the NIST Cybersecurity Framework, the Secure Software Development Framework (SP 800-218), and ISO/IEC 42001:2023 exist and should be used to avoid duplicate compliance work. The Generative AI Profile (NIST AI 600-1) adds twelve generative-specific risks including CBRN information, confabulation, dangerous recommendations, and information integrity. Applying RMF is more than a paper exercise: it changes which documents get produced, which committees review them, and which decisions are delegated versus retained at the Chief AI Officer or Deputy Secretary level.

Practical Checklist for AI Program Managers

(1) Before the contract is signed, articulate what will be known at the end of the discovery phase that is not knowable today, and structure the contract so that the discovery phase produces a validated production requirement. (2) Draft acceptance criteria as statistical distributions, not points. (3) Budget 40-60 percent of effort for data work: collection, cleaning, labeling, documentation, lineage, and ongoing governance. (4) Register the use case in the agency AI inventory per OMB M-24-10. (5) Classify as rights-impacting, safety-impacting, or neither, and apply the associated minimum practices. (6) Complete or update the Privacy Impact Assessment and relevant System of Records Notices before production. (7) Specify red-teaming and adversarial testing responsibilities in the Statement of Work. (8) Plan for Authority to Operate conditions that trigger re-authorization on drift, not just on a three-year clock. (9) Establish a human-in-the-loop or human-on-the-loop design for all rights-impacting and safety-impacting systems. (10) Define sunset criteria: under what conditions will the AI be turned off, not just upgraded. This checklist does not replace agency-specific policy but provides a floor that any credible AI program manager should be meeting in 2026.

Summary and Next Steps

AI projects differ from traditional IT along at least a dozen structural axes: determinism, data dependency, iteration style, requirement specificity, validation technique, governance structure, workforce composition, contracting vehicle, operational monitoring, incident response, accountability framework, and sunset criteria. Treating AI as a new kind of IT produces predictable failures at predictable intervals: MIDAS, ID.me, toeslagenaffaire, COMPAS, Epic Sepsis, Watson Oncology, Optum. Treating AI as genuinely new, and applying OMB M-24-10, NIST AI RMF, EO 14110, ISO/IEC 42001, FedRAMP, FISMA, HIPAA where applicable, and agency-specific guidance together, produces the conditions under which AI can deliver its promise without destroying public trust. The next lecture, 2.4.2 Requirements Gathering for AI, applies these principles to the concrete artifact of the AI requirements document, distinguishing capability requirements, data requirements, performance requirements, fairness requirements, and operational requirements, and showing how to negotiate them with stakeholders who expect traditional IT determinism.