Learning Objectives

After completing this L3 lecture, senior federal AI engineers, Chief Data Officers, and program leads will be able to: first, articulate the reference architecture for federal enterprise AI data infrastructure, including ingestion, storage, lineage, catalog, feature store, evaluation harness, and serving, mapped onto FedRAMP-authorized cloud services and aligned with FISMA and NIST 800-53 controls; second, diagnose whether an agency's existing data infrastructure can support production AI or requires targeted investment in specific layers such as the data catalog, lineage, feature store, or model evaluation harness; third, design a data governance program that satisfies OMB Circular A-108 on Privacy Act records, the Evidence-Based Policymaking Act, the DATA Act's spending transparency requirements, the agency-specific data strategy, and OMB M-24-10's AI data governance obligations; fourth, implement data quality, lineage, and provenance controls sufficient to support the NIST AI Risk Management Framework's MAP and MEASURE functions and the EU AI Act Article 10 requirements on training data quality for high-risk systems; fifth, navigate FedRAMP authorization categories for data-intensive AI workloads, including FedRAMP High for CUI and DoD IL5 for defense workloads, and understand the inheritance and supplement model; and sixth, coordinate with the Chief Data Officer Council, the Federal Data Strategy, and agency Data Governance Boards to align AI-specific infrastructure with broader federal data modernization.

Key Topics Covered

The lecture covers nine topics. First, the federal enterprise AI reference architecture from ingestion through serving, with explicit tie-ins to FedRAMP-authorized services (AWS GovCloud, Azure Government, Google Public Sector) and federal engineering standards. Second, the federal data governance landscape including the Chief Data Officer Council, the Federal Data Strategy, the DATA Act, Evidence-Based Policymaking Act, OMB Circular A-108, Paperwork Reduction Act, and OMB M-24-10. Third, data catalog and discovery, using tools such as DataHub, Alation, and OpenMetadata, with federal deployment patterns. Fourth, data lineage and provenance, including OpenLineage, Apache Airflow metadata, and git-based lineage for model artifacts. Fifth, feature stores including Feast, Tecton, and AWS SageMaker Feature Store in federal environments. Sixth, data quality pipelines including Great Expectations, Monte Carlo, and custom statistical drift detectors. Seventh, model evaluation harness design covering offline evaluation, backtesting, fairness testing, and shadow deployment. Eighth, streaming and real-time AI data infrastructure, covering Apache Kafka, AWS Kinesis, and agency-grade event mesh patterns. Ninth, classification boundary design and data movement between classification levels, including cross-domain solutions and DoD IL5 inheritance.

Why This Matters for Government

Enterprise AI in the federal government is ultimately a data problem. A brilliant model trained on poor, ungoverned, or incomplete data produces decisions the agency will regret. A mediocre model trained on clean, well-governed, well-understood data produces decisions the agency can defend. The difference is rarely the algorithm; it is almost always the data infrastructure. This lesson is well documented in GAO reports, Inspector General audits, and Chief Data Officer Council after-action reports. Agencies that invest in their data infrastructure before scaling AI capture the benefits. Agencies that skip the data investment and lunge directly at AI build systems they cannot trust, cannot audit, and cannot defend.

Federal data infrastructure is not the same as commercial data infrastructure. Commercial firms can assemble data pipelines largely on their own terms. Federal agencies operate inside a specific stack of statutory, regulatory, and policy constraints. The Privacy Act of 1974 and OMB Circular A-108 constrain how personally identifiable information moves through federal systems. The Evidence-Based Policymaking Act of 2018 and the DATA Act require specific data standards and transparency. FISMA and NIST 800-53 impose security control families across every data layer. FedRAMP authorization is required for cloud-hosted infrastructure. OMB M-24-10 layered specific AI data governance obligations on top of these in 2024. And agency-specific statutes, such as 26 USC 6103 for tax data or HIPAA for health data, constrain specific data categories further. A data engineer designing federal AI infrastructure who does not know this stack will build something that cannot be authorized. A data engineer who does know it can build an infrastructure that moves at the speed of modern AI while meeting every one of these constraints.

The downstream consequences of weak data infrastructure are predictable. Without a data catalog, nobody knows what data exists, and AI projects repeatedly rediscover or duplicate datasets. Without lineage, nobody can explain where a model's training data came from, which is fatal in an IG or GAO audit. Without a feature store, every AI team invents its own features, producing inconsistency across systems and hidden bias. Without a quality pipeline, silent data issues produce silent model degradation, often discovered only when a beneficiary complains or an auditor asks. Without an evaluation harness, models drift without anyone noticing, and shipping a new model to production is a terrifying leap of faith. Each of these weak spots is a concrete audit finding waiting to happen. Each of them is also addressable with proven tools and design patterns that this lesson covers.

The Federal Enterprise AI Data Reference Architecture

A defensible federal enterprise AI data architecture has seven layers, each FedRAMP-authorized at an appropriate level for the data it holds. Layer one is ingestion, bringing data from authoritative source systems into the AI environment under documented data use agreements and SORN coverage. Layer two is the raw data lake or bronze layer, storing data in its original form with immutable logs of what arrived and when. Layer three is the curated data warehouse or silver layer, where data is cleaned, conformed, and governed with schemas, access controls, and documented provenance. Layer four is the data catalog, which makes everything in bronze and silver discoverable, with metadata about owner, lineage, quality, sensitivity, and authorized uses. Popular federal-deployed tools include DataHub, Alation, and OpenMetadata.

Layer five is the feature store, which exposes model-ready features consistently across training, online serving, and batch inference. Feast is the open-source reference; Tecton and AWS SageMaker Feature Store are commercial options available in federal environments. The feature store is where an agency ensures that the feature a model was trained on is identical to the feature it sees at inference, a subtle issue that causes more silent model failures than any other single cause. Layer six is the model training and evaluation harness, which takes features, runs training, and captures evaluation metrics including accuracy, fairness, and calibration. This layer integrates with experiment tracking tools such as MLflow and Weights & Biases in federal-approved variants. Layer seven is serving, where the model is exposed to calling applications at the latency and throughput the use case requires. Serving must support model versioning, shadow deployment, rollback, and audit logging.

Cutting across these seven layers are four cross-cutting concerns. Security is the FedRAMP/FISMA/800-53 control implementation across every layer. Privacy is the Privacy Act and agency-specific statute compliance, with SORN coverage and CMA execution as needed. Quality is the data and feature validation pipeline that catches drift before it corrupts decisions. Observability is the metric, log, and trace infrastructure that lets engineers diagnose failures. An agency can assemble this architecture from FedRAMP-authorized building blocks; it does not need to build everything from scratch. The design choice is which components to use, how to integrate them, and what governance to wrap around them.

Federal Data Governance for AI

Data governance is the non-glamorous discipline that determines whether the data architecture actually serves mission. The baseline federal data governance stack includes the Chief Data Officer Council (interagency), the Federal Data Strategy (the cross-government plan published by OMB), the agency Chief Data Officer (required by the Evidence-Based Policymaking Act of 2018), the agency Data Governance Board, and the Senior Agency Official for Privacy under 5 CFR Part 5b. For AI, OMB M-24-10 adds AI-specific data governance obligations, including documented data provenance, data minimization, and disaggregated monitoring for rights-impacting uses. An agency whose data governance does not integrate these bodies and requirements will drift, and drift is expensive.

A strong agency AI data governance program has five practices. First, a canonical data inventory maintained by the Chief Data Officer's office, cross-referenced to authoritative sources, agency SORNs, and the Evidence-Based Policymaking Act data compendium. Second, a Data Governance Board that reviews new data uses for legal basis, privacy impact, security posture, and mission alignment. Third, a documented data minimization discipline, where AI projects must justify any PII or CUI they ingest and confirm that no less-sensitive alternative would serve the mission. Fourth, disaggregated quality and performance monitoring for any rights-impacting use, as required by OMB M-24-10 and aligned with NIST AI RMF MEASURE. Fifth, a data incident response plan that names specific actions when data quality or privacy issues are detected, with coordination paths to the agency CISO, Privacy Office, and the CAIO.

Cross-agency data governance amplifies this. The Chief Data Officer Council shares templates and practices across agencies. The Federal Data Strategy provides a shared frame. Agencies that participate actively move faster and avoid reinventing governance, while agencies that go it alone spend years on problems peer agencies have already solved. A federal data engineer designing AI infrastructure should know which interagency resources apply and where to plug in.

Data Lineage, Provenance, and Quality

Lineage and provenance are the audit trail of federal AI. For any model decision that affects a citizen, the agency must be able to answer three questions: what data was used to train the model, what data was used at inference time, and how has the training-to-inference distribution changed over time. Without lineage infrastructure, these questions cannot be answered except by labor-intensive manual reconstruction, which is slow, error-prone, and indefensible under audit. OpenLineage, an open standard for data lineage, is increasingly deployed in federal environments. Apache Airflow, Prefect, and Dagster all integrate with OpenLineage to automatically capture lineage as pipelines run. Model lineage extends this to the model artifact, using tools like MLflow Model Registry and DVC to track which code, data, and hyperparameters produced each model version.

Provenance is closely related but includes external context: what authoritative source produced the data, what agreements govern its use, and what statutes apply. Provenance is what an IG or GAO auditor asks for when they want to understand whether an AI decision was based on lawful data. A federal data infrastructure should maintain provenance metadata in the catalog layer, with explicit links between datasets and the agreements, SORNs, and statutes that authorize their use.

Data quality is the active practice of detecting issues before they corrupt decisions. A federal data quality pipeline runs automated checks on each data flow: schema conformance, value ranges, uniqueness constraints, referential integrity, and statistical drift from baseline distributions. Great Expectations is the canonical open-source tool. Monte Carlo and Databand are commercial options. Drift detection is specifically important for AI: even if data conforms to schema, the distribution of values may shift, causing model performance to degrade silently. Federal AI teams should alert on drift at the 95th percentile or higher, with clear escalation paths that tie into the agency's incident response.

Feature Stores and Model Evaluation Harness

A feature store is the layer that ensures consistency between training-time and inference-time data. Without a feature store, a data scientist who computes 'average income last 12 months' in training using one calculation, and an engineer who computes it at inference using a slightly different calculation, produces silent training-serving skew, which can substantially degrade model performance. The feature store centralizes the definition and execution of features, so training and inference use the same code path. Feast is the reference open-source implementation; Tecton and SageMaker Feature Store are commercial. In federal environments, feature stores must be deployed within the FedRAMP boundary, integrated with the agency's data catalog, and subject to the same access controls as the underlying data.

The model evaluation harness is the layer that tells the agency whether a model is good enough to ship. A federal-grade evaluation harness includes offline evaluation on a held-out test set, backtesting on historical data, fairness evaluation with disaggregated metrics across protected classes, calibration analysis, robustness testing for distributional shift, and shadow deployment where the new model runs alongside the current one without affecting decisions. Results from the harness feed into the AI Governance Board's review and the CAIO's sign-off. OMB M-24-10 requires documented evaluation for rights-impacting uses, and ISO/IEC 42001 adds ongoing operational evaluation expectations. Agencies that industrialize this harness can ship new models in days with confidence; agencies without it take weeks and still make leaps of faith.

A common trap is to treat evaluation as a pre-production task only. Ongoing production evaluation is equally important. A federal AI system should have continuous evaluation pipelines that recompute key metrics daily or weekly on recent production data, with alerts when metrics diverge from acceptance thresholds. This is how an agency catches model drift, data shift, or subtle regressions before they produce harm. Tools like Evidently AI, Arize, and custom dashboards built on the data warehouse all serve this function. The discipline is the same as other quality disciplines: automated detection, documented thresholds, clear escalation.

FedRAMP Authorization and Classification Boundaries

Federal AI data infrastructure must be hosted in FedRAMP-authorized environments at the appropriate impact level. FedRAMP Moderate is the common baseline for unclassified non-sensitive data. FedRAMP High is required for Controlled Unclassified Information, including PII in many categories. DoD IL5 is required for certain defense-sensitive unclassified workloads. Classified workloads require accredited classified environments outside of the civilian FedRAMP marketplace. An agency's data architecture must be explicit about which data lives at which level, with clear rules about data movement between levels. Moving data from High to Moderate is a potential CUI spill; moving data from IL5 to IL4 requires formal cross-domain solutions.

The FedRAMP authorization inheritance model lets an agency reuse another agency's authorization for a specific service, reducing duplicate ATO work. The authorization inheritance requires the receiving agency to review the control implementations, accept the residual risks, and add agency-specific supplements where needed. For AI infrastructure components, inheritance can cut authorization time significantly, though inheritance is not a substitute for independent review. Agencies that set up authorization leveraging as a standing practice move faster than agencies that re-ATO every service.

Classification boundary design is specifically important for AI because AI pipelines often combine data from multiple sources that may live at different levels. The design discipline is to identify the highest classification any data will touch, architect the pipeline to that level, and provide explicit cross-domain solutions where lower-classification data flows up or higher-classification results must flow down with appropriate redaction. DoD agencies have well-developed practice here, and civilian agencies increasingly adopt analogous patterns for CUI boundary management. A data engineer designing for federal AI should plan classification boundaries at the architecture stage, not retrofit them after the fact.

Related Lectures

L3 3.5.2 Cross-Agency AI Coordination. L3 3.5.3 Moving from Pilot to Production. L2 2.5.2 AI Metrics and KPIs. L4 4.4.1 Enterprise AI Risk Management.