Learning Objectives

After completing this lecture, you will be able to:

• Understand the key concepts of nist ai rmf: map, measure, manage in a government context
• Participate in structured workshop activities with real-world scenarios
• Connect nist ai rmf: map, measure, manage to your agency's AI initiatives
• Identify next steps for applying these concepts in your role

Key Topics Covered

-
Risk mapping, measurement tools, management strategies

-
Practical application to government scenarios

-
Government context for nist ai rmf: map, measure, manage

-
Practical applications and next steps

Why This Matters for Government

Overview

Government agencies face unique challenges when it comes to AI adoption. This lecture addresses these challenges head-on by providing analysts, project leads, team supervisors with the knowledge and frameworks needed to navigate AI in the public sector responsibly and effectively.

As part of the L2 (AI Practitioner) curriculum, this lecture builds on the foundational principle that every AI system in government ultimately serves citizens. Whether you are working with AI tools daily or setting strategy for your agency, understanding nist ai rmf: map, measure, manage is essential for responsible, effective government AI adoption.

======================================================================

TRANSCRIPT

======================================================================

Chapter: 3

What you will learn:

• The MAP function: Identifying and inventorying AI systems, documenting risk profiles
• The MEASURE function: Systematically assessing AI system performance, fairness, and risks
• The MANAGE function: Implementing controls, responding to identified risks, continuous improvement
• How MAP, MEASURE, MANAGE connect to GOVERN
• Practical tools and processes for each function
• How to integrate these functions into ongoing operations

Welcome back. In the previous lecture, you learned about GOVERN--the foundational governance structures, policies, and accountability mechanisms that make responsible AI possible. Now we shift focus to the three operational functions that sit on top of that foundation: MAP, MEASURE, and MANAGE.

If GOVERN is about asking "How will we decide?", then MAP, MEASURE, and MANAGE are about actually executing on those decisions. MAP answers "What AI systems do we have, and what are their risk profiles?" MEASURE answers "How is each system actually performing? Is it meeting our standards?" MANAGE answers "What do we do when we find gaps or problems?"

These three functions work together in a continuous cycle. You map systems to understand what you're working with. You measure to understand how they're actually performing. You manage to address problems and continuously improve. Then you go around the cycle again. With each cycle, your understanding deepens, your risk management becomes more sophisticated, and your systems become more robust.

In government, this three-function cycle is not optional--it's required by OMB guidance. Federal agencies must know what AI systems they're operating, must assess their performance and risks, and must have plans to manage identified risks. This lecture will show you how to do it in a way that's compliant, practical, and actually valuable to your organization.

WHY THIS MATTERS FOR GOVERNMENT

Before MAP, MEASURE, MANAGE became standard practice, many government agencies would deploy AI systems with minimal understanding of what they were doing. A department might implement a system to screen job applications. Years later, when someone investigates, they discover the system has been systematically disadvantaging certain populations. But by then, hundreds or thousands of decisions have been made using that system. The damage is done.

Or an agency would inherit an AI system from a previous administration, not know exactly how it worked or what it was trained on, and keep using it because removing it seemed harder than maintaining it. Problems would emerge in production, but there was no process for detecting and responding to them systematically.

MAP, MEASURE, MANAGE prevent these scenarios. They create organizational visibility into what's actually happening with AI systems. They establish early detection mechanisms for problems. They create systematic processes for fixing problems when they're found. The result is lower risk, faster problem resolution, and higher confidence in AI system deployment.

THE MAP FUNCTION--IDENTIFYING AND INVENTORYING AI SYSTEMS

The MAP function answers a deceptively simple question: "What AI systems do we have?" Deceptively simple, because actually answering it in a large organization is surprisingly difficult.

Many government agencies have no comprehensive inventory of their AI systems. Some systems were developed in-house. Some were procured from vendors. Some are internally built but licensed to external partners. Some are legacy systems that have been in place so long no one remembers they exist. Some are cloud-based SaaS systems that different departments have adopted independently. Mapping requires going through the organization systematically and asking: "What counts as an AI system for governance purposes?"

What Counts as an AI System

For governance purposes, an AI system is any system that uses machine learning, deep learning, statistical models, or other computational methods to make or support decisions. This is broader than many people realize. It includes:

• Predictive models that score or rank decisions (risk assessment, resume screening, benefit eligibility)
• Classification systems that categorize inputs (email spam filtering, document classification, incident categorization)
• Anomaly detection systems that flag unusual patterns (fraud detection, network intrusion detection, benefits fraud)
• Recommendation systems that suggest actions or resources
• Natural language processing systems that extract information or generate text
• Computer vision systems that analyze images

It includes both "high-tech" deep learning systems and more traditional statistical models. It includes systems that make decisions entirely on their own and systems that just support human decision-makers.

The Mapping Process

To map AI systems, most organizations take these steps:

• Send a questionnaire to all departments and units asking about systems they use to automate decisions or analyze data.
• Conduct interviews with technical leads to surface systems that haven't been formally recorded.
• Review cloud service logs and SaaS subscriptions to find independently-adopted AI services.
• Consolidate into a central inventory listing all identified AI systems with key information.
• Validate with system owners to ensure accuracy and completeness.

What to Document for Each System

For each AI system, document:

• Name and purpose
• Owner and responsible team
• Data sources and data characteristics
• Decisions it influences
• Population affected
• Risk classification (safety-impacting vs. rights-impacting)
• Current accuracy and performance metrics
• Fairness testing status and findings
• Governance approval status
• Monitoring and measurement status

THE MEASURE FUNCTION--SYSTEMATIC ASSESSMENT OF AI SYSTEMS

Once you know what AI systems you have, you need to measure how they're actually performing. Measurement is foundational to responsible risk management.

What to Measure

Key measurement dimensions:

Accuracy and Performance: Is the system producing the right outputs? Measure through accuracy, precision, recall, F1 score, or domain-specific metrics. Crucially, measure across the full population and within demographic subgroups.

Fairness Metrics: Is the system producing equitable outcomes? Measure disparate impact (similar approval rates across groups?), calibration (similar outcome rates for given prediction scores?), equalized odds (similar true/false positive rates?), and individual fairness (similar individuals treated similarly?).

Robustness and Security: How does the system handle unusual inputs or adversarial attacks? Test with edge cases, typos, adversarial inputs, and domain-specific challenges.

Data Quality: How good is the training and operational data? Assess completeness (missing values?), accuracy (is the data correct?), currency (is data recent?), and representativeness (does it represent current populations?).

Drift Detection: Has performance changed over time? Detect data drift (input distribution changes), label drift (outcome distribution changes), and concept drift (relationships between inputs and outputs change).

Explainability: Can the system explain its decisions? For consequential systems, people need to understand why decisions were made. Use feature importance, SHAP values, LIME explanations, or manual audits.

Measurement Frequency

Measurement cadence depends on risk and environment stability:

• High-risk systems: Continuous automated monitoring with weekly or monthly manual reviews
• Medium-risk systems: Monthly or quarterly measurement and review
• Low-risk systems: Quarterly or annual measurement

Rapidly-changing environments warrant more frequent measurement than stable ones.

Measurement Tools

Options include fairness toolkits (IBM AI Fairness 360, Google Fairness Indicators), monitoring platforms (Fiddler, Evidently AI, WhyLabs), statistical analysis tools (Python, R, SQL), and custom monitoring solutions tailored to specific systems.

THE MANAGE FUNCTION--RESPONDING TO RISKS AND CONTINUOUS IMPROVEMENT

Measurement identifies problems. MANAGE is what you do about them.

Risk Response Options

• Mitigate: Implement controls while keeping the system operational (e.g., manual review for affected populations)
• Adjust: Change the system (retrain, adjust thresholds, redesign process)
• Escalate: Flag for human decision-makers when technical fixes alone are insufficient
• Retire: Stop using the system if risks exceed benefits
• Restrict: Use the system for lower-risk applications only

Choice depends on problem severity, fix feasibility, and system value.

Incident Response Process

When serious problems emerge:

• Detection through monitoring or user escalation
• Documentation of what happened, when, scope of impact
• Containment through offline, restriction, or heightened review
• Investigation and root cause analysis
• Remediation to fix the problem and prevent recurrence
• Retrospective to improve governance and measurement
• Communication to affected parties and stakeholders

Continuous Improvement

As you measure and manage systems, you learn:

• Model retraining on new or refined data
• Parameter tuning as you understand fairness-accuracy tradeoffs
• Measurement evolution as your understanding deepens
• Process improvement based on managing actual systems

HOW MAP, MEASURE, MANAGE CYCLE TOGETHER

Overview

The framework is powerful in its cyclical nature:

Cycle 1:

• MAP: Identify 15 AI systems
• MEASURE: Find System 7 has a fairness issue
• MANAGE: Implement fix and establish monitoring

Cycle 2:

• GOVERN: Update fairness assessment policies based on learning
• MAP: Improved policies uncover additional systems
• MEASURE: Measure newly-mapped systems, find issues
• MANAGE: Address new issues, continue learning

With each cycle, governance sophistication increases, understanding of the AI footprint becomes more complete, and risk management becomes more effective.

ANTI-PATTERNS AND MISUSE RISKS

Risk 1: Measurement Theater

Creating sophisticated-looking dashboards that don't drive decisions. Measurement is time-consuming, and if findings don't result in action, credibility erodes. Avoid by tying each metric to specific decisions: "If this exceeds X, we do Y." Have clear escalation for findings and demonstrate that measurement drives improvements.

Risk 2: Incomplete Measurement

Measuring some dimensions (accuracy) while ignoring others (fairness, robustness). Unmeasured failures surprise you later. Instead, define comprehensive measurement requirements upfront and implement incrementally over time.

Risk 3: Manage Without Escalation

Problems handled operationally without escalation to governance. Systemic issues never get recognized. Fix by requiring documentation and escalation, expecting escalation from leadership, and reviewing incident trends for patterns.

Risk 4: Measurement Without Accountability

Findings identified but no one responsible for fixing them. Assign clear ownership for each finding with specific timelines. Track progress. Include measurement and remediation in performance expectations.

PRACTICE AND REFLECTION PROMPTS

Prompt 1: Map Your Inventory

Identify 3-5 AI systems in your area. Document: name, purpose, owner, data sources, risk profile. Mini-version of the mapping process.

Prompt 2: Design a Measurement Plan

For one of the systems, design measurement: What metrics? How often? What tools? What fairness metrics are essential?

Prompt 3: Scenario Planning

A fairness issue is discovered in one of your systems. Walk through incident response: detection, investigation, decision-making, remediation.

Prompt 4: Governance Integration

How do MAP, MEASURE, MANAGE connect to your agency's governance? What information flows between them?

Prompt 5: Tool Evaluation

Research measurement tools (Evidently AI, Fiddler, WhyLabs). Which suits your context? What are costs and implementation timelines?

KEY TAKEAWAYS

• MAP creates visibility. Comprehensive inventory is foundational. You can't manage what you don't know about.
• MEASURE is systematic assessment. Include accuracy, fairness, robustness, data quality, and drift. Connect measurement to decision-making.
• MANAGE includes multiple options. Mitigate, adjust, escalate, retire, or restrict based on severity and feasibility.
• They cycle continuously. Each cycle deepens sophistication and effectiveness.
• Measurement drives action. Findings without ownership and action undermine credibility.
• Scaling requires tools and process. Manual approaches work for small numbers of systems. Larger scale requires automation and clear processes.
• Documentation and escalation matter. They create organizational memory and ensure problems don't disappear.

GLOSSARY

AI System Inventory -- Comprehensive catalog of all AI systems, including purpose, data sources, risks, and governance status.

Disparate Impact -- Different outcomes for different demographic groups, even without explicit discrimination.

Model Drift -- Performance change over time due to data, population, or environment shifts.

Fairness Metric -- Quantitative measure of equitable outcomes across demographic groups.

Incident Response -- Systematic process for detecting, investigating, and remediating AI system problems.

Root Cause Analysis -- Investigation into underlying causes of problems, not just symptoms.

Automated Monitoring -- Continuous systematic assessment using tools and alerts, not manual periodic review.

MAP, MEASURE, MANAGE are where NIST AI RMF's promises become reality. While GOVERN establishes structures and policies, these three functions execute on them in practice.

The framework's power lies in its iterative design. You don't do these once and finish. They repeat continuously, each cycle adding sophistication. Organizations that embrace this cyclical approach develop increasingly mature governance and increasingly confident AI deployment.

As subsequent lectures explore specific governance challenges--risk classification, impact assessments, data governance, incident response--remember that all rest on this foundational MAP, MEASURE, MANAGE framework.

Assess your organization across these three functions:

MAP: Do you have a comprehensive AI system inventory? What gaps exist?

MEASURE: What metrics are currently tracked? What measurement gaps exist?

MANAGE: What happens when problems are identified? Is there clear incident response?

Next step: What would have highest impact--completing inventory, adding measurement, improving incident response?

This lecture explored the three operational functions of NIST AI RMF: MAP, MEASURE, and MANAGE. Together with GOVERN, you now have a complete picture of the framework.

In the next lecture (2.3.3), we'll examine risk classification: distinguishing between safety-impacting and rights-impacting AI systems, and why that distinction shapes governance approaches.

The frameworks here are scalable and adaptable--they work for small teams starting AI governance and scale to large federal agencies with complex AI ecosystems. Principles remain constant. Implementation scales with context.

End of Transcript

Source: GOVT.CLUB

Visit: https://govt.club/learn/lectures/l2/232-nist-ai-rmf-map-measure-manage.html

Government AI CLUB Certification Program

Level 2: AI Ready | NIST AI RMF: MAP, MEASURE, MANAGE | Lecture 2.3.2

A GOVT.CLUB initiative

<- 2.3.1 NIST AI RMF: The GOVERN Function
2.3.3 Your Agency's AI Governance Structure ->

Start Your CLUB Certification

This lecture is part of L2: AI Practitioner -- 40 hours of comprehensive government AI training.

Explore CLUB Certification

Related Lectures

L2
2.3.1 -- NIST AI RMF: The GOVERN Function
60 min - Video + Workshop

L2
2.3.3 -- Your Agency's AI Governance Structure
60 min - Reading + Discussion

L2
2.3.4 -- AI Use Case Inventory and Documentation (OMB M-24-10)
60 min - Workshop + Template