Learning Objectives

By the end of this lecture you will (1) define AI red-teaming distinctly from traditional penetration testing and from vanilla software QA, drawing on the NIST AI 600-1 Generative AI Profile published July 2024 and the MITRE ATLAS knowledge base first released in 2021; (2) map the adversarial machine learning taxonomy to federal threat scenarios including prompt injection, indirect prompt injection through retrieved context, jailbreaks, data poisoning, model inversion, membership inference, model extraction, and supply-chain attacks against foundation models; (3) apply Executive Order 14110 Section 4.1(a)(ii) which directed the Secretary of Commerce through the Director of NIST to coordinate guidelines for AI red-teaming, later operationalized through the U.S. AI Safety Institute and the International AI Safety Institute network; (4) describe the DEF CON 31 Generative Red Team (August 2023) and DEF CON 32 (August 2024) collaborations with the White House OSTP that produced the largest public AI red-teaming exercises to date with more than 2,200 participants; (5) plan a federal AI red-team engagement including scope, authorization, rules of engagement, and disclosure pathway consistent with CISA coordinated vulnerability disclosure; (6) connect red-teaming outputs to NIST AI RMF MEASURE and MANAGE functions and to OMB M-24-10 minimum practices for rights-impacting and safety-impacting AI; and (7) evaluate common pitfalls including 'theater' red-teaming where findings do not drive remediation, single-shot exercises without continuous follow-up, and over-reliance on automated tools to the exclusion of creative adversarial thinking.

What AI Red-Teaming Is and Is Not

Red-teaming originated in Cold War military planning and entered cybersecurity in the 1990s as adversarial penetration testing. AI red-teaming borrows the adversarial posture but operates on a fundamentally different attack surface: the model, its training data, its inference pipeline, and the sociotechnical system that surrounds it. NIST AI 600-1 defines AI red-teaming as 'a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI'. This is different from traditional penetration testing which targets networks, hosts, and applications; from vulnerability scanning which is largely automated; and from user acceptance testing which verifies that documented requirements are met. AI red-teaming covers three tiers: (1) model-level attacks such as prompt injection, jailbreaks, and adversarial examples; (2) system-level attacks including indirect prompt injection via retrieved context, tool abuse, and data exfiltration through agentic behavior; and (3) sociotechnical-level attacks including disinformation generation, bias exploitation, and misuse by authorized users. A federal AI red-team that looks only at the model misses two-thirds of the attack surface. The ACM FAccT and NeurIPS communities have generated substantial academic literature on each tier, but operational red-teaming in government requires synthesizing across all three.

Federal Directives Driving AI Red-Teaming

Executive Order 14110 (October 30, 2023) Section 4.1(a)(ii) directed NIST to coordinate AI red-team guidelines, with the dual-use foundation model developer reporting requirements under the Defense Production Act in Section 4.2 creating concrete reporting obligations on red-team results for covered models. OMB Memorandum M-24-10 (March 28, 2024) requires pre-deployment AI testing in its minimum practices for rights-impacting and safety-impacting AI and instructs agencies to conduct or require red-teaming where appropriate. NSM-10 on critical and emerging technologies and the National Security Memorandum on AI from October 24, 2024 extended red-teaming requirements to national-security AI. The DoD Responsible AI Strategy and Implementation Pathway includes Test and Evaluation, Verification and Validation (TEVV) with explicit red-team elements, and the Chief Digital and AI Office (CDAO) operates Task Force Lima for generative AI experimentation. CISA published 'Guidelines for Secure AI System Development' in collaboration with the UK NCSC in November 2023 and has issued subsequent advisories covering AI-specific threat patterns. The Department of Homeland Security's 'Safety and Security Guidelines for Critical Infrastructure Owners and Operators' (April 2024) applies to 16 critical infrastructure sectors. International coordination runs through the Bletchley Declaration (November 2023), the Seoul Declaration (May 2024), and the emerging International Network of AI Safety Institutes. Agencies contracting AI services should require red-team evidence aligned with these instruments or face GAO audit findings.

MITRE ATLAS and the Adversarial ML Taxonomy

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems), first published 2021 and continuously updated, structures adversarial ML in an ATT&CK-style matrix of tactics and techniques. Reconnaissance tactics include gathering information about model architecture, training data, and deployment environment. Resource Development includes obtaining adversarial examples, building proxy models, and acquiring data. Initial Access covers ML Supply Chain Compromise, Valid Accounts, and Exploit Public-Facing ML Application. ML Attack Staging covers Create Proxy ML Model, Backdoor ML Model, and Verify Attack. Execution covers ML-Enabled Product or Service and Command and Scripting. Persistence includes Poison Training Data and Backdoor ML Model. Defense Evasion covers Evade ML Model and Craft Adversarial Data. Discovery covers Discover ML Model Ontology and Discover ML Model Family. Collection covers ML Artifact Collection and Data from Information Repositories. ML Attack Staging uses synthetic examples, craft adversarial data, and proxy models. Exfiltration covers Exfiltration via ML Inference API and via Cyber Means. Impact covers Evade ML Model, Denial of ML Service, Spam ML System, Erode ML Model Integrity, Cost Harvesting, External Harms, and ML Intellectual Property Theft. The ATLAS matrix enables red teams to plan structured campaigns rather than ad hoc attempts. Case studies in ATLAS include the Clearview AI scraping pattern, the ProofPoint evasion of Cylance, the Microsoft Tay data poisoning (2016), the tumbler-labeled STOP sign manipulation (Eykholt et al. 2018), and the ChatGPT data exfiltration patterns of 2023. Federal red teams should adopt ATLAS as their baseline taxonomy.

Generative AI-Specific Threat Classes

NIST AI 600-1 (July 2024) defines twelve generative AI risk categories that federal red teams must probe. (1) CBRN Information: ability to provide meaningful uplift to a non-expert seeking chemical, biological, radiological, or nuclear harm. (2) Confabulation: fabricating plausible but false content, especially citations and numerical facts. (3) Dangerous, Violent, or Hateful Content. (4) Data Privacy: leaking training data, memorized PII, or exposed context. (5) Environmental: carbon and water costs at scale. (6) Harmful Bias or Homogenization. (7) Human-AI Configuration: over-reliance, misaligned mental models. (8) Information Integrity: deepfakes, synthetic media, provenance. (9) Information Security: prompt injection, jailbreaks, agentic misuse. (10) Intellectual Property. (11) Obscene, Degrading, and/or Abusive Content, especially non-consensual intimate imagery and child sexual abuse material. (12) Value Chain and Component Integration. Each risk class has distinct red-team techniques. For prompt injection, the canonical paper is Greshake et al. 'Not what you've signed up for' (2023) on indirect prompt injection via retrieved documents. For jailbreaks, Wei et al. 'Jailbroken' (2023) and Zou et al. 'Universal and Transferable Adversarial Attacks' (2023) are foundational. For data leakage, Carlini et al. 'Extracting Training Data from Large Language Models' (2021) is the seminal reference. For model extraction, Tramer et al. 'Stealing Machine Learning Models via Prediction APIs' (2016) established the threat model. For deepfakes, the Partnership on AI Synthetic Media Framework (2023) and Content Authenticity Initiative provide mitigations. Federal red teams should maintain a living catalog of known-failing prompts and refresh monthly as new techniques emerge.

DEF CON Generative Red Team and Public Exercises

In August 2023 at DEF CON 31, the AI Village, Humane Intelligence, SeedAI, and the White House OSTP coordinated the largest public AI red-teaming exercise to date, with approximately 2,200 participants probing models from OpenAI, Anthropic, Google, Hugging Face, Meta, Microsoft, NVIDIA, and Stability AI across eight risk categories. The exercise yielded the first systematic public corpus of successful attacks against production models. DEF CON 32 in August 2024 extended the exercise with a focus on AI-powered agentic systems and federal use cases, coordinated with NIST U.S. AI Safety Institute. Findings from these exercises demonstrated that (a) prompt injection remains trivially successful against most models without hardened scaffolding; (b) jailbreaks transfer across model families; (c) role-play and persona attacks are highly effective against alignment; (d) models are vulnerable to crafted unicode and non-English language attacks; (e) models leak training data when prompted with partial matches; and (f) models produce unsafe biological and cybersecurity content when framed as fiction. Federal programs running their own red-teaming should budget for techniques documented in these public exercises and adopt a 'findings become tests' discipline in which every vulnerability becomes a regression-test. The Association for the Advancement of Artificial Intelligence (AAAI) and USENIX Security also host peer-reviewed venues where new attack categories emerge; subscribing to these venues is part of a mature federal red-team posture.

Planning a Federal AI Red-Team Engagement

A federal AI red-team engagement has five phases. PLAN: define scope (model, system, sociotechnical), set objectives (find safety-impacting vulnerabilities before deployment, validate mitigations, generate regression tests), assemble a mixed team including security engineers, data scientists, domain experts (clinicians for HHS, tax specialists for IRS, field operators for DHS), and representatives of affected communities for bias testing. Document rules of engagement. AUTHORIZE: obtain written authorization from the system owner and authorizing official; clarify what data the red team can use, whether live prompts are permitted against production, and how findings will be handled. CISA's Coordinated Vulnerability Disclosure policy applies to vulnerabilities affecting federal systems. EXECUTE: perform attacks against each ATLAS tactic and each AI 600-1 risk category, document successful and unsuccessful attempts, capture artifacts. ANALYZE: categorize findings by severity using an AI-adapted CVSS or the forthcoming NIST AI 100-3 severity schema, estimate exploitability and impact. REPORT: produce a structured report with executive summary, technical findings, mitigations, residual risk, and regression-test suite; route to the Chief AI Officer, AI Governance Board, and Authorizing Official. For DoD programs, results feed the TEVV process and the Milestone Decision Authority. Typical engagement duration is 2-6 weeks for a production system, with continuous lightweight red-teaming layered on top of periodic deep engagements.

Case Studies in Federal and Adjacent AI Security Failures

Microsoft Tay (2016): A Twitter chatbot was induced through coordinated prompting to produce racist content within 24 hours; the failure was a lack of red-teaming against adversarial user populations. Clearview AI (2020 onwards): Massive web scraping of social media photos for a facial recognition database used by many law enforcement agencies illustrated the supply-chain integrity problem; ACLU litigation under the Illinois Biometric Information Privacy Act produced a 2022 settlement. ChatGPT March 2023 bug: A Redis client library defect exposed other users' chat titles and partial payment information, illustrating that AI systems inherit the full infrastructure attack surface. Samsung 2023 source code leak: Engineers pasted confidential code into ChatGPT, illustrating the social-engineering attack surface around allowed employee AI use. Air Canada chatbot (2024): Tribunal ruling that the airline was liable for incorrect refund information provided by its chatbot, establishing a principle of principal liability for agentic misstatements. NYC MyCity chatbot (2024): Provided legally incorrect advice to small businesses including encouraging illegal actions; correction required multiple rounds of media scrutiny. IRS ID.me (2022): Illustrates supply-chain and vendor-review failures around AI-adjacent biometrics. Michigan MIDAS and Dutch toeslagenaffaire remain cautionary tales for fraud-detection red-teaming. Red teams should compile and internalize these cases; lessons-learned libraries compound over time.

Tools and Infrastructure for AI Red-Teaming

Open-source tooling includes Microsoft PyRIT (Python Risk Identification Tool, 2024), Protect AI's garak LLM vulnerability scanner, Azure AI Studio safety evaluations, HELM (Stanford Holistic Evaluation of Language Models), BIG-bench, TruthfulQA, ToxiGen, BOLD for bias, the AdvBench jailbreak prompt dataset, and the NIST ARIA (Assessing Risks and Impacts of AI) evaluation program launched by the U.S. AI Safety Institute in 2024. Commercial platforms include Robust Intelligence, HiddenLayer, Credo AI, and Calypso AI. Red teams should not rely on tools alone; automated scanners catch only a fraction of the attack space, and creative human adversarial thinking remains essential. Infrastructure requirements include isolated test environments, logging at prompt and response granularity, version control for attack artifacts, and chain-of-custody for evidence. For classified environments, cross-domain solutions must be considered and the NSA CSfC program may apply. FedRAMP-authorized red-team-as-a-service offerings are emerging but coverage gaps remain, particularly for sociotechnical red-teaming.

Anti-Patterns in Federal AI Red-Teaming

(1) Red-Team Theater: engaging a contractor for a one-week sprint that produces a glossy report but no regression tests, no remediation plan, and no follow-up. (2) Scope Collapse: limiting scope to model-level attacks and ignoring sociotechnical failures, vendor supply chain, and agentic tool abuse. (3) Findings-Without-Remediation: documenting vulnerabilities that are never fixed because no one owns the model. (4) Single-Shot Engagement: one pre-deployment red-team and nothing after, even though the threat landscape evolves weekly. (5) Homogeneous Team: no domain experts, no affected-community representation, no sociotechnical perspective. (6) Tool-Only Red-Teaming: relying on garak or PyRIT without human creativity. (7) Over-Sharing: publishing attack artifacts publicly before coordinated disclosure, enabling downstream abuse. (8) Under-Sharing: hoarding findings within the originating program and not contributing to cross-agency learning. (9) Metric Tyranny: counting attempts rather than measuring actual risk reduction. (10) Confusing Red-Team with Evaluation: running standard benchmarks and calling it a red-team exercise. CISA and the NIST U.S. AI Safety Institute have published guidance distinguishing red-teaming from standard evaluation, and federal program managers should internalize the difference.

Integrating Red-Team Outputs with NIST AI RMF and M-24-10

Red-team findings feed the MEASURE function of AI RMF by providing empirical evidence of trustworthiness characteristics including safety, security and resilience, robustness, and managed bias. They feed the MANAGE function by prioritizing risk responses: mitigate, avoid, transfer, or accept with documentation. Under M-24-10 minimum practices for safety-impacting AI, agencies must conduct AI impact assessments, perform testing (including red-team testing where appropriate), conduct ongoing monitoring, and provide human oversight. Red-team findings should be tracked in a risk register that is reviewed at AI Governance Board cadence and summarized in the annual AI Use Case Inventory. For vendor-provided AI, agencies should require delivery of red-team evidence as a contract deliverable, with rights to repeat red-teaming on the delivered system. Post-incident, red-team capabilities provide root-cause analysis and input to remediation planning. Over a multi-year horizon, mature federal AI programs develop internal red-team cadres with domain depth that external contractors cannot match, supplemented by periodic external engagements for fresh perspective.

Summary and Next Steps

AI red-teaming is a structured adversarial testing discipline distinct from traditional penetration testing, software QA, or benchmark evaluation. It operates across model, system, and sociotechnical tiers, maps to the MITRE ATLAS taxonomy and the NIST AI 600-1 risk categories, and produces outputs that feed the NIST AI RMF MEASURE and MANAGE functions and OMB M-24-10 minimum practices. Federal programs should plan, authorize, execute, analyze, and report red-team engagements with documented rules of engagement and coordinated vulnerability disclosure. Case studies from Microsoft Tay, Clearview AI, Air Canada, NYC MyCity, IRS ID.me, Michigan MIDAS, and the Dutch toeslagenaffaire illustrate both AI-specific and AI-adjacent failure modes. Tools including PyRIT, garak, HELM, ARIA, and commercial platforms accelerate red-teaming but do not replace creative adversarial thinking. Mature programs avoid anti-patterns including red-team theater, findings-without-remediation, and single-shot engagements. The next lecture on Bias Detection and Mitigation at Scale deepens one specific red-team thread: how to systematically measure and mitigate algorithmic bias across protected classes and intersectional groups.