โ†
AI Governance, Risk & Red Teaming
Capable ยท M17 ยท lesson 17 of 22 ยท queued
Preview โ€” browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll โ†’
The Generative AI Use Policy - Aligned to NIST AI 600-1
๐Ÿ“–
now learning

The Generative AI Use Policy - Aligned to NIST AI 600-1

15 min

The General AI Acceptable Use Policy answers the question "what AI can our employees use, and how?" The Generative AI Use Policy answers a different question: "now that we have permitted generative AI, what additional guardrails govern its unique risks: confabulation, IP exposure, training-data leakage, deepfake creation, machine-readable provenance marking?" The general AUP is the umbrella. The GenAI Policy is the GenAI-specific addendum that turns the AUP from a permissions document into an operational substrate aligned to NIST AI 600-1 GenAI Profile suggested actions GV-1.1-001 through MG-4.3-005, EU AI Act Article 50(2) and 50(4) marking obligations, and the GPAI Code of Practice copyright chapter. This lesson ships the GenAI Use Policy template, scope, confabulation-review requirement, IP and copyright handling, training-data exclusion, output classification, human-review thresholds, prompt-engineering restrictions, incident reporting, with NIST AI 600-1 action references in policy footnotes the auditor will check line by line.

Why a GenAI Policy on Top of the General AUP

The general AI Acceptable Use Policy from Lesson 022 governs the full enterprise AI surface: predictive models, decision-support systems, classification systems, traditional ML pipelines, and generative AI. It establishes the permission model (sanctioned, sandboxed, prohibited), the data-handling tiers, the prohibited use cases under Article 5, the human-in-the-loop expectations under Article 14, and the Article 4 literacy linkage. It does not, and cannot, name the GenAI-specific risks at the operational depth the NIST AI 600-1 GenAI Profile requires.

The case for a separate GenAI Policy comes down to four facts the general AUP cannot carry without becoming unreadable. First, confabulation is a category of risk that has no analog in traditional ML; "the model is wrong with high confidence" requires a different review mechanism than "the classifier mis-labeled a record." Second, IP exposure for generative outputs is mediated by training-data licensing, output similarity, and prompt-injected exfiltration patterns that don't apply to a fraud-scoring model. Third, training-data flow, whether a vendor uses customer prompts to train future models, is a vendor-configuration question specific to GenAI APIs (OpenAI's opt-out, Anthropic's training-data exclusion, Google's Gemini-via-Vertex settings). Fourth, EU AI Act Article 50(2) requires machine-readable marking of GenAI outputs and Article 50(4) requires deepfake disclosure, obligations that do not apply to non-GenAI systems at all.

The NIST AI 600-1 GenAI Profile names 12 risks (CBRN, confabulation, dangerous content, data privacy, environmental, human-AI configuration, IP, obscene content, information integrity, information security, value chain, harmful bias) and ships 200+ suggested actions across Govern / Map / Measure / Manage. The GenAI Use Policy is the deployer-side translation: which suggested actions become enforceable policy obligations on the employees and the systems that use generative AI. The auditor reading the policy will check whether GV-1.1-001 (AI governance roles defined), MP-2.3-002 (use-case characterization), MS-2.5-001 (output verification), MG-2.2-001 (response to identified risks), and similar are reflected in policy text. The cross-walk is the audit defense.

GenAI Policy Structure - Eight Sections With NIST AI 600-1 Footnotes

The recommended GenAI Use Policy structure carries eight numbered sections, each footnoted to the NIST AI 600-1 suggested actions it operationalizes. The full structure runs 8-14 pages depending on organizational complexity. Each section below names the operational depth and the regulator cross-walk.

Section 1 - Scope and Definitions

Scope names generative AI tools specifically: large language models (text generation), generative image (DALL-E, Midjourney, Adobe Firefly, Stable Diffusion), generative video (Sora, Runway, Pika), generative audio (ElevenLabs voice synthesis, Suno, Udio), and multimodal models (Claude with vision, Gemini Pro, GPT-4o). Definitions cross-reference EU AI Act Article 3(63) "general-purpose AI model" and Article 3(44) "deep fake." Non-GenAI systems route to the general AUP. Cross-walk: NIST AI 600-1 GV-1.2-001 (scope determination); ISO 42001 A.6.2.2 (objectives definition).

Section 2 - Confabulation Review Requirement

The operational core. The policy requires output verification for any externally-distributed content: content published to customers, regulators, courts, public web, financial filings, contracts, legal advice, medical advice, marketing collateral, or any artifact carrying organizational liability. Verification means a human source-checks claims against authoritative sources before distribution. The named "high-stakes use cases" require human review regardless of distribution channel: legal advice (citations, holdings, precedent), medical advice (diagnosis, dosing, contraindications), financial advice (suitability, projections, regulatory disclosures), credit decisions (eligibility, terms, adverse-action notices), employment decisions (interview content, candidate evaluation, performance review summaries). Cross-walk: NIST AI 600-1 MS-2.5-001 / MS-2.5-002 (output verification); MG-2.2-001 (response to identified risks); EU AI Act Article 14 (human oversight); OWASP LLM09 (misinformation).

Output indemnification analysis required for any commercial use, the policy names which vendor agreements include indemnification (OpenAI Copyright Shield for ChatGPT Enterprise; Microsoft Copilot Copyright Commitment; Anthropic for Claude Enterprise; Google Cloud's Generated Output Indemnification for Vertex AI / Gemini Enterprise) and which do not (most open-source models, most consumer tiers). Training-data attribution where claimed, citation of upstream provenance when the vendor publishes it. Cross-reference to the EU AI Act Article 53(1)(c) copyright policy and the GPAI Code of Practice copyright chapter for upstream providers' obligations. Output-similarity testing required for marketing imagery before public distribution. Cross-walk: NIST AI 600-1 GV-6.1-003 (IP risk management); MG-3.2-002 (third-party content controls); EU AI Act Article 53(1)(c); U.S. Copyright Office AI-assisted-works guidance.

Section 4 - Training-Data Exclusion (No-Train Flags)

The vendor-configuration core. Policy requires "no-train" flags configured on every vendor API that handles confidential or restricted-tier data. Named configurations as of May 2026:

  • OpenAI API: data submitted via the API is not used for training by default (Mar 2023 policy change); enterprise tier explicitly excludes; Zero Data Retention available on request for additional sensitivity.
  • Anthropic Claude API: data submitted via the API is not used for training by default; commercial agreements explicitly exclude; Workspaces feature for tenant-isolated logging.
  • Google Gemini API / Vertex AI Gemini: enterprise Vertex AI explicitly excludes from training; consumer Gemini may use prompts for training unless opted out; the policy names the Vertex AI deployment path as the sanctioned one for confidential data.
  • Microsoft Azure OpenAI / Copilot for M365: enterprise tier excludes; commercial-data-protection covers Copilot Chat with work account; consumer M365 Copilot Pro may differ.
  • Amazon Bedrock: no training on inference inputs / outputs; enterprise-default.

The policy requires verification on enrollment and periodic re-verification (quarterly recommended; annually minimum) because vendors update terms periodically and a previously-excluded tier can shift. The Procurement / AI Officer team owns the verification log. Cross-walk: NIST AI 600-1 GV-6.1-005 (third-party data controls); MP-4.1-003 (data-collection mapping); EU AI Act Article 53(1)(d) Annex XI training-data summary cross-reference; OWASP LLM02 (sensitive information disclosure).

Section 5 - Output Classification and Provenance Marking

Confidence-level disclosure to users for any user-facing GenAI output: the model's stated confidence (where available), the recommendation that the user verify before acting, the disclaimer that the output may be incorrect. Provenance marking for any synthetic content distributed externally: C2PA content credentials for images and video (Adobe / Microsoft / Google / OpenAI consortium); SynthID for Gemini-generated images; IPTC PhotoMetadata for image metadata; watermarking for ElevenLabs synthetic voice. These marking requirements operationalize EU AI Act Article 50(2), which mandates machine-readable marking of synthetic content by Dec 2, 2026. Article 50(4) deepfake disclosure applies to any synthetic content depicting natural persons; the policy requires a visible, audible, or otherwise-perceivable disclosure that the content is AI-generated. Cross-walk: NIST AI 600-1 MS-1.1-004 (output transparency); MG-4.1-001 (information sharing with users); EU AI Act Article 50(2), Article 50(4); GPAI Code of Practice transparency commitments.

Section 6 - Human-Review Thresholds

The policy names three triggers for mandatory human review:

  • Confidence-below-threshold automatic escalation, if the model's stated confidence (or a downstream confidence signal, log-probability, retrieval-augmentation match quality, output classifier score) falls below a use-case-specific threshold, the output is held for human review before user-facing rendering.
  • High-stakes use case always reviewed, the five named high-stakes categories from Section 2 are always reviewed regardless of confidence; the model output is treated as a draft for the human reviewer.
  • Sampling protocol for QA: even for low-stakes, high-confidence outputs, a sampling protocol routes a percentage (typically 1-5%) to human review for quality monitoring and drift detection. Sampling results feed the Article 26 deployer monitoring and Article 72 post-market monitoring artifacts.

Cross-walk: NIST AI 600-1 MS-2.3-004 (human review pathways); MG-2.4-002 (escalation criteria); EU AI Act Article 14 (human oversight); Article 26(5) deployer monitoring; ISO 42001 A.6.2.7 (system intended use and reasonably foreseeable misuse).

Section 7 - Prompt-Engineering Restrictions

The user-side behavioral guardrails. Policy prohibits: jailbreak attempts (DAN prompts, role-play bypasses, attempts to elicit prohibited content via persona injection); prohibited-content elicitation (any attempt to generate Article 5 prohibited outputs: social-scoring narratives, predictive-policing rationales, CSAM, NCII, dangerous-substance synthesis); proprietary-prompt-extraction (attempts to extract system prompts or training-data through prompt-injection patterns); training-data exfiltration (attempts to extract memorized training data via membership-inference or extraction prompts). Violations trigger the disciplinary pathway named in the general AUP plus, for prohibited-content attempts, mandatory incident-reporting to the AI Officer. Cross-walk: NIST AI 600-1 MS-2.6-001 (prompt-injection testing); MG-3.1-001 (harmful-use prevention); OWASP LLM01 (prompt injection); OWASP LLM07 (system prompt leakage); EU AI Act Article 5 prohibited practices.

Section 8 - Incident Reporting (GenAI-Specific)

Cross-references the general Article 73 serious-incident runbook for high-risk system incidents (15-day deadline for non-fatal; 10-day for serious infrastructure disruption; 2-day for death). Adds generative-specific incident patterns the AI Officer wants surfaced even when they do not rise to Article 73:

  • Confabulated harm: any incident where a GenAI output contributed to a customer, employee, or third-party harm because the output was wrong with high confidence.
  • IP infringement: any third-party IP claim arising from GenAI output (DMCA notice, cease-and-desist, copyright infringement allegation).
  • Deepfake exposure, any incident where the organization's synthetic-content marking failed (Article 50(2) compliance failure) or where an unauthorized deepfake was created using organizational tools.
  • Training-data leakage: any confirmed or suspected case where confidential information was found in a model output that should not have been there (memorization, retrieval-augmentation leak, prompt-injection exfiltration).
  • No-train flag violation, any case where confidential data flowed to a vendor tier that did not have training-exclusion configured.

Cross-walk: NIST AI 600-1 MG-4.2-001 (incident response); MG-4.3-001 through MG-4.3-005 (post-incident analysis); EU AI Act Article 73 (serious-incident reporting); ISO 42001 A.10.4 (incident management).

Integration With Vendor Procurement

The GenAI Use Policy is not a standalone document. It integrates with three other artifacts that procurement and legal own. The Vendor Risk Policy (Lesson 024) names the diligence questions for GenAI vendors specifically: training-data exclusion configuration, Annex XII downstream-deployer information delivery (for GPAI-backed vendors), Article 50(2) marking commitments by Dec 2, 2026, indemnification scope, substantial-modification change-control. The procurement contract templates embed those answers as enforceable contractual obligations, the policy's "no-train flag must be configured" becomes a contractual representation by the vendor that no-train is configured and a notification obligation if the vendor changes its training-data policy. The Annex XII receivable verification for GPAI-model-backed deployments confirms the upstream-foundation-model documentation is in hand, the policy's confabulation-review requirement assumes the deployer has the model-card / system-card evidence to know the model's known limitations.

The chain reads: Vendor Risk Policy โ†’ Procurement Contract โ†’ Annex XII receivables โ†’ GenAI Use Policy operating evidence. A failure at any layer creates a gap. The L4 governance lead's quarterly audit-committee briefing reports on the chain end-to-end.

Cross-Walks Throughout - NIST 600-1 + EU AI Act + ISO 42001 + OWASP

The GenAI Use Policy is the most cross-walk-dense policy in the AI governance stack because of the convergence of frameworks on generative AI. The policy footnotes should cite at minimum:

  • NIST AI 600-1 GenAI Profile: GV-1.1-001 (AI governance roles); GV-1.2-001 (scope); GV-6.1-003 (IP risk); GV-6.1-005 (third-party data); MP-2.3-002 (use-case characterization); MP-4.1-003 (data-collection mapping); MS-1.1-004 (output transparency); MS-2.3-004 (human review pathways); MS-2.5-001 / 002 (output verification); MS-2.6-001 (prompt-injection testing); MG-2.2-001 (response to identified risks); MG-2.4-002 (escalation criteria); MG-3.1-001 (harmful-use prevention); MG-3.2-002 (third-party content controls); MG-4.1-001 (information sharing); MG-4.2-001 (incident response); MG-4.3-001 through MG-4.3-005 (post-incident analysis).
  • EU AI Act: Article 3(63) GPAI model definition; Article 3(44) deepfake definition; Article 5 prohibited practices; Article 14 human oversight; Article 26 deployer obligations; Article 50(2) machine-readable marking (Dec 2, 2026); Article 50(4) deepfake disclosure; Article 53(1)(c) copyright policy; Article 53(1)(d) Annex XI training-data summary; Article 72 post-market monitoring; Article 73 serious-incident reporting.
  • ISO/IEC 42001:2023: A.2.2 AI policy; A.6.2.2 objectives; A.6.2.7 intended use; A.10.4 incident management.
  • OWASP LLM Top 10 (2025): LLM01 prompt injection; LLM02 sensitive information disclosure; LLM05 improper output handling; LLM07 system prompt leakage; LLM09 misinformation.
  • GPAI Code of Practice: copyright chapter (TDM opt-out compliance); transparency chapter (model documentation); safety-and-security chapter (incident reporting).
  • NIST AI 600-1 12 risks especially: Risks 2 (confabulation), 4 (data privacy), 7 (IP), 9 (information integrity), 10 (information security), 12 (harmful bias).

The cross-walk density is the audit defense. An auditor opening the policy and finding GV-1.1-001 cited next to the AI Officer role definition, MS-2.5-001 cited next to the confabulation review section, MG-2.2-001 cited next to the escalation paragraph, confirms the policy is operationalizing NIST AI 600-1 systematically, not name-checking it.

Six Common GenAI Policy Mistakes

Mistake 1 - Copying the General AUP Without GenAI Specifics

The most common failure pattern. The general AUP says "use AI responsibly" and the organization assumes that covers GenAI. It does not. The general AUP cannot carry the confabulation review requirement, the no-train flag configuration, the Article 50(2) marking commitments, the high-stakes human-review thresholds at the operational depth NIST AI 600-1 requires. The audit defense fails because the auditor opens the AUP and finds no policy text that operationalizes the GenAI Profile suggested actions.

Mistake 2 - Missing Training-Data Exclusion Configuration

A GenAI Policy without explicit no-train flag configuration leaves the organization exposed to a training-data leakage incident with no policy basis for asserting the vendor was contractually required to exclude. The verification log, quarterly re-verification of the vendor's training-data exclusion configuration, is the operational evidence. Missing this section is one of the top three audit findings in 2026 customer-assurance reviews of GenAI programs.

Mistake 3 - No Human-Review Threshold

"Use AI judgment" is not a human-review threshold. The policy must name confidence thresholds, high-stakes use cases requiring mandatory review, and QA sampling protocols. Without thresholds, the Article 14 human-oversight obligation is unenforceable and the Article 26 deployer monitoring evidence is incomplete. The L4 governance lead cannot demonstrate at the audit committee that human review is operating systematically.

Mistake 4 - No Article 50(2)/(4) Marking Commitments

The Dec 2, 2026 Article 50(2) deadline is the most-imminent GenAI-specific regulatory acceleration. A policy that does not commit the organization to C2PA / SynthID / IPTC / equivalent marking on synthetic content distributed externally creates an Article 50(2) compliance gap visible from the outside (regulators can test public-facing synthetic content for marking). Article 50(4) deepfake disclosure has been in effect since Aug 2, 2026. Missing either is a foreseeable Article 99 penalty exposure.

Mistake 5 - No Confabulation Review Requirement

Confabulation is Risk 2 in the GenAI Profile, the highest-density risk with ~30 suggested actions. A policy that does not require output verification for externally-distributed content leaves the organization exposed to confabulated-output-driven customer harm (legal misadvice, medical misinformation, financial misrepresentation, factually-incorrect public statements). The L1 tort defense, "we had a confabulation review requirement and the human reviewer caught the issue", depends on the policy text existing first.

Mistake 6 - Static Policy

The GenAI landscape moves fast: new vendor offerings, new training-data exclusion settings, new model versions with different safety properties, new EU AI Act guidance, new GPAI Code of Practice clarifications, new NIST 600-1 updates, new OWASP entries. A policy not refreshed at minimum semi-annually with triggered updates on new vendor onboarding, new model versions, regulatory guidance changes, and incident retrospectives quickly drifts out of step with the operational reality. The L4 governance lead's policy-management calendar names the refresh cadence and the triggered-refresh criteria.

Key Takeaways

  • The GenAI Use Policy sits on top of the general AUP. The AUP is the umbrella permission and risk-tier framework; the GenAI Policy is the GenAI-specific addendum operationalizing NIST AI 600-1 suggested actions, Article 50(2)/(4) marking, and the GPAI Code of Practice copyright chapter.
  • Eight-section structure with NIST AI 600-1 footnotes throughout. Scope; confabulation review; IP and copyright; training-data exclusion (no-train flags); output classification and provenance marking; human-review thresholds; prompt-engineering restrictions; incident reporting. Each section cites GV-1.1-001 through MG-4.3-005 actions it operationalizes.
  • Confabulation review is the operational core. Output verification required for externally-distributed content; mandatory human review for the five named high-stakes use cases (legal, medical, financial, credit, employment); operationalizes Risk 2 in the GenAI Profile.
  • Training-data exclusion (no-train flags) is the vendor-configuration core. Named configurations for OpenAI API, Anthropic Claude API, Google Gemini / Vertex AI, Microsoft Azure OpenAI / Copilot, Amazon Bedrock. Quarterly re-verification on the Procurement / AI Officer log.
  • Output classification and provenance marking operationalize EU AI Act Article 50(2)/(4). C2PA / SynthID / IPTC / equivalent for synthetic content; visible deepfake disclosure for natural-person depictions; Dec 2, 2026 Article 50(2) acceleration is the forcing function.
  • Three human-review triggers. Confidence-below-threshold escalation; high-stakes use case always reviewed; QA sampling protocol (1-5%) for monitoring and drift detection. Sampling results feed Article 26 deployer monitoring evidence.
  • Prompt-engineering restrictions name four prohibited behaviors. Jailbreak attempts; prohibited-content elicitation; proprietary-prompt-extraction; training-data exfiltration. Violations route to the AUP disciplinary pathway plus incident-reporting for prohibited-content attempts.
  • Incident reporting adds five GenAI-specific patterns to the Article 73 runbook. Confabulated harm; IP infringement; deepfake exposure; training-data leakage; no-train flag violation. The AI Officer wants these surfaced even when they don't rise to Article 73.
  • Integration with Vendor Risk Policy + procurement contracts + Annex XII receivables. The chain: Vendor Risk Policy โ†’ Procurement Contract โ†’ Annex XII receivables โ†’ GenAI Use Policy operating evidence. A failure at any layer creates a gap.
  • Six mistakes to avoid. Copying the general AUP without GenAI specifics; missing training-data exclusion configuration; no human-review threshold; no Article 50(2)/(4) marking commitments; no confabulation-review requirement; static policy. Semi-annual minimum refresh cadence with triggered updates.