Learning Objectives

After completing this lecture, you will be able to:

• Understand the key concepts of pii and ai: the bright red lines in a government context

• Connect pii and ai: the bright red lines to your agency's AI initiatives

• Identify next steps for applying these concepts in your role

Key Topics Covered

• What NEVER goes into an AI system
• Real examples of PII exposure
• How to sanitize inputs before using AI

Why This Matters for Government

Government agencies face unique challenges when it comes to AI adoption. This lecture addresses these challenges head-on by providing all government employees with the knowledge and frameworks needed to navigate AI in the public sector responsibly and effectively.

As part of the L1 (AI Aware) curriculum, this lecture builds on the foundational principle that every AI system in government ultimately serves citizens. Whether you are working with AI tools daily or setting strategy for your agency, understanding pii and ai: the bright red lines is essential for responsible, effective government AI adoption.

Lecture URL: https://skill.re/learn/govt/pii-and-ai-the-bright-red-lines.php

======================================================================

TRANSCRIPT: PII and AI: The Bright Red Lines

======================================================================

What you will learn: What NEVER goes into an AI system. Real examples of PII exposure. How to sanitize inputs before using AI.

I want to start this lecture by being absolutely clear: there are bright red lines when it comes to using AI with personally identifiable information.

Some things you absolutely cannot do. You cannot upload a citizen's social security number into an AI system. You cannot use personal medical information to train a model. You cannot feed an AI system the names and addresses of vulnerable populations.

Not because it's inconvenient. Not because it's a best practice that might be worth reconsidering. Not because in your specific case the risk might be low.

No. These are hard lines because the risks are existential—to individuals, to your agency, and to public trust.

In this lecture, I'm going to be very specific about what those red lines are and why they exist. I'm also going to give you concrete strategies for doing AI work safely when you legitimately need to work with data that contains PII.

WHY THIS MATTERS FOR GOVERNMENT

PII is the most personal information an individual shares with government. When you share your social security number, medical history, or family composition, you're trusting government to keep that information secure and use it only for the purposes you understand.

When government fails at that trust—when PII is breached, misused, or exposed—the consequences are profound:

• Identity theft

• Fraud victimization

• Stalking or harassment

• Wrongful investigation

• Discrimination

• Loss of privacy and dignity

And for government: legal liability, loss of public trust, operational disruption, regulatory scrutiny.

When it comes to AI and PII, the risks are amplified:

• AI systems process data at scale—thousands of records at once

• AI systems combine data in new ways—creating inferences and predictions that might reveal sensitive facts

• AI systems are sometimes opaque—hard to audit, hard to know what's happening

• AI systems are sometimes shared or deployed broadly—increasing the surface area for exposure

This is why the red lines exist.

THE BRIGHT RED LINES

Here's what you absolutely must not do with PII and AI:

Red Line 1: Never Upload PII to Unauthorized External AI Systems

This is the clearest red line. You cannot copy a spreadsheet containing citizen names, addresses, and personal information, and upload it to a public AI service like ChatGPT, Gemini, or Claude.

Not even in a test environment. Not even to quickly see if the AI can help you. Not because you trust the service and think they're probably secure. Not if you "anonymize" the names but leave the other details intact.

Why: Once the data leaves your secure government systems, it's out of your control. It can be:

• Accessed by foreign adversaries

• Used to train other models

• Retained by the service longer than you expect

• Breached

• Subject to legal discovery if the company is sued

• Shared with data brokers

The data is no longer yours to protect.

Real examples of this happening:

• Employees at tech companies uploaded code containing PII, which became available to competitors

• A hospital employee uploaded patient data to a cloud AI service to test a workflow. The data was retained and later included in model training without explicit consent.

• Government employees shared citizen data with commercial AI services, which then used it for research and shared it with affiliates.

The penalty: disciplinary action, termination for cause, loss of security clearance, legal liability, criminal charges in severe cases.

Red Line 2: Never Use an AI System Without Explicit Legal Authority and Security Review

Even internal government AI systems need approval before you use them with PII.

You cannot decide on your own that your agency's needs are urgent enough to skip the security review and legal check. "We need this AI output today" does not override security requirements.

Before you use any AI system with PII:

• Get legal approval (does your statute authorize this use?)

• Get security approval (is the system secure enough for this data?)

• Get privacy office approval (does using this data respect privacy expectations?)

• Document the approval

Red Line 3: Never Combine Multiple Data Sources to Re-identify De-identified Data

You have a de-identified dataset (names removed, age generalized, etc.). You want to link it to other databases to get more information, using characteristics like age, zip code, and medical condition.

Do not do this. This is a direct violation of the de-identification promise. You're re-identifying data that was supposed to be anonymous.

Red Line 4: Never Use One Purpose's Data for Another Purpose

You collect data from citizens for one purpose (benefits determination). You cannot then use that data for a different purpose (immigration enforcement, fraud detection in a different program, etc.) without:

• Legal authority for the new use

• Notice to the individuals (usually)

• New privacy protections specific to the new purpose

Reusing data for new purposes without authorization is a breach of trust and often a violation of law.

Red Line 5: Never Train a Model on Sensitive Data and Then Share the Model

You train an AI model on data containing PII. The model is accurate and useful. You want to share it with another agency or contractor so they can use it.

Do not do this without extensive review. The model might memorize PII from the training data, or it might be possible to extract training data from the model through attacks like membership inference (determining whether a specific individual's data was in the training set) or model inversion (reconstructing original data from the model).

REAL EXAMPLES OF PII EXPOSURE

Let's ground this in reality with specific cases.

Example 1: The Facial Recognition Case

A law enforcement agency used an AI facial recognition system to identify suspects. The system was trained on mugshot databases—millions of images, with names and identifying information attached.

An investigative journalist obtained the training dataset and discovered that the facial images of ordinary citizens—people who had never been convicted of anything—were included. Their faces had been collected at driver's license bureaus and were used to train a system that could be used to track them.

The citizens had never consented to this. They had no idea their images were being used to train a surveillance system.

The PII exposure: facial images + identities, used for purposes the subjects didn't understand or approve.

The outcome: legal suits, regulatory scrutiny, reputational damage, system retirement.

Example 2: The Predictive Model Leak

A state agency developed an AI model to predict which welfare applicants might be engaging in fraud. The model was trained on years of case data containing detailed personal information: employment history, family composition, financial details, even mental health information.

The model was deployed and performed well. So the agency wanted to share it with other states. But before they did, a security researcher analyzed the model and discovered that membership inference attacks were possible—an attacker could determine whether a specific individual's data was in the training set. This meant the "secrets" of the training data (identities of people investigated for fraud) could be extracted.

The PII exposure: personal information of fraud investigation subjects could be extracted from the model.

The outcome: model sharing plans abandoned, training data anonymized and re-done, reputational damage.

Example 3: The Contractor Leak

A government agency contracted with a vendor to build an AI system. The vendor needed training data, so the agency provided it—including de-identified case records with detailed personal information.

The vendor developed the system and passed it back to the government. But no one had explicitly documented that the vendor could not reuse the data for other purposes.

Years later, the vendor used the same dataset to train a different model for a private company. The private company used the data in ways that violated the original individuals' privacy expectations.

The PII exposure: personal information, originally collected by government for one purpose, was used by private companies for other purposes without consent.

The outcome: legal liability, breach notification requirements, loss of vendor trust.

HOW TO SANITIZE INPUTS BEFORE USING AI

Here's the reality: you probably will need to use AI systems with some information that contains or relates to personal data. The question is how to do it safely.

Strategy 1: Use Aggregated Data Instead of Individual Data

Instead of "Here's 10,000 individual benefit applications, analyze them": "Here's summary statistics: 40% of applicants in region X are single parents, 25% are unemployed, average age is 34."

This gives you insights without exposing individual PII.

Risk: Aggregation can sometimes be reversed to reveal individuals (especially if groups are small). But it's generally much safer than individual-level data.

Strategy 2: De-identify Carefully

If you need individual-level data, de-identify it:

• Remove direct identifiers (name, SSN, exact address)

• Generalize quasi-identifiers (exact age to age range, exact address to zip code)

• Test whether re-identification is possible through linkage to other datasets

• Document your de-identification method

• Have a security expert review it

Strategy 3: Use Synthetic Data

Train an AI model on real data containing PII, but don't share or use the original data. Instead, use the model to generate "synthetic data" - artificial data that has the same statistical properties as the real data but doesn't represent real individuals.

Example: Train a model on 100,000 employment records containing sensitive information. Use the model to generate 100,000 synthetic employment records that have the same patterns but aren't actual people.

Risk: Even synthetic data can sometimes be reverse-engineered or linked to real data. But it's generally much safer than original data.

Strategy 4: Use Only Approved Government AI Systems

Some government agencies have developed secure AI systems specifically for working with sensitive data. These systems:

• Are hosted on secure government servers

• Have multiple layers of security

• Include audit logging

• Are staffed by security-cleared personnel

• Comply with all data handling requirements

Using these systems is vastly safer than using commercial AI services or unsecured systems.

Strategy 5: Minimize Data Retention

If you do use PII in an AI system:

• Keep it only as long as you need it

• Delete it when the project is complete

• Don't retain it "just in case"

• Document your deletion and have it verified

The less PII you keep, the less PII can be exposed if something goes wrong.

ANTI-PATTERNS / MISUSE RISKS

Anti-Pattern 1: "Just One Quick Test"

An employee needs to test an AI approach but is impatient with the approval process. They do a "quick test" with real data containing PII.

The risk: That quick test ends up revealing a security vulnerability, exposing PII, or creating precedent for doing things the wrong way.

Anti-Pattern 2: "Anonymized So It's Safe Now"

An organization removes names from data and assumes it's now safe to use anywhere. They don't realize age + zip code + medical condition can re-identify individuals.

The risk: De-identification is incomplete. PII is exposed when the data is linked to other sources.

Anti-Pattern 3: "The Vendor Promised Confidentiality"

An agency shares sensitive data with a contractor with only a verbal promise or a weak contract that the contractor won't misuse it.

The risk: The contractor misuses the data, or sells it, or gets hacked. The agency has no clear contractual recourse.

Anti-Pattern 4: "Everyone Handles PII; Why Can't AI?"

An organization argues that because people handle PII every day, surely AI can too. They don't recognize the scale and opacity differences.

The risk: PII is exposed at scale. The exposure is harder to detect because it's automated.

PRACTICE / REFLECTION PROMPTS

• Think about an AI system your agency uses or is considering. Does it use PII? If so, do you know if it has legal approval and security approval? If not, what would be your first step to address this?

• Have you ever been tempted to do something with data "just as a test" without going through proper approval? What stopped you (or didn't)? What would you do differently if that situation came up again?

• If you had a legitimate need to analyze data containing PII, but couldn't use an approved system, how would you approach de-identification or sanitization?

• In your agency, who would you ask if you had a question about whether something is safe to do with PII and AI? Make sure you know.

KEY TAKEAWAYS

• There are bright red lines with PII and AI. Never upload PII to unauthorized external AI systems. Never use an AI system with PII without explicit legal and security approval.

• De-identification is not a simple fix. Data can be re-identified through linkage to other datasets. De-identification requires expertise and testing.

• Reusing data for new purposes without authorization is a violation of trust and often law. Once you collect data for one purpose, you can't use it for another without proper legal authority and notice.

• Models trained on sensitive data can leak that data. Attackers can sometimes extract training data from models. Sharing a model trained on PII is as risky as sharing the original data.

• The consequences of PII exposure are severe: legal liability, loss of public trust, disruption, regulatory action, personal consequences for those involved.

• When in doubt, ask. Your agency should have a privacy office or security office. Ask them before you proceed.

• Compliance is not the enemy—it's the foundation. Taking time to get proper approval actually saves time by preventing incidents.

TERMS / GLOSSARY ITEMS

PII (Personally Identifiable Information): Information that can identify an individual (name, SSN, address, phone, email, etc.).

De-identification: Removing direct identifiers from data so individuals cannot be identified.

Re-identification: Determining the identity of individuals in supposedly de-identified data through linkage to other datasets.

Membership Inference: An attack that determines whether a specific individual's data was used to train a machine learning model.

Model Inversion: An attack that reconstructs original training data from a trained model.

Quasi-identifier: A piece of information that is not itself personally identifiable but can be used to identify individuals when combined with other data (e.g., age + zip code + medical condition).

Data Minimization: The principle of collecting and using only the minimum data necessary for a stated purpose.

You work in a social services agency. You want to use AI to improve intake—to help caseworkers understand which clients need which services.

The data you'd like to use: casework notes (often containing detailed personal information), assessment scores, service history, client demographics.

The bright red line: you cannot upload this data to a public AI service.

Here's what you can do:

• Get legal approval for using this data for this purpose. Does your statute allow it? Do you need to notify clients first?

• Get security approval. Can the data be used on an approved government AI system?

• Establish data handling procedures. Who has access? How long is data retained? How is it deleted?

• Test the system for privacy leakage. Can anyone reverse-engineer the system outputs to figure out what personal information was in the training data?

• Limit deployment. Use the system in controlled environments first, before going agency-wide.

• Monitor and audit. Who's using the system? What data are they querying? Has anything suspicious happened?

That's how you do legitimate AI work with sensitive data.

Time yourself. You have 10 minutes.

Answer these questions:

• In your agency, what PII do you work with?

• How is it currently stored and protected?

• Has anyone proposed using this PII in an AI system?

• If so, do you know if it has legal and security approval?

• If not, what would need to happen first?

If you can't answer all these questions, identify who you need to talk to.

The bright red lines exist for a reason. They're not bureaucratic obstacles—they're the minimum standards for trustworthy government.

Every time you respect these lines, you're protecting a citizen. Every time you violate them, you're risking real harm to real people.

Do the right thing. Follow the red lines. Ask for help when you're uncertain. Your agency will be stronger for it.

Thank you.

Government AI CLUB Certification Program

Level 1: AI Aware | Government AI Policy Landscape | Lecture 2.4

A GOVT.CLUB initiative.

<- 1.2.2 Data Sensitivity and Classification 1.2.4 The Blueprint for an AI Bill of Rights ->

Start Your CLUB Certification

This lecture is part of L1: AI Aware—8 hours of comprehensive government AI training.

Explore CLUB Certification

Related Lectures

L1 1.2.1—Government AI Policy Landscape 20 min - Video + Reading

L1 1.2.2—Data Sensitivity and Classification 15 min - Video + Checklist

L1 1.2.4—The Blueprint for an AI Bill of Rights 20 min - Video + Reading

Frequently Asked Questions

What will I learn in PII and AI: The Bright Red Lines?

In this 15 min video + scenarios lecture, you will What NEVER goes into an AI system. Real examples of PII exposure. How to sanitize inputs before using AI

What level is PII and AI: The Bright Red Lines?

This is a Level 1 (AI Aware) lecture, part of Chapter 1.2 \u2014 Responsible AI Use. It is designed for all government employees.

How long is lecture 1.2.3?

Lecture 1.2.3 (PII and AI: The Bright Red Lines) takes 15 min. It is delivered as a video + scenarios format.

Do I need prerequisites for PII and AI: The Bright Red Lines?

This lecture is part of L1 (AI Aware). Prerequisites: None.

What is the CLUB Certification?

CLUB (Community Leading Unified Benchmarks) is a maturity-based AI certification for government professionals with 5 levels (L1-L5), 215 lectures, and 25 chapters aligned with NIST AI RMF, OMB, and GAO frameworks.