The vendor demonstration was impressive. The sales engineer pulled up a sample case, typed a plain-language question into the AI assistant, and watched it summarize eighteen months of contact notes, prior reports, and service history into a clean two-paragraph briefing in eleven seconds. The agency director, a deputy commissioner who had spent twenty years in child welfare before moving to administration, asked one question that stopped the room: "Where does that case go when the model reads it?" The engineer said the data was processed securely. The director asked it differently. "When your model reads a note that says a nine-year-old disclosed sexual abuse, with the child's name and address and school in the same record, where does that text physically go, who can see it, how long is it kept, and is it ever used to train anything?" The engineer said he would have to check with the product team. The director thanked him and ended the meeting. She later told her privacy officer the deal was dead, not because the tool was bad but because nobody on the vendor side could answer, in plain language, the only question that mattered: what happens to the most sensitive data the government holds about the most vulnerable people it serves.

Why This Data Is Different

Human-services agencies hold a category of data that has almost no equal in government. A child-protective-services (CPS, the agency function that investigates child abuse and neglect) case file contains a child's full legal identity, home address, school, medical history, and a narrative record of the worst things that have happened to them. A benefits file under SNAP (the Supplemental Nutrition Assistance Program, federal food assistance), TANF (Temporary Assistance for Needy Families, time-limited cash assistance), or Medicaid (the joint federal-state health-coverage program for low-income people) holds household income, immigration status of household members, disability diagnoses, and the financial fragility of a family at its most exposed. A domestic-violence safety plan holds the one piece of information that, if leaked, can get a person killed: where they are hiding.

This is not ordinary personally identifiable information (PII, any data that can identify a specific person). It is the most sensitive PII the state collects, held about people who have the least power to protect themselves and the most to lose if it is exposed. A data breach in a retail company costs customers a credit-card number and some inconvenience. A data breach in a child-welfare agency can expose a child's abuse history to the person who abused them, reveal a survivor's safe address to an abuser, or out a family's immigration status to the wrong audience. The harm is not financial. It is physical, and sometimes irreversible.

That asymmetry, maximum sensitivity held about minimum power, is why privacy in this field cannot be treated as a compliance checkbox bolted onto an AI deployment after the fact. It has to be the foundation the whole transformation is built on. An agency that gets the documentation use case right, the equity auditing right, and the change management right, but gets privacy wrong, has not modernized. It has built a faster way to harm the people it exists to protect.

This is the most sensitive data the government holds, about the people least able to protect themselves. Privacy is not a feature of the AI program. It is the ground the program stands on.

The Legal Perimeter That Already Governs You

Before a single AI tool enters the building, an agency is already bound by a dense web of confidentiality law. AI does not create new legal obligations here so much as it stresses the ones that already exist, and a leader running an AI program has to know exactly which walls the new technology is being asked to operate inside.

In child welfare, federal law under the Child Abuse Prevention and Treatment Act (CAPTA) and Title IV-E of the Social Security Act conditions federal funding on strict confidentiality of CPS records, with disclosure limited to specifically authorized purposes. In benefits, federal regulation restricts the use and disclosure of SNAP, TANF, and Medicaid applicant information to the administration of the program itself. Medicaid records additionally fall under health-privacy rules. For families receiving substance-use treatment, an especially strict federal confidentiality regulation governs those records and limits redisclosure even within the same agency. Layered on top of all of this is state confidentiality law, which is often stricter than the federal floor, and court orders that seal specific case records.

The practical meaning for an AI program is concrete. Every one of these regimes restricts who may access the data and for what purpose. When you send a case record to an AI tool, you are disclosing that record to whatever systems and people sit behind that tool. If the tool runs on a vendor's servers, you have disclosed protected case information to the vendor. If the tool uses your data to improve its model, you have disclosed it to every future use of that model. If the tool is accessible to a contractor's support staff for debugging, you have disclosed it to them. Each of those disclosures must be authorized under the law that governs that specific category of record, or it is a violation, regardless of how useful the AI output was.

Consider the worked consequence. A caseworker carrying 24 families pastes a substance-use treatment summary into a general-purpose public AI chatbot to get help drafting a court report. That single action may have violated the federal substance-use confidentiality regulation, the state child-welfare confidentiality statute, and a sealing order on the case, all at once, in the four seconds it took to hit paste. The worker was trying to save twenty minutes on a report. The agency now faces a reportable disclosure, a potential funding finding, and a family whose most protected records have left the perimeter and cannot be recalled. The legal perimeter is not abstract. It is the difference between a defensible program and a catastrophic one.

Data Minimization Is the First Control

The single most powerful privacy control available to an AI program is also the oldest privacy principle there is: do not put data where it does not need to be. Data minimization means giving any tool, AI or otherwise, the least sensitive data sufficient to do the job, and no more. In an AI context this principle does most of the protective work, because data that never reaches the model cannot be leaked by the model, retained by the vendor, or used to train a future system.

Apply it to the documentation use case, the field's safest and most widespread application. A worker wants AI help drafting a home-visit note. Does the tool need the child's full legal name and address to help structure the narrative? In most workflows it does not. It needs the substance of the observations. A program designed with minimization in mind strips or tokenizes direct identifiers before the record reaches the model, so the AI works on "the older child" rather than a named, addressed, school-identified human being. If the tool's output is then merged back into the identified record inside the secure case-management system, the family gets the time-saving benefit while the most dangerous identifiers never crossed the perimeter.

What Minimization Looks Like in Practice

• Strip direct identifiers before processing. Names, addresses, dates of birth, Social Security numbers, and case numbers can often be removed or replaced with role labels ("the mother," "child 1") before text reaches an AI tool, then restored inside the secure system.
• Scope the input to the task. A tool drafting a single contact note needs that visit's notes, not the entire eighteen-month file. Sending the whole record "just in case" expands the exposure for no benefit and, as the hallucination lessons show, raises the risk of fabricated history too.
• Separate the most dangerous fields. A domestic-violence safe address, an immigration status, an HIV status, a child's sexual-abuse disclosure: these are fields where the cost of exposure is so high that many programs forbid them from reaching any external tool at all.
• Prefer structured extraction to bulk dumping. Asking a tool to work on a defined, minimized excerpt is safer than pasting an entire file and hoping the relevant part is in there.

The reason minimization is the first control, not the fifth, is that it is the only control that works even when everything else fails. Encryption can be misconfigured. Access controls can be bypassed. A vendor's retention promise can be broken. But data that was never sent to the tool is protected against all of those failures simultaneously. A program that minimizes well has a smaller blast radius when, not if, something else goes wrong.

Where the Data Physically Goes

The director's question in the opening, "where does that case go when the model reads it," is the question every AI procurement in this field has to answer in writing before deployment. There are meaningfully different architectures, and the privacy posture of each is different.

A model that runs inside the agency's own controlled environment, on infrastructure the agency or a tightly contracted government-cloud provider operates, keeps the data inside a perimeter the agency can describe and audit. This is the strongest posture for the most sensitive records, and it is increasingly achievable.

A model that runs in a vendor's cloud under a contract with specific data-handling terms is the most common arrangement. Here the case text leaves the agency and is processed on the vendor's systems. Whether this is acceptable depends entirely on the contract: what the vendor may do with the data, how long they keep it, whether any humans on their side can read it, whether it is used for training, and where in the world the servers physically sit. None of these can be assumed; all must be specified.

A general-purpose consumer AI service with no agency contract is the worst case and the one that causes the most accidental harm. When a worker uses a free public chatbot, the case data goes to a company under that company's standard consumer terms, which may permit the data to be retained, reviewed by staff, and used to train future models. There is no business contract, no data-protection agreement, and no enforceable confidentiality commitment scoped to the agency's legal obligations. For the data this field holds, this is categorically unacceptable, and the most important single rule an agency can give its workforce is that protected case information never goes into a consumer AI service. That one rule prevents a large share of the worst incidents.

The worked distinction matters because the same task, drafting a court report, can be done in any of these three architectures, and the family in the report has no idea which one their caseworker used. The agency owns that choice. The deputy commissioner who killed the deal in the opening was not being difficult. She was refusing to expose a child's abuse disclosure to an architecture nobody could describe.

Vendor Due Diligence That Stays With the Agency

When an agency contracts an AI tool, it does not contract away its legal responsibility for the data. The obligation to protect a child's CPS record stays with the agency even when a vendor is doing the processing. This is the principle that should drive vendor due diligence: the agency remains accountable to the court, the family, and the funding source for everything that happens to the data, so the contract has to give the agency the visibility and control its accountability requires.

A serious due-diligence process gets written, specific, plain-language answers to a defined set of questions before signing, not after a breach.

The Questions That Must Be Answered in Writing

• Training. Will our data ever be used to train or improve your models, or any models? The required answer for this data is no, in contract language, not a marketing assurance.
• Retention. How long is our data stored after processing, and can we require deletion? "Processed and discarded" is very different from "retained for ninety days for quality."
• Human access. Can any of your employees, contractors, or subprocessors read the case data we send? Under what conditions, with what logging?
• Subprocessors. Who else touches the data? Many AI vendors are themselves built on another company's model, so the agency's data may flow to a fourth party the agency never evaluated.
• Location. Where do the servers physically sit, and does any processing occur outside the jurisdictions our law permits?
• Breach. What is your notification timeline and process, and does it meet our statutory reporting deadlines?
• Audit. Can we, or an independent auditor, verify these answers rather than take them on faith?
• Exit. When the contract ends, is our data returned and deleted, with proof?

The consequence of skipping this is not hypothetical. History in this field is full of expensive failures where automated systems harmed vulnerable people at scale: the Dutch childcare-benefits scandal, where a fraud-detection system wrongly accused tens of thousands of families and pushed many into financial ruin, and Michigan's MiDAS unemployment system, which issued tens of thousands of false fraud determinations. Those were not primarily AI privacy failures, but they share the root cause that good due diligence exists to prevent: a government deployed a powerful automated system into the lives of vulnerable people without keeping the visibility and control its accountability demanded. The lesson transfers directly. If you cannot answer, in writing, what happens to the data, you are not ready to send it.

Transparency to the People in the Record

Privacy in human services is not only about keeping data inside a perimeter. It is also about being honest with the people whose lives are in the record. The non-negotiable that due process and privacy form the perimeter has a transparency component: families have a right to challenge determinations and to understand how decisions about them were made, and that right is hollow if the agency hides its use of AI.

Transparency does not mean burdening a family in crisis with a technical lecture. It means that the agency's use of AI is disclosed and documented in a way that a parent, an advocate, a guardian ad litem, and a judge can see and question. When a court report was drafted with AI assistance and verified by the worker, the record should be honest about the workflow if asked, and the agency should have a clear, public-facing account of where and how AI is used in its work. An advocate challenging a determination has a legitimate interest in knowing whether an AI tool touched the record, just as they would want to know any other material fact about how the determination was reached.

The worked tension is real. A privacy officer worries that disclosing AI use invites challenges. The opposite is true over any meaningful time horizon. A program that uses AI quietly and is later discovered to have done so, without disclosure, will face a far more damaging crisis of legitimacy than one that was transparent from the start. In a field where every consequential decision is bound by due process, the defensible posture is the transparent one. Hidden AI in a child-welfare or benefits determination is a scandal waiting to surface. Disclosed, verified, well-governed AI is a defensible practice. The cardinal rule that AI informs and humans decide is also a transparency commitment: the agency can stand in front of a court and account for exactly how the human decision was made and what role the tool played.

Building Privacy Into the Program, Not Onto It

Everything above becomes real only when it is built into how the agency operates, not written into a binder nobody reads. Privacy by design means the protective controls are the default path of least resistance, so a tired worker carrying 28 families at 9 PM does the safe thing because it is the easy thing, not because they remembered a training slide.

In practice this is an operating model, not a memo. It means the approved AI tools are integrated into the secure case-management system so workers never need to copy case text out to an external service, which removes the temptation that causes the worst incidents. It means minimization and identifier-stripping happen automatically in the workflow rather than depending on each worker to remember to do it by hand. It means access to AI features is governed by the same role-based controls that govern access to the records themselves, and every AI interaction with a protected record is logged so the agency can answer, after the fact, exactly what was sent where. It means a named privacy officer owns the AI privacy posture, sits on the governance board, and has the authority to stop a deployment, the way the deputy commissioner stopped the deal in the opening.

It also means the workforce understands the why, not just the rule. A worker who understands that pasting a substance-use summary into a public chatbot can expose a family's most protected records and trigger a funding finding will not do it, even under deadline pressure. A worker who was only handed a prohibited-tools list without the reasoning will paste it the first time the deadline is bad enough. Privacy training in this field is not a compliance exercise. It is harm prevention, and it has to land with the same weight as the verification discipline that keeps fabricated observations out of court reports.

Done this way, privacy stops being the brake on the AI program and becomes the thing that makes the program survivable. An agency can return thousands of hours to direct work with families through AI-assisted documentation and protect the most vulnerable people's most sensitive data at the same time. Those goals are not in tension when privacy is the foundation. They are in tension only when privacy is an afterthought, and in this field an afterthought is how a child's abuse history ends up somewhere it can never be recalled.

Key Takeaways

• Human-services agencies hold the most sensitive personally identifiable information (PII) the government collects, about the people least able to protect themselves. A breach here is not financial inconvenience; it can be physical, irreversible harm, which is why privacy must be the foundation of an AI program, not a later compliance step.
• A dense legal perimeter already governs this data: CAPTA and Title IV-E confidentiality for child welfare, federal restrictions on SNAP, TANF, and Medicaid information, strict substance-use treatment confidentiality, plus stricter state law and court sealing orders. Sending a record to an AI tool is a disclosure that must be authorized under the law governing that record.
• Data minimization is the first and most powerful control because it is the only one that protects the data even when every other control fails. Strip direct identifiers, scope inputs to the task, and keep the most dangerous fields (safe addresses, immigration status, abuse disclosures) out of external tools entirely.
• Architecture determines privacy posture. An agency-controlled environment is strongest; a vendor cloud under a specific contract is acceptable only on its terms; a general-purpose consumer AI service is categorically unacceptable for protected case data, and the single most important workforce rule is that such data never goes into a consumer chatbot.
• Contracting an AI tool does not transfer the agency's legal responsibility for the data. Vendor due diligence must get written, plain-language answers on training, retention, human access, subprocessors, location, breach notification, audit rights, and exit, before signing.
• The Dutch childcare-benefits scandal and Michigan's MiDAS system show the root failure due diligence prevents: deploying a powerful automated system into vulnerable people's lives without keeping the visibility and control the agency's accountability demands.
• Transparency is part of the privacy perimeter. Families, advocates, guardians ad litem, and courts have a legitimate interest in knowing whether AI touched a record. Disclosed, verified, well-governed AI is defensible; hidden AI is a legitimacy crisis waiting to surface.
• Privacy by design makes the safe path the easy path: approved tools integrated into the secure case-management system, automatic minimization, role-based access, full logging, a named privacy officer with stop authority, and a workforce that understands the harm, not just the rule.