The vendor demo had been impressive. The AI documentation tool transcribed a mock home visit, drafted a clean case note in ninety seconds, and the account executive closed with a line the agency's deputy director would repeat for weeks: "Your data never trains our models, and we are fully compliant." Six months into the pilot, a county attorney preparing for a contested termination hearing asked a question nobody on the agency side could answer: where, physically, are our case notes stored, who at the vendor can read them, and what happens to the transcript audio of a child describing abuse after the note is drafted? The agency had a signed contract and a glossy security one-pager. It did not have a data flow diagram, a subprocessor list, a breach-notification clause with a deadline, or a record of having ever tested the vendor's claims. The deputy director discovered, in the worst possible setting, that "the vendor said it was compliant" is not the same thing as the agency knowing it was true. In human services, the agency is the legal custodian of the most sensitive personal data the government holds, and that custody does not transfer to a vendor no matter what the contract says.

The Obligation That Never Leaves the Agency

Start with the principle that governs everything else in this lesson, because every practical step flows from it. When an agency contracts with an AI vendor to process case data, the legal and ethical obligation to protect that data does not move to the vendor. It stays with the agency. The vendor becomes a processor acting on the agency's instructions, but the agency remains the custodian, the entity accountable to the court, the auditor, the advocate, and above all the family whose data it holds. A data breach at the vendor is the agency's breach. A misuse of case data by the vendor is the agency's failure to control its processor. A subpoena for case records lands on the agency, and the agency must be able to produce, account for, and protect those records regardless of where the vendor stores them.

This is not an abstraction. Consider the data that flows through an AI documentation tool in this field. A home visit transcript contains a child's voice describing what happens in their home. An intake assessment contains allegations of abuse, a parent's mental health history, a household's immigration status, a teenager's disclosure of sexual abuse. An eligibility record contains a family's income, their Social Security numbers (SSNs), their medical conditions, their criminal history, their substance use treatment records. This is, by any measure, among the most sensitive categories of personal data that exists. PII (personally identifiable information, any data that can identify a specific person) in this context is not a marketing email list. It is the raw material of a family's most vulnerable moments, collected under legal compulsion, and a leak of it can endanger physical safety, not just privacy.

Federal and state law recognize this. Child welfare records are governed by confidentiality provisions tied to federal funding, including the requirements attached to the CCWIS (Comprehensive Child Welfare Information System, the federal framework that funds and regulates state child-welfare data systems). Substance use treatment records carry their own heightened federal protection. Health information may fall under HIPAA (the Health Insurance Portability and Accountability Act, the federal medical-privacy law) when the agency operates a covered program. Benefits data carries program-specific confidentiality rules tied to SNAP, Medicaid, and TANF. Each of these regimes imposes obligations that the agency cannot delegate away. Signing a vendor contract does not exempt the agency from any of them. The due diligence in this lesson is how an agency satisfies obligations it cannot escape.

You can outsource the processing. You can never outsource the accountability. The breach is always yours.

Mapping the Data Flow Before the Contract

The single most common failure in human-services AI procurement is that nobody at the agency can draw, on one page, where the data actually goes. The vendor demo shows a clean interface. It does not show the journey a child's home-visit transcript takes from the caseworker's phone to the model that processes it to the storage where it rests to the third parties who touch it along the way. Before any contract is signed, the agency must produce that map, and the vendor must confirm it in writing.

A data flow map for an AI documentation tool answers concrete questions. When a caseworker records a home visit, where does the audio go first? Is it transmitted to the vendor's servers, or to a cloud provider the vendor uses, or to a separate transcription service? Is the audio retained after the transcript is produced, and if so for how long and where? When the transcript is sent to the model that drafts the note, is that model hosted by the vendor, or is it a third-party model API the vendor calls? If it is a third-party API, the case data is now flowing to a fourth party the agency may never have evaluated. Where does the finished draft rest? Is it inside the agency's case-management system, or does a copy persist on the vendor's infrastructure? Who, at every one of these stops, can technically access the data?

Work an example. A mid-size county runs a 90-day pilot of an AI note tool across 40 caseworkers handling roughly 900 open cases. The deputy director, doing the data flow map properly for the first time during the second pilot, discovers that home-visit audio is stored unencrypted on the vendor's servers for 30 days "for quality assurance," that the drafting model is a general-purpose third-party API the vendor calls rather than a model the vendor controls, and that a separate analytics subprocessor receives metadata about every case for usage reporting. None of this was hidden. All of it was in documents the agency had not read closely, because the demo answered a different question than the data flow map asks. That 30-day unencrypted audio store, holding 900 families' most sensitive disclosures including children describing abuse, is a breach exposure the agency created without knowing it. The data flow map is what surfaces it before a breach does, not after.

The map should be drawn for each distinct data type the tool touches: the raw audio, the transcript, the drafted document, the case metadata, and any data used for the vendor's own purposes. For each, the agency records the origin, every system it passes through, every party that can access it, where it rests, how long it is retained, and how it is eventually destroyed. A vendor that cannot help the agency produce this map, or that resists the exercise, has answered the most important due-diligence question already.

The Questions That Go Past the Security One-Pager

Vendors arrive with a security one-pager: a list of certifications, a sentence about encryption, a logo for a compliance standard. The one-pager is the floor of due diligence, not the substance of it. The substance is in the questions a vendor would rather not be asked, and an agency protecting families must ask them and require answers in the contract, not in the sales conversation.

Data Handling and Model Training

The first cluster of questions concerns what the vendor does with the agency's data. Is case data ever used to train, fine-tune, or improve the vendor's models, either now or under a future policy change? "We do not train on your data" must be a contractual commitment with no exception for "service improvement," "aggregated insights," or "anonymized analysis," because true anonymization of narrative case data is far harder than vendors imply, and a re-identifiable transcript of a child's abuse disclosure is not protected by stripping the name field. Does the vendor retain any copy of the data after the contract ends, and is there a contractual deletion obligation with a deadline and a certification of destruction? Where, geographically, is the data stored and processed, and does that location satisfy any state requirement that resident data stay in-country or in-state?

Subprocessors and the Chain of Custody

The second cluster concerns the parties behind the vendor. Modern AI tools are rarely built by a single company end to end. The vendor uses a cloud host, very likely uses a third-party model provider, and may use additional services for transcription, storage, or analytics. Each of these is a subprocessor, and each one touches the case data. The agency must require a current, complete subprocessor list, a contractual right to be notified before a new subprocessor is added, and assurance that every subprocessor is bound by the same data-protection terms the vendor agreed to. A chain of custody for case data is only as strong as its weakest link, and a subprocessor the agency never evaluated is an unexamined link holding a child's disclosure.

Access Controls and the Insider Risk

The third cluster concerns who, in human terms, can read the data. Encryption protects data from outside attackers, but it does not address the vendor employee with legitimate access. The agency must ask: which vendor personnel can access agency case data, under what controls, with what logging, and for what stated purposes? Is access role-based and least-privilege, or can any support engineer pull up a family's transcript? Is every access logged in a tamper-evident record the agency can review? In a field where a leaked location can endanger a domestic-violence survivor or a child placed in confidential foster care, the insider-access question is not paranoia. It is a direct safety control.

Breach Notification and Incident Response, In Writing

A breach will eventually be tested, and the time to negotiate the response is before it happens, not during. The agency's worst position is a contract that is silent or vague on breach notification, because in that silence the vendor's incentive is to minimize, delay, and characterize the incident in the least alarming terms while the agency's legal duty to notify affected families runs on a statutory clock the agency may not even know has started.

The contract must specify a breach-notification deadline measured in hours, not "promptly" or "as soon as practicable." A common and defensible standard is notification to the agency within 24 to 72 hours of the vendor becoming aware of a suspected breach, with a definition of "breach" broad enough to include unauthorized access, not just confirmed exfiltration. The contract must obligate the vendor to provide the specific information the agency needs to meet its own notification duties: what data was affected, which individuals, when, and how. It must require the vendor to cooperate fully with the agency's investigation and to preserve evidence rather than remediate in a way that destroys the forensic trail.

Consider the downstream consequence of getting this wrong. An agency learns, four months after the fact, that a vendor subprocessor was breached and that transcripts including children's abuse disclosures and families' addresses were exposed. State law required the agency to notify affected individuals within a set window of the agency learning of the breach, but the contract had no vendor-notification deadline, so the vendor's four-month delay became the agency's notification failure. The families, including a domestic-violence survivor whose confidential address was in the exposed data, were not warned in time to take protective steps. The due-process and safety harm here is not hypothetical. A breach-notification clause with a 24-to-72-hour deadline is the difference between an agency that can protect the people in its system and one that learns of the danger to them long after they could have acted.

The Contract Clauses That Actually Protect the People Served

Due diligence becomes real only when it is written into the contract, because a verbal assurance in a sales call has no legal weight when a family's data is exposed. The data-protection terms belong in the contract or a binding data-processing addendum, and an agency without the in-house expertise to draft them should not sign until legal counsel with privacy experience has reviewed the terms. The clauses below are the load-bearing ones.

• Purpose limitation. The vendor may use agency data only to provide the contracted service, with an explicit prohibition on training, product development, marketing, or resale.
• No-training commitment. A clear statement that case data is never used to train or improve the vendor's models, with no carve-outs for anonymized or aggregated use.
• Data residency and ownership. The agency retains ownership of all case data; the vendor acquires no rights; data is stored and processed in agreed jurisdictions.
• Subprocessor control. A current subprocessor list, advance-notice rights, the right to object, and flow-down of all data-protection terms to every subprocessor.
• Breach notification. A defined deadline in hours, a broad definition of breach, and a duty to provide the information the agency needs to meet its own legal notice obligations.
• Audit rights. The right to audit or to receive independent audit reports, so the agency can test the vendor's claims rather than trust them.
• Deletion and return. On termination, the vendor returns all data in a usable format and certifies destruction of every copy, including in backups and subprocessor systems, within a defined window.
• Liability and indemnification. The vendor bears defined responsibility for breaches caused by its failures, so the cost does not fall entirely on the public agency and the families it serves.

One clause deserves emphasis because agencies routinely concede it: the audit right. Due diligence is not a one-time event at signing. The vendor's subprocessors change, its policies update, its security posture drifts. An audit right, or at minimum a contractual right to current independent audit reports on a defined cadence, is what lets the agency verify on a continuing basis rather than trust on a single occasion. The deputy director in the opening scene could not answer the county attorney's questions precisely because the agency had treated due diligence as a signing-day checkbox rather than a standing obligation.

Due Diligence as a Continuing Discipline, Not a Signing-Day Checkbox

The final move is to stop treating vendor due diligence as a procurement gate the agency passes once and to treat it as a standing function the agency operates for the life of the contract. The reason is simple: the risks change after signing. A vendor adds a new third-party model subprocessor to cut costs and never tells the agency, because the contract did not require notice. A vendor's quietly updated privacy policy now permits "service improvement" use of customer data. A subprocessor suffers a breach the vendor does not consider material enough to report under a vague contract. Each of these moves the risk after the procurement decision was made, and an agency that filed its due diligence away at signing will not see any of them.

A continuing discipline looks like a named owner, a calendar, and a record. Someone at the agency owns vendor oversight as part of their actual job, not as a hope. On a defined cadence, the agency reviews the current subprocessor list against the last one, requests the latest independent audit report, confirms the no-training and purpose-limitation commitments still hold under the vendor's current policies, and tests the breach-notification path. Every review is recorded, because the audit trail of due diligence is itself part of what makes the agency's AI use defensible to a court and an advocate. When the contested-hearing question comes, and it will, the agency that operated due diligence as a discipline can answer it: here is where the data lives, here is who can touch it, here is our subprocessor list as of last quarter, here is the breach clause, here is the last audit. The agency that treated due diligence as a one-time checkbox can only say what the vendor told it.

This connects directly to the program's spine. Privacy and due process are the perimeter, and the most vulnerable people's most sensitive data is what that perimeter protects. An agency that cannot account for where a child's abuse disclosure travels after an AI tool processes it has not protected that perimeter, regardless of how good the note-drafting was. The hours an AI tool returns to caseworkers are a genuine benefit, but they are only defensible if the data that made them possible was held to the standard the people in the system are owed. Vendor due diligence is how the agency keeps the benefit without surrendering the obligation it can never delegate away.

Key Takeaways

• The legal and ethical obligation to protect case data never transfers to the vendor. The vendor is a processor; the agency remains the custodian and is accountable to the court, the advocate, and the family. A breach at the vendor is the agency's breach.
• Human-services case data, including home-visit transcripts of children describing abuse, abuse allegations, immigration status, and treatment records, is among the most sensitive PII (personally identifiable information) the government holds. A leak can endanger physical safety, not just privacy, and is governed by CCWIS, HIPAA, and program-specific confidentiality rules the agency cannot delegate away.
• Before signing, the agency must produce a one-page data flow map for each data type: where audio, transcript, draft, and metadata originate, every system and party they pass through, where they rest, how long they are retained, and how they are destroyed. The map surfaces hidden exposures, such as a 30-day unencrypted audio store, that a vendor demo never reveals.
• Due diligence goes past the security one-pager into the questions vendors prefer to avoid: is data ever used for training (with no carve-out for anonymized or aggregated use), who are the subprocessors and is the full chain of custody bound by the same terms, and which vendor employees can access a family's data with what logging.
• Breach notification must be a contractual deadline measured in hours (commonly 24 to 72 hours of vendor awareness), with a broad definition of breach, because a vendor's delay becomes the agency's failure to notify affected families in time for them to take protective steps.
• The protections must be written into the contract or data-processing addendum: purpose limitation, a no-training commitment, data residency and ownership, subprocessor control, breach notification, audit rights, deletion and return with certified destruction, and liability and indemnification. A sales-call assurance has no legal weight when data is exposed.
• The audit right matters most because it converts trust into verification: the agency can test the vendor's claims on a cadence rather than trust them once at signing.
• Due diligence is a continuing discipline with a named owner, a calendar, and a recorded review of subprocessors, audit reports, policy changes, and the breach path. The recorded trail is part of what makes the agency's AI use defensible when a contested hearing asks where the data lives and who can touch it.