The home visit had gone well. The caseworker's notes from the visit were brief because the family had been cooperative, the apartment was clean, and the two children were doing fine in school. She typed her observations into the case-management system that evening, then used the agency's new AI documentation tool to help draft the formal home-visit note. The draft came back in under two minutes. It captured the general shape of the visit accurately. But it also included this sentence: "Worker observed that the older child, approximately age 9, had visible bruising on the left forearm." The caseworker had observed no such thing. She had not written anything about bruising in her raw notes. The child did not have bruising. The model had invented a clinical observation, stated it in the precise professional language of a case note, and inserted it seamlessly between two paragraphs of accurate reporting. If that note had been filed without review, it would have entered the case record as a documented observation of a licensed professional. It would have traveled from there to the next court review, the next safety assessment, the next worker who picked up the file. It would have been, in the record, true.

What Hallucination Is and Why It Happens

Before this lesson can be useful, the word "hallucination" needs to be understood precisely. In the world of large language models (LLMs, the AI systems that generate text from prompts and documents), hallucination refers to a specific and documented failure: the model produces output that is internally coherent, professionally worded, and confident in tone but that is factually wrong, unsupported by the source material, or entirely fabricated. It is not a crash. It is not an error message. It is fluent, plausible prose that happens to be false.

The reason it happens is structural, not incidental, and understanding the structure is the point. An LLM does not retrieve facts from a database and then check whether those facts match the source document before producing output. It does not work the way a search engine works, pulling from a verified index. It works by predicting the most probable next word, or more precisely the next "token," given everything it has seen: its training data (billions of pages of text), the instructions it was given, and the document or conversation in front of it. That prediction process is extraordinarily powerful at producing fluent, coherent prose. It is not a verification process. The model cannot distinguish between text it accurately extracted from the document it was given and text it generated because that kind of text typically follows in documents like this one.

Put it in casework terms. The AI tool reads your field notes from a home visit. Those notes are fragmentary, the way real field notes are. They include factual observations, quick impressions, a few abbreviations. The model's job is to turn that into a formal case note. It knows what formal case notes look like because it was trained on millions of documents, many of them in the language of social work and child protective services (CPS). When it encounters a home visit with two children and a parent and some ambiguity in the notes, it predicts the language that typically appears in such notes. Some of that predicted language will be accurate to your visit. Some of it will be language that is statistically plausible in this context but was never part of what you saw. The model does not know the difference. Neither will you unless you read every word with the source notes in hand.

This is not a bug that will be patched in the next software update. The architecture of the technology produces this behavior. Improvements in training data, in retrieval techniques, and in model design can reduce the frequency and severity of hallucinations. They cannot eliminate the underlying tendency of a next-token prediction system to generate plausible text where accurate text is unavailable. Every AI documentation tool in this field today carries this risk. The question is not whether hallucination can happen; it is whether your verification practice will catch it before it harms someone.

Three Failure Modes in a Case Record

Within a case record, hallucination shows up in three specific and distinct ways. Each one has a different source, a different appearance, and a different path to harm. Treating them as one undifferentiated problem makes it harder to catch them. Understanding each one separately makes the verification practice more targeted and more reliable.

Invented Observations

An invented observation is what appeared in the opening of this lesson: a specific, concrete detail about what the worker saw, heard, or documented during a contact or visit that was never part of the worker's actual notes or experience. It appears in the draft as if it belongs there. It uses the specific, professional vocabulary of a case note. It does not stand out as strange in tone or register. It is wrong.

Why does this happen specifically with observations? Because the model is trained on large volumes of case notes, court reports, and human-services documentation. That training data contains many home visit notes that describe conditions in the home, the appearance and behavior of children, and the observations of the worker. The model has learned the patterns: what sentence follows "Worker observed that the children appeared..." or "The home environment showed evidence of...". When your field notes are ambiguous or when the model cannot confidently extract a specific observation from what you gave it, it fills the gap with a statistically plausible continuation. The result reads exactly like the real observations around it.

The harm pathway is direct and severe. Case notes are legal records. They are submitted to courts, reviewed by supervisors, read by attorneys, and used as the basis for safety assessments and placement decisions. A fabricated observation of bruising, or of a chaotic home environment, or of a parent's intoxicated behavior, or of a child's distressed affect carries evidentiary weight that the observation does not deserve, because it never happened. In a dependency hearing, in a termination of parental rights proceeding, in a safety assessment that determines whether a child can remain in the home, a fabricated observation can tip a decision. The family facing that decision will never know that the detail on page 4 of the case record was invented by a text-prediction system at 9:47 PM on a Tuesday.

An invented observation in a court report is not a clerical error. It is a false statement in a legal record with the worker's name on it.

Misapplied or Wrong Policy

The second failure mode occurs when an AI tool is used to apply policy to a case and applies the wrong policy, applies policy to a situation it does not govern, or states a policy rule incorrectly. This is distinct from invented observations because it operates on the regulatory and eligibility side of the work rather than on the factual-description side.

Consider a benefits-eligibility context. A caseworker uses an AI tool to check whether a family qualifies for the Supplemental Nutrition Assistance Program (SNAP), the federal food-assistance benefit commonly known as food stamps, under their state's specific rules. The model returns a determination: "Based on household income and size, this family does not meet the gross income threshold for SNAP eligibility under 7 CFR 273.9." The rule is cited. The output looks exactly like a researched policy determination. But the specific income threshold cited is for a household that does not have a categorically eligible member. This family has a child receiving Supplemental Security Income (SSI), which makes them categorically eligible under a separate provision, meaning the gross income test does not apply to them at all. The family would have qualified for SNAP. The AI applied the wrong rule because the rules are complex, the exceptions are nested, and the model generated a plausible-looking policy citation without accurately navigating the structure of the regulation.

Policy misapplication can run in multiple directions. The model may apply a rule that was superseded by a state administrative update that postdates its training data. It may apply a federal standard where a more generous state option applies. It may apply an eligibility rule for one program to a determination under a related but distinct program. It may confabulate a specific income figure or asset threshold that is close to but not equal to the actual rule. In every case, the output will look like a researched policy determination. It will cite the regulation. It will use the correct program names and the correct legal vocabulary. And it will be wrong.

The harm in a benefits context is concrete: a family that should receive food assistance does not. A person who should qualify for Medicaid, or for housing voucher priority, or for a disability benefit, receives a denial based on a policy determination that was never correct. They may not have the knowledge or the resources to challenge it. If they request a fair hearing, the erroneous determination may not be discovered until the hearing itself, at which point the agency must explain how it made a denial on the basis of a policy rule that did not apply. The due-process costs of that mistake extend well beyond the original determination.

The same failure mode applies in child welfare. An AI tool used to help draft a safety assessment may cite a statutory substantiation standard that applies in a different context, or may mis-state the threshold for a mandatory report under the state's mandated reporter law. A worker who relies on that citation without verifying it against the actual statute or policy manual is working from a wrong foundation.

Fabricated History

The third failure mode is in some ways the most insidious because it operates directly on the record's past rather than its present. Fabricated history occurs when an AI summarization or drafting tool produces text that refers to a prior incident, a prior service, a prior contact, or a prior finding that is not in the case record and did not happen.

This failure mode is particularly likely when a tool is given a partial record or a summary to work from rather than the complete file. Imagine a worker preparing a court report for a family that has been open for six months. She inputs her most recent case notes and asks the AI to help draft the background section of the report. The model, drawing on the patterns of court reports it was trained on, knows that court reports typically include a prior history section. It generates one. That section may accurately reflect the limited information in the notes the worker provided. It may also include a reference to "a 2024 TANF (Temporary Assistance for Needy Families) sanction for failure to comply with the work requirement" that never appears in the actual case record because it never happened. Or a reference to a prior CPS report that was subsequently unsubstantiated, with the substantiation status omitted. Or a prior drug and alcohol evaluation that the client underwent as part of a different case that has since been closed, cited as if it were a current finding.

In each case, the fabricated history carries implications that are not just factual errors but characterizations: this is a family with a history of benefits non-compliance, or with prior CPS involvement, or with substance use concerns. Those characterizations affect how a judge reads the report, how a guardian ad litem advocates for or against the family's position, and how the next worker who opens the file approaches the family. The family has no mechanism to discover that a particular sentence was generated by a predictive text system rather than extracted from the record, because the sentence looks like every other sentence in the report.

Fabricated history can also harm in the opposite direction: it can create the appearance of services delivered that were not delivered, contacts made that were not made, or safety planning completed that was never done. In those cases, the fabrication obscures a gap in service delivery rather than impugning a family's character, but the effect on court review is equally distorting.

The model generates what typically appears in documents like this one. If what typically appears in court reports is a prior-history section, the model will provide one, whether the history is in the record or not.

Why the Confident Tone Is the Specific Danger

There is a feature of AI hallucinations that makes them substantially more dangerous than, say, a typo or a wrong date. That feature is tonal indistinguishability. When an LLM hallucinates, it produces the false output in exactly the same confident, professional tone as its accurate output. There is no asterisk. There is no hedged language like "possibly" or "I believe." There is no change in formatting or register. The sentence "Worker observed visible bruising on the left forearm of the older child" reads identically whether the model accurately extracted that observation from the worker's notes or invented it from scratch. The sentence "This family was sanctioned under TANF in 2024" reads identically whether the model found that fact in the file or generated it because it statistically follows the kind of text around it.

This is the specific failure that makes AI hallucination different from every other source of documentation error a caseworker might catch through normal review. A worker who writes down the wrong date usually knows the right date. A worker who misremembers a number can check the source. A worker who writes something ambiguously can be asked to clarify. But a worker reviewing an AI-drafted document looking for errors is looking for something that does not look like an error. The false sentences are well-constructed, grammatically correct, and contextually plausible. They belong to the same professional register as the surrounding text. The only way to find them is to check every specific factual claim against the source.

This is also different from the familiar experience of a draft that was wrong because the worker who wrote it was misinformed. If a colleague drafts a court report with an incorrect history section because they misread the file, you can ask them to show you where they found the information. The answer will reveal the error. If an AI drafts a report with an incorrect history section, asking the AI where it found the information may produce a confident-sounding answer that also does not correspond to any specific page in the actual record. The model may hallucinate its own sourcing.

Some AI documentation tools include citation features, sometimes called grounding or retrieval-augmented generation (RAG, a technique that connects the model to a specific document set before generating output), that attempt to address this problem by linking each output statement to a specific source passage. These features reduce the risk of hallucination. They do not eliminate it. A grounded model can still hallucinate; it can produce a citation that links to a source passage that does not actually support the claim, or it can generate content in areas where the source document has gaps. Grounding is a mitigation, not a guarantee, and the verification discipline remains essential even when the tool includes citation support.

What Hallucination Means for Real Families

This lesson has so far described hallucination as a technical failure mode. It is worth spending some time on the people those technical failures affect, because in this field the distance between a generated sentence and a harmed family is shorter than it is in almost any other domain where AI is being deployed.

Consider the case of a family in a dependency proceeding where custody is at issue. The judge in that case will read the caseworker's court report. They will read it as the documented, professional judgment of a licensed worker who observed the family, reviewed the record, and applied their training to what they saw. If the report includes a fabricated observation of a child in distress during a home visit, the judge will weigh that observation. If the family's attorney is thorough and the worker is available for cross-examination, the fabricated observation may be identified and the record corrected. But if the report was filed, accepted, and the judge ruled before the error was identified, the damage to the family cannot be automatically undone. A child who was separated from a family on the basis of a case record that included a hallucinated observation by an AI system was harmed in a way that no subsequent correction of the record can make whole.

Consider the family that was denied SNAP benefits because an AI tool misapplied the eligibility rule. The immediate harm is material and immediate: they have less food. But the downstream harms extend further. They may have been in a period of acute financial stress when the denial arrived, making the timing particularly damaging. They may lack the knowledge, language access, or time off from work to request a fair hearing and challenge the denial. They may not know that the denial was based on a policy that did not apply to them. They will experience the denial as a statement by the government that they do not qualify, without any visible signal that the determination was the product of a text-generation system making a statistical guess about policy.

Consider the worker who filed the AI-drafted case note with the fabricated observation of bruising and did not notice it. That worker will likely face a review. The note bears their name and their professional signature. Whether or not the agency has an AI policy that addresses review obligations, the worker is accountable for what went into the record under their name. The defense that "the AI wrote it" does not transfer legal or professional liability. A social worker who submits documentation that contains false statements is accountable for those statements regardless of the tool that generated the first draft. The obligation to verify is not an optional enhancement to the workflow; it is a basic requirement of professional practice.

The Only Reliable Control

If hallucination is structural, and if it is tonally indistinguishable from accurate output, what does a caseworker actually do? The answer is not reassuring in terms of speed, but it is clear in terms of what works: human verification of every factual claim against the source record, before any AI-assisted document is filed.

Verification in this context means something specific. It means not just reading through the AI-drafted document for general sense and flow. It means locating every specific factual claim in the draft and confirming that claim against the original source: the worker's own field notes, the case file, the policy manual, the prior records in the case-management system. It means treating the AI draft as a first draft from a very capable but unreliable colleague, one who writes beautifully and confidently and who cannot be trusted not to invent a detail or misapply a rule.

Verifying Observations

For home visit notes and contact notes, verification means reading each specific observation in the AI draft and asking: did I observe this, or did the model add it? The comparison is between the draft and the worker's own field notes, not between the draft and the worker's general memory of the visit. Memory is fallible. If an observation appears in the AI draft that does not appear in the worker's raw field notes, and the worker does not independently recall making that observation, the observation must be removed. Adding observations that were not in the original notes, even if they seem plausible or consistent with the visit, crosses the same line as inventing them. The case note documents what was observed, not what the model inferred was probably observed.

This verification step requires that workers maintain their own field notes with enough specificity to serve as a check against the AI draft. If the field notes consist only of a few key words and the worker is relying on memory to fill in the rest, the verification step is compromised. Good AI-assisted documentation practice starts with good field notes, and those notes should be specific enough that any claim in the AI draft can be traced back to them or identified as an addition.

Verifying Policy

For any AI-assisted output that applies a policy rule, the verification step is to go to the actual policy source. That means the state policy manual, the relevant federal regulation, the agency's operating procedures, or the specific statutory provision. It does not mean asking the AI to confirm the policy it cited. The model that misapplied the policy in the first place can generate a plausible-sounding confirmation of the same wrong rule. The verification must go to a source that is independent of the model.

This step is particularly important for rules that change: state benefit rules that are updated through annual legislative sessions or administrative actions, income thresholds that are indexed to federal poverty levels and updated each year, eligibility criteria for programs that have been modified since the model's training data was collected. An AI tool trained on data from 18 months ago may be applying rules that have since changed. The worker's obligation to verify against a current source is independent of when the AI's training data was collected.

Verifying History

For any AI-assisted document that includes a summary of prior history, the verification step is to go to the actual case record and confirm each historical claim against what is actually there. This is the slowest and most demanding verification task because the history section of a court report may include references to events across months or years of case activity. But it is also the most important, because fabricated history is the failure mode most likely to escape notice. A worker reviewing an AI-drafted court report for accuracy may check the recent observations carefully and verify the current service plan, but skim through the history section because it looks familiar. That skim is where the fabricated prior incident or invented service episode lives.

In practice, verifying the history section means opening the case-management system alongside the AI draft and tracing each specific historical reference to a specific entry in the system. "A CPS report was received in March 2024" should correspond to a specific intake record in the system. "The family completed a parenting class in the fall of 2025" should correspond to a documented service record. "A substance use assessment was completed and returned negative" should correspond to a specific document in the file. Claims that cannot be traced to a specific record should be removed from the draft.

The Role of Workflow and Supervision

Individual verification discipline is necessary. It is not sufficient at scale. An agency deploying AI documentation tools across a unit of 15 caseworkers, each carrying 20 to 30 families, each generating home visit notes and court reports under constant time pressure, cannot rely solely on individual workers maintaining their verification practice when they are tired, when their caseload spikes, and when the AI draft is almost entirely correct. The verification discipline has to be built into the workflow, not just required of individual workers.

That means supervisory review that treats AI-assisted documentation the same way it treats all documentation: not as a completed product but as a draft to be checked. It means agencies maintaining clear, written policies about which documents require verification before filing and what that verification must include. It means training that teaches workers to verify as a specific skill, not just instructs them to be careful. It means AI tool procurement and deployment decisions that prioritize tools with grounding and citation features over tools that generate free-form prose from prompts alone.

It also means creating the conditions under which verification is actually possible: workers need enough time in their workflow to check their documentation. If the documentation burden is so heavy that the only way to keep up is to trust the AI draft and file it, the verification step will be skipped. One of the genuine benefits of AI documentation tools is that they can return hours to caseworkers by reducing the time needed to produce a complete draft. That time dividend should go to verification and to direct time with families, not to filling the saved time with more files. An agency that deploys AI documentation tools without examining whether workers have the capacity to verify every draft has not reduced its documentation risk; it has moved the risk to a new form.

Key Takeaways

• AI hallucination is a structural feature of how large language models (LLMs) work, not a bug that will be patched away. LLMs generate statistically plausible text rather than retrieving and verifying facts, which means they can and do produce confident, professionally worded output that is factually false.
• In a case context, hallucination appears in three specific failure modes: invented observations (a detail in a home-visit note that the worker never made), misapplied or wrong policy (an eligibility rule cited incorrectly or applied to a situation it does not govern), and fabricated history (a prior incident, service, or finding that is not in the case record and did not happen).
• Each failure mode has a distinct and real harm pathway. An invented observation in a court report can contribute to a removal or a placement decision affecting a real child. A wrong eligibility rule can deny SNAP, Medicaid, TANF, or housing benefits to a family that qualified. A fabricated history can shape a judge's reading of a case for months or years.
• The most dangerous feature of hallucinations is tonal indistinguishability: a fabricated sentence reads exactly like an accurate sentence in the same document. The error does not announce itself. The worker reviewing the draft is the error-detection mechanism, which means verification is not optional.
• Verification means tracing every specific factual claim in an AI-assisted document to a specific source: the worker's own field notes for observations, the actual policy manual or regulation for policy claims, and the case-management system record for historical claims. A claim that cannot be traced to a source should be removed from the draft.
• Grounding techniques such as retrieval-augmented generation (RAG) reduce but do not eliminate hallucination risk. Verification discipline remains essential even when the AI tool includes citation features.
• The caseworker who files an AI-assisted document bears professional and legal accountability for its contents regardless of what tool generated the first draft. "The AI wrote it" is not a defense in a court, a licensing review, or an agency investigation.
• Agencies deploying AI documentation tools must build verification into the workflow at the supervisory and policy level, not rely solely on individual workers under caseload pressure to maintain their verification practice. The time AI returns from drafting should go to verification and to direct time with families.