โ†
AI for Healthcare & Clinical Practice
Aware ยท M1 ยท lesson 1 of 19 ยท in progress
Preview โ€” browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll โ†’
AI Hallucinations in a Clinical Context
๐Ÿ“–
now learning

AI Hallucinations in a Clinical Context

15 min

A resident asked an AI tool for the maximum daily dose of a medication and got a clean, specific, confident answer: a number, a unit, a frequency. It was wrong, and not by a rounding error but by a factor that would have mattered. The resident, trained to trust a reference, nearly wrote it. What saved the patient was a habit, not the tool: she cross-checked it against a real formulary because it was a dose, and doses get checked. This is the clinical hallucination in its natural habitat, not a bizarre glitch but a normal-looking answer with a lie inside it, delivered in the same voice as a hundred correct answers before it. We have named hallucination already. This lesson goes deeper, into the specific forms it takes in medicine, why it happens exactly where it is most dangerous, and the concrete habits that catch it before it reaches a patient. By the end you should be able to look at any AI output in your workflow and know, in a second, whether it belongs to the small set of answers you must verify before acting.

What a Clinical Hallucination Really Is

A hallucination is a confident, plausible output that is not true, and the word is slightly misleading because it suggests something rare and pathological. In a generative model, producing a hallucination is not a malfunction; it is the same next-token process that produces every correct answer, applied to a case where the most plausible-sounding continuation happens not to match reality. The model is not lying, because lying requires knowing the truth and choosing to distort it. The model has no access to truth at all; it has patterns of language. When those patterns point at reality, you get a correct answer, and when they do not, you get a hallucination, and from inside the model and on the page these two are identical events. This is the single most important thing to internalize: the tool cannot tell you which one it just did, because it does not know there is a difference.

In a clinical setting this matters more than almost anywhere else, because medicine runs on specifics, and specifics are exactly what a model is most likely to fabricate. A vague, hedged, generally-true statement is easy for a model to produce accurately. A precise number, a named study, an exact dose, a specific interaction, these require the model to have the real fact, and when it does not, it does not fall silent or hedge; it generates a specific-looking answer anyway, because a specific answer is the plausible shape of a response to a specific question. The precision that makes a clinical answer useful is the very thing that makes a hallucinated one dangerous, because a confident, precise, wrong number is far more likely to be acted on than a vague one.

Fluent Is Not Accurate

Hold two words apart in your mind, because a generative model has taught a generation of users to collapse them: fluency and accuracy. Fluency is how smoothly and confidently the text reads, whether the grammar is clean, the tone is authoritative, the structure is orderly. Accuracy is whether the content corresponds to reality. A language model is optimized, at its core, for fluency: its whole training objective is to produce the most plausible next token given everything before it. Accuracy is something we hope emerges from that process, and it usually does when the true answer is also the most plausible-sounding one. But when a fluent continuation and a true continuation diverge, the model has no mechanism that prefers the true one. It will hand you the fluent falsehood with the same polish it hands you the fluent truth.

Consider a concrete micro-scenario. A clinician asks a tool to summarize the mechanism by which a common antihypertensive lowers blood pressure. The answer is three tidy sentences, grammatically flawless, using the right vocabulary: vasodilation, afterload, receptor selectivity. It reads like a textbook. Two of the three sentences are correct and one quietly attributes the drug to the wrong receptor class. Nothing in the prose signals which sentence is the weak one. The fluency is uniform across all three because fluency is what the model produces uniformly. The reader who equates a smooth read with a reliable read has no way to locate the error, and that is precisely the trap: the smoothness that makes the output pleasant to read is the same smoothness that camouflages the mistake inside it.

The Clinical Taxonomy: The Shapes It Takes

Hallucinations are not all the same, and recognizing the specific forms helps you know where to look. In clinical work they cluster into a handful of recurring types.

The Fabricated Fact

The tool states a specific clinical fact that is simply false: a wrong maximum dose, an invented drug interaction, a lab reference range that does not match reality, a mechanism of action that sounds right and is not. These are the most directly dangerous because they can be acted on immediately. They cluster around numbers and pharmacology, the domains where precision is required and the model is most tempted to supply it.

Return to the opening scene and slow it down. The resident asked for a maximum daily dose. The tool returned a value with a unit and a dosing interval, formatted exactly the way a formulary would format it. Nothing about the shape of the answer was wrong; only the number was. Had she pasted it into an order, the order would have looked clinically ordinary. That is the signature of a fabricated fact: it wears the costume of the correct answer perfectly, down to the units and the sig, and the only thing missing is correspondence to the real reference. The fabricated fact does not announce itself by being oddly phrased or hedged. It announces itself, if at all, only when you set it beside a trusted source and the two numbers fail to match.

The Phantom Citation

Ask for evidence and the model produces a reference: authors, a journal, a year, sometimes a plausible-sounding title, that does not correspond to any real paper, or that garbles a real one into something it never said. This is so common and so well documented that a named citation from a generative tool should be treated as a claim to verify, never as evidence in itself. The danger is amplified in medicine because a citation carries authority; a fabricated one lends false weight to whatever claim it was attached to.

A worked example makes the shape unmistakable. A clinician drafting a teaching point asks the tool to support a claim about a treatment threshold. It replies with a reference that looks impeccable: a real journal name, a plausible author surname, a year within the last decade, a volume and page range, a title that reads exactly like a title a study on that topic would carry. Every surface cue of legitimacy is present. When the clinician searches the actual database, no such article exists, or one exists with those authors but says something different, or the page range points to an unrelated paper. The phantom citation is not a lazy fabrication; it is a fluent reconstruction of what a supporting reference should look like, assembled from the statistical shape of thousands of real citations. That is why it is so persuasive and why the only reliable response is to look the reference up rather than to judge whether it seems real.

The Invented Patient Detail

When a tool summarizes or drafts and lacks a piece of information, it may fill the gap with a plausible detail rather than leaving it blank. An ambient scribe writes a normal exam finding for a system that was not examined. A summary states a history the record does not support. This is the most insidious form in documentation, because the invented detail is plausible, is embedded among true details, and becomes part of the record the moment it is signed.

Picture the ambient scribe in a real encounter. A clinician sees a patient for a cough, listens to the chest, and documents the lungs. The clinician never examined the abdomen, and never said a word about it out loud. The ambient tool, generating a complete-looking SOAP note from the conversation, writes into the physical exam: "Abdomen soft, non-tender, no organomegaly." It is a perfectly normal finding. It is the finding that appears in the overwhelming majority of notes, so it is the most plausible next token when the note reaches the abdominal exam section. It is also a pure fabrication, because no abdominal exam happened. The line is not flagged, not italicized, not set apart. It sits in the objective section between two things that are true, and if the clinician skims and signs, an exam that never occurred becomes a documented finding in the legal record.

Now trace what the invented detail does downstream. A week later the patient returns with abdominal pain and a covering clinician pulls the prior note. It says the abdomen was soft and non-tender at the last visit. That phantom finding now anchors a clinical trajectory: it suggests the pain is new, that nothing was present before, that a rapid change has occurred. The reasoning built on a fabricated pertinent negative can send the next clinician down the wrong path. This is why the invented patient detail is uniquely corrosive: it does not just misinform the moment it is written, it contaminates every decision that later relies on the record as ground truth.

The most defensible attestation on an ambient note is not "AI-assisted documentation." It is the clinician's confirmation that every finding in the note was actually observed or obtained. Signing a note means asserting that its contents are true, and a normal-looking finding you did not examine is not made true by being plausible.

The Confident Over-Reach

Asked something at or beyond the edge of what is known, or specific to a case it cannot see, the model answers confidently anyway rather than saying it cannot know. Ask it to interpret a specific patient's constellation of findings it was never given, and it will produce a confident clinical impression built on nothing. The failure here is the absence of appropriate uncertainty; a good consultant says "I am not sure, let us look," and the model, by default, does not.

The tool cannot tell you whether it just gave you a fact or a fabrication, because to the model there is no difference. That judgment is yours, and it is the whole job. No setting, no model version, and no vendor assurance transfers that judgment off you, because the tool simply has nothing to make that judgment with in the first place.

Why It Happens Exactly Where It Is Most Dangerous

There is a cruel logic to where hallucinations concentrate, and understanding it lets you predict your own risk. Hallucination rises with specificity, because specific answers require specific facts the model may not have. It rises with rarity, because the rarer the true answer, the less it appeared in training, and the more the model must improvise a plausible substitute. It rises when the model is asked about information it cannot actually access, such as a particular patient's data that was never provided, because it will fill the void rather than report the gap. And it rises under leading prompts, because a question that presumes an answer pushes the model to produce it.

Now overlay that on clinical work and the danger becomes clear. The questions that matter most in medicine are often the specific ones (what exact dose, what is the interaction), the rare ones (an unusual presentation, a niche drug), and the patient-specific ones (what is going on with this person). Those are precisely the conditions under which hallucination is most likely. The model is most trustworthy on the common, general, well-trodden question and least trustworthy on the specific, rare, individual one, which is the inverse of what a busy clinician often wants to lean on it for. The safe use of these tools requires holding that inversion constantly in mind: the more the answer matters to a specific patient, the less you can take the model's word for it.

The table below sorts common clinical uses by where they sit on that risk gradient. Read it not as a fixed rulebook but as a way to train your instinct for which outputs pass through and which get stopped for verification. The statistics you might encounter about hallucination rates, whatever figure a vendor or study cites, are a number to verify against your own experience and your institution's data, not a reassurance to repeat blindly; the useful lesson is the direction of the gradient, not any single percentage.

AI use in the workflowRisk of harmful hallucinationWhat the clinician does
Rewriting your own text into plainer languageLowRead for fidelity, use freely
Drafting a general patient-education explanationLow to moderateRead for accuracy, then release
Summarizing a source document you providedModerateSpot-check summary against the source
Reciting a guideline from memory, unprompted by a sourceHighGround in the real guideline and verify
Supplying an exact dose or renal adjustmentHighVerify against a trusted formulary before acting
Producing a named citation or a statisticHighLook it up; treat as an unverified claim
Interpreting a specific patient's findings it cannot seeVery highDo not rely on it; the impression is baseless

The pattern in the right two columns is the whole discipline: as you move down the table toward the specific, the patient-relevant, and the numerically precise, the correct clinician behavior shifts from "use freely" to "verify before acting." Nothing about the tool's tone changes as you move down that list. The change is entirely in the character of the question, which is exactly why the question, not the answer, is what should trigger your caution.

A Worked Example: Catching It in Real Time

Watch the resident from the opening handle a second question well, because the contrast is the whole lesson. She asks the tool to explain a drug interaction for patient education, and it produces a clear, general explanation of how two common drug classes interact. This is a well-trodden, general question, and the answer is sound; she uses it as a draft, reads it for fidelity, and moves on. Then she asks the specific, high-stakes question: the exact dose adjustment for this interaction in a patient with reduced kidney function. The tool answers with the same confidence and a precise number. But she has internalized the inversion: this is specific, patient-relevant, and high-stakes, exactly where hallucination lives. She treats the number as an unverified claim and checks it against a trusted renal-dosing reference, where she finds the tool's figure was wrong. Same tool, same session, same confident tone, two answers, and she trusted them exactly as far as each deserved. She did not distrust the tool globally, which would have thrown away the useful general explanation, and she did not trust it globally, which would have carried the wrong dose to the patient. She calibrated, and calibration is the skill.

Notice what did the work. It was not detecting a tell in the text, because there is no tell; the wrong answer looked exactly like the right one. It was a rule applied before reading the answer at all: this kind of question, specific and high-stakes, gets verified regardless of how confident the output sounds. The defense against hallucination is not a better eye for spotting fabrication in the moment. It is a habit of deciding, based on the question, how much verification the answer requires before you ever see it.

Catching It Before It Reaches the Chart

The costliest place for a hallucination to survive is the signed note, because at that moment it stops being a draft and becomes the legal record. So the practical skill is catching it in the narrow window between generation and attestation. Consider an ambient note again, and read it the way a defensible clinician reads it: not for whether it sounds right, but for whether every objective finding was actually obtained. A useful reading pass asks a single question of the objective section: did I do this exam, or is this the note describing an exam I did not perform. The moment a line describes a system you did not examine, you delete it or replace it with what you actually found, including the honest pertinent negatives you truly elicited.

The contrast in note language is instructive. A before-and-after makes it concrete. Before, the AI-drafted line reads, with fluent false completeness: "Neuro: alert and oriented x3, cranial nerves II to XII intact, no focal deficits, gait normal." If you did not test cranial nerves or watch the patient walk, that line is a fabrication dressed as thoroughness. After, the defensible line reads only what happened: "Neuro: alert and oriented; formal cranial nerve and gait exam deferred." The second version is shorter, less impressive, and true, and truth is the only property that protects the patient and the clinician when the record is read later. The goal of the pre-attestation pass is not to make the note look complete. It is to make the note match reality, so that grounding, retrieval, and every downstream decision rest on findings that were genuinely observed.

A note is not a summary of what a typical encounter contains. It is an attestation of what this encounter contained. An ambient tool generates the former by default; the clinician's job is to convert it into the latter before signing.

The Defenses That Actually Work

Because you cannot detect a hallucination by how it looks, the defenses are structural rather than perceptual, and they are learnable habits rather than technology. The first and most powerful is grounding: give the model the real source to work from rather than asking it to recall. A model summarizing a guideline you pasted in is far safer than one reciting a guideline from memory, because the former can be checked against the text and the latter is improvising from patterns. Most safe clinical AI workflows are built on this single move, and later levels turn it into real systems.

Grounding deserves one concrete illustration because it is the move most people underuse. Compare two prompts. The first asks, "What does the guideline recommend for this situation," and the model recites from memory, improvising from patterns, with no source you can check. The second pastes in the actual guideline text and asks, "According to the text above, what is recommended for this situation, and quote the relevant sentence." The second is dramatically safer, not because the model became more honest, but because retrieval put a checkable source in front of it and asked it to point at that source rather than invent one. When the model quotes a sentence, you can confirm the sentence exists and says what the model claims. Grounding does not make hallucination impossible, but it converts a memory recitation you cannot check into a claim about a document you can.

The second is verification of specifics against a trusted source. Any specific, consequential claim, a dose, a number, a citation, an interaction, gets confirmed against a real reference before it informs care. This is not a burden if you target it correctly; you are not re-checking everything, only the specifics that would change a decision, which is a small set. The third is neutral prompting: ask open questions rather than leading ones, because presuming an answer manufactures one. The fourth is calibrated skepticism by question type: reflexively trust the model less as a question becomes more specific, rarer, and more patient-relevant, and trust it more on the common, general, easily-verified. And the fifth, underlying them all, is the mindset that the model's confidence is not evidence; a fluent, precise, assured answer has earned exactly as much trust as an unverified claim, which on anything that matters is: verify first. These five habits, grounding, verifying specifics, prompting neutrally, calibrating by question type, and refusing to read confidence as truth, are the entire practical defense, and none of them requires you to understand a neural network. They require you to be a clinician who checks.

The Special Danger of the Confident Tone

It is worth dwelling on why hallucination is so much more dangerous than an ordinary error, because the difference is psychological and it is working against you. When a human colleague is unsure, they signal it: they slow down, they hedge, they say "I would double-check that." Those signals are how we allocate our own scrutiny; we lean in when someone sounds uncertain and relax when they sound sure. A generative model breaks this instinct completely, because it delivers a fabrication in exactly the same assured, articulate voice as a fact. The uncertainty signal that you have relied on your entire clinical life is simply absent, and worse, it is replaced by a uniform confidence that actively invites trust. You are not merely lacking a warning; you are being given a false all-clear.

This is why clinicians who are otherwise careful get caught. They are not careless; they are applying a lifetime of well-calibrated instincts to a source those instincts were never built for. The mental adjustment required is genuinely unnatural: you must decouple your sense of how much to trust an answer from how confident it sounds, and instead attach it to what kind of question you asked. That is hard, because sounding confident and being right have been correlated for your whole career when the speaker was a human. With a generative model the correlation vanishes, and the clinician who keeps relying on tone as a proxy for reliability will eventually trust a confident fabrication. Naming this trap is half of defeating it: the next time an AI answer sounds authoritative on a specific, high-stakes clinical question, let that very authority be the thing that prompts you to check, rather than the thing that reassures you.

Building the Habit Into Your Day

The good news is that a full defense against clinical hallucination does not require constant, exhausting suspicion of everything, which would be unworkable and would negate the tool's value. It requires a small, targeted reflex applied at the right moments. In practice, this looks like a quiet internal question you ask before acting on any AI output: is this specific, consequential, and about a particular patient? If it is a general explanation you will read and reshape, use it freely. If it is a dose, a number, a citation, an interaction, or a claim about the patient in front of you, tag it as unverified and confirm it against a real source before it touches care. That single reflex, triggered by the character of the question rather than the confidence of the answer, catches the large majority of dangerous hallucinations while leaving the tool's genuine usefulness intact.

Over time this becomes automatic, the way checking a high-alert medication or confirming a patient's identity became automatic. You stop experiencing it as extra work and start experiencing it as simply how one uses these tools, the same way a good clinician does not experience double-checking an insulin dose as a burden but as part of what giving insulin means. The clinicians who will thrive alongside generative AI are not the ones with a magical ability to spot fabrication; no one has that, because there is nothing to spot. They are the ones who have built the reflex to verify the specifics that matter, so reliably that a confident wrong dose never gets past them, not because they caught the lie in the text, but because they were always going to check that kind of answer anyway.

One caution as you build the habit: do not let it curdle into the opposite error of refusing the tool entirely because it sometimes fabricates. That overcorrection throws away real value, the general explanations, the first drafts, the plain-language rewrites, that the model produces reliably and that you can verify cheaply. The goal is neither trust nor distrust but a precise, question-shaped allocation of skepticism, spending your verification effort on the specific and consequential while using the tool freely for the general and low-stakes. A clinician who verifies a dose but happily uses an AI-drafted patient handout after reading it has struck exactly the right balance, capturing the efficiency where it is safe and standing guard where it is not. That balance, and not any fear of the technology, is what a mature relationship with these tools looks like.

Key Takeaways

  • A clinical hallucination is a confident, plausible, false output produced by the same process as every correct answer, so the tool cannot tell you which it just gave you. That judgment is yours.
  • Medicine runs on specifics, and specifics are exactly what a model fabricates most, because a precise answer requires a precise fact it may not have and it will supply a specific-looking substitute rather than hedge.
  • Clinical hallucinations cluster into recognizable forms: the fabricated fact (wrong dose or interaction), the phantom citation, the invented patient detail in documentation, and confident over-reach beyond what is known or accessible.
  • Hallucination rises with specificity, rarity, questions about data the model cannot see, and leading prompts, which is the inverse of what a busy clinician wants to lean on the tool for.
  • The model is most trustworthy on common, general, well-trodden questions and least trustworthy on the specific, rare, patient-individual ones that matter most.
  • You cannot catch a hallucination by how it looks; the defense is a rule applied before reading, based on the question, for how much verification the answer requires.
  • The five structural defenses: ground the model in real sources, verify specifics against a trusted reference, prompt neutrally, calibrate skepticism by question type, and never read confidence as truth.
  • None of these defenses requires understanding the technology. They require being a clinician who checks the specifics that would change a decision.