By Level 3, you are no longer the writer who verifies a paragraph by hand. You are the person who designs the machine that verifies thousands of paragraphs and signs off on whether that machine is good enough to put in front of an FDA Office of New Drugs reviewer. A hallucination detection protocol is that machine: a documented, repeatable, validated pipeline that takes AI-generated regulatory content and the source corpus it was supposed to be grounded in, and returns a defensible determination of whether every factual claim, every number, and every cross-reference traces to truth. This lesson builds that pipeline as five named stages run in sequence over a Module 2.5 Clinical Overview and its supporting Module 2.7 summaries: citation existence checks, fact extraction-and-verification, contradiction detection against the source Clinical Study Reports, TLF round-tripping, and cross-Module consistency. Each stage catches a failure class the previous one cannot, and the protocol is only defensible when all five run, log their results, and gate the document from advancing until every flag is resolved. The goal is not to catch hallucinations by reading more carefully. The goal is to make catching them a process with an audit trail, because a process is what survives an inspection and a careful read is not.

Why a Protocol and Not a Careful Read

A careful read does not scale, does not document itself, and does not survive personnel change, which makes it the wrong control for regulated AI output. The Module 2.5 lead who reconciles every claim by hand on a Tuesday is a single point of failure, and when that person is on leave during submission lock, the verification quality silently drops to whatever the backup happens to do. A protocol fixes the verification logic in a written specification, so the same five checks run the same way on every document regardless of who operates the pipeline, and each run leaves a record that an inspector can read. This is the difference between a control that depends on a person and a control that is engineered into the workflow, and only the latter satisfies the FDA-EMA Guiding Principles' expectation of governance and documentation around AI use.

There is a second reason the protocol beats the read, and it is about failure-class coverage. A human reading a fluent draft is good at catching prose that sounds wrong and terrible at catching a plausible hazard ratio that is simply the wrong number, because the wrong number reads exactly as confidently as the right one. The protocol decomposes the verification into five orthogonal checks, each targeting a failure class that a single linear read tends to miss, and the union of the five catches dramatically more than any reader can. The mechanical fact from Level 1, that the model completes patterns rather than retrieving truth, is the entire justification for this structure: because the model can invent a citation, a number, a contradiction, a table reference, and a cross-Module inconsistency independently, you need five independent detectors, not one attentive set of eyes. The protocol is the operational answer to the architecture.

Stage One: Citation Existence Checks

The first stage answers the cheapest and highest-yield question in the entire pipeline: does every cited target actually exist? A citation existence check parses the AI-generated document for every reference token, the TLF table numbers like Table 14.2.1.4, the CSR section pointers like CSR Section 11.4.2.1, the Module cross-references like Module 2.7.3.2, and the literature citations, and it confirms that each named target is present in the controlled source inventory. This is a set-membership test, not a content test: the stage does not yet ask whether Table 14.2.1.4 contains the result the sentence claims, only whether a table with that number exists in the final TLF package. Run first because it is fast, deterministic, and eliminates the single most dangerous failure mode before any expensive semantic checking begins.

The implementation discipline matters more than the concept. You build the existence check against the controlled inventory, which means a frozen, version-stamped list of every real table, figure, listing, CSR section, and Module reference in the actual submission, not against the model's belief about what exists. The check produces three outcomes per citation: exists, does not exist, or unresolvable because the inventory is incomplete, and the third outcome is treated as a hard stop, not a pass, because an unverifiable citation is wrong until proven right. A citation to Table 14.2.1.4 that does not appear in the frozen TLF inventory is flagged as a fabricated cross-reference, the exact failure that surfaces as an Office of New Drugs Information Request on Day 74 if it escapes. This stage alone, run rigorously, would have caught the running example from the Level 1 lesson before it ever propagated into Module 2.7.3.

Stage Two: Fact Extraction-and-Verification

The second stage moves from structure to content by decomposing each sentence into atomic factual claims and verifying each one against the source. Fact extraction takes a sentence like "In the intent-to-treat population, the hazard ratio for progression-free survival was 0.68 (95% CI 0.54 to 0.86, p=0.001)" and decomposes it into a structured set of assertions: the analysis population is ITT, the endpoint is PFS, the point estimate is 0.68, the confidence interval bounds are 0.54 and 0.86, and the p-value is 0.001. Each extracted fact carries its claimed source locator, and the verification step retrieves the actual value from that source and compares. The decomposition is the load-bearing move, because a single fluent sentence hides four or five independently falsifiable claims, and verifying the sentence as a whole lets a wrong number ride along with the correct ones.

The verification comparison must be exact and typed, not approximate. A hazard ratio of 0.68 against a source value of 0.71 is a mismatch even though both are plausible, and the protocol does not get to round or shrug, because a transcription error and a generated value are indistinguishable on the page and both are wrong in the submission. The discipline is to verify the value, the analysis set, the direction, and the units as separate fields, because the most dangerous failures are not wrong numbers but right numbers attached to the wrong population or the wrong arm. A correctly transcribed median of 18.2 months that the model silently assigned to the control arm when the source assigns it to the treatment arm passes a naive number-match and fails a typed comparison, and the typed comparison is the one that protects the benefit-risk story. Every fact that cannot be matched to a located source value is flagged and routed to human reconciliation before the document advances.

Stage Three: Contradiction Detection Against Source CSRs

The third stage catches a failure the first two cannot: a claim that is internally well-formed, cites a real target, and reports a number that appears nowhere as an exact match, but contradicts what the source CSR actually says. Contradiction detection runs each AI-generated claim against the relevant CSR text and the Statistical Analysis Plan and asks not "does this value match" but "does the source assert something incompatible with this." The model can write "the treatment effect was consistent across all prespecified subgroups" when the CSR forest plot shows a clear interaction in the elderly subgroup, and that sentence cites a real table, reports no specific number to mismatch, and is simply false against the source. Existence and fact-matching both pass it; only contradiction detection catches it.

This stage is where retrieval and semantic comparison earn their keep, because contradiction is a relationship between meanings, not a string difference. The protocol retrieves the source passages most relevant to each generated claim and evaluates entailment: does the source support, contradict, or stay silent on the claim. A claim the source contradicts is a hard flag; a claim the source is silent on is also a flag, because an unsupported claim in a Clinical Overview is a claim the model added without grounding, and the named author cannot sign what the source does not support. The subtle, high-value cases live here: a directional overstatement ("strongly favored" when the source says "numerically favored"), a quietly dropped caveat (omitting that the result was nominal and not multiplicity-controlled), and a generalization the data does not license. These are the hallucinations that pass spell-check, pass a number audit, and lose the submission its credibility when a reviewer reads the source.

Stage Four: TLF Round-Tripping

The fourth stage closes the loop between prose and tables with a bidirectional check called TLF round-tripping, and it is the most regulatory-specific control in the protocol. Forward round-tripping takes each numeric claim in the narrative and traces it to the exact cell in the exact TLF table that should contain it, confirming the prose value equals the table value. Reverse round-tripping takes the key results in the TLF and confirms that the narrative reports them faithfully and does not silently omit a material result the reviewer will expect to see summarized. The round trip is what makes the check complete: forward catches invented and mis-transcribed values, and reverse catches the omission of an inconvenient result, which is a different and equally serious integrity problem in a benefit-risk document.

Round-tripping depends on a clean linkage between narrative claims and table cells, which is why this stage is most powerful when the AI workflow that produced the draft was built to emit source-cell links in the first place, the structured-output discipline taught in the previous lesson. When the draft carries a machine-readable map from each sentence to a TLF coordinate, round-tripping becomes a deterministic cell-by-cell comparison rather than a fuzzy search, and the audit log records the comparison for every claim. The stage also enforces the rule that a claim may only round-trip to a table that survived stage one's existence check, which prevents the protocol from cheerfully verifying a number against a table the model invented. The output of this stage is the single most reviewer-legible artifact the protocol produces: a claim-to-cell reconciliation table that an Office of New Drugs reviewer would recognize as exactly the reconciliation they perform by hand on Day 74.

Stage Five: Cross-Module Consistency

The fifth stage steps up a level to ask whether the documents agree with each other: does Module 2.5 say what Module 2.7 says, and do both agree with the CSR. Cross-Module consistency detection compares the same result as it is stated in every place it appears, the integrated efficacy claim in the Module 2.5.4 Clinical Overview, the detailed result in the Module 2.7.3 efficacy summary, and the underlying CSR Section 11, and flags any divergence in value, population, direction, or emphasis. This matters because the dossier is read as a coherent whole, and a reviewer who finds the Clinical Overview claiming a hazard ratio of 0.68 while the 2.7.3 reports 0.71 has found either a transcription error or a fabrication, and cannot tell which without an Information Request. AI workflows make this failure more likely, not less, because the same model generating both summaries from the same loaded sources reproduces the same invention consistently in some runs and inconsistently in others.

The check is structural and must be built around a canonical result registry, a single controlled list of the key results with their authoritative values, populations, and source cells, against which every Module's statement is compared. Building the registry first turns cross-Module consistency from an all-pairs comparison nightmare into a star comparison against one source of truth, and the registry becomes a reusable asset that also feeds the round-tripping stage. The highest-value catches here are the quiet ones: a subgroup result emphasized in 2.7.3 but dropped from the 2.5 integrated view, a safety signal characterized as "manageable" in one Module and "requires monitoring" in another, and a numerical rounding inconsistency that, while individually trivial, signals to a reviewer that the documents were not reconciled. Consistency across Modules is not cosmetic; it is the visible evidence that the submission was assembled with discipline, and its absence invites exactly the line-by-line scrutiny every sponsor wants to avoid.

Severity Tiering and the Residual Risk You Accept

Not every flag carries the same weight, and a protocol that treats a rounding discrepancy in a demographic table the same as a fabricated primary-endpoint hazard ratio will drown its operators in noise and lose the deadline argument the first time it is run under pressure. Severity tiering assigns each flag a criticality level based on what the claim does in the submission: a flag on a primary or key secondary efficacy result, a safety signal, or a benefit-risk statement is critical and blocks the gate unconditionally, while a flag on a non-decision-driving descriptive statistic is minor and may be batched for correction without halting the document. The tiering must be defined in the protocol specification ahead of time, not negotiated per flag at lock, because a severity scheme invented under deadline pressure is indistinguishable from rationalizing a defect through. The discipline is to write down, before the first run, which claim types can never ship with an open flag, and to make that list match exactly the claims a reviewer reads first.

Tiering also forces an honest conversation about residual risk, which is the risk that survives the protocol because no detector is perfect. The protocol catches what its five stages are built to catch, and a sophisticated failure, a number that is internally consistent across all Modules because the same wrong value was propagated everywhere from a single mis-loaded source, can pass all five stages and still be wrong. The mature protocol names this residual risk explicitly in its risk register, identifies the compensating control (independent human reconciliation of a sampled subset of critical claims against the raw data, not against the AI-consistent corpus), and records the accepted level. An inspector does not expect a protocol that catches everything, because no such protocol exists; an inspector expects a protocol that knows what it catches, names what it does not, and has a documented compensating control for the gap. Claiming zero residual risk is itself a credibility failure, and the honest risk register is the stronger position.

Wiring the Five Stages Into a Gated Pipeline

A protocol is not five checks run ad hoc; it is five checks wired into a gated pipeline where the document cannot advance until every flag is dispositioned. The stages run in a deliberate order: existence first because it is cheap and eliminates the worst failures, then fact verification, then contradiction, then round-tripping, then cross-Module consistency, with each stage's flags accumulating in a single reconciliation log keyed to the document version under change control. A flag has exactly three legal dispositions: corrected, justified with a recorded rationale, or escalated, and there is no fourth disposition called "accepted because the deadline is close." The gate is the control that gives the protocol teeth, because an ungated checklist is advisory and a gated pipeline is a barrier, and only a barrier holds when the PDUFA clock is running and the pressure to ship is maximal.

The pipeline must also be validated, which is the Level 3 obligation that separates a real protocol from a clever script. Following the IQ/OQ/PQ mindset, you establish that the pipeline is installed and configured correctly, that each stage operates as specified against known-good and known-bad test documents, and that it performs to acceptance criteria on representative real content, including a seeded set of deliberately planted hallucinations the protocol must catch. The validation evidence, the test documents, the expected and actual results, and the acceptance determination, becomes part of the audit trail that makes the protocol's outputs trustworthy to an inspector. A hallucination detection protocol that has never been challenged with a known fabrication has not been validated; it has merely been run, and the difference is exactly what an Establishment Inspection Report will probe. The protocol you can defend is the one whose own correctness you can demonstrate.

Key Takeaways

• A hallucination detection protocol is a validated pipeline, not a careful read. It fixes the verification logic in a written specification so the same five checks run identically on every document and leave an inspectable record, which is what governance and documentation under the FDA-EMA principles require and what a careful read can never provide.
• The five stages target orthogonal failure classes and must all run. Citation existence checks catch fabricated targets, fact extraction-and-verification catches wrong and mis-attributed values, contradiction detection catches claims the source refutes or is silent on, TLF round-tripping catches invented values and omitted results, and cross-Module consistency catches divergence between 2.5, 2.7, and the CSR.
• Existence checks run first against a frozen controlled inventory, and an unverifiable citation is a hard stop. A citation to a table that does not appear in the version-stamped TLF inventory is a fabricated cross-reference, the exact failure that becomes a Day 74 Information Request if it propagates into Module 2.7.3.
• Typed, exact verification beats number-matching because the worst failures are right numbers in the wrong place. Verify value, analysis set, direction, and units as separate fields, and build round-tripping bidirectionally so the protocol catches both invented values and the quiet omission of an inconvenient result.
• The protocol is only defensible when it is gated and validated. Every flag must be corrected, justified, or escalated before the document advances, and the pipeline itself must be challenged with seeded hallucinations under an IQ/OQ/PQ mindset, because a protocol that has never caught a planted fabrication has been run, not validated.