Six months in, the readmission-prediction pilot looked like a triumph. Adoption was high, care managers loved the interface, the tool shaved minutes off their morning huddle, and the steering committee was ready to scale it across all fourteen sites. Then a quality analyst asked the question no one had built the pilot to answer: did it actually work, and did it work for everyone? Nobody knew. The pilot had measured how many people used the tool and how fast it ran. It had never measured whether the predictions were right, or whether they were right for the Black patients and the non-English-speaking patients who made up a third of the panel. A pilot that measures adoption and speed but not safety and equity is not a pilot. It is a very expensive way to talk yourself into deploying something you never tested. This lesson is about designing the other kind.

The Pilot That Is Actually a Trap

Most failed clinical AI deployments do not fail at the demo or the contract. They fail at the pilot, and specifically at a pilot that was designed to succeed rather than to test. The trap is seductive because the metrics that are easiest to collect, adoption, user satisfaction, time saved, are also the metrics a vendor most wants you to celebrate and the ones a busy team is most eager to see. Those metrics are not worthless; a tool no one will use is a failed tool regardless of its accuracy. But they answer a different and lesser question. They tell you whether people like the tool. They tell you nothing about whether the tool is right, and nothing at all about whether it is right for the patients most likely to be harmed when it is wrong.

A pilot that only measures adoption or speed is a trap for a precise reason: it manufactures confidence without evidence. At the end of it you have a room full of enthusiastic users, a slide showing minutes saved, and an implicit claim, "it works," that the pilot never actually tested. You then scale that untested claim across your whole system, at which point the accuracy and equity questions you skipped do not disappear. They simply get answered later, in production, by your patients, in the form of missed deteriorations, false alarms that erode trust, and disparate harm to the groups your model was quietly worse at. The pilot did not de-risk the deployment. It laundered the risk into something that felt safe.

The reframe is simple and demanding. A pilot exists to try to prove that a tool is both accurate and fair on your specific population, under conditions that could genuinely fail. If it cannot fail, it cannot inform you. The whole design discipline of a good pilot is the discipline of building something that would honestly tell you no.

It helps to name the two pilots side by side, because the contrast is the whole lesson in miniature. The same tool, run two ways, produces two records: one that feels like proof and is not, and one that is uncomfortable and is.

Define Success and Stopping Before You Start

The single most important decision in a pilot is made before it begins: what would count as success, and what would make you stop. Written down, in advance, in numbers. This sounds obvious and is almost never done, because defining success in advance is uncomfortable. It forecloses the option, deeply human and deeply dangerous, of looking at whatever the pilot produced and constructing a story in which it succeeded. A pilot without pre-specified success criteria will almost always be judged a success, because the people who ran it want it to be, and the data can usually be arranged to agree.

So specify it up front. Success criteria should state the minimum performance the tool must hit on your patients, in clinical terms: a sensitivity floor, a PPV floor appropriate to your prevalence, a maximum acceptable false-alarm rate, and, critically, those thresholds met not just in aggregate but in each subgroup you care about. Equally important, and almost always omitted, are the stopping criteria: the conditions under which you halt the pilot regardless of enthusiasm. A safety signal, a subgroup where performance falls below the floor, an alert burden that is driving alarm fatigue, a workflow failure that is causing near-misses. Stopping criteria are what separate a pilot from a slow-motion uncontrolled rollout. Without them, a pilot has no brakes, and a pilot with no brakes is just deployment with a friendlier name.

There is a discipline to setting these numbers that is worth naming, because a floor pulled from the air is nearly as dangerous as no floor at all. The thresholds should be anchored to the clinical decision the tool informs and to the harm of being wrong in each direction. For a deterioration model whose false negative means a missed crash, the sensitivity floor should be set high, because the cost of a miss is a code that could have been prevented. For a care-gap list whose false positive merely adds a chart to review, a lower PPV may be tolerable, but for one that triggers an outreach call or a medication change, the false-positive cost climbs and the PPV floor must climb with it. The point is that success is not a single generic number; it is a set of thresholds reasoned from what the tool does and what it costs to be wrong. A pilot whose success criteria were chosen by asking "what will the tool probably hit" rather than "what must it hit to be safe" has quietly let the tool grade its own exam.

The most useful form for these numbers is a table that ties each metric to a floor, an action if the floor is breached, and a named decision owner. This is the stop/go criteria table, and it is the single artifact that most reliably converts good intentions into a real gate. Notice that "who decides" is a column, not an afterthought: a threshold with no owner is a suggestion, and a suggestion does not stop a rollout. The specific numbers below are illustrative and must be reasoned locally from your prevalence and the harm of being wrong; treat any floor you inherit from a vendor or a paper as a number to verify against your own decision, not a number to adopt on faith.

Duration and sample size matter just as much, and are just as often fudged. A pilot too short or too small to produce a stable estimate of subgroup performance has not really tested the subgroup; it has produced a number with confidence intervals so wide they include both success and disaster. When your most important subgroup is a modest fraction of your patients, an honest pilot has to run long enough, or enroll deliberately enough, to say something real about that group. Otherwise the equity gate becomes theater: a subgroup row on the dashboard with a number underneath it that no one should actually trust. Decide before you start how much data you need to make a defensible call for each group, and treat an underpowered subgroup estimate as a reason to keep going, not a reason to declare victory.

Sample-Size and Power Discipline

Here is the mechanic that separates a serious equity gate from a decorative one. Suppose a subgroup is 8 percent of your panel and your pilot enrolls a few hundred patients. The number of subgroup events, the actual deteriorations or actual readmissions in that group, may be a handful. A sensitivity estimated from a handful of events swings wildly: a single missed case can drop your point estimate by ten or fifteen points, and the confidence interval can easily span from clearly unsafe to apparently excellent. An estimate like that cannot clear a floor and cannot fail one either; it simply is not informative. The rigorous response is not to average the subgroup into the majority, and not to wave it through on the aggregate. It is to keep going: extend enrollment, run a targeted sub-study, or oversample the group until the estimate is tight enough to make a defensible call. An underpowered subgroup number is a reason to continue the pilot, never a reason to end it in the tool's favor. The discipline is to decide, before you start, how many subgroup events you need to be able to say something true, and to treat falling short of that as an unmet stopping condition rather than a rounding error.

If you did not write down what would make you stop, you did not run a pilot. You ran the first phase of a deployment you had already decided to do.

The Safety Gate: Prove It Works Here

The first of the two gates a pilot must clear is safety, which in this context means accuracy on your patients under real conditions. A tool validated elsewhere has, at best, earned the right to be tested here; it has not earned deployment. The safety gate asks the tool to demonstrate, prospectively and locally, that its predictions are accurate enough on your population to be trusted in the workflow you intend.

Concretely, that means measuring the tool's real performance during the pilot against a ground truth you can establish: for a deterioration model, did the patients it flagged actually deteriorate, and did it miss patients who did? It means watching the PPV your clinicians actually experience, because that determines whether the tool earns trust or trains people to ignore it. It means tracking near-misses and any safety events where the tool contributed, whether by a wrong output that was acted on or a missed one that should have been caught. And it means measuring not just the model in isolation but the human-plus-model system, because a tool that is accurate but confusing, or accurate but ignored, or accurate but over-trusted, fails in practice even when it succeeds on paper. The safety gate is cleared only when the tool has shown it performs on your patients, in your workflow, well enough that deploying it reduces risk rather than adding it.

It is worth laying the two gates out as parallel checklists, because in practice teams pass one and quietly skip the other, then call the pilot done. The safety gate and the equity gate ask different questions of the same pilot, and a tool must clear both to scale. Neither substitutes for the other: a tool can be accurate on average and unfair to a subgroup, or fair across subgroups but not accurate enough to be worth deploying at all.

The human-plus-model point deserves a moment, because it is where technically sound pilots most often go wrong. The thing you are deploying is never just the model; it is the model plus a clinician plus a workflow plus a moment of time pressure. A model with a respectable sensitivity can still produce net harm if its interface buries the important flag among ten trivial ones, if it fires so often that staff learn to click past it, or if it is so authoritative that clinicians stop applying their own judgment and inherit the model's errors. Conversely, a middling model wrapped in a workflow that presents its output as one checkable input among several, at the right moment, to a clinician who still decides, can be genuinely safe. So the safety gate must observe the whole system in action: does the flag reach the right person at the right time, do they verify before acting, does the tool change behavior in the direction of better care, and does it do so without breeding the automation bias that earlier lessons warned turns a model error into a patient harm. A pilot that measures only the model's numbers and never watches what clinicians actually do with them has tested half of the thing it is supposed to test.

The Equity Gate: Prove It Works for Everyone

The second gate is the one most pilots skip entirely, and it is the one most likely to produce a harm you cannot defend. A model trained on a non-representative population underperforms for the patients already underserved, and the terrible feature of that underperformance is that it is invisible in aggregate numbers. A tool can have excellent overall accuracy and a sensitivity that quietly collapses for one racial group, or for women, or for non-English-speaking patients, and every aggregate metric on your dashboard will look fine while the tool systematically fails a subpopulation.

The equity gate closes that blind spot by requiring that performance be measured, and meet the floor, in each subgroup that matters for your population, defined by race, ethnicity, sex, age, language, and payer, and any others your patient mix makes salient. This is not a compliance checkbox; it is a clinical safety requirement, because a disparity in a risk model is not an abstraction. It is a concrete group of patients whose deterioration is caught less often, or whose care-gap list is systematically wrong, precisely because they were underrepresented in the data the model learned from. The CHAI and Joint Commission guidance names bias evaluation before and after deployment as a foundational element for exactly this reason. A pilot that cannot report subgroup performance has not tested for the harm most likely to fall on the patients your institution most needs to protect, and it cannot clear the equity gate, because it has no data with which to clear it.

A Subgroup Table That Passes Aggregate and Fails a Group

The mechanic is easiest to see in a worked subgroup table. Below is the kind of readout the equity gate demands: not a single aggregate number, but a row per subgroup with its size, its measured sensitivity and PPV, and an explicit yes or no on whether it meets the pre-set floors (sensitivity at least 0.70, PPV at least 0.40 in this illustration). Read the aggregate row first, then read down. The tool looks like a clear pass in aggregate and is unacceptable for the Spanish-preferred group, whose sensitivity sits well below the floor. The numbers here are illustrative and exist to show the shape of the reasoning, not to be repeated as fact.

Two things are true at once in that table, and holding both is the discipline. The aggregate is genuinely good, and the tool genuinely fails a group that is a quarter of the panel. The aggregate does not lie; it simply averages the strong majority result over the weaker one, and the weaker one disappears into the mean. If the pilot had reported only the top row, the disparity would have shipped invisibly and been discovered later by the Spanish-preferred patients whose deteriorations were caught less often. Notice too that the low-event subgroups deserve a second look for power: if the Spanish-preferred group had shown 0.58 on only six events rather than dozens, the honest read would be that the estimate is not yet trustworthy and the pilot must continue, rather than either passing or failing the group on noise.

Framed plainly, the equity gate treats "it works well overall" as an incomplete sentence, and insists on finishing it: it works well overall, and here is the proof that it also works for each group of patients we serve. There is a hard corollary. Sometimes the equity gate will tell you no. The tool will perform beautifully overall and fail a subgroup that is a meaningful share of your patients, and the honest response is to withhold or restrict deployment until that is fixed, even when the aggregate story is a winner and the champions are impatient. That is the entire point of the gate. It exists to catch precisely the case where the tool is good on average and unacceptable for the people you are obligated not to fail, and to make that catch before those patients are the ones who discover it.

A Worked Example: The Same Tool, Two Pilots

Two hospital systems pilot the identical readmission-risk tool. System A runs the trap. It measures adoption (85 percent of care managers used it daily), satisfaction (high), and time saved (twelve minutes per huddle). After ninety days it declares victory and scales to all sites. Eight months later, a quality review finds the tool's PPV in production is 22 percent, so most flagged patients were not actually high risk, and worse, its sensitivity for the system's large Spanish-speaking population is dramatically lower than for English speakers, meaning the patients hardest to reach were the ones the tool most often failed to flag. The disparity had been present the whole time. The pilot simply never looked.

System B runs the real pilot. Before starting, it writes success criteria: sensitivity at least 0.80 and PPV at least 0.40 overall and within each major subgroup, false-alarm burden below a defined threshold, and stopping criteria including any subgroup falling below a 0.70 sensitivity floor. It runs the tool prospectively on a defined cohort, establishes ground truth by tracking actual readmissions, and measures performance by subgroup weekly. At week six, the equity gate trips: sensitivity for Spanish-speaking patients is 0.58, well below the floor. System B does not scale. It pauses, works with the vendor to understand the gap, requires a mitigation and a re-test, and only proceeds for that subgroup once the floor is met. System B spent more and moved slower, and System B is the one that did not deploy a tool that systematically failed its most vulnerable patients. The difference was not the tool. The tool was identical. The difference was that one pilot was built to discover the truth and the other was built to confirm a decision already made.

Making the Gates Real in Your Governance

Gates only work if they have teeth, which means the authority to stop a rollout has to sit with someone who is not the person whose success is measured by the rollout happening. Build the safety and equity gates into your governance so that clearing them is a formal, documented precondition for scaling, decided by a body, your AI governance committee or its equivalent, that includes quality, safety, and equity voices and is empowered to say no. The pre-specified success and stopping criteria become the committee's checklist. The subgroup performance data becomes a required deliverable, not an optional appendix. And the decision to scale, or not, is written down with the evidence it rested on.

Concretely, a gate with teeth has four moving parts, and it is worth being explicit about each because a gate that is missing any one of them fails quietly. First, the criteria are written and signed before enrollment, so no one can renegotiate the bar after seeing the data. Second, the go or no-go decision belongs to a body that does not report to the champion and is not measured by the rollout shipping, so the person most invested in scaling is not the person who gets to declare the pilot a success. Third, the subgroup performance table is a required deliverable, listed in the pilot charter, so its absence blocks the decision rather than being excused as an appendix that ran out of time. Fourth, the decision and the evidence it rested on are documented and retained, so that a Joint Commission surveyor, a plaintiff, or your own future quality review can see exactly what you knew and when. Strip out any one of those and the gate reverts to a suggestion: unsigned criteria get renegotiated, a conflicted decider rationalizes, missing subgroup data gets waved through, and an undocumented decision cannot be defended.

This maps directly onto the seven foundational elements the Joint Commission and CHAI named in their 2025 responsible-use guidance: a designated governance structure, patient safety and quality, and bias evaluation before and after deployment are precisely the muscles the gated pilot exercises. Treat that guidance as a floor to verify against your own practice, not a ceiling to admire from a distance.

This does two things. It protects patients, by ensuring that no tool scales across your system without proving it is accurate and fair on the people it will touch. And it protects the institution, because when a surveyor or a plaintiff asks how you knew the tool was safe and equitable to deploy at scale, you can produce a pilot with pre-specified criteria, prospective local performance data, and documented subgroup results, rather than a satisfaction survey and a slide about minutes saved. The gated pilot is slower and more expensive than the trap, and that cost is the price of actually knowing. A pilot that only measures adoption and speed is cheaper right up until the day it is not, and that day arrives in production, paid for by the patients the tool was quietly worst at.

Key Takeaways

• A pilot that measures adoption, satisfaction, and speed but not accuracy and fairness is a trap: it manufactures confidence without evidence and launders risk into production, where your patients pay for the questions the pilot skipped.
• A pilot exists to try to prove a tool is accurate and fair on your specific population under conditions that could genuinely fail; if it cannot fail, it cannot inform you.
• Define success and stopping criteria in numbers, in advance: sensitivity and PPV floors appropriate to your prevalence, a maximum false-alarm burden, and the explicit conditions under which you halt regardless of enthusiasm. A pilot with no stopping criteria is deployment with a friendlier name.
• The safety gate requires the tool to prove, prospectively and locally, that it performs accurately enough on your patients in your workflow, measured against real ground truth, including the human-plus-model system, not the model in isolation.
• The equity gate requires performance to meet the floor in each subgroup that matters (race, ethnicity, sex, age, language, payer), because aggregate accuracy hides a sensitivity that can quietly collapse for an underserved group.
• Sometimes the equity gate says no: a tool can be excellent overall and unacceptable for a meaningful subgroup, and the honest response is to withhold or restrict deployment until it is fixed, which is the entire purpose of the gate.
• The CHAI and Joint Commission guidance names bias evaluation before and after deployment as a foundational element; a pilot that cannot report subgroup performance has not tested for the harm most likely to fall on your most vulnerable patients.
• Gates only work with teeth: the authority to stop a rollout must sit with a governance body that includes quality, safety, and equity voices and is empowered to say no, with the criteria, subgroup data, and scaling decision documented as the defensible record.