A sepsis prediction model has been live in shadow mode at your health system for six weeks. The headline metric on the vendor's slide is clean: aggregate sensitivity of 0.83, aggregate specificity of 0.81, calibrated on a national dataset. Your governance committee is minutes from voting to move it into silent alerting. Then your equity officer asks a single question that nobody on the vendor call can answer: does that 0.83 hold for the patients who arrive through your emergency department speaking Spanish, for your Medicaid population, for the older adults on your medical-surgical floors? The room goes quiet. The truth is that nobody has looked, because the aggregate number never made them look. This lesson is about the discipline of making yourself look, on purpose, at the subgroups you actually serve, before and after you turn the model on.

Why Equity Is a Safety Issue, Not a Values Statement

It is tempting to file bias and equity under corporate values, somewhere near the mission statement. That framing is a mistake, and it is a dangerous one. Disparate performance is a patient-safety failure with a body count, a legal exposure, and an accreditation consequence. When an AI tool works well for the average patient but poorly for a subgroup, the patients in that subgroup receive worse care specifically because you deployed the tool. You did not inherit that harm. You introduced it.

The mechanism is not mysterious. A model learns the patterns present in its training data. When the training population does not resemble the population you serve, or when it underrepresents a subgroup, the model has seen fewer examples of how disease presents in those patients and it performs worse for them. The cruelty of this pattern is that it compounds existing inequity. The patients most likely to be underrepresented in training data are frequently the patients who are already underserved: racial and ethnic minorities, people with limited English proficiency, patients on public insurance, rural patients, people with disabilities. A model that underperforms for them widens a gap that was already there.

Regulators and accreditors have caught up to this. The Coalition for Health AI (CHAI) and the Joint Commission framework for responsible health AI use include, among their seven elements, an explicit requirement to evaluate for risk and bias both before and after deployment. This is not a suggestion to be equitable in spirit. It is an operational requirement to produce evidence that you tested for disparate performance, that you set thresholds, and that you kept watching. If an auditor asks how you tested for bias and your answer is that the vendor validated the model nationally, you do not have an answer. You have a gap.

An aggregate accuracy number is where a subgroup's harm goes to hide. If you have not disaggregated, you have not measured.

Two Load-Bearing Examples of How Bias Enters

Two documented cases should live permanently in the mind of anyone who governs clinical AI. They are not cautionary hypotheticals. They are the real thing, and they teach two different entry points for bias: the label, and the measurement.

The cost-as-need algorithm: bias in the label

A widely used population-health algorithm, deployed across tens of millions of patients, was designed to identify patients who would benefit from high-risk care management programs. To rank patients by need, it predicted future healthcare cost, on the reasonable-sounding logic that sicker patients cost more. That single design choice, cost as a proxy for need, encoded a structural inequity. Because the health system had historically spent less money on Black patients at any given level of illness, due to reduced access, mistrust, and other barriers, those patients generated lower predicted costs. The algorithm read lower cost as lower need. It therefore scored Black patients as healthier than equally sick white patients and under-referred them to the very programs meant to help them. Obermeyer and colleagues documented this in Science in 2019. Correcting the target to a direct measure of illness rather than cost dramatically increased the share of Black patients flagged for extra care.

The lesson is not that the engineers were malicious. The lesson is that the bias lived in the label, the thing the model was trained to predict. The model was accurate at predicting cost. Cost was simply the wrong stand-in for need, and it was wrong in a way that tracked race. This is called proxy bias or label bias: when the outcome the model predicts is a biased substitute for the outcome you actually care about. No amount of accuracy on the wrong target rescues you. You have to interrogate the target itself.

Pulse oximetry: bias in the measurement

Pulse oximeters estimate blood oxygen saturation (SpO2) optically, and that optical measurement is affected by skin pigmentation. Multiple studies have shown that these devices systematically overestimate true oxygen saturation in patients with darker skin. The clinical consequence is occult hypoxemia: the patient is genuinely hypoxemic, their tissues are oxygen-starved, but the device reads a reassuring number and the true hypoxemia goes undetected. Patients with darker skin were found to have occult hypoxemia at meaningfully higher rates, which in turn affected eligibility for oxygen therapy and other interventions gated on SpO2 thresholds.

Now follow the contamination downstream. Any algorithm or AI tool that ingests SpO2 as an input, and many do, inherits that measurement bias. The model is not doing anything wrong statistically. It is faithfully learning from a signal that is itself biased against a subgroup. This is measurement bias, and it is insidious because the model can look perfectly calibrated on the recorded data while being systematically wrong about the underlying reality for the patients the sensor mismeasures. A biased input silently becomes a biased output.

Hold these two together. Bias can enter through the target you predict (cost for need) and through the inputs you feed the model (a sensor that mismeasures by skin tone). Both are invisible to an aggregate accuracy number. Both require you to look at subgroups deliberately.

Define the Subgroups That Matter for Your Population

Equity testing begins before any statistics, with a definition step that is easy to shortchange. You cannot test for disparate performance across subgroups you have not named. The subgroups that matter are the ones present in your patient population and plausibly at risk of differential performance. A useful starting list, which you should tailor rather than copy:

• Race and ethnicity. The most studied axis of disparate performance, and the one where training-data underrepresentation is best documented. Use your own recorded categories, and note where they are incomplete or self-reported.
• Language and limited English proficiency (LEP). Patients with limited English proficiency, meaning they do not speak English well enough to manage their care in it, frequently have thinner, more fragmented documentation, which starves text-based models of signal. Why you care: sparse notes can degrade any model that reads the chart.
• Sex and gender. Disease presentation, physiology, and historical trial representation all differ, and models can encode those gaps.
• Age. Pediatric and geriatric extremes are frequently underrepresented, and a model tuned on middle-aged adults may miscalibrate at the edges.
• Insurance and payer. A structural proxy for access and for the cost-based label problem above. Medicaid, Medicare, uninsured, and commercial populations can experience the model differently.
• Disability status. Patients with disabilities may have atypical baselines that a model reads as noise.
• Rural versus urban residence. Access patterns, care fragmentation, and documentation density differ, and this often intersects with the axes above.

Two disciplines matter here. First, use intersectional thinking where your sample allows it: an older, Spanish-speaking, Medicaid patient may sit at the confluence of three separate weaknesses. Second, be honest about your data quality. Race and ethnicity fields are frequently missing, misclassified, or aggregated in ways that hide small groups. You cannot test what you cannot see, so documenting the completeness of your subgroup variables is itself part of the work.

The Disaggregated Evaluation: How You Actually Test

The core technical move of equity testing is simple to state and demanding to execute: compute your performance metrics separately for each subgroup, not just overall. This is a disaggregated evaluation. The overall number is the average, and the average is exactly where a subgroup's harm hides.

The metrics, and why each one can betray a subgroup differently

You need a panel, not a single number, because a model can be equitable on one metric and inequitable on another. Define each on first use so the whole committee reads the same table:

• Sensitivity (true-positive rate): of the patients who truly have the condition, what fraction does the model catch. Why you care: low sensitivity in a subgroup means missed cases, the most direct route to harm.
• Specificity (true-negative rate): of the patients who truly do not have the condition, what fraction the model correctly clears. Why you care: low specificity means false alarms, alert fatigue, and unnecessary workups concentrated in a group.
• Positive predictive value (PPV): when the model fires, how often it is right. This depends on prevalence, so it can differ across subgroups even when the model is behaving identically.
• Negative predictive value (NPV): when the model stays silent, how often the patient is truly clear.
• False-negative rate and false-positive rate: the complements you should report explicitly, because a false-negative rate that is twice as high in one subgroup is a sentence about who gets missed.
• Calibration: when the model says thirty percent risk, do thirty percent of those patients actually have the event, and does that hold within each subgroup. A model can be well calibrated overall and badly miscalibrated for a subgroup, systematically over- or under-stating their risk.

Report every one of these per subgroup, side by side, in a single table. The disparity you are hunting is a gap between rows.

Interrogating the label and the inputs

Numbers alone are not enough. Two structural questions run alongside the metrics. First, the label question, straight from the cost-as-need case: is the outcome the model predicts actually the thing you care about, or a biased stand-in? Ask the vendor what the training target was, in plain words, and ask whether that target could correlate with access, spending, or any variable that tracks a protected group. Second, the input question, straight from the pulse-oximetry case: does any model input carry a known measurement bias, such as SpO2 by skin tone, that would contaminate the output for a subgroup?

What you require from the vendor

A CHAI element you can operationalize immediately: require the vendor to disclose the training population and any known disparate performance. Concretely, ask for the demographic composition of the training and validation data, the subgroup-level performance the vendor has already measured, and a plain statement of known limitations. A vendor who cannot or will not describe who the model was trained on has told you something important. Vendor disclosure does not replace your own local testing, because their population is not yours, but it is the floor.

Sample size in small subgroups

A real trap: your smallest and most vulnerable subgroups are exactly where estimates are noisiest. A sensitivity of 0.60 computed on eleven patients is not a finding, it is a coin flip with a decimal point. Before you act on a subgroup number, check whether you have enough cases to trust it, report confidence intervals, and where the sample is too thin, say so explicitly and pool data over a longer window rather than pretending precision you do not have. Underpowered does not mean safe. It means you do not yet know, which is its own reason to keep watching.

Equity Thresholds, a Response Plan, and a Worked Example

Testing without a pre-committed decision rule invites rationalization after the fact. Before go-live, your governance committee sets equity thresholds: how much subgroup disparity is acceptable, and what happens when it is exceeded. A common form is a bounded ratio, for example that the false-negative rate in any subgroup with adequate sample size must not exceed the best-performing subgroup by more than a set relative margin. The exact number is a clinical and ethical judgment for your committee. Having a number, decided before you see the results, is the non-negotiable part. Pair it with a response plan naming the options in advance: do not deploy, deploy with a mitigation, deploy for some subgroups and not others, or deploy conditionally with heightened monitoring.

The worked example: catching disparate performance in shadow mode

Return to the sepsis model in the opening scene. You are the governance and equity owner. The tool has run in shadow mode, generating predictions on live patients without those predictions reaching clinicians, for six weeks. You now run the disaggregated evaluation you committed to.

The aggregate looks fine: sensitivity 0.83, specificity 0.81, well calibrated. Then you break it down by primary language and payer, and a row jumps out. For patients with limited English proficiency, sensitivity falls to 0.61 while specificity holds. Translated into human terms: the model misses nearly two in five true sepsis cases in your LEP patients, versus roughly one in six overall. You check the sample: 47 true sepsis cases in the LEP subgroup over six weeks, wide but not trivial confidence interval, enough to take seriously and not enough to be certain. You look for a mechanism and find a plausible one. Your LEP patients have sparser free-text documentation in the early hours of admission, and the model leans on note-derived features. The signal it depends on is thinner for exactly these patients.

Your pre-set equity threshold said no subgroup's false-negative rate may exceed the best subgroup by more than a defined relative margin. This breach clears that bar. Now the response plan, decided in advance, does its job. You do not silently launch, because that would knowingly deliver worse sepsis detection to an already-underserved group. You do not simply reject the tool either, because it performs well for most patients and sepsis detection has real value. You issue a conditional response: hold the general go-live, keep the model in shadow mode, formally document the disparity and its likely mechanism, return to the vendor with the specific subgroup finding and a request for their remediation plan, and require a re-test on a larger LEP sample before any deployment that includes those patients. You also add a compensating control, flagging LEP admissions for the standard clinical sepsis screen independent of the model, so patient safety does not wait on the vendor. Every step is written down, dated, and reported to the committee. That documentation is precisely what you hand an auditor who asks how you tested for bias.

Notice what made this possible. Not goodwill. A named subgroup, a disaggregated metric, a sample-size check, a pre-committed threshold, and a response plan that existed before the number did.

After Deployment: Bias Is Not a One-Time Test

The most common failure after a careful pre-deployment evaluation is to treat equity as certified and move on. It is not certified, because the conditions that produced your results do not hold still. Your patient population changes. Documentation practices change. Upstream data sources and even the model itself may change. A model that was equitable at launch can drift into disparity as the world underneath it shifts, and drift does not announce itself.

So the disaggregated evaluation becomes a recurring instrument. Re-run the subgroup metrics on a defined cadence, watch the same rows you watched before go-live, and set alerts on the equity thresholds so a widening gap surfaces as a signal rather than as an incident. Small subgroups that were underpowered at launch may accumulate enough cases over months to finally support a trustworthy estimate, so revisit the questions you had to defer. Ongoing equity monitoring connects directly to your model-drift program, which the next lesson takes up in full.

Documentation is the deliverable

Every element here produces a record: the subgroups you defined and why, the vendor disclosures you obtained, the disaggregated results with sample sizes and confidence intervals, the thresholds you set, the decision you made, the compensating controls, and the monitoring cadence. This documentation is not paperwork after the real work. It is the artifact that proves the real work happened. It is what satisfies the CHAI and Joint Commission expectation of before-and-after bias evaluation, what protects your organization when a disparity surfaces, and what lets a successor understand what you decided and why. If it is not written down, for governance purposes it did not happen.

The throughline of this entire discipline is one refusal: refuse to assume the absence of disparate performance. It is not absent because the vendor is reputable, not absent because the aggregate number is high, and not absent because you would prefer it to be. Disparate performance is a claim about your patients, and claims about patients are tested, not assumed. You look, on purpose, at the subgroups you serve, before you turn the model on and for as long as it stays on.

Key Takeaways

• Disparate performance is a patient-safety, legal, and accreditation issue, not a values statement. The CHAI and Joint Commission seven elements require evaluating for risk and bias both before and after deployment, and an aggregate accuracy number is where a subgroup's harm hides.
• Bias enters through the target and through the inputs. The cost-as-need algorithm (Obermeyer et al., 2019) shows label bias: predicting cost as a proxy for need under-referred Black patients because less had historically been spent on them at the same level of illness.
• Pulse oximetry shows measurement bias: devices overestimate oxygen saturation in darker skin, producing occult hypoxemia that then contaminates any AI tool built on SpO2. A biased input silently becomes a biased output.
• Name the subgroups that matter for your population before you compute anything: race and ethnicity, language and limited English proficiency, sex and gender, age, insurance and payer, disability, and rural versus urban, with intersectional combinations where your sample allows.
• Run a disaggregated evaluation: compute sensitivity, specificity, PPV, NPV, false-negative and false-positive rates, and calibration separately per subgroup, and interrogate whether the label is a biased proxy and whether any input carries measurement bias.
• Require the vendor to disclose the training population and any known disparate performance, and treat that disclosure as a floor, never a substitute for testing on your own patients.
• Check sample size in small subgroups, report confidence intervals, and never act on a number computed on too few cases. Underpowered means you do not yet know, which is its own reason to keep watching.
• Set equity thresholds and a response plan before go-live, keep re-running the disaggregated evaluation after deployment because bias can emerge with drift, and document every step, because for governance purposes what is not written down did not happen.