Why Accuracy Alone Fails for Modern AI Systems

When a retail bank benchmarks a new underwriting model and proudly reports 92 percent accuracy on its hold-out set, accuracy by itself tells you almost nothing about whether the system is safe to deploy. The 8 percent error rate may concentrate entirely in one demographic subgroup; the model may be well-calibrated at the 0.5 threshold and catastrophically miscalibrated at the 0.9 threshold used in production; it may hallucinate an approval rationale that looks plausible but cites a nonexistent income history. Modern AI evaluation has to answer a much richer question: across which subpopulations, under which distribution shifts, at which operating thresholds, against which adversaries, and measured by which construct, is this system good enough to ship?

The failure mode of single-number accuracy became infamous with the Gender Shades study published by Joy Buolamwini and Timnit Gebru in 2018, which showed that commercial face analysis systems from IBM, Microsoft, and Face++ had overall accuracy above 90 percent but accuracy as low as 65 percent on darker-skinned female faces. Since then the field has accumulated a long list of similar lessons: Amazon's abandoned 2018 resume screening model that down-ranked applications containing the word women's; the Dutch toeslagenaffaire childcare-benefits scandal (2013 to 2019) where an opaque fraud-detection model led to more than 26,000 families being wrongly accused; the 2016 COMPAS recidivism investigation by ProPublica; and the 2023 retraction of a Stanford lung-cancer CT screening model that collapsed when radiology scanners were changed. In every case the headline metric was fine. The system was not.

A comprehensive evaluation framework, as taught in this chapter, treats evaluation as a socio-technical program spanning pre-deployment benchmarks, post-deployment monitoring, adversarial probing, human oversight, and governance artifacts. The framework you build must be reproducible, version-controlled, tied to specific risk tiers, auditable by a regulator, and cheap enough that engineers will actually run it on every pull request. Anything less, and evaluation becomes a ritual performed before launch and then quietly abandoned while the model drifts. The remainder of this chapter walks through the standards, tools, metric families, and organizational practices that separate real evaluation programs from theater.

A useful mental model is the four-layer evaluation stack: construct validity at the bottom (are we measuring the right thing?), technical validity above it (is the measurement statistically sound?), operational validity (does the eval reflect production conditions?), and governance validity at the top (can we prove to a regulator and an affected user that we did this?). Most teams invest heavily in the middle two layers and neglect the bookends. That is where failures hide.

Construct validity failures are the most embarrassing because they mean the metric you optimized was not measuring the thing you cared about. The Epic sepsis prediction model (studied by Wong et al., JAMA Internal Medicine, 2021) is the canonical example: vendor-reported AUC near 0.76, independent external validation showed 0.63 on true sepsis onset rather than sepsis billing codes; the training label was an artifact of billing, not clinical reality. The second canonical example is the 2020 pandemic-era surge of COVID-19 CT screening papers where internal AUCs above 0.95 collapsed to random on external data because the models had learned hospital-specific markers in image metadata. Construct validity is not a metric; it is an argument tied to how labels were produced, how the deployment population differs from the training population, and what downstream decision the prediction drives.

Operational validity is the twin that most engineering teams skip. A question-answering system that scores 0.82 on SQuAD in a notebook may score 0.49 in production because production users type two-word fragments instead of full questions, because the retrieval layer returns chunks the QA model never saw during training, because users ask questions in languages underrepresented in the eval set, and because latency budgets force a quantized model at inference time. A comprehensive framework therefore requires shadow deployments where the new model sees real production traffic in parallel with the incumbent and its outputs are scored retrospectively. Shadow results are almost always worse than offline results, and the gap (sometimes called the offline-online delta) is itself a metric worth tracking over time.

NIST AI RMF, ISO/IEC 42001, and ISO/IEC 24028 in Practice

The NIST AI Risk Management Framework (AI RMF 1.0, released January 2023, with the Generative AI Profile added July 2024) organizes risk work into four functions: Govern, Map, Measure, and Manage. For evaluation purposes, the Measure function is the heart of the framework. Measure contains four categories and nineteen subcategories. MEASURE 2 focuses on trustworthy-AI characteristics: accuracy, reliability, robustness, safety, security, privacy, fairness, explainability, and accountability. MEASURE 2.3 in particular requires that systems are evaluated for performance and trustworthiness using measurements that are consistent across development and production. MEASURE 3 and 4 address feedback loops and tracking effectiveness over time.

A concrete way to operationalize MEASURE 2.3 is to construct an evaluation matrix with characteristics as rows (for example, fairness, robustness, calibration, faithfulness) and lifecycle stages as columns (unit tests, integration tests, pre-release sign-off, shadow production, canary, full production, post-incident review). Each cell names a specific test, a data slice, an acceptable threshold, and the owner. For an LLM assistant, a fairness cell in the pre-release row might point to a BBQ bias benchmark run against the current model snapshot, with a threshold that accuracy disparity across demographic contrasts must not exceed 5 points, and the owner is the responsible AI lead. When a regulator asks how NIST AI RMF MEASURE 2.3 is satisfied, you hand them the matrix and the signed artifacts.

ISO/IEC 42001:2023 is the first certifiable AI Management System standard, analogous to ISO 27001 for information security. It requires an organization to define AI objectives, assess risks and impacts (Annex B and the linked ISO/IEC 23894), implement controls (Annex A, with 38 controls grouped into nine sets), and run internal audits and management reviews. Critically, control A.6.2.4 requires evaluation of AI system performance throughout the lifecycle, and A.6.2.6 requires operational monitoring. Certification bodies such as BSI, DNV, and TUV Rheinland began issuing 42001 certificates in 2024; AWS announced its AI services reached 42001 conformance in early 2025, and SAP followed shortly after. For practitioners, the value of 42001 is that it forces a written statement of applicability linking each control to auditable evidence.

ISO/IEC TR 24028:2020 on trustworthiness of AI is older and not certifiable, but it catalogs threats and mitigations in a way engineers find useful. It breaks trustworthiness into transparency, explainability, controllability, robustness, resilience, reliability, availability, accuracy, privacy, and security, and it enumerates specific engineering mitigations for each. When you are designing test cases, 24028 is a better starting checklist than the RMF because it is more concrete at the technique level. A typical stack in a mature organization looks like this: NIST AI RMF as the overarching governance language, ISO/IEC 42001 as the certifiable management system, ISO/IEC 24028 as the engineering threat catalog, ISO/IEC 23894 for the underlying risk process, and domain-specific rules on top (EU AI Act Annex III conformity assessments, FDA Good Machine Learning Practice, SR 11-7 for US bank model risk).

Experiment Tracking: MLflow, Weights & Biases, and the Discipline of Reproducibility

An evaluation result that cannot be reproduced is a rumor. The first engineering problem of a comprehensive evaluation program is therefore not which metric to compute but how to bind every metric to the exact code, data, model weights, prompt, seed, and hardware that produced it. Two tools dominate this space in 2026. MLflow is open source, originated at Databricks in 2018, and is now a Linux Foundation project; it is the default choice when your stack is mostly Python and Spark and when you want total control over storage. Weights & Biases, usually written W&B or wandb, is a commercial SaaS (with a self-hosted tier) that pairs richer interactive dashboards with a strong artifact and sweep system.

In MLflow, an evaluation run is an MLflow Run that logs parameters (mlflow.log_param for things like temperature, top_p, seed), metrics (mlflow.log_metric for scalars such as exact_match, BLEU, expected calibration error), artifacts (mlflow.log_artifact for the full predictions CSV, confusion matrix PNGs, and the eval dataset hash), and a model signature. The MLflow Model Registry then promotes a specific Run's model through stages (None, Staging, Production, Archived) with required approval; a well-run organization ties the Staging-to-Production transition to an eval gate that checks the Run has logged a fairness report, a calibration report, and a red-team summary. MLflow 2.x added mlflow.evaluate() with built-in evaluators for classifier, regressor, and question-answering tasks, and MLflow 3.0 (2025) added GenAI-focused evaluators with LLM-as-judge metrics.

Weights & Biases offers a similar Runs and Artifacts model, but its differentiator is Sweeps for hyperparameter search and Reports for shareable markdown/interactive analyses. W&B Launch lets you queue eval jobs to Kubernetes or SageMaker, and W&B Weave, released in 2024, is specifically designed for LLM evaluation traces where each call, tool invocation, and scorer becomes a node in a tree you can inspect. In practice, teams pick based on deployment constraints: regulated healthcare and finance shops that cannot send data to SaaS choose MLflow self-hosted or Vertex AI Experiments; research teams and startups that want the UI lean W&B.

Reproducibility requires more than tracking. It requires pinning randomness (seeding PyTorch, NumPy, CUDA, and Python hash), freezing tokenizer and tokenization settings (subtle changes in Hugging Face tokenizers between versions have swung benchmark scores by one or two points), pinning the inference library (vLLM 0.5 and 0.6 produce different outputs for the same prompt because of kernel changes), and recording the exact checkpoint hash (SHA-256 of the safetensors file). A useful discipline is to publish an Eval Card alongside a model card: the Eval Card lists dataset version, metric implementation library version, prompt template SHA, judge-model name and version, date, and a link to the Run. Teams at Anthropic, OpenAI, and Google DeepMind all publish eval cards or equivalents in their system cards; enterprise teams should copy the pattern.

Hold-out Test Sets, Contamination, and Slice-Based Evaluation

The central statistical tool of evaluation is the hold-out test set: a sample of data that the model has never seen during training or tuning, drawn from the same distribution as production. Everything else is a variation on this idea. For classical ML, a clean 60/20/20 train/validation/test split with stratification across the label and important demographic features works. For foundation models, hold-out is much harder because we cannot verify what was in the pretraining corpus. A 2024 study by Sainz et al., Data Contamination Quiz, found that Llama 2 70B could reproduce the opening paragraphs of about 12 percent of the MMLU test items verbatim, which strongly implies memorization. The MMLU, HellaSwag, GSM8K, and HumanEval benchmarks are now partially contaminated for every frontier model.

The practical response is a portfolio of evaluation datasets: public benchmarks (for comparability, with contamination disclaimers), private held-out versions maintained by the organization (never shared with any vendor), time-shifted evaluations that use only items created after the model's training cutoff (LiveCodeBench, introduced by Jain et al. in 2024, rotates its test set monthly for exactly this reason), and task-specific golden sets curated by domain experts. The Center for AI Safety's WMDP benchmark for dangerous-capability eval and the FrontierMath benchmark for quantitative reasoning are examples of benchmarks that attempt to keep items secret from crawlers.

Single aggregate numbers on a hold-out set hide the failure modes you most need to see. The discipline of slice-based evaluation, popularized by the Snorkel team and widely adopted since, requires enumerating meaningful slices of your data and reporting per-slice metrics. A mortgage underwriting model is sliced by applicant gender, race, age bracket, geographic zip code, loan amount bucket, first-time-buyer status, and application channel. A clinical NLP model is sliced by patient age, language, site of care, time of day, and note type. If the aggregate F1 is 0.88 but the F1 on pediatric discharge notes in Spanish is 0.54, the aggregate is lying to you. Open-source tools to operationalize this include TensorFlow Model Analysis, Fairlearn's MetricFrame, Aequitas from the University of Chicago, and Giskard for LLM slices.

Finally, hold-out hygiene requires versioning data like code. The dataset should have a semver version (3.2.1), a hash over the sorted record IDs, a documented provenance, and a labeling-quality report. DVC, LakeFS, and Hugging Face Datasets with the datasets library's built-in fingerprinting are the common tools. When a regulator asks you how you know a recent performance drop is not a measurement artifact, the answer is that you re-ran v3.2.1 of the eval set against both the old and new model and confirmed the drop reproduces. Without version pinning, you cannot make that statement.

Adversarial Evaluation and Structured Red-Teaming Protocols

Adversarial evaluation asks a different question than benchmark evaluation: not how well does the model perform on typical inputs, but how badly can it fail under inputs designed to break it. For classical models this includes perturbation tests (FGSM and PGD attacks on vision models, HotFlip and TextAttack on NLP), robustness to covariate shift (WILDS, DomainBed), and membership-inference attacks on privacy. For LLMs, adversarial evaluation focuses on jailbreaks, prompt injection, toxic output under coaxing, capability elicitation for dangerous tasks, and data-exfiltration via tool use. The distinction between an adversarial eval and a red-team exercise is roughly scale and structure: adversarial evals are automated, run continuously, and produce metrics; red-teams are human-driven, time-boxed, and produce narratives plus a taxonomy of findings.

The best public protocol for LLM red-teaming as of 2026 is the one published by Anthropic in its Responsible Scaling Policy and refined through the UK AI Safety Institute's Inspect framework and the US AI Safety Institute's TRAINS network. A structured red-team has five phases: threat modeling (enumerate what could go wrong and who benefits), target selection (choose specific capabilities or harms to probe), attack design (build a bank of attempts, both manual and automated via tools like PAIR, GCG, and Cisco's robust-intelligence-style fuzzers), execution with observation (run attacks against the model with full logging), and remediation tracking (each finding gets a severity rating such as the one in NIST's SP 800-30, an owner, and a fix SLA).

Several named benchmarks codify parts of this: HarmBench from the Center for AI Safety (2024) with 510 harmful behaviors across seven categories; AdvBench from Zou et al. for automated suffix attacks; MITRE ATLAS for adversarial threat modeling; CyberSecEval 3 from Meta for code-agent cyber risks; and the UK AISI's Inspect-Evals repository. For specific risks, WMDP evaluates proxy questions for biosecurity, cybersecurity, and chemical weapons uplift; BOLD and RealToxicityPrompts probe toxic continuation; TruthfulQA probes sycophancy and plausible falsehoods.

A cautionary detail: attack success rate (ASR) on a benchmark is a floor, not a ceiling. A model with 0 percent ASR on HarmBench may still have a 12 percent ASR in a live product because the product introduces tool use, file uploads, or long-context memory that HarmBench does not cover. The protocol should therefore include deployment-specific red-teaming: run the same attack catalog through the actual product surface, including voice modes, mobile apps, and any agentic tool loop. A 2024 postmortem of a major AI customer-service deployment found that its pre-launch red-team was against the raw model API; the jailbreaks that reached customers used the voice channel, which the red-team had not tested.

Calibration, Uncertainty Quantification, and Selective Prediction

A classifier that outputs 0.9 for a positive prediction is well-calibrated if, across all predictions with probability 0.9, roughly 90 percent are actually positive. Calibration is cheap to measure and expensive to ignore. Two standard metrics are Expected Calibration Error (ECE), introduced by Guo et al. in 2017, and Brier score. ECE bins predictions by confidence and averages the absolute gap between confidence and empirical accuracy within each bin. Modern deep networks are systematically overconfident: a ResNet on CIFAR-100 often has an accuracy of 70 percent but an average confidence of 87 percent. Temperature scaling, a one-parameter logistic fit on a validation set, typically reduces ECE by 5 to 10x at zero accuracy cost; it is the first intervention to try and belongs in every evaluation report.

For LLMs, calibration is harder because the probability is over tokens, not over correctness. Two widely used proxies are verbalized confidence (the model is asked to output a numeric confidence) and sampled agreement across n temperatures (self-consistency). Research from OpenAI (Let's Verify Step by Step, 2023) and Google DeepMind (2024) showed that verbalized confidence in GPT-4-class models is modestly calibrated after fine-tuning but sharply overconfident out of the box. Techniques like P(IK) probes (Kadavath et al., 2022) and constrained decoding improve calibration on short-form QA.

Calibration feeds directly into selective prediction: the system abstains when confidence is below a threshold. Selective prediction is often the fastest path to shipping a model that would otherwise be too risky. An underwriting model with 88 percent accuracy that only acts on the 60 percent of loans where its calibrated confidence exceeds 0.85 can yield 97 percent accuracy on the portion it covers, with a human reviewer handling the rest. Evaluating a selective system requires a coverage-risk curve and area-under-risk-coverage (AURC); SelectiveNet (Geifman and El-Yaniv, 2019) formalized the training side.

In regulated settings, calibration is increasingly required. The EU AI Act's Article 15 on accuracy, robustness, and cybersecurity for high-risk systems has been interpreted by ENISA and the AI Office as implying calibrated confidence outputs where feasible. The FDA's 2024 Total Product Lifecycle guidance for AI-enabled device software explicitly calls out calibration drift as a Predetermined Change Control Plan trigger. Calibration has moved from statistical nicety to regulatory expectation.

Human Evaluation, LLM-as-Judge, and Faithfulness versus Correctness

Automated metrics are cheap and reproducible but narrow; human evaluation is rich and expensive. The right evaluation program combines both and understands their failure modes. For LLM tasks, the dominant automated approach in 2024 to 2026 is LLM-as-judge: a second model scores the first model's outputs against a rubric. Papers like Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023), and systems like G-Eval (Liu et al., 2023), Prometheus (Kim et al., 2024), and OpenAI's Evals framework, popularized the pattern. Judge models are fast enough for CI but have well-documented biases: position bias (first option preferred), verbosity bias (longer answers preferred), self-preference (Claude judges rate Claude outputs higher), and calibration drift across versions.

Mitigations include randomizing option order, running pairwise with both orderings and taking the majority, using a stronger judge than the system under test (GPT-5 or Claude Opus 4.7 judging a smaller model), prompting the judge to explain before scoring (chain-of-thought judgment reduces position bias by roughly 40 percent per Zheng et al.), and periodically validating judge scores against human gold labels. A healthy rule of thumb is that no LLM-as-judge metric should be deployed to production gating without an initial human-judge correlation study on at least 200 items, targeting Kendall tau above 0.6.

Human evaluation itself has subspecialties. Expert evaluation (for example, board-certified radiologists reading model outputs) is the gold standard for clinical and legal applications but costs on the order of 50 to 300 dollars per item. Crowd evaluation via Scale AI, Surge, Toloka, or Prolific is cheaper (1 to 5 dollars per item) but needs rigorous guidelines, qualification tests, inter-rater reliability thresholds (Krippendorff alpha or Fleiss kappa above 0.6), and adversarial QA. Head-to-head preference evaluation, as used by LMSYS Chatbot Arena since 2023, produces Bradley-Terry or Elo rankings from pairwise preferences; the Arena leaderboard crossed 2 million votes in 2025 and has been influential despite known issues such as style-over-substance preference.

A subtle but critical distinction is faithfulness versus correctness, especially for retrieval-augmented generation and summarization. A response is correct if it matches ground truth; it is faithful if it is supported by the provided sources, regardless of whether the sources themselves are correct. RAGAS (Es et al., 2023) and TruLens operationalize both: faithfulness decomposes the answer into atomic claims and checks each against retrieved passages, while answer relevance checks alignment with the question. A model that is 95 percent correct but 70 percent faithful is drawing on parametric memory rather than the retrieved corpus, which means source updates will not flow through. Many enterprise RAG failures are faithfulness failures disguised as correctness wins.

Building the Evaluation Program: Governance, Cases, and Tradeoffs

Putting it all together, a comprehensive evaluation program has four durable components. First, a written Evaluation Policy, approved by the AI governance committee, that maps risk tiers to evaluation depth. Microsoft's Responsible AI Standard (v2, 2022) and Google's AI Principles review process are public exemplars. A tiered policy might specify that Tier 1 (general-purpose assistants, low-stakes content) requires benchmark suite plus red-team; Tier 2 (productivity inside regulated workflows) adds fairness and calibration reports; Tier 3 (credit, employment, clinical) adds prospective shadow deployment with a human-labeled outcome study.

Second, an Evaluation Platform: the CI integration, the dashboards, the alerting. LangSmith, Arize Phoenix, Evidently AI, and Fiddler occupy this space for LLMs; Arize, Fiddler, Aporia, and WhyLabs cover classical and tabular ML; internal platforms at Booking.com (Experimentation Platform), Netflix (Metaflow plus internal eval), and Stripe (Capture) show the pattern of treating eval as infrastructure. A mid-sized team should expect a 4 to 8 FTE investment to build and run this.

Third, a Catalog of Evaluations: concrete test suites with owners. The catalog approach avoids the trap of reinventing tests for each model. A mature catalog at a bank might include 40 tabular-model tests (KS drift, PSI, AUC by protected attribute, calibration slope), 25 NLP tests (prompt-injection corpus, jailbreak corpus, policy classifier agreement), 15 RAG tests (faithfulness, context relevance, answer relevance, latency tail), and 10 agent tests (tool-use safety, loop detection, cost bounds). Each entry has a unique ID, a linked notebook, and an expected runtime.

Fourth, Incident Response and Learning. When an eval finding or live incident surfaces, the program captures it with root-cause analysis and feeds it back as a regression test. The template is borrowed from SRE: blameless postmortem, 5 whys, action items with owners and dates, and a permanent regression case.

Real cases ground the framework. The 2023 Air Canada chatbot case (Moffatt v. Air Canada), where a tribunal held the airline liable for a chatbot's false bereavement-refund policy, is an evaluation failure: the chatbot had no faithfulness gate against policy documents. The 2024 Mata v. Avianca case, where a New York lawyer cited six fabricated cases produced by ChatGPT, was a calibration and faithfulness failure amplified by no human oversight. iTutor Group's 365,000-dollar settlement with the EEOC in 2023 for age discrimination in AI-assisted hiring illustrates what happens when fairness slice evaluation is absent. Conversely, the UK AISI's 2024 pre-deployment evaluation of Anthropic, OpenAI, and Google DeepMind models, published in its May 2024 update, is a public example of rigorous adversarial evaluation done well.

The fundamental tradeoffs you will manage: (1) speed versus rigor, where every additional gate adds hours or days to release but prevents incidents; (2) automated versus human, where LLM-as-judge lets you run 10,000 items a day for a dollar while humans give you ground truth at 100x the cost; (3) breadth versus depth, where covering more slices diffuses attention and covering fewer misses failure modes; (4) public benchmarks versus private sets, where public ones give comparability but leak to training data. There is no single right answer. The discipline is to make the tradeoffs visible, version them, and revisit them on a schedule. That is what separates an evaluation program from an evaluation checkbox.