There is a moment in every eval-set conversation where the question gets specific: "okay, but what do we actually measure?" The novice answer is "we'll have an LLM judge each output." It is wrong — not entirely wrong, but expensively wrong, wasteful of the most signal-rich layers of evaluation, and almost guaranteed to produce a fragile system. The right answer is four layers, each catching a different class of failure, each cheaper or more abundant than the one above it. Deterministic checks run on 100% of outputs and cost essentially zero. Heuristic scores catch the next layer of quality issues for fractions of a cent. LLM-as-judge handles nuance — tone, helpfulness, hallucination — at roughly one-tenth the cost of human review for 90% of the signal. Human calibration runs on a stratified sample and is the truth check that keeps the whole stack honest. Stack the four layers in this order and you have an eval system that scales. Skip any one and you have a system that fails on some predictable class of input. This lesson is the recipe.

Why One Layer Is Never Enough

The single-layer eval is the most common failure mode in agent teams. Usually it is "use GPT-5 to score every output." It looks pragmatic. It catches some things. It misses entire classes of failures.

One example. A finance agent that summarizes customer accounts. The team set up a GPT-5 judge that scored "is this summary good?" on every output. The eval scored 92%. The team shipped. Two weeks later they discovered that 8% of the summaries were returning malformed JSON that broke the downstream dashboard — a problem a deterministic schema check would have caught on day one. Another 4% were missing key fields that the user needed but the judge did not realize were missing. The "92% good" eval had been hiding a 12% breakage rate behind a vague quality dimension.

The opposite failure mode: a team that built five different deterministic checks, no LLM judge, and an eval that scored 98% on every run. They shipped with confidence. Stakeholders started complaining within a week that the answers were technically correct but condescending in tone. The deterministic checks caught the schema and the field presence. They missed the entire dimension of how the output read to a human. No judge layer meant no quality signal beyond mechanical correctness.

One-layer evals fail in the dimensions they do not measure. Two-layer evals are better. Four-layer evals are the production standard in 2026. Each layer catches a different class of failure, costs a different amount of money, and produces a different signal. Stack them. The cost is small. The robustness is large.

The four layers, in cost order

Order matters. Run the cheap layers first, the expensive layers last. Most failures are caught by the cheap layers — the LLM-judge and human calibration only run on outputs that pass the earlier filters. This is the architectural shape that makes the four-layer pattern affordable.

1. Layer 1 — Deterministic checks. Schema, format, length, parse-ability. Cost: essentially zero. Coverage: 100% of outputs. Catches: hard breakage.
2. Layer 2 — Heuristic scoring. Key-field presence, regex matches, structured-content rules. Cost: fractions of a cent per output. Coverage: 100% of outputs that pass layer 1. Catches: missing required content.
3. Layer 3 — LLM-as-judge. Tone, helpfulness, hallucination, factuality, the soft dimensions. Cost: 10x cheaper than human review. Coverage: 100% of outputs that pass layers 1 and 2 (or a sample, if cost is a concern). Catches: quality nuance.
4. Layer 4 — Human calibration. Stratified sample of 5-10% of outputs, reviewed by domain experts. Cost: the most expensive per output but the smallest sample. Coverage: 5-10% stratified. Catches: anything the other three missed; calibrates the judge prompt.

Layer One: Deterministic Checks — The Free Pass

Deterministic checks are the binary pass/fail rules that have no judgment. They either pass or they don't. They run on every output. They cost nothing. They catch the mechanical failures that destroy production pipelines.

The checks that belong in layer one

• Schema validation. Does the output match the expected JSON schema? Use Pydantic, Zod, or JSON Schema validators. If the agent is supposed to return {"answer": str, "citations": list, "confidence": float}, the schema check verifies every output has those fields with those types.
• Format validation. Markdown well-formed? Email addresses syntactically valid? Dates parseable? Phone numbers in the right country format? Use libraries — email-validator, dateutil, phonenumbers.
• Length checks. Output between expected min and max? Empty outputs caught? Single-character outputs caught? Outputs exceeding the downstream rendering limit caught?
• Parse-ability checks. If the output is supposed to be valid Markdown for downstream rendering, does it parse? If it is SQL, does the SQL parser accept it? If it is a structured tool call, does it match the tool's input schema?
• Reference integrity. If the output cites chunk IDs, do all cited chunks exist in the retrieved set? If it references customer IDs, do they exist in the database (for non-PII checks)?
• Profanity / toxicity gates. Cheap toxicity classifiers (Perspective API, Detoxify, OpenAI's moderation endpoint) run as a binary gate. Cost: pennies per thousand calls.

Why layer one is the most under-invested in

Teams skip deterministic checks because they feel beneath the dignity of an LLM-quality eval. "We're measuring intelligence; why are we checking JSON parse?" Because 30% of agent failures in production are mechanical — malformed output, missing fields, invalid format. The LLM judge does not notice these (it judges "is the answer good?" and a malformed answer can still look "good"). The deterministic check catches them in milliseconds.

The Anthropic and OpenAI 2026 model APIs both ship with structured output enforcement at decoding time, which reduces malformed JSON to a rounding error. But "reduces" is not "eliminates." Layer one is still required.

What layer one does not catch

Layer one only catches mechanical failure. A schema-valid output that is factually wrong passes layer one. A well-formed email that says "Dear Sir/Madam" to a customer named Lisa passes layer one. The deterministic layer cannot judge content quality. That is the job of the upper layers.

Layer Two: Heuristic Scoring — Cheap Content Checks

Heuristics are rule-based scoring that judges content, not just format. They are still cheap — fractions of a cent per output — and they catch a class of failure that deterministic checks miss but that LLM-judges over-spend on.

The heuristics that earn their keep

• Key-field presence. For a golden case "What was Q3 revenue?", the answer must contain the actual Q3 revenue number ($48.3M). A simple substring check. If the number is missing, the answer is wrong even if it sounds reasonable.
• Required-element checks. A customer-service email must have a greeting, a body, a closing, and a signature. Four substring checks. If any is missing, fail.
• Forbidden-content checks. The agent must never mention competitors by name. Regex against a list of competitor names. Hard fail.
• Citation requirement. The agent must include at least one citation in every factual answer. Regex for citation markers ([1], [Source: ...]). If zero citations on a factual claim, fail.
• Numeric-range checks. If the agent estimates an ROI, the number must be in the range [0.5, 5.0] — anything outside is suspect. Parse and bounds-check.
• Embedding-similarity floor. The output's embedding must be within some cosine distance of the expected output's embedding. Catches drastically off-topic answers cheaply. Use only as a floor, not a verdict.

The heuristic that surprises every team

The most useful heuristic is also the simplest: output length sanity. Agents that are stuck in retry loops, agents that hallucinate long stories when they should be brief, agents that produce one-character outputs because of decoding errors — all caught by a min/max length check. A finance summary agent that should output 100-300 words is producing 1500 words? Something is wrong; the LLM judge will not catch it, the length check will.

What heuristics cannot do

Heuristics cannot judge nuance. They cannot tell a polite refusal from a rude refusal. They cannot tell a well-written email from a stilted one. They cannot tell a hallucinated fact from a real fact. That is layer three.

Layer Three: LLM-as-Judge — 10x Cheaper Than Human

The LLM judge is a separate LLM call that takes the agent's output and scores it on quality dimensions a human would otherwise score. It is the layer that handles tone, helpfulness, factuality, hallucination, adherence to brand voice — the soft dimensions that rules cannot capture.

The cost math that makes layer three essential

In 2026, an LLM-judge call on Claude Haiku, GPT-5 Mini, or Gemini Flash costs roughly $0.0005 to $0.002 per evaluation, depending on context size. A human reviewer costs $20-$50 per hour, evaluating maybe 30-60 outputs per hour depending on complexity. That is $0.30-$1.50 per output for human review versus $0.001 for LLM-judge — a 100-1500x cost ratio.

The signal ratio is the other side of the math. Calibrated LLM-judges, after the work covered in lesson 3, achieve 80-90% agreement with human reviewers on the dimensions they measure. That means roughly 90% of the signal at 1% of the cost. The 10% gap is where layer four (human calibration) lives.

LLM-as-judge is the workhorse of layer three: 10x cheaper than human review at 90% of the signal. Use a smaller model (Haiku, Mini, Flash) — large models are not meaningfully better at judging and cost 10x more. The cost ratio is what makes the four-layer pattern affordable at scale.

The judge dimensions that matter most

• Factuality. Does the answer match the ground truth (when ground truth exists) or contradict it? Critical for RAG agents.
• Faithfulness. Does the answer stay within the retrieved sources, or does it introduce facts not in the retrieved chunks? The hallucination check.
• Helpfulness. Does the answer address the user's actual question, or does it dodge?
• Conciseness. Is the answer the right length for the question? Not padding, not too terse.
• Tone-match. Does the answer match the brand voice or the persona the agent should embody?
• Safety. Does the answer respect safety boundaries (no PII leak, no unsafe advice, no policy violations)?
• Refusal appropriateness. When the agent refuses, is the refusal appropriate, or is it over-refusing (refusing things it should answer)?

The judge does not need to score all seven on every output. Pick three or four that matter most for your agent. Run them in parallel (one LLM call per dimension is cleaner than one prompt asking for all dimensions — the model focuses better).

The judge prompt template

A solid judge prompt has six components:

1. Role and context. "You are evaluating the output of a customer-support agent for [company]. The agent's job is [job]."
2. The dimension being judged. "You are scoring this output on [factuality / helpfulness / etc.]."
3. Explicit criteria. "Factuality is the degree to which the answer's claims are supported by the provided sources. A score of 5 means every claim is fully supported. A score of 1 means most claims are unsupported or contradicted." (Five-point scale is more reliable than 0-1 binary or 0-100 continuous.)
4. The input. The user's query.
5. The output. The agent's answer.
6. The references (when applicable). The ground truth, the retrieved sources, the expected output. The judge needs evidence to ground its scoring.

Output format: JSON with {"score": 1-5, "reasoning": "..."}. The reasoning field is critical — it lets you spot-check the judge and detect when the judge is wrong. The score alone is opaque; the score plus the reasoning is auditable.

The failure modes of LLM-judges (full coverage in lesson 3)

• Self-consistency. The same judge run twice on the same output gives different scores. Mitigation: run 3 times and average, or use a higher-capability judge for the calibration set.
• Length bias. Judges tend to score longer outputs higher. Mitigation: explicitly tell the judge to ignore length.
• Position bias (for comparison judges). When judging A vs B, judges prefer whichever was presented first. Mitigation: randomize position, or use single-output scoring rather than comparison.
• Same-model self-agreement. The judge model and the agent model are the same — the judge agrees with the agent's mistakes because they share biases. Mitigation: use a different model family as judge.

Layer Four: Human Calibration — The Truth Check

The fourth layer is human review of a stratified sample. It is the smallest sample (5-10% of outputs) but the most expensive per output. Its job is not to evaluate every output — it is to calibrate the other layers and catch the failures that all three lower layers missed.

What stratified sampling means in practice

Random sampling of 10% of outputs gives you 10% of the typical-case signal and almost nothing of the rare-case signal. Stratified sampling deliberately oversamples the cases that matter most. The strata that earn their seats:

• Layer-3 borderline scores. Outputs the LLM-judge scored 3/5 — the middle scores are where the judge is least certain and where human review is most valuable.
• Adversarial cases. Every adversarial case in the eval gets human review. Always. The cost of a missed adversarial failure is high.
• New case categories. When you add a new edge case or a new tool, the first 10-20 outputs in that category get human review.
• High-stakes outputs. Anything the agent flagged as financial impact above a threshold, or anything customer-facing with sensitive content.
• Production samples. A weekly draw of 20 real production queries, reviewed by hand. The eval set should reflect production reality.

Who does the human calibration

Not the agent's developer. The developer has confirmation bias — they wrote the prompt, they want the agent to score well. The reviewer should be:

• A domain expert (the actual customer success manager, the actual analyst, the actual policy author).
• A QA engineer who is not on the agent team and is paid to find failures.
• The product manager for the agent (their reputation is on the line; they will read carefully).

Rotate reviewers. One reviewer's biases become the eval's biases.

The calibration loop

Human review is not a one-way scoring activity. It is a feedback loop. Every human-reviewed case produces two artifacts: a score, and a delta (the difference between the human score and the LLM-judge score). The deltas drive the calibration:

1. If the LLM-judge agrees with human on more than 80% of cases: the judge is calibrated. Continue. Lesson 3 covers the calibration process.
2. If the LLM-judge disagrees with human on more than 20% of cases: the judge prompt needs work. Look at the disagreements; usually the criteria are not specific enough.
3. If the LLM-judge systematically scores higher than humans: the judge is being lenient. Tighten the criteria.
4. If the LLM-judge systematically scores lower than humans: the judge is too strict, or the criteria penalize things humans accept.

The Cost and Coverage Table

Putting the four layers side by side clarifies why the stack works. Numbers below are typical 2026 values for an agent handling 100,000 queries per month. Adjust for your scale.

Layer 1 — Deterministic

• Coverage: 100% of outputs
• Cost: ~$0.00 per output (a few CPU cycles)
• Total monthly cost: $0
• Catches: schema breakage, format violations, length anomalies, missing references

Layer 2 — Heuristic

• Coverage: 100% of outputs that pass layer 1
• Cost: ~$0.0001 per output (regex, substring, simple embeddings)
• Total monthly cost: ~$10
• Catches: missing key fields, forbidden content, numeric range violations

Layer 3 — LLM-as-judge

• Coverage: 100% of outputs that pass layers 1 and 2 (or sampled to 10% for cost-sensitive ops)
• Cost: ~$0.001-$0.005 per output (Haiku, Mini, or Flash, 3-4 dimensions)
• Total monthly cost: $100-$500 at 100% coverage, $10-$50 at 10% sample
• Catches: factuality, helpfulness, hallucination, tone

Layer 4 — Human calibration

• Coverage: 5-10% stratified sample
• Cost: ~$0.50 per output (skilled reviewer)
• Total monthly cost: $2,500-$5,000
• Catches: anything the other three missed; recalibrates the judge weekly

Total cost for 100K queries/month with full stack: roughly $3,000-$6,000. Total cost of the same eval done by humans alone: $50,000+. The cost ratio is the case for the four-layer pattern.

The Stacking Discipline — Rules of the Recipe

Rule one: cheaper layers run first

An output that fails layer one (broken JSON) does not need to be scored by an LLM-judge. Run cheap layers first; cascade. Most teams get this wrong by running the LLM-judge on every output, including ones that failed the schema check. Wasteful.

Rule two: layers do not substitute, they compose

Each layer catches a different class of failure. The temptation is to drop layer one because "we have layer three, the judge will catch any problems." It will not. The judge does not check JSON syntax. The judge does not check schema. The judge will rate a malformed JSON output "okay" because it does not know to check.

Rule three: the judge is calibrated, not assumed

An uncalibrated LLM-judge is worse than no judge — it produces confident-looking scores that lead to confident-looking decisions that are wrong. Lesson 3 covers calibration. Do not skip it.

Rule four: human review never stops

The temptation as cost-pressure grows is to drop human review because "the LLM-judge is good enough." It is not. The judge drifts. The agent changes. The criteria evolve. The 5-10% human sample is the immune system of the eval. Keep it.

Rule five: weight the layers

The final eval score is not "did all four layers agree." It is a weighted aggregate. Typical weights:

• Layer 1 (deterministic): hard gate. Fail = fail the whole case.
• Layer 2 (heuristic): 20% of score for required-element checks; hard gate for forbidden content.
• Layer 3 (LLM-judge): 60% of score, weighted across the dimensions.
• Layer 4 (human): used to calibrate layer 3, not directly weighted in the score (unless human disagrees significantly, in which case the human score overrides).

The exact weights depend on your domain. A safety-critical agent weights layer 1 higher. A creative-writing agent weights layer 3 higher. Tune.

Case Study: The Four-Layer Eval of a Customer Support Agent

An e-commerce company built a customer support agent in late 2025. The agent answered shipping questions, refund inquiries, and product details. The team's eval architecture, after iteration, looked like this:

Layer 1 — Deterministic. Schema check (response, citations, intent_classification fields required), profanity gate (Perspective API), customer-ID reference integrity (cited order IDs must exist for the asking customer), email-format check on the customer email if generated. Caught 4% of outputs as hard fails.

Layer 2 — Heuristic. Key-field presence (refund-question outputs must mention timeline; shipping-question outputs must mention carrier and tracking number availability), forbidden-content check (no mention of competitor brands), citation requirement (every factual claim has a chunk reference), length sanity (50-400 words). Caught another 6% of outputs.

Layer 3 — LLM-judge. Three dimensions, each scored 1-5 by Claude Haiku: factuality (against the retrieved sources), tone-match (against the brand voice document), helpfulness (does it answer the actual question). Average score 4.1/5 on golden cases, 3.6/5 on edge cases, 2.9/5 on adversarial.

Layer 4 — Human review. Weekly stratified sample of 30 outputs: 10 from the score-3 borderline of layer 3, 10 random production samples, 10 adversarial. The customer success manager reviewed. The judge-human agreement on the borderline-3 samples started at 64% and climbed to 87% after three rounds of judge-prompt refinement (the calibration loop of lesson 3).

Total monthly cost for an agent handling 80,000 queries/month: $2,800 for layer 3 (Haiku at 100% coverage) plus $1,200 for 30 hours of CSM time on layer 4. Total: $4,000. The CSM time replaced what had previously been about $25,000/month of escalated-ticket handling — net savings of $21,000/month plus a measurable agent that was getting better every week.

Anti-Patterns of the Four-Layer Pattern

Anti-pattern one: only-judge eval

"We use GPT-5 to score every output." Misses mechanical failures. Spends 100x more than necessary. Common in teams that conflate "we have an eval" with "we have an LLM-judge."

Anti-pattern two: only-deterministic eval

"Our eval is rule-based — it's reliable." Misses tone, helpfulness, hallucination. The agent passes schema but writes stilted, unhelpful answers. Common in engineering-heavy teams that distrust LLMs as evaluators.

Anti-pattern three: uncalibrated judge

"We have an LLM-judge, it scored the agent 92%." Without human calibration, the score is meaningless — the judge could be lenient, strict, or systematically wrong. Lesson 3.

Anti-pattern four: skipping human review to save money

"LLM-judge is good enough, we cancelled the human review budget." The judge drifts. The criteria evolve. The agent changes. Without ongoing human calibration the eval degrades. Cost of human review is small relative to the cost of an undetected agent regression.

Anti-pattern five: running all layers on every change indiscriminately

Running the full 4-layer stack on every commit gets expensive. Use a smaller "smoke" subset (layers 1-3 on 50 cases) for every commit, and the full stack (all 4 layers, full eval set) nightly or weekly. Lesson 4 covers the CI architecture.

Key Takeaways

• One-layer evals fail in the dimensions they do not measure. Four layers stack to cover the full failure surface at a manageable cost.
• Layer 1 (deterministic) is essentially free, runs on 100% of outputs, catches schema/format/length failures — the mechanical 30% of bugs.
• Layer 2 (heuristic) costs fractions of a cent, catches missing key fields, forbidden content, length sanity — content rules without judgment.
• Layer 3 (LLM-as-judge) is 10x cheaper than human review at 90% of the signal — use Claude Haiku, GPT-5 Mini, or Gemini Flash; don't pay for large models as judges.
• Layer 4 (human) reviews a 5-10% stratified sample — borderline judge scores, adversarial cases, new categories, production samples. Calibrates the judge.
• Cheaper layers run first. Each layer composes; layers do not substitute for each other.
• An uncalibrated judge is worse than no judge — confident-looking but unreliable scores.
• The full four-layer stack runs nightly; a smoke subset (layers 1-3 on 50 cases) runs on every commit. CI architecture in lesson 4.