Every team that adopts LLM-as-judge has the same moment of false confidence in week two. The judge prompt is written. The eval is running. The scores look reasonable — most cases score 3 or 4, the failure cases score 1 or 2. The team starts trusting the numbers. Six weeks later someone hand-reviews 30 cases that scored 4/5 and discovers that 12 of them are actually wrong — the judge was systematically lenient on a specific failure mode and nobody noticed. The eval was confidently telling them the agent was getting better while the agent was actually getting worse. This is the most expensive failure mode in eval architecture: the uncalibrated judge that produces confident-looking but unreliable scores. The fix is calibration — a one-week project, not a PhD. Take 50 cases. Have a human score them. Have the LLM-judge score them. Compute agreement. Tune the prompt until agreement crosses 80%. Document the calibrated prompt as a versioned artifact. Repeat monthly. This lesson is how, including the failure modes that make naive calibration go sideways.

The Illusion of the Confident Judge

The LLM-judge prompt looks like a quality scorecard. "Score this output on factuality from 1 to 5. Provide reasoning." The judge returns a 4. The reasoning sounds plausible. The team moves on.

What the team has not done: verified that the judge's 4 corresponds to what a human would call a 4. Maybe the judge calls almost everything a 4. Maybe the judge calls everything that mentions a specific keyword a 4. Maybe the judge is doing keyword-matching against the input and not actually evaluating factuality. Maybe the judge is over-weighting length and considering any long answer a 4. Without calibration, you do not know what the score means.

The uncalibrated judge fails in a particular way that makes it especially dangerous: it fails confidently. Every score looks right. Every reasoning looks plausible. Every comparison between runs looks like it carries signal. The eval system feels healthy. The agent is shipping changes that score better. And six weeks later you discover the metric has been moving in the wrong direction the whole time.

An uncalibrated LLM-judge is worse than no judge. No judge means you know you are flying blind and you act accordingly. An uncalibrated judge means you have confident-looking numbers leading to confident-looking decisions — and you do not know until something breaks badly that the numbers were noise.

The bar conversation that ends 'I trust my judge'

A data analytics team at a mid-stage company built an internal SQL agent in early 2026. The team wrote an LLM-judge prompt that scored factuality on a 1-5 scale. The eval ran nightly. Scores trended up over three months. The team celebrated. The agent was getting better.

An analyst with deep domain knowledge spent an afternoon reading 30 randomly sampled agent answers. Twelve of the thirty were technically wrong in ways the judge had marked as 4 or 5. The wrong answers all shared a pattern: the agent was reusing the schema names from the question even when joining the wrong tables. The judge was scoring the structure-of-the-answer (which looked correct) instead of the meaning (which was wrong).

The eval had been telling the team the agent was getting better while the agent was getting worse on the specific failure mode the team cared about most. The fix was calibration — three days of work. The agreement between judge and analyst rose from 53% to 84% after two rounds of prompt refinement. The eval became trustworthy. The agent's actual quality started moving in the same direction as the eval score.

What Calibration Actually Is

Calibration is the process of measuring and improving the agreement between the LLM-judge's scores and a trusted human's scores on the same set of examples. The process is mechanical. The PhD is not required.

The four steps that complete one calibration cycle:

1. Build a 50-case calibration set. Sampled from your real eval corpus. Stratified across score levels (you need cases that humans would call 1, 2, 3, 4, and 5 — not 50 cases that are all clearly correct).
2. Score the calibration set by hand. A domain expert reads each case, scores it 1-5, writes a one-sentence reason. Two-to-three hours of focused work.
3. Score the calibration set with the judge prompt. Run the LLM-judge on the same 50 cases. Capture scores and reasoning.
4. Compute agreement and identify disagreements. Tabulate where the judge and human matched, where they didn't. The disagreement patterns are the calibration signal.

If agreement is above 80%, the judge is calibrated. Document the prompt, version it, ship it. If agreement is below 80%, the judge prompt needs work — and the disagreements tell you what to fix.

Building the 50-Case Calibration Set

The stratification that matters

The biggest mistake in calibration set construction is sampling 50 random cases. The score distribution of random samples is heavily skewed — usually 70-80% of cases would be 4 or 5, 10-15% would be 2 or 3, almost none would be 1. The calibration set ends up measuring the judge's behavior at the top of the scale and ignoring the middle and bottom, where judges most often fail.

Stratify deliberately. Aim for roughly:

• 10 cases at score 5 (clearly excellent)
• 10 cases at score 4 (good, minor issues)
• 10 cases at score 3 (borderline — the hardest cases)
• 10 cases at score 2 (clearly poor)
• 10 cases at score 1 (clearly wrong or harmful)

The borderline-3 cases are the most valuable. The judge's behavior on the obvious-5 and obvious-1 cases is usually correct. The judge fails on the cases that require nuanced judgment. Over-sample the middle.

Where to source the cases

• Production traffic samples. Real queries the agent has handled. The closest match to production reality.
• The existing eval set. Golden, edge, and adversarial cases. Re-use what you have.
• Synthetic cases for under-represented score bands. If you cannot find enough natural score-1 examples, synthesize a few (lesson 5 covers synthetic generation in depth).

What metadata to capture per case

Each calibration case gets four fields beyond the input/output:

• Human score: 1-5 from the domain expert.
• Human reasoning: a one-sentence explanation of the score. ("Wrong customer ID; used customer 4421 instead of 4412.")
• Category tag: what kind of case is this — factuality failure, tone problem, refusal issue, multi-step reasoning failure?
• Difficulty rating: easy / medium / hard. Easy cases anyone would score consistently; hard cases experts disagree on.

The metadata enables disagreement analysis. If the judge disagrees on 30% of cases and 25% of the disagreements are in the "multi-step reasoning failure" category, you know exactly where to focus prompt refinement.

The Human Scoring Protocol

Who scores

The human scorer must be a domain expert. Not the agent developer (confirmation bias). Not a generic crowd-worker (lacks domain context). The right scorers:

• A senior practitioner in the agent's domain (customer success manager for a support agent, financial analyst for a finance agent, policy author for a compliance agent).
• An internal QA engineer who is paid to find failures and has domain context.
• The product manager for the agent, when their stake in correctness is real.

Two scorers are better than one. Two scorers calibrated against each other (inter-rater agreement) tell you how much of the disagreement is human-vs-human noise versus judge-vs-human bias. Aim for 90%+ inter-rater agreement before you trust the human scores as a baseline.

The scoring rubric

A scoring rubric is the most under-invested artifact in calibration. The temptation is to ask the scorer "is this good, 1-5?" and accept their gut feel. Their gut feel will be inconsistent. Their score-3 today is a score-4 next week.

A scoring rubric specifies what each score level means in concrete terms. For factuality:

• 5 — Fully factual: every claim in the answer is supported by the retrieved sources. No unsupported claims. No contradictions.
• 4 — Mostly factual: most claims supported, one or two minor unsupported claims that do not change the answer's substance.
• 3 — Mixed: some claims supported, others not. The core claim may be unsupported. Reader cannot rely on the answer without verification.
• 2 — Mostly unfactual: majority of claims are unsupported or contradicted by sources. Core answer is wrong but partially related to the question.
• 1 — Fully wrong: answer contradicts the sources or makes up facts wholesale. Harmful if relied on.

Concrete criteria at each level. Then the human scorer applies the same rubric to all 50 cases, and you give the same rubric to the LLM-judge. Both are scoring the same thing.

The two-hour focused work session

Human scoring is fatigue-sensitive. After 90 minutes of close reading, scoring quality drops noticeably. Plan the session: 50 cases, two 45-minute sessions with a break in between, total work 90-120 minutes. Avoid scoring in chunks across days — context-shift drift makes the scores inconsistent.

Running the Judge Against the Calibration Set

This is the easy part. Wrap the existing judge prompt around the calibration set inputs, run it, capture the scores and reasoning. Five minutes of work in any of the eval platforms (Braintrust, LangSmith, Langfuse all support this directly).

What to capture per judge call

• The judge's score (1-5).
• The judge's reasoning (the text the judge wrote to justify the score).
• The judge model and version (Claude Haiku 4, GPT-5 Mini, Gemini 2.5 Flash — versions matter; the same prompt scored by different models scores differently).
• The exact judge prompt used (versioned).
• Token cost (for tracking spend per calibration cycle).

Run each judge call three times if you can afford it. The self-consistency check (covered below) is the cheapest signal about judge reliability you can buy.

Computing Agreement — The Numbers That Matter

The simplest agreement metric: exact match

The judge's score equals the human's score. Count how many cases agreed exactly out of 50. Divide. Percent agreement. Below 80% the judge is uncalibrated; tune it. Above 80% the judge is good enough for production.

The slightly-more-honest metric: off-by-one tolerance

Exact match is harsh. A judge that scores 4 when the human scores 5 is still useful — both agree the answer was good. A judge that scores 1 when the human scores 5 is a disaster. Off-by-one agreement counts a match if scores are within 1 of each other. A more practical metric for production decisions.

Use both. Exact match for the strict calibration target (80%+). Off-by-one match for sanity checks (should be 95%+).

The bias metrics: systematic skew

Beyond agreement, look for systematic bias:

• Mean delta: average (judge_score - human_score) across the 50 cases. If the mean delta is +0.5, the judge is systematically lenient (scoring higher than humans). If -0.5, systematically strict. Zero is calibrated.
• Score distribution comparison: what fraction of judge scores are 5, 4, 3, 2, 1? Compare to the human distribution. If the judge's distribution is much flatter or more peaked than the human's, the judge is not discriminating the right way.
• Per-category agreement: which case categories show the worst agreement? Factuality cases? Tone cases? Adversarial cases? The pattern tells you where the prompt is failing.

Cohen's kappa for the statistically inclined

Cohen's kappa is the inter-rater agreement statistic that corrects for chance agreement. Useful but not required. Above 0.6 is substantial agreement; above 0.8 is almost-perfect. Compute it if your stakeholders like statistics. The off-by-one agreement plus mean delta plus per-category breakdown is enough for most production decisions.

The Prompt Refinement Loop

Calibration is not a one-pass measurement. The first run will likely show agreement in the 50-70% range. The work is to iterate the prompt and measure again.

Looking at the disagreement cases

Print every case where judge and human disagree by 2 or more points. Look at the judge's reasoning. Three patterns appear repeatedly:

• Vague criteria. The judge prompt said "factuality" without defining it. The judge interpreted factuality differently than the human. Fix: add explicit per-score-level criteria.
• Missing context. The judge did not have access to the ground truth, the retrieved sources, or the expected output. It was guessing. Fix: include the reference material in the judge prompt.
• Misleading examples. If the judge prompt includes example scores, those examples bias the judge toward similar-looking outputs. Fix: review the examples, ensure they span the score range, remove examples that overlap with the test cases.

The prompt-version-and-rerun cycle

1. Identify the top disagreement pattern (largest cluster of bad scores).
2. Add a specific clause to the judge prompt addressing the pattern. ("Note: a multi-step query that succeeds on the first step but fails on the second is a score-2, not score-4.")
3. Rerun the judge on the 50-case calibration set.
4. Recompute agreement. Did it improve? Did anything regress?
5. If agreement crossed 80%: stop. Document. Version. Ship.
6. If not: identify the next disagreement pattern, repeat.

Three rounds is typical. Most prompts converge from 50-60% initial agreement to 80%+ after three iterations. Some require five rounds. If you are past round seven with no convergence, the underlying dimension may be inherently subjective and you need a different evaluation approach (multiple judges, ensemble scoring, or accepting a lower agreement bar with more human review).

Failure Mode: The Judge That Agrees With the Agent Because They're the Same Model

The most common — and most insidious — calibration failure is using the same model family for both the agent and the judge. The judge agrees with the agent because they share biases, training data, and reasoning patterns.

The concrete failure mode

The agent is GPT-5. The judge is GPT-5. The agent produces an answer that subtly hallucinates. The judge reads the answer and finds it plausible — same reasoning patterns produced the hallucination as would evaluate it. The judge scores it 4/5. The human sees the hallucination immediately. The judge missed it because it shared the agent's blind spot.

This is not theoretical. Multiple 2024-2025 academic papers documented same-model self-agreement at 5-15% higher than cross-model agreement. The judge inflates the agent's apparent quality by 5-15% just because they are the same model.

The fix: cross-family judging

Use a different model family for the judge than for the agent. Concrete pairings that work:

• Agent: GPT-5 / GPT-5 Pro. Judge: Claude Haiku 4 or Sonnet 4.5.
• Agent: Claude Sonnet 4.5 / Opus. Judge: GPT-5 Mini or Gemini 2.5 Flash.
• Agent: Gemini Pro 2.5. Judge: Claude Haiku 4 or GPT-5 Mini.

Cross-family judges are usually within 2-3% of same-family judges on agreement metrics on cases the agent gets right. They diverge on the cases the agent gets wrong — which is exactly where you need divergence. The cross-family judge catches the hallucinations the same-family judge would miss.

The ensemble option for high-stakes evaluation

For agents in regulated or safety-critical contexts, use two judges from different families and average (or require both to agree above a threshold for a positive verdict). The cost roughly doubles. The bias-reduction is real. Worth it when the cost of an undetected failure is high.

The Versioned Prompt as a Shipping Artifact

The calibrated judge prompt is an artifact. Version it. Treat it like code. Anything else is asking for silent drift.

What versioning looks like in practice

• Store the judge prompt in your eval platform's prompt registry (Braintrust prompts, LangSmith prompt hub, Langfuse prompts) or in your code repo as a Python/TypeScript constant with a version string.
• Each prompt version has a calibration record: the 50-case set used, the agreement score, the date, the human scorer, the judge model and version.
• Major prompt changes get a new version (judge_factuality_v3 → v4). The calibration must be re-run for each version.
• The version that ships to production must have a documented calibration record. No undocumented prompt changes ship.

The monthly recalibration cadence

Calibration is not a one-time activity. The judge prompt may stay the same, but the model behind it can update (a new Claude Haiku version ships every 3-6 months in 2026), the production traffic distribution can shift, and your agent itself changes. Recalibrate monthly. The recalibration is cheap — 50 cases, three hours of human time, an hour of analysis — and catches drift before it becomes a problem.

The recalibration trigger list

• Judge model version changes (e.g. Claude Haiku 4.0 → 4.1).
• Major agent model change (GPT-5 → Claude Opus 5, or similar).
• New tools added to the agent (the dimensions of correctness may shift).
• Quarterly even if nothing changed (drift detection).
• Any time stakeholder reports of agent quality diverge from eval scores (a tell that the judge is drifting).

Judge Failure Modes Beyond Same-Model

Length bias

Judges tend to score longer outputs higher. The mechanism: longer outputs look more thorough, and judges read "thorough" as "good." Mitigation: tell the judge explicitly in the prompt that length is not a factor ("ignore output length in your evaluation"). Spot-check with a calibration case that is short-and-correct versus long-and-wordy — the short answer should score higher.

Position bias (comparison judging)

When the judge compares two outputs A and B and picks the better one, judges prefer whichever is shown first. The bias is 5-15% in published studies. Mitigation: randomize position. Or run the comparison twice with positions swapped and accept only verdicts where both runs agree.

Self-consistency

The same judge on the same input gives different scores across runs. This is sampling-driven; temperature > 0 introduces variation. Mitigation: temperature = 0 in the judge call (different from the agent call). Run three times and take the mode if budget allows.

Anchoring on the input

The judge weights the input (the user's query) too heavily and pattern-matches the answer to the input rather than checking the actual claim. Mitigation: include the ground truth (or retrieved sources) in the prompt; instruct the judge to evaluate against the ground truth, not the input.

Refusal-blindness

Judges sometimes score refusals (the agent's "I cannot answer this from the sources") as low because they "didn't answer." Refusals are correct behavior when sources are insufficient. Mitigation: include a separate criterion for refusal appropriateness; tell the judge that a correct refusal is a 5 and an over-confident wrong answer is a 1.

The Three-Day Calibration Sprint

If you have never calibrated a judge before, here is the work plan that gets you from zero to a calibrated production judge in three days.

Day one

• Pick the agent dimension to evaluate first (factuality is usually the right starting point).
• Write the v1 judge prompt with five-point rubric, explicit per-score criteria, reasoning field.
• Build the 50-case calibration set (stratified, sourced from production samples + eval set).
• Schedule the two-hour human scoring session for day two.

Day two

• Human scores the 50 cases in a focused 90-120 minute session.
• Run the v1 judge prompt on the 50 cases (15 minutes of compute).
• Compute agreement, mean delta, per-category breakdown.
• Identify the top disagreement patterns (1-2 hours).
• Write v2 judge prompt addressing the patterns. Rerun.

Day three

• Analyze v2 results. If agreement crossed 80%, document and ship. If not, write v3 and rerun.
• Document the calibrated prompt with version, agreement score, date, human scorer, judge model.
• Set up the monthly recalibration calendar event.

Three days. Production-grade calibrated judge. No PhD required.

Key Takeaways

• An uncalibrated LLM-judge produces confident-looking unreliable scores. It is worse than no judge because it leads to confident wrong decisions.
• Calibration is mechanical: 50-case set, human scores, judge scores, compute agreement, iterate the prompt until agreement crosses 80%.
• Stratify the calibration set across all five score levels — over-sampling the borderline-3 cases where judges most often fail.
• Use a five-point rubric with explicit per-score criteria. Document each score level concretely. Same rubric to human and judge.
• The biggest insidious failure: using the same model family for agent and judge. Same-model self-agreement inflates scores by 5-15%. Use cross-family judging.
• Other judge failure modes to watch: length bias, position bias in comparisons, self-consistency drift, refusal-blindness.
• Version the calibrated prompt. Treat it like code. Each version has a calibration record (50-case set, agreement, date, scorer, model).
• Recalibrate monthly, or sooner on model-version changes, major agent changes, or stakeholder-eval divergence reports.
• The three-day calibration sprint: day one set up + write v1, day two human + judge run + v2, day three v2 analysis + ship or v3.