Why Calibration Is the Difference Between Scores and Signal

Your LLM judge gives Response A a 4.2 and Response B a 4.0. You pick A. But what does 4.2 actually mean? If you ran the same evaluation ten times, the score would range from 3.6 to 4.8. Your 0.2-point difference is noise, not signal. An uncalibrated LLM judge is like an uncalibrated thermometer. It gives you numbers, but those numbers do not map reliably to the quantity you are trying to measure. Calibration is the process of ensuring your judge's scores are consistent (same input produces same score), discriminating (different quality levels produce different scores), and aligned (scores match human expert judgment). A 2025 analysis of 15 production LLM judge deployments found that only 3 had calibration procedures in place. The other 12 were making decisions based on scores whose precision they had never measured. This lesson teaches you how to calibrate any LLM judge from scratch.

What Calibration Means Formally

A calibrated judge has three properties. Consistency (reliability): If you evaluate the same output N times, the scores cluster tightly. Measure this with the coefficient of variation (CV = standard deviation / mean). Target: CV < 0.10 for a well-calibrated judge. Discrimination: The judge assigns meaningfully different scores to outputs of different quality. Measure with the effect size (Cohen's d) between your 'known good' and 'known bad' examples. Target: d > 1.5 for a well-discriminating judge. Alignment: The judge's rankings match human expert rankings. Measure with Kendall's tau or Spearman's rho between judge scores and human scores. Target: tau > 0.60.

A judge can be consistent without being aligned (it reliably gives wrong scores). It can be aligned on average without being discriminating (it gets the overall ranking right but assigns very similar scores to different quality levels). You need all three.

| Property | Metric | Formula Intuition | Target |
|---|---|---|---|
| Consistency | CV | stdev / mean | < 0.10 |
| Discrimination | Cohen's d | (mean_good - mean_bad) / pooled_stdev | > 1.5 |
| Alignment | Kendall's tau | Rank correlation with humans | > 0.60 |

Building Your Calibration Set

Everything depends on the quality of your calibration set. Build it systematically. Step 1: Collect outputs across the full quality spectrum. You need examples rated 1/5 through 5/5 by human experts. If your evaluator only sees good outputs, it cannot learn to discriminate. Aim for 200+ examples minimum, distributed roughly uniformly across quality levels (40 per level for a 5-point scale). Step 2: Get multi-rater human labels. Each example needs ratings from at least 3 qualified raters. Compute inter-rater reliability (Krippendorff's alpha). If alpha < 0.67, your rubric is ambiguous, fix it before proceeding. Step 3: Create difficulty tiers. Label each example as 'easy' (unanimous human agreement), 'medium' (majority agreement), or 'hard' (split). Your calibration analysis should report accuracy separately for each tier. A judge that scores 95% on easy cases but 40% on hard cases is very different from one that scores 70% across the board. Step 4: Include adversarial examples. Add outputs that are designed to trigger known biases: verbose-but-wrong, well-formatted-but-shallow, confident-but-inaccurate. These test whether your judge falls for surface-level quality signals.

Reading and Building Calibration Curves

A calibration curve plots the judge's predicted quality against actual quality (human ratings). Perfect calibration is a 45-degree line: when the judge says 3.0, human experts also say 3.0. In practice, LLM judges exhibit two characteristic distortions. Score compression: The judge uses only a narrow range of the scale (typically 3.0-4.5 on a 1-5 scale). The calibration curve is flat, small changes in actual quality produce no change in judge scores. This destroys discrimination. Ceiling effects: The judge rarely assigns scores below 3.0 or above 4.5, compressing the extremes. Excellent outputs are underrated and terrible outputs are overrated.

To build your calibration curve: (1) Plot human scores (x-axis) against judge scores (y-axis) for your 200+ calibration examples. (2) Fit a linear regression. If the slope is significantly less than 1.0, you have compression. If the intercept is significantly above 0, you have inflation. (3) Use the regression line as a correction function: corrected_score = (raw_score - intercept) / slope. This simple linear correction often improves alignment (Kendall's tau) by 0.10-0.15 points. For non-linear distortions, use isotonic regression, which makes no assumptions about the shape of the correction.

Calibration Through Prompt Engineering

The evaluation prompt is your primary calibration lever. Small changes produce large effects. Rubric specificity: Vague rubrics ('Rate the quality from 1-5') produce compressed, unreliable scores. Specific rubrics with behavioral anchors for each score level produce better discrimination. Define exactly what 1, 2, 3, 4, and 5 look like with concrete examples. Chain-of-thought: Requiring the judge to reason step-by-step before scoring improves calibration significantly. G-Eval showed that CoT prompting increased Spearman correlation with human judgments by 0.12 on summarization tasks. Few-shot calibration examples: Include 3-5 example outputs with their correct scores in the prompt. Choose examples that span the full range (one each at 1, 3, and 5 minimum). This anchors the judge's scale to your intended distribution. Score definition: Instead of 'Rate 1-5,' define scores as: '1 = Completely wrong or harmful. 2 = Major errors that make the response unreliable. 3 = Adequate but with notable issues. 4 = Good with minor issues. 5 = Excellent, no meaningful improvements possible.' Test your prompt against your calibration set. If tau < 0.60, revise the prompt and retest.

Temperature, Sampling, and Score Aggregation

The judge's inference parameters directly affect calibration. Temperature: At temperature 0, scores are deterministic but potentially biased by greedy decoding. At temperature 0.3-0.7, you get variance that can be aggregated for better estimates. The optimal strategy: run each evaluation N times at temperature 0.3-0.5 and take the mean. How many samples? Diminishing returns set in quickly. For most judge models, N=5 gives 80% of the variance reduction of N=20. Use N=3 for cost-sensitive applications, N=5-7 for critical decisions. Aggregation method matters: Mean is sensitive to outliers. Median is more robust. For categorical decisions (pass/fail), use majority vote. For ordinal scores, use trimmed mean (drop the highest and lowest, average the rest).

Score precision formula: Standard error = stdev / sqrt(N). If your judge has stdev = 0.6 at temperature 0.3, then: N=1 gives SE = 0.60 (useless for 0.2-point differences). N=5 gives SE = 0.27 (marginal). N=10 gives SE = 0.19 (can detect 0.4-point differences). Choose N based on the minimum score difference you need to detect reliably.

Calibrating Across Multiple Judges

Multi-judge ensembles reduce bias but introduce a new challenge: different judges use different scales. GPT-4 might center at 3.8 while Claude centers at 3.2 on identical outputs. You cannot simply average their raw scores. Z-score normalization: For each judge, compute its mean and standard deviation on the calibration set. Normalize: z_score = (raw_score - judge_mean) / judge_stdev. Now all judges operate on the same scale (standard deviations from their respective means). Average the z-scores. Weighted ensemble: Not all judges are equally good. Weight each judge by its alignment with human ratings: weight_i = tau_i / sum(tau). This gives more influence to more accurate judges. Disagreement routing: When judges disagree by more than 1.5 standard deviations, do not average, route to human review. These high-disagreement cases are where automated evaluation is most likely to fail. In practice, 10-20% of cases trigger disagreement routing, and human review on those cases improves overall pipeline accuracy by 8-12%.

Monitoring Calibration Drift

Calibration is not a one-time event. Judge performance drifts over time due to three factors. Model updates: When the judge model is updated (GPT-4 to GPT-4-turbo, Claude 3 to Claude 3.5), calibration can shift dramatically. Always re-run your calibration set after any model update. Distribution shift: If the outputs being evaluated change in character (new domain, different model generating them), the judge's calibration may not transfer. A judge calibrated on customer service responses may be poorly calibrated on technical documentation. Rubric evolution: As your team's quality standards evolve, the rubric changes, and the judge's alignment with the new rubric must be re-established. Monitoring system: Run your calibration set through the judge weekly. Track three metrics over time: consistency (CV), discrimination (Cohen's d), and alignment (tau). Alert if any metric drops more than 10% from its baseline. Plot all three on a dashboard. When you see degradation, diagnose whether it is a model change, distribution shift, or rubric evolution, and recalibrate accordingly.

Advanced Calibration Techniques

Beyond the fundamentals, these techniques push calibration further. Probability calibration: Instead of asking for a score, ask the judge to estimate the probability that a human expert would rate the output above a threshold. For example: 'What is the probability that a human expert would rate this response 4 or higher?' Then apply Platt scaling or isotonic regression to map predicted probabilities to observed frequencies. This yields better-calibrated uncertainty estimates than raw scores. Dimension-specific calibration: Calibrate each evaluation dimension (accuracy, fluency, helpfulness) separately. Different dimensions have different bias profiles, accuracy judgments tend to be better calibrated than helpfulness judgments. Contrastive calibration: Instead of absolute scoring, present the judge with an anchor response of known quality and ask whether the target response is better, equal, or worse. This relative judgment is often more calibrated than absolute scoring because it removes scale interpretation differences. The Prometheus framework uses contrastive calibration to achieve near-human agreement on open-ended evaluation tasks.

Try This Now

Build a minimal calibration set and test your LLM judge. Step 1: Gather 30 outputs from your AI system. Have 2-3 team members rate each on a 1-5 scale using your existing rubric. Compute inter-rater agreement (Cohen's kappa for 2 raters, Krippendorff's alpha for 3+). Step 2: Run your LLM judge on all 30 outputs. Compute Kendall's tau between the judge's rankings and the human consensus rankings. Step 3: Plot the calibration curve (human score x-axis, judge score y-axis). Does it look like a 45-degree line? If not, fit a linear regression and use it as a correction. Step 4: Run the judge 5 times on 10 of the outputs (with temperature 0.3). Compute the coefficient of variation. Is it below 0.10? Step 5: Record your baseline: tau = ___, CV = ___, slope = ___, intercept = ___. These four numbers tell you exactly how calibrated your judge is and what corrections to apply.

Key Takeaways

Calibration transforms an LLM judge from a number generator into a measurement instrument. Without it, your scores are arbitrary. With it, they are actionable. Three properties define a calibrated judge: consistency (CV < 0.10), discrimination (Cohen's d > 1.5), and alignment (Kendall's tau > 0.60). Build a calibration set of 200+ outputs with multi-rater human labels across the full quality spectrum. Use calibration curves to detect and correct score compression and ceiling effects, a simple linear correction often improves alignment by 0.10-0.15 tau points. Calibrate through prompt engineering: specific rubrics, chain-of-thought, few-shot examples, and explicit score definitions. For critical decisions, run the judge N=5 times and aggregate with trimmed mean. When using multi-judge ensembles, z-score normalize before aggregating and route high-disagreement cases to human review. Monitor calibration weekly: model updates, distribution shifts, and rubric evolution all cause drift. Recalibrate proactively, not after you discover your scores have been meaningless for a month.