Choosing the Wrong Metric Costs More Than Choosing No Metric

A search team optimizes for NDCG@10. Rankings improve by 8%. User satisfaction drops 12%. What happened? NDCG measured relevance at position 10, but users only looked at the first 3 results, and the optimization moved the best results from position 1-2 to positions 4-6 while improving positions 7-10 nobody cared about. The metric improved while the experience degraded. This is not a rare edge case; it is the default outcome when you pick metrics without a systematic selection process. The right metric aligns your optimization target with your actual quality objective. The wrong metric creates a divergence that grows silently until users revolt. This lesson gives you a rigorous checklist for selecting evaluation metrics, one that prevents the 'improving the wrong thing' failure mode that derails more AI projects than any technical limitation.

Step 1: Define Your Quality Dimensions

Before selecting metrics, you need to know what you are measuring. Most teams skip this step and jump straight to 'let us use BLEU' or 'let us use an LLM judge.' Start by listing every quality dimension that matters for your application. Common dimensions for generative AI:

• Correctness: Is the output factually accurate?
  - Faithfulness: Does the output stay grounded in provided sources?
  - Relevance: Does the output address what the user asked?
  - Completeness: Does the output cover all aspects of the request?
  - Conciseness: Is the output appropriately brief?
  - Safety: Does the output avoid harmful content?
  - Fluency: Is the language natural and well-formed?
  - Consistency: Does the output contradict itself?
  - Format compliance: Does the output match the required structure?

Rank these dimensions by importance for YOUR use case. A medical Q&A system ranks correctness and safety first. A creative writing assistant ranks fluency and relevance first. Your metric selection must reflect this ranking, never measure a low-priority dimension with a high-fidelity metric while measuring a high-priority dimension with a rough proxy.

Step 2: Assess Reference Availability

The single most important factor in metric selection is whether you have reference outputs (ground truth). This splits the metric landscape in two.

Reference-based metrics compare generated output against a known-correct answer. They require curated test sets with gold-standard outputs. Examples: BLEU, ROUGE, BERTScore, exact match, F1. Best for: translation, factoid QA, structured extraction, summarization of short documents.

Reference-free metrics evaluate output quality without a ground truth comparison. They assess intrinsic properties or use a judge (human or LLM). Examples: LLM-as-judge scores, perplexity, self-consistency, faithfulness metrics (NLI-based). Best for: open-ended generation, creative tasks, conversational AI, any task where multiple correct answers exist.

| Question | If Yes | If No |
|---|---|---|
| Is there exactly one correct answer? | Reference-based (exact match, F1) | Reference-free |
| Can you enumerate acceptable answers? | Reference-based (BLEU, ROUGE) | Reference-free |
| Is the source document the reference? | Faithfulness metrics (NLI, QA-based) | LLM-as-judge |
| Is quality purely subjective? | Human eval or LLM-as-judge | N/A |

Most production AI systems need both types: reference-based for testable facts and reference-free for subjective quality.

Step 3: Validate Correlation with Human Judgment

A metric is only useful if it correlates with what humans consider quality. Never assume, measure. The validation protocol: (1) Collect 100+ outputs. (2) Have 3+ human raters score each output on your target quality dimension. (3) Compute your candidate metric for each output. (4) Calculate Kendall's tau or Spearman's rho between metric scores and human scores. Interpret the results:

• tau > 0.60: Strong correlation. The metric is a reliable proxy for human judgment on this dimension.
  - tau 0.40-0.60: Moderate correlation. Use as a signal but not as the sole decision-maker.
  - tau 0.20-0.40: Weak correlation. Use only for coarse filtering or alerting.
  - tau < 0.20: No meaningful correlation. Do not use this metric for this dimension.

Known correlation ranges from the literature: BLEU with human quality (translation): tau 0.30-0.50. BERTScore with human quality (summarization): tau 0.40-0.55. G-Eval with human quality (open-ended): tau 0.55-0.70. Human-human agreement: tau 0.60-0.80. If your metric correlates worse than BLEU (tau < 0.30), it is adding noise, not signal. If it approaches human-human agreement (tau > 0.60), you have a strong metric.

Step 4: Test Sensitivity and Specificity

A good metric must be sensitive (it changes when quality changes) and specific (it does not change when quality stays the same). Sensitivity test: Create 20 output pairs where one is clearly better than the other on your target dimension (validated by humans). Compute your metric for both. The metric should assign a higher score to the better output at least 80% of the time. If it does not, the metric cannot distinguish quality levels. It lacks sensitivity. Specificity test: Create 20 pairs of outputs that differ on a NON-target dimension (e.g., formatting or length) but have equal quality on your target dimension. The metric scores should be approximately equal. If the metric assigns systematically different scores, it is measuring the wrong thing. It lacks specificity. Minimum detectable effect: Determine the smallest quality difference your metric can reliably detect. Run your metric on 50 pairs with a known 1-point quality difference (on a 5-point human scale). If the metric scores overlap significantly (Cohen's d < 0.5), it cannot detect this difference. You need a larger quality gap or a more sensitive metric.

Step 5: Evaluate Cost and Latency Constraints

Every metric has a cost profile. Map it before committing.

| Metric Type | Cost per 1K evals | Latency | Scale Limit |
|---|---|---|---|
| String match / regex | ~$0 | < 1ms | Unlimited |
| BLEU / ROUGE | ~$0.01 | < 10ms | Unlimited |
| BERTScore | ~$0.10 | 50-200ms | GPU-bound |
| NLI-based faithfulness | ~$0.50 | 100-500ms | GPU-bound |
| LLM-as-judge (GPT-4) | $2-10 | 2-10s | API rate limits |
| LLM-as-judge (open-source) | $0.10-0.50 | 1-5s | GPU-bound |
| Human evaluation | $50-500 | Hours-days | Annotator availability |

Decision rules: Real-time production gating (every output) → must be < 100ms and < $0.01. Batch quality monitoring (daily sample) → can be up to $5/eval and 10s latency. Strategic benchmarking (monthly) → human evaluation is feasible. Match each quality dimension to a cost tier. High-priority dimensions get expensive, high-fidelity metrics. Lower-priority dimensions get cheap, approximate metrics. Never spend your entire evaluation budget on one dimension while leaving others unmeasured.

Step 6: Assess Gaming and Goodhart Resistance

Any metric you optimize against will eventually be gamed, by humans tuning the system or by the model itself through RLHF. Goodhart's Law is not theoretical; it is the default outcome. Rate each candidate metric on gaming resistance. High gaming risk: Metrics that can be improved by surface-level tricks. BLEU can be gamed by repeating n-grams from the reference. ROUGE can be gamed by extracting sentences verbatim. Length-based metrics incentivize padding. These metrics degrade under optimization pressure. Medium gaming risk: LLM-as-judge scores can be gamed by learning the judge's formatting preferences (markdown, bullet points, longer responses). This is exactly what happens during RLHF when the reward model has formatting bias. Low gaming risk: Human evaluation with rotating raters and evolving rubrics. Observational metrics tied to real outcomes (task completion, user retention). Multi-metric composites that require improvement across multiple orthogonal dimensions simultaneously. The test: Ask yourself, if a team optimized solely for this metric for 6 months, would the resulting system actually be better? If you can imagine a degenerate 'hack' that improves the metric without improving quality, the metric has a gaming vulnerability. Document it and add a complementary metric that catches the hack.

Step 7: Build Composite Metrics Carefully

Rarely does a single metric capture everything you need. Composite metrics combine multiple signals, but they introduce new risks. Weighted sum: quality_score = w1*accuracy + w2*fluency + w3*safety. Simple and interpretable, but the weights are arbitrary, how do you decide that accuracy is worth 0.5 and fluency is worth 0.3? Threshold gating: First apply hard constraints (safety score > 0.95, format compliance = 100%), then rank by quality metrics. This prevents trading safety for quality. Hierarchical evaluation: Evaluate correctness first. Among correct outputs, evaluate helpfulness. Among helpful outputs, evaluate conciseness. This creates a lexicographic ordering that respects your priority ranking.

Pitfalls to avoid: (1) Do not average metrics on different scales without normalization. (2) Do not combine metrics that measure the same underlying construct (double-counting). (3) Do not use weights you have not validated, run A/B tests to confirm that higher composite scores predict better user outcomes. (4) Always report component metrics alongside the composite. A composite score of 0.78 is meaningless without knowing that accuracy = 0.92, fluency = 0.85, and safety = 0.57, the safety component is failing and the composite hides it.

The Complete Metric Selection Checklist

Use this checklist for every metric you adopt. Print it, tape it to your monitor, and do not skip steps.

• [ ] Quality dimensions listed and ranked by importance for your use case
  - [ ] Reference availability assessed for each dimension (reference-based vs. reference-free)
  - [ ] Correlation validated, tau > 0.40 against human judgment on 100+ examples
  - [ ] Sensitivity confirmed, metric distinguishes better from worse outputs 80%+ of the time
  - [ ] Specificity confirmed, metric does not change when non-target dimensions change
  - [ ] Cost and latency mapped to your budget and speed requirements
  - [ ] Gaming scenarios documented, at least 3 ways the metric could be hacked
  - [ ] Complementary metrics identified, each gaming vulnerability covered by another metric
  - [ ] Composite construction validated, component weights tested against user outcomes
  - [ ] Monitoring plan in place, metric-human correlation re-checked quarterly
  - [ ] Sunset criteria defined, conditions under which you replace this metric

A metric that passes all 11 checks is ready for production. A metric that fails any of the first 4 should not be used at all.

Metric Selection Anti-Patterns

Avoid these five patterns that teams fall into repeatedly. The legacy metric: Using BLEU for open-ended generation because 'that is what the literature uses.' BLEU was designed for machine translation with multiple references. Its correlation with human quality on free-form generation is often below 0.20. The single metric: Reducing all quality to one number. This always hides critical failures. A system with accuracy = 0.95 and safety = 0.60 looks fine as a weighted average (0.81) but is unacceptable. The leaderboard metric: Adopting whatever metric a public leaderboard uses without checking if it is relevant to your task. MMLU measures knowledge, not helpfulness. HumanEval measures code correctness, not code quality. The untested metric: Using an LLM-as-judge score without ever validating it against human judgment on your specific task. A judge calibrated for customer service may be poorly calibrated for legal document review. The static metric: Setting up metrics once and never revisiting them. Your system changes, your users change, your quality bar changes. Metrics that were valid six months ago may no longer correlate with user satisfaction.

Try This Now

Apply the checklist to your current evaluation setup. Step 1: List every metric you currently use. For each one, write down which quality dimension it measures. Step 2: Identify gaps, are there quality dimensions you care about that have no metric? Are there metrics measuring dimensions you do not care about? Step 3: For your most important metric, run the correlation check: take 30 outputs, have 2+ people rate them on the quality dimension, compute your metric, and calculate Kendall's tau. Is it above 0.40? Step 4: For the same metric, run the gaming test: can you think of a way to improve the metric score without improving actual quality? If yes, what complementary metric would catch that hack? Step 5: If you use a composite metric, decompose it and check whether any critical component (especially safety) is being averaged away. Create a dashboard that shows component scores alongside the composite.

Key Takeaways

Metric selection is a design decision with the same weight as model architecture or data pipeline design, possibly more, because a wrong metric corrupts every downstream decision. Follow the seven-step checklist: define quality dimensions, assess reference availability, validate human correlation, test sensitivity and specificity, evaluate cost constraints, assess gaming resistance, and build composites carefully. Every metric must pass four non-negotiable checks: it measures a dimension you care about, it correlates with human judgment (tau > 0.40), it distinguishes quality levels (sensitivity > 80%), and it does not respond to irrelevant changes (specificity). Document gaming vulnerabilities for every metric and cover each with a complementary metric. Never reduce quality to a single number, always report components alongside composites. Revisit your metric selection quarterly, because your system, users, and quality bar all evolve. The right metric makes optimization productive. The wrong metric makes optimization destructive.