One Size Fits None

A team evaluates their RAG-based document Q&A system using the same metrics they used for their chatbot. BLEU scores look reasonable. They ship. Within a week, users report that the system confidently answers questions using information not in the source documents, classic hallucination. BLEU never caught it because BLEU measures surface overlap, not faithfulness. The chatbot metrics were irrelevant for a grounded generation system. Every AI system type has a distinct failure mode profile, and your metrics must target those specific failures. A code generation system fails differently than a summarization system, which fails differently than a multi-turn conversational agent. Using generic metrics across system types is like using a thermometer to measure blood pressure. You get a number, but it tells you nothing about the condition you need to diagnose. This lesson maps the right metrics to seven common AI system types.

Chatbots and Conversational Agents

Conversational agents fail through irrelevance, inconsistency, and turn-level degradation. Your metrics must target each.

Core metrics:
- Turn-level relevance: Does each response address the user's actual question? LLM-as-judge with a relevance rubric, targeting Kendall's tau > 0.55 with human ratings.
- Multi-turn consistency: Does the agent contradict itself across turns? Use NLI between the current response and all previous responses. Flag entailment contradictions.
- Conversation completion rate: Does the user accomplish their goal? Track task completion through observational signals (explicit confirmation, session end without escalation).
- Engagement quality: Not just engagement duration, longer is not better if the user is frustrated. Measure resolution time: shorter = better for task-oriented agents.

Anti-metrics (metrics that mislead for this system type): BLEU/ROUGE (no single reference exists for open-ended conversation), perplexity (low perplexity does not mean helpful), response length (longer is not better).

Recommended stack: LLM-as-judge (relevance + helpfulness) + NLI consistency checker + observational (completion rate, escalation rate, regeneration rate).

Retrieval-Augmented Generation (RAG) Systems

RAG systems have a unique dual failure mode: the retrieval can fail (wrong documents) and the generation can fail (hallucinating beyond the documents). You need metrics for both.

Retrieval metrics:
- Recall@K: What fraction of relevant documents appear in the top K retrieved? Target: > 0.85 for K=10.
- Precision@K: What fraction of retrieved documents are actually relevant? Target: > 0.50 for K=10.
- Mean Reciprocal Rank (MRR): How high is the first relevant document ranked? Target: > 0.70.

Generation metrics (given correct retrieval):
- Faithfulness: NLI-based (SummaC, AlignScore) or LLM-based verification that every claim is supported by retrieved documents. This is the most critical RAG metric.
- Answer completeness: Does the response use all relevant information from retrieved documents? Measure as recall of key facts.
- Attribution accuracy: Can every claim be traced to a specific retrieved chunk? Use citation verification.

End-to-end metrics:
- Correct answer rate: Across retrieval + generation, does the user get a correct answer? Requires ground truth Q&A pairs.
- Hallucination rate: Percentage of responses containing claims not supported by any retrieved document. Target: < 5%.

Critical insight: always measure retrieval and generation separately. A low end-to-end score does not tell you which component to fix.

Code Generation Systems

Code generation has a rare advantage: you can objectively verify correctness by executing the code. Exploit this.

Functional correctness:
- pass@k: Generate k code samples; pass@k = probability that at least one passes all test cases. HumanEval and MBPP are standard benchmarks. pass@1 is the most practical metric (users see one suggestion).
- Test case pass rate: For each problem, what percentage of test cases pass? More granular than pass@k.

Code quality (beyond correctness):
- Cyclomatic complexity: Does the generated code have reasonable control flow complexity? Unusually high complexity suggests the model is generating convoluted solutions.
- Lint compliance: Does the code pass standard linters (pylint, eslint)? Measures adherence to style and best practices.
- Edit distance: How much does the user modify the generated code before accepting it? Lower is better. This is the strongest observational signal for code gen quality.

Security metrics:
- Static analysis findings: Run SAST tools (Semgrep, CodeQL) on generated code. Count critical and high vulnerabilities per 1K generations.
- Dependency safety: If the code imports packages, are they real packages or hallucinated names? Hallucinated package names are a supply chain attack vector.

Recommended stack: pass@1 (correctness) + lint compliance (quality) + static analysis (security) + edit distance (observational).

Summarization Systems

Summarization evaluation is a mature subfield with well-established metrics, but most teams still use them incorrectly.

Faithfulness (most critical):
- SummaC / AlignScore: NLI-based metrics that check whether summary claims are entailed by the source. This is non-negotiable, an unfaithful summary is worse than no summary.
- QAFactEval: QA-based faithfulness checking. Generates questions from the summary, checks if the source answers them consistently.

Coverage:
- ROUGE-L: Measures longest common subsequence between summary and reference. Useful for extractive summarization but weak for abstractive. Treat as a sanity check, not a quality metric.
- BERTScore: Semantic similarity between summary and reference. Better than ROUGE for abstractive summaries (tau ~ 0.45 vs ROUGE's 0.30 with human judgment).
- Key fact recall: Define the 5-10 most important facts in the source. What fraction appear in the summary? This manual metric has the highest correlation with human quality ratings.

Conciseness:
- Compression ratio: Summary length / source length. Optimal depends on use case but typically 0.10-0.30.
- Information density: Key facts per sentence. Higher = more concise and informative.

Recommended stack: SummaC (faithfulness) + key fact recall (coverage) + compression ratio (conciseness) + LLM-as-judge (overall quality).

Classification and Extraction Systems

Classification and extraction tasks have clear ground truth, making evaluation more straightforward, but the choice of metric still matters enormously.

Classification:
- Accuracy: Only appropriate for balanced datasets. With 95% negative and 5% positive, a classifier that always says 'negative' gets 95% accuracy.
- Precision / Recall / F1: Essential for imbalanced classes. Choose your emphasis based on the cost of errors. False positives costly (spam filter blocking real email) → optimize precision. False negatives costly (cancer screening missing tumors) → optimize recall.
- PR-AUC vs ROC-AUC: For imbalanced datasets, PR-AUC is more informative than ROC-AUC. A model can have ROC-AUC of 0.95 while PR-AUC is 0.40 on a heavily imbalanced dataset.
- Calibration (Brier score): Does the model's confidence match its accuracy? If it says 80% confident, is it correct 80% of the time? Essential for downstream decision-making.

Extraction (NER, slot filling, structured output):
- Exact match: The extracted value matches ground truth exactly. Strict but appropriate for structured fields (dates, IDs, amounts).
- Partial match (token F1): Overlap between predicted and ground truth spans. More forgiving for entity boundaries.
- Schema compliance: Does the extracted output conform to the expected schema? Validate JSON structure, field types, and required fields before measuring accuracy.

For LLM-based classification and extraction, always add a refusal audit: does the model appropriately decline when the input does not contain extractable information, rather than hallucinating a plausible-sounding answer?

Multi-Agent and Agentic Systems

Agentic AI systems, where models plan, use tools, and execute multi-step tasks, require fundamentally different evaluation from single-turn generation. Standard metrics do not capture the unique failure modes.

Task completion metrics:
- Success rate: Does the agent complete the assigned task? Binary, but the most important metric. Measure on a diverse task suite covering easy, medium, and hard tasks.
- Step efficiency: How many steps does the agent take compared to the optimal path? Ratio > 2.0 suggests inefficient planning.
- Tool use accuracy: When the agent calls a tool, does it pass correct parameters and handle the result correctly? Measure as: correct tool calls / total tool calls.

Safety and control metrics:
- Scope compliance: Does the agent stay within its authorized actions? Count out-of-scope tool calls or file accesses.
- Error recovery rate: When a tool call fails, does the agent recover gracefully or enter a failure loop?
- Cost per task: Total API calls, tokens, and tool invocations per completed task. Optimize for cost-effectiveness, not just success.

Trajectory-level metrics:
- Plan coherence: LLM-as-judge evaluation of whether the agent's plan is logical and efficient before execution.
- Intermediate state correctness: At each step, is the agent's working state consistent with the task requirements?

Recommended stack: Success rate (primary) + step efficiency + tool use accuracy + scope compliance + cost per task. Evaluate on a benchmark suite with 100+ diverse tasks at three difficulty levels.

Translation and Cross-Lingual Systems

Machine translation has the most mature evaluation ecosystem in NLP, but the shift to LLM-based translation demands updated approaches.

Traditional metrics:
- BLEU: The classic n-gram overlap metric. Still useful as a sanity check and for comparability with published results. But BLEU has known weaknesses: it penalizes valid paraphrases, ignores fluency, and correlates poorly with human quality for high-resource language pairs (tau ~ 0.30).
- chrF: Character-level F-score. More robust than BLEU for morphologically rich languages (Finnish, Turkish, Arabic). Consistently higher correlation with human judgment.
- COMET: Neural metric trained on human quality judgments. State-of-the-art correlation with human ratings (tau ~ 0.55-0.65). Should be your primary automated metric for translation in 2025-2026.

LLM-era metrics:
- GEMBA: Uses GPT-4 as a translation quality evaluator. Achieves segment-level correlation comparable to COMET at higher cost. Useful for low-resource language pairs where COMET's training data is sparse.
- Error span annotation: LLM-as-judge identifies specific error spans and categorizes them (accuracy, fluency, terminology, style). More actionable than a single quality score.

Critical for LLM translation: Measure hallucination rate, LLMs sometimes generate fluent translations that add content not in the source. Use cross-lingual NLI or back-translation consistency to detect this. Also measure language contamination: does the translation accidentally include words from the source language?

The Complete Decision Matrix

Use this reference table to quickly identify your starting metric set based on system type.

| System Type | Primary Metric | Secondary Metrics | Must-Have Check | Anti-Metric |
|---|---|---|---|---|
| Chatbot | LLM-judge relevance | Consistency, completion rate | Multi-turn coherence | BLEU/ROUGE |
| RAG | Faithfulness (NLI) | Recall@K, answer correctness | Hallucination rate < 5% | Perplexity |
| Code Gen | pass@1 | Lint compliance, edit distance | Security scan | BLEU |
| Summarization | SummaC faithfulness | Key fact recall, compression | Source attribution | ROUGE alone |
| Classification | F1 (or PR-AUC) | Calibration (Brier score) | Refusal audit | Accuracy alone |
| Multi-Agent | Success rate | Step efficiency, tool accuracy | Scope compliance | Token count |
| Translation | COMET | chrF, error span analysis | Hallucination check | BLEU alone |

This matrix is a starting point. Every deployment needs customization based on your specific quality dimensions, failure modes, and user population. But if you are using an anti-metric as your primary evaluation signal, you are measuring the wrong thing.

Try This Now

Identify which system type best describes your AI application (it may be a hybrid, pick the closest). Look up its recommended metric stack in this lesson. Now audit your current metrics against the recommendation. Step 1: List your current metrics. Step 2: For each recommended metric you are NOT using, determine why. Is it a deliberate choice (justified by your specific use case) or an oversight? Step 3: For each metric you ARE using that appears in the 'anti-metric' column, evaluate whether it is genuinely informative for your application or whether you are using it out of habit. Step 4: If you are building a RAG system and not measuring faithfulness, implement SummaC or an LLM-based faithfulness check this week. It is the single highest-impact metric addition you can make. Step 5: If you are building an agentic system, add a scope compliance check before your next release, unauthorized tool calls are the most dangerous failure mode for agents.

Key Takeaways

Different AI system types have fundamentally different failure modes, and your metrics must target the specific failures that matter for your system. Using generic metrics across system types is a reliability hazard. For chatbots, measure relevance and multi-turn consistency, not BLEU. For RAG systems, faithfulness is the non-negotiable primary metric, measure retrieval and generation quality separately. For code generation, exploit the advantage of executable verification with pass@k, but add security scanning and edit distance. For summarization, faithfulness (SummaC) and key fact recall matter far more than ROUGE scores. For classification on imbalanced data, use PR-AUC and F1, never accuracy alone. For agentic systems, success rate and scope compliance are critical metrics that traditional NLG evaluation ignores entirely. For translation, COMET has replaced BLEU as the primary automated metric. Always check the anti-metric column, if your primary metric appears there, you need to change course immediately.