Your Evaluation Is Only as Good as Your Dataset

In 2024, researchers discovered that several top-performing models on the Winogrande benchmark had been inadvertently trained on data contaminated with test set examples. Their scores were meaningless. This is the dataset construction problem in miniature: if your evaluation dataset is flawed -- contaminated, biased, ambiguous, or unrepresentative -- every conclusion you draw from it is suspect. Building high-quality evaluation datasets is a craft that sits at the intersection of data science, domain expertise, and annotation methodology. It is the most underleveraged skill in AI evaluation. This lesson teaches you how to construct eval datasets that produce reliable, actionable measurements of model quality.

Core Principles of Eval Dataset Design

Effective evaluation datasets follow five design principles. First, fitness for purpose: the dataset must measure the specific capability you care about, not a proxy. If you want to evaluate customer support quality, generic QA benchmarks are not sufficient. Second, distributional representativeness: the dataset's difficulty distribution, topic distribution, and input format should mirror your production traffic. If 40% of your production queries are about billing, 40% of your eval set should be too. Third, label quality: every example needs a defensible gold-standard answer. Fourth, adversarial coverage: include examples designed to exploit known model weaknesses -- edge cases, ambiguous inputs, inputs that trigger common failure modes. Fifth, contamination resistance: design your dataset to be difficult to memorize from internet scraping. Use paraphrased versions of common questions, novel scenarios, and time-stamped facts that did not exist during model training.

Sampling Strategies: Random, Stratified, and Adversarial

How you sample examples determines what your eval dataset can tell you. Random sampling from production logs gives you a representative picture of average model performance but underrepresents rare, high-impact failure modes. Stratified sampling ensures coverage across important dimensions: sample proportionally across topic categories, difficulty levels, and user segments. Adversarial sampling specifically targets model weaknesses: use error analysis from production to identify failure patterns, then construct examples that stress-test those patterns. The optimal eval dataset combines all three. A practical split: 60% stratified production-representative examples, 20% adversarial examples targeting known weaknesses, 10% examples from the tails of your difficulty distribution (very easy and very hard), and 10% from novel scenarios not yet seen in production. For dataset size, 200-500 examples per capability give you statistically meaningful results with 95% confidence intervals of plus or minus 3-4 percentage points.

Annotation Methodology: Guidelines, Training, and Quality Control

Annotation quality is the single largest source of variance in evaluation results. Start with annotation guidelines: a document of 5-15 pages that defines every label in your schema with concrete examples and boundary cases. For each category, include 3-5 clear positive examples, 3-5 clear negative examples, and 3-5 boundary cases with explanations. Train annotators in a calibration session: have them label a shared set of 30-50 examples, then discuss disagreements. After calibration, measure inter-annotator agreement using Cohen's kappa (two annotators) or Fleiss' kappa (three or more). Target kappa above 0.8 for factual tasks and above 0.6 for subjective tasks. If agreement falls below these thresholds, your guidelines are ambiguous -- revise them, do not just average the disagreements. Use a three-annotator setup with majority vote for production eval sets. Track individual annotator quality by embedding known gold-standard examples (10% of the annotation batch) and flagging annotators who miss them.

Choosing Your Label Schema: Binary, Likert, Comparative, and Rubric

The label schema you choose shapes what you can measure. Binary labels (correct/incorrect, acceptable/unacceptable) are the simplest and most reliable, with highest inter-annotator agreement. Use them when there is a clear right answer. Likert scales (1-5 quality rating) capture gradations but introduce noise: annotators interpret scale points differently. Anchor every point with a concrete example. Comparative labels (A is better than B) are more reliable than absolute Likert scores for subjective quality and are the basis of Elo-style rankings. Rubric-based labels score across multiple dimensions independently, giving you the richest signal but requiring the most annotator effort. For model selection decisions, binary pass/fail on critical requirements plus comparative ranking on quality dimensions gives you the most actionable data. Avoid free-text annotations as primary labels -- they are expensive to aggregate and difficult to compute metrics from.

Using LLMs to Generate and Augment Eval Datasets

LLM-generated synthetic data has become a standard tool for eval dataset construction, but it requires careful methodology. Use a strong model (Claude Opus, GPT-4) to generate candidate examples, then have human annotators verify and filter them. A generation prompt should specify: the capability being tested, the desired difficulty level, the format and length, and 3-5 reference examples. Generate 3-5x more candidates than you need and filter aggressively. Measure the quality of your synthetic pipeline by comparing model performance on synthetic-only vs. human-curated eval sets. If scores diverge by more than 5 percentage points, your synthetic data has systematic biases. Common pitfalls: LLM-generated questions tend to be more formulaic and less ambiguous than real user queries, creating artificially easy eval sets. Counteract this by seeding generation with real production examples and prompting for edge cases, unusual phrasings, and adversarial inputs.

Preventing and Detecting Data Contamination

Data contamination -- when evaluation examples appear in a model's training data -- is the most serious threat to evaluation validity. Prevention strategies: keep eval datasets private (do not publish them on GitHub or Hugging Face without considering contamination risk), use dynamically generated examples that did not exist before the model's training cutoff, and include canary strings or unique identifiers that let you detect if the dataset appears in training corpora. Detection strategies: compare model performance on your private eval set vs. public benchmarks -- suspiciously higher scores on public benchmarks suggest contamination. Run n-gram overlap analysis between your eval examples and known training corpora. Test with paraphrased versions of your eval questions: if a model answers the original perfectly but fails on a semantically equivalent paraphrase, it may have memorized the answer rather than learned the capability. The HELM framework's contamination analysis is a good methodological reference.

Versioning, Maintenance, and Dataset Lifecycle

Evaluation datasets are living artifacts that require systematic maintenance. Version every dataset release with semantic versioning: major version for schema changes or large-scale re-annotation, minor version for adding new examples, patch version for fixing individual label errors. Store datasets in version control alongside their annotation guidelines, annotator agreement metrics, and change logs. Schedule quarterly reviews: re-annotate a random 5% sample to check for label drift, review whether the difficulty distribution still matches production, and add examples covering new model capabilities or newly discovered failure modes. Retire examples that models have saturated (consistently scored correctly across multiple model generations). Track dataset provenance: for every example, record its source (production logs, synthetic generation, manual creation), annotation date, annotator IDs, and any edits. This provenance trail is essential for debugging evaluation anomalies and for EU AI Act Article 10 compliance.

Documenting Your Dataset: Data Cards and Datasheets

Every evaluation dataset needs structured documentation. Use the datasheet framework (Gebru et al.) or data cards (Pushkarna et al.) to document: motivation (why was this dataset created and for what evaluation purpose?), composition (how many examples, what categories, what demographic or topical distribution?), collection process (production logs, manual creation, synthetic generation?), annotation process (how many annotators, what guidelines, what agreement scores?), intended uses (what evaluation questions can this dataset answer?), limitations (what does it not cover, what biases might it have?), and maintenance plan (who will update it, on what schedule?). A well-documented dataset is reusable across teams and time. An undocumented dataset becomes a liability: nobody knows what it measures, whether its labels are trustworthy, or whether it is still relevant. Treat documentation as a required deliverable, not an afterthought.

Statistical Power: How Many Examples Do You Actually Need?

Eval datasets are often too small to support the conclusions drawn from them. The number of examples you need depends on the effect size you want to detect and the confidence level you require. For comparing two models, use a power analysis: to detect a 5-percentage-point difference in accuracy with 95% confidence and 80% power, you need approximately 400 examples per comparison. For rarer failure modes (occurring in less than 5% of cases), you need 500-1000 examples to estimate the failure rate with reasonable precision. Always report confidence intervals, not just point estimates: saying 'accuracy is 0.87 with a 95% CI of [0.84, 0.90]' is far more honest than 'accuracy is 87%.' For stratified analyses (performance by category, demographic subgroup), you need sufficient examples in each stratum -- 50 examples minimum per cell. This is why evaluation datasets for production systems typically need 1,000-5,000 examples: not because bigger is always better, but because stratified analysis with statistical rigor demands it.

Try This Now: Audit an Existing Eval Dataset

Take an evaluation dataset your team currently uses (or select a public benchmark like TruthfulQA or MMLU). Audit it against these criteria: (1) Re-annotate 20 random examples yourself. How many of the original labels do you agree with? If agreement is below 90%, you have a label quality problem. (2) Check distributional coverage: what topic categories are represented, and does this match your production traffic distribution? (3) Look for contamination risk: are these examples available on the public internet? Could a web-scraped training set include them? (4) Check for adversarial coverage: are there examples specifically designed to test failure modes, or is the dataset entirely 'happy path'? (5) Is there documentation (datasheet or data card)? This 30-minute audit almost always reveals actionable quality issues.

Key Takeaways

Evaluation datasets should follow five design principles: fitness for purpose, distributional representativeness, label quality, adversarial coverage, and contamination resistance. Use a mix of stratified sampling (60%), adversarial sampling (20%), and tail/novel examples (20%) for balanced coverage. Target inter-annotator agreement of kappa above 0.8 for factual tasks and above 0.6 for subjective tasks, using calibrated annotators with anchored guidelines. LLM-generated synthetic eval data is useful but systematically easier than real data -- always validate with human annotation and compare synthetic vs. human eval set results. Prevent contamination by keeping eval sets private, using dynamic generation, and testing with paraphrased variants. Version datasets with semantic versioning, review quarterly, and document with datasheets or data cards. Use power analysis to determine dataset size: 400+ examples for model comparison, 1,000-5,000 for stratified production evaluation.