Overview

Data leakage is the silent killer of AI evaluation. When an LLM has seen your test items during pretraining or fine-tuning, its evaluation scores reflect memorization, not generalization. This lesson gives you a systematic framework for detecting and preventing leakage: membership inference attacks, contamination testing, n-gram overlap scans, canary strings, and provenance tracking.

We cover three distinct leakage types (train-test, target leakage, and indirect signal leakage), quantitative detection methods with concrete thresholds, and the regulatory landscape (EU AI Act, NIST AI RMF) that increasingly treats leakage as a reportable evaluation defect.

Table of Contents:
- The Three Types of Data Leakage
- Why Leakage Invalidates Evaluation
- Membership Inference Attacks
- Contamination Testing for LLMs (n-gram, exact-match, canary)
- Indirect and Target Leakage Patterns
- Data Provenance Tracking
- Regulatory Requirements
- Remediation Playbook

The Three Types of Data Leakage

Leakage takes three distinct forms. Each requires different detection and remediation.

Type 1: Train-Test Contamination
Test items (or near-duplicates) appear in the training corpus. A model evaluated on MMLU after pretraining on Common Crawl has likely seen many MMLU questions verbatim. Detection: n-gram overlap, exact-match canaries, membership inference. Remediation: decontaminate via substring or perplexity filters; rebuild held-out sets post-hoc.

Type 2: Target Leakage
Features available at training time would not be available at inference time. Classic example: predicting loan default using a feature that is itself computed from the default event. Detection: feature-availability audit; temporal splits. Remediation: remove post-outcome features; enforce temporal holdout.

Type 3: Indirect or Signal Leakage
No direct overlap, but a proxy gives away the label. Examples: image metadata encoding the label; watermark tokens; dataset-specific artifacts (e.g., a benchmark always uses a specific punctuation pattern). Detection: shuffle test, adversarial probes, feature importance audits. Remediation: normalize formatting, strip metadata, use blind evaluation.

Most practitioners only check for Type 1. Types 2 and 3 silently inflate scores without tripping simple overlap detectors.

Why Leakage Invalidates Evaluation

Leakage breaks the fundamental claim of evaluation: that measured performance predicts real-world performance on unseen data.

The inflation effect. Contaminated benchmarks overstate capability. A 2023 study of GPT-4 on GSM8K found a 5-12 percentage-point gap between contaminated and held-out subsets. On MMLU, the gap can exceed 15 points for contaminated subjects.

Differential inflation. Leakage does not inflate all subsets equally. Models memorize popular, frequently copied items more reliably. This means leakage also corrupts comparative evaluation: Model A might look better than Model B not because it generalizes better but because A's training set happened to include more benchmark copies.

Downstream harm. A leaky benchmark leads to a leaky leaderboard. Deployment decisions, procurement, and regulatory submissions cite these numbers. In regulated sectors (medical, legal, financial), a leaked eval can be an auditable defect.

Rule of thumb. If the gap between a reference benchmark score and a freshly collected held-out score exceeds 5%, treat leakage as a strong hypothesis and investigate before publishing results.

Membership Inference Attacks

A membership inference attack (MIA) asks: was this specific example in the training set? MIAs are the canonical quantitative test for Type 1 leakage.

Loss-based MIA. Members have lower per-example loss than non-members. Given a candidate example x, compute the model's loss L(x). If L(x) is below a threshold τ (chosen on a calibration set with known members and non-members), classify x as member.

Calibrated / reference-model MIA. The naive loss test is confounded by intrinsic example difficulty. Calibrate by comparing L_target(x) to L_ref(x) from a reference model that did not train on x. The signal is L_ref(x) - L_target(x); large positive values indicate membership.

Min-K% Prob attack. For LLMs, compute the average log-probability of the K% least-likely tokens in x. Members tend to have fewer very-low-probability tokens than non-members. K=20% is a common default.

Metrics. Report AUC of the member vs non-member classifier. AUC > 0.6 indicates meaningful leakage; AUC > 0.75 is severe. Also report true positive rate at low false positive rates (TPR@1%FPR), which catches worst-case memorization even when average leakage is low.

Limitations. MIAs have high false-positive rates on near-duplicates and common text. Pair them with overlap tests and canaries for a defense-in-depth story.

Contamination Testing for LLMs

Direct contamination testing checks whether benchmark items appear (verbatim or near-verbatim) in training data or in model outputs.

Method 1: n-gram overlap. Compute the fraction of benchmark 13-grams (or 8-grams for shorter items) that appear in the training corpus. GPT-3 used 13-gram overlap; many follow-ups use 8-grams for sensitivity. Flag any item whose overlap exceeds 50% of its n-grams.

Method 2: exact-match prompting. Prompt the model with a prefix of the benchmark item and check whether it completes with the exact continuation. Use perplexity or BLEU-score thresholds. Works when you cannot inspect training data directly (closed-weight models).

Method 3: canary strings. Seed a unique, random canary string into your evaluation artifacts before publishing (e.g., 'BENCHMARK_CANARY_7f3a2b91'). Months later, prompt candidate models with a prefix of the canary and check whether it can complete the rest. A positive result is near-definitive evidence of training-data exposure.

Method 4: perplexity comparison. Compare perplexity on a benchmark to perplexity on a known-clean contemporaneous corpus. Abnormally low perplexity on the benchmark suggests memorization.

Method 5: shuffled/perturbed replay. Paraphrase benchmark items while preserving semantics. If accuracy drops sharply (>10pts), the original version was likely memorized.

Publish your contamination report as part of any benchmark release: which methods you ran, what thresholds you used, which items were flagged, and how you handled them.

Indirect and Target Leakage Patterns

Indirect leakage is subtler and more common than teams expect.

Pattern A: Feature computed from the label. In fraud detection, a feature like 'days_since_chargeback' is only computable if a chargeback occurred. Including it during training gives near-perfect train accuracy and zero deployment value. Audit each feature: is it available at inference time for the prediction target?

Pattern B: Temporal leakage. Using future data to predict the past. Random train/test splits leak temporal signal. Use strict temporal holdouts: train on t ≤ T, evaluate on t > T.

Pattern C: Group leakage. The same subject (patient, user, document) appears in both train and test. Models learn subject-specific shortcuts. Use group-aware splits (GroupKFold).

Pattern D: Annotator artifacts. Labels encode annotator idiosyncrasies (specific vocabulary, punctuation). Models learn the annotator, not the construct. Mitigation: blind annotation, multiple annotators per item, adversarial probes that predict the annotator from text.

Pattern E: Metadata leakage. File names, byte offsets, image EXIF tags encode the label or source. Strip metadata before evaluation.

Detection: the shuffle test. Randomly permute labels in training data. If the model still achieves above-chance accuracy on test, you have indirect leakage (the model learned a proxy unrelated to the true label).

Data Provenance Tracking

Provenance is the audit trail: for each evaluation item, where it came from, when it was collected, and what transformations it underwent.

Minimum provenance record.
- item_id (stable, globally unique)
- source (URL, internal system, synthesis pipeline)
- collection_date
- collection_method (scraped, human-written, LLM-synthesized)
- transformations applied (deduplication, normalization, filtering)
- license / consent status
- hash of final content (SHA-256)

Why hashes matter. If a benchmark item's hash appears in a training corpus's hash index, you have deterministic contamination detection. Hash every item before release and publish the hash list.

Versioning. Treat benchmark datasets like code: semantic versioning, changelogs, and immutable releases. When you discover contamination, release a 'v2' with contaminated items removed and a clear diff, rather than silently patching.

Cutoff dates. For every model under evaluation, record its training data cutoff. An item collected after the cutoff cannot be a Type 1 contamination risk (but can still have Type 2/3 issues).

Provenance enables defense. When a regulator asks 'how do you know this eval is clean?', you answer with the provenance ledger, canary results, and overlap scan report.

Regulatory Requirements for Leakage Prevention

Leakage prevention is moving from best practice to regulatory requirement.

EU AI Act (high-risk systems). Article 10 requires 'data governance and management practices' including examination for 'possible biases and relevant gaps or shortcomings'. Case law and guidance are treating train-test contamination as a reportable data-quality defect for high-risk classifiers.

NIST AI RMF (Measure 2.3). Requires evidence that evaluation data is representative and non-leaky. Explicit call-out: 'datasets used for measurement should not include items used in training'.

FDA (medical AI). 510(k) and De Novo submissions increasingly require evidence of temporal holdouts and patient-level (not record-level) splits. Group leakage in clinical AI can invalidate a submission.

ISO/IEC 5259-3. Data quality for analytics and ML. Requires documentation of provenance, preprocessing, and leakage checks.

Practical implication. Treat the contamination report as a regulatory artifact. Retain it, version it, and be prepared to produce it on audit. The cost of discovery (a regulator finds leakage you missed) greatly exceeds the cost of disclosure (you report it yourself with remediation).

Remediation Playbook

When you detect leakage, the response should be proportional and documented.

Step 1: Scope. Quantify the fraction of items affected. Is it 0.5% or 30%? Quarantine the flagged items; do not delete them yet.

Step 2: Re-score without flagged items. Report both the contaminated and the decontaminated scores. The gap is the leakage premium.

Step 3: Replace with fresh items. Generate new test items from post-cutoff sources. Re-validate the benchmark on a sample to ensure the new items measure the same construct (correlation > 0.8 with original construct expert ratings).

Step 4: Rotate private canaries. If canaries tripped, rotate to new canaries for the next release cycle.

Step 5: Publish a contamination advisory. Describe detection method, scope, and corrective action. This is required in regulated sectors and best practice everywhere.

Step 6: Root-cause the pipeline. Did you scrape benchmark data into a training corpus? Did a partner ship you data of unknown provenance? Fix the pipeline, not just the artifact.

Key takeaways.
- Assume leakage until proven otherwise; default to defense-in-depth.
- Always run at least n-gram overlap AND a membership-inference-style probe.
- Track provenance with hashes and immutable versioned releases.
- Treat contamination reports as regulatory evidence.
- Remediate by decontaminating, replacing, and publicly disclosing, not by silent patching.