The email from the CEO landed at 4:52 on a Friday, and it contained one line and one attachment. The line read: "Vendor says their engine scores 61.3 BLEU on our content, competitors are in the 40s, let's sign Monday." The attachment was a two-page PDF with a bar chart, a smiling stock photo of a globe, and a footnote in six-point type. Dana, the head of localization for a medical-device maker shipping into nineteen markets, had seen this movie before. She knew, the way you know a card trick is a trick without knowing which hand hid the ace, that a single number chosen by the vendor and measured on data the vendor selected was not a reason to route German drug-label copy through an engine. She also knew that "trust me, it feels worse" would lose the argument on Monday. So she did the only thing that wins these arguments: she built a bake-off. She assembled a test set from her own content, scored three engines against it with a rubric a regulator would recognize, counted the one thing the vendor's chart hid, and walked into Monday with a decision that survived the room. This lesson is that bake-off, built slowly, because the difference between a real engine evaluation and a vendor demo is the difference between an operation that ships safely and one that ships a flipped dosage with a 61.3 next to it.

Why a Single Number Cannot Choose Your Engine

Start with the vocabulary, because half of every bad engine decision is made by people who do not know what the numbers they are quoting actually measure. Machine translation (MT) is any system that turns source-language text into target-language text with no human writing the words. Neural machine translation (NMT) is the dedicated-model form of MT, an engine trained specifically to translate, which is what most CAT and TMS pipelines have run for years. A large language model (LLM) is a general-purpose text predictor that translates as a side effect of its broad fluency, which makes it more fluent, more steerable by instruction, and more confidently wrong than a classic NMT engine. A CAT tool is the computer-assisted translation editing environment a linguist works in; a TMS is the translation-management system that orchestrates the pipeline around it. When this lesson says "engine," it means any of these: a hosted NMT service, an LLM behind a translation prompt, a custom-tuned model, or a hybrid. The evaluation method is the same regardless of what is under the hood, because you are scoring the output, not the architecture.

Now the number on the CEO's chart. BLEU (Bilingual Evaluation Understudy) is an automatic metric from the early 2000s that scores a machine translation by measuring how many word sequences (n-grams) it shares with one or more human reference translations, then applying a penalty for being too short. It produces a number, usually reported on a 0-to-100 scale, and it is fast, cheap, and language-agnostic, which is exactly why it became the default and exactly why it is dangerous as a purchasing criterion. BLEU answers one narrow question: how closely does this output overlap, word for word, with one particular human translation someone happened to write? It does not know what the words mean. It cannot tell a fluent correct sentence from a fluent sentence that flipped a negation, because both can share the same n-grams with the reference except for the one word that matters. A translation that drops the word "not" can score nearly identically to the correct one, because "not" is a single token in a long segment, and BLEU is counting tokens, not truth.

BLEU measures surface overlap with one human reference. It cannot distinguish a correct sentence from a fluent lie that shares the same words minus the one that inverts the meaning. A number that cannot see the error that ships a lawsuit cannot choose your engine.

There is a deeper structural problem than "BLEU is imperfect," and a strategist has to understand it to defend the decision. BLEU rewards resemblance to a reference, so it is biased toward whatever phrasing that one human translator chose. A different, equally correct translation that used synonyms or a different clause order scores lower, penalized for being right in a different way. Meanwhile the metric is blind to the failures that actually matter in your content: a swapped drug name, an inverted contraindication, a dropped dose unit, a mistranslated indemnity. These are the errors that fail files and end relationships, and BLEU assigns them almost no penalty because they are small in token terms and large only in consequence. So the vendor's 61.3 is not lying, exactly. It is answering a question you did not ask, on data you did not choose, using a yardstick that is structurally blind to your worst-case failure. The modern automatic metrics, the neural learned metrics that score meaning similarity rather than n-gram overlap, are meaningfully better than BLEU at correlating with human judgment, and they belong in your toolkit as a triage signal. But no automatic metric, learned or not, produces the one thing your decision requires: a count of how many critical, ship-blocking errors the engine made on content that looks like yours.

The Vendor Demo Is the Second Trap

If the automatic number is the first trap, the curated demo is the second, and it catches even people who know better than to trust BLEU. A vendor demo is a performance. The sentences shown are the sentences the engine handles well, in the language pair the engine is strongest on, on content that resembles the engine's training distribution. Nobody demos the segment where their engine drops a negation. The demo is not fraudulent; it is selection bias operating exactly as designed. The antidote is not to distrust the vendor personally. The antidote is to take the choice of test data out of the vendor's hands entirely and put it in yours, which is the whole premise of the bake-off this lesson builds. You never let the party being evaluated choose the exam. You write the exam from your own content, you grade it with your own rubric, and you count the failure mode that ends your relationships. Everything that follows is the machinery for doing that defensibly.

Building a Domain-Representative Test Set

An engine evaluation is only as trustworthy as the content you score it on, so the test set is the load-bearing decision, and it is the one most operations get wrong by grabbing whatever text is convenient. The principle is in the name: the test set must be representative of your domain, meaning it mirrors the content you will actually route through the engine in the mix and proportions you will actually route it. If eighty percent of your volume is user-interface strings and financial disclosures and you build a test set out of blog posts because they were easy to export, you have measured the engine on content you do not send it and stayed blind to the content you do. Representativeness is not a nicety; it is the entire validity of the exercise.

What Goes Into the Set, and in What Proportion

Assemble the set from your real production content, sampled to reflect three things at once: your content types, your risk tiers, and the hard cases that expose engine weaknesses. Content types means the genres you translate: UI strings, help articles, marketing copy, regulated instructions-for-use, contracts, release notes, support macros. Risk tiers means the consequence classification you already apply at intake, where a marketing string, a drug-label instruction, and an indemnity clause sit in different tiers because a fluent error costs a shrug, a life, and a lawsuit respectively. The test set must over-sample the high-consequence tiers relative to raw volume, because that is where a single engine failure is unaffordable, and an evaluation that scored the engine only on its average behavior across low-stakes bulk would tell you nothing about whether it is safe on the content that can hurt you. Weight the set toward the decisions that matter.

Hard cases are the segments you deliberately include because they stress the failure modes MT and LLMs are known to produce. Seed the set with negations and polarity ("do not exceed," "unless," "except where"), numbers and units and dosages, dates and ranges, named entities and product names that must not be translated, obligations and conditionals in legal language, approved terminology that the engine is tempted to render with a common synonym, and placeholder and tag structures the engine can break. You are not trying to trick the engine; you are trying to sample the segments where the difference between engines actually shows up, because on easy declarative sentences every modern engine looks equally competent. The signal lives in the hard cases, and if your test set is all easy cases you will conclude, wrongly, that the engines are interchangeable.

How Big, How Clean, and the Reference Question

Size is a trade between statistical stability and evaluation cost, because a human scores this set segment by segment and human evaluation is the expensive part. A few hundred segments per language pair per content type is a defensible working target for a purchasing decision: large enough that one lucky or unlucky segment does not swing the result, small enough that a qualified evaluator can score it carefully rather than skimming. Resist the urge to make the set enormous; a thousand carelessly scored segments are worth less than three hundred scored with full attention, because a rushed human evaluation reintroduces exactly the "looks fine" vibe the whole method exists to eliminate. The set must also be clean and stable: fixed source segments, frozen so every engine sees the identical input, with any personally identifying or confidential content handled per your data rules before it goes near a hosted engine.

One discipline that separates a rigorous test set from a sloppy one: keep it unseen and held out. If the segments you evaluate on are already in a public corpus, or worse, in the vendor's training data, you are measuring memorization, not translation, and the engine that "wins" may simply have seen the answer. Draw from your own recent production content, which no public engine has trained on, and keep the set private so it stays a valid exam for the next evaluation too. On references: BLEU and the learned automatic metrics need a human reference translation to compare against, so if you want an automatic triage signal you will need a trusted human translation of the set. But your primary method, the human error scoring that actually decides the purchase, is reference-free in the sense that the evaluator judges the engine output against the source segment and your termbase directly, the way a post-editor does, not against one pre-written "correct answer." That is a feature, not a gap: it means your decision does not inherit the bias of a single reference, and it measures the thing you care about, whether the output is correct and safe, rather than whether it matches one person's phrasing.

Scoring With MQM and ISO 5060

Once you have a test set, you need a rubric, and the rubric is where a real evaluation diverges permanently from a vendor demo. You are not going to read the output and form an impression. You are going to mark errors, one at a time, by category and severity, against a standard a regulator and a client recognize. That standard is MQM scored under ISO 5060. MQM is Multidimensional Quality Metrics, an analytic error-typology framework that classifies each translation error by dimension (what kind) and severity (how much). ISO 5060:2024 is the international standard, published in 2024, that formalizes an MQM-aligned model for the human analytic evaluation of translation output: it tells you to mark errors by dimension and severity against an agreed typology and to compute a score from those marks, rather than forming a holistic gut read. Together they turn "this engine feels better" into a defensible, reproducible number built from named errors.

The Four Dimensions and Three Severities

The evaluator marks each error in the engine output under one of four dimensions. Accuracy is whether the target means what the source means, judged against the source segment: mistranslation, omission (the dropped negation lives here), addition (a hallucinated number), and untranslated or over-translated content. Terminology is whether the output uses the client's approved term from the termbase consistently, a defect that can be technically accurate and still wrong because the approved term is itself a requirement. Locale conventions cover dates, numbers, decimal separators, units, currency, quotation style, and formality register for the target market. Fluency and style is whether the target language is itself well-formed, judged on the target alone: grammar, spelling, punctuation, register. This last dimension is the one holistic reading is good at and the one machines fail least, which is precisely the trap: an evaluation that drifts toward counting fluency slips is measuring the thing the engine reliably gets right while the accuracy failures that fail files slip past.

Each marked error also gets a severity. Critical is an error with severe consequences: a safety hazard, a legal or financial misstatement, an error that could cause harm or make the content unfit for use. A flipped dosage, an inverted contraindication, a negated obligation reversed, a wrong drug name. Major is an error that significantly affects usability or meaning but stops short of the catastrophic: a genuinely confusing mistranslation, a broken UI reference, an inconsistent regulated term. Minor is a real but low-consequence defect: an awkward-but-clear phrasing, a stylistic slip, a missing comma. The severities carry weights when you compute a score, with Critical weighted far above Major and Major above Minor, so that the arithmetic reflects the consequence hierarchy. And above the arithmetic sits the rule that governs everything: one Critical error fails the file, regardless of how clean the rest of the output looks, because a single shipped hazard is not offset by a thousand tidy sentences.

MQM under ISO 5060 turns "feels better" into a count of named errors by dimension and severity. The score is arithmetic, but the gate is categorical: one Critical fails the file no matter how high the rest of the numbers look.

Normalizing So Engines Compare Fairly

To compare engines you need the score expressed per unit of content, not as a raw error count, because a longer test set naturally accumulates more errors. The ISO 5060 approach normalizes: you sum the severity-weighted penalties across the evaluated segments and express the result against the volume evaluated, commonly as a penalty per some number of words, so that a quality score falls on a comparable scale regardless of set size. The exact penalty weights and the words-per-unit are parameters you set in your evaluation profile and, crucially, hold constant across all engines in the bake-off. The single most important fairness rule in the entire method is this: every engine is scored on the identical test set, by the same evaluator or evaluators, against the same typology, with the same severity weights, under the same instructions. The moment one engine gets an easier set or a more lenient marker, the comparison is dead. Freeze the exam, freeze the rubric, then let the engines differ only in their output.

The Metric That Actually Decides: Critical-Error Rate

Here is the heart of the lesson, the number the vendor's chart did not contain and the number Dana walked into Monday with. The critical-error rate is the frequency of Critical-severity errors in the engine's output, expressed per unit of content: Critical errors per thousand words, or per hundred segments, whichever unit your operation reports in. It is a different kind of measurement from a quality score, and understanding why is the difference between a strategist and someone who quotes metrics. A quality score is an average: it blends all the errors, weighted, into one number that describes typical behavior. The critical-error rate is a tail measurement: it counts only the catastrophic failures and ignores everything else. You need both, because they answer different questions, and the vendor showed you neither.

The reason the critical-error rate is the decisive metric for high-consequence content is that quality in localization is not governed by the average, it is governed by the worst case. An engine can have a beautiful average quality score, a wonderful learned-metric number, a high BLEU, and a critical-error rate that makes it categorically unusable for drug labels, because it flips a negation once every few thousand words and once is enough to kill someone or trigger a recall. Averages hide tails. A metric that blends the one lethal error into nine hundred and ninety-nine clean segments produces a reassuring number that describes a dangerous engine. The critical-error rate refuses to let the good segments buy back the lethal one. It isolates the failure mode that governs whether the engine is fit for your highest-consequence tier, and it is the metric you weight most heavily precisely for the content where a single failure is unaffordable.

A quality score is an average and averages hide tails. The critical-error rate is a tail metric: it counts only the failures that end relationships and refuses to let clean segments buy them back. For high-consequence content, the tail is the decision.

Reading the Two Numbers Together

In practice you carry both numbers for each engine and read them as a pair, because the combination tells you something neither tells alone. A high average quality score with a nonzero critical-error rate is the most dangerous profile of all, because the strong average lulls the buyer into trusting an engine that occasionally ships a hazard, and the strong average is exactly what the vendor will put on the chart. A slightly lower average quality score with a zero or near-zero critical-error rate is the safer engine for regulated content, even though it "loses" on the number the vendor prefers, because it does not produce the failure that fails files. The strategist's judgment is in weighting these by content tier: for low-stakes bulk you may legitimately prefer the higher-average engine and accept a small critical-error rate you will catch in post-editing; for the drug label you weight the critical-error rate so heavily that any nonzero value on your test set eliminates the engine from that tier outright. Same two numbers, opposite decisions, driven by consequence. That weighting judgment, made explicit and defensible, is the thing you are actually being paid for.

Terminology-Adherence Testing as Its Own Track

Terminology deserves a dedicated measurement track alongside the error scoring, because it is a distinct engine behavior with a distinct fix, and because it is one of the most common and most underestimated ways an engine drifts. Your client's termbase is their controlled glossary of approved terms: the exact rendering they require for a device name, a feature, a regulated substance, a legal concept. An engine, especially an LLM, is trained to prefer the most probable word, which is frequently a common synonym rather than the client's mandated term. It will render the approved "concentrator" as "device," the approved trade name as a descriptive phrase, or three different translations of one source term across a single document, all fluently, all technically almost-right, and all wrong against the requirement.

Measuring Adherence, Not Forming an Impression

Terminology-adherence testing measures this directly and numerically instead of trusting the general error pass to catch it. Build a subset of your test set where the segments contain known termbase entries, run each engine, and compute a terminology-adherence rate: of the approved terms that appeared in the source, what percentage did the engine render with the mandated target term. A ninety-eight-percent adherence rate and an eighty-percent adherence rate describe two very different engines for a client whose regulatory submission depends on one device name meaning one thing on every page, and that difference is invisible in an average quality score where a wandering term costs only a Minor or Major here and there. Measure it on its own, per engine, and read it as a first-class result.

Two refinements make this measurement decision-grade rather than superficial. First, test adherence with and without a glossary constraint supplied to the engine, because modern engines and LLM pipelines can be given the termbase as an instruction or a constraint, and the interesting result is not just the raw adherence but how much the engine's adherence improves when you feed it the glossary. An engine that scores poorly unconstrained but leaps to near-perfect when handed the termbase is a very different, and often better, purchasing proposition than one that ignores the glossary even when given it, because the first is controllable and the second is not. Second, weight terminology failures by the risk tier of the term: a drifting marketing adjective is a Minor annoyance, while a drifting regulated substance name or a legally defined term is a failure that can invalidate a document, so a raw adherence percentage should be read alongside where the failures fell. An engine at ninety-five-percent adherence whose five percent of misses are all on the highest-risk regulated terms is worse than an engine at ninety-two percent whose misses are all on low-stakes labels.

Comparing Per Language Pair and Per Content Type

Now the finding that overturns most naive engine decisions, and the reason a single overall winner is usually the wrong output of an evaluation. Engine quality is not a single property of the engine. It is a property of the engine on a specific language pair for a specific content type, and it varies enormously across both axes. An engine that is excellent at English into German for technical documentation can be mediocre at English into Japanese for marketing, and a different engine can be the reverse. The training data, the language-pair coverage, and the content style all interact, so the question "which engine is best" is malformed. The answerable question is "which engine is best for this language pair on this content type at this risk tier," and the honest answer is frequently different engines for different cells.

Build a Matrix, Not a Ranking

This is why the output of a serious bake-off is a matrix, not a leaderboard. You lay the engines against a grid whose rows are language pairs and whose columns are content types (or risk tiers), and you score each cell independently: quality score, critical-error rate, terminology-adherence rate, for that engine, in that cell. What emerges is almost never "Engine A wins." What emerges is "Engine A wins for German and French technical content, Engine B wins for Japanese and Korean, and no engine clears the critical-error bar for the regulated tier in any language, so that tier goes to full human translation." That last clause is not a failure of the evaluation; it is the most valuable thing the evaluation can produce, because it tells you where the machine is forbidden before a single hazard ships. A ranking hides this. A matrix reveals it, and it lets you route each cell to the engine or the workflow that actually serves it, which is a categorically better operating posture than sending everything through one contract because it "won" on average.

Engine quality is a property of the engine on a language pair for a content type, not a single score. The output of a real evaluation is a matrix that routes each cell to its best engine or to human translation, not a leaderboard with one winner.

The matrix also changes your commercial posture. Instead of a single all-or-nothing engine contract chosen on a vendor's number, you can run a multi-engine routing strategy where content flows to the engine that wins its cell, hold the regulated cells to human workflows, and re-run the evaluation periodically because engines change under you as vendors retrain them. It reframes engine selection from a one-time purchasing event into an ongoing measurement discipline, which is what it actually is, because the engine you bought is not the engine you will have in a year. The matrix is the artifact that makes that discipline legible to leadership and defensible to an auditor.

A Worked Engine Bake-Off, Start to Decision

Return to Dana and watch the method run end to end on a real decision, because the abstractions settle only when you see them produce an answer. Dana's operation ships a portable medical concentrator into nineteen markets. The content splits into three types she cares about: regulated instructions-for-use and label copy (highest risk tier), the device's companion app UI strings (medium tier), and marketing and support content (low tier). She is evaluating three engines: Engine A, a mature NMT service (the one on the CEO's 61.3 chart); Engine B, an LLM-based translation pipeline that accepts a glossary constraint; and Engine C, the incumbent she already runs. She scopes the bake-off to her two highest-volume language pairs, English into German and English into Japanese, because doing all nineteen at purchasing-decision rigor would cost more than the decision is worth, and these two cover the majority of her regulated volume.

The Setup, Held Constant

Dana builds one test set per language pair: three hundred segments each, drawn from her own recent production content so no engine has trained on them, proportioned to over-sample the regulated tier and deliberately seeded with hard cases, negations, dosages, dose units, the approved device term, do-not-translate product names, and date ranges. She freezes the source segments so all three engines see identical input. She runs each engine on the frozen set, twice for Engine B, once with the termbase supplied as a constraint and once without, to measure the glossary lift. Then a single qualified evaluator scores every output segment under MQM and ISO 5060: dimension, category, severity, source, target, fix, one structured row per error. Same set, same evaluator, same typology, same severity weights, same instructions, across all engines. The exam and the grader are frozen; only the engines vary.

The Results, Read as a Matrix

The numbers come back and they do not tell the story the CEO's chart told. On English into German, regulated tier: Engine A posts the highest average quality score of the three, comfortably, consistent with its shiny automatic number, but its critical-error rate is two Criticals across the German regulated segments, both dropped or softened negations in dosage instructions, either one of which fails a file. Engine B, constrained with the glossary, posts a slightly lower average quality score but zero Criticals on the German regulated segments and a terminology-adherence rate of ninety-seven percent constrained versus seventy-nine percent unconstrained. Engine C, the incumbent, sits between them on quality, one Critical, ninety-one percent adherence. On English into Japanese, regulated tier: the order inverts. Engine B, so strong in German, produces three Criticals in Japanese including a hallucinated dosage figure, while Engine A produces one and Engine C produces one; none of the three is clean on Japanese regulated content. On the app UI and marketing tiers in both languages, all three engines are close, low critical-error rates, and Engine A's higher average quality makes it genuinely attractive for the low-stakes bulk.

Read as a matrix, the decision writes itself, and it is nothing like "sign Engine A on Monday." For the German regulated tier, Engine B with the glossary constraint wins outright: zero Criticals and the highest terminology adherence, which is exactly the profile regulated content demands, even though it "loses" on the vendor's preferred average. For the Japanese regulated tier, no engine clears the bar: every one of them produced at least one Critical on content where a single Critical is unaffordable, so that cell routes to full human translation, and the evaluation has just prevented a shipped hazard that the 61.3 chart would have caused. For the app UI and marketing tiers, Engine A's strong average and low critical-error rate make it the sensible choice for the bulk, captured cheaply through MTPE. And Engine B's dramatic glossary lift (seventy-nine to ninety-seven percent) tells Dana something operational: this engine is controllable, it respects the termbase when fed it, which makes it the better long-term bet for any tier where terminology governs, provided the glossary is always supplied.

The Decision, and Why It Survives the Room

Dana walks into Monday not with a feeling but with a matrix. Her recommendation: route German regulated content to Engine B under a mandatory glossary constraint; route all regulated Japanese to human translation until an engine clears zero Criticals on a re-test; route app UI and marketing bulk to Engine A via post-editing; retire nothing about the incumbent until migration is scored; and re-run the bake-off every two quarters because the engines will change. When the CEO holds up the 61.3, Dana does not argue the number is wrong. She agrees it is real, then shows that it is an average on the vendor's data measured with a metric blind to negation drops, and that on her frozen test set that same Engine A dropped two negations in German dosage instructions, either of which ships a recall. The 61.3 does not survive contact with a critical-error count on representative content. The matrix does survive, because every cell in it is a named, severity-scored, reproducible result an auditor could re-derive. That is the difference between a vendor demo and an engine evaluation, and it is the entire reason the strategist, not the vendor, chooses the engine.

Key Takeaways

• BLEU measures word-sequence overlap with one human reference; it is fast and cheap but structurally blind to the failures that fail files, because it cannot distinguish a correct sentence from a fluent one that dropped the single word inverting the meaning. A vendor's single BLEU number answers a question you did not ask, on data you did not choose, with a yardstick blind to your worst case.
• Take the choice of test data out of the vendor's hands. Build a domain-representative test set from your own recent, unseen production content, proportioned to your real content types and risk tiers, over-sampling high-consequence tiers, and deliberately seeded with hard cases (negations, numbers, dosages, dates, approved terms, do-not-translate names) where engines actually differ.
• Score the output with MQM under ISO 5060: mark each error by dimension (accuracy, terminology, locale, fluency) and severity (Critical, Major, Minor), reference-free against the source and termbase the way a post-editor works, and normalize the severity-weighted penalties per unit of content so engines compare on a common scale.
• The critical-error rate, Critical errors per unit of content, is the decisive metric for high-consequence tiers, because a quality score is an average and averages hide tails. An engine with a beautiful average and a nonzero critical-error rate is the most dangerous profile there is, and the rule that one Critical fails the file governs the arithmetic.
• Carry the quality score and the critical-error rate as a pair and weight them by content tier: prefer the higher average for low-stakes bulk, but let any nonzero critical-error rate eliminate an engine from the regulated tier outright. That explicit, defensible weighting judgment is the strategist's actual contribution.
• Test terminology adherence as its own numeric track: the percentage of approved terms rendered with the mandated target term, measured both with and without a glossary constraint supplied to the engine, and weighted by the risk tier of the term. An engine's glossary lift (adherence gain when fed the termbase) tells you whether it is controllable.
• Engine quality is not a single property; it is a property of the engine on a language pair for a content type. The output of a real bake-off is a matrix that routes each cell to its best engine or to human translation, not a leaderboard with one winner, and it turns engine selection into an ongoing measurement discipline because engines change as vendors retrain them.
• In the worked bake-off, the vendor-favored engine posted the best average yet dropped two negations in German dosage instructions, the LLM engine won German regulated with zero Criticals and a strong glossary lift but failed Japanese, and no engine cleared the bar for Japanese regulated content, so that tier went to human translation, the most valuable output an evaluation can produce.