The decision had already been made before the meeting started, and the meeting was there to make it feel earned. A global consumer-health company was moving its regulatory and marketing localization onto a single machine-translation stack, and the head of localization had run what everyone in the room agreed was a rigorous pilot. Three engines had been tried on a batch of Spanish content over four weeks. The numbers on the slide were clean: engine B had post-edited fastest, lifted the pool from roughly 2,000 words a day to over 5,000, and cut the per-word cost by nearly half. The recommendation was engine B, the room was nodding, and the head of localization was thirty seconds from a signature on a two-year rollout across nine languages when the newly hired quality lead asked one question: "How many Critical errors did engine B produce, and how would we know?" The room went quiet, because the honest answer was that nobody had scored a single file. The pilot had measured throughput, cost, and edit distance, and every one of those numbers said engine B was the winner. Not one number said engine B was correct. The pilot had proven speed, and the company had been about to scale speed across its drug-safety content on the strength of a metric that would happily reward an engine that produced fluent, confident, catastrophic mistranslations faster than its rivals. This lesson is about the thing that meeting did not have: a proof of concept designed around a quality gate, so that the pilot proves the one thing scaling actually depends on, which is whether the output is safe to ship, and not merely how fast it is wrong.

The Pilot That Only Proved Speed

Start with the trap, because almost every localization pilot walks into it, and the ones that walk in confidently are the dangerous ones. A proof of concept (PoC), in this program, is a small, time-boxed, deliberately designed trial that scores one or more candidate engines or workflows against pre-declared criteria on representative content, so a scaling decision rests on evidence instead of a vendor demo or a gut feeling. The word "designed" is load-bearing. A PoC that is not designed still produces numbers, and the numbers it produces by default are the easy ones: words per hour, cost per word, edit distance, turnaround time. Those are the metrics a project-management tool hands you for free. They are all measures of speed and effort, and not one of them is a measure of correctness. A pilot that collects only the free metrics is not neutral. It is quietly biased toward the fastest engine, which is very often not the safest engine, because the two properties are unrelated and sometimes actively opposed.

Consider why speed and correctness pull apart. An engine that produces confident, fluent, over-smoothed output post-edits fast, because the post-editor reads clean prose, feels no friction, and moves on. That same fluency is exactly what hides the silent critical error, the flipped negation or the swapped dosage that reads perfectly and means the opposite. So the very property that makes an engine score well on throughput, its fluency, is the property that makes its worst errors hardest to catch. A speed-only pilot does not just fail to measure quality. It systematically rewards the engine whose failures are the most dangerous, because dangerous failures are the invisible ones, and invisible failures do not slow a post-editor down. The consumer-health company was about to standardize on engine B partly because it was smooth, without any evidence that the smoothness was accuracy rather than a well-upholstered lie.

A pilot that measures only speed, cost, and edit distance is not measuring quality by omission; it is actively selecting for the engine whose worst errors are the hardest to see. Fluency lowers post-editing friction and hides the silent critical error at the same time.

There is a second, subtler failure in the undesigned pilot, and it is the one that survives even after someone adds a quality check. The check gets added at the end, informally, as a spot-read: someone senior looks at a few files, says "these read fine," and signs off. "Reads fine" is not a quality gate. It is the absence of one dressed as diligence. It is not reproducible, because a different reader would spot-read different segments and reach a different verdict. It is not severity-scored, so a Critical error and a stiff adjective carry the same weight, which is to say no weight, because both dissolve into a general impression. And it is not set against a threshold declared in advance, so whatever the reader felt after reading becomes the standard, which means the standard was written to fit the result. The fix is not to spot-read harder. The fix is to build the quality gate into the design of the PoC before it runs, so the pilot measures correctness with the same rigor it measures speed, and the go/no-go decision has a real number on both axes.

The Two Axes a PoC Must Measure

Every localization PoC lives on two axes, and the entire discipline of this lesson is refusing to collapse them into one. The first axis is efficiency: throughput, cost per word, turnaround, post-editing effort. This axis is easy to measure, and every stakeholder already wants it, because it is where the savings live. The second axis is quality: the severity-scored error profile of the output, the critical-error rate, terminology adherence, locale correctness. This axis is hard to measure, requires a human evaluation, and is the one every undesigned pilot drops. A defensible PoC produces a coordinate on both axes for every candidate, so the decision reads "engine A: 4,800 words/day, error score 9 per thousand, zero Criticals" against "engine B: 5,400 words/day, error score 22 per thousand, three Criticals," and the fast engine's speed is visible right next to the fact that it is unsafe. Collapse the axes and you get a race to the bottom on price with the quality risk hidden inside the winning bid. Keep them separate and the trade-off becomes a decision a leadership team can actually make with its eyes open.

What a Quality Gate Actually Is

Before the PoC can be designed around a quality gate, the gate itself has to be defined precisely, because "quality gate" is a phrase people use loosely and then implement as a vibe. A quality gate is a pre-declared, severity-scored pass/fail rule that output must clear before it is allowed through a stage of the pipeline. In this lesson the stage is the scaling decision itself: the gate is what a candidate engine must pass on the pilot content before it earns the right to be scaled. The gate has three components, and all three must exist before a single file is scored, or the gate is retrofitted to the result and proves nothing.

The vocabulary has to be exact, because the gate is assembled from several standards-anchored pieces and the names carry weight. Machine translation (MT) is any system that renders text from one language into another with no human writing the words. A large language model (LLM) is a general-purpose text predictor that translates as a side effect of broad training and is fluent even when it is wrong. Machine-translation post-editing (MTPE), shortened to PE, is the workflow where a human edits machine output rather than translating from scratch. A segment is the unit a translation tool works in, usually a sentence, the row a linguist sees in a CAT tool (a computer-assisted translation tool, the linguist's editing environment). A termbase is the client's controlled glossary of approved terms. MQM is Multidimensional Quality Metrics, the analytic error-typology framework that classifies translation errors by dimension (the kind of error: accuracy, terminology, locale, fluency) and severity (how much it matters). ISO 5060:2024 is the international standard that formalizes an MQM-aligned model for the human analytic evaluation of translation output, including the Critical, Major, and Minor severity bands and the computation of a normalized error score. A Critical error renders content dangerous, unusable, or legally exposed: a flipped dosage, a dropped negation in a safety warning, an inverted indemnity clause. The critical-error rate is the count of Critical errors per unit of scored content, and it is the single most important number a localization PoC produces. Quality estimation (QE) is an automatic, model-produced confidence signal, distinct from the human MQM evaluation the gate is built on. Define each once, then use it precisely.

The Three Components of the Gate

The first component is the scoring model: the MQM/ISO 5060 error typology the output is scored against, with its dimensions, its severity bands, and its weights. This is the machinery that turns a marked error into a number. The classical MQM weighting most ISO 5060-aligned profiles start from is 1 penalty point per Minor, 5 per Major, and a large penalty (commonly 25) per Critical, with the total normalized to an error score per thousand words: penalty points divided by the scored word count, times one thousand. The scoring model must be the same one the production pipeline will use, because a PoC scored on a different rubric than production proves the engine against a standard it will never actually face.

The second component is the threshold: the declared pass line the score must beat. A quality threshold is a number, not an adjective, and it is set in advance. "Error score not exceeding 15 per thousand words" is a threshold. "Good enough" is not. The threshold is chosen from the content's risk tier and often mirrors the acceptance clause the eventual client contract will carry, because a PoC that passes an engine at a threshold looser than production will demand has proven nothing that survives contact with a real delivery.

The third component, and the one that makes a localization gate different from a generic quality check, is the absolute Critical rule: a single Critical error fails the file regardless of the normalized score. The normalized error score is an average, and averages forgive. One catastrophic mistranslation buried in ten thousand fluent words can produce a beautiful average and still be the error that kills a patient or loses a lawsuit. The Critical rule refuses to average away the one error that matters most. In a PoC this rule does double duty: it decides individual files, and across the pilot it makes the critical-error rate a first-class result, because an engine that produced even a few Criticals on representative content has told you something no throughput number can, which is that it will produce them at scale on content that ships.

A quality gate is three things declared before scoring begins: the MQM/ISO 5060 scoring model, a numeric threshold, and the absolute rule that one Critical error fails the file. A gate assembled after the results are in is not a gate; it is a rationalization.

Designing the PoC to Measure the Right Things

With the gate defined, the PoC is designed backward from it. The design decisions are not administrative details; each one is a place where a lazy choice quietly rigs the outcome toward the fast-and-unsafe engine, and a disciplined choice keeps the result honest. There are five design decisions that matter, and a PoC that gets all five right produces evidence a leadership team and an auditor can both trust.

Representative Content, Not Convenient Content

The first decision is what content the PoC runs on, and it is the one most often made by convenience. The temptation is to pilot on whatever content is easiest to get: a clean marketing batch, a set of UI strings, the material that happens to be sitting in the queue. That content is easy precisely because it is low-risk and low-complexity, which means the engine performs well on it, which means the pilot flatters every candidate and discriminates between none of them. A PoC must run on content that is representative of the production mix the engine will actually face, which for a consumer-health company means the pilot has to include the drug-safety copy, the dosage instructions, the contraindications, and the regulated label text, not just the campaign taglines. The representative sample deliberately includes the hard, high-liability content, because the whole point of the exercise is to find out how the engine behaves where behaving badly costs the most. Piloting an engine only on the content it is good at is not a test; it is a sales demo you ran yourself.

Representativeness has a second dimension beyond risk tier: content type and linguistic difficulty. The sample should span the genres the production stream contains, marketing prose, technical instructions, UI strings, legal boilerplate, because engines fail differently across genres. An engine that handles fluent marketing beautifully can fall apart on terse UI strings stuffed with placeholders, or on the dense nominal syntax of regulatory text. A single-genre pilot hides that variance. The sample must also cover the real language pairs in scope, because engine quality is wildly uneven across languages, and an engine that is excellent into Spanish can be mediocre into Japanese or Finnish. A PoC that tests one easy language pair and generalizes to nine is not a proof of concept; it is a hope wearing a lab coat.

The Sample Must Be Big Enough to Mean Something

The second decision is sample size, and it is where PoCs quietly lose their statistical footing. A critical-error rate computed on 300 words is noise: an engine can produce zero Criticals on a tiny sample by pure luck and one Critical on the next tiny sample by the same luck, and neither number tells you what the engine does at scale. The sample has to be large enough that the critical-error rate is a stable estimate rather than a coin flip, and large enough that each risk tier and each language pair inside it carries enough scored words to be judged on its own. There is no universal magic number, but the discipline is concrete: size the sample so that the highest-risk tier, the content where a Critical actually matters, has enough scored words that a single lucky-clean file cannot carry the verdict. A PoC that scores 5,000 representative words per language pair, weighted toward the high-risk content, tells you far more than one that scores 20,000 words of easy marketing in a single language. Volume in the wrong place is not evidence; volume in the high-risk place is.

Severity-Scored Evaluation, Not a Spot-Read

The third decision is how the output is evaluated, and this is where the PoC either has a quality gate or merely claims one. The evaluation must be a full severity-scored MQM/ISO 5060 pass: a qualified human evaluator reads each scored segment against the source, marks every error with its dimension and its exact span, assigns each a severity band against the content's risk tier, and the workflow weights and normalizes those marks into an error score per thousand words. The evaluation must be run identically across every candidate engine, same sample, same rubric, same evaluators, same profile, or the comparison is meaningless. If engine A is scored by a strict evaluator on the hard content and engine B by a lenient one on the easy content, the pilot compares evaluators, not engines. Blinding helps here: where possible, the evaluator scores the output without knowing which engine produced it, so no reputational halo or vendor relationship tilts the severity calls. The evaluation is the single most expensive part of the PoC and the single most important, because it is the only part that measures the axis every other metric ignores.

The Metrics the PoC Actually Collects

The fourth decision is which numbers the PoC records, and a well-designed pilot collects a specific, balanced set spanning both axes. On the quality axis, the load-bearing metrics are these. The critical-error rate: Criticals per thousand words, or per file, the number that most directly predicts a shipped disaster, and the one a speed-only pilot never sees. The normalized error score: the weighted MQM/ISO 5060 score per thousand words, the overall quality density against the threshold. Terminology adherence: the rate at which the engine honored the client's approved termbase versus drifting to a fluent synonym, measured as a percentage of term instances rendered correctly, because an engine that ignores the glossary will propagate the wrong term across every future project. The dimension breakdown: how the errors distribute across accuracy, terminology, locale, and fluency, because it tells you why an engine fails and whether the fix is a glossary, a prompt, or a hard no. On the efficiency axis, the pilot still records throughput (words per day per linguist), post-editing effort (edit distance or time), and cost per word, because the trade-off is real and leadership needs both coordinates. The design rule is that no efficiency metric is collected without its quality counterpart, so speed never appears on a slide without the correctness it bought or failed to buy sitting right beside it.

Success Criteria Declared in Advance

The fifth decision is the one that separates a PoC from a demo, and it is deceptively simple: the success criteria are written down before the pilot runs. This is the discipline that makes the whole exercise honest. Before any engine touches any content, the PoC design states the pass conditions in numbers: the error-score threshold each engine must beat, the maximum acceptable critical-error rate (very often zero on the high-risk tier), the minimum terminology adherence percentage, and the efficiency floor below which speed savings do not justify the switch. Written in advance, these criteria cannot be bent to fit whichever engine the organization already wanted. Written after the results, they will be, every time, because human beings rationalize toward the answer they were leaning toward. The consumer-health company's pilot had no written criteria, which is exactly why "engine B is fastest" was silently allowed to become "engine B wins." Had the design declared "no engine passes with a nonzero critical-error rate on drug-safety content," engine B's three Criticals would have failed it before its throughput ever entered the conversation.

Success criteria written before the pilot are a gate; success criteria written after are a rationalization. If the pass line is not on paper before an engine touches content, the organization will draw the line wherever the engine it already preferred happens to land.

The Critical-Error Rate Is the Headline

Among all the metrics a PoC collects, the critical-error rate deserves its own treatment, because it is the number the entire quality gate exists to surface and the number a speed-obsessed organization most wants to look away from. Everything else the pilot measures is a matter of degree: a slightly higher error score means slightly more post-editing, a lower terminology adherence means slightly more glossary work. The critical-error rate is not a matter of degree. It is a measure of how often the engine produces the specific class of error that ends relationships, triggers recalls, and lands in litigation. A single Critical shipped in a drug label is not a quality-of-degree problem to be traded against throughput; it is a categorically different kind of event, and the PoC must treat it that way.

This is why the critical-error rate is reported separately and never folded into the normalized average. Folding it in lets a strong average absorb a Critical: an engine with a gorgeous error score of 6 per thousand and one Critical looks, on the blended number, better than an engine with a score of 11 and zero Criticals, and that ranking is exactly backward for anyone whose content can hurt someone. The PoC readout puts the critical-error rate on its own line, at the top, before the average, because it is a different question with a different answer. The average asks "how much post-editing will this engine cost me?" The critical-error rate asks "how often will this engine try to kill someone in fluent prose?" Those are not the same question, and a pilot that answers only the first has answered the easy one and ducked the one that matters.

What a Nonzero Critical Rate Tells You at Scale

The temptation, when an engine produces one or two Criticals across a pilot, is to wave them off as post-editable: "the post-editor will catch them." Resist it, because the pilot is precisely the evidence that they often will not. Recall the failure mode: Critical errors in fluent output are the hardest to catch because they read perfectly. An engine that produced two Criticals in a 10,000-word pilot has not shown you two isolated slips; it has shown you a rate, and a rate scales. Two Criticals in 10,000 words is roughly one every 5,000 words, and a consumer-health operation shipping millions of words a year is now looking at hundreds of fluent, confident, catastrophic mistranslations flowing into its content stream, each one relying on a tired post-editor to notice the sentence that gives them no reason to slow down. The critical-error rate in a PoC is not a tally of the pilot's Criticals. It is a forecast of the production disaster, expressed in the only units a leadership team should be allowed to scale on.

Studies of LLM output on medical content found error rates around 59% on drug names, 60% on dates and times, and 66% on adverse events, every one delivered in grammatically perfect prose. Those numbers are why the critical-error rate is the headline and why zero is the only defensible target on high-liability content. An engine's average error score can be managed with more post-editing budget. Its critical-error rate on drug-safety content, at scale, is managed only by not scaling it there. The PoC is the cheapest possible place to learn that an engine has a nonzero critical rate, because learning it in production means learning it from a recall notice.

The Go/No-Go Decision

A PoC exists to produce a decision, and the decision has a specific shape that the design has been building toward from the first paragraph. It is a go/no-go: for each candidate engine or workflow, the pilot returns either "scale this" or "do not scale this," and the answer is read directly off the pre-declared criteria rather than argued in the room. The whole reason the criteria were written in advance is so that this decision is a comparison, not a negotiation. The evidence meets the threshold or it does not.

The decision is not a single yes/no, though. Because the PoC measures two axes and multiple risk tiers, the honest output is often tiered and conditional, and a mature strategist reports it that way. An engine can be a clear "go" on marketing and UI content, a clear "no-go" on regulated drug-safety copy, and a "conditional go, with full post-editing and a mandatory terminology override" on technical documentation. That is not indecision; it is the actual answer, and it maps directly onto the risk-tiered pipeline the organization already runs. The go/no-go from a well-designed PoC is not "which engine wins" but "for which content, in which language, at which effort level, does each engine clear the gate." Flattening that into a single winner is how the consumer-health company nearly put a fluency-optimized engine on its drug labels: the engine that wins on the easy content is allowed to inherit the hard content by default, and the default is where people get hurt.

The Decisions the Criteria Force

Written criteria force four clean verdicts, and naming them keeps the decision from sliding back into vibes.

• Go. The engine clears the error-score threshold, produces zero Criticals on every risk tier including the highest, meets the terminology-adherence floor, and its efficiency justifies the switch. Scale it, on the content and languages it was proven on, not beyond them.
• No-go. The engine produced any Critical on high-liability content, or missed the error-score threshold, or fell below the terminology floor by a margin no glossary work will close. Its throughput is irrelevant; a fast engine that fails the gate is a fast way to ship a liability. Do not scale it, and do not let its speed reopen the question.
• Conditional go. The engine clears the gate on some tiers or languages and fails on others, or clears it only with a heavier post-editing effort or a stricter terminology override than planned. Scale it precisely where it passed, at the effort level it required, and route the tiers it failed elsewhere. Write the conditions into the rollout, because a conditional go with unwritten conditions decays into an unconditional one within a quarter.
• Extend the PoC. The sample was too small in a tier that matters, or two engines are close enough that the difference is inside the noise, or a promising engine failed on something a glossary or prompt change might fix. Run more, do not guess. The cost of extending a PoC is a rounding error against the cost of scaling the wrong engine across nine languages of regulated content.

The discipline in all four is the same: the decision is read off the evidence against the pre-declared line, and the efficiency axis is never allowed to override a quality-axis failure. Speed is a tiebreaker between engines that both pass the gate. It is never a reason to lower the gate.

Speed is a tiebreaker among engines that pass the quality gate, never a reason to lower it. The moment throughput is allowed to argue an engine past a failed critical-error criterion, the PoC has stopped being a proof and become a permission slip.

A Worked PoC Design and Readout

Abstraction becomes a method only when you run it once, end to end, on something concrete. Return to the consumer-health company, and design the PoC the second meeting should have had before anyone said "engine B." The strategist's job is to write the design first, run it, and produce a readout that decides.

The Design, Written Before Anything Runs

Scope. Three candidate engines (call them A, B, and C) evaluated for a rollout across nine languages, piloted first on the two highest-volume pairs, English into Spanish and English into German, with a documented plan to extend to the remaining seven before any pair scales. The content stream is classified into three risk tiers already used in production: Tier 1 regulated drug-safety and label copy (dosages, contraindications, warnings), Tier 2 technical and support documentation, Tier 3 marketing and UI strings.

Representative sample. For each engine and each language pair, a stratified sample of 6,000 scored words: 2,500 from Tier 1, 2,000 from Tier 2, 1,500 from Tier 3, drawn from real production content, not a curated demo set. Tier 1 is deliberately the largest slice, because it is where a Critical costs the most and where the pilot most needs statistical footing. Every Tier 1 segment is scored at 100%; the lower tiers are sampled.

Scoring model. The production MQM/ISO 5060 profile: classical weights (Minor 1, Major 5, Critical 25), normalized to an error score per thousand words. Terminology adherence measured against the client termbase as a percentage of term instances rendered correctly. Evaluation run blind to engine identity by two calibrated evaluators, with a calibration pass on a shared sample first to confirm they agree within two points.

Success criteria, declared now. These are on paper before an engine runs:

• Tier 1: zero Criticals, non-negotiable, and error score not exceeding 8 per thousand words.
• Tier 2: zero Criticals, error score not exceeding 15 per thousand words.
• Tier 3: error score not exceeding 25 per thousand words; a single Critical fails the file but the tier tolerates a low nonzero rate given its low liability.
• Terminology adherence: at least 98% on Tier 1, at least 95% on Tiers 2 and 3.
• Efficiency floor: to justify a switch, the winning engine must lift throughput at least 40% over the current baseline; below that, the quality risk of changing engines is not repaid by the speed.

The Readout

The pilot runs. The evaluation is scored identically across all three engines. The readout puts the critical-error rate first, on its own line, per tier, before any average, and reports both axes side by side. The English-into-Spanish results come back like this.

• Engine A. Tier 1: zero Criticals, error score 6.4 per thousand, terminology adherence 99%. Tier 2: zero Criticals, error score 11.0, adherence 97%. Tier 3: error score 18.0, one Critical (a mistranslated promotional claim, low liability), adherence 96%. Throughput lift: 42% over baseline.
• Engine B. Tier 1: three Criticals (a dropped negation in a contraindication, a swapped dosage unit, an inverted "do not exceed" instruction), error score 5.1 per thousand, terminology adherence 91%. Tier 2: one Critical, error score 9.0, adherence 90%. Tier 3: error score 14.0, zero Criticals, adherence 93%. Throughput lift: 58% over baseline.
• Engine C. Tier 1: zero Criticals, error score 9.2 per thousand (misses the 8 threshold), terminology adherence 96%. Tier 2: zero Criticals, error score 13.0, adherence 95%. Tier 3: error score 22.0, zero Criticals, adherence 95%. Throughput lift: 31% over baseline.

Now read the decision off the criteria, and watch the speed-only pilot's answer invert. Engine B, the winner of the original undesigned pilot, is a hard no-go. It is the fastest engine by a wide margin, with a lift of 58% and the lowest Tier 1 error score of the three, because it is fluent and its output post-edits fast. And it produced three Criticals on drug-safety content, plus a fourth on Tier 2, plus terminology adherence below every floor. Its low error score is exactly the trap the lesson opened on: the fluency that makes its average look good is the fluency that hid three catastrophic errors from the fast, comfortable post-editors in the first pilot. Scaling engine B across nine languages of regulated health content would have been scaling a machine that produces a fluent, confident, life-threatening mistranslation roughly once per contraindication batch, faster than any rival.

Engine A is the go, on Tiers 1 and 2, in this language pair. It clears every threshold, produces zero Criticals on the high-liability tiers, meets the terminology floors, and its 42% lift clears the efficiency floor. Its single Tier 3 Critical on a low-liability promotional claim is noted, fails that one file per the rule, and is acceptable for the tier; the tier's own criteria tolerate a low nonzero rate. Engine A is not the fastest engine. It is the fastest engine that passes the gate, which is the only speed ranking that means anything.

Engine C is a conditional go, and an instructive one. It produced zero Criticals everywhere, which is the thing that matters most, but it missed the Tier 1 error-score threshold (9.2 against a ceiling of 8) and its 31% throughput lift falls below the 40% efficiency floor. The verdict: do not scale C as the primary engine, but it is the safe fallback for any tier where A later stumbles, and it is worth a short PoC extension to see whether a terminology override and a domain prompt pull its Tier 1 score under 8. It is the engine that is safe but not yet efficient enough, the exact case a written efficiency floor is designed to catch before it quietly becomes the default.

The engine that won the speed-only pilot was the one hard no-go in the designed one. Engine B was fastest and produced three Criticals on drug-safety content; the fluency that made its average look best was the fluency that hid its worst errors. The designed PoC did not just add a number. It reversed the decision.

What the Readout Hands Leadership

The readout that decides is not a slide with one winner. It is a two-axis table, engine by engine, tier by tier, with the critical-error rate on its own line at the top of every cell, the error score and terminology adherence beside it, and the efficiency numbers in their own column so the trade-off is visible rather than hidden. Under the table sits the go/no-go per engine per tier, read directly off the pre-declared criteria, and under that the conditions attached to every conditional go. That artifact does three things the original pilot's single slide could not. It defends the decision, because every verdict traces to a number and a criterion set before the engine ran, which survives a leadership challenge and a client audit alike. It routes the rollout, because the tiered go/no-go maps straight onto the risk-tiered pipeline: engine A on Tiers 1 and 2, engine C held as fallback, engine B excluded from regulated content entirely. And it prevents the disaster, because the one number the original pilot never collected, the critical-error rate on drug-safety content, is the number that stopped a fluent, fast, catastrophic engine thirty seconds before its signature. The PoC did not slow the organization down. It stopped it from scaling the wrong thing at speed, which is the only kind of speed that is ever worth stopping.

Key Takeaways

• A proof of concept (PoC) is a small, time-boxed, deliberately designed trial that scores candidate engines or workflows against pre-declared criteria on representative content. An undesigned pilot defaults to the free metrics (throughput, cost, edit distance), which measure only speed and effort and never correctness, and that default actively selects for the fluent engine whose worst errors are the hardest to catch.
• Every localization PoC lives on two axes that must never collapse into one: efficiency (throughput, cost, effort) and quality (severity-scored error profile, critical-error rate, terminology adherence). A defensible PoC produces a coordinate on both axes for every candidate, so a fast engine's speed always appears next to the correctness it did or did not buy.
• A quality gate is three things declared before any file is scored: the MQM/ISO 5060 scoring model with its weights, a numeric threshold (a number, not "good enough"), and the absolute rule that one Critical error fails the file regardless of the average. A gate assembled after the results is a rationalization, not a gate.
• Design the PoC backward from the gate with five disciplined decisions: run on content representative of the real production mix including the high-liability tiers, size the sample large enough that the critical-error rate is stable in the tiers that matter, run a full severity-scored MQM/ISO 5060 evaluation identically (and where possible blind) across every engine, collect balanced metrics so no efficiency number appears without its quality counterpart, and write the success criteria down before the pilot runs.
• The critical-error rate is the headline, reported on its own line before any average, because a Critical is a categorically different event from a quality-of-degree problem and averages forgive the one error that must never be forgiven. Two Criticals in a 10,000-word pilot is not two slips; it is a rate, and a rate scales into hundreds of fluent, confident, catastrophic mistranslations across a year of production.
• Zero is the only defensible critical-error target on high-liability content. An engine's average error score can be managed with more post-editing budget; its critical-error rate on drug-safety content at scale is managed only by not scaling it there, and the PoC is the cheapest place to learn that, because the alternative is learning it from a recall notice.
• The go/no-go decision is read directly off the pre-declared criteria, not argued in the room, and it is usually tiered and conditional rather than a single winner: an engine can be a go on marketing, a no-go on regulated copy, and a conditional go on documentation. Speed is a tiebreaker among engines that pass the gate, never a reason to lower it.
• The readout that decides is a two-axis, engine-by-engine, tier-by-tier table with the critical-error rate on its own line, the go/no-go and its conditions traceable to criteria set before the pilot ran. In the worked case, the engine that won the speed-only pilot (engine B, fastest, three Criticals on drug-safety content) was the one hard no-go in the designed one, and the designed PoC did not add a metric so much as reverse the decision it was about to make.