The pilot had been running for eight months, and everyone had quietly stopped calling it a pilot. It was just how the German support content got localized now. A global enterprise-software company had spun up a small experiment eighteen months earlier: one LLM engine, one content type, one language pair, a couple of enthusiastic post-editors, and a Slack channel where the results looked genuinely good. Throughput was up, the linguists were happy, the client complaints had not spiked. So the team kept feeding content into it, and kept meaning to write up the results, and kept not writing them up, because the experiment was working and nobody wanted to stop a working thing to document it. Then the annual quality-governance review arrived, and the head of localization walked in expecting to present a success and instead spent forty minutes unable to answer four questions. What exactly was the hypothesis this pilot was testing, and had it been confirmed or not? What was the critical-error rate on the content that had been running through it for eight months, measured against what baseline? Under whose sign-off had regulated adjacent content been allowed into a pilot that was never approved to touch it? And if leadership asked tomorrow to scale this across nine languages, what evidence pack would move with the decision, and would the governance group trust it? The honest answers were: there was no written hypothesis, nobody had scored a representative sample, the scope had crept without a control owner noticing, and there was no evidence pack at all, only a warm feeling and a Slack channel. The experiment had been quietly succeeding and quietly ungoverned for eight months, which meant it had been quietly accumulating risk the whole time, and now it could neither be scaled (no evidence) nor killed (too embedded) nor defended (no record). This lesson is about the discipline that prevents exactly this: running a localization-AI pilot so that it satisfies quality governance while it runs and produces evidence leadership can act on when it ends, instead of becoming a permanent, un-scaled, undocumented experiment that everyone depends on and nobody can account for.

A Pilot Is Not a PoC

Enterprises conflate two things that live at different altitudes, and the conflation is where the trouble starts. A proof of concept (PoC) is a small, time-boxed, deliberately designed trial that scores one or more candidate engines or workflows against pre-declared criteria on representative content, so a technical selection rests on evidence rather than a vendor demo. A PoC answers a narrow, comparative, technical question: which engine, on this content, produces the lowest critical-error rate against our threshold? You run it in a controlled corner, off to the side, and its output is a ranking. A pilot, at the enterprise level, is a different animal entirely. A pilot is a production-like, governance-observed, evidence-generating run of an already-selected approach, conducted at limited but real scope, to test whether that approach holds up under the conditions of the actual operation and to produce the evidence a leadership team needs to decide whether to scale, adjust, or stop. The PoC asks "does this work in a lab?" The pilot asks "does this hold in our house, watched by our governance, and can I prove it to the people who sign the budget?"

Three words in that definition carry the entire distinction, and each one is a discipline the enterprise-software company's eight-month drift violated. Production-like means the pilot runs inside, or close to, the real pipeline: the actual CAT tool (computer-assisted translation tool, the linguist's editing environment) and TMS (translation-management system, the platform that routes and tracks localization work), the real intake and routing, the real post-editors doing their real jobs under real deadlines, not a curated corner staffed by volunteers who care more than the average vendor linguist will. A pilot that runs in an artificial best-case setting proves that the approach works when everything is ideal, which is precisely the condition production never provides. Governance-observed means the quality-governance function is watching the pilot while it runs, not auditing it afterward: the scope, the risk tiers touched, the sign-offs, and the quality signals are visible to the people accountable for the operation's quality posture from day one. The eight-month experiment was governance-invisible, which is why regulated-adjacent content crept into it without anyone with authority noticing. Evidence-generating means the pilot is instrumented to produce a specific, pre-planned body of evidence, so that when it ends there is an evidence pack (the assembled, decision-ready record of what the pilot did, measured, and proved) ready to hand to leadership, rather than a scramble to reconstruct eight months of undocumented runs from memory and log files.

A PoC selects an approach in a lab; a pilot proves an already-selected approach in your house, under your governance, and produces the evidence to scale it. If your "pilot" is a controlled comparison off to the side, it is a PoC. If it is a production-like run that nobody is watching and nobody is documenting, it is not a pilot; it is an unsupervised experiment accumulating liability.

Why the Altitude Matters

The reason the distinction is not pedantic is that the two run at different stages of the same decision, and using one where you need the other produces a specific, predictable failure. Run a PoC when you need a pilot, and you scale a lab result into an operation whose real conditions the lab never contained: the engine that scored beautifully on a clean stratified sample meets the messy, deadline-compressed, scope-creeping reality of production and behaves nothing like the lab said it would. Run a pilot when you needed a PoC, and you have committed real production content and real post-editor time to an approach you never technically vetted, which means the pilot is testing an engine choice that a two-week comparison should have made for a fraction of the cost. The mature sequence is PoC first, pilot second: the PoC narrows the field to a defensible technical choice, and the pilot proves that choice under production conditions and generates the evidence to scale it. The enterprise-software company skipped straight to something that looked like a pilot, never ran a real PoC, and then let the pilot run without any of the three disciplines that make a pilot a pilot rather than a leak.

There is a governance consequence that follows directly. Because a pilot is production-like, it touches real content that ships to real users, which means the quality posture of the operation is actually exposed during the pilot in a way it never is during a PoC. A PoC's output is scored and discarded; a pilot's output ships. That single fact is why governance must observe a pilot from the start: the pilot is not a rehearsal, it is a limited-scope live performance, and if it drifts onto content it was never cleared for, the exposure is real, not simulated. Treating a pilot as a low-stakes experiment because it is "just a pilot" is the exact error that let eight months of regulated-adjacent content flow through an unvetted engine under nobody's sign-off.

The Pilot as a Hypothesis Test

The single mental shift that turns a drifting experiment into a disciplined pilot is this: a pilot is a hypothesis test, not an activity. An activity runs until someone stops it; a hypothesis test runs until it has confirmed or refused a specific, falsifiable claim, and then it ends. The enterprise-software company's experiment was an activity. It had no hypothesis, so it had no natural endpoint, so it ran forever, because an activity with no failing condition cannot fail and therefore cannot conclude. The first artifact of a real pilot, written before a single segment is post-edited, is a hypothesis stated precisely enough that it could be proven wrong.

A localization-AI pilot hypothesis has a rigid shape, and the rigidity is the point. It names the approach, the content, the language, the effort level, the quality bar, and the efficiency claim, all in numbers or declared conditions, so that the evidence at the end can be checked against it without interpretation. A vague hypothesis ("LLM post-editing will improve our German support localization") is untestable, because "improve" has no threshold, and an untestable hypothesis is how an experiment becomes permanent. A testable hypothesis reads like a contract:

"Using engine A with full post-editing on English-into-German support documentation (Tier 2), a two-person qualified post-editing team can sustain at least a 40% throughput lift over the current human-translation baseline while holding zero Critical errors and an error score at or below 12 per thousand words, with terminology adherence at or above 95%, measured on a stratified representative sample over an eight-week run."

Every clause in that sentence is load-bearing. It names one engine, because a pilot that tries three engines is a PoC wearing a pilot's clothes, and the engine choice should already be made. It names one content type and one risk tier, Tier 2 support documentation, because a pilot that spans regulated drug-safety copy and marketing taglines at once cannot produce a clean verdict on either. It names one language pair, because engine quality is wildly uneven across languages and a claim proven into German says nothing certain about Japanese or Finnish. It names the effort level (full post-editing), the quality bar (zero Criticals, error score at or below 12, terminology at or above 95%), and the efficiency claim (40% lift), each as a number the evidence will either clear or miss. And it names a duration, eight weeks, because a hypothesis test that does not end is not a test.

The Quality Gate Inside the Pilot

The quality bar in the hypothesis is not a vibe; it is a quality gate, a pre-declared, severity-scored pass/fail rule that output must clear, and it must be the same gate production will use, or the pilot proves the approach against a standard it will never actually face. The vocabulary has to be exact because the gate is assembled from standards-anchored pieces. MQM is Multidimensional Quality Metrics, the analytic error-typology framework that classifies each translation error by dimension (accuracy, terminology, locale, fluency) and severity (how much it matters). ISO 5060:2024 is the international standard that formalizes an MQM-aligned model for the human analytic evaluation of translation output, with the Critical, Major, and Minor severity bands and a normalized error score per thousand words. A Critical error renders content dangerous, unusable, or legally exposed: a flipped dosage, a dropped negation in a safety warning, an inverted indemnity clause. The critical-error rate is the count of Critical errors per unit of scored content. A segment is the unit a translation tool works in, usually a sentence. A termbase is the client's controlled glossary of approved terms. Quality estimation (QE) is an automatic, model-produced confidence signal, distinct from the human MQM evaluation the gate is built on. And the revised ISO 18587, expanded to cover AI and LLM "non-human translation output" and hybrid human-in-the-loop workflows, is the post-editing standard that insists the post-editor hold the same linguistic competence as a professional translator, which is why a pilot's post-editors must be qualified linguists, not the fastest available hands.

The gate inside a pilot carries the same three components a PoC's gate does, but with one added enterprise obligation. The three components are the scoring model (the MQM/ISO 5060 typology with its weights, classically 1 point per Minor, 5 per Major, 25 per Critical, normalized per thousand words), the threshold (a number, not "good enough"), and the absolute Critical rule (one Critical fails the file regardless of the average, because averages forgive the one error that must never be forgiven). The added enterprise obligation is that in a pilot the gate is live: it does not merely score a discarded sample at the end, it governs the content that actually ships during the run. If a file fails the gate mid-pilot, that file does not ship, the same way it would not ship in production. A pilot whose gate is only retrospective is not production-like, because production gates in real time, and a pilot that lets failed files ship "because it's just a pilot" has already proven that the approach is unsafe in exactly the way that matters.

Controls, Scope, and the Kill Criterion

A hypothesis alone does not make a pilot rigorous. Three further design elements separate a pilot that produces trustworthy evidence from one that produces a flattering anecdote: a control to compare against, a scope fence that governance can see, and a kill criterion that lets the pilot fail cleanly.

The Control: What Are You Comparing To?

An efficiency or quality claim is meaningless without a baseline to measure it against, and the most common way a pilot lies to its sponsors is by comparing the new approach to nothing. "The pilot hit 4,800 words a day" is a number, not evidence, until you know what the same team produced on the same content without the new approach. A pilot needs a control: the current-state process, measured on comparable content over the same period, so the lift and the quality delta are differences, not absolutes. The strongest control is a genuine parallel arm, where a portion of comparable content runs through the existing human-translation or existing-workflow process during the pilot window, scored on the identical rubric by the same calibrated evaluators, so the comparison is contemporaneous and blind to the outcome the sponsor wants. Where a full parallel arm is too expensive, a documented historical baseline, the measured throughput, cost, and error profile of the same content type before the pilot, is the minimum acceptable control. What is never acceptable is the absent control, because without it every number the pilot produces floats free of meaning, and "it feels faster" reenters through the side door the whole exercise was meant to lock.

The control also disciplines the quality claim, which is the half sponsors forget. It is not enough to show the pilot's error score; you must show it against the baseline's error score, because a new approach that is 40% faster and 30% worse on quality is not an improvement, it is a trade the organization must decide on with both numbers visible. The eight-month experiment had no quality baseline at all, so even its warm feeling was unanchored: the linguists felt productive, but nobody could say whether the critical-error rate was better, worse, or the same as the human process it had quietly replaced.

The Scope Fence Governance Can See

A pilot's scope is a fence, and the fence has to be visible to governance and enforced by intake, because the defining failure of the enterprise-software case was scope creep nobody could see. The scope declaration names, explicitly and in writing, the content types in scope, the risk tiers in scope, the language pairs in scope, and everything out of scope, and it names a control owner responsible for enforcing the fence at intake. "Tier 2 English-into-German support documentation only; Tier 1 regulated content is explicitly out of scope and must be routed to the existing human process; any content whose tier is ambiguous is escalated to the terminology lead before it enters the pilot" is a scope fence. "German support content" is not, because it has no tier boundary and no out-of-scope clause, which is exactly how regulated-adjacent material walks in.

The governance-observed property lives here. The scope fence is not a note in a project plan; it is a control the quality-governance group can see and check while the pilot runs. That means the pilot reports, on a regular cadence, what content actually entered it against what was declared in scope, so that drift is caught in week two rather than discovered in an annual review eight months late. A pilot whose scope is declared once and never reconciled against reality is governance-invisible in practice even if a governance group nominally exists, because the group cannot govern what it cannot see.

Scope creep is not a project-management annoyance in a localization-AI pilot; it is a quality-governance breach. Every tier of content that enters the pilot beyond its declared fence is content being localized by an approach that was never cleared for it, under nobody's sign-off, and the exposure is real because the pilot is production-like and the output ships.

Success and Kill Criteria, Both in Advance

A PoC needs success criteria written before it runs. A pilot needs that and its opposite: an explicit kill criterion, a pre-declared condition under which the pilot is stopped, not adjusted, not extended, but stopped and the approach sent back. The kill criterion is the single most important governance instrument in the entire design, because it is the mechanism that prevents the permanent un-scaled experiment. An experiment with only a success condition has no way to end in failure; it can only succeed or persist, and persistence disguised as patience is exactly how the enterprise-software company ran an ungoverned pilot for eight months. A kill criterion gives the pilot the ability to fail honestly and quickly.

Success and kill criteria are declared together, in numbers, before the pilot runs. Success might read: "over the eight-week run, on the representative sample, zero Critical errors on any file, error score at or below 12 per thousand words, terminology adherence at or above 95%, and a sustained throughput lift at or above 40% over the documented baseline." The kill criterion is not merely the logical negation of success; it is a set of conditions severe enough to warrant stopping mid-run rather than waiting for the end: "any Critical error on shipped content that reaches a user; a critical-error rate on the scored sample that projects to more than one Critical per 20,000 production words; terminology adherence below 90%; or evidence that the post-editors are clearing the gate only by working unsustainable hours." Between clean success and outright kill sits the honest middle, the conditional outcome: the approach clears the quality bar but misses the efficiency claim, or clears both on the easy half of the sample and struggles on the hard half, in which case the evidence pack says "scale with these conditions and constraints" rather than a flat yes. Naming all three in advance, success, kill, and the conditional band between them, is what makes the pilot's verdict a reading of the evidence rather than a negotiation in the room.

The Evidence Pack Leadership Can Act On

Everything the pilot does converges on one deliverable: the evidence pack, the assembled, decision-ready record that the governance group and leadership can trust and act on. A pilot that runs beautifully and produces no evidence pack has failed at its primary job, because the purpose of a pilot is not to run, it is to decide, and a decision needs evidence in a form the deciders can use. The eight-month experiment's fatal weakness was not that the approach was bad; it may well have been good. Its fatal weakness was that it could produce no evidence pack, so it could neither be scaled nor defended, and a genuinely good approach was rendered unusable by the absence of the record that would let anyone act on it.

An evidence pack is not a slide with a headline number. It is a structured artifact with a specific anatomy, and each part answers a question a skeptical governance member or a budget-holding executive will actually ask. The pack must contain, at minimum:

• The hypothesis, verbatim, as written before the pilot ran. This is first because it frames everything after it as either confirmation or refutation, and because a hypothesis that appears for the first time in the results section was written to fit them.
• The scope, as declared and as actually run. Both, side by side, so the governance group can see whether the fence held. A pilot that ran exactly its declared scope earns trust; one that drifted must account for the drift explicitly rather than hiding it.
• The quality axis, headed by the critical-error rate. The critical-error rate on its own line at the top, before any average, then the normalized error score against the threshold, the dimension breakdown (how errors distributed across accuracy, terminology, locale, fluency, because it explains why the approach failed or held), and terminology adherence. Every quality number is shown against the baseline, not in isolation.
• The efficiency axis, beside the quality axis, never instead of it. Throughput lift, post-editing effort, and cost per word, each against the documented baseline, so the trade-off is a visible coordinate rather than a buried assumption.
• The sample and method, so the numbers can be trusted. Sample size per tier and language, the scoring rubric, evaluator qualification and calibration, and whether scoring was blind to the approach. Evidence with no method is an anecdote with a decimal point.
• The verdict against the pre-declared criteria. Success, kill, or conditional, read directly off the criteria set in advance, with the conditions named explicitly for any conditional outcome.
• The governance trail. Who observed the pilot, who signed off on the scope, who evaluated, and where the failed files are recorded, so the whole run is reconstructable by an auditor or a governance member who was not there.

An evidence pack leadership can act on is not a claim; it is a reconstructable record. Its test is simple: could a governance member who never saw the pilot run read the pack, check every verdict against a criterion set before the pilot began, and reach the same decision without trusting anyone's memory? If yes, it is evidence. If it rests on "we saw it work," it is an anecdote, and an anecdote cannot be scaled or defended.

The Dual-Axis Discipline the Pack Enforces

The evidence pack's structure enforces the one discipline the whole program turns on: efficiency and quality are two axes that must never collapse into one number. Every efficiency metric in the pack sits beside its quality counterpart, so throughput never appears without the critical-error rate it did or did not buy. This is not a formatting preference; it is the structural defense against the failure the L4 PoC lesson opened on, where a speed-only readout let the fastest engine win because its fluency hid its worst errors. At the enterprise scale, the stakes of collapsing the axes are higher, because the pack is what leadership scales on, and a pack that reports a 58% throughput lift without the three Critical errors sitting on the line above it is not a summary, it is a trap that will put a fluent, fast, unsafe approach onto regulated content at nine-language scale. The dual-axis structure of the pack is what makes the trade-off a decision leadership makes with its eyes open rather than a risk it inherits blind.

Avoiding the Permanent Un-Scaled Experiment

The most common way an enterprise localization-AI pilot fails is not that it produces a bad result. It is that it produces no result at all, because it never ends. It becomes a permanent, un-scaled experiment: content flows through it indefinitely, it is neither formally scaled nor formally stopped, it accumulates dependency and risk in equal measure, and it sits in a governance blind spot because it has no status a governance group can act on. It is not a pilot and not production; it is a limbo, and limbo is where quality risk goes to hide. The enterprise-software company lived in this limbo for eight months. Naming the mechanisms that produce it lets you build the design elements that prevent it.

The permanent experiment is produced by four specific absences, and every one of them is prevented by an element already described in this lesson:

• No hypothesis, so no endpoint. An activity with no falsifiable claim cannot conclude; it can only run. The written hypothesis with a duration is the prevention, because it defines the moment the pilot is done.
• No kill criterion, so no way to fail. An experiment that can only succeed or persist will persist whenever it is merely mediocre, because "not yet good enough" reads as "give it more time" indefinitely. The pre-declared kill criterion is the prevention, because it defines the condition under which "more time" is refused.
• No evidence plan, so no decision. A pilot that is not instrumented to produce an evidence pack cannot hand leadership a decision, so the decision never gets made, so the pilot never resolves. The evidence-generating design is the prevention, because it makes the pack a planned output rather than an afterthought.
• No governance observation, so no owner of the ending. A pilot nobody is watching has nobody responsible for calling it done. The governance-observed property is the prevention, because it gives the ending an owner who is accountable for reaching it.

There is a fifth, quieter driver worth naming on its own, because it is the one that feels virtuous while it does the damage: the pilot that is succeeding is the hardest one to end. A failing experiment gets killed; a succeeding one gets fed, because stopping a working thing to formally evaluate and scale it feels like bureaucratic friction against a good outcome. But a succeeding pilot that is never converted to a governed, scaled operation is not a success. It is an ungoverned production process wearing a pilot's exemption from the controls production content is supposed to carry. The eight-month experiment was succeeding the entire time, and that is precisely why it was dangerous: its success bought it eight months of exemption from the scoring, the scope enforcement, and the sign-off that governed content is required to have. A pilot must have a defined conversion event, a moment where the evidence pack is presented and the approach is formally scaled into governed production, adjusted, or stopped. A pilot with no conversion event does not graduate; it squats.

A succeeding pilot with no conversion event is more dangerous than a failing one, because failure gets stopped and success gets fed. Every week a working pilot runs without being formally scaled into governed production is a week of production content localized under a pilot's exemption from the controls it should carry. The pilot must graduate or die; it must not be allowed to squat.

A Worked Pilot-to-Evidence Run

Method becomes real only when it is run once, end to end, on something concrete. Return to the enterprise-software company, and design the pilot the eight-month experiment should have been, from hypothesis to evidence pack, with the discipline that would have made it either scalable or dead within its window instead of a permanent limbo.

The Design, Written First

Precondition. A PoC has already run and selected engine A for this content type and language, so the pilot is not re-litigating the engine choice; it is proving engine A under production conditions and generating scaling evidence. This is the sequence the original team skipped.

Hypothesis, written before anything runs. "Using engine A with full post-editing on English-into-German Tier 2 support documentation, a two-person qualified post-editing team, working inside the production TMS under normal deadlines, can sustain at least a 40% throughput lift over the documented human-translation baseline while holding zero Critical errors, an error score at or below 12 per thousand words, and terminology adherence at or above 95%, over an eight-week run scored on a stratified representative sample."

Scope fence, visible to governance. Tier 2 English-into-German support documentation only. Tier 1 regulated and safety-adjacent content is explicitly out of scope and routes to the existing human process. Any content of ambiguous tier is escalated to the terminology lead before entering the pilot. The intake control owner reconciles actual-entered content against declared scope weekly and reports it to the governance group. This is the exact control whose absence let regulated-adjacent content drift in for eight months.

Control. A parallel arm: 20% of comparable Tier 2 content runs through the existing human-translation process during the same eight weeks, scored on the identical MQM/ISO 5060 rubric by the same two calibrated evaluators, blind to which arm produced each file, so the lift and the quality delta are contemporaneous differences, not comparisons to a warm memory.

Quality gate, live. Production MQM/ISO 5060 profile, classical weights (Minor 1, Major 5, Critical 25), normalized per thousand words, one Critical fails the file, and the gate governs shipped content in real time: any file that fails does not ship, exactly as in production. Two calibrated evaluators, blind to arm, calibrated on a shared sample first to agree within two points.

Success, kill, and conditional criteria, all declared now.

• Success: zero Criticals on every scored file across the run, error score at or below 12 per thousand, terminology adherence at or above 95%, and a sustained throughput lift at or above 40% over the parallel-arm baseline.
• Kill (stop mid-run): any Critical error reaching a user on shipped content; a scored critical-error rate projecting to more than one Critical per 20,000 production words; terminology adherence below 90%; or evidence the team is clearing the gate only through unsustainable overtime.
• Conditional: quality bar met but efficiency lift between 20% and 40%, or quality and efficiency both met on standard support content but not on the placeholder-dense UI-string subset, yielding a "scale with named constraints" verdict rather than a flat go.

The Run and the Readout

The pilot runs its eight weeks. The scope reconciliation catches, in week two, three files of installation-safety content (Tier 1-adjacent) that intake had misrouted into the pilot; the control owner pulls them, routes them to the human process, and logs the near-miss in the governance trail. That single caught event is the difference between this pilot and the original experiment, in which the same kind of content flowed unnoticed for months. At the end of the run, the evidence is scored identically across both arms, and the readout reports both axes with the critical-error rate first.

• Engine A, full PE arm (the pilot). Critical-error rate: zero across all scored files. Error score: 9.8 per thousand words (clears the 12 threshold). Dimension breakdown: errors concentrated in terminology and locale, light on accuracy, which points the fix at glossary and locale rules rather than at the engine's comprehension. Terminology adherence: 94% (misses the 95% floor by a point, driven by drift on a cluster of product-feature names). Throughput lift: 46% over the parallel-arm baseline. Post-editor hours: within normal load, no unsustainable overtime.
• Human-translation arm (the control). Critical-error rate: zero. Error score: 7.1 per thousand words. Terminology adherence: 98%. Throughput: the baseline, by definition.

Now read the verdict off the criteria, and notice how much richer the answer is than "it feels like it's working." The pilot is not a clean success and not a kill; it is a conditional go, and naming the conditions is the entire value of the exercise. The quality bar is almost met: zero Criticals (the non-negotiable, cleared) and an error score comfortably under threshold, but terminology adherence at 94% sits a point below the floor, and the dimension breakdown shows exactly why, drift on a specific cluster of feature names that a termbase enforcement pass and a domain prompt can close. The efficiency claim is met and then some, at 46%. So the evidence pack's verdict is precise: "Scale engine A with full post-editing on Tier 2 English-into-German support documentation, conditional on a terminology-enforcement override for the product-feature name cluster and a re-score confirming adherence at or above 95% before general rollout; hold the placeholder-dense UI-string subset for a follow-on pilot; Tier 1 remains out of scope." That is a decision leadership can act on, a rollout the risk-tiered pipeline can route, and a record an auditor can reconstruct.

What the Pack Hands Leadership, and the Ending It Forces

The evidence pack that lands on the governance group's table is the anatomy from earlier, filled in: the verbatim hypothesis at the top; the declared scope beside the actually-run scope, with the week-two near-miss logged as proof the fence held; the quality axis led by the zero critical-error rate, then the error score against threshold, the terminology gap with its dimension-level cause, all shown against the control arm's numbers; the efficiency axis beside it at 46% against the baseline; the sample, rubric, evaluator calibration, and blinding that make the numbers trustworthy; the conditional verdict with its named conditions; and the governance trail naming the observers, sign-offs, and the location of the pulled Tier 1 files. Every number traces to a criterion written before the pilot ran. A governance member who never watched a segment go by could read this pack and reach the same conditional-go decision without trusting anyone's memory.

And crucially, the design forces an ending. The eight-week duration, the kill criterion, and the conversion event mean this pilot cannot become the eight-month squat. At week eight the evidence pack is presented, and the pilot must resolve into one of three governed states: scaled into production under the named conditions, adjusted and re-piloted (as the UI-string subset will be), or stopped. There is no fourth state in which content quietly keeps flowing through an unvetted, ungoverned, undocumented process because nobody remembered to decide. The discipline did not slow the organization down. It did the opposite: it turned eighteen months of accumulating, undocumented, un-scalable risk into an eight-week run that produced, on schedule, a defensible decision the enterprise could actually act on, scale on, and stand behind in front of an auditor. That is the whole difference between running a pilot to evidence and running an experiment into limbo.

Key Takeaways

• A PoC and a pilot run at different altitudes: a PoC is a small, off-to-the-side comparison that selects an approach in a lab against pre-declared criteria, while an enterprise pilot is a production-like, governance-observed, evidence-generating run of an already-selected approach at limited real scope. The mature sequence is PoC first (to select), pilot second (to prove under production conditions and generate scaling evidence). Skipping the PoC commits real content to an unvetted approach; treating a pilot as a low-stakes experiment exposes the operation's real quality posture because a pilot's output ships.
• A pilot is a hypothesis test, not an activity. The first artifact, written before a segment is post-edited, is a falsifiable hypothesis naming one engine, one content type and risk tier, one language pair, the effort level, the quality bar (in numbers), the efficiency claim, and a duration. A vague hypothesis has no threshold and therefore no endpoint, which is how an experiment becomes permanent.
• The quality gate inside a pilot is the same three components as a PoC's (the MQM/ISO 5060 scoring model with weights, a numeric threshold, and the absolute rule that one Critical fails the file) but it must be live: it governs shipped content in real time, exactly as production does. A pilot that lets failed files ship "because it's just a pilot" has already proven the approach unsafe in the way that matters.
• Three design elements turn a flattering anecdote into trustworthy evidence: a control (a parallel arm scored on the identical rubric, or at minimum a documented historical baseline, because a lift against nothing is not evidence), a scope fence visible to governance and reconciled against actual intake weekly (because scope creep in a production-like pilot is a governance breach, not a project annoyance), and a kill criterion declared in advance alongside the success criteria.
• The kill criterion is the single most important governance instrument in the design, because it is what lets a pilot fail cleanly and prevents the permanent un-scaled experiment. Declare success, kill, and the conditional band between them all in numbers before the pilot runs, so the verdict is a reading of the evidence rather than a negotiation in the room.
• The evidence pack is the pilot's primary deliverable, and its test is reconstructability: could a governance member who never saw the run read the pack, check every verdict against a criterion set before the pilot began, and reach the same decision without trusting anyone's memory? Its anatomy is the verbatim hypothesis, declared-versus-actual scope, the quality axis led by the critical-error rate and shown against the control, the efficiency axis beside it, the sample and method, the verdict against pre-declared criteria, and the governance trail. Every efficiency number sits beside its quality counterpart so the axes never collapse into one.
• The permanent un-scaled experiment is produced by four absences (no hypothesis so no endpoint, no kill criterion so no way to fail, no evidence plan so no decision, no governance observation so no owner of the ending) plus a fifth quiet driver: a succeeding pilot is the hardest to end, because failure gets stopped and success gets fed. A succeeding pilot with no conversion event is more dangerous than a failing one, because it runs production content under a pilot's exemption from the controls governed content must carry. The pilot must graduate or die; it must not squat.
• In the worked run, the disciplined pilot caught misrouted Tier 1 content in week two (where the original experiment let it flow for months), returned a precise conditional-go (zero Criticals and a 46% lift, but terminology adherence a point under floor with a named, fixable cause), and forced an ending at week eight into a governed state (scale with conditions, re-pilot the UI subset, hold Tier 1 out) rather than an eighteen-month limbo. Running a pilot to evidence converts accumulating, undocumented, un-scalable risk into a scheduled, defensible decision the enterprise can act on and stand behind before an auditor.