A learning team runs a 30-day AI pilot and declares victory: build time dropped 70%, the executive sponsor is thrilled, the rollout to 40 courses is greenlit. Then a SME reviews the first scaled module and finds three invented policy thresholds. Then an accessibility auditor finds the AI-generated interactions are unreachable by keyboard. The pilot measured the one thing that was easy to measure, speed, and never tested the two things that actually decide whether the tool is safe to scale: is the content correct, and is the experience conformant. A pilot that measures only speed has not de-risked anything. It has accelerated the risk. Designing the pilot that catches what speed hides is the whole subject of this lesson.

The Pilot That Proves the Wrong Thing

The default learning-AI pilot is a speed pilot, because speed is visible, immediate, and flattering. You point the tool at a course, it produces a draft in a fraction of the usual time, and the time saved is trivial to put in a slide. The sponsor sees the number, feels the win, and approves the scale-up. The problem is that speed was never the question. The question was whether the tool produces content a SME will sign, an auditor will pass, and a learner can actually use, and none of those were tested. A speed pilot answers a question nobody was worried about and stays silent on the questions that end careers.

Here is the term that anchors the lesson. A pilot gate is a pass-or-fail criterion the pilot must clear before the tool is allowed to scale, defined before the pilot starts and tied to quality rather than speed. Why you care: without a gate, a pilot becomes a demo that ran for thirty days, and a demo cannot tell you whether the tool is safe at scale. With a gate, the pilot becomes an experiment with a falsifiable hypothesis: this tool produces verifiable, conformant content at a quality bar we defined, or it does not, and we find out on 2 courses instead of 40.

The reframe is simple and decisive. A pilot is not a speed test. It is a quality experiment that happens to be fast. The speed is real and worth noting, but it is the byproduct, not the result. The result is a yes-or-no answer to "can this tool produce content we can defend," measured against gates you set in advance, on a scope small enough that a failure is cheap and a SME has time to actually look.

A pilot that measures only speed has not reduced your risk. It has scaled your confidence ahead of your evidence, which is the most expensive thing a learning function can do.

The Two Gates a Real Pilot Must Have

Two gates separate a real pilot from a fast demo, and a pilot must pass both before scale. They map to the two failure modes that ship silently and surface expensively: the wrong fact and the inaccessible experience.

The Verification Gate: Is the Content Correct

The verification gate asks whether the AI-produced content is correct, claim by claim, against an approved source of truth. This is not a vibe check or a quick read. It is a SME tracing every load-bearing claim, every threshold, every procedure step, every citation, back to the source it must match, and recording the result. The pilot measures the verification yield: of the claims the tool produced, what fraction were correct as generated, what fraction were wrong, and what fraction were unverifiable because the tool could not show a source. Why you care: a tool that generates fast but produces a 10% error rate on regulated claims has not saved you time, it has manufactured a verification workload that scales with every course, and the pilot is where you measure that workload honestly before committing to it forty times over.

The critical discipline is that the verification gate is measured on regulated, load-bearing content, not on the easy prose. Any tool can write a passable introduction. The question is what happens at the lockout/tagout step, the policy threshold, the compliance citation, the safety procedure, the place where a confident wrong sentence becomes a liability. Pick the pilot scope so it includes that hard content, because a pilot run only on low-stakes material proves nothing about the high-stakes material you will inevitably run through the same tool.

The Accessibility Gate: Is the Experience Conformant

The accessibility gate asks whether the AI-generated experience meets WCAG 2.2 AA, tested the way an auditor will test it, not eyeballed on a designer's monitor. This means running the AI-generated content through a screen reader, navigating it by keyboard alone, checking the auto-generated captions for accuracy against the audio, and confirming the generated alt text actually describes the images. Why you care: AI generates the experience on the fly, so a tool can pass a static accessibility check and still produce an inaccessible interaction on the next generation. The gate is not "did the player pass once" but "does the generated content pass every time, and who is checking."

The bright line holds in the pilot exactly as it holds in production: an AI-generated experience that fails WCAG 2.2 AA does not ship, and a pilot that does not test conformance has not earned the right to scale. The pilot is the cheapest place to discover that the tool's auto-captions are 88% accurate on your accented technical narration, or that its generated drag-and-drop interaction is invisible to a screen reader. Discovering it on 2 courses is a finding. Discovering it on 40 is an incident.

Designing the Pilot Scope and Measures

A pilot that can pass or fail honestly needs a deliberate scope and pre-defined measures. A speed-only pilot is, at heart, a demo that happened to run for thirty days, and a demo cannot fail, which is precisely what makes it useless as evidence. The gated pilot is the opposite: it is built to be able to fail, on a small enough scope that a failure costs little and a SME has the time to actually look at every claim rather than skim for plausibility. The table below contrasts the speed-only pilot with the gated pilot across the dimensions that matter.

Three design choices make the gated pilot honest. First, set the gates and their thresholds before the pilot starts, so the sponsor cannot move the goalposts when the number is disappointing. Decide in advance that, say, regulated content must be 100% verifiable with zero unsourced claims, and conformance must be full WCAG 2.2 AA on every generated artifact, because those are the bars production will hold to. Second, scope small but representative: few enough courses that a SME can actually trace every claim, but including the hard, regulated content that the tool will eventually have to handle. Third, measure the verification workload, not just the build speed, because the real economics of the tool are build time minus verification time, and a tool that halves the build but triples the verification is not the win the speed number implies.

A Worked Example: Two Pilots, Same Tool

The same AI authoring tool runs through two pilots at two organizations evaluating it for a safety-training refresh.

Before (the speed pilot). Organization A runs the tool on a convenient onboarding course, measures a 70% reduction in build time, and scales to 40 courses on the strength of that number. The pilot never traced a single claim to a source and never ran the content through a screen reader. When the safety curriculum scales, a SME finds invented thresholds in three modules and the accessibility team finds keyboard-trapped interactions in a dozen. The remediation costs more than the build time saved, the rollout pauses, and the sponsor who approved on the speed number is the one explaining the incident. The pilot proved the tool was fast. It never proved the tool was safe, and fast-but-wrong is not a saving.

After (the gated pilot). Organization B runs the same tool on two courses chosen because they include regulated safety procedures and a policy threshold. A SME traces every load-bearing claim: the verification yield comes back at 82% correct, 11% wrong, 7% unverifiable because the tool could not cite a source. The accessibility gate finds auto-captions at 88% accuracy and one generated interaction unreachable by keyboard. The pilot has a clear finding: the tool is genuinely fast, but it requires a mandatory SME verification pass and a human caption-and-interaction check, and with those in place the net economics still favor adoption. Organization B scales with eyes open, the verification workflow built in from the start, because the pilot measured quality and surfaced the real cost before the commitment, not after.

The lesson is not that the tool was bad. It was the same tool in both pilots, with the same real speed. The lesson is that one pilot measured the easy thing and scaled blind, and the other measured the hard things and scaled informed. The 18% of claims that were wrong or unsourced existed in both pilots. Only one pilot was designed to find them, and finding them on 2 courses instead of 40 is the entire value of doing a pilot at all.

Reading the Yield and Deciding Honestly

A verification yield of 82% correct, 11% wrong, and 7% unverifiable is not a single verdict; it is three findings that point to three different decisions, and learning to read them apart is what turns a pilot into a real evaluation. The 11% wrong is a verification workload: those claims are catchable by a SME tracing them to the source, so they translate into a staffing cost you can price and a mandatory verification pass you can build in. The 7% unverifiable is a different and more serious signal, because it means the tool could not show a source at all, which is a source-transparency weakness that no amount of SME diligence fully fixes, since you cannot verify what the tool cannot ground. A pilot that reports only an aggregate "82% good" hides exactly this distinction, and the distinction is where the real decision lives.

The decision rule follows from reading the three numbers honestly. If the wrong fraction is catchable and the unverifiable fraction is near zero, the tool is adoptable with a built-in verification pass, and the pilot has priced that pass precisely. If the unverifiable fraction is meaningful, you either confine the tool to non-regulated content where unsourced claims carry less liability, or you reject it for regulated use entirely, because shipping unverifiable regulated claims at scale is indefensible regardless of how fast they were produced. And there is a halt condition that overrides any speed gain: if the verification yield on regulated content cannot reach the production threshold even with a human pass, or if generated experiences cannot be made WCAG 2.2 AA conformant on every generation, no workflow makes the tool safe, and adoption would knowingly ship undefendable or inaccessible content. The pilot's job is to make that halt condition visible before the commitment, not after.

An aggregate score hides the decision. Split the yield into correct, wrong, and unverifiable, because wrong is a workload you can staff and unverifiable is a liability you may not be able to.

Why the Gates Connect to the Rest of the Chapter

The pilot does not stand alone; it is the empirical test of promises the earlier chapter lessons gathered on paper. Procurement gated whether a tool can cite a source and prove conformance in principle, but a contract clause is a promise, and the pilot is where you find out whether the promise holds on your real regulated material. A tool that passed procurement on the strength of a citation feature but yields 7% unverifiable claims in the pilot has just told you the feature does not perform on your content, and that is a finding you could only get by running it. In this sense the verification gate and the procurement grounding gate reinforce each other: one checks the capability exists, the other checks the capability works, and you need both because a capability that exists on a slide and fails on your SOP is no capability at all.

The same logic ties the pilot to the privacy and the build-versus-buy decisions. The pilot runs on real learner-adjacent content, so the data terms must already be settled before it starts, or the pilot itself becomes an uncontrolled data exposure. And the path you choose, buy, compose, or build, determines how much of the verification and accessibility workload the pilot reveals will land on your function permanently, which is exactly the cost the pilot exists to measure. Run the four disciplines in isolation and they are four hoops. Run them as one sequence, procurement to privacy to pilot to path, and they become a single defensible adoption decision in which the pilot is the moment the abstract promises meet your actual content and either survive or do not. That meeting, on a small scope where failure is cheap, is the whole reason a strategist runs a pilot rather than a demo.

The Pilot Design Checklist

• Gates set first. Define the verification and accessibility pass criteria and their thresholds before the pilot starts, so they cannot move.
• Hard content in scope. Include regulated, load-bearing claims, not just easy prose, because that is what the tool must eventually handle.
• Real verification. A SME traces every load-bearing claim to a source and records the yield, not a plausibility read.
• Real accessibility testing. Screen reader, keyboard, caption accuracy, and alt-text accuracy on the generated content, not a glance.
• Measure verification workload. Net economics are build time minus verification time, so capture both.
• Pass or fail, not vibes. The pilot outputs a defensible yes or no against the pre-set gates, with speed noted as a byproduct.

Key Takeaways

• The default pilot measures speed because speed is visible and flattering, but speed was never the question; the question is whether the tool produces content a SME will sign, an auditor will pass, and a learner can use.
• A pilot is not a speed test, it is a quality experiment that happens to be fast; the speed is the byproduct, the defensible yes-or-no is the result.
• A real pilot has two gates: verification (is every load-bearing claim correct against an approved source) and accessibility (does the generated experience meet WCAG 2.2 AA the way an auditor tests it).
• Verification yield, the fraction of claims correct, wrong, and unverifiable, is the honest measure; a tool with a 10% error rate on regulated claims has manufactured a verification workload, not saved time.
• Test the gates on hard, regulated content, not easy prose, because a pilot run on low-stakes material proves nothing about the high-stakes content you will inevitably run through the same tool.
• Set the gates and thresholds before the pilot starts so the sponsor cannot move the goalposts, scope small but representative, and measure the verification workload because net economics are build time minus verification time.
• The bright line holds in the pilot: an AI-generated experience that fails WCAG 2.2 AA does not earn the right to scale, and discovering a conformance failure on 2 courses is a finding while discovering it on 40 is an incident.
• The iron rule threads through: AI assists and may be fast, but the human verifies the content and owns the accessibility decision, so the pilot exists to measure quality and surface the real cost before the commitment, not after.