โ†
AI for Instructors & Learning Professionals
Visionary ยท M17 ยท lesson 17 of 19 ยท queued
Preview โ€” browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll โ†’
The Verify-by-Design, Measure-by-Default Learning Function
๐Ÿ“–
now learning

The Verify-by-Design, Measure-by-Default Learning Function

15 min

Two learning functions ship the same compliance refresh in 2027, both using the same AI tooling. The first builds fast, then stops the line for a two-week verification-and-measurement scramble at the end: someone chases the SME for sign-off, someone hand-builds an evaluation plan nobody budgeted time for, someone discovers the AI-narrated video never got a caption review. The second ships in four days with a SME sign-off log, a WCAG 2.2 AA conformance report, and a Level 3 behavior measure already wired in, because verification and measurement were not steps at the end. They were properties of the build from the first minute. Same tools, same people, same speed on the production. The difference is where rigor lives. This lesson is about moving it from the end of the process into the design of the process, so speed and rigor stop being a trade-off and become one system.

The Bolt-On Tax That Kills the Speed

The most common failure in an AI-accelerated learning function is not a hallucinated fact. It is subtler and more expensive: the function captures the production speed and then loses most of it back to a verification-and-measurement scramble bolted onto the end of every build. The demo promised six weeks down to an afternoon. In practice the afternoon of production is followed by two weeks of chasing sign-offs, retrofitting accessibility, and inventing an evaluation plan under deadline, and the net saving is a fraction of what was advertised. Worse, because the rigor is a scramble, it is inconsistent: some builds get a real SME review and some get a rushed one, some ship with a Level 3 measure and most ship with a smile sheet, and nobody can predict which.

Call this the bolt-on tax. It is what you pay when verification and measurement are afterthoughts rather than defaults. And it produces exactly the wrong political dynamic inside the function: because rigor is slow and painful and inconsistent, the organization starts to experience verification as the enemy of speed, and the pressure builds to relax it. That is the road to shipping a confident wrong module at scale. The bolt-on tax does not just cost time. It quietly makes the case for cutting the one thing that must never be cut.

The way out is not to choose speed or rigor. It is to notice that the trade-off is an artifact of when the rigor happens. A verification pass that is retrofitted onto a finished build is slow, because it is fighting a build that was never designed to be verified. A verification pass that is designed into the build, where the source is captured as the draft is generated and the sign-off is a gate the work cannot pass without, is fast, because the artifacts it needs already exist. The same is true of measurement: an evaluation plan invented after launch is a research project, but an evaluation plan drafted alongside the objectives is a byproduct. Verify-by-design and measure-by-default are the names for moving rigor to the front, where it is cheap, instead of the back, where it is expensive.

Rigor is only slow when it is retrofitted. Designed in from the first minute, verification and measurement run at the speed of the production, because the artifacts they need are already there.

What Verify-by-Design Actually Means

Verify-by-design means the verification a claim needs is built into the workflow that produces the claim, so nothing can reach a learner without passing through the gate that checks it. Why you care: the alternative is verification-by-hope, where a busy team is trusted to remember to check every regulated claim on every build, and the one build a substitute covers is the one where an invented policy threshold slips through. Verify-by-design replaces a habit that fails under pressure with a structure that holds.

Concretely, it has four moving parts, and each one converts a good intention into a property of the system.

Grounding Is the Default, Not a Setting

In a verify-by-design function, the model drafts from the approved source of truth by construction. The policy library, the SOP repository, and the SME interview corpus are wired in, and every load-bearing claim carries a citation to the source it came from. Ungrounded generation of a regulated fact is not something the team is trusted to avoid; it is something the workflow makes hard to do by accident. Grounded generation, also called RAG (retrieval-augmented generation, forcing the model to answer from your approved material rather than its training data), stops being a technique a skilled designer applies and becomes the water the whole function swims in.

The Source Is Captured as the Draft Is Made

The single most important design move is capturing provenance at the moment of generation rather than reconstructing it afterward. When the model drafts screen 18 about the lockout procedure, the workflow records which SOP version that step traces to, right then. Nobody has to reverse-engineer where a claim came from three weeks later, because the answer was written down as the claim was born. This is the difference between a verification pass that takes an afternoon and one that takes two weeks: in a verify-by-design function, the question "where did this come from" was answered before it was asked.

The Gate Is a Gate, Not a Suggestion

A regulated claim cannot move from draft to published without a named human signing that they checked it against the current approved source, and the sign-off is logged in a tamper-evident record. This is not a checkbox someone can wave through under deadline; it is a boundary the work cannot cross unsigned. The SME sign-off log is the artifact that answers "who verified this?" before anyone has to ask, and in a verify-by-design function it is produced automatically as a byproduct of the gate, not assembled by hand at audit time.

Accessibility Is a Gate Too

An AI-generated experience that fails WCAG 2.2 AA (Web Content Accessibility Guidelines, version 2.2, conformance level AA, the W3C Recommendation from 5 October 2023 that is the accessibility target for learning content) does not pass the build gate, full stop. Captions, alt text, contrast, and reading level are checked as part of the flow, per language, and the conformance result is captured as a VPAT or ACR (a Voluntary Product Accessibility Template or Accessibility Conformance Report, the document that states how a product meets accessibility standards). Accessibility stops being the rework cycle discovered after launch and becomes a condition of shipping at all.

What Measure-by-Default Actually Means

Measure-by-default means every build ships with its evaluation designed in, so the question "did it work?" has an instrumented answer instead of a scramble. Why you care: the smile-sheet trap (measuring whether learners liked the course instead of whether they can now do the job) survives almost entirely because measurement is hard to design under deadline, so teams default to the easy Level 1 reaction data and never reach behavior or results. Measure-by-default removes the deadline excuse by drafting the measurement plan alongside the objectives, before the content is even built.

The mechanics mirror verify-by-design. When the objectives are written, the workflow drafts the matching measurement: the Level 2 learning check that proves the objective was met, the Level 3 behavior measure that proves the skill transferred to the job, and, where the program warrants it, the Level 4 results and Phillips Level 5 ROI. The course is instrumented to emit the data it needs, with xAPI statements (Experience API records of actor-verb-object learning events, stable at v1.0.3 since 2016) capturing behavior beyond the LMS. The leading indicators (completion, practice performance, confidence) and the lagging indicators (on-the-job behavior, error rates, business results) are both named at design time, so the function is never left, sixty days after launch, trying to invent a way to prove impact it never instrumented for.

The Kirkpatrick four levels (Reaction, Learning, Behavior, Results) are not a report you write after the fact in a measure-by-default function. They are a design constraint you apply before the fact, which is the only way the higher levels ever actually get measured. A designer who has to bolt Level 3 on after launch almost never does. A designer whose workflow drafts the Level 3 measure alongside the objective does it every time, because doing it is the path of least resistance.

A course that cannot answer "did it change behavior?" was not measured. It was completed. Measure-by-default is the discipline that never lets a build ship as merely completed.

The Operating Model in One Table

Here is the whole idea as a contrast between the bolt-on function and the verify-by-design, measure-by-default function, stage by stage across the pipeline. Read the two right-hand columns against each other: the same work, the same tools, rigor at the back versus rigor in the design.

Pipeline stageBolt-on function (rigor at the end)Verify-by-design, measure-by-default function (rigor in the design)
ObjectivesObjectives written; measurement figured out later, usually never past Level 1Objectives written with the matching Level 2 and Level 3 measures drafted at the same time
Grounded draftingTeam is trusted to ground each build; some do, some forget under deadlineGrounding is the default; every load-bearing claim carries a citation by construction
ProvenanceReconstructed weeks later when someone asks where a claim came fromCaptured at the moment of generation, so the answer exists before the question
SME verificationA sign-off chased at the end, sometimes rushed, sometimes skippedA gate the claim cannot pass unsigned, logged automatically in a tamper-evident record
AccessibilityDiscovered as rework after launch; the AI video fails a 508 reviewA build gate, checked per language, producing a VPAT as a byproduct
Bias check on peopleAd hoc, dependent on whether someone happens to noticeA required release gate on any scenario about people, before a learner runs it
MeasurementAn evaluation plan invented sixty days post-launch, if at allInstrumented at design time; xAPI emits the behavior data the plan already named

Notice that the difference in every row is not a different tool or a different person. It is when the rigor happens. The bolt-on function does the same work, later, slower, and inconsistently. The verify-by-design function does it earlier, faster, and every time, because the rigor is a property of the design rather than a task on a to-do list that a busy team can drop. That is the entire operating model: move the rigor from the back of the pipeline to the front, where it is cheap and unskippable.

A Before and After: The Same Build, Two Functions

Watch one build, a quarterly safety refresh for 6,000 field technicians, run through both functions.

Before (bolt-on). A designer generates the refresh from a brief in an afternoon. It looks finished, so it goes to review. The SME is on a plant visit, so sign-off waits a week. When it comes, the SME flags that two procedure steps trace to a superseded SOP, but nobody recorded which SOP the model used, so the designer spends two days reconstructing provenance to find out. The AI-narrated video went straight to the LMS staging area; a screen-reader user on the accessibility team catches that the captions drift out of sync on the Spanish version, triggering a caption rebuild. There is no measurement plan, so when the head of learning asks how they will know it worked, the answer is a completion report and a satisfaction survey. Net result: an afternoon of production, three weeks of scramble, and a build that can prove people finished it but not that it changed a single behavior at a live panel. The speed was real and then it evaporated.

After (verify-by-design, measure-by-default). The same designer generates the same refresh in the same afternoon, but the workflow is built differently. Grounding is on by default, so every procedure step drafts from the current approved SOP with the version captured at generation. The SME sign-off is a gate: the build cannot reach staging until the SME signs each regulated step against the current source, and the log records who and when. Because provenance was captured as the draft was made, the SME review is a confirmation pass, not an investigation, and it takes an hour. The video passes an accessibility gate per language before it can advance, producing a VPAT; the caption drift is caught inside the gate, not after launch. The measurement plan was drafted alongside the objectives: a Level 2 check on the procedure knowledge, a Level 3 behavior measure that samples whether technicians actually perform the corrected sequence at ninety days, instrumented with xAPI. Net result: an afternoon of production, a day of confirmation, and a build that ships with a sign-off log, a conformance report, and a behavior measure already running. When the regulator, the auditor, or the CFO asks their question, the answer already exists.

The two builds used the identical model and captured the identical production speed. The bolt-on function gave most of that speed back and still could not answer the three questions that matter. The verify-by-design function kept the speed and answered all three, because rigor was a property of the pipeline, not a scramble at the end of it. That gap, repeated across a thousand builds a year, is the difference between a function that survives an audit and one that survives on hope.

How a Leader Builds This

A transformation leader does not create a verify-by-design, measure-by-default function by exhorting the team to be more careful. Carefulness is a habit, and habits fail under deadline. The leader builds it by changing the pipeline so that the careful thing is the default thing, the fast thing, and the only thing the work can do. That is a design job, and it has a recognizable shape.

First, make grounding and provenance-capture the default in the tooling, so a designer would have to work to produce an ungrounded, unsourced draft. Second, make the SME sign-off and the accessibility conformance check actual gates in the workflow, boundaries the work cannot cross unsigned or unconformant, with the log and the VPAT produced as byproducts. Third, wire the measurement design into the objective-writing step, so drafting the Level 3 measure happens at the same moment as drafting the objective it measures. Fourth, and most important, invest in making the verification itself fast, because a gate that is slow will be resented and eventually relaxed, and the whole point is that rigor must run at the speed of production or the organization will trade it away.

The measure of success is a specific and unglamorous one. It is not "we adopted AI." It is that a random build pulled from the function on a random day already has its provenance log, its conformance report, and its behavior measure attached, with no scramble required, because those artifacts are byproducts of how the work is done rather than deliverables someone had to remember to produce. When that is true, speed and rigor have stopped being a trade-off. They are one system. And that system is the thing a regulator, an auditor, and a CFO actually trust, because it does not depend on anyone remembering to be careful on the day it counts.

This is where the whole program has been heading. The iron rule, AI assists, the human verifies, the human owns the decision, and "the AI wrote it" is never a defense to a compliance officer, an accessibility auditor, or a CFO, is not a slogan you post on a wall. It is an operating model you build, one gate and one default at a time, until verification and measurement are so woven into how the function works that being wrong at scale is something the pipeline itself resists. The final lesson turns that operating model into a concrete first ninety days.

Key Takeaways

  • The most common failure in an AI-accelerated learning function is not a hallucination; it is the bolt-on tax, where the production speed is given back to a verification-and-measurement scramble retrofitted onto the end of every build.
  • Rigor is only slow when it is retrofitted; designed into the build from the first minute, verification and measurement run at the speed of production because the artifacts they need already exist.
  • Verify-by-design has four parts: grounding as the default, provenance captured at the moment of generation, the SME sign-off as an unskippable gate, and accessibility conformance as a build gate producing a VPAT.
  • Capturing provenance as the draft is made is the single move that turns a two-week verification investigation into a one-hour confirmation pass, because the answer to "where did this come from" exists before the question.
  • Measure-by-default drafts the Level 2 and Level 3 measures alongside the objectives, defeating the smile-sheet trap by removing the deadline excuse that keeps teams stuck at Level 1 reaction data.
  • The difference between the bolt-on function and the verify-by-design function is never a different tool or person; it is when the rigor happens, moved from the expensive back of the pipeline to the cheap, unskippable front.
  • A leader builds this by changing the pipeline, not by exhorting carefulness: make grounding and provenance the default, make sign-off and accessibility real gates, wire measurement into objective-writing, and invest in making verification fast so it is never resented and relaxed.
  • The whole program lands here: the iron rule is not a slogan but an operating model, built one gate and one default at a time until being wrong at scale is something the pipeline itself resists.