There is a moment, after Maze AI hands you a tidy summary of a thirty-person study, where you have to decide whether to trust it. Most designers resolve that moment one of two ways, and both are wrong. Some trust it completely, paste the summary into the readout, and ship findings they never checked. Others trust it not at all, re-watch every recording by hand, and throw away the entire point of using AI. The replicable pattern this lesson teaches is the third way: a structured workflow where Maze AI does the synthesis it is genuinely good at, you sample-verify a small, deliberately chosen set of clips against the raw recordings, and you write findings that carry their own audit trail. You walk away with a published findings doc and a verification audit trail that shows exactly what the AI claimed, what you checked, and what you found - the document that lets a senior IC ship AI-synthesized research without supervision and defend it in any review.

The Two Ways This Goes Wrong

Watch a design team adopt Maze AI and you will see the failure modes within a month. The first is over-trust. Maze AI produces a clean summary - "Task 3 had a 67 percent completion rate; the most common point of failure was the date picker; users described the flow as straightforward" - and it reads so confidently that the designer pastes it into the readout and moves on. The summary is mostly right, which is exactly what makes the wrong part dangerous, because nobody is looking for it. Somewhere in there is a claim the model inflated, a quote it paraphrased into something the user never said, or a "most common" that was actually two people. The readout ships, a roadmap decision gets made on it, and the error only surfaces months later when the feature built on it underperforms and someone finally re-watches the tape.

The second failure is over-verification, and it is quieter because it looks like diligence. Burned once, or just temperamentally cautious, the designer re-watches all thirty recordings, re-transcribes the key moments, and rebuilds the synthesis by hand. The findings are now bulletproof and the designer spent three days producing what Maze AI produced in three minutes. They have paid the full manual cost and captured none of the AI benefit, and worse, they have trained themselves to believe the tool is useless when in fact they just used it without a verification strategy. Do this a few times and you quietly opt out of the speed that the rest of the org now assumes you have.

The replicable pattern threads between these. It accepts that Maze AI's synthesis is a strong first draft that is usually right and occasionally, consequentially wrong, and it spends a fixed, small verification budget exactly where being wrong would cost the most. The goal is not certainty - certainty costs three days - but calibrated confidence at a known, affordable price.

What Maze AI Is Actually Good At, and Where It Drifts

To verify intelligently you have to know which parts of the synthesis to trust on sight and which parts to check. Maze AI is reliable on the quantitative scaffolding because that part is not synthesis at all - it is counting. Completion rates, time-on-task, misclick counts, the per-screen heatmaps: these come straight from the captured behavioral data, and the model is reporting numbers, not interpreting them. You can trust a stated completion rate the way you trust a calculator, because that is essentially what produced it.

Where Maze AI drifts is in the qualitative interpretation, and the drift has a predictable shape. It paraphrases open-text answers and verbatim comments, and in paraphrasing it smooths - "I had no idea what that button did" becomes "user expressed uncertainty about the control," which is blander, safer, and quietly less true. It clusters comments into themes, and the clustering can over-merge ("usability concerns") or invent a theme that three loud responses suggested and twenty-seven contradicted. It picks representative quotes, and a representative quote is a judgment call the model makes without your context about which user matters. And it occasionally states a "most users felt" where the actual count was a minority - the qualitative summary inherits no denominator unless you force one.

This map is the whole basis of the verification strategy. You do not verify uniformly, because the synthesis is not uniformly risky. You trust the counts and you check the interpretations, and within the interpretations you check the ones whose error would be expensive. The drift is concentrated; so is your verification.

Maze AI is a calculator on the numbers and a paraphraser on the quotes. Trust the calculator; audit the paraphraser - and audit it hardest exactly where a smoothed-over verbatim would change a decision.

The Pattern: Run, Synthesize, Sample-Verify, Write

Here is the workflow end to end, on a real thirty-respondent unmoderated study. It has four stages, and the discipline is in the third.

Stage one, run the study. Thirty respondents through a study designed to be synthesizable - behavioral tasks with success conditions, a difficulty rating after each, step-anchored open-text. (If you have not designed the study for synthesis, this whole pattern degrades, which is why the previous lesson comes first.) Maze captures completion, time, misclicks, and the open-text automatically.

Stage two, take the AI synthesis as a first draft. Run Maze AI's summary. Read it once, fully, and mark it as a draft - literally, in the document, label it "AI synthesis, unverified." This labeling is not bureaucratic; it changes how you and everyone downstream read it, because an unverified draft invites scrutiny where a finished summary invites trust.

Stage three, sample-verify five clips. This is the heart of the pattern and the part everyone skips. You do not verify everything and you do not verify randomly. You select five clips to check against the raw recordings, chosen by where verification has the most leverage. Five is not arbitrary: it is small enough to do in twenty minutes and large enough to catch the systematic errors that matter, and it forces you to prioritize, which is the actual skill.

Stage four, write the findings with the audit trail attached. You write the findings doc using the verified synthesis, and you attach the verification audit trail - which clips you checked, what the AI claimed about each, what you found, and whether the claim held. The findings doc is the deliverable; the audit trail is what makes it defensible and what lets a reviewer trust the parts you did not check, because they can see your verification was targeted, not random.

How to Choose the Five Clips

The selection is the craft. Verifying the wrong five clips is barely better than verifying none, because you spend your budget confirming things that were never at risk. Choose the five by these criteria, roughly in priority order.

The Claim That Will Drive the Biggest Decision

Find the single finding in the synthesis that, if wrong, would most damage the decision the study is feeding. If the study is meant to validate shipping the new flow, the claim "Task 3 had a 67 percent completion rate and users found it straightforward" is load-bearing - the whole go decision leans on it. Verify it first. Pull the recordings for the failures specifically and watch what actually happened: did they fail because of the date picker the AI named, or because of something the AI missed entirely? This single check is worth more than the other four combined, because it sits directly under the decision.

The Quote the Synthesis Leans On

Synthesis summaries almost always feature one or two verbatim quotes doing emotional and rhetorical heavy lifting - the quote that "captures the user frustration." Find the most load-bearing quote and check it against the recording word for word. This is where the smoothing failure from the previous lesson lives: the model may have paraphrased a sharp, specific complaint into a vague one, and the sharp original might be the single most important sentence in the study. The classic case is a verbatim like "I stopped because I couldn't tell what it just did" getting rendered as "users had concerns about feedback clarity." If your headline quote turns out to be a paraphrase, you have found a real problem and possibly a better finding.

The "Most Users" With No Visible Denominator

Any qualitative claim phrased as "most users," "several respondents," or "a common theme" without a number attached is a candidate, because that phrasing is exactly where the model inflates a minority into a majority. Pick the most consequential one and count it manually against the open-text. If "most users struggled with X" turns out to be four of thirty, the finding is not wrong but its weight is, and weight drives prioritization. Forcing the denominator is often the single highest-value twenty seconds in the whole verification.

A Contradiction and a Confirming Outlier

Spend the last two clips on robustness. One on a place where the synthesis seems internally inconsistent - a high completion rate paired with heavy reported difficulty, say, which usually means something interesting is hiding (people completed it but hated it, or completed it wrong). And one on a strong claim you expect to hold, as a control: if a clip you predicted would confirm actually contradicts, your trust in the unchecked remainder should drop and you verify more. The control clip calibrates how much to trust everything you did not check, which is the entire purpose of sampling.

The Verification Audit Trail

The second named artifact is the audit trail, and its structure is what makes the findings doc survive scrutiny. For each of the five verified clips, the trail records four things: the AI's claim, the verification action you took, what you actually found, and the resolution. A row reads like this. AI claim: "Task 3 completion 67 percent, users found it straightforward." Verified: re-watched all 10 failure recordings. Found: 7 of 10 failed at the date picker as claimed; 3 failed earlier, at the address field, which the synthesis missed. Resolution: completion rate confirmed; added address-field failure as a second finding; corrected "straightforward" to "completable but with two distinct friction points."

Notice what this row does. It confirms the number, catches a missed finding, and corrects an over-smoothed adjective, all transparently, so anyone reading the findings can see precisely what was checked and what changed. The audit trail is not a confession of the AI's errors; it is the evidence that your findings are trustworthy. When a stakeholder pushes on a finding in the review, you do not defend it with "the AI said so" or with your authority - you point to the row that shows you checked it against the recordings and what you saw. That is a categorically stronger position than either over-trust or over-verification leaves you in.

The trail also documents what you did not verify, honestly. The findings doc should carry a short note: "Verified 5 of N synthesized claims, selected by decision-impact; remaining claims are AI-synthesized and unverified, and carry the confidence of the quantitative data they rest on." This is not a weakness to hide. It is the honest scope of the work, and stating it is what separates a senior IC's defensible findings from a junior's unexamined paste.

When Verification Finds a Real Problem

Sometimes the sample turns up an error large enough that you cannot trust the rest, and the pattern has to tell you what to do then. The rule is proportional escalation. If your five clips come back clean, you ship the findings with the audit trail and the unverified-remainder note, confident. If one clip reveals a contained, local error - a paraphrased quote, a missed minor finding - you correct that finding and ship. But if a clip reveals a systematic error - the completion rate itself is wrong, or the AI's theme-clustering merged two genuinely different problems across the whole study - then the sample has told you the synthesis is unreliable in a way that is not local, and you escalate: verify more clips, or in the worst case, re-synthesize. The five-clip sample is a smoke detector. Most days it confirms there is no fire and you move on; occasionally it catches a real one, and then you do the extra work, but only then.

This is what makes the pattern honest rather than a rubber stamp. A verification budget that can never trigger more work is theater. The five clips have real teeth precisely because they can, and sometimes do, send you back. But the design of the pattern - check the highest-leverage clips first - means that if there is a systematic problem, you are most likely to hit it in the first one or two checks, where it is cheapest to discover, rather than after you have already shipped.

Why Five, and Why a Sample at All

The instinct to verify everything comes from a real place: any unchecked claim could be the wrong one. But verifying everything is not actually more rigorous - it is a failure to think about risk. The five-clip sample is a bet, and the bet is a good one because of how AI synthesis errors distribute. They are not uniformly scattered; they concentrate in the qualitative interpretation, in the load-bearing claims, and in the "most users" phrasings. If you check the highest-leverage points and they hold, the probability that a low-leverage unchecked claim is both wrong and consequential is small - and a low-leverage claim that is wrong is, by definition, one whose error does not change a decision. You are not verifying everything because you are verifying everything that matters, which is a different and better thing.

There is also a calibration argument. By verifying a sample and tracking how often the synthesis holds, you build, over many studies, an empirical sense of your tooling's actual error rate on your kind of work. After a dozen studies you know whether Maze AI's completion rates are dead reliable (they are) and whether its theme-clustering needs heavy checking (it usually does), and you can tune the sample accordingly - more clips on the parts that have burned you, fewer on the parts that never have. The sample is not just verification of this study; it is a continuous measurement of how much to trust the tool, which is knowledge a designer who either always-trusts or always-rechecks never acquires.

The Replicable Part: This Is a Template, Not a One-Off

The word "replicable" in the title is the point. This is not a heroic one-time audit; it is a pattern you run identically on every AI-synthesized study, so that the verification stops being a decision you agonize over and becomes a routine you execute. Run the study, take the synthesis as a draft, verify five clips chosen by decision-impact, attach the audit trail, note the unverified scope, escalate proportionally. Same shape every time. The replicability is what makes it survivable at senior-IC throughput: you cannot afford to reinvent your verification strategy for every study, and you do not have to, because the strategy is fixed and only the five clips change.

Done consistently, the pattern also changes how your team treats AI-synthesized research. When every findings doc ships with an audit trail, "verified five clips by decision-impact" becomes the team's shared standard for what trustworthy AI synthesis looks like, and the over-trust and over-verification failure modes both fade, because there is now a named middle path everyone runs. You have turned a personal discipline into a team norm, which is the actual mark of an AI-integrated designer: not that you verify well, but that you made verifying well cheap, repeatable, and standard.

Key Takeaways

• The two failure modes are over-trust (paste the AI summary, ship unchecked findings, discover the error months later) and over-verification (re-watch all thirty recordings, pay the full manual cost, capture none of the AI benefit). The replicable pattern threads between them with a small, fixed, well-targeted verification budget.
• Maze AI is a calculator on the numbers (completion rates, time, misclicks come straight from captured data - trust them) and a paraphraser on the quotes (it smooths verbatims, over-merges themes, and inflates minorities into "most users"). Trust the counts; audit the interpretations.
• The pattern is four stages: run a synthesis-ready study, take the AI summary as a labeled draft, sample-verify five clips against the raw recordings, and write findings with the audit trail attached.
• Choose the five clips by leverage, not randomly: the claim driving the biggest decision, the load-bearing quote (check it word for word for smoothing), the "most users" with no denominator (count it), one internal contradiction, and one confirming control to calibrate trust in the unchecked remainder.
• The audit trail records the AI's claim, what you verified, what you found, and the resolution for each clip, plus an honest note of what you did not verify. It is what lets you defend findings without saying "the AI said so" and lets a reviewer trust the unchecked parts because your verification was targeted.
• Escalate proportionally: clean sample, ship; local error, correct and ship; systematic error, verify more or re-synthesize. The five-clip sample is a smoke detector with real teeth - it can and sometimes must send you back, which is what keeps it from being a rubber stamp.
• It is a replicable template, not a heroic one-off. Running the same shape every study makes verification routine, builds an empirical sense of your tooling's true error rate over time, and turns a personal discipline into a team standard for trustworthy AI synthesis.