It is 6:40 on a Tuesday morning and the night-shift quality lead is standing at your desk before you have taken your coat off. A customer in Ohio has opened a containment. They found four cracked brackets in a lot of 3,000 that shipped last Thursday, and they are asking the question that turns a bad morning into a bad quarter: how many more are out there, and how did they get past your inspection. Your plant runs an AI vision system on that line, a camera and a model that grades every bracket for surface cracks at line speed and lights a green light when the part passes. The night shift trusted the green light. The green light was wrong, or the line was not running the model the customer thinks it was, or the model drifted three weeks ago and nobody caught it. You do not know which yet. What you do know is that in about ninety minutes you will be on a call with the customer's supplier quality engineer, and "the model flagged it as good" is going to be the worst sentence you could possibly say. This lesson is about the document that should already be written before that Tuesday: the incident-response runbook for an AI-related quality event. You write it cold, on a calm afternoon, so that the people running the containment on the worst morning of the quarter are reading from a plan instead of inventing one.

Why an AI Event Is Not a Normal Containment

Every plant that ships to a serious customer already has a containment process. When a defect escapes, you sort, you contain, you do an 8D (the eight-discipline structured problem-solving report the customer expects), you find root cause, you put in corrective action, and you close it out. So the natural reaction to an AI-related quality event is to say: we already have a runbook, it is the 8D, we will just run it. That reaction is half right and it is the half that gets plants in trouble.

A normal containment investigates a physical or process failure. A weld was cold, a die was worn, an operator skipped a step. An AI-related quality event has all of that plus a second layer: a decision made or assisted by a model that you cannot interview, that has no memory of why it did what it did, and that may have been quietly changing behavior for weeks before the escape. The questions a customer asks about an AI event are different in kind. They will not just ask what the defect was. They will ask: was the model in the loop on this decision, what version of the model was running, when was it last validated, what was its false-reject and escape performance on a holdout set, who was accountable for the green light, and can you show me the log. If you cannot answer those questions in the first day, you are not running a containment, you are improvising under audit, and the customer can hear the difference.

Remember the cardinal rule that runs through this entire program: the customer audits you, not the vendor. When a bracket cracks in Ohio, the customer's supplier quality engineer is not going to call the company that sold you the vision system. They are going to call you. The accountability for an AI-touched quality decision stays with the plant and the human who signed the record. Your incident runbook exists to make that accountability survivable, by making sure that when an AI decision goes wrong, you can reconstruct exactly what happened, prove what you have contained, and show the customer a disciplined response instead of a black box and a shrug.

Write the AI incident runbook on a calm afternoon, because the containment morning is the one morning you will not have time to think clearly.

There is a money reason this matters and it is not small. A single defect escape that becomes a customer containment can cost more than a month of your entire AI program: sort labor across thousands of parts, expedited freight to replace good stock, the customer's charge-back, and the soft cost of a supplier scorecard downgrade that quietly moves the next contract to a competitor. The same talent cliff that pushed you toward AI in the first place is working against you here. With 85% of manufacturers reporting that staffing shortages are hurting product quality, the night shift that trusted the green light was probably short an inspector, which is exactly why the model was watching that line in the first place. The runbook does not prevent the escape. It prevents the escape from compounding into a crisis you cannot explain.

What Counts as an AI-Related Quality Event

The first job of the runbook is to define its own trigger, because the worst incidents are the ones nobody declared an incident. If your team is debating at 9 a.m. whether this even counts as an AI event, you have already lost two hours. The definition has to be written down and it has to be broad enough that the obvious cases and the sneaky cases both trip it.

An AI-related quality event is any quality or downtime event where a model made, assisted, or should have caught the decision in question. That phrasing is deliberate. It covers three distinct failure shapes, and your runbook should name all three so nobody argues about scope while the clock runs.

The escape: the model said good and the part was bad. This is the cracked-bracket case. The vision system passed parts it should have rejected, and they shipped. The escape is the most visible failure mode because a customer usually finds it for you. The investigation has to determine whether the model genuinely misjudged the parts, whether the parts it saw were different from the parts that shipped (a sampling or handling gap), or whether the model was never actually evaluating these parts at all because of an integration fault.

The false-reject storm: the model said bad and the parts were good, at a rate that hurt. This one rarely triggers a customer call, so plants tend not to treat it as an incident, and that is a mistake. A vision model whose false-reject rate climbs from 2% to 9% overnight after a lighting change is scrapping good product and burning operator trust at the same time. The dollar damage is real and quiet: at 9% false reject on a line running 3,000 parts a shift, you are pulling 270 good parts a shift, and an operator who watches the system cry wolf 270 times will reach up and disable the green light by Thursday. The day they disable it is the day your next escape ships. A false-reject storm is an AI-related quality event even though no customer ever sees a bad part, because it is actively dismantling the control you depend on.

The hallucinated-record event: the AI wrote something into a quality record that was not true. Your team uses a generative model (a model that drafts text, the kind that writes a great work instruction and can also invent a torque spec that was never on the drawing) to draft 8Ds, dispositions, and corrective-action language. If that model fabricated a root cause, cited a procedure that does not exist, or stated a verification step that never happened, and that record went out to a customer or into your quality management system, that is an AI-related quality event of a different and dangerous kind. It is dangerous because it can pass every visual check. A confidently written, professionally formatted 8D that contains an invented root cause looks exactly like a good one until an auditor pulls the thread.

Your runbook should also handle the near-miss explicitly: the model failed but a human caught it before any harm reached the customer. Treat near-misses as full incidents for investigation purposes, because a near-miss is a free lesson. The operator who overrode a bad green light and stopped a shipment just told you the model has a problem, for the price of zero containment. A plant that logs and learns from near-misses fixes drift before it becomes an escape. A plant that ignores them learns the same lesson later, in Ohio, with a charge-back attached.

The First Hour: Contain and Preserve

The opening move of any quality containment is the same with or without AI: stop the bleeding. Quarantine suspect product, widen the net to every lot that could share the defect, and notify the customer per the timeline in your quality agreement. Your existing process covers this and the runbook should not reinvent it. What the AI runbook adds to the first hour is a parallel action that is easy to forget under pressure and impossible to recover once it is gone: preserve the evidence about the model.

Models and their surrounding systems overwrite their own evidence constantly. A vision system may keep only the last 48 hours of inference images before they roll off. A historian (the time-series database that logs your sensor and process tags) may downsample older data so the fine detail you need is gone by next week. A vendor cloud service may auto-update the model on a schedule, so the version that made the bad call could be replaced by Friday and you will be investigating a model that no longer exists. The first hour of an AI incident has to include a hard freeze on the evidence, and the runbook should list exactly what to grab.

The evidence preservation checklist, in plain floor terms, is this. Lock the model version. Record the exact version or build identifier of the model that was running on that line at the time of the event, and if the vendor pushes updates automatically, freeze updates on that line until the investigation is done. Pull the inference log. Export the model's decisions for the suspect period: every part it graded, the pass or reject call, and the confidence score if the system reports one. Save the images or the raw input. For a vision system, copy the actual images the model saw for the suspect parts before they roll off the buffer. For a predictive-maintenance model, export the historian tags it was reading. Capture the configuration. Lighting settings, camera exposure, decision threshold, any recipe or recipe change tied to the product. Record the human side. Who was on the line, what the operators saw on the screen, and whether anyone overrode or noticed anything unusual.

The reason this is hour-one work and not day-three work is that day three is too late. A plant I will call a composite of several real ones lost the ability to prove its case because the vision vendor's cloud auto-updated the model 60 hours into the investigation, and the team had never recorded the old version number. They could not tell the customer whether the model that escaped the cracked brackets was the same model running now, which meant they could not prove the new corrective action actually addressed the failure. The containment dragged an extra three weeks and the sort had to be widened twice, turning a 3,000-part problem into an 11,000-part problem. The fix was a single line in a runbook that did not exist yet: freeze the model version in the first hour.

Reconstructing What the Model Did and Why

Once the bleeding is stopped and the evidence is frozen, the investigation turns to the question only an AI incident asks: what did the model actually do, and was it the model that failed or something around it. This is where teams without a runbook waste days chasing the wrong layer, and where a structured set of questions earns its keep. Walk the decision from the outside in.

Was the model even in the loop? The first and most common surprise is that the model was not running the way everyone assumed. The line was in a manual bypass mode after a jam. The camera was fouled with coolant mist and the system was returning a default pass. An integration fault meant the model's reject signal was not actually stopping the part. Before you blame the model's judgment, confirm the model was making the decision you think it was. The inference log answers this: if the log shows the model rejected the brackets but they shipped anyway, your failure is downstream of the model, in the handling or the line logic, not in the model's eyes.

What did the model see versus what shipped? A vision model can only grade what is in front of the camera. If the cracked brackets cracked after inspection, during a downstream press or in transit, the model did its job correctly and your problem is a process step the camera never watched. Compare the saved inference images against the returned defective parts. This single comparison resolves a large share of vision escapes and it protects you from putting a corrective action on a model that was never the cause.

Did the model drift? Drift is the slow killer. A vision model validated in spring against bright morning light can degrade by autumn when the sun angle changes and the supplier switches to a slightly shinier raw material. The model did not break on Thursday; it had been getting quietly worse for weeks, and Thursday was just the day a real defect landed in the gap. Check the model's recent performance trend if you have been monitoring it, compare current false-reject and escape rates against the validation baseline, and look for the environmental change that started the slide: a new material lot, a relamped fixture, a shifted camera, a seasonal light change. If you find drift, your corrective action is not just sort-and-ship, it is a re-validation and a drift-monitoring plan, because the same drift will recur next season.

For a generative record, where did the claim come from? If the event is a hallucinated 8D or disposition, the reconstruction is different. Pull the prompt and the source documents the model was given, and check whether the disputed claim was grounded in a real record or invented. A model asked to write a root cause without being given the historian trace and the traveler will fill the gap with a plausible story. The investigation here is tracing every factual claim in the AI-drafted record back to a source document, and the ones that trace to nothing are your hallucinations.

The discipline that makes all of this possible is one you build long before the incident: keep AI advisory and logged. A vision system that recommends and lets a human or a governed line logic make the final call, with every decision written to a log, is reconstructable. A model wired directly into control with no log is a black box that cannot be investigated, which is exactly why this program insists AI stay advisory and out of direct control of anything that moves unless it is properly governed. You cannot run an incident response on a plant you cannot see, and 78% of OT networks (the operational-technology side, the controls-and-sensors network as opposed to the office IT network) lack the centralized monitoring that would let you see it. The runbook's reconstruction phase is only as good as the logging you set up on a calm day.

Roles, Decision Authority, and What You Tell the Customer

An incident is half investigation and half coordination, and the coordination half is where calm afternoons pay off most. The runbook has to name who does what, who decides what, and who talks to whom, so that nobody is figuring out the org chart while the customer waits.

Name the incident owner. One person owns the event end to end. On a quality escape this is usually the quality engineer or quality manager, not the person who deployed the AI. The owner is accountable for the containment, the investigation, and the customer communication, and crucially the owner is a human whose name goes on the record. The model is never the owner. "The model flagged it" is not a sentence the owner is allowed to end on.

Pull in the AI and OT roles fast. The investigation needs someone who can read the inference log and the model configuration, which on most plants is the person who stood up the vision or predictive system, plus whoever holds the OT/IT boundary so that pulling logs and freezing versions does not itself break the line or open a security hole. If your plant has stood up an AI governance forum (the standing group of quality, maintenance, OT, and EHS that owns floor AI), the incident runbook is one of its core deliverables, and the forum members are your incident team. If you have not stood that forum up yet, the incident is the argument for it.

Set decision authority in advance. The runbook should state who can take the AI system offline, who can widen the containment, who can release product back to flow, and who signs the customer-facing 8D. These are decisions that should not require a meeting to schedule a meeting. The authority to pull the vision system off a line and run 100% manual inspection should sit with the quality manager and be exercisable in minutes, because the alternative is shipping more suspect parts while the team debates.

The customer communication piece deserves its own discipline because it is where plants most often turn a recoverable incident into a relationship-ending one. Tell the customer what you know, what you do not yet know, and what you are doing about both. Do not speculate about the model on the first call, and never offer the model as an excuse. A strong opening to a supplier quality engineer sounds like: we have contained all suspect lots, here is the quantity and the lot range, we have frozen the inspection system and preserved the records from the period in question, and we will have a verified root cause to you by the agreed date. That answer says you have a process. The answer that ends careers is: we are not sure what happened, the AI system passed them, we are looking into it. Same facts, opposite credibility.

There is a documentation standard underneath all of this. Every AI-touched quality decision in the suspect window should be reconstructable from your logs, and the incident record itself becomes part of your audit trail. When the customer or a registrar auditing your IATF 16949 (the automotive quality management standard) or AS9100 (the aerospace equivalent) system asks how you control AI in quality, the answer is your governance: here is our incident runbook, here is a closed incident that proves it works, here is the log that let us reconstruct the decision. A clean, well-run incident is not just damage control. It is the single best evidence you can show an auditor that your AI is under control.

Closing the Loop and Fixing the System, Not Just the Parts

A normal containment closes when the parts are sorted, the root cause is found, and the corrective action is verified. An AI incident has the same finish line plus one more lap, because the root cause of an AI event usually lives in two places at once: the physical or process cause of the actual defect, and the reason the AI failed to catch it or invented something false. Close both or the incident will reopen.

Take the cracked brackets. Suppose the investigation finds two true root causes. The brackets cracked because an upstream die wore past its limit, a normal mechanical root cause with a normal corrective action: replace the die, add a die-life counter, tighten the preventive schedule. But they also escaped because the vision model had drifted, its false-reject rate had quietly dropped as it was retrained on too many borderline-pass examples, and it had lost the sensitivity to flag the hairline cracks. That second root cause needs its own corrective action: re-validate the model against a fresh holdout set of known-good and known-bad parts, restore the decision threshold, and stand up a drift monitor that compares weekly performance against the validation baseline so the next slide trips an alarm instead of a customer. If you only fix the die, you ship again the first time a different defect appears, because the model is still blind.

The corrective action for an AI failure is usually one of a short list. Re-validate the model against a current, representative holdout set. Adjust the decision threshold to rebalance false-reject against escape, with the trade-off priced in dollars and signed off by a human, not left to a vendor default. Add or fix drift monitoring so degradation is caught early. Tighten the human-in-the-loop step so a person confirms borderline cases instead of the line auto-passing them. Improve grounding for a generative tool so it can only write what the source documents support, and add a verification checklist that traces every claim before the record leaves the building. Retire the model from that decision entirely if the investigation shows it should never have been load-bearing there in the first place.

Then do the thing that separates a plant that learns from a plant that repeats: feed the incident back into governance. Every closed AI incident should update your when-not-to-trust list, your validation cadence, and your runbook itself. If this event revealed that nobody owned drift monitoring on that line, the corrective action assigns an owner. If it revealed that the model auto-updated mid-investigation, the corrective action freezes auto-updates plant-wide until a change-control process exists. The runbook you wrote cold gets better every time it runs hot, and that compounding is the whole point.

The payback math closes the case. The runbook costs you an afternoon to write and a few hours a quarter to drill. The first real incident it shortens by even a week, by preserving evidence you would otherwise have lost and structuring an investigation you would otherwise have improvised, pays for itself many times over against a containment that can already cost more than a month of the program. The plant with the runbook turns its worst Tuesday into a closed incident and an audit exhibit. The plant without it turns the same Tuesday into a black box, a widened sort, and a supplier scorecard it spends a year recovering from.

Key Takeaways

• An AI-related quality event is a normal containment plus a second layer: a model decision you cannot interview, that may have drifted for weeks, and that a customer will ask pointed questions about. Write the runbook on a calm afternoon so the containment morning runs from a plan, not an improvisation.
• The customer audits you, not the vendor. Accountability for an AI-touched quality decision stays with the plant and the human who signs the record. "The model flagged it" is never a sufficient answer, and a single escape-turned-containment can cost more than a month of the whole AI program.
• Define the trigger broadly: the escape (model said good, part was bad), the false-reject storm (model said bad at a rate that scraps good product and kills operator trust), and the hallucinated-record event (a generative tool wrote a false root cause or invented a spec into a quality record). Treat near-misses as full incidents because they are free lessons.
• The first hour adds one AI-specific action to normal containment: preserve the evidence before it overwrites itself. Lock the model version and freeze auto-updates, pull the inference log, save the images or historian tags, capture the configuration, and record the human side. Day three is too late.
• Reconstruct from the outside in: was the model even in the loop, did it see what actually shipped, did it drift against its validation baseline, and for a generative record, does every factual claim trace to a real source. This is only possible if AI was kept advisory and logged on a calm day, because you cannot investigate a black box wired into control.
• Name a human incident owner, pull in the AI and OT roles fast, and set decision authority in advance so taking the system offline or widening a sort does not wait on a meeting. Tell the customer what you know, what you do not, and what you are doing, and never offer the model as an excuse.
• Close both root causes, the physical defect and the AI failure. The corrective action for the model is usually re-validation against a fresh holdout, threshold rebalancing priced in dollars, drift monitoring, a tighter human-in-the-loop step, better grounding for generative tools, or retiring the model from a decision it should never have owned.
• Feed every closed incident back into governance: update the when-not-to-trust list, the validation cadence, and the runbook itself. A clean, well-run AI incident is also the single best audit exhibit you can show that your floor AI is genuinely under control.