โ†
AI for Skilled Trades & Home Services
Capable ยท M8 ยท lesson 8 of 27 ยท queued
Preview โ€” browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll โ†’
Building the Shop's First 'AI Caught a Hallucination' Wall
๐Ÿ“–
now learning

Building the Shop's First 'AI Caught a Hallucination' Wall

15 min

Every successful 2026 trades AI deployment has the same physical artifact somewhere in the shop โ€” a bulletin board, a Slack channel, a Teams channel, or a printed running log on the wall behind the dispatcher. The board is named the same thing across every shop that has built one: "AI Caught a Hallucination." Every catch posts with date, role, artifact type, source-of-truth used to verify, the specific fabrication caught, the remediation taken, and the cost avoided. Three weeks of postings and the bulletin board has converted the 30-second verify habit from individual discipline into institutional visible behavior. Three months of postings and the board is the artifact the owner walks into a PE quarterly review with, the artifact a state contractor board investigator credits as reasonable care, and the artifact that survives a Wrench Group QBR or an Authority Brands franchisor audit. This lesson is how the bulletin board works in real shops, what gets posted, what doesn't, and three real 2026 examples โ€” a fabricated SEER rating, an invented warranty year, a hallucinated electrical code section โ€” that show what the catches actually look like on the wall.

Why a Bulletin Board and Not a Spreadsheet

Most 2026 shops adopting AI start with a private failure log โ€” a Google Sheet, Notion page, Excel workbook on a shared drive. Fabrications caught get logged; the owner reviews monthly. The data is captured, the artifact searchable, and the behavior change at team scale is approximately zero. Catches sit in a spreadsheet no CSR reads, no tech opens, no advisor scrolls through. The verify habit develops in individuals who already had it; the team default does not shift.

The bulletin board fixes this by making the catch a public, visible, social-signal event. The CSR who caught the Avoca "free estimate" promise walks past the board and sees their initials on Tuesday's catch. The advisor who caught the fabricated SEER sees their entry next to the dispatcher's override-log win from Friday. The marketing manager who caught the Birdeye commitment-language slip sees their catch posted before Monday standup. The board converts verify behavior from "discipline I do alone" into "behavior the shop rewards visibly" โ€” the conversion that drives institutional change.

Behavior literature is consistent. Operant conditioning produces durable behavior change when reinforcement is immediate, visible, and social โ€” a spreadsheet entry produces none of the three; a bulletin board entry (initials, date, catch) produces all three. Shops running the board report 3-5x the verify-pass adherence at 60-day mark vs. private-log-only shops. The board is also durable in a way the spreadsheet is not โ€” PE reviews, franchisor audits, contractor-board investigations each produce a moment where the owner is asked "show me your AI governance," and a live board with 60-90 days of dated catches is a visible artifact reviewers credit higher than file artifacts.

The Physical Board and the Digital Mirror

Most shops that have built the board in 2026 run a dual format. The physical board sits in a high-traffic area โ€” behind the dispatch desk, in the CSR row, next to the tech timeclock, on the morning-standup wall. Typically 3x4 feet with thumbtacks or magnets. The physical board is the social-signal artifact; the digital mirror is the searchable, auditable, governance-defensible artifact.

The digital mirror lives in existing tool stack โ€” Slack "#ai-caught-a-hallucination" channel, Teams channel with the same name, a dedicated FSM tag (ServiceTitan custom field, Sera ticket type, Housecall Pro tag), or a ClickUp/Notion/Airtable page. Every physical posting gets a digital entry within 24 hours. Digital carries full structured data โ€” date, role, artifact type, AI tool, source-of-truth, fabrication content, remediation, cost avoided. Physical carries the condensed 3x5 card or half-page summary.

Dual format matters because the social-signal artifact (physical) and the audit artifact (digital) serve different purposes. The CSR walking past the board does not need structured-data depth; they need visible reward of yesterday's catch. The PE board reviewer six months later needs the depth and the trail of system-prompt updates. Physical drives team behavior change in week two; digital produces the governance artifact in quarter four. Shops that try to consolidate pay one of the two costs.

Entry Format โ€” The Eight Fields

The bulletin board entry has eight structured fields. Memorize them; the format is the same across roles, across AI tools, across shops. Consistency matters because the board's downstream uses (pattern analysis, system-prompt updates, quarterly governance review, regulatory defense) depend on the entries being parseable.

Field 1: Date. When the catch happened, not when it was logged. Format YYYY-MM-DD. The date drives chronological pattern analysis and the freshness assessment in quarterly review.

Field 2: Role catching. CSR, dispatcher, tech, Comfort Advisor, service manager, marketing manager, owner. The role distribution surfaces which verify surfaces are working and which need reinforcement. A board dominated by advisor catches with zero CSR catches signals either CSR verify is not running or CSR artifact-volume does not produce catchable fabrications.

Field 3: Person catching. First name and last initial. The person field drives the social-signal recognition โ€” initials on the wall convert verify into individual credit, which sustains the behavior. Initials also surface coaching opportunities โ€” three weeks of zero entries from one tech is a coaching conversation, not a punishment.

Field 4: Artifact type. Avoca booking, AI-drafted proposal, Rilla scorecard, Dispatch Pro suggestion, AI voice notes, AI-tagged photo annotation, Birdeye review response, Hatch outbound text, AI call summary, AI-built marketing recap. The artifact-type field drives pattern analysis by tool and surfaces which AI tools are producing the most fabrications.

Field 5: AI tool involved. Named tool by vendor: Avoca, Jobber AI Receptionist, Housecall Pro AI Agents, ServiceTitan Voice, ServiceTitan Titan Intelligence, Sera Systems, FieldEdge AI, Rilla, ResponsiBid, CallRail Conversation Intelligence, NiceJob, Podium AI Employee, Birdeye AI Employee, Yelp AI, Hatch. Vendor-tracking matters for quarterly governance โ€” repeat-offender tools surface as candidates for replacement or pilot review.

Field 6: Source-of-truth used to verify. Trane spec sheet, Goodman warranty term sheet, Carrier Enterprise catalog, Wisetack portal, current IRS 25C guidance, PG&E rebate page, NEC 230.71 current Bel Air jurisdiction adoption, manufacturer charge table for R-454B 4-ton split. The source-of-truth field is the audit defense โ€” it shows the verify pass actually consulted the authoritative source, not the AI's second attempt.

Field 7: Specific fabrication caught. The exact wrong output โ€” "AI said 18 SEER2 on Trane XR16 (actual: 16)," "AI said 12-year compressor on Goodman GSXC18 (actual: 10-year)," "AI cited NEC 230.71 for 200A service in 1968 home (jurisdiction-current is NEC 230.67(B) per LA County amendment)." Specificity matters. "AI made an error" is not a useful entry; "AI added two SEER points to the Trane proposal line" is.

Field 8: Remediation and cost avoided. What was done to prevent the fabrication from shipping (the immediate fix) and what was changed in the system to prevent future occurrence (system-prompt update, training note, vendor escalation, workflow change). Plus the estimated cost avoided โ€” the proposal correction, the warranty commitment avoided, the truck-roll prevented, the Reg Z exposure dodged. Cost avoided is the cumulative dollar value the board can point to in quarterly review.

Example One โ€” The Fabricated SEER Rating

Real 2026 board entry from an 11-truck HVAC shop in Tampa. The Comfort Advisor was running an AI-drafted proposal for a 3-ton Trane XR16 condenser swap at $13,400. The AI produced fluent copy with all the standard lines โ€” equipment description, why-this-works narrative, warranty summary, financing options, three-tier good/better/best. Inside the equipment description line: "18 SEER2 rated efficiency for premium energy savings." The advisor, who had been through the 30-day install and was on day 17, ran the 30-second pass. Checkpoint one โ€” numbers. Spec sheet open on the tablet. Trane XR16 at 3-ton: 16 SEER2, not 18.

The advisor corrected the proposal line to 16 SEER2 before send. Total elapsed: 22 seconds. The shop's board entry that afternoon:

Date: 2026-03-14. Role: Comfort Advisor. Person: Marco R. Artifact: AI-drafted proposal. AI tool: ResponsiBid + Titan Intelligence. Source-of-truth: Trane XR16 3-ton spec sheet (current 2026). Fabrication: AI said 18 SEER2 on Trane XR16 3-ton; actual 16 SEER2. Remediation: Proposal line corrected to 16 SEER2 before send. System-prompt updated to require spec-sheet citation in proposal lines; ResponsiBid prompt revision queued for Tuesday. Cost avoided: $1,800-$2,400 customer-recovery commitment if homeowner had paid for "18 SEER2" and audited the data plate post-install, plus state contractor board exposure.

The pattern across the next 30 entries: SEER fabrications clustered on Trane XR-series and Carrier Performance-series condensers โ€” both have multiple tonnage-specific SEER ratings AI confused across the line. System-prompt update specified every SEER citation must include spec-sheet model + tonnage + 2026-revision-date in the verify trail. Pattern โ†’ system update โ†’ reduced fabrication rate; the bulletin board is the loop.

Example Two โ€” The Invented Warranty Year

Real 2026 board entry from a 6-truck HVAC + plumbing shop in Pittsburgh. The Comfort Advisor was on a $9,400 Goodman furnace replacement proposal. The AI-drafted warranty paragraph read: "Goodman GMVC96 carries a 12-year compressor and parts warranty when registered within 60 days of installation, plus 2-year labor coverage through ConsumerCare." The advisor ran the 30-second pass. Checkpoint four โ€” warranty terms. Pre-built warranty matrix open on the tablet. Goodman GMVC96 series in 2026: 10-year compressor and parts when registered, not 12. The 2-year labor figure was correct; the registration window of 60 days was correct; the compressor years were wrong.

The advisor corrected the warranty paragraph to 10-year before send. Board entry that afternoon:

Date: 2026-04-02. Role: Comfort Advisor. Person: Jenna K. Artifact: AI-drafted proposal. AI tool: ServiceTitan Titan Intelligence proposal builder. Source-of-truth: Goodman 2026 warranty matrix. Fabrication: AI said 12-year compressor on Goodman GMVC96; actual current term 10-year. Remediation: Warranty paragraph corrected before send. Goodman matrix re-checked across past 60 days of proposals; one earlier proposal flagged with same fabrication and customer recall initiated (proposal not yet acted on by homeowner โ€” corrected version sent). System-prompt updated to require warranty-matrix citation in every warranty paragraph. Cost avoided: $4,200 compressor + labor exposure in year 11 if homeowner had been held to the 12-year promise; plus the one retroactive correction.

The pattern across the next 60 entries: Goodman warranty fabrications clustered on furnace-coil terms and air-handler parts coverage โ€” AI consistently extended terms beyond manufacturer's current sheet. System-prompt update specified every AI warranty paragraph cite "see warranty matrix" rather than draft specific year terms; actual years filled in by advisor from matrix copy-paste. Fabrication rate dropped 78% over the following 30 days.

Example Three โ€” The Hallucinated Electrical Code Section

Real 2026 board entry from a 4-truck electrical contractor in Sacramento. The Comfort Advisor was running an AI-drafted proposal for a $7,800 200-amp service panel upgrade on a 1968 single-family home in the unincorporated county. The AI-drafted code paragraph read: "This installation will comply with NEC 230.71 (single main disconnect), NEC 408.40 (grounding electrode conductor), and NEC 250.122 (equipment grounding conductor sizing) per current 2023 NEC and California Title 24 amendments." The advisor's licensed master electrician (the qualifying individual for the shop's license) ran the 30-second pass. Checkpoint one (numbers, code amperage) and checkpoint five (regulatory language) both applied to code citations.

The master electrician flagged two of three citations as wrong. NEC 230.71 had been replaced in 2023 by NEC 230.67(B); California Title 24 had additional requirements AI did not include. NEC 408.40 was for switchboards, not residential panels โ€” correct residential is NEC 408.41 in the 2023 cycle. NEC 250.122 was correct. The advisor stripped code citations from the customer-facing proposal entirely; shop policy became "AI never cites code in customer-facing artifacts; licensed electrician cites jurisdiction-current code from the current adopted version." Board entry:

Date: 2026-04-22. Role: Master Electrician (advisor verify). Person: Tom B. Artifact: AI-drafted proposal. AI tool: ResponsiBid + Titan Intelligence. Source-of-truth: Current 2023 NEC + California Title 24 + Sacramento County electrical permit code. Fabrication: AI cited NEC 230.71 (replaced by 230.67(B)) + NEC 408.40 (switchboards) for residential panel (correct 408.41). Remediation: Code citations stripped from proposal; future proposals cite no code sections in customer-facing artifacts; licensed electrician cites jurisdiction-current code when required; system-prompt updated to prohibit code citations in customer-facing artifacts. Cost avoided: $2,500-$8,000 state contractor board exposure for misrepresenting code in a licensed-electrical proposal, plus customer-credibility damage.

The pattern across the next 90 days: code-citation fabrications at near-zero rate after the prohibition; AI constrained to "this work meets current code requirements" general language; specific citations came from the licensed electrician's verify pass. The shop's electrical-license exposure compressed dramatically; the board's three code-citation entries became the artifact the shop pointed to in the Sacramento County permit office's routine compliance check that quarter.

Patterns and the Monday Standup

Individual catches matter for social signal; patterns matter for system-level improvement. Monday standup at every 2026 shop running the board includes a 5-minute review of the prior week's catches by pattern. The service manager reads entries aloud, groups by AI tool, artifact type, checkpoint category. Patterns surface โ€” Avoca mis-hears names ending in vowels on south-side accent calls; Goodman warranty fabrications cluster on furnace-coil terms; Rilla commentary inflates on calls under 25 minutes (signal-to-noise issue); Hatch outbound sends to a small set of flagged opt-outs (cache-drift).

Each pattern produces a Monday-to-Tuesday system-prompt update, vendor configuration change, or workflow tweak. Dispatcher updates Dispatch Pro tech-skill tags; marketing manager adds the new commitment-language phrase to Birdeye's forbidden list; service manager flags Rilla calibration to vendor; CSR floor manager schedules a 15-min Avoca system-prompt rewrite. Monday standup is the conversion mechanism โ€” board entries become operational improvements within 48 hours.

The 5-min time-box matters. Longer and the standup turns adversarial โ€” team starts hiding catches. Five minutes, fact-pattern review, named system updates, queue for Tuesday. Owner does not adjudicate individual catches in standup; owner sees the weekly summary Friday and quarterly pattern at governance review.

What Doesn't Go on the Board

The bulletin board is for AI catches, not AI failures. Three categories do not post โ€” the distinction preserves signal-to-noise and protects psychological safety.

Tool downtime or vendor outages. Avoca down for 90 minutes Tuesday is an incident, not a hallucination catch โ€” belongs in IT log or vendor tracker. Board is for AI generating wrong output, not AI failing to generate output.

Process errors by the human. Advisor forgetting to verify is a coaching conversation, not a board entry. CSR skipping the 4 p.m. review is a calendar issue, not a catch. The board credits catches; it does not punish misses. Posting "Marco didn't verify" destroys the social-signal architecture.

Customer complaints without an AI fabrication. A 1-star review about slow dispatch is a complaint, not an AI catch. Goes through the existing customer-recovery workflow; the board stays focused on AI-specific fabrications. Mixing dilutes the operating signal.

The clean rule: AI produced a specific wrong output, human caught it before customer exposure, catch generated a remediation. Everything else lives elsewhere.

The Board and the Quarterly Governance Review

The downstream artifact is the quarterly governance review the owner runs in 90 minutes. The digital mirror feeds the first 20-min block โ€” pull catches, categorize by tool/role/artifact type, surface patterns. Patterns drive block two (tool retention โ€” keep, drop, replace based on catch rate and metric movement) and block three (policy updates โ€” forbidden-phrase list refresh, system-prompt updates, data-policy revisions). Block four plans next-quarter pilots; block five documents decisions and locks the calendar.

The board is the artifact the owner hands to the PE board, franchisor, coach, peer group. Ninety days of dated catches + a categorized digital mirror + Monday-standup cadence + quarterly review memo is what an Authority Brands regional president, Wrench Group portfolio reviewer, Nexstar peer coach, CertainPath member-family advisor, or CFPB examiner credits as reasonable care. The board is not theater; it is the operating system made visible.

Cost and the Cheap Insurance Frame

Hard cost is approximately $40 โ€” a 3x4-foot corkboard, thumbtacks, 3x5 index cards, label maker. Digital mirror is $0 (every shop has Slack/Teams/ClickUp/Notion/FSM tags). Labor is Monday standup (5 min/week) + quarterly governance review (90 min/quarter) โ€” both already on the calendar at any 2026 trades shop with operations maturity.

Against that input: customer commitments avoided ($1,500-$8,000 per warranty catch ร— dozens/quarter), Reg Z/FCRA exposure dodged ($500-$5,000 per regulated-language catch), state contractor board exposure dodged ($2,500-$25,000 per code/warranty catch), license-suspension events avoided (existential), 1-star Google reviews avoided, and compounding effect on team verify discipline. Cheapest insurance any 2026 trades shop carries; highest-leverage artifact any AI deployment produces.

Key Takeaways

  • The bulletin board is the social-signal artifact โ€” Public visibility, immediate reinforcement, named credit drive 3-5x verify-pass adherence at 60-day mark vs. private log alone.
  • Dual format โ€” Physical board (high-traffic area, 3x4 ft, thumbtacks) for social signal; digital mirror (Slack/Teams/ClickUp/Notion/FSM tag) for audit and governance defense.
  • Eight structured fields โ€” Date, role, person (first name + last initial), artifact type, AI tool involved, source-of-truth used, specific fabrication caught, remediation + cost avoided.
  • Example one โ€” Fabricated SEER rating โ€” Trane XR16 3-ton: AI said 18 SEER2, actual 16. ResponsiBid + Titan Intelligence. Spec sheet source-of-truth. Cost avoided $1,800-$2,400 + state contractor board exposure.
  • Example two โ€” Invented warranty year โ€” Goodman GMVC96: AI said 12-year compressor, actual 10. Titan Intelligence proposal builder. Goodman warranty matrix source-of-truth. Cost avoided $4,200 year-11 compressor commitment.
  • Example three โ€” Hallucinated NEC code section โ€” 1968 home 200A panel upgrade: AI cited 230.71 (replaced by 230.67(B) in 2023) and 408.40 (switchboards, not residential panels โ€” correct is 408.41). ResponsiBid + Titan Intelligence. Master electrician verify. System-prompt prohibits all code citations in customer-facing artifacts.
  • Monday 5-minute standup โ€” Read week's catches aloud; group by tool, artifact, checkpoint; surface patterns; queue system-prompt updates for Tuesday. Time-box matters โ€” longer turns adversarial and team hides catches.
  • What doesn't go on the board โ€” Tool downtime, human process misses, customer complaints without AI fabrication. Clean rule: AI produced wrong output, human caught before customer exposure, catch generated remediation.
  • Quarterly governance review โ€” Digital mirror feeds first 20-min block; patterns drive tool retention, policy updates, next-quarter pilots. Board is the artifact the owner hands to PE, franchisor, coach, peer group, regulator.
  • Cost โ€” $40 hard cost (corkboard, tacks, cards, label maker) + 5 min/week Monday + 90 min/quarter review. Returns: warranty commitments avoided ($1,500-$8,000 per catch), Reg Z/FCRA exposure dodged ($500-$5,000 per regulated-language catch), state contractor board exposure dodged ($2,500-$25,000 per code/warranty catch).
  • The board is the operating system made visible โ€” Survives PE quarterly, franchisor audit, contractor-board investigation, CFPB examiner, FTC inquiry. The cheapest insurance any 2026 trades shop carries.