A vendor's agent demo on a LinkedIn livestream is the most beautiful 90 seconds of marketing in 2026. Then you take the same agent into your real environment and it falls over inside a week. The gap isn't bad luck. It is structural — and it's diagnosable with a fifteen-minute audit.

The Shape of the Gap

Every operator-builder has watched the same demo a dozen times. A founder or product person shares their screen, types a casual instruction into a chat window, and the agent gracefully reads three SaaS systems, drafts a beautiful response, files it in the right place, and gets a Slack thumbs-up. The audience claps. The pricing page gets traffic. A purchase order gets approved.

Then the buyer's team installs it. In their environment, the agent connects to a tenant that has 18 months of edge-case CRM data, not the demoer's clean sandbox. The user prompts vary — some are typos, some are passive-aggressive Slack messages, some are forwarded customer emails with PDF attachments. The agent's tools point at production data instead of read-only fixtures. Within a week, the agent has gotten a customer's record confused with another customer's, triggered an embarrassing email, and silently rolled up a $1,200 token bill that nobody can break down.

This is the demo-to-production gap. Vendors don't lie in their demos; they just show you the version of the system that works. Your job as the operator-builder is to know what to ask for to see the version that breaks.

The demo is a story about the system. Production is a stress test of the story. Almost every demo-to-production failure traces back to three missing artifacts: an eval set, an identity model, and observability.

Reason One: Missing Eval Set

In the demo, the demoer asked the question they knew worked. They typed it carefully. The example was hand-curated. Maybe the demo has been polished over fifty practice runs. Maybe the prompt was adjusted three minutes before going live. None of that is dishonest — it is just the nature of a demo. But it means the demo proves this question, in this state, produces this output. It does not prove anything about your questions, your state, or your output.

What an eval set actually is

An eval set is a list of input/expected-output pairs that you score the agent against. For a customer-support agent, it might be 200 real customer emails (sanitized) with the human-reviewed correct action for each. For a lead-routing agent, it might be 500 inbound leads with the correct rep assignment. The agent runs the eval set, and you score the percentage of outputs that match the expected answer (or fall within an acceptable range).

This is the single thing that separates "we built an agent" from "we shipped an agent that works." Without an eval set, you have a vibe. With an eval set, you have a number that goes up or down when you change things.

The hand-curated demo

Here is a real pattern from a vendor demo in early 2026: the demoer typed "Can you check why this customer hasn't paid?" The agent gracefully pulled from Stripe, found a failed payment with a clear error code, and drafted a polite email. In the audience Q&A, someone asked, "What happens if the customer name doesn't match what's in Stripe exactly?" The demoer paused, ran a different version of the question, got a different (less impressive) result, and pivoted. That pause is the demo-to-production gap visible in real time. The system worked for the curated example and not for the off-curve one.

How to ask for an eval set in a vendor pitch

Five questions that expose whether a vendor has an eval set or just demos:

1. "Can you share a benchmark of how this agent performs on a fixed eval set of 100+ examples?"
2. "What's the agent's accuracy on the eval set? What's the failure breakdown?"
3. "How often do you re-run the eval set? After every model update?"
4. "Can I add 20 of my own examples to the eval set and see how it scores?"
5. "What was the eval set score on the most recent model version vs. the previous one?"

If the vendor doesn't have crisp answers, they're shipping vibes. That's not necessarily a deal-breaker, but it tells you exactly how much production engineering you'll be doing yourself.

Reason Two: Missing Identity Model

In the demo, the agent ran on the demoer's account, with the demoer's credentials, in the demoer's tenant. The demoer is an admin. The credentials have god-mode. The tenant is one company.

In your production environment: you have many users, many roles, many tenants (if you're a SaaS), data segmentation by territory or team, audit requirements, compliance constraints (SOC2, HIPAA, GDPR), and an EU AI Act Article 26 deadline coming in August 2026. The same agent that worked in the demo cannot just be plugged into your environment. It needs an identity model.

What an identity model is

An identity model answers: when this agent takes an action, on whose behalf is it acting, and what is that identity allowed to do? Three layers:

• User identity: the agent is acting on behalf of a specific user. Tool calls inherit that user's permissions. If the user can't see record X, the agent can't see record X.
• Agent identity: the agent itself has a service account with its own permission set. Even when acting "on behalf of" a user, the agent's actions are scoped to what the agent role is allowed to do.
• Tenant scoping: in multi-tenant SaaS, every tool call must be scoped to the correct tenant. Cross-tenant data leaks are a category-of-existence-ending bug.

The demo's credential problem

The demoer used personal credentials with broad scope. When you install the agent, you're typically asked to authenticate the agent with your account. The most convenient path is to give it an OAuth scope that mirrors the demoer's god-mode access. Don't. Instead, create a dedicated service account, scope it to the minimum required, and let the agent inherit user identity via downstream tool calls (OAuth-on-behalf-of, SCIM-style sub-claims, or whatever your stack supports).

The cross-tenant nightmare

A 2025 incident at a B2B SaaS company (anonymized in the engineering postmortem): the company built an AI customer-success agent for its own customers. The agent had a tool, get_account_summary(account_id), that queried the production database. It worked in testing. In production, a customer's user triggered the agent, which called get_account_summary — and because the agent was running with the SaaS company's god-mode service account, the agent could query any account, including other customers'. The agent didn't intend to leak data. But because the identity model was wrong, the tool was capable of returning data from another tenant. The company caught it in QA before public release. It would have been a six-figure incident.

EU AI Act Article 26 implications

From August 2, 2026, EU AI Act Article 26 binds deployers of high-risk AI systems to maintain identity-traceable logs, ensure human oversight, and conduct fundamental rights impact assessments. An agent without an identity model — where you can't say who took an action and what their authorization was — cannot be deployed in a regulated EU context. The demo never shows the identity model because there isn't one to show.

Reason Three: Missing Observability

In the demo, the agent's output was the visible artifact. The drafted email. The completed Notion page. The Slack message. What you didn't see: the trace, the cost, the tool calls, the retries, the errors, the time-to-first-token, the loop iterations, the memory state.

In production, the artifact is one tenth of what matters. The other nine tenths is the operational data: did this agent do what it was supposed to do, at what cost, with what side effects, and how can we replay it when it goes wrong?

The observability hierarchy

For an operator-builder shipping an agent in 2026, observability has five levels of maturity:

1. Level 0 — Nothing: the agent runs, you see the final output, that's it. (Most demos.)
2. Level 1 — Logs: tool calls and model calls are logged somewhere. You can grep them.
3. Level 2 — Traces: structured traces per run, with thought/action/observation, viewable in a UI (LangSmith, Helicone, Langfuse, Arize, Datadog LLM, OpenAI logs UI).
4. Level 3 — Metrics: aggregate metrics per agent — runs/day, success rate, p50/p99 latency, cost/run, token use distribution, error rate.
5. Level 4 — Alerting: push alerts on metric thresholds and anomalies; on-call rotation owns response.

A demo is at level 0. A production agent should be at level 3 or 4. The audit asks: what level is the system you're being sold? If the answer is level 0 and the vendor expects you to bolt on observability yourself, that's significant engineering work you didn't budget.

Why observability is the first thing to cut

Observability is invisible to the buyer. It produces no visible feature. It doesn't show up in the demo. It is the thing engineering teams cut first when they're under pressure to ship. So vendors routinely ship Level 0 or Level 1, then promise "you can integrate with your SIEM" as the answer. That promise is doing a lot of work. Verify before signing.

The replay problem

When an agent does something wrong in production, the first question is "can you show me what happened?" Without traces, you cannot. You see the final output and have to guess. Replay — the ability to reconstruct a run from logged traces — is the single most useful capability for an ops team running agents. Vendors that don't have replay are vendors whose incidents you'll be diagnosing by candlelight.

The Rollback Question

A fourth, less-celebrated gap that comes up in real deployments: what's the rollback path?

In a demo, the agent runs once, produces an output, and the demoer moves on. In production, the agent runs thousands of times and occasionally takes irreversible actions — sends an email, issues a refund, closes a ticket, posts to social. When the agent makes a mistake at scale, the question is not "can we stop it?" (that's the kill switch). The question is "can we undo what it did?"

Reversibility per action

Every action your agent can take should be classified for reversibility:

• Trivially reversible: updating an internal Airtable status, marking a Slack message read. Just write the previous value back.
• Reversible with friction: sending a Slack message (delete it; people may have seen it). Updating a CRM stage (revert the stage; reports may have run).
• Hard to reverse: sending an external email to a customer. You can send a correction, but the original was seen.
• Irreversible: issuing a payment, closing an account, posting to public social media, deleting data.

Irreversible actions should require HITL approval before execution. Hard-to-reverse actions should be rate-limited and require manual override above a threshold. Trivially reversible actions can be automatic.

The rollback test

Before you ship an agent, ask: "If this agent did something wrong to 100 records yesterday, what's our rollback?" If the answer involves a script someone will write reactively, the rollback path is missing. If it involves restoring from a backup that's 24 hours old, you've discovered the data-loss cost of every incident. The right answer is: "We have a documented rollback procedure that's been tested, and the irreversible actions are gated behind HITL, so the blast radius of any incident is bounded to N reversible-with-friction actions."

The Demo Reality Audit (Fifteen Minutes)

This is the artifact. Use it on any vendor pitch, any internal proposal, any "let's just try Lindy for this." Fifteen minutes. Five sections. Three questions each.

Section 1: Eval Set

1. What's the eval set this agent is benchmarked against? (Looking for: a real list of inputs, not "we tested it.")
2. What's the accuracy on the eval set, and what's the failure breakdown by category? (Looking for: a number and a histogram, not "it's pretty good.")
3. Can we add 20 of our own real examples to the eval set and see the score live? (Looking for: yes within an hour, not "we'll need engineering.")

Section 2: Identity Model

1. When the agent takes an action, whose identity is it acting under? Show us in the audit log. (Looking for: a real principal, not "the service account.")
2. How does the agent inherit user permissions when it queries data? (Looking for: OAuth on-behalf-of, SCIM scopes, or explicit per-call authorization — not "it has admin.")
3. What prevents the agent from querying data outside the current tenant? (Looking for: tenant scoping enforced at the tool layer, not "we trust the prompt.")

Section 3: Observability

1. Show us the trace UI. What does an individual run look like? (Looking for: per-iteration thought/action/observation with cost, identity, latency.)
2. What metrics are aggregated, and where do alerts go? (Looking for: dashboards, push alerts, on-call rotation.)
3. Can we replay yesterday's run X with a single click? (Looking for: yes — replay is a real feature.)

Section 4: Cost Controls

1. What's the per-run cost ceiling, enforced by the runtime? (Looking for: a real cap, not an aspiration.)
2. What's the daily aggregate $-cap, and what's the alert threshold? (Looking for: numbers, not "we'll monitor.")
3. Where is the kill switch, and when was it last tested? (Looking for: a tested mechanism, not "we can revoke the API key if we have to.")

Section 5: Rollback

1. Classify the agent's actions by reversibility. Which are irreversible? (Looking for: an explicit map.)
2. What HITL gates exist for irreversible actions? (Looking for: gates, not "we trust the prompt.")
3. If the agent mis-handled 100 records yesterday, what's the rollback? (Looking for: a documented, tested procedure.)

If any vendor or internal proposal cannot answer 80% of these crisply, you have just identified the demo-to-production gap. Send the audit back; ask for the remediations before you sign or deploy.

Three Illustrative Demo Gaps

The "agent that worked because the demoer hand-curated the inputs"

A sales-enablement vendor in Q1 2026 demoed an agent that "writes personalized outbound based on the prospect's LinkedIn activity." Watching the demo, the agent produced a strikingly good email referencing the prospect's recent post about expanding into Europe. The Q&A revealed that the demoer had pre-curated the prospect's LinkedIn data — pulled it that morning, cleaned it, and fed the agent a structured summary. When a prospect was randomly selected from the audience's request, the agent produced an email that confused two different people with the same name. The demo proved the system worked on cleaned inputs. Production has dirty inputs. The eval set would have shown this.

The "agent that had no tool boundaries because it ran on the demoer's personal credentials"

An ops-automation vendor showed an agent that "manages your entire RevOps stack." The agent reached into HubSpot, Salesforce, Slack, Notion, and Stripe in a single workflow. It worked beautifully. When the audience asked "what scopes does it need?", the answer was "admin access to all the connected apps." When asked "how does it know which user is asking?", the answer was "we use the connected admin's credentials." This is not malicious. It is a demo built without an identity model. Plugged into a regulated environment, this agent would be a SOC2 violation in week one.

The "agent with no rollback path"

A customer-support vendor demoed an agent that auto-resolves 60% of inbound tickets — closing them, sending the resolution email, and tagging the ticket. When asked "what happens if the agent misclassified a P1 ticket as P3 and closed it?", the demoer said "well, the customer would re-open it." That's the rollback: the customer notices and complains. For low-stakes products, that might be acceptable. For a healthcare or fintech customer base, that's a regulatory incident. The rollback question would have surfaced this in fifteen minutes.

The Builder Mindset Shift

The deepest lesson of demo-to-production is a mindset shift that every operator-builder eventually internalizes: shipping the demo is the easy 10%. The hard 90% is everything you don't see in the demo.

That 90% is the eval set, the identity model, the observability, the rollback, the cost controls, the on-call rotation, the documentation, the runbooks, the compliance paperwork, and the periodic re-validation. Vendors don't show this in their pitches because it's boring. Buyers don't ask about it because they're excited. The result is the demo-to-production gap.

The Demo Reality Audit is the operator-builder's defense. Run it cold, ask the awkward questions, and require remediation before deployment. Every minute you spend on the audit saves a week of incidents in production.

The vendors and internal builders who pass a real fifteen-minute audit are the ones whose agents survive twelve months of production. Everyone else is selling you a demo that will become your problem.

Key Takeaways

• Demos are stories about systems. Production stress-tests the stories. The gap is structural, not a vendor failure of integrity.
• Three core gaps cause most demo-to-production failures: missing eval set, missing identity model, missing observability. A fourth — missing rollback — bites you on irreversible actions.
• Eval sets separate "we built an agent" from "we shipped an agent that works." Without a benchmark, you have a vibe. With one, you have a number that changes when you change things.
• Identity models prevent cross-tenant nightmares and regulatory disasters. User identity flows through tool calls; agent has its own service account; tenant scoping is enforced at the tool layer. EU AI Act Article 26 makes this non-optional from Aug 2026.
• Observability is the first thing vendors cut and the first thing you need. Trace UI, structured metrics, push alerts, on-call. Level 0 (nothing) is the demo default. Level 3-4 is production minimum.
• Rollback paths matter at production scale. Classify actions by reversibility. Irreversible actions require HITL. Hard-to-reverse actions need rate limits and thresholds.
• The Demo Reality Audit is a 15-minute, 5-section, 15-question framework. Run it on every vendor pitch and every internal proposal. Require remediation before deployment.
• Shipping the demo is the easy 10%. The hard 90% is eval sets, identity, observability, rollback, cost controls, on-call, documentation, compliance, and re-validation. The audit makes the invisible visible.