In late 2025 an outbound-email agent at a mid-market SaaS company sent forty-seven copies of the same follow-up message to a single prospect inside eleven minutes. The cause was a partial timeout on the email-send tool: the upstream provider had succeeded on each send, but had returned a slow response, and the agent's framework, configured to retry on slow responses, kept retrying. Each retry was a fresh send. The prospect, a senior buyer at a Fortune 500 account, posted a screenshot of the inbox flood on LinkedIn with the caption "this is how you lose a deal." The deal closed for a competitor. The incident is now the most-quoted example of what the industry calls the "Run Twice" problem — the failure mode where an agent action that should run exactly once runs many times. It is one of the most common production agent incidents of 2025 and 2026, and it is preventable. This lesson is the design discipline that prevents it: idempotency keys, dedup tables, write-once contracts, and the wrapper code that makes "Run Twice" simply not a possible failure.

The "Run Twice" Incident: What Actually Happened

The full anatomy of the 47-email incident is worth reconstructing because the same shape happens in every "Run Twice" case we audit. Specifics anonymized; the sequence is real.

The agent and the tool

The agent was a sales-outreach assistant built on a popular framework. Its job: when a prospect responds to a campaign email, draft a personalized follow-up and send it via the company's SMTP relay. The send was wrapped as a single tool — call it send_followup_email — with parameters (recipient, subject, body, campaign_id). The wrapper called the SMTP relay's REST API.

The trigger

The prospect, John, replied to a campaign at 4:47 p.m. The agent drafted a follow-up at 4:48. The agent called send_followup_email. The SMTP relay was experiencing a slow regional issue. The actual send succeeded on the relay's side but the API response took 11 seconds to return.

The framework default

The agent framework had a default retry policy: retry the tool call if it takes longer than 8 seconds. At 4:48:08, the framework retried the call. The SMTP relay sent another email — and again took 11 seconds. Retry. Retry. Retry. The agent's loop was structured to count retries as agent-step iterations, and the loop ceiling was 50.

By 4:59 p.m., 47 emails had reached John's inbox. The 48th retry returned cleanly within 8 seconds (the regional issue resolved), the agent marked the tool as succeeded, and the loop exited. The agent did not know anything had gone wrong. The audit log showed 47 successful sends; the wrapper's failure logic only triggered on framework-level errors, of which there had been zero.

The damage

By 5 p.m. John had screenshotted the inbox and posted on LinkedIn. By 6 p.m. the post had 8,000 engagements. By Monday morning, the company's CRO had received forwarded copies from three different sales executives at customers. The deal John was in moved to a competitor inside two weeks. Inside the company, a $2 million annual contract was lost; outside the company, the post became referenced in customer-trust conversations for the rest of the quarter.

Total cost: $2M ARR, 4 weeks of incident response, an external customer communication, and a permanent line item in the company's agent-governance review titled "the John incident."

The fix and what it should have been

The fix was straightforward and took 90 minutes of engineering to ship. The wrapper now requires an idempotency_key parameter. The wrapper records (key, result) before calling the SMTP relay. On a retry with the same key, the wrapper returns the prior result without calling the relay again. Same wrapper, three extra fields in a database table, one query before the send call. Total lines of code added: 18.

The lesson — the one this lesson exists to communicate — is that the prevention is cheap, and the absence is expensive. Every team that ships a write tool without an idempotency story is gambling with the same dollar-figure exposure.

The agent that runs the same action 47 times in 11 minutes is not a smart agent doing something dumb. It is a stupid pipe with no memory. The fix is to give the pipe a memory.

Why This Happens in Agents More Than Anywhere Else

The "Run Twice" problem is not new. Distributed systems engineers have been writing about it since the 1980s. What is new is how often it shows up in agent systems, because agents combine three risk factors that are individually manageable and jointly catastrophic.

Risk factor one: the retry loop

Every agent framework retries. LangChain retries on tool errors and on certain timeout conditions. LangGraph retries by default in its node-execution model. The OpenAI Assistants API retries internally on certain transient errors. Lindy, Relevance AI, and n8n all expose retry settings that default to "on." Retries are correct in principle — transient errors deserve retries — but they multiply through tool calls that are not retry-safe.

Risk factor two: tools that do real-world side effects

Pre-agent automation systems often called tools that were already idempotent (an UPSERT in a database, an HTTP GET). Agent tools are different. They send emails. They charge cards. They create JIRA tickets. They post to Slack. These are real-world side effects. The first time you call them, something happens. The second time you call them, something happens again — twice.

Risk factor three: opaque tool internals

The agent does not know what the tool does internally. From the LLM's perspective, every tool is a black box. If the framework decides to retry a tool, the LLM has no way to inspect whether the prior call had a side effect. The LLM cannot reason about safety; only the wrapper code can.

Combine the three: a system that retries by default, a tool surface full of side effects, and a model that cannot inspect the side effects. "Run Twice" is the default failure mode. Idempotency is the non-default design.

Idempotency Keys: The Primary Defense

An idempotency key is a unique identifier the caller provides with each logical operation. The wrapper records (key, result) on the first call. On subsequent calls with the same key, the wrapper returns the prior result without re-executing the side effect.

The contract in three sentences

An idempotent tool guarantees: (1) two calls with the same idempotency key produce the same observable result; (2) only one set of side effects occurs in the external world; (3) the second call returns the result of the first call, not an error.

This contract is well-defined in REST API design — Stripe popularized it in 2014 with their idempotency-key header — and is well-supported by major SaaS vendors. The 2026 version of agent tooling makes idempotency keys a first-class parameter on every write tool.

Where the key comes from

The agent's wrapper is responsible for generating or accepting the key. There are three common patterns:

1. Caller-provided. The agent's orchestrator generates a UUID per logical operation and passes it as the idempotency_key. This is the most flexible and is the pattern most production teams use.
2. Derived from request content. The wrapper hashes the request parameters (recipient + body hash + campaign_id) and uses that hash as the key. This works for content-deterministic operations but breaks when the same content is intentionally sent twice (e.g., a deliberate re-send).
3. Time-windowed. The key combines content with a time bucket (e.g., the same email to the same recipient within a 24-hour window has the same key). This is a compromise that prevents accidental duplicates while allowing intentional re-sends after the window expires.

The right choice depends on the tool's semantics. Email sending is usually pattern 2 or 3. Payment authorization is always pattern 1, because the same charge for the same amount on the same card on the same day might legitimately be two separate transactions.

Storing the key

The wrapper needs a store for (key, result) tuples. Options ranked by adoption in production agent systems:

• Redis with TTL. Most common. 24-hour to 30-day TTL depending on the tool's semantics. Fast, simple, fits in any infrastructure that already has Redis.
• Postgres table. Best when you want the idempotency log to be queryable for audit. Schema: (key, tool_name, request_hash, result_json, created_at, ttl_expires_at).
• Vendor-native. Stripe, Square, PayPal, and most modern payment APIs support an Idempotency-Key HTTP header natively. The wrapper passes through; the vendor handles dedup. This is the simplest path when the vendor supports it.
• DynamoDB or similar key-value store. Used in larger deployments where Redis-as-cache is not durable enough. Conditional writes guarantee single-execution semantics.

The store's job is to be authoritative for "did this key already produce a result." Whichever store you pick, the wrapper checks the store first, returns the prior result if present, and writes the new result before returning if absent.

The Three Design Patterns That Prevent Run Twice

Idempotency keys are the primary defense. Two adjacent patterns close the remaining gaps. Together they prevent essentially every "Run Twice" failure mode we have audited.

Pattern one: idempotency keys (as above)

The wrapper requires a key. The store returns prior results. The vendor never sees a duplicate.

Pattern two: dedup tables for content-based deduplication

Some tools cannot be made idempotent through keys alone, because the "duplicate" is defined by content rather than by call. Example: the agent might generate the same outbound email body for two different prospects within an hour because the prompt produced similar drafts. The keys differ (each prospect has their own key), but the system should still recognize that 47 emails from the same agent to the same prospect inside an hour is suspicious.

The dedup table sits between the agent and the tool. Before the wrapper sends, it queries: "has this agent sent to this recipient with this content hash within window W?" If yes, the wrapper either blocks the call (strict mode) or sends and records a warning (audit mode). The threshold is configurable per tool; for outbound email, 1 send per recipient per 24-hour window is typical.

This is the pattern that would have caught the "John incident" even in the absence of idempotency keys. The wrapper would have seen 1 send recorded at 4:48, then refused to send a second at 4:48:08 because the dedup window had not expired.

Pattern three: write-once contracts

Some operations should structurally only be possible once. Creating a customer record. Issuing a refund. Closing a JIRA ticket. The write-once contract is enforced at the wrapper level: the wrapper records the operation in a dedicated "completed operations" table, keyed by the operation's natural identifier (e.g., refund_id, customer_id, ticket_id). Any subsequent call with the same natural identifier returns the prior result without re-executing.

Write-once is stricter than idempotency keys because it ignores the key and uses the natural identifier. If you call "issue refund for transaction T" once and then again, the second call returns the first refund — even if the agent provided a fresh idempotency key. This protects against agent retries even when the wrapping is imperfect.

Implementation: a completed_operations table with (operation_type, natural_id, result, created_at). The wrapper checks before executing and writes after. A unique constraint on (operation_type, natural_id) guarantees database-level enforcement even if the wrapper has a race condition.

Seven Tools and How to Make Each Idempotent

Real tools have different idempotency strategies. Here are seven we have seen audited in 2026, each with the recommended approach.

send_email

Strategy: idempotency key + dedup table. The key prevents framework-retry duplicates. The dedup table (recipient + content_hash + 24h window) prevents content-similar duplicates from any source. Vendors like Postmark and SendGrid support an idempotency header natively; pass through where available.

charge_payment

Strategy: vendor-native idempotency key. Stripe and Square both support the Idempotency-Key header. The wrapper generates a UUID per logical charge, stores (key, result) in Postgres for audit, and passes the header to the vendor. The vendor guarantees single-execution.

create_jira_ticket

Strategy: write-once on a natural identifier. The natural identifier is (project, customer_id, issue_description_hash, 24h window). The wrapper checks completed_operations first; if a matching ticket exists in the window, returns its ID. Prevents duplicate tickets when the agent retries.

update_salesforce_field

Strategy: idempotency key + state check. The wrapper records (key, prior_value, new_value, success). On retry with the same key, returns the prior result. The state check is a guard: the wrapper reads the current field value, and if it already equals the new value, returns success without writing. This handles the case where the original write succeeded but its response was lost.

post_slack_message

Strategy: dedup table. Slack messages are intentionally allowed to repeat (an agent might legitimately post status updates), so an idempotency key is too strict. The dedup table catches "exact same message to the same channel within 5 minutes" and is the right fit. Pattern is "send and record; on duplicate within window, log a warning and skip."

schedule_calendar_event

Strategy: write-once. The natural identifier is (calendar_id, start_time, attendee_emails_hash). Duplicate scheduling is a common failure mode and write-once eliminates it. The wrapper returns the existing event ID rather than creating a second event.

execute_database_write

Strategy: never expose a freeform write tool to an agent. Wrap specific writes (update_customer_status, log_event, archive_record) and apply idempotency keys to each. This is also Lesson 1's "kitchen-sink query" anti-pattern in another form.

Testing Idempotency: The Replay Test

Every wrapped tool that claims to be idempotent needs an explicit test. We call this the replay test, and it is the single test that catches the most "this looked idempotent in design but isn't" bugs.

The replay test in five lines

1. Call the tool with a specific set of parameters and idempotency_key K.
2. Record the response and the external side effect (the email that was sent, the row that was inserted, the charge that was made).
3. Call the tool again with the same parameters and the same K.
4. Assert that the response is byte-identical to the first response (allowing for timestamp fields if those are documented).
5. Assert that no additional external side effect occurred (no second email sent, no second row inserted, no second charge).

The fifth assertion is the part teams skip and is the part that catches the most bugs. The first four can pass while the underlying vendor silently created a duplicate; only the explicit side-effect check confirms idempotency end-to-end.

How to verify side effects

For email: check the SMTP relay's send log; should show exactly one send for the (recipient, body_hash) pair. For database writes: run a SELECT COUNT before and after; difference should be zero on the retry. For payments: check the payment gateway's transaction log via API; should show exactly one transaction with the idempotency key. For external webhooks: subscribe to the vendor's audit feed and confirm one fire per logical call.

Each verification path is vendor-specific. Build it once per tool, run it in CI, and the "Run Twice" failure mode becomes structurally impossible.

When to Use Which Pattern

A simple decision flow we use in audit reviews:

1. Is the tool a read? No defense needed beyond standard rate limits. Reads are idempotent by definition.
2. Is the tool a write to an external system (email, payment, JIRA)? Require an idempotency key. If the vendor supports an Idempotency-Key header, pass through. Otherwise, store (key, result) in Redis or Postgres.
3. Is the duplicate-detection concern about retries (same call, accidentally) or about content similarity (different calls, similar content)? Retries → idempotency keys. Content similarity → dedup table.
4. Does the operation correspond to a "happens once in the business meaning" event (refund, ticket close, calendar event creation)? Add a write-once contract on the natural identifier.
5. Should the operation be repeatable within some window (status updates, notifications)? Dedup table with a short window. Idempotency keys would be too strict.

Apply at design time. Each tool gets one or more of the three patterns explicitly chosen. The wrapper documents the choice in its description so the next maintainer can see it.

The Cost of Getting This Wrong: Real Numbers from 2025-2026 Incidents

"Run Twice" is not a hypothetical concern. From our audit reviews and from public incidents, here are real costs from 2025-2026:

• The John incident (47 emails, late 2025): $2M ARR lost, 4 weeks of incident response, permanent customer-trust impact. 18 lines of code prevented.
• Duplicate Stripe charges (mid-2025): A travel-booking agent retried 4 times on a slow Stripe response. Each retry charged the card. The customer was charged 4x for one trip. Refund process took 11 business days. Public Trustpilot review, 1 star. Vendor-native idempotency key would have prevented entirely.
• Duplicate JIRA tickets (Q1 2026): A customer-support agent created 312 duplicate tickets in 48 hours because the retry was at the framework level and the wrapper had no dedup. The engineering team spent 22 hours triaging and merging the duplicates.
• Duplicate Slack notifications (Q2 2026): An incident-response agent re-posted the same alert 18 times to a channel. On-call engineers muted the channel during a real incident. Two-hour delay in real-incident response. Direct customer impact.
• Duplicate calendar events (Q1 2026): A scheduling agent created 7 copies of the same customer call on a sales executive's calendar. Customer's calendar refused new bookings for the rest of the week because of "too many conflicts." Three meetings missed.

Pattern across all five: the prevention was cheap (lines of code, hours of design). The absence was expensive (revenue, time, customer trust). Every team that ships a write tool without idempotency is one bad retry from one of these stories.

The Six-Line Idempotency Wrapper

The defensive pattern in pseudocode, applied to any write tool:

def call_idempotent_tool(params, idempotency_key):
    cached = store.get(idempotency_key)
    if cached:
        return cached
    result = vendor.execute(params)
    store.set(idempotency_key, result, ttl=30_days)
    return result

Six lines. The store is Redis or Postgres. The vendor is the SaaS API. The wrapper is the agent tool. This is the entire defense for the "retried tool call" case.

Production wrappers add: a race-condition guard (set the key with a "pending" sentinel before calling the vendor, swap to the actual result after success), an error policy (don't cache vendor errors that should be retried; do cache vendor errors that should not be), and an audit log row per call (including whether it was a cache hit or a fresh call).

The expanded production version is roughly 35 lines. Still cheap. Still prevents every retry-induced "Run Twice" incident.

The Handoff to Permission Scoping

Idempotency is the defense against running the right action too many times. Lesson 4 of this chapter covers the adjacent defense: not running the wrong action at all. Permission scoping is the discipline of stripping write access from tools where read is enough, applying OAuth scope discipline (crm.read instead of crm.full_access), and walking through a real scope review. Together, idempotency and scoping cover most of the production-action failure modes that have made the news in 2025-2026.

Build the wrap (Lesson 2), make it idempotent (this lesson), scope it tightly (Lesson 4). That is the three-layer discipline behind every production-grade agent tool surface in 2026.

Key Takeaways

• "Run Twice" is the failure mode where an agent action that should run exactly once runs many times. The famous late-2025 incident: an outbound-email agent sent 47 copies of the same follow-up to one prospect in 11 minutes because the SMTP relay's slow responses triggered framework-level retries. Cost: $2M ARR lost, four weeks of incident response, permanent customer-trust impact. Prevention: 18 lines of code.
• Three risk factors make "Run Twice" the default failure mode for agents: frameworks retry by default (LangChain, LangGraph, OpenAI Assistants, Lindy, Relevance AI, n8n); tools have real-world side effects (email, payment, ticket creation); the LLM cannot inspect tool internals to know whether a retry is safe.
• The primary defense is an idempotency key: a unique identifier the caller provides with each logical operation. The wrapper stores (key, result) on first call and returns the prior result on retries with the same key. Stripe popularized the pattern in 2014; vendor-native support is common across modern SaaS APIs.
• Three key-generation patterns: caller-provided UUID (most flexible, most common), derived from request content hash (works for content-deterministic operations), time-windowed key (compromise that prevents accidental duplicates while allowing intentional re-sends after window expires).
• Storage options for (key, result) tuples: Redis with TTL (most common, 24h-30d), Postgres table (when audit queryability matters), vendor-native Idempotency-Key header (Stripe, Square, PayPal), DynamoDB or similar (durable key-value store for larger deployments).
• Three design patterns together prevent essentially every audited "Run Twice" case: (1) idempotency keys for retry-induced duplicates, (2) dedup tables for content-based duplicates, (3) write-once contracts for operations that structurally happen once in business meaning (refund, ticket close, calendar event creation).
• Per-tool strategy: send_email = key + dedup table; charge_payment = vendor-native key; create_jira_ticket = write-once on natural identifier; update_salesforce_field = key + state check; post_slack_message = dedup table only (repeats sometimes legitimate); schedule_calendar_event = write-once; execute_database_write = never expose freeform, wrap specifics.
• The replay test catches "looked idempotent but isn't" bugs: call with key K, record response and side effect; call again with same K; assert response byte-identical AND no additional side effect occurred. The fifth assertion (no additional side effect) is the part teams skip and the part that catches the most bugs.
• Real 2025-2026 incident costs: 47-email incident ($2M ARR), duplicate Stripe charges (11-day refund, 1-star Trustpilot), 312 duplicate JIRA tickets (22 hours triage), 18 Slack re-posts (2-hour delay in real-incident response, customer impact), 7 duplicate calendar events (3 meetings missed). All preventable; prevention was cheap.
• The six-line idempotency wrapper: check store for key; return cached if present; execute vendor; store result; return result. Production version expands to ~35 lines with race-condition guard, error policy, and audit log row per call.
• Decision flow: reads need no defense; writes require idempotency keys; retries vs content-similarity drives keys-vs-dedup-table choice; operations that "happen once in business meaning" need write-once contracts on the natural identifier; repeatable-within-window operations use dedup tables with short windows instead of keys.