On a Wednesday in February 2026, n8n shipped a routine LangChain-node update. The release notes mentioned a tool-arg schema change. Nothing in the changelog suggested the change would simultaneously break OpenAI and Anthropic tool calls across upgraded workspaces. The teams that auto-upgraded woke up Thursday morning to silent failures: agents calling tools and getting back schema-mismatch errors that the platform was swallowing, agents falling back to text outputs instead of structured tool invocations, and zero visible signal in dashboards that did not include eval. By Friday afternoon, a handful of teams had diagnosed the issue. By the following Monday, half the affected teams had still not noticed. The teams that caught it within hours had one thing in common: they had a post-upgrade eval run wired into their platform-version change management. The teams that took weeks all had the same gap: they treated platform upgrades as routine maintenance instead of as model changes. This lesson is the discipline that closes that gap. The full eval set on every platform upgrade, every model version change, every prompt change. The discipline that distinguishes platforms from utilities. The story of the February incident and the system the teams built afterward.

The February 2026 n8n Incident

The trigger was a minor version update to n8n's LangChain node — the component that wraps LangChain agent loops inside the n8n workflow runtime. The update modified how tool-call arguments were serialized into the request payload sent to OpenAI and Anthropic.

What changed

The previous version sent tool-call schemas in a format both OpenAI and Anthropic accepted. The new version changed the serialization in a way that caused both providers to reject the tool definitions as malformed. Specifically: an enum field in the JSON Schema definition was being serialized as a nested object instead of as an inline array, which caused OpenAI's function-calling validator to silently strip the tool from the request and Anthropic's tool_use validator to return an unhelpful 400.

The bug was caught by both providers' validators differently. OpenAI's behavior: the API returned a 200, the LLM ignored the malformed tool definition, and the agent responded with a natural-language answer instead of calling the tool. The agent appeared to be working. Users got plausible-looking responses. The fact that the tool was never called was invisible without a trace.

Anthropic's behavior: the API returned a 400 error, which the n8n node was configured to retry. Three retries failed. The node fell back to its error-handling branch — which for many workflows meant returning an empty result or a default response. Again: users got plausible-looking responses. The fact that the agent had failed to call any tools was invisible.

Who noticed and when

The teams running an eval suite as part of their post-upgrade verification caught the issue within hours. Their nightly eval suite scored materially lower on the morning after the upgrade. The smoke eval that ran on a sample of recent production traffic — what some teams call the "shadow eval" — detected the tool-call regression and fired an alert.

The teams that did not run a post-upgrade eval did not notice for days or weeks. The dashboard signals were ambiguous: latency was normal, error rate was within tolerance, cost was actually lower (no tool calls meant cheaper runs). The agent was clearly doing something different than before, but no automated signal told the team what or when.

For some teams, the discovery came from a user complaint: "the agent told me X instead of doing Y." A support engineer reproduced the case, found the missing tool call, traced it back to the platform upgrade. Eight to fourteen days of degraded service had elapsed.

The February 2026 n8n incident is the canonical example of why platform upgrades are not maintenance — they are model changes. The platform's reasoning runtime, the wrapper that mediates between the agent's logical flow and the underlying model API, is part of the agent's behavior contract. When the runtime changes, the agent's behavior can change in ways that no team-side commit would predict. The eval run on every platform upgrade is the only reliable signal that catches the failure mode.

The two failure-detection patterns that worked

Teams that detected within hours used one of two patterns. The first: a scheduled CI job runs the full eval suite immediately after a platform upgrade is applied, before traffic is routed back to the upgraded environment. The eval fails, the deployment is rolled back, the team is paged. The platform upgrade is treated like any other change to the agent's runtime — it gates on the eval.

The second: continuous shadow eval on production traffic. A sampled fraction (1-5% of runs) is scored in real-time by an LLM-as-judge. The judge does not block the production response; it produces an asynchronous score that goes into a rolling-window trend. When the trend drops below a threshold, an alert fires. The shadow eval caught the February incident within hours because the trend dropped immediately after the upgrade.

Teams that had neither of these patterns relied on user complaints. Average time to detect: 8-14 days.

Why Platform Upgrades Deserve the Full Eval

The instinct: platform upgrades are infrastructure changes. The agent's behavior is defined by the prompt, the model, and the tools. None of those changed. Why would the agent's behavior change?

The reality: platforms are agents' runtime. The runtime mediates between the agent's logical flow and the underlying model and tool APIs. When the runtime changes — in serialization, in retry behavior, in default settings, in how it parses model outputs, in how it constructs tool definitions — the agent's behavior changes. The platform is part of the system being tested.

The list of runtime layers that affect agent behavior

• Tool-call serialization. How tool definitions are formatted in the request. The February n8n incident lived here.
• Tool-result parsing. How the response from a tool is processed before being passed back to the LLM. Changes in parsing can change what the LLM sees and how it reasons.
• Retry logic. When the underlying provider returns an error, what does the platform do? Changes in retry counts, retry timing, or fallback behavior cascade into agent behavior.
• System prompt assembly. Platforms often inject system-level prompts (date stamps, agent identity, tool descriptions) on the team's behalf. Changes in this assembly are functionally changes to the system prompt.
• Response streaming. Streaming behavior — how the platform buffers, when it surfaces partial responses, how it handles mid-stream tool calls — affects user-perceived behavior.
• Output parsing. The platform parses the model's response to determine whether it is a tool call, a final answer, or something else. Parsing changes affect whether the agent loops correctly.
• Token counting and limits. How the platform counts tokens for context-window management can change which messages are included or truncated.
• Tracing and logging. Less directly behavior-affecting but: changes here affect the team's ability to diagnose issues after the fact.

Any of these layers can shift in a minor platform release. The release notes typically describe only the surface change. The behavioral consequences are not usually documented. The only reliable detection: run the eval suite against the upgraded runtime.

The minor-version trap

Major version upgrades (n8n 1.x to 2.x, Lindy's redesigned agent runtime, Make's "Agentic Studio" rollout, LangGraph 0.x to 1.0) tend to get appropriate scrutiny. Teams plan migrations, they test, they read release notes carefully.

The trap is the minor versions. n8n 1.62.1 to 1.62.2. Make's weekly silent updates to its platform. Lindy's continuous improvements that ship without team-side notification. The minor versions are where the February incident lived — a routine version bump that should have been routine.

The discipline: treat minor versions as changes-to-the-agent's-runtime, not as platform maintenance. The full eval runs on every minor version because the team has no way to know which minor version will be the one that breaks something.

The Three Platform-Version Triggers

Three categories of upgrade trigger the full eval. Teams write these into the change-management policy explicitly.

Platform/runtime upgrades

n8n. Lindy. Make. Zapier (when the agent uses Zapier AI Actions). Microsoft Copilot Studio. Salesforce Agentforce. LangGraph Platform. LangChain SDK. LlamaIndex. Anthropic SDK. OpenAI SDK. Any package or platform whose code runs between the agent's prompt and the model API.

For self-hosted platforms, version is straightforward. For SaaS platforms that auto-upgrade, the team has to subscribe to release notes, monitor the platform's public changelog, and where possible disable auto-upgrade in favor of a controlled rollout. Some platforms allow workspace-level version pinning (n8n self-hosted, certain Lindy enterprise tiers); others do not (Make's silent updates).

For SaaS platforms that do not allow version pinning, the production pattern is the continuous shadow eval: the team cannot prevent the upgrade, so they detect the consequences immediately.

Model version changes

Direct: the team explicitly swaps GPT-5 for Claude Sonnet 4.5, or upgrades from gpt-5-2026-04 to gpt-5-2026-05. These are obvious changes that warrant a full eval.

Silent: the provider upgrades the model behind a non-versioned endpoint. Calling "gpt-4o" or "claude-3-5-sonnet" without an explicit version suffix gets the provider's current preferred version, which may change at any time.

The discipline: pin model versions explicitly. Every model call specifies the version (gpt-5-2026-04-15, claude-sonnet-4-5-2026-03-20, gemini-2-5-2026-02-10). When the team decides to upgrade, they do so as an intentional change with an eval run, not as an accident.

Prompt changes

Already covered in detail in Level 3, Chapter 5. Restated here for completeness: every prompt change runs the smoke eval. Every prompt change that touches the system prompt, the user prompt template, or any instructional text triggers the eval. A single word can move 5-15 cases on the eval set.

The Eval Architecture for Platform Upgrades

The eval architecture has three modes that map to platform-upgrade scenarios.

Mode 1: pre-upgrade staging eval

For platforms that allow controlled upgrades (self-hosted n8n, planned LangGraph Platform deployments, Anthropic SDK upgrades managed in the team's repo), the eval runs in staging before production is touched.

The flow: deploy the new platform version to a staging environment. Wire the same agent (same prompts, same tools, same model versions) to the staged platform. Run the full eval suite. Compare against the production baseline. If the eval passes, proceed with the production upgrade. If it fails, diagnose, file a bug with the platform vendor if appropriate, decide whether to upgrade with the workaround or stay on the previous version.

This is the cleanest model. The team controls the upgrade timing and can defer if needed.

Mode 2: post-upgrade verification eval

For platforms that auto-upgrade (most SaaS), the eval runs immediately after the upgrade is applied. The eval becomes the verification step.

The flow: the platform's release notes (or monitoring webhook) indicates a new version is rolling out. The team's CI pipeline schedules the full eval to run within 30 minutes of the rollout. The eval scores the agent against the baseline. If it fails, the team is paged, the incident is opened, the platform vendor is contacted, and (if available) the team's fallback strategy executes — switch to a backup platform, revert the workspace to a snapshot, or pause the agent.

This model accepts that platform upgrades happen without team control. The team's job is fast detection and rapid response.

Mode 3: continuous shadow eval

For platforms where upgrades are continuous (Make's weekly silent updates, certain managed services), the team runs a continuous shadow eval on production traffic. A sampled fraction of production runs (typically 1-5%) is scored in real-time by an LLM-as-judge that is itself isolated from the platform under test.

The shadow eval produces a rolling-window score. When the score drops below a threshold for a sustained period (15-60 minutes depending on the platform's traffic volume), an alert fires. The platform may be the cause; a recent provider-side change may be the cause; a content drift may be the cause. The alert prompts investigation regardless.

The shadow eval is the catch-all detector for changes the team did not anticipate.

The three modes coexist

A mature 2026 agent team runs all three modes. Pre-upgrade staging eval for platforms under team control. Post-upgrade verification eval for auto-upgrade SaaS. Continuous shadow eval as the final safety net. Each catches a different class of failure; the combination catches all of them.

The Eval Set Composition for Platform Regression

The eval set for platform-version regression has overlapping cases with the accuracy eval set but emphasizes different concerns.

Tool-call cases

Cases that exercise every tool the agent uses. Each tool gets at least three cases: one where the tool should be called and is called correctly, one where the tool should not be called (testing refusal/restraint), and one with adversarial input that the tool-call validation should reject.

The February incident lived in this category. The eval cases that called OpenAI or Anthropic tool definitions would have failed immediately after the upgrade because the tool was no longer being called even though the case prompt clearly required it.

Tool-result handling cases

Cases that exercise how the agent handles different tool responses. A successful tool call. A tool call that returns an error. A tool call that times out. A tool call that returns malformed data. A tool call that returns unexpected schema. Each variation exercises a different layer of the platform's runtime.

Multi-step cases

Cases that require multiple tool calls in sequence. These exercise the platform's loop runtime: how it parses each tool response, decides whether to continue the loop, manages context across iterations.

Streaming and non-streaming cases

If the agent uses streaming responses, both modes are tested. Streaming changes how the platform buffers and surfaces output, and platform upgrades sometimes change streaming behavior subtly.

Token-budget edge cases

Cases near the context-window limit. Cases with large tool responses. Cases with long conversation history. These exercise the platform's token-counting and context-management logic.

The total set size

For platform-upgrade verification, a 30-50 case eval suite is typical. Smaller than the full accuracy eval (which has 500-1000 cases at maturity) because the platform-upgrade eval focuses on the layers most likely to be affected. The full accuracy eval still runs nightly; the platform-upgrade eval is a targeted subset that runs on every platform change.

Instrumentation: The Traces That Let You Diagnose

Detection is the first half. Diagnosis is the second. The eval failing tells the team something is wrong. The traces tell them what.

The trace stack

An agent run produces a trace that captures: the input prompt and any preprocessing, the model invocation (including the exact request payload sent), the model response (including any tool calls invoked), each tool call (with the exact request and response), each iteration of the agent loop, the final response. Tools like LangSmith, Langfuse, Arize Phoenix, Helicone, Braintrust, and (for some platforms) the platform's native trace UI capture this stack.

The trace is the diagnostic instrument. When the February incident hit, the teams that had the trace captured were able to compare a pre-upgrade trace against a post-upgrade trace, see that the tool-call request payload differed in the serialization of the enum field, and diagnose the platform regression within minutes.

Teams without traces could see only the input and the output. They could see the output had changed but could not see why. Diagnosis took days because every component had to be ruled out manually.

The pre-upgrade snapshot

The discipline: before every platform upgrade, capture and archive a snapshot of representative traces. Twenty or thirty trace exports from production. After the upgrade, run the eval and pull traces for any failing cases. Compare trace-by-trace against the pre-upgrade snapshot to identify exactly where the behavior diverged.

This is the equivalent of database backups before a schema migration. The teams that skip it pay for it.

The Platform Incident Response Pattern

When the eval fires after a platform upgrade and the team confirms the platform is at fault, the response pattern is specific.

Step 1: confirm and contain

Reproduce the failure with a manual test on a few eval cases. Confirm the issue is consistently reproducible. Pause the agent or roll back to the previous platform version if available. Containment first; diagnosis after.

Step 2: file with the platform vendor

Open a support ticket with the platform vendor immediately. Include the eval results, the trace comparison (pre-upgrade vs post-upgrade), and the specific behavior change observed. The platform vendor usually responds within hours for paying customers; the response time for community-edition users is longer.

Public communication: if the platform is open-source or has a community forum, post the issue publicly. Other teams are likely affected and may have already noticed. Cross-referencing reports accelerates the vendor's prioritization.

Step 3: implement a workaround

While waiting for the platform fix, implement a workaround. Common patterns: bypass the affected node by replacing it with a custom code node that does the equivalent work, route around the broken platform layer (e.g., call the model API directly instead of through the platform's wrapper), or pause the agent if no workaround is feasible.

The workaround is captured in the change log as a Class 3 or Class 4 change depending on the blast radius.

Step 4: communicate to stakeholders

The owner is informed. If the impact is user-visible, end users are communicated to. The audit trail captures the incident timeline. The postmortem (Lesson 5) is opened.

Step 5: re-test on the platform fix

When the platform vendor ships the fix, the team upgrades to the patched version, runs the full eval, and confirms the regression is resolved. The workaround is removed. The change log is updated.

Vendor Management: The Platform Relationship

The February incident also highlighted the difference between platforms and utilities — and the implications for vendor management.

Platforms vs utilities

A utility (Stripe, Twilio, Slack API) is an interface. The team uses it; it works; the team has no operational dependency on the utility's internal architecture. Utility upgrades are infrequent and well-documented because the utility's contract is the API surface.

A platform (n8n, Lindy, Make, Salesforce Agentforce) is part of the team's runtime. The platform's internal architecture is the team's operational concern. Platform upgrades affect agent behavior even when the team's code does not change.

The vendor relationship for a platform is correspondingly different from a utility. The team needs: release-note subscriptions, advance notice of breaking changes, the ability to pin versions where possible, an escalation channel for platform-specific incidents, and a roadmap conversation about behavior-stability commitments.

The platform-stability negotiation

For enterprise customers, platform vendors will typically commit to behavior-stability guarantees as part of the contract. The 2026 emerging standard: 30 days advance notice of any change to platform behavior that could break existing agents, version pinning available, and a designated technical contact who can be paged during incidents.

Teams that have not negotiated these terms have less leverage when an incident hits. The platform vendor's general support flow may not understand the specific behavior change. The escalation chain may not exist. The pinning option may not be discussed until the incident is already in progress.

The multi-platform hedge

Some teams hedge their platform risk by building agents that can run on more than one platform. The agent's prompt, tool definitions, and behavior contract are defined in a platform-neutral format (often as a set of Markdown files plus a tools manifest). The team can run the agent on n8n today and Lindy tomorrow if needed.

The hedge is expensive — the team is essentially maintaining two implementations. For agents critical enough to warrant the cost, the hedge is justified. For most agents, the hedge is overkill and the team commits to a single platform.

The Platform Changelog Discipline

The discipline that scales: every platform the team uses has a documented changelog subscription. The team reads the release notes. The team flags any change that could affect agent behavior. The team runs the eval before or immediately after the upgrade.

The release-note review checklist

For every platform release note, the team checks: does the release touch the LLM runtime? The tool-call layer? The retry logic? The streaming behavior? The token counting? Any layer flagged means an eval run before or immediately after the upgrade.

For platform release notes that do not touch any of these layers (UI changes, dashboard improvements, billing fixes), the eval is not strictly required — but most teams run it anyway because the cost of running the eval is low and the cost of missing a behavior change is high.

The platform-version log

The team maintains a log of platform versions in use, dated, with eval results. When the team is investigating a recent regression, the log answers the question "what platform version are we on?" in seconds. When the team is planning a rollback, the log identifies the last known-good version.

The log overlaps with the change log from Lesson 1 but is more specific. The change log captures all changes; the platform-version log captures specifically which platform versions the agent has been verified to run on.

Anti-Patterns to Avoid

The auto-upgrade trust trap

The team treats platform auto-upgrade as a feature: "great, the vendor handles updates for us." Then the February-style incident hits and there is no detection mechanism. The fix is the post-upgrade verification eval or the continuous shadow eval; without one of those, auto-upgrade is a risk.

The "minor version is safe" assumption

The team scrutinizes major version upgrades and waves through minor version upgrades. The February incident lived in a minor version. The discipline: every version change runs the eval.

The single-detector reliance

The team relies only on the nightly eval and does not run a post-upgrade eval. The nightly eval catches the regression — but not until the next morning, after 12+ hours of degraded service. The fix is multiple detection mechanisms running at different cadences.

The no-trace diagnostic gap

The team has the detection (eval failed) but no traces. Diagnosis takes days because every layer of the stack has to be ruled out manually. The fix is the trace stack and the pre-upgrade snapshot discipline.

The skipped vendor escalation

The team detects the incident, implements a workaround, and never reports it to the platform vendor. The vendor never ships a fix because they do not know about the issue. Other teams continue to be affected. The fix is the vendor support ticket — even if the team has a workaround in place.

Key Takeaways

• The February 2026 n8n LangChain-node tool-arg schema change broke OpenAI and Anthropic tool calls simultaneously in upgraded workspaces. Teams with post-upgrade eval runs caught it in hours; teams without took 8-14 days to notice. The incident is the canonical example of why platform upgrades are model changes, not maintenance.
• The platform is part of the agent's runtime. Eight layers can shift in a minor version: tool-call serialization, tool-result parsing, retry logic, system prompt assembly, response streaming, output parsing, token counting, tracing. Each is a potential behavior change.
• Three platform-version triggers for full eval: platform/runtime upgrades (n8n, Lindy, Make, LangGraph, SDKs), model version changes (direct or silent), and prompt changes. Pin model versions explicitly. Treat minor versions with the same scrutiny as major versions.
• Three eval modes that coexist: pre-upgrade staging eval (for platforms under team control), post-upgrade verification eval (for auto-upgrade SaaS), continuous shadow eval (1-5% of production traffic, LLM-as-judge). Each catches a different class of failure.
• The platform-upgrade eval set: 30-50 cases targeting tool calls, tool-result handling, multi-step loops, streaming, and token-budget edges. Smaller than the full accuracy eval but specifically designed to exercise platform runtime layers.
• Traces are the diagnostic instrument. Capture pre-upgrade trace snapshots. After upgrade, compare trace-by-trace to identify exactly where behavior diverged. Without traces, diagnosis takes days; with traces, minutes.
• Platform incident response: confirm and contain, file with the vendor, implement a workaround, communicate to stakeholders, re-test on the fix. The workaround is logged as a Class 3 or Class 4 change.
• Vendor management: subscribe to release notes, negotiate behavior-stability commitments for enterprise contracts (30-day advance notice, version pinning, escalation channel), maintain a designated technical contact. Some teams hedge with multi-platform implementations for critical agents.
• The platform changelog discipline: read release notes, flag changes touching LLM runtime / tool calls / retry / streaming / token counting / parsing. Maintain a platform-version log of versions in use and eval results.
• Anti-patterns: auto-upgrade trust trap (no detection mechanism), "minor version is safe" assumption (where February lived), single-detector reliance (only nightly eval), no-trace diagnostic gap (detection without diagnosis), skipped vendor escalation (workaround without report means no fix).