Every production agent has a day it got dumber and the team did not notice. Sometimes the cause is a one-word prompt change made to fix one stakeholder complaint that quietly broke three other things. Sometimes it is a tool-schema migration that changed an enum value and the agent started picking the wrong action. Sometimes it is the provider silently upgrading the model behind their API and the same call producing measurably different outputs. The agent does not announce that it has regressed. Users notice over weeks. Stakeholders complain quietly. Trust erodes. By the time someone diagnoses the cause, the regression is months old. The fix is regression detection as code — a CI-style eval that runs on every prompt change, every model change, every tool-schema change, and blocks the merge below a threshold. This lesson is the architecture, the GitHub Actions and Jenkins recipes, the threshold policies, and the human-in-the-loop overrides that keep the system shipping when the regression is intentional.

The Day the Agent Got Dumber and Nobody Noticed

A B2B SaaS company shipped a customer-service agent in late 2025. The agent handled tier-1 tickets — return policy, shipping status, refund timing. Quality was good. Stakeholder feedback was positive. The team was iterating quickly.

On a Tuesday in March 2026, an engineer pushed a small prompt change to fix one stakeholder complaint about over-formal language. The change shipped. The eval was not run. The agent got measurably worse on three other dimensions that the engineer did not realize were sensitive to that wording. By the time the support team caught a wave of new complaints six weeks later, no one remembered the prompt change. The reproduction took eight working days. The fix took another two days. The total cost: two weeks of degraded service, a stakeholder who had championed the project quietly questioning whether the agent was reliable, and the team's velocity reset to "be cautious."

The fix that should have been in place: a CI eval that ran on every prompt commit, scored the agent against the 50-case eval set, and blocked the merge if the score dropped below the regression threshold.

The day the agent got dumber is the silent failure mode of agent operations. Users notice over weeks. Stakeholders complain quietly. Trust erodes. The CI-style eval — running on every change that touches the agent — is the discipline that catches the regression the day it happens, not six weeks later when the damage compounds.

Why agent regressions are subtle

Software regressions are mostly binary. A test fails. A type-check errors. A user-facing button breaks. Engineers notice immediately because the system loudly tells them.

Agent regressions are continuous. The agent still works. It still produces grammatically-correct outputs. The outputs are 7% worse on tone, 4% worse on factuality, 12% worse on multi-step queries. None of those changes are visible to a human-eyeball review of any single output. They emerge as patterns across many outputs over weeks. By then it is too late.

The CI-style eval makes the regression visible in the way software regressions are visible: a number that drops, an automated check that fails, a merge that gets blocked. The continuous quality problem becomes a discrete signal.

What Triggers a Re-Eval

The four high-risk changes

Not every code change requires a full eval run. The cost would be prohibitive. The list of changes that absolutely demand re-eval:

• Prompt change. Any modification to the system prompt, the user prompt template, or any instruction the agent receives. The most common and most underestimated. A single word can move 5-15 cases.
• Model change. Swapping GPT-5 for Claude Sonnet 4.5 — obviously. But also: a provider's silent minor-version upgrade ("gpt-5-2026-04" → "gpt-5-2026-05"). Pin the model version explicitly; treat version bumps as model changes.
• Tool-schema change. Adding a new tool, deprecating an old one, modifying a parameter, changing an enum value, renaming. Tools change the agent's reasoning patterns in subtle ways. Re-eval.
• Framework or SDK upgrade. LangChain 0.4 to 0.5. Anthropic SDK 1.x to 2.x. Underlying retries, default settings, or trace formats may shift agent behavior. Re-eval.

The fifth trigger: time

Even when nothing changed in the team's code, run the full eval nightly. Models drift. Provider infrastructure changes. Embedding models are silently re-trained. The nightly eval catches the changes the team did not make. A red signal at 3am beats a stakeholder complaint at 3pm.

The Two-Tier Eval Architecture: Smoke and Full

Running 500 eval cases on every commit is expensive and slow. Running zero cases is the failure mode. The compromise is two tiers.

Tier one: smoke eval

50-100 cases. Targets the most critical golden + edge + adversarial cases. Runs on every commit to the agent codebase. Completes in 2-5 minutes. Blocks the merge if any layer-1 deterministic check fails, or if the aggregate score drops below threshold.

The smoke eval is the gate. It catches obvious regressions in minutes. It is fast enough to live in the developer's loop — they can run it before pushing, can run it again after a code review comment, can see the diff in their PR.

Tier two: full eval

500-1000+ cases. Comprehensive coverage. Runs nightly and on every merge to main. Completes in 30-60 minutes. Generates the trend reports stakeholders see. Catches the regressions the smoke eval misses (the long-tail edge cases that don't fit in 50 examples).

The full eval is the heartbeat. It produces the weekly score-up tickets, the monthly stakeholder reports, the quarterly trend analyses.

The cost tradeoff

Smoke eval cost: ~$0.50-$2.00 per run (50 cases through the agent + 4-layer eval stack with a small judge model). Run 20 times a day across the team: $10-$40/day. Negligible.

Full eval cost: ~$5-$20 per run. Run nightly: ~$200-$600/month. Still small compared to the cost of an undetected regression.

GitHub Actions: The Canonical Workflow

GitHub Actions is the most common CI runner for agent teams. A canonical workflow looks like this:

The smoke workflow on every PR

A .github/workflows/eval-smoke.yml triggered by pull-request events on branches that touch the agent code (paths-filter on prompts/, tools/, agent/**). The workflow runs in three jobs:

1. Install and set up. Check out the PR branch. Install Python or Node dependencies. Cache the eval set artifacts.
2. Run smoke eval. Execute the eval driver script (Braintrust, LangSmith, Langfuse, or a custom script). Capture exit code. Output the score summary as a job artifact.
3. Comment on the PR. Post the score summary as a PR comment using actions/github-script. Include the diff vs. the main-branch baseline: which cases improved, which regressed, the aggregate delta.

The merge gate via branch protection

In GitHub's branch protection rules for main, mark the smoke-eval workflow as required. PRs cannot merge to main unless the smoke eval passes (exit code 0). The eval driver returns non-zero when the aggregate score is below the configured threshold.

The threshold has two flavors:

• Absolute threshold. The eval must score >= 80%. Simple but doesn't adjust for case-set evolution.
• Relative threshold. The eval must score >= (main_branch_score - 2 percentage points). Adapts as the agent improves; protects against regression specifically.

Most teams settle on relative threshold for the smoke eval — protects against day-over-day regressions — and absolute floor for catastrophic catches.

The nightly full eval workflow

A .github/workflows/eval-full.yml triggered by schedule: cron: '0 4 * * *' (4am UTC daily) and by push: branches: [main]. The workflow runs the full eval set, uploads results to your eval platform, sends a Slack notification with the score summary, and opens an issue if the score regressed below threshold.

Secrets management

The eval workflow needs API keys for the model provider and the eval platform. Use GitHub Actions Secrets. Rotate quarterly. Scope to read-only where possible. The eval platform API keys often have a separate "evaluation" scope that does not allow production traffic — use that scope for CI.

Jenkins and the Self-Hosted Option

Some teams run on Jenkins, GitLab CI, CircleCI, or self-hosted CI runners. The pattern translates directly.

Jenkins pipeline definition

A Jenkinsfile with a pipeline block. Stages:

1. Checkout. Standard SCM step.
2. Install deps. A shell step running pip install -r requirements.txt or npm ci.
3. Run smoke eval. A shell step running the eval driver. Set currentBuild.result = 'FAILURE' if exit code != 0.
4. Publish results. Use the archiveArtifacts step to save the JSON output. Use the JUnit publisher if your eval driver supports JUnit XML output (Promptfoo does natively; Braintrust and Langfuse have JUnit adapters).
5. Notify. Send a Slack notification with the summary using the Slack plugin.

The PR-comment equivalent on Jenkins

Use the GitHub PR Comment plugin or a similar one. Render the score summary as a Markdown comment on the PR linked to the build. Same UX as GitHub Actions.

Self-hosted runner advantages

For enterprise teams: self-hosted CI runners can sit inside the corporate VPC, reach internal vector stores, and have controlled secret access. The eval cost is lower (no per-minute CI billing) but you maintain the runners. Worth it when the security perimeter requires it.

Promptfoo: The Eval-as-CLI Pattern

Promptfoo is the CI-first eval tool in the 2026 landscape. It is a CLI plus a YAML config. Designed to slot directly into any CI runner.

The promptfoo config file

A promptfooconfig.yaml declares the prompts, the providers (the model APIs), the test cases (your eval set), and the assertions (your scoring functions). The CLI promptfoo eval reads the config, runs the cases, evaluates the assertions, exits with non-zero if any assertion below threshold.

Why Promptfoo pairs well with Braintrust or Langfuse

Promptfoo is strong at the CI integration: declarative YAML, exit codes, JUnit XML output, PR comment integrations. Promptfoo is weaker on the dashboard side — limited diff views, less polished historical analysis. The common pattern: Promptfoo runs the CI eval, Braintrust or Langfuse provides the dashboard and longitudinal trend analysis. Both can read the same eval set.

Promptfoo built-in CI integrations

• GitHub Action: promptfoo/promptfoo-action drops the eval into a workflow with one block.
• GitLab CI integration: native via the CLI plus the GitLab MR comment API.
• CircleCI orb: promptfoo/promptfoo orb provides ready-made commands.

The Threshold Policy: Block Merge vs. Warn

What happens when the smoke eval score drops? Two policies:

Block-merge policy

If the smoke eval score is below threshold, the PR cannot merge. Engineer must either fix the regression or override (covered below). Strict; protects production absolutely.

Use block-merge for: production-facing agents, safety-critical domains, agents with audit obligations (EU AI Act).

Warn-only policy

If the score drops, the workflow posts a warning comment on the PR but does not block. Engineer can merge anyway with explicit acknowledgment. Looser; relies on engineer discipline.

Use warn-only for: early-stage internal agents, demos, agents in active heavy iteration where false-positive blocks would slow development too much.

The hybrid policy

Most production teams settle on hybrid:

• Hard fail (block merge): any layer-1 deterministic check fails (schema break, format violation), any adversarial case regresses (security regression), or aggregate score drops by more than 5 percentage points (major regression).
• Soft fail (warn): aggregate score drops by 1-5 points. Engineer reviews the diff; merges with acknowledgment if intentional.
• Pass: aggregate score holds or improves.

The Human-in-the-Loop Override for Intentional Regressions

Sometimes the regression is intentional. The agent was scoring 95% on a case the team has now decided was the wrong behavior. The new prompt scores lower on that case because the agent is doing the right thing — refusing where it used to over-answer, asking for clarification where it used to guess.

A rigid block-merge policy on these cases creates friction that breeds workarounds. Teams start commenting out test cases instead of acknowledging intentional behavior changes.

The label-based override

Add a PR label like eval-regression-acknowledged. When present, the smoke eval workflow checks the label, posts the regression detail to the PR, and allows the merge. The label can only be applied by someone other than the PR author (forces a code-review style check on the override). The label is captured in the PR history — auditable.

The case-deprecation flow

When a case is genuinely obsolete (the desired behavior has changed), the case should be retired from the eval set, not commented out and not bypassed. Open a separate PR that updates the eval set: removes the obsolete case, adds the new desired-behavior case. Reviewable, auditable, and the eval set evolves with the agent's behavior contract.

The Trend Dashboard: The Stakeholder Artifact

The CI integration produces signals for developers. Stakeholders need a different artifact: the trend dashboard.

What the dashboard shows

• Weekly aggregate eval score with the trend line. Going up = good story. Going down = action item.
• Per-category breakdown: golden, edge, adversarial. Stakeholders can see "we're getting better on edge cases" or "we're regressing on adversarial."
• Per-dimension breakdown: factuality, helpfulness, tone, safety. The dimensions matter to different stakeholders (security cares about safety; product cares about helpfulness).
• Recent failures: which cases dropped from passing to failing in the last week? The actionable list.
• Production-traffic eval (when supported): how does the agent score on real production queries this week vs. last week?

The weekly score-up ticket

Friday afternoon: the eval system auto-generates a ticket or a Slack post titled "Week of [date]: agent eval +1.2 points." The post includes the headline trend, the biggest improvements, the biggest regressions, and any cases that flipped from pass to fail. Stakeholders see this every Friday. The agent stops being a black box.

Anti-Patterns of Regression Detection

Anti-pattern one: no CI eval

"We run the eval before each release." Releases happen every two weeks. The regression that ships on Monday's prompt change doesn't get caught until the release-week eval. Three weeks of degraded service. The team learns to blame "the model" instead of the change. Fix: CI eval on every commit.

Anti-pattern two: ignored CI failures

The smoke eval fails. The engineer overrides with no investigation. The CI signal becomes noise. Three weeks later the team turns the CI eval off because "it always fails." Fix: the override requires a reason; the reason gets a code-review comment; ignored overrides surface in the weekly score-up ticket.

Anti-pattern three: too-strict threshold

The threshold is set at "no regression at all" — any case that flips from pass to fail blocks the merge. Engineers experience false-positive blocks on natural agent variability. Workarounds proliferate. Fix: tolerance of 1-2 percentage points for natural variability; hard fail only on multi-point regressions.

Anti-pattern four: stale eval set

The eval set was built six months ago. The agent and the product have evolved. The eval is testing irrelevant cases and missing the new failure modes. CI eval passes; stakeholders complain. Fix: monthly eval-set review, weekly production-traffic sampling into the set.

Anti-pattern five: opaque CI output

The CI eval prints "FAILED" and the developer has no idea why. They override. Or they ignore. Or they spend an hour digging through logs. Fix: rich PR comment with the diff view, link to the failed cases, score breakdown by category and dimension. Make the regression diagnosis fast.

The Pre-Commit Eval: The Developer Loop

The CI eval catches regressions at PR time. The pre-commit eval catches them before the developer pushes — the tightest loop possible.

The pre-commit hook

A Git pre-commit hook (via pre-commit, husky, or a custom hook) runs a 10-case smoke eval before allowing the commit. 30 seconds. Catches "I broke the agent" in real-time. The developer fixes before pushing.

The smaller-set tradeoff

10 cases is fast but loose. The pre-commit eval catches 60-70% of regressions the full smoke would catch. The remaining 30-40% are caught by the PR-time smoke eval. Two-tier in-loop checking.

When to skip the pre-commit eval

Some commits don't touch the agent (documentation, UI, deployment config). The pre-commit hook should detect via path-filter and skip the eval for non-agent commits. Otherwise developers learn to bypass the hook entirely.

Case Study: The Regression That Cost Two Weeks of Service

The B2B SaaS team from the opening story rebuilt their CI eval architecture after the Tuesday-in-March incident. The system that emerged:

Pre-commit hook: 10-case smoke eval. 30 seconds. Catches obvious breakage. Developers run it before every push.

PR-time smoke eval: 75-case smoke eval via Promptfoo. Runs on every PR that touches prompts/, tools/, or agent/. 3-4 minutes. Posts the diff view as PR comment. Blocks merge on >5-point regression or any adversarial failure.

Nightly full eval: 500-case eval via Braintrust. Runs at 3am UTC. Posts results to Slack #agent-quality channel. Opens GitHub issue if aggregate regresses below threshold.

Weekly score-up ticket: Auto-generated Friday afternoon. Three months later the team had not had another silent regression. Score-up tickets had become a routine artifact stakeholders looked forward to. The stakeholder who had questioned the agent's reliability after the incident became one of its strongest advocates.

The investment: roughly two weeks of engineering work to build the CI pipeline. The return: never paying the two-week silent-regression cost again, plus the stakeholder trust that comes from a measurable improvement story.

Key Takeaways

• Agent regressions are continuous and silent: the agent still works but degrades 5-15% on dimensions no one notices until weeks of complaints accumulate.
• The CI-style eval — running on every prompt, model, tool, or framework change — makes the silent regression discrete: a number that drops, a check that fails, a merge blocked.
• Two tiers: smoke eval (50-100 cases, 2-5 min, every commit) plus full eval (500+ cases, nightly).
• GitHub Actions: a workflow on PR events runs the smoke eval, posts the diff as a PR comment, blocks merge below threshold via branch protection.
• Jenkins and other CI runners follow the same pattern. Promptfoo is the CI-first eval tool of choice; pairs with Braintrust or Langfuse for dashboards.
• Threshold policy: hybrid — hard fail on any L1 break, any adversarial regression, or aggregate drop >5 points; soft warn for 1-5 point drops; pass at hold-or-improve.
• Intentional regressions get a label-based override that requires a code-reviewer's acknowledgment. The override is auditable in PR history.
• Trend dashboard plus weekly score-up ticket turns eval into the stakeholder-facing artifact that builds trust.
• Pre-commit hook adds a 10-case smoke in the developer loop — 30 seconds, catches 60-70% of regressions before push.
• Anti-patterns to avoid: no CI eval, ignored CI failures, too-strict threshold, stale eval set, opaque CI output.