By the time you have the right HITL mode (Lesson 1) and the right approval card with structured override-reason capture (Lesson 2), you have built a signal-generation machine. Every human edit, every rejection, every approval-with-modification produces a labeled data point: the agent proposed X, the human shipped Y, the reason was Z. The question is what you do with that signal. Most teams in 2024–2025 did nothing. The data flowed into a logs warehouse and sat there. The agent did not get better. Reviewers eventually started asking why they were doing the same overrides every week, then asking whether the agent was worth keeping. The 2026 practice closes the loop. Every override becomes an eval case. Every category aggregation becomes a fine-tune target or a prompt update. The weekly review and the 90-day rolling baseline are the named rituals that make this work — not as theory but as repeating calendar events that produce incremental measurable improvement. This lesson is how to build the feedback loop that compounds.

The Three Flavors of Feedback and Why They Matter

Not all human feedback is the same shape. Conflating them is why most feedback loops fail to compound. The 2026 practice distinguishes three flavors and treats each differently.

1. Explicit edits (the agent proposed X, the human shipped Y)

When a reviewer hits Edit on an approval card and modifies the draft before approving, you get a paired example: agent's draft as the "before," reviewer's edited version as the "after," and the structured override-reason as the label. This is the highest-quality feedback signal in the system because every component is captured. The before, the after, and the why are all explicit. Use this for fine-tuning, for prompt updates, and for eval cases.

2. Explicit rejections (the agent proposed X, the human shipped nothing)

When a reviewer hits Reject, you get half a paired example: the agent's draft and the reason it was rejected, but no positive counterexample. This is medium-quality feedback. You know the agent was wrong but you don't always know what right would have been. Use this for negative-example training (DPO/RLHF-style) and for identifying the categories of failure that need attention.

3. Implicit signals (the agent proposed X, the human approved without modification)

When a reviewer hits Approve directly, you get a positive signal: this draft was good enough to ship as-is. But you don't know how carefully the reviewer read it. The signal is noisy because of approve-spam concerns (Lesson 1). High-volume implicit approval is useful in aggregate but should never be treated as ground truth on individual cases. Use this for tracking distribution and detecting drift, not for training.

Three flavors: explicit edits (highest signal — the before, after, and why), explicit rejections (medium signal — the before and why, no after), implicit approvals (low signal — useful in aggregate, suspect on individual cases). Treat each according to its quality.

The Eval Store as the Loop's Foundation

The 2026 feedback loop is built on top of an eval store. Without one, the override data goes into logs and dies. The eval store is where the override becomes a labeled row that can be evaluated against future model versions, future prompt versions, and future fine-tunes.

The four 2026 eval-store choices

• Braintrust. SaaS, eval-first, used by many production agent teams in 2026. Strong UI for inspecting individual cases, good API for ingesting overrides. The choice for teams that want eval as a first-class concern with minimal infra.
• LangSmith. Tightly integrated with LangChain/LangGraph. Excellent if you're already in that ecosystem; clean dataset management and run-vs-eval surface.
• Langfuse. Open-source option. Self-hostable. Used by teams that need data residency control or want to avoid SaaS lock-in. Slightly thinner UI than Braintrust but solid bones.
• Helicone. Started as an LLM proxy with logging; expanded into eval and dataset management. Lower friction to set up for teams that already use it as a proxy. Strong for cost-and-latency-with-quality joint analysis.

Pick one. Stick with it. The eval-store-shopping pattern (try one, try another, never standardize) is a common 2026 anti-pattern that destroys feedback-loop momentum.

The override-to-eval-row pipeline

The pipeline from approval card to eval store has five concrete steps:

1. Card action triggers webhook. Reviewer clicks Edit/Reject; the approval-card webhook fires with structured payload (agent draft, human edit/reject, override-reason category, free-text note, timestamp, reviewer ID, original input that produced the draft).
2. Webhook normalizes the payload. Map to a canonical schema: {input, expected_output, agent_output, override_reason_category, override_reason_text, reviewer_id, timestamp, original_run_id}. The canonical schema is what your eval store understands.
3. Append to the right dataset. One dataset per task (refund-approval, customer-email, segment-classification). Append the row, tagged with the override category for slicing later.
4. Auto-tag for review. New override rows are flagged for the next weekly review. The reviewer in the weekly meeting decides whether to keep, edit, or discard each candidate row before it becomes part of the canonical eval set.
5. Promote to canonical eval set. After human review, the row joins the eval set used for regression testing on every prompt or model change. Now your changes are gated against the patterns the human reviewers actually corrected for.

The Weekly Review as the Named Ritual

The weekly review is the most important calendar event in the 2026 agent operator's week. It is the difference between "we have feedback data" and "we have a feedback loop." A 45–60 minute meeting, every week, with a fixed agenda.

The participants

• The agent owner. The person responsible for the agent's quality. Usually a citizen developer, an AI engineer, or an ops lead.
• One or two reviewer representatives. Humans who actually clicked the buttons during the week. They bring the qualitative texture that the dashboard misses.
• (Optional) A platform/eval engineer. If the team has dedicated infra, this person handles the dataset operations and prompt iteration.

Three to four people. No more. Standing meeting; reschedule but never skip.

The fixed agenda

1. Volumes and rates (5 min). How many cards last week? Approval, edit, reject rates. Compare to prior 4 weeks. Any anomalies? This is the dashboard-driven part of the review.
2. Override category breakdown (10 min). Pull the structured override-reason dashboard. What was the top category this week? Has it changed from prior weeks? A consistent top category over 3+ weeks is a fine-tune target.
3. Three case deep-dives (20 min). Pick three actual override cases — typically the worst-feeling rejection, the most-edited approval, and one randomly sampled. Read the agent's draft. Read the reviewer's edit. Discuss why the agent got it wrong. This is where the qualitative texture lives.
4. Eval set additions (10 min). Of the new override rows auto-tagged this week, which become canonical eval cases? Edit any rows that need adjustment. Discard any that were one-off / reviewer error / out-of-scope. Promote the rest to the eval set.
5. Action items (5 min). One or two concrete changes to ship before the next review. "Update the prompt to handle X." "Run a fine-tune candidate against the new eval cases." "Add a new risk flag for Y." Concrete, owned, dated.

Why weekly specifically

Daily is too frequent — the signal hasn't accumulated enough to see patterns. Monthly is too slow — patterns become entrenched before they're addressed and reviewers feel ignored. Weekly is the cadence where the volume is high enough to see signal and the response is fast enough that reviewers see their override-reason notes producing actual changes within two weeks.

What kills the weekly review

Three failure modes that turn the weekly review from a working ritual into a performative meeting:

• Skipping when nothing seems broken. If the agent is at 95% approval and there are no fires, the temptation is to skip. Resist it. The weekly review's purpose includes catching the slow drift before it becomes a fire.
• Letting the meeting become a stand-up. "Last week we had X cards" without case deep-dives is a dashboard review. The weekly is supposed to be where humans look at actual cases. If the case deep-dives get squeezed out, the texture is lost.
• No follow-through on action items. If the weekly produces action items that don't get done before the next weekly, the reviewers learn the meeting is theater. Two or three action items per week, owned and dated, ship by the next review.

The 90-Day Rolling Baseline

The weekly review handles week-over-week change. The 90-day rolling baseline handles structural quality assessment. The two work together; they are not interchangeable.

What the rolling baseline is

A held-out evaluation suite that runs against the agent on a regular cadence (typically twice a month) using a 90-day window of accumulated eval cases. The eval set is the union of: original launch cases, every promoted override case from the last 90 days, and any synthetic edge cases the team has added. As cases age past 90 days they roll off (unless flagged "permanent" — e.g., known regulatory cases that should never break).

Why 90 days specifically

Three reasons. First, 90 days is long enough that real-world distribution has been sampled: typical customer behavior, edge cases, seasonal patterns, the rare-but-real flags. Second, 90 days is short enough that the eval set reflects the current state of the world; older cases drift out as policy, product, and customer base evolve. Third, 90 days aligns with quarterly review cadences at most operator teams, so the baseline numbers show up in the metrics the business already tracks.

The baseline numbers you track

• Approval rate. What percentage of the eval cases would the agent's current output have been approved if the historical human reviewer saw it? Run the agent on the eval set; compare each output to the human-shipped version using an LLM-as-judge graded on semantic equivalence.
• Edit-similarity score. For cases where the historical human edited, how close is the new agent's output to the human's edited version vs. the agent's original? A score near 1.0 means the agent has internalized the edit pattern; a score near 0 means the model hasn't learned.
• Override-category breakdown. Of the eval cases the new agent fails, which categories dominate? This tells you what fine-tune or prompt update to do next.
• Latency and cost per eval run. Drift on these is a separate signal but tracked in the same baseline.

How the baseline rolls forward

Every two weeks the eval suite runs against the current agent. Numbers are logged. The rolling baseline is the trend: is approval rate going up over the last 90 days or down? Is the edit-similarity score on a specific category trending toward 1.0 (improvement) or away from it (regression)? The trend is the signal; any single run is noise.

The pre-deploy gate

Before any prompt change, model change, or fine-tune ships to production, it runs against the 90-day rolling baseline. The criteria: approval rate must not drop more than 2 percentage points; edit-similarity score on any category must not drop more than 0.05; no eval case flagged "permanent" can fail. If the gate fires, the change does not ship without an explicit override and a documented decision. This is what turns an eval store into a regression-prevention system.

From Overrides to Fine-Tunes: The Decision Matrix

Not every override pattern needs a fine-tune. Most patterns can be fixed with a prompt update or a guardrail addition. The decision of when to fine-tune is the practitioner's discrimination.

The four categories of fix

• Prompt update. The most common fix. The agent's instructions can be amended to address the override pattern. "When a customer has been a regular for more than 2 years, the agent should consider tenure adjustment." Two minutes of work, deploys in minutes, easy to revert.
• Guardrail addition. The agent should refuse to act when certain conditions are met. "Refuse to draft a refund without the order record attached." Refuse-to-act guardrails are cheap and catch entire categories of failure.
• Few-shot example injection. The override examples themselves become few-shot examples in the prompt. Most platforms (Braintrust, LangSmith) can pull the latest N examples of a specific override category into the prompt automatically.
• Fine-tune. When the pattern is consistent over 3+ weeks of weekly reviews, the override category is the top category for multiple weeks, and prompt-update / few-shot / guardrail approaches have plateaued. Fine-tune is the heavyweight move. Use it last.

The fine-tune trigger criteria

Fine-tune is worth the cost (time, infra, evals re-run) when all four conditions hold:

1. The override category has been the top category for 3+ consecutive weekly reviews.
2. The volume in that category is at least 50 examples in the 90-day window (otherwise insufficient training signal).
3. Prompt update, few-shot, and guardrail approaches have been tried and have not closed the gap.
4. The improvement target is concrete (specific edit-similarity score, specific approval-rate increase).

When all four conditions hold, fine-tune. Most teams in 2026 use parameter-efficient fine-tuning (LoRA-style) on the override-paired data, which costs $50–$500 in compute and produces a deployable adapter in hours. Fine-tune the small classifier model if the pattern is classification; fine-tune the main generator model only if the pattern is generation behavior.

The fine-tune-then-eval ritual

Every fine-tune runs against the rolling baseline before ship. If it doesn't improve the target metric without regressing the others, it doesn't ship. This is the closed loop: override → eval row → fine-tune → baseline check → ship → measure.

Closing the Loop: The Meta-Pattern

The feedback loop is not a one-time setup. It is a meta-pattern that has its own quality signal. The signal is whether reviewers see their override-reason notes producing changes in the agent's behavior.

The visibility check

At each weekly review, surface one or two cases from previous weeks where a reviewer's override directly drove a change. "Three weeks ago, @sarah.k flagged that the agent was using stale segment data. We updated the segment-lookup tool. Last week, segment-related overrides dropped from 12 to 3." Reviewers seeing this loop close is what keeps them engaged.

The reviewer pulse

Every six weeks, a short async survey to reviewers: do you feel like your overrides are making the agent better? Do you trust the system more or less than six weeks ago? Three-question pulse, two-minute response. A declining pulse is the leading indicator that the loop is not actually closing, even if dashboard numbers look fine.

The story-of-the-quarter

Each quarter, write up the single highest-impact override-driven improvement as a one-page story. Share with the team, the stakeholders, the executive sponsor. The story has four parts: the override pattern that was discovered, the size of the problem (how many cases / how much money / how much customer impact), the change shipped, the measurable impact post-change. The story-of-the-quarter is what makes the feedback loop visible to non-practitioners and protects the program when budget review season hits.

Story: The Refund Agent That Learned

A consumer fintech in late 2025 launched a refund-recommendation agent. The agent proposed refund amounts to a CS team lead who approved, edited, or rejected. The first three months were turbulent — approval rate hovered around 60%, override rate was high, and reviewers were frustrated. The team had built the override-reason capture but did not have a weekly review or a 90-day baseline. The data was accumulating in Braintrust without being acted on.

In January 2026 they instituted the weekly review. The first review pulled the override-reason data for the prior four weeks. The top category was "wrong tenure consideration" — the agent was treating all customers the same regardless of tenure, and reviewers were consistently editing for customers with 2+ years of history. Volume in that category was 78 cases in the 90-day window.

Action items from week 1: update the prompt to consider tenure (shipped Tuesday). Add tenure as a structured input from the customer record (shipped Thursday). Add five tenure-edge cases to the eval set (shipped Friday). Re-run the baseline next Monday.

Week 2 review: tenure-related overrides dropped from 22 in the prior 7 days to 5 in the new 7 days. The fix was working. New top category emerged: "wrong policy reference" — the agent was citing policies that had been deprecated. Action items: update the agent's policy-reference tool to filter by active status; add a guardrail that refuses to cite a deprecated policy.

Week 4 review: policy-reference overrides cleared. New top category: "wrong communication tone for high-tier customers." This one persisted through weeks 4, 5, 6. After three weeks at the top, the team triggered a fine-tune. They pulled 92 tone-edited examples from the eval store, fine-tuned a LoRA adapter on the base model, evaluated against the rolling baseline. Edit-similarity score on tone cases improved from 0.41 to 0.78 without regressing other categories. They shipped the adapter.

By April 2026, the agent's approval rate had moved from 60% to 89%. The override rate dropped from 30%+ to 11%. The team had completed eight 90-day baseline runs, shipped one fine-tune, made 23 prompt updates, and added 41 cases to the canonical eval set. The story-of-the-quarter for Q1 2026 was titled "How tenure, policy, and tone took the refund agent from 60% to 89%."

The interesting part of the story is not the 60-to-89% improvement. It is the ritual that made the improvement possible. Without the weekly review, the team had three months of accumulating data and no progress. With it, they had two months of measurable compounding improvement.

Anti-Patterns: Feedback Loops That Don't Loop

The "we'll review it monthly" trap

Monthly is too slow. Patterns get entrenched. Reviewers feel ignored. By the time the monthly review happens, the override-reasons from week 1 are no longer fresh in anyone's memory and the action items don't get prioritized. Weekly or it doesn't work.

The "we have the data" delusion

"We're capturing override reasons, we have a Braintrust account, we're all set." Capturing data is not the loop. The loop is review → action → measure → review. Data without ritual is dead weight in a logs database.

The "fine-tune first" mistake

The team's first reaction to override data is to fine-tune. Fine-tunes are expensive and slow to revert. Most patterns can be fixed with prompt updates or guardrails. Reserve fine-tunes for patterns that prompt-update has failed on after 3+ weeks.

The "no baseline, no gate" failure

The team ships prompt updates and fine-tunes without running against the rolling baseline. Improvements on one category cause regressions on another category. The agent gets better at refunds and worse at escalations. Without the baseline gate, the team is playing whack-a-mole. By month 6 the agent's overall quality is worse despite "improvements."

The "reviewer isolation" failure

The reviewers see no evidence that their overrides matter. Override-reason notes pile up; no changes appear in the agent. Reviewers stop writing meaningful override-reasons (or stop overriding at all and slip into approve-spam). The fix is the weekly review's visibility check — every meeting surfaces one or two cases where a prior override drove a change.

The "infinite eval set" failure

Every override gets promoted to the canonical eval set. After a year, the eval set has 4,000 cases. Each baseline run takes hours. The team starts running it less frequently. The 90-day window prevents this — cases age out unless explicitly marked permanent, and the eval set stays in the 200–500 range that keeps baseline runs fast.

Instrumentation: the 2026 Stack

The reference architecture

The reference architecture in 2026 has four components, two paths, and a clear flow:

1. Approval surface (Slack Block Kit / Teams Adaptive Cards / embedded). Captures the structured override-reason on edit/reject.
2. Webhook + normalizer (n8n / LangGraph node / custom). Receives the action, normalizes to the canonical schema, forwards to the eval store.
3. Eval store (Braintrust / LangSmith / Langfuse / Helicone). Stores the override row tagged with the category. Maintains datasets for each task.
4. Review surface (the eval store's UI, plus the weekly review meeting). Where humans promote rows to canonical, run baseline evals, and gate deployments.

The flows

• Capture flow. Card action → webhook → normalize → eval store dataset (tagged as "candidate").
• Promotion flow. Weekly review → human promotes candidate rows to "canonical." Canonical rows are the regression eval set.
• Gate flow. Prompt change / model change / fine-tune → re-run canonical eval → compare to 90-day rolling baseline → deploy if passes gate, otherwise return for fixes.

The minimum viable feedback loop in 90 minutes

If you have an approval card with structured override capture (Lesson 2), you can wire the minimum viable feedback loop in a Tuesday afternoon. n8n flow: webhook receives card action → JSON transform → HTTP POST to Braintrust dataset endpoint. Schedule the weekly review on the team calendar. Run the first baseline at the next session. The loop is live. The improvement compounds from there.

Key Takeaways

• Three flavors of feedback: explicit edits (highest signal — before, after, why), explicit rejections (medium — before, why, no after), implicit approvals (low — useful in aggregate, suspect on individual cases).
• The eval store is the foundation. Pick one (Braintrust, LangSmith, Langfuse, Helicone). Stick with it. Override-shopping is a feedback-loop killer.
• The override-to-eval pipeline has five steps: card action → webhook → normalize → append-to-dataset → human-promotes-to-canonical.
• The weekly review is the named ritual. 45–60 minutes, 3–4 people, fixed agenda (volumes, override breakdown, three case deep-dives, eval set additions, action items). Weekly because daily is too noisy and monthly is too slow.
• The 90-day rolling baseline is the structural quality measure. Held-out eval suite, runs twice a month, tracks approval rate, edit-similarity, category breakdown, latency/cost.
• The pre-deploy gate uses the baseline. No prompt change or fine-tune ships without passing the gate (approval rate within 2pp, edit-similarity within 0.05, no permanent case fails).
• Four fix categories: prompt update (most common), guardrail addition (cheap, catches whole categories), few-shot example injection (auto-pulled from eval store), fine-tune (last resort, used when the pattern has been top-category for 3+ weeks with 50+ examples).
• Closing the loop has its own quality signal: reviewers seeing their overrides produce changes. Visibility check at each weekly review; six-week reviewer pulse; quarterly story-of-the-quarter.
• The refund-agent story: 60% to 89% approval in 90 days via weekly review + 90-day baseline. The ritual made the improvement possible.
• Six anti-patterns: monthly cadence, "we have the data" delusion, fine-tune-first, no-baseline-no-gate, reviewer isolation, infinite eval set. Each looks plausible and kills compounding.