A Service Level Objective is a promise you can defend. Classical SRE has decades of practice picking SLOs that matter, defining error budgets that govern release velocity, and wiring alerts that fire before users notice. Agent ops is still figuring out which SLOs survive the transition from "metric we look at" to "promise we will defend." By the middle of 2026, four service levels have emerged as the ones serious teams commit to: p95 latency, accuracy against a 50-case eval baseline, cost-per-run with a two-sigma alert, and escalation rate treated as a quality metric rather than a failure metric. This lesson is the architecture: how each SLO is defined, what the 2026 production targets are, how to wire the alerting so it fires before users notice, and which SLOs are the wrong ones to commit to even though they sound right. The discipline is not picking metrics. The discipline is picking metrics you will actually defend at the QBR.

The SLO Discipline

An SLO is not a metric. An SLO is a metric plus a target plus a consequence. When the SLO is breached, something happens — alerts fire, releases pause, the team has a conversation about the error budget. A metric without the target and the consequence is a dashboard widget, not an SLO.

Why most agent teams have metrics, not SLOs

Walk into the average agent team in 2026 and you will find ten dashboards. Latency dashboard, cost dashboard, eval-score dashboard, error-rate dashboard, tool-call-distribution dashboard, model-usage dashboard. Ten dashboards is great for situational awareness. Ten dashboards is terrible for accountability.

The team has metrics. The team does not have SLOs. When the latency dashboard goes red, nothing automatic happens. Someone might notice. Someone might mention it in Slack. Someone might file a ticket. The team is operating reactively.

The SLO discipline forces a different posture. Pick the four numbers you will commit to. Define the targets in writing. Wire alerts so the moment a target is breached, the on-call is paged or the release gate is closed. Make the consequence automatic.

Why four SLOs and not eight

Classical SRE wisdom: fewer SLOs, defended more rigorously, beat more SLOs defended weakly. The teams that try to commit to eight SLOs end up enforcing two and ignoring six. The signal-to-noise ratio collapses. The alerts get muted. The team is back to running on dashboards-as-vibes.

Four is the sweet spot for production agents. Each of the four SLOs covers a distinct dimension that, when breached, requires a different response. Latency breach: model or routing problem. Accuracy breach: prompt or retrieval problem. Cost-per-run breach: loop or pricing problem. Escalation rate breach: agent autonomy mismatched to use case.

The four SLOs are the four questions an agent owner answers at the QBR. Is it fast enough? Is it accurate enough? Is it cheap enough? Are we handing off appropriately to humans? Pick four numbers you will defend in front of a CFO and a VP of Customer Success and you have an operational vocabulary. Pick more and you have a dashboard. Pick fewer and you have an opinion.

SLO 1: Latency — p95 Under 8 Seconds for Conversational Agents

The most defensible latency SLO in 2026 production is p95 under 8 seconds for conversational agents. Async agents (those that run on a schedule, batch process, or operate via Slack with no real-time user waiting) have a different SLO — typically p95 under 5 minutes or a hard-cap on a 60-second per-run budget — but conversational agents converge on the 8-second p95 target.

Why p95 and not p50 or p99

p50 (the median latency) is too forgiving. Half the user requests are above it. The agent that has a 2-second p50 but a 30-second p95 is providing a terrible experience for one in twenty users.

p99 is too punitive. The outliers in p99 are usually the long-tail of model retries, tool timeouts, and the occasional pathological loop. Chasing p99 means optimizing for cases that may not be worth the engineering investment.

p95 hits the right balance. It captures the user experience the vast majority of users actually have. It is sensitive enough to alert on real degradation but not so sensitive that it fires on every transient blip.

Why 8 seconds and not 5 seconds or 12 seconds

The number is empirical, not theoretical. Conversational agent research throughout 2024 and 2025 converged on roughly 8-10 seconds as the threshold where user perception shifts from "the agent is thinking" to "the agent is broken." Below 8 seconds, users continue engaging. Above 10 seconds, abandonment rates rise sharply.

The 8-second target is the production floor. Some agents can do better — voice agents (Vapi, Retell, Bland, ElevenLabs Agents) demand sub-2-second turn-by-turn latency. Some agents have to do worse — multi-step research agents that require five tool calls and an LLM-as-judge verification step routinely exceed 30 seconds.

For the canonical conversational agent — Slack-embedded, Teams-embedded, web-chat, in-app — 8 seconds is the SLO the user-experience research supports.

How the alert fires before users notice

The naive alerting rule: if p95 latency exceeds 8 seconds, page on-call. The naive rule is wrong because it fires after the threshold has already been breached, which means the user experience has already degraded.

The 2026 production pattern: alert on the leading indicator, not the threshold. The leading indicators are p95 latency exceeding 6 seconds for 5 minutes (75% of the threshold), or p99 latency exceeding 15 seconds (the outlier wave that usually precedes a p95 shift), or the rate of latency increase exceeding 20% over a 15-minute rolling window.

Wire all three as page-worthy. The on-call gets the page when the slope of latency is bad, not when latency is already bad. They have time to identify the cause — usually model provider degradation, a tool returning slower, or a recently shipped prompt change that increased token volume — before users start complaining.

The streaming vs. non-streaming question

Streaming responses change the latency calculation. The user perceives time-to-first-token, not time-to-final-token. A 12-second response streamed at 200ms-to-first-token feels faster than a 4-second response delivered all-at-once at the end.

For streaming agents, define two SLOs: time-to-first-token p95 under 1 second, and time-to-final-token p95 under 8 seconds. The first SLO governs perceived responsiveness; the second governs total experience. Some teams pin only the time-to-first-token SLO and accept that long total-completion times are acceptable as long as the stream stays live.

SLO 2: Accuracy — Above Human Baseline on the 50-Case Eval

The accuracy SLO is the trickiest of the four because there is no universal definition of accuracy for an agent. Accuracy depends on the task. A customer-support agent's accuracy is different from a lead-enrichment agent's accuracy is different from a code-review agent's accuracy.

Why the 50-case eval anchor

The 2026 convention is to define accuracy as a score against a fixed, well-curated eval set of 50 cases. The 50-case eval is small enough to maintain rigorously, large enough to be statistically meaningful, and stable enough to provide a fixed reference point quarter-over-quarter.

The 50 cases are not chosen arbitrarily. The set is composed of golden cases (the agent should clearly succeed), edge cases (subtle situations where good agents succeed and bad ones fail), and adversarial cases (security or behavior-contract violations the agent must refuse). A typical breakdown: 25 golden, 15 edge, 10 adversarial. The eval is graded by a combination of deterministic checks, structured-output validation, LLM-as-judge scoring, and a sample of human review.

Why "above human baseline"

The accuracy target is not "above 80%" or "above 90%." Those numbers are arbitrary and quickly become political — every stakeholder will argue for a higher number, and the engineering team will spend cycles chasing improvements that do not change user experience.

The defensible anchor is the human baseline: how well does a human operator perform on the same 50-case eval? If the agent's accuracy is at or above the human baseline, the agent is providing value at scale and the SLO is met. If the agent's accuracy is below the human baseline, the agent should not be making autonomous decisions on those cases.

Establishing the human baseline takes one-time effort. Have three trained operators independently process the 50 cases. Average their scores. Use the average as the baseline. Re-run the human baseline calibration annually because the eval set evolves and so does what the team considers "correct."

How to wire the accuracy alert

The accuracy SLO is enforced through two mechanisms.

The first mechanism is the CI eval that runs on every change — every prompt change, every model change, every tool-schema change. When the smoke eval drops below the human baseline, the CI gate blocks the merge. The accuracy SLO becomes a release gate, not just a metric.

The second mechanism is the nightly full eval, which runs the complete 50-case set against the current production agent and emits a score. When the nightly score drops below threshold, a GitHub issue opens, the team is notified in Slack, and the on-call may be paged if the drop is more than 5 points. The shadow eval against production traffic adds a third signal — sampled production runs are scored, and a sustained drop in production scoring fires an alert.

The accuracy decomposition stakeholders see

The single accuracy score is the SLO number. The dashboard stakeholders look at decomposes it into the three categories: golden score, edge score, adversarial score. When the aggregate drops, the decomposition tells the team where the drop is. A drop in golden cases means basic competence regressed. A drop in edge cases means subtle handling regressed. A drop in adversarial cases means a safety regression — the most urgent class.

The owner cares about the single SLO number. The builder cares about the decomposition. The reviewer cares about case-by-case diffs. Three different artifacts, one source of truth.

SLO 3: Cost-Per-Run with a Two-Sigma Alert

Cost-per-run is the SLO that defends the business case for the agent. If cost-per-run drifts upward, the agent becomes unprofitable even if every other metric is fine. The cost-per-run SLO is the early-warning system for runaway loops, expensive tool calls, and silent provider pricing changes.

Defining the cost-per-run baseline

The baseline is empirical: measure the cost of each agent run over a representative period (typically two weeks of production traffic), compute the distribution (mean, median, standard deviation), and pin the baseline.

The cost includes everything: prompt tokens, completion tokens, tool-call API costs (per-call charges from MCP servers or third-party APIs), embedding generation, vector-store queries, observability platform overhead, and any LLM-as-judge or guardrail costs. The total per-run cost is the number the SLO defends.

A representative baseline for a 2026 conversational agent: $0.04-$0.18 per run, with the variance heavily dependent on whether the agent uses GPT-5, Claude Sonnet 4.5, Gemini 2.5, or a smaller model like Claude Haiku or GPT-5-mini. Multi-step research agents are typically $0.30-$2 per run. Voice agents add the STT/TTS cost layer and run $0.40-$1.50 per session-minute.

Why two-sigma instead of a hard threshold

A hard threshold ($0.20 per run, alert if exceeded) is fragile. The legitimate variance in cost-per-run is high — long conversations cost more than short ones, complex queries cost more than simple ones, retries cost more than first-time successes. A hard threshold either alerts constantly (false positives) or never alerts on real regressions (false negatives).

The two-sigma alert is adaptive. The alerting system computes the mean and standard deviation of cost-per-run over the trailing seven days. If any individual run's cost exceeds the mean plus two standard deviations, an alert fires. If the cohort cost-per-run (e.g., the rolling hourly average) exceeds two-sigma above the trailing baseline, an alert fires.

Two-sigma captures roughly 2.5% of natural variance — the alert fires when something is genuinely anomalous, not when the agent is having a busy day. The alert routes to the on-call. The diagnostic is usually one of three: runaway loop, expensive tool call, or silent provider pricing change.

The runaway loop signature

The most common cost SLO breach is the runaway loop. A tool returns a payload the agent finds interesting. The agent reasons about the payload, calls the tool again with a refined query. The tool returns another interesting payload. The cycle compounds. A single run consumes 100x the baseline tokens before the timeout or token-limit kills it.

The signature: cost-per-run jumps from $0.08 to $4 on a single run. Tool-call count for that run is 40 or 60 instead of the baseline 3-5. Token count is 200,000 or 500,000 instead of the baseline 5,000. The two-sigma alert catches the single anomalous run within minutes; the on-call investigates; the loop condition is identified; a tool-call-budget guardrail is added.

The silent provider pricing change

The most strategic cost SLO breach is the silent pricing change. The provider raises per-million-token pricing. Or adds a tier-pricing structure. Or changes how tool-use tokens are counted. No HTTP error fires. No latency alert fires. Just the cost dashboard begins to creep upward run-over-run.

The two-sigma alert catches this within a few days. The diagnosis is straightforward: check the provider's pricing page, check the platform's API change log, compare token counts before and after. The remediation may involve renegotiating the provider contract, swapping to a cheaper model, or eating the cost while the team plans the migration.

The cost-per-task vs. cost-per-run distinction

Some teams track cost-per-task instead of cost-per-run. A task is the business unit — resolving a customer ticket, enriching one lead, drafting one email — which may take multiple agent runs (each with its own retries and tool calls).

Cost-per-task is the metric the business case is built on. Cost-per-run is the metric the engineering team operates against. Both matter; track both. The two-sigma alert is wired to cost-per-run because that is where the anomaly surfaces first. Cost-per-task is the quarterly metric for the QBR.

SLO 4: Escalation Rate as a Quality Metric, Not a Failure Metric

The escalation rate is the percentage of agent runs that hand off to a human — either because the agent refused to act, requested approval, or because the user explicitly escalated. The way most teams think about escalation rate is wrong. They treat it as a failure metric: "the agent escalated 12% of the time; we need to get that to 5%."

The 2026 production reframing: escalation rate is a quality metric. An agent that escalates appropriately is doing the right thing. An agent that escalates too little is overstepping its competence. An agent that escalates too much is underutilizing its capability. The SLO target is a range, not a ceiling.

Why the failure-metric framing is wrong

The failure framing assumes the agent should handle everything. Escalation is treated as the agent giving up. The team optimizes for "fewer escalations" and the agent learns — through prompt engineering, through model swap, through tool addition — to push through cases it should be deferring.

The result: the agent's competent-zone accuracy drops because it is now operating outside the zone. Adversarial cases that should have been refused get acted on. Edge cases that should have been escalated get handled with confident-but-wrong outputs. The "fewer escalations" optimization has degraded the overall quality.

The quality framing inverts the logic. The agent should escalate exactly the cases that fall outside its competent zone. The right escalation rate depends on the case distribution — if 15% of production traffic is genuinely outside the agent's competence, the right escalation rate is 15%. Optimizing to 5% means forcing the agent to handle cases it should not handle.

The escalation rate SLO range

The 2026 production convention is to set escalation rate as a range, not a single target. For a customer-support agent: 8-18% escalation rate, with both edges of the range as alerting boundaries. Below 8% may indicate the agent is handling things it should not. Above 18% may indicate the agent is over-escalating, possibly because of an overly-conservative prompt or a model swap that made the agent less capable.

For a sales-research agent: 5-12% escalation rate. Lower because more of the work is routine and the cases requiring human judgment are rarer.

For a code-review agent: 20-40% escalation rate. Higher because much of the value of the agent is filtering — the agent is supposed to escalate the cases the human reviewer needs to look at.

The exact range is calibrated empirically and re-calibrated as the agent evolves. The range goes in the SLO document and is reviewed quarterly.

The escalation-rate-by-category breakdown

The aggregate escalation rate is the SLO number. The dashboard decomposes it. Escalation rate by case category (refused-action, requested-approval, user-initiated). Escalation rate by case type (refund processing, technical issue, billing question, account change). Escalation rate by confidence level (high-confidence vs. low-confidence agent responses that nevertheless escalated).

When the aggregate escalation rate moves, the decomposition tells the team why. A spike in user-initiated escalations means the agent's outputs are not being trusted by users. A drop in refused-action escalations means the agent is acting on cases it previously refused — a behavior change worth investigating.

Alert Design: Fires Before Users Notice

An SLO is only useful if the alerting design lets the team respond before users have noticed the breach. The 2026 production alerting patterns:

Leading indicators, not threshold breaches

Alert on the slope, not the value. For latency: alert when p95 is at 75% of threshold and rising at >20%/15min. For cost-per-run: alert on individual two-sigma anomalies (single-run signal) and on rolling-hour two-sigma (cohort signal). For accuracy: alert on the production-traffic shadow eval scoring, not just the nightly run. For escalation rate: alert on either edge of the range being approached, not breached.

Multi-signal correlation, not single-signal alerts

Single-signal alerts produce false positives. Latency might spike for 5 minutes because the model provider had a regional issue. Cost might spike because someone is testing an expensive query. Accuracy might dip because a synthetic eval case was added that the agent has not been tuned for.

The 2026 pattern: correlate signals. The page fires when latency is high AND error rate is rising AND cost is rising. Or when accuracy drops AND production traffic volume is normal AND no recent change explains it. The correlation logic filters out the obvious false positives.

This is implemented via the observability stack (Datadog monitors with composite conditions, Honeycomb triggers, Grafana alerting with multi-condition logic) or via the eval/observability platforms (Langfuse alerts, LangSmith monitoring, Arize Phoenix anomaly detection, Helicone budgets, Braintrust trend alerts).

Alert routing by class

Not every alert pages the on-call. The 2026 convention: alerts are tiered.

• Page-worthy. Pages the on-call. Includes the five page-worthy signals from Lesson 1: latency breach, cost-per-run two-sigma, error rate above 1%, eval regression, manual escalation. Acknowledged in 5 minutes.
• Ticket-worthy. Files a ticket. Includes drift signals that are concerning but not actionable in 5 minutes: gradual cost creep, slow accuracy drift, escalation rate approaching edge of range. Reviewed at the daily standup or weekly review.
• FYI. Posts to a Slack channel. Includes routine deployment notifications, eval-set evolution, dashboard milestones. Read on the next workday.

The alert design discipline: every alert has a defined class, a defined routing destination, and a defined ack expectation. Alerts that violate any of those are recategorized or retired.

Error Budgets and Release Velocity

The classical SRE concept of an error budget translates directly to agent ops. The error budget is the inverse of the SLO: if the SLO is 95% accuracy, the error budget is 5% of runs allowed to fall below the accuracy threshold before the team is in violation.

How the error budget governs releases

When the error budget is healthy (well above zero), the team can ship changes freely. Class 2 and Class 3 changes go out as the team has reviewed them. The agent is in good standing.

When the error budget is depleted (the SLO has been violated and the cumulative damage is close to the allowable threshold), the team's release velocity is throttled. Class 2 changes still ship but with extra scrutiny. Class 3 changes pause until the budget is replenished. Class 4 changes pause unconditionally.

The error budget creates a natural tension between feature velocity and reliability. The team that pushes too hard burns through the budget and has to slow down. The team that is too cautious accumulates budget that is never used. The dynamic is self-correcting.

Error budget tracking in practice

Each of the four SLOs has its own error budget. The latency error budget tracks minutes-of-SLO-breach against the monthly allowance. The accuracy error budget tracks cumulative accuracy-drop-points-below-baseline against monthly allowance. The cost error budget tracks dollars-over-baseline against monthly allowance. The escalation budget tracks days-out-of-range against monthly allowance.

Each budget is tracked in a dashboard the on-call sees. Each budget triggers a different policy when depleted: latency budget depleted means model swap or routing change is high priority; accuracy budget depleted means prompt and retrieval work; cost budget depleted means guardrail tuning; escalation budget depleted means the agent's autonomy boundary needs re-thinking.

The SLOs Not to Commit To

Some metrics sound like SLOs but should not be SLOs. Picking the wrong SLOs is worse than picking too few.

User satisfaction (NPS, CSAT)

Satisfaction metrics are too slow and too sparse. The NPS survey runs quarterly. The CSAT response rate is 10-20%. The signal is noisy on a per-week basis and useless for real-time alerting. Satisfaction is a quarterly outcome metric, not an operational SLO.

Throughput (runs per second)

Throughput sounds like an SLO. It is actually a capacity metric. The agent can handle X runs per second; if traffic exceeds X, the queue grows. The right operational response is autoscale, not page-the-on-call. Throughput goes on the capacity dashboard, not the SLO dashboard.

Tool-call success rate

This sounds operational but is too fine-grained to defend. The agent calls 5-15 tools per run. Some calls fail and are retried. Some calls fail and are routed around. A tool-call success rate of 92% might be perfect (the agent recovers from all failures) or terrible (the agent gives up on 8% of calls). The metric is too lossy without decomposition.

Model-token-count

Tokens are an implementation detail. The cost-per-run SLO covers what the team actually cares about. Optimizing for fewer tokens for the sake of fewer tokens is a distraction.

Time-to-first-action

For some agent designs, this is meaningful. For most, it is a redundant decomposition of latency. The p95 latency SLO captures the user experience without the added complexity.

Putting It Together: The SLO Document

The four SLOs go in an SLO document that is owned by the agent's owner (business stakeholder) and maintained by the builder. The document is reviewed quarterly. The structure:

1. Agent identification. Name. Use case. Owner. Builder. Reviewer.
2. SLO 1: Latency. Target (p95 under 8s for conversational, or alternative for async). Alerting thresholds (leading indicators at 6s p95, 15s p99, >20%/15min slope). Error budget (minutes of breach per month).
3. SLO 2: Accuracy. Target (above human baseline on 50-case eval). Eval set composition (25 golden, 15 edge, 10 adversarial). Human baseline value and date of last calibration. Alerting thresholds (CI gate, nightly eval, shadow eval).
4. SLO 3: Cost-per-run. Baseline value (computed empirically, dated). Components included (tokens, tool calls, embedding, observability). Two-sigma alert configuration. Error budget (dollars over baseline per month).
5. SLO 4: Escalation rate. Range (8-18% for support, 5-12% for research, etc.). Categorization breakdown. Alerting thresholds (both edges of range). Quarterly recalibration date.
6. Release policy. How error-budget depletion throttles Class 2, 3, 4 changes.
7. Review cadence. Quarterly with the owner. Annually with governance. Monthly with the builder team.

The SLO document is the artifact the QBR refers to. It is the artifact the on-call references when an alert fires. It is the artifact the new hire reads to understand what the team is committed to.

Key Takeaways

• An SLO is a metric plus a target plus a consequence. Dashboards without SLOs are situational-awareness widgets, not accountability instruments. Pick four SLOs you will defend at the QBR.
• Four 2026 production SLOs: p95 latency, accuracy against 50-case eval, cost-per-run with two-sigma alert, escalation rate as a quality range. Each covers a distinct failure dimension with a distinct response.
• Latency: p95 under 8 seconds for conversational agents (the empirical user-experience threshold). Voice agents demand sub-2s turn-by-turn; async agents track a 60-second per-run budget. Streaming agents pin time-to-first-token p95 under 1s.
• Accuracy: above the human baseline on a fixed 50-case eval (25 golden, 15 edge, 10 adversarial). The human baseline is established by averaging three trained operators and re-calibrated annually. CI gate, nightly eval, and shadow eval are the three enforcement mechanisms.
• Cost-per-run: empirically baselined over two weeks of production traffic. The two-sigma alert catches anomalous runs within minutes — typically a runaway loop, an expensive tool call, or a silent provider pricing change. Cost-per-task is the QBR metric; cost-per-run is the operational metric.
• Escalation rate is a quality metric, not a failure metric. The SLO is a range (8-18% for support, 5-12% for research, 20-40% for code review). Below the range means the agent is overstepping; above means the agent is over-deferring. Optimize for the right range, not zero escalations.
• Alert on leading indicators (slope, 75% of threshold) rather than threshold breaches. Correlate signals (latency AND error rate AND cost) to filter false positives. Tier alerts into page-worthy, ticket-worthy, and FYI with defined routing for each.
• Error budgets translate from SRE to agent ops. Each SLO has its own budget. Depletion throttles release velocity: Class 4 changes pause unconditionally, Class 3 paused until replenished, Class 2 shipped with extra scrutiny.
• SLOs not to commit to: NPS/CSAT (too slow), throughput (capacity not SLO), tool-call success rate (too lossy), token count (implementation detail), time-to-first-action (usually redundant with latency).
• The SLO document is the QBR-grade artifact. Owned by the business owner, maintained by the builder, reviewed quarterly. The on-call references it during incidents. The new hire reads it to understand the team's commitments.