Every production agent in 2026 needs three dashboards. Not five. Not twelve. Three. Latency by step (p50, p95, p99) so you can find the slow nodes. Cost per run (with model breakdown) so you can find the expensive runs. Pass-rate over time (against your eval set) so you can see quality moving in real time. That's it. Build those three with live URLs you can send to your CRO at 10 PM during a board meeting, and you have the agent-observability discipline that separates teams shipping reliable agents from teams who hope. This lesson walks through exactly what goes on each dashboard, which platforms make them easy in 2026, and the specific failure modes each dashboard surfaces.

Why Three Dashboards (and Not Twelve)

The temptation when you stand up observability is to build a dashboard for everything. Dashboards for token usage by model. Dashboards for prompt-by-prompt latency. Dashboards for cache hit rate. Dashboards for tool-call counts. Dashboards for user feedback rates. Dashboards for retrieval recall. You end up with twelve dashboards nobody looks at except the engineer who built them.

The discipline that works in 2026: pick three dashboards that answer the three questions a leader will actually ask, build them well, and put the URLs in your team's Slack header. The three questions:

1. Is the agent slow? When users complain about latency, where in the run is the time going? (Latency by step.)
2. Is the agent expensive? Are we on track for the budget? Which model contributes how much? (Cost per run with model breakdown.)
3. Is the agent good? Is quality holding, improving, or regressing? Against what reference? (Pass-rate over time against the eval set.)

These three questions are what the CRO, the CTO, the product lead, and the on-call engineer all want to know. The same three dashboards serve all four audiences. The rest can be ad hoc — trace inspection in LangSmith, queries in Helicone, drill-downs as needed.

Three dashboards covers 90% of agent-observability decisions. The other 10% lives in trace inspection. Twelve dashboards covers 100% of decisions and is looked at by no one, which is the same as covering 0%.

Dashboard 1: Latency by Step (p50, p95, p99)

The first dashboard answers: when the agent is slow, where in the run is the time going?

What goes on this dashboard

One panel per major step in the agent's flow. For a typical multi-step agent — classify, retrieve, plan, generate, validate, send — you have six panels. Each panel shows p50, p95, and p99 latency over time (rolling 24-hour or 7-day window). The Y-axis is milliseconds; the X-axis is time. Each line tells you a different story:

• p50 is the typical user's experience. If p50 doubles, the agent has gotten noticeably slower.
• p95 is the bad-day user's experience. If p95 is 5x p50, you have a long tail problem.
• p99 is the SLA risk. If p99 spikes past your SLA, some users are timing out.

You also need a top-level panel: total agent latency p50/p95/p99. The per-step panels tell you where the time goes; the top-level panel tells you whether anyone should care.

The bug-finding pattern

The shape that screams "go look at this": one step's p95 or p99 spiking while p50 stays flat. A retrieval step that's normally 200 ms at p95 spiking to 4 seconds at p99 means a class of queries is going wrong — a too-broad search returning too many candidates, a slow embedding model, a vector DB taking a hot path through cold storage. The trace dashboard from Lesson 1 is where you'd then drill in; the latency dashboard told you which step to drill into.

The other shape that matters: a single step's p50 quietly trending up over a week. The agent isn't broken, but it's getting slower. Often this is "the prompt got longer" (someone added context that should have been retrieval) or "the model got slower" (a provider released a heavier default).

How to build it

Three options in 2026:

1. Native platform dashboards. LangSmith, Langfuse, and Phoenix each have built-in latency-by-step dashboards. Tag your steps in the trace (run_type for LangSmith, observation name for Langfuse, span name for Phoenix) and the dashboards build themselves. This is the fastest path; the dashboards are usable but generic.
2. Helicone for LLM-call latency. Helicone gives you p50/p95/p99 by model and by feature tag, with nice charts. Limited to LLM calls; not great for tool steps and retrieval. Pair with one of the SDK platforms for the full picture.
3. Datadog or Honeycomb for whole-stack latency. If you've wired traces to Datadog LLM Observability or Honeycomb LLM Observability (covered in Lesson 4), you can build the latency dashboards in those tools and overlay them with HTTP layer, database, and infrastructure metrics. Most polished; takes more setup.

The pitfalls

Two specific traps:

• Plotting only averages. The p50 hides the tail. A 1 second average can be 200 ms p50 and 8 second p99 — very different operationally. Always plot all three.
• Lumping all steps together. A single "total agent latency" panel without per-step breakdown tells you the agent is slow but not where. The per-step breakdown is where the debugging signal lives.

Dashboard 2: Cost Per Run with Model Breakdown

The second dashboard answers: what does it cost to run this agent, and where does the spend go?

What goes on this dashboard

Four panels at minimum:

1. Cost per run distribution. A histogram of per-run cost in dollars. Most runs cost a few cents; the tail goes to the runs that cost dollars or more. The right tail is where surprises live.
2. Total daily cost. A bar chart of daily spend. Tracks against your budget. The line a CFO actually wants to see.
3. Cost breakdown by model. A stacked area chart. Today's spend by gpt-5.5, gpt-5.5-mini, claude-opus-4.5, claude-sonnet-4.5, fine-tuned-haiku, etc. Tells you which model is responsible for which share of cost.
4. Top expensive runs. A leaderboard. The 10 most expensive runs in the last 24 hours. Each row links to the full trace. This is where you click in when cost spikes.

Optional but valuable: cost per feature, cost per tenant, cost per user. If you serve multiple customers or feature surfaces, knowing which one is burning the budget is the difference between an actionable dashboard and a wall of numbers.

The bug-finding pattern

The shapes to watch:

• Daily cost rising faster than usage. Cost-per-run is growing. Either the prompts got longer, the model got more expensive, or the agent is doing more retries. The model-breakdown panel tells you which model's share is growing.
• A single run in the top-expensive leaderboard 10x larger than the rest. Pathological run. Click into the trace, find the runaway loop or the document the agent dumped into context. A common 2026 bug: a retrieval step returning a 200-page PDF that all got loaded into the next LLM call.
• Sudden spike in expensive-model share. The fallback path is firing. Something is failing on the cheaper model and falling through to the expensive one. The latency dashboard usually shows the same incident; the cost dashboard tells you what it's costing.

How to build it

Two options in 2026 that are sufficient:

1. Helicone. Helicone's cost dashboards are best-in-class for the proxy pattern. You get distribution, total, model-breakdown, and top-spend out of the box. If you've followed Lesson 2, this dashboard is essentially free.
2. LangSmith / Langfuse / Phoenix cost views. Each platform tracks cost natively (computed from token counts and model pricing). The dashboards are usable; LangSmith's is the most polished for LangChain stacks. Pair with the trace tree so you can click from a cost outlier to its full trace.

The pitfalls

• Aggregating to monthly only. Monthly is too coarse. Daily is the right granularity for catching incidents before they compound.
• Forgetting model pricing updates. Provider pricing changes; the dashboard's computed cost can drift from your invoice if the pricing data isn't current. Use a platform that updates pricing automatically (all three SDK platforms do; Helicone does).
• No tenant attribution. If you serve multiple customers and don't attribute cost per tenant, you can't make sane pricing decisions. The Helicone-Property-Tenant pattern (Lesson 2) earns its keep here.

Dashboard 3: Pass-Rate Over Time (Against Your Eval Set)

The third dashboard answers: is the agent's quality holding, improving, or regressing? This is the dashboard most teams skip and the one that pays the most.

What goes on this dashboard

One primary panel:

• Pass rate against eval set over time. A line chart. Y-axis is percentage passing your eval set (0-100). X-axis is time, with markers for each agent version deployed. The line should climb gently or plateau. A drop is a regression you need to investigate.

Three supporting panels:

1. Pass rate by eval category. Your eval set has categories (the ones we built in Chapter 3.5: happy path, ambiguous queries, adversarial inputs, edge cases). Plot each category separately. A regression in "adversarial inputs" but not "happy path" tells a different story than a regression in both.
2. Failed cases leaderboard. Top 10 eval cases that failed the most recent run, with click-through to the failing trace. The on-call engineer reads this on Monday morning.
3. Latency-to-evaluate. How long does it take to run your eval set against a new version? If it's hours, your iteration speed is slow. If it's minutes, you can run evals on every deploy. Latency here drives velocity downstream.

The bug-finding pattern

The most important shape on this dashboard: a drop in pass rate after a deploy. If yesterday's version passed 92% of evals and today's passes 79%, you have a regression. The version markers on the X-axis let you locate the deploy that caused it. The eval-category panel tells you whether the regression is general or category-specific.

The slower-moving shape: gradual drift downward over weeks. This often reflects evolving production traffic the agent wasn't built for; the eval set captures the agent's static notion of "good," but the actual users have moved on. You need to refresh the eval set with new edge cases (the synthetic-eval generation lesson before this chapter is the playbook).

How to build it

The eval-driven pieces:

1. Eval set as a versioned dataset. Stored in LangSmith Datasets, Langfuse Datasets, or Phoenix Datasets. Each example has input, expected output, and category. Versioned so you know which version of the eval set was used.
2. Evaluator function. Some examples have exact-match expected outputs; others use LLM-as-judge for fuzzy correctness. The platforms support both.
3. Scheduled runs. The eval suite runs against production agent versions on schedule (typically nightly) and on every deploy. Results write to the dataset's run history.
4. Dashboard. Reads from the run history. Plotted over time with deploy markers.

The mechanic is the same in all three SDK platforms; the UIs differ. LangSmith's experiments view is the most polished for LangChain stacks; Langfuse's is the most cleanly versioned (datasets have explicit versions); Phoenix's is the most flexible with custom evaluator code.

The pitfalls

• An eval set that doesn't reflect production traffic. The pass rate is high; users still complain. The eval set is testing what the agent could do six months ago, not what users actually ask. Refresh quarterly at minimum.
• LLM-as-judge evaluators that drift. If your evaluator is itself an LLM call, the evaluator's behavior changes as the underlying model updates. Pin the evaluator's model version, and run a sanity check (does the evaluator agree with human judgments on a held-out set) periodically.
• Pass rate as a single number. 87% pass rate tells you nothing about what's failing. The by-category and failed-cases panels are where the actionable signal lives.

The Live Dashboard URL Rule

An organizing principle that we've watched separate the best agent teams from the rest in 2026: every dashboard has a stable URL, and the URL is in the Slack channel header.

Why this matters operationally:

• When the CRO asks "how's the agent doing?" at a Tuesday standup, you don't open a tool and navigate; you paste a URL. The conversation happens around live data, not memory.
• When the on-call engineer gets paged at 2 AM, the URL is the first thing they open. They see latency by step in 15 seconds; if a single step's p99 has spiked, they know where to drill.
• When you onboard a new engineer, the dashboards are the second thing you show them after the codebase. They're documentation by graph.

The URL discipline is operationally simple but culturally important. A dashboard that lives behind three clicks in someone's bookmarks is a dashboard nobody references during incidents. A dashboard you can paste in chat is a tool everyone uses.

A Real Dashboard-Driven Incident Story

A 30-person fintech runs a credit-decisioning agent that scores small-business loan applications. The agent has been in production since November 2025. They built the three dashboards in December using LangSmith for latency-by-step and pass-rate, and Helicone for cost.

In late February 2026, the head of underwriting pinged the data team: "credit recommendations feel weird lately. Are we good?" The team's lead engineer opened the pass-rate dashboard. The line had drifted from 91% to 84% over the prior two weeks. The drop was concentrated in the "ambiguous credit history" category — applications with missing or contradictory income data, the hardest cases.

She opened the failed-cases leaderboard, clicked into the top failing trace, walked it in LangSmith. The model was now recommending denials for cases that previously triggered a "request more documentation" recommendation. The model had been updated — the team had switched from Claude Sonnet 4 to Claude Sonnet 4.5 ten days earlier, on the assumption that newer-is-better. The latency dashboard confirmed the deploy date (latency p50 dropped slightly when the model changed, a telltale signature).

The fix: revert to Sonnet 4 for the ambiguous-case branch, keep 4.5 for the clean-data branch. Pass rate recovered to 92% within three days. Without the dashboards, the head of underwriting's "feels weird" would have been hard to debug — it might have taken weeks of manual case review to spot the pattern.

This is the value of the three-dashboards discipline. The vague feeling becomes a measured drop, the drop becomes a category, the category becomes a deploy date, the deploy date becomes a model version, and the model version becomes a fix. Four data points, fifteen minutes.

Dashboards You Do Not Need (Yet)

Three dashboards we've watched teams build and regret:

• Token usage by model. Already covered by the cost-breakdown panel in Dashboard 2. A separate token dashboard is duplicative.
• Cache hit rate over time. Useful as a sanity check, but Helicone's main view shows it as a number; you don't need a time-series panel until you're actively tuning caching.
• User-feedback rate over time. Important signal, but most teams confuse it with quality. Feedback rate reflects users' willingness to click a thumbs-down, which is influenced by UX placement and incentive — not a clean quality signal. Use feedback as a discovery mechanism for bad traces, not a top-level dashboard.

The principle: every dashboard you build is a dashboard someone has to look at. Bias toward fewer, sharper dashboards rather than comprehensive coverage.

How the Three Dashboards Pay Off

Three specific operational moments where the three-dashboard discipline shows its value:

The Monday morning routine

The on-call engineer opens the three URLs at 9 AM Monday. Latency by step: anything spiking? Cost per run: anything anomalous? Pass rate: anything dropping? Five minutes of scanning catches 80% of the prior week's incidents that nobody alerted on.

The board prep moment

The CRO needs to show the board that the AI investment is paying off. She doesn't need engineering metrics; she needs the storytelling versions. The cost dashboard (down 20% month-over-month after caching) and the pass-rate dashboard (climbing) and the latency dashboard (steady under SLA). Three URLs, three slides, board done.

The deploy gate

Every deploy triggers an eval suite run. The pass-rate dashboard updates with a new data point. If pass rate dropped, the deploy is rolled back automatically (some teams) or flagged for review (most teams). The dashboard isn't reactive; it's a deploy-gate driver.

Key Takeaways

• Three dashboards, not twelve. Latency by step (p50, p95, p99). Cost per run with model breakdown. Pass rate over time against your eval set. These three answer the three questions a CRO, CTO, product lead, and on-call engineer all want to know: Is the agent slow? Is it expensive? Is it good?
• Latency-by-step dashboard. One panel per major step, p50/p95/p99 over time. The shape to watch: a single step's p99 spiking while p50 stays flat. The platforms that build it for free: LangSmith, Langfuse, Phoenix (tag your spans). Pair with Datadog/Honeycomb for whole-stack overlay.
• Cost-per-run dashboard. Distribution histogram, daily total, model breakdown stacked area, top-expensive leaderboard with trace links. Optional: cost per feature, per tenant, per user. Helicone makes this dashboard essentially free; SDK platforms cover it too.
• Pass-rate dashboard. Pass rate against eval set over time, with deploy markers. Plus by-category breakdown, failed-cases leaderboard, time-to-evaluate. The dashboard most teams skip and the one that pays the most.
• Three pitfalls. Plotting only averages (hides the tail). Lumping all steps together (no debugging signal). An eval set that doesn't reflect production traffic (pass rate is high; users complain).
• The live URL rule. Every dashboard has a stable URL, and the URL is in the Slack channel header. The conversation happens around live data, not memory. The on-call engineer's first action at 2 AM is opening the URL.
• A real story. A 30-person fintech caught a model-upgrade regression by walking down the dashboard chain: vague complaint → pass rate dropped 91% to 84% → ambiguous-credit-history category specifically → deploy date matches → Sonnet 4 → 4.5 swap → revert for that branch → recovery in three days. Four data points, fifteen minutes of debugging.
• Dashboards you do not need yet. Token usage by model (duplicates cost). Cache hit rate over time (Helicone shows the number). User-feedback rate (signal is noisy; use feedback as a discovery mechanism for bad traces, not a top-level dashboard).
• Three operational moments where the discipline pays. Monday morning scan catches 80% of unsurfaced incidents in five minutes. Board prep becomes three URLs/three slides. Deploy gate uses pass-rate as the auto-rollback signal.
• The principle: every dashboard is one someone has to look at. Bias toward fewer, sharper dashboards. Three live URLs you can show your CRO at 10 PM beats twelve dashboards nobody opens.