The first time you read a complete agent trace — not a log file, not a sampled span, but the full thing: every model call, every tool invocation, every retry, every token of input and output, laid out as a tree you can click through — is the moment debugging an agent stops being divination and starts being engineering. This lesson walks you through doing that in the three platforms that own the agent-tracing category in May 2026: LangSmith for LangChain and LangGraph teams, Langfuse for open-source and self-hosted setups, and Arize Phoenix for ML-grade rigor. You'll learn how to instrument a multi-step run, how to read the trace top-to-bottom, and how to identify the exact step that produced bad output in under three minutes — even on a 47-step run with two parallel tool calls and a retry storm.

Why Tracing Is the First Observability Investment

Agents fail in ways that no single log line can explain. A customer-support agent answers wrong; the prompt looks fine, the model is fresh, the tool returned data, and yet the final reply hallucinated a refund policy that doesn't exist. The reason is somewhere in the middle of 14 steps: the retrieval tool returned the wrong document, or the agent misread a JSON field, or the prompt-compression step dropped the citation, or the second model call ignored the first model's caveat. Without a trace, you read 14 log lines in 14 different formats and guess. With a trace, you see the tree, click the suspicious node, and read its inputs and outputs in seconds.

By May 2026, "trace your agent" is no longer optional advice. Every production agent we've seen ship in 2025 and 2026 ran with a tracing platform attached from week one. The teams that skipped it lost weeks to debugging incidents they could have solved in an afternoon if they'd been able to read the trace.

An agent without a trace is a debugger without a stack. You can still run it. You just can't reason about it when it breaks. And it will break — agents always break in production in ways you didn't anticipate.

What a trace gives you that logs don't

Structured logs tell you what happened, in order, in flat lines. A trace tells you what happened in tree — which parent step called which children, what those children did, how long each took, what they cost, and which one produced the output that flowed into the next. The tree is the unit of agent observability because agents are recursive: a planner calls a subagent which calls a tool which calls another tool. A flat log loses the parent-child relationship and you waste an hour reconstructing it.

The other thing a trace gives you: full payloads. The prompt that actually went to the model after all the templating. The response the model actually returned before parsing. The tool's actual JSON argument. The actual JSON result. No "[truncated]" in the middle of the part you care about.

The Three Platforms: Positioning

By mid-2026, the agent-tracing market has stratified into three platforms with overlapping but distinct positioning. Most builders pick one based on stack and team philosophy.

LangSmith — for LangChain and LangGraph teams

LangSmith is LangChain's first-party observability platform. If your agent is built on LangChain or LangGraph, LangSmith is the path of least resistance. Tracing is essentially automatic: import the SDK, set an environment variable, and every chain, agent, tool, and LLM call shows up in the LangSmith UI without further instrumentation. The platform is opinionated about LangChain's mental model — chains, runnables, agent executors, graph nodes — and the UI maps directly to those concepts.

LangSmith's strengths: zero-friction setup for LangChain stacks, native LangGraph visualization (the graph itself shows up as a graph in the UI, not just a list of spans), built-in evaluation runs that compare a new agent version against a reference dataset, and prompt-management features that let you edit, version, and A/B test prompts inside the platform.

Pricing as of May 2026 is per-trace and per-eval, with a free tier sufficient for solo builders and small teams. The platform is cloud-hosted by LangChain, with a self-hosted option for enterprise plans.

Langfuse — for open-source and self-hosted

Langfuse is the open-source agent-tracing platform that won the "I need to self-host this" segment. The code is on GitHub under an MIT-compatible license, the docker-compose setup runs locally in two minutes, and the cloud-hosted version (langfuse.com) is available for teams who'd rather not run their own. Langfuse is framework-neutral — it works with LangChain, LlamaIndex, the OpenAI SDK, the Anthropic SDK, and custom Python or TypeScript code via decorators or context managers.

Strengths: framework-neutral instrumentation, full data ownership when self-hosted (compliance teams love this), prompt management with version control and deployment labels, and a mature evaluation product (datasets, LLM-as-judge evaluators, human annotation queues). Pricing is open source for self-hosted; the managed cloud uses a per-event metering with a generous free tier.

Builders pick Langfuse when they want a non-LangChain stack, when they need their data to stay inside their VPC, or when they want to avoid vendor lock-in to LangChain's ecosystem.

Arize Phoenix — for ML-grade rigor

Arize Phoenix comes from the ML observability world — Arize AI's lineage is structured-ML monitoring (drift, performance, fairness) for tabular ML systems, and Phoenix is their open-source LLM-and-agent extension. Phoenix is the platform you choose when you want OpenTelemetry-native traces, evaluation harnesses that look like research code, and an interactive notebook-friendly UI (Phoenix runs inside a Jupyter notebook session as well as as a standalone service).

Strengths: OpenTelemetry-first instrumentation (every Phoenix trace is also a valid OTel trace, which means it can flow into any OTel-compatible observability backend in addition to Phoenix), strong evaluation primitives (token-level evaluators, retrieval-quality metrics like context relevance and faithfulness, hallucination detection), and a Python-notebook-native workflow that data scientists feel at home in.

Builders pick Phoenix when their team has an ML-rigor bias, when they want OTel as a first-class concern from day one, or when they need to evaluate retrieval quality with rigor (Phoenix has the best out-of-the-box retrieval evaluators in the category as of mid-2026).

Which to pick

Three rules of thumb that hold up in 2026:

1. If your stack is LangChain or LangGraph, default to LangSmith. The integration is free, and switching later is cheap.
2. If you need self-hosting or you're using a non-LangChain stack, default to Langfuse. The docker-compose setup is the fastest path to a self-hosted UI.
3. If your team thinks like ML researchers and OpenTelemetry-native matters to you, default to Phoenix. The eval primitives are best-in-class.

The three platforms have heavily overlapping feature sets in 2026 — all three do tracing, all three do evaluation, all three do prompt management. The choice is more about which tool fits your team's existing muscle memory than about who has the better feature checklist.

Instrumenting a Multi-Step Agent

Let's walk through the same multi-step agent in each platform. The agent: a customer-support triage agent that ingests an inbound email, classifies urgency, looks up the customer's history, drafts a reply, and either auto-sends (low risk) or queues for human review (high risk). Six tool calls, two LLM calls, one branch.

LangSmith instrumentation

If you're on LangChain, instrumentation is two lines:

import os
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "ls__..."
os.environ["LANGCHAIN_PROJECT"] = "support-triage-agent"

Every RunnableLambda, every ChatModel.invoke, every Tool.run is automatically traced and tagged with the project name. The trace tree mirrors your chain or graph structure exactly. For LangGraph specifically, the trace UI renders the graph topology — you see the actual state-graph diagram with each node colored by latency or status, not a flat span list.

If you're not on LangChain but want to use LangSmith anyway, the @traceable decorator wraps arbitrary Python functions:

from langsmith import traceable

@traceable(run_type="tool", name="lookup_customer")
def lookup_customer(customer_id: str) -> dict:
    return crm.get(customer_id)

Langfuse instrumentation

Langfuse's primary integration is decorators and context managers:

from langfuse.decorators import observe, langfuse_context

@observe()
def triage_inbound(email: dict):
    classification = classify_urgency(email)
    history = lookup_customer(email["from"])
    draft = draft_reply(email, history, classification)
    return route(draft, classification)

@observe(as_type="generation")
def classify_urgency(email: dict):
    # LLM call here; Langfuse auto-captures input/output/usage
    ...

The @observe decorator captures function inputs, outputs, latency, and errors. as_type="generation" tells Langfuse this is an LLM call (so it captures tokens and cost). For OpenAI and Anthropic SDKs, Langfuse ships a drop-in client wrapper that captures every call automatically — no decorators needed on the function that wraps the LLM.

Phoenix instrumentation

Phoenix uses OpenTelemetry instrumentation libraries. For OpenAI:

from phoenix.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor

tracer_provider = register(project_name="support-triage-agent")
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

Once registered, every OpenAI call is auto-traced as an OTel span. The openinference family of instrumentors covers OpenAI, Anthropic, LangChain, LlamaIndex, DSPy, Haystack, Bedrock, and Vertex. Custom functions use the OTel API directly:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("lookup_customer") as span:
    span.set_attribute("customer_id", customer_id)
    result = crm.get(customer_id)
    span.set_attribute("output", json.dumps(result))

The Phoenix UI is the consumer of those OTel spans. You can also send the same spans to Datadog, Honeycomb, or any other OTel-compatible backend — that's the OTel-native advantage Phoenix brings.

Reading a Full Trace Top-to-Bottom

Here's what reading a trace actually looks like in practice. The agent failed: it auto-sent a reply that quoted a refund policy from a competitor's docs. Walk through the trace with me.

The root span

You open the trace and the root span is support-triage-agent. Duration: 4.2 seconds. Status: success (no exception was thrown — but the output was wrong, which is why you're here). Total cost: $0.034. Total tokens: 12,847. You expand the root span and see six children: classify_urgency, lookup_customer, retrieve_kb, draft_reply, risk_check, send_reply.

Walk down the tree

You click classify_urgency. It's a generation span — model: gpt-5.5-mini, prompt tokens 412, completion tokens 18, duration 320 ms. The input prompt is shown in full. The output: {"urgency": "medium", "category": "refund_question"}. Looks right. Move on.

You click lookup_customer. It's a tool span — input {"customer_id": "ACC-2218"}, output a customer record with name, plan, and history. Looks right.

You click retrieve_kb. It's a retrieval span — input {"query": "refund policy for annual plans"}, output a list of three documents. You scan the documents. Document one: your own refund policy. Document two: a competitor's blog post about their refund policy. Document three: an internal Notion page from 2023.

Found it. The retrieval step pulled in a contaminated document that shouldn't have been in the knowledge base. The downstream model dutifully synthesized from all three. The "bad" output isn't a model failure — it's a data hygiene failure two steps upstream from where the wrong words appeared.

The 30-second debugging principle

Most agent failures look like model failures and are actually data, retrieval, or tool failures. Without a trace, you spend an hour examining the prompt and the model output and conclude "the model hallucinated." With a trace, you spend 30 seconds walking the tree and find the contaminated document in the retrieval span. The model didn't hallucinate; it accurately synthesized from bad inputs.

This is the value proposition of agent tracing in one paragraph: you stop debugging at the wrong layer.

Filtering and Searching Traces at Scale

A production agent runs hundreds or thousands of traces a day. You can't read them all. You read the ones that matter, and the platforms give you filters for "the ones that matter."

Filter by outcome

Every platform lets you filter by status (error vs success), latency (top 1% slowest), cost (top 1% most expensive), and feedback (thumbs-down from users). Start there. The traces that fail, run slow, cost too much, or get bad feedback are the high-value debugging targets.

Filter by tag and metadata

All three platforms let you attach arbitrary metadata to traces. Tag traces with the user ID, the tenant ID, the agent version, the model used, the experiment flag, the deployment environment. Filter by any combination. "Show me all errored traces from tenant ACME on agent-v4.2 in the last 24 hours."

LangSmith uses tags and metadata on the runnable. Langfuse uses metadata on the observation. Phoenix uses OTel span attributes. The mental model is the same across all three.

Filter by step content

The harder feature: searching inside trace content. "Show me all traces where any retrieval step returned a document with the word 'competitor' in it." LangSmith supports content filters via its query DSL. Langfuse has full-text search across observations. Phoenix integrates with its own SQL query layer over span attributes. All three support this in 2026; LangSmith's UX is the most polished, Phoenix's SQL is the most powerful, Langfuse's is the most approachable for non-engineers.

A Real Multi-Step Debugging Story

A 40-person B2B SaaS shipped a sales-prospecting agent in March 2026. The agent: scrape a prospect's LinkedIn and website, draft an outbound email, check the email against a tone guide, and send via Outreach. Six steps, three LLM calls, four tool calls. The agent had been running for two weeks. Then sales started complaining: "the emails are mentioning the prospect's competitors. Why is the agent talking about competitors?"

The team was using Langfuse (self-hosted; they're a fintech). The lead engineer opened the trace dashboard, filtered for "last 7 days, thumbs-down feedback," and got 12 traces. She opened the first one.

Root span: prospect-outreach-agent. She expanded. Step 1: scrape_linkedin — looks fine, returns the prospect's profile JSON. Step 2: scrape_website — looks fine, returns the website's homepage text. Step 3: extract_company_context — an LLM call. She opened it. Input prompt: "Given the LinkedIn profile and website text below, extract the prospect's company, role, and key buying signals." Output: "Company: Acme Co. Role: VP Engineering. Buying signals: hiring 12 SDRs (LinkedIn), comparing Datadog vs. Honeycomb (website blog)."

The website's blog post — a comparison article — leaked into the buying-signals output. The blog mentioned competitor names. Downstream, the email draft picked it up and wrote "I noticed you're comparing Datadog and Honeycomb..."

Fix in 12 minutes: update the extract_company_context prompt to ignore comparison blog content, add a regex filter on the buying-signals output to strip product names that aren't the prospect's, ship. Without the trace, this would have been a multi-hour investigation across logs, the email tool, the LLM provider, and three Slack channels.

Trace-Driven Development

The teams that move fastest with agents in 2026 develop trace-first. Every change to the agent — new prompt, new tool, new model — goes through a flow that explicitly includes the trace:

1. Make the change locally.
2. Run the agent on a fixed set of test inputs (your eval dataset; see Chapter 3.5).
3. Open the LangSmith/Langfuse/Phoenix UI, look at the traces produced by the new run, and compare against the reference run.
4. If the trace tree looks right and the outputs match expectations, ship.
5. If anything's off, the trace tells you exactly which step went sideways.

This is what "evaluation-driven development" looks like in practice for agents. The trace is the artifact. The eval dataset is the test suite. The diff between runs is the regression report. You stop guessing.

The LangSmith comparison view

LangSmith's "comparison view" lets you run the same dataset through two agent versions and see, side by side, where the outputs differ and which steps cost or took longer. The view is essentially a diff UI for trace trees. The same diff workflow exists in Langfuse (via the experiments feature) and in Phoenix (via the comparison playground). All three are usable; LangSmith's is the most polished as of mid-2026 because it's the most opinionated about the LangChain mental model.

Five Trace-Reading Habits That Pay Off

From watching dozens of teams ramp up on tracing platforms in 2025-2026:

1. Always open the trace before opening the prompt. If an output is wrong, the trace is faster than reading the prompt. Almost always, the bug is somewhere other than where you'd first look.
2. Read top-down, not bottom-up. Start at the root span and walk down. Don't start at the failing step and walk up — you'll miss context that shapes the failure.
3. Inspect full payloads on retrieval and tool steps. Truncated views hide the document that contaminated the output. Always expand the full text.
4. Tag your traces. Tenant, agent version, user ID, model. Five minutes of tagging discipline buys you weeks of filtering power later.
5. Add a feedback widget to your UI. When users thumbs-down a response, the feedback should attach to the trace ID. This is how you find the bad traces you don't see in error filters.

When to Use Which Platform (and When to Use More Than One)

A subtlety we've watched teams discover: it's not unusual to use more than one. A team might run LangSmith on their LangChain-backed customer-facing agent (the path of least resistance) while also running Phoenix on a Python-DSPy-built retrieval-quality experiment (because Phoenix's retrieval evaluators are best-in-class). Both traces flow to their respective UIs; the team's "incident review" process tags both.

Or: a team uses Langfuse for production tracing because they self-host for compliance, and uses LangSmith's prompt-management features in a separate workspace for prompt iteration. The prompt source-of-truth lives in LangSmith; production traces live in Langfuse; the team manages the seam.

You don't have to commit to one. The three platforms are friendly enough to coexistence that mixed setups are common in 2026. The wrong move is using zero of them. The right move is picking one that fits your stack and shipping the tracing instrumentation as part of week one, not week six.

Key Takeaways

• An agent without a trace is a debugger without a stack. You can run it; you can't reason about it when it breaks. The teams that ship reliable agents in 2026 instrument tracing on week one, not week six.
• Three platforms own the agent-tracing category in May 2026. LangSmith for LangChain and LangGraph teams (zero-friction setup, native graph visualization). Langfuse for open-source and self-hosted (framework-neutral, full data ownership). Arize Phoenix for ML-grade rigor (OpenTelemetry-native, best retrieval evaluators).
• Rules of thumb: LangChain stack → LangSmith. Self-hosting or non-LangChain → Langfuse. ML-rigor team or OTel-native priority → Phoenix.
• A trace is a tree, not a log. Parent-child relationships, full payloads, latency and cost per node, with a UI that lets you click through the tree in seconds. Logs are flat; agents are recursive; trees fit agents.
• The 30-second debugging principle: most "bad output" failures look like model failures and are actually data, retrieval, or tool failures two or three steps upstream. The trace tells you which step in 30 seconds; without it, you debug at the wrong layer for an hour.
• Instrumentation is minutes, not days. LangSmith: two environment variables. Langfuse: an @observe decorator or an SDK wrapper. Phoenix: the openinference instrumentor for your framework. All three are production-ready out of the box.
• The contaminated-document story (competitor blog leaks into outbound email) was solved in 12 minutes with Langfuse. Without it, the same fix would have taken hours of cross-referencing logs.
• Filter by outcome, tag, and content. Status, latency, cost, feedback — start there. Then narrow by tenant ID, agent version, model. Then search inside payloads for specific strings. All three platforms support this in 2026.
• Trace-driven development. Every agent change runs through a fixed eval set; you compare trace trees in LangSmith's, Langfuse's, or Phoenix's comparison views; you only ship when the diff matches expectations. This is what "evaluation-driven development" looks like for agents.
• Use more than one platform if it fits. The three coexist well. The wrong move is using zero. Pick one that fits your stack and ship the instrumentation as part of week one.