Overview

Imagine a bug report arrives at 3 AM. Your support system's AI agent reads it. It searches your codebase for similar issues. It reviews logs. It traces the call stack. It identifies the root cause. It writes a fix. It runs the test suite. Tests pass. It opens a pull request with a description and links related issues. All of this happens automatically, without anyone waking up.

That's an AI agent in action. Not a chatbot answering questions. Not a recommendation system suggesting products. An autonomous system that takes a goal, reasons about how to accomplish it, executes multi-step workflows, observes the results, handles failures, and adapts when plans don't work.

Agents represent a qualitative shift in what AI systems can do. They're not passive: they don't wait for you to tell them what to do. They're goal-directed. They reason about dependencies. They coordinate multiple tools. They recover from errors. They accomplish objectives that would take humans hours to complete.

This power comes with real risks. Agents that go wrong don't just give you bad advice. They take bad actions. They delete data, charge customers millions, send millions of emails, leak confidential information. If chatbots are dangerous, agents are dangerously autonomous. This guide covers both the power and the guardrails.

What Makes an AI Agent

A Goal or Objective

Not "answer a question." A specific goal: "Reduce production incidents for this service by 20%." Or: "Onboard a new customer and ensure they have all necessary access." Or: "Analyze this dataset and identify anomalies."

The goal is what drives everything. Without a clear goal, an agent is just a chatbot answering questions. A goal is specific: "Reduce production incidents for this service by categorizing and alerting on error spikes" not "improve reliability."

Tools and Capabilities

Tools are how agents interact with systems you care about. An incident response agent needs: fetch metrics, search alert history, access runbooks, post to incident channels, trigger auto-remediation scripts. A customer onboarding agent needs: create accounts in CRM, provision infrastructure, configure access controls, send emails.

The agent's power is directly proportional to its tools. Give it read-only tools, it can analyze. Give it write tools, it can execute. Give it delete tools, it can cause disasters. Choose tools carefully. They define what the agent can and cannot do.

Reasoning and Planning

The agent examines the goal and reasons backward about steps. "To fix this production incident, I must: 1) identify the root cause via logs, 2) find the responsible service, 3) locate the recent commit that broke it, 4) revert or apply a fix, 5) verify with metrics, 6) notify the on-call engineer." This plan is not hardcoded; the agent generates it based on the current situation.

Critical difference from scripts: the agent adapts. If log analysis shows a different root cause, the plan changes. If a revert fails, the agent tries a fix instead. The agent is solving the problem, not following a flowchart.

Monitoring and Feedback

After each action, the agent observes results. "I reverted the commit. Now let me check if metrics are improving." It monitors progress toward the goal. If progress stalls or fails, it adjusts strategy. This feedback loop is what makes agents powerful and what makes them risky. They can adapt to unexpected situations, but they can also adapt in dangerous directions.

The Autonomy-Safety Tradeoff: Agents that are fully autonomous are fast and efficient but risky. Agents with human approval on every action are safe but slow and don't save time. The goal is to find the sweet spot: automate 70% of routine cases fully, require human review for 20% of uncertain cases, and escalate 10% of complex cases immediately. This maximizes both speed and safety.

Agent Architectures and Reasoning Patterns

Different agent architectures make different tradeoffs between reasoning depth, computational cost, and reliability. Understanding these patterns helps you choose the right approach for your problem.

ReAct (Reasoning + Acting)

The agent alternates between thinking and doing. "I need to understand this bug. Let me search the codebase." (Action: search) "I found three similar failures in this file." (Reasoning: pattern identified) "Let me examine the specific line that changed." (Action: read file) This loop continues until the goal is achieved. ReAct makes agents more interpretable because you can trace their reasoning. It also makes them more reliable: they're forced to verify assumptions rather than hallucinate.

The downside: ReAct requires multiple LLM calls, which adds latency and cost. For simple tasks, it's overkill.

Plan-and-Execute

The agent builds a complete plan upfront, then executes it step by step. First: "To migrate this database, I need to: 1) backup current data, 2) run migrations, 3) verify schema, 4) run smoke tests, 5) rollback if needed." Then it executes each step in order. If a step fails, it can either retry, adjust the plan, or escalate.

Plan-and-Execute works well for workflows with clear dependencies. It reduces the number of LLM calls (better for cost and latency). The downside: if the plan is wrong, the agent is committed to a bad path. Less adaptive than ReAct.

Tree of Thoughts

The agent explores multiple possible paths simultaneously, evaluating which looks most promising. When solving a complex problem, instead of picking one action and hoping it works, it reasons: "I could approach this via path A, path B, or path C. Path A is fast but risky. Path B is thorough but slow. Path C is moderate on both dimensions." It then explores the most promising paths, pruning branches that look bad.

Tree of Thoughts is powerful for complex reasoning but computationally expensive. Use it when the cost of a wrong answer is high.

Multi-Agent Collaboration

For complex objectives, break the problem into domains and assign agents to each domain. A content creation workflow might have: a research agent (finds sources), a draft agent (writes content), a fact-check agent (verifies claims), and an editor agent (polishes prose). These agents coordinate, passing work between each other. One agent might say: "I found these sources, but I need interpretation. Research agent, can you summarize their findings?" The research agent responds, and the workflow continues.

Multi-agent systems handle specialization well and can be more robust (if one agent fails, others can compensate). They're also complex to orchestrate and debug. Use when you have truly distinct domains that benefit from specialized reasoning.

Framework Comparison: Real Tradeoffs

Choosing a framework isn't just about popularity. Each has real tradeoffs that affect your ability to debug, scale, and integrate with existing systems.

LangChain: Flexibility vs. Complexity

LangChain is the most established agent framework. It gives you tools for ReAct patterns, memory management, and orchestration. The advantage: it's battle-tested. The disadvantage: it's a layer on top of language models, which means you're learning LangChain's abstractions in addition to learning agent design.

Use LangChain when: you need multi-step agent workflows, you're already using it for other tasks, you value the ecosystem of integrations. Skip it when: you need direct control over model interactions or when your requirements are simple enough that the abstraction overhead isn't worth it.

Anthropic Tool Use: Direct and Debuggable

Claude has native tool-use capability. You define tools in the system prompt. Claude decides when to use them. This is the simplest approach: you don't need a framework. You're directly using the model's built-in reasoning.

Advantages: direct control, easy to debug (you can see exactly what the model is reasoning), no framework overhead, works with any Claude model. Disadvantages: you have to implement agent logic yourself (planning, memory, tool organization). Better for simpler agents; less suitable for complex multi-agent orchestration.

OpenAI Assistants: Fully Managed State

OpenAI's Assistants API manages agent state for you. The service persists memory, handles tool calling, and manages long-running agent threads. This is useful if you want someone else managing infrastructure.

Disadvantages: vendor lock-in, less visibility into agent reasoning (harder to debug), can't run locally or on your infrastructure, token limits affect large tool definitions.

LlamaIndex: For Structured Knowledge Workflows

LlamaIndex specializes in agents that work with structured data, documents, and knowledge bases. If your agent needs to reason over large collections of documents or structured databases, LlamaIndex provides query engines and retrieval patterns.

Use when: your agent's primary role is understanding and retrieving from domain-specific knowledge. Skip when: you need general-purpose agent orchestration.

CrewAI: Role-Based Multi-Agent Teams

CrewAI abstracts agents as roles in a team. You define a "research agent," a "writing agent," a "fact-check agent", each with a description, tools, and goals. The framework orchestrates them. This is elegant for human-centric workflows.

Advantages: high-level abstractions make agent definitions readable. Disadvantages: less control over coordination, not ideal for technical system tasks, more opinionated about workflow structure.

Recommendation: Start with Anthropic Tool Use, graduate to LangChain

For your first agent, use Claude's native tool use. It's simple, debuggable, and teaches you how agents actually work. Once you need multi-agent coordination or complex memory management, move to LangChain. Avoid OpenAI Assistants and CrewAI unless those specific features (vendor management or role-based teams) are central to your problem.

Why Agents Fail: The Reliability Problem

Studies on agent reliability reveal a hard truth: agents fail on 20-40% of moderately complex tasks. Not catastrophically. They don't delete your database. But they get stuck, take wrong paths, hallucinate tool usage, or refuse to proceed when they should. Understanding why helps you design better agents.

The Hallucination Problem

An agent reports: "I checked the logs and found error code 42 at timestamp 14:32:15." But it never actually called the logging tool. It inferred what the logs might contain based on training data. This is hallucination: the model confidently making things up.

Why it happens: language models are trained to be helpful and complete-looking answers. Saying "I don't know" feels wrong to them. When an agent is under pressure to accomplish a goal, this tendency intensifies.

How to prevent it: force tool use. Tell the agent "You must use a tool before making any factual claim." Log every tool call. Verify tool results before acting. Require agents to cite which tool provided information before using it. This overhead is worth it, hallucination in autonomous systems is dangerous.

Looping and Dead-End States

An agent tries to fix an issue. It calls a tool. The tool fails with an obscure error. The agent, unsure what to do, tries the same tool again. Same error. It tries a third time. Fourth time. It's stuck in a loop, burning tokens and getting nowhere.

Why it happens: agents don't have good introspection. They can't easily recognize when they're repeating themselves or when they've hit an unsolvable problem.

How to prevent it: implement loop detection. Track the last 5 actions. If the same action appears twice, the agent must try something different or escalate. Set hard attempt limits per task (max 5 attempts total). Maintain a "graveyard" of failed approaches so the agent knows not to retry them.

Poor Tool Selection

You've given the agent 15 tools. The agent needs to categorize a customer complaint. Instead of using the classification tool, it tries to write SQL to query the database, which fails, which it then tries to debug using the logging tool. It's using the wrong tools in the wrong order.

Why it happens: agents don't have semantic understanding of tools. They're pattern-matching on names and descriptions. If tool descriptions are vague, the agent guesses wrong.

How to prevent it: make tool descriptions extremely specific. Not "classify_input" but "use this tool to classify customer complaints into: billing, technical_issue, feature_request, or other. Provide the raw complaint text. This tool only works on customer complaints, do not use it for other classification tasks." Include examples. Group related tools conceptually and describe their relationships.

The Cost Spiral

An agent is working on a task. It's using expensive tools. It makes a mistake, retries, makes another mistake, retries again. After 50 tool calls, you've spent $200. The agent is still stuck.

Why it happens: agents have no cost awareness. They don't know that calling the search tool 100 times to find a single result is expensive.

How to prevent it: track cost per agent execution. Set hard limits: "If this agent spends more than $1, shut it down." Use cheaper models for initial exploration, switch to expensive models only for final reasoning. Batch tool calls when possible. Cache results from expensive tools (if you searched for "database migration patterns" once, don't search again in the same session).

Production Hardening Checklist: Before an agent touches production: 1) Are goals absolutely clear? 2) Does the agent have hard constraints (not suggestions, hard rules)? 3) Is every action logged with timestamp and justification? 4) Can you kill it in under 10 seconds? 5) Is there human approval for any action affecting data or customers? 6) Do you have cost limits and alerts? 7) Have you tested failure modes (API down, tool returns garbage, timeout)? 8) Is there a monitoring dashboard showing success rate, cost, execution time, and error patterns?

Building Your First Agent: A Practical Example

Choose a Real Problem

Start with a problem that's currently solved manually and eats a lot of time. Not "analyze customer feedback" (vague). "Triage incoming support tickets into: billing, technical, feature request, other. Assign to correct team. Escalate if complexity is high." This is specific, measurable, and scoped.

Define the Agent's Scope Tightly

Your agent will have limitations. Be explicit about them. "This agent can: read tickets, search knowledge base, assign to teams. This agent cannot: modify tickets, contact customers directly, make refunds." This clarity prevents the agent from trying to do things you didn't authorize.

List the Tools

What does the agent need to access to accomplish the goal?

• read_ticket(ticket_id) - fetch ticket content and metadata
  - search_knowledge_base(query) - find relevant docs and past tickets
  - classify_issue(description) - predict category (billing, technical, etc.)
  - get_team_info(category) - find which team handles this category
  - assign_ticket(ticket_id, team_id) - update ticket assignment
  - escalate_ticket(ticket_id, reason) - mark for human review
  - log_action(ticket_id, action, result) - record what the agent did

Write Tool Definitions Carefully

Each tool description must be precise. Bad: "search_knowledge_base(query) - Searches the knowledge base." Good: "search_knowledge_base(query: str, limit: int = 5) - Searches the knowledge base for documents matching the query. Returns up to 'limit' results, each with title, relevance score, and a 100-word excerpt. Use this to find relevant documentation and past ticket solutions when classifying a new issue. Only searches English documents."

Specify Behavior Rules

Tell the agent how to think:

You are a support ticket triage agent. Your goal is to automatically
categorize and route tickets to the correct team.

BEHAVIOR:
- Always read the entire ticket before classifying
- If the ticket mentions multiple issues, focus on the primary issue
- If you're unsure about classification (confidence

Test on Real Historical Data

Don't launch the agent on tomorrow's tickets. Test it on the past 100 tickets your support team already triaged. For each historical ticket, run the agent and check: did it assign to the same team a human did? If not, was it wrong or just a different valid choice? Track accuracy. If accuracy is below 85%, improve the prompt or tools before deploying.

Deploy With Human Review

Don't trust the agent immediately. For the first week, every assignment goes to a human for review before the customer sees it. Track: how many does the human override? Track which types the agent gets wrong. Use this feedback to improve the prompt.

Real Examples: What Works and What Doesn't

Case Study: Klarna's AI Customer Service (Success)

Klarna, the payment company, deployed an AI agent to handle customer service. The agent accesses Klarna's customer data, payment history, dispute records, and knowledge base. When a customer contacts support, the agent reads their inquiry, searches the knowledge base, pulls relevant account info, and generates a response. For simple issues (refund status, payment confirmation, general questions), the agent responds directly. For complex issues (disputes, fraud concerns, unusual situations), it escalates to a human agent.

The result: the AI agent now handles approximately two-thirds of Klarna's customer support conversations. It's been deployed in production for over a year. Why it works: clear scoping (only simple inquiries), fallback to humans for complexity, continuous monitoring and retraining. The agent isn't trying to be perfect; it's trying to handle the cases it's good at and know when to pass to humans.

Case Study: The Chevy Dealer Chatbot (Failure)

A Chevrolet dealership deployed a chatbot to handle customer inquiries. The system took customer requests and attempted to fulfill them. A customer asked: "What's your best deal on a Corvette?" The chatbot responded: "I can sell you a new Corvette for $1. Would you like to proceed?" The customer, thinking this was genuine, agreed. Now the dealership had to either honor the offer or deal with a very angry customer.

What went wrong: no safety constraints. The agent had access to pricing and order systems but no guardrails preventing it from offering illegal deals. No escalation for unusual requests. No human review for high-value transactions. The cost to the dealership in reputation and legal exposure was high.

Lesson: agents that interact with real customers need hard constraints. Not "try to make good deals", "never offer a price more than 50% below MSRP," "require human approval for any deal below $5,000 margin," "if a customer requests something outside normal parameters, escalate immediately."

Case Study: Internal Bug Triage (Success)

A software company built an agent to triage bug reports. When a bug is filed, the agent: reads the description, searches issue history for duplicates, checks if the reported behavior exists in recent commits, assigns a preliminary severity level, and notifies the appropriate team. The agent doesn't fix bugs; it organizes them.

Why it works: narrow scope, no irreversible actions, human reviews the agent's work before it affects anything. The triage agent saves humans 10 minutes per bug, which adds up across hundreds of daily reports. It's fast enough to triage in real-time, so issues get routed immediately.

Evaluating Agent Performance

Accuracy and Correctness

Does the agent accomplish its goal correctly? For a ticket classifier: what percentage of classifications match human judgment? For a bug fixer: what percentage of proposed fixes actually pass tests and don't introduce new bugs? Accuracy should be your primary metric. If an agent is only 60% accurate, it's making work for humans rather than saving it.

Reliability

How often does the agent fail completely (crash, get stuck, timeout)? A 99% completion rate is table stakes. 99.9% is good. Below 95% and you shouldn't deploy to production. Track: completed vs. failed runs, average time to failure, types of failures (hallucination, looping, timeout, tool error).

Cost and Efficiency

How much does the agent cost per execution? If your agent costs $0.50 per ticket and humans cost $2.00 per ticket, great. If your agent costs $5.00 per ticket, it's not worth deploying. Track: tokens used, tool calls made, cost per execution. Compare to human cost.

Speed

Is the agent faster than a human? A human takes 5 minutes per ticket; your agent takes 10 seconds. The agent is winning. But if your agent takes 2 minutes (due to multiple LLM calls and tool latency), the advantage is marginal. Track: end-to-end execution time, time per tool call, latency breakdown.

Failure Modes

Understand how your agent fails. Does it fail gracefully (escalates and logs the issue) or catastrophically (takes a bad action)? Does it fail consistently on certain types of inputs (tickets with multiple issues, non-English text, ambiguous cases)? Use failure analysis to improve the system.

Human Override Rate

In production, how often do humans override the agent's decision? If humans override 40% of decisions, the agent isn't saving time. If they override 5%, great. High override rates indicate: agent doesn't understand the domain, scope is too broad, or tool descriptions are confusing.

Human-in-the-Loop Patterns That Work

The Approval Pattern

The agent proposes an action. A human approves or rejects it before execution. Used for: high-stakes actions (deleting data, charging customers, sending communications), actions that are rare enough that approval doesn't create a bottleneck (happens 10 times per day, not 10,000 times). Implementation: agent builds the action, presents it to a queue, waits for human decision, executes only if approved.

The Review and Veto Pattern

The agent executes the action immediately, but a human can veto it within a time window (e.g., within 2 hours). Used for: actions that benefit from speed (customer support responses should go out quickly) but are reversible. Implementation: agent acts immediately, logs to a review queue, humans can still revert within 2 hours. After 2 hours, changes are locked.

The Escalation Pattern

The agent handles routine cases automatically but escalates to humans when it hits complexity. Used for: tasks with a clear "easy" and "hard" version. Implementation: agent attempts to handle, but if a decision seems uncertain or requires judgment, it escalates. Humans only review the escalated cases.

The Audit Pattern

The agent works fully autonomously, but humans audit the work after the fact. Used for: tasks where errors are low-impact and can be fixed retroactively. Implementation: agent runs fully unsupervised, logs all actions, humans periodically sample and review recent agent work (every 100th action, or all actions from the past week). If problems are found, revert recent changes and retrain the agent.

Best Practice: Combine Patterns

For most production agents, use a hybrid: automation for routine cases (Escalation pattern), immediate action with human review/veto for common cases (Review and Veto), and full approval for high-stakes cases (Approval pattern). This maximizes efficiency (most common cases proceed fast) while protecting against catastrophic failure.

Adversarial FAQ: Hard Questions About Agent Reliability

Q: How do I know my agent isn't hallucinating?

A: You can't know with 100% certainty. But you can make it very hard. Require the agent to use tools before making any factual claim. Log every tool call. Verify tool outputs are accurate. Create a tool that fact-checks the agent: before it acts, it must cite which tool provided the information. Make hallucination detectable even if you can't prevent it.

Q: What if the agent encounters an error from a tool and retries forever?

A: Set hard attempt limits. After 3 failed attempts at the same action, the agent must either try something different or escalate. Implement cycle detection: if the same tool call fails twice, ban it from trying that tool again in this session. The agent should learn from failures, not repeat them.

Q: How do I know when an agent is actually making a mistake vs. when it's just doing something I didn't expect?

A: Define correctness upfront. For a ticket classifier, correctness is: assigns to the same team a human would assign to (or a valid alternative). For a bug fixer, correctness is: proposed fix is correct, passes all tests, doesn't introduce new bugs. Without a clear definition, you can't evaluate performance, and you shouldn't deploy.

Q: My agent is making good decisions 90% of the time, but the 10% of failures are catastrophic. Is it production-ready?

A: No. High average performance doesn't matter if failures are catastrophic. You need to: 1) reduce failure modes to 90%), and low override rate (<10%). If accuracy is 90% but override rate is 40%, the agent has a trust problem, humans don't believe in it even when it's right.

Q: What if I deploy an agent and it starts making unexpected mistakes?

A: You should have a kill switch. Within 10 seconds of realizing something is wrong, you can completely stop the agent. No "drain existing requests", immediate termination. Then: 1) assess the damage, 2) understand what went wrong, 3) fix the issue, 4) test extensively, 5) redeploy cautiously. You should also have continuous monitoring that alerts you automatically before users notice problems.

What to Do Monday Morning

• Identify a problem that's currently handled manually: incident triage, support ticket routing, bug classification, expense report categorization. Something that happens frequently and follows repeatable patterns.
  - Write down exactly what a human does to solve it. Not "analyze the ticket." Specific: "read title and description, search knowledge base for similar issues, determine if issue is billing-related or technical, assign to correct team." Explicit decisions matter.
  - List the tools the agent needs: what data must it read? What systems must it access? What can it modify?
  - Choose Claude's native tool use if this is your first agent. It's simpler than frameworks and teaches you how agents actually work. Graduate to LangChain only if you need multi-agent coordination.
  - Build the minimal working version: implement the tools, write clear tool descriptions, define agent constraints in the system prompt. Don't overengineer.
  - Test on historical data. Run the agent on 100 past tickets and compare its decisions to human decisions. Track accuracy. If below 85%, identify why and improve before deploying.
  - Deploy with human review for one week. Every agent decision goes to a human first. Track override rate. Use this data to improve the prompt.
  - Set up monitoring immediately: log every action, track cost, track success rate, alert if anything looks wrong. Monitoring should happen before the agent touches production.

A Second Callout on Agent Failure Modes: Agents fail in specific ways. Common failures: hallucination (inventing data), looping (repeating failed actions), poor tool selection (using the wrong tool for the job), cost explosion (making thousands of expensive API calls), and unsafe actions (deleting data without authorization). Understanding these failure modes and building guardrails for each one is what separates agents that work from agents that cause damage.

AI agents are systems that accomplish multi-step goals autonomously: reading state, reasoning about actions, executing workflows, observing results, adapting when things fail. They're fundamentally different from chatbots. They require clear goals, explicit tools, hard safety constraints, comprehensive logging, and human oversight for high-stakes actions. Start small: pick one specific problem, build a minimal agent, test thoroughly, deploy with human review, monitor obsessively. Agents are powerful. They can save hours of human labor. They're also risky. They can cause damage if deployed carelessly. The difference is in the design and safety thinking you put in upfront.

On This Page

Watch the Lecture
What Makes an Agent
Agent Architectures
Framework Comparison
Why Agents Fail
Real Case Studies
Building Your First Agent
Evaluating Performance
Human-in-the-Loop
Adversarial FAQ

Chapter Details

Part ofChapter 6