Your AI Agent Just Told a Customer the Wrong Refund Policy. Now What?

A Fortune 500 retailer deploys an AI customer support agent handling 50,000 conversations daily. In the first week, the agent confidently tells 340 customers they are eligible for refunds they cannot actually receive. The cost: $180,000 in honored incorrect refunds, plus unmeasured brand damage. This failure was entirely preventable with proper evaluation. Customer support is one of the highest-stakes domains for AI evaluation because errors have immediate financial consequences, every conversation is a brand interaction, and the failure modes -- hallucinated policies, incorrect escalation, tone-deaf responses to frustrated users -- are specific and testable. This lesson gives you the evaluation framework to deploy AI support agents with confidence.

The Six Dimensions of Support Agent Evaluation

Evaluating a customer support AI requires measuring six distinct dimensions, and optimizing for one often degrades another. Resolution accuracy: did the agent provide the correct answer or take the correct action? Policy compliance: did the response align with current company policies, terms, and procedures? Tone and empathy: did the agent match the appropriate emotional register for the customer's state? Escalation judgment: did the agent correctly identify when to hand off to a human, and did it avoid unnecessary escalations? Efficiency: did the agent resolve the issue in minimal turns without sacrificing quality? Safety: did the agent avoid sharing sensitive information, making unauthorized commitments, or engaging with abusive inputs inappropriately? Build separate eval sets for each dimension. A response can be factually correct but tonally wrong, or empathetic but policy-noncompliant. Single-score evaluations hide these tradeoffs.

Evaluating Policy Grounding and Factual Accuracy

The most critical evaluation for support agents is whether they accurately represent company policies. Build a policy grounding eval set by extracting every factual claim from your knowledge base: refund windows, warranty terms, pricing tiers, feature availability, service hours. For each claim, construct test conversations where a customer asks about that specific policy. The gold-standard answer comes directly from your documentation. Measure grounding accuracy: what percentage of policy-related responses are fully consistent with source documents? Use an LLM-as-judge to check: provide the judge with the source policy document and the agent's response, and ask whether any claims in the response contradict or go beyond the source. Target 99%+ grounding accuracy for high-stakes policies (billing, legal obligations) and 95%+ for general information. Refresh this eval set every time policies change -- stale evals are worse than no evals because they provide false confidence.

Evaluating Multi-Turn Conversation Quality

Customer support conversations are multi-turn, and single-turn evaluation misses critical failure modes. Build multi-turn eval scenarios that test: information gathering (does the agent ask the right clarifying questions?), context retention (does the agent remember what the customer said three turns ago?), state tracking (does the agent correctly track the issue's resolution status?), and graceful recovery (when the customer corrects the agent, does it update its understanding?). A concrete approach: create 50 scripted conversation flows with branching paths. At each branch, define what the agent should do given the customer's input. Run the conversations programmatically using a test harness that simulates the customer side. Score each turn for correctness, and compute conversation-level success rate (percentage of conversations where every turn was acceptable). A common finding: agents that score 95% on single-turn accuracy drop to 70% on multi-turn conversations due to context window management failures.

Measuring Tone, Empathy, and Emotional Intelligence

Customer support evaluation must assess emotional competence. A factually correct response delivered in a dismissive tone to a frustrated customer is a failure. Build a tone evaluation rubric with anchored examples: Empathetic acknowledgment (1-5) -- does the agent validate the customer's frustration before problem-solving? Appropriate formality (1-5) -- does the register match the brand voice and customer's communication style? De-escalation skill (1-5) -- when a customer is angry, does the agent reduce tension or inflame it? Use LLM-as-judge for tone evaluation at scale, providing the judge with your brand's tone guidelines and specific examples of appropriate and inappropriate responses. Test specifically with adversarial emotional scenarios: irate customers, customers who have been bounced between channels, customers experiencing distress related to your product. These edge cases reveal whether your agent defaults to robotic policy recitation or demonstrates genuine emotional intelligence.

Evaluating Escalation Decisions

Escalation judgment is a binary classification problem with asymmetric costs. A false negative (failing to escalate when needed) can result in a customer churning or a compliance violation. A false positive (unnecessary escalation) wastes human agent time and increases cost-per-contact. Build an escalation eval set with clear labels: conversations that definitely require human intervention (legal threats, safety issues, complex billing disputes, emotional distress) and conversations the AI should handle autonomously (standard FAQ, simple account changes, status checks). Measure precision and recall separately. For most support operations, target escalation recall above 95% (catch almost all necessary escalations) even at the cost of lower precision (some unnecessary escalations). Compute the confusion matrix quarterly using production data: sample 500 escalated conversations and 500 non-escalated ones, have human reviewers judge whether the escalation decision was correct, and track the false-negative rate over time.

Production Monitoring: CSAT, Resolution Rate, and Containment

Pre-deployment evaluation is necessary but not sufficient. In production, track these key performance indicators. CSAT (Customer Satisfaction Score): post-conversation surveys on a 1-5 scale, benchmarked against your human agent baseline. First-contact resolution rate: percentage of issues resolved without follow-up contact within 72 hours. Containment rate: percentage of conversations handled entirely by AI without human escalation. Average handle time: total conversation duration compared to human agents. Cost per resolution: total AI system cost divided by resolved conversations. Set up anomaly detection on these metrics: if CSAT drops more than 0.3 points day-over-day, or containment rate shifts more than 5 percentage points, trigger an investigation. Sample and review conversations from anomalous periods. The most insidious production failures are gradual: a policy change that the AI's knowledge base does not reflect, causing a slow increase in incorrect responses that CSAT surveys catch weeks later.

Domain-Specific Evaluation Challenges in Support

Customer support evaluation has unique challenges that generic LLM evaluation does not address. Knowledge base currency: your eval set is only valid if it reflects current policies. Build automated checks that compare eval set answers against the live knowledge base and flag discrepancies. Channel-specific behavior: the same agent may need different evaluation criteria for chat (concise, fast), email (thorough, formal), and voice (conversational, clear). Build separate eval rubrics per channel. Personalization vs. consistency: the agent should personalize responses to the customer's history but remain consistent in policy application. Test with scenarios where personalization could lead to policy exceptions that should not be granted. Multilingual support: if you serve customers in multiple languages, your eval coverage must include each language. Do not assume that a model that scores well in English performs equivalently in Portuguese or Japanese -- build language-specific eval sets with native-speaker annotators.

A/B Testing AI Support Agents in Production

Before full deployment, A/B test your AI agent against your current support operation. Randomly assign incoming conversations to AI-handled or human-handled (or AI-assisted) groups. Run the test for at least two weeks to capture weekly patterns in inquiry volume and type. Measure the same KPIs across both groups: CSAT, resolution rate, handle time, escalation rate, and cost per resolution. Use statistical significance testing (two-sample t-test for continuous metrics, chi-squared for proportions) with p < 0.05 as the deployment threshold. Watch for Simpson's paradox: the AI may appear to perform equivalently overall but fail badly on specific issue categories. Segment your A/B results by issue type, customer tier, and complexity level. A common pattern: AI matches or beats human agents on simple, high-volume issues (password resets, order status) but underperforms significantly on complex, ambiguous cases. Use this segmentation to define your AI agent's scope.

Try This Now: Build a Policy Grounding Eval

Select one section of your company's customer support knowledge base -- refund policy, shipping policy, or account management procedures. Extract every distinct factual claim (aim for 20-30 claims). For each claim, write a customer question that would require the agent to state that fact. Write the gold-standard answer based on the policy document. Now test your AI agent with these questions and score each response: fully correct (matches policy), partially correct (some accurate information but missing key details or adding unsupported claims), or incorrect (contradicts policy). Calculate your grounding accuracy. If it is below 95%, you have identified specific policy areas where retrieval or grounding needs improvement. This exercise takes about 45 minutes and produces an immediately actionable eval set you can automate.

Key Takeaways

Customer support AI evaluation requires six distinct dimensions: resolution accuracy, policy compliance, tone and empathy, escalation judgment, efficiency, and safety. Policy grounding evaluation is the highest-priority dimension -- target 99%+ accuracy on high-stakes policies using LLM-as-judge against source documents. Multi-turn evaluation is essential because agents that score 95% on single turns often drop to 70% in full conversations. Build separate tone rubrics with adversarial emotional scenarios to test de-escalation ability. Measure escalation decisions using precision and recall separately, targeting recall above 95% for safety-critical escalations. In production, monitor CSAT, first-contact resolution rate, containment rate, and cost per resolution with anomaly detection. A/B test your AI agent segmented by issue type to define its optimal scope.