โ†
AI Agent Builders & Citizen Developers
Aware ยท M17 ยท lesson 17 of 19 ยท queued
Preview โ€” browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll โ†’
When Not to Build an Agent
๐Ÿ“–
now learning

When Not to Build an Agent

15 min

The most valuable governance artifact your team can produce in 2026 is a written list of conditions under which you will not build an agent. Not "should not." Not "carefully consider." Will not. A kill-criteria list โ€” pinned to the wall, signed by leadership, applied without exception โ€” is the single document that has saved more organizations from regulatory and operational disasters than any framework, vendor, or model. In this lesson we will build the list together. We will cover the five conditions that should trigger an automatic "no" before any feasibility scoping starts: ambiguous SLA, missing source-of-truth, no rollback, regulated employment-or-credit-or-similar decision without legal sign-off, no possible eval set. We will walk through real cases where the right answer was no โ€” and where saying no saved a healthcare system $4M, a HR-tech vendor a Colorado AI Act class action, and a Series B fintech its banking partner. The work of an Aware Operator is not just to ship good agents. It is to refuse to ship bad ones. This lesson teaches the refusal.

Why the Kill-Criteria List Exists

Most teams I work with already know, intuitively, that some agent ideas are bad. The product manager who insists the agent should make credit decisions "to reduce friction." The executive who wants the agent to "handle everything in HR." The vendor that promises their AI can "replace your customer service team." Operators sense the problem. They struggle to articulate the refusal in a way that doesn't sound like "we're being too cautious."

The kill-criteria list solves this by moving the refusal from individual judgment (which can be questioned, lobbied, overridden) to organizational policy (which has been adopted, signed, and applies uniformly). When the next "but can't we just..." conversation starts, the answer is not "I personally think no." The answer is "we have a written policy that this triggers kill criteria; here's the list; we'd need to amend the policy to proceed." The conversation moves from emotion to procedure.

This is not anti-velocity. It is anti-incident. The cost of saying no to one bad agent idea in a planning meeting is a missed sprint. The cost of saying yes is โ€” well, look at the incidents in this chapter. The kill-criteria list converts the cost asymmetry into a procedural one. That's the magic.

An organization that cannot say no to bad agent ideas will eventually be unable to say yes to good ones โ€” because the cost of cleaning up incidents will consume the velocity the good agents were supposed to deliver.

The Five Kill Criteria

The list is short. Five conditions. Each one, present, triggers automatic refusal pending policy override. Memorize them.

Criterion 1: ambiguous SLA

The agent's job has no clear, measurable, agreed-upon definition of success. "The agent should help with customer support" is not an SLA. "The agent should reduce first-response time to under 4 minutes for 95% of tier-1 tickets with a customer satisfaction score of 4.2+ on a 5-point scale" is an SLA. If the team building the agent cannot articulate the latter, the agent's success is unmeasurable, its failures are undefined, and its eventual incident is unattributable.

Why this kills: an agent whose success cannot be measured cannot be improved, audited, or defended. Under EU AI Act Article 26 monitoring obligations, "we couldn't measure if it was working" is not a defense. Under ISO/IEC 42001 8.3, "establish criteria for the processes" is a control requirement. The SLA is the criterion.

Criterion 2: missing source-of-truth

The agent needs to answer questions or take actions for which no authoritative source exists in the organization. Examples: "what's our return policy?" โ€” but the actual policy is different across regions and the published version is stale. "Who owns this account?" โ€” but Salesforce's owner field hasn't been kept current in years. "What's our SLA for enterprise?" โ€” but it depends on the contract version and no central repository tracks contract versions.

Why this kills: an agent without source-of-truth either fabricates answers (the made-up policies failure from Lesson 1.4.2) or escalates everything (which defeats the purpose). The fix is not to build the agent. The fix is to build the source-of-truth first, then consider the agent. Most organizations skip the source-of-truth work because it's unglamorous; this is exactly why it must be in the kill criteria. The agent project sponsors the source-of-truth work, or it doesn't proceed.

Criterion 3: no rollback

The agent will take actions that cannot be undone within the org's operational window. The customer service agent that sends emails: the email is sent; recall is at best partial; the brand consequence is not rollback-able. The financial agent that posts irrevocable transactions to an external system. The supply chain agent that issues orders to vendors. The legal agent that files documents with regulators.

Why this kills: irreversible actions amplify confident-wrong incidents to permanent consequences. The blast radius (Lesson 1.4.1) is not just "what happens in the next hour" โ€” it's "what stays happened forever." The fix is not to ship the agent without rollback. The fix is to either add rollback (delayed-send queues, transactional commits with hold windows, staged confirmation) or to not build the agent.

Criterion 4: regulated employment, credit, or analogous decision without legal sign-off

The agent makes decisions that affect human rights, employment, credit, education, healthcare access, public benefits, or other Annex III high-risk categories โ€” and you do not have explicit legal sign-off on the deployment, the FRIA, and the regulatory posture. This is broader than the EU AI Act: U.S. state laws (Colorado AI Act, NYC bias-audit law, California successor legislation), federal sector laws (EEOC, ECOA, HUD, HIPAA), and international equivalents (UK, Singapore, Brazil) all apply.

Why this kills: Annex III deployments carry FRIA, oversight, and disclosure obligations that operators are not authorized to interpret unilaterally. Legal sign-off is not a formality; it's a regulatory requirement and a personal-liability question for the deployer. Operators who deploy Annex III agents without explicit legal sign-off are exposed personally and organizationally. The kill criterion exists because operators sometimes assume the legal review will happen "eventually." It must happen before, not eventually.

Criterion 5: no possible eval set

You cannot construct a representative evaluation set of inputs with known correct outputs against which to measure the agent's performance. This happens when the problem is too subjective ("write an inspiring email"), too unique-per-instance ("decide what to do with this novel customer escalation"), or too lacking in ground truth ("predict which sales lead will convert without historical data").

Why this kills: without an eval set, you cannot tell if the agent is working. You cannot detect regression when the model changes. You cannot run A/B tests against alternatives. You cannot satisfy Article 26 monitoring or ISO 42001 measurement criteria. "We trust the model" is not a substitute for measurement. If the problem genuinely cannot be evaluated, the problem is not an agent fit. Find the parts of the problem that can be evaluated, build agents for those, and keep humans on the rest.

Real Cases Where "No" Was the Right Answer

Let's make this concrete with three cases where saying no โ€” explicitly, via the kill criteria โ€” saved the organization. I've changed identifying details; the cases are otherwise as they happened.

Case 1: the healthcare scheduling agent that wasn't

A Boston-area healthcare system in late 2025 wanted to deploy an agent to handle patient scheduling. The pitch was attractive: front-desk staff were overwhelmed, no-show rates were high, and a 24/7 agent could fill gaps in coverage. The team had selected a major no-code platform, had a sponsor at the COO level, and had begun implementation. Three months in, an operator on the team (who happened to have read an earlier version of this lesson) ran the kill criteria.

The result: missing source-of-truth (provider availability was stored across three EMR systems with frequent inconsistencies and a manual override workflow that wasn't tracked), no rollback (sent appointment confirmations couldn't be unsent; misscheduled patients showed up to the wrong location), and regulated decision without legal sign-off (HIPAA implications of an AI handling PHI hadn't been formally reviewed; OCR guidance on AI in healthcare was still evolving). Three of five criteria failed.

The team brought the analysis to the COO. Initial response: "we've already invested three months." The operator's response: "the alternative is shipping an agent that schedules people incorrectly, can't unschedule them, exposes PHI, and we can't defend in a HIPAA audit. The investment to date is sunk; the future investment is unbounded." After a week of escalation, the project was paused. The team pivoted to a smaller scope: an agent that suggested appointment times to front-desk staff (decision support) instead of scheduling them autonomously. The smaller scope passed the kill criteria. Six months later, the smaller scope was operationally successful.

Estimated value of saying no: the health system later learned that a similar autonomous-scheduling deployment at another organization led to $4M in remediation, a state attorney general investigation, and a six-month operational moratorium. The kill criteria prevented a parallel incident.

Case 2: the HR-tech vendor and the Colorado AI Act

An HR-tech vendor in early 2026 was preparing to ship an "AI candidate scoring" feature to their ATS product. The feature would rank candidates against job descriptions, with a score from 0-100. The product team was eager; the GTM team was eager; the engineering team was capable. A general counsel new to the company ran the kill criteria.

The result: regulated employment decision without legal sign-off (Annex III employment under EU AI Act; Colorado AI Act consumer-facing decision; NYC bias-audit law applicability uncertain), ambiguous SLA (what does "rank candidates accurately" mean? against what ground truth?), and no possible eval set without protected-class proxies (constructing the eval set required attributes that could leak into protected-class proxies, creating the very bias problem the feature was trying to avoid).

The kill criteria triggered. The general counsel and product team negotiated a redesign: the feature would surface candidate profile summaries (not scores) and would not rank. Recruiters would still see candidates in their preferred order, with summaries to help review. Eval sets could be built for summary quality (factual accuracy, completeness) without protected-class proxies. The feature shipped. It was less ambitious than the original; it was also legally defensible.

Estimated value of saying no: a competitor's similar ranking feature was named in a class action under the Colorado AI Act in mid-2026. Estimated litigation cost: $3-7M. The kill criteria positioned this vendor to compete in the market by offering a different, defensible product.

Case 3: the Series B fintech that almost lost its banking partner

A San Francisco Series B fintech in early 2026 wanted to deploy an agent to handle customer "balance inquiries and small adjustments." The agent would have read/write access to the customer's transaction ledger and would handle requests up to \$500 without human review. The team had a working prototype and a launch date. A risk-and-compliance manager ran the kill criteria.

The result: no rollback (ledger adjustments fed downstream to a banking partner whose API didn't support rollback within the standard 24-hour window) and ambiguous SLA (what's a "small adjustment"? what's an acceptable false-positive rate for adjustments the customer didn't actually request? what's the customer-confirmation requirement?).

Two of five criteria failed. The compliance manager escalated to the banking partner relationship team, who immediately recognized the issue: the banking partner's compliance team had a specific stance on autonomous AI adjustments to customer ledgers, which was effectively "don't." Shipping the agent as designed would have triggered a banking partner review and potentially the suspension of the partnership.

The team redesigned: the agent would propose adjustments and present them to the customer for explicit confirmation before any ledger write. Adjustments above \$200 also required compliance team review with an SLA of 4 business hours. The banking partner was consulted in advance and signed off on the redesign. The agent shipped. The banking partnership remained intact.

Estimated value of saying no: the banking partner is the fintech's only path to operations; losing the partnership would have been an existential event. The kill criteria caught the issue before it reached the banking partner's compliance team in an uncontrolled way.

The Soft Criteria: Yellow Flags Worth Considering

Beyond the five hard criteria, there are soft criteria โ€” yellow flags that don't automatically kill the project but should trigger deeper scoping. Worth listing because they catch issues the hard criteria miss.

Yellow flag 1: the workflow has long-tail edge cases

The agent's job is mostly straightforward, but the edge cases are operationally critical and not well-understood. Customer support agents handling 80% routine questions and 20% complex situations are the canonical example. The 20% is where incidents live. Yellow flag: scope the agent to the 80% and route the 20% to humans, explicitly.

Yellow flag 2: the value depends on speed, not accuracy

The agent is being deployed to be fast rather than to be correct. Speed without an accuracy floor means the agent's failures are loud. Yellow flag: define the accuracy floor first; speed is a secondary metric.

Yellow flag 3: the agent is replacing rather than augmenting

The agent is positioned as a replacement for human work rather than an augmentation. Replacement framing tends to remove the humans before the agent is ready to operate without them. Yellow flag: ship in augmentation mode first; replacement is a year-later decision once the agent has proven itself.

Yellow flag 4: the org has no recent agent operating experience

The team building the agent has not operated an agent in production before. There is a gap between knowing how to build and knowing how to operate. Yellow flag: pair with someone who has operated, or start with a low-blast-radius agent to build that muscle first.

Yellow flag 5: the platform is novel

The agent is being built on a platform the team has not used before, in a stack the team has not used before, with a model the team has not used before. Stack novelty multiplies incident risk. Yellow flag: stabilize the stack before adding agent complexity, or work with a partner who has done this stack before.

The Policy-Override Process

The kill criteria list works because it's hard to override. But not impossible โ€” there are legitimate cases where the right answer is to proceed despite a criterion failure. The override process matters as much as the criteria themselves.

The recommended structure:

  1. Documented rationale: the team requesting override must write a one-page document describing which criterion is failing, why it's failing in this case, what the residual risk is, and what mitigations are in place.
  2. Cross-functional review: the override request goes to a review group (typically engineering, ops, legal, risk, and a senior product or business sponsor). Each reviewer signs off explicitly or denies.
  3. Time-bound: overrides expire. A typical timing is 90 days after which the agent must either resolve the original criterion failure or re-apply for override.
  4. Logged: every override is logged and visible in the agent governance inventory. Patterns of overrides on the same criterion are signal that the criterion may need refinement.

The process is deliberately friction-ful. The goal is not to prevent all overrides; it is to ensure overrides are explicit, considered, documented decisions rather than implicit drift.

"When Not To Build" vs. "When To Defer"

A subtle but important distinction. The kill criteria don't always mean "never build this agent." Sometimes they mean "not this agent at this time." The difference matters because operators who hear "no" as final can become resistant; operators who hear "not yet" understand the path forward.

Reframing the five criteria as deferral conditions:

  • Ambiguous SLA โ†’ defer until SLA is defined and agreed.
  • Missing source-of-truth โ†’ defer until the source-of-truth project is funded and delivered.
  • No rollback โ†’ defer until rollback mechanism (delayed-send, transactional commit, staged confirmation) is designed in.
  • Regulated decision without legal sign-off โ†’ defer until legal review is complete and FRIA is on file.
  • No possible eval set โ†’ defer until the problem is scoped to a part that can be evaluated; build for that part now.

Framing the kill criteria as deferral conditions converts "we're not building this" into "we're building this on a longer path with the prerequisite work funded." The agent often does ship โ€” just six months later, with the foundation that makes it succeed.

The Cultural Implication: Saying No Is Leadership

One final reflection. The willingness to say no to bad agent ideas is, in 2026, the most important cultural attribute of high-performing AI deployment organizations. The reason is straightforward: AI is a force multiplier on both good and bad decisions. An organization that ships agents indiscriminately multiplies its mistakes faster than it multiplies its wins. An organization that ships agents discriminately compounds. The discipline of refusal is the discipline of compounding.

This is uncomfortable. Saying no carries career risk: the executive who sponsored the agent project may not appreciate being told their idea fails the kill criteria. The team that invested three months of work doesn't want to hear that the work was misdirected. The vendor that sold the platform doesn't benefit from your discipline. These are real costs.

The kill criteria list shifts these conversations from personal to procedural. The operator is not saying "I think this is a bad idea." The operator is saying "the policy we've all agreed to flags this idea โ€” here's the procedure to override if you believe override is warranted." That phrasing is what makes the discipline survivable. Without it, individual operators get worn down and the org learns to ship bad ideas.

The most valuable artifact in a 2026 AI program is not the agents that shipped. It is the list of agents that didn't. The discipline of refusal is the discipline of compounding. The kill criteria list is the document that makes the discipline organizational rather than personal.

Putting the List Into Practice

To close, here's the operational rollout I recommend:

  1. Draft the list. Use the five criteria from this lesson as a starting point. Adapt to your org's specific context โ€” sector, regulatory exposure, scale.
  2. Get executive sign-off. The list must be adopted at a level above any individual agent project sponsor. CEO, COO, or board level for most orgs.
  3. Train the team. Every person who proposes, builds, or operates an agent should know the list and the override process. 15-minute training session, mandatory.
  4. Apply to new proposals. Every new agent proposal runs through the kill criteria before scoping work begins. Failures trigger the override process or reshape the proposal.
  5. Apply to existing agents. Run the criteria against existing production agents quarterly. Failures trigger remediation or shutdown.
  6. Track and refine. Log every criterion application, every override, every shutdown. After a year, refine the list based on real-world results.

That's the work. It is the most important work this chapter teaches. The other lessons โ€” blast radius, confident-wrong, Kiteworks, governance โ€” are all valuable, but they presume you're shipping the right agents. The kill criteria list ensures you're shipping the right agents.

Now go write your list.

Key Takeaways

  • A kill-criteria list โ€” written, signed by leadership, applied without exception โ€” is the single most valuable governance artifact your team can produce in 2026. It moves refusal from individual judgment to organizational policy.
  • The five hard criteria: ambiguous SLA, missing source-of-truth, no rollback, regulated employment/credit/analogous decision without legal sign-off, no possible eval set. Each, alone, triggers automatic refusal pending override.
  • Real cases where saying no was the right answer: the healthcare scheduling agent that became decision-support ($4M+ incident avoided), the HR-tech vendor that shifted from ranking to summarizing (Colorado AI Act class action avoided), the fintech that added customer confirmation (banking partner relationship saved).
  • Five soft criteria (yellow flags) worth considering: long-tail edge cases, speed-over-accuracy framing, replacement-vs-augmentation positioning, no recent agent operating experience, novel stack.
  • The policy-override process: documented rationale, cross-functional review, time-bound (90-day default), logged. Friction-ful by design; explicit decisions, not implicit drift.
  • The reframe: "when not to build" often means "not this agent at this time." Criteria can be deferral conditions, converting refusal into a longer path with prerequisites funded.
  • The cultural implication: saying no is leadership. AI multiplies both good and bad decisions; the discipline of refusal is the discipline of compounding. The kill criteria list is what makes the discipline organizational rather than personal.
  • Rollout: draft the list, get executive sign-off, train the team, apply to new proposals and existing agents, track and refine.
  • The most valuable artifact in a 2026 AI program is not the agents that shipped. It is the list of agents that didn't.