Overview

You're managing an AI that helps with infrastructure tasks. It's doing well: generating documentation, drafting RFCs, analyzing logs. Then one day you realize: someone just asked it to "create a firewall rule" and AI obliged. The rule is now live in production. Nobody reviewed it. Now you're panicking.

This is a production boundary problem. The production boundary is the line between work AI can do independently and work that requires human approval before it affects live systems. On the safe side: documentation, drafting, analysis. On the dangerous side: making changes to production systems.

The production boundary is not always obvious. Is creating a DNS entry a production change? What about updating an SSL certificate? Modifying a firewall rule? Rotating credentials?

Understanding the production boundary helps you design AI systems that are helpful without being reckless.

Purpose

The production boundary is the safety line. Work that crosses the boundary must have human approval before execution. Work on the safe side can be more autonomous.

The production boundary serves two purposes:

• Safety: Prevents AI from making changes that could break systems, violate compliance, or expose data
  - Accountability: Ensures humans are accountable for changes that affect production

The boundary is organizational-specific. What's production for a bank (every change is critical) differs from a startup (move fast, fix breaks). But every organization has a boundary somewhere.

Why This Matters

Without a clear production boundary, you either over-restrict AI (making it useless) or under-restrict it (making it dangerous):

• Over-restriction: AI can only do low-risk work, but even low-risk work requires review. Result: minimal productivity gain, AI is bottleneck waiting for approval.
  - Under-restriction: AI makes production changes independently. Result: incidents occur that didn't need to, systems are less stable, compliance is at risk.
  - Ambiguous boundary: Team doesn't agree what's production-critical. Some changes get approved, similar changes don't. Inconsistency and confusion result.

A clear boundary solves this: work on the safe side moves fast, work on the dangerous side requires approval.

Key Insight: The Boundary Is About Control, Not Risk

The production boundary is not about how risky the work is. It's about whether you want human control before the change goes live. A low-risk change might still require approval because you want visibility. A high-risk change might be pre-approved if it follows a standard procedure.

Core Concepts

1. Categories of Work

Work falls into categories based on impact and reversibility:

Tier 0: Approved Autonomously

• Documentation: runbooks, procedures, architecture docs
• Drafting: RFC outlines, change request templates, communication templates
• Analysis: trend analysis, cost analysis, security scans that don't make changes
• Suggestions: recommendations, optimization ideas, process improvements
• Research: finding information, summarizing knowledge

*Why autonomous:* These don't change systems. They're reversible (edit, delete, rewrite). Low risk.

Tier 1: Recommended Review, Can Proceed

• Staging/test deployments: infrastructure in non-production environments
• Non-sensitive configurations: updating a proxy's cache timeout, enabling logging
• Adding monitoring: new monitoring rules, alerts, dashboards
• Creating tickets/records: change requests, incident reports, support tickets
• Bulk data operations: backups, archiving, exports (if no deletion)

*Why light review:* These might impact development/testing but not production. Reversible. Low risk to live systems. Still good to review for quality/correctness, but don't block on review.

Tier 2: Requires Approval

• Production deployments: software releases, config changes to live systems
• Access control changes: adding/removing users, changing permissions, elevating privileges
• Infrastructure changes: network rules, firewall rules, routing, DNS
• Database operations: schema changes, permission modifications (not data operations)
• Security controls: policy changes, encryption settings, authentication rules
• Backup/recovery operations: affecting live data, changeability impacts recovery

*Why approval required:* These affect live systems. Mistakes are expensive. Reversibility might be limited (DNS caching, certificate replacement). High risk.

Tier 3: Requires Explicit Pre-Approval and Detailed Procedure

• Data destruction: deleting live data, purging logs, archiving
• Emergency changes: breaking glass procedures, incident response changes
• Bulk privilege escalation: granting admin access temporarily for incident response
• System takeover: assuming control of a system in failure state

*Why strict control:* Irreversible or nearly so. High risk. Requires explicit pre-approval plus detailed procedure.

Key insight: Clear tiers help you decide what AI can do independently vs. what needs approval.

2. Production vs. Staging vs. Development

Same type of work has different risk profile depending on environment:

DEPLOY A NEW VERSION:
- Development environment: Tier 0-1 (minimal approval)
- Staging environment: Tier 1 (light review recommended)
- Production environment: Tier 2 (requires approval)

MODIFY A CONFIGURATION:
- Development: Tier 0-1 (autonomous)
- Staging: Tier 1 (light review)
- Production: Tier 2 (requires approval)

ADD A FIREWALL RULE:
- Test environment: Tier 1 (light review)
- Production: Tier 2 (requires approval, CAB process)

ADD A MONITORING ALERT:
- Non-critical system: Tier 1 (light review)
- Critical system: Tier 2 (approval required)

The same work requires different oversight depending on environment and criticality. A script in staging is low-risk. The same script in production is high-risk.

Key insight: Risk is environmental. Production is always higher risk than staging.

3. Reversibility and Recovery

Some changes are reversible, others are not:

REVERSIBLE (can be undone):
- Adding a monitoring rule (delete it)
- Deploying a new version (rollback to old version)
- Modifying a non-critical config (revert to previous)

PARTIALLY REVERSIBLE (can be undone, takes effort/time):
- DNS change (eventually propagates backward, but takes 24+ hours)
- Certificate replacement (old cert might be needed briefly)
- Adding a user (can be deleted, but history might remain)

IRREVERSIBLE (cannot be undone):
- Deleting data (unless you have backups)
- Destructive migrations (data format changes)
- Purging logs (audit trail is gone)

LONG-TAIL REVERSIBILITY:
- Rotating credentials (old creds no longer work, can cause service failures if something still uses them)
- Changing DNS (applications might cache results)

Non-reversible changes warrant more scrutiny because mistakes can't be undone. Reversible changes warrant less because rollback is available.

Key insight: Non-reversible changes should require approval. Reversible changes can proceed with lighter review.

4. Compliance and Audit Requirements

Some changes are compliance-critical:

COMPLIANCE-CRITICAL CHANGES:
- Access control modifications (SOC 2, HIPAA, PCI require approval)
- Encryption policy changes (compliance requirement)
- Data retention modifications (legal/compliance)
- Audit logging changes (compliance)
- User privilege changes (compliance)

NON-COMPLIANCE-CRITICAL CHANGES:
- Monitoring configuration (good practice, not compliance)
- Documentation updates (informational)
- Non-sensitive config updates (operational)

Compliance-critical changes need evidence of approval and authorization. You need to show the auditor: "We changed this access control, it was approved by [person], on [date], for [reason]." AI-independent changes won't have that paper trail.

Key insight: Compliance-critical work requires approval and documentation.

5. Build a Production Boundary Classification

Create a matrix that classifies work:

PRODUCTION BOUNDARY CLASSIFICATION

Work Item: ________________
Environment: Dev / Staging / Production

RISK ASSESSMENT:
□ Affects production systems (live data, live users): Yes / No
□ Is reversible (can be rolled back): Yes / Partially / No
□ Compliance-critical (SOC 2, HIPAA, etc.): Yes / No
□ High-stakes (business-critical system): Yes / No

CLASSIFICATION:
□ Tier 0: Autonomous (documentation, analysis, drafting)
□ Tier 1: Light review (staging changes, non-critical changes)
□ Tier 2: Approval required (production changes)
□ Tier 3: Strict approval (data destruction, emergency changes)

APPROVAL PROCESS:
[If Tier 2 or 3]
□ Requires CAB approval (change advisory board)
□ Requires security review (if compliance-critical)
□ Requires manager approval (if high-stakes)
□ Requires documented evidence of approval before execution

NOTES:
[Reason for classification, special considerations]

This matrix makes the decision explicit and documentable.

Practical Use Cases

Before: Unclear what AI can do independently; some changes bypass review.

After: Clear boundary; work moves fast on safe side, gets approval on dangerous side.

Use Case 1: Documentation Automation (Tier 0)

You build an AI system to generate runbooks from your infrastructure.

Boundary decision:

• Generate runbook outlines from existing procedures: Tier 0 (autonomous)
• Generate complete runbooks: Tier 0 (autonomous)
• Publish runbooks to wiki: Tier 1 (light review: tech writer reviews for clarity, owner reviews for accuracy)
• Deploy runbook to live support system: Tier 0 (it's documentation, not a system change)

Why Tier 0? Runbooks are documentation. Mistakes are caught before use. Reversible (edit, delete). No compliance impact.

Risk: Wrong runbook steps. Mitigation: owner spot-checks before publication. Light review, not blocking.

Use Case 2: Infrastructure Configuration in Staging (Tier 1)

You build an AI system to help configure test environments.

Boundary decision:

• Generate infrastructure-as-code (Terraform) for staging: Tier 1 (light review)
• Apply Terraform to staging: Tier 1 (light review: someone checks the plan, then applies)
• Test the staging deployment: Tier 1 (autonomous)
• Report test results: Tier 0 (autonomous)

Why Tier 1? It's staging, so low risk to live systems. But still review the plan before applying (catches obvious mistakes). Reversible (can redeploy).

Risk: Breaking staging (delays dev work). Mitigation: peer review of Terraform plan before apply.

Use Case 3: Production Firewall Rules (Tier 2)

You want AI to help draft firewall rules.

Boundary decision:

• Analyze current firewall rules and suggest improvements: Tier 0 (autonomous)
• Draft new firewall rules: Tier 1 (light review: security team reviews for correctness)
• Test new rules in staging: Tier 1 (light review: network team validates)
• Apply rules to production: Tier 2 (requires CAB approval and security sign-off)

Why Tier 2? Firewall rules affect production. Mistakes can break connectivity. Compliance-critical (audit trail required). Non-reversible (takes time to update).

Risk: Misconfiguration breaks production connectivity. Mitigation: CAB review, security sign-off, staged rollout.

Use Case 4: User Access Provisioning (Tier 2)

You want AI to help with user access requests.

Boundary decision:

• Draft access request from description: Tier 0 (autonomous)
• Send request for approval: Tier 0 (autonomous)
• Execute access change if approved: Tier 2 (requires documented approval before execution)

Why split? Drafting is safe. Executing a pre-approved change is safe (approval is the gate). But AI shouldn't independently execute access changes without proof of approval.

Risk: Unauthorized access granted. Mitigation: approval requirement before execution.

Examples

Example 1: Tier Classification for Common IT Tasks

TASK: Database backup

• Autonomous Tier 0: Generate backup script, document backup procedure
  - Light Review Tier 1: Run backup in staging, validate backup integrity,
  restore from backup in test
  - Approval Tier 2: Change backup schedule in production, modify retention policy

WHY: Script/docs are safe. Staging backups are low-risk. Production backup
changes affect compliance/recovery, warrant approval.

---

TASK: SSL certificate renewal

• Autonomous Tier 0: Identify expiring certificates, generate renewal
  documentation, draft renewal procedure
  - Light Review Tier 1: Generate CSR, submit renewal request to CA
  - Approval Tier 2: Deploy new certificate to production, retire old certificate

WHY: Planning is safe. Submission is safe. Deployment affects production
availability and requires approval.

---

TASK: Create new user account

• Autonomous Tier 0: Generate account request from description
  - Approval Tier 2: Approve account creation in directory, provision access

WHY: Draft request is safe. Execution requires approval (compliance requirement).

---

TASK: System monitoring

• Autonomous Tier 0: Analyze logs, identify trends, generate alerts
  for new patterns
  - Light Review Tier 1: Create new monitoring dashboards (reviewed for clarity)
  - Approval Tier 2: Modify SLA-critical alerts (affects incident response)

WHY: Analysis is safe. Dashboards are informational. Alert changes affect
incident response, warrant review.

This classification helps teams understand what AI can do safely.

Example 2: Boundary Policy Document

PRODUCTION BOUNDARY POLICY

TIER 0: AUTONOMOUS (No Approval Needed)
AI can execute independently:
- Generate documentation, procedures, runbooks
- Analyze systems and data (no changes)
- Draft change requests, communication, procedures
- Create issues, tickets, records
- Generate reports and summaries
- Test in development environment
- Suggest improvements (recommendations, not execution)

TIER 1: LIGHT REVIEW (Recommended, Non-Blocking)
AI can proceed after light review:
- Deploy to staging/test environments
- Add monitoring/alerts (non-SLA-critical)
- Run tests and validation
- Create backups (non-production)
- Modify non-critical configurations
- Bulk data operations on test data
- Documentation in shared systems

Who reviews: Team lead, relevant owner, technical peer
Timeline: Same day preferred, next day acceptable
Blocking: No (proceed if review incomplete, but document)

TIER 2: APPROVAL REQUIRED (Blocking)
AI must have approval before executing:
- Production deployments
- Production configuration changes
- Access control modifications (user provisioning, privilege changes)
- Infrastructure changes (network, DNS, firewall, storage)
- Database schema changes
- Security control modifications
- Backup/recovery operations on live data
- Certificate replacement
- Any change affecting compliance or audit trail

Who approves: CAB for infrastructure, Security for security/compliance,
Manager for access control
Timeline: CAB meets weekly; emergency changes can be expedited
Blocking: Yes (must have approval before execution)
Evidence: Approval must be documented (ticket, email, CAB notes)

TIER 3: STRICT CONTROL (Explicit Procedure + Approval)
AI cannot execute even with approval without explicit procedure:
- Data destruction/purging (permanently deletes data)
- Credential rotation affecting production (can cause service failures)
- Emergency access grants (break-glass procedures)
- System takeover (assuming control in failure state)
- Audit log modification

Who approves: Security + Compliance + Manager
Procedure: Documented emergency procedure must be followed
Evidence: Post-incident review after execution

BOUNDARY EXAMPLES:
- Fix documentation: Tier 0
- Test script in dev: Tier 0
- Deploy to staging: Tier 1
- Deploy to production: Tier 2
- Create user account: Tier 2
- Delete user account: Tier 2 (or Tier 3 if data purging)
- Add firewall rule: Tier 2
- Rotate certificates: Tier 2
- Modify SSL policy: Tier 2
- Create monitoring alert: Tier 1 (non-critical) or Tier 2 (SLA-critical)
- Delete production data: Tier 3

ENFORCEMENT:
- Tier 0/1 work: Execute and document
- Tier 2 work: Require approval before execution, document approval evidence
- Tier 3 work: Require procedure + approval + post-execution review

VIOLATIONS:
- Tier 2 work executed without approval: Incident investigation required
- Tier 3 work executed outside procedure: Incident investigation + security review

This policy is clear and enforceable.

Anti-Patterns

Anti-Pattern 1: No Boundary (Everything Requires Approval)

Every change requires CAB approval. Even documentation changes, which is silly. Result: bureaucracy, slowness, people avoid using AI.

Prevention: Create tiers. Keep Tier 0 and 1 moving fast. Only gate Tier 2 and 3.

Anti-Pattern 2: Unclear Boundary (Everyone Interprets Differently)

Person A thinks "create a monitoring alert" is Tier 1. Person B thinks it's Tier 2. Same work gets approved inconsistently.

Prevention: Explicit classification. "SLA-critical alerts are Tier 2. Non-critical alerts are Tier 1."

Anti-Pattern 3: Boundary Based on AI vs. Human

"All AI-generated changes require approval, but human-made changes don't."

Problem: AI changes are not inherently riskier than human changes. A human can make the same mistake an AI makes. Inconsistent policy.

Prevention: Boundary based on change impact, not source. An unsafe change is unsafe whether human or AI.

Anti-Pattern 4: Boundary That's Too Restrictive

Everything is Tier 2. AI is basically useless because every change requires approval.

Prevention: Be realistic about risk. Documentation, testing, drafting are low-risk. Let those go autonomous.

Anti-Pattern 5: Boundary That's Too Permissive

AI can make production changes independently. Result: incident that didn't need to happen.

Prevention: If a change affects live systems, requires compliance evidence, or is non-reversible, it's Tier 2 at minimum.

Human Judgment Checkpoints

• 
  Reversibility assessment: "Can we actually undo this?" If no, Tier 2 at minimum.
• 
  Compliance impact: "Does this affect compliance?" If yes, Tier 2.
• 
  Business criticality: "If this breaks, how bad?" If very bad, Tier 2.
• 
  Audit trail: "Do we need evidence of approval?" If yes, Tier 2.
• 
  Organizational culture: "What's our appetite for risk?" Conservative org → more Tier 2. Aggressive org → more Tier 1.

Key Takeaways

Define a clear production boundary. Without it, decisions are inconsistent and risk is unclear.

Create tiers based on impact and reversibility. Tier 0 (autonomous) and Tier 1 (light review) for low-risk work. Tier 2 (approval) for production-affecting work. Tier 3 (strict) for non-reversible work.

Environment matters. Same work is different risk in dev vs. production. Classify by environment.

Compliance work is always Tier 2 at minimum. Compliance-critical changes need approval and audit trail.

Non-reversible work is always Tier 2 at minimum. If you can't undo it, you need approval before doing it.

Make boundaries explicit. Don't rely on judgment calls. List what's Tier 0, Tier 1, Tier 2, Tier 3.

Educate team on boundaries. If everyone understands the boundary, decisions are faster and more consistent.

Enforce the boundary in tooling. Set up automation so Tier 2 changes require approval before execution. Make the boundary part of the workflow, not just policy.

Review boundary periodically. After incidents, ask: "Should this have required approval?" Update boundaries accordingly.

Boundary is not about AI capability, it's about organizational risk tolerance. A high-quality AI change to a critical system still needs approval because the impact is high, not because AI made it.