Overview

Six months after deploying an AI-assisted change, an auditor asks: "What human judgment went into this decision?" You look at your change record and it says: "AI suggested network policy change. Approved and deployed." There's no record of what a human actually reviewed, what they checked, what they changed, or why they approved it. Was there actually human judgment? Or did someone just rubber-stamp an AI suggestion?

Now the auditor is asking harder questions: "Was this change properly validated?" "Could the human who reviewed it actually understand what the AI recommended?" "What was the reasoning for approval?" You realize you have no answers. The change looks good technically, but your documentation doesn't prove that human judgment was actually applied.

This is the core problem: when AI is involved in decisions, you need documentation that proves humans made actual decisions, not just approved AI outputs. Without clear documentation of AI involvement and human decision-making, you can't demonstrate proper controls to auditors, you can't learn from failures, and you can't trust that your team is actually reviewing AI recommendations instead of automatically accepting them.

Purpose

Documenting AI involvement means more than logging that AI was involved. It means documenting what the human actually did: what they checked, what they changed, why they approved or rejected the recommendation. It means building a record that shows human judgment was applied, not bypassed. This documentation serves your team (for learning and improvement), your auditors (for compliance), and your incident investigation (for understanding what went wrong).

This lesson covers practical documentation frameworks for AI-assisted decisions, how to integrate AI involvement into your existing change management and incident management systems, how to prove that humans actually reviewed and judged AI recommendations, and how to use that documentation to improve both AI suggestions and human decision-making. By the end, you'll have a documentation standard that works for your organization's processes.

Why This Matters

Documentation of AI involvement matters for different reasons to different stakeholders:

For compliance: Auditors need to verify that proper controls were in place. If your documentation shows that a human reviewed an AI recommendation, what they checked, and why they approved, that's a control. If it just says "AI suggested, we deployed," that's not a control.

For learning: When an AI-assisted decision fails, documentation lets you understand why. Was the AI wrong? Did the human miss something? Did the situation change after the decision? Without documentation, incident investigation is guesswork.

For accountability: When something goes wrong, people ask: Who was responsible? If you can document what each person (and the AI) contributed to the decision, you can answer that clearly.

For team trust: If your team doesn't see documentation of actual human review, they lose trust that AI is being properly governed. They assume decisions are automated when they should be collaborative.

For continuous improvement: Over time, documentation reveals patterns. Are humans reviewing AI recommendations carefully or rubber-stamping them? Are humans' changes to AI recommendations usually improvements or usually unnecessary? Use this data to improve your process.

Core Concepts

Key insight: Documentation of human involvement is as important as documentation of AI involvement. You need to document what the AI contributed AND what the human contributed, making clear that both were part of the decision.**

1. Documentation Framework: What to Record

Part 1: The Problem

What was the actual situation that needed a decision?

Problem Documentation:
- Specific issue or request (not vague: "Database slow" is vague;
"Query_X running 10s against table_Y with 5M rows" is specific)
- Who identified the problem
- When it was identified
- Business impact (how does this affect the organization?)
- Why this problem matters now
- Any constraints on the solution (time, cost, risk, dependencies)

Example:

Problem: Payment service is experiencing 500ms latency when fetching
transaction history for accounts with >1M transactions.
Identified by: Performance monitoring alert
When: 2026-04-09 14:32 UTC
Impact: Affects 1,200 daily users; may cause payment processing delays
Constraints: Can't modify database schema (other systems depend on it);
must roll back within 5 minutes if it breaks anything

Part 2: The AI Contribution

What did the AI recommend and what was its reasoning?

AI Contribution Documentation:
- Exact recommendation (the full artifact, not a summary)
- AI's reasoning (why it recommends this)
- AI's confidence level
- Assumptions the AI made
- Alternative approaches the AI considered
- Known limitations of the suggestion

Example:

AI Recommendation: Add covering index on transactions(account_id,
created_at) with INCLUDE(amount, status)
AI Reasoning: Query scans full table for account_id matches; covering
index eliminates the need to look up individual rows
Confidence: 87%
Assumptions:
- Index creation time is <30 seconds (untested)
- No other processes write to this table during index creation
- Query plan will use this index (estimated 3x throughput improvement)
Alternatives Considered:
- Partition table by account_id (more complex, slower to implement)
- Cached results in Redis (requires application change)
- Materialized view (adds data duplication)

Part 3: The Human Review

What did a human actually do to review the AI recommendation?

Human Review Documentation:
- Who reviewed the recommendation and their expertise
- What validation or checks they performed
- What concerns they identified
- What modifications they made to the AI recommendation
- Why they approved or rejected the recommendation
- What questions they asked about the AI's assumptions

Example:

Reviewed by: Sarah Chen (Senior DBA, 8 years experience)
Validation Performed:
✓ Verified assumption about index creation time (tested: 8 seconds)
✓ Checked for other writers during index creation (none found)
✓ Analyzed query plan with proposed index (confirmed 4x improvement,
better than AI estimate)
✓ Checked impact on existing indexes (no conflicts)
✓ Estimated disk space impact (42MB; acceptable)

Concerns Identified:
- Index on created_at might become stale if timestamps are modified
(SOLUTION: Added check constraint to prevent timestamp updates)
- Performance may degrade if account IDs become very large
(SOLUTION: Documented for future migration planning)

Modifications Made:
- Added created_at DESC for descending order
(reason: most queries fetch recent transactions first)
- Added index statistics refresh schedule
(reason: ensure query plan stays optimal)

Approval Decision: APPROVED
Reasoning: Addresses the performance problem; modifications make the
solution more robust. Risk is low; easily reversible.

Part 4: The Approval

Who authorized the decision and under what conditions?

Approval Documentation:
- Who approved the decision and their authority
- Date and time of approval
- Any conditions or constraints on implementation
- Sign-off that the reviewer's assessment was considered

Example:

Approved by: James Rodriguez (Infrastructure Manager)
Date/Time: 2026-04-09 15:30 UTC
Conditions:
- Must be deployed during business hours (easier to troubleshoot)
- DBA must be on-call during deployment
- Rollback must be tested before deployment
- Performance must be verified immediately after deployment

Approval Notes: Sarah's review was thorough. Changes to the AI
recommendation improve safety. Ready to proceed.

Part 5: The Implementation

How was the decision actually executed?

Implementation Documentation:
- Who implemented the change
- How they implemented it (specific commands/process)
- When it was implemented
- Pre-implementation verification (rollback plan tested? conditions met?)
- Post-implementation verification (did it actually work?)

Example:

Implemented by: automation_service (on-call: Sarah Chen)
Date/Time: 2026-04-09 16:02 UTC
Method: kubectl apply -f index_creation.yaml

Pre-Implementation:
✓ Rollback procedure tested (index deletion completed in 2 seconds)
✓ On-call resources ready
✓ Business hours confirmed
✓ Monitoring alerts configured

Implementation:
- Index creation started at 16:02:05
- Index creation completed at 16:02:13
- Query plan updated successfully
- Statistics refreshed

Post-Implementation:
✓ Latency reduced to 120ms (target: <200ms achieved)
✓ Throughput increased 4.2x (AI estimated 3x, actual was better)
✓ No errors in error logs
✓ Monitoring normal
Conclusion: SUCCESS

Part 6: Lessons Learned (if needed)

If something went wrong or there were surprises, document what you learned.

Lessons Learned:
- AI estimate was conservative (3x improvement); actual was 4x
(suggests AI may underestimate index effectiveness)
- Assumption about index creation time was conservative
(could have deployed larger changes more aggressively)
- Sarah's modification to add DESC ordering was crucial
(suggests AI doesn't always optimize for query patterns it can't see)

2. Integration with Change Management Systems

Your documentation should integrate with your existing change management processes:

If using ServiceNow or similar:
- Change Request document has standard fields
- Add "AI Involved" checkbox
- If checked, link to AI Recommendation attachment
- "Implementation Notes" section includes human review summary
- "Change Justification" references both AI reasoning and human judgment

If using Git + pull requests:
- Pull request includes both the AI-suggested change and human modifications
- PR description includes AI recommendation and human reasoning
- Commits reference the change ID
- Approval requires explicit human reviewer
- Merge commit message documents the decision

If using tickets + runbooks:
- Ticket includes "AI Assisted: Yes"
- Ticket documents what the human did to verify AI recommendation
- Ticket includes why the human approved or rejected AI suggestion

3. Accessibility: Making Documentation Useful

Documentation only matters if it's accessible:

Good documentation:
- Structured format (JSON, YAML, standardized fields)
- Searchable and indexed
- Linked from relevant systems
- Summarizable (you can extract the key points quickly)
- Accessible to auditors, but not exposing sensitive details

Bad documentation:
- Free-form text that's hard to parse
- Buried in lengthy documents
- No index or search capability
- Requires understanding lots of context to grasp the decision

4. Privacy and Sensitivity

Some AI involvement involves sensitive information:

Example: Security incident response
The AI identified suspicious access patterns and recommended disabling
an account. But the account belongs to an employee under investigation
for something unrelated. The documentation needs to record:
- That the decision involved sensitive factors
- That human judgment was applied carefully
- But not expose the sensitive investigation details

Solution: Tiered documentation
- Public record: "Decision made; account reviewed for security risk"
- Audit-accessible record: Full details available to auditors under NDA
- Investigation-accessible record: Details available only to security team

Practical Use Cases

Use Case 1: Documenting AI-Assisted Security Decision

Scenario: An AI flags suspicious behavior and recommends access restriction. You need to document that a human carefully reviewed this decision, not just rubber-stamped the AI's suggestion.

Documentation approach:

{
"incident_id": "SEC-2026-04-09-042",
"decision_type": "security_incident_response",
"problem": {
"description": "Unusual access pattern: account accessing 15
databases in 20 minutes; all read-only",
"severity": "medium",
"detected_at": "2026-04-09T14:22:33Z",
"business_impact": "Potential data exfiltration risk"
},
"ai_recommendation": {
"action": "temporary_access_restriction",
"details": "Restrict database access for 24 hours pending investigation",
"reasoning": "Access pattern doesn't match user's normal behavior;
multiple databases in short time suggests automated
scanning",
"confidence": 0.62,
"risk_if_wrong": "User cannot access systems they need; may impact
productivity"
},
"human_review": {
"reviewer": {
"name": "security_lead",
"team": "security",
"expertise": "incident response"
},
"review_process": [
{
"check": "Is this user typically on at this time?",
"result": "Yes, works on classified project with off-hours access"
},
{
"check": "Are these databases accessed in their normal work?",
"result": "Yes; role includes data migration responsibilities"
},
{
"check": "Does the pattern match the migration project timeline?",
"result": "Yes, exactly matches scheduled data migration"
},
{
"check": "Was the user notified about this activity?",
"result": "Will verify with their manager"
}
],
"concerns_identified": [
"AI doesn't have context about classified project",
"Unusual pattern is explained by legitimate work",
"Restricting access could disrupt critical migration"
],
"decision": "REJECT_AI_RECOMMENDATION",
"reasoning": "Pattern is suspicious only without context. User's
work is legitimate and documented. Restriction would
damage critical project. Add to monitoring but don't
restrict access.",
"actions_taken": [
"Notified user's manager about suspicious pattern",
"Verified legitimate project context",
"Added account to enhanced monitoring",
"Will alert on similar patterns going forward"
]
},
"follow_up": {
"monitoring_level": "enhanced",
"review_date": "2026-04-16",
"notes": "Demonstrates importance of context in security decisions.
AI flagged legitimate activity as suspicious."
}
}

Use Case 2: Documenting AI-Assisted Performance Decision

Scenario: AI suggests infrastructure optimization. You need to document the reasoning for the decision so you can audit whether it was effective and learn from outcomes.

Documentation template:

Change: Database Connection Pool Optimization

Problem
- Symptom: Occasional connection timeout errors under peak load
- Frequency: 2-3 times per week
- Impact: 0.2% of transactions fail; mostly recoverable
- Identified: Performance alerting system

AI Recommendation
- Suggestion: Increase max_pool_size from 100 to 150
- Confidence: 85%
- Reasoning: Peak concurrent connections reach 120; increasing to 150
provides headroom
- Assumptions:
- Peak load of 120 concurrent connections is stable
- Connection pool size is the bottleneck (not query execution)

Human Review (by Sarah Chen, Senior DBA)
- Validation performed:
✓ Verified peak concurrent connections (confirmed: 115-125)
✓ Checked if pool size is actually the bottleneck (yes, queue builds
when pool exhausted)
✓ Reviewed connection leak risk (mitigated by idle timeout)
✓ Estimated memory impact (acceptable)

• Concerns identified:
  - Should also optimize connection reuse (don't just add more)
  - Current idle timeout (15 min) might be too long
• Modifications:
  - Keep AI suggestion for pool size increase
  - Add: reduce idle timeout from 900s to 300s (recycle connections faster)
  - Add: enable connection validation to catch stale connections
• Approval: YES, with modifications

Implementation
- Deployed: 2026-04-09 16:00 UTC
- Method: Configuration change + service restart
- Verification: Timeout errors reduced from 2.3/day to 0.1/day
- Lesson: Combination of pool size + idle timeout optimization was
more effective than pool size alone

Audit Notes
- Decision made with appropriate human review
- Human's modifications improved on AI suggestion
- Outcome was positive
- Documentation is complete for compliance purposes

Use Case 3: Documenting AI Decision That Was Rejected

Scenario: An AI makes a recommendation that you decide not to follow. You need to document why the human judgment overrode the AI, so auditors know this was a deliberate decision, not a failure.

decision_record:
id: "DEC-2026-04-10-003"
context: "Database schema change request"

ai_recommendation:
action: "Normalize customer table structure"
reasoning: "Current schema has data redundancy; normalization would improve
consistency and reduce storage"
confidence: 0.91

human_judgment:
reviewer: "database_architect"
decision: "REJECT_AI_RECOMMENDATION"
reasoning: |
While technically correct, the AI recommendation doesn't account for:
1. Downstream systems expect denormalized structure
2. Reporting system has 6-month migration plan
3. Historical queries depend on current structure
4. Risk of breaking changes outweighs storage savings

documentation: |
The AI's analysis was technically sound, but it optimized for a metric
(storage efficiency) without considering other factors that are critical
in our context (cross-system compatibility, downstream dependencies,
planned migrations).

This is a good example of AI suggesting technically correct solutions
that aren't appropriate in a complex environment where human context
matters.

approval: "ACCEPTED (decision to reject AI recommendation)"
auditor_notes: "This demonstrates human judgment overriding AI when
context-dependent factors are present."

Examples

Example 1: Documentation Template for IT Operations

Decision Record: [Brief Description]

Decision ID: [DEC-YYYY-MM-DD-###]

1. Problem Statement
- What was the situation?
- Who identified it?
- Why does it matter?

2. AI Involvement
- Was AI used? [Yes/No]
- If yes, what was the AI's recommendation?
- What was the AI's confidence level?
- What were the AI's key assumptions?

3. Human Review
- Who reviewed the decision?
- What did they check?
- What concerns did they identify?
- What modifications did they make?
- Did they approve, reject, or modify?

4. Final Decision
- What decision was made?
- Who authorized it?
- What were the conditions?

5. Implementation
- How was it implemented?
- Did it work as expected?
- Any unexpected outcomes?

6. Follow-Up
- Any lessons learned?
- Would you do this differently next time?
- What metrics show the outcome?

7. Audit Certification
- [ ] All required approvals obtained
- [ ] Documentation is complete
- [ ] Human judgment was applied
- [ ] Decision is justified and traceable

Example 2: Structured Decision Record (JSON)

#!/usr/bin/env python3
"""
Structured documentation of AI-involved decisions
"""

import json
from datetime import datetime

def create_decision_record(
problem_description,
ai_recommendation=None,
human_review=None,
final_decision=None,
implementation=None
):
"""Create a structured decision record"""

record = {
"metadata": {
"record_id": f"DEC-{datetime.now().strftime('%Y-%m-%d-%H%M%S')}",
"created_at": datetime.now().isoformat(),
"system": "IT Operations AI Decision Log"
},
"problem": {
"description": problem_description["description"],
"identified_by": problem_description.get("identified_by"),
"impact": problem_description.get("impact"),
"constraints": problem_description.get("constraints", [])
},
"ai_involvement": {
"ai_used": ai_recommendation is not None,
"recommendation": ai_recommendation.get("action") if ai_recommendation else None,
"confidence": ai_recommendation.get("confidence") if ai_recommendation else None,
"reasoning": ai_recommendation.get("reasoning") if ai_recommendation else None,
"assumptions": ai_recommendation.get("assumptions", []) if ai_recommendation else []
},
"human_review": {
"reviewer": human_review.get("reviewer") if human_review else None,
"checks_performed": human_review.get("checks") if human_review else [],
"concerns": human_review.get("concerns") if human_review else [],
"modifications": human_review.get("modifications") if human_review else [],
"recommendation": human_review.get("recommendation") if human_review else None
},
"decision": {
"action": final_decision.get("action") if final_decision else None,
"authorized_by": final_decision.get("authorized_by") if final_decision else None,
"conditions": final_decision.get("conditions", []) if final_decision else []
},
"implementation": {
"method": implementation.get("method") if implementation else None,
"date_time": implementation.get("date_time") if implementation else None,
"outcome": implementation.get("outcome") if implementation else None,
"verification": implementation.get("verification") if implementation else []
}
}

return record

def save_decision_record(record, filepath):
"""Save decision record to file for audit"""

with open(filepath, 'a') as f:
json.dump(record, f)
f.write('\n') # Newline for separation

def audit_decision_record(record):
"""Verify decision record is complete for audit"""

required_fields = [
("metadata.record_id", True),
("problem.description", True),
("human_review.reviewer", True), # Human judgment required
("human_review.recommendation", True),
("decision.action", True),
("decision.authorized_by", True)
]

issues = []

for field, required in required_fields:
parts = field.split('.')
value = record
for part in parts:
if isinstance(value, dict):
value = value.get(part)
else:
value = None
break

if required and not value:
issues.append(f"Missing required field: {field}")

return len(issues) == 0, issues

Example usage
problem = {
"description": "API response times degrading",
"identified_by": "monitoring_system",
"impact": "User-facing delay affecting all API clients",
"constraints": ["Must roll back within 5 minutes"]
}

ai_rec = {
"action": "increase_cache_ttl",
"confidence": 0.78,
"reasoning": "Cache hit rate declining; increasing TTL reduces backend load",
"assumptions": ["cache_invalidation_is_working", "no_stale_data_issues"]
}

human_review_result = {
"reviewer": "api_team_lead",
"checks": [
"Verified cache implementation is working",
"Checked for stale data issues (none found)",
"Analyzed typical cache lifetimes for this data"
],
"concerns": [
"Cache might be too aggressive for frequently-changing data",
"Need to monitor cache hit rate after change"
],
"modifications": [
"Increase TTL but add cache invalidation for critical updates"
],
"recommendation": "APPROVE_WITH_MODIFICATIONS"
}

final_decision_result = {
"action": "increase_cache_ttl_with_smart_invalidation",
"authorized_by": "api_manager",
"conditions": ["Monitor cache hit/miss rates", "Alert if invalid data served"]
}

implementation_result = {
"method": "configuration_change",
"date_time": "2026-04-10T10:00:00Z",
"outcome": "SUCCESS",
"verification": [
"API response time improved",
"Cache hit rate increased",
"No stale data reported"
]
}

record = create_decision_record(
problem, ai_rec, human_review_result, final_decision_result, implementation_result
)

Verify completeness
complete, issues = audit_decision_record(record)
if complete:
print("Decision record is complete and audit-ready")
else:
print("Issues found:")
for issue in issues:
print(f" - {issue}")

save_decision_record(record, "/var/log/decisions/decisions.log")

Anti-Patterns

Anti-Pattern 1: Documenting only the AI's decision, not the human's

If your documentation shows "AI recommended X, we did X," you haven't documented human judgment. Document what the human did: what they checked, what they changed, why they approved.

Anti-Pattern 2: Generic documentation that could apply to any decision

"Reviewed by: John. Approved." tells you nothing. "John checked the deployment impact on dependent services, verified no breaking changes, and approved because risk is low and rollback is easy" tells you the human actually reviewed it.

Anti-Pattern 3: Documentation that's hard to access or search

If someone investigating an incident can't easily find the documentation of how a decision was made, it's not useful. Make documentation searchable and accessible.

Anti-Pattern 4: Documenting only decisions that succeeded

Document failures too. When an AI-assisted decision fails, that's when documentation is most valuable. You need to understand why it failed.

Anti-Pattern 5: Separating AI documentation from human documentation

If you document the AI's reasoning separately from the human's judgment, you lose the context that connects them. Keep them together so the relationship is clear.

Anti-Pattern 6: Not making documentation part of your process

If documentation is optional, people won't do it. Make it part of your change management process: document or don't deploy.

Human Judgment Checkpoints

1. Deciding what level of documentation is appropriate

Not every decision needs a full audit trail. Minor changes might need less documentation. Your judgment: what documentation level matches the risk and impact?

2. Evaluating whether the human actually reviewed the AI recommendation

Documentation says a human reviewed it, but did they really? Evaluate: Did they check for plausibility or just check that it compiled? Did they understand what they were approving?

3. Recognizing when documentation reveals process problems

Good documentation can show that humans are rubber-stamping AI decisions, that they're not actually reviewing them. Use that data to improve your process.

Key Takeaways

Document the complete decision chain: Problem → AI recommendation → Human review → Human judgment → Final decision → Implementation → Outcome.

Make human judgment visible: Document what the human did, not just that a human was involved. What did they check? What did they change? Why did they approve?

Integrate with existing systems: Add AI documentation to your change management, incident management, and approval processes. Don't create a separate system.

Make documentation searchable and accessible: Structure it so you can find decisions by system, date, outcome, or type of change.

Use documentation to improve: Analyze decisions over time. Are humans actually reviewing? Are modifications usually improvements? Use this data to improve your process.

Prove human judgment was applied: Documentation is your evidence that proper controls were in place. Make it clear that humans made decisions, not just approved AI outputs.

Document rejections too: When you reject an AI recommendation, document why. That's valuable learning for the AI and for auditors.