Overview

Your monitoring tool fires an alert. It's 3 AM. An on-call engineer wakes up. They check the alert. It's a blip, a temporary metric spike that resolved itself. The engineer goes back to sleep. You just wasted 30 minutes of their time and burned through on-call fatigue.

This lesson teaches you to automate the alert-to-incident journey: detecting real incidents, filtering false positives, auto-classifying severity, and notifying the right people. When done right, your alert noise disappears, your MTTR drops, and your on-call engineers sleep better.

Purpose

Modern IT systems generate thousands of alerts. Most are noise. Your job is to find the signal. The naive approach: alert on everything and let humans filter. The result: alert fatigue, on-call burnout, real incidents buried in noise.

The AI approach: correlation, anomaly detection, business impact analysis, and intelligent classification. The result: real incidents bubble up, false positives disappear, your on-call team focuses on actual problems.

This lesson covers the end-to-end workflow: raw alerts in → AI processing → classified incident out. You'll learn how to configure thresholds, tune sensitivity, reduce false positives, and build a pipeline that understands your systems and your business.

Why This Matters

A 500-person company got 2,000 alerts per day. Their on-call team ignored 95% of them. When a real incident hit (database down), it took 2 hours to notice because the alert was lost in the noise. The company lost $200k in revenue before they got the database back up.

They fixed it with incident detection AI. Same 2,000 alerts, but AI filtered to 15 real incidents per day. Real incidents got paged to on-call within 1 minute. Their next database outage was fixed in 12 minutes because the team knew about it immediately.

This isn't about deploying "AI." It's about building a detection pipeline that understands correlation, false positive patterns, and business impact. The on-call engineer's sleep is worth the effort.

Core Concepts

Key insight: Incident detection is a pipeline, not a model

You don't plug in a machine learning model and call it done. You build a multi-step pipeline:

Step 1: Alert Ingestion
Sources: Prometheus, Datadog, Splunk, CloudWatch, PagerDuty, etc.
Input: Metric violation (CPU >90%, latency >1s, error rate >5%)
Output: Alert event (timestamp, metric, threshold, value)

Step 2: Alert Enrichment
Add context: service metadata, ownership, criticality, dependencies
Input: Alert event
Output: Enriched alert (now knows: this is the payment service, it's critical, affects 1000+ users)

Step 3: Alert Deduplication
Is this the same alert firing from the same service?
Or a different service affected by the same root cause?
Input: Enriched alert
Output: Deduplicated alert (don't create 50 incidents for the same problem)

Step 4: Correlation
Are multiple alerts related?
Alert A: Database error rate up. Alert B: API latency up.
Are they the same incident (one upstream alert, one downstream symptom)?
Input: Multiple alerts
Output: Grouped incidents (this is one problem, not two)

Step 5: Anomaly Detection
Is this metric actually abnormal, or expected variation?
Example: CPU 85% at 2 PM during batch job = normal.
CPU 85% at 3 AM during quiet hours = anomalous.
Input: Metric + historical context
Output: Is this truly anomalous? (Boolean)

Step 6: Business Impact Analysis
How many users affected? Is this production or staging? Critical path or non-critical?
Input: Enriched alert + affected service metadata + usage patterns
Output: User count, revenue impact, SLA risk

Step 7: Severity Classification
Critical, High, Medium, Low?
Uses: Business impact + alert severity + historical patterns
Input: Alert details + impact analysis
Output: Severity label

Step 8: Notification & Escalation
Who should be notified? Page or ticket only?
Input: Severity + service + team assignments
Output: Notification sent (page + email + Slack)

Step 9: Incident Ticket Creation
Create ticket with all context pre-populated
Input: Severity + enriched alert + impact analysis
Output: Incident ticket ready for investigation

Each step is a quality gate. If a step fails, the incident gets stuck in that step. If step 4 (correlation) fails, you might create two separate incidents for the same problem. If step 6 (impact analysis) fails, you might page the wrong team.

Key insight: False positive elimination is 80% of the work

Thousands of alerts fire. Most are false positives:

• Metric spike that resolves in 30 seconds? False positive.
• Temporary network blip? False positive.
• Known maintenance window? False positive, should be suppressed.
• Metric at 85% but the threshold was configured for 80% three years ago? False positive.

Building incident detection is mostly about eliminating false positives, not catching more incidents.

Technique 1: Alert Suppression Rules

Suppress known false positives:

• If timestamp in maintenance_window, suppress
  - If alert fired in last 30 min and resolved itself, don't create incident
  - If error rate >5% but recovering quickly (<30 sec), might be false positive
  - If service is non-critical and metric is at 70% (expected variance), suppress

Technique 2: Threshold Tuning

Static thresholds are wrong. They don't account for:

• Time of day (peak hours vs. quiet hours)
• Day of week (weekday vs. weekend)
• Business cycles (quarter-end batch jobs cause load spikes)
• Recent changes (new feature causing expected load increase)

Dynamic thresholds adapt:

CPU threshold:
- Weekday 9-5: 85% (normal business hours)
- Weekday 5-9: 80% (after-hours, should be quiet)
- Weekday 9-11 PM: 75% (batch job window, expect high CPU but don't alert unless it exceeds historical norm by 10%)
- Weekday midnight-6 AM: 70% (quiet hours)
- Weekends: 70% (lower expected load)

Technique 3: Correlation & Dependency Mapping

If the database is having issues, you'll see alerts from:

• Database metrics (CPU, memory, query time)
• Dependent services (API latency, error rate)
• Client applications (timeouts, failures)

Naive alerting: 50 separate incidents.

Smart alerting: "Database problem with 5 downstream impacts" = 1 incident.

Correlation uses service dependency graph:

If Database alert fires:
Check: What services depend on the database?
Expected alerts: API (should see latency), App (should see timeout)
If API and App also fired alerts → Correlate into single incident
Reduce from 3 incidents to 1

Technique 4: Baseline Learning

What's normal for your system?

• This service handles 1000 requests/sec on average. 1200 requests/sec is normal variance.
• This database query usually takes 50ms. 100ms is a spike, but 150ms is an anomaly.
• This error rate usually sits at 0.01%. 0.1% is a problem.

Build baselines from historical data:

Normal range for CPU on service-api:
Average: 45%
95th percentile (normal spikes): 65%
99th percentile (extreme but expected variance): 80%

Alert threshold: 85% (above 99th percentile, truly anomalous)

This approach eliminates 90% of false positives.

Key insight: Severity classification is pattern-based and learnable

An alert fires. You need to classify it: Critical, High, Medium, Low.

Critical: User-facing, affecting many users, production, immediate response needed

• Payment service down
• Authentication service down
• Database affecting 1000+ users

High: User-facing or important system, moderate impact, quick response needed

• Batch service failing (50 users affected, but critical for operations)
• Cache cluster degraded (performance issue, not outage)
• API latency +50% (users notice, but still working)

Medium: Non-critical or limited impact, can wait a few hours

• Non-production service down
• Admin tool failing (affects 5 internal people)
• Disk usage at 85% (not immediate risk)

Low: Minor issues, informational

• Backup job took longer than usual
• Development environment down
• Log ingestion slightly delayed

Patterns for classification:

IF service is PAYMENT and status is DOWN → Critical
IF service is PAYMENT and latency >1s → High
IF service is INTERNAL and DOWN → Medium
IF service is STAGING and DOWN → Low

IF users_affected >1000 and status is DOWN → Critical
IF users_affected 100-1000 and status is DOWN → High
IF users_affected <100 or status is DEGRADED → Medium or Low

IF revenue_impact >$1k/min → Critical
IF revenue_impact >$100/min → High
IF revenue_impact minimal → Medium or Low

Build these rules from your organization's incident history. What incidents did you respond to immediately? Those should classify as Critical. What incidents did you handle during business hours? Those are High or Medium.

Key insight: Confidence thresholds determine auto-escalation

An incident is classified as "High severity" with 75% confidence. Do you auto-page the team, or send a ticket?

Set thresholds:

• Confidence >90%: Auto-page immediately
• Confidence 80-90%: Page, but also notify incident commander
• Confidence 70-80%: Create high-priority ticket, don't page yet
• Confidence <70%: Create normal-priority ticket, no page

This prevents alert fatigue while ensuring real incidents get attention quickly.

Key insight: Feedback loops improve incident detection over time

After every incident:

• What did the detection system predict?
• What was the actual severity?
• Should it have been escalated earlier or later?

Use this feedback to improve the system:

Incident: "Payment service latency spike"
Detection predicted: High severity (80% confidence)
Actual severity: Critical (caused 10 minutes of payment failures)
Learning: Payment service latency >50% should be higher confidence + higher severity
Next time, 90% confidence → Auto-page immediately

Incident: "Database CPU spike during backup"
Detection predicted: High severity (75% confidence)
Actual severity: Low (expected, backup completes in 15 min)
Learning: Database CPU spike during backup window = normal, suppress or lower severity
Next time, suppress alerts during 2-4 AM backup window

Incident: "API error rate spike"
Detection predicted: Critical severity (85% confidence)
Actual severity: Medium (only test traffic affected, no production users)
Learning: Error rate spike should check if traffic is production or test
Next time, filter for production traffic only

Practical Use Cases

Use Case 1: Multi-Step Detection Pipeline for Microservices

You run 50 microservices. Each generates metrics: CPU, memory, error rate, latency, queue depth. That's 250+ metrics, each with an alert threshold.

Before AI: Every service has static thresholds. CPU >80% = alert. Error rate >5% = alert. During business hours, you fire 100+ alerts. Most are false positives. Your on-call team stops responding.

After AI detection pipeline:

Step 1: Alert Ingestion (Prometheus → Detection Pipeline)
Incoming alerts: 500/day from 50 services

Step 2: Alert Enrichment
For each alert, look up:
- Service ownership (who owns this service?)
- Service criticality (production, staging, development?)
- Service dependencies (what services depend on this?)
- Recent deployments (did something change?)
- Business impact (affects paying customers?)

Example enrichment:
Alert: "api-gateway CPU >85%"
Enriched: "api-gateway (critical, 50 dependencies, 100k users affected)
Recent deployment: 30 min ago"

Step 3: Deduplication
Is this alert already firing from this service?
Or is it a new alert?

If already firing (CPU spike ongoing): Update existing alert, don't create new incident
If new alert (new service experiencing CPU spike): Create new alert event

Step 4: Correlation
Did the api-gateway alert cause dependent service alerts?

Check:
- api-gateway CPU spike at 2:00 PM
- payment-service latency spike at 2:01 PM
- checkout-service timeout at 2:02 PM

Correlate: All three are dependent on api-gateway. Likely single incident.
Output: "Gateway problem with downstream impacts" (1 incident, not 3)

Step 5: Anomaly Detection
Is this CPU spike truly anomalous, or expected?

Context:
- Time: 2:00 PM (business hours, typically busy)
- Day: Tuesday (typical load day)
- Recent events: Deployment 30 min ago

Analysis:
- Baseline CPU for api-gateway: 45% (business hours average)
- Spike to 85% = 89th percentile (elevated, but not extreme)
- Post-deployment spikes are normal (new code warming up)

Verdict: Elevated, but expected post-deployment. Anomaly score: Medium (6/10)

Step 6: Business Impact Analysis
How many users affected by api-gateway degradation?

Input: api-gateway latency increased from 50ms to 150ms (3x slower)
100k requests/sec flowing through gateway
5% of requests timing out

Analysis:
- Users experiencing timeouts: 5% × 100k = 5,000 users
- Revenue impact: Checkout service failed, ~$500 lost in failed transactions
- SLA risk: Response time SLA is 200ms, we're at 150ms (within SLA but degraded)

Output: 5,000 affected users, $500 revenue at risk, SLA: within limits but degraded

Step 7: Severity Classification
Pattern match:
- Production service: Yes (api-gateway)
- Users affected: 5,000
- Revenue impact: $500
- SLA breached: No (150ms < 200ms threshold)
- Recent deployment: Yes

Rules:
- If CPU spike post-deployment + within SLA → High severity
- If revenue impact <$1k + user impact <10k → High

Prediction: High severity (78% confidence)
Reasoning: Unexpected CPU spike, affecting many users, but within SLA and expected post-deployment

Step 8: Notification
Confidence 78% → Page team lead (not full incident commander)
Notification: "api-gateway latency elevated post-deployment. 5k users affected. Investigating."

Step 9: Incident Ticket
Ticket created with pre-populated data:
- Title: "api-gateway latency spike post-deployment v2.3.1"
- Severity: High
- Affected users: 5,000
- Affected services: api-gateway, payment-service, checkout-service (dependent)
- Business impact: $500 in failed transactions
- Recent changes: Deployment api-gateway v2.3.1 30 min ago
- Suggested actions:
1. Check api-gateway logs for errors
2. Compare metrics before/after deployment
3. Check dependent services for cascading failures
4. If critical, rollback deployment
- Estimated MTTR: 15 minutes (based on similar incidents)

On-call team sees ticket + page notification.
Engineer checks ticket, sees pre-populated context.
Instead of 15 minutes of investigation, they have context in 30 seconds.
MTTR: 45 minutes (before) → 12 minutes (after)

Feedback loop:
What happened? Team rolled back api-gateway to v2.0.
Was severity correct? Yes, customers were affected.
Did we catch it quickly enough? Yes, 2 minutes to notification.
Learn: api-gateway deployments are high-risk. Increase monitoring during rollouts.
Next deployment, lower confidence threshold (alert at >75% instead of >85%).

Use Case 2: Self-Healing for Known Issues

You have 10 known issues that fire alerts regularly. Each one has a known fix.

Examples:

• Cache cluster drops a node every Friday during backup. Alert fires. Engineer logs in, adds node back. Takes 5 minutes.
• Database connection pool exhaustion every quarter-end. Alert fires. Engineer increases pool size. Takes 10 minutes.
• Batch job timeout during heavy load. Alert fires. Engineer extends timeout. Takes 2 minutes (config change).

Detection pipeline with self-healing:

Known Issue: Cache node drop during backup

Alert fires: "cache-cluster: 2 nodes down, peer replication lag >500ms"

Detection pipeline:
Step 1-4: Ingest, enrich, deduplicate, correlate (standard)

Step 5: Anomaly Detection
Time: Friday 2:00 AM (backup window)
Historical data: Cache node drops every Friday 2-4 AM
Pattern match: 95% confidence this is backup-related node drop
Verdict: Expected anomaly, not a real incident

Step 6: Business Impact
Impact: Replication lag, but cache has redundancy
User impact: Minimal (cache hit rate stays high, some requests slower)
Revenue impact: None

Step 7: Severity
Pattern match: Known issue + expected + minimal impact
Classification: Informational (not even Low severity)

Step 8: Action (Self-Healing)
This is a known issue. Trigger auto-remediation:
1. Check: Is this backup-related node drop? (Yes, 95% confidence)
2. Execute: Add dropped node back to cluster
3. Verify: Replication lag drops below 100ms? (Check in 30 seconds)
4. Report: "Cache node auto-recovered. No user impact."

Step 9: Incident Ticket
Not created (no incident, just normal operation)
Log entry: "2024-05-17 02:00 - Cache node drop (auto-recovered, backup-related)"

Feedback loop: Every occurrence feeds data. If drop happens at different time or impacts users differently, update the pattern.

Result: Friday cache node drops used to trigger pages. Now they auto-remediate with zero human intervention.

Use Case 3: Intelligent Escalation for Ambiguous Incidents

Some incidents are clearly Critical. Some are clearly Low. Some are ambiguous.

Example: Ambiguous incident

Alert: "Database query latency P95: 500ms (normally 50ms)"

Is this Critical?

• Not a total outage (queries still running)
• Queries slow (users might notice)
• Could escalate (if slowdown continues, queries timeout)

Severity ambiguity: Could be High or Medium depending on context.

Detection pipeline:

Alert: Database latency spike

Enrichment:
- Database is critical for checkout flow
- Affects 100k users
- SLA is 200ms response time

Anomaly Detection:
- Latency usually 50ms
- Spike to 500ms = 90th percentile (extreme)
- Anomaly score: 9/10 (very anomalous)

Business Impact:
- Is checkout path blocked? No, still responding <200ms SLA
- Are users affected? Some might see slowness, but not failures
- Revenue at risk: None immediate, but risk if slowdown worsens

Severity Classification:
- Multiple signals
- Strong anomaly (9/10) but within SLA
- Pattern doesn't clearly map to Critical or High

AI Confidence: 65% (too low to auto-escalate)

Decision:
Create High-priority ticket + notify incident commander (not full page)
Incident commander reviews: "This is unusual. Let me check what's happening."

Incident commander sees:
- Latency spike timeline
- Suggested investigation: Check query logs, check for missing indexes
- Similar incidents from history

Incident commander assesses: "This could be a slow query or missing index. Keep watching."
- Set up auto-remediation: If latency doesn't improve in 5 min, escalate to database team
- Set monitoring: If latency worsens to >1s, auto-page DBA

Results:
- Incident commander used contextual judgment to escalate appropriately
- AI gave context upfront (no 15-minute investigation needed)
- MTTR: 25 minutes (incident commander decision + DBA investigation)

Feedback loop:
Root cause: Missing index on newly added query
Learning: Queries using newly added columns → higher risk
Next time, watch for missing indexes post-deployment
Increase anomaly threshold if recent schema change

Anti-Patterns

Anti-Pattern 1: "Alert on everything, let humans filter"

You monitor 50 microservices. You set up alerts for everything: CPU, memory, disk, error rate, latency, queue depth. That's 250+ metrics.

You fire 100+ alerts per day. Your on-call team ignores 95% of them. When a real incident happens, it's lost in the noise.

Why it fails: Alert fatigue is a real cognitive issue. After the 100th false positive, humans stop responding to alerts. You've broken your detection system.

How to avoid it: Invest in correlation, deduplication, and false-positive elimination. Alert on 15 real incidents, not 100 false ones.

Anti-Pattern 2: "We'll tune thresholds once, then we're done"

You set CPU threshold at 80%. It works fine for 6 months. Then you deploy a feature that legitimately uses more CPU. Now every day during business hours, the threshold fires false positives.

You don't update it because "it worked before," and you forget about it. Or you disable alerting because it's too noisy.

Why it fails: Systems change. Your baseline expectations change. Static thresholds don't adapt.

How to avoid it: Use dynamic thresholds that adapt to your system's behavior. Or re-tune quarterly. Review alert history: what % of alerts are false positives? If it's >20%, it's time to retune.

Anti-Pattern 3: "Severity classification is manual"

Each incident that comes in, a human manually assigns severity. "Is this Critical or High? Let me think... I'll say High."

Different people assign different severities to the same incident. One engineer says "Critical," another says "High." No consistency.

Why it fails: Without consistent severity, you can't prioritize. You don't know which incidents matter most. And manual assignment takes time that AI could automate.

How to avoid it: Build severity classification rules from your incident history. "Critical" incidents are the ones you responded to immediately. "High" incidents are the ones you handled in the next hour. Use these definitions to build rules that AI can apply consistently.

Anti-Pattern 4: "We have no feedback loop"

You deploy incident detection AI. It runs for 6 months. You don't track what AI predicted vs. what humans decided. You don't measure how often AI was right or wrong.

After 6 months, someone asks: "Is the AI system working?" You have no idea. You assume it's fine because nobody complained loudly.

Why it fails: Without feedback, you can't improve. Without measurement, you can't detect degradation. The AI might be getting worse over time, and you wouldn't know.

How to avoid it: Capture feedback at every incident. Log AI prediction, human decision, actual outcome. Review monthly. Adjust thresholds or retraining based on what you learn.

Anti-Pattern 5: "Dependency mapping is too hard, skip it"

Correlation and dependency mapping seem complicated. You think, "We'll start simple. Just alert on direct metrics."

But your payment service depends on the database and cache. When the database slows, payment service slows. When cache drops, payment service fails. If you alert on all three separately, you create 3 incidents for 1 problem.

Why it fails: Without dependency mapping, you get alert storms. You lose signal in noise.

How to avoid it: Start with the critical path. Map what services matter for your core business (payments, authentication, search). For those services, build dependency maps. Your correlation rules might be simple at first (5 rules), but they're better than zero.

Human Judgment Checkpoints

• 
  Confidence threshold review: Have you set thresholds that match when to page vs. create tickets?
• 
  False positive analysis: Review your last 30 alerts. What % were false positives? If >20%, thresholds need tuning.
• 
  Severity consistency: Do your incident severities make sense? Would your team respond to Critical incidents differently than High? If not, your classifications are wrong.
• 
  Correlation coverage: What are your critical dependencies? Have you mapped them for correlation?
• 
  Feedback loop review: Can you show the data on how often AI severity predictions matched actual severity? If not, set up the logging.

Key Takeaways

Build a multi-step detection pipeline, not just a model. Enrichment, deduplication, correlation, and anomaly detection each filter out false positives. Together, they reduce noise dramatically.

Dynamic thresholds beat static thresholds. Account for time of day, day of week, recent deployments, and business cycles. A static threshold configured 2 years ago is probably wrong now.

Correlation and dependency mapping eliminate alert storms. When database goes down, 10 downstream services fire alerts. That's 1 incident, not 10. Map dependencies.

Build severity classification from incident history. What incidents did you respond to immediately? Critical. What incidents did you handle during business hours? High. Use incident history to define rules.

Confidence thresholds drive notification decisions. High confidence (>90%) = auto-page. Medium confidence (70-85%) = notify incident commander. Low confidence (<70%) = create ticket. Set thresholds that match your risk tolerance.

Capture feedback loops. Log AI prediction, human decision, and actual outcome. Review monthly. Use feedback to improve thresholds, retrain models, and adapt rules.

Suppress known false positives explicitly. Backup windows, maintenance periods, and expected load spikes should not fire incidents. Build suppression rules.

Start with critical path. Don't try to correlation-map all 50 services at once. Start with payment, authentication, and core services. Expand incrementally.