Overview

Your AI system is in production. It's working great. Then, suddenly, customers start complaining. "Recommendations are terrible." "Accuracy is way down."

You investigate. The model is fine. The code is fine. But somehow, quality degraded.

What happened? Data drift. The distribution of input data changed. The model was trained on historical user behavior. Now users behave differently (seasonality, new feature adoption, market change). The model still outputs predictions, but they're calibrated for the old distribution. Accuracy plummets.

Or maybe the upstream system changed. The data pipeline that feeds your model broke. A schema changed. A field that was supposed to be numeric is now null 50% of the time. Now you're getting junk input. Output is junk. Customers don't see errors. They see bad recommendations.

These problems are invisible until customers tell you about them. You wish you'd caught them before they affected users. You could have if you were monitoring the right metrics.

This is why monitoring AI systems is critical. You can't just monitor like traditional software (CPU, memory, latency). You have to monitor AI-specific things: data quality, model accuracy, data drift, model confidence, business metrics. This lecture is about what to monitor, how to monitor it, and how to alert before customers notice problems.

The Three Layers of AI Monitoring

Layer 1: System Monitoring

Traditional monitoring. Is the service up? Latency? Error rate? CPU? Memory?

If your AI service goes down, that's bad. But if it's up and giving bad results, that's worse.

Layer 2: Data Monitoring

Is the input data what you expect? Distribution, format, values.

Data drift: input distribution changes. Model was trained on old distribution, fails on new.

Data quality: are there nulls? Are values in expected ranges? Is required data missing?

Monitoring data is critical because garbage in = garbage out.

Layer 3: Model Monitoring

Is the model performing as expected? Accuracy. Precision. Recall. Confidence.

Model degradation: accuracy decreases over time. Why? Data drift? Something changed? Model needs retraining?

Monitoring model is critical because models degrade silently if you don't watch them.

Key insight: A system can be up (layer 1), get good data (layer 2), but still give bad results (layer 3). Monitor all three layers.

Specific Metrics to Monitor

Data Quality Metrics

Null/Missing: What percentage of fields are missing? If critical field is missing, everything downstream fails.

Data types: Are values the right types? Ages should be numbers 0-120. Emails should match email pattern.

Value ranges: Are values in expected ranges? Temperature 0-50C? Revenue -0 to billions? Flag outliers.

Distributions: Does the distribution match historical patterns? If usually 60/40 male/female but suddenly 90/10, something changed.

Freshness: Is data current? If yesterday's data is still showing up today, your pipeline is slow or broken.

Model Performance Metrics

Accuracy: Percentage correct. If accuracy drops 10% in a day, something is wrong.

Precision/Recall: For classification, are you getting false positives or false negatives? Different is worse for different applications.

Confidence: Models give confidence scores. Are scores still calibrated? If model says 95% confident but is actually 60% accurate, confidence is miscalibrated.

Latency: How long to generate prediction? If latency doubles, users notice.

Cost: Cost per prediction. If costs spike, something is wrong (extra tokens? slower model?)

Drift Detection

Data drift: Input distribution changes. Detect statistical differences between historical data and recent data.

Prediction drift: What your model predicts changes. Could indicate data drift. Could indicate model degradation.

Concept drift: What you're actually trying to predict changes. Example: fraud detection. What constitutes fraud changes. Model needs retraining.

Business Metrics

Track business impact. If AI is supposed to increase sales, track sales. If supposed to reduce support costs, track support costs.

Some models can have perfect accuracy but still fail business metrics because they're answering the wrong question.

Alerting Strategy

Alert Thresholds

Set thresholds. If accuracy drops below 85%, alert. If data quality score drops below 90%, alert. If latency exceeds 1 second, alert.

Make sure thresholds are meaningful. Not so sensitive you get alert storm. Not so loose you miss real problems.

Alert Prioritization

Critical: Model is completely broken. Zero accuracy. All nulls. Service down. Needs immediate response.

High: Model accuracy down 20%. Data quality degraded. Respond within an hour.

Medium: Model accuracy down 5%. Minor data quality issues. Investigate within a day.

Low: Informational. Latency slightly up. Cost slightly higher. Note for future reference.

Escalation

Who gets alerted? On-call ML engineer? Manager? CEO? Define escalation paths.

Critical: immediate alert to on-call + manager.

High: alert to on-call, escalate to manager if not resolved within an hour.

Medium/Low: log for team to review later.

Root Cause Diagnosis

When something goes wrong, you need to figure out why quickly.

Check data: Is input data what you expect? Check distributions. Check for nulls. Check freshness.

Check model: Is the model the one you think it is? Did it get updated recently? Is there a versioning issue?

Check code: Did deployment introduce a bug? Is there a configuration issue?

Check upstream: Did the system that feeds your data change? Is it still working?

Good monitoring makes diagnosis faster. If you're already tracking all these metrics, you know what changed and when.

What to Do Monday Morning

• Identify what matters: What metrics would tell you if your AI system is working? Accuracy? User satisfaction? Business metrics? For each metric, define: what's normal? what's concerning? what's critical?
  - Set baselines: What are normal values for these metrics? Use historical data to establish baselines. For accuracy: what was last month's accuracy? Is it stable or trending?
  - Implement monitoring: Start tracking these metrics. Logging (store predictions + actual outcomes), dashboards (visualize trends), alerts (notify when thresholds broken).
  - Set alert thresholds: At what value should you get alerted? Make sure thresholds are meaningful, not noise. Example: alert if accuracy drops > 5% from baseline.
  - Test your alerts: Simulate failures. Inject bad data. Make sure alerts fire. Make sure the right people get notified. Make sure runbooks are clear.
  - Create a runbook: When you get an alert, what's the procedure? "Accuracy dropped, now what?" Steps: (1) check if it's a false positive, (2) check data quality, (3) check model version, (4) escalate if needed.
  - Plan for degradation: What's your fallback when the model fails? Can you use the old model? Can you return a default answer? Can you route to a human? Have a plan.

Real-World Monitoring Case Studies

Case Study 1: Recommendation Engine at E-commerce Company

An online retailer with $50M revenue built a recommendation engine using AI. They monitored accuracy (precision of top-10 recommendations). Baseline: 72% of top-10 recommendations resulted in purchases. Six months in, they noticed accuracy dropping: 68%, then 65%. They didn't catch it immediately. Customers started complaining about recommendation quality. Revenue impact: 3-4% drop in conversion rate on recommendations ($1.5M/month). Root cause: user behavior shifted during market downturn. Wealthy customers stopped buying premium items (which the model was trained on). The model was still predicting premium items for everyone. Fix: retrain on recent data (two weeks). New accuracy: 71% within a week. Lesson: they should have alerted at 70% (5% drop threshold), not waited for customer complaints. Cost of delay: $6M in lost revenue over one month.

Case Study 2: Classification Model at SaaS Company

A customer support AI classified incoming tickets as urgent/non-urgent. Accuracy baseline: 88%. Three months in production, accuracy silently dropped to 79% (undetected). Why? A new competitor had launched. Customer complaints had different patterns. The model was trained on historical data from when they had different competitors. Impact: 9% of tickets were misclassified, leading to slow response times on actual urgent issues. Customers noticed support got slower. They didn't know why. Root cause: data drift in input distribution (customers now complained about different features). Fix: retraining took one week. New accuracy: 86%. Lesson: they monitored system metrics (uptime, latency) but not data quality metrics (input distribution) or model accuracy. Three months of degraded service before detection.

Key Lesson: The worst AI failures are silent. The system is still running. It's giving bad results. Customers notice before you do. Monitor early. Alert often. Catch degradation before customers complain.

When This Goes Wrong: Monitoring Failures

Failure Mode 1: You set thresholds too loose. You alert when accuracy drops >20%. Accuracy gradually drops 15% but never exceeds your threshold. Customer experience degrades. You never get alerted. Lesson: set thresholds based on business impact, not "reasonableness." If 5% accuracy drop costs you $100k, alert at 5%.

Failure Mode 2: You don't monitor data quality, only model accuracy. You track accuracy. You don't track input data distribution. Bad data arrives (missing fields, wrong values). Model still produces confident predictions. Accuracy appears stable (because it's still predicting on bad data consistently). Eventually, customers report incorrect results. Lesson: monitor data quality as your first line of defense. Bad data is the root cause of 70% of model failures.

Failure Mode 3: You alert on everything and get alert fatigue. You set 50 different alerts. 40 of them are noise (normal variation). Everyone ignores alerts. When a real critical issue happens, the alert is lost in the noise. Lesson: alert ruthlessly on critical issues. Make most things observable (visible on dashboards) but only alert when action is needed.

Failure Mode 4: You don't have a runbook, so incident response is chaos. An alert fires. Your team investigates for 2 hours. Should we retrain? Should we rollback? Should we increase thresholds? No clear answer. Finally someone decides to rollback. But rollback takes another hour. Total incident time: 3 hours. Service was degraded the whole time. Lesson: write runbooks before you need them. "If accuracy alert fires: (1) Check data quality, (2) Check if data drift detected, (3) If both OK, retrain model, (4) If retraining takes >2 hours, rollback to previous version."

FAQ: Monitoring Questions

Q: How often should I retrain the model?

A: Monitor data drift. When drift magnitude exceeds threshold (usually 0.15-0.25 Wasserstein distance or equivalent), retrain. Could be daily, weekly, or monthly depending on your data stability. Some models never need retraining. Don't retrain just on schedule if data hasn't changed.

Q: What if I detect a problem but can't fix it immediately?

A: Have a degradation plan. Can you use the old model? Can you return a default answer? Can you route to human review? Can you reduce confidence threshold and only show predictions you're >95% certain about? Multiple fallbacks are better than one.

Q: What's acceptable accuracy drift?

A: Depends entirely on your application and business impact. For a recommendation engine where 1% accuracy drop = $50k/day revenue loss, alert at 1%. For an internal analytics tool, maybe 10% is acceptable. Calculate the business impact of accuracy loss. That's your alert threshold.

Q: How do I detect data drift if my data distribution is supposed to change seasonally?

A: Expected changes vs unexpected changes. Seasonality is expected, track it explicitly. "December has 3x holiday gift purchases" is expected. Alert only when data looks different from seasonal baseline. Sudden spike outside seasonal pattern? Alert. Use statistical tests (Kolmogorov-Smirnov test, chi-square test) to detect unexpected drift while tolerating expected seasonal variation.

Q: This sounds expensive. Do we really need all this monitoring?

A: Yes, but start simple. Start with: accuracy (one metric that matters most to you), latency, and one data quality metric (% nulls, distribution shift). That's three metrics and probably 2-4 hours of setup. ROI on that setup is usually positive within the first month when it catches your first problem.

Q: What if we have multiple models in production?

A: Monitor each separately. Track which model version is active. Log results by model. When you upgrade a model, run both models in shadow mode for a week (both make predictions, only old one is used). Compare metrics. Only switch when new model is clearly better. This prevents "upgrade went wrong" incidents.

Q: How do we monitor models we didn't build (third-party APIs)?

A: You still monitor output quality. You can't see their internal metrics, but you can see if their outputs are accurate for you. Log every API call and result. Sample results for accuracy. Track latency, cost, error rate. You can't monitor their model performance, but you can monitor impact on your system.

The Monitoring Operations Playbook

This is what world-class teams actually do:

Week 1: Baseline Deploy model. Log everything for a week (without alerting). Understand normal behavior. What does accuracy look like? Data distribution? Latency? Errors?

Week 2: Alert Setup Based on baselines, set alert thresholds. Conservative at first (alert on 15% degradation, not 1%). Test alerting. Make sure right people get notified.

Week 3-4: Monitor Observe. Adjust thresholds based on false positive rate. If you're getting 5+ alerts per day that don't warrant action, thresholds are too loose. If you're missing obvious problems, they're too tight.

Month 2+: Iteration Real data comes in. Update monitoring based on what matters. Maybe you learn that latency variations don't matter but accuracy variations do. Adapt.

Quarterly: Review Spend an hour reviewing your monitoring setup. Which alerts are actually valuable? Which get ignored? Which problems are you still catching too late? Update for next quarter.

Cost: ~40 hours of engineering time per deployed model in first month, then 2-3 hours/month for maintenance. Payoff: catching problems before customers do, understanding model behavior, making informed decisions about retraining/updates.

Key Insight

AI systems degrade silently. Data changes. Models lose accuracy. Customers notice before you do. Monitor at three levels: system (latency, errors), data (quality, drift), and model (accuracy, confidence). Set meaningful thresholds. Alert on problems. Diagnose quickly. This is how you catch problems before customers do.

On This Page

Introduction
Three Layers
Specific Metrics
Alerting Strategy
Root Cause Diagnosis
Case Studies
When This Goes Wrong
Monday Morning Action
FAQ

Chapter Details

Part ofChapter 5**