Overview

You've trained a model. It performs well. You deploy it to production. It's working great.

Two years later, the model is still in production. But nobody knows when it was trained. Nobody knows who trained it. Nobody knows the test accuracy. Nobody knows why it was chosen over alternatives.

When something breaks, you can't quickly revert to a known-good version because you don't have version control on models.

When you want to improve it, you don't know what experiments have already been tried.

When you need to explain a prediction for compliance, you can't because you have no provenance.

This is the model lifecycle problem. Most organizations treat models like ad-hoc scripts. No governance. No versioning. No audit trail.

The solution is to manage models like any other critical asset. Version them. Track them. Audit them. Retire them properly.

This lecture is about model lifecycle management. From creation to retirement.

The Model Lifecycle

Phase 1: Experimentation

Data scientist has a hypothesis. "This model architecture will improve accuracy." They experiment. Train different versions. Track results.

This phase is messy. Many experiments. Most fail. Some succeed.

You need experiment tracking. What hyperparameters? What data? What accuracy? What precision/recall? Able to reproduce results.

Use a tool like MLflow, Weights & Biases, or Neptune. Log every experiment.

Phase 2: Development

Best experiment from phase 1 becomes the development candidate. You run more rigorous tests. Integration tests. Load tests. Fairness tests.

You get stakeholder sign-off. Product manager. Security team. Legal team (for fairness/bias).

You build the model registry. Official version of the model. Training script. Data used. Test metrics. Requirements for deployment.

Phase 3: Staging

Model is deployed to staging environment. You run parallel tests. New model vs. old model. Same data. Compare results.

You run load testing. How much traffic can it handle?

You run stress testing. What happens when data is bad? When traffic spikes?

You get operational sign-off. Ops team confirms they can deploy and monitor this model.

Phase 4: Production

Model is deployed to production. Initially to a small percentage of traffic (canary). If metrics are good, expand gradually.

You monitor everything. Accuracy. Latency. Cost. Error rates. Business metrics.

You compare to baseline. Is the new model better than the old one?

Phase 5: Maintenance

Model is in production. You monitor it. If accuracy degrades (data drift), you retrain.

You collect feedback. Users reporting issues? Specific input types where accuracy is bad?

You plan improvements. Next version should address these issues.

Phase 6: Retirement

Eventually, the model is no longer needed. A newer model replaced it. A rule-based system replaced it. The product was discontinued.

You retire the model properly. Archive data. Archive training artifacts. Document why it was retired. Save for audit trail.

Key principle: Every model should have complete provenance. Who trained it? When? Why? What data? What accuracy? All documented and searchable.

Model Versioning and Registry

Versioning Strategy

Models need semantic versioning. Major.Minor.Patch.

Major: significant change in behavior (new architecture, new data distribution).

Minor: incremental improvement (hyperparameter tuning, more data).

Patch: bug fix, no behavior change.

Example: v2.3.1 is the third patch to the second major version, third minor release.

Model Registry

Central place to store and manage models. Every model in the registry has:

• Name and version
• Training date
• Training data version
• Hyperparameters
• Test metrics (accuracy, precision, recall, AUC, etc.)
• Training code version
• Deployment history
• Status (development, staging, production, retired)

Use MLflow Model Registry or similar tool. Keep models organized and discoverable.

Model Promotion

Models move through stages. Development → Staging → Production.

Each stage transition requires approval. Security review. Performance review. Operational review.

Approval is documented. Who approved? When? Why?

Rollback Strategy

If production model breaks, you need to quickly rollback to previous version.

Keep at least the previous three versions available for quick rollback.

Rollback should be automatic for critical failures. Manual decision for minor issues.

Model Governance and Compliance

Fairness and Bias

Before a model goes to production, ensure it's fair. Not discriminating against protected groups.

Test for bias. Split data by protected characteristics (gender, race, age). Do model metrics vary significantly?

If bias is detected, investigate. Is it in the data? Is it in the model? Fix before production.

Explainability

For regulated domains, models need to be explainable. Why did the model make that prediction?

Use explainability tools. SHAP, LIME, etc. Be able to generate explanations for predictions.

Documentation

Every model needs documentation. What problem does it solve? What data does it use? What are the limitations?

Who is the owner? Who do you contact with questions?

What are the training requirements? How often should it be retrained?

Create a model card (internal documentation) for every model.

Audit Trail

Every model needs an audit trail. When was it trained? Who trained it? What data? What accuracy?

When was it deployed? Who approved? When did it change?

This is critical for compliance and debugging.

Access Control

Who can train models? Who can deploy models? Who can retire models?

Define roles and permissions. Data scientists can train. ML engineers can deploy to staging. Directors can approve production deployment.

Operational Considerations

Retraining

Models degrade over time. Data distribution changes. Accuracy drops. When should you retrain?

Monitor data drift. When drift exceeds threshold, retrain.

Monitor model accuracy. When accuracy drops below threshold, retrain.

Have a retraining pipeline. Automated or semi-automated. New data flows in. Model is retrained. New version is tested. If passes tests, deployed.

Monitoring and Alerting

In production, monitor the model continuously. Accuracy. Latency. Cost. Input data quality.

Alert on degradation. If accuracy drops 10%, investigate. If latency doubles, investigate.

Cost Management

Models have costs. Inference cost (API calls). Compute cost (retraining). Storage cost (model artifacts, data).

Track costs per model. If a model is expensive, can you optimize it? Use a cheaper model? Cache results?

Scaling

Can the model scale to production traffic? If millions of inferences per day, does the system handle it?

Model optimization: quantization, distillation, pruning. Serve more efficiently.

Infrastructure optimization: batch processing, caching, CDN, load balancing.

What to Do Monday Morning

Audit your current models: What models do you have in production? Who trained them? When? What data? Document it.

Set up experiment tracking: If you don't have one, use MLflow or similar. Start logging experiments. Hyperparameters. Metrics. Code versions.

Create a model registry: Centralized place for all models. Every model registered. Versioned. Documented.

Define governance: Who can train? Who can deploy? What approvals are needed? Document it.

Set up monitoring: Track model accuracy in production. Set alerts for degradation.

Document procedures: How to train a new model? How to deploy? How to rollback? Write runbooks.

Lifecycle Management Case Studies

Case Study 1: Fraud Detection Model at Payments Company

A payments processor had one fraud model in production. No versioning. No tracking. When something broke, they had no way to rollback. Incident: fraud model suddenly started flagging legitimate transactions. False positive rate jumped from 0.1% to 5%. They couldn't quickly rollback because they had no version control. Resolution: retraining took 3 days. During those 3 days, 0.5% of legitimate transactions were rejected. Revenue impact: $200k. Cost per day of incident: $67k.

Solution: implement model registry. Track every model. Keep last 5 versions available. Now when something breaks, they can rollback to previous version in 5 minutes. Same incident 6 months later: rollback to v3.2, incident resolved in 6 minutes instead of 3 days. Saved: $200k. Setup cost: 80 hours engineering. ROI: 2.5x in one incident.

Case Study 2: Recommendation Model at E-commerce (Scaling Problem)

They had one recommendation model handling 10M requests/day. Model was accurate but slow (200ms per inference). Business needed faster responses (50ms requirement for real-time) to compete. Options: (1) buy faster model, (2) optimize current model, (3) build new model.

They chose option 3 + 2: trained a smaller, faster model (50ms latency, 89% accuracy vs current 92%). Ran canary test: 10% of traffic to new model. Business impact: conversions from fast recommendations were 2% higher (users prefer fast+good to slow+perfect). Full rollout: 50M/day requests through new model. Accuracy regression offset by latency gains. New model now baseline.

Lesson: systematic model management allowed them to iterate. They could A/B test. They had versions to rollback to. This speed of iteration created competitive advantage.

Lifecycle Principle: Model lifecycle isn't just about compliance or safety. It's about enabling fast iteration. Version control lets you experiment, rollback, and optimize. Without it, you're slow. With it, you're fast.

When Lifecycle Management Fails

Failure Mode 1: You don't track what you deployed.** You deployed v1.3 last month. Accuracy degraded. You want to rollback. But you don't know what version it was. Lesson: every deployment should be in a registry with timestamp, who deployed, which version.

Failure Mode 2: You can't quickly rollback because old versions are gone.** You deployed v2.0. It's bad. You want v1.9. But you deleted it. You have to retrain v1.9 (takes 1 week) instead of restoring from archive (takes 5 minutes). Lesson: keep last 5+ versions available for quick rollback.

Failure Mode 3: You have 100 models but don't track them.** CEO asks "how many models do we have in production?" You don't know. Someone has a model running in production that nobody knows about. It breaks. It takes 3 days to find who owns it. Lesson: model registry + audit. You must know what you have.

Failure Mode 4: You retrain constantly and never reach stable state.** New data arrives daily. You retrain daily. New version always has minor differences from previous. Accuracy drifts. You can't tell if it's improvement or noise. Lesson: set retraining thresholds. Don't retrain daily. Retrain when drift magnitude exceeds threshold (e.g., Wasserstein distance > 0.2).

FAQ: Model Lifecycle Questions

Q: How often should I retrain a model?

A: Not on schedule. On signal. Monitor data drift. When drift magnitude exceeds threshold (0.15-0.25 statistical distance), retrain. Could be daily, weekly, monthly, or never. Depends on your data. Don't retrain just because the calendar says it's retraining day.

Q: Can I automate the entire lifecycle?

A: Mostly, but not completely. Automation: training (yes), evaluation (yes), deployment to staging (yes), monitoring (yes), rollback (yes). Manual: deployment to production (human decision), retirement (human decision), handling edge cases (human judgment). You can automate 80% of the workflow.

Q: What if I have hundreds of models?

A: Use a model registry (MLflow, Weights & Biases, etc.). You can manage hundreds of models if they're organized, versioned, and monitored. Without a registry, you can't manage more than 5-10.

Q: How long should I keep old models?

A: For compliance: at least 1-2 years. For business: keep last 5 versions available for rollback. Archive older versions for compliance/audit but don't need them in production. After your retention period (usually 2-3 years), delete to save storage costs.

Q: What if we're not sure if a new model is better than the old one?**

A: A/B test. Deploy new to 5-10% of traffic. Compare key metrics (accuracy, latency, business impact). If better, increase to 50%, then 100%. If worse, rollback. This is why versioning matters. You can rollback instantly if something is wrong.

Key Insight

Treat models like critical infrastructure. Version them. Track them. Govern them. Monitor them in production. Retrain when they degrade. Retire when no longer needed. This gives you the ability to quickly understand what models you have, what they do, why they were deployed, and how they're performing. This is how large organizations manage AI systems reliably.

On This Page

Introduction
Lifecycle Phases
Versioning and Registry
Governance and Compliance
Operational Considerations
Case Studies
Failure Modes
Monday Morning Action
FAQ

Chapter Details

Part ofChapter 5