Hook

You've built a good model. It works great in the lab. Now you need to deploy it to production so the business can use it.

You spin up an API server and expose the model. One internal team starts using it. Then another team wants to use it. Then your CEO asks why inference is slow. You check and realize one team is sending 1,000 requests per minute, and they've added 500ms latency to everyone else's requests. You don't know which team. You have no rate limiting. You have no quota management. You have no way to bill them.

Now you have multiple models in production, and they sometimes need to work together. The recommendation model calls the ranking model calls the personalization model. If one is slow, all are slow. You have no way to route traffic intelligently or handle failures gracefully.

This lesson is about building an internal AI platform, treating AI services like the infrastructure they are, not like a pet project. You'll learn API management, rate limiting, cost allocation, multi-model orchestration, and the patterns that let you scale AI to serve the whole organization.

Purpose

As you go from "we have one model in production" to "we have 20 models serving the organization," you need to think like a platform engineer, not a data scientist.

Specifically, you need:

• API Gateways, A single entry point for AI services, with rate limiting, authentication, monitoring
  - Cost Attribution, Know which team is using which model, and charge them for it
  - Service Discovery, Know where models live, which version is running, how to call them
  - Orchestration, Chain multiple models together, route requests intelligently, handle failures
  - Monitoring, See request volumes, latency, error rates, model performance

Without these, AI becomes a bottleneck. With them, it becomes an internal utility that enables the whole organization.

Why This Matters

Most organizations treat AI services as standalone projects. Each model is deployed separately, exposed via REST API, and operated independently.

This creates three problems:

First, cost becomes invisible. Teams don't know what they're paying for AI infrastructure. They over-use because there's no cost signal. The AI team can't justify infrastructure spending to finance.

Second, performance degrades. One team uses 80% of GPU capacity, making models slow for everyone else. Without rate limiting, you have no way to enforce fair sharing.

Third, reliability suffers. If one model depends on another, and the second model goes down, the first fails too. Without intelligent error handling and fallbacks, cascading failures happen.

Organizations that succeed treat AI as infrastructure:

• Expose services via consistent APIs
• Rate-limit and quota consumers
• Monitor usage and cost
• Route intelligently
• Handle failures gracefully

This turns AI from a bottleneck into an enabler.

Core Concepts

Key Insight: The API Gateway Pattern

An API gateway is a single entry point for all AI services. Instead of teams calling models directly, they call the gateway, which routes to the appropriate model.

Benefits:

• Single authentication point, Verify credentials once, grant access to multiple models
• Rate limiting, Enforce quota per team or per user
• Monitoring, See all traffic in one place
• Versioning, Route requests to different model versions based on rules
• Caching, Cache model predictions to reduce redundant computation

Example flow:

Request: GET /api/v1/churn-prediction?customer_id=123
↓
API Gateway (authenticate, rate-limit, route)
↓
Churn Prediction Service v1.2.3
↓
Response: {churn_probability: 0.73, confidence: 0.95}

Tools: Kong, AWS API Gateway, Azure API Management, Nginx

Key Insight: Rate Limiting and Quota Management

Rate limiting prevents one team from monopolizing resources. It enforces fair sharing.

Three patterns:

Pattern 1: Per-Team Rate Limiting

Each team has a quota (e.g., 1,000 requests/minute).

Implementation:

Team A: 1,000 req/min
Team B: 500 req/min
Team C: 250 req/min

If Team A exceeds 1,000 req/min, subsequent requests are rejected or queued.

Benefits:

• Fair sharing (everyone has guaranteed access)
• Cost attribution (easy to bill teams)
• Prevents runaway consumption

Pattern 2: Per-Model Rate Limiting

Each model has a quota (e.g., churn model can handle 500 req/min).

Implementation:

Churn Prediction: 500 req/min
Recommendation: 1,000 req/min
Fraud Detection: 2,000 req/min

Benefits:

• Prevents overloading a single model
• Scales quota based on model infrastructure
• Predictable performance

Pattern 3: Priority-Based Rate Limiting

Different consumers have different priorities.

Implementation:

High priority (batch): unlimited, but queued
Normal priority (real-time): 1,000 req/min
Low priority (experimental): 100 req/min

Benefits:

• Batch jobs don't interfere with real-time
• Experimental work doesn't block production
• Flexibility for different use cases

Key Insight: Model Versioning and Routing

As you improve models, you need to deploy new versions without breaking existing consumers.

Canary Deployment

Deploy new model to 5% of traffic, monitor, gradually increase.

Flow:

Route 95% → Model v1.0
Route 5% → Model v1.1 (new)

Monitor:
- Model v1.1 latency acceptable?
- Model v1.1 accuracy same as v1.0?
- Any errors?

If yes: gradually increase to v1.1
If no: rollback to v1.0

Benefits:

• Detect issues on small subset before rolling out
• Rollback quickly if needed
• Zero downtime deployment

Consumer-Specified Versioning

Let consumers specify which version they want.

Flow:

Request: GET /api/v1/churn-prediction?customer_id=123&version=v1.1

Benefits:

• Consumers can test before upgrading
• Gradual migration path

A/B Testing

Route some users to old model, some to new, compare performance.

Flow:

User group A (control): Model v1.0
User group B (treatment): Model v1.1

Compare:
- Which version performs better?
- Which has better business outcome?

Key Insight: Orchestration and Chaining

Often, models work together. Recommendation → Ranking → Personalization.

Problem: If you call them sequentially, latency multiplies.

• Recommendation takes 100ms
• Ranking takes 80ms
• Personalization takes 60ms
• Total: 240ms (slow for real-time)

Solution 1: Parallel Execution

Run non-dependent models in parallel.

├─ Recommendation model (100ms)
├─ User context enrichment (50ms)
└─ Personalization model (60ms, depends on context)

Total latency: max(100, 50) + 60 = 160ms (faster)

Solution 2: Model Chaining

Some models feed into others. Orchestration manages the flow.

Flow:

Input: user_id
↓
User Embedding Model (50ms)
↓ (uses embeddings)
Recommendation Model (100ms)
↓ (uses recommendations)
Ranking Model (80ms)
↓
Output: ranked recommendations

Tools: Seldon, KServe, BentoML for model serving and orchestration

Key Insight: Cost Attribution and Chargeback

Who pays for AI infrastructure?

Options:

Option 1: Centralized Cost

• AI infrastructure is central IT cost
• No chargeback to teams
• Problem: Teams have no cost signal, over-consume

Option 2: Per-Team Chargeback

• Track usage per team
• Bill based on: compute hours × $rate
• Example: Team A used 50 GPU-hours at $1.50/hour = $75

Option 3: Per-Prediction Chargeback

• Track cost per prediction
• Bill based on: predictions × $rate_per_prediction
• Example: 1M predictions at $0.001 = $1,000

Option 4: Hybrid

• Base cost (reserved capacity) + usage cost
• Example: $500/month base + $0.001 per prediction

Implementation:

• Track API calls per team/model
• Calculate resource usage (GPU hours, storage)
• Generate monthly reports
• Bill or allocate cost

Benefits:

• Cost visibility
• Teams make conscious decisions about usage
• Can justify infrastructure investment

Practical Use Cases

Use Case 1: Building an Internal AI Platform

Scenario: Financial services company has 3 models in production. Teams are calling them directly. There's no monitoring, no quotas, no versioning.

Current State:

• Credit scoring model: REST API at http://model1:8000
• Fraud detection model: REST API at http://model2:8000
• Customer segmentation model: REST API at http://model3:8000

Problem: No central point of control. Teams call models directly.

Redesigned Architecture:

Layer 1: API Gateway (Kong)

• Single entry point: http://api.internal/v1/
• Authentication: OAuth2
• Rate limiting: Per-team quotas
• Monitoring: Logs all requests

Layer 2: Service Registry (Consul)

• Models register their address
• Gateway discovers models via registry
• Easy to move/upgrade models

Layer 3: Model Services

• Credit Score Service (v1.2, v1.1, v1.0)
• Fraud Detection Service (v2.0, v1.9)
• Segmentation Service (v3.1)

Layer 4: Monitoring & Logging

• Prometheus: Metrics (latency, errors, throughput)
• ELK: Logs (request details, errors)
• Cost tracking: GPU usage per team

Routing Rules:

GET /api/v1/credit-score
- Auth: check API key
- Rate limit: check team quota
- Route: 90% → v1.2, 10% → v1.3 (canary)
- Response: 120ms avg latency
- Cost: $0.0008 per prediction

GET /api/v1/fraud-detection
- Auth: check API key
- Rate limit: check team quota
- Route: v2.0
- Response: 200ms avg latency
- Cost: $0.0015 per prediction

GET /api/v1/orchestrate/recommendation
- Auth: check API key
- Orchestration: run chained models
1. User embedding (50ms)
2. Recommendation (80ms, parallel with #1)
3. Ranking (60ms)
- Total latency: 140ms
- Cost: $0.003 per prediction (sum of three models)

Deployment Result:

Metric
Before
After
Change

Models deployable / month
1
5
5x

Mean latency (orchestrated)
300ms
140ms
2.1x faster

Uptime
95%
99.9%
higher

Cost visibility
none
full
trackable

Rate limit enforcement
no
yes
fair sharing

Use Case 2: Implementing Intelligent Routing

Scenario: E-commerce company has two recommendation models in production.

• Model v1.0: Old, stable, optimized for latency (<50ms)
  - Model v2.0: New, better accuracy, slower (100ms)

Goal: Roll out v2.0 gradually while maintaining performance.

Implementation:

Week 1: Route 5% to v2.0

GET /api/v1/recommendations
- If user hash % 100 < 5: route to v2.0
- Else: route to v1.0
- Monitor: v2.0 latency, accuracy, errors

Week 2: Route 25% to v2.0

GET /api/v1/recommendations
- If user hash % 100 < 25: route to v2.0
- Else: route to v1.0
- Metrics: v2.0 performing well? CTR improving?

Week 3: Route 50% to v2.0

• A/B testing active
  - v2.0 showing 2% improvement in CTR
  - No performance issues

Week 4: 100% on v2.0

• v2.0 is stable
  - v1.0 in archive (can rollback if needed)

Outcome:

• Zero downtime deployment
• Detected issues on 5% subset before full rollout
• A/B test validated performance improvement
• Could rollback instantly if needed

Use Case 3: Cost Attribution and Chargeback

Scenario: Mid-size company runs 10 AI models. C-suite wants to know: "What are we spending on AI?"

Setup:

Track all API requests:

{
timestamp: "2024-01-15T10:30:45Z",
team: "fraud",
model: "fraud_detection",
version: "v2.1",
prediction_type: "real_time",
latency_ms: 150,
gpu_compute_time_ms: 120,
tokens_used: 0,
request_status: "success"
}

Calculate costs:

Fraud Team Usage (January):
- 50M fraud detection predictions
- Avg 150ms latency, 120ms compute per prediction
- GPU compute: 50M × 120ms / 1000 / 3600 = 1,666 GPU-hours
- Cost: 1,666 hours × $1.50/hour = $2,499

Recommendation Team Usage (January):
- 100M recommendations
- 3 models chained (50ms + 80ms + 60ms compute)
- GPU compute: 100M × (50+80+60)ms / 1000 / 3600 = 4,166 GPU-hours
- Cost: 4,166 hours × $1.50/hour = $6,249

Customer Success Team Usage (January):
- 10M churn predictions
- 80ms compute per prediction
- GPU compute: 10M × 80ms / 1000 / 3600 = 222 GPU-hours
- Cost: 222 hours × $1.50/hour = $333

Total: $9,081/month

Chargeback:
- Fraud: $2,499 (27%)
- Recommendation: $6,249 (69%)
- Customer Success: $333 (4%)

Outcome:

• Cost visibility: knows exactly what's being spent and on what
• Cost allocation: teams see what they're using
• Optimization opportunities: can see which models/teams are expensive
• Budget planning: can forecast future costs based on usage trends

Examples

Example 1: API Gateway Configuration

Kong API Gateway Configuration

services:
- name: credit_score
host: credit_score_service
port: 8000
routes:
- paths: [/api/v1/credit-score]
methods: [GET, POST]
plugins:
- name: authentication
config:
credential_type: oauth2
- name: rate-limiting
config:
minute: 1000 # 1000 req/min
policy: local
- name: request-size-limiting
config:
size: 1000 # 1KB limit
- name: cors
config:
origins: ["*"]
- name: prometheus
config:
# Track metrics

• name: recommendation
  host: recommendation_service
  port: 8000
  routes:
  - paths: [/api/v1/recommendations]
  plugins:
  - name: authentication
  - name: rate-limiting
  config:
  minute: 500 # Lower quota
  - name: request-transformer
  config:
  add:
  headers:
  - "X-Request-ID: $(uuid)"

upstream_targets:
credit_score:
- target: credit_score_v1_2:8000
weight: 90
- target: credit_score_v1_3:8000
weight: 10 # Canary

Example 2: Model Orchestration Definition

BentoML Orchestration Configuration

name: recommendation_pipeline
version: v1.0

steps:
- name: user_embedding
service: user_embedding_service
input: user_id
timeout: 100ms

• name: recommendation
  service: recommendation_service
  input: user_embedding.output
  timeout: 120ms
  depends_on: [user_embedding]
• name: ranking
  service: ranking_service
  input:
  - recommendations: recommendation.output
  - user_context: user_embedding.output
  timeout: 100ms
  depends_on: [recommendation]
• name: personalization
  service: personalization_service
  input:
  - ranked: ranking.output
  - user_id: user_id
  timeout: 80ms
  depends_on: [ranking]

output: personalization.output
total_timeout: 400ms # SLA

Anti-Patterns

Anti-Pattern 1: Exposing Models Directly Without a Gateway

Teams call models directly. One team hammers a model with 100K requests, making it slow for everyone. No authentication, no rate limiting.

Instead: Use an API gateway. Single entry point, rate limiting, monitoring.

Anti-Pattern 2: No Cost Tracking

Infrastructure costs spiral. No one knows what's being spent or why. Can't justify budget.

Instead: Track usage per team/model. Generate cost reports monthly. Use chargeback or allocation to create cost awareness.

Anti-Pattern 3: Deploying New Models Without Versioning Strategy

You release a new model version. It breaks clients who weren't prepared for the change. You have to rollback fast, but rolling back is painful.

Instead: Version all models. Use canary deployment for gradual rollout. Keep multiple versions running.

Anti-Pattern 4: Chains Without Error Handling

Recommendation → Ranking → Personalization. If Ranking fails, whole chain fails.

Instead: Implement fallback strategies. If Ranking is down, skip it and return recommendations. If Personalization is slow, use cached result.

Anti-Pattern 5: No Monitoring of Model Performance

Models degrade slowly. Users start getting bad predictions. You don't know until someone complains.

Instead: Monitor model predictions continuously. Alert if accuracy drops below threshold or latency exceeds SLA.

Human Judgment Checkpoints

Checkpoint 1: Do You Have a Single Entry Point for AI Services?

If teams are calling models directly, add an API gateway. It's a high-ROI investment.

Checkpoint 2: Can You Easily Add a New Rate-Limit Rule?

Implementing new quotas should be a config change, not a code change.

Checkpoint 3: Can You Trace Request Through the System?

From API gateway to model to inference to response, can you see what happened? If not, add request tracing.

Checkpoint 4: Are You Monitoring Cost?

Can you answer: "What did we spend on AI this month? Which teams used it? Which models cost most?"

Checkpoint 5: Do You Have a Versioning Strategy?

Can you deploy a new model version without breaking existing clients?

Key Takeaways

Use an API gateway as the single entry point for AI services. This gives you authentication, rate limiting, monitoring, and versioning in one place.

Implement rate limiting and quotas per team. This prevents one team from monopolizing resources and creates cost signals.

Version all models and use canary deployment for rollouts. This lets you deploy improvements without breaking existing clients.

Track cost per team and per model. Cost visibility drives good behavior. Teams consume responsibly when they see what they're using.

Use intelligent routing for multi-model systems. Canary deployment, A/B testing, and fallback strategies make systems more reliable and let you improve continuously.

Monitor everything: latency, errors, cost, model performance. If you can't see it, you can't manage it.

Automate orchestration for chained models. Parallel execution, smart routing, and error handling reduce latency and improve reliability.

Document your API contracts. Make it easy for teams to use your services. Good documentation reduces support burden.