Overview

You've deployed Claude to every AI task in your application. It works well. But you're spending a lot on API calls. And you're noticing something: not every task needs Claude's full reasoning capability.

Some tasks are simple classification. "Is this email spam or legitimate?" Claude is overkill for this. You could use something cheaper and faster and get the same answer.

Some tasks need specialized domain knowledge. "What does this medical imaging report indicate?" You might need a model that's been fine-tuned on medical data, not a general-purpose model.

Some tasks are latency-sensitive. Real-time user interactions. You need a model that responds in milliseconds, not seconds. Smaller models are faster.

Some tasks are cost-sensitive. You're processing millions of items. Using an expensive model multiplies your costs across the entire dataset. A cheaper model could handle 90% of your volume.

This is the multi-model strategy: using the right model for the right job. Not one model for everything. A portfolio of models optimized for different purposes.

The payoff is enormous. Cost drops 30-70%. Quality stays the same or improves. Latency improves. You're no longer paying for reasoning power you don't need.

This is how large organizations actually run AI at scale. Not with one model, but with a carefully orchestrated portfolio of models.

The Model Landscape

General-Purpose Reasoning Models

These are the powerhouses. Claude 3.5 Sonnet. GPT-4 Turbo. They're good at everything: reasoning, writing, coding, analysis, planning.

Cost: high. $0.001-0.01 per 1K input tokens, $0.003-0.03 per 1K output tokens. For a 2K input and 500 output, you're paying $0.02-0.05 per request.

Latency: 1-5 seconds typical. Fine for asynchronous tasks. Not great for real-time interactions.

Best for: complex reasoning, novel problems, anything where correctness is critical.

Use when: you need the model to think carefully, consider edge cases, balance multiple requirements.

Fast, Cheap Models

Claude 3 Haiku. GPT-3.5 Turbo. Llama 2-7B. These are smaller models that trade some reasoning capability for speed and cost.

Cost: low. $0.00008-0.0002 per 1K input tokens. For the same 2K input and 500 output, you're paying $0.0002-0.0005 per request.

So Haiku is 50-100x cheaper than Sonnet.

Latency: 100-500ms typical. Good for real-time interactions.

Best for: simple classification, routing, extraction, summarization of straightforward content.

Use when: you know the task is well-defined and doesn't require deep reasoning.

Specialized Models

These are models optimized for specific domains or tasks:

Vision Models: Claude 3.5 Sonnet, GPT-4 Turbo, or Claude 3.5 Sonnet with vision. These understand images and can answer questions about them.

Code Models: Claude (especially with code context), specialized coding models. These are trained on massive amounts of code and understand programming patterns better than general models.

Domain Models: Medical models, legal models, financial models trained on domain-specific data. These understand the domain better than general models.

Embedding Models: Text embedding models that convert text to vectors. Used for semantic search, clustering, recommendation systems. Cheap and fast.

Retrieval Models: Models optimized for finding relevant information from large bodies of text. Used for RAG (retrieval-augmented generation) systems.

Cost and latency vary widely depending on the model and task.

Open Source Models

Llama 2, Llama 3, Mistral, and others. You can run these yourself or use services that host them.

Advantage: you control the model, you control the data (privacy), you can fine-tune them.

Disadvantage: you have to run them (hardware cost), maintain them, and they're often not as capable as proprietary models.

Best for: situations where you need privacy, control, or deep customization.

Key insight: There's no single best model. Different models are optimized for different cost/performance/latency tradeoffs. Your job is to match models to tasks based on requirements.

Building Your Multi-Model Routing Strategy

Categorize Your Tasks

List every AI task your application performs. "Classify emails as spam or legitimate." "Extract key information from documents." "Generate product descriptions." "Analyze user sentiment." "Write code based on specifications." "Answer customer support questions."

For each task, estimate three things:

Complexity: Does this require reasoning or is it straightforward pattern matching? On a scale of 1-10, where is it?

Accuracy requirement: If the model gets this wrong, what's the impact? Spam misclassification is annoying but not critical (maybe 95% accuracy is fine). Fraud detection is critical (need 99.9% accuracy).

Volume and cost sensitivity: How many times is this task called? If it's millions of times a day, cost matters. If it's hundreds of times a day, cost is less important.

Match Tasks to Models

Now you're building a decision matrix. For each task, which model makes sense?

Rule 1: Use the cheapest model that meets your accuracy requirement.

If Haiku gets 95% accuracy and that's good enough, use Haiku. Don't pay for Sonnet.

Rule 2: For novel or complex tasks, start with Claude. Then optimize.

You don't know what the answer is yet. Use a powerful model. Once you understand the problem, you might optimize to something cheaper.

Rule 3: For high-latency-sensitive tasks, use fast models or local models.

Real-time user interactions need millisecond latency. Haiku or local models. Not Sonnet which might take 5 seconds.

Rule 4: For privacy-sensitive tasks, use models you control.

Patient data, financial data, proprietary information. Run a local model or work with a provider that guarantees privacy.

Rule 5: For volume tasks (millions per day), optimize for cost aggressively.

Every 1% cost savings multiplies across millions. If you're calling Claude a million times a day for something that could be done with Haiku, you're wasting money.

Build the Router

Your application needs to route tasks to the right model. The router could be simple or complex.

Simple router (rule-based):

If task == "spam_classification" → Haiku

Else if task == "document_analysis" → Claude

Else if task == "image_understanding" → Vision model

Else → Claude (default to powerful for unknown tasks)

Complex router (learned):

You build a small classifier that predicts which model is best for a given task based on the task description and metadata. This lets you optimize without hardcoding every task.

Start simple. Add complexity only if you need it.

Handle Failures and Retries

Sometimes a cheaper model fails. What do you do?

Retry with a more powerful model. If Haiku fails or returns low-confidence answer, escalate to Claude.

Use ensemble voting. Run multiple models, take the majority vote. More expensive but higher quality.

Accept failure. If the task is low-impact (like content filtering), accept that some things slip through.

Most applications use a combination. Haiku for speed. Escalate to Claude if confidence is low. Use ensemble voting for critical tasks.

Monitor Cost and Quality

Once you've deployed your multi-model system, monitor everything.

Cost metrics: Total cost per day. Cost per task type. Which models are most expensive. Are there opportunities to optimize further?

Quality metrics: Accuracy of each model on each task. Error rates. User complaints. Are cheaper models actually meeting quality requirements?

Latency metrics: Response time for each model. Are latency-sensitive tasks fast enough?

Monthly reviews: are you hitting your cost targets? Quality targets? Latency targets? If not, adjust routing.

Advanced Cost Optimization Techniques

Batch Processing

If your task doesn't need real-time response, batch it. Process thousands of items at once with batch APIs. Many providers offer discounts for batch jobs (50% cheaper).

Example: customer support transcripts. You don't need to analyze them in real-time. You can analyze them in a daily batch job. Save 50% on cost.

Caching

If the same input is processed multiple times, cache the result. Second time someone asks the same question, return the cached answer instead of calling the model.

Cost savings can be 80%+ if your workload is repetitive.

Context Reuse

If you're processing a conversation, reuse context. Send the previous messages with new user message, not just the new message alone. Reduces redundant processing.

Prompt Optimization

Shorter prompts = cheaper responses. Can you say the same thing more concisely? Can you move some instructions to system prompts instead of the message?

Cost savings: 10-20% by writing prompts more efficiently.

Model Fine-Tuning

If you have a specific task you're doing thousands of times, fine-tune a smaller model on examples of that task. A fine-tuned small model might outperform a large model on your specific task.

Cost savings: 50-90% if fine-tuning is successful.

Structured Output

If you're extracting data, use structured output (JSON format). The model generates less text. Cheaper and easier to parse.

Cost optimization rule: A 10% cost savings multiplies across your entire volume. If you're running a billion model calls a year, 10% savings is six figures. Optimization is worth the effort.

Real-World Example: E-Commerce Platform

Imagine you're running an e-commerce platform with several AI tasks:

Task 1: Spam classification on customer reviews

Volume: 100K reviews per day. Accuracy needed: 95%. Real-time? No.

Model choice: Haiku + batch processing. Cost: ~$1/day. Alternative (Claude): ~$50/day.

Task 2: Product description generation

Volume: 50 new products per week. Accuracy needed: High (customers see it). Real-time? No (background job).

Model choice: Claude. Cost: ~$0.50/product. Worth paying for quality.

Task 3: Customer support chatbot

Volume: 10K conversations per day. Accuracy needed: Medium-high. Real-time? Yes (customer waiting).

Model choice: Haiku for initial response (fast), escalate to Claude if confidence is low. Average cost: ~$0.05 per conversation.

Task 4: Search query classification (what category is customer looking for?)

Volume: 1M searches per day. Accuracy needed: 90%. Real-time? Yes (search latency matters).

Model choice: Fine-tuned Haiku. Cost: ~$0.001 per search.

Task 5: Product image understanding (what is in this product image?)

Volume: 50K images ingested per week. Accuracy needed: High. Real-time? No.

Model choice: Claude with vision (batch). Cost: ~$0.10 per image.

Total cost with multi-model strategy: ~$2K/month

If they used Claude for everything: ~$50K/month.

Savings: 96% while maintaining quality.

Building a Multi-Model System

Architecture

Your application has an AI request layer. Instead of directly calling Claude, requests go to a router. The router decides which model to use. Requests are sent to the appropriate model.

Results come back. Results are cached if applicable. Results are returned to the application.

Logging happens at every step. Cost, latency, accuracy, errors.

Implementation

Use a service like LiteLLM or LangChain that abstracts away the differences between models. Single interface, multiple backends.

Or build your own thin wrapper that handles routing and logging.

Key features to build in:

• Router that can make intelligent decisions about which model to use
• Fallback logic (if primary model fails, retry with backup)
• Caching layer (detect repeated requests, return cached results)
• Comprehensive logging (cost, latency, accuracy, errors)
• Easy way to tune routing rules without code changes

Testing and Validation

For each task, run multiple models and compare quality. Create a test set. Which model performs best? Which is cheapest?

Plot the Pareto frontier: for each accuracy level, which model is cheapest? Choose your router based on this.

Continuously monitor: as models improve or pricing changes, re-evaluate.

What to Do Monday Morning

Catalog your AI tasks: List every AI task in your application. How many times per day? What accuracy is needed? Is latency critical?

Test alternatives: For each task, run it with different models (Haiku, Claude, specialized models if applicable). Compare accuracy and cost.

Calculate potential savings: If you switched from Claude to Haiku where appropriate, how much would you save? What would quality impact be?

Build your routing rules: Based on the analysis, define rules for which model handles which task.

Implement a basic router: Could be simple if/else logic at first. Route requests to appropriate models.

Monitor and iterate: Track cost and quality. Fine-tune routing. Look for optimization opportunities.

Case Study: E-Commerce Platform Multi-Model Optimization

An e-commerce platform was using Claude for all AI tasks: product categorization, quality scoring, fraud detection, recommendation ranking, and customer support. Monthly API spend: $85k. They implemented a multi-model strategy. Process: (1) Cataloged 18 distinct AI tasks. (2) Tested each task with Claude, Haiku, and specialized models. (3) Created routing rules. Result by task:

• Product categorization (5M calls/month): Switched from Claude to Haiku + embedding model. Accuracy dropped from 98% to 96.5%. Cost dropped from $50k/month to $2k/month. Net savings: $48k/month.
  - Fraud detection (2M calls/month): Stayed with Claude because accuracy is critical (99.5% required) and cost is not price-sensitive relative to fraud loss.
  - Recommendation ranking (10M calls/month): Switched to a specialized ranking model. Cost dropped from $60k/month to $8k/month. Ranking quality improved because specialized model understands e-commerce patterns better.
  - Customer support routing (500k calls/month): Switched from Claude to Haiku for initial categorization, then escalate complex cases to Claude. Latency improved (Haiku returns in 200ms vs Claude at 3 seconds). Cost dropped from $15k/month to $1.5k/month.

Total impact: Monthly spend dropped from $85k to $22k. Quality improved on 3 out of 4 task categories. Latency improved significantly. They also implemented continuous monitoring: every quarter, re-test models to see if new models or pricing changes create new opportunities. Year 1 savings: $756k. The engineering effort to implement routing was ~400 hours (one engineer, ~10 weeks). ROI: savings paid for investment in 2 months.

Key insight from implementation: the hardest part wasn't building the router (surprisingly simple logic). The hard part was getting domain experts to trust that 96.5% accuracy from Haiku was acceptable when Claude achieved 98%. Once they measured business impact (almost no customer-visible difference), the decision became obvious.

FAQ

Q: How do we test if a cheaper model is good enough for our task?

A: Create a test set of 100-1000 examples. Run both models. Compare results. Measure accuracy. If cheaper model is 95%+ as good, it's probably fine to use.

Q: What if we need to switch models mid-task if quality is low?

A: Build a confidence threshold. If the model's confidence is below threshold, escalate to a more powerful model. Costs more sometimes, but ensures quality.

Q: Can we use local models to save cost?

A: Yes. Run Llama or other open-source models locally. Huge cost savings if you have the infrastructure. Tradeoff: maintenance and hardware cost.

Q: How often should we reevaluate model choices?

A: Quarterly. Model quality and cost improve constantly. New models get released. Your application's requirements change. Stay on top of it.

Q: What about latency when switching models?

A: Smaller models are faster. If latency is critical, use small models or local models. Can always escalate to more powerful models asynchronously if needed.

Q: Doesn't using different models for different tasks create consistency issues?

A: Only if you're not intentional about it. Define what consistency means for your use case. For some tasks (content moderation), consistency across models matters. For others (product recommendations), consistency doesn't matter, what matters is quality. Test across models to ensure they behave acceptably. If you need consistency, use the same model for that task class, or test combinations to ensure they produce aligned outputs.

Q: What if the cheap model performs unexpectedly poorly on edge cases?

A: This is why monitoring matters. Track error rates, not just average accuracy. If you notice the cheap model performs poorly on specific input types, you have options: (1) adjust the router to escalate those inputs to a better model, (2) improve your test set to catch these cases earlier, (3) live with the edge case if the business impact is acceptable. Document these failures. They're valuable training data for understanding model limitations.

Q: Can we dynamically route based on request complexity?

A: Yes, this is advanced routing. Analyze the input first (quick, cheap analysis), determine complexity, then choose the appropriate model. Example: long customer support tickets → use Claude. Short tickets → use Haiku. This requires confidence estimation or complexity scoring, which adds a step but can save significant cost.

Q: What if new models are released and our existing routing is suboptimal?

A: Set a quarterly evaluation schedule. Test new models against your test sets. If a new model is better/cheaper, update your routing rules. Don't be rigid, be opportunistic. The e-commerce case study had to update routing twice in the first year because new models were released. Each time, they saved additional money.

Key Insight

One model for every task is simple but expensive. A multi-model strategy costs 50-90% less while maintaining or improving quality. Route simple tasks to cheap, fast models. Route complex tasks to powerful models. Route latency-sensitive tasks to fast models. Route privacy-sensitive tasks to models you control. Monitor cost and quality continuously. This is how large organizations run AI at scale efficiently.

On This Page

Watch the Lecture
The Model Landscape
Routing Strategy
Cost Optimization
Real-World Example
Building the System
Monday Morning Action
FAQ

Chapter Details

Part ofChapter 8