Overview

One of the core decisions you'll make building with AI: run your own models or use proprietary APIs?

This isn't a binary choice, most sophisticated organizations do both. But understanding the tradeoffs is critical because they're different in 2026 than they were in 2024. The landscape shifted. Capabilities improved. Costs changed. What was true 18 months ago might not be true now.

This lecture is about helping you make that decision for your specific context. We'll cover the model types, the advantages and disadvantages of each, cost calculations, and real-world trade-off scenarios. Then we'll discuss a hybrid approach that many organizations use successfully.

Proprietary, API-Based Models

How it works: You send a request to OpenAI, Anthropic, or another provider. They run the model on their infrastructure. They send back the response. You pay per token used.

Advantages:

• Best-in-class capability: Frontier models (latest, most capable) are with commercial vendors. Claude Opus, GPT-4 Turbo, Gemini Pro. These are superior to open-source in capability.
  - No infrastructure burden: You don't run servers, manage GPUs, handle scaling. Vendor handles all of that.
  - Simple pricing: Pay for what you use. Predictable. Scale up or down instantly.
  - Zero operational overhead: Vendor handles updates, patches, maintenance, scaling. You don't have a midnight page when the model serving goes down.
  - Integration ready: SDKs, documentation, examples. Everything you need to integrate.
  - Latest models: Vendors update models regularly. You get improvements automatically.

Disadvantages:

• Cost scales with volume: High-volume applications become expensive. If you're processing billions of tokens per month, costs add up.
  - Latency: Network round-trip adds delay. Not suitable for sub-100ms latency requirements.
  - Data leaves your systems: Privacy and compliance implications. Some organizations can't send data to third parties.
  - Dependent on vendor reliability: Outages affect you. Rate limits cap your throughput.
  - Limited by vendor's rate limits: You can't request more than they allow. During high demand, you might get throttled.
  - Vendor lock-in: Switching to a different vendor's API requires rewriting integration code.

When to use: Interactive applications, applications where freshness matters (models updated regularly), applications where ultimate capability matters, anything user-facing, rapid prototyping, teams without ML ops expertise.

Real example: A SaaS company building an AI-powered writing assistant. They use Claude API. Users type in their app. Request goes to Anthropic. Response comes back. User gets suggestions. Simple, fast to market, no infrastructure work.

Open Source, Self-Hosted Models

How it works: You download a model (like Llama 3 from Meta). You run it on your infrastructure (cloud servers or on-prem). You send requests to your own servers. Zero external API costs per token (you pay for compute infrastructure only).

Advantages:

• Data stays on-prem: No privacy concerns. No compliance issues with sending data to third parties.
  - Pricing is predictable: You pay for servers, not per token. 1 million requests = same cost as 10 million requests once servers are paid for.
  - No external dependencies: Your performance isn't affected by vendor outages. Your throughput isn't capped by vendor rate limits.
  - Fully customizable: You can fine-tune the model on your data. You can modify the architecture. You control everything.
  - Potential cost advantage at scale: If you have high volume and can keep servers busy (high utilization), self-hosting becomes cheaper than APIs.
  - Full control: Deploy where you want, modify as needed, integrate however you want.
  - Latency: Depends on your hardware, but can be fast if you co-locate inference and application.

Disadvantages:

• Infrastructure burden: You manage servers, GPUs, scaling, monitoring, security, updates. This is work.
  - Operational complexity: Monitoring, troubleshooting, updates, capacity planning. You need ops expertise.
  - Inferior capability: Open-source models usually lag frontier models. Llama 3 is good but not as good as Claude Opus.
  - Latency depends on hardware: Depends on GPUs, network, optimization. Faster than API round-trip but requires investment.
  - Team expertise required: You need people who know how to operate ML infrastructure. Hard to hire. Expensive.
  - Capital costs upfront: GPUs cost $3k-10k per month. You pay whether you use them or not.
  - Maintenance burden: Models need to be retrained, updated, monitored. This is ongoing work.

When to use: Batch processing (non-real-time workloads), applications with extreme privacy requirements, cost-critical applications with high volume, specialized applications needing fine-tuning, applications where latency is critical and you can't rely on network round-trips.

Real example: A financial services company doing risk analysis on confidential trading data. They can't send data to external APIs (regulatory requirement). They self-host Llama. Accepts slightly lower quality to maintain privacy.

The Cost Crossover Point: At some volume, self-hosting becomes cheaper than APIs. This depends on utilization, idle servers cost the same as busy ones. If you can keep GPU utilization high (70%+), the crossover point is lower. Typical crossover: 5-10 billion tokens per month, depending on model size and utilization.

The Cost Calculation (Get Specific)

Let's do real math. Claude 3 Haiku (via API): roughly $0.25 per million input tokens, $1.25 per million output tokens. Sounds cheap until you do the math.

Scenario: Processing 1 billion tokens per month.

• Input cost: $250/month
  - Output cost (assuming 1:1 input:output ratio): $1,250/month
  - Total: $1,500/month or $18k/year

Self-hosted equivalent: NVIDIA A100 GPU in cloud (AWS, GCP, Azure)

• GPU rental: $3,000/month
  - Can process: approximately 10,000 tokens per second
  - 1 billion tokens per month: 385 tokens/second on average
  - Utilization: 3.85% of GPU capacity
  - Cost allocated: $3,000 × 0.0385 = $115/month

Wait, that's cheaper! But...

You also need:

• Server to run the model ($500/month)
  - Monitoring, security, networking infrastructure ($200/month)
  - Engineering time for setup, monitoring, maintenance (0.5 FTE = $6,250/month)
  - On-call rotation for failures

Real all-in cost for self-hosting: roughly $7,000/month for this volume, assuming you can amortize some costs across multiple applications.

Comparison:

• API cost: $1,500/month
  - Self-hosted cost: $7,000/month
  - Winner: API, by 4.7x

But also consider: The API gives you Claude Opus-level quality. Self-hosted gives you Llama quality (maybe 70-80% as good). Is the quality difference worth the cost savings? Probably not at 1B tokens/month.

At what volume does self-hosting win?

At 10 billion tokens/month:

• API cost: $15k/month ($180k/year)
  - Self-hosted cost: $8k/month (engineering time spread across multiple applications, better utilization)
  - Self-hosted wins

At 100 billion tokens/month:

• API cost: $150k/month ($1.8M/year)
  - Self-hosted cost: You can afford multiple GPUs, specialist team, better utilization. Maybe $30k/month.
  - Self-hosted wins by a lot

Key insight: Most companies aren't at 10+ billion tokens/month. They're at 1-5 billion. For them, APIs are cheaper and simpler. Self-hosting makes sense only at scale.

The Hybrid Approach (Most Companies Do This)

Smart organizations don't pick one. They do both:

• APIs for interactive, low-latency, high-capability applications: User-facing features where quality matters and speed matters. Claude API.
  - Self-hosted for batch processing, sensitive data, or extreme cost sensitivity: Overnight jobs, confidential data, cost-critical workloads. Llama or Mistral.
  - Sometimes, open-source models via API providers: Companies like Together AI, Replicate, or RunwayML offer Llama, Mistral via API. Cost efficiency without infrastructure burden. Middle ground.

This approach requires an abstraction layer: Code that doesn't care which vendor you use. You route requests to the right model/provider based on requirements (cost, privacy, quality, speed). This takes engineering work but gives flexibility.

Real example: E-commerce company uses Claude API for product descriptions and chat (interactive, quality matters). Uses self-hosted Llama for batch processing of customer feedback (non-time-sensitive, cost-sensitive). Uses Together AI's Llama API for some middle workloads. One abstraction layer handles routing.

The Practical Reality in 2026

Honest assessment: for most organizations, API-based proprietary models are the right choice. Here's why:

• Proprietary models are significantly better. Haiku vs. Llama 3 is a meaningful quality gap. For most applications, proprietary is worth it.
  - Operational complexity of self-hosting is being underestimated. Teams that haven't done it think it's just "rent a server and run it." Reality: monitoring, retraining, updating, capacity planning, on-call, security updates. It's work.
  - API pricing has become competitive and predictable. Not what people expected 2 years ago. It's affordable for most workloads.
  - The cost crossover point is higher than people expect. You need very high volume before self-hosting makes financial sense. Open-source models are catching up in cost but not in capability.
  - Time to market advantage of APIs is huge. You can launch in weeks. Self-hosting takes months to do well.

But there are exceptions:

• Data that can't leave your infrastructure: Some financial, healthcare, government contexts. Self-hosting might be required.
  - Extreme scale: Billions of tokens per month. Only then does self-hosting's long-term cost efficiency kick in.
  - Deep customization needed: Model fine-tuning, architectural modifications. You need control.
  - Unreliable internet: You need local capability because external APIs aren't reliable.

For everyone else: use APIs. It's simpler, cheaper, faster to market. Revisit in 18 months if your volume changes dramatically.

The Decision Framework: Does your data require on-prem hosting? No = use APIs. Yes = self-host. Are you processing >5B tokens/month cost-sensitively? No = use APIs. Yes = self-host and/or use open-source API providers. Do you need model customization? No = use APIs. Yes = self-host.

What to Do Monday Morning

Step 1: Calculate your actual AI token volume. How many tokens are you processing per month? Go look at your API bills or estimates.

Step 2: Audit your data sensitivity. What data are you sending to APIs? Is any of it sensitive? That determines whether API usage is even acceptable in your organization.

Step 3: Evaluate your team capability. Do you have people who can operate ML infrastructure? If not, self-hosting becomes much more expensive. If yes, it might be viable.

Step 4: Calculate TCO for your workload. API cost vs. self-hosted cost. Be honest about engineering time and operational overhead.

Step 5: Make a decision based on data, not hype. APIs for most companies. Self-hosting for specific scenarios. Hybrid if you have workloads in both buckets.

Step 6: Build an abstraction layer if hybrid. One abstraction handles routing to different models/providers. Enables flexibility.

FAQ: Open Source vs. Proprietary

Q: Are open-source models catching up to proprietary?**

A: In some dimensions, yes. Llama 3 is respectable. But there's still a quality gap for complex reasoning tasks. Proprietary models still have an edge.

Q: What about privacy with APIs?**

A: Depends on your contract with the vendor and your data. Many vendors have enterprise agreements that don't use your data for training. But this should be explicit in your contract.

Q: Can we start with APIs and switch to self-hosted later?**

A: Yes, that's reasonable. Start fast with APIs. If volume grows to justify it, switch to self-hosted. Need a good abstraction layer though.

Q: What's the best open-source model right now?**

A: Llama 3 (70B) is solid. Mistral 8x7B (mixture of experts) is efficient. But "best" depends on your use case. Test with your data.

Q: Should we fine-tune a model?**

A: Only if your domain is very specialized or you need specific output format. Most of the time, prompting is sufficient. Fine-tuning is expensive and complex.

Q: What happens if the API vendor raises prices?**

A: You can switch to a different vendor (if you have abstraction layer). You can increase self-hosting if you hit breakeven. You can optimize prompts to reduce tokens. You can live with higher costs if the alternative is worse. API vendors know switching costs exist, so price increases are usually modest (5-10%/year, not 50%). But this risk is why you build abstraction layers, to maintain optionality.

Case Study: Fintech's Open-Source vs. Proprietary Decision

A fintech company needed AI for risk analysis and document processing. They evaluated three architectures:

Option A (Full Proprietary API): Use Claude API for all work. Estimated cost at 500M tokens/month: $75K/month. Estimated engineering time: 2 months to build. Upside: high accuracy, fast to market. Downside: vendor lock-in, costs scale with volume.

Option B (Full Self-Hosted Open-Source): Deploy Llama 70B on Kubernetes. Estimated infrastructure cost: $35K/month + $200K initial setup + $80K/year operational overhead (1 FTE). Estimated engineering time: 5 months to build and stabilize. Upside: complete control, costs don't scale with volume. Downside: lower accuracy on complex analysis, slower time to market, operational burden.

Option C (Hybrid): Use Claude API for complex risk analysis (high-value, accuracy-critical, 15% of tokens). Use self-hosted Llama for document classification and simple document processing (lower-value, lower complexity, 85% of tokens). Estimated cost: Claude API (10K/month) + self-hosted infrastructure (35K/month) + setup/operational costs = $55K/month total. Estimated engineering time: 3 months.

Decision:** They chose Option C (hybrid).

Year 1 costs:**
- Option A: $900K (API) + $200K (engineering) = $1.1M
- Option B: $35K * 12 (infra) + $80K (operations) + $400K (engineering) = $1.04M
- Option C: $55K * 12 (blended) + $250K (engineering) = $910K

All three are roughly equivalent in total cost, but Option C wins because:
1. Lower operational burden (hybrid uses 0.5 FTE vs. Option B's 1 FTE)
2. Better accuracy than Option B (Claude for complex analysis)
3. Cost control vs. Option A (self-hosting handles 85% of volume)
4. Faster time to market than Option B (3 months vs. 5 months)

Year 2 dynamics:** Volume doubled. With Option A, costs would be $1.8M. With Option C's hybrid approach, it was $1.2M (self-hosted scales, API usage relatively flat). Option C was clearly the right choice.

Key lesson:** The decision wasn't "proprietary OR open-source." It was "what mix of approaches maximizes value for our situation?" Hybrid won.

A Decision Matrix for Your Situation

Use this to decide: (1) What's your token volume per month? (2) Do you have data that can't leave your infrastructure? (3) Do you have dedicated ML ops engineers? (4) What's your accuracy requirement?

Low volume (1B/mo), privacy critical, strong ML ops team, demanding accuracy: Full self-hosted or hybrid heavy. Total cost: $50K+/month but includes infrastructure + people. Time to market: 4-6 months.

Key Insight

For most companies, APIs are simpler and cheaper than self-hosting. Self-hosting cost is $7k/month at 1B tokens/month (vs. $1.5k API), breaks even at 10B tokens/month. Exceptions: data privacy requirements (self-host), extreme scale 50B+ tokens/month (self-host), deep model customization needed (self-host). Hybrid approach: APIs for user-facing work, self-hosted for batch processing of sensitive data.

Making the Choice for Your Context

There's no universal right answer. It depends on your volume, your data sensitivity, your team capability, your timeline, and your requirements. Use the framework above to decide for your specific situation.

Most companies will choose APIs. That's the right choice for most. Some will need self-hosting for specific workloads. A few will do hybrid. All three approaches are viable.

Make an informed decision. Revisit it regularly as circumstances change.

On This Page

Introduction
Proprietary, API-Based
Open Source, Self-Hosted
Cost Calculation
Hybrid Approach
Practical Reality in 2026
Monday Morning Action
FAQ
Key Insight
Making the Choice

Chapter Details