Overview

Here's the open source AI conversation happening everywhere: "We should use Llama because it's free and we control it. We should avoid vendor lock-in."

This logic makes sense until you actually try it. Then you realize free has hidden costs. Control comes with operational burden. And the complexity of self-hosting often outweighs the savings.

But open source does have a place. The question isn't "open source or API?" It's "where does open source provide real value versus where does it create technical debt?"

This is about being pragmatic. Not ideological.

The Economics of Self-Hosting AI

Let's calculate the real cost. People usually only count compute cost. They miss everything else. This is where most organizations fail to do proper due diligence.

Compute Cost

Running Llama-70B on a single H100 GPU costs roughly $0.50-$1.00 per million tokens depending on your infrastructure and utilization. This assumes you're paying about $3-5k per month for the GPU.

Running Claude API costs roughly $0.01-$0.03 per million tokens depending on model size and request type. That's Opus at $0.03 per million input tokens.

So self-hosting looks 20-100x cheaper by this metric. So obvious you should do it, right? But wait.

The Full Cost Picture

• Infrastructure: GPU server costs $3-5k/month. You need two for redundancy. That's $6-10k/month minimum. Add networking, storage, backup, and cooling. Add 20%. Total: $7-12k/month or $84-144k/year just for hardware.
  - Operational overhead: Someone monitors this. Handles failures. Updates models. Scales when you hit traffic spikes. That's 0.5 FTE minimum. Cost: $50-70k/year in salary plus 30% benefits. Realistically $70k/year total.
  - Opportunity cost: Your best engineer could be building features. Instead they're patching a model serving stack, debugging CUDA errors, and managing deployments. What's that engineer worth? $250-350k/year in fully-loaded cost. Even spending 25% time on this is a $60-90k drag.
  - Scaling complexity: When you 10x traffic, your single-GPU setup breaks. Now you need distributed inference, load balancing, caching. A major rewrite. That's 2-3 months of senior engineering time. $120-200k project.
  - Model updates: New versions of Llama come out (currently every 3-4 months). You want to use them. But they might not fit your GPU. Or they change output format. Unexpected dependencies break. You need to test, upgrade, validate. Ongoing work that takes 1-2 weeks per update. That's 10% of engineer time.
  - Security and compliance: Your model is a service exposed on your network. You need authentication, authorization, rate limiting, DDoS protection, audit logging. Another 2-4 weeks of engineering.

Real cost of self-hosting: $200-300k/year for a moderately-scaled setup. Plus 25-30% of a senior engineer's time.

Real cost of Claude API for equivalent throughput: $30-80k/year, depending on token volume.

Case Study: Fintech Startup

A fintech startup decided to self-host Llama-70B for document analysis. Initial cost estimate: $50k/year (compute only). They budgeted $75k with "some operational overhead." Year one actual: $220k. Year two: $280k (added redundancy, observability, security). By year two, they realized Claude API would have cost $45k and they'd have less operational burden. They migrated. The setup time? 2 weeks. The regret? Significant. They'd spent half a million dollars to avoid $90k in API costs.

This math changes if you have massive volume (millions of tokens per day). It changes if you have privacy requirements that make self-hosting essential. It changes if you have specialized needs where Llama's performance is 2x better than Claude.

For most companies? The math doesn't work in favor of self-hosting.

The Economics Principle: Don't optimize for cost. Optimize for total cost of ownership including engineering time. Often that means using the API, even if raw compute is more expensive.

When Self-Hosting Actually Makes Sense

Not never. There are legitimate cases where the economics actually work.

High Volume with Consistent Load

If you're doing 500M+ tokens per month on a consistent workload, self-hosting can be cheaper. But you need the volume to be truly predictable and stable. If it spikes seasonally (e.g., 10x during holidays), you're overprovisioning hardware most of the year and the economics break.

Concrete example: a support team that processes 5M support queries per month through summarization, classification, and routing. Consistent load (varies by day of week but nothing dramatic), massive volume. Cost with Claude: $150-200k/year. Cost to self-host: $180k infrastructure + $50k ops + $30k opportunity = $260k first year, then $80k ongoing. At 3x that volume (15M queries/month), self-hosting becomes cheaper year two.

Another example: a data processing pipeline that summarizes 100M documents. One-time workload, no ongoing traffic. Self-hosting wins here because you can buy compute for 3 months, do the work, and turn it off. API would cost millions.

Privacy/Data Residency Requirements

If your data can't leave your infrastructure due to regulation or customer requirements, you have to self-host. The cost math doesn't matter. It's a requirement.

In this case, the question becomes: what's the minimum cost way to self-host? Not "how do we make self-hosting cheap" but "what's the least expensive path that meets the requirement?"

Often: run proprietary models (Claude, GPT) internally via API gateway. Or, use a smaller open source model that fits your privacy requirements.

Specialized Modeling Advantage

If you've fine-tuned a model on your domain data and it's 2-3x better than off-the-shelf models, self-hosting makes sense. The superior performance is worth the operational cost.

Medical imaging models. Legal document analysis. Domain-specific classifiers. These cases justify self-hosting.

But be honest: is your model actually 2-3x better? Or are you overestimating? Test it. Compare it to Claude or GPT-4 on your exact use case. Real data.

Offline/Low-Latency Requirements

If you need AI inference when there's no internet or you need sub-100ms latency with no dependency on an external API, you need to self-host.

Example: edge computing on mobile devices, real-time scoring in a trading engine, inference on closed networks.

In these cases, self-hosting isn't a cost optimization. It's a capability requirement.

The Hybrid Approach: The Real Strategy

Most successful companies don't go "all self-hosted" or "all API." They use a hybrid strategy optimized for their specific constraints.

Use APIs for:
- Complex reasoning, chain-of-thought, multi-step problems (Claude, GPT-4)
- Latest model capabilities and research
- Variable or low-volume workloads (cost/performance predictable)
- Mission-critical features (vendor support matters)
- Features where output quality matters more than cost

Self-host for:
- Commodity tasks at massive scale (summarization of 100M+ documents)
- Privacy-critical operations (data can't leave your infrastructure)
- Specialized models you've fine-tuned on your domain (2-3x better than off-the-shelf)
- Cost-sensitive, high-volume operations where you've optimized infrastructure
- Offline/low-latency requirements (edge computing, mobile)

Example architecture for a B2B SaaS company with $50M ARR:
- Customer-facing intelligent features: Claude API. You're not optimizing for cost, you're optimizing for quality and reliability. Customer satisfaction matters more than saving 20% on compute.
- Internal analytics dashboards: Llama-70B self-hosted. Low-latency isn't critical. Cost matters because this touches every customer. Volume is massive (100M daily summarizations). Self-hosting wins. Infrastructure cost is $150k/year. API cost would be $500k/year.
- Data processing pipeline: Hybrid. High-volume commodity work (classification of documents) runs on self-hosted Llama. Tricky edge cases (unusual document types, ambiguous classifications) get routed to Claude API. 80% self-hosted, 20% API.
- Customer support: Llama-7B for first-pass analysis and routing. Claude for complex reasoning when human review is needed anyway.

This architecture gives you: cost optimization on high-volume commodity work, reliability and quality on customer-facing features, flexibility to use the best tool for each job, and reduces monthly AI spend by 35% compared to all-API.

The Hybrid Principle: Use APIs for what you can't do well yourself. Self-host for commodity work at massive scale or where you have competitive advantage. Most companies should be 70/30 API/self-hosted.

The Operational Reality of Self-Hosting

If you do self-host, understand what you're signing up for. Many teams underestimate these costs because they're not obvious in a spreadsheet.

Model Updates

New versions of Llama are released every 3-4 months. Mixtral variants come out. You face a constant decision: do we upgrade? Each upgrade requires testing your full pipeline with the new model, validating output quality, checking for API changes. Llama 3.1 to 3.2 changed tokenization. If your pipeline depended on specific token counts, you had to retest everything. You're committing to ongoing maintenance work, about 2-3 weeks per year per model.

Hardware Changes and Obsolescence

GPUs evolve. H100s are faster and cheaper than A100s now. But your system might have been optimized for the A100. Migrating means testing, potential downtime, re-optimization. And your original H100 becomes harder to source in 3 years. Planning a 5-year infrastructure lifecycle is expensive.

Scaling Headaches

One H100 can handle 100 requests per second. When you 2x traffic, one GPU isn't enough. Now you need distributed inference, load balancing, cache invalidation, request queuing. This is non-trivial engineering, probably 4-8 weeks of senior engineer time.

Monitoring and Reliability

Self-hosted models fail differently than APIs. A GPU can overheat (thermal failure). A model can get stuck (stuck in forward pass). Memory leaks can develop over days. CUDA OOM errors happen at 3am. You need sophisticated monitoring, alerting, and runbooks. And you need someone on-call to handle failures. Most teams don't budget for this.

Security

Your self-hosted model is a service exposed on your network. You need to secure it. Authentication to prevent unauthorized access. Authorization to limit who can call it. Rate limiting to prevent abuse. DDoS protection. Audit logging. This is another operational layer most teams don't plan for, and if you skip it, you've built a security vulnerability.

Case Study: Series B Company Operational Costs

A Series B company self-hosted Llama to save on API costs. Year one: infrastructure $120k, ops 0.7 FTE ($60k), opportunity cost $50k = $230k total. They processed 200M tokens/year. Year two, traffic grew. They added another GPU ($120k), ops was now 1.5 FTE ($150k). By year three, they had three GPUs, 2 FTE ops, distributed inference complexity, and annual costs of $500k. At that scale, Claude API would cost $140k/year. They'd spent $1M to avoid $280k in API costs. They've since migrated back to APIs with self-hosting for only their highest-volume, lowest-quality-requirement tasks.

If you're going to self-host, budget for all of this operational cost. Not after launch. Before. Include it in your business case. Often, you'll find it doesn't pencil out.

What to Do Monday Morning

Step 1: Calculate your actual token volume. How many tokens do you actually process per month? Use real data from logs, not estimates. Count both input and output tokens. Segment by use case (customer-facing vs. internal).

Step 2: Calculate API cost. Take your token volume. Multiply by Claude API rates. Account for which models you'd use (Opus vs. Sonnet vs. Haiku). What's the real cost? Include support/SLA costs if applicable. Most teams underestimate because they assume one model. You'll use different models for different workloads.

Step 3: Calculate self-hosting cost realistically. Infrastructure ($7-12k/month minimum for H100s, redundancy, storage). Operations (0.5-1 FTE, $50-80k/year). Opportunity cost of engineer time (0.25 FTE, $60-80k/year). Scaling costs when you grow (assume 50% more engineering in 18 months). Be honest. Most teams underestimate by 40-50%.

Step 4: Compare over 3 years. Calculate total cost of ownership for both paths. Include the cost of switching later if you choose wrong. If API cost is within 2x self-hosting cost, use APIs. The flexibility, reduced operational risk, and ability to switch models matters. If self-hosting is dramatically cheaper (3x+) because of your specific volume and stability, consider it for specific workloads.

Step 5: Build a hybrid strategy. Which workloads are customer-facing and need quality? (API). Which are internal and tolerant of lower quality? (Could be self-hosted). Which have massive volume where cost matters? (Self-hosted if stable, API if variable). Which are unpredictable in volume? (API, because you avoid overprovisioning). Build that architecture. Probably: 60% API, 40% self-hosted for commodity work.

Step 6: Document the decision. Write down why you chose what you chose. What assumptions did you make? Token volume, growth rate, quality requirements, operational appetite. Revisit this quarterly. Markets shift. Your answer might change.

FAQ: Open Source AI Decisions

Q: Should we be concerned about vendor lock-in with Claude?

A: If Claude's API became unavailable, you'd need to migrate. But if you've designed properly, it's mostly a config change. You change your LLM provider in your abstraction layer, revalidate, and you're done. The lock-in risk is real but overstated. The operational risk of self-hosting is significantly higher. Vendor risk is something to manage, not something to panic about.

Q: What about the "free software" principle? Shouldn't we avoid vendor lock-in for ethical reasons?

A: That's a valid principle if it matters to your company. But implementing it has real costs: $200-500k annually in operational burden. If you care about free software, that's fine, budget accordingly. But don't pretend it's free. Some companies decide the principle matters and budget for it. Others decide to be pragmatic and use APIs. Both are defensible. The indefensible path: pretending self-hosting is cheap.

Q: Can we use open source models via paid APIs to get the best of both worlds?

A: Yes. Together AI, Replicate, vLLM hosting, and others offer APIs for open source models. You get model choice and community flexibility with the operational simplicity of APIs. Cost is 2-5x higher than raw compute but still lower than engineering overhead of self-hosting. This is often the right middle ground.

Q: How do we know if our fine-tuned model is actually better than off-the-shelf?

A: A/B test it rigorously. Run the same inputs through your fine-tuned model and through Claude. Compare outputs on your specific domain tasks. Score them. If yours is better 85%+ of the time on your specific domain, it's worth self-hosting. If it's better 65-70%, probably not, the operational burden outweighs the benefit. If it's better 50-60%, definitely not.

Q: What if our data is sensitive and can't leave our infrastructure?

A: Then self-hosting or an on-premises model becomes a requirement, not a choice. The economics don't matter. It's a constraint. In this case, your question shifts from "should we self-host?" to "what's the minimum cost way to self-host?" Consider smaller models (Llama-7B, not 70B), less redundancy if you can tolerate downtime, or managed on-premises options. Cost is still $150-250k/year, but at least you're optimizing within the constraint.

Q: Are there cases where self-hosting actually wins financially?

A: Yes, at high volume. 500M+ tokens per month with stable, predictable workload and low quality requirements. At that scale, self-hosting cost drops per token and eventually becomes cheaper. But most companies never reach that volume. Be honest about your volume trajectory before you commit.

Key Insight

Self-hosting looks cheap ($0.50-1/M tokens) until you account for infrastructure ($84-144k/year), operations (0.5-1 FTE, $50-80k/year), and opportunity cost ($60-90k/year). Real cost: $200-300k/year. Use APIs for complex reasoning and variable workloads. Self-host only for massive predictable volume (500M+ tokens/month), privacy requirements, or domain-specific models 2-3x better than Claude. Hybrid strategy (70% API, 30% self-hosted commodity work) works best.

Being Pragmatic About Open Source

Open source is valuable. It accelerates innovation. It gives you options. But it's not free. It's not a magic cost reduction. It's a tradeoff with real costs.

Understand the tradeoff. Calculate your specific numbers honestly. Make the call based on your actual constraints, volume, and costs. That's how winning companies approach it.

The winners use APIs for most workloads and self-host for specific high-volume, low-quality-requirement, high-stability workloads. The losers try to self-host everything for "cost savings" and end up spending 3x more on operational overhead than they would have on APIs.

Default to APIs. Self-host when the numbers clearly justify it. Hybrid is best.

On This Page

Introduction
Economics of Self-Hosting
When It Makes Sense
Hybrid Approach
Operational Reality
Monday Morning Action
FAQ
Key Takeaway

Chapter Details

Part ofCh 2: Build, Buy, or Partner