Overview

Here's the budget disaster that happens: You estimate token costs at $5k/month. You launch. Actual cost is $50k/month. You're over budget by 10x. In one documented case, a SaaS platform experiencing unexpected scale discovered their API token spend had increased 10x after just three months of production use.

Or: you budget for one engineer to build and maintain. It takes three, plus a data engineer, plus DevOps time. Your "AI project" is now consuming 20% of the eng team.

These aren't failures of the AI initiative. They're failures of estimation. Almost no one correctly estimates the total cost of AI ownership.

This is about having a framework to get the estimate right the first time.

The Cost Categories: What to Budget For

1. Direct Model Costs

This is what everyone thinks about. Token costs for API calls or compute costs for self-hosted.

How to estimate:
- Token per request: test with real data
- Requests per day/month: get historical metrics or make an educated guess
- Token cost: use published pricing
- Calculate: (tokens per request × requests × token cost) × 1.2 (20% buffer for growth)

Example:
- Customer support summaries: 2,000 tokens per summary
- 100 summaries per day
- $0.02 per 1M tokens (Claude)
- Cost: 2,000 × 100 × 30 × 0.02 / 1M = $120/month for tokens

Reality check: Get historical usage data from your pilot. Don't extrapolate from guesses.

2. Infrastructure and Hosting

If you self-host or have custom infrastructure:

• Compute (GPUs, CPUs): $5-50k/month depending on scale (based on typical cloud pricing for AI workloads)
  - Storage (model weights, outputs, logs): $1-10k/month (varies by data retention and volume)
  - Networking (egress, APIs): $1-5k/month (typical range for production AI systems)
  - Monitoring and observability tools: $1-2k/month (industry standard tooling)
  - Backups and redundancy: included in above but plan for 20% overhead

If you use APIs: mostly zero. Your cost is token costs.

3. Data Preparation and Management

This is where most budgets go wrong. People think: "we'll use our existing data." Actual: data is messy, needs cleaning, needs feature engineering.

• Data labeling (if needed): $10-50k depending on volume
  - Data infrastructure (pipelines, cleaning): $5-20k/month (this is a recurring cost, not one-time)
  - Feature engineering: $20-50k in the first year
  - Data validation and quality checks: $2-5k/month

For a moderately complex AI system, expect data costs to be 40-50% of total AI costs. This aligns with common patterns across organizations building custom ML systems, data preparation typically consumes more resources than the engineering build itself.

4. Personnel Costs

This is the big one most CTOs underestimate.

• ML Engineers: $150-250k salary + 30% benefits = ~$200-325k per person per year. You'll likely need 1-3 depending on complexity.
  - Data Engineers: $130-180k + 30% = ~$170-235k per year. You'll need 1 for data infrastructure.
  - Product Manager (dedicated to AI): $120-180k + 30% = ~$156-234k per year. Even partial time (0.5 FTE) is $70-120k/year.
  - Domain expert (part-time): Depends on domain. Could be 10 hours/week at $150/hour = ~$80k/year.
  - QA/Testing (dedicated): $80-120k + 30% = ~$105-156k per year for 0.5 FTE.

**Real example for a moderately complex AI system:
- 1 ML Engineer: $250k
- 0.5 Data Engineer: $120k
- 0.5 Product Manager: $120k
- Total: $490k/year in person costs**

This is the cost nobody mentions when they talk about "adding AI." But it's real and it's massive. A Series A company estimated their AI customer support project would need "one AI engineer." Actual: they needed 1 ML engineer, 0.5 data engineer, 0.5 product manager, plus 10 hours/week of customer support expertise for training data. Total: $450k in personnel. They budgeted $80k (for one engineer). The surprise derailed the project when they had to request additional headcount mid-year.

5. Vendor and Tool Costs

• Fine-tuning platform (if using external): $1-5k/month
  - Evaluation frameworks and testing tools: $1-3k/month
  - Monitoring and alerting for AI models: $2-5k/month
  - Security scanning for AI outputs: $1-3k/month
  - A/B testing and experimentation: $1-2k/month

6. Ongoing Maintenance and Iteration

Your system doesn't ship and then run itself. You'll need:

• Bug fixes: ~10% of initial build cost per quarter
  - Model retraining: depends on how often data changes, but budget $5-20k per retraining cycle
  - Monitoring and updates: 20% of an engineer's time ongoing
  - Documentation and knowledge transfer: not usually budgeted but significant

Plan for maintenance to be 20-30% of initial build cost annually.

7. Hidden Costs

These kill budgets:

• Latency/performance: If your AI feature makes the application 2x slower, you need infrastructure upgrades. Add $10-50k.
  - Hallucinations and false positives: If your AI makes mistakes, you need human review processes. That's people cost.
  - Compliance and auditing: Depending on industry, you might need audit trails, compliance documentation. $5-30k depending on rigor.
  - Privacy and data residency: If data can't leave your country or needs special handling, add $20-100k to infrastructure costs.
  - Integration surprises: Systems don't integrate as cleanly as you expect. Budget 20% extra engineering time.

The Total Cost Principle: Based on analysis of successful deployments across industries, token costs are typically 10-20% of total AI ownership cost. Personnel is 40-60%. Infrastructure is 10-20%. Data is 10-15%. Everything else is 5-10%. Budget accordingly.

Cost Failure Mode: A fintech company budgeted $200k for an AI feature thinking they'd buy an API and slap it in. Actual costs: $400k in first year (double initial estimate). Why? Data labeling cost 3x what they expected, models needed more tuning, and they underestimated integration work. Lesson: use a detailed cost model. Budget 20% for contingency. Review monthly against actual. Adjust scope if costs creep.

Building a Cost Model for Your AI Project

Use this template. Fill in real numbers. Update quarterly.

Model Costs (Year 1):
- Tokens/month: [X]
- Token cost: [Y]
- Total annual token cost: [X × Y × 12]

Infrastructure (Year 1):
- Compute: $[X]/month
- Storage: $[Y]/month
- Networking: $[Z]/month
- Total annual: $[X+Y+Z] × 12

Data (Year 1):
- Labeling/annotation: $[X] (one-time mostly)
- Infrastructure: $[Y]/month
- Feature engineering: $[Z] (mostly Year 1)
- Total annual: $X + (Y × 12) + Z

Personnel (Year 1):
- ML Engineers (FTE): [#] × $250k
- Data Engineers (FTE): [#] × $150k
- Product Manager (FTE): [#] × $150k
- Domain expert (FTE): [#] × salary
- Total annual: $[sum]

Tools and Vendors (Year 1):
- Fine-tuning: $[X]/month
- Monitoring: $[Y]/month
- Testing tools: $[Z]/month
- Total annual: $[X+Y+Z] × 12

Contingency (20%):
- Total × 0.2

GRAND TOTAL YEAR 1: $[sum all above]**

This is your real budget. Share it with the board. Adjust for your specific situation (cheaper labor markets, different model choices, etc.) but the framework is right.

Cost Optimization Strategies

Once you understand costs, optimize them.

Model Selection

Cheaper models might save 50-70% on token costs. But if accuracy drops 10%, is that worth it? Calculate the trade-off: cost savings vs. business impact of lower quality. These ranges reflect typical delta between frontier and mid-tier models in production environments.

Prompt Efficiency

Well-optimized prompts use 20-30% fewer tokens. This is free optimization and a commonly observed improvement in production deployments. Invest time here.

Caching Strategies

If you're hitting the same prompts repeatedly, cache them. Prompt caching (supported by Claude) can save 50-80% on repeated requests for the same context, making it valuable for retrieval-heavy workloads.

Hybrid Models

Use expensive models (Claude) only for complex reasoning. Use cheap models (open source) for commodity tasks. Organizations using tiered model strategies typically report 60-80% cost reductions compared to using a single frontier model across all tasks.

Async Processing

Not everything needs synchronous AI. Pre-generate summaries offline. Use batch processing for classification. Move expensive work to off-peak hours.

Feature Importance**

Not all AI features deliver equal ROI. Track which features generate value. Cut the ones that don't. Reallocate budget to winners.

When This Goes Wrong: Cost Overrun Scenarios

Scenario 1: The "Cheap Model" Trap**

A SaaS company decided to use cheaper open-source models instead of Claude to save money. Model cost: $200/month vs. $2k/month with Claude. Savings: $21,600 annually. Sounds great.

But the open-source model had 15% lower accuracy. Their AI feature (code generation suggestions for developers) was less useful. Engineering support tickets about AI suggestions tripled. They had to hire additional customer support staff. What started as a $21k savings turned into a $60k cost increase (extra support staff). Plus: reputational damage, developers started avoiding the feature. They ended up switching back to Claude 8 months later, having wasted $40k.

The lesson: token cost savings of 50% might require 200% spending on mitigation (support, retraining, etc.). Model selection isn't just about token price; it's about total cost of ownership including support burden.

Scenario 2: The Hidden Data Cost**

A healthcare company estimated their AI project at $300k: 2 engineers, infrastructure, token costs. Data preparation: "we have good internal data, shouldn't cost much."

Reality: their data was messy. Patient records had inconsistent formatting (20 years of legacy data entry). Medical terminology wasn't standardized. 40% of data had missing critical fields. Cleaning cost $120k (3 months of a data engineer). They also needed to hire a medical domain expert to validate labeled data. Total actual cost: $480k.

The lesson: assume data is worse than it looks. Budget 40-50% of cost for data prep. If you skip this budget item, you'll discover it mid-project when you're out of time and money.

Scenario 3: The Integration Nightmare**

A fintech company estimated: AI model development 60%, infrastructure 15%, integration 5%. Budget: $200k (60%), $50k (15%), $17k (5%).

Integration took 4 months instead of 2 weeks. Why? Their trading system was decades old. APIs didn't exist. Had to build middleware. Had to deal with regulatory requirements (audit trails, compliance). Had to coordinate with legacy system teams. Integration went from $17k to $140k.

Total project: $290k instead of $267k. More importantly, launch delayed 3 months, missing a market window. The savings from under-budgeting integration cost them millions in lost first-mover advantage.

The lesson: integration work is unpredictable. Add 30-50% buffer for "integration surprises" especially if integrating with legacy systems.

Scenario 4: The False Start**

A company committed to an AI approach, built for 6 months, spent $400k, then discovered the approach wouldn't work at scale. Performance degraded too much with larger datasets. Had to restart with a different architecture.

Total spend: $800k for one year of work, with nothing shipped to customers. The failure wasn't technical; it was estimation. They should have done a 4-week spike ($50k) to validate the approach before committing $400k.

The lesson: expensive approaches need spike validation early. Never commit 6 months to an unvalidated architecture. Spend 2-4 weeks validating the approach on real data. If it works: commit. If it doesn't: find a different approach, having wasted only $50k instead of $400k.

What to Do Monday Morning

Step 1: Build a cost model. Use the template above. Get real numbers from pilots if you have them.

Step 2: Break it out by category.** Tokens, infrastructure, data, people, tools, contingency. See where money actually goes.

Step 3: Identify biggest costs.**** Usually it's people. Usually it's data. Design your approach to minimize these.

Step 4: Get board alignment.** Show them the real number. If it's $500k+ in personnel alone, they need to know that going in, not discovering it mid-year.

Step 5: Build financial tracking.** Monthly, review actual vs. budgeted. Where are you surprised? Adjust.

FAQ: Cost Questions

Q: Isn't using a cheaper open-source model just smarter budgeting?**

A: Only if you account for the true cost of ownership. Open-source might save 80% on tokens but costs 50-100% more in personnel (you need ML engineers for tuning and DevOps), infrastructure (you need GPU servers), and support (lower accuracy = more support burden). True savings: 20-40% at best, and only if your team is skilled at managing models. For most organizations, paying 10x more for Claude but needing 40% fewer engineers is actually cheaper. Compare: $2k token cost vs. $3k token cost with 1 fewer engineer ($125k annually). The cheaper tokens cost more overall.

Q: Can we reduce costs by outsourcing the ML work to contractors?**

A: Yes, if you know exactly what you want built. No, if you're exploring (which most AI projects are). Contractors are cheap per hour but expensive per outcome when requirements are unclear. You end up paying for rework, unclear deliverables, knowledge walking out the door. Better approach: hire one full-time ML engineer ($250k) who understands your business deeply and can pivot quickly. Spending $250k on one great engineer beats spending $80k each on three contractors who don't understand your domain.

Q: If we use off-the-shelf AI features from vendors (like Salesforce Einstein), isn't that cheaper than building?**

A: Usually yes, but only if they solve your exact problem. You save on engineering (no ML team needed) but pay vendor licensing (often $10-50k+ annually). Integration is usually cleaner. But: you lose differentiation (competitors can buy the same tool), you're locked into the vendor's approach, and features you need that the vendor doesn't support require workarounds. The math: off-the-shelf saves $200-400k in build cost but loses competitive advantage. Build only if you need differentiation. Buy if you need commodity features fast and don't care about being unique.

Q: Our CFO says this is too expensive. What do we do?

A: Either reduce scope (fewer AI features, smaller team, longer timeline), reduce quality targets (accept lower accuracy), or reframe the ROI (show what you can't do without this). Usually it's a combination. Don't try to fund it at lower cost; that's how you end up with failed projects.

Q: Can we do this cheaper with open source?

A: Potentially. You save on token costs but add infrastructure and personnel costs (you need ML engineers and ops). Net savings are often 20-40%, not 80%. The trade-off: you own more risk and complexity.

Q: What's reasonable to spend on AI as % of engineering budget?

A: Depends on your strategy. If AI is core to your business: 20-30% of engineering budget. If AI is a feature (one of many): 5-10%. If you're spending more than 30% on AI relative to business value, you're probably over-investing.

Q: How do we track whether we're over budget?

A: Monthly financial review: actual vs. budgeted spend by category. If tokens are 2x budget, why? Are you using the model inefficiently? Need to optimize prompts? If personnel is 50% over, why? Is scope creep happening? Need to add headcount or replan?

Key Takeaway

Token costs are 10-20% of total AI cost. Personnel is 40-60%. Everything else is 20-30%. Most budgets fail because they underestimate people and data costs. Build a detailed model: tokens, infrastructure, data, people, tools, contingency. Review monthly. Adjust based on actual vs. budget.

Real-World Cost Examples

Example 1: E-Commerce Search Optimization A mid-market e-commerce company budgeted $150k for an AI-powered search ranking system. Reality: tokens $2k/month, infrastructure $3k/month, data engineering $80k (one-time for pipeline setup), ML engineer 0.5 FTE ($125k), contingency $35k. Total year 1: $280k. They discovered data engineering (cleaning product descriptions, building training datasets) was 30% of cost, not included in initial estimate. Lesson: allocate specifically for data work.

Example 2: Customer Support AI Support team wanted AI chatbot. Estimated: token costs $5k/month. Actual: tokens $8k, infrastructure $2k, data annotation (creating training data from past tickets) $40k, ongoing human review (for accuracy) $25k/month, model monitoring $2k/month, customer support for the AI feature itself $3k/month. Year 1 total: $282k. They expected $60k. The surprise: human-in-the-loop (reviewing AI responses) cost way more than the AI itself.

Example 3: Demand Forecasting (Winner)** Finance team built demand forecasting AI for inventory planning. Initial estimate: $200k. Actual: tokens $1k/month, data infrastructure $4k/month, one data engineer 0.5 FTE ($75k), one ML engineer ($150k), contingency ($50k). Year 1: $245k. They came in under estimate. Why? They didn't need annotation (historical sales data was clean). They had one dedicated engineer instead of requiring extensive cross-team coordination. Clear ROI (saved $2M in inventory cost) meant organizational support and smooth implementation. Lesson: clear ROI enables realistic budgeting.

Budgeting as Strategy

Getting the budget right isn't just about money. It's about setting expectations. It's about making realistic promises. It's about knowing what you can actually afford. Most AI initiatives fail not because the technology is bad, but because the investment doesn't match the ambition. Get the budget right and everything else gets easier. The organizations that succeed with AI don't just have the best technology; they have the most realistic expectations and best-funded teams.

On This Page

Introduction
Cost Categories
Cost Model
Optimization
Monday Morning Action
FAQ
Real-World Examples
Key Takeaway

Chapter Details

Part ofCh 2: Build, Buy, or Partner