Overview

Infrastructure decisions are strategic decisions, not technical ones. Where you train models, where you serve them, what hardware you use. These decisions cascade through your entire organization, affecting cost, latency, compliance, and team velocity.

Ten years ago, the choice was binary: on-premises or cloud. Today, it's more nuanced: cloud training, on-prem serving. Edge inference, cloud orchestration. Hybrid everything. Multiple models running in parallel. Some fine-tuned, some off-the-shelf.

Getting these decisions right is critical. Wrong infrastructure choices cost millions per year and create latency bottlenecks. A startup paid $5M/year for infrastructure they only needed $1.2M of because they over-provisioned. Another couldn't serve predictions in <100ms so their product felt slow. Get it right and you move fast. Get it wrong and you're constantly battling cost and latency.

The Infrastructure Decision Tree

Here's how to think about infrastructure strategy:

Training: Where do you train models? Almost always cloud. Why? Because you need massive compute for weeks at a time, then you're done. On-prem GPUs are too expensive to keep sitting idle. Cloud lets you spin up 100 GPUs for a week, then turn them off. You pay for what you use.

The only exception is if you have models so large that cloud costs become prohibitive (like training a trillion-parameter LLM), or if you have sensitive data you can't move to the cloud (regulated financial data, medical records). In those cases, on-prem training makes sense.

Serving: Where do you serve models (i.e., get predictions)? This depends on latency, volume, and cost.

• Cloud serving: For most use cases, cloud is fine. You get auto-scaling (traffic spikes, you automatically get more capacity), reliability (managed databases, automatic backups), and simplicity (you use a managed service). Cost is higher per request, but operational simplicity is worth it.
  - On-prem serving: For high-volume, low-latency use cases, on-prem makes sense. If you're serving millions of recommendations per second with latency requirements of

Orchestration: Where do you orchestrate everything? Usually cloud. You need a control plane that manages training jobs, monitors serving infrastructure, handles deployment pipelines. Cloud platforms (AWS, GCP, Azure) are better at this than building on-prem.

The Decision Framework: Train in cloud (unless you have massive models or extreme data sensitivity). Serve where latency and cost tradeoffs favor it (usually cloud, sometimes edge, rarely on-prem). Orchestrate in cloud. This is the pattern that scales from startups to enterprises.

Cost Optimization

The biggest mistake technology leaders make is overspending on infrastructure. They provision for peak load all the time. They use premium hardware when commodity hardware would work. They don't take advantage of discounting because it's annoying.

Here's a better approach:

For training: Use spot instances (spare cloud capacity that's much cheaper). You might get pre-empted, but for training jobs that can checkpoint, that's fine. You save 70% of the cost. Use commodity hardware unless your model requires specialized hardware.

For serving: Right-size your instances. Start with small instances, monitor actual usage, scale up only if needed. Use reserved instances (you commit to 1-3 years, get a discount) for baseline load, and on-demand for spikes.

For storage: Tiered storage (hot data on fast storage, warm data on cheaper storage, cold data on archival storage) reduces costs significantly. Not all of your data needs to be in expensive databases.

For data transfer: Data egress from cloud is expensive. Minimize it. Process data where it lives. If you're moving data to on-prem for processing, that costs money. Do the processing in cloud instead.

The companies that are best at cost optimization don't just provision infrastructure. They engineer for efficiency: they write code that uses less compute, they design algorithms that are faster, they structure data so queries are cheaper. Infrastructure cost is ultimately a software engineering problem.

Latency and Performance

Some applications need sub-millisecond latency. Real-time fraud detection needs to block transactions in milliseconds. Autonomous vehicles need decisions in milliseconds. High-frequency trading needs microseconds.

For these applications, cloud infrastructure is not acceptable. Latency from network round-trips is too high. You need on-prem or edge infrastructure.

But here's the thing: most applications don't need this. Most applications can tolerate 100-500ms latency. Your recommendation engine can take 200ms to compute. Your chatbot can take 1 second. Your content moderation can take 5 seconds. For these, cloud serving is fine.

The discipline is: explicitly define your latency requirements. "This API needs <50ms response time." Then design infrastructure to meet that. Don't over-build just because latency is theoretically important.

Compliance and Data Locality

Some industries have regulations that restrict where data can be stored. Healthcare data must stay in the US (sometimes a specific state). Financial data must stay within the country. European data must stay in Europe.

If you're in one of these industries, your infrastructure strategy is constrained. You might need on-prem training because you can't move data to cloud. You might need regional cloud deployments.

The complexity is when you want to leverage modern AI platforms (like large language models) that are cloud-hosted. Some companies solve this by:

• Private deployments: Deploy the model on your own infrastructure. You lose some benefits of the shared platform, but you maintain control.
  - Anonymization: Anonymize data before sending to cloud platforms. You lose some signal, but you maintain compliance.
  - Federated learning: Train models across distributed data without centralizing it. This is more complex, but it's becoming more common.

Compliance constraints are real. Don't ignore them. But don't over-optimize for them either. Most data isn't sensitive enough to require special handling.

Hybrid and Multi-Cloud

Some organizations run "hybrid" infrastructure: some systems on-prem, some on cloud, trying to get the best of both worlds. In practice, hybrid often means: high operational complexity, high costs, no real benefits.

The companies that get hybrid right are very thoughtful: they have a clear reason for each piece being where it is. And they have excellent DevOps so that deploying to hybrid doesn't require different tools and processes.

Multi-cloud (using AWS and GCP and Azure) is similarly complex. The only reason to do it is if you have a specific requirement (one vendor is too expensive for X, another vendor is too expensive for Y). Otherwise, stick with one cloud. Complexity costs money.

Building Your Serving Infrastructure

Model serving is harder than it looks. You need:

• Model hosting: Where the model runs. This might be a Docker container on Kubernetes, a managed service like SageMaker, or custom code.
  - Feature serving: Getting features to the model in milliseconds. This requires caching and fast lookups.
  - Request handling: HTTP API or gRPC endpoint that accepts requests, gets features, runs the model, returns predictions.
  - Versioning: Multiple versions of the model running at once. New versions get A/B tested. Old versions get rolled back if new ones perform poorly.
  - Monitoring: Is the model running? Is latency good? Is accuracy good? Alerting when something goes wrong.
  - Auto-scaling: Traffic spikes? Add more instances. Traffic drops? Remove them.

Building this from scratch takes months. Using a managed service (AWS SageMaker, Google Vertex AI, or open-source frameworks like KServe) accelerates it. You lose some flexibility, but you save huge amounts of time.

The Multi-Model World

As you scale, you'll have dozens or hundreds of models in production. Managing them becomes complex. You need a model registry: what models exist, what version is in production, who owns each one, what's the SLA.

You need a model serving platform that can efficiently host all of them. You can't spin up a separate server for each model. You need consolidation: one server runs multiple models, models share compute, you're ruthlessly efficient with hardware.

This is where infrastructure becomes a competitive advantage. If you can serve 10,000 models efficiently on modest hardware, your cost per model is negligible. Competitors who spin up a server per model have cost per model that limits their ambition. You can afford to build more models. You'll have better products.

The Monday Morning Action: Audit your infrastructure. Where are your models trained? Where are they served? What's your cost per model? What's your latency? Are you over-provisioned? These are the levers that determine whether you're agile or slow.

Failure Mode: The Over-Provisioned Starter

A startup built on AWS with best practices: high availability, auto-scaling, managed databases, CDN, the whole package. Cost: $40k/month for what should've been $5k/month. Why? They provisioned for 10M monthly users when they had 10k. They used premium database tiers when they had 1GB of data. They didn't need high availability yet. They needed cheap.

By month 6, they'd run through their runway fighting infrastructure costs. They never got to find product-market fit because they were fighting the cloud bill.

The lesson: Start simple and cheap. You can optimize when you have real load to optimize for. Don't pay for future scale you might never reach.

Real Case Study: ML Platform Cost Reduction

A Series B company training 200 models per month spent $800k/month on cloud GPU costs. Their infrastructure team ran experiments: batch training on cheaper regions, spot instances for fault-tolerant jobs, consolidating small models onto shared GPUs, using quantization to reduce compute requirements.

Result: Same throughput, $240k/month. Savings: $560k/month. This was engineering work, not magic. Attention to cost. They hired one person focused on ML infrastructure optimization. Payback: 1 week.

FAQ

Q: Should we build our own model serving platform or use AWS/GCP/Azure?

A: Unless you have very specialized requirements (extreme scale like >100k predictions/sec, extreme latency like $100k/month on GPUs.

Q: Should we do on-device or edge deployment?

A: Edge deployment (models on phones, browsers, IoT) is increasingly important. Advantages: privacy (data stays on device), offline capability, low latency (no network round trip). Constraints: models must be small (< 100MB for phones, <5MB for browsers), less compute available, harder to update. Use case: features where latency is critical (AR filters) or privacy is required (medical, financial). For most cases, cloud inference is fine.

Q: What if I'm wrong about infrastructure requirements?

A: You'll be. Nobody predicts scale perfectly. Build for current requirements + 50% buffer (not 10x buffer). When you hit limits, optimize or upgrade. This costs less than over-provisioning from day one. Plan to iterate.

Key Takeaway

Infrastructure decisions are strategic. Train in cloud for flexibility and cost. Serve where latency and volume dictate (usually cloud, sometimes edge). Optimize ruthlessly for cost. Use managed services except where you have specialized requirements. Get infrastructure strategy right, and you move fast and stay efficient. Get it wrong, and you're slow and expensive.

Now that you have the infrastructure, let's talk about how to turn it into a platform that others can build on.

On This Page

Watch the Lecture
Decision Tree
Cost Optimization
Latency and Performance
Compliance
Hybrid and Multi-Cloud
Serving Infrastructure
Multi-Model World
Failure Modes
Case Study
FAQ

Chapter Details

Part ofAI Platform Strategy