Hook

You're shopping for GPU infrastructure and realize there are hundreds of options. Tesla V100? H100? A100? AMD? Nvidia? On-premises or cloud? Batch training or real-time inference? The specifications feel meaningless, and the vendors' marketing doesn't help.

Your infrastructure engineer says "we need enterprise-grade GPUs for production." Your data scientist says "any GPU works fine for research." Your CFO says "get the cheapest option." They're all partially right, which means they're all talking past each other.

This lesson cuts through the noise. It shows you what different GPU types are actually good at, how to size infrastructure for your actual workload, how to optimize costs, and when to buy versus rent. By the end, you'll understand what your teams are actually asking for and why.

Purpose

AI infrastructure is fundamentally different from traditional IT infrastructure. Traditional servers do roughly similar work (run applications, serve requests). AI infrastructure needs to handle drastically different workloads:

• Training, Move massive amounts of data through compute, often for hours or days. GPUs shine here.
  - Inference. Take a trained model and make predictions on new data. Could be batch (process 1M records overnight) or real-time (respond in milliseconds).
  - Serving. Keep a model running, responding to requests, handling varying loads.

Each workload has different needs:

• Training needs raw compute power and memory bandwidth
• Real-time inference needs low latency and high throughput
• Batch inference needs sustained compute at lower cost

This lesson teaches you how to right-size infrastructure for your actual workload, not some generic "AI infrastructure."

Why This Matters

Getting GPU infrastructure wrong is expensive and operationally painful:

First, you can buy the wrong GPUs. You spec high-end A100s for a workload that fits on V100s. You spend 2x on hardware you don't need. Or you buy cheap K80s that are too slow for your workload, and training takes twice as long.

Second, you can provision the wrong amount. You buy 10 GPUs for a workload that needs 40. Your data scientists queue their jobs and wait days to get GPU time. Or you buy 100 GPUs for a workload that uses 10, and the rest sit idle, costing money.

Third, you can pick the wrong GPU type for the workload. Some GPUs are optimized for dense matrix operations (training). Others are optimized for inference. Using the wrong GPU type means poor utilization and higher cost per prediction.

Fourth, you can structure inference badly. You run one model per GPU server. Real traffic spikes cause 10x demand. You panic and buy more GPUs. Then traffic normalizes and GPUs sit idle. With proper serving infrastructure (batching, load balancing, multi-model serving), you can handle the same demand on 1/3 the GPUs.

Getting this right means:

• Right GPU type for your workload
• Right number of GPUs (neither over- nor under-provisioned)
• Right serving infrastructure to use GPUs efficiently
• Right cost optimization strategies

This saves 30-50% on infrastructure costs and prevents scaling crises.

Core Concepts

Key Insight: GPU Types and Their Purpose

Training-Optimized GPUs (Nvidia A100, H100, L40S; AMD MI300)

• High memory bandwidth (for moving training data through compute)
  - Large memory (80GB+ to fit large models and batches)
  - Good for mixed-precision training (FP32, FP16, TF32)
  - Cost: $10K-$15K per GPU
  - Cloud: $2.00-$4.00/hour

Use for:

• Training large models (GPT-scale, vision transformers)
• High-batch-size training
• Multi-GPU distributed training

Inference-Optimized GPUs (Nvidia L4, T4, L40; AMD MI25)

• Lower memory footprint (enough for model weights + batch)
  - Optimized for lower latency at inference
  - Good for batched inference
  - Cost: $2K-$8K per GPU
  - Cloud: $0.35-$1.00/hour

Use for:

• Real-time or near-real-time inference
• High-throughput batched inference
• Cost-sensitive inference workloads

Multi-Purpose GPUs (Nvidia A100, H100)

• Powerful enough for training or inference
  - More expensive than specialized options
  - Makes sense if you do both training and inference

Avoid:

• Older GPUs (K80, M40, P100) unless you have a huge existing fleet. They're power-hungry and slow.

Key Insight: Inference Patterns and Infrastructure

Most organizations underestimate the complexity of inference. Training is straightforward: load model, load data, run training loop. Inference is trickier because it needs to be:

• Low latency, Respond quickly to requests
• High throughput, Handle thousands of requests per second
• Cost-efficient, Not waste GPU resources
• Reliable, Handle failures gracefully

Pattern 1: Batch Inference

Process a batch of data at once (e.g., 100K predictions/night). Infrastructure:

• Simple: Load model once, process batch, return results
• Cost-efficient: One GPU serves entire batch
• Latency not critical: Can process overnight
• Example: Nightly churn prediction on entire customer base

Pattern 2: Real-Time Single-Request Inference

Each request is a single prediction (e.g., "is this email spam?"). Infrastructure:

• Simple API that takes a request, returns prediction
• Latency critical: Need to respond in <100ms
• Challenge: Many single requests come in throughout the day
• Problem: One request per GPU is wasteful (GPU throughput unused)

Solution: Batching layer: Queue requests, send them to GPU in batches of 32-64, reduce latency with smart scheduling.

Pattern 3: Real-Time Streaming Inference

Continuous stream of data (e.g., IoT sensors, event streams). Infrastructure:

• Data arrives continuously, model must score continuously
• Latency critical: Need fresh predictions quickly
• Infrastructure: Message queue (Kafka) → Batch processor → Results

Pattern 4: Multi-Model Serving

Multiple models running on same infrastructure (e.g., one model for fraud, one for churn, one for recommendations). Infrastructure:

• Can't give each model its own GPU (too expensive)
• Need model serving platform (KServe, Seldon, TFServing) that:
• Loads/unloads models on-demand
• Routes requests to right model
• Batches requests across models
• Handles model versioning

Key Insight: Capacity Planning Formula

To figure out how many GPUs you need:

GPUs_needed = (Requests_per_second × Inference_latency_seconds) / (GPU_throughput × Batch_size × Utilization_target)

Example:

• 1,000 requests/second
• Inference latency: 50ms (0.05 seconds)
• GPU can process 1,000 inferences/second for a batch of 32
• Batch size: 32
• Target utilization: 70% (leave headroom for traffic spikes)

GPUs = (1000 × 0.05) / (1000 × 32 × 0.7) = 50 / 22,400 = 0.002 GPUs

Wait, that's less than one GPU! This means one GPU can easily handle 1,000 req/sec if you batch properly.

Now add a traffic spike (3x peak):

• 3,000 requests/second → need 0.007 GPUs → still less than 1

But if you don't batch and process each request separately:

• 3,000 requests/second, 50ms per request, 1 request per GPU inference
• GPUs = 3,000 × 0.05 = 150 GPUs needed!

The difference: with batching, 1 GPU. Without batching, 150 GPUs. This is why serving infrastructure matters so much.

Key Insight: Cost Optimization Strategies

Right-Sizing

Most organizations over-provision. They buy GPUs "just in case." Instead:

• Measure actual utilization for 3 months
• Right-size infrastructure to actual demand
• Build autoscaling to handle peaks

Savings: 20-30% by removing unused capacity

Batch Processing

Use batch processing wherever possible instead of real-time:

• Nightly batch inference: $50 for 1 hour of GPU time
• Real-time inference: $500/day to keep GPU warm

Batch costs 10x less. If latency tolerance is >1 hour, batch.

Inference Optimization

Smaller models, smarter serving:

• Quantization (16-bit or 8-bit instead of 32-bit): 2-4x faster, lower memory
• Model distillation (smaller model that mimics larger one): 10x smaller, nearly same accuracy
• Batching and multi-model serving: use GPU capacity more efficiently

Savings: 30-50% by optimizing inference

Cloud Reserved Instances

If you have predictable, steady-state workloads, buy reserved capacity instead of on-demand:

• On-demand: $4.00/hour (P100)
• 1-year reserved: $2.50/hour (38% savings)
• 3-year reserved: $1.80/hour (55% savings)

Only use if you're confident you'll need that capacity for 1-3 years.

Spot/Preemptible Instances

For workloads that can tolerate interruption (training, batch processing), use cheap, interruptible cloud instances:

• On-demand: $4.00/hour
• Spot (AWS) or preemptible (GCP): $1.20/hour (70% savings)

Risk: can be interrupted anytime, but good for non-critical workloads.

Practical Use Cases

Use Case 1: Planning GPU Infrastructure for Distributed Training

A company is training large language models. They need to figure out infrastructure.

Model: 7B parameters, training on 500B tokens

Team: 5 ML engineers doing parallel experiments

Timeline: Training takes 20 days on 8 GPUs

Capacity Plan:

Option A: Buy enough GPUs for one full training (8 GPUs, on-premises)

• Cost: $120K CapEx
• One person can train at a time
• Others wait (unproductive)

Option B: Buy for 3 concurrent trainings (24 GPUs)

• Cost: $360K CapEx
• Higher utilization
• Still might not be enough (everyone wants to train at same time)

Option C: Use cloud, pay per use

• Cost: $4/hour × 8 GPUs × 480 hours = $15,360 per training
• 5 concurrent trainings (if everyone trains same time) = $75K
• After 5 trainings, paid for on-premises infrastructure

Better approach: Hybrid

• Build small on-premises cluster (8 GPUs, $120K) for quick experiments
• Use cloud for full-scale training (when model is ready)
• Dev/test on on-prem (quick iteration)
• Production training on cloud (cost-efficient at scale)

This balances cost, utilization, and speed.

Use Case 2: Sizing Inference Infrastructure for E-Commerce

An e-commerce company needs real-time recommendations. Current state:

• 100M daily active users
• 1M recommendations/day (1 per 100 users)
• Average latency must be <100ms
• Model is ResNet-50 (122M parameters, 100MB)

Requirement: 12 req/sec average (100M/day ÷ 86,400 seconds)

Peak: 50 req/sec (assume 4x traffic spike at peak hours)

Single-GPU Inference Calculation:

ResNet-50 throughput on T4 GPU: ~500 images/sec at batch size 32

Batching strategy:

• Queue requests
• Send batch of 32 to GPU every 10ms
• Throughput: 32 inferences / 10ms = 3,200 inferences/sec
• Latency: P99 = <100ms (batching introduces <50ms overhead)

Capacity needed:

• Average load: 12 req/sec → 1 T4 GPU handles 3,200 req/sec → massive headroom
• Peak load: 50 req/sec → still easily handled by 1 GPU
• Add 50% headroom for spikes: 1 GPU is enough

Cost:

• Cloud on-demand: 1 × T4 at $0.35/hour × 730 hours/month = $255/month
• Plus data transfer and model serving platform ($100/month)
• Total: ~$400/month

Alternative (bad): Single-request inference without batching

• P99 latency 100ms → GPU can handle 10 req/sec
• Peak of 50 req/sec → need 5 GPUs
• Cost: 5 × T4 at $0.35/hour = $1,277/month
• Bad decision: 3x cost for same latency!

This shows why serving infrastructure matters.

Use Case 3: Planning Infrastructure for Multiple Projects

An organization has 3 projects:

1. Demand Forecasting - Trains daily, needs 1 GPU for 2 hours
2. Churn Prediction - Trains weekly, needs 4 GPUs for 3 hours
3. Anomaly Detection - Continuous inference, needs steady compute

Daily Schedule:

• 2am-4am: Demand forecasting (1 GPU)
• 4am-7am: Churn prediction (4 GPUs, happens 1x/week)
• 24/7: Anomaly detection inference

Approach 1: Static allocation

• Allocate 1 GPU for forecasting
• Allocate 4 GPUs for churn
• Allocate 2 GPUs for anomaly detection
• Total: 7 GPUs × 24/7
• Cost: 7 × $1/hour × 730 hours = $5,110/month

Approach 2: Shared, elastic allocation

• Anomaly detection: 2 GPUs (can flex to 3-4 during high-traffic periods)
• Training jobs (forecasting + churn): shared 2 GPU pool, auto-scaling
• Prediction: use inference batching, run on 1-2 GPUs
• Peak capacity: 4 GPUs
• Average: 2.5 GPUs
• Cost: 2.5 × 730 = $1,825/month

Cost difference: $3,285/month (64% savings) by sharing infrastructure and using elastic allocation.

Examples

Example 1: GPU Sizing Worksheet

Project Name: [Project]

Workload Type: [ ] Training [ ] Inference [ ] Both

For Training:

• Model size: ***_* parameters
• Batch size: ***** samples
• Training dataset size: ***__ GB
• Desired training time: _____ days
• Calculate: (Dataset_size + Model_size + Activations) / (GPU_memory)
• If result > 1, you need multiple GPUs or gradient checkpointing
• If result > number of GPUs you have, training won't fit

For Inference:

• Requests/second: ***_* avg, ***** peak
• Latency requirement: ***__ ms
• Model size: ***_* MB
• Expected batch size: **___ samples
• Calculate: (Req/sec_peak × Latency_ms) / (GPU_throughput × Batch_size)

Infrastructure Recommendation:

• GPU type: ***_* (training vs. inference optimized)
• Quantity: *****
• On-premises or cloud: ***__
• Estimated cost: _____/month

Example 2: Cost-Benefit of Optimization

Scenario: Your inference infrastructure needs optimization.

Current State:

• 10 T4 GPUs
• Cost: $3,500/month
• Average utilization: 20%
• P99 latency: 150ms

Optimization 1: Add Batching (1 week engineering effort)

• Reduces GPU count needed by 4x (due to efficiency gains)
• New count: 2-3 GPUs
• Cost: $700-1,050/month
• Latency: same or better (50-100ms with smart batching)
• Savings: $2,450-2,800/month
• ROI: 2-3 weeks (engineering cost amortized over 1 month)

Optimization 2: Model Quantization (2 weeks engineering effort)

• 8-bit quantization: 2x faster inference
• Allows 1 fewer GPU
• Cost: $350/month additional savings
• Accuracy loss: <0.5% (usually negligible)
• ROI: 1 month

Total Optimizations: 3 GPUs → 1 GPU, $3,500 → $350/month, saves $37K/year

This is why infrastructure optimization is a high-ROI engineering project.

Anti-Patterns

Anti-Pattern 1: Buying GPUs "for the Future"

You buy 100 GPUs because you might need them in a year. Most sit idle today. Cost: $100K/month in unused capacity.

Instead: Buy what you need now. Use cloud for future peaks. As demand grows and you understand actual needs, buy on-premises if ROI justifies it.

Anti-Pattern 2: Treating All Inference As Real-Time

You built real-time serving infrastructure for all workloads. Batch processing that could run nightly instead runs continuously, wasting GPU time.

Instead: Ask "does this need to be real-time?" If latency tolerance is >1 hour, batch it. Save 10x on infrastructure costs.

Anti-Pattern 3: Not Measuring GPU Utilization

You think GPUs are "always busy" but never measure. Actually, they're at 15% utilization. You're paying for 6 GPUs to do 1 GPU of work.

Instead: Measure utilization weekly. If below 50%, you're over-provisioned. Right-size infrastructure.

Anti-Pattern 4: Buying One Expensive GPU Instead of Multiple Cheaper Ones

You buy 1 A100 ($15K) instead of 3 T4s ($6K each). The A100 is "more powerful," so it's better, right?

Not necessarily. If your workload is inference-heavy, 3 T4s are better:

• Each T4 handles inference well (optimized for it)
• Better fault tolerance (one fails, you have 2 others)
• Better load balancing (distribute requests across 3)
• Same cost, better performance for inference workload

Instead: Choose GPU type based on workload, not raw power.

Anti-Pattern 5: Not Accounting for Infrastructure Overhead

You calculate "we need 4 GPUs." You buy 4 GPUs. You also need: network cards (1 GPU per card), switches, cooling, storage for models, monitoring infrastructure. Total cost is 2x what you budgeted.

Instead: Budget 50-100% infrastructure overhead for networking, cooling, monitoring, storage, redundancy.

Human Judgment Checkpoints

Checkpoint 1: Have You Measured Actual Demand?

Don't plan infrastructure based on hypotheticals. Run the workload for 3 months, measure actual demand, then plan infrastructure. Many organizations 10x over-provision based on assumptions.

Checkpoint 2: Can You Articulate Why You Need This GPU Type?

For each GPU you're recommending, can you say "we chose this because [specific reason]"? If you're not sure, you might be over-specifying.

Checkpoint 3: Have You Modeled Inference Batching?

If you're building inference infrastructure, have you calculated the impact of batching? It often reduces GPU count needed by 3-4x. If you haven't modeled it, you're probably over-provisioned.

Checkpoint 4: Is Cloud or On-Prem Right for Your Utilization?

High, consistent utilization → on-prem or reserved cloud instances. Variable or bursty utilization → cloud on-demand or spot instances. What's your actual utilization pattern?

Checkpoint 5: Have You Included Infrastructure Overhead in Budgeting?

GPUs are 50% of the cost. Cooling, power, networking, redundancy, monitoring add another 50%+. Did you budget for all of it?

Key Takeaways

Right-size GPUs to your workload, not your ambition. Training-optimized GPUs (A100, H100) are expensive and overkill for inference. Inference-optimized GPUs (T4, L4) are cheaper and better for serving. Choose based on what you actually do.

Inference infrastructure design matters as much as GPU choice. Batching, multi-model serving, intelligent request routing can reduce GPU count by 3-4x. This is a high-ROI area for engineering optimization.

Measure actual utilization before provisioning. Don't assume. Run the workload, measure, then plan infrastructure. Most organizations over-provision by 3-5x.

Batch processing instead of real-time saves 10x on infrastructure. If you can tolerate 1-hour latency, batch it nightly instead of serving continuously. This is often the highest-ROI cost optimization.

Choose between on-premises and cloud based on utilization patterns. High, predictable utilization → on-premises or reserved cloud is cheaper. Variable utilization → cloud on-demand is more cost-efficient.

Budget for infrastructure overhead beyond GPUs. Cooling, power, networking, monitoring, storage, redundancy add 50-100% to GPU costs. Include it in your budget.