Overview

A single AI model call is useful. But real power comes from chaining multiple steps together into a pipeline. Fetch data → process with AI → validate → store → notify.

An AI pipeline is a sequence of steps where each step produces output that becomes input for the next step. The pipeline runs automatically, either on schedule or triggered by events.

Pipelines are how you scale AI from toys to production systems. One model call is fragile. A well-designed pipeline is robust, validated, monitored, and can handle thousands of items per day.

This lecture is about designing and building production AI pipelines.

Pipeline Components and Architecture

Stage 1: Data Intake

Where do you get input data? Files uploaded by users, database records, API endpoints, message queues, streaming data.

Your pipeline needs to reliably ingest data. Handle backpressure (too much data arriving at once). Handle failures (if data source is down, pipeline should retry or fail gracefully).

Stage 2: Data Preparation

Raw data isn't ready for AI. Clean it. Validate it. Format it. Extract relevant fields.

Example: you get customer emails. Raw, unstructured. You extract customer name, account number, issue description. Format into structured data.

Stage 3: AI Processing

Feed prepared data to AI model. Get result. This is the core of the pipeline.

But you're doing this for thousands of items, so efficiency matters. Batch processing when possible. Use the right model for the job. Monitor costs.

Stage 4: Validation and Post-Processing

AI result needs validation. Does it make sense? Does it meet quality standards? Format the result for downstream consumption.

Example: AI classified an email as low priority. Validate that it has high confidence. Format result as JSON.

Stage 5: Storage and Retrieval

Store results somewhere. Database, data lake, cache, message queue. Make them accessible to downstream systems.

Stage 6: Notifications and Monitoring

Notify relevant systems that processing is complete. Update dashboards. Alert if something went wrong. Log everything for debugging.

Orchestration

Connect all stages together. A pipeline is a graph of stages where each stage triggers the next. Some stages run in parallel. Some depend on others.

Use a workflow orchestration tool: Airflow, Prefect, Temporal, or custom solution. The tool handles sequencing, retries, monitoring, and error handling.

Key principle: Each stage should be idempotent (running it twice with same input produces same output) and retriable (if it fails, running it again should succeed if the underlying issue is fixed).

Common Pipeline Patterns

Batch Processing Pipeline

Data accumulates. On a schedule (daily, hourly), process the batch. All items go through the pipeline together.

Advantage: efficient, can use batch pricing discounts, easy to debug and monitor.

Disadvantage: latency (if data arrives at 10am and batch runs at 10pm, 12 hour delay).

Use case: generate daily reports, process daily backlog, bulk data transformation.

Streaming Pipeline

Data is processed as it arrives. Low latency (seconds or milliseconds).

Advantage: real-time processing, responsive.

Disadvantage: higher complexity, harder to debug, more expensive.

Use case: real-time classification, live chat bots, fraud detection, alert systems.

Event-Triggered Pipeline

Pipeline runs when a specific event happens. Customer submits form, pipeline processes form. File uploaded, pipeline analyzes file.

Advantage: runs only when needed, no wasted compute.

Disadvantage: harder to batch, scaling can be tricky if events spike.

Use case: on-demand analysis, request-response systems, file processing.

Chain of Responsibility Pipeline

Multiple stages where each decides if to process or pass to next stage. Router model decides: is this request simple or complex? Route to fast model or powerful model. Not powerful enough? Try again with larger model.

Advantage: cost optimization, handles varying complexity.

Disadvantage: latency unpredictable, complex logic.

Feedback Loop Pipeline

Results from stage N feed back into stages 1-M for refinement. Human corrects mistakes. Corrections improve future results.

Advantage: gets better over time, handles ambiguous cases.

Disadvantage: complex logic, humans in the loop slows throughput.

Building Production Pipelines

Step 1: Design the Pipeline

What are your stages? What's the sequence? What stages run in parallel?

Draw it out. Data → Clean → AI Process → Validate → Store → Notify.

Define the contract between stages. What's the input to each stage? What's the output? What's the format?

Step 2: Implement and Test Each Stage

Build and test stages independently. Test data → Clean data → AI Process with test data. Each stage works in isolation before you chain them.

Step 3: Connect Stages

Use workflow orchestration tool. Define the graph. "After Clean, run AIProcess. After AIProcess, run Validate. Then Store, then Notify."

The tool handles sequencing and error handling.

Step 4: Handle Failures

Your pipeline will fail. Network glitches, API errors, unexpected data. Plan for it.

Retry logic: if stage fails, retry N times with exponential backoff.

Dead letter queue: if all retries fail, put the item in a queue for human review.

Monitoring: alert if error rate exceeds threshold. Someone needs to investigate.

Step 5: Optimize and Scale

Once working, optimize. Parallelize stages that don't depend on each other. Batch where possible. Use cheaper models where possible. Cache results that don't change.

Monitor costs. Monitor latency. Adjust based on performance.

Step 6: Monitor in Production

Track metrics: success rate, latency, cost. Track quality: does the output still match quality standards?

Set up alerts. If error rate spikes, get notified. If cost balloons, get notified. If latency degrades, get notified.

Production readiness checklist: Error handling, retries, monitoring, alerting, dead letter queue, cost tracking, quality metrics, documentation, runbook for common failures.

Cost Management in Pipelines

Pipelines process thousands or millions of items. Small inefficiencies multiply.

Batch processing: Process 100 items in one batch call instead of 100 individual calls. Cheaper and faster.

Caching: If same input appears twice, use cached result instead of calling model again. Saves 50-90% for repetitive data.

Model selection: Use cheaper models for simple tasks. Reserve expensive models for complex tasks. Route intelligently.

Monitoring and alerting: Set budget alerts. If daily costs exceed threshold, get notified. Something might be broken (infinite loop, sending data incorrectly).

Data validation: Validate data before sending to AI. Bad data wastes money. Clean, valid data is cheaper and gets better results.

What to Do Monday Morning

Identify a repetitive process: Something that processes data on a schedule or in response to events. Good candidate for pipelining.

Design the pipeline: What are the stages? What's the sequence? Draw it out.

Pick an orchestration tool: Airflow, Prefect, or custom. Start simple.

Build Stage 1: Data intake and preparation. Get data flowing into the pipeline.

Add AI processing: Stage 2. Process data with AI.

Add validation and storage: Stages 3-4. Make sure data is correct before storing.

Deploy and monitor: Run in production. Monitor success rate and costs. Iterate.

Pipeline Case Studies: Real Production Examples

Case Study 1: Content Moderation Pipeline (SaaS Platform, 50k daily items)

A community platform implemented an AI content moderation pipeline:

• Stage 1: Ingest (user-uploaded content, database records)
  - Stage 2: Classify (is this potentially harmful? Use fast model: Haiku)
  - Stage 3: Evaluate (if flagged, do more detailed evaluation with Opus)
  - Stage 4: Action (delete, warn user, flag for human review)
  - Stage 5: Feedback (collect human override decisions, improve future model)

Architecture: Batch process at night (50k items → 8 hours, cost-optimized). Streaming for real-time items (small percentage, high latency tolerance). Results: 98% accuracy on obvious cases (handled by fast model), 85% accuracy on ambiguous cases (handled by better model). Human reviewers reduced 90%. Cost: $0.002/item average. Volume: 50k/day. Monthly cost: $3k. Saved: ~$40k/month in manual moderation. ROI: positive within 2 weeks of operation.

Case Study 2: Customer Feedback Analysis Pipeline (SaaS, B2B, 5k daily feedback items)

A CRM company implemented feedback analysis pipeline. Customer sends feedback → AI extracts sentiment, intent, topic → stores in database → alerts if issue > priority threshold.

• Stage 1: Ingest (email, support tickets, surveys)
  - Stage 2: Extract (sentiment, intent, topic, customer segment)
  - Stage 3: Classify (bug report? feature request? compliment? complaint?)
  - Stage 4: Route (send bug reports to engineering, feature requests to product, complaints to success team)
  - Stage 5: Monitor (track quality, alert if pattern changes)

Key learning: Stage 2 was initially doing too much. The model was trying to extract 15 fields. Accuracy was 65%. They simplified: extract only 3 most important fields. Accuracy jumped to 92%. Then added specialized models for specific fields in separate stages. Cost: $0.01/item. Volume: 5k/day. Monthly cost: $1.5k. Value: product and engineering teams no longer have to manually read all feedback. They see prioritized summaries. Time saved: 40 hours/week. Cost savings: $50k+/month in salary time. ROI: massive.

Key lesson: simpler pipelines are more reliable. Don't try to do everything in one stage.

Pipeline Principle: Pipelines fail when you try to do too much in one stage or one model. Split complex tasks into simpler stages. Use cheaper models where possible. Expensive models only for hard decisions. This maximizes accuracy and minimizes cost.

When Pipelines Fail: Common Failure Modes

Failure Mode 1: Cascading failures.** Stage 2 fails. Stage 3 doesn't execute. Stage 4 doesn't execute. The whole pipeline stalls. You don't realize until 12 hours later. Impact: 12 hours of items in queue. Lesson: monitor each stage independently. Alert if any stage has >1% error rate. Don't wait for end-to-end failure.

Failure Mode 2: Quality degrades silently.** A schema changes upstream. Your Stage 2 now receives bad data. It produces bad output. The pipeline still runs. But quality is trash. Nobody notices for days. Lesson: validate data at intake stage. Reject bad data. Alert if rejection rate spikes.

Failure Mode 3: You don't have retry logic.** Transient failure (API timeout). Pipeline item is discarded. Data is lost. You don't realize. Lesson: idempotency + retries + dead letter queue. If stage fails, retry. If all retries fail, queue for human review. Never silently discard.

Failure Mode 4: Costs balloon unexpectedly.** You didn't predict traffic spike. Suddenly 10x items flowing through. Monthly cost goes from $5k to $50k. Lesson: budget alerts. If daily cost exceeds threshold, alert. Something might be broken (infinite loop, wrong data, misconfiguration).

Failure Mode 5: You don't monitor quality.** Accuracy was 90%. Now it's 75%. You didn't notice. Lesson: sample output periodically. Compare to baseline. Alert if quality drops >5%.

FAQ: Pipeline Questions

Q: How do we handle failures in the middle of a pipeline?

A: Idempotency + retries + dead letter queue. Structure each stage so running it twice with same input produces same output. If stage fails, retry (with exponential backoff). If retries exhaust, put in dead letter queue for human review. This handles transient failures and prevents data loss.

Q: How many items can a pipeline realistically process?

A: Depends on your infrastructure. A single serverless function: 1000-5000/minute. A dedicated server: 5000-50000/minute. A distributed cluster: millions/minute. Start with serverless, upgrade if you hit limits.

Q: How do we know if quality is degrading?

A: Sample output periodically (1% of items). Have human review samples. Track accuracy. Compare to baseline. Alert if drops >5%. For critical pipelines, review daily. For non-critical, weekly is fine.

Q: Can we update pipeline logic without reprocessing all historical data?

A: Usually you don't need to. But if you change the model or logic, good practice is to reprocess last 2-4 weeks of data (ensures consistency) and spot-check against old output. Reprocess all historical data only if the change is significant.

Q: This sounds operationally complex. What's the minimum viable pipeline?

A: Stage 1 (ingest), Stage 2 (process), Stage 3 (store). Data flows: input → process → output. Add monitoring for success rate. Add alerts for errors. Add one dead letter queue for failures. That's 80/20. Everything else is optimization.

Q: How do we handle rate limits when using external AI APIs?

A: Implement backoff and queuing. If you hit rate limits, queue items temporarily. Retry after delay. Use distributed rate limiting if you have multiple pipelines. Most AI providers let you request higher rate limits if you have high volume, contact them first rather than hitting limits in production.

Advanced Pipeline Patterns

Adaptive Routing: Route items to different model based on complexity. Simple items → fast/cheap model. Complex items → slow/expensive model. Example: short text comments → Haiku. Long detailed feedback → Claude. Saves 70% of costs while maintaining quality.

Feedback Loops: Use output from production pipeline to improve the model. Collect human feedback on pipeline output. Periodically retrain on corrected examples. Pipeline gets better over time.

Parallel Processing: Run independent stages in parallel. Stage 2 sentiment analysis and Stage 3 topic classification don't depend on each other, run both after Stage 1. Reduces latency significantly.

Incremental Pipelines: Only reprocess items that changed or need updating. Don't reprocess everything every run. Track what's been processed. Update selectively. Saves time and cost.

Key Insight

Pipelines scale AI from batch experiments to production systems processing thousands of items. Sequence stages logically. Implement error handling and retries. Monitor costs and quality carefully. Start simple, add complexity as needed. This is how you actually run AI at scale in production.

On This Page

Introduction
Pipeline Basics
Common Patterns
Building Pipelines
Cost Management
Case Studies
Failure Modes
Monday Morning Action
FAQ

Chapter Details

Part ofChapter 8