The Layers of the AI Stack

When you use AI, you're usually not using just a language model. You're using a stack of technologies. Let me walk you through the layers, from the model itself up to the application layer.

Layer 1: The Model

At the bottom is the model itself. Claude, GPT-4, Llama, Gemini. These are language models. They're trained, frozen neural networks that take text in and predict text out.

You access models through APIs (Application Programming Interfaces). You send a request with text and get back a response. The API is the contract, "here's what I'll give you, here's what I'll get back."

Model choice matters. Different models have different characteristics:

• Size: Smaller models are cheaper, faster, but less capable. Larger models are expensive and slow but can handle more complex tasks.
  - Training data: Models trained on different data behave differently. A model trained on code will be better at coding than one trained on general web text.
  - Fine-tuning: Models fine-tuned for specific domains (medical, legal) are better at those domains.
  - Cost vs. capability: GPT-4 is more capable than GPT-3.5, but also 15x more expensive. Sometimes a cheaper model is good enough.

As a practitioner, you're usually choosing between commercial APIs (use their models, pay per token) and open source (run your own models, pay for compute).

The Model Choice Is Not The Biggest Decision: Which model you use matters, but usually less than how you architect around it. A good architecture using GPT-3.5 usually beats a bad architecture using GPT-4. Pick a reasonable model and focus on everything else.

Layer 2: APIs and SDKs

You don't call the model directly. You use an API. OpenAI has their API. Anthropic has theirs. Fireworks and Together have theirs. The API is the interface layer.

APIs handle:

• Authentication: proving you have permission
  - Rate limiting: controlling how many requests you make
  - Billing: tracking usage and charging you
  - Routing: directing your request to the right GPU cluster
  - Monitoring: logging requests for debugging

Most APIs have SDKs (Software Development Kits) in common languages (Python, JavaScript, Go). These wrap the HTTP calls and give you a nice interface. For example, the Anthropic SDK lets you write:

client.messages.create(model="claude-3-opus", messages=[...])

Instead of manually building HTTP requests. SDKs handle serialization, error handling, retries. They're worth using.

Layer 3: Prompting and Context

Above the API is the prompting layer. This is where you decide what to ask the model and how to ask it.

Prompting is both simple and complex:

Simple: "What's 2+2?" gets you an answer.

Complex: How do you structure a prompt to get consistent, high-quality output? How do you give the model enough context to be accurate without exceeding the context window? How do you handle edge cases?

There are prompting patterns that work:

• Few-shot learning: showing examples before asking for the real task
  - Chain-of-thought: asking the model to reason step-by-step
  - System prompts: giving the model instructions about how to behave
  - Structured output: asking for JSON or specific formats

This is where the 1.6x-to-higher-multipliers gap often lives. A thoughtful prompt gets you 10x better results than a naive prompt. This is Level 2-3 on the capability spectrum. You're problem-solving with the model, not just using it.

Layer 4: Retrieval-Augmented Generation

One of the most important patterns: RAG (Retrieval-Augmented Generation). The problem it solves: language models have knowledge cutoffs. They can't access real-time data. They don't know about your proprietary documents.

Solution: fetch relevant information from a database, include it in the prompt, ask the model to answer based on that information. Example flow:

• User asks: "What are our Q3 financials?"
  - System searches documents for Q3 financials
  - System includes the relevant document excerpts in the prompt: "Here are our Q3 financials: [excerpts]. Now answer the user's question."
  - Model sees the context and generates an answer based on your actual data

RAG requires:

• A vector database (Pinecone, Weaviate, Milvus) that stores embeddings of your documents
  - An embedding model that converts documents and queries to vectors
  - A retrieval process that finds similar documents quickly

This is how you make language models work with your proprietary data without needing to fine-tune. And it's how you keep them current without retraining.

RAG is Level 3-4 on the capability spectrum. It's a structural change to how you use AI. Instead of "ask the model a question," it's "find relevant context, ask the model using that context." The improvement in accuracy and reliability is dramatic.

Layer 5: Function Calling and Tools

Another important pattern: giving the model access to tools. The model can output function calls (not just text), your system executes them, and the results get fed back to the model.

Example: You have a calculator function. The user asks "What's 234 * 567?" The model instead of estimating, outputs "call calculate(234, 567)." Your system calls the function, gets 132,678. The model incorporates that into its response: "The answer is 132,678."

Tools can be:

• Deterministic functions (calculator, string processors)
  - APIs (fetch from your database, call external services)
  - Code execution (Python, SQL queries)
  - Custom tools (domain-specific operations)

This is powerful because it lets the model delegate work to systems that are reliable. The model handles reasoning and language. The tools handle computation and data access.

This is also Level 4-5 territory. You're building AI systems, not just using AI as a tool.

RAG + Tools = Real Systems: The combination of retrieval-augmented generation and tool use is where you move from "fun chatbot" to "reliable system." The model provides reasoning and language. RAG provides context. Tools provide reliable computation. Together, they're powerful.

Layer 6: Monitoring and Evaluation

When you deploy AI, you need to know if it's working. This requires monitoring and evaluation.

Monitoring tracks:

• Cost: how much is this AI system spending?
  - Latency: how fast are responses?
  - Error rates: what percentage of requests fail?
  - Token usage: how much are we using relative to what we expected?
  - Model quality: is the output getting worse over time?

Evaluation is trickier. You need to know if the AI is accurate. Some evals are automated (does the output match the expected format?). Some require human judgment (is this a good code review?).

This is why many teams build evaluation frameworks, datasets of examples with expected outputs, automated tests to check accuracy, human review of edge cases.

Monitoring and evaluation separate production AI from experimental AI. Experimental, you might not worry. Production, you need visibility and confidence.

Layer 7: User Interface

At the top is the user interface. This is how people actually interact with your AI system. It might be a chat interface, a web form, an API endpoint, or something custom.

UI is often where you solve the real problem. A language model is powerful, but useless if you can't make it do what users need. The UI is how you shape the model's output into something valuable.

Putting It Together

A typical AI application stack looks like this:

User Interface → Application Logic (prompting, RAG, tools) → API (to the model) → Model → API Response → Post-processing → User Interface

But also in parallel: Vector Database ← Embedding Pipeline ← Document Ingestion

And underneath everything: Monitoring, Logging, Evaluation

This sounds complex, and it can be. But most of the complexity is optional. You don't need all of these layers. Simple applications might just be: User → Prompt → API → Model → Response. That's fine.

But as you move from 1.6x to higher multipliers, you gradually need more sophisticated layers. RAG helps you ground outputs in reality. Tools help you delegate computation. Monitoring helps you trust the system. Evaluation helps you iterate.

When This Goes Wrong: Stack Failures

The Cost Explosion

You deploy an AI feature without monitoring (Layer 6). It works great. One week later, you notice your API bills are 10x higher than expected. A bug in your prompting caused infinite loops. Or a user found an edge case that causes the model to generate 100K tokens per request. You're now paying $10K/day instead of $1K/day. Solution: monitor token usage, cost, and latency from day one. Set alerts at 50% of expected cost. When something spikes, investigate immediately.

RAG Gone Wrong**

You implement retrieval-augmented generation (Layer 4). Documents get embedded and stored. Your search works... sometimes. Random queries return wrong documents. The embedding model is finding syntactic matches but not semantic relevance. You realize you're using a generic embedding model trained on web text, but your documents are technical/specialized. Solution: use domain-specific embedding models (or fine-tune). Test retrieval quality with a small eval dataset before deploying. Verify that top-5 results are actually relevant 80%+ of the time.

Silent Quality Degradation**

You deploy an AI system. It works well. Months later, someone notices the quality has drifted, same prompts, worse outputs. What happened? The model behavior changed (vendor pushed an update). Or real-world data has shifted (users asking different questions). Or feedback loops amplified bad behavior. You never noticed because you weren't monitoring Layer 6. Solution: evaluate model output regularly. Sample outputs monthly. Compare to baseline. If quality drops 10%+, investigate immediately.

The Missing Tool**

You build a system with function calling (Layer 5) but forget to implement rate limiting on tool execution. A user exploits this to call a function 10K times in 5 seconds. The function calls your database, your database gets overwhelmed, service goes down. Solution: every tool needs rate limiting, cost controls, and timeout protections. Don't just allow the model to call anything it wants.

Case Study: Logistics Company Stack Evolution

A logistics company built AI features to help optimize routes. Early version (Layer 3 only): user uploads route data, gets optimization suggestions. Good. But customers complained: "The AI suggests routes that violate our constraints (no late-night deliveries, prefer local drivers)." The AI didn't know about their business rules.

First Fix (Added Layer 4 - RAG): They created a vector database of customer constraints. When optimizing, the system retrieves the customer's constraints, includes them in the prompt, and generates routes respecting them. Quality improved 40%.

Second Problem: "The AI suggests routes but we need to verify feasibility." Verification took 20 minutes per route (manual database lookups). The system was slower than the old system.

Second Fix (Added Layer 5 - Tools): The AI got access to tools: get_customer_constraints(), verify_driver_availability(), check_vehicle_capacity(), estimate_delivery_time(). When suggesting a route, the AI uses these tools to verify feasibility in real-time. The system is now 10x faster because the AI delegates deterministic checks to tools instead of guessing.

Third Problem: They deployed to production. One week later, cost was 3x higher than expected. A specific customer kept asking for "optimize with additional constraints" that caused 50K token prompts.

Third Fix (Added Layer 6 - Monitoring): They added cost monitoring. Alert if a single request costs >$1. They also added caching: if the same constraints are used, return cached route instead of calling the API. Cost dropped back to normal.

Result: By gradually building out the stack (3 → 4 → 5 → 6), they built a system that was 40% more accurate, 5x faster, and cost-efficient. Starting with just Layer 3 wouldn't have worked. The full stack was necessary.

Building vs. Using

One key distinction: are you using AI (like you use a library) or building AI systems (architecting a solution)?

Using AI (Layers 1-3): Call Claude through an API, get a response, use it. Simple. Takes hours to implement. Limited control and customization.

Building with AI (Layers 1-7): Design a system where AI is one component. Manage data pipelines (embeddings, retrieval). Handle edge cases. Monitor quality. Iterate based on results. Takes weeks to months but gives you deep value.

Most organizations that move from 1.6x to higher multipliers shift from "using AI" to "building with AI." It's more complexity, but it's where the real value is. And the complexity is worth it. You go from "fun chatbot" to "reliable system that creates business value."

What to Do Monday Morning

• Map your current stack. Document all the layers you're currently using (Layers 1-7). What's missing?
  - Identify your biggest blocker. Is it accuracy (Layer 4 RAG might help)? Speed (Layer 5 tools might help)? Cost (Layer 6 monitoring might help)? Visibility (Layer 6 evaluation)?
  - Pick one layer to improve. Don't try to implement all 7 at once. Pick the layer that will have the biggest impact and invest there.
  - For layer 3 (prompting): test 3-5 different prompt approaches. Measure which one gives the best quality. Document the best one.
  - For layer 4 (RAG): if you have proprietary documents, test embedding them and retrieving them. Does retrieval work? If not, what's wrong?
  - For layer 6 (monitoring): set up basic monitoring for cost, latency, error rate. If you notice anomalies, you want to know immediately.

FAQ

Q: Do I need all 7 layers?

A: No. A simple chatbot needs 3 layers. A production system needs 5-7. Start simple, add layers as you need them. The most common progression: 3 → 4 (add retrieval) → 5 (add tools) → 6 (add monitoring) → 7 (polish UI).

Q: Which model should I use?

A: For most use cases, Claude 3.5 Sonnet or GPT-4o are good starting points. If cost is critical, Claude 3.5 Haiku. If you need speed, Haiku. For very complex reasoning, Opus/GPT-4. Don't overthink this. Pick one, build your system, measure, iterate.

Q: Is open-source (Llama, Mistral) or commercial APIs (Claude, GPT) better?

A: Commercial APIs: easier, better models, higher cost. Open-source: lower cost, more control, harder to deploy. For most teams, start with commercial APIs. Once you have product-market fit, consider open-source if cost is a problem.

Q: How much does this stack cost?

A: Layer 1-3 (using Claude API): $0.001-0.01 per request, so $0.1-10/user/month depending on usage. Layer 4 (RAG): add $50-500/month for vector database. Layer 5 (tools): cost depends on what the tools are. Layer 6 (monitoring): $100-1000/month depending on volume. Total: $100-2000/month for a modest production system.

What Comes Next

The next lesson is about confidence and hallucinations, the real constraints of language models. Understanding the stack is important, but understanding what's not possible is equally important.

Key Insight

Language models are just one layer in the stack. The full stack includes APIs, prompting, retrieval, tools, monitoring, and evaluation. Most organizations at 1.6x are using layers 1-3. Moving to 10x usually means effectively using layers 4-7.

On This Page

The Layers of the AI Stack
Layer 1: The Model
Layer 2: APIs and SDKs
Layer 3: Prompting and Context
Layer 4: Retrieval-Augmented Generation
Layer 5: Function Calling and Tools
Layer 6: Monitoring and Evaluation
Layer 7: User Interface
Putting It Together
Failure Modes
Case Study
Building vs. Using
Monday Morning Action
FAQ
What Comes Next

Chapter Details