Skill.re
How Large Language Models Work
Hook
Your CTO just told you the company is "moving to LLM-powered systems" and wants to know what infrastructure investments are needed. Your budget analyst wants to know why one AI solution costs 10x more to run than another. Your security team wants to understand what data moves through these systems and where it gets stored. And you're realizing that "it's an AI system" doesn't actually tell you anything about what's happening under the hood, or what resources you need to support it.
Here's what you need to know: Large Language Models are not magic. They're massive statistical engines that run on your infrastructure and have very specific requirements. Understanding how they actually work, tokens, parameters, inference, context windows, is the difference between a smart infrastructure decision and a budget disaster.
Let's look inside these systems and understand what's actually happening when you deploy one.
Purpose
This lesson explains how Large Language Models (LLMs) work from an infrastructure perspective. You'll understand the computational requirements, the constraints that affect performance and reliability, the actual data flows, and the limitations that matter when you're responsible for deployment and operations.
Why This Matters for IT Professionals
When you understand how LLMs actually work, several things become clear:
- Resource planning: You can estimate compute, memory, and storage requirements instead of just trusting vendor claims. You know why one model requires 48GB of GPU memory and another requires 4GB.
- Performance troubleshooting: When response latency becomes an issue, you understand whether the problem is your infrastructure, your model choice, or your usage pattern.
- Cost optimization: You can make intelligent decisions about which models to use for which tasks, how to batch requests, whether to use smaller specialized models instead of larger general ones.
- Security and data governance: You understand where user data goes, what happens to context windows, how long information persists, and what privacy implications exist.
- Vendor conversations: When a vendor claims their LLM is more efficient or faster, you can ask intelligent questions about inference speed, model size, quantization, and actual performance metrics.
- Incident response: When an LLM system starts misbehaving or generating unexpected outputs, you know how to investigate and understand whether it's a model issue, a data issue, or a system configuration issue.
Core Concepts
What Tokens Actually Are
When you send text to an LLM, the system doesn't process words. It processes tokens, small pieces of text, typically 3-4 characters on average but varying wildly.
Here's the concrete reality: if you write "infrastructure," the system might break it into tokens like ["infra", "structure"] or ["in", "fra", "structure"] depending on how the tokenizer works. A word like "cat" is one token. An emoji might be multiple tokens. Punctuation is its own token. Whitespace matters.
Key insight: This matters operationally because tokens directly map to computational cost and response time. If your system processes tokens at a rate of 100 tokens per second and a user query is 500 tokens, that's a minimum of 5 seconds of processing time just for input. An LLM-generated response of 500 tokens adds another 5 seconds. Add network latency, and suddenly your "AI-powered help desk" has multi-second response times.
More critically, tokens determine billing for most LLM services. If you don't understand tokenization, you can't predict costs. A task that seems simple (summarizing an incident report) might tokenize into far more tokens than you expect because incident reports often contain code, log snippets, and technical jargon that tokenize inefficiently.
Parameters and Model Size
An LLM's parameters are numbers in mathematical equations. A model with 7 billion parameters has 7 billion numbers that got optimized during training. A model with 70 billion parameters has 70 billion numbers. These numbers are what give the model its knowledge and behavior.
Here's the infrastructure implication: each parameter typically requires 2-4 bytes of memory to store and run. A 7-billion-parameter model needs roughly 14-28GB of memory just to exist. A 70-billion-parameter model needs 140-280GB. If your infrastructure has 64GB of GPU memory, you can't run a 70B model. You need a smaller one, or you need a different approach.
Key insight: Bigger models are generally smarter and more capable, but they're also exponentially more expensive to run. A 70B model isn't 10x better than a 7B model; it's 10x larger and 10x slower to run, but maybe 20-30% more accurate for complex tasks. This creates a critical tradeoff in operational design.
Parameter size also determines batch processing. If you can fit only one 70B model instance in your available memory, you can't parallelize requests. If you can fit ten 7B instances, you can handle ten requests simultaneously even if each individual request is slightly less intelligent.
Training vs. Inference
Training is what happened to the model before you got it. During training, engineers fed it gigabytes of text data (books, articles, code repositories, web pages, whatever the model was trained on), and the model learned patterns from that data. Training is expensive, slow, and requires massive compute resources. You don't do this. The vendor did.
Inference is what happens when you use the model. You send it some text (the "prompt"), and the model generates output. This is what runs in your infrastructure.
Key insight: Inference is still expensive, but it's orders of magnitude cheaper than training. However, it's not free. Every request consumes compute, memory, and network bandwidth. If you have thousands of concurrent users, that adds up quickly.
This distinction matters because you might hear someone say "we're fine-tuning the model," and you need to understand what that actually means. Fine-tuning is a lighter version of training. You take an already-trained model and do a smaller round of training on domain-specific data. It's cheaper than training from scratch but still expensive compared to inference.
Context Window: The Limits of What the Model Can See
An LLM can't actually read all of your documentation or all of your previous tickets. It has a context window, the maximum amount of text it can process in a single request. Current models typically have context windows between 4,000 and 200,000 tokens.
This matters operationally because it's the hard constraint on what the model can consider. If your help desk ticket includes context from three previous related tickets (the complete history), that context eats into your context window. If you want the model to consider your company's security policies before answering a question, that context eats into your window.
Key insight: Context window creates a real tradeoff between breadth of knowledge (how much history or reference material you can include) and task performance (how complex a task you can solve with the remaining window for actual problem-solving).
A practical example: you want to use an LLM to evaluate server configurations against your infrastructure standards. If your standards document is 50,000 tokens and your context window is 100,000 tokens, that leaves 50,000 tokens for the actual server configuration and the model's response. If your server configuration is very large (lots of services, many config files), it might not fit. You either need a larger context window model (more expensive, slower) or a different approach (RAG, which we'll cover later).
Temperature: Controlling Randomness
Temperature is a parameter that controls how deterministic the model's output is. Low temperature (like 0.1) makes the model more predictable and consistent. It tends to output the same thing for the same input. High temperature (like 1.0+) makes it more creative and random. It explores more varied responses.
Key insight: For IT operations work, temperature matters for reliability and consistency. If you're using an LLM to classify help desk tickets, you want low temperature (consistent classifications). If you're using it to brainstorm infrastructure designs, higher temperature might be useful to explore varied approaches. But higher temperature also means less predictability, which can be problematic for automated decision-making.
Temperature directly affects whether you need human review of the output. At very low temperature, you might trust automated classification more (though you still shouldn't fully). At high temperature, every output is different and less reliable for automation.
How Inference Actually Works (Simplified)
When you send a prompt to an LLM, here's what happens:
- Your text gets tokenized (broken into tokens)
- The model processes all those tokens and outputs a probability distribution across all possible next tokens
- The system samples from that distribution (or picks the most likely one) and outputs a token
- That token becomes part of the new context, and the model repeats
This happens token by token. If you ask a question and the model generates a 500-token response, it runs this loop 500 times. Each iteration is a neural network computation across billions of parameters.
This is why inference takes time and why larger models are slower. They have more parameters, so each token takes longer to compute. It's also why batch processing is more efficient than single requests. You can process multiple prompts in parallel, keeping the GPU busy.
Key insight: This token-by-token generation means LLMs are inherently sequential. You can't parallelize a single response. If you need the full response in 2 seconds, you need infrastructure fast enough to compute tokens at that rate. Latency is a hard constraint based on your hardware.
Hallucination: Why Models Generate False Information
Because LLMs are predicting the statistically most likely next token based on patterns in training data, they don't have a mechanism for "I don't know." If they encounter a situation their training data didn't cover, they don't output an error. They just predict the most likely token anyway, generating something that sounds reasonable but might be completely false.
This is hallucination: the model confidently generates information that sounds plausible but is incorrect.
Key insight: Hallucination isn't a bug you can fix by trying harder or using a better prompt. It's inherent to how these systems work. No LLM can be trusted to generate factual information without verification. This matters critically for IT operations, if an LLM suggests a security vulnerability mitigation or a configuration change, you must verify it's actually correct before deploying it.
Practical Use Cases
Case 1: Infrastructure Documentation Q&A
The Goal: Your infrastructure team has 5MB of documentation about your systems, standards, and configurations. You want an LLM to answer questions about this documentation so engineers don't have to dig through it manually.
The Problem: Your documentation is 5 million tokens. Most models have 4K-8K context windows. You can't fit both the documentation and the question in a single request.
The Solution: RAG (Retrieval Augmented Generation). You store the documentation in a vector database (we'll cover this later). When someone asks a question, you search for the most relevant documentation sections, pass those into the LLM's context along with the question, and get an answer. This way, the model only sees the relevant parts of documentation (maybe 2,000 tokens) plus the question, leaving room for a useful response.
Infrastructure Implications: You need vector database infrastructure (managed service or self-hosted), persistent storage for embeddings, and a search pipeline. The LLM itself is simpler. You can use a smaller, faster model because you're not asking it to know all of your documentation, just to answer questions about the parts you feed it.
Case 2: Code Review and Vulnerability Scanning
The Goal: Use an LLM to review infrastructure-as-code (Terraform, Kubernetes configs) for potential security issues and misconfigurations.
The Problem: Some infrastructure files are large. A single Terraform module might be 10,000+ tokens. Your context window might be 8K. But even if it fits, the LLM's training data includes a lot of insecure examples, so it might miss issues or normalize the insecure patterns it saw during training.
The Reality: LLMs are useful for this, but not as a replacement for specialized security scanners. They catch things that look wrong to human patterns (unused variables, overly permissive settings) but miss domain-specific issues (a configuration that looks fine but doesn't work with your specific cloud provider version). They also require human review because they generate false positives.
Infrastructure Implications: You run the LLM asynchronously, not inline with your deployment process. It generates suggestions that your security team reviews rather than blocking deployments. For critical infrastructure, you treat the LLM as a secondary check, not the primary one.
Case 3: Incident Response and Log Analysis
The Goal: During an incident, feed logs from multiple systems into an LLM to get a rapid initial assessment of what went wrong.
The Problem: Production logs can be massive. A single hour of logs from a busy system might be 100,000+ tokens. You need to summarize and filter the logs before passing them to the LLM, or you exceed context window.
The Solution: Extract the most relevant logs (errors, warnings, timeouts) and feed those to the LLM. The model can quickly identify patterns (e.g., "the database query latency spiked before the service timeouts started, suggesting a database issue"). But this is an initial hypothesis, not a diagnosis. Your team must verify it.
Infrastructure Implications: You need log aggregation and filtering infrastructure. The LLM accelerates diagnosis, but you still need traditional monitoring and domain expertise. The LLM is a tool for the human incident commander, not a replacement for incident response procedures.
Examples
Example 1: The Context Window Surprise
Scenario: You deploy an LLM-based help desk assistant. It's supposed to look at a customer question and generate a response by considering the customer's account history.
What Happens: The system works for new customers. But when a customer has six months of ticket history (the context for understanding their specific issues), the context window fills up with history. The model's response becomes shorter and less useful because less of the context window is available for actually solving the current problem.
The Issue: You designed the system without accounting for the relationship between context window size and your use case. Small context windows force you to choose: include historical context, or have space for a detailed response. You can't optimize both.
What You Must Do: Either increase context window (costs more, slower inference), summarize historical context instead of including it verbatim (requires preprocessing), or split the task (one model reads history and creates a summary, another model uses that summary). Understand the actual tradeoff before deploying.
Infrastructure Lesson: Context window isn't just a capability. It's a constraint that drives architecture decisions.
Example 2: The Token Cost Explosion
Scenario: You implement a system where an LLM reviews every help desk ticket for sensitive information (credit cards, SSNs, passwords) before the ticket gets stored. Seems reasonable, catch sensitive data before it's in your system.
What Happens: You track costs monthly and discover the LLM review is 3x more expensive than you budgeted. Why? Because you're processing every ticket, including the metadata and ticket templates. A ticket that looks like 500 words is actually 3,000 tokens once it's fully tokenized with headers, field names, and metadata.
The Issue: You didn't account for how your specific text tokenizes. Different systems tokenize differently. Technical text (with code, URLs, log snippets) tokenizes inefficiently. Your estimate was off by 6x.
What You Must Do: Tokenize a real sample of your actual data before budgeting. Don't estimate based on word count. Count actual tokens. If you're building a system that processes lots of text, build in an initial phase where you sample your data, tokenize it, and refine your cost estimates.
Infrastructure Lesson: Tokenization is where estimated costs diverge from real costs. Validate your assumptions with actual data.
Example 3: The Latency Commitment
Scenario: You promise your help desk that the new AI system will provide responses in under 2 seconds. Your chosen LLM has 70 billion parameters and can generate 20 tokens per second on your infrastructure.
What Happens: Users ask questions and wait 5+ seconds for responses. Why? Because a typical question is 100 tokens, and a useful response is 200+ tokens. That's 300 tokens minimum. At 20 tokens per second, that's 15 seconds minimum processing time. Add network latency, context retrieval, and orchestration, and you're easily at 20+ seconds.
The Issue: You made a promise based on ideal conditions, not your actual infrastructure and use case. Your hardware can handle the bandwidth, but not the latency.
What You Must Do: Before committing to latency SLOs, measure actual end-to-end latency with real prompts. Account for: tokenization, inference, context retrieval (if using RAG), post-processing, and network round trips. Larger models are slower. Longer responses are slower. Real-world latency is rarely what vendors claim.
Infrastructure Lesson: Latency scales with parameters and response length. You can't hit aggressive latency targets with very large models unless you have exceptional hardware or use smaller specialized models.
Anti-Patterns
Anti-Pattern 1: Running Full LLMs When Smaller Models Would Work
Risk: You deploy a 70-billion-parameter model for a task that a 7-billion-parameter model handles perfectly.
Why It Happens: Bigger models are generally better, and "better" feels like a safe choice. You want to hedge against unknowns.
What Goes Wrong: Massive infrastructure costs (10x the compute), slower response times (10x latency for the same task), and more difficult deployment (bigger models require more resources, fail more gracefully if you run out, and are harder to scale horizontally).
How to Avoid: Start with the smallest model that handles your use case. For most IT operations tasks (ticket classification, log analysis, alert triage), 7B or 13B models are sufficient. Reserve larger models for tasks that genuinely need them (very long reasoning, novel problems, complex multi-step tasks). Run A/B tests comparing models on your actual use cases and measure the tradeoff between accuracy and cost/latency. Usually, 80% of performance at 10% of the cost is the right answer.
Anti-Pattern 2: Assuming Inference is Cheap
Risk: You deploy LLMs widely, assuming that once you've bought the infrastructure, running queries is essentially free.
Why It Happens: Training costs are extremely high (millions of dollars). Inference feels free by comparison. And API-based LLM services often quote costs-per-query that seem small.
What Goes Wrong: Your inference usage explodes (tokens per month scale with users and query frequency). Your infrastructure costs become dominated by LLM inference. Or your API bills become the largest IT expense in the department.
How to Avoid: Treat LLM queries like any other scarce resource. Budget them, monitor them, and optimize them. A query that costs 0.001 cents seems free until you're running a million of them per month. Implement request batching where possible (it's more efficient), set limits on context window size, and regularly audit whether tasks actually need LLM inference or could be solved with simpler methods.
Anti-Pattern 3: Not Validating the Model's Training Data
Risk: You deploy an LLM for help desk ticket routing or severity classification without understanding what data the model was trained on.
Why It Happens: You trust that major models are trained on high-quality data and general knowledge. For a general-purpose task, this seems reasonable.
What Goes Wrong: The model's training data might not represent your environment well. A model trained mostly on public internet data (Reddit posts, GitHub issues, Stack Overflow) learns different patterns than a model trained on enterprise support tickets. It might misclassify your specific issues because they don't match the patterns in its training data.
How to Avoid: Before deploying, understand where the model comes from and what it was trained on. Test it on your actual data and measure its accuracy. If it's significantly worse on your domain (security tickets, database issues, infrastructure problems) than on general tasks, fine-tune it on a sample of your data, or use a specialized model if available. For critical IT operations tasks, don't assume a general-purpose model will work without validation.
Anti-Pattern 4: Building Systems That Depend on Perfect Inference
Risk: You design a system where an LLM makes autonomous decisions without human review, perhaps automatically closing tickets, approving changes, or escalating alerts.
Why It Happens: Removing human review seems efficient and aligns with the promise of "intelligent automation."
What Goes Wrong: When the LLM makes mistakes (misclassifies severity, misunderstands a ticket, hallucinates a solution), those mistakes have consequences. Tickets get closed prematurely. Changes get approved that break production. False alerts go unchecked. The system becomes unreliable.
How to Avoid: Design for human collaboration, not autonomous decision-making. The LLM can suggest, filter, summarize, and flag. A human decides. Or the LLM makes decisions only on clearly low-risk cases (auto-closing a resolved ticket for the fifth time) while humans review edge cases. This prevents the system from making critical mistakes while still reducing human workload.
Anti-Pattern 5: Ignoring Quantization and Model Optimization
Risk: You deploy the exact model files from the model publisher without considering quantization or optimization techniques.
Why It Happens: You want to ensure maximum quality and assume the published models are already optimized.
What Goes Wrong: You're running at 10x the infrastructure cost you need to. Quantization (reducing numerical precision from 32-bit to 8-bit or 4-bit) reduces model size by 4-8x with only 1-5% loss in accuracy. For most IT operations tasks, this is an excellent tradeoff. But it requires conversion and testing, so it's easy to skip.
How to Avoid: After validating a model works for your use case, explore quantized versions. Measure the accuracy impact. If it's minimal (which it usually is for IT operations tasks), deploy the quantized version. You get 4-8x faster inference and 4-8x less memory usage. This is often the highest ROI infrastructure optimization you can make.
Human Judgment Checkpoints
Before deploying an LLM in your infrastructure, ask these questions:
What's my actual context window requirement? Measure your typical prompt length and response length. Add them together. If it exceeds the available context window, you need a different approach (RAG, smaller prompts, or a larger context window model).
What tokens per second do I need? Multiply response length by number of concurrent users. If you need to handle 100 concurrent users expecting 200-token responses, you need 20,000 tokens per second throughput. Can your infrastructure deliver that? What latency will that create?
What's the actual inference cost? Calculate tokens per month. Multiply by the cost-per-token from your vendor or infrastructure. Is this in your budget? Will it grow with usage?
How accurate does this model need to be for this task? Test on your actual data. If accuracy is above 95% for your use case, the smallest model that meets that threshold is the right choice. If accuracy needs to be 99%+, you might need human review, not a larger model.
Can I quantize this model without unacceptable accuracy loss? Try a quantized version on your actual use cases. Most IT operations tasks tolerate 1-5% accuracy loss for 4-8x efficiency gains.
What happens when the model fails? The most dangerous assumption is that LLMs won't make mistakes. They will. Design your system to fail gracefully. For critical decisions, build in human review.
Key Takeaways
Understand that tokens determine infrastructure requirements. Your compute needs scale with the number of tokens you process. Token counts matter more than word counts for estimating cost and latency.
Model size and inference speed are a fundamental tradeoff. Bigger models are generally smarter but exponentially slower and more resource-intensive. Start with the smallest model that handles your use case, then optimize from there.
Context window is a hard constraint. You can't process more text than your context window allows. Design your system around this limitation. Use RAG, summarization, or smaller context windows rather than trying to force large amounts of text through a small window.
Inference isn't free, even if it feels cheap. Monitor token consumption and infrastructure costs. Implement batching and optimization. The efficiency gains from optimization usually exceed the cost of the optimization effort.
Hallucination is inherent, not a bug. No LLM can be trusted to generate factual information without verification. Design systems that verify outputs or require human review, especially for high-consequence decisions.