Overview

You're sitting with your team, and someone pulls up a generative AI tool. They ask it to draft a procurement policy, and in three seconds, out comes a 800-word document with sections on approvals, spend limits, vendor selection, and compliance. Your team reads it and someone says: "How does it do that so fast? Does it have access to your company files? How much of this is real?"

You don't need to be an ML engineer to use AI effectively, but you do need to understand the basic mechanics of how it works. Specifically: what is it actually doing when you type something in and press send? What are tokens, and why does everyone say "token limits"? What happens when you adjust temperature? Why does it sometimes make things up confidently?

The answer matters because it changes how you use the tool. A well-tuned prompt from someone who understands the mechanics will give you 10x better results than a random request from someone who doesn't. And you'll know when to trust the output and when to be skeptical.

The Fundamental Concept: Predicting What Comes Next

An LLM, a Large Language Model, is a statistical prediction machine. Here's the simplified version of what it does.

Imagine you have a well-trained operations manager who has processed 10,000 purchase orders. You show her an incomplete sentence: "The customer submitted a PO for 50 units, the unit price is $200, and the delivery location is..." and you ask her: "What word comes next?" She might say: "Chicago" (because that's the customer's main distribution center) or "warehouse" or "Texas" or "the fulfillment center in Dallas." The probability that it's "Chicago" might be 35%, "warehouse" 20%, "Texas" 15%, and so on.

She's not guessing randomly. She's applying patterns she learned from thousands of previous orders. She knows which customers have which primary locations. She knows seasonal patterns. She knows exception codes.

An LLM does the same thing, but with text on a massive scale. It's been trained on billions of documents, books, articles, contracts, code, emails, research papers. It learned patterns in language from all of that. Now, when you give it a prompt, it predicts: "Given this input, what is the most likely next token?" Then it predicts the next one. Then the next one. It keeps doing this until it either reaches your requested length, hits a stop signal, or hits its context limit.

This is the mental model you need: an LLM is a highly sophisticated word-completion engine. Not conscious. Not thinking. Not "understanding" in the human sense. Completing patterns based on statistical probability.

Why does this matter? Because it explains both why LLMs are useful and why they fail.

They're useful because language patterns encode knowledge. If the model learned "procurement policies usually have these sections" from training data, it can reproduce something similar. If it learned "when a vendor fails audit, the next step is usually escalation to compliance," it can predict the next step. The patterns in language compress an enormous amount of useful information.

But they fail because statistical probability isn't truth. Just because words often appear together doesn't mean what the model generates is accurate. An LLM can confidently generate a plausible-sounding vendor scorecard with completely made-up numbers. This is called "hallucination", not because the model is confused, but because it predicted tokens that sounded coherent but weren't grounded in fact.

Core Understanding: An LLM predicts the next token based on patterns in training data. This makes it excellent at generating coherent text on almost any topic. It makes it dangerous when you treat the output as factual without verification. The model doesn't know whether what it generated is true. It just knows that the tokens fit the pattern.

Tokens: Why They Matter

A token is a small unit of text. Usually one word. Sometimes part of a word. Sometimes punctuation. It depends on the tokenizer, the algorithm that breaks text into tokens.

Here's why you need to understand this: AI tools limit how much text you can process at once. This limit is called your "context window," and it's measured in tokens, not words.

A rough rule of thumb: 1 token ≈ 0.75 words. So 1,000 words ≈ 1,300 tokens.

Different models have different context windows:

• GPT-4: up to 8,000 tokens (fast) or 128,000 tokens (slower, expensive)
  - Claude: up to 100,000 or 200,000 tokens depending on version
  - Gemini: up to 1,000,000 tokens

The context window is your working memory. If you paste in a 50,000-word operations manual and ask the AI to reference it, the AI can only "see" as much of it as fits in the context window. The newer models have much larger windows, which is why they're better at working with long documents.

For operations work, this matters in a few ways:

First, batch processing. If you want to analyze 100 vendor contracts and you paste all of them into a prompt, you'll quickly hit token limits. Solution: process them in batches. Feed the AI 5-10 contracts, get results, repeat.

Second, prompt complexity. Your prompt itself takes up tokens. If you paste in company policy, vendor history, and current data all in one request, you've used up a chunk of your context window before you even ask a question. Good prompts are efficient. They include what's necessary and exclude what's not.

Third, output length. When you ask the AI to generate something, it uses tokens for the output too. If you ask for a 5,000-word SOP and your context window is 8,000 tokens, you're left with very little room for the input. Longer context windows give you more flexibility.

Token Efficiency: If you're hitting token limits, you have three options: use a model with a larger context window (newer models, but more expensive), break the task into smaller batches (more requests, but each one is fast and cheap), or strip down the prompt to essentials (only include what the AI truly needs to answer your question).

For operations-specific examples: if you're asking an AI to analyze a procurement process, you don't need to paste the entire 30-page policy manual. Paste the relevant sections. If you're asking it to draft a vendor scorecard, you don't need the last five years of vendor communications. Paste this year's data. Be efficient.

Training Data: Why LLMs Know What They Know

An LLM's knowledge comes from its training data. It was trained on text available on the internet plus books plus research papers plus code repositories, generally data that was public and available when the model was trained.

This creates several constraints you should know about:

Knowledge cutoff: The model's knowledge cuts off at a certain date. GPT-4 was trained through early 2024. Claude was trained through early 2024. Gemini through late 2024. They don't know about events after their training cutoff. If you ask them about a regulatory change in January 2025, they won't know about it. This matters for operations because compliance rules change, and outdated knowledge is dangerous.

No access to proprietary data: The model wasn't trained on your company's internal files, your vendor agreements, or your process documents. It doesn't "know" your specific workflows. This is actually good (your data stays private) but it means the model works from general patterns, not from your specific context. You have to supply that context in your prompt.

Training data reflects its era: The model learned patterns from data that was public. This means it learned biases from that data. If the training data overrepresents certain industries or perspectives, the model will reflect that bias. It also means the model may not reflect recent shifts in thinking, practice, or best practices. Train it on data, and it learns what was common in the data, not necessarily what's optimal now.

No continuous learning: The model doesn't learn from your conversations. Each request is independent. If you correct the model once ("That's not right, here's the actual policy"), the next time you ask a similar question, it won't remember the correction. It will make the same mistake. This is why you either need to include the correction in every prompt or use specialized techniques like Retrieval-Augmented Generation (RAG, see the terminology lesson for details).

For operations leaders, this means: use the model for general knowledge and pattern generation, but treat its output as needing verification against your current requirements and your specific data. The model is a capable assistant, not an expert in your operations.

How Prompting Works: The Input Matters

The quality of what an LLM generates depends heavily on the quality of the prompt. This is where you as the operator make a huge difference.

A bad prompt: "Draft a procurement policy."

A good prompt: "Draft a procurement policy for a mid-sized manufacturing company with $20M annual procurement spend, 15 key vendors representing 70% of spend, and ISO 9001 compliance requirements. The policy should include (1) approval authority by spend tier, (2) vendor evaluation criteria, (3) contract terms requirements, and (4) compliance checkpoints. Format as a word-document-ready outline with executive summary."

The difference is specificity. The good prompt tells the AI:

• Who is this for (mid-sized manufacturing)
  - Scale of spend ($20M)
  - Vendor concentration (70% from 15 vendors)
  - Constraints (ISO 9001)
  - What sections to include
  - Format requirements

With all this information, the AI generates something useful. Without it, it generates generic boilerplate that requires heavy editing.

Here are the elements of a strong operations prompt:

Context: Who is this for? What's the business context? (manufacturing, services, financial, regulated industry?)

Constraints: What matters? Budget limits? Compliance requirements? Scale? (Are we talking 10 vendors or 1,000? $1M or $100M?)

Output spec: What format do you want? Outline? Draft document? Analysis? Comparison table? Decision matrix?

Specific requirements: What sections, points, or elements should be included?

Exclusions (if relevant): What should NOT be included?

Example for vendor analysis:

*"I have three vendor proposals for our annual procurement. Please create a comparison matrix that evaluates each against: (1) price per unit, (2) minimum order quantity, (3) lead time, (4) payment terms, (5) quality certifications, (6) our current relationship history (favorable/neutral/problematic). Format as a table. Include a summary recommendation based on risk-adjusted total cost of ownership, not just price. Here's the data: [paste data]"*

This prompt tells the AI exactly what dimensions to evaluate, what format you want, and what your decision framework is. The output will be more useful because the prompt set clear expectations.

Prompting Pro Tip: If the first response isn't quite right, iterate. Tell the model what to adjust: "That's good, but add a risk column," or "Remove the history section and add a compliance certification column instead," or "Can you format this as a one-page summary?" The model can edit and refine much faster than it can generate from scratch. Iteration is part of the process.

Temperature: Creative vs. Consistent

Temperature is a parameter that controls how much randomness is in the model's output. Think of it as a dial from "stick to the most likely answer" to "be creative and vary your responses."

Temperature ranges from 0 to 1 (usually):

• Temperature 0.0: Deterministic. The model always picks the single most likely next token. You ask it the same question three times, you get the same answer every time. It's consistent and predictable but boring. Hallucinations are still possible but less likely because the model sticks to obvious patterns.
  - Temperature 0.3: Conservative. The model is mostly deterministic but occasionally picks the second or third most likely token. Still pretty consistent, but with slight variation. Good for fact-based tasks.
  - Temperature 0.7: Balanced. The model has freedom to be creative while still staying coherent. Good for drafting documents, brainstorming, or generating options.
  - Temperature 1.0: High creativity. The model picks from a wide distribution of possible tokens. Much more variation, more creative outputs, but also higher chance of going off-topic or generating nonsense.

For operations work, what temperature should you use?

Use low temperature (0.1-0.3):

• Analyzing data or spotting patterns (you want consistent, reliable analysis)
  - Answering factual questions (who approved this vendor? what's our spend?)
  - Classifying or categorizing items (is this high-risk or low-risk?)
  - Extracting information from documents

Use medium temperature (0.5-0.7):

• Drafting documents or SOPs (you want the structure to be sound but have some variation so it doesn't feel templated)
  - Generating multiple options (e.g., "Generate three different approaches to vendor consolidation")
  - Brainstorming process improvements
  - Writing communications (emails, policy summaries)

Use high temperature (0.8-1.0):

• Creative brainstorming with no bounds
  - Generating diverse perspectives on a problem
  - Exploring "what if" scenarios with wide latitude

Most operations leaders should default to lower temperature because you care about reliability more than creativity. You want the analysis to be right, not interesting.

Hallucinations: The Dark Side of Statistical Prediction

A hallucination is when an LLM confidently generates something that is false, misleading, or completely made up. It's not a bug. It's a feature of how the model works.

Remember: the model predicts the next token based on patterns. If a pattern in the training data is "when people talk about vendor management, they often mention creating a scorecard," the model might predict "scorecard" even if no scorecard actually exists in your scenario. It completed the pattern, not your specific need.

For operations, hallucinations are dangerous because you might not catch them immediately. Here's an example:

You ask an AI: "What are our three biggest cost drivers in procurement?" The AI might respond: "Based on typical manufacturing operations, your three biggest cost drivers are usually: (1) raw material costs, (2) shipping and logistics, (3) labor for assembly." This sounds confident and reasonable. But it's not based on your actual data. The model generated plausible-sounding information without checking whether it's true for you.

Another example: you ask an AI to generate a compliance checklist for GDPR. The model generates a checklist that sounds authoritative. But it includes requirements that were changed in 2024, and your copy is outdated. You implement the checklist, you think you're compliant, you're actually not.

How do you prevent hallucinations?

First, always specify sources. If you want the AI to use specific data, paste it in. "Based on the attached vendor scorecard, which vendor should we renew with?" is better than "Which vendor should we renew?" because it constrains the AI to your actual data.

Second, use low temperature for fact-based tasks. Higher temperature increases hallucinations because the model is being more creative.

Third, fact-check outputs that matter. If the AI generates an analysis that will affect a decision, you verify it. Does it match your data? Are the numbers right? Did it miss any important factors?

Fourth, be skeptical of confidence. An LLM that says "I don't know" is trustworthy. An LLM that confidently answers a question it shouldn't be able to answer is hallucinating. Learn to notice when the model is operating outside its competence.

Hallucination Reality: LLMs will hallucinate. You can reduce the frequency and severity by prompting carefully and using low temperature for factual tasks, but you cannot eliminate hallucinations. This is why human review is not optional. It's fundamental to safe deployment.

Fine-Tuning vs. RAG: Two Ways to Add Your Data

You have historical data, vendor contracts, SOPs, process documents, approved templates. You want the AI to know about your specific operations, not just general patterns.

There are two main approaches: fine-tuning and RAG. Both matter for operations, but they work differently.

Fine-Tuning

Fine-tuning means retraining the model on your specific data. You take a base model and feed it examples from your operations. The model learns your patterns, your terminology, your style.

In theory, this sounds ideal. The model becomes specialized in your operations.

In practice, fine-tuning is expensive and slow. It requires significant technical setup. You need hundreds or thousands of examples in the right format. The fine-tuned model is proprietary to you (which is good for privacy, bad if you need updates). If your underlying data is wrong or biased, the fine-tuned model learns the wrong patterns.

For most operations teams, fine-tuning is overkill. You don't need a permanently specialized model. You just need better results on specific tasks.

RAG (Retrieval-Augmented Generation)

RAG is simpler. Instead of retraining the model, you give it access to your documents. When you ask a question, the system finds the relevant documents in your knowledge base and includes them in the prompt.

Example: your operations manual has 150 pages. You ask the AI "What's our approval authority for contracts over $100K?" The RAG system searches your manual, finds the relevant section, includes it in the prompt, and the AI answers based on your actual policy, not general patterns.

RAG is much faster and cheaper than fine-tuning. It scales easily. You can add documents without retraining. It's more transparent. You can see which documents the system used for its answer. And it avoids hallucinations about your specific policies because it's working from your actual documents.

For operations teams, RAG is usually the right approach. You get the accuracy benefits of having your data available without the cost and complexity of fine-tuning.

The limitation: RAG only works well if your documents are clear and well-organized. If your operations manual is a mess of outdated PDFs and scattered Google Docs, RAG will struggle to find the right information.

Context Window: Your Working Memory Limit

The context window is the amount of text the model can see at once. It's measured in tokens.

Think of it as the model's working memory. If you're having a conversation with a teammate, they can keep the whole conversation in mind. If the conversation lasts 10 hours, eventually they'll forget what you said at hour 1 (unless you remind them). Same with LLMs. The context window is how much of the conversation the model can hold in mind.

Recent models have much larger context windows:

• GPT-4: 128,000 tokens (much larger than it used to be)
  - Claude: 200,000 tokens
  - Gemini: 1,000,000 tokens (!). You can paste an entire novel into a prompt

Larger context windows are better for operations because you can:

• Paste entire SOPs and ask questions about them
  - Analyze a full year of vendor performance data in one request
  - Feed multiple contracts at once for comparison
  - Have longer conversations where the model remembers earlier context

But they're slower and more expensive. A request with 200,000 tokens takes longer to process and costs more than a request with 10,000 tokens.

The strategy: use the smallest context window that solves your problem. If you're analyzing one contract, don't paste ten. If you're asking a question that needs specific data, include only the relevant data, not everything you have.

The Mental Model Recap

Here's what you need to hold in your mind to use LLMs effectively:

LLMs are pattern-prediction machines. They're trained on billions of examples of text and they predict the next token based on statistical patterns. This makes them extraordinarily good at generating fluent, coherent text on almost any topic. It makes them prone to hallucination because they don't distinguish between "this pattern is common" and "this pattern is true."

Your prompt is the spec. The better you specify what you want (context, format, constraints, specific requirements), the better the output.

Temperature controls the dial from predictable to creative. Use lower temperature for factual tasks, higher for exploratory tasks.

Tokens are the unit of cost and capacity. Understand your token limits so you don't hit walls. Be efficient with your prompts.

Context window is your working memory. Larger is better if you need to work with big documents or long conversations, but it's slower and more expensive.

Hallucinations are inevitable. Prompt carefully, fact-check outputs, keep a human in the loop for decisions that matter.

Common Questions

What is a token?

A token is a small unit of text, usually a word, part of a word, or punctuation. One token is roughly 0.75 words. Your model has a token limit (context window), and both your input and the model's output count against that limit. If you need to work with large documents, use a model with a large context window.

What does temperature mean?

Temperature controls how creative vs. predictable the model is. Low temperature (0.1-0.3) makes the model consistent and conservative. High temperature (0.8-1.0) makes it creative and varied. For operations work, use low temperature for analysis and factual tasks, medium temperature for drafting, high for brainstorming.

How is an LLM trained?

An LLM is trained on billions of documents from the internet and books. It learns statistical patterns in how text is structured and how words relate to each other. Then it's fine-tuned to follow instructions better. You don't retrain the base model. You use it as-is or you add your data via RAG. Fine-tuning is expensive and usually not necessary for operations work.

Why does the model make things up?

Because it predicts the next token based on patterns, not truth. If a pattern in training data is "contracts usually include an indemnification clause," the model might predict "indemnification" even if your specific contract doesn't have one. It completed the pattern. To prevent hallucinations, be specific in your prompt ("Here's our contract, what clauses does it include?"), use low temperature for factual tasks, and always fact-check outputs that matter.

Should I fine-tune an LLM on our operations data?

Probably not. Fine-tuning is expensive, slow, and requires technical infrastructure. Instead, use RAG (retrieval-augmented generation) to give the model access to your documents without retraining. When you ask a question, RAG retrieves relevant documents and includes them in the prompt, so the model answers based on your actual data.

What to Do Monday Morning

• Write down one task you regularly do that involves analyzing data or drafting documents. This is your use case.
• Write a detailed prompt for that task. Include context (who is this for? what's the business situation?), constraints (budget? compliance? scale?), output format (table? document? outline?), and specific requirements. Don't just ask the question, describe the job.
• Test different temperatures. Run your prompt with temperature 0.3, then 0.7. Notice the difference in output. Which works better for your task? (Spoiler: for operations work, lower is usually better.)
• Fact-check the output. Does it match your actual data? Did it make assumptions that aren't true? Did it miss something important? This is where you catch hallucinations before they cause problems.
• Iterate one time. Tell the model what to adjust. "Add a cost column," or "Focus on compliance issues," or "Make this more concise." Notice how fast the model adapts. This is your real productivity edge, iteration, not perfection on the first try.

Key Takeaways

• Understand the core mechanic: LLMs predict the next token based on patterns. This makes them good at coherent text generation. It makes them prone to hallucination. There's no magic, just statistical patterns.
• Prompt specificity matters enormously: A vague prompt gets vague output. A detailed prompt with context, constraints, and format requirements gets much better output. The time spent on a good prompt is time saved in editing and iteration.
• Temperature is your creativity dial: Lower temperature for factual analysis and consistent output. Higher temperature for brainstorming and options generation. Know which you need and set it accordingly.
• Tokens are your unit of capacity and cost: Understand what a token is, know your model's context window limit, and be efficient with prompts to avoid hitting limits or running up costs.
• Context windows are growing but speed/cost tradeoffs exist: Larger context windows are better for working with big documents, but they're slower and more expensive. Use the smallest window that solves your problem.
• Hallucinations are a feature, not a bug: You cannot prevent them, only reduce them and catch them. Low temperature, specific prompts, and human fact-checking are your defense.
• Use RAG for your data, not fine-tuning: Give the model access to your documents via RAG so it answers based on your actual operations, not just general patterns. Much cheaper and easier than fine-tuning.