Lesson 1.2: How Generative AI Works

Purpose

You don't need to understand neural networks or calculus to use AI effectively. You do need an accurate mental model of how generative AI actually works—why it behaves the way it does, why it makes certain mistakes, why context matters, and why longer conversations might get off-track.

This lesson builds that model. By the end, you'll understand why AI generates text the way it does, what "training" means, and what a "token" is (and why it matters).

Why This Matters for Managers

Many AI misconceptions come from not understanding the mechanism. You might think:

• "AI should remember everything I've told it" (misunderstanding context windows)
• "AI made an error because it's not trying hard enough" (misunderstanding probability)
• "This AI must be lying because it said different things in different conversations" (misunderstanding lack of memory)
• "AI understands language like humans do" (misunderstanding token-level prediction)

These misconceptions lead to:

• Frustration when AI behaves differently than expected
• Unrealistic trust in AI consistency or accuracy
• Poor decisions about what tasks to delegate to AI
• Difficulty explaining AI's behavior to your team

Understanding the mechanism prevents these mistakes.

Core Concepts

How Generative AI Is Trained

Generative AI models like ChatGPT, Claude, and Gemini are trained on vast amounts of text data—books, articles, websites, code repositories. The training process is not like teaching a human. It's not about explaining concepts. It's about exposing the AI to patterns.

The Training Process (Simplified)

1. Ingest text: The model reads billions of words from diverse sources.
2. Learn patterns: It identifies statistical relationships—which words typically follow other words, which topics relate to which concepts, which writing styles match which contexts.
3. Predict-and-adjust: During training, the model practices: "Given these words, predict the next word." When it's wrong, its predictions are adjusted.
4. Result: The model learns to predict plausible next words given any input.

What This Means

• Generalization: The AI can generate entirely new text it never saw during training.
• Language patterns: The AI understands common word associations ("hospital" often appears near "doctor," "surgery," "patient").
• Broad patterns: The AI learned patterns across millions of documents, so it can tackle diverse topics.
• No actual understanding: The AI has no semantic understanding of what these words mean. It predicts based on statistics.

How AI Generates Text (The Token-by-Token Process)

When you ask an AI to write an email, it doesn't generate the whole thing at once. It generates one token at a time.

What's a Token?

A token is roughly a word or word-piece. For practical purposes, think of it as a word. When you write "The quick brown fox," that's about 4 tokens. "It's a beautiful day" is about 5 tokens (the apostrophe in "It's" might be a separate token). One token is roughly 4 characters.

The Generation Process

1. You provide a prompt: "Write a brief email requesting a meeting."
2. AI sees all the tokens in your prompt: It analyzes what you've asked and the context.
3. AI calculates probabilities: For the next position, what word is most likely? It doesn't pick randomly—it evaluates thousands of possible words and assigns probability to each. "Dear" might have 45% probability, "Hi" might have 20%, "I" might have 8%, and so on.
4. AI picks a word: It selects one word based on those probabilities. Often the highest-probability word, but not always (there's randomness to avoid repetition).
5. AI repeats: That word becomes part of the context. AI now calculates the next word's probability given "Dear [the email so far]." It picks the next word.
6. Continues until: The AI generates a stop token (essentially, it "decides" the text is complete).

Context Windows and Memory Limits

AI doesn't have memory, but it has something called a context window. This is the amount of text it can "see" at once.

How Context Windows Work

When you have a conversation with an AI:

1. Message 1: You write a message. The AI sees it, responds.
2. Message 2: You ask a follow-up. The AI sees Message 1 + Message 2 + the previous response. It can reference earlier parts of the conversation.
3. Message 5: You ask another question. The AI sees all previous messages within its context window.

But—and this is critical—if the conversation gets long enough, the AI eventually "forgets" the beginning.

Context Window Limitations

Modern AI models have context windows measured in tokens. For example:

• Claude has a 200K token context window (roughly 150,000 words)
• GPT-4 has a 128K token context window
• Older models had 2K-8K tokens (roughly 1,500-6,000 words)

Within that window, the AI can reference earlier parts of the conversation. Beyond it, it can't—that text is not available.

What this means:

• Long conversations get harder for AI to track
• If you have a 2-hour conversation and expect the AI to remember something from 90 minutes ago, it might not—context limits
• Uploading a 300-page document is risky if the AI's context window is smaller than the document size

Practical implication: For important information, remind the AI repeatedly rather than assuming it remembers.

Temperature, Determinism, and Variability

You might have noticed that if you ask the same question twice, the AI gives slightly different answers. This isn't randomness in the chaotic sense. It's intentional.

Temperature Settings

AI models have a "temperature" setting that controls randomness:

• Low temperature (closer to 0): The AI picks the most likely word almost every time. Output is consistent, predictable, sometimes repetitive.
• High temperature (closer to 1): The AI takes more probabilistic risks, picking less-obvious words more often. Output is more creative but less predictable.

Most AI tools default to a middle temperature for balance—creative enough to sound natural, consistent enough to be reliable.

What this means: The same prompt can generate different outputs, even if you don't change anything. This is by design, not a malfunction.

What AI Actually Learned vs. What It Didn't

This distinction is critical and frequently misunderstood.

What AI Learned

• Language patterns: How words relate to each other, common phrases, natural writing flow
• Topic associations: Topics that appear together, concepts that relate
• Reasoning patterns: How humans typically structure arguments, explanations, problem-solving
• Style variation: How different people write differently, formal vs. casual, professional vs. conversational
• Factual patterns: Associations between topics and facts (though not necessarily accurate facts)

What AI Didn't Learn (Or Learned Poorly)

• Truth: It learned patterns about how facts are discussed, but not whether those facts are actually true
• Current events: If trained on data up to April 2024, it doesn't know about major events after that
• Proprietary information: Details specific to your organization weren't in the training data
• Reasoning from first principles: It learned how reasoning looks, but it doesn't derive truth logically
• Causation: It learned correlations, not why things cause other things
• Your preferences: Unless you tell it, it doesn't know how you like things written

Hallucinations: Why They Happen

Given what you now understand, hallucinations make sense.

An hallucination occurs when AI confidently generates false information.

Why Hallucinations Happen

The AI is doing what it's designed to do: predict plausible next words based on patterns. When you ask "What was the founding date of Company X?" the AI:

1. Recognizes the pattern of a founding date question
2. Recalls patterns about how founding dates are typically discussed
3. Generates a plausible-sounding date

If "Company X" wasn't well-represented in the training data, or if the training data contained incorrect information, the AI might generate an incorrect date with confidence.

It's not lying. The AI isn't trying to deceive. It's generating plausible text based on patterns. The problem is that "plausible" doesn't mean "true."

Why This Is Particularly Dangerous

An AI that says "I don't know" is actually less likely to hallucinate. But many AI systems are designed to sound confident and complete, because that's how humans expect text to be written. Confident-sounding text is more useful—unless it's wrong.

Why More Data Doesn't Always Mean Better

You might think bigger AI models with more training data are always better. Not entirely.

Model Size Tradeoffs

Larger models (more parameters):

• More general patterns, broader knowledge
• Better at nuanced tasks
• Slower and more expensive to run
• More capacity for complex reasoning

Smaller models (fewer parameters):

• Faster and cheaper
• Better at narrow, specific tasks
• Better for rapid response
• Sometimes more factually grounded for specific domains

Best practice: Use the right-sized model for the task, not always the biggest one.

Training Data Biases and Limitations

The AI's training data determines everything it "knows." This creates systematic blind spots and biases.

What's in the Training Data?

For large AI models:

• Heavily weighted toward English, European, and North American perspectives
• Includes published text (books, articles, websites), which has its own biases
• Skewed toward recent historical events (more recent data often overrepresented)
• Reflects the internet's gender, racial, and cultural biases
• Includes outdated information (your knowledge of current events is limited if training ended in 2023)

What This Means

• The AI reproduces biases from its training data
• The AI has gaps where its training data has gaps (e.g., knowledge of niche industries, recent developments)
• The AI's "common sense" is actually the common sense of its training data's authors—which may not be universal
• Important: This is not the AI "being biased." It's the AI learning from biased data. The fix requires intentional effort.

Practical Managerial Use Cases

Understanding Why AI Sometimes "Forgets"

Scenario: You've been working with an AI on a complex project planning task over 20 messages. In message 21, you reference something from message 3, expecting the AI to remember.

What's happening: If the conversation has gone longer than the context window, the AI can't see message 3 anymore. It's not deliberately forgetting—it literally can't see it.

What you do: Save important details. "As we discussed earlier [repeat the detail from message 3], what should we do next?" This re-grounds the AI in the context.

Understanding Variable Responses

Scenario: You ask your AI assistant to summarize a meeting. You ask the same question three times and get three slightly different summaries.

What's happening: This is expected behavior. The AI is working probabilistically. Each word choice is a statistical prediction, not a lookup.

What you do: If you need consistency, you can:

• Ask the AI to refine the best version ("Which of these three is best? Make it even more concise.")
• Use a lower temperature setting if your tool allows
• Ask for a specific format that constrains variation

Understanding When to Worry About Training Data Limits

Scenario: You ask the AI about a market development or regulatory change from last month.

What you know now: If the AI was trained on data ending in April 2024, it has no idea about May 2024 events. It might confidently generate something wrong rather than say "I don't know."

What you do: Always verify AI statements about recent events. Use AI to draft and summarize, but verify facts against current sources.

Understanding When AI Might Hallucinate

Scenario: You ask the AI to describe how your specific company's performance evaluation process works.

What you know now: Your company's evaluation process wasn't in the training data. The AI learned how performance evaluations generically work. It will try to generate something plausible based on patterns, not something accurate about your process.

What you do: Never ask AI about proprietary information without providing the specific details. Instead: "Our evaluation process works like this: [explain]. How should I describe this to a new manager?"

Examples

Example 1: Token-by-Token Generation in Action

You ask: "Write a one-sentence meeting summary."

What AI does internally:

        Prompt: "Write a one-sentence meeting summary."
        
        [AI calculates token probabilities]
"The" (25% probability)
"During" (18% probability)
"At" (12% probability)
        ...
        
        [AI selects "The" (or sometimes "During")]
        
        Now the context is "The"...
        [AI calculates next token]
"meeting" (45%)
"team" (15%)
"discussion" (12%)
        ...
        
        [AI selects "meeting"]
        
        Now the context is "The meeting"...
        [AI calculates next token]
        ...and so on until the sentence is complete.
        

Result: A complete sentence that reads naturally, generated word-by-word.

Example 2: Why Context Window Matters

Scenario: You upload a 100-page annual report to an AI and ask it to summarize it.

What happens:

• If the model's context window is 128K tokens (roughly 96,000 words), the 100-page report (roughly 25,000 words) fits.
• The AI can read the whole thing and provide an accurate summary.

But if:

• You then ask the AI about a specific detail from page 1 after having loaded 50 more messages
• The first 10 pages might have scrolled past the context window
• The AI can't see those pages anymore
• It might confidently generate something wrong

What you do: For important documents, keep them easily referenceable or re-upload them if the conversation gets long.

Example 3: Hallucination Risk in Specialized Domains

Scenario: You ask an AI to recommend a data privacy law for a specific European country.

What happens:

• The AI learned general patterns about data privacy laws
• If your country's specific law isn't well-represented in training data, the AI generates something plausible-sounding
• It sounds like it knows (because it's written confidently)
• It might be completely wrong

What you do: Verify with an authoritative source. AI is great for drafts, less reliable for factual accuracy in specialized domains.

Anti-Patterns / Misuse Risks

Misuse Risk 1: Assuming AI Consistency Within Context Window

"I'll have the AI track this project over multiple messages so it stays consistent."

Why it doesn't work as expected: While the AI can reference earlier messages within the context window, its probabilistic generation means even the same question can produce slightly different answers.

Misuse Risk 2: Overloading Context Windows

"I'll upload the entire codebase, plus the requirements document, plus the current project status, plus our design guidelines."

Misuse Risk 3: Expecting AI to Know Proprietary Information

"The AI should understand our process. I told it once."

Misuse Risk 4: Relying on AI for Facts Without Verification

"The AI said this is how the regulation works, so it must be."

Human Judgment Checkpoints

Before trusting AI's factual claims, ask:

1. Is this factual? Is the AI making a claim about objective reality, or generating creative text?
2. Is this recent? If this fact is recent, did the AI's training data include it?
3. Is this specialized? If this is specialized knowledge, was it well-represented in training data?
4. Is this proprietary? If this involves your organization, did the AI actually have access to this information?
5. Can I verify it? Can I check this fact against a reliable source?

If you answer "no" to verification, treat AI output as draft/exploratory, not final.

Responsible AI Considerations

Understanding Why AI Errors Aren't Carelessness

When AI makes an error, it's not because it's not "trying hard enough" or made a typo. It's because of how the system fundamentally works—probabilistic prediction based on training data. Understanding this builds realistic expectations.

Transparency About Limitations

When you explain AI to your team, help them understand that AI works statistically, not semantically. "The AI is excellent at mimicking writing styles, but it doesn't actually understand context the way we do" builds appropriate skepticism.

Recognizing That Better Doesn't Mean Perfect

Larger, more sophisticated AI models are better in most ways. But "better" doesn't mean "reliable for every use case." Match the tool to the task.

Practice / Reflection Prompts

1. Token Intuition: Think of a task you want AI to help with. Roughly how many tokens would be involved? (Estimate: most pages are 300-500 tokens.) Is that easily within a typical context window?

1. Hallucination Risk: What's something you'd ask AI where hallucination would be particularly dangerous? Why? How would you verify?

1. Training Data Limitations: What important knowledge about your industry or organization would an AI not have learned? How would you handle that?

1. Context Management: In a long project, how would you structure AI conversations to avoid context window issues?

1. Verification Responsibility: For a task you're considering giving AI, what would verification look like? What sources would you trust?

Key Takeaways

1. AI generates token-by-token: Text is built word-by-word based on probabilistic predictions.
2. Training determines everything: What AI "knows" comes entirely from its training data.
3. Context windows are real limits: Beyond the window, the AI can't see earlier text.
4. Hallucinations are structural: False confidentiality is built into how probabilistic generation works.
5. Verification is non-negotiable: Especially for factual claims, recent information, or proprietary details.