From Neural Networks to Language Models

Remember that neural networks are just mathematical functions that learn patterns? Language models apply that same basic principle to text. They learn patterns in how language works, what word typically follows what word, how sentences structure themselves, how ideas connect.

But there's a specific architecture that makes language models work well: transformers. Every large language model you've heard of (Claude, GPT-4, Llama, Gemini) uses some variation of the transformer architecture. Understanding transformers is the key to understanding how these systems work.

The Transformer Architecture

A transformer processes text by breaking it into tokens (pieces of text, usually words or subwords) and then running them through multiple layers of processing. The key innovation is something called "attention."

Imagine you're reading a sentence: "The bank executive told the bank teller to count the money from the bank." The word "bank" appears three times, but it means something different each time (financial institution, riverbank, savings). How do you know which meaning is correct?

You read the context. You look at the surrounding words and figure out which meaning makes sense. "Bank executive" and "bank teller" suggest financial institution. "Riverbank" would make the other meaning clear.

Attention is how transformers do this. Each token (word) in the sequence can "look at" every other token and figure out which ones are relevant. "Bank teller" pays attention to "executive" and "money" to figure out it means financial institution. This happens through weighted connections, some words are attended to more strongly than others.

The actual mechanism is mathematical (it involves matrices, dot products, and learned weights), but the concept is simple: each word can attend to context, and the strength of attention is learned during training.

This is why transformers are so good at language. Language is fundamentally about context. The same word means different things in different contexts. Attention lets the model capture these contextual relationships.

Deep Architecture: How Transformer Layers Build Understanding

Modern language models like Claude or GPT-4 stack many transformer layers (50-100+) on top of each other. Each layer refines the representation of tokens. Early layers learn simple patterns: grammatical structure, word relationships, noun-verb agreement. Middle layers learn semantic concepts: if a token represents a person, other tokens referring to pronouns get updated to reflect this. Late layers learn abstract concepts: sentiment, reasoning chains, domain-specific knowledge.

In early layers, a mention of "he" in a sentence pays strong attention to nearby nouns to figure out which person "he" refers to. In middle layers, the same position might pay attention to action verbs to understand what the person did. In late layers, the representation might encode abstract knowledge: this is a decision-making scenario, the person is in a position of authority, the outcome is important.

This layered processing is why transformers can handle complex language understanding. It's not that the model reasons step-by-step (it doesn't). It's that the layered architecture allows patterns to build on each other, creating increasingly sophisticated representations of meaning as information flows through layers.

Attention is the Secret Sauce: The transformer architecture's attention mechanism is what makes language models work. It's the difference between a model that just predicts words based on patterns and a model that actually captures meaning. Everything else in a transformer (and they're actually pretty simple architecture-wise) supports attention.

How Language Models Predict Words

Here's what a language model actually does: it predicts the next word (or token, technically) in a sequence.

You give it "The quick brown" and it predicts the next word is probably "fox." You give it "She opened the door and" and it predicts the next word is probably something like "saw" or "found."

The model does this by running the input through its transformer layers and producing a probability distribution over all possible next tokens. "Fox" might get probability 0.7. "Dog" might get 0.1. "House" might get 0.05. And so on.

Then it samples from that distribution. Usually it picks the highest probability word (greedy decoding). Sometimes it samples randomly according to the probabilities (which introduces variety). Sometimes it uses a hybrid approach.

Once it predicts the next token, it adds that to the sequence and repeats. "The quick brown fox" → predict next word → "The quick brown fox jumps" → predict next word → "The quick brown fox jumps over" ... and so on until it decides to stop.

This is why language models are called "language models". They literally model the probability distribution of language. They're statistical engines that have learned, from examples, what text is likely to come next.

Tokens and Context Windows

An important limitation: language models can only look back so far. They have a maximum context length, often called the context window. GPT-3.5 had a 4k token context window. GPT-4 has up to 128k. Claude has 200k.

This matters because tokens don't map 1:1 to words. "Unbelievable" might be 1 token. "Don't" might be 2 tokens ("do" and "n't"). A token is roughly 4 characters on average.

So a 200k token context window is about 150,000 words, or roughly 500 pages of text. That's a lot, enough to include a full document and have the model reference it throughout the conversation.

Context window matters for your use case because it determines what information the model can have access to. If you're asking Claude to analyze a 100-page report, it fits in the context window. If you're asking it to analyze 500 pages of history, you need to feed it in chunks or use a tool that retrieves relevant excerpts.

Temperature and Randomness

Language models have a parameter called "temperature" that controls how random their outputs are. Low temperature (near 0) means the model picks the most likely next token every time. It's deterministic and consistent. High temperature (1.0+) means the model samples more randomly from the probability distribution. It's more creative but less predictable.

This is why you get different outputs each time you ask the same question to Claude, temperature introduces randomness. If you want reproducible outputs (like for testing), you can set temperature to 0.

Temperature is a lever. Use low temperature when you want reliable, consistent answers (like code generation, fact retrieval). Use higher temperature when you want creative outputs (like brainstorming, writing variations).

Why They Seem to Understand

People often say "ChatGPT understands my question" or "Claude seems to really get what I'm asking." This is natural language, but technically misleading.

What's actually happening: the model has learned statistical patterns about language so well that its predictions happen to align with human understanding. When you ask "How do I implement a binary search?", the model predicts tokens that form a coherent explanation of binary search. It predicts those tokens because, in its training data, explanations of binary search follow a similar structure and contain similar concepts.

The model isn't reading your question and then reasoning about the answer. It's processing your question (encoding it into patterns) and predicting the statistical likely continuation of that text (which happens to be an explanation of binary search).

This is an important distinction because it sets expectations. The model can predict plausible-sounding answers to questions it's never seen before. But if you ask it something that isn't well-represented in its training data, or something that requires genuine reasoning beyond pattern matching, it might confidently generate something that sounds right but is actually wrong.

Understanding vs. Pattern Matching: Large language models don't "understand" in the human sense. They do pattern matching at such high sophistication that the outputs seem like understanding. This is useful to remember when something surprising happens, the model isn't being clever, it's just predicting statistically likely text given its training.

Training Data and Knowledge Cutoffs

Language models learn from training data. They can't know anything that wasn't in their training data (unless it's mathematically derivable from patterns in the training data).

Claude's training data includes examples up to early 2024. That's the knowledge cutoff. It doesn't know about events that happened in mid-2024 or later (unless you tell it). It doesn't have real-time access to the internet. If you ask it about a current event that happened after its training, it will tell you it doesn't know.

This is why many AI systems have tools to access real-time information. The model itself is frozen (it doesn't learn from conversations or internet). But the system around it can fetch current information and feed it to the model, so the outputs can reference recent events.

Hallucinations and Confidence

One of the most important things to know about language models: they can be confidently wrong.

A hallucination is when a language model generates text that seems plausible and is presented with confidence, but is actually false or fabricated. It might cite a paper that doesn't exist, make up statistics, or describe events that never happened.

This happens because the model's training objective is to predict the next token that's likely to follow, not to be accurate. If your training data includes a lot of text about how to [do something dangerous], the model learns to predict tokens that continue that text, even if the instructions are wrong or will cause harm.

Similarly, if you ask the model about something obscure, it might generate a plausible-sounding answer because it's predicting tokens that follow questions about that topic, even if no actual answer is in the training data.

This is why you need to validate AI output. Always assume it might be wrong, especially on factual claims, citations, or code it hasn't tested. Use AI to generate drafts and starting points, not truth.

Case Study: Hallucination in Legal Context

A legal team in 2023 asked GPT-4 to research case law about a specific tort. The model cited "Anderson v. Baker, 2019" as supporting their position. The team included this in a brief filed with the court. The judge later informed them the case didn't exist. The model had generated a plausible citation that sounded like real case law (using the right format, reasonable date range, plausible names) but was entirely fabricated. The model had learned the statistical pattern of how case citations look but had no knowledge that this particular case existed (or didn't exist). This is a hallucination: the model's prediction of "what tokens should follow" produced something that looked authoritative but was false.

This happened because: (1) Legal databases include discussion of cases that don't exist yet (hypothetical discussion, proposed legislation), (2) Discussion of false cases sometimes appears in training data, (3) The model's training objective is to predict plausible text, not verify citations exist, (4) The model's architecture has no mechanism to "verify" against an external database.

Solutions: (1) Use retrieval-augmented generation where the model queries a legal database and cites only cases it finds, (2) Have humans verify any citations before use, (3) Build validation into the system: "Verify this citation exists before including it in output", (4) Understand this is a fundamental limitation of pure language models and design workflows around it.

Fine-Tuning and Instruction-Following

Raw language models just predict the next token. They're not optimized for being helpful or following instructions. That requires additional training called fine-tuning.

After the base model is trained on a large corpus of text, companies like Anthropic do additional training on curated data where the model is shown questions and ideal answers. The model learns to predict these helpful answers rather than just continuing text.

This is why Claude is better at following instructions than the base model it came from. It's been fine-tuned to be helpful, harmless, and honest. The same is true for ChatGPT, Gemini, and other commercial language models.

Fine-tuning is also why different models have different personalities. A model fine-tuned on medical Q&A will behave differently than one fine-tuned on customer service. They're the same architecture with different learned behaviors.

What Comes Next

Now that you understand how language models work, the next lesson talks about the broader AI stack, how these models are deployed, what APIs and infrastructure sit around them, and how you actually use them in production.

Before You Move On

Understand prediction, not reasoning: Language models predict plausible text. They don't reason in the sense of following logical chains. This understanding prevents you from assuming they can do things they can't.

Internalize the token concept: Understand that context windows are limited. 200k tokens sounds like a lot until you realize it's only 500 pages. This affects how you structure prompts and what information you can pass to the model.

Remember hallucinations are baked in: The model will confidently generate wrong information sometimes. Always validate output, especially factual claims. Treat AI as a starting point, not ground truth.

Frequently Asked Questions

Q: If language models just predict the next token, how can they write coherent essays?

A: Because predicting the next token at scale, across many tokens, produces coherent output. Each token prediction is based on all previous tokens. The model learns patterns from training data like "essays have introductions, body paragraphs, and conclusions," so it naturally generates structures that follow those patterns. Coherence emerges from statistical patterns, not conscious intent.

Q: Does increasing temperature make the model smarter or just more creative?

A: Just more creative (and less predictable). Temperature doesn't change what the model knows; it changes how it selects from what it knows. High temperature makes it more likely to pick less probable tokens. Low temperature makes it pick the most probable token every time. For factual questions, use low temperature. For creative tasks, higher temperature can be better.

Q: Can I fine-tune a model to make it genuinely better at something, or just change its style?

A: Fine-tuning can do both. For style (follow instructions, communicate differently), fine-tuning is perfect. For capability (making it actually better at solving hard problems), fine-tuning has limits. You can't teach a model something entirely outside its base knowledge. But you can teach it to apply what it knows differently.

Q: Why don't language models just look up facts on the internet instead of hallucinating?

A: Because they can't, by design. Language models run on token prediction, not database lookup. They don't have access to the internet or external systems (unless you build that in). Retrieval-augmented generation (RAG) solves this by feeding the model documents to draw from. But pure language models can only work with what's in their training data and your prompt.

Q: If a model is trained on data from 2023, can it ever know about events after that?

A: Not from its training. If you tell it about post-2023 events in your prompt (retrieval augmentation), it can reason about them. But it can't spontaneously know about them. This is why real-time systems pair models with search or document retrieval, to close the knowledge gap.

Q: Why does the model seem to "understand" code structure if it's just predicting tokens?

A: Code has very clear structure and patterns. If statements always have "if", "condition", "then code". Function definitions have "function name", "parameters", "body". The training data includes millions of code examples. The model learns these patterns so well it can predict syntactically valid code. Understanding? Not really. Pattern matching at such high fidelity that the output is valid code. This is why AI is good at code completion but bad at novel algorithms. It can predict patterns it's seen, but can't invent fundamentally new solutions.

Q: If the model has 200k token context, why can't it reliably summarize 200k token documents?

A: Because not all tokens are used equally. Early tokens in a long context window get "diluted" in attention. The model's attention mechanism is good at focusing on relevant parts of text, but for very long documents, important information in the middle or beginning might get less attention than recent information. This is why many systems don't send entire large documents to the model. Instead, they break them into chunks, analyze each chunk, and aggregate results. Or they use retrieval to find the most relevant passages and send those instead of the whole document.

Q: What's the difference between a "base model" and an "instruction-tuned" model?

A: Base models are trained purely on next-token prediction. They're good at continuing text (complete the pattern) but not at following instructions (respond to a question). Instruction-tuned models are fine-tuned on question-answer pairs, so they've learned to predict helpful responses to user queries. Base models might start hallucinating random text when given a question. Instruction-tuned models try to answer the question. All commercial models (Claude, ChatGPT, Gemini) are instruction-tuned. If you use an API, you're using the instruction-tuned version.

Q: Is there a way to make a language model more "certain" about its answers instead of hallucinating?

A: Partially. Lowering temperature makes outputs more consistent and less random, but doesn't reduce hallucinations (both occur at low temperature). What helps: (1) Including verifiable information in the prompt (RAG), (2) Asking the model to cite sources, (3) Training the model on data where citations are common, (4) Building validation into your workflow (checking outputs against reliable sources), (5) Using smaller models fine-tuned on your domain instead of general models. None of these eliminate hallucinations, but they reduce the likelihood and make them easier to catch.

Key Insight

Language models predict the next token based on statistical patterns learned from training data. They don't understand, reason, or access the internet. They predict plausible continuations of text. Everything they do follows from this fundamental mechanic.

On This Page

From Neural Networks to Language Models
The Transformer Architecture
How Language Models Predict Words
Tokens and Context Windows
Temperature and Randomness
Why They Seem to Understand
Training Data and Knowledge Cutoffs
Hallucinations and Confidence
Fine-Tuning and Instruction-Following
What Comes Next
Before You Move On

Chapter Details