The transformer is the fundamental architecture underlying modern LLMs. Understanding its structure explains both capabilities and limitations.

Key points:

• Transformers process all input tokens simultaneously, rather than sequentially (unlike older neural network designs)
• This parallel processing is why they can handle long documents and why they're fast
• They consist of multiple "layers" (typically 40-100 layers in modern LLMs) that progressively refine understanding
• Each layer adds another level of pattern recognition on top of previous layers
• This layered architecture means early layers might catch surface patterns while deeper layers recognize abstract concepts
• The architecture is fundamentally statistical: it assigns probabilities, not certainties
• For researchers, this means: transformers work well for tasks fitting their parallel processing nature (analyzing documents, extracting information) but struggle with tasks requiring sequential, step-by-step logic

Attention is what enables LLMs to "focus" on relevant parts of long documents rather than treating every word equally.

Key points:

• Attention mechanisms allow the model to weight which input tokens are most relevant to understanding each part of the text
• When processing a sentence about "banks," attention helps distinguish between financial banks and river banks based on context
• Each token can attend to every other token, creating a map of which parts of the text are related
• Attention heads (multiple parallel attention processes) can specialize: some might track pronouns while others track numerical quantities
• This is why LLMs can handle documents with multiple threads of discussion and keep track of which statement relates to which topic
• Token limit constraints partly exist because attention's computational cost grows with the square of the number of tokens
• For researchers: attention is why you can ask an AI to "focus on the results section" or "compare these two papers"—the system can selectively attend to relevant information

Everything an AI system processes is broken into tokens—and this has direct practical consequences for your research use.

Key points:

• A token is not exactly a word; typically, one token 0.75 words in English (some words are multiple tokens)
• Non-English text often requires more tokens per semantic unit
• Code and mathematical notation are token-inefficient (they often require multiple tokens per symbol)
• The model's context window is stated in tokens, not words (Claude 3.5 Sonnet has 200,000 tokens; ChatGPT-4 has 128,000)
• If your context window is 200,000 tokens, and your document uses 150,000 tokens, you have 50,000 tokens left for analysis
• When you ask follow-up questions, previous conversation tokens still count against your limit
• Researchers need to know this to understand why pasting massive datasets requires being selective about prompt length
• One paper 5,000-15,000 tokens; you can fit 10-40 typical papers into a 200,000 token context window

LLMs generate text probabilistically, not deterministically. Understanding this prevents frustration.

Key points:

• Temperature is a parameter controlling how much randomness (creativity) is in the model's output
• Low temperature (0.2-0.5): model picks the most likely next token consistently; outputs are more repetitive and deterministic
• High temperature (0.7-1.0): model explores less likely alternatives; outputs are more creative and varied
• For research tasks requiring consistency (extracting specific data, analyzing code), low temperature is better
• For creative tasks (brainstorming research directions, generating multiple hypotheses), higher temperature makes sense
• Even with temperature set to 0, there can be small variations due to computational rounding
• When you ask the same question three times and get three different analyses, the system isn't being inconsistent—that's the designed behavior
• Researchers should note: for data analysis, extract specific numbers at low temperature; for synthesis, use higher temperature to see multiple perspectives

The context window is the total amount of text the model can see and analyze at once.

Key points:

• Context windows have grown dramatically: from 2,000 tokens (2021) to 200,000+ tokens (2024)
• Longer context windows let you ask the model to analyze entire datasets, books, or multiple papers at once
• However, longer context windows have trade-offs: they're more expensive (tokens cost money), slower, and can make it harder for the model to find information in very long contexts
• Some research suggests that models pay more attention to the beginning and end of their context (the "lost in the middle" problem)
• For researchers, this means: yes, you can paste your entire dissertation, but the model might miss details in the middle; sometimes it's better to ask about specific sections
• Very long context windows are powerful for tasks like: comparing multiple papers, analyzing entire datasets, or reviewing a full literature review
• They're less effective for: finding one specific piece of information in a very long text (because the signal gets diluted)

—

Practical Research Use Cases

Scenario: Your lab has conducted a multi-year study measuring gene expression in patient tissue samples. You have 500 samples with 20,000 genes measured per sample, producing CSV files totaling 2 GB.

Without AI understanding: You try to paste the entire dataset into ChatGPT and ask "What patterns do you see?" The system either crashes, hits token limits, or spits out useless analysis because it's trying to process too much data as text.

With understanding of tokens and context: You recognize that 2 GB of raw CSV data is millions of tokens—way beyond any context window. Instead, you use Python with AI assistance to generate summary statistics: per-sample means, variance analysis, correlation matrices. These summaries might be 10,000 tokens. You ask the AI to identify the most interesting patterns in the summary, then ask follow-up questions about specific patterns. You've translated your problem into a form the AI can actually process.

Scenario: You're reviewing 15 papers on machine learning applications in medical imaging to understand the state of the field.

Without understanding attention and context: You ask ChatGPT about all 15 papers at once. The model begins analyzing them but your context window fills up partway through, or the model's attention becomes scattered because processing 15 complex papers simultaneously is demanding.

With understanding: You recognize that 15 papers x 10,000 tokens per paper = 150,000 tokens for papers alone. Claude's 200,000 token window gives you 50,000 tokens for your questions and the AI's responses. You group papers by methodology (5 using deep learning, 5 using traditional ML, 5 using hybrid approaches) and ask the model to analyze each group, then synthesize across groups. You structure your comparative analysis to match how attention mechanisms work—keeping related information together.

Scenario: You need to extract reported sample sizes, effect sizes, and p-values from 10 published studies for a meta-analysis.

Without understanding temperature and sampling: You paste all 10 papers and ask "Extract all sample sizes." You run this three times to verify consistency. Two times you get N=245, one time you get N=252 for the same study. You're frustrated by the "inconsistency" and decide the AI is unreliable.

With understanding: You recognize that the model is working probabilistically. For extracting specific numerical data, you set temperature to 0 (or as close as possible) to reduce variation. You ask the model to format output as a structured table with citations. You still verify a few data points by checking the original papers, but you understand why variation exists—not because the AI is malfunctioning, but because this is how probabilistic systems work.

Scenario: You're developing a research hypothesis about how social isolation affects disease outcomes in your population. You want to brainstorm potential mechanisms and confounding variables.

Without understanding sampling and temperature: You ask an LLM "What mechanisms could explain the relationship between social isolation and disease?" You get one answer, accept it, move on.

With understanding: You recognize this is a creative, exploratory task where multiple perspectives are valuable. You set temperature higher (0.7-0.8) and ask the same question three different times, generating different hypotheses. You ask about confounding variables multiple times, getting varied perspectives. You use this as brainstorming input—not definitive answers, but generative material to feed your own critical thinking. You also ask the AI to identify gaps in existing literature and push back on its own suggestions.

—

Hands-On Exercise

Objective: Understand how token limits and context windows constrain what you can ask AI systems to do.

Steps:

1. Find a token counter: Use an online token counter (search "OpenAI token counter" or use Claude's built-in counter) to count tokens in sample research documents

1. Estimate your context window usage:

• Count tokens in your most recent paper (typically 5,000-15,000)
• Count tokens in a typical research question (100-500 tokens)
• Multiply to understand: if your context is 200,000 tokens, how many papers can you analyze simultaneously?

1. Test the lost-in-the-middle effect:

• Take a medium-length paper (10,000-15,000 tokens)
• Ask Claude or ChatGPT to extract the main finding from the abstract (beginning)
• Ask it to extract the main finding from the conclusion (end)
• Ask it to extract a specific methodological detail from the middle section
• Document which questions it answers correctly
• Reflect: Did it struggle more with middle content?

1. Observe temperature effects:

• Choose a small research task (extract key variables from a methods section)
• Run it three times and note variation (if using a system where you can adjust temperature, set to 0)
• Then ask an open-ended question three times at higher temperature
• Document the difference in consistency
• Reflect: How would you want these settings adjusted for different research tasks?

1. Estimate token efficiency:

• Paste 2-3 papers into an AI system
• Ask: "What methodology was used?" (requires finding information in middle of context)
• Ask: "What datasets are mentioned?" (might be scattered throughout)
• Ask: "Compare these papers' sample sizes" (requires synthesizing information)
• Note which questions are answered well versus poorly
• Reflect: Which types of questions are efficient use of context windows?

Time required: 20-30 minutes

—

Common Mistakes and Misconceptions

This is like saying a microscope's magnification doesn't matter because you can get closer to specimens. Token limits are hard constraints. When you hit them, the model stops processing. Understanding your token budget prevents asking for analysis the system literally cannot perform. A researcher wanting AI to analyze a 500-page book learns this the hard way when the system only analyzes the first 200 pages.

Researchers often think more randomness means more creative thinking. For some tasks (brainstorming, generating multiple hypotheses), that's true. But for data extraction, analysis, and fact-finding, higher temperature introduces noise. A model set to temperature 1.0 asked to extract effect sizes will hallucinate numbers more readily than at temperature 0.2. Matching temperature to task type matters.

The "lost in the middle" phenomenon is real: models often pay less attention to information buried deep in long contexts. If you paste a 15-paper literature review and ask about the 7th paper's findings, the model might miss details that would have been caught if that paper was positioned first. Researchers working with long contexts should strategically organize their input and sometimes repeat the most important information.

Attention mechanisms are sophisticated pattern-weighting systems, not consciousness or understanding. The model weights which input tokens correlate with output—but correlation is not understanding. The model attending to the word "disease" when processing "cancer" doesn't mean it understands disease the way a doctor does. For researchers, this means attention mechanisms are powerful for finding relevant information, not for drawing novel causal conclusions.

The transformer excels at parallel processing of sequential information (text, code, time series data) but has inherent limitations. It's not designed for spatial reasoning (3D structures), causal reasoning (what truly causes what), or tasks requiring genuine arithmetic. A researcher expecting an LLM to do complex mathematical proofs will be disappointed. Understanding that transformers are specialized tools prevents misdirection.

—

Key Takeaways

• Transformers process all tokens in parallel, enabling fast analysis of long documents while attention mechanisms allow models to focus on relevant information and ignore noise
• Tokens are the currency of AI processing (typically 0.75 words per token), and your context window is your total processing budget—understand this to avoid hitting limits mid-analysis
• Temperature controls the randomness of outputs: low temperature (0.2-0.5) for consistent data extraction, higher temperature (0.7-1.0) for creative brainstorming
• Context windows have grown to 200,000+ tokens but introducing the "lost in the middle" problem where middle-positioned information receives less attention than beginning and end
• LLMs generate text probabilistically, not deterministically: the same question asked twice may yield different answers, which is design behavior not a bug
• Different research tasks require different settings and strategies: data extraction differs from synthesis differs from hypothesis generation

—

Reflection Questions

1. Estimate your typical research tasks: When you ask an AI system questions about your work, are you usually asking it to extract specific information (needs low temperature), synthesize across sources (higher temperature), or create novel ideas (highest temperature)? How should you adjust your approach?

1. Context window constraints: For your current research project, how many of your typical research documents (papers, datasets, notes) could you fit into a 200,000-token window? Where would you hit limits if you tried to analyze everything at once?

1. Information architecture: If you wanted an AI system to synthesize information across 5 papers, how would you structure that request to work with attention mechanisms rather than against them? (Hint: Should you paste them in order of importance, or grouped by theme?)

1. Reproducibility and consistency: In your own research, where do you need the AI system to be highly consistent (extraction tasks) versus where do you want varied perspectives (brainstorming)? How would you design prompts differently for each?