How do large language models work?

What is a large language model?

On the surface, a large language model (LLM) does something deceptively simple: given the beginning of a text, it predicts the next piece of it. It receives a sequence of tokens — “The cat climbed up the” — and returns a probability distribution over every possible next token in its vocabulary: “tree” 62%, “stairs” 11%, “fence” 7% (illustrative numbers), and so on across tens of thousands of further options. Generating text is nothing but the repetition of that single operation: the model draws a token from the distribution, appends it to the text, then computes the next-token distribution for the extended text. In practice part of the earlier computation can be reused from a cache; the chapter on inference covers this. This is called autoregressive generation. This guide focuses mainly on such autoregressive transformer models that produce text token by token.

What the definition hides is this: to predict the next token well, you implicitly need to know a great deal. Continuing “Paris is the capital of France, and Berlin is …” takes geographical facts; continuing “if 3 apples cost $6, then 7 apples …” takes arithmetic; continuing a half-finished Python function takes programming. Next-token prediction is therefore not a narrow task: learning many linguistic and factual regularities can help with it. Over the course of training, the model's parameters can organize into internal patterns that represent grammar, facts, style and patterns of inference, because these improve prediction. Some studies have also found internal representations of the task's world: for example the state of a board game in a model trained only on move sequences, or directions for places and times in large language models (Li et al., 2023; Gurnee & Tegmark, 2024). This does not imply that every model forms a unified or human-like world model.

Why “large”?

“Large” refers to the number of parameters: the learned numbers that determine the model's behavior. A modern LLM has billions of them — open models typically range from ~1 billion to several hundred billion, and the largest systems go beyond that. The overwhelming majority of parameters sit in a handful of enormous matrices (embedding, attention and MLP weights; see chapter 4). The size is not an end in itself: in section 5.6 we will see that performance follows surprisingly regular scaling laws — more parameters and more data give predictably better prediction.

The life cycle of a model

A modern LLM is built in two large stages. During pretraining the model learns next-token prediction on an enormous corpus of trillions of tokens — this is where the raw capability comes from (chapter 5). Post-training then shapes it into a usable assistant: supervised fine-tuning, preference-based training (RLHF, DPO) and safety training (chapter 6). Running the finished model — inference — is an engineering field of its own, with its own tricks (chapter 7). Finally, chapter 8 is about what researchers still understand only partly: what actually happens inside.

Tokenization: from text to a sequence of numbers

A neural network does not see letters, it sees numbers. The first step is therefore to cut the text into the entries of a fixed vocabulary — these are the tokens — and then to replace each piece with its index in that vocabulary. The question is: how big should the pieces be?

Two extremes present themselves, and both are bad. If we tokenize character by character, the vocabulary is tiny (a few hundred entries), but the sequences become very long and the model would have to reassemble the meaning of every word from characters every single time. If we tokenize word by word, the sequences are short, but the vocabulary explodes (even in English, “play, plays, played, playing, player, players, replay, unplayable…” would each need a separate entry — and in morphologically richer languages the explosion is far worse), and every unfamiliar word — a new name, a typo, a rare compound — becomes unmanageable. The practical answer lies in between: subword tokenization, where frequent words get a single token and rare ones break into meaningful pieces. The dominant algorithm is byte-pair encoding (BPE).

The core idea of BPE is a compression principle: whatever frequently occurs together, merge into one symbol. The vocabulary is not written by hand but learned from data: we start from characters (more precisely, bytes) and repeatedly merge the most frequently adjacent pair in the corpus until the vocabulary reaches the desired size — typically somewhere between 32,000 and 260,000 entries.

The vocabulary and special tokens

A vocabulary may contain learned subwords and reserved special token IDs for text endings, role boundaries or tool calls. Typing a marker as text is different from inserting the actual control token. The tokenizer API and chat serializer determine when special tokens are allowed; BPE alone does not guarantee this separation. Untrusted input must be encoded as content rather than as a role boundary. This alone does not prevent prompt injection. Example: tiktoken allowed_special / disallowed_special. A larger vocabulary may shorten sequences but increases the embedding table.

# corpus: byte sequences; goal: a vocabulary of size V
vocab = {all 256 bytes}
merge_rules = []
while |vocab| < V:
    pair = the most frequent adjacent (x, y) token pair in the corpus
    new_token = concat(x, y)
    vocab.add(new_token)
    merge_rules.append((x, y) → new_token)
    corpus = replace every occurrence of (x, y) with new_token

Embeddings: meaning as a direction in a vector space

Token IDs on their own are meaningless serial numbers: token 9042 is no more “similar” to 9043 than it is to 517. The network needs a representation in which similarity is computable. So we assign every token a learned vector — a sequence of several thousand numbers — and from then on the model works with these vectors rather than with tokens. This assignment is called the embedding.

The decisive point: embeddings are not designed by hand but learned by the model, with the same gradient method as every other parameter. Under the pressure of training the geometry starts to carry meaning: tokens that occur in similar contexts — and therefore have to contribute similarly to similar predictions — receive vectors close to one another. The vectors for “dog” and “cat” end up near each other; “dog” and “integral” far apart. What is more, the directions carry meaning too: moving along a particular direction changes grammatical number, gender, tense, or even formality.

How large is this space?

The length of the embedding vectors is the model's hidden dimension, d_model — a few hundred in small models, around 4,000–20,000 in large ones. The embeddings of all tokens form a single table: a matrix of size V × d_model (where V is the vocabulary size). With a 128k vocabulary and a dimension of 8,192, that alone is more than a billion parameters. An important look ahead: this vector is only the starting point — the dictionary meaning of the token. The transformer layers (chapter 4) gradually rewrite it into the contextual meaning: the vector of the token “bank” becomes something different after passing through the layers in “on the river bank” than in “the bank approved the loan”.

The transformer architecture

4.1 · The block: attention and MLP

The transformer is the engine of modern LLMs. Its structure is surprisingly repetitive: the same block repeats one after another, a few dozen or a few hundred times. Every block consists of two sub-layers: an attention layer, responsible for the flow of information between tokens — the only place where positions of the same sequence can “talk” to one another — and an MLP layer, which processes every position on its own, but with large capacity. The division of labor between them is the heart of the whole architecture: attention moves information, the MLP processes it.

4.2 · The residual stream: the backbone of the architecture

Before going into the details of attention, it is worth fixing the best mental model of the whole. Every token position carries a d_model-dimensional vector that receives additive updates through the layers — this is the residual stream. The blocks do not “process and pass on” the data like a conveyor belt: instead, every sub-layer reads the current contents of the stream, computes something useful from it, and adds the result back to the stream. Moving toward the output, the stream carries the contributions of more and more layers, as a sum. Additions can also cancel previous components: information does not necessarily accumulate or improve monotonically.

This seemingly minor design decision — the residual connection — matters in three ways. Trainability: addition provides an identity path for gradients, helping optimize deep networks without guaranteeing stable training. Incremental updates: each sub-layer updates the existing state, potentially reinforcing or cancelling earlier components. Interpretability: the additive structure helps separate component contributions; this is one important tool in chapter 8.

4.3 · Attention: which token should read from where?

Attention moves information between tokens, but causal direction matters. In “The crane nested high”, the position of “crane” cannot yet see the later word “nested”. The position of “nested” and subsequent positions can combine the earlier “crane” with evidence for the bird meaning. Attention can help through learned retrieval. Each position has three roles:

• Query — “what am I looking for?”: the token states what information it needs.
• Key — “what do I offer?”: every token advertises what information it can provide.
• Value — “what do I hand over if chosen?”: the content actually to be passed on.

The mechanism: the token's query vector is compared with the key vectors of the current and all earlier tokens (by a dot product: q · k = ‖q‖ ‖k‖ cos θ, so both direction and vector lengths matter). Wherever the match is strong, a large weight goes; softmax normalizes the weights into a distribution; and finally the token receives the weighted average of the value vectors and adds it to its own stream. So it does not read from a single addressee but from everyone at once, in proportion to relevance.

The causal mask and multi-head attention

Two important additions. First: since the task is predicting the next token, a position may read only from the past — attention toward future tokens is forbidden by the causal mask (before the softmax we write −∞ into the scores of future positions). This is what lets training practice prediction at every position at once: the output at position t depends only on the first t tokens, so a single forward pass yields T training examples.

Second: attention does not run once but many times in parallel — this is multi-head attention. Every head works with its own query/key/value projections in a smaller subspace, so different heads can specialize in different relations: one attends to the immediately preceding token, another to the subject of the sentence, a third to earlier occurrences of the same word (this will be the induction head of 8.3). The outputs of the heads are concatenated and returned to the stream through a shared output projection.

4.4 · The MLP layer: the processing and knowledge unit

The other half of the block, the MLP (multi-layer perceptron, also called the feed-forward layer), runs at every position separately, with the same weights — no new information flows directly between positions here. Its structure is simple: a large matrix projects up the stream vector into a much wider space (typically about 4× the model dimension), an element-wise nonlinearity acts on it there, and another matrix projects it back down into the original dimension. Most of the parameters — roughly two thirds — sit in these matrices.

What is it for? In a conventional two-projection MLP, removing the activation collapses the projections into one affine map (linear when there are no biases). This reduces the MLP's expressiveness; the full transformer remains nonlinear, because attention and normalization remain nonlinear. Gated variants also introduce nonlinearity through multiplication. The original architecture describes these as separate components. More intuitively: the MLP can also be read as a key–value memory. In the row-vector convention below, each column of the up-projection matrix is a pattern detector (a key): to what degree is a particular combination of features present in the input? The nonlinearity modulates detector activations — and the corresponding row of the down-projection matrix (the value) writes the content belonging to the activated detectors into the stream. Many factual associations — “Eiffel Tower → Paris” — are demonstrably stored in the MLPs of the middle layers.

The key–value expression is algebraically exact, but attaching human meaning is an interpretation. Neuron input-weight vectors are concrete parts of the matrix; nevertheless, a neuron can respond to multiple patterns and a concept can be distributed across components. Superposition is one researched explanation. A mathematical “key” should not be equated with a unique human concept.

4.5 · Normalization: keeping the signal in shape

Normalization helps control the scale of sublayer inputs. LayerNorm subtracts the component mean and divides by √(variance + ε); RMSNorm divides by √(mean(x²) + ε) without centering. A learned elementwise gain follows, usually with a bias for LayerNorm. Pre-norm places normalization before the sublayer, inside the residual branch. The direct residual path remains an identity, which can improve gradient flow. Xiong et al. obtained good results without warmup in certain settings; this is not a general stability guarantee, and warmup requirements depend on the full training recipe.

4.6 · Positional encoding: how does the model know what is where?

Without positional information or a mask, self-attention is permutation equivariant: permuting input rows permutes output rows in the same way. Both rows and columns of the attention matrix are reordered; the matrix does not remain unchanged. Explicit positional encodings provide location or distance information. A causal mask itself already introduces an ordering asymmetry by restricting which positions are available.

First-generation solutions added position as a vector to the embedding (a fixed sinusoidal pattern or a learned positional embedding). Today's standard is RoPE (rotary position embedding), which is more elegant: it adds nothing to the content but rotates the query and key vectors by an angle proportional to their position. The dot product of two rotated vectors depends on position solely through the difference of the two rotation angles — formally ⟨R_mq, R_nk⟩ = ⟨q, R_n−mk⟩ — that is, the attention score automatically senses the relative distance of the tokens rather than their absolute places. (The value of the dot product of course also depends on the two vectors themselves; the point of the claim is that position enters only through the difference m−n — the rotation is orthogonal, so it leaves the norms untouched.) This is exactly the property language calls for: the relation “next to each other” means the same at the start of a text as in the middle.

4.7 · The output head: from vector to probability

After the last block, the stream vector passes through final normalization and the unembedding matrix (W_U ∈ ℝ^d×V). Each token’s logit is a dot product with the corresponding matrix column, plus an output bias if used. Dot products depend on both direction and magnitude: the highest cosine similarity alone need not give the highest logit. Softmax converts logits into probabilities:

Exponentiation gives positive weights, and normalization makes the probabilities sum to 1. Softmax turns logit differences into ratios: pᵢ / pⱼ = exp(zᵢ − zⱼ). Increasing one logit by 1 multiplies that token's probability ratio relative to each other token by e (~2.7×), not its own probability. With two initially equal logits, its probability goes from 50% to e / (e + 1) ≈ 73%. Chapter 7 lets you also explore temperature.

4.8 · Modern variants: what has the industry changed?

The core of the 2017 recipe is surprisingly stable, but almost every component has been optimized since. The most important directions:

Training: where do the weights come from?

The weights of a freshly initialized model are random numbers and its output is meaningless noise. The aim of training is that out of this noise — purely from examples and a measure of error — the good parameters of the machinery in chapter 4 should emerge. The principle of the recipe is simple, and essentially all of deep learning is built on this one loop:

1. Take a batch of text from the corpus and run the model on it.
2. Measure with a loss function how bad the prediction was.
3. Compute, for every single parameter, how much a small change to it would improve the loss — this is the gradient, and backpropagation computes it.
4. Move every parameter a little in the direction of improvement (the optimizer).
5. Repeat — for large models, over many millions of steps and many trillions of tokens.

5.1 · The loss function: cross-entropy

What does “a bad prediction” mean? At every position the model produced a distribution over the next token — and the corpus tells us what the next token actually was. The cross-entropy loss is simply the negative logarithm of the model's probability for the correct token: if the model gave 90% to the correct token, the loss is small (−ln 0.9 ≈ 0.105); if it gave 1%, the loss is large (−ln 0.01 ≈ 4.6). Training therefore literally optimizes for the texts that were actually written to be as probable as possible according to the model — this is the maximum likelihood principle.

5.2 · The computational graph and automatic differentiation

To compute the gradient, notice that the loss is a function of the parameters — a monstrously complex one, but built out of elementary steps (matrix multiplication, addition, softmax…). Modern frameworks (PyTorch, JAX) record this structure during the forward pass into a computational graph: every node is an operation, every edge a data dependency. Computing the gradient is then mechanical: moving backwards through the graph, each operation multiplies the incoming gradient by its own (known, simple) derivative. This is automatic differentiation (reverse-mode autodiff) — backpropagation is its name in the neural-network world.

5.3 · Backpropagation: industrializing the chain rule

The essence of backprop is the chain rule, familiar from school: the derivative of a composite function is the product of the partial derivatives. In the computational graph, a long chain of operations stands between the loss and a deep parameter — and the gradient multiplies its way back along that chain. The genius is in the organization: instead of walking the chain separately for every parameter (which would be one forward pass per parameter — unworkable at a billion parameters), moving backwards shares the common partial products: the gradient at every node is computed once, and every parameter picks up its own at its own node. This is how all the gradients together come out for roughly the price of two forward passes.

5.4 · Optimizers: from the gradient to the step

AdamW maintains two exponential moving averages per parameter: the gradient and its square. The first provides momentum; the second supports coordinatewise scaling. Step size still depends on the learning rate, these statistics and ε: updates are not uniformly unit-sized, and good conditioning or convergence is not automatic. SGD remains a valid method, although adaptive optimizers are often more practical for large language models.

5.5 · Training in practice: data, precision, parallelism

All of the above works on a single GPU — but training a large model is an engineering operation running on tens or hundreds of thousands of accelerators for months. The main topics:

Data. The quality of the multi-trillion-token corpus (web crawl, books, code, scientific text, and increasingly synthetic data) is the number one determinant of the end result. The pipeline: language identification, quality filtering (heuristics and learned classifiers), deduplication (duplicates cause memorization and distorted capabilities), PII and toxicity filtering, and then careful design of the mixture ratios — toward the end of training often a “curriculum”: over-weighting the highest-quality data.

Precision. bfloat16 uses 1 sign bit, 8 exponent bits and 7 stored fraction bits; float32 uses 1, 8 and 23 respectively. Their exponent ranges match, but bfloat16 has lower precision and half the storage. It does not have “half the mantissa”. Mixed-precision training can use different formats for matrix operations and sensitive accumulations, including FP8. Speedups depend on hardware and operations rather than being automatically 2–4×. Loss scaling is particularly relevant to FP16’s narrower range. BFLOAT16 study.

Parallelism. A large model fits on a single GPU neither in weights nor in activations, so the work is sliced along three axes, typically all three at once (“3D parallelism”):

Memory tricks. Backprop requires keeping the intermediate activations of the forward pass — at long sequences this dominates memory. Gradient checkpointing strikes a bargain: it stores the activations of only every k-th layer and recomputes the rest during the backward pass (about a third more computation, a fraction of the memory).

5.6 · Scaling laws: the physics of the field

In many experiments, test loss is well approximated by power laws in model size, data and compute. With a nonzero floor E, the ideal straight line on logarithmic axes applies to excess loss L − E. Fits support planning within the studied data, architecture and scale ranges; extrapolation adds uncertainty. A lower loss does not automatically establish success on a particular new task.

Fine-tuning and alignment: from raw model to assistant

The pretrained “base model” has formidable knowledge but is not an assistant: asked “How do I bake bread?”, it may well answer with a list of further questions — because on the web, questions often continue that way. The base model continues what is likely, not what is useful. The job of post-training is to close that gap: to shape the model's behavior to be helpful, honest and harmless — without damaging the raw capabilities.

The three stages

1 · Supervised fine-tuning (SFT). The model is trained further on high-quality example dialogues — with the same next-token loss as before, only now on examples of “correct behavior”, and with the loss typically computed on the assistant's response tokens alone. This is what establishes the role: a question is followed by an answer, not by another question.

2 · Reward model. The notion of a “good answer” is hard to write down as rules, but people find it easy to pick the better of two answers. From such pairwise judgments a separate model learns — the reward model — which assigns a score to any answer. It is a compressed, machine substitute for human taste.

3 · Reinforcement learning. The model now learns from its own generated answers: it responds, the reward model scores, and the parameters shift so that highly scored behavior becomes more likely. A critical component is the KL brake: the model is penalized for drifting too far from the starting (reference) model — without it the policy finds the reward model's weak spots and exploits them (reward hacking: confident nonsense, flattery, artificial length).

Inference: running the finished model

Training gave us the weights — but using the model is an engineering world of its own. Prefill processes the prompt: positions can be computed in parallel, so sufficiently long prompts are often compute-bound. Decoding then proceeds token by token. At small batch sizes, reading weights and the KV cache often makes memory bandwidth the bottleneck. This is not universal: batch size, context length, architecture, and hardware change the balance. The KV cache reuses keys and values for earlier tokens. These differences motivate measuring time to first token and generation speed separately. Detailed cost model and assumptions.

Sampling: from a distribution to a token

The model's output is a distribution — but a concrete token has to be chosen. The greedy choice (always the most likely) is deterministic, but often produces flat, repetitive text. In practice we sample from the distribution, and the sharpness of the sampling is governed by the temperature parameter: the logits are divided by T before the softmax. T<1 sharpens the distribution toward the peak (more conservative, more prone to repetition), T>1 flattens it (more creative, more prone to error). As a complement, top-p (nucleus) filtering cuts off the long, noisy tail of the distribution: we sample only from the smallest set of tokens whose combined probability reaches p.

The KV cache: so we do not recompute everything

For an unchanged model and prefix, earlier K/V vectors in a causal decoder do not depend on newly appended tokens. The KV cache stores them per layer. Each new position still requires Q, K and V projections, attention and the MLP; its K/V vectors are appended, and its query attends to permitted previous and current keys. Dense attention for one new position is linear in context length, without rerunning the entire prefix. Cache storage also grows linearly and can exceed weight storage at long contexts.

Mechanistic interpretability: what happens inside?

8.1 · The reverse-engineering program

The previous chapters told us how the model computes — but not what. The weights are learned, not designed: nobody prescribed what any given neuron should mean. The research program of mechanistic interpretability (mech interp) is to reverse-engineer the trained network as though it were an unknown machine: to identify its internal variables (features) and the algorithms that connect them (circuits). The stakes are not merely scientific curiosity: if we can see what the model computes, we can audit its behavior from the inside as well as from the outside by testing — we can investigate possible internal differences between honest answers and strategic compliance (chapter 6), and look for causes of errors. This is a research goal: current methods do not provide a general, reliable lie detector or safety guarantee.

8.2 · The residual stream view and the logit lens

The natural coordinate system for reverse engineering is the residual stream from 4.2. Every component — the embedding, the attention heads, the MLPs — writes into the same shared vector space, additively; and output logits are linear maps of the vector after final normalization. Direct attribution fixes the normalization scale observed in that run; removing a component and rerunning the model can have a different effect. Two immediate, practical tools fall out of this. Direct logit attribution: since every component's contribution is a term of a sum, it can be computed component by component how much each pushed the “Paris” logit up. And the logit lens: the intermediate state of the stream can be passed through the unembedding after any layer — so we can watch, layer by layer, what the model currently “thinks” the continuation is. The typical picture: early layers still work in the surface world of tokens, the semantics come together in the middle layers (often along with the correct answer), and the late layers sharpen the phrasing.

An important limitation: the raw logit lens easily misleads. The representations of early layers do not yet “live” in the space of the unembedding — applying the matrix W_U to them directly often yields uninterpretable noise or a systematically distorted prediction, and how far it remains usable varies strongly by model family. This is why the tuned lens variant was developed: an affine transformation is trained per layer to fit the intermediate state into the output space before the unembedding is applied. This gives a substantially smoother and more reliable layer trajectory. The logit lens is therefore best treated as a fast, cheap probe rather than a measuring instrument — and every conclusion drawn from it should be confirmed by causal intervention (8.6).

8.3 · Induction heads: the first circuit we understood

The finest textbook example that “circuit” is not a metaphor. The phenomenon: in many transformers a simple but broadly useful algorithm can be observed — “if the pattern [A][B] occurred earlier and I now see [A] again, predict [B]”. It is implemented by a circuit of two heads spanning two layers: a previous-token head in an earlier layer copies onto every position the identity of the token preceding it; building on that, the query of an induction head in a later layer uses the current [A] token to find the position whose “previous token was A” — that is, the earlier [B] — and forwards the content of the token it found toward the output. The composition of two learned heads = a search-and-copy algorithm.

8.4 · Superposition: more concepts than dimensions

Why is it not enough to look at neurons? According to the superposition hypothesis, a network can store more features than it has representational dimensions if they are encoded in overlapping directions. This can work when features are sparse — only a few are active at any time — since a high-dimensional space has room for exponentially many nearly orthogonal directions. In simple toy models this can be studied in detail: there, optimization does pack features into slightly overlapping directions. Extending the result to real language models needs separate evidence; the methods of the next section (8.5) try to provide exactly that. The price: interference noise (simultaneously active features “talk into” one another a little) — and the fact that individual neurons become polysemantic, since a neuron sees the projection of many feature directions.

8.5 · Sparse autoencoders: a dictionary for thoughts

If features hide in overlapping directions, we need a tool that disentangles them. That is the sparse autoencoder (SAE): an auxiliary network trained to decompose the model's activations into a much wider but sparse code — every activation should be expressible as the sum of a few “dictionary elements”. The learned dictionary elements are surprisingly often monosemantic: they correspond to individual human concepts — the word “often” carries weight here; the known limitations of the method (reconstruction residual, feature splitting, seed dependence) are covered in this chapter's research block. From large models, with dictionaries of a million entries, features have been extracted such as “Golden Gate Bridge”, “security vulnerability in code”, “flattery”, “inner conflict” — and the features work causally too: artificially amplifying their activation shifts behavior predictably (the famous demo, “Golden Gate Claude”, steered every conversation to the bridge).

8.6 · Methodology: how do we know we are right?

The methodological backbone of mech interp is causal intervention: seeing a correlation is not enough (“this head gives a large weight to X”), one has to show that the component causes the behavior. The toolkit:

• Ablation: switch the component off (zero it out or replace it with the mean) — what breaks? If nothing, the component is not critical to the behavior under study (or it is redundantly backed up).
• Activation patching: run the model on two inputs (for example “The city of the Colosseum:” / “The city of the Eiffel Tower:”) and transplant the activation of a single component from one run into the other. If the answer switches from Rome to Paris, you have found where the information travels. Scanning systematically maps out the flow of information.
• Attribution graphs / circuit tracing: the latest generation of tools (2025) locally approximates the model with an interpretable “replacement model” and draws, for a concrete prompt, the feature→feature influence graph: which feature activated which along the path to the answer. This is how it first became visible that a model plans the rhyming word ahead while writing verse, or that mental addition runs parallel approximate and exact-digit paths.

Capabilities and limits: what can we expect from the machinery?

A few characteristic behaviors follow necessarily from the machinery of the previous chapters — good and bad alike. This chapter ties the two together: it derives both the capabilities and the failure modes from the mechanism.

In-context learning: learning without a weight update

One of the strangest capabilities: from a few examples given in the prompt, the model “learns” a new task — even though not one of its weights changes. The explanation is architectural: attention can move information from the examples to the current question at runtime, and induction-style circuits (8.3) implement exactly pattern matching and continuation. It is important, however, that the induction head is not the complete explanation of in-context learning: Olsson et al. demonstrated a strong co-occurrence between the formation of these heads and the abrupt appearance of the ability, but on the current picture copying heads, MLP layers carrying task and format representations, and considerably more complex attention patterns all take part in ICL as well. The induction circuit is the best understood component, not the only one. The context is therefore a kind of fast, temporary memory in which the model tunes its learned general algorithms to the task at hand. In a research reading: under certain conditions the transformer's forward pass implements implicit optimization steps over the examples in the context (the “mesa-optimization” literature).

Step-by-step thinking

A forward pass is a computation of fixed depth: N layers, exactly that many. Whatever cannot be computed in that many steps, the model cannot produce correctly in a single token — but the generated tokens can themselves be turned into a computational scratchpad. The chain of thought (writing out the reasoning) is exactly this: the model externalizes its intermediate results as tokens and then conditions on them at the next step — “deepening” the computation along the sequence dimension. Modern reasoning models make a system of this: they learn through RL (5.6, chapter 6) to use thinking tokens efficiently — to undo an error, to try a branch, to check.

Why does it hallucinate?

Hallucination — a falsehood delivered with confidence — is not a mysterious malfunction but the shadow of the training objective. Several of its sources can be derived from the mechanism:

• The loss rewards continuation, not truth. Cross-entropy rewards plausible text; a well-sounding false claim has a low loss if its surface pattern is common. On top of that, “I don't know” scores worse than a guess on most benchmarks and in most preference data — so post-training can inadvertently teach confident guessing (the exam-taker's strategy: leaving it blank is a guaranteed loss of points).
• Knowledge is compressed into a distribution. Rarely seen facts are weak, noisy associations (chapters 3 and 4.4); after one bad token from sampling (chapter 7) the model conditions on its own mistake and coherently builds on it.
• Recall and phrasing come apart. Interpretability case studies (8.6) find a separate “I know this entity” signal in the network: if it switches on mistakenly, the inhibition on the answer-generating circuits is released and the model fills in the details — the structure is there, the content is not.

Mitigation can combine training for uncertainty and abstention, retrieval from external sources (RAG), tool use and verification. The mere presence of a citation does not establish that it exists or supports the claim. There is no established general guarantee eliminating hallucinations across open-ended tasks. Probabilistic modeling alone does not prove impossibility, however: constrained, verifiable tasks can admit strong correctness guarantees through formal restrictions or verification.

Further characteristic limits

Glossary

Attention

The mechanism by which a token position reads information from other positions with learned relevance weights (query–key matching, value transfer). 4.3.

Autoregressive

Generation proceeding token by token, where every new token is conditioned on the entire text so far. Chapter 1.

Backpropagation

The efficient, graph-based execution of the chain rule: the gradient of every parameter is produced in a single backward pass. 5.3.

BPE (byte-pair encoding)

A vocabulary-building algorithm: the repeated merging of the most frequent adjacent pairs of symbols. Chapter 2.

Chain of thought (CoT)

Writing out the intermediate reasoning as tokens, which deepens the computation along the sequence dimension. Chapter 9.

Circuit

An identifiable cooperation of network components (heads, neurons, features) that implements an algorithm describable in human terms. Chapter 8.

Cross-entropy loss

The negative logarithm of the model's probability for the correct token; the objective of training. 5.1.

DPO (direct preference optimization)

An alignment method that optimizes directly from preference pairs, without a separate reward model or RL loop. Chapter 6.

Embedding

The assignment of a learned vector to a token; the starting point of the geometric representation of meaning. Chapter 3.

Feature

A unit of the model's internal representation — on the working hypothesis, to a good approximation a direction in activation space. Chapter 8.

Gradient

The vector of partial derivatives of the loss with respect to the parameters; it shows how much a small change to each parameter would improve things. Chapter 5.

GQA (grouped-query attention)

Several query heads share fewer key/value heads — shrinking the KV cache. 4.8.

Induction head

An attention head that finds and continues the repetition of a pattern seen earlier in the context; one of the foundations of in-context learning. 8.3.

Inference

Running the trained model (prefill plus token-by-token decoding). Chapter 7.

KV cache

Storing the key/value vectors of earlier tokens so that they need not be recomputed during decoding. Chapter 7.

KL divergence

A measure of the difference between two distributions; in RLHF, the brake on drift away from the reference model. Chapter 6.

Logit

The raw pre-softmax score for a given vocabulary entry. 4.7.

Logit lens

A diagnostic: passing an intermediate state of the residual stream through the unembedding — the forming prediction becomes visible layer by layer. 8.2.

LayerNorm / RMSNorm

Standardizing the scale of the vectors at each layer, for the stability of training. 4.5.

MLP (feed-forward layer)

A per-position up- and down-projection with a nonlinearity; most of the parameters and stored associations live here. 4.4.

MoE (mixture of experts)

Many expert MLPs, of which only a few activate per token — large capacity, small per-token computation. 4.8.

Perplexity

The exponential of the loss: how many ways the model is “undecided” per token on average. 5.1.

Polysemantic neuron

A neuron that activates for several unrelated concepts — a symptom of superposition. 8.4.

Pretraining

Next-token training on a giant corpus, which gives the raw capabilities. Chapter 5.

Residual stream

The per-position vector that receives additive updates through the layers — the communication backbone of the architecture. 4.2.

RLHF

Reinforcement learning from human feedback: policy optimization guided by a reward model learned from human preferences. Chapter 6.

RoPE (rotary position embedding)

Positional encoding by rotating the query/key vectors in proportion to position; attention thereby senses relative distance. 4.6.

SAE (sparse autoencoder)

An auxiliary network that decomposes activations into a wide, sparse, often monosemantic dictionary of features. 8.5.

Softmax

The transformation of a vector of scores into a probability distribution, by exponentiation and normalization. 4.7.

Superposition

Storing more features than there are dimensions — in overlapping, nearly orthogonal directions, exploiting sparsity. 8.4.

SFT (supervised fine-tuning)

Supervised fine-tuning on example dialogues; teaching the form of assistant behavior. Chapter 6.

Temperature

A sampling parameter: the divisor of the logits, sharpening or flattening the distribution. Chapter 7.

Token

The atomic unit of text for the model — typically a word piece (a BPE fragment). Chapter 2.

Transformer

An architecture of attention and MLP blocks built on a residual stream, trainable in parallel. Chapter 4.

Unembedding

The matrix that projects the final stream vector onto the logits of the vocabulary. 4.7.

Further reading — the canon, in reading order

Scientific review · 19 September 2026 (v1.1.1). The review covered the 12 learning stages, 60 quiz questions, worked examples and 14 interactive labs. Corrections and primary sources are recorded in the review notes. Toy illustrations are not measurements from a real LLM; this review is not independent academic peer review.

The list is deliberately short: the foundational texts of the field, roughly in the order of this material's chapters. The foundational texts are freely available; linked titles lead directly to the original sources.

1. Vaswani et al. (2017): Attention Is All You Need — the original transformer paper. (Chapter 4)
2. Radford et al. (2019): Language Models are Unsupervised Multitask Learners (GPT-2) and Brown et al. (2020): Language Models are Few-Shot Learners (GPT-3) — the discovery of scaling and in-context learning. (Chapters 1 and 9)
3. Sennrich et al. (2016): Neural Machine Translation of Rare Words with Subword Units — the introduction of BPE into NLP. (Chapter 2)
4. SentencePiece: official documentation — BPE and unigram implementations and tokenizer training. (Chapter 2)
5. Rumelhart, Hinton & Williams (1986): Learning representations by back-propagating errors — the classic of backprop; as a modern complement, Karpathy: Neural Networks: Zero to Hero, a video series from a hand-built autograd to a GPT. (Chapter 5)
6. Kingma & Ba (2015): Adam and Loshchilov & Hutter (2019): Decoupled Weight Decay Regularization (AdamW). (5.4)
7. Kaplan et al. (2020): Scaling Laws for Neural Language Models and Hoffmann et al. (2022): Training Compute-Optimal Large Language Models (Chinchilla). (5.6)
8. Su et al. (2021): RoFormer — RoPE. (4.6)
9. Ouyang et al. (2022): Training language models to follow instructions with human feedback (InstructGPT) — the RLHF recipe. (Chapter 6)
10. Bai et al. (2022): Constitutional AI: Harmlessness from AI Feedback. (Chapter 6)
11. Rafailov et al. (2023): Direct Preference Optimization. (Chapter 6)
12. Dao et al. (2022): FlashAttention and Kwon et al. (2023): Efficient Memory Management for LLM Serving with PagedAttention (vLLM). (Chapter 7)
13. Leviathan et al. (2023): Fast Inference from Transformers via Speculative Decoding. (Chapter 7)
14. Elhage et al. (2021): A Mathematical Framework for Transformer Circuits — the residual stream / QK–OV formalism. (8.2)
15. Olsson et al. (2022): In-context Learning and Induction Heads. (8.3)
16. Elhage et al. (2022): Toy Models of Superposition. (8.4)
17. Li et al. (2023): Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task and Gurnee & Tegmark (2024): Language Models Represent Space and Time — studies of internal “world” representations. (Chapter 1)
18. Olshausen & Field (1996): Emergence of simple-cell receptive field properties by learning a sparse code for natural images — the classic of sparse coding, the conceptual ancestor of SAEs. (8.5)
19. Bricken et al. (2023): Towards Monosemanticity and Templeton et al. (2024): Scaling Monosemanticity — SAEs at small and frontier scale. (8.5)
20. Lindsey et al. (2025): On the Biology of a Large Language Model — attribution graphs, case studies (planning, mental arithmetic, unfaithful CoT). (8.6)
21. Schaeffer et al. (2023): Are Emergent Abilities of Large Language Models a Mirage? — the emergence debate. (5.6)
22. Berglund et al. (2023): The Reversal Curse. (Chapter 9)
23. Greenblatt et al. (2024): Alignment Faking in Large Language Models. (Chapter 6)

This material was written for teaching. Its subject matter reflects the state of the field as of July 2026; the last update to this edition was 27 September 2026 — the difference between the two is set out in the changelog. The numerical examples (sizes, ratios) illustrate typical orders of magnitude, not the specifications of particular products. The figures are schematic. Corrections and suggestions are welcome: csaplar.d@gmail.com. ◆

Changelog

This material covers a fast-moving field. The list below is in reverse chronological order, so you can see how current the version you are reading is.

v1.2.0 — 27 September 2026

Shared AI-tananyagok design, dark mode by default. Scientific clarifications after an external review: the guide’s scope (autoregressive transformers), generation loop consistent with caching, the scope of world modeling and superposition (as a hypothesis, separating toy and real models), the induction-figure caption; language fixes; direct links in the bibliography.

19 September 2026 — Labels

The navigation, links and example labels now consistently use “Interactive experiments”.

v1.1.3 — 19 September 2026

Order-independent quiz explanations; corrected family-relation, SAE, Adam, memory and emergence questions; more beginner-friendly wording. Fixed language-specific builds with new regression coverage. Saved progress is unchanged.

v1.1.2 — 19 September 2026

Quizzes: length-based clues to the correct answer were substantially reduced; the explanations once again say why the right answer is right and why the wrong one is tempting. The learning components are built separately per language, so the English edition carries no Hungarian text, and vice versa. New share card. Your progress is kept.

v1.1.1 — 19 September 2026

Scientific review: corrected attention/MLP formulas, bfloat16, scaling laws, speculative decoding, KV caching and calibration; revised 28 quiz items. Replaced the scaling sketch with computed curves. See the review notes for corrections and primary sources. Existing progress is preserved.

v1.1 — 19 September 2026

Fourteen new interactive workshops; worked foundations; a twelve-week practice path with local checklists; free reading and guided learning modes; distinct depth and progress labels; scientific corrections and balanced introductory quiz choices. Both editions remain self-contained and work offline.

v1.0 — 30 August 2026

First public release. The twelve chapters were turned into a level-based learning system: each chapter closes with a five-question quiz, a 4/5 pass mark and unlimited retries; chapters you have not yet completed stay locked. Progress lives in your browser's local storage — nothing is sent to a server. Light and dark modes designed separately.

English edition, translated from the Hungarian original.