Transformers llm architecture

An interpretability experiment where an internal activation from one run of a model is copied into another run, and the change in output shows which component carries the relevant information. It helps locate the heads or layers responsible for a behaviour.

Example. Copying one attention head's output from a prompt about Delhi into a prompt about Mumbai flips the predicted answer, revealing that head's role.

Related: Mechanistic interpretability, Attention head, Residual stream, Induction head

The number of parameters actually used to process a single token. In dense models it equals the total; in mixture-of-experts models it is much smaller, and it drives compute per token while total parameters drive memory.

Example. A model with 47B total parameters that uses 13B per token needs memory for 47B but computes like a 13B model.

Related: Mixture of experts, Dense model, Parameter count

An early attention scoring method, introduced for machine translation, where the relevance of an input to the current state is computed by a small feed-forward network over both, passed through tanh. It preceded the dot-product form used in Transformers, which is faster on matrix hardware.

Example. In a GRU translation model, score = v^T tanh(W1*decoder_state + W2*encoder_state) is computed for each source word, then normalised to attention weights.

Related: Attention mechanism, Scaled dot-product attention, Sequence-to-sequence model

Attention with Linear Biases: a position method that adds no position vectors, but subtracts a penalty from attention scores that grows linearly with the distance between query and key, with a different slope per head. It helps models handle inputs longer than seen in training.

Example. A key 10 tokens back gets a penalty of 10 * m added to its score, where m is that head's slope.

Related: Positional encoding, Rotary position embedding, Relative position encoding, Context window

One of the parallel attention computations inside a multi-head attention layer, with its own query, key and value projection matrices. Each head produces its own pattern of weights, and researchers have found heads that track things like previous tokens, syntax links or repeated text.

Example. Llama-style 7B models typically use 32 heads per layer, so a 32-layer model has 1,024 attention heads in total.

Related: Multi-head attention, Induction head, Head dimension, Mechanistic interpretability

A way for a model to decide which parts of its input matter most for the current step. It scores how relevant each input item is, turns the scores into weights that sum to 1, and takes a weighted average of the items. This lets the model focus on distant but relevant information.

Example. When translating 'The cat sat on the mat' word by word, the model puts most attention weight on 'cat' while producing the matching verb form.

Related: Self-attention, Scaled dot-product attention, Attention weights, Cross-attention

The matrix of attention weights of one head for a given input, showing which token each position attends to. Inspecting patterns can reveal behaviours like attending to the previous token, but they are only part of what a head computes.

Example. A plotted pattern showing a bright diagonal line one step below the main diagonal indicates a previous-token head.

Related: Attention weights, Attention head, Mechanistic interpretability, Induction head

A phenomenon in which autoregressive Transformers place a large share of attention weight on the first few tokens, regardless of their meaning, giving attention a place to put weight when nothing is relevant. Streaming methods keep these tokens in the cache so generation stays stable.

Example. When a chat runs long and old tokens are evicted, keeping the first 4 tokens in the cache prevents quality from collapsing.

Related: KV cache, Sliding window attention, Attention weights, Context window

The numbers produced by applying softmax to attention scores, one per pair of positions, that sum to 1 across each row. They show how much each token draws from every other token and can be plotted as a heatmap, though they are only a partial explanation of model behaviour.

Example. For a 6-token sentence, each token has 6 attention weights, for example [0.05, 0.60, 0.10, 0.10, 0.10, 0.05].

Related: Attention mechanism, Scaled dot-product attention, Attention pattern, Softmax

Producing a sequence one element at a time, where each new element is conditioned on all the ones generated before it and is then fed back as input for the next step. Each token needs a full forward pass, so long outputs take proportionally longer.

Example. To write a 200-token reply, an LLM performs 200 sequential forward passes, appending one token each time.

Related: Next-token prediction, KV cache, Decoding, Teacher forcing

A model that generates or scores a sequence by factorising its probability into a product of each element given the earlier ones. In language modelling it means predicting tokens left to right, using the model's own outputs as inputs when generating.

Example. P('I love India') = P(I) * P(love | I) * P(India | I love).

Related: Autoregressive generation, Decoder-only model, GPT, Causal masking

An encoder-decoder Transformer pretrained by corrupting text, for example by deleting spans, shuffling sentences or masking tokens, and training the decoder to reconstruct the original. It suits generation tasks such as summarisation and translation.

Example. A summarisation model fine-tuned from BART reads a 700-word news article and writes a 3-sentence summary.

Related: Encoder-decoder model, BERT, GPT, T5

A decoding method that keeps the several most probable partial sequences at each step, called the beam, extends each, and keeps the best again, finally returning the highest-scoring sequence. It explores more than greedy decoding and suits translation, but tends to be bland for open-ended chat.

Example. With beam width 4, the decoder tracks 4 candidate sentences at each step instead of 1.

Related: Greedy decoding, Length penalty, Decoding, Sequence-to-sequence model

Bidirectional Encoder Representations from Transformers: an encoder-only model pretrained by masking random words and predicting them, plus a sentence-pair task. Its representations are fine-tuned for many language-understanding tasks and it popularised the pretrain-then-fine-tune approach.

Example. A BERT model is fine-tuned on 5,000 labelled Hindi-English sentiment examples to classify tweets.

Related: Encoder-only model, Masked language modelling, Bidirectional attention, GPT

Attention in which every token can see all other tokens, both earlier and later, with no causal mask. It gives richer representations for understanding tasks such as classification and tagging, but cannot be used directly for left-to-right text generation.

Example. In BERT, the word 'bank' in 'river bank' is encoded using words both before and after it.

Related: Causal masking, Encoder-only model, BERT, Masked language modelling

A sparse-attention Transformer that combines a sliding window, a few global tokens and random connections, so cost grows linearly with length while keeping theoretical properties of full attention. It was designed for long documents such as articles and genomic sequences.

Example. BigBird lets an encoder process a 4,096-token research paper in one pass.

Related: Sparse attention, Longformer, Sliding window attention, Quadratic attention cost

In a mixture-of-experts layer, the setting that fixes how many tokens each expert can accept per batch, as a multiple of the average even share. Tokens beyond an expert's capacity are dropped or passed through unchanged, so it trades quality against efficiency.

Example. With 1,000 tokens and 8 experts the even share is 125; a capacity factor of 1.25 lets each expert take up to 156 tokens.

Related: Switch Transformer, Load-balancing loss, Top-k expert routing

Blocking each position in attention from seeing later positions, done by setting scores for future tokens to negative infinity before softmax so their weights become zero. It ensures a model trained to predict the next token cannot cheat by looking ahead, and is what makes decoder-only models generative.

Example. For a 4-token input, token 2 can attend only to tokens 1 and 2; the score matrix is lower-triangular after masking.

Related: Decoder-only model, Attention mask, Next-token prediction, Bidirectional attention

Decoding in which allowed tokens are restricted at each step, by masking logits, so the output must follow a grammar, schema or pattern such as valid JSON. It guarantees the format at the cost of some extra computation.

Example. To get valid JSON, the decoder masks every token that would break the schema, so after '{"age":' only digits are allowed.

Related: Logit bias, Decoding, Log probabilities

The maximum number of tokens a model can consider at once, covering the prompt, any documents, the conversation so far and the reply. Beyond it, earlier text must be dropped or summarised. Longer windows allow bigger documents but cost more memory and compute.

Example. A model with a 128,000-token window can take a roughly 300-page book in one prompt, while a 4,096-token window cannot.

Related: Sequence length, KV cache, Lost in the middle, Position interpolation

A decoding method that favours tokens a strong model finds much more likely than a weaker model does, while filtering out implausible tokens. It aims to reduce generic, repetitive or hallucinated text by subtracting what a small model would say anyway.

Example. A large model's log-probability minus a small model's log-probability is used to rank the next tokens.

Related: Decoding, Log probabilities, Top-k sampling

Attention in which queries come from one sequence and keys and values come from another. In an encoder-decoder model, decoder positions use it to read the encoder's representation of the input. It is also used to condition image generators on text and to inject retrieved passages.

Example. In English-to-Hindi translation, each Hindi word being generated queries the encoded English sentence through cross-attention.

Related: Encoder-decoder model, Self-attention, Sequence-to-sequence model

The part of a Transformer that generates output one token at a time, using causal self-attention over what it has produced so far and, in encoder-decoder models, cross-attention to the encoder output. It ends with a layer that converts vectors into next-token probabilities.

Example. A translation decoder produces Hindi tokens one by one, each conditioned on earlier Hindi tokens and the English encoding.

Related: Encoder, Decoder-only model, Encoder-decoder model, Causal masking

A Transformer with only a decoder stack and causal attention, trained to predict the next token. It generates text by repeatedly predicting and appending tokens. This is the design behind GPT-style and most current chat models.

Example. Given 'The capital of India is', a decoder-only model predicts ' New' then ' Delhi' one token at a time.

Related: Causal masking, Next-token prediction, GPT, Encoder-only model

The process of turning a language model's next-token probabilities into actual output text by choosing a token at each step under some strategy, such as always taking the most likely one or sampling. The strategy strongly affects how creative, repetitive or accurate the output is.

Example. The same prompt gives a fixed answer with greedy decoding but varied answers when sampling at temperature 1.0.

Related: Greedy decoding, Beam search, Top-p sampling, Temperature

A model in which every parameter is used to process every token, in contrast with sparse designs such as mixture of experts that use only a subset per token. Dense models are simpler to train and serve, but their compute cost grows in step with their size.

Example. A 70B dense model uses all 70 billion parameters for each generated token.

Related: Mixture of experts, Active parameters, Parameter count

A variant of beam search that splits the beam into groups and penalises later groups for choosing tokens already chosen by earlier ones, giving a set of outputs that differ from one another instead of near-copies.

Example. Asking for 4 image captions returns four distinct phrasings rather than four small variations of the same sentence.

Related: Beam search, Decoding, Length penalty

An encoder pretraining method in which a small generator replaces some tokens with plausible fakes and the main model learns to detect which tokens were replaced. Because it learns from every token, not just masked ones, it trains more efficiently than masked language modelling.

Example. In 'The chef cooked the meal', the generator swaps 'cooked' for 'ate' and the model must flag that position as replaced.

Related: BERT, Encoder-only model, Bidirectional attention

The part of a Transformer that reads the whole input with bidirectional attention and turns it into a set of context-aware vectors, one per token. Those vectors can be used for classification, search embeddings or as input to a decoder.

Example. A BERT encoder turns a 20-token customer review into 20 vectors of size 768.

Related: Decoder, Encoder-only model, Encoder-decoder model, Bidirectional attention

A Transformer with two stacks: an encoder that reads the full input and a decoder that generates the output while attending to it. Suited to tasks that map one sequence to another, such as translation and summarisation. T5 and BART are well-known examples.

Example. A summariser encodes a 1,000-word article and its decoder writes a 60-word summary.

Related: Encoder, Decoder, Cross-attention, T5

A Transformer that uses only the encoder stack with bidirectional attention. It does not generate text but produces rich representations for understanding tasks such as classification, entity tagging, and semantic search. BERT and its relatives are typical examples.

Example. A fine-tuned encoder-only model labels support tickets as billing, delivery or refund.

Related: BERT, Encoder, Masked language modelling, Decoder-only model

One of the parallel feed-forward sub-networks inside a mixture-of-experts layer. Each token is processed by only a few experts. Experts are not usually specialised in neat human topics, though they can develop tendencies for certain kinds of tokens.

Example. In a layer with 64 experts, a given token might be processed by experts 7 and 41 only.

Related: Mixture of experts, Router, Shared expert, Transformer MLP block

A way of spreading a mixture-of-experts model over many GPUs by placing different experts on different devices, with tokens sent across the network to whichever device hosts the experts chosen for them. It makes very large MoE models fit and run but adds communication cost.

Example. With 64 experts on 8 GPUs, each GPU holds 8 experts and receives the tokens routed to them.

Related: Mixture of experts, Expert, Active parameters, Top-k expert routing

A training method in which a document is split into prefix, middle and suffix and reordered so a left-to-right model learns to generate the missing middle given both sides. It lets decoder-only models perform code completion inside existing files without hurting normal generation.

Example. An editor sends the code before and after the cursor, and the model writes the function body between them.

Related: Causal language modelling, Decoder-only model, Autoregressive generation

An exact attention algorithm that reorganises the computation into tiles that fit in fast on-chip GPU memory, avoiding writing the huge attention score matrix to slower memory. The output is the same as standard attention but runs faster and uses much less memory.

Example. Switching an attention implementation to FlashAttention lets a 32,000-token sequence train without running out of GPU memory.

Related: Scaled dot-product attention, Quadratic attention cost, Sparse attention, Context window

Two API settings that reduce repetition by subtracting from a token's logit. The presence penalty applies once if the token has appeared at all; the frequency penalty grows with how many times it has appeared.

Example. A token used 3 times gets logit - 3 * frequency_penalty - presence_penalty.

Related: Repetition penalty, Logit bias, Decoding

Generative Pre-trained Transformer: a family of decoder-only models trained to predict the next token on large text corpora. The name is now often used for the general approach of generative pretraining and for OpenAI's series of models built on it.

Example. A GPT-style model trained on web text can continue a prompt like 'Dear hiring manager,' into a full cover letter.

Related: Decoder-only model, Next-token prediction, Large language model, BERT

A decoding strategy that picks the single most probable token at every step. It is fast and deterministic, but can produce repetitive text and may miss sequences that are better overall because an early choice was only locally best.

Example. If probabilities are 'Delhi' 0.6 and 'Mumbai' 0.3, greedy decoding always outputs 'Delhi'.

Related: Decoding, Beam search, Temperature, Top-k sampling

A compromise between multi-head and multi-query attention: query heads are split into groups and each group shares one key and value head. It keeps most of the quality of full attention while cutting KV cache size, and is common in current open models.

Example. With 32 query heads and 8 key-value heads, each key-value head serves 4 query heads and the cache is 4 times smaller.

Related: Multi-query attention, Multi-head attention, KV cache, Head dimension

The size of the query, key and value vectors in each attention head, usually the model dimension divided by the number of heads. Common values are 64 and 128. It affects the shape of the KV cache and the efficiency of attention kernels.

Example. 4,096 model dimension with 32 heads gives a head dimension of 128.

Related: Multi-head attention, Model dimension, KV cache, Attention head

The vectors that each layer of a Transformer produces for every token position, carrying its evolving representation. The last layer's hidden states feed the output head, and intermediate ones are often extracted as features or inspected for analysis.

Example. A 12-layer BERT gives 13 sets of hidden states (embeddings plus 12 layers), each of size tokens x 768.

Related: Residual stream, Model dimension, Encoder, LM head

A pair of cooperating attention heads found in Transformers that complete repeated patterns: on seeing token A, they look for an earlier occurrence of A and predict the token that followed it. They are thought to underlie part of in-context learning.

Example. In 'Asha Rao ... Asha', an induction head attends to the earlier 'Asha' and predicts 'Rao'.

Related: In-context learning, Attention head, Mechanistic interpretability

A base model that has been further trained on examples of instructions and good responses, and often on human preference data, so it follows requests and holds conversations. Chat assistants are instruct models, typically released alongside the base model.

Example. Llama releases a base checkpoint and an 'Instruct' checkpoint, and the Instruct one answers questions directly.

Related: Base model, Large language model, Foundation model

Memory that stores the key and value vectors of every token already processed at each layer, so that during generation each new token only computes its own keys and values instead of recomputing the whole prompt. It speeds up decoding but its size grows with sequence length, layers and batch.

Example. Without it, generating token 501 would reprocess all 500 earlier tokens; with it, only the new token is computed and attends to cached keys and values.

Related: Autoregressive generation, Prefill and decode, Grouped-query attention, Context window

A Transformer-based language model with billions of parameters, trained on very large text collections. Its scale lets it follow instructions, write code, summarise and reason about text, often with no task-specific training, and it is usually adapted for chat with further tuning.

Example. A chat assistant that drafts a job application in Hindi and English is powered by an LLM.

Related: Foundation model, Decoder-only model, Emergent abilities, In-context learning

A position scheme where each position index has its own trainable vector, looked up like a word embedding and added to the token embedding. It is simple and works well within the trained length, but the model has no vectors for positions beyond the maximum it saw in training.

Example. GPT-2 uses a table of 1,024 learned position vectors, so it cannot process inputs longer than 1,024 tokens.

Related: Positional encoding, Sinusoidal positional encoding, Context window

An adjustment used in beam search that changes how sequence scores depend on length. Because summed log-probabilities always fall as text grows, unadjusted search favours very short outputs; dividing by a power of the length compensates.

Example. score = sum(log p) / length^alpha, with alpha = 0.6, stops a 5-token answer from beating a better 20-token one.

Related: Beam search, Decoding, Log probabilities

A family of approximations that replace softmax attention with kernel-style computations, letting the model reorder operations so cost grows linearly with sequence length. Some can be run like recurrent networks with fixed-size memory, but usually with quality trade-offs.

Example. Rather than an n x n score matrix for 100,000 tokens, a running summary of fixed size is updated for each new token.

Related: Quadratic attention cost, Sparse attention, State space model, RWKV

A family of open-weight Transformer language models released by Meta. They are decoder-only models that use pre-norm with RMSNorm, SwiGLU MLPs and rotary position embeddings, and have become a common base for fine-tuning and for other open models.

Example. A developer fine-tunes a Llama model on Hindi customer-support chats using LoRA.

Related: Open-weight model, Rotary position embedding, SwiGLU, Decoder-only model

The final linear layer of a language model that maps the last hidden vector of each position to one score, called a logit, for every token in the vocabulary. Softmax on these logits gives next-token probabilities. Its weights are sometimes tied to the input embeddings.

Example. A model with vocabulary of 32,000 and d_model 4,096 has an LM head of shape 4,096 x 32,000.

Related: Logits, Softmax, Weight tying, Next-token prediction

An extra training term added to the loss in mixture-of-experts models that pushes the router to spread tokens evenly across experts. Without it, the router tends to favour a few experts, leaving others unused and wasting capacity.

Example. If 90 percent of tokens go to 2 of 8 experts, the auxiliary loss becomes large and nudges the router to distribute more evenly.

Related: Router, Mixture of experts, Capacity factor, Top-k expert routing

The natural logarithms of the probabilities the model assigns to tokens. Many APIs can return them for generated and alternative tokens, and they are used to gauge confidence, rank outputs, compute perplexity and build classifiers on model output.

Example. A token with probability 0.5 has a log probability of about -0.69; one with 0.01 has about -4.6.

Related: Logits, Perplexity, Length penalty, Softmax

An interpretability technique that applies the model's final output layer to intermediate layers' residual stream, to see what token the model would predict if it stopped early. It shows how the prediction is refined across layers.

Example. Decoding layer 12 of a 32-layer model shows 'Paris' as top token for 'The Eiffel Tower is in', sharpening by layer 20.

Related: Mechanistic interpretability, Residual stream, LM head, Logits

A language model designed or extended to accept very long inputs, often hundreds of thousands of tokens, through position-scaling methods, efficient attention kernels and training on long sequences. It lets whole codebases or books be placed in a single prompt.

Example. A long-context model reads a 400-page annual report in one prompt to answer a question about a footnote.

Related: Context window, Position interpolation, FlashAttention, Lost in the middle

An encoder Transformer for long documents that combines sliding-window local attention with a few tokens that attend globally, so cost grows linearly with length. It allows inputs of thousands of tokens where standard BERT is limited to 512.

Example. A legal-tech team classifies 4,000-token contracts with a Longformer instead of splitting them into 512-token pieces.

Related: Sliding window attention, Sparse attention, BERT

A state space model architecture whose parameters depend on the current input, so it can choose what to remember or forget, and which uses a hardware-aware scan. It offers linear scaling with length and competes with Transformers on some language tasks.

Example. A Mamba model generates text with constant memory per step rather than a KV cache that grows with every token.

Related: State space model, Linear attention, KV cache, Transformer

A limit on how many new tokens the model may generate in one response. Generation ends when it is reached, even mid-sentence. It caps cost and latency, and input plus output must fit within the context window.

Example. Setting it to 256 ensures a summary is cut off after at most about 190 words.

Related: Context window, Stop sequence, Decoding

A research approach that tries to reverse-engineer neural networks by finding the internal features and circuits, such as specific attention heads and MLP neurons, that carry out particular behaviours. The aim is to explain how a model computes an answer, not just what it outputs.

Example. Researchers trace how a model finds 'Delhi' after 'The capital of India is' by tracking which heads and MLP layers add it to the residual stream.

Related: Induction head, Residual stream, Logit lens, Attention head

A sampling method that keeps tokens whose probability is at least a fraction of the most likely token's probability, so the cut-off scales with how confident the model is. It aims to stay coherent even at higher temperatures.

Example. If the top token has probability 0.5 and min-p is 0.1, tokens below 0.05 are removed.

Related: Top-p sampling, Top-k sampling, Temperature

A family of open-weight mixture-of-experts language models from Mistral AI, where each layer has several experts and a router selects two of them per token. It demonstrated that a sparse model can match larger dense models at lower per-token compute.

Example. Mixtral 8x7B uses 2 of its 8 experts per token in each layer, so it computes with only a fraction of its total parameters.

Related: Mixture of experts, Open-weight model, Active parameters, Top-k expert routing

A technique in which a router decides, per token and per layer, whether the token goes through the block's attention and MLP or skips it via the residual path. Compute is spent only on tokens that need it, with a fixed budget per layer.

Example. In each layer only the top 12.5 percent of tokens are processed by the block; the rest pass through unchanged.

Related: Mixture of experts, Residual stream, Transformer block

An architecture in which each layer's feed-forward block is replaced by several parallel sub-networks called experts, plus a router that sends each token to only a few of them. The model has a large total parameter count but uses only a fraction per token.

Example. Mixtral 8x7B has 8 experts per layer and uses 2 per token, so far fewer parameters are active than in total.

Related: Router, Expert, Active parameters, Top-k expert routing

The width of the vectors that flow through a Transformer, often written d_model. Token embeddings, attention outputs and the residual stream all have this size. Along with layer count it largely determines a model's parameter count.

Example. GPT-2 small uses d_model = 768; a 7B Llama-style model uses 4,096.

Related: Head dimension, Parameter count, Residual stream, Multi-head attention

Running several attention operations in parallel, each with its own learned query, key and value projections over a smaller dimension, then concatenating their outputs and mixing them with one more linear layer. Different heads can specialise in different relationships, such as syntax or nearby words.

Example. A model with d_model = 768 and 12 heads gives each head 64 dimensions; the 12 outputs are concatenated back to 768.

Related: Attention head, Self-attention, Head dimension, Grouped-query attention

An attention design used in the DeepSeek models that compresses keys and values into a small shared latent vector for caching, and expands them back when needed. It sharply reduces KV cache size while aiming to retain multi-head quality.

Example. Instead of caching full keys and values for every head, only a compact latent per token is stored.

Related: KV cache, Grouped-query attention, Multi-query attention

A variant of multi-head attention in which all query heads share a single set of key and value heads. It shrinks the KV cache and memory traffic dramatically, speeding up generation, with some quality loss compared with full multi-head attention.

Example. With 32 query heads, the KV cache is 32 times smaller than standard multi-head attention because only 1 key-value head is stored.

Related: Grouped-query attention, Multi-head attention, KV cache

A training approach where the model predicts several future tokens at each position rather than only the next one, using extra prediction heads. It can give a richer training signal and enable faster generation by drafting multiple tokens.

Example. At position t, heads predict tokens t+1, t+2 and t+3 at once during training.

Related: Next-token prediction, Causal language modelling, Speculative decoding

A model whose trained parameters are published so anyone can download, run and often fine-tune it on their own hardware. It is not always open source, since training data, code or licence terms may be restricted. Llama and Mistral models are examples.

Example. A Bengaluru startup runs an open-weight model on its own GPUs so customer data never leaves its servers.

Related: Llama, Foundation model, Parameter count

A mask that stops attention from using padding tokens, the filler added to shorter sequences so a batch has equal length. Masked positions get negative-infinity scores so they receive zero weight and do not affect real tokens or the loss.

Example. Batching sentences of length 5 and 8 pads the first with 3 filler tokens; the padding mask marks those 3 positions as ignored.

Related: Causal masking, Sequence length, Self-attention

The total number of learned numbers (weights and biases) in a model, commonly stated in millions or billions, such as 7B. It roughly indicates model capacity and sets memory needs: at 16-bit precision each parameter takes 2 bytes.

Example. A 7-billion-parameter model in 16-bit precision needs about 14 GB just to store its weights.

Related: Dense model, Active parameters, Model dimension, Large language model

A technique to extend a model's usable context by scaling down position indices so that longer inputs fit within the position range seen in training, followed by brief fine-tuning. It avoids the poor behaviour that occurs when rotary positions are pushed past their trained range.

Example. To take a model trained on 4,096 tokens to 16,384, positions are multiplied by 0.25 so they map back into 0-4,096.

Related: Rotary position embedding, Context window, YaRN

Extra information added to or mixed into token representations so the model knows the order of tokens. Attention by itself treats its input as an unordered set, so without position information 'dog bites man' and 'man bites dog' would look the same.

Example. A token embedding of size 768 has a position vector of size 768 added to it before the first Transformer block.

Related: Sinusoidal positional encoding, Learned positional embedding, Rotary position embedding, Relative position encoding

The block layout of the original Transformer, where normalisation is applied after adding the sublayer output to its input. It can give good results but training deep stacks usually needs careful learning-rate warmup, so most large language models moved to pre-norm.

Example. x = LayerNorm(x + Attention(x)); then x = LayerNorm(x + MLP(x)).

Related: Pre-norm, Transformer block, Layer normalisation

A block layout where normalisation is applied to the input of each sublayer, before attention or the MLP, and the sublayer output is added back to the unnormalised residual path. It makes very deep models much more stable to train than the original arrangement and is standard in modern LLMs.

Example. x = x + Attention(LayerNorm(x)); x = x + MLP(LayerNorm(x)).

Related: Post-norm, Residual stream, Transformer block, RMSNorm

The two phases of LLM inference. In prefill, the model processes the entire prompt in parallel and fills the KV cache; in decode, it generates output tokens one at a time. Prefill is compute-heavy, while decode is limited mostly by memory bandwidth.

Example. A 2,000-token prompt is handled in one prefill pass, followed by 300 decode steps to write a 300-token answer.

Related: KV cache, Autoregressive generation, Sequence length

A Transformer setup in which the input prefix, such as an instruction or document, is read with full bidirectional attention while the continuation is generated with causal attention. It sits between encoder-decoder and decoder-only designs and uses a single stack of layers.

Example. In a prefix LM, the 300-token question is attended to in both directions, and the answer tokens are then generated left to right.

Related: Decoder-only model, Bidirectional attention, Causal masking, Encoder-decoder model

Applying normalisation to the query and key vectors before their dot product in attention. It keeps attention scores from growing too large, which stabilises training of big models and helps avoid loss spikes caused by saturated softmax.

Example. Each head's query and key vectors are normalised to controlled scale before computing Q K^T.

Related: Scaled dot-product attention, Query, key and value, Pre-norm, Layer normalisation

The fact that standard self-attention compares every token with every other, so compute and memory for the score matrix grow with the square of sequence length. Doubling the input length roughly quadruples attention cost, which is the main obstacle to very long contexts.

Example. A 1,000-token input produces 1 million scores per head; a 10,000-token input produces 100 million.

Related: Sequence length, FlashAttention, Sparse attention, Linear attention

The three learned projections of each token used in attention. The query says what a token is looking for, the key advertises what a token contains, and the value is the information actually passed on. A query is compared to all keys, and the resulting weights mix the values.

Example. For token 'it', the query vector matches strongly with the key of 'animal' and the output is mostly the value vector of 'animal'.

Related: Self-attention, Scaled dot-product attention, Attention head

An efficient Transformer that uses locality-sensitive hashing to group similar queries and keys so attention is computed only within buckets, and reversible layers to save memory. It reduces long-sequence cost from quadratic towards near-linear.

Example. Tokens whose query vectors hash to the same bucket attend to each other only, instead of to all 64,000 tokens.

Related: Sparse attention, Quadratic attention cost, Longformer

A family of position schemes in which attention depends on the distance between two tokens rather than their absolute indices. This generalises better to different lengths and captures patterns like 'the word two positions back'. RoPE, ALiBi and T5's bias buckets are examples.

Example. Whether a verb is 2 tokens after its subject matters the same way at positions 5 and 500.

Related: Rotary position embedding, ALiBi, Positional encoding, T5

The main vector for each token that flows through the layers of a Transformer, to which every attention and MLP sublayer adds its output. Viewing the model this way, layers read from and write to a shared channel, which is a common framing in interpretability work.

Example. Layer 5's attention adds a small vector to the residual stream, which layer 6 then reads.

Related: Skip connection, Transformer block, Mechanistic interpretability, Pre-norm

A method for training and running attention over very long sequences by splitting the sequence across many devices and passing blocks of keys and values around a ring of devices, overlapping communication with computation. Memory per device stays small while context grows with device count.

Example. A 1-million-token sequence is split over 64 GPUs, each holding about 16,000 tokens and receiving key-value blocks from its neighbour.

Related: FlashAttention, Quadratic attention cost, Long-context model, Context window

A robustly optimised retraining of BERT that trains longer on more data with larger batches, drops the next-sentence prediction task and masks tokens dynamically, changing the hidden words each time a sequence is seen. It improved results without changing the architecture.

Example. A team swaps a BERT checkpoint for RoBERTa in its ticket-classification pipeline and sees a small accuracy gain with no code changes.

Related: BERT, Encoder-only model, Bidirectional attention

A method that encodes position by rotating the query and key vectors by an angle proportional to their position, in pairs of dimensions. The dot product between a query and a key then depends on their relative distance. It is widely used in open models such as Llama.

Example. A query at position 10 and a key at position 7 are rotated by different angles, and their dot product reflects the distance of 3.

Related: Positional encoding, Relative position encoding, Position interpolation, Llama

A language model architecture that can be trained in parallel like a Transformer but run like a recurrent network at inference, with a fixed-size state instead of a growing cache. It aims to keep quality while making long generation cheap.

Example. An RWKV model running on a small device keeps one fixed-size state no matter how long the conversation grows.

Related: Recurrent neural network, Linear attention, State space model, KV cache

The core attention formula of the Transformer: multiply queries by transposed keys to get similarity scores, divide by the square root of the key dimension, apply softmax to get weights, then multiply by values. The scaling keeps scores from growing large and making softmax saturate.

Example. attention(Q,K,V) = softmax(Q K^T / sqrt(d_k)) V. With d_k = 64 the scores are divided by 8 before softmax.

Related: Query, key and value, Attention weights, Multi-head attention, Softmax

Attention where the queries, keys and values all come from the same sequence, so each token gathers information from the other tokens around it. The result is a new representation for each token that reflects its context, such as which noun a pronoun refers to.

Example. In 'The bank raised its rates', self-attention lets 'its' pull information from 'bank', helping the model treat 'bank' as a financial institution.

Related: Attention mechanism, Multi-head attention, Query, key and value, Cross-attention

The number of tokens in a single input the model processes at once, including the prompt and, when generating, the output so far. Compute and memory for attention grow with it, and each model has a maximum it can handle.

Example. A 3-page document of about 1,500 words is roughly 2,000 tokens, so the sequence length is about 2,000.

Related: Context window, Padding mask, Quadratic attention cost

A model that reads one sequence and produces another sequence whose length may differ. Early versions used a recurrent encoder and decoder; modern ones use Transformers. Typical uses are translation, summarisation and speech recognition.

Example. A seq2seq model converts a 12-word English sentence into a 10-word Tamil sentence.

Related: Encoder-decoder model, Additive attention, Cross-attention, T5

An expert in a mixture-of-experts layer that every token always passes through, alongside the routed experts. It captures common knowledge so that the routed experts can specialise. DeepSeek's MoE models use this idea.

Example. A layer has 1 shared expert plus 64 routed experts, of which 6 are chosen per token.

Related: Expert, Mixture of experts, Router

A fixed, non-learned position scheme from the original Transformer that uses sine and cosine waves of different frequencies at each embedding dimension. Each position gets a unique pattern, and offsets between positions correspond to simple transformations of these patterns.

Example. PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)).

Related: Positional encoding, Learned positional embedding, Rotary position embedding

A pattern in which each token attends only to a fixed number of most recent tokens instead of the whole sequence. Cost grows linearly with length, and information can still travel further because stacked layers extend the effective reach.

Example. With a window of 4,096, token 10,000 attends to tokens 5,905 through 10,000 only.

Related: Sparse attention, Longformer, Quadratic attention cost, KV cache

Attention in which each token attends to only a chosen subset of positions, such as a local window, a few global tokens or a strided pattern, instead of all of them. It cuts the cost of long sequences at the price of some flexibility.

Example. A pattern where each token sees its 512 neighbours plus 4 global tokens replaces a full 16,000 x 16,000 score matrix.

Related: Sliding window attention, Longformer, Quadratic attention cost, Linear attention

A small neural network trained to reconstruct a language model's internal activations through a wide layer where only a few units are active at once. The active units often correspond to human-interpretable features, making it a main tool for interpretability research.

Example. An autoencoder trained on layer-20 activations finds a feature that fires on Indian currency amounts such as rupee prices.

Related: Mechanistic interpretability, Residual stream, Logit lens, Activation patching

A sequence model that keeps a fixed-size hidden state updated at each step by learned linear dynamics, and can be computed either recurrently or as a convolution. It handles long sequences with linear cost, making it an alternative to attention.

Example. An S4-style layer processes a 100,000-step signal with a state of a few hundred numbers instead of comparing every pair of steps.

Related: Mamba, Linear attention, RWKV, Recurrent neural network

A mixture-of-experts Transformer from Google that simplifies routing by sending each token to just one expert (top-1). It showed that sparse models with very many parameters can train efficiently and introduced a capacity limit and an auxiliary balancing loss.

Example. In a Switch layer with 128 experts, each token is handled by exactly one expert feed-forward network.

Related: Mixture of experts, Top-k expert routing, Capacity factor, Load-balancing loss

Text-to-Text Transfer Transformer: an encoder-decoder model that casts every task as text in, text out, using a prefix to say what to do. The same model, loss and decoding process handle translation, summarisation and classification.

Example. Input 'summarize: <article>' produces a short summary, while 'translate English to German: Hello' produces 'Hallo'.

Related: Encoder-decoder model, Sequence-to-sequence model, Relative position encoding

A setting that divides the logits by a number before softmax, controlling how spread out the next-token probabilities are. Below 1 makes the model more confident and predictable, above 1 flattens the distribution and adds variety, and near 0 approaches greedy decoding.

Example. At temperature 0.2 the top token might get 95 percent probability; at 1.5 the same token might get 40 percent.

Related: Softmax, Top-p sampling, Top-k sampling, Decoding

The computation spent while a model answers, rather than during training. It can be raised by generating longer reasoning, sampling multiple answers and selecting the best, or searching. More of it can lift accuracy on hard problems without a bigger model.

Example. Sampling 16 solutions to a maths question and taking the majority answer uses 16 times the inference compute of a single answer.

Related: Reasoning model, Best-of-n sampling, Decoding

A routing rule in which, for each token, only the k experts with the highest router scores are used and their outputs are combined with the normalised scores as weights. Common choices are k = 1 or 2 to keep compute low.

Example. With k = 2, a token's output is 0.6 * expert_3(x) + 0.4 * expert_5(x).

Related: Router, Mixture of experts, Switch Transformer, Capacity factor

A sampling method that keeps only the k most probable next tokens, renormalises their probabilities, and picks one at random. It removes the unlikely long tail that can cause nonsense, but a fixed k is too loose when the model is sure and too tight when it is not.

Example. With k = 40, only the 40 highest-probability tokens are candidates; all others are excluded.

Related: Top-p sampling, Temperature, Min-p sampling, Decoding

A sampling method that sorts tokens by probability and keeps the smallest set whose probabilities add up to p, then samples from that set. The number of candidates adapts: few when the model is confident, more when it is uncertain.

Example. With p = 0.9, if 'Delhi' has 0.7 and 'Mumbai' 0.25, only those two are kept because 0.95 exceeds 0.9.

Related: Top-k sampling, Temperature, Min-p sampling, Decoding

A neural network design that processes a whole sequence at once by letting every position look at every other position through attention, instead of reading step by step like a recurrent network. Stacked identical blocks make it easy to train in parallel on GPUs, which is why it underlies almost all modern language models.

Example. GPT, BERT and Llama are all Transformers; a 12-block model reads a 500-token paragraph in one parallel pass rather than 500 sequential steps.

Related: Attention mechanism, Self-attention, Transformer block, Recurrent neural network

The repeating unit of a Transformer: an attention sublayer followed by a position-wise feed-forward sublayer, each wrapped with normalisation and a residual connection. The model is built by stacking many identical blocks, with each block refining the token representations.

Example. A 7B-parameter Llama-style model stacks 32 such blocks, each with attention followed by an MLP.

Related: Multi-head attention, Transformer MLP block, Residual stream, Pre-norm

The feed-forward sublayer in each Transformer block, applied independently to every token: a linear layer that expands the dimension (often 4x), a nonlinearity, and a linear layer that projects back. It holds most of a block's parameters and is thought to store much factual knowledge.

Example. With d_model = 768, the MLP expands to 3,072 units and projects back to 768.

Related: Transformer block, SwiGLU, Gated linear unit, Model dimension

A Transformer variant that processes long text in segments while letting each segment attend to cached hidden states from the previous one, and uses relative positions to make this consistent. It extends the effective context beyond a single fixed segment.

Example. A 512-token segment attends to its own tokens plus the cached states of the prior 512-token segment.

Related: Relative position encoding, Context window, Transformer, KV cache

A sampling method that keeps tokens whose surprise, the negative log-probability, is close to the expected surprise of the distribution, instead of just the most probable ones. It aims for text that reads like natural human writing, avoiding both dull and bizarre tokens.

Example. Very probable filler tokens and very improbable oddities are both filtered, leaving tokens with typical information content.

Related: Top-p sampling, Top-k sampling, Temperature, Log probabilities

A Transformer variant that applies the same block repeatedly, sharing its weights across depth, and can adaptively decide how many times to apply it to each position. It combines parallel attention with recurrence in depth.

Example. Instead of 6 different blocks, a Universal Transformer applies one block up to 6 times, stopping earlier for easy tokens.

Related: Transformer block, Mixture of depths, Transformer

A method for extending the context window of models using rotary position embeddings. It rescales different frequency components by different amounts and adjusts attention temperature, so a model can handle much longer inputs with relatively little extra training compared with plain position interpolation.

Example. A team applies YaRN to an open model trained on 8K tokens and fine-tunes briefly to serve 64K-token contracts.

Related: Position interpolation, Rotary position embedding, Context window