Llm training alignment

Combining data, tensor and pipeline parallelism at once to train models too big for any single strategy. Each axis maps to a different network layer of the cluster, with tensor parallelism inside fast nodes.

Example. A 512-GPU job might use tensor parallel 8, pipeline parallel 4 and data parallel 16, since 8 x 4 x 16 = 512.

Related: Tensor parallelism, Pipeline parallelism, Data parallelism

An optimiser such as Adam that stores its running statistics in 8-bit precision instead of 32-bit, cutting the memory used for optimiser state by roughly three-quarters with little loss in final quality.

Example. Adam keeps two extra values per parameter; storing them in 8 bits saves gigabytes on a multi-billion-parameter model.

Related: AdamW, Paged optimisers, Mixed precision training

A small trainable module inserted into a frozen pretrained network, typically a down-projection, non-linearity and up-projection with a residual connection. Only adapters are trained and stored per task, and they are swapped in as needed.

Example. A single base model keeps one adapter file each for Hindi support and Tamil support, each only a few megabytes.

Related: Parameter-efficient fine-tuning, LoRA, Prefix tuning

A measure of how much better an action is than the average action in that situation, computed as expected return of the action minus the value of the state. Subtracting this baseline lowers the variance of policy-gradient updates, so learning is steadier.

Example. If an answer earns reward 0.9 and the model would normally earn 0.6 for that prompt, the advantage is +0.3, so the answer is reinforced.

Related: Value function, Policy gradient, Generalised advantage estimation, Proximal policy optimisation

The effort to make an AI system pursue the goals and follow the values its designers and users actually intend. For language models this covers following instructions, being truthful and refusing harmful requests, using methods such as fine-tuning on human feedback.

Example. Two models can be equally capable, but the aligned one declines to give instructions for fraud and admits when it does not know an answer.

Related: RLHF, Helpful, honest and harmless, Constitutional AI, Post-training

The loss in performance on some capabilities that can occur after alignment training, such as lower benchmark scores after RLHF. Techniques like mixing in pretraining data or careful regularisation aim to reduce it.

Example. After RLHF, a model becomes more polite but its score on a reading-comprehension benchmark falls by a few points.

Related: RLHF, Catastrophic forgetting, KL penalty

An early open instruction-tuned model and dataset from Stanford, produced by fine-tuning a LLaMA model on about 52,000 instruction-following examples generated by prompting another model. It popularised the recipe of fine-tuning on model-generated instructions.

Example. Alpaca-style datasets pair an instruction, optional input and output in a simple JSON record.

Related: Self-instruct, Synthetic data, Instruction tuning

A language model that has finished pretraining but has not been trained to follow instructions or chat. It simply continues text in the style it has seen, so a question may be answered with more questions. It is the starting point for instruction tuning, alignment and domain fine-tuning.

Example. Prompt a base model with 'What is GST?' and it may continue with 'What is VAT? What is income tax?' instead of answering.

Related: Instruction tuning, Pretraining, Supervised fine-tuning, Foundation model

A statistical model for pairwise comparisons that says the chance item i beats item j is sigmoid(score_i - score_j). Reward models in RLHF are typically trained with this model: the loss is -log sigmoid(r_chosen - r_rejected), which pushes the chosen response's score above the rejected one's.

Example. If the chosen score is 1.5 and the rejected score is 0.5, the modelled win probability is sigmoid(1.0), about 0.73.

Related: Reward model, Preference data, Direct preference optimisation, Sigmoid function

A training objective for decoder-style models in which each position may only attend to earlier positions, so the model learns to predict the next token from the left context alone. A causal mask inside attention enforces this rule. It is the objective behind GPT-style and most modern chat models.

Example. For the sentence 'Ram ate rice', the model predicts 'ate' from 'Ram' and 'rice' from 'Ram ate'; the mask stops position 2 from seeing position 3 during training.

Related: Next-token prediction, Masked language modelling, Pretraining

The exact text format that wraps system, user and assistant messages with special role tokens before they are fed to a chat model. Training and inference must use the same template, otherwise the model sees an unfamiliar format and quality drops.

Example. A template may render a conversation as '<|user|> Hi <|end|> <|assistant|>', and the model continues after the assistant marker.

Related: Instruction tuning, Special tokens, Supervised fine-tuning, Loss masking

The finding from DeepMind's Chinchilla work that, for a fixed compute budget, model size and training tokens should grow together, roughly twenty tokens per parameter, rather than making models larger on limited data. Earlier large models were undertrained by this measure.

Example. By this rule a 7B-parameter model is compute-optimal at about 140 billion tokens.

Related: Scaling laws, Compute-optimal training, Overtrained model

The two answers stored in a preference example: the chosen (preferred) response and the rejected (dispreferred) one. Preference training pushes the model to make the chosen answer more likely than the rejected answer for the same prompt.

Example. For the prompt 'Is it safe to share my OTP?', the chosen response says never share it and the rejected one says it is fine with a bank employee.

Related: Preference data, Direct preference optimisation, Reward model

The PPO loss that multiplies the advantage by the probability ratio between new and old policies, then clips that ratio to the range 1 - epsilon to 1 + epsilon and takes the smaller value. This removes any incentive for a single update to change the policy too much.

Example. With epsilon = 0.2, a ratio of 1.4 on a positive-advantage token is treated as 1.2, so the gradient from that token vanishes.

Related: Proximal policy optimisation, Advantage function, Policy gradient

A non-profit's public archive of web pages gathered by crawling the internet regularly. It is a major raw source for pretraining text, but needs heavy filtering for quality, duplicates and unsafe content.

Example. A pipeline extracts text from Common Crawl snapshots, filters by language, and removes near-duplicates.

Related: Data curation, Deduplication, Pretraining

Choosing model size and training data amount to reach the lowest loss for a given compute budget, using fitted scaling laws. It optimises training cost only and ignores inference cost, which is why many production models are trained on more data than this rule suggests.

Example. With a fixed GPU budget, a compute-optimal plan may pick a 13B model on 260B tokens over a 40B model on 30B tokens.

Related: Chinchilla scaling, Scaling laws, Overtrained model

An alignment approach from Anthropic in which a model critiques and revises its own outputs according to a written list of principles, and preferences generated by AI under those principles train the final model. It reduces reliance on human labels for harmlessness.

Example. A model drafts an unsafe reply, critiques it against the principle 'avoid helping harm', rewrites it politely refusing, and the revised answers become training data.

Related: Reinforcement learning from AI feedback, Alignment, RLHF

Further training of an existing pretrained model on new unlabelled text with the same next-token objective, typically to add a domain or language. It differs from fine-tuning in using raw text rather than labelled examples.

Example. Continuing pretraining a general model on 20 billion tokens of Indian legal text before instruction tuning.

Related: Domain-adaptive pretraining, Pretraining, Catastrophic forgetting

The batch size beyond which making batches larger yields sharply diminishing gains in learning per example. Below it, doubling the batch nearly halves the steps needed; above it, extra compute is largely wasted.

Example. If the critical size is about 4 million tokens, using 16 million tokens per batch costs more compute for little speed-up.

Related: Batch size, Scaling laws, Linear scaling rule

The process of selecting, cleaning, filtering and balancing training data to raise its quality, using heuristics, classifiers and human review. For language models it often matters as much as model size.

Example. Removing boilerplate pages and low-quality forum text from a crawl before pretraining.

Related: Data mixture, Deduplication, Common Crawl

The proportions of different sources, such as web text, code, books and multilingual text, sampled during pretraining. Changing the mixture shifts what a model is good at, so it is tuned with small experiments.

Example. A mixture of 70 percent web, 15 percent code, 10 percent books and 5 percent Indian-language text.

Related: Data curation, Pretraining, Continued pretraining

A proposed oversight method in which two AI systems argue opposite sides of a question and a human judge decides who is more convincing, on the idea that it is easier to judge arguments than to solve hard problems directly.

Example. Two models argue whether a code change is safe; each points to lines that support its claim, and the judge picks a winner.

Related: Scalable oversight, Alignment, Weak-to-strong generalisation

An open-source library from Microsoft for large-scale deep-learning training and inference. It provides ZeRO memory partitioning, CPU offloading, mixed precision and parallelism helpers so very large models fit on limited GPU clusters.

Example. Enabling a DeepSpeed ZeRO config lets a 13B model fine-tune on GPUs that could not hold it under plain data parallelism.

Related: ZeRO, Fully sharded data parallel, Megatron-LM

An alignment method that trains a model directly on chosen-versus-rejected pairs with a simple classification-style loss, removing the separate reward model and the reinforcement learning loop. It raises the relative likelihood of chosen answers compared with a frozen reference model, and is cheaper and steadier than PPO.

Example. loss = -log sigmoid(beta * [(log pi(chosen) - log ref(chosen)) - (log pi(rejected) - log ref(rejected))]), with beta commonly 0.1.

Related: Preference data, Reference model, RLHF, Identity preference optimisation

Continuing unsupervised training on text from one specific field, such as medicine, finance or law, so the model absorbs its vocabulary and facts before it is fine-tuned on a labelled task in that field.

Example. A base model is further pretrained on clinical notes and then fine-tuned to extract diagnoses.

Related: Continued pretraining, Transfer learning, Fine-tuning

Weight-Decomposed Low-Rank Adaptation, a LoRA variant that splits a pretrained weight into a magnitude and a direction, applies the low-rank update to the direction and trains the magnitude separately, often improving quality at similar cost.

Example. Swapping LoRA for DoRA in a fine-tuning config aims at accuracy closer to full fine-tuning with a similar number of trainable parameters.

Related: LoRA, Parameter-efficient fine-tuning, Rank (LoRA)

A QLoRA memory-saving step that quantises the quantisation constants themselves, so the per-block scaling factors also use fewer bits. It saves a fraction of a bit per parameter, which adds up for very large models.

Example. Instead of storing a 32-bit scale for each block of 64 weights, the scales are stored in 8 bits with a second-level constant.

Related: QLoRA, NormalFloat 4-bit, Quantisation

Abilities that seem absent in smaller models and appear at larger scale. Whether they are truly sudden or an artefact of the metric used is debated, since smoother metrics often show gradual improvement.

Example. Multi-digit arithmetic accuracy stays near zero for small models and then rises in larger ones on an exact-match metric.

Related: Scaling laws, Pretraining, In-context learning

A method that uses a language model to rewrite instructions into progressively harder or more complex versions, for instance by adding constraints or requiring deeper reasoning, producing a more varied and difficult instruction dataset.

Example. 'Write a function to add two numbers' evolves into 'Write a function that adds two numbers given as strings, handling negative values'.

Related: Self-instruct, Synthetic data, Instruction tuning

A mismatch in models trained with teacher forcing: during training they always see correct previous tokens, but at generation time they see their own possibly wrong outputs, so errors can compound.

Example. A translation model trained on perfect prefixes drifts off after one wrong word at inference.

Related: Teacher forcing, Next-token prediction, Sequence-level distillation

Finetuned Language Net, a line of work that instruction-tunes models on a large collection of existing NLP tasks rewritten as natural-language instructions. It showed that training on many tasks improves zero-shot performance on unseen tasks.

Example. A sentiment dataset becomes 'Is this review positive or negative? [review]' with the label as the answer.

Related: Instruction tuning, Zero-shot learning, Supervised fine-tuning

Floating-point operations, a count of arithmetic operations used to measure compute. Training cost is often quoted as total FLOPs; a common estimate for a transformer is about 6 x parameters x tokens.

Example. Training a 7-billion-parameter model on 1 trillion tokens costs roughly 6 x 7e9 x 1e12 = 4.2e22 FLOPs.

Related: Scaling laws, Model FLOPs utilisation, Compute-optimal training

A large model trained on broad data at scale so that it can be adapted to many downstream tasks, rather than built for one task. Language models, image models and multimodal models can all be foundation models. Their value is that one expensive training run supports many products through prompting or fine-tuning.

Example. A single pretrained model can be prompted to summarise Hindi news, or fine-tuned to answer questions on bank policy documents.

Related: Pretraining, Base model, Fine-tuning, Transfer learning

Training with 8-bit floating-point numbers for some matrix operations, which roughly doubles arithmetic throughput and halves memory versus 16-bit on supporting GPUs, while keeping sensitive values such as master weights in higher precision.

Example. Matrix multiplications run in FP8 while weights are kept in higher precision to preserve accuracy.

Related: Mixed precision training, Bfloat16, Loss scaling

Updating all of a pretrained model's weights on new data. It can give the best task fit but needs memory for weights, gradients and optimiser state, and produces a full-size model copy for every task.

Example. Fully fine-tuning a 7B model in 16-bit needs multiple GPUs' worth of memory, while a LoRA run may fit on one.

Related: Parameter-efficient fine-tuning, LoRA, Catastrophic forgetting

A PyTorch training strategy that shards model parameters, gradients and optimiser state across GPUs and gathers each layer's weights just in time for its computation, allowing much larger models than plain data parallelism.

Example. On 8 GPUs, each holds one-eighth of the weights and temporarily gathers a layer's full weights during its forward pass.

Related: ZeRO, Data parallelism, DeepSpeed

A technique that computes advantages as a weighted sum of temporal-difference errors, with a parameter lambda that trades bias against variance. Setting lambda near 0 relies on the value estimate, while lambda near 1 relies on the observed returns. It is commonly used with PPO.

Example. With lambda = 0.95 and gamma = 1, the advantage at each token blends the next few steps' value errors instead of just one step or the whole episode.

Related: Advantage function, Value function, Proximal policy optimisation

The principle that when a measure becomes a target, it stops being a good measure. In alignment, pushing hard on a proxy such as a reward-model score eventually improves the proxy while true quality gets worse, which is why over-optimisation needs limits.

Example. Optimising a chatbot for a reward-model score keeps raising the score, but past a point human raters find the answers worse.

Related: Reward hacking, Reward model, KL penalty

A reinforcement learning method that samples a group of answers for each prompt and computes each answer's advantage relative to the group's mean and spread, so no separate value network is needed. It was popularised for training reasoning models with verifiable rewards.

Example. For one maths question, 8 answers are sampled; those scoring above the group's average reward get positive advantage and are reinforced.

Related: Proximal policy optimisation, Reinforcement learning with verifiable rewards, Advantage function

A commonly cited three-part target for assistant behaviour: give useful answers, tell the truth and admit uncertainty, and avoid causing harm. The goals can conflict, for example when a helpful answer is risky, so alignment work balances them.

Example. An assistant asked for medicine dosage gives general information, notes uncertainty and advises seeing a doctor for a specific case.

Related: Alignment, RLHF, Sycophancy

An open-source library, short for Transformer Reinforcement Learning, providing ready-made trainers for supervised fine-tuning, reward modelling, PPO, DPO and related alignment methods, built on top of the Hugging Face Transformers library.

Example. A practitioner runs an SFT trainer and then a DPO trainer from the same library on a chat dataset.

Related: Supervised fine-tuning, Direct preference optimisation, PEFT library

A variant of direct preference optimisation that replaces the sigmoid-based loss with a squared-error loss on the log-ratio margin, designed to reduce overfitting when preferences are nearly deterministic. It regularises the gap between chosen and rejected log-ratios toward a fixed target.

Example. When nearly every pair in a dataset has an obvious winner, plain DPO can push the margin to extremes, while IPO holds it near a set value.

Related: Direct preference optimisation, Preference data, Reference model

A 2022 OpenAI research model and paper that fine-tuned GPT-3 with supervised demonstrations and then reinforcement learning from human feedback, showing that a smaller aligned model could be preferred by people over a much larger unaligned one. It established the standard three-stage recipe of SFT, reward model and RL.

Example. The recipe: collect human-written demonstrations, train a reward model on ranked outputs, then optimise the model against that reward model with PPO.

Related: RLHF, Supervised fine-tuning, Reward model, Proximal policy optimisation

A method that takes unlabelled text such as web documents, has a model guess what instruction would have produced each one, and then keeps the best pairs as training data. It builds instruction datasets from existing writing.

Example. A blog post on Diwali recipes gets the generated instruction 'Give me a Diwali sweets recipe', forming a training pair.

Related: Synthetic data, Instruction tuning, Self-instruct

Fine-tuning a base model on many examples of instructions paired with good responses, across varied task types, so it learns to follow directions given in plain language. It turns a text-continuer into an assistant-like model and helps it generalise to instructions it has never seen.

Example. A dataset of 50,000 pairs such as 'Translate this to Tamil: ...' with the correct translation, 'Summarise this email' with a summary, and so on.

Related: Supervised fine-tuning, FLAN, Base model, Chat template

An alignment method that uses only a binary signal per response, 'desirable' or 'undesirable', instead of paired comparisons. Its loss is inspired by prospect theory from behavioural economics, treating gains and losses asymmetrically. It suits data where thumbs-up and thumbs-down are easier to collect than rankings.

Example. App feedback with thumbs-up on 40,000 replies and thumbs-down on 15,000 others can be used directly, without matching them into pairs.

Related: Direct preference optimisation, Preference data, RLHF

A term added to the RLHF reward that subtracts a multiple of the KL divergence between the tuned model and a frozen reference model. It stops the policy from drifting into odd text that games the reward model, and preserves fluency and knowledge from earlier training.

Example. reward_total = reward_model_score - beta * KL(policy || reference), with beta = 0.05 penalising answers whose token probabilities depart far from the reference.

Related: Reference model, KL divergence, RLHF, Reward hacking

Training a smaller student model to imitate a larger teacher model, using the teacher's output probabilities or generated text as targets. The student keeps much of the teacher's ability at lower serving cost and latency.

Example. A 70-billion-parameter teacher's answers train a 7-billion-parameter student that runs on a single GPU.

Related: Teacher-student learning, Soft labels, Synthetic data, Quantisation

'Less Is More for Alignment', a study that fine-tuned a large pretrained model on roughly one thousand carefully curated prompt-response pairs, without reinforcement learning, and found it performed competitively. It supports the view that data quality matters more than quantity for instruction tuning.

Example. A 1,000-example curated set can rival one with 50,000 noisy examples for teaching an assistant style.

Related: Superficial alignment hypothesis, Supervised fine-tuning, Instruction tuning

Extending a model to handle longer inputs by continuing training on long sequences, often with adjusted positional-encoding settings, after pretraining on shorter ones. It is cheaper than pretraining at full length.

Example. A model pretrained on 4,096-token windows is further trained on 32,768-token documents.

Related: Continued pretraining, Sequence packing, Pretraining

Low-Rank Adaptation, a fine-tuning technique that freezes the original weight matrix W and learns a small update written as the product of two thin matrices B and A, so the effective weight is W + (alpha/r) * B*A. It trains far fewer parameters and the update can be merged into W afterwards.

Example. For a 4096x4096 matrix, rank 8 needs 2 x 4096 x 8 = 65,536 trainable values instead of about 16.8 million.

Related: Rank (LoRA), QLoRA, Adapter, Parameter-efficient fine-tuning

Excluding chosen tokens from the training loss so that gradients come only from the parts you want the model to learn. In supervised fine-tuning the prompt tokens are usually masked, so the model is scored only on the assistant's reply, not on reproducing the user's question.

Example. For a 30-token question and a 50-token answer, only the 50 answer tokens contribute to the loss; the 30 question tokens are given label -100 in common libraries.

Related: Supervised fine-tuning, Cross-entropy loss, Chat template, Sequence packing

A sudden jump in training loss during large-model training, often caused by unstable optimisation, bad data batches or numerical problems. Teams mitigate by lowering learning rate, clipping gradients, skipping batches or resuming from an earlier checkpoint.

Example. At step 40,000 the loss jumps from 2.1 to 6.5; the team restarts from the step-39,000 checkpoint and skips that data.

Related: Gradient clipping, Checkpoint, Learning rate warmup

A training objective where some tokens in the input are hidden and the model must recover them using context from both the left and the right. It suits encoder models that build rich representations of a sentence, and was popularised by BERT. Unlike next-token prediction, it does not generate text left to right.

Example. From 'UPI payments are [MASK] in India', a model fills the blank with 'popular' by using words on both sides.

Related: Causal language modelling, Span corruption, Pretraining

A way of scaling initialisation and learning rates with model width so that good hyperparameters found on a small model transfer to a much larger one, cutting the cost of tuning big runs.

Example. Tune the learning rate on a small width-256 model, then reuse the same value at width 4096 under this parametrisation.

Related: Hyperparameter tuning, Scaling laws, Learning rate

NVIDIA's open-source framework for training very large transformer models efficiently across many GPUs, using tensor, pipeline and sequence parallelism, which has influenced many later training libraries and tools. Its techniques have influenced many later large-model training libraries and tools.

Example. A team uses Megatron-LM to split a large transformer across many GPU nodes with tensor and pipeline parallelism.

Related: Tensor parallelism, Pipeline parallelism, DeepSpeed

The degradation that occurs when models are trained repeatedly on data produced by earlier models, so rare patterns in the original data vanish and outputs become more uniform, repetitive and less accurate with each generation.

Example. After several rounds of training on its own generated text, a model loses uncommon words and unusual facts.

Related: Synthetic data, Data curation, Pretraining

The share of a hardware system's peak theoretical FLOPs that a training job turns into useful model computation, excluding recomputation overhead. Values of 30-50 percent are considered good for large runs; low values signal communication or data bottlenecks.

Example. A cluster with 1,000 units of peak capacity delivering 400 units of model math has an MFU of 40 percent.

Related: FLOPs, Data parallelism, Pipeline parallelism

Combining the weights of several fine-tuned models that share a base into one model, for example by averaging or by adding weight differences, without extra training. It can blend skills cheaply, though results depend on how compatible the models are.

Example. Averaging a coding-tuned and a maths-tuned version of the same base yields one model with both abilities.

Related: Task arithmetic, Full fine-tuning, LoRA

The training task in which a model reads a sequence of tokens and outputs a probability for every possible next token, and is penalised by cross-entropy when the true next token gets a low probability. Doing this over enormous text teaches the model language structure and world knowledge without any human labels.

Example. Given the tokens 'The capital of India is', the model should put high probability on ' New' and then ' Delhi'; a low probability on 'banana' produces a large loss.

Related: Causal language modelling, Pretraining, Cross-entropy loss

A 4-bit number format introduced with QLoRA whose levels are placed at quantiles of a normal distribution, matching how neural network weights are typically distributed. It represents weights more accurately than evenly spaced 4-bit integers.

Example. Sixteen NF4 levels are denser near zero, where most weights lie, and sparser at the extremes.

Related: QLoRA, Quantisation, Double quantisation

A method that merges supervised fine-tuning and preference alignment into one stage by adding an odds-ratio penalty that favours chosen over rejected responses to the ordinary language-modelling loss. It needs no reference model, saving memory.

Example. Instead of running SFT and then DPO, one training run on chosen-and-rejected pairs applies both losses together.

Related: Direct preference optimisation, Supervised fine-tuning, Reference model

A model trained on far more tokens than compute-optimal scaling suggests, accepting higher training cost to obtain a smaller model that is cheaper to serve at similar quality. It is a common choice when many users will run the model.

Example. A 8B-parameter model trained on 15 trillion tokens is about 1,900 tokens per parameter, far above the compute-optimal ratio.

Related: Chinchilla scaling, Compute-optimal training, Scaling laws

A memory technique that stores optimiser state in pageable memory so it can move between GPU and CPU RAM when GPU memory spikes, preventing out-of-memory crashes during training with long sequences.

Example. A QLoRA run with occasional long batches keeps training instead of crashing because Adam's state pages to CPU memory.

Related: QLoRA, 8-bit optimiser, Gradient checkpointing

A group of methods that adapt a large pretrained model by training only a small number of added or selected parameters while the rest stay frozen. They cut memory, storage and compute, and let one base model serve many tasks with small swappable add-ons.

Example. Instead of updating 7 billion weights, train about 20 million adapter weights for a legal-document task.

Related: LoRA, Adapter, Prefix tuning, Full fine-tuning

Hugging Face's open-source library implementing parameter-efficient fine-tuning methods such as LoRA, prefix tuning and prompt tuning. It attaches small trainable adapters to models loaded from the Transformers library and saves them separately.

Example. Wrapping a model with a LoRA config in PEFT makes only the adapter weights trainable.

Related: LoRA, Parameter-efficient fine-tuning, Hugging Face TRL

Placing consecutive groups of layers on different GPUs and passing micro-batches through them like an assembly line. It reduces per-GPU memory, but idle time, called pipeline bubbles, must be managed by using many micro-batches.

Example. Layers 1-16 sit on GPU 0 and layers 17-32 on GPU 1, and micro-batches flow through GPU 0 then GPU 1.

Related: Tensor parallelism, Data parallelism, Model parallelism

The rule an agent uses to choose actions given a state. In language-model reinforcement learning, the policy is the model itself: the state is the prompt plus text generated so far, and the action is the next token, sampled from the model's probability distribution.

Example. When a chat model is trained with RLHF, the model being updated is called the policy, while a frozen copy is the reference model.

Related: Policy gradient, Reference model, Reinforcement learning, Value function

A family of reinforcement learning methods that directly adjust a policy's parameters to make actions with high reward more likely. The basic estimate multiplies the gradient of the log-probability of an action by the return it earned. For language models, the policy is the model and an action is a generated token.

Example. If a sampled answer earns reward +1 and the average is 0, the update raises the probability of every token in that answer slightly.

Related: Proximal policy optimisation, Advantage function, Policy (reinforcement learning), Reinforcement learning

All the training stages applied after pretraining to make a base model useful and safe, typically supervised fine-tuning, preference optimisation, reinforcement learning and safety tuning. It uses far less compute than pretraining but strongly shapes how the model behaves with users.

Example. After pretraining, a lab runs SFT on curated conversations, then preference tuning on ranked answers, then safety tuning; this whole sequence is post-training.

Related: Pretraining, Supervised fine-tuning, RLHF, Alignment

A dataset where each prompt comes with two or more responses and a label saying which one people (or another model) prefer. It is the raw material for training reward models and for direct methods such as DPO. Quality, diversity and annotator agreement determine how good the resulting alignment is.

Example. A row holds the prompt 'Suggest a savings plan', a chosen answer that asks about income and goals, and a rejected answer that gives generic advice.

Related: Reward model, Direct preference optimisation, Chosen and rejected responses, RLHF

A parameter-efficient method that learns continuous vectors prepended to the attention keys and values at every layer while the model itself is frozen. Only these vectors are trained and stored for each task.

Example. 20 learned prefix vectors per layer steer a frozen model to produce table-to-text summaries.

Related: Prompt tuning, Parameter-efficient fine-tuning, Adapter

The first and most expensive stage of building a large language model, where the network learns from a huge unlabelled text collection by repeatedly predicting missing or next pieces of text. It produces a general-purpose model that knows grammar, facts and patterns, and is later specialised by fine-tuning and alignment.

Example. A team trains a model on trillions of tokens of web pages, books and code for weeks on thousands of GPUs; the result can continue any text but does not yet follow instructions well.

Related: Base model, Foundation model, Supervised fine-tuning, Next-token prediction

A reward model that scores each intermediate step of a reasoning chain rather than only the final answer. Step-level feedback helps locate where a solution goes wrong, and can guide search or training, at the cost of expensive step-level labels.

Example. In a five-step algebra solution, the model gives steps 1-3 high scores and step 4 a low score where a sign error occurs.

Related: Outcome reward model, Reward model, Reasoning model, Best-of-n sampling

A parameter-efficient method that learns a short sequence of continuous embedding vectors placed at the input of a frozen model, so the task is encoded in the learned 'soft prompt' rather than in weights or hand-written text.

Example. 100 learned soft-prompt vectors are prepended to every input for a topic-classification task.

Related: Prefix tuning, Parameter-efficient fine-tuning, Soft prompt

A reinforcement learning algorithm that improves a policy in small, stable steps by clipping how far the new policy's probabilities may move from the old ones in each update. It was the standard optimiser in early RLHF pipelines, using a value model to estimate advantages.

Example. With clip range 0.2, if a token's probability under the new policy rises to 1.5 times the old, the clipped objective stops rewarding the extra increase beyond 1.2 times.

Related: Clipped surrogate objective, Advantage function, RLHF, Group relative policy optimisation

A method that fine-tunes LoRA adapters on top of a base model whose weights are frozen and stored in 4-bit precision. Gradients flow through the quantised weights into the adapters, allowing large models to be fine-tuned on a single GPU with little quality loss.

Example. A 65B-scale model that would need hundreds of gigabytes in 16-bit can be fine-tuned with QLoRA on one 48 GB GPU.

Related: LoRA, NormalFloat 4-bit, Double quantisation, Paged optimisers

The inner dimension r of the two small matrices in a LoRA update. Higher rank gives the adapter more capacity and more parameters; low ranks such as 8 or 16 are common defaults. Alpha, a scaling constant, is set together with it.

Example. Raising r from 8 to 64 multiplies the adapter's parameter count by 8 for the same layer.

Related: LoRA, QLoRA, Parameter-efficient fine-tuning

A language model trained, often with reinforcement learning on checkable problems, to produce a long chain of intermediate thinking before its final answer. Spending more tokens on thinking tends to improve accuracy on maths, coding and logic tasks, at higher latency and cost.

Example. Asked a puzzle, a reasoning model writes several hundred tokens of working, checks itself, and only then gives the answer.

Related: Test-time compute scaling, Reinforcement learning with verifiable rewards, Chain-of-thought, Group relative policy optimisation

A variant of RLHF where the preference labels come from another AI model, guided by written criteria, instead of from human annotators. It scales cheaply and speeds up iteration, but inherits the judging model's biases.

Example. A strong model compares two answers to 100,000 prompts using a rubric on politeness and accuracy, producing preference labels for training.

Related: RLHF, Constitutional AI, Preference data, LLM-as-judge

A method for aligning a model with human preferences: people rank model answers, a reward model is trained to predict those rankings, and the language model is then optimised with reinforcement learning to score highly under the reward model while staying close to its starting behaviour.

Example. Annotators compare two answers to a loan query and pick the clearer one; thousands of such picks train a reward model that then steers the chat model.

Related: Reward model, Proximal policy optimisation, KL penalty, Direct preference optimisation

Reinforcement learning where the reward comes from an automatic, checkable rule rather than a learned preference model, such as whether a maths answer matches the key or code passes unit tests. It gives cheap, hard-to-game feedback and is used to train reasoning behaviour.

Example. A model gets reward 1 if its final answer to a JEE-style numerical problem equals the known answer, else 0.

Related: Group relative policy optimisation, Reasoning model, Reward model, Process reward model

A training loop where a model generates many answers, a reward model or checker keeps only the best or correct ones, and the model is then fine-tuned on the kept answers. Repeating it gradually raises quality without needing reinforcement learning machinery.

Example. Generate 16 solutions per coding problem, keep those passing the tests, and fine-tune on the passing ones.

Related: Best-of-n sampling, Supervised fine-tuning, Self-Taught Reasoner

When a model finds ways to score highly on the reward signal without truly doing what people wanted, exploiting flaws in the reward model or metric. Typical symptoms are excessive length, flattery or confident nonsense. It is one reason RLHF uses KL penalties and reward-model refreshes.

Example. A summariser rewarded for 'sounding helpful' learns to add long, polite filler that earns higher scores but adds no information.

Related: Goodhart's law, Reward model, KL penalty, Sycophancy

A model that reads a prompt and a response and outputs a single score estimating how much humans would prefer it. It is usually a language model with a scalar output head, trained on ranked pairs of responses. It supplies the training signal in RLHF and can also rank candidate answers.

Example. Given two replies to 'Explain EMI', the reward model scores the clearer one 2.1 and the vague one -0.4, matching the human ranking.

Related: Preference data, Bradley-Terry model, RLHF, Reward hacking

Post-training that teaches a model to refuse or safely handle harmful requests, using examples of unsafe prompts paired with appropriate refusals or safer alternatives, and often followed by preference tuning to refine the balance.

Example. Training on prompts asking for phishing scripts, each paired with a polite refusal and safe alternative.

Related: Alignment, Supervised fine-tuning, Constitutional AI

Research on how humans can reliably supervise AI systems whose outputs are too complex or numerous to check directly, for example by using AI assistants, task decomposition or debate to help evaluators. It matters as models surpass human ability on some tasks.

Example. A reviewer checks a 500-line program by asking an assistant model to point out its riskiest lines.

Related: Weak-to-strong generalisation, Debate (AI safety), Alignment

Empirical rules showing that a language model's loss falls smoothly, following a power law, as model size, dataset size and training compute increase. They let teams predict the payoff of a larger run from small experiments.

Example. Training several small models and plotting loss against compute allows forecasting the loss of a run 100 times larger.

Related: Chinchilla scaling, Compute-optimal training, Emergent abilities, FLOPs

A method for bootstrapping instruction data: start from a small seed set of human-written tasks, ask a language model to generate new instructions and answers, filter poor or duplicate ones, and use the result to fine-tune. It reduces the need for manual annotation.

Example. 175 seed tasks are expanded to tens of thousands of new instructions written by the model itself.

Related: Synthetic data, Evol-Instruct, Instruction tuning, Alpaca

A training approach where a model both generates answers and judges them, using its own judgements as preference labels for further preference training, so its answering and judging abilities improve together over iterations.

Example. Round one: the model writes 4 answers to each prompt and scores them; the best and worst form a preference pair for the next round.

Related: Reinforcement learning from AI feedback, Direct preference optimisation, Synthetic data

A bootstrapping method in which a model writes step-by-step rationales for problems, keeps the rationales that lead to correct answers, and is fine-tuned on them; for failed problems it is shown the correct answer and asked to rationalise. Repeating this improves reasoning without human-written explanations.

Example. A model solves 60 percent of arithmetic word problems, is fine-tuned on its own correct rationales, and then solves more in the next round.

Related: Rejection sampling fine-tuning, Reasoning model, Synthetic data

Concatenating several short training examples into one fixed-length sequence, separated by end tokens, so that no compute is wasted on padding. Attention masks often stop examples from seeing each other. It raises throughput noticeably when data has many short samples.

Example. Instead of 8 padded sequences of length 512 each holding a 100-token sample, pack 5 samples into each 512-token row and use far fewer rows.

Related: Loss masking, Batch size, Supervised fine-tuning, Pretraining

Distillation in which the student is trained on complete outputs generated by the teacher, for instance its decoded translations or answers, rather than on per-token probabilities. It is the common approach when only text outputs, not logits, are available.

Example. Collect 200,000 answers from a large model via its API and fine-tune a smaller open model on those texts.

Related: Knowledge distillation, Synthetic data, Supervised fine-tuning

A reference-free preference method that uses the length-normalised average log-probability of a response as its implicit reward and adds a target margin between chosen and rejected answers. Removing the reference model lowers memory use and curbs a tendency toward longer answers.

Example. Scoring each response by its mean per-token log-probability lets a 20-token and an 80-token answer be compared fairly.

Related: Direct preference optimisation, Reference model, Odds ratio preference optimisation

Target probability distributions over classes or tokens, as opposed to a single correct answer. In distillation the teacher's soft labels reveal how similar wrong answers are, giving the student richer information than one-hot labels.

Example. Teacher gives 'cat' 0.85, 'lynx' 0.10, 'dog' 0.05, so the student learns that a lynx is closer to a cat than a dog is.

Related: Knowledge distillation, Temperature scaling, Cross-entropy loss

A denoising pretraining objective, used by T5, in which contiguous stretches of tokens are replaced by single sentinel placeholders and the model must generate the missing spans in order. It trains an encoder-decoder model to reconstruct text and is a variant of masked language modelling.

Example. The input 'Monsoon <X> Kerala in <Y>' has the target '<X> reaches <Y> June', so the decoder learns to output the removed spans.

Related: Masked language modelling, Pretraining, Causal language modelling

The claim, from the LIMA paper, that almost all knowledge and ability is learned in pretraining, and alignment mainly teaches the style and format of interaction, so a small set of high-quality examples can be enough.

Example. LIMA fine-tuned a large model on only about a thousand carefully chosen examples and still obtained strong chat behaviour.

Related: LIMA, Supervised fine-tuning, Pretraining

Continuing training of a pretrained model on labelled input-output examples written or curated by people, using the same next-token loss but only on the desired responses. It teaches format, tone and task behaviour and is normally the first post-training step before preference-based alignment.

Example. Fine-tuning on 10,000 customer-support chats where the target is the agent's reply teaches a model to answer in a company's style.

Related: Instruction tuning, Fine-tuning, RLHF, Loss masking

A tendency of a model to tell users what they seem to want to hear, agreeing with wrong claims or flattering them, rather than giving accurate answers. It can arise because human raters tend to prefer agreeable replies during preference training.

Example. A user insists that 7 x 8 is 54; a sycophantic model apologises and agrees instead of correcting to 56.

Related: Reward hacking, Preference data, Alignment, RLHF

Training data generated by a model or program instead of collected from people or the real world. It is used to add rare cases, reasoning traces or instruction pairs cheaply, but poorly filtered synthetic data can spread errors and reduce diversity.

Example. A team generates 100,000 synthetic customer questions in Hinglish with a strong model, then filters them before fine-tuning.

Related: Model collapse, Self-instruct, Knowledge distillation, Data curation

Editing a model by computing a task vector, the difference between fine-tuned and base weights, and adding, scaling or subtracting it from the base model to give or remove a skill.

Example. base + 0.5 * (math_finetuned - base) adds half of the maths skill to the base model.

Related: Model merging, Full fine-tuning, Parameter-efficient fine-tuning

The setup used in distillation: a trained, usually larger teacher model provides targets, and a smaller student model learns to reproduce them. The student may match output distributions, hidden features or generated text.

Example. A large multilingual model labels text and a small model for a phone app learns from those labels.

Related: Knowledge distillation, Soft labels, Synthetic data

Splitting individual weight matrices across GPUs so each computes part of every layer's matrix multiplication, then combining results. It lets layers too large for one GPU run, but needs fast interconnects because it communicates at every layer.

Example. A 16384 x 16384 matrix is split column-wise across 4 GPUs, each holding 16384 x 4096.

Related: Pipeline parallelism, Data parallelism, Model parallelism, Megatron-LM

Improving answer quality by spending more computation at inference time, for example by generating longer reasoning, sampling many answers and voting, or searching with a verifier. It complements pretraining scale as a way to raise capability.

Example. Sampling 16 answers to a maths problem and picking the majority answer often beats a single answer from the same model.

Related: Best-of-n sampling, Reasoning model, Process reward model, Scaling laws

A reinforcement learning algorithm that improves a policy while constraining the KL divergence between old and new policies to stay under a threshold, guaranteeing steadier progress. PPO was designed as a simpler, first-order approximation of the same idea.

Example. An update is accepted only if the average KL divergence between the old and new policies is below a small limit such as 0.01.

Related: Proximal policy optimisation, Policy gradient, KL divergence

A large open preference dataset in which many prompts have several model-generated responses scored by a strong model on criteria like helpfulness and honesty, widely used to train open reward models and DPO-tuned chat models.

Example. Its scored response pairs are converted to chosen and rejected pairs for DPO training.

Related: Preference data, Direct preference optimisation, Reinforcement learning from AI feedback

In reinforcement learning, an estimate of the total future reward expected from a given state when following the current policy. In PPO-based RLHF a separate value network, called the critic, learns this so that advantages can be computed for each token.

Example. Halfway through a generated answer, the value function might predict a final reward of 0.4, and the actual reward of 0.7 then yields a positive surprise.

Related: Advantage function, Proximal policy optimisation, Policy (reinforcement learning), Generalised advantage estimation

A learning-rate schedule with three phases: a short warmup, a long stretch at a constant rate, and a final decay to a small value. Since the constant phase has no fixed end, training can be extended or branched flexibly.

Example. Learning rate rises to 3e-4 over 2,000 steps, stays there for most of the run, then falls to near zero over the last 10 percent.

Related: Learning rate warmup, Cosine annealing, Learning rate schedule

A research setting that asks whether a strong model trained on labels from a weaker supervisor can perform better than the supervisor itself. It serves as an analogy for humans supervising future systems smarter than they are.

Example. A small model's noisy labels are used to fine-tune a much larger model, and researchers check whether the large model beats its teacher.

Related: Scalable oversight, Alignment, Supervised fine-tuning

Zero Redundancy Optimizer, a memory-saving method for data-parallel training that partitions optimiser state, then gradients, then parameters across GPUs instead of copying them on each. Stages 1, 2 and 3 partition progressively more.

Example. With 8 GPUs, ZeRO stage 1 stores one-eighth of optimiser state on each GPU instead of a full copy.

Related: Fully sharded data parallel, DeepSpeed, Data parallelism