Deep learning core

A non-linear function applied to each neuron's weighted sum. Without it, stacked layers would collapse into one linear transformation; with it, a network can model curves and complex boundaries. Common choices include ReLU, sigmoid, tanh and GELU.

Example. Applying ReLU to the values [-2, 0.5, 3] gives [0, 0.5, 3].

Related: ReLU, Sigmoid function, Neuron

A memory-efficient adaptive optimiser that stores compact row and column statistics of the squared gradients instead of one value per weight. It is popular for training large transformer models where Adam's extra memory is costly.

Example. For a 4,096 x 4,096 weight matrix it keeps two vectors of 4,096 numbers rather than a 16-million-entry array.

Related: Adam, RMSprop, Optimiser

A model made of many small numeric units arranged in layers, where each unit takes weighted inputs, adds them up and passes the result through a simple function. Connection strengths are learned from data, so the network can approximate very complicated input-to-output mappings without hand-written rules.

Example. A network with 784 inputs (a 28x28 pixel image), one hidden layer of 128 units and 10 outputs can learn to read handwritten digits.

Related: Neuron, Layer, Deep learning

A network trained to reproduce its own input by first compressing it into a small code and then reconstructing it. The bottleneck forces it to learn compact features, useful for denoising, anomaly detection and compression.

Example. An autoencoder squeezing 784 pixel values into 32 numbers, then rebuilding the image, can flag unusual inputs by high reconstruction error.

Related: Bottleneck, Representation learning, Loss function

PyTorch's automatic differentiation engine. It records tensor operations as they run and, when asked, walks that record backward to fill each tensor's gradient. It makes writing new network layers straightforward because only the forward computation must be coded.

Example. x = torch.tensor(3.0, requires_grad=True); y = x**2; y.backward() leaves x.grad equal to 6.0.

Related: Automatic differentiation, Tensor, Backward pass

The algorithm that computes how much each weight contributed to the error. It applies the chain rule of calculus backward from the loss through every layer, producing a gradient for each parameter, which an optimiser then uses to update the weights.

Example. If raising a weight by 0.01 would raise the loss by 0.002, backpropagation reports a gradient near 0.2 for that weight.

Related: Forward pass, Automatic differentiation, Gradient descent

The way recurrent networks are trained: the network is unrolled across the sequence steps into a long feedforward graph and standard backpropagation is applied through all of them. Long sequences make gradients vanish or explode, so it is often truncated.

Example. Truncating to windows of 100 steps limits how far back gradients travel while keeping cost manageable.

Related: Recurrent neural network, Vanishing gradient, Backpropagation

The half of a training step in which gradients of the loss are computed for all parameters, moving from the output layer toward the input. It typically costs about as much as, or somewhat more than, the forward pass because it reuses stored activations.

Example. After computing the loss on a batch, calling loss.backward() in PyTorch runs this step.

Related: Forward pass, Backpropagation, Gradient

A layer that standardises each feature across the mini-batch to zero mean and unit variance, then rescales it with learnable parameters. It stabilises training, allows larger learning rates and acts as mild regularisation, but depends on batch statistics.

Example. For a mini-batch of 64, it computes the mean and variance of each channel over those 64 examples, then applies gamma*x_hat + beta.

Related: Layer normalisation, Batch size, Group normalisation

An extension of batch normalisation that corrects the mini-batch statistics using running averages of the mean and variance, so training-time and test-time behaviour match more closely. It helps when mini-batches are very small or not evenly mixed.

Example. With batches of only 4 examples, it keeps outputs consistent with what the model will see at inference.

Related: Batch normalisation, Batch size, Group normalisation

The number of training examples processed together before the weights are updated. Larger batches give smoother gradient estimates and better hardware use but need more memory, while smaller batches add noise that can help generalisation.

Example. Setting batch size to 64 on a dataset of 6,400 samples gives 100 updates per epoch.

Related: Epoch, Iteration, Gradient accumulation

A 16-bit number format with the same exponent range as 32-bit floating point but fewer precision bits. Its wide range makes training more stable than standard FP16 without loss scaling, and it is supported on modern GPUs and TPUs.

Example. Values up to about 3e38 fit in it, whereas FP16 overflows above 65,504.

Related: Mixed precision training, Loss scaling, Tensor

A learnable constant added to a neuron's weighted sum before the activation function. It lets the neuron shift its output up or down independently of the inputs, so it can fire even when every input is zero. Not to be confused with statistical bias or unfairness.

Example. A neuron computing w*x + b with w = 2 and b = -6 only produces a positive value once x exceeds 3.

Related: Weights, Neuron, Activation function

A recurrent network that runs two passes over a sequence, one forward and one backward, and combines their states at every position. Each output can then use both past and future context, though it needs the whole sequence in advance.

Example. To tag a word in a sentence, it uses words both before and after it.

Related: Recurrent neural network, LSTM, Hidden state

The cross-entropy loss for two-outcome problems: -[y*ln(p) + (1-y)*ln(1-p)], where y is 0 or 1 and p is the predicted probability of class 1. It is normally paired with a sigmoid output.

Example. For a true label 1 and prediction 0.8 the loss is -ln 0.8, about 0.22; a prediction of 0.2 costs about 1.61.

Related: Cross-entropy loss, Sigmoid function, Logits

A deliberately narrow layer in a network that forces information through far fewer units than the layers around it. It compresses data into its essential features, reduces parameter counts and, in autoencoders, is where the compact code lives.

Example. A layer of 32 units between layers of 784 and 784 is a bottleneck in an autoencoder.

Related: Autoencoder, Hidden layer, Representation learning

The tendency of a network trained on a new task to abruptly lose performance on earlier tasks, because updated weights overwrite what was learned before. It is a central obstacle in continual learning and fine-tuning.

Example. A model fine-tuned only on legal text may lose much of its general chat ability.

Related: Multi-task learning, Regularisation, Weights

The internal memory vector of an LSTM that runs along the sequence with only small, gated modifications. Because it is updated mostly by addition, information and gradients can travel through many steps with little loss.

Example. At each step the forget gate multiplies the cell state by values between 0 and 1 to discard outdated information.

Related: LSTM, Hidden state, Recurrent neural network

A saved snapshot of a model's weights, and usually optimiser state and epoch number, written to disk during training. It allows resuming after an interruption and keeping the best-performing version.

Example. Saving a file every epoch lets a Colab run that disconnects at epoch 12 resume from epoch 12.

Related: Training loop, Epoch, Early stopping

A loss for classification that measures the gap between predicted probabilities and the true labels: for one example it is minus the log of the probability assigned to the correct class. Confident wrong answers are punished heavily.

Example. If the model gives the true class probability 0.9 the loss is -ln 0.9, about 0.105; at 0.1 it is about 2.30.

Related: Softmax, Binary cross-entropy, Negative log-likelihood

A training strategy that presents examples in a meaningful order, typically from easy to hard, like a school syllabus. It can speed convergence or improve final quality on difficult tasks.

Example. A speech model might first train on clean, short clips and later on noisy, long ones.

Related: Training loop, Epoch, Batch size

A policy that swings the learning rate between a lower and an upper bound repeatedly instead of only decreasing it. It can reduce tuning effort and help training cross flat regions or saddle points.

Example. The rate might rise from 0.001 to 0.01 and back over every 2,000 iterations.

Related: Learning rate schedule, One-cycle policy, Learning rate

A way to train faster by copying the model onto several devices, giving each a different slice of the batch, and averaging their gradients before every update. It scales training with more GPUs as long as the model fits on one.

Example. Eight GPUs each processing 32 examples behave like one step of batch size 256.

Related: Batch size, Gradient accumulation, Training loop

A branch of machine learning that trains neural networks with many layers so they learn useful features directly from raw data such as pixels, audio or text, instead of relying on hand-crafted features. It powers modern image, speech and language systems.

Example. Instead of engineers designing features to spot a cat, a deep network learns them from millions of labelled photos.

Related: Artificial neural network, Deep neural network, Representation learning

A neural network with several hidden layers stacked between input and output. The extra depth lets it compose simple features into progressively richer ones, which often generalises better than a very wide shallow network for structured data like images and text.

Example. A 50-layer image network is deep; a network with a single hidden layer of 1,000 units is usually called shallow.

Related: Hidden layer, Deep learning, Skip connection

A relative of dropout that randomly zeroes individual weights, instead of whole neuron outputs, during each training step. It gives a finer-grained form of noise-based regularisation for fully connected layers, since every neuron sees a different random subset of its connections.

Example. With probability 0.2, each of a layer's 10,000 weights is masked on a given forward pass.

Related: Dropout, Weights, Regularisation

A regularisation technique that randomly switches off a fraction of neurons during each training step, so the network cannot rely on any single unit. At test time all neurons are used, with values scaled to compensate.

Example. With a rate of 0.5, about half of a 100-unit layer outputs 0 on each step.

Related: Overfitting, DropConnect, Regularisation

A failure in which a ReLU neuron's input becomes negative for all training examples, so it outputs zero and receives zero gradient, and can no longer recover. A too-high learning rate or bad bias values often cause it.

Example. After one huge update, a neuron with bias -50 outputs 0 for every input and never changes again.

Related: ReLU, Leaky ReLU, Vanishing gradient

The exponential linear unit, an activation that behaves like the identity for positive inputs and follows a smooth exponential curve, a*(e^x - 1), for negative ones. Its outputs can be negative, pushing average activations closer to zero, which can speed learning.

Example. ELU(-1) with a = 1 is e^(-1) - 1, about -0.63; ELU(2) = 2.

Related: ReLU, SELU, Activation function

A layer that maps integer identifiers, such as word or item ids, to dense learnable vectors by table lookup. Similar items end up with similar vectors after training, giving the network a compact numeric representation of categories.

Example. A vocabulary of 30,000 tokens with 256-dimensional embeddings is a 30,000 x 256 table of weights.

Related: Representation learning, Weights, Tensor

One complete pass of the learning algorithm over the whole training dataset. Training usually runs for many epochs, and progress is commonly tracked by evaluating loss at the end of each one.

Example. With 50,000 training images and a batch size of 100, one epoch consists of 500 weight updates.

Related: Batch size, Iteration, Training loop

A framework setting that switches layers whose behaviour differs between training and testing to their test-time form: dropout is disabled and batch normalisation uses stored running statistics. Forgetting to set it gives noisy, inconsistent predictions.

Example. In PyTorch, model.eval() turns off dropout before validating.

Related: Dropout, Batch normalisation, Validation loss

A training problem where gradients grow larger and larger while moving backward, causing enormous weight updates, unstable loss and often numerical overflow. It is common in deep or recurrent networks and is typically controlled by clipping and careful initialisation.

Example. A loss that jumps from 2.3 to 10^9 to NaN within a few steps often indicates this failure.

Related: Vanishing gradient, Gradient clipping, Weight initialisation

A smoothed copy of a model's weights updated after each step as a weighted average, ema = decay*ema + (1 - decay)*weights. Evaluating with the averaged weights often gives steadier and slightly better results than the raw final weights.

Example. With decay 0.999, the average reflects roughly the last 1,000 updates.

Related: Stochastic weight averaging, Checkpoint, Optimiser

The number of inputs feeding into a neuron or layer (fan-in) and the number of outputs it sends signals to (fan-out). Initialisation formulas use these counts to scale the starting weights so that signals keep a sensible size.

Example. A dense layer from 300 units to 100 units has fan-in 300 and fan-out 100.

Related: Xavier initialisation, He initialisation, Fully connected layer

A network in which information moves in one direction only, from input through hidden layers to output, with no loops or memory of earlier inputs. It is the simplest network family and the base on which recurrent and transformer designs are built.

Example. Predicting whether an Indian loan applicant will default from 30 tabular features using three stacked dense layers is a feedforward setup.

Related: Multilayer perceptron, Recurrent neural network, Forward pass

Minima in the loss landscape surrounded by a broad low-loss region, as opposed to sharp narrow valleys. They are widely believed, though debated, to generalise better because small shifts in weights change predictions little.

Example. A solution whose loss stays near 0.2 when weights are perturbed by 5 percent is flatter than one that jumps to 2.0.

Related: Sharpness-aware minimisation, Loss landscape, Generalisation gap

A modified cross-entropy that shrinks the loss contribution of examples the model already gets right, multiplying it by (1 - p)^gamma. It focuses training on hard examples and helps when one class hugely outnumbers the others.

Example. In fraud detection with 1 fraud in 1,000 transactions, it keeps easy legitimate cases from dominating the gradient.

Related: Cross-entropy loss, Loss function, Label smoothing

The step in which an input is pushed through the network layer by layer to produce an output and, during training, a loss value. No weights change during it; it just computes predictions using the current weights and stores intermediate values needed later.

Example. Feeding a batch of 32 images through the network to get 32 sets of class scores is one forward pass.

Related: Backward pass, Backpropagation, Loss function

A layer in which each output neuron receives a weighted contribution from every input value. It is implemented as one matrix multiplication plus a bias, which makes it fast on GPUs. It is also called a dense or linear layer.

Example. A fully connected layer mapping 512 inputs to 10 outputs has 512*10 weights plus 10 biases = 5,130 parameters.

Related: Multilayer perceptron, Weights, Parameter

A layer component that multiplies one linear projection of the input by a sigmoid-gated second projection: (xW + b) * sigmoid(xV + c). The gate learns to let information through selectively and is the basis of several transformer feed-forward variants.

Example. With projections giving 4.0 and a gate value of 0.25, the unit outputs 1.0.

Related: SwiGLU, Sigmoid function, Gating mechanism

A learned multiplier between 0 and 1, usually produced by a sigmoid, that decides how much of a signal passes through. Gates let networks such as LSTMs and GRUs choose what to remember, forget or expose.

Example. A forget gate value of 0.1 keeps only 10 percent of the old memory.

Related: LSTM, GRU, Sigmoid function

The Gaussian error linear unit, a smooth activation that multiplies the input by the probability that a standard normal variable is below it: x * Phi(x). Unlike ReLU it curves gently near zero, and it is widely used in transformer models such as BERT and GPT.

Example. GELU(1) is about 0.84, while GELU(-1) is about -0.16.

Related: ReLU, SiLU, Activation function

A memory-saving technique that stores only some layer activations during the forward pass and recomputes the rest when needed in the backward pass. It trades extra computation for a large reduction in memory use.

Example. Recomputing activations may add roughly a third to step time but lets a much deeper model fit on one GPU.

Related: Backward pass, Forward pass, Mixed precision training

The overall size of the gradient vector across all parameters, commonly the square root of the sum of squared entries. Tracking it reveals exploding or vanishing gradients and is the quantity used in norm-based clipping.

Example. A gradient norm that suddenly jumps from 1.2 to 800 is an early warning of instability.

Related: Gradient clipping, Exploding gradient, Vanishing gradient

A normalisation that splits a layer's channels into groups and standardises within each group for every example separately. It does not rely on batch size, so it works well when memory limits force very small batches.

Example. With 32 channels in 8 groups, each normalisation covers 4 channels of one image.

Related: Batch normalisation, Layer normalisation, Batch size

The gated recurrent unit, a simpler alternative to the LSTM that merges memory and state into one vector and uses just two gates (update and reset). It has fewer parameters and often performs comparably.

Example. A GRU layer with 128 units has fewer weights than an LSTM layer of the same size.

Related: LSTM, Recurrent neural network, Hidden state

A scheme that draws initial weights with variance 2 / fan_in, compensating for ReLU zeroing about half its inputs. It keeps activations from shrinking layer after layer in deep ReLU networks.

Example. For a layer with 512 inputs, weights have standard deviation sqrt(2/512) = 0.0625.

Related: Xavier initialisation, ReLU, Weight initialisation

Any layer between the input and output of a network. Its neurons are not directly observed in the data; they learn intermediate features that make the final prediction easier. More hidden layers let a network build more complex features from simpler ones.

Example. In a face model, an early hidden layer may respond to edges while a later one responds to eyes or noses.

Related: Layer, Input layer, Output layer, Deep neural network

The vector a recurrent network carries from one time step to the next, summarising what it has seen so far. It is recomputed at each step from the new input and the previous state.

Example. After reading 'the monsoon has', an RNN's 128-number state encodes context used to predict the next word.

Related: Recurrent neural network, Cell state, LSTM

A regression loss that acts like squared error for small mistakes and like absolute error for large ones, switching at a threshold delta. It stays smooth near zero yet is less thrown off by outliers than plain squared error.

Example. With delta = 1, an error of 0.5 costs 0.125 while an error of 10 costs 9.5, not 50.

Related: Loss function, Gradient descent, Exploding gradient

The first layer of a network, which simply receives the raw feature values. It performs no learning of its own; it defines the shape of the data the model accepts, such as one slot per pixel or per tabular feature.

Example. For a house-price model with 12 features (area, bedrooms, locality score and so on), the input layer has 12 slots.

Related: Hidden layer, Output layer, Layer

A normalisation that standardises each channel of each individual example separately, using its own spatial mean and variance. It became popular in image style-transfer and generation models where per-image contrast should be removed.

Example. Each feature map of one photo is normalised alone, unaffected by other images in the batch.

Related: Batch normalisation, Group normalisation, Layer normalisation

One single update step in training: a forward pass, a loss computation, a backward pass and a weight update on one mini-batch. The number of iterations per epoch equals the dataset size divided by the batch size.

Example. 10,000 samples with batch size 250 mean 40 iterations per epoch, so 10 epochs equal 400 iterations.

Related: Epoch, Batch size, Training loop

A regularisation trick that replaces hard 0/1 target labels with slightly softened ones, such as 0.9 for the true class and the remaining 0.1 spread over other classes. It discourages overconfident predictions and can improve calibration.

Example. With 10 classes and smoothing 0.1, the target for the right class is about 0.91 and each wrong class gets 0.01.

Related: Cross-entropy loss, Softmax, Overfitting

A layer-wise adaptive optimiser that rescales each layer's Adam-style update by the ratio of the layer's weight size to its update size. It allows very large batch sizes to be used without losing accuracy, as in fast BERT pretraining.

Example. A layer with small weights takes proportionally smaller steps than a layer with large weights.

Related: Adam, Batch size, Linear scaling rule

A group of neurons that process their inputs in parallel at the same depth of a network. Data flows from one layer to the next, and each layer transforms the representation produced by the previous one, typically into something more abstract.

Example. A small classifier might stack layers of 256, 128 and 10 neurons, each layer feeding the next.

Related: Hidden layer, Input layer, Output layer

A layer that standardises the values across the features of a single example rather than across the batch, then applies a learned scale and shift. It works with any batch size, which is why transformers and recurrent networks use it.

Example. Each token vector of 768 numbers in a transformer is normalised using its own mean and variance.

Related: Batch normalisation, RMSNorm, Group normalisation

A variant of ReLU that lets a small fraction of negative inputs through instead of zeroing them: f(x) = x if x > 0, otherwise a*x with a small constant such as 0.01. This keeps a gradient flowing for negative values and reduces neurons getting stuck.

Example. With slope 0.01, an input of -5 outputs -0.05, while ReLU would output 0.

Related: ReLU, Dying ReLU, Parametric ReLU

A quick experiment that trains for a short time while increasing the learning rate exponentially and records the loss. A good rate is picked from the region where loss falls fastest, before it blows up.

Example. Sweeping from 1e-7 to 1 over 100 steps and seeing steepest drop near 3e-3 suggests trying about 1e-3.

Related: Learning rate, Learning rate schedule, Hyperparameter

A scheme that draws weights with variance 1 / fan_in so that layer outputs keep roughly unit variance. It is the recommended starting point for SELU networks and an older basis for later methods.

Example. A layer with 100 inputs uses weights with standard deviation 0.1.

Related: SELU, He initialisation, Xavier initialisation

A rule of thumb from large-batch training: when the batch size is multiplied by k, multiply the learning rate by k too, usually with a warmup period at the start. It keeps training behaviour similar as hardware is scaled up.

Example. Moving from batch size 256 at rate 0.1 to batch size 1,024 suggests trying a rate near 0.4.

Related: Batch size, Learning rate warmup, Data parallelism

An optimiser found by automated program search that updates each weight using only the sign of a momentum-based direction, so every step has the same magnitude. It stores less state than Adam, saving memory, and has been reported to work well on some large models.

Example. Instead of scaling by gradient size, each weight moves by plus or minus the learning rate.

Related: Adam, Momentum, Optimiser

A numerical technique for computing log(sum of e^x_i) safely by first subtracting the largest x from every value. It avoids overflow from huge exponentials and underflow from tiny ones, and underlies stable softmax and cross-entropy implementations.

Example. Computing softmax of [1000, 1001] directly overflows; subtracting 1001 first gives [-1, 0] and works fine.

Related: Softmax, Logits, Cross-entropy loss

The raw, unnormalised scores a network produces just before softmax or sigmoid turns them into probabilities. They can be any real number, and losses are often computed directly from them for better numerical stability.

Example. A three-class model outputs logits [3.1, -0.5, 0.2]; softmax then converts them into probabilities.

Related: Softmax, Cross-entropy loss, Output layer

A companion to 16-bit training that multiplies the loss by a large constant before the backward pass so that small gradients do not underflow to zero, then divides the gradients back before updating weights.

Example. Scaling the loss by 1,024 turns a gradient of 1e-8, which FP16 would round to zero, into a representable value.

Related: Mixed precision training, Vanishing gradient, Backward pass

The proposal that a large randomly initialised network contains a small sub-network which, trained alone from its original starting weights, can match the full network's accuracy. It motivated research on pruning and on why large networks train well.

Example. Pruning 90 percent of the weights and resetting the rest to their initial values can still reach similar accuracy in the original paper's experiments.

Related: Overparameterisation, Weight initialisation, Parameter

Long short-term memory, a recurrent unit with a separate memory cell and three gates (input, forget, output) that control what is written, kept or read. Its gating lets gradients flow over long spans, easing the vanishing-gradient problem of plain recurrent networks.

Example. An LSTM can hold on to a subject seen 40 words earlier to choose the correct verb form.

Related: GRU, Cell state, Recurrent neural network

An activation that computes several linear functions of its input and returns the largest of them. It can learn its own piecewise-linear shape, of which ReLU is a special case, at the cost of multiplying the parameter count.

Example. A maxout unit with 2 pieces takes max(w1*x + b1, w2*x + b2).

Related: ReLU, Activation function, Weights

A method that runs most calculations in 16-bit floating point while keeping a 32-bit copy of the weights for updates. It cuts memory use and speeds up GPUs that have hardware for half precision, with little effect on accuracy when done right.

Example. Using autocast in PyTorch with a Colab T4 GPU can noticeably shorten epochs versus pure 32-bit.

Related: Loss scaling, Tensor, Batch size

A way of training a network too large for one device by splitting the model itself, such as its layers or weight matrices, across several devices that pass activations between them. It contrasts with data parallelism, which copies the whole model.

Example. Layers 1-12 of a 24-layer network sit on GPU 0 and layers 13-24 on GPU 1.

Related: Data parallelism, Parameter, Batch size

Training a single network to solve several related tasks at once, sharing early layers and using separate output heads. Shared representations can improve accuracy and data efficiency on each task.

Example. One shared network might predict both sentiment and product category of a review.

Related: Representation learning, Hidden layer, Output layer

A feedforward network made of fully connected layers, where every neuron connects to every neuron in the next layer, with non-linear activations in between. With at least one hidden layer it can learn non-linear patterns that a single-layer model cannot.

Example. An MLP with layers 20 -> 64 -> 32 -> 1 can predict a customer's monthly spend from 20 features.

Related: Feedforward neural network, Fully connected layer, Hidden layer

A loss equal to minus the log of the probability a model gives to the observed data. Minimising it is the same as maximising likelihood, and for classifiers it coincides with cross-entropy when labels are one-hot.

Example. Three independent predictions with probabilities 0.5, 0.8 and 0.9 give NLL = -(ln 0.5 + ln 0.8 + ln 0.9), about 1.02.

Related: Cross-entropy loss, Loss function, Softmax

The removal of weights, neurons or whole channels that contribute little, usually followed by fine-tuning. It shrinks model size and can speed up inference with a modest loss in accuracy.

Example. Removing the 50 percent smallest-magnitude weights of a layer and retraining briefly can leave accuracy nearly unchanged.

Related: Lottery ticket hypothesis, Parameter, Weights

A theoretical tool showing that, for extremely wide networks with suitable scaling, training by gradient descent behaves like fitting a fixed kernel method. It helps analyse why overparameterised networks train and generalise.

Example. In the infinite-width limit, a network's predictions evolve as if governed by a fixed kernel matrix.

Related: Overparameterisation, Universal approximation theorem, Gradient descent

The basic computing unit of a neural network. It multiplies each of its inputs by a weight, sums them, adds a bias and applies an activation function to produce one output number. Loosely inspired by biological nerve cells, but it is just a small formula.

Example. A neuron with inputs [2, 3], weights [0.5, -1] and bias 1 computes 2*0.5 + 3*(-1) + 1 = -1 before its activation is applied.

Related: Weights, Bias term, Activation function

A schedule that raises the learning rate from a small value to a high peak and then lowers it far below the start, all within one training run. It often allows fast training at large rates, sometimes called super-convergence.

Example. A run might rise from 0.004 to 0.04 in the first 30 percent of steps, then fall toward 0.00004.

Related: Cyclical learning rate, Learning rate warmup, Learning rate schedule

The algorithm that uses gradients to decide how to update weights after each step. Options range from plain gradient descent to methods with momentum or per-parameter learning rates, such as Adam.

Example. torch.optim.SGD(model.parameters(), lr=0.01) creates one; Adam is a common alternative.

Related: Adam, RMSprop, Learning rate

A scheme that starts a weight matrix as an orthogonal matrix, one whose columns are perpendicular unit vectors. Such matrices preserve vector length, so signals neither grow nor shrink through many layers or recurrent steps.

Example. Initialising a 256x256 recurrent weight matrix orthogonally helps keep its repeated products stable.

Related: Weight initialisation, Recurrent neural network, Exploding gradient

The final layer of a network, whose values are the prediction. Its size and activation match the task: one linear unit for a regression value, a sigmoid unit for a yes/no probability, or one softmax unit per class for multi-class problems.

Example. A network sorting support tickets into 5 categories ends with 5 output neurons followed by softmax.

Related: Softmax, Logits, Layer

A version of Leaky ReLU in which the slope applied to negative inputs is itself a learnable parameter, trained along with the weights. It lets each layer or channel decide how much negative signal to keep.

Example. If training learns a slope of 0.25 for a layer, an input of -4 becomes -1.

Related: Leaky ReLU, ReLU, Parameter

A network that processes a sequence one element at a time while carrying a hidden state forward, so earlier inputs can influence later outputs. It handles variable-length data but struggles with long-range dependencies and cannot be parallelised across time.

Example. Reading a sentence word by word, updating a state vector after each word, is a recurrent computation.

Related: LSTM, Hidden state, Backpropagation through time

The rectified linear unit, an activation that outputs the input if it is positive and zero otherwise: max(0, x). It is cheap to compute, avoids saturation for positive values and became the default in many deep networks.

Example. ReLU(-3) = 0 and ReLU(4.2) = 4.2.

Related: Leaky ReLU, Dying ReLU, Activation function

A simplified layer normalisation that rescales a vector by its root mean square and a learned gain, skipping the mean subtraction. It is slightly cheaper than full layer normalisation and is used in several recent large language models.

Example. For a vector [3, 4], the root mean square is sqrt((9+16)/2), about 3.54, so the output is about [0.85, 1.13] before gain.

Related: Layer normalisation, Batch normalisation, Weights

A state in which an activation function is so far into its flat region that changing the input barely changes the output, so its gradient is nearly zero. Saturated units learn very slowly, a key reason sigmoid and tanh hurt deep networks.

Example. Sigmoid(10) is about 0.99995, and its slope there is roughly 0.00005, so updates almost stop.

Related: Vanishing gradient, Sigmoid function, Tanh

The scaled exponential linear unit, an activation with fixed constants chosen so that, in a suitably initialised feedforward network, layer outputs tend to keep zero mean and unit variance without explicit normalisation layers. It is called self-normalising.

Example. A deep MLP using SELU with LeCun-normal initialisation can keep stable activations across many layers.

Related: ELU, LeCun initialisation, Batch normalisation

The sigmoid linear unit, an activation defined as x * sigmoid(x). It is smooth, non-monotonic near zero and behaves like ReLU for large inputs. It is also known as Swish and appears in many modern vision and language networks.

Example. SiLU(2) = 2 * 0.88, about 1.76; SiLU(-2) is about -0.24.

Related: GELU, Sigmoid function, ReLU

A shortcut that adds a layer's input directly to a later layer's output, bypassing the layers in between. It gives gradients a direct path backward, which makes very deep networks trainable, and it is central to residual networks and transformers.

Example. Output = F(x) + x, so if F(x) is near zero the block simply passes x along.

Related: Residual connection, Vanishing gradient, Deep neural network

A function that turns a vector of raw scores into probabilities that are positive and sum to 1, by exponentiating each score and dividing by the total: p_i = e^(z_i) / sum of e^(z_j). It is the standard last step for multi-class classification.

Example. Scores [2, 1, 0] become roughly [0.67, 0.24, 0.09].

Related: Logits, Cross-entropy loss, Output layer

A smooth approximation of ReLU defined as ln(1 + e^x). It is always positive and differentiable everywhere, which makes it useful when a network must output a strictly positive quantity such as a variance or a rate.

Example. softplus(0) = ln 2, about 0.69; softplus(10) is almost exactly 10.

Related: ReLU, Activation function, Output layer

The multi-class cross-entropy loss computed when labels are given as integer class indices rather than one-hot vectors. It gives the same value but avoids building large one-hot arrays, saving memory when there are many classes.

Example. With label 2 and predicted probabilities [0.1, 0.2, 0.7], the loss is -ln 0.7, about 0.36.

Related: Cross-entropy loss, Softmax, Logits

A technique that divides a layer's weight matrix by its largest singular value so the layer cannot stretch inputs by more than a fixed factor. It stabilises training of models such as GAN discriminators.

Example. If a weight matrix has largest singular value 4, every entry is divided by 4 so the layer scales inputs by at most 1.

Related: Weight normalisation, Weights, Exploding gradient

A training technique for very deep residual networks that randomly skips entire residual blocks on each step, so the network is effectively shallower while training and full depth at test time. It reduces training time and overfitting.

Example. In a 100-block network, roughly 20 blocks might be bypassed on a given batch.

Related: Skip connection, Dropout, Deep neural network

A technique that averages the weights from several points late in training, typically taken with a constant or cyclical learning rate. The averaged model tends to sit in a flatter region and generalise better.

Example. Averaging weights saved at the end of each of the last 10 epochs gives one final model.

Related: Exponential moving average of weights, Flat minima, Checkpoint

A trick for training through operations with zero or undefined gradients, such as rounding or thresholding. It uses the operation in the forward pass but pretends it was the identity in the backward pass so learning can continue.

Example. In quantisation-aware training, rounding weights to integers is bypassed by copying the gradient straight through.

Related: Backpropagation, Automatic differentiation, Activation function

A gated feed-forward variant that uses SiLU (Swish) as the gate's activation, multiplying it by a second linear projection. It is used in the feed-forward blocks of several recent large language models as a stronger alternative to plain ReLU or GELU layers.

Example. A block computes SiLU(xW) * (xV) and then projects the result back down with a third matrix.

Related: Gated linear unit, SiLU, GELU

The need to start neurons with different weights so they learn different features. If every neuron in a layer began with identical values, they would get identical gradients and stay identical forever, wasting the layer's capacity.

Example. Setting all weights to zero makes all 128 units of a hidden layer behave as a single unit; small random values fix this.

Related: Weight initialisation, Hidden layer, Neuron

The hyperbolic tangent activation, which squashes any input into the range -1 to 1 with a smooth S-shaped curve. Because its output is centred on zero it often trains better than sigmoid in hidden layers, but it still flattens out for large inputs.

Example. tanh(0) = 0, tanh(2) is about 0.96 and tanh(-2) about -0.96.

Related: Sigmoid function, Saturation, Activation function

A training method for sequence models in which the true previous token, rather than the model's own earlier prediction, is fed as the next input. It speeds and stabilises learning but creates a mismatch with generation time.

Example. When training a translator, the correct first word is provided when predicting the second, even if the model guessed wrongly.

Related: Recurrent neural network, Training loop, Loss function

The repeated procedure that trains a network: fetch a batch, run the forward pass, compute the loss, run the backward pass, update weights with an optimiser and clear gradients. Evaluation on validation data is usually interleaved.

Example. In PyTorch, a loop calls optimizer.zero_grad(), loss.backward() and optimizer.step() for every batch.

Related: Epoch, Iteration, Optimiser

A mathematical result showing that a feedforward network with one hidden layer and enough neurons, using a suitable non-linear activation, can approximate any continuous function on a bounded region to any desired accuracy. It says such a network exists, not that training finds it.

Example. In principle a wide single-hidden-layer network can approximate sin(x) on [0, 10] to any accuracy.

Related: Multilayer perceptron, Activation function, Hidden layer

The loss measured on held-out data that the network does not train on. Watching it alongside training loss reveals overfitting: training loss keeps falling while validation loss stops improving or rises.

Example. Training loss of 0.1 with validation loss climbing from 0.4 to 0.6 across epochs signals memorisation.

Related: Overfitting, Early stopping, Learning curve

A training problem in which gradients shrink toward zero as they are propagated back through many layers, so early layers barely learn. It is caused by repeatedly multiplying small numbers, for example slopes of saturating activations below 1.

Example. Multiplying twenty factors of 0.25, the maximum slope of sigmoid, gives about 10^-12, so early-layer weights effectively freeze.

Related: Exploding gradient, Saturation, Skip connection

A regularisation method that shrinks every weight by a small fraction at each update, discouraging very large weights. In plain SGD it is equivalent to L2 regularisation, which adds a penalty proportional to the squared weights to the loss.

Example. With a decay of 0.01 and learning rate 0.1, each update multiplies weights by roughly 0.999 before adding the gradient step.

Related: Regularisation, AdamW, Overfitting

The choice of starting values for a network's weights before training. Good starting values keep signals and gradients at a workable scale from layer to layer; poor ones cause vanishing or exploding gradients or leave neurons identical.

Example. Drawing weights from a normal distribution with standard deviation 0.01 in a 50-layer network would shrink signals to nothing.

Related: Xavier initialisation, He initialisation, Vanishing gradient

A reparameterisation that writes each weight vector as a direction times a separate learnable length, w = g * v / ||v||. Decoupling scale from direction can speed up optimisation without depending on batch statistics.

Example. A neuron's weight vector is stored as a unit direction v/||v|| and a scalar g learned independently.

Related: Batch normalisation, Weights, Gradient descent

Reusing the same set of weights at multiple positions or time steps in a network. It reduces the number of parameters and builds in the assumption that the same pattern can matter anywhere, as in convolutional and recurrent layers.

Example. A recurrent network applies the same weight matrix at every word of a sentence, however long.

Related: Recurrent neural network, Parameter, Weights

Using one shared weight matrix for two roles in a network, most often the input embedding table and the output projection of a language model. It cuts the parameter count and can improve results.

Example. A model with a 50,000-token vocabulary and 512-dimensional embeddings saves about 25.6 million parameters by reusing the table.

Related: Embedding layer, Weight sharing, Parameter

A scheme that sets the spread of initial weights from the number of inputs and outputs of a layer, roughly variance = 2 / (fan_in + fan_out), so that signal size stays similar forwards and backwards. It suits tanh and sigmoid activations.

Example. A layer with 256 inputs and 256 outputs draws weights with variance 2/512, about 0.0039.

Related: He initialisation, Fan-in and fan-out, Weight initialisation