Calculus optimisation

An optimiser that scales each parameter's step by the inverse square root of the sum of all its past squared gradients. Rarely updated parameters get larger steps, which suits sparse features, but the accumulating sum can shrink steps too much over long training.

Example. A word-embedding weight for a rare word gets large updates when it finally appears, while one for a common word takes small steps.

Related: RMSProp, Adam, Learning rate

A popular optimiser that keeps exponential moving averages of the gradient (first moment) and of its square (second moment), corrects their start-up bias, and scales each parameter's step accordingly. It combines momentum with adaptive step sizes and works well with little tuning.

Example. Typical defaults are learning rate 0.001, beta1 = 0.9, beta2 = 0.999 and epsilon = 1e-8; in PyTorch: torch.optim.Adam(model.parameters(), lr=1e-3).

Related: AdamW, RMSProp, Momentum, Learning rate schedule

A version of Adam that applies weight decay directly to the parameters, separately from the gradient-based update, instead of folding it into the gradient as an L2 penalty. This decoupling makes regularisation behave consistently and is the standard choice for training transformers.

Example. torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01) shrinks each weight by a small fraction every step in addition to the Adam update.

Related: Adam, Weight decay, Learning rate schedule

A function whose derivative is a given function. It is determined only up to an added constant, because constants vanish when differentiated. Finding one is the main way to evaluate exact integrals.

Example. An antiderivative of 3x^2 is x^3, since d/dx of x^3 = 3x^2; x^3 + 10 works equally well.

Related: Integral, Fundamental theorem of calculus, Derivative

A sufficient-decrease test for step size: a step t along direction d is accepted only if f(x + t*d) <= f(x) + c * t * (gradient . d), with c a small constant such as 1e-4. It ensures each step lowers the objective by an amount proportional to the step and slope.

Example. If f(x) = 10, gradient . d = -4, t = 0.5 and c = 0.0001, the new value must be at most 10 - 0.0002 = 9.9998.

Related: Backtracking line search, Line search, Gradient descent

A technique that computes exact derivatives of a program by recording elementary operations and applying the chain rule to each of them, without deriving formulas by hand or using approximations. It is what PyTorch, TensorFlow and JAX use to obtain gradients of losses.

Example. In PyTorch, setting x = torch.tensor(3.0, requires_grad=True), y = x**2 and calling y.backward() leaves x.grad equal to 6.0.

Related: Reverse-mode automatic differentiation, Forward-mode automatic differentiation, Computational graph, Chain rule

A line search that starts with a large trial step and shrinks it by a constant factor until a sufficient-decrease condition is met. It is cheap, simple and needs only function evaluations.

Example. Begin at step 1, halve repeatedly (1, 0.5, 0.25) until the Armijo condition holds.

Related: Line search, Armijo condition, Gradient descent

A form of gradient descent that computes the gradient of the loss over the entire training set before every parameter update. Each step is accurate and smooth but expensive, and it becomes impractical when data does not fit in memory.

Example. With 1 million training rows, every single update needs a pass over all 1 million before weights change once.

Related: Stochastic gradient descent, Mini-batch gradient descent, Gradient descent

A strategy for optimising expensive black-box functions by fitting a probabilistic surrogate model, such as a Gaussian process, to the evaluations so far, then using an acquisition function to choose the next point that balances exploration and exploitation.

Example. To tune 5 hyperparameters of a model that takes 2 hours per training run, it picks each new configuration from a surrogate rather than at random.

Related: Derivative-free optimisation, Objective function, Global minimum

The best-known quasi-Newton algorithm, named after Broyden, Fletcher, Goldfarb and Shanno. It maintains a dense approximate inverse Hessian updated with rank-two corrections, and is typically paired with a line search. It suits problems with up to a few thousand variables.

Example. scipy.optimize.minimize(f, x0, method="BFGS") runs it on a smooth function of a few dozen parameters.

Related: L-BFGS, Quasi-Newton method, Line search

The rule for differentiating a function built from other functions: the derivative of f(g(x)) is f'(g(x)) * g'(x). Slopes of the inner and outer steps multiply. It is the mathematical engine of backpropagation, where gradients flow through many stacked layers.

Example. For y = (3x + 1)^2, let u = 3x + 1; then dy/dx = 2u * 3 = 6(3x + 1). At x = 1 the slope is 24.

Related: Multivariable chain rule, Automatic differentiation, Computational graph, Derivative

A directed graph in which nodes are values or operations and edges show how results feed later steps. Automatic differentiation records one while a program runs, then walks it backward applying the chain rule to obtain gradients.

Example. For loss = (w*x - y)^2, the graph has multiply, subtract and square nodes; gradients flow backward through them to reach w.

Related: Automatic differentiation, Chain rule, Reverse-mode automatic differentiation

The ratio of the largest to the smallest curvature of an objective, given at a minimum by the ratio of the extreme Hessian eigenvalues. A large value makes the surface a long narrow valley, forcing small steps and slow zig-zagging convergence.

Example. For f(x, y) = x^2 + 100y^2, the Hessian eigenvalues are 2 and 200, so the condition number is 100 and plain gradient descent crawls.

Related: Hessian matrix, Convergence rate, Momentum, Preconditioning

An iterative method that minimises quadratic functions, or solves symmetric positive definite linear systems, by choosing successive search directions that do not undo earlier progress. It needs no matrix storage or inversion and takes at most n steps for n variables in exact arithmetic.

Example. To solve A x = b for a huge sparse matrix A, it uses only matrix-vector products with A, never forming the inverse.

Related: Newton's method, Hessian matrix, Line search

Optimisation where the variables must also satisfy equalities or inequalities, so the best answer must lie in the allowed region. Standard tools include Lagrange multipliers, the KKT conditions, projections and penalty or barrier methods.

Example. Minimise x^2 + y^2 subject to x + y = 2: the best point is (1, 1) with value 2, not the unconstrained minimum at (0, 0).

Related: Lagrange multiplier, KKT conditions, Projected gradient descent, Feasible region

A condition that candidate solutions must satisfy, such as an equation or inequality on the variables. Such conditions shrink the set of allowed answers and often change where the best answer lies.

Example. Maximise profit subject to 2x + 3y <= 120, where x and y are units of two products and 120 is the available labour hours.

Related: Feasible region, Constrained optimisation, Lagrange multiplier

A property of a function whose graph has no jumps, gaps or breaks: a tiny change in the input causes only a tiny change in the output. Differentiation requires it, and most smooth losses and activations used in training are continuous.

Example. f(x) = x^2 is continuous everywhere, while a step function that jumps from 0 to 1 at x = 0 is not continuous at 0.

Related: Limit, Differentiability, Lipschitz continuity

The point at which an iterative algorithm settles, so that further steps no longer change the answer or objective meaningfully. Practitioners watch for the loss to flatten or the gradient norm to become tiny.

Example. Training loss falling 2.3, 1.1, 0.62, 0.55, 0.549, 0.549 suggests convergence after the fifth step.

Related: Convergence rate, Gradient descent, Learning rate

A measure of how quickly an iterative method approaches its solution, described as sublinear, linear (error shrinks by a constant factor each step) or quadratic (error roughly squares each step). It lets you compare optimisers on theory.

Example. Gradient descent on a strongly convex smooth function has a linear rate, while Newton's method near the minimum has a quadratic rate, e.g. error 0.1 becoming 0.01 then 0.0001.

Related: Convergence, Strong convexity, Newton's method, Condition number

A function where the line segment between any two points on its graph lies on or above the graph, so it is bowl-shaped. Every local minimum of such a function is a global minimum, which makes gradient-based methods reliable and provably convergent.

Example. f(x) = x^2 is convex. Linear regression with squared error is a convex problem, so gradient descent will find the best coefficients.

Related: Convex optimisation, Strong convexity, Global minimum

The study of minimising a convex function over a convex set. Such problems have no spurious local minima and can be solved reliably and efficiently. Linear programs, least squares, logistic regression and support vector machines are convex problems.

Example. Fitting logistic regression with cross-entropy loss is convex in the weights, so different starting points end at the same solution.

Related: Convex function, Convex set, Linear programming, KKT conditions

A collection of points in which the straight line segment between any two members lies entirely inside the collection. Circles, half-planes and boxes qualify; a crescent shape does not. Optimising over one is far better behaved.

Example. The unit disc x^2 + y^2 <= 1 is convex: any two points inside it are joined by a segment that stays inside.

Related: Convex function, Feasible region, Convex optimisation

An optimiser that minimises an objective by adjusting one variable (or block) at a time while holding others fixed, cycling through them. Each update can be very cheap, so it suits problems such as Lasso regression and support vector machine solvers.

Example. To minimise f(x, y), fix y and minimise over x, then fix x and minimise over y, and repeat until nothing changes.

Related: Gradient descent, Proximal gradient method, Convergence

A learning-rate schedule that lowers the rate from an initial value to a minimum following half a cosine curve, decreasing slowly at first, faster in the middle and gently at the end. It is widely used for training vision and language models.

Example. lr(t) = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T)); with lr_max = 0.1 and lr_min = 0 at halfway, lr = 0.05.

Related: Learning rate schedule, Warm restarts, Learning rate warmup

The instantaneous rate of change of a function at a point, defined as the limit of (f(x+h) - f(x)) / h as h shrinks to zero. Geometrically it is the slope of the tangent line. It tells you how much the output moves per tiny nudge of the input.

Example. For f(x) = x^2, the derivative is 2x, so at x = 3 the function rises about 6 units per unit step of x.

Related: Tangent line, Partial derivative, Gradient, Rate of change

Methods that optimise a function using only its values, with no gradient information. They are used when the function is a black box, non-differentiable or noisy, as with hyperparameter tuning or simulation. Nelder-Mead and evolutionary methods are examples.

Example. Tuning a learning rate and batch size by trying combinations and measuring validation accuracy needs no gradient of accuracy with respect to those settings.

Related: Bayesian optimisation, Simulated annealing, Genetic algorithm

A function is differentiable at a point if its slope exists and is the same whether you approach from the left or the right. Sharp corners, jumps and vertical tangents break it. Deep-learning frameworks still handle non-smooth points, such as ReLU at zero, by choosing a convention.

Example. f(x) = |x| is continuous but not differentiable at x = 0 because the slope is -1 on the left and +1 on the right.

Related: Continuity, Derivative, Subgradient

The rate at which a function changes when you move from a point in a chosen direction. It equals the dot product of the gradient with a unit vector in that direction, and it is largest when that direction matches the gradient.

Example. For f(x, y) = x^2 + y^2 at (3, 4), moving along the unit vector (1, 0) gives 6 * 1 + 8 * 0 = 6 per unit step.

Related: Gradient, Partial derivative, Level set

The principle of pairing an optimisation problem, the primal, with a related problem, the dual, whose solution bounds the primal one. For convex problems under mild conditions the two optimal values coincide, and the dual is sometimes easier to solve, as in kernel SVMs.

Example. The dual of a support vector machine expresses training in terms of one multiplier per training example, allowing the kernel trick.

Related: Lagrangian, KKT conditions, Convex optimisation

The constant e, about 2.71828, defined as the limit of (1 + 1/n)^n as n grows without bound. It is the unique base for which an exponential function has slope equal to its own value, so it is the natural base for calculus.

Example. If Rs 1 earned 100% annual interest compounded continuously, it would grow to about Rs 2.718 after one year.

Related: Exponential function, Natural logarithm, Limit

A function of the form a^x where the input sits in the exponent; the most important case is e^x, whose derivative equals itself. It models growth and decay and appears inside softmax, sigmoid and probability densities.

Example. e^x has slope e^x everywhere: at x = 0 both the value and the slope are 1. Money compounding at a fixed rate follows an exponential curve.

Related: Natural logarithm, Euler's number, Chain rule

The set of all points that satisfy every constraint of an optimisation problem. The best feasible point is the answer. For linear constraints it is a polygon or polyhedron, and for convex problems it is a convex set.

Example. With x >= 0, y >= 0 and x + y <= 4, the feasible region is the triangle with corners (0, 0), (4, 0) and (0, 4).

Related: Constraint, Convex set, Linear programming

An approximation of a derivative computed from function values at nearby points, such as (f(x + h) - f(x)) / h. It needs no formula for the derivative but suffers from rounding and truncation error, so it is mostly used to verify analytic gradients.

Example. For f(x) = x^2 at x = 3 with h = 0.001: (9.006001 - 9) / 0.001 = 6.001, close to the true derivative 6.

Related: Numerical differentiation, Gradient checking, Derivative

The necessary requirement that the gradient of a differentiable function equals zero at an unconstrained optimum. It identifies candidate points but cannot by itself say whether each is a minimum, maximum or saddle.

Example. For f(x, y) = x^2 + y^2 - 4x, setting the gradient (2x - 4, 2y) to zero gives the candidate point (2, 0).

Related: Second-order condition, Stationary point, Gradient

A minimum surrounded by a wide region where the loss stays low, meaning small changes in the parameters barely affect it. Such minima are commonly linked to better generalisation than sharp minima, though the link depends on how sharpness is measured.

Example. Two models reach the same training loss of 0.10; nudging the first one's weights by 1% keeps loss near 0.10, but the second jumps to 0.60, so the first sits in a flatter minimum.

Related: Loss landscape, Sharpness-aware minimisation

A variant that propagates derivatives alongside values from inputs toward outputs, computing one directional derivative per pass. It is efficient when a function has few inputs and many outputs, and it computes Jacobian-vector products.

Example. A simulation with 3 tunable parameters and 10,000 outputs needs only 3 forward passes to get its full Jacobian.

Related: Reverse-mode automatic differentiation, Jacobian-vector product, Automatic differentiation

A rule that assigns exactly one output to each allowed input. Written f(x), it describes how one quantity depends on another. Calculus studies how such rules change, and machine learning models are ultimately large rules mapping inputs to predictions.

Example. f(x) = 3x + 2 maps x = 4 to 14. A model that predicts a house price in lakhs from its area in square feet is a function of that area.

Related: Derivative, Limit, Objective function

The theorem linking derivatives and integrals: accumulating the area under a rate-of-change curve recovers the total change, and differentiating an accumulated area recovers the original curve. It turns integration into evaluation of an antiderivative at two endpoints.

Example. If a car's speed is v(t) = 2t km/h, the distance from t = 0 to t = 3 is the integral of 2t, which is 3^2 - 0 = 9 km.

Related: Integral, Antiderivative, Derivative

An algorithm for non-linear least squares that approximates the Hessian by J^T J, using only first derivatives of the residuals. It avoids second derivatives and works well when residuals are small near the solution.

Example. Fitting the curve y = a * exp(b * x) to data points by repeatedly solving a linear least-squares problem built from the Jacobian of the residuals.

Related: Levenberg-Marquardt algorithm, Jacobian matrix, Newton's method

An evolution-inspired search that maintains a population of candidate solutions and repeatedly selects the fittest, combines them (crossover), and randomly alters them (mutation) to produce better generations. It needs no gradients.

Example. To tune a delivery schedule, encode each schedule as a list, keep the 20 best of 100 each generation, and breed them.

Related: Simulated annealing, Derivative-free optimisation

The point where a function takes its lowest value over its entire domain. Finding it is easy for convex functions, but for non-convex ones such as neural network losses optimisers usually settle for a good local minimum.

Example. f(x) = x^2 - 4x + 7 has its global minimum at x = 2, where f = 3.

Related: Local minimum, Convex function, Saddle point

The vector of all partial derivatives of a scalar function of several variables. At any point it points in the direction of steepest increase, and its length gives the steepness. Training a model means repeatedly stepping opposite this vector for the loss.

Example. For f(x, y) = x^2 + y^2 at (3, 4), the gradient is (6, 8), so moving along (6, 8) raises f fastest and moving along (-6, -8) lowers it fastest.

Related: Partial derivative, Gradient descent, Directional derivative, Jacobian matrix

A technique that sums gradients from several small mini-batches before making one parameter update, imitating a larger batch when memory is limited. The effective batch size is the micro-batch size times the number of accumulation steps.

Example. A GPU that fits 8 examples at a time can run 4 micro-batches and then update once, giving an effective batch size of 32.

Related: Mini-batch gradient descent, Stochastic gradient descent, Gradient clipping

A debugging test that compares an analytically or automatically computed gradient with a finite-difference estimate. A large relative gap suggests a bug in the gradient code. It is slow, so it is used on small examples rather than during real training.

Example. If autodiff gives 6.000 and finite differences give 6.001, the relative error is about 0.00008, so the gradient passes the check.

Related: Finite difference, Numerical differentiation, Automatic differentiation

A safeguard that limits the size of gradients before an update, either by capping each component or by rescaling the whole gradient when its norm exceeds a threshold. It prevents sudden huge steps that could destabilise training, especially in recurrent and transformer models.

Example. If the gradient norm is 50 and the threshold is 1, the gradient is multiplied by 1/50 so its new norm is 1: torch.nn.utils.clip_grad_norm_(params, 1.0).

Related: Gradient descent, Learning rate, Gradient accumulation

An iterative algorithm that minimises a function by repeatedly moving its variables a small step in the direction opposite the gradient, the direction of steepest decrease. It is the basis of how nearly every machine learning model is trained.

Example. Minimise f(w) = (w - 3)^2 from w = 0 with learning rate 0.1: the update w - 0.1 * 2(w - 3) moves w to 0.6, then 1.08, then 1.46, approaching 3.

Related: Learning rate, Stochastic gradient descent, Gradient, Loss landscape

The square matrix of all second-order partial derivatives of a scalar function. It describes curvature: how the slope itself changes in every direction. Its eigenvalues reveal whether a stationary point is a minimum, maximum or saddle, and it drives Newton-type optimisers.

Example. For f(x, y) = x^2 + 3y^2, the Hessian is [[2, 0], [0, 6]], so the surface curves three times more sharply along y than along x.

Related: Newton's method, Second-order condition, Saddle point, Condition number

A quantity that adds up infinitely many infinitesimal pieces, most commonly the area under a curve between two points. It is the reverse operation of differentiation and is used to compute totals, probabilities from densities and expected values.

Example. The integral of f(x) = 2x from 0 to 3 is 9, the area of a triangle with base 3 and height 6 under y = 2x.

Related: Antiderivative, Riemann sum, Fundamental theorem of calculus

The matrix of all first-order partial derivatives of a vector-valued function: entry (i, j) is the derivative of output i with respect to input j. It describes the best linear approximation of the function near a point and generalises the gradient.

Example. For f(x, y) = (x*y, x + y), the Jacobian is [[y, x], [1, 1]]; at (2, 3) it is [[3, 2], [1, 1]].

Related: Gradient, Hessian matrix, Vector-Jacobian product, Multivariable chain rule

The product of a Jacobian matrix with a vector, giving how all outputs change when the inputs move in one chosen direction. Forward-mode differentiation computes it without forming the Jacobian.

Example. In JAX, jax.jvp(f, (x,), (v,)) returns f(x) together with J(x) times v for a direction v.

Related: Vector-Jacobian product, Forward-mode automatic differentiation, Jacobian matrix

The Karush-Kuhn-Tucker conditions, necessary requirements for optimality in problems with inequality constraints: stationarity of the Lagrangian, feasibility, non-negative multipliers for inequalities, and complementary slackness. They are also sufficient for convex problems and underlie support vector machines.

Example. For minimising x^2 subject to x >= 1, the multiplier is 2 at x = 1 with the constraint active; complementary slackness says the multiplier times the constraint slack is zero.

Related: Lagrange multiplier, Duality, Convex optimisation, Constrained optimisation

A limited-memory version of BFGS that stores only the last handful of parameter and gradient changes, rather than a full matrix, to implicitly represent curvature. It scales to many variables and is used for classical models like logistic regression.

Example. scikit-learn's LogisticRegression uses the "lbfgs" solver by default, keeping perhaps 10 recent updates.

Related: BFGS, Quasi-Newton method, Line search

A property of a differentiable function whose gradient is Lipschitz continuous with constant L, meaning the slope cannot change faster than L allows. It implies gradient descent with step size 1/L is guaranteed to decrease the objective every iteration.

Example. f(x) = 2x^2 has gradient 4x, which is 4-Lipschitz, so it is 4-smooth and a step size of 0.25 is safe.

Related: Lipschitz continuity, Strong convexity, Learning rate, Convergence rate

A number attached to a constraint that converts a constrained problem into an unconstrained one via the Lagrangian. At the solution it measures how sensitive the optimal value is to loosening that constraint.

Example. To minimise x^2 + y^2 subject to x + y = 2, set gradient f = lambda * gradient of constraint: 2x = lambda and 2y = lambda, so x = y = 1 and lambda = 2.

Related: Lagrangian, KKT conditions, Constrained optimisation

The function formed by adding each constraint, multiplied by its multiplier, to the objective: L(x, lambda) = f(x) + lambda * g(x). Its stationary points give candidate constrained optima and it is the starting point for duality.

Example. For f = x^2 + y^2 and constraint x + y - 2 = 0, L = x^2 + y^2 + lambda * (x + y - 2).

Related: Lagrange multiplier, Duality, KKT conditions

The scalar that sets how large a step an optimiser takes along the negative gradient. Too small and training crawls; too large and it overshoots or diverges. It is usually the most important setting to tune.

Example. With gradient 4 and learning rate 0.01, a weight changes by 0.04 per step; with learning rate 1.0 it would jump by 4 and could bounce past the minimum.

Related: Gradient descent, Learning rate schedule

A rule that changes the learning rate during training, usually starting larger for fast progress and decreasing so the model can settle precisely. Common forms include step decay, exponential decay, cosine annealing and warmup followed by decay.

Example. Start at 0.1 and multiply by 0.1 at epochs 30 and 60, or follow a cosine curve from 0.1 to nearly 0 across 100 epochs.

Related: Learning rate warmup, Cosine annealing, Step decay, Learning rate

A schedule phase that starts training with a very small learning rate and raises it gradually over the first steps to its target value. It stabilises early updates for adaptive optimisers and large models, where initial gradient estimates are unreliable.

Example. Increase the rate linearly from 0 to 3e-4 across the first 2,000 steps, then apply cosine decay.

Related: Learning rate schedule, Adam, Cosine annealing

The set of all inputs at which a function takes one fixed value. In two dimensions these are contour lines, like height contours on a trekking map. The gradient is always perpendicular to the one passing through a point, which helps visualise a loss landscape.

Example. For f(x, y) = x^2 + y^2, the level set f = 25 is a circle of radius 5 centred at the origin.

Related: Gradient, Loss landscape, Directional derivative

A non-linear least-squares method that blends Gauss-Newton with gradient descent by adding a damping term to J^T J. Large damping behaves like cautious gradient descent; small damping behaves like Gauss-Newton. It is a standard curve-fitting routine.

Example. scipy.optimize.curve_fit uses it by default for unbounded problems when fitting a model to measurements.

Related: Gauss-Newton method, Trust region method, Jacobian matrix

The value that a quantity gets closer and closer to as its input approaches some point, whether or not it actually reaches it there. This idea gives a rigorous meaning to instantaneous change and underpins both slopes and areas under curves.

Example. As x approaches 2, f(x) = (x^2 - 4) / (x - 2) approaches 4, even though the formula is undefined at x = 2 itself.

Related: Continuity, Derivative, Integral

A step-size selection procedure that, once a search direction is fixed, tests candidate step lengths along it until the objective decreases enough. Exact and backtracking variants are common in classical optimisers such as BFGS.

Example. Try step 1.0; if the loss does not fall enough, try 0.5, then 0.25, until it does.

Related: Backtracking line search, Armijo condition, Learning rate

Estimating a function near a point by its tangent line or plane: f(x) is approximately f(a) + f'(a)(x - a). It is accurate for small steps and is the idea behind first-order optimisers, which trust the gradient only locally.

Example. sqrt(4.1) is approximately 2 + (1/4)(0.1) = 2.025; the exact value is about 2.0248.

Related: Tangent line, Taylor series, Gradient descent

Optimising a linear objective subject to linear equality and inequality constraints. The feasible region is a polyhedron and an optimum occurs at a corner. Solvers include the simplex method and interior-point methods.

Example. Maximise profit 40x + 30y over units of two products subject to 2x + y <= 100 and x + 2y <= 80; the best plan sits at a corner of the feasible region.

Related: Feasible region, Convex optimisation

A function is Lipschitz continuous with constant L if the difference in outputs is never more than L times the distance between inputs. It caps how fast the function can change and is used to bound step sizes and prove convergence.

Example. f(x) = 3x + 1 is Lipschitz with L = 3, since |f(a) - f(b)| = 3|a - b|.

Related: L-smoothness, Continuity, Convergence rate

A point whose function value is lower than at every other point in a small neighbourhood, though a lower value may exist elsewhere. Gradient-based methods can settle there because the slope is zero.

Example. f(x) = x^4 - 3x^2 + x has a shallow local minimum near x = 1.13 and a deeper global minimum near x = -1.30.

Related: Global minimum, Saddle point, Stationary point

The surface formed by plotting a model's loss against its parameters. Its hills, valleys, plateaus and saddles determine how easy training is. It is high-dimensional, so it is studied through slices, contours and curvature statistics.

Example. Slicing the loss of a small network along two random directions through the trained weights yields a 3D surface with a broad basin.

Related: Level set, Saddle point, Flat minimum, Objective function

A compromise that computes each gradient on a small random subset of the data, typically tens to thousands of examples. It reduces noise compared with single-example updates while exploiting GPU parallelism, and is the standard way deep networks are trained.

Example. With 50,000 images and batch size 100, each pass over the data makes 500 parameter updates.

Related: Stochastic gradient descent, Batch gradient descent, Gradient accumulation

An optimisation trick that keeps a running velocity, an exponentially weighted sum of past gradients, and updates parameters using it. Steps in consistent directions build speed while oscillating directions cancel, which accelerates progress along narrow valleys.

Example. With momentum 0.9, velocity = 0.9 * velocity + gradient, then weights change by learning rate times velocity.

Related: Nesterov momentum, Stochastic gradient descent, Condition number

The extension of the chain rule to functions of several variables: the derivative of an output with respect to an input is the sum, over every path connecting them, of the products of local partial derivatives along that path. It is how gradients are summed when a value feeds several later computations.

Example. If z = x*y with x = a + b and y = a * b, then dz/da = y*1 + x*b, adding one term per path from a to z.

Related: Chain rule, Computational graph, Jacobian matrix, Total derivative

The logarithm with base e, written ln(x) or log(x). It is the inverse of the exponential function, and its derivative is 1/x. It turns products into sums, which keeps probabilities numerically stable and simplifies gradients of likelihoods and cross-entropy.

Example. ln(e^3) = 3, and ln(0.5 * 0.4 * 0.9) = ln(0.5) + ln(0.4) + ln(0.9). The slope of ln(x) at x = 4 is 0.25.

Related: Exponential function, Euler's number, Chain rule

A refinement of momentum that evaluates the gradient at the look-ahead point where the current velocity would take the parameters, rather than at the current point. This early correction reduces overshoot and gives better theoretical rates for convex problems.

Example. With velocity 0.5 and weight 2.0, it computes the gradient at about 2.0 + 0.9 * 0.5 rather than at 2.0.

Related: Momentum, Stochastic gradient descent, Learning rate

An iterative optimiser that uses both the gradient and Hessian to model the objective as a quadratic and jump straight to that quadratic's minimum: x_new = x - H^-1 * g. It converges very fast near a solution but each step is costly for many parameters.

Example. For f(x) = x^2 - 4x, at x = 0 the gradient is -4 and the Hessian is 2, so one step gives x = 0 - (-4/2) = 2, the exact minimum.

Related: Hessian matrix, Quasi-Newton method, Convergence rate, Gradient descent

Estimating derivatives by evaluating a function at nearby inputs rather than deriving a formula. It works for any function you can run, but it needs one or more evaluations per input dimension and is sensitive to the step size chosen.

Example. For a model with 1,000 parameters, estimating the full gradient this way needs at least 1,001 loss evaluations, which is why training uses automatic differentiation instead.

Related: Finite difference, Automatic differentiation, Symbolic differentiation

The quantity an optimisation seeks to minimise or maximise, written as a function of the variables being tuned. In machine learning it is usually a loss measuring how wrong a model is on the training data.

Example. Mean squared error, (1/n) * sum of (prediction - actual)^2, is the objective when fitting a line to house prices.

Related: Optimisation problem, Constraint, Loss landscape

The task of finding inputs that make some quantity as small or as large as possible, possibly subject to rules the inputs must obey. Training a model, planning delivery routes or allocating an ad budget can all be posed this way.

Example. Choose how many of 3 delivery vans to send to each of 5 areas in Bengaluru so total travel time is minimised.

Related: Objective function, Constraint, Global minimum

The derivative of a function of several variables taken with respect to one variable while every other variable is treated as a constant. It measures sensitivity of the output to that single input. Collecting all of them gives the gradient.

Example. For f(x, y) = x^2 * y, the partial with respect to x is 2xy and with respect to y is x^2. At (3, 2) they are 12 and 9.

Related: Gradient, Derivative, Total derivative

A differentiation rule stating that the derivative of x raised to a constant power n is n times x raised to n - 1. It is the workhorse for differentiating polynomials and is applied term by term.

Example. d/dx of x^5 = 5x^4, and d/dx of (3x^2 + 7x) = 6x + 7.

Related: Derivative, Sum rule, Chain rule

Transforming an optimisation problem, usually by rescaling variables or the gradient, so that its curvature is more uniform and the condition number is smaller. Adaptive optimisers such as Adam act as cheap diagonal preconditioners.

Example. Standardising two features from Rs and square feet to zero mean and unit variance reshapes an elongated loss valley into a rounder bowl.

Related: Condition number, Adam, Newton's method

A differentiation rule for a product of two functions: the derivative of u(x) * v(x) is u'(x) * v(x) + u(x) * v'(x). Each factor is differentiated in turn while the other is held fixed.

Example. d/dx of (x^2 * e^x) = 2x * e^x + x^2 * e^x = e^x * (2x + x^2).

Related: Quotient rule, Chain rule, Power rule

A constrained version of gradient descent that takes an ordinary gradient step and then projects the result back to the nearest point in the feasible set. It suits simple constraints such as boxes or norm balls.

Example. To keep weights within [0, 1], take a step, then clip any value below 0 to 0 and above 1 to 1.

Related: Constrained optimisation, Gradient descent, Feasible region

An algorithm for minimising a smooth function plus a simple non-smooth term, such as an L1 penalty. It alternates a gradient step on the smooth part with a proximal step that handles the other part exactly, for example soft-thresholding.

Example. For Lasso, take a gradient step on the squared error, then shrink each coefficient toward zero by lambda * lr, setting small ones exactly to zero.

Related: Coordinate descent, Subgradient method, Convex optimisation

A family of optimisers that build up an approximation of the inverse Hessian from successive gradient differences instead of computing second derivatives directly. They keep much of Newton's fast convergence at a much lower cost per step.

Example. BFGS updates its curvature estimate after every step using only the change in the parameters and the change in the gradient.

Related: BFGS, L-BFGS, Newton's method

A differentiation rule for a ratio of two functions: the derivative of u/v is (u'v - uv') / v^2. It is needed whenever a variable appears in a denominator, though it can often be avoided by rewriting the ratio as a product.

Example. d/dx of (x / (x + 1)) = ((1)(x + 1) - x(1)) / (x + 1)^2 = 1 / (x + 1)^2.

Related: Product rule, Chain rule, Derivative

How much one quantity changes per unit change in another. The average version is measured over an interval as change in output divided by change in input; the instantaneous version is the derivative. It is the core idea behind slopes, speeds and sensitivities.

Example. If a delivery app grows from 2,000 to 2,600 orders in 3 weeks, the average rate of change is 200 orders per week.

Related: Derivative, Secant line, Tangent line

A variant that first runs the computation forward, then sweeps backward from the output propagating sensitivities to every input, giving the whole gradient of a scalar output in one backward pass. It is efficient for losses with millions of parameters and is the basis of backpropagation.

Example. For a loss over 100 million parameters, one forward pass plus one backward pass yields all 100 million partial derivatives, at a cost of a few forward passes.

Related: Forward-mode automatic differentiation, Vector-Jacobian product, Computational graph, Automatic differentiation

An approximation of an integral made by splitting the interval into thin strips, computing each strip's height times width, and adding them. As the strips get thinner the sum approaches the exact integral. It is the basis of numerical integration.

Example. To estimate the integral of x^2 on [0, 2] with four strips of width 0.5 and left-hand heights: 0.5 * (0 + 0.25 + 1 + 2.25) = 1.75, against the exact 2.67.

Related: Integral, Limit, Antiderivative

An adaptive optimiser that divides each step by the square root of an exponentially decaying average of recent squared gradients. Unlike AdaGrad it forgets old history, so the effective learning rate does not vanish, making it suitable for non-stationary deep-learning objectives.

Example. With decay 0.9, running average s = 0.9 * s + 0.1 * g^2, and the update is lr * g / (sqrt(s) + epsilon).

Related: AdaGrad, Adam

A stationary point that curves upward in some directions and downward in others, so it is neither a minimum nor a maximum. In high-dimensional non-convex problems such as neural network training, these are far more common than poor local minima.

Example. f(x, y) = x^2 - y^2 has zero gradient at the origin, yet moving along x raises f while moving along y lowers it.

Related: Stationary point, Hessian matrix, Second-order condition, Local minimum

A straight line through two distinct points on a curve. Its slope is the average rate of change between them. As the two points move together, it approaches the tangent line, which is how the derivative is built from a limit.

Example. On f(x) = x^2, the secant through x = 1 and x = 3 passes through (1, 1) and (3, 9), so its slope is (9 - 1) / (3 - 1) = 4.

Related: Tangent line, Rate of change, Finite difference

Approximating a function near a point using its value, gradient and Hessian: f(x + d) is roughly f(x) + g.d + 0.5 * d.H.d. The quadratic term captures curvature, letting an optimiser choose better step sizes and directions than the gradient alone.

Example. For f(x) = x^4 at x = 1, the approximation is 1 + 4d + 6d^2; for d = 0.1 it gives 1.46 against the true value 1.4641.

Related: Taylor series, Hessian matrix, Newton's method

A test using the Hessian at a stationary point: if it is positive definite (all eigenvalues positive) the point is a strict local minimum, if negative definite a local maximum, and if it has mixed signs a saddle point.

Example. For f(x, y) = x^2 + 3y^2 the Hessian is diag(2, 6), both positive, so the stationary point at the origin is a minimum.

Related: First-order condition, Hessian matrix, Saddle point

A training method that seeks parameters lying in neighbourhoods with uniformly low loss. Each step first perturbs the weights in the direction that raises the loss most, then takes the gradient there, which steers optimisation toward flat minima.

Example. Each SAM update needs two gradient computations, one at the current weights and one at the perturbed weights, roughly doubling the cost per step.

Related: Flat minimum, Stochastic gradient descent

A randomised search that explores solutions by sometimes accepting worse moves with a probability that falls as a temperature parameter is lowered, in analogy with slowly cooling metal. It can escape local minima on non-differentiable problems.

Example. When routing a delivery tour, a swap that lengthens the route by 2 km may be accepted early in the search but almost never near the end.

Related: Genetic algorithm, Local minimum, Derivative-free optimisation

A point where the gradient of a function is zero, so the surface is momentarily flat. It may be a minimum, a maximum or a saddle point. Setting the gradient to zero is the first step in finding optima analytically.

Example. f(x) = x^3 - 3x has f'(x) = 3x^2 - 3 = 0 at x = -1 and x = 1, so both are stationary points.

Related: Saddle point, First-order condition, Local minimum

A learning-rate schedule that cuts the rate by a fixed factor after a set number of epochs or steps. It is simple and effective, producing a staircase-shaped curve as training progresses and letting weights settle precisely.

Example. Learning rate 0.1 is halved every 10 epochs: 0.1, 0.05, 0.025, 0.0125.

Related: Learning rate schedule, Cosine annealing, Learning rate

A variant that updates parameters using the gradient computed from a single randomly chosen training example (or, loosely, a small random batch). Updates are cheap and noisy; the noise can help escape shallow traps and makes learning on huge datasets feasible.

Example. For linear regression on 1 million rows, pick one row, compute its error gradient, adjust the weights, and repeat with another random row.

Related: Mini-batch gradient descent, Batch gradient descent, Momentum

A property meaning a function curves upward at least as fast as a quadratic with some positive constant mu: its Hessian has all eigenvalues at least mu. It guarantees a unique minimum and lets gradient descent converge at a geometric rate.

Example. f(x) = 3x^2 is strongly convex with mu = 6, since its second derivative is always 6.

Related: Convex function, L-smoothness, Convergence rate, Condition number

A generalisation of the gradient for convex functions that are not differentiable everywhere: the slope of any line that touches the graph at the point and stays beneath it. At a kink there are many valid choices.

Example. For f(x) = |x| at x = 0, any value between -1 and 1 is a subgradient; frameworks typically use 0 for ReLU at 0.

Related: Differentiability, Subgradient method, Convex function

A gradient-descent-like algorithm for convex functions with kinks, using any subgradient in place of the gradient and diminishing step sizes. It converges slowly but is simple and handles non-smooth objectives such as hinge loss.

Example. Minimising |x - 2| from x = 0 with steps 1/k: each step moves toward 2 by the diminishing size.

Related: Subgradient, Gradient descent, Proximal gradient method

A differentiation rule saying the derivative of a sum of functions is the sum of their derivatives, and constant multiples pass straight through. It lets you differentiate a long expression piece by piece, which is how per-example loss terms are handled.

Example. d/dx of (4x^3 + 2 sin x) = 12x^2 + 2 cos x. A total loss summed over 100 samples has gradient equal to the sum of 100 per-sample gradients.

Related: Power rule, Product rule, Derivative

Producing an exact algebraic formula for a derivative by applying calculus rules mechanically to an expression, as computer algebra systems like SymPy do. The results are exact but can grow enormously large for deep, nested functions.

Example. SymPy computes diff(x**2 * sin(x), x) and returns x**2*cos(x) + 2*x*sin(x).

Related: Automatic differentiation, Numerical differentiation, Chain rule

The straight line that just touches a curve at a single point and has the same slope as the curve there. Its slope equals the derivative at that point. It gives the best straight-line approximation of the function near that point.

Example. For f(x) = x^2 at x = 3, the tangent line is y = 6x - 9, which touches the curve at (3, 9) with slope 6.

Related: Derivative, Secant line, Linear approximation

An expansion of a smooth function around a point as an infinite sum of terms built from its derivatives at that point: f(a) + f'(a)(x - a) + f''(a)(x - a)^2 / 2! + ... Truncating it gives polynomial approximations used to derive optimisers and analyse errors.

Example. Around 0, e^x = 1 + x + x^2/2 + x^3/6 + ..., so e^0.1 is about 1.10517 using the first four terms.

Related: Linear approximation, Second-order approximation, Hessian matrix

The derivative of a function with respect to a variable when all its inputs may themselves depend on that variable. It adds up the direct effect and every indirect effect flowing through the other inputs.

Example. If f(x, y) = x*y with x = t and y = t^2, then f = t^3 and df/dt = 3t^2, which equals y*1 + x*2t = t^2 + 2t^2.

Related: Partial derivative, Multivariable chain rule, Chain rule

An optimisation approach that builds a simple model, usually quadratic, of the objective and trusts it only within a bounded radius of the current point. The radius grows or shrinks depending on how well predictions match reality.

Example. If the model predicts a loss drop of 2.0 but the true drop is 1.9, the radius may be enlarged; if the true drop is negative, the step is rejected and the radius shrunk.

Related: Newton's method, Levenberg-Marquardt algorithm, Line search

Optimisation in which the variables may take any real values, with no restrictions. Most neural-network training is treated this way, and it is solved by following gradient information to a point where the slope vanishes.

Example. Minimising f(w) = (w - 5)^2 over all real w gives w = 5.

Related: Constrained optimisation, Gradient descent, Stationary point

A family of techniques such as SVRG and SAGA that lower the noise of stochastic gradient estimates by using stored or periodically computed full gradients as a reference. They allow larger steps and faster, linear convergence on convex finite-sum problems.

Example. SVRG recomputes the full-data gradient once per epoch and corrects each mini-batch gradient using the difference from that snapshot.

Related: Stochastic gradient descent, Convergence rate

The product of a row vector with a Jacobian matrix, giving how a weighted combination of outputs changes with each input. Reverse-mode differentiation computes it directly without ever building the full Jacobian, which keeps memory small.

Example. In PyTorch, y.backward(gradient=v) computes v^T J for a vector output y, where v holds upstream sensitivities.

Related: Jacobian matrix, Reverse-mode automatic differentiation, Jacobian-vector product

A schedule technique that periodically resets the learning rate to a high value and decays it again, often with cosine annealing, so training can leave one basin and explore others. Each cycle can end with a good candidate solution.

Example. A cosine schedule of 10 epochs is repeated three times, each cycle starting again from lr = 0.1 and decaying to near zero.

Related: Cosine annealing, Learning rate schedule