Linear algebra

A linear transformation followed by adding a fixed vector, so it can also shift points. A neural network layer computes one, W x + b, before applying its non-linear activation.

Example. y = [[2,0],[0,3]] x + [1, 1] stretches the axes and then moves every point by [1, 1].

Related: Linear transformation, Homogeneous coordinates, Matrix-vector product

A matrix formed by attaching the right-hand-side column of a linear system to its coefficient matrix, so row operations can be applied to the whole system at once during elimination.

Example. For x + 2y = 5 and 3x - y = 1, the augmented matrix is [[1,2 | 5],[3,-1 | 1]].

Related: System of linear equations, Gaussian elimination, Row echelon form

A minimal set of linearly independent vectors that spans a whole space, so every vector in the space can be written in exactly one way as a combination of them. Choosing a basis is choosing a coordinate system.

Example. [1, 1] and [1, -1] form a basis of the 2-D plane; the vector [3, 1] equals 2*[1, 1] + 1*[1, -1].

Related: Standard basis, Span, Linear independence, Change of basis

Multiplying many pairs of matrices at once, where a leading batch axis picks out each pair. Deep learning frameworks use it to run the same layer on a whole batch of inputs, or on many attention heads, in one efficient GPU call.

Example. Multiplying tensors of shape (8, 64, 32) and (8, 32, 16) gives (8, 64, 16): eight independent matrix products.

Related: Matrix multiplication, Tensor, Tensor contraction

Re-expressing vectors and transformations using a different set of basis vectors. The same object gets new coordinates through a matrix whose columns are the new basis written in the old coordinates.

Example. The vector [3, 1] in the standard basis has coordinates [2, 1] in the basis {[1, 1], [1, -1]}.

Related: Basis, Linear transformation, Diagonalisation

The polynomial obtained as the determinant of (A minus lambda times the identity). Its roots are exactly the eigenvalues of A, which gives the classical hand method of finding them.

Example. For [[2,1],[1,2]] the polynomial is (2 - lambda)^2 - 1, with roots 1 and 3.

Related: Eigenvalue, Determinant, Trace

A factorisation of a symmetric positive definite matrix into a lower triangular matrix times its own transpose. It takes about half the work of general elimination and is used to solve equations and sample from multivariate normal distributions.

Example. For [[4,2],[2,3]], L = [[2,0],[1,1.414]] and L times L transposed reproduces the matrix.

Related: Positive definite matrix, LU decomposition, Covariance matrix

The set of all outputs a matrix can produce, equal to all linear combinations of its columns. The system A x = b has a solution exactly when b lies in this set, and its dimension is the rank.

Example. For A = [[1,0],[0,1],[0,0]], the column space is the x-y plane inside 3-D space.

Related: Row space, Null space, Rank, Span

A vector written as a single column, so an n x 1 matrix. Textbooks treat this as the default orientation, so a matrix acting on a vector is written A x with the vector on the right.

Example. With a 2 x 3 matrix A and column vector x of 3 numbers, A x is a column vector of 2 numbers.

Related: Row vector, Transpose, Matrix-vector product

A score between -1 and 1 for how closely two vectors point in the same direction, ignoring their lengths. It equals the dot product divided by the product of the two norms, and is widely used to compare text embeddings.

Example. For a = [1, 0] and b = [1, 1], cos = (1) / (1 * 1.414) = 0.707, an angle of 45 degrees.

Related: Dot product, L2 norm, Euclidean distance, Unit vector

A square symmetric table in which entry (i, j) is the covariance between feature i and feature j, and diagonal entries are the variances. Its eigenvectors point along the directions of greatest spread in the data.

Example. For heights and weights of 200 students, the 2 x 2 covariance matrix shows each variance on the diagonal and how they vary together off it.

Related: Symmetric matrix, Positive semi-definite matrix, Eigenvector, Whitening transformation

An operation on two 3-D vectors that returns a third vector perpendicular to both, whose length equals the area of the parallelogram they form. It is used in geometry and graphics but is defined only in three (and seven) dimensions.

Example. [1, 0, 0] x [0, 1, 0] = [0, 0, 1].

Related: Dot product, Orthogonal vectors, Determinant

A matrix stored as a full grid, where every entry is kept explicitly whether or not it is zero. It suits data with mostly non-zero values, such as neural network weights, and works well with GPU hardware.

Example. A 1024 x 1024 weight matrix is stored as 1,048,576 numbers regardless of their values.

Related: Sparse matrix, Matrix, Tensor

The matrix of input features in a regression problem, with one row per observation and one column per feature, often plus a column of ones for the intercept. Model fitting formulas such as the normal equations are written in terms of it.

Example. For 3 houses with area and rooms, X = [[1,1200,3],[1,900,2],[1,1500,4]], the first column being the intercept.

Related: Normal equations, Matrix, Least squares, Projection matrix

A single number computed from a square matrix that tells how much the matrix scales area or volume, with a negative sign if it flips orientation. A zero value means the matrix collapses space and cannot be inverted.

Example. det([[1,2],[3,4]]) = 1*4 - 2*3 = -2, so areas double in size and orientation flips.

Related: Singular matrix, Inverse matrix, Eigenvalue, Trace

A square matrix whose only non-zero entries lie on the main diagonal. Multiplying by it simply scales each coordinate separately, so products, powers and inverses are cheap to compute.

Example. diag(2, 3) turns [4, 5] into [8, 15].

Related: Identity matrix, Eigendecomposition, Triangular matrix

The process of expressing a square matrix as a diagonal matrix in a well-chosen basis of its eigenvectors. Not every matrix can be diagonalised, but when it can, complex repeated actions reduce to simple per-axis scaling.

Example. A 2 x 2 matrix with two distinct eigenvalues always has two independent eigenvectors and can be diagonalised.

Related: Eigendecomposition, Change of basis, Eigenvector

The number of vectors in any basis of a space, which equals the number of independent directions it has. For a vector, it is often used loosely to mean how many components the vector contains.

Example. A sentence embedding with 768 numbers lives in a 768-dimensional space; a plane inside 3-D space has dimension 2.

Related: Basis, Vector space, Rank

An operation on two equal-length vectors that multiplies matching components and sums the results into one number. It measures how strongly two vectors point the same way, and it is the core of neuron computations, attention scores and similarity search.

Example. [1, 2, 3] . [4, 5, 6] = 1*4 + 2*5 + 3*6 = 32.

Related: Inner product, Cosine similarity, Orthogonal vectors, Matrix multiplication

Writing a diagonalisable square matrix as V D V^-1, where columns of V are eigenvectors and D is a diagonal matrix of eigenvalues. It shows the matrix as a stretch along its natural axes and makes powers of the matrix easy to compute.

Example. For symmetric A, A = Q D Q^T; A^10 = Q D^10 Q^T needs only raising the diagonal entries to the tenth power.

Related: Eigenvalue, Diagonalisation, Spectral theorem, Singular value decomposition

The set of all eigenvectors sharing one eigenvalue, together with the zero vector. It is a subspace, and its dimension counts how many independent directions are scaled by that same factor.

Example. For the identity matrix in 3-D, every vector is scaled by 1, so the eigenspace for eigenvalue 1 is the entire 3-D space.

Related: Eigenvector, Subspace, Eigenvalue

A number by which a matrix stretches a special direction that it does not turn. A negative value also flips the direction, and a value of zero collapses it. Eigenvalues reveal the variance along principal directions, stability of systems, and the shape of loss surfaces.

Example. For [[2,0],[0,3]], the eigenvalues are 2 and 3, because the x-axis is stretched by 2 and the y-axis by 3.

Related: Eigenvector, Eigendecomposition, Characteristic polynomial, Determinant

A non-zero vector that a matrix only rescales, never rotates: A v = lambda v, where lambda is its eigenvalue. Eigenvectors mark the natural axes of a transformation, such as the directions of greatest variance in a covariance matrix.

Example. For A = [[2,0],[0,3]], v = [1, 0] gives A v = [2, 0] = 2 v, so v is an eigenvector with eigenvalue 2.

Related: Eigenvalue, Eigendecomposition, Eigenspace, Covariance matrix

A compact notation that describes tensor products, contractions and transposes by naming each axis with a letter and summing over any letter that repeats. Array libraries offer it through an einsum function.

Example. np.einsum('ij,jk->ik', A, B) performs ordinary matrix multiplication of A and B.

Related: Tensor contraction, Matrix multiplication, Tensor

The size of a matrix computed as the square root of the sum of the squares of all its entries, as if it were one long vector. It is the usual measure for weight-decay penalties and for the error of matrix approximations.

Example. For [[1,2],[3,4]], the norm is sqrt(1 + 4 + 9 + 16) = sqrt(30), about 5.48.

Related: Matrix norm, L2 norm, Trace, Low-rank approximation

A matrix whose rank equals the smaller of its row and column counts, meaning it has as much independent information as its shape allows. A square full-rank matrix is invertible.

Example. A 100 x 5 feature matrix whose 5 columns are independent has rank 5, so it is full rank.

Related: Rank, Singular matrix, Multicollinearity

A step-by-step method for solving linear systems by adding multiples of rows to other rows and swapping rows until the matrix becomes triangular, then finding unknowns from the bottom up. It is the basis of many numerical solvers.

Example. Subtracting 3 times row 1 from row 2 in [[1,2],[3,-1]] leaves [0, -7], from which y and then x follow.

Related: Row echelon form, LU decomposition, System of linear equations, Pivot

The matrix of all pairwise dot products among a set of vectors, entry (i, j) being vector i dotted with vector j. It is symmetric and positive semi-definite, and kernel methods use it to hold all similarities between training examples.

Example. For three vectors, the Gram matrix is 3 x 3; its diagonal holds each vector's squared length.

Related: Dot product, Positive semi-definite matrix, Symmetric matrix

A procedure that turns a list of independent vectors into an orthonormal list spanning the same space: take each vector, subtract its projections onto those already processed, and scale the remainder to length 1. It underlies QR decomposition.

Example. From [1, 1] and [1, 0], the first becomes [0.707, 0.707]; subtracting its projection from [1, 0] and normalising gives [0.707, -0.707].

Related: Orthonormal basis, QR decomposition, Projection

Element-wise multiplication of two matrices or vectors of the same shape: each entry is multiplied by the entry in the same position. It is used for masking and gating in neural networks and differs from ordinary matrix multiplication.

Example. [1, 2, 3] element-wise times [4, 5, 6] gives [4, 10, 18].

Related: Matrix multiplication, Kronecker product, Broadcasting

A way of writing points with one extra coordinate so that shifts as well as rotations and scalings become single matrix multiplications. It is common in graphics and computer vision and is how bias terms can be folded into weights.

Example. The 2-D point (3, 4) is written as [3, 4, 1], and a translation by (5, 2) becomes a 3 x 3 matrix acting on it.

Related: Affine transformation, Rotation matrix, Linear transformation

A flat subspace one dimension smaller than the space around it, defined by w . x = c for a normal vector w. In machine learning it acts as the boundary that a linear classifier draws between two classes.

Example. In 2-D, the line x + 2y = 4 is a hyperplane; in 3-D, a plane; a linear classifier with 50 features uses a 49-dimensional one.

Related: Dot product, Linear transformation, Vector

A square matrix with ones on the main diagonal and zeros elsewhere. Multiplying any compatible matrix or vector by it leaves that object unchanged, so it plays the role that the number 1 plays in ordinary multiplication.

Example. For I = [[1,0],[0,1]] and v = [7, 9], I v = [7, 9].

Related: Inverse matrix, Diagonal matrix, Standard basis

A matrix that is nearly singular, having a very large condition number, so that solving equations with it is unstable and rounding errors get amplified. Highly correlated features are a common cause in regression.

Example. Two features that are 99.99 percent correlated give a design matrix whose near-dependent columns make coefficients swing wildly.

Related: Condition number, Singular matrix, Multicollinearity

The size of a vector defined as the largest absolute value among its components. It is the limit of the Lp family as p grows and is used to bound the worst-case error in any single coordinate, for example in gradient clipping.

Example. ||[3, -7, 2]||_inf = 7.

Related: Lp norm, L1 norm, Norm

A generalisation of the dot product to any vector space: a rule taking two vectors to a number that is linear in each, symmetric, and positive for any vector paired with itself. It lets one define length and angle in settings beyond ordinary coordinates.

Example. For functions on an interval, the integral of f(x)*g(x) is an inner product; for ordinary vectors it reduces to the dot product.

Related: Dot product, Inner product space, Norm, Orthogonal vectors

For a square matrix A, the matrix that undoes its effect: multiplying A by it in either order gives the identity matrix. It exists only when A does not collapse any direction, and it lets you solve A x = b as x = inverse times b.

Example. For A = [[2,0],[0,4]], the inverse is [[0.5,0],[0,0.25]], and their product is the identity.

Related: Identity matrix, Singular matrix, Determinant, System of linear equations

An operation that builds a large block matrix by multiplying every entry of one matrix by the whole of another. It appears in structured covariance models and approximations of curvature in large networks.

Example. For a 2 x 2 matrix A and a 3 x 3 matrix B, the product is a 6 x 6 matrix made of 4 blocks, each block being an entry of A times B.

Related: Hadamard product, Block matrix, Outer product

The size of a vector measured by adding up the absolute values of its components. It gives the distance you would walk along axes only, and as a penalty it pushes many weights to exactly zero, producing sparse models.

Example. ||[3, -4, 2]||_1 = 3 + 4 + 2 = 9.

Related: L2 norm, Manhattan distance, Norm, Sparse vector

The straight-line length of a vector, found by squaring each component, adding them and taking the square root. It matches ordinary geometric distance and is used in normalisation, weight decay and Euclidean distance.

Example. ||[3, 4]||_2 = sqrt(9 + 16) = 5.

Related: L1 norm, Euclidean distance, Unit vector, Frobenius norm

A method for finding the vector x that makes A x as close as possible to b by minimising the sum of squared differences, used when a system has more equations than unknowns. It is the mathematical core of linear regression.

Example. Fitting a line through 50 noisy points means finding the slope and intercept that minimise the total squared vertical gaps.

Related: Normal equations, Pseudoinverse, Projection, QR decomposition

A new vector formed by multiplying each vector in a set by a number and adding the results. It is the basic building step of linear algebra, and a neural network neuron computes one when it weights and sums its inputs.

Example. With v1 = [1, 0] and v2 = [0, 1], the combination 3*v1 + 2*v2 equals [3, 2].

Related: Span, Linear independence, Basis, Matrix-vector product

A property of a set of vectors where none can be built as a combination of the others; the only way to combine them into the zero vector is to use zero for every multiplier. Dependent vectors carry redundant information, which is why correlated features can destabilise models.

Example. [1, 0], [0, 1] and [1, 1] are dependent in 2-D because [1, 1] = [1, 0] + [0, 1]; the first two alone are independent.

Related: Basis, Span, Rank, Multicollinearity

A function between vector spaces that preserves addition and scaling: transforming a sum equals summing the transforms, and scaling before or after gives the same result. Every such function on coordinate vectors is multiplication by some matrix.

Example. Rotating every point of the plane by 30 degrees is linear and is represented by a 2 x 2 rotation matrix.

Related: Matrix, Matrix-vector product, Affine transformation, Change of basis

Replacing a matrix by another of much smaller rank that stays as close as possible to the original, typically built from the top singular values. It compresses data and reduces parameters, and underlies fine-tuning methods that store updates as products of thin matrices.

Example. Approximating a 1000 x 1000 matrix by two factors of 1000 x 10 and 10 x 1000 stores 20,000 numbers instead of 1,000,000.

Related: Singular value decomposition, Truncated SVD, Rank, Matrix factorisation

A family of vector sizes defined as the sum of absolute components each raised to the power p, then taking the p-th root, for p at least 1. Setting p to 1 gives the sum of absolute values and p to 2 gives ordinary length; larger p emphasises the biggest components.

Example. With p = 3 and v = [1, 2], the norm is (1^3 + 2^3)^(1/3) = 9^(1/3), about 2.08.

Related: L1 norm, L2 norm, Infinity norm

A factorisation of a square matrix into a lower triangular matrix L and an upper triangular matrix U, usually with row swaps. Once done, many systems with the same matrix and different right sides are solved fast by two triangular solves.

Example. A = L U lets you solve A x = b by first solving L y = b and then U x = y.

Related: Gaussian elimination, Triangular matrix, Cholesky decomposition, QR decomposition

A rectangular grid of numbers arranged in rows and columns. It can store a whole dataset (one row per example), a set of weights in a neural network layer, or represent a linear transformation that moves and reshapes vectors.

Example. A dataset of 500 customers with 12 features is a 500 x 12 matrix; the weight matrix of a layer mapping 12 inputs to 4 outputs is 12 x 4.

Related: Vector, Tensor, Matrix multiplication, Linear transformation

Combining an m x n matrix with an n x p matrix to give an m x p matrix, where each entry is the dot product of a row from the first and a column from the second. It represents applying one transformation after another and is the main workload of neural networks.

Example. [[1,2],[3,4]] times [[5],[6]] = [[1*5+2*6],[3*5+4*6]] = [[17],[39]].

Related: Dot product, Matrix-vector product, Linear transformation, Batch matrix multiplication

A function giving the overall size of a matrix, satisfying the rules for norms. Different versions measure total entry size or the maximum stretch the matrix applies to a vector, and they are used to bound errors and regularise weights.

Example. The matrix [[1,2],[3,4]] has a Frobenius size of sqrt(1 + 4 + 9 + 16), about 5.48.

Related: Frobenius norm, Spectral norm, Norm

The result of multiplying a matrix by a vector: each output entry is the dot product of a matrix row with the vector, or equivalently a linear combination of the matrix columns weighted by the vector entries. It is how a dense neural layer computes its outputs.

Example. W = [[1,0,2],[0,1,1]] and x = [3,4,5] give W x = [3+10, 4+5] = [13, 9].

Related: Matrix multiplication, Linear combination, Linear transformation

A function that assigns a non-negative length or size to a vector, being zero only for the zero vector, scaling with multiplication, and obeying the triangle inequality. Different norms measure size differently and appear in distances and regularisation penalties.

Example. For v = [3, 4], the L2 norm is 5 and the L1 norm is 7.

Related: L1 norm, L2 norm, Lp norm, Matrix norm

The equations X^T X b = X^T y whose solution gives the least-squares coefficients. They express that the residual must be perpendicular to every column of X, but forming X^T X can worsen numerical problems.

Example. For X with columns [1, 1, 1] and [1, 2, 3] and y = [2, 3, 5], solving the equations gives intercept 0.33 and slope 1.5.

Related: Least squares, Projection, Condition number, Design matrix

The set of all vectors that a matrix sends to the zero vector. Its size shows how much information the matrix destroys, and it exists nontrivially exactly when the columns are dependent.

Example. For A = [[1,2],[2,4]], the vector [2, -1] gives A x = [0, 0], so it lies in the null space.

Related: Column space, Rank-nullity theorem, Singular matrix, Subspace

A square matrix whose columns are mutually perpendicular unit vectors, so its transpose is also its inverse. Multiplying by it rotates or reflects vectors without changing their lengths or angles.

Example. A 2-D rotation by 90 degrees, [[0,-1],[1,0]], is orthogonal: it turns [1, 0] into [0, 1] and both have length 1.

Related: Orthonormal basis, Rotation matrix, Transpose, Inverse matrix

A projection in which the leftover difference between the vector and its image is perpendicular to the target subspace, making the image the closest point in it. Least squares fitting is exactly this operation applied to the target vector.

Example. The orthogonal projection of b = [1, 2, 3] onto the x-y plane is [1, 2, 0], and the leftover [0, 0, 3] is perpendicular to the plane.

Related: Projection, Projection matrix, Least squares, Orthogonal complement

Two vectors whose dot product is zero, meaning they meet at a right angle and carry no shared direction. Orthogonal directions are independent of each other, which makes computations and coordinate systems cleaner.

Example. [1, 2] . [2, -1] = 2 - 2 = 0, so these vectors are orthogonal.

Related: Dot product, Orthonormal basis, Orthogonal matrix, Projection

A basis whose vectors are all of unit length and mutually orthogonal. Coordinates in such a basis are found simply by taking dot products, which makes them numerically stable and easy to work with.

Example. [0.6, 0.8] and [-0.8, 0.6] are both length 1, have dot product 0, and so form an orthonormal basis of the plane.

Related: Basis, Orthogonal matrix, Unit vector, Gram-Schmidt process

An operation that multiplies a column vector of length m by a row vector of length n to give an m x n matrix whose entry (i, j) is the i-th component times the j-th. Every outer product has rank one and is used in gradient updates and low-rank models.

Example. [1, 2] as a column times [3, 4, 5] as a row gives [[3, 4, 5], [6, 8, 10]].

Related: Inner product, Rank-one matrix, Matrix multiplication

A square matrix of zeros and ones with exactly one 1 in each row and column. Multiplying by it reorders the rows or entries of a matrix or vector, and it is used to record row swaps in decompositions.

Example. [[0,1],[1,0]] times [7, 9] gives [9, 7].

Related: Orthogonal matrix, LU decomposition, Identity matrix

The first non-zero entry in a row used to eliminate the entries below it during row reduction. The number of pivots equals the rank, and picking large pivots keeps calculations numerically stable.

Example. In [[2,1],[4,3]], using 2 as pivot lets you subtract 2 times row 1 from row 2 to get [0, 1].

Related: Gaussian elimination, Row echelon form, Rank

A symmetric matrix A for which x^T A x is greater than zero for every non-zero vector x; equivalently all its eigenvalues are positive. It describes a bowl-shaped surface with a single minimum and can be inverted.

Example. [[2,0],[0,3]] is positive definite: x^T A x = 2*x1^2 + 3*x2^2 is positive unless x is the zero vector.

Related: Positive semi-definite matrix, Quadratic form, Cholesky decomposition, Eigenvalue

A symmetric matrix A for which x^T A x is never negative, so all eigenvalues are zero or positive. Covariance matrices and kernel matrices always have this property, and it marks a valid, non-negative measure of spread.

Example. The covariance matrix [[1,1],[1,1]] of two identical features is positive semi-definite, with eigenvalues 2 and 0.

Related: Positive definite matrix, Covariance matrix, Gram matrix, Eigenvalue

An iterative method that multiplies a vector by a matrix over and over, rescaling each time, until it settles on the eigenvector with the largest-magnitude eigenvalue. It is simple and works on huge sparse matrices, as in ranking web pages.

Example. Starting from [1, 1] and applying [[2,1],[1,2]] repeatedly with rescaling converges to [1, 1] scaled, with eigenvalue 3.

Related: Eigenvector, Spectral radius, Sparse matrix

The operation of dropping a vector onto a line or subspace to find its closest point there. The result is the part of the vector lying along that subspace, and the leftover part is perpendicular to it.

Example. Projecting [3, 4] onto the x-axis gives [3, 0].

Related: Orthogonal projection, Projection matrix, Least squares, Dot product

A matrix P that performs a projection when it multiplies a vector, satisfying P times P equals P because projecting twice changes nothing. For projection onto the columns of X it equals X (X^T X)^-1 X^T, the 'hat matrix' of regression.

Example. P = [[1,0],[0,0]] projects any 2-D vector onto the x-axis: P [3, 4] = [3, 0].

Related: Projection, Least squares, Idempotent matrix, Design matrix

A generalisation of the inverse that exists for any matrix, including non-square or singular ones. It gives the least-squares solution of A x = b with the smallest norm, and is usually computed with the SVD.

Example. For a tall 100 x 3 data matrix X, the pseudoinverse times y yields the linear regression coefficients.

Related: Least squares, Singular value decomposition, Inverse matrix, Normal equations

A factorisation of a matrix into an orthogonal matrix Q and an upper triangular matrix R. It is numerically stable and is used to solve least-squares problems and to compute eigenvalues.

Example. For a tall 100 x 3 data matrix X, X = Q R with Q having 3 orthonormal columns; regression coefficients solve R b = Q^T y.

Related: Gram-Schmidt process, Orthogonal matrix, Least squares, Triangular matrix

An expression x^T A x that turns a vector into a single number by multiplying its components in pairs with weights from a matrix. It describes the shape of quadratic loss surfaces, and its sign pattern shows whether they curve up or down.

Example. For A = [[1,0],[0,4]], x^T A x = x1^2 + 4*x2^2, so x = [2, 1] gives 8.

Related: Positive definite matrix, Symmetric matrix, Eigenvalue

The number of linearly independent columns (equivalently rows) in a matrix, telling how many dimensions its output actually covers. A rank below the full possible value signals redundancy or collapsed information.

Example. [[1,2,3],[2,4,6],[1,0,1]] has rank 2, because its second row is twice the first.

Related: Full rank, Linear independence, Column space, Low-rank approximation

A rule stating that for a matrix with n columns, the rank plus the dimension of the null space equals n. It says every input dimension is either carried through to the output or lost.

Example. A 3 x 5 matrix of rank 3 has a null space of dimension 5 - 3 = 2.

Related: Rank, Null space, Dimension

A matrix that can be written as a single column vector times a single row vector, so all its rows are multiples of one another. It is the simplest non-zero matrix and the building block of low-rank and SVD sums.

Example. The matrix [[2,4],[3,6]] equals [2, 3] as a column times [1, 2] as a row, so its rank is 1.

Related: Outer product, Rank, Singular value decomposition, Low-rank approximation

A refined row echelon shape in which each pivot equals 1 and is the only non-zero entry in its column. Every matrix has exactly one such form, so solutions to a system can be read off directly.

Example. [[1,0,2],[0,1,3]] means x = 2 and y = 3 for the system it came from.

Related: Row echelon form, Gaussian elimination, System of linear equations

An orthogonal matrix with determinant 1 that turns vectors around the origin by a fixed angle without changing their lengths. In 2-D it is [[cos t, -sin t],[sin t, cos t]] for angle t.

Example. With t = 90 degrees the matrix is [[0,-1],[1,0]] and sends [1, 0] to [0, 1].

Related: Orthogonal matrix, Linear transformation, Determinant

A matrix shape reached by row operations in which every row starts further right than the row above it and any all-zero rows sit at the bottom. It makes the rank readable directly from the count of non-zero rows.

Example. [[1,2,3],[0,1,4],[0,0,0]] is in row echelon form and has rank 2.

Related: Gaussian elimination, Reduced row echelon form, Pivot, Rank

The set of all linear combinations of a matrix's rows. Its dimension equals that of the column space, so row and column counts of independent directions always agree.

Example. For [[1,2],[2,4]], every row is a multiple of [1, 2], so the row space is that single line.

Related: Column space, Rank, Span

A vector written as a single row, so a 1 x n matrix. Row vectors are the natural way to store one data example when a dataset matrix has one example per row, and their orientation matters when multiplying with matrices.

Example. x = [2, 5, 7] as a 1 x 3 row can be multiplied on the left of a 3 x 2 matrix to give a 1 x 2 row.

Related: Column vector, Transpose, Matrix multiplication

A single number, as opposed to a list or grid of numbers. Scalars are the ordinary quantities you multiply vectors and matrices by to stretch or shrink them, and they are also the results of operations such as a dot product or a loss calculation.

Example. A learning rate of 0.01 is a scalar: multiplying a gradient vector by it shrinks every component of the vector by the same factor.

Related: Vector, Scalar multiplication, Tensor

Multiplying every component of a vector or matrix by the same number. It stretches or shrinks the arrow, and a negative number also flips its direction without changing the line it lies on.

Example. 2.5 * [4, -2] = [10, -5].

Related: Scalar, Vector addition, Linear combination

A square matrix that has no inverse because it squashes some direction to zero; its determinant is zero and its columns are linearly dependent. Attempting to solve equations with it gives no unique answer or a numerically unstable one.

Example. [[1,2],[2,4]] is singular: its second row is twice its first, and its determinant is 1*4 - 2*2 = 0.

Related: Inverse matrix, Determinant, Rank, Linear independence

A non-negative number in the decomposition of a matrix that measures how strongly the matrix stretches along one of its principal directions. It equals the square root of an eigenvalue of the matrix times its transpose, and small ones mark near-redundant directions.

Example. A matrix with singular values 10, 3 and 0.01 does almost all its stretching along the first two directions.

Related: Singular value decomposition, Eigenvalue, Rank, Condition number

A factorisation of any matrix into U S V^T, where U and V are orthogonal matrices and S is diagonal with non-negative singular values in decreasing order. It exposes the matrix's most important directions and is used for compression, denoising and recommendation.

Example. A 1000 x 300 matrix is split into U (1000 x 300), 300 singular values and V^T (300 x 300); keeping the top 20 gives a compact approximation.

Related: Singular value, Truncated SVD, Eigendecomposition, Low-rank approximation

The set of all vectors you can reach by taking linear combinations of a given list of vectors. It describes how much of a space those vectors cover: two vectors not on the same line span a plane, while two along the same line span only that line.

Example. [1, 0] and [0, 1] span the whole 2-D plane, but [1, 2] and [2, 4] span only the line through [1, 2].

Related: Linear combination, Basis, Linear independence, Subspace

A matrix in which most entries are zero, so it is stored by recording only the non-zero values and their positions. This saves memory and speeds up arithmetic for data such as word counts or user-item ratings.

Example. A 1,00,000 x 50,000 user-by-product matrix where each user bought about 20 items is over 99.9 percent zeros and is stored as a sparse matrix.

Related: Dense matrix, Sparse vector, Matrix factorisation

A vector in which most components are zero, stored compactly as index-value pairs. Bag-of-words and keyword-weight representations of text are sparse vectors, unlike the dense vectors produced by embedding models.

Example. A 50,000-word vocabulary vector for a tweet with 12 distinct words has 12 non-zero entries and 49,988 zeros.

Related: Sparse matrix, L1 norm, Vector

The matrix norm equal to the largest singular value, showing the maximum factor by which the matrix can lengthen any vector. It is used to control how sensitive a layer is to input changes, for example in spectral normalisation.

Example. A matrix with singular values 5, 2 and 0.3 has spectral norm 5, so no unit vector is stretched beyond length 5.

Related: Singular value, Matrix norm, Spectral radius

The largest absolute value among a square matrix's eigenvalues. It tells whether repeatedly applying the matrix makes vectors grow or shrink, so it governs stability of iterations and of recurrent network dynamics.

Example. A matrix with eigenvalues 0.9 and -0.5 has spectral radius 0.9, so repeated multiplication shrinks vectors towards zero.

Related: Eigenvalue, Power iteration, Spectral norm

A result stating that a real symmetric matrix can be written using orthogonal eigenvectors and real eigenvalues. It guarantees clean, stable eigen-analysis for covariance and Gram matrices, which is why those matrices behave so well in machine learning.

Example. Any covariance matrix of features can be written as Q D Q^T with orthogonal Q and non-negative diagonal D.

Related: Symmetric matrix, Eigendecomposition, Orthogonal matrix, Covariance matrix

The default basis made of unit vectors that have a single 1 and zeros elsewhere, each pointing along one coordinate axis. The numbers in an ordinary vector are its multipliers on these axis vectors, and one-hot encodings are standard basis vectors.

Example. In 3-D the standard basis is [1,0,0], [0,1,0], [0,0,1]; a one-hot label for class 2 of 3 is [0,1,0].

Related: Basis, One-hot encoding, Identity matrix

A smaller vector space living inside a bigger one: it contains the zero vector and stays closed under addition and scaling. Lines and planes through the origin are subspaces of ordinary 3-D space.

Example. In 3-D space, the set of all points with z = 0 is a subspace: the flat x-y plane through the origin.

Related: Vector space, Span, Column space, Null space

A square matrix that equals its own transpose, so entry (i, j) equals entry (j, i). Covariance matrices and Gram matrices are symmetric, and such matrices have real eigenvalues and orthogonal eigenvectors.

Example. [[2,1],[1,3]] is symmetric because the entry in row 1, column 2 equals the entry in row 2, column 1.

Related: Transpose, Covariance matrix, Spectral theorem, Positive definite matrix

A set of equations in which each unknown appears only to the first power and is multiplied by a constant. It can be written compactly as A x = b, and may have one solution, none, or infinitely many.

Example. x + 2y = 5 and 3x - y = 1 give x = 1, y = 2, written as [[1,2],[3,-1]] [x, y] = [5, 1].

Related: Gaussian elimination, Inverse matrix, Augmented matrix, Least squares

A general array of numbers with any number of axes: a scalar has zero axes, a vector one, a matrix two, and higher-order arrays three or more. Deep learning frameworks store inputs, weights and activations as tensors so operations can run in bulk on GPUs.

Example. A batch of 32 colour images of 224 x 224 pixels is a tensor of shape (32, 3, 224, 224).

Related: Scalar, Vector, Matrix, Tensor rank

Summing the products of entries of tensors over a shared axis, which reduces the total number of axes. Matrix multiplication, dot products and traces are all special cases.

Example. Contracting a (2, 3) tensor with a (3, 4) tensor over the shared axis of size 3 yields a (2, 4) tensor.

Related: Einstein summation, Matrix multiplication, Tensor

The number of axes of a tensor, also called its order. A scalar has rank 0, a vector rank 1, a matrix rank 2; this differs from matrix rank, which counts independent columns.

Example. A tensor of shape (32, 128, 768), such as a batch of token embeddings, has rank 3.

Related: Tensor, Rank, Dimension

The sum of the entries along the main diagonal of a square matrix. It also equals the sum of the matrix's eigenvalues and is unchanged by cyclic reordering of a product, which makes it handy in derivations and loss formulas.

Example. The trace of [[2,7],[1,5]] is 2 + 5 = 7.

Related: Determinant, Eigenvalue, Frobenius norm

The matrix obtained by swapping rows and columns, so entry (i, j) moves to (j, i). It turns row vectors into column vectors, appears in the formulas for dot products and least squares, and reverses the order when applied to a product.

Example. The transpose of [[1,2,3],[4,5,6]] is [[1,4],[2,5],[3,6]].

Related: Symmetric matrix, Row vector, Column vector, Orthogonal matrix

A square matrix with all entries below the diagonal zero (upper triangular) or all entries above it zero (lower triangular). Systems with this shape are solved quickly by back or forward substitution, which is why decompositions aim to produce them.

Example. [[2,3],[0,5]] is upper triangular; solving it for a given right side needs only two steps.

Related: LU decomposition, Gaussian elimination, Cholesky decomposition

A vector whose length equals exactly 1. Any non-zero vector can be turned into one by dividing it by its own norm, which keeps its direction but removes its magnitude; this is common before comparing embeddings.

Example. [3, 4] divided by its length 5 gives the unit vector [0.6, 0.8].

Related: Norm, Cosine similarity, Orthonormal basis

An ordered list of numbers that can be viewed as a point in space or as an arrow with a length and direction. In machine learning a vector usually holds the features of one example, or the coordinates of an embedding, so that distances and angles can be computed.

Example. A house described by [area in sq ft, bedrooms, age in years] = [1200, 3, 8] is a 3-dimensional vector.

Related: Scalar, Matrix, Dimension, Dot product

Combining two vectors of the same length by adding their matching components. Geometrically it places one arrow at the tip of the other and draws the arrow from the start to the end.

Example. [2, 5] + [1, -3] = [3, 2].

Related: Scalar multiplication, Linear combination, Vector

A collection of vectors closed under addition and scaling by numbers: adding any two members, or multiplying one by a scalar, always gives another member. Two-dimensional and three-dimensional coordinate space are the familiar examples, and the idea underlies feature spaces and embedding spaces.

Example. All vectors with 3 real components form a vector space; the sum of [1, 2, 3] and [4, 0, 1] is [5, 2, 4], still in the space.

Related: Subspace, Basis, Dimension, Span

A linear transformation applied to data so that its features become uncorrelated and have unit variance, making its covariance matrix the identity. It is usually computed from the eigendecomposition of the covariance matrix.

Example. After whitening, a 2-feature dataset stretched along a diagonal becomes a round cloud whose covariance matrix is [[1,0],[0,1]].

Related: Covariance matrix, Eigendecomposition, Linear transformation