Classical ML algorithms

Adaptive Boosting: a boosting algorithm that trains weak learners in turn, raising the weight of rows that were misclassified so the next learner focuses on them. Final predictions are a weighted vote in which more accurate learners count for more.

Example. With 1,000 rows, each starts at weight 0.001; after round one, wrongly classified rows might rise to 0.003 while correct ones fall.

Related: Boosting, Decision stump, Weak learner, Gradient boosting

The bottom-up form of hierarchical grouping. It starts with each point as its own cluster and repeatedly merges the two closest clusters until one remains or a target count is reached.

Example. With 6 points, 5 merge steps are needed to reach a single cluster.

Related: Hierarchical clustering, Linkage criterion, Dendrogram

Finding observations that differ strongly from the majority of data, without or with few labelled examples. It is used for fraud, faults and intrusion, and methods include isolation forests, density-based and distance-based techniques.

Example. Flagging a card transaction of 3 lakh from a new country on an account that usually spends under 5,000.

Related: Isolation forest, Local outlier factor, One-class SVM, DBSCAN

An algorithm for finding frequently occurring item sets. It builds larger sets from smaller ones and prunes any candidate containing an infrequent subset, since every subset of a frequent set must itself be frequent.

Example. If {milk, bread} is rare, {milk, bread, eggs} is skipped without counting.

Related: Association rule learning, Support, Confidence

A technique for finding co-occurrence patterns in transactions, expressed as rules like 'if items A are bought then B is likely'. Rules are scored by support, confidence and lift. It is used in market-basket analysis.

Example. Finding that customers who buy bread and eggs also buy butter in a supermarket's bills.

Related: Apriori algorithm, Support, Confidence, Lift

A method that trains many models on different random samples of the data, drawn with replacement, and averages their predictions or lets them vote. The averaging cancels out much of each model's random variation, lowering variance without raising bias much.

Example. Train 100 decision trees each on a resampled copy of 10,000 rows and average their outputs.

Related: Bootstrap sampling, Random forest, Ensemble learning, Variance

A graph in which nodes are variables and arrows show direct probabilistic dependence, with a probability table at each node. It represents the joint distribution compactly and allows reasoning from evidence to causes.

Example. A medical network linking 'smoker', 'lung disease' and 'cough' lets you update disease probability once cough is seen.

Related: Naive Bayes, Conditional independence, Bayes' theorem

A naive Bayes version for binary features that records whether each feature is present or absent, and explicitly penalises absence. It suits short texts where word presence matters more than counts.

Example. Representing each tweet as a 0/1 vector over a 5,000-word vocabulary and predicting sentiment.

Related: Naive Bayes, Multinomial naive Bayes, One-hot encoding

An ensemble strategy that builds models one after another, each concentrating on the examples earlier models got wrong, then adds them up with weights. It steadily reduces bias and usually wins on structured tabular data, but is more sensitive to noise than bagging.

Example. After tree 1 misclassifies 100 of 1,000 rows, tree 2 is trained with those rows emphasised.

Related: AdaBoost, Gradient boosting, Weak learner, Bagging

Creating a new dataset by drawing rows at random from the original one with replacement, so some rows repeat and others are left out. A sample of size n contains about 63% of the distinct original rows. It underlies bagging and uncertainty estimation.

Example. From 5 rows [A,B,C,D,E], one bootstrap sample could be [B,B,D,A,E], leaving C out.

Related: Bagging, Out-of-bag error, Random forest, Resampling

Quinlan's successor to ID3 that handles numeric features by choosing thresholds, copes with missing values and prunes the tree afterwards. It uses gain ratio, which corrects information gain's bias toward features with many distinct values.

Example. A feature like customer_id with unique values would look perfect by information gain, but gain ratio penalises it so C4.5 avoids that split.

Related: ID3, Information gain, Tree pruning, Decision tree

Classification and Regression Trees: a tree-building algorithm that always makes binary splits, uses Gini impurity for classes and squared error for numbers, and then prunes back. It is the basis of scikit-learn's tree models and most forest and boosting libraries.

Example. DecisionTreeClassifier in scikit-learn implements an optimised version of CART.

Related: Decision tree, Gini impurity, Tree pruning, Regression tree

A gradient boosting library from Yandex designed to handle categorical features well by encoding them with ordered statistics computed without target leakage. It uses symmetric trees and often gives good results with little tuning.

Example. Passing cat_features=['city','plan_type'] lets CatBoost encode text columns directly without one-hot encoding.

Related: Gradient boosting, XGBoost, LightGBM, Target encoding

The average position of a group of points, calculated by averaging each coordinate separately. In clustering it acts as the representative centre of a group, and algorithms such as k-means keep moving it as assignments change.

Example. The centroid of points (1,2), (3,4), (5,6) is (3,4).

Related: K-means, Nearest centroid classifier, Euclidean distance

An unsupervised task of grouping items so that those in the same group are more alike than those in different groups, with no labels provided. It is used for segmentation, exploration and summarising data.

Example. Grouping 1 lakh telecom customers into segments by usage without knowing segments beforehand.

Related: K-means, Hierarchical clustering, DBSCAN, Unsupervised learning

A relationship in which two variables carry no extra information about each other once a third is known: P(A, B | C) = P(A | C) * P(B | C). Naive Bayes relies on assuming it for all features given the class.

Example. Given that a message is spam, the presence of 'free' is treated as unrelated to the presence of 'offer'.

Related: Naive Bayes, Bayes' theorem, Bayesian network

In association rules, how often the right side appears when the left side does: support(A and B) / support(A). It estimates the conditional probability of B given A, but it can look high simply because B is common.

Example. If 200 baskets have tea and biscuits and 250 have tea, confidence of tea to biscuits is 0.8.

Related: Association rule learning, Support, Lift

Density-Based Spatial Clustering of Applications with Noise: it grows clusters from points that have at least min_samples neighbours within radius eps, and labels isolated points as noise. It finds odd-shaped clusters and needs no preset cluster count.

Example. On delivery GPS data, eps=0.5 km and min_samples=10 group busy neighbourhoods and mark stray one-off points as noise.

Related: K-means, OPTICS, HDBSCAN, Anomaly detection

The surface in feature space where a classifier switches from predicting one class to another. In two dimensions it is a line or curve. Its shape depends on the algorithm: straight for logistic fitting, jagged and box-like for trees, curved for kernel methods.

Example. For a loan model using income and debt, the boundary might be the line debt = 0.4 * income; applicants above it are predicted to default.

Related: Logistic regression, Support vector machine, Decision tree, Classifier

A tree with just one split, which makes a prediction from a single feature and a single threshold. It is extremely simple and cheap to train, and it is the classic weak learner used inside AdaBoost, where many are combined into a stronger model.

Example. A stump that predicts 'spam' whenever the count of the word 'lottery' is above 2.

Related: Weak learner, AdaBoost, Decision tree

A model that predicts by asking a chain of yes/no questions about features, splitting the data at each step until it reaches a final leaf holding a prediction. It is easy to read, needs little data preparation and captures non-linear patterns, but a deep tree can memorise noise.

Example. A tree may ask 'is age > 30?' then 'is income > 8 lakh?' and end at a leaf saying 'approve loan'.

Related: Gini impurity, Information gain, Tree pruning, Random forest

A tree diagram showing the order and distance at which clusters are merged in a hierarchy. The height of a join shows how dissimilar the merged groups were, and a horizontal cut gives a chosen number of clusters.

Example. A tall vertical gap in the diagram before two last branches suggests the data naturally holds two clusters.

Related: Hierarchical clustering, Agglomerative clustering, Linkage criterion

Transforming data with many features into a smaller set of features while keeping as much useful structure as possible. It speeds up learning, removes noise, reduces storage and makes it possible to draw 2-D or 3-D plots of the data.

Example. Compressing 1,000 gene-expression columns into 20 derived columns.

Related: Principal component analysis, t-SNE, Feature selection, Curse of dimensionality

A fitting method that blends the ridge and lasso penalties, controlled by a mixing ratio. It keeps lasso's ability to zero out weights while borrowing ridge's stability when features are correlated, where lasso alone tends to pick one arbitrarily from a group.

Example. ElasticNet(alpha=0.5, l1_ratio=0.3) uses 30% lasso-style and 70% ridge-style penalty on gene-expression data with thousands of correlated columns.

Related: Ridge regression, Lasso regression, Multicollinearity, Regularisation

A heuristic for picking the number of clusters: plot inertia against k and choose the point where the curve bends and further clusters give diminishing gains. It is subjective and sometimes shows no clear bend.

Example. Inertia drops steeply from k=1 to k=4 then flattens, so k=4 is chosen.

Related: K-means, Inertia, Silhouette score

Combining the predictions of several models so that the group is more accurate or stable than any single member. Main styles are bagging, boosting and stacking. It works best when members are individually decent yet make different mistakes.

Example. Averaging the outputs of a logistic model, a forest and a boosted model to predict loan default.

Related: Bagging, Boosting, Stacking, Voting classifier

A measure of uncertainty or disorder in a set of outcomes: the negative sum of p * log2(p) over class proportions. It is 0 when everything is one class and 1 bit for a 50/50 split of two classes. Decision trees use it to judge how mixed a node is.

Example. A node with 8 yes and 2 no has entropy = -(0.8*log2 0.8 + 0.2*log2 0.2), about 0.72 bits.

Related: Information gain, Gini impurity, Decision tree, Cross-entropy loss

The straight-line distance between two points: the square root of the sum of squared differences in each coordinate. It is the default similarity measure in neighbour and clustering methods and is sensitive to feature scale.

Example. Between (1, 2) and (4, 6) the distance is sqrt(3^2 + 4^2) = 5.

Related: Manhattan distance, k-nearest neighbours, K-means, Feature scaling

An iterative method for fitting models with hidden variables. The E-step estimates the hidden values given current parameters, and the M-step updates parameters to best fit those estimates. Each cycle never lowers the likelihood, and it is used for mixture models.

Example. Fitting a two-component mixture: estimate each point's probability of belonging to component A, then recompute A's mean and spread using those probabilities.

Related: Gaussian mixture model, K-means, Maximum likelihood estimation

The fraction of a dataset's total variation captured by each principal component, so the sum over kept components shows how much information survives. It is used to decide how many components to retain.

Example. Components explain 0.55, 0.25, 0.10 and 0.05 of the variance, so the first three keep 90%.

Related: Principal component analysis, Scree plot, Principal component

A forest variant that goes one step beyond random subsets: at each split it draws random thresholds for candidate features and keeps the best of those, instead of searching for the optimal cut. It usually trains faster and adds extra smoothing.

Example. ExtraTreesClassifier in scikit-learn trains 500 such trees on the same data as a random forest, often in less time.

Related: Random forest, Decision tree, Bagging

A score saying how much each input contributes to a model's predictions. For trees it is commonly the total impurity reduction a feature brings across all splits, averaged over the forest. It guides feature selection and debugging but can mislead with correlated or high-cardinality features.

Example. A forest on loan data reports importance 0.35 for credit_score and 0.02 for city, hinting that credit_score drives most decisions.

Related: Permutation importance, Random forest, Gradient boosting, Feature selection

A probabilistic model that assumes data come from a blend of several bell-shaped (normal) distributions, each with its own mean and spread. It gives soft cluster membership as probabilities and handles elliptical clusters, and is usually fitted with expectation-maximisation.

Example. A customer may belong 70% to the 'high spender' cluster and 30% to 'occasional'.

Related: Expectation-maximisation, K-means, Normal distribution, Soft clustering

A naive Bayes version for continuous features that models each feature within each class as a normal distribution, estimating a mean and a variance per class. It is quick and works when features roughly follow bell curves.

Example. For classifying flowers, petal length in one species might be modelled as mean 4.3 cm and standard deviation 0.5 cm.

Related: Naive Bayes, Multinomial naive Bayes, Normal distribution

A flexible model that defines a probability distribution over functions using a kernel to describe how outputs relate. It gives predictions along with uncertainty and works well on small datasets, but scales poorly with rows.

Example. Fitting 30 expensive lab experiments and getting both a predicted yield and a confidence band at untested settings.

Related: RBF kernel, Bayesian optimisation, Kernel trick, Prediction interval

A measure of how mixed the classes are in a group of examples: 1 minus the sum of squared class proportions. It is 0 when all items share one class and highest when classes are evenly mixed. Tree building picks splits that reduce it the most.

Example. A node with 6 spam and 4 not-spam emails has Gini = 1 - (0.6^2 + 0.4^2) = 0.48.

Related: Decision tree, Information gain, CART, Entropy

A boosting method that adds trees one at a time, each fitted to the errors (the negative gradient of the loss) of the current ensemble. A learning rate scales each tree's contribution. It is highly accurate on tabular data and underlies XGBoost, LightGBM and CatBoost.

Example. For house prices, tree 1 predicts 50 lakh where truth is 58; tree 2 learns the +8 residual and adds a fraction of it.

Related: XGBoost, LightGBM, CatBoost, Shrinkage

A hierarchical version of density-based clustering that builds a tree of clusters over all density thresholds and extracts the most stable ones. It handles varying densities and needs mainly a minimum cluster size.

Example. Using min_cluster_size=20 finds tight groups of support tickets and labels the rest as noise.

Related: DBSCAN, OPTICS, Hierarchical clustering

A family of clustering methods that build a nested tree of groups rather than one flat split. Bottom-up methods start with every point alone and merge the closest pair repeatedly; top-down ones split. Cutting the tree at a level gives the clusters.

Example. Grouping 200 products by similarity, then cutting the tree at height 5 gives 8 clusters.

Related: Agglomerative clustering, Dendrogram, Linkage criterion, K-means

A loss function for margin classifiers: max(0, 1 - y * f(x)), where y is +1 or -1 and f(x) is the raw score. It is zero for points correctly classified beyond the margin and grows linearly for points inside it or misclassified.

Example. A positive example (y=+1) with score 0.3 has loss 0.7; with score 1.5 it has loss 0.

Related: Support vector machine, Soft margin, Cross-entropy loss, Loss function

An early tree-building algorithm by Ross Quinlan that grows a tree top-down, choosing at every node the categorical feature with the highest information gain. It cannot handle numeric features or missing values directly and has no pruning step, which later methods fixed.

Example. On a weather dataset for 'play cricket?', ID3 picks 'outlook' first because it yields the largest information gain.

Related: C4.5, Information gain, Entropy, Decision tree

In k-means, the sum of squared distances from every point to the centre of its assigned cluster. Lower means tighter clusters, but it always falls as k rises, so it is compared across k rather than minimised.

Example. With k=3 inertia is 1,200 and with k=4 it is 900; the gain from adding a cluster shows whether it is worthwhile.

Related: K-means, Elbow method, Silhouette score

The drop in disorder achieved by splitting a group on a feature, computed as the entropy before the split minus the weighted average entropy of the resulting parts. Tree algorithms such as ID3 pick the split with the largest gain.

Example. Splitting 10 customers (5 buy, 5 don't) into two pure groups of 5 reduces entropy from 1 bit to 0, a gain of 1 bit.

Related: Entropy, Decision tree, ID3, Gini impurity

An anomaly-detection method that builds random trees which split data at random feature values. Unusual points get separated from the rest after only a few splits, so a short average path length signals an outlier.

Example. Flagging an unusual UPI transaction of 9 lakh among typical 200 to 2,000 rupee payments, since it is isolated in just 2 splits.

Related: Anomaly detection, Random forest, Local outlier factor, One-class SVM

A clustering algorithm that splits data into k groups by alternating two steps: assign each point to its nearest centre, then move each centre to the average of its points. It repeats until assignments stop changing. It is fast but assumes roundish clusters of similar size and needs k in advance.

Example. Grouping 50,000 shoppers into k=4 segments using annual spend and visit frequency.

Related: Centroid, K-means++, Elbow method, Inertia

A method that predicts for a new point by finding the k most similar training points and taking their majority class or average value. It has no training phase beyond storing the data, works with any distance, and is simple, but slows down on large data and suffers from irrelevant features.

Example. With k=5, if 4 of the 5 closest customers to a new one churned, the model predicts churn.

Related: Euclidean distance, Lazy learning, KD-tree, Feature scaling

A tree that repeatedly splits space along one coordinate axis at a time so that nearby points can be found without checking every stored point. It speeds up neighbour search in low to moderate dimensions but loses its advantage as dimensions grow.

Example. Finding the 5 closest of 1 million 3-D points takes roughly logarithmic time using this tree rather than a full scan.

Related: k-nearest neighbours, Ball tree, Approximate nearest neighbour

An extension of principal component analysis that uses the kernel trick so it can find curved, non-linear structure. It runs the analysis in an implicit high-dimensional space rather than on original coordinates.

Example. Unrolling two concentric rings so a linear classifier can separate them after projection.

Related: Principal component analysis, Kernel trick, RBF kernel

A way to get the effect of mapping data into a much higher-dimensional space without computing that mapping. It replaces dot products with a kernel function that measures similarity directly, allowing curved boundaries while keeping computation manageable.

Example. Two rings of points, not separable by a line in 2-D, become separable using an RBF kernel, with no explicit new coordinates computed.

Related: Support vector machine, RBF kernel, Polynomial kernel, Kernel PCA

Adding a small constant, usually 1, to every count before turning counts into probabilities, so an event never seen in training does not get probability zero. Without it, one unseen word can wipe out a whole naive Bayes product.

Example. If 'cashback' never appeared in non-spam among 1,000 words with vocabulary 5,000, its probability is (0+1)/(1000+5000), not 0.

Related: Naive Bayes, Multinomial naive Bayes, Zero-frequency problem

A least-squares variant that adds a penalty equal to the sum of absolute weight values. Because of the sharp corner of that penalty, some weights become exactly zero, so the model both fits and selects features. It suits datasets with many features where only a few matter.

Example. With 200 candidate features and Lasso(alpha=0.1), perhaps only 12 keep non-zero weights, giving a compact model that is easy to explain.

Related: Ridge regression, Elastic net, Feature selection, Regularisation

An approach where a model postpones work until a prediction is requested, keeping the training data and comparing against it each time, rather than building a summary in advance. Neighbour-based methods are the standard example.

Example. A neighbours model 'trains' in milliseconds by storing 1 million rows, but each prediction must search them.

Related: k-nearest neighbours, Eager learning, KD-tree

In association rules, the ratio of observed co-occurrence to that expected if items were independent: confidence(A to B) / support(B). Above 1 means the items appear together more than chance, 1 means no link.

Example. If confidence is 0.8 and biscuits appear in 0.4 of all baskets, lift is 2.

Related: Association rule learning, Confidence, Support

A gradient boosting library from Microsoft that speeds up training by bucketing feature values into histograms and growing trees leaf-wise, expanding the leaf with the biggest loss reduction. It handles large datasets efficiently and supports categorical columns natively.

Example. LGBMClassifier(num_leaves=31) can train on millions of rows of click data in minutes.

Related: Gradient boosting, XGBoost, Histogram-based gradient boosting, CatBoost

A classifier and dimensionality reducer that assumes each class is a normal cloud sharing the same spread, and finds the directions that best separate class averages relative to within-class spread. Its boundaries are straight lines or planes.

Example. Projecting 4 flower measurements onto 2 axes that separate the three iris species as much as possible.

Related: Quadratic discriminant analysis, Principal component analysis, Gaussian naive Bayes, Logistic regression

A supervised method that predicts a number as a weighted sum of input features plus a constant. Training picks the weights that make predictions as close as possible to the known answers, usually by minimising squared error. It is fast, easy to interpret and a standard baseline for numeric prediction.

Example. Predicting a flat's monthly rent in Bengaluru as 4000 + 25 * area_sqft + 3000 * bedrooms; a 900 sqft 2-bedroom flat gives 4000 + 22500 + 6000 = 32,500 rupees.

Related: Ordinary least squares, Residual, Multiple linear regression, Ridge regression

The rule for measuring distance between two clusters when merging: single (closest pair), complete (farthest pair), average (mean of all pairs) or Ward (smallest increase in within-cluster variance). It strongly shapes the clusters found.

Example. Single linkage can chain far-apart points through a thin bridge, while Ward tends to give compact, similar-sized clusters.

Related: Agglomerative clustering, Hierarchical clustering, Dendrogram

An anomaly score that compares a point's local density with the densities of its neighbours. A point sitting in a much sparser area than its neighbours gets a score well above 1 and is flagged.

Example. A house priced far from similar properties in its own locality scores high even if it is normal for the whole city.

Related: Anomaly detection, Isolation forest, k-nearest neighbours, DBSCAN

The natural logarithm of the odds of an event, ln(p / (1 - p)), where p is its probability. It stretches probabilities across the whole number line, which is why a straight weighted sum can model them. Each feature weight in logistic fitting changes the log-odds by a fixed amount.

Example. A probability of 0.8 means odds of 4 to 1, so the log-odds is ln(4), about 1.386.

Related: Logistic regression, Odds ratio, Sigmoid function, Link function

A supervised method for classification that computes a weighted sum of features and squeezes it through an S-shaped curve to give a probability between 0 and 1. A cut-off, often 0.5, turns that probability into a class label. Despite the name it predicts categories, and it is a strong, interpretable baseline.

Example. An email with weighted score 2.2 gets probability 1 / (1 + e^-2.2) = 0.90 of being spam, so it is filtered.

Related: Sigmoid function, Log-odds, Decision boundary, Cross-entropy loss

The distance between two points measured by summing the absolute differences of their coordinates, like walking along city blocks. It is less affected by a single large difference than straight-line distance and is used in neighbour and clustering methods.

Example. Between (1, 2) and (4, 6) it is |4-1| + |6-2| = 7.

Related: Euclidean distance, k-nearest neighbours, Minkowski distance

Approximating a large matrix by the product of two smaller matrices whose entries are learned from the known values. It extracts hidden factors, and is widely used to predict missing entries in user-item rating tables for recommendations.

Example. A 1 million by 50,000 rating matrix is approximated by user vectors and item vectors of length 32.

Related: Non-negative matrix factorisation, Singular value decomposition, Collaborative filtering

A linear classifier that picks, among all lines or planes that separate two classes perfectly, the one farthest from the nearest training points on both sides. A wide gap tends to generalise better. It only works when the classes can be separated exactly.

Example. Among many lines splitting red and blue dots, choose the one leaving a 3-unit-wide empty corridor.

Related: Support vector machine, Soft margin, Support vector

A faster variant of k-means that updates the centres using small random batches of data at each step instead of all the rows. It trades slight accuracy for large speed and memory savings on very big datasets.

Example. MiniBatchKMeans(batch_size=1024) clusters 20 million app-event rows without loading them all at once.

Related: K-means, Stochastic gradient descent, Inertia

A situation in which two or more input features are strongly correlated with each other. The model cannot tell their effects apart, so fitted weights become unstable and swing widely with small data changes, even though overall predictions may stay fine. Dropping or combining features, or using penalised fitting, helps.

Example. Including both height in centimetres and height in inches as features makes the weights on each meaningless individually, since one is an exact multiple of the other.

Related: Variance inflation factor, Ridge regression, Linear regression, Principal component analysis

A naive Bayes version for count data, such as how often each word appears in a document. It models word counts per class as a multinomial distribution and is a long-standing text classification baseline.

Example. Classifying news articles as sports or politics from word counts, where 'wicket' appears 5 times in one article.

Related: Naive Bayes, Bernoulli naive Bayes, Laplace smoothing, TF-IDF

A straight-line style model with two or more input features, written y = b0 + b1*x1 + b2*x2 + ... Each weight shows the effect of its feature while the others are held fixed. It extends the one-feature case to realistic problems with many predictors.

Example. Predicting delivery time from distance_km, orders_in_queue and rain_flag: time = 8 + 3.5 * distance_km + 1.2 * orders_in_queue + 6 * rain_flag minutes.

Related: Linear regression, Multicollinearity, Ordinary least squares

A probabilistic classifier that applies Bayes' rule and assumes features are independent of each other once the class is known. That simplification makes training a matter of counting, so it is very fast, works with little data and is a classic baseline for text classification.

Example. Classifying an SMS as spam by multiplying the class prior by the probability of each word given spam and given not-spam, then picking the larger.

Related: Bayes' theorem, Conditional independence, Laplace smoothing, Multinomial naive Bayes

A method that splits a matrix of non-negative values into two smaller non-negative matrices whose product approximates it. Because parts can only add up, factors tend to be interpretable, such as topics or parts of images.

Example. Factorising a document-word count matrix into topics, each a set of words with positive weights.

Related: Matrix factorisation, Truncated SVD, Topic modelling

The closed-form formula that gives the least-squares weights in one step: w = (X^T X)^-1 X^T y, where X is the matrix of features and y the vector of targets. It needs no learning rate or iteration but becomes slow or unstable with many features or near-duplicate columns.

Example. With 3 features and 10,000 rows, NumPy solves w = np.linalg.solve(X.T @ X, X.T @ y) directly instead of running gradient descent.

Related: Ordinary least squares, Gradient descent, Multicollinearity, Matrix inverse

The factor by which the odds of an outcome multiply when a feature rises by one unit. In logistic fitting it equals e raised to that feature's weight. A value above 1 means the feature raises the odds, below 1 lowers them, and exactly 1 means no effect.

Example. A weight of 0.69 on has_coupon gives an odds ratio of about 2, so customers with a coupon have roughly double the odds of buying.

Related: Log-odds, Logistic regression, Coefficient, Relative risk

A margin-based method trained only on normal data that learns a boundary around it. Points falling outside are flagged as anomalies. It suits cases where abnormal examples are rare or unavailable.

Example. Training on a month of normal server logs and alerting when a new log vector lies outside the learned region.

Related: Anomaly detection, Support vector machine, Isolation forest, Kernel trick

A strategy for many-class problems that trains a binary classifier for every pair of classes, then lets the pairwise models vote. It needs k(k-1)/2 models for k classes but each trains on a small slice of data, which suits algorithms that scale badly with data size.

Example. With 5 classes you train 10 pairwise models; the class that wins the most votes is predicted.

Related: One-vs-rest, Support vector machine, Multi-class classification

A strategy that handles many classes with a binary classifier by training one model per class, each separating that class from all others combined. At prediction time the model reporting the highest confidence decides the label.

Example. For 10 digit classes, train 10 classifiers; the '7 vs everything else' model returns the highest score for a handwritten 7.

Related: One-vs-one, Logistic regression, Support vector machine, Multi-class classification

The classic way to fit a line or plane: choose the weights that minimise the sum of squared differences between actual and predicted values. It has a closed-form solution when the feature matrix is well behaved, and it gives the best linear unbiased estimates under the Gauss-Markov assumptions.

Example. With points (1,2), (2,3), (3,5), OLS gives the line y = 0.5 + 1.5x, which leaves smaller total squared error than any other straight line.

Related: Linear regression, Normal equation, Residual, Mean squared error

An error estimate for bagged models that uses the rows each tree did not see in its bootstrap sample. Every row is predicted only by trees that left it out, so the score behaves like validation data without setting any data aside.

Example. Setting oob_score=True in RandomForestClassifier reports 0.86 accuracy measured on the left-out rows of each tree.

Related: Random forest, Bootstrap sampling, Bagging, Cross-validation

The oldest learning algorithm for a linear classifier: it multiplies inputs by weights, adds a bias and outputs class +1 or -1, then nudges the weights whenever it makes a mistake. It converges only if the classes are separable by a straight line.

Example. On a mistake with input (2, 1) and label +1, add the input to the weights: w = w + 1 * (2, 1).

Related: Logistic regression, Support vector machine, Neuron, Activation function

A way to measure how much a model depends on a feature: shuffle that feature's values in held-out data and see how much the score drops. A large drop means the model relies on it. It works with any model and avoids biases of split-based importance.

Example. Shuffling the 'tenure' column drops accuracy from 0.90 to 0.82, so tenure has an importance of 0.08.

Related: Feature importance, Random forest, Feature selection, Explainability

A kernel of the form (x . z + c)^d that implicitly considers all feature combinations up to degree d. It lets a margin classifier learn curved boundaries built from products of features.

Example. A degree-2 kernel lets an SVM separate points inside a circle from those outside using products such as x1*x2 and x1^2.

Related: Kernel trick, RBF kernel, Support vector machine, Polynomial regression

Fitting a curve by adding powers of a feature (x^2, x^3, ...) as extra columns and then running a straight-line fit on them. The model is still linear in its weights, yet it can bend to match curved relationships. High degrees tend to wiggle wildly and memorise noise.

Example. Fitting y = 2 + 0.5x + 0.1x^2 to stopping distance versus speed of a two-wheeler captures the fact that braking distance grows faster than speed.

Related: Linear regression, Overfitting, Feature engineering, Ridge regression

One of the new axes found in dimensionality reduction, given as a direction in the original feature space. The first captures the most variation, the second the most remaining variation at right angles to it, and so on.

Example. For height and weight data, the first axis roughly points along overall body size.

Related: Principal component analysis, Eigenvector, Explained variance ratio

A technique that finds new axes, ordered by how much of the data's variation they capture, by eigen-decomposition of the covariance matrix. Keeping only the first few axes compresses data with little information loss, aiding visualisation, noise removal and speed.

Example. Reducing 50 correlated survey answers to 5 components that keep 90% of the variance.

Related: Principal component, Explained variance ratio, Eigenvalue, Singular value decomposition

A generative classifier like linear discriminant analysis but allowing every class its own spread, which produces curved quadratic boundaries. It is more flexible but needs more data to estimate each class's covariance.

Example. With classes of different shapes, one tight and one wide, it draws an elliptical boundary between them.

Related: Linear discriminant analysis, Gaussian naive Bayes, Decision boundary

An ensemble of many decision trees, each trained on a bootstrap sample and allowed to consider only a random subset of features at every split. Predictions are averaged or voted. It is accurate, resists overfitting and needs little tuning, which makes it a popular tabular-data baseline.

Example. RandomForestClassifier(n_estimators=300) builds 300 trees on churn data and predicts the class most trees choose.

Related: Bagging, Decision tree, Out-of-bag error, Gradient boosting

The radial basis function kernel, K(x, z) = exp(-gamma * ||x - z||^2), giving similarity that decays with distance. A larger gamma means each point influences only its close neighbours, producing wiggly boundaries; a small gamma gives smoother ones.

Example. SVC(kernel='rbf', gamma=0.1) draws a smooth curved boundary around clusters of customers.

Related: Kernel trick, Support vector machine, Gaussian process, Polynomial kernel

A decision tree that predicts a number. Each leaf holds the average target of the training rows that land in it, and splits are chosen to cut squared error the most. It gives step-shaped predictions and captures interactions without any feature scaling.

Example. A tree predicting house price might split on area, then locality; a leaf with 12 training flats averaging 48 lakh predicts 48 lakh for any new flat landing there.

Related: Decision tree, CART, Mean squared error, Random forest

The gap between an actual observed value and the value a model predicted for it, calculated as actual - predicted. Plotting residuals reveals whether a model misses curved patterns, has unequal error spread or is distorted by outliers.

Example. A house sold for 60 lakh but the model predicted 55 lakh, so its residual is +5 lakh.

Related: Linear regression, Mean squared error, Ordinary least squares

A variant of least-squares fitting that adds a penalty equal to the sum of squared weights, scaled by a strength lambda. The penalty shrinks weights towards zero without making them exactly zero, which steadies the fit when features overlap or when there are many features compared with rows.

Example. In scikit-learn, Ridge(alpha=1.0).fit(X, y) trains a model whose weights are smaller than plain least squares would give on correlated columns.

Related: Lasso regression, Elastic net, Multicollinearity, Regularisation

A line chart of the variance explained by each component in order, used to spot where the curve levels off and adding components stops helping. The bend is often called the elbow.

Example. A plot dropping from 40% to 8% and then flattening near 3% suggests keeping about 3 components.

Related: Principal component analysis, Explained variance ratio, Elbow method

Scaling down each new tree's contribution in boosting by a small factor, the learning rate, so the ensemble improves in tiny steps. Smaller values need more trees but usually generalise better.

Example. Using learning_rate=0.05 with 600 trees often beats learning_rate=0.5 with 60 trees.

Related: Gradient boosting, Learning rate, Overfitting, XGBoost

The S-shaped function sigma(z) = 1 / (1 + e^-z) that maps any real number to a value between 0 and 1. It turns a raw score into something readable as a probability. It is smooth, so its slope can be used for training, but it flattens for large positive or negative inputs.

Example. sigma(0) = 0.5, sigma(4) is about 0.982, and sigma(-4) is about 0.018.

Related: Logistic regression, Log-odds, Softmax regression, Activation function

A measure of clustering quality for each point: (b - a) / max(a, b), where a is the average distance to points in its own cluster and b the average distance to the nearest other cluster. Values near 1 mean well separated, near 0 overlapping, negative misassigned.

Example. A point with a=2 and b=6 has silhouette (6-2)/6, about 0.67.

Related: K-means, Elbow method, Inertia, Clustering

The one-feature case of fitting a straight line to data: y = b0 + b1 * x. The slope b1 says how much the prediction changes for a one-unit rise in x, and the intercept b0 is the prediction when x is zero. It is the easiest model to plot and explain.

Example. Fitting exam score against hours studied gives score = 35 + 6 * hours, so each extra study hour adds about 6 marks and a student who studies 5 hours is predicted 65.

Related: Linear regression, Ordinary least squares, Residual

A relaxation that lets a margin-based classifier make some mistakes or let points fall inside the gap, in return for a wider margin. The parameter C controls the trade-off: small C tolerates more violations, large C punishes them heavily.

Example. SVC(C=0.1) permits many margin violations and gives a smoother boundary; SVC(C=1000) fits training data tightly.

Related: Support vector machine, Hinge loss, Regularisation, Maximum margin classifier

A multi-class extension of logistic fitting that scores every class with its own weighted sum and converts the scores into probabilities that add up to 1 using the softmax function. The predicted class is the one with the highest probability.

Example. Scores of 2.0, 1.0 and 0.1 for cat, dog and bird become probabilities of about 0.66, 0.24 and 0.10.

Related: Logistic regression, Sigmoid function, Cross-entropy loss, One-vs-rest

A clustering method that builds a graph of point similarities, computes eigenvectors of its Laplacian matrix, and runs k-means on those eigenvectors. It can separate non-convex shapes such as intertwined rings that k-means cannot.

Example. Two concentric circles of points are separated correctly, though a straight-line centre method would mix them.

Related: K-means, Eigenvalue, Graph Laplacian, Hierarchical clustering

An ensemble technique where predictions from several different base models become the inputs to a final model that learns how best to combine them. Base predictions are produced out-of-fold to avoid leakage.

Example. Feed forest, SVM and logistic predictions into a small logistic model that outputs the final answer.

Related: Ensemble learning, Voting classifier, Cross-validation, Blending

In association rules, the fraction of all recorded transactions that contain a given set of items. It shows how common a pattern is, and rules with very low values are usually ignored as too rare to trust.

Example. If 200 of 1,000 baskets contain both tea and biscuits, support is 0.2.

Related: Association rule learning, Confidence, Lift

A training point that lies on or inside the gap around a margin-based classifier's boundary, so it directly fixes the position of the boundary. Moving or deleting any other point changes nothing. Few support vectors relative to data size usually signals a simple boundary.

Example. In a dataset of 10,000 rows, the fitted model may keep only 300 support vectors.

Related: Support vector machine, Maximum margin classifier, Kernel trick

A classifier that finds the boundary separating two classes with the widest possible gap between them. Only the points closest to the boundary, the support vectors, determine it. With kernels it can draw curved boundaries, and it works well on medium-sized, high-dimensional data.

Example. Separating cancerous from normal cells using 30 measurements: the model finds the widest-margin hyperplane in that 30-dimensional space.

Related: Support vector, Kernel trick, Hinge loss, Maximum margin classifier

An adaptation of margin-based learning to predict numbers. It fits a function that stays within a tolerance band epsilon of the targets, ignoring small errors and penalising only points outside the band.

Example. SVR(epsilon=2.0) predicting temperature treats errors under 2 degrees as costless.

Related: Support vector machine, Kernel trick, Linear regression

t-distributed Stochastic Neighbour Embedding: a technique that maps high-dimensional data to 2-D or 3-D so that similar points stay close, by matching neighbour probabilities. It gives striking cluster plots but distances between clusters and their sizes are not reliable.

Example. Plotting 784-pixel handwritten digits in 2-D where each digit forms its own island.

Related: UMAP, Principal component analysis, Dimensionality reduction

Reducing the size of a decision tree by cutting branches that add little predictive value, either by stopping growth early or by trimming a full tree afterwards. It lowers the chance of memorising noise and yields simpler, more robust models.

Example. In scikit-learn, setting max_depth=5 or ccp_alpha=0.01 restricts the tree so it cannot grow a leaf for every training row.

Related: Decision tree, Overfitting, CART, Cost-complexity pruning

A dimensionality reduction method that keeps only the largest singular values of the data matrix. Unlike principal component analysis it does not centre the data first, so it works directly on sparse matrices such as word counts.

Example. Compressing a 100,000 by 50,000 sparse TF-IDF matrix to 200 columns.

Related: Singular value decomposition, Principal component analysis, TF-IDF

Uniform Manifold Approximation and Projection: a dimensionality reduction method that builds a graph of nearest neighbours and lays it out in low dimensions. It is usually faster than t-SNE and keeps more global structure, and can be used for general reduction too.

Example. Reducing 768-dimensional text vectors to 2-D to inspect topic clusters.

Related: t-SNE, Principal component analysis, k-nearest neighbours

A score that measures how much a feature's weight variance is inflated because that feature can be predicted from the other features. It equals 1 / (1 - R^2), where R^2 comes from regressing that feature on the rest. Values above roughly 5 to 10 are commonly treated as a warning.

Example. If income can be predicted from the other features with R^2 = 0.9, its score is 1 / (1 - 0.9) = 10, flagging heavy overlap.

Related: Multicollinearity, Linear regression, Feature selection

An ensemble that combines several classifiers by majority vote (hard voting) or by averaging their predicted probabilities (soft voting). It is the simplest way to blend different model types, and it works best when members make different kinds of mistakes.

Example. If a tree, an SVM and a logistic model predict spam, spam, not-spam, hard voting returns spam.

Related: Ensemble learning, Stacking, Random forest

A simple model that performs only slightly better than random guessing, such as a decision tree with one split. Boosting combines many of these in sequence to form a strong model.

Example. A one-split tree ('salary > 5 lakh?') that is right 58% of the time on a two-class problem.

Related: Boosting, AdaBoost, Decision stump, Gradient boosting

A widely used open-source library for gradient boosted trees. It adds a regularisation term, handles missing values, uses second-order gradient information and offers parallel, fast training. It has been a common winner in structured-data competitions.

Example. xgb.XGBClassifier(n_estimators=400, max_depth=6, learning_rate=0.05) trains a churn model on a customer table.

Related: Gradient boosting, LightGBM, CatBoost, Regularisation