Ml fundamentals

An experiment that removes or disables one component of a system at a time, such as a feature or a module, to measure how much each contributes to performance. It shows what really matters and what is unnecessary.

Example. Dropping the 'location' feature lowers accuracy from 86% to 79%, showing it is important; dropping 'day of week' changes nothing.

Related: Baseline model, Feature, Model selection, Reproducibility

A strategy in which the algorithm chooses which unlabelled examples a human should label next, usually those it is most uncertain about. It aims to reach good accuracy with far fewer labels, which saves annotation cost.

Example. Out of 100,000 unlabelled support tickets, the model picks the 500 it is least sure about for agents to categorise first.

Related: Semi-supervised learning, Label, Weak supervision, Annotation

A score for comparing models that rewards fit but penalises the number of parameters, calculated as AIC = 2k - 2 ln(L), where k is the parameter count and L the maximised likelihood. Lower values are preferred, and it estimates relative information lost by each model.

Example. Model A has AIC 210 and Model B 204, so B is preferred as the better trade-off of fit and complexity.

Related: Bayesian information criterion, Model selection, Occam's razor, Regularisation

The process of attaching labels or other information to raw data, such as tagging images, marking spans of text or rating responses. It is usually done by people following written guidelines, and its consistency and cost strongly affect supervised projects.

Example. Annotators mark each of 5,000 customer reviews as positive, neutral or negative following a one-page guide.

Related: Label, Inter-annotator agreement, Ground truth, Active learning

A simple reference model used to judge whether a more complex approach is actually worth it. It might predict the most common class or the average value, or use a basic rule. Any serious model has to beat it clearly, and comparing against it prevents self-deception.

Example. On a dataset where 92% of transactions are genuine, always predicting 'genuine' gives 92% accuracy; a fraud model must be judged against that.

Related: Majority-class baseline, Model selection, Ablation study, Test set

A model comparison score similar to the Akaike criterion but with a heavier penalty on complexity for larger datasets: BIC = k ln(n) - 2 ln(L), where n is the number of observations. It tends to favour simpler models and lower values are better.

Example. With 1,000 rows, BIC punishes each extra parameter by about 6.9 units, so a marginal feature is often not worth including.

Related: Akaike information criterion, Model selection, Occam's razor, Regularisation

The part of a model's prediction error caused by wrong or overly simple assumptions, so that even with unlimited data it would miss the true relationship. It is the average gap between what the model predicts and the truth. Not to be confused with unfairness towards groups of people.

Example. A straight-line model of house prices in a market where prices rise steeply with size will consistently underestimate large homes, whatever data it sees.

Related: Model variance, Bias-variance tradeoff, Underfitting, Irreducible error

The tension between two sources of prediction error. Simple models make strong assumptions and miss real patterns (high bias); flexible models react to random quirks of the sample and change a lot when data changes (high variance). Good models balance the two so that total error on new data is lowest.

Example. Raising a tree's depth from 2 to 20 lowers bias but raises variance; validation error is typically lowest somewhere in between.

Related: Bias error, Model variance, Overfitting, Underfitting

A supervised task where the goal is to assign each input to one of a fixed set of categories. The model outputs a class, often together with a probability for each class. It appears everywhere from email filtering to medical diagnosis, and its quality is judged with measures such as accuracy or precision.

Example. Deciding whether a customer review is positive, neutral or negative is a three-class task; an email spam filter is a two-class one.

Related: Regression, Decision boundary, Label, Supervised learning

A change over time in the relationship between the inputs and the target itself, so patterns the model learned no longer hold. It requires retraining or adaptation, and is detected by watching live performance.

Example. After a new regulation, the same transaction features no longer signal fraud in the way they did before.

Related: Distribution shift, Covariate shift, Online learning, Temporal leakage

Training or deciding in a way that treats different mistakes as having different costs, rather than counting all errors equally. It fits problems where missing a case costs far more than a false alarm.

Example. Missing a cancer case is set to cost 20 times more than a false alarm, so the model is pushed to flag borderline scans.

Related: Class imbalance, Decision threshold, Loss function, Classification

A kind of distribution shift where the distribution of the input features changes between training and use, while the rule linking inputs to outputs stays the same. The model may be asked about regions of input space it saw rarely, so errors rise.

Example. A credit model trained mainly on salaried urban applicants starts receiving many rural self-employed applicants.

Related: Distribution shift, Concept drift, Out-of-distribution, Selection bias

A way of estimating how well a model will perform on new data by repeatedly splitting the data, training on some parts and testing on the remaining part, then averaging the results. It uses all the data for both purposes and is more reliable than one split, especially for small datasets.

Example. Rather than one 80/20 split, you train five times, each time testing on a different fifth of the data, and average the five scores.

Related: k-fold cross-validation, Holdout method, Nested cross-validation, Model selection

A group of problems that appear when data has many features: the space grows so fast that data becomes sparse, distances between points lose meaning and far more examples are needed to cover it. It harms methods such as nearest neighbours and encourages feature reduction.

Example. Covering a 1-D range with 10 points is easy, but keeping the same density in 10 dimensions would need 10 billion points.

Related: Dimensionality reduction, Feature space, Overfitting, Sample complexity

A mistake where information that would not be available at prediction time gets into training, so the model looks excellent in testing but fails in real use. Common causes are features that reveal the answer, preprocessing on the full dataset, and duplicates across splits.

Example. A churn model uses 'account closed date' as a feature; it scores 99% in testing but the field is empty for customers who are still active.

Related: Target leakage, Temporal leakage, Train-test contamination, Data snooping

Repeatedly using the same held-out data to make modelling choices, so that the reported score reflects choices tuned to that data rather than true performance. Each look at the test results makes the estimate a little less trustworthy.

Example. After trying 40 model variants and always picking the one with the best test score, the winning score is inflated by selection.

Related: Test set, Train-test contamination, Nested cross-validation, Validation set

An approach that focuses on improving the quality, coverage and consistency of the data, rather than mainly tweaking model architecture. It holds that better labels and data often bring larger gains than model changes.

Example. Fixing 300 inconsistent labels in a defect dataset raises accuracy more than trying three new architectures.

Related: Label noise, Annotation, Ground truth, Dataset

An organised collection of examples used to build or judge a model, usually arranged as a table where rows are examples and columns are features and labels. Its size, quality and how well it represents the real world largely determine what any model trained on it can do.

Example. A CSV file of 50,000 customer records with 12 feature columns and one 'churned' column forms a dataset for a churn model.

Related: Training example, Training set, Test set, Ground truth

The cut-off applied to a model's predicted probability to turn it into a class decision, commonly 0.5. Moving it trades false alarms against misses, so it should be set according to the cost of each type of mistake.

Example. Flagging a loan as risky when the predicted default probability exceeds 0.3 instead of 0.5 catches more defaulters but rejects more good customers.

Related: Cost-sensitive learning, Decision boundary, Probabilistic model, Classification

A model that learns directly the boundary between classes, or the probability of a label given the inputs, without modelling how the inputs themselves are distributed. It usually gives better classification accuracy than generative approaches when plenty of labelled data exists.

Example. A logistic regression that outputs the probability an email is spam given its words does not model how emails are written.

Related: Generative model, Decision boundary, Classification, Probabilistic model

A change between the data a model was trained on and the data it faces later, so its learned patterns stop matching reality. Shifts can come from changing customers, seasons, sensors or policies, and they quietly degrade performance unless monitored.

Example. A demand model trained before a festival season sees very different buying patterns during it and loses accuracy.

Related: Covariate shift, Concept drift, Out-of-distribution, Independent and identically distributed

A pattern in which test error first falls, then rises as a model nears the point where it can just fit the training data exactly, and then falls again as the model grows even larger. It challenges the simple view that more capacity always hurts generalisation.

Example. Plotting test error against network size shows a peak at the size where the model barely memorises the data, then improvement beyond it.

Related: Bias-variance tradeoff, Overparameterisation, Model capacity, Overfitting

A regularisation method for iterative training: watch the error on a validation set and stop when it has stopped improving for a set number of rounds, keeping the best version seen. It prevents the model from continuing to memorise training data after it has learned the general pattern.

Example. Training halts at epoch 34 because validation loss has not fallen for 5 epochs, and the epoch-29 weights are kept.

Related: Regularisation, Overfitting, Validation set, Learning curve

The principle of choosing the model whose average loss on the training data is smallest, as a stand-in for its unknowable average loss on all future data. Most supervised training follows it. It works well only when the model class is not so flexible that it simply memorises the sample.

Example. Fitting a line by choosing slope and intercept that minimise the mean squared error over 1,000 training houses applies this principle.

Related: True risk, Structural risk minimisation, Overfitting, Training error

A single measurable input property that a model uses to make a prediction. Features can be numbers, categories, text pieces or pixels, and together form the description of each example. Choosing informative features often matters more to accuracy than choosing a fancier algorithm.

Example. For predicting house rent, features could be area in square feet, number of bedrooms, floor number and distance to the nearest metro station.

Related: Label, Feature space, Training example, Representation learning

The abstract space in which each example is a point and each feature is one axis. With 3 features data lives in 3-D space; with 300 it lives in 300 dimensions. Many algorithms are described as geometric operations in it, such as distances or separating surfaces.

Example. A customer with age 30, income 8 lakh and 4 orders is the point (30, 8, 4) in a three-dimensional space.

Related: Feature, Decision boundary, Curse of dimensionality, Dimensionality reduction

A model's ability to perform well on new data it has never seen, not just on the examples it learned from. It is the real goal of machine learning. A model that only memorises its training data is useless in practice, however impressive its training results look.

Example. A model trained on invoices from 2023 that still reads 2024 invoices accurately is generalising.

Related: Overfitting, Generalisation gap, Test set, Bias-variance tradeoff

The expected error of a model on new data drawn from the same source as its training data. It is approximated by error on a held-out test set, and it is what training is really trying to make small.

Example. A model with 4% training error and 11% error on a fresh test set has an estimated value of about 11%.

Related: Training error, True risk, Test set, Generalisation gap

The difference between a model's performance on data it trained on and on unseen data. A large gap signals memorisation of the training set. Tracking it is a quick way to spot trouble and decide whether more data, simpler models or regularisation would help.

Example. Accuracy of 99% on training data but 82% on validation data shows a gap of 17 percentage points.

Related: Generalisation, Overfitting, Training error, Generalisation error

A model that learns how the data itself is produced, typically the joint distribution of inputs and outputs, so it can create new samples resembling the training data as well as classify. Examples include language models and Gaussian mixture models.

Example. A model trained on Hindi news text learns which words tend to follow which, then generates new sentences.

Related: Discriminative model, Probabilistic model, Latent variable, Unsupervised learning

A tuning method that tries every combination from a fixed list of values for each hyperparameter. It is simple and thorough within the grid but the number of runs multiplies quickly as more hyperparameters are added.

Example. Three depths times four regularisation strengths gives 12 combinations, each scored with cross-validation.

Related: Random search, Hyperparameter tuning, Bayesian optimisation, Cross-validation

The verified correct answers against which a model's predictions are compared. It may come from expert annotation, measurements or recorded outcomes. It is not always perfect, so errors in it limit how reliable any evaluation can be.

Example. Doctors' confirmed diagnoses for 2,000 scans serve as the reference against which the model's outputs are checked.

Related: Label, Annotation, Label noise, Test set

A cross-validation method that keeps all rows belonging to one group, such as a single patient or user, together in the same fold. It stops a model from being tested on people whose other records it already saw during training, which would inflate scores.

Example. In a dataset with 5 scans per patient, all 5 scans of a patient stay either in training or in testing, never split across both.

Related: k-fold cross-validation, Data leakage, Train-test contamination, Cross-validation

The simplest way to estimate performance on new data: split the dataset once into a training part and a held-out part, train on the first and evaluate on the second. It is fast, but the estimate depends on which rows happened to land in each part, so it can be noisy on small datasets.

Example. An 80/20 random split of 5,000 records trains on 4,000 and evaluates on 1,000.

Related: Cross-validation, Training set, Test set, Stratified k-fold

A setting chosen by the practitioner before training rather than learned from data. It controls how the model is built or how learning proceeds, for example tree depth, regularisation strength or number of neighbours. Good values are found by trying options and checking validation performance.

Example. Setting the maximum tree depth to 6 or the regularisation strength to 0.1 are hyperparameter choices tried on the validation set.

Related: Parameter, Hyperparameter tuning, Grid search, Validation set

The process of searching for the hyperparameter values that give the best validation performance. Methods range from trying a grid of values to random sampling or smarter guided search. Because each trial means training a model, budget and time matter.

Example. Trying 20 combinations of tree depth and learning rate, each scored by 5-fold cross-validation, and keeping the best.

Related: Grid search, Random search, Bayesian optimisation, Hyperparameter

The full set of candidate functions a learning algorithm is able to choose from. A linear model can only pick straight-line relationships, while a deep network can pick vastly more complex ones. Its size affects both how well data can be fit and how much data is needed to generalise.

Example. Straight lines y = mx + c form the hypothesis space for simple linear regression, so a curved relationship cannot be represented exactly.

Related: Model capacity, Inductive bias, VC dimension, Overfitting

The standard assumption that every example is drawn separately from the same underlying source of data, and no example influences another. Most learning theory and validation methods rely on it, and real-world violations such as time trends or repeated users cause surprises.

Example. Customers drawn at random from one city's database roughly fit it; daily stock prices, which depend on the previous day, do not.

Related: Distribution shift, Generalisation, Group k-fold, Time-series split

The set of assumptions a learning algorithm makes so that it can choose one answer over the many that fit the training data. Without such assumptions, no learning beyond memorising is possible. Examples include preferring smooth functions, or assuming nearby points share a label.

Example. A nearest-neighbour method assumes points close together have similar labels; a convolutional network assumes patterns matter wherever they appear in an image.

Related: Hypothesis space, No free lunch theorem, Occam's razor, Generalisation

Using an already trained model to produce predictions for new inputs. It does not change the model's parameters. Speed, cost and reliability at this stage matter greatly in real products.

Example. Sending a new customer's details to the trained churn model and receiving a 0.31 churn probability is inference.

Related: Training, Model, Batch learning, Online learning

A measure of how consistently different people give the same label to the same items, often summarised with a statistic such as Cohen's kappa. Low agreement suggests unclear guidelines or an inherently subjective task, and puts a ceiling on model quality.

Example. Two annotators agree on 88 of 100 review labels; kappa adjusts that figure for agreement expected by chance.

Related: Annotation, Ground truth, Label noise, Label

The portion of prediction error that no model can remove, because the outcome depends on random factors or unmeasured influences. It sets a floor on achievable performance. Knowing it exists prevents wasted effort chasing perfection on inherently noisy targets.

Example. Two identical-looking customers may differ in whether they buy simply because of mood or timing that the data never records.

Related: Noise, Bias error, Model variance, Bias-variance tradeoff

The most common form of cross-validation. The data is divided into k equal folds; each fold takes one turn as the test portion while the other k-1 folds train the model. The k scores are averaged, and their spread hints at how stable the estimate is. Typical values of k are 5 or 10.

Example. With k = 5 and 1,000 rows, each round trains on 800 rows and tests on 200, giving five scores.

Related: Cross-validation, Stratified k-fold, Leave-one-out cross-validation, Holdout method

A penalty equal to the sum of the absolute values of the model weights, scaled by a strength setting. It tends to push some weights exactly to zero, so it effectively selects features and yields sparse models. In linear regression it is known as lasso.

Example. loss_total = loss + lambda x (|w1| + |w2| + ...); increasing lambda drives more weights to exactly zero.

Related: L2 regularisation, Regularisation, Feature, Overfitting

A penalty equal to the sum of the squared model weights, scaled by a strength setting and added to the loss. It pushes weights toward small values without usually making them exactly zero, giving smoother, more stable models. In linear regression it is called ridge regression.

Example. loss_total = loss + lambda x (w1^2 + w2^2 + ...); with lambda = 0.1 large weights are discouraged.

Related: L1 regularisation, Regularisation, Weight decay, Overfitting

The correct answer attached to a training example, which the model is asked to predict. For classification it is a category; for regression it is a number. Labels usually come from human annotation or recorded outcomes, and their quality places a ceiling on how good a supervised model can be.

Example. In a set of X-ray images, each image is labelled 'pneumonia' or 'normal' by a radiologist; those tags are the labels.

Related: Feature, Ground truth, Label noise, Supervised learning

Errors or inconsistency in the labels of a training set, caused by human mistakes, ambiguous cases or faulty automatic tagging. It can mislead learning and cap achievable accuracy, so cleaning labels often helps more than changing the model.

Example. About 5% of images in a cat-versus-dog dataset are tagged with the wrong animal.

Related: Ground truth, Annotation, Weak supervision, Overfitting

A hidden quantity that is never observed directly but is inferred from what is observed, and that helps explain the data. Models with such variables can capture underlying factors such as topics or customer types.

Example. A customer's true 'interest in fashion' is not recorded, but is inferred from browsing and purchase behaviour.

Related: Generative model, Representation learning, Dimensionality reduction, Probabilistic model

A plot of a model's error or score against the amount of training data, or against training time. Comparing the training and validation curves shows whether more data would help or whether the model is too simple or too flexible.

Example. If validation error is still falling at 5,000 examples while training error is low, collecting more data is likely to help.

Related: Validation curve, Overfitting, Underfitting, Generalisation gap

An extreme form of cross-validation where each single example takes a turn as the test set and the model trains on all the rest. It uses nearly all data each time and has little bias, but requires as many training runs as there are examples, so is only practical for small datasets or cheap models.

Example. With 50 patient records, the model is trained 50 times, each time predicting the one patient left out.

Related: k-fold cross-validation, Cross-validation, Holdout method, Model selection

A property of a labelled dataset meaning that a single straight line, or flat plane in higher dimensions, can perfectly split the classes. Real data rarely satisfies it, which is why flexible boundaries or feature transformations are often needed.

Example. Points of two classes on either side of the line x + y = 5 are separable; two classes arranged in concentric rings are not.

Related: Decision boundary, Feature space, Classification, Model capacity

A way of building software in which a program improves at a task by finding patterns in example data instead of following hand-written rules. The result is a fitted model that can make predictions on new inputs. It matters because many problems, such as spam filtering or fraud detection, are too messy to capture with explicit rules.

Example. Instead of coding rules for spotting fake UPI payment requests, a bank trains a program on thousands of labelled genuine and fraudulent transactions so it learns the patterns itself.

Related: Supervised learning, Unsupervised learning, Model, Training

A baseline that always predicts the most frequent class in the training data. It shows how much accuracy comes free from class proportions alone, and reveals why a high accuracy number can be meaningless on skewed data.

Example. If 90% of loan applicants repay, predicting 'repay' for everyone scores 90% accuracy without learning anything.

Related: Baseline model, Class imbalance, Classification, Decision threshold

The idea that real high-dimensional data such as images actually lies close to a much lower-dimensional curved surface within the full space. It explains why learning is possible despite huge feature counts and motivates methods that discover compact representations.

Example. Images of a rotating object have millions of pixels, yet only a few numbers, such as angle and lighting, describe the variation.

Related: Dimensionality reduction, Curse of dimensionality, Representation learning, Latent variable

Learning how to learn: training across many tasks so that a model can adapt quickly to a new task from very few examples. It is sometimes described as learning the learning procedure itself.

Example. A model trained on hundreds of small classification tasks can learn a new task, such as recognising a rare disease, from five examples.

Related: Transfer learning, Multi-task learning, Inductive bias, Generalisation

The mathematical function, along with its learned numbers, that maps inputs to predictions. An algorithm produces it by fitting data. Once trained, it can be saved and used to predict on new inputs without seeing the training data again.

Example. A fitted equation rent = 3,000 + 25 x area + 4,000 x bedrooms is a small model that estimates monthly rent from two features.

Related: Parameter, Hyperparameter, Training, Inference

A measure of how complicated a pattern a model is able to fit. Low-capacity models such as a straight line can only represent simple relationships; high-capacity ones such as large networks can fit almost anything, including noise. Matching capacity to the amount and complexity of data is central to avoiding both extremes of fit.

Example. A polynomial of degree 1 has low capacity; degree 15 fitted to 20 points has high capacity and can pass through every point.

Related: Hypothesis space, Overfitting, Underfitting, VC dimension

The task of choosing among candidate models or settings, using validation results or criteria such as information measures. It should be judged on data not used to fit the candidates, and reported honestly with a final untouched test.

Example. Comparing a linear model, a tree and a small neural network by 5-fold cross-validation, then picking the best.

Related: Cross-validation, Hyperparameter tuning, Akaike information criterion, Baseline model

The part of prediction error that comes from a model being too sensitive to the particular training sample. Retrain a high-variance model on a slightly different sample and its predictions change a lot. Flexible models with little data suffer most, and averaging or regularisation reduces it.

Example. Two deep trees trained on different random halves of the same customer data give quite different predictions for the same new customer.

Related: Bias error, Bias-variance tradeoff, Overfitting, Regularisation

A two-level cross-validation: an inner loop tunes hyperparameters and an outer loop measures performance of the whole tuning-plus-training procedure on data the tuning never touched. It avoids the optimistic estimate that arises when the same folds are used both to pick settings and to report the score.

Example. An outer 5-fold loop tests each model, while inside every outer training part a 3-fold loop chooses the best regularisation strength.

Related: Cross-validation, Hyperparameter tuning, Data snooping, Model selection

A theoretical result stating that, averaged over all possible problems, no learning algorithm is better than any other. It means an algorithm is only good to the extent its assumptions match the problem at hand, so practitioners must test several approaches on their own data.

Example. A method that excels on image data can perform worse than a simple rule on some tabular data; neither is best everywhere.

Related: Inductive bias, Model selection, Hypothesis space, Occam's razor

Random variation in data that does not reflect the real pattern, arising from measurement error, chance or unmeasured influences. Models that chase it overfit, so a good model separates the signal from it.

Example. Two thermometers reading 30.1 and 29.8 degrees for the same room differ because of measurement noise.

Related: Irreducible error, Outlier, Overfitting, Label noise

A model whose complexity grows with the amount of training data rather than being fixed in advance. It makes fewer assumptions about the shape of the relationship, but needs more data and can be slower or more memory-hungry at prediction time.

Example. A nearest-neighbour predictor keeps all training points and uses them directly, so it gets larger as data grows.

Related: Parametric model, Model capacity, Inductive bias, Instance-based learning

The principle that among explanations or models that fit the data about equally well, the simpler one should be preferred. In machine learning it motivates regularisation and model selection criteria, since simpler models tend to generalise better and are easier to interpret.

Example. If a 3-feature model and a 60-feature model both reach about 88% validation accuracy, the 3-feature one is the better choice.

Related: Regularisation, Model selection, Inductive bias, Bayesian information criterion

Training in which the model is updated continuously as new examples arrive, one at a time or in small groups, rather than being retrained from scratch on a fixed dataset. It suits streams of data and changing environments.

Example. A news recommender adjusts its parameters after each click instead of waiting for a nightly full retrain.

Related: Batch learning, Concept drift, Training, Inference

Describes inputs that differ substantially from the data a model was trained on. Models often behave unpredictably or confidently wrongly on such inputs, so detecting them is important for safe deployment.

Example. A model trained on clear daylight road photos is shown a snowstorm night image; that is far outside what it learned from.

Related: Distribution shift, Generalisation, Anomaly detection, Covariate shift

A data point that lies unusually far from the rest. It may be a recording error or a genuine rare event, and it can distort averages and fitted lines, so it must be examined before it is removed or kept.

Example. In a salary column mostly between Rs 3 and 12 lakh, an entry of Rs 4 crore is an outlier.

Related: Anomaly detection, Noise, Feature, Robustness

When a model learns the quirks and random noise of its training data so closely that it performs worse on new data. It shows up as very low training error but much higher error on held-out data. Causes include overly flexible models, too little data and too many features.

Example. A decision tree grown until every training row has its own leaf scores 100% on training data but only 70% on new customers.

Related: Underfitting, Regularisation, Bias-variance tradeoff, Early stopping

Having more adjustable parameters in a model than training examples. Such models can fit the training data perfectly, yet modern large networks often still generalise well, in part because of how training methods steer toward simpler solutions.

Example. A network with 10 million weights trained on 50,000 images is heavily overparameterised in this sense.

Related: Double descent, Model capacity, Overfitting, Inductive bias

A formal framework asking when an algorithm can, with high probability, find a model that is approximately correct after seeing a reasonable number of examples. It ties the needed sample size to accuracy demands and model class complexity, giving a mathematical basis for why learning from data can work at all.

Example. PAC results say roughly how many training examples suffice so that, with 95% confidence, a learned rule has error under 5%.

Related: VC dimension, Sample complexity, Generalisation, Hypothesis space

A number inside a model that is learned from data during training, such as a weight in a linear equation. The learning algorithm adjusts parameters to reduce error. Their values, not the code around them, hold what the model has learned.

Example. In price = 2.5 x area + 40, the values 2.5 and 40 are parameters fitted from data.

Related: Hyperparameter, Model, Parametric model, Training

A model that summarises the data in a fixed number of parameters chosen before seeing data, no matter how many training examples there are. It is compact and quick to use but makes strong assumptions about the form of the relationship.

Example. Linear regression with 5 features always has 6 parameters, whether it is trained on 100 or 100,000 rows.

Related: Non-parametric model, Parameter, Inductive bias, Model capacity

A chained sequence of steps, such as cleaning, scaling, feature creation and the model, packaged so that they run in order as one unit. It makes experiments repeatable and prevents leakage, since preprocessing is fitted only on the training part of each split.

Example. Pipeline([('scale', StandardScaler()), ('clf', LogisticRegression())]) scales and classifies in one fit call.

Related: Data leakage, Estimator, Training, Inference

A model that expresses its predictions as probabilities or distributions rather than single answers, capturing uncertainty. It allows decisions to weigh how confident the prediction is and to combine with prior knowledge.

Example. Instead of saying 'default', the model reports a 23% chance the applicant will default.

Related: Generative model, Discriminative model, Latent variable, Decision threshold

Turning a business need into a well-defined machine learning task: deciding what to predict, what data exists, how success is measured and what action follows. Good framing avoids building an accurate model for the wrong question.

Example. Reducing customer churn is framed as predicting which customers will leave in the next 30 days so that retention offers can be targeted.

Related: Label, Baseline model, Classification, Regression

A tuning method that samples hyperparameter combinations at random from chosen ranges instead of trying every grid point. It often finds good settings faster than exhaustive search because usually only a few hyperparameters matter much, and random draws explore those more varied values.

Example. Draw 30 random pairs of learning rate between 0.001 and 1 and depth between 2 and 12, then keep the best-scoring pair.

Related: Grid search, Hyperparameter tuning, Bayesian optimisation, Hyperparameter

A starting number given to a random number generator so that random steps, such as shuffling or weight initialisation, produce the same sequence each time. Fixing it makes experiments repeatable, and varying it shows how sensitive results are to chance.

Example. Setting random_state = 42 when splitting data gives the same train and test rows on every run.

Related: Reproducibility, Holdout method, Training, Model variance

A supervised task where the goal is to predict a continuous number rather than a category. The model learns how the inputs relate to the numeric target and outputs an estimate for new cases. Errors are measured by how far predictions fall from actual values, for example with mean squared error.

Example. Predicting the resale price in rupees of a used car from its age, kilometres driven and brand is a regression problem.

Related: Classification, Supervised learning, Label, Loss function

Any technique that deliberately restricts or penalises a model's complexity so it fits noise less and generalises better. Typical methods add a penalty on large weights to the loss, stop training early or drop parts of the network. It trades a little training accuracy for better performance on new data.

Example. Adding a penalty of 0.01 times the sum of squared weights to the loss keeps a regression model from using extreme coefficients.

Related: L1 regularisation, L2 regularisation, Early stopping, Overfitting

Methods where a model learns useful features from raw data automatically, rather than a person designing them by hand. The learned representations can then be reused for several tasks, which is the basis of modern deep learning and pretrained models.

Example. Instead of hand-crafting edge detectors, a model learns internal features from millions of raw images and reuses them for new tasks.

Related: Self-supervised learning, Transfer learning, Latent variable, Feature

The ability to obtain the same results again from the same data, code and settings. It requires recording versions, seeds and configurations, and it is essential for trusting experiments and debugging.

Example. Logging the dataset version, code commit and random seed lets a colleague rerun your experiment and get the same 0.87 score.

Related: Random seed, Baseline model, Ablation study, Data snooping

A distortion that arises when the examples in a dataset are not representative of the population the model will serve, because of how they were collected or chosen. Models trained on such data can fail on the groups that were left out.

Example. A survey taken only online overrepresents younger, urban respondents and misses many rural households.

Related: Survivorship bias, Covariate shift, Generalisation, Dataset

Learning in which the training signal is created from the data itself rather than from human labels. A model hides or corrupts part of the input and learns to predict it, such as the next word in a sentence. It lets models learn from huge unlabelled collections and underpins modern language and vision systems.

Example. A language model is shown 'The monsoon arrives in ___' and learns to predict the missing word, using only raw text with no human annotation.

Related: Unsupervised learning, Representation learning, Transfer learning, Semi-supervised learning

Learning that combines a small amount of labelled data with a large amount of unlabelled data. The unlabelled examples reveal the shape of the input distribution, which helps the model draw better boundaries than the few labels could alone. It is attractive when labelling requires expensive expert time.

Example. A hospital has 500 X-rays labelled by radiologists and 50,000 unlabelled ones; a model uses both, for example by assigning confident guesses to the unlabelled images and retraining.

Related: Supervised learning, Weak supervision, Active learning, Label

When a model solves a benchmark by using easy but unintended cues in the data rather than the skill the designers wanted, and then fails in slightly different settings. It is a common cause of impressive tests but disappointing real-world results.

Example. A model identifies cows only by the green grass background and fails on a cow on a beach.

Related: Spurious correlation, Distribution shift, Out-of-distribution, Generalisation

A pattern in the training data that predicts the label by coincidence or through an unrelated factor, not because of a real causal link. Models that rely on it break when conditions change.

Example. In a set of X-rays, a hospital tag in the corner correlates with disease because sicker patients went to that hospital.

Related: Shortcut learning, Distribution shift, Generalisation, Overfitting

A version of k-fold cross-validation that keeps the proportion of each class roughly the same in every fold as in the full data. It prevents a fold from having too few or no examples of a rare class, which would make scores misleading.

Example. If 5% of 2,000 transactions are fraud, every fold has about 20 fraud cases out of 400 instead of some folds getting almost none.

Related: k-fold cross-validation, Class imbalance, Cross-validation, Group k-fold

A principle that chooses a model by balancing how well it fits the training data against how complex its model class is, rather than only minimising training error. It underlies regularisation and margin-based methods and links to theoretical bounds on generalisation.

Example. Choosing the degree of a polynomial by minimising training error plus a penalty that grows with degree.

Related: Empirical risk minimisation, Regularisation, VC dimension, Model capacity

Learning from examples where every input comes with the correct answer attached. The algorithm studies these pairs and learns a mapping it can apply to new inputs whose answers are unknown. It covers the two most common tasks, predicting categories and predicting numbers, and needs labelled data, which can be costly to produce.

Example. A dataset of 10,000 home-loan applications, each marked 'repaid' or 'defaulted', is used to teach a model to predict default for new applicants.

Related: Unsupervised learning, Label, Classification, Regression

A form of data leakage in which a feature is directly or indirectly produced by the outcome being predicted, or recorded after it. The model then learns a shortcut that is impossible to use when making genuine predictions.

Example. Predicting hospital readmission with a column 'discharge medication changed after readmission' gives the answer away.

Related: Data leakage, Temporal leakage, Feature, Label

Leakage that occurs when data from the future is used to predict the past, for example by randomly shuffling time-ordered data before splitting or by computing features with later information. It makes forecasts look far more accurate than they can ever be live.

Example. Using the full-year average sales as a feature for predicting sales in March lets the model know about December.

Related: Time-series split, Data leakage, Target leakage, Concept drift

Data reserved for one final, honest estimate of how the chosen model performs on unseen cases. It must not influence training or tuning decisions in any way. If you keep peeking at it and adjusting, its estimate becomes too optimistic.

Example. After choosing a model using validation results, you score it once on the 1,500 untouched test emails and report that accuracy.

Related: Training set, Validation set, Data snooping, Generalisation error

A validation scheme for data ordered in time in which the model is always trained on earlier periods and tested on later ones, never the reverse. Random shuffling would let the model see the future, so this ordered approach gives a realistic estimate of forecasting performance.

Example. Train on January to June, test on July; then train on January to July, test on August, and so on.

Related: Temporal leakage, Cross-validation, Data leakage, Holdout method

When information from the test data influences training, whether through preprocessing fitted on all rows, duplicate records across the split or repeated tuning against the test set. It makes reported performance unrealistically high.

Example. Computing the mean used for scaling on the whole dataset before splitting lets test statistics shape the training features.

Related: Data leakage, Data snooping, Test set, Group k-fold

The process of fitting a model to data by adjusting its parameters so that its predictions become closer to the known answers. It repeats until improvement stops or a budget is used up, and produces the model later used for prediction.

Example. Running the fitting routine on 7,000 labelled emails until validation error stops improving produces the spam model.

Related: Inference, Model, Parameter, Training set

The average error a model makes on the same data it was fitted to. It is easy to make small by using a very flexible model, so on its own it says little about how the model will do on new cases.

Example. A tree that memorises 1,000 rows has 0% training error yet may make many mistakes on new rows.

Related: Generalisation error, Generalisation gap, Overfitting, Training set

One record in a dataset: the feature values for a single case, plus its label when the task is supervised. A dataset is a collection of such records. Also called an instance, sample or observation, it is the basic unit a learning algorithm learns from.

Example. One row of a loan dataset: age 34, monthly income Rs 60,000, existing loans 1, outcome 'repaid'.

Related: Dataset, Feature, Label, Training set

The portion of the data the algorithm is allowed to learn from. The model's parameters are adjusted to fit these examples. Because the model has seen them, performance measured on this portion is optimistic and cannot show how well the model will handle new data.

Example. From 10,000 labelled emails, 7,000 go into the training set so the spam model can learn from them.

Related: Validation set, Test set, Training error, Overfitting

Reusing a model trained on one task or dataset as the starting point for a different but related task. It saves data, time and compute because the model already knows general patterns, and only needs adapting to the new problem.

Example. A network trained on millions of general photos is fine-tuned on 800 photos of Indian crop diseases.

Related: Representation learning, Self-supervised learning, Multi-task learning, Meta-learning

The expected loss a model would incur over the whole distribution of data it could ever meet, not just the sample it was trained on. It is the quantity we really care about but can never compute exactly; test data gives an estimate of it.

Example. A spam filter's true risk is its average error over every email that will ever arrive, whereas 2,000 test emails give only an estimate.

Related: Empirical risk minimisation, Generalisation error, Test set, Training error

When a model is too simple or too little trained to capture the real pattern in the data, so it performs poorly even on its own training set. Fixes include using a more flexible model, adding informative features or reducing regularisation.

Example. Fitting a straight line to data that clearly follows a curve leaves large errors on both training and test data.

Related: Overfitting, Model capacity, Bias error, Learning curve

Learning structure from data that has no answers attached. The algorithm looks for groupings, compressed representations or unusual points on its own. It is useful for exploring data and for cases where labelling is too expensive, but there is no single correct answer to check the result against.

Example. A retailer feeds purchase histories of 2 lakh customers into a grouping algorithm and discovers a cluster of festival-only buyers nobody had defined beforehand.

Related: Supervised learning, Dimensionality reduction, Anomaly detection, Self-supervised learning

A plot of training and validation scores as one hyperparameter varies, such as tree depth or regularisation strength. It reveals the range where the model underfits, the range where it overfits and the sweet spot between.

Example. Plotting scores for depth 1 to 15 shows validation accuracy peaking at depth 6 while training accuracy keeps climbing.

Related: Learning curve, Hyperparameter tuning, Overfitting, Underfitting

A portion of data kept apart from training and used to compare models and tune settings such as hyperparameters. Because decisions are made using it, it gradually stops being an unbiased check, which is why a separate final test set is still needed.

Example. With 10,000 emails, 1,500 form a validation set used to decide whether a tree depth of 5 or 10 works better.

Related: Training set, Test set, Hyperparameter tuning, Model selection

A theoretical measure of a model class's flexibility: the largest number of points it can label in every possible way without error. Higher values mean a richer class that needs more data to generalise reliably. It connects capacity to guarantees on the gap between training and test error.

Example. A straight line in a plane can separate any 3 points in every labelling but not all labellings of 4 points, so its value is 3.

Related: Model capacity, Hypothesis space, PAC learning, Sample complexity

Building training labels cheaply from imperfect sources such as rules, heuristics, existing databases or other models rather than from careful manual labelling. The labels are noisy, but at scale they can train useful models, often after their errors are modelled and combined.

Example. Tagging support tickets as 'refund' whenever they contain the word 'refund' or 'money back' produces rough labels for thousands of tickets in minutes.

Related: Label noise, Semi-supervised learning, Active learning, Annotation