Evaluation metrics

The fraction of all predictions that are correct: (TP + TN) / (TP + TN + FP + FN). It is simple and easy to explain, but it misleads when one class is far more common than the other, because a model that always predicts the majority class can still score very high.

Example. If a model gets 985 of 1,000 transactions right, its accuracy is 0.985 or 98.5%, even though it may have missed a quarter of the frauds.

Related: Accuracy paradox, Balanced accuracy, Confusion matrix, F1 score

The situation where a model with high accuracy is useless because the classes are heavily imbalanced. A model that always predicts the common class scores well on accuracy while never finding the rare class the task is about. It is why precision, recall and related metrics are preferred for imbalanced problems.

Example. If only 1% of 10,000 loan applicants default, predicting 'no default' for everyone gives 99% accuracy but catches zero defaulters.

Related: Accuracy, Recall, Balanced accuracy, Majority-class baseline

A measure of agreement between two ways of grouping the same items, such as clusters versus true classes, that counts pairs placed together or apart in both and corrects for chance. It is 1 for identical groupings, about 0 for random ones, and can be negative.

Example. K-means clusters of 1,000 customers are compared with known segments; an ARI of 0.72 indicates strong agreement well beyond chance.

Related: Normalised mutual information, V-measure, Silhouette score, K-means

The area under the ROC curve, a number from 0 to 1 that summarises ranking quality across all thresholds. It equals the probability that a randomly chosen positive case gets a higher score than a randomly chosen negative one. 0.5 is random and 1.0 is perfect separation.

Example. An AUC of 0.85 means that if you pick one defaulter and one non-defaulter at random, the model scores the defaulter higher 85% of the time.

Related: ROC curve, Gini coefficient, Average precision, Partial AUC

A single number that summarises the precision-recall curve, computed as the weighted mean of the precision achieved at each threshold, with the weights being the increase in recall. Higher is better, and it is preferred to ROC AUC when the positive class is rare.

Example. A rare-defect detector scoring average precision 0.72 on a test set is doing much better than one at 0.30, even if both have ROC AUC above 0.9.

Related: Precision-recall curve, Mean average precision, AUC-ROC, Recall

The average of recall and specificity: (TPR + TNR) / 2. It gives each class equal weight regardless of how many examples it has, so a model cannot look good simply by favouring the majority class. It is a simple fix for accuracy on imbalanced data.

Example. With recall 0.75 and specificity 0.99, balanced accuracy is about 0.87, far below the 98.5% plain accuracy on the same predictions.

Related: Accuracy, Recall, Specificity, Accuracy paradox

A text-generation metric that embeds each word of the candidate and reference with a pretrained contextual language model and matches words by cosine similarity, then reports precision, recall and F1. It credits paraphrases with different wording, which pure word overlap cannot.

Example. 'The film was fantastic' versus the reference 'The movie was great' scores low on word overlap but high on BERTScore, since the embeddings of matching words are close.

Related: BLEU, ROUGE, METEOR, Cosine similarity

A language-model quality measure that converts the model's total negative log-likelihood, in base 2, into bits divided by the number of raw bytes or characters in the text. Because the divisor is fixed by the text, not by the tokeniser, it allows fair comparison between models that split text differently.

Example. Two models with different vocabularies cannot be compared by per-token perplexity, but both can report bits per byte on the same 1 MB of text.

Related: Perplexity, Log loss, Entropy, Entropy

A metric for machine translation that compares a system's output to one or more human reference translations. It computes modified n-gram precision (typically for 1- to 4-word sequences), combines them with a geometric mean and multiplies by a brevity penalty for outputs that are too short. Scores run 0 to 1, or 0 to 100.

Example. A Hindi-to-English system output shares many 1- and 2-word sequences with the reference but few 4-word ones, so its BLEU is moderate. Very short outputs get the brevity penalty.

Related: ROUGE, METEOR, chrF, BERTScore

The mean squared difference between predicted probabilities and the actual 0/1 outcomes: (1/N) * sum of (p - y)^2. It ranges from 0 to 1 for binary events, lower is better, and it rewards predictions that are both accurate and well calibrated. It is a proper scoring rule.

Example. A forecaster says 80% chance of rain and it rains (y = 1): the squared error is (0.8 - 1)^2 = 0.04.

Related: Log loss, Calibration, Proper scoring rule, Reliability diagram

The property that a model's stated confidence matches how often it is actually right. A well-calibrated classifier that says 70% should be correct on about 70% of such predictions. It matters wherever probabilities feed decisions, such as risk pricing or triage.

Example. Among 1,000 loan applicants each given a 20% default probability, roughly 200 should default if the model is calibrated.

Related: Reliability diagram, Expected calibration error, Platt scaling, Brier score

An internal clustering measure equal to the ratio of between-cluster dispersion to within-cluster dispersion, adjusted for the number of clusters and points. Higher values indicate dense, well-separated clusters, and it is used to compare numbers of clusters without labels.

Example. Running K-means for k = 2 to 8 and picking the k with the highest Calinski-Harabasz score is a common heuristic.

Related: Davies-Bouldin index, Silhouette score, K-means, Adjusted Rand index

The fraction of items in a catalogue that a recommender ever recommends over a test period. Low coverage means the system keeps surfacing the same popular items and ignores the long tail, even if its accuracy metrics look fine.

Example. If a shop lists 10,000 products but the recommender only ever shows 2,000 of them, catalogue coverage is 20%.

Related: Hit rate, Recall at k, Normalised discounted cumulative gain, Precision at k

The same edit-distance idea as word error rate applied to characters: (substitutions + deletions + insertions) / number of reference characters. It suits handwriting and OCR tasks and languages without clear word boundaries, and it gives partial credit for near-miss words.

Example. An OCR system reads 'Bengaluru' as 'Bengalura'; that is 1 substitution over 9 characters, so CER = 1 / 9, about 11%.

Related: Word error rate, Translation edit rate, chrF, Exact match

A metric for generated text that computes an F-score over overlapping character n-grams instead of words. Character matching handles rich word endings and spelling variation, so it suits morphologically rich languages such as Hindi, Tamil or Finnish. The variant chrF++ also adds word n-grams.

Example. A Tamil output with a slightly different word ending still gets partial credit under chrF because most character sequences match.

Related: BLEU, METEOR, Translation edit rate, BERTScore

A metric for image captioning that compares a generated caption to several human captions using TF-IDF weighted n-gram similarity. Words common across all images get low weight, so distinctive, informative words that humans agree on drive the score.

Example. A caption 'a red auto-rickshaw on a busy road' scores well if several human captions also mention 'auto-rickshaw', a distinctive word, more than for generic words like 'a'.

Related: BLEU, METEOR, ROUGE, Cosine similarity

A measure of agreement between two raters, or between a model and the true labels, that corrects for the agreement expected by chance: kappa = (observed agreement - chance agreement) / (1 - chance agreement). A value of 1 is perfect agreement and 0 is chance level. It is useful when classes are imbalanced.

Example. Two doctors label 100 scans and agree on 90, but chance alone would give 70 agreements, so kappa = (0.90 - 0.70) / (1 - 0.70), about 0.67.

Related: Weighted kappa, Inter-annotator agreement, Fleiss' kappa, Matthews correlation coefficient

A table that counts how a classifier's predictions line up with the true labels. For two classes it has four cells: true positives, false positives, false negatives and true negatives. Nearly every classification metric is computed from these counts, and the table shows which kinds of mistakes a model makes, which a single accuracy number hides.

Example. A fraud model reviews 1,000 UPI transactions, 20 of them fraud. It catches 15 (TP), misses 5 (FN), wrongly flags 10 genuine ones (FP) and correctly clears 970 (TN).

Related: True positive, False positive, Accuracy, Recall

A measure of how good a full probabilistic forecast is, comparing the predicted cumulative distribution to the single value that occurred. Lower is better. It generalises mean absolute error to distributions, and for a single-number forecast it reduces to the absolute error.

Example. Two weather models both centre on 30 degrees C, but the one whose forecast distribution is narrower and correct gets the lower CRPS when the actual temperature is 31.

Related: Pinball loss, Brier score, Proper scoring rule, Mean absolute error

A table assigning a price to each type of outcome, such as the cost of a false positive versus a false negative. Multiplying it with the confusion matrix gives an expected cost, which lets a team choose a threshold or compare models by business impact instead of a generic score.

Example. A bank estimates a missed fraud costs Rs 20,000 and a wrongly blocked payment costs Rs 500, so 5 misses and 10 false alarms cost 5 * 20,000 + 10 * 500 = Rs 1,05,000.

Related: Confusion matrix, Operating point, Decision threshold, F-beta score

The classic method for evaluating search systems using a fixed test collection: a set of documents, a set of queries and human relevance judgments. Any system can be re-run against it and scored repeatably, which makes results comparable across systems and over time.

Example. To compare two search engines, run the same 100 queries on both against one document set and judgments file, then compare their MAP scores.

Related: Relevance judgments, Mean average precision, Normalised discounted cumulative gain, Hit rate

A chart that ranks cases from highest to lowest predicted score and plots the share of all true positives captured against the share of the population contacted. It shows how much a model beats random selection and is common in marketing and credit risk.

Example. If a marketing model's top 20% of customers by score contain 60% of all buyers, the gains chart passes through the point (20%, 60%).

Related: Lift, Gini coefficient, ROC curve, Recall

An internal clustering measure that needs no true labels: for each cluster it finds the worst ratio of the sum of two clusters' spreads to the distance between their centres, then averages. Lower values mean compact, well-separated clusters.

Example. Comparing k = 3, 4 and 5 clusters on customer data, the choice with the lowest Davies-Bouldin index is preferred.

Related: Silhouette score, Calinski-Harabasz index, K-means, Purity

A single number for a test's discriminating power: the odds of a positive result in truly positive cases divided by the odds in truly negative cases, equal to (TP * TN) / (FP * FN). Higher is better, and 1 means the test tells nothing.

Example. For TP = 15, TN = 970, FP = 10, FN = 5, DOR = (15 * 970) / (10 * 5) = 291.

Related: Likelihood ratio, Confusion matrix, Odds ratio, Youden's J statistic

A ranking score that sums the relevance grade of each result divided by the log of its position: DCG = sum of rel_i / log2(i + 1). Highly relevant items near the top add the most, and gains shrink further down the list. It handles graded, not just yes-or-no, relevance.

Example. Relevance grades 3, 2, 0 at ranks 1, 2, 3 give DCG = 3/1 + 2/1.585 + 0 = about 4.26.

Related: Normalised discounted cumulative gain, Mean average precision, Expected reciprocal rank, Rank-biased precision

An internal clustering measure equal to the smallest distance between points in different clusters divided by the largest distance across points within a single cluster. Higher values indicate compact, well-separated clusters. It is sensitive to outliers because it depends on extreme distances.

Example. If clusters are at least 6 units apart but the widest cluster spans 3 units, the Dunn index is 6 / 3 = 2.

Related: Davies-Bouldin index, Calinski-Harabasz index, Silhouette score, K-means

The practice of manually inspecting a model's mistakes, grouping them by type or cause, and counting each group, in order to decide what to fix next. It turns an aggregate score into concrete actions such as collecting more data for one case or fixing a labelling problem.

Example. Reviewing 100 wrong predictions of an invoice reader shows 40 come from blurry scans, so improving image quality is prioritised over changing the model.

Related: Confusion matrix, Slice-based evaluation, Recall, Precision

The fraction of predictions that are identical to a reference answer, usually after normalising case, punctuation and spacing. It is strict and unambiguous, which suits questions with a single short answer, but it gives no credit for correct answers worded differently.

Example. On 200 trivia questions, a model whose answer string matches the reference on 130 gets an exact match score of 65%.

Related: Token-level F1, Accuracy, ROUGE, Hamming loss

A single number summarising miscalibration: predictions are split into bins, and the gap between each bin's average confidence and accuracy is averaged, weighted by bin size: ECE = sum of (n_b / N) * |accuracy_b - confidence_b|. Lower is better, and 0 is perfect.

Example. If a model is 90% confident on a bin holding half the data but only 80% accurate there, that bin contributes 0.5 * 0.10 = 0.05 to ECE.

Related: Maximum calibration error, Reliability diagram, Calibration, Temperature scaling

A ranking metric built on a cascade user model: the reader scans down the list and stops once satisfied, with the chance of satisfaction growing with an item's relevance grade. The score is the expected reciprocal of the position where the user stops. It rewards a strong result early and discounts redundancy.

Example. If a very relevant page sits at rank 1, most simulated users stop there, so later pages add little to the ERR score.

Related: Normalised discounted cumulative gain, Reciprocal rank, Rank-biased precision, Mean reciprocal rank

A regression metric equal to 1 - Var(y - y_hat) / Var(y): the fraction of the target's variance that the model's predictions account for. A value of 1 is perfect. Unlike R-squared it ignores a constant offset in the errors, so a biased model can still score well.

Example. If the target's variance is 100 and the variance of the model's errors is 20, the explained variance score is 0.8.

Related: R-squared, Mean bias error, Mean squared error, Residual

A generalisation of the F1 score that weights recall beta times as much as precision: F_beta = (1 + beta^2) * P * R / (beta^2 * P + R). Beta above 1 favours recall (F2 is common), while beta below 1 favours precision (F0.5). It lets the metric reflect which error is costlier.

Example. For a cancer-screening model where misses are worse than false alarms, a team reports F2 (beta = 2) so that recall counts four times as much as precision in the formula's weighting.

Related: F1 score, Precision, Recall, Cost matrix

The harmonic mean of precision and recall: F1 = 2 * precision * recall / (precision + recall). Because a harmonic mean is dominated by the smaller number, it is high only when both precision and recall are decent. It gives a single figure for imbalanced classification where accuracy is unhelpful.

Example. With precision 0.60 and recall 0.75, F1 = 2 * 0.60 * 0.75 / 1.35, about 0.667.

Related: Precision, Recall, F-beta score, Macro averaging

A case that truly belongs to the class of interest but that the model labels as not belonging. It is a missed detection. In screening, safety and fraud settings a missed case is often costlier than a false alarm, and they drive down recall.

Example. A tuberculosis-screening model clears a patient's chest X-ray who actually has the disease. That missed case is a false negative.

Related: False positive, Recall, False negative rate, Confusion matrix

The share of real positive cases that the model misses: FN / (FN + TP). It equals one minus recall. It is used when a regulator or clinician cares about the proportion of true cases that slip through undetected.

Example. The fraud model misses 5 of 20 frauds, so FNR = 5 / 20 = 0.25.

Related: Recall, False negative, False positive rate, Detection error tradeoff curve

A case that does not belong to the class of interest but that the model wrongly labels as belonging. It is a false alarm. Each false positive costs something real, such as a blocked genuine payment or an unnecessary medical test, and they drive down precision.

Example. A bank's fraud model blocks a customer's genuine Rs 45,000 rent payment. The customer is annoyed and calls support.

Related: False negative, Precision, False positive rate, Confusion matrix

The share of real negative cases that the model wrongly flags as positive: FP / (FP + TN). It equals one minus specificity and forms the horizontal axis of the ROC curve. It tells you how often the system raises false alarms among cases that were actually fine.

Example. 10 genuine transactions are blocked out of 980 genuine ones, so FPR = 10 / 980, about 0.0102 or 1%.

Related: Specificity, ROC curve, False positive, Recall

An extension of Cohen's kappa that measures chance-corrected agreement among a fixed number of raters (more than two) assigning categories to items. Each item need not be rated by the same people, but each must have the same number of ratings. 1 is perfect agreement and 0 is chance level.

Example. Five annotators each label 300 reviews as positive, neutral or negative; Fleiss' kappa summarises how far their agreement exceeds chance.

Related: Cohen's kappa, Krippendorff's alpha, Inter-annotator agreement, Weighted kappa

The geometric mean of recall and specificity: sqrt(TPR * TNR). Like balanced accuracy it treats both classes equally, but it drops sharply if either rate is very low, so it punishes a model that ignores the minority class. It is common in imbalanced-learning research.

Example. With recall 0.75 and specificity 0.99, G-mean = sqrt(0.75 * 0.99), about 0.86.

Related: Balanced accuracy, Recall, Specificity, Accuracy paradox

In credit scoring and ranking models, a rescaling of AUC: Gini = 2 * AUC - 1. It runs from 0 for random ranking to 1 for perfect ranking. Lenders quote it to describe how well a scorecard separates good and bad borrowers.

Example. A credit scorecard with AUC 0.80 has a Gini coefficient of 2 * 0.80 - 1 = 0.60.

Related: AUC-ROC, Kolmogorov-Smirnov statistic, Cumulative gains chart, ROC curve

For multi-label classification, the fraction of individual label slots that are predicted wrongly, averaged over all examples and labels. It counts each wrong label separately instead of demanding that the whole label set match, so it is more lenient than exact-match accuracy.

Example. 4 news articles each have 5 possible topic tags. If 3 of the 20 tag decisions are wrong, Hamming loss = 3 / 20 = 0.15.

Related: Exact match, Jaccard index, F1 score, Accuracy

The fraction of queries or users for whom at least one relevant item appears in the top k results. It is a simple yes-or-no success measure used in recommendation and retrieval evaluation.

Example. A recommender is tested on 100 users with one held-out favourite film each; that film is in the top 10 for 70 users, so hit rate at 10 is 0.70.

Related: Recall at k, Precision at k, Mean reciprocal rank, Top-k accuracy

Having people rate or compare model outputs on criteria like fluency, accuracy or helpfulness, often on a Likert scale or through pairwise preference. It is slow and costly, but remains the reference standard against which automatic metrics are validated.

Example. Three annotators rate 200 chatbot answers from 1 to 5 for helpfulness, and their average is reported.

Related: Inter-annotator agreement, BLEU, Spearman's rank correlation, Cohen's kappa

A calibration method that fits a monotonic, step-shaped function mapping raw scores to probabilities, using held-out data. It is more flexible than Platt scaling and can fix distorted probability shapes, but needs more data and can overfit small sets.

Example. On a 50,000-row validation set, isotonic calibration maps scores to probabilities with a staircase curve that never decreases.

Related: Platt scaling, Temperature scaling, Calibration, Reliability diagram

A measure of overlap between two sets: the size of their intersection divided by the size of their union. Applied to classification it equals TP / (TP + FP + FN), ignoring true negatives. It runs from 0 (no overlap) to 1 (identical sets).

Example. A model predicts tags {python, sql, git} and the true tags are {python, sql, docker}. The intersection has 2 items and the union 4, so the Jaccard index is 0.5.

Related: Dice coefficient, F1 score, Hamming loss, Exact match

A measure of how similar two rankings are, based on pairs: (concordant pairs - discordant pairs) / total pairs. A pair is concordant if both rankings put the two items in the same order. It ranges from -1 to +1 and is used to compare rankings, such as a model's ordering against ground truth.

Example. Comparing two orderings of 5 items gives 10 pairs; 8 agree and 2 disagree, so tau = (8 - 2) / 10 = 0.6.

Related: Spearman's rank correlation, Normalised discounted cumulative gain, Mean average precision, Correlation coefficient

The largest vertical gap between two cumulative distribution curves. In scoring models it compares the score distributions of positives and negatives: a larger gap means better separation. In credit risk it is a common single-number check of scorecard quality.

Example. If at some score cutoff 80% of defaulters but only 30% of non-defaulters lie below it, the KS statistic is 0.50.

Related: Gini coefficient, AUC-ROC, ROC curve, Cumulative gains chart

A flexible agreement measure that works with any number of raters, missing ratings and different types of scale (nominal, ordinal, interval or ratio). It is defined as 1 minus the ratio of observed to expected disagreement. It is often preferred for real annotation projects with incomplete data.

Example. In a project where each of 1,000 sentences is rated by only 3 of 8 annotators, alpha can still be computed, while simpler kappas cannot.

Related: Fleiss' kappa, Cohen's kappa, Inter-annotator agreement, Weighted kappa

Metrics that say how much a test result shifts the odds of the condition. The positive ratio is sensitivity / (1 - specificity) and the negative ratio is (1 - sensitivity) / specificity. They do not depend on prevalence, so they transfer across populations.

Example. With sensitivity 0.75 and specificity 0.99, the positive likelihood ratio is about 73, meaning a positive flag makes fraud about 73 times more likely in odds terms.

Related: Diagnostic odds ratio, Base rate fallacy, Bayes' theorem, Specificity

A metric for probabilistic classifiers that averages the negative natural log of the probability assigned to the true class: -(1/N) * sum of ln(p_true). It heavily penalises confident wrong predictions and rewards well-calibrated probabilities. Lower is better, and 0 is perfect.

Example. If the true class gets probability 0.9 the loss is -ln(0.9), about 0.105; if it gets 0.1 the loss is about 2.30.

Related: Brier score, Calibration, Proper scoring rule, Accuracy

A way to combine per-class metrics in multiclass problems by computing the metric separately for each class and taking a plain average, so every class counts equally regardless of size. It exposes poor performance on rare classes that micro averaging would hide.

Example. Three classes have recall 0.9, 0.9 and 0.3. The macro-averaged recall is 0.7, even if the weak class is tiny.

Related: Micro averaging, Weighted averaging, F1 score, Confusion matrix

A single score for binary classification that uses all four confusion-matrix cells: (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)). It ranges from -1 to +1, where 0 means no better than chance. It stays informative on imbalanced data where accuracy and F1 can mislead.

Example. For TP = 15, FN = 5, FP = 10, TN = 970 the score is 14,500 / sqrt(25 * 20 * 980 * 975), about 0.66.

Related: F1 score, Confusion matrix, Balanced accuracy, Cohen's kappa

A metric for open-ended text generation that measures how close the distribution of machine-generated text is to human-written text. It embeds both sets, groups them into clusters and compares cluster proportions along a divergence frontier, capturing both low-quality and low-diversity output. Higher is better.

Example. Comparing 5,000 story continuations from a model with 5,000 human ones, a MAUVE near 1 suggests the model's text is statistically hard to tell apart from human writing.

Related: Perplexity, BERTScore, Human evaluation, BLEU

The single largest absolute difference between a prediction and its true value across the test set. It captures the worst case and is used where one big miss is unacceptable, for instance in safety or engineering tolerances.

Example. With absolute errors 5, 2, 2 and 20 lakh, the max error is 20 lakh.

Related: Mean absolute error, Root mean squared error, Median absolute error, Residual

The largest gap between confidence and accuracy over all confidence bins, rather than the average gap. It measures the worst-case miscalibration and is relevant for high-stakes uses where a single badly calibrated confidence range could cause harm.

Example. If bin gaps are 0.02, 0.05 and 0.18, the ECE-style average is small but the maximum calibration error is 0.18.

Related: Expected calibration error, Reliability diagram, Calibration, Brier score

A statistical test for comparing two classifiers evaluated on the same test set. It looks only at the cases where exactly one model is right and asks whether the two disagreement counts differ more than chance would explain. It suits paired binary outcomes.

Example. Model A is right and B wrong on 30 cases; B is right and A wrong on 12. McNemar's test asks whether 30 versus 12 is too lopsided to be chance.

Related: P-value, Accuracy, Confusion matrix, P-value

The average size of the prediction errors, ignoring their direction: MAE = (1/N) * sum of |y - y_hat|. It is in the target's own units and treats every unit of error equally, so it is less swayed by outliers than squared-error metrics.

Example. For actual [50, 60, 70, 80] and predictions [55, 58, 72, 100], the absolute errors are 5, 2, 2, 20, so MAE = 29 / 4 = 7.25.

Related: Root mean squared error, Median absolute error, Mean absolute percentage error, Mean squared error

The average of the absolute errors expressed as a percentage of the true values: (100/N) * sum of |y - y_hat| / |y|. It is easy to communicate but breaks down when true values are zero or near zero, and it penalises over-forecasts more than under-forecasts.

Example. For actuals 50, 60, 70, 80 and predictions 55, 58, 72, 100 the percentage errors are 10%, 3.3%, 2.9% and 25%, giving MAPE of about 10.3%.

Related: Symmetric mean absolute percentage error, Mean absolute error, Mean absolute scaled error, Root mean squared error

A scale-free forecasting metric that divides the model's mean absolute error by the mean absolute error of a simple naive forecast on the training data, such as predicting the last observed value. A value below 1 means the model beats the naive method. It works when series have different scales.

Example. A model with MAE 4 on a series where the naive forecast has MAE 5 gets MASE = 0.8.

Related: Mean absolute error, Mean absolute percentage error, Baseline model, Symmetric mean absolute percentage error

For each query, the average precision is the mean of the precision values at the rank of every relevant result; MAP is that figure averaged across queries. It rewards ranking relevant items early and covers the whole list, so it is a standard for search and retrieval systems.

Example. For a query with 3 relevant items found at ranks 1, 3 and 5, the precisions are 1, 0.67 and 0.6, so AP is about 0.76. MAP averages such values over all queries.

Related: Average precision, Mean reciprocal rank, Normalised discounted cumulative gain, Precision at k

The average signed prediction error, with predictions minus actuals: (1/N) * sum of (y_hat - y). Unlike absolute-error metrics, positive and negative errors cancel, so it reveals whether a model systematically over- or under-predicts.

Example. Signed errors +5, -2, +2, +20 average to +6.25, showing the model tends to over-predict prices by about 6 lakh.

Related: Mean absolute error, Residual, Residual plot, Bias error

The average of the reciprocal rank over many queries. It summarises how high, on average, the first relevant result appears. A value near 1 means the right answer is usually first. It is best when each query has one right answer.

Example. Three queries have their first correct result at ranks 1, 2 and 4, so MRR = (1 + 0.5 + 0.25) / 3, about 0.58.

Related: Reciprocal rank, Mean average precision, Normalised discounted cumulative gain, Hit rate

The average of the squared differences between predictions and true values: MSE = (1/N) * sum of (y - y_hat)^2. Squaring makes all errors positive and punishes large errors far more than small ones. It is the standard training loss for regression, but its units are the square of the target's units.

Example. For actual prices [50, 60, 70, 80] lakh and predictions [55, 58, 72, 100], the squared errors are 25, 4, 4, 400, so MSE = 108.25.

Related: Root mean squared error, Mean absolute error, Residual, R-squared

The mean of squared differences between the natural logs of (1 + prediction) and (1 + actual). It measures relative rather than absolute error, so it treats being off by a factor as equally bad at small and large scales. It suits targets spanning orders of magnitude, but needs non-negative values.

Example. Predicting 200 for an actual 100 gives a squared log error of about 0.47; predicting 1,100 for an actual 1,000 gives only about 0.009.

Related: Root mean squared logarithmic error, Mean squared error, Mean absolute percentage error, Root mean squared error

The middle value of all the absolute prediction errors. Because it looks at the median rather than the mean, a handful of extreme mistakes barely moves it, making it a robust description of typical error when the data contains outliers.

Example. Absolute errors 5, 2, 2, 20 sort to 2, 2, 5, 20, so the median absolute error is (2 + 5) / 2 = 3.5, while the mean is 7.25.

Related: Mean absolute error, Max error, Root mean squared error, Residual

A translation metric that aligns words between candidate and reference by exact match, then stems and synonyms, and combines unigram precision and recall (weighted toward recall) with a penalty for scrambled word order. It tends to correlate better with human judgment than plain n-gram precision.

Example. A translation that says 'automobile' where the reference says 'car' still earns a match through synonym alignment, which BLEU would count as a miss.

Related: BLEU, ROUGE, chrF, BERTScore

A way to combine per-class results in multiclass problems by pooling the true positives, false positives and false negatives from all classes first and then computing one metric. Large classes dominate the result. For single-label multiclass tasks, micro-averaged precision, recall and F1 all equal accuracy.

Example. Pooling counts over 3 classes with 900 correct out of 1,000 predictions gives micro precision, recall and F1 all equal to 0.90.

Related: Macro averaging, Weighted averaging, Accuracy, F1 score

Of all cases the model labels negative, the share that truly are negative: TN / (TN + FN). It is the negative-side counterpart of precision and tells you how far to trust a 'clear' verdict. It depends heavily on how common the positive class is.

Example. The model clears 975 transactions and 970 of them are genuine, so NPV = 970 / 975, about 0.995.

Related: Precision, Specificity, Prevalence, Confusion matrix

The discounted cumulative gain divided by the best score achievable with a perfect ordering of the same items, giving a value between 0 and 1 that can be averaged across queries. It is the standard metric for graded-relevance search and recommendation ranking.

Example. A list with grades [2, 3, 0] has DCG about 3.89 while the ideal order [3, 2, 0] has 4.26, so NDCG = 0.91.

Related: Discounted cumulative gain, Mean average precision, Mean reciprocal rank, Hit rate

A clustering metric that measures how much knowing the cluster assignment tells you about the true class, using mutual information scaled to lie between 0 and 1. It works even if the number of clusters differs from the number of classes.

Example. If clusters perfectly reproduce the true news categories, NMI = 1; if they are unrelated to the categories, NMI is near 0.

Related: Adjusted Rand index, V-measure, Entropy, Silhouette score

The RMSE divided by a reference scale, commonly the mean or the range of the true values, so that errors can be compared across targets with different units or magnitudes. Which scale is used must always be stated, because the number changes with the choice.

Example. An RMSE of 10.4 lakh on prices averaging 65 lakh gives a normalised RMSE of 10.4 / 65 = 0.16.

Related: Root mean squared error, Mean absolute percentage error, Mean absolute error, Explained variance score

The particular decision threshold, and therefore the particular precision-recall or TPR-FPR pair, at which a classifier is actually deployed. Metrics such as AUC summarise all thresholds, but a live system must commit to a single one, chosen from business costs or constraints.

Example. A fraud team picks the threshold at which the model flags 0.5% of transactions, because the review team can only handle about 500 alerts a day.

Related: Decision threshold, ROC curve, Precision-recall trade-off, Cost matrix

The area under only a chosen portion of the ROC curve, typically a low false-positive-rate range, rather than the whole curve. It focuses evaluation on the operating region that matters in practice, since a model can have a good full AUC yet perform badly where false alarms must be rare.

Example. A screening tool must keep false positives under 5%, so the team computes the area under the ROC curve only for FPR between 0 and 0.05.

Related: AUC-ROC, ROC curve, Operating point, False positive rate

A metric for code generation: the probability that at least one of k sampled programs passes all unit tests for a problem. It is usually estimated from n samples with c correct as 1 - C(n-c, k) / C(n, k), then averaged across problems.

Example. If 3 of 10 generated solutions pass, pass@1 is 0.3, and pass@5 is higher because any of five attempts may succeed.

Related: Exact match, Accuracy, Unit test, Hit rate

A measure of how well a language model predicts text: the exponential of the average negative log-probability per token. Roughly, it is the number of equally likely choices the model is torn between at each step, so lower is better. It cannot be compared across different tokenisers.

Example. A model that is equally unsure among 4 possible next words at every step has a perplexity of 4.

Related: Bits per byte, Log loss, Entropy, BLEU

The loss used to train and evaluate quantile forecasts. For quantile level tau it charges tau * (y - q) when the actual value is above the prediction q and (1 - tau) * (q - y) when it is below. It is asymmetric, so low or high quantiles are judged appropriately.

Example. For the 0.9 quantile, an actual of 100 and prediction of 90 costs 0.9 * 10 = 9, while a prediction of 110 costs only 0.1 * 10 = 1.

Related: Mean absolute error, Prediction interval coverage, Continuous ranked probability score, Mean squared error

A post-processing method that fits a logistic regression on a model's raw scores, using held-out data, to convert them into calibrated probabilities. It has just two parameters, so it works with little data. It was introduced for support vector machines.

Example. An SVM outputs distances from the decision boundary; Platt scaling learns a sigmoid mapping those distances to probabilities such as 0.83.

Related: Isotonic calibration, Temperature scaling, Calibration, Logistic regression

Among the top k results returned, the fraction that are relevant. It is easy to interpret and matches how users look only at the first page, but it ignores the order within the top k and the total number of relevant items.

Example. If 3 of the top 5 search results are relevant, precision at 5 is 3 / 5 = 0.6.

Related: Recall at k, Mean average precision, Hit rate, Precision

The single point on the precision-recall curve where precision equals recall, used as a one-number summary of a classifier's performance. It removes the need to choose a threshold by hand, and was common in early text-classification and information-retrieval studies.

Example. If a spam filter at some threshold has precision 0.82 and recall 0.82, its break-even point is 0.82.

Related: Precision-recall curve, F1 score, Precision-recall trade-off, Average precision

A plot of precision against recall as the decision threshold varies. It ignores true negatives, so it is more informative than the ROC curve when positives are rare. A curve that stays high toward the top-right indicates a model that finds most positives with few false alarms.

Example. For a fraud model where only 2% of transactions are fraud, the team plots precision versus recall over thresholds to pick a setting that keeps precision above 0.5.

Related: ROC curve, Average precision, Precision-recall trade-off, Recall

The tendency for precision and recall to move in opposite directions as the decision threshold changes. Lowering the threshold catches more true cases, raising recall, but also admits more false alarms, lowering precision. Choosing the threshold means deciding which error is more acceptable.

Example. A resume-screening model set to shortlist anyone scoring above 0.3 finds nearly all suitable candidates but shortlists many unsuitable ones; raising the cut-off to 0.7 gives a cleaner list that misses more good people.

Related: Decision threshold, Precision, Recall, Operating point

The share of true values that actually fall inside the intervals a model predicts. A 90% interval is well calibrated if roughly 90% of outcomes land inside it. Coverage is checked alongside interval width, since very wide intervals cover well but are not useful.

Example. A demand forecaster issues 90% intervals for 100 weeks and only 78 actual values fall inside, so its coverage of 78% shows it is overconfident.

Related: Pinball loss, Confidence interval, Calibration, Continuous ranked probability score

The proportion of cases in a population that truly belong to the positive class. It strongly affects how a test's positive results should be interpreted, because even an accurate test produces mostly false alarms when the condition is very rare.

Example. If 20 of 1,000 transactions are fraudulent, prevalence is 2%.

Related: Base rate fallacy, Precision, Negative predictive value, Accuracy paradox

A scoring function for probability forecasts that is best achieved, in expectation, by reporting one's true beliefs. It cannot be gamed by hedging. Log loss and the Brier score are proper, whereas plain accuracy applied to thresholded predictions is not.

Example. Under the Brier score, if you truly believe rain has a 70% chance, stating 70% earns a better expected score than exaggerating to 100%.

Related: Log loss, Brier score, Continuous ranked probability score, Calibration

Precision measured at a cutoff equal to the number of relevant documents R that exist for the query. Because the cutoff adapts to each query, the precision and recall at that point are equal. It gives a single order-aware figure without choosing k.

Example. A query has 8 relevant documents; 5 of the top 8 results are relevant, so R-precision is 5 / 8 = 0.625.

Related: Precision at k, Recall at k, Mean average precision, Average precision

The share of all real positive cases that the model manages to find: TP / (TP + FN). It answers 'of everything that was truly there, how much did we catch?'. Catching most cases matters when missing one is costly, as in disease screening or fraud detection.

Example. There are 20 fraudulent transactions and the model flags 15 of them, so recall = 15 / 20 = 0.75.

Related: Precision, F1 score, False negative, Specificity

Among all the relevant items that exist for a query, the fraction that appear in the top k results. It measures whether retrieval brings the needed material into view at all, which matters for the first stage of a search or retrieval pipeline.

Example. A query has 6 relevant documents and 3 of them are in the top 10, so recall at 10 is 0.5.

Related: Precision at k, Hit rate, Mean average precision, Recall

For a single ranked result list, one divided by the position of the first relevant item. It rewards putting a correct answer near the top and ignores everything after the first hit. It is used for question answering and lookup-style search.

Example. If the first correct document appears at rank 4, the reciprocal rank is 1/4 = 0.25.

Related: Mean reciprocal rank, Hit rate, Precision at k, Mean average precision

Human labels stating which documents are relevant to each test query, sometimes on a graded scale. Together with the queries and the document collection they form the ground truth that ranking metrics such as MAP and NDCG are computed against.

Example. An Indian legal-search test set has 50 queries, each with 20 judged court cases marked 0 (irrelevant) to 3 (highly relevant).

Related: Cranfield paradigm, Normalised discounted cumulative gain, Mean average precision, Inter-annotator agreement

A plot for checking calibration: predictions are grouped into confidence bins, and each bin's average predicted probability is plotted against the actual fraction of positives. Points on the diagonal mean perfect calibration, below it means overconfident, above it underconfident.

Example. For the bin of predictions near 0.8, only 60% turn out positive, so the point falls well below the diagonal, showing overconfidence.

Related: Calibration, Expected calibration error, Platt scaling, Brier score

A scatter plot of a regression model's residuals (actual minus predicted) against the predicted values or a feature. A healthy plot is a shapeless cloud around zero. Curves, funnels or clusters signal missing non-linearity, changing error size or other problems the summary metrics hide.

Example. A funnel-shaped residual plot for house-price predictions shows errors growing with price, suggesting a log transform of the target.

Related: Residual, Mean bias error, Root mean squared error, Error analysis

A plot of the true positive rate against the false positive rate as the decision threshold of a scoring classifier sweeps from strict to lenient. Each point is one threshold. A curve hugging the top-left corner means a strong classifier, and the diagonal means random guessing.

Example. A credit-risk model that scores applicants can be run at 50 thresholds; plotting each threshold's (FPR, TPR) pair gives its ROC curve.

Related: AUC-ROC, Decision threshold, Precision-recall curve, Operating point

The square root of the mean squared error, which brings the error back into the same units as the target. It still weights large errors heavily. It is one of the most common regression metrics and reads as a typical size of the prediction error.

Example. With MSE 108.25 (in lakh squared), RMSE = sqrt(108.25), about 10.4 lakh.

Related: Mean squared error, Mean absolute error, Normalised RMSE, Max error

The square root of the mean squared logarithmic error. It reports relative error on a scale comparable to percentage difference and is popular in demand and price forecasting competitions where the target ranges widely and under-prediction should not be ignored.

Example. If MSLE is 0.09, RMSLE = 0.3, meaning predictions are typically off by a factor of about e^0.3, or 1.35.

Related: Mean squared logarithmic error, Root mean squared error, Mean absolute percentage error, Mean squared error

A family of metrics for automatic summarisation that measure how much of the reference summary's word sequences also appear in the generated summary. ROUGE-N counts overlapping n-grams and is recall-oriented, though precision and F-scores are also reported.

Example. Reference: 'the cat sat on the mat' (6 words); summary: 'the cat on the mat'. Five of the six reference unigrams are matched, so ROUGE-1 recall = 5/6, about 0.83.

Related: ROUGE-L, BLEU, BERTScore, Exact match

A ROUGE variant based on the longest common subsequence between the generated and reference texts: the longest run of words appearing in both in the same order, though not necessarily adjacent. It captures sentence-level word order without fixing an n-gram length.

Example. Between 'the cat sat on the mat' and 'the cat on the mat', the longest common subsequence has 5 words, giving recall 5/6 and precision 5/5.

Related: ROUGE, BLEU, METEOR, Token-level F1

Computing metrics separately for meaningful subsets of the data, such as by language, region, device or demographic group, rather than only overall. A model can look strong on average while failing badly on a slice, so this reveals hidden weaknesses and fairness gaps.

Example. A speech model has 8% word error rate overall but 21% on Hindi-accented English, which slice-based evaluation reveals.

Related: Error analysis, Word error rate, Accuracy, Confusion matrix

The correlation between the ranks of two variables instead of their raw values. It captures any consistently rising or falling relationship, not only straight-line ones, and is less affected by outliers. It is often used to compare a metric's ordering with human judgments.

Example. If an automatic quality metric ranks 10 summaries in almost the same order as human raters, Spearman's rho is close to +1, such as 0.9.

Related: Kendall's tau, Correlation coefficient, Normalised discounted cumulative gain, Human evaluation

The share of all real negative cases that the model correctly labels negative: TN / (TN + FP). It is the mirror image of recall and equals one minus the false positive rate. It is widely used in medical testing alongside sensitivity.

Example. Of 980 genuine transactions, 970 are cleared correctly, so specificity = 970 / 980, about 0.99.

Related: Recall, False positive rate, True negative, ROC curve

A variant of the percentage error that divides each absolute error by the average of the actual and predicted magnitudes: (100/N) * sum of |y - y_hat| / ((|y| + |y_hat|) / 2). It softens the over-forecast bias of MAPE and is used in forecasting competitions, though it still misbehaves near zero.

Example. For actuals 50, 60, 70, 80 and predictions 55, 58, 72, 100, sMAPE is about 9.5%.

Related: Mean absolute percentage error, Mean absolute scaled error, Mean absolute error, Root mean squared error

A calibration method for neural networks that divides the model's logits by a single learned number T before the softmax. T above 1 softens overconfident probabilities and T below 1 sharpens them. It leaves the predicted class unchanged, so accuracy stays the same.

Example. A classifier's 99% confidence drops to 85% after dividing its logits by T = 2, better matching its real accuracy.

Related: Expected calibration error, Platt scaling, Logistic regression, Calibration

An answer-scoring metric for question answering that treats the predicted and reference answers as bags of tokens and computes F1 on their overlap. It gives partial credit when the answer is close but not identical, unlike exact match.

Example. Reference answer 'the Reserve Bank of India' and prediction 'Reserve Bank of India' share 4 tokens: precision 1.0, recall 0.8, so F1 is about 0.89.

Related: Exact match, F1 score, ROUGE-L, Precision

A translation metric counting the minimum number of edits (insertions, deletions, substitutions and shifts of word blocks) needed to turn the system output into a reference, divided by the reference length. Lower is better, and it approximates how much human post-editing is needed.

Example. If 3 edits fix a 12-word output, TER = 3 / 12 = 0.25.

Related: Word error rate, BLEU, chrF, METEOR

A case that does not belong to the class of interest and that the model correctly labels as not belonging. It is a correct negative prediction. Large counts of them inflate plain accuracy on imbalanced data, which is why many metrics ignore them.

Example. A loan-default model says 'will repay' for a customer who does repay. In the fraud example, 970 genuine transactions correctly passed are true negatives.

Related: True positive, False negative, Specificity, Accuracy paradox

A case that really belongs to the class of interest and that the model also labels as that class. It is a correct positive prediction. Such cases sit in the numerator of recall and precision, so raising their count without adding false alarms is the usual goal of improving a classifier.

Example. A spam filter marks an email as spam and it truly is spam. Out of 200 real spam emails, catching 180 gives 180 true positives.

Related: False positive, True negative, Confusion matrix, Recall

The harmonic mean of two clustering scores against known classes: homogeneity (each cluster holds members of only one class) and completeness (all members of a class fall in the same cluster). It ranges from 0 to 1 and needs ground-truth labels.

Example. Splitting one true class across many pure clusters has high homogeneity but low completeness, which lowers the V-measure.

Related: Normalised mutual information, Adjusted Rand index, Purity, Silhouette score

A way to combine per-class metrics by averaging them with weights proportional to how many true examples each class has. It reflects the class distribution, so a weak rare class has little effect on the final number, unlike macro averaging.

Example. Class A (900 samples) has F1 0.95 and class B (100 samples) has F1 0.50; the weighted F1 is 0.9 * 0.95 + 0.1 * 0.50 = 0.905.

Related: Macro averaging, Micro averaging, F1 score, Accuracy paradox

A version of Cohen's kappa for ordered categories in which disagreements are penalised according to how far apart the two labels are, usually linearly or quadratically. Confusing 'mild' with 'severe' counts as worse than confusing 'mild' with 'moderate'. It is common for graded ratings and essay scoring.

Example. Two examiners grade essays 1 to 5. Disagreeing by one grade costs little under quadratic weighting, while a 1 versus a 5 costs sixteen times as much.

Related: Cohen's kappa, Inter-annotator agreement, Krippendorff's alpha, Fleiss' kappa

The standard metric for speech recognition: the number of substituted, deleted and inserted words needed to turn the recognised text into the reference, divided by the number of reference words. Lower is better, and it can exceed 100% if there are many insertions.

Example. A 100-word reference with 6 substitutions, 3 deletions and 1 insertion gives WER = 10 / 100 = 10%.

Related: Character error rate, Translation edit rate, Slice-based evaluation, Exact match

A single-number summary of a classifier at one threshold: J = sensitivity + specificity - 1. It ranges from 0 to 1 for a useful test. Picking the threshold that maximises J is a common way to choose the ROC point farthest above the chance diagonal.

Example. With sensitivity 0.75 and specificity 0.99, J = 0.75 + 0.99 - 1 = 0.74.

Related: ROC curve, Recall, Specificity, Decision threshold