Statistics inference

A controlled online experiment that randomly splits users into two groups, shows each a different version, and compares an outcome metric to judge which version is better. It is the standard way to estimate the causal effect of a product change.

Example. Half of 40,000 app users see a green Pay button and half see blue; you compare payment completion rates.

Related: Control group, Randomisation, Two-sample t-test

A version of R-squared that penalises extra predictors, computed as 1 - (1 - R^2)(n - 1)/(n - p - 1). It rises only if a new predictor improves the fit more than chance would.

Example. Adding a random column raises R-squared from 0.72 to 0.721 but lowers the adjusted value from 0.71 to 0.709.

Related: R-squared, Akaike information criterion

The claim accepted in place of the null hypothesis when the data give strong evidence against it, typically that an effect or difference exists. It can be two-sided (any difference) or one-sided (a specific direction).

Example. The new page's conversion rate differs from the old page's rate.

Related: Null hypothesis, One-tailed and two-tailed tests, P-value

Analysis of variance: a test that asks whether three or more group averages are all equal by comparing variation between groups with variation within groups. A large ratio suggests at least one group differs.

Example. Compare average test scores of students from four coaching centres in one test instead of six separate pairwise tests.

Related: F-test, Post-hoc test, Multiple comparisons problem

The mean difference in outcome between receiving a treatment and not receiving it, averaged over a population: ATE = E[Y(1) - Y(0)]. Randomised experiments estimate it by comparing group averages.

Example. If treated users spend Rs 620 on average and control users Rs 590 in a randomised test, the estimated effect is Rs 30 per user.

Related: Potential outcomes framework, Randomised controlled trial, Propensity score

An approach that treats parameters as uncertain quantities described by probability distributions, updating a prior belief with observed data via Bayes' theorem to get a posterior belief. It gives direct probability statements about parameters.

Example. Start with a prior about a conversion rate, observe 30 conversions in 500 visits, and update to a posterior centred near 6 percent.

Related: Prior distribution, Posterior distribution, Credible interval

A step-up method that controls the false discovery rate. Sort the p-values from smallest to largest, find the largest rank k where p(k) <= (k / m) * q, and reject all hypotheses up to that rank.

Example. With 100 tests and q = 0.05, the 10th smallest p-value is compared against 10/100 * 0.05 = 0.005.

Related: False discovery rate, Bonferroni correction, Multiple comparisons problem

The gap between the average value of an estimator over many samples and the true quantity it targets. A nonzero gap means the estimator is systematically too high or too low, no matter how many times you repeat the study.

Example. Dividing the sum of squared deviations by n instead of n - 1 gives a variance estimate that is on average too small.

Related: Unbiased estimator, Estimator, Consistency

A simple adjustment for many tests: divide the significance level by the number of tests, or equivalently multiply each p-value by that number. It controls the family-wise error rate but can be too strict when there are many tests.

Example. Testing 5 metrics at an overall level of 0.05 means judging each one against 0.05 / 5 = 0.01.

Related: Multiple comparisons problem, Family-wise error rate, False discovery rate

A resampling method that builds many new datasets by drawing observations from the original sample with replacement, recomputing the statistic each time. The spread of those recomputed values approximates the statistic's uncertainty.

Example. Resample 500 order values 2,000 times with replacement, compute each median, and use the middle 95 percent of those medians as an interval.

Related: Resampling, Sampling distribution, Confidence interval

A diagram where nodes are variables and arrows point from causes to effects, with no cycles. It makes assumptions explicit and shows which variables to adjust for to estimate a causal effect.

Example. Draw arrows Income -> Loan approval and Income -> Repayment to show that income is a confounder of approval and repayment.

Related: Confounding variable, Causal inference

The set of methods for working out whether and how much one factor changes another, rather than merely moving with it. It combines study design, assumptions and statistical adjustment. It matters in AI work whenever you must decide what to change, not just predict what will happen.

Example. Estimating how much a loyalty programme raises repeat purchases, after accounting for the fact that keen shoppers join it.

Related: Counterfactual, Potential outcomes framework, Average treatment effect

A test of whether observed category counts match a claimed distribution. It sums (observed - expected)^2 / expected across categories and compares the total with a chi-square reference distribution. Small totals mean the observed counts match the claim well.

Example. A die rolled 600 times should give about 100 of each face; the test judges whether the observed counts deviate too much for a fair die.

Related: Chi-square test of independence, Test statistic, Degrees of freedom

A test that checks whether two categorical variables are related, by comparing observed counts in a contingency table with the counts expected if they were unrelated. Large gaps produce a large statistic and a small p-value.

Example. Test whether payment method (UPI, card, cash) and product category are related using a 3 by 4 table of order counts.

Related: Chi-square goodness-of-fit test, Degrees of freedom, Fisher's exact test

A standardised effect size for the difference between two averages, computed as the gap divided by the pooled standard deviation. Conventionally 0.2 is called small, 0.5 medium and 0.8 large.

Example. Averages of 72 and 68 with a pooled standard deviation of 10 give d = 4 / 10 = 0.4.

Related: Effect size, Two-sample t-test, Power analysis

A range of values computed from sample data by a procedure that captures the true population value in a stated percentage of repeated studies. It expresses the precision of an estimate; it does not give the chance that this one interval contains the truth.

Example. An average order value of Rs 850 with a 95% confidence interval of Rs 820 to Rs 880.

Related: Confidence level, Margin of error, Standard error of the mean

The long-run percentage of intervals, built the same way from repeated samples, that would contain the true value. Common choices are 90, 95 and 99 percent; a higher level gives a wider interval.

Example. At 95 percent, about 95 of 100 independently built intervals would cover the true value and about 5 would miss.

Related: Confidence interval, Significance level, Critical value

A property of an estimator: as the sample grows without limit, its value converges to the true quantity being estimated. It guarantees that more data eventually brings the estimate arbitrarily close to the truth.

Example. The sample proportion of clicks from 100 users may be off by 5 points, but from 10 million users it lies extremely close to the true click rate.

Related: Estimator, Bias of an estimator, Efficiency

A grid that counts how many observations fall into each combination of categories of two or more variables. It is the starting point for tests of association between categorical variables.

Example. A 2 by 2 table of Android versus iOS against purchased versus not purchased.

Related: Chi-square test of independence, Fisher's exact test, Odds ratio

The group in an experiment that does not receive the change being tested, providing the baseline against which the changed group is compared. Randomly assigned, it shows what would have happened without the change.

Example. In a test of a new recommendation model, the control group keeps seeing recommendations from the old model.

Related: A/B test, Randomisation, Randomised controlled trial

What would have happened to the same unit under a different treatment than the one it actually received. It can never be observed directly, which is the core difficulty of causal questions.

Example. A customer got a discount and bought; the counterfactual is whether the same customer would have bought without it.

Related: Potential outcomes framework, Causal inference, Average treatment effect

A lower limit on the variance of any unbiased estimator of a parameter, equal to the reciprocal of the Fisher information. It shows the best precision that is theoretically possible from the data.

Example. For a coin with p = 0.5 and n = 100 flips, no unbiased estimator of p can have variance below 0.25 / 100 = 0.0025.

Related: Fisher information, Efficiency, Unbiased estimator

A range that contains a parameter with a stated posterior probability, such as 95 percent. Unlike a confidence interval, it can be read directly as the probability that the parameter lies inside.

Example. A posterior for conversion rate gives a 95 percent credible interval of 4.6 to 7.4 percent.

Related: Confidence interval, Posterior distribution, Bayesian inference

The cut-off on a test statistic's reference distribution that marks the boundary of the rejection region for a chosen significance level. Values beyond it are considered too unlikely under the null hypothesis.

Example. For a two-sided z-test at the 5 percent level, the critical values are -1.96 and +1.96.

Related: Rejection region, Significance level, Test statistic

The number of values in a calculation that are free to vary once constraints are fixed. It sets the exact shape of reference distributions such as the t and chi-square, and is often n - 1 or n minus the number of fitted parameters.

Example. With 10 observations and a known average, only 9 values can vary freely, so a one-sample t-test uses 9 degrees of freedom.

Related: t-distribution, Student's t-test, Chi-square test of independence

A method that compares the change over time in a treated group with the change over the same period in an untreated group. It removes stable group differences and shared trends, under the assumption of parallel trends.

Example. Sales in cities where a new delivery service launched rose by 12 percent while other cities rose by 5 percent, giving an estimated effect of 7 points.

Related: Causal inference, Average treatment effect, Confounding variable

A measure of how large a difference or relationship is, independent of sample size. Unlike a p-value, it tells you whether the change is big enough to matter. Reporting it alongside the p-value shows both whether an effect exists and how big it is.

Example. A new onboarding flow raises seven-day retention from 40 to 43 percent, an effect of 3 percentage points.

Related: Cohen's d, Practical significance, Statistical power

A measure of how small the variability of an unbiased estimator is compared with the best achievable. Between two unbiased estimators, the more efficient one has the smaller variance and so needs less data for the same precision.

Example. For normally distributed data the sample average has lower variance than the sample median, so it is the more efficient estimator of the centre.

Related: Cramér-Rao bound, Unbiased estimator, Estimator

A rule or formula that turns sample data into a guess for an unknown population quantity. Different rules can target the same quantity, and they are compared by properties such as bias, spread and consistency.

Example. The sample average is an estimator of the population average; the sample median is another estimator for the same target.

Related: Point estimate, Bias of an estimator, Consistency

A test whose statistic is a ratio of two variance estimates and follows the F-distribution when the null hypothesis holds. It is used in ANOVA, to compare two variances, and to check whether a regression model explains anything.

Example. In ANOVA, F = between-group variance estimate / within-group variance estimate; F near 1 suggests no group effect.

Related: ANOVA, Degrees of freedom, Ordinary least squares

The expected fraction of rejected null hypotheses that are actually false alarms. Controlling it is less strict than controlling the family-wise error rate and suits screening many candidates, such as genes or features.

Example. If 50 features are flagged as significant and the rate is controlled at 10 percent, about 5 of them are expected to be false leads.

Related: Benjamini-Hochberg procedure, Family-wise error rate, Multiple comparisons problem

The probability of making at least one false positive across a whole set of related tests. Corrections such as Bonferroni aim to keep it below a chosen level. It grows quickly as the number of tests increases.

Example. In a test with 10 metrics, keeping this rate at 0.05 requires stricter thresholds for each individual metric.

Related: Bonferroni correction, False discovery rate, Multiple comparisons problem

A measure of how much information a sample carries about an unknown parameter, defined as the expected curvature (negative second derivative) of the log-likelihood at the true value. Sharper curvature means more precise estimation.

Example. For n coin flips, the information about p is n / (p(1 - p)), so more flips give sharper estimates.

Related: Cramér-Rao bound, Log-likelihood, Maximum likelihood estimation

A test of association between two categorical variables in a 2 by 2 table that computes the exact probability of the observed table, rather than using an approximation. It is preferred when counts are small.

Example. With 3 of 8 users converting under A and 7 of 9 under B, the counts are too small for the chi-square approximation, so this exact test is used.

Related: Chi-square test of independence, P-value, Odds ratio

An approach that treats parameters as fixed unknowns and defines probability as long-run frequency over repeated sampling. Confidence intervals and p-values belong to this approach. Its results are judged by how the procedure behaves across many hypothetical repetitions.

Example. Saying 'this procedure covers the true value in 95 percent of repeated studies' is a frequentist statement.

Related: Bayesian inference, Confidence interval, P-value

A family of models that links a linear combination of predictors to an outcome through a link function and a chosen distribution for the outcome, such as binomial for yes/no or Poisson for counts. Logistic regression is one example.

Example. Modelling daily complaint counts with a Poisson distribution and log link is a generalised linear model.

Related: Odds ratio, Maximum likelihood estimation, Regression coefficient

A metric monitored during an experiment to ensure a change does not damage something important, even if the main target metric improves. A guardrail breach can block a launch. Teams typically set an acceptable threshold for each one before the test starts.

Example. A test that boosts sign-ups also tracks page-load time and refund rate as guardrails.

Related: A/B test, Practical significance, Lift

A situation in which the spread of a regression model's errors changes with the predictors, often widening as values grow. It leaves coefficients unbiased but makes usual standard errors wrong.

Example. Errors in predicting salary are small for freshers but large for senior managers, giving a fan-shaped residual plot.

Related: Homoscedasticity, Residual, Ordinary least squares

The assumption that the spread of a regression model's errors is the same across all values of the predictors. When it holds, standard errors and tests from ordinary least squares are reliable.

Example. Prediction errors for house prices are about Rs 2 lakh whether the house is cheap or expensive.

Related: Heteroscedasticity, Residual, Ordinary least squares

A variable that affects the treatment but influences the outcome only through that treatment and is unrelated to hidden confounders. It allows causal estimation when direct comparison is contaminated. Finding a valid one is hard, and its assumptions cannot be fully tested from data.

Example. Using a random nudge email that changes whether users adopt a feature to estimate the feature's effect on retention.

Related: Confounding variable, Causal inference, Ordinary least squares

A term in a model that lets the effect of one predictor depend on the value of another, usually built by multiplying the two. Without it the model assumes each predictor works independently.

Example. In sales = b0 + b1 * ad_spend + b2 * festival + b3 * ad_spend * festival, b3 shows how much ads work better during festivals.

Related: Regression coefficient, Ordinary least squares, Generalised linear model

A resampling method that recomputes a statistic n times, each time leaving out one observation, and uses the variation among results to estimate bias and standard error. It is older and simpler than the bootstrap.

Example. For 30 observations, compute the median 30 times with each observation left out in turn.

Related: Bootstrap, Resampling, Standard error of the mean

A test that measures the largest vertical gap between two cumulative distribution curves, either a sample against a reference distribution or one sample against another. It is often used to detect changes in a feature's distribution.

Example. Compare a feature's values in last month's data and this month's data to see whether the two distributions differ.

Related: Non-parametric test, P-value

A function of the model parameters that gives the probability (or density) of the observed data as those parameters vary, with the data held fixed. Higher values mean the parameters explain the data better.

Example. For 7 heads in 10 coin flips, the likelihood of head-probability p is proportional to p^7 * (1 - p)^3, highest near p = 0.7.

Related: Maximum likelihood estimation, Log-likelihood, Bayesian inference

A test that compares two nested models by looking at twice the difference in their log-likelihoods, which follows a chi-square distribution under the simpler model. It asks whether extra parameters improve the fit significantly.

Example. Comparing a logistic model with and without a city predictor: a large statistic favours keeping city.

Related: Log-likelihood, Degrees of freedom

The natural logarithm of the likelihood function. It turns products over independent observations into sums, which are easier to differentiate and numerically stable, and it has its maximum at the same parameter values.

Example. For 7 heads in 10 flips, log-likelihood = 7 ln(p) + 3 ln(1 - p), which peaks at p = 0.7.

Related: Likelihood function, Maximum likelihood estimation, Fisher information

A rank-based test for whether values in one independent group tend to be larger than in another. It replaces the two-sample t-test when data are skewed or ordinal. Because it uses ranks, extreme values have little pull on the result.

Example. Compare app-rating scores from Android and iOS users, both on a 1 to 5 scale, without assuming normality.

Related: Two-sample t-test, Non-parametric test, Wilcoxon signed-rank test

Half the width of a confidence interval: the distance from the point estimate to either end. It is the critical value multiplied by the standard error, and it shrinks as the sample gets larger.

Example. A poll finds 52 percent support with a margin of error of 3 points, so the interval runs from 49 to 55 percent.

Related: Confidence interval, Standard error of the mean, Critical value

A method that picks the parameter value with the highest posterior probability, combining the data likelihood with a prior belief. It equals maximum likelihood with an added penalty term, which is why L2 weight decay corresponds to a Gaussian prior.

Example. Adding a normal prior centred at zero to a regression's weights and maximising gives the same fit as ridge regression.

Related: Maximum likelihood estimation, Prior distribution, Posterior distribution

The smallest true change an experiment is designed to reliably detect at the chosen significance level and power. Smaller effects need much larger samples, so it is fixed at design time from business value.

Example. Deciding that only a lift of 1 percentage point in signup rate is worth shipping and sizing the experiment to detect exactly that.

Related: Power analysis, Statistical power, A/B test

The rise in false positives when many hypotheses are tested at once: each test has its own small chance of a false alarm, and these accumulate. Without adjustment, some significant results are expected purely by luck.

Example. Testing 20 unrelated metrics at the 5 percent level gives a 1 - 0.95^20, about 64 percent, chance of at least one false alarm.

Related: Bonferroni correction, False discovery rate, Family-wise error rate

A hypothesis test that makes few assumptions about the shape of the underlying distribution, often working with ranks rather than raw values. It suits skewed data, outliers and small samples.

Example. With heavily skewed Rs-per-order data, comparing groups by ranks is safer than comparing averages.

Related: Mann-Whitney U test, Wilcoxon signed-rank test

A temporary change in behaviour caused only by users seeing something new, which fades as they get used to it. It can make a short experiment overstate the lasting benefit of a change.

Example. A redesigned home screen lifts clicks in week one because it is unfamiliar, then the lift decays by week three.

Related: A/B test, Practical significance, Peeking problem

The default claim in a test, usually that there is no effect or no difference. The test asks whether the data are surprising enough to cast serious doubt on it; failing to do so does not prove it true.

Example. For a new checkout page, the null hypothesis is that its conversion rate equals the old page's rate.

Related: Alternative hypothesis, P-value, Significance level

Two ways of setting the alternative hypothesis. A two-tailed test looks for a difference in either direction; a one-tailed test looks only in a pre-specified direction and puts all rejection area on that side.

Example. Testing whether a new model is different from the old one is two-tailed; testing only whether it is faster is one-tailed.

Related: Alternative hypothesis, Critical value, Rejection region

Trying many analyses, subgroups, metrics or stopping points until one yields a small p-value, then reporting only that one. It inflates false positives and produces findings that fail to replicate.

Example. Slicing an experiment by 30 user segments and announcing the one segment with p = 0.04 as a win.

Related: Multiple comparisons problem, Type I error, Peeking problem

The probability, assuming the null hypothesis is true, of getting a result at least as extreme as the one observed. It is not the chance that the null hypothesis is true, and it says nothing about effect size.

Example. A p-value of 0.03 means that if the new page had no real effect, results this large would appear in about 3 percent of experiments.

Related: Significance level, Null hypothesis, Effect size

A test for the average difference between two related measurements on the same units, such as before and after. Working on within-unit differences removes variation between units and gives more power.

Example. Compare each of 40 students' scores before and after a coaching course by testing whether the mean of the 40 differences is zero.

Related: Student's t-test, Two-sample t-test, Statistical power

The inflation of false positives that happens when an experiment's p-value is checked repeatedly and stopped as soon as it looks significant. A fixed-sample test is valid only for one look at the planned sample size.

Example. Checking a test daily and stopping the first time p < 0.05 can push the true false-positive rate far above 5 percent.

Related: Sequential testing, P-hacking, Type I error

A hypothesis test that randomly shuffles group labels many times to build the distribution of a statistic under the null hypothesis of no group effect. The p-value is the share of shuffles at least as extreme as the real data.

Example. Shuffle 'A' and 'B' labels across 200 users 10,000 times; if only 120 shuffles show a gap as large as observed, p = 0.012.

Related: Bootstrap, P-value, Non-parametric test

A single number offered as the best guess of an unknown population value, produced by applying an estimator to observed data. It carries no information about how uncertain it is unless paired with an interval or standard error.

Example. A trial gives an average conversion of 4.2 percent, so 4.2 percent is the point estimate of the true rate.

Related: Estimator, Confidence interval, Standard error of the mean

A fixed but usually unknown number that describes a whole population, such as its average or its proportion. Because it cannot be observed directly, it is approximated from sample data.

Example. The true share of all Indian smartphone users who use UPI is a parameter; a survey of 2,000 users only approximates it.

Related: Statistic, Population, Estimator

A follow-up comparison run after an overall test such as ANOVA finds a difference, to identify which specific groups differ. It adjusts for the many pairwise comparisons so the error rate stays controlled.

Example. After ANOVA shows the four coaching centres differ, Tukey's HSD indicates that centre B and centre D are the pair that differ.

Related: ANOVA, Multiple comparisons problem, Family-wise error rate

The probability distribution of a parameter after combining the prior with observed data through Bayes' theorem. It holds everything the analysis knows about the parameter and is used for estimates and intervals.

Example. A Beta(2, 38) prior updated with 30 conversions and 470 non-conversions gives a Beta(32, 508) posterior.

Related: Prior distribution, Bayesian inference, Credible interval

A formal model of causality in which each unit has an outcome under each possible treatment, but only one is ever seen. Causal effects are defined as differences between these outcomes.

Example. For user i, Y(1) is the spend if shown the offer and Y(0) is the spend if not; only one of them is observed.

Related: Counterfactual, Average treatment effect, Causal inference

A calculation done before collecting data to find the sample size needed to detect an effect of a chosen size with a chosen power and significance level. It prevents experiments that are too small to be informative.

Example. Detecting a rise in conversion from 5 to 6 percent at 5 percent significance and 80 percent power needs roughly 8,000 users per group.

Related: Statistical power, Minimum detectable effect, Effect size

Whether an effect is large enough to matter in the real world, as opposed to merely being statistically detectable. With huge samples, trivial differences produce tiny p-values. Judging it requires business or domain context, not just a statistical calculation.

Example. A test on 10 million users shows p < 0.001 for a 0.01 percent lift in clicks, yet the gain may not repay the engineering cost.

Related: Effect size, P-value, Minimum detectable effect

A probability distribution expressing what is believed about a parameter before seeing the current data. It can encode earlier studies or expert judgement, and its influence fades as more data arrive.

Example. Before an experiment, a Beta(2, 38) prior expresses a belief that conversion is around 5 percent.

Related: Posterior distribution, Bayesian inference

The probability that a unit receives a treatment given its observed characteristics. Matching, weighting or stratifying on it balances treated and untreated groups in observational data. Adjusting on it can only balance the characteristics that were actually measured.

Example. Estimate each customer's chance of joining a loyalty programme from age, city and past spend, then compare members with non-members who had similar scores.

Related: Confounding variable, Average treatment effect, Causal inference

The proportion of variation in the outcome that a regression model accounts for, between 0 and 1, computed as 1 - (residual sum of squares / total sum of squares). It never falls when predictors are added, even useless ones.

Example. An R-squared of 0.72 means the model accounts for 72 percent of the variation in house prices in the data.

Related: Adjusted R-squared, Residual, Ordinary least squares

Assigning experimental units to groups purely by chance. It makes the groups comparable on both measured and unmeasured characteristics on average, so differences in outcomes can be attributed to the treatment.

Example. Hash each user ID and send users with even hashes to version A and odd hashes to version B.

Related: A/B test, Confounding variable, Randomised controlled trial

An experiment in which participants are randomly assigned to a treatment or control condition and outcomes are compared. It is regarded as the strongest design for establishing cause and effect.

Example. Randomly giving 500 of 1,000 job seekers a skills course and tracking how many get jobs in six months.

Related: Randomisation, Control group, Average treatment effect

A number in a regression model giving the expected change in the outcome for a one-unit increase in a predictor, holding the other predictors fixed. Its uncertainty is reported by a standard error, confidence interval and p-value.

Example. In rent = 5,000 + 40 * area, each extra square foot is associated with Rs 40 more monthly rent.

Related: Ordinary least squares, Standard error of the mean, Interaction term

The tendency for extreme measurements to be followed by measurements closer to the average, because part of any extreme value is luck. It creates false impressions that an intervention worked.

Example. A salesperson with a record month usually sells less the next month, even with no change in effort.

Related: Spurious correlation, Selection bias, Confounding variable

The set of test statistic values for which the null hypothesis is rejected. Its boundaries are the critical values, and its total probability under the null equals the significance level.

Example. At the 5 percent level in a two-sided z-test, any z below -1.96 or above 1.96 falls in the region.

Related: Critical value, Significance level, Test statistic

A family of techniques that repeatedly draw new samples from the observed data to study the behaviour of a statistic, without relying on formulas that assume a specific distribution. Bootstrap, permutation and jackknife methods belong here.

Example. Shuffling group labels 5,000 times to see how often chance alone produces a gap as large as the observed one.

Related: Bootstrap, Permutation test, Jackknife

A warning sign in an experiment where the observed split of users between groups differs from the intended split by more than chance would allow. It usually points to a bug in assignment or logging, and invalidates the results.

Example. A planned 50/50 split that shows 50,900 versus 49,100 users out of 100,000 fails a chi-square check and needs investigation.

Related: A/B test, Chi-square goodness-of-fit test, Randomisation

The pattern of values a statistic would take if you repeated the same sampling process many times. Its spread tells you how much the statistic wobbles from sample to sample, which is the basis for standard errors, intervals and tests.

Example. Drawing 1,000 samples of 100 customers and recording each sample's average spend gives 1,000 values; their histogram is the sampling distribution of the average.

Related: Standard error of the mean, Statistic, Bootstrap

Methods that let you examine experiment data at multiple times and stop early while still controlling the overall false-positive rate. They adjust thresholds at each look instead of using one fixed cut-off.

Example. A group sequential design with pre-set looks at 25, 50, 75 and 100 percent of the sample lets a clear winner be declared early.

Related: Peeking problem, Type I error, A/B test

The threshold chosen before a test for the largest p-value at which the null hypothesis will be rejected, equal to the accepted chance of a false alarm. The usual choice is 0.05.

Example. With a level of 0.05, a p-value of 0.03 leads to rejecting the null hypothesis, while 0.08 does not.

Related: P-value, Type I error, Confidence level

A correlation computed on the ranks of the values instead of the raw values. It captures any consistently rising or falling relationship, not only straight lines, and is less affected by outliers.

Example. Ranking 10 cities by both rent and salary and correlating the ranks gives a Spearman value close to 1 if the orderings match.

Related: Correlation coefficient, Non-parametric test, Spurious correlation

The standard deviation of a statistic across repeated samples. It measures how precisely the statistic estimates the population value; for a sample average it is s / sqrt(n), so quadrupling the data halves it.

Example. With s = 20 minutes and n = 100 deliveries, the standard error of the average is 20 / 10 = 2 minutes.

Related: Sampling distribution, Confidence interval, Margin of error

A number computed from sample data, such as the sample average or sample proportion. It changes from one sample to the next, and it is used to approximate an unknown population value.

Example. In a sample of 50 orders, the average delivery time is 32 minutes; that 32 is a statistic.

Related: Population parameter, Estimator, Sampling distribution

The probability that a test will detect an effect of a given size when that effect truly exists, equal to 1 - the Type II error rate. It rises with sample size, effect size and the significance level.

Example. A test with 80 percent power finds a real 2 point lift in 8 of 10 repeated experiments.

Related: Type II error, Effect size, Power analysis

A hypothesis test that checks whether a sample average differs from a claimed value, or whether two averages differ, using the t-distribution because the spread is estimated from the data. It assumes roughly normal data or a fair sample size.

Example. Delivery times for 25 orders average 34 minutes; testing against a promised 30 minutes gives t = (34 - 30) / (s / sqrt(25)).

Related: Two-sample t-test, Paired t-test, t-distribution

A bell-shaped, symmetric distribution with heavier tails than the normal curve, used when the population spread is estimated from a small sample. As the degrees of freedom increase it approaches the normal curve.

Example. With 9 degrees of freedom, the 95 percent two-sided critical value is about 2.26, versus 1.96 for the normal curve.

Related: Degrees of freedom, Student's t-test, Confidence interval

A single number computed from sample data that summarises how far the result lies from what the null hypothesis predicts, scaled by the expected noise. Its position in a known reference distribution yields the p-value.

Example. A t statistic of 2.5 means the observed average is 2.5 standard errors from the claimed value.

Related: P-value, Critical value, Null hypothesis

A hypothesis test that asks whether the averages of two independent groups differ by more than chance would explain. It compares the observed gap in averages with the standard error of that gap.

Example. Compare average session time of 30 users shown design A with 30 different users shown design B.

Related: Welch's t-test, Paired t-test, A/B test

Rejecting a null hypothesis that is actually true, a false positive. Its long-run rate is controlled by the significance level. It is the risk you accept, in advance, of acting on an effect that is not real.

Example. Launching a feature because a test showed a lift, when the lift was only random noise.

Related: Type II error, Significance level, Statistical power

Failing to reject a null hypothesis that is actually false, a missed real effect. Its probability is called beta, and power equals 1 - beta. It is more likely when the sample is small or the true effect is subtle.

Example. Discarding a genuinely better checkout design because a small test could not detect its improvement.

Related: Type I error, Statistical power, Effect size

An estimator whose average value across all possible samples equals the true quantity it targets. Any single result can still miss the target, but there is no systematic push in one direction.

Example. The sample average is unbiased for the population mean: across many samples its results centre exactly on the true mean.

Related: Bias of an estimator, Estimator, Efficiency

A version of the two-group averages test that does not assume the groups have equal spread and adjusts its degrees of freedom accordingly. It is a safe default because it loses little when spreads are equal.

Example. Comparing checkout times where one group has standard deviation 5 seconds and the other 20 seconds calls for this test rather than the pooled version.

Related: Two-sample t-test, Degrees of freedom, Student's t-test

A rank-based test for paired data that asks whether the differences between paired measurements are centred at zero. It is the distribution-free counterpart of the paired t-test. It is useful for small samples where the differences look skewed.

Example. Rank the absolute before-and-after differences in 15 users' page-load times and check whether positive and negative ranks balance.

Related: Paired t-test, Non-parametric test, Mann-Whitney U test

A hypothesis test that compares a sample average or proportion with a claimed value using the normal distribution. It is appropriate when the population spread is known or the sample is large.

Example. Testing whether a site's conversion rate differs from the historic 3 percent using 50,000 visitors is normally done with a z-test for a proportion.

Related: Test statistic, P-value, Student's t-test