Gauss–Markov theorem

The Gauss–Markov theorem states that, under specified linear-model assumptions, ordinary least squares is the best linear unbiased estimator of the regression coefficients, meaning it has the lowest variance among linear unbiased estimators.

Ordinary least squares: An estimation method that chooses coefficients to minimize the sum of squared residuals. The theorem identifies this estimator as best among linear unbiased alternatives.
Linear model: A model in which the expected response is a linear combination of unknown parameters and known quantities. The theorem assumes the response follows a linear model in its coefficients.
Heteroscedasticity: A condition in which a random variable’s variance changes with predictor values or other covariates. Unequal error variances remove the classical efficiency guarantee for ordinary least squares.
Carl Friedrich Gauss: A German mathematician and scientist whose work included least-squares methods and probability theory. The theorem’s name reflects Gauss’s connection to least-squares estimation.
Unbiased estimator: An estimator whose expected value equals the parameter it estimates. The theorem compares estimators only when they are unbiased.
Design matrix: A matrix whose rows encode the predictor values used to model each observation. Its rank determines whether regression coefficients are identifiable.
Autocorrelation: Dependence between values of a variable and its values at different times or positions. Correlated regression errors violate the theorem’s uncorrelated-error condition.
Andrey Markov: A Russian mathematician known for foundational work in probability and stochastic processes. Markov’s name is attached to the theorem’s classical linear-estimation result.
Homoscedasticity: A condition in which a random variable has constant variance across relevant values of its predictors. Equal error variance is a key condition for the classical result.
Full column rank: A matrix property in which every column is linearly independent of the others. This condition makes the least-squares coefficient estimate unique.

Linked from 4 pages

Sum of squaresRelated: It explains an optimality property of the estimator produced by minimizing residual squares.