Ordinary least squares

Ordinary least squares (OLS) estimates model parameters by minimizing the sum of squared differences between observed and predicted values. In linear regression, it chooses the fitted line or hyperplane with the smallest squared residuals.

Residual: A residual is the difference between an observed value and its model-predicted value. OLS squares these observed-minus-predicted differences before adding them.
Linear regression: A modeling approach that represents a response as a linear combination of predictors and parameters. OLS is the standard coefficient-fitting method for linear regression.
Simple linear regression: A linear regression model with one predictor and one response. OLS estimates its intercept and slope by minimizing squared residuals.
Median regression: A regression method that estimates conditional medians by minimizing absolute residuals. It is less sensitive than OLS to large residuals and targets a different conditional quantity.
Adrien-Marie Legendre: A French mathematician who published an early account of the method of least squares in 1805. Legendre gave the method an explicit mathematical formulation and name.
Least-squares estimator: An estimator that chooses parameters by minimizing the sum of squared residuals. It expresses OLS as a parameter-estimation rule.
Design matrix: A matrix whose rows encode observations and whose columns encode predictors and, often, an intercept. Its columns define the candidate fitted values available to OLS.
Multiple linear regression: A linear regression model with multiple predictors and one response. OLS extends directly to estimate its several predictor coefficients.
Ridge regression: A regression method that adds a squared-coefficient penalty to the residual sum of squares. Its penalty trades coefficient bias for reduced variance, unlike unregularized OLS.
Carl Friedrich Gauss: A German mathematician who developed least-squares methods and connected them to error distributions. Gauss used least squares in astronomical calculations and derived its statistical rationale.

Linked from 10 pages