Least squares

Least squares estimates unknown parameters by minimizing the sum of squared differences between observed values and model predictions. It is widely used for fitting models and solving overdetermined linear systems.

Residual: The difference between an observed value and the value predicted by a model. Least squares assigns each observation’s prediction error a squared contribution.
Linear system: A set of equations in which each equation is linear in the unknowns. Least squares handles linear systems that have no exact solution.
Linear regression: A statistical model that represents a response as a linear function of predictors and an error term. Ordinary linear regression estimates coefficients by minimizing squared residuals.
Least absolute deviations: A fitting method that minimizes the sum of absolute residuals. Its linear penalty is less sensitive to large residuals than squared loss.
Adrien-Marie Legendre: A French mathematician who published the method of least squares in 1805. Legendre gave the first published account of the method and applied it to geodesy.
Normal equations: A linear system whose solution gives the least-squares fit for a linear model. They express the condition that the residual is orthogonal to the model’s column space.
Matrix rank: The dimension of the column space of a matrix. Full column rank ensures a unique parameter estimate in linear least squares.
Curve fitting: The construction of a function that approximates observed data points. Least squares provides a standard criterion for choosing curve parameters.
Ridge regression: Linear regression with a squared-coefficient penalty added to the residual sum of squares. It regularizes least squares when predictors are highly correlated or parameters are unstable.
Carl Friedrich Gauss: A German mathematician and astronomer who developed influential work on least-squares estimation. Gauss connected least squares with astronomical observation and the Gaussian error model.

Linked from 68 pages