Numerical optimization

Numerical optimization uses computational algorithms to find approximate minima or maxima of objective functions, often subject to constraints.

Gradient descent: An iterative method that updates variables in the direction opposite an objective’s gradient. It is the basic derivative-based route toward a local minimum.
Objective function: A function that assigns a numerical score to each candidate decision or parameter setting. It is the quantity an optimizer is tasked with minimizing or maximizing.
Linear programming: Optimization of a linear objective subject to linear equality and inequality constraints. It is a structured special case with efficient algorithms and strong global guarantees.
Nonlinear least squares: The problem of minimizing squared residuals when predictions depend nonlinearly on unknown parameters. It is a common numerical task in fitting models to measured data.
Convergence of optimization algorithms: The study of whether and how an optimization algorithm’s iterates approach a solution. Convergence conditions determine when an algorithm’s output can be trusted.
Newton's method: An iterative root-finding method that uses first and second derivatives to estimate a solution. Optimization variants use curvature information to make more informed steps.
Decision variable: A quantity whose value can be chosen in a mathematical model or optimization problem. Algorithms alter these values to improve the objective.
Combinatorial optimization: Optimization over a finite or discrete set of combinations, arrangements, or selections. Discrete decisions often require different algorithms from continuous numerical methods.
Maximum likelihood estimation: A method of estimating model parameters by maximizing the likelihood of observed data. Numerical optimizers often find parameter estimates when likelihoods lack closed-form solutions.
Condition number: A measure of how sensitive a problem’s solution is to small changes in its input. Poor conditioning can slow optimization and amplify numerical error.

Linked from 5 pages