Gradient descent

Gradient descent is an iterative optimization method that updates parameters opposite an objective function’s gradient. Repeated updates seek a local minimum of that function.

Gradient: A vector of partial derivatives that points toward the steepest local increase of a differentiable scalar function. Its negative direction determines each gradient descent update.
Multivariable calculus: The branch of calculus studying functions of several variables and their derivatives and integrals. Gradient descent uses derivatives of objectives with respect to multiple parameters.
Linear regression: A statistical method that models a response as a linear combination of explanatory variables. Its squared-error parameters can be fitted by minimizing an objective with gradient descent.
Newton's method: An iterative method that uses first and second derivatives to approximate a function near a solution. It uses curvature information that ordinary gradient descent ignores.
Learning rate: A scalar controlling the size of each parameter update in an iterative optimization algorithm. It sets how far each step moves along the negative gradient.
Objective function: A function that assigns a numerical value to each candidate solution in an optimization problem. The method repeatedly changes parameters to lower this function.
Logistic regression: A statistical model that estimates class probabilities using a logistic function of a linear predictor. Its likelihood-based objective is commonly optimized with gradient methods.
Conjugate gradient method: An iterative method for solving linear systems or minimizing quadratic functions using conjugate search directions. Its directions can avoid the repeated zigzagging of steepest descent on quadratic problems.
Line search: A method for choosing a step length along a selected direction to reduce an objective function. It can replace a fixed learning rate with a step chosen for the current direction.
Local minimum: A point where a function’s value is no greater than at sufficiently nearby points. Gradient descent may settle at a local minimum rather than the global one.

Linked from 26 pages