Backpropagation

Backpropagation computes how a neural network’s output error changes with each parameter by applying the chain rule backward through its layers. These gradients guide parameter updates during training.

Chain rule: A rule for differentiating composite functions by multiplying derivatives of their component functions. It lets backpropagation combine layer-by-layer derivatives into a gradient from output to earlier parameters.
Artificial neuron: A computational unit that combines inputs with weights, applies a bias, and transforms the result with an activation function. Backpropagation differentiates each neuron’s weighted sum and activation to assign parameter gradients.
Stochastic gradient descent: An optimization method that updates model parameters using gradients estimated from individual examples or small batches. It commonly applies backpropagated gradients to update neural-network parameters.
Finite difference: A numerical method that estimates derivatives from changes in function values at nearby inputs. Unlike backpropagation, it estimates gradients through repeated evaluations rather than exact chain-rule propagation.
Computational graph: A directed graph representing a computation as operations and the values passed between them. Backpropagation traverses this graph in reverse, reusing intermediate values to calculate derivatives.
Activation function: A function applied to a unit’s input to determine its output, often introducing nonlinearity into a neural network. Its derivative controls how strongly a neuron passes the backward error signal.
Mini-batch gradient descent: An optimization method that estimates gradients from small subsets of training examples before updating parameters. Backpropagation is usually run across a mini-batch to form each training update.
Forward-mode automatic differentiation: An automatic differentiation method that propagates derivatives forward alongside computed values. It is less efficient than reverse mode when one scalar loss depends on many parameters.
Jacobian matrix: A matrix of first-order partial derivatives describing how a vector-valued function changes with its inputs. Each layer’s local Jacobian maps downstream error signals to derivatives with respect to its inputs.
Weight (machine learning): A learnable scalar that scales a connection or feature contribution in a machine-learning model. Backpropagation calculates how changing each weight would change the loss.

Linked from 13 pages