Expectation–maximization algorithm

An iterative method for maximum-likelihood estimation with latent variables or incomplete data. It alternates estimating latent-variable distributions under current parameters with maximizing expected complete-data log-likelihood.

Expectation step: The EM step that computes a distribution over latent variables using current parameter estimates and observed data. It supplies expected latent assignments while holding the current parameters fixed.
Gaussian mixture model: A probabilistic model representing data as a weighted mixture of Gaussian distributions. EM alternates soft component assignments with updates to mixture weights, means, and covariances.
Arthur P. Dempster: A statistician who coauthored the 1977 paper that formalized the expectation–maximization algorithm. His paper with Laird and Rubin established EM as a general statistical procedure.
Direct maximum-likelihood optimization: Numerical optimization of an observed-data likelihood without introducing EM’s alternating latent-variable steps. It can converge faster, but its objective may be harder to optimize or differentiate.
Maximization step: The EM step that updates parameters by maximizing expected complete-data log-likelihood. It turns the expectation step’s latent-variable estimates into revised model parameters.
Baum–Welch algorithm: An EM algorithm for estimating parameters of a hidden Markov model from observation sequences. Its forward–backward calculations perform the expectation step over hidden state sequences.
Nan Laird: A statistician known for work on statistical methods, including the 1977 formalization of EM. The foundational paper she coauthored gave the algorithm its standard formulation.
Gradient descent: An optimization method that updates parameters in the direction of decreasing objective gradient. Unlike EM’s latent-variable decomposition, it takes steps directly on an objective function.
Complete-data likelihood: The likelihood of observed and latent variables treated together as if all data were available. EM optimizes its expected log-likelihood rather than directly solving the incomplete-data objective.
Missing data: Values that are unavailable for some observations in a dataset. EM can estimate model parameters by integrating over unobserved values rather than discarding incomplete cases.

Linked from 8 pages