Gibbs' inequality

Gibbs' inequality states that the expected log ratio between two probability distributions is nonnegative, with equality only when the distributions are identical.

Connect

Kullback–Leibler divergence: A measure of the discrepancy between two probability distributions, defined as an expected log ratio. Gibbs' inequality is precisely the nonnegativity of this quantity.
Probability distribution: A mathematical assignment of nonnegative probabilities to outcomes, with total probability one. The inequality compares two normalized distributions over the same outcomes.
Maximum likelihood estimation: A method that chooses model parameters maximizing the probability of observed data. The inequality helps relate expected log-likelihood to the data-generating distribution.
Total variation distance: A distance between probability distributions defined by the largest probability difference over events. Unlike relative entropy, it is symmetric and bounded for probability distributions.
Jensen's inequality: An inequality relating a convex function of an average to the average of that function. Applying it to the logarithm gives a standard proof of Gibbs' inequality.
Support (mathematics): The set of inputs where a function or measure is nonzero or concentrated. If the second distribution vanishes where the first is positive, the divergence is infinite.
Evidence lower bound: A variational objective that lower-bounds a model's log evidence using an approximate posterior. Its gap from log evidence is a nonnegative divergence by Gibbs' inequality.
Jensen–Shannon divergence: A symmetric, finite divergence formed by averaging relative entropies to a mixture distribution. It modifies the one-way divergence in Gibbs' inequality to obtain a symmetric comparison.
Log-sum inequality: An inequality bounding a sum of terms involving logarithms by a corresponding aggregate expression. It yields a discrete proof and generalizations of Gibbs' inequality.
Shannon entropy: The expected information content of outcomes drawn from a probability distribution. Gibbs' inequality implies that cross-entropy cannot fall below this entropy.