Shannon entropy

Shannon entropy quantifies the average uncertainty of a probability distribution as the expected logarithm of the inverse event probability. It is largest for equally likely outcomes and zero when one outcome is certain.

Self-information: The information associated with a specific event, commonly measured as the negative logarithm of its probability. Entropy averages this event-level quantity across all possible outcomes.
Probability distribution: A function assigning probabilities to the possible outcomes of a random variable. Entropy takes this distribution as its input.
Source coding theorem: A theorem stating the limits on average code length for losslessly encoding outcomes from a source. Shannon entropy sets the asymptotic minimum average number of bits per source symbol.
Rényi entropy: A family of entropy measures parameterized by an order that changes how probabilities are weighted. Shannon entropy is its limiting case at order one.
Kullback–Leibler divergence: A measure of how one probability distribution differs from a reference distribution. It equals the entropy increase incurred when a distribution is encoded using another distribution.
Random variable: A function that assigns a numerical value to each outcome in a probability space. The outcomes whose probabilities are averaged are represented by random variables.
Data compression: The representation of data using fewer bits than its original encoding. Entropy estimates the irreducible coding cost when a source distribution is known.
Hartley entropy: The logarithm of the number of possible outcomes, assuming all are considered equally possible. It ignores unequal probabilities, unlike Shannon entropy.
Joint entropy: The Shannon entropy of a joint probability distribution over two or more random variables. It extends the same averaging rule to outcomes described by multiple variables.
Expected value: The probability-weighted average of a random variable's possible values. Entropy is the expected value of the negative log probability.

Linked from 26 pages