Source coding theorem

The source coding theorem characterizes the minimum rate required to represent a random source at a specified distortion level. It distinguishes lossless compression from lossy compression by the fidelity constraint imposed.

Rate–distortion function: The minimum mutual information between a source and its reconstruction needed to keep expected distortion below a chosen threshold. It gives the central rate-versus-fidelity bound for lossy source coding.
Shannon entropy: A measure of the average uncertainty of a random variable, calculated from its probability distribution. For lossless coding, entropy supplies the minimum average rate.
Transform coding: A compression method that transforms data into coefficients before quantizing and encoding them. It underlies image and audio codecs that allocate bits according to perceptual distortion.
Shannon channel coding theorem: A theorem characterizing the maximum reliable communication rate of a noisy channel. It bounds channel capacity, whereas source coding bounds the rate needed to represent data.
Shannon source coding theorem: A theorem stating that lossless source compression can approach the source entropy rate, but cannot beat it. It is the zero-distortion, lossless counterpart to the fidelity-constrained result.
Discrete memoryless source: A sequence-generating model in which each symbol is drawn independently from the same finite probability distribution. It is a standard setting in which source coding bounds are stated.
Vector quantization: A quantization method that maps input vectors to representative codewords from a finite codebook. It approximates rate–distortion-optimal block representations with a concrete codebook.
Finite-blocklength information theory: The study of coding performance when block lengths are finite rather than asymptotically large. The source coding theorem's limiting rates do not capture finite-block penalties.
Distortion measure: A function assigning a cost to representing one source outcome by a reconstruction. The chosen cost defines which reconstruction errors the rate bound permits.
Expected value: The probability-weighted average of the possible values of a random variable. Fidelity constraints commonly bound expected distortion across source outcomes.

Linked from 8 pages