Information and communication theory

The mathematical study of representing, transmitting, and recovering information through communication systems, especially when signals are noisy or constrained.

Connect

Shannon entropy: A measure of the average uncertainty in a random variable, expressed in bits when logarithms have base two. It quantifies the information produced by a source and underlies source-coding limits.
Probability theory: The mathematical framework for describing uncertain events and random variables. Communication sources and channel noise are modeled as random variables.
Huffman coding: A lossless coding algorithm that assigns shorter binary codewords to more frequent symbols. It is a practical variable-length code grounded in source probabilities.
Claude Shannon: An American mathematician and electrical engineer whose work founded modern information theory. His mathematical framework unified source compression and reliable communication.
Kolmogorov complexity: A measure of the length of the shortest program that produces a given finite object. Unlike Shannon entropy's average uncertainty over a distribution, it measures individual descriptions.
Channel capacity: The maximum rate at which information can be transmitted with arbitrarily low error over a communication channel. It sets the reliable communication limit that coding schemes seek to approach.
Random variable: A function that assigns numerical values to outcomes of a random experiment. Source symbols and noisy channel outputs are often represented as random variables.
Reed–Solomon code: A block error-correcting code that represents data as evaluations of polynomials over finite fields. It corrects symbol errors in storage and communication systems, including optical media.
A Mathematical Theory of Communication: Claude Shannon's 1948 paper establishing core mathematical results for communication and information. It introduced foundational definitions and the channel coding theorem.
Algorithmic information theory: The study of information through program descriptions, computation, and randomness. It extends information questions from probabilistic ensembles to individual strings and objects.