Information theory

Information theory is the mathematical study of uncertainty, information, compression, and communication. It establishes limits on how efficiently data can be represented and how reliably messages can be transmitted through noisy channels.

Shannon entropy: A measure of the average uncertainty of a random variable, defined from the probabilities of its possible outcomes. It is the central measure of information in Shannon’s framework.
Source coding theorem: Shannon’s theorem stating the fundamental limit on lossless compression of a random source. It explains why average code length cannot fall below source entropy.
Data compression: The encoding of data using fewer bits than its original representation. Information-theoretic limits distinguish achievable lossless and lossy compression.
Claude Shannon: An American mathematician and engineer who founded modern information theory. His 1948 work established the field’s core measures and coding limits.
Quantum information theory: The study of information processing and communication using quantum states and operations. Quantum states introduce information resources and limits absent from classical channels.
Conditional entropy: The average uncertainty remaining about one random variable when another is known. It quantifies uncertainty that remains after side information is available.
Shannon–Fano coding: A prefix-code construction that recursively divides outcomes into groups of similar total probability. It illustrates early probability-based compression, though it is not always optimal.
Digital communication: The transmission of information using discrete symbols represented by digital signals. Its designs balance transmission rate, bandwidth, and errors using channel capacity.
A Mathematical Theory of Communication: Claude Shannon’s 1948 paper that formalized information measures and fundamental communication limits. It introduced the framework that gave information theory its modern form.
Quantum entanglement: A quantum correlation in which the joint state of systems cannot be represented as independent states. Entanglement can serve as a resource for communication and information processing.

Linked from 64 pages