Source-linked AI summary
A Mini-Introduction To Information Theory
Edward Witten
TL;DR
The article introduces classical and quantum information-theoretic quantities while addressing how these concepts support hypothesis testing, quantum-state description, teleportation, and information encoding. It develops key properties and quantum applications, including asymptotic links between quantum and classical relative entropy and bounds on encoded classical information.
Problem
The article addresses how probability distributions and quantum states can be described and compared, and how quantum information measures relate to operational tasks such as teleportation and encoding.
Method
The article gives a short conceptual introduction to Shannon and von Neumann entropy, relative entropy, conditional entropy, and mutual information, then develops selected quantum applications using measurements, many-copy asymptotics, and entropy properties.
Results
The article derives that quantum relative entropy can asymptotically match classical relative entropy after suitable measurements, and that an k-dimensional quantum state carries at most log k bits of classical information.
Takeaways & Limitations
Entropy and relative-entropy properties provide a framework for analyzing quantum teleportation, hypothesis testing, and limits on classical information encoded in quantum states.
Takeaways & Limitations
The relation between formally defined quantum mutual information and information gained about system A by observing system B is not obvious and is explored only in one aspect.
Abstract
from arXiv · showhide
This article consists of a very short introduction to classical and quantum information theory. Basic properties of the classical Shannon entropy and the quantum von Neumann entropy are described, along with related concepts such as classical and quantum relative entropy, conditional entropy, and mutual information. A few more detailed topics are considered in the quantum case.
1 Introduction
This article provides a very short introduction to classical and quantum information theory. It presents core entropy concepts and explores selected quantum topics in more detail.
- The article is intentionally brief and points readers to introductory books and lecture notes for broader coverage.
- Classical information theory: Section 2 introduces Shannon entropy and related classical concepts, including conditional entropy, relative entropy, and mutual information.
- Quantum information theory: Section 3 presents the corresponding quantum concepts: von Neumann entropy, quantum conditional entropy, quantum relative entropy, and quantum mutual information.
- Detailed quantum topics: Section 4 examines selected quantum topics to test how closely these concepts match the intuition suggested by their names.
2 Classical Information Theory
The classical introduction develops Shannon entropy and related measures for messages and random variables, then uses relative-entropy properties to explain information and inference. It establishes nonnegativity, monotonicity, and mutual-information principles while noting asymptotic and modeling qualifications.
- 2.1 Shannon Entropy: The entropy of a long message is asymptotic because the derivation uses only Stirling’s leading term and ignores fluctuations in letter frequencies.
- 2.1 Shannon Entropy: Shannon entropy measures the information in a probability distribution and determines the asymptotic number of possible messages and bits needed to encode long messages.For an alphabet with k letters, entropy is maximized by the uniform distribution; the message length is asymptotically N S_A bits.
- 2.2 Conditional and Mutual Information: Conditional entropy S(X|Y) is the information remaining about X after observing Y, while mutual information measures how much observing Y reveals about X.
- 2.3 Relative Entropy: Relative entropy quantifies disagreement between a hypothesized and correct distribution, is nonnegative, and vanishes only when the two distributions coincide.
- 2.3 Relative Entropy: 2^-N S(P_X||Q_X) describes the large-N decay of the chance of falsely excluding a correct hypothesis under the stated noise-free analysis.
- 2.4 Monotonicity of Relative Entropy: Strong subadditivity extends to quantum mechanics, while mutual-information monotonicity states that observing Y and Z reveals at least as much about X as observing Y alone.
- 2.4 Monotonicity of Relative Entropy: Relative entropy decreases when variables are integrated out, expressing monotonicity under loss of observations.
3.1 Density Matrices
Quantum states of subsystems are represented by density matrices, which support correct predictions without describing an entire universe or environment. Density matrices can be purified, reduced by partial trace, and admit nonunique ensemble decompositions.
- Pure and mixed states: A pure product state gives a pure subsystem state, whereas entanglement generally produces a mixed reduced state.A reduced density matrix has rank greater than one when the subsystem is mixed.
- Density matrices: A subsystem can be described by a density matrix when only measurements on that subsystem are performed.This avoids tracking a wavefunction for the universe or all outgoing systems interacting with it.
- Purification: Every density matrix can be realized as the reduced state of a pure state on a bipartite system, called a purification.The purification is not unique, but any two purifications of the same state are related by a unitary transformation on the auxiliary system.
- Ensemble decompositions: An ensemble decomposition of a density matrix is generally not unique, so measurements cannot reveal how the system was prepared.Uniqueness requires distinct probabilities together with a minimal orthonormal decomposition; otherwise multiple decompositions describe the same measurement statistics.
- Partial trace: The reduced density matrix of a subsystem is obtained by taking the partial trace over the unobserved subsystem.Operationally, this sums over the unobserved states while retaining the state needed for measurements on the observed system.
3.2 Quantum Entropy
The von Neumann entropy extends Shannon entropy to density matrices through their eigenvalue probabilities. It retains key entropy properties while exhibiting specifically quantum behavior, including equal subsystem entropies for pure bipartite states.
- Definition: The von Neumann entropy of a density matrix is defined analogously to Shannon entropy using its eigenvalue probabilities.After diagonalization, it equals the Shannon entropy of the corresponding probability distribution.
- Basic properties: Entropy reaches its upper bound only when the density matrix is proportional to the identity, corresponding to a maximally mixed state.The upper-bound equality condition parallels the classical maximum-entropy condition.
- Quantum analogues: Quantum conditional and relative entropy are introduced as quantum analogues whose properties require more subtle explanations and can differ from classical intuition.The article notes that these quantities are invariant under suitable unitary transformations.
- Bipartite systems: For a pure bipartite state, the two reduced density matrices have the same entropy while the joint entropy is zero.Thus a subsystem and its purifying partner share entropy even though the combined system is pure.
3.3 Concavity
Von Neumann entropy is concave: mixing density matrices cannot reduce the entropy below the corresponding mixture of their individual entropies. Dephasing a state by removing off-diagonal elements therefore increases entropy, strictly unless the state was already diagonal.
- Concavity: For ρ(t)=tρ1+(1−t)ρ2, concavity places the entropy of the mixture at least as high as the weighted endpoint entropies.The same property extends to arbitrary finite mixtures with nonnegative weights summing to one.
- Mixing: Entropy can only increase under mixing, with the nonnegative difference identified as the Holevo information χ.This generalizes the two-state concavity inequality to multiple density matrices.
- Dephasing: Removing off-diagonal elements from a density matrix in any basis cannot decrease its entropy.The argument interpolates between the diagonal state and the original state and uses concavity along that path.
- Dephasing: The entropy increase from removing off-diagonal elements is strict unless the original density matrix is already diagonal in the chosen basis.The second derivative at the interpolation endpoint is strictly negative unless ρ=ρD.
3.4 Conditional and Relative Quantum Entropy
Quantum information theory formally extends classical conditional entropy, relative entropy, and mutual information, but conditional entropy can become negative and lacks a direct conditional-probability interpretation. Relative entropy remains nonnegative, yielding nonnegative mutual information and related entropy inequalities.
- Conditional entropy: Quantum conditional entropy is formally defined by imitating the classical case, but it is not an entropy conditioned on quantum probabilities.The paper notes that quantum conditional probabilities do not have a good general notion, while discussing an analogous behavior later.
- Conditional entropy: In an entangled pure state, S(A|B) is negative because S_AB = 0 while S_B > 0.
- Mutual information: Quantum mutual information is nonnegative and vanishes exactly when the bipartite density matrix factorizes.
- Purifications: Purifications provide bounds and equivalent formulations involving conditional entropy, mutual information, and related entropy inequalities.For a pure ABC state, the passage relates S_AB to S_C and S_B to S_AC before invoking mutual-information positivity.
- Relative entropy: Quantum relative entropy is nonnegative, with equality precisely when the two density matrices are identical.
- Mutual information: Positivity of quantum relative entropy implies positivity of mutual information, also called subadditivity of entropy.
3.5 Monotonicity of Relative Entropy
Monotonicity of quantum relative entropy under partial trace establishes monotonicity of mutual information and strong subadditivity. These results preserve selected classical intuitions despite the absence of a general joint probability distribution for quantum observables.
- Monotonicity: Taking a partial trace can only reduce quantum relative entropy, a statement also called the Data Processing Inequality.
- Strong subadditivity: Monotonicity of relative entropy is equivalent to monotonicity of mutual information and strong subadditivity.
- Purifications: Purifications generate equivalent entropy relations, including cases where conditional entropies are negative.
- Interpretation: The monotonicity result supports the classical intuition that observing B and C provides at least as much information about A as observing B alone.
- Significance: Strong subadditivity is a key source of many useful statements in quantum information theory.
3.6 Generalized Measurements
Generalized quantum measurements are constructed by coupling a system to an auxiliary system, applying a unitary, and then performing a projective measurement. The resulting operators define outcome probabilities and post-measurement states, including POVMs as a broader measurement class.
- POVMs: POVM elements are nonnegative Hermitian operators summing to the identity, generalizing orthogonal projection operators.
- Construction: A generalized measurement couples the system to an auxiliary system, applies a unitary, and measures the auxiliary system projectively.
- Measurement operators: The operators E_s satisfy a completeness relation, while outcome probabilities and conditional post-measurement states are computed from them.
- Extensions: For the direct-sum extension, the same operators determine outcome probabilities and conditional density matrices after measurement.
3.7 Quantum Channels
Quantum channels provide the general physically sensible evolution of density matrices by combining unitary dynamics, auxiliary systems, and partial traces. Relative entropy cannot increase under these channels.
- Construction: The channel construction initializes an auxiliary system, applies a unitary, and obtains the output by partial trace.
- Quantum channels: A quantum channel maps density matrices through Kraus operators, with unitary evolution as the special case of one Kraus operator.
- Quantum channels: The most general physically sensible density-matrix evolution has the quantum-channel form, including maps between different Hilbert spaces.
- Monotonicity: Quantum relative entropy can only decrease under a quantum channel because initialization and unitary conjugation preserve it, while partial trace reduces it.
- Examples: The section concludes with exercises on channels that prepare pure or maximally mixed states, diagonalize density matrices, compose channels, and implement partial traces.
3.8 Thermodynamics And Quantum Channels
The section connects quantum relative entropy with free energy and entropy under quantum channels that preserve thermal equilibrium. These channels cannot increase free energy, and in the infinite-temperature limit they cannot decrease entropy.
- Thermal relative entropy: For a thermal state σ at temperature T = 1/β, quantum relative entropy expands as S(ρ||σ) = −S(ρ) + Trρ(βH + log Z).The partition-function term is independent of ρ and ensures S(σ||σ) = 0.
- Free-energy monotonicity: A quantum channel preserving thermal equilibrium can only reduce the free energy of an arbitrary state.This is identified as an aspect of the second law of thermodynamics.
- Infinite-temperature limit: At T →∞, reducing free energy is equivalent to increasing entropy, so channels preserving the maximally mixed state can only increase entropy.The maximally mixed state is the thermal state at infinite temperature.
- Unital channels: A channel mapping every density matrix to its diagonal form preserves the maximally mixed state and therefore satisfies S(ρ) ≤ S(ρD).This channel provides an explicit example of entropy increase under a unital quantum channel.
4 More On Quantum Information Theory
The section develops quantum-information concepts through teleportation, hypothesis testing, and classical communication encoded in quantum states. It interprets conditional and relative entropy operationally, including teleportation criteria, distinguishability rates, and a dimension-based communication bound.
- Quantum Teleportation and Conditional Entropy: Teleportation is possible in the generalized state-merging sense exactly when quantum conditional entropy satisfies S(A|B) ≤ 0.The necessity follows from preserving a reference system’s state, while sufficiency is established asymptotically using many copies.
- Quantum Teleportation and Conditional Entropy: Positive S(A|B) counts the maximally entangled qubit pairs Alice must send to enable teleportation or state merging without further quantum communication.Each transmitted half-pair leaves SAB unchanged, increases SB by 1, and lowers S(A|B) by 1.
- Quantum Teleportation and Conditional Entropy: Negative S(A|B) permits teleportation or state merging initially and leaves −S(A|B) maximally entangled qubit pairs afterward.The operational statement is understood asymptotically over many copies.
- Quantum Relative Entropy and Hypothesis Testing: Quantum relative entropy controls the asymptotic ability to distinguish density matrices by measurement, with suitable measurements making quantum and classical relative entropies asymptotically equal.For N copies, the quantum relative entropy scales as NS(ρ||σ).
- Quantum Relative Entropy and Hypothesis Testing: The measurement achieving the distinguishability interpretation depends only on σ, the initial hypothesis, not on ρ, the unknown answer.Monotonicity and optimal distinguishability bounds are presented as mutually related statements.
- Encoding Classical Information In A Quantum State: An k-dimensional quantum state carries at most log k bits of classical information, and Bob’s processing cannot increase the mutual information.The bound follows from the chain of mutual-information inequalities and strong subadditivity or equivalent results.