Source-linked AI summary
On the Difference Between the Information Bottleneck and the Deep Information Bottleneck
Aleksander Wieczorek, Volker Roth
TL;DR
The paper asks how the two Markov assumptions used in information-bottleneck models relate, since requiring both can restrict possible joint distributions. It derives and interprets a lower bound on I(T;Y) that needs only X-T-Y, then presents an IB interpretation using directed graphical models.
Problem
The relationship between T-X-Y and X-T-Y, and its effect on the set of possible joint distributions, had been neglected; requiring both can be limiting.
Source: §1 · pp. 1–2Method
The paper optimizes a lower bound on I(T;Y) that follows from DVIB’s construction and requires X-T-Y, then interprets IB models as directed graphical models.
Source: §1 · p. 2, §4 · p. 11Results
The lower-bound approach lets standard DVIB implementations optimize over more joint distributions P(X,T,Y) while specifying only the marginal T|X.
Source: §1 · p. 2, §4 · p. 11Takeaways & Limitations
The paper identifies a fundamental necessary feature of IB in directed-graphical-model terms, with the original and deep IB models satisfying it.
Source: §4 · p. 11The standard DVIB derivation uses T-X-Y, while its construction supplies X-T-Y; requiring both constrains the available joint distributions.
Source: §1 · p. 1, §2.2 · p. 5
Abstract
from arXiv · showhide
Combining the Information Bottleneck model with deep learning by replacing mutual information terms with deep neural nets has proved successful in areas ranging from generative modelling to interpreting deep neural networks. In this paper, we revisit the Deep Variational Information Bottleneck and the assumptions needed for its derivation. The two assumed properties of the data $X$, $Y$ and their latent representation $T$ take the form of two Markov chains $T-X-Y$ and $X-T-Y$. Requiring both to hold during the optimisation process can be limiting for the set of potential joint distributions $P(X,Y,T)$. We therefore show how to circumvent this limitation by optimising a lower bound for $I(T;Y)$ for which only the latter Markov chain has to be satisfied. The actual mutual information consists of the lower bound which is optimised in DVIB and cognate models in practice and of two terms measuring how much the former requirement $T-X-Y$ is violated. Finally, we propose to interpret the family of information bottleneck models as directed graphical models and show that in this framework the original and deep information bottlenecks are special cases of a fundamental IB model.
1 Introduction
The paper revisits the assumptions behind the Deep Variational Information Bottleneck (DVIB), focusing on how the Markov chains T-X-Y and X-T-Y constrain possible joint distributions. It shows that DVIB can optimize a lower bound on I(T;Y) without requiring T-X-Y, and outlines the paper’s structure.
1 Introduction
- The original IB assumes T-X-Y, while DVIB also satisfies X-T-Y by construction.The paper investigates whether requiring both chains limits the available joint distributions P(X,Y,T).Source: §1 · p. 1
- The relationship between the two Markov assumptions and their effects on possible solutions had not been examined, motivating the paper’s analysis.The paper’s proposed route is to optimize a lower bound on I(T;Y) that follows from the model’s construction.Source: §1 · p. 2
- The paper proceeds from IB and DVIB models to a lower bound on I(T;Y), a directed-graphical-model interpretation, and concluding remarks.These topics are assigned to Sections 2, 3, and 4, respectively.Source: §1 · p. 2
2 Related Work on the Deep Information Bottleneck Model
This section reviews the Information Bottleneck (IB), its deep and Gaussian variants, and related approaches to bounding mutual information. It introduces the assumptions and model constructions relevant to the paper’s later analysis.
2 Related Work on the Deep Information Bottleneck Model
- The IB has also been applied to deep-network analysis, including quantifying information between layers and proposing explanations of network behavior.The cited work derives an information-theoretic limit on DNN efficiency.Source: §2 · p. 2
2.1 Information Bottleneck
- The IB compresses X while preserving information about Y; its objective trades off compression and preservation, and its derivation uses T-X-Y.For general IB, no analytical solution is available; discrete cases can be approximated with Blahut-Arimoto.Source: §2.1 · p. 3
- For jointly Gaussian X and Y, the Gaussian IB has a Gaussian solution and an analytical optimum for fixed β, whereas the general IB lacks an analytical solution.The analytical solution depends on covariance structure and eigenvectors and eigenvalues of Σ_X|Y Σ_X^-1.Source: §2.1 · pp. 3–4
- The sparse Gaussian IB restricts the projection matrix A to be diagonal, reducing the optimization to diagonal matrices with positive entries.The reduction follows from the stated log-determinant identity.Source: §2.1 · p. 4
2.2 Deep Variational Information Bottleneck
- DVIB parameterizes P(T|X) and P(Y|T) with neural networks to recover the IB mutual-information terms; its derivation uses T-X-Y.The decoder samples from T, while the stochastic encoder samples T from X.Source: §2.2 · pp. 4–5
- The deep copula IB transforms inputs using empirical and Gaussian CDFs to depend on the copula rather than the marginals; prior work reports improved interpretability and disentanglement.The transformation addresses concerns about monotonic-transformation invariance and marginal specifications in DVIB.Source: §2.2 · p. 5
2.3 Bounds on Mutual Information in Deep Latent Variable Models
- Prior work derives variational bounds for I(T;Y) and rate-distortion bounds for I(X;T), alongside Gaussian relaxations and mutual-information regularizers in generative models.These approaches include comparisons to Canonical Correlation Analysis and bounds involving a latent space and generator network.Source: §2.3 · pp. 5–6
- Work connecting dropout to DVIB notes both Markov properties but does not analyze their impact or relationship.That work states T-X-Y is enforced by construction while X-T-Y is approximated by the optimal joint distribution.Source: §2.3 · p. 6
3 The Difference Between Information Bottleneck Models
Section 3 examines how the Information Bottleneck and Deep Variational Information Bottleneck use different Markov assumptions, then interprets their models as directed graphical models.
3.1 Clarifying the Discrepancy between the Assumptions in IB and DVIB
- DVIB can retain the constructed X-T-Y chain while optimizing a lower bound on I(T;Y), avoiding the need to enforce T-X-Y.The bound uses the model’s construction without requiring the full conditional P(T|X,Y).Source: §3.1 · pp. 6–8
- The exact I(T;Y) decomposes into the optimized lower bound, H(Y), and two terms that vanish when both Markov assumptions hold.The two additional terms, I(Y;T|X) and L(Y;T|X), measure how much T-X-Y is violated during training.Source: §3.1 · p. 8
- In DVIB, I(Y;T|X) and L(Y;T|X) track violation of T-X-Y; bound tightness measures distance to distributions satisfying both assumptions.Source: §3.1 · p. 8
3.2 The Original IB Assumption Revisited
- The T-X-Y condition is sufficient but not necessary to exclude T=Y; the necessary restriction is no direct Y→T arrow, while collider DAGs yield degenerate bottlenecks.Source: §3.2 · pp. 9–10
- A DAG view yields 10 candidate models across T-X-Y, X-T-Y, or neither; standard IB and Gaussian IB, and DVIB, occupy different categories.The paper identifies probabilistic CCA as another model in the X-T-Y category.Source: §3.2 · p. 10
3.3 Comparing IB and DVIB Assumptions
- The comparison frames original IB and DVIB as different DAGs built on different Markov assumptions, with DVIB’s bound tightness indicating violation of T-X-Y.Source: §3.3 · p. 11
4 Conclusion
The conclusion shows that DVIB can lift the original IB Markov assumption while retaining the constructionally satisfied X-T-Y chain, and frames both models as cases of a fundamental IB feature in directed graphical models.
4 Conclusion
- DVIB lifts the original T-X-Y assumption while retaining the constructionally satisfied X-T-Y chain.This allows optimisation over a larger set of joint distributions P(X,T,Y) by specifying only the marginal T|X and optimising a lower bound on I(T;Y).Source: §4 · p. 11
- The lower bound on I(T;Y), rather than the full mutual information, makes this broader optimisation possible.Source: §4 · p. 11
- The conclusion describes IB as a directed graphical model and identifies a fundamental necessary feature satisfied by both original and deep IB.Source: §4 · p. 11