Source-linked AI summary
Relevant sparse codes with variational information bottleneck
Matthew Chalk, Olivier Marre, Gasper Tkacik
TL;DR
The paper addresses the computational difficulty of information bottleneck learning for high-dimensional or non-gaussian data. It develops a variational lower-bound method, derives sparse and kernelized IB algorithms, and demonstrates recovery of sparse features and improved performance in nonlinear settings.
Problem
Information bottleneck learning becomes intractable for high-dimensional or non-gaussian data.
Method
The paper uses a variational lower-bound approximation to derive sparse IB and kernelized extensions for nonlinear input–relevance relationships.
Results
The sparse model recovered original bar features, encoded more relevant information than the Gaussian model across bottleneck strengths, and sparse kernel IB showed a small performance improvement over Gaussian kernel IB.
Takeaways & Limitations
Kernel IB provides an intermediate representation for visualizing features jointly accounting for inputs and relevance variables, with potential generalization across tasks sharing latent features.
Takeaways & Limitations
The framework's scope is bounded by choices of internal coding constraints and external inputs, and the sparse marginal is treated as an assumption motivated by analogy to sparse priors.
Abstract
from arXiv · showhide
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.
1 Introduction
The information bottleneck approach extracts aspects of complex data that are relevant to a specified variable while discarding other information. This paper develops variational, sparse, and kernelized extensions to address computationally difficult settings.
- The information bottleneck approach uses a relevance variable to determine which aspects of complex data should be retained or discarded.
- Previous IB applications include filtering, prediction, and learning, but high-dimensional or non-gaussian data make the method intractable.
- A variational approximation extends IB to sparse data, while kernelized versions address nonlinear relations between inputs and relevance variables.
2 Variational IB
The variational IB formulation replaces intractable exact updates with a lower bound optimized through alternating variational and encoding updates. Restricting the decoding distribution can make inference tractable while favoring representations whose relevance information is readily extracted.
- The bottleneck strength is controlled by the Lagrange multiplier γ, with 0 < γ < 1.
- High-dimensional or non-gaussian joint distributions make the exact IB updates intractable, motivating variational approximations.
- The variational method maximizes a lower bound on the IB objective by replacing exact distributions with tractable approximations.
- For fixed encoder parameters, it optimizes the variational decoding and marginal distributions, then optimizes encoder parameters with those distributions fixed.
- A restricted decoder such as a linear Gaussian makes information about Y easier to extract, for example through linear regression.
3 Sparse IB
Sparse IB uses variational approximations to make information bottleneck optimization tractable with a heavy-tailed response marginal, producing sparse representations that preserve relevant information. Simulations show that it recovers relevant image features and adapts representations to the relationship between inputs and targets.
- 3 Sparse IB: Sparse IB replaces the analytically tractable Gaussian response marginal with a heavy-tailed Student-t marginal, requiring variational approximations to optimize the bottleneck objective.The encoder and decoder remain linear Gaussian, while the Student-t marginal promotes sparse responses.
- 3.1 Iterative algorithm: The variational algorithm alternates updates for the decoding distributions, response-marginal parameters, and encoding distribution, with direct updates for the encoding weights.When the variational parameters ξni are unity, the updates reduce to the iterative Gaussian IB algorithm.
- 3.2 Simulations: Only sparse IB recovered the original bar features from noisy patches, while its response histograms were considerably more heavy-tailed than those of Gaussian IB.The denoising task used 10,000 patches with Gaussian noise variance σ2 = 0.005.
- 3.2 Simulations: Sparse IB encoded more relevant information than Gaussian IB across a range of bottleneck strengths, whereas a noisy-input null model performed almost identically to Gaussian IB.The comparison is shown by the information curves in Figure 1E.
- 3.2 Simulations: Under strong and weak bottlenecks, Gaussian IB produced a smooth spectrum of response magnitudes, whereas sparse IB produced a more all-or-nothing response pattern.Figure 1F reports the fraction of response variance attributed to signal fluctuations for each unit.
- 3.2 Simulations: With vertically correlated input noise, sparse IB avoided representing features co-oriented with the noise, so isolated vertical bars could not be reconstructed even without noise.The learned filters remained qualitatively similar to those obtained with uncorrelated noise.
4 Kernel IB
Kernelizing the variational IB algorithm enables nonlinear encoding through high-dimensional feature spaces while retaining tractable optimization. In occlusion tasks, sparse kIB learned nonlinear, extended features and heavy-tailed responses, with similar patterns on handwritten digits.
- Kernel formulation: Kernel IB replaces linear encoding weights with expansion coefficients over mapped training data, allowing high- or infinite-dimensional feature spaces without explicitly computing the embedding.The dual-space formulation represents responses using columns of the kernel Gram matrix.
- Kernel formulation: An L2 regularization term prevents degeneracy in the dual-space solution, while iterative solvers avoid directly inverting the large coefficient matrix.The implementation restricts coefficients to a subset of training instances to reduce computational complexity.
- Kernel formulation: The kernel method uses a two-stage procedure: tune the kernel and regularization through kernel ridge-regression performance, then run variational IB with them fixed.The kernel parameters and regularization constant are selected using held-out data.
- Occlusion task: Sparse kIB produced a small but significant performance improvement over gaussian kIB in the occlusion task, while yielding much more heavy-tailed responses.The comparison used information curves, and the response distributions were assessed for the highest-variance units.
- Occlusion task: The nonlinear sparse kIB code filled in an occluded bar when complementary segments appeared on both sides, unlike reconstructions from single segments.Some encoding units responded strongly to both segments together but near zero to either segment alone.
- Handwritten digits: On handwritten-digit patches, sparse kIB again produced heavier-tailed responses and decoding filters resembling digits that extended into the occluded region, unlike alternative IB algorithms.The task provided the left side of each patch as X and reconstructed the right side as Y.
5 Discussion
The discussion situates sparse IB among latent-variable, infomax, CCA, and kernel methods while emphasizing its interpretability and task-transfer advantages. It also connects IB to sensory-processing models by varying coding and task constraints.
- Sparse IB is closely related to sparse-coding latent-variable models with student-t priors and linear-Gaussian likelihoods, but its recognition model is conditioned differently.
- Unlike infomax, IB derives constraints on encodings automatically from its bottleneck term rather than requiring additional ad hoc constraints.
- Compared with CCA, IB lets the bottleneck parameter γ determine the number and scale of features, while kernel CCA embeds X and Y symmetrically.
- Kernel IB provides an intermediate representation that visualizes features jointly accounting for X and Y and supports generalization across tasks sharing latent features.These capabilities are identified as advantages over kernel ridge regression.
- IB offers a framework for studying sensory codes as internal parameter γ and external input X constraints vary, with suggested applications beyond early sensory processing.The discussion cites perceptual filling in as an example and proposes future modulation through changes in the relevance variable Y.