PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Robust Representations via Multi-View Information Bottleneck

Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, Zeynep Akata

arXiv:2002.07017v2cs.LGstat.ML

TL;DR

Unsupervised information bottleneck learning lacks labels for identifying superfluous information. This paper uses shared information between redundant views, implements the idea as MIB, and reports strong performance across multi-view and augmented single-view settings.

  • Problem

    Without labels, unsupervised information bottleneck methods cannot directly determine which information is relevant to the predictive task and which is superfluous.

  • Method

    MIB maximizes mutual information between representations of two views while minimizing view-specific information through a symmetrized KL-based objective, with data augmentation providing views in the single-view setting.

  • Results

    MIB achieves strong performance on sketch-based image retrieval and outperforms Multi-View InfoMax when fewer MIR-Flickr labels are available.

  • Takeaways & Limitations

    Shared information across views can provide a practical unsupervised criterion for learning robust representations, including when mutual redundancy holds only approximately.

  • Takeaways & Limitations

    The mutual redundancy condition does not trivially extend beyond two views, and the authors identify multi-view extension as future work.

Abstract

from arXiv · show

The information bottleneck principle provides an information-theoretic method for representation learning, by training an encoder to retain all information which is relevant for predicting the label while minimizing the amount of other, excess information in the representation. The original formulation, however, requires labeled data to identify the superfluous information. In this work, we extend this ability to the multi-view unsupervised setting, where two views of the same underlying entity are provided but the label is unknown. This enables us to identify superfluous information as that not shared by both views. A theoretical analysis leads to the definition of a new multi-view model that produces state-of-the-art results on the Sketchy dataset and label-limited versions of the MIR-Flickr dataset. We also extend our theory to the single-view setting by taking advantage of standard data augmentation techniques, empirically showing better generalization capabilities when compared to common unsupervised approaches for representation learning.

1 INTRODUCTION

The paper extends the information bottleneck principle to unsupervised multi-view learning by treating view-specific information as superfluous. It proposes maximizing shared information between views while eliminating information not shared by them.

  • Unsupervised representation learning cannot directly identify task-relevant information without labels, making selective removal of superfluous information challenging.
  • The method assumes both views contain the same task-relevant information, so view-specific information can be discarded to improve generalization.
  • MIB maximizes mutual information between the two representations while eliminating information that is not shared between views.
  • The authors provide a theoretical analysis, define a new model, and evaluate it on Sketchy, MIR-Flickr, and single-view augmentation settings.

2 PRELIMINARIES AND FRAMEWORK

The framework defines useful representations through sufficiency for the label and minimality of information beyond the task. Without labels, it uses redundancy across views to identify information that can be safely removed.

  • A representation z is sufficient for y exactly when I(x; y|z) = 0, meaning encoding preserves all label information accessible from x.
  • Sufficient representations preserve task information while potentially removing information in x that is irrelevant for predicting y.
  • I(x; z|y) measures superfluous information in z, whereas I(y; z) measures accessible predictive information.
  • Without labels, minimizing superfluous information requires additional assumptions, motivating the use of redundant information from an additional view.

3 MULTI-VIEW INFORMATION BOTTLENECK

MIB learns robust representations by preserving information shared across mutually redundant views while discarding view-private information. Its loss balances shared-information maximization with robustness, and data augmentation extends the approach to single-view inputs.

  • 3.1 SUFFICIENCY AND ROBUSTNESS IN THE MULTI-VIEW SETTING: Mutual redundancy means each view provides no additional label information once the other view is observed: I(y; v1|v2) = 0.
  • 3.1 SUFFICIENCY AND ROBUSTNESS IN THE MULTI-VIEW SETTING: Under mutual redundancy, a representation containing all information shared by both views is as predictive for y as observing both views jointly.
  • 3.1 SUFFICIENCY AND ROBUSTNESS IN THE MULTI-VIEW SETTING: The objective reduces view-private information while preserving information that makes one representation sufficient for the other view.
  • 3.2 THE MULTI-VIEW INFORMATION BOTTLENECK LOSS FUNCTION: When the two views have identical marginals, the encoders can share parameters, requiring only one encoder.
  • 3.2 THE MULTI-VIEW INFORMATION BOTTLENECK LOSS FUNCTION: The MIB loss combines representation mutual-information maximization with a symmetrized KL divergence weighted by β, which controls the sufficiency–robustness trade-off.
  • Single-view extension: For single-view data, independent label-preserving augmentations create artificial views whose unshared variations are discarded by the training objective.

4 RELATED WORK

Prior unsupervised objectives either preserve observational information or compress without identifying which information is task-relevant. The paper positions MIB as explicitly discarding superfluous information in the unsupervised multi-view setting while retaining information shared across views.

  • Information Plane: The Information Plane characterizes representations by observational information I(x; z) and predictive information I(y; z).A desirable representation retains label information while minimizing information from the raw observations.
  • Existing objectives: InfoMax maximizes I(x; z), encouraging representations to preserve information about raw observations.It targets the high-observational-information region of the Information Plane.
  • Existing objectives: Multi-View InfoMax maximizes mutual information between views but does not explicitly remove information about the first view.Its target satisfies I(z; x) ≥ I(x; v2).
  • Existing objectives: β-VAE objectives compress representations indiscriminately, without identifying which information is relevant to the predictive task.The compression–reconstruction balance depends on β and architectural inductive biases.
  • MIB: MIB explicitly identifies and discards superfluous information in unsupervised multi-view learning without additional label supervision.Among representations optimal according to Multi-View InfoMax, it targets the least superfluous and most robust representation.

5 EXPERIMENTS

Experiments evaluate MIB in multi-view retrieval and classification, then test its information-theoretic predictions in an augmented single-view setting. Across the supplied results, compression is associated with stronger performance when labels are scarce, while MIB competes well with or exceeds comparison methods.

  • Experimental scope: MIB is evaluated on sketch-based image retrieval, Flickr multiclass image classification, and an augmented MNIST single-view task.The experiments compare MIB with domain-specific, prior multi-view, and common unsupervised representation-learning methods.
  • Sketch-based image retrieval: Sketchy training pairs an image and sketch from the same class, enabling both views to contain equivalent label information.Retrieval ranks 73,002 natural images using Euclidean distance between query-sketch and image representations.
  • Sketch-based image retrieval: MIB achieves strong mAP@all and Prec@200 on Sketchy, suggesting that its representation captures common class information between paired images and sketches.The reported effectiveness is mostly attributed to regularization from the symmetrized KL divergence between encoded views.
  • MIR-Flickr: MIB outperforms Multi-View InfoMax when fewer MIR-Flickr labels are available, while performing similarly with many labels.Larger β improves scarce-label accuracy but slightly reduces accuracy when all labels are observed.
  • Interpretation: MIB’s effectiveness may partly reflect mutual-information estimators that avoid reconstruction, unlike MVAE and VCCA.The paper notes that reconstruction terms can introduce bias that decreases performance.
  • Self-supervised single-view task: On augmented MNIST, representations retaining less data information while preserving predictive information achieve better low-label classification performance.The MIB model with β = 1 retains almost exclusively label information and hardly decreases classification performance with one label per data point.

6 CONCLUSIONS AND FUTURE WORK

The paper concludes that Multi-View Information Bottleneck performs strongly across several representation-learning tasks, while identifying extensions to more than two views and richer augmentation choices as future work.

  • 6 CONCLUSIONS AND FUTURE WORK: MIB performs strongly on sketch-based image retrieval, multi-view representation learning, and unsupervised representation learning.The experiments compare MIB empirically with other approaches across these three tasks.
  • 6 CONCLUSIONS AND FUTURE WORK: Positive MIR-Flickr results indicate that MIB can work when mutual redundancy holds only approximately.
  • 6 CONCLUSIONS AND FUTURE WORK: Future work includes extending the framework beyond two views and studying how augmentation choices connect information bottlenecks with invariant neural networks.The paper notes that mutual redundancy does not trivially extend beyond two views.
  • B THEOREMS AND PROOFS: The theory assumes representations are conditionally independent of other system variables once the source observation is observed.This assumption does not require the representation to be a deterministic function of the observation.

B.2 NO FREE GENERALIZATION

This section formalizes why reducing information in a representation can harm some predictive tasks, then shows how mutual redundancy and augmentation support sufficient representations without labels.

  • B.2 NO FREE GENERALIZATION: If a representation z′ retains less information about x than z, some label y can make z predictive while z′ is not.The theorem establishes that no information reduction guarantees preservation of predictive usefulness for every possible label.
  • B.2 NO FREE GENERALIZATION: A representation that discards observational information can therefore fail to predict a label that remains predictable from the original observations.The corollary states this under the condition I(z′; x) < H(x).
  • B.3 MULTI-VIEW REDUNDANCY AND SUFFICIENCY: For mutually redundant views, any representation of one view that is sufficient for the other is also sufficient for the target label.This follows because the second view contributes no additional label information once the first view is considered.
  • B.4 SUFFICIENCY AND AUGMENTATION: Augmentations that preserve label information can create mutually redundant views, allowing a representation sufficient for one augmented view to retain the predictive information available from x.The graphical model specifies the independence assumptions used for this argument.
  • B.5 INFORMATION PLANE: For continuous x, the first three representation information bounds remain valid, but discreteness is required to prove the fourth bound.

D NON-TRANSITIVITY OF MUTUAL REDUNDANCY

Mutual redundancy between views is not transitive: two adjacent view pairs may satisfy the condition while the outer pair does not. This higher-order interaction prevents a trivial extension of the theory beyond two views.

  • Counterexample: The example demonstrates that pairwise redundancy assumptions can coexist with a failure of redundancy for the remaining pair.Thus, reasoning from two-view relationships alone is insufficient for arbitrary multi-view systems.
  • Non-transitivity: Mutual redundancy between v1 and v2 and between v2 and v3 does not imply mutual redundancy between v1 and v3.The failure arises from higher-order interactions among multiple views and the label.
  • Counterexample: With y defined as v1 XOR v3, v1 and v3 individually contain no information about y, yet each is conditionally related through the other view.The construction uses three fair, independent binary variables.
  • Implication: The paper’s broader objective framework includes supervised Information Bottleneck and InfoMax as special cases of Multi-View Information Bottleneck.These equivalences are established under specific assumptions about shared label information or identical views.

E.1 MULTI-VIEW INFORMATION BOTTLENECK AND SUPERVISED INFORMATION BOTTLENECK

Multi-View Information Bottleneck coincides with supervised Information Bottleneck when redundant views share only label information, while identical views reduce the objective to InfoMax. The model operationalizes this through compression and cross-view information preservation.

  • Assumptions: The equivalence requires mutual redundancy and the condition that the views share only label information.These assumptions are stated as H1 and H2.
  • Supervised equivalence: A sufficient representation of v1 that is minimal for v2 is also minimal for y when mutually redundant views share only label information.This follows from the equivalence I(v1; z1|v2) = I(v1; z1|y).
  • InfoMax equivalence: When the two views are identical, no superfluous information can be identified through cross-view conditioning.In this case, a minimal sufficient representation is any representation maximizing mutual information, yielding InfoMax.
  • Objective: The practical objective combines symmetric conditional-information terms with mutual-information terms weighted by λ1 and λ2.The derivation upper-bounds the objective using a symmetrized KL divergence and a representation mutual-information term.
  • Optimization: The encoders use Normal distributions parameterized by neural networks, while the cross-representation mutual information requires an estimator.The reported experiments use Jensen-Shannon estimation, and β is annealed to avoid representation collapse.

G.2 SKETCHY EXPERIMENTS

The Sketchy experiments train on frozen sketch and image features from two views and evaluate cross-modal retrieval using 64-dimensional representations. The MIR-Flickr example illustrates that tags are only approximately redundant with images for labels.

  • Input: The sketch-based task supplies 4096-dimensional sketch and image features extracted from separate frozen VGG-16 models.Both feature extractors were pretrained on TU-Berlin images and sketches.
  • Evaluation: Evaluation compares 64-dimensional sketch and image representations using Euclidean distance for retrieval.For comparison with binary-hashing methods, the representations are also binarized and evaluated with Hamming distance.
  • MIR-Flickr caveat: MIR-Flickr tags are not always predictive of category labels, so the mutual redundancy assumption holds only approximately.The figure contrasts pictures, tags, and labels to expose this limitation of the two-view assumption.
  • Single-view extension: The MNIST single-view setup creates paired views through translation, rotation, scaling, shearing, and pixel corruption augmentations.These transformations are used to construct two views for training.

G.4.1 RESULTS AND VISUALIZATION

The MNIST results compare information content and linear-classification accuracy across models and mutual-information estimators. MIB embeddings form digit-specific clusters, supporting strong performance when labels are scarce.

  • Supplementary analysis: The supplementary section provides additional single-view MNIST visualizations and quantitative results for the models in Figure 4.These results include comparisons across mutual-information estimators.
  • Estimator comparison: The Jensen-Shannon estimator generally performs better for InfoMax, MV-InfoMax, and MIB, so it is used in the main experiments.The supplementary comparison reports estimator-dependent performance differences.
  • Quantitative results: Table 2 compares I(x; z), I(z; y), and linear-classifier accuracy across labeled-example regimes and two mutual-information estimators.The estimators are Jensen-Shannon and InfoNCE.
  • Visualization: MIB’s 64-dimensional embedding projects into roughly ten clusters corresponding to the ten MNIST digit classes.The visualization is a two-principal-component projection colored by digit class.
  • Visualization: I(x; z) ≈ I(z; y) ≈ log 10 accompanies the clustered embedding and the linear classifier’s performance in scarce-label regimes.The paper states that 10 labeled examples suffice to align cluster centroids with digit labels.

H.1 DIFFERENT RANGES OF DATA AUGMENTATION

Data augmentation helps MIB remove irrelevant information while preserving task-relevant information, but excessive corruption violates the shared-information assumption and degrades performance. The β hyperparameter controls the trade-off between representation sufficiency and minimality, as visualized against Multi-View InfoMax and β-VAE.

  • Data augmentation: MIB representations move from the top right toward the top left of the Information Plane as corruption increases, indicating removal of irrelevant input information.MV-InfoMax remains approximately in the same Information Plane region and gains little from increasing corruption.
  • Data augmentation: Near 100% corruption, MIB performance deteriorates because the mutual redundancy assumption between views is violated.
  • Hyperparameter trade-off: β controls the trade-off between sufficiency and minimality for the second view, with β = 0 reducing MIB to the Multi-View InfoMax objective.For 0 < β ≤ 1, superfluous information is gradually removed while the sufficiency constraint is enforced.
  • Hyperparameter trade-off: Values of β > 1 can violate sufficiency because minimizing I(x; z|v2) is prioritized, with the resulting trade-off compared against β-VAE in Figure 9.Figure 8 compares Information Plane coordinates and prediction accuracy across label amounts and corruption percentages; Figure 9 compares these quantities across β values.
Loading 2002.07017v2…