PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Minimal Sufficient Representation in Contrastive Learning

Haoqing Wang, Xun Guo, Zhi-Hong Deng, Yan Lu

arXiv:2203.07004v2cs.CV

TL;DR

Contrastive learning can discard task-relevant information that is not shared between views, limiting downstream representations. The paper analyzes this insufficiency and proposes increasing representation–input mutual information; it reports theoretical and empirical evidence that minimal sufficient representations are insufficient for downstream tasks.

  • Problem

    Some downstream tasks require information that is not shared between views, so eliminating all non-shared information risks damaging representation performance.

    Source: §3.1 · p. 3

  • Method

    The paper proposes increasing mutual information between representation and input to approximately introduce more task-relevant information when downstream task information is unavailable.

    Source: §1 · p. 2

  • Results

    The authors report theoretical and empirical evidence that minimal sufficient representations are insufficient for downstream tasks because they lose non-shared task-relevant information.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Downstream sufficiency may require retaining non-shared task-relevant information rather than relying only on what views share.

    Source: §3.1 · p. 3

    The assumption that non-shared task-relevant information cannot be ignored usually holds for cross-domain transfer but may not hold for tasks on the training dataset.

    Source: §5 · p. 8

Abstract

from arXiv · show

Contrastive learning between different views of the data achieves outstanding success in the field of self-supervised representation learning and the learned representations are useful in broad downstream tasks. Since all supervision information for one view comes from the other view, contrastive learning approximately obtains the minimal sufficient representation which contains the shared information and eliminates the non-shared information between views. Considering the diversity of the downstream tasks, it cannot be guaranteed that all task-relevant information is shared between views. Therefore, we assume the non-shared task-relevant information cannot be ignored and theoretically prove that the minimal sufficient representation in contrastive learning is not sufficient for the downstream tasks, which causes performance degradation. This reveals a new problem that the contrastive learning models have the risk of over-fitting to the shared information between views. To alleviate this problem, we propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information, since we cannot utilize any downstream task information during training. Extensive experiments verify the rationality of our analysis and the effectiveness of our method. It significantly improves the performance of several classic contrastive learning models in downstream tasks. Our code is available at https://github.com/Haoqing-Wang/InfoCL.

1. Introduction

Contrastive learning approximately retains information shared between views, but that representation can omit task-relevant information needed downstream. The paper analyzes this risk and proposes increasing representation–input mutual information to retain more information when downstream tasks are unknown.

1. Introduction

  • Contrastive learning maximizes mutual information between views to obtain a sufficient representation, while its cross-view supervision approximately yields a minimal sufficient representation that removes non-shared information.The paper defines a sufficient representation as containing all shared information and a minimal sufficient representation as containing only shared information.Source: §1 · p. 1
  • When task-relevant information is not shared between views, the minimal sufficient representation contains less task-relevant information and has higher achievable classification and regression error than other sufficient representations.The paper assumes non-shared task-relevant information cannot be ignored and identifies a non-ignorable gap from the optimal representation.Source: §1 · p. 1
  • Because downstream task information is unavailable during training, the paper proposes increasing mutual information between representation and input to approximately introduce more task-relevant information.It considers both input reconstruction and high-dimensional mutual-information estimation as implementations.Source: §1 · p. 2
  • The paper theoretically analyzes contrastive learning’s risk of over-fitting to shared information and evaluates SimCLR, BYOL, and Barlow Twins on classification, detection, and segmentation tasks.It also reports analytical experiments to examine its hypotheses, theoretical analysis, and model.Source: §1 · p. 2

2. Related works

Related work frames contrastive learning as learning across views and connects the paper’s analysis to task-dependent view choice and information bottleneck theory. The paper argues that minimal sufficient representations can omit downstream task-relevant information.

2. Related works

  • Prior work finds that views optimal for one downstream task may not suit another, while this paper analyzes how contrastive learning may over-fit to information shared between views.The related-work discussion contrasts this with approaches that assume almost all downstream-relevant information is shared between views.Source: §2 · p. 2
  • Contrastive views provide supervision information to each other, a mechanism that underlies the paper’s account of the learned representation.The discussion also relates this mechanism to sufficient and minimal sufficient representations.Source: Figure 2 · p. 2, §2 · p. 2
  • In the information bottleneck account, learning first extracts task-relevant information for sufficiency and then compresses task-irrelevant information; the paper places contrastive learning in the first phase.It argues that more task-relevant information is needed to achieve sufficiency.Source: §2 · p. 2

3. Theoretical analysis and model

The section analyzes how contrastive learning’s focus on shared view information can omit information needed by downstream tasks. It proves this can disadvantage minimal sufficient representations and proposes increasing representation–input mutual information as a task-agnostic remedy.

3.1. Contrastive learning

  • Contrastive learning maximizes mutual information between view representations, and because each view supervises the other, it approximately learns a sufficient representation that retains shared information while discarding non-shared information.The account assumes sufficiently capable, well-learned encoders and sufficient data when treating the representations as sufficient.Source: §3.1 · p. 3
  • Because downstream tasks may require information absent from the other view, eliminating all non-shared information can damage task performance.The paper illustrates this with video and audio views: shared information may suffice for identity recognition but not object tracking.Source: §3.1 · p. 3

3.2. Analysis on minimal sufficient representation

  • When non-shared task-relevant information matters, minimal sufficient representations contain less task-relevant information than other sufficient representations and are expected to perform worse downstream.The analysis attributes the gap to information in the input that is not shared between views.Source: §3.2 · pp. 3–4
  • For classification, minimal sufficient representations are expected to have higher Bayes error; for regression, sufficient representations achieve lower squared prediction error under stated residual-distribution assumptions.The regression result assumes the conditional estimation error is uniform, Laplacian, or Gaussian.Source: §3.2 · p. 4

3.3. More non-shared task-relevant information

  • Without downstream-task labels, the proposed remedy adds representation–input mutual information to approximately introduce more task-relevant information.The authors also note that this term can help extract shared information early in optimization and use coefficients to control its amount.Source: §3.3 · pp. 4–5
  • One implementation reconstructs the input from its representation and maximizes an approximate conditional log-likelihood because the true conditional distribution is intractable.The approximation uses q(v|z) as a lower bound; its distribution can be chosen for the input type.Source: §3.3 · p. 5
  • A second, optional implementation uses an InfoNCE mutual-information lower bound and a stochastic representation obtained through reparameterization.It is offered as an alternative when input reconstruction is challenging or adds model parameters.Source: §3.3 · p. 5

4. Experiments

Increasing mutual information between representations and inputs improves contrastive learning across classification, detection, and segmentation, with especially clear gains on transfer tasks. Analytical experiments show that the effect depends on the task, the regularization strength, and training duration.

4. Experiments

  • The experiments evaluate SimCLR, BYOL, and Barlow Twins using reconstruction and lower-bound-estimate implementations of the input-information objective.Pretraining uses CIFAR10, STL-10, or ImageNet; linear evaluation includes source and transfer datasets, while detection and instance segmentation use VOC07+12 and COCO.Source: §4 · p. 5, §4.1 · p. 6

4.1. Effectiveness of increasing I(z, v)

  • Increasing I(z, v) improves classification on transfer datasets and significantly improves object detection and instance segmentation across the evaluated contrastive models.The paper reports that transfer gains are especially notable on DTD, VGG Flower, and Traffic Signs; dense prediction gains are reported on VOC07+12 and COCO.Source: §4.1 · p. 6

4.2. Analytical experiments

  • Removing non-shared information changes downstream classification accuracy little, consistent with contrastive learning already producing an approximately minimal sufficient representation.The comparison includes regularizers for eliminating non-shared information and SimCLR with added Gaussian noise.Source: §4.2 · p. 7
  • Increasing I(z, v) yields a reverse-U accuracy trend as λ varies, suggesting that excessive increases may introduce noise alongside useful information.The study evaluates λ values from 0.001 to 10 and reports source-dataset and averaged transfer-dataset accuracy.Source: §4.2 · p. 7, Figure 4 · p. 7
  • With longer training, transfer-dataset accuracy declines as representations increasingly emphasize shared information; increasing I(z, v) significantly improves transfer accuracy.On CIFAR10, source accuracy rises with more epochs and the added objective makes no difference; on STL-10, source accuracy declines and the added objective improves it without that decline over training.Source: Figure 5 · p. 7, §4.2 · p. 8
  • In supervised pretraining, increasing I(z, x) improves transfer-dataset performance while achieving comparable results on the source dataset.The authors describe this as helping alleviate over-fitting to label information and leave further study of more general supervised-pretraining representations for future work.Source: §4.2 · p. 8, Table 4 · p. 8

5. Limitations

The paper identifies three limitations: the assumption about non-shared task-relevant information may not hold for training-dataset tasks, increasing input information can add noise, and the best SimCLR ImageNet results were not reproduced.

5. Limitations

  • The assumption that non-shared task-relevant information cannot be ignored usually holds for cross-domain transfer tasks but may not hold for tasks on the training dataset.This limits the assumption’s stated applicability across task settings.Source: §5 · p. 8
  • Increasing I(z, v) can add task-irrelevant noise and increase downstream data demand, so coefficients λ1 and λ2 may need adjustment for different tasks.The limitation concerns the trade-off between adding task-relevant and irrelevant information.Source: §5 · p. 8
  • The authors could not reproduce SimCLR’s best ImageNet results because those results require batch size 4096 and more training epochs than their resources allowed.Source: §5 · p. 8

6. Conclusions

The paper concludes that contrastive learning can discard non-shared task-relevant information, and proposes increasing mutual information between representations and inputs when downstream tasks are unknown. It also identifies combining reconstruction and contrastive learning as future work.

6. Conclusions

  • Theoretically and empirically, minimal sufficient representations can be insufficient for downstream tasks because they lose non-shared task-relevant information.Contrastive learning approximately obtains these representations, risking over-fitting to information shared between views.Source: §6 · p. 8
  • The proposed method increases mutual information between the representation and input to introduce more non-shared task-relevant information when downstream tasks are unknown.Source: §6 · p. 8
  • Future work could combine reconstruction models with contrastive learning for convolutional neural networks or vision transformers.The conclusion associates reconstruction with more sufficient information and contrast with more discriminative representations.Source: §6 · p. 8

A. Proofs of theorems

The appendix gives theorem proofs under a conditional-independence assumption and describes a choice of mutual-information estimator. Its table caption identifies linear-evaluation accuracy across source and transfer datasets.

A. Proofs of theorems

  • The proofs assume z1 is conditionally independent of other system variables once v1 is observed; the text says this holds for deterministic encoders and explains the randomized case.Source: A. Proofs of theorems · p. 11
  • Theorem 1's proof decomposes task information into a minimal-representation term and a conditional-information term, then uses an inequality for the sufficient representation.The proof states I(v1, T) = I(zmin_1, T) + I(v1, T|v2) and I(v1, T|v2) ≥ I(zsuf_1, T|v2) ≥ 0.Source: A. Proofs of theorems · p. 12

B. Choice of mutual information estimate

  • The estimator discussion selects the InfoNCE lower bound because modeling a critic jointly on input and representation is described as difficult given their different distributions.The candidates include the Nguyen–Wainwright–Jordan and MINE bounds.Source: B. Choice of mutual information estimate · p. 12
  • Table 5 reports linear-evaluation accuracy on a source dataset and other transfer datasets.Source: Table 5 · p. 13

C. More experiments

This section introduces additional experiments intended to support the paper.

C. More experiments

C.1. Results on Barlow Twins

The Barlow Twins experiments report improved downstream classification accuracy when representation–input mutual information is increased. The authors take this result as evidence that their analysis applies across different contrastive losses.

C.1. Results on Barlow Twins

  • The evaluation uses Barlow Twins, whose objective makes the cross-correlation matrix between view representations approach the identity, alongside SimCLR and BYOL discussed in the main text.The supplemental experiments use the same pre-training schedule and linear-evaluation protocol as the main text.Source: §C.1 · p. 13
  • Increasing I(z, v) improves Barlow Twins representation accuracy in downstream classification, supporting applicability of the analysis across different contrastive losses.The cited results are shown in Table 5.Source: §C.1 · p. 13

C.2. Reconstructed samples

The reconstructed images retain the original images’ shape and outline information, while appearing blurry because the reconstruction module uses mean square error loss.

C.2. Reconstructed samples

  • The reconstructed images retain shape and outline information from the original inputs.The example uses SimCLR contrastive loss and CIFAR10 training data.Source: §C.2 · p. 13, Figure 6 · p. 13
  • Mean square error loss makes the reconstructed images blurry.The passage notes that this phenomenon is also observed in vanilla variational auto-encoders [27].Source: §C.2 · p. 13
  • Figure 6 compares original input images with reconstructed images.The demonstration is for Implementation I.Source: Figure 6 · p. 13

D. Derivation of LMIB and LIP

This appendix derives specific forms of regularizers proposed to eliminate non-shared information between views, including an analytical expression and a discussion of fixing encoder variance.

D. Derivation of LMIB and LIP

  • Federici et al. [15] and Tsai et al. [44] propose regularizers to eliminate non-shared information between views and obtain minimal sufficient representations.The appendix derives the specific forms used in the main text.Source: D. Derivation of LMIB and LIP · p. 13
  • The appendix presents the regularization term from Federici et al. [15] and identifies the inverse predictive loss defined by Tsai et al. [44].The supplied passages do not provide enough detail to characterize the full forms of both terms.Source: D. Derivation of LMIB and LIP · pp. 13–14
  • For LMIB, equal variances minimize the stated ratio-sum term; with fixed upper bounds, the variances converge to their maximum during training.The authors therefore propose fixing the variance; they warn that very large variance can produce drastically changing sampled representations and poor downstream performance.Source: D. Derivation of LMIB and LIP · p. 14
  • The derivation evaluates a regularization term over the data distribution p(v1, v2) and gives an analytical expression.The passage identifies d as the dimension of z1 and z2.Source: D. Derivation of LMIB and LIP · p. 14
Loading 2203.07004v2…