PaperlayerView exact revision on arXiv

Source-linked AI summary

What Makes for Good Views for Contrastive Learning?

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, Phillip Isola

arXiv:2005.10243v3cs.CVcs.LG

TL;DR

The paper studies how view choice affects contrastive representations and argues that views should retain task-relevant information while minimizing irrelevant mutual information. It analyzes task-dependent optimal views, proposes methods for learning them, and reports 73.0% ImageNet linear-readout accuracy with ResNet-50.

  • Problem

    Contrastive learning lacks sufficient understanding of how view choices affect representations, while useful views must preserve downstream task information without retaining irrelevant variation.

  • Method

    The paper combines theoretical and empirical analysis with unsupervised and semi-supervised methods that learn task-effective views by reducing mutual information, including stronger data augmentation.

  • Results

    73.0% ImageNet linear-readout accuracy was achieved with a ResNet-50 using the proposed data augmentation and view-selection understanding.

  • Takeaways & Limitations

    Good contrastive views should share the minimal information necessary for downstream performance while minimizing irrelevant nuisances, and their optimal construction depends on the task.

  • Takeaways & Limitations

    View construction is not bias-free: choices such as the color space can hide or reveal different properties of the data.

Abstract

from arXiv · show

Contrastive learning between multiple views of the data has recently achieved state of the art performance in the field of self-supervised representation learning. Despite its success, the influence of different view choices has been less studied. In this paper, we use theoretical and empirical analysis to better understand the importance of view selection, and argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. To verify this hypothesis, we devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. We also consider data augmentation as a way to reduce MI, and show that increasing data augmentation indeed leads to decreasing MI and improves downstream classification accuracy. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification ($73\%$ top-1 linear readout with a ResNet-50). In addition, transferring our models to PASCAL VOC object detection and COCO instance segmentation consistently outperforms supervised pre-training. Code:http://github.com/HobbitLong/PyContrast

1 Introduction

Contrastive learning must choose views that preserve task-relevant information without retaining excessive nuisance variation. The paper argues for an InfoMin principle, showing task-dependent optimal views, a mutual-information sweet spot, learned view-selection methods, and 73.0% ImageNet accuracy.

  • Contrastive multiview learning brings views of the same scene together and pushes views of different scenes apart in representation space.
  • The central design question is which viewing conditions should be treated as invariant, because excessive or insufficient invariance can discard task-relevant information or preserve nuisance-specific distinctions.
  • Optimal views depend critically on the downstream task, so task knowledge can guide the design of effective views.
  • The InfoMin principle proposes views sharing the minimal information necessary for downstream performance while discarding nuisance information.The paper relates this principle to minimal sufficient statistics and Information Bottleneck theory.
  • The paper introduces a semi-supervised method for learning task-effective views and applies the principle through stronger data augmentation to reduce mutual information.
  • 73.0% top-1 accuracy is achieved on the ImageNet linear readout benchmark with a ResNet-50.

2 Related Work

Related work establishes contrastive representation learning as a competitive self-supervised paradigm and describes views as a primary way to construct positive pairs. Prior studies also show that labels can guide representations toward task-relevant features, while this paper uses labels for view learning but not contrastive representation learning.

  • Contrastive losses learn representations by pulling similar pairs together and pushing dissimilar pairs apart, and have outperformed other approaches in several studies.
  • Positive pairs are commonly generated by creating multiple views, including channel decompositions, random image augmentations, video time steps, and image patches.
  • Prior work shows that labeled data can guide contrastive representations toward task-relevant features that improve performance.
  • This paper uses labeled data to learn better views while performing contrastive learning with only unlabeled data, leaving joint use of labels for both stages to future work.

3 What Are the Optimal Views for Contrastive Learning?

The paper frames view selection as controlling which information contrastive representations retain, proposing views that share minimal task-relevant information while discarding irrelevant information. Theory and experiments identify a task-dependent sweet spot, with performance varying across view constructions and downstream tasks.

  • 3.1 Multiview Contrastive Learning: Contrastive learning estimates mutual information between two views and learns representations by distinguishing joint-distribution positives from product-of-marginals distractors.InfoNCE maximizes a lower bound on I(v1; v2), with representations produced by encoders applied to the two views.
  • 3.1 Multiview Contrastive Learning: Minimal sufficient encoders retain information required by the contrastive objective while discarding other irrelevant information.This is useful when the information needed by the downstream task is shared between views.
  • 3.2 Three Regimes of Information Captured: View quality depends on shared information: missing task information degrades performance, a task-complete noise-free overlap is optimal, and excess shared information adds irrelevant noise.The three regimes are defined by whether shared information is below, equal to, or above the task-relevant information needed by the task.
  • 3.2 Three Regimes of Information Captured: The InfoMin principle proposes views sharing the minimal information necessary for strong downstream performance, subject to retaining task-relevant information.The proposition assumes minimal sufficient encoders and a known downstream task.
  • 3.3 View Selection Influences Mutual Information and Accuracy: Because downstream task information is often unavailable during unlabeled training, the paper evaluates view choices empirically using INCE while holding other settings fixed.The authors note that INCE is a neural proxy for mutual information and depends on network architecture.
  • 3.3 View Selection Influences Mutual Information and Accuracy: Patch-based views show a reverse-U relationship between INCE and accuracy, while color-space views improve downstream classification and segmentation as INCE decreases.For patch views, the sweet spot occurs at d = 128; color-space experiments use STL-10 classification and NYU-V2 segmentation.
  • 3.4 Data Augmentation to Reduce Mutual Information between Views: InfoMin Aug. applies stronger, varied augmentations to reduce mutual information and achieves 73.0% ImageNet linear-readout accuracy with ResNet-50, outperforming SimCLR by nearly 4%.The resulting models also consistently outperform supervised ImageNet pre-training when transferred to PASCAL VOC detection and COCO instance segmentation.

4 Learning views for contrastive learning

The paper develops unsupervised and semi-supervised methods for learning views that retain task-relevant information while reducing irrelevant shared information. Experiments show that view quality depends on the downstream task and that semi-supervised views can outperform raw inputs.

  • Optimal views depend critically on the downstream task, so view construction should preserve information needed for the target task.
  • The toy dataset combines digit class, digit position, and background-image factors to test which shared information contrastive learning extracts.The evaluation freezes the backbone and trains linear heads for digit classification, digit localization, and background classification.
  • Sharing only background information can prevent representations from capturing digit class and location, while shared background can also overwhelm other factors.
  • Flow-based view generators transform images into novel pixel-wise color spaces and split channels into two views for contrastive learning.The generator uses invertible volume-preserving or non-volume-preserving flows, while encoders maximize and the generator minimizes INCE between views.
  • The unsupervised objective often reduces INCE too far and is unstable because the generator lacks downstream-task information.This motivates adding supervised classifiers that preserve label information in each learned view.
  • 94% classification accuracy is reached by learned views g(RGB), which consistently outperform raw inputs on larger backbones.The semi-supervised generator significantly outperforms the supervised generator, validating the value of reducing inter-view mutual information.

5 Conclusion

The conclusion characterizes good views through the InfoMin principle: retain task-relevant information while minimizing irrelevant nuisances. It reports task-dependent optimal views, semi-supervised view learning, and improved ImageNet linear-readout performance.

  • Good views retain task-relevant information while minimizing irrelevant nuisances, an objective the paper calls the InfoMin principle.
  • Optimal views are task-dependent in both theory and practice, motivating a semi-supervised method for learning effective views for a given task.
  • A new set of data augmentations achieves state-of-the-art top-1 accuracy on the ImageNet linear-readout benchmark with a ResNet-50.

Broader Impact

The paper frames view selection as relevant to the broader practice of representation learning and downstream use of uncurated data. It also acknowledges that view constructions can encode biases by hiding or revealing different data properties.

  • Self-supervised representation learning may reduce reliance on labeled data and enable more cost-effective use of uncurated data.
  • Uncurated data may contain errors and biases that human curation would otherwise uncover.
  • View constructions are not bias-free: choosing one color space over another may hide or reveal different properties of the data.The paper compares view choice with the choice of training data and annotations in supervised learning.

A.2 Proof

The proof establishes that optimal views and minimal sufficient encoders preserve all task-relevant information while discarding task-irrelevant information. The resulting representations are sufficient and minimal for predicting the downstream label.

  • Optimal views yield representations that are sufficient statistics of their respective views for predicting the task label.This gives I(z1; y) = I(v1; y) or I(z2; y) = I(v2; y).
  • The proof uses conditional-independence and sufficiency properties to show that no label-relevant information is lost in z1.Because z1 is a function of v1 and the optimal views satisfy the required conditional-independence relation, I(y; v1|z1) = 0.
  • The proof derives sufficiency through nonnegativity of mutual information and the equality I(v1; v2) = I(v2; z1).These identities establish the zero conditional mutual-information terms needed for sufficiency.
  • The representations z1 and z2 are minimal for the task label y.A minimal sufficient encoder minimizes I(z1; v1) to I(v1; y), yielding I(z1; v1|y) = 0; the same applies to z2.
  • The experimental evaluation uses STL-10 and CIFAR-10 classification, plus STL-10 and NYU-Depth-v2 segmentation tasks.The learned backbone is evaluated with linear classifiers or a decoder trained on frozen representations.

B.3 Reducing I(v1; v2) with Frequency Separation

Frequency separation creates views by splitting an image into blurred low-frequency and residual high-frequency components. The experiments show that mutual information and task accuracy depend on both the amount and trajectory of frequency separation.

  • Frequency separation: v1 = Blur(x, σ) and v2 = x − v1 separate an image into low- and high-frequency views controlled by σ.Extremely small or large σ can leave one frequency component with little information.
  • Frequency separation: σ = 0.7 maximizes INCE on STL-10, while increasing σ produces a reverse-U accuracy curve with a sweet spot at σ = 1.0.The accuracy first improves and then drops as σ increases from 0.7.
  • Frequency separation: Decreasing σ from 0.7 causes accuracy to keep diminishing, despite also reducing INCE.The two trajectories indicate that mutual information alone does not determine representation quality; the shared information matters too.
  • Task-dependent views: The Moving-MNIST setup fixes v1 as past frames and constructs v2 from a later frame to test task-dependent shared factors.Downstream tasks predict digit class, digit location, or background class from frozen representations.
  • Learned views: The invertible view generator includes volume-preserving and non-volume-preserving building blocks, followed by contrastive learning and evaluation.Unsupervised view learning was relatively unstable and used different learning rates for the view generator and INCE approximator.

C.2 Analysis of Data Augmentation as it relates to MI and Transfer Performance

Stronger augmentation changes mutual information and downstream accuracy along reverse-U curves. The experiments identify augmentation-specific sweet spots and show that nonlinear projection heads can be more sensitive near those optima.

  • Augmentation strength: Increasing augmentation strength produces practical reverse-U relationships between INCE and linear classification accuracy.This pattern appears when varying individual augmentation functions, including cropping and color jittering.
  • Cropping: c = 0.2 is the cropping sweet spot for both linear and nonlinear critic heads.Decreasing the RandomResizedCrop lower bound c means more aggressive augmentation; the commonly used value is 0.08.
  • Color jittering: x = 1.0 is the color-jittering sweet spot, matching the value used in SimCLR.The reverse-U curve appears with both linear and nonlinear projection heads.
  • Projection heads: Accuracy is more sensitive around the sweet spot for nonlinear projection heads than for linear heads.The same sensitivity pattern also occurs for cropping, emphasizing the importance of tuning augmentation strength.

C.3 Results on ImageNet Benchmark

InfoMin pre-training improves ImageNet linear evaluation and transfers effectively to detection and segmentation. The appendix also notes that manual augmentation design has not yet reached the identified sweet spot.

  • ImageNet benchmark: InfoMin reaches 70.1% with 200 epochs, outperforming SimCLR trained for 1000 epochs.The reported 73.0% result is obtained after extending training to 800 epochs.
  • ImageNet benchmark: Adding Jigsaw to RA-CJ-Blur improves linear-classifier accuracy from 63.6% to 65.9%, while a 2-layer MLP raises it to 67.3%.These results use 100 epochs of pre-training.
  • Limitations: Manual augmentation design has not yet reached the sweet spot, leaving room for learned or improved synthetic views.The authors propose extending semi-supervised view learning to synthesize better augmentations.
  • Architectures: Increasing network capacity significantly improves InfoMin ImageNet linear-readout performance.The appendix reports this trend across different architectures.
  • Transfer learning: InfoMin pre-trained models outperform supervised cross-entropy pre-training across the tested detection frameworks and fine-tuning schedules.This transferability result includes varied backbones and detectors; one ResNeXt-152 comparison uses ImageNet-1K versus supervised ImageNet-5K.

D.4 ResNet-101 with Mask R-CNN, FPN architecture

This section reports Mask R-CNN experiments using R-101 and ResNeXt FPN backbones across several training schedules. The referenced tables compare COCO detection and instance segmentation results with supervised ImageNet pre-training counterparts.

  • Mask R-CNN with an R-101 FPN backbone was evaluated using 1x, 2x, and 6x schedules.
  • The referenced COCO tables report object detection and instance segmentation results relative to ImageNet supervised pre-training counterparts.
  • Cascade Mask R-CNN with an R-101 FPN backbone was also evaluated using 1x, 2x, and 6x schedules.
  • Mask R-CNN with an X-101 FPN backbone was evaluated using 1x and 2x schedules.

D.7 ResNeXt-152 with Mask R-CNN, FPN architecture

This section evaluates Mask R-CNN with an X-152 FPN backbone under a 1x schedule and compares InfoMin with supervised pre-training. InfoMin still outperforms the supervised model despite using ImageNet-1K rather than the supervised model’s ImageNet-5K pre-training.

  • InfoMin outperforms supervised pre-training with Mask R-CNN and an X-152 FPN backbone under a 1x schedule.The comparison is reported for COCO object detection and instance segmentation.
  • The InfoMin model was pre-trained on ImageNet-1K, whereas the supervised model was pre-trained on ImageNet-5K.ImageNet-5K is described as 6x larger than ImageNet-1K.
  • The X-152 FPN experiment used a 1x schedule for Mask R-CNN.
  • The section’s referenced tables describe COCO object detection and instance segmentation results as gaps to ImageNet supervised pre-training counterparts.
Loading 2005.10243v3…