PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Contrastive Learning

Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, Yann LeCun

arXiv:2110.06848v3cs.LGcs.CV

TL;DR

Contrastive learning methods often depend on large batches, long training, or specialized components, motivating a more efficient baseline. The paper identifies negative-positive coupling in InfoNCE and removes the positive denominator term to form DCL. DCL delivers competitive results with reduced computational requirements, including 72.3% ImageNet-1K top-1 accuracy for NNCLR with batch size 512 in 400 epochs.

  • Problem

    Contrastive learning methods can require large batches, extensive training epochs, specialized encoders or queues, and are sensitive to hyperparameters.

  • Method

    DCL removes the positive term from the InfoNCE denominator to eliminate negative-positive coupling and improve training efficiency.

  • Results

    72.3% ImageNet-1K top-1 accuracy is achieved by combining DCL with NNCLR using batch size 512 in 400 epochs.

  • Takeaways & Limitations

    DCL provides a simple, robust contrastive-learning baseline that reaches competitive performance without large batches, momentum encoding, or long epochs.

  • Takeaways & Limitations

    DCL provides less gain over InfoNCE when batch sizes are large because the NPC multiplier approaches zero.

Abstract

from arXiv · show

Contrastive learning (CL) is one of the most successful paradigms for self-supervised learning (SSL). In a principled way, it considers two augmented "views" of the same image as positive to be pulled closer, and all other images as negative to be pushed further apart. However, behind the impressive success of CL-based techniques, their formulation often relies on heavy-computation settings, including large sample batches, extensive training epochs, etc. We are thus motivated to tackle these issues and establish a simple, efficient, yet competitive baseline of contrastive learning. Specifically, we identify, from theoretical and empirical studies, a noticeable negative-positive-coupling (NPC) effect in the widely used InfoNCE loss, leading to unsuitable learning efficiency concerning the batch size. By removing the NPC effect, we propose decoupled contrastive learning (DCL) loss, which removes the positive term from the denominator and significantly improves the learning efficiency. DCL achieves competitive performance with less sensitivity to sub-optimal hyperparameters, requiring neither large batches in SimCLR, momentum encoding in MoCo, or large epochs. We demonstrate with various benchmarks while manifesting robustness as much less sensitive to suboptimal hyperparameters. Notably, SimCLR with DCL achieves 68.2% ImageNet-1K top-1 accuracy using batch size 256 within 200 epochs pre-training, outperforming its SimCLR baseline by 6.4%. Further, DCL can be combined with the SOTA contrastive learning method, NNCLR, to achieve 72.3% ImageNet-1K top-1 accuracy with 512 batch size in 400 epochs, which represents a new SOTA in contrastive learning. We believe DCL provides a valuable baseline for future contrastive SSL studies.

1 Introduction

Contrastive self-supervised learning improves representation learning by bringing augmented views of the same instance together, but existing methods can require costly training settings. The paper identifies negative-positive coupling in InfoNCE and proposes DCL to improve efficiency and robustness.

  • Contrastive self-supervised methods pull different views of the same instance together and push different instances apart.
  • Existing contrastive methods may require momentum encoders, large memory queues, large batches, and many training epochs.
  • InfoNCE contains an NPC multiplier q_B that couples positive and negative contributions, reducing learning efficiency when positives or negatives are easy, especially with small batches.
  • Larger SimCLR batches shift q_B toward 1 and reduce its variation, motivating a fixed q_B to lessen batch-size dependence.
  • DCL removes the coupling term from InfoNCE, improving training efficiency with less sensitivity to suboptimal hyperparameters.
  • DCL achieves competitive performance without large batches, long training, momentum encoding, or additional techniques, and can further improve SOTA contrastive methods.

2 Related Work

Related work establishes contrastive learning as a framework based on positive and negative pairs, while prior approaches address representation quality and batch-size sensitivity in different ways. DCL contributes a decoupling perspective that removes the positive term from InfoNCE's denominator.

  • Contrastive Learning: Contrastive learning constructs positive and negative sample pairs to learn representations from the data itself.
  • Contrastive Learning: The SimCLR gradient includes a negative force formed as a weighted average of negative samples, whose effect is coupled to the positive force.
  • Contrastive Learning: SimCLR encodes augmented views with a shared encoder and uses the remaining batch views as classification targets, with one positive label.
  • Contrastive Learning: MoCo maintains a dictionary of negative samples, while SimCLR and related methods use large numbers of negatives to avoid collapsing outputs.
  • Batch Size Sensitivity on InfoNCE: The paper argues theoretically that an intertwined positive-negative multiplier q_B is a main issue in contrastive learning.
  • Batch Size Sensitivity on InfoNCE: DCL offers a novel InfoNCE perspective by decoupling positive and negative terms and making the objective invariant to batch-size sensitivity.

3 Decouple Negative and Positive Samples in Contrastive Learning

The paper analyzes SimCLR's gradient and identifies a batch-size-dependent coupling between positive and negative samples. DCL removes the positive pair from the denominator, while weighted variants emphasize harder positive pairs.

  • SimCLR Formulation: SimCLR encodes two augmented views of each sample with a shared encoder and normalized MLP projector.
  • SimCLR Formulation: Each view uses the other augmented view as its sole positive target and the remaining batch views as negatives in a cross-entropy loss.
  • Negative-Positive Coupling: The InfoNCE loss combines positive similarity with a sum of negative exponentials, where U_i,k denotes the negative-term sum.
  • Negative-Positive Coupling: The gradient contains a common NPC multiplier q_B that modifies all partial gradients, coupling positive and negative learning signals.
  • Negative-Positive Coupling: When negatives are easy, positive gradients are reduced; when positives are easy, negative gradients are reduced, and smaller batches further reduce learning efficiency.
  • Negative-Positive Coupling: Smaller batches shift q_B toward 0 and increase its fluctuation, whereas larger batches push the distribution toward 1.
  • DCL Loss: DCL removes the positive pair from InfoNCE's denominator, thereby removing the NPC multiplier and decoupling positive and negative terms.

4 Experiments

Experiments show that DCL improves contrastive learning efficiency and robustness across large- and small-scale vision benchmarks, especially with small batches, queues, or few epochs. DCL also benefits from re-weighting, tuning, and stronger augmentation.

  • Batch and queue sensitivity: DCL keeps performance steadier than InfoNCE baselines as ImageNet batch size or queue size decreases.On ImageNet, performance drops were SimCLR: −4.1% vs. −8.3% and MoCo: −0.4% vs. −5.9%.
  • Batch and queue sensitivity: 61.8% to 65.9%: DCL improves SimCLR top-1 accuracy under batch size 256; 54.7% to 60.8%: it improves MoCo under 256 queues.At batch size 1024, DCL reaches 66.1% versus 65.1% for the SimCLR baseline.
  • Small-scale benchmarks: DCL outperforms its baseline on small-scale benchmarks at batch size 32, with gains across CIFAR10, CIFAR100, and STL10.The reported kNN / Linear gains are 4.8% / 5.3% on CIFAR10, 1.7% / 4.4% on CIFAR100, and 7.9% / 9.0% on STL10.
  • Few epochs and convergence: 64.6%: DCL- SimCLR reaches ImageNet-1K top-1 accuracy after 100 epochs, compared with 57.5% for the SimCLR baseline.With 100 epochs, DCL also yields 64.4% for MoCo-v2 versus 63.6% for its baseline, while early convergence is faster on CIFAR10 and STL10.

5 Discussion

DCL provides a robust, efficient alternative to InfoNCE-based contrastive learning across vision and other domains, without relying on large batches, long training, or momentum encoding. Its advantage is smaller at large batch sizes, where InfoNCE converges toward DCL.

  • Comparison with other SOTA SSL Approaches: DCL achieves competitive results with minimal requirements and is less sensitive to suboptimal hyperparameters across multiple SSL benchmarks.It does not require large batch sizes, long training epochs, momentum encoding, negative-sample queues, stop-gradient, or multi-cropping.
  • Comparison with other SOTA SSL Approaches: DCL can be combined with NNCLR to improve performance without large batch sizes or long training epochs.The paper evaluates this combination on ImageNet-1K alongside comparisons with other state-of-the-art SSL approaches.
  • Generalization of DCL to Different Domains: DCL generalizes to speech and language domains, including improved downstream performance when applied to wav2vec 2.0.The paper describes DCL as adaptable to different domains and demonstrates its combination with a transformer-based speech model.
  • DCL Convergence for Large Batch Sizes: At large batch sizes, DCL provides less gain over InfoNCE because the NPC multiplier approaches zero and the two losses converge.The paper identifies this convergence as a consequence of the batch-size behavior analyzed theoretically and illustrated empirically.
  • DCL Convergence for Large Batch Sizes: 69.9% ImageNet-1K top-1 accuracy is reached at batch size 1024, compared with 69.5% at batch size 256 after 400 training epochs.The reported increase is slight as batch size grows from 256 to 1024.

6 Conclusion

The paper identifies negative-positive coupling in InfoNCE and removes it through the DCL objective, yielding efficient and competitive contrastive learning. DCL improves SimCLR with minimal modification and can also be combined with NNCLR for strong ImageNet-1K performance.

  • 6 Conclusion: DCL removes the NPC effect from InfoNCE and requires neither momentum encoding, large batch sizes, nor long training epochs for competitive performance.The paper presents DCL as a simple objective with efficient and reliable improvements across benchmarks.
  • 6 Conclusion: DCL provides efficient, reliable, and nontrivial performance improvement over the SimCLR baseline with minimal modification.
  • 6 Conclusion: 72.3% ImageNet-1K top-1 accuracy is achieved by combining DCL with NNCLR using batch size 512 for 400 epochs.
  • 6 Conclusion: The paper proposes maintaining SSL task complexity as batch size decreases to improve learning efficiency.This is stated as an important lesson from the DCL loss.

A.1 Proof of proposition 1

The proof of Proposition 1 derives an NPC multiplier in the InfoNCE gradient. This multiplier scales both the positive-gradient contribution and the negative-sample contribution.

  • A.1 Proof of proposition 1: The anchor-gradient expression factors the positive-view term by q_B,i^(1)/τ.
  • A.1 Proof of proposition 1: The multiplier is defined by the positive-pair exponential divided by the positive-pair exponential plus the negative-sample exponentials.
  • A.1 Proof of proposition 1: Proposition 1 identifies an NPC multiplier q_B,i^(1) in the gradient of the InfoNCE loss.
  • A.1 Proof of proposition 1: By symmetry, a corresponding NPC multiplier q_B,i^(k) exists for the other view.
  • A.1 Proof of proposition 1: The negative-sample gradient contribution is likewise scaled by q_B,i^(1)/τ.

A.2 Proof of proposition 2

The proof of Proposition 2 constructs DCL by removing the positive pair from the InfoNCE denominator. This removes the NPC coupling term from the resulting loss and gradient.

  • A.2 Proof of proposition 2: Removing the positive pair from the InfoNCE denominator leads to the decoupled contrastive learning loss.
  • A.2 Proof of proposition 2: The DCL objective contains the positive similarity term separately from the log negative-sample sum.
  • A.2 Proof of proposition 2: Removing the positive denominator term makes the coupling term disappear when the gradient derivation is repeated.

A.3 Linear classification on ImageNet-1K

DCL is evaluated against state-of-the-art self-supervised learning methods on ImageNet-1K using linear evaluation. The setup uses a ResNet-50 backbone and compares methods under their reported batch-size and epoch settings.

  • DCL is compared with state-of-the-art SSL approaches on ImageNet-1K using linear classifiers trained on ResNet-50 representations.The comparison lists each method’s original batch size and learning epoch settings.
  • The pre-training setup uses a ResNet-50 backbone and two views of size 224 × 224.
  • DCL reaches competitive performance without relatively huge batch sizes, epochs, momentum encoding, or other pre-training schemes.

A.4 Implementation details

The implementation follows SimCLR-style augmentations, with an asymmetric augmentation variant improving ImageNet-1K performance. The paper also emphasizes DCL’s effectiveness under reduced computational requirements.

  • Implementation details: Default DCL augmentations combine random resized crops, horizontal flips, color jittering, grayscale conversion, and Gaussian blur.The stated crop scale is [0.08, 1.0], and the augmentation probabilities and blur range follow the specified settings.
  • Implementation details: 68.2% ImageNet-1K top-1 performance is achieved with asymmetric augmentations, compared with 67.8% using default augmentations.
  • Implementation details: DCL’s effectiveness does not rely on large batch sizes and learning epochs, momentum encoding, or other training tricks.
  • Implementation details: Linear evaluation commonly trains a classifier with batch size 256 for 100 epochs using SGD, momentum 0.9, and zero weight decay.

A.5 Relation to alignment and uniformity

DCL is related to alignment and uniformity objectives but differs in how negative terms are aggregated. The comparison argues that DCL avoids negative-negative coupling and performs especially well relative to Hypersphere at smaller batch sizes.

  • Relation to alignment and uniformity: DCL and Hypersphere both avoid negative-positive coupling, but differ because they swap the order of expectation and exponential operations.
  • Relation to alignment and uniformity: Uniformity introduces negative-negative coupling: similar negative samples can attenuate the gradient for an anchor.
  • Relation to alignment and uniformity: DCL’s negative component differs from uniformity because it sums negative terms separately rather than coupling negatives across different positive samples.
  • Relation to alignment and uniformity: DCL outperforms Hypersphere in every reported experiment, with the advantage becoming larger under smaller batch sizes.
  • Relation to alignment and uniformity: Replacing InfoNCE with DCL improves wav2vec 2.0 representations on speaker identification and intent classification downstream tasks.
  • Relation to alignment and uniformity: 85.4% CIFAR10 kNN top-1 accuracy is obtained with δ = 3 and σ = 0.5, outperforming the SimCLR InfoNCE baseline by 4%.
Loading 2110.06848v3…