PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised Contrastive Learning

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, Dilip Krishnan

arXiv:2004.11362v5cs.LGcs.CVstat.ML

TL;DR

Cross-entropy remains dominant in supervised deep classification despite known shortcomings, while contrastive losses had mainly advanced self-supervised learning. This paper develops supervised contrastive learning with multiple same-class positives and reports stronger accuracy, corruption robustness, and hyperparameter stability than cross-entropy.

  • Problem

    Alternatives to cross-entropy have rarely outperformed it on large-scale datasets, while supervised contrastive learning lacks a way to use multiple known same-class positives.

  • Method

    SupCon extends contrastive learning to supervision by pulling normalized same-class embeddings together, using multiple positives per anchor, and evaluating two loss formulations.

  • Results

    81.4% top-1 accuracy on ImageNet with ResNet-200 is a 0.8% improvement over cross-entropy, with consistent accuracy gains, greater corruption robustness, and lower hyperparameter sensitivity.

  • Takeaways & Limitations

    Supervised contrastive learning provides a simple, stable alternative to cross-entropy that improves classification accuracy and robustness across the reported evaluations.

  • Takeaways & Limitations

    The study leaves experiments with MixUp and CutMix as future work.

Abstract

from arXiv · show

Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models. Modern batch contrastive approaches subsume or significantly outperform traditional contrastive losses such as triplet, max-margin and the N-pairs loss. In this work, we extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture. We show consistent outperformance over cross-entropy on other datasets and two ResNet variants. The loss shows benefits for robustness to natural corruptions and is more stable to hyperparameter settings such as optimizers and data augmentations. Our loss function is simple to implement, and reference TensorFlow code is released at https://t.ly/supcon.

1 Introduction · 2 Related Work · 3 Method

The paper extends contrastive learning to supervised classification by using labels to pull same-class embeddings together while separating different classes. It introduces SupCon, compares two supervised formulations, and reports strong accuracy, robustness, and training stability.

  • 1 Introduction: The method addresses limitations of cross-entropy, including reduced robustness to noisy labels and potentially poor margins, while proposed alternatives often underperform on ImageNet.The paper also reports consistent accuracy boosts, corruption robustness, and lower sensitivity to hyperparameters.
  • 1 Introduction: SupCon adapts contrastive learning to supervision by using multiple same-class positives per anchor and negatives from different classes.Normalized embeddings from the same class are pulled closer together than embeddings from different classes.
  • 1 Introduction: 81.4% top-1 accuracy on ImageNet with ResNet-200 is a 0.8% improvement over the state-of-the-art cross-entropy loss on that architecture.The gain is also accompanied by increased robustness on ImageNet-C.
  • 2 Related Work: Prior work spans cross-entropy, self-supervised representation learning, pixel-predictive methods, metric learning, and triplet-based losses.Triplet losses use exactly one positive and one negative pair per anchor, unlike contrastive losses that can use more pairs.
  • 2 Related Work: Relative to soft-nearest-neighbor approaches, SupCon normalizes embeddings, uses inner products, increases data augmentation, discards the contrastive head, and employs two-stage training.The cited comparison also notes that earlier experiments were conducted at a much smaller scale.
  • 3.1 Representation Learning Framework: The framework applies two random augmentations, encodes both views into 2048-dimensional normalized embeddings, and uses a projection network discarded at inference.The same encoder processes both augmented samples, while the projection network is used only during training.
  • 3.2 Contrastive Loss Functions: Supervised contrastive learning defines positives as all distinct same-label samples in the multiviewed batch, generalizing self-supervised learning beyond its single positive.Both candidate formulations retain negatives in the denominator and support arbitrary numbers of positives.
  • 3.2.2 Supervised Contrastive Losses: The formulation summing over positives outside the logarithm, Lsup_out, significantly outperforms the inside-log formulation, Lsup_in, so the paper uses Lsup_out thereafter.The analysis attributes this advantage to reduced positive bias and a more favorable gradient structure.

4 Experiments

Experiments show that SupCon outperforms cross-entropy and other contrastive or margin-based methods on image classification, while also improving corruption robustness and hyperparameter stability. Its transfer-learning performance is comparable to cross-entropy and self-supervised contrastive learning when architectures match.

  • Classification accuracy: 78.7% top-1 accuracy on ImageNet with ResNet-50 and AutoAugment establishes a new state-of-the-art result for that setting.SupCon also slightly improves over CutMix, a state-of-the-art augmentation strategy.
  • Classification accuracy: 79.1% top-1 accuracy on ImageNet is achieved with ResNet-50 using memory size 8192, batch size 256, and SGD, exceeding 78.7% with batch size 6144 without memory.The memory-based setup uses 128-dimensional vectors, 8 Nvidia V100 GPUs, and significantly reduces compute and memory footprint.
  • Hyperparameter stability: SupCon has significantly lower output variance than cross-entropy when augmentations, optimizers, and learning rates are varied.The comparison covers RandAugment, AutoAugment, SimAugment, Stacked RandAugment, LARS, SGD with Momentum, and RMSProp.
  • Robustness: SupCon models show increased robustness to ImageNet-C corruptions and lesser accuracy degradation as corruption severity increases.Robustness is evaluated using Mean Corruption Error, relative Mean Corruption Error, and mean accuracy across corruption severities.
  • Transfer learning: SupCon is on par with cross-entropy and self-supervised contrastive learning for transfer learning when trained on the same architecture.The evaluation covers fine-tuning on 12 natural image datasets, with architecture identified as the dominant factor in transfer performance.

Supplementary · 5 Training Setup

The training setup contrasts cross-entropy, self-supervised contrastive, and supervised contrastive learning. SupCon learns representations with a contrastive loss while using labels to sample positives alongside same-image augmentations.

  • 5 Training Setup: The number of parameters in the inference models always stays the same across the compared training setups.
  • 5 Training Setup: A linear classifier is not necessary in the second stage for evaluating learned representations.
  • 5 Training Setup: Previous works have evaluated representations using k-Nearest Neighbor classification or prototype classification.
  • 5 Training Setup: The linear classifier can also be trained jointly.
  • 5 Training Setup: Cross-entropy uses labels and a softmax loss to train a classifier.
  • 5 Training Setup: Self-supervised contrastive learning uses a contrastive loss and data augmentations to learn representations.
  • 5 Training Setup: Supervised contrastive learning uses a contrastive loss to learn representations while incorporating label information.
  • 5 Training Setup: SupCon samples positives from label information in addition to augmentations of the same image.

6 Gradient Derivation

This section derives the gradients of the two supervised contrastive losses with respect to normalized projection representations. The derivations confirm that both gradients have a nearly identical mathematical form and can be written collectively.

  • Gradient derivation: The section derives the gradients of both considered supervised contrastive losses with respect to a normalized projection network representation, z_i.It is sufficient to establish the claim for L_sup^in,i before deriving the corresponding expressions.
  • Gradient derivation: P_i^p captures the likelihood of z_p among positives and negatives, whereas X_i^p considers only positives; P_i^n analogously defines the likelihood of z_n.The passage states P_i^p ≤ X_i^p.
  • Gradient derivation: Both gradients, given in Eqs. 9 and 12, share a very similar form and can be expressed collectively.This collective expression corresponds to Eq. 4 and its subsequent analysis in the paper.

7 Intrinsic Hard Positive and Negative Mining Properties

Contrastive-loss gradients intrinsically mine hard positives and negatives, eliminating the need for complicated hard-mining algorithms. This property holds for self-supervised and supervised contrastive losses only when the projection outputs are normalized, with larger batches increasing hard-example impact.

  • Intrinsic Mining: Contrastive-loss gradients provide intrinsic hard positive/negative mining, removing the need for complicated hard-mining algorithms.The paper establishes this analytically and notes that hard mining is crucial for triplet and max-margin losses.
  • Intrinsic Mining: The intrinsic mining property applies to the self-supervised loss and both supervised contrastive-loss variants.The self-supervised contrastive loss is a special case of the supervised formulations under P(i) = j(i).
  • Normalization: Normalization of projection outputs structures gradients to focus learning on hard positives and negatives.The analysis defines pre-normalization outputs as w_i and normalized representations as z_i = w_i/∥w_i∥.
  • Gradient Contributions: Hard positive and negative examples produce large gradient contributions, whereas weak positives and negatives produce small contributions.Weak positives have z_i • z_p ≈1 and weak negatives have z_i • z_n ≈−1; hard examples have similarities near 0.
  • Batch Size: Larger batches increase the probability of selecting hard positives and negatives, whose larger gradients can improve performance.Increasing the number of negatives increases hard-positive response for either supervised loss, while increasing positives can further increase it for Lsup_out.

8 Triplet Loss Derivation from Contrastive Loss

With one positive and one negative, the contrastive loss reduces to a triplet-loss form with margin α = 2τ. Unlike triplet loss, supervised contrastive learning naturally reinforces hard negatives without expensive mining.

  • Derivation: With one positive and one negative, contrastive loss is a special case of triplet loss when the anchor-positive similarity exceeds anchor-negative similarity.The derivation assumes z_a • z_p ≫ z_a • z_n.
  • Derivation: The contrastive objective becomes log (1 + exp ((z_a • z_n − z_a • z_p) /τ)) and approximates exp ((z_a • z_n − z_a • z_p) /τ).This uses a Taylor expansion of the logarithm under the stated alignment assumption.
  • Derivation: A further approximation yields 1 + 1/τ · (z_a • z_n − z_a • z_p), matching the linear similarity structure of triplet loss.The approximation follows the exponential form derived from the contrastive objective.
  • Implications: The resulting triplet-loss margin is α = 2τ, and empirical results indicate contrastive loss generally outperforms triplet loss on representation tasks.The comparison is attributed to prior empirical results.
  • Implications: Supervised contrastive gradients naturally reinforce hard negatives during training, avoiding triplet loss’s computationally expensive hard-negative mining.This behavior is discussed in Section 7 of the paper.

9 Supervised Contrastive Loss Hierarchy

The SupCon loss subsumes self-supervised contrastive, N-Pairs, and triplet losses under progressively stronger restrictions on positive and negative samples. On ImageNet with batch size 6144, SupCon substantially outperformed N-Pairs in an otherwise identical ResNet50 setup.

  • Loss hierarchy: SupCon subsumes multiple commonly used losses as special cases under additional restrictions on its positive and negative sets.The section frames these restrictions as a hierarchy of loss formulations.
  • Positive sets: SupCon positives include views of the anchor’s source image and views of different same-label images.These are the two disjoint positive categories used to derive related losses.
  • Self-supervised contrastive loss: Self-supervised contrastive loss is recovered when each anchor’s positives contain only another view of the same source image.Under this restriction, P(i) = j(i), where j(i) indexes the paired view.
  • N-Pairs loss: N-Pairs loss requires one positive from a different same-label image and omits temperature in its original formulation.Temperature can nevertheless be generalized into the N-Pairs formulation.
  • Triplet loss: Triplet loss is recovered when the numbers of positives and negatives are each restricted to one.This restriction also yields special cases of self-supervised and N-Pairs losses.

10 Effect of Temperature in Loss Function

Temperature is an important supervised contrastive learning hyperparameter, balancing optimization smoothness against emphasis on hard positives and negatives; the optimal setting can improve performance by nearly 3%. Empirically, temperature 0.1 was optimal for ResNet-50 top-1 accuracy and was used for all ResNet-200 experiments.

  • 10 Effect of Temperature in Loss Function: Temperature plays an important role in supervised contrastive learning, with the optimal value improving performance by nearly 3%.Changing temperature creates competing effects involving optimization smoothness and hard positives/negatives.
  • 10 Effect of Temperature in Loss Function: Larger temperatures reduce gradient magnitudes, simplifying optimization by permitting larger learning rates.The gradient norm scales as ||∇L|| ∝ 1/τ; with one positive and one negative, the loss is equivalent to a triplet loss with margin proportional to τ.
  • 10 Effect of Temperature in Loss Function: Lower temperatures emphasize hard positives and negatives, whose structure can improve supervised contrastive and triplet-loss classification performance.For a fixed batch and anchor, lowering temperature relatively increases the value of P_ik.
  • 10 Effect of Temperature in Loss Function: 0.1 was optimal for top-1 accuracy on ResNet-50, and the same temperature was used for all ResNet-200 experiments.Results across temperatures are reported in Figure 4 of the main paper.

11 Effect of Number of Positives

The ablation tests how capping the number of positives per sample affects the loss, excluding selected positives from the denominator so they are not treated as negatives. With batch size 6144, training otherwise provides 13 positives in expectation, and Table 5 reports a steady benefit from adding more positives.

  • Ablation setup: The ablation caps each sample at at most k positives and removes those positives from the loss denominator, preventing them from being treated as negatives.The model is trained with batch size 6144 for 350 epochs.
  • Ablation setup: 13 positives occur in expectation without capping, comprising six positives with two augmentations each plus other augmentations of the anchor image.
  • Results: Adding more positives produces a steady benefit in the ablation reported in Table 5.

12 Robustness · 13 Two stage training on Cross Entropy · 14 Training Details

The supervised contrastive loss is more robust to corruption severity and augmentation choices than cross-entropy, while training ablations identify effective optimizer and augmentation configurations. Cross-entropy representations are also evaluated through a two-stage linear-classifier protocol.

  • 12 Robustness: Increasing the maximum number of positives improves final Top-1 accuracy, whereas one positive recovers the self-supervised augmented-sample setting.The comparison varies |P(i)| from 1 to uncapped.
  • 12 Robustness: Supervised contrastive models maintain higher accuracy across ImageNet-C corruption severities without the increasing calibration error observed with cross-entropy.Performance and calibration degrade as corruption severity increases, but the contrastive loss avoids increasing calibration error.
  • 13 Two stage training on Cross Entropy: Cross-entropy representations are tested in a two-stage protocol by reinitializing the final layer and training a fixed representation with a linear classifier.The first stage learns representations with cross-entropy; the second trains the reinitialized final layer while keeping earlier weights fixed.
  • 14 Training Details: The training ablations disentangle the effects of optimizer and data augmentation on downstream performance.The study evaluates optimizer and augmentation choices separately.
  • 14.1 Optimizer: LARS performs best for embedding-network training, using cosine learning-rate decay, while RMSProp performs best for linear-classifier training with exponential decay.The optimizer combinations are reported in Table 8.
  • 14.2 Data Augmentation: AutoAugment gives the highest Top-1 accuracy for both losses on ResNet-50, whereas Stacked RandAugment does so for both losses on ResNet-200.The authors conjecture that Stacked RandAugment benefits from the larger model capacity of ResNet-200.
  • 14.2 Data Augmentation: AutoAugment trains faster than RandAugment, SimAugment, and Stacked RandAugment, while MixUp and CutMix remain future work.The runtime observation concerns augmentation schemes used in training.
  • 14.2 Data Augmentation: Supervised contrastive methods consistently outperform cross-entropy training across RandAugment magnitudes on ResNet-50 and ResNet-200.The comparison varies augmentation magnitude rather than holding it at a single setting.

15 Change Log

The change log records successive revisions that expanded the empirical and analytical content, added implementation resources, and corrected presentation and mathematical errors. The main substantive update added supervised-contrastive-loss analyses, experiments, transfer learning, tuning, augmentation, and batch-size and training-duration results.

  • Version 1: Version 1 established the initial arXiv version on 2020-04-23.This was the initial release recorded in the change log.
  • Version 2: Version 2 added analyses of supervised contrastive loss forms and gradients, supporting experiments, transfer learning experiments, and additional tuning for top-1 accuracy.It also moved accuracy-versus-number-of-positives results to the supplementary material and added results for batch size and training epochs.
  • Version 2: Version 2 added StackedRandAugment, a GitHub code link, and results comparing batch size and the number of training epochs.The same revision reported that heavier model tuning deteriorated ECE.
  • Version 3: Version 3 removed a deprecated sentence from the abstract on 2020-10-13.The change log identifies this as an abstract-only revision.
  • Versions 4–6: Versions 4–6 corrected equation references, added first-page author footnotes, and fixed a sign mistake in Jensen’s Inequality.These revisions were recorded on 2020-12-10, 2020-12-27, and 2020-03-10, respectively.
Loading 2004.11362v5…