PaperlayerView exact revision on arXiv

Source-linked AI summary

Parametric Contrastive Learning

Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, Jiaya Jia

arXiv:2107.12028v2cs.CV

TL;DR

Long-tailed recognition is challenging because supervised contrastive loss favors high-frequency classes and worsens imbalance. PaCo introduces learnable class-wise centers to rebalance optimization, and experiments report state-of-the-art long-tailed results while also benefiting balanced datasets.

  • Problem

    Supervised contrastive loss assigns higher importance to high-frequency classes, creating bias and making imbalanced learning more difficult.

  • Method

    PaCo extends supervised contrastive learning with parametric class-wise learnable centers to rebalance optimization.

  • Results

    PaCo creates new state-of-the-art results across CIFAR-LT, ImageNet-LT, Places-LT, and iNaturalist 2018, and benefits balanced ImageNet and CIFAR datasets.

  • Takeaways & Limitations

    PaCo can rebalance contrastive learning for long-tailed recognition and adaptively strengthen same-class attraction to benefit hard-example learning on balanced data.

  • Takeaways & Limitations

    The theoretical analysis of Balanced Softmax as a practical center-learning rebalance remedy remains future work.

Abstract

from arXiv · show

In this paper, we propose Parametric Contrastive Learning (PaCo) to tackle long-tailed recognition. Based on theoretical analysis, we observe supervised contrastive loss tends to bias on high-frequency classes and thus increases the difficulty of imbalanced learning. We introduce a set of parametric class-wise learnable centers to rebalance from an optimization perspective. Further, we analyze our PaCo loss under a balanced setting. Our analysis demonstrates that PaCo can adaptively enhance the intensity of pushing samples of the same class close as more samples are pulled together with their corresponding centers and benefit hard example learning. Experiments on long-tailed CIFAR, ImageNet, Places, and iNaturalist 2018 manifest the new state-of-the-art for long-tailed recognition. On full ImageNet, models trained with PaCo loss surpass supervised contrastive learning across various ResNet backbones, e.g., our ResNet-200 achieves 81.8% top-1 accuracy. Our code is available at https://github.com/dvlab-research/Parametric-Contrastive-Learning.

1. Introduction

Long-tailed recognition is difficult because high-frequency classes overwhelm low-frequency classes, and supervised contrastive learning further biases optimization toward frequent classes. PaCo addresses this with learnable class-wise centers and achieves strong results across imbalanced and balanced benchmarks.

  • Long-tailed distributions contain few classes with many instances and many classes with few instances, allowing frequent classes to overwhelm rare ones.
  • Supervised contrastive learning assigns greater importance to high-frequency classes, increasing optimization difficulty for imbalanced datasets.
  • PaCo theoretically balances optimal same-class positive-pair probabilities and organizes loss lower bounds so low-frequency classes receive more attention.
  • With more samples clustered around class centers, PaCo intensifies same-class attraction and benefits hard-example learning under balanced data.
  • PaCo extends supervised contrastive loss with parametric class-wise learnable centers to rebalance contrastive optimization.
  • PaCo establishes new records across long-tailed CIFAR, ImageNet, Places, and iNaturalist 2018, while also outperforming supervised contrastive learning on balanced datasets.

2. Related Work

Prior long-tailed recognition methods rebalance data or training stages, while contrastive learning learns similarity through positive and negative pairs. These approaches address representation or classifier learning in different ways, with practical optimization and end-to-end limitations.

  • Re-sampling balances class exposure by over-sampling rare classes or under-sampling frequent classes, but can overfit or discard useful data.
  • Re-weighting adjusts loss contributions for sparse or difficult classes, but can make optimization difficult on large-scale data.
  • Two-stage methods separate representation learning from classifier learning by first using uniform sampling and then class-balanced fine-tuning.
  • The two-stage design is not end-to-end, and prior analysis attributes its issue partly to bad momentum causal effects.
  • Non-parametric contrastive learning organizes data into similar and dissimilar pairs, with InfoNCE comparing a query against positive and negative key samples.
  • Unlike cross-entropy with classifier weights, InfoNCE does not use parametric learnable parameters and is therefore termed non-parametric contrastive loss.

3. Parametric Contrastive Learning

Supervised contrastive learning is biased toward frequent classes in long-tailed data. PaCo adds learnable class centers to rebalance optimization and adaptively coordinate contrastive and center-based learning.

  • 3.1. Supervised Contrastive Learning: Supervised contrastive loss concentrates more on high-frequency classes, making it poorly suited to imbalanced learning.The analysis attributes this to higher loss lower bounds and greater training importance for frequent classes.
  • 3.2. Theoretical Motivation: PaCo introduces parametric class-wise learnable centers into supervised contrastive learning to rebalance class optimization.The centers are incorporated into the original supervised contrastive formulation as the PaCo algorithm.
  • 3.4. PaCo under Balanced Setting: On balanced data, PaCo becomes an adaptive combination of supervised cross-entropy and supervised contrastive losses rather than a fixed-weight multitask objective.Unlike fixed-weight multitask learning, PaCo adjusts the relative intensities of the two losses during training.
  • 3.4.1 Analysis of PaCo under Balanced Setting: For full ImageNet, the PaCo example uses α=0.05 and reaches its smallest extra-loss value at P_sup=0.71.The example sets q*=0.001, queue length=8192, and αK*=0.41.
  • 3.3. Rebalance in Contrastive Learning: As more samples cluster around class centers, PaCo increases the intensity of pulling same-class samples together and benefits hard-example learning.The mechanism is linked to stronger supervised contrastive intensity as P_sup increases.

4. Experiments

Experiments evaluate PaCo across augmentation settings, long-tailed benchmarks, and balanced datasets. PaCo consistently improves recognition, with augmentation choices and dataset domain affecting performance.

  • Ablation Study: 55.0% versus 52.2% shows severe ImageNet-LT degradation when the crop hyper-parameter changes from (0.08,1) to (0.2,1).The paper attributes this sensitivity to PaCo's center learning.
  • Ablation Study: Strategy (3), using RandAugment for the encoder and SimAugment for the momentum encoder, gives PaCo its best augmentation performance.The comparison includes SimAugment for both inputs and RandAugment for both inputs.
  • Long-tailed Recognition: PaCo surpasses Balanced Softmax by 2.6% on Places-LT under fair training without RandAugment.PaCo uses the same network architecture as the compared pretrained models, whereas RIDE's architecture change prevents the same pretrained-model setup.
  • Long-tailed Recognition: PaCo exceeds Balanced Softmax by 1.4% on iNaturalist 2018 and reaches 75.3% top-1 accuracy with ResNet-152.The reported iNaturalist result transfers ImageNet-LT hyper-parameters without tuning.
  • Long-tailed Recognition: On CIFAR-100-LT, PaCo surpasses Balanced Softmax by 1.2%, 1.8%, and 1.2% at imbalance factors 100, 50, and 10.These comparisons use the same strong training setting with Cutout and AutoAugment.
  • Ablation Study: RandAugment raises PaCo's ImageNet-LT ResNet-50 performance to 79.3% under the tested augmentation strategy.Using SimAugment achieves 78.7% top-1 accuracy, while stronger augmentation further improves performance.
  • Full ImageNet and CIFAR Recognition: On balanced data, PaCo reaches 79.3% with RandAugment on full ImageNet and outperforms supervised contrastive learning by 2.6% on CIFAR-100.The full ImageNet ResNet-50 SimAugment result is 78.7%, 0.8% above supervised contrastive learning.

5. Conclusion

The paper concludes that PaCo uses learnable class-wise centers to address long-tailed recognition and adaptively strengthens same-class attraction on balanced data. Experiments report new state-of-the-art results on long-tailed benchmarks and gains on balanced datasets.

  • Conclusion: PaCo introduces parametric class-wise learnable centers to tackle long-tailed recognition.The method is motivated by theoretical analysis of supervised contrastive learning.
  • Conclusion: On balanced data, PaCo adaptively increases the intensity of pulling same-class samples together as more samples cluster around their centers.The analysis indicates this can benefit hard-example learning.
  • Conclusion: PaCo creates a new state-of-the-art for long-tailed recognition across CIFAR-LT, ImageNet-LT, Places-LT, and iNaturalist 2018.The conclusion reports additional benefits on full ImageNet and CIFAR.

A. Proof to Remark 1

The proof analyzes expected positive-pair counts and the supervised contrastive objective under i.i.d. sampling. It uses a constrained optimization formulation to derive the optimal positive-pair probabilities and loss value.

  • Proof to Remark 1: For sample X_i, the proof begins by analyzing the expected number of positive pairs under i.i.d. training samples.The positive representations are denoted z⁺_1, z⁺_2, ..., z⁺_K_yi.
  • Proof to Remark 1: The approximation K_yi ≈ length(queue) · q(y_i) uses batchsize ≪ length(queue), while the analysis also holds for the precise K_yi.Here q(y_i) denotes the class frequency over the whole dataset.
  • Proof to Remark 1: The supervised contrastive loss is rewritten in terms of positive-pair probabilities p⁺_1, ..., p⁺_K_yi.The derivation then introduces a Lagrange multiplier to enforce the relevant constraint.
  • Proof to Remark 1: The first-order conditions with respect to the multiplier and positive-pair variables yield the optimal solution and minimal loss value.The proof identifies the resulting probability as the probability that two samples of the same class form a true positive pair.

B. Proof to Remark 2

The proof applies the same positive-pair analysis to the parametric contrastive loss. It derives an optimization involving sample-positive terms and the class center, then obtains the corresponding optimal probability expressions.

  • Proof to Remark 2: For PaCo, the proof retains the expected positive-pair count relation used for the image X_i with label y_i.The parametric loss is then rewritten analogously to the supervised contrastive loss.
  • Proof to Remark 2: A Lagrange-multiplier formulation and first-order conditions are used to derive the optimal solution for the parametric loss.The derivation proceeds through the constrained objective and its stationarity conditions.
  • Proof to Remark 2: The parametric contrastive loss includes positive-pair terms together with the sample's relation to its corresponding class center.The proof states that the sample is closest to its corresponding center c_yi among C.
  • Proof to Remark 2: The derived expression gives the true-positive-pair probability as 1/(1 + αK_yi) for two samples of the same class.The proof also reports an α-scaled expression for the optimal positive-pair term.

C. Gradient Derivation

As P_sup increases from 0 to 0.71, PaCo increasingly pulls samples toward their corresponding class centers. Gradient analysis shows this raises the probability of nearest-center assignment to the correct center while reducing assignment to other centers.

  • C. Gradient Derivation: As P_sup increases from 0 to 0.71, the intensity of the supervised contrastive loss increases.The analysis uses full ImageNet as a balanced-setting example.
  • C. Gradient Derivation: Increasing P_sup raises the probability that a sample is closest to its corresponding center and lowers the probability that it is closest to other centers.Consequently, more samples are pulled together with their correct centers during training.

D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.

The supplied experimental passages document comprehensive evaluations across ImageNet-LT and iNaturalist 2018, including different backbones, RIDE settings, and training protocols. They also specify a comparison against supervised-contrastive re-weighting and the training schedule used for several models.

  • D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.: ImageNet-LT results cover ResNet-50, ResNeXt-50, and ResNeXt101 backbones under 400-epoch RandAugment training.Inference time is measured with batches of 64 images on an Nvidia GeForce 2080Ti using PyTorch 1.5 and Python 3.6.
  • D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.: ImageNet-LT evaluations with RIDE use 400-epoch RandAugment training and report inference time for batches of 64 images.The table metadata specifies Nvidia GeForce 2080Ti, PyTorch 1.5, and Python 3.6.
  • D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.: iNaturalist 2018 evaluations include ResNet-50 and ResNet-152, with a marker identifying models trained without RandAugment.Inference-time measurements use batches of 64 images and the stated GPU and software configuration.
  • D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.: iNaturalist 2018 RIDE models are trained for 400 epochs with RandAugment and without knowledge distillation.Inference time is again calculated with batches of 64 images under the stated hardware and software setup.
  • D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.: The supervised-contrastive re-weighting comparison on ImageNet-LT with ResNet-50 uses models trained without RandAugment.The re-weighting strategy is applied directly to the supervised contrastive loss.
  • D. More Experimental Results on Many-shot, Medium-shot, and Few-shot.: Cross-entropy and parametric contrastive-loss models are trained for 400 epochs, while supervised contrastive learning uses separate 400-epoch representation and classifier stages.For supervised contrastive learning, the backbone is fixed before linear-classifier training.

F. Ablation Study

The ablation study compares re-weighting strategies applied to contrastive learning and center learning. PaCo outperforms the contrastive-learning re-weighting baselines, while center-learning rebalancing raises performance to 58.2%.

  • F. Ablation Study: PaCo significantly surpasses re-weighting and Balanced Softmax baselines when rebalancing is applied to supervised contrastive learning.The comparison is summarized in Table 13.
  • F. Ablation Study: Center learning also requires balancing, so the study incorporates Balanced Softmax into center learning for comparison with long-tailed-recognition methods.The passage situates center-learning rebalancing alongside prior approaches.
  • F. Ablation Study: 58.2%: after center-learning rebalancing, PaCo surpasses baselines by a large margin.This result is reported for ImageNet-LT with ResNeXt-50 in Table 14.
Loading 2107.12028v2…