PaperlayerView exact revision on arXiv

Source-linked AI summary

Probabilistic Contrastive Learning for Long-Tailed Visual Recognition

Chaoqun Du, Yulin Wang, Shiji Song, Gao Huang

arXiv:2403.06726v2cs.LGcs.CV

TL;DR

Long-tailed data makes minority categories difficult to learn, while supervised contrastive learning needs large batches that may not contain enough tail examples. ProCo models normalized features with a vMF mixture and derives a closed-form expected contrastive loss. The paper reports consistent improvements across long-tailed recognition tasks, as well as gains on balanced datasets.

  • Problem

    Supervised contrastive learning requires sufficiently large batches to form class-covering pairs, a difficult requirement when tail classes are infrequent in imbalanced data.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    ProCo models normalized features with a mixture of vMF distributions, estimates parameters across batches, and derives a closed-form expected contrastive loss.

    Source: §1 · p. 2

  • Results

    ProCo consistently improves generalization over existing competitive methods across supervised and semi-supervised classification and object detection, and also enhances performance on balanced datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The closed-form surrogate avoids explicitly sampling numerous contrastive pairs and can be optimized without additional inference overhead.

    Source: §1 · p. 2

    The method exhibits numerical instability when κ is not sufficiently large.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Long-tailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Such imbalance issue considerably impairs the performance of standard supervised learning algorithms, which are mainly designed for balanced training sets. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. However, the performance of supervised contrastive learning is plagued by an inherent challenge: it necessitates sufficiently large batches of training data to construct contrastive pairs that cover all categories, yet this requirement is difficult to meet in the context of class-imbalanced data. To overcome this obstacle, we propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space, and samples contrastive pairs accordingly. In fact, estimating the distributions of all classes using features in a small batch, particularly for imbalanced data, is not feasible. Our key idea is to introduce a reasonable and simple assumption that the normalized features in contrastive learning follow a mixture of von Mises-Fisher (vMF) distributions on unit space, which brings two-fold benefits. First, the distribution parameters can be estimated using only the first sample moment, which can be efficiently computed in an online manner across different batches. Second, based on the estimated distribution, the vMF distribution allows us to sample an infinite number of contrastive pairs and derive a closed form of the expected contrastive loss for efficient optimization. Our code is available at https://github.com/LeapLabTHU/ProCo.

1 INTRODUCTION

Long-tailed data limits minority-class learning, while supervised contrastive learning depends on large batches that may rarely include tail classes. ProCo models feature distributions across batches and optimizes a closed-form expected contrastive loss; the paper reports consistent gains across recognition tasks.

1 INTRODUCTION

  • Long-tailed data can hinder generalization to infrequent classes, and SCL's large-batch requirement is especially difficult when tail classes rarely appear in a batch.The resulting gradients can be dominated by head classes, leaving tail classes underrepresented.Source: §1 · p. 1
  • ProCo models normalized contrastive features with a vMF distribution, estimating its parameters across batches to sample contrastive pairs without relying on large batches.The distribution parameters can be estimated using the first sample moment, computed across batches during training.Source: §1 · p. 2
  • ProCo extends to imbalanced semi-supervised learning by generating pseudo-labels from the ProCo loss for unlabeled samples.Those pseudo-labels can subsequently be used to estimate the feature distribution inversely.Source: §1 · p. 2
  • ProCo consistently improves the generalization performance of existing competitive methods across supervised and semi-supervised classification and object detection.Evaluations cover CIFAR-10/100-LT, ImageNet-LT, iNaturalist 2018, and LVIS v1.Source: §1 · p. 2
  • ProCo also achieves enhanced performance on balanced datasets.The paper reports this because ProCo is theoretically independent of imbalanced class distributions.Source: §1 · p. 2
  • The closed-form expected supervised contrastive loss avoids explicitly sampling numerous pairs and is optimized as a surrogate loss.The derivation considers the sampling number approaching infinity; the surrogate adds no inference overhead.Source: §1 · p. 2

2 RELATED WORK

Related work addresses long-tailed recognition through rebalancing, classifier adjustments, augmentation, contrastive learning, distillation, and imbalanced semi-supervised learning. The paper positions its vMF approach against these strands and prior uses of contrastive learning.

Long-tailed Recognition

  • Long-tailed recognition methods include re-sampling, re-weighting, classifier modification, post-hoc weight normalisation, margin adjustment, and minority-class augmentation.The section contrasts these approaches with ProCo's unit-sphere vMF modeling and closed-form expected contrastive loss.Source: §2 · p. 3

Contrastive Learning for Long-tailed Recognition

  • Prior contrastive methods balance pairs or introduce class complements so all classes appear in training iterations, while long-tailed contrastive learning can still over-focus on head classes.KCL uses equal numbers of positive pairs per class; other approaches use class complements,,,,,.Source: §2 · p. 3
  • Multimodal approaches bring visual-linguistic representations to long-tailed recognition, including class-level pretraining and a language-guided recognition head in VL-LTR.The section connects this direction to contrastive foundation models such as CLIP.Source: §2 · p. 3

Knowledge Distillation for Long-tailed Recognition

  • Long-tailed knowledge distillation spans expert ensembles and distillation, class-balanced teachers, nested expert learning, and unbiased test-agnostic modeling.Examples include LFME, RIDE, DiVE, NCL, and xERM.Source: §2 · p. 3

Imbalanced Semi-supervised Learning

  • Imbalanced semi-supervised learning generates pseudo-labels from scarce labels and addresses head-biased predictions with refinement, iterative retraining, or blended linear and semantic labels.The section describes DARP, CReST, and DASO as examples.Source: §2 · pp. 3–4

3 METHOD

ProCo replaces the need for large batches of observed contrastive pairs with a feature-distribution model and a closed-form expected loss. The method also extends to semi-supervised learning, provides theoretical error bounds, and is designed to work with existing long-tailed recognition methods.

3.1 Preliminaries

  • Supervised contrastive learning needs adequate same-class and different-class pairs, but long-tailed batches often undersample tail classes and require larger, more costly batches.The section defines SCL through positive pairs from the same class and negative pairs from different classes; its two loss formulations are not equivalent, and SCL adopts the upper-bound form.Source: §3.2 · p. 4, §3.1 · p. 4

3.2 Probabilistic Contrastive Learning

  • ProCo models normalized class features with a mixture of vMF distributions, estimates distribution parameters across batches, and derives a closed-form expected supervised contrastive loss.This avoids explicitly sampling numerous contrastive pairs and optimizes a surrogate loss.Source: §3.2 · pp. 4–5
  • The method combines logit-adjustment classification loss with ProCo loss in a two-branch model that shares a backbone and adds no inference overhead.The representation branch uses a projection head, while the classification branch uses a linear classifier.Source: §3.2 · p. 6

3.3 Theoretical Error Analysis

  • Under its stated assumptions, the generalization error bound depends primarily on sample size and feature-distribution variance, while excess risk is governed mainly by first-order parameter-estimation error.The generalization analysis considers binary classification and states that its temperature and feature-dimension assumption is reasonable in the experimental setting.Source: §3.3 · pp. 6–7

3.4 ProCo for Semi-supervised Learning

  • For semi-supervised learning, ProCo computes class posterior probabilities from weakly augmented features to generate pseudo-labels for strongly augmented samples.The implementation uses FixMatch to produce the two views and pseudo-labels.Source: §3.4 · p. 7

3.5 Connection with Related Work

  • ProCo uses feature-estimated, non-learnable class complements and derives its loss from contrastive loss under a feature-distribution assumption, with margin modification.The paper distinguishes this design from related methods using learnable class representations and reports comparison in Section 4.4.Source: §3.5 · pp. 7–8

4 EXPERIMENT

The experiments evaluate ProCo across long-tailed and balanced classification, semi-supervised learning, and object detection, alongside component and implementation analyses. Results report improvements over competitive methods across several settings.

4.1 Dataset and Evaluation Protocol

  • Experiments assess supervised and semi-supervised learning, long-tailed classification on CIFAR-10/100-LT, ImageNet-LT, and iNaturalist 2018, and applicability to balanced classification and long-tailed object detection.Long-tailed classification reports top-1 accuracy on balanced validation sets; instance segmentation uses mask AP aggregated across IoU thresholds 0.5–0.95, while detection-box AP is APb.Source: §4 · p. 8, §4.1 · p. 8

4.2 Implementation Details

  • The implementation uses a two-branch setup for CIFAR-10/100-LT, while ImageNet-LT and iNaturalist use ResNet-50 backbones and equal branch loss weights.For CIFAR, the reported standard setting uses 200 epochs and batch size 256; large-scale classification uses 90 epochs and batch size 256.Source: §4.2 · pp. 8–9

4.3 Components Analysis

  • BCL and SupCon performance is significantly limited by batch size, while ProCo mitigates SupCon’s limitation; estimating each class’s feature distribution beats prototype-only estimation, and vMF models feature distributions better than normal.On CIFAR-100-LT, Lin outperforms Lout; α = 1.0 yields optimal performance with equal branch loss weights. On CIFAR100-LT at imbalance factor 100, strong augmentation yields greater performance enhancement.Source: §4.3 · pp. 9–10

4.4 Long-Tailed Supervised Image Classification

  • On CIFAR-100-LT, ProCo improves over BCL by 1.2% and 2.0% for tail classes after 200 and 400 epochs, respectively, while improving head categories too.On ImageNet-LT, it surpasses BCL by 1.3% with ResNet-50 and ResNeXt-50 after 90 epochs; on iNaturalist 2018, it outperforms BCL by 1.7% under the same setting.Source: §4.4 · pp. 10–11
  • ProCo outperforms BCL on CIFAR-10/100-LT across imbalance levels, and combining its representation and classification branches further improves performance.On CIFAR-100-LT, the reported tail-class gains over BCL are 1.2% after 200 epochs and 2.0% after 400 epochs; head categories also improve.Source: §4.4 · p. 10

4.5 Balanced Supervised Image Classification

  • ProCo also demonstrates effectiveness on balanced CIFAR-10/100 and CUB-200-2011 datasets.The authors relate its utility to addressing both imbalance in P(y) and intra-class distribution variance in P(z|y).Source: §4.5 · p. 11

4.6 Long-Tailed Semi-Supervised Image Classification

  • In long-tailed semi-supervised classification, ProCo outperforms DASO across imbalance levels and dataset sizes under the same training conditions.With γ = 20 and N1 = 50, the reported improvement with LA is up to 2.8% over DASO.Source: §4.6 · p. 11

4.7 Long-Tailed Object Detection

  • On LVIS v1, integrating ProCo loss into Faster R-CNN and Mask R-CNN improves performance across categories.The experiments use the original baseline training settings.Source: §4.7 · pp. 11–12

5 CONCLUSION

ProCo models normalized features with von Mises–Fisher distributions, enabling efficient distribution estimation and a closed-form expected supervised contrastive loss. It also extends to semi-supervised learning, and the paper reports theoretical and empirical support for the approach.

5 CONCLUSION

  • ProCo models normalized feature distributions with von Mises–Fisher distributions, estimating their parameters across batches by maximum likelihood.The paper says parameter estimation is efficient across batches.Source: §5 · p. 12
  • A closed-form expected supervised contrastive loss lets ProCo optimize using infinitely many samples drawn from the estimated distributions.Source: §5 · p. 12
  • ProCo extends to semi-supervised learning by generating pseudo-labels for unlabeled data and using them to estimate distributions inversely.Source: §5 · p. 12
  • The paper reports a proven error bound and effectiveness on multiple image-classification and object-detection datasets.Source: §5 · p. 12

APPENDIX A PROOF OF PROPOSITION 2

The proof develops an asymptotic expansion of the ProCo loss and uses variance comparisons and concentration arguments to establish a generalization bound.

APPENDIX A PROOF OF PROPOSITION 2

  • The ProCo loss has an asymptotic expansion under Assumption 1, which the proof uses in its argument.Source: Appendix A · p. 12
  • The conditional variance of the logistic loss is bounded by that of its linear counterpart, using the stated 1-Lipschitz property.The proof introduces the logistic and linear losses and invokes Bennett’s inequality.Source: Appendix A · p. 13
  • The proof derives a conditional-distribution error bound and extends it across classes with a union bound, yielding probability at least 1 −δ.Source: Appendix A · pp. 13–14

PROOF OF PROPOSITION 3

The excess-risk proof combines an asymptotic expansion of the ProCo loss with its 1-Lipschitz property, then uses convexity and an integral inequality.

PROOF OF PROPOSITION 3

Related papers

Loading 2403.06726v2…