Source-linked AI summary
Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification
Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, Lei Wang
TL;DR
Long-tailed data underrepresents rare classes, and cross-entropy may learn skewed features that produce biased classifiers. The paper introduces a curriculum-trained hybrid network using supervised contrastive feature learning and cross-entropy classifier learning, with SC and prototype-based PSC variants. Across three datasets, the approach outperforms cross-entropy counterparts and establishes state-of-the-art long-tailed classification performance.
Problem
Long-tailed class distributions underrepresent rare categories, while the suitability of typical cross-entropy for feature learning from imbalanced data remains unresolved.
Method
A hybrid network progressively transitions from supervised contrastive feature learning to cross-entropy classifier learning, using SC and memory-efficient PSC strategies.
Results
Experiments on three long-tailed image classification datasets show the hybrid networks outperform cross-entropy counterparts and establish new state-of-the-art performance.
Takeaways & Limitations
Supervised contrastive learning is presented as a better substitute for cross-entropy for feature learning in long-tailed classification.
Takeaways & Limitations
Standard SC has memory consumption linear to positive-size times negative-size, so shrinking negatives under limited memory can compromise feature quality, especially with many classes.
Abstract
from arXiv · showhide
Learning discriminative image representations plays a vital role in long-tailed image classification because it can ease the classifier learning in imbalanced cases. Given the promising performance contrastive learning has shown recently in representation learning, in this work, we explore effective supervised contrastive learning strategies and tailor them to learn better image representations from imbalanced data in order to boost the classification accuracy thereon. Specifically, we propose a novel hybrid network structure being composed of a supervised contrastive loss to learn image representations and a cross-entropy loss to learn classifiers, where the learning is progressively transited from feature learning to the classifier learning to embody the idea that better features make better classifiers. We explore two variants of contrastive loss for feature learning, which vary in the forms but share a common idea of pulling the samples from the same class together in the normalized embedding space and pushing the samples from different classes apart. One of them is the recently proposed supervised contrastive (SC) loss, which is designed on top of the state-of-the-art unsupervised contrastive loss by incorporating positive samples from the same class. The other is a prototypical supervised contrastive (PSC) learning strategy which addresses the intensive memory consumption in standard SC loss and thus shows more promise under limited memory budget. Extensive experiments on three long-tailed classification datasets demonstrate the advantage of the proposed contrastive learning based hybrid networks in long-tailed classification.
1. Introduction
Long-tailed image data makes unbiased classifier learning difficult, while typical cross-entropy feature learning can produce skewed representations. The paper proposes contrastive-learning-based hybrid networks that progressively learn features and classifiers, with SC and memory-efficient PSC variants, and reports superior performance on three datasets.
- Rare tail classes are underrepresented, creating a major challenge for learning unbiased classifiers.
- Typical cross-entropy can learn skewed features that lead to biased classifiers, leaving its suitability for imbalanced feature learning unresolved.
- The hybrid network combines supervised contrastive feature learning with cross-entropy classifier learning and progressively transitions from features to classifiers.
- SC pulls same-class samples together and separates different-class samples, while PSC performs the analogous operation using class prototypes.
- PSC avoids explicitly sampling positives and negatives, addressing SC's memory bottleneck and offering more flexible sampling under limited memory.
- Experiments on three long-tailed datasets show the hybrid networks outperform cross-entropy counterparts and establish new state-of-the-art performance.
2. Related Work
Long-tailed classification addresses severe class imbalance, while prior approaches rebalance data, losses, margins, or representations. Contrastive learning extends representation learning with labeled positives, but existing decoupled approaches remain centered on cross-entropy.
- Rebalancing methods include resampling, loss re-weighting, margin modification, and data augmentation to mitigate tail-class data shortages.
- Decoupled learning: Decoupled methods separate representation learning from classifier learning and commonly favor random sampling for features and class-balanced sampling for classifiers.
- Decoupled learning: These decoupled studies use cross-entropy for both stages, leaving whether it is ideal for feature learning under imbalance unresolved.
- Contrastive learning: Supervised contrastive learning uses label information to form positives and negatives, following a two-stage feature-learning and classifier-learning procedure.
3. Main Approach
The proposed hybrid network combines supervised contrastive feature learning with cross-entropy classifier learning and progressively shifts training toward the classifier branch. It includes standard supervised contrastive and prototype-based variants, with prototypes addressing standard SC's memory bottleneck.
- A Hybrid Framework for Long-tailed Classification: The hybrid framework uses a contrastive branch for representation learning and a cross-entropy branch for classifier learning, sharing a backbone.
- A Hybrid Framework for Long-tailed Classification: A curriculum adjusts branch weightings during training so discriminative feature learning precedes classifier learning.
- A Hybrid Framework for Long-tailed Classification: The shared backbone produces representations, while an MLP projection head maps them for contrastive loss and a linear layer predicts classification logits.
- Supervised Contrastive Loss: Supervised contrastive loss treats same-class samples as positives and contrasts them against samples from other classes.
- Supervised Contrastive Loss: Standard supervised contrastive learning consumes memory proportional to positive-size times negative-size, restricting negative samples under limited GPU memory.
- Prototypical Supervised Contrastive Loss: Prototypical supervised contrastive loss learns one prototype per class, pulling samples toward their class prototype and away from other-class prototypes.
- Prototypical Supervised Contrastive Loss: Multiple-prototype PSC models multimodal within-class distributions, while its detailed evaluation is left for future work.
4. Experiments
Experiments on three long-tailed datasets evaluate the proposed hybrid networks, their implementation choices, and key ablations. The results show advantages over cross-entropy-based alternatives, support PSC under different sampling schemes, and favor curriculum-based joint training.
- Datasets: Experiments use long-tailed CIFAR-10, long-tailed CIFAR-100, and the real-world iNaturalist 2018 dataset.The CIFAR variants are created by reducing training examples per class, while iNaturalist contains 8,142 species with severe sample imbalance.
- Comparison to state-of-the-art methods: The proposed hybrid networks outperform compared methods on almost all long-tailed CIFAR settings.Comparisons include loss re-weighting, margin modification, data augmentation, decoupling, and other imbalanced-classification methods.
- Comparison to state-of-the-art methods: Hybrid-PSC outperforms BBN by 1.8% on iNaturalist and Decoupling by nearly 3%.These comparisons concern closely related decoupled-learning approaches.
- Ablation studies and discussions: Hybrid-PSC performs better than Hybrid-SC when the batch size is too small to provide sufficient negative samples for SC.PSC avoids this issue in the iNaturalist experiments, which involve many classes.
- Ablation studies and discussions: Hybrid-PSC achieves comparable performance with random and class-balanced sampling, indicating reduced sensitivity to oversampling.The comparison is reported on long-tailed CIFAR-100; the authors associate this behavior with alleviating overfitting from class-balanced sampling.
- Ablation studies and discussions: Replacing supervised contrastive feature learning with cross-entropy causes a significant performance drop.The comparison evaluates long-tailed CIFAR-100 and supports supervised contrastive losses for feature learning in the hybrid networks.
- Ablation studies and discussions: Curriculum-based joint training outperforms two-stage or fixed-weight training because it preserves compatibility between features and classifiers.The curriculum lets supervised contrastive losses dominate early training before classifier learning becomes more influential.
5. Conclusion
The paper proposes a supervised-contrastive hybrid network with curriculum training for long-tailed classification, including a prototypical loss designed for limited GPU memory.
- The hybrid network combines supervised contrastive loss for image features with cross-entropy loss for classifier learning.
- Curriculum training smoothly transitions optimization from feature learning to classifier learning, reflecting the goal that better features make better classifiers.
- The prototypical supervised contrastive loss offers an advantage under limited GPU memory budgets.
- Experiments on three long-tailed classification datasets showed that the proposal significantly outperformed existing methods.
- The paper identifies deeper exploration of MPSC as a direction for future work.