Source-linked AI summary
The Majority Can Help The Minority: Context-rich Minority Oversampling for Long-tailed Classification
Seulki Park, Youngkyu Hong, Byeongho Heo, Sangdoo Yun, Jin Young Choi
TL;DR
Long-tailed training can bias classifiers toward majority classes because minority examples are scarce and repetitive oversampling provides limited context diversity. CMO pastes minority-class foregrounds onto majority-class backgrounds to create diversified minority samples, improving results across long-tailed benchmarks and integrating with existing methods without architectural changes.
Problem
Imbalanced datasets can bias classifiers toward majority classes and reduce generalization on minority classes, while naive oversampling can intensify overfitting through limited context diversity.
Method
CMO crops minority-class images and pastes the resulting foreground patches onto majority-class backgrounds, using a minor-class-weighted sampling strategy.
Results
CMO significantly improves performance across long-tailed benchmarks, including a 7.9%p increase on iNaturalist 2018 from 61.0% to 68.9% with cross-entropy loss.
Takeaways & Limitations
CMO can be combined with existing losses and recognition methods to achieve state-of-the-art performance without architectural changes or complex algorithms.
Abstract
from arXiv · showhide
The problem of class imbalanced data is that the generalization performance of the classifier deteriorates due to the lack of data from minority classes. In this paper, we propose a novel minority over-sampling method to augment diversified minority samples by leveraging the rich context of the majority classes as background images. To diversify the minority samples, our key idea is to paste an image from a minority class onto rich-context images from a majority class, using them as background images. Our method is simple and can be easily combined with the existing long-tailed recognition methods. We empirically prove the effectiveness of the proposed oversampling method through extensive experiments and ablation studies. Without any architectural changes or complex algorithms, our method achieves state-of-the-art performance on various long-tailed classification benchmarks. Our code is made available at https://github.com/naver-ai/cmo.
1. Introduction
Long-tailed data can bias classifiers toward majority classes, while naive minority oversampling repeats context-limited examples and intensifies overfitting. CMO addresses this by combining minority objects with majority backgrounds to diversify minority contexts.
- Imbalanced training data bias models toward majority classes and impair minority-class generalization.
- Repeatedly oversampling minority examples produces low-diversity images with nearly identical contexts, intensifying overfitting.
- CMO crops minority objects at varied sizes and pastes them onto majority images to create diverse minority contexts.
- The method transfers information-rich majority contexts into minority samples and generates diversified data around the decision boundary.
- CMO is designed as a simple, low-cost method that integrates with existing long-tailed recognition algorithms without architectural changes or complex algorithms.
2. Related Work
Related work addresses long-tailed recognition through re-weighting, re-sampling, feature-space augmentation, multi-stage or multi-branch methods, and spatial image mixing. These approaches differ in whether they alter sampling, representations, architectures, or augmentation distributions.
- Re-weighting methods adjust the importance of training samples at the class or instance level.
- Re-sampling methods modify training distributions through undersampling or oversampling, but undersampling can discard valuable majority information.
- Feature-space oversampling augments minority classes using in-class neighbors, pretrained features, or implicit semantic augmentation.
- The paper positions its method as simpler and more cost-efficient than prior feature-space approaches while outperforming them.
- Some methods use additional training stages, branches, experts, or meta-learning to address long-tailed recognition.
- Spatial augmentation methods include Cutout, CutMix, and mixup, which remove, replace, or interpolate image regions.
3. Context-rich Minority Oversampling
CMO creates minority-centric training samples by combining minority foreground patches with majority-biased backgrounds using CutMix. A minor-class-weighted distribution controls foreground sampling, while experiments select its strategy for long-tailed classification.
- 3.1. Algorithm: CMO combines a majority-background image with a minority foreground patch to generate a new sample and label.
- 3.1. Algorithm: CutMix supplies the image-combination mechanism, using a binary mask to paste a foreground patch onto the background image.
- 3.1. Algorithm: The combined label is a soft target formed from the two source labels using the image-combination ratio λ.
- 3.1. Algorithm: CMO samples backgrounds from majority-biased data while sampling foregrounds from a minor-class-weighted distribution.
- 3.2. Minor-class-weighted Distribution Q: The foreground sampling distribution assigns weights inversely related to class frequency, with larger r increasingly favoring minority classes.
- 3.2. Minor-class-weighted Distribution Q: The paper evaluates sampling strategies on long-tailed CIFAR-100 and selects q(1,k) for CMO’s minor-class-weighted distribution.
- 3.1. Algorithm: CMO uses a soft-target label inherited from CutMix, which the paper interprets as penalizing over-confident outputs.
4. Experiments
Experiments across CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 evaluate CMO with diverse baselines, architectures, training settings, and ablations. CMO consistently improves long-tailed recognition, particularly for few-shot classes, while using simple input-space augmentation.
- Experimental settings: CMO is evaluated on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 using overall and many-, medium-, and few-shot accuracy.CIFAR-100-LT and ImageNet-LT are artificially imbalanced, whereas iNaturalist 2018 is naturally long-tailed.
- Main results: CMO consistently boosts baseline performance across benchmarks and achieves state-of-the-art accuracy when combined with existing long-tailed recognition methods.The reported gains span CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018.
- Experimental settings: The experiments compare CMO with minority oversampling, re-weighting, two-stage, multi-branch, and other state-of-the-art long-tailed methods.Baselines include ROS, Remix, FSA, LDAM, IB, Balanced Softmax, LADE, DRW, Decouple, BBN, RIDE, Causal Norm, and MiSLAS.
- Main results: CMO consistently improves all evaluated long-tailed recognition methods, whereas ROS severely degrades performance on CIFAR-100-LT.CMO with basic cross-entropy is comparable to complex methods, and applying it to RIDE further boosts performance, especially at imbalance ratios 50 and 100.
- Main results: 7.9%p: CMO improves cross-entropy accuracy on iNaturalist 2018 from 61.0% to 68.9%.The paper attributes this improvement to CMO’s use of abundant training-data context and also reports gains in few-shot classes.
- Analysis and ablations: CMO produces diverse minority samples with varied contexts, such as sky or sea backgrounds for snow geese instead of only grass.Ablations show CMO outperforms CutMix, while pixel-level augmentations provide little comparable gain because they do not add new context.
5. Conclusion
CMO addresses context-limited minority oversampling by transferring majority-class contexts to minority samples. Extensive benchmark experiments show improved performance and state-of-the-art results, while minority gains can sometimes reduce majority-class performance.
- CMO transfers rich majority-class contexts to minority samples, addressing the context limitation of previous oversampling methods.The method is described as simple and intuitive for augmenting minority samples.
- Extensive experiments on various benchmark datasets show that CMO significantly improves performance and advances state-of-the-art results when added to basic losses.These improvements require no architectural changes according to the conclusion.
- In some cases, improved minority-class performance is accompanied by degraded majority-class performance.Future work is identified as improving all classes without sacrificing many-shot performance.
- CMO may require more computation because generated samples benefit from longer training and deeper architectures.The paper identifies possible environmental degradation as a negative societal impact.
A. Implementation details
The experiments use standard long-tailed benchmarks and conventional deep-learning training configurations. Settings vary by dataset in backbone, augmentation, training duration, hardware, and learning-rate schedule.
- The implementation details cover CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 experiments.CIFAR-100-LT uses ResNet-32, ImageNet-LT uses ResNet-50, and iNaturalist uses multiple ResNet and Wide ResNet backbones.
- CIFAR-100-LT: CIFAR-100-LT training uses SGD with momentum 0.9, weight decay 2 × 10−4, 200 epochs, and five-epoch linear learning-rate warm-up.Simple augmentation includes padding, horizontal flipping, and random cropping to 32 × 32.
- ImageNet-LT: ImageNet-LT uses 224 × 224 random crops, horizontal flips, color jittering, batch size 256, four GTX 1080Ti GPUs, and 100 training epochs.The initial learning rate is 0.1 and decays by 0.1 at epochs 60 and 80.
- iNaturalist 2018: iNaturalist 2018 uses ImageNet-style augmentation, batch size 512, eight Tesla V100 GPUs, and 200 training epochs.Experiments use ResNet-50, ResNet-101, ResNet-152, and Wide ResNet-50; the learning rate decays at epochs 75 and 160.
B.1. Comparison with oversampling methods
CMO consistently improves performance across long-tailed recognition methods on CIFAR-100-LT, including imbalance ratios of 50 and 10, with results reported against baseline comparisons.
- CMO consistently improves performance across long-tailed recognition methods on CIFAR-100-LT at imbalance ratios 50 and 10.The comparison is reported in Table 13.
- Table 13 reports classification accuracy (%) for ResNet-32 on CIFAR-100-LT and marks the best results in bold.The table compares CMO against baselines.
B.2. Results on longer training epochs
Under longer training schedules and stronger augmentation settings, CMO remains effective against PaCo on CIFAR-100-LT and iNaturalist 2018. The combined BS + CMO method surpasses PaCo in most cases and reaches new state-of-the-art performance.
- BS + CMO surpasses PaCo in most cases and achieves new state-of-the-art performance under the evaluated longer-training settings.The results are reported in Tables 14 and 15.
- The evaluation trains for 400 epochs with AutoAugment on CIFAR-100-LT and uses RandAugment on iNaturalist 2018.These experiments follow the same setting as PaCo.
- Table 14 reports CIFAR-100-LT classification accuracy across different imbalance ratios using ResNet-32 and 400-epoch AutoAugment training.The table marks the best results in bold and identifies results from PaCo with an asterisk.
B.3. Impact of α
CMO improves baseline accuracy across all tested α values, with best performance at α = 1.0. Its implementation combines original-distribution background batches with minority-weighted foreground batches using a random mask, while adding only 3.94% training time.
- Impact of α: CMO improves the 38.6% baseline accuracy for every tested α ∈ {0.1, 0.25, 0.5, 1.0, 2.0, 4.0}.The evaluation is conducted on CIFAR-100-LT with an imbalance ratio of 100.
- Impact of α: The best performance is achieved when α = 1.0.
- Computational cost: CE+CMO takes 0.369s per batch versus 0.355s for CE on ImageNet-LT, an increase of 3.94%.The paper characterizes this as a low computational cost because CMO loads only one additional batch from the minority-class-weighted loader.
- Implementation: The authors describe CMO as easy to implement and applicable to different losses, networks, and algorithms.Algorithm 2 presents the method in PyTorch-style pseudocode.
- Algorithm: CMO loads background images from the original distribution and foreground images from a minority-class-weighted distribution, then attaches the foreground through a random binary mask.The mask coordinates and mixing ratio are sampled before the foreground patch replaces part of the background image.
- Algorithm: The mixed image is passed through any backbone network and the loss can combine background and foreground labels before the optimization step.The pseudocode lists ResNet and multi-branch networks as example backbones, and CE, LDAM, balanced softmax, and RIDE loss as examples.