Source-linked AI summary
Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, Jiashi Feng
TL;DR
Long-tail object detection lacks effective solutions for extremely skewed data, where few-shot classes produce imbalanced classifiers. The paper introduces balanced group softmax (BAGS), which trains groups of similarly frequent categories separately without extra tail sampling, and reports gains across LVIS detection and segmentation benchmarks.
Problem
Long-tail object detection is under-explored, and existing detectors struggle to model few-shot classes under extremely skewed data distributions.
Method
BAGS groups categories with similar training-instance counts and applies separate group-wise softmax training, with others and background handling to limit false positives.
Results
BAGS improves tail-class performance by 9%–19% and overall mAP by around 3%–6% across LVIS detectors and instance-segmentation frameworks.
Takeaways & Limitations
BAGS provides a detection-specific approach that balances classifier training without extra tail-instance sampling and generalizes across strong backbones and frameworks.
Takeaways & Limitations
Resampling and loss re-weighting remain limited by overfitting, computation overhead, weight-design sensitivity, and poor background handling.
Abstract
from arXiv · showhide
Solving long-tail large vocabulary object detection with deep learning based models is a challenging and demanding task, which is however under-explored.In this work, we provide the first systematic analysis on the underperformance of state-of-the-art models in front of long-tail distribution. We find existing detection methods are unable to model few-shot classes when the dataset is extremely skewed, which can result in classifier imbalance in terms of parameter magnitude. Directly adapting long-tail classification models to detection frameworks can not solve this problem due to the intrinsic difference between detection and classification.In this work, we propose a novel balanced group softmax (BAGS) module for balancing the classifiers within the detection frameworks through group-wise training. It implicitly modulates the training process for the head and tail classes and ensures they are both sufficiently trained, without requiring any extra sampling for the instances from the tail classes.Extensive experiments on the very recent long-tail large vocabulary object recognition benchmark LVIS show that our proposed BAGS significantly improves the performance of detectors with various backbones and frameworks on both object detection and instance segmentation. It beats all state-of-the-art methods transferred from long-tail image classification and establishes new state-of-the-art.Code is available at https://github.com/FishYuLi/BalancedGroupSoftmax.
1. Introduction
Long-tail object detection remains difficult because detectors trained on skewed data underrepresent few-shot classes. The paper attributes this failure to imbalanced classifier parameters and introduces BAGS to balance head and tail training without extra tail sampling.
- LVIS exposes a realistic long-tailed object-recognition setting that balanced datasets such as PASCAL VOC and COCO do not represent.
- Directly training detectors designed for balanced datasets on long-tail data causes a substantial performance drop, whose causes are initially entangled.
- Classifier weight norms become severely imbalanced because low-shot categories are activated too infrequently during detection training.
- Resampling and loss re-weighting can address class imbalance but introduce training overhead, overfitting risk, or deployment-sensitive weighting choices.
- BAGS groups categories with similar training-instance counts and applies separate group-wise softmax losses, adding others and background mechanisms to reduce false positives.
- 9%–19% tail-class gains and roughly 3%–6% overall mAP gains are reported across multiple detection and segmentation frameworks on LVIS.
2. Related Works
Prior long-tail approaches largely come from classification or address different detection settings. The paper argues that object detection needs a dedicated solution and presents group-wise training as that approach.
- Long-tail object detection is less explored than balanced-distribution and few-shot detection, with LVIS adding a few-shot tail setting absent from some earlier work.
- Detection frameworks use proposal-level or keypoint-level classification, and mainstream detectors employ softmax classifiers in these components.
- BAGS can replace the original softmax classifier in mainstream detection and corresponding instance-segmentation frameworks.
- Long-tail classification methods include data resampling and cost-sensitive learning through category- or instance-level loss adjustments.
- The paper presents group-wise training as a detection-specific way to address classifier imbalance without extra sampling from tail classes.
3. Preliminary and Analysis
The analysis separates proposal representation from classification and examines how long-tail data affect classifier norms and recognition. It motivates BAGS by showing why tail predictions lose competition against head categories.
- Faster R-CNN generates proposal features from backbone and ROI features, then maps each proposal to C object classes plus background.
- The classification head produces z = Wh+b, where each classifier-weight column corresponds to one category.
- Training applies softmax cross-entropy to each proposal's ground-truth category prediction.
- Current detectors can perform well on head classes yet fail rapidly on tail classes under long-tailed training distributions.
- COCO-to-LVIS Faster R-CNN performance falls from 36.4% to 20.9% mAP, while tail-class AP decreases to 0 rapidly.
- COCO classifier norms remain relatively balanced at 0.75–1.25, whereas LVIS tail-class norms become very small or nearly zero.
- Head-class proposals repeatedly suppress tail classifiers during softmax training, lowering tail prediction scores and selection likelihood.
- Resampling and re-weighting have practical drawbacks, including overfitting, computation overhead, weight-design sensitivity, and difficulty handling background.
4. Balanced Group Softmax
Balanced Group Softmax divides categories with similar training frequencies into separate softmax groups, then calibrates cross-group predictions with an “others” category. Sampling “others” instances keeps group training balanced and improves final detection probabilities.
- Group softmax: BAGS partitions categories by training-instance counts so classes with similar frequencies compete within separate softmax groups.The groups are disjoint, and their boundaries are chosen empirically to keep category instance totals similar.
- Group softmax: The background group G0 uses sigmoid cross entropy, while foreground groups use softmax cross entropy and activate alongside G0 for each labeled proposal.G0 contains only background because background typically has 10–100 times more proposals than object categories.
- Calibration via category “others”: Adding an “others” category to every group calibrates predictions across groups and suppresses false positives when all groups are evaluated at test time.Labels are remapped so “others” is the target in groups that exclude the proposal’s ground-truth category.
- Balancing training samples in groups: BAGS samples “others” proposals for foreground groups in proportion to current-batch category counts, preventing the added class from becoming a dominating outlier.The sampling ratio is β, with β = 8 used normally; groups without activated normal categories are ignored.
- Calibration via category “others”: 2.7% improvement over the baseline results from adding the “others” category.The calibrated probability vector is then passed to post-processing such as NMS, although it is not technically normalized to sum to 1.
5. Experiments
Experiments on LVIS and COCO-LT show that BAGS improves long-tail detection and segmentation across frameworks, backbones, and class-frequency bins. Analyses attribute gains to better classifier balance and carefully designed background, others, grouping, and sampling components.
- Main results on LVIS: On LVIS, the baseline Faster R-CNN achieves 20.98% mAP but 0 AP1, while longer training leaves AP1 around 0.Fine-tuning tail samples also leaves AP1 at 0 and decreases AP4 by 2.5%.
- Main results on LVIS: RFS improves overall mAP by 2.5% while preserving head-class AP, but increases training time by 1.7×.COCO initialization followed by fine-tuning drops mAP by 0.8% due to over-fitting.
- Main results on LVIS: With stronger models, BAGS gains 3.2% on ResNeXt-101-64x4d Faster R-CNN and increases Cascade R-CNN overall mAP by 5.6%.Replacing all three Cascade R-CNN stage classifiers with BAGS yields persistent gains across the three heads.
- Results for instance segmentation: On LVIS instance segmentation, BAGS raises bounding-box and mask mAP, surpassing cited methods by at least 1.8%, 1.4%, and 2.3% across evaluated settings.The models fit tail classes better while head-class AP drops slightly.
- Model analysis: Classifier norms for BAGS groups G1–G3 are nearly level, while G4 is better balanced than in the original model.The analysis compares baseline, RFS, re-weighting, and BAGS classifier weight norms.
- Model analysis: Adding category others and background handling prevents false positives, while four groups outperform two by 0.6 overall mAP.Splitting the smallest tail bin further causes insufficient training, and increasing the number of groups does not help.
- Model analysis: mAP increases with the others-sampling ratio until β = 8; using all others proposals improves head classes but substantially reduces tail performance.Sampling all others across groups creates imbalance within each group.
6. Conclusion
The paper identifies classifier imbalance from insufficient training of low-shot classes as a reason for poor long-tail detection, then proposes BAGS to address it across detection and instance segmentation.
- Classifier weight norms become imbalanced because low-shot classes receive insufficient training on long-tail data.The paper links this imbalance to poor detection performance.
- BAGS addresses classifier imbalance through a balanced group softmax module that improves long-tail detection and instance segmentation across strong backbones.
7.1. Implementation details
The experiments use LVIS and compare several transferred long-tail classification methods under specified training and implementation settings, including reweighting, focal loss, NCM, and τ-normalization.
- Implementation details: Experiments use MMDetection and PyTorch with eight V100 GPUs, SGD, learning-rate warm-up, and scheduled learning-rate decay.The stated batch size is two images per GPU, except one image per GPU for HTC.
- Transferred methods: RFS oversamples images containing tail-category instances, using the best-practice setting t = 0.001.
- Transferred methods: Re-weighting scales category losses according to inverse training frequency, normalizes the weights, caps them between 0.01 and 5, and keeps the background weight at 1.
- Transferred methods: Focal loss is applied directly at proposal level, with variants fine-tuning either the whole backbone and classifier or only the classifier parameters.
- Transferred methods: NCM computes a mean feature for each training class and applies cosine-similarity 1-nearest-neighbor classification to normalized mean features at inference.
- Transferred methods: τ-normalization rescales classifier weights by their L2 norms, and the best searched value is τ = 1 with special handling for background predictions.
7.2. How to train our model
The paper tests alternative BAGS training strategies, including training from random initialization or tuning pretrained components, while noting that β = 1 produces lower results than the main setting.
- 7.2. How to train our model: Adding “others” categories changes the classifier output dimension, so classifier weights and biases must be randomly initialized.
- 7.2. How to train our model: With β = 1, the authors compare training modes and report results below those obtained in the main paper with β = 8.
- 7.2. How to train our model: One training mode freezes feature extraction and trains only the classifier, while another freezes the backbone and trains the full classification head.
7.3. Comparison with winners of LVIS 2019
Because the LVIS test evaluation server is closed, the paper reports validation results and compares them with the LVIS 2019 winner’s reported results; it also describes another winner’s classification change.
- 7.3. Comparison with winners of LVIS 2019: All reported results come from the LVIS validation set because the test-set evaluation server is closed.
- 7.3. Comparison with winners of LVIS 2019: The paper compares against lvlvisis using its reported results and states that its own results surpass them largely.
- 7.3. Comparison with winners of LVIS 2019: Strangeturtle’s Equalization Loss replaces softmax with sigmoid classification and blocks some backpropagation for tail classes.
7.4. Results on COCO-LT
The authors construct COCO-LT by sampling COCO images and annotations to reproduce a long-tail distribution aligned with LVIS. They use this dataset to test BAGS generalization.
- COCO-LT is constructed by sampling images and annotations from the COCO train 2017 split.
- COCO categories are aligned to LVIS categories by sorted training-instance counts, with each COCO category assigned its corresponding LVIS target count.The construction samples target numbers of instances while using as many instances from each sampled image as possible.
7.5. Main results
On COCO-LT, the authors compare Faster R-CNN and Mask R-CNN with ResNet50-FPN, initializing baselines from LVIS-trained models because the training set is small. BAGS improves mean average precision for both bounding-box and mask tasks, particularly on tail classes.
- Faster R-CNN and Mask R-CNN with ResNet50-FPN are evaluated on COCO-LT, using LVIS-trained models to initialize baselines.The initialization addresses the small number of training images in COCO-LT.
- More than 2% improvements in mAP are obtained for both bounding-box and mask results.
- The improvements are especially large for tail classes.