PaperlayerView exact revision on arXiv

Source-linked AI summary

Distribution Alignment: A Unified Framework for Long-tail Visual Recognition

Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, Jian Sun

arXiv:2103.16370v1cs.CVcs.AIcs.LG

TL;DR

Long-tail visual recognition remains difficult because highly imbalanced class frequencies bias predictions toward head classes. The paper diagnoses a two-stage bottleneck and introduces confidence-aware distribution alignment with adaptive calibration and generalized re-weighting. Across image classification, semantic segmentation, object detection, and instance segmentation, the method achieves state-of-the-art results.

  • Problem

    Long-tail class distributions create severe imbalance between head and tail categories, causing naïve models to favor head classes and degrade tail-class performance.

  • Method

    The method calibrates classifier scores through adaptive, input-dependent magnitude and margin, then applies generalized re-weighting to incorporate a balanced class prior.

  • Results

    The method achieves state-of-the-art performance across image classification, semantic segmentation, object detection, and instance segmentation.

  • Takeaways & Limitations

    A unified two-stage distribution-alignment strategy provides a flexible solution for long-tail recognition across diverse visual tasks.

  • Takeaways & Limitations

    The semantic-segmentation evaluation distinguishes head, medium, and tail classes using pixel-frequency thresholds, including tail classes below 0.1% of total pixels.

Abstract

from arXiv · show

Despite the recent success of deep neural networks, it remains challenging to effectively model the long-tail class distribution in visual recognition tasks. To address this problem, we first investigate the performance bottleneck of the two-stage learning framework via ablative study. Motivated by our discovery, we propose a unified distribution alignment strategy for long-tail visual recognition. Specifically, we develop an adaptive calibration function that enables us to adjust the classification scores for each data point. We then introduce a generalized re-weight method in the two-stage learning to balance the class prior, which provides a flexible and unified solution to diverse scenarios in visual recognition tasks. We validate our method by extensive experiments on four tasks, including image classification, semantic segmentation, object detection, and instance segmentation. Our approach achieves the state-of-the-art results across all four recognition tasks with a simple and unified framework. The code and models will be made publicly available at: https://github.com/Megvii-BaseDetection/DisAlign

1. Introduction

Long-tail recognition must handle severe differences in class frequency, which can bias predictions toward head classes and degrade tail-class performance. The paper identifies biased decision boundaries as a key two-stage bottleneck and proposes unified distribution alignment to improve classifier learning across visual tasks.

  • Long-tail datasets range from one example for some tail classes to hundreds or thousands for head classes.
  • Naively trained models are dominated by head classes, degrading performance on many tail classes.
  • The empirical study finds a substantial gap between balanced-head retraining and the baseline despite sharing the same first-stage representation.This indicates substantial room for improvement in the second-stage classifier because of a biased decision boundary.
  • The proposed strategy calibrates classifier outputs by aligning them with a reference class distribution that favors balanced prediction.It uses class prior and data input to learn decision boundaries without tedious hyper-parameter tuning.
  • Adaptive calibration provides input-dependent magnitude and margin, while generalized re-weighting incorporates a balanced class prior across imbalance scenarios.
  • State-of-the-art performance is reported for image classification, semantic segmentation, object detection, and instance segmentation.

2. Related Works

Prior work addresses long-tail recognition through resampling, loss re-weighting, knowledge transfer, and decoupled representation and classifier learning. These approaches span multiple visual recognition tasks and include both one-stage and two-stage strategies.

  • One-stage Imbalance Learning: One-stage methods rebalance training data by oversampling minority categories, undersampling frequent categories, or using class-aware and repeat-factor sampling.
  • One-stage Imbalance Learning: Other one-stage approaches re-weight losses at the class or sample level and transfer knowledge from head classes to improve tail categories.
  • Two-stage Imbalance Learning: Two-stage methods decouple representation learning from classifier-head learning and then rebalance the classifier.
  • Visual Recognition Tasks: The study considers image classification, object detection, semantic segmentation, and instance segmentation as representative visual recognition tasks.The strategy is described as extendable to other deep-network methods for these tasks.

3. Our Approach

The approach uses two-stage learning for long-tail recognition, diagnosing classifier calibration as the main bottleneck and addressing it through adaptive distribution alignment with generalized re-weighting.

  • 3. Our Approach: The two-stage framework first learns representations and a classifier from unbalanced data, then calibrates classification scores toward a balanced reference distribution.The framework is applied across visual recognition tasks by adapting the input modality, including images, pixels, and bounding boxes.
  • 3.1. Problem Setting and Empirical Study: Ablation shows that imbalanced-data training produces strong representations, while biased classifier decision boundaries leave substantial performance headroom.The analysis retrains the classifier head with balanced data while keeping the first-stage representation fixed.
  • 3.2.1 Adaptive Calibration Function: The adaptive calibration function applies class-specific score transformations and input-dependent confidence to combine original and calibrated classifier outputs.The class-specific parameters are learned from data, while the confidence function controls how much calibration each input receives.
  • 3.3. Connection with Recent Work: The unified strategy offers a simple alternative to resampling for instance-level and pixel-level recognition tasks, where resampling is difficult to apply.The paper reports validation on image classification, semantic segmentation, object detection, and instance segmentation, with consistent state-of-the-art performance across benchmarks.

4. Experiments

Experiments evaluate DisAlign across image classification, semantic segmentation, object detection, and instance segmentation, with ablations probing backbones, components, and re-weighting. The method consistently improves performance across these long-tail recognition settings.

  • Image classification: 52.9% per-class average accuracy on ImageNet-LT with ResNet-50 exceeds state-of-the-art methods by 2.5%.With ResNeXt-50, DisAlign reaches 53.4%, exceeding prior methods by 1.6%.
  • Image classification: 39.3% per-class average accuracy on Places-LT with ResNet-152 yields a 1.4% gain over prior methods.The comparison uses the same setting as the referenced prior work.
  • Semantic segmentation: DisAlign improves semantic segmentation by 2.0 and 2.3 mIoU with FCN-8s using ResNet-50 and ResNet-101, respectively.With DeepLabV3+ and ResNeSt-101, it improves performance by 0.5 and reaches 47.8 mIoU on ADE20k.
  • Detection and instance segmentation: DisAlign achieves 25.6 APbbox and 26.3 APmask with Mask R-CNN+FPN and an ImageNet-pretrained ResNet-50.Using a COCO-pretrained model, the scores increase to 27.6 APbbox and 27.9 APmask, while common and rare categories improve.

5. Conclusion

The conclusion presents DisAlign as a unified two-stage strategy for large-scale long-tail visual recognition. It calibrates biased predictions through confidence-aware distribution alignment and generalized re-weighting, outperforming prior work across several tasks.

  • Conclusion: DisAlign uses a unified two-stage learning strategy for large-scale long-tail visual recognition.The paper frames the approach around a joint learning stage followed by distribution calibration.
  • Conclusion: Confidence-aware distribution alignment calibrates initial classification predictions to address biased label prediction.The method uses a generalized re-weight scheme to leverage category priors during alignment.
  • Conclusion: The method outperforms previous work with a large margin across image classification, semantic segmentation, and object detection and segmentation.The conclusion summarizes results across multiple long-tail visual recognition tasks.

A.2. Training Configuration

Training uses standard SGD-based configurations across long-tail image-classification datasets, followed by a short distribution-alignment stage with fixed backbone and original classifier parameters.

  • Backbones: ImageNet-LT uses ResNet-{50,101,152} and ResNeXt-{50,101,152}, while iNaturalist 2018 uses ResNet-{50,101,152}.Places-LT uses ResNet-152 pretrained on the full ImageNet-2012 dataset.
  • Optimization: Experiments use SGD with momentum 0.9, batch size 256, cosine decay from 0.1 to 0, and 224×224 images.These settings are reported for the image-classification experiments.
  • Joint learning stage: The joint learning stage trains the backbone and original classifier for 90 epochs on ImageNet-LT and 90/200 epochs on iNaturalist-2018.Places-LT training lasts 30 epochs with all layers frozen except the last ResNet block in the first stage.
  • Distribution alignment stage: The second alignment stage fixes the backbone and original classifier head while training for 10 epochs on ImageNet-LT and Places-LT or 30 epochs on iNaturalist-2018.The learning rate is restarted for this stage.
  • Alignment parameters: The generalized re-weight scale is set to ρ = 1.2 for dot-product heads and ρ = 1.5 for cosine-normalized heads.The calibration parameters are initialized as α = 1.0 and β = 0.0.

A.3. Detailed Experimental Results

Detailed experimental results cover ImageNet-LT and two additional large-scale long-tail benchmarks. The reported comparisons extend evaluation beyond the principal ImageNet-LT results.

  • ImageNet-LT: Detailed quantitative results for ImageNet-LT are reported in Table 10.The paper presents ImageNet-LT results as a separate detailed comparison.
  • Additional benchmarks: iNaturalist and Places-LT provide additional large-scale long-tail benchmark evaluations in Tables 13 and 14.These experiments further demonstrate the method beyond ImageNet-LT.

A.4. Ablation Study

The ablations show that both adaptive calibration and generalized re-weighting improve performance. Calibration learns larger magnitude and margin values for tail and body classes, while input-aware calibration is strongest among tested variants.

  • Both adaptive calibration and generalized re-weighting contribute to performance improvement in the component ablation.
  • Learned calibration magnitude and margin values are larger for tail and body classes than for head classes.This pattern boosts tail scores and alleviates bias in the original predictions.
  • Input-aware calibration outperforms input-agnostic calibration and baselines using only magnitude or margin.Examples with low biased-prediction probability on the ground-truth class tend to improve with higher confidence.
  • DisAlign is also applied to semantic segmentation on the ADE-20K benchmark to validate its effectiveness beyond classification.

B.1. Dataset and Evaluation

This section describes ADE-20K as a 150-category scene-parsing dataset and reports mIoU and mAcc, while identifying the table and figure contexts used for analysis and ablation.

  • ADE20K contains 150 stuff/object categories, with 20K, 2K, and 3K images for training, validation, and testing.Its imbalance ratio is 788 based on the maximum-to-minimum category frequency.
  • Tables 10, 11, and 12 cover ImageNet-LT accuracy, component influence, and confidence-score ablations, respectively.
  • Figure 6 analyzes calibration using a ResNeXt-50 model trained on ImageNet-LT.
  • ADE-20K categories are divided into head, body, and tail groups using pixel-frequency ratios.Head classes exceed 1.0%, body classes range from 0.1% to 1%, and tail classes are below 0.1%.
  • Segmentation evaluation uses mean intersection over union (mIoU) and mean pixel accuracy (mAcc), including results for each frequency group.

B.2. Training Configuration

The segmentation experiments use MMSegmentation with several backbones and fixed training schedules, then evaluate FCN and DeepLabV3+ configurations across backbone choices.

  • MMSegmentation experiments train models for 160k iterations with batch size 32 on eight V100 GPUs.Images are cropped to 512 × 512, augmented by random scaling from 0.5 to 2.0 and flipping, and use a poly learning-rate schedule.
  • ResNet-50, ResNet-101, and ResNeSt-101 serve as segmentation backbones.
  • The DisAlign stage trains for 8k iterations and uses ρ = 0.3 across experiments.
  • FCN and DeepLabV3+ are evaluated with ResNet-50, ResNet-101, and ResNeSt-101 backbones.
  • LVIS v0.5 contains 56K training images, 5K validation images, and 1230 categories for object detection and instance segmentation.Categories are grouped as rare, common, and frequent according to training-image counts.
  • LVIS evaluation uses COCO-style average precision, including AP, APr, APc, and APf for rare, common, and frequent subsets.

C.2. Training Configuration

The detection and instance-segmentation configurations use Detectron2 with ResNet-FPN baselines, then freeze network parameters while learning calibration parameters; results are reported for multiple benchmark tables.

  • Baseline detection and instance-segmentation models use Detectron2, ImageNet-pretrained ResNet backbones, and FPN.Training uses eight GPUs, two images per GPU, and 90K iterations with learning-rate drops at 60K and 80K iterations.
  • DisAlign freezes all network parameters and learns magnitude and margin for an additional 9K iterations.Generalized re-weight applies only to foreground categories, with scale ρ set to 0.8.
  • ADE-20K split categories are defined by dataset-wide frequency ratios for head, body, and tail groups.
  • Detailed results are provided in Tables 17 and 18.
Loading 2103.16370v1…