PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Calibration for Long-Tailed Recognition

Zhisheng Zhong, Jiequan Cui, Shu Liu, Jiaya Jia

arXiv:2104.00466v1cs.CV

TL;DR

Long-tailed recognition suffers from degraded performance and increased miscalibration, including in two-stage methods. MiSLAS combines mixup, label-aware smoothing, and shifted batch normalization to address class-dependent over-confidence and sampling-induced dataset bias. Across multiple benchmarks, it achieves strong recognition accuracy and confidence calibration, setting a new state-of-the-art.

  • Problem

    Long-tailed datasets cause poorer recognition and more miscalibrated, over-confident models, while two-stage methods additionally face dataset bias between sampling stages.

  • Method

    MiSLAS combines mixup, label-aware smoothing for class-dependent over-confidence, and shifted batch normalization for dataset bias in two-stage training.

  • Results

    MiSLAS achieves strong top-1 recognition accuracy and confidence calibration across various long-tailed benchmarks, setting a new state-of-the-art.

  • Takeaways & Limitations

    Calibration and recognition performance can be improved together in long-tailed recognition through the proposed combination of regularization and batch-normalization shift learning.

Abstract

from arXiv · show

Deep neural networks may perform poorly when training datasets are heavily class-imbalanced. Recently, two-stage methods decouple representation learning and classifier learning to improve performance. But there is still the vital issue of miscalibration. To address it, we design two methods to improve calibration and performance in such scenarios. Motivated by the fact that predicted probability distributions of classes are highly related to the numbers of class instances, we propose label-aware smoothing to deal with different degrees of over-confidence for classes and improve classifier learning. For dataset bias between these two stages due to different samplers, we further propose shifted batch normalization in the decoupling framework. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets, including CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, Places-LT, and iNaturalist 2018. Code will be available at https://github.com/Jia-Research-Lab/MiSLAS.

1. Introduction

Long-tailed datasets degrade recognition performance and produce more miscalibrated, over-confident models, including two-stage methods. MiSLAS addresses class-dependent over-confidence and sampling-induced dataset bias with label-aware smoothing and shifted batch normalization, achieving new state-of-the-art results.

  • Long-tailed class imbalance notably degrades CNN recognition performance compared with artificially balanced datasets.
  • Two-stage methods decouple representation learning from classifier learning through different sampling strategies, creating a dataset bias between stages.Stage 1 uses instance-balanced data, whereas Stage 2 uses class-balanced data, so their data distributions differ.
  • Models trained on long-tailed datasets are more miscalibrated and over-confident than balanced-data models, including cRT and LWS.The comparison uses CIFAR-100-LT with imbalance factor 100 and reliability diagrams with 15 bins.
  • MiSLAS proposes label-aware smoothing to handle different class-specific degrees of over-confidence and shifted batch normalization to address two-stage dataset bias.The model combines mixup, label-aware smoothing, and shift learning on the batch-normalization layer.

2. Related Work

Prior work addresses long-tailed recognition through resampling, reweighting, regularization, and two-stage decoupling. These approaches motivate MiSLAS’s focus on calibration, mixup behavior, and classifier weight patterns under different sampling strategies.

  • Re-sampling and re-weighting: Resampling methods either oversample tail classes, risking overfitting, or undersample head classes, discarding data and potentially reducing generalization.
  • Re-sampling and re-weighting: Reweighting assigns different weights to classes or instances, but vanilla approaches can be difficult to optimize on large-scale data.Effective-number weighting and focal loss represent refinements of class- and instance-level reweighting.
  • Confidence calibration and regularization: Mixup interpolates inputs and labels and has been reported to improve calibration, while label smoothing reduces over-confidence through soft targets.
  • Two-stage methods: Two-stage decoupling trains representations with instance-balanced sampling, then retrains or rescales classifiers using class-balanced sampling.cRT learns classifier weights, whereas LWS learns a weight-scaling vector with fewer parameters.
  • Two-stage methods: The paper evaluates top-1 accuracy and ECE across CE, cRT, and LWS variants while varying mixup across the two training stages.
  • Two-stage methods: Classifier weight norms are examined by class frequency to compare cRT and LWS with or without mixup.Classes are sorted by descending training-sample counts, with true norms and smoothed versions shown.

3. Main Approach

MiSLAS addresses miscalibration and classifier-learning limitations in long-tailed recognition by combining mixup-based representation learning, label-aware smoothing, and shifted batch normalization within two-stage training. The approach adapts smoothing to class instance counts and updates normalization statistics across differently sampled stages.

  • Mixup Strategy: Mixup improves cRT and LWS performance when used in Stage-1, while Stage-2 mixup provides no clear gain or can damage performance.This supports using mixup primarily for representation learning rather than classifier learning.
  • Mixup Strategy: Stage-1 mixup consistently improves top-1 accuracy and lowers ECE for cRT and LWS across different backbones.The paper therefore introduces label-aware smoothing to improve classifier learning and calibration further.
  • Label-aware Smoothing: Label-aware smoothing assigns stronger smoothing to classes with more instances, reducing over-confidence especially for head and medium classes.Its smoothing factor is negatively correlated with class instance number, so head classes receive stronger smoothing than tail classes.
  • Shifted Batch Normalization: Different instance-balanced and class-balanced samplers create biased batch-normalization statistics, so shifted batch normalization updates running mean and variance in Stage-2.The learnable linear transformation parameters α and β remain fixed during this normalization update.

4. Experiments

Experiments evaluate MiSLAS and its components across long-tailed benchmarks, showing improvements in accuracy and calibration. Ablations examine label-aware smoothing, shifted batch normalization, and mixup.

  • Calibration: Mixup and label-aware smoothing improve calibration and recognition performance, with similar trends reported on CIFAR-10-LT, ImageNet-LT, and Places-LT.
  • Label-Aware Smoothing: 3.3% higher accuracy is obtained on CIFAR-10-LT with IF 100 using ϵ1 = 0.3 and ϵK = 0.0 versus conventional cross-entropy.
  • Shifted Batch Normalization: 45.3% accuracy is achieved with shifted batch normalization, 1.1% higher than without BN shifting on CIFAR-100-LT with IF 100.
  • Ablation Study: Each ablated module improves accuracy and relieves over-confidence on CIFAR-100-LT across imbalance factors 100, 50, and 10.
  • Benchmark Results: MiSLAS consistently outperforms previous methods in top-1 accuracy and ECE on CIFAR-10-LT and CIFAR-100-LT across imbalance factors 100, 50, and 10.
  • Benchmark Results: MiSLAS achieves better accuracy and confidence calibration than competing approaches on ImageNet-LT, iNaturalist 2018, and Places-LT.

5. Conclusion

The paper identifies severe over-confidence and miscalibration in long-tailed recognition and proposes calibration- and performance-oriented remedies. MiSLAS combines mixup, label-aware smoothing, and shifted batch normalization, achieving strong benchmark results.

  • The paper finds that long-tailed training produces greater miscalibration and over-confidence than balanced training.
  • MiSLAS combines mixup and label-aware smoothing with shifted batch normalization to address class-dependent over-confidence and dataset bias between resampling stages.
  • Extensive experiments report strong top-1 recognition accuracy and confidence calibration, establishing a new state-of-the-art across multiple benchmarks.

A. Experiment Setup

Experiments use established evaluation protocols across five long-tailed recognition benchmarks. Accuracy is reported overall and by class-frequency groups, with detailed settings provided separately.

  • Evaluation Metrics: Top-1 accuracy is reported overall and for Head-Many, Medium, and Tail-Few class splits.
  • Implementation Settings: Table 6 provides detailed experiment settings, including learning rate, batch size, weight decay, learning-rate schedule, and ΔW learning-rate ratio.

B. Exponential Form of the Related Function f(·)

The exponential related function provides an alternative shape-controlled formulation, while the concave form remains the default because it performs best with only limited differences among variants. Reliability diagrams compare calibration across models and long-tailed datasets.

  • B. Exponential Form of the Related Function f(·): For CIFAR-100-LT with imbalanced factor 100, the concave related function achieved the best accuracy after 10 Stage-2 epochs.The comparison used K = 100, N1 = 500, N100 = 5, ϵ1 = 0.4, and ϵ100 = 0.1.
  • B. Exponential Form of the Related Function f(·): The exponential related function uses hyperparameter p to control its shape, producing a concave function when p < 1 and a convex function otherwise.The function is illustrated alongside the other related-function variants in Fig. 8.
  • B. Exponential Form of the Related Function f(·): The related-function form had limited influence on final performance, with variants differing by 0.3% in accuracy.Because the concave function performed best, it was selected as the default for subsequent experiments.
  • C. Calibration Performance: Figure 9 compares 15-bin reliability diagrams for plain and two-stage methods on CIFAR-10-LT with imbalanced factor 100.The figure includes a plain ResNet-32 trained on original CIFAR-10, a plain long-tailed model, cRT, LWS, and MiSLAS.
  • C. Calibration Performance: The calibration figures collectively use 15 bins to compare model reliability across CIFAR-10-LT, ImageNet-LT, Places-LT, and iNaturalist 2018.The compared models include plain baselines, cRT, LWS, mixup variants, and MiSLAS.
  • C. Calibration Performance: Figure 10 compares 15-bin reliability diagrams for plain, cRT, LWS, and MiSLAS models on ImageNet and ImageNet-LT.The comparison includes a plain ResNet-50 trained on the original ImageNet dataset and long-tailed variants.
  • C. Calibration Performance: Figure 11 compares 15-bin reliability diagrams for cRT, LWS, mixup variants, and MiSLAS on Places-LT using ResNet-152.The plotted models are cRT, LWS, cRT with mixup, LWS with mixup, and MiSLAS.
  • C. Calibration Performance: Figure 12 compares 15-bin reliability diagrams for cRT, LWS, mixup variants, and MiSLAS on iNaturalist 2018 using ResNet-50.The LWS, LWS-with-mixup, and MiSLAS entries are identified as under-confidence variants.

D. More Results on ImageNet-LT, iNaturalist 2018, and Places-LT

The appendix reports additional accuracy evaluations across ImageNet-LT, iNaturalist 2018, and Places-LT using specified backbones and training setups.

  • ImageNet-LT: Table 7 reports comprehensive ImageNet-LT accuracy results for ResNet-50, ResNet-101, and ResNet-152 trained for 180 epochs.The table compares results across three backbone networks.
  • iNaturalist 2018: Table 8 reports comprehensive iNaturalist 2018 accuracy results using ResNet-50 trained for 200 epochs.The table provides detailed accuracy evaluation for this dataset and backbone.
  • Places-LT: Table 9 reports detailed Places-LT accuracy results starting from an ImageNet-pre-trained ResNet-152.The table focuses on this dataset and initialization setup.

E. Proof of Eq. (2), the Optimal Solution of LAS

The proof compares optimal solutions for cross-entropy, re-weighting, and label-aware smoothing, showing that LAS yields finite outputs rather than the infinite solutions of the other two losses.

  • Optimal-solution comparison: The section derives optimal solutions for cross-entropy, re-weighting, and LAS and compares the three methods.The comparison is formulated through a general K-class loss and its Lagrange multiplier conditions.
  • Label-aware smoothing: LAS uses class-dependent target probabilities, assigning 1 − ϵy to the ground-truth class and ϵy/(K−1) to each other class.The smoothing factor is defined through the related function f(Ny).
  • Cross-entropy and re-weighting: For cross-entropy and re-weighting, the optimal predicted distribution assigns probability 1 to the ground-truth class and 0 to every other class.Their corresponding solutions drive the correct-class weight sufficiently large and the others sufficiently small.
  • Cross-entropy and re-weighting: Cross-entropy and re-weighting cannot effectively change the predicted distribution or relieve over-confidence because their optimal weight solutions are infinite.The proof states that both losses make the correct-class weight large enough while shrinking the others.
  • Label-aware smoothing: LAS encourages a finite output that refines predicted distributions for head, medium, and tail classes and remedies over-confidence effectively.This contrasts with the infinite optimal solutions obtained for cross-entropy and re-weighting.

F. More Results about the Effect of mixup on cRT and LWS

Additional experiments examine how mixup changes classifier weight norms and accuracy for cRT and LWS, finding stronger benefits in Stage 1 than in Stage 2.

  • Classifier weight norms: On Places-LT and iNaturalist 2018, mixup produces similar weight-norm patterns for cRT and LWS variants.The figures sort classes by descending training-instance counts and show true and smoothed norms.
  • Classifier weight norms: Mixup increases tail-class weight norms and decreases head-class weight norms, reducing head-class dominance.This pattern is reported for both cRT and LWS on the two datasets.
Loading 2104.00466v1…