PaperlayerView exact revision on arXiv

Source-linked AI summary

Balanced Meta-Softmax for Long-Tailed Visual Recognition

Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, Hongsheng Li

arXiv:2007.10740v3cs.LGcs.CVstat.ML

TL;DR

Long-tailed training data conflicts with class-balanced testing, and standard Softmax gives a biased estimation under shifted label distributions. The paper introduces Balanced Softmax and a meta-learned sampler, reporting superior performance across image classification and instance segmentation while acknowledging computational and approximation limits.

  • Problem

    Long-tailed training distributions mismatch balanced test distributions, making standard Softmax biased under label-distribution shift.

  • Method

    BALMS combines Balanced Softmax, which accommodates label-distribution shift and minimizes a generalization error bound, with a Meta Sampler that learns re-sampling through meta-learning.

  • Results

    BALMS outperforms state-of-the-art methods across four image-classification datasets and one instance-segmentation dataset, especially at high imbalance.

  • Takeaways & Limitations

    Balanced Softmax addresses distribution shift, while the Meta Sampler improves optimization for large long-tailed datasets with high imbalance factors.

  • Takeaways & Limitations

    Meta Sampler is computationally expensive, large-dataset optimization is slow, and Balanced Softmax only approximately guarantees the generalization error bound.

Abstract

from arXiv · show

Deep classifiers have achieved great success in visual recognition. However, real-world data is long-tailed by nature, leading to the mismatch between training and testing distributions. In this paper, we show that the Softmax function, though used in most classification tasks, gives a biased gradient estimation under the long-tailed setup. This paper presents Balanced Softmax, an elegant unbiased extension of Softmax, to accommodate the label distribution shift between training and testing. Theoretically, we derive the generalization bound for multiclass Softmax regression and show our loss minimizes the bound. In addition, we introduce Balanced Meta-Softmax, applying a complementary Meta Sampler to estimate the optimal class sample rate and further improve long-tailed learning. In our experiments, we demonstrate that Balanced Meta-Softmax outperforms state-of-the-art long-tailed classification solutions on both visual recognition and instance segmentation tasks.

1 Introduction

Long-tailed data creates a mismatch between imbalanced training distributions and class-balanced evaluation, making standard optimization unreliable. BALMS addresses this through Balanced Softmax and a complementary Meta Sampler, with strong results across long-tailed tasks.

  • Motivation: Long-tailed datasets contain many under-represented tail classes alongside a few high-frequency head classes.Class frequencies generally decrease exponentially from head to tail, and tail classes can be thousands of times rarer than head classes.
  • Motivation: Empirical-risk minimization on long-tailed training data often underperforms on class-balanced test data.This distribution mismatch poses difficulties for vision tasks as datasets scale.
  • Existing approaches: Class-balanced sampling and loss re-weighting can under-represent head classes or create gradient problems during optimization.These methods aim to simulate balanced training but may introduce optimization difficulties.
  • Proposed approach: Balanced Meta-Softmax combines Balanced Softmax, which models label-distribution shift, with a Meta Sampler that learns class re-sampling rates.The method is designed to address both moderate and extremely large imbalance factors.
  • Proposed approach: Balanced Softmax improves moderate-imbalance classification, while the Meta Sampler helps optimization when imbalance is extremely large.The paper gives CIFAR-10-LT with maximum imbalance factor 200 and LVIS with imbalance factor 26,148 as examples.
  • Results: BALMS outperforms state-of-the-art methods on five datasets spanning long-tailed image classification and instance segmentation.The evaluated datasets are CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, Places-LT, and LVIS; the LVIS advantage is reported as large.

2 Related Works

Prior long-tailed learning methods rebalance data, engineer losses, use meta-learning, or decouple training. Each strategy addresses imbalance differently but carries specific representation, stability, data, or scope limitations.

  • Data Re-Balancing: Re-sampling and re-weighting attempt to restore balanced training distributions from imbalanced data.Over-sampling may overfit tail classes, under-sampling may lose head-class variation, and re-weighting can destabilize severe-imbalance training.
  • Loss Function Engineering: Loss-function methods modify gradients, margins, uncertainty, or class-dependent costs to improve long-tailed optimization.Examples include dropping tail-class scores, increasing tail margins, modeling Bayesian uncertainty, and balancing gradient flow.
  • Meta-Learning: Meta-learning approaches learn sample weights through bi-level optimization using a clean, unbiased development set.The meta set is usually a fixed subset of training images.
  • Decoupled Training: Decoupled training improves generalization by separating representation learning from classifier training, but is inadequate for extremely high imbalance factors such as LVIS.The authors report that it complements BALMS and yields additional improvements when combined with it.

3 Balanced Meta-Softmax

BALMS addresses the mismatch between imbalanced training distributions and balanced test metrics by combining Balanced Softmax with a meta-learned sampler. Balanced Softmax models label-distribution shift and aligns with a multiclass generalization-bound objective, while Meta Sampler addresses re-sampling difficulties under extreme imbalance.

  • 3.1 Balanced Softmax: Standard Softmax gives a biased estimation of balanced-test conditional probabilities when training and testing label distributions differ.
  • 3.1 Balanced Softmax: Balanced Softmax accommodates label-distribution shifts between imbalanced training data and balanced test data.
  • 3.1 Balanced Softmax: The optimal margin is larger for classes with fewer samples, requiring lower training loss for tail classes.
  • 3.1 Balanced Softmax: Optimizing the Balanced Softmax loss is equivalent to minimizing a multiclass generalization upper bound.
  • 3.2 Meta Sampler: Class-balanced sampling can worsen performance when naively combined with Balanced Softmax because tail-dominated optimization may favor minority classes.
  • 3.2 Meta Sampler: Meta Sampler learns the optimal class sample rate through meta-learning to address over-balancing during re-sampling.

4 Experiments

Experiments evaluate BALMS across long-tailed classification and instance segmentation benchmarks, reporting strong performance and component-specific benefits under severe imbalance.

  • Experimental Setup: BALMS is evaluated on four long-tailed classification datasets and the LVIS instance segmentation dataset, using balanced test or validation evaluation for classification.Classification reports top-1 accuracy, including many-shot, medium-shot, and few-shot splits; LVIS reports detection and segmentation metrics.
  • Long-Tailed Image Classification: 3.4%: BALMS exceeds LWS on CIFAR-10 with imbalance factor 200, while also significantly outperforming cRT on the larger Places-LT dataset.The paper identifies high imbalance and larger dataset scale as settings where BALMS has a clear advantage.
  • Robustness Analysis: Balanced Softmax produces more balanced test-set likelihoods than Softmax and EQL across imbalance factors, while Meta Sampler reduces CBS’s over-balancing toward tail classes.Figure 1 uses end-to-end training; Figure 2 uses decoupled training with imbalance factor 200.
  • Overall Results: BALMS achieves state-of-the-art performance across all evaluated datasets and especially outperforms competing methods on LVIS under its extreme imbalance.LVIS has an imbalance factor of 26148, substantially higher than the other datasets.
  • Long-Tailed Instance Segmentation: 19.6 AP: BALMS achieves rare-class average precision on LVIS versus 14.6 for the best competing method.The authors attribute the result to more balanced gradients from Balanced Softmax and the learnable Meta Reweighter.
  • Component Analysis: Balanced Softmax improves overall performance in both end-to-end and decoupled training, and Meta Sampler generally improves performance, especially at higher imbalance factors.Meta Sampler also outperforms Meta Reweighter in the reported comparisons and addresses over-balancing.

5 Conclusion

BALMS addresses the training–testing distribution shift in long-tailed visual recognition by combining Balanced Softmax with a learned Meta Sampler. It generally outperforms state-of-the-art methods, especially under high imbalance, while incurring computational and approximation limitations.

  • 5 Conclusion: BALMS combines Balanced Softmax, which theoretically minimizes a generalization error bound, with a Meta Sampler for large long-tailed datasets.The method targets distribution shift between training and testing and uses meta-learning to improve optimization.
  • 5 Conclusion: BALMS generally outperforms state-of-the-art methods on 4 image classification datasets and 1 instance segmentation dataset by a large margin, especially when the imbalance factor is high.
  • 5 Conclusion: Meta Sampler is computationally expensive in practice, and optimization on large datasets is slow.Balanced Softmax only approximately guarantees a generalization error bound.

Broader Impact

BALMS is presented as a lightweight approach intended to connect long-tailed research benchmarks with industrial visual-recognition applications. The paper also notes potential misuse and possible negative effects on democratic privacy.

  • Broader Impact: BALMS is described as a lightweight method that adds minimal computational cost during training and is compatible with most existing visual-recognition works.The authors suggest this could help bridge research benchmarks and industrial applications.
  • Broader Impact: Applications of stronger recognition on long-tailed distributions may extend to real-life scenarios, where misuse could cause negative effects on democratic privacy.
  • Broader Impact: Multiclass classification can use k binary logistic regressions, with the predicted label selected as the class having the maximum model output.

A.4 Proof to Corollary 2.1

The proof develops an approximation to the loss that minimizes a generalization bound for balanced-test-set error. It uses margin-based class terms and relates the approximation to the training gradients under class-balanced sampling.

  • A.4 Proof to Corollary 2.1: A base margin term γ_base resolves the contradiction caused by requiring the estimated class probabilities to sum exactly to 1 without changing the optimal γ* value.
  • A.4 Proof to Corollary 2.1: Minimizing the bound reduces to minimizing its class-dependent terms, and the derived approximation approximately minimizes that bound under stated constraints.
  • A.4 Proof to Corollary 2.1: Under class-balanced sampling, each class contributes equally many samples, changing class counts to B/k in the batch-gradient analysis.

B.1 Meta Sampler

The Meta Sampler learns sample rates through a differentiable approximation to discrete sampling, connecting sampled images to classification loss via external memory. It uses bi-level optimization and differs from Meta Reweighter in its treatment of multi-label images.

  • B.1 Meta Sampler: The Meta Sampler assigns learnable class rates to instance-label pairs, normalizes them into an instance distribution ρ, and draws discrete image indexes to form batches.
  • B.1 Meta Sampler: Because discrete sampling and image augmentation are normally nondifferentiable, the method back-propagates loss gradients directly to the learnable sample rates.
  • B.1 Meta Sampler: The Straight-through Gumbel Estimator discretizes samples during the forward pass and uses the sample gradient during the backward pass.The estimator uses Gumbel noise and a temperature coefficient τ.
  • B.1 Meta Sampler: An external memory connects the sampler’s selected one-hot image vectors with the classification loss so gradients can reach the sample rates.
  • B.1 Meta Sampler: For LVIS, Meta Reweighter assigns each instance a learnable class-based loss weight because one image can contain instances from multiple categories.

C.4 Meta-learned sample rates with Softmax and Balanced Softmax

With Meta Sampler, Balanced Softmax produces a more balanced distribution of learned class sample rates than standard Softmax on CIFAR-100-LT.

  • Under standard Softmax, tail-class sample rates approach 1 while head-class rates approach 0.This extreme divergence could challenge meta-learning optimization and make very low optimal learning rates numerically unstable.
  • Balanced Softmax reduces the converged sample-rate variance from 0.13 with Softmax to 0.03.The sample rates are initialized at 0.5 and constrained to (0,1).
  • Figure 3 compares learned sample rates across classes ordered by decreasing training-set size.The experiment uses CIFAR-100-LT with imbalance factor 200.

D.2 Long-tailed datasets generation

The paper constructs long-tailed benchmarks by reducing or sampling class sizes while retaining specified balanced test sets or official dataset splits.

  • CIFAR-10-LT and CIFAR-100-LT are created by randomly reducing training samples to imbalance factors 200, 100, and 10.Their test sets remain unchanged and balanced.
  • ImageNet-LT samples class sizes from a Pareto distribution with α = 6, yielding 115,846 training images across 1,000 classes.Its imbalance factor is 256, and the original validation set supplies 50 test images per class.
  • Places-LT is generated similarly, with 62,500 training images across 365 classes and imbalance factor 996.The test set contains 100 images per class.
  • LVIS uses its official training and validation split without modification.

Appendix E Comparisons with Reported SOTA Results on CIFAR-LT

The appendix explains why the paper uses reproduced CIFAR-LT results for empirical analysis and presents Table 7 as a comparison with additional results reported in corresponding papers.

  • The empirical analysis uses the authors’ reproduced results on CIFAR-LT.
  • Prior works used different baselines, preventing fair direct comparison with those results.
  • Table 7 compares the authors’ method with additional results originally reported in the corresponding papers.

F.1 Visualization and analysis on the feature space of Balanced Softmax

The feature-space analysis contrasts Balanced Softmax with Softmax and class-balanced sampling, while a toy study examines how resampling affects optimization and classifier solutions.

  • Feature-space visualization: Balanced Softmax and CBS produce a more concentrated t-SNE center area than Softmax, whose features are described as more suitable for classification.The visualization uses CIFAR-10-LT with imbalance factor 200 and ten colors for the ten classes.
  • Decoupled training: Softmax-pretrained features generally outperform Balanced Softmax-pretrained features in decoupled CIFAR-10-LT training at imbalance factor 200.
  • Resampling analysis: In a three-class two-dimensional toy problem, resampling does not change the linear classifier’s solution but changes the optimization process.CBS causes an over-balance problem during optimization relative to Balanced Softmax.
  • Decision-boundary visualization: Both Softmax and Softmax+CBS converge to symmetric decision boundaries, with Softmax+CBS converging faster than Softmax.The toy dataset contains 10,000 red, 100 yellow, and 1 blue sample.
Loading 2007.10740v3…