PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-Tailed Recognition via Weight Balancing

Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, Shu Kong

arXiv:2203.14197v1cs.CV

TL;DR

Long-tailed recognition is difficult because naive training favors common classes and produces disproportionately large common-class weights. The paper studies weight regularization and a two-stage training procedure, reporting improved LTR performance and state-of-the-art accuracy across five benchmarks. The authors note that interactions with existing techniques and alternative regularizers remain open.

  • Problem

    LTR lacks balanced recognition under long-tailed data, where naive training favors common classes and produces artificially larger common-class weights.

  • Method

    The approach trains features with cross-entropy and tuned weight decay, then trains classifiers with class-balanced loss, weight decay, and MaxNorm.

  • Results

    The approach outperforms prior art on five long-tailed benchmarks, with ablations reaching 53.35% when combining MaxNorm and weight decay.

  • Takeaways & Limitations

    Simple parameter regularization provides a strong baseline for improving LTR without new losses, aggressive augmentation, or new network modules.

  • Takeaways & Limitations

    The study does not examine how weight balancing complements existing techniques or whether alternative regularizers such as Lp-norm weight decay perform better.

Abstract

from arXiv · show

In the real open world, data tends to follow long-tailed class distributions, motivating the well-studied long-tailed recognition (LTR) problem. Naive training produces models that are biased toward common classes in terms of higher accuracy. The key to addressing LTR is to balance various aspects including data distribution, training losses, and gradients in learning. We explore an orthogonal direction, weight balancing, motivated by the empirical observation that the naively trained classifier has "artificially" larger weights in norm for common classes (because there exists abundant data to train them, unlike the rare classes). We investigate three techniques to balance weights, L2-normalization, weight decay, and MaxNorm. We first point out that L2-normalization "perfectly" balances per-class weights to be unit norm, but such a hard constraint might prevent classes from learning better classifiers. In contrast, weight decay penalizes larger weights more heavily and so learns small balanced weights; the MaxNorm constraint encourages growing small weights within a norm ball but caps all the weights by the radius. Our extensive study shows that both help learn balanced weights and greatly improve the LTR accuracy. Surprisingly, weight decay, although underexplored in LTR, significantly improves over prior work. Therefore, we adopt a two-stage training paradigm and propose a simple approach to LTR: (1) learning features using the cross-entropy loss by tuning weight decay, and (2) learning classifiers using class-balanced loss by tuning weight decay and MaxNorm. Our approach achieves the state-of-the-art accuracy on five standard benchmarks, serving as a future baseline for long-tailed recognition.

∗MIT ♮UIUC †Argo AI ♯CMU

Figure 1 compares naive and weight-balanced LTR on CIFAR100-LT, showing higher accuracy with balancing and class accuracy varying with class cardinality.

  • Per-class classification accuracy is plotted against class cardinality for CIFAR100-LT with imbalance factor 100.
  • 53.35% overall accuracy with weight balancing exceeds the naive model's 38.38% on CIFAR100-LT with imbalance factor 100.The plotted comparison also includes per-class accuracy against class cardinality.

1. Introduction

The introduction frames LTR as recognition under long-tailed data, where naive models favor common classes, and proposes balancing network weights with simple regularization in a two-stage approach.

  • Motivation: LTR targets high accuracy averaged across classes when real-world data contains many rare classes and a few common classes.Examples include autonomous-vehicle recognition of rare strollers and animals and bioimage recognition of rarely seen species.
  • Status quo: Naive models favor common classes because their larger data volume dominates training losses and gradients, producing worse rare-class accuracy.
  • Motivation: Naive training yields artificially large common-class weights, motivating weight balancing as an orthogonal direction for LTR.
  • Contribution: L2-normalization fixes classifier weights at unit norm, while weight decay learns small balanced weights and MaxNorm grows small weights while capping all weights.
  • Key Findings: The proposed two-stage approach tunes weight decay for cross-entropy backbone training and weight decay plus MaxNorm for class-balanced classifier training.

2. Related Work

Related work covers LTR, regularization, and stage-wise training, positioning this paper's weight-balancing approach within established strategies for imbalanced recognition.

  • Long-Tailed Recognition (LTR): LTR addresses long-tailed class distributions in which few common classes have much more data than many rare classes.
  • Parameter Regularization: Parameter regularization adds information to ill-posed problems, improving generalizability and preventing overfitting.
  • Parameter Regularization: The paper examines L2-normalization, weight decay, and MaxNorm as established but underexplored regularizers in LTR.
  • Stage-wise Training: Stage-wise training decouples feature learning with cross-entropy from classifier learning with a class-balancing loss.

3. Weight Balancing for Long-Tailed Learning

Long-tailed recognition can be improved by balancing weight norms rather than only balancing data, losses, or gradients. L2-normalization, weight decay, and MaxNorm offer different constraints, with weight decay and MaxNorm supporting flexible balanced weights and strong performance.

  • Setup: LTR trains a K-way classifier on imbalanced data, where class cardinality is measured by IF and accuracy is averaged across classes.The classifier parameters include filters at intermediate layers and class-specific classifier filters.
  • Motivation: Naive training produces artificially larger-norm classifier weights for common classes, while intermediate-layer filters can also have imbalanced norms.The paper attributes classifier-weight growth to the greater amount of training data for common classes and reports similar imbalance among hidden-layer filters.
  • L2-normalization: L2-normalization fixes every classifier weight to unit norm, but its strict constraint can reduce classifier flexibility and sometimes lower overall LTR performance.Post-hoc normalization may favor rare-class accuracy while sacrificing common-class accuracy; τ-normalization allows varied norms and can perform better.
  • Weight decay and MaxNorm: Weight decay penalizes large weights more heavily and learns small, balanced weights, whereas MaxNorm permits weights to grow inside an L2-norm ball while capping their norms.MaxNorm can be implemented with projected gradient descent, which projects weights outside the ball back onto the constraint set.
  • Further discussion: Weight decay and MaxNorm dynamically balance norms, unlike L2-normalization, which strictly fixes them; their joint use can outperform either regularizer alone.Weight decay regularizes small weights and reduces overfitting, while MaxNorm prevents large weights from dominating training.
  • Training pipeline: The proposed two-stage approach tunes weight decay while training a backbone with cross-entropy, then trains the classifier with class-balanced loss, weight decay, and MaxNorm.Weight decay can conveniently balance all network weights, whereas applying MaxNorm across layers requires per-layer thresholds.

4. Experiments

Experiments evaluate weight balancing across five long-tailed benchmarks and show that tuned regularization and two-stage classifier learning substantially improve recognition. The strongest results combine weight decay, class-balanced loss, and MaxNorm, while performance on large-scale datasets remains below methods using ensembles or aggressive self-supervised augmentation.

  • Experiment Setup: Experiments use five long-tailed benchmarks, including CIFAR100-LT, ImageNet-LT, and iNaturalist2018, with balanced validation or test evaluation.CIFAR100-LT is evaluated at imbalance factors 100, 50, and 10; ImageNet-LT and iNaturalist2018 provide large-scale settings.
  • Ablation Study: τ-normalization improves the weight-decay backbone from 42.00% to 51.31%, demonstrating the importance of learning a backbone with balanced weights.The experiments also report that L2-normalization reaches 49.60% in post-hoc classifier modification.
  • Ablation Study: Classifier learning with class-balanced loss and weight decay reaches 52.42%, while adding MaxNorm achieves 53.35% and fine-tuning two layers reaches 53.55%.Using class-balanced loss alone improves only from 46.08% to 47.09%; regularization provides the larger additional gain.
  • Weight Analysis: MaxNorm substantially improves classifier weight and marginal-likelihood balance, whereas L2-normalization equalizes norms without producing uniform marginal likelihood.The final MaxNorm, weight-decay, and class-balanced-loss model yields nearly uniform marginal likelihood with a small bias toward rare classes.
  • Weight Analysis: Weight decay limits excessive norm growth, while MaxNorm caps weights within a norm ball and allows smaller weights to grow during training.Without regularization, classifier weights in the naive model grow rapidly in norm.
  • Ablation Study: Tuned weight decay raises CIFAR100-LT top-1 accuracy from 38.38% for naive training to 46.08%, outperforming most compared methods.The 46.08% result exceeds SSD at 46.00% and DiVE at 45.35% on CIFAR100-LT (IF100).
  • Benchmark Results: On ImageNet-LT and iNaturalists, the proposed methods rival prior work but underperform ensemble and self-supervised methods using multiple models or aggressive augmentation.These comparisons qualify the state-of-the-art results reported across the benchmark suite.

5. Conclusion

The paper presents parameter regularization as an effective, simple direction for long-tailed recognition, while identifying unresolved combinations with other techniques and broader societal risks.

  • 5. Conclusion: Parameter regularization substantially improves long-tailed recognition and outperforms prior methods across five long-tailed benchmarks.The approach focuses on balancing classifier weights rather than introducing new losses, aggressive augmentation, or new network modules.
  • 5. Conclusion: The authors recommend practitioners consider parameter regularization early when addressing real-world long-tailed distributions.
  • 5. Conclusion: The study does not examine how weight balancing complements expert models, sophisticated augmentation, or self-supervised pretraining.It also leaves alternative regularizers, such as Lp-norm weight decay where p≠2, for future work.
  • 5. Conclusion: Improving long-tailed classification can support fairness research but may also enable discriminatory systems targeting data-scarce subgroups.

Appendix

The appendix supplements the ablation evidence, provides executable tutorial code, and includes a demonstration of how regularizers change weights during training.

  • Appendix: The appendix adds ablation results supporting regularizers for learning with long-tailed recognition.
  • Appendix: It provides open-source Jupyter Notebook code as a self-explanatory tutorial for the approach.
  • Appendix: A video demo shows how weights change during training with different regularizers.

A. Detailed Ablation Study

Table 4 reports additional results beyond the ablation study in the main paper. Its caption contains the salient conclusions.

  • Table 4 presents results in addition to the ablation study reported in the main paper.
  • The table’s caption identifies the salient conclusions from these additional results.
  • Readers should consult the caption for the key interpretations of Table 4.

B. Open-Source Code

The released notebooks reproduce the paper’s CIFAR100-LT demonstrations, compare regularization choices across training stages, and support interactive educational use.

  • B. Open-Source Code: The first notebook compares naive first-stage training with tuned weight decay on CIFAR100-LT at imbalance factor 100.It should achieve approximately 39% overall accuracy for naive training and 46% with tuned weight decay.
  • B. Open-Source Code: The second notebook compares various regularizers during second-stage training and should achieve over 52% overall accuracy.
  • B. Open-Source Code: Jupyter Notebook is used because it enables interactive demonstrations for educational purposes.A command is provided for converting notebooks into Python scripts.
  • B. Open-Source Code: Running the code requires common packages including Pandas, torchvision, and PyTorch, with Python 3.7.4 listed among the reported versions.
  • B. Open-Source Code: The code requires more than 1GB of storage, saves checkpoints after every training epoch, and is released under the MIT License.

C. Video Demo

This section demonstrates how class-specific weight norms evolve during training using a 2-dimensional visualization setup on CIFAR100-LT.

  • The experiment trains models on CIFAR100-LT with imbalance factor 100.
  • A modified ResNet34 inserts an additional 2-dimensional pre-logit layer before the logits.
  • The added layer projects 2-dimensional pre-logit features into K-dimensional logits through weights W.
  • Each row of W is a class-specific filter weight whose norm can be examined during training.
Loading 2203.14197v1…