PaperlayerView exact revision on arXiv

Source-linked AI summary

Long-tail learning via logit adjustment

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, Sanjiv Kumar

arXiv:2007.07314v2cs.LGstat.ML

TL;DR

Long-tailed label distributions make rare-label generalisation difficult and bias naïve learning toward dominant labels. The paper introduces post-hoc and training-time logit adjustment based on label frequencies, with Fisher consistency for balanced error and strong empirical performance. Its scope includes unresolved practical variants and cautions against combining the two adjustments incoherently.

  • Problem

    Long-tailed data makes rare-label generalisation difficult and naïve learning biased toward dominant labels.

  • Method

    The paper applies label-frequency-based logit adjustment either post-hoc to trained models or within the softmax cross-entropy loss.

  • Results

    Logit adjustment is Fisher consistent for minimising balanced error and shows strong empirical performance on real-world datasets.

  • Takeaways & Limitations

    Logit adjustment unifies several recent long-tail proposals while providing statistical grounding for balanced-error optimisation.

  • Takeaways & Limitations

    The paper leaves further exploration of data-augmentation variants for future work.

Abstract

from arXiv · show

Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.

1 Introduction

Long-tailed data makes rare-label generalisation difficult and biases naïve learning toward dominant labels. The paper proposes statistically grounded logit adjustment methods that address limitations of weight normalisation and loss modification.

  • Long-tailed distributions leave many labels with few samples, making rare-label generalisation difficult and naïve learning biased toward dominant labels.
  • Existing approaches use post-hoc weight normalisation or loss modification, but each has important limitations.Weight normalisation depends on optimiser-sensitive weight norms, while loss modification can sacrifice softmax cross-entropy consistency.
  • The paper presents two logit-adjustment modifications of softmax cross-entropy, applied either post-hoc or during training.These methods unify several recent proposals and use label frequencies to modify logits.
  • Logit adjustment increases the relative margin between rare positive labels and dominant negative labels.
  • Logit adjustment is Fisher consistent for minimising balanced error and achieves strong empirical performance on real-world datasets.Balanced error averages per-class errors, and the paper reports efficacy on real-world datasets.

2 Problem setup and related work

Long-tail learning evaluates performance under highly skewed class priors, where balanced error is preferred to ordinary misclassification error. Existing methods modify inputs, outputs, or losses, but weight normalisation and loss modification have identified limitations.

  • Long-tail learning involves highly skewed class probabilities, with many rare labels occurring infrequently.
  • Balanced error averages per-class error rates because majority-label prediction can achieve low ordinary misclassification error under imbalance.
  • Class-imbalance methods modify model inputs, outputs, or internals such as the loss function.
  • Weight normalisation upweights rare labels through class-dependent scaling, but its rationale depends on weight norms correlating with class priors.
  • Under Adam, per-class weight norms are anti-correlated or independent of class frequency, making weight normalisation ineffective for class imbalance.
  • Per-label margin losses can shift decision boundaries but are not Fisher consistent for balanced error, even for binary problems.

3 Logit adjustment for long-tail learning: a statistical view

The paper derives logit adjustment from the Bayes-optimal rule for balanced error. This yields post-hoc and training-time procedures whose adjusted logits target balanced class probabilities and are Fisher consistent.

  • For balanced error, the Bayes-optimal prediction selects the label with the largest balanced class-probability, equivalently the largest class-conditional likelihood.
  • Balanced-error optimisation is invariant to arbitrary changes in class priors when class-conditional distributions remain fixed.
  • If native class probabilities are represented by logits, balanced probabilities are obtained by translating logits according to class priors.
  • The resulting procedures either train a standard model and adjust logits post-hoc or train directly for balanced class probabilities.
  • Logit adjustment is statistically grounded because its optimum coincides with the Bayes-optimal balanced-error solution, giving Fisher consistency.

4 Post-hoc logit adjustment

Post-hoc logit adjustment modifies a trained classifier’s prediction rule by subtracting a class-prior-dependent offset from each logit. It extends standard adjustment through a tunable τ to address neural-network calibration and differs from weight normalisation’s multiplicative update.

  • Procedure: Post-hoc logit adjustment predicts labels using logits shifted by a prior-dependent term, with τ > 0 controlling the adjustment.The class-prior estimates π may be empirical training frequencies.
  • Procedure: When τ = 1, post-hoc adjustment applies a plug-in estimate of P(y | x) using the class priors.
  • Calibration: For τ ≠ 1, the procedure can be viewed as applying the adjustment to temperature-scaled probability estimates.Temperature scaling is relevant because neural-network probability estimates are often uncalibrated.
  • Calibration: τ can be tuned using holdout calibration measures or fixed at 1 while calibration is encouraged during training.Suggested measures include expected calibration error, probabilistic sharpness, and proper scoring rules.
  • Comparison to existing post-hoc techniques: Post-hoc logit adjustment with τ = 1 has precedents in class-imbalance methods, while τ ≠ 1 is important for practical neural-network use because of calibration issues.
  • Comparison to existing post-hoc techniques: Unlike weight normalisation, logit adjustment makes an additive rather than multiplicative logit update and is consistent for minimising balanced error.The two updates can therefore produce different label orderings.

5 The logit adjusted softmax cross-entropy

The training-based approach incorporates class-prior offsets directly into softmax cross-entropy rather than applying them after training. It yields pairwise margins that protect rare labels and has Fisher-consistency grounding for balanced error, unlike some related losses.

  • The logit adjusted softmax cross-entropy: The logit-adjusted softmax cross-entropy directly incorporates class-prior offsets while learning the logits.This contrasts with post-hoc adjustment, which applies the offset after training.
  • The logit adjusted softmax cross-entropy: The loss is equivalent to learning shifted scores while making predictions with the corresponding prior-dependent correction.
  • Pairwise margins: For τ = 1, the loss corresponds to a pairwise margin formulation that demands larger gaps between rare positive and dominant negative labels.The intended effect is to prevent dominant-class scores from overwhelming rare-class scores.
  • Relation to existing losses: Several existing balanced and margin-based losses are special cases of the general pairwise margin formulation, but they may use only positive- or negative-label frequencies.
  • Consistency: The logit-adjusted loss has firm statistical grounding because it is Fisher consistent for balanced error.
  • Discussion and extensions: Combining the logit-adjusted loss with post-hoc adjustment is statistically incoherent because post-hoc correction distorts the balanced class-probability estimates.
  • Discussion and extensions: Logit adjustment can instead be combined with loss weighting or generalized through separate positive- and negative-label frequency parameters.
  • Discussion and extensions: The inconsistency of some adaptive-margin losses concerns non-separable multiclass data, including settings involving label noise or limited model capacity.

6 Experimental results

Experiments on synthetic and real-world long-tailed datasets show that logit adjustment approaches Bayes-optimal balanced error, outperforms weight normalisation and other corrections, and improves rare-class performance. The loss-based version generally has a slight advantage, while stronger architectures and selected combinations provide further gains.

  • 6.1 Results on synthetic dataset: The synthetic binary experiment evaluates affine classifiers under 5% positive-class prevalence using balanced error over 100 independent trials.Each method is trained on 10,000 instances and evaluated on a separate 10,000-sample test set.
  • 6.1 Results on synthetic dataset: Logit adjusted loss closely matches the Bayes-optimal balanced error and separator, whereas weight normalisation can hamper performance.Post-hoc logit adjustment approaches Bayes performance with suitable scaling, while weight normalisation fails in the synthetic comparison.
  • 6.2 Results on real-world datasets: Post-hoc logit adjustment consistently outperforms weight normalisation across scaling-parameter comparisons and improves weight-normalisation gains over ERM by an 8% relative reduction on CIFAR-10.Scaling further improves post-hoc logit adjustment beyond its unscaled performance.
  • 6.2 Results on real-world datasets: Logit adjusted softmax cross-entropy generally outperforms loss-correction baselines, including a 6% relative reduction on iNaturalist, with a slight advantage over post-hoc adjustment.The experiments use long-tailed CIFAR, ImageNet, and iNaturalist datasets with standard ResNet architectures and SGD-based training.
  • 6.2 Results on real-world datasets: On CIFAR-10, CIFAR-100, and iNaturalist, logit adjusted loss systematically improves over ERM, particularly for rare classes, while dominant classes generally have lower error across methods.The per-class analysis aggregates classes into ten frequency-sorted groups for CIFAR-100 and iNaturalist.
  • 6.2 Results on real-world datasets: Further data augmentation variants remain an empirical direction left for future work.The discussion identifies data augmentation as one possible complementary extension rather than evaluating it in the reported experiments.

A Proofs of results in body

The analysis characterizes when weighted and margin-modified losses are Fisher consistent for balanced error, and shows how logit adjustment satisfies this objective. It also distinguishes finite-temperature behavior from limiting cases and identifies limitations of arbitrary parameter choices.

  • Bayes analysis: Under constant weights, the Bayes-optimal score is linked to the class-conditional probabilities, while suitable weighting changes the effective class base rates.Weighted-loss learning is equivalent to learning under a distribution with modified priors.
  • Consistency: Balanced-error consistency requires the Bayes-optimal score to predict the positive class exactly when η(x) exceeds its prior π.This condition connects loss minimization to the balanced-error decision rule.
  • Consistency: A simple condition on weights and margins characterizes Fisher consistency for the binary pairwise margin loss.The paper states this condition as an if-and-only-if result.
  • Parameterizations: Logit adjustment uses an unweighted loss with a margin biased toward the rare class, providing one admissible consistent parameterization.Other admissible choices include standard class weighting with a constant margin and combined weighting-plus-margin modifications.
  • Limitations: Margin and weight choices cannot be selected arbitrarily: some cited losses fail the balanced-error consistency requirement, while finite-temperature and infinite-temperature behavior can differ.Constant positive margins may become irrelevant as γ approaches infinity, whereas consistency at finite γ depends on the full parameter choice.

B.2 Properness of the pairwise margin loss

The pairwise margin family is proper composite and can achieve balanced-error consistency through different link functions. Its weights determine the consistency threshold, while margins alter conditional Bayes risk and the loss geometry.

  • Properness: Every member of the pairwise margin family considered is proper composite with an invertible link function.The Bayes-optimal score therefore has the form f*(x)=Ψ(η(x)).
  • Link functions: The balanced loss, unequal-margin loss, and balanced-plus-margin loss use different link functions, although their inverse links can satisfy the balanced-error threshold.The first two choices satisfy Ψ^-1(0)=π for any γ>0, while the third does so as γ approaches infinity.
  • Bayes risk: For balanced error, the conditional Bayes risk is a tent-shaped concave function with its maximum at p=π.The curves are normalized to have maximum 1 for comparison.
  • Bayes risk: Unequal margins alone leave the conditional Bayes risk unchanged from standard log-loss, whereas weighting or combined weighting and margins better approximate the cost-sensitive curve.Thus margins can change the link function without changing the underlying conditional Bayes risk in the unequal-margin case.
  • Binary case: In the binary hinge-loss construction, consistency reduces to choosing weights so the lower-risk endpoint corresponds to η>c.The expected loss is piecewise linear, with possible minima at δ+1 and −δ−1.

D Experimental setup

The experiments use long-tailed CIFAR, ImageNet, and iNaturalist settings with standardized optimization and architecture choices. Additional evaluations cover alternative imbalance profiles, optimizer effects, ImageNet-LT correction, and per-class errors.

  • Common setup: All methods use SGD with momentum 0.9, five-epoch linear warm-up, and weight decay 10^-4 unless dataset-specific details override them.The setup is intended to provide a common comparison across methods.
  • CIFAR: CIFAR-10 and CIFAR-100 use ResNet-32 models trained for 200 epochs with batch size 128 and scheduled learning-rate decay.The base learning rate is 0.1, decayed at epochs 160 and 180.
  • ImageNet: ImageNet uses ResNet-50 for 90 epochs with cosine decay, batch size 512, standard cropping and flipping, and weight decay 5 × 10^-4.Its base learning rate is 0.4.
  • iNaturalist: iNaturalist uses ResNet-50 for 90 epochs with cosine decay, base learning rate 0.4, batch size 512, and the ImageNet augmentation procedure.The augmentation procedure is shared with the ImageNet experiment.
  • Additional experiments: Additional experiments evaluate CIFAR Step profiles, Adam-related weight-norm behavior, and post-hoc correction plus per-class errors on ImageNet-LT.These extensions test robustness across profiles, optimizers, and class-frequency groups.

E.1 Results on CIFAR-LT with Step-100 profile

On the Step-100 profile, tuned post-hoc logit adjustment outperforms weight normalization, while ImageNet-LT analyses show gains concentrated on rarer classes and consistent across frequency groups.

  • Step-100 results: 54.80 is achieved by logit adjustment with τ>1 on the Step-100 profile, whereas weight normalization is slightly better when τ=1.The reported comparison concerns the evaluated Step-100 results; lower balanced error is better in the table context.
  • Step-100 results: Logit adjustment outperforms weight normalization on Step-100 CIFAR-10 and CIFAR-100 when its scaling is suitably tuned.Figure 8 reports the tuned comparison across the Step profile, CIFAR-10, and CIFAR-100.
  • ImageNet-LT: ImageNet-LT per-class analysis sorts classes by frequency and shows significant gains from logit adjustment on rarer classes.The analysis groups classes into ten frequency buckets.
  • ImageNet-LT: On ImageNet-LT, tuned post-hoc logit adjustment is competitive with weight normalization as τ varies.The comparison is presented in Figure 10.
  • ImageNet-LT: ImageNet-LT results show consistent gains from logit adjustment across Many, Medium, and Few class groups.Many, Medium, and Few are defined by at least 100, 20–100, and at most 20 training examples, respectively.

F Does weight normalisation increase margins?

Weight normalisation is intended to increase tail-class classification margins, but this intuition is not guaranteed in multiclass settings. Functional and geometric margins need not align, so controlling class-specific weight norms does not necessarily control geometric margins.

  • Normalising classifier weights is motivated by the expectation that it increases classification margins for tail classes.This intuition arises when weight norms track class priors under momentum-based SGD.
  • The classifier uses scores of the form f_y(x) = w_y^T Φ(x), with class weight vectors w_y and representation Φ(x).
  • The per-group error comparison divides classes into Many, Medium, and Few according to their training-example counts.Many has at least 100 examples, Medium has 20–100, and Few has at most 20.
  • In multiclass classification, functional and geometric margins do not generally align.Consequently, a fixed functional margin does not imply that reducing a weight norm increases the geometric margin.
  • Controlling each class weight norm does not necessarily control the geometric margin.
  • For binary logistic models, the bias term corresponds to the log-odds of the class priors.
Loading 2007.07314v2…