Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

• Long-tailed training data can produce poor performance on underrepresented classes, especially when testing uses uniform-label or minimum-class accuracy.Source: §1 · p. 1
• Re-weighting and re-sampling can improve class-accuracy trade-offs but may overfit minority classes because those classes provide less information.Source: §1 · pp. 1–2
• The paper regularizes minority classes more strongly than frequent classes to improve their generalization without sacrificing fit on frequent classes.Source: §1 · p. 2
• LDAM uses per-class margins motivated by a uniform-label generalization bound, while remaining compatible with re-weighting and re-sampling.Source: §1 · p. 2
• Deferred re-balancing postpones re-weighting, and the combined methods improve performance on artificially imbalanced CIFAR and Tiny ImageNet and real-world iNaturalist’18.Source: §1 · p. 2

Re-sampling

• Re-sampling either discards frequent-class data or repeatedly exposes minority examples, with under-sampling becoming infeasible under extreme imbalance and over-sampling risking minority overfitting.Source: §2 · pp. 2–3

Re-weighting

• Re-weighting can make deep-model optimization difficult under extreme imbalance and may reduce performance on frequent classes when inverse class frequency is used.Source: §2 · p. 3
• Focal loss and related methods adjust weights using individual-example properties such as classification difficulty or gradient magnitude.Source: §2 · p. 3

Novelty

• The paper’s approach adds minority-focused regularization orthogonal to re-weighting and proposes deferred re-balancing to improve a generic re-weighting scheme.Source: §2 · p. 3

Margin loss

• Unlike class-independent angular-margin methods, the paper uses class-distribution-aware margins to target minority-class generalization.Source: §2 · p. 3

Related settings

• The paper assumes the test label distribution is known, unlike distributionally robust formulations that only bound the amount of label shift.Source: §2 · p. 3

Meta-learning

• The authors regard loss-based methods as generally more computationally efficient than meta-learning approaches.Source: §2 · p. 3

Problem setup and notations

• The analysis assumes class-conditional distributions are unchanged between training and testing and evaluates error under a balanced test distribution.The balanced distribution samples classes uniformly, then samples inputs from each class-conditional distribution.Source: §3.1 · p. 4

Theoretical motivations

• Per-class margin bounds motivate larger margins for minority classes, whose smaller sample counts make generalization more difficult.The paper studies per-class and balanced-test error bounds rather than only the minimum margin across all examples.Source: §3.1 · pp. 4–5

Label-distribution-aware margin loss

• The proposed class-dependent margins scale as ∆j = C n_j^-1/4, assigning larger margins to classes with fewer training examples.This trade-off is derived from the binary-class analysis and extended to multiple classes.Source: §3.2 · pp. 5–6, §3.1 · p. 5
• LDAM extends soft-margin classification by enforcing label-distribution-dependent margins and is orthogonal to re-weighting and re-sampling.The implementation uses a smooth cross-entropy relaxation of the multiclass hinge loss, with normalized logits to simplify margin tuning.Source: §3.2 · pp. 5–6

Deferred Re-balancing Optimization Schedule

• Deferred re-balancing first trains with vanilla ERM and LDAM, then applies re-weighted LDAM after learning-rate annealing at a smaller learning rate.The first stage is intended to provide a good initialization, while the second stage changes the weights relatively little; this addresses optimization and over-fitting issues associated with re-balancing.Source: §3.3 · p. 6

Datasets

• Experiments cover artificially imbalanced IMDB, CIFAR-10, CIFAR-100, and Tiny ImageNet, plus real-world iNaturalist 2018.The evaluation uses controllable imbalance settings for the artificial datasets and a large-scale naturally imbalanced dataset.Source: §4 · p. 7

Methods

• LDAM-DRW combines label-distribution-aware margins with deferred re-weighting, while DRW and DRS apply re-balancing only after the initial ERM stage.LDAM is also evaluated with other losses and schedules, and LDAM-DRW is the main proposed algorithm.Source: §4 · p. 7

CIFAR results

• LDAM-DRW achieves the best top-1 validation-error performance on imbalanced CIFAR-10 and CIFAR-100, with each component beneficial in combinations with other losses or schedules.Table 2 compares ResNet-32 models under long-tailed and step imbalance settings.Source: Table 2 · p. 8, §4.2 · p. 8
• LDAM is superior to cross-entropy and focal loss without a data-rebalance schedule, while the full pipeline outperforms previous state-of-the-art methods by a large margin.Uniform-margin alternatives and hinge loss are included in the ablation analysis; hinge loss has optimization issues with 100 classes.Source: §4.2 · p. 8

Analysis

• On CIFAR-10 with step imbalance, the proposed algorithm generalizes better on minority classes while leaving frequent-class performance almost unaffected; under long-tailed imbalance, DRW becomes superior after learning-rate annealing.Figure 2 contrasts RW under-fitting and RS minority over-fitting, while Figure 3 marks annealing at epoch 160.Source: Figure 2 · p. 9, Figure 3 · p. 9, §4.4 · p. 10

iNaturalist 2018

• 10.86% and 6.88% are the reported improvements in iNaturalist 2018 top-1 error over ERM and the previous state of the art, respectively.The experiment uses 437,513 training images, 8,142 classes, balanced validation labels, and a ResNet-50 backbone.Source: §4.3 · p. 9, Table 3 · p. 9

5 Conclusion

• LDAM encourages larger margins for minority classes and is theoretically justified through a uniform-label generalization error bound.The analysis motivates class-dependent margins from per-class generalization bounds.Source: §5 · p. 10
• Deferred re-weighting postpones re-weighting until after the model learns an initial representation.The authors state that this may avoid drawbacks of re-weighting or re-sampling during early training, although its precise explanation remains unclear.Source: §5 · p. 10
• LDAM and deferred re-weighting achieve significantly improved performance across varied imbalanced-dataset vision tasks.The conclusion reports broad benchmark improvements while identifying the theoretical explanation of deferred re-weighting as an open direction.Source: §5 · p. 10

A Missing Proofs and Derivations in Section 3.1

• The proof defines class-specific hard-margin losses and empirical Rademacher complexities to control generalization from each class’s samples.The class-j sample count enters the complexity-based bound through the corresponding margin analysis.Source: A Missing Proofs and Derivations in… · p. 14
• Theorem 2 bounds balanced-class generalization for all positive class-dependent margins and hypotheses in the function class.The proof applies a classwise margin bound and combines the class-specific results by a union bound and averaging.Source: A Missing Proofs and Derivations in… · p. 14
• For binary classification, adding bias terms can shift the two class margins while preserving their total and reach the bound’s optimal margin allocation.The construction starts from a classifier with positive margins, selects bias terms, and obtains the optimized margins before applying Theorem 2.Source: A Missing Proofs and Derivations in… · pp. 14–15
• Substituting γ2 = β − γ1 and setting the derivative to zero yields the margin allocation that optimizes the two-class bound.The resulting optimized margins are then used to obtain the stated generalization guarantee.Source: A Missing Proofs and Derivations in… · pp. 14–15

B Implementation details

• Artificial imbalance distributions include long-tailed and step configurations for CIFAR datasets.Figure 4 illustrates the number of training examples per class for these generated distributions.Source: B Implementation details · p. 15
• CIFAR experiments train ResNet-32 for 200 epochs with batch size 128, momentum 0.9, and weight decay 2 × 10−4.Training uses four-pixel padding, random crops, and horizontal flips.Source: B Implementation details · p. 15
• Tiny ImageNet experiments use ResNet-18, 64 × 64 crops, batch size 128, and a 120-epoch training schedule.The passage specifies horizontal flips, eight-pixel padding, and an initial learning rate of 0.1.Source: B Implementation details · p. 15
• iNaturalist 2018 experiments train ResNet-50 for 90 epochs and begin rebalancing at epoch 60 in the two-stage schedule.Images are resized to a 256-pixel shorter side, cropped to 224 × 224, and evaluated with margins normalized so the largest enforced margin is 0.3.Source: B Implementation details · pp. 15–16
• Table 4 reports validation error on imbalanced Tiny ImageNet across different loss functions and training schedules.The supplied table caption identifies the comparison structure but does not provide numerical entries.Source: Table 4 · p. 16

C.1 Feature visualization

• The feature visualization constrains representations to three normalized dimensions, with training distributions in the top row and validation distributions in the second row.The setup enables scattering features on a unit hyper-sphere for visual comparison.Source: §C.1 · p. 16
• Tiny ImageNet has 200 classes with 500 training images and 50 validation images per class, and is evaluated under generated long-tailed and step imbalance.The results are presented in Table 4.Source: §C.2 · p. 17
• LDAM and deferred re-weighting show consistent improvements over ERM on imbalanced Tiny ImageNet, whereas Class-Balanced Softmax performs worse than ERM.This comparison extends evaluation beyond artificially imbalanced CIFAR.Source: §C.2 · p. 17

Feature extractor comparison

• Under step imbalance with ρ = 100 and µ = 0.5, ERM-trained backbone networks produce the most expressive feature embedding among the three methods.This comparison uses a linear classifier trained on features extracted from differently pretrained backbones.Source: §C.3 · p. 17

Deferred re-balancing

• Re-weighting and re-sampling exhibit similar performance when combined with deferred re-balancing.The authors use re-weighting as the default choice in the main paper.Source: §C.4 · p. 17

LDAM adaptation

• The LDAM extension adapts the class-specific margin ∆i using the training and test class counts ni and n′i.The cited passages introduce the relevant counts and state that LDAM is adapted by changing the margin for class i.Source: §C.5 · pp. 17–18

Test-distribution construction

• Test labels are generated with an imbalanced distribution and their frequencies are randomly permuted so the training and test distributions differ substantially.For the long-tailed setting with ρ = 100, the training distribution follows Figure 7a, while test distributions are shown in Figure 7b and Figure 7c.Source: §C.5 · p. 18, Figure 7 · p. 18

Robustness across test distributions

• Across two random seeds for generating test label distributions, the reported settings show qualitatively similar results.The precise generated label distributions are provided in the authors’ code.Source: §C.5 · p. 18

Experimental setting

• The extended algorithm is evaluated with both imbalanced training and imbalanced validation distributions for CIFAR-10.Table 6 reports Top-1 validation error for this scheme.Source: Table 6 · p. 18