PaperlayerView exact revision on arXiv

Source-linked AI summary

Class-Balanced Loss Based on Effective Number of Samples

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, Serge Belongie

arXiv:1901.05555v1cs.CV

TL;DR

Long-tailed class distributions make visual recognition perform poorly on weakly represented classes, while inverse-frequency reweighting can be inadequate. This paper models sample overlap to define effective sample counts and derives a class-balanced loss that significantly improves performance across CIFAR, ImageNet, and iNaturalist.

  • Problem

    Long-tailed datasets make models perform poorly on weakly represented classes, while existing inverse-frequency class balancing can yield poor performance.

  • Method

    The paper models samples as neighboring regions to quantify overlap, then reweights loss inversely by each class’s effective sample count.

  • Results

    The class-balanced loss significantly improves performance across long-tailed CIFAR, ImageNet, and iNaturalist datasets and improves several existing loss functions.

  • Takeaways & Limitations

    The framework offers a generally applicable way to quantify data overlap and construct class-balanced losses for long-tailed visual recognition.

  • Takeaways & Limitations

    The framework simplifies overlap by assuming each new sample is either entirely inside or entirely outside previously sampled data.

Abstract

from arXiv · show

With the rapid increase of large-scale, real-world datasets, it becomes critical to address the problem of long-tailed data distribution (i.e., a few classes account for most of the data, while most classes are under-represented). Existing solutions typically adopt class re-balancing strategies such as re-sampling and re-weighting based on the number of observations for each class. In this work, we argue that as the number of samples increases, the additional benefit of a newly added data point will diminish. We introduce a novel theoretical framework to measure data overlap by associating with each sample a small neighboring region rather than a single point. The effective number of samples is defined as the volume of samples and can be calculated by a simple formula $(1-β^{n})/(1-β)$, where $n$ is the number of samples and $β\in [0,1)$ is a hyperparameter. We design a re-weighting scheme that uses the effective number of samples for each class to re-balance the loss, thereby yielding a class-balanced loss. Comprehensive experiments are conducted on artificially induced long-tailed CIFAR datasets and large-scale datasets including ImageNet and iNaturalist. Our results show that when trained with the proposed class-balanced loss, the network is able to achieve significant performance gains on long-tailed datasets.

1. Introduction

Long-tailed real-world datasets make CNNs biased toward dominant classes, while inverse-frequency re-weighting and re-sampling have important limitations. The paper addresses this by modeling sample overlap, defining an effective sample number, and using it to construct a class-balanced loss.

  • Problem: Real-world visual-recognition datasets often have long-tailed class distributions, with a few dominant classes containing most examples.This imbalance biases models toward dominant classes.
  • Limitations of Existing Methods: Re-sampling can duplicate data and cause overfitting or discard valuable examples, motivating a focus on re-weighting for CNN training.Over-sampling also slows training, while under-sampling removes examples useful for feature learning.
  • Limitations of Existing Methods: Inverse-frequency class weighting often performs poorly on large-scale, real-world long-tailed datasets, motivating smoothed weighting heuristics.Recent work uses weights empirically proportional to the inverse square root of class frequency.
  • Proposed Approach: The paper models information overlap among samples to study effective sample size and design a class-balanced term for long-tailed training data.The framework is intended to explain why additional samples provide diminishing benefit as overlap increases.
  • Contributions: The proposed class-balanced term improves existing softmax cross-entropy, sigmoid cross-entropy, and focal losses.The contribution explicitly reports significant performance improvements when the term is added to these commonly used losses.

2. Related Work

Prior work on long-tailed imbalance primarily uses re-sampling or cost-sensitive learning, while also exploring knowledge transfer, metric-learning objectives, and covering-based formulations. These approaches address imbalance through data manipulation, weighting, alternative objectives, or theoretical sample-overlap concepts.

  • Related Work: Previous efforts on long-tailed imbalanced data mainly fall into re-sampling and cost-sensitive learning.Re-sampling includes over-sampling and under-sampling.
  • Re-Sampling: Over-sampling can overfit through repeated minority samples, while interpolation or synthesis introduces noise; under-sampling has therefore been preferred in some work.These methods target minority classes but involve risks from repetition, synthetic-sample noise, or removing important samples.
  • Cost-Sensitive Learning: Cost-sensitive learning assigns sample weights to match target distributions and includes inverse class-frequency and smoothed inverse-square-root-frequency weighting.This line of work builds on importance sampling and adjusts decision boundaries through weighting.
  • Related Work: Other imbalance strategies transfer knowledge from major to minor classes or design improved training objectives through metric learning.These methods provide alternatives to direct re-sampling and class-frequency weighting.
  • Covering and Effective Sample Size: The paper’s covering-based framework is inspired by random covering, and its effective number of samples differs from statistical effective sample size.Statistical effective sample size measures variance when samples are correlated, whereas the proposed framework concerns covering through small random sets.

3. Effective Number of Samples

The section models sampling as random covering, defining the effective number of samples as expected covered volume. This framework captures diminishing marginal benefit through an exponential function controlled by β.

  • Random-covering formulation: Each sample is modeled as a unit-volume neighboring region that may overlap other samples, rather than as a single point.Sampling is formulated as a random covering process over a class’s feature-space set.
  • Definition: The effective number of samples is defined as the expected volume of sampled data.This definition measures coverage while accounting for overlap among sampled regions.
  • Overlap assumption: The model simplifies overlap by assuming each new sample is either entirely inside previously sampled data or entirely outside it.This assumption makes the expected-volume calculation tractable.
  • Growth behavior: The effective number grows exponentially with sample count n, while β ∈ [0, 1) controls its growth rate.The formulation reflects diminishing marginal benefits as additional samples increasingly overlap existing data.
  • Asymptotic properties: When β = 0, En = 1; as β → 1, En → n, corresponding respectively to complete overlap and increasingly unique samples.For large N, effective sample count approaches n; for N = 1, all class data can be represented by one prototype.

4. Class-Balanced Loss

The Class-Balanced Loss re-weights examples inversely to each class’s effective sample count, using a factor (1 − β)/(1 − β^n_i) with β ∈ [0, 1). The model-agnostic and loss-agnostic term smoothly spans no re-weighting to inverse-frequency re-weighting and applies to several standard losses.

  • Core formulation: The Class-Balanced Loss addresses imbalanced training by weighting examples inversely to the effective number of samples and can be applied across deep networks and loss functions.The weighting term is designed as a general re-balancing component rather than a model-specific loss.
  • Core formulation: The normalized class weight α_i is inversely proportional to the effective sample number and is scaled so that the weights sum to C across classes.This normalization keeps the total loss roughly on the same scale after re-weighting.
  • Re-weighting behavior: The hyperparameter β smoothly controls re-weighting: β = 0 gives no re-weighting, whereas β → 1 approaches inverse class-frequency re-weighting.The effective-number term therefore provides a continuum between the two weighting extremes.
  • Applications: The class-balanced term is independent of the loss function L and predicted probabilities p, and is demonstrated with softmax cross-entropy, sigmoid cross-entropy, and focal loss.Its model-agnostic and loss-agnostic design supports use with multiple common classification objectives.
  • Applications: For focal loss, the class-balanced formulation is equivalent to the α-balanced variant when α_t = (1 − β)/(1 − β^n_y), explicitly setting α_t from the effective sample count.This connects the proposed term to focal-loss re-weighting through a data-dependent α_t.

5. Experiments

Experiments evaluate class-balanced losses on synthetic long-tailed CIFAR datasets and large-scale datasets including iNaturalist and ImageNet, using residual networks trained from scratch. The proposed loss improves performance across long-tailed settings, with β sensitivity depending on dataset and strong large-scale results using class-balanced focal loss.

  • Datasets and setup: Experiments cover long-tailed CIFAR-10 and CIFAR-100, iNaturalist 2017 and 2018, and ILSVRC 2012 using residual networks trained from scratch.The iNaturalist datasets contain 579,184 images from 5,089 classes in 2017 and 437,513 images from 8,142 classes in 2018; ILSVRC 2012 contains 1,281,167 training and 50,000 validation images.
  • Long-Tailed CIFAR: Class-balanced loss significantly improves commonly used loss functions on long-tailed CIFAR datasets when hyperparameters are properly selected.Comparisons include softmax cross-entropy, sigmoid cross-entropy, focal loss, and the proposed class-balanced loss.
  • Long-Tailed CIFAR: β = 0.9999 is unanimously optimal on CIFAR-10, whereas CIFAR-100 favors smaller, imbalance-dependent β values.On CIFAR-10, larger β consistently provides greater gains; on CIFAR-100, β = 0.99 or β = 0.999 improves the original loss, while larger β hurts performance.
  • Large-Scale Datasets: On large-scale datasets, class-balanced focal loss with β = 0.999 and γ = 0.5 outperforms softmax cross-entropy, by large margins on iNaturalist.ResNet-50 achieves comparable performance with ResNet-152 on iNaturalist and ResNet-101 on ILSVRC 2012.

6. Conclusion and Discussion

The work introduces a theoretically grounded framework that quantifies effective sample numbers by accounting for data overlap and yields a class-balanced re-weighting loss. Experiments validate the loss on CIFAR, iNaturalist, and ImageNet, while the non-parametric framework remains extensible through data-distribution assumptions.

  • Contributions: The framework quantifies effective sample numbers by accounting for data overlap in long-tailed training data.It is presented as a theoretically sound approach to long-tailed distributions.
  • Contributions: The proposed class-balanced loss re-weights loss inversely with each class’s effective number of samples.Extensive studies on artificially induced long-tailed CIFAR datasets analyze the loss, with benefits verified on CIFAR, iNaturalist, and ImageNet.
  • Discussion: The framework is non-parametric, makes no assumptions about data distribution, and is broadly applicable to existing models and loss functions.Future work will incorporate reasonable assumptions about the data distribution to improve effective-number estimation.

Appendix A: More Experimental Results

The appendix reports comprehensive long-tailed CIFAR experiments, evaluating ResNet-32 models with softmax, sigmoid, focal, and class-balanced loss variants across multiple imbalance factors and datasets.

  • Visual Recognition on Long-Tailed CIFAR: ResNet-32 models were evaluated on long-tailed CIFAR-10, CIFAR-20, and CIFAR-100 with imbalance factors of 200, 100, 50, 20, and 10.The experiments included original and class-balanced variants of softmax, sigmoid, and focal losses, using β = 0.9, 0.99, 0.999, and 0.9999; focal loss also used γ = 0.5, 1.0, and 2.0.
  • Visual Recognition on Long-Tailed CIFAR: Classification error rates for the three long-tailed CIFAR datasets are presented in Figures 8, 9, and 10.Rows correspond to specific loss-function, γ, and β configurations, while columns correspond to datasets with specific imbalance factors; lighter colors indicate lower error rates.
Loading 1901.05555v1…