Source-linked AI summary
Influence-Balanced Loss for Imbalanced Visual Classification
Seulki Park, Jongin Lim, Younghan Jeon, Jin Young Choi
TL;DR
Class imbalance can bias DNN decision boundaries toward majority classes, while class- or difficulty-based weighting may not account for each sample’s influence. The paper derives influence-balanced loss for a two-phase training scheme, using influence-based inverse weighting during fine-tuning. Across multiple benchmark data sets, it reports improved generalization, minority-class performance, and results exceeding state-of-the-art cost-sensitive methods.
Problem
Imbalanced data can cause DNNs to overfit majority classes, while conventional weighting assigns identical weights within classes despite unequal sample influence.
Method
The method derives influence-balanced loss from influence functions and applies it during fine-tuning after normal training to re-weight samples inversely to their decision-boundary influence.
Results
The proposed method improves generalization and minority-class performance across multiple benchmark data sets while outperforming state-of-the-art cost-sensitive re-weighting methods.
Takeaways & Limitations
IB loss is presented as easy to implement and integrate with existing imbalance-learning methods, without restriction to a specific task, model, or training method.
Takeaways & Limitations
The influence-function derivation assumes a near-stationary optimum and a strictly convex local basin with positive-definite Hessian.
Abstract
from arXiv · showhide
In this paper, we propose a balancing training method to address problems in imbalanced data learning. To this end, we derive a new loss used in the balancing training phase that alleviates the influence of samples that cause an overfitted decision boundary. The proposed loss efficiently improves the performance of any type of imbalance learning methods. In experiments on multiple benchmark data sets, we demonstrate the validity of our method and reveal that the proposed loss outperforms the state-of-the-art cost-sensitive loss methods. Furthermore, since our loss is not restricted to a specific task, model, or training method, it can be easily used in combination with other recent re-sampling, meta-learning, and cost-sensitive learning methods for class-imbalance problems.
1. Introduction
Imbalanced data can make DNNs overfit majority classes, while existing re-weighting strategies may overfit hard samples. The paper proposes influence-balanced loss to re-weight samples by their decision-boundary influence and reports improved generalization and minority-class performance.
- Highly imbalanced training data can produce satisfactory overall performance while models perform poorly on minority classes.
- Existing cost-sensitive methods use global class frequencies and assign identical fixed weights to samples within each class.
- Hard-example re-weighting can cause high-capacity DNNs to memorize overlapping-region samples and overfit them.
- Influence-balanced loss estimates each sample’s influence on the decision boundary and re-weights its loss inversely to that influence.Training uses standard training followed by fine-tuning with the influence-balanced loss.
- The method outperforms state-of-the-art cost-sensitive re-weighting methods across multiple benchmark data sets and improves all reported minority classes.
- Simply replacing standard cross-entropy with the proposed loss significantly improves generalization on highly imbalanced data.
2. Related Work
Imbalanced-learning research includes data-level, cost-sensitive, and meta-learning approaches, each with practical trade-offs. This work applies influence functions during training to make sample re-weighting responsive to decision-boundary influence.
- Data-level approach: Data-level methods balance distributions through under-sampling, over-sampling, or synthetic-sample generation.
- Data-level approach: Under-sampling can discard valuable information, while over-sampling and data generation can overfit repetitive samples.
- Re-weighting approach: Cost-sensitive methods commonly weight samples using class frequency or effective sample number, but class-based schemes give the same weight to samples in one class.
- Re-weighting approach: Difficulty- or loss-based re-weighting down-weights well-classified samples and emphasizes hard examples, which can lead DNNs to memorize them.
- Meta-learning approach: Meta-learning methods enhance data-level or cost-sensitive approaches but can require additional unbiased data, meta-samplers, or substantial computation.
- Influence function: Unlike prior diagnostic uses of influence functions, this work uses them during training to design influence-balanced loss.
3. Method
The method re-weights training samples according to their influence on the decision boundary, then applies the influence-balanced loss during fine-tuning to reduce overfitting by influential majority samples.
- Influence-based weighting: The method re-weights samples by their influence on the decision boundary to form a more generalized boundary under class imbalance.Influence is estimated using an influence function, which measures parameter change when a training sample is removed without retraining.
- Influence-based weighting: The influence function is approximated by a sample’s parameter influence, under the assumption that the empirical-risk gradient is approximately zero near the optimum.The derivation uses a locally strictly convex setting with a positive-definite Hessian.
- Influence-balanced loss: Because inverse-Hessian computation is expensive, the IB weighting factor uses the L1 magnitude of the loss gradient as a relative influence measure.For the final fully connected layer, this factor becomes ||f(x,w)-y||_1 · ||h||_1, whose inverse down-weights influential samples.
- Influence-balanced loss: The influence-balanced loss re-weights sample loss inversely to the gradient-based influence factor, constraining the boundary against overfitting to influential majority samples.The formulation is derived for softmax cross-entropy and can also apply to sigmoid cross-entropy or mean squared error.
- Class-wise re-weighting: Class-wise weights λ_k complement sample-wise influence weighting by slowing majority-loss minimization and further down-weighting influential majority samples.The class-wise term is normalized across classes and adjusted with hyper-parameter α.
- Influence-balanced training scheme: Influence-balanced training has normal-training and fine-tuning phases, with IB loss applied after transition time T1 without requiring a new architecture or training scheme.The paper generally recommends setting T1 at half of the total training schedule, near model convergence.
4. Experiments
Experiments across imbalanced image benchmarks evaluate influence behavior, training choices, class-wise accuracy, and comparisons with existing cost-sensitive methods. The proposed IB loss improves minority-class performance and generalization, including when combined with other methods.
- Experimental setup: Experiments used CIFAR-10, CIFAR-100, Tiny ImageNet, and iNaturalist 2018, spanning imbalance ratios from 10 to 500.CIFAR and Tiny ImageNet were artificially imbalanced, while iNaturalist 2018 has a naturally long-tailed imbalance ratio of 500.
- Influence analysis: In imbalanced CIFAR-10, minority samples had significantly less influence than majority samples, whereas influence distributions differed little in the balanced dataset.The comparison used ResNet-32 trained with standard cross-entropy on long-tailed CIFAR-10 with imbalance ratio ρ = 100.
- Influence analysis: Using the L1 norm for influence magnitude achieved the best classification accuracy across multiple CIFAR-10 imbalance ratios.The study compared L1, L2, and L∞ norms.
- Training schedule: Transitioning to influence-balanced fine-tuning after training loss convergence produced the best performance, with robust results across transition times.For the tested long-tailed CIFAR-10 settings, transitioning after the 100th epoch was best.
- Influence analysis: Setting ϵ = 1e−3 yielded the best performance, while removing the IB weighting factor greatly decreased accuracy.The epsilon analysis used three random seeds on long-tailed CIFAR-10 with ρ = 100.
- Comparison with baselines: IB loss significantly improved generalization, improved all minority classes, outperformed other baselines on Tiny ImageNet, and achieved considerable improvement on iNaturalist 2018.On CIFAR benchmarks, IB alone could achieve the best performance, while combining IB with other methods could further improve accuracy on multiple datasets.
5. Conclusion
The paper concludes that influence-balanced loss addresses majority-class overfitting and improves generalization under class imbalance. It is also designed for integration with existing methods, with experiments including Tiny ImageNet and iNaturalist 2018 evaluations.
- Conclusion: IB loss addresses majority-class overfitting by assigning weights according to each sample’s influence on the model.The method directly focuses on sample influence rather than relying only on class-level weighting.
- Conclusion: Experiments demonstrate improved generalization performance under class-imbalance settings.
- Conclusion: IB loss is easy to implement and integrate with existing methods for imbalanced learning.
- Conclusion: Table 5 reports class accuracy (%) for ResNet-18 on Tiny ImageNet.
- Conclusion: Table 6 reports class accuracy (%) for ResNet-50 on iNaturalist 2018.