PaperlayerView exact revision on arXiv

Source-linked AI summary

Well-classified Examples are Underestimated in Classification with Deep Neural Networks

Guangxiang Zhao, Wenkai Yang, Xuancheng Ren, Lei Li, Yunfang Wu, Xu Sun

arXiv:2110.06537v6cs.LGcs.AIcs.CLcs.CV

TL;DR

Well-classified examples receive less emphasis under CE, and the paper argues this hinders representation learning, energy reduction, and margin growth. It proposes adding bonuses to reward these examples. The authors report improved learning performance across image recognition and machine translation, alongside positive optimization effects in experiments.

  • Problem

    CE gives well-classified examples smaller gradient contributions, which the paper argues hinders representation learning, energy reduction, and margin growth.

    Source: §1 · pp. 1–2

  • Method

    The paper proposes Encouraging Loss, which adds a bonus to CE to increase the learning emphasis on well-classified examples.

    Source: §1 · p. 2, §4.1 · p. 3

  • Results

    Experiments report consistently improved learning performance on image recognition and machine translation, and positive optimization effects from valuing well-classified examples.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors report that enhancing learning from well-classified examples improves model capacity in imbalanced classification, OOD detection, and adversarial settings.

    Source: §1 · p. 2

    With the normal bonus, EL does not improve accuracy in some settings and slightly increases ECE; reducing the learning rate or adding label smoothing may help.

    Source: §6 · p. 8

Abstract

from arXiv · show

The conventional wisdom behind learning deep classification models is to focus on bad-classified examples and ignore well-classified examples that are far from the decision boundary. For instance, when training with cross-entropy loss, examples with higher likelihoods (i.e., well-classified examples) contribute smaller gradients in back-propagation. However, we theoretically show that this common practice hinders representation learning, energy optimization, and margin growth. To counteract this deficiency, we propose to reward well-classified examples with additive bonuses to revive their contribution to the learning process. This counterexample theoretically addresses these three issues. We empirically support this claim by directly verifying the theoretical results or significant performance improvement with our counterexample on diverse tasks, including image classification, graph classification, and machine translation. Furthermore, this paper shows that we can deal with complex scenarios, such as imbalanced classification, OOD detection, and applications under adversarial attacks because our idea can solve these three issues. Code is available at: https://github.com/lancopku/well-classified-examples-are-underestimated.

1 Introduction

The paper challenges the practice of down-weighting well-classified examples, arguing that it limits representation learning, energy reduction, and margin growth. It proposes rewarding these examples and reports improved learning performance across diverse tasks and settings.

1 Introduction

  • The authors argue that CE down-weights well-classified examples, hindering representation learning, energy reduction, and margin growth.They report that up-weighting these examples sharpens the energy surface, enlarges margins, and helps improve adversarial robustness.Source: §1 · pp. 1–2
  • Experiments on image recognition and machine translation show that the proposed counterexample can consistently improve learning performance.The authors also report benefits in imbalanced classification, OOD detection, and applications under adversarial attacks.Source: §1 · p. 2
  • Encouraging Loss (EL) counteracts this practice by up-weighting well-classified examples relative to CE.The authors theoretically show that EL revives their contribution to representation learning, reduces energy on the data manifold, and enlarges classification margins.Source: §1 · p. 2
  • The paper tests its counterexample on canonical and state-of-the-art models and reports benefits in imbalanced classification, OOD detection, and adversarial settings.The reported tasks include image recognition and machine translation based on heterogeneous data types.Source: §1 · p. 2

2 Related Work

The related work situates the paper among approaches to representation learning, energy-based interpretations of classifiers, and margin enlargement. It distinguishes its focus on improving CE’s learning from well-classified examples from these neighboring directions.

Representation learning

  • Contrastive learning mitigates representation-learning shortages through augmentation, while Kang et al. (2020) turn off class-level re-weighting during representation learning without improving CE itself.The paper frames its own focus as improving representation learning from well-classified examples.Source: §2 · p. 2

Using energy-based models to interpret the classifier

  • Prior work re-interprets classifiers as energy-based models for generative modeling, whereas this paper separately investigates energy on the data and data optimized by CE.The related work names classifier and conditional-EBM interpretations by Grathwohl et al. (2020), Xie et al. (2016), and Du and Mordatch (2019).Source: §2 · p. 2

Enlarging the margin

  • Margin-focused methods include hinge loss, which focuses on boundary-near points, and CE variants that adjust logits before softmax.The paper says its method can be combined with these large-margin approaches by replacing the original CE loss with EL after softmax.Source: §2 · p. 2

3 Exploring Theoretical Issues of CE Loss

The section sets up classification using logits, softmax probabilities, and negative log-likelihood, then identifies three limitations of CE: saturated gradients for well-classified examples, weak energy reduction on data, and limited margin growth.

3.1 Setup and Notations

  • For classification, the model maps inputs to K logits, softmax yields class probabilities, and CE is expressed as the negative log-likelihood of the true class.The section uses p for prediction correctness and defines loss steepness as ∂L/∂p.Source: §3.1 · p. 2

3.2 Limitations of CE loss in Three Aspects

  • As predictions become correct, softmax normalization brings gradient saturation back, hindering representation learning from well-classified examples.The section attributes this to the diminishing gradient contribution of those examples.Source: §3.2 · p. 3
  • CE has insufficient power to reduce energy on the data manifold because its terms pull the true-class energy both down and up.It can push up energy for non-target classes, but the true-class energy faces a competing term in the softmax denominator.Source: §3.2 · p. 3
  • CE’s incentive to enlarge the margin approaches zero as predictions near the target, limiting further margin growth.The section links larger likelihood to larger margins and explains that CE becomes less effective as likelihood rises.Source: §3.2 · p. 3

4 Gaining from Reviving the Learning of Well-classified Examples

This section introduces Encouraging Loss (EL), which increases the relative importance of well-classified examples compared with cross-entropy loss (CE). It describes bonus variants and explains how EL addresses CE’s representation-learning, energy-optimization, and margin-growth issues.

4.1 The Counterexample: Encouraging Loss

  • EL adds a bonus to CE, with the normal bonus log(1 − p) making the loss steeper again at higher likelihoods.The logarithm is clamped by a small epsilon to avoid numerical instability.Source: §4.1 · p. 3, Figure 2 · p. 4
  • Conservative bonuses end the log curve at LE and continue with its tangent, with well-classified-example emphasis increasing from right to left.The original CE loss corresponds to a constant bonus.Source: §4.1 · p. 4, Figure 3 · p. 4

4.2 The Counterexample Can Solve the Identified Issues in CE Loss

  • For normal-bonus EL, gradients are independent of likelihood; conservative variants also give larger gradients to well-classified examples.The section identifies enhanced representation learning from well-classified examples as an EL benefit.Source: §4.2 · p. 4
  • Unlike CE, EL does not need to push up data energy to minimize its second term, reducing the obstacle to lowering energy on the data.Conservative bonuses retain an obstacle, but it is smaller than with CE.Source: §4.2 · p. 4
  • EL increases margin faster than CE, and the margin-growth ratio for conservative bonuses remains larger than 1.For the normal bonus, the ratio becomes large as the model improves and the negative gap exponent approaches 0.Source: §4.2 · p. 4

5 Practical Effect of Encouraging the Learning of Well-classified Examples

The experiments evaluate Encouraging Loss (EL) across image, graph, and translation tasks, then examine representation learning, energy, margins, and more complex settings. They also test whether rewarding well-classified examples helps with imbalanced classification, OOD detection, adversarial attacks, and MSE.

5.2 Improving Classification Performance

  • On ImageNet, ResNet-50 accuracy rose 0.94 points and EfficientNet-B0 rose from 77.8 to 78.28.The paper also compares these improvements with other training methods, including Noisy Student and Mixup.Source: §5.2 · p. 5
  • Across tasks, EL improved performance, including graph accuracy by 0.44 on PROTEINS and 0.51 on NCI1, and BLEU by 0.41 on De-En and 0.63 on Fr-En.Table 1 summarizes CE and EL across tasks; Figure 4 plots performance under various settings.Source: Table 1 · p. 5, §5.2 · p. 5, Figure 4 · p. 6
  • With conservative bonuses, EL consistently improves performance; peaks occur between LE=0.5 and LE=0.75. In preliminary Transformer translation experiments, LE=1 made gradient norms >5× larger and brought a 1–2 BLEU drop.For LE=0.75, gradient norms were <1.7× larger than CE.Source: §5.2 · p. 6

5.3 Addressing the Issues of CE Loss in Practice

  • With representation learning, EL achieved accuracy 92.98 ± 0.01 versus 91.69 ± 0.04 for CE; EL also made conditional energy around data sharper and margins several times larger than CE.Source: §5.3 · p. 7

5.4 Coping with Complex Application Scenarios

  • On iNaturalist 2018, EL outperformed CE with conservative and aggressive bonuses; combining it with CRT improved results by 1.9 points.In the representation-learning phase with CRT, the method also revived learning from well-classified examples while keeping classifier-phase settings unchanged.Source: Table 3 · p. 7, §5.4 · p. 7

OOD detection

  • For MNIST versus Fashion-MNIST OOD detection, FPR95 fell from 8.10% to 0.95% with EL using minimum conditional energy as the indicator.The paper reports better OOD detection performance than CE on both its small- and large-scale tasks.Source: OOD detection · pp. 7–8, Table 4 · p. 8
  • On MNIST, EL was more robust than CE under FGSM and PGD attacks, including PGD with L∞ and L2 bounds.Figure 7 compares accuracy under FGSM and PGD attacks.Source: OOD detection · p. 8, Figure 7 · p. 8

5.5 Well-classified Examples Are also Underestimated in MSE

  • Adding a mirror bonus improved MSE performance, and MSE beat CE on MNIST, indicating that MSE also underestimates well-classified examples.Table 5 summarizes CE, MSE, and the mirror-bonus result on MNIST.Source: Table 5 · p. 8, §5.5 · p. 8

6 Limitations

EL has two reported practical difficulties: accuracy may not improve in some settings, and calibration error may increase.

6 Limitations

  • EL with the normal bonus may not improve accuracy in some settings, while its more confident predictions slightly increase ECE; reducing the learning rate or adding label smoothing are possible mitigations.The authors report that EL combined with label smoothing yields lower ECE than CE or CE with label smoothing.Source: §6 · p. 8

7 Conclusion

The paper argues that well-classified examples can help improve deep classification models, and supports this claim with theoretical analysis and experiments.

7 Conclusion

  • The paper theoretically and empirically finds that well-classified examples help further improve deep classification models, and verifies counterexamples that value their learning through varied experiments.The authors first analyze how common practice weakens learning from these examples, then evaluate counterexamples’ positive effects in optimization.Source: §7 · p. 8

Ethical Statement

The authors report no potential ethical considerations, citing their use of existing datasets, minority-class performance improvements, and training time similar to CE.

Ethical Statement

  • The work uses existing public datasets, can improve minority-class performance, and has training time similar to CE, which the authors say presents no potential ethical considerations.The authors state that the work does not facilitate social bias or add energy cost.Source: Ethical Statement · p. 8

A Complete Calculation of Equations

The calculations describe how EL affects gradients, conditional energy optimization, and margin growth; the accompanying appendix specifies experiment setups and a representation-learning verification procedure.

A Complete Calculation of Equations

  • EL’s normal bonus is log(1 − p), a mirror flip of CE, and gradients are larger for well-classified examples with conservative bonuses too.The appendix describes the normal-bonus EL and notes the larger gradients for conservative bonuses.Source: A Complete Calculation of Equations · p. 11
  • In the conditional energy form, EL avoids the CE incentive to push up data energy, creating more incentive to lower it, while the conservative bonus leaves a smaller obstacle.The conservative bonus does not remove the obstacle in the second term, but the passage says it is smaller than CE’s.Source: A Complete Calculation of Equations · p. 12
  • As the negative logit gap exponent A approaches zero, EL’s margin-growth ratio becomes large; the conservative-bonus ratio is smaller but remains above 1.The passage compares the margin-growth rate with CE and defines A as the exponential of the negative logit gap.Source: A Complete Calculation of Equations · p. 12

B Details of the Experiment Setups

  • Experiments use public datasets and, except for graph data ending through early exit, the same training rounds as the baseline; most listed tasks average five runs, while graph runs use 50 data splits.The image-recognition, translation, and imbalanced-classification results average five runs; graph runs each contain 50 train/validation/test splits.Source: §B.1 · p. 12

B.1 Experiment Setups for Main Results

  • The main setups include balanced image recognition, PROTEINS and NCI1 graph classification, and IWSLT De-En and Fr-En translation with Transformer models.Image datasets include MNIST, CIFAR-10, CIFAR-100, and ImageNet; the translation tasks use the Transformer model.Source: §B.1 · pp. 12–13

B.2 Experiment Setups for Verifying the Properties

  • To verify representation learning, the authors train the full model with CE or EL, then reinitialize and train the output layer with CE while keeping representations fixed.The classifier retraining stage lasts 20 epochs; the procedure separates representation learning from classification.Source: §B.2 · p. 13, Representation Learning in Balanced… · p. 13

B.3 Experiment Setups for Complex Application Scenarios

The appendix specifies protocols for imbalanced classification, OOD detection, adversarial robustness, and object detection, along with the study’s estimated training cost.

B.3 Experiment Setups for Complex Application Scenarios

  • The iNaturalist 2018 setup evaluates many-, medium-, and few-shot classes using Kang et al. (2020)’s setting, with Deferred Re-weighting and CRT as baselines.Training lasts 200 epochs; Deferred Re-weighting starts after 180 regular-training epochs, while CRT fixes the representations before classifier retraining.Source: §B.3 · p. 13
  • OOD detection uses minimum conditional energy across classes to distinguish in-distribution MNIST or ImageNet samples from Fashion-MNIST or iNaturalist 2018 samples.The appendix describes both a small-scale and a large-scale dataset pair.Source: §B.3 · p. 13

Adversarial Robustness

  • Robustness evaluation compares CE- and EL-trained models under FGSM and PGD attacks, and also tests Auto Attack.MNIST settings come from TorchAttacks; the paper uses Liu et al. (2016)’s settings for the main-paper Tables 1 and 5.Source: Adversarial Robustness · p. 14

Object Detection

  • The COCO object-detection experiments use RetinaNet-R-50-FPN-1x with a ResNet50 backbone, trained for 90,000 updates with eight images per batch.Source: Object Detection · p. 14

Adversarial Training

  • Adversarial training generates attack-based inputs iteratively at each training step, then evaluates the model with the corresponding attack methods.Source: Adversarial Training · p. 14

B.5 Training Costs

  • The study estimates its total training cost at 300 days on an RTX 2080TI GPU.Source: §B.5 · p. 14

C.1 Well-classified examples are also underestimated in the improved variants of CE loss

The appendix tests whether EL complements adversarial training and large-margin softmax, including its effects on margins and accuracy under attacks.

C.1 Well-classified examples are also underestimated in the improved variants of CE loss

  • The adversarial-training experiments generate attack examples at every step and test robustness with FGSM and PGD, with Auto Attack also evaluated in the appendix.Source: §C.1 · p. 14
  • EL is additive with adversarial training, although it provides no further benefit under L2-norm PGD.The authors suggest models trained with EL on clean data may already be robust to that attack.Source: §C.1 · p. 14
  • EL increases margins for most MNIST samples compared with CE, including under adversarial training.Source: §C.1 · p. 14, Figure 8 · p. 14

Large margin softmax

  • On MNIST, adding EL to large-margin softmax raises accuracy from 99.62± 0.04 to 99.64 ± 0.02.Under attack, accuracy rises from 92.89 to 93.81 at epsilon 0.2 and from 82.68 to 86.02 at epsilon 0.3; the authors also report ImageNet improvement where other margin-increasing methods do not.Source: Large margin softmax · p. 14

Label Smoothing

The appendix reports EL results across label smoothing, long-tailed classification, margins, energy-based OOD detection, and related loss designs.

Label Smoothing

  • In translation, EL is additive to pure CE loss, with improvements on par with label smoothing.Source: Label Smoothing · p. 14, Table 8 · p. 16
  • Halted Focal Loss (HFL) returns to CE when correct-label likelihood exceeds 0.5, while retaining focal loss in the low-likelihood region.The appendix reports improved results from HFL on imbalanced classification and COCO detection; its high-likelihood gradients match CE’s.Source: Label Smoothing · pp. 14–15

EL Reduces Excess Risk

  • EL reduces the train–test accuracy gap by 2, 3, and 2.5 points for ResNet50 on CIFAR-10, CIFAR-100, and ImageNet, respectively.Source: EL Reduces Excess Risk · p. 15

EL with Conservative Bonus also Reduces Energy on the Data and Enlarges Margins

Maximum Conditional Energy is a Good Indicator for OOD Detection

  • For OOD detection, EL with LE=0.75 lowers FPR95 from 75.13% for CE to 71.86%, and EL with LE=1 beats CE on two energy indicators.The paper describes minimum conditional energy as the maximum value among class-conditional energies, contrasting it with free energy’s log-sum-exp over energies.Source: Maximum Conditional Energy is a Good… · p. 17, Figure 11 · p. 17

Label Smoothing Helps reduce ECE in EL

  • With label smoothing, EL’s data-point energy remains lower than CE’s on MNIST, while its non-data-point energy is slightly higher.The reported averages are -12.6 versus -4 for data points and -0.07 versus -0.08 for non-data points; on CIFAR-10, EL shifts data energy from -5 to -12.3 and the lowest out-of-data energy from -2.3 to -0.8.Source: Label Smoothing Helps reduce ECE in EL · p. 17, Table 13 · p. 17

The combination of EL and label smoothing can significantly improve OOD detection

D Other Related Work

  • The related-work discussion connects EL’s emphasis on well-classified examples to Mixup and Poly Loss, while distinguishing EL’s likelihood-based reward design.The authors say their theory can explain Poly Loss’s success and that its additional polynomial term also increases the learning weight of well-classified samples.Source: D Other Related Work · pp. 17–18
Loading 2110.06537v6…