Source-linked AI summary
Symmetric Cross Entropy for Robust Learning with Noisy Labels
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, James Bailey
TL;DR
Training DNNs with noisy labels is difficult because CE both overfits noisy labels on easy classes and underlearns hard classes. The paper proposes SL, which combines CE with noise-tolerant RCE, and reports improved robustness, class-wise learning, and compatibility with existing methods.
Problem
The paper addresses the limited understanding and handling of CE’s class-biased learning under noisy labels, where easy classes overfit while hard classes are underlearned.
Method
SL symmetrically boosts CE with the noise-tolerant Reverse Cross Entropy loss, supported by theoretical and empirical analysis.
Results
SL improves robustness over state-of-the-art methods across benchmark and real-world datasets and further enhances existing methods.
Takeaways & Limitations
SL is a simple, easily implemented loss framework for robust DNN training with noisy labels and combination with other techniques.
Abstract
from arXiv · showhide
Training accurate deep neural networks (DNNs) in the presence of noisy labels is an important and challenging task. Though a number of approaches have been proposed for learning with noisy labels, many open issues remain. In this paper, we show that DNN learning with Cross Entropy (CE) exhibits overfitting to noisy labels on some classes ("easy" classes), but more surprisingly, it also suffers from significant under learning on some other classes ("hard" classes). Intuitively, CE requires an extra term to facilitate learning of hard classes, and more importantly, this term should be noise tolerant, so as to avoid overfitting to noisy labels. Inspired by the symmetric KL-divergence, we propose the approach of \textbf{Symmetric cross entropy Learning} (SL), boosting CE symmetrically with a noise robust counterpart Reverse Cross Entropy (RCE). Our proposed SL approach simultaneously addresses both the under learning and overfitting problem of CE in the presence of noisy labels. We provide a theoretical analysis of SL and also empirically show, on a range of benchmark and real-world datasets, that SL outperforms state-of-the-art methods. We also show that SL can be easily incorporated into existing methods in order to further enhance their performance.
1. Introduction
The paper identifies class-biased learning under CE: noisy labels cause easy classes to overfit while hard classes remain underlearned. It proposes SL, combining CE with noise-tolerant RCE, and reports improved class-wise learning and robustness.
- DNN training with noisy labels is practically important because large-scale label annotation is costly, error-prone, and rarely perfectly clean.
- CE learning is class-biased: easy classes converge faster, while hard classes can remain significantly underlearned, especially with noisy labels.Under 40% noise, class 3 reaches about 60% accuracy while class 6 exceeds 90%.
- SL combines CE with noise-tolerant Reverse Cross Entropy to address both hard-class underlearning and noisy-label overfitting.
- Under 40% noise, SL narrows class-wise accuracy variation to 20%, with 95% highest and 75% lowest accuracy, close to clean-label behavior.
- The paper provides theoretical analysis and empirical understanding of SL, identifying hard-class underlearning as a key bottleneck in noisy-label learning.
- SL improves robustness over state-of-the-art methods and can be incorporated into existing methods with minimal training-process intervention.
2. Related Work
Existing noisy-label methods modify labels, losses, or training strategies, with different requirements and complexities. Label correction may require clean data or expensive noise-model estimation, while loss correction can depend on an unavailable or inaccurately estimated transition matrix.
- Noisy-label methods are commonly categorized as label correction, loss correction, or refined training strategies.
- Label correction methods infer cleaner labels using noise models, but may require extra clean data or expensive detection processes.
- Loss correction methods modify the loss using a noise transition matrix, whose ground truth may be unavailable and whose estimation can be difficult.
- Refined training strategies include sample weighting, disagreement-based updates, and co-teaching between two networks.
3. Weakness of Cross Entropy
Under noisy labels, CE learns classes unevenly: easy classes can overfit while hard classes remain substantially underlearned. This class bias is linked to weak representations and contributes to overall performance degradation.
- CE learning is class-biased, with easy and hard classes converging at different rates.The analysis examines this behavior on CIFAR-10 using an 8-layer CNN and symmetric label noise.
- Under 40% noisy labels, hard classes such as classes 2 and 3 can lag the clean setting by up to 20%, while easy classes such as 1 and 6 begin overfitting.Easy-class accuracy drops from epoch 50 to 100, whereas hard-class under learning persists later in training.
- For noisy labels, class-wise accuracy remains highly variable at the later training stage, unlike the relatively flat clean-label accuracy curve.Figure 3 compares class-wise test accuracy at epochs 10, 50, and 100 over 120 total epochs.
- Hard-class examples receive substantial probability on similar classes, and CE fails to form a visible cluster for class 3 under noise.Class similarity makes hard-class predictions diffuse across competing classes, weakening learned representations.
- Even among class 3’s 60% correct labels, confidence in the true class is only around 0.5, while other classes receive around 0.05 or 0.1.Hard classes also have fewer true-positive samples throughout intermediate training stages.
- CE alone is insufficient for hard-class learning under noisy labels, revealing both under learning and noisy-label overfitting.This finding differs from the view that DNNs generally overfit across all classes.
4. Symmetric Cross Entropy Learning
Symmetric Cross Entropy Learning (SL) combines CE with Reverse Cross Entropy (RCE) to balance convergence, sufficient learning, and robustness to noisy labels. The paper analyzes RCE's noise tolerance and SL's class-adaptive gradient behavior.
- RCE reverses the cross-entropy arguments, placing the ground-truth distribution inside the logarithm and requiring log 0 to be replaced by a negative constant A.The replacement addresses zero values for one-hot labels and supports the robustness analysis.
- SL combines Cross Entropy with Reverse Cross Entropy, using separate hyperparameters to balance CE's convergence benefits against noise robustness.The framework assigns α to CE's overfitting issue and β to flexible exploration of RCE's robustness.
- Under the paper's stated conditions, RCE is noise tolerant for symmetric noise and, with zero clean risk, for asymmetric class-dependent noise.For symmetric noise, the condition is η < 1 − 1/K; the asymmetric result uses the stated class-dependent noise-rate condition.
- RCE adds adaptive acceleration to learning: it increases the labeled-class probability more strongly below p_j = 0.5 and reduces unlabeled probabilities faster when labeled-class confidence is higher.The acceleration for unlabeled classes is proportional to confidence at the labeled class and disappears when that confidence is zero.
- Increasing CE's coefficient alone can intensify overfitting, whereas MAE is exactly a special case of RCE when A = −2 for a single true label.The paper contrasts this with SL's combination of two symmetrical learning terms rather than a simple CE rescaling or weighted MAE.
5. Experiments
The experiments evaluate SL across synthetic symmetric and asymmetric noise settings and the real-world Clothing1M dataset. SL improves class-wise learning, learned representations, and robustness, while its combination with existing methods further enhances performance.
- Experimental setup: SL is evaluated on MNIST, CIFAR-10, CIFAR-100, and Clothing1M under symmetric and asymmetric label noise.Symmetric noise uniformly flips labels, whereas asymmetric noise is class-dependent.
- Class-wise learning: SL achieves considerable improvement on each class compared with earlier CIFAR-10 results under noisy labels.The experiments use an 8-layer CNN and examine class-wise test accuracy.
- Representations: SL learns more separated and clearly bounded CIFAR-10 representation clusters than CE under 40% and 60% noisy labels.Representations are extracted from the second-last dense layer and projected into two dimensions with t-SNE.
- Parameter analysis: When α is too small, convergence becomes slow; with α = 0.1 controlling CE overfitting, SL is not sensitive to A.The sensitivity to A increases when CE overfitting is not properly addressed.
- Ablation study: Upscaling CE does not improve learning and can increase overfitting, whereas combining CE and RCE in SL drastically improves performance.RCE alone does not overfit when upscaled but converges slowly.
- Baselines and extensions: SL can be incorporated into Forward, D2L, and LSR, with the resulting methods still not as good as SL in some cases.The authors suggest adaptive incorporation and parameter tuning could further improve combined methods.
- Real-world noisy data: SL obtains the highest performance on the real-world Clothing1M noisy dataset compared with the evaluated baselines.Clothing1M contains 1 million clothing images with labels generated from surrounding online-shopping text and approximately 61.54% label accuracy.
6. Conclusions
The paper identifies CE's hard-class under learning under noisy labels and proposes SL, which combines CE with noise-robust RCE. Across benchmark and real-world datasets, SL addresses CE's under learning and overfitting problems and is simple to incorporate with other techniques.
- CE under learns hard classes when training DNNs with noisy labels.
- SL combines CE with noise-robust RCE to address hard-class under learning and noisy-label overfitting.
- SL is effective across various noise types and rates on benchmark and real-world datasets.
- Because SL is simple and easy to implement, it can be used alongside other techniques for datasets containing noisy labels.
A. Proof for Theorem 1
Theorem 1 establishes noise tolerance for RCE under symmetric noise below a stated rate and under asymmetric noise when the correct class remains sufficiently more probable than each incorrect class.
- Symmetric noise: Under symmetric noise, RCE is noise tolerant when the noise rate satisfies the theorem's stated bound.
- Asymmetric noise: Under asymmetric noise, RCE is noise tolerant when each incorrect-class transition probability is below the probability of the correct label, assuming R(f*) = 0.
- Interpretation: The asymmetric-noise condition corresponds to a noise-transition matrix that is diagonally dominated by its correct-class probabilities.
B. Gradient Derivation of SL
The appendix derives SL's gradient with respect to the logits by combining the cases for the target and non-target classes, including the special case of perfect target prediction.
- Gradient cases: The derivation combines separate gradient expressions for k = j and k ≠ j into the SL gradient.
- Combined expression: The derivation includes a term involving pk log qk − log qj.
- Perfect target prediction: When qj = qy = 1, the appendix gives the corresponding simplified SL gradient.