Normalized Loss Functions for Deep Learning with Noisy Labels

Xingjun Ma; Hanxun Huang; Yisen Wang; Simone Romano; Sarah Erfani; James Bailey

Source-linked AI summary

Normalized Loss Functions for Deep Learning with Noisy Labels

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, James Bailey

arXiv:2006.13554v1Jun 24, 2020cs.LGcs.CVstat.ML

TL;DR

Noisy labels make commonly used CE unsuitable, while existing robust losses are only partially effective. The paper proves that simple normalization can make any loss noise-robust, then proposes APL to address underfitting by combining complementary loss terms. On benchmarks, APL consistently outperforms state-of-the-art methods, especially at high noise rates.

Problem
Cross Entropy is not robust to noisy labels, and existing robust loss functions do not fully ensure accurate DNN training.
Method
The paper normalizes arbitrary losses for theoretical robustness and combines active and passive loss terms in the APL framework.
Results
APL losses consistently outperform state-of-the-art methods on benchmark datasets, with especially large gains under 60% or 80% noise.
Takeaways & Limitations
Robust loss design should address both theoretical noise tolerance and sufficient learning, rather than robustness alone.
Takeaways & Limitations
The theoretical guarantees require noise conditions generally leaving correct labels in the majority, and asymmetric-noise analysis additionally assumes zero clean risk.

Abstract

from arXiv · show

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.

1. Introduction

Robust loss functions offer a simple approach to learning with noisy labels, but existing methods have limitations. The paper shows that normalization can make any loss robust, identifies underfitting in robust losses, and proposes APL to improve learning and performance.

Motivation: Robust loss functions provide a simpler solution than label correction, loss correction, and refined training strategies for noisy-label learning.The alternative approaches may require inaccurate noise estimation or sophisticated training modifications.
Motivation: Cross Entropy is not robust to noisy labels, motivating the design of inherently robust alternatives.Mean Absolute Error is theoretically robust, while Generalized Cross Entropy is robust only when reduced to Mean Absolute Error.
Contributions: A simple normalization can theoretically make any loss function robust to noisy labels.This result provides the paper’s central theoretical insight.
Contributions: Existing robust loss functions suffer from underfitting, so robustness alone does not guarantee accurate DNN training.The paper addresses this limitation through a framework that combines complementary loss behaviors.
Contributions: APL combines active and passive loss components to create robust losses with improved learning properties.The framework is designed to retain robustness while addressing insufficient learning.
Contributions: APL losses outperform state-of-the-art methods by considerable margins, especially with 60% or 80% incorrect labels.The paper reports this result empirically on benchmark datasets.

2. Related Work

Prior noisy-label methods include label correction, loss correction, refined training strategies, and robust loss functions. The paper focuses on robust losses as a simpler and more generic alternative, while introducing normalization as a broader design principle.

Label correction: Label correction methods seek to identify and correct wrong labels but may require extra clean data or expensive noise-model estimation.They use complex models such as graphical models, conditional random fields, neural networks, or knowledge graphs.
Loss correction: Loss correction methods modify the training loss using label-dependent weights or an estimated noise transition matrix.Some methods augment the network with an additional linear layer.
Refined training strategies: Refined training strategies adapt sample weighting, network interactions, labels, or supervision to improve robustness.Several methods require auxiliary networks or complex intervention procedures.
Robust loss functions: Robust loss functions are simpler and more generic, but MAE can converge slowly because of gradient saturation, while CE is not noise-robust.GCE and other robust-loss designs address parts of this trade-off without resolving it completely.
Paper contribution: The paper proposes simple normalization as a basic principle that can make any loss robust and shift loss design toward properties beyond robustness.This theoretical insight broadens the design space for robust losses.

3. Any Loss can be Robust to Noisy Labels

The paper formalizes noisy-label classification and reviews common losses before showing that normalization makes losses noise tolerant under stated conditions. It also identifies that robustness alone can cause underfitting, motivating further loss design.

Preliminaries: The noisy-label setting assumes labels are conditionally independent of inputs given the true labels, with symmetric or class-conditional noise.The overall noise rate is η, while η_jk denotes the class-wise rate from true class j to mislabeled class k.
Preliminaries: The classifier outputs class probabilities from network logits and is trained by minimizing empirical loss over labeled samples.The labeled target may be incorrect, and the label distribution is represented as one-hot in the common single-label case.
Existing losses: MAE and RCE are robust to label noise, whereas CE and FL are not robust under the cited theoretical characterization.RCE can be combined with CE to form Symmetric Cross Entropy.
Normalized losses: Normalization constrains a loss to [0, 1] and produces normalized variants such as NCE, NMAE, NRCE, and NFL.For MAE and RCE, normalization yields scaled versions that preserve robustness; for CE and FL, it derives new losses.
Theoretical justification: Any normalized loss is noise tolerant under symmetric noise when η < (K−1)/K, and under asymmetric noise when η_jk < 1−η_y under additional assumptions.The asymmetric result assumes zero clean risk and bounded normalized loss values.
Theoretical justification: The theoretical noise-rate conditions generally require correct labels to remain the class majority, while the zero-risk assumption may fail for nonseparable classes.The authors note that empirical robustness may still be achieved in practice.
Theoretical justification: Robustness alone is insufficient for good performance because normalized losses can underfit during training.This motivates examining additional learning properties beyond noise tolerance.

4. Robustness Alone is not Sufficient

Robustness alone does not ensure accurate training: normalized and inherently robust losses can underfit, motivating APL’s combination of complementary active and passive terms.

Robustness Alone is not Sufficient: Four tested robust losses—NCE, NFL, MAE, and RCE—suffer from underfitting and are insufficient by themselves for accurate DNN training.The issue persists across learning rates, schedulers, weight decay, and training epochs.
Robustness Alone is not Sufficient: On CIFAR-100 with 0.6 symmetric noise, normalized CE and FL become robust without becoming more accurate, while NCE and NFL perform worse than CE and FL.Unnormalized MAE and RCE even fail to converge in this setting.
Active and Passive Losses: Active losses explicitly maximize the labeled-class probability, whereas passive losses also explicitly minimize probability at one or more other class positions.CE exemplifies an active loss, while MAE exemplifies a passive loss.
Proposed Active Passive Loss (APL): APL combines robust active and passive losses, with the two terms optimizing the same objective from complementary directions.Examples include αNCE + βMAE, αNCE + βRCE, αNFL + βMAE, and αNFL + βRCE.
Proposed Active Passive Loss (APL): Normalization can make any loss noise robust, and Lemma 3 shows that weighted combinations of noise-tolerant active and passive losses remain noise tolerant.The weights α and β are positive parameters balancing the two terms.
More Insights into APL Loss Functions: APL addresses underfitting because its passive term explicitly minimizes components that normalization can otherwise allow to increase while the labeled-class probability remains fixed.This is intended to combine robustness with convergence advantages.

5. Experiments

Experiments show that normalization makes non-robust losses noise-tolerant, while APL combines complementary robust losses to address underfitting and achieve strong performance across noisy-label benchmarks.

Normalized losses are robust: Normalization removed the overfitting observed in CE and FL under 0.6 symmetric noise, while preserving robustness for MAE and RCE.The same pattern held across MNIST, CIFAR-10, and CIFAR-100 with noise rates η ∈[0.2, 0.8].
Scaling and sufficient learning: Scaled NCE and NFL only slightly improved learning on CIFAR-100, whereas proper scaling helped on CIFAR-10 but could be replaced by learning-rate adjustment.The limited benefit was especially apparent on the more challenging CIFAR-100 dataset.
APL losses: APL combines active and passive losses, yielding a clear advantage over active-plus-active and passive-plus-passive combinations, especially for high accuracy.The four evaluated combinations were NCE+MAE, NCE+RCE, NFL+MAE, and NFL+RCE; α and β were set to 1.0 without tuning in this comparison.
Benchmark evaluation: On CIFAR-10 with 0.6 symmetric noise, NFL+RCE and NCE+RCE exceeded NLNL’s 72.85% accuracy by more than 6%.On CIFAR-100 with 0.8 symmetric noise, NCE+MAE and NCE+RCE exceeded GCE and NLNL by at least 9%, while CE and FL failed to converge.
Benchmark evaluation: APL methods achieved the top two results across all symmetric- and asymmetric-noise test scenarios, with the best combination varying by dataset but remaining consistent across noise types and rates within each dataset.For CIFAR-100 with 0.4 asymmetric noise, NCE+MAE was 5% and NCE+RCE 4% above SCE’s 42.19%.
Overall findings: NCE+RCE showed consistently strong performance across datasets, supporting APL as a general principle for developing robust loss functions.The experiments covered MNIST, CIFAR-10, CIFAR-100, and the real-world noisy WebVision dataset.

6. Conclusions

The paper concludes that normalization can make any loss robust, but robustness alone does not ensure accurate training. APL addresses underfitting by combining active and passive losses and performs strongly against state-of-the-art methods.

A simple normalization can make any loss function robust to noisy labels.
Existing robust losses can underfit, so robustness alone is insufficient for training accurate DNNs.
APL mutually boosts active and passive losses to provide theoretical robustness together with effective sufficient learning.
Experiments verified excellent APL performance compared with state-of-the-art methods on benchmark datasets.

A. Proofs for Lemma 1, Lemma 2 and Lemma 3

The proofs establish that normalized losses preserve clean-risk minimizers under symmetric and asymmetric label noise, and that APL inherits this tolerance when combining noise-tolerant active and passive losses.

Lemma 1: Any normalized loss is noise tolerant under symmetric label noise when η < K−1.The proof shows noisy and clean risks differ by a positive scaling factor, preserving the global minimizer.
Lemma 2: Under asymmetric label noise, normalized losses remain noise tolerant if η_yk < 1 − η_y and R(f*) = 0.The condition requires the correct class to retain higher probability than each specified incorrect class.
Lemma 3: APL is noise tolerant whenever its active and passive component losses are each noise tolerant, for arbitrary coefficients α and β.The proof combines their noise risks as α·CActive + β·CPassive = C.
Proof strategy: The proofs rely on showing that the clean global minimizer also minimizes the corresponding noisy risk.This establishes robustness by comparing Rη(f*) − Rη(f) with the clean-risk difference.
Lemma 2: The asymmetric-noise proof concludes that the clean-risk minimizer is also a noisy-risk minimizer for normalized loss.The argument uses the bounded-loss condition 0 ≤ Lnorm(f*(x), k) ≤ 1/(K−1).