Source-linked AI summary
Robust Loss Functions under Label Noise for Deep Neural Networks
Aritra Ghosh, Himanshu Kumar, P. S. Sastry
TL;DR
Noisy labels challenge deep-network training, motivating loss functions that tolerate noise without changing architectures or algorithms. The paper derives multiclass sufficient conditions and finds that MAE is robust, with experiments showing much smaller accuracy degradation than CCE under label noise.
Problem
Deep networks require large labeled datasets, but labels from crowdsourcing, the web, and subjective annotation can be unreliable.
Method
The paper derives sufficient conditions for multiclass noise-tolerant losses and evaluates MAE, MSE, and CCE under label noise.
Results
MAE satisfies the sufficient conditions and shows greater robustness than CCE, retaining near-zero-noise MNIST test accuracy even under 80% symmetric label noise.
Takeaways & Limitations
Robust loss functions can support multiclass risk minimization with noisy labels without changing the learning algorithm or network architecture.
Takeaways & Limitations
The distribution-independent robustness bound may be loose at high noise rates.
Abstract
from arXiv · showhide
In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.
Introduction
Large-scale deep-network classification requires extensive labeled data, but labels can be unreliable because of human, measurement, crowdsourcing, and web-based errors. This paper studies risk minimization under multiclass label noise, deriving sufficient conditions for noise-tolerant losses and showing that MAE satisfies them.
- Deep neural networks achieve impressive classification performance but require very large training datasets, making accurate labeling a serious challenge.
- Labels become noisy through human labeling errors, measurement errors, subjective bias, crowdsourcing, and automatic collection from web information.
- The paper frames robustness to label noise as risk minimization and emphasizes that robustness depends on the loss function.Risk minimization underlies Bayes classification with 0–1 loss and backpropagation-based neural-network learning with losses such as squared error or cross entropy.
- The paper derives sufficient conditions for noise-tolerant loss functions in multiclass classification and shows that mean-absolute error satisfies them.It also presents empirical investigations comparing robustness.
- Unlike data cleaning, heuristic algorithm modification, or hidden-label estimation, inherently noise-tolerant methods seek robustness from algorithmic properties without estimating true labels.The paper notes that existing approaches may perform well but do not guarantee robustness to label noise.
Preliminaries and Problem Statement
This section formalizes multiclass classification with clean and corrupted labels, defines loss-based risk minimization, and specifies noise models and noise tolerance. It distinguishes symmetric, class-conditional, non-uniform, and simple noise settings.
- Classification setup: The framework draws iid examples from an unknown distribution over feature space X and labels Y = [k], representing classifiers as h(x) = pred ◦ f(x).The score function maps X to C ⊆ R^k, while h maps X to the class labels Y.
- Risk minimization: A loss function maps C × Y to nonnegative real values, and its L-risk is minimized globally; with 0–1 loss, this risk is the usual Bayes risk.The risk depends on the chosen loss function and is defined using expectation over the relevant random variables or distribution.
- Noise model: Label noise replaces clean labels with corrupted labels, producing a noisy distribution Dη distinct from the clean distribution D.The true label is denoted yx, the corrupted label is ˆyx, and the total corruption rate at x is ηx.
- Noise models: Symmetric noise uses constant η and assigns η/(k−1) probability to every incorrect class.This definition applies uniformly across examples and incorrect labels.
- Noise models: Class-conditional noise depends on the true class, whereas non-uniform noise allows corruption rates to depend on x; simple non-uniform noise distributes errors uniformly among incorrect classes.Fixed per-class rates define the special case called simple class conditional noise.
- Noise tolerance: Risk minimization is noise-tolerant when a noisy-risk minimizer has the same misclassification probability as the clean-risk minimizer.The paper also states that equality of the corresponding minimizers is sufficient for noise tolerance.
Theoretical Results
The paper gives distribution-independent sufficient conditions under which multiclass risk minimization remains robust to symmetric, simple non-uniform, and class-conditional label noise. Symmetric losses guarantee uniform-noise tolerance, while additional risk and noise-rate conditions handle non-uniform settings.
- Symmetric and uniform noise: For symmetric or uniform label noise, a loss satisfying Eq 2 is noise tolerant when η < k−1.Theorem 1 states this sufficient condition for multiclass classification.
- Symmetric and uniform noise: Symmetric-loss robustness to uniform label noise is distribution independent and generalizes the existing two-class result.The only stated condition is the noise-rate bound η < k−1.
- Simple non-uniform noise: Under simple non-uniform noise, Eq 2 is sufficient when RL(f ∗) = 0 and ηx < k−1 for every x.When the minimum clean risk is positive, Theorem 2 instead provides a bound involving the maximum noise rate ηmax.
- Simple non-uniform noise: The positive-risk case remains useful when classes are non-separable, although its distribution-independent bound may be loose at high noise rates.For 0–1 loss, zero clean risk corresponds to separability; general losses may have nonzero optimal risk even for separable classes.
- Class-conditional noise: Under class-conditional noise, robustness requires Eq 2, zero clean risk, bounded losses, and a diagonally dominant label-noise matrix.The theorem’s noise-rate condition is stricter, implying ηyx < (k−1)/k.
- Loss-function implications: Among categorical cross entropy, mean square error, and mean absolute error, only MAE satisfies the symmetry condition, while MSE is bounded and more robust than unbounded CCE.MAE and MSE are also described as classification calibrated.
Classification Calibrated Losses
The section establishes that MAE and MSE are classification calibrated in multiclass settings, alongside previously established logistic and exponential losses. It also proves that ERM with label-noise-robust losses is uniformly consistent under symmetric noise over finite-VC function classes.
- Classification Calibration: Logistic and exponential losses are classification calibrated in multiclass settings, and the paper shows that MAE and MSE share this property.These results extend the set of commonly used losses known to satisfy classification calibration.
- Classification Calibration: For MAE and MSE, the unique minimizer is pi = ui, making conditional surrogate regret positive whenever predictions disagree with the true class probabilities.The disagreement set is non-empty unless pi = 1/k for every k.
- Consistency of ERM under Label Noise: Under symmetric label noise, ERM with a noise-robust loss uniformly converges to the noise-free minimizer’s error rate over any finite-VC-dimension function class.The theorem concerns empirical-risk minimizers from noisy samples and compares them with risk minimizers under the noise-free distribution.
- Consistency of ERM under Label Noise: The consistency argument combines distribution-independent VC bounds with the derived transformation of error rates under symmetric label noise.The proof uses the robustness condition on the loss to equate the noisy and noise-free minimizer error rates.
Empirical Results
Experiments across image and text datasets evaluate CCE, MSE, and MAE under symmetric and class-conditional label noise. MAE is highly robust, retaining near-clean-data test accuracy even with 80% symmetric noise and showing fair robustness under class-conditional noise.
- Experimental setup: Experiments evaluate CCE, MSE, and MAE using backpropagation across two image and four text datasets with varied label-noise rates.Noise is added symmetrically or through class-conditional probability matrices, and results average six runs.
- Experimental setup: Networks use dataset-specific architectures with ReLU hidden layers and a softmax output layer.Architectures vary because the image and text datasets have different feature-space dimensions.
- Symmetric noise: MAE remains highly robust to symmetric noise, achieving test accuracy close to its zero-noise performance even at 80% noise.Figure 1 compares MAE and CCE on MNIST and RCV1 using η = 0, 0.4, and 0.8.
- Noise-rate comparison: Under 60% noise, MAE drops very little on datasets such as MNIST and RCV1 when its zero-noise accuracy is high.Table 2 reports average test accuracy and standard deviation over six runs for multiple noise rates and class-conditional noise.
- Loss-function comparison: MSE provides intermediate robustness between MAE and CCE, while MAE also shows fair robustness under class-conditional noise.Higher dropout rates reduce CCE’s sensitivity to label noise, consistent with dropout acting as a regularizer.
Conclusion
The paper derives general theoretical results showing that robust loss functions can support multiclass classifier learning under noisy labels without modifying the algorithm or network architecture. These results apply to any multiclass classifier trained through risk minimization.
- The paper derives theoretical results on the robustness of loss functions for multiclass classification.
- Robust loss functions can enable learning a good classifier when training labels are noisy, without changing the algorithm or network architecture.
- Although discussed for neural networks, the theoretical results apply generally to multiclass classifiers learned through risk minimization.