PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels

Zhilu Zhang, Mert R. Sabuncu

arXiv:1805.07836v4cs.LGcs.CVstat.ML

TL;DR

Noisy labels can impair DNN classification, while MAE may be difficult to optimize on challenging datasets. This paper develops generalized MAE/CCE loss functions and reports improved accuracy across several datasets and both closed- and open-set noise scenarios.

  • Problem

    Noise-robust loss functions for DNNs remain limited, while MAE can be difficult to optimize and reduce performance on challenging datasets.

  • Method

    The paper proposes theoretically grounded Lq and truncated Lq losses that generalize MAE and CCE for closed- and open-set noisy labels.

  • Results

    The proposed losses improved classification accuracy over MAE and CCE baselines across CIFAR-10, CIFAR-100, and FASHION-MNIST under closed- and open-set noise.

  • Takeaways & Limitations

    Lq and truncated Lq provide broadly applicable noise-robust losses for DNN classification with closed- and open-set noisy labels.

  • Takeaways & Limitations

    The approach depends on correctly labeled data being distinguishable from mislabeled data through softmax outputs, and its results can improve with optimized q and k values and more training epochs.

Abstract

from arXiv · show

Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100 and FASHION-MNIST datasets and synthetically generated noisy labels.

1 Introduction

Deep neural networks require many training samples, while noisy labels can undermine their performance. The paper proposes a theoretically analyzed generalization of CCE and evaluates it against MAE and CCE across multiple datasets and noise settings.

  • Motivation: DNN classification often requires large numbers of training samples, including millions of hand-annotated images in datasets such as ImageNet.ImageNet is cited as containing 3.2 million hand-annotated images.
  • Motivation: Although MAE can be robust to noisy labels under certain assumptions, that robustness can make DNN training more difficult and reduce performance.The paper identifies increased training difficulty as a limitation of MAE’s noise robustness.
  • Contributions: The proposed loss functions can be promptly applied with existing algorithms and support both closed-set and open-set noisy labels.Closed-set errors come from known dataset labels, whereas open-set errors may not belong to any known training class.
  • Contributions: The paper proposes a novel generalization of CCE and theoretically analyzes its loss functions for learning with noisy labels.The proposed losses are intended to address noise robustness directly in the DNN setting.
  • Contributions: Experiments on CIFAR-10, CIFAR-100, and FASHION-MNIST report improved classification accuracy over MAE and CCE under both closed-set and open-set noise.The empirical evaluation is described as thorough and compares against MAE and CCE baselines.

2 Related Work

Related work addresses noisy labels through confusion-matrix correction, theoretically motivated robust losses, label-cleaning and sample-reweighting strategies, latent-variable modeling, and knowledge distillation. Several approaches require estimating a confusion matrix or access to a small clean dataset, while alternative robust losses for DNNs have received limited attention.

  • Confusion-matrix correction: Confusion-matrix methods modify cross-entropy to account for noisy labels, but the true matrix is usually unknown and difficult to estimate.Even with the real confusion matrix, training with the resulting loss may be suboptimal for sufficiently capacious DNNs.
  • Noise-robust losses: Robust losses including ramp, unhinged, and savage losses were developed for support vector machines, while Natarajan et al. generalized noise-robust surrogate-loss modification to binary classification.The passage notes that alternative noise-robust loss functions for DNNs have received little attention.
  • Label cleaning: Label-cleaning approaches use a parallel label-cleaning network, gradually replace noisy labels with predictions, or prune samples judged correct from softmax outputs.The parallel-network method requires a small set of clean labels.
  • Latent modeling and reweighting: Other methods model true labels as latent variables for EM-like learning or re-weight confident samples using an LSTM or a small clean dataset.The cited reweighting approaches learn sample weights or emphasize samples using clean data.
  • Knowledge distillation: Knowledge distillation has also been proposed for noisy labels, and the discussed reweighting and distillation methods require a smaller clean dataset.The clean-data requirement applies to both methods described across the passages.

3 Generalized Cross Entropy Loss for Noise-Robust Classifications

The section motivates generalized cross entropy loss by showing why CCE overemphasizes potentially noisy, difficult samples while MAE can impede learning, then introduces Lq and its truncated variant to balance robustness and learnability. The proposed losses interpolate between CCE and MAE, with theoretical bounds connecting larger q or threshold k to stronger noise tolerance under stated conditions.

  • Motivation: CCE emphasizes difficult samples through inverse-probability weighting, which can cause overfitting to noisy labels, whereas MAE’s robustness can make useful features harder to learn.Experiments on CIFAR datasets found that MAE took significantly longer to converge and compromised classification accuracy relative to CCE.
  • Generalized cross entropy: Lq uses the negative Box-Cox transformation with q ∈ (0, 1], recovering CCE as q approaches 0 and MAE/unhinged loss at q = 1.Thus, Lq generalizes both commonly used losses while retaining a tunable parameter for the robustness–learnability trade-off.
  • Theoretical robustness: For q values approaching 1, the noisy-risk optimum has a clean-data risk close to the clean-risk optimum, implying increasing noise tolerance; at q = 1, the bound constant is A = 0.Under class-dependent noise, a related bound holds when RLq(f ∗) = 0 and qij < qii for i ≠ j.
  • Generalized cross entropy: Relative to CCE, Lq downweights weak label–prediction agreement, while relative to MAE it preserves greater emphasis on challenging datapoints.Its gradient applies an additional fyi(xi; θ)^q weighting relative to CCE and fyi(xi; θ)^(q−1) relative to MAE.
  • Truncated Lq: Truncated Lq introduces a threshold k to tighten the loss bounds and improve noise tolerance, with the inequality satisfied for all q and c when k ≥ 0.3.When the provided-label softmax output falls below k, the loss becomes constant and its gradient is zero, so the sample is discarded from learning dynamics.
  • Truncated Lq: The truncated objective can be optimized with binary sample weights, and ACS alternately optimizes model parameters and weights when Lq is convex in the parameters.The optimal weight is 1 for losses at or below the threshold and 0 otherwise.

4 Experiments

Experiments across CIFAR-10, CIFAR-100, and FASHION-MNIST show that Lq-based losses improve robustness to noisy labels, especially by delaying overfitting, while truncation and q selection affect performance. The method outperformed baselines under uniform and open-set noise, but MAE performed poorly across datasets.

  • Experimental setup: Experiments used artificially corrupted training and validation labels, true-label test accuracy, 10% validation data, mini-batches of 128, and ReLU hidden layers.
  • Lq behavior: Increasing q delayed overfitting and improved classification accuracy on noisy data, although it slowed convergence and reduced accuracy on clean data.With CCE (q = 0), models first learned predictive patterns before strongly overfitting noisy labels.
  • Lq behavior: At η = 0.2, q = 0.8 produced an overfitting-free training process, but excessively large q values harmed classification accuracy.The q threshold increased with noisier data, making q a tunable hyper-parameter.
  • Closed-set noise: Under closed-set noise, proposed losses significantly outperformed baselines for uniform noise, while truncated Lq loss improved results and approached Forward T for class-dependent noise.Forward T generally performed best for class-dependent noise because it used the ground-truth confusion matrix.
  • Limitations: Classification accuracy could improve further with optimized q and k values and more epochs, whereas MAE performed poorly across all datasets.The approach is expected to work when softmax outputs distinguish correctly from incorrectly labeled data.
  • Open-set noise: For CIFAR-10 open-set noise, Lq noticeably prevented overfitting, and truncated Lq loss outperformed the reported state-of-the-art method while being easier to implement.The open-set experiment used CIFAR-100 images assigned random CIFAR-10 labels and trained for 100 epochs.

5 Conclusion

The paper proposes theoretically grounded, easy-to-use noise-robust loss classes—Lq and truncated Lq—for noisy-label classification with existing DNN algorithms, and empirically verifies their robustness across datasets and noise scenarios.

  • 5 Conclusion: The Lq loss and truncated Lq loss are theoretically grounded, easy-to-use noise-robust functions applicable with any existing DNN algorithm.They are proposed for classification with noisy labels.
  • 5 Conclusion: Noise robustness was empirically verified across various datasets under both closed-set and open-set noise scenarios.

Appendix

The appendix establishes theoretical properties of the proposed losses, including their connection to categorical cross entropy, bounded classwise sums, and robustness under specified noise conditions. It also derives bounds for truncated Lq loss and notes that these bounds support noise-robust training.

  • Theoretical properties: As q→0, the Lq loss converges to categorical cross entropy loss.Lemma 1 states limq→0 Lq(f(x), ej) = LC(f(x), ej).
  • Theoretical properties: For q ∈(0, 1], the sum of Lq loss over all classes is bounded.This is stated in Lemma 2 for any x.
  • Noise robustness: When q = 1, A = 0, and f ∗ is also a minimizer of risk under uniform noise.This is given as a remark following the uniform-noise result.
  • Noise robustness: Under class-dependent noise satisfying ηij < (1 −ηi) for all j≠i, if RLq(f ∗) = 0, the theorem guarantees the stated robustness property.The condition uses ηij = p(ey = j|y = i) and (1 −ηi) = p(ey = i|y = i).
  • Truncated loss: For 1/c ≤k < 1 and q ∈(0, 1), the sum of truncated Lq loss over all classes has the bounds stated in Lemma 3.The bound uses ˜d = max(1, (1−q)1/q k ).
  • Truncated loss: The truncated loss is asserted to yield more noise-robust training by applying the same arguments as in Theorems 1 and 2.This conclusion is stated in the remark following Lemma 3.
Loading 1805.07836v4…