PaperlayerView exact revision on arXiv

Source-linked AI summary

Does label smoothing mitigate label noise?

Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv Kumar

arXiv:2003.02819v1cs.LGstat.ML

TL;DR

The paper studies whether label smoothing can cope with label noise, addressing limited systematic evidence and its apparent equivalence to injecting symmetric label noise. It connects smoothing to loss-correction techniques and evaluates its denoising and distillation effects. The results show competitive denoising performance and benefits from smoothing teachers trained on noisy data.

  • Problem

    Whether label smoothing mitigates label noise remains unclear because it may prevent overconfidence or inject uniform noise, and systematic study has been lacking.

  • Method

    The paper connects label smoothing to label-noise loss-correction techniques and analyzes its denoising role as regularisation.

  • Results

    Label smoothing is competitive with loss correction under label noise, and smoothing teachers trained on noisy data improves distilled students.

  • Takeaways & Limitations

    Label smoothing can be beneficial for noisy-label training and for distillation from teachers trained on noisy data.

  • Takeaways & Limitations

    The paper treats the smoothing parameter as a tuning parameter rather than using its theoretical optimal value.

Abstract

from arXiv · show

Label smoothing is commonly used in training deep learning models, wherein one-hot training labels are mixed with uniform label vectors. Empirically, smoothing has been shown to improve both predictive performance and model calibration. In this paper, we study whether label smoothing is also effective as a means of coping with label noise. While label smoothing apparently amplifies this problem --- being equivalent to injecting symmetric noise to the labels --- we show how it relates to a general family of loss-correction techniques from the label noise literature. Building on this connection, we show that label smoothing is competitive with loss-correction under label noise. Further, we show that when distilling models from noisy data, label smoothing of the teacher is beneficial; this is in contrast to recent findings for noise-free problems, and sheds further light on settings where label smoothing is beneficial.

1 Introduction

The paper asks whether label smoothing mitigates label noise despite competing intuitions that it may either prevent overconfidence or inject additional noise. It connects smoothing to loss correction and reports benefits under noisy-label training and distillation.

  • 1 Introduction: Label smoothing mixes one-hot labels with a uniform label vector and is commonly used to improve deep-learning performance and calibration.The paper frames smoothing as a form of regularisation that improves generalization and model calibration.
  • 1 Introduction: Label smoothing may mitigate noise by preventing overconfidence, but it may also accentuate noise because it injects uniform label noise.The paper presents these as competing intuitions motivating its empirical study.
  • 1 Introduction: A systematic study of whether label smoothing works for noisy labels was previously lacking, despite its similarity to popular loss-correction techniques.The paper identifies both a literature gap and a precise relationship requiring investigation.
  • 1 Introduction: The paper connects label smoothing to existing loss-correction techniques and finds it competitive for denoising under label noise.It further interprets smoothing through a regularisation view that complements the loss-correction perspective.
  • 1 Introduction: Label smoothing significantly improves performance under label noise, which the paper explains by relating smoothing to ℓ2 regularisation.This is presented as an empirical result alongside the theoretical connection to loss correction.
  • 1 Introduction: When distilling from noisy labels, smoothing the teacher improves the student, reversing findings that smoothing can harm distillation in noise-free settings.The contrast with prior noise-free findings is explicitly highlighted in the paper’s contributions.

2 Background and notation

The section introduces multiclass classification, class-conditional and symmetric label noise, loss correction, and knowledge distillation. The goal under noisy labels is to minimize clean-data risk despite observing corrupted labels.

  • Multiclass classification: Multiclass classification maps instances into one of L labels and seeks a predictor minimizing expected loss under the data distribution.
  • Learning under label noise: Label noise occurs when observed class probabilities differ from the ground-truth probabilities, making clean-risk minimization difficult.
  • Learning under label noise: Class-conditional noise models label flips with a row-stochastic transition matrix T, where T_y,y′ is the probability of changing y to y′.
  • Learning under label noise: Symmetric noise assumes a constant probability ρ of changing each label uniformly to one of the other classes.
  • Loss correction: Backward correction uses the inverse noise-transition matrix to produce an unbiased clean-risk estimate, whereas forward correction modifies predictions and is not unbiased.
  • Knowledge distillation: Knowledge distillation trains a student on teacher-generated class-probability estimates and can improve performance over training directly on the original sample labels.

3 Label smoothing meets loss correction

Label smoothing and loss-correction methods can be expressed through a common label-smearing framework, clarifying both their relationships and differing loss behavior. Smoothing modifies the effective class distribution while preserving classification consistency, unlike backward correction, which can recover clean probabilities under the assumed noise model.

  • General framework: Label smearing applies a matrix M to the vector of per-class losses, potentially combining losses for all possible labels.
  • Special cases: The framework includes standard training, label smoothing, and backward correction as special cases defined by different choices of M.
  • Special cases: Label smoothing uses M = (1−α)·I + α/L·J, mixing the true label with a uniform distribution over classes.
  • Statistical consistency: Minimizing a smeared loss is equivalent to minimizing the original loss on modified class probabilities p_SM(x) = M^T p*(x).
  • Statistical consistency: Under label smoothing, maximal class probability is preserved for α < 1, so smoothing remains classification-consistent.
  • Statistical consistency: Under class-conditional noise, backward correction maps noisy probabilities back to clean probabilities when M = T^-1.
  • Loss behavior: Label smoothing and backward correction combine identity and all-ones matrices differently, by addition versus subtraction, producing fundamentally different losses.
  • Loss behavior: Forward correction smooths logits rather than labels, making the loss bounded for all predictions.

4 Effect of label smoothing on label noise

Under symmetric label noise, label smoothing improves accuracy and denoises noisy examples while reducing confidence in noisy labels; it remains competitive with loss correction and can preserve calibration at suitable smoothing levels.

  • Accuracy: Choosing α much larger than the true noise rate improves performance, making α a tuning parameter rather than simply the theoretically optimal noise-rate value.The paper contrasts α ≫ ρ∗ with the approximate theoretical choice α ≈ ρ∗ for loss correction.
  • Accuracy: Label smoothing is often competitive with loss correction, although loss correction generally achieves the best overall accuracy with high α.Forward correction outperforms label smoothing in the reported comparison, while backward correction varies with α.
  • Denoising: Increasing α improves accuracy on both clean and noisy training examples, with a larger improvement on the noisy portion.The results indicate that smoothing helps classify some noisy examples neither as the correct class nor as their observed noisy class without smoothing.
  • Denoising: Label smoothing reduces confidence in noisy labels and makes noisy instances’ pre-logits more uniform, whereas correction methods increase confidence, especially for true labels.Smoothing reduces confidence in both noisy and correct labels overall, while forward and backward correction sharpen confidence in both.
  • Calibration: Label smoothing improves calibration over baseline at α = 0.1, but larger α values worsen calibration, while loss correction degrades it relative to smoothing.The paper’s overall summary characterizes smoothing as particularly successful in denoising while preserving calibration.
  • Label smoothing as regularisation: The paper connects label smoothing to loss-correction techniques but leaves a full theoretical combination of the loss and regularisation perspectives as future work.The smoothing loss is biased, unlike unbiased backward correction, despite smoothing’s empirical denoising success.
  • Label smoothing as regularisation: A regularisation analysis explains denoising through shrinkage: label smoothing can shrink weights, increase margins, and move a noisy-data decision boundary toward the Bayes-optimal separator.For centered data, the smoothing effect reduces to weight shrinkage; an explicit L2 regulariser has a similar effect.

5 Distillation under label noise

Under label noise, denoising the teacher before distillation improves student performance, including when denoising uses label smoothing. This reverses the reported noise-free setting, where smoothing the teacher hurts the student.

  • Motivation and contrast: The noisy-data result contrasts with noise-free distillation, where smoothing improves the teacher but hurts the student by erasing relative information in teacher logits.The paper investigates whether noisy-data smoothing can instead transfer denoising effects to the student.
  • Motivation and contrast: The experiments test whether teacher smoothing can transfer its denoising effect to a student despite possible information erasure in the teacher logits.The question is motivated by smoothing’s denoising effect on noisy-data models and its information-erasing effect on clean-data teachers.
  • Experimental design: The study trains noisy-label teachers without correction, with label smoothing, or with forward correction, then distills each into a same-complexity student.A separate setup distills a vanilla teacher while applying smoothing or forward correction to the student.
  • Distillation results: Distilling a teacher trained with noisy labels, label smoothing or forward correction significantly improves students over vanilla distillation on CIFAR-10 and CIFAR-100.Applying smoothing or correction only to the student has mixed results: benefits appear on CIFAR-10 but essentially disappear on CIFAR-100.
  • Distillation results: For all tested smoothing strengths α, smoothing the teacher improves student accuracy over a plain teacher with α = 0.The comparison uses CIFAR-100 with ResNet-32 teacher and student models at distillation temperature T = 1.

6 Conclusion

The paper finds that label smoothing is effective for coping with label noise and beneficial when distilling from noisy data. It connects smoothing to loss correction and regularisation while identifying denoising theory as a direction for future work.

  • Main findings: Label smoothing is competitive with existing loss-correction techniques and exhibits strong denoising effects under label noise.The conclusion presents this as an empirical result of the study.
  • Theoretical interpretation: The paper theoretically relates label smoothing to a loss-correction technique and reinterprets smoothing as regularisation.This connection provides the paper’s theoretical framing for its denoising behavior.
  • Distillation: When distilling models from noisy data, smoothing the teacher is beneficial.The conclusion identifies teacher smoothing as beneficial specifically in noisy-data distillation.
  • Future work: The authors suggest formal exploration of label smoothing’s denoising properties as an interesting topic for future work.This is presented as a future research direction rather than a completed result.

A Proof of Theorem 1

For linear models, the paper uses convexity and differentiability of the regularisation objective to characterize its minimizer through a vanishing gradient.

  • Proof strategy: For linear models, Ω(f) is convex in Wy′, so its minimizer can be found by solving where the gradient vanishes.The proof then evaluates the gradient at Wi = 0 for every i.
  • Proof strategy: The proof swaps differentiation and expectation because Ω(f) is differentiable in both Wi and x.This supports the subsequent gradient calculation.
  • Stationary point: The gradient is shown to evaluate to zero at Wi = 0 for all i.This is the proof’s stated stationary-point calculation.

B.1 Architecture

The experiments use batch-normalized ResNet architectures, with configurations varying by dataset and listed in terms of layers, filters, and stride for each block.

  • Model family: The experiments use ResNet architectures with batch normalization.The cited implementation passage identifies ResNet with batch norm as the experimental model family.
  • Dataset configurations: CIFAR-10 and CIFAR-100 experiments use ResNet-32 and ResNet-56, while ImageNet experiments use ResNet-v2-50.These are the dataset-specific architecture choices reported by the paper.
  • Configuration notation: Architecture configurations are specified per ResNet block using the tuple (nlayer, nfilter, stride).The configurations are listed in Table 6.

B.2 Training

CIFAR-10 and CIFAR-100 experiments use mini-batches of 128 for 64k steps with Nesterov SGD, while ImageNet uses large-batch LARS training.

  • CIFAR-10 and CIFAR-100 use mini-batches of 128 and train for 64k steps.Training uses stochastic gradient descent with Nesterov momentum of 0.9.
  • The CIFAR learning rate starts at 0.1 and drops by a factor of 10 at 32k and 48k steps.Weight decay is set to 1e-4.
  • ImageNet uses ResNet-v2-50 with LARS, a batch size of 3500, and 32 training steps.
  • Distillation experiments train only against the teacher’s logits using cross-entropy and a temperature of 2 unless otherwise specified.
  • CIFAR experiments run on 4 TPU v2 chips, while ImageNet uses 128 TPU v3 chips.CIFAR training takes under 15 minutes and ImageNet training around 1.5 hours.

C Experiments: additional results

Additional experiments visualize how label smoothing and loss-correction methods alter logits and pre-logits under noisy-label settings across CIFAR datasets and ResNet models.

  • With 20% label-flip noise and α = 0.1 smoothing, Table 7 evaluates label-smearing results using Patrini matrix estimates.The estimates use logits from examples in the 99.9th activation percentile for each label.
  • Label smoothing reduces confidence, whereas backward and forward correction increase it, especially for the correct-label logit.The comparison uses density plots of maximum or true/noisy-label logits against average logits.
  • Figure 8 visualizes pre-logits for three classes on CIFAR-100 with ResNet-56 and CIFAR-10 with ResNet-32.The plots use the procedure of Müller et al.; black markers indicate instances mislabeled by noise.
  • Smoothing makes noisy instances’ pre-logits more uniform, indicating a denoising effect in the visualizations.
  • Figure 9 visualizes backward and forward correction logits for three CIFAR-100 classes, with black points marking label-noisy instances.Both correction methods are reported to denoise noisy instances.
Loading 2003.02819v1…