Source-linked AI summary
When Does Label Smoothing Help?
Rafael Müller, Simon Kornblith, Geoffrey Hinton
TL;DR
Label smoothing is widely used, yet its effects and underlying mechanisms remain poorly understood. This paper empirically studies its calibration, distillation, and representation consequences using experiments and penultimate-layer visualizations. It finds that smoothing improves calibration but can impair distillation by erasing information about similarities among examples and classes.
Problem
Label smoothing is widely used, but the reasons for and conditions of its effects on neural-network performance remain poorly understood.
Method
The paper combines empirical studies of calibration and knowledge distillation with visualizations of penultimate-layer representations and analyses of logit information.
Results
Label smoothing improves calibration and generalization-related performance, but teachers trained with smoothing produce inferior student networks because information in relative logits is erased.
Takeaways & Limitations
Label smoothing creates tight, equally separated class clusters that support calibration but remove inter-class resemblance information needed for effective distillation.
Abstract
from arXiv · showhide
The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.
1 Introduction
Label smoothing is widely used to improve neural-network performance, but its benefits and underlying behavior remain incompletely understood. The paper introduces a visualization approach and examines how smoothing changes training objectives and learned representations.
- The paper studies label smoothing alongside earlier loss functions, including quadratic loss and cross entropy, as objectives affecting classification performance and convergence.
- Label smoothing has improved deep-learning accuracy across image classification, speech recognition, and machine translation.
- The paper addresses limited understanding of when and why label smoothing improves network performance.
- The authors introduce a visualization method based on linear projections of penultimate-layer activations.
- Label smoothing replaces hard-target cross-entropy with cross-entropy against modified targets controlled by a smoothing parameter α.
2 Penultimate layer representations
Label smoothing changes penultimate-layer geometry by tightening same-class representations and constraining relative logits, while preserving or improving classification accuracy across examined settings.
- Representation geometry: Label smoothing encourages penultimate-layer activations to approach the correct-class template while remaining similarly distant from incorrect-class templates.The visualization projects activations onto the plane crossing three class templates.
- Representation geometry: The visualization compares activations across AlexNet/CIFAR-10, ResNet-56/CIFAR-100, and Inception-v4/ImageNet, including semantically different and similar classes.Figure 1 uses separate rows for these dataset-architecture and class selections.
- CIFAR-10: With label smoothing, CIFAR-10 clusters become tighter and organize into regular triangles, unlike the broader clusters produced by hard targets.The networks have similar accuracies despite their qualitatively different activation clustering.
- CIFAR-100: On CIFAR-100, label smoothing produces better accuracy and limits class-logit differences, whereas hard targets permit larger projection values associated with over-confident predictions.The observed projection scale differs between the two training regimes.
- ImageNet: For ImageNet, semantically similar classes form an arc with label smoothing but a close isotropic cluster under hard targets.The comparison uses toy poodle, miniature poodle, and the semantically different tench class.
3 Implicit model calibration
Label smoothing implicitly improves calibration in image classification and translation. In translation, this calibration benefits BLEU despite worse NLL, while temperature scaling of hard-target models does not recover the same BLEU improvement.
- Calibration motivation: Label smoothing is evaluated as a way to make prediction confidence better reflect prediction accuracy, addressing over-confidence in neural networks.The paper measures calibration using expected calibration error and reliability diagrams.
- Image classification: On CIFAR-100 and ImageNet, hard-target networks are over-confident, while label smoothing and temperature scaling reduce calibration error.On ImageNet, hard targets have ECE 0.071, temperature scaling reaches 0.022, and label smoothing reaches 0.035.
- Image classification: The tight training clusters induced by label smoothing can spread on validation data, representing a range of prediction confidences while maintaining calibration and generalization.This behavior is illustrated for CIFAR-100 representations.
- Machine translation: For English-to-German translation, a Transformer trained with label smoothing is better calibrated than the hard-target alternative.The reliability diagram evaluates next-token predictions assuming a correct prefix.
- Machine translation: Label smoothing improves BLEU despite worse NLL, and temperature scaling of hard-target models is insufficient to match its BLEU score.Changing temperature for the label-smoothed model degrades both calibration and BLEU.
4 Knowledge distillation
The paper finds that label smoothing can improve teacher accuracy while making knowledge distillation less effective. Experiments and mutual-information analysis attribute this to erased logit information about similarities among examples.
- Distillation results: 0.59% teacher test error became 0.91% student test error after distillation with label smoothing, versus 0.67% and 0.74% with hard-target training.The label-smoothed teacher was more accurate, but its distilled student was substantially worse.
- Experimental design: The CIFAR-10 experiment trains a ResNet-56 teacher and distills it into an AlexNet student across teacher and student smoothness settings.The comparisons include nondistilled baselines, hard-target teachers with temperature scaling, and label-smoothed teachers at fixed temperature.
- Distillation results: Increasing α improves teacher accuracy up to α = 0.6, while label smoothing slightly degrades baseline student performance.This separates the teacher’s direct accuracy benefit from the student’s nondistilled baseline behavior.
- Distillation results: Students distilled from hard-target teachers outperform the label-smoothed baseline, whereas better-performing label-smoothed teachers provide no improvement and can perform worse.The authors attribute this to relative information between logits being erased by label smoothing.
- Information erasure: Label smoothing tightens same-class representation clusters, reducing distinctions among training examples that could otherwise be conveyed through teacher logits.The visualization and mutual-information analyses connect representation collapse with poorer distillation.
5 Related work
Related work connects label smoothing to entropy-based regularization, alternative target distributions, calibration research, and information compression in learned representations.
- Label-smoothing variants: Label smoothing is related to confidence penalty, unigram label smoothing, and DisturbLabel, including equivalences or extensions described by prior work.Confidence penalty reverses the KL-divergence order, while unigram smoothing replaces the uniform distribution when labels are imbalanced.
- Calibration: Prior studies investigated calibration in image classification and sequence models, including parametric improvements to next-token calibration in machine translation.The cited work distinguishes image-classification calibration from more recent sequence-level and next-token analyses.
- Information and transfer: Earlier research found that label smoothing impairs transfer learning, while other work linked softmax-representation compression with mutual-information estimates.These findings provide related context for the paper’s account of information loss affecting downstream use of representations.
6 Conclusion and future work
The conclusion explains label smoothing through tight, equally distant penultimate-layer clusters and reduced input information in logits. It highlights improved generalization and calibration alongside impaired distillation, and proposes further study of information relationships.
- Conclusion: Label smoothing encourages training examples to form tight, equally distant clusters in the penultimate layer, visualized with a new low-dimensional projection scheme.The paper presents this geometry as an emergent inductive bias of label smoothing.
- Conclusion: Despite improving generalization and calibration, label smoothing can hurt distillation by reducing mutual information between input examples and output logits.Hard targets permit more logit variation across classes and examples, whereas smoothing treats incorrect classes more equally.
- Future work: The paper identifies the relationship between label smoothing and information as a direction for future research.
A.1 AlexNet
The AlexNet experiments used CIFAR-10 for visualization and distillation, with validation results reported under a fixed training protocol.
- AlexNet visualization and distillation experiments used CIFAR-10 with 40k training and 10k validation examples.
- Five random seeds were used for each plotted point, reporting minimum, average, and maximum values.
- Distillation used β = 1, meaning students were trained fully with teacher targets.
A.2 ResNet
The ResNet experiments covered visualization, calibration, and distillation on CIFAR-10 and CIFAR-100 using a shared validation-based training setup.
- ResNet was used for visualization, calibration, and distillation experiments on CIFAR-10 and CIFAR-100.
- The datasets were split into 40k training and 10k validation examples, with results reported on validation data.
- Training used stochastic gradient descent with Nesterov momentum of 0.9 and an initial learning rate of 0.1.
A.3 Transformer
The Transformer calibration experiment reused a public implementation for TranslateEndeWmt32k, evaluating calibration on about 10k validation tokens and BLEU on the full validation set.
- The Transformer calibration experiment used the TranslateEndeWmt32k dataset with around 32k tokens.
- The experiment compared label smoothing values of 0.1 and 0.0 using the authors’ provided single-GPU hyperparameters.
- ECE and reliability diagrams used around 10k validation tokens, while uncased BLEU used the full validation set.
- The Inception-v4 experiments reused a public implementation and modified batch normalization to include a scale parameter.
B Penultimate layer representation for translation
For translation, next-token prediction is treated as classification, but imbalanced token frequencies, training errors, and dimensional constraints complicate direct comparison with image classification.
- Translation maximizes p(y_t|x, y_0:t−1) for the next token, paralleling image classification’s p(y|x) objective.
- Unlike balanced image classes, translation token distributions are highly imbalanced, which can affect visualization and suggests unigram label smoothing as a possible adjustment.
- Translation training accuracy is around 80%, so visualizations include errors from both training and validation sets.
- The translation setting uses a roughly 30k-token alphabet in 512 dimensions, preventing token templates from forming a regular simplex.
- The paper focuses visualization and distillation intuition on image classification, leaving translation distillation for future work.
- Figure 7 visualizes penultimate representations of a Transformer trained for English-to-German translation.