PaperlayerView exact revision on arXiv

Source-linked AI summary

Delving Deep into Label Smoothing

Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, Ming-Ming Cheng

arXiv:2011.12562v2cs.CV

TL;DR

The paper addresses the limited reliability of conventional soft labels, which treat non-target categories uniformly. It proposes Online Label Smoothing, updating class-level soft labels from intermediate model predictions to encode category relationships. Across CIFAR-100, ImageNet, and fine-grained datasets, OLS consistently improves classification performance and is reported to improve calibration and noisy-label robustness.

  • Problem

    Conventional label smoothing regularizes DNNs but assigns identical probabilities to non-target categories, omitting relationships among categories.

  • Method

    OLS continuously updates class-level soft labels during training using model-prediction statistics, then uses them to supervise the model.

  • Results

    OLS consistently improves performance across CIFAR-100, ImageNet, and four fine-grained datasets, with reported gains including 2.11% on CIFAR-100 and 1.4% on ImageNet.

  • Takeaways & Limitations

    OLS provides a practical regularization tool that represents category similarities and can be applied during training across classification models.

Abstract

from arXiv · show

Label smoothing is an effective regularization tool for deep neural networks (DNNs), which generates soft labels by applying a weighted average between the uniform distribution and the hard label. It is often used to reduce the overfitting problem of training DNNs and further improve classification performance. In this paper, we aim to investigate how to generate more reliable soft labels. We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category. The proposed OLS constructs a more reasonable probability distribution between the target categories and non-target categories to supervise DNNs. Experiments demonstrate that based on the same classification models, the proposed approach can effectively improve the classification performance on CIFAR-100, ImageNet, and fine-grained datasets. Additionally, the proposed method can significantly improve the robustness of DNN models to noisy labels compared to current label smoothing approaches.

I. INTRODUCTION

The paper argues that conventional label smoothing regularizes over-confident DNNs but assigns identical probabilities to all non-target categories. It proposes Online Label Smoothing, which uses model-prediction statistics to represent category relationships, and reports consistent gains across benchmarks.

  • I. INTRODUCTION: Conventional label smoothing averages hard labels with a uniform distribution to regularize DNNs and reduce over-confidence.Its fixed non-target probabilities can limit the usefulness of category relationships.
  • I. INTRODUCTION: OLS maintains class-level label distributions that update during training from intermediate model predictions.These evolving distributions supervise the model until convergence and assign non-uniform confidence to related categories.
  • I. INTRODUCTION: 1.57% and 2.11% top-1 performance gains were obtained on CIFAR-100 with ResNet-56 and ResNeXt29-2x64d, respectively.The reported gains come from directly applying OLS to the two backbones.
  • I. INTRODUCTION: 1.4% and 1.02% performance improvements were obtained on ImageNet with ResNet-50 and ResNet-101, respectively.OLS also achieved an average 1.0% performance improvement over LS across four fine-grained datasets and four backbones.
  • I. INTRODUCTION: OLS is positioned as a label-based regularization tool that can also address noisy labels by reducing overfitting to training sets.The paper describes it as conveniently applicable during training across many classification models.

III. METHOD

The method first defines hard-label training and conventional label smoothing, then identifies uniform treatment of non-target classes as the key limitation motivating OLS.

  • III. METHOD: For each sample, the DNN predicts p(k|xi) over K classes, while the hard-label distribution assigns probability 1 to yi and 0 to other classes.The standard cross-entropy setup uses these target distributions for image classification.
  • III. METHOD: Label smoothing replaces hard labels with a mixture of the hard-label distribution and a uniform distribution over classes.The smoothing parameter ε is usually set to 0.1 in practice.
  • III. METHOD: Conventional label smoothing treats non-target categories equally and therefore does not capture genuine relationships among categories.OLS is introduced to account for those relationships instead of using a static uniform treatment.

B. Online Label Smoothing

Online Label Smoothing maintains class-level soft labels that are updated from correctly classified samples and used across successive training epochs. It combines these soft labels with hard-label supervision to encourage more meaningful within-class representations.

  • Online Label Smoothing: OLS maintains a K×K matrix whose columns are class-level soft labels updated across training epochs.The previous epoch’s labels supervise the current epoch, while current predictions generate labels for the next epoch.
  • Online Label Smoothing: OLS combines hard-label and soft-label supervision because using soft labels alone makes convergence difficult under random initialization.The balancing coefficient α controls the contributions of Lhard and Lsoft.
  • Online Label Smoothing: Correctly classified samples update the soft-label column corresponding to their target category using their predicted probability distributions.The accumulated columns are normalized at the end of each epoch.
  • Online Label Smoothing: Uniform soft labels initialize the first epoch, while subsequent epochs use accumulated correct predictions to generate the class-level distributions.This initialization is equivalent to vanilla label smoothing and avoids early training divergence.
  • Online Label Smoothing: The resulting training constraint encourages samples from the same category to become closer in representation space.The paper visualizes these representations with t-SNE on CIFAR-100 using ResNet-56.

IV. DISCUSSION

The discussion contrasts OLS with teacher-free knowledge distillation and model ensembling. OLS uses prediction knowledge from prior epochs to supervise current learning, while remaining compatible with ensemble prediction strategies.

  • Comparison with Tf-KD: Tf-KDreg uses a hand-designed distribution with correct-class information but remains uniform across non-target categories.OLS instead seeks a non-uniform distribution that reflects relationships among classes.
  • Comparison with Tf-KD: OLS can outperform Tf-KD because its soft labels benefit from similarities between classes.The comparison is reported in Table VI.
  • Connection with the model ensemble: Model ensembling averages outputs from models at different epochs, whereas OLS uses prior-epoch knowledge to supervise current training.The paper states that the two strategies do not conflict.
  • Connection with the model ensemble: With the same ensemble strategy on CIFAR-100 using ResNet-56, OLS achieves 25.27% Top-1 Error.The setup selects models uniformly across a 300-epoch training schedule.

V. EXPERIMENTS

The experiments evaluate OLS across classification benchmarks, noisy-label tolerance, adversarial robustness, object detection, and ablation settings. Implementations use PyTorch and Jittor.

  • V. EXPERIMENTS: Experiments evaluate OLS on CIFAR-100, ImageNet, fine-grained datasets, symmetric noisy labels, adversarial attacks, object detection, and ablation settings.These evaluations are organized across Sections V-A through V-E.
  • V. EXPERIMENTS: All experiments are implemented using PyTorch and Jittor.

A. General Image Recognition

OLS consistently improves classification across CIFAR-100, ImageNet, and fine-grained datasets, while remaining compatible with other regularization and distillation strategies.

  • CIFAR Classification: OLS improves CIFAR-100 classification across lightweight and complex architectures, outperforming LS by about 1.5% on ResNeXt29-2x64d.The comparison uses three runs and reports mean and standard deviation of Top-1 Error.
  • ImageNet Classification: OLS improves ImageNet results, achieving 22.28% Top-1 Error on ResNet-50 and improving ResNet-101 by 1.02%.On ResNet-101, OLS also improves over LS by 0.42%.
  • Generalization Across Strategies: Combining OLS with CutOut or BYOT brings extra performance gains on ResNet-50 and ResNet-101.The paper presents OLS as a plugin regularization module that can be combined with other methods.
  • Fine-grained Classification: OLS remains effective on four fine-grained recognition datasets evaluated with the same experimental setup.The datasets are CUB-200-2011, Flowers-102, Cars, and Aircrafts; results report average Top-1 and Top-5 Error.

B. Tolerance to Noisy Labels

OLS is evaluated under symmetric label noise and shows stable advantages over LS while reducing the model’s fit to incorrectly labeled samples.

  • Noisy-label classification: OLS achieves comparable results to methods specifically designed for noisy labels across noisy rates from 0% to 80%.The evaluation uses ResNet-56 and reports mean and standard deviation of Top-1 Error over three runs.
  • Noisy-label classification: OLS consistently improves over LS under different noisy rates.Noise is introduced by uniformly flipping selected labels to incorrect labels before training.
  • Training and test behavior: OLS has higher training errors but lower test errors than hard-label and LS models under noisy labels.This pattern is presented as evidence that OLS reduces overfitting to noisy samples.
  • Training on wrong labels: OLS fits incorrectly labeled training samples worse than baselines, as measured by Top-1 Error calculated against the wrong labels.Higher error under this calculation means weaker fitting to the wrong labels.

C. Robustness to Adversarial Attacks

OLS improves robustness to adversarial attacks on CIFAR-10 and ImageNet, with intra-class constraints encouraging samples from the same class to remain closer.

  • Attack setup: FGSM searches within an ϵ-ball for perturbations that cross a sample’s decision boundary and cause misclassification.The adversarial example is constructed using the loss gradient and a signed optimization step.
  • Mechanism: OLS accumulates same-class predictions into soft labels, imposing intra-class constraints that encourage samples in the same class to become closer.These constraints are described as making current training samples farther from the decision boundary.
  • Adversarial evaluation: OLS-trained models achieve the lowest Top-1 Error after FGSM and PGD attacks on CIFAR-10.The evaluation attacks ResNet-29 models using FGSM and PGD, with 20 PGD iterations.
  • Adversarial evaluation: 17.9% average Top-1 Error gain and 13.9% average Top-5 Error gain are reported against hard labels on ImageNet attacks.OLS also outperforms LS by 2.3% on Top-1 Error and 2.4% on Top-5 Error.

D. Object Detection

OLS extends beyond image classification to object detection, improving YOLO’s PASCAL VOC performance over both hard labels and LS.

  • Experimental setup: The object-detection experiment trains on combined PASCAL VOC 2012 and 2007 training and validation data for 120 epochs.Training uses SGD, warmup, multi-scale training, random crop, and cosine learning-rate decay.

E. Ablation Study

The ablation study examines hyperparameters, category relationships, sample-level aggregation, and calibration. OLS performs best with moderate label weighting and frequent updates, while aggregated predictions improve relationships and calibration.

  • Impact of Hyper-parameters: α=0.5 yields the lowest Top-1 Error, while α=0 causes difficult convergence.As α increases from 0.1 to 0.5, the error rate gradually decreases, indicating that original hard-label information remains necessary.
  • Impact of Hyper-parameters: One epoch is the best updating period; performance drops sharply when soft-label updates occur less frequently.Performance is similar for periods shorter than one epoch, approximately 384 iterations, but declines for longer periods.
  • Importance of relationships among categories: OLS predictions are closer to human uncertainty distributions and indicate more reasonable category relationships.The comparison uses average KL divergence on CIFAR-10H for correctly predicted samples.
  • Sample-level soft labels: 26.18 ± 0.30 versus 25.24 ± 0.18 Top-1 Error shows OLS outperforming OLS-Single on ResNet-56.The result supports accumulating predictions across different samples rather than using a single sample’s predictive distribution.
  • Calibration effect: OLS lowers Top-1 Error by an average of 1.14% versus LS and achieves lower ECE on three depth models.The authors interpret these results as improved prevention of over-confident predictions and better calibration.

VI. CONCLUSION

The paper proposes Online Label Smoothing, which uses statistics from intermediate model predictions to generate relationship-aware soft labels. OLS improves performance across CIFAR, ImageNet, and fine-grained datasets.

  • VI. CONCLUSION: OLS generates soft labels from statistics of intermediate model predictions and uses them to supervise the model.The labels incorporate relationships among categories to address overfitting to the training set.
  • VI. CONCLUSION: 18.81% Top-1 Error on CIFAR-100 with ResNeXt-2x64d corresponds to a 2.11% performance gain.On ImageNet, OLS brings 1.4% and 1.02% gains to ResNet-50 and ResNet-101, respectively.
  • VI. CONCLUSION: OLS outperforms the hard label by 2% in Top-1 Error across four fine-grained datasets.The evaluation covers CIFAR, ImageNet, and four fine-grained datasets.
Loading 2011.12562v2…