PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Knowledge Distillation with Progressive Refinement of Targets

Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, Sangheum Hwang

arXiv:2006.12000v3cs.LGstat.ML

TL;DR

Deep neural networks can overfit and produce overconfident, miscalibrated predictions, motivating better ways to soften hard targets. PS-KD progressively combines ground-truth labels with past predictions from the model itself. Across image classification, object detection, and machine translation, it consistently improves generalization and confidence estimation over baselines.

  • Problem

    Deep neural networks can overfit and produce overconfident, miscalibrated predictions, while existing label smoothing can degrade performance with advanced regularization.

  • Method

    PS-KD makes the model its own evolving teacher, adaptively combining hard targets with past predictions to soften training targets.

  • Results

    PS-KD consistently improves state-of-the-art baselines across image classification, object detection, and machine translation, including accuracy, confidence estimation, and BLEU scores.

  • Takeaways & Limitations

    PS-KD provides a simple, broadly applicable regularizer that implicitly emphasizes hard examples and improves both predictive performance and confidence quality.

  • Takeaways & Limitations

    Label smoothing is not complementary to some advanced regularization techniques, with simultaneous LS and CutMix substantially degrading classification and confidence-estimation performance.

Abstract

from arXiv · show

The generalization capability of deep neural networks has been substantially improved by applying a wide spectrum of regularization methods, e.g., restricting function space, injecting randomness during training, augmenting data, etc. In this work, we propose a simple yet effective regularization method named progressive self-knowledge distillation (PS-KD), which progressively distills a model's own knowledge to soften hard targets (i.e., one-hot vectors) during training. Hence, it can be interpreted within a framework of knowledge distillation as a student becomes a teacher itself. Specifically, targets are adjusted adaptively by combining the ground-truth and past predictions from the model itself. We show that PS-KD provides an effect of hard example mining by rescaling gradients according to difficulty in classifying examples. The proposed method is applicable to any supervised learning tasks with hard targets and can be easily combined with existing regularization methods to further enhance the generalization performance. Furthermore, it is confirmed that PS-KD achieves not only better accuracy, but also provides high quality of confidence estimates in terms of calibration as well as ordinal ranking. Extensive experimental results on three different tasks, image classification, object detection, and machine translation, demonstrate that our method consistently improves the performance of the state-of-the-art baselines. The code is available at https://github.com/lgcnsai/PS-KD-Pytorch.

1. Introduction

Deep neural networks can overfit and produce overconfident, miscalibrated predictions, motivating improved target-softening methods. PS-KD adaptively combines ground-truth labels with past model predictions and improves generalization and confidence estimation across tasks.

  • Scaling deep networks can increase overfitting, while their incorrect predictions are often overconfident because they are miscalibrated.
  • Label smoothing improves generalization and calibration but can degrade performance when combined with advanced regularization such as CutMix.
  • PS-KD progressively softens hard targets by adaptively combining ground-truth labels with the model’s past predictions during training.
  • PS-KD reweights gradients toward hard-to-learn examples and can be combined with existing regularization methods for supervised tasks with hard targets.
  • Experiments on image classification, object detection, and machine translation show improved generalization, confidence estimates, representations, and BLEU scores over state-of-the-art baselines.

2. Related Work

Prior self-knowledge-distillation methods use feature matching, same-class samples, pretrained generations, auxiliary classifiers, or snapshot teachers. PS-KD instead uses evolving past predictions without pretraining and is presented as broadly applicable and simple.

  • Self-knowledge distillation methods have used feature matching between distorted or same-class inputs within a single network.
  • Other approaches distill outputs from same-architecture teachers through pretrained generations, auxiliary classifiers, or snapshot models across mini-generations.
  • PS-KD uses a model’s own predictions as teacher knowledge to enhance generalization performance.
  • Unlike BANs and TF-KD, PS-KD does not require pretraining and is designed for broad supervised-learning applicability with practical simplicity.

3. Self-Knowledge Distillation

Knowledge distillation trains students with teacher probabilities alongside hard labels, while PS-KD makes the evolving model its own teacher. Past predictions are blended progressively into targets, yielding adaptive regularization and implicit hard-example mining.

  • 3.1. Knowledge Distillation as Softening Targets: Knowledge distillation transfers a teacher’s predictive probabilities to a student alongside one-hot labels, typically from a larger model to a smaller one.
  • 3.1. Knowledge Distillation as Softening Targets: Temperature scaling softens teacher and student probabilities, and the distillation loss combines hard-label cross-entropy with softened teacher-student cross-entropy.
  • 3.1. Knowledge Distillation as Softening Targets: When τ=1, the distillation loss is equivalent to training against a soft target formed by linearly combining the hard target and teacher prediction.
  • 3.2. Distilling Knowledge from the Past Predictions: PS-KD uses the model at epoch t−1 as a teacher and trains epoch t with targets (1−α)y+αP^S_{t−1}(x).
  • 3.2. Distilling Knowledge from the Past Predictions: The teacher evolves during training, and α increases gradually because early predictions contain less reliable knowledge.
  • Theoretical support: PS-KD’s gradient rescaling gives greater weights to hard-to-learn examples than easy examples, an effect confirmed empirically on CIFAR-100.
  • Implementation: The method requires past predictions, which can be obtained by retaining the previous model in memory or saving predictions for later use.

4. Experimental Results

Across image classification, object detection, and machine translation experiments, PS-KD improves predictive performance and confidence estimation, while complementing advanced regularization methods and ensembles.

  • CIFAR-100 Classification: PS-KD improves accuracy and confidence estimation across CIFAR-100 architectures, outperforming baseline, label smoothing, CS-KD, and TF-KD.On DenseNet-121, accuracy improves by 1.74% and 1.15% over CS-KD and TF-KD; on PyramidNet, ECE decreases by 12.87% and 8.65%.
  • CIFAR-100 Classification: PS-KD remains the only method with consistent and robust improvement across all evaluated metrics, whereas CS-KD sometimes underperforms label smoothing or the baseline.The authors attribute possible CS-KD degradation to pulling same-class samples together, potentially accelerating overfitting.
  • Complementarity with Regularization: 14.82% top-1 error is achieved when Cutout is combined with PS-KD, improving Cutout by 1.23%.Combining PS-KD with CutMix and stochastic depth reduces top-1 error by 0.48% versus CutMix and stochastic depth, while improving NLL, ECE, and AURC.
  • Ensembles: PS-KD improves ensembles without degrading the diversity of independently trained models.The ensembles use three trained models from the CIFAR-100 comparison.
  • ImageNet Classification: 21.41% top-1 error is achieved on ImageNet, with PS-KD reducing ECE by 3.28% versus CS-KD and 2.19% versus TF-KD.PS-KD also improves all metrics when combined with CutMix, especially ECE.
  • Object Detection: 1.06%, 1.17%, and 1.22% mAP improvements are obtained over LS, CS-KD, and TF-KD for Faster R-CNN with a PS-KD-trained ResNet-152 backbone.PS-KD also yields better mAP when combined with CutMix.
  • Machine Translation: PS-KD achieves the best BLEU scores on all evaluated machine-translation datasets, outperforming the baseline Transformer and label smoothing.The results are consistent with image-classification and object-detection findings.

5. Conclusion

PS-KD distills a model’s past predictions into more informative training targets and implicitly performs hard-example mining through gradient rescaling. It improves generalization across diverse tasks and can be implemented with Fairseq.

  • PS-KD distills a model’s own knowledge to generate more informative targets for training.Past predictions from the previous epoch soften the targets.
  • PS-KD implicitly performs hard-example mining during training through gradient rescaling.
  • Experiments used the Fairseq toolkit.
  • PS-KD improves the generalization capability of DNNs across diverse tasks.

A.1. Evaluation Metrics

The evaluation metrics assess confidence calibration and confidence-based prediction ordering. ECE compares accuracy with confidence within bins, while AURC evaluates selective prediction using coverage and risk.

  • ECE: ECE estimates the gap between accuracy and confidence by partitioning samples into confidence-based bins.Each bin contains samples whose confidence falls within a specified interval.
  • ECE: Lower ECE indicates that a model is well-calibrated.
  • Calibration visualization: Reliability diagrams and calibration plots visualize calibration by plotting accuracy against confidence values.
  • AURC: AURC measures how well predictions are ordered by confidence using selective classification, coverage, and risk.Coverage is the proportion of retained samples, while risk is the error rate among those samples.

A.2. Methods

The methods include label smoothing, input occlusion, sample mixing, and ShakeDrop regularization. These techniques modify targets, inputs, or residual-block behavior to regularize training.

  • Label smoothing: Label smoothing replaces one-hot targets with soft targets by combining hard targets with a uniform class distribution.The smoothing parameter is ϵ, and K is the number of classes.
  • Cutout: Cutout randomly selects a square region of an input image and zeroes it during training.The resulting partially occluded image acts as augmented data through structured input-space dropout.
  • CutMix: CutMix constructs a training sample by replacing a rectangular region of one image with a patch from another image.Its target is a weighted combination of the two source targets, with λ sampled uniformly from (0, 1).
  • ShakeDrop: ShakeDrop regularizes ResNet variants by replacing residual blocks with ShakeDrop blocks.The block uses distinct train-forward, train-backward, and test-phase output rules.

A.3. Datasets

The evaluation uses CIFAR-100 and ImageNet for image classification. CIFAR-100 is a balanced 100-class dataset, while ImageNet is a large-scale 1K-class dataset with varied image resolutions.

  • CIFAR-100: CIFAR-100 contains 50K training images and 10K test images at 32×32 resolution across 100 classes.Each class has the same number of images.
  • ImageNet: ImageNet contains 1.2M training images and 50K validation images across 1K classes.The dataset includes images with various resolutions and some images containing multiple objects.
  • ImageNet: ImageNet training uses random 224×224 crops from images resized to 256×256, while inference uses center crops of the same size.

A.4. Experimental Results on CIFAR-100

On CIFAR-100, PS-KD benefits from adaptive target softening, with αT = 0.8 selected by jointly considering error and calibration. Its improvements extend to calibration and advanced regularization combinations.

  • Hyperparameter selection: αT = 0.8 is selected as the best value using validation top-1 error and ECE on CIFAR-100 with ResNet-18.The same value is used for the other experiments.
  • Target softening: Fixed αt = 0.1 already outperforms LS with ϵ = 0.1 in NLL and top-1 error, while larger fixed values further improve performance.These results support using model predictions rather than static label smoothing.
  • Target softening: Linear growth toward αT = 0.8 performs surprisingly well compared with fixed αt = 0.8, providing a simple regularization schedule.The comparison isolates the effect of adjusting αt during training.
  • Calibration: PS-KD improves calibration for advanced regularization methods including Cutout, CutMix, and CutMix+SD on CIFAR-100.The calibration benefit is reported across several existing augmentation and regularization methods.

A.5. Experimental Results on ImageNet

On ImageNet, PS-KD improves calibration relative to comparison target-softening methods and adds calibration benefits to CutMix, while also improving human interpretability and AP for many detection classes.

  • Calibration: PS-KD improves CutMix calibration on ImageNet compared with the existing advanced regularization method alone.The result is reported in the calibration comparison plots.
  • Calibration: PS-KD is slightly better calibrated than LS, CS-KD, and TF-KD on ImageNet with ResNet-152.It also provides additional calibration benefits when combined with CutMix.
  • Prediction quality: PS-KD produces predicted probabilities judged better in terms of human interpretation for additional ImageNet validation samples.The examples compare baseline and PS-KD outputs across several labeled images.
  • Object detection: PS-KD has higher AP than the baseline, LS, CS-KD, and TF-KD for 10 of 20 PASCAL VOC classes.The comparison is summarized over all classes in Table S9.

C.1. Evaluation Metrics

BLEU evaluates machine translation using reference overlap and sentence-length agreement.

  • BLEU: BLEU measures machine-translation quality using 4-gram overlap with human references and a penalty based on translated-sentence length.It combines n-gram matching with a length comparison to avoid rewarding overly short outputs.

C.2. Dataset

The experiments use the IWSLT15 English–German and German–English dataset, with designated training, validation, and combined test splits.

  • Dataset: IWSLT15 contains 191K training sentence pairs, with 8,300 training pairs reserved for validation.The test set concatenates dev2010, dev2012, tst2010, tst2011, tst2012, and tst2013.
Loading 2006.12000v3…