PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective

Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, Qian Zhang

arXiv:2102.00650v1cs.LGcs.CV

TL;DR

Soft labels regularize student networks, but how distillation changes bias and variance remains unclear. The paper analyzes this tradeoff sample-wise, proposes weighted soft labels, and reports effective results on standard evaluation benchmarks.

  • Problem

    The bias-variance changes caused by training with soft labels, and their effect on distillation performance, remain unclear.

  • Method

    The paper decomposes distillation behavior through bias-variance analysis, identifies sample-wise tradeoffs, and proposes weighted soft labels for adaptive handling.

  • Results

    Experiments on standard evaluation benchmarks validate the effectiveness of weighted soft labels, including new state-of-the-art results on CIFAR-100.

  • Takeaways & Limitations

    Regularization samples should be handled adaptively rather than completely filtered out, because they can remain valuable for distillation.

  • Takeaways & Limitations

    The analysis assumes that KD produces smaller variance than direct training.

Abstract

from arXiv · show

Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network. The outputs from the teacher network are used as soft labels for supervising the training of a new network. Recent studies \citep{muller2019does,yuan2020revisiting} revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network. From the perspective of statistical learning, regularization aims to reduce the variance, however how bias and variance change is not clear for training with soft labels. In this paper, we investigate the bias-variance tradeoff brought by distillation with soft labels. Specifically, we observe that during training the bias-variance tradeoff varies sample-wisely. Further, under the same distillation temperature setting, we observe that the distillation performance is negatively associated with the number of some specific samples, which are named as regularization samples since these samples lead to bias increasing and variance decreasing. Nevertheless, we empirically find that completely filtering out regularization samples also deteriorates distillation performance. Our discoveries inspired us to propose the novel weighted soft labels to help the network adaptively handle the sample-wise bias-variance tradeoff. Experiments on standard evaluation benchmarks validate the effectiveness of our method. Our code is available at \url{https://github.com/bellymonster/Weighted-Soft-Label-Distillation}.

1 INTRODUCTION

The paper examines why soft labels help student networks by treating them as both supervisory signals and regularizers, focusing on their sample-wise bias-variance tradeoff. It proposes weighted soft labels after finding that regularization samples can hurt performance when overrepresented but remain valuable.

  • Motivation: Knowledge distillation uses teacher predictions as soft labels to guide student-network training.Soft labels have been effective in empirical and theoretical studies.
  • Motivation: The benefit of soft labels lacks a clear theoretical explanation, although prior work identifies their regularization effect.Label smoothing regularizes penultimate-layer activations, and related work hypothesizes regularization as a main reason soft labels work.
  • Motivation: Soft labels act as both supervisory signals and regularizers, creating a bias-variance tradeoff whose behavior in distillation remains unclear.The paper investigates whether this tradeoff exists and how it affects distillation performance.
  • Analysis: The analysis finds that distillation has larger bias and smaller variance than direct training, with the tradeoff varying across samples during training.The authors rewrite the distillation loss as direct-training loss plus a regularization loss and inspect the two terms’ gradients.
  • Contribution: Regularization samples are associated with lower distillation performance when their number increases at a fixed temperature, motivating adaptive weighting.The proposed weighted soft labels aim to alleviate their negative effects while retaining their value.

2 RELATED WORKS

Prior work studies soft-label regularization and adaptive distillation, while this paper distinguishes itself by analyzing sample-wise bias-variance behavior and tuning that tradeoff during training.

  • Knowledge distillation: Knowledge distillation transfers outputs from large models to smaller, faster models, while soft labels can carry useful information and act as regularizers.Label smoothing’s adverse effect on distillation motivated further study of soft-label regularization.
  • Distillation loss: Adaptive distillation methods have weighted hard-to-mimic samples using teacher-student prediction gaps, but may be harmed when teachers provide incorrect guidance.A large prediction gap does not necessarily indicate that emphasizing a sample is beneficial.
  • Bias-variance tradeoff: Existing bias-variance studies mainly examine variance from network-model choice, whereas this paper focuses on sample behavior during training.Its goal is adaptive tuning of the sample-wise tradeoff rather than only theoretical analysis.

3 BIAS-VARIANCE TRADEOFF FOR SOFT LABELS

The paper decomposes distillation into bias and variance effects, finding that soft-label training reduces variance while increasing bias and that this tradeoff varies across samples and training time. Samples whose variance-reduction effect dominates are associated with poorer performance when more are present, yet filtering them entirely also harms distillation, motivating adaptive weighting.

  • Bias-variance decomposition: Knowledge distillation produces larger bias error and smaller variance than direct training with one-hot labels.The decomposition separates intrinsic noise, bias, and variance using KL divergences.
  • Bias-variance decomposition: Writing Lkd = Lkd − Lce + Lce identifies Lkd − Lce as the variance-reduction term and Lce as the bias-reduction term.The two terms induce opposing changes: variance decreases while bias increases through the distillation-specific component.
  • Sample-wise tradeoff: The bias-variance tradeoff varies sample-wisely because gradients from bias and variance reduction can oppose each other during training.The relative importance changes over training: bias dominates shortly after training begins, while variance becomes more relevant later.
  • Regularization samples: A regularization sample is one for which the variance-reduction gradient dominates, operationally defined by b > a.Here a represents the bias-reduction impact and b represents the variance-reduction impact on the ground-truth-related logit.
  • Regularization samples: Under fixed temperature, more regularization samples are associated with worse distillation performance, and label-smoothed teachers produce more such samples.Their count also increases faster with training epochs under teacher label smoothing, widening the gap over time.
  • Adaptive handling: Completely filtering out regularization samples deteriorates performance, indicating that these samples remain valuable despite their variance-dominant effect.The experiments motivate tuning their contribution rather than removing them entirely, leading to weighted soft labels.

4 WEIGHTED SOFT LABELS

The method lowers the contribution of regularization samples by assigning sample-wise weights based on teacher and student predictions, while keeping the direct training loss.

  • 4 WEIGHTED SOFT LABELS: Weighted soft labels lower the weight of regularization samples because discarding them entirely hurts distillation.The weighting strategy is motivated by the finding that regularization samples are harmful in excess but still outperform direct training when retained.
  • 4 WEIGHTED SOFT LABELS: The weighting factor for each sample is determined by comparing teacher and student predictions at temperature τ = 1.This comparison makes the weighting scheme independent of the distillation temperature hyperparameter.
  • 4 WEIGHTED SOFT LABELS: A relatively better-trained student receives a smaller weight for that sample’s distillation loss.The method assigns importance according to the relative values of the teacher and student predictions for the ground-truth class.
  • 4 WEIGHTED SOFT LABELS: The total supervision loss combines the direct training loss with the weighted soft-label loss using a balancing hyperparameter α.The resulting objective is Ltotal = Lce + αLwsl.

5 EXPERIMENTS

Experiments evaluate weighted soft labels across teacher-student settings on CIFAR-100 and against state-of-the-art methods on ImageNet, with ablations targeting regularization samples. The method achieves new state-of-the-art results on CIFAR-100 and outperforms existing methods on ImageNet, while weighted labels improve performance on selected subsets and with label-smoothed teachers.

  • 5 EXPERIMENTS: Experiments cover varied teacher-student pairs on CIFAR-100, state-of-the-art comparisons on ImageNet, and subset ablations for bias-variance tradeoff handling.The datasets are CIFAR-100 and ImageNet, with temperature τ = 4 and τ = 2, respectively.
  • 5.2 MODEL COMPRESSION: Weighted soft labels reach new state-of-the-art Top-1 classification accuracy on CIFAR-100 for same- and different-architecture distillation.The method outperforms standard KD by a large margin in both architecture settings.
  • 5.2 MODEL COMPRESSION: Weighted soft labels outperform all existing methods in the reported ImageNet comparisons.The ImageNet table reports Top-1 and Top-5 validation accuracy, with one comparison using a CRD model trained for 10 extra epochs.
  • 5.3 ABLATION STUDIES: Ablations evaluate weighted soft labels on different training subsets, including regularization samples, with results averaged over five runs.These experiments test whether performance gains reflect handling of the sample-wise bias-variance tradeoff.
  • 5.3 ABLATION STUDIES: Weighted soft labels significantly improve distillation, especially when the teacher was trained with label smoothing.Teachers trained with label smoothing still perform worse than teachers without label smoothing.

6 CONCLUSION

The paper concludes that soft-label distillation involves a sample-wise bias-variance tradeoff and proposes weighted soft labels to address it. Experiments on standard benchmarks verify the method’s effectiveness.

  • 6 CONCLUSION: The bias-variance tradeoff in soft-label distillation varies across samples.This sample-wise variation motivates adapting the treatment of soft labels rather than applying one uniform strategy.
  • 6 CONCLUSION: Weighted soft labels handle the sample-wise tradeoff and are effective on standard evaluation benchmarks.The conclusion summarizes the proposed approach and its experimental validation.

A.1 VISUALIZATION OF THE RESEMBLANCES INTRODUCED BY SOFT LABEL REGULARIZER

The visualization analyzes class-level relations implied by the soft-label regularizer at τ = 4. The reported variance reduction remains associated with semantic resemblances among class labels.

  • A.1 VISUALIZATION OF THE RESEMBLANCES INTRODUCED BY SOFT LABEL REGULARIZER: At τ = 4, the regularizer’s variance reduction still implies resemblances among labels consistent with class-name semantic distance.The paper examines these relations through class-level gradient-derived quantities visualized from a 1000 × 1000 matrix.

A.2 INTERMEDIATE STATES BETWEEN EXCLUDING AND ONLY ON REGULARIZATION SAMPLES

The experiments examine intermediate training settings between excluding all regularization samples and using only regularization samples. They vary the inclusion of regularization and non-regularization samples in two complementary ways.

  • Two training settings vary the inclusion of regularization samples and non-regularization samples.The first excludes progressively fewer regularization samples; the second progressively adds non-regularization samples while retaining all regularization samples.
  • Regularization samples are gradually retained, moving from excluding all to excluding 25% of them.
  • Non-regularization samples are gradually added while all regularization samples remain included.

A.3 COMBINING WITH RKD (PARK ET AL., 2019).

The paper applies weighted soft labels to relational knowledge distillation, where transferred knowledge depends on relations among samples rather than individual samples. Averaging the weights of involved sample pairs still improves results on CIFAR-100, though less than applying weighted soft labels to standard KD.

  • RKD transfers knowledge through L2 distances between sample features or angles formed by three samples.
  • Because RKD is sample-dependent, the method averages weighting factors for the sample pairs involved in each distance or angle matrix.
  • Weighted soft labels improve RKD on CIFAR-100, although the gain is smaller than for weighted soft labels applied to KD.The reported results are averaged over 5 runs.

A.4 OTHER VARIANTS OF WEIGHTING.

The paper compares alternative sample-weighting forms and evaluates weighting sensitivity and transfer beyond image classification. The results support adaptive sample-wise weighting, with limited sensitivity to α and improvements on an NLP distillation task.

  • The weighting scheme converts the distillation loss ratio into [0, 1], assigning lower weights to regularization samples than to non-regularization samples.
  • The adopted weighting form outperforms the Sigmoid baseline and standard KD when the sample-wise bias-variance tradeoff is adaptively tuned.The authors state that the proposed form is not mathematically optimal, but its weights are not difficult to tune.
  • Performance is not very sensitive to α across the tested CIFAR-100 and ImageNet settings, and searching for α is reported as inexpensive.
  • On MultiNLI, the method is evaluated with BERT-base-cased as teacher and T3 as student against standard KD.The teacher has 12 layers and 108M parameters, while the student has 3 layers and 44M parameters.
Loading 2102.00650v1…