PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling the Knowledge in a Neural Network

Geoffrey Hinton, Oriol Vinyals, Jeff Dean

arXiv:1503.02531v1stat.MLcs.LGcs.NE

TL;DR

The paper addresses the deployment cost of ensembles and develops distillation to transfer their knowledge into smaller models. It also introduces independently trainable specialist ensembles, achieving strong transfer results and improving large-network performance.

  • Problem

    Ensembles improve predictions but are cumbersome and computationally expensive to deploy and, for very large datasets, expensive to train.

  • Method

    The paper distills ensemble knowledge into smaller models using soft targets and trains specialist networks independently to discriminate among highly confusable classes.

  • Results

    Nearly all ensemble improvement for a deep acoustic model was distilled into a same-sized, easier-to-deploy network, while 61 specialists produced a 4.4% relative test-accuracy improvement.

  • Takeaways & Limitations

    Distillation can preserve ensemble benefits in a single deployable model, while parallel specialist models can improve very large networks without requiring a full ensemble.

  • Takeaways & Limitations

    The paper has not shown that specialist knowledge can be distilled back into the single large network.

Abstract

from arXiv · show

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.

1 Introduction

Distillation separates training from deployment: a cumbersome model can extract structure from large data, then transfer its learned generalization to a smaller model. Soft class probabilities preserve information about how the cumbersome model generalizes beyond hard labels.

  • Training versus deployment: Training can use cumbersome models and extensive computation, whereas deployment requires low latency and limited computational resources.The paper motivates using different models for extracting structure during training and serving many users at deployment.
  • What is transferred: Distillation transfers knowledge from a cumbersome model to a smaller model without tying that knowledge to the original parameterization.The paper characterizes knowledge more abstractly as a learned mapping from input vectors to output vectors.
  • Soft targets: Incorrect-class probabilities reveal how the cumbersome model generalizes because their relative values encode similarities among classes.For example, different probabilities assigned to confusing digit or object classes can distinguish how inputs resemble alternative classes.
  • Soft targets: A small model can be trained to generalize in the same way as a large model, including when the large model benefits from ensemble averaging.This directly aligns the transfer objective with the larger model’s learned generalization behavior.
  • Soft targets: Soft targets from the cumbersome model provide more information per case and lower gradient variance when they have high entropy.They can allow the small model to use less data and a higher learning rate than the original cumbersome model.
  • Soft targets: On MNIST, ratios among very small soft-target probabilities encode a rich similarity structure that hard-label training largely overlooks.These ratios can indicate whether one instance of a digit resembles a 3 or a 7.

2 Distillation

Distillation trains a student with softened class distributions from a teacher, optionally combining them with correct labels. At high temperature, the procedure becomes squared-logit matching under a zero-mean assumption, while intermediate temperatures can be advantageous for undersized students.

  • Temperature: Softmax converts each class logit into a probability by comparing it with the other logits.The resulting probabilities are the basis for the soft targets used in distillation.
  • Temperature: Increasing the temperature above 1 produces a softer probability distribution over classes.The paper denotes temperature by T and normally sets it to 1 for ordinary prediction.
  • Transfer objective: Basic distillation trains the student on teacher-generated soft targets at high temperature, then uses temperature 1 after training.The same high temperature is used for the teacher’s targets and the student’s transfer training.
  • Transfer objective: When labels are available, combining soft-target cross-entropy with hard-label cross-entropy improves distillation.The soft-target term uses the high temperature, while the hard-label term uses the student’s logits at temperature 1; the soft-target gradients are scaled by T^2.
  • Logit matching: In the high-temperature limit, distillation minimizes 1/2(z_i − v_i)^2 when logits are separately zero-meaned for each transfer case.Here z_i are student logits and v_i are teacher logits.
  • Logit matching: Intermediate temperatures work best when the student is too small to capture all teacher knowledge, suggesting that ignoring very negative logits can help.At lower temperatures, distillation pays less attention to logits far below the average, which may be noisy or informative.

3 Preliminary experiments on MNIST

MNIST experiments show that soft targets transfer substantial knowledge from a strongly regularized large network to a smaller one. Distillation remains effective with a class absent from the transfer set, while the best temperature depends on student capacity.

  • Distillation performance: 74 test errors for the smaller network after matching soft targets at temperature 20 transferred much of the large network’s advantage.The transfer also conveyed generalization learned from translated training data, despite no translations in the transfer set.
  • Temperature sensitivity: Temperatures above 8 performed similarly for students with at least 300 units per hidden layer.This result contrasts with the sharper temperature dependence of much smaller students.
  • Temperature sensitivity: For students with 30 units per hidden layer, temperatures from 2.5 to 4 significantly outperformed higher or lower temperatures.The preferred temperature therefore changed when student capacity was radically reduced.
  • Missing classes: With all 3s absent from transfer training, bias adjustment yielded 98.6% accuracy on test 3s.The distilled model initially made 133 errors among 1010 test 3s, but increasing the 3-class bias by 3.5 reduced this to 14 errors.
  • Missing classes: Using only 7s and 8s in the transfer set produced 47.3% test errors, reduced to 13.2% after adjusting their biases.The biases for 7 and 8 were reduced by 7.6 to optimize test performance.

4 Experiments on speech recognition

The speech-recognition experiments test whether distillation can compress an ensemble of DNN acoustic models into one model while retaining its gains. The distilled model transfers most of the ensemble’s frame-accuracy improvement and also transfers its WER improvement.

  • Acoustic modeling: The ASR system uses a DNN to map temporal acoustic features to HMM-state probabilities, which a decoder uses to produce transcriptions.Training commonly minimizes frame-level cross-entropy against forced-alignment state labels.
  • Experimental setup: The baseline has 8 hidden layers of 2560 rectified linear units, 14,000 HMM-state labels, about 85M parameters, and 2000 hours of training data.It achieves 58.9% frame accuracy and 10.9% WER on the development set.
  • Results: The distilled single model performs about as well as the averaged predictions of the 10 models used to produce its soft targets.This indicates that the compression retains the ensemble’s reported performance while avoiding ensemble inference.
  • Experimental setup: Ten separately initialized models form an ensemble whose averaged predictions significantly outperform individual models.The models use the same architecture and training procedure, with diversity arising from different random initializations.
  • Results: More than 80% of the ensemble’s frame-classification accuracy improvement transfers to the distilled model trained from its soft targets.The ensemble’s WER improvement also transfers, although the WER gain is smaller because the training objective is mismatched with the ultimate objective.
  • Related work: A related temperature-1 approach using a large unlabeled dataset transferred only 28% of the large-small hard-label error gap.The comparison concerns another reported acoustic-model distillation method.

5 Training ensembles of specialists on very big datasets

Specialist ensembles address the excessive training cost of large ensembles by focusing models on confusable class subsets, while retaining a generalist for broad coverage. On JFT, specialists were trained independently and rapidly, and combining 61 specialists improved overall test accuracy by 4.4% relatively.

  • Motivation: Large-neural-network ensembles can exploit parallel computation, but their training cost becomes excessive on very large datasets.The JFT baseline used 100 million labeled images and training over a large number of cores, making additional ensemble training impractical at the same scale.
  • Specialist design: Specialist ensembles reduce computation by training models on highly confusable subsets of classes while retaining a generalist model for uncovered classes and specialist selection.Specialists use a smaller softmax with a dustbin class for classes outside their focused subset.
  • Training: Specialists initialized from the generalist are trained with balanced focused and random examples, while dustbin-logit correction compensates for the biased training distribution.This initialization and sampling strategy is used to reduce overfitting and share lower-level feature detectors.
  • Specialist design: Specialist class groups are derived by clustering the covariance matrix of generalist predictions, identifying classes often predicted together without requiring true labels.An online K-means algorithm applied to covariance-matrix columns produced reasonable clusters.
  • Inference: Inference first selects the generalist’s most probable classes, then activates specialists whose focused class subsets intersect that candidate set.With n = 1, the active specialist set includes models whose subsets overlap the generalist’s top predicted class.
  • Results: 4.4% relative improvement in overall test accuracy was achieved with 61 specialist models, which trained in a few days instead of many weeks for JFT.The specialists were trained completely independently, making the training process easy to parallelize.
  • Results: Accuracy improvements were generally larger for classes covered by more specialists.The JFT experiments used 61 specialists, each covering 300 classes plus a dustbin class.

6 Soft Targets as Regularizers

Soft targets let a model trained on only 3% of the speech data recover nearly all information from the full set, while specialist models face overfitting risks that soft-target training may address.

  • 57% was reached with soft targets using only 3% of the speech data, whereas hard-target training peaked at 44.5% after early stopping.The baseline speech model had 85M parameters, and the soft-target system converged without early stopping.
  • Soft targets communicated regularities learned from the full dataset to another model, recovering performance within about 2% of full-data training.
  • Specialists trained on highly enriched data tend to overfit their special classes because their effective training sets are much smaller.Making specialists smaller sacrifices helpful transfer effects from modeling non-specialist classes.
  • Initializing a specialist from a generalist and matching soft targets for non-special classes may preserve the generalist’s knowledge, but this approach remains under exploration.

7 Relationship to Mixtures of Experts

Specialist models resemble mixture-of-experts systems in focusing on subsets of classes, but their subsets are defined from a generalist’s confusion matrix and their training is independent.

  • Mixtures of experts use a gating network that assigns examples to experts while learning from their relative discriminative performance.
  • Specialist subsets are defined from the generalist’s confusion matrix, after which specialists can be trained entirely independently.
  • At test time, the generalist’s predictions determine which specialists are relevant, so only those specialists need to run.

8 Discussion

The paper shows that distillation transfers ensemble knowledge into smaller, easier-to-deploy models, while specialist nets improve very large models but have not yet been distilled back into them.

  • Nearly all ensemble-derived improvement in a deep acoustic model was distilled into a same-size neural net that is far easier to deploy.
  • Distillation transferred knowledge from ensembles or highly regularized large models into smaller distilled models.
  • A single very large net improved significantly by learning specialist nets for highly confusable class clusters when training a full ensemble was infeasible.
  • The paper had not yet shown that specialist knowledge could be distilled back into the single large net.
Loading 1503.02531v1…