Source-linked AI summary
Unifying distillation and privileged information
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, Vladimir Vapnik
TL;DR
The paper asks how machines can learn from other machines when useful representations or explanations are available during training but not at test time. It unifies distillation and privileged information as generalized distillation, analyzes why it works, and extends it across learning settings. Numerical simulations report improved performance, including distilled accuracy near privileged accuracy and teacher-matched multitask performance.
Problem
The paper addresses how to exploit teacher-provided representations or explanations that are available during training but unavailable at test time.
Method
It frames distillation and privileged information as generalized distillation, training a teacher on privileged data and transferring soft predictions to a student using regular inputs and hard labels.
Results
Across simulations, generalized distillation is effective; relevant-feature classification reached 97±1% distilled accuracy versus 89±1% regular accuracy, while multitask distillation matched teacher performance.
Takeaways & Limitations
Generalized distillation provides a common framework for leveraging multiple machines and data representations in semi-supervised, multitask, curriculum, and reinforcement learning scenarios.
Takeaways & Limitations
Privileged information is unavailable at test time, so the student must rely on regular features after training.
Abstract
from arXiv · showhide
Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines. This paper unifies these two techniques into generalized distillation, a framework to learn from multiple machines and data representations. We provide theoretical and causal insight about the inner workings of generalized distillation, extend it to unsupervised, semisupervised and multitask learning scenarios, and illustrate its efficacy on a variety of numerical simulations on both synthetic and real-world data.
1 INTRODUCTION
The paper motivates machines-teaching-machines by showing how teacher-provided information can accelerate learning, then unifies distillation and privileged information as generalized distillation.
- Privileged information consists of additional teacher-provided explanations available during training but unavailable to the learner at test time.
- The biopsy example represents regular features as images, privileged information as oncologist reports, and labels as cancerous-versus-healthy tissue.
- Teacher guidance can potentially accelerate learning from O(n^-1/2) to O(n^-1), reducing the examples needed for comparable accuracy from 10^6 to 1000.
- Learning using privileged information and distillation are presented as instances of machines-teaching-machines, where machines learn from other machines alongside training data.
- The proposed framework, generalized distillation, supports learning from multiple machines and data representations and connects to semi-supervised, domain-adaptation, transfer, Universum, reinforcement, and curriculum learning.
2 DISTILLATION
Distillation transfers a flexible teacher’s learned representation to a simpler student that combines soft teacher predictions with hard labels, reducing prediction costs for production systems.
- The framework focuses on c-class classification with softmax outputs, cross-entropy loss, and an increasing regularizer, while noting that the ideas also apply to regression.
- Distillation is related to model compression and is adopted as the manuscript’s term for this transfer process.
- Distillation transfers a flexible teacher function into a simpler student function for computationally cheaper prediction.The motivation is that ensemble prediction at test time can be prohibitive for production systems.
- The student learns from both teacher soft predictions and true hard labels through an imitation parameter λ ∈ [0, 1].
- The temperature parameter T > 0 controls the softness of the teacher’s class-probability predictions.Higher temperatures produce softer predictions that expose additional label information.
3 VAPNIK’S PRIVILEGED INFORMATION
Privileged information is used to train a teacher on information unavailable at test time, and distillation transfers its useful knowledge to a student using regular features.
- The central problem is building a better test-time classifier from privileged information x⋆ unavailable to the student at deployment.
- Predicting privileged representations from regular inputs can be cumbersome because medical reports may be harder to predict than cancer classification.
- The proposed procedure trains a teacher on privileged inputs, computes soft labels, and distills them into a student using regular features and hard labels.
- A teacher is helpful when the capacity of its correcting functions is much smaller than the capacity of the student decision boundary.
- Privileged information is assumed to support accurate teacher estimation from small sample sizes when the teacher function class has small capacity.
- Distillation differs from similarity control by supporting arbitrary learning algorithms, selectable student complexity, and sequential rather than simultaneous teacher-student training.
4 GENERALIZED DISTILLATION
Generalized distillation unifies distillation and privileged information as machines teaching machines through multiple representations. It provides a three-step framework, theoretical and causal explanations, and extensions to semi-supervised, multitask, curriculum, and reinforcement learning.
- Generalized distillation: Generalized distillation learns a teacher from privileged representations, obtains soft predictions, and trains a simpler student from regular inputs, labels, and those predictions.The numerical simulations implement these steps sequentially, although they could also be jointly optimized.
- Generalized distillation: The framework reduces to Hinton’s distillation when privileged and regular representations coincide and the student class has much lower capacity than the teacher class.It reduces to privileged-information learning when the teacher uses a privileged description and has much greater capacity than the student.
- Why generalized distillation works: Teacher benefits arise when teacher capacity is small, teacher approximation error is lower, and the learning exponent satisfies α > 1/2.The analysis identifies the strongest benefits in low-data regimes and connects these conditions to assumptions in privileged information and distillation.
- Why generalized distillation works: Soft labels can expose label dependencies and uncertainty, while smoothing the decision boundary may let students focus on easier examples.The imitation parameter λ balances imitation against prediction of hard labels, allowing fallback to regular learning.
- Extensions: Semi-supervised generalized distillation applies the same three steps to clean subsets, allowing teachers to produce soft labels for unlabeled examples.The framework also accommodates missing regular features, privileged features, labels, or combinations of these elements.
- Extensions: In multitask settings, source-task modalities or labels can serve as privileged information, while teacher uncertainty can rank examples for curriculum learning.The paper also describes distillation as resembling imitation in reinforcement learning and discusses causal conditions affecting when privileged information helps.
5 NUMERICAL SIMULATIONS
The simulations evaluate when distilling privileged information helps by comparing teacher, regular-feature, and distilled classifiers across synthetic experiments. One experiment shows that clean decision-boundary information can nearly match teacher performance after distillation despite corrupted labels.
- Experimental setup: The experiments use synthetic logistic-regression simulations repeated over 100 random partitions, with 200 training and 10,000 testing samples.Regular features have dimensionality d = 50, and the comparisons include privileged, regular, and distilled classifiers.
- Clean labels as privileged information: 96 ± 0% privileged accuracy, 88 ± 1% regular accuracy, and 95 ± 1% distilled accuracy were obtained when clean decision-boundary distances accompanied corrupted labels.The result illustrates effective detection of outliers in label space through distillation of privileged information.
2. Clean features as privileged information
When privileged explanations are clean versions of noisy regular features, the privileged classifier outperforms both regular and distilled classifiers, but the distillation improvement is not statistically significant.
- Clean features as privileged information: 90 ± 1% privileged accuracy, 68 ± 1% regular accuracy, and 70 ± 1% distilled accuracy were obtained on the test set.The privileged explanations are clean versions of the regular features available at test time.
- Clean features as privileged information: The distilled improvement over regular features is not statistically significant because the explanations are independent of the noise corrupting the regular features.Consequently, the teacher provides no additional transferable information in this setup.
3. Relevant features as privileged information
Using relevant variables as privileged information can substantially improve classification after distillation, approaching the performance achieved with privileged features directly.
- 98 ± 0% privileged accuracy, 89±1% regular accuracy, and 97±1% distilled accuracy demonstrate effective feature selection through privileged-information distillation.The teacher explanations identify three variables relevant for classification, reducing the dimensionality learned from.
4. Sample-dependent relevant features as privileged information
Sample-dependent privileged features can help distillation, but their benefit depends on how they relate to the target and on the learning setup.
- Sample-dependent relevant features: 96 ± 2% privileged accuracy, 55 ± 3% regular accuracy, and 0.56 ± 4% distilled accuracy illustrate performance under sample-dependent relevant features.The privileged features are interpreted as pixels inside bounding boxes, while the misspecified regular function class does not deteriorate final performance through distillation.
- Causal interpretation: Privileged features that add no information about the target-function mapping justify the observed lack of improvement.This is presented as a consequence of the paper’s causal hypothesis.
- Causal interpretation: Adding information about the target function is beneficial for distillation in some experiments but is not sufficient for improvement in the fourth example.
- Experimental comparisons: Figure 1 reports MNIST results for training with 300 samples on the left and 500 samples on the right.
- Experimental comparisons: Figure 2 reports results for CIFAR 10 on the left and SARCOS on the right.
5. MNIST handwritten digit image classification
The MNIST experiment uses high-resolution images as privileged features and downscaled images as regular features to evaluate generalized distillation with limited training data.
- The teacher receives original 28x28 MNIST images, while the student receives the same images downscaled to 7x7 pixels.Both models use 300 or 500 training samples and are evaluated across temperature and imitation levels.
6. Semisupervised learning
The CIFAR10 experiment tests generalized distillation for semisupervised learning by using privileged clean images to provide soft labels for a large unlabeled training set.
- The teacher trains on 300 labeled clean CIFAR10 images and produces soft labels for all 50,000 training images.The student distills the 300 original hard labels together with the teacher’s soft labels while using noisy images as regular features.
7. Multitask learning
Generalized distillation is applied to SARCOS as a multitask learning problem, using teachers trained to predict robotic-arm torques and transferring their knowledge to a student. At the proper temperature, the student matches the teacher’s performance.
- The SARCOS dataset contains 7 robotic-arm joint-torque regression tasks based on 21 real-valued features.
- Teachers trained on 300 samples predict each torque from the other six torques, while the student uses the 21 regular features.
- At the proper distillation temperature, the student matches the teacher’s performance.The reported performance measure is mean squared error.