Source-linked AI summary
Born Again Neural Networks
Tommaso Furlanello, Zachary C. Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar
TL;DR
Knowledge distillation is usually used to compress a high-capacity teacher into a smaller student, but this paper tests whether identical-capacity students can also benefit. It introduces Born-Again Networks and finds teacher-surpassing students across vision and language-modeling settings, while probing the roles of confidence weighting and dark knowledge.
Problem
Knowledge distillation is commonly studied as a way to transfer a high-capacity model’s knowledge into a more compact student, leaving its benefits apart from compression less established.
Method
The paper trains identical-architecture or similar-capacity students using teacher output distributions alongside label supervision, and analyzes dark knowledge with CWTM and DKPP.
Results
BAN students consistently outperform their teachers across computer vision and language modeling, including 16.95% error with 6.5 M parameters versus 16.87% for a teacher with almost eight times more parameters.
Takeaways & Limitations
Knowledge distillation can improve students without requiring a stronger or larger teacher, including across comparable but different architectures.
Takeaways & Limitations
The LSTM experiments did not use the state-of-the-art training techniques or the recently introduced methods noted by the authors.
Abstract
from arXiv · showhide
Knowledge Distillation (KD) consists of transferring âknowledgeâ from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the studentâs compactness, without sacrificing too much performance. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating the effect of the teacher outputs on both predicted and non-predicted classes.
1. Introduction
The paper revisits knowledge distillation without compression, training identical-capacity students that can outperform their teachers. It introduces Born-Again Networks and examines how teacher outputs contribute to distillation.
- Motivation: Knowledge distillation is reframed from model compression to transferring knowledge between models with identical capacity.The goal is to disentangle distillation’s benefits from using smaller students.
- Born-Again Networks: Born-Again Networks train a new student after teacher convergence using both correct labels and the teacher’s output distribution.Each student is initialized with a different random seed and trained from the earlier generation’s supervision.
- Results: BANs consistently achieve lower validation errors than their teachers across DenseNets, ResNets, and LSTM-based sequence models.For DenseNets, the procedure can be repeated across multiple generations, although returns diminish.
- Distillation Components: The analysis decomposes the KD gradient into dark knowledge from wrong outputs and a ground-truth component related to teacher-confidence weighting.The experiments aim to quantify the contribution of dark knowledge to KD’s success.
- Cross-Architecture Transfer: BANs can transfer knowledge across comparable architectures: BAN-ResNets surpass DenseNet teachers, while DenseNet students improve over standard ResNets.These results indicate that KD can improve students even when the teacher is weaker.
2. Related Literature
The related literature frames knowledge transfer around compression, interpretability, and specialized distillation objectives. This paper distinguishes its approach by showing that standard dark-knowledge distillation can improve identical-architecture students across domains.
- Knowledge Transfer Goals: Earlier work transferred knowledge to compact models to retain larger models’ accuracy under resource constraints.Other approaches targeted transparency or interpretability by approximating neural networks with simpler models.
- Distillation Methods: Knowledge distillation commonly trains students to match a teacher’s full softmax distribution, including information in non-predicted classes.Subsequent work also matched attention maps, input derivatives, or ensembles of teacher models.
- Broader Applications: KD has been applied beyond supervised learning, including reinforcement learning, continual learning, and adversarial training.These applications use distillation for model consolidation, reducing forgetting, or incorporating adversarial objectives.
- Closest Prior Work: Yim et al. reported faster training and greater accuracy for identical-architecture students using a loss based on matching activation-statistic relationships.Their method used inner products between activation tensors at paired layers.
- Key Differences: This work differs by demonstrating that dark knowledge enables self-distillation gains without softening logits and by evaluating the effect beyond images.The authors also report current second-best CIFAR-100 performance for identical-architecture distillation.
3. Born-Again Networks
Born-Again Networks use KD to train new models from teacher outputs, including sequential generations, ensembles, and analyses separating dark knowledge from confidence weighting. The experiments extend transfer across architectures and model complexities.
- Core Formulation: Born-Again Networks modify the original loss with cross-entropy between student outputs and the outputs of an original teacher.The paper focuses on identical architectures and also studies similar-capacity models with different architectures.
- Sequential Training and Ensembles: Sequential BAN training transfers knowledge from each model to the next generation, followed by prediction averaging across multiple generations for BANE.The sequence’s improvements saturate, whereas ensembling produces significant gains.
- Gradient Decomposition: KD gradients separate into a correct-class component and dark knowledge from wrong outputs across the minibatch.The teacher’s probability for the correct class can act as a weighting factor on the ground-truth gradient.
- Confidence Weighting: When teacher confidence is near 1, the correct-class gradient approaches the ground-truth gradient; lower-confidence samples receive reduced gradient contributions.This interprets teacher confidence as a sample-level rescaling factor.
- Distillation Treatments: CWTM weights ground-truth cross-entropy examples by teacher confidence, while DKPP permutes non-argmax outputs to destroy their original class associations.DKPP therefore tests whether dark knowledge depends on pairwise similarities among output categories.
- Architectural Scope: DenseNet architectural changes trade off parameters, memory use, and sequential operations, while DenseNet teachers are also used to improve comparable ResNet students.The study varies depth, growth, compression, and residual-block design to construct comparable models.
4. Experiments
The experiments evaluate Born-Again Networks across architectures, distillation variants, and language modeling, including tests of whether teacher outputs help students beyond conventional dark knowledge.
- Distillation objectives: CWTM retains only the teacher’s argmax output and uses its maximum value as a sample-importance weight.The weighted value rescales each student loss term.
- Distillation objectives: DKPP preserves the teacher-output moments while randomly permuting every dimension except the argmax dimension.The remaining training scheme and architecture are unchanged.
- Distillation objectives: Both CWTM and DKPP alter covariance between outputs, so their improvements cannot be fully attributed to the classical dark-knowledge interpretation.
- Cross-architecture experiments: BAN-ResNets share the teacher’s first and last layers while varying residual depth and, in some cases, halving network width.The experiments match channel counts to corresponding DenseNet block outputs and explore mostly equal-width architectures.
- Cross-architecture experiments: DenseNet students trained from Wide-ResNet teachers test whether weaker masters can improve a different architecture, using multiple teacher configurations.
- Language modeling: The BAN framework is also evaluated on Penn Tree Bank with a 1500-unit single-layer LSTM and a smaller CNN-highway-LSTM model.For both language models, KD is applied between softmax outputs of the unrolled teacher and student.
5. Results
Across vision and language-model experiments, Born-Again students generally improve over their teachers, including across architectures and altered network configurations. The results also show that teacher outputs support gains beyond ordinary label supervision, though language models require both teacher outputs and labels.
- Students improve over their teachers across configurations, including identical-architecture knowledge distillation.
- 16.95% error is reached by the smallest BAN-DenseNet-112-33 using only 6.5M parameters, versus 16.87% for its teacher with almost eight times more parameters.
- 15.5% error is achieved by BAN-3-DenseNet-80-80 with 22M parameters on CIFAR-100.
- 15.5% error is the best single-model CIFAR-100 result, while the largest BAN ensemble reaches 14.9% error.
- Teacher-output variants CWTM and DKPP generally improve over baselines, while three of four CWTM models improve despite removing wrong-logit information.
- 17.39% and 17.13% test errors are reached by BAN-Pre-ResNet and BAN-Wide-ResNet students with parameter budgets comparable to their DenseNet teachers.
- 16.79% error is obtained by a DenseNet-90-60 student trained from a Wide-ResNet-28-10 teacher, improving over simple label supervision.
- Test perplexity decreases from 80.05 to 76.97 for the smaller BAN-LSTM-CNN and from 71.87 to 68.56 for the larger BAN-LSTM.
6. Discussion
Minsky’s sequence of teaching selves offers an analogy for Born-Again Networks: newer models may be trained under guidance from older internal models. The analogy frames long-term identity as an ensemble of multiple generations.
- Minsky connected sudden childhood intelligence spurts to hidden training of new student models guided by an older self.
- The analogy treats a long-term self as an ensemble of multiple generations of internal models.
- The current model can use earlier internal models for guidance.