Source-linked AI summary
Net2Net: Accelerating Learning via Knowledge Transfer
Tianqi Chen, Ian Goodfellow, Jonathon Shlens
TL;DR
Training each new neural network from scratch wastes time during experimentation. Net2Net transfers knowledge through function-preserving transformations, enabling larger or modified networks to converge faster while retaining accuracy, including 78.5% on ImageNet.
Problem
Training many different neural networks from scratch makes experimentation and model-design workflows wasteful.
Method
Net2Net uses function-preserving transformations to transfer knowledge from smaller or shallower networks into wider or deeper networks.
Results
78.5% on the ImageNet validation set established a new state of the art, while Net2Net-initialized models converged faster than standard or baseline approaches.
Takeaways & Limitations
Net2Net enables faster exploration of model families and reduces the time required for new machine-learning experiments.
Takeaways & Limitations
The demonstrated knowledge transfer applies under some architectural constraints.
Abstract
from arXiv · showhide
We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.
3 EXPERIMENTS
The experiments evaluate Net2Net for widening, deepening, and redesigning Inception networks. Function-preserving initialization accelerates convergence and supports a 78.5% ImageNet validation result without training the larger models from scratch.
- 3 EXPERIMENTS: Net2Net operators were evaluated for widening, deepening, and broader model-design exploration using Inception-BN networks trained on ImageNet.The comparisons included random-pad and random-initialization baselines.
- 3.2 NET2WIDERNET: Net2WiderNet reaches the same final accuracy as random initialization while converging faster, indicating no accuracy loss from mimicking a smaller model.The narrower teacher used 30% of the original parameter count in most layers before initializing a standard-sized student.
- 3.3 NET2DEEPERNET: Net2DeeperNet adds vertical-horizontal convolutional layer pairs initialized to implement identity transformations, yielding good training and validation accuracy faster than random initialization.The inserted layers are complete convolutional layers with rectification and batch normalization.
- 3.2 NET2WIDERNET: 3 × 10^6 minibatches suffice for Net2WiderNet to approach final validation accuracy, whereas the randomly initialized baseline needs roughly 2 × 10^6 additional minibatches.Both approaches reach roughly the same validation-set accuracy.
- 3.4 EXPLORING MODEL DESIGN SPACE WITH Net2Net: 78.5% ImageNet validation accuracy was achieved after enlarging Inception modules, while Net2Net-initialized models converged faster than the standard reference model.The larger models were not trained from scratch because of resource and time constraints.
- 4 DISCUSSION: The discussion concludes that Net2Net can rapidly transfer knowledge to significantly larger networks under architectural constraints and reduce model-experimentation time.The authors call for more general knowledge-transfer methods.
A CHOICE OF HYPERPARAMETERS FOR FINE TUNING Net2Net TRANSFORMED MODELS
Net2Net-transformed models can generally use the hyperparameters employed for training from scratch. The teacher needs no changes, while the student requires only one modification.
- A CHOICE OF HYPERPARAMETERS FOR FINE TUNING Net2Net TRANSFORMED MODELS: Nearly all hyperparameters used to train networks from scratch can also train Net2Net-transformed student networks.The teacher network requires no hyperparameter changes, while the student needs only one modification.
B RELATED WORK
Related work spans architecture growth, knowledge transfer for deeper networks, model compression, and function-preserving transformations. Net2Net differs by rapidly adapting a pre-existing model while preserving its represented feed-forward function, within architectural constraints.
- B RELATED WORK: Cascade-correlation enlarges a pre-existing architecture by adding hidden units, but newly added components create a temporary low-performance period and restrict continued adaptation of earlier units.Net2Net’s function-preserving initialization is presented as avoiding that period, aside from possible brief noise-induced degradation.
- B RELATED WORK: Prior knowledge-transfer methods incrementally train deeper convolutional networks, whereas the authors report training considerably deeper models without transfer using a strong Inception-BN-RMSProp baseline.The standard-size models have 25 layers on their shortest path and 47 on their longest path.
- B RELATED WORK: Model compression transfers knowledge from many models to one model for regularization, while Net2Net uses a pre-existing model to train rapidly without seeking extra regularization.The two techniques therefore pursue different objectives.
- B RELATED WORK: Net2DeeperNet inserts layers initialized as identity functions, extending prior uses of identity weights to function-preserving transformations of pre-existing neural networks.The authors present this use as novel to their knowledge.
- B RELATED WORK: Function-preserving transformations in Net2Net preserve a feed-forward network’s represented function, unlike the distribution-preserving but differently constrained layer growth discussed for deep belief networks.The related comparison notes that DBN growth is known to preserve distributions only for one specific new-layer size.