PaperlayerView exact revision on arXiv

Source-linked AI summary

Introspection: Accelerating Neural Network Training By Learning Weight Evolution

Abhishek Sinha, Mausoom Sarkar, Aahitagni Mukherjee, Balaji Krishnamurthy

arXiv:1704.04959v1cs.LG

TL;DR

Training deep neural networks is time-consuming because many parameters must be learned. This paper trains a predictor on weight evolution from one network and uses it to forecast weights in other networks; it reports faster convergence across MNIST, CIFAR-10, and ImageNet, with limitations around jump timing and generalization.

  • Problem

    Training deep networks is time-consuming because they require learning many parameters, creating a bottleneck for large-scale applications.

    Source: §1 · p. 1

  • Method

    A neural network trained on weight-evolution histories predicts future weights in other in-training networks to accelerate their convergence.

    Source: §1 · p. 1, §3.1 · p. 4, §4.1 · p. 5

  • Results

    Introspection led to faster convergence across MNIST, CIFAR-10, and ImageNet; on CIFAR-10, 85.7% validation accuracy took approximately 28,000 steps with introspection versus 40,000 iterations with normal SGD.

    Source: §4.2.2 · p. 9, §6 · p. 13

  • Takeaways & Limitations

    The method has a small memory footprint, is computationally efficient, and can be used in practical settings.

    Source: §6 · p. 13

    Early jumps while training AlexNet tended to degrade final outcomes, and the predictor's generalization and optimal jump points remain open questions.

    Source: §5 · pp. 12–13

Abstract

from arXiv · show

Neural Networks are function approximators that have achieved state-of-the-art accuracy in numerous machine learning tasks. In spite of their great success in terms of accuracy, their large training time makes it difficult to use them for various tasks. In this paper, we explore the idea of learning weight evolution pattern from a simple network for accelerating training of novel neural networks. We use a neural network to learn the training pattern from MNIST classification and utilize it to accelerate training of neural networks used for CIFAR-10 and ImageNet classification. Our method has a low memory footprint and is computationally efficient. This method can also be used with other optimizers to give faster convergence. The results indicate a general trend in the weight evolution during training of neural networks.

1 INTRODUCTION

Training deep neural networks is time-consuming because they require learning many parameters, making training time a bottleneck. The paper proposes predicting weights of in-training networks to accelerate convergence, with low memory use and parallel layer updates.

1 INTRODUCTION

  • Training deep networks can take days even on GPUs, making training time a bottleneck for large-scale applications.The introduction calls for more efficient optimization to reduce training time.Source: §1 · p. 1
  • Even newer optimizers such as Adagrad, Adadelta, RMSProp, and Adam leave a need for faster convergence.The paper describes an optimizer as iteratively updating network weights to minimize an objective function.Source: §1 · p. 1
  • The proposed method predicts weights of other in-training networks to accelerate convergence, using little memory and updating all layers in parallel.Source: §1 · p. 1

2 RELATED WORK

The paper positions its approach as forecasting future weights from weight history rather than gradients, and as reusing learned weight-evolution information to accelerate novel networks. It also describes compatibility with existing optimizers and contrasts its approach with learned optimizers and reinforcement-learning-based tuning.

2 RELATED WORK

  • The method can be used alongside Momentum, AdaGrad, AdaDelta, RMSProp, and Adam (Kingma & Ba, 2014) to further improve convergence time.Source: §2 · p. 2
  • Unlike learned update functions that take gradients as input, the forecasting network uses weight history to predict values many steps ahead and can be invoked asynchronously.The paper also states that this approach generalizes across architectures and datasets without additional retraining and has a smaller memory footprint than Andrychowicz et al. (2016).Source: §2 · p. 2
  • Unlike Q-gradient descent, which trains a Deep-Q Network using data from a target network, the paper's pre-trained forecasting network can optimize a network without training on its data.Source: §2 · p. 2
  • The paper distinguishes its approach from prior methods by using information from existing networks' training to accelerate novel networks.It also contrasts with synthetic-gradient prediction, which still estimates gradients, whereas its update method does not.Source: §2 · p. 2

3 PATTERNS IN WEIGHT EVOLUTION

The paper reports a general trend in weight evolution across tasks, layers, and optimization rules: most weights change little, while the few with large changes follow predictable trajectories. It uses histories of weight evolution to train a predictor that forecasts future weights in unseen networks.

3 PATTERNS IN WEIGHT EVOLUTION

  • Weight evolution followed a general trend across tasks, layers, and optimization rules, and a major proportion of weights changed little.The authors analyze MNIST and CIFAR-10, fully connected and convolutional networks, and different optimization rules.Source: §3 · pp. 2–3
  • The analysis measures change as final-minus-initial weight values and oscillation as the square root of the second moment about the initial value.The reported histograms indicate that most weights undergo neither significant magnitude changes nor significant oscillations.Source: §3 · p. 2, Figure 4 · p. 4, Figure 5 · p. 4
  • A small subset of weights changed substantially, and those weights tended to increase or decrease predictably as training progressed.The paper illustrates sampled histories from MNIST convolutional and fully connected networks and a CIFAR-10 convolutional network.Source: §3 · p. 3, Figure 1 · p. 3, Figure 2 · p. 3

3.1 WEIGHT PREDICTION

  • The authors train a forecasting network on collected weight histories to predict future values of weights in an unseen network during its training.The predictor's inference time is described as negligible relative to the target network's training time.Source: §3.1 · pp. 3–4
  • The forecasting network is a one-layer feedforward network with four history inputs, a 40-neuron fully connected ReLU hidden layer, and one predicted-weight output.Four history samples were the minimum number for which its training converged in the reported experiments.Source: §3.1 · p. 4
  • After a prediction jump, the target network continues training normally with SGD until the predictor is used again.Source: §3.1 · p. 4

4 EXPERIMENTS

The experiments train an introspection network on MNIST weight histories to predict future weight values, then use it to update weights in other networks. Training data are balanced across weight histories with different amounts of variation, and the network is trained to predict a weight at step 2t.

4.1 TRAINING OF INTROSPECTION NETWORK

  • The introspection network is trained on weight histories from an MNIST network with three convolutional and two fully connected layers, using Adam.The source network uses ReLU activations, max pooling after convolutional layers, dropout after the first fully connected layer, a learning rate of 1e-4, and batch size 50.Source: §4.1 · p. 4
  • The report includes log-frequency distributions of weight changes for MNIST and CIFAR-10 networks and of the square root of the second moment about initialized weights.The described cases include a fully connected MNIST network trained with Adam and a CIFAR-10 CNN trained with SGD.Source: Figure 6 · p. 5, Figure 8 · p. 5, Figure 7 · p. 5
  • Training examples pair four values from a weight’s history with its value at step 2t as the prediction target.The inputs are the weight at steps t, 7t/10, and 4t/10, plus its initialized value; predicting at 2t was an empirical choice, and k = 2.2 was slightly better on MNIST validation than k = 2.0, which was more consistent across training.Source: §4.1 · pp. 4–5
  • To prepare about 0.8 million training examples, weight histories are sampled in proportions that favor weights with larger changes while retaining examples from less-varying weights.The selection is 50% from the top 50th percentile by variation, 25% from the next 25th percentile, and 25% from the remainder.Source: §4.1 · p. 5
  • Figure 10 illustrates an example of a weight update using the introspection network.Source: Figure 10 · p. 6
  • The introspection network’s final training loss was 3.1 and validation loss was 3.4, corresponding to average L1 weight prediction errors of 0.0031 and 0.0034, respectively.Weight values were multiplied by 1000 before being input to the network.Source: §4.1 · p. 6

4.2 USING PRE-TRAINED INTROSPECTION NETWORK TO TRAIN UNSEEN NETWORKS

A single introspection network trained on MNIST weight histories is applied at intermediate training steps to guide other networks. Across MNIST, CIFAR-10, and ImageNet, these updates reach reported accuracy targets in fewer training steps than ordinary training.

4.2 USING PRE-TRAINED INTROSPECTION NETWORK TO TRAIN UNSEEN NETWORKS

  • The same introspection network, trained on MNIST weight evolution, is applied to unseen networks at intermediate jump points; later training steps generally yield a better accuracy increase.The jump schedule depends on the distribution of training steps used to train the introspection network.Source: §4.2 · p. 6

4.2.1 MNIST

  • MNIST1 reached 98.22% validation accuracy in 20,000 steps with introspection, versus 97.22% in the same steps with normal training; MNIST2 reached 99.11% in 3,500 versus 6,000 steps.For MNIST1, normal training reached 97.92% in the same 251 seconds used by introspection.Source: Figure 11 · p. 7, Figure 12 · p. 7, §4.2.1 · p. 7
  • After 15,000 steps, MNIST3 reached 96.89% with introspection versus 95.71% with normal Adam; MNIST4 reached 98.85% versus 98.65% after 7,000 steps.The MNIST4 result applies an introspection network trained with ReLU to an RNN using sigmoid and tanh activations.Source: Figure 13 · p. 8, Figure 14 · p. 8, §4.2.1 · p. 8

4.2.2 CIFAR-10

  • CIFAR1 reached 85.7% validation accuracy in approximately 28,000 steps with introspection, versus 40,000 steps with normal SGD across all four jump schedules.The choice of jump points affected final accuracy, but not greatly; later jumps generally performed better.Source: Figure 16 · p. 9, Figure 18 · p. 9, Figure 17 · p. 9

4.2.3 IMAGENET

  • For AlexNet1 on ImageNet, introspection reduced the steps to 72% top-5 accuracy from 196,000 to 179,000, with negligible prediction time relative to SGD.The validation-accuracy and loss plots are shown in Figures 20 and 21.Source: Figure 20 · p. 10, §4.2.3 · p. 10, Figure 21 · p. 10

4.3 COMPARISON WITH BASELINE TECHNIQUES

The section compares introspection-based weight updates with other optimizers and simple heuristics for updating weights during training.

  • The comparison considers other optimizers and simple weight-update heuristics as alternatives to updates by the introspection network.Source: §4.3 · p. 10

4.4 COMPARISON WITH ADAM OPTIMIZER

Introspection updates are compared with Adam and with quadratic and linear weight extrapolation baselines. The reported comparisons favor introspection in convergence speed or robustness, while linear fitting shows no significant improvement.

4.4 COMPARISON WITH ADAM OPTIMIZER

  • On MNIST1 and MNIST3, Adam with introspection reached normal Adam’s maximum accuracy in 7,200 versus 10,000 steps and 8,800 versus 15,000 steps, respectively.Adam outperformed normal SGD and SGD with introspection, but introspection also accelerated Adam.Source: §4.4 · p. 10, Figure 22 · p. 11, Figure 23 · p. 11

4.4.1 FITTING QUADRATIC CURVE

  • On MNIST1, introspection reached 99.21% maximum accuracy versus 99.19% for quadratic fitting, whose best jump ratio required experimentation and did not generalize to CIFAR-10.Quadratic updates could sharply reduce accuracy by up to 9.8%, with training often not recovering.Source: Figure 24 · p. 11, §4.4.1 · p. 11
  • On CIFAR-10, introspection reached 85.6% maximum accuracy after 25,500 steps, versus 85.45% after 27,200 steps for quadratic fitting.For the normal-SGD accuracy of 85.29%, introspection reached the target in 21,200 steps, versus 26,500 for SGD and 27,000 for quadratic updates.Source: Figure 25 · p. 11, §4.4.1 · p. 11

4.4.2 FITTING LINEAR CURVE

  • Linear fitting on MNIST1 showed no significant performance improvement, so the experiment was not repeated on CIFAR-10.Higher tested jump ratios caused divergence after two or three jumps.Source: Figure 26 · p. 12, §4.4.2 · p. 12

4.5 LINEAR INTROSPECTION NETWORK

A linear introspection network and Gaussian-noise weight updates were tested on MNIST1. Neither baseline showed the performance needed to justify testing it on other datasets.

4.5 LINEAR INTROSPECTION NETWORK

  • Removing ReLU gave the linear introspection network some advantage over normal SGD, but it underperformed the standard introspection network and was not tested on other datasets.Source: §4.5 · p. 12

4.5.1 ADDING NOISE

  • Gaussian-noise weight updates showed no significant improvement on MNIST, so the experiment was not extended to CIFAR-10.The updates added small zero-mean Gaussian noise at two standard deviations.Source: Figure 28 · p. 12, §4.5.1 · p. 12

5 LIMITATIONS AND OPEN QUESTIONS

The section identifies unresolved questions about when to apply weight-update jumps and how broadly the introspection network generalizes. It also reports that very early jumps during AlexNet1 training tended to hurt final outcomes.

  • Optimal jump points and generalization to RNNs and non-image tasks remain open questions.Source: §5 · p. 12
  • Applying jumps in very early AlexNet1 training steps tended to degrade final outcomes.Source: §5 · p. 13
  • The authors suggest that the early-jump issue may reflect a simple introspection network trained only on MNIST weight evolution, and that a more powerful network trained on diverse data may ameliorate it.Source: §5 · p. 13

6 CONCLUSION

The conclusion describes a neural network trained on one network’s weight evolution and then used to accelerate training across other tasks and architectures. It reports faster convergence across the tested cases and a general trend in weight evolution.

  • A network trained on one neural network’s weight evolution was used to update deep networks across three tasks and varied architectures, activations, optimizers, and normalization strategies.Source: §6 · p. 13
  • The introspection network led to faster convergence than existing methods in all reported cases.Source: §6 · p. 13
  • The reported results indicate a general underlying pattern in neural-network weight evolution, alongside a small memory footprint and computational efficiency.Source: §6 · p. 13

A APPENDIX

Preliminary Inception V1 results on ImageNet report faster accuracy gains with introspection, but very early jumps did not produce eventual gains. Training was still in progress.

A APPENDIX

  • These were preliminary Inception V1 results on ImageNet using an introspection network trained on MNIST weight evolution, with network training still in progress.Source: A APPENDIX · p. 14
  • At least 730,000 steps were gained; normal training reached 68.40% maximum accuracy after about 1.5 million steps, while introspection every 300k steps reached 69.06%.With jumps at 200k, 400k, and 600k steps, maximum accuracy was 68.69%, and the normal-training model’s 68.40% was reached after 944k steps.Source: A APPENDIX · p. 14, Figure 29 · p. 14, Figure 30 · p. 14
  • Choosing jump points early in training did not lead to eventual gains despite an initial accuracy increase; test accuracy flattened after the early jumps.Source: A APPENDIX · p. 14, Figure 31 · p. 15

Related papers

Loading 1704.04959v1…