PaperlayerView exact revision on arXiv

Source-linked AI summary

The Early Phase of Neural Network Training

Jonathan Frankle, David J. Schwab, Ari S. Morcos

arXiv:2002.10365v1cs.LGcs.NEstat.ML

TL;DR

The paper asks what changes deep neural networks undergo during the earliest training phase and how those changes depend on network state and data. Using extensive telemetry and IMP-based perturbation experiments, it finds that deep networks become non-robust to sign-preserving random reinitialization and develop highly non-independent weight distributions. Pre-training with input-distribution information or an auxiliary self-supervised task can approximate supervised changes, although labels accelerate the process.

  • Problem

    The early phase of deep-network training is comparatively underexplored despite evidence that consequential learning changes occur there.

  • Method

    The authors measure early-training network statistics and use IMP with rewinding plus state perturbations to probe weight distributions and data dependence.

  • Results

    Deep networks are not robust to sign-preserving random reinitialization, and early weight distributions are highly non-independent; pre-training using only p(x) can approximate supervised changes but takes roughly 32× longer.

  • Takeaways & Limitations

    The early network state cannot be easily approximated from simple distributional properties, while labels are not necessary to approximate it and may accelerate its formation.

Abstract

from arXiv · show

Recent studies have shown that many important aspects of neural network learning take place within the very earliest iterations or epochs of training. For example, sparse, trainable sub-networks emerge (Frankle et al., 2019), gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the network undergoes a critical period (Achille et al., 2019). Here, we examine the changes that deep neural networks undergo during this early phase of training. We perform extensive measurements of the network state during these early iterations of training and leverage the framework of Frankle et al. (2019) to quantitatively probe the weight distribution and its reliance on various aspects of the dataset. We find that, within this framework, deep networks are not robust to reinitializing with random weights while maintaining signs, and that weight distributions are highly non-independent even after only a few hundred iterations. Despite this behavior, pre-training with blurred inputs or an auxiliary self-supervised task can approximate the changes in supervised networks, suggesting that these changes are not inherently label-dependent, though labels significantly accelerate this process. Together, these results help to elucidate the network changes occurring during this pivotal initial period of learning.

1 INTRODUCTION

The paper focuses on the comparatively unexplored earliest phase of deep-network training, where several consequential changes may occur. It uses IMP-based analysis and controlled perturbations to characterize these changes across architectures.

  • The study targets early training because prior work identified critical periods, weight reshaping, and sparse trainable subnetworks, while later training had received more attention.
  • The authors provide an in-depth overview of statistics summarizing learning during the early part of training.
  • They evaluate perturbations of the early network state, including reinitialization, weight permutations, and noise-based approximations.
  • Deeper networks are not robust to reinitialization with random weights while maintaining signs, unlike observations in smaller networks.
  • After the early phase, weight distributions are highly non-independent, and permuting weights substantially harms performance even when signs are maintained.
  • Pre-training using only p(x) can approximate early supervised changes, but requires roughly 32× longer training and must avoid misleading labels.

2 KNOWN PHENOMENA IN THE EARLY PHASE OF TRAINING

Prior studies identify several rapid and consequential phenomena during early neural-network training, including lottery-ticket rewinding, Hessian reshaping, and critical periods. Their timing and robustness depend on the setting.

  • Lottery ticket rewinding: IMP performance benefits from rewinding to later training points, with gains saturating roughly between 500 and 2000 iterations for ResNet-20 on CIFAR-10.
  • Hessian eigenspectrum: Early training rapidly reshapes the Hessian spectrum, after which gradient descent occurs largely within its top subspace.
  • Critical periods in deep learning: Critical periods can make early corrupted data cause irrevocable damage, although their timing extends beyond the period studied here and varies with training choices.

3 PRELIMINARIES AND METHODOLOGY

The paper studies standard convolutional networks and uses IMP with rewinding to extract subnetworks from different early-training states. Its preliminary results establish the rewinding timescale and sparsity context.

  • Networks: Experiments use five standard convolutional networks for CIFAR-10, with ResNet-20 emphasized in the main text and qualitatively similar results across networks.
  • Networks: All reported experiments use means and standard deviations across five replicates with different random seeds.
  • Iterative magnitude pruning with rewinding: IMP trains a network to completion, globally prunes the 20% lowest-magnitude weights, and rewinds the remaining weights to an earlier iteration k.
  • Iterative magnitude pruning with rewinding: For k ≥500, subnetworks match the original network with 16.8% of weights remaining, while k >2000 yields essentially no further improvement.

4 THE STATE OF THE NETWORK EARLY IN TRAINING

Measurements of ResNet-20 during its first 4000 iterations reveal substantial early changes organized into three sub-phases. Accuracy rises rapidly while gradients, weights, and signs change most sharply at the beginning.

  • The analysis focuses on ResNet-20 during the first 4000 iterations, or 10 epochs, and summarizes the observations as an early-training timeline.
  • First phase: During the earliest ten iterations, large gradients drive rapid movement from initialization and many weight sign changes.
  • Second phase: Accuracy reaches 55% by 400 iterations and approaches 80% by 2000 iterations, compared with a final accuracy of 91.5%.
  • Three sub-phases: The first phase lasts a few iterations, the second about 500 iterations, and the third continues changing while decelerating.
  • Measurements: Telemetry tracks accuracy/loss, weight magnitude, sign changes, sampled weights, gradient magnitude, distances, and cosine similarities.

5 PERTURBING NEURAL NETWORKS EARLY IN TRAINING

The paper tests which aspects of early weight transformations matter for sparse-subnetwork training, finding that magnitudes, signs, weight dependencies, and perturbation scale all influence performance.

  • 5.1 ARE SIGNS ALL YOU NEED?: Changes in weight magnitudes over the first 500 iterations are substantially more important than sign changes for enabling subsequent training.
  • 5.1 ARE SIGNS ALL YOU NEED?: By iteration 2000, combining both signs and magnitudes from that iteration outperforms mixing either component with initialization values.Only 9% of signs change between iterations 500 and 2000, yet signs become important during this interval.
  • 5.2 ARE WEIGHT DISTRIBUTIONS I.I.D.?: Shuffling weights globally or within layers substantially reduces performance, indicating that early weight distributions are not independently interchangeable.The experiments also consider filter-wise and other structural permutations.
  • 5.2 ARE WEIGHT DISTRIBUTIONS I.I.D.?: Same-sign shuffling improves filter-wise and layer-wise permutation performance, especially at k = 2000, but also reduces the perturbation magnitude.At k = 500, the reported perturbation is 0.0 ± 0.049.

6 THE DATA-DEPENDENCE OF NEURAL NETWORKS EARLY IN TRAINING

The paper investigates whether early network changes require labels, intact inputs, or task-specific information. Self-supervised rotation and blurred-input pre-training can approximate favorable states, whereas combining both information restrictions provides too little information.

  • 6 THE DATA-DEPENDENCE OF NEURAL NETWORKS EARLY IN TRAINING: The central question is whether early weight changes depend on the input distribution p(x), the conditional label distribution p(y|x), or both.
  • 6.1 RANDOM LABELS: Pre-training with randomized labels tests whether the input distribution alone can establish a favorable sparse-training state despite incorrect labels.
  • 6.2 SELF-SUPERVISED ROTATION PREDICTION: Pre-training with self-supervised rotation produces well-trainable sparse subnetworks, but requires 40 epochs instead of 1.25, a factor of 32× longer than supervised training.The rotation task classifies images by randomly applied 90n-degree rotations.
  • 6.3 BLURRED INPUTS: Pre-training on blurred inputs with correct labels succeeds in producing subnetworks similar to those obtained after supervised early training.Inputs are downsampled by 4× and then upsampled to full size.
  • 6.4 COMBINING BLURRING AND ROTATION: Combining blurred inputs with self-supervised rotation gives the network too little information for the considered pre-training durations.The resulting IMP subnetworks are not tangibly better than those obtained by rewinding to initialization.

7 DISCUSSION

The discussion consolidates the study's findings: early training has three dynamical sub-phases, deeper networks retain sensitivity to initialization perturbations, and self-supervised pre-training can approximate late rewinding.

  • 7 DISCUSSION: The earliest training period contains three sub-phases: rapid motion with unusually large gradients, gradient overshoot followed by rapid performance gains, and gradual deceleration.
  • 7 DISCUSSION: Deeper networks are not robust to reinitializing with random weights while maintaining signs, and their early-phase weight distributions become highly non-independent.
  • 7 DISCUSSION: Self-supervised pre-training yields performance equivalent to late IMP rewinding, while labels are not necessary to approximate the rewinding state.The discussion connects this result to the possibility that appropriately pre-trained networks may not require rewinding.
  • 7 DISCUSSION: Highly non-independent weights make direct approximation of the rewinding point difficult, challenging interpretations that identify lottery-ticket subnetworks at initialization.

A MODEL DETAILS

The study reports network coverage and a shared CIFAR-10 training protocol. The main text focuses on ResNet-20, with other architectures summarized for comparison.

  • A MODEL DETAILS: The network summary includes ResNet-20 in the main body and the remaining studied networks in Appendix B.
  • A MODEL DETAILS: All networks are trained for 160 epochs with SGD, initial learning rate 0.1, momentum 0.9, and tenfold learning-rate drops at epochs 80 and 120.Training uses weight decay 1e-4 and normalized random-flip and random-crop augmentation.

B EXPERIMENTS FOR OTHER NETWORKS

Experiments characterize early training dynamics and test how initialization, weight structure, perturbations, and pre-training conditions affect network behavior across settings.

  • Gradient magnitudes are very large during the early training phase.
  • Large sign changes occur early, with 6% appearing in the first 10 iterations, while gradient magnitudes converge to roughly constant magnitude.
  • Weight magnitudes decrease linearly during early training.
  • The experiments vary IMP rewinding iteration, sign and magnitude initialization, structural weight shuffling, sign-preserving shuffling, and Gaussian noise.
  • Additional experiments examine early-training telemetry, perturbation strength, random-label pre-training, self-supervised rotation, blurring, and combined blurring with rotation across network settings.
Loading 2002.10365v1…