Source-linked AI summary
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, Richard Socher
TL;DR
Deep-learning heuristics improve training and performance, but their mechanisms remain insufficiently understood and existing empirical analyses have limitations. The paper studies cosine restarts, warmup, and distillation with mode connectivity and CCA, finding unsupported standard explanations for cosine annealing, deeper-layer stabilization from warmup, and predominantly deeper-layer knowledge transfer.
Problem
The mechanisms behind effective deep-learning heuristics remain relatively unknown, while existing empirical analyses rely on limited linear-interpolation and dimensionality-reduction tools.
Method
The paper uses mode connectivity and canonical correlation analysis to study cosine annealing, learning-rate warmup, and knowledge distillation.
Results
The analysis suggests that standard cosine-annealing explanations lack evidence, warmup prevents deeper-layer training instability, and teacher knowledge is primarily disbursed to deeper layers.
Takeaways & Limitations
The findings support using controlled loss-surface and representation analyses to refine explanations of these heuristics and motivate further investigation.
Takeaways & Limitations
The authors note that cosine or linear decay over the full training budget consistently performed well and hypothesize that decay, more than restarts, drives success.
Abstract
from arXiv · showhide
The convergence rate and final performance of common deep learning models have significantly benefited from heuristics such as learning rate schedules, knowledge distillation, skip connections, and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit such analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz., mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons for the success of the heuristics. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed to the deeper layers.
1 INTRODUCTION
The paper examines why widely used deep-learning heuristics work despite limited theoretical understanding, using mode connectivity and CCA to study cosine annealing, warmup, and distillation. It reports that cosine-annealing explanations lack experimental support, warmup stabilizes deeper layers, and teacher knowledge is mainly disbursed to deeper student layers.
- Heuristics such as normalization, residual connections, and learning-rate strategies have accelerated deep-learning progress, but their effectiveness remains relatively unexplained.
- The study uses mode connectivity and CCA to analyze cosine annealing, learning-rate warmup, and knowledge distillation.
- Cosine annealing’s commonly quoted explanations are not substantiated by the experiments, while iterates move over barriers after restarts.
- Learning-rate warmup primarily limits weight changes in deeper layers, and freezing those layers achieves similar outcomes.
- The latent knowledge shared by a teacher during distillation is primarily disbursed to deeper layers.
2 EMPIRICAL TOOLS
The paper develops empirical tools for studying deep-learning training dynamics: mode connectivity probes low-loss paths between independently obtained modes, while CCA compares layer representations despite parameterization invariances. Experiments establish robust connecting curves across varied training choices and use these tools to analyze representation and loss-surface behavior.
- 2.1 MODE CONNECTIVITY: Mode connectivity finds low-loss piecewise-linear curves connecting independently obtained optima, challenging the view that comparable minima are isolated by barriers.
- 2.1 MODE CONNECTIVITY: The curve objective minimizes expected loss along the path by sampling t uniformly and using an unbiased gradient estimate for optimization.
- 2.1 MODE CONNECTIVITY: The study tests mode connectivity across different initializations, optimizers, augmentation choices, regularization settings, and training hyperparameters.
- 2.1 MODE CONNECTIVITY: Within a few curve-training epochs, all six mode pairs admit curves whose points generalize almost as well as the connected endpoint models.
- 2.2 CCA FOR MEASURING REPRESENTATIONAL SIMILARITY: CCA compares activation representations rather than weights, avoiding interpretability problems caused by neuron permutations and parameter rescaling.
- 2.2 CCA FOR MEASURING REPRESENTATIONAL SIMILARITY: SVCCA reduces dimensionality with SVD before CCA and summarizes layer similarity using the average of canonical correlations.
3 STOCHASTIC GRADIENT DESCENT WITH RESTARTS (SGDR)
SGDR combines cosine-decayed learning rates with warm restarts, producing trajectories that cross training-loss barriers but do not clearly support a local-minima escape explanation. Mode-connectivity and CCA analyses further show lower-loss connecting curves can generalize worse than SGDR iterates, while SGDR and SGD exhibit similar layerwise representational trends.
- Learning-rate schedule: SGDR decays the learning rate along a cosine curve and restarts it after each period, with successive periods scaled by Tmult.ηmin and ηmax bound the learning rate, while Tcur counts epochs since the last restart.
- Experimental setup: The VGG16 CIFAR-10 experiment uses T0 = 10 epochs, Tmult = 2, ηmax = 0.05, and ηmin = 10^-6 for SGDR.The SGD baseline uses momentum 0.9 with initial learning rate η0 = 0.05 and step reductions at epochs 60 and 150.
- Loss-landscape analysis: For SGDR pairs {w30, w150} and {w70, w150}, line segments have much higher training loss than their endpoints, indicating trajectories that cross barriers.The SGD baseline shows no corresponding line-segment training-loss barriers for its tested pairs.
- Loss-landscape analysis: The SGDR iterates reach a different region without appearing to occupy distinct local-minimum basins, making the local-minima escape explanation an oversimplification.The mode-connectivity curve has lower training loss than w150 but appears to overfit relative to the SGDR iterates on validation loss.
- Representation analysis: CCA comparisons show that SGDR and standard SGD have similar representational layer trends between early and late training despite SGDR trajectories crossing barriers.In both schemes, shallower-layer activations are more similar than deeper-layer activations between partially and fully trained models.
4 WARMUP LEARNING RATE SCHEME
The study examines warmup in large-batch training using CCA comparisons across training configurations. Results suggest warmup stabilizes deeper layers, with freezing the fully connected stack producing comparable outcomes.
- Warmup is motivated by the instability caused when large batch sizes require learning-rate increases that exceed loss-function curvature support.
- CCA compares representations across small-batch, large-batch without warmup, and large-batch with warmup configurations at selected training iterations.
- Large-batch training without warmup lacks the late-layer similarity increase observed with warmup and small-batch training.
- Freezing the fully connected stack during the first 200 iterations of large-batch training without warmup achieves comparable performance at the end of warmup.The comparison supports the proposition that warmup limits unstable changes in the fully connected layers.
- The analysis suggests that warmup primarily prevents deeper layers from creating training instability under large-batch training.
5 KNOWLEDGE DISTILLATION
The paper studies knowledge distillation by comparing teacher–student representations with CCA. The observed differences between distilled and independently trained students are concentrated mainly in deeper, discriminative layers.
- Knowledge distillation trains a smaller student to match a teacher’s output probability distribution rather than hard labels.
- The experiment compares a VGG-16 teacher with shallow students trained either by distillation or standard hard-label supervision.
- CCA scores are similar for shallow student layers relative to the teacher, while differences emerge mainly in the deeper fully connected stack.
- The results suggest that knowledge distillation’s distinctive effect is restricted primarily to deeper, discriminative layers rather than feature-extraction layers.
6 CONCLUSION
The conclusion revisits cosine annealing, learning-rate warmup, and knowledge distillation through mode connectivity and CCA. The empirical analysis questions the usual explanation for cosine annealing, links warmup to deeper-layer stability, and localizes transferred knowledge mainly in deeper layers.
- The paper investigates cosine annealing, learning-rate warmup, and knowledge distillation using mode connectivity and CCA.
- The commonly cited reasons for cosine annealing’s success are not evidenced in the paper’s empirical analysis.
- Learning-rate warmup is associated with preventing deeper layers from creating training instability.
- The latent knowledge shared by the teacher is primarily disbursed to deeper layers.
7 ADDITIONAL RESULTS ON ROBUSTNESS OF MC
Additional experiments assess the robustness of mode connectivity across training settings and visualize connected modes with t-SNE. The loss results support connectivity, while the t-SNE visualization has limited interpretability.
- The robustness experiments vary training choices including initialization, optimization, augmentation, regularization, and hyperparameters when connecting independently obtained modes.
- Validation loss, training accuracy, and training loss are reported for curves joining six pairs of modes.
- The reported curve results confirm that models along the connectivity curve can exhibit overfitting or poor generalization.
- t-SNE is used to visualize seven modes and their connecting points in two dimensions.
- Because t-SNE preserves local but not global information, the visualization is presented only to establish connected modes.
8 ADDITIONAL SGDR RESULTS
Additional SGDR analysis projects iterates onto planes and examines training and validation loss surfaces to test whether observations generalize across SGDR trajectories.
- SGDR Comparisons: Figure 11 reports validation loss, validation accuracy, and training accuracy for curves and line segments joining SGDR and SGD iterates.The figure compares mode-connectivity curves with line segments across the two learning-rate strategies.
- Projecting Iterates: The projection Wn is the point on the plane spanned by w70, w150, and θ with minimum l-2 distance from the original iterate.This projection represents the closest point on the plane to the iterate being analyzed.
- Loss Surface Analysis: Training and validation loss surfaces are plotted on a plane defined by SGDR iterates w30, w70, and their connection w30−70.The additional plane is used to check whether observations from another SGDR trajectory are general.
9 SGDR CCA HEATMAPS
CCA heatmaps compare representations before and after SGDR restarts, while accompanying loss-surface figures visualize training and validation behavior on iterate-defined planes.
- CCA Comparisons: CCA heatmaps compare models at epochs 10 and 150 and at epochs 150 and 155, with each cell correlating layers across the two models.The epoch-150-to-155 comparison measures the immediate effect of restarting.
- Interpreting CCA: High CCA correlation indicates similar layer representations, whereas low correlation indicates dissimilar representations.The analysis uses activation representations rather than weights to compare models.
- Observed SGDR Pattern: Despite a restart, SGDR retains the tendency for shallower layers to resemble one another more closely than deeper layers.The authors interpret this as evidence that restarting does not greatly affect the model, especially in shallower layers.
- Loss Surfaces: Figures 12 and 13 show logarithmic training and validation loss surfaces on planes defined by SGDR iterates and their connection.The planes include {w30, w70, w30−70} and projections of iterates.
10 WARMUP EXPERIMENTS ON RESNET-18 AND RESNET-32
Warmup experiments on VGG-11 and two ResNet architectures test whether freezing fully connected layers can reproduce warmup’s stabilizing effect under large batches and learning rates.
- Warmup and Freezing: Freezing the fully connected stack can obtain the stability induced by warmup when training with large batches and learning rates.The comparison was first shown on VGG-11 and then extended to ResNet-18 and ResNet-32.
- Experimental Setup: For the ResNet experiments, the learning rate was set to 5.0, and warmup increased it linearly from 0 to 5 over 20 epochs.The learning rate was scaled for the larger batch-size setting, with the warmup schedule applied for 20 epochs.
- Generality and Scope: Experiments on other configurations yielded similar results for the warmup and fully connected-layer-freezing comparison.The authors note that whether the results extend to larger datasets such as ImageNet remains future work.