Source-linked AI summary
FreezeOut: Accelerate Training by Progressively Freezing Layers
Andrew Brock, Theodore Lim, J. M. Ritchie, Nick Weston
TL;DR
FreezeOut addresses the training cost of early layers, which consume substantial computation despite having relatively few parameters. It progressively freezes layers and excludes them from backward passes, yielding speedups for DenseNets and Wide ResNets but appearing less suitable for VGG.
Problem
Early layers consume much of the training budget despite having few parameters and tending toward simple configurations, motivating reduced fine-tuning for those layers.
Method
FreezeOut progressively freezes hidden layers after a set portion of training and excludes them from the backward pass.
Results
Up to 20% speedup was observed with at most a 3% increase in test error for DenseNets; FreezeOut appeared better suited to Wide ResNets and less well-suited to VGG.
Takeaways & Limitations
FreezeOut may be tenable for quickly ranking many prototype designs, but is not desirable when the primary goal is maximizing test-set performance.
Takeaways & Limitations
The practical speedups differ across environments because experiments use controlled hardware while real runs use shared servers, varied GPUs, and other programs.
Abstract
from arXiv · showhide
The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at https://github.com/ajbrock/FreezeOut
1 INTRODUCTION
FreezeOut is motivated by evidence that not every unit must participate in every training step, especially because early layers often consume substantial computation despite having few parameters and simple representations.
- Motivation: Prior techniques suggest that selectively omitting layers or units during training can reduce computational costs.Stochastic Depth drops whole layers, whereas DropOut does not ordinarily provide a computational speedup.
- Motivation: FreezeOut progressively freezes early layers and excludes them from the backward pass to reduce training time.The approach targets early layers because they take up much of the computation budget, contain fewer parameters, and often converge to simple configurations such as edge detectors.
- Motivation: The method is also motivated by an unverified hypothesis that near-optimal layers may remain near local optima after learning-rate restarts.The authors connect this hypothesis to potentially reduced internal covariate shift and faster convergence, but explicitly identify it as unverified.
2 FREEZEOUT
FreezeOut uses layer-wise cosine annealing to reduce each layer’s learning rate to zero at a scheduled time, then removes that layer from future backward passes. The paper varies learning-rate scaling and timing schedules, including linear and cubed schedules.
- Core procedure: FreezeOut anneals each layer’s learning rate to zero at a scheduled time, then places the layer in inference mode and excludes it from future backward passes.The first layer freezes at t0, while later layers freeze at subsequent scheduled times, producing an immediate per-iteration speedup.
- Linear scheduling: In the simplest schedule, layer Li uses a fixed initial learning rate and anneals it to zero at ti, with ti linearly spaced from t0 to the total training duration.The schedule is illustrated for a five-hidden-layer network with t0 = 0.5.
- Linear scheduling: The per-layer learning rate follows cosine annealing until its layer-specific freeze time.The equation defines αi(t) using the initial rate αi(0), time t, and freeze time ti.
- Learning-rate scaling: Scaled schedules set αi(0) = α/ti so each layer’s learning-rate curve integrates to the same value.This gives layers comparable total movement in weight space despite receiving different numbers of training steps.
- Timing schedules: Cubing the linearly scheduled freeze times gives later layers more training time than the linear schedule.The paper evaluates both unscaled and scaled variants, and reports cubed times relative to their uncubed values; for t0 = 0.5, the cubed value is 0.125.
- Configuration: FreezeOut adds the choices of t0 and scheduling strategy to standard learning-rate and iteration choices, while requiring approximately 15 unique PyTorch lines.The experiments compare four strategies and seek a default configuration to reduce hyperparameter tuning.
3 EXPERIMENTS
Experiments evaluate FreezeOut across DenseNets, Wide ResNets, and VGG networks on CIFAR, comparing speedups with test performance. FreezeOut offers a speed–accuracy tradeoff for DenseNets, performs better in Wide ResNets, and is less well-suited to VGG.
- Experiments test four FreezeOut scheduling strategies across DenseNets, Wide ResNets, and VGG on CIFAR-10 and CIFAR-100.DenseNet experiments use 100-epoch runs and compare wall-clock speedups with test-set performance.
- DenseNet experiments: Up to 20% speedup accompanies a maximum 3% relative increase in test error across the evaluated strategies.Lower speedup levels generally perform better, while occasional baseline improvements are treated as insignificant because of training nondeterminism.
- Practical considerations: The acceptability of the speed–accuracy tradeoff depends on the use case: prototyping may tolerate higher FreezeOut levels, whereas maximizing test performance may not.The authors state that reduced training time is likely of no value when the architecture and hyperparameters are already fixed for maximum test performance.
- DenseNet experiments: Cubic scheduling with learning-rate scaling and t0 = 0.8 before cubing is recommended for maximizing speed within a 3% relative-error envelope.Linear scheduling without scaling at t0 = 0.5 is offered as a close alternative.
- Wide ResNet experiments: FreezeOut appears better suited to Wide ResNets than DenseNets, achieving higher accuracy than the baseline even when trained for the same number of epochs.The comparison uses WRN40-4 and also considers matching FreezeOut and baseline training times.
- VGG experiments: FreezeOut appears less well-suited to VGG, suggesting that skip connections may enable its effectiveness in the other evaluated architectures.The VGG comparison uses a more limited Linear Unscaled experiment against a no-FreezeOut baseline.
4 CONCLUSION
The paper presents FreezeOut as a simple technique for accelerating neural-network training by progressively freezing hidden layers.
- FreezeOut accelerates neural-network training by progressively freezing hidden layers.