Source-linked AI summary
Improving Generalization Performance by Switching from Adam to SGD
Nitish Shirish Keskar, Richard Socher
TL;DR
Adaptive optimizers can train quickly but may generalize worse than SGD, motivating a hybrid approach. SWATS switches from Adam to SGD using a monitored projection-based trigger and learned post-switch learning rate, with experiments showing results comparable to SGD while retaining Adam's rapid initial progress and hyperparameter insensitivity.
Problem
Adaptive methods can outperform SGD early in training yet generalize poorly later, creating a need to combine their rapid progress with SGD's generalization properties.
Method
SWATS starts with Adam, monitors the projection of Adam steps onto the gradient subspace, and automatically switches to SGD while learning the switch point and SGD learning rate.
Results
Across standard benchmarks, SWATS produces results comparable to SGD while retaining Adam's rapid initial progress and hyperparameter insensitivity.
Takeaways & Limitations
Hybrid training can combine Adam's rapid initial progress and hyperparameter insensitivity with performance comparable to SGD across the reported benchmarks.
Takeaways & Limitations
The proposed deep-network generalization explanation remains a hypothesis requiring further empirical and theoretical evidence.
Abstract
from arXiv · showhide
Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks.
1. Introduction
SGD offers strong empirical and theoretical performance but uses a uniform learning rate, while adaptive methods address ill-scaled problems yet may generalize worse than SGD. The paper motivates SWATS, which automatically switches from Adam to SGD without adding hyperparameters.
- SGD performs well across applications and has theoretical foundations including saddle-point avoidance and improved generalization.
- SGD uniformly scales gradients, which can hinder optimization on ill-scaled problems and make learning-rate tuning laborious.
- Adaptive methods use parameter-specific learning rates based on curvature estimates, unlike the scalar learning rate used by SGD and SGDM.
- Adam improves on initialization-induced noise in Adagrad and RMSProp through bias correction and exponential moving averages.
- Adam often makes faster initial progress than SGD but can stagnate later, motivating a hybrid strategy combining Adam's early performance with SGD's generalization properties.
- SWATS automatically switches from Adam to SGD without additional hyperparameters, learning both the switchover point and post-switch SGD learning rate during training.
2. SWATS
SWATS combines Adam’s rapid initial progress with SGD’s later generalization by learning both when to switch and the SGD learning rate from Adam’s steps. Its projection-based trigger and averaging add no hyperparameters, while experiments show that timing the switch is important for closing the generalization gap.
- Motivation: Adam reaches rapid initial progress, but SGD achieves lower final testing error on DenseNet/CIFAR-10, motivating a hybrid strategy.SGD reaches approximately 5% testing error, whereas Adam stagnates around 7%.
- Motivation: Adam-Clip(1, ∞) partially closes the gap by lower-bounding Adam’s parameter-wise step sizes, while Adam-Clip(0, 1) has no tangible final effect.Adam-Clip(1, ∞) reaches approximately 6% final accuracy in the reported experiment.
- Learning rate for SGD after the switch: Switching too early forfeits Adam’s initial progress, whereas switching too late preserves a generalization gap similar to Adam.On DenseNet/CIFAR-10, switching after 10 epochs resembles SGD, while switching after 80 epochs yields approximately 6.5% testing accuracy.
- Learning rate for SGD after the switch: SWATS estimates the post-switch SGD learning rate by projecting Adam’s step onto the gradient direction and averaging the resulting scalar estimate.The projection is interpreted as the scaling needed for the gradient’s projection onto the Adam step to reproduce that step; orthogonal projection underestimates the required rate.
3. Numerical Results
Experiments compare Adam, SGD, and SWATS across image-classification and language-modeling benchmarks. SWATS generally matches the better optimizer while retaining good early progress, though switching can temporarily degrade performance.
- Experimental setup: Experiments evaluate Adam, SGD, and SWATS on image classification and language modeling across multiple architectures and datasets.Image classification includes CIFAR-10, CIFAR-100, and Tiny-ImageNet; language modeling uses PTB and WT-2 with AWD-LSTM and AWD-QRNN.
- Experimental setup: The experiments tune optimizer learning rates and report the best-performing configurations in terms of generalization.Adam and SWATS use recommended β1 and β2 values of 0.9 and 0.999; SWATS switches to SGDM with momentum 0.9.
- Image classification: Across image-classification datasets and architectures, Adam makes good initial progress but fails to find solutions that generalize well.This pattern is reported for all three image-classification datasets.
- Image classification: SWATS switches within the first 20 epochs for most CIFAR datasets and at epoch 49 for Tiny-ImageNet, where accuracy temporarily drops from 80% to 52% before recovery.Despite the temporary degradation, Tiny-ImageNet achieves a better peak testing accuracy than Adam.
- Language modeling: In language modeling, SWATS switches later and achieves comparable generalization to Adam by lowest validation perplexity, although Adam reaches its final performance in fewer epochs.The switch may temporarily degrade performance, followed by recovery.
- Overall findings: Overall, SWATS performs almost as well as the better of SGD and Adam across the described tasks while often providing a good initial decrease in error.The estimated SGD learning rate is noisy but convergent in mean and reaches a scale similar to the tuned SGD learning rate.
4. Discussion and Conclusion
The paper argues that adaptive methods retain useful early-training behavior but can generalize worse than SGD, motivating SWATS as a hybrid strategy. Across several benchmarks, SWATS achieves results comparable to SGD while preserving adaptive optimization benefits, although the mechanism behind this behavior remains incompletely established.
- Discussion: Adaptive methods can generalize substantially worse than SGD because their per-variable learning-rate scaling differs from SGD’s uniform scaling.For convex quadratic functions, prior work reports orders-of-magnitude worse generalization for adaptive methods.
- Experiments: The evaluation compares SGD(M), Adam, and SWATS across multiple architectures and datasets, including CIFAR, Tiny-ImageNet, PTB, and WikiText-2.Figures 4 and 5 cover image-classification architectures, while the broader experiments include language-modeling benchmarks.
- Discussion: SWATS starts with an adaptive method and switches to SGD without adding hyperparameters, determining both the switch point and post-switch learning rate during training.The strategy is intended to retain rapid initial progress and hyperparameter insensitivity while improving generalization.
- Results: The reported results show SWATS produces results comparable to SGD while retaining Adam’s rapid initial progress and relative hyperparameter insensitivity.The strategy is presented as effective across several standard benchmarks, though language-modeling behavior differs from image classification.
- Limitations: The authors hypothesize that early adaptive training helps deep networks reach basins with better generalization, but further empirical and theoretical evidence is necessary.The paper identifies this mechanism as an open direction rather than an established explanation.
- Future work: The strategy may extend analogously to Adagrad and RMSProp, while smoother or reversible switching remains a proposed direction for future research.The hard switch may cause short-term performance degradation, motivating convex combinations of Adam and SGD directions.