Source-linked AI summary
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, Quanquan Gu
TL;DR
Adaptive gradient methods can converge quickly yet generalize worse than momentum SGD, leaving open how to close that gap. The paper introduces Padam, which partially adapts momentum through a parameter p, and reports fast convergence with generalization comparable to SGD.
Problem
Adaptive methods such as Adam can generalize worse than momentum SGD in deep neural networks, motivating the question of how to retain fast convergence while closing this generalization gap.
Method
Padam introduces a partially adaptive parameter p that unifies Adam/Amsgrad with momentum SGD by controlling the degree of adaptiveness.
Results
Padam maintains fast convergence while generalizing as well as momentum SGD in the reported experiments.
Takeaways & Limitations
The results suggest that adaptive gradient methods can again be considered for faster deep-network training.
Takeaways & Limitations
The paper leaves Padam’s performance on other neural-network types, including GANs and GCNs, for future work.
Abstract
from arXiv · showhide
Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter $p$, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain a fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.
1 Introduction
Adaptive methods can converge quickly but often generalize worse than momentum SGD, especially in over-parameterized networks. The paper proposes Padam to control adaptiveness and combine fast convergence with SGD-like generalization.
- Adaptive methods automatically adjust learning rates using historical gradients, but Adam and related methods can generalize worse than momentum SGD despite better training performance.
- Learning-rate decay can make adaptive methods’ effective learning rates too small after several rounds, a phenomenon called the “small learning rate dilemma.”
- Padam introduces a partially adaptive parameter that unifies Adam/Amsgrad with momentum SGD and controls the degree of adaptiveness.
- The paper provides a convergence guarantee for Padam to a stationary point in stochastic nonconvex optimization.
- Experiments report that Padam achieves fast convergence while generalizing as well as momentum SGD and outperforming Yogi, AdamW, and AdaBound in generalization.
2 Review of Adaptive Gradient Methods
Adaptive gradient methods modify coordinate-wise learning rates using accumulated or exponentially averaged gradient information. Adam and Amsgrad improve adaptation and convergence guarantees, but their theoretical guarantees remain limited mainly to convex settings.
- Adagrad uses an adaptive learning rate for each dimension based on accumulated gradient information.
- RMSprop replaces Adagrad’s arithmetic averages with exponential moving averages, while Adam combines this idea with momentum acceleration.
- Adam can fail to converge when rarely encountered large gradients are forgotten by its short-memory exponential moving average.
- Amsgrad adds a maximum second-moment term to ensure decay of the effective learning rate and addresses some convergence issues in Adam.
- Existing theoretical guarantees for Adagrad, Adam, and Amsgrad are restricted to convex functions, motivating nonconvex analysis.
3 The Proposed Algorithm
Padam controls adaptiveness through a parameter p, interpolating between momentum SGD and Amsgrad. The paper argues that smaller adaptiveness can reduce the small-learning-rate dilemma while preserving rapid early progress.
- Padam introduces a partial adaptive parameter p to control the level of adaptiveness in the optimization procedure.
- When p approaches 0, Padam reduces to momentum SGD; when p = 1/2, it becomes exactly Amsgrad.
- The effective learning rate is α_t/bv_t^p, so larger p increases the severity of the small-learning-rate dilemma under learning-rate decay.
- Choosing p < 1/2 can potentially reduce the small-learning-rate dilemma relative to Amsgrad while retaining adaptive behavior.
- The experiments fix the learning-rate decay schedule across methods to compare control of adaptiveness rather than schedules.
- On ResNet with CIFAR-10 and CIFAR-100, p = 1/8 is reported as an example that achieves fast convergence and strong later generalization.
4 Convergence Analysis of the Proposed Algorithm
The analysis establishes Padam’s convergence for stochastic nonconvex optimization under bounded-gradient, smoothness, and cumulative-gradient growth assumptions. Its rate matches nonconvex SGD in the worst case and is strictly better when stochastic gradients are sparse.
- Problem setting: Padam is analyzed for minimizing an expected stochastic objective with an L-smooth nonconvex function and unbiased stochastic gradient estimators.The analysis assumes coordinatewise bounded stochastic gradients and L-smoothness of the expected objective.
- Assumptions: The convergence theorem assumes p ∈ [0, 1/2], β1 < β2p, a constant step size, and cumulative-gradient growth bounded by G∞T^s for 0 ≤ s ≤ 1/2.The theorem also imposes the bounded-gradient and smoothness assumptions introduced earlier.
- Rate dependence: The convergence rate depends on the cumulative-gradient growth exponent s, with sparse gradients corresponding to s < 1/2.The worst case is s = 1/2, while practical sparse-gradient settings may yield smaller s.
- Comparison with SGD: When s = 1/2, Padam’s rate matches the O(1/√T) rate of nonconvex SGD in its dependence on T.This comparison is made under the worst-case cumulative-gradient growth exponent.
- Comparison with SGD: When stochastic gradients are sparse, s < 1/2, Padam’s convergence rate is strictly better than nonconvex SGD’s rate.The theorem’s constants include terms whose dependence on iteration count and dimension is discussed separately.
5 Experiments
Experiments across image classification and language modeling show that Padam combines rapid convergence with generalization matching or exceeding strong baselines. Across CIFAR-10, ImageNet, and Penn Treebank, Padam generally outperforms existing adaptive methods.
- Overall results: Padam maintains a fast convergence rate while generalizing as well as SGD with momentum and better than Adam and Amsgrad.The experiments cover modern deep learning models and several standard benchmarks.
- Experimental setup: Padam is compared with SGD-Momentum, Adam, Amsgrad, AdamW, Yogi, and AdaBound on CIFAR-10, ImageNet, and Penn Treebank.The evaluation includes CNN architectures for image classification and LSTM models for language modeling.
- Image classification: On CIFAR-10, adaptive methods initially reduce train loss and test error rapidly, but after learning-rate decays Adam and Amsgrad generalize poorly; Padam retains strong performance.After the second decay, Adam and Amsgrad suffer from the small learning rate dilemma, while SGD eventually catches up with Padam.
- Image classification: On ImageNet, Padam again achieves the fastest convergence while generalizing as well as SGD with momentum.AdamW, Yogi, and AdaBound improve over Adam but still exhibit a large generalization gap.
- Language modeling: On Penn Treebank, Padam achieves the best final test perplexity for both 2-layer and 3-layer LSTM experiments despite slower early-stage progress than some adaptive methods.Adam, Amsgrad, and AdamW converge faster initially, especially in the more complicated setting.
- Quantitative results: Padam achieves the highest CIFAR-10 test accuracy for VGGNet and WideResNet, matches SGD closely for ResNet, and leads several ImageNet and Penn Treebank comparisons.For ResNet on CIFAR-10, the final accuracy difference from SGD with momentum is less than 0.2%.
6 Conclusions and Future Work
The paper concludes that Padam unifies Adam/Amsgrad with SGD-Momentum and can maintain fast convergence while closing the generalization gap. It identifies evaluation on GANs and GCNs as future work.
- Padam unifies Adam/Amsgrad with SGD-Momentum through an appropriate partially adaptive parameter.
- Padam is reported to maintain a fast convergence rate while closing the generalization gap.
- The paper provides theoretical analysis of Padam's convergence rate to a stationary point in stochastic nonconvex optimization.
- Future Work: Performance on generative adversarial networks and graph convolutional networks remains future work.
A Proof of the Main Theory
This section provides a detailed version of the proof of Theorem 4.3.
- The section supplies a detailed proof of Theorem 4.3.
- Theorem 4.3 is the central result whose proof is expanded in this section.
- The proof details are presented as part of the paper's main theory.
A.1 Proof of Theorem 4.3
The appendix proves Theorem 4.3 by introducing auxiliary lemmas, bounding separate terms, and combining the resulting inequalities through rearrangement and telescoping.
- The proof begins with the auxiliary sequence z_t and develops lemmas concerning its relationships and bounds.
- Under bounded stochastic-gradient assumptions, the proof establishes bounds for the momentum and maximum second-moment estimates.
- A corollary is obtained by selecting p in [0, 1/4] and setting q = 0.
- The main smoothness inequality is decomposed into terms I1, I2, and I3, which are bounded separately.
- The proof uses matrix norm inequalities, Young's inequality, conditional expectation, and auxiliary lemmas to control its terms.
- For later iterations, the resulting inequality is telescoped over t = 2 to T, combined with the initial case, and rearranged to complete the theorem.
B Proof of Technical Lemmas
The technical-lemma proofs derive relations for the auxiliary sequence, control gradient and momentum quantities, and apply standard inequalities to establish the bounds used by the main theory.
- The appendix derives formulas for z_t and z_{t+1} − z_t from the definition of the auxiliary sequence.
- One lemma bounds the difference between gradients evaluated at z_t and x_t using smoothness-related terms.
- Bounded stochastic gradients imply uniform bounds on the momentum sequence m_t and the maximum second-moment estimate b_v_t.
- The proof of Lemma A.5 controls coordinatewise quantities using Cauchy, Hölder, and gradient-bound inequalities.
C.1 Datasets
The experiments use popular image-classification datasets, the Penn Treebank language-modeling dataset, and several modern neural architectures.
- Datasets: The image-classification datasets are CIFAR-10, CIFAR-100, and ImageNet (ILSVRC2012).CIFAR-10 has 50,000 training and 10,000 test images; CIFAR-100 has 100 classes; ImageNet has 1.28 million training and 50,000 validation images across 1,000 classes.
- Datasets: Penn Treebank is used for natural-language processing, with models evaluated by perplexity, where lower is better.
- Architectures: The image-classification architectures include modified VGG-16, ResNet-18, and WRN-16-4.VGGNet uses stacked 3×3 convolutions and max pooling; ResNet uses skip connections and batch normalization; Wide ResNet increases residual-network width.
- Architectures: The experiments also include an LSTM network for language modeling.The LSTM is described as a recurrent neural network capable of learning long-term dependencies.
D Additional Experiments
On CIFAR-100, Padam is evaluated across three CNN architectures using training curves and final test accuracy, showing fast convergence and strong generalization.
- Training Curves: Padam maintains faster convergence while generalizing as well as SGD with momentum on CIFAR-100.Figure 4 reports this pattern across three CNN architectures.
- Final Test Accuracy: Padam achieves the highest test accuracy in the VGGNet and ResNet CIFAR-100 experiments.
- Final Test Accuracy: Less than 0.1% separates Padam and SGD with momentum on WideResNet for CIFAR-100 at the final epoch.
- Final Test Accuracy: Table 4 reports final test accuracy for all algorithms, with bold numbers marking the best result.