PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda, Nicha Dvornek, Xenophon Papademetris, James S. Duncan

arXiv:2010.07468v5cs.LGcs.CVstat.ML

TL;DR

Adaptive optimizers can converge quickly but often generalize worse than SGD, while GAN training remains unstable. AdaBelief adapts stepsizes to prediction error and achieves fast convergence, SGD-comparable ImageNet accuracy, and strong GAN performance.

  • Problem

    Adaptive methods trade faster convergence for weaker generalization, while optimizer stability remains limited in GAN training.

  • Method

    AdaBelief scales stepsizes by the deviation between observed gradients and their EMA predictions, using smaller steps for less-believed observations.

  • Results

    70.08 versus 70.23 validation accuracy on ImageNet matched SGD, while AdaBelief achieved the lowest FID among tested optimizers for GANs.

  • Takeaways & Limitations

    AdaBelief combines fast convergence, good generalization, and training stability across image classification, language modeling, and GAN experiments.

  • Takeaways & Limitations

    The convergence bounds may be loose and rely on the assumption that s_t is nondecreasing, implemented with an elementwise maximum.

Abstract

from arXiv · show

Most popular optimizers for deep learning can be broadly categorized as adaptive methods (e.g. Adam) and accelerated schemes (e.g. stochastic gradient descent (SGD) with momentum). For many models such as convolutional neural networks (CNNs), adaptive methods typically converge faster but generalize worse compared to SGD; for complex settings such as generative adversarial networks (GANs), adaptive methods are typically the default because of their stability.We propose AdaBelief to simultaneously achieve three goals: fast convergence as in adaptive methods, good generalization as in SGD, and training stability. The intuition for AdaBelief is to adapt the stepsize according to the "belief" in the current gradient direction. Viewing the exponential moving average (EMA) of the noisy gradient as the prediction of the gradient at the next time step, if the observed gradient greatly deviates from the prediction, we distrust the current observation and take a small step; if the observed gradient is close to the prediction, we trust it and take a large step. We validate AdaBelief in extensive experiments, showing that it outperforms other methods with fast convergence and high accuracy on image classification and language modeling. Specifically, on ImageNet, AdaBelief achieves comparable accuracy to SGD. Furthermore, in the training of a GAN on Cifar10, AdaBelief demonstrates high stability and improves the quality of generated samples compared to a well-tuned Adam optimizer. Code is available at https://github.com/juntang-zhuang/Adabelief-Optimizer

1 Introduction

The introduction contrasts accelerated SGD and adaptive learning-rate methods, highlighting adaptive methods’ faster early convergence but weaker generalization and instability in GANs. It proposes AdaBelief, an Adam-compatible optimizer that scales steps according to confidence in observed gradients, targeting fast convergence, generalization, and stability.

  • Background: First-order optimizers divide broadly into accelerated SGD methods and adaptive learning-rate methods.Examples include NAG, momentum SGD, and heavy-ball for accelerated SGD, and Adagrad, AdaDelta, RMSProp, and Adam for adaptive methods.
  • Motivation: Adaptive methods usually converge faster early in training than SGD but generalize worse, motivating approaches that combine their respective benefits.Prior approaches include scheduled or smooth transitions from Adam to SGD and other Adam modifications.
  • Contributions: The proposed optimizer is designed to provide fast convergence, good generalization, and training stability in complex settings such as GANs.The paper also theoretically analyzes convergence in convex and non-convex stochastic optimization and validates performance through extensive experiments.
  • Method: AdaBelief modifies Adam by replacing its second-moment normalization with the exponential moving average of squared prediction errors, without extra parameters.Adam divides the first-moment estimate by √v_t, whereas AdaBelief divides it by √s_t, where s_t tracks (g_t − m_t)^2.
  • Method: AdaBelief takes smaller steps when observed gradients deviate from their EMA prediction and larger steps when they closely match it.The inverse of √s_t represents belief in the current gradient observation.

2 Methods

AdaBelief modifies Adam by replacing the gradient-magnitude accumulator with an EMA of prediction errors, adapting stepsizes to deviations between observed and predicted gradients. This mechanism incorporates curvature- and sign-related information, producing large steps in flat or low-curvature directions and small steps in oscillatory or high-variance directions.

  • Comparison with Adam: AdaBelief replaces Adam’s denominator accumulator vt with st, the EMA of (gt−mt)^2, without introducing extra parameters.Both methods use the EMA gradient mt and bias correction; AdaBelief adds ϵ to st during bias correction.
  • Comparison with Adam: AdaBelief takes a large step when the observed gradient gt is close to its EMA prediction mt and a small step when they greatly deviate.The denominator is √st, where st measures the squared prediction error.
  • AdaBelief uses curvature information: AdaBelief uses gradient changes related to the Hessian, thereby incorporating loss-curvature information into its stepsizes.In a steep, narrow valley, large gradient changes produce a large denominator and a small step; with large gradients but small curvature, AdaBelief retains a large step.
  • AdaBelief considers the sign of gradient in denominator: AdaBelief considers both gradient magnitude and sign, taking a large step in the x direction and a small step in the y direction in the 2D L1 example.Adam has equal denominators across coordinates because vt ignores gradient sign, whereas AdaBelief has 1/√st,x ≫ 1/√st,y.
  • AdaBelief considers the sign of gradient in denominator: When coordinate-wise gradient variance is uniform, AdaBelief’s update direction matches the gradient; with nonuniform variance, it takes smaller steps where variance is larger.This contrasts with Adam’s sign-descent behavior, which can deviate from the true gradient direction.

3 Experiments

Experiments across image classification, language modeling, and GAN training show that AdaBelief combines fast convergence with competitive accuracy and improved stability. Additional evaluations report advantages over established optimizers in translation, object detection, and reinforcement learning.

  • CNNs on image classification: On Cifar image classification, AdaBelief achieves fast convergence like adaptive methods while targeting SGD-like accuracy.Experiments use VGG11, ResNet34, and DenseNet121 on Cifar10 and Cifar100, reporting mean and standard deviation across 3 runs under optimized hyperparameters.
  • CNNs on image classification: On ImageNet, AdaBelief outperforms other adaptive methods and achieves accuracy comparable to SGD.AdaBelief uses Adam’s default parameters with decoupled weight decay, while other optimizers use their best literature results.
  • LSTM on language modeling: For 2-layer and 3-layer LSTM language modeling, AdaBelief achieves the lowest perplexity, while its 1-layer performance is close to other optimizers.Perplexity is evaluated on the Penn TreeBank test set using mean and standard deviation across 3 runs.
  • Generative adversarial networks: AdaBelief achieves the lowest FID among compared optimizers for both large and small GANs.The study evaluates WGAN, WGAN-GP, and SN-GAN variants; FID measures generated-image quality and diversity.
  • Remarks: Compared with Padam, AdaBelief achieves a much lower GAN FID and slightly higher ImageNet classification accuracy.AdaBelief uses the same number of parameters as Adam, whereas Padam has one additional parameter.
  • Additional experiments: In additional experiments, AdaBelief outperforms Adam and RAdam on Transformer translation, achieves higher mAP on PASCAL VOC detection, and obtains higher reward than Adam in SAC reinforcement learning.AdaBelief is slightly worse than AdaHessian on the translation task.

4 Related works

AdaBelief is positioned among first-order optimizer extensions, variance-reduction and layer-wise scaling methods, and alternative Adam variants. The paper also contrasts these approaches with second-order methods, which use curvature information but incur heavy computational costs.

  • First-order methods: AdaBelief can be combined with Lookahead, variance-reduction methods, and LARS, which respectively separate fast and slow weights, reduce gradient variance, and scale learning rates layer-wise.These methods represent other directions for modifying or augmenting first-order optimization updates.
  • Adam variants: Other Adam variants include NosAdam, Sadam, and Adax.The paper lists these as related alternatives to Adam.
  • Second-order methods: Second-order methods, including Newton, Quasi-Newton, Gauss-Newton, L-BFGS, Natural-Gradient, and Conjugate-Gradient methods, are widely used in conventional optimization.Hessian-free optimization uses second-order methods to train neural networks.
  • Second-order methods: Second-order methods typically use curvature information and are invariant to scaling but have heavy computational burden, limiting their use in deep learning.The paper contrasts their computational cost with their curvature-based scaling invariance.

5 Conclusion

The paper proposes AdaBelief, which scales stepsizes using the difference between predicted and observed gradients. It is presented as combining fast convergence, SGD-like generalization, and stability in complex settings while retaining Adam’s parameters.

  • 5 Conclusion: AdaBelief adaptively scales the stepsize according to the difference between predicted and observed gradients.The optimizer uses this prediction-error-based mechanism to adjust update sizes.
  • 5 Conclusion: AdaBelief is presented as the first optimizer to achieve fast convergence, good generalization, and training stability simultaneously.These goals correspond respectively to adaptive methods, SGD, and complex settings such as GANs.
  • 5 Conclusion: AdaBelief uses the same parameters as Adam, making it easy to tune.The conclusion explicitly identifies parameter compatibility with Adam as a practical benefit.

Broader Impact · Appendix · A. Detailed Algorithm of AdaBelief

The paper presents AdaBelief as an optimizer intended to combine fast convergence, good generalization, and training stability, while providing a detailed algorithm applicable to models with numerically estimated gradients. Its appendix defines the optimizer’s notation, typical hyperparameters, and update equations.

  • Broader Impact: AdaBelief is presented as the first optimizer to achieve fast speed, good generalization, and training stability.The paper states that its social impact is mainly determined by each application.
  • Broader Impact: AdaBelief can train any model whose parameter gradients can be numerically estimated.The authors suggest this broad applicability could support development and application of deep learning models.
  • Appendix: The appendix introduces the loss function f(θ), parameter vector θ ∈ R^d, gradient g_t, learning rate α, and small constant ϵ.The default learning rate is 10^-3, while ϵ is typically 10^-8.
  • A. Detailed Algorithm of AdaBelief: The algorithm uses β_1 and β_2 as smoothing parameters, with typical values β_1 = 0.9 and β_2 = 0.999.m_t is the EMA of g_t, v_t is the EMA of g_t^2, and s_t is the EMA of (g_t − m_t)^2.
  • A. Detailed Algorithm of AdaBelief: At each step, the algorithm increments t and computes g_t as the parameter-gradient estimate at the preceding iterate.The gradient is written as g_t ← ∇_θ f_t(θ_{t−1}).
  • A. Detailed Algorithm of AdaBelief: The first-moment estimate is updated as m_t ← β_1m_{t−1} + (1 − β_1)g_t.This is the exponential moving average update for the observed gradient.
  • A. Detailed Algorithm of AdaBelief: The belief-based variance estimate is updated as s_t ← β_2s_{t−1} + (1 − β_2)(g_t − m_t)^2 + ϵ.The displayed procedure then branches to an AMSGrad case.

B. Convergence analysis in convex online learning case (Theorem 2.1 in main paper)

The convex online-learning analysis establishes a regret bound for AdaBelief under bounded-gradient, positive-stabilizer, monotone-second-moment, and convex-feasible-set assumptions. A corollary specializes the result to exponentially decaying β1,t = β1λ^t with 0 < λ < 1.

  • Assumptions: The analysis absorbs ϵ into s_t, assumes s_t ≥ c > 0, and omits debiasing without limiting applicability to the de-biased version.These are stated as notational and analytical simplifications.
  • Theorem 2.1: Theorem 2.1 considers convex f over a convex feasible set F with bounded diameter D∞ and derives a regret bound for the algorithm-generated sequence.The theorem also assumes 0 ≤ β2 < 1, α_t = α / √t, nondecreasing s_t, bounded gradients, and s_t,i ≥ c > 0.
  • Proof: The proof combines a coordinatewise inequality, convexity of f, recursive term bounds, and substitution of the resulting series estimate.The derivation explicitly uses θ_t+1 and θ∗ as inputs to a lemma before applying the recursive bounds.
  • Corollary 2.1: Corollary 2.1 applies Theorem 2.1 when β1,t = β1λ^t with 0 < λ < 1 and derives the corresponding specialized result.Its proof uses the sum of an arithmetico-geometric series before substituting the estimate into the theorem’s bound.

C. Convergence analysis for non-convex stochastic optimization (Theorem 2.2 in main paper) · Assumptions

The convergence analysis assumes smoothness, lower boundedness, bounded true and noisy gradients, and unbiased independent gradient noise. It then specifies momentum conditions and analyzes Adam-type convergence through effective stepsizes, bounded variance, and auxiliary bounds.

  • Assumptions: The analysis assumes an L-Lipschitz gradient, a lower-bounded objective, and bounded true and noisy gradients with norm at most H.These are assumptions A1 and A2.
  • Assumptions: The noisy gradient is modeled as gt = ∇f(θt) + ζt with zero-mean noise independent across time.This is assumption A3.
  • Assumptions: The convergence statement uses constants C1, C2, and C3 independent of d and T, plus C4 independent of T, with expectation over all gradient randomness.The displayed bound itself is not included in the supplied passage.
  • Assumptions: The analysis defines the minimum effective stepsize at time t over coordinates and all possible past gradients {gi}t.The supplied definition is fragmentary but identifies the quantity being minimized.
  • Assumptions: It characterizes the convergence rate of an Adam-type algorithm using s1(T), which is defined through an upper bound on the right-hand side of formula (8).The corresponding rate expression is not fully visible in the supplied passages.
  • Assumptions: A supporting theorem additionally assumes minj∈[d](s1)j ≥ c > 0 and bounded gradient-noise variance, Var(gt) = σ2.These conditions support the subsequent convergence analysis.
  • Assumptions: The proof derives an upper bound for the right-hand side and a lower bound for the left-hand side of formula (8), yielding expressions involving Q1 + Q2 log T.A corollary further considers the condition c > C1H.

D. Proof of Theorem .3 · Proof of Theorem .3

The proof combines Lemmas 5–11 to bound the objective under Theorem 3’s assumptions, with separate bounds for terms T1–T5 and constants defined afterward.

  • D. Proof of Theorem .3: Lemma 6 states a result under the conditions of Theorem 3.The supplied passage does not include the lemma’s bound.
  • D. Proof of Theorem .3: Lemma 7 bounds T1 under the condition in Theorem 3.The supplied passage gives the bound’s existence but not its expression.
  • D. Proof of Theorem .3: Lemma 8 bounds T3 when the conditions of Theorem 3 are satisfied.The supplied passage does not include the explicit bound.
  • D. Proof of Theorem .3: Lemma 9 bounds T4 under the assumptions of Theorem 3.The supplied passage does not include the explicit bound.
  • D. Proof of Theorem .3: Lemma 10 bounds T5 under the assumptions of Theorem 3.The supplied passage does not include the explicit bound.
  • D. Proof of Theorem .3: Lemma 11 bounds T2 when the assumptions of Theorem 8 are satisfied.The supplied passage refers to Theorem 8 rather than Theorem 3.
  • Proof of Theorem .3: The proof is provided from [27] for completeness and combines Lemmas 5–11 to bound the objective.This establishes the overall proof strategy.
  • Proof of Theorem .3: The constants used in the proof are defined below.No constant expressions are included in the supplied passage.

E. Bayesian interpretation of AdaBelief

The section gives AdaBelief a Bayesian interpretation by modeling gradients with Gaussian prior and observation distributions, deriving the corresponding posterior update direction. It explains AdaBelief’s centered covariance estimate, contrasts it with Adam’s uncentered estimate, and interprets ε as both numerical stabilizer and prior strength.

  • Bayesian model: Under Gaussian prior and observation assumptions, the analysis derives a posterior for the true gradient using Bayes’ rule.The prior has uniform diagonal covariance, while the observed gradient is Gaussian around the true gradient with covariance C.
  • Bayesian update: The maximum-expected-gain direction is (εI + C)^−1g, implemented in practice as (εI + C)^−1m_t using the gradient EMA.Adaptive optimizers further apply a predefined learning-rate scaling for numerical stability.
  • Covariance estimation: Adam estimates covariance with an uncentered EMA, whereas AdaBelief uses the centered EMA approximation EMA diag[(g_t − E g_t)(g_t − E g_t)⊤].The section notes that covariance is defined as a centered quantity.
  • Comparison with Adam: Because C appears in the denominator, AdaBelief behaves closer to the ideal and takes a larger step than Adam.This comparison follows the Bayesian interpretation of the adaptive update direction.
  • Interpretation and limitation: ε serves as a numerical term preventing division by zero and, Bayesianly, represents the prior on g_t, with larger ε indicating larger σ^2.The Gaussian prior may mismatch the distorted gradient distribution as training evolves.

F. Experimental Details · 1. Image classification with CNNs on Cifar · 2. Image Classification on ImageNet

The experiments replicate AdaBound under standardized CNN training settings and compare optimizers through extensive hyperparameter search. On ImageNet, ResNet18 experiments use distinct learning-rate schedules for SGD and AdaBelief, with AdaBelief reaching accuracy very close to SGD.

  • F. Experimental Details: The study exactly replicated AdaBound’s reported results using its official implementation.
  • 1. Image classification with CNNs on Cifar: All Cifar CNN experiments trained for 200 epochs with batch size 128.
  • 1. Image classification with CNNs on Cifar: The Cifar learning rate was multiplied by 0.1 at epoch 150 across experiments.
  • 1. Image classification with CNNs on Cifar: The optimizer comparisons used the same training setting and extensive hyperparameter search.The main paper reported only test accuracy for these experiments.
  • 2. Image Classification on ImageNet: ImageNet experiments used a ResNet18 classification task with decoupled weight decay set as 10^-4 for both SGD and AdaBelief.
  • 2. Image Classification on ImageNet: SGD used initial learning rate 0.1, decayed by 0.1 at epochs 30 and 60, while AdaBelief started at 0.001 and decayed at epochs 70 and 80.
  • 2. Image Classification on ImageNet: AdaBelief achieved ImageNet accuracy very close to SGD, closing the generalization gap.

3. Robustness to hyperparameters · 4. Experiments with LSTM on language modeling · 5. Experiments with GAN

AdaBelief remains effective across broad hyperparameter ranges, performs competitively on simple LSTM language models and better on complicated ones, and is evaluated in multiple GAN settings under controlled experimental protocols.

  • 3. Robustness to hyperparameters: Across learning rates from 5 × 10^-4 to 3 × 10^-3 on VGG11 Cifar10, AdaBelief produces higher test-accuracy curves and is more robust than Adam.
  • 4. Experiments with LSTM on language modeling: LSTM experiments on Penn-TreeBank report training and test curves using results averaged across 3 independent runs with independent initialization.
  • 4. Experiments with LSTM on language modeling: For 1-layer LSTMs, AdaBelief’s perplexity is very close to other optimizers, whereas on complicated models it achieves significantly lower test perplexity.The experiments use α = 0.001, β1 = 0.9, β2 = 0.999, and ϵ = 10^-8 for 2-layer and 3-layer models; 1-layer models use ϵ = 10^-12.
  • 3. Robustness to hyperparameters: AdaBelief achieves its highest Cifar10 accuracy at the default ϵ = 10^-8 and exceeds 94% accuracy across all tested ϵ values, while Adam remains below 94%.The tested ϵ range is 10^-4 to 10^-9; AdaBelief is slightly more sensitive to ϵ than Adam.
  • 5. Experiments with GAN: GAN experiments cover WGAN and WGAN-GP, with each optimizer evaluated in 5 independent runs over 100 epochs.The protocol generates 64,000 fake samples and compares them with 60,000 real Cifar10 images.
  • 5. Experiments with GAN: For WGAN, discriminator weights are clipped to [−0.01, 0.01], while WGAN-GP uses a gradient-penalty weight of 10.0.
  • 5. Experiments with GAN: The study also evaluates AdaBelief with Spectral Normalization GAN using ϵ = 10^-16 and the rectification technique from RAdam.Other hyperparameters and training schemes follow the public repository used for the experiment.
Loading 2010.07468v5…