PaperlayerView exact revision on arXiv

Source-linked AI summary

Quasi-hyperbolic momentum and Adam for deep learning

Jerry Ma, Denis Yarats

arXiv:1810.06801v4cs.LGstat.ML

TL;DR

Deep-learning optimization commonly relies on momentum, but stochastic settings motivate reducing update noise without making gradient estimates stale. The paper introduces QHM and QHAdam, connects QHM to several existing optimizers, and reports improved training, including 29.45 BLEU on WMT16 EN-DE. These methods are presented as computationally cheap, interpretable replacements, while formal convergence analysis for the QH algorithms remains future work.

  • Problem

    Deep-learning momentum methods lack many of their deterministic theoretical advantages in stochastic settings, motivating improved optimization methods.

  • Method

    QHM averages plain SGD and momentum updates, while QHAdam applies the same quasi-hyperbolic idea to Adam and QHM connects to several existing optimizers.

  • Results

    29.45 BLEU is achieved on WMT16 EN-DE, while QHAdam converges on all 10 seeds compared with Adam exploding on 4.

  • Takeaways & Limitations

    QHM and QHAdam are computationally cheap, intuitive, simple to implement, and can replace momentum/NAG and Adam in varied settings.

  • Takeaways & Limitations

    Formal convergence analysis is left to future work, and multiple momentum buffers showed negligible preliminary value over one slow-decaying buffer with an immediate discount.

Abstract

from arXiv · show

Momentum-based acceleration of stochastic gradient descent (SGD) is widely used in deep learning. We propose the quasi-hyperbolic momentum algorithm (QHM) as an extremely simple alteration of momentum SGD, averaging a plain SGD step with a momentum step. We describe numerous connections to and identities with other algorithms, and we characterize the set of two-state optimization algorithms that QHM can recover. Finally, we propose a QH variant of Adam called QHAdam, and we empirically demonstrate that our algorithms lead to significantly improved training in a variety of settings, including a new state-of-the-art result on WMT16 EN-DE. We hope that these empirical results, combined with the conceptual and practical simplicity of QHM and QHAdam, will spur interest from both practitioners and researchers. Code is immediately available.

1 INTRODUCTION

The paper introduces QHM as a simple average of momentum and plain SGD, then extends the idea to QHAdam. It reports broad empirical gains, including improved stability and a new WMT16 EN-DE result.

  • QHM averages momentum’s update with plain SGD’s update, motivated initially by gradient variance reduction.
  • QHM connects to numerous optimization methods, including Nesterov acceleration, PID control, noise-robust momentum, Triple Momentum, and least-squares acceleration.These connections support analysis and allow QHM to recover several algorithms in a simpler implementation.
  • 40% improvement in stability and 29.45 BLEU are reported for the WMT16 EN-DE translation model after replacing the vanilla optimizer with its QH counterpart.
  • The paper also reports that momentum often yields negligible improvement over plain SGD.
  • QHM and QHAdam require little additional computation, are interpretable as interpolations between familiar algorithms, and can be tuned using existing practical intuitions.

2 PRELIMINARIES

The preliminaries define optimization algorithms through per-step update rules for model parameters and auxiliary buffers. Momentum uses an exponentially weighted gradient buffer whose discount factor controls both update memory and stochastic variance.

  • Optimization algorithms are represented as sequences of parameter updates, with update rules describing one step and optional auxiliary operations.
  • Momentum maintains a normalized exponentially weighted average of stochastic gradients and updates parameters using that buffer.
  • β = 0 recovers plain SGD from the momentum update rule.
  • The discount factor β controls how slowly the momentum buffer changes and, in stochastic optimization, the variance of the normalized buffer.
  • Normalizing the buffer by 1 − β removes dependence of update magnitude on β and supports interpreting the buffer as an average of past gradients.

3 ALGORITHM: QUASI-HYPERBOLIC MOMENTUM (QHM)

QHM replaces momentum’s exponentially weighted moving average with a quasi-hyperbolically weighted average. Its immediate discount factor interpolates between plain SGD and momentum while decoupling current-gradient contribution from buffer discounting.

  • QHM uses parameters α, β, and ν and replaces the exponentially weighted momentum buffer with a quasi-hyperbolically weighted moving average.
  • The recommended rule of thumb is ν = 0.7 and β = 0.999.
  • Interpretation: ν = 0 gives plain SGD, ν = 1 gives momentum, and intermediate ν values average the momentum and plain-SGD update steps.
  • QHM vs. momentum: QHM decouples the buffer discount factor β from the current gradient’s contribution, unlike standard momentum.
  • Efficiency: QHM requires one auxiliary buffer, one in-place scalar-vector multiplication, and three scaled vector additions per update step.

4 CONNECTIONS TO OTHER ALGORITHMS

The paper presents QHM as a unifying two-state optimizer that recovers or relates to several momentum-based algorithms. These connections provide convergence results and an efficient, accessible implementation, while multi-buffer extensions showed negligible preliminary benefit.

  • Nesterov acceleration: QHM recovers Nesterov’s accelerated gradient when ν = β, interpreting NAG as a β-weighted average of momentum and plain SGD.
  • PID control: QHM is a superfamily of PID optimizers and can recover nearly all PID coefficients when β is treated as a free variable.
  • Synthesized Nesterov variants: QHM and synthesized Nesterov variants recover each other, allowing QHM to recover Robust Momentum and Triple Momentum.
  • AccSGD: QHM recovers AccSGD, whereas AccSGD does not recover QHM; the paper also disproves the claim that AccSGD recovers NAG.
  • Theoretical convergence results: In deterministic optimization, QHM inherits Triple Momentum’s global linear convergence rate of 1 −1/√κ for strongly convex, smooth losses.
  • Unifying two-state optimization algorithms: Many two-state algorithms are functionally similar or equivalent, but QHM offers a more accessible and efficient implementation.
  • Unifying two-state optimization algorithms: Appendix D characterizes the two-state optimization algorithms recoverable by QHM for routine conversion to its accessibility and efficiency benefits.
  • Many-state optimization algorithms: Preliminary experiments found negligible value from multiple momentum buffers compared with one slowly decaying buffer and an appropriate immediate discount.

5 ALGORITHM: QHADAM

QHAdam replaces Adam’s two moment estimators with quasi-hyperbolic terms, while retaining connections to several established optimizers. It adds limited computational overhead and leaves formal convergence analysis for future work.

  • QHAdam replaces both of Adam’s moment estimators with quasi-hyperbolic terms.
  • QHAdam recovers Adam when ν1 = ν2 = 1, RMSProp when ν1 = 0 and ν2 = 1 modulo bias correction, and NAdam when ν1 = β1 and ν2 = 1.
  • QHAdam incurs four extra scaled vector additions over Adam.
  • When Adam training is stable, setting ν2 = 1 and leaving β2 unchanged is usually reasonable; ν2 < 1 may improve stability when training is unstable.The paper associates ν2 < 1 with a tighter step-size bound.
  • Formal convergence analysis for QHAdam is left to future work.

6 EXPERIMENTS

The experiments evaluate QHM and QHAdam through parameter sweeps and case studies spanning image recognition, language modeling, reinforcement learning, and translation. Across these settings, QH methods generally improve optimization, convergence speed, robustness, or final metrics over vanilla optimizers.

  • 6 EXPERIMENTS: The experiments comprise parameter sweeps on relatively small models and case studies on diverse large or state-of-the-art settings.Case studies cover image recognition, language modeling, reinforcement learning, and neural machine translation.
  • 6.1 PARAMETER SWEEPS: QH algorithms improve both training and validation metrics, and the default ν = 0.7 and β = 0.999 outperforms optimal NAG or Adam parameterizations in all settings.
  • 6.1 PARAMETER SWEEPS: QHM converges much faster than vanilla methods, even when the final performance gap becomes small.The paper suggests that a more aggressive learning-rate schedule could reduce training time in such settings.
  • 6.1 PARAMETER SWEEPS: Plain SGD performs similarly to NAG and momentum in most experiments, while QHM provides significant benefits without changing the effective step size.The paper attributes much of the common unnormalized momentum and NAG benefit to increased effective step size.
  • 6.2 CASE STUDIES: QHM trains much faster than NAG on ResNet152 and reaches a marginally better validation error.This result uses three seeds.
  • 6.2 CASE STUDIES: QHM improves validation perplexity by half a point over NAG on WikiText-103 across 10 seeds.
  • 6.2 CASE STUDIES: QHAdam improves average reward on four of seven MuJoCo environments and virtually ties on another.The comparison uses 10 seeds.
  • 6.2 CASE STUDIES: QHAdam converges for all 10 WMT16 English-German seeds while Adam explodes on 4, achieving 29.45 BLEU.The paper reports this as a new state-of-the-art result and attributes the robustness to controlling the maximum per-step update with ν2.

7 DISCUSSION

The paper presents QHM and QHAdam as simple, inexpensive replacements for momentum, NAG, and Adam across several settings. It also identifies limited empirical coverage, missing convergence theory, and open questions about tuning and asynchrony.

  • 7 DISCUSSION: Practitioners should tune QHM when feasible; ν = 0.7 and β = 0.999 are offered as a rule of thumb, while QHAdam settings are more situational.
  • 7 DISCUSSION: The empirical evaluation covers only a limited set of tasks and architectures, leaving the breadth of performance gains and hyperparameter insights for future work.
  • 7 DISCUSSION: Formal convergence results for QHAdam and reasonably general stochastic convergence results for QHM remain open.
  • 7 DISCUSSION: Extending asynchronous-optimization analyses to QHM could clarify how asynchrony affects ν and β.
  • 7 DISCUSSION: QHM and QHAdam are computationally cheap, intuitive, and simple to implement, and can replace momentum, NAG, and Adam in varied settings.
  • 7 DISCUSSION: QHM and QHAdam enable high exponential discount factors through immediate discounting, while QHM recovers numerous other algorithms.

APPENDICES

The appendices motivate QHM through discounted-sum variance reduction, document its practical and theoretical connections, and state important computational and analytical boundaries.

  • Organization: The appendices organize parameter-sweep, case-study, derivation, algorithm-connection, and stability analyses supporting the paper’s main development.They include experimental setups and results, PID and two-state analyses, miscellaneous derivations, and Adam/QHAdam stability bounds.
  • Variance-reduction motivation: Increasing β reduces EWMA covariance, with β = 0.9 yielding 1/19 · Σ and β = 0.99 yielding 1/199 · Σ.The appendix interprets momentum as variance reduction but notes that larger β also increases gradient staleness.
  • Hyperbolic discounting: Pure HWMA can benefit stochastic optimization but is impractical because its infinite total discount requires decay and its history must be recomputed each iteration.Preliminary experiments observed a marked benefit over EWMA when the number of past gradients was limited for tractability.
  • Quasi-hyperbolic discounting: QHWMA preserves discounted-average interpretation and can be computed online as a ν-weighted average of EWMA and the current vector without additional memory.When ν = 1, QHWMA becomes EWMA; otherwise its discounting is time-inconsistent.
  • Caveats and corrections: The appendix reports that QHM does not formally analyze stochastic convergence from a variance-reduction standpoint and that a cited PID derivation appears inconsistent by β factors.It also states that QHM recovers AccSGD, whereas AccSGD cannot recover NAG or fully recover QHM.
  • Connections to other optimizers: QHM recovers nearly all PID coefficients when β is free, while QHM and SNV recover each other and QHM also recovers Robust Momentum and Triple Momentum.The PID relationship is restricted when β is fixed, but the restriction largely disappears when β is treated as a free variable.

C.5 ACCSGD

This section relates QHM to AccSGD and argues that QHM is more expressive in the relevant recovery direction, while also contrasting their practical and stability considerations.

  • Motivation: AccSGD was proposed after reported failures of momentum and NAG in stochastic least-squares optimization, where it can converge faster in certain regression settings.The comparison concerns the formulation of Kidambi et al. (2018).
  • NAG recovery: AccSGD requires ξ ≤ √κ and ϵ ∈ (0, 1), whereas the parameter value needed to recover NAG gives ξ > √κ, making recovery impossible.The text reports that empirical simulations confirm this finding.
  • Relationship with QHM: QHM recovers AccSGD, but AccSGD cannot recover NAG and therefore cannot fully recover QHM.The section specifically disputes the claim that AccSGD recovers NAG.
  • Derivation: The AccSGD-to-QHM relationship is established by matching their unrolled update rules and corresponding transition-matrix coefficients under stated parameter constraints.The derivation introduces a generic two-state optimizer and matches its state evolution to QHM.
  • Related optimizer setting: The alternative PID setting uses exponential decay in its I term to avoid parameter blowup, but its D term becomes dependent on P and I, making the setting degenerate.The resulting controller is characterized as either PI or PD and is costlier than QHM because it uses two auxiliary buffers.
  • Stability context: The stability appendix identifies rare gradient spikes as a source of Adam’s large updates and proposes a tight bound for Adam and QHAdam.It reports that lowering ν2 below 1 can lower QHAdam’s maximum per-step update.

F.2 IMPLICIT UPDATE BOUND

The appendix derives a tight per-coordinate upper bound for QHAdam updates, then examines its limit cases and practical implications for Adam stability. It also compares QHM with AggMo on an EMNIST autoencoder, where hand-converted QHM outperforms the swept AggMo configuration.

  • QHAdam bound: QHAdam has a tight per-coordinate upper bound under 0 < β1 < √β2 < 1 and nonzero ˜st+1 coordinates.The bound depends on β1, β2, ν1, and ν2.
  • Limit cases: For vanilla Adam, the tight bound contradicts Kingma and Ba’s claimed upper bound on Adam’s per-coordinate step size.With β1 = 0.9 and β2 = 0.999, the claimed bound is ≲3.16 · α, whereas the tight bound is ≲7.27 · α.
  • Practical implications: Decreasing ν2 below 1 can impose a tighter QHAdam update-magnitude constraint without decreasing β2.The appendix motivates this as an alternative to lowering β2, which can slow training.
  • Practical implications: Increasing β2 far beyond 0.98 led to training explosion in experiments on models with rare inputs or labels, such as machine translation.The authors suspect instability is especially prevalent in these settings.
  • AggMo comparison: QHM-Converted outperforms AggMo-Best on mean squared error across the training, validation, and testing datasets of the EMNIST autoencoder task.QHM-Converted uses α = 28, ν = 0.97, and β = 0.999 without task-specific parameter sweeps.

H.3 DISCUSSION

The discussion argues that hand-converted QHM can match or exceed a more complex multi-buffer optimizer while retaining a simpler parameterization. It frames additional momentum buffers as offering marginal value in the examined comparison, but notes limitations in the supporting investigation.

  • Empirical comparison: QHM-Converted outperforms the extensively swept AggMo-Best parameterization on the EMNIST autoencoder task.The QHM configuration was constructed by hand rather than optimized on that task.
  • Interpretation: The results indicate that multiple momentum buffers with arbitrary weighting provide negligible benefit over one slow-decaying buffer with an appropriate QHM weight.This conclusion is stated for the AggMo-versus-QHM comparison.
  • Interpretation: The authors conjecture that the current gradient already provides adequate damping for a slow-decaying momentum buffer.Under this interpretation, damping from additional momentum buffers has marginal value.
  • Connections: QHM recovers extended AggMo with K = 2 and discount factors [0, β], while using fewer momentum buffers in its standard formulation.The connection provides an algebraic link between the two methods.
  • Limitations: Preliminary experiments found no material improvement from alternate weighting schemes with K > 2, but the investigation was neither rigorous nor conclusive.The number of hyperparameters also scales linearly with the number of momentum buffers K.
  • Future work: The authors suggest developing theoretically grounded or empirically tractable methods to select extended-AggMo weighting schemes before relying on that optimizer in practice.They meanwhile surmise that QHM may be preferable given AggMo’s added costs and complexity.

I.2.1 EXPERIMENT: RN152-IMAGENET-QHM

This section specifies experiments spanning ImageNet, WikiText-103, MuJoCo control, and WMT16 English–German translation. It compares QHM or QHAdam against NAG or Adam using task-specific models, schedules, and repeated-seed evaluations.

  • RN152-ImageNet-QHM: The RN152-ImageNet experiment uses a 152-layer residual network and evaluates QHM against the NAG baseline on ImageNet.The QHM setting uses ν = 0.7, β = 0.999, and a learning rate increased tenfold relative to baseline.
  • RN152-ImageNet-QHM: The RN152-ImageNet evaluation runs three seeds and reports validation top-1 error.All other details are identical between the baseline and QHM configurations.
  • WikiText-103: The WikiText-103 experiment compares QHM with ν = 0.98 and β = 0.998 against a 60-epoch NAG baseline.The QHM initial learning rate is increased 100-fold relative to baseline, with other details unchanged.
  • MuJoCo control: The MuJoCo experiments compare QHAdam with ν1 = 0.9 and ν2 = 1 against default-parameter Adam across seven continuous-control environments.Each optimizer is evaluated with ten seeds and average reward reported every 5000 training iterations.
  • WMT16 EN-DE: The WMT16 English–German experiment compares QHAdam with ν1 = 0.8, β1 = 0.95, ν2 = 0.7, and β2 = 0.98 against Adam.Four of ten Adam seeds failed to converge, so both optimizers are compared using their six best seeds.

J FULL PARAMETER SWEEP RESULTS

The full parameter-sweep appendix summarizes selected and optimal vanilla and QH parameterizations across the sweep experiments. Because each setting contains nearly 200 parameterizations with three seeds, complete results are distributed as CSV data.

  • Summary graphs: Figure 5 summarizes selected and optimal parameterizations of both vanilla and QH algorithms across all parameter-sweep experiments.The full experimental settings are provided in Appendix I.
  • Data release: Each experimental setting contains nearly 200 parameterizations with three seeds each, preventing complete presentation through graphs or tables.The authors instead provide data files describing all runs in CSV format.
  • Summary graphs: The sweep results are presented in two figure parts, with shaded bands indicating ±1 standard deviation.The two parts together cover the full parameter-sweep results.
Loading 1810.06801v4…