Adam: A Method for Stochastic Optimization

1 INTRODUCTION

• Stochastic objectives arise from data subsampling and other noise, while higher-order methods are ill-suited to the high-dimensional parameter spaces considered here.The paper therefore restricts its discussion to first-order methods.Source: §1 · p. 1
• Adam computes parameter-specific adaptive learning rates from first- and second-moment estimates of gradients using only first-order gradients and little memory.Its update magnitudes are invariant to gradient rescaling, and its stepsizes are approximately bounded by the stepsize hyperparameter.Source: §1 · p. 1
• Adam is designed to combine AdaGrad’s handling of sparse gradients with RMSProp’s suitability for online and non-stationary settings.The introduction also notes that Adam naturally performs a form of step-size annealing.Source: §1 · p. 1
• The paper reports that Adam consistently outperforms other methods across a variety of models and datasets, and describes it as versatile for large-scale, high-dimensional machine learning.Its analysis covers initialization bias correction and convergence in online convex programming.Source: §1 · p. 2

2 ALGORITHM

• Adam updates exponential moving averages of the gradient and squared gradient as estimates of the gradient’s first moment and second raw moment.The decay rates β1 and β2 control the averages; zero initialization biases them toward zero, especially early and when the rates are small.Source: §2 · p. 2
• The zero-initialized moving averages are corrected to counter their early bias toward zero.The algorithm uses bias-corrected estimates in its parameter update.Source: §2 · p. 2

2.1 ADAM’S UPDATE RULE

• Adam’s effective step magnitude is approximately bounded by α, making α an upper bound on parameter-step size in common scenarios.A smaller signal-to-noise ratio yields a smaller effective step, including near an optimum, where this acts as automatic annealing.Source: §2.1 · p. 3
• Rescaling gradients by a factor c scales the first-moment estimate by c and the second-moment estimate by c^2, which cancel in the update ratio.Thus the effective step is invariant to gradient scale.Source: §2.1 · p. 3

3 INITIALIZATION BIAS CORRECTION

• The bias correction follows from comparing the expected exponential average with the true second moment.The derivation considers gradients drawn at successive timesteps from an underlying distribution and initializes the average at zero.Source: §3 · p. 3
• Adam divides the running average by its initialization factor to correct the bias caused by starting the average at zero.The derivation identifies the factor (1 − β2^t) as the source of initialization bias in the second-moment estimate.Source: §3 · p. 3
• For sparse gradients, reliable second-moment estimates require averaging many gradients with a small β2.Without initialization-bias correction, that small β2 can produce much larger initial steps.Source: §3 · p. 3

4 CONVERGENCE ANALYSIS

• The regret framework compares Adam’s predictions on an unknown sequence of convex costs with the best fixed feasible parameter.The paper describes its bound as comparable to the best known bound for this general convex online-learning problem.Source: §4 · p. 4
• For sparse data features and bounded gradients, the summation term in the theorem can be much smaller than its upper bound.The paper notes that the adaptive-method result can achieve O(log d T), improving on O(d T) for a non-adaptive method.Source: §4 · p. 4
• The theoretical analysis requires β1,t to decay toward zero, a condition the paper says also matches prior empirical findings.Those findings suggest reducing the momentum coefficient near the end of training can improve convergence.Source: §4 · p. 4
• Adam’s average regret is bounded under bounded gradients, bounded distances between generated parameters, and stated conditions on β1 and β2.The theorem also requires a decaying learning rate αt and exponentially decaying β1,t.Source: §4 · p. 4

5 RELATED WORK

• Other first-order-information methods set stepsizes by estimating curvature, while SFO is a minibatch quasi-Newton method with memory that may be infeasible on constrained systems.Adam’s memory requirement is lower than SFO’s partition-linear storage.Source: §5 · pp. 4–5
• Adam uses a preconditioner that adapts to data geometry through a diagonal Fisher-information approximation, with a more conservative square-root inverse scaling than vanilla NGD.SFO instead uses minibatch-based quasi-Newton updates and requires memory linear in the number of minibatch partitions.Source: §5 · p. 5
• RMSProp with momentum applies momentum to rescaled gradients, whereas Adam directly averages first and second gradient moments; RMSProp also lacks bias correction.The correction matters especially for small β2, where omitting it can produce very large stepsizes and divergence.Source: §5 · p. 5
• AdaGrad corresponds to Adam with β1 = 0, infinitesimal (1 − β2), and an annealed stepsize; without bias correction, that correspondence breaks down.As β2 approaches 1, removing correction can yield infinitely large bias and parameter updates.Source: §5 · p. 5

6 EXPERIMENTS

• The experiments cover logistic regression, fully connected networks, and deep CNNs, using shared parameter initialization and dense searches over learning rates and momentum.The authors report the best hyperparameter setting for each optimizer.Source: §6 · p. 5

6.1 EXPERIMENT: LOGISTIC REGRESSION

• Adam matches momentum SGD on MNIST logistic regression, with both faster than AdaGrad; on sparse IMDB features, Adam matches AdaGrad, which beats momentum SGD.Figure 1 reports training negative log likelihood for MNIST and IMDB with 10,000 bag-of-words features.Source: §6.1 · pp. 5–6, Figure 1 · p. 6

6.2 EXPERIMENT: MULTI-LAYER NEURAL NETWORKS

• On deterministic multilayer-network training, Adam progresses faster than SFO in iterations and wall-clock time; SFO is 5–10x slower per iteration and stores memory linear in minibatches.Figure 2 compares training on MNIST; the experiment uses deterministic cross-entropy with L2 weight decay.Source: §6.2 · p. 6, Figure 2 · p. 7
• With dropout noise in multilayer networks, Adam shows better convergence than the other stochastic first-order methods; SFO fails because it assumes deterministic subfunctions.Figure 2 contrasts dropout training with the deterministic-cost setting.Source: §6.2 · p. 6, Figure 2 · p. 7

6.3 EXPERIMENT: CONVOLUTIONAL NEURAL NETWORKS

• On CIFAR-10 CNNs, Adam and SGD eventually converge considerably faster than AdaGrad; Adam only marginally improves on momentum SGD while adapting layerwise rates.The network uses alternating convolution and pooling stages followed by a 1,000-unit ReLU layer; the second-moment estimate vanishes after a few epochs, while first-moment variance reduction contributes to speedup.Source: §6.3 · pp. 6–7, Figure 3 · p. 7

6.4 EXPERIMENT: BIAS-CORRECTION TERM

• With β2 near 1, omitting bias correction destabilizes early VAE training; corrected Adam performs equal or better than RMSProp across tested hyperparameter settings.Figure 4 compares loss with and without correction across stepsizes and β settings after 10 and 100 epochs.Source: Figure 4 · p. 8, §6.4 · p. 8

7.1 ADAMAX

• AdaMax generalizes Adam’s L2-based scaling through Lp norms, with the infinity-norm variant providing a simple, stable algorithm.The Lp variants become numerically unstable for large p; AdaMax updates an exponentially weighted infinity norm.Source: §7.1 · pp. 8–9
• AdaMax needs no initialization-bias correction, and its parameter-update magnitude is bounded by α.The stated bound is |∆t| ≤ α.Source: §7.1 · p. 9

7.2 TEMPORAL AVERAGING

• Averaging can improve generalization over a noisy last iterate, using either Polyak-Ruppert averaging or an exponential moving average of parameters.The exponential average weights recent parameter values more heavily and can be bias-corrected.Source: §7.2 · p. 9

• Adam combines AdaGrad’s ability to handle sparse gradients with RMSProp’s ability to handle non-stationary objectives.Source: §8 · p. 10
• The method is straightforward to implement, requires little memory, and is aimed at large datasets or high-dimensional parameter spaces.Source: §8 · pp. 9–10
• Experiments confirm the convergence analysis in convex problems, while Adam is reported as robust across a wide range of non-convex machine-learning problems.Source: §8 · p. 10

10.1 CONVERGENCE PROOF

• The proof uses the convex tangent-hyperplane inequality with Adam’s update rules, then sums the resulting bounds across dimensions and iterations.Source: §10.1 · pp. 12, 14
• Auxiliary inequalities support the theorem, including a lemma proved by induction and a further regret-bound step.Source: §10.1 · pp. 12, 15
• Adam’s guarantee assumes bounded gradients and bounded distances between generated iterates, together with conditions on the moment-decay parameters.Source: §10.1 · pp. 13–14