Aggregated Momentum: Stability Through Passive Damping

• Momentum’s damping coefficient creates a speed–stability trade-off: values near 1 increase terminal velocity but are prone to oscillations and instability.Source: §1 · p. 1
• AggMo maintains several velocity vectors with different β parameters and averages them for each parameter update.Large β values build velocity along low-curvature directions, while smaller values damp oscillations; implementation adds little computational overhead.Source: §1 · p. 1
• AggMo uses passive damping inspired by combining materials with different resonant frequencies, preventing any single frequency from driving catastrophic resonance.Source: §1 · p. 1
• The paper analyzes quadratic convergence, online convex programming, and empirical performance on autoencoders, convolutional networks, and LSTMs.Source: §1 · p. 2
• AggMo works at least as well as classical momentum for a given β and often converges substantially faster than classical and Nesterov momentum when its maximum β is tuned.Source: §1 · p. 2

• Increasing classical momentum’s damping coefficient to accelerate low-curvature progress can destabilize high-curvature directions and produce oscillations.Source: §2 · p. 2
• Nesterov momentum improves convergence and stability by correcting the error after moving in the velocity direction.Source: §2 · p. 2
• The quadratic analysis studies optimizer behavior along eigenvectors, using quadratic functions as a proxy for behavior near a local minimum.Source: §2 · p. 2
• On quadratics, β = 0.9 requires many iterations, whereas β = 0.999 causes oscillations that prevent classical momentum from converging.Source: §2 · p. 2

• AggMo updates several velocities separately, then averages them to obtain the final velocity used for the parameter update.Source: §3 · p. 2
• AggMo dampens oscillations quickly across eigenvalues and converges faster than classical and Nesterov momentum on the quadratic example.Source: §3 · p. 3, Figure 1 · p. 3
• During oscillation, the β = 0.9 velocity reverses direction while β = 0.999 remains pointed away from the minimum, allowing the former to damp the latter.Source: §3 · p. 3, Figure 2 · p. 3
• The damping vector spaces terminal velocities exponentially, with a default K = 3 choice of β = [0, 0.9, 0.99] reported as stable and effective.Source: §3.1 · pp. 3–4
• AggMo adds a handful of arithmetic operations and stores K parameter-sized velocity vectors, while gradient evaluation remains the dominant computation.Source: §3.1 · p. 4

• AggMo recovers Nesterov momentum by introducing separate learning rates for each velocity and using a damping vector containing 0 and β.Source: §4 · p. 4
• With constant learning rate, Nesterov’s update can be written in a form that separates the momentum contribution from the gradient update.Source: §4 · p. 4
• The reparameterization φ_t = θ_t + γβv_t makes the Nesterov update identical to the corresponding AggMo update, with approximate equivalence when γ^(1)_t = 2γ and β is sufficiently close to 1.Source: §4 · p. 4
• This equivalence interprets Nesterov momentum as a weighted average of gradient and momentum updates and provides access to Nesterov’s theoretical convergence results.Source: §4 · p. 4

Quadratic convergence

• Figure 3 evaluates convergence rates for quadratics with condition numbers κ from 10^1 to 10^7 using fixed hyperparameters.The optimal classical-momentum curve assumes knowledge of the condition number, while the classical-momentum curves reach it when β is critical.Source: §5.1 · p. 5
• Approximately three-times faster convergence than Nesterov momentum occurs for AggMo in the under-damped regime without sacrificing performance at larger condition numbers.AggMo smoothly interpolates between classical momentum with β = 0.9 and β = 0.99 as the condition number varies.Source: §5.1 · p. 5, Figure 3 · p. 5

Online convex programming

• AggMo is evaluated in online convex programming, a setting equivalent to stochastic convex optimization for convergence-rate analysis.The analysis considers unknown convex cost functions and aims to minimize regret over time.Source: §5.2 · p. 5
• AggMo has asymptotically comparable regret to the best known bound and its average regret converges to zero under bounded-gradient assumptions.The theorem requires strict assumptions, and the paper notes that Adam is not guaranteed to have converging average regret under the same assumptions.Source: §5.2 · pp. 5–6

Momentum methods

• Prior work studies momentum convergence theoretically and empirically, including Nesterov’s momentum interpretation and analyses of existing methods’ failure modes.Sutskever et al. (2013) introduced the momentum view of Nesterov’s accelerated gradient.Source: §6 · p. 6

Gradient-history methods

• AggMo uses a higher-order finite-difference recurrence to access more gradient history, whereas lagged-momentum methods introduce many additional hyperparameters.The paper describes AggMo as evolving according to a (K+1)-th order finite difference equation.Source: §6 · p. 6

Adaptive methods

• Adaptive gradient methods address ill-conditioned curvature by approximating or preconditioning against the objective’s local geometry.Examples include diagonal adaptive methods and natural gradient descent based on the Fisher information matrix.Source: §6 · p. 6

Oscillation control

• Oscillation-aware methods detect instability and remove it by restarting momentum or adapting its schedule.The simplest restarting scheme sets the momentum velocity to zero whenever the loss increases.Source: §6 · p. 6

7 Evaluation

• The evaluation used autoencoders, convolutional networks, and LSTMs across MNIST, CIFAR-10, CIFAR-100, and Penn Treebank, comparing AggMo with classical momentum, Nesterov momentum, and Adam.Learning rate and damping-coefficient grid searches were used, and validation/test performance was reported alongside training loss where meaningful.Source: §7 · p. 6, §7.2 · p. 7, §7.3 · p. 8

7.1 Autoencoders

• AggMo achieved the best MNIST autoencoder performance across the reported training, validation, and test settings.The comparison used averages and standard deviations over 15 runs, with training and validation-selected hyperparameters reported separately.Source: §7.1 · p. 7, Table 1 · p. 7
• AggMo remained stable at learning rates an order of magnitude or more larger than those possible for classical or Nesterov momentum at the same maximum damping coefficient.With fixed learning rate 0.1 and varying damping configurations, AggMo using maximum damping 0.999 achieved the fastest overall convergence, while Nesterov was unstable at 0.999 and 0.9999.Source: §7.1 · p. 7, Figure 5 · p. 7

7.2 Classification

• On CIFAR classification, AggMo significantly outperformed other methods on the unregularized small convolutional network, while classical momentum had the best ResNet-32 validation accuracy.AggMo nevertheless performed consistently well and had the fastest overall convergence on the ResNet experiments.Source: §7.2 · p. 8, Table 2 · p. 8
• AggMo’s default settings converged faster than classical and Nesterov momentum with β = 0.9 without sacrificing final ResNet-32 validation performance.For CIFAR-100 without batch normalization, AggMo remained stable at larger learning rates but reached 69.32% test error versus 67.26% for classical momentum.Source: §7.2 · p. 8, Figure 6 · p. 8

7.3 Language modeling

• On Penn Treebank language modeling, Adam achieved the best training, validation, and test performance, while AggMo finished close to Adam and other momentum methods outperformed momentum-free SGD.Momentum methods converged after 300 epochs, whereas momentum-free methods converged more slowly; all momentum methods performed better without gradient clipping.Source: §7.3 · p. 9

• AggMo often delivered substantially faster convergence than both classical and Nesterov momentum because of its stability at higher β values.Source: §8 · p. 9
• AggMo can replace existing optimizers with little-to-no additional hyperparameter tuning on the explored tasks.Source: §8 · p. 9
• Nesterov momentum can be viewed as a special case of AggMo.Source: §8 · p. 9
• AggMo is easy to implement and has negligible computational overhead on modern deep learning tasks.Source: §8 · p. 9

• The equivalence demonstrations use β = 0.999 and initialize both optimizers at the same point.Source: A Nesterov Equivalence · p. 12
• When β = 0.999, AggMo and Nesterov follow visibly identical trajectories on a 2D quadratic function.The plotted circles and squares overlap, with markers offset only for readability.Source: A Nesterov Equivalence · p. 12, Figure 8 · p. 12
• On the Rosenbrock function, the two methods are initially indistinguishable but begin to differ as optimization proceeds toward the origin.Source: A Nesterov Equivalence · p. 12

• AggMo can be written as a linear dynamical system in K + 1 variables for quadratic objectives.Source: B Quadratic Convergence Analysis · p. 12
• The spectral norm of the system matrix B determines the convergence rate and bounds the distance to the optimum.Source: B Quadratic Convergence Analysis · p. 13
• For classical momentum, low damping causes slow monotone convergence, whereas high damping can induce oscillations that prevent convergence.Source: B Quadratic Convergence Analysis · p. 13
• The convergence-rate curves use direct eigenvalue computation and a fine-grained grid search for AggMo's learning rate because no closed-form optimum was available.Source: B Quadratic Convergence Analysis · p. 13
• AggMo combines multiple velocities to damp oscillations quickly across eigenvalues and can converge faster than classical and Nesterov momentum.Source: B Quadratic Convergence Analysis · p. 14, Figure 10 · p. 14

• The authors identify deriving closed-form eigenvalues and conditions for under-damped behavior as open theoretical goals.Source: §C.1 · p. 17
• AggMo dynamics form a (K + 1)-th order finite-difference equation rather than the second-order form common to most momentum methods.Source: §C.1 · p. 18
• This higher-order structure makes existing convergence proof techniques difficult to apply to AggMo.Source: §C.1 · p. 18
• On a non-convex toy problem, AggMo traversed flat regions and funnels quickly while remaining stable, unlike classical momentum with β = 0.999, which became unstable and overshot.Source: §E.1 · p. 20
• Default AggMo settings significantly outperformed classical and Nesterov momentum and appeared less sensitive to hyperparameter tuning in the reported task.Source: §E.2 · p. 20
• AggMo achieved better training, validation, and test perplexity than classical and Nesterov defaults in the LSTM experiments.Source: §E.2 · p. 20