Source-linked AI summary
AdamX: Cosine similarity meets gradient descent
Francisco Caldas, Ruben Belo, Cláudia Soares
TL;DR
Adaptive optimizers primarily adjust gradient magnitudes, leaving successive-gradient alignment comparatively underused. AdamX adds a bounded cosine controller to Adam/AMSGrad-style updates and evaluates it with regret analysis and epoch-based benchmarks. It is competitive with established optimizers and achieves the best result in the considered CIFAR-10 setting, while its analyzed envelope can limit step-size re-increases after alignment improves.
Problem
Existing adaptive optimizers primarily modify gradient-magnitude information, while the use of successive-gradient alignment as a direct update-magnitude signal remains comparatively limited.
Method
AdamX adds a bounded cosine-similarity controller to Adam/AMSGrad-style adaptive updates and studies a simplified variant under online-convex-optimization regret analysis.
Results
AdamX is competitive with ten established optimizers and achieves the best result in the considered CIFAR-10 setting.
Takeaways & Limitations
Directional alignment is presented as a promising lightweight source of adaptivity for first-order optimization.
Takeaways & Limitations
The regret analysis uses a non-increasing envelope that can prevent effective step sizes from increasing again when gradients become strongly aligned.
Abstract
from arXiv · showhide
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.
1 Introduction
AdamX addresses the limited use of gradient-direction alignment in adaptive optimization by adding a bounded cosine controller to an Adam/AMSGrad-style optimizer. The paper combines this lightweight geometric signal with regret analysis and benchmark evaluation.
- 1 Introduction: AdamX targets a gap in existing optimizers, which primarily modify gradient-magnitude statistics while comparatively underusing alignment between successive gradients.The introduction contrasts this focus with methods including AMSGrad, AdamW, RAdam, AdaBelief, Lion, and Muon.
- 1 Introduction: Cosine similarity provides a scale-independent signal: aligned gradients suggest consistent progress, whereas opposing gradients may indicate oscillation or rapidly changing trajectories.The paper motivates this signal as computationally inexpensive and directly related to update-magnitude adaptation.
- 1 Introduction: The method investigates whether a lightweight scalar geometric signal can add adaptivity while preserving the implementation simplicity and scalability of first-order Adam-type methods.This objective is contrasted with richer curvature or preconditioning approaches such as Shampoo and SOAP.
- 1 Introduction: AdamX integrates a bounded cosine-similarity controller into an Adam-style moment-normalized update with a monotone second-moment envelope.The controller uses alignment between consecutive gradients to modulate update magnitudes while retaining the practical structure of Adam-type methods.
- 1 Introduction: The paper evaluates AdamX across benchmark datasets and architectures using epochs required to reach predefined test-performance thresholds, alongside an online-convex-optimization analysis.The evaluation uses default settings and measures progress through test-performance thresholds rather than only training metrics.
2 Method
AdamX combines Adam-style moment normalization and an AMSGrad variance envelope with a bounded controller based on cosine similarity between consecutive gradients. The method scales updates according to directional agreement while a simplified variant supports online-convex regret analysis.
- 2 Method: AdamX retains Adam’s exponential moving averages of gradients and squared gradients, with bias-corrected estimates used for adaptive normalization.The moving averages use decay rates β1 and β2, and squared-gradient operations are coordinate-wise.
- 2 Method: The cosine controller multiplies AdamX updates according to consecutive-gradient alignment, increasing effective magnitude for aligned gradients and reducing it for opposing gradients.Its strength is controlled by λ, while δ prevents division by zero and γ1 is set to 1 because no previous gradient exists initially.
- 2 Method: Setting λ = 0 removes the alignment controller and recovers AMSGrad, making the alignment mechanism an additive component over the baseline adaptive structure.This isolates the role of directional alignment within the optimizer design.
- 2 Method: The AMSGrad-style variance envelope makes the adaptive denominator coordinate-wise non-decreasing, preventing effective coordinate-wise stepsizes from increasing solely because second-moment estimates decrease.The envelope is formed from coordinate-wise maxima of past estimates.
- 2 Method: The resulting update combines the cosine controller, moment normalization, and monotone variance envelope to transform parameters using a base learning rate η.Algorithm 1 computes gradients, updates moments, applies the controller and envelope, then stores the current gradient for the next iteration.
- 2 Method: The regret analysis studies a simplified momentum-free OCO variant that uses projection, an adaptive diagonal metric, and a non-increasing envelope for alignment-scaled steps.These modifications isolate the bounded controller while enabling standard adaptive online-learning analysis.
3 Regret Guarantees for AdamX-OCO
AdamX-OCO combines an AMSGrad-style monotone denominator with cosine-based alignment control in an online convex optimization framework. The analysis provides regret guarantees for an envelope-based variant and a variation-dependent guarantee for the raw practical alignment update, while exposing limitations from momentum and adversarial variation.
- Algorithm 2 AdamX for OCO: AdamX-OCO uses a monotone second-moment envelope, an alignment-based step controller, and adaptive projected updates over a convex feasible set.Algorithm 2 defines the cosine multiplier, monotone step-size envelope, diagonal metric, and projected parameter update.
- Convex Regret Bound for AdamX-OCO: The envelope-based AdamX-OCO variant retains a convex regret bound with the standard sublinear rate, while alignment changes only the constants.The alignment parameter scales constants by eλ because the multiplier is bounded between e−λ and eλ.
- Using Raw Alignment Instead of the Envelope: The monotone envelope is theoretically convenient but prevents effective step sizes from increasing after prior decreases, even when gradients later align strongly.Replacing it with the raw step size removes monotonicity of the adaptive metric and requires a data-dependent variation bound.
- Using Raw Alignment Instead of the Envelope: The practical raw-alignment update admits a variation-dependent regret bound when metric variation is controlled, but oscillating cosine signals can make that term large.This formulation is more faithful to the practical optimizer than the monotone envelope used for the main proof.
- Momentum and the Full AdamX Algorithm: The full momentum-based AdamX analysis introduces a momentum-bias term that can scale linearly with T for adversarial gradient sequences.The simplified OCO guarantee therefore does not directly cover the full practical optimizer with momentum.
4 Experiments
The experiments compare AdamX with established first-order optimizers using epochs to a target test accuracy, emphasizing generalization rather than runtime. Across MNIST and CIFAR-10, AdamX is competitive with or outperforms leading baselines, while training-loss behavior does not always predict test performance.
- Evaluation protocol: The protocol measures epochs needed to reach predetermined test-set accuracy targets rather than wall-clock runtime.This treats gradient evaluations as the main computational-cost proxy and emphasizes generalization.
- Training loss: AdamX, AMSGrad, and Yogi show lower or smoother training-loss trajectories, whereas some alternatives exhibit higher loss or larger oscillations.RMSProp is omitted from the MNIST loss plot because of significantly higher loss values; on CIFAR-10, RAdam, Adam, and AdamW show larger oscillations.
- MNIST: On MNIST, AdamX is among the fastest optimizers to reach 0.994 test accuracy, alongside AMSGrad and Yogi.The evaluation uses a three-layer CNN, default hyperparameters, five seeds, and a 100-epoch limit.
- CIFAR-10: AdamX achieves the lowest mean number of epochs on CIFAR-10 and compares favorably with AMSGrad and Adam.Five of eleven optimizers fail to reach the 0.84 test-accuracy target in the 100-epoch budget.
- Interpretation: Lower training loss does not necessarily imply better generalization: Adagrad attains low MNIST training loss but needs more epochs to reach the test-accuracy target.Table 1 summarizes the cross-experiment results, with AdamX and AMSGrad requiring the fewest gradient evaluations overall.
5 Conclusions
AdamX augments Adam/AMSGrad-style updates with a bounded cosine-similarity controller and performs competitively across the evaluated optimizers. It achieves the best result in the considered CIFAR-10 setting while motivating further work on robustness and scale.
- AdamX combines Adam/AMSGrad-style optimization with a bounded cosine-similarity controller, supported by a simplified OCO analysis.
- AdamX obtains the lowest mean number of epochs to reach the test-accuracy target, with similar values for AMSGrad and Adam.
- Training-loss results on CIFAR-10 show lower loss for AdamX, AMSGrad, and Yogi, while RMSProp is omitted because its loss is significantly higher.
- AdamX achieves the best result in the considered CIFAR-10 setting while remaining competitive with ten established optimizers.