Ranger21: a synergistic deep learning optimizer

1 Introduction

• Ranger21 combines AdamW with eight selected enhancements, testing and integrating ideas that the authors found composable.The enhancements address areas including gradient normalization, momentum, weight decay, and step size.Source: §1 · p. 1
• Ranger21 improved validation accuracy and training speed over AdamW on ImageNet2012 and trained a ResNet50 without Batch Normalization layers.Training was smoother, and certain validation levels were reached up to 3x faster.Source: §1 · p. 1
• The paper describes its components, full algorithm, ImageNet2012 experiments, and conclusions across five sections.Source: §1 · p. 1

2.1 Adam (adaptive moment estimation)

• AdamW is the core of Ranger21, with Adam computing an adaptive step size for each parameter from two moving averages of stochastic gradients.Source: §2.1 · p. 2
• The algorithm specifies learning rate, weight decay, Adam decay rates, numerical-stability epsilon, and an iteration limit as inputs.Source: §2.1 · p. 2
• Its update sequence maintains and bias-corrects moment estimates, forms an update vector, applies weight decay, and updates parameters.Source: §2.1 · p. 2

2.2 Adaptive Gradient Clipping

• Large gradients from sporadic high-loss minibatches can destabilize training, especially with smaller batches and higher learning rates.Source: §2.2 · p. 2
• Raw gradient clipping can destabilize training, and choosing its threshold requires fine-grained tuning to model depth, batch size, and learning rate.Source: §2.2 · p. 2
• Adaptive Gradient Clipping sets its threshold dynamically in proportion to unit-wise gradient-to-parameter norm ratios.The method works on individual rows rather than a full layer; ε defaults to 10^-3 and τ to 10^-2.Source: §2.2 · pp. 2–3
• Adaptive Gradient Clipping accelerated training while requiring only minimal tuning.Source: §2.2 · p. 3

2.3 Gradient Centralization

• Gradient Centralization computes each multidimensional layer’s gradient mean along all but the first axis and subtracts it from the gradient.Source: §2.3 · p. 3
• Gradient Centralization was associated with improved generalization, smoother training curves, and faster convergence on networks with fully connected and/or convolutional layers.The authors describe it as a regularizer that constrains the loss function and should smooth the optimization landscape.Source: §2.3 · p. 3

2.4 Positive-Negative Momentum

• Positive-Negative Momentum maintains separate first-moment estimates for odd and even iterations, then combines them with positive weight on the current estimate and a negative step on the previous one.Source: §2.4 · p. 3
• Positive-Negative Momentum improved performance across a variety of datasets and integrated complementarily with Ranger21’s other algorithms.Source: §2.4 · p. 3

2.5 Norm loss

• AdamW-style weight decay subtracts a learning-rate-scaled decay term from parameters during the update.Source: §2.5 · p. 4
• Norm Loss pushes a weight matrix toward unit norm, whereas traditional weight decay pushes weights toward zero.The proposed loss uses the matrix’s Euclidean norm.Source: §2.5 · p. 4
• In testing, Norm Loss acted as a soft regularizer for the weight space across a large variety of hyper-parameters.Source: §2.5 · p. 4

2.6 Stable Weight Decay

• Because the effective step size depends on the gradient magnitude as well as the learning rate, decay calibrated early can be too large later as the estimated second moment falls.Source: §2.6 · p. 4
• Stable Weight Decay reweights AdamW-style decay to account for the effective step size.The reweighting uses the layer-wide mean, as described for equation 6.Source: §2.6 · p. 4
• Stable Weight Decay gave significant generalization improvements on vision tasks, and its benefits were additive with Norm Loss.The two methods approach weight regularization from different perspectives.Source: §2.6 · p. 4

2.7 Linear learning rate warm-up

• The warm-up is linear for 2 · (1 − β2)^−1 iterations, capped at twarmup, defaulting to 22% of total iterations.With default hyperparameters, the uncapped duration is about 2000 iterations.Source: §2.7 · p. 4
• The cap limits the warm-up when the prescribed duration would be too long for a shorter training run.Source: §2.7 · p. 4
• Linear warm-up behaved similarly to Rectified Adam, avoiding excessive early step sizes while being simpler to implement.Source: §2.7 · p. 4

2.8 Explore-Exploit learning rate schedule

• The schedule combines linear warm-up, flat exploration, and linear warm-down; the cited Fastai leaderboard reports cosine annealing, described as extremely similar, dominating.The default warm-down occupies 28% of iterations; the figure illustrates the schedule over 10,000 iterations.Source: §2.8 · p. 5, Figure 1 · p. 5

2.9 Lookahead

• Lookahead maintains an exponential moving average of weights and substitutes it for current weights every klookahead steps, five by default.Source: §2.9 · p. 5
• In Ranger21, Lookahead’s benefit persists through the added components, with a net upward shift in training validation curves and higher final validation accuracy across various datasets.Source: §2.9 · p. 5

• Algorithm 4 combines all previously introduced components into Ranger21.Source: §3 · p. 5
• The eight ideas are applied to Adam because the authors found them orthogonal, compatible with Adam, and synergistic in their tests.Source: §3 · p. 6
• The authors report that Ranger21’s additional hyperparameters perform well at default values across a wide range of tasks without tuning.They attribute this partly to components such as Stable Weight Decay reducing hyperparameter sensitivity.Source: §3 · p. 6

4 Experiments

• The ImageNet2012 comparison trained Ranger21 and AdamW for 60 epochs with a default learning rate of 3e−3 and label smoothing of 0.1.The normalizer-free experiment stopped at 60 epochs; the authors note that 300-epoch ImageNet training is computationally expensive and adds little value to this comparison.Source: §4 · p. 6, §4.2 · p. 6

4.1 ResNet50

• 73.69% validation accuracy for Ranger21 was 2.61% higher than AdamW’s highest accuracy of 71.08% on ResNet50.Ranger21 also had consistently better loss and a smoother training curve.Source: §4.1 · p. 6
• At 10 epochs, Ranger21 reached loss 0.03579, which AdamW reached only at epoch 35.Source: §4.1 · p. 6

4.2 Normalizer-Free ResNet50

• On a ResNet50 without batch normalization, AdamW remained at high loss (0.1), while Ranger21 improved steadily through the 60-epoch cutoff.The authors repeated the experiment five times with different random seeds, and all attempts failed similarly for AdamW.Source: §4.2 · p. 6

5 Conclusion

• Ranger21 usually accelerates learning and achieves higher validation accuracy without compromising generalization by combining improvements across optimizer sub-areas.The authors frame this combination as a way to make the optimizer significantly better than its individual parts.Source: §5 · p. 7
• Ranger21 trained a Normalizer-Free ResNet50, while Adam was stopped at 10 epochs because it was not converging.The authors report that other optimizers fail to train this architecture.Source: §5 · p. 7, Figure 4 · p. 9
• ImageNet2012 was chosen to showcase Ranger21, with results intended as a representative picture of its possible real-world applicability.The authors report stronger outperformance on smaller datasets but focus on ImageNet2012.Source: §5 · p. 8
• Experiments are only a proxy for real-world behavior because each dataset and network architecture creates a unique loss landscape.Source: §5 · p. 8
• A finer-grained learning-rate scheduler is a future target for improving performance and reducing dependence on the default learning rate.Source: §5 · p. 8
• The authors want to test whether some optimizer components benefit transformers more than other architectures.They report having experimented with transformer-type architectures.Source: §5 · p. 8
• A broader future direction is a modular optimizer paired with a meta-optimizer that selects components for a given task.Source: §5 · p. 9