PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta, Yifan Wang, Andy Liu, Niccolò Ajroldi, David Tweedle, Xingyu, Yang, Isaac Patole, Frank Schneider, George E. Dahl

arXiv:2610.13036v1cs.LG

TL;DR

Optimizer performance is difficult to compare when tuning, co-design, and limited task coverage confound results. This report evaluates 13 submissions under AlgoPerf’s updated self-tuning benchmark, using one configuration across nine workloads. Schedule-Free AdamW v2 (PyTorch) ranked first, while further analyses show trade-offs in step efficiency, target sensitivity, and tuning retention.

  • Problem

    Optimizer performance is difficult to assess because weak baselines, optimizer–model co-design, limited task coverage, and obscured tuning recipes confound comparisons.

    Source: §1 · p. 1

  • Method

    The report evaluates 13 submissions on updated AlgoPerf using a single self-tuning configuration across nine workloads and scores time-to-target performance with performance profiles.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    Schedule-Free AdamW v2 (PyTorch) ranked first; preconditioned and two-loop methods made strong per-step progress but required systems optimizations to offset step latency.

    Source: §7 · p. 15

  • Takeaways & Limitations

    Tuning remains a substantial contributor to practical performance, and algorithms differ in how much specialized performance they retain under a single default configuration.

    Source: §7 · p. 15

    Equal relative target relaxations are not equally lenient across workloads because targets lie at different distances from metric saturation.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Evaluating deep learning optimizer is challenging because confounding factors such as hyperparameter tuning and optimizer-model co-design often obscure true algorithmic performance. We present results of the updated AlgoPerf V1 benchmark, in which we evaluate a new set of training algorithms on wall-clock time-to-target across nine workloads on fixed hardware. This report focuses on the results of the benchmark's self-tuning ruleset, which requires submissions to use a single configuration across all workloads. Out of the 13 submissions in PyTorch and JAX, Schedule-Free AdamW ranked first. In addition to the wall-clock scores, we also analyze the trade-offs between step efficiency and wall-clock execution, revealing that matrix-preconditioned and two-loop methods achieve good sample efficiency but require more aggressive computational optimizations to be competitive in runtime. Finally, our hyperparameter saliency and retention analyses show that the learning rate and momentum are highly sensitive across domains and algorithms, highlighting the need for methods that are more robust against hyperparameter tuning and can generalize well to new tasks.

1 Introduction

The report updates AlgoPerf to evaluate training algorithms while reducing confounds that make their practical performance difficult to compare. Across 13 self-tuning submissions, updated Schedule-Free AdamW ranked first.

1 Introduction

  • Optimizer comparisons are difficult because weak baselines, optimizer–model co-design, limited task coverage, and obscured tuning recipes confound algorithmic performance.Training also relies on fragile empirical recipes and substantial computational budgets.Source: §1 · p. 1
  • Earlier AlgoPerf results ranked Distributed Shampoo first under external tuning and Schedule-Free AdamW first under self-tuning; newer interest includes Muon, SNOO, and AdEMAMix.External tuning samples from a supplied search space, whereas self-tuning evaluates without hyperparameter sampling.Source: §1 · p. 1
  • 13 self-tuning submissions were evaluated on the updated AlgoPerf benchmark, and Schedule-Free AdamW v2 (PyTorch) ranked first, followed by an AdamW-equivalent AdEMAMix submission.The update adds a decoder-only language-model workload, moves to four A100 GPUs, and recalibrates runtime budgets.Source: §1 · p. 2

2 Background

Prior optimizer benchmarks differ in workload, architecture control, training horizon, and evaluation target. The report positions AlgoPerf’s fixed-architecture, cross-workload self-tuning track as a stringent test of generalization.

2 Background

  • The modded nanogpt challenge benchmarks FineWeb-Edu training speed, but permits simultaneous optimizer and architecture changes and uses short horizons that may miss longer-run behavior.This can couple architectural and algorithmic effects in its reported speedups.Source: §2 · p. 2
  • Language-model studies report horizon- and batch-dependent rankings: Muon excels on shorter runs, while SOAP and Kron overtake it on longer runs; matrix methods or momentum-heavy methods lead in large-batch findings.Wen et al. [2026] report the horizon and matrix-method results; Semenov et al. [2025] report large-batch scaling for AdEMAMix and MARS.Source: §2 · p. 2
  • Other benchmarks report lower final losses for Muon and SOAP on diffusion tasks, while tabular-learning results identify Muon as a consistent top performer and EMA as a strong AdamW baseline.The tabular study evaluates held-out generalization rather than time to a target loss.Source: §2 · p. 2
  • AlgoPerf’s self-tuning track tests cross-domain generalization with one fixed hyperparameter set across nine workloads, unlike the related benchmark of Zhang et al. [2026].AlgoPerf permits batch-size tuning per optimizer while keeping the language-model architecture fixed.Source: §2 · p. 2

3 Methodology

The updated benchmark measures time-to-target on fixed workloads and hardware, while limiting workload-specific behavior to batch size. Its self-tuning track applies a single configuration across nine workloads and scores submissions using performance profiles.

3 Methodology

  • The benchmark update adds a decoder-only NanoDO-style FineWeb-Edu 10B workload and changes hardware, runtime budgets, JAX sharding, and evaluation controls.The setup moves from eight V100 GPUs to four A100 GPUs; the submission API also adds controls for batch-normalization updates and evaluation parameters.Source: §3 · p. 2

3.1 Workloads

  • Each workload specifies a dataset, model, loss, target metric and value, and runtime budget; the training objective is to reach its validation target in the least wall-clock time.The benchmark covers nine base workloads across multiple domains.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Submissions

  • The fixed API limits submissions to five functions, with batch size as the only workload-dependent parameter, isolating algorithmic speedups from pipeline or model changes.The functions cover parameter updates, optimizer-state initialization, data selection, batch size, and evaluation preparation.Source: §3.2 · p. 3

3.3 Self-Tuning track

  • Self-tuning requires one fixed hyperparameter configuration across all nine workloads, unlike external tuning’s sampled configurations; the track is intended to reward methods that need less tuning.A submission may adjust hyperparameters during training using a procedure that generalizes, but its score includes time spent trying configurations.Source: §3.3 · pp. 3–4

3.4 Scoring

  • The benchmark score is the normalized area under performance profiles, which track the fraction of workloads solved within a time factor τ of the fastest submission.A score of 1.0 corresponds to training fastest on all workloads.Source: §3.4 · p. 4

4 AlgoPerf leaderboard

Schedule-Free AdamW v2 (PyTorch) leads the wall-clock leaderboard, while step-based and relaxed-target analyses reveal distinct efficiency and ranking patterns. The benchmark also notes that equal target relaxations are not equally lenient across workloads.

4.1 Evaluated algorithms

  • The leaderboard evaluates a diverse set of first-order, momentum-augmented, and matrix-preconditioned algorithms submitted by the community and working group members.Table 2 summarizes the evaluated submissions.Source: §4.1 · p. 4, Table 2 · p. 5

4.2 AlgoPerf scores

  • Schedule-Free AdamW v2 (PyTorch) ranked first, followed by an AdamW-equivalent AdEMAMix submission; the top three leaderboard submissions were PyTorch implementations.Schedule-Free AdamW v2 (JAX) reached targets on a different workload set, with fast (≤1.25τ) time-to-target on five of nine workloads and target attainment on six.Source: §4.2 · p. 6, Figure 1 · p. 6

4.3 AlgoPerf wall-clock vs step efficiency

  • Step-based scores can favor methods that take fewer, more expensive steps: DiLoCo and Muon (JAX) score slightly better by steps than by wall-clock time.More optimized implementations, such as Muon (PyTorch), can perform better on the wall-clock leaderboard.Source: §4.3 · pp. 6–7, Figure 2 · p. 7

4.4 Relaxed AlgoPerf leaderboards

  • At 12% target relaxation, Muon (PyTorch) climbs from sixth to first, while Schedule-Free AdamW v2 (PyTorch) performs consistently across relaxation levels.The same relative relaxation is not equally lenient across workloads because targets lie at different distances from metric saturation; AlgoPerf scores are more stable than ranks as targets change.Source: §4.4 · pp. 7–8, Figure 3 · p. 8

5 Tuning

The tuning analyses measure hyperparameter sensitivity and compare tuning performance on individual workloads with the performance retained by one universal configuration. Learning rate and momentum are sensitive across workloads, while tuning difficulty and retention vary across optimizers.

5.1 Hyperparameter saliency analysis

  • The base learning rate ranks among the two most important hyperparameters across workloads and algorithms, and most algorithms are also sensitive to momentum.Figure 4 shows importance indices at the top 10% performance threshold; higher values indicate a narrower band is needed for top-tier performance.Source: Figure 4 · p. 9, §5.1 · p. 9
  • Top-5% learning-rate ranges can conflict across workloads: NadamW’s LibriSpeech ranges are nearly disjoint, while AdEMAMix’s fastMRI range is disjoint from two workloads.The figure’s Muon learning rate is for its paired optimizer and serves as the basis for its learning-rate multiplier.Source: §5.1 · p. 9, Figure 5 · p. 10

5.2 Hyperparameter tuning overhead and retention analysis

  • DiLoCo had the lowest failure rate (1.0%) and highest median speedup (0.427x), versus 0.063x for Muon and 0x for NAdamW and AdEMAMix.The comparison uses distributions of nine-workload mean time-based speedup scores across 100 configurations per optimizer.Source: §5.2 · p. 10, Figure 6 · p. 11
  • NAdamW (65.5%) and DiLoCo (61.4%) retained the largest shares of oracle performance under one universal configuration, ahead of Muon (47.0%) and AdEMAMix (41.5%).Muon had the highest per-workload oracle score (1.973); universal-configuration scores were 0.989 for NAdamW, 0.786 for DiLoCo, 0.927 for Muon, and 0.583 for AdEMAMix.Source: Figure 7 · p. 11, §5.2 · p. 11
  • DiLoCo had the highest expected score at minimal tuning budgets (k ≤ 5); NAdamW and Muon overtake it as budgets grow, while AdEMAMix needs substantially larger budgets to approach its golden configuration.The budget analysis estimates expected best scores from subsamples of 100 configurations.Source: §5.2 · pp. 11–12

6 Submission Implementation Case Studies

The case studies examine how implementation choices affected benchmark outcomes for Schedule-Free AdamW, Muon, and AdEMAMix. They cover batch-normalization calibration, distributed sharding, and warmup choices.

6.1 Schedule-Free AdamW: Batch Normalization Calibration

  • Schedule-Free AdamW v2 ranked first and reached target on six workloads, while a batch-normalization buffer mismatch had degraded validation on BN-dependent tasks.Calibrating buffers at evaluation weights xt stabilized ResNet validation; ImageNet ResNet-50 changed from a failure mode to a stable run.Source: §6.1 · p. 12, Figure 8 · p. 13

6.2 Muon: Distributed Sharding and Computational Overhead

  • Muon sharding reduced wall-clock time on matrix-dominated workloads, but gains were negligible on compute-dominated workloads where forward and backward passes dominated step time.The PyTorch implementation balanced orthogonalization work across GPUs; systems optimizations lowered per-step latency enough for good configurations to finish within budget.Source: §6.2 · p. 14, Figure 9 · p. 14

6.3 AdEMAMix: Slow-EMA Warmup Horizons and AdamW Equivalence

  • A 500,000-step AdEMAMix warmup kept the second-place submission from meaningfully engaging its slow EMA, making it effectively a tuned AdamW baseline.A budget-scaled warmup activated the slow EMA and reached the FineWeb-Edu 10B target in 0.27× budget versus 0.38× for the baseline, but performed poorly on other workloads; retention was 41.5%.Source: §6.3 · pp. 14–15

7 Conclusion

Schedule-Free AdamW v2 (PyTorch) ranks first on the updated AlgoPerf v1 self-tuning leaderboard. The analyses show trade-offs in step efficiency, hyperparameter sensitivity, and performance retention under a single configuration.

7 Conclusion

  • Schedule-Free AdamW v2 (PyTorch) places first on the updated leaderboard, retaining its crown from the first benchmark.The evaluation covers 13 self-tuning submissions implemented in PyTorch and JAX.Source: §7 · p. 15
  • Preconditioned and two-loop methods such as Muon and Single-Worker DiLoCo make strong per-step progress but need systems optimizations to offset step latency.The report compares wall-clock rankings with step-based rankings and relaxed target thresholds.Source: §7 · p. 15
  • Learning rate and momentum dominate optimizer hyperparameter sensitivity, and top-performing learning-rate intervals are often disjoint across workloads.Source: §7 · p. 15
  • Muon benefits substantially more from per-workload oracle configurations than Single-Worker DiLoCo or NAdamW, which retain more specialized performance under a single default.Source: §7 · p. 15
  • The results indicate substantial room for self-tuning optimizers to improve, with tuning still responsible for a large share of practical performance.Source: §7 · p. 15

A Training Time Tables

The section presents tables for time to target, step execution time, and step-based scoring, alongside a leaderboard and performance profiles for steps to target.

A Training Time Tables

  • Table 5 reports time to target as a fraction of each workload’s self-tuning runtime budget, with a dash indicating an unreached target.Source: Table 5 · p. 18
  • Table 6 gives each submission’s average step time relative to NAdamW (Baseline v0.5) (JAX), excluding the first evaluation interval and using each submission’s own batch size.Source: Table 6 · p. 18

B Step-Based Leaderboard

  • Table 7 compares wall-clock and step-based benchmark scores using the same performance-profile scoring and reports rank changes between them.Source: Table 7 · p. 19
  • The step-based leaderboard and performance profiles report steps to target; Table 8 gives median optimizer steps, which compare updates rather than examples because submissions choose their own batch sizes.Source: Figure 10 · p. 19, Table 8 · p. 20

C Relaxed Leaderboard

The section presents leaderboard comparisons under targets relaxed by 10%, including time-to-target fractions and rank changes.

C Relaxed Leaderboard

  • Table 9 compares the standard and 10%-relaxed self-tuning leaderboards and reports rank changes after targets are relaxed.Source: Table 9 · p. 21
  • Figure 11 presents the self-tuning leaderboard and performance profiles with 10% relaxed targets.Source: Figure 11 · p. 21
  • Table 10 reports time to target as a fraction of each workload’s self-tuning runtime budget; bold values mark targets reached only after relaxation.Source: Table 10 · p. 22

D Training Curves

The training-curve figures compare validation metrics against wall-clock time across nine base workloads. They report distinct patterns for Muon, NAdamW variants, DiLoCo, Schedule-Free AdamW, and AdEMAMix.

D Training Curves

  • Muon (JAX) reaches higher accuracy than Muon (PyTorch) on ImageNet ResNet-50 and ImageNet ViT, but levels off below target on OGBG GNN and WMT Transformer.The implementations use different fixed hyperparameters; Figure 12 plots means across three studies with shaded standard-deviation bands.Source: Figure 12 · p. 23
  • Single-worker DiLoCo trails the NAdamW variants on ImageNet ViT, LibriSpeech Conformer, and OGBG GNN; Cautious NAdamW spikes on Criteo 1TB DLRMsmall and regresses after about four hours on LibriSpeech DeepSpeech.Figure 13 shows means across three studies with shaded standard-deviation bands.Source: Figure 13 · p. 24
  • On ImageNet ResNet-50, both Schedule-Free AdamW v2 submissions train stably while the first versions oscillate; on ImageNet ViT, JAX nearly reaches the target while PyTorch levels off near 0.65 accuracy.Figure 14 also plots two AdEMAMix submissions across the nine base workloads.Source: Figure 14 · p. 25

E Hyperparameter Sweep Details and Search Spaces

The hyperparameter sweeps evaluate shared configurations across all nine workloads, and the accompanying distributions compare sampled settings with the top-performing trials.

E Hyperparameter Sweep Details and Search Spaces

F Top-K Distributions

  • The distributions compare sampled hyperparameter values with the top-10% best-performing trials; greater concentration or peaking in the latter indicates favorable settings.The figures encode the sample space with dashed lines and light fill, and top-10% trials with solid lines and dark fill.Source: Figure 15 · p. 27, Figure 16 · p. 28, Figure 17 · p. 29

G Efficiency and Systems Analysis

The framework comparison finds workload-dependent differences in median step time: DeepSpeech favors PyTorch, while ResNet and ViT are faster in JAX.

G.1 JAX vs. PyTorch

  • The comparison uses Schedule-Free AdamW v1 and v2 submissions with shared core optimizer hyperparameters and holds batch sizes and hyperparameters constant across frameworks.fastMRI, FineWeb-Edu, and Librispeech Conformer are excluded because their training batch sizes differ.Source: §G.1 · p. 31, Figure 19 · p. 31
  • The faster JAX cuDNN LSTM implementation was disabled because of a gradient-calculation bug, and the ResNet and ViT framework disparity remains under investigation.Source: §G.1 · p. 31
  • DeepSpeech is significantly advantaged toward PyTorch, whereas ResNet and ViT are faster in JAX by a large margin.The comparison reports median time per training step.Source: Figure 19 · p. 31, §G.1 · p. 31
Loading 2610.13036v1…