PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards

Zhipeng Chen, Wayne Xin Zhao, Ji-Rong Wen

arXiv:2610.16533v1cs.CLcs.LG

TL;DR

RLVR optimizers are largely inherited from supervised training, raising the question of whether RLVR has a distinct optimization geometry. The paper characterizes that geometry and introduces Ares, which adapts update rank to gradient spectra; across model scales, tasks, and RLVR algorithms, it reports gains over widely used optimizers.

  • Problem

    RLVR optimizers are largely inherited from supervised training, leaving open whether RLVR’s distinct training dynamics induce a different optimization geometry.

    Source: §1 · p. 1

  • Method

    Ares estimates effective rank from gradient spectral energy, tracks it over training, and confines updates to the resulting low-rank subspace.

    Source: §4.2 · p. 5, §4.3 · p. 6, §4.4 · p. 6

  • Results

    Across model scales, tasks, and RLVR algorithms, Ares demonstrates consistent gains over widely used optimizers, with ablations supporting component contributions.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The findings point to adapting optimizer update dimensionality to the evolving spectral structure of gradients rather than fixing or scheduling rank externally.

    Source: §1 · p. 2, §4.4 · p. 6

    The current implementation does not yet realize the potential resource savings from maintaining optimizer moments in factored form.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) is reshaping how large language models (LLMs) acquire reasoning capabilities, yet the optimizers used remain largely inherited from teacher-forcing training. This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a different optimizer? We investigate this question through the spectral structure of LLM gradients and find that gradient energy progressively concentrates into a shrinking effective subspace, which is characteristic of RLVR, as it is absent under supervised fine-tuning (SFT) and is largely removed when the verifiable reward is replaced by a random one. Moreover, the rank required to capture a fixed fraction of gradient energy steadily decreases over training, revealing that the effective dimensionality of useful updates itself evolves with training. These findings motivate a new optimization principle: rather than prescribing the update rank externally, an optimizer should adapt its rank to the evolving spectral structure of the gradient. We introduce Ares (Adaptive Rank-Effective Spectral Optimization), which estimates the effective update rank from the gradient's spectral energy distribution, tracks its evolution, and confines the gradient signal driving each update to the resulting low-rank subspace. Across LLMs of different scales, Ares improves accuracy over widely used optimizers and generalizes across RLVR algorithms and tasks. Results suggest that effective optimization may require optimizers to be designed around the geometry and dynamics of the training process. We release the code and data at https://github.com/RUCAIBox/Ares.

1. Introduction

The introduction asks whether RLVR’s distinct training dynamics create a gradient geometry that inherited optimizers do not address. It reports RLVR-specific spectral concentration and a declining effective rank, motivating Ares, which adapts its update subspace to the gradient spectrum.

1. Introduction

  • RLVR updates aggregate signed contributions from sampled trajectories, so their direction depends on the policy’s evolving sampling distribution.This contrasts with teacher forcing, whose signal is anchored to reference continuations.Source: §1 · p. 1
  • After a brief early transient, the rank needed to capture a fixed fraction of gradient energy declines steadily during RLVR.The effective dimensionality of useful updates therefore evolves during training, rather than merely reflecting a fixed compression choice.Source: Figure 1 · p. 2, §1 · p. 2
  • Top-32 gradient energy rises during RLVR, unlike SFT, while replacing the verifiable reward with a random one removes most of the increase.The random-reward control leaves rollouts and configuration intact, indicating that on-policy sampling alone does not explain the trend.Source: Figure 2 · p. 2, §1 · p. 2
  • Ares estimates effective rank from gradient energy, tracks its evolution, and confines each update to the resulting low-rank subspace.It uses randomized subspace iteration to avoid a full SVD.Source: §1 · pp. 2–3
  • Across model scales, tasks, and RLVR algorithms, Ares improves accuracy over widely used optimizers, with ablations supporting component contributions.The introduction presents this as validation of the spectral-driven principle.Source: §1 · p. 3

2. Related Work

Prior RLVR work has largely changed policy-optimization objectives while leaving the optimizer inherited from supervised training. Existing optimizer families address curvature, update orthogonalization, or memory efficiency, but do not infer update rank from RLVR gradient spectra.

2. Related Work

  • RLVR methods including PPO, GRPO, REINFORCE++, and DAPO reshape training signals while typically leaving the inherited AdamW optimizer untouched.PPO uses a learned value model, which GRPO removes by estimating advantages from sampled responses.Source: §2 · p. 3
  • Optimizer research improves Adam through curvature methods, update orthogonalization, symbolic search, and optimizer-state factorization.Examples include Shampoo, SOAP, Sophia, Muon, Adafactor, and CAME.Source: §2 · p. 3
  • Existing low-rank methods such as GaLore use rank as an externally fixed compression budget, whereas this work infers update rank from evolving RLVR gradient spectra.The authors characterize existing methods as not designed for RLVR.Source: §2 · p. 3

3. Preliminary

The preliminary section introduces rule-based reward optimization and GRPO, then contrasts that training setup with the AdamW baseline and previews Ares’s gradient-subspace workflow.

3. Preliminary

  • RLVR trains a policy to produce correct solutions scored by a rule-based verifier, with an objective that also keeps the policy close to a reference policy.The reward is binary: it returns 1 only when the response reaches the correct final answer.Source: §3 · p. 3
  • GRPO samples a group of responses per prompt and estimates group-relative advantages without a separately learned value model.Positive-advantage responses are reinforced and negative-advantage responses suppressed; importance sampling and clipping reduce variance and support stability.Source: §3 · pp. 3–4
  • AdamW uses bias-corrected first- and second-moment estimates for adaptive updates with decoupled weight decay, and serves as the primary baseline.Ares’s overview instead decomposes 2-D gradients, estimates effective rank, reconstructs gradients in the effective subspace, and updates 1-D tensors with standard AdamW.Source: §3 · p. 4, Figure 3 · p. 4

4. Approach

Ares estimates the gradient’s effective rank from its evolving spectral energy, tracks that estimate, and confines updates to the resulting low-rank subspace. Its approach combines efficient leading-subspace factorization, spectral rank adaptation, and an adaptive parameter update.

4.1. Efficient Gradient Factorization

  • Randomized block power iteration extracts the leading gradient subspace at a cost governed by target rank, avoiding computation of the full spectrum.The stated cost is O(qmnℓ), compared with O(mn^2) for dense SVD, when q(k + p) ≪ n.Source: §4.1 · pp. 4–5

4.2. Spectral Energy-based Rank Estimation

  • Ares converts retained spectral-energy proportions into an entropy-based effective rank, then smooths and clips the estimate to a stable integer rank within [1, k].The rank is tracked over steps with an exponential moving average; its estimate is limited to the target-rank budget.Source: §4.2 · pp. 5–6

4.3. Adaptive Low-Rank Optimization

  • Ares truncates the factorization to the selected leading components, preserving their singular-value weights, and applies an AdamW-style adaptive update with decoupled weight decay.One-dimensional parameters instead receive a standard AdamW step.Source: §4.3 · p. 6

4. Approach

  • The approach combines randomized factorization, entropy-based rank estimation, and a rank-adaptive low-rank update, with per-step cost governed by target rank rather than ambient dimension.Table 1 characterizes Ares as low-rank and spectral-driven, unlike methods using fixed or scheduled rank.Source: §4.4 · p. 6, Table 1 · p. 7

4.4. Summarization and Discussion

  • Ares infers update rank from the observed gradient spectrum rather than fixing or scheduling it, so update dimensionality follows energy concentration over training.Table 1 contrasts this spectral-driven choice with fixed-rank and scheduled-rank alternatives.Source: §4.4 · p. 6, Table 1 · p. 7

5. Experiment

Ares leads the reported accuracy comparisons across three model scales, while ablations and analyses examine its components, computational costs, and evolving rank adaptation.

5.1. Experiment Settings

  • Experiments use DAPO training data, seven mathematical reasoning benchmarks, and a shared RLVR pipeline to compare Ares with standard and low-rank optimizers.The benchmarks are AIME24, AIME25, AIME26, AMC23, MATH500, Minerva Math, and OlympiadBench; average accuracy uses eight sampled responses per problem.Source: §5.1 · p. 7

5.2. Main Results

  • Ares achieves the best average accuracy at 1.7B, 4B, and 8B (27.2, 38.5, 42.1), beating AdamW by 5.0, 2.7, and 2.3 points.The comparison uses GRPO-trained models and reports accuracy across mathematical benchmarks.Source: §5.2 · p. 7, Table 2 · p. 8
  • GaLore is on par with AdamW and worse at 4B (32.8 vs. 35.8), while SGD, RMSprop, and Muon are consistently weak across scales.The authors attribute the insufficiency of fixed-rank projection to RLVR's evolving spectral concentration.Source: §5.2 · p. 8

5.3. Detailed Analysis

  • Removing low-rank updates collapses average accuracy to 2.7; removing dynamic rank yields 26.6, while removing momentum yields 24.1.The ablation on Qwen3-1.7B-Base finds momentum smooths noisy rank estimates and stabilizes the update subspace.Source: §5.3 · pp. 8–9, Table 3 · p. 9
  • Ares update time stays near standard optimizers and far below an exact spectral method, but its current dense moment tensors do not realize optimizer-state memory savings.Randomized subspace iteration extracts leading components without a full SVD; factored moment states are left for future work.Source: Figure 4 · p. 9, §5.3 · p. 9
  • Effective rank declines during RLVR, while the 4B model remains at higher effective rank than the 1.7B model throughout training.The authors connect the decline to progressively concentrated gradient energy and the higher rank to adapting Ares across model scales.Source: §5.3 · p. 9, Figure 5 · p. 10
  • The exponential rank mapping performs best overall, Threshold is consistently worst, and adaptation strategies affect the smaller model more strongly.The comparison covers Exp, Linear, and Threshold strategies across Qwen3-1.7B and Qwen3-4B.Source: Figure 6 · p. 10, §5.3 · p. 10

6. Conclusion

The conclusion characterizes RLVR gradients as having an evolving spectral geometry distinct from the studied controls. It presents Ares as an optimizer built around that geometry and reports consistent gains across the tested scales, tasks, and RLVR algorithms.

6. Conclusion

  • RLVR gradients concentrate energy in leading singular components as training proceeds, lowering effective update rank relative to the studied SFT and random-reward controls.The controls in this study do not show the same pattern.Source: §6 · p. 10
  • Ares infers update rank from the gradient spectrum, adapts it during training, and confines each update's gradient signal to the resulting low-rank subspace.Source: §6 · p. 10
  • Experiments report consistent gains over widely used optimizers across model scales, tasks, and RLVR algorithms, with ablations supporting contributions from each component.The conclusion draws the broader principle that optimizer design should account for training-induced gradient geometry.Source: §6 · p. 10

A. Implementation Details

The implementation trains Qwen3-Base models with GRPO under a shared configuration and sets common Ares hyperparameters across scales. Its step procedure applies spectral low-rank updates to matrices and falls back to AdamW for one-dimensional parameters.

A. Implementation Details

  • Qwen3-Base models at 1.7B, 4B, and 8B train with GRPO for 300 steps using shared rollout settings and separately tuned optimizer learning rates.Low-rank optimizers use rank 64; Ares starts at rank 64 and adapts it online from the gradient spectrum.Source: A. Implementation Details · p. 15
  • Ares uses shared hyperparameters across layers and scales: RBPI oversampling p=10, power iterations q=20, rank-EMA smoothing c=0.9, and scale s=1.Its learning rate is 5 × 10^-6, and rank-to-index mapping uses nearest-integer rounding.Source: A. Implementation Details · p. 15

B. Pseudo Code of Ares

  • Each step uses a spectral low-rank update for two-dimensional weights and standard AdamW for one-dimensional parameters, maintaining moment estimates and tracked rank.RBPI estimates the leading singular subspace, and an EMA smooths and tracks rank.Source: B. Pseudo Code of Ares · p. 15

C. Analysis of the Randomized Block Power Iteration

Ares uses randomized block power iteration (RBPI) to approximate the gradient’s leading spectral subspace without computing a dense SVD. The analysis characterizes RBPI’s cost and approximation behavior, including when it is expected to work well for Ares.

C. Analysis of the Randomized Block Power Iteration

  • RBPI repeatedly applies Aᵀ and A with QR re-orthonormalization, amplifying separation between leading and trailing singular values to approximate the leading subspace.Its block size is ℓ=k+p, with p as oversampling and q as the number of power iterations.Source: C. Analysis of the Randomized Block… · pp. 15–16

C.2. Approximation Guarantee

  • RBPI is exact for rank-k matrices; otherwise, its projection error is bounded relative to the best rank-k error, with the bound improving as power iterations increase and the spectral gap widens.Oversampling adds robustness, while the expected accuracy in Ares is motivated by RLVR gradients’ pronounced spectral gap.Source: §C.2 · pp. 17–18
  • The projection bound controls reconstruction error for Ares’s compressed representation, but accurate individual singular vectors and values additionally depend on a spectral gap.When the leading values are well separated, perturbation results also support accurate recovery of the quantities used to form updates and estimate effective rank.Source: §C.2 · p. 18

C.1. Time Complexity

  • RBPI reduces leading-order cost from O(mn^2) to O(qmnℓ), favorable when q(k+p)≪n, but wall-clock speedup depends on hardware and implementation.The practical comparison also depends on matrix shape and power-iteration overhead.Source: §C.1 · p. 17, §C.3 · p. 18

C.3. Implication for Ares

  • RBPI is suited to Ares because RLVR gradients are spectrally concentrated, making a moderate number of power iterations and a small oversample expected to track the leading subspace closely.The expected cost advantage is substantial when q(k+p)≪n, although practical speedup must be measured.Source: §C.3 · p. 18

D.1. Comparison of Training Efficiency

On Qwen3-4B-Base, Ares reaches AdamW’s accuracy in fewer optimizer steps and finishes at a higher accuracy. The authors link this efficiency to restricting updates to the gradient’s concentrated spectral subspace.

D.1. Comparison of Training Efficiency

  • Because rollout and reward-verification pipelines are shared per step and optimizer overhead is described as modest, step-level efficiency translates to a comparable wall-clock advantage, but not a claimed compute-matched speedup.Source: §D.1 · p. 18
  • Ares reaches AdamW’s accuracy in substantially fewer steps and converges to a higher final accuracy on Qwen3-4B-Base.The comparison is in optimizer steps, not a fully compute-matched speedup.Source: §D.1 · p. 18, Figure 7 · p. 19
  • Restricting updates to the spectral subspace carrying concentrated gradient energy suppresses uninformative directions and yields a more effective update per step.The authors report faster and better convergence from optimizing within the observed RLVR gradient geometry.Source: §D.1 · p. 19

D.2. Detailed Results of Ablation Study

The ablations indicate that Ares’s low-rank update is central to its performance, while dynamic rank and the other components contribute additional gains. The full-space removal is a controlled ablation, not an AdamW-equivalent baseline.

D.2. Detailed Results of Ablation Study

  • 2.7 on Qwen3-1.7B-Base and 8.6 on Qwen3-4B-Base: removing low-rank projection collapses average accuracy, below separately tuned AdamW at 22.2 and 35.8, respectively.The full-space ablation keeps other Ares settings tuned for low-rank gradients, so it is not an AdamW-equivalent comparison.Source: §D.2 · p. 19
  • Dynamic rank, spectral normalization, adaptive learning rate, and momentum each provide smaller but consistent gains at both model scales.Source: §D.2 · p. 19
  • Removing dynamic rank also trails the full model, indicating that inferring rank from the gradient spectrum rather than fixing it accounts for part of the improvement.Source: §D.2 · pp. 19–20

D.3. Detailed Results of Various Rank Adaptation Methods

Smooth rank mappings outperform a hard cutoff, Ares’s advantage extends beyond mathematical training, and rank momentum is beneficial across a broad coefficient range. Together, these findings cover rank adaptation, domain transfer, and rank tracking.

D.3. Detailed Results of Various Rank Adaptation Methods

  • Average accuracy is 27.2 / 38.5 for Exp and 24.1 / 38.8 for Linear, versus 22.3 / 34.6 for Threshold; the smooth mappings outperform the hard cutoff on both scales.The paper attributes the difference to smooth mappings tracking energy concentration continuously rather than making an abrupt rank change.Source: §D.3 · p. 20

D.4. Performance on Various Domain Tasks

  • Ares achieves the best average accuracy across MMLU-Pro sub-benchmarks and GPQA, including distant domains, with a wider margin at 4B.The model is trained only on mathematical data; the reported gains span STEM and fields such as philosophy, law, and business.Source: Figure 8 · p. 21, §D.4 · p. 21

D.5. Effectiveness of Momentum

  • Rank momentum improves performance over the momentum-free variant, and accuracy remains strong across a broad range of coefficients rather than requiring narrow tuning.Larger coefficients smooth noisy per-step rank estimates, while smaller ones allow sharp fluctuations that the authors associate with degraded performance.Source: §D.5 · p. 21

D.6. Sensitivity Analysis of Initial Rank

Ares improves average accuracy across additional RLVR algorithms, while its initial-rank budget can be set loosely once it covers most gradient energy; on SFT, it shows no consistent gain over AdamW.

D.7. Adaptation on Various RLVR Algorithms

  • Across RLOO [1], REINFORCE++ [14], and GDPO [21], Ares improves average accuracy at both scales by up to +5.1 points, with only a handful of per-benchmark regressions.These algorithms extend evaluation beyond the GRPO setting of the main results.Source: Table 7 · p. 22, §D.7 · pp. 22–23

D.6. Sensitivity Analysis of Initial Rank

  • Once k covers most gradient energy, Ares remains stable and accurate across a wide range, making k a loose upper bound rather than a sensitive tuning choice.When k is too small, the retained subspace misses non-negligible gradient components and performance is poor.Source: §D.6 · pp. 21–22, Figure 10 · p. 22

D.8. Effectiveness of Ares on SFT Process

  • 35.3 vs. 34.8 average accuracy for Ares and AdamW, respectively, accompanies nearly identical SFT loss curves and no consistent downstream gain.The authors attribute the contrast with RLVR to SFT gradients being less spectrally concentrated, leaving little low-rank structure for Ares to exploit.Source: Table 8 · p. 23, §D.8 · p. 23, Figure 11 · p. 23

Related papers

Loading 2610.16533v1…