PaperlayerView exact revision on arXiv

Source-linked AI summary

On Predictability of Reinforcement Learning Dynamics for Large Language Models

Yuchen Cai, Ding Cao, Xin Xu, Zijun Yao, Yuqing Huang, Zhenyu Tan, Benyi Zhang, Guangzhong Sun, Guiquan Liu, Junfeng Fang

arXiv:2510.00553v3cs.LGcs.AI

TL;DR

RL-trained LLMs lack a clear account of how parameter updates produce reasoning gains. The paper analyzes update matrices, identifies dominant and predictable Rank-1 dynamics, and introduces AlphaRL, which accelerates training while preserving most reasoning performance. These findings provide a predictive lens on RL dynamics and a practical acceleration approach, though their theoretical foundations remain incomplete.

  • Problem

    The paper addresses the limited understanding of parameter dynamics during RL training and their relationship to reasoning gains in LLMs.

  • Method

    The authors analyze singular subspaces and early-training trajectories of RL parameter updates, then use Rank-1 dynamics to forecast final updates with AlphaRL.

  • Results

    Rank-1 updates recover nearly all reasoning improvements, Rank-1 trajectories are highly linear, and AlphaRL achieves up to 2.5× acceleration while retaining more than 96% of final reasoning capability.

  • Takeaways & Limitations

    RL reasoning gains can be studied through a low-dimensional, predictable update structure that supports plug-in acceleration without extra modules or hyperparameter tuning.

  • Takeaways & Limitations

    The conclusions are primarily empirical and lack rigorous theoretical foundations, while AlphaRL remains constrained by RL algorithm design and stability.

Abstract

from arXiv · show

Recent advances in reasoning capabilities of large language models (LLMs) are largely driven by reinforcement learning (RL), yet the underlying parameter dynamics during RL training remain poorly understood. This work identifies two fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1 Dominance, where the top singular subspace of the parameter update matrix nearly fully determines reasoning improvements, recovering over 99\% of performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace evolves linearly throughout training, enabling accurate prediction from early checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the generalizability of these properties. More importantly, based on these findings, we propose AlphaRL, a plug-in acceleration framework that extrapolates the final parameter update using a short early training window, achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning performance without extra modules or hyperparameter tuning. This positions our finding as a versatile and practical tool for large-scale RL, opening a path toward principled, interpretable, and efficient training paradigm for LLMs.

1 Introduction

The paper investigates whether RL-induced parameter updates follow consistent principles that explain reasoning gains. It identifies Rank-1 Dominance and Rank-1 Linear Dynamics, then uses them to motivate AlphaRL acceleration.

  • The study asks whether RL-guided parameter updates obey consistent principles and how those principles give rise to reasoning capabilities.
  • Adding only the top singular component of the parameter update recovers nearly all RL reasoning improvements throughout training.This property is formalized as Rank-1 Dominance.
  • Rank-1 Subspace trajectories evolve almost linearly, with R2 exceeding 0.96, enabling forecasts from short early training windows.This property is formalized as Rank-1 Linear Dynamics.
  • Experiments across diverse LLMs and training algorithms report 99.17% average reasoning recovery for Rank-1 Dominance and average R2 of 0.914 for Rank-1 evolution.The supplied passage is truncated before the complete prediction result, so only these reported values are stated.
  • AlphaRL uses an early training window to estimate Rank-1 initialization and growth, achieving up to 2.5× acceleration while retaining more than 96% of final reasoning capability.The method requires no extra modules or hyperparameter tuning.

2 Dominance of Rank-1 Subspace (Property 1)

This section shows that RL reasoning gains are concentrated in the leading singular subspace of parameter updates, unlike SFT and distillation. Analyses of training progression, scaling, update norms, and embeddings explain this concentration.

  • Results on Fully Trained Models: A single Rank-1 Subspace recovers performance close to the fully trained RL model and can surpass it for RLOO, GRPO, and DAPO.SFT and DIST require more subspaces to obtain gains, whereas RL concentration appears after full-parameter training.
  • Results across the RL Process: At later RL checkpoints, Rank-1 performance matches the fully trained model, while early checkpoints remain slightly lower.The authors hypothesize that training progressively concentrates update directions into stable reasoning-enhancement subspaces.
  • Ablation Study: Rank-1 dominates other individual subspaces, whose reasoning contributions generally decline with their corresponding singular values.Several higher-ranked subspaces still contribute substantially, indicating that the leading direction is central but not exclusive.
  • Scaling Experiment: Performance rises rapidly with scaling factor λ and saturates near λ≈0.7, slightly below the full-update norm.Further scaling may yield diminishing returns after the principal reasoning contribution is realized.
  • Update Characteristics: RL updates have smaller norms than SFT and DIST but allocate a larger fraction of update information to Rank-1 and Rank-1% Subspaces.Figure 5 compares update norms and captured information across methods.
  • Distribution Shifts of Embedding Space: RL causes minimal embedding-space distribution shift, whereas SFT and DIST produce noticeable global shifts, especially DIST for some tokens.The authors associate RL improvements primarily with high-level information-flow adjustments rather than broad lower-level representation changes.
  • Approximate Low-rank of ΔW: The parameter update ΔW exhibits approximate low-rank structure in RL but not in SFT or DIST.The paper further reports that modifying a small number of tokens through Rank-1 can achieve reasoning performance comparable to the fully trained model.

3 Linear Dynamics of Rank-1 Subspace (Property 2)

The Rank-1 Subspace follows smooth, nearly linear trajectories during RL, and its relationship with reasoning accuracy can be modeled using PLS regression. Linearity varies across modules and tracks their functional contribution to reasoning updates.

  • 3.1 Exploring the Dynamics of Rank-1 Subspace: Rank-1 trajectories are smooth and nearly linear, with color progression aligned to training progress, indicating a stable update direction.
  • 3.2 Rank-1 Linearity & Module Importance: PLS regression models Rank-1 trajectories against checkpoint accuracy, with some modules achieving R^2 values close to 1.
  • 3.2 Rank-1 Linearity & Module Importance: Modules with low linearity often exhibit fragmented trajectories, directional shifts, and unstable relationships with accuracy.
  • 3.2 Rank-1 Linearity & Module Importance: MLP modules, especially in mid-to-high layers, generally have higher R^2, whereas self-attention modules show lower and noisier linearity.
  • 3.2 Rank-1 Linearity & Module Importance: Performance declines as the minimum R^2 of selected module windows decreases, supporting R^2 as a measure of module functional contribution.

4 AlphaRL: A Free Acceleration for RL Training

AlphaRL predicts later Rank-1 updates from early training trajectories and their accuracy relationships, then applies the predicted updates to accelerate RL training. Across six reasoning benchmarks, it approaches fully trained performance while reducing training time.

  • 4 AlphaRL: A Free Acceleration for RL Training: AlphaRL accelerates RL by predicting the final parameter update from early Rank-1 Subspace dynamics, bypassing full training.
  • 4.1 Acceleration Method: AlphaRL scales Rank-1 vectors, fits their relationship with relative accuracy using one-component PLS, and inverts the fit for a target accuracy.
  • 4.2 Main Results: At 40% training, AlphaRL-enhanced RLOO and GRPO models reach 96% of fully trained reasoning performance and surpass the fully trained model on MATH-500.
  • 4.2 Main Results: For DAPO, AlphaRL reaches 102% relative accuracy on GPQA, while evaluation responses remain broadly similar to those of the fully trained model.
  • 4.2 Main Results: AlphaRL accelerates training by up to 2.5 times while retaining reasoning performance close to that of the fully trained model.

5 Related Work

Prior work has used RL to improve instruction following, alignment, and reasoning, while studying emergent behaviors such as sampling efficiency and entropy collapse. These studies motivate examining the parameter dynamics of RL itself.

  • Reinforcement Learning for LLMs: RL progressed from RLHF for instruction following and preference alignment to RLVR for mathematical and programming reasoning.
  • Emergent Behaviors of Reinforcement Learning: Related studies report improved pass@1 sampling efficiency without direct reasoning enhancement and identify entropy collapse as an early loss of exploratory capacity.

6 Limitations and Future works

The paper’s laws are supported primarily by empirical observations and lack rigorous theoretical foundations. AlphaRL’s effectiveness is also constrained by the design and stability of the underlying RL algorithms.

  • 6 Limitations and Future works: The proposed low-rank dynamics remain primarily empirical and lack rigorous theoretical foundations.
  • 6 Limitations and Future works: Future work proposes nonlinear forecasting, robust reward modeling, causal analysis, and combining Rank-1 regularities with high-rank corrections.
  • 6 Limitations and Future works: AlphaRL’s effectiveness is constrained by the design and stability of RL algorithms.

7 Conclusion

The paper identifies Rank-1 Dominance and Rank-1 Linear Dynamics as two laws of RL updates, then uses them in AlphaRL to forecast later updates from early checkpoints while preserving reasoning performance.

  • Rank-1 Dominance concentrates reasoning improvements in the top singular direction of RL-induced parameter updates.
  • Rank-1 Linear Dynamics means the dominant direction evolves predictably and approximately linearly throughout training.
  • AlphaRL accelerates training by up to 2.5 times while retaining most reasoning performance by forecasting later updates from early checkpoints.The method is described as plug-and-play and reduces computational cost while preserving reasoning performance close to full training.

Ethics Statement

The study reports no human subjects, personal data, sensitive information, conflicts of interest, or ethical risks.

  • The research uses publicly available benchmark datasets and model parameters without human subjects or personal data.
  • The authors state that the study follows the ICLR Code of Ethics and has no potential conflicts of interest or ethical risks.

Use of LLMs

LLMs were used only to polish the manuscript's language and style, without changing its technical content, results, or scientific claims.

  • LLMs were used for clarity, grammar, readability, and stylistic refinement rather than for technical or scientific contributions.
  • The authors state that the ideas, methods, experiments, and conclusions are their original contributions.

A Additional Experiment

Additional experiments report that AlphaRL's extrapolation and Rank-1 update structure persist across larger and diverse model initializations, as well as adversarial self-play settings.

  • Larger-scale models: AlphaRL accurately reproduces later-stage performance gains from early checkpoints on 9B and 14B models, supporting predictable RL dynamics at larger scales.
  • Diverse base models: Rank-1 update structure and AlphaRL effectiveness remain consistent across distilled and instruction-tuned base models.
  • Adversarial self-play: In adversarial self-play, RL updates retain a stable Rank-1 structure and AlphaRL extrapolates late-stage improvements from early checkpoints.

B Experimental Setup

The experiments compare multiple training methods, models, datasets, and update representations to evaluate low-rank RL dynamics across diverse settings.

  • Model Setup: The setup defines pretrained parameters θ_init and updated parameters θ_full produced by a training method M.These parameters provide the base and trained models used to analyze parameter updates.
  • Methods: Experiments include DIST, SFT, PPO, RLOO, GRPO, Dr.GRPO, DAPO, on-policy distillation, DPO, and Spiral.The methods span supervised, reinforcement-learning, distillation, preference-optimization, and adversarial-training paradigms.
  • Models: The evaluation spans models from 7B to 32B, including DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, GLM-9B, Qwen3-14B, and DAPO-Qwen-32B.Several model families and initialization strategies are represented, including cold-start training for Llama3.1-8B-Instruct.
  • Evaluation: The main low-rank evaluations report update norms and performance under Rank-1 and Rank-k% subspaces on MATH-500.Table 7 covers update norms and captured update information, while Table 8 covers MATH-500 performance.
  • Training Procedure: RLVR experiments use Qwen3-8B-Base trained on DAPO-Math-17K, with checkpoints saved after each training batch for later evaluation.The shared RL configuration uses 2,048-token prompts, 20,480-token responses, mini-batches of 32, and 16 gradient-accumulation iterations.

C In-depth Analysis of the Low-rank Phenomenon

The analysis attributes RL’s low-rank effectiveness to constrained, concentrated updates that alter critical reasoning-token preferences, while tracking Rank-1 trajectories with PLS.

  • Potential Causes: RL gains arise through limited, focused parameter updates within critical subspaces despite stabilization constraints such as KL, logits, and gradient clipping.The analysis contrasts this concentrated update pattern with broad parameter drift and relates it to low-rank effectiveness.
  • Token Patterns: Non-greedy tokens cluster around initial-answer and reasoning-transition positions, such as “Alright,” “Let,” “But,” and “Wait.”These positions correspond to early problem representation and later reasoning adjustments.
  • Token-Level Effects: Rank-1 information reshapes token-level decoding preferences and significantly reduces non-greedy tokens, aligning outputs with RL-enhanced reasoning patterns.The effect is interpreted as activating and stabilizing latent reasoning trajectories already present in the base model.
  • Behavioral Comparison: RL changes reasoning behavior less than DIST and SFT, while Rank-1 updates recover the critical token-level adjustments associated with fully trained RL models.The reported mechanism emphasizes sparse, high-impact positions rather than wholesale changes to the output distribution.
  • Linear Projection Methods: PLS models checkpoint-wise Rank-1 vectors against reasoning accuracy by extracting a covariance-maximizing predictive direction and fitting accuracy in that latent space.The vectors are standardized after being stacked into the design matrix Ũ1.
Loading 2510.00553v3…