Source-linked AI summary
Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration
Zhipeng Chen, Tao Qian, Wayne Xin Zhao, Ji-Rong Wen
TL;DR
RLVR’s extensive exploration makes training costly, while the dynamics of its parameter updates remain insufficiently understood and may not be linear. NExt models nonlinear trajectories in a LoRA-induced rank-1 subspace to extrapolate future parameters, reducing computational overhead by approximately 37.5% while maintaining or improving performance across settings.
Problem
RLVR requires extensive exploration, while existing rank-1 linear extrapolation methods rely on an underexplored assumption that one dominant direction sufficiently characterizes the parameter transformation.
Method
NExt extracts rank-1 parameter-difference subspaces from LoRA-based RLVR checkpoints, trains a trajectory predictor, and uses predict-extend extrapolation for future model parameters.
Results
37.5% lower computational overhead is reported while maintaining or improving model performance across different model scales, hyperparameters, downstream tasks, and RLVR algorithms.
Takeaways & Limitations
NExt provides a nonlinear low-rank approach for accelerating RLVR while remaining compatible with varied training algorithms and tasks.
Takeaways & Limitations
Further investigation is needed into internal parameter-update patterns to support additional reductions in computational overhead and test-time scaling.
Abstract
from arXiv · showhide
Recently, scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm for significantly improving model capabilities, which requires guiding the model to perform extensive exploration and learning, leading to substantial computational overhead and becoming a key challenge. To reduce the number of training steps, Prior work performs linear extrapolation of model parameters. However, the dynamics of model parameter updates during RLVR training remain insufficiently understood. To further investigate the evolution of LLMs during RLVR training, we conduct empirical experiments and find that the rank-1 subspace of the model does not evolve linearly, and its dominance over the original parameters is further amplified during LoRA training. Based on the above insights, we propose the \textbf{N}onlinear \textbf{Ext}rapolation of low-rank trajectories (\textbf{NExt}), a novel framework that models and extrapolates low-rank parameter trajectories in a nonlinear manner. Concretely, we first train the model using LoRA and extract the rank-1 subspace of parameter differences at multiple training steps, which is then used for the subsequent nonlinear extrapolation. Afterward, we utilized the extracted rank-1 subspace to train a predictor, which can model the trajectory of parameter updates during RLVR, and then perform the predict-extend process to extrapolate model parameters, achieving the acceleration of RLVR. To further study and understand NExt, we conduct comprehensive experiments that demonstrate the effectiveness and robustness of the method. Our method reduces computational overhead by approximately 37.5\% while remaining compatible with a wide range of RLVR algorithms and tasks. We release our code in https://github.com/RUCAIBox/NExt.
1 INTRODUCTION
RLVR improves LLM reasoning but its extensive exploration and iterative updates create a major computational bottleneck. NExt addresses this by modeling nonlinear low-rank trajectories to predict future parameters and reduce intermediate training steps.
- RLVR improves LLM reasoning through extensive exploration and learning, but this process creates prohibitive computational costs as model scale and reasoning complexity grow.
- Prior acceleration methods improve sampling or rewards while leaving RLVR’s iterative training process fundamentally unchanged.
- Rank-1 subspaces become increasingly dominant during RLVR, especially with LoRA fine-tuning, but their evolution is nonlinear rather than strictly linear.
- NExt models and extrapolates low-rank parameter optimization trajectories nonlinearly to predict future model states.
- 37.5% lower time cost is achieved while NExt maintains or improves performance across models and remains robust to hyperparameters, tasks, and RLVR algorithms.
2 RELATED WORK
Related work frames RLVR acceleration as a response to substantial rollout and reward-feedback costs. Existing approaches improve data, exploration, rewards, or parameter extrapolation, including linear methods for predicting later checkpoints.
- RLVR acceleration is motivated by substantial training time and computational demands from iterative rollout and reward-feedback loops.
- Researchers accelerate RLVR by filtering data, improving exploration strategies, and designing more effective reward functions.
- Model parameter extrapolation leverages patterns across training stages, model scales, or checkpoints to improve capability and training efficiency.
- Prior work uses linear extrapolation of parameter evolution to predict later-stage parameters and reduce training time.
3 EFFECTIVENESS AND DYNAMICS OF LLM RANK-1 SUBSPACE DURING RLVR
This section introduces RLVR, low-rank representation, and the rank-1 subspace, then shows that its dominance grows during training while parameter updates are not uniformly linear.
- Preliminaries: RLVR optimizes a policy from sampled solutions and verifier rewards for question–answer pairs.GRPO estimates token-level advantages by normalizing rewards within each response group.
- Preliminaries: SVD represents parameter matrices through singular values and vectors, with the largest singular component defining the most influential rank-1 subspace.The rank-1 subspace is the component associated with the largest singular value.
- Rank-1 subspace dominance: The rank-1 energy ratio measures the largest singular value divided by the sum of singular values in a parameter update matrix.The analysis compares this quantity under Full FT and LoRA fine-tuning.
- Rank-1 subspace dominance: During early RLVR training, the rank-1 energy ratio increases, and LoRA produces stronger rank-1 dominance than full-parameter fine-tuning.This increasing dominance supports using the rank-1 subspace to approximate parameter updates.
- Nonlinearity of parameter updates: More than 50% of parameter updates have R2 < 0 when predicting the next five checkpoints from the first ten, showing that many updates are not well modeled linearly.Some parameters have predicted-versus-true R2 below −0.5, indicating that linear extrapolation can degrade in later training stages.
4 EXTRAPOLATION WITH LOW-RANK PARAMETER OPTIMIZATION TRAJECTORIES
NExt accelerates RLVR by extracting low-rank representations from LoRA-based optimization trajectories and nonlinearly predicting future parameter updates. It combines rank-1 approximation, trajectory prediction, and a predict-extend procedure to reduce extrapolation cost while modeling nonlinear training dynamics.
- Optimization Trajectory Collection: NExt first performs LoRA-based RLVR training and collects intermediate checkpoints and their parameter-update differences.These differences include global, local, and target deltas used to construct trajectory-prediction instances.
- LLM RLVR via LoRA Fine-tuning: LoRA fine-tuning keeps the original matrices fixed while optimizing low-rank adapters, which are later merged back into the model parameters.The adapter update is represented as BiAix within the matrix transformation.
- Low-rank Approximation: Rank-1 SVD approximations reduce each delta from O(n × m) parameters to O(n + m), lowering storage and predictor-computation overhead.The approximation retains the largest singular value and its corresponding singular vectors.
- Constructing Trajectory Predictor: The trajectory predictor takes global and local low-rank representations as inputs and predicts the target representation using an encoder–decoder architecture.Its training objective minimizes prediction discrepancy with an L1 norm.
- Predict-Extend Paradigm: NExt predicts a future target delta from the latest global and local deltas, scales it with an extending coefficient, and adds it to the current parameters.This predict-extend process directly models nonlinear optimization trajectories instead of using predefined linear combinations.
- Summarization and Discussion: NExt bridges low-rank modeling with nonlinear trajectory prediction to reduce RLVR training steps while targeting both efficiency and model performance.The framework is designed specifically for RLVR rather than solely for improving training effectiveness.
5 EXPERIMENT
This section introduces the experimental evaluation and its subsequent analysis of NExt.
- The experiments report main results first, followed by detailed analyses of NExt.
5.1 EXPERIMENTAL SETTINGS
The experiments use mathematical reasoning data, multiple baselines, GRPO as the backbone algorithm, and repeated accuracy and cost evaluation.
- Datasets: RLVR training uses approximately 17k mathematical reasoning problems, while evaluation covers five tasks spanning multiple difficulty levels.
- Baselines: Baselines include GRPO with full-parameter or LoRA fine-tuning, plus AlphaRL and RL-Extra acceleration methods.
- Implementation Details: GRPO serves as the backbone RLVR algorithm, with α=1.5 and 15 checkpoints collected during the first 150 training steps for NExt extrapolation.
- Evaluation Metrics: Evaluation reports training steps, average accuracy across tasks, and ICER; each task is repeated eight times for stability.
5.2 MAIN RESULTS
Across models and mathematical tasks, NExt reaches higher performance with fewer training steps than vanilla fine-tuning and linear-extrapolation baselines.
- NExt is evaluated on four model scales and five challenging mathematical tasks using the reported accuracy tables.
- 250 additional training steps after 150-step training let NExt surpass vanilla RLVR trained for 400 steps across all four model scales.
- NExt achieves higher accuracy and lower ICER than AlphaRL and RL-Extra at the same number of training steps.
- LoRA and full-parameter fine-tuning yield comparable performance, including 21.1% versus 20.8% for 1.5B and 24.0% versus 23.5% for 7B models.
5.3 DETAILED ANALYSIS
Detailed analyses examine NExt’s components, resource use, hyperparameter sensitivity, algorithm compatibility, and transfer across domains.
- Ablation Study: Ablation experiments compare extrapolation with and without LoRA, global delta, and local delta components.
- Extending Process: Model performance remains relatively stable for α values from 0.5 to 2.5, while overly large coefficients cause significant fluctuations.
- Adaptation of Other RLVR Algorithms: NExt applies to RLOO and REINFORCE++, reducing training steps by 37.5% while achieving comparable or better performance than vanilla RLVR.
- Adaptation to Other Domain Tasks: On MMLU-Pro and GPQA Diamond, NExt transfers across domains and reaches comparable performance with 250 rather than 400 RLVR steps.
6 CONCLUSION
The paper investigates RLVR parameter dynamics and proposes NExt, which extrapolates low-rank trajectories to improve performance with low computational overhead. Experiments demonstrate robustness and generalization, while further investigation of internal parameter updates remains future work.
- NExt uses LoRA-based RLVR checkpoints and parameter differences to train a trajectory predictor for extrapolating future model parameters.
- NExt achieves better LLM performance within a small number of training steps and with low computational overhead.
- Additional experiments show that NExt has strong robustness and generalization ability.
- Future work will investigate internal parameter-update patterns to support further computational-overhead reduction and test-time scaling.