You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

Zhepei Wei; Xinyu Zhu; Wei-Lin Chen; Chengsong Huang; Jiaxin Huang; Yu Meng

Source-linked AI summary

You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories

Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Chengsong Huang, Jiaxin Huang, Yu Meng

arXiv:2605.21468v1May 20, 2026cs.LGcs.CL

TL;DR

RLVR training is costly, and it is unclear whether future checkpoints can be predicted from early update dynamics. RELEX estimates a rank-1 trajectory subspace and linearly extrapolates its coefficients, matching or improving RLVR performance with 15–20% of training cost across three models.

Problem
RLVR training is computationally expensive, motivating the question of whether future checkpoints can be predicted from a short prefix of training dynamics.
Method
RELEX estimates a rank-1 update subspace with SVD and linearly extrapolates projected coefficients to predict future RLVR checkpoints without a learned model.
Results
With 15–20% of training cost, RELEX matches or exceeds RLVR on in-domain MATH and matches or improves out-of-distribution accuracy across three Qwen models.
Takeaways & Limitations
RLVR weight updates follow low-rank, near-linear trajectories, enabling substantial checkpoint extrapolation from limited observed training.
Takeaways & Limitations
The study covers GRPO-based mathematical reasoning on three Qwen-family models, so generalization to other algorithms, tasks, and model families remains open.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving reasoning in large language models (LLMs), yet the underlying geometry of the resulting parameter trajectories remains underexplored. In this work, we demonstrate that RLVR weight trajectories are extremely low-rank and highly predictable. Specifically, we find that the majority of downstream performance gains are captured by a rank-1 approximation of the parameter deltas, where the magnitude of this projection evolves near-linearly with training steps. Motivated by this, we propose a simple and compute-efficient method RELEX (REinforcement Learning EXtrapolation), which estimates the rank-1 subspace from a short observation window and extrapolates future checkpoints via linear regression, with no learned model required. Across three models (i.e., Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base), RELEX produces checkpoints that match or exceed RLVR performance on both in-domain and out-of-domain benchmarks, requiring as few as 15% steps of full RLVR training. Remarkably, RELEX is able to extrapolate far beyond the observation window at no training cost, predicting checkpoints up to 10-20$\times$ beyond the observed prefix with continued improvement (e.g., observe only the first 50 steps and extrapolate to 1000 steps). Our ablation analysis confirms the minimalist sufficiency of RELEX: neither increasing the subspace rank nor employing non-linear modeling yields further gains in extrapolation. Finally, we show that RELEX's success stems from a "denoising" effect: by projecting updates onto the rank-1 subspace, the model discards stochastic optimization noise that would otherwise degrade performance during extrapolation. Our code is available at https://github.com/weizhepei/RELEX.

1 Introduction

RLVR training is computationally expensive, yet its weight-update trajectories are highly structured: a rank-1 direction captures most downstream-relevant change and evolves near-linearly. RELEX exploits this structure to extrapolate future checkpoints without further training, achieving strong performance with as few as 15% of full RLVR cost.

RLVR typically requires massive optimization runs that produce progressively improving checkpoints, making the process computationally expensive.
Rank-1 SVD captures the dominant RLVR update direction, with reconstructed checkpoints closely matching RLVR checkpoints across training steps.The rank-1 reconstruction preserves most downstream performance on MATH across models.
RELEX estimates the rank-1 subspace from the first Tcut steps, fits a line to projected coefficients, and linearly extrapolates future checkpoints without a learned model.Once the subspace is estimated, predicting any future checkpoint is training-free.
15% of training cost is sufficient for RELEX to match and often exceed full RLVR on in-domain and out-of-domain math benchmarks across three backbone models.The method matches or exceeds full RLVR performance while using only a short observed training prefix.
Increasing the subspace rank or using non-linear modeling yields no further extrapolation gains, supporting RELEX’s minimalist design.

2 Background

RLVR trains language-model policies with programmatically verifiable rewards, using GRPO to sample, score, and update responses under a clipped objective with KL regularization. The paper analyzes checkpoint parameter deltas through SVD, separating parameter-change directions from their temporal coefficients.

RLVR and GRPO: RLVR maximizes programmatically verifiable rewards, and this work uses GRPO to sample multiple responses, score them with a verifier, and update the policy.GRPO uses a token-level clipped objective with a KL penalty toward a reference policy.
Trajectory representation: Checkpoint deltas Δθ_t = θ_t − θ_0 are flattened and stacked into a trajectory matrix M, with each row representing one training step.The construction applies to each parameter tensor, such as an attention weight matrix.
Subspace decomposition: SVD decomposes the trajectory into a parameter-change subspace and temporal coefficients, while rank-r truncation retains the dominant directions and dynamics.This factorization enables independent analysis and prediction of where parameters move and when or how much they move.

3 Method

Section 3 shows that RLVR parameter updates are effectively rank-1, while their dominant coefficients evolve nearly linearly, enabling RELEX to extrapolate future checkpoints from a short observed prefix using closed-form computations.

Trajectory structure: Rank-1 reconstruction preserves nearly all task-relevant RLVR change across all three models, closely tracking the original trajectory and downstream MATH accuracy gains.A single component per tensor accounts for nearly all task-relevant change despite the tensors’ high dimensionality.
Trajectory structure: R2 > 0.98 across most tensors, as rank-1 coefficients closely follow a linear function of training step.The coefficient trajectory is obtained by projecting observed deltas onto v1 and fitting c(t) = at + b via least squares.
RELEX method: RELEX estimates each tensor’s dominant rank-1 direction from the observed prefix, extrapolates its scalar coefficient linearly, and reconstructs the target checkpoint.The predicted coefficient is c_hat_T = aT + b, with the predicted delta added back to the base weights.
RELEX method: One truncated SVD per tensor plus a two-parameter least-squares fit gives RELEX zero additional RLVR training cost beyond the Tcut observation window.The method is closed-form, has no learnable parameters, and retains only the top singular vector.

4 Experiments

Across three models trained for 500 RLVR steps, RELEX matches or exceeds full RLVR and other extrapolation baselines using 15–20% of the training cost. Ablations attribute its performance to rank-1 SVD projection and linear modeling, which enable stable extrapolation far beyond the observed window.

RLVR training and evaluation: Three models are trained with GRPO on MATH for 500 steps, with evaluation on in-domain MATH and five out-of-distribution benchmarks.The models are Qwen2.5-Math-1.5B, Qwen3-4B-Base, and Qwen3-8B-Base.
Main results: 71.6% vs. 71.5% and 85.6% vs. 85.5% show RELEX matching or slightly exceeding RLVR on MATH for the two smaller models, while remaining within 1.1% on Qwen3-8B-Base.On Qwen3-8B-Base, RELEX reaches 87.4% versus 88.5% for RLVR.
Baseline comparison: 15–20% of RLVR training cost is sufficient for RELEX to uniformly outperform the other extrapolation methods on MATH.For Qwen2.5-Math-1.5B, RELEX beats Weight Extrapolation by +1.2 points, Logits Extrapolation by +6.7 points, ExPO by +3.9 points, and AlphaRL by +4.3 points.
Ablation studies: 81.4% of the representative rank-5 subspace variance is explained by the leading component, whose coefficient evolves near-linearly, while components 2–5 are noisier.Switching from SVD space to raw weight space lowers accuracy at every reported step, and adding components beyond rank 1 does not help.
Ablation studies: Step 200 reaches 69.5% for the neural fit versus 70.0% for linear extrapolation, while polynomial extrapolation collapses outside the observation window.The neural network offers no consistent intermediate-horizon advantage and incurs additional complexity.
Long-horizon stability: Step 1000 reaches 71.6% for Qwen2.5-Math-1.5B with Tcut=125, exceeding the 71.5% RLVR step-500 reference after extrapolating roughly 8× the observation window.Under a well-chosen observation window, RELEX remains close to peak accuracy across all three models as far as step 1000.

5 Related Work

Prior work studies RLVR dynamics through principal directions, update direction, partial parameter adaptation, implicit curricula, and linear extrapolation. Related research also explores low-rank weight updates and static weight-space combination, whereas this work extrapolates along a single evolving training trajectory.

Structure of RLVR training dynamics: Recent RLVR studies analyze principal components, update direction, partial parameter adaptation, and implicit curricula induced by mixed-difficulty data.These works provide complementary views of RLVR training geometry and optimization dynamics.
Structure of RLVR training dynamics: Wang et al. [2026] observe linear evolution of both weights and logits during RLVR and propose Weight Extrapolation and Logits Extrapolation to reduce training cost.The present work shares this core linearity observation but differs in its extrapolation approach.
Low-rank structure and weight-space modeling: Low-rank update structure appears in supervised fine-tuning, LoRA, classical deep RL, and RL-induced updates, including rank-1 linear dynamics exploited by AlphaRL.These studies establish precedent for modeling learning through a small number of parameter directions.
Model merging and scaling laws: Task arithmetic, model soups, and LoraHub exploit linear weight-space structure for model combination or cross-task generalization.Unlike these methods, which operate on static endpoints, this work extrapolates along one model’s evolving training trajectory.

6 Discussion

The discussion identifies model-dependent sensitivity in observation-window and subspace selection, with early extrapolations sometimes unstable at long horizons. It also limits the conclusions to GRPO-based mathematical reasoning on three Qwen models and flags broader generalization of rank-1 designs as open.

Observation-window sensitivity: Observation-window sensitivity is model-dependent: Qwen2.5-Math-1.5B benefits from longer prefixes, whereas Qwen3 models can favor small or intermediate windows.Early-window extrapolations may become unstable at long horizons.
Subspace selection is critical: Subspace selection is critical: Qwen3-4B-Base favors early subspaces such as Tcut = 75, while Qwen3-8B-Base favors intermediate-to-large windows Tcut ∈{100, 125}.Larger cutoffs degrade Qwen3-4B-Base, whereas small windows underperform on Qwen3-8B-Base.
Limitations: The study focuses on RLVR with GRPO for mathematical reasoning across three Qwen-family models, leaving other algorithms, tasks, and model families untested.The authors identify adaptive rank selection as a future direction because rank-1 may not generalize universally.

7 Conclusion

RLVR weight updates follow low-rank, predictable trajectories, with parameter deltas concentrated in per-tensor rank-1 subspaces and scalar coefficients evolving near-linearly. RELEX exploits this structure through rank-1 SVD and linear regression to extrapolate future checkpoints without a learned model.

Conclusion: Parameter deltas concentrate in a rank-1 subspace per tensor, while their scalar coefficient evolves near-linearly with training step.This geometric regularity makes RLVR weight trajectories low-rank and predictable.
Conclusion: RELEX extrapolates future checkpoints using rank-1 SVD followed by linear regression, with no learned model required.The method is parameter-free and directly leverages the observed trajectory geometry.
Conclusion: 15–20% of training observed is sufficient for RELEX to match RLVR on in-domain MATH for Qwen2.5-Math-1.5.The passage reports this result as evidence that only a short observation window is needed.

A Implementation Details

The study uses standardized GRPO-based RLVR training across three models, with model-specific decoding settings for evaluation. Efficient trajectory analysis relies on compact Gram-matrix SVDs and cached singular directions, while baselines follow specified checkpoint or logit extrapolation procedures.

RLVR training details: All three models train for 500 optimization steps on 8xH200 GPUs using verl, GRPO, and the MATH training split.Unless otherwise stated, training uses AdamW with learning rate 10−6, KL coefficient 0.001, group size G = 8, and mini-batch size 256.
Inference details: Inference uses model-specific in-domain decoding, while OOD benchmarks use avg@8 sampling at temperature 0.7.Qwen2.5-Math-1.5B uses greedy decoding with a 4K-token budget; Qwen3-family models use sampling with a 16K-token budget.
Trajectory analysis: Trajectory SVDs are computed through a T × T Gram matrix, with cached per-tensor singular directions and coefficient trajectories avoiding recomputation across cutoff sweeps.The compact approach exploits T ≪dℓ and recovers right singular directions as needed.
Comparison baselines: Weight and Logits Extrapolation use two-endpoint linear extrapolation, ExPO amplifies partial-training displacement, and AlphaRL estimates rank-1 SVDs independently at observed checkpoints.These methods serve as comparison baselines for RELEX under their respective extrapolation procedures.

B Weight-Space Alignment Analysis

The analysis compares rank-1 reconstruction and extrapolation against the true RLVR trajectory in raw weight space. Reconstruction closely recovers the dominant direction and update magnitude within the observed prefix, whereas extrapolation increasingly drifts with horizon.

Extrapolation: Extrapolated checkpoints drift in both direction and magnitude as the prediction horizon grows.Extrapolation fits only the first 75 steps and predicts future checkpoints without observing WRLVR(t).
Alignment analysis: The study evaluates weight-space alignment using mean per-tensor direction similarity and magnitude ratio against the true RLVR trajectory.Reconstruction uses Tcut = 500 and fits SVD on the full trajectory; extrapolation uses Tcut = 75 and predicts future checkpoints without seeing RLVR weights.
Reconstruction: Reconstruction’s rank-1 projection recovers the dominant RLVR direction with cosine similarity increasing from 0.50 to 0.91 and peaking near step 400.The lower similarity at step 100 is attributed to noisier early RLVR updates.
Reconstruction: Under reconstruction, the magnitude ratio remains close to 1.0, indicating that rank-1 projection faithfully captures the trajectory within the observed prefix.The evaluated checkpoint steps are t ∈{100, 200, 300, 400, 500}.