PaperlayerView exact revision on arXiv

Source-linked AI summary

The Path Not Taken: RLVR Provably Learns Off the Principals

Hanqing Zhu, Zhenyu Zhang, Hanxian Huang, DiJia Su, Zechun Liu, Jiawei Zhao, Igor Fedorov, Hamed Pirsiavash, Zhizhou Sha, Jinwon Lee, David Z. Pan, Zhangyang Wang, Yuandong Tian, Kai Sheng Tai

arXiv:2511.08567v1cs.LGcs.AI

TL;DR

The paper addresses why RLVR can improve reasoning while apparently changing few parameters. It explains this through a Three-Gate Theory and parameter-level validation, finding that RLVR learns off principal directions with minimal spectral disruption, unlike SFT, and that SFT-era PEFT priors can be misaligned.

  • Problem

    RLVR improves reasoning despite apparently sparse parameter updates, while the mechanisms shaping its parameter evolution remain poorly understood.

  • Method

    The paper proposes a Three-Gate Theory and validates it by comparing RLVR and SFT parameter dynamics, including principal-weight overlap, spectral structure, and geometry interventions.

  • Results

    RLVR learns off principal directions while preserving spectral structure and avoiding principal weights, whereas SFT targets principal directions and distorts the spectrum.

  • Takeaways & Limitations

    SFT-era principal-aligned sparse and low-rank PEFT methods can be misaligned with RLVR, motivating geometry-aware, RLVR-native algorithms.

  • Takeaways & Limitations

    Precision amplifies visible sparsity but does not cause the optimization bias, because optimizer states and gradient accumulation remain in float32.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) reliably improves the reasoning performance of large language models, yet it appears to modify only a small fraction of parameters. We revisit this paradox and show that sparsity is a surface artifact of a model-conditioned optimization bias: for a fixed pretrained model, updates consistently localize to preferred parameter regions, highly consistent across runs and largely invariant to datasets and RL recipes. We mechanistically explain these dynamics with a Three-Gate Theory: Gate I (KL Anchor) imposes a KL-constrained update; Gate II (Model Geometry) steers the step off principal directions into low-curvature, spectrum-preserving subspaces; and Gate III (Precision) hides micro-updates in non-preferred regions, making the off-principal bias appear as sparsity. We then validate this theory and, for the first time, provide a parameter-level characterization of RLVR's learning dynamics: RLVR learns off principal directions in weight space, achieving gains via minimal spectral drift, reduced principal-subspace rotation, and off-principal update alignment. In contrast, SFT targets principal weights, distorts the spectrum, and even lags RLVR. Together, these results provide the first parameter-space account of RLVR's training dynamics, revealing clear regularities in how parameters evolve. Crucially, we show that RL operates in a distinct optimization regime from SFT, so directly adapting SFT-era parameter-efficient fine-tuning (PEFT) methods can be flawed, as evidenced by our case studies on advanced sparse fine-tuning and LoRA variants. We hope this work charts a path toward a white-box understanding of RLVR and the design of geometry-aware, RLVR-native learning algorithms, rather than repurposed SFT-era heuristics.

1 Introduction

RLVR’s apparent parameter sparsity reflects a stable, model-conditioned optimization bias rather than uniformly small learning. The paper explains and validates an off-principal, low-curvature regime distinct from SFT, with implications for RLVR-native parameter-efficient tuning.

  • 1 Introduction: RLVR updates consistently localize to a stable subset of parameters across datasets and RL variants, resolving the apparent sparsity paradox.The bias is conditioned on the fixed pretrained model, while finite precision amplifies its visible sparsity.
  • 1 Introduction: The Three-Gate Theory attributes this regime to KL-constrained updates, pretrained geometry steering toward low-curvature off-principal subspaces, and precision masking micro-updates.The gates respectively constrain, steer, and reveal the optimization pattern.
  • 1 Introduction: RLVR preserves spectral structure, avoids principal weights, and rotates principal subspaces less, whereas SFT targets principal directions and distorts them.The paper uses parameter-level evidence and geometry-disrupting interventions to validate this distinction.
  • 1 Introduction: SFT-era principal-aligned sparse and low-rank PEFT methods can be misaligned with RLVR, motivating geometry-aware RL-native alternatives.Principal-weight updates yield weak trajectories, PiSSA offers no obvious gain over standard LoRA, and aggressive principal-direction scaling can destabilize training.
  • 1 Introduction: The work frames RLVR as a distinct optimization regime and provides a parameter-space account linking reinforcement-learning dynamics to weight evolution.This complements prior work focused primarily on policy-level or distributional effects.

2 A Persistent, Model-Conditioned Optimization Bias in RLVR

RLVR updates exhibit a persistent, model-conditioned routing bias: changes localize to stable parameter regions across runs, datasets, and algorithms, while bfloat16 precision amplifies their apparent sparsity.

  • 2.1 A Robust, bfloat16-aware Analysis of Update Sparsity: 36%–92% RL sparsity versus 0.6%–18.8% SFT sparsity across models confirms that RL updates appear substantially sparser under the bfloat16-aware probe.Recent checkpoints show lower absolute sparsity than earlier reports, motivating precision-aware re-evaluation.
  • 2.2 RLVR Exhibits Model-Conditioned Update Locality: High cross-run Jaccard similarity confirms that independent RLVR runs from the same base model target a shared update footprint.The comparison uses five runs trained with different datasets and RLVR algorithms.
  • 2.2 RLVR Exhibits Model-Conditioned Update Locality: Consensus maps reveal contiguous row and column stripes in attention projections, showing localized rather than scattered routing of RLVR updates.Row-wise stripes are especially evident in Q/K/V, while O exhibits column-wise stripes.
  • 2.2 RLVR Exhibits Model-Conditioned Update Locality: The routing bias emerges early and remains reinforced through training, with persistent peaks and troughs indicating a structured, temporally stable phenomenon.The temporal profiles are tracked across checkpoints at t∈{240, 720, 1200}.
  • 2.2 RLVR Exhibits Model-Conditioned Update Locality: Similar stripe-structured footprints on Llama and Mistral suggest that the routing bias is generic to RLVR across model families.The paper reports this comparison in an appendix figure.
  • 2.3 Sparsity Is a Superficial Artifact of the Optimization Bias: bfloat16 precision hides sub-ULP changes in non-preferred regions, so observed sparsity is a readout of optimization bias rather than precision alone.The training pipeline keeps optimizer states and gradient accumulation in float32, so precision alone cannot explain the effect.

3 A Mechanistic Theory of RL’s Unique Optimization Dynamics

The Three-Gate Theory explains RLVR’s parameter dynamics: a KL leash limits each update, pretrained geometry steers it toward low-curvature spectrum-preserving directions, and precision amplifies visible sparsity.

  • 3.1 Gate I: On-Policy RL Imposes a One-Step KL Leash: RL updates are constrained by an implicit per-step KL leash that keeps the new policy close to its current policy.This bound limits parameter movement without specifying the update’s location.
  • 3.2 Gate II: Model Geometry Determines Where a KL-Bounded Step Goes: The pretrained model’s structured geometry steers KL-constrained updates toward low-curvature directions rather than principal directions.The theory attributes this location preference to the pretrained optimization landscape.
  • 3.2 Gate II: Model Geometry Determines Where a KL-Bounded Step Goes: RL’s conservative updates preserve singular values and limit rotation of the model’s principal subspaces.Theoretical results bound subspace rotation and singular-value changes under the KL-constrained update.
  • 3.2 Gate II: Model Geometry Determines Where a KL-Bounded Step Goes: Figure 4 shows RLVR maintaining a stable top-k spectrum with minimal subspace rotation, unlike SFT.The comparison uses principal angles, singular-value curves, maximum principal angle, and normalized spectral drift.
  • 3.3 Gate III: Precision Acts as a Lens Revealing the Compass: Limited bfloat16 precision hides sufficiently small updates in some regions, amplifying visible sparsity without causing the underlying optimization bias.The precision gate is described as a visibility amplifier, while optimizer states remain in float32.

4 Theory-Guided Validation of RLVR’s Optimization Dynamics

Theory-guided experiments show that RLVR updates remain off principal directions, preserve pretrained spectral geometry, and depend causally on that geometry across models and tasks.

  • 4.1 Spectral Geometry: RLVR checkpoints show minimal spectral drift and principal-subspace rotation, while SFT produces substantially larger changes on the same metrics.The evaluation compares singular-value profiles and top-k subspace rotations between base and fine-tuned models.
  • 4.2 Update–Principal Misalignment: Principal weights serve as a computationally practical proxy for high-curvature directions because low-rank reconstructions identify influential pathways.Perturbation studies link changes to these weights with sharp reasoning-performance degradation.
  • 4.2 Update–Principal Misalignment: RL updates overlap sub-randomly with principal weights but super-randomly with low-magnitude weights.Removing weights that are both principal and low-magnitude further reduces the residual principal-weight overlap.
  • 4.3 RLVR Relies on Model Geometry, Disrupting Geometry Destroys the Bias: Intervening on pretrained geometry collapses update overlap to random levels in affected layers while leaving untouched layers unchanged.The interventions use orthogonal rotations and head permutations, providing causal evidence that geometry generates the optimization bias.
  • 4.4 Beyond RLVR: Agents and RLHF: The same spectrum-preserving, off-principal pattern extends from verifiable math and code to agent tasks and RLHF.Agent and RLHF checkpoints show stable spectra, minimal rotation, and consistent avoidance of principal weights.

5 Theory-Guided Rethinking of Learning Algorithms for RL

Because RLVR and SFT occupy distinct parameter-space regimes, SFT-oriented PEFT methods can transfer poorly; geometry-aware masks and off-principal adapters better match RL dynamics.

  • 5.1 Probing Sparse Fine-Tuning in RL: The safe sparse mask tracks dense RLVR’s KL trajectory and reaches comparable final accuracy, whereas the principal-only mask has the worst optimization trajectory.The safe mask favors non-principal and low-magnitude weights; the principal-only mask yields slowly rising KL and degraded training dynamics.
  • 5.1 Probing Sparse Fine-Tuning in RL: Updating non-principal, low-magnitude weights reproduces dense RLVR behavior using roughly 70% of the parameters.The fixed mask is constructed directly from the pretrained model without additional training.
  • 5.2 Revisiting LoRA Through the Lens of Our Theory: Across tested ranks and learning rates, principal-targeted PiSSA provides no clear gain over LoRA and often collapses earlier at higher learning rates.The experiments report pass@1 on AIME24, AMC23, and MATH500; LoRA remains more stable.
  • 5.2 Revisiting LoRA Through the Lens of Our Theory: Principal-aligned adapters impose SFT-style update geometry that conflicts with RLVR’s off-principal dynamics, explaining their brittle optimization.The paper attributes early PiSSA collapse to higher-curvature, spectrum-distorting principal-direction updates.
  • 5.3 Implications for RL Learning Algorithms: The results motivate RL-native, geometry-aware PEFT methods rather than direct reuse of SFT-era sparse or low-rank heuristics.The paper presents the mismatch between SFT geometry and RL dynamics as a design constraint for future methods.

6 Conclusion

The paper explains RLVR’s apparent update sparsity as a model-conditioned, geometry-aligned optimization bias and distinguishes RLVR’s parameter dynamics from SFT. Its conclusion motivates RLVR-native, geometry-aware parameter-efficient learning algorithms.

  • 6 Conclusion: RLVR’s visible updates localize to stable model-specific parameter regions across runs and datasets, with bfloat16 amplifying the apparent sparsity.This resolves the sparsity paradox by distinguishing update visibility from the underlying optimization bias.
  • 6 Conclusion: The Three-Gate Theory explains apparent sparsity through KL-constrained updates, off-principal low-curvature movement, and finite-precision masking of micro-updates.The theory links optimization constraints, pretrained geometry, and bfloat16 visibility into one mechanism.
  • 6 Conclusion: RLVR preserves spectral structure and avoids principal weights, whereas SFT targets principal directions and distorts the spectrum.The conclusion frames these as contrasting parameter-space signatures of the two training regimes.
  • 6 Conclusion: SFT-era principal-aligned sparse and low-rank PEFT methods can misalign with RLVR’s off-principal regime.The paper uses sparse and low-rank variants as case studies of this transfer problem.
  • 6 Conclusion: The work provides a parameter-level account of RLVR training dynamics and motivates geometry-aware, RLVR-native algorithms.It positions this account as a step toward white-box understanding of how parameters evolve under RLVR.

C.1 Training Settings

The experiments use three open language models, verifiable-reward RLVR training, two mathematics training corpora, and specified distributed-training configurations. LoRA and PiSSA studies use a reduced-cost DAPO setup.

  • C.1 Training Settings: Experiments use DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-Math-7B, and Qwen3-4B-Base with model-specific context lengths.The maximum context lengths are 8192 for the first two models and 20480 for Qwen3-4B-Base.
  • C.1 Training Settings: The study uses DAPO+MATH and DS+SR to reduce dataset-specific confounds in mathematics evaluation.DS+SR combines DeepScaler with high-difficulty SimpleRL problems, while DAPO+MATH combines DAPO-Math-17k with MATH.
  • C.1 Training Settings: Training uses DAPO on VeRL with vLLM rollouts, FSDPv2 mixed precision, a global batch size of 256, and four gradient updates per step.Unless otherwise noted, experiments omit an explicit reference-KL penalty and use ratio clipping as in DAPO.
  • C.1 Training Settings: LoRA and PiSSA sweeps use global batch size 128, mini-batch 32, four gradient updates per step, and 200 training steps.Both DeepSeek-R1-Distill-Qwen-1.5B and LLaMA-3.2-3B use this reduced-cost configuration.
  • C.1 Training Settings: RLVR training uses verifiable rewards of +1.0 for correct extracted answers and −1.0 otherwise, with an over-length penalty.The penalty adds a 1024-token budget and uses factor 1.0.

C.2 Evaluation settings

The models are evaluated on four mathematics benchmarks using averaged accuracy, reported as pass@1(avg@k) under fixed sampling settings.

  • C.2 Evaluation settings: Evaluation covers AIME24, AIME25, AMC23, and MATH-500 using averaged accuracy, pass@1(avg@k).AIME24 and AIME25 average over 64 samples; Eval-Chemy uses temperature 0.7 and top-p 0.8.

D Intervention details

The interventions alter attention-head coordinates or labels while preserving the model’s function, enabling tests of whether RL update locations depend on pretrained geometry. The analysis reports overlap and post-intervention update-position behavior.

  • D Intervention details: Loss-preserving V/O rotations apply orthogonal transformations along attention-head dimensions while maintaining the attention computation.The intervention is formalized with an exact-invariance proposition.
  • D Intervention details: Head shuffling permutes key/value heads and expands the permutation across query heads while inversely permuting output columns.The construction relabels which head carries each subspace without changing the block function.
  • D Intervention details: After weight intervention, the model’s update-position overlap becomes sub-random, while untouched weights retain high overlap.These observations distinguish changed parameter coordinates from preserved model function.

E Examples of why previous identified method fails

The section formalizes why fixed absolute-tolerance probes misclassify bf16 updates and develops a scale-aware alternative grounded in ULP spacing. It also outlines the RLVR objective and supporting local analytical assumptions.

  • E.1 Failures of a Fixed Absolute Tolerance Rule: A fixed 10^-5 tolerance also produces false positives at large scales, flagging values that differ numerically but round to the same bf16 code.The example 1024.001 versus 1024.002 differs by 10^-3 but has no storage-level change.
  • E.1 Failures of a Fixed Absolute Tolerance Rule: A fixed 10^-5 absolute tolerance produces false negatives at small scales, where distinct bf16 values can differ by many ULPs yet be marked equal.At approximately 10^-6, values separated by 10^-6 are about 134 ULPs apart and quantize to different bf16 codes.
  • E.2 Justification of our probe: 0.195%–0.391% is the minimal realized relative update range for normalized bf16 values, with larger magnitudes requiring larger absolute steps to register.The threshold follows the magnitude-dependent bf16 spacing and motivates a scale-aware probe.
  • E.2 Justification of our probe: The scale-aware test uses bf16 representability gaps to make equality equivalent to unchanged stored values under a sufficiently small threshold.The stated corollary identifies η = 10^-3 as safe for bitwise-identical bf16 entries or both zeros.
  • Optimization objective and assumptions: The DAPO setup samples candidate completions under an old policy and uses importance-weighted advantages with trust-region clipping in its token-level objective.The analysis assumes a fixed input, full-support reference distribution, and locally smooth C3 policy family.
  • Optimization objective and assumptions: The appendix establishes local second-order identities linking frozen-policy surrogates, M-projections, Hessian–Fisher proximity, and KL-based update analysis.The frozen-policy surrogate matches the original objective and gradient at the current policy, with a second-order remainder under Lipschitz gradients.

F.2.2 Proof of Proposition 3.2

The proof derives local Fisher-norm and Euclidean bounds for policy updates under a KL budget, then extends the analysis to layer-conditioned parameter changes and spectral stability. It also connects clipped ratios to an implicit KL leash.

  • F.2.2 Proof of Proposition 3.2: The proof bounds the update through the Fisher quadratic form, yielding a local relationship between policy KL and the Fisher norm.The derivation controls cubic remainders using local strong convexity and small-step assumptions.
  • F.2.2 Proof of Proposition 3.2: A sufficiently small KL budget gives a Fisher-norm bound proportional to 2K and an accompanying Euclidean bound, with asymptotically negligible remainder terms.The proof uses a quadratic policy-KL expansion and shows the cubic remainder is o(K).
  • F.2.2 Proof of Proposition 3.2: Ratio clipping constrains policy KL, tightening to O(ε^2) in the small-step, mean-zero-advantage regime.Autoregressive factorization converts per-token ratio bounds into a sequence-level KL bound.
  • F.2.2 Proof of Proposition 3.2: Layer-conditioned Fisher constraints bound the Frobenius change of a selected parameter block and support spectral perturbation analysis.The blockwise setup partitions parameters into W and its complement and imposes a quadratic budget on vec(ΔW).
  • F.2.2 Proof of Proposition 3.2: Singular-value perturbation bounds control principal-subspace rotation, while Weyl/Mirsky and Hoffman–Wielandt bounds control spectrum changes.The subspace guarantee depends on a positive singular-value gap γ_k.
  • F.2.2 Proof of Proposition 3.2: Across five independent RLVR runs, final update masks show high pair-wise Jaccard similarity, supporting a consistent model-conditioned optimization bias.The comparison spans different data and algorithms and targets the same parameter subset.

G.2 Spectrum shift for DS-1.5B and Qwen3-1

The spectrum experiments compare RLVR and SFT across DS-Distill-Qwen-1.5B and Qwen3-14B while examining update-mask structure and its temporal emergence. RLVR is characterized by stable leading spectra and limited principal-subspace rotation.

  • G.2 Spectrum shift for DS-1.5B and Qwen3-1: Pair-wise Jaccard similarity is used to compare update masks from five independent RLVR runs on Layer 13 of DS-Distill-Qwen-1.5B.The figure defines the run-level mask-consistency comparison used in the analysis.
  • G.2 Spectrum shift for DS-1.5B and Qwen3-1: Principal-only masking leaves a clear accuracy gap versus the dense baseline and the complementary mask, while principal-plus-low masks come closest to dense performance.The masking comparisons are reported at both 320 and 500 training steps.
  • G.2 Spectrum shift for DS-1.5B and Qwen3-1: Update-mask structure is examined across Llama and Mistral models using zero-RL checkpoints, extending the analysis beyond the primary Qwen experiments.The figure plots weight update masks for both model families.
  • G.2 Spectrum shift for DS-1.5B and Qwen3-1: The temporal analysis tracks row- and column-wise update ratios in the 13th attention block across three gradient-update steps.Column-wise Q and row-wise O ratios show a much weaker bias than the principal structured pattern.
  • G.2 Spectrum shift for DS-1.5B and Qwen3-1: The token-wise KL loss rises steadily during a DAPO run without an explicit KL penalty, rather than remaining unconstrained.This diagnostic accompanies the spectrum and update-mask analyses.
  • G.2 Spectrum shift for DS-1.5B and Qwen3-1: RLVR shows stable top-k spectra, minimal principal-subspace rotation, and small top-k eigenvalue changes on both DS-Distill-Qwen-1.5B and Qwen3-14B.The same spectral pattern is also reported under an RLHF setting, where layer spectra remain preserved across checkpoints.
Loading 2511.08567v1…