PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Parameter Efficient Methods for RLVR

Qingyu Yin, Yulun Wu, Zhennan Shen, Sunbowen Li, Zhilin Wang, Yanshu Li, Chak Tou Leong, Jiale Kang, Jinjin Gu

arXiv:2512.23165v2cs.LG

TL;DR

RLVR makes the choice of parameter-efficient method consequential, yet it remains unclear whether standard LoRA suits RL’s distinct optimization dynamics. This paper benchmarks more than 12 PEFT methods and finds structural variants outperform LoRA, while SVD-informed initialization and extreme compression show failure modes.

  • Problem

    It is unclear which PEFT method best suits RL’s distinct optimization dynamics, despite standard LoRA’s predominance in RL applications.

    Source: §1 · p. 2

  • Method

    The paper establishes a large-scale benchmark evaluating over 12 PEFT methods for RL.

    Source: §1 · p. 3

  • Results

    Structural variants consistently outperform LoRA, SVD-based initializations can collapse from spectral misalignment, and extreme compression bottlenecks reasoning capacity.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight magnitude-direction decoupling as important for handling complex policy shifts in RL.

    Source: §5 · p. 10

    The evaluation remains within the DeepSeek-R1-Distill family and current short-horizon schedules, with broader architectures and prolonged training left for future work.

    Source: §4 · p. 10

Abstract

from arXiv · show

We systematically evaluate Parameter-Efficient Fine-Tuning (PEFT) methods under the paradigm of Reinforcement Learning with Verifiable Rewards (RLVR). RLVR incentivizes language models to enhance their reasoning capabilities through verifiable feedback; however, while methods like LoRA are commonly used, the optimal PEFT architecture for RLVR remains unidentified. In this work, we conduct the first comprehensive evaluation of over 12 PEFT methodologies across the DeepSeek-R1-Distill families on mathematical reasoning benchmarks. Our empirical results challenge the default adoption of standard LoRA with three main findings. First, we demonstrate that structural variants, such as DoRA, AdaLoRA, and MiSS, consistently outperform LoRA. Second, we uncover a spectral collapse phenomenon in SVD-informed initialization strategies (\textit{e.g.,} PiSSA, MiLoRA), attributing their failure to a fundamental misalignment between principal-component updates and RL optimization. Furthermore, our ablations reveal that extreme parameter reduction (\textit{e.g.,} VeRA, Rank-1) severely bottlenecks reasoning capacity. We further conduct ablation studies and scaling experiments to validate our findings. This work provides a definitive guide for advocating for more exploration for parameter-efficient RL methods.

1. Introduction

Because RLVR uses sparse supervision, parameter-efficient methods may reduce redundant full-parameter training, but it is unclear whether standard LoRA suits RL’s optimization dynamics. The evaluation finds structural variants outperform LoRA, while SVD-informed initialization and extreme compression have distinct failure modes.

  • Standard LoRA’s prevalence in RL leaves open whether it best suits RL’s distinct optimization dynamics.The paper frames this as its primary research question amid a proliferation of PEFT variants.Source: §1 · p. 2
  • The study evaluates over 12 PEFT variants on DeepSeek-R1-Distill mathematical reasoning tasks under RLVR with unified hyperparameters.The benchmark includes MATH-500, AIME, and AMC, among other tasks.Source: §1 · pp. 2–3
  • Extreme compression methods, including VeRA and Rank-1 adapters, bottleneck the expressivity needed to sustain complex reasoning.Moderate parameter reduction, such as freezing half the weights in LoRA-FA, remains tolerable.Source: §1 · pp. 2–3
  • Structural variants retain their superiority at the 7B scale, and ablations across batch sizes, ranks, and learning rates support the findings’ robustness.The scaling experiment uses DeepSeek-R1-Distill-Qwen-7B.Source: §1 · p. 3
  • Structural variants consistently outperform standard LoRA, with DoRA notably surpassing full-parameter fine-tuning.The reported variants include DoRA, MiSS, and AdaLoRA.Source: §1 · pp. 2–3
  • SVD-informed initializations such as PiSSA fail because principal-component updates conflict with RLVR’s off-principal update dynamics.The paper attributes the failure to structural misalignment.Source: §1 · pp. 2–3

2. Preliminaries & Setup

This section defines the RLVR training framework and lays out the PEFT methods, models, data, training recipe, and mathematics evaluation used in the benchmark.

2.1. Reinforcement Learning with Verifiable Rewards (RLVR)

  • GRPO estimates advantages from groups of sampled responses, avoiding a separate critic model within the RLVR framework.RLVR uses deterministic verifiers to provide sparse binary rewards.Source: §2.1 · p. 3
  • DAPO is the standard training algorithm; its variants address exploration, reward-homogeneous prompts, and biases in advantage estimation.DAPO uses Clip-Higher and dynamic prompt sampling, while Dr. GRPO removes length normalization and group-level standard deviation.Source: §2.1 · pp. 3–4

2.2. PEFT Methods

  • The benchmark compares full fine-tuning and LoRA with structural, initialization-based, efficiency-oriented, and other PEFT mechanisms.The methods include DoRA, AdaLoRA, MiSS, PiSSA, MiLoRA, LoRA-FA, VeRA, IA3, and LayerNorm Tuning.Source: §2.2 · pp. 4–5

2.3. Models and Datasets

  • Experiments use the 1.5B and 7B DeepSeek-R1-Distill-Qwen models and the approximately 17.4k-query DAPO-Math-17k-Processed training dataset.The dataset uses structured reasoning traces and boxed final answers.Source: §2.3 · p. 5

2.4. Training Settings

  • Training uses binary outcome rewards and DAPO settings, with different batch sizes and step counts for the two model scales.The setup includes eight rollouts per prompt, a 1e-5 learning rate, and no KL coefficient.Source: §2.4 · pp. 5–6
  • Table 3 compares accuracy and pass scores, with all values reported as percentages.The supplied caption identifies the metrics and units but does not provide table values.Source: Table 3 · p. 6

2.5. Evaluation

  • Evaluation covers MATH-500, AMC23, AIME24/25, Minerva, and HMMT using benchmark-specific metrics.Table 2 summarizes the datasets, test-sample counts, and evaluation metrics.Source: §2.5 · p. 6, Table 2 · p. 6

3. Results and Analysis

Structural PEFT variants outperform standard LoRA in the reported RLVR results, while extreme parameter reduction and SVD-based initialization can impair reasoning performance. Ablations and 7B scaling tests examine how these findings vary with training settings and model scale.

3.1. Main Results

  • DoRA reaches 46.6% overall average accuracy, above full-parameter fine-tuning (44.9%) and standard LoRA (42.5%); AdaLoRA (44.2%) and MiSS (43.4%) also beat LoRA.The authors attribute the variants’ advantage to mitigation of standard LoRA’s optimization rigidity.Source: §3.1 · p. 7
  • VeRA reaches 40.7% accuracy and IA3 22.3%, while moderate reduction such as LoRA-FA remains competitive; vector-only updates are identified as an expressivity bottleneck.The authors argue that reasoning requires a minimum threshold of trainable adapter capacity.Source: §3.1 · p. 7, Table 4 · p. 7
  • PiSSA collapses to 0.2% accuracy and MiLoRA reaches 18.0%; both end up with principal-component updates despite RLVR’s off-principal update tendency.PiSSA imposes principal-subspace updates, whereas MiLoRA’s negligible initialization lets gradient flow reorient updates toward dominant principal components.Source: §3.1 · pp. 7–8

3.2. Ablation Studies

  • Across GRPO, DAPO, and Dr. GRPO, PEFT performance shows no statistically significant reasoning-accuracy deviation, suggesting algorithmic invariance across these RLVR objectives.The compared algorithms use different strategies for advantage estimation and regularization.Source: §3.2 · pp. 8–9
  • Learning-rate magnitude is decisive for RLVR stability, while rank 1 consistently underperforms ranks 16 and 32.The ablation varies LoRA ranks 1, 8, 16, and 32; the authors recommend moderate rather than extreme rank reduction.Source: §3.2 · p. 9
  • Batch size 32 yields average accuracy of 42.5%, but batch 128 outperforms it on AIME 2024, so the small-batch SFT heuristic transfers less strongly to RLVR.The reported comparison appears in Table 5 and Figure 5.Source: §3.2 · p. 8, Figure 5 · p. 9

3.3. Scaling on Stronger Models

  • At 7B, DoRA and LoRA+ each reach 55.0% overall average accuracy versus standard LoRA’s 54.8%; DoRA scores 83.1% on AMC and 38.7% on AIME25.The authors report that the relative performance hierarchy remains largely consistent at this larger scale.Source: §3.3 · pp. 9–10, Table 6 · p. 10

4. Future Work

The authors identify infrastructure, theoretical, and deployment questions for future PEFT-RL work. They plan to expand model and training coverage, investigate adapter dynamics, and test broader settings and practical stability issues.

Advanced Infrastructure and Scaling

  • The current TRL setup limits large-scale distributed training; planned work would broaden evaluation beyond DeepSeek-R1-Distill and short-horizon schedules, including R1-Zero-like cold starts and diverse architectures.The authors plan to migrate to a higher-performance framework such as VeRL.Source: §4 · p. 10

Mechanistic Interpretability of Adapter Dynamics

  • The precise mathematical reasons for structural variants’ alignment and SVD-based initialization collapse remain unresolved; planned work will examine spectral evolution and optimization landscapes.The stated aim is to move from empirical observations toward a grounded theory.Source: §4 · p. 10

Broader Frontiers and Deployment Stability

  • Future validation will extend to multimodal, multi-turn, and asynchronous RL, while addressing weight-merging stability and training–inference inconsistencies.The authors describe these engineering issues as relevant to robust real-world application.Source: §4 · p. 10

5. Conclusion

This work presents a first systematic, large-scale evaluation of PEFT methods for RLVR. Its findings challenge reliance on standard LoRA, identify failure modes in SVD-informed initialization and extreme parameter reduction, and favor geometry-aware adapters such as DoRA.

5. Conclusion

  • Structural variants consistently outperform standard LoRA and can surpass full-parameter fine-tuning, highlighting magnitude-direction decoupling for RL policy shifts.The study evaluates over 12 PEFT variants across multiple model scales.Source: §5 · p. 10
  • SVD-informed initializations such as PiSSA and MiLoRA misalign with RLVR’s off-principal updates, leading to training instability or collapse.The proposed mechanism is their focus on principal components, which conflicts with RLVR’s intrinsic off-principal update dynamics.Source: §5 · p. 10
  • Extreme parameter reduction, including VeRA and Rank-1 adapters, creates a structural bottleneck that limits reasoning plasticity.Source: §5 · p. 10
  • The authors advocate moving beyond default LoRA toward geometry-aware adapters such as DoRA, which balance efficiency and reasoning capability.Source: §5 · pp. 10–11

Related papers

Loading 2512.23165v2…