PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Reason under Off-Policy Guidance

Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, Yue Zhang

arXiv:2504.14945v5cs.LGcs.AIcs.CL

TL;DR

Existing RLVR methods are inherently on-policy, limiting models to reasoning behaviors available from their own outputs. LUFFY adds stronger-policy traces through mixed-policy GRPO and policy shaping, achieving broad benchmark gains, stronger out-of-distribution performance, and successful training of weaker models where on-policy RLVR fails.

  • Problem

    On-policy RLVR restricts learning to a model’s self-generated outputs, limiting acquisition of reasoning behaviors beyond its initial capabilities.

  • Method

    LUFFY combines stronger-policy reasoning traces with on-policy rollouts through Mixed-Policy GRPO and regularized-importance-sampling policy shaping.

  • Results

    +6.4 points average gain across six math benchmarks, over +6.2 points on out-of-distribution tasks, and successful training of weak models where on-policy RL fails.

  • Takeaways & Limitations

    The results support off-policy guidance as a promising path for more generalizable reasoning and for extending RLVR beyond on-policy capability limits.

  • Takeaways & Limitations

    Mixed-Policy training can reduce exploration through rapid entropy collapse by reinforcing high-probability off-policy tokens and ignoring low-probability actions.

Abstract

from arXiv · show

Recent advances in large reasoning models (LRMs) demonstrate that sophisticated behaviors such as multi-step reasoning and self-reflection can emerge via reinforcement learning with verifiable rewards~(\textit{RLVR}). However, existing \textit{RLVR} approaches are inherently ``on-policy'', limiting learning to a model's own outputs and failing to acquire reasoning abilities beyond its initial capabilities. To address this issue, we introduce \textbf{LUFFY} (\textbf{L}earning to reason \textbf{U}nder o\textbf{FF}-polic\textbf{Y} guidance), a framework that augments \textit{RLVR} with off-policy reasoning traces. LUFFY dynamically balances imitation and exploration by combining off-policy demonstrations with on-policy rollouts during training. Specifically, LUFFY combines the Mixed-Policy GRPO framework, which has a theoretically guaranteed convergence rate, alongside policy shaping via regularized importance sampling to avoid superficial and rigid imitation during mixed-policy training. Compared with previous RLVR methods, LUFFY achieves an over \textbf{+6.4} average gain across six math benchmarks and an advantage of over \textbf{+6.2} points in out-of-distribution tasks. Most significantly, we show that LUFFY successfully trains weak models in scenarios where on-policy RLVR completely fails. These results provide compelling evidence that LUFFY transcends the fundamental limitations of on-policy RLVR and demonstrates the great potential of utilizing off-policy guidance in RLVR.

1 Introduction

RLVR has enabled sophisticated reasoning behaviors but remains constrained by models’ self-generated outputs. LUFFY introduces stronger-policy guidance and mixed-policy training to expand reasoning beyond initial capabilities, with substantial benchmark gains and successful training of weaker models.

  • Motivation: On-policy RLVR limits learning to a model’s self-generated outputs, bounding reasoning improvement by the base LLM’s initial capabilities.This limitation motivates seeking external reasoning guidance.
  • Method: LUFFY introduces external guidance from stronger policies, such as DeepSeek-R1, to expose models to otherwise inaccessible reasoning patterns and cognitive structures.The framework combines off-policy reasoning traces with on-policy rollouts during advantage computation.
  • Method: LUFFY extends GRPO into Mixed-Policy GRPO with an off-policy objective and importance sampling to calibrate policy gradients.This design combines off-policy demonstrations with on-policy exploration during training.
  • Results: +6.4 points average gain across six math benchmarks compared with previous RLVR methods.The benchmarks include AIME24/25, AMC, OlympiadBench, Minerva, and MATH-500.
  • Results: +6.2 points average advantage on out-of-distribution tasks, while LUFFY successfully trains LLaMA3.1-8B where on-policy RL fails.These results support improved generalization and training beyond on-policy capability limits.

2 Reinforcement Learning with Verifiable Rewards

RLVR uses verifiable answer rewards, while GRPO estimates advantages from normalized rewards across groups of sampled solutions. Its clipped objective and importance sampling calibrate updates while keeping the current policy near the old policy.

  • Verifiable Reward Function: Verifiable rewards compare an extracted model answer with a predefined golden answer, assigning reward based on correctness.The example uses a required \boxed{} format and regex extraction.
  • Verifiable Reward Function: This reward design reduces reward-hacking risk and supports scaling reinforcement-learning training.The passage attributes this benefit to verifiable reward design.
  • Group Relative Policy Optimization: GRPO estimates each solution’s advantage by normalizing its reward relative to the mean and standard deviation of N group samples.Because it uses group reward scores, GRPO does not require an additional value model.
  • Group Relative Policy Optimization: GRPO inherits PPO’s clipped objective and uses importance sampling to calibrate gradients because sampled solutions come from the old policy.KL divergence and clipping keep the current policy within the old policy’s trust region.

3 Learning to Reason under Off-Policy Guidance

LUFFY extends GRPO with off-policy reasoning traces and policy shaping, combining guided learning with on-policy exploration. The shaping mechanism addresses reduced exploration by emphasizing unfamiliar, low-probability actions.

  • 3.1 Mixed-Policy GRPO: Mixed-Policy GRPO adds off-policy trajectories to on-policy rollouts for advantage estimation and calibrates their gradients with importance sampling.The off-policy ratio uses the behavior policy πϕ, while the on-policy ratio uses πθold.
  • 3.1 Mixed-Policy GRPO: Higher-reward off-policy traces receive greater advantage when the model struggles, while successful on-policy traces increasingly drive exploration.This adaptive group computation shifts emphasis from demonstrations toward the model’s own successful reasoning.
  • 3.2 Policy Shaping via Regularized Importance Sampling: Mixed-Policy GRPO accelerates convergence but significantly reduces exploration because entropy collapses faster than in on-policy RL.The objective can reinforce off-policy tokens already likely under πθ while ignoring low-probability tokens carrying unfamiliar reasoning capabilities.
  • 3.2 Policy Shaping via Regularized Importance Sampling: Policy shaping replaces the off-policy ratio with a transformation f(·) that increases gradient emphasis on low-probability actions.The chosen shaping function is f(x) = x/(x + γ), with γ = 0.1.
  • 3.2 Policy Shaping via Regularized Importance Sampling: Regularized importance sampling assigns more weight to unfamiliar but effective off-policy decisions and can achieve lower sampling-weight variance for more stable training.The variance result is supported by a first-order approximation and a special-case derivation.

4 Experimental Setup

The experiments train on filtered DeepSeek-R1 reasoning traces and evaluate math reasoning across six in-distribution benchmarks, three out-of-distribution benchmarks, and several RLVR baselines.

  • Dataset Construction: Training uses 45k filtered prompts and off-policy traces from OpenR1-Math-220k, after removing generations longer than 8192 tokens or verified incorrect by Math-verify.The default source subset contains 94k prompts before filtering.
  • RL Practice: RL practice removes the KL loss, sets the entropy coefficient to 0.01, and uses eight rollouts per prompt.Policy shaping uses γ = 0.1; rollout and update batch sizes are 128 and 64, respectively.
  • Evaluation: Evaluation covers AIME 2024, AIME 2025, AMC, Minerva, OlympiadBench, and MATH-500, reporting avg@32 for the smaller tests and pass@1 for the others.The setup also evaluates generalization on three out-of-distribution benchmarks.
  • Baseline Methods: Compared RLVR baselines include Simple-RL, Oat-Zero, PRIME-Zero, and OpenReasonerZero, using rule-based rewards or modified policy-optimization procedures.The baselines are used to compare LUFFY with prior RLVR methods.

5 Experimental Results

LUFFY outperforms prior RLVR and alternative off-policy methods across in-distribution and out-of-distribution mathematics benchmarks, while also training weaker models where on-policy RLVR fails. Its training dynamics indicate adaptive imitation, sustained exploration, and later-stage gains from policy shaping.

  • 5.1 Main Results: 50.1 average score across six challenging competition-level benchmarks, outperforming existing RLVR methods by +6.4 points.LUFFY establishes a new state-of-the-art on RLVR with Qwen2.5-Math-7B under the unified evaluation setting.
  • 5.1 Main Results: +6.2 points over OpenReasoner-Zero across three out-of-distribution benchmarks, where LUFFY achieves a 57.8 average score.The result supports stronger generalization on diverse out-of-distribution tasks.
  • 5.1 Main Results: +10.3 points on out-of-distribution tasks over three alternative off-policy baselines, while LUFFY uses 59% of the GPU hours of SFT+RL.SFT+RL also trails LUFFY by +1.9 points in-distribution and +16.1 points out-of-distribution.
  • 5.2 LUFFY Across Models: LUFFY surpasses SFT and on-policy RL across Qwen2.5-Math-1.5B, Qwen2.5-Instruct-7B, and LLaMA-3.1-8B.These experiments indicate applicability across small, instruction-tuned, and weak models.
  • 5.2 LUFFY Succeeds Where On-Policy Fails: On the LLaMA-3.1-8B Hard subset, on-policy RL training rewards collapse to zero, whereas LUFFY achieves stable reward improvements on both Easy and Hard subsets.On-policy RL performs well on Easy but struggles to obtain positive feedback on Hard.
  • 5.3 Training Dynamics: LUFFY initially imitates off-policy trajectories, then increasingly relies on on-policy rollouts while sustaining higher entropy than on-policy RL.On-policy entropy converges near zero after approximately 200 steps; LUFFY’s policy shaping supports continued exploration and later validation gains.

6 Related Work

Related work situates LUFFY within reinforcement-learning approaches for improving language-model reasoning and contrasts on-policy experience use with off-policy experience reuse. The supplied passages also identify policy shaping as a distinct component of LUFFY.

  • Policy Shaping: Figure 6 examines the effects of policy shaping within LUFFY’s mixed-policy training approach.The supplied figure text identifies the comparison but does not state the figure’s axes or specific outcome.
  • RL for LRMs: Recent work improves language-model reasoning through RL, including RLVR with purely verifiable rewards and test-time adaptation mechanisms.The cited related-work passage names DeepSeek-R1, OpenAI-o1, and Kimi-1.5 among recent advances.
  • On-Policy and Off-Policy RL: On-policy methods update from current-policy trajectories, whereas off-policy methods reuse experiences from diverse policies.The distinction frames stability and exploration-space trade-offs in reinforcement-learning experience utilization.

7 Conclusion

LUFFY integrates off-policy reasoning guidance into RLVR while balancing imitation with exploration. It outperforms strong baselines on math benchmarks and out-of-distribution tasks, supporting off-policy learning as a promising direction for reasoning models.

  • LUFFY dynamically balances imitation and exploration by integrating external reasoning traces into RLVR.The framework uses off-policy guidance without sacrificing novel-solution discovery.
  • LUFFY outperforms strong baselines across competitive math benchmarks and generalizes robustly to out-of-distribution tasks.
  • The results position off-policy learning as a scalable and principled path toward more general, capable, and self-improving reasoning models.

A Limitations

The theoretical analysis studies importance-weighted policy-gradient optimization for nonconvex finite-sum objectives under smoothness, bounded-gradient, and bounded-weight assumptions. It establishes an O(1/K) convergence rate and O(1/ϵ^2) iteration complexity, while requiring advance knowledge of K for the stated step size.

  • The analysis considers nonconvex finite-sum objectives and an importance-weighted policy-gradient estimator.The objective and component functions may be nonconvex and are assumed Lipschitz smooth.
  • The convergence guarantee assumes Lipschitz smoothness, σ-bounded gradients, and bounded positive instance weights.
  • O(1/K) is the convergence rate of the importance-weighted policy-gradient estimator.
  • O(1/ϵ^2) is the time complexity for obtaining an ϵ-accurate solution.
  • The stated step-size choice requires knowing the total iteration count K in advance, while a time-decayed alternative is suggested.

B.2 Informal Analysis on Variance of Regularized Importance Sampling

The informal analysis regularizes importance weights with a shaping function and uses Taylor approximation under a special weight distribution. It concludes that regularization reduces variance and supports more stable off-policy training.

  • Vanilla importance sampling can have large weight variance that harms estimate accuracy.
  • LUFFY regularizes importance weights with a shaping function to improve learning from low-probability tokens.
  • The analysis approximates the shaping function with a first-order Taylor expansion around u = 1.
  • The special-case derivation models the original importance weight with p(x) = e^-x and relates this choice to off-policy trajectory probabilities.
  • Var[f(x)] < Var[x], indicating reduced variance for the regularized weight in the analyzed case.

C Experimental Details

Experiments evaluate LUFFY across multiple foundation models, training configurations, datasets, prompts, and statistical tests. LUFFY consistently outperforms SFT and on-policy RL, while the setup includes model-specific context and prompting choices.

  • Datasets: The experiments use Easy and Hard subsets of OpenR1-MATH-220K, split by the length of correct DeepSeek-R1 solutions.
  • Training: LUFFY is evaluated on Qwen2.5-Math-7B, Qwen2.5-Math-1.5B, Qwen2.5-Instruct-7B, and LLaMA 3.1-8B.
  • Model configuration: Owing to lengthy off-policy samples, Qwen2.5-Math uses rope theta 40000 and a 16384-token window, while Qwen2.5-Instruct keeps its existing configuration.
  • Model-specific setup: LLaMA3.1-8B uses the Easy subset and a simplified CoT prompt without <think> tokens because on-policy RL fails on harder or broader subsets.
  • Evaluation: Significance tests use paired bootstrap resampling with 1000 samples, and LUFFY and LUFFY† significantly outperform all baseline methods.
  • Case study: LUFFY produces balanced reasoning in the case study, unlike SFT’s repetition and GPRO’s concise but unfounded deductions.
  • Policy updates: LUFFY removes the on-policy clip because off-policy guidance causes more frequent clipping that can suppress learning from high-quality traces.
  • Results: On Qwen2.5-Math-1.5B, LUFFY reaches 38.0 average score, gaining +6.1 over SFT and +8.0 over On-Policy RL.

E.3 Ablation Study

The ablation study finds that shaping and NoClip improve Mixed-Policy training only when paired with off-policy guidance. Policy shaping performs best at γ = 0.1, while nearby choices reduce accuracy.

  • Component ablations: Shaping and NoClip both improve Mixed-Policy training, but neither enhancement helps without off-policy guidance.On-Policy + No Clip/Shaping does not yield improvement, underscoring the necessity of external signals.
  • Policy-shaping coefficient: 50.1 accuracy scores are achieved when γ = 0.1, while increasing or decreasing γ causes a notable performance decline.The study evaluates γ values from 0.05 to 0.5 and uses γ = 0.1 throughout the experiments.

F.1 LUFFY Learns Strategically from Off-Policy Traces, While SFT Imitates Rigidly

LUFFY selectively adopts useful reasoning patterns from off-policy traces, whereas SFT tends to imitate their superficial form. This difference appears in generation length, trace similarity, and temperature-based exploration behavior.

  • Generation length: 2,832 tokens is LUFFY’s average generation length, versus 4,646 tokens for SFT on six mathematical reasoning benchmarks.The shorter LUFFY generations suggest a reasoning process that balances imitation and exploration more effectively.
  • Trace similarity: 57.5 BLEU for SFT, 8.8 for On-Policy RL, and 44.8 for LUFFY quantify differing similarity to DeepSeek-R1 traces.SFT shows strong imitation, while LUFFY exhibits more selective but substantial imitation on 1,000 held-out samples.
  • Training dynamics: RL w/ SFT Loss rapidly produces excessively long outputs, whereas LUFFY’s generation length increases more gradually during training.The contrast reflects rigid imitation by RL w/ SFT Loss and more strategic learning through LUFFY’s dynamic advantage balancing.
  • Temperature and exploration: Higher temperatures improve pass@8 for On-Policy RL and LUFFY, indicating exploratory behavior that may support test-time compute scaling.SFT performs comparably to LUFFY at temperature 0.1 but deteriorates as temperature increases.
Loading 2504.14945v5…