PaperlayerView exact revision on arXiv

Source-linked AI summary

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models

Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, Yi Dong

arXiv:2505.24864v1cs.CLcs.AI

TL;DR

The paper asks whether RL genuinely expands reasoning beyond a base model or merely sharpens latent solutions. It introduces prolonged RL with stability controls and diverse tasks, finding new reasoning pathways, broad performance gains, and continued improvement over extended training, while noting boundaries on task coverage and some high-baseline tasks.

  • Problem

    The paper addresses limited evidence about whether RL creates new reasoning capabilities or only improves sampling of solutions already present in base models.

  • Method

    ProRL combines prolonged training, KL regularization, reference-policy resets, improved GRPO, and diverse verifiable tasks spanning math, code, STEM, logic, and instruction following.

  • Results

    ProRL-trained models discover solution pathways absent from base models and outperform baselines across diverse domains, with continued gains beyond 2k training steps.

  • Takeaways & Limitations

    The findings support extended RL as a way to expand reasoning boundaries, particularly on tasks where the base model initially struggles and on out-of-distribution problems.

  • Takeaways & Limitations

    Some high-baseline tasks show unchanged or diminished broader reasoning capacity, and the limited task scope prevents guarantees across all untrained reasoning domains.

Abstract

from arXiv · show

Recent advances in reasoning-centric language models have highlighted reinforcement learning (RL) as a promising method for aligning models with verifiable rewards. However, it remains contentious whether RL truly expands a model's reasoning capabilities or merely amplifies high-reward outputs already latent in the base model's distribution, and whether continually scaling up RL compute reliably leads to improved reasoning performance. In this work, we challenge prevailing assumptions by demonstrating that prolonged RL (ProRL) training can uncover novel reasoning strategies that are inaccessible to base models, even under extensive sampling. We introduce ProRL, a novel training methodology that incorporates KL divergence control, reference policy resetting, and a diverse suite of tasks. Our empirical analysis reveals that RL-trained models consistently outperform base models across a wide range of pass@k evaluations, including scenarios where base models fail entirely regardless of the number of attempts. We further show that reasoning boundary improvements correlates strongly with task competence of base model and training duration, suggesting that RL can explore and populate new regions of solution space over time. These findings offer new insights into the conditions under which RL meaningfully expands reasoning boundaries in language models and establish a foundation for future work on long-horizon RL for reasoning. We release model weights to support further research: https://huggingface.co/nvidia/Nemotron-Research-Reasoning-Qwen-1.5B

1 Introduction

The paper examines whether reinforcement learning expands reasoning beyond a base model or mainly improves sampling of latent solutions. It proposes prolonged, diverse-task RL and reports new solution pathways, continued gains, and stronger performance across reasoning domains.

  • The central debate is whether RL unlocks new reasoning capabilities or only improves sampling efficiency for solutions already embedded in the base model.
  • Prior studies may underestimate RL by focusing on overtrained mathematics domains and terminating training after only hundreds of steps.
  • ProRL extends training beyond 2k steps and uses diverse math, code, STEM, logic, and instruction-following tasks.
  • 14.7% math, 13.9% coding, 54.8% logic-puzzle, 25.1% STEM, and 18.1% instruction-following average pass@1 improvements are reported over DeepSeek-R1-1.5B.
  • RL-trained models discover solution pathways absent from base models, including tasks where the base model has no correct samples but the trained model reaches 100% pass rates.
  • The reported findings associate stronger RL gains with weaker base-model task performance and show higher trajectory novelty after prolonged training.
  • The authors argue that RL can enhance capabilities without additional training data through sustained exploration of new reasoning strategies.

2 ProRL: Prolonged Reinforcement Learning

ProRL combines GRPO with exploration-preserving mechanisms for stable, long-horizon reinforcement learning. Its design addresses entropy collapse, reference-policy drift, and diminishing updates during extended training.

  • Group Relative Policy Optimization: GRPO removes PPO’s value model and estimates advantages from relative scores within groups of sampled responses.
  • Exploration and entropy: Entropy collapse narrows the output distribution, reducing the diverse samples needed for reliable relative-advantage estimates and causing training stagnation.
  • Exploration and entropy: Higher rollout temperature initially encourages exploration but only delays, rather than prevents, continued entropy decline.
  • Exploration and entropy: DAPO-style decoupled clipping raises probabilities of unlikely tokens, while dynamic sampling prioritizes intermediate-difficulty prompts with useful learning signals.
  • KL regularization: A KL penalty regularizes the online policy toward a reference policy, preserving entropy and reducing drift or overfitting to spurious rewards.
  • KL regularization: KL regularization is especially beneficial here because training starts from a checkpoint already capable of coherent chain-of-thought outputs.
  • Reference policy reset: Because the KL term can dominate later training, periodic reference-policy resets and optimizer reinitialization restore effective policy updates.

3 Nemotron-Research-Reasoning-Qwen-1.5B: The World’s Best 1.5B Reasoning Model

Nemotron-Research-Reasoning-Qwen-1.5B is a generalist model trained with prolonged reinforcement learning on 136K verifiable problems across five domains. It achieves broad gains over its base model and domain-specialized baselines, including on out-of-distribution tasks.

  • Model and training: Nemotron-Research-Reasoning-Qwen-1.5B uses reinforcement learning on 136K verifiable problems spanning math, code, STEM, logical puzzles, and instruction following.The training data provides binary or continuous reward signals across the five task domains.
  • Results: The model improves over DeepSeek-R1-Distill-Qwen-1.5B by 15.7% on math, 14.4% on code, 25.9% on STEM, 22.0% on instruction following, and 54.8% on logic puzzles.These results are reported across the paper’s multi-domain evaluation.
  • Training dynamics: During prolonged training, KL divergence loss and DAPO enhancements helped avoid entropy collapse, while validation performance improved in stages without requiring longer responses.A final approximately 200-step stage increased the context window to 16k tokens and produced measurable improvements.
  • Evaluation: The evaluation covers math, coding, STEM reasoning, instruction following, logical puzzles, and out-of-distribution Reasoning Gym tasks.Math and coding use pass@1 estimates from 16 samples, while logical puzzles and instruction following use average continuous reward scores.
  • Generalization: The model shows significant improvements on three out-of-distribution Reasoning Gym tasks, indicating stronger generalization beyond the training distribution.The reported OOD tasks are acre, boxnet, and game_of_life_halting.
  • Results: The model achieves superior pass@1 scores over domain-specialized baselines by 4.6% on math and 6.5% on code.The specialized comparators are DeepScaleR-1.5B for mathematics and DeepCoder-1.5B for competitive programming.

4 Analysis: Does ProRL Elicit New Reasoning Patterns?

ProRL’s reasoning-boundary effects depend on task difficulty, base-model competence, and training duration: gains may diminish, plateau, or continue with prolonged RL. Extended training also improves generalization to out-of-distribution and harder tasks, while shifting pass@1 distributions toward successful solutions.

  • Analysis setup: Increasing inference to 256 samples compares the base, intermediate, and final prolonged-RL models across selected Reasoning Gym and other benchmarks.The evaluation samples 18 of 96 Reasoning Gym tasks because of compute limits.
  • Base competence and reasoning-boundary gains: ProRL expands reasoning boundaries most on tasks where the base model initially struggles, whereas familiar tasks show minimal or negative breadth gains.Minimal-gain tasks tend to have lower creativity indices, suggesting greater overlap with pretraining data.
  • Diminish, Plateau, and Sustained Gains: Some benchmarks diminish because RL sharpens a narrow output distribution, while others plateau after early gains or sustain improvement with prolonged training.The sustained regime is especially associated with complex tasks such as coding.
  • Out-of-distribution reasoning: On the unseen OOD task boxnet, the base model shows no solving capability, whereas ProRL produces significant ability and extended training sustains the improvement.This supports generalization beyond the training distribution.
  • Increased task difficulty: On graph_color, ProRL maintains higher pass@1 and pass@128 accuracy than the base and intermediate models as graph size and task difficulty increase.Training used graphs of size 10, while evaluation tested larger graphs.
  • Pass@1 distribution evolution: ProRL shifts pass@1 distributions rightward, including a novel reasoning task that moves from predominantly zero accuracy to peaks at perfect accuracy.The shifts increase expected pass@1 enough to offset negative effects from increased variance.

5 Related Work

Prior work frames reasoning models as systems that use long chain-of-thought and verifiable-reward reinforcement learning, while earlier RL systems demonstrate continued improvement through iterative data collection and policy updates.

  • Reasoning Models: Reasoning models generate detailed long chain-of-thought before final answers and are commonly trained with reinforcement learning using verifiable rewards.The related work discusses GRPO, Mirror Descent, RLOO, and other RL variants.
  • RL Reasoning Boundary: DeepQ, AlphaGo, and AlphaZero exemplify reinforcement-learning systems that improve through repeated interaction, data collection, and policy improvement.AlphaGo and AlphaZero alternate Monte Carlo Tree Search data collection with policy improvement.

6 Conclusion

The paper argues that reinforcement learning can expand language-model reasoning boundaries. It presents ProRL as evidence that extended, stable training develops novel reasoning patterns beyond a base model’s initial capabilities.

  • The paper addresses whether reinforcement learning can truly expand language models’ reasoning boundaries.
  • ProRL uses KL divergence penalties, periodic reference-policy resets, and diverse training data to support stable long-duration training of a 1.5B generalist reasoning model.The data span mathematics, coding, STEM, logical puzzles, and instruction following.

A Limitations

The paper identifies substantial resource, scalability, training-process, and task-scope limitations for ProRL. These constraints affect affordability, transfer to larger models, reproducibility, and generalization beyond evaluated domains.

  • Computational Resources: Extended RL training requires substantial computational resources that may be prohibitive for smaller organizations and limited-budget researchers.Multiple training stages, periodic resets, and long reasoning-chain sampling intensify these requirements.
  • Scalability Concerns: ProRL’s scalability to larger models remains unclear despite effective training demonstrated for a 1.5B-parameter model.Computational requirements become more pronounced as parameter counts increase.
  • Training Process Challenges: Periodic hard resets of the reference policy and optimizer parameters add training complexity and may produce inconsistent results.The paper contrasts this process with more stable training methods.
  • Limited Task Scope: The training dataset covers only a subset of possible reasoning tasks, limiting guarantees of improvement across all unrepresented domains.Promising OOD performance does not establish similar gains for every potential reasoning domain.

B Societal Impacts

ProRL is presented as expanding reasoning capabilities across domains while creating access, governance, resource-inequality, and misuse concerns. The paper emphasizes that enhanced capabilities require monitoring, safeguards, and evolving evaluation.

  • Societal Impacts: ProRL creates opportunities and challenges by enhancing language-model reasoning capabilities across domains.The paper frames these effects as relevant to both the AI research community and society.
  • Potential Benefits and Opportunities: The approach potentially achieves superhuman reasoning with sufficient compute, while 1.5B-parameter models may broaden access to advanced AI capabilities.The paper highlights educational settings as especially relevant under resource constraints.
  • Ethical Considerations and Challenges: Substantial computational requirements may exacerbate resource inequality in AI development.This concern is presented alongside the broader opportunities of enhanced reasoning capabilities.
  • Ethical Considerations and Challenges: Enhanced reasoning capabilities could enable more sophisticated misuse without appropriate safeguards, making ongoing monitoring essential.The paper recommends value alignment approaches and dynamic benchmarks that evolve with model capabilities.
  • Examples: The box-coloring example requires returning a JSON map that assigns colors to graph vertices.The task asks for a coloring in which adjacent vertices do not share a color.
  • Examples: The box-moving example asks a central planner to produce efficient multi-step action plans matching colored boxes to corresponding targets.Plans must assign one action per agent, avoid conflicts, and minimize moves.

D Training Dataset

The training dataset combines diverse verifiable-reward tasks spanning mathematics, coding, STEM, logical puzzles, and instruction following. This breadth exposes the model to varied reasoning patterns and supports evaluation across multiple domains.

  • Training Dataset: Training spans mathematics, code generation, STEM problems, logical puzzles, and instruction following, with task details summarized by domain and reward type.The dataset overview categorizes sources, dataset sizes, and binary or continuous rewards.
  • Mathematics: The math component contains 40K competition problems with verifier-based binary rewards.The pipeline uses DeepScaleR-curated data, the original verifier, and an improved math verifier.
  • Code: The coding component comprises 24K competition problems and assigns continuous rewards based on the fraction of test cases passed.Compilation, syntax, and timeout failures receive zero reward.
  • STEM: SCP-116K supplies 274K scientific problem-solution pairs across physics, chemistry, biology, and mathematics.Because the data were automatically extracted from heterogeneous sources, the authors applied rigorous filtering.
  • Logical Puzzles: Reasoning Gym contributes approximately 100 logical-puzzle tasks and 37K synthetic training samples across varied reasoning domains.Covered domains include algebra, arithmetic, computation, cognition, geometry, graph theory, logic, and popular games.
  • Instruction Following: Synthetic Llama-Nemotron data pair tasks with randomly chosen instructions to improve instruction following.Examples combine a requested task with constraints such as producing a three-paragraph response.

E Training Recipe

The ProRL recipe monitors validation performance during extended training, uses KL divergence control with periodic reference-model and optimizer resets, and iteratively adjusts data, rewards, rollout counts, and context length.

  • Training Monitoring: Validation monitoring tracks AIME2024, Codeforces, GPQA-diamond, IFEval, and Reasoning Gym graph_color during training.Evaluation uses sampling parameters similar to final evaluation while retaining the training context window.
  • Reference Model and Optimizer Reset: Hard resets of the reference model and optimizer are triggered when validation metrics degrade substantially or improvements plateau.Resets restore stability and permit changes such as added data, reward shaping, and hyperparameter adjustments.
  • Training Stability: KL divergence is tracked across training runs while the reference policy and optimizer state are periodically reset.These controls support stability during prolonged training.
  • Run Schedule: The initial run trains four tasks with an 8k response-length limit, after which validation instability motivates later intervention.The base model’s sequential length is 128k, but rollouts are constrained to avoid long sequences.
  • Run Schedule: Run 2 resumes after a hard reference-policy reset while retaining the 8k maximum response length.The authors report that this limit was sufficient for learning and validation improvement.
  • Run Schedule: Instruction-following data are added in Run 3, while later runs penalize improper termination to reduce repeated, non-terminating responses.Reward shaping produces a modest reduction in response length.
  • Run Schedule: Runs 6 and 7 increase rollouts from 16 to 32 with two hard resets, while Run 8 extends context to 16k tokens and reduces rollouts to 16.The extended context window yields marginal AIME improvements and larger gains in other domains.

F.1 Reasoning Gym

The model outperforms both DeepSeek models across the evaluated Reasoning Gym tasks, with results reported through subcategory analyses and pass@k comparisons. Evaluation also spans mathematics, code, logic, IFEval, GPQA, OOD, and graph-coloring tasks, though some Reasoning Gym tasks were sampled due to compute limits.

  • Reasoning Gym evaluation categorizes logic puzzles into 96 tasks according to the official repository.
  • The evaluation notes that the model can format answers with \boxed{} instead of the required <answer> </answer> tags, affecting some Reasoning Gym results.
  • The model demonstrates superior performance across all reported Reasoning Gym reasoning-task subcategories compared with both DeepSeek models.
  • Pass@k comparisons cover three models across evaluated tasks, with a randomly selected subset of Reasoning Gym tasks used because of compute limitations.
  • The supplementary figures report pass@k for Math, IFEval, GPQA, OOD, Code, Reasoning Gym, and graph_color tasks at varying node counts.The graph_color training default is 10 nodes.
  • Pass@1 distributions are reported separately for Math, IFEval, GPQA, OOD, Code, and Reasoning Gym tasks.
Loading 2505.24864v1…