PaperlayerView exact revision on arXiv

Source-linked AI summary

BroRL: Scaling Reinforcement Learning via Broadened Exploration

Jian Hu, Mingjie Liu, Ximing Lu, Fang Wu, Zaid Harchaoui, Shizhe Diao, Yejin Choi, Pavlo Molchanov, Jun Yang, Jan Kautz, Yi Dong

arXiv:2510.01180v1cs.LGcs.CL

TL;DR

RLVR performance can plateau when scaling only training steps, motivating a complementary way to scale exploration. BroRL increases rollouts per example and analyzes how sampled and unsampled tokens affect correct-token mass. Larger rollout sizes suppress harmful unsampled coupling, support continuous gains beyond ProRL saturation, and yield strong 1.5B-model reasoning results with improved efficiency.

  • Problem

    ProRL-style step scaling can plateau after thousands of steps, leaving effective scaling of RLVR unresolved.

  • Method

    BroRL scales the number of rollouts per example, using mass-balance analysis to characterize sampled and unsampled contributions to correct-token probability mass.

  • Results

    BroRL sustains improvement beyond ProRL saturation, achieves state-of-the-art results for a 1.5B model across complex reasoning tasks, and nearly doubles throughput in the reported hardware setup.

  • Takeaways & Limitations

    Rollout size N is a critical and efficient RL scaling axis, and sufficient exploration can revive models that have stagnated under longer training.

  • Takeaways & Limitations

    The theoretical analysis relies on a simplifying one-step RLVR assumption, while the unsampled contribution can remain positive or negative depending on batch mood and unsampled-token concentration.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key ingredient for unlocking complex reasoning capabilities in large language models. Recent work ProRL has shown promise in scaling RL by increasing the number of training steps. However, performance plateaus after thousands of steps, with clear diminishing returns from allocating more computation to additional training. In this work, we investigate a complementary paradigm for scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds to exhaustively Broaden exploration, which yields continuous performance gains beyond the saturation point observed in ProRL when scaling the number of training steps. Our approach is motivated by a mass balance equation analysis allowing us to characterize the rate of change in probability mass for correct and incorrect tokens during the reinforcement process. We show that under a one-step RL assumption, sampled rollout tokens always contribute to correct-mass expansion, while unsampled tokens outside rollouts may lead to gains or losses depending on their distribution and the net reward balance. Importantly, as the number of rollouts per example N increases, the effect of unsampled terms diminishes, ensuring overall correct-mass expansion. To validate our theoretical analysis, we conduct simulations under more relaxed conditions and find that a sufficiently large rollout size N-corresponding to ample exploration-guarantees an increase in the probability mass of all correct tokens. Empirically, BroRL revives models saturated after 3K ProRL training steps and demonstrates robust, continuous improvement, achieving state-of-the-art results for the 1.5B model across diverse benchmarks.

1 INTRODUCTION

BroRL scales RLVR through hundreds of rollouts per example, targeting the exploration limits that cause step-scaling methods such as ProRL to plateau. Its analysis and experiments indicate that larger rollout sizes reduce harmful unsampled effects, sustain improvement beyond ProRL saturation, and improve efficiency.

  • Approach: BroRL increases rollouts per example to hundreds or thousands, using broadened exploration as a complementary RL scaling axis.The approach is motivated by mass-balance analysis of correct-token probability mass.
  • Theory: As rollout size N grows, unsampled effects diminish and the overall correct-token mass change approaches ΔQpos ≥0.The sampled portion always provides a nonnegative gain, whereas the unsampled portion can add or remove mass.
  • Empirical motivation: BroRL (N = 512) continues improving math performance, while ProRL (N = 16) plateaus at 3k steps and degrades with prolonged training.Figure 1 compares the two continued-training strategies.
  • Simulation validation: Sufficiently large rollouts increase the probability mass of all correct tokens and eliminate knowledge shrinkage in simulations.The simulations use a TRPO-style linear surrogate objective under more relaxed conditions.
  • Efficiency and significance: Under identical hardware, BroRL nearly doubles throughput versus ProRL while reducing filtered samples and shifting generation from memory-bound to compute-bound operation.The reported efficiency gains combine algorithmic and hardware-level effects.

2 THEORETICAL ANALYSIS

The theoretical analysis decomposes one-step RLVR changes in correct-token probability mass into sampled and unsampled contributions. Sampled terms are nonnegative, while the potentially negative unsampled coupling shrinks with rollout size, linking larger N to more reliable pass@k improvement.

  • Setup: The analysis models one-step RLVR logit updates and tracks the probability mass assigned to correct tokens within a vocabulary partitioned into correct and incorrect sets.The treatment uses a mass-balance argument and distinguishes sampled from unsampled tokens.
  • Mass-balance decomposition: The correct-mass change decomposes into a nonnegative sampled term and an unsampled coupling term that can be negative but vanishes as rollout size N grows.This decomposition identifies rollout size as a key trade-off for performance.
  • Sampled terms: Sampled-correct tokens gain mass through (Rc − SR)QnegA2, while sampled-incorrect tokens free mass for correct tokens through (SR − Rw)QposB2.Both effects are nonnegative under the stated reward ordering and become stronger with relevant concentration.
  • Unsampled term: The unsampled contribution may help or hurt depending on batch mood SR and whether unsampled incorrect or correct mass is more concentrated.For positive-reward batches, downward unsampled updates help when incorrect mass is more concentrated and hurt when correct mass is more concentrated; signs reverse for negative-reward batches.
  • Implications: Increasing N shrinks unsampled second moments monotonically, driving ΔQpos toward nonnegativity and strengthening pass@k improvements because pass@k increases with Qpos.The analysis also notes positivity for balanced batches or sufficiently small unsampled mass, even at small N.

3 BRORL: BROAD REINFORCEMENT LEARNING

BroRL scales reinforcement learning by increasing rollouts per prompt while retaining prolonged PPO training and exploration-focused sampling techniques. Larger rollout sizes reduce harmful unsampled-token coupling, producing more stable policy updates and faster correct-mass growth.

  • Background: Prolonged Reinforcement Learning: BroRL builds on clipped PPO with decoupled advantage normalization across the global sample batch.Trajectory advantages subtract the corresponding prompt group's mean return before global-batch normalization.
  • Background: Prolonged Reinforcement Learning: Dynamic Sampling filters entirely correct or entirely incorrect trajectories so training focuses on informative samples.The filtered batch is constructed from rollout trajectories using predictions and an indicator function.
  • Background: Prolonged Reinforcement Learning: Additional techniques include periodic reference-policy resets, Clip-Higher exploration, and truncated importance sampling for engine mismatch correction.Clip-Higher uses ε_high > ε_low, while truncated importance sampling addresses off-policy mismatch between inference and training engines.
  • BroRL: Larger rollout sizes reduce the detrimental influence of unsampled coupling, increasing rollout diversity and stabilizing policy optimization.The framework links this effect inversely to rollout size N and uses large N to minimize variance and potential negativity from unsampled action-space regions.
  • BroRL: Figure 3 tracks correct-action mass, the fraction of improving correct actions, and the worst negative probability change under varying rollout sizes.These measures evaluate whether larger N produces faster, more stable updates and removes knowledge shrinkage.

4 EXPERIMENTS

Experiments show that increasing rollout size to N = 512 improves exploration, stabilizes correct-token learning, and revives performance after ProRL’s 3,000-step plateau. BroRL outperforms or continues improving over ProRL across most trajectories while using fewer training hours and higher generation throughput.

  • 4.1 TOKEN-LEVEL SIMULATIONS: At large N, correct-token mass grows more reliably, the share of improving correct tokens increases, and knowledge shrinkage disappears.Simulations varied N from 4 to 51,200 and found that sufficiently large rollouts eliminate worst-case probability drops among correct tokens.
  • 4.2.2 ANALYSIS OF PASS@1 SUCCESS RATE: BroRL consistently outperforms ProRL when both improve, continues improving when ProRL degrades, and rarely shares ProRL’s failure to produce consistent gains.The third trajectory suggests N = 512 may remain insufficient for some harder problems; results are measured on test data and therefore also reflect generalization.
  • 4.2.2 ANALYSIS OF PASS@1 SUCCESS RATE: BroRL’s final-checkpoint Pass@1 advantage over ProRL was Δ = 0.0033 with t = 4.84 and one-tailed p = 6.5 × 10^-7 under approximately 140 hours of equal training compute.The comparison used over 10,000 problem instances and a paired t-test.
  • 4.2.3 PUSHING REASONING BOUNDARIES BEYOND STEPS SCALING: 63.03 in Math, 54.20 in Code, and 63.09 in Reasoning Gym were BroRL’s highest scores, surpassing ProRL after 98.1 hours—about 35 fewer hours.BroRL improved continuously across all three benchmarks, whereas ProRL stagnated or degraded.
  • 4.2.4 ROLLOUT SIZE SCALING’S IMPACT ON GPU COMPUTE EFFICIENCY: N = 512 raised the dynamic-sampling pass rate to 62% versus 41% for N = 16, while generation throughput reached 72.4 versus 36.5 samples/s.Larger batches reduce wasted samples and shift generation from memory-bound toward compute-bound execution.

5 RELATED WORK

Prior RL scaling work primarily increases training steps, whereas BroRL formalizes rollout size as a complementary scaling axis. Its analysis links larger rollouts to a more reliable learning signal by reducing unsampled coupling.

  • Reinforcement Learning for Reasoning: RLVR supports reasoning models through verifiable-reward training and includes methods such as GRPO, RLOO, REINFORCE++, and DAPO.These methods adapt reinforcement-learning approaches for long chain-of-thought reasoning.
  • Scaling Axes in Reinforcement Learning: ProRL scales reinforcement learning through prolonged training, but BroRL instead scales the number of rollouts sampled per prompt at each update.The paper presents rollout size N as a distinct scaling dimension for RLVR.
  • Scaling Axes in Reinforcement Learning: Increasing N dampens the negative unsampled coupling term, improving the reliability of the learning signal and addressing training instability.The paper describes this mechanism as a formal analysis of why broader exploration can support more stable reasoning-model training.

6 CONCLUSION

The paper concludes that rollout size is an efficient scaling axis for RL in language models, offering an alternative to simply extending training steps. BroRL combines theoretical mitigation of unstable learning signals with empirical gains and improved hardware efficiency.

  • 6 CONCLUSION: BroRL establishes rollout size N as a critical and efficient RL scaling axis beyond longer training.The conclusion frames steps-scaling plateaus as potentially arising from insufficient exploration rather than fundamental limits.
  • 6 CONCLUSION: Increasing N mitigates the unsampled coupling term, which the analysis identifies as a primary source of learning-signal instability.The proposed mechanism connects broader exploration with more stable policy improvement.
  • 6 CONCLUSION: BroRL also doubled hardware throughput in some cases by shifting generation’s bottleneck from memory to compute.This efficiency result supports the method’s practicality for real-world deployment.

A LIMITATIONS

The study’s main limitation is that it examines only a narrow range of rollout sizes, leaving the broader relationship between N and model improvement unresolved. It also notes potential societal risks from more capable reasoning and coding systems.

  • A LIMITATIONS: The experiments compare N = 16 with N = 512 but do not fully characterize how intermediate rollout sizes affect model improvement.The authors suggest evaluating values such as 64, 256, and 1024 to clarify the performance curve and cost-benefit trade-off.
  • A LIMITATIONS: Simulation results suggest gains may be monotonic but concave, yet this trend remains unvalidated on large-scale language models because broader sweeps are computationally demanding.
  • B BROADER IMPACT: BroRL may improve sample efficiency and hardware utilization, potentially lowering barriers for smaller organizations to train capable reasoning models.
  • B BROADER IMPACT: More capable reasoning and coding models could create risks in cybersecurity, information ecosystems, and labor markets, motivating continued safety and ethics work.

C.1 THEOREM 1

Theorem 1 analyzes one-step RL through a mass-balance decomposition of correct-token probability mass. It separates an always-nonnegative sampled contribution from an unsampled coupling term that can be negative and diminishes as rollout size increases.

  • C.1 THEOREM 1: The analysis uses a TRPO-style linear surrogate with rewards assigned to sampled correct, sampled incorrect, and unsampled tokens.The sampled-token contribution is summarized by SR = RcPpos + RwPneg.
  • C.1 THEOREM 1: The one-step change in correct-token mass decomposes into a sampled positive term and an unsampled coupling term.

C.2 LEMMA 2

Lemma 2 derives how an unsampled token’s second-moment contribution scales under independent Bernoulli sampling. The contribution decreases with rollout size through the probability that the token is never selected.

  • C.2 LEMMA 2: The lemma assumes each token is sampled independently and identically as a Bernoulli random variable across N rollout draws.This assumption provides a tractable formula for the scaling of the unsampled second moment.
  • C.2 LEMMA 2: For X ~ Bin(N, p), the probability that a token is never sampled is (1 − p)^N.
  • C.2 LEMMA 2: The expected unsampled second-moment contribution is E[S] = p^2(1 − p)^N.

D EMPIRICAL EVALUATION

Across math, code generation, and Reasoning Gym, BroRL’s larger rollout size outperforms continued small-rollout training after the ProRL plateau. It achieves these gains with comparable generated samples and fewer wall-clock GPU hours.

  • D EMPIRICAL EVALUATION: BroRL reaches 62.85 on mathematical reasoning after 134 steps, while ProRL moves from 61.69 to 62.08 before slightly declining to 62.02.The comparison is reported in the math results table.
  • D EMPIRICAL EVALUATION: BroRL improves code-generation performance by +1.48 points, exceeding ProRL’s +0.74-point gain.The comparison is reported in the code-generation results table.
  • D EMPIRICAL EVALUATION: On Reasoning Gym, BroRL improves by over 1.5 points, whereas ProRL provides almost no meaningful gain.The comparison is reported in the Reasoning Gym results table.
  • D EMPIRICAL EVALUATION: BroRL achieves superior results with a comparable number of generated samples while consuming fewer wall-clock GPU hours.
Loading 2510.01180v1…