PaperlayerView exact revision on arXiv

Source-linked AI summary

The Art of Scaling Reinforcement Learning Compute for LLMs

Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, Rishabh Agarwal

arXiv:2510.13786v1cs.LGcs.AI

TL;DR

RL compute is expanding, but the field lacks a principled way to predict how RL methods scale. The paper studies RL scaling over more than 400,000 GPU-hours, fits sigmoid performance curves, and derives ScaleRL; its curves accurately extrapolate performance, including a 100,000 GPU-hour run, while recipe choices produce different ceilings and efficiency profiles.

  • Problem

    RL compute has grown substantially, but researchers lack a reliable methodology for evaluating and predicting how RL methods scale with compute.

  • Method

    The paper fits sigmoid validation-performance curves and systematically ablates RL recipe choices to quantify asymptotic performance and compute efficiency.

  • Results

    ScaleRL scales better than existing recipes in the experiments, with extrapolated curves closely matching observed performance in a 100,000 GPU-hour run.

  • Takeaways & Limitations

    Smaller-scale ablations can be used to predict larger-scale RL performance and construct a recipe with predictable scaling.

  • Takeaways & Limitations

    The study primarily characterizes in-distribution validation curves, leaving full generalization from training distributions to held-out test sets beyond its scope.

Abstract

from arXiv · show

Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL, and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.

1 Introduction

RL compute is growing rapidly, but RL scaling remains less systematic and predictable than pre-training scaling. The paper introduces a sigmoid-based framework, uses large-scale ablations to identify scalable choices, and develops ScaleRL for predictable extrapolation.

  • Motivation: RL compute is increasing rapidly, while methods for understanding and scaling RL remain more ad hoc than pre-training methodologies.The paper notes major increases in RL budgets, including 100,000 H800 GPU-hours for DeepSeek-R1-Zero and more than 10× growth across some model generations.
  • Predictive framework: The framework models expected validation reward as a sigmoid-like function of training compute, with A representing asymptotic performance and B compute efficiency.Cmid sets the curve midpoint, while A and B characterize the performance ceiling and scaling efficiency.
  • Predictive framework: The fitted curves allow researchers to extrapolate higher-compute performance from lower-compute runs without running every experiment to its computational limit.This is intended to provide a systematic way to evaluate RL scalability more cost-effectively.
  • Empirical principles: More than 400,000 GPU-hours of empirical study showed that different RL methods can reach different performance ceilings, while many design choices mainly affect compute efficiency.Loss aggregation, curriculum, length penalty, and advantage normalization primarily modulate B rather than materially changing the asymptotic ceiling.
  • Empirical principles: Early training dynamics can identify scalable methods, because stable recipes follow predictable trajectories that remain accurate when extrapolated to larger compute budgets.The paper reports close agreement between extrapolated curves and longer runs for stable methods, including ScaleRL and MiniMax.
  • ScaleRL: ScaleRL combines existing RL components and achieves higher asymptotic performance and compute efficiency than established recipes while preserving predictable scaling across larger compute axes.The recipe was validated through leave-one-out ablations and tested across batch size, generation length, multitask RL, and larger MoE models.

2 Preliminaries & Setup

The paper studies RL scaling for LLM reasoning with a generator–trainer setup, clipped policy optimization, and validation-based sigmoid fitting. Its setup separates rollout generation from training and evaluates both asymptotic performance and compute efficiency.

  • RL setup: The RL system uses prompts sampled from a data distribution, with an old generator policy producing rewarded candidate completions for policy optimization.Policy optimization maximizes a clipped surrogate objective over prompts and rollouts from the old generator policy.
  • RL setup: The generator–trainer setup assigns rollout generation and parameter updates to separate GPU groups, enabling high-throughput inference alongside FSDP training.Generators use optimized inference kernels, while trainers run the training backend and update parameters.
  • RL objective: The base algorithm resembles GRPO without KL regularization and uses asymmetric DAPO clipping to help avoid entropy collapse and maintain output diversity.This serves as the starting algorithm for the later ablations.
  • RL objective: Advantages are computed from each prompt’s group of rewarded completions, and token-level importance-sampling ratios use asymmetric upper and lower clipping thresholds.The setup assigns scalar rewards to completions before computing advantages and group-normalized advantages.
  • RL objective: Losses are aggregated by averaging per-sample token losses before averaging across samples.This aggregation choice is part of the training objective studied in the paper.
  • Training regimen: Forced interruptions stop overly long reasoning generations by appending an end-of-thinking phrase, addressing length-related stability and efficiency concerns.The paper later compares this approach with length penalties.
  • Scaling fits: The paper fits pass rate against log(compute) with a sigmoid because the fit is empirically more robust and stable than a power law for bounded metrics.Fits generally exclude the earliest low-compute regime, beginning after approximately 1.5k GPU hours.
  • Scaling fits: The scaling curve summarizes saturating returns: A is asymptotic performance, while B and Cmid primarily describe compute efficiency and curve position.The experiments evaluate RL design axes using asymptotic performance A and compute efficiency B.

3 An Empirical Study of RL Scaling

The study evaluates RL design choices through asymptotic performance and compute efficiency, finding that some choices change the ceiling while others primarily affect how quickly it is reached.

  • Experimental setup: Experiments on an 8B dense model measure RL scaling through asymptotic performance A and compute efficiency B.The study uses verifiable math problems and initially tests choices at 3.5k–4k GPU-hours, extending stable variants further.
  • Asynchronous RL setup: PipelineRL matches PPO-off-policy in asymptotic performance A but substantially improves compute efficiency B by reducing idle time.PipelineRL-8 is selected because the experiments find k = 8 optimal for maximum off-policyness.
  • Algorithmic choices: GSPO and CISPO substantially outperform DAPO in asymptotic pass-rate A, with CISPO marginally better than GSPO later in training.The authors therefore choose CISPO as the preferred loss type.
  • Algorithmic choices: FP32 computation at the language-model head improves asymptotic performance A from 0.52 to 0.61 by addressing generator–trainer probability mismatches.These mismatches directly affect the importance-sampling ratio used in the surrogate objective.
  • Algorithmic choices: Prompt-level loss aggregation and batch-level advantage normalization are adopted, with batch-level normalization described as theoretically sound and marginally better.The loss-aggregation comparison reports prompt averaging as having the highest asymptotic performance.
  • Algorithmic choices: Zero-variance filtering and No-Positive-Resampling improve asymptotic performance by removing samples or prompts that provide little useful policy-gradient signal.No-Positive-Resampling permanently removes prompts whose historical pass rate reaches at least 0.9.

4 ScaleRL: Scaling RL Compute Effectively & Predictably

ScaleRL combines the study’s selected design choices and tests whether their scaling behavior remains effective and predictable. Leave-one-out experiments and extrapolation show that efficiency differences matter even when asymptotic rewards are similar.

  • Leave-one-out ablations: Leave-one-out variants usually reach similar asymptotic rewards, while ScaleRL remains slightly more effective through asymptotic reward or compute efficiency.The comparison fixes the average asymptotic reward and refits slopes to make efficiency differences visible.
  • Leave-one-out ablations: Plotting log F(Rc) against log C exposes slope B directly and shows ScaleRL achieving the highest compute efficiency.This transformation is used because most leave-one-out variants have similar asymptotic pass rates.
  • Predictability: Fits using the first 8,000 GPU-hours closely predict performance through 16,000 GPU-hours for leave-one-out experiments and independent ScaleRL runs.The alignment supports predictable scaling for stable, scalable recipes under large-scale RL training.
  • Robustness across regimes: The study argues that apparently redundant components can improve stability, robustness, or efficiency across models and RL regimes.This motivates retaining components for recurring instability and variance rather than only for marginal gains in one configuration.
  • Robustness across regimes: FP32 precision can provide large gains beyond the dense 8B ScaleRL setting, improving overall scalability in the Scout 17B×16 MoE experiment.The authors also report that CISPO is more robust to the importance-sampling clipping parameter and more efficient than DAPO, with B = 2.01 versus B = 1.77.

5 Predictable Scaling Returns Across RL Compute Axes

The study evaluates RL scaling across context length, batch size, model scale, and generation allocation using early fitted curves and extended runs. Larger context, batch size, and model scale can improve eventual performance, while some gains trade early efficiency for higher asymptotes.

  • Predictive methodology: Early fitted curves aligned with extended trajectories across the tested compute axes, supporting predictive extrapolation.The procedure fits at half the target budget, extrapolates, then extends training to verify the forecast.
  • Model scale: 17B×16 MoE ScaleRL preserved predictable, stable scaling and exceeded the 8B dense model’s performance using 1/6 of its RL compute.The extended points aligned with the fitted curve, with low truncation rates and no instability pathologies.
  • Generation length: Increasing generation length from 14k to 32k tokens slowed early progress but raised the fitted asymptote and eventual performance.The higher 32k-token trajectory was correctly forecast when training was extended.
  • Global batch size: Larger batches improve the asymptote and avoid downstream stagnation, although smaller batches can appear better during early compute.In the largest run, batch size 2048 stabilized training and supported extrapolation from 50k to 100k GPU-hours.
  • Generations per prompt: At moderate batch sizes, reallocating a fixed total batch across 8, 16, 24, or 32 generations per prompt leaves fitted curves essentially unchanged.The authors identify this allocation as a second-order choice for both asymptotic performance and efficiency.

6 Related Work

Related work reports useful RL algorithms, heuristics, and comparative findings, but generally does not study compute-scaling behavior. This work instead emphasizes predictive fits and larger-scale validation.

  • ProRL: ProRL uses prolonged RL fine-tuning and stability heuristics to uncover strategies beyond base capabilities on a 1.5B model.The reported regimen used approximately 2,000 optimization steps, batch size 64, and 16K GPU-hours.
  • LitePPO: LitePPO provides comparative ablations and a minimalist recipe that outperforms GRPO and DAPO on smaller models and compute scales.Its focus is comparative empirical findings rather than scaling behavior.
  • Positioning: The authors distinguish their work by developing predictive compute-performance fits, operating at larger compute and model scales, and scaling beyond 100,000 GPU-hours.They also use in-distribution held-out evaluation to study predictable scaling rather than relying only on downstream comparisons.

7 Discussion & Conclusion

The discussion frames ScaleRL as a recipe derived from predictive ablations, emphasizing extrapolation, decision priorities, cumulative efficiency effects, and remaining generalization and multitask questions.

  • Contribution: The paper fits validation-set scaling curves to quantify asymptotic performance and compute efficiency, then combines ablation choices into ScaleRL.The resulting recipe scales better than existing recipes in the reported experiments.
  • Compute scaling extrapolation: Smaller-scale ablations can systematically predict larger-scale performance, providing the basis for constructing the final scalable recipe.This is presented as a central insight of the scaling methodology.
  • Most important decisions: Off-policy algorithm, loss function, and model precision are the most important individual decisions, while other choices contribute cumulatively to efficiency.Leave-one-out experiments indicate that the additional choices still have some combined impact.
  • Asymptotic performance vs. efficiency: Many ablations improve both efficiency and asymptotic performance, but some, such as FP32 precision, do not; leave-one-out results show robust cumulative efficiency effects.The backward leave-one-out experiments found little individual impact on asymptotic performance from each decision.
  • Generalization: The primary scaling analysis uses in-distribution validation curves, and the paper reports correlation with downstream generalization without fully characterizing held-out test-set transfer.Larger batches, fewer truncations, longer generations, and larger models appeared more helpful for generalization.
  • Multi-task RL: Preliminary math-and-code multitask results show aligned extrapolated trends, but predictability across different data mixtures remains for future study.The experiments focus mainly on the math domain.
  • Future work: Future work includes scaling laws across pre-training compute, model size, and RL data, plus structured rewards, generative verifiers, and other post-training regimes.The paper also states that ScaleRL is not the end of the story and releases curve-fitting code for future studies.

A.1 Extended Related Work

The extended related work covers the RL methods associated with recent reasoning progress, including GRPO and DAPO, while motivating the paper’s scaling-focused perspective.

  • Recent RL progress: Recent RL work has improved LLM reasoning, but major model reports often provide limited training details.The section cites OpenAI’s o1 series and DeepSeek R1 as examples of this recent wave.
  • GRPO: GRPO is a critic-free, group-relative policy-gradient method with PPO-style clipping and group baselines for lower cost and stabilized long-chain credit assignment.It catalyzed rapid progress in verifiable-reward reasoning development.
  • DAPO: DAPO modifies GRPO through decoupled clipping, Clip-Higher, dynamic prompt sampling, and token-level loss aggregation.These changes target entropy collapse, zero-variance samples, and the loss-aggregation choice.

A.2 RL for LLMs: GRPO and DAPO

GRPO adapts PPO for verifiable-reward LLM fine-tuning, while DAPO modifies clipping, aggregation, and sampling to change how tokens contribute to training.

  • GRPO generates multiple completions per prompt and normalizes their scalar rewards to emphasize relative quality within each group.
  • GRPO averages its objective across completions and tokens, preserving token-level policy ratios with group-normalized advantages.This design is intended to stabilize learning under sparse rewards.
  • DAPO replaces symmetric clipping with asymmetric clipping using distinct thresholds for upward and downward policy deviations.
  • DAPO aggregates loss at the prompt level so each token contributes equally regardless of sampled completion count or length.
  • DAPO dynamically drops zero-variance prompts and fills batches with additional prompts, although this change is omitted here because its effect resembles a larger batch.

A.3 Training Setup

The training setup uses curated reasoning and math/code data, fixed SFT and RL hyperparameters, and a generator–trainer GPU split for distributed RL updates.

  • Datasets: The RL stage primarily uses Polaris-53K, adding Deepcoder for experiments combining math and code.
  • Supervised Fine-tuning: SFT uses 2M-token batches, 12,288-token maximum sequences, a 3 × 10^-5 learning rate, AdamW, 32 H100 GPU nodes, approximately four epochs, and 32B total tokens.
  • Reinforcement Learning: RL allocates 14k generation tokens per sample, with 12k for thinking and 2k for the final answer, and uses 48 prompts with 16 generations each.
  • Reinforcement Learning: Each RL update therefore contains 768 completions, with rewards of +1 for correct traces and −1 for incorrect traces.
  • Reinforcement Learning: Automated math checkers evaluate final answers after removing thinking traces, while coding tasks use a custom execution environment with unit tests and desired outputs.
  • Reinforcement Learning: Experiments use 80 Nvidia GB200 GPUs, split into 64 generators and 16 trainers that exchange trajectories, perform updates, and broadcast parameters.

A.4 What curve to fit?

The paper fits RL performance with a sigmoid rather than a pre-training-style power law because RL metrics are bounded and training spans a short regime with saturation.

  • Power-law fits model performance as RC = A − D/C^B above a compute threshold C0, implying constant proportional gains from multiplicative compute increases.
  • Sigmoidal fits are preferred for bounded accuracy or reward metrics because they provide better predictive fits and support extrapolation from low compute.
  • The sigmoid captures slow low-compute growth, rapid mid-range improvement, and saturation toward a finite high-compute ceiling.
  • At high compute, the sigmoidal curve approaches the same form as a power law when C is much larger than Cmid.

A.5 Fitting scaling curves

The fitting framework reveals two-phase RL dynamics, identifies stability and asymptotic performance as central concerns, and links several design choices to predictable scaling or instability.

  • Fitting regime: The mean reward rises rapidly during approximately the first epoch, then follows sigmoidal behavior in the later training regime.The fits are applied to this latter portion rather than the initial rapid-growth phase.
  • Fitting regime: Very large runs are impractical for robust fitting because ablations are expensive and some compared algorithms saturate early or degrade.
  • Fitting regime: Stable design variants scale to approximately 5k–7k GPU hours, enabling leave-one-out experiments at approximately 1600 GPU hours per run.
  • Fitting regime: For the 100k GPU-hour 8B run, fitted asymptotes remain approximately A = 0.645–0.655 across fitting regimes, while B ranges from 1.56 to 1.70.
  • Interpreting parameters: B and Cmid primarily affect scaling efficiency, whereas A determines the asymptotic performance achievable at large compute.
  • Design choices: Replacing interruption with length penalty in the final ScaleRL recipe does not improve performance at 16k GPU-hours.
  • Design choices: The off-policy algorithm can shift the scaling-curve asymptote, making it one of the most consequential RL post-training design decisions.
  • Diagnostics: Entropy trajectories can remain nearly identical despite stronger downstream performance, so entropy is not necessarily a reliable predictor of downstream results.
Loading 2510.13786v1…