PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Yuante Li, Junwei Quan, Kai Qiu, Kate Zhang, Yicheng Tao

arXiv:2610.18853v1cs.LGcs.AI

TL;DR

Costly regenerated rollouts motivate larger batches, but the square-root learning-rate rule is less understood in practical multi-step RLVR than in one-step training. The paper develops a two-clock model and tests it in controlled GRPO experiments. Larger batches slow progress per sample while unclipped failure remains approximately sample-invariant, motivating a schedule that switches batches before failure.

  • Problem

    One-step GRPO results suggest square-root scaling can preserve progress per sample, but its behavior in practical multi-step RLVR is less understood despite the cost of regenerating policy rollouts.

    Source: §1 · p. 1

  • Method

    A two-clock framework combines a local quadratic progress model with a drift-threshold hypothesis for failure, and tests their predictions in controlled GRPO experiments.

    Source: §1 · p. 2

  • Results

    Larger batches learn more slowly per sample, yet without clipping both 512- and 4,096-response runs fail after 30 to 50k samples; clipping delays collapse more for the larger batch.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The two clocks suggest starting below the critical batch and switching to a larger batch before failure when throughput demands it.

    Source: §8 · p. 8

    Coverage is limited to one model family and one task domain, the theory is local, and response efficiency does not by itself capture token, compute, or wall-clock efficiency.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a $16\times$ batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it. Widening the clip trades stability for a transient speed gain. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model. Finally, we use the two-clock framework to derive a simple multi-step RLVR schedule: start below the critical batch and switch to a larger batch before failure. Our code is available at: https://github.com/WinstonLiyt/two-clocks-rlvr.

1 Introduction

Multi-step GRPO separates learning progress from failure timing: larger rollout batches learn more slowly per sample, while unclipped runs fail after similar sample counts. A two-clock model explains this split and motivates a batch-ramp schedule.

1 Introduction

  • Earlier one-step GRPO results found learning curves within about one accuracy point across 512-to-8K-response batches, but practical recipes split rollouts into multiple optimizer steps.DAPO, for example, uses 16 optimizer steps per rollout, allowing policy movement from the data-generating policy and activating ratio clipping after the first step.Source: §1 · pp. 1–2
  • The square-root rule preserves first-order drift per sample, not progress: fewer, longer updates lose more to curvature and create a learning-rate-dependent critical batch.A drift-threshold hypothesis predicts a complementary failure clock that is invariant in samples along the square-root rule.Source: §1 · p. 2
  • With the square-root rule, larger batches learn more slowly per sample, but unclipped 512- and 4,096-response runs both fail after 30 to 50k samples.Clipping delays collapse twofold for the small batch and fourfold for the large one, while the progress gap persists with or without clipping.Source: §1 · p. 2
  • The progress gap reappears on a second dataset, a second checkpoint, and a 7B model, while failure-law fits favor drift scaling over noise-driven alternatives.The experiments vary batch size, learning rate, clipping, model scale, and dataset.Source: §1 · p. 2
  • The proposed schedule starts below the critical batch and switches to a larger batch before failure; it approaches the best fixed-batch performance while generating three quarters of samples at the large batch.This schedule follows from separating the sample cost of progress from the sample count before collapse.Source: §1 · p. 2

2 Preliminaries

The preliminaries specify clipped GRPO, the rollout and optimizer batch relationship under square-root learning-rate scaling, and response-based measures of sample cost and failure.

2 Preliminaries

  • GRPO normalizes rule-based rewards within each response group into advantages and uses token-level importance ratios between the current and data-generating policies.The DAPO objective averages the clipped policy objective over tokens and uses a dual clip for negative advantages.Source: §2 · pp. 2–3
  • With clipping bounds (εlow, εhigh) = (0.2, 0.28) and dual clip c=10, a token's gradient contribution is zero when its ratio crosses the bound in the direction favored by its advantage.At the first optimizer step after a rollout, ratios equal one, so clipping can act only from the second step onward.Source: §2 · p. 3
  • A rollout batch Br contains P prompts with G responses each, is split into µ optimizer steps of size b = Br/µ, and uses learning rate η ∝ √Br at fixed µ.Adam's other hyperparameters stay fixed.Source: §2 · p. 3
  • Sample efficiency compares validation accuracy after equal numbers of consumed responses, and sample cost Nτ is the first response count reaching target accuracy τ.The sample-cost exponent α is zero under perfect invariance and one when doubling rollout batch doubles the samples required.Source: §2 · p. 3
  • The failure point Nfail is the consumed-response count at the first prespecified failure criterion; a run that does not fail within its budget is right-censored.Its training budget is then a lower bound on its failure point.Source: §2 · p. 3

3 Method: A Two-Clock Model of Batch Scaling

The two-clock model asks which per-sample quantities square-root scaling preserves in multi-step GRPO. It derives a curvature-limited progress clock and a drift-threshold failure clock, then predicts how clipping affects each.

3 Method: A Two-Clock Model of Batch Scaling

  • Under the square-root rule, the larger optimizer batch takes fewer, longer steps, preserving drift per consumed response while losing more progress to curvature.The analysis assumes noise-dominated normalization and a locally quadratic loss, and treats mini-batches within a rollout as fresh for progress.Source: §3 · p. 3
  • Progress per sample falls as batch grows because longer steps lose more to curvature, so reaching a target loss decrease requires more responses.The critical batch B* decreases as the learning rate assigned by the square-root rule grows and can lie below the gradient noise scale.Source: §3 · p. 4
  • The failure clock assumes collapse when accumulated drift reaches a threshold while the mean-gradient direction changes slowly; under the square-root rule, its sample count is batch-independent.This is a drift-threshold hypothesis, rather than a consequence of the progress model alone.Source: §3 · p. 4
  • Within a rollout, larger batches spread token log-ratios faster, so ratio clipping acts more strongly on them by zeroing contributions beyond the clip bound.Ratios reset at the next rollout, so clipping limits added drift rather than undoing accumulated drift, and it does not generate the curvature term behind the progress gap.Source: §3 · pp. 4–5
  • The model predicts that progress degrades beyond a critical batch, unclipped failure remains sample-invariant, and clipping delays failure most for larger batches without removing the progress gap.Lowering the learning rate is predicted to delay failure linearly under the drift law, rather than quadratically under a noise-driven 1/η^2 law.Source: §3 · p. 5

4 Experimental Setup

The experiments vary rollout batch size, learning rate, and clipping under controlled GRPO settings. They evaluate learning curves across batches and define when a run counts as failed.

Training setup

  • Runs use Qwen2.5-Math-1.5B with GRPO on DAPO-Math-17k, with a budget of 98,304 responses each.The setup disables KL penalty, entropy bonus, dynamic sampling, and gradient-norm clipping to isolate batch-size, learning-rate, and clipping effects.Source: §4 · p. 5

Experimental design

  • Four rollout batches are tested across a 16× range, with µ=8 optimizer steps per rollout unless stated otherwise.Learning rates follow the square-root rule; µ=4 and µ=16 runs are trained without clipping.Source: §4 · p. 5, Table 1 · p. 5

Learning-curve comparison

  • At ηS=5×10−6 and µ=8, the 1,536 batch closely tracks the 512 batch, while larger batches require progressively more samples.Figure 3 bands span two seeds where available.Source: Figure 3 · p. 5

Failure criterion

  • A run fails when accuracy falls below half its prior best, or when a sustained gradient-norm excursion meets the specified threshold.Runs meeting neither condition within the training budget are right-censored.Source: §4 · p. 5

5 Experimental Results

The results show that larger batches can lose sample efficiency beyond a learning-rate-dependent critical scale, while unclipped failure remains tied to consumed samples. Ratio clipping delays failure, especially for larger batches, but widening it trades away stability for only transient speed.

5.1 Progress slows above a critical batch (P1)

  • At µ=1, the large batch needs 20–45% more samples than the small batch to reach the same accuracy, and the gap persists without clipping.The progress gap also appears at µ=8 in the conservative regime, so it does not require the ratio clip.Source: §5.1 · p. 6
  • The sample-cost exponent rises from 0.15 to 0.44 as the learning rate increases, while the critical batch falls from about 9k to 2k responses.Increasing the batch from 512 to 1,536 costs little, but further increases become progressively less sample-efficient; at active rates, large batches can still finish higher because small batches decay or fail late.Source: §5.1 · p. 6
  • The positive sample-cost gap recurs on DeepScaleR-Preview, Qwen2.5-1.5B-Instruct, and Qwen2.5-Math-7B.The direction holds in every tested setting.Source: §5.1 · p. 6

5.2 Failure keeps time in samples (P2)

  • Without clipping, the 4,096 batch fails after 56–80 optimizer steps and the 512 batch after 648, yet both fail after 30–50k responses.Across 21 unclipped runs, failure along the square-root rule is approximately invariant in samples; the fitted drift form survives a likelihood-ratio test while the other candidate forms do not.Source: §5.2 · p. 6
  • Failure also keeps time in samples on Qwen2.5-Math-7B: the two batches are 6× apart in rollouts but only 1.3× apart in samples.With a norm cap of 1.0, the small batch fails after 36k samples, within its seeds’ 31–46k range.Source: §5.2 · p. 6

5.3 The clip buys stability rather than sustained speed (P3)

  • Clipping delays failure most for large batches, but widening the clip yields only a transient speed gain before performance decays and one of two seeds collapses.At twice the scaled rate, the large batch lasts four times longer with clipping than without; the small batch gains a factor of two. The overall trade favors stability over sustained speed.Source: §5.3 · p. 7

6 Implications

The two-clock framework motivates using the largest efficient batch below the critical scale, then switching to a larger batch before predicted failure when throughput requires it. The tested ramps outperform fixed large-batch curves and track the best fixed batch.

Below the critical batch

  • At the tested learning rates, 1,536 responses is the largest fixed batch below the critical regime and matches the small batch’s sample efficiency within seed noise.It does not fail and achieves the best final accuracy among the fixed batches.Source: §6 · pp. 7–8

Ramp results

  • The tested ramps lie well above fixed large-batch curves, track the best fixed batch, and generate three quarters of their samples with the large batch.Switches between 25k and 74k samples perform similarly; the failure law bounds how late the switch can come.Source: Figure 7 · p. 8, §6 · p. 8

Above the critical batch

  • When throughput requires a larger batch, begin with a small batch and switch before its predicted failure point.The optimizer state is reset and the one-rollout warm-up is repeated at the switch.Source: §6 · p. 8

7 Related Work

Prior work studies batch-size scaling, critical batches, collapse, and ratio clipping in RL optimization. This paper’s related-work framing distinguishes its multi-step GRPO setting and separates progress from failure in consumed samples.

Batch-size scaling and invariance

  • Prior scaling work reports comparable trajectories under linear SGD and square-root Adam rules, and batch-size invariance for PPO or one-step-per-rollout GRPO under specified conditions.The paper contrasts those settings with multi-step GRPO, where drift preservation does not imply progress preservation.Source: §7 · p. 8

Critical batch

  • Under tuned learning rates, the critical batch equals the gradient noise scale; under square-root scaling, it depends on the learning-rate scale.The paper connects this distinction to reports that larger batches can reach higher final accuracy while learning more slowly.Source: §7 · p. 8

Training collapse and ratio clipping

  • Prior collapse work links RLVR collapse to step size and asynchronous GRPO collapse time to roughly 1/η; this paper adds batch size and sample-count invariance along the square-root rule.Related clipping work has treated clipping as a bottleneck, whereas this paper finds it mainly buys stability at a transient speed cost.Source: §7 · p. 8

8 Conclusion

The square-root rule preserves first-order drift per sample, not progress per sample, producing separate progress and failure clocks. The framework supports training below the critical batch for efficient progress, then switching to a larger batch before failure when throughput demands it.

8 Conclusion

  • The two clocks separate the samples needed to reach a target from the approximately sample-invariant failure point under the square-root rule.The progress clock slows beyond a learning-rate-dependent critical batch, while the drift-threshold hypothesis predicts an approximately invariant failure point.Source: §8 · p. 8
  • Larger batches lose progress to curvature despite preserving first-order drift per sample.The square-root rule makes larger batches take fewer, longer updates; the progress gap persists even without ratio clipping.Source: §8 · p. 8
  • The proposed schedule starts below the critical batch and switches to a larger batch before failure when throughput demands it.Ratio clipping buys stability, especially for larger batches, but does not remove the progress gap.Source: §8 · p. 8

Limitations

The evaluation spans multiple checkpoints, two datasets, and a 7B model, but remains limited to one model family and one task domain. The theory is local, curvature is not directly measured, failure points vary across seeds, and response-count efficiency does not capture all costs.

Limitations

  • Compute and resource constraints limit evaluation to one model family and one task domain.The study broadens evaluation across multiple checkpoints, two datasets, and a 7B model.Source: Limitations · p. 9
  • The theory is intentionally local, and direct measurement of curvature is computationally expensive at this scale.The authors evaluate the theory through its testable predictions instead.Source: Limitations · p. 9
  • Failure points exhibit intrinsic variation across seeds.Source: Limitations · p. 9
  • Sample efficiency is measured in responses, while token, compute, and wall-clock efficiency also depend on generation lengths, hardware, and system implementation.Source: Limitations · p. 9

Ethical Considerations

The appendix provides derivations for progress, failure, ratio spread, clipping, and critical batches, alongside experimental details. Its analysis specifies the local quantities and assumptions used to connect batch size with progress and failure.

Ethical Considerations

  • The study releases no new model or dataset and analyzes RLVR on public math datasets with open-weight models.Its societal relevance is environmental: the schedule aims to reduce generated samples needed to reach a given accuracy, although the study required many training runs.Source: Ethical Considerations · p. 9

A.1 Progress per sample

  • The progress-rate derivation identifies a batch-growing curvature term, while the curvature-weighted noise scale yields a learning-rate-dependent critical batch under square-root scaling.With learning rate re-tuned at each batch, the critical batch instead equals the curvature-weighted noise scale and is learning-rate independent.Source: §A.1 · p. 11

A.2 Failure law

  • Under the drift-threshold assumption, the failure point is independent of rollout batch size, while samples to a target increase with batch size.The threshold model predicts scaling exponents distinct from noise-driven and batch-blind alternatives; the reported fit excludes the noise-driven alternative.Source: §A.2 · p. 12

A.3 Ratio spread

  • At matched optimizer steps, larger batches have greater token-level ratio spread; in active runs, clipped tokens carry 50–70% of the full gradient norm for L.The measured ratio-spread comparison and clipped-gradient share characterize how clipping acts across batches.Source: §A.3 · p. 13, §A.4 · p. 13

A.4 Share of the clipped tokens

  • In the active regime, clipping removes a dominant update component for the large batch: 0.5% of its mini-batch carries 50–70% of the full gradient norm.For the small batch, 0.05% of clipped tokens contribute 10–20% of the norm; in the conservative regime, the share is at most 0.14.Source: §A.4 · p. 13

A.5 Per-coordinate normalization and larger batches

  • With per-coordinate normalization, drift per sample remains independent of batch size, and the failure law retains its batch-scaling form.The measured optimizer batches are 60 to 2000 times below Bsimple, so the alternative regime near Bsimple does not explain the observed gaps.Source: §A.5 · p. 14

A.6 Implied critical batch

  • At µ=8, the implied critical batch falls from about 9.3k to 4.0k to 1.9k as ηS rises from 10−6 to 5×10−6 to 10−5.All three values lie well below Bsimple.Source: §A.6 · p. 14

B Experimental Details

  • The experimental design disables gradient-norm clipping because a fixed norm cap acts differently across batch sizes and interferes with the square-root rule.Runs use accuracy-first failure checks because some runs collapse while gradient norm remains at baseline.Source: B Experimental Details · p. 14

C Additional Results for the Progress Clock

Across accuracy targets, larger batches require more samples to make progress, with the gap widening as the learning rate rises and sometimes closing only when the small batch begins to fail. The appendix also examines failure-law fits, within-rollout clipping, replayed updates, and a widened clip radius.

C Additional Results for the Progress Clock

D Failure-Law Fits and Failure Points

  • Failure detection uses accuracy halving alongside gradient-norm onset, since some runs lose accuracy or collapse without a gradient-norm excursion.The accuracy criterion takes precedence and captures collapse followed by recovery as well as KL blow-up cases.Source: D Failure-Law Fits and Failure Points · pp. 16–17, Table 5 · p. 17
  • The drift-form failure law passes the likelihood-ratio test (p ≈0.3), while the random-walk and batch-blind forms fail (p < 10−4).The conclusion is based on aggregate scaling across 21 runs because individual failure points vary across seeds.Source: D Failure-Law Fits and Failure Points · p. 17, Table 4 · p. 17

E.1 Replay

  • Replayed updates show that at ηS=10−5, clipping rotates the large-batch update by about 0.2 from k=2 onward, 6–50 times the small batch’s rotation.Removing the clip gives dclip ≈2–3×10−5 at every step, treated as the measurement floor; a small-batch mini-batch with only 7×10−5 clipped tokens can have a gradient component comparable in norm to the full mini-batch gradient.Source: Figure 13 · p. 18, §E.1 · p. 18
  • The unclipped gradient signal is not resolvable against noise, with a simple noise scale above 6×10^4 responses at checkpoints after the first few steps.The implied critical batches lie well below this noise scale.Source: §E.1 · p. 18

E.2 Widened clip radius

  • Widening the large batch’s clip gives faster early learning, but both seeds then decay and one collapses.The widened radius halves the clipped fraction while allowing greater ratio drift; one seed ends at 0.078, below the plain large batch’s final 0.23 in the other seed.Source: §E.2 · p. 18, Figure 14 · p. 18

F Checkpoint Robustness

The progress gap also appears on Qwen2.5-1.5B-Instruct, where the small batch leads at both tested settings and reaches 8% accuracy much earlier at µ=8. In a batch-ramp sweep, switching from small to large batches at 25k–74k samples performs similarly, while the earliest switch gives up part of the small batch’s lead.

F Checkpoint Robustness

  • The small batch leads early at both µ=1 and µ=8 on Qwen2.5-1.5B-Instruct, despite this checkpoint training in a markedly different regime.At matched learning rate, its per-step KL is about ten times larger, within-rollout log-ratio spread 2.7 times larger, and entropy 0.5–0.8 rather than about 0.2.Source: F Checkpoint Robustness · p. 18, Figure 15 · p. 19
  • At µ=8, the small batch reaches 8% accuracy after 14k samples versus 86k for the large batch, and leads at every evaluation.Its α is 0.67; the large-batch run remains stable but is much slower.Source: F Checkpoint Robustness · p. 19, Figure 15 · p. 19

G Batch-Ramp Switch-Point Sweep

  • Switches from 25k to 74k samples perform similarly if they precede failure; the 12k switch gives up part of the small batch’s progress advantage.The sweep varies only the switch point along a common small-batch trajectory.Source: G Batch-Ramp Switch-Point Sweep · p. 19, Figure 16 · p. 19

Related papers

Loading 2610.18853v1…