PaperlayerView exact revision on arXiv

Source-linked AI summary

Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?

Bangji Yang, Hongbo Ma, Ruihan Guo, Jiajun Fan, Xi Zhu, Ge Liu

arXiv:2610.22887v2cs.LGcs.AIcs.CL

TL;DR

Long reasoning raises RLVR experience-collection cost, while truncating every single-problem rollout before an answer can eliminate group-relative accuracy advantages. CCR sequentially packs independently verifiable problems into one shared capped completion, averaging correctness while retaining credit for completed answers after suffix truncation. CCR reduces tokens per step by 82.1% on 4B and 79.1% on 8B.

  • Problem

    Long reasoning raises RLVR experience-collection cost, while truncating every single-problem rollout before an answer can eliminate group-relative accuracy advantages.

    Source: §1 · p. 1

  • Method

    CCR sequentially packs independently verifiable problems into one shared capped completion, averaging correctness while retaining credit for completed answers after suffix truncation.

    Source: §1 · p. 2

  • Results

    CCR reduces tokens per step by 82.1% on 4B and 79.1% on 8B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Increasing reward, shorter training completions, and retained single-problem reasoning quality support the conclusion that multiple verifiable answers can make short-window RLVR effective and efficient.

    Source: §7 · p. 10

    CCR can still starve if the window cannot finish the first answer or candidates complete the same number correctly; compression assumes shortening preserves correctness on the shared prefix.

    Source: §3.2 · p. 4, §3.3 · p. 5

Abstract

from arXiv · show

Long rollout windows make reinforcement learning with verifiable rewards (RLVR) expensive, yet simply shortening them can remove the correctness signal: when every single-problem rollout truncates before an answer, group-relative accuracy advantages vanish. As a technical continuation of Batched Contextual Reinforcement (BCR), we study how concurrent rewards relax this dependence on long windows. Concurrent Constraint Reinforcement (CCR) places three independently verifiable problems in one capped training completion. Correct answers completed before truncation retain credit, while a shared budget encourages shorter useful reasoning without an explicit length penalty. We connect this mechanism to conditional reward- and update-variance analyses and study 4B and 8B mathematical reasoning models. Against single-problem GRPO with a 32k window, CCR with an 8,000-token window reduces tokens per training step from 975,464 to 174,957 on 4B and from 793,832 to 165,788 on 8B, saving 82.1% and 79.1%, respectively. An 8B training trace shows increasing accuracy reward alongside shorter completions. Single-problem evaluations provide supporting evidence of retained reasoning quality. These results identify reward-preserving training structure as a practical route to lower per-step cost in short-window RLVR.

1 Introduction

Long rollout windows make RLVR costly, but shortening them can erase group-relative correctness feedback when every single-problem answer is truncated. CCR groups independently verifiable problems under a shared cap so completed answers retain credit, and the paper studies its stability and training efficiency.

1 Introduction

  • When every candidate in a single-problem group truncates before answering, relative accuracy advantages vanish, motivating a smaller-window RLVR training signal.Long reasoning raises the cost of collecting training experience.Source: §1 · p. 1
  • CCR sequentially packs independently verifiable problems into one capped completion, averaging correctness while preserving credit for completed answers despite suffix truncation.An unfinished suffix contributes zero without erasing earlier correct answers.Source: §1 · p. 2
  • The shared cap can encourage useful compression because unnecessary reasoning on one problem may displace another correct answer, without an explicit token penalty.The analysis gives conditional reward-noise and approximate update-variance bounds, but does not guarantee convergence for arbitrary windows.Source: §1 · p. 2
  • The study evaluates 4B and 8B models at fixed concurrency N = 3 using an 8,000-token CCR window against single-problem, 32k-window GRPO.Evidence includes per-step token cost, an 8B training trace, and ordinary single-problem evaluations.Source: §1 · p. 2
  • 82.1% fewer tokens per step on 4B and 79.1% fewer on 8B are reported for CCR, with a learning trace and single-problem evaluations supporting retained reasoning quality.The comparison is against single-problem GRPO with a 32k window; CCR uses an 8,000-token window.Source: §1 · p. 2

2 Rollout Windows and Reward Starvation

A rollout window caps generated tokens per completion, but does not determine how many tokens are actually used. In standard GRPO, groups with equal rewards cannot produce relative accuracy advantages, while CCR's grouped setup preserves credit for completed answers under truncation.

2 Rollout Windows and Reward Starvation

  • The rollout window caps each completion, including reasoning and answer tokens, while prompt tokens are outside the cap and total context capacity is distinct.The centered accuracy advantage is based on candidate completions for one problem; equal rewards still produce zero advantages after division by a stabilized group standard deviation.Source: §2 · p. 2
  • Standard GRPO assigns terminal correctness reward to one problem per candidate, whereas CCR attempts grouped problems sequentially under a shorter shared cap and retains credit for correct completed answers.CCR averages correctness over all assigned problems for the update; the figure's shorter ribbons illustrate token efficiency, not measured ratios.Source: Figure 1 · p. 3
  • Realized tokens per training step vary with early stopping, problem difficulty, and policy, so the window is an upper bound rather than consumed-token count.The standard GRPO comparison uses a 32k completion window.Source: §2 · p. 3

3 Concurrent Rewards for Short Windows

CCR generates ordered answers to three problems under one shared completion cap, with rewards computed across the assigned problems. Completed answers can preserve group-relative feedback after truncation, while the shared cap makes reasoning tokens costly when they displace other answers.

3.1 Training unit and reward

  • CCR samples joint candidates for the same problem group, verifies each recoverable answer, and computes within-group advantages from combined accuracy and format rewards.The reported experiments use three problems per completion; normalization includes unfinished problems.Source: §3.1 · pp. 3–4

3.2 Completed answers survive truncation

  • For N > 1, variation in correct completed-answer counts yields nonzero accuracy advantages even when every candidate has an unfinished suffix.The signal still vanishes if all candidates have the same reward; retaining it also requires a compatible truncation mask and policy-loss contribution.Source: §3.2 · pp. 4–5

3.3 Why the same constraint encourages compression

  • When concise trajectories complete more problems than verbose ones with equal success on their shared prefix, their expected accuracy reward is higher if an added problem has positive success probability.This does not reward brevity by itself: shortening must preserve correctness, and useful extra reasoning remains valuable when it does not displace another answer.Source: §3.3 · p. 5
  • Unlike an explicit token penalty, CCR makes reasoning costly when it crowds out another answer, favoring productive allocation under the hard shared budget.The shared budget encourages useful compression without requiring uniformly shorter responses.Source: §3.3 · p. 6

4 Stability Within the Feasible Regime

The analysis studies reward and update variance within a feasible shared-budget regime. Its guarantees depend on independence or approximation assumptions, and the number of problems must remain compatible with the token cap.

4.1 Reward noise and accuracy advantages

  • With conditionally independent correctness indicators, averaging across N problems gives the reward-noise term variance O(1/N) for fixed S.A shared cap and sequential generation can induce dependence, adding covariance that modifies this stability benefit.Source: §4.1 · p. 6
  • The linear signal-to-noise improvement SNR(a) = Θ(N) requires nonvanishing latent-quality variation across candidates and reward-noise variance of order 1/N.This stronger conclusion is conditional on those additional assumptions.Source: §4.1 · p. 6

4.2 Averaged updates and residual coupling

  • When the joint update is approximated by an average of problem-level updates, averaging reduces the non-shared variance component if directional correlation is below one.The shared component remains, and a shared budget alone does not ensure that residual coupling is small.Source: §4.2 · p. 7

4.3 Choosing a feasible shared budget

  • Fitting all N solutions requires a budget on the order of Nb when each solution needs roughly b tokens, so the study fixes N = 3 for two backbones.Actual lengths vary by problem and trajectory; increasing N can make the cap infeasible and destroy the intended signal.Source: §4.3 · p. 7

5 Experiments

With three problems sharing an 8,000-token cap, CCR uses substantially fewer tokens per training step than single-problem GRPO with a 32k window. The 8B training trace shows accuracy reward rising as completions shorten, while single-problem evaluations report higher average accuracy for CCR checkpoints.

5.1 Training and evaluation setup

  • The experiments train Qwen3-4B-Thinking and Qwen3-8B-ODA-Math with N = 3 on a reported 3,000-problem subset of DeepMath-103K.The setup uses an 8,000-token shared completion window, eight NVIDIA A100 GPUs, and evaluations on nine mathematical reasoning benchmarks.Source: §5.1 · p. 7

5.2 Training-token efficiency at 4B and 8B

  • CCR reduces tokens per training step from 975,464 to 174,957 on 4B (82.1%) and from 793,832 to 165,788 on 8B (79.1%).The comparison is single-problem GRPO with a 32k window versus CCR with three problems and an 8,000-token shared window; both use eight A100 GPUs and matched remaining training parameters.Source: §5.2 · pp. 7–8, Figure 2 · p. 8

5.3 Reward improves while completions shorten

  • In the 8B CCR run, accuracy reward rises from 0.142 to 0.509 as mean joint-completion length falls from 6,867 to 5,034 tokens.These are first-50 versus last-50-step averages; the completion cap stays fixed, and the shortening reflects changes in generated trajectories.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 Reasoning quality after short-window training

  • Average single-problem evaluation accuracy increases from 75.3% to 77.7% on 4B and from 68.8% to 69.7% on 8B, with variation across benchmarks.Table 2 evaluates CCR checkpoints trained with N = 3 on single-problem prompts; it reports final-answer correctness and evaluation lengths.Source: §5.4 · pp. 8–9, Table 2 · p. 9

6 Related Work

Related work spans RLVR reward and sampling design, grouped reasoning, inference-time compute, and alternative feedback or evaluation regimes. CCR’s distinction is to vary how many independently verifiable answers fit within a trajectory.

RLVR and rollout design

  • Related RLVR approaches address long-chain training, verifier noise, failed rollouts, and sharing verified trajectories across agents [7], [3], [8], [9].GRPO uses relative rewards among candidate completions [1], while DeepSeek-R1 demonstrates a broader role for verifiable RL in eliciting reasoning [2].Source: §6 · p. 9
  • CCR changes how many independently verifiable answers are available within a trajectory, so an unfinished suffix need not eliminate all accuracy feedback.Prior batched contextual reinforcement introduced multiple problems in a shared context; this paper focuses on the coupling among completion caps, reward availability, and measured training-step cost.Source: §6 · p. 10

Reasoning and inference compute

  • Inference-time compute and reasoning-efficiency work shape individual prediction trajectories, whereas CCR targets repeated on-policy generation during training [10] [11], [12], [13],, [15] [16].Shorter evaluation traces are supporting observations; the primary measurements concern training steps under a restricted rollout window.Source: §6 · p. 10

Other feedback and evaluation settings

  • Preference optimization, positive-unlabeled RL distillation, and arena-ranking stability address feedback or evaluation regimes different from CCR’s verified multi-problem rewards [17], [18], [19].Source: §6 · p. 10

7 Conclusion

BCR II develops the training-efficiency perspective of batched contextual reinforcement through CCR, which preserves credit for completed answers and rewards productive use of a shared token budget. With N = 3 and an 8,000-token window, it uses fewer tokens per step than single-problem, 32k-window GRPO.

7 Conclusion

  • CCR uses 82.1% fewer tokens per step on 4B and 79.1% fewer on 8B than single-problem, 32k-window GRPO, with N = 3 and an 8,000-token window.Source: §7 · p. 10
  • CCR preserves credit for completed answers under suffix truncation and rewards productive use of the shared token budget.Source: §7 · p. 10
  • Increasing reward, shorter training completions, and retained single-problem reasoning quality support the conclusion that multi-answer rollout structure can make short-window RLVR effective and efficient.Source: §7 · p. 10

A Derivations and Assumptions

The derivations connect prefix rewards to group-relative feedback, reward averaging to conditional stability, and update averaging to assumptions about cross-problem terms. Their bounds depend on reward diversity, independence, and small residual contributions.

A.1 Prefix rewards and opportunity cost

  • When candidates have different counts of correct completed answers, centered accuracy advantages cannot all be zero; the opportunity-cost comparison counts added and displaced correct answers.The expected gain is the added success probability divided by N, less the expected correctness of each displaced answer divided by N.Source: §A.1 · p. 13

A.2 Reward variance and centering

  • Under conditional independence, reward noise is O(1/N) for fixed S ≥ 2, but correlated problem outcomes can add a variance term and prevent that decay.The argument also requires a nonzero latent correctness signal; if all answers truncate and rewards coincide, it does not apply.Source: §A.2 · p. 13

A.3 Update averaging and cross-problem terms

  • The update-variance comparison isolates an averaged accuracy-gradient component and assumes residual contributions are small; sequential conditioning and a shared budget may violate that assumption.Format-gradient effects, normalization, and clipping enter through the full GRPO update.Source: §A.3 · p. 14

B Experimental Configuration and Log Processing

Table 3 summarizes the comparison settings, while the log-processing procedure aligns training observations by global step and derives summaries from steps 1–1,000.

B Experimental Configuration and Log Processing

C Training Procedure and Reward Interface

CCR trains on ordered groups of independently verifiable problems under a shared generation cap, preserving credit for completed answers while defining reward and update handling for truncated responses.

C Training Procedure and Reward Interface

  • Each training prompt contains an ordered group of three independently verifiable problems, and candidates generate an ordered sequence of solutions under a shared 8,000-token cap.Source: C Training Procedure and Reward… · p. 16
  • CCR assigns zero correctness to missing or incorrect answers but retains earlier valid answers when a candidate's final suffix is unfinished.Accuracy is averaged over the fixed number of assigned problems, then combined with a format term using the reported reward weights.Source: C Training Procedure and Reward… · p. 16
  • Group-relative advantages are computed among candidates for the same problem group and used in the configured GRPO loss.Loss masking must retain the policy contribution associated with the reward-bearing prefix of a truncated trajectory.Source: C Training Procedure and Reward… · p. 16
  • A separate answer after each problem, marked with distinct identifiers, provides the interface for recovering each answer.Format reward and accuracy reward serve distinct roles; gating all answers on whole-response formatting would discard earlier correctness feedback.Source: C Training Procedure and Reward… · p. 16

D Additional Reward and Budget Derivations

The additional derivations distinguish reward noise, token-cost mechanisms, and conditions for informative feedback; qualitative trajectories illustrate how CCR solutions differ from base-checkpoint traces.

D.1 Heterogeneous answer probabilities

  • The heterogeneous-probability analysis separates Bernoulli observation noise from variation in mean trajectory quality and explains why the 1/(4N) bound is loose for near-certain outcomes.Source: §D.1 · p. 17

D.2 Explicit length penalties versus a shared cap

  • An explicit length penalty assigns a prescribed cost to every added token, whereas a shared cap assigns opportunity cost through correctness displaced from later answers.The local-edit comparison assumes no other effects on remaining solutions; shared autoregressive context can violate that simplification.Source: §D.2 · p. 17

D.3 Feasible completion and conditional signal

  • Nonzero accuracy advantages can arise even when no candidate completes all problems, provided candidate completed-answer counts vary; if all candidates finish none, informative accuracy feedback is unavailable.Ordering matters because an unusually long first solution can block later answers, and fixed B/N is only a rough allocation heuristic.Source: §D.3 · p. 17

E.1 Torus–Sphere Tangency (AIME 2024); E.2 Date Digit Parity (AMC 2023)

  • In the torus–sphere example, CCR compresses the derivation and uses the tangency-circle formulas, while the base trace ends with an incorrect value; for date parity, both reach 9, with CCR using month-wise parity vectors.Source: §E.1 · p. 18, §E.2 · p. 18

E.3 Party Graph with Enemy Closure

  • For the enemy-closure problem, the base trace misses intermediate possibilities, while CCR lists divisor cases and sums the possible values to 100.Source: §E.3 · p. 19

E.4 Vector Projection (MATH-500); E.5 Polytropic Atmosphere Hydrostatic Equilibrium (Minerva)

  • In the vector-projection and atmosphere examples, CCR selects the vector-valued interpretation and writes the hydrostatic equation with K^-3/5, unlike the base trace's scalar interpretation and mishandled K factor.Source: §E.4 · p. 19, §E.5 · p. 20

E.6 Three Non-overlapping Squares; E.7 Sphere Tangent to Six Tetrahedra; E.8 Trigonometric Cubic

  • Across the squares, tetrahedra, and trigonometric-cubic examples, CCR identifies the shared boundary, chooses the closest tangent edge, and obtains 567 from the root triple rather than a numerical guess.Source: §E.6 · p. 20, §E.7 · p. 21, §E.8 · p. 21

E.9 Product of Factorials

  • For the factorial problem, the base trace reaches its token limit without an answer, whereas CCR verifies the constraints and computes abcd = 2240.Source: §E.9 · p. 22
Loading 2610.22887v2…