PaperlayerView exact revision on arXiv

Source-linked AI summary

Double-Rollout Reinforcement Learning with Sampling-based Reward Estimation

Haifeng Qian, David Mosallanezhad, Christian Munley, Adithya Renduchintala, Oleksii Kuchaiev

arXiv:2610.16593v1cs.AI

TL;DR

Outcome rewards can misrepresent reasoning quality when final answers and their traces disagree, creating a credit-assignment challenge for reasoning RL. The paper estimates trace quality by averaging rewards from sampled continuations, and reports superior accuracies and accuracy-efficiency trade-offs across math and coding tasks and model families.

  • Problem

    A correct final answer may follow an inconclusive trace, while an incorrect rollout may contain reasoning on the right path, making outcome rewards an imperfect measure of reasoning quality.

    Source: §1 · p. 1

  • Method

    Double-rollout RL truncates each rollout’s reasoning trace, samples multiple continuations, and replaces the original reward with their average verifiable reward.

    Source: §1 · pp. 1–2

  • Results

    The method achieves superior accuracies and accuracy-efficiency trade-offs across math and coding tasks and model families, including 95.4% accuracy on AIME 2026.

    Source: §3 · p. 4, §5 · p. 6

  • Takeaways & Limitations

    The authors conclude that double-rollout offers an affordable-complexity approach to addressing credit assignment in long reasoning tasks.

    Source: §5 · p. 6

    Experiments were capped at 80 RL steps because of resource constraints.

    Source: §3 · p. 5

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) implemented with Group Relative Policy Optimization (GRPO) has driven substantial progress in training reasoning language models. However, verifiable outcome rewards provide an imperfect measure of reasoning quality: a correct final answer may not be supported by the reasoning trace, whereas an incorrect rollout may contain a nearly correct reasoning trajectory. We present empirical evidence for this mismatch and hypothesize that the resulting reward noise contributes to instability in reinforcement learning for both reasoning accuracy and token efficiency. To address this credit assignment challenge, this paper proposes double-rollout reinforcement learning. For each sampled rollout, we truncate its reasoning trace, sample multiple continuations from the truncated prefix, and replace the original reward with the mean continuation reward, yielding a more robust estimate of reasoning quality. This double-rollout algorithm incurs only a moderate increase in compute cost and yet reaches substantially higher accuracies across math and coding tasks and across Qwen and Nemotron models than baseline algorithms with fewer RL steps. For example, using only DAPO data it trains Nemotron-3-Nano-30B-A3B to reach 95.4% on AIME 2026, versus 91.5% by GRPO and substantially above state of the art for this model size, within 30 RL steps. When combined with length-based rewards, the same algorithm further improves the accuracy-efficiency Pareto frontier over existing efficiency optimizations. These findings suggest that the double-rollout reward estimation is an effective and cost-efficient way to improve both the accuracy and token efficiency of reasoning RL.

1 Introduction

Outcome verifiers can reward or penalize reasoning traces whose quality does not match the final answer, creating a credit-assignment challenge that is especially relevant to token-efficiency training. The paper addresses this by estimating rollout quality from sampled continuations while retaining a simple GRPO-style approach.

1 Introduction

  • A correct final answer can rest on an inconclusive trace, while an incorrect answer can follow reasoning on the right path, so outcome rewards can misassign credit.The resulting updates may reinforce questionable traces or suppress promising reasoning.Source: §1 · p. 1
  • Shorter rollouts have less certain quality, which the authors hypothesize helps explain undesirable trade-offs in existing token-efficiency methods.Rewarding a short correct rollout may reinforce a subpar trace when its answer was correct by chance.Source: §1 · p. 1
  • The method estimates a partial sequence’s value by sampling multiple continuations and averaging their verifiable rewards.For example, truncating at 90% of the thinking trace and sampling eight continuations approximately doubles rollout compute.Source: §1 · pp. 1–2
  • 95.4% accuracy on AIME 2026 was reached by Nemotron-3-Nano-30B-A3B using only DAPO data, while the method outperformed baseline GRPO across Qwen and Nemotron models and math and coding tasks.The authors also report superior accuracy-efficiency trade-offs with length-based rewards and fewer RL steps to best accuracy.Source: §1 · p. 2
  • The method is described as applicable to RLHF on reasoning LLMs, but applying it there is left to future work.The authors identify this as future work.Source: §1 · p. 2

2 Double-Rollout RL

The empirical study shows that continuation correctness can diverge from the original rollout’s correctness, with larger discrepancies for shorter traces. The proposed algorithm uses sampled continuations to revise rewards and selectively replaces some positively advantaged incorrect rollouts; its token-efficiency variant adds rewards for short, reliably correct rollouts.

An Empirical Study on the Reliability of Rewards

  • 74% was the average continuation pass rate for short correct rollouts on hard DAPO problems, illustrating that a correct answer may follow a low-quality trace.The study truncates rollouts at 90% of their thinking traces and samples continuations, grouping results by original correctness, difficulty, and rollout length.Source: An Empirical Study on the Reliability… · p. 2, Figure 1 · p. 2
  • A substantial fraction of incorrect rollouts contain traces on the right path, so outcome-based updates can push down the probability of good reasoning tokens.The study presents these continuation results separately for originally incorrect rollouts.Source: An Empirical Study on the Reliability… · p. 2, Figure 2 · p. 3
  • Across all six plots, shorter original rollouts show larger discrepancies between original correctness and average continuation correctness.The authors hypothesize that this uncertainty contributes to undesirable trade-offs in token-efficiency methods.Source: An Empirical Study on the Reliability… · p. 2

Double-Rollout Algorithm for Max Accuracy

  • The max-accuracy algorithm replaces each rollout’s original reward with the average reward of continuations sampled from its truncated trace.This extends the spirit of GRPO without adding a process reward model.Source: Double-Rollout Algorithm for Max… · pp. 2–3
  • For a positively advantaged rollout with an incorrect final answer, the algorithm replaces it with a correct continuation, while retaining punishment for correct answers based on inconclusive traces.The authors say correcting the former mismatch helps RL results, whereas correcting the latter is unnecessary because premature guessing should be discouraged.Source: Double-Rollout Algorithm for Max… · p. 3
  • Truncating at 90% of the thinking trace and sampling eight continuations approximately doubles rollout compute, with a token cap set to limit costly long tails.The cap is 10% of the original thinking-trace length plus a task-dependent buffer; the authors report 1K for math and 4K for coding.Source: Double-Rollout Algorithm for Max… · p. 3

Double-Rollout Algorithm for Token Efficiency

  • The token-efficiency variant applies an upward reward revision to short, reliably correct rollouts rather than a length-based penalty.It calculates the length mean and standard deviation among positively advantaged rollouts after reward revisions, and scales the added reward by the rollout’s length z-score.Source: Double-Rollout Algorithm for Token… · p. 3

3 Experiments

Across four settings, double-rollout achieves the best accuracies, including 95.4% on AIME 2026 for a 30B MoE model. It also improves the accuracy-efficiency frontier over the range tested.

3 Experiments

  • 95.4% accuracy on AIME 2026 was reached by the 30B MoE model using only outdated training data.Across all four settings, double-rollout reaches the best accuracies.Source: Table 1 · p. 4, §3 · p. 4
  • Experiments cover four settings combining two starting models and two training datasets; GRPO with large rollouts costs eight times as much per step as vanilla GRPO.The tested settings use Nemotron and Qwen models with DAPO math data and OpenCodeReasoning coding data.Source: §3 · pp. 3–4
  • Below 50% trace retention, reward estimates become noisy and RL degrades the model; 50% and 75% are not far worse than 90%.The authors identify 90% as the best choice in this table, noting that higher retention saves second-stage sampling compute.Source: Table 2 · p. 4, §3 · p. 5
  • Sampling two continuations already brings substantial accuracy benefit at low extra compute cost, while eight is described as a reasonable middle ground.The number of continuations trades off compute cost per RL step against reward-estimation precision.Source: Table 3 · p. 4, §3 · p. 5
  • Peak accuracy took 30, 18, 16, and 30 RL steps with double-rollout, versus 30, 45, 44, and 45 steps with GRPO across the four settings.The authors hypothesize higher-quality training data per step improves data efficiency and helps offset extra per-step compute.Source: §3 · p. 4
  • Double-rollout dominates the accuracy-efficiency frontier over the range covered in DAPO-trained Nemotron experiments.Figure 3 plots average tokens per AIME problem against mean accuracy on AIME 2024, 2025, and 2026.Source: Figure 3 · p. 5, §3 · p. 5
  • All experiments were capped at 80 steps because of resource constraints, leaving longer double-rollout runs with length-based rewards for future work.The authors identify those runs as a possible way to explore further left in the trade-off space.Source: §3 · p. 5

4 Related Work

Related work addresses credit assignment and token efficiency through diverse methods, but the cited approaches face compute or accuracy-efficiency trade-offs. The paper positions double-rollout as a practical alternative for long reasoning tasks.

4 Related Work

  • GRPO and its variants are the mainstream approach for reasoning LLMs, while several proposed variants target different aspects of the algorithm.The baseline GRPO implementation used in Section 3 already includes some variants.Source: §4 · p. 5
  • Prior credit-assignment methods include Monte Carlo value estimation, tree-based sample reuse, and PRM-based approaches, with reported constraints on long-task compute or accuracy.VinePPO [Kazemnejad et al. 2024] and SPO are described as limited to short tasks; PURE reports fairly low math accuracy.Source: §4 · p. 6
  • Length-based rewards, penalties, and token budgets are common efficiency techniques, but some report undesirable accuracy-efficiency trade-offs.The passage cites length overfitting under token budgets and model merging to mitigate accuracy loss from length penalties.Source: §4 · p. 6

5 Conclusion

The paper proposes double-rollout reinforcement learning as a practical middle ground for credit assignment in long reasoning tasks. Experiments across math and coding tasks and model families report higher accuracy and better accuracy-efficiency trade-offs.

5 Conclusion

  • Double-rollout reinforcement learning targets credit-assignment challenges while maintaining affordable computational complexity on long reasoning tasks.The conclusion frames the algorithm as a practical middle ground between addressing detrimental credit-assignment symptoms and computational cost.Source: §5 · p. 6
  • Empirical results span math and coding reasoning tasks and multiple model families.The reported results demonstrate superior accuracies and accuracy-efficiency trade-offs.Source: §5 · p. 6
  • The paper concludes that the method achieves superior accuracy-efficiency trade-offs alongside superior accuracies.These are the conclusion's reported empirical outcomes across the evaluated tasks and model families.Source: §5 · p. 6

Related papers

Loading 2610.16593v1…