PaperlayerView exact revision on arXiv

Source-linked AI summary

Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning

Rohan Surana, Gagan Mundada, Xunyi Jiang, Chuhan Wang, Zhenwei Tang, Difan Jiao, Zihan Huang, Yuxin Xiong, Junda Wu, Sheldon Yu, Xintong Li, Raghav Jain, Nikki Kuang, Sizhe Zhou, Bowen Jin, Zhendong Chu, Tong Yu, Ryan Rossi, Kuan-Hao Huang, Jingbo Shang, Jiawei Han, Julian McAuley

arXiv:2605.02913v1cs.LG

TL;DR

Rollout design is central to the data and learning signals in LLM reinforcement learning, yet it is often underreported and lacks shared comparison tools. This survey introduces GFCR to organize rollout pipelines and synthesizes methods, criteria, case studies, diagnostics, and open challenges across reasoning and agentic settings. GFCR makes rollout choices explicit and comparable across these settings.

  • Problem

    Rollout strategies are often underreported and lack shared vocabulary, making it difficult to compare methods or isolate components associated with reported gains.

  • Method

    The survey formalizes rollout pipelines with unified notation and introduces GFCR, an optimizer-agnostic taxonomy covering Generate, Filter, Control, and Replay.

  • Results

    The survey synthesizes rollout methods across verifiable rewards, process supervision, guided search, adaptive compute, throughput optimization, replay, and self-evolution.

  • Takeaways & Limitations

    GFCR makes rollout choices explicit and comparable while supporting diagnostic analysis of recurring failure modes and mitigation levers across reasoning and agentic settings.

  • Takeaways & Limitations

    Many methods omit comparable accounting for tool calls, verifier runtime, branch/prune overhead, and replay refresh.

Abstract

from arXiv · show

Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs). In these systems, the rollout, the trajectory sampled from a prompt to termination, including intermediate reasoning steps and optional tool or environment interactions, determines the data the optimizer learns from, yet rollout design is often underreported. This survey provides an optimizer-agnostic view of rollout strategies for RL-based post-training of reasoning LLMs. We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR), a lifecycle taxonomy that decomposes rollout pipelines into four modular stages: Generate proposes candidate trajectories and topologies; Filter constructs intermediate signals via verifiers, judges, critics; Control allocates compute and makes continuation/branching/stopping decisions under budgets; and Replay retains and reuses artifacts across rollouts without weight updates, including self-evolving curricula that autonomously generate new training tasks. We complement GFCR with a criterion taxonomy of reliability, coverage, and cost sensitivity that characterizes rollout trade-offs. Using this framework, we synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement. We ground the framework with case studies in math, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks that evaluate skill induction, reuse, and cross-task transfer. Finally, we provide a diagnostic index that maps common rollout pathologies to GFCR modules and mitigation levers, alongside open challenges for building reproducible, compute-efficient, and trustworthy rollout pipelines.

1 Introduction

This survey treats rollout design as a first-class, optimizer-agnostic object in RL post-training for reasoning LLMs, because rollout-generated data shapes learning quality and training cost. It introduces GFCR and complementary criteria to organize methods, case studies, diagnostics, and open challenges for reproducible rollout pipelines.

  • Motivation: Rollout-generated data strongly affects both training cost and learning-signal quality, yet rollout configurations are often underreported, confounding comparisons and optimizer attribution.Optimizers can only learn from the data supplied by online model-generated rollouts.
  • Framework: GFCR decomposes rollout pipelines into Generate, Filter, Control, and Replay stages that propose trajectories, construct signals, allocate compute, and retain reusable artifacts.The stages are modular and composable but may interleave through feedback, control decisions, and replay-seeded generation.
  • Framework: The survey pairs GFCR with reliability, coverage, and cost sensitivity to compare trajectory proposal, filtering, compute control, and reuse independently of the optimizer.It presents this organization as a unified taxonomy for rollout pipelines.
  • Survey Scope: It synthesizes methods spanning verifiable rewards, process supervision, judge gating, guided and tree/segment rollouts, adaptive compute, partial rollouts, throughput optimization, and replay-based self-improvement.The surveyed space includes early-exit strategies and replay, recomposition, and self-evolution.
  • Evaluation and Practice: Case studies cover mathematics, code/SQL, multimodal reasoning, tool-using agents, and agentic skill benchmarks, while diagnostics map rollout pathologies to GFCR mitigations.The benchmarks include skill induction, library management, and cross-task transfer; open challenges include calibration, compute accounting, safe self-evolution, and provenance tracking.

2 Related Work

Prior surveys emphasize feedback modeling, preference learning, alignment, evaluation, RL families, and optimization mechanics, while leaving rollout strategy largely implicit. This section situates rollout design as a distinct focus involving trajectory sampling, structuring, scoring, adaptive stopping, tool interaction, and replay under compute budgets.

  • Rollout focus: Rollout-focused concerns include grouped sampling, branching search, tool interactions, stepwise or terminal scoring, adaptive stopping, and replay under compute budgets.
  • Related surveys: Prior surveys organize RL post-training mainly around feedback modeling, reward learning, and optimization objectives, leaving rollout strategy implicit.
  • Related surveys: Surveys of RLHF and preference learning emphasize feedback collection, alignment loops, and evaluation protocols.
  • Related surveys: Technical surveys cover RLHF, RLAIF, direct preference methods, RL algorithms, and training mechanics for LLM fine-tuning.

3 Foundations: Rollouts, Criteria, and the GFCR Framework

This section frames rollout-centric RL as an end-to-end pipeline and introduces GFCR, which decomposes rollout behavior into Generate, Filter, Control, and Replay modules. It also formalizes trajectories, prefixes, group-based supervision, compute budgets, and the pipeline-induced distribution used to analyze these systems.

  • Motivation: Rollout strategies span trajectory proposal, scoring, compute allocation, and artifact retention rather than a single algorithmic choice.Modern post-training systems generate candidate trajectories, evaluate them with verifiers or judges, and convert the resulting signals into training supervision.
  • GFCR Framework: GFCR decomposes rollout pipelines into Generate, Filter, Control, and Replay: proposing trajectories, constructing supervision, allocating compute, and reusing artifacts.Replay retains and reuses artifacts across rollouts without weight updates.
  • GFCR Framework: GFCR modules interleave: Filter signals can trigger pruning or stopping, Replay artifacts can seed generation, and Control governs storage and reuse.Control also determines when replayed artifacts are trusted, refreshed, or discarded.
  • Notation: Rollouts represent interaction trajectories with actions, observations, and a stopping time, while prefixes support partial-rollout scoring, pruning, and tree expansion.In text-only settings, observations are empty and a rollout reduces to a completion y ≡u1:T.
  • Notation: Training systems sample individual or grouped rollouts, assign intermediate Filter signals and training supervision, and track costs under compute budgets.Signals may include validity, verification outcomes, process scores, judge ranks, and learning-value signals, including group-dependent supervision.
  • Pipeline-Induced Distributions: The full GFCR pipeline induces qθ,GFCR(T | x, B), which can differ from pθ(τ | x) through filtering, budget decisions, adaptive sampling, and persistent replay state.The replay state B stores and reuses artifacts from past rollouts, making the pipeline distribution history-dependent.

4 Generate: How Trajectories Are Proposed

The Generate module proposes candidate rollout sets through a distribution shaped by guidance, sampling configuration, and rollout topology. Its design space spans linear, group, tree/graph, and multi-turn/tool rollouts, with structured topologies enabling branching, prefix reuse, and interaction-dependent trajectories.

  • Generate formalization: Generate maps a prompt to a candidate rollout set that is passed to Filter and shaped by subsequent Control decisions.The proposal distribution depends on model parameters, rollout-time guidance z, sampling configuration κG, and topology Topo.
  • Tree/Graph Rollouts: Tree/graph rollouts branch from partial histories, reuse shared prefixes, and use terminal-node feedback to support step-level supervision and budget allocation.Expansion and pruning determine how rollout budgets are allocated, while shared prefixes amortize computation.
  • Multi-Turn, Tool-Using, and Multi-Agent Rollouts: Multi-turn and tool-using rollouts place the policy in an action–observation loop whose trajectories include actions, observations, and state carried across turns.Learning signals are often sparse and delayed, motivating environments with verifiable endpoints or other practical reward mechanisms.
  • Generate design space: The design space varies topology and interaction from single and group candidates to tree/graph and multi-turn/tool rollouts.Guidance includes exemplars, rubrics, plans, critiques, repairs, tool traces, and retrieved examples; sampling includes temperature, top-p, and diversity controls.
  • Group Rollouts: Group rollouts sample K independent candidates and compute normalized advantages from within-group reward mean and standard deviation, reducing variance without a learned critic.The normalization is Â_i = (r_i − μ_G)/σ_G, where μ_G and σ_G are the group reward mean and standard deviation.

5 Filter: From Rollouts to Learning Signals

Filter converts sampled rollout candidates into intermediate signals and optimizer-facing supervision for pruning, selection, credit assignment, and compute allocation. It can score complete trajectories or prefixes using validity, verifier, process, judge, and learning-value signals.

  • Filter maps sampled candidates into intermediate signals and optimizer-facing supervision for pruning, selection, credit assignment, and compute allocation.
  • Filtering can operate on complete rollouts or prefixes, enabling step- or segment-level scoring during long traces and tree expansion.
  • Signals may include validity indicators, verifier outcomes, process or step scores, judge preferences or ranks, disagreement, entropy, and novelty.
  • A useful decomposition separates filtering into gating, semantic evaluation, and mapping outputs into training targets, potentially using the entire candidate group.

SIGNAL CONSTRUCTION

Filter converts sampled rollouts into optimizer-facing supervision through structural gating, evaluation, and mapping. Signals range from verifier-based correctness to dense process scores and comparative judgments, but dense-signal quality requires direct calibration against reasoning errors.

  • Signal pipeline: Filter transforms rollouts into optimizer-facing weights, shaped rewards, advantages, or preference labels through gating, evaluation, and mapping.Mappings can include masking or reweighting, within-group normalization, abstention or tie handling, and aggregation of per-step scores.
  • Structural gating: Structural gates prevent malformed, unsafe, unparsable, or non-executable rollouts from being misclassified and injecting systematic noise into training.They may require delimiters, single-answer formats, valid tool calls, or executable programs and queries before semantic scoring.
  • Semantic verification: Hard semantic gates use task-grounded pass/fail or bounded scalar signals from deterministic checkers, including unit tests, SQL execution, and normalized exact-answer verification.This verifier-centered paradigm underlies RLVR and is intended to be relatively objective, cheap, and difficult to exploit.
  • Process scoring: Dense process scores evaluate steps, segments, or nodes, enabling early pruning, partial credit, credit assignment, branch continuation decisions, and advantage shaping.Scores may be aggregated using minimum or first failure, discounted sums, or calibrated averages.
  • Process-score evaluation: ProcessBench directly evaluates earliest-error identification and calibration, exposing the risk that process scores track surface heuristics rather than reasoning quality.This shifts process-reward evaluation beyond indirect end-task improvements such as best-of-K reranking.

6 Control: Compute Allocation, Decision Rules, and On/Off-Policy Knobs · 6.1 Problem Formulation: Control (C) — Compute Allocation and Decision Rules

Control allocates rollout compute and selects continuation, branching, pruning, configuration, and data-mixing decisions under per-prompt or global budgets. It uses Filter signals and cost accounting to shape the realized rollout distribution while targeting learning utility per unit compute.

  • 6 Control: Compute Allocation, Decision Rules, and On/Off-Policy Knobs: Control decides which prompts to roll out, how many candidates to generate, when to stop or continue, when to branch or prune, and how to configure rollouts.Configuration includes maximum length, temperature, and concision.
  • 6.1 Problem Formulation: Control (C) — Compute Allocation and Decision Rules: Control operates as a budgeted sequential decision process over evolving partial-rollout frontiers, observing Filter signals, accumulated cost, and remaining budget.Signals include step scores, validity flags, and verifier outcomes.
  • 6 Control: Compute Allocation, Decision Rules, and On/Off-Policy Knobs: Control actions include prompt selection, adaptive sampling, continuation or early-exit rules, selective continuation, and branch or subtree pruning.Adaptive sampling can choose candidate count, tree depth or width, or selective rollout.
  • 6.1 Problem Formulation: Control (C) — Compute Allocation and Decision Rules: Control also sets rollout length, temperature, concision, and sampling or selection rules that alter the mix of positive and informative-negative examples.These configuration choices are part of the action space rather than separate pipeline stages.
  • 6 Control: Compute Allocation, Decision Rules, and On/Off-Policy Knobs: At the pipeline level, Control allocates compute by deciding which rollouts are completed or abandoned to maximize learning utility per unit compute.Utility can summarize usable-sample mass, expected signal strength, or other signal-quality surrogates.
  • 6.1 Problem Formulation: Control (C) — Compute Allocation and Decision Rules: Control includes on/off-policy knobs that mix fresh on-policy rollouts with reused or off-policy data, often through Replay.It also encompasses systems-level scheduling and token-accounting decisions related to throughput.

6.2 Prompt and Task Selection (What Do You Roll Out On?)

Prompt and task selection is a consequential rollout design choice because uniform sampling can waste computation on prompts that provide little useful training signal. Recent methods target informativeness through reward-variance prediction, uncertainty-driven sampling, semantic entropy, and category-level curriculum selection.

  • Prompt and Task Selection: Prompt selection determines which trajectories supply training signal, yet early RL post-training work often sampled uniformly from the training distribution.The survey identifies prompt choice as an important, sometimes overlooked design decision.
  • Prompt and Task Selection: GRESO targets wasted computation by predicting prompts likely to avoid identical rewards, whose zero within-group variance collapses GRPO-style advantages and produces near-zero policy-gradient signal.Its motivation is that many prompts contribute little useful training signal, especially when all samples in a rollout group receive the same reward.
  • Prompt and Task Selection: VADE uses online Beta-posterior correctness estimates and Thompson sampling to favor informative prompts while preserving exploration, whereas SEED-GRPO uses semantic entropy across multiple answers.SEED-GRPO measures meaning diversity rather than explicitly deciding which prompts to sample.
  • Prompt and Task Selection: SEC learns a distribution over prompt categories as a non-stationary multi-armed bandit, using absolute advantage as reward and updating the curriculum policy to maximize immediate learning gain.Categories can represent difficulty levels or task types, with updates performed, for example, via TD(0).

6.3 Budgeting and Scheduling (How Much Do You Roll Out?) · 6.4 Rollout Configuration Control (Length, Temperature, Concision, Positive or Negative Ratio) · 6.5 Partial Rollouts, Early Exit, and Continuation Rules

The paper presents rollout compute as a controllable resource, moving from fixed or uniformly scaled budgets toward adaptive scheduling and configuration choices that balance correctness, reasoning length, and efficiency. It also treats early stopping, partial continuation, and rollout selection as mechanisms for reducing rollout and update costs.

  • 6.3 Budgeting and Scheduling (How Much Do You Roll Out?): Early GRPO-style methods commonly use a fixed number of sampled trajectories per prompt, while other approaches increase rollout breadth or repeated optimization depth.DeepSeekMath is cited as an example of GRPO-style scaling, though the supplied passage truncates before its specific configuration.
  • 6.3 Budgeting and Scheduling (How Much Do You Roll Out?): Adaptive schedulers allocate rollout budgets non-uniformly, using predicted success probabilities, outcome variance, or trajectory diversity to improve optimization efficiency.VIP minimizes expected policy-gradient variance; MMR1 targets high-variance, diverse prompts to mitigate gradient vanishing.
  • 6.4 Rollout Configuration Control (Length, Temperature, Concision, Positive or Negative Ratio): Longer Chain-of-Thought traces can help difficult problems but often cause redundant overthinking, making reasoning length and thinking mode explicit configuration decisions.The section frames configuration control as shaping training experience through sampling and filtering under a compute budget.
  • 6.4 Rollout Configuration Control (Length, Temperature, Concision, Positive or Negative Ratio): ShorterBetter learns an instance-adaptive optimal reasoning length from the shortest correct response among multiple generations, while DECS addresses the mismatch between trajectory-level rewards and token-level optimization.ShorterBetter names this shortest correct response the Sample Optimal Length (SOL) and uses it as a dynamic reward signal.
  • 6.4 Rollout Configuration Control (Length, Temperature, Concision, Positive or Negative Ratio): AdaptThink learns to select between direct-answer and reasoning modes according to problem difficulty, reflecting a broader effort to train hybrid thinking decisions.The passage states that NoThinking can outperform long reasoning on simpler queries while maintaining overall performance.
  • 6.4 Rollout Configuration Control (Length, Temperature, Concision, Positive or Negative Ratio): Train-time sampling and filtering can reduce test-time computation by selecting responses with favorable length and token-efficiency properties, although this may require increased training-time compute.GFPO uses larger groups per problem and filters responses by length and reward per token.
  • 6.5 Partial Rollouts, Early Exit, and Continuation Rules: Rollout control reduces cost by learning when to stop, pausing or continuing partial generations, and down-selecting rollouts before communication-heavy policy updates.S-GRPO samples a single reasoning path and uses serial-group decaying rewards to encourage earlier exits and concise thoughts.

6.6 Branching and Pruning Control (Trees and Agents) · 6.7 On- and Off-Policy Controls and Correctness Constraints · 6.8 Systems to Accelerate Rollout Throughput

The surveyed controls allocate computation across branching trees, balance on-policy correctness with off-policy reuse, and accelerate rollout throughput through adaptive decoding and systems optimization. Together, these methods target exploration, dense supervision, data efficiency, training stability, and infrastructure constraints.

  • 6.6 Branching and Pruning Control (Trees and Agents): Branching and pruning allocate compute across reasoning paths by choosing nodes to expand, paths to prune, and exploration depth under fixed budgets.Tree topology can also provide denser supervision.
  • 6.6 Branching and Pruning Control (Trees and Agents): DeepSearch integrates Monte Carlo Tree Search into RLVR training, embedding structured search directly within the training loop.This design aims to improve exploration and enable fine-grained credit assignment.
  • 6.6 Branching and Pruning Control (Trees and Agents): TreeRPO uses tree sampling to estimate expected rewards at reasoning steps and computes group-relative step-level rewards without a separate step reward model.The resulting signals are dense training signals.
  • 6.7 On- and Off-Policy Controls and Correctness Constraints: On-policy rollouts align updates with the current policy but are compute intensive, whereas replay and mix-policy methods reuse past rollouts for greater data efficiency.RePO augments GRPO with a replay buffer and replay strategies to reuse off-policy rollouts.
  • 6.8 Systems to Accelerate Rollout Throughput: Throughput systems treat long rollouts and batched generation as control problems, optimizing token usage, scheduling, and infrastructure constraints in RL pipelines.These systems make rollout throughput an explicit systems-level objective.
  • 6.8 Systems to Accelerate Rollout Throughput: ReSpec adapts speculative decoding to RL by dynamically tuning decoding configurations, evolving the drafter to avoid staleness, and reward-weighting updates to preserve training stability.Drafter evolution can use distillation to reduce staleness.

7 Replay: Retention, Reuse, and Self-Evolution

Replay retains rollout-derived artifacts without updating model parameters, making future rollout behavior history-dependent and enabling reuse across responses, segments, and self-generated tasks. Retrieved artifacts can condition generation, control compute, and stabilize filtering, while policy drift creates off-policy and evaluator-drift risks.

  • Replay foundations: Replay maintains a persistent buffer of cached responses, trajectories, prefixes, verified segments, and self-generated tasks for reuse across rollouts.Because the buffer evolves over time, the realized rollout distribution depends on both the current policy and retained-and-retrieved artifacts.
  • Replay foundations: Replay introduces off-policy bias and evaluator drift when stored artifacts come from earlier policies or changing verifiers and judges.Reusing data from prior behavior policies can improve sample efficiency but creates distribution and calibration mismatches under policy drift.
  • Replay foundations: Retrieved artifacts can condition Generate, warm-start Control, reduce sampling when strong cached solutions exist, and stabilize Filter signals.Retrieval priority may depend on similarity, correctness, diversity, cost, and freshness.
  • Replay mechanisms: The surveyed reuse granularities are full-response resampling and retention, trajectory recomposition through verified segments or prefixes, and self-evolving curricula with intrinsic feedback.These mechanisms respectively stabilize learning, amortize shared computation, and generate new tasks or data through iterative rollout-to-training loops.
  • Empirical evidence: 1-shot RLVR with replay matches training on DeepScaleR’s 1.2k-example math subset containing that example, demonstrating substantial replay-based data efficiency.By contrast, on-policy GRPO does not reuse samples from previous policy iterations, limiting data efficiency.

8 Domains and Case Studies

Benchmarks define rollout interfaces by specifying trajectory structure, feedback, verification, and stopping conditions. The survey organizes case studies across verifiable language, multimodal, interactive agentic, and agentic skill interfaces, each imposing distinct rollout and evaluation requirements.

  • Cross-domain implications: Across domains, benchmark interfaces determine where feedback enters trajectories, how verification is computed, and how budgets are allocated across depth, branching, and replay.Interactive post-training can additionally couple tool-logged trajectories with multi-turn control and self-generated curricula.
  • Verifiable language interfaces: Verifiable language benchmarks use text-only rollouts with deterministic or execution-based verification for math, code, and SQL.Math commonly uses exact-answer rewards, while code and SQL use compilation, runtime execution, unit tests, or database execution; execution failures can trigger retry and repair.
  • Multimodal reasoning interfaces: Multimodal reasoning benchmarks score completions from non-text inputs using structured answer extraction, modality-aware verifiers, annotations, or rule-based checks.Instances may combine images or video with textual queries, while the generated completion remains the primary scored object.
  • Agentic interactive interfaces: Interactive agentic benchmarks alternate model actions with tool or environment observations, making correctness trajectory-level and rollout cost sensitive to long horizons.Software agents use code edits and tests; web agents use browser actions and page state; dialogue simulators derive rewards from evolving user-state signals.
  • Agentic skill interfaces: Agentic skill benchmarks evaluate whether agents induce, store, manage, and transfer reusable procedures across tasks rather than merely solve independent episodes.Representations include retrieved natural-language workflows, executable Python functions, and evolving memory or hierarchical skill banks evaluated across web, embodied, memory, and question-answering tasks.

9 Failure Modes and Open Problems

The survey identifies recurring rollout failures as diagnosable across the GFCR lifecycle and provides a troubleshooting index linking symptoms to initial modules and mitigation guidance. Open problems concern verifier calibration, reward transfer, comprehensive compute accounting, and safe replay-driven self-evolution.

  • Open problems: Verifier and judge evaluation lacks standardized, domain-spanning protocols for error rates, formatting robustness, normalization robustness, and calibration drift.These gaps affect filtering reliability and control decisions that depend on early feedback.
  • Troubleshooting index: Table 8 maps recurring rollout pathologies from observed symptoms to a primary GFCR module and relevant diagnostic or mitigation subsections.The primary module is a starting point rather than an exclusive attribution.
  • Open problems: Reward misalignment remains unresolved because RLVR systems can learn brittle heuristics or exploit reward artifacts, limiting understanding of supervision that transfers across tasks and domains.The issue is coupled to training-signal construction and generation diversity.
  • Open problems: Comparable cost accounting remains incomplete because methods often report tokens while omitting tool calls, verifier runtime, branch/prune overhead, and replay refresh.This makes true cost/benefit trade-offs difficult to assess and affects Control and Systems choices.
  • Open problems: Replay and self-evolution still require safeguards for provenance tracking, contamination, and bounding self-generated data while preserving autonomous curricula.The unresolved issue spans Replay and Filter.

10 Conclusion

The survey identifies rollout design as the link between optimization objectives and post-training experience, and introduces GFCR to unify rollout pipelines and organize recent methods across reasoning settings.

  • 10 Conclusion: Rollout design connects optimization objectives to the experience driving post-training for reasoning-oriented LLMs.The survey presents rollout design as the missing link in this process.
  • 10 Conclusion: GFCR decomposes rollout pipelines into four composable modules: Generate, Filter, Control, and Replay.The framework uses unified notation to formalize these pipelines.
  • 10 Conclusion: The framework synthesizes recent methods across verifiable language interfaces, multimodal reasoning, interactive agentic settings, and agent applications.The verifiable language interfaces include math, code, and SQL.
Loading 2605.02913v1…