PaperlayerView exact revision on arXiv

Source-linked AI summary

ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

Zhiwei Zhang, Huayu Deng, Fei Zhao, Jiayan Fu, Bin Liang, Kam-Fai Wong, Mu Chuan

arXiv:2609.35954v1cs.LGcs.AI

TL;DR

Historical rollouts from LLM post-training are often treated as stale, although they may retain compatible behaviors that later policies do not express reliably and may mix useful and undesirable steps. ROSS keeps full trajectory context while selectively supervising useful segments; across domain-specific RL, MOPD, and agentic RL, it improves upstream checkpoints.

  • Problem

    Historical self-generated rollouts are often discarded as training advances, despite potentially retaining behaviors that later policies no longer reliably recover and containing mixed-quality steps.

    Source: §1 · pp. 1–2

  • Method

    ROSS preserves complete historical trajectories as context while selectively supervising informative segments within successful rollouts.

    Source: §1 · p. 2

  • Results

    Across domain-specific RL, MOPD, and agentic RL, ROSS consistently improves upstream checkpoints; the six-benchmark MOPD average rises from 58.40% to 62.20%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Historical self-generated rollouts can remain a reusable source of behavioral experience for later checkpoints.

    Source: §1 · p. 2, §6 · p. 9

Abstract

from arXiv · show

Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.

1 Introduction

Post-training accumulates self-generated trajectories, but earlier rollouts may retain useful behaviors that later checkpoints no longer express reliably. ROSS selectively supervises valuable historical experience while preserving full-trajectory context, improving upstream checkpoints across several settings.

1 Introduction

  • RL and on-policy distillation accumulate self-generated experience, yet historical rollouts are often treated as stale as training advances.The paper asks whether later checkpoints can continue learning from behaviors discovered earlier.Source: §1 · p. 1
  • Historical rollouts may preserve underrepresented behaviors, but successful trajectories can also interleave useful reasoning with mistakes, redundant steps, and detours.Outcome-level success alone does not establish that every intermediate step is desirable supervision.Source: §1 · pp. 1–2
  • 58.40% to 62.20% on the six-benchmark MOPD average and 64.20% to 68.40% on SWE-bench Verified: ROSS improves upstream checkpoints across domain-specific RL, MOPD, and agentic RL.The evaluations span reasoning, coding, instruction following, and software engineering.Source: §1 · p. 2
  • ROSS selects valuable historical rollouts and useful segments within successful trajectories while retaining each complete trajectory as context.This aims to consolidate previously discovered behaviors without imitating entire trajectories indiscriminately.Source: §1 · p. 2

2 Preliminary Analysis

Historical experience can be worth revisiting when it remains compatible with the current policy and adds behavioral coverage that the policy does not reliably express. The analysis finds both near-on-policy compatibility and complementary behaviors, sometimes localized to particular trajectory segments.

2 Preliminary Analysis

  • Historical experience is assessed along two dimensions: compatibility with the current policy and complementarity to its behavioral coverage.Their combination distinguishes useful histories from misaligned behaviors or behaviors already well covered.Source: §2 · p. 2

2.1 Compatibility Analysis

  • Historical rollouts have response-level NLL close to current-policy rollouts and lower than external-teacher trajectories, supporting their compatibility as supervision.The analysis compares responses on 100 mathematical problems using token-level NLL under the current policy.Source: §2.1 · p. 3, Figure 3 · p. 3

2.2 Complementarity Analysis

  • 37.7% mean pass@1 versus 99.0% pass@32: across 98 historically solved problems, behaviors remain available but are not reliably expressed by the current policy.For 74 of the 98 problems, the per-problem success rate is at most 50%.Source: §2.2 · p. 3, Figure 3 · p. 3
  • The rollout reaching the correct count of 1007 first makes an incorrect 210 = 1024 count, illustrating that useful recovery and undesirable reasoning can coexist.The recovery segment identifies an overlooked constraint, so trajectory-level correctness alone does not identify which segments to learn.Source: §2.2 · p. 3

3 Method

ROSS relearns from saved rollouts after the original training run, selecting positive trajectories and useful model-generated spans while retaining full-trajectory context. Its procedure combines outcome verification, staged review, token alignment checks, and masked teacher-forcing.

3 Method

  • ROSS starts from the original run’s final checkpoint, filters saved rollouts by verified outcome, and applies imitation loss only to selected model-generated tokens while preserving full history as context.The overview summarizes this procedure as an offline relearning stage.Source: §3 · p. 3, Figure 4 · p. 4

3.1 Problem Formulation

  • The historical pool contains rollouts from earlier checkpoints of the same training run, which may include behaviors the final policy does not reliably express.ROSS can use rollouts from RL, on-policy distillation, or agentic RL procedures.Source: §3.1 · p. 4

3.2 Selective Supervision over Historical Rollouts

  • Outcome verification retains positive trajectories, then staged LLM review identifies useful ordered spans and deterministic checks validate source consistency and token alignment.Candidates are rejected if no valid target can be recovered without retaining a confirmed error or adding missing reasoning.Source: §3.2 · pp. 4–5

3.3 ROSS Training Objective

  • ROSS uses masked teacher-forcing on selected historical rollouts: unselected tokens remain in the sequence as context, while the loss targets selected tokens.The same post-annotation subset can instead be trained with all eligible policy-generated tokens in the ROSS w/o mask variant.Source: §3.2 · p. 5, §3.3 · p. 5

4 Experiments

ROSS is evaluated on historical single-turn and agentic rollouts across domain-specific RL and MOPD. It improves upstream performance, while experiments examine transfer, selective masking, and the content excluded from supervision.

4.1 Experimental Setup

  • Experiments reuse historical single-turn RL and MOPD rollouts and agentic trajectories, applying masked SFT from final checkpoints; evaluation spans mathematics, code, instruction following, and SWE-bench Verified.The rollout judge reviews saved trajectories without generating replacement targets; agentic training and relearning use a separate long-context schedule.Source: §4.1 · pp. 5–6

4.2 Main Results

  • ROSS raises domain-specific Math Avg. from 75.19 to 76.98, Code Avg. from 42.09 to 44.21, and the MOPD six-benchmark average from 58.40 to 62.20.ROSS also improves several off-domain capabilities, including Avg. Code from 41.21 to 44.24 after Math RL and Avg. Math from 74.19 to 75.03 after Code RL.Source: §4.2 · p. 6

4.3 Relearning from Agentic Trajectories

  • ROSS raises the SWE-bench Verified resolved-issue rate from 64.20 to 68.40 using successful OpenSWE trajectories from agentic RL.The reported gain is 4.20 points over the Upstream RL checkpoint.Source: §4.3 · p. 7

4.4 What Drives Relearning from Historical Rollouts?

  • With the replay set fixed, selective masking improves over ROSS without a mask by 3.71 points on MOPD, 1.91 on LCB Gen, 0.43 on OJBench, and 0.66 on Math.Trajectory filtering also helps: ROSS without a mask raises the MOPD average from 57.70 to 58.49, slightly above Upstream.Source: §4.4 · p. 7, Table 3 · p. 7
  • Base-initialized ROSS performs comparably to Upstream-initialized ROSS across evaluated settings, suggesting historical rollouts can transfer behaviors without the original parameter updates.The comparison uses identical trajectories, masks, and SFT configurations for both initializations.Source: §4.4 · pp. 7–8, Table 4 · p. 8

4.5 Characterizing Excluded Supervision

  • In 1,185 sampled trajectories, Math exclusions were mostly corrected mistakes, while Code exclusions were mostly redundant exploration and grew from 67.9% early to 76.6% late.Corrected mistakes accounted for 72.7% of early and 68.3% of late Math samples.Source: §4.5 · p. 8
  • Excluded supervision persists as RL progresses: TER shifts from 11.60% to 9.27% in Math and from 22.65% to 28.07% in Code, despite declining entropy.In a separate 4B Math RL run, ROSS suppresses a repetitive-answer mode while improving Avg. Math from 52.05% to 63.65%.Source: Figure 5 · p. 8, §4.5 · p. 9

5 Related Work

Prior self-generated supervision methods commonly select or reweight whole rollouts, while other work studies useful supervision within trajectories. ROSS focuses on selective supervision of historical rollouts from intermediate policies.

Learning from Self-Generated Experience

  • Rejection sampling, reward-ranked fine-tuning, iterative self-training, and reward-based reweighting use self-generated experience, but largely treat each rollout as an indivisible unit [26] [3] [31] [11] [35] [15].These approaches select, refine, or reweight trajectories by overall outcomes or rewards.Source: §5 · p. 9

Selecting Useful Supervision from Rollouts

  • Process supervision, step verification, agent refinement, and loss masking investigate which parts of a rollout are useful, including segment selection in successful software-engineering trajectories [14] [33] [10] [22] [2] [18] [34].SWE-Prime selects segments by contribution, learnability, and risk while keeping the full trajectory as SFT context [34].Source: §5 · p. 9
  • ROSS addresses successful historical rollouts from intermediate policies, selecting both trajectories and the behaviors within them for supervision.Its focus differs from work that selects or refines whole rollouts based on outcomes or rewards [26] [3] [31] [11] [35] [15].Source: §5 · p. 9

6 Conclusion

The conclusion presents historical self-generated rollouts as reusable behavioral experience and ROSS as a way to supervise informative verified segments while retaining full trajectories as context. Its procedure filters trajectories, audits imitation targets, compiles masks, and updates the final checkpoint.

6 Conclusion

  • Across domain-specific RL, MOPD, and agentic RL, ROSS outperforms upstream checkpoints and competing relearning baselines while preserving or enhancing out-of-domain capabilities.The conclusion frames historical rollouts as a reusable behavioral footprint accumulated during policy evolution.Source: §6 · p. 9
  • ROSS keeps complete historical trajectories as context while supervising only informative, verified segments to avoid imitating intermediate mistakes and redundant exploration.This selective procedure is presented as the basis for consolidating reusable behaviors.Source: §6 · p. 9

B ROSS Procedure

  • The procedure retains verifier-approved trajectories, audits reviewer-proposed targets, compiles and validates masks, then updates the final checkpoint.The verifier defines trajectory selection; the reviewer proposes and audits within-trajectory targets, with mask compilation deterministic.Source: B ROSS Procedure · p. 12

C.1 Historical Data and Controlled Comparisons

The protocol reuses saved self-generated rollouts for masked SFT from final checkpoints, with controlled comparisons and evaluation settings specified for single-turn experiments. Data pools, training configurations, and evaluation protocols are described for the Math, Code, and MOPD settings.

C.1 Historical Data and Controlled Comparisons

  • ROSS reuses saved responses from Math, Code, and MOPD training pools, with 3,000 Math, 12,000 Code, and 10,000 instruction-following prompts.The data come from DAPO-Math-17K, CodeI/O, and the data released with Nemotron-Cascade 2; ROSS starts from the final checkpoint.Source: §C.1 · p. 12
  • The three contrasts separate replay, filtering, and masking; the masking contrast holds examples and input context fixed, while filtering changes dataset composition and size.Positive-Rollout SFT minus Upstream measures replay; ROSS without masking minus Positive-Rollout SFT measures filtering; ROSS minus ROSS without masking measures masking.Source: §C.1 · p. 12, Table 5 · p. 13

C.2 Optimization and Infrastructure

  • Single-turn Math, Code, and MOPD use no-thinking responses, while agentic rollouts retain thinking traces and use a longer context and separate schedule.The configurations for ROSS and its matched full-supervision control are specified in Table 6.Source: Table 6 · p. 13
  • ROSS applies masked SFT to saved rollout text, with accepted supervision masks and no replacement target responses from an annotation model.Single-turn results use the three-epoch endpoint rather than each method’s best test score.Source: §C.2 · p. 13

C.3 Single-Turn Rollout Evaluation Protocol

  • Single-turn Math, Code, and MOPD evaluation uses no-thinking inference with a 16,384-token output limit.Avg. Math is the unweighted mean of three listed math benchmarks; LCB Gen is pass@1 averaged over six runs, excluding execution and output-prediction subtasks.Source: §C.3 · p. 13

C.4 Agentic Training and Evaluation Protocol

The agentic protocol trains on filtered OpenSWE tasks and reuses successful interaction histories with selective loss masking. SWE-bench Verified evaluates the resulting policies under a fixed agent scaffold, environment, interface, and execution budget.

C.4 Agentic Training and Evaluation Protocol

  • The upstream agentic RL tasks come from a filtered OpenSWE set of 4,048 training tasks, and SWE-bench Verified is reserved for evaluation.The training set excludes tasks with exposed .git histories or other reward-hacking artifacts, as well as low-quality examples.Source: §C.4 · pp. 13–14
  • The earlier historical rollout recovers from a shared overcounting error, whereas the later rollout retains it; ROSS keeps the full prefix as context and supervises selected continuation text.The overcount in the later rollout is described as an undesirable imitation target.Source: Table 7 · p. 14, §C.4 · p. 14
  • A Codex CLI agent executes repository-level actions in Harbor environments, where a task verifier converts the final repository state into terminal reward.Saved trajectories interleave model-generated assistant content with tool results and other environment observations.Source: §C.4 · p. 14
  • Positive-Rollout SFT trains on eligible assistant tokens in successful histories, while ROSS excludes assistant spans associated with identified errors and leaves the full history visible as context.Neither method replaces recorded model actions with annotation-model responses.Source: §C.4 · p. 14
  • Agentic relearning uses thinking-enabled trajectories with a 65,536-token sequence limit, compared with 32,768 tokens for single-turn rollouts.Both agentic methods start from upstream RL iteration 29 and use trajectory datasets filtered to the longer limit.Source: §C.4 · p. 14
  • SWE-bench Verified measures the percentage of repository issues resolved, with the Codex scaffold, Harbor environments, tool interfaces, and execution budgets fixed across policies.The evaluation comparison includes Base, Upstream, Positive-Rollout SFT, and ROSS.Source: §C.4 · p. 14

D A Segment-Level Complementarity Case

A sampled problem illustrates how a historically successful solution can be missing from later rollouts, while selective-supervision protocols identify usable spans without rewriting saved trajectories. The procedures compile those selections into token-level masks for training.

D A Segment-Level Complementarity Case

  • 6/40 = 15.0% of sampled responses solve the problem, versus 74.3% averaged over the full training set; none of 16 sampled rollouts at the final two observed steps solves it.At steps 2, 42, 67, 86, and 109, correct responses number 1, 3, 2, 0, and 0; the problem remains among the hardest 1–11% at every sampled step.Source: D A Segment-Level Complementarity Case · pp. 14–15

E Selective-Supervision Annotation Protocols

  • The staged reviewer first forms a verifier-positive candidate pool, then accepts candidates with a valid supervision mask into the ROSS pool.For Math and Code, the reviewer extracts a self-contained correct solution; verifier-positive instruction-following responses retain full-response supervision.Source: E Selective-Supervision Annotation… · p. 15

E.1 Shared Annotation Contract

  • Only eligible model-generated assistant tokens can receive imitation loss; user messages, tool observations, and protocol wrappers remain context.Changing the mask leaves the original sequence intact.Source: §E.1 · p. 15

E.2 Single-Turn Rollouts: Positive-Span Selection

  • Instruction-following rollouts use the outcome verifier directly: positive responses keep all eligible assistant tokens, while failed responses are discarded.The procedure uses outcome verification because these responses are typically short answers with little extended reasoning.Source: §E.2 · p. 15
  • Single-turn span selection retains correct, self-contained reasoning and the complete final answer while excluding confirmed errors; an independent judge audits the selected text itself.If a clean solution cannot be recovered from text already in the response, the candidate is dropped rather than supplied with invented reasoning.Source: §E.2 · pp. 15–16

E.3 Agentic Trajectories: Semantic Review and Defect Localization

  • Agentic review distinguishes errors with source tokens from omissions, which have no source token to exclude; later success does not erase a confirmed earlier error.A second judge localizes each confirmed error to the smallest source substring expressing it, protecting independently correct neighboring text.Source: §E.3 · pp. 16–17

E.4 Common Token-Level Mask Compilation

  • Single-turn supervision uses Aτ ∩ Kτ, while agentic supervision uses Aτ \ Eτ; both compile to a binary mask without changing the sequence.Aτ denotes positions eligible under the original training mask; Kτ projects accepted keep intervals and Eτ projects localized agentic defects.Source: §E.4 · p. 18

F Analysis of Excluded Supervision

The analysis samples masked trajectories from early and late rollout windows and classifies excluded text by primary category. It reports both trajectory-level category frequencies and token-weighted excluded-token rates.

F Analysis of Excluded Supervision

  • 300 early and 300 late Math trajectories, plus 290 early and 295 late Code trajectories, form the sampled analysis.The windows are rollout steps 1–20 and 81–100, so the analysis does not separately sample the middle stage.Source: F Analysis of Excluded Supervision · p. 18, Figure 6 · p. 19
  • Figure 6 reports the percentage of sampled trajectories assigned to each primary masked-content category, with percentages summing to approximately 100% within each column.Each trajectory receives one primary label; these proportions measure trajectory frequency, unlike TER, which measures the fraction of excluded tokens.Source: F Analysis of Excluded Supervision · pp. 18–19, Figure 6 · p. 19
  • 9.98% for Math and 26.28% for Code are the token-weighted TER values across all 100 rollout steps.Window-level TER pools excluded and total token counts.Source: F Analysis of Excluded Supervision · pp. 19–20

G Cross-Domain Performance after Domain-Specific Relearning

Cross-domain evaluations show that Math ROSS improves all measured code and instruction-following benchmarks, while Code ROSS raises aggregate Math and keeps IFBench close to Upstream. BFCL multi-turn scores rise for both, though some Code ROSS subsets decline.

G Cross-Domain Performance after Domain-Specific Relearning

  • Cross-domain benchmarks are outside the ROSS training corpus, and changes are measured against the corresponding Upstream checkpoint.The main comparison uses third-epoch ROSS models; BFCL uses Math iteration 650 and Code iteration 3389.Source: G Cross-Domain Performance after… · p. 20
  • Math ROSS increases LCB Gen by 2.19 points, OJBench by 3.88, and IFBench by 1.30.These are improvements on all measured code-generation and instruction-following benchmarks.Source: Table 8 · p. 20, G Cross-Domain Performance after… · p. 20
  • Code ROSS raises Avg. Math from 74.19 to 75.03 while IFBench changes from 35.45 to 35.07 (−0.38 points).AIME 2025 falls by 1.77 points, while AIME 2026 and HMMT-November rise by 1.61 and 2.70 points.Source: Table 8 · p. 20, G Cross-Domain Performance after… · p. 20
  • BFCL multi-turn overall rises from 44.12 to 45.62 after Math ROSS and from 46.25 to 49.38 after Code ROSS.All four Math subcategories increase; for Code, Missing Function and Missing Parameter improve while Base and Long Context decline.Source: Table 8 · p. 20, G Cross-Domain Performance after… · p. 20
  • For Math cross-domain tasks, Positive-Rollout SFT exceeds ROSS, although ROSS improves over Upstream.Positive-Rollout SFT scores 60.76 on LCB Gen, 29.96 on OJBench, and 36.78 on IFBench; the corresponding Code evaluation is unavailable.Source: G Cross-Domain Performance after… · p. 20

H Mitigating Repetitive Answer Emission after 4B Math RL

In an earlier 4B Math RL run, ROSS reduces repetitive answer emission and raises Avg. Math from 52.05% to 63.65%. Doubling Upstream’s output limit to 32,768 tokens does not improve Avg. Math.

H Mitigating Repetitive Answer Emission after 4B Math RL

  • The 4B study retains 25,869 verifier-positive trajectories from rollout steps 1–48 and reports third-epoch SFT at a 16,384-token output limit.The run uses the same mathematics dataset and a similar RL recipe to the 35B setting, with architecture-specific settings omitted.Source: H Mitigating Repetitive Answer… · pp. 20–21
  • ROSS reduces the repetition rate to 21.5%, lowers the mean boxed-answer marker count from 628.3 to 63.0, and shortens mean output length by approximately 30%.Repetition is defined as at least five boxed-answer markers per response; Upstream has 84.5% of responses meeting this threshold.Source: Table 9 · p. 21, H Mitigating Repetitive Answer… · p. 21
  • Upstream-initialized ROSS raises Avg. Math from 52.05% to 63.65%.Base-initialized ROSS reaches 65.59% on the same annotated data.Source: Table 10 · p. 21, H Mitigating Repetitive Answer… · p. 21
  • Doubling Upstream’s output limit to 32,768 tokens does not improve Avg. Math (52.05% versus 51.62%).Inspected outputs continue to show repeated boxed answers after a correct derivation.Source: H Mitigating Repetitive Answer… · p. 21
Loading 2609.35954v1…