PaperlayerView exact revision on arXiv

Source-linked AI summary

From Imitation to Reward Discovery: On-Policy Warmup for Agentic RL

Yitong Qiao, Tiantian He, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu, Zhixuan Chu

arXiv:2609.39436v1cs.LGcs.AI

TL;DR

Sparse verifier rewards can leave early agent training without useful feedback, and fixed teacher trajectories may miss states induced by student decisions. The paper studies teacher-guided OPW on student-generated trajectories before RLVR. Its comparisons report earlier high performance and higher average and final performance, while its theory links on-policy distillation to initial reward discovery under stated conditions.

  • Problem

    Sparse outcome rewards provide limited guidance about intermediate decisions, and fixed teacher-generated trajectories do not cover histories induced by an agent’s own actions.

    Source: §1 · p. 2

  • Method

    OPW supervises student-generated interaction trajectories with a fixed teacher before transitioning to RLVR without further teacher queries.

    Source: §1 · p. 2

  • Results

    On-policy distillation warmup reaches high performance earlier and achieves higher average and final performance during subsequent RLVR than the compared baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings support OPW as an effective warmup for agentic RLVR and identify initial reward discovery as a mechanism that can contribute to the observed acceleration.

    Source: §1 · p. 2, §6 · p. 9

    OPW’s gains do not extend uniformly across scientific tasks, despite higher scores on Properties and Mechanics/Energy.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Reinforcement learning with a verifiable reward (RLVR) offers a scalable approach to training language-model agents, yet sparse outcome rewards can leave early training with little signal for policy improvement. We identify an On-Policy Acceleration Phenomenon: in our main comparisons, RLVR initialized with on-policy distillation reaches high performance earlier in training and achieves both higher average performance during subsequent RLVR and higher final performance than the alternative baselines. Motivated by this observation, we study On-Policy Warmup (OPW), a teacher-guided stage in which the student trains with teacher supervision on its own interaction trajectories before transitioning to RLVR. Unlike imitation on fixed teacher-generated trajectories, OPW targets states induced by the student's own decisions, including imperfect actions and recovery situations. We provide a theoretical explanation by connecting on-policy reverse-KL distillation to trajectory-level distribution matching. Under a competent teacher and sufficiently small population distillation loss, this connection yields a lower bound on initial verifier success and a corresponding bound on reward-discovery complexity. For group-relative RLVR, we further characterize when increased success probability produces more reward-informative groups. Together, our findings support on-policy distillation as an effective warmup for agentic RLVR and identify initial reward discovery as a mechanism that can contribute to the observed acceleration.

1 INTRODUCTION

Sparse verifier outcomes can leave early agent training without useful reward contrast, while fixed teacher traces miss histories induced by student actions. The paper proposes OPW and reports earlier, higher RLVR performance, with theory connecting on-policy distillation to initial reward discovery.

1 INTRODUCTION

  • Sparse binary outcome rewards can leave all-failure groups without within-group task-reward contrast for group-relative methods such as GRPO.Early learning therefore needs successes frequent enough for the verifier to distinguish useful behavior.Source: §1 · p. 2
  • Fixed teacher-generated trajectories supervise teacher-visited states, whereas student decisions can create different tool outputs, information, and recovery opportunities.This distribution mismatch motivates applying teacher supervision to student-generated trajectories.Source: §1 · p. 2
  • On-policy distillation warmup reaches high performance earlier and achieves higher average and final performance during subsequent RLVR than the compared baselines.These are the main comparisons reported for the On-Policy Acceleration Phenomenon.Source: §1 · p. 2
  • OPW trains on student-generated trajectories with teacher supervision before switching to RLVR without further teacher queries.This exposes supervision to states induced by the student’s decisions, including unsuccessful actions.Source: §1 · p. 2
  • Under a competent teacher and sufficiently small population distillation loss, trajectory-level distribution matching bounds initial verifier success and reward-discovery complexity.The analysis also characterizes when higher success probability makes group-relative RLVR groups more reward-informative.Source: §1 · p. 2

2 RELATED WORK

Prior work studies warmup for RLVR through offline supervision, on-policy distillation, and combinations of teacher guidance with reinforcement learning. This paper focuses on teacher supervision over student-generated interactive trajectories before teacher-free RLVR.

2 RELATED WORK

  • Warmup approaches include fixed offline supervision, filtering or reweighting, sequential OPD–RLVR training, and response-level alignment before RLVR.The cited methods include PEAR [Zhang et al., 2026], TailSFT [Malladi et al., 2026], and PRISM [Wang et al., 2026a].Source: §2 · p. 3
  • Interactive-agent distillation includes student-generated sequences and mixed teacher–student turns; this work studies student-trajectory supervision before teacher-free RLVR.The related work also describes teacher guidance combined with ongoing RL, interaction feedback, or completed-interaction self-distillation.Source: §2 · p. 3

3 METHOD

The method connects on-policy reverse-KL distillation to trajectory matching and initial reward discovery, then uses student-generated trajectories for teacher-guided warmup before verifier-based RLVR.

3.1 THEORETICAL MOTIVATION FOR ON-POLICY WARMUP

  • Under shared environment dynamics, cumulative on-policy reverse-KL loss equals the KL divergence between student and teacher trajectory distributions, bounding their binary-verifier success gap.The success-transfer guarantee assumes finite population loss; average token-level loss can be loose over long interactions because loss accumulates across action positions.Source: §3.1 · p. 4
  • On-policy distillation targets the student’s state distribution directly, avoiding the additional state-coverage coefficient required to transfer an offline loss bound.The distinction matters when student actions lead to novel tool outputs, execution failures, or recovery states.Source: §3.1 · p. 4

3.2 IMPLICATIONS FOR EARLY RLVR REWARD DISCOVERY

  • A positive lower bound on warm-start success bounds expected rollouts to first success by 1/ℓ under the fixed policy.For binary rewards, increased success probability raises the frequency of mixed success–failure groups only when success remains in the sparse-success regime; the lower bound alone does not guarantee this increase.Source: §3.2 · p. 4

3.3 ON-POLICY WARMUP FOR RLVR

  • OPW first trains the student on its own interaction trajectories with teacher supervision, then initializes RLVR from the warmed-up policy without further teacher queries.The student encounters imperfect intermediate solutions, failed tool calls, and recovery states during warmup.Source: §3.3 · p. 5
  • OPW samples tasks from the warmup distribution, queries the teacher at histories along current-student trajectories, and resamples trajectories as the student changes.Experiments use disjoint warmup and RLVR task pools; teacher supervision is removed after the prescribed warmup duration.Source: §3.3 · p. 5
  • The theoretical bounds concern initial reward discovery under task-specific population reverse-KL control and do not establish that finite warmup attains that condition or transfers across tasks or decoding settings.The guarantees are conditional on the stated control and policy-distribution setting.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments compare OPW with off-policy and teacher-free warmups, using ALFWorld and ScienceWorld tasks. OPW+GRPO leads the reported final task and repeated-success comparisons, though gains are not uniform across scientific task categories.

4.1 EXPERIMENTAL SETUP

  • The benchmarks cover multi-step household and scientific tasks, with separate Seen and Unseen test sets in ALFWorld and ScienceWorld.Evaluation reports ALFWorld pass@1 and ScienceWorld normalized scores; success coverage and repeated success are measured with pass@k and pass^k.Source: §4.1 · pp. 5–6
  • The evaluation uses Qwen3-1.7B and Qwen3-4B students with a Qwen3-32B teacher; 40-step warmups are followed by 60 GRPO steps.Baselines include off-policy SFT, SFT with rejection sampling, and teacher-free GRPO warmup; direct OPD and direct GRPO are also compared over 100 steps.Source: §4.1 · pp. 5–6

4.2 RESULTS

  • 96.7% vs 86.6% (ALFWorld Unseen) and 68.2 vs 61.2/55.2 (ScienceWorld Unseen): OPW+GRPO beats direct GRPO and OPD; OPW also leads the warmup-strategy comparison.The strategy comparison reports 96.7% on ALFWorld Unseen for OPW, versus 87.8% for SFT-RS and 80.3% for SFT.Source: §4.2 · p. 6, Table 3 · p. 6
  • 24.0 percentage points: OPW improves Seen Clean and Cool success over GRPO warmup, but it does not lead every ScienceWorld task category.Clean and Cool add object-state changes before placement; OPW does not lead the cited ScienceWorld category despite higher scores on Properties and Mechanics/Energy.Source: §4.2 · pp. 6–7
  • 88.8% vs 61.9% vs 28.4% pass^8 on ALFWorld Unseen; after warmup on ALFWorld Seen, OPW has lower pass@64 but higher pass^64 than GRPO.OPW+GRPO also reaches 43.8% pass^8 on ScienceWorld Seen and 8.5% on Unseen; repeated success remains difficult on ScienceWorld Unseen.Source: Table 4 · p. 6, §4.2 · p. 7, Table 5 · p. 7

5 MORE ANALYSIS

The additional analyses examine post-warmup learning, training cost, warmup duration, action errors, teacher supervision, and successful-sequence diversity. They show that warmup success alone does not predict later learning, and that outcomes depend on task and evaluation measure.

Learning Efficiency and Training Cost

  • 84.7% vs 69.5% average success over 60 GRPO steps: OPW leads on ALFWorld 4B Seen despite starting below GRPO warmup, at 46.8% vs 56.0%.In the first 20 GRPO steps, OPW gains 41.8 percentage points versus 11.4; OPW also has the highest average task performance across the reported environments, model sizes, and splits.Source: §5 · p. 8
  • 19.6 recorded training GPU-hours: OPW exceeds 80% Seen success on ALFWorld 4B, versus at least 29.6 hours for other methods; only OPW exceeds 90% within 100 steps.Over 100 steps, OPW costs less than GRPO warmup at 4B but more at 1.7B.Source: §5 · p. 8

Repeated Invalid Actions

  • 3.19% vs 10.10% vs 6.78%: repeated invalid commands account for a smaller share of actions after OPW than after SFT or GRPO warmup on ALFWorld.Removing supervision at repeated invalid actions lowers early ALFWorld 4B Seen GRPO gain to 34.6 percentage points, versus 42.0 under random removal of a similar number of action tokens.Source: §5 · p. 8

Teacher Supervision on Student-Generated Trajectories

  • 3.6 points Seen and 5.2 Unseen: resampling student trajectories improves average success during GRPO, though fixed trajectories yield higher final Seen success.In all-failure ALFWorld 4B groups, one update raises mean teacher-target log probability at teacher–student disagreements by 0.183 with OPD, versus 0.128 with SFT and 0.004 with GRPO.Source: §5 · pp. 8–9

Task Value and Retention of Teacher Corrections

  • 45.00% to 46.75%: continuation success rises over 40 GRPO steps after OPW, compared with 45.93% to 46.47% after GRPO warmup at evaluated histories.With Base fixed, the evaluation measures action selection at those histories; in a separate selected-disagreement test, teacher substitution raises continuation success from 26.48% to 29.70%.Source: §5 · p. 9

Diversity of Successful Action Sequences

  • 2.23 to 4.00: OPW’s mean distinct successful action sequences among eight rollouts rises over 60 GRPO steps on ALFWorld 4B Seen, while failure-only sequences fall from 2.95 to 0.31.For four selected successful trajectories, expected distinct-sequence count falls from 2.54 to 2.14 on ALFWorld 4B but rises from 1.83 to 2.32 on ScienceWorld 4B.Source: §5 · p. 9

Choosing the Warmup Duration

  • 10.8% to 22.0% pass^8: extending ALFWorld 1.7B OPW from 20 to 40 steps more than doubles repeated success while validity rises only from 89.4% to 91.8%.On Seen tasks, longer warmup raises final pass^8 from 69.3% to 80.7% and reaches 80% success at total step 60 rather than 75; a separate ScienceWorld Unseen comparison favors switching at 40 steps over adding 20 (4.5% vs 3.5%).Source: §5 · p. 9, Figure 4 · p. 9

6 CONCLUSION

The paper studies On-Policy Warmup (OPW), which supervises student-generated trajectories before RLVR, and reports earlier and stronger performance than compared baselines. Its theory links on-policy reverse-KL distillation to initial verifier success and reward-discovery complexity under stated conditions.

6 CONCLUSION

  • OPW uses teacher supervision on student-generated trajectories before transitioning to RLVR.Empirical analyses also report persistent teacher-target probability advantages during teacher-free GRPO, alongside improved action selection and repeated task success.Source: §6 · p. 9
  • In the main comparisons, OPW reaches high performance earlier and achieves higher average and final task performance than the compared baselines.The reported comparisons cover ALFWorld and ScienceWorld.Source: §6 · p. 9
  • The theory connects on-policy reverse-KL distillation to trajectory-level distribution matching, initial verifier success, and reward-discovery complexity under its stated conditions.The conclusion identifies initial reward discovery as a mechanism that can contribute to the observed acceleration.Source: §6 · p. 9

AI USE STATEMENT

The authors report using AI assistants to refine manuscript language and clarity, while retaining responsibility for the final manuscript.

AI USE STATEMENT

  • AI assistants were used to refine the manuscript’s language and improve its clarity.The statement says the authors reviewed and verified all scientific claims, theoretical analyses, experimental results, and references.Source: AI USE STATEMENT · p. 10
  • The authors reviewed and verified the scientific claims, theoretical analyses, experimental results, and references.The statement assigns full responsibility for the final manuscript to the authors.Source: AI USE STATEMENT · p. 10
  • The authors take full responsibility for the final manuscript.This responsibility statement appears alongside the disclosure of AI assistance for language and clarity.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility materials specify the theoretical conditions and proof details, experimental settings and evaluation procedures, and supplementary analyses supporting the reported warmup comparisons.

A.1 TRAJECTORY KL AND SUCCESS TRANSFER

  • Under shared environment dynamics, cumulative reverse-KL distillation on student-visited states equals trajectory-level KL, which bounds the student–teacher success-probability gap.A sufficiently successful teacher and sufficiently small population loss yield a positive lower bound on initial student success and an upper bound on expected success-discovery rollouts; guarantees are for the specified task, environment, verifier, and sampling policies.Source: §A.1 · pp. 14–15

A.2 REWARD DISCOVERY AND REWARD-INFORMATIVE GROUPS

  • For independent rollouts, a positive lower bound on warmup success bounds the rollouts needed to discover a success, while mixed-reward groups are not guaranteed to increase with success probability.The mixed-group probability rises below p = 0.5 and falls above it; all-success groups indicate reliable completion but have zero direct GRPO advantages.Source: §A.2 · p. 16

B.1 DATA, MODELS, AND AGENT INTERACTION

  • The setup uses released Qwen3 weights without additional agent-task training and describes the benchmarks’ observations, actions, task partitions, and interaction histories.ALFWorld reserves 64 of 3,553 training tasks; ScienceWorld reserves 64 of 2,294, with separate task pools for warmup and GRPO.Source: §B.1 · pp. 16–17, Table 8 · p. 16

B.2 TRAINING OBJECTIVES AND OPTIMIZATION

  • The training description specifies batch construction, optimizer steps, warmup and GRPO pool use, and token accounting for SFT, SFT-RS, and OPW.GRPO after warmup uses the second task pool without teacher supervision; direct GRPO instead uses that pool throughout without restarting.Source: §B.1 · p. 17, §B.2 · p. 17

B.3 EVALUATION AND STATISTICAL SUMMARIES

  • Evaluation procedures define learning-curve averaging, early gain, average performance, and task-level pass@k and pass^k measures.Learning curves average eight rollouts per task; target-reaching times use the first scheduled evaluation meeting the target, without interpolation.Source: §B.3 · pp. 17–18

C.1 FULL WARMUP RESULTS

  • OPW has the highest overall repeated task success and average task performance across environments, model sizes, and splits after warmup and GRPO.The reported measures are pass^8 after 40 warmup and 60 GRPO steps, and average performance over 60 GRPO steps.Source: §C.1 · p. 18

C.2 SENSITIVITY TO TRAINING LENGTH AND SEED

The supplementary analyses test OPW across training durations and seeds, account for training costs, and examine teacher-target probabilities, trajectory resampling, and supervision at repeated invalid actions.

C.2 SENSITIVITY TO TRAINING LENGTH AND SEED

  • With 20 warmup and 80 GRPO steps, OPW retains the highest final ALFWorld 4B success and pass^8 on both splits.After 100 GRPO steps, SFT-RS approaches OPW’s final Seen success (95.2% versus 96.2%), but average GRPO success is lower (71.0% versus 89.4%); Unseen averages are 68.3% and 90.9%, respectively.Source: §C.2 · p. 18, Table 12 · p. 19, Figure 5 · p. 20
  • Under both ALFWorld 4B seeds, OPW achieves larger early gains and higher average success than GRPO warmup on both splits.OPW starts lower on Seen and higher on Unseen; the comparison uses 40 warmup and 100 GRPO steps.Source: §C.2 · p. 19, Table 14 · p. 21

C.3 TRAINING COMPUTATION AND TARGET-REACHING COST

  • OPW costs 27.3 versus 29.6 GPU-hours at 4B, but 23.9 versus 22.2 at 1.7B, relative to GRPO warmup.Cost accounting includes trajectory generation, environment interaction, teacher scoring, and student optimization; only 4B OPW exceeds 90% within 100 steps.Source: §C.3 · pp. 20–21, Table 15 · p. 21

D.1 TEACHER-TARGET PROBABILITIES AFTER ONE OPTIMIZER STEP

  • Disagreements occupy 6.8% of response positions but account for 97.7% of teacher-target loss and 95.2% of OPD’s positive log-probability change after one optimizer step.In all-failure groups, GRPO advantages are zero, while OPD directly supervises student histories.Source: §D.1 · p. 21, Table 16 · p. 22

D.2 RESAMPLING STUDENT TRAJECTORIES

  • Resampling student trajectories improves early gains and average success, while fixed trajectories yield higher final Seen success.The comparison jointly changes histories, diversity, and task coverage.Source: Table 17 · p. 22, §D.2 · p. 22

D.3 REMOVING SUPERVISION AT REPEATED INVALID ACTIONS

  • Targeted removal of supervision at repeated invalid actions yields smaller early GRPO gains than random removal despite similar warmup validity.The result supports a contribution from supervision at repeated errors.Source: Table 18 · p. 22, §D.3 · p. 22

E.1 TEACHER ALTERNATIVES, TASK VALUE, AND RETENTION

The analysis tests whether teacher alternatives improve action choices and whether those gains persist during teacher-free GRPO. OPW’s effects differ across selected disagreements and broader action histories, while common-history evaluations show improved continuation success during GRPO.

E.1 TEACHER ALTERNATIVES, TASK VALUE, AND RETENTION

  • Teacher alternatives raise action validity from 67.44% to 82.22% and continuation success from 26.48% to 29.70%.On 89 added tasks, continuation success still improves by 2.39 percentage points.Source: §E.1 · p. 23
  • During 40 teacher-free GRPO steps, OPW’s mean teacher-target probability rises from 39.3% to 47.2% at selected positions but stays near its warmup level on broader disagreements, 43.8% to 43.2%.Selected positions show further gains, while the broader set shows retention.Source: §E.1 · p. 23, Table 19 · p. 23

E.2 COMPLETE-ACTION VALUE AT COMMON HISTORIES

  • The common-history evaluation samples policy actions at shared positions and uses Base to supply continuations for each sampled action.The evaluation averages positions and sources within tasks, then weights tasks equally.Source: §E.2 · p. 23
  • At active common histories, OPW continuation success rises from 45.00% to 46.75% over 40 GRPO steps, versus 45.93% to 46.47% after GRPO warmup.Base supplies every continuation, so the reported gains reflect improved action selection at the evaluated histories.Source: §E.2 · p. 23, Table 20 · p. 24

F.1 EXECUTION ERRORS AND REPETITION

Across execution-error and action-sequence analyses, OPW is associated with fewer repeated invalid actions and more successful action sequences. During warmup, repeated task success continues to improve even when validity gains slow or partial-credit differences are already common.

F.1 EXECUTION ERRORS AND REPETITION

  • Validity requires membership in the current valid-action list, while repetition means an action appeared earlier in the trajectory, whether valid or not.ScienceWorld navigation aliases absent from the list count as invalid even if execution succeeds.Source: §F.1 · p. 24
  • On ALFWorld Seen, consecutive identical invalid actions account for 0.66% of actions after OPW versus 3.93% after SFT; on ScienceWorld Unseen, the shares are 4.17% and 13.83%.The fractions of ALFWorld Seen trajectories with invalid actions are similar for SFT and OPW: 50.9% and 50.0%.Source: §F.1 · p. 24
  • For ALFWorld 4B, OPW validity rises only 0.9 points during GRPO while success rises 49.3 points.Table 21 compares execution behavior and task performance before and after 60 GRPO steps.Source: §F.1 · p. 24, Table 21 · p. 24

F.2 SUCCESSFUL AND FAILURE-ONLY ACTION SEQUENCES

  • In ALFWorld 4B, distinct action sequences fall from 5.18 to 4.31 after OPW, while successful sequences rise from 2.23 to 4.00 and failure-only sequences fall from 2.95 to 0.31.Across both environments and model sizes, total sequence counts decrease after OPW while successful counts increase.Source: §F.2 · p. 25, Figure 7 · p. 25
  • With four successful trajectories sampled, OPW’s expected distinct-sequence count falls from 2.54 to 2.14 in ALFWorld but rises from 1.83 to 2.32 in ScienceWorld.More successful sequences among eight rollouts can accompany either less or more variation among equally many successes.Source: §F.2 · p. 25, Table 22 · p. 26

G.1 WHAT CONTINUES TO IMPROVE DURING WARMUP?

  • On ALFWorld 1.7B, tasks solved in every rollout increase from 2 to 11 between OPW steps 20 and 40, while tasks solved at least once rise from 52 to 53.The analysis uses 32 rollouts per task on reserved analysis tasks distinct from the full-training Seen comparison.Source: §G.1 · pp. 25–26, Table 23 · p. 26
  • In ScienceWorld 1.7B, pass@8 rises from 18.8% to 56.2% between OPW steps 15 and 40, although 92.2% of groups already have different partial-credit scores at both steps.The figure contrasts partial-credit reward differences with complete-task success.Source: §G.1 · p. 26, Figure 8 · p. 26

G.2 LEARNING SPEED AND REPEATED TASK SUCCESS

Warmup duration comparisons show that longer OPW can improve ALFWorld repeated success, while ScienceWorld outcomes vary with the task set and evaluation metric. Long-trajectory examples illustrate OPW completing tasks after longer searches, and the paper limits its experiments to two controlled testbeds.

G.2 LEARNING SPEED AND REPEATED TASK SUCCESS

  • On ALFWorld Seen, extending OPW from 20 to 40 steps raises final pass@1 from 88.3% to 94.9% and pass^8 from 69.3% to 80.7%, with the first 80%-success evaluation moving from step 75 to 60.Both paths end at 100 total steps, but GPU-hours differ.Source: §G.2 · p. 26, Table 24 · p. 27
  • On ScienceWorld, switching to GRPO after 40 OPW steps yields higher Unseen pass@1 and pass^8 than 20 further OPW steps, while longer warmup improves Seen mean score and pass@8.The preferred allocation depends on task set and whether evaluation rewards partial progress or complete success.Source: §G.2 · p. 27
  • Within a separate ScienceWorld seed-3 run, extending OPW from 20 to 40 steps raises Unseen pass@1 from 28.3% to 34.1% and pass^8 from 1.0% to 2.0%, while mean score changes little.These comparisons support choosing duration through validation on the required outcome.Source: §G.2 · p. 27

H LONG-TRAJECTORY EXAMPLES

I LIMITATIONS AND SCOPE

  • Experiments use ALFWorld and ScienceWorld as controlled testbeds for multi-step interactive tasks; broader interaction settings remain future work.Warmup duration and end-to-end efficiency depend on task, model configuration, and teacher-supervision cost.Source: I LIMITATIONS AND SCOPE · p. 29

Related papers

Loading 2609.39436v1…