PaperlayerView exact revision on arXiv

Source-linked AI summary

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

Zhentao Zhang, Yanzhao Zheng, Yanlin Liu, Yuanqiang Yu, Tianze Xu, Chao Ma, Jihuai Zhu, Lelun Wu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

arXiv:2610.13238v1cs.LG

TL;DR

Multi-turn agentic OPD can face unstable or less informative supervision as interactions lengthen. State-OPD grounds teacher supervision in interaction states while retaining full-history student generation and sampling complete responses at high-entropy states. Across three benchmarks, it achieves the highest macro-average success among evaluated methods for both student sizes.

  • Problem

    Multi-turn OPD supervision can become unstable or less informative as interactions lengthen, challenging reliable transfer along student-generated trajectories.

    Source: §1 · p. 2

  • Method

    State-OPD conditions teacher supervision on interaction states and reallocates sampling toward multiple complete student responses at high-response-entropy states, while retaining full-history student generation.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    31.74% macro-average success for Qwen3-4B exceeds 26.07% with Vanilla OPD, and State-OPD leads both student-size groups across the three benchmarks.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    Interaction states provide a common organizing unit for grounding teacher supervision and exploring alternative student decisions at the same state.

    Source: §6 · p. 10

    The interaction-state sufficiency result assumes responses affect the environment only through executable actions and all environment feedback is recorded in observations.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and executed action-observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning-action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. We introduce State-Grounded On-Policy Distillation, a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning-action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07\% with the strongest baseline to 31.74\% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.

1 Introduction

Multi-turn OPD faces less informative or unstable supervision as interactions lengthen. State-OPD uses interaction states to ground teacher supervision and diversify student responses, achieving leading macro-average success across benchmarks and student sizes.

1 Introduction

  • Prior multi-turn OPD work reports supervision becoming unstable or less informative as interactions lengthen, challenging reliable transfer along student-generated trajectories.Existing approaches reshape rollout histories or reallocate rollout and loss budgets across turns.Source: §1 · p. 2
  • At matched states, inherited student reasoning weakens teacher guidance, while resampling current reasoning with actions exposes more action alternatives than resampling actions under a fixed reasoning trace.Removing inherited reasoning improves teacher continuation success, with larger gains at later turns.Source: §1 · p. 2
  • 31.74% macro-average success for Qwen3-4B exceeds the 26.07% strongest-baseline result, while State-OPD leads both evaluated student-size groups.On ALFWorld Seen, Qwen3-4B also reaches 43.39%, surpassing its Qwen3-32B teacher at 38.39%.Source: §1 · p. 2
  • State-OPD makes interaction states the unit of distillation, grounding teacher targets in executed actions and observations while resampling complete student responses at high-entropy states.The student still generates from its full history.Source: §1 · p. 2
  • The paper formalizes interaction states and reports fewer persistent failed-action loops alongside leading macro-average success across benchmarks and student scales.Its ALFWorld analyses also report sustained gains with fewer supervised turn responses.Source: §1 · p. 2

2 Preliminaries

A multi-turn agent generates a response at each turn from its interaction history, after which the environment returns an observation. In OPD, the student samples a trajectory and the teacher supplies response-level targets at each turn.

2 Preliminaries

  • State-OPD’s overview contrasts the teacher’s interaction-state conditioning with the student’s full-history generation and additional complete-response sampling at high-entropy states.The figure frames distillation around the interaction reached while preserving the student’s history.Source: Figure 1 · p. 3
  • At each of T turns, the agent conditions on its history, generates a reasoning-and-action response, and receives the environment’s next observation after executing the action.The trajectory ends when the task is completed or the maximum horizon is reached.Source: §2 · pp. 2–3
  • In multi-turn OPD, the student samples a trajectory and the teacher provides a target distribution over the student’s response at every turn.The response-level objective uses reverse KL.Source: §2 · p. 3

3 Method

The section separates interaction records from student reasoning, showing how inherited reasoning affects teacher guidance and current reasoning affects sampled actions. State-OPD grounds teacher supervision in interaction states and reallocates response sampling toward high-entropy states.

3.1 Interaction State as the Unit of Agentic Distillation

  • Interaction states retain the task instruction and executed action–observation record but omit prior reasoning, separating the realized interaction from the student's interpretation.Histories with different reasoning can share the same state when their executed actions and returned observations match.Source: §3.1 · pp. 3–4

3.2 Removing Inherited Reasoning Improves Teacher Guidance

  • For Qwen3-4B, removing prior reasoning raises paired continuation success gain from 0.8 percentage points at early anchors to 20.3 points at late anchors; reverse-KL depth trends also reverse by teacher context.Removing prior reasoning improves continuation success for both student models in every anchor range.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3 A Single Reasoning Path Limits Action Diversity

  • Fixed-reasoning probes yield one parsed action in 88.5% of cases, versus 44.3% for complete-response probes; higher response entropy accompanies more additional distinct actions.The probes compare 16-response sets at the same interaction state.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 State-OPD: Distillation over Interaction States

  • State-OPD grounds teacher targets in interaction states while retaining full-history student generation, and reallocates the nominal response budget to extra complete responses at high-entropy states.The allocation concentrates responses at fewer states while approximately matching the reference turn-response budget.Source: §3.4 · p. 6

4 Experiments

The experiments evaluate State-OPD against trained baselines across three benchmarks, then examine its training dynamics, failed-action loops, and component contributions. State-OPD leads macro-average success for both student sizes, and its ALFWorld ablations show gains from both state grounding and response allocation.

4.1 Experimental Setup

  • Experiments distill Qwen3-1.7B and Qwen3-4B students from a frozen Qwen3-32B teacher across ALFWorld, ScienceWorld, and WebShop.The evaluation reports success rate and average interaction rounds; State-OPD allocates additional responses to states in the highest-entropy half of root trajectories.Source: §4.1 · pp. 6–7

4.2 Main Results

  • 31.74% average success for Qwen3-4B versus 26.07% with Vanilla OPD; State-OPD leads macro-average success for both student sizes.The comparison covers five evaluation settings; Table 1 marks the best and second-best trained methods within each student group.Source: Table 1 · p. 7, §4.2 · p. 8
  • At matched budgets, State-OPD reaches 40.26% success after 146k supervised turn responses, 11.77 percentage points above interpolated Vanilla OPD.At step 200, it reaches 44.55% success in 23.02 rounds versus 32.42% in 24.46 rounds for Vanilla OPD, after approximately 286k versus 302k supervised turn responses.Source: Figure 4 · p. 7, §4.2 · p. 8
  • At step 200, State-OPD produces 638 thinking tokens per root-trajectory turn versus 336 for Vanilla OPD.In a task-matched trace, the student revisits searched locations and observations before choosing its next action.Source: Figure 4 · p. 7, §4.2 · p. 8
  • At step 200, only 0.07% of State-OPD failure starts reach eight further identical failures, versus 4.29% for Vanilla OPD.From step 40 onward, fewer than 0.9% of State-OPD failure starts reach five further repeats, versus more than 4.9% for Vanilla OPD at every checkpoint.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 Ablation Studies

  • State grounding raises average SR from 32.42% to 41.73% without response allocation and from 35.35% to 44.55% with it.Response allocation raises average SR from 32.42% to 35.35% under full-history teacher conditioning and from 41.73% to 44.55% under state-grounded conditioning.Source: §4.3 · p. 8, Table 2 · p. 9
  • With both components, State-OPD reaches 43.39% Seen and 45.71% Unseen SR, averaging 44.55%, 12.13 points above Vanilla OPD.The table reports the mean of Seen and Unseen success rates.Source: §4.3 · p. 9, Table 2 · p. 9
  • Across β ∈{2, 3, 4}, State-OPD outperforms Vanilla OPD on both ALFWorld splits.Entropy-ranked state selection recovers more distinct actions than random selection at every tested β under matched budgets.Source: §4.3 · p. 9

5 Related Work

Related work extends on-policy distillation to multi-turn settings and addresses context, rollout, and supervision challenges in different ways. State-OPD distinguishes itself by separating inherited reasoning from the executed interaction record and using interaction states to organize teacher conditioning and response sampling.

On-policy distillation

  • Prior OPD work studies token-level objectives, reasoning-prefix supervision, data dependence, and how teacher conditioning shapes supervision.Context-cleaned dialogue methods remove prior assistant replies or condition teachers on complete user evidence; State-OPD instead preserves executed actions and observations while separating inherited reasoning.Source: §5 · p. 9

Agentic on-policy distillation

  • Agentic OPD methods shape trajectories with curricula or interventions, or vary supervision across turns and steps.State-alignment methods route guidance to supported states or reconstruct model-invocation context after memory updates.Source: §5 · p. 9
  • State-OPD identifies distinct roles for inherited and current reasoning at a fixed interaction state.Inherited reasoning distorts teacher targets, while resampling reasoning together with actions increases action diversity; this motivates state-grounded teacher conditioning and complete-response sampling.Source: §5 · p. 9

6 Conclusion

State-OPD organizes agentic distillation around interaction states, grounding teacher targets in executed actions and observations while sampling complete responses at high-entropy states. Across three benchmarks, it leads macro-average success at both student scales and reduces persistent failed-action loops on ALFWorld.

6 Conclusion

  • State-OPD grounds teacher targets in executed actions and observations and resamples complete responses at high-entropy states while retaining full-history student generation.Source: §6 · p. 10
  • Across three benchmarks, State-OPD achieves the highest macro-average success among trained methods at both student scales.Its Qwen3-4B student also surpasses the Qwen3-32B teacher on ALFWorld Seen.Source: §6 · p. 10
  • On ALFWorld, State-OPD sustains higher success with fewer supervised turn responses and sharply reduces persistent failed-action loops.The paper concludes that separating the realized interaction from student reasoning provides a common foundation for grounding supervision and exploring alternatives at the same state.Source: §6 · p. 10

A Proof of Interaction Sufficiency

The proof shows that conditioning on the interaction state preserves the posterior over latent environment states available from the full history. The training procedure then selects high-entropy states, samples additional student responses, and scores them with a state-conditioned teacher.

A Proof of Interaction Sufficiency

  • The factorization separates student-response probabilities from environment transitions, which depend on responses through executed actions.The proof sums over histories in the interaction state's equivalence class and shows the resulting terms cancel under normalization.Source: A Proof of Interaction Sufficiency · p. 13
  • For reachable histories, conditioning on the interaction state yields the same posterior over the latent state path as conditioning on the full history.Marginalizing earlier latent states gives p(ξ_t | h_t) = p(ξ_t | s_t).Source: A Proof of Interaction Sufficiency · p. 13

B State-OPD Training Procedure

  • Each iteration rolls out tasks, computes turn-level entropy, selects top-ranked states, samples additional responses there, and scores responses with the teacher conditioned on the interaction state.The student is then updated with the sampled policy surrogate.Source: B State-OPD Training Procedure · p. 13

C.1 Reasoning-History Interference

The probes isolate teacher-context effects at matched ALFWorld anchors and compare state-local action diversity under alternative reasoning-path resampling.

C.1 Reasoning-History Interference

  • 32 anchors from 30 source episodes receive four paired teacher continuations per condition to compare continuation success from matched interaction points.The conditions are full-history and interaction-state contexts; the table reports success rates and their paired difference with a source-episode bootstrap interval.Source: §C.1 · p. 13, Table 3 · p. 14
  • Teacher reverse-KL is estimated on fixed response tokens under both contexts, with turn-depth curves smoothed and paired heatmaps left unsmoothed.The analysis uses two student sizes and a frozen Qwen3-32B teacher; it reports a sampled estimate, not exact full-vocabulary KL.Source: §C.1 · p. 14

C.2 State-Local Action Diversity

  • 1,615 matched states compare distinct parsed actions from complete-response resampling against action-only resampling after a fixed reasoning prefix.Each condition uses 16-response sets, and states are retained only when both conditions have at least two valid parsed actions.Source: §C.2 · p. 14
  • 1,831 states relate root-response entropy to additional action diversity across 16 responses, with markers averaging ten equal-count entropy bins.Additional diversity counts distinct valid actions beyond the realized root action, when that action is valid.Source: §C.2 · pp. 14–15

D.1 Training Hyperparameters

Training uses a sampled-token policy surrogate and retains full interaction histories; evaluation shares prompts and context-budget rules across methods on three interactive benchmarks.

D.1 Training Hyperparameters

  • State-OPD training uses a sampled-K1 policy surrogate with clipped token-level advantages and a dual-clipped loss averaged over supervised student tokens.Environment rewards are not used, and each configuration has one independent run with seed 42, so success rates do not estimate variation across training seeds.Source: §D.1 · p. 15
  • The 2,048-token limit applies to the initial prompt, while responses and observations share a cumulative 32,768-token transcript budget without sliding-window truncation.An episode ends when the remaining budget cannot support further interaction; observation and formatting tokens consume the budget but are excluded from training loss.Source: §D.1 · p. 15

D.2 Evaluation Hyperparameters

  • All methods use the same prompts and complete splits, with training context budgets and history retention also applied at evaluation.The evaluation hyperparameter table lists shared decoding settings and environment-specific limits.Source: §D.2 · p. 15, Table 5 · p. 16

E.1 Benchmark Environments

  • The benchmarks cover household-scene action, interactive scientific procedures, and simulated shopping through a natural-language website.ALFWorld includes seen and unseen environments, ScienceWorld separates trained and held-out task types, and WebShop evaluates held-out sessions.Source: §E.1 · p. 16

E.2 Dataset Sizes and Split Construction

The evaluation uses benchmark-specific training and held-out splits, then reports task-weighted success and failure-penalized interaction rounds.

E.2 Dataset Sizes and Split Construction

  • 3,553/140/134 ALFWorld games and 2,294/456/782 ScienceWorld instances form train/seen/unseen splits; WebShop uses 500 test sessions.The task-instance table summarizes the training and evaluation sets.Source: §E.2 · p. 16, Table 6 · p. 16

E.3 Metrics

  • Success rate averages binary outcomes across sampled trajectories per task and then across tasks; ALFWorld and WebShop use four trajectories per task, ScienceWorld one.Success is defined by environment-specific completion signals, including a final reward of 1 in WebShop.Source: §E.3 · p. 17
  • Rounds averages executed actions for successful episodes but assigns failures the maximum horizon: 30 in ALFWorld and ScienceWorld, 15 in WebShop.This makes Rounds a failure-penalized measure rather than the mean actions actually executed.Source: §E.3 · p. 17

F Baseline Implementations

The appendix describes baseline mechanisms and paired task-level uncertainty procedures, and tracks persistent failed-action repetition across training checkpoints.

F Baseline Implementations

  • Vanilla OPD uses full-history student rollouts; baselines add temporal curricula, teacher-guided turns, or adaptive depth and loss normalization, alongside untrained student and teacher references.TCOD-F2B changes rollout horizon, Guided-OPD gradually withdraws teacher-generated turns, and TurnOPD retains rollout-depth and loss-normalization controllers.Source: F Baseline Implementations · pp. 17–18

G.1 Paired Bootstrap Uncertainty of Main Results

  • Paired bootstrap intervals resample matched task identities across methods, conditional on fixed checkpoints; they do not measure training-seed variation or simultaneous coverage.The reported intervals are pointwise 95% intervals without multiple-comparison correction.Source: §G.1 · pp. 18–19

G.2 Failed-Action Repetition Across Training

  • At repeat depth k = 5, State-OPD stays below 0.9% from step 40 onward, while Vanilla OPD stays above 4.9% at every checkpoint.Figure 6 extends the step-200 comparison to all ten evaluation checkpoints; separation is stronger at deeper repeat depths than at the first retry.Source: §G.2 · p. 19

G.3 State Allocation Analysis and Sensitivity

This section examines State-OPD’s allocation parameter β and state selection in ALFWorld, including their effects on success and action-diversity capture. It also contrasts State-OPD’s action revision after failures with Vanilla OPD’s repeated failed actions.

G.3 State Allocation Analysis and Sensitivity

  • With reference multiplicity M=4, State-OPD selects Pi = ⌈Ti/β⌉ states and samples n = β(M −1) additional responses at each; β=2 assigns six additional responses to about half of each trajectory’s states.Larger β values sample more responses at fewer interaction states under a nominally matched turn-response budget.Source: §G.3 · p. 19
  • At β=2, entropy ranking captured 49.09% of observed action alternatives versus 30.99% for random selection, and captured more diversity at β=3 and β=4.β=2 yielded the highest captured fraction, supporting its use as the common default.Source: §G.3 · pp. 19–20, Figure 7 · p. 20
  • Across β=2–4, Seen success ranged from 40.18% to 44.46% versus 30.89% for Vanilla OPD, and Unseen success ranged from 44.59% to 46.83% versus 33.96%; β=3 peaked on both splits.Performance declined at β=4, but the gain over OPD persisted across the tested breadth–depth allocations.Source: §G.3 · p. 20, Figure 7 · p. 20

I.2.1 ALFWorld: Recovering from a Failed Pickup

  • 6 turns: State-OPD completes the ALFWorld task by moving cd 1 to a drawer before retrying cd 2, while Vanilla OPD repeats the failed pickup through turn 30.The revised sequence succeeds after the environment reports that the initial pickup did nothing.Source: §I.2.1 · p. 29

I.2.2 ScienceWorld: Recovering from Invalid Navigation

  • 10 turns: State-OPD completes ScienceWorld by routing back through the kitchen, while Vanilla OPD repeats the invalid bathroom-to-hallway move through turn 30.After the navigation error, State-OPD returns to the kitchen before continuing; Vanilla OPD repeats the invalid move.Source: §I.2.2 · pp. 29–31

I.2.3 WebShop: Selecting Options Before Purchasing

Both students reach the same product page with the requested red and size-8 options unselected. State-OPD selects both before purchasing, while Vanilla OPD buys immediately; success across four evaluations is 4/4 versus 0/4.

I.2.3 WebShop: Selecting Options Before Purchasing

  • 4/4 versus 0/4 evaluations succeeded for State-OPD and Vanilla OPD, respectively.State-OPD selected the requested options before purchasing; Vanilla OPD purchased immediately.Source: §I.2.3 · p. 31
  • Both students reached a product page with size 8 and red available, but neither option was selected yet.They shared the same task and interaction through turn 3.Source: §I.2.3 · p. 31
  • State-OPD selected red and size 8 before buying, whereas Vanilla OPD's immediate purchase omitted the requested variants.The State-OPD trace records both selections before the purchase; the Vanilla OPD trace records the purchase without the requested color or size.Source: §I.2.3 · pp. 31–32

Related papers

Loading 2610.13238v1…