PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation

Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu, Xiangjun Fan, Zhuokai Zhao

arXiv:2606.19659v1cs.CL

TL;DR

Multi-turn OPD must contend with errors that alter later observations and with brittle dense token-level supervision. SAGE-OPD selectively weights student turns using intervention judgments and teacher confidence; it improves over baselines across agent tasks, including on ALFWorld.

  • Problem

    Most OPD studies focus on single-turn settings, while multi-turn errors can alter later observations and dense token-level supervision can mishandle valid alternatives or unreliable histories.

    Source: §1 · p. 1, §2 · p. 3, §3.1 · p. 4

  • Method

    SAGE-OPD keeps student-induced trajectories and weights turn-level OPD using teacher intervention judgments and predictive confidence, with loss normalization to preserve scale.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across ALFWorld, ScienceWorld, and SearchQA, SAGE-OPD improves over base students, off-policy SFT, and standard OPD while remaining competitive with strong OPD variants; on ALFWorld it achieves a 13.3% relative improvement in success rate over standard OPD with Qwen-3 models.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    The supported conclusion is that multi-turn OPD can retain on-policy teacher supervision without applying uniformly dense distillation to every student-generated turn.

    Source: §1 · p. 2, §7 · p. 13

    The normalization result does not guarantee lower total variance than standard OPD, because vanilla-scale normalization may produce Z > 1.

    Source: §4 · p. 7

Abstract

from arXiv · show

On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training. However, most OPD studies focus on single-turn settings, while realistic LLM agents interact with environments over multiple turns. In this regime, early errors can alter future observations and compound across the trajectory, and standard dense token-level OPD becomes brittle, as it may over-penalize semantically valid alternatives, reinforce local degeneracies such as repeated actions, and propagate unreliable teacher supervision on off-distribution histories. We propose SAGE-OPD, a verifier-free selective intervention framework specifically designed for multi-turn OPD. Instead of applying teacher supervision uniformly across all turns, SAGE-OPD first observes environment feedback and uses teacher judgment to decide whether each student response should be skipped or intervened on. To further address compounding errors, SAGE-OPD weights token-level distillation by teacher confidence, reducing the influence of uncertain teacher distributions on corrupted or ambiguous histories. Finally, SAGE-OPD applies loss normalization to preserve the overall loss scale of standard OPD while retaining selective turn-level weighting. Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD. Ablation studies further demonstrate that turn-level intervention, teacher confidence weighting, and loss normalization provide complementary benefits. Our results suggest that effective multi-turn OPD should remain on-policy, but teacher supervision should be selectively allocated to turns where intervention is necessary and reliable.

1 Introduction

The introduction motivates multi-turn OPD as a way to train on student-induced histories, but highlights brittle token-level supervision and compounding errors. SAGE-OPD selectively allocates supervision, and the paper reports gains over standard OPD and across several agent-task domains.

Motivation

  • Most existing OPD studies evaluate single-turn reasoning, while multi-turn errors can change later observations and compound across an interaction.Dense token-level alignment can over-penalize semantically valid alternatives, reinforce repeated or malformed patterns, and propagate less reliable teacher supervision on off-distribution histories.Source: §1 · pp. 1–2

Approach

  • Teacher confidence weighting and loss normalization complement turn-level intervention by reducing uncertain supervision while preserving the overall training-signal scale.The introduction reports that these components provide complementary benefits in ablation studies.Source: §1 · p. 2

Results

  • 13.3% relative improvement in success rate over standard OPD with Qwen-3 models on ALFWorld.The introduction also reports gains across embodied interaction, scientific task solving, and question answering, with strong gains over standard OPD and other state-of-the-art OPD baselines.Source: §1 · p. 2

Approach

  • SAGE-OPD is a verifier-free selective intervention framework that preserves on-policy supervision while avoiding uniformly dense distillation over student-generated turns.The teacher judges whether each turn should be skipped or intervened on, avoiding unnecessary dense supervision on acceptable or semantically flexible responses.Source: §1 · p. 2

2 Related Work

Related work frames multi-turn OPD against multi-turn SFT and reward-based RL, then describes brittleness in dense token-level OPD. SAGE-OPD differs from trajectory-depth curricula by deciding where supervision is applied and how strongly it is trusted.

Multi-turn Agent Interaction

  • Multi-turn agents must act on evolving observations, so an early suboptimal action can change the future observation stream and compound errors.The related work situates these challenges in embodied simulators, web navigation, tool use, and search-based question answering.Source: §2 · p. 3
  • Multi-turn SFT imitates teacher-generated states, while RLVR optimizes final outcomes but typically needs external rewards and faces sparse-reward credit assignment.These limitations motivate OPD methods that supervise student-induced trajectories without relying solely on outcome-based rewards.Source: §2 · p. 3

On-Policy Distillation and Multi-turn Limitations

  • OPD trains on student-sampled trajectories and gives teacher token-level feedback on student-induced states, aligning training with inference-time behavior.The cited work presents this as a way to mitigate exposure bias from offline imitation.Source: §2 · p. 3
  • Prior OPD analyses identify brittle token-level alignment, which can over-penalize valid alternatives, reinforce degenerate patterns, and become less reliable as errors shift later histories.In multi-turn agents, repeated thoughts or actions can affect subsequent environment feedback.Source: §2 · p. 3
  • TCOD controls trajectory depth with a temporal curriculum, whereas SAGE-OPD delegates turn-level intervention to the teacher and weights supervision by teacher confidence.SAGE-OPD decides where OPD applies and how strongly its signal should be trusted, rather than manually setting how long it applies.Source: §2 · p. 3

3 Methodology

The methodology retains student-generated multi-turn trajectories but makes supervision selective at the turn level. Teacher judgments and confidence set each turn’s weight, while normalization preserves the scale of standard OPD.

3.1 Preliminaries

  • In multi-turn OPD, the student acts from histories formed by prior observations and actions, and the teacher supervises states induced by the student’s own policy.Environment feedback makes early actions consequential for later observations and histories.Source: §3.1 · pp. 3–4

3.2 SAGE-OPD

  • SAGE-OPD uses environment feedback and teacher judgment to decide whether each response should be skipped or intervened on, then combines that decision with teacher confidence and normalizes the loss.This concentrates supervision where intervention is needed and the teacher signal is reliable, while retaining selective turn-level weighting at standard OPD scale.Source: §3.2 · p. 4, Figure 1 · p. 5

3.3 Turn-Level Intervention

  • Turn-level intervention assigns Skip, Weak, or Strong weights; clear execution failures receive Strong, while the teacher judges remaining executable but ambiguous responses.Skip covers acceptable responses or plausible alternatives; Weak marks possible suboptimality with uncertainty, and Strong marks responses judged clearly detrimental.Source: §3.3 · p. 5

3.4 Teacher Confidence Weighting

  • Teacher confidence is the average top-1 probability across response tokens, and it weakens supervision when the teacher’s predictive distribution is uncertain.The confidence factor is combined with the intervention weight and applied across the turn’s token-level OPD loss.Source: §3.4 · p. 6

3.5 Loss Normalization

  • Loss normalization keeps the average effective token weight close to one, preserving standard OPD’s scale while retaining relative turn-level weighting.Without normalization, selective weights reduce the overall gradient magnitude as well as changing the relative contribution of turns.Source: §3.5 · p. 6

4 Theoretical Analysis

The analysis studies how selective weighting changes noisy teacher supervision and how its objective approximates an ideal oracle intervention objective. It characterizes relative noise suppression and approximation error while limiting what those results guarantee.

Selective Reweighting of Supervision Noise

  • Selective weighting suppresses noisy-turn contributions relative to more heavily weighted turns, with the common normalization factor canceling in pairwise comparisons.The noise analysis models per-turn OPD loss as ideal supervision plus zero-mean noise from teacher uncertainty, token-level mismatch, or off-distribution histories.Source: §4 · p. 7
  • Vanilla-scale normalization does not guarantee lower total variance, since Z > 1 can increase the normalized objective's absolute variance in some batches.The result preserves relative suppression: turns with smaller selective weights contribute quadratically less noise than turns with larger weights.Source: §4 · p. 7

Connection to Oracle Intervention

  • The selective objective's deviation from oracle supervision is controlled by how closely soft weights s_t approximate ideal intervention decisions m*_t.The analysis frames the soft weights as an approximation, not as recovery of the oracle mask.Source: §4 · pp. 7–8
  • Deterministic environment failures provide high-precision positive intervention signals, while teacher judgments approximate oracle decisions for executable but ambiguous turns.Teacher confidence further softens the intervention weight when the token-level teacher distribution is uncertain.Source: §4 · p. 8

5 Experiments

Experiments evaluate SAGE-OPD across three multi-turn agent benchmarks and compare it with OPD and other baselines. It performs particularly well on ALFWorld, remains competitive on ScienceWorld and SearchQA, and continues improving on ALFWorld’s unseen split as standard OPD plateaus.

5.1 Experimental Setup

  • The evaluation uses Qwen3 teacher–student pairs on ALFWorld, ScienceWorld, and SearchQA, comparing SAGE-OPD with off-policy SFT, standard OPD, TCOD-F2B, Entropy-Aware OPD, and compute-matched OPD rollout-2.The benchmarks span embodied interaction, scientific task solving, and search-based question answering.Source: §5.1 · p. 8

5.2 Main Results

  • SAGE-OPD is strongest on ALFWorld and competitive on ScienceWorld and SearchQA; it leads unseen SR in three of four ALFWorld settings and ties seen SR in two.On SearchQA, Qwen3-1.7B/Qwen3-8B average EM rises from 30.09 under OPD to 30.77; on ScienceWorld, Qwen3-0.6B/Qwen3-8B mean score rises from −12.88 to +3.89, with more turns.Source: Table 1 · p. 9, §5.2 · p. 9, Table 2 · p. 10
  • Qwen3-0.6B/Qwen3-8B improves from 55.00 to 57.86 seen SR and from 56.72 to 61.94 unseen SR over standard OPD.For Qwen3-1.7B/Qwen3-8B, unseen SR improves from 61.94 to 70.15; SAGE-OPD also outperforms compute-matched OPD rollout-2 in most ALFWorld settings.Source: Table 1 · p. 9, §5.2 · p. 9
  • The benefits are more consistent for Qwen3-1.7B, while Qwen3-0.6B shows more task-dependent effects on ScienceWorld and SearchQA.The authors hypothesize that stronger teacher signals are not always more reliable when applied densely.Source: §5.2 · p. 10

5.3 Training Dynamics

  • On ALFWorld unseen success, standard OPD plateaus around 60–63% after roughly 20–30 steps, while SAGE-OPD reaches 70.15% by training’s end.Training-batch success tracks closely between methods, while SAGE-OPD’s distillation loss decreases steadily.Source: §5.3 · pp. 10–11, Figure 2 · p. 11

6 Ablation Studies and Analysis

The analyses examine how intervention labels vary across training and turns, then isolate component and configuration effects on ALFWorld. They find complementary roles for intervention, confidence weighting, and normalization, with moderate intervention strength and deterministic confidence performing best among tested settings.

6.2 Ablation Studies

  • Intervention-only raises unseen SR from 56.72 to 62.69 for Qwen3-0.6B/Qwen3-8B and from 68.66 to 79.10 for Qwen3-1.7B/Qwen3-32B.The authors interpret this as evidence that turn-level labels can identify useful correction points without a task-level verifier.Source: Table 4 · p. 12, §6.2 · p. 12
  • For Qwen3-1.7B/Qwen3-8B, normalization raises unseen SR from 64.93 under NoNorm to 70.15 with full SAGE-OPD; for Qwen3-0.6B/Qwen3-32B, it rises from 69.40 to 70.90.The ablations report complementary roles for intervention weighting, teacher confidence, and loss-scale normalization.Source: Table 4 · p. 12, §6.2 · p. 12

6.1 Intervention Analysis

  • Skip labels generally increase over training but decrease at later turn positions, while Det-Fail becomes more frequent as interaction horizons grow.Det-Fail covers clear failures such as malformed actions, failed tool calls, missing executable actions, or environment-reported execution failures, which receive full intervention without teacher judgment.Source: §6.1 · p. 11, Figure 3 · p. 13

6.3 Sensitivity Analysis

  • With Qwen3-8B/Qwen3-1.7B, α = 0.5 achieves the best seen and unseen SR; larger α values degrade performance and increase average turns.The authors report that α approaching 1 makes weak interventions resemble strong interventions and the objective closer to dense OPD on ambiguous turns.Source: §6.3 · p. 12, Table 5 · p. 13
  • Teacher temperature 0.0 performs best, while higher temperatures reduce success rate; active-token-only normalization reaches 68.66% unseen SR versus 70.15% with default normalization.The default normalization counts all student tokens.Source: §6.3 · p. 13, Table 5 · p. 13

7 Conclusion

SAGE-OPD selectively allocates on-policy teacher supervision across turns and weights token-level distillation by teacher confidence. Experiments across three agent benchmarks show improvements over several baselines, with complementary benefits from the framework’s components.

7 Conclusion

  • SAGE-OPD improves over base students, off-policy SFT, and standard OPD across ALFWorld, ScienceWorld, and SearchQA, while remaining competitive with strong OPD variants.Ablation and training-dynamics analyses find complementary benefits from intervention labels, teacher confidence weighting, and vanilla-scale normalization.Source: §7 · p. 13
  • SAGE-OPD estimates whether each response needs intervention, then weights token-level distillation by teacher confidence rather than supervising every turn uniformly.The framework is verifier-free and designed for multi-turn on-policy distillation.Source: §7 · p. 13
  • The results suggest multi-turn OPD can remain on-policy without dense token-level supervision, allocating teacher guidance where intervention is necessary and reliable.The conclusion also identifies intervention labels, confidence weighting, and vanilla-scale normalization as complementary components.Source: §7 · p. 13

A Implementation Details

The implementation uses a shared asynchronous training pipeline, benchmark-specific evaluation protocols, and distinct weighting schemes for agentic OPD baselines and SAGE-OPD.

A Implementation Details

  • SAGE-OPD and its baselines use a fully asynchronous verl pipeline with a staleness-bounded queue, AdamW, learning rate 10^-6, weight decay 0.1, and one training epoch.The setup uses rollout number n = 1, global batch size 64, response length 4096, and synchronization and staleness settings of 4 and 0.5.Source: A Implementation Details · p. 16
  • Evaluation caps episodes at 30 turns for ALFWorld and ScienceWorld and 16 for SearchQA, using ReAct prompts with full chat history and benchmark-specific test splits.SearchQA allows up to eight search calls and retrieves three documents; the training subset randomly samples 10K SearchQA examples.Source: A Implementation Details · p. 16
  • TCOD-F2B truncates rollouts after assistant turns, while Entropy-Aware OPD scales token weights when cached teacher entropy exceeds 0.8, using α = 1.0.TCOD-F2B uses kstart = 1 and η = 1; its maximum horizon is 30 turns for ALFWorld and ScienceWorld and 16 for SearchQA.Source: A Implementation Details · p. 16
  • SAGE-OPD obtains a per-turn intervention weight from one teacher query per assistant turn, defaulting to 0.5 for Weak and using full intervention for deterministic failures.If the teacher output is unparseable, the implementation falls back to full intervention; empty, unparseable, or schema-invalid student turns also receive full intervention directly.Source: A Implementation Details · p. 16
  • Evaluation uses ALFWorld valid-seen and valid-unseen success rate, ScienceWorld success rate and mean score, and SearchQA exact match with a macro-average across seven datasets.ALFWorld and ScienceWorld also report average turns; SearchQA reports average search and turn statistics.Source: A Implementation Details · p. 16

B.1 Case Study: Turn-Level Intervention Examples

The case studies illustrate how SAGE-OPD distinguishes acceptable exploration, uncertain but executable actions, harmful responses, and deterministic execution failures. Environment feedback and teacher judgment guide these turn-level supervision choices.

B.1 Case Study: Turn-Level Intervention Examples

  • Acceptable exploratory actions, such as looking for objects, receive Skip and intervention weight 0 even when they do not immediately complete the task.Plausible actions can remain Skip even when the environment reports that they found nothing, avoiding unnecessary token-level correction.Source: §B.1 · p. 17
  • Executable but potentially inefficient actions receive Weak supervision, while actions inconsistent with the required task procedure receive Strong intervention.For example, navigating to a cabinet instead of cooling a plate is inconsistent with the stated procedure and receives full intervention weight.Source: §B.1 · p. 17
  • Environment- or parser-detected deterministic failures bypass teacher judgment and receive full intervention weight.The examples include actions that fail to produce a valid state transition; the paper describes this pathway as a high-precision correction signal for malformed, invalid, or ineffective executable actions.Source: §B.1 · p. 18
  • The case studies distinguish turns to leave unchanged, guide weakly, or correct strongly using environment feedback and teacher judgment.The discussion notes the trade-off: over-correcting acceptable exploration can reduce behavioral diversity, while missing harmful or invalid actions can let errors compound.Source: §B.1 · p. 18

C Prompt Templates

The prompt templates support benchmark-specific ReAct interactions and turn-level teacher judgments. The training procedure combines turn-level intervention labels with teacher confidence to form selective supervision weights.

C Prompt Templates

  • The benchmark prompts use fixed output protocols so actions can be parsed and sent to each environment.The templates cover ALFWorld, ScienceWorld, and SearchQA; SearchQA prompts the agent to issue Wikipedia queries and end with a concise answer.Source: C Prompt Templates · p. 18, Table 6 · p. 19, Table 7 · p. 19
  • The separate intervention prompt gives the teacher the task goal, current observation, admissible actions, and student response for a Skip, Weak, or Strong judgment.The prompt template presents this information as a user turn followed by the assistant’s thought and action.Source: C Prompt Templates · pp. 18–19, Table 9 · p. 20
  • The training procedure assigns Strong with full intervention to deterministic execution failures and otherwise asks the teacher to label each turn.The label mapping assigns intervention weights 0 to Skip, α to Weak, and 1 to Strong.Source: C Prompt Templates · p. 21
  • The procedure combines the per-turn intervention weight with teacher confidence to obtain the selective weight used for supervision.The listed procedure computes teacher confidence before setting the selective weight to the product of the two weights.Source: C Prompt Templates · p. 21
Loading 2606.19659v1…