When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

Junzhuo Liu; Weiwei Li; Jun Ling; Peng Wang

Source-linked AI summary

When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

Junzhuo Liu, Weiwei Li, Jun Ling, Peng Wang

arXiv:2608.05219v1Aug 5, 2026cs.AI

TL;DR

Multi-turn agents can receive unreliable privileged guidance when their execution states diverge from successful reference trajectories. SMRC-SD routes distillation only at matched states and constructs state-conditioned teacher context, improving task success from 0.746 to 0.865 on ALFWorld and from 0.574 to 0.693 on WebShop.

Problem
Successful reference trajectories may not support locally compatible continuations after the student's actions change its execution state.
Method
SMRC-SD matches the reached state to the reference, applying distillation only on matched turns and constructing locally grounded teacher context.
Results
0.746 to 0.865 and 0.574 to 0.693 task success rates on ALFWorld and WebShop, respectively, outperforming unconditional FullPath-SD.
Takeaways & Limitations
Across embodied and web interaction, the results support state-compatible routing and teacher-context construction as components of improved policy performance.

Abstract

from arXiv · show

Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student's response at every turn with access to training-only references, such as successful trajectories. In interactive environments, however, the student's preceding actions continually change the execution state. As the student takes different actions or completes subgoals in a different order, its rollout may reach states not covered by the reference, making the reference an unreliable source of guidance for the state actually reached. Applying privileged distillation indiscriminately therefore creates state--reference mismatch. This mismatch motivates a central objective: providing privileged reference guidance that remains compatible with the student's current execution state. We introduce State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which explicitly determines when and how a privileged trajectory should guide an on-policy student. At each turn, SMRC-SD verifies whether the student's current execution state matches a supported state along the reference trajectory. Distillation is applied only at matched states, filtering out turns for which the reference lacks locally compatible guidance. For each matched state, SMRC-SD further constructs state-conditioned teacher context from the successful trajectory, grounding supervision in the state actually reached. Across ALFWorld and WebShop, SMRC-SD consistently outperforms unconditional successful full-path distillation. With Qwen3-1.7B, it improves task success from $0.746$ to $0.865$ on ALFWorld and from $0.574$ to $0.693$ on WebShop. Controlled routing and context ablations support both selecting locally supported turns and constructing state-compatible teacher context as contributors to these gains. Code is available at https://github.com/liujunzhuo/SMRC-SD.

State-Matched Routing and Contextualized Self-Distillation · Introduction

SMRC-SD addresses state–reference mismatch in privileged on-policy distillation by routing guidance only to matched execution states and constructing locally grounded teacher context. Across ALFWorld and WebShop, it outperforms unconditional FullPath-SD, with gains supported by routing and context ablations.

Introduction: Outcome rewards are sparse about intermediate decisions, whereas on-policy distillation preserves exploration while providing dense token-level supervision.The student’s responses are re-scored by a synchronized teacher conditioned on training-only feedback or references.
Introduction: FullPath-SD exposes the complete successful trajectory to the teacher at every turn, despite student actions continually changing execution state.Relevant state variables include location, inventory, object properties, active page, and completed subgoals.
Introduction: State–reference mismatch makes a task-correct trajectory locally invalid when the student reaches a state with no compatible demonstrated continuation.A transition is tied to its pre-action state, so different actions or subgoal orders can invalidate unconditional privileged guidance.
Introduction: Prior methods reshape rollout distributions or select, mask, and weight teacher signals, but do not directly verify whether references support the reached state.These approaches regulate states presented to distillation or how the resulting signal is applied.
Introduction: SMRC-SD matches each reached execution state against reference states, routes distillation only on matches, and constructs teacher context from the matched continuation and reached-state summary.Unmatched turns remain optimized by the GRPO objective.
Introduction: 0.865 versus 0.746 task success rate is achieved by SMRC-SD versus unconditional FullPath-SD on ALFWorld with Qwen3-1.7B.On WebShop, task success rate likewise improves from 0.574 to 0.693.
Introduction: Controlled ablations show that both state-compatible routing and state-compatible teacher context contribute to consistent gains across embodied and web interaction.The reported environments are ALFWorld and WebShop.

Related Work

Related work spans privileged on-policy and self-distillation, methods for reliable supervision under changing multi-turn states, and approaches that maintain or reconstruct state-aware agent context.

Privileged on-policy and self-distillation: Privileged self-distillation conditions a synchronized self-teacher on verified traces, feedback, or task knowledge, extending classical and student-generated-sequence distillation.The passage also identifies Skill-SD and SDAR as multi-turn agent methods.
Reliable supervision in multi-turn interaction: Recent multi-turn distillation methods improve supervision by reshaping rollout horizons or prefixes, or by selecting and weighting supervision using confidence, discrepancy, structure, or outcomes.Examples include TCOD, ReOPD, SAGE-OPD, SDAR, StepOPSD, HINT-SD, TurnOPD, and SERL.
State-aware agent context: State-aware agent methods preserve decision context through histories, feedback, goal-state reflection, prior trajectories, workflows, state compaction, state-level retrieval, or proxy-state reconstruction.These approaches address evolving interaction state through complementary forms of context maintenance and reuse.

Preliminaries

The preliminaries define on-policy interactive rollouts and privileged teacher scoring, then formalize routed distillation over turns. They also distinguish state-certified reference guidance from unconditional full-path supervision.

Interactive rollout: At turn t, the environment provides observation o_t and admissible actions A_t, while the agent retains interaction history h_t.The ordinary prompt is formed from the task, history, observation, and admissible actions; the policy samples a response and parses an action.
Interactive rollout: GRPO uses group-wise trajectory advantages and applies LGRPO to every rollout after the episode receives terminal reward r(τ).
Privileged distillation: After rollout, a detached synchronized teacher scores the student’s response under privileged input containing training-only context, without generating actions.
Privileged distillation: Routing w_t ∈ {0, 1} controls which turns contribute the chosen-token K3 distillation term, while the on-policy trajectory and token support remain unchanged.Because the denominator uses all response tokens, selecting fewer turns reduces total SDL weight rather than renormalizing selected turns.
Reference-state compatibility: A successful reference path certifies actions only at their demonstrated pre-action states, so SMRC-SD derives routing and teacher context from the reached state and reference path.The stored reference contains the canonical action sequence rather than simulator states, whereas FullPath-SD supplies the entire reference path at every turn.

State-Compatible

SMRC-SD routes privileged distillation only to reference-supported states and localizes teacher context to the reached state and grounded continuation. Fixed-state ALFWorld interventions show that compatibility preserves agreement on matched turns and strengthens candidate-relative preference beyond generic successful-path context.

State-Matched Routing: A turn is matched only when the successful reference contains a compatible pre-action state whose next action can be grounded in the reached state.Unmatched turns are excluded from reference-conditioned supervision, while matched turns identify the corresponding continuation.
Controlled Evaluation: The fixed-state study uses 400 anchors per stratum, retaining admissible, successfully executed, progress-making actions while varying only teacher context.A contains matched agreeing turns, B matched divergent turns, and C unmatched turns with valid progress-making actions.
Routing Results: +0.070 [+0.012, +0.153] is the A–C contrast under FullPath-SD, with positive mean shift on matched A and negative shift on unmatched C.Across 800 A anchor–scorer evaluations, localized context decreased an already agreeing action’s score by more than 0.01 only once.
Contextualized Guidance: +1.158 is the matched FullPath-SD preference shift, while SMRC-SD adds +0.266 [+0.206, +0.328] by identifying reached progress and the grounded continuation.Abstract skill and another same-type path shift preference much less, supporting state-contextualized guidance rather than generic task-relevant text.

State-Matched Routing and Contextualized · Self-Distillation

SMRC-SD routes privileged self-distillation only when the student reaches a reference-supported, action-grounded state, then conditions teacher context on that reached state. Unmatched turns retain ordinary on-policy learning, while inference removes all reference-dependent components.

Self-Distillation: SMRC-SD couples state-matched routing with contextualized self-distillation to withhold unsupported guidance and localize supervision at supported transitions.Routing selects the latest compatible reference position before contextualized guidance is constructed.
Reconstructing State Signatures: Reference and student signatures compactly encode task metadata, reference prefixes, actual history, observations, and admissible actions.The environment adapter constructs signatures at reference position k and student turn t.
Reconstructing State Signatures: Environment adapters share deterministic reconstruction, directional support checks, action grounding, and context rendering while isolating environment semantics from the learning objective.The signatures remain environment-specific but expose a common interface.
Structured-State Matching and Routing: Matching requires exact-task retrieval, state compatibility, and a nonempty grounded admissible candidate before a reference position can support distillation.The asymmetric support relation checks continuation-critical fields exactly while tolerating irrelevant extra progress.
Structured-State Matching and Routing: SMRC-SD selects the latest verified match, allowing valid alternative histories and preventing repetition of completed subgoals.Every selected position must pass both state and action-grounding checks.
State-Contextualized Guidance Construction: For matched turns, teacher context combines the complete reference path, a summary of the reached student state, and the grounded candidate.The matched position selects the candidate, but the prompt preserves the student’s actual reached state.
Integration with On-Policy Learning: SMRC-SD applies GRPO+SDL on matched turns and GRPO alone on unmatched turns, removing only the privileged loss from unsupported turns.At inference, the ordinary prompt is used without references, signatures, matching, candidates, or teacher context.

Experimental Setup

The experiments evaluate SMRC-SD on two interactive benchmarks, compare it with progressively more targeted distillation baselines, and test two Qwen model variants under fixed training configurations.

Datasets and metrics: ALFWorld tests embodied household tasks, while WebShop tests web-based product search and purchase, with actions continually changing execution state and task progress.ALFWorld state changes include location, inventory, object properties, and subgoal progress; WebShop changes the active page and task progress through navigation and option selection.
Baselines and comparisons: FullPath-SD supplies the complete successful path at every turn, whereas Routing restricts the same context to matched turns and SMRC-SD also changes the context.Vanilla uses terminal rewards only, GRPO is the base reinforcement-learning comparison, Skill-SD adds self-distillation, and SDAR gates token signals.
Implementation details: Experiments study Qwen2.5-3B-Instruct and Qwen3-1.7B across task-specific horizons and fixed checkpoints, with no validation-based checkpoint selection.Path variants share GRPO, references, initial-state rollouts, chosen-token K3, λSDL = 0.01, and no prefix replay.

Main Results

SMRC-SD consistently outperforms unconditional FullPath-SD across model families and environments, improving task performance while producing shorter, less repetitive responses and maintaining competitive training dynamics.

0.746 to 0.865: SMRC-SD raises Qwen3-1.7B ALFWorld Average@4 over unconditional FullPath-SD; Pass@4 rises from 0.836 to 0.914.
0.694 to 0.825 and 0.574 to 0.693: SMRC-SD improves Qwen3-1.7B WebShop Score and Acc over unconditional FullPath-SD.
0.766 to 0.883: With Qwen2.5-3B, SMRC-SD improves ALFWorld Average@4; Pass@4 rises from 0.852 to 0.938, and WebShop Score rises from 0.842 to 0.863.
78.8 tokens: SMRC-SD’s mean full response length is close to GRPO’s 79.5 and below FullPath-SD’s 142.6 and Skill-SD’s 255.6.SMRC-SD overtakes baselines by update 100 and remains strongest through the latter half of the run.

Analysis and Ablations

Ablations show that matched-only routing is the primary source of improvement, while complete state-contextualized guidance adds further gains. Turn-identity controls, component tests, and matcher audits support selecting executable, locally compatible continuations rather than merely increasing coverage or sparsity.

Guidance routing and construction: Routing raises Average@4 from 0.746 to 0.836, while adding complete SMRC context reaches 0.865 under the matched-only route.Dynamic Context without Routing reaches only 0.695, indicating that routing dominates and contextualized guidance adds a further gain.
Turn-selection controls: Matched and random turns have nearly identical counts, yet matched selection improves Average@4 by 0.113, showing gains depend on turn identity rather than sparsity.Applying FullPath-SD to all unmatched turns reaches only 0.750 despite greater coverage.
Teacher-context components: Neither candidate-only nor state-summary-only guidance improves over FullPath-SD + Routing, supporting their complete bundle.The state summary locates reached progress, while the candidate specifies the corresponding continuation within the full path.
Matcher comparison: On 35,712 archived turns, structured-state matching expands candidate coverage from 15.4% to 20.2% while recovering 98.8% of history matches.The coverage difference includes 1,775 structured-state-only turns and 68 history-only turns; 63 history-only turns are rejected because inventory no longer supports the continuation.
Executable-continuation audit: Replay succeeds for 792/800 history matches (99.0%) and 781/781 structured-state matches (100%), supporting executable continuations from structured matching.The two methods select the same candidate on 5,395 of 5,428 shared matches (99.4%).

Conclusion · A Chosen-Token SDL Estimator

SMRC-SD addresses state–reference mismatch by routing privileged teacher scoring only after matching the student’s execution-progress state. The chosen-token SDL estimator preserves ordinary response tokens, applies routing only to SDL, and keeps GRPO active across all trajectories.

Conclusion: SMRC-SD identifies local state–reference mismatch as the source of unreliable privileged supervision during on-policy execution.A globally successful reference may not support a continuation from the student’s reached state.
Conclusion: SMRC-SD matches hand-engineered execution-progress signatures before routing path-conditioned teacher scoring.The compatibility decision is used to determine whether the reference can guide the current student state.
A Chosen-Token SDL Estimator: The inherited K3 estimator defines token-level teacher loss, a non-negative K3 estimate, and a current-versus-rollout policy correction.The estimator uses the synchronized detached teacher and the rollout policy alongside the current student.
A Chosen-Token SDL Estimator: Numerical stability is enforced by upper-clipping −δt,i at 20 and ηt,i at 10 before exponentiation.The implementation passage specifies both clipping thresholds and their purpose.
A Chosen-Token SDL Estimator: The eligible-token mask retains ordinary response tokens, removes tokenizer-defined special tokens, and does not discard reasoning tokens.It also leaves the denominator based on the unmodified response-token mask, including routed-out turns.
A Chosen-Token SDL Estimator: Selecting fewer turns reduces aggregate SDL contribution rather than strengthening selected tokens because normalization counts all original response tokens.The route mask suppresses only SDL, while GRPO remains active on every trajectory.
A Chosen-Token SDL Estimator: FullPath-SD and SMRC-SD use the same estimator and normalization, while the Random Turns control samples its turn set before loss evaluation.The teacher scores only chosen tokens in the student’s response.

B Matcher-Conditioned Teacher Intervention Details … J Interpretive Limits

The paper’s supplementary analyses operationalize SMRC-SD with matcher-conditioned teacher interventions, task-indexed references, environment-specific state adapters, and matched-only routing. Audits, ablations, overhead measurements, and interpretive limits support state-compatible guidance while clarifying that matcher labels and replay validation do not establish action correctness or recall.

B Matcher-Conditioned Teacher Intervention Details: The fixed intervention suite contains 1,200 turns from 458 trajectories and 79 training games, split by matched agreement, matched divergence, and unmatched terminal-success turns.Retained turns satisfy execution, admissibility, formatting, and progress checks; teacher scoring changes only privileged context while holding state, prompt, and sampled response fixed.
B Matcher-Conditioned Teacher Intervention Details: +1.424 is SMRC-SD’s pooled margin change over no privilege, versus +1.158 for matched FullPath; SMRC-SD exceeds FullPath by +0.266.The ordering SMRC-SD > FullPath > shuffled path > other same-family path holds across outcomes and scorer endpoints.
B Matcher-Conditioned Teacher Intervention Details: +0.070 is the A–C FullPath contrast, while adverse SMRC-SD effects occur in only 1 of 800 anchor–scorer evaluations, or 0.1%.Unmatched C anchors show selective interference: FullPath lowers sampled-action score by more than 0.01 on 8.3% and 6.5% of anchors at the two scorer endpoints.
C Reference Availability and Construction: Training retrieves one canonical reference per stable task identity after rollout, while evaluation removes references, adapters, matchers, summaries, candidates, and teachers from ordinary benchmark starts.ALFWorld indexes 3,553 training-game references; WebShop constructs 6,910 deterministic goal traces and trains only on indices 500–6909.
D State Adapters and Guidance Construction: Environment-specific adapters reconstruct symbolic execution progress and expose directional support, admissible-action grounding, and teacher-context rendering rather than learning a universal state estimator.SMRC-SD represents local guidance as a singleton or compatible action set; unmatched turns receive wt = 0 and no SDL.
E Structured-State Matcher Audit: 95.1% of 1,604 successful WebShop option-set turns have sampled actions inside the rendered compatible set, confirming admissible local guidance in the offline audit.Structured-state matching also supports continuation despite histories lacking sufficiently similar suffixes.
F Training and Prompt Overhead: All 781 structured-state replays succeed, compared with 792 of 800 history replays; these audits validate sampled continuation matches but not matcher recall or unmatched-turn action quality.The structured-state and history samples are matcher-specific, and the archived-selector comparison is descriptive rather than an independent significance test.
G Response Length and Style: SMRC-SD remains close to GRPO in response length, whereas FullPath-SD and especially Skill-SD are longer and more repetitive; training-only indexing and matching add no deployment prompt tokens or inference component.Shared experiments use GRPO for every rollout, add SDL only for distillation methods, and do not replay reference prefixes into the environment.