Predicting Privileged Information Effectiveness for On-Policy Self-Distillation

1 Introduction

• PI formats include reference trajectories with actions, observations, and thoughts, alongside compressed Summary, Skill, and Checklist hints.Checklists derive task requirements from reference and failed student trajectories.Source: Figure 1 · p. 2
• Prior studies covered limited PI formats, models, and tasks, leaving unclear what produces useful student learning signals and whether PI quality can be assessed before training.OPSD conditions the same model on PI unavailable at inference time, without a separate teacher model.Source: §1 · pp. 1–2
• The study evaluates 16 PI configurations across ALFWorld, AppWorld, and τ2-retail using three student models that vary in scale and reasoning mode.The models are Qwen3-4B-Instruct, Qwen3-4B-Thinking, and Qwen3-30B-A3B-Instruct.Source: §1 · pp. 2–3
• PI effectiveness depends on content, generation source, and student model: information-dense trajectories and more capable generators generally yield stronger gains, while thinking models benefit less consistently.Skills are generally strongest among hint formats, and adding observations consistently improves Instruct students on ALFWorld and AppWorld.Source: §1 · p. 3
• Error-step KL contrast better distinguishes useful PI and improves per-task selection over random selection when candidates vary in source or strength.Sampled-token reverse KL tracks performance in eight of nine settings but misses shift direction; PI utility is unreliable because of information leakage.Source: §1 · p. 3
• Combining OPSD with the most effective PI can outperform a strong GRPO baseline.The introduction also presents the combination as outperforming matched GRPO baselines.Source: §1 · p. 3

2.1 On-Policy Self-Distillation and PI Construction

• OPSD uses shared student–teacher parameters, with the PI-conditioned teacher supplying token-level supervision on trajectories generated by the student.Because the policies differ only in access to PI, PI construction directly determines the training signal.Source: §2.1 · pp. 3–4
• PI includes action-only, action-plus-observation, action-plus-reasoning, and combined trajectories, alongside summaries, skills, and checklists.Skills combine strategies, mistake analysis, and reusable guidance; checklists provide structured task criteria rather than a target response.Source: §2.1 · p. 4

2.2 Offline Metrics

• The study compares PI utility, sampled-token reverse KL, and error-step KL contrast as offline measures of PI effectiveness.Error-step contrast compares suppression at judged error steps with task- and position-matched baselines; positive values indicate stronger targeting of identified errors.Source: §2.2 · pp. 4–5

2.3 Experimental Setup

• Experiments evaluate three Qwen3 students on ALFWorld, AppWorld, and τ2-retail, with offline metrics computed on training or development tasks disjoint from evaluation sets.Reported performance is pass@1 from the best three-checkpoint window.Source: §2.3 · p. 5

3.1 Which PI Formats Work Best?

• Trajectory PI generally outperforms compressed hints, but richer context can hurt the already-capable thinking model, especially on τ2-retail.Adding observations and teacher thinking tokens benefits Instruct models, while the thinking model’s pass@4 of 88.9 on τ2 suggests less room for additional context to help.Source: Figure 2 · p. 6, §3.1 · p. 6
• Skills are often the strongest hint format, leading on ALFWorld for all three students and on several AppWorld and τ2-retail student settings.Skills add mistake analysis and reusable strategies beyond a solution description.Source: §3.1 · p. 6
• GLM-5.2 PI almost always beats Qwen3.7-Plus PI; for 4B-Instruct, GLM wins 20 of 21 comparisons and leads by 36.0 points on ALFWorld hints.The source gap stays within 3.2 points for trajectory formats, making source choice especially consequential when generating hints.Source: §3.1 · pp. 6–7
• For 4B-Instruct, self-generated Summary and Skill hints improve performance on all three benchmarks.The results indicate that the student can generate its own hints from a reference solution and benefit from OPSD.Source: §3.1 · p. 7

3.2 What Do PI Metrics Reveal?

• PI utility can reward leaked answer-specific information rather than transferable training signals, so higher utility does not imply larger OPSD gains.On ALFWorld and τ2-retail, the PI-conditioned model can exploit answer information without reproducing the required interaction sequence.Source: §3.2 · p. 7
• Sampled-token reverse KL correlates positively with post-training performance in eight of nine settings, but a large shift can still overwrite useful behavior.For 4B-Thinking on τ2, the correlation is ρ = −0.84; shift direction and location matter alongside magnitude.Source: §3.2 · p. 7
• Across all nine student–benchmark combinations, error-step KL contrast is positively correlated with post-training performance (ρ = 0.10–0.94).The contrast measures whether PI more strongly suppresses sampled tokens at judged error steps than at comparable positions from the same task.Source: Figure 3 · p. 8, §3.2 · p. 8

4.1 Combined Training Objective

• The combined objective weights the GRPO and OPSD advantages, with λ ∈ [0, 1] controlling their relative contribution.The experiments use Qwen3-4B-Instruct-2507 and compare four λ values across three PI formats with matched GRPO-only controls on ALFWorld, AppWorld, and τ2.Source: §4.1 · p. 9

4.2 Effect of Objective Weighting

• Trajectory PI remains strongest across most λ values, followed by Skill and Checklist PI, with that ordering consistent through λ = 0.99 on all benchmarks.The paper concludes that adding outcome-based rewards does not eliminate differences in PI quality.Source: §4.2 · pp. 9–10
• Trajectory PI exceeds matched GRPO-only scores on ALFWorld, AppWorld, and τ2 at λ = 0.99.Scores are 95.13 versus 84.96 on ALFWorld, 43.90 versus 42.91 on AppWorld, and 55.04 versus 48.27 on τ2.Source: §4.2 · p. 10
• Increasing λ generally improves all three PI formats, with the largest gains for weaker Checklist and Skill formats on ALFWorld and AppWorld.Source: §4.2 · p. 10
• PI-guided training, particularly with Trajectory PI, improves rapidly early, but all three PI-guided ALFWorld curves plateau or decline after step 100 while GRPO continues improving.The section suggests repeated static PI may become redundant after one epoch; on AppWorld, Trajectory PI reaches nearly 100% training success at λ = 0.9 but 39.14 test pass@1, indicating possible overfitting on small, less diverse training sets.Source: §4.2 · p. 10

5 Which Offline Metric Should Guide PI Choice?

• The controlled studies isolate answer leakage and test per-task selection from candidate pools scored by D and error-step KL contrast against random selection.The Passcode task is scored only on the submitted task-specific key; the selection study compares held-out pass@1 through training.Source: §5.1 · pp. 10–11, §5.2 · pp. 11–12

5.1 ALFWorld-Passcode: Leaked Information Can Distort PI Utility

• A PI containing only the correct key yields 98.1 pass@1 for the student seeing it, yet OPSD training leaves performance unchanged; both KL metrics rank it lowest among the four PIs.The passage explains that copying a key available only in the prompt cannot transfer to a student trained without seeing that PI.Source: §5.1 · p. 11
• Changing the key alone shifts utility by 25 points while leaving training performance and both KL metrics essentially unchanged; changing the trajectory instead alters training performance and KL rankings.The mismatched trajectory nearly halves training performance to 45.0, even though its utility is higher than the task’s own trajectory in standard ALFWorld or with the decoy key.Source: §5.1 · p. 11

5.2 Selecting PI per Task with Offline Metrics

• Error-step KL contrast improves on random PI selection when candidates come from different sources, with the greatest benefit in pools mixing winning and failed rollouts.Among the student’s winning rollouts, it leads random selection by 5.1 points at step 40, but the gap closes by the end of the third epoch; for winning trajectories from one strong generator, it matches random selection.Source: §5.2 · p. 12, Figure 6 · p. 12
• Reverse-KL selection helps little except for candidates from the single strong GLM-5.3-Flash generator, performs slightly worse than random in other pools, and fails badly on mixed self-win and self-loss rollouts.Failed rollouts shift the student’s distribution most, so ranking candidates by D can favor misleading PI.Source: §5.2 · p. 12, Figure 6 · p. 12

5.3 Practical Recommendations

• The authors recommend using error-step KL contrast when PI quality is inconsistent, combining selected PI with a strong GRPO signal, and reducing or removing its weight when its benefit plateaus.For small training sets, they recommend choosing PI schedules and diversity using held-out performance rather than training success alone.Source: §5.3 · pp. 12–13

On-policy self-distillation and PI construction

• Conventional distillation trains on teacher-generated sequences, whereas OPD supervises student-generated trajectories on the student’s own state distribution [1] [31].Source: §6 · p. 13
• Prior OPSD work uses answers, reasoning traces, environment feedback, demonstrations, successful trajectories, skills, mistake analyses, hindsight feedback, and structured rubrics [35] [7] [22] [19] [26] [33] [6] [20].The authors note potential context-transfer effects, reduced output diversity, suppressed uncertainty expression, and disrupted self-correction in thinking models [9] [11] [13] [17].Source: §6 · p. 13

Combining OPSD with reinforcement learning

• Prior combinations of RL and self-distillation vary in how they route or weight signals, while this study uses a simple weighted combination of GRPO and OPSD to examine complementarity [29] [14] [18] [26] [16] [27] [30] [10].Source: §6 · p. 13

7 Conclusion

• Across 16 PI configurations, three agentic benchmarks, and three student models, information-dense trajectories, environment observations, mistake-aware strategies, and more capable generators often improve learning, but thinking models benefit less consistently.Long reasoning traces can make supervision difficult to target at actual errors.Source: §7 · p. 13
• Sampled-token reverse KL measures the magnitude of PI-induced distribution shifts, while error-step KL contrast measures whether shifts target identified mistakes; the metrics track post-training performance more strongly than unreliable PI utility.Error-step KL selection improves over random, whereas reverse-KL selection can favor misleading PI.Source: §7 · pp. 13–14
• Strong PI can improve on matched GRPO baselines and accelerate early learning, but static PI may become redundant or encourage overfitting on small datasets.The conclusion calls for exploring diverse PI instances and adaptive schedules that reduce or reshape distillation as the student learns.Source: §7 · p. 14

A PI Construction

• Reference trajectories retain actions, following observations, preceding reasoning, or combinations of these components; the appendix describes successful rollouts sampled from expert models.Source: A PI Construction · p. 17
• Hint constructions include summaries of actions and strategy, skills combining success and mistake analysis with a workflow, and checklists derived from paired successful and failed trajectories.Source: A PI Construction · p. 17

B.1 Benchmarks

• ALFWorld uses text commands for household tasks, AppWorld uses Python APIs across simulated apps, and τ2-retail tests customer-service interactions with database tools.ALFWorld evaluates pooled success on valid_seen and valid_unseen tasks; AppWorld evaluates on test_normal, and τ2-retail on 114 retail base tasks.Source: §B.1 · pp. 17–18

B.2 Training Configuration / B.3 Evaluation

• Training uses AdamW, vLLM, and distributed training with FSDP2 or Megatron; evaluation samples four no-hint rollouts per task using benchmark-specific settings.Checkpoints are evaluated every 25 steps on AppWorld and τ2-retail and every 10 steps for ALFWorld hint arms.Source: §B.2 · p. 18, §B.3 · p. 18

B.4 GRPO and GRPO + OPSD

• GRPO normalizes each rollout’s reward within eight task rollouts, while GRPO + OPSD mixes the GRPO and per-token OPSD advantages.For ALFWorld, GRPO omits reward standard-deviation normalization; λ = 1 recovers GRPO.Source: §B.4 · p. 18

C Offline Metric Protocols

• Offline metrics use 200 sampled ALFTrain tasks, AppWorld train and dev splits, and 132 matched τ2-retail training tasks, disjoint from post-training evaluation sets.Source: C Offline Metric Protocols · p. 18

C.1 PI Utility

• PI utility is the paired per-task performance difference when the model is evaluated with versus without PI.Each condition uses four rollouts per task, and Figure 3 and Table 4 report Δpass@1.Source: §C.1 · p. 19

C.2 Sampled-token Reverse KL

• Sampled-token reverse KL averages the absolute log-ratio over student rollouts scored by the same model conditioned on PI.Every PI is scored on the same sampled rollouts, so differences in D reflect the PI.Source: §C.2 · pp. 19–20

C.3 Error-step KL Contrast

• Error-step KL contrast compares teacher–student divergence at judge-labeled error steps in failed rollouts against a task- and position-matched baseline.Labels are produced by a model judging serialized trajectory steps, and every PI is evaluated on the same flagged steps.Source: §C.3 · p. 20

• 51.9–96.5% of ALFWorld rollouts under longer trajectory hints begin with the reference’s goal-completing final action, versus at most 1% under Action hints.Nearly every such first action fails, and the model repeats the final action in 51–95% of longer-format rollouts.Source: D How Information Leakage Distorts PI… · p. 20, Table 2 · p. 21
• Action hints yield the highest utility for almost every student and source despite training worst, because the model copies 71–89% of its actions from the leaked sequence.Longer hints can prompt out-of-order copying; for the 4B thinking student, those rollouts succeed 7–11 points less often than other rollouts of the same task.Source: D How Information Leakage Distorts PI… · p. 21
• 33% of the 4B thinking student’s τ2-retail episodes with Action + Obs. + Thou. make no tool call, and 116 of 130 such episodes quote an identifier or price from the hint.The no-tool-call rate falls to 0.7% without reasoning and is 1.5% for 4B-Instruct with the same hint.Source: D How Information Leakage Distorts PI… · p. 22
• Utility can rise or fall with inference-time reuse of leaked solutions, although the OPSD student never sees PI and learns only from the teacher’s signal.Source: D How Information Leakage Distorts PI… · pp. 20, 22

E PI Ablations

• Failed self-rollouts can shift the teacher about as much as successful rollouts, yet have lower error-step KL contrast; large shifts alone do not indicate useful PI.On self-rollout selection pools, failed rollouts have larger D but lower Δd^-_err than successful ones.Source: E PI Ablations · p. 22
• Padding Action to full-trajectory length changes D and Δd^-_err only slightly, and OPSD performance remains close to unpadded Action.The larger shifts from trajectory PIs therefore appear to reflect content more than length.Source: E PI Ablations · p. 22
• 87.9 pass@1 with the task’s own trajectory falls to 34.8 with mismatched PI and 5.4 with other-domain PI, below the untrained baseline.All three metrics place both corrupted PIs below the two references, with other-domain PI lowest on each.Source: E PI Ablations · p. 22, Figure 7 · p. 23

F Per-Task PI Selection

• The candidate pools include winning trajectories from eight generators, eight GLM-5.3-Flash trajectories, and successful or mixed success-and-failure rollouts from the untrained student.Some tasks have fewer candidates when a source did not solve them or the student produced fewer than eight failures.Source: F Per-Task PI Selection · p. 23
• Each candidate is scored using error-step KL on other task rollouts, with self-pool candidates excluded from their own scoring rollouts.Scoring uses 128 rollouts per task on 200 offline tasks and 16 on the remaining training tasks; self-pool scoring uses the other 15 rollouts.Source: F Per-Task PI Selection · p. 23
• Runs fix one PI per task, drawn uniformly from its pool or from the top three candidates by D or Δd−.Source: F Per-Task PI Selection · p. 23
• Source pools train for one epoch on 1,600 tasks, while self pools train for four epochs on 483 tasks; evaluation uses multiple seeds and checkpoints.The source pools are evaluated every 25 steps and self pools every 20 steps; reported self-pool checkpoints stop at step 100.Source: F Per-Task PI Selection · p. 23

G.1 OPSD Performance

• Table 3 reports best-window performance for every student and PI used in Section 3.1.Source: §G.1 · p. 24

G.2 Offline Metrics

• Table 4 defines utility U (∆pass@1), sampled-token reverse KL D, and error-step KL contrast ∆d− err as offline PI metrics.Table 3 separately reports student and PI OPSD performance using best-window pass@1 and pass@4.Source: §G.2 · p. 24, Table 3 · p. 25, Table 4 · p. 26

H.1 Agent Prompts

• Teacher inputs prepend privileged hints derived from successful trajectories to the student’s chat-templated prompt, while students are evaluated without seeing the hints.The example ALFWorld, AppWorld, and τ2-retail prompts use different interaction formats, including action commands, Python code, and customer-service messages or tool calls.Source: §H.1 · pp. 27–28

H.2.1 Summary

• The summary prompt asks for a concise plain-text hint describing key actions and strategy, focusing on what an agent achieved if it did not complete the task.Source: §H.2.1 · p. 28

H.2.2 Skill

• Benchmark-specific skill analyzers distill transferable strategies and mistakes from completed trajectories and outcomes, while avoiding literal task-specific solutions.The AppWorld and τ2-retail prompts additionally emphasize domain mechanics, such as scoping records and following authentication, policy, and confirmation protocols.Source: §H.2.2 · pp. 28–29

H.2.3 Checklist

• The checklist analyzer contrasts a stronger agent’s successful attempt with a weaker agent’s failed attempt to produce transferable, imperative pre-action guidance.It organizes guidance into before_acting, while_acting, before_finishing, and pitfalls, and instructs the analyzer not to pad empty sections or repeat instructions.Source: §H.2.3 · p. 30

H.3 Example Hints

• The ALFWorld example presents an action-and-observation hint and a checklist that guides object search, cleaning, placement, and success confirmation.The checklist also warns against acting from the wrong location, repeating failed commands, or assuming that finding an object completes the task.Source: §H.3 · pp. 31, 33
• For the Spotify task, the agent used genre and follower-count API filters, paginated through all results, and followed every qualifying artist.The hint emphasizes using built-in filters and exhausting result pages before applying the mutation.Source: §H.3 · p. 34
• The τ2-retail agent authenticated the user, clarified the abbreviated city and missing state, obtained explicit confirmation, and successfully updated the default address.It compared the current and proposed addresses before the change and confirmed the update afterward.Source: §H.3 · p. 35
• The address-change hints also stress checking each field-to-parameter mapping and inspecting the returned object, correcting misplaced values before reporting success.They specify assigning an empty string when a field, such as an apartment number, has no value.Source: §H.3 · p. 36