OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

1 INTRODUCTION

• Outcome-based RL provides stable task-level optimization, but sparse, delayed rewards do not identify which intermediate decisions caused an outcome, limiting fine-grained credit assignment.In long-horizon interaction, early mistakes or repeated invalid actions can derail an episode before their effects are reflected in a terminal reward.Source: §1 · pp. 1–2
• OPID achieves the strongest average performance on ALFWorld and WebShop while remaining competitive on Search-based QA, improving over outcome-only RL and skill-distillation baselines.The evaluation covers ALFWorld, WebShop, and Search-based QA with models at different scales.Source: Figure 1 · p. 2, §1 · p. 3
• Unlike skill-conditioned methods relying on external libraries, retrieved files, or maintained memories, OPID extracts distribution-matched hindsight skills directly from completed on-policy trajectories.External skills require maintenance and may mismatch the current policy's state distribution, especially when multi-turn deviations cause state drift.Source: §1 · pp. 2–3
• OPID uses episode-level skills for global workflows or failure-avoidance rules and step-level skills for local guidance at critical timesteps, routing to step-level skills first.When no critical timestep is identified, the method falls back to the broader episode-level skill.Source: §1 · pp. 2–3
• The routed skill is added to interaction history so the old policy can re-score the same response; its token-level log-probability shift supplies a distillation advantage combined with the episode advantage.This preserves outcome-based RL as the primary objective, and inference requires no analyzer, external skill retrieval, or privileged context.Source: §1 · p. 3

• Outcome-based agent RL, including GRPO, provides stable on-policy optimization, but sparse terminal rewards do not identify which intermediate decisions drive success or failure (Shao et al., 2024).OPID keeps outcome-based RL as its optimization backbone while adding decision-level supervision from completed trajectories.Source: §2 · p. 3
• On-policy self-distillation supplies token-level targets, while OPID builds its privileged branch from hindsight about states, actions, and failures encountered by the current policy.Existing approaches may use generic revision contexts, external hints, or task-level feedback transformations; OPID’s dense signal complements task-level RL and is tied to the current policy’s experience.Source: §2 · p. 3
• Unlike external skill libraries, retrieved files, or persistent memories, OPID extracts hierarchical hindsight skills from completed on-policy trajectories and routes them by decision criticality.Natural-language skills can encode subgoal decompositions, action templates, and failure-avoidance rules (Lu et al., 2026b; Wang et al., 2026; Lu et al., 2026a; Wu et al., 2026b). Figure 2 describes prioritizing step-level skills at critical states, re-scoring the same response with and without the routed skill, and using the token-wise log-probability difference as a dense advantage complementary to episode-level RL; on-policy extraction avoids retrieval costs and state-distribution mismatch.Source: §2 · pp. 3–4, Figure 2 · p. 4

3.1 PROBLEM FORMULATION

• OPID models agentic tasks as partially observable decision processes in which the agent acts from interaction history and receives trajectory-level outcome rewards.The formulation includes hidden states, observations, actions, transitions, and rewards; benchmark feedback is often sparse and terminal.Source: §3.1 · pp. 4–5

3.2 ON-POLICY SKILL EXTRACTION

• OPID extracts episode-level skills as global workflows or failure-avoidance rules and step-level skills as local knowledge for pivotal, outcome-sensitive decisions.An LLM-based analyzer identifies critical timesteps in completed trajectories; episode skills provide broad default guidance, while step skills are precise but sparse and state-dependent.Source: §3.2 · p. 5

3.3 CRITICAL-FIRST SKILL-CONDITIONED SELF-DISTILLATION

• At critical timesteps OPID routes step-level skills, otherwise episode-level skills, then compares the old policy’s scores for the same response under original and skill-augmented histories.The resulting token-level log-probability shift supplies dense guidance without regenerating the sampled response or requiring an external expert action.Source: §3.3 · pp. 5–6

3.4 POLICY OPTIMIZATION WITH SKILL ADVANTAGE

• OPID combines group-relative outcome feedback with token-level skill supervision in a clipped policy objective, keeping outcome reward as the primary RL signal.The analyzer, routed skills, and skill-conditioned scoring are training-only; at inference, the policy acts from ordinary interaction history without retrieval or privileged context.Source: §3.4 · pp. 6–7

4.1 EXPERIMENTAL SETTING

• Experiments evaluate Qwen2.5-3B/7B-Instruct and Qwen3-1.7B-Instruct on ALFWorld, WebShop, and Search-based QA.Training uses batch sizes of 16 for ALFWorld and WebShop and 128 for Search-based QA, with 150 steps across environments; comparisons include prompting-based and training-based baselines.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

• OPID improves over GRPO in most model–domain combinations and matches or surpasses strong hybrid and self-distillation baselines in several aggregate settings.On Qwen2.5-3B, OPID scores 84.3 vs. 75.0 on ALFWorld, 45.0 vs. 36.4 on Search-based QA, and 74.2 vs. 63.3 on WebShop. OPID is evaluated without skill input and exceeds Skill-GRPO on ALFWorld by +24.1, +20.5, and +37.8 points for Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B, respectively.Source: §4.2 · pp. 8–9

4.3 TRAINING DYNAMICS

• On ALFWorld, OPID diverges from GRPO during middle-stage training and maintains higher performance while reducing average episode length to 15–16 steps versus GRPO’s 17–18.The analysis attributes more direct action sequences to episode-level workflows and step-level guidance at critical decisions.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 SAMPLE EFFICIENCY

• With 60% of ALFWorld training data, OPID reaches 71.9, near full-data GRPO’s 75.0; with 80%, OPID scores 78.9 and surpasses it.OPID improves over GRPO across all data scales, with absolute gains ranging from +9.3 to +20.3 points.Source: §4.4 · pp. 9–10, Figure 4 · p. 9

4.5 CROSS-DOMAIN GENERALIZATION

• OPID transfers to ALFWorld Unseen, achieving an average success rate of 78.6 and outperforming GRPO by +7.7 points.Gains are concentrated on Look (+26.7) and Heat (+18.5), with competitive performance on the remaining task types.Source: Figure 5 · p. 9, §4.5 · p. 10

4.6 ABLATION STUDIES AND ANALYSIS

• The complete skill hierarchy performs best on both domains, and critical-first routing improves ALFWorld average performance over non-routed guidance.Removing episode-level skills lowers ALFWorld average from 84.3 to 74.1 and WebShop success rate from 74.2 to 67.2; removing step-level skills lowers them to 79.1 and 65.6. Critical-first routing improves ALFWorld average by +6.8 points (84.3 vs. 77.5).Source: §4.6 · p. 10, Table 2 · p. 10, Table 3 · p. 10
• In an ALFWorld clean-and-place example, OPID completes the task in six steps, while GRPO hallucinates a target object and fails within the 30-step limit.OPID follows a locate–clean–place workflow, whereas GRPO substitutes a spoon for the spatula and does not complete placement.Source: §4.6 · p. 11, Figure 6 · p. 11

5 CONCLUSION

• Across embodied, web, and search-based benchmarks, OPID improves agent learning with trajectory-derived hindsight skills while retaining outcome-based RL and avoiding external skill libraries, retrieval, or privileged inference context.The paper frames agent trajectories as reusable records of decision knowledge distilled back into the policy.Source: §5 · p. 11

A THEORETICAL ANALYSIS

• The analysis relates OPID’s teacher advantage to sampled-token reverse-KL updates, motivates on-policy context collection, and justifies critical-first routing under a natural specialization assumption.Source: A THEORETICAL ANALYSIS · p. 14

A.1.1 NOTATION

• OPID defines behavior and detached skill-conditioned teacher distributions at each token context, with the observed token sampled from behavior; their log-probability gap gives the skill advantage.A positive gap means the skill-conditioned teacher assigns greater probability to that token than the behavior policy.Source: §A.1.1 · p. 15

A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES

• On-policy distillation trains at contexts generated by the student or a behavior policy, and output-space supervision can be full-vocabulary, Top-K, or sampled-token; OPID uses sampled tokens.Source: §A.1.2 · p. 15
• Full-vocabulary matching can minimize forward KL, reverse KL, or generalized Jensen–Shannon divergence; forward KL emphasizes teacher-supported coverage, reverse KL is typically more mode-seeking, and α = 1/2 recovers symmetric JSD.Source: §A.1.2 · p. 15
• Top-K matching retains multi-token distribution supervision at reduced cost, but discards probability mass outside its selected support and is a support-dependent truncation of full reverse KL.Common supports are selected by either the student or teacher.Source: §A.1.2 · pp. 15–16
• OPID’s sampled-token signal estimates per-context reverse KL from one behavior-sampled token with importance weighting, requiring support coverage; it uses less teacher information and has higher Monte Carlo variance.Only the teacher probability of the realized token is required.Source: §A.1.2 · p. 16
• The skill-only unclipped surrogate matches the skill-induced component of PPO’s first-order update at the behavior policy, but away from that point clipping couples outcome and skill signals.The unclipped decomposition therefore does not describe the complete clipped objective globally.Source: §A.1.2 · p. 17

A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE

• With rollout histories, routed skills, and teachers held fixed and common support, the unclipped OPID loss equals λskill times reverse-KL minus behavior-relative KL.The behavior-relative KL is minimized at the behavior policy; common support holds for standard finite-logit softmax models.Source: §A.2 · pp. 17–18
• Unlike direct reverse-KL, OPID is not globally equivalent; around the behavior policy, the losses share value and gradient, with second-order discrepancy.The losses can have different global optima, so the equivalence is local rather than global.Source: §A.2 · pp. 18–19
• Adding a behavior-KL penalty with coefficient β = λskill recovers reverse-KL exactly; a different reference distribution or coefficient leaves a residual term.Exact cancellation requires the penalty to use the same behavior distribution under the ordinary context.Source: §A.2 · p. 19
• The relative-KL decomposition applies to the unclipped loss, while combined PPO clipping matches its value and first derivative at the behavior policy but couples signals at clipping boundaries.Beyond a clipping boundary, the exact decomposition no longer applies to the complete clipped objective.Source: §A.2 · pp. 19–20
• When outcome rewards tie, a skill teacher supplies a per-context token-level signal whenever qi ≠ bi, although gradients across contexts may cancel.At the behavior policy, the per-context logit gradient is zero for every token if and only if qi = bi.Source: §A.2 · p. 20

A.3 ON-POLICY OCCUPANCY MATCHING FOR DISTILLATION

• For bounded per-context losses, the occupancy-induced expectation gap is controlled by context-distribution mismatch in total variation and KL, and is zero when dµ = db.The proposition isolates changes in ordinary autoregressive context distribution.Source: §A.3 · p. 21
• Applying the occupancy bound to the signed OPID skill loss requires a bounded range, such as probability flooring, log-ratio clipping, or a compact parameter neighborhood.More general bounds require suitable moment or tail conditions.Source: §A.3 · p. 21
• The occupancy result controls only the outer context distribution for the same per-context loss map, not policy-induced changes in hindsight skills or routed teachers.It does not by itself control other trajectory-dependent quantities that change with the data-collection policy.Source: §A.3 · p. 21

A.4 CRITICAL-FIRST HIERARCHICAL ROUTING

• Critical-first routing selects between episode- and step-level teachers, and the routed teacher supplies q_i in the OPID skill advantage.The routing analysis characterizes this selection relative to an oracle choice between candidate teachers.Source: §A.4 · p. 23
• Under teacher specialization, perfect criticality detection recovers the lower-error candidate pointwise; imperfect detection incurs excess error controlled by detector error and the candidate-error gap.The comparison is between teacher approximation errors relative to the ideal teacher.Source: §A.4 · pp. 22–23
• The routing result concerns teacher approximation error, not the student–teacher reverse-KL loss; without further assumptions, it does not directly bound that loss.The candidate-teacher criterion is distinct from D_KL(pθ,i∥q_i) in L_RKL.Source: §A.4 · p. 23

A.5 SUMMARY

• The unclipped OPID skill loss is an exact relative-KL loss under fixed rollout histories, routed skills, and detached distributions.This is the decomposition analyzed in Proposition 1.Source: §A.5 · p. 23
• When on-policy collection matches the behavior-policy distribution, the occupancy term for ordinary autoregressive contexts is zero, eliminating outer context-distribution mismatch.This is the condition analyzed in Proposition 2.Source: §A.5 · p. 23
• At the behavior policy, the unclipped skill loss is first-order equivalent to scaled reverse-KL distillation.The summary identifies this as a property of the unclipped OPID skill loss.Source: §A.5 · p. 23

• The protocol details are organized by datasets, baselines, and implementation.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 23

B.1 DATASETS

• Experiments span embodied reasoning, web navigation, and search-augmented QA, with 2,400 ALFWorld, 2,400 WebShop, and 19,200 search-QA training examples.Training is conducted separately for each benchmark setting.Source: §B.1 · pp. 23–24
• ALFWorld tests six task types—Pick, Look, Clean, Heat, Cool, Pick2; WebShop reports normalized completion and binary exact success; search QA spans seven benchmarks.The search-QA benchmarks are Natural Questions, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.Source: §B.1 · pp. 23–24

B.2 BASELINES

• Unless starred, methods use ordinary environment history without privileged context; starred methods receive a natural-language skill at validation or test, and reproduced baselines match OPID’s training and evaluation setup.The matched setup includes backbone, environment wrappers, rollout budget, task batch, training steps, and evaluation protocol when applicable.Source: §B.2 · pp. 24–25
• Vanilla uses the unmodified backbone; Skill-Prompt∗ adds a retrieved test-time skill, while GRPO (Shao et al., 2024) uses outcome rewards and Skill-GRPO uses skills during training only.Skill-GRPO∗ retains the skill at validation and test, matching its train-time conditioning.Source: §B.2 · pp. 24–25
• OPSD (Zhao et al., 2026) uses privileged teacher context for dense token targets; GRPO+OPSD combines that distillation loss directly with sequence-level GRPO.The OPSD teacher re-scores the student’s on-policy prefixes, with the privileged context absent at inference.Source: §B.2 · p. 25
• Skill-SD (Wang et al., 2026) conditions only its teacher on retrieved skills; RLSD (Yang et al., 2026) weights GRPO updates by log-probability gaps, while SDAR (Lu et al., 2026a) gates auxiliary distillation.RLSD keeps update direction anchored to outcome advantage; SDAR leaves the GRPO advantage unchanged and evaluates the student without privileged skill context.Source: §B.2 · p. 25

B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES

• Algorithm 1 collects on-policy rollout groups and episode advantages, then performs hierarchical skill extraction, critical-first routing, paired scoring, and clipped policy optimization.The procedure is summarized as including all these stages; the supplied algorithm excerpts show rollout collection, routing, scoring, and optimization steps.Source: §B.3 · pp. 25–26
• Extracted episode-level skills capture reusable global workflows, whereas critical-step skills focus on sparse local decisions that influence the final outcome.Table 5 presents representative skills from successful and failed trajectories across ALFWorld, WebShop, and Search-based QA.Source: §B.3 · pp. 25–26

B.4 IMPLEMENTATION DETAILS

• ALFWorld uses the macro-average of success rates across task types; Search-based QA uses unweighted macro-average accuracy across seven datasets; WebShop reports normalized Score and exact-success Succ.WebShop Score is the environment’s mean normalized task score multiplied by 100, while Succ. is the percentage of tasks with exact success.Source: §B.4 · p. 26
• After each episode, an LLM analyzer receives an ordered record of the task prompt, step-indexed observations, policy responses/actions, environment feedback, and terminal outcome, then produces an episode-level skill and sparse critical-step skills.Step indices are zero-based, consistent with Table 5.Source: §B.4 · pp. 26–27
• Table 5 shows one successful and one failed trajectory per dataset, with episode-level skills summarizing reusable global behavior and critical-step skills targeting sparse decision points.Step indices are zero-based analyzer keys.Source: Table 5 · p. 27
• The default analyzer is GLM-5.2 (Z.ai, 2026), run at temperature 0.4 with a maximum output length of 4096 and at most five critical steps for ALFWorld and WebShop or two for Search-based QA.Source: §B.4 · p. 27
• Training uses Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct (Yang et al., 2024), and Qwen3-1.7B-Instruct (Yang et al., 2025) for 150 update steps; batch sizes are 16 for ALFWorld and WebShop and 128 for Search-based QA.Table 6 records the remaining hyperparameters required for exact reproduction.Source: §B.4 · p. 27, Table 6 · p. 28
• Training is conducted on 8 Nvidia A800 80G GPUs.Source: §B.4 · p. 27

C.1 DETAILED SAMPLE EFFICIENCY COMPARISON

• Trajectory-derived hindsight skills help OPID extract more supervision from each rollout, making outcome-based RL less dependent on many environment interactions.The passage presents this as an explanation for OPID’s sample-efficiency gains.Source: §C.1 · p. 28
• OPID gains +15.6 points with 60% of the data and +20.3 points with 80% of the data over GRPO.These gains are especially large in the low- and mid-data regimes.Source: Table 7 · p. 28, §C.1 · p. 28
• OPID improves ALFWorld success rates over GRPO across all training-data budgets.Table 7 reports success rates under different fractions of the training data.Source: Table 7 · p. 28, §C.1 · p. 28

C.2 CROSS-DOMAIN GENERALIZATION

• The unseen-split gains indicate that OPID’s distilled workflows and decision rules retain value under unseen environment configurations.The results suggest OPID does not merely fit observed training trajectories.Source: §C.2 · p. 28
• OPID improves average success rate on ALFWorld Unseen over GRPO by +7.7 points, with particularly clear gains on Look and Heat.Table 8 reports success rates across six unseen task types and their average.Source: Table 8 · p. 28, §C.2 · p. 28
• OPID selects a relatively small number of critical steps for local skill supervision, relying on episode-level skills as default guidance for non-critical decisions.Figure 7 reports the average number of timesteps selected by the analyzer for step-level hindsight skills in each ALFWorld trajectory.Source: Figure 7 · p. 29

D CASE STUDY

• The case study provides illustrative OPID trajectories from ALFWorld, Search-QA, and WebShop.The figures show full trajectories, with examples from each benchmark.Source: D CASE STUDY · p. 29, Figure 10 · p. 32, Figure 11 · p. 33

E ADDITIONAL DISCUSSION

• Broader interactive benchmarks could test whether trajectory-derived skills help with longer horizons, richer interfaces, latent rules, and open-ended exploration, including OdysseyArena (Xu et al., 2026), AgentBench (Liu et al., 2023), WebArena (Zhou et al., 2023), Mind2Web (Deng et al., 2023), and VisualWebArena (Koh et al., 2024).These are proposed stress tests beyond the embodied, shopping, and search-based settings studied in the paper.Source: E ADDITIONAL DISCUSSION · p. 29
• Future extensions could combine on-policy skill extraction with search-discovered reasoning patterns (Wu et al., 2024) and policy-aware exploration (Wu et al., 2026a; Lu et al., 2026a) to aggregate skills, capture recurring failures, and form compositional rules.The discussion retains OPID’s design choice that skills shape training rather than serve as privileged inference-time context.Source: E ADDITIONAL DISCUSSION · pp. 29–30

C.3 TRAINING DIAGNOSTICS AND SKILL EXTRACTION PATTERNS

• OPID applies step-level supervision sparsely to critical decisions, while Figure 8 contrasts episode-level outcome advantages with skill-guided advantages during optimization.Figure 7 reports the average number of critical steps identified on ALFWorld; Figure 8 shows the magnitudes of the two advantage signals.Source: §C.3 · p. 29, Figure 8 · p. 30

E ADDITIONAL DISCUSSION

• Because analyzer and skill-conditioned scoring run only during training, the learned policy requires no inference-time skill-retrieval cost, though more efficient scoring could reduce training cost.The discussion proposes speculative and retrieval-parallel decoding methods such as DOUBLE (Shen et al., 2026), and extending OPID to perceptual and embodied benchmarks such as RobotEQ (Fang et al., 2026).Source: E ADDITIONAL DISCUSSION · p. 30

C.3 TRAINING DIAGNOSTICS AND SKILL EXTRACTION PATTERNS

• The analyzer prompt requests successful-workflow or failure-avoidance episode skills and action-oriented guidance for at most a specified number of critical steps.It also requests an episode summary and has the analyzer infer step guidance from the full episode.Source: §C.3 · p. 29, Figure 9 · p. 31