PaperlayerView exact revision on arXiv

Source-linked AI summary

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Shuo Yang, Jinyang Wu, Zhengxi Lu, Yuhao Shen, Fan Zhang, Lang Feng, Shuai Zhang, Haoran Luo, Zheng Lian, Zhengqi Wen, Jianhua Tao

arXiv:2606.26790v1cs.CL

TL;DR

Outcome-based RL gives language agents sparse feedback that does not identify which intermediate decisions matter. OPID distills hierarchical skills from completed on-policy trajectories into dense token-level guidance, improving performance, sample efficiency, and robustness across agentic benchmarks over outcome-only RL and skill-distillation baselines.

  • Problem

    Sparse, delayed outcome rewards provide stable task-level pressure but limited fine-grained credit assignment for intermediate decisions in long-horizon agentic tasks.

  • Method

    OPID extracts hierarchical episode- and step-level hindsight skills from on-policy trajectories, routes them by decision criticality, and converts their effects into token-level advantages combined with outcome advantages.

  • Results

    Across ALFWorld, WebShop, and Search-based QA, OPID improves agent performance, sample efficiency, and robustness over outcome-only RL and skill-distillation baselines.

  • Takeaways & Limitations

    Completed on-policy trajectories can provide distribution-matched hindsight supervision that agents internalize without external skill libraries or privileged inference-time context.

  • Takeaways & Limitations

    At inference time, OPID relies on the learned policy acting from ordinary interaction history without analyzer calls, skill retrieval, or privileged context.

Abstract

from arXiv · show

Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose \textbf{OPID} (\textbf{O}n-\textbf{P}olicy Sk\textbf{i}ll \textbf{D}istillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.

1 INTRODUCTION

OPID addresses the coarse, delayed supervision of outcome-based agentic RL by extracting hierarchical hindsight skills directly from completed on-policy trajectories. It routes these skills into token-level self-distillation while retaining outcome-based RL as the primary objective, improving performance across long-horizon agentic benchmarks.

  • Motivation: Outcome-based agentic RL provides sparse, delayed supervision that reveals trajectory success but not which intermediate decisions caused it.This limitation is particularly severe in long-horizon interaction, where early mistakes, repeated invalid actions, and delayed consequences can derail episodes.
  • Motivation: Existing skill-conditioned self-distillation methods commonly depend on external skill libraries, retrieved skill files, or maintained skill memories.Such privileged context can be costly to maintain and mismatched with the current policy’s induced state distribution.
  • OPID: OPID extracts distribution-matched hindsight skills from completed on-policy trajectories, avoiding external skill libraries and off-policy retrieval.The framework treats trajectories sampled by the current policy as the source of skill supervision.
  • OPID: Hierarchical skills represent global workflows or failure-avoidance rules at the episode level and critical local decisions at the step level, with critical-first routing selecting the most specific available skill.Step-level skills are used at identified critical timesteps; episode-level skills provide default guidance otherwise.
  • OPID: OPID converts routed skills into dense token-level shaping signals by comparing the old policy’s scores for the same response under original and skill-augmented contexts.The resulting skill-based self-distillation advantage is combined with the episode advantage while outcome reward optimization remains primary.
  • Evaluation: OPID improves long-horizon agent performance over outcome-only RL and skill-distillation baselines on ALFWorld, WebShop, and Search-based QA, with better sample efficiency and fewer repetitive or invalid behaviors.The evaluation uses models at different scales across these benchmarks.

2 RELATED WORK

Related work spans outcome-based reinforcement learning for interactive language agents, on-policy self-distillation for token-level supervision, and skill-conditioned learning with natural-language procedural knowledge. OPID addresses the maintenance and state-distribution limitations of external skill resources by extracting hierarchical hindsight skills from completed on-policy trajectories and routing them to decisions.

  • Reinforcement learning for agentic LLMs: Outcome-based reinforcement learning has become a natural post-training paradigm for interactive language agents operating over long horizons, using methods such as GRPO for stable critic-free on-policy optimization.These agents invoke tools and receive feedback from environments or verifiers.
  • On-policy self-distillation: On-policy distillation learns from a model’s own sampled outputs with auxiliary teacher signals, while self-distillation compares the same policy under different contexts or feedback conditions.This provides token-level learning targets without requiring a separate teacher.
  • Skill-conditioned agent learning: Natural-language skills provide compact procedural knowledge through subgoal decompositions, action templates, and failure-avoidance rules.These capabilities motivate skill-conditioned approaches for agent learning.
  • Skill-conditioned agent learning: OPID extracts hierarchical hindsight skills directly from completed on-policy trajectories and routes step-level skills to critical decisions before falling back to episode-level guidance.It re-scores the same sampled response with and without the routed skill, converting token-wise log-probability differences into a dense skill advantage.
  • Skill-conditioned agent learning: Existing skill-based methods commonly rely on external skill libraries, retrieved files, or persistent memories, creating maintenance and retrieval costs and risking mismatch with the current policy’s state distribution.The mismatch can worsen in long-horizon interaction because small deviations may cause substantial state drift.

3 METHODS

OPID extracts hierarchical hindsight skills from completed on-policy trajectories, routes them by decision criticality, and converts their effects into dense token-level self-distillation supervision. It combines this skill signal with outcome-based RL during training while requiring no skill-related machinery at inference.

  • Hierarchical Skill Extraction: OPID extracts episode-level workflows or failure-avoidance rules and step-level local decision knowledge from completed on-policy trajectories.Episode-level skills provide broad default guidance, while step-level skills target pivotal, state-dependent decisions.
  • Skill Routing: Critical-first routing selects precise step-level skills at analyzer-identified critical timesteps and otherwise falls back to robust episode-level skills.This prevents the two skill granularities from being blindly combined and assigns each step the appropriate guidance.
  • Skill-Conditioned Self-Distillation: OPID injects the routed skill into the interaction history and has the old policy re-score the same sampled response under original and skill-augmented contexts.The response is not regenerated; token-level probability shifts indicate alignment or misalignment with the routed hindsight skill.
  • Policy Optimization: The final advantage combines group-relative outcome feedback with token-level skill supervision, keeping outcome reward as the primary RL signal while adding token-level shaping.OPID optimizes the standard clipped policy objective.
  • Training-Inference Boundary: At inference time, the learned policy uses ordinary interaction history alone without analyzer calls, skill retrieval, or privileged context.The analyzer, routing, and skill-conditioned scoring are used only to construct the training advantage.

4 EXPERIMENT

Across ALFWorld, WebShop, and Search-based QA, OPID consistently improves outcome-only RL and often matches or surpasses stronger baselines. Its gains include better sample efficiency, cross-domain transfer, and performance from hierarchical, critically routed skills.

  • Experimental Setup: Experiments cover ALFWorld, WebShop, and Search-based QA using Qwen2.5-3B/7B-Instruct and Qwen3-1.7B-Instruct for 150 training steps.Batch size is 16 for ALFWorld and WebShop and 128 for Search-based QA.
  • Main Results: +9.3 points on ALFWorld, +8.6 on Search-based QA, and +10.9 on WebShop are Qwen2.5-3B OPID gains over GRPO.For Qwen2.5-7B, corresponding improvements are +8.8, +7.2, and +7.1 points; Qwen3-1.7B gains +12.8 on ALFWorld and +26.5 on WebShop.
  • Main Results: +1.7 points and +5.0 points are OPID’s ALFWorld improvements over the strongest baselines on Qwen2.5-7B and Qwen3-1.7B, respectively.OPID achieves the best average on both model scales in these settings, with 90.0 vs. 88.3 and 58.9 vs. 53.9.
  • Efficiency and Generalization: +9.3 to +20.3 points are OPID’s absolute gains over GRPO across ALFWorld training-data scales.With 60% of the data, OPID reaches 71.9 versus full-data GRPO at 75.0; with 80%, it reaches 78.9 versus 75.0.
  • Ablations and Analysis: Removing episode-level skills lowers ALFWorld from 84.3 to 74.1 and WebShop success from 74.2 to 67.2, while removing step-level skills lowers them to 79.1 and 65.6.Critical-first routing improves ALFWorld by +6.8 points over the non-routed variant, and OPID reduces average episode length to 15-16 steps versus GRPO’s 17-18.

5 CONCLUSION

OPID distills completed on-policy trajectories into hierarchical hindsight supervision, providing dense, distribution-matched token-level guidance while retaining outcome-based reinforcement learning as the primary objective. Experiments across embodied, web, and search-based benchmarks show improved agent learning without external skill li.

  • Conclusion: OPID converts completed agent trajectories into hierarchical hindsight supervision.The framework extracts both episode-level and step-level skills from the current policy’s own rollouts.
  • Conclusion: The framework provides dense, distribution-matched token-level guidance while preserving outcome-based RL as the primary objective.
  • Conclusion: Experiments across embodied, web, and search-based agentic benchmarks show that OPID improves agent learning without external skill li.

A THEORETICAL ANALYSIS … A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES

The analysis formalizes OPID’s token-level skill signal and situates it among on-policy distillation objectives. It shows that sampled-token supervision corresponds to reverse-KL optimization, while clarifying the local validity and global limitation of its PPO surrogate interpretation.

  • A.1.1 NOTATION: OPID defines behavior, skill-conditioned teacher, and trainable policy distributions at each valid token position, with the observed response token sampled from the behavior policy.The standard context is ci, the skill-augmented context is eci, and the observed token ai is sampled from bi.
  • A.1.1 NOTATION: The token-level skill advantage is the log-likelihood gap between the skill-conditioned teacher and behavior policy for the observed token.A positive gap means the skill-conditioned teacher assigns greater probability to that token.
  • A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES: On-policy distillation reduces context-distribution mismatch by applying teacher supervision at contexts generated by the student or a behavior policy, independently of supervision granularity.The analysis organizes output-space supervision into full-vocabulary, top-K, and sampled-token forms.
  • A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES: Full-vocabulary matching may minimize forward KL, reverse KL, or generalized Jensen–Shannon divergence; forward KL emphasizes coverage, whereas reverse KL is typically more mode-seeking.Generalized JSD compares both models with a mixture distribution, with α = 1/2 recovering standard symmetric JSD.
  • A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES: Top-K matching retains distribution-level supervision on a restricted local support selected by either the student or teacher.It preserves multi-token information at lower computational or communication cost but discards probability mass outside the selected support, making it a support-dependent approximation to full reverse KL.
  • A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES: Sampled-token distillation expresses token-level reverse KL as an expectation over student-sampled tokens and estimates it with an importance-weighted realized-token sample.This requires the support condition pθ,i ≪ bi and uses only the teacher probability of the realized token, at the cost of higher Monte Carlo variance and less information than full-vocabulary matching.
  • A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES: OPID combines outcome and skill advantages before applying PPO clipping, so its clipped objective generally does not decompose into independently clipped outcome and skill losses.The corresponding unclipped surrogate is therefore used to isolate the skill-distillation signal and is implemented as an empirical average over observed valid tokens.
  • A.1.2 REPRESENTATIVE ON-POLICY DISTILLATION OBJECTIVES: At θ0 = θold, the clipped and unclipped objectives have identical value and gradient, making the skill term exactly the skill-induced component of PPO’s first-order update.Away from this local region, clipping couples outcome and skill signals through the sign of their combined advantage, so the unclipped decomposition is not globally complete.

A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE · A.3 ON-POLICY OCCUPANCY MATCHING FOR DISTILLATION

The unclipped OPID skill loss is exactly a behavior-relative-KL surrogate, locally matching reverse-KL distillation but diverging globally and under PPO clipping. On-policy context occupancy removes distribution mismatch, while trajectory-dependent teacher changes remain uncontrolled.

  • A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE: The unclipped OPID skill loss decomposes exactly into scaled reverse-KL minus a behavior-relative KL term when rollout quantities remain detached and fixed.The analysis assumes common support between the student, behavior, and teacher distributions.
  • A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE: At the behavior policy, the behavior-relative KL vanishes and is globally minimized, so the OPID loss has the same value and gradient as scaled reverse-KL distillation.This equivalence holds at θ0 when pθ0,i = bi.
  • A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE: The OPID skill loss is an exact relative-KL loss but only a local surrogate for direct student–teacher reverse-KL matching.Their discrepancy is second order near the behavior policy, while their global optima can differ.
  • A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE: Exact recovery of direct reverse-KL distillation requires a behavior-KL penalty evaluated against the same bi and the matching coefficient β = λskill.Different reference distributions or coefficients leave a residual behavior-relative term.
  • A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE: PPO clipping preserves the unclipped skill component’s local value and first derivative at the behavior policy but later couples outcome and skill signals through their combined advantage.Thus, the complete clipped objective does not globally decompose into independently clipped outcome and skill losses.
  • A.2 THE UNCLIPPED OPID SKILL LOSS AS A RELATIVE-KL SURROGATE: Even with tied outcome rewards, a nontrivial teacher supplies a token-level signal whenever qi ≠ bi, although shared-parameter gradients may still cancel across contexts.At a context, the gradient is zero for every token if and only if qi = bi.
  • A.3 ON-POLICY OCCUPANCY MATCHING FOR DISTILLATION: On-policy occupancy matching bounds the difference between expected bounded per-context losses under db and dµ by total variation and KL divergence, reaching zero when dµ = db.The result applies to both nonnegative distillation losses and signed surrogate losses when the latter satisfy bounded-range conditions.
  • A.3 ON-POLICY OCCUPANCY MATCHING FOR DISTILLATION: The occupancy result controls only ordinary-context distribution mismatch and does not control changes in hindsight skills, routed teachers qi, or other trajectory-dependent quantities.It assumes the same per-context loss map is evaluated under both context distributions.

A.4 CRITICAL-FIRST HIERARCHICAL ROUTING · A.5 SUMMARY

Critical-first routing selects between episode-level and step-level teachers, recovering the lower-error candidate under perfect detection and incurring detector-dependent excess error otherwise. The summary establishes exact skill-loss decomposition, on-policy removal of outer context mismatch, and conditions for oracle-like routing, while distinguishing teacher approximation error from reverse-KL loss.

  • A.4 CRITICAL-FIRST HIERARCHICAL ROUTING: Critical-first routing selects the step-level teacher for predicted critical decisions and otherwise uses the episode-level teacher.The routed teacher is the qi used in the OPID skill advantage Δi(v) = log qi(v) − log bi(v).
  • A.4 CRITICAL-FIRST HIERARCHICAL ROUTING: Under teacher specialization and perfect criticality detection, routing recovers the oracle pointwise choice between the two candidate teachers.The selected candidate has the smaller approximation error.
  • A.4 CRITICAL-FIRST HIERARCHICAL ROUTING: With imperfect detection, excess teacher-approximation error is controlled by detector error probability and the difference between candidate-teacher errors.A routing mistake selects the nonspecialized teacher, whose excess error over the oracle choice is characterized in the proposition.
  • A.4 CRITICAL-FIRST HIERARCHICAL ROUTING: The routing criterion measures candidate-teacher approximation relative to the ideal teacher q⋆i, not the student–teacher reverse-KL loss DKL(pθ,i∥qi).Without assumptions relating candidate likelihood ratios, the routing result is not a direct upper bound on LRKL.
  • A.5 SUMMARY: Proposition 1 gives an exact decomposition for the unclipped OPID skill component when rollout histories, routed skills, and detached distributions bi and qi are fixed.This result concerns the unclipped skill component of the OPID policy loss.
  • A.5 SUMMARY: On-policy collection removes the outer context-distribution mismatch when the collection distribution equals the behavior-policy distribution, dµ = db.Under this condition, the occupancy term in Eq. 12 is zero.
  • A.5 SUMMARY: Together, the results establish exact relative-KL skill loss, first-order equivalence to scaled reverse-KL distillation at the behavior policy, mismatch removal, and oracle-approaching routing under specialization and accurate detection.These conclusions summarize Propositions 1–3 and preserve their stated conditions.

B ADDITIONAL EXPERIMENTAL DETAILS

This section describes the experimental protocol underlying the main-paper results and organizes the details by datasets, baselines, and implementation.

  • The section presents the experimental protocol used for the main-paper results.
  • The experimental details are organized by datasets.
  • The organization also covers baselines and implementation.

B.1 DATASETS · B.2 BASELINES · B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES

The study evaluates OPID across embodied reasoning, web navigation, and search-augmented question answering, comparing it with prompting, outcome-RL, and distillation baselines. Its procedure combines on-policy rollouts, hierarchical skill extraction, critical-first routing, paired scoring, and clipped optimization.

  • B.1 DATASETS: Experiments span embodied reasoning, web navigation, and search-augmented question answering.ALFWorld, WebShop, and search-augmented QA provide the three agentic evaluation domains.
  • B.1 DATASETS: ALFWorld requires sequential admissible actions across six task types: Pick, Look, Clean, Heat, Cool, and Pick2.The environment aligns text interaction with the ALFRED household environment and supplies natural-language goals and textual observations.
  • B.1 DATASETS: WebShop evaluates product search, attribute selection, and purchasing using normalized task-completion and binary exact-success signals.Agents interact with product pages to satisfy natural-language requests, with partial credit for matching requested attributes.
  • B.1 DATASETS: Search-augmented QA covers seven benchmarks, with agents interacting with a configured search environment before answering.The benchmarks are Natural Questions, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle.
  • B.2 BASELINES: Baselines include prompting-only, outcome-based reinforcement learning, and self- or skill-distillation methods under matched backbone and evaluation protocols.Unless marked with an asterisk, methods use ordinary environment histories without skills or privileged context; asterisks denote validation/test-time skill access.
  • B.2 BASELINES: GRPO assigns group-normalized scalar outcome advantages uniformly across trajectory tokens, while Skill-GRPO adds training-time task-relevant skills and removes them at test time.Skill-GRPO* instead retains skills during validation and testing, matching train-time and test-time conditioning.
  • B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES: OPID training performs on-policy rollout collection, hierarchical skill extraction, critical-first routing, paired scoring, and clipped policy optimization.Extracted episode-level skills represent reusable global workflows, whereas critical-step skills target sparse local decisions influencing final outcomes.

B.4 IMPLEMENTATION DETAILS · C SUPPLEMENTARY RESULTS

The implementation defines task-specific metrics, extracts hierarchical skills from completed on-policy trajectories, and specifies analyzer, model, training, and computing configurations for reproducibility. Supplementary materials include examples of episode-level and critical-step skills alongside RL hyperparameters.

  • B.4 IMPLEMENTATION DETAILS: ALFWorld performance is measured by task-type success rates aggregated with a macro-average.The metric is computed separately for each task type before averaging.
  • B.4 IMPLEMENTATION DETAILS: Search-based QA performance is measured by per-dataset answer accuracy aggregated with an unweighted macro-average across seven datasets.Accuracy is computed separately for each of the seven datasets.
  • B.4 IMPLEMENTATION DETAILS: WebShop reports Score as mean normalized task score multiplied by 100 and Succ. as the percentage of exactly successful tasks.The two metrics respectively capture normalized task performance and exact task success.
  • B.4 IMPLEMENTATION DETAILS: After each on-policy episode, the trajectory analyzer serializes the prompt, indexed observations, actions, feedback, and terminal outcome into an ordered record.An LLM-based analyzer maps the record to one episode-level skill and sparse critical-step skills.
  • B.4 IMPLEMENTATION DETAILS: Hierarchical skill examples pair successful and failed trajectories, with episode-level skills summarizing reusable global behavior and critical-step skills targeting sparse decisions.Step indices are 0-based analyzer keys and are consistent with the implementation description.
  • B.4 IMPLEMENTATION DETAILS: The default analyzer is GLM-5.2 with temperature 0.4, maximum output length 4096, and at most 5 critical steps for ALFWorld and WebShop or 2 for Search-based QA.The analyzer is identified as GLM-5.2 from Z.ai (2026).
  • B.4 IMPLEMENTATION DETAILS: Training uses Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-1.7B-Instruct for 150 update steps, with batch size 16 on ALFWorld and WebShop and 128 on Search-based QA.Table 6 records the remaining hyperparameters required for exact reproduction.
  • B.4 IMPLEMENTATION DETAILS: Training is conducted on 8 Nvidia A800 80G GPUs.This is the reported computing configuration.

C.1 DETAILED SAMPLE EFFICIENCY COMPARISON · C.2 CROSS-DOMAIN GENERALIZATION

OPID improves ALFWorld sample efficiency over GRPO, with especially large gains under limited training data, and transfers to unseen task configurations. Its selective use of step-level skills complements episode-level guidance for non-critical decisions.

  • C.1 DETAILED SAMPLE EFFICIENCY COMPARISON: OPID consistently improves ALFWorld success rates over GRPO across all evaluated training-data fractions.Table 7 compares success rates under different fractions of the training data.
  • C.1 DETAILED SAMPLE EFFICIENCY COMPARISON: +15.6 points with 60% of the data and +20.3 points with 80% of the data are reported gains for OPID over GRPO.The largest improvements occur in the low- and mid-data regimes, indicating stronger supervision extraction from each rollout.
  • C.1 DETAILED SAMPLE EFFICIENCY COMPARISON: The Table 7 ∆row reports absolute OPID-over-GRPO improvements in success rate across training-data budgets.These comparisons identify where OPID provides stronger gains under reduced data.
  • C.2 CROSS-DOMAIN GENERALIZATION: OPID improves average success rate on the ALFWorld unseen split over GRPO by +7.7 points.The clearest task-level gains are reported for Look and Heat.
  • C.2 CROSS-DOMAIN GENERALIZATION: Table 8 evaluates OPID and GRPO across six unseen ALFWorld task types and their average success rates.The results indicate that trajectory-derived skill supervision transfers beyond the training environments.
  • C.2 CROSS-DOMAIN GENERALIZATION: OPID selects a relatively small number of critical timesteps for step-level hindsight skills in each ALFWorld trajectory.Episode-level skills remain default guidance for non-critical decisions.

C.3 TRAINING DIAGNOSTICS AND SKILL EXTRACTION PATTERNS · D CASE STUDY · E ADDITIONAL DISCUSSION

The supplementary material diagnoses OPID’s sparse critical-step supervision and skill-guided learning signals, illustrates trajectories across ALFWorld, Search-QA, and WebShop, and discusses extensions to broader environments, richer skills, and efficient training-time scoring.

  • C.3 TRAINING DIAGNOSTICS AND SKILL EXTRACTION PATTERNS: OPID identifies critical steps sparsely rather than assigning step-level supervision to every decision, while diagnostics visualize its training-advantage dynamics.Figure 7 reports average critical steps on ALFWorld; Figure 8 contrasts episode-level outcome advantages with skill-guided advantages during optimization.
  • D CASE STUDY: Supplementary case studies provide illustrative OPID trajectories from ALFWorld, Search-QA, and WebShop.The examples include household manipulation, nationality and book-identification questions, and constrained product searches.
  • E ADDITIONAL DISCUSSION: Future work could evaluate OPID in broader interactive environments requiring latent-rule discovery, long-term state maintenance, and extended adaptation.Suggested benchmarks include OdysseyArena, AgentBench, WebArena, Mind2Web, and VisualWebArena.
  • E ADDITIONAL DISCUSSION: Future skill extensions could aggregate skills across trajectories, identify recurring failures, and form compositional rules while preserving training-only skill use.The discussion proposes combining on-policy extraction with higher-level reasoning abstractions and policy-aware exploration.
  • E ADDITIONAL DISCUSSION: Because analyzer and skill-conditioned scoring occur only during training, OPID adds no inference-time skill-retrieval cost, although faster scoring could reduce distillation overhead.Speculative and retrieval-parallel decoding methods such as DOUBLE are proposed for repeated model scoring.
  • C.3 TRAINING DIAGNOSTICS AND SKILL EXTRACTION PATTERNS: The analyzer prompt extracts episode summaries, successful workflows, failed-trajectory avoidance rules, and action-oriented guidance for critical steps.It distinguishes episode-level workflow and avoidance skills from step-level guidance.
  • D CASE STUDY: The Search-QA examples show OPID agents issuing targeted searches for questions about shared nationality and a bestselling serial-killer book.The supplementary trajectory text includes the initial search actions and retrieved information for these tasks.
  • D CASE STUDY: The WebShop examples show constrained product-search tasks requiring agents to combine category, fit, material, color, size, sleeve, and price criteria.The examples cover teen-girl tops under 40.00 dollars and men’s sleepwear under 60.00 dollars.
Loading 2606.26790v1…