PaperlayerView exact revision on arXiv

Source-linked AI summary

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang, Yuhao Shen, Lang Feng, Haoran Luo, Zheng Lian, Shuai Zhang, Zhengqi Wen, Jianhua Tao

arXiv:2607.14777v1cs.CL

TL;DR

Sparse trajectory-level rewards provide limited guidance for intermediate decisions in long-horizon agentic RL. SEED extracts evolving hindsight skills from completed on-policy trajectories and distills their effects into token-level supervision, consistently improving performance, sample efficiency, and robustness across agentic benchmarks.

  • Problem

    Outcome-based agentic RL offers coarse supervision because sparse, delayed trajectory-level rewards do not identify which intermediate observations, actions, or tool calls matter.

  • Method

    SEED uses the current policy to analyze completed on-policy trajectories into hindsight skills, then distills skill-induced probability shifts as dense token-level supervision jointly with outcome-based RL.

  • Results

    SEED consistently improves performance, sample efficiency, and robustness across embodied interaction, web navigation, search-based QA, visual reasoning, and planning benchmarks.

  • Takeaways & Limitations

    SEED shows that policy-synchronized hindsight distillation can provide reusable guidance for long-horizon agentic RL without requiring skills at inference time.

  • Takeaways & Limitations

    Because the actor and analyzer share one policy, internally generated supervision can inherit shared model errors and plateau below oracle-supervised training.

Abstract

from arXiv · show

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.

1 INTRODUCTION

SEED addresses the supervision gap in long-horizon agentic RL by turning completed on-policy trajectories into hindsight skills and distilling their behavioral effects into dense token-level guidance. Its self-evolving loop jointly improves policy decision-making and trajectory analysis, with experiments reporting gains in task performance, sample efficiency, and robustness across diverse agentic benchmarks.

  • Motivation: Outcome-based agentic RL leaves a supervision gap because sparse, delayed trajectory-level rewards do not identify which intermediate observations, actions, or tool calls caused success or failure.Failed trajectories may contain useful partial behaviors, while successful trajectories may contain reusable strategies that scalar rewards do not expose.
  • SEED Framework: SEED converts completed on-policy trajectories into natural-language hindsight skills describing reusable workflows, decisive observations, and failure-avoidance rules.The current policy both collects trajectories and analyzes them, so policy updates improve decision-making and skill analysis together.
  • On-Policy Distillation: SEED transforms skill-induced log-probability shifts on sampled actions into dense token-level supervision and jointly optimizes it with outcome-based RL.This policy-synchronized on-policy distillation mechanism converts trajectory-level hindsight into decision-level learning signals.
  • Design Principles: The framework is designed to remain on-policy, dense, and self-evolving as the policy’s capabilities and induced trajectory distribution change.Fixed teachers, static skill datasets, and one-time distillation cannot continually adapt to the evolving policy.
  • Experiments: SEED improves task performance, sample efficiency, and robustness across embodied interaction, web navigation, search-based QA, and visual perception and planning.The evaluation spans diverse long-horizon agentic benchmarks and compares SEED with representative baselines.

2 RELATED WORK

Prior work applies outcome-based reinforcement learning, hindsight learning, and on-policy self-distillation to agentic language models, but privileged supervision can remain static or mismatched as policies evolve. SEED combines hindsight skill extraction with skill-conditioned on-policy distillation in a self-evolving training loop.

  • Reinforcement learning for agentic LLMs: Outcome-based reinforcement learning directly optimizes task-level outcomes for interactive language agents that reason, use tools, and act over long horizons.
  • Hindsight learning for language agents: Completed trajectories reveal reusable strategies, decisive observations, and failure causes, motivating hindsight relabeling, return decomposition, process supervision, verbal reflection, and experience memory.
  • On-policy self-distillation for agentic RL: On-policy self-distillation transfers teacher behavior through token- or sequence-level supervision while reducing distribution mismatch by training on outputs sampled from the learner.
  • On-policy self-distillation for agentic RL: SEED equips the policy to extract hindsight skills from completed trajectories, then jointly optimizes outcome-based RL and skill-conditioned on-policy distillation in a self-evolving loop.
  • On-policy self-distillation for agentic RL: Privileged supervision in recent agentic RL methods is often static, externally generated, or updated independently of the policy, risking staleness or mismatch as behavior and encountered failure modes change.

3 METHOD

SEED is a self-evolving on-policy distillation framework that converts completed trajectories into natural-language hindsight skills and distills their behavioral effects into token-level supervision. It first trains a shared policy to analyze trajectories, then jointly optimizes on-policy RL with skill-conditioned distillation while removing all analyzer components at deployment.

  • Overview: SEED converts sparse trajectory-level feedback into token-level supervision by extracting reusable hindsight skills from completed agent trajectories and distilling their behavioral effects into the ordinary policy.Completed trajectories can reveal behavioral patterns, failure causes, and reusable strategies unavailable at intermediate steps.
  • Inference: Hindsight skills are used only during training, so deployment requires neither the analyzer nor skill storage, retrieval, or augmented decision prompts.The analyzer is removed at inference time after its behavioral effect has been distilled into the policy.
  • Hindsight-skill supervised fine-tuning: Stage 1 fine-tunes a single autoregressive policy to generate natural-language hindsight skills from completed trajectories, including strategies from successes and corrective or avoidance guidance from failures.The resulting checkpoint initializes both the later RL policy and synchronized trajectory analyzer.
  • On-policy reinforcement learning: Stage 2 freezes the current policy snapshot to collect on-policy trajectories and analyze them into skills, while a trainable copy is optimized with environment-driven GRPO and OPD objectives.Refreshing the snapshot evolves both the actor’s experience distribution and the model’s skill-analysis capability together.
  • On-policy distillation objective: SEED re-scores each sampled action under ordinary and skill-augmented contexts, using the detached skill-induced log-probability shift as a confidence-gated token-level distillation signal.The skill-conditioned branch acts as a training-time teacher, gradients flow only through the ordinary student branch, and positive shifts receive larger gates.

4 EXPERIMENT

Across ALFWorld, Search-based QA, and WebShop, SEED consistently improves over outcome-only, prompt-based, and static distillation baselines. Its gains extend to faster optimization, better sample efficiency, unseen-task transfer, and improvements from each core component.

  • Main results: SEED consistently outperforms outcome-only RL across ALFWorld, Search-based QA, and WebShop on all three backbones.Relative to GRPO, SEED improves ALFWorld macro-average by 14.9–45.9 points, Search-based QA by 1.4–9.3 points, WebShop task-completion by 8.7–19.8 points, and WebShop success rate by 5.5–39.0 points.
  • Main results: SEED’s internalized hindsight skills outperform evaluation-time prompting and static distillation across aggregate metrics.SEED beats Skill-Prompt on every aggregate metric, exceeds Skill-GRPO* in 11 of 12 comparisons, and achieves the best or tied-best result in 10 of 12 static-distillation comparisons.
  • Training dynamics: SEED reaches roughly 57% success by training step 40 versus about 35% for GRPO, while reducing final mean episode length to approximately 13 versus 16 turns.The success advantage persists throughout training, and shorter trajectories coincide with higher success, indicating more efficient task execution.
  • Sample efficiency: With 60% of the training data, SEED achieves 80.7 versus 75.0 for full-data GRPO, demonstrating stronger sample efficiency.At 40% data, SEED reaches 58.9, closely matching GRPO’s 58.6 at 80%.
  • Generalization: On ALFWorld Unseen, SEED raises macro-average success from 70.9 to 86.2, outperforming GRPO by 15.3 points in five of six task families.The largest gains are on Heat (+35.0), Look (+18.3), and Pick (+16.5).
  • Ablation analysis: Ablations show that hindsight-skill SFT, self-evolving OPD, and on-policy skills each materially contribute to SEED’s performance.Removing them lowers ALFWorld average performance to 86.0, 87.0, and 84.4, corresponding to drops of 5.8, 4.8, and 7.4 points, respectively.

5 CONCLUSION · A THEORETICAL ANALYSIS · A.1 ON-POLICY HINDSIGHT PRODUCES AN OCCUPANCY-MATCHED ADAPTIVE TARGET

SEED bridges trajectory-level outcomes and token-level learning by extracting hindsight skills from on-policy experience and distilling their effects into dense supervision. The analysis formalizes how this supervision is occupancy-matched, skill-selective, and sensitive to distribution staleness.

  • 5 CONCLUSION: SEED’s latest policy acts as both trajectory actor and analyzer, allowing behavior and the supervision derived from experience to evolve together.This design bridges sparse trajectory-level outcomes and token-level policy learning through dense hindsight supervision.
  • 5 CONCLUSION: In ALFWorld, SEED completes “put a candle in toilet” in five steps, whereas GRPO searches inefficiently, takes an irrelevant item, and enters an off-task loop.The qualitative comparison shows SEED systematically checking plausible shelves, finding the candle, and completing the required placement.
  • 5 CONCLUSION: SEED consistently improves performance, sample efficiency, and robustness across embodied interaction, web navigation, search-based QA, visual reasoning, and planning tasks.The framework extracts hindsight skills from completed on-policy trajectories and distills their behavioral effects without requiring skills at inference time.
  • A THEORETICAL ANALYSIS: The theoretical analysis establishes that SEED’s hindsight supervision is on-policy, dense, and self-evolving through occupancy matching, decision-specific credit, and analyzer refreshing.These properties correspond to the framework’s stated requirements for aligning auxiliary supervision with current policy behavior.
  • A THEORETICAL ANALYSIS: At iteration k, the frozen current policy π_k generates trajectories and the synchronized analyzer A_k converts each completed trajectory into a hindsight skill used in skill-augmented contexts.The skill may depend on the realized token, subsequent rollout, environment feedback, and analyzer randomness, so the analysis retains that conditional dependence.
  • A.1 ON-POLICY HINDSIGHT PRODUCES AN OCCUPANCY-MATCHED ADAPTIVE TARGET: The expected OPD update distills toward a skill-reweighted target on the current policy’s own token-context occupancy rather than uniformly imitating sampled tokens.The occupancy factor focuses learning on contexts, actions, and failure modes currently visited, while the conditional gate gives greater relative mass to skill-supported actions.
  • A.1 ON-POLICY HINDSIGHT PRODUCES AN OCCUPANCY-MATCHED ADAPTIVE TARGET: Using earlier trajectories biases the auxiliary gradient according to divergence from the current trajectory-token distribution, whereas on-policy collection eliminates rollout-stage distribution mismatch.Thus, synchronized data generation is essential for preserving the occupancy-matched adaptive target.
  • A.1 ON-POLICY HINDSIGHT PRODUCES AN OCCUPANCY-MATCHED ADAPTIVE TARGET: The reweighted target has higher local expected value only when hindsight support is positively correlated with current-policy action value.The theorem guarantees an on-policy target, while empirical improvement depends on skills assigning greater support to better decisions.

A.2 DENSE SKILL CREDIT REMAINS INFORMATIVE UNDER SPARSE OR TIED REWARDS · A.3 SELF-EVOLVING SYNCHRONIZATION CONTROLS ANALYZER STALENESS · B ADDITIONAL EXPERIMENTAL DETAILS

SEED retains informative token-level credit when outcome rewards are sparse or tied, and synchronizes its analyzer with the current policy to limit supervision staleness. Additional experimental details document the datasets, baselines, algorithm, extracted skills, and implementation.

  • A.2 DENSE SKILL CREDIT REMAINS INFORMATIVE UNDER SPARSE OR TIED REWARDS: Under tied rewards, the reward-driven policy gradient is zero, but the OPD gradient remains nonzero exactly when expected hindsight support varies across candidate tokens.The separate KL regularizer may still contribute a gradient, but it does not provide task-outcome credit.
  • A.2 DENSE SKILL CREDIT REMAINS INFORMATIVE UNDER SPARSE OR TIED REWARDS: Softmax normalization converts variation in hindsight support into signed relative credit, promoting tokens above expected support and suppressing tokens below it.The variance of hindsight gates quantifies the strength of this token-level signal.
  • A.2 DENSE SKILL CREDIT REMAINS INFORMATIVE UNDER SPARSE OR TIED REWARDS: SEED assigns context- and token-dependent credit, preserving locally useful behavior and attenuating inefficient choices even when terminal rewards cannot distinguish them.Outcome-based RL broadcasts one trajectory-level advantage, whereas skill-conditioned OPD varies its coefficient through w_k(c, v).
  • A.3 SELF-EVOLVING SYNCHRONIZATION CONTROLS ANALYZER STALENESS: A fixed analyzer can become mismatched to later policies, so Proposition 3 bounds analyzer-induced OPD-gradient discrepancy through differences in skill-induced probability shifts.The bound uses a score-norm bound and the β_opd/4 Lipschitz factor of the sigmoid gate.
  • A.3 SELF-EVOLVING SYNCHRONIZATION CONTROLS ANALYZER STALENESS: Using the analyzer from the current checkpoint makes cross-iteration analyzer mismatch zero during rollout and analysis, without assuming newer analyzers are always more accurate.Staleness is measured by behavioral effects on the current policy’s trajectories rather than parameter distance.
  • A.3 SELF-EVOLVING SYNCHRONIZATION CONTROLS ANALYZER STALENESS: SEED resets both experience and supervision staleness each outer iteration by pairing on-policy trajectories μ_k with the refreshed analyzer A_k.The analyzer remains fixed during subsequent inner policy optimization, so lag is not identically zero throughout optimization.
  • B ADDITIONAL EXPERIMENTAL DETAILS: Additional experimental details cover datasets, baseline methods, the complete SEED algorithm, representative extracted skills, and implementation details.These materials provide the experimental and procedural context for the method.

B.1 DATASETS · B.2 BASELINES · B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES

The experiments span embodied household reasoning, web navigation, and search-augmented question answering, with comparisons against prompting, outcome-based RL, and self-distillation baselines. SEED uses synchronized on-policy skill analysis and paired contextual re-scoring to jointly optimize dense distillation with KL-regularized GRPO.

  • B.1 DATASETS: The evaluation covers ALFWorld, WebShop, and Search-Augmented QA, representing household reasoning, interactive e-commerce, and search-based question answering.ALFWorld reports 140 seen tasks and 134 unseen tasks; WebShop evaluates 128 test samples, while Search-Augmented QA combines seven datasets.
  • B.1 DATASETS: Training uses 180 selected tasks per benchmark with eight rollouts each for SFT, followed by benchmark-specific RL sets of 2,400, 2,400, and 19,200 instances.The resulting SFT construction set contains 1,440 trajectories per benchmark configuration, and SFT and RL data are built separately for each configuration.
  • B.2 BASELINES: SEED is compared with prompting-only, outcome-based RL, and self-distillation or skill-distillation methods under standardized backbones, interfaces, and aligned training budgets.Unless marked with ∗, methods use only the standard task prompt and environment interaction history during evaluation; reproduced baselines are aligned where supported on batch size, rollout budget, group size, updates, and protocol.
  • B.2 BASELINES: The baseline suite includes Vanilla and Skill-Prompt∗, GRPO and skill-conditioned GRPO variants, plus OPSD, GRPO+OPSD, Skill-SD, RLSD, and SDAR.These baselines distinguish inference-time skills, training-time skill conditioning, generic on-policy self-distillation, and privileged-teacher objectives.
  • B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES: The algorithm converts the skill-induced token probability difference into a gated OPD signal and combines it with the clipped GRPO objective.The combined loss is L_SEED = L_rl + λ_opd L_opd, with detached skill-branch outputs used in the OPD term.
  • B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES: SEED’s frozen policy snapshot collects trajectories and analyzes completed interactions, while the trainable policy re-scores identical sampled tokens under ordinary and skill-augmented contexts.Gradients flow only through the ordinary branch, and the resulting OPD loss is jointly optimized with KL-regularized GRPO before updating the next snapshot.
  • B.3 ALGORITHM AND EXTRACTED SKILL EXAMPLES: Table 4 presents representative skills extracted from both successful and failed trajectories across ALFWorld, WebShop, and Search-based QA.The examples illustrate the extracted-skill material used across the three benchmark families.

B.4 IMPLEMENTATION DETAILS · C SUPPLEMENTARY RESULTS

The implementation details define benchmark-specific evaluation metrics, staged trajectory-to-skill annotation, RL prompting, and reproducibility settings for SEED. Supplementary materials illustrate skills extracted from successful and failed episodes.

  • B.4 IMPLEMENTATION DETAILS: ALFWorld averages success rates equally across six task categories, while Search-based QA uses a dataset-balanced aggregate of per-dataset accuracies.
  • B.4 IMPLEMENTATION DETAILS: Successful episodes yield reusable workflows, whereas failed episodes yield actionable failure-avoidance rules in the trajectory analyzer’s extracted skills.
  • B.4 IMPLEMENTATION DETAILS: WebShop reports complementary completion metrics: normalized requirement satisfaction scores and the percentage of episodes satisfying every requirement exactly.
  • B.4 IMPLEMENTATION DETAILS: At the SFT stage, an external analyzer converts serialized completed rollouts—task, full trajectory, and terminal outcome—into natural-language hindsight skills offline.
  • B.4 IMPLEMENTATION DETAILS: During RL, the actor uses benchmark-specific environment prompts, while the analyzer reuses the trajectory-analysis prompt from SFT.
  • B.4 IMPLEMENTATION DETAILS: Training hyperparameters are recorded for exact reproduction, and training uses 8 Nvidia A800 80G GPUs.

C.1 DETAILED SAMPLE EFFICIENCY COMPARISON

Table 6 shows that SEED consistently outperforms GRPO across five training-data fractions on both ALFWorld and WebShop, with substantial gains even when using less data.

  • ALFWorld: 80.7 with 60% of ALFWorld data exceeds GRPO’s 75.0 with the full dataset, while SEED’s gains range from 13.4 to 30.2 points.These results demonstrate strong ALFWorld sample efficiency relative to full-data GRPO training.
  • WebShop: 75.0 with 80% of WebShop data exceeds GRPO’s 63.3 with the full dataset, with improvements ranging from 5.5 to 15.6 points.SEED therefore achieves higher WebShop performance while using a smaller training-data fraction.
  • Sample efficiency comparison: SEED consistently outperforms GRPO across all five training-data fractions on both ALFWorld and WebShop.Table 6 reports absolute performance improvements over GRPO across both benchmarks.

C.2 CROSS-DOMAIN GENERALIZATION · C.3 MULTIMODAL EXTENSION

SEED generalizes beyond training environments, improving unseen ALFWorld task families and extending effectively to visually grounded agentic benchmarks. Across both settings, its hindsight supervision transfers reusable behavioral guidance to text-only and multimodal tasks.

  • C.2 CROSS-DOMAIN GENERALIZATION: SEED raises ALFWorld Unseen average success from 70.9 to 86.2, outperforming GRPO on five of six task families.Table 7 reports a 15.3-point average improvement using Qwen2.5-3B-Instruct.
  • C.2 CROSS-DOMAIN GENERALIZATION: On ALFWorld Unseen, the largest task-family gain is 35.0 points on Heat, followed by 18.3 on Look and 16.5 on Pick.Clean is the sole reported decrease, falling by 2.9 points.
  • C.2 CROSS-DOMAIN GENERALIZATION: Broad gains across unseen ALFWorld families indicate that SEED learns reusable behavioral guidance that transfers beyond the training environments.The result holds despite the 2.9-point decrease on Clean.
  • C.3 MULTIMODAL EXTENSION: Sokoban evaluates visual state tracking and long-horizon spatial planning because irreversible dead ends can result when boxes are pushed poorly.Each state is a 6 × 6 visual grid containing the player, boxes, walls, and targets; boxes cannot be pulled.
  • C.3 MULTIMODAL EXTENSION: The multimodal evaluation uses Qwen2.5-VL-3B-Instruct (Bai et al., 2025) on Sokoban and EZPoints, covering spatial planning and visual arithmetic.Sokoban is described by Schrader (2018).
  • C.3 MULTIMODAL EXTENSION: SEED achieves 82.0% on Sokoban and 100.0% on EZPoints, improving over GRPO by 14.9 and 13.1 points, respectively.Its average success rises from 77.0% to 91.0%, while ReAct reaches only 7.4% on average.
  • C.3 MULTIMODAL EXTENSION: Figure 7 illustrates a representative Sokoban trajectory through six consecutive actions, with temporal progression and each action shown beneath its observation.The visualization documents the sequence of agent-environment interaction rather than reporting an additional performance comparison.

C.4 ADDITIONAL TRAINING DYNAMICS

SEED improves success rates across all tested backbones and domains, while OPD losses decline and stabilize across the same nine settings. These dynamics indicate consistent learning and support the stability of SEED’s self-evolving loop.

  • Training dynamics: The consistent success-rate gains across model families and environments show that SEED is not tied to a particular backbone or form of agentic interaction.Convergence patterns vary by domain: Search-based QA improves rapidly early, while WebShop follows a steadier upward trend.
  • Training dynamics: Success rates increase across all three backbones and domains, with ALFWorld approaching 0.9, Search-based QA stabilizing at 0.47–0.55, and WebShop finishing at 0.68–0.75.The results cover nine backbone–domain settings and show consistent learning across model scales and agentic tasks.
  • Training dynamics: OPD losses decrease and stabilize in all nine settings, indicating that ordinary policies increasingly assign probability to actions favored by hindsight supervision.Qwen2.5 models converge gradually, whereas Qwen3-1.7B shows a sharper early decline, especially on ALFWorld.

D CASE STUDY

The case studies show SEED internalizing behavioral guidance and applying it without skill inputs across extended ALFWorld interactions, evidence-adaptive Search-based QA, and constrained WebShop purchasing. OPD loss dynamics further indicate that this guidance is progressively internalized during training.

  • D CASE STUDY: Without skill inputs, SEED tracks state and manages preconditions across extended ALFWorld tasks, completing multi-step placement and repeated book-retrieval subgoals.The examples include locating and cleaning a ladle before opening a drawer, and retrieving two books while preserving progress between subgoals.
  • D CASE STUDY: OPD loss generally decreases and stabilizes during training, indicating progressively internalized behavioral guidance from hindsight skills.This dynamic is shown in Figure 9.
  • D CASE STUDY: Search-based QA adapts information gathering to available evidence, while WebShop preserves requested attributes and price limits from search through purchase.The policy answers directly when retrieved passages suffice, but identifies Finding Neverland and searches specifically for its director before answering the second question.

E ADDITIONAL DISCUSSION

SEED’s additional discussion highlights open challenges in scaling self-evolving supervision: broader long-horizon benchmarks, shared actor–analyzer errors, and training-time computational cost. Illustrative trajectories show task decomposition, progress tracking, targeted search, and constraint preservation across agentic environments.

  • Limitations: SEED should be evaluated on broader benchmarks with longer workflows, richer state spaces, greater interaction complexity, and rarer terminal success.The discussion names DeepPlanning, Long-Horizon-Terminal-Bench, OdysseyArena, and RobotEQ as more demanding tests.
  • Limitations: Shared actor–analyzer policies can propagate blind spots: inaccurate analyses may turn recurring mistakes into reusable rules and reinforce them.On-policy alignment and confidence gating reduce distribution mismatch and noisy supervision, but neither guarantees semantic correctness; self-preference may also bias evaluation.
  • Limitations: SEED adds no deployment overhead, but training cost increases with trajectory length and multimodal context because it analyzes trajectories and scores actions under paired contexts.Speculative decoding, cached representations, and batched paired scoring are proposed to reduce rollout, skill-generation, and scoring costs.
  • Examples: Illustrative trajectories demonstrate SEED completing multi-step embodied tasks, targeted search, and constrained shopping through decomposition, progress tracking, verification, and preserved requirements.Examples include a seven-step ladle task, repeated book placement, two-stage factual search, and WebShop purchases satisfying color, size, and price constraints.
Loading 2607.14777v1…