PaperlayerView exact revision on arXiv

Source-linked AI summary

PRACTICE: From Experience to Expertise in Self-Evolving Embodied Agents

Ziyi Bai, Siqi Li, Tinglei Huang, Börje F. Karlsson

arXiv:2608.30760v1cs.LG

TL;DR

Embodied agents need to improve and adapt from interaction, yet experience-based methods commonly rely on fixed prompting workflows for skill maintenance. PRACTICE trains a skill learner to update a persistent skill library while keeping the executor frozen, using staged curriculum training and online skill-edit distillation. It consistently improves multiple frozen executors and outperforms strong experience-based baselines on EB-ALFRED and EB-Habitat.

  • Problem

    Most experience-augmented methods use fixed prompt-engineered workflows whose update policies do not learn from accumulated experience or downstream execution outcomes.

  • Method

    PRACTICE learns a skill-update policy that extracts, consolidates, and revises persistent procedural skills through oracle training, failure-aware contrasting, and online skill-edit distillation.

  • Results

    PRACTICE consistently improves multiple frozen executors and exceeds the strongest experience-based baselines by 9.7 and 2.6 percentage points on EB-ALFRED and EB-Habitat, respectively.

  • Takeaways & Limitations

    Sustained improvement depends on learning how procedural knowledge should evolve from interaction experience, not merely on expanding what the agent knows.

  • Takeaways & Limitations

    The framework keeps the executor policy parameters frozen throughout training.

Abstract

from arXiv · show

Recent studies have shown that multimodal large language models (MLLMs) can serve as embodied agents, translating language instructions and visual observations into executable plans. However, building agents that can continually improve through interaction and rapidly adapt to their environments remains challenging. Summing up experience from past interaction trajectories provides a promising solution, but existing experience-based methods often rely on manually designed prompting workflows to extract and update skills. Such fixed procedures may struggle to learn updated skills from new and diverse experiences. We introduce PRACTICE, which trains a skill learner to discover and maintain a persistent skill library from past interaction trajectories while keeping the task executor frozen. Given the historical accumulated skills and incoming trajectories, the skill learner produces structured batch-edits that add, refine, merge, or remove skills, and then hierarchical consolidate all collected edits into a consistent updated skill library. We train the learner with a two-stage curriculum. First, it learns basic skill generation and library maintenance from oracle trajectories. Then, by contrasting successful and failed trajectories from heterogeneous executors on the same tasks, it learn to identify invalid action patterns and recovery strategies. Finally, we apply online skill-edit distillation to align the skill learner with a stronger teacher on its current edit distribution to further improves the policy. Experiments demonstrate that a compact skill learner delivers consistent performance improvements across successive library-update rounds for multiple frozen executors. On EB-ALFRED and EB-Habitat, PRACTICE further outperforms the strongest experience-based baselines. Project resources are publicly available at: https://baai-agents.github.io/PRACTICE

1 Introduction

Embodied agents can externalize experience as reusable skills, but fixed prompting workflows do not learn how to update those skills. PRACTICE makes skill-library maintenance learnable and iteratively co-evolves the learner with persistent procedural knowledge while executors remain frozen.

  • Motivation: Experience-augmented methods externalize trajectories as persistent memories or reusable skills, but most use fixed prompt-engineered update workflows.These workflows can add, refine, merge, or remove entries without learning their update policy from accumulated experience or execution outcomes.
  • Approach: PRACTICE treats the skill library as evolving procedural knowledge and the skill learner as a learnable update policy.The learner extracts, consolidates, and revises skills while the executor remains frozen.
  • Iterative Co-evolution: The learner and library improve together across iterations: the current library shapes execution trajectories, which then train improved skill-refinement behavior.Updated skills guide the next execution round and provide new experience for further learner updates.
  • Training: PRACTICE trains the learner through oracle-grounded skill editing, failure-aware learning, and online skill-refinement policy distillation.Successful oracle trajectories establish basic maintenance; contrasting successes and failures teaches invalid-condition detection and recovery; Skill OPD aligns edits with a stronger teacher on learner-induced contexts.
  • Results: 9.7 and 2.6 percentage points: PRACTICE outperforms the strongest experience-based baselines on EB-ALFRED and EB-Habitat, respectively.It also consistently improves multiple frozen executors.

2 Related Works

Related work improves embodied planning either by post-training executors or by externalizing experience into reusable skills and memories. Prompt-based approaches maintain these artifacts through predefined operations, while memory methods organize experience into structured stores.

  • Executor Post-training: Executor post-training uses expert or generated trajectories, environment-grounded supervision, and interaction rewards to improve task planning.Representative methods apply supervised learning, imitation learning, or reinforcement learning to VLM or MLLM executors.
  • Externalized Experience: Experience-externalization methods store reusable skills or memories because executor post-training typically requires task-specific trajectories.Skill methods distill trajectories into lessons or executable rules, while memory methods organize records into memory structures.
  • Prompt-based Maintenance: Prompt-based methods maintain external knowledge through predefined operations such as trajectory consolidation, skill addition, merging, and removal.Trace2Skill and AutoSkill exemplify this approach.

3 Methodology

PRACTICE couples a frozen embodied task executor with a persistent skill library and a learnable skill learner that converts trajectories into structured library updates. Its curriculum trains skill generation, failure-aware editing, and library consistency from oracle and heterogeneous-executor experience.

  • System overview: The agent system combines an embodied task executor, persistent skill library, and learnable skill learner in a closed loop of skill-level self-evolution.The library guides execution, while the learner converts resulting trajectories into library updates.
  • Skill representation: Skill cards encode reusable action patterns with arguments plus usage specifications covering applicability and failure recovery.The library provides procedural guidance including action patterns, execution preconditions, expected effects, and recovery strategies.
  • Library evolution: At each update round, the learner reads the current library and new trajectories, predicts ADD, REVISE, MERGE, or REMOVE edits, and applies them to form the next library.The frozen executor remains unchanged, while different library versions induce different action and trajectory distributions.
  • Library evolution: PRACTICE replaces fixed prompt-engineered maintenance with a coupled cycle in which updated libraries change future trajectories and provide new evidence for learner improvement.The learner and library are improved iteratively across successive execution and update rounds.
  • Curriculum training: The curriculum proceeds from oracle-grounded SFT to failure-aware SFT and online skill-refinement policy distillation, with each stage inheriting the preceding learner and library.Stage 0 establishes basic generation and maintenance; Stage 1 contrasts successes and failures from heterogeneous executors; Stage 2 aligns the learner with a stronger teacher.
  • Curriculum training: Stage-0 supervision teaches generation from an empty library, localized editing of an initialized library, and consolidation that removes duplicates and reconciles compatible conditions.The consolidation task preserves distinct skills when their preconditions or effects differ, maintaining consistency beyond a single local update.
  • Curriculum training: Failure-aware batches use multiple executors attempting the same task, so shared instructions and success criteria make behavioral differences evidence for generalizable skill content.This controlled comparison supports identifying invalid action patterns and recovery strategies.

4 Experiments

PRACTICE is evaluated across EB-ALFRED and EB-Habitat, where it improves frozen executors, outperforms experience-augmented baselines, and benefits from progressive and failure-aware training.

  • Evaluation setup: PRACTICE evaluates two EmbodiedBench benchmarks spanning six capability dimensions, including complex instruction following and long-horizon planning.The benchmarks are EB-ALFRED and EB-Habitat, with Base, Commonsense, Complex, Visual, Spatial, and Long splits.
  • Main results: 49.7% and 58.3% average success rates on EB-ALFRED and EB-Habitat, respectively, exceed the strongest prior experience-augmented baseline.Under the same Qwen3-VL-32B-Instruct executor, PRACTICE leads experience-augmented methods and performs best on five EB-ALFRED and four EB-Habitat splits.
  • Main results: PRACTICE improves multiple frozen executors and surpasses stronger foundation-model comparisons, including Qwen3.5-Flash and state-of-the-art EB-Habitat performance with GPT-5.4.The method trains a separate skill learner while leaving executor parameters unchanged.
  • Cross-executor generalization: 2.4, 30.3, 15.3, and 25.4 percentage points improve GPT-5.4, GPT-5.2, Gemini, and Qwen average success rates, respectively.The cross-executor study compares each skill-augmented executor with its matched unaugmented baseline; PRACTICE has the largest reported matched improvements on GPT-5.2 and Qwen among shown experience-augmented methods.
  • Progressive training: 42.3% after Stage 0, 45.3% after Stage 1, and a further 4.4-point gain after Stage 2 show consistent improvement across training stages.Stage 1 raises Long performance from 6% to 14%, while Stage 2 produces 29 skill cards.
  • Failure-aware learning: 2.6 percentage points separate failure-aware learning from success-only training, with Long improving from 8% to 14%.Failure-aware supervision exposes invalid action patterns, violated preconditions, and corrective behaviors for complex and long-horizon tasks.
  • Failure-aware learning: A failed PickUp triggers an “object not found” recovery strategy, redirects the search, and enables completion in 16 steps.The trajectory then uses WashObject and PlaceObjectReceptacle.

5 Conclusion

PRACTICE frames sustained embodied-agent improvement as learning how interaction experience becomes persistent procedural knowledge. Its learner–library cycle uses applicability conditions and recovery strategies while keeping the executor frozen.

  • PRACTICE reframes sustained improvement as learning how interaction experience should be transformed into persistent procedural knowledge.
  • Reusable action patterns become more effective when paired with explicit applicability conditions and failure-recovery strategies.Together, they help executors decompose complex instructions, avoid repeated failure modes, and recover from deviations.
  • A co-evolving learner and skill library improves future trajectories and the procedural knowledge retained in subsequent library updates without changing the executor.

A.1 Training Data Construction

PRACTICE constructs training supervision from oracle trajectories and same-task rollouts across heterogeneous executors. Its learner updates libraries through batched add, revise, merge, and remove edits that are progressively consolidated.

  • Stage 0: Stage 0 uses successful oracle trajectories to train skill generation, editing, and consolidation capabilities.
  • Stage 0: Stage 0 generation samples N = 10 trajectories from 300 oracle trajectories, then samples M = 15 skill candidates for consolidation supervision.
  • Stage 1: Stage 1 contrasts successful and failed executions of the same tasks across heterogeneous executors to produce failure-aware edits.
  • Progressive co-evolution: Algorithm 1 iteratively rolls out a frozen executor, updates the library, collects heterogeneous same-task trajectories, and constructs supervision for learner refinement.
  • Library updates: For each trajectory batch, the learner emits ADD, REVISE, MERGE, or REMOVE edits, which are applied jointly to form the updated library.

A.2 Training Algorithm

Skill OPD trains the skill learner on its own generated edits while a frozen teacher supplies token-level distributions. The objective directly matches student and teacher distributions without task rewards or policy-gradient losses.

  • On-policy rollout: The student first samples a skill update, after which the frozen teacher evaluates each student-induced response context.
  • Teacher supervision: At each response position, the teacher provides a Top-K token set for supervision.
  • Objective: The loss aggregates only valid response-token positions, using K = 32 in the implementation.
  • Objective: Skill OPD directly backpropagates a truncated forward-KL distribution-matching objective through student probabilities while keeping the teacher frozen.
  • Objective: Task rewards and PPO/GRPO policy-gradient losses are disabled, so the total optimization objective is Ltotal = LOPD.
  • On-policy rollout: Because responses are resampled from the latest student before scoring, distillation states evolve with the student policy rather than an offline teacher-generated dataset.

B Implementation Details

The implementation initializes the learner with curriculum SFT and then applies Skill OPD using a fixed stronger teacher. Distillation uses pure Top-K forward-KL without auxiliary reward or policy-gradient objectives.

  • Curriculum SFT: Qwen3-VL-8B-Instruct is first trained with curriculum SFT supervised by Qwen3.7-Max.
  • Skill OPD: Skill OPD then uses a fixed Qwen3-VL-32B-Instruct teacher to provide token-level soft targets for current-student editing rollouts.
  • Skill OPD: The implementation uses pure Top-K forward-KL distillation without auxiliary reward or policy-gradient objectives.

C.1 What We Learned

PRACTICE learns libraries containing both primitive and composite procedures, including long-horizon macros. The resulting skill granularity differs across EB-ALFRED and EB-Habitat in ways aligned with their reported procedural structures.

  • Learned libraries: After all three training stages, the EB-ALFRED library contains 29 skills, while EB-Habitat contains 15 skills.
  • Learned libraries: EB-ALFRED has 13 single-action and 16 composite skills, whereas EB-Habitat has 5 single-action and 10 composite skills.
  • Procedural abstraction: HeatAndStoreFoodItem expands into 15 actions, integrating retrieval, appliance operation, heating, and storage.
  • Procedural abstraction: MultiObjectSearchAndTransfer contains 8 actions covering repeated search, retrieval, and delivery.
  • Skill hierarchy: In EB-ALFRED, 11 of 16 composite skills contain 4 to 7 actions, while 9 of 10 EB-Habitat composite skills contain 2 to 4 actions.
  • Skill hierarchy: The reported distributions associate EB-ALFRED with longer manipulation routines and EB-Habitat with compact search-and-transfer abstractions.

C.2 Skill Execution Analysis

Learned skills improve execution through both proactive failure avoidance and recovery after rejected actions. The library concentrates usage on broadly reusable skills with stable positive returns.

  • Execution outcomes: Failure-free episodes increase from 40/300 (13.3%) to 77/300 (25.7%) with learned skills.Successful failure-free episodes also rise from 38/300 (12.7%) to 73/300 (24.3%), indicating more executable plans that satisfy action preconditions.
  • Execution outcomes: Successful episodes containing rejected actions increase from 35/300 (11.7%) to 76/300 (25.3%).Episodes with rejected actions decrease from 260 to 223, so the lower branch-level percentages should not be interpreted as conditional recovery rates.
  • Execution outcomes: The combined improvement reflects proactive failure avoidance and post-failure recovery.Practice helps produce valid plans while preserving progress and completing tasks after execution failures.
  • Skill benefit–risk analysis: Practice raises task success from 24.3% to 40.0%, with positive improvements for TurnOff, Close, TurnOn, Open, PutDown, Find, and PickUp.Find, PickUp, and PutDown combine high invocation frequency with broad task coverage, while TurnOff, Close, and TurnOn achieve larger gains.
  • Skill benefit–risk analysis: Frequent, broadly applicable skills form the library’s core strength, while frequently used harmful skills are nearly absent.Low-frequency negative estimates remain uncertain for skills such as Drop and HoldObjectWhileTurningOnLight because of limited samples.
  • Skill benefit–risk analysis: The learned library improves success while concentrating usage on broadly reusable skills with stable positive returns.These results support extracting transferable behavioral patterns from trajectories to improve execution reliability across diverse tasks.

C.3 Skill Usage in Different Splits

Paired evaluations show consistent skill advantages across EB-ALFRED and EB-Habitat, while case studies reveal that remaining failures arise from grounding, state tracking, and long-horizon coordination.

  • Paired benchmark comparison: On EB-ALFRED, Practice recovers 91 baseline-failed episodes and regresses on 15, yielding ∆SR = +25.3 percentage points.The largest gains are on base and complex-instruction tasks (+36 points), followed by spatial tasks (+30 points).
  • Paired benchmark comparison: On EB-Habitat, Practice recovers 58 failed episodes and causes 32 regressions, producing ∆SR = +8.7 points.The largest gains occur on visual-appearance tasks (+20 points), followed by complex-instruction (+12 points) and base tasks (+10 points).
  • Paired benchmark comparison: Across both benchmarks, skills reduce planning complexity for recurring multi-step behaviors and improve execution consistency.Remaining failures concentrate in visual grounding, indirect object search, spatial reasoning, and long-horizon coordination.
  • Split-level execution: Complex Instruction and Base achieve the highest EB-ALFRED task success rates, at 70.0% and 66.0%, respectively.These splits also show reliable skill expansion and action acceptance.
  • Split-level execution: Long Horizon reaches only 24.0% task success despite high expansion validity and recovery from every local failure.Local recovery is effective but does not fully prevent error accumulation or preserve state consistency across extended compositions.
  • Qualitative cases: Successful EB-ALFRED trajectories combine reusable composite skills with feedback-aware primitive execution and modular long-horizon composition.Examples include light-assisted inspection and transformations involving slicing, cooling or heating, and final placement.
  • Qualitative cases: Containment and holding-state failures expose incomplete nested-transport coverage and insufficient executor-side tracking of the held object.These failures can disrupt required cup–knife or pan–apple relations.

D.2 Case Study in EB-Habitat

In EB-Habitat, SearchAndRetrieve structures indirect object retrieval and repeated transport, with local search enabling recovery from failed pickups. Its main weakness is uncertain source-receptacle inference and ineffective exploration after failure.

  • Feedback-guided search: SearchAndRetrieve recovers from failed pickup by switching to another candidate receptacle while preserving the retrieval task structure.In Base-36, the agent moves from the sofa to a table, retrieves the toy airplane, and delivers it to the left counter.
  • Repeated transport: The same abstraction supports repeated multi-object transport by instantiating SearchAndRetrieve with different objects, sources, and destinations.This reuse reduces planning complexity while allowing failed pickups to trigger local search.
  • Failure cases: Incorrect source-receptacle hypotheses cause repeated unreachable pickup attempts and unstructured exploration.Base-26 lacks a coverage-aware strategy for deciding which receptacle to inspect next.
  • Failure cases: The skill schema can represent object transport even when failures are primarily caused by uncertain source arguments and executor-side exploration.The navigation–pickup–placement pattern itself is not identified as the main failure source.
  • Environment-specific observations: EB-Habitat failures are dominated by source-receptacle inference, indirect exploration, semantic grounding, identity tracking, and search-cost accumulation.Across benchmarks, robust grounding and state-aware execution remain necessary to realize the library’s full benefit.
Loading 2608.30760v1…