PaperlayerView exact revision on arXiv

Source-linked AI summary

ASPIRE: Agentic /Skills Discovery for Robotics

Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi "Jim" Fan, Guanzhi Wang

arXiv:2607.00272v1cs.ROcs.AIcs.MA

TL;DR

Robot coding agents often receive only coarse task-level feedback, making it difficult to localize failures across interacting subsystems. ASPIRE iteratively writes and refines robot programs while accumulating reusable skills; across benchmarks it outperforms baselines and reaches 31% success on LIBERO-Pro Long [Zhou et al., 2025] versus 4% for prior methods.

  • Problem

    Existing robotic coding agents receive only coarse task-level feedback, while failures can arise across interacting components, making fine-grained root-cause diagnosis difficult.

    Source: §1 · pp. 1–2

  • Method

    ASPIRE combines closed-loop multimodal execution traces, a reusable skill library, and evolutionary search to iteratively write and refine code-as-policy robot programs.

    Source: §1 · p. 2, §2.2 · p. 6, §2.3 · p. 6

  • Results

    Across diverse benchmarks, ASPIRE substantially outperforms baselines; on LIBERO-Pro Long [Zhou et al., 2025], it achieves 31% success versus prior methods’ 4%.

    Source: §1 · p. 3, §6 · p. 12

  • Takeaways & Limitations

    Simulation-discovered skills provide initial evidence of cross-embodiment sim-to-real transfer while reducing real-robot programming token cost.

    Source: §6 · p. 12

    ASPIRE is not yet a fully autonomous real-world lifelong learner; real-world deployment still requires robust success detection, safe resets, safety monitoring, and calibration maintenance.

    Source: §5 · p. 11

Abstract

from arXiv · show

Traditional robot programming is challenging: it requires orchestrating multimodal perception, managing physical contact dynamics, and handling diverse configurations and execution failures. We introduce ASPIRE (Agentic Skill Programming through Iterative Robot Exploration), a continual learning system that autonomously writes and refines robot control programs in a code-as-policy paradigm while compounding experience into a reusable skill library. ASPIRE discovers skills that persist across tasks, simulation and real-world settings, and embodiments. It operates in an open-ended loop with three components: (1) a closed-loop robot execution engine that exposes fine-grained multimodal traces, enabling autonomous failure diagnosis, repair synthesis, and validation; (2) a continually expanding skill library that distills validated fixes into reusable, transferable knowledge; and (3) evolutionary search that generates diverse task sequences and control programs to explore beyond single-trajectory refinement. ASPIRE surpasses prior methods by up to 77% on LIBERO-Pro manipulation under perturbation, 72% on Robosuite bimanual handover, and 32% on BEHAVIOR-1K long-horizon household tasks. Its accumulated library also enables zero-shot generalization to unseen long-horizon tasks: on LIBERO-Pro Long, ASPIRE achieves 31% success versus 4% for prior methods despite their use of test-time reasoning and retries. Finally, simulation-discovered skills provide initial evidence of sim-to-real transfer, substantially reducing real-robot programming effort across different embodiments and robot APIs.

1. Introduction

Aspire is a continual-learning system that writes and refines robot programs while turning execution experience into reusable skills. Its results span benchmark gains, zero-shot transfer to longer-horizon tasks, and initial sim-to-real transfer.

Motivation

  • Coarse task-level feedback obscures failure causes, while existing systems discard fixes across tasks; Aspire instead aims to compound debugging experience into transferable knowledge.Robot failures may stem from perception, planning, grasping, contact dynamics, or long-horizon coordination, making fine-grained diagnostic evidence important.Source: §1 · pp. 1–2

System

  • Aspire combines per-primitive multimodal execution traces, a growing library of validated transferable fixes, and evolutionary search over diverse task sequences and control programs.Agents inspect traces, diagnose failures, synthesize repairs, validate them by re-execution, and consolidate successful recovery patterns as skills for future tasks.Source: §1 · p. 3

Results

  • Against prior coding agents [Fu et al., 2026], Aspire improves success rate by up to 77 points on LIBERO-Pro perturbation suites, 72 points on Robosuite contact-rich manipulation tasks, and 32 points on BEHAVIOR-1K long-horizon household tasks.The BEHAVIOR-1K tasks use procedurally generated layouts.Source: §1 · p. 3
  • On LIBERO-Pro Long, Aspire reaches 31% success versus 4% for prior methods; three simulation-discovered skills also reduce real-world reasoning tokens and enable programs where naive debugging fails.The LIBERO-Pro Long transfer uses a skill library accumulated on LIBERO-90; prior methods rely on test-time reasoning and retries. The real-robot evaluation uses a bimanual robot.Source: §1 · p. 3

2. Method

ASPIRE combines trace-guided execution and repair, a growing library of validated reusable skills, and evolutionary search in an open-ended learning loop that lets future tasks inherit prior experience.

2. Method

  • The coordinator dispatches task-focused coding actors, while transferable experience is distilled into a shared skill library rather than exchanged as raw trajectories or chat histories.Each actor writes, executes, diagnoses, and repairs programs using the task specification, current program, and structured traces from the current failure.Source: §2 · p. 3

2.1. Robot Execution Engine

  • Per-primitive multimodal traces expose perception, planning, and control evidence, letting agents diagnose failures, make targeted repairs, and validate them through closed-loop execution.In the radio example, repeated PLANNING_ERRORs reveal targets too close to the table edge; sampling alternative approach directions clears the collision buffer and enables a successful grasp.Source: §2.1 · pp. 4, 6, Figure 2 · p. 4

2.2. Skill Library

  • The skill library stores heterogeneous repair patterns induced from validated fixes, not whole task programs or a prescribed taxonomy.Compact in-context entries encode a failure signature, application condition, repair strategy, and sometimes a code sketch, enabling reuse across tasks and embodiments.Source: §2.2 · p. 6

2.3. Evolutionary Search

  • Evolutionary search proposes and executes diverse robot programs conditioned on top-performing candidates and failure traces, reducing repeated local repair of the same strategy.Validated repairs enter the library only if they generalize across environment variations and tasks; search stops when debugging configurations are solved or the budget is exhausted.Source: §2.3 · p. 6

3. Experiments

ASPIRE outperforms prior methods across simulation benchmarks, transfers accumulated skills to unseen tasks and real-robot programming, and benefits from its execution engine and evolutionary search. The experiments evaluate these results across benchmark families, held-out seeds, and a different real-robot embodiment and API.

3.1. Experimental Setup

  • Simulation experiments use Claude Opus 4.6 with a 1M-token context window to write CaP-X Python robot programs, while real-robot transfer uses Codex GPT-5.5 on a bimanual YAM station.The real-robot study supplies three simulation-discovered skills—soda-can pickup, bowl-on-plate placement, and drawer push/pull—as in-context guidance and compares tokens to first success and held-out success rates.Source: §3.1 · p. 7

3.2. Benchmarks and Baselines

  • The evaluation spans LIBERO-Pro perturbation robustness, Robosuite contact-rich single- and dual-arm manipulation, and BEHAVIOR-1K long-horizon household mobile manipulation, against CaP-Agent0 and end-to-end VLAs.The VLA baselines include OpenVLA, π0, and π0.5; zero-shot transfer is tested on held-out LIBERO-Pro Long tasks using skills accumulated on LIBERO-90.Source: §3.2 · p. 7

3.4. Main Evaluation Results

  • 77% on LIBERO-Pro Object, 41.5% on Goal, and 42.5% on Spatial are Aspire’s gains over the strongest baselines; handover rises from 20% to 92%, and radio task success from 56% to 88%.Aspire also outperforms human and CaP-Agent0 on BEHAVIOR-1K navigation and task success, and improves over baselines on all three LIBERO-Pro suites.Source: Figure 4 · p. 8, §3.4 · p. 9

3.3. Evaluation Protocol

  • Aspire learns on disjoint debug seeds and evaluates one generated program per LIBERO-Pro and Robosuite task on held-out seeds, while CaP-Agent0 regenerates per seed with retries.For BEHAVIOR-1K, Aspire uses incremental block execution, generating each next code block from the current multimodal input.Source: §3.3 · p. 8

3.5. Zero-Shot Transfer to Unseen Tasks

  • 23% position-perturbation and 38% task-perturbation success on LIBERO-Pro Long outperform CaP-Agent0 and π0.5, with success increasing as the library grows.The N=90 library transfers from LIBERO-90 to held-out long-horizon tasks; each task uses one program with no further debugging, retries, or task-specific library updates.Source: Figure 5 · p. 9, §3.5 · p. 9

3.6. Real-Robot Skill Transfer Across Embodiments

  • 13/20 to 19/20 soda-can lifting success and nearly an order-of-magnitude fewer tokens show that simulation-discovered skills can help real-robot programming across embodiment and API changes.Bowl placement succeeds with and without retrieval but uses fewer tokens with the skill; drawer manipulation reaches 11/20 with guidance, while the baseline exhausts a larger token budget without a successful evaluation program. This is in-context guidance, not direct policy deployment.Source: Table 1 · p. 10, §3.6 · p. 10

3.7. Ablation Studies

  • 14% to 62% macro-average success comes from adding the robot execution engine, and evolutionary search raises success to 72% while improving remaining hard tasks.Across low-performing tasks, success rises steadily in the first evolutionary-search rounds and then more gradually, indicating diminishing returns from further rounds.Source: §3.7 · p. 10, Figure 6 · p. 11

4. Related Work

ASPIRE builds on agentic robot control and self-improving agents, distinguishing itself through persistent embodied improvement: it diagnoses failures, validates repairs, and reuses validated repair knowledge across tasks.

Agentic robot control

Self-improving agents and skill libraries

5. Limitations

ASPIRE shows selected simulation-discovered skills can support real-robot debugging, but its current loop remains limited by real-world deployment requirements, dependence on a frontier LLM and predefined primitives, library-management challenges, and compute costs.

5. Limitations

  • Selected simulation-discovered skills support debugging on another embodiment, but ASPIRE is not yet a fully autonomous real-world lifelong learner because success detection, safe resets, monitoring, and calibration remain open.Future work should close the evaluation-and-reset loop and scale sim-to-real transfer across broader real-world task suites.Source: §5 · pp. 11–12
  • ASPIRE relies on frozen Claude Opus 4.6 with a 1M-token context window, and whether smaller or weaker LLMs can sustain its debugging loop is unverified.The LLM interprets multimodal traces, writes program repairs, and proposes evolutionary-search candidates.Source: §5 · p. 12
  • ASPIRE’s predefined perception, planning, and control primitives make debugging tractable and safe but bound expressible behavior, sometimes requiring inefficient approximations or human API extensions.Future systems should study safe agent proposals, validation, and incorporation of new robot primitives.Source: §5 · p. 12
  • The skill library prioritizes validated reusable repairs but lacks full long-term memory management, so stale, overly specific, redundant, or misleading entries may contribute to non-monotonic zero-shot transfer.More robust retrieval, pruning, ranking, and re-validation are needed as the library grows.Source: §5 · p. 12
  • ASPIRE’s debugging and evolutionary-search loop consumes many LLM calls and simulator or robot rollouts per task, making large-suite scaling costly.Scaling will require cheaper inference, more sample-efficient search, or stronger reuse of prior repairs.Source: §5 · p. 12

6. Conclusion · A. Skill Library Details · B. Main Benchmark Result Tables

ASPIRE continually writes and refines robot control programs, compounding experience into a reusable skill library. Its appendix details representative skill entries across debugging, localization, navigation, grasping, motion primitives, and scene reasoning.

6. Conclusion

  • Across diverse benchmarks, ASPIRE outperforms existing VLA and coding-agent baselines, shows zero-shot transfer to unseen long-horizon tasks, and offers initial evidence of sim-to-real transfer that reduces real-robot programming token cost.The conclusion attributes this transfer to skills discovered in simulation and notes that it spans different robot embodiments and APIs.Source: §6 · p. 12

A. Skill Library Details

  • Each skill-library entry records the triggering failure problem, a situational retrieval guard, a validated repair snippet, and the task or tasks that produced it.The appendix expands representative Figure 3 entries into the library taxonomy.Source: A. Skill Library Details · p. 17
  • Debugging entries encode reusable strategies with failure signatures, when-to-apply guards, and validated repair sketches.Source: Figure 7 · p. 17
  • Localization entries ground ambiguous language and object references in robust perception and localization routines.Source: Figure 8 · p. 17
  • Navigation entries recover from motion-planning failures and select collision-aware approach poses.Source: Figure 9 · p. 18
  • Strategic grasping entries choose task-appropriate grasp points and adapt strategy to object geometry and scene context.Source: Figure 10 · p. 18
  • Motion-primitive entries capture reusable low-level motion patterns, contact-rich alignment, and execution-time recovery.Source: Figure 11 · p. 18
  • Scene-reasoning entries address spatial relations, support surfaces, occlusions, and scene-level task constraints.Source: Figure 12 · p. 19

B.1. Macro-Averaged Main Results · C. LIBERO-Pro Long Zero-Shot Transfer

The supplied passages describe held-out evaluations across LIBERO-Pro, Robosuite, and BEHAVIOR-1K, specifying benchmarks, perturbations, tasks, and aggregation methods but not numerical outcomes.

B.1. Macro-Averaged Main Results

  • Table 2 compares Aspire with four baselines across three LIBERO-Pro suites under position and instruction perturbations.It reports macro-averaged performance across 10 tasks per suite, with overall columns averaged across the three suites.Source: Table 2 · p. 20
  • Aspire’s LIBERO-Pro results use a 50-seed held-out evaluation on seeds 1–50.The table caption points to Appendix D.1 for per-task breakdowns.Source: Table 2 · p. 20
  • The supplied passages provide no LIBERO-Pro Long zero-shot transfer results or numerical benchmark outcomes.The captions establish evaluation scopes and conditions only; they do not show table cell values.Source: Table 2 · p. 20, Table 3 · p. 20, Table 4 · p. 20
  • Table 3 evaluates Aspire and CaP-Agent0 on seven Robosuite manipulation tasks using 100 held-out trials.It reports task success rates in [0, 1].Source: Table 3 · p. 20
  • Robosuite’s Average is the unweighted mean across all seven tasks.The caption defines the evaluation measure but supplies no result values.Source: Table 3 · p. 20
  • Table 4 covers BEHAVIOR-1K Soda Can pick-up and Radio pick-up, reporting navigation and task success separately.The held-out evaluation uses 25 seeds (seeds 1–25); Aspire uses interactive block-by-block generation and a skill library accumulated on seeds 26–35.Source: Table 4 · p. 20

C.1. Macro-Averaged Main Results

Table 5 describes macro-averaged zero-shot transfer of the Aspire skill library, accumulated on LIBERO-90, to LIBERO-Pro Long. It evaluates success across position and instruction perturbations without additional test-time debugging.

C.1. Macro-Averaged Main Results

  • Table 5 reports macro-averaged zero-shot transfer of the Aspire skill library, accumulated on LIBERO-90, to LIBERO-Pro Long [Zhou et al., 2025].The supplied passage gives the table scope but not its cell values.Source: Table 5 · p. 21
  • The library is seeded by repair skills from N LIBERO-90 tasks, with no additional debugging performed at test time.N denotes the number of LIBERO-90 tasks used to seed the library.Source: Table 5 · p. 21
  • Success is macro-averaged over 10 tasks per axis: Pos for position perturbation and Task for instruction perturbation.Source: Table 5 · p. 21

C.2. Per-Task Breakdown · D. Ablation Details

The supplied passage describes a per-task analysis of zero-shot transfer from an ASPIRE skill library accumulated on LIBERO-90 to LIBERO-Pro Long, across snapshot sizes and perturbation variants.

C.2. Per-Task Breakdown

  • Table 6 covers per-task zero-shot transfer to LIBERO-Pro Long across snapshot sizes N, using seeds 1–50 (n=50), with positional- and semantic-perturbation variants.The library was accumulated on LIBERO-90; the caption provides no per-task result values.Source: Table 6 · p. 21

D.1. LIBERO-Pro Position Perturbation (Swap) Per-Task Breakdown

The section presents a per-task ablation of ASPIRE on LIBERO-Pro position perturbation, comparing zero-shot, execution-engine-repaired, evolutionary-search, and validation-selected programs.

D.1. LIBERO-Pro Position Perturbation (Swap) Per-Task Breakdown

  • ASPIRE selects each task’s winner on held-out seeds 66–80 between the repaired program and evolutionary-search result, while evaluation uses seeds 1–50 (n=50).The ablation also defines the zero-shot baseline as Claude Opus 4.6 with 15 example programs and the repaired-program variant as the execution engine plus skill library.Source: Table 7 · p. 22

D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown

Table 8 reports LIBERO-Pro task-perturbation per-task ablations over 50 seeds. It compares zero-shot, execution-engine-repaired, evolutionary-search, and validation-selected Aspire programs.

D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown

  • The zero-shot variant uses Claude Opus 4.6 with 15 example programs, while Aspire without evolutionary search uses the execution-engine-repaired program and skill library.The table evaluates seeds 1–50 (n=50).Source: Table 8 · p. 23
  • Aspire selects each task’s winner on held-out seeds 66–80 between the repaired program and the best validated evolutionary-search candidate.The table reports the raw evolutionary-search result as “–” when evolutionary search was not run for that task.Source: Table 8 · p. 23

D.3. Evolutionary Search Progress Per-Task Breakdown … E.3. ASPIRE Fix Loop Skills

This section documents per-task evolutionary-search progress and ASPIRE’s repair process, from diagnosing robot-trial failures and developing task-level fixes to validating them and promoting reusable patterns into the skill library. It also specifies constraints on debugging data, seeds, attempts, and reporting.

D.3. Evolutionary Search Progress Per-Task Breakdown

  • Table 9 reports held-out success rates across iterations for the best evolutionary-search candidate on selected LIBERO-Pro tasks; blank cells mark early search termination.Rates are measured over seeds 1–50.Source: Table 9 · p. 24

E.1. Skill Library Prompts

  • Actor agents report each repair’s failure mode, validated fix, transferable patterns, task-specific quirks, and debug-configuration success rate; the coordinator admits only audited, validated reusable fixes.The coordinator serializes skill admission to prevent conflicting library writes.Source: §E.1 · p. 25

E.2. Agent System Prompt and Persistent Memory

  • ASPIRE’s persistent guidance describes a self-improving coding agent that debugs failures, accumulates reusable skills, and improves perception tools using a curated robot API and simulator feedback.The guidance forbids simulator ground-truth access in fix code, keeping diagnosis based on transferable observations and traces.Source: §E.2 · pp. 25–26

E.3. ASPIRE Fix Loop Skills

  • The fix loop dispatches one task per free device, redispatches after completion, and promotes only generalizable patterns from findings into shared skills.Completed tasks are not redispatched, avoiding duplicate trial directories that could corrupt aggregate results.Source: §E.3 · pp. 28–29
  • Debugging uses non-held-out seeds only; the finalized repair is evaluated once on held-out validation seeds, and violations invalidate the benchmark result.Stage 2 is explicitly one-shot and must not be debugged.Source: §E.3 · pp. 30, 32
  • The repair loop diagnoses failed trials from traces, code, keyframes, and summaries, using signals such as gripper width, perception masks, and IK failures to guide fixes.Repairs are replayed on debugging seeds and iterated from new traces; debugging is capped at three replay attempts per seed.Source: §E.3 · p. 31
  • The final task-level repair program must generalize across seeds and use observation-derived perception and manipulation APIs, not simulator internals or ground-truth state.The protocol says simulator internals would not transfer to real robots and would invalidate the result.Source: §E.3 · pp. 30, 32
  • The findings format records concise descriptions of each failure mode, changes that resolved each root cause, effective perception prompts with caveats for each object, generalizable patterns likely to apply beyond the task, and task-specific quirks.Source: §E.3 · p. 32

E.4. ASPIRE Evolutionary Search Skills · E.5. Initial Skill Library Templates

This appendix describes an evolutionary debugging workflow alongside initial reusable skill-library templates for manipulation and robot motion. The templates separate shared patterns from task-specific parameters and validated discoveries, and are designed to grow through validated experience.

E.4. ASPIRE Evolutionary Search Skills

  • Evolutionary search carries a task-analysis document across rounds, recording scene geometry, hypotheses, tested and blocked directions so later rounds can avoid repeats or retry blocked approaches.UpdateAnalysis revises the document from new traces, keyframes, and library retrievals.Source: §E.4 · p. 33
  • For low-performing tasks, evolutionary debugging evaluates multiple candidates per iteration, each testing a distinct hypothesis, to explore strategies a single-pass actor may miss.Search stops when a candidate reaches the debug-seed threshold, the iteration limit is reached, or the task is blocked; stalled search should explore structurally new approaches.Source: §E.4 · pp. 33, 34, 36
  • Held-out evaluation seeds are excluded from iterations; after selecting code, Stage 2 runs it once per held-out seed without using those results to revise the code.Iterations use the same fixed debug seeds, which may be unrepresentative, so candidates should target unseen seeds rather than fit debug-seed patterns.Source: §E.4 · pp. 35–37

E.5. Initial Skill Library Templates

  • The initial grasp skill is a structural pick-and-place template comprising topdown-quaternion construction, pre-grasp approach, lower-and-close, and lift-transport-place, without task-specific parameters.Task-specific parameters are intended to grow beyond the template.Source: §E.5 · p. 38
  • The pick-and-place template sequences grasp planning, pre-grasp approach, grasp, lift, transit, release, and settling, while excluding shelves, stacking, and narrow containers.Its guidance favors scene-relative heights, a top-down grasp with an argmax fallback, and task-specific offsets and yaw outside the shared skeleton.Source: §E.5 · pp. 38–39
  • The localization skill tries SAM3 text prompts in order, converts valid masks to 3D points and an oriented-box center, and supports Molmo grounding as a fallback.It also records reusable prompts and disambiguation guidance as discoveries are validated.Source: §E.5 · pp. 39–41
  • A separate manipulation skill covers drawers, knobs, hinged doors, and pushing, with validated object- or action-specific parameters added to its registry.The template requires physical interaction rather than forcing joint positions or reading task specifications for parameters.Source: §E.5 · pp. 41–42
  • The transport skill uses a lift-transit-descend pattern to move grasped objects above clutter before lowering to the target release height.Its example lifts 0.15 above the grasp position and sets release height 0.05 above the target surface.Source: §E.5 · p. 42
  • Three cardinal waypoints—up, over, then down—keep a grasped object clear of obstacles during transit, unlike a direct grasp-to-release move that can clip tall objects or openings.The passage identifies basket walls and drawer openings as examples of collision risks.Source: §E.5 · p. 42
  • For LIBERO-Long-Pro subtasks, open the gripper, return the arm home, and collect three observations after release before localizing the next object.The reset provides a known-safe configuration, lets physics settle, and prevents the arm from obstructing the next perception step.Source: §E.5 · p. 43
  • In the far workspace, use solve_ik and move_to_joints instead of goto_pose; if inverse kinematics fails, retry with a target shifted 0.02 lower before reporting infeasibility.The template treats goto_pose as unreliable for targets such as the back of a counter or far side of a table.Source: §E.5 · p. 43
  • A waypoint registry is designed to grow per chunk with validated multi-step transit patterns for scenes where direct motion fails.Its format records the scenario, waypoint pattern, first task, and why the default three-waypoint pattern failed; the registry starts empty for coordinator population.Source: §E.5 · p. 43
  • The templates warn against whole-scene direct goto_pose moves, lifting at grasp height, descending without rim clearance, stale post-release observations, and goto_pose in the far workspace.These failure patterns respectively risk clutter collisions, sideways object dragging, container-wall collisions, unsettled objects, and unreliable motion.Source: §E.5 · p. 43

Related papers

Loading 2607.00272v1…