PaperlayerView exact revision on arXiv

Source-linked AI summary

ASPIRE: Agentic /Skills Discovery for Robotics

Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi "Jim" Fan, Guanzhi Wang

arXiv:2607.00272v1cs.ROcs.AIcs.MA

TL;DR

Robot coding agents struggle with coarse feedback and do not retain debugging experience across tasks. ASPIRE addresses this with an open-ended code-as-policy learning loop that diagnoses and repairs programs while storing validated fixes in a transferable skill library and exploring alternatives evolutionarily. Across benchmarks, it substantially outperforms prior methods, including 31% versus 4% zero-shot success on LIBERO-Pro Long, while showing initial sim-to-real transfer evidence.

  • Problem

    Robotic coding agents have coarse execution feedback and discard debugging fixes, limiting failure localization and experience accumulation across tasks.

  • Method

    ASPIRE combines closed-loop multimodal execution traces, continual storage of validated repairs as reusable skills, and evolutionary search over robot programs.

  • Results

    ASPIRE substantially outperforms existing VLA and coding-agent baselines across diverse benchmarks, reaching 31% versus 4% success on LIBERO-Pro Long.

  • Takeaways & Limitations

    Validated skills transfer to unseen long-horizon tasks and provide initial evidence of sim-to-real transfer across embodiments and robot APIs.

  • Takeaways & Limitations

    Real-world deployment still requires robust success detection, safe resets, safety monitoring, and calibration maintenance, and the method relies on a frozen frontier LLM and predefined robot APIs.

Abstract

from arXiv · show

Traditional robot programming is challenging: it requires orchestrating multimodal perception, managing physical contact dynamics, and handling diverse configurations and execution failures. We introduce ASPIRE (Agentic Skill Programming through Iterative Robot Exploration), a continual learning system that autonomously writes and refines robot control programs in a code-as-policy paradigm while compounding experience into a reusable skill library. ASPIRE discovers skills that persist across tasks, simulation and real-world settings, and embodiments. It operates in an open-ended loop with three components: (1) a closed-loop robot execution engine that exposes fine-grained multimodal traces, enabling autonomous failure diagnosis, repair synthesis, and validation; (2) a continually expanding skill library that distills validated fixes into reusable, transferable knowledge; and (3) evolutionary search that generates diverse task sequences and control programs to explore beyond single-trajectory refinement. ASPIRE surpasses prior methods by up to 77% on LIBERO-Pro manipulation under perturbation, 72% on Robosuite bimanual handover, and 32% on BEHAVIOR-1K long-horizon household tasks. Its accumulated library also enables zero-shot generalization to unseen long-horizon tasks: on LIBERO-Pro Long, ASPIRE achieves 31% success versus 4% for prior methods despite their use of test-time reasoning and retries. Finally, simulation-discovered skills provide initial evidence of sim-to-real transfer, substantially reducing real-robot programming effort across different embodiments and robot APIs.

1. Introduction

ASPIRE addresses the limits of coarse robot feedback and discarded debugging experience by combining fine-grained execution traces, continual skill accumulation, and autonomous program refinement. It evaluates this approach across diverse robot benchmarks and reports substantial gains over prior coding agents.

  • Motivation: Existing robotic coding agents receive coarse task-level feedback, making it difficult to localize failures across perception, planning, grasping, contact dynamics, and recovery.Fine-grained traces are needed to determine what evidence to inspect, where failures originate, and which repair strategy to attempt.
  • Motivation: Debugging experience is not accumulated across tasks, so discovered fixes and recovery strategies are discarded instead of becoming reusable skills.Human robotics engineers instead internalize transferable strategies such as grasp recovery heuristics, navigation strategies, prompting recipes, and procedural fixes.
  • ASPIRE: ASPIRE autonomously writes and refines code-as-policy robot programs while accumulating experience in a reusable skill library.Its open-ended loop lets the agent inspect traces, diagnose failures, synthesize repairs, validate corrected behaviors, and consolidate recovery patterns.
  • ASPIRE: ASPIRE combines a closed-loop execution engine with per-primitive multimodal traces, a growing skill library, and evolutionary search over diverse candidate programs.The execution engine records observations, inputs, outputs, and visual evidence for perception, planning, grasping, and control calls; evolutionary search conditions later candidates on surviving programs and failure traces.
  • Results: ASPIRE improves success rate by up to 77 points on LIBERO-Pro, 72 points on Robosuite, and 32 points on BEHAVIOR-1K against prior coding agents.Its accumulated LIBERO-90 skill library also reaches 31% zero-shot success on LIBERO-Pro Long versus 4% for prior methods.

2. Method

ASPIRE combines trace-rich closed-loop execution, a reusable skill library, and evolutionary program search in an open-ended robot-learning loop. Actors diagnose and repair programs, while validated fixes are consolidated for reuse across tasks and embodiments.

  • ASPIRE combines a robot execution engine, skill library, and evolutionary search procedure into an open-ended learning loop.The engine supports failure attribution and validation, the library accumulates reusable repairs, and search broadens exploration beyond single-trajectory repair.
  • The coordinator dispatches task-specific coding agents, while transferable experience is distilled into a shared skill library rather than exchanged as full histories or raw trajectories.Each actor writes, executes, diagnoses, and repairs robot programs within the execution engine.
  • A BEHAVIOR-1K radio episode localizes repeated PLANNING_ERRORs to navigation targets inside a table’s collision buffer, then repairs the program with a multi-angle approach.The robot re-perceives the radio from a reachable side, completes the grasp, and admits the validated repair as a reusable skill.
  • Per-primitive multimodal traces expose API calls, inputs, outputs, statuses, and evidence for diagnosing and validating agent-written repairs.Evidence includes RGB keyframes, overlays, grasp candidates, object poses, and motion information.
  • The skill library stores validated, heterogeneous repair patterns—including localization, grasping, navigation, motion, scene understanding, and debugging knowledge—in compact reusable guidance.Entries encode failure signatures, application conditions, repair strategies, and sometimes representative code sketches; only validated reusable patterns are admitted.
  • Evolutionary search proposes and executes populations of candidate programs conditioned on strong prior programs, failure traces, and the skill library.Validated repairs enter the library after generalizing across environment variations and tasks; search ends when debugging configurations are solved or the budget is exhausted.

3. Experiments

Experiments evaluate Aspire across short- and long-horizon simulation benchmarks, zero-shot transfer, real-robot cross-embodiment transfer, and component ablations. Aspire improves performance broadly, while accumulated skills reduce real-robot debugging cost and execution-engine traces provide the largest ablation gain.

  • Benchmarks and Protocol: Aspire is evaluated on LIBERO-Pro, Robosuite, and BEHAVIOR-1K using executable robot programs and held-out evaluation settings.The benchmarks cover perturbation robustness, contact-rich manipulation, and long-horizon household mobile manipulation.
  • Main Evaluation Results: 77% on LIBERO-Pro Object, 41.5% on Goal, and 42.5% on Spatial exceed the strongest baseline gains across the perturbation suites.On Robosuite bimanual handover, success rises from 20% to 92%; on BEHAVIOR-1K navigate-and-pick-up-radio, task success rises from 56% to 88%.
  • Zero-Shot Transfer: 23% position-perturbation success and 38% task-perturbation success are achieved on LIBERO-Pro Long with the full N=90 skill library.Aspire generates one program per held-out task and evaluates it without additional debugging, retries, or task-specific library updates.
  • Zero-Shot Transfer: Success increases consistently as the LIBERO-Pro Long skill library grows, indicating reuse of validated short-horizon repairs in longer-horizon compositions.The transfer study evaluates library sizes N∈{0, 25, 50, 90}.
  • Real-Robot Skill Transfer: Transferred skills consistently reduce real-robot debugging cost, while their effect on final success is task-dependent across embodiment and API changes.Soda-can lifting improves from 13/20 to 19/20 while reducing total tokens by nearly an order of magnitude; drawer manipulation reaches 11/20 with skill guidance.
  • Ablation Studies: 72% macro-average success is reached with both the robot execution engine and evolutionary search, up from 14% without either component.The execution engine raises success to 62%, providing the largest average improvement, while evolutionary search further improves remaining hard tasks with diminishing returns across later iterations.

4. Related Work

Related work connects Aspire to executable robot programs, software-engineering agents, self-improving agents, and skill libraries. Aspire differs by preserving validated repair knowledge from embodied failures and reusing it for future program repair.

  • Agentic Robot Control: Executable robot-control systems compose perception, planning, and control APIs, while software-engineering agents use write-execute-debug loops for code.Aspire extends this paradigm toward persistent embodied improvement through trace selection, repair validation, and experience preservation.
  • Self-Improving Agents and Skill Libraries: Aspire stores validated repair knowledge extracted from attributed embodied failures rather than only success memories, textual reflections, or reward functions.Its open-ended skills span heterogeneous categories and are reused as in-context guidance; evolutionary search broadens executable repair exploration before admission.

5. Limitations

ASPIRE remains limited in real-world autonomy, model dependence, exposed robot capabilities, and long-term skill-library management. Broader sim-to-real deployment requires closing evaluation and reset challenges.

  • Real-world deployment is not yet fully autonomous because it still requires robust success detection, safe resets, safety monitoring, and calibration maintenance.
  • Future work should close the evaluation-and-reset loop and scale sim-to-real skill transfer across broader real-world task suites.
  • The method relies on a frozen frontier LLM, and the authors have not verified equivalent debugging performance with smaller or weaker models.
  • A predefined perception, planning, and control API bounds the behaviors ASPIRE can express when tasks require capabilities outside its exposed primitives.
  • The skill library does not fully solve long-term memory management, so stale, overly specific, redundant, or misleading entries may affect zero-shot transfer.

6. Conclusion

ASPIRE is a continual-learning robotic system that combines multimodal execution traces, a reusable skill library, and evolutionary search. It outperforms existing baselines across diverse benchmarks and shows zero-shot and initial sim-to-real transfer evidence.

  • ASPIRE autonomously writes and refines robot control programs while compounding experience into a reusable skill library.
  • Its open-ended loop combines fine-grained multimodal traces, validated transferable fixes, and evolutionary search over diverse task sequences and control programs.
  • Across diverse benchmarks, ASPIRE substantially outperforms existing VLA and coding-agent baselines.
  • ASPIRE demonstrates strong zero-shot transfer to unseen long-horizon tasks and initial evidence of sim-to-real skill transfer across embodiments and APIs.

A. Skill Library Details

The skill library organizes validated, reusable robot-program repairs across debugging, localization, navigation, grasping, motion, and scene reasoning. Entries pair task-specific conditions with concrete recovery strategies.

  • Debugging skills: Skill-library entries encode failure signatures, when-to-apply guards, validated repair sketches, and the origin tasks that produced them.
  • Localization skills: Localization skills ground ambiguous language and object references into robust perception and localization routines.
  • Navigation skills: Navigation skills recover from motion-planning failures and select collision-aware approach poses.
  • Strategic grasping skills: A tall-cylinder grasp aligns gripper yaw with the object’s oriented-bounding-box major axis and uses staged closing before a slow lift.The strategy targets cylindrical objects with aspect ratio > 2.0, including wine bottles, tall cans, and spray bottles.
  • Motion-primitive skills: Motion-primitive skills provide reusable low-level motion patterns, contact-rich alignment, and execution-time recovery.
  • Scene-reasoning skills: Scene-reasoning skills address spatial relations, support surfaces, occlusions, and scene-level task constraints.

B.1. Macro-Averaged Main Results

The main results tables report macro-averaged performance on LIBERO-Pro, Robosuite, and BEHAVIOR-1K under held-out evaluations. They compare ASPIRE with prior methods across perturbation, manipulation, and household-task settings.

  • LIBERO-Pro: Table 2 compares OpenVLA, π0, π0.5, CaP-Agent0, and ASPIRE on three LIBERO-Pro suites under position and instruction perturbations.Results are macro-averaged across 10 tasks per suite and across the three suites overall, using 50 held-out seeds.
  • Robosuite: Table 3 reports CaP-Agent0 and ASPIRE success rates on seven Robosuite manipulation tasks using a 100-trial held-out evaluation.The average is the unweighted mean across all seven tasks.
  • BEHAVIOR-1K: Table 4 separately reports navigation and task-success rates for Soda Can pick-up and Radio pick-up in BEHAVIOR-1K.Evaluation uses 25 held-out seeds, while ASPIRE uses a skill library accumulated on seeds 26–35.

C.1. Macro-Averaged Main Results

Table 5 reports macro-averaged zero-shot transfer from a skill library accumulated on LIBERO-90 to LIBERO-Pro Long across positional and semantic perturbations. It evaluates 10 tasks per perturbation axis without additional debugging at test time.

  • C.1. Macro-Averaged Main Results: Macro-averaged success is reported separately for Pos and Task perturbation axes across 10 tasks each.N denotes the number of LIBERO-90 tasks whose repair skills seed the library.
  • C.1. Macro-Averaged Main Results: Zero-shot transfer is measured using a library accumulated on LIBERO-90, with no additional debugging performed at test time.
  • C.1. Macro-Averaged Main Results: The table varies snapshot size N to characterize how the number of library-seeding tasks relates to transfer performance.

C.2. Per-Task Breakdown

Table 6 breaks down zero-shot transfer from the LIBERO-90 skill library to LIBERO-Pro Long at the individual-task level. Results are shown across snapshot sizes N and two perturbation variants.

  • C.2. Per-Task Breakdown: Per-task success rates are reported for LIBERO-Pro Long under positional and semantic perturbations.Pos denotes positional perturbation, while Task denotes semantic perturbation.
  • C.2. Per-Task Breakdown: The breakdown evaluates seeds 1–50 with n=50 for each task and snapshot size N.
  • C.2. Per-Task Breakdown: Snapshot size N indexes the LIBERO-90 skill-library configurations used for zero-shot transfer.

D.1. LIBERO-Pro Position Perturbation (Swap) Per-Task Breakdown

Table 7 compares per-task LIBERO-Pro position-perturbation performance across ASPIRE ablations and evolutionary-search results. Evaluation uses seeds 1–50, with ASPIRE winners selected on held-out validation seeds 66–80.

  • D.1. LIBERO-Pro Position Perturbation (Swap) Per-Task Breakdown: Per-task position-perturbation results are reported on seeds 1–50 with ASPIRE selection based on held-out seeds 66–80.
  • D.1. LIBERO-Pro Position Perturbation (Swap) Per-Task Breakdown: The ablation compares ASPIRE against versions without the robot execution engine and evolutionary search.
  • D.1. LIBERO-Pro Position Perturbation (Swap) Per-Task Breakdown: Evolutionary search is also shown as a raw best validated candidate, while ASPIRE selects a per-task winner on held-out validation data.

D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown

Tables 8 and 9 examine LIBERO-Pro task-perturbation performance and evolutionary-search progress. The comparisons distinguish ablated systems, raw evolutionary-search candidates, and ASPIRE’s held-out-selected winners.

  • D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown: Table 8 reports per-task LIBERO-Pro task-perturbation results using seeds 1–50 and held-out validation seeds 66–80.
  • D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown: The ablation compares ASPIRE without the robot execution engine and evolutionary search against ASPIRE without evolutionary search.The latter uses a robot-execution-engine repaired program with the execution engine and skill library.
  • D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown: Table 9 tracks the held-out success rate of the best candidate at each evolutionary-search iteration on selected LIBERO-Pro tasks.Blank cells indicate that search terminated early.
  • D.2. LIBERO-Pro Task Perturbation (Goal) Per-Task Breakdown: Candidate repairs are structured around failure modes, validated repairs, transferable patterns, task-specific quirks, and validation success rates.

E.4. ASPIRE Evolutionary Search Skills

ASPIRE’s evolutionary search explores diverse control programs through iterative, multi-candidate debugging while preserving task-specific analysis across rounds. It uses fixed debug seeds for fair comparison, then validates converged code on held-out seeds without evaluation-set leakage.

  • Search state: Persistent task analysis records scene details, hypotheses, candidate metadata, eliminated directions, and blocked directions across search rounds.New traces, keyframes, and skill-library retrievals rewrite the analysis so later rounds avoid eliminated branches while revisiting blocked ones when new techniques become available.
  • Candidate generation: Evolutionary search runs multiple structurally distinct candidates per iteration, each testing a different hypothesis and documenting its expected failure mode.Existing baseline repair code seeds candidate_A when available, while candidates are designed to avoid duplicating the same failure reason.
  • Distributed execution: The workflow dispatches one subagent per task and compute device, keeps available devices occupied, and redispatches the next pending task after completion.Each subagent runs debugging iterations and then Stage 2 before returning a structured report with both results.
  • Stopping and exploration: Stage 1 stops when a candidate reaches the debug success threshold, the iteration budget is exhausted, or the task is blocked; blocked tasks skip Stage 2.When progress stalls, the remaining budget is used to explore structurally new approaches rather than stopping immediately.
  • Evaluation protocol: Fixed debug seeds support fair cross-iteration comparisons, while Stage 2 evaluates the selected final code on held-out seeds.Held-out seeds are locked out during iterations; violating this separation invalidates the benchmark.
  • Generalization: Debug seeds may be small and unrepresentative, so candidates should target mechanistic robustness rather than fitting observed seed-specific patterns.The final evaluation uses unseen held-out seeds, discouraging hard-coded thresholds, masks, or offsets derived from debug observations.
Loading 2607.00272v1…