PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboAuto: Autonomous Robot Program Search through Multi-Level Physical Understanding

Buzhao Liu, Jianing Ye, Xiu Yuan, Yangtong Feng, Chengsong Huang, Tyler Dinh, Yevgeniy Vorobeychik, Ning Zhang, Chongjie Zhang

arXiv:2610.36533v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Robot program search can leave physical questions unresolved when it relies mainly on execution feedback. RoboAuto combines object-centric knowledge, scene grounding, verified multi-stage planning, and targeted experiments; it reports more than twice ASPIRE’s mean success on LIBERO-PRO Long and LIBERO-Plus. Its knowledge also transfers across simulators and embodiments, though real-robot rollouts still require human resets and safety monitoring.

  • Problem

    Existing robot program-search methods rely mainly on execution feedback, which can leave unresolved how objects should be manipulated, how geometry constrains actions, and why interactions succeed or fail.

    Source: §1 · pp. 2–3

  • Method

    RoboAuto combines a hierarchical knowledge base and scene grounding with stage-wise verification and self-designed experiments to investigate physical uncertainties.

    Source: §1 · p. 3, §3 · p. 4, §3.3 · p. 5

  • Results

    RoboAuto more than doubles ASPIRE’s performance on LIBERO-PRO Long and LIBERO-Plus, improves task success across five benchmark families, and transfers knowledge to a new simulator and embodiment.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The results support the complementary roles of transferable physical knowledge and executable robot programs in program search.

    Source: §5 · p. 8

    RoboAuto searches autonomously only in simulation; real-robot rollouts still require a human to reset scenes and keep the arm and surroundings safe.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recently, LLM-based autonomous agents have shown strong capabilities in iteratively generating, evaluating, and refining executable solutions. In robotic manipulation, Code-as-Policy lets these agents search over modular and editable robot programs. However, effective program search requires physical understanding: agents must understand how objects can be manipulated, how scene geometry constrains actions, and why interactions succeed or fail. Existing methods rely mainly on execution feedback for program refinement, often leaving these physical questions unresolved. We introduce RoboAuto, a framework for robot program search that explicitly organizes physical understanding at the object, scene, and interaction levels through three components. First, a hierarchical knowledge base combines executable skills with object-centric knowledge of affordances, spatial relations, and action effects, learned from embodied data and online task solving. Second, scene grounding connects this knowledge to the current environment by identifying relevant objects and resolving action-critical quantities. Third, multi-stage planning with self-designed experiments checks whether each stage produces its expected physical effects and actively tests competing hypotheses to resolve physical uncertainties. RoboAuto achieves superior performance across five benchmark families, attaining an 89.1% success rate on short-horizon LIBERO-PRO tasks and more than twice the mean success rate of ASPIRE on LIBERO-PRO Long and LIBERO-Plus. It also outperforms ASPIRE and $\pi_{0.5}$ on three challenging long-horizon real-robot tasks requiring precise manipulation. We further show that the knowledge base transfers across simulators and embodiments, and that program search performance scales with its size. Project page: https://roboauto-ai.github.io/

1. Introduction

RoboAuto argues that robot program search needs physical understanding in addition to execution feedback. It combines object-centric knowledge, scene grounding, and verified planning with experiments, and reports gains across five benchmark families.

1. Introduction

  • Baseline runs expose gaps in affordance knowledge, scene-specific geometric grounding, and verification of intermediate physical effects.A failed knob rotation also leaves uncertainty about how far the knob must turn, motivating experiments that test specific hypotheses.Source: §1 · pp. 2–3
  • RoboAuto combines object-centric affordances, spatial relations, and action effects with executable skills, grounds them in the current scene, and verifies stage outcomes.When uncertainty remains, it tests competing hypotheses; interaction evidence refines physical knowledge, while verified programs expand the skill library.Source: Figure 2 · p. 2, §1 · p. 3
  • 73.4% mean success on LIBERO-PRO Long versus 30.8% for ASPIRE, and 90.3% on LIBERO-Plus versus 44.7% for ASPIRE; RoboAuto also reaches 89.1% on short-horizon LIBERO-PRO.It obtains 90.6% mean success on Robosuite versus ASPIRE’s 81.0%, and 91.5% on RoboTwin 2.0 versus 24.5% without a knowledge base.Source: §1 · p. 3
  • The framework’s contributions include grounding and multi-stage planning with grounded verification, alongside autonomously designed experiments.Its evaluation spans five benchmark families, and the paper reports knowledge scaling and transfer across simulators and embodiments.Source: §1 · p. 3

2. Related Work

Prior work spans learned robotic policies, editable robot programs, and reusable physical knowledge, but RoboAuto makes physical understanding itself a persistent, improvable substrate for program search.

Robotic foundation models

  • Unlike robotic foundation models trained to acquire behavior through parameterized policies, RoboAuto searches and refines explicit programs through grounded physical understanding, verification, and exploration.The prior models include diffusion-based policies, vision-language-action models, and world-action models.Source: §2 · p. 3

Code-as-Policy and autonomous robot self-improvement

  • Code-as-Policies [23] makes robot behavior inspectable and revisable, while later work extends program interfaces, execution-based revision, and reuse of validated skills and experience.Examples include CaP-X [13], CaP-Agent0, RHO [11], ASPIRE [25], RATs [46], and RoboRSI [36].Source: §2 · p. 3

Physical knowledge and scene grounding

  • Knowledge-based robotics and scene-grounding methods provide reusable action knowledge or spatial representations, while RoboAuto combines object-centric knowledge with task-specific grounding and online experimental refinement.Related directions include MEMORA [43], general-purpose 3D representations [15] [20], and task-specific spatial constraints [17] [18] [28].Source: §2 · p. 4

3. Method

RoboAuto organizes physical understanding into a hierarchical knowledge base, scene grounding, and search with exploration. These components connect reusable object knowledge and executable skills to the current scene, then verify and investigate physical changes during program search.

3. Method

  • RoboAuto decomposes long-horizon tasks into stages, verifies each intended object-state change before proceeding, and uses targeted experiments to investigate unresolved physical uncertainties.Its three components are a hierarchical knowledge base, physical scene grounding, and search and exploration.Source: §3 · p. 4, §3.3 · p. 5

3.1. Hierarchical Knowledge Base

  • The knowledge base separates transferable physical regularities from executable skills, which are verified programs retrievable as candidate implementations or reusable building blocks.Verified programs from online task solving expand the skill library.Source: §3.1 · p. 4
  • Object-centric entries encode affordances, spatial relations, action effects, preconditions, applicability conditions, and measurement procedures, drawing on embodied demonstrations and online task-solving evidence.Their scene-dependent quantities are measured for current objects and the robot during physical scene grounding.Source: §3.1 · p. 4

3.2. Physical Scene Grounding

  • Scene grounding identifies task-relevant objects and their interaction roles, then measures action-critical quantities rather than constructing an exhaustive representation of the scene.It can use visual perception, depth observations, and active viewpoint selection when an initial view is insufficient.Source: §3.2 · pp. 4–5
  • Grounded measurements and unresolved conditions inform program generation and further investigation, while execution outcomes can trigger remeasurement and reassessment of knowledge applicability.This keeps grounding revisable as execution reveals new conditions or changes the scene.Source: §3.2 · p. 5

3.3. Search and Exploration

  • Sequential stage search uses a fresh-observation verifier and stops at an unverified stage, preserving the verified program prefix and localizing revision to an incomplete physical transition.An independent auditor reviews unsuccessful executions using the stage plan, physical knowledge, logs, and synchronized scene and wrist-camera videos; its judgments can prompt revisions to the verifier, stage program, or plan.Source: §3.3 · p. 5
  • Dedicated experiment code tests physical hypotheses through predicted observations and targeted interventions, and the resulting evidence can refine both the current program and object-centric knowledge.Controlled grasp-and-lift trials can compare candidate grasp poses without repeatedly executing the full task.Source: §3.3 · p. 5

4. Experiments

Across five benchmark families, RoboAuto evaluates program search under varied manipulation, scene, and embodiment conditions. Its experiments compare the full system with baselines and component variants, and test knowledge transfer, scaling, and real-robot performance.

4.1. Experimental Setup

  • The evaluation spans five benchmark families, uses disjoint search and test scenes, and compares RoboAuto with autonomous Code-as-Policy methods and a vision-language-action baseline.The suites cover perturbations, long-horizon manipulation, scene variation, contact-rich tasks, and cross-embodiment transfer.Source: §4.1 · p. 5

4.2. Main Results

  • RoboAuto achieves the highest mean success across evaluated benchmark suites, with strong results on LIBERO-PRO Long, LIBERO-Plus, and contact-rich Robosuite tasks.It also leads in every evaluation category grouped by similarity to LIBERO-90 demonstrations.Source: §4.2 · pp. 6–7, Figure 4 · p. 6
  • With unseen layouts and objects, RoboAuto achieves 84.3% mean success versus 68.8%, while the knowledge base contains general strategies and measurement targets rather than instance-specific commands.Adding the knowledge base to ASPIRE improves several tasks but does not close the gap, indicating that scene grounding and grounded multi-stage planning also contribute to RoboAuto’s gains.Source: §4.2 · p. 7

4.3.1. Ablation Studies

  • Mean final-program success over five tasks rises from 5.2% for the base loop to 19.2% with Search and Exploration, 38.4% with the Hierarchical Knowledge Base, and 91.2% with full RoboAuto.The full system adds Physical Scene Grounding, producing the largest gain.Source: §4.3.1 · p. 7, Figure 5 · p. 7

4.3.2. Hierarchical Knowledge Base Transferability

  • On RoboTwin 2.0, success rises from 24.5% without retained knowledge to 91.5% with the knowledge base learned from LIBERO, despite changes in simulator, embodiment, layouts, tasks, and control primitives.The accompanying figure compares success on four tasks and marks mean success with dashed lines.Source: §4.3.2 · pp. 7–8, Figure 6 · p. 7

4.3.3. Knowledge Scaling

  • After 20 rollouts, the full knowledge-base condition reaches 90.0% mean success versus 57.2% with half the entries, and all five tasks first reach 80% within 19 rollouts.With half the entries, only three tasks reach 80% over the full budget; without the knowledge base, one does.Source: §4.3.3 · p. 8, Figure 7 · p. 8

4.4. Real-Robot Experiments

  • On three long-horizon WidowX AI tasks, RoboAuto achieves near-perfect stage-completion scores, while ASPIRE makes limited progress and π0.5 makes none.The table reports mean stage-completion scores over 20 trials per task.Source: §4.4 · p. 8, Table 1 · p. 8, Figure 8 · p. 8

5. Conclusion

RoboAuto combines a hierarchical knowledge base, scene grounding, stage-wise verification, and self-designed experiments, improving results across five benchmark families and transferring knowledge across simulators and embodiments. Real-robot autonomy remains limited by human scene resets and safety monitoring.

5. Conclusion

  • Knowledge learned in LIBERO transfers to RoboTwin 2.0, reaching 91.5% success versus 24.5% without retained knowledge.The conclusion attributes this result to complementary roles for transferable physical knowledge and executable robot programs.Source: §5 · p. 8
  • On real robots, a human operator still resets the scene after each rollout and keeps the arm and surroundings safe.Automating both tasks would extend the same autonomy to real-world deployment.Source: §5 · p. 8

A. Detailed Experimental Setup

A.1.1. LIBERO-PRO

  • LIBERO-PRO covers Object, Spatial, and Goal suites under position and task perturbations, for 60 settings total.The suites test object selection, spatial relations, and varied interactions.Source: §A.1.1 · p. 12

A.1.2. LIBERO-PRO Long

  • LIBERO-PRO Long samples ten task–perturbation settings requiring several dependent physical state changes in the correct order.Source: §A.1.2 · p. 12

A.1.3. LIBERO-Plus

  • LIBERO-Plus uses six high-difficulty camera or layout variation settings, requiring re-identification of objects and re-estimation of scene-dependent quantities.Source: §A.1.3 · p. 12

A.1.4. Robosuite

  • Robosuite evaluates seven single- and dual-arm tasks spanning placement, contact, geometric alignment, and manipulator coordination.These tasks introduce interaction structures and configurations that differ from LIBERO despite both using MuJoCo.Source: §A.1.4 · p. 12

A.1.5. RoboTwin 2.0

  • RoboTwin 2.0 tests transfer to a bimanual robot in a different simulator, with changed gripper geometry, control primitives, contact dynamics, layouts, and task definitions.Source: §A.1.5 · p. 12

A.2. RoboAuto Evaluation Protocol

The evaluation uses separate search and test scenes, with shared rollout accounting for program search, scene grounding, and experiments. Configuration and knowledge checks define the agent, baselines, and the abstraction level of stored object-centric entries.

A.2. RoboAuto Evaluation Protocol

  • Search uses training scenes only, while final programs run on disjoint test scenes; each rollout is one program executed in one training scene.Active viewpoint selection and self-designed experiments count toward the rollout budget, and final evaluation follows the stated scene-count protocol.Source: §A.2 · p. 12

A.3. RoboAuto Configuration

  • The configuration uses 15 training scenes and fixed per-task initial knowledge conditions; online knowledge and skills derive only from search evidence.Initial object-centric knowledge comes from one successful LIBERO demonstration video per LIBERO-90 source task.Source: §A.3 · p. 13
  • RoboAuto uses Claude Opus 4.6 for coding and combines Molmo2, SAM 3, and depth observations to identify relevant objects and estimate action-critical quantities.Source: §A.3 · p. 13

A.4. Baselines and Comparison Protocol

  • Comparisons include program-revision, evolutionary-search, and policy-repository methods plus a direct-action VLA baseline; autonomous coding methods share perception and control primitives.Some baseline results are published rather than reproduced, and π0.5 is unavailable for Robosuite two-arm tasks.Source: §A.4 · pp. 13–14

B. Additional Experiment Results

B.1. Generality and Transferability of Object-Centric Knowledge

  • The knowledge base is organized around object classes and reusable interaction requirements, while program search and scene grounding supply concrete programs and parameters.Source: §B.1 · p. 14, §B.1.4 · p. 14
  • Object-centric entries encode qualitative interaction strategies, not executable actions, scene-specific physical values, task instructions, or ordered task plans.The checks also verify that entries are organized by general object categories rather than particular instances.Source: §B.1.1 · p. 14, §B.1.2 · p. 14, §B.1.3 · p. 14

B.1.5. Results by Similarity to the Knowledge Construction Data

RoboAuto performs strongly across categories that differ in layout and object overlap with LIBERO-90 demonstrations. Adding its knowledge base to ASPIRE improves some Spatial tasks but does not reproduce RoboAuto’s results.

B.1.5. Results by Similarity to the Knowledge Construction Data

  • RoboAuto leads in all three similarity categories, with mean success rates of 74.5%, 90.9%, and 84.3%, including when layout and object set are both unseen.The table groups evaluation settings by whether layout and manipulated object set match the LIBERO-90 demonstrations.Source: §B.1.5 · p. 15, Table 3 · p. 15

B.1.6. Hierarchical Knowledge Base with ASPIRE

  • Adding the knowledge base raises ASPIRE’s success on three Spatial tasks, but ASPIRE remains at 0% on Goal tasks and does not close the gap to RoboAuto.On task-perturbed bowl placement on the stove, ASPIRE remains at 0% while RoboAuto reaches 98%.Source: §B.1.6 · p. 15

B.2. Complete Benchmark Results

  • The following tables provide task- and perturbation-level results underlying the main benchmark comparison.Source: §B.2 · p. 16

B.2.1. Per-Task Results

Across benchmark settings, RoboAuto’s results exceed ASPIRE’s aggregate performance, and component and transfer analyses show gains associated with grounding, staged search, and broader retained knowledge.

B.2.1. Per-Task Results

  • Across 83 settings, RoboAuto averages 87.4% vs 65.6% for ASPIRE, outperforming on 56, tying on 9, and trailing on 18.On the 28 settings where ASPIRE is below 50%, RoboAuto averages 76.6% versus 19.5%; both average 97.3% on 37 settings where ASPIRE is at least 90%.Source: §B.2.1 · p. 16

B.3. RQ1: How Does Each Component Contribute?

  • Across five tasks, adding Search and Exploration, the knowledge base, and Physical Scene Grounding raises mean success from 5.2% to 19.2%, 38.4%, and 91.2%.The results describe the components as complementary: staged search structures improvement, the knowledge base supplies strategies, and grounding instantiates them in the current scene.Source: §B.3 · p. 16

B.4. RQ2: How Transferable Is the Hierarchical Knowledge Base?

  • On RoboTwin 2.0, success rises from 24.5% without retained knowledge to 91.5% with the hierarchical knowledge base.The entries are re-grounded in the target scene and implemented with the new robot’s control primitives.Source: §B.4 · p. 16

B.5. RQ3: Does Scaling Knowledge-Base Entries Improve Search?

  • After 20 rollouts, mean success is 22.0%, 57.2%, and 90.0% with no, half, and full knowledge; final-program success is 19.2%, 65.2%, and 91.2%.This experiment compares broader knowledge subsets rather than attributing causal effects to individual entries; with half the entries, bowl placement on the stove remains at 0%, versus 98% with the full set.Source: §B.5 · pp. 17–18

B.6. RQ4: Does RoboAuto Remain Effective with a Stronger Coder?

With a stronger coder, RoboAuto retains a substantial advantage over the base loop, while the task-level cases show that coding skill alone does not resolve physical uncertainty. The section also describes repeated-search and token-efficiency analyses, but the supplied passages do not report their outcomes.

B.6. RQ4: Does RoboAuto Remain Effective with a Stronger Coder?

  • 77.6% mean success for RoboAuto versus 29.6% for the base loop across five LIBERO-PRO settings with Opus 5.The comparison indicates that RoboAuto's advantage holds with a stronger coder.Source: §B.6 · p. 18
  • Opus 5 still failed on physically uncertain tasks, spending effort tuning execution details without establishing reliable grasps, push distances, or collision-free drawer pulls.The base loop achieved 78% on picking a bowl from a drawer but reached 12% on plate pushing and failed on the moka-pot and cabinet-drawer tasks.Source: §B.6 · pp. 18–19
  • RoboAuto improves success from 0% to 96% on moka placement, 12% to 74% on plate pushing, 0% to 36% on the cabinet drawer, and 58% to 84% on mug-and-pudding.Stage checks and targeted experiments exposed carry faults, primitive limits, collision causes, and finger-placement issues behind these failures.Source: §B.6 · p. 20

B.7. RQ5: How Sensitive Is RoboAuto to Search Randomness?

  • The repeated-search analysis runs three independent searches on three randomly selected tasks and evaluates each final program on the same 50 test scenes.This holds test scenes constant so run-to-run differences primarily reflect search variation.Source: §B.7 · p. 20
  • The ablation adds Search and Exploration, the hierarchical knowledge base, and Physical Scene Grounding cumulatively; the final column is the full system.Search and Exploration comprises verified stages and experiments.Source: Figure 9 · p. 20, §B.7 · p. 21

B.8. RQ6: How Token-Efficient Is RoboAuto?

  • The token-efficiency analysis compares success with recorded inference usage under each method's native search procedure and stopping criteria.It summarizes per-task usage with geometric means to reduce the influence of exceptionally long searches.Source: §B.8 · p. 21

Matched comparison with autonomous Code-as-Policy

On matched LIBERO-PRO Long and LIBERO-Plus settings, RoboAuto achieves higher success than ASPIRE and CaP-Agent0 with substantially lower recorded token usage than either baseline. In cross-embodiment transfer, retaining the knowledge base coincides with higher success and greater inference usage.

Matched comparison with autonomous Code-as-Policy

  • The knowledge-base-size comparison uses the last program within the rollout budget for each size; its 0% condition is the Search and Exploration run without entries.With no entries, scene grounding has nothing to measure.Source: Matched comparison with autonomous… · p. 22, Table 15 · p. 22
  • With transferred knowledge on RoboTwin 2.0, success is 91.5% using 23.6M tokens, versus 24.5% using 9.3M tokens without retained knowledge.The knowledge-base condition uses more inference while grounding knowledge and developing programs for the new simulator and embodiment.Source: Matched comparison with autonomous… · pp. 22–23
  • RoboAuto more than doubles ASPIRE's mean success on both benchmarks while using approximately 26% of its tokens on LIBERO-PRO Long and 25% on LIBERO-Plus.Across the 16 complete-record settings, mean success is 79.8% for RoboAuto, 36.0% for ASPIRE, and 17.9% for CaP-Agent0; geometric-mean total usage is 35.1M, 135.6M, and 118.8M tokens per task, respectively.Source: Matched comparison with autonomous… · pp. 21–22, Figure 11 · p. 23
  • Token aggregates include only settings with complete output and total-token records for every compared method, and use each method's native search and stopping criteria.Total counts reproduce recorded accounting without inferring an unavailable breakdown across model calls.Source: Matched comparison with autonomous… · p. 23

C. Method Details

RoboAuto combines reusable object-centric physical knowledge and verified executable skills, grounds retrieved knowledge in the current scene, and searches through verified task stages. When execution leaves physical causes unresolved, targeted experiments gather diagnostic evidence; this section also outlines behavioral analyses of the search process.

C.1. Hierarchical Knowledge Base

  • The hierarchical knowledge base maintains transferable object-centric interaction knowledge alongside executable skills, and both layers expand through online task solving.Entries encode affordances, spatial relations, or action effects, applicability conditions, candidate strategies, required scene measurements, and supporting events; evidence can support, challenge, or refine them.Source: §C.1 · pp. 24, 25, 27
  • Demonstration videos become evidence records of entities, temporally segmented interaction events, event states and visual evidence, and episode outcomes, from which object-centric knowledge is distilled.Figure 12 illustrates the process and shows scene-dependent quantities that must be measured again when applying an entry in a new scene.Source: §C.1 · p. 25, Figure 12 · p. 28

C.2. Physical Scene Grounding

  • Physical Scene Grounding identifies task-relevant objects, associates retrieved entries, checks applicability, and establishes their action-critical quantities in the current scene.Visual reasoning, segmentation, and depth-camera observations support a revisable record of established quantities, observations, and unresolved conditions.Source: §C.2 · p. 27

C.3. Search and Exploration

  • Search and Exploration decomposes tasks into dependency-ordered stages with intended object-state changes and verifiers, stopping at a failed stage so later actions do not compound its failure.Revision targets the earliest incomplete stage; an independent auditor reviews failed executions, and self-designed experiments test competing physical hypotheses when ordinary execution evidence is insufficient.Source: §C.3 · p. 27

C.3.1. Execution and Revision Protocol

  • Experiments use hypotheses, predicted observations, and controlled test actions to gather diagnostic evidence that informs program revision and may update object-centric knowledge.The diagnosis notebook persists across iterations; once the task is solved across the search pool, verified programs are retained as skills and physical evidence is consolidated into the knowledge base.Source: §C.3 · p. 27, §C.3.1 · p. 28

D. Behavioral Analysis of RoboAuto

  • The behavioral analysis combines baseline case studies with controlled cumulative ablations while holding task, rollout budget, and scene seeds fixed.Its cases examine interaction-strategy selection, grounding required quantities, and verifying intended physical effects; Figure 13 diagrams the stage execution, independent audit, revision, and experiment loop.Source: D. Behavioral Analysis of RoboAuto · pp. 28–29, Figure 13 · p. 28

D.1. Behavior Across Component Ablations

This section examines how interaction knowledge, scene-specific measurements, staged plans, and evidence-based verification support robot-program development and manipulation diagnosis. It covers component ablations and case studies, with an independent auditor judging scenes and stages from observed execution.

D.2.1. An Unsuitable Book Grasp Persists Across Revisions

  • RoboAuto reaches 74% success on held-out book scenes by using a measured cover pinch, while none of 44 CaP-Agent0 trials without service errors completes the task.The thin-slab entry prescribes a top-edge cover pinch, backed by 14 demonstrations; RoboAuto measures the book’s 2.8 cm thickness and checks it against the 5.2 cm caddy slot. Repeated top-down grasps topple the book, and parameter changes within that contact strategy do not recover the task.Source: §D.2.1 · p. 29

D.3.1. A Grasp-Height Rule Is Copied from Another Object

  • ASPIRE’s inherited mug grasp-height rule produces an air grasp in approximately half of seeds, and its repair loop ends at 0/50 success.RoboAuto instead measures the current mug’s rim radius and top height, offsets the pads using the measured radius, and checks predicted pad height before closing; ASPIRE’s reported 20% on this setting comes from subsequent evolutionary search.Source: §D.3.1 · p. 30

D.4.1. An Executed Action Is Mistaken for an Achieved Physical State

  • RoboAuto reaches 92% success on stove test scenes by verifying burner activation before moving the pan.Its knob strategy twists until the burner’s appearance changes, checks the red fraction, and re-grasps and reverses direction if the burner remains off; this localizes failure to the knob stage rather than inferring success from the executed rotation.Source: §D.4.1 · p. 30

D.1. Behavior Across Component Ablations

  • By turn 9, the full system solves 14 of 15 stove-task training scenes after 36 candidate-program rollouts.The ablation progression moves from revising whole programs based on task outcomes, to localizing failures by stage, adding interaction knowledge, and finally measuring scene-specific quantities; Figure 14 depicts the four variants.Source: §D.1 · p. 29, Figure 14 · p. 31

E.1. Coder Session Prompt

  • The coder maintains a running diagnosis, checks prior hypotheses against new evidence, and uses small experiments with explicit predictions to distinguish causes of failure.Experiments address one question, their outcomes are recorded as supporting or refuting a prediction, and experiments never change stage verdicts.Source: §E.1 · p. 35
  • The coder first relates the setup image to relevant knowledge-base entries, probes required open quantities, records measurements, and then writes the staged plan.The knowledge base covers affordance constraints, spatial relationships, and action effects and dependencies; scene-specific quantities must be measured because the base records no numbers.Source: §E.1 · p. 35

E.2. Auditor Prompt

  • The auditor independently assesses every stage in every scene from videos and logs, requiring the target object to reach the intended stage state rather than trusting printed verdicts.The auditor combines the agent-view and wrist-camera evidence, gives time-ordered factual scene notes, and treats skipped stages as failures.Source: §E.2 · p. 36

E.3. Per-Turn Feedback and Experiment Protocol

Each turn gives the coder execution results and an independent auditor’s stage judgments; it can revise the focus stage or run an experiment to test a diagnosis. Experiments return logs and video, while stage verdicts remain separate from experiment outcomes.

E.3. Per-Turn Feedback and Experiment Protocol

  • The feedback combines per-scene task and stage outcomes, auditor notes, video evidence, and a remaining rollout budget.The example reports 4 of 45 scene runs used and 41 left.Source: §E.3 · pp. 36–37
  • The diagnostic tests whether reach or collision prevented rim access; its hypothesis is that the arm cannot reach the plate rim at x≈0.71, where pad-bottom z=0.015 and rim-top z=0.007.A straddle from the -x direction at x≈0.65, 6.5 cm closer, should permit 3–5 cm lower reach and place the pads at rim level; a diagonal 135° straddle avoids the bowl.Source: §E.3 · p. 37
  • In the plate-grasp example, the gripper pads were above the rim and the plate slipped during the test lift, indicating marginal contact.The log reports the pads 10.5 mm above the rim and grip falling from 0.0474 after closing to 0.0146 after the lift.Source: §E.3 · p. 37
  • Experiments document a hypothesis, prediction, observations, and whether the prediction held; their results do not change stage verdicts.The coder then revises the focus stage or proposes another experiment.Source: §E.3 · p. 37

F. Real-Robot Experimental Details

Real-robot experiments reuse RoboAuto’s LIBERO-acquired knowledge base on held-out tasks, with search and evaluation conducted on a WidowX AI arm. Performance is measured by stage completion across three long-horizon tasks.

F. Real-Robot Experimental Details

  • The real-robot setup uses a WidowX AI arm with RealSense D405 RGB-D and Logitech C920 RGB cameras, shared across methods.The cameras provide complementary geometric and visual observations.Source: F. Real-Robot Experimental Details · p. 38
  • RoboAuto reuses its LIBERO-acquired hierarchical knowledge base on held-out real-robot tasks and has a substantial advantage over ASPIRE despite the simulation-to-hardware shift.ASPIRE also transfers its LIBERO-acquired skill library for comparison.Source: F. Real-Robot Experimental Details · p. 38
  • Search compares RoboAuto and ASPIRE with 10 rollouts each; evaluation tests RoboAuto, π0.5, and ASPIRE on three WidowX tasks over 20 trials per task.Human judgments replace simulator rewards during search, and evaluation averages each trial’s fraction of completed stages.Source: F. Real-Robot Experimental Details · p. 38
  • Sort Balls and Stack Dishes each score four stages equally, while Erase Board scores three stages and awards proportional credit for erased letters.Erasing n of eight letters earns n/8 of the erasing-stage score.Source: F. Real-Robot Experimental Details · p. 38
  • Figure 18 presents three WidowX task rollouts as time-ordered scene-camera keyframes plus a wrist-camera frame, marking active actions and verified stages.Checked stages are verified before the next stage begins.Source: Figure 18 · p. 39

Related papers

Loading 2610.36533v1…