PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenRUA: Robot-Use Agents Are Zero-Shot Visuomotor Policies

Zhaoyang Chu, Earl T. Barr, Claire Le Goues, Peter O'Hearn, Mark Harman, Federica Sarro, He Ye

arXiv:2610.02459v1cs.RO

TL;DR

Robot control requires substantial engineering, while learned visuomotor policies need large-scale demonstrations and have limited generalization beyond training distributions. OpenRUA gives off-the-shelf coding agents terminal access to ROS 2 in a minimalist workspace; it achieves 99.0% on CaP-Bench and 87.0% on LIBERO-PRO.

  • Problem

    Robot control programming requires substantial human effort, while learned visuomotor policies need large-scale demonstrations and have limited generalization beyond training distributions.

    Source: §1 · p. 1

  • Method

    OpenRUA gives off-the-shelf coding agents terminal access to ROS 2 in a minimalist workspace, leaving them to organize their work and code perception and control.

    Source: §1 · pp. 2–3

  • Results

    99.0% on CaP-Bench and 87.0% on LIBERO-PRO; OpenRUA exceeds leading systems by 18.0 and 4.6 percentage points, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Image-processing, metric-measurement, motion-control, and feedback tools emerge as agents code during execution, without task-specific primitive libraries.

    Source: §6 · p. 12

    Evaluation is simulation-only, and task success does not establish safe execution because agent-written safeguards are not guaranteed present or correct.

    Source: §5 · pp. 11–12

Abstract

from arXiv · show

Coding agents are extending their reach into the physical world by writing and executing robot control programs. One might expect the agents to use the existing mature software stack that engineers have developed over decades to access sensors and control motion. Yet prior work primarily engineers complex custom harnesses to orchestrate agents for robot use, particularly by prescribing specialized workflows and providing bespoke interfaces. This raises the question: "Is such additional harness engineering necessary?" We introduce OpenRUA, a zero-abstraction harness that bypasses bespoke abstraction layers by providing off-the-shelf coding agents with only terminal access to the robot's native software interface ROS 2. OpenRUA employs a minimalist workspace-as-harness design, only offering ROS 2 documentation and basic tools while leaving the coding agent to organize its own work without orchestrating any agentic workflow. Within this workspace, OpenRUA recasts perception as file I/O and manipulation as coding. With Claude Code powered by Claude Opus 5, OpenRUA achieves success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO, demonstrating that an off-the-shelf coding agent can serve as a zero-shot visuomotor policy through the robot's native interface, without bespoke primitives or task-specific training. Under this minimalist design, further analysis reveals striking emergent behaviors of coding agents: (1) For perception, the agent spontaneously writes programs that process raw sensory inputs and derive metric measurements in 96.80% of episodes. (2) For manipulation, the agent spontaneously builds motion-control clients (e.g., gripper control) in 95.87% of episodes and closed-loop control programs (e.g., adjusting motion based on sensor feedback) in 50.13% of episodes. Our code is available at https://github.com/terminalworld/OpenRUA.

1 INTRODUCTION

Traditional robot programming demands substantial engineering, learned visuomotor policies require demonstrations and generalize poorly beyond training distributions, and the need for custom robot-agent harnesses remains unclear. OpenRUA tests a minimalist ROS 2 workspace design; experiments report strong benchmark results and emergent perception and control behaviors.

1 INTRODUCTION

  • Existing robot-use systems engineer custom interfaces, primitive libraries, or workflows, leaving it unclear whether such harness engineering is necessary for effective robot control.OpenRUA asks whether an off-the-shelf coding agent can act as a zero-shot visuomotor policy through the robot’s native interface without bespoke primitives or learned policies.Source: §1 · p. 2
  • OpenRUA gives off-the-shelf coding agents terminal access to ROS 2 and a workspace, where perception becomes file I/O and manipulation becomes coding without a prescribed workflow.The agent interprets images, computes metric measurements, writes control programs, and refines actions using execution feedback.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • 99.0% on CaP-Bench and 87.0% on LIBERO-PRO; OpenRUA exceeds ASPIRE and Harness VLA by 18.0 and 4.6 percentage points, respectively.With Claude Code powered by Claude Opus 5, it also achieves 28.1% on unseen composite RoboCasa365 tasks, 13.1 percentage points above Harness VLA.Source: §1 · p. 3
  • Perception programs deriving metric measurements appear in 96.80% of episodes; motion-control clients in 95.87%, and closed-loop control programs in 50.13%.Perception programs process raw sensory inputs; control programs can adjust motion or stop based on sensor feedback.Source: §1 · p. 3
  • On RoboCasa365, OpenRUA achieves 26.3% on seen and 28.1% on unseen composite tasks, while RLDX-1 and Harness VLA achieve 5.0% and 13.8% on unseen tasks.The passage describes OpenRUA’s seen and unseen results as comparable and identifies Harness VLA as orchestrating RLDX-1.Source: §1 · p. 3
  • More test-time interaction or reasoning effort improves task success without robot training or bespoke primitives, though performance eventually saturates at levels that vary across models.The findings describe improvements from increasing test-time computation, not a single saturation level shared by all models.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORK

Prior work includes learned visuomotor policies and robot-use agents, alongside CLI coding agents and the ROS 2 interfaces that expose robot sensing and control.

Visuomotor Policies

  • Learned visuomotor policies map raw sensory observations to low-level actions, but their generalization remains bounded by training distributions; OpenRUA instead uses an off-the-shelf coding agent without robot demonstrations or fine-tuning.Examples of learned policies include vision-language-action models and world action models.Source: §2 · p. 3

Robot-Use Agents

  • Robot-use agents operate robots like computers through tool calls or executable programs; prior approaches wrap learned policies or expose low-level actions through predefined, hand-crafted primitives.OpenRUA instead supplies terminal access to native ROS 2 interfaces and a workspace for building task-specific tools during execution.Source: §2 · p. 3

CLI Coding Agents

  • CLI coding agents navigate codebases, run commands and programs, interpret terminal output, and recover from errors through autonomous iteration.OpenRUA directs these capabilities toward robot control by treating the robot as a live software project.Source: §2 · p. 4

Robot Operating System

  • ROS 2 exposes robot sensing, state, and control through topics, services, and actions, accessible with ros2cli and the rclpy client library.Prior LLM-to-ROS work added custom tools or predefined primitives and evaluated on in-house tasks rather than standard robot-control benchmarks.Source: §2 · p. 4

3 OPENRUA: WORKSPACE AS HARNESS FOR ROBOT-USE AGENTS

OpenRUA makes a workspace, rather than a prescribed agent workflow or bespoke interface layer, the harness for robot use. The agent uses native ROS 2 access to gather observations, write control code, and organize execution artifacts.

Workspace as Harness

  • The workspace provides ROS 2 documentation, basic tools, and an opening task prompt, leaving the agent to explore the machine, organize its work, and verify completion.The prompt asks the agent to survey the machine and finish the task physically, but does not prescribe perception or manipulation steps.Source: §3 · p. 4, Figure 2 · p. 4
  • Starter tools are source files the agent can inspect, adapt, or use as examples for native ROS 2 programs.The initial workspace includes a README entry point directing the agent to robot configuration and interface documentation.Source: §3 · pp. 4–5
  • Workspace files persist during an episode, letting the agent revisit observations, programs, notes, and logs as external memory.The agent can recover findings and reuse or revise earlier programs as the task progresses.Source: §3 · p. 5

Perception as File I/O

  • Perception becomes file I/O: the agent acquires sensor data through ROS 2 topics, saves it to workspace files, and chooses when and how to inspect or process it.Files can include visual data, proprioceptive and kinematic states, and force or effort feedback.Source: §3 · p. 5

Manipulation as CLI Coding

  • Manipulation becomes CLI coding: ros2cli commands or rclpy programs send trajectory, velocity, or gripper requests through native ROS 2 interfaces.The agent can inspect feedback between stepwise commands or run closed-loop programs using sensor feedback, then revise its actions.Source: §3 · p. 5

4 EXPERIMENTS

Across three simulation benchmarks, OpenRUA evaluates coding agents through zero-shot access to native robot interfaces and reports strong benchmark results. Analyses describe agent-written perception and control programs, generalization, test-time scaling, and remaining failure modes.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers CaP-Bench, LIBERO-PRO, and RoboCasa365; episodes use fresh workspaces without demonstrations, task rehearsal, prior memory, privileged state, or success-predicate access.CaP-Bench uses 100 trials per task, LIBERO-PRO ten per task, and RoboCasa365 ten trials for each of 50 target tasks.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • 99.0% on CaP-Bench and 87.0% on LIBERO-PRO; OpenRUA exceeds ASPIRE’s 81.0% and Harness VLA’s 82.4% without training, prior task experience, or predicate access.The results support coding agents as zero-shot visuomotor policies without a learned robot policy or task-specific primitive library.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 HOW THE AGENT PERCEIVES

  • 72.67% of episodes included self-written image processing, and 96.80% included self-written metric programs for deriving task-specific measurements.The agent crops or enlarges images and overlays grids, while depth, camera intrinsics, and frame transforms support measurements such as world-coordinate targets.Source: §4.3 · p. 7

4.4 HOW THE AGENT MANIPULATES

  • 95.87% of episodes included agent-written motion clients, while 50.13% executed self-written feedback-control programs.Programs used sensor readings to adjust motion or stop; the agent also revised a failed grasp using new observations and reconstructed geometry.Source: §4.4 · pp. 8–9, Table 3 · p. 8

4.5 GENERALIZATION UNDER DISTRIBUTION SHIFT

  • 28.1% unseen versus 26.3% seen composite-task success; RLDX-1 falls from 21.3% to 5.0%, while Harness VLA drops from 61.0% to 13.8%.The RoboCasa365 comparison concerns composite tasks whose templates are held out from or included in benchmark pretraining data.Source: §4.5 · pp. 9–10, Table 4 · p. 10

4.6 SCALING WITH CODING AGENTS AND MODELS

  • 50%→70% as Opus 5’s turn budget rises from 100 to 200; GPT-6 Astra reaches 69.0% at high versus 62.5% at medium reasoning effort.Gains flatten at higher budgets, and model results differ: Sonnet 5 reaches 24.0%, versus 72.5% for Opus 5 and 75.0% for Fable 5.1.Source: Figure 8 · p. 10, §4.6 · p. 10

4.7 FAILURE STUDY

  • 58% of 413 failures involved task criteria or undocumented behavior; physical constraints accounted for 18%, and control, perception, or waiting errors for 24%.Failures included missed benchmark thresholds, limited reach or grip margins, incorrect control choices, missed targets, and excessive polling.Source: Table 5 · p. 11, §4.7 · p. 11

5 LIMITATIONS

The evaluation is limited to simulation, leaving real-world timing, sensing robustness, and safe deployment unestablished.

5 LIMITATIONS

  • The evaluation is limited to simulation, so physical-robot experiments are needed to assess OpenRUA’s practical viability.A shared ROS 2 interface alone does not establish generalization to real-world deployment.Source: §5 · p. 11
  • Simulation pauses during agent deliberation, so real-time responsiveness—especially for dynamic tasks—remains unevaluated.Feedback controllers can react within a running program but do not eliminate delays between agent-level decisions.Source: §5 · p. 11
  • Simulated sensor observations omit real-world variability, including sensor noise, missing depth values, and calibration errors.The demonstrated perception capabilities therefore do not establish robustness to real-world sensor data.Source: §5 · p. 12
  • Task success does not establish safe execution; agent-written thresholds and stopping conditions are not guaranteed to be present or correct.Physical deployment would require independently enforced motion limits, collision checks, and emergency stops; validating these controls remains future work.Source: §5 · p. 12

6 CONCLUSION

OpenRUA uses coding agents as zero-shot visuomotor policies through a minimalist workspace and native ROS 2 access. The conclusion and supporting sections describe how this setup relates to prior approaches and how agents acquire observations and issue control requests.

6 CONCLUSION

  • OpenRUA shows coding agents can serve as zero-shot visuomotor policies without learned robot policies or task-specific primitive libraries.Image-processing tools, metric-measurement programs, motion-control clients, and feedback controllers emerge as agents write code during execution.Source: §6 · p. 12

A EXTENDED RELATED WORK

  • Prior approaches use learned policies or predefined primitives, while OpenRUA lets agents construct perception and control programs through native ROS 2 interfaces.VLA and WAM approaches require large-scale demonstrations and have bounded generalization [Brohan et al., 2023; Kim et al., 2024; Team, 2025; Bjorck et al., 2025; Intelligence et al., 2026; Agarwal et al., 2026; Ye et al., 2026; Li et al., 2026; Zhou et al., 2025; Fei et al., 2025; Kim et al., 2026]; other RUA systems may use privileged state or custom primitives [Tsui et al., 2026; Xie et al., 2026; Fu et al., 2026; Seker et al., 2026; Zhang et al., 2026a].Source: A EXTENDED RELATED WORK · p. 18

B.1 NATIVE ROS INTERFACES

  • ROS 2 exposes robot sensing, state, and control through topics, services, and actions, accessible from the terminal with ros2cli or programmatically with rclpy.Actions can provide feedback and cancellation, while rclpy programs can keep connections active and process messages in feedback loops; OpenRUA adds no agent-specific API.Source: §B.1 · p. 19

B.2 WORKSPACE

  • The workspace supplies machine documentation, basic operation tools, and starter examples, but no task solutions or prescribed workflow.The prompt asks the agent to survey the machine, perform the task, and verify the result; workspace files persist within an episode, including resumed sessions, but do not transfer between episodes.Source: §B.2 · p. 19

B.3 OBSERVATION ACQUISITION AND FILE REPRESENTATIONS

  • Agents select or combine observation methods according to need, using available sensor modalities and preserving metric depth values for calculations.Geometric calculations require calibration and coordinate-frame relations; aligned moving-scene observations may require an agent-written acquisition program that accounts for timestamps and pixel correspondence.Source: §B.3 · p. 20

B.4 NATIVE CONTROL REQUESTS

  • Native ROS 2 control requests include joint-trajectory goals, gripper actions, velocity commands, and inverse-kinematics service requests.Agents can issue these through ros2cli or rclpy, and the corresponding robot controller executes the requested motion.Source: §B.4 · p. 20

C.1 BENCHMARKS AND TRIAL SAMPLING

The evaluation uses simulation benchmarks with specified trial sets, fresh agent sessions, ROS 2 interfaces, and independent success scoring. Baseline comparisons draw on published configurations whose evaluation protocols and preparation can differ.

C.1 BENCHMARKS AND TRIAL SAMPLING

  • The evaluation covers 2,000 simulation episodes across CaP-Bench, LIBERO-PRO, and RoboCasa365, with benchmark-specific task and trial sampling.CaP-Bench uses 700 episodes, LIBERO-PRO 800, and RoboCasa365 500.Source: §C.1 · p. 21

C.2 AGENT CONFIGURATION AND EXECUTION PROTOCOL

  • Each episode starts from a fresh agent session without carried-over files or task findings; ROS 2 exposes sensor, state, trajectory, gripper, and available velocity interfaces.The simulation pauses between control-triggered steps, including during agent reasoning; arm trajectories use joint-position control with 640 × 480 image observations.Source: §C.2 · p. 21

C.3 SCORING AND AGGREGATION

  • Success is checked against the benchmark predicate at every simulation step and retained if achieved before termination, while success rates use benchmark-specific aggregation.LIBERO-PRO reports both six-cell and eight-cell averages; RoboCasa365 weights its overall rate by task counts.Source: §C.3 · p. 22

C.4.1 SOURCES AND COMPARISON SCOPE

  • Baseline results are reproduced from published sources rather than rerun, so comparisons involve configurations that may differ in models, budgets, tools, experience, and evaluation protocols.The CaP-Bench human reference is based on CaP-X expert-written programs; baseline sources and preparation protocols vary across systems.Source: §C.4.1 · pp. 22–23

C.4.2 TRAINING, EXPERIENCE, AND ORACLE ACCESS

The table annotations distinguish training, prior task or environment experience, and oracle access. OpenRUA's task-specific work occurs within an episode, while baseline trial counts and evaluation rules vary by source.

C.4.2 TRAINING, EXPERIENCE, AND ORACLE ACCESS

  • Training, experience, and oracle annotations separately describe demonstration training, prior task or environment preparation, and access to the benchmark success predicate.The annotations can overlap and characterize reported configurations.Source: §C.4.2 · p. 23
  • OpenRUA's task-specific programs and notes are produced within the current episode and do not count as prior experience under the paper's definition.Source: §C.4.2 · p. 23
  • Baseline trial counts, initial states, stopping rules, and models differ across sources, so the paper retains source-specific settings rather than treating every comparison as protocol-matched.RHO uses 50 initial states on six LIBERO-PRO cells while this evaluation uses ten; Pigey also applies its own completion protocol.Source: §C.4.2 · p. 23

D.1 DETAILED ANALYSIS OF MAIN RESULTS

OpenRUA serves as a zero-shot visuomotor policy through native ROS 2 interfaces, with strong results on CaP-Bench and perturbed LIBERO-PRO conditions. Comparisons span systems using learned policies or prior program development, but their reported configurations differ.

D.1 DETAILED ANALYSIS OF MAIN RESULTS

  • 99.0% on CaP-Bench and 91.8% on the six LIBERO-PRO conditions are achieved without a learned robot policy, task-specific primitive library, or prior task experience.The expert-written programs score 82.6% on CaP-Bench, while OpenVLA, π0, and π0.5 remain at or below 12.8% on the six LIBERO-PRO conditions after scene perturbations.Source: §D.1 · p. 23
  • 18.2% for CaP-X and 43.8–71.7% for RHO, ASPIRE, and RATS on the six LIBERO-PRO conditions are reported after using provided primitives and accumulated repositories or skills.OpenRUA constructs its programs within the current episode through native ROS 2 interfaces.Source: §D.1 · p. 23
  • 91.8% for OpenRUA is reported against 87.7% for Harness VLA and 53.3% for Pigey, which combine coding agents with learned policies.Harness VLA uses task-specific memory from reference-seed exploration and access to the success predicate; the compared systems also differ in models and evaluation settings.Source: §D.1 · p. 24

D.2 NATIVE INTERFACES AND THE LIMITS OF ACCUMULATED SKILLS

Accumulated policy experience does not resolve every difficult task, and the capabilities available through an interface constrain what that experience can encode. OpenRUA's native sensor and control interfaces support agent-written feedback programs during manipulation.

D.2 NATIVE INTERFACES AND THE LIMITS OF ACCUMULATED SKILLS

  • 100% for OpenRUA versus 0%, 1%, and 9% for CaP-Agent0, RHO, and ASPIRE on Nut shows that accumulated experience leaves some difficult tasks unresolved.On LIBERO-PRO Spatial position/task conditions, OpenRUA reports 98%/100%, versus 12%/14% for CaP-X, 53.0%/61.6% for RHO, and 51%/60% for ASPIRE.Source: §D.2 · p. 24
  • A primitive interface limits what accumulated experience can encode; in the evaluated CaP-X simulation interface, no force-reading or force-conditioned motion primitive is listed.The text distinguishes this simulation restriction from CaP-X's separate real-robot force-wrench access and notes reported insertion difficulties requiring continuous feedback.Source: §D.2 · p. 24
  • OpenRUA's ROS 2 sensor and control interfaces let the agent write feedback programs; the Nut insertion example uses seven trajectory segments with a force reading after each.The paper presents this as a mechanism for using sensing during manipulation without a pre-specified task primitive or learned robot policy.Source: §D.2 · p. 24

D.3 GENERALIZATION AND DEMONSTRATION COVERAGE

OpenRUA’s composite-task results show similar success on Seen and Unseen templates, while composite tasks remain harder than atomic tasks. The appendix also contrasts OpenRUA’s no-rehearsal results with learned-policy results that use target-task demonstrations.

D.3 GENERALIZATION AND DEMONSTRATION COVERAGE

  • GR00T N1.5 rises from 0.2% to 42.1% on the same 16 Composite-Unseen tasks after post-training on their target-task demonstrations.“Unseen” templates were absent from benchmark pretraining but included in the post-training target data; OpenRUA’s 28.1% uses no such training.Source: §D.3 · p. 24
  • OpenRUA reaches 26.3% on Composite-Seen and 28.1% on Composite-Unseen tasks without a learned robot policy or task-specific rehearsal.Harness VLA reaches 61.0% and 13.8% with Codex, and 47.5% and 15.0% with Claude Code, respectively; both retain a Seen–Unseen gap.Source: §D.3 · p. 25
  • OpenRUA’s success falls from 61.1% on Atomic-Seen tasks to 26.3% and 28.1% on the two composite groups, and all compared systems do worse on composite tasks.The Seen–Unseen comparison concerns template coverage within composite tasks, while the atomic-to-composite comparison reflects the added demands of composing and executing multiple steps.Source: §D.3 · p. 25

E AGENT PERCEPTION AND CONTROL

  • The appendix expands perception and control analyses with statistics, measurement definitions, and examples spanning all three benchmarks.Examples include saved or processed sensor images, control loops plotted from program logs, and the agent’s terminal output and programs.Source: E AGENT PERCEPTION AND CONTROL · p. 25

E.1 PERCEPTION STATISTICS

Across CaP-Bench and LIBERO-PRO, agent-written programs commonly process sensory inputs and derive metric measurements; computed coordinates also support motion. The appendix details how the agent prepares views for inspection and checks actions using visual or computed representations.

E.1 PERCEPTION STATISTICS

  • Metric-program code appears in 1,452 of 1,500 episodes, image-processing code in 1,090, and both behaviors in 1,050 (70.00%).Counts use source-pattern and execution-trace matching, which can miss indirect implementations and cannot establish that every computed measurement is correct.Source: §E.1 · p. 25
  • 90–94% of Cartesian-target motion commands use coordinates traced to an earlier agent-written metric program or computed within the motion program.The programs use depth, camera intrinsics, or frame transforms; observation and parking poses account for remaining targets, connecting measurement programs to physical actions.Source: §E.1 · pp. 25–26, Table 6 · p. 26
  • 16–40% of image reads are processed views; among those, 68–94% involve crops or enlargements and 3–11% involve grids or markers.Processing categories can overlap, and reads are classified by preceding events such as scene understanding, checking an action, or inspecting an intermediate result.Source: §E.1 · p. 26, Table 7 · p. 26
  • Agent-written perception code includes geometric and image-processing techniques, including switch-state frame differencing and depth-based point-cloud reconstruction.Table 8 reports episode shares for techniques, which can co-occur; code presence alone does not establish successful execution.Source: §E.1 · p. 26, Table 8 · p. 27, Figure 9 · p. 27

E.2 VISUAL INSPECTION AND PROGRAMMATIC IMAGE PROCESSING

  • The agent crops, enlarges, or annotates images for inspection, checks action outcomes with before-and-after views, and renders computed representations such as point clouds.The harness does not prescribe a fixed image-processing sequence; processed views can expose visual details while geometric relations are needed for physical dimensions.Source: §E.2 · pp. 26–27

E.3 METRIC MEASUREMENTS FROM SENSOR DATA

OpenRUA combines native sensor readings with quantities computed from saved observations, without prescribing a task-specific measurement pipeline. The appendix describes depth-to-coordinate conversion, episode-specific measurement checks, and two further perception examples.

E.3 METRIC MEASUREMENTS FROM SENSOR DATA

  • The agent can combine native joint, effort, force, and frame-transform readings with quantities computed from saved sensor observations.Available observations depend on the robot’s sensors and interfaces; locating an object without a tracked frame requires estimation from observations.Source: §E.3 · p. 28
  • Depth pixels map to camera-frame points through the pinhole intrinsic matrix, then into a chosen reference frame using rotation and translation.The calculation assumes calibrated pixels, documented depth scale and convention, valid positive depth, matching color/depth pixels, and calibration and transforms aligned to the observations.Source: §E.3 · p. 28
  • In the Nut example, an overhead-camera height of 2.0 m yields world heights as 2.0 − z, and episode-specific thresholds select components for coordinate estimates.The agent cross-checks measurements from an oblique camera and compares a 30.35 mm finger gap with an estimated 30.4 mm handle width as grasp evidence, not a universal success rule.Source: §E.3 · p. 28

E.4 ADDITIONAL PERCEPTION EXAMPLES

  • For CaP-Bench Wipe, the agent intersects three camera-derived, 5 mm world grids to select a wipe region and repeats the measurement after the stroke.In the kettle example, it grids knob crops, projects depth-derived burner centers into the wrist image, and combines color thresholds with world-coordinate bounds to detect flame.Source: §E.4 · pp. 28–29, Figure 10 · p. 29

E.5 CONTROL STATISTICS

Agent-written motion clients are prevalent, and many episodes also contain programs that use sensor readings to alter motion. Examples show the agent combining discrete actions, program-level feedback, and recovery or corrective motion across varied tasks.

E.5 CONTROL STATISTICS

  • Agent-written motion clients appear in 95.87% of episodes, while feedback programs appear in 50.13%.Counts identify agent-written client source and sensor-dependent control logic; they do not establish successful client execution, and indirect implementations may be missed.Source: §E.5 · p. 29
  • On contact-rich CaP-Bench tasks, 99% of motion commands use agent-written programs, 67% of episodes contain sensor-dependent control, and 86% read force.Across six LIBERO-PRO perturbation cells, 68% of commands use agent-written programs and feedback programs appear in 37% of episodes.Source: §E.5 · p. 29, Table 9 · p. 30
  • Starter-tool use varies by operation: CaP-Bench executes its inverse-kinematics tool in 3% of episodes, while LIBERO-PRO executes its trajectory tool in 63%.The corresponding source-only read rates are 41% and 13%, showing starter scripts can serve as references as well as executable tools.Source: §E.5 · p. 30, Table 10 · p. 30

E.6 COMMAND EXECUTION AND FEEDBACK PROGRAMS

  • The agent can inspect an action’s outcome and choose another, or let a running program adjust motion from sensor feedback; one task can combine both.A drawer episode includes a failed grasp, a revised approach, and a reusable client for IK, trajectories, gripper actions, joint states, and velocity commands.Source: §E.6 · pp. 30–31
  • A drawer approach applies proportional velocity corrections until within 2.5 mm of its target; pulling stops on gap, displacement, target, or iteration checks.The 20-message velocity bursts are followed by state checks; the nominal 20 Hz parameter does not mean the full feedback loop runs at 20 Hz.Source: §E.6 · p. 31, Figure 12 · p. 31
  • A Restack descent reads force after each 2 mm segment and stops when force change exceeds 1.5 N or the lower position limit is reached.The running program processes feedback between trajectory segments; its threshold and step size are program-specific, not a harness-wide policy.Source: §E.6 · pp. 31–32

E.7 ADDITIONAL CONTROL EXAMPLES

  • Examples include force-guarded descent, aborting and correcting a slipping two-arm lift, leveling a drooping handle, and treating an 11 mm jar error as contact.The examples combine checks inside running programs with corrections selected by the agent between executions.Source: §E.7 · p. 32, Figure 13 · p. 32
  • For mobile hot-dog transfers, the agent accounts for base pose in inverse kinematics and checks lifts; for Two-Arm Lift, it coordinates staged motions and retries after a slip.The two-arm program sends both trajectory goals before waiting and verifies the lift stages with gripper-gap checks.Source: §E.7 · p. 32, Figure 14 · p. 33, Figure 16 · p. 33

F SCALING ANALYSIS

Success, interaction, and cost profiles vary across the evaluated agent–model configurations; higher success can coincide with lower cost per success. The failure study classifies 413 Opus 5 failures by decisive-error stage, with scope and annotation limits.

F SCALING ANALYSIS

  • Raising GPT-6 Astra reasoning effort from medium to high increased success from 62.5% to 69.0% and reduced cost per success from $19.1 to $14.4.Median output tokens per turn rose from 90 to 122, while mean episode cost fell from $11.8 to $10.0.Source: F SCALING ANALYSIS · pp. 35–36
  • Within 100 turns, Fable 5.1 solved 61% of episodes, Opus 5 solved 50%, and Sonnet 5 solved 1%; Opus and Fable flattened by about 300 turns.Sonnet continued gaining successes to about 400 turns and finished at 24.0%; the curves describe recorded runs retrospectively, not separate budget-limited runs.Source: F SCALING ANALYSIS · p. 35
  • At API list prices, cost per success was $93.3 for Sonnet 5, $18.9 for Opus 5, and $11.2 for Fable 5.1, with success rates of 24.0%, 72.5%, and 75.0%.Runs were billed on subscriptions; list prices are used for comparability.Source: F SCALING ANALYSIS · pp. 35–36
  • Claude Code and Codex used the same workspace and native ROS 2 interface, yet their output-token and failed-episode turn profiles differed across the evaluated agent–model combinations.Codex with GPT-6 Astra produced median output tokens of 90 or 122 per turn, versus 760 for Fable 5.1 and 1,058 for Opus 5; the comparison does not isolate agent from model.Source: F SCALING ANALYSIS · p. 36

G FAILURE STUDY

  • The failure study analyzed 413 failed Opus 5 episodes: 104 from LIBERO-PRO, 7 from CaP-Bench, and 302 from RoboCasa365.Researchers compared the agent-measured end state with the benchmark predicate, reconstructing RoboCasa365 scenes from episode seeds when needed.Source: §G.1 · p. 37, G FAILURE STUDY · p. 37

G.1 CLASSIFICATION PROCEDURE AND SCOPE

  • Failures were assigned to the stage of the decisive error: undocumented or contradictory predicates, budget exhaustion, physical limits, or agent errors in perception, control, or waiting.The procedure also recorded whether the agent’s last message announced completion.Source: §G.1 · p. 37
  • The failure analysis covers only one model, and five LIBERO-PRO episodes remained unresolved between class A and a drawer-opening residual below point-cloud resolution.Classes were assigned by reading and recorded with confidence levels.Source: §G.1 · p. 37

G.2 DETAILED FINDINGS

This section details how agents use the workspace’s native ROS 2 interfaces and guidance to program perception and manipulation, with examples spanning measurement, feedback-driven motion, and coordinated control. It also documents failure causes, a false-completion case, and the workspace materials provided to agents.

G.2 DETAILED FINDINGS

  • Classes A–C account for 238 of 413 analyzed failures (58%), including unstated or conflicting completion criteria and budgets spent handling undocumented machine behavior.Examples include a drawer threshold above 140 mm, a predicate checking the wrong pot, and a self-closing freezer door; replayed demonstrations show conditions a zero-shot agent must otherwise infer.Source: §G.2 · pp. 37–38
  • Class D accounts for 75 failures (18%) involving physical constraints, while class E accounts for 100 failures (24%): 46 control, 27 perception, and 27 waiting errors.Examples include objects incompatible with gripper dimensions, force-displaced bases, missed targets or joints, and budget spent polling background jobs; writing measurement and feedback programs does not ensure correct use or verification.Source: §G.2 · p. 38
  • By benchmark, classes A and B account for 59 of 104 LIBERO-PRO failures, six of seven analyzed CaP-Bench failures occur in Handover, and classes C and D account for 188 of 302 RoboCasa365 failures.All 59 LIBERO-PRO episodes ended with the agent’s own completion message; RoboCasa365 failures include unbraked-base displacement and objects dropped below the arm’s reach.Source: §G.2 · p. 38

G.3 FAILURE CASES

  • In a LIBERO-10 episode, the agent placed a tomato can instead of the requested alphabet-soup can and reported completion after verifying placement without checking object identity.The requested blue can remained on the table.Source: §G.3 · p. 38, Figure 24 · p. 41

H.1 INITIAL MATERIALS AND FILE INVENTORY

  • The initial workspace template comprises six Markdown documents and six Python starter tools, with a machine-specific machine.yaml generated from the resolved robot configuration at startup.The starter utilities cover camera capture, depth-pixel-to-world conversion, trajectory and gripper actions, inverse kinematics, and base motion.Source: §H.1 · p. 42

H.2 WORKSPACE DOCUMENTATION

  • The workspace documentation explains how to inspect the machine manifest and use ROS 2 CLI or rclpy for sensing and control, and advises verifying motion by rereading sensors rather than waiting.It covers machine interfaces, perception, action, and generic working patterns; the paused simulation clock means waiting alone does not advance the physical world.Source: §H.2 · pp. 43–44
  • Convert image pixels and depth to camera-frame 3D points using camera intrinsics, then transform them into world coordinates.The documented pick pattern snapshots cameras and locates the target before moving above it.Source: §H.2 · pp. 45, 47
  • The documented arm controls include joint-trajectory actions for primary motion, Cartesian-velocity servoing for small precise moves, and gripper actions for opening and closing.Servo motion should be streamed in bursts with sensor checks between them, rather than as one long blind stream.Source: §H.2 · pp. 45–46
  • Use command–sense–compare–correct cycles, combining large trajectory moves with short servo bursts for a coarse-to-fine approach.Re-read relevant sensors rather than waiting: compare joint state with motion targets, finger gap and camera frames for manipulation, and before/after frames for world changes.Source: §H.2 · p. 47
  • For picking, approach above the target, descend in short sensed servo bursts, close the gripper, then verify the grasp and lift visually.Grasp evidence includes finger positions stopping above the gripper's closed width or a wrench change on contact; confirm with a fresh camera frame after lifting.Source: §H.2 · pp. 46–47
  • The workspace documentation describes native-interface tools for converting selected depth pixels to world coordinates and sending joint trajectories, gripper commands, IK moves, and base motions.The documented base motion uses odometry feedback.Source: §H.2 · p. 48

H.3 STARTER TOOL SOURCE CODE

  • The starter tools are ordinary, inspectable or modifiable Python source files, and the agent can write additional programs using the same native interfaces.The examples include camera capture, pixel-to-world conversion, trajectory and gripper commands, IK moves, and closed-loop base motion.Source: §H.3 · pp. 48, 49, 50, 51, 53

I AGENT-WRITTEN PROGRAMS

  • The programs shown are verbatim examples from mainline Claude Code with Opus 5 episodes and illustrate the code behind Tables 2 and 3.Imports and repeated blocks are elided in the listings.Source: I AGENT-WRITTEN PROGRAMS · p. 54

I.1 CAP-BENCH NUT

  • For the CaP-Bench Nut task, the agent segmented overhead depth data and cross-checked measurements from an oblique camera before inserting in monitored trajectory segments.It also cropped and enlarged an image to inspect the nut’s orientation and handle width; during insertion it read wrench and finger-gap feedback and checked a path-height floor.Source: §I.1 · pp. 54–55

I.2 LIBERO-PRO DRAWER

  • For the LIBERO-PRO drawer task, the agent built a reusable client for native robot interfaces and revised an unsuccessful grasp into feedback-based servoing.It used finger gap as grasp evidence and guarded the drawer pull by checking grip retention, progress, and travel limits.Source: §I.2 · p. 56

I.3 CAP-BENCH WIPE

  • For CaP-Bench Wipe, the agent ray-cast colour-mask blobs onto the table plane, counted a blob as spill only when at least 90% landed on the table, and intersected 5 mm grids across three cameras.The implementation filtered small connected components and rasterized qualifying hits into the grid.Source: §I.3 · pp. 56–57

I.4 CAP-BENCH RESTACK

  • For CaP-Bench Restack, the agent descended in 2 mm increments and stopped when the wrist-wrench change exceeded a 1.5 threshold, using force feedback to detect contact.The program returned the contact height and whether contact was detected.Source: §I.4 · p. 57

I.5 CAP-BENCH TWO-ARM LIFT

  • For the CaP-Bench two-arm lift, the agent sent both arms’ trajectory goals before waiting for either result, enabling concurrent execution.It then collected each arm’s error code and refreshed state.Source: §I.5 · p. 57

I.6 CAP-BENCH HANDOVER

In the CaP-Bench handover example, the agent levels a drooping hammer handle by measuring its pitch and counter-rotating the wrist. It resolves a pose that places the handle center at the target height.

I.6 CAP-BENCH HANDOVER

  • The agent resolves the pose so the handle center lands at the target height.This follows the handover, after the hammer handle droops.Source: §I.6 · p. 57
  • The agent measures the handle’s pitch from its top face and counter-rotates the wrist.The code applies a rotation of -pitch around the x-axis.Source: §I.6 · p. 57
  • The code computes a new wrist pose, solves for joint positions, and sends the resulting joint target.It uses the handle midpoint in the hand frame and clips the commanded duration based on the maximum joint change.Source: §I.6 · p. 57
Loading 2610.02459v1…