PaperlayerView exact revision on arXiv

Source-linked AI summary

RobotAgentBench: Towards Open-world Robot Autonomy

Lingjun Mao, Jinglin Cao, Lukun He, Wenpeng Xu, Yuchen Yan, Hengyu Jin, Zijun Zhang, Jinzhou Tang, Songyao Jin, Ziche Liu, Yifei Shao, Junbo Huang, Fang Nan, Ruobing Han, Ziqiao Xi, Ziming Xu, Shuang Liang, Sibo Zhu, Wenyi Wu, Xinyue Wang, Yiming Huang, Xinkai Zou, Gao Zhu, Kun Zhou, Biwei Huang

arXiv:2610.34649v1cs.ROcs.AI

TL;DR

Existing benchmarks often focus on predefined manipulation or isolated capabilities, although open-world household autonomy requires discovery, contextual reasoning, execution, and adaptation. RobotAgentBench evaluates this broader loop in realistic household settings; experiments reveal a manipulation–autonomy gap and benefits from agentic approaches, while real-world evaluations show performance degradation.

  • Problem

    Existing benchmarks largely assess predefined manipulation tasks or isolated capabilities, leaving open-world robot autonomy insufficiently evaluated.

    Source: §1 · p. 2

  • Method

    RobotAgentBench uses a real-to-sim household benchmark and multi-level task and scenario assessments, comparing standalone models and robot-agent architectures in a planning-execution-verification framework.

    Source: §1 · p. 2, §2.3 · p. 7

  • Results

    Evaluations reveal a substantial manipulation–autonomy gap; coding-agent harnesses, hybrid execution, and RSI improve performance, while real-world evaluations show degradation but broadly consistent rankings.

    Source: §1 · pp. 2–3, §5 · p. 15

  • Takeaways & Limitations

    The benchmark highlights the promise of agentic architectures and recursive self-improvement, while open-world robot autonomy remains challenging.

    Source: §5 · p. 15

    Real-world evaluations show performance degradation, despite broadly consistent relative rankings.

    Source: §1 · p. 3

Abstract

from arXiv · show

Recent advances in vision-language-action (VLA) and world-action models (WAMs) have significantly improved robotic manipulation, yet deploying robots in the open world demands a higher level of autonomy beyond executing predefined tasks. Robot agents offer a promising paradigm for this transition, integrating task discovery, contextual reasoning, physical execution, and adaptive recovery into a closed interaction loop. However, existing benchmarks largely evaluate isolated capabilities, failing to capture the complexity of open-world robot autonomy. We introduce RobotAgentBench (RAB), a comprehensive benchmark for evaluating robot agents in realistic, open-world household environments. Through an agent-assisted, human-in-the-loop real-to-sim pipeline, RAB constructs 50 diverse household layouts with interactive objects and physically grounded tasks, encompassing 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and 4 open-ended scenarios. RAB provides a multi-level evaluation of robot agent capabilities: (1) task-level assessments systematically examine five complementary dimensions: task discovery, context reasoning, manipulation, failure recovery, and robustness; (2) scenario-level evaluation assesses how agents integrate these capabilities to autonomously fulfill high-level user needs. Furthermore, RAB enables systematic comparisons of standalone VLA/WAM models and diverse robot agent architectures under a unified planning-execution-verification framework. Extensive experiments reveal substantial gaps between manipulation and autonomy, demonstrate the advantages of agentic architectures and recursive self-improvement, and highlight the remaining challenges toward open-world robot autonomy.

1 Introduction

Open-world household autonomy requires robots to decide what needs doing, reason about context, execute physical tasks, and adapt to unexpected outcomes; existing benchmarks often test predefined tasks or isolated capabilities. RobotAgentBench evaluates this broader loop, and its experiments expose a gap between manipulation and autonomy alongside benefits from agentic approaches.

1 Introduction

  • Robot agents are framed as closed-loop systems that combine contextual planning, physical execution, feedback verification, and adaptation.This shifts evaluation from only how to execute toward also deciding what to do and how to respond to outcomes.Source: §1 · p. 2
  • Existing benchmarks largely target predefined manipulation or isolate reasoning, planning, and execution, leaving open-world autonomous interaction insufficiently assessed.The paper argues that autonomy requires evaluating how these capabilities work together, including task discovery, recovery, and adaptation.Source: §1 · p. 2
  • RobotAgentBench combines 50 household layouts with 90 atomic tasks, 18 short-horizon tasks, 8 long-horizon tasks, and four open-ended scenarios to assess five autonomy dimensions.The dimensions are task discovery, context reasoning, manipulation, failure recovery, and robustness.Source: §1 · p. 2
  • Standalone policies show a gap between manipulation ability and autonomy, while coding-agent harnesses improve performance, especially with hybrid code-and-learned-policy execution.The introduction also reports benefits from recursive self-improvement and performance degradation in real-world evaluation despite broadly consistent relative rankings.Source: §1 · pp. 2–3

2 RobotAgentBench

RobotAgentBench builds a realistic household benchmark from real-to-sim environments and evaluates autonomy at both task and scenario levels. It also compares robot-agent architectures that combine planning, execution, verification, and recursive improvement.

2.1 Real-to-Sim Pipeline for Environment and Data Construction

  • Real-room reconstruction, scene variation, and agent-assisted trajectory generation produce diverse simulated household environments and scalable, physically grounded data.Human reviewers validate reconstructed scenes against real-world visual fidelity and interaction consistency; demonstrations are collected on the AgiBot G1 and replayed in simulation.Source: §2.1.1 · p. 4, §2.1.2 · p. 4

2.2 Multi-level Evaluation of Robot Autonomy

  • Task-level evaluation diagnoses discovery, context reasoning, manipulation, failure recovery, and robustness, while scenario-level evaluation tests their integration toward user needs.The benchmark pairs fine-grained capability assessment with end-to-end evaluation.Source: §2.2 · p. 5

2.2.1 Composable Household Task Suite

  • The task foundation comprises 90 atomic tasks, 18 short-horizon tasks, and 8 long-horizon tasks spanning varied household objects, materials, and manipulation skills.Training data include 50 trajectories per atomic and short-horizon task and 10 demonstrations per long-horizon task and open-ended scenario, totaling 101.8 hours.Source: §2.2.1 · pp. 5–6

2.2.2 Task-level Capability Evaluation

  • Discovery and context tests assess target identification through object contact; manipulation uses explicit instructions, while failure recovery and robustness introduce execution errors or environmental changes.The settings use capability-specific measures and controlled task variants, including missed grasps, human interventions, and lighting changes.Source: §2.2.2 · p. 6

2.2.3 Scenario-level Integrated Evaluation

  • Four open-ended scenarios require agents to infer tasks from sampled environments and user information, plan dependencies, and coordinate multiple physical interactions.End-to-end progress is aggregated across task checkpoints; additional metrics measure coverage, progress, recovery, and stagnation.Source: §2.2.3 · pp. 6–7

2.3 Robot Agent Framework for Evaluation

  • The evaluation framework separates a planner that interprets and adapts tasks, an executor that acts through learned or code-based control, and a verifier that detects failures and guides correction.The components can be flexibly combined to examine their roles in robot autonomy.Source: §2.3 · p. 7
  • Compared architectures include learned policies with coding-agent verification or planning, fully code-based control, hybrid code-and-learned execution, and code-based control with recursive self-improvement.The RSI variant iteratively creates, refines, and reuses tools, retaining successful improvements across subsequent attempts.Source: §2.3 · pp. 7–8

3 Experiments

The experiments compare standalone VLA/WAM policies and complete robot agents across task-level capabilities and open-ended household scenarios, then examine harness design, recursive self-improvement, and sim–real performance. Integrated agent designs generally outperform standalone policies on task-level evaluation, but both struggle to complete open-ended scenarios.

3.1 Experimental Setup

  • The evaluation covers five task-level assessments and four open-ended scenarios across 30 instances per task or scenario, with shared environments, a 22-dimensional command interface, and randomized benchmark conditions.The experiments include standalone VLA/WAM models and five agent harness designs; RSI runs for ten iterations and reports the final iteration.Source: §3.1 · pp. 8–9

3.2 Main Results

  • Standalone policies generally underperform integrated agents, and combining code-based control with selectively invoked learned policies yields the best reported performance.Among fully code-based systems, Antigravity CLI has the highest reported mean score, followed by Claude Code.Source: Table 1 · p. 9, §3.2.1 · pp. 9–10

3.2.1 Task-Level Evaluation

  • Standalone policies struggle with task discovery and contextual reasoning, while InternVLA-A1.5 has the strongest overall standalone performance and notably higher robustness and failure recovery scores.It even surpasses some complete robot agents on those latter dimensions.Source: §3.2.1 · p. 10

3.2.2 Scenario-Level Evaluation

  • Both standalone policies and robot agents perform poorly in open-ended scenarios, where policies often idle or move unproductively and agents can become stuck retrying individual tasks.Figure 3 gives representative cases: InternVLA-A1.5 does not initiate a task, while Codex repeatedly attempts one subtask without completing it before timeout.Source: §3.2.2 · pp. 10–11, Figure 3 · p. 11

3.3 Agent Harness Analysis

  • Antigravity CLI produces the largest gains across evaluated policy executors; in sampled episodes, its instructions are shorter, switch less often, and contain fewer execution details than Codex’s.The authors suggest concise, high-level instructions may be easier for learned-policy executors to follow.Source: §3.3 · p. 11, Figure 4 · p. 12, Figure 5 · p. 12
  • InternVLA-A1.5 performs best as an executor when paired with the same planner, with stronger manipulation, robustness, failure recovery, and following of revised instructions.The authors note its extensive VQA training may help preserve language understanding.Source: §3.3 · p. 12, Figure 4 · p. 12
  • All three fully code-based agents improve at task and scenario levels over successive recursive self-improvement iterations, while Codex’s performance gap gradually narrows.Agents refine reusable control functions, including decomposing motions into feasible intermediate poses.Source: §3.3 · pp. 12–13, Figure 6 · p. 13

3.4 Sim–Real Performance Gap

  • Both standalone models and Codex generally perform worse in the real world than in simulation on the evaluated matched subsets.The study covers selected tasks and four scenario contexts, with 30 evaluation instances per task or scenario in each environment.Source: Table 3 · p. 13, §3.4 · p. 13

4 Related Works

Prior work covers manipulation, embodied planning, and robot-agent architectures, but the related benchmarks often leave open-world household assistance or physical execution and recovery insufficiently evaluated. RobotAgentBench is positioned as a unified evaluation of agentic capabilities and designs.

Robot Manipulation Benchmarks

  • Manipulation benchmarks extend from individual skills to household and longer-horizon tasks, but often retain predefined goals and demonstrations; RobotAgentBench evaluates capability dimensions and open-ended household scenarios.Its demonstrations primarily teach reusable skills for agents to select and compose according to the situation.Source: §4 · p. 14

Embodied Planning Benchmarks

  • Embodied planning benchmarks assess household instruction following, planning, memory, or interaction, but abstracting physical control limits assessment of adaptation to execution outcomes and failure recovery.Source: §4 · p. 14

Robot Agents

  • Robot-agent architectures combine language-model planning, generated programs, learned policies, analytic controls, or execution-time correction; RobotAgentBench offers a unified benchmark for comparing these designs.Source: §4 · p. 14

5 Conclusion

RobotAgentBench evaluates open-world robot autonomy beyond predefined manipulation tasks through a real-to-sim pipeline and multi-level assessment. Experiments expose a substantial gap between manipulation and autonomous task completion, while coding-agent harnesses, hybrid execution, and recursive self-improvement show benefits; reliable open-ended autonomy remains unsolved.

5 Conclusion

  • RobotAgentBench is a comprehensive benchmark for open-world robot autonomy, spanning task-level capabilities and open-ended household scenarios.Its evaluations cover task discovery, contextual reasoning, physical execution, and adaptive recovery.Source: §5 · p. 15
  • Experiments reveal a substantial gap between manipulation capabilities and autonomous task completion, alongside benefits from coding-agent harnesses, hybrid execution, and recursive self-improvement.Source: §5 · p. 15
  • Planning–execution coordination, failure recovery, and real-world deployment remain challenges, and reliable autonomy in open-ended scenarios is far from solved.Source: §5 · p. 15
  • The authors aim to extend agent-level recursive self-improvement into the physical open world through autonomous exploration, learning from failures, and transfer of reusable experience.They describe this as a possible path toward continual improvement with minimal human intervention.Source: §5 · p. 15

A Benchmark Setup

The benchmark setup specifies shared information and action interfaces, diverse household layouts, and persistent simulation conditions for evaluation. Systems receive the same disclosure rules, while their representations and use of the information depend on their model and agent design.

A Benchmark Setup

  • The setup describes the shared simulation settings, robot action space, and information available to evaluated systems.Source: A Benchmark Setup · p. 21

A.1 Provided Information

  • Systems receive task instructions or scenario context, synchronized RGB views, robot-state feedback, and a progressively revealed navigation map under shared disclosure rules.Context can include user preferences, household routines, and storage arrangements; map-assisted systems do not receive target poses or planned routes.Source: §A.1 · p. 21

A.2 Action Space

  • Each action is a 22-dimensional whole-body command covering arm, torso, head, gripper, and mobile-base controls.Joint channels specify target positions; base channels specify forward velocity and yaw rate.Source: §A.2 · p. 21

A.3 Simulation and Execution Settings

  • The scene catalog contains 50 layouts based on a real-home digital twin, with altered floor plans and variation in furniture appearance.Layouts are manually reviewed for geometric validity and plausible furniture arrangements.Source: §A.3 · p. 21, Figure 8 · p. 22
  • Robot, camera, asset, and scene settings remain fixed within each episode, while physics runs at 60 or 240 Hz and the action interface at 20 Hz.Grasping relies on physical contact and friction; cloth manipulation may use a dedicated soft-body solver.Source: §A.3 · p. 22
  • Task transitions preserve scene and object states and accumulated progress, and final placement credit reflects the current physical state at episode end.Disturbing an earlier placement can reduce its final credit.Source: §A.3 · p. 22

B Task and Scenario Descriptions

This section describes manipulation tasks and open-ended household scenarios for evaluating robot autonomy, specifying activity requests or procedures and their completion conditions. Its coverage includes short- and long-horizon tasks and activities such as placement, tidying, cloth replacement, search and packing, and food preparation.

B Task and Scenario Descriptions

  • The benchmark evaluates manipulation, task discovery, context reasoning, failure recovery, and robustness, plus four open-ended scenarios combining discovery, reasoning, and manipulation under high-level requests.The figures show task settings and procedures using demonstrations and diagnostic renders, not evaluated-agent results.Source: B Task and Scenario Descriptions · p. 23

B.1 Manipulation

  • Manipulation comprises 18 short and 8 long tasks, scored with task-specific process measures; long tasks require continuous execution across subgoals without resetting the scene.Episodes provide a task instruction, camera observations, and robot-state feedback, and begin in task-specific scenes.Source: §B.1 · p. 23

1. Prepare coffee; 8. Move food to the microwave

  • Tasks span multi-step appliance use and object transfer, with completion requiring specified object states, such as retrieving the same cup upright or leaving both appliance doors closed.Coffee preparation requires placing a cup, adding beans, and starting and stopping brewing; microwave transfer requires moving the same food box inside and closing both doors.Source: §1 · p. 23, §8 · p. 25

6. Hang the cap; 7. Clear table waste

  • Other tasks test precise placement and destination constraints, including hanging a cap on a valid hook and releasing all designated waste inside the correct bins.Merely placing the cap near the rack or removing waste from the table is insufficient.Source: §6 · p. 25, §7 · p. 25

1. Stock the tray (Multi)

  • The final listed task begins a multi-item tray activity in which requested snacks and drinks are specified by type and quantity and brought from a shelf to a wooden tray.The supplied passage gives the request and setting, but not the task procedure or completion condition.Source: §1 · p. 29
  • Track requested products while placing them in the small wooden tray on the coffee table, without displacing earlier deliveries or using the large meal tray.Completion requires all requested products to be in the small wooden tray.Source: §1 · p. 29

2. Tidy the sofa area

  • Tidy the sofa area by placing the book on the table, storing the wallet in the closed drawer, and hanging the cap on a coat-rack hook.All three placements must remain satisfied at the end.Source: §2 · p. 30

3. Tidy the table

  • Sort designated table waste into assigned bins and place requested belongings in the storage basket, preserving earlier placements and not discarding reusable items.The tote bag is explicitly not the storage destination.Source: §3 · p. 30

4. Replace the tablecloth

  • Replace the used tablecloth by folding and storing it, then retrieving and spreading the clean cloth over the intended tabletop region.Obstructions may need to be cleared first.Source: §4 · p. 31

5. Find and pack an item

  • Search the room for the requested wallet or sunglasses case, then guide that same object into the designated open tote bag without resetting the scene.The instruction identifies the target but does not reveal its location.Source: §5 · p. 31

6. Prepare a meal

  • Retrieve the food from the refrigerator, heat it in the microwave, and place it with the requested cutlery on the large meal tray; tray delivery is unnecessary.Close both the refrigerator and microwave after retrieving their contents.Source: §6 · p. 32

7. Prepare coffee and pastry

  • Prepare coffee and load then retrieve the target pastry using the air fryer; neither heating the pastry nor returning it to its plate is required.The basket must be removed completely and set down securely before retrieving the pastry.Source: §7 · p. 32

8. Pack bags and baskets

  • Transfer each requested object to its assigned bag or storage basket, track remaining objects, and preserve prior placements until all targets are supported inside their containers.Assignments may use one container type throughout or mix bags and baskets.Source: §8 · p. 33

Identify task-relevant objects

The benchmark evaluates task discovery, contextual object selection, and recovery from predefined manipulation failures through household tasks that require identifying targets, applying context, or adapting an action.

Identify task-relevant objects

  • Agents explore from a scenario and household context, touch inferred target objects, and receive object-coverage credit under a limited contact budget without completing the underlying tasks.Each episode permits contact with at most N+2 distinct objects; repeated contacts count once and non-target contacts receive no credit.Source: Identify task-relevant objects · p. 33

B.3 Context Reasoning

  • Context reasoning tests whether agents use preferences, intended uses, classification rules, or revised requests to select the appropriate visible object by touch.Selection ends at contact; grasping, carrying, placement, and disposal are not required.Source: §B.3 · p. 34

1. Drink selection; 2. Snack selection

  • Drink and snack selection require matching stated properties, with confirmed requests taking priority and broadly similar products rejected when they conflict with requirements.Examples distinguish sweet zero-sugar cola from sparkling water and smoky barbecue crisps from sour-cream-and-onion crisps.Source: §1 · p. 34, §2 · p. 35

3. Waste selection; 4. Sofa object selection

  • Waste and sofa selection apply supplied category rules or current user needs, including following a latest request when an earlier plan is superseded.These are contact-based selection tasks, not waste transport or sofa-to-table transfer.Source: §3 · p. 35, §4 · p. 36

B.4 Failure Recovery

  • Failure recovery starts from a predefined failed state with the target still present and evaluates adjustment and task completion across four missed grasps and one tipped-cup state.The five settings cover a shelf drink, coffee-bean bag, TV remote, upright used cup, and tipped cup.Source: §B.4 · p. 36

1. Shelf pickup (Missed grasp); 2. Pour coffee beans (Missed grasp)

  • Shelf and coffee-bean recovery require adjusting the grasp; the drink must be lifted and held steadily, while the bag must be used to pour beans into the hopper and returned upright.Tray delivery and making coffee are not required.Source: §1 · p. 36, §2 · p. 37

3. Move sofa items to the table (Missed grasp); 4. Clear table waste (Missed grasp)

  • Remote and upright-cup recovery require a new grasp and placement at the destination, while remote recovery accepts any valid supported position on the tabletop.The standard sofa-to-table task instead specifies a restricted placement area; the cup must go inside its assigned waste bin.Source: §3 · p. 37, §4 · p. 38

5. Clear table waste (Tipped cup)

  • Tipped-cup recovery requires adapting pickup to the cup’s position and orientation, then releasing it inside its assigned waste bin.The starting cup lies on its side and is not held by either gripper.Source: §5 · p. 38

B.5 Robustness

The appendices specify disturbance-robustness tests, four multi-subtask household scenarios, and adaptation of policy baselines to the AgiBot G1.

B.5 Robustness

  • Six robustness settings test adaptation to object relocation, changing illumination, coffee-bag movement, machine displacement, combined disturbances, and button-operation lighting changes.The tests include shelf pickup and coffee preparation; agents receive instructions, camera observations, and robot-state feedback, and are scored with task-specific process scores.Source: §B.5 · p. 38

B.6 Open-ended Scenarios

  • Open-ended episodes combine subtasks in one persistent scene, with agents inferring tasks from household memory and observations without receiving the subtask list or target bindings.Scenarios cover dinner preparation, hosting guests, preparing to leave, and room tidying; earlier results must be preserved.Source: §B.6 · p. 41
  • The scenarios use a shared scene and object instances without resets between subtasks, and score verified physical progress with partial credit while checking completed placements.Episode scores are normalized by the sum of subtask maxima; disturbing earlier placements can reduce the score.Source: §B.6 · pp. 41–42

1. Prepare dinner

  • Dinner preparation combines a preference-informed meal and accompaniments with serving on a large tray and clearing misplaced belongings and waste.The example uses household memory to infer Ryan’s usual meal and coffee.Source: §1 · p. 42

2. Host guests

  • Hosting guests requires inferring a visitor’s usual refreshments from household memory, serving them together, and putting away belongings and waste while leaving other stocked products in place.Source: §2 · p. 43

3. Prepare to leave

  • Preparing to leave requires packing activity-specific belongings, putting away other misplaced objects, removing waste, and cleaning table stains when present.Objects already packed or correctly stored need no transfer.Source: §3 · pp. 43–44

4. Tidy the room

  • Room tidying requires returning belongings, sorting waste, and cleaning table stains when present, while leaving correctly placed objects undisturbed.The example includes storage, disposal, and recycling tasks but requires no table cleaning.Source: §4 · p. 44

C Policy training and model-specific adaptation

C.1 Shared training setup

The appendix standardizes the demonstration data and training split across eleven baselines, then documents model-specific fine-tuning budgets and adaptations.

C.1 Shared training setup

  • All eleven policy baselines use demonstrations from the 101.8-hour simulation dataset and the same episode-level training/validation split.Source: §C.1 · p. 45
  • Demonstrations use 23-dimensional robot states and 22-dimensional whole-body actions, with model-specific input formatting and training-split normalization statistics.Image augmentation is applied only during training.Source: §C.1 · p. 45
  • Selected continuation phases draw 80% of samples from short- and long-task demonstrations and 20% from atomic-action demonstrations.Fine-tuning otherwise follows each model’s native recipe, including prescribed frozen components.Source: §C.1 · p. 46
  • Table 5 reports trainable parameter counts, final-phase global batch sizes, and total optimizer updates; its batch sizes include gradient accumulation.Each training step denotes one optimizer update, and batch sizes describe the final phase rather than the full training history.Source: Table 5 · p. 46, §C.1 · p. 46

C.2 Model-specific settings

  • Model-specific settings adapt state and action interfaces, camera inputs, or training objectives, with listed training budgets ranging from 16,000 to 120,000 optimizer updates.The settings include a tau-0 action-policy configuration rather than the complete world-model hierarchy, and OpenWAM-Alpha retains its joint video-and-action objective.Source: §C.2 · pp. 46–47

D Detailed Analysis of Open-ended Scenarios

The analysis pairs scenario scores with behavioral measures of agents’ interaction patterns and responses to execution difficulties.

D Detailed Analysis of Open-ended Scenarios

D.1 Behavioural metrics

The section defines behavioral measures for task engagement, process-score attainment, physical-failure recovery, and stagnation, alongside scenario-level behavior analysis.

D.1 Behavioural metrics

  • Task Coverage measures the proportion of required tasks physically engaged, counting each task once and excluding workspace selection and background monitoring.Engagement requires eligible gripper-finger contact with a registered target or a task-specific operation-start event.Source: §D.1 · p. 47
  • Task Progress measures process-score attainment among engaged tasks, including target-contact credit but excluding final-state score channels and unengaged tasks.Source: §D.1 · p. 47
  • Recovery Rate is the percentage of observable physical-failure segments followed by task-specific success on the same task at a later control step.Repeated failures before recovery form one segment; API errors, formatting errors, and termination-only settlement events are excluded.Source: §D.1 · p. 47
  • Coverage, progress, and recovery percentages pool numerators and denominators across scenarios, while Figure 55 encodes task-event distributions and execution issues by action purpose and issue family.Zero denominators are reported as undefined; figure percentages are rounded independently.Source: §D.1 · p. 47, Figure 55 · p. 48
  • Stagnation measures simulation time spent in repeated physical execution without progress, applying a 30-second grace period and excluding successful navigation above specified motion thresholds.Repeated state loops can also qualify under the stated call-count, duration, and matching-tolerance conditions.Source: §D.1 · p. 47

D.2 Behavioural patterns and execution issues

Agents differ in how they allocate task events, but all more often replan from feedback than explicitly select tasks or goals. Motion-feasibility issues and execution shortfalls dominate identified execution issues.

D.2 Behavioural patterns and execution issues

  • 42.4% of Codex task events involve object manipulation, compared with 16.1% for Claude Code and 28.6% for Antigravity CLI.Claude Code instead devotes 57.3% of its events to exploration, grounding, and base navigation combined.Source: §D.2 · p. 48
  • All three agents replan from feedback more often than they explicitly select tasks and goals, including by shortening approach steps or revising wrist orientations.Task-event shares are averaged equally across scenarios after within-episode normalization.Source: §D.2 · p. 48
  • 82.1% of identified issues are motion-feasibility issues or execution shortfalls, versus 15.3% code or interface errors.These percentages describe the composition of issue-bearing calls, not the overall error rate.Source: §D.2 · p. 48
  • Reliable execution requires coordinating navigation, stance adjustment, and manipulation, then verifying that revisions produce the intended physical progress.The identified difficulties span both positioning and object manipulation, not just tool-call correctness.Source: §D.2 · p. 48
Loading 2610.34649v1…