PaperlayerView exact revision on arXiv

Source-linked AI summary

RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments

Zhiqin Yang, Chenxin Li, Xiaomeng Hu, Yibin Liu, Weidong Huang, Jiankai Sun, Haitao Li, Zijian Wu, Yuzhi Huang, Fanding Huang, Hanwen Sun, Jiashun Liu, Jingqi Tong, Mingxin Huang, Shaoli Hu, Shijue Huang, Tianyi Bai, Xinyuan Wang, Yunlong Lin, Zhengyang Tang, Zhexin Zhang, Zhuo Chen, Xierui Song, Juntao Dai, Boyuan Chen, Jiaming Ji, Fangneng Zhan, Mengkang Hu, Wei Xue, Yonggang Zhang, Han Hu, Tsung-Yi Ho, Yike Guo

arXiv:2610.10409v1cs.ROcs.LG

TL;DR

General-purpose agents are increasingly capable in digital tasks, raising the question of how far those capabilities extend to robot use. RobotWorld evaluates them across diverse simulated tasks and analyses execution traces, finding capable perception and control workflows that do not consistently yield reliable completion. The results identify capability gaps and differences in model strengths across physical demands.

  • Problem

    The paper asks how far general-purpose agents’ digital capabilities transfer to physical robot use and where execution breaks down.

    Source: §1 · p. 2

  • Method

    RobotWorld brings together 84 task specifications with documented robot interfaces, task budgets, and executable success checks, and analyses recorded actions, tool interactions, and outcomes.

    Source: §1 · p. 2

  • Results

    Agents construct perception and control workflows, but these capabilities do not consistently compose into task completion, with failures involving lost task-relevant states, ineffective action correction, temporal coordination, and completion assessment.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    RobotWorld’s execution analysis identifies concrete targets for improving reliable physical action, including preserving task-relevant states, correcting ineffective actions, meeting temporal constraints, and recognising completed goals.

    Source: §8 · p. 26

    The suite’s finite, uneven coverage characterises performance on its tasks rather than a representative distribution of all robot use.

    Source: §7 · p. 26

Abstract

from arXiv · show

General-purpose agents increasingly write code, use tools, and complete complex digital tasks, raising the question of how far these capabilities carry into the physical world. To investigate this, we introduce RobotWorld, a challenging simulation testbed for robot use: turning instructions and observations into physical task execution through robot interfaces. Its 84 tasks span manipulation, mobile manipulation, locomotion, driving, and aerial control, with explicit interaction budgets and executable success checks. By analysing task outcomes alongside execution traces, we identify both the capabilities that transfer and the gaps that prevent reliable completion. Furthermore, we find that current agents can construct sophisticated perception and control workflows, including image segmentation, camera calibration, spatial estimation, and dynamics-based computation. These capabilities, however, do not consistently compose into successful behaviour: agents lose task-relevant object states despite reaching commanded poses, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion. This uneven transfer also differs across models: Astra succeeds more often on spatial and constrained-contact goals, whereas Opus 5.5 succeeds more often on continuous-balance and timed-interaction goals. By linking these outcomes to execution behaviour, RobotWorld provides both a rigorous proving ground and an empirical account of the remaining capability gaps, thereby establishing concrete targets for training and designing more reliable physical-world agents.

1 Introduction

RobotWorld asks how far general-purpose agents’ digital capabilities transfer to robot use, and where execution breaks down. It combines a diverse, checked simulation benchmark with trajectory analysis, revealing capable perception and control workflows that do not consistently compose into task completion.

1 Introduction

  • The benchmark examines whether general-purpose agents can turn digital capabilities into reliable physical action across robot bodies and demands.The motivating question is which capabilities transfer and what should improve next.Source: §1 · p. 2
  • RobotWorld’s trajectory analysis finds agents using colour segmentation, camera-geometry fitting, spatial estimation, and numerical dynamics calculations.Agents also use feedback to revise motions and recover from execution errors.Source: §1 · p. 2
  • Agents build perception and control workflows, yet lose task-relevant states, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion.These failures point to gaps in state interpretation, action revision, temporal coordination, and completion assessment.Source: §1 · pp. 2–3
  • Across analysed task groups, Astra more often succeeds on spatial and constrained-contact goals, while Opus 5.5 more often succeeds on continuous-balance and timed-interaction goals.Among eight tasks solved by both, Astra uses fewer tool calls in seven but fewer control steps in only six.Source: §1 · p. 3
  • RobotWorld provides 84 tasks from 20 source projects across diverse embodiments and control demands, with explicit interfaces, budgets, and executable success checks.Recorded actions, tool interactions, and outcomes support analysis beyond aggregate scores.Source: §1 · pp. 2, 4

2 Related Work

Related work studies foundation models for action generation and prediction, robot-use agents that connect instructions to execution, and benchmarks for embodied capabilities. RobotWorld focuses on breadth and reliability across physical demands, examining how execution mechanisms relate to task completion.

Foundation Models for Robotics

  • Vision–language–action models generate actions from visual observations and language, while world–action models connect action generation with predictions of future observations.RobotWorld examines how reasoning, programs, and feedback combine these capabilities during execution.Source: §2 · p. 4

Agents for Robot Use

  • Robot-use agents use reasoning, programs, and tools to connect instructions to physical action, with prior work developing execution feedback, reusable skills, and robotics research workflows.RobotWorld focuses on how agents combine available capabilities and which mechanisms support or obstruct completion.Source: §2 · p. 5

Benchmarks for Embodied Agents

  • Embodied-agent benchmarks cover planning, navigation, manipulation, and other capabilities, while RobotWorld spans diverse embodiments and physical demands including sustained stabilisation and moving targets.It analyses feedback, recovery, completion assessment, and interaction cost alongside outcomes; it can interleave direct commands and generated programs within an episode.Source: §2 · p. 5

3 RobotWorld Environment

RobotWorld provides a common protocol for robot use across heterogeneous simulators and embodiments while preserving each source system’s physics, action semantics, and assistance. Its task contracts and feedback loop specify how agents observe, act, and are evaluated.

3.1 Task Definition and Interaction Loop

  • Each task contract specifies an environment, initial conditions, goal, observation and action interfaces, assistance, budgets, and an evaluator.At each interaction, the agent may request execution, workspace computation, or text; evaluation separately checks the resulting state and events.Source: §3.1 · pp. 5–6

3.2 Observations and Analysis Workspace

  • Task-specific observations include permitted images, state, and feedback, while evaluator-only state and review cameras remain outside the agent interface.The offline workspace supports image inspection, geometric estimation, calculations, and notes, but cannot access simulator internals or advance physics.Source: §3.2 · pp. 6–7

3.3 Robot Actions and Execution Feedback

  • Direct actions execute bounded commands, while optional code control updates actions from fresh observations at each step before the agent replans between calls.Code control is available only where supported and enabled; both modes preserve the task’s documented command semantics and controller assistance.Source: §3.3 · p. 8, Figure 5 · p. 8

3.4 Episode Execution and Records

  • Physics advances during robot execution and pauses during reasoning, so control steps, tool interactions, and wall-clock time measure different resource costs.A completed call or textual completion claim does not establish success; the evaluator checks required states and events during execution.Source: §3.4 · pp. 8–9

4 RobotWorld Benchmark

RobotWorld combines tasks from diverse robot domains under explicit instructions, interaction conditions, and executable success criteria. The benchmark specifies both the physical demands agents face and the evidence required to establish completion.

4 RobotWorld Benchmark

  • The benchmark comprises 84 tasks from 20 source projects spanning manipulation, mobile manipulation, locomotion, driving, and aerial control.Tasks are evaluated under explicit interaction conditions and executable success criteria.Source: §4 · p. 9

4.1 Task Scope and Selection

  • The suite targets overlapping requirements such as spatial goals, constrained contact, multiple subgoals, balance, and timed interaction.It contains 38 manipulation, 20 mobile manipulation, 11 locomotion, 11 driving, and four aerial tasks; manipulation and mobile manipulation comprise 69.0% of the suite.Source: §4.1 · pp. 9–10

4.2 Task Construction and Adaptation

  • Task construction combines source assets with an instruction, initialisation, interaction contract, and success checker, distinguishing objective changes from interface or scenario changes.Selected locomotion and aerial tasks receive explicit completion goals when a continuing reward alone does not establish instruction completion.Source: §4.2 · p. 10

4.3 Success Criteria

  • Success checkers assess physical outcomes rather than reference action sequences, using task-relevant geometry, object state, contact, and motion conditions.Other checker components can require persistence, event order, or path and safety constraints; the required evidence depends on the task definition.Source: §4.3 · p. 11
  • Persistence, event order, and path constraints can make completion depend on the trajectory, not just the final pose.A completed call, favourable image, or agent success claim can be insufficient when the checker requires additional evidence.Source: §4.3 · pp. 11–12

4.4 Validation and Coverage

  • Checker validation does not establish reachability for every task or robustness across initial conditions, and representative embodiment coverage does not isolate morphology effects.Success claims are bounded by the physical relations represented in a checker and its sampling rate.Source: §4.4 · p. 12

5 Benchmarking Multimodal Agents on RobotWorld

RobotWorld evaluates multimodal agents on 84 robot tasks under common interfaces, budgets, and executable checks. Results show limited, model-specific task coverage, while resource measures do not straightforwardly track success.

5.1 Experimental Setup

  • Five models each attempted 84 tasks (420 episodes) under shared initialization and horizons; agents received task goals, tool documentation, and permitted observations and replanned from feedback.Environment-side code control was disabled.Source: §5.1 · p. 13

5.2 Evaluation Protocol and Metrics

  • Episodes ended at task termination or budget exhaustion; executable checks assessed success at the evaluation boundary, while resource summaries included unsuccessful episodes and full recorded interactions.The protocol also imposed task-specific control-step horizons and separate limits on non-action events.Source: §5.2 · p. 13

5.3 Main Results

  • 21/84 tasks (25.0%) are covered by at least one evaluated model, leaving 63 unsolved; this union is not a best-model-selection policy.The aligned task outcomes show which tasks each model solved and where successes overlapped.Source: Figure 8 · p. 14, §5.3 · p. 14

5.4 Performance Across Task Domains

  • Astra solves 9/38 manipulation tasks (23.7%) and 4/20 mobile-manipulation tasks (20.0%), while Opus 5.5 solves 7/38 (18.4%) and none, respectively.In fixed-base manipulation, the two models share five successes; their combined coverage across the two manipulation domains is 15/58.Source: §5.4 · p. 15
  • Opus 5.5 completes 3/4 aerial tasks (75.0%) and the only locomotion success (1/11, 9.1%), while nine driving and ten locomotion tasks remain unsolved.The aerial set is small: one task changes its success rate by 25 percentage points.Source: §5.4 · pp. 15–16

5.5 Recorded Resource Consumption

  • Astra's median total tool calls are 70 versus Opus 5.5's 57.5 despite shorter median elapsed time, so interaction frequency and duration rank differently.Elapsed-time records include setup, waiting, and execution and may extend beyond retrospective scoring cutoffs; they do not measure time to success.Source: §5.5 · pp. 16–17, Figure 10 · p. 16

5.6 Success, Token Consumption, and Estimated Cost

  • Astra reaches 19.0% success at an estimated $9,913 and 941M tokens, while Opus 5.5 reaches 15.5% at $2,916 and 1.63B tokens.Astra's estimated expenditure is about 3.4 times Opus 5.5's, although its reported token volume is lower; prices and cache accounting differ.Source: §5.6 · p. 17

6 Analysis

The analysis links retained execution traces to evaluator outcomes, using episode case studies to examine mechanisms and cross-task profiles for broader descriptive comparisons.

6.1 Feedback-Driven Adaptation and Recovery

  • Effective feedback changed actions and checked recovery: K3 shortened drawer pushes, Astra reconfigured before retrying a rejected camera rotation, and Opus 5.5 sustained computed drone control.The charger episode illustrates reconfiguration but does not isolate it as the cause of success.Source: §6.1 · p. 18

6.2 Failure Modes

  • Failures persisted when agents lost object state, remained in preparation, stopped task-directed correction, or recovered after temporal thresholds.In chopping, the observed bottleneck was reaching the workspace, so the episode does not diagnose the unattempted cutting skill.Source: §6.2 · pp. 18–19, Figure 13 · p. 20

6.3 Control Strategies Across Tasks

  • In drone juggling, Opus 5.5 sustained 15 valid and 14 height-qualified contacts versus Astra’s 3 and 1; stacking showed distinct perception and computation sequences.Both juggling agents used a median of three executed steps per action; Astra completed stacking at step 635, while Opus 5.5 reached its budget at step 313 without completion.Source: §6.3 · pp. 20, 22, Figure 14 · p. 21, Figure 15 · p. 21

6.4 Completion Assessment and Decisions to Continue

  • Continuing only helps when it resumes corrective search: agents may stop because they believe a task is complete or further progress infeasible, while some goals require sustained conditions.In white-mug placement, the release may itself have triggered success, so the episode does not establish unnecessary post-success action.Source: §6.4 · pp. 22–23, Table 4 · p. 22

6.5 Interaction Efficiency

  • Interaction and physical costs can diverge: in cube placement, Astra used fewer tool calls than Opus 5.5 (24 versus 31) but more control steps (286 versus 239).The matched-scene comparisons separate robot, shell, and image-view calls from executed control steps.Source: §6.5 · p. 23, Figure 16 · p. 23

6.6 Cross-Task Model Profiles

  • Astra succeeds more often on spatial and constrained-contact tasks and uses fewer calls on seven of eight shared successes; Opus 5.5 leads on continuous-balance and timed-interaction tasks.The proposed explanation—that the models compose perception, computation, and control differently—remains exploratory.Source: §6.6 · pp. 24–25, Figure 19 · p. 25, Figure 18 · p. 25

7 Discussion and Limitations

The discussion frames reliable robot use as a matter of composing capabilities across an episode, while identifying boundaries on what the simulation results establish.

Discussion

  • Reliable completion requires agents to track action consequences, revise failing strategies, and assess progress; the paper proposes testing visual access, auxiliary computation, and controller support separately.The contrasting model profiles do not isolate the contribution of vision, coding, or control.Source: §7 · p. 26

Limitations

  • The evaluation measures performance in simulation interfaces that pause physics during inference, so it does not establish transfer to real-time loops or hardware.Physical deployment adds inference delays, sensor noise, and contact or dynamics discrepancies.Source: §7 · p. 26
  • RobotWorld’s task coverage is finite and uneven, with related tasks inherited from shared source projects; results therefore do not represent all robot use.Public task descriptions, code, or demonstrations may also have appeared in model training data.Source: §7 · p. 26

8 Conclusion

RobotWorld connects measurable outcomes with execution patterns to characterize emerging robot-use capabilities and the remaining gaps in reliable action.

8 Conclusion

  • Across simulation tasks, agents build perception and control workflows, but reliable action also requires preserving task states, correcting ineffective actions, meeting temporal constraints, and recognizing completion.The conclusion presents model differences as evidence that strengths in one physical-demand class do not imply uniformly strong robot use.Source: §8 · p. 26

A.1 Task inventory and reading guide

  • The appendix inventories 84 tasks from 20 source groups across five domains and links task objectives to observations, tools, and evaluator checks.The domains are manipulation, mobile manipulation, locomotion, driving, and aerial tasks.Source: §A.1 · p. 31

A.2 Primary task types

  • Task types form an editorial partition by primary objective; secondary demands may overlap, and the executable checker remains the authority for success.The inventory distinguishes task objective from embodiment and from overlapping requirements such as constrained contact or multiple subgoals.Source: §A.2 · p. 32

A.3 Task profile 01: Ordered block stacking

These task profiles examine how success depends on satisfying each task’s full state-based conditions, rather than merely reaching a commanded pose, including maintaining joint stability of a drone and suspended payload over time. The retained episodes include successes, failures, and delayed recovery from ineffective actions.

A.3 Task profile 01: Ordered block stacking

  • Astra completed the ordered red–blue–green–yellow stack at step 635, after grasping and lifting a block and making an initial placement.The episode illustrates the need to track object retention and verify support after release; reaching a pose or closing the gripper alone does not establish placement.Source: §A.3 · p. 33

A.4 Task profile 02: Plugging in a charger

  • Astra’s charger insertion succeeded at step 323 after a rejected rotation was retried successfully following an adjustment to the idle arm.The retained trajectory passed insertion and return conditions; lateral alignment alone does not satisfy the task’s depth, orientation, and robot-return requirements.Source: §A.4 · p. 34

A.5 Task profile 03: Closing a kitchen drawer

  • Kimi K3 closed the drawer at step 265, when the environment terminated its requested 12-step withdrawal after only two steps.The late pushing requests shortened from 20, 20, 12, 6, to 5 steps; the truncated retreat reflects checker termination, not a requirement to complete the full retreat.Source: §A.5 · p. 35

A.6 Task profile 04: Pouring into a cup

  • Kimi K3 failed to pour liquid within the 400-step horizon after toppling the bottle at step 42 and delaying a substantial strategy change until step 240.Repeated left-arm attempts did not establish a usable grasp, and the right-arm attempt remained unfinished; moving a grasp point does not verify object retention or liquid transfer.Source: §A.6 · p. 36

A.7 Task profile 05: Following a courtyard route

  • Kimi K3 reached all six route checkpoints and completed the required 0.8-second parking hold, succeeding at step 1,881 of 2,000.Completion required the ordered route as well as destination, heading, speed, and dwell conditions; reaching the green region alone was insufficient.Source: §A.7 · p. 37

A.8 Task profile 06: Quadruped disturbance recovery

  • Opus 5.5 completed quadruped recovery with final error 0.192 m, speed 0.0022 m/s, and tilt 4.48 degrees, meeting the checker’s thresholds throughout the final 2 seconds.Endpoint values alone are insufficient: the required error, speed, and tilt bounds must hold together throughout that window.Source: §A.8 · p. 38

A.9 Task profile 07: Stabilising a suspended payload

  • Opus 5.5 stabilised the suspended payload, passing with final payload error 0.0394 m and speed 0.0147 m/s.Success requires payload error and speed, swing, and drone tilt bounds to hold together through the final 2 seconds.Source: §A.9 · p. 39
  • The controller must track payload displacement and velocity alongside drone tilt because thrust corrections can excite swing and motor lag delays their physical effect.An upright drone alone is not sufficient evidence of success, and a single passing frame does not meet the temporal stability window.Source: §A.9 · p. 39
  • Opus 5.5 achieves 0.0394 m payload error and 0.0147 m/s speed, sustaining all four bounds together for 2.15 seconds against a two-second requirement.The trajectory maintains the recovered state through the end rather than stopping at the first apparently stable observation.Source: §A.9 · p. 39
  • Among the retained episodes, Opus 5.5 succeeds; Astra, Kimi K3, DeepSeek V4.1 Flash, and Gemini 3.8 Flash fail.There is one episode per model.Source: §A.9 · p. 39

A.10 Task profile 08: Repeated drone ball juggling

In repeated drone ball juggling, Opus 5.5 completes the full horizon and meets the checker’s hit and final-airborne requirements, while the other retained model episodes fail.

A.10 Task profile 08: Repeated drone ball juggling

  • The agent controls four rotors through bounded 50 Hz action segments using native drone, ball, and episode-state observations.The review camera is unavailable to the policy, and thinking pauses simulator time.Source: §A.10 · p. 40
  • The checker requires at least four height-qualified hits and a final 0.6-second airborne interval without support contact.A qualifying arc exceeds 3.5 m ball-centre height, while contacts separated by at most 25 steps are invalid.Source: §A.10 · p. 40
  • The trajectory alternates ball-contact and return-preparation phases; visible proximity alone cannot establish a legal hit or reconstruct the intervening flight arc.Frequent robot requests and accumulated hits do not replace the checker’s distinct final-window requirement.Source: §A.10 · p. 40
  • Opus 5.5 succeeds, while Astra, Kimi K3, DeepSeek V4.1 Flash, and Gemini 3.8 Flash fail; each outcome is one retained episode.Opus 5.5 records 15 valid hits, 14 height-qualified hits, 274 robot requests with a median of three executed steps, and satisfies the final airborne interval across the 800-step horizon.Source: §A.10 · p. 40

B Execution, Observations, and Evaluation Records

This section documents RobotWorld’s execution records, tool and scoring contracts, observation conventions, and driving and flight interfaces and constraints. It also covers task-level results and resource distributions for five models, distinguishing model observations from evaluator and reviewer records and clarifying that feedback or apparent completion alone does not establish what action occurred or whether a task succeeded.

B.1 What the archive contains

  • The archive contains 420 verified task-record pages, and its retained score, embedded historical result, and event/video stream represent distinct readings.The pages were SHA-256-verified against the five-model result audit; when the historical terminal success occurs after the scored boundary, the retained score is not replaced with a later plausible frame.Source: §B.1 · p. 41

B.2 Registered robot tools across the benchmark

  • The catalog covers 20 source groups, with tool-name sets consistent across five models within each group even though parameters and action maps can vary by task.Shell and image viewing belong to the frontier harness rather than the robot-tool catalog; environment-side coding control, reset, score-query, and give-up tools were not registered in retained runs.Source: §B.2 · p. 41

B.3 Manipulation tools: arguments and execution meaning

  • Robot commands with the same name can differ in coordinate frame, controlled point, units, persistence, and whether repeated inputs hold a pose or accumulate motion.For example, RoboLab uses absolute flange poses while RoboCasa repeats normalized end-effector increments; BEHAVIOR-1K combines absolute arm poses with physical base velocities, unlike RoboCasa’s normalized base and torso commands.Source: §B.3 · p. 42, §B.4 · p. 43

B.5 Native-vector tools and action maps

  • Across 13 native-action groups, valid action vectors use task-specific schemas and active native dynamics, so receipts, executed steps, observations, and evaluator outcomes must be inspected separately.A valid request can still cause tracking error, collision, lost grasp, or task failure; an invalid request can consume an interaction without advancing physics.Source: §B.5 · p. 44

B.6 Control-step and interaction accounting

  • The scored protocol imposes task-specific interaction limits, distinguishes requested from executed steps, and marks some resource accounting incomplete.For 83 tasks, the consecutive non-action limit is 15 and total limits are 30, 60, or 120; Volleyball 1v1 uses 20 consecutive and 150 total. Auxiliary and action-free turns count under the observable-event policy, while API retries are excluded; the final evaluator step count can differ from requested or confirmed steps.Source: §B.6 · p. 45

B.7 Success checks and their observation window

  • Temporal checks cover logged control ticks, including recorded resets, not unobserved continuous physics; payload hovering requires four bounds throughout the final two seconds, while juggling requires event count and final airborne window.Native numerical rewards remain diagnostics and are not averaged across heterogeneous tasks.Source: §B.7 · p. 45

B.8 Frame provenance

  • Appendix frames come from timestamped, hashed source videos, while model observations, retained history, and reviewer video remain separate records.Video timestamps indicate playback time rather than elapsed model time or a guaranteed tool-event match; external driving and aerial views are reviewer views, and late frames may precede terminal success.Source: §B.8 · p. 45

C Prompts and Agent Instructions

  • The prompt examples specify embodiment-specific sensing and action contracts, including grasp-point versus flange control, incremental versus absolute mobile manipulation, and onboard driving versus direct rotor control.The examples also distinguish supplied observations from reviewer views; the driving and flight descriptions pair onboard visual/proprioceptive input with speed-and-steering commands or native actor measurements with rotor inputs.Source: C Prompts and Agent Instructions · p. 46, §C.1 · p. 46, §C.2 · p. 47

C.3 Driving and flight

  • Driving uses onboard camera images and ideal wheel, steering, and body-IMU sensors, while global position, map and hazard state, progress, and review-camera video are withheld.The policy must infer road shape, hazards, and stopping locations from onboard imagery; sensor noise is not added.Source: §C.3 · p. 48
  • Non-action interaction limits are 15 consecutive and 30 total calls for driving, versus 15 consecutive and 120 total for flight; the evaluator ends the episode at either limit.Auxiliary calls and zero-control-step robot calls count toward these budgets; confirmed control steps reset only the consecutive counter.Source: §C.3 · p. 48
  • The Hummingbird is controlled through four simultaneous normalized rotor commands without an automatic stabilizer, and zero command means half maximum steady-state thrust.Only the original actor policy group is supplied, and the observation contract determines whether simulator-derived state is included; critic and review-video inputs are excluded.Source: §C.3 · p. 48

D Full Quantitative Results

  • Section 5 reports complete task-level outcomes and numerical resource distributions for all five models, but this excerpt supplies no specific values.The passage identifies the results’ scope without reporting their outcomes or numbers.Source: D Full Quantitative Results · p. 49

D.1 All task–model outcomes

The appendix reports task-level outcomes and resource distributions across RobotWorld’s 84 tasks and five models, then illustrates selected successful trajectories. The trajectory cases are qualitative and do not estimate behavioural prevalence.

D.1 All task–model outcomes

  • Table 11 inventories 84 tasks and retained outcomes for five models; the listed rows show task-specific successes across manipulation, driving, and aerial tasks.Astra and Opus 5.5 both succeed on several listed manipulation tasks, while the listed driving and aerial successes vary by task and model.Source: Table 11 · p. 49, §D.1 · pp. 49–51

D.2 Resource distributions

  • Tables 12 and 13 report across-task medians and interquartile ranges for time, interaction use, physical execution, and auxiliary-tool use.The intervals describe variation across tasks, not uncertainty from repeated trials.Source: Table 12 · p. 51, Table 13 · p. 51, §D.2 · p. 51

D.3 Aggregate token and cost accounting

  • Table 14 reports aggregate consumption using website list-price estimates; Kimi’s token and cost totals extrapolate 57 valid records to 84 tasks.Provider token definitions, cache accounting, and unit prices differ, so token and cost rankings need not coincide.Source: Table 14 · p. 52, §D.3 · p. 52

E Successful Trajectories

  • Four deliberately selected success cases illustrate sequential manipulation, dynamic stabilization, recovery after a rejected action, and competitive interaction.The cases link observations, agent decisions, and execution feedback, but do not estimate how often these behaviours occur.Source: E Successful Trajectories · p. 53

E.1 Astra: building and checking an ordered stack

  • Astra’s ordered stack succeeds at step 635 after repeated approach–grasp–lift–place sequences, although the final client receipt is missing.The retained evaluator reports success despite the missing receipt for the final release request.Source: §E.1 · p. 53, Figure 20 · p. 53

E.2 Opus 5.5: repeated contacts under rotor-level control

The examples trace three successful episodes through robot feedback: Opus 5.5 sustains repeated contacts, Gemini revises an unreachable lift, and DeepSeek wins a volleyball episode. Each illustrates a particular run, not a general success rate.

E.2 Opus 5.5: repeated contacts under rotor-level control

  • 15 true hits and 14 height-qualified hits in Opus 5.5’s 800-step juggling run exceed the four-hit requirement, with the final airborne condition satisfied.The retained record reports 274 robot requests; the figure’s rendered views are not the policy’s input, and the retained checker establishes the outcome.Source: Figure 21 · p. 54, §E.2 · p. 54

E.3 Gemini: recovering from an unreachable lift

  • After a 0.95 m lift is rejected, Gemini lowers the request to 0.85 m and receives confirmation of progress from step 501 to 508.The episode ends at step 511 with retained success; consecutive tool receipts establish the recovery sequence.Source: Figure 22 · p. 55, §E.3 · p. 55

E.4 DeepSeek: correction, interception and a terminal win

  • DeepSeek’s volleyball episode ends in a native terminal win at step 241, after four total contacts, two by each side.This is one winning episode, not an estimate of match win probability; the figure’s outcome is the controlled actor’s terminal win.Source: Figure 23 · p. 56, §E.4 · p. 56

F Observable Behavioural Patterns

The behavioural analysis treats linked observations, agent outputs, tool requests, and receipts as descriptive evidence. It distinguishes visible correction from verified control correctness and agent intent from evaluator-confirmed outcomes.

F Observable Behavioural Patterns

  • The cases illustrate calibration revision, contact or grasp reassessment, pre-action analysis saturation, and post-boundary continuation, but do not establish corpus-wide frequencies.A visible self-correction is not independent verification of the entire controller, and late footage does not count as success under the retained budget.Source: F Observable Behavioural Patterns · p. 57

F.1 What counts as using feedback

  • A fresh observation, an explicit image view, and an action change after feedback are distinct events; the Gemini sequence links a rejection to a revised lift and confirmed execution.A low image-view count alone does not show that an agent ignored images already supplied in robot feedback.Source: §F.1 · p. 57, F Observable Behavioural Patterns · p. 57

F.2 How to read agent claims

  • Tool notes and messages record intent or interpretation, while retained evaluator fields establish outcomes such as success or failure.Astra’s stacking result is retained as success despite missing terminal client feedback for the final release request.Source: §F.2 · p. 57

G Matched-Task Comparisons

Matched-task outcomes support observational comparisons, not causal claims about interaction strategies; selected execution traces show how failures can arise and how budget boundaries affect scoring.

G Matched-Task Comparisons

  • The five-model snapshot compares matched tasks observationally, with provider, collection-time, and build-metadata differences remaining possible confounders.It does not isolate the effects of history, code control, or reasoning budget.Source: G Matched-Task Comparisons · p. 58

G.1 Success overlap

  • Twenty-one unique tasks are solved by at least one model, and the overlap matrix does not assume independent model errors or an executable selection oracle.Source: §G.1 · p. 58

G.2 Astra and Opus 5.5 on shared successes

  • Eight tasks are successful for both Astra and Opus 5.5, but tool-request counts and retained result steps measure different costs, not time to first success.A request can hold an action for a longer segment, while auxiliary analysis can increase total calls without advancing physics.Source: §G.2 · p. 58, Table 16 · p. 58

H Failure Cases and Outcome Adjudication

  • Failure adjudication distinguishes post-interaction failure, exhaustion before robot execution, and disagreement between historical outcomes and the scored budget boundary.The examples clarify the evidence chain, not an exhaustive taxonomy or a measured distribution of failure causes.Source: H Failure Cases and Outcome Adjudication · p. 59

H.1 Kimi: repeated grasp revision without a completed pour

  • Kimi noticed that contact had knocked the bottle over, but repeated closure revisions and later arm switching never established a stable grasp before failure at step 400.The trace supports failed acquisition and budget exhaustion, not a claim that Kimi declared the pour complete.Source: §H.1 · p. 59, Figure 24 · p. 59

H.2 DeepSeek: analysis consumes the budget before any action

The failure cases show analysis without robot execution, limits on attributing causes from scored outcomes, and a mismatch between historical success and budget-limited scoring. The adjudication ledger records how such boundaries change retained outcomes.

H.2 DeepSeek: analysis consumes the budget before any action

  • DeepSeek made zero robot requests and confirmed control steps while preparatory scripts and image inspection consumed the interaction limit before execution.The terminal ledger counts 15 charged auxiliary units, while the visible trace contains 14 events and flags incomplete accounting.Source: §H.2 · p. 60

H.3 Attribution limits

  • A scored label alone cannot identify whether a malformed request, planner rejection, dropped event, simulator failure, or unmet task predicate caused the outcome.The raw response, converted request, validation receipt, and environment state must be traced before assigning a more specific cause.Source: §H.3 · p. 60
  • The 14 visible auxiliary events versus 15 charged units is an accounting limitation, not evidence of an unlogged physical action.Source: §H.3 · p. 60, §H.2 · p. 60

H.4 Opus 5.5: late success beyond the scored boundary

  • Task-82 is a retained failure at step 313 under the non-action limit, despite historical success after 602 executed steps and merged events through step 601.The later stacking work lies beyond the scored budget boundary.Source: §H.4 · p. 61, Figure 25 · p. 61

H.5 Complete adjudication ledger

  • Twenty-one records are marked adjudicated: nine missing an original Boolean success value are assigned retained failure, and three historical successes become failures at an earlier budget boundary.The ledger preserves unavailable original values rather than silently treating them as native failures; some records also have incomplete invocation accounting or lack terminal client feedback.Source: §H.5 · p. 62

Related papers

Loading 2610.10409v1…