PaperlayerView exact revision on arXiv

Source-linked AI summary

VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control

Zhongbo Zhang, Jiayi Jin, Yifan Wang, Zaibin Zhang, Haiwen Diao, Lijun Wang, Huchuan Lu

arXiv:2609.19554v1cs.ROcs.CV

TL;DR

Under partial observation, spatial reasoning must support acquiring missing evidence and acting in a common spatial frame, yet existing evaluations do not jointly test evidence acquisition, inference, and execution. VA-Bench evaluates general-purpose MLLMs using visual demonstrations, active views, metric robot commands, and execution feedback. Across its task families, the best macro-average success is 53.93±3.17%, and summary transfer and long-horizon composition remain limited.

  • Problem

    Under partial observation, agents must acquire missing evidence and interpret it in a common spatial frame, while existing benchmarks leave evidence acquisition, inference, and execution coupled.

    Source: §1 · p. 2

  • Method

    VA-Bench evaluates MLLMs that use demonstration-derived summaries to select views and generate metric robot commands, without privileged poses, oracle trajectories, or learned action heads.

    Source: §1 · p. 3, §3.4 · p. 6

  • Results

    Across 14 task families and three runs, the best macro-average success is 53.93±3.17%, with strong localization but less consistent spatial relations and weak online correction.

    Source: §6 · p. 11

  • Takeaways & Limitations

    VA-Bench characterizes limitations in closed-loop competence, including mismatch between spatial diagnostics and execution, geometric-transfer degradation, and failure to complete long-horizon composition.

    Source: §1 · p. 3

    End-to-end outcomes combine summary quality, live-scene re-grounding, and action generation, so they do not isolate summary-independent online control.

    Source: §B.3 · p. 25

Abstract

from arXiv · show

Spatial intelligence requires more than describing object locations. Under incomplete observation, models must identify and acquire missing evidence, interpret it in a common spatial frame, and act on it. We introduce VA-Bench to evaluate the complete observe-reason-act-revise loop. General-purpose MLLMs learn procedural context from RGB-only demonstrations, actively select camera viewpoints, issue metric Cartesian commands, and revise them from execution feedback. Models receive no privileged object poses, oracle trajectories, or learned action heads. A fixed model-agnostic controller executes only model-specified targets. VA-Bench contains 14 base task families (11 single-arm and three dual-arm), seven held-out geometry/layout variants, and a long-horizon five-object composition track. We evaluate 12 primary model conditions in three independent runs over the same 20 physically verified seeds per base task, reporting terminal success, nine trajectory-level behavioral diagnostics, and subtask progress. First, the best-performing model scores 100.0% on target localization and 78.9% on spatial relations in the annotated run. Its three-run macro-average task success is only 53.93+/-3.17%. Second, active camera control significantly improves task success over passive multi-view observation. In one matched comparison, success rises from 27.86% to 57.50%. Third, held-out geometric transfer can reduce task success by over 30 percentage points. No model completes a strict long-horizon episode, despite substantial partial progress. VA-Bench thus tests whether general-purpose MLLMs can turn visual demonstrations and actively acquired evidence into successful embodied action.

1 INTRODUCTION

Under partial observation, spatially capable agents must acquire missing evidence and interpret it in a common frame before acting. VA-Bench evaluates whether general-purpose MLLMs can turn visual demonstrations and active observation into executable, revisable robot actions.

1 INTRODUCTION

  • VA-Bench jointly tests evidence acquisition, spatial inference, and executable action in a closed loop, addressing gaps left when these capabilities are evaluated separately.A correct relation description may still be too imprecise for contact, and predefined views or skills do not test whether models resolve missing evidence or generate metric motion.Source: §1 · p. 2
  • VA-Bench uses RGB demonstrations for procedural context, then requires MLLMs to select viewpoints, issue metric Cartesian commands, and revise actions from feedback without privileged poses or oracle trajectories.A fixed model-agnostic controller executes only model-specified targets, so evaluation includes instance-specific re-grounding as well as procedure interpretation.Source: §1 · p. 2
  • The benchmark covers 14 task families, seven held-out geometry/layout variants, and a five-object composition track, with nine behavioral diagnostics and subtask-progress measures.Its 12 primary model conditions are evaluated in three independent runs over the same 20 physically verified seeds per base task.Source: §1 · pp. 2–3
  • The best-performing model reaches 100.0% on target localization and 78.9% on spatial relations, yet its three-run 14-task macro-average success is 53.93±3.17%.The evaluation also finds active camera control improves success over passive multi-view observation, while held-out transfer and long-horizon composition remain challenging.Source: §1 · p. 2

2 RELATED WORK

Prior work studies spatial reasoning, embodied manipulation, active perception, and demonstration-based control, but often evaluates them separately or relies on structured inputs or learned policies. VA-Bench combines active evidence acquisition, metric arm control, and execution feedback for general-purpose MLLMs.

2.1 SPATIAL REASONING UNDER PARTIAL OBSERVATION

  • Spatial benchmarks range from static-image and supplied multi-view reasoning to interactive exploration, but interactive work primarily evaluates beliefs, semantic decisions, or high-level outcomes rather than metric end-effector control.VA-Bench requires one model to acquire task-relevant visual evidence, translate it into metric arm motion, and revise decisions from execution outcomes.Source: §2.1 · p. 3

2.2 EMBODIED MANIPULATION AND ROBOT BENCHMARKS

  • Embodied manipulation benchmarks often use structured spatial information or predefined execution components, while learned visuomotor policies rely on specialized action-generation modules.VA-Bench instead tests whether a general-purpose MLLM actively selects observations and grounds them into metric arm commands without supplied object geometry or a learned robot action head.Source: §2.2 · p. 3

2.3 ACTIVE PERCEPTION AND DEMONSTRATION-GROUNDED CONTROL

  • Learned visuomotor work integrates sensing and control, and demonstration-based systems train policies from action-bearing data; VA-Bench tests explicit MLLM decisions in a joint view–arm loop.Its demonstrations provide contextual guidance rather than training signals, and the model must infer reusable procedures and re-ground them in unseen scenes.Source: §2.3 · p. 4

3 VA-BENCH: BENCHMARK DESIGN AND EVALUATION PROTOCOL

VA-Bench evaluates general-purpose MLLMs in a physics-based observe–reason–act–revise protocol, using demonstrations for procedural context and requiring model-generated metric actions. Its tracks and diagnostics measure task success and observable progress across base tasks, transfer, and long-horizon composition.

3 VA-BENCH: BENCHMARK DESIGN AND EVALUATION PROTOCOL

  • The standard protocol evaluates whether an MLLM infers task context from visual demonstrations, acquires evidence, generates metric robot commands, and revises them from execution feedback.Each episode is scored by a physics-based task-success predicate.Source: §3 · p. 4

3.1 EVALUATION TRACKS AND TASK SUITE

  • The suite has 14 base task families with 20 verified scene seeds each, seven held-out single-arm transfer families, and a separate five-object basket composition track.Transfer variants change geometry, container or box instance, or layout; the basket task combines four atomic skills and requires sustained progress tracking.Source: §3.1 · p. 5

3.2 INTERACTIVE EPISODE PROTOCOL

  • Each episode fixes a task, scene seed, instruction, demonstration context, action budget, and success predicate, while camera actions alter only the observation pose and robot actions alter physical state.Camera and robot actions share the budget; an episode ends on success, model stop, or budget exhaustion.Source: §3.2 · p. 5

3.3 DEMONSTRATION-CONDITIONED RE-GROUNDING

  • For base tasks, models derive a textual procedure from sampled RGB frames and must re-ground it in a new seed without machine-readable actions, poses, or a metric solution.Transfer variants reuse the base summary, while the long-horizon task supplies four atomic-skill videos but no full-task demonstration.Source: §3.3 · p. 5

3.4 OBSERVATION, ACTION, AND PRIVILEGE BOUNDARY

  • The model receives RGB observations and filtered proprioception, generates numerical robot targets and camera adjustments, and is withheld object geometry, poses, task waypoints, and other evaluator-only signals.Robot translations and rotations are metric targets generated by the model, while camera actions select bounded semantic viewpoints and fixed-step adjustments.Source: §3.4 · p. 6

3.5 AGENT HARNESS AND FIXED EXECUTION

  • A shared harness validates and dispatches commands, while a fixed executor checks reachability and executes trajectories; the MLLM selects objects, grasp regions, roles, waypoints, and corrections.The primary direct-control condition uses no learned RGB-D or pre-grasp spatial module.Source: §3.5 · p. 6

3.6 TASK CONSTRUCTION AND INSTANCE VALIDATION

  • Task families must have a visually recoverable spatial bottleneck, support direct control and deterministic checking, and expose subtask checkpoints; all 280 retained instances pass expert execution to success.The suite is fixed before evaluation and reused unchanged across models and runs.Source: §3.6 · p. 6

3.7 OUTCOMES AND BEHAVIORAL DIAGNOSTICS

  • Terminal success is the primary outcome, complemented by nine human-annotated trajectory criteria spanning spatial perception, manipulation, and error recovery.Error detection, online correction, and post-failure adjustment apply only when an observable error occurs; the other six criteria apply to every annotated base episode.Source: §3.7 · pp. 6–7

4 MAIN RESULTS AND ANALYSIS

The evaluation compares 12 model conditions across base-task success, subtask progress, and trajectory-level diagnostics. The top three conditions have similar overall success, while diagnostic profiles and performance across arm types vary.

4.1 BASE-TASK PERFORMANCE

  • 68.57%: Opus-5 leads annotated-run micro-averaged subtask completion, ahead of Qwen3.8-max at 66.07% and GPT-5.6-sol at 65.89%.The progress measure uses checkpoint-level weighting and is not an upper bound on task-family success.Source: §4.1 · pp. 7–8, Figure 2 · p. 7
  • 23.10±1.25%: GPT-5.6-terra is well below the top three, every remaining condition reaches at most 15.00%, and the best dual-arm mean is 22.22%.Only the top three conditions exceed a 10% mean on dual-arm tasks.Source: §4.1 · p. 8
  • 53.93±3.17%: Qwen3.8-max has the highest three-run 14-task mean, but overlapping ranges mean the authors claim no reliable ordering among the top three.Qwen3.8-max leads the single-arm mean at 65.61%, while GPT-5.6-sol leads the dual-arm mean at 22.22%.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 INTERACTION AND FAILURE ANALYSIS

  • 100.0%: The top three score this on target localization and at least 99.6% on manipulation semantics, while their spatial-relation scores range from 70.4% to 78.9%.The authors note that identifying targets and selecting meaningful motions does not establish accurate robot–object geometry.Source: §4.2 · p. 8
  • 83.2%: Opus-5 leads active exploration and scores 77.9% on fine-grained pre-contact analysis, which records analysis before contact rather than successful completion.GPT-5.6-sol and Qwen3.8-max score 74.6% and 73.9% on fine-grained pre-contact analysis, respectively.Source: Figure 2 · p. 7, §4.2 · p. 8
  • 46.7%: Qwen3.8-max leads online correction, while error detection is similar across Qwen3.8-max, GPT-5.6-terra, and Opus-5 at 73.6%, 72.6%, and 69.9%.Gemini-3.6 leads post-failure performance at 61.9% despite near-zero online correction; detecting errors does not imply correcting or replanning after failure.Source: §4.2 · p. 8
  • 95.7%: The three annotators agree unanimously on applicable labels, with Fleiss κ = 0.942.Terminal success remains the primary outcome; the behavioral diagnostics show where behaviors occur or stop within a trajectory.Source: §4 · p. 7, §4.2 · p. 8

5 GENERALIZATION AND ABLATION

VA-Bench extends evaluation beyond base tasks to active versus passive observation, held-out re-grounding, long-horizon composition, and demonstration-summary transfer.

5.1 ACTIVE EVIDENCE ACQUISITION VS. PASSIVE MULTI-VIEW INPUT

  • Removing camera control reduces success by 9.64 to 29.64 points across four conditions, while passive observation performs worse on most tested task families.The comparison changes only the camera interface; Table 4 reports pooled active-versus-passive results for four conditions.Source: §5.1 · p. 9, Table 4 · p. 9
  • On held-out geometry and layout variants, Qwen3.8-max retains 67.86% success with a 10.00-point drop, while GPT-5.6-sol loses 32.14 points.The transfer track changes geometry, container or box instance, or layout while preserving the instruction; Table 5 also reports the composition track.Source: §5.1 · p. 9, Table 5 · p. 10
  • No model completes a strict long-horizon episode; Qwen3.8-max records 61/100 successful placements and GPT-5.6-sol 53/100.The five-object task combines four independently demonstrated atomic skills without a full-task demonstration.Source: §5 · p. 8, §5.1 · p. 10, Table 5 · p. 10
  • Under controlled transfer, spatial-perception and understanding scores remain at or above 99.3%, while robot-manipulation scores range from 54.3% to 86.7%.High perception scores coexist with lower manipulation scores and incomplete task execution.Source: §5.1 · p. 10

5.2 DEMONSTRATION-SUMMARY ABLATION

  • Qwen3.8-max’s summary lowers success for the other four models by 31.25 points on average, while the Opus-5 summary improves theirs by 7.50 points on average.The model with the highest task completion does not produce the most transferable summary; the summary contents suggest explicit constraints may help explain the difference.Source: §5.2 · pp. 10–11
  • Under the Opus-5 summary, Gemini-3.6 rises from 23.33% to 35.00% and GPT-5.6-terra from 41.67% to 55.00%, with gains concentrated on placement.The comparison covers three tasks in a single run, so the summary-source ordering is a suggestive trend rather than a precise estimate.Source: §5.2 · p. 11

5.3 SUMMARY-GUIDED LOCAL SMALL-MODEL EVALUATION

  • With Qwen3.8-max summaries, locally deployed models achieve at most 9.64% success, and no condition completes a placement or dual-arm task.The corresponding unguided controls for Gemma-4-31B-it and the two Qwen3.6 variants obtain zero success.Source: §5.3 · p. 11

6 CONCLUSION

The conclusion emphasizes that strong localization does not ensure reliable physical action, and that active camera control improves task success over passive observation.

6 CONCLUSION

  • Across 14 task families and three runs, the best macro-average is 53.93±3.17%, despite strong target-localization and manipulation-semantics scores; spatial-relation diagnostics and online correction remain weaker.The conclusion states that trajectory diagnostics show where models stop using spatial evidence effectively during execution.Source: §6 · p. 11
  • Removing camera control reduces terminal success by 9.64 to 29.64 percentage points across tested conditions; Qwen3.8-max falls from 57.50% to 27.86%.Broader passive coverage does not substitute for task-conditioned evidence acquisition.Source: §6 · p. 11

A COMPLETE EVALUATION RESULTS

  • Appendix results tables split per-task base-benchmark results by model group and behavioral diagnostics by module, ordering model rows and columns by 14-task macro-average.Bold blue cells mark the best score within each directly comparable task row or metric column, including ties.Source: A COMPLETE EVALUATION RESULTS · p. 19

A.1 BASE-EVALUATION SETTINGS AND AGGREGATION

  • The primary comparison uses 12 models’ own demonstration-derived summaries and three runs on the same 20 seeds per task; success is macro-averaged over task families.The reported mean and sample standard deviation summarize three run-level macro-averages; diagnostics and subtask progress use one annotated run, and checkpoint-weighted progress is not a relaxed upper bound on terminal success.Source: §A.1 · p. 19

A.2 BASE-TASK SUCCESS RATES

The appendix reports disaggregated base-task success rates across the benchmark’s tasks and primary model conditions.

A.2 BASE-TASK SUCCESS RATES

  • Tables 7–9 report three-run mean and sample standard deviation for each of 14 base tasks across 12 primary model conditions, split into three model groups.Each run uses the same 20 retained seeds per task, and the final row aggregates each run across all 14 tasks before summary statistics are computed.Source: §A.2 · p. 19, Table 7 · p. 20, Table 8 · p. 20
  • Table 10 compares per-task active and passive successes out of 20 seeds, using each condition’s own summary and identical seeds and execution accounting.The active protocol permits camera actions; the passive protocol supplies five predefined views without camera control.Source: Table 10 · p. 21

A.3 BASE-TASK BEHAVIORAL DIAGNOSTICS

The base-task diagnostics record observable behaviors and recovery criteria, while the held-out section reports controlled transfer setups and descriptive module profiles.

A.3 BASE-TASK BEHAVIORAL DIAGNOSTICS

  • ED, OC, and PF are scored only when an observable error occurs; the other six criteria apply to every annotated base episode.Episodes without an observable error are NA and excluded from the ED, OC, and PF denominators; terminal success remains the primary outcome.Source: §A.3 · p. 21
  • Tables 11–14 report pooled or component behavioral diagnostics for the twelve primary conditions, with error-free episodes excluded from recovery percentages.The tables cover pooled trajectory-level scores, perception and understanding, manipulation, and error recovery.Source: Table 11 · p. 22, Table 12 · p. 22, Table 13 · p. 22

A.4 GENERALIZATION AND LONG-HORIZON DIAGNOSTICS

  • Held-out single-arm transfer rates use 20 seeds and reuse the corresponding base-task demonstration summary.Table 16 gives equal-weight module means as descriptive profiles, not an overall ranking; SPU, RM, and ER average perception/exploration/relations, manipulation, and recovery criteria, respectively.Source: Table 15 · p. 23, §A.4 · p. 23, Table 16 · p. 23

A.5 DEMONSTRATION-SUMMARY SOURCE ABLATION

The appendix reports summary-source comparisons for Qwen3.7-plus and cross-model shared summaries, with macro-average and per-task breakdowns.

A.5 DEMONSTRATION-SUMMARY SOURCE ABLATION

  • The Qwen3.7-plus summary-source comparison reports macro-averages over 11 single-arm tasks, three dual-arm tasks, and all 14 tasks.Table 17 provides the subset breakdown for the matched comparison summarized in Section 5.Source: Table 17 · p. 23, §A.5 · p. 23
  • The shared-summary comparison reports per-task success over the same 20 verified seeds as the corresponding first base-evaluation run, changing only the demonstration summary.Each summary author’s own-summary block reproduces its base run exactly.Source: §A.5 · p. 23

A.6 LOCALLY DEPLOYED SMALL-MODEL RESULTS

The appendix reports locally deployed models under summary guidance and limits what the shared-summary and generalization comparisons establish.

A.6 LOCALLY DEPLOYED SMALL-MODEL RESULTS

  • Without summary guidance, Gemma-4-31B-it and both Qwen3.6 variants score 0.00% on every task.Table 20 reports per-task rates for summary-guided locally deployed models, with 20 trajectories per task.Source: §A.6 · p. 23, Table 20 · p. 24
  • Table 19 reports summary-guided success as counts and rates, with denominators of 220 single-arm, 60 dual-arm, and 280 overall trajectories.Table 18’s per-task comparison covers three tasks and uses shared demonstration summaries.Source: Table 19 · p. 24, Table 18 · p. 24

B.1 WHAT THE DIAGNOSTICS AND PROTOCOL COMPARISONS IDENTIFY

  • The cross-model summary comparison varies the source over three tasks in a single run, and summary style differences prevent isolating constraint explicitness as the source of the observed effect.The source matters, but the ordering of the three sources differs from the ordering of their authors’ task success.Source: §B.1 · p. 25

B.2 GENERALIZATION SCOPE

  • Held-out variants are finite and controlled, so they do not establish open-set generalization across arbitrary objects, robots, cameras, or real-world calibration.The long-horizon result covers one compositional family, and the local-model comparison covers four summary-guided deployments.Source: §B.2 · p. 25

B.3 LIMITATIONS: DEMONSTRATION-SUMMARY CONDITIONING

The primary results combine summary quality with live grounding and action generation, and the benchmark evaluates direct metric control rather than modular VLM–VLA control. Its protocol specifies the task instance, observations, available actions, and the passive-view comparison.

B.3 LIMITATIONS: DEMONSTRATION-SUMMARY CONDITIONING

  • Primary outcomes combine each model’s demonstration summary with its live-scene re-grounding and action generation, rather than measuring summary-independent online control.Separating these factors would require no-summary and expert-written-summary controls across multiple models on the full suite.Source: §B.3 · p. 25

B.4 LIMITATIONS: MODULAR VLM–VLA CONTROL

  • The benchmark evaluates direct metric robot and camera commands through a fixed harness, not a VLM that instructs a separately trained VLA.The results therefore do not measure benefits or failure modes of a VLM–VLA interface, learned action policy, or division of control responsibilities.Source: §B.4 · p. 25

C.1 INTERACTIVE TASK FORMULATION

  • An evaluation instance is defined by task family, scene seed, goal, optional observation-only demonstration, action budget, and a physics-based success predicate.The seed determines the scene but supplies no privileged geometry to the model.Source: §C.1 · p. 26
  • At each decision, the model receives the current RGB image, filtered proprioception, interaction history, and prior validity and planning outcomes.The simulator’s physical state is unobserved.Source: §C.1 · p. 26
  • Robot actions update physical state, whereas camera actions update only the observation pose; both consume the task-dependent action budget.The episode ends on success, model stop, or budget exhaustion.Source: §C.1 · p. 26
  • The passive comparison replaces active camera control with five predefined RGB views and removes camera actions, while keeping robot actions, history limits, seeds, and scoring the same.The model cannot select, reorder, or request another view.Source: C BENCHMARK SPECIFICATION DETAILS · p. 25, §C.1 · p. 26
  • The model receives clean RGB frames and robot-state proprioception but no depth, object poses, oracle grasp points, task waypoints, or numerical camera calibration.The robot numerical state anchors the metric reference frame; task-relevant object geometry must still be inferred from RGB and interaction.Source: §C.1 · p. 26

C.2 TASK-LEVEL SPATIAL DEMANDS

VA-Bench frames embodied spatial competence as a set of coupled demands: acquire useful visual evidence, ground it in metric robot–object relations, and act and revise across task stages. Its protocols pair procedural demonstrations with physically verified task instances, bounded action interfaces, and trajectory-level behavioral diagnostics.

C.2 TASK-LEVEL SPATIAL DEMANDS

  • The seven task-level demands span active evidence acquisition, cross-view state tracking, metric grounding, contact geometry, action sequencing and revision, bimanual coordination, and demonstration-conditioned re-grounding.These are suite design dimensions, not extra scores or independently identified latent variables; end-to-end outcomes also depend on numerical action generation, the fixed motion planner, and simulated dynamics.Source: §C.2 · pp. 26–27

C.3 TASK-FAMILY COVERAGE

  • The 14 base families cover 11 single-arm and three dual-arm tasks, while a separate five-object basket task tests long-horizon composition.Seven base families also have paired held-out variants that reuse their base-task demonstrations while changing geometry, instances, or layout.Source: §C.3 · pp. 27–28

C.4 SEED CONSTRUCTION AND PHYSICAL VERIFICATION

  • All 280 retained seeds have a matching ready=true probe with the full planner chain and terminal checks passing, certifying end-to-end physical solvability from robot home.The 20 accepted seeds per task are drawn from ordered candidate pools, which are screened until the per-task target is met.Source: §C.4 · p. 28

C.5 METRIC ROBOT ACTION SPACE

  • The robot interface exposes numerical Cartesian target changes rather than coarse motion labels or low-level motor commands.A fixed IK and trajectory layer attempts model-specified metric targets; it does not choose the object, grasp region, arm role, or task waypoint.Source: §C.5 · p. 30, Table 26 · p. 30

C.6 BOUNDED ACTIVE CAMERA ACTION SPACE

  • Active camera actions provide bounded, gripper-centered semantic views and camera-frame adjustments that change observation pose without changing world-frame robot controls.The interface evaluates viewpoint selection rather than physical camera-arm motion, reachability, collision, or continuous trajectory cost; camera actions consume episode steps.Source: §C.6 · pp. 30–31, Table 27 · p. 31

C.7 PASSIVE FIVE-VIEW OBSERVATION PROTOCOL

  • The passive control supplies five fixed RGB views at matched control points, whereas the active condition can request further views and has no passive image bundle.Robot-action semantics and environment transitions are matched, making the comparison one of model-directed active acquisition versus passively supplied multi-view evidence.Source: §C.7 · p. 31

C.8 DEMONSTRATION-CONDITIONED EVALUATION PROTOCOL

  • For each standard task, the model derives a structured procedural summary from sampled frames of a successful RGB-only execution, but receives no metric solution for the live seed.At test time it must reacquire the scene and generate numerical parameters; held-out variants reuse the base summary, while the basket task uses four atomic-skill videos and no full-task demonstration.Source: §C.8 · pp. 31–32

C.9 TRAJECTORY-BASED CAPABILITY DIAGNOSTICS

  • The diagnostic rubric divides trajectory competence into spatial perception and understanding, robot manipulation, and error recovery, with three behavioral criteria per module.Complete episodes are reviewed by three annotators; ED, OC, and PF are gated on observable errors, while the figure’s examples indicate that final placement alone does not establish pre-contact analysis.Source: §C.9 · pp. 32–33, Figure 3 · p. 33

C.9.1 ANNOTATION RELIABILITY

The paper reports high overall agreement for its trajectory annotations and checks how aggregation choices affect scores. It also distinguishes observable behaviors within trajectories from terminal success, with separate criteria for perception, manipulation, and recovery.

C.9.1 ANNOTATION RELIABILITY

  • 95.69% of applicable labels were unanimous, with Fleiss κ = 0.942 across the annotated episodes.Every disagreement was a 2–1 split, and reported labels follow the majority vote.Source: §C.9.1 · pp. 33–34, Table 28 · p. 34
  • Across three voting rules, the largest within-condition range in nine-criterion means was 6.25 points, and condition rankings remained similar.Individual criteria were more sensitive: FG leadership differed between unanimity and majority voting.Source: §C.9.1 · p. 34
  • FG agreed with terminal success on 82.08% of the matched episodes, while TL and MS agreed on 31.1% and 30.7%, respectively.The record includes FG-positive failures and FG-negative successes, so its labels are not simple restatements of the checker outcome.Source: §C.9.1 · p. 34

C.9.2 SPATIAL PERCEPTION AND UNDERSTANDING

  • Spatial criteria distinguish target localization, informative evidence acquisition and use, and correct interpretation of robot–object height, distance, direction, and alignment.Informative exploration requires a view responding to ambiguity, an observation that reduces it, and a subsequent action or estimate consistent with the new evidence.Source: §C.9.2 · p. 35

C.9.3 ROBOT MANIPULATION

  • Manipulation criteria separate purposeful use of commands, coherent goal-directed metric sequencing, and fine-grained pre-contact analysis.Pre-contact analysis is scored independently of task completion; success alone does not establish it.Source: §C.9.3 · pp. 35–36

C.9.4 ERROR RECOVERY

  • Recovery criteria distinguish recognizing an error, correcting a developing deviation before failure, and materially revising an attempt after explicit failure.Recovery scores use the shared observable-error set, while other criteria use episodes with a valid assessment opportunity; module means are descriptive, not an overall capability score.Source: §C.9.4 · p. 36

D AGENT HARNESS IMPLEMENTATION DETAILS

The harness keeps the MLLM as the only model-dependent control component and routes its decisions through a persistent, validated tool loop. It supplies observations and execution feedback while a fixed layer executes model-specified targets, and records environment-checked outcomes.

D AGENT HARNESS IMPLEMENTATION DETAILS

D.1 PERSISTENT DECISION LOOP

  • At each decision, the harness requests one structured model output, validates and routes it, and returns tool results in the same thread without advancing the simulator.Robot and camera actions undergo schema and range checks; invalid outputs receive bounded repair attempts.Source: §D.1 · pp. 37–38

D.2 MODEL OBSERVATIONS AND ACTIONS

  • The model receives current RGB observations, robot self-geometry, action validity, and binary planning feedback, but not target-object poses, oracle grasp points, or waypoints.It supplies bounded world-frame translations, gripper-local rotations, and gripper commands; the fixed IK and trajectory layer attempts requested motions without choosing semantic targets.Source: §D.2 · p. 38

D.3 DEMONSTRATION AND PROMPT CONDITIONING

  • The model derives a task summary from sampled frames of a successful RGB demonstration, which conveys procedure but no machine-readable actions or metric solution for the live scene.Model-specific summaries are the primary condition; a shared-summary condition is a controlled diagnostic, and task scaffolding includes instructions and safety constraints.Source: §D.3 · pp. 38–39

D.4 EXECUTION RECORDS AND OUTCOME

  • Reported success is the environment-defined physics-simulated outcome, not a visual judgment by the MLLM or harness.The end-to-end outcome also depends on the fixed planner, action interface, and simulator dynamics.Source: §D.4 · p. 39

Related papers

Loading 2609.19554v1…