PaperlayerView exact revision on arXiv

Source-linked AI summary

VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation

Siyi Chen, Hugo Hadfield, Alex Zook, Mikaela Angelina Uy, Chan Hee Song, Erwin Coumans, Xuning Yang, Faisal Ladhak, Qing Qu, Stan Birchfield, Jonathan Tremblay, Valts Blukis

arXiv:2606.07723v1cs.RO

TL;DR

Open-vocabulary long-horizon manipulation requires robots to reason over complex instructions and scenes while planning, monitoring, and recovering during execution. VoLoAgent treats a VLA/WAM as an interruptible tool within a VLM-managed loop and substantially outperforms single-model, code-as-policy, and TAMP baselines across RoboVoLo suites.

  • Problem

    Open-vocabulary long-horizon manipulation requires planning, language and spatial reasoning, world knowledge, scene memory, embodied perception, and precise actions.

  • Method

    VoLoAgent uses a VLM-managed closed loop to orchestrate an interruptible VLA/WAM alongside perception models and grasp/place primitives.

  • Results

    VoLoAgent outperforms single-model, code-as-policy, and TAMP baselines on every RoboVoLo suite, with gains of +38.9% to +2.1% over the strongest suite-specific baselines.

  • Takeaways & Limitations

    The results support physical orchestration as an effective approach for open-vocabulary long-horizon manipulation, with orchestration driving the gains in ablations.

  • Takeaways & Limitations

    Cloud-VLM latency of approximately 1–5 seconds can limit reaction time and miss fast failures, and the system was demonstrated only on a single-arm parallel-jaw manipulator.

Abstract

from arXiv · show

Open-vocabulary long-horizon manipulation requires robots to reason over flexible instructions and complex multi-object scenes while adaptively planning, executing, monitoring, and recovering from failures. We address these demands with a closed agent loop in which a VLM orchestrates heterogeneous robot capabilities as interruptible tools. Unlike in virtual AI agents, the timing of decisions, actions and tool calls is important in a physical world that does not pause for reasoning. We refer to this setting as Physical Orchestration, and propose VoLoAgent, a VLM that plans, monitors, and recovers by treating a VLA/WAM as an interruptible tool it steers mid-rollout alongside vision models and action primitives. To evaluate these long-horizon capabilities, we introduce RoboVoLo, a high-fidelity benchmark for open-vocabulary long-horizon manipulation across common sense, memory/state tracking, complex references, and world knowledge, with both task-level success and failure-mode diagnostics. Experiments show VoLoAgent substantially outperforms single VLA/VLM or tool-based systems, with validation on real-robot experiments. Project page: https://chicychen.github.io/VoLo/

1. Introduction

Open-vocabulary long-horizon manipulation requires robots to understand flexible references, plan across multiple objects, monitor subgoals, and recover from failures. VoLo frames this setting as physical orchestration, introducing VoLoAgent and RoboVoLo to combine adaptive tool use with evaluation of these capabilities.

  • Motivation: Open-vocabulary long-horizon tasks require negative-reference understanding, multi-object planning, subgoal monitoring, and recovery from mistakes.The example instruction excludes specific objects while requiring sequential placement and correction after picking the wrong item.
  • Limitations of existing approaches: End-to-end VLA/WAM approaches provide precise manipulation but lack robust planning, monitoring, and perception in typical multi-object scenes.Code-as-policy methods support explicit reasoning but are limited by fixed toolsets and control APIs for contact-rich manipulation.
  • Physical orchestration: Physical orchestration treats manipulation as decisions made while the world keeps moving, rather than pausing during reasoning.VoLoAgent unifies a VLA/WAM, perception models, and grasp/place primitives as callable tools in a flexible VLM-managed loop.
  • Benchmark: RoboVoLo is a high-fidelity benchmark designed to study open-vocabulary long-horizon manipulation, state tracking, and adaptive recovery in complex scenes.It spans common sense, memory, complex references, and world knowledge, addressing limitations of benchmarks focused on short-horizon skills or simplified scenes.
  • Contributions: RoboVoLo contains 126 tasks spanning common sense, memory, references, and world knowledge, while VoLoAgent composes an interruptible VLA/WAM with perception models and classical action primitives.The benchmark is designed independently of the system, and the agent uses a VLM to plan, reason, monitor, and recover in a single closed loop.

2. Related Work

Prior work includes end-to-end VLAs and WAMs, agentic and hierarchical robot frameworks, and benchmarks for long-horizon open-vocabulary manipulation. These approaches cover direct action prediction, programmatic or verified reasoning, task-and-motion planning, and diverse manipulation settings, while some frameworks remain limited by fixed primitive interfaces and limited attention to real-time monitoring.

  • Vision-Language-Action and World Action Models: VLAs map observations and instructions directly to robot actions, while WAMs jointly predict future video and actions.
  • Agentic and Hierarchical Robot Frameworks: LLM- and VLM-driven frameworks ground high-level reasoning in robotic primitives through code generation, closed-loop verification, or TAMP-augmented symbolic planning.
  • Agentic and Hierarchical Robot Frameworks: These agentic frameworks remain limited by fixed primitive interfaces and largely overlook real-time monitoring.
  • Long-Horizon Open-vocabulary Manipulation Benchmarks: Manipulation benchmarks cover tabletop, household, kitchen, and language-conditioned long-horizon tasks, alongside real-to-sim suites measuring policy transfer.

3. RoboVoLo Benchmark

RoboVoLo is a high-fidelity benchmark for open-vocabulary, long-horizon manipulation that evaluates reasoning over changing scenes, fine-grained references, world knowledge, and failure-aware execution. It contains 126 tasks across 15 categories organized into four capability suites, supported by an expanded simulation asset library.

  • Benchmark scope: RoboVoLo targets the coupled reasoning-and-execution demands of long-horizon manipulation, including scene grounding, state tracking, reference resolution, world knowledge, monitoring, and recovery.The benchmark is designed to address capabilities that current benchmarks do not isolate.
  • Capability suites: 126 tasks span 15 categories grouped into Common Sense, Memory, Complex References, and World Knowledge capability suites.The suites cover inferring intent from context, tracking state across actions, resolving spatial and linguistic cues, and applying external knowledge.
  • Capability suites: Memory tasks track earlier scene states, while Complex References tasks disambiguate objects using spatial, ordinal, relational, size-based, or negative cues.Examples include restoring arrangements, undoing changes, swapping objects, and rearranging objects relative to their initial configuration.
  • Simulator: The simulator adds 501 new objects to RoboLab, including 247 household assets and 254 task-specific assets for chemistry, geometric art, and mathematical manipulation tasks.The task-specific assets include 118 periodic-table element cubes, 120 geometric art objects, and 16 wooden math cubes.

4. VoLoAgent and Physical Orchestration

VoLoAgent frames physical orchestration as a closed loop in which one VLM plans, monitors, interrupts, and redirects heterogeneous tools while the world continues changing. Its design combines fast monitoring, deliberative planning, safety-aware idling, and complementary visuomotor, perception, and action tools.

  • Physical Orchestration: Physical orchestration requires detecting divergence, halting in-flight actions, and redirecting through replanning, reissuing actions, or switching tools.Unlike virtual agents, physical agents must reason while the world keeps moving.
  • VoLoAgent: VoLoAgent uses one VLM to plan subtasks, monitor execution, and continuously decide whether to continue, switch tools, advance, or recover.The VLA is treated as a callable tool rather than a separately controlled executor.
  • Control Loop: The agent combines a fast monitor context with fuller deliberation at planning points and holds the robot still when reasoning must continue mid-task.Monitoring reads the latest observation with memory, while safety-aware idling pauses active execution during recovery reasoning.
  • Tool Orchestration: VoLoAgent orchestrates VLA/WAM rollouts, open-vocabulary perception models, and grasp/place primitives as complementary tool families.Perception tools support detection and segmentation, while action primitives combine perception, GraspGen, and inverse kinematics.
  • Recovery: During recovery, the agent can resume, replan, rewrite the VLA subgoal, or invoke grasp/place primitives on perception-grounded targets.This routing allows the same agent to recover without a fixed planner–executor split.

5. Experimental Results

VoLoAgent achieves the strongest long-horizon open-vocabulary manipulation performance across RoboVoLo suites, combining improved task success with substantially better failure recovery. Ablations and real-robot trials further show that its full agent loop benefits from coordinated VLA, perception, monitoring, and recovery components.

  • Simulation benchmarks: VoLoAgent outperforms single-model, code-as-policy, and TAMP baselines on every evaluated suite.The comparison includes standalone action-model policies, CaP-X code-as-policy variants, and TiPToP TAMP + VLM.
  • Qualitative analysis: The full system combines VLA steering with grasp-tool recovery, repositioning the gripper after a wrong-object selection so contact-rich manipulation can continue.The component behaviors are illustrated in representative long-horizon tasks.
  • Failure analysis: VoLoAgent has 5× more failure-free episodes than π0.5, 20 vs. 4, and recovers from 54% (38/70) of failed episodes versus 13% (11/86).The analysis traces episodes through world failures, recovery events, and outcomes.
  • Failure analysis: Completion-monitor errors exceed 67% of VLM failure events and increase 4.3× across VLM capability, while planning errors remain at ≤9 events per 90 episodes.Failure-monitor errors are also substantial for most backends, with GPT-5.5 at 44, Gemini 2.5 Flash at 81, and Qwen3-VL-8B at 89.
  • Real-robot validation: 42.9% real-robot success versus 14.3% for π0.5 represents a 3× improvement across 168 rollouts spanning 14 RoboVoLo tasks.Each task used 3 matched-initial-state trials for π0.5, VoLoAgent variants, and full VoLoAgent.

6. Conclusion and Limitations … B. Agentic System — Tools and API

VoLoAgent unifies heterogeneous manipulation capabilities in a VLM-managed closed loop and is evaluated with the 126-task RoboVoLo benchmark. The framework’s main limitations concern monitoring latency and embodiment scope, while supplementary sections document future directions, implementation details, prompts, benchmarks, simulation, statistics, and extended analyses.

  • 6. Conclusion and Limitations: VoLoAgent unifies VLA/WAM rollouts, perception models, and grasp/place primitives in a VLM-managed closed loop for open-vocabulary long-horizon manipulation.The RoboVoLo benchmark contains 126 tasks, and ablations attribute performance gains to orchestration.
  • 6. Conclusion and Limitations: Completion monitoring accuracy is a key improvement target because cloud-VLM calls take approximately 1–5 s and may miss fast failures.The paper calls for fast local monitors to improve reaction time.
  • 6. Conclusion and Limitations: VoLoAgent was demonstrated on a single-arm manipulator with a parallel-jaw gripper, while broader embodiments require retraining or swapping the VLA.The framework is described as supporting bimanual, dexterous-hand, and mobile embodiments.
  • Appendix: The appendix covers future directions, agentic-system implementation, VLM prompt templates, RoboVoLo details, simulation, statistical tests, real-robot breakdowns, other environments, ablations, and Sankey diagrams.These topics are organized across Appendices A–J.
  • A. Future Directions: Future extensions include bimanual, dexterous-hand, mobile-base, and humanoid embodiments, requiring action-primitive re-implementations for new kinematics and contact models.The agent loop and tool API are described as embodiment-agnostic.
  • B. Agentic System — Tools and API: The agent loop can accommodate new simulators or policy backends because a proxy architecture isolates the orchestrator from the evaluation client and VLA.The tool catalog and action-primitive pipelines are specified separately.
  • B. Agentic System — Tools and API: Appendix B specifies the tool catalog and action-primitive pipelines used by the agentic system.These details complement the proxy architecture that separates the orchestrator from evaluation and policy components.

B.1. Proxy Architecture and Extensibility · B.2. Tool Catalog and API · C. VLM Prompts

VoLoAgent is an extensible proxy that decouples evaluation, orchestration, policy, and perception processes through wire-level interfaces, while exposing interruptible tools and configurable simulator, strategy, monitoring, and recovery components. Its fixed-template JSON prompts support planning, monitoring, recovery, and destination localization with state-aware action vocabularies.

  • B.1. Proxy Architecture and Extensibility: VoLoAgent runs eval client, orchestrator, VLA/WAM policy server, and tool server as separate processes, typically isolating dependencies across GPUs and environments.The processes share no Python imports and couple only on the wire, allowing GraspGen and a 14B WAM to coexist without dependency conflicts.
  • B.1. Proxy Architecture and Extensibility: Frontend and Backend interfaces translate different client and policy protocols into one canonical schema, so strategies remain independent of transport implementations.The repository includes OpenPI WebSocket, GR00T ZMQ, OpenVLA REST, and File-IPC protocol modules.
  • B.1. Proxy Architecture and Extensibility: The --env flag adapts image keys, action-chunk lengths, and monitor cadence per simulator, with overrides that support new environments without changing the agent loop, prompts, or tool catalog.This path has been exercised with robolab/IsaacLab, LIBERO, RoboCasa, and VLABench.
  • B.2. Tool Catalog and API: The vla, grasp, and place APIs expose asynchronous policy chunks and natural-language targets, while grasp/place hide perception outputs from the VLM before motion execution.grasp and place ground targets, plan 6-DoF or 3-D motions, solve IK, and stream actions in the same chunk format as vla.
  • B.2. Tool Catalog and API: Both action primitives use damped least-squares IK with null-space joint centering and multi-start seeds, enforcing Franka limits and explicit convergence tolerances.The solver uses damping λ=5×10^-3, null-space gain α=0.5, up to 500 iterations, and retries fixed and random seed variants.
  • C. VLM Prompts: The prompt system uses fixed JSON templates for one-time subgoal planning, per-step monitoring and acting, conditional replanning, and place-tool 2-D point queries.Planning stores an ordered atomic subgoal list in slow-context memory, while replanning selects remaining subgoals from the original list without inventing or rephrasing them.
  • C. VLM Prompts: Monitoring prompts share task, subgoal, and image inputs but expose variant-specific actions: tool-chain control, VLA continuation/replanning, or VLA-plus-tool recovery.The full variant adds grasp_tool for wrong-object recovery and place_tool for wrong-destination recovery, while otherwise preferring continue or replan.
  • C. VLM Prompts: Prompts require visual evidence for completion, distinguish in-progress execution from genuine failure, and use BEFORE/NOW images to preserve state during recovery and camera-frame changes.Front-camera notes account for left/right and front/behind flips and require describing targets by visual features rather than directional words.

D. RoboVoLo Benchmark Details · E. Simulation Setup Details · F. Statistic Test Details

RoboVoLo evaluates open-vocabulary long-horizon manipulation through 126 tasks spanning four diagnostic suites and 15 categories. The simulation uses a DROID-style embodiment and controlled paired trials, while significance claims rely on paired sign-flip tests and Wilson intervals.

  • D. RoboVoLo Benchmark Details: RoboVoLo contains 126 tasks across 15 categories in four suites targeting common sense, memory, complex references, and world knowledge.Each task prevents success through instruction-independent behavior such as always putting everything in one bin.
  • D. RoboVoLo Benchmark Details: Representative task examples show initial scenes and natural-language instructions for each category from the orchestrator’s front-camera view.Figures 8–11 provide two representative initial-scene views per category for each suite.
  • E. Simulation Setup Details: Each task uses T=3 seeded trials with shared canonical scenes and controlled pose, lighting, and distractor randomization for paired system comparisons.Trial k for system A and trial k for system B starts from identical states.
  • E. Simulation Setup Details: Actions are streamed in 8-step chunks at 15 Hz, and orchestrator grasp/place primitives emit the same shape as VLA outputs.This lets the evaluation client treat primitive and VLA actions identically.
  • E. Simulation Setup Details: Three RGB cameras mirror the real DROID cell, while front-camera depth supports grasping and placing; the VLA receives RGB but not depth.The place primitive additionally consumes a Molmo2 2-D point.
  • F. Statistic Test Details: Statistical claims use paired sign-flip randomization tests for simulation results and component ablations, plus Wilson score confidence intervals for real-robot results.These procedures are documented for every significance claim in the main paper.

F.1. Paired Sign-Flip Randomization Test (Main-Results and Component-Ablation tables in the main paper) · F.2. Wilson Score Confidence Intervals

The paper uses paired sign-flip randomization tests for main-result and component-ablation comparisons, with exact or Monte Carlo computation depending on the number of paired tasks. For real-robot results, it reports Wilson score intervals rather than paired ablation significance tests because the sample is small and intervals overlap heavily.

  • F.1. Paired Sign-Flip Randomization Test (Main-Results and Component-Ablation tables in the main paper): The paired test compares two methods across matched-seed task trials using per-task success fractions and the mean per-task difference.Each task has K=3 matched-seed trials, and the statistic is the mean difference in per-task success rates.
  • F.1. Paired Sign-Flip Randomization Test (Main-Results and Component-Ablation tables in the main paper): The two-sided sign-flip p-value treats per-task differences as independently exchangeable in sign, while zero differences contribute zero under every flip.This preserves the standard sign-flip convention and defines the null randomization distribution for the observed mean difference.
  • F.1. Paired Sign-Flip Randomization Test (Main-Results and Component-Ablation tables in the main paper): For N≤24 paired tasks, the test enumerates all 2^N sign assignments; for larger N, it uses B=2×10^5 Monte Carlo sign vectors with an unbiased estimator.The estimator adds one to the numerator and denominator, preventing a zero finite-sample p-value and preserving valid p-value behavior.
  • F.1. Paired Sign-Flip Randomization Test (Main-Results and Component-Ablation tables in the main paper): +32.37 percentage points is the largest reported main-table difference, for MolmoAct2 versus Full, with p<10−4 by Monte Carlo.Other main-table comparisons are also significant at p<10−4, except VoLoAgent (Only VLA), which reports +4.41 and p=0.0599.
  • F.1. Paired Sign-Flip Randomization Test (Main-Results and Component-Ablation tables in the main paper): +6.83 percentage points is reported for VoLoAgent (Only VLA) in the ablation table, with p=0.0466 across the four RoboVoLo suites.The perception swap GDino+SAM2 / Molmo2 reports +3.28 with p=0.3012, and the exterior-camera variant reports +5.28 with p=0.0942; both are non-significant.
  • F.2. Wilson Score Confidence Intervals: Real-robot results use per-system success rates with two-sided 95% Wilson score intervals over n=42 trials per system.Wilson intervals are used because normal approximations are unreliable for small tail counts, such as π0.5 with p̂=0.143.
  • F.2. Wilson Score Confidence Intervals: The paper reports no paired significance test for real-robot ablations because intermediate-variant intervals overlap heavily with Full.More tasks or trials per task would be needed to distinguish the three VoLoAgent variants with adequate statistical power.

G. Real-Robot Setup and Results · H. Other Simulation Environments

VoLoAgent transfers unchanged from simulation to a single physical DROID cell, enabling matched real-robot evaluation across 14 tasks and 168 total trials. The stack was also explored through a common WebSocket proxy on several MuJoCo-based benchmarks and author-designed LIBERO composite tasks.

  • G. Real-Robot Setup and Results: The real-robot platform is a single DROID cell with a 7-DoF Franka Research 3 arm, Robotiq 2F-85 gripper, exterior ZED 2i, and wrist-mounted ZED mini.Its camera extrinsics, lighting, and joint-position plus binary-gripper action space match simulation.
  • G. Real-Robot Setup and Results: The same VLA checkpoints, orchestrator code, and grasp/place primitives run from simulation to hardware without retraining or per-cell calibration.This preserves the simulation-to-real software and capability stack unchanged.
  • G. Real-Robot Setup and Results: 14 tasks were sampled from the four RoboVoLo suites and evaluated with three trials per system under identical reset scenes.Objects, poses, and lighting were matched across systems before each trial.
  • G. Real-Robot Setup and Results: 42 trials per system and 168 trials total formed the real-robot evaluation.These totals follow from 14 tasks × 3 trials and 4 systems × 42 trials.
  • G. Real-Robot Setup and Results: Table 5 reports real-robot per-task success counts for n=3 trials per cell with matched initial states across all four systems.Its column totals match the per-system rates in the main-paper real-robot table.
  • H. Other Simulation Environments: VoLoAgent was explored on LIBERO, RoboCerebra, and VLABench through the same WebSocket proxy.These are widely used MuJoCo-based benchmarks.
  • H. Other Simulation Environments: The authors also created long-horizon composite tasks on LIBERO scenes with literal, vague, and creative phrasings of the same scene-level goal.The author-designed tasks use the same kitchen and desk scenes as LIBERO and instructions resembling the RoboVoLo suites.

H.1. Finding 1: VLAs over-fit to scene, not generalizable to instruction · H.2. Finding 2: vision tools do not transfer to non-photoreal sims

The experiments find that VLAs largely respond to scene configuration rather than instruction changes, while real-world vision tools fail in non-photorealistic MuJoCo-rendered scenes. Language-conditional, broadly trained VLAs and photorealistic rendering instead support transfer in RoboLab, motivating RoboVoLo.

  • H.1. Finding 1: VLAs over-fit to scene, not generalizable to instruction: Binary success was 0/10 for every prompt condition, because 3-step chains exhausted the 700-step budget.PSR therefore measures predicate satisfaction over 24 underlying goal predicates.
  • H.1. Finding 1: VLAs over-fit to scene, not generalizable to instruction: PSR varied only 5 percentage points, from 30% with an empty prompt to 35% with a literal composite prompt.The comparison held the LIBERO scene configuration and BDDL goal predicates fixed while varying only language.
  • H.1. Finding 1: VLAs over-fit to scene, not generalizable to instruction: Removing language still yielded 30% predicate satisfaction, essentially matching the literal multi-step description.The four prompted conditions were within a few points of one another and the empty-prompt condition.
  • H.1. Finding 1: VLAs over-fit to scene, not generalizable to instruction: The same no-prompt-versus-prompted gap appeared across other benchmark-model combinations and spot-checks on RoboCerebra and VLABench.These results indicate limited generalist behavior in the released simulation-trained checkpoints.
  • H.2. Finding 2: vision tools do not transfer to non-photoreal sims: In MuJoCo-rendered scenes, GroundingDINO produced empty or wrong-class boxes, SAM2 and SAM3 failed or selected wrong instances, and Molmo2 pointed toward image centers.These tools were trained almost exclusively on real-world imagery.
  • H.2. Finding 2: vision tools do not transfer to non-photoreal sims: The same three vision failures occurred on RoboCerebra and VLABench without tuning, implicating the renderer rather than benchmark asset choices.This provides empirical support for the paper’s insufficient-realism claim.
  • H.2. Finding 2: vision tools do not transfer to non-photoreal sims: RoboLab’s language-conditional DROID VLA checkpoints and Isaac Sim PathTracer rendering allow instruction rewrites and vision tools to transfer.These properties motivated designing RoboVoLo on top of RoboLab.

I. Per-suite Component Ablations … K.1. World failures (outcome events, task_failures.jsonl)

The appendix expands component ablations, outcome-flow diagnostics, and passive failure-taxonomy definitions. It reports per-suite success-rate comparisons, intermediate ablation flows, and state-based event tracking for long-horizon tasks.

  • I. Per-suite Component Ablations: Table 7 provides the full per-suite breakdown of component ablations while holding all other components at their defaults.The final VoLoAgent row is the full-system reference, and all entries report success rate (%), where higher is better.
  • J. Additional Outcome-flow Diagrams: Figure 13 compares outcome flows for VoLoAgent (No VLA) and VoLoAgent (Only VLA) on Common Sense.The No VLA variant replaces the policy with VLM-driven primitives, while Only VLA retains the VLA and VLM monitor but disables tool-augmented recovery.
  • K. Failure-mode Taxonomy and Definitions: The diagnostic streams run passively, logging per-episode events without influencing orchestration and using the simulator’s gt_state export.These streams expand the two diagnostic streams referenced in the main paper.
  • K.1. World failures (outcome events, task_failures.jsonl): Each long-horizon task is represented as an ordered sequence of sub-tasks, each with target objects and required end-state predicates.At any instant one sub-task is active, and its objects are tracked in parallel until every required predicate is satisfied.
  • K.1. World failures (outcome events, task_failures.jsonl): Previously completed objects remain tracked for regression, except when they are targets of the active sub-task, where regression is deferred until completion.This exception treats an in-flight re-grasp as part of the plan rather than a failure.
  • K.1. World failures (outcome events, task_failures.jsonl): Stateless rules consume per-step gt_state snapshots to emit five event types against the active sub-task and object-tracking state.Defined events include wrong_object_picked, wrong_target_place, object_regression, and stuck; the supplied passage truncates the fifth definition.
  • K.1. World failures (outcome events, task_failures.jsonl): A failure spell spans a failure event and its paired recovery, while unrecovered spells are those lacking recovery before episode end.Successful episodes are post-hoc labeled “all recovered,” including cases where the recovery event was cut off by the success terminal.

K.2. VLM failures (metrics.jsonl)

VoLo evaluates VLM failures by matching each VLM call against hand-authored ground-truth predicates evaluated from simulator state. Its metrics partition failures across planning, completion monitoring, failure monitoring, and tool use, while an alias-and-grounding resolver removes semantically equivalent target-name mismatches.

  • Ground-truth evaluation: Hand-authored success and invariant predicates define task correctness and are evaluated against simulator physics state at every step.The resulting gt_state stream includes predicate results alongside poses, gripper contact, and per-subtask completion flags; detectors consume it passively.
  • Ground-truth evaluation: Each VLM planning, monitoring, or tool-dispatch call is paired with the ground-truth predicate it should have verified at that step.This compares verdicts such as subgoal completion and grasp targets against the active task state.
  • Failure taxonomy: Eight leaf metrics partition VLM failures into planning, completion monitoring, failure monitoring, and tool-use taxonomy groups.The group sum equals the total VLM-failure count, and the groups correspond to the four segments in the main-paper failure figure.
  • Tool-use resolution: A two-stage offline resolver reclassifies alias-equivalent grasp-target mismatches using exact matching, an alias dictionary, and residual VLM grounding.The resolver is applied once per sweep to distinguish naming variation from true semantic mismatches.
  • Tool-use resolution: Resolved events enter the vlm_grasp_target_qa: target_aliased success bucket and are excluded from vlm_tool_use_error.Table 8 and the main-paper VLM-failure figure report post-resolver counts.
Loading 2606.07723v1…