PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Generalizable Visually Grounded Exploration of Household Devices

Linhao Zheng, Zeming Liu, Wangke Chen, Li Zeng, Wanxiang Che, Heyan Huang, Yuhang Guo

arXiv:2609.00845v1cs.AI

TL;DR

Existing embodied benchmarks and tool-learning paradigms do not adequately evaluate how agents operate unfamiliar household devices without manuals by grounding general knowledge in fine-grained affordances. VGEBench addresses this gap with a deterministic Logic-Driven State Machine and multi-turn feedback loops, and experiments show that most current VLMs struggle with precise physical interaction and long-horizon state tracking.

  • Problem

    Existing benchmarks and embodied paradigms rely on explicit documentation, annotated trajectories, or static evaluation rather than testing documentation-free, feedback-driven operation of unfamiliar devices.

  • Method

    VGEBench uses a Logic-Driven State Machine to simulate deterministic multi-turn interactions in which VLMs ground visual affordances, act, and respond to environmental feedback.

  • Results

    Most VLMs struggle to translate semantic knowledge into precise physical interaction and maintain long-horizon state consistency; Gemini-3-Flash reaches 54.27% Success Rate, while all remaining models stay below 17%.

  • Takeaways & Limitations

    Generalizable visually grounded exploration remains a difficult open problem requiring fine-grained grounding, active exploration, and feedback-driven refinement.

  • Takeaways & Limitations

    Full-scale evaluation requires substantial computation and inference time, and high-resolution observations demand efficient visual encoding and large context windows.

Abstract

from arXiv · show

Recent advancements in Vision-Language Models (VLMs) have demonstrated impressive capabilities in static visual recognition and high-level semantic reasoning. However, current embodied exploration paradigms still heavily rely on imitation learning from human-annotated trajectories, which severely limits agents' generalization ability. The key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration: the ability to operate novel devices without manuals or specific training by actively grounding abstract world knowledge into fine-grained visual affordances. Yet, existing benchmarks fail to evaluate this capability: they generally rely on explicit documents and annotated trajectories, neglecting the dynamic Hypothesis-Interaction-Refinement process essential for functional device operation. To bridge this gap, we introduce VGEBench, a comprehensive benchmark designed to evaluate the generalizable visually grounded exploration capabilities of VLMs. Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction. Experimental results demonstrate that existing VLMs face significant challenges in translating semantic knowledge into physical execution and maintaining long-horizon state tracking.

1 Introduction

VGEBench addresses the gap between abstract world knowledge and precise interaction with unfamiliar household devices by evaluating hypothesis-driven, feedback-informed exploration. Experiments show that current VLMs struggle with visual grounding, physical execution, and long-horizon state tracking.

  • Motivation: Humans operate unfamiliar devices by forming affordance hypotheses, interacting, and refining them from physical feedback.This Hypothesis-Interaction-Refinement process supports generalizable manipulation.
  • Research Gap: Existing robotic benchmarks often depend on imitation or reinforcement learning in specific environments, limiting generalization to unseen devices.Tool-learning systems typically use structured APIs and explicit documentation, bypassing instance-specific physical interfaces.
  • VGEBench: VGEBench evaluates documentation-free operation of 968 diverse devices across 26 categories using visual perception and interaction.Its challenges include knowledge grounding, fine-grained localization, and feedback-driven correction.
  • Contributions: VGEBench introduces a Logic-Driven State Machine framework for deterministic, multi-turn interaction and long-horizon exploratory reasoning.The benchmark is designed to evaluate generalizable visually grounded exploration rather than static recognition or rote execution.
  • Results: Experiments reveal that existing VLMs struggle to translate general world knowledge into precise physical interaction during multi-turn scenarios.The evaluation also exposes limitations in visual grounding, generalizable exploration, and long-horizon state tracking.

2 Related Work

Related benchmarks cover language-centric tool use, static visual reasoning, and embodied perception, but generally do not test active discovery of device affordances through interaction. VGEBench targets their intersection by combining fine-grained grounding, feedback-driven exploration, and functional generalization.

  • From APIs to Physical Interaction: API-based tool learning relies on structured interfaces or documentation rather than discovering functionality through physical exploration.This leaves a gap between language-centric tool use and undocumented device interaction.
  • Fine-grained Visual Grounding: VQA and referring-expression benchmarks emphasize holistic recognition or region-language alignment but do not fully model actionable component localization under interaction constraints.Small, functionally meaningful parts can remain difficult to distinguish across instances and viewpoints.
  • VGEBench Positioning: VGEBench evaluates training-free generalization, interactive feedback and refinement, error correction, device logic modeling, manual-free operation, fine-grained grounding, and multi-view perception.These dimensions are summarized in its comparison with existing related benchmarks.
  • Embodied Exploration: Embodied benchmarks often assess passive observation, static functional reasoning, or trained task execution rather than active state-changing manipulation.Household-device operation additionally requires validating actions through observable state changes and feedback.
  • VGEBench Positioning: VGEBench focuses on functional and interactive generalization: selecting actions from visual components and revising hypotheses through environment feedback.This differs from appearance-level novelty in static image recognition.

3 VGEBench

VGEBench constructs a device benchmark from high-fidelity views, component annotations, and executable state machines, then evaluates agents in deterministic feedback-driven interaction loops. Its tasks use validated state-transition paths and assess both action validity and ordered goal completion.

  • Data Construction: The benchmark construction pipeline comprises raw view collection, state-machine annotation, and instruction generation.The pipeline progresses from visual observations to executable logic and natural-language tasks.
  • Raw View Collection: Annotators capture high-fidelity multi-view observations and verify interactive components, producing 968 3D models across 26 household-device categories and 7,888 views.Each object has eight views: six orthographic and two axonometric.
  • State Machine Design: A Universal Category State Machine is pruned into an object-specific state machine according to visually present components and their valid transitions.The resulting logic graph binds transitions to atomic actions and parameters.
  • Scope and Quality Control: VGEBench evaluates functional and interactive generalization rather than appearance-level novelty, while its high-resolution, closed-loop design increases computational and visual-context demands.The benchmark also cannot guarantee that closed-source VLMs have never encountered related public assets.
  • Instruction Generation: Instruction tasks are generated by finding loop-free paths through validated state machines and verbalizing their symbolic transitions with an LLM.The initial state, target state, and valid path remain determined by the state machine, while the LLM verbalizes them.
  • Environment Feedback and Evaluation: At each step, the simulator validates action type, coordinates, and parameters against the current state, then emits outcome-dependent feedback without requiring a fixed trajectory.Tasks require ordered sub-goals, and any valid interaction sequence reaching the required states is accepted within the interaction budget.

4 Experiment

VGEBench evaluates VLMs with a multidimensional protocol spanning task performance, execution efficiency, visual grounding, and reasoning stability. Results show substantial difficulty in translating visual and semantic knowledge into precise physical interaction, although stronger grounding and exploration correlate with better task completion.

  • Evaluation Protocol: VGEBench reports nine metrics across task performance, execution efficiency, visual grounding, and reasoning stability.The protocol includes success, subtask, path-length, state-alignment, interaction, navigation, and exploration measures.
  • Task Performance: Success Rate measures completed episodes, while Sub-task Success Rate measures step-wise correctness based on achieved sub-goals.These metrics distinguish full-task completion from partial progress in multi-turn tasks.
  • Exploration: Effective Exploration Rate is the ratio of valid operations to total operation steps, measuring reasoning efficiency and safety awareness.The broader protocol also evaluates interaction validity, target grounding, and navigation cost.
  • Main Results: 54.27% Success Rate is achieved by Gemini-3-Flash, while Doubao-1.5-Thinking-Vision-Pro reaches 16.68% and all remaining models remain below 15%.Only Gemini-3-Flash exceeds 25% on Sub-task Success Rate.
  • Main Results: Higher visual grounding and exploration scores accompany stronger task completion, whereas MiMo-Embodied-7B reaches 0.88 VSPS but only 1.48% SR.The contrast indicates that navigation efficiency alone does not substitute for fine-grained visual grounding and reasoning.

5 Analysis

The analysis tests whether VLMs generalize immediately and how additional interaction budget affects exploration. Models show weak one-shot understanding but can improve through feedback-driven interaction, with stronger models exploiting longer exploration more effectively.

  • RQ1: Generalization Capacity: 12.94% PSR versus 62.86% SSR for Gemini-3-Flash shows a pronounced gap between exact one-shot completion and subtask success.Across models, successful performance mainly arises from iterative feedback-driven exploration rather than immediate operational understanding.
  • RQ2: Interaction Budget: Additional interactions improve task completion for all models, but the gains gradually taper as the interaction limit increases.Figure 3 normalizes performance at λ = 2 to zero and shows relative gains for larger budgets.
  • RQ2: Interaction Budget: Weaker models such as Qwen3-VL-8B reach earlier, more pronounced performance plateaus than Gemini-3-Flash.Gemini-3-Flash shows greater ability to exploit deep exploration for self-correction, alongside stronger exploratory reasoning and long-term memory maintenance.

RQ3: Where lies the visual bottleneck: Coarse-grained navigation or fine-grained grounding?

The performance gap is moderate for coarse-grained navigation but substantially larger for fine-grained grounding, especially in coordinate accuracy and correction after failures. These results identify precise interactive-component localization as the primary visual bottleneck in VGEBench.

  • Coarse-grained navigation: GPT-5-mini’s successful navigation (NP) is 12.5% lower than Gemini-3-Flash’s, indicating a moderate coarse-grained navigation gap.Navigation is evaluated through view-switching outcomes, including successful navigation from a negative to a positive view.
  • Fine-grained grounding: GPT-5-mini’s coordinate error (Avg Dist) is 63.9% worse than Gemini-3-Flash’s, showing a drastic fine-grained grounding disparity.Avg Dist measures average distance to the closest target bounding box on a normalized 1000 × 1000 scale.
  • Fine-grained grounding: GPT-5-mini’s Avg W-Dist gap is 77.3%, indicating weaker correction of coordinates after initial failures.Avg W-Dist penalizes repeated failed attempts on the same target.
  • Bottleneck diagnosis: The primary VGEBench bottleneck is precisely localizing the interactive component within the correct view rather than navigating to that view.Human analyses likewise associate fewer invalid-coordinate errors with more informative component-level feedback for hypothesis refinement.

6 Conclusion

VGEBench evaluates generalizable visually grounded exploration without manuals through deterministic multi-turn interaction loops. Experiments show that current VLMs retain semantic knowledge but struggle with fine-grained grounding and long-horizon state tracking.

  • 6 Conclusion: VGEBench evaluates manual-free device exploration using a Logic-Driven State Machine with deterministic feedback in multi-turn interaction loops.The benchmark is designed to assess generalizable visually grounded exploration of household devices.
  • 6 Conclusion: Current VLMs struggle significantly with fine-grained visual grounding despite strong semantic knowledge.The conclusion frames precise grounding as a persistent difficulty in the benchmark setting.
  • 6 Conclusion: Current VLMs also struggle to maintain state tracking over long-horizon interactions.The benchmark’s multi-turn structure exposes this difficulty through sequential interaction and feedback.

Limitations

VGEBench’s interactive evaluation is computationally expensive and demands efficient visual encoding and large context windows. Its deterministic abstraction and rendered observations also limit how fully it represents physical-robot deployment and real-world visual conditions.

  • Limitations: Full-scale VGEBench evaluation requires substantial computational resources and inference time because it uses multi-turn agent-environment feedback loops.The limitation applies to the benchmark’s dynamic interactive evaluation framework.
  • Limitations: High-resolution observations needed for fine-grained grounding place heavy demands on VLM visual encoding efficiency and context-window capacity.The constraint follows from resolving fine-grained grounding details in the visual inputs.
  • Limitations: The benchmark’s low-level-control abstraction and fixed viewpoint switching do not fully capture sim-to-real challenges in physical robotic deployment.The design intentionally focuses evaluation on high-level reasoning by avoiding continuous control and freeform camera navigation.
  • Limitations: Device-centered rendered images omit the full complexity of real environments, including complex backgrounds, natural illumination, and object occlusion.Controlled compression, blur, and random-occlusion tests cannot fully substitute for real-world observations.

Ethics Statement

VGEBench reports ethical and data-compliance practices for its household-device assets and human annotation. The benchmark prioritizes common, compact devices and combines licensed 3D sources across 26 categories.

  • Ethics Statement: Human annotators received category-specific training, informed consent about data usage, and fair compensation.The paper links these measures to annotation quality and appropriate treatment of participants.
  • Category Selection Criteria: Device categories emphasize common household objects while excluding deeply nested or highly articulated structures that could impair visual coverage.Examples include CoffeeMachine, Microwave, Lamp, and DigitalAlarmClock.
  • Data Source Distribution: VGEBench contains 968 tool instances across 26 categories, sourced from Sketchfab and 3DWarehouse, with Sketchfab contributing the larger share of high-quality models.The source distribution is attributed to Sketchfab’s greater availability of high-quality 3D models.
  • Action Type Distribution: Press accounts for approximately 45.20% of atomic actions, while Rotate, Grasp, and Timed_wait appear sparsely in device-dependent categories.The action statistics describe SSM transitions across the 26 device categories.

B Error Analysis

VGEBench error analysis groups failures into fine-grained visual grounding, action execution, and visual navigation, with spatial grounding dominating model errors. The analysis also identifies parameter handling as a notable reasoning weakness despite competent global navigation and planning.

  • Error Taxonomy: Fine-grained Visual Grounding Errors include invalid-coordinate clicks and selecting incorrect or non-optimal components.These failures concern pixel-level localization of interactive regions.
  • Error Taxonomy: Action Execution Errors occur when targeting is correct but the operation logic is wrong, including parameter, atomic-action, and invalid-waiting errors.Examples include rotating a button that should be pushed and waiting when no state change is possible.
  • Error Taxonomy: Visual Navigation Errors comprise ineffective view searching and losing the target component during view switching.The taxonomy labels these failures Switch View NN and Switch View PN.
  • Error Distribution: 66.1% of errors are fine-grained visual grounding failures, compared with 17.2% for Action Execution and 16.7% for Visual Navigation.Coordinate-related errors are identified as the primary bottleneck in device-centric interaction.
  • Interpretation: Agents show competent global navigation and high-level planning but substantial weakness in fine-grained perception, with parameter errors accounting for 10.8%.The parameter-error rate indicates difficulty respecting precise physical constraints even after identifying the correct component.

C Human Baseline Analysis

The human baseline evaluates 149 episodes under the same interface and interaction budgets as Gemini-3-Flash. Humans outperform the model across major metrics, while the benchmark remains non-trivial for humans and uses feedback, views, and budgets to structure exploration.

  • Human Baseline: Humans consistently outperform Gemini-3-Flash across task performance, visual grounding, and exploration metrics.The comparison uses 149 human episodes under the same interface and interaction budgets.
  • Error Distribution Comparison: Humans make fewer invalid-coordinate errors, enabling more precise localization of interactive regions than Gemini-3-Flash.This difference allows humans to obtain more informative component-level feedback.
  • Feedback Mechanism: The environment returns successful-transition, operational-error, or null-effect feedback based on the action’s match with the specific state machine.These feedback levels support error recovery by distinguishing state changes, valid-but-wrong operations, and non-interactive coordinates.
  • Interaction Budgets: Episodes terminate when all sub-tasks are completed or either the global or local interaction budget is violated.The global budget limits total episode interactions, while the local budget prevents inefficient looping within a sub-task.
  • Dashboard View: The dashboard view provides closer inspection for large devices whose small text or controls are difficult to recognize globally.It is unlocked actively through a dashboard-region view switch or passively by attempting an interaction within the locked region.
  • History-Length Analysis: 55.04% SR and 63.29% SSR with complete history exceed 22.61% SR and 31.83% SSR with no history for Gemini-3-Flash.Long interaction traces support evidence accumulation and self-correction for this model, whereas Doubao performs best with a shorter three-turn history.

E.2 Visual Perturbation Robustness

Visual perturbation and instruction-robustness experiments examine whether VGEBench remains informative under degraded observations and alternative instruction generation or rewriting styles. Gaussian blur causes the strongest degradation, while instruction choices and rewriting styles produce stable evaluations.

  • Visual Perturbations: Compression, Gaussian blur, and random occlusion simulate low-quality transmission, defocusing or motion blur, and environmental clutter.Occlusion uses eight black patches covering total area ratios of 10%, 25%, or 40%.
  • Visual Perturbations: Under strong Gaussian blur with σ = 5.0, Gemini-3-Flash SR falls from 55.04% to 38.53% and SSR from 63.29% to 48.12%.Doubao-1.5-Thinking-Vision-Pro shows the same direction, with SR decreasing from 16.30% to 10.16%.
  • Visual Perturbations: The stronger sensitivity to blur supports fine-grained visual grounding as a central bottleneck when component boundaries, icons, and text cues are degraded.These high-frequency visual details are needed for reliable interaction.
  • Instruction Robustness: Changing the instruction generator yields only minor variation: Gemini-3-Flash reaches 55.04%–56.86% SR and 63.29%–64.19% SSR.GPT-5-mini reaches 10.71%–12.58% SR and 17.20%–19.20% SSR across generators.
  • Instruction Robustness: Instruction rewriting styles produce standard deviations below 0.6% across both metrics, with narrow confidence intervals preserving stable relative model ordering.This experiment addresses potential template and stylistic bias in the fixed language pipeline.
  • Evaluation Metrics: VGEBench measures episode success, sub-goal success, path efficiency, state alignment, grounding, navigation, exploration, and annotation quality through complementary metrics.The benchmark also evaluates valid interaction coordinates, goal-oriented grounding, view-switch efficiency, and the valid-operation ratio.
Loading 2609.00845v1…