PaperlayerView exact revision on arXiv

Source-linked AI summary

Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding

Haibo Wang, Zihao Lin, Zhiyang Xu, Lifu Huang

arXiv:2604.00528v2cs.CVcs.AI

TL;DR

Zero-shot 3D-VG methods often depend on preprocessed point clouds and static proposal matching, limiting grounding from raw visual streams. TAB uses an agentic Think-Act-Build process that combines 2D VLM semantic reasoning with deterministic multi-view geometry on RGB-D streams. On ScanRefer and Nr3D, it is reported to outperform prior zero-shot methods and fully supervised baselines.

  • Problem

    Existing zero-shot 3D-VG frameworks often rely on preprocessed point clouds or limited semantic tracking, while supervised methods require costly dense 3D annotations.

  • Method

    TAB dynamically invokes visual tools through an iterative Think-Act-Build loop, using 2D VLMs for semantic tracking and multi-view geometry for 3D reconstruction from RGB-D streams.

  • Results

    TAB outperforms prior zero-shot methods and fully supervised baselines on ScanRefer and Nr3D; on Nr3D it achieves 68.0% overall accuracy.

  • Takeaways & Limitations

    TAB provides a zero-shot 3D-VG paradigm that operates without pre-scanned point clouds and supports rigorous evaluation through refined benchmarks.

Abstract

from arXiv · show

3D Visual Grounding (3D-VG) aims to localize objects in 3D scenes via natural language descriptions. While recent advancements leveraging Vision-Language Models (VLMs) have explored zero-shot possibilities, they typically suffer from a static workflow relying on preprocessed 3D point clouds, essentially degrading grounding into proposal matching. To bypass this reliance, our core motivation is to decouple the task: leveraging 2D VLMs to resolve complex spatial semantics, while relying on deterministic multi-view geometry to instantiate the 3D structure. Driven by this insight, we propose "Think, Act, Build (TAB)", a dynamic agentic framework that reformulates 3D-VG tasks as a generative 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams. Specifically, guided by a specialized 3D-VG skill, our VLM agent dynamically invokes visual tools to track and reconstruct the target across 2D frames. Crucially, to overcome the multi-view coverage deficit caused by strict VLM semantic tracking, we introduce the Semantic-Anchored Geometric Expansion, a mechanism that first anchors the target in a reference video clip and then leverages multi-view geometry to propagate its spatial location across unobserved frames. This enables the agent to "Build" the target's 3D representation by aggregating these multi-view features via camera parameters, directly mapping 2D visual cues to 3D coordinates. Furthermore, to ensure rigorous assessment, we identify flaws such as reference ambiguity and category errors in existing benchmarks and manually refine the incorrect queries. Extensive experiments on ScanRefer and Nr3D demonstrate that our framework, relying entirely on open-source models, significantly outperforms previous zero-shot methods and even surpasses fully supervised baselines.

1 Introduction

3D-VG localizes objects from natural-language descriptions, but annotation costs and point-cloud-dependent zero-shot workflows limit scalability and flexibility. TAB instead combines VLM semantic reasoning with geometric reconstruction directly from RGB-D streams.

  • Motivation: 3D-VG localizes a target object in 3D space from a free-form natural-language query.The task supports applications including human-robot interaction, embodied navigation, and AR/VR.
  • Motivation: Dense 3D annotations make supervised methods costly to scale and difficult to generalize to open-world, open-vocabulary scenarios.
  • Limitations of prior work: Existing zero-shot methods often depend on preprocessed point clouds and reduce grounding to selecting among predefined 3D proposals.This restricts deployment where preprocessed point clouds are unavailable.
  • Limitations of prior work: Some image-based methods use heuristic 2D semantic matching to associate multi-view observations without fully exploiting geometric information.
  • TAB: TAB reformulates zero-shot 3D-VG as active semantic reasoning and geometric reconstruction without relying on preprocessed point clouds.Its agent iteratively reasons and invokes visual tools while operating on RGB-D streams.
  • TAB: TAB introduces Semantic-Anchored Geometric Expansion and refines benchmark queries to address multi-view coverage, reference ambiguity, and category errors.The framework uses open-source models and is reported to outperform prior zero-shot methods and fully supervised baselines.

2 Related Works

Related work covers supervised 3D-VG pipelines and efforts to extend VLMs into 3D understanding. Existing approaches either process point clouds with specialized 3D encoders or aggregate multi-view 2D features into 3D representations.

  • 3D Visual Grounding: Supervised 3D-VG methods use either two-stage proposal generation or single-stage fusion of point-cloud and textual features.Both approaches are described as limited by dense 3D annotation costs and restricted generalization.
  • VLMs for 3D Understanding: 3D multimodal models commonly build 3D-aware representations with specialized point-cloud encoders or aggregated multi-view image features.

3 Method

TAB reformulates zero-shot 3D visual grounding as a dynamic agentic loop that combines VLM semantic reasoning with multi-view geometric reconstruction from RGB-D streams. It tracks a reference target, expands coverage geometrically, and builds a 3D representation while adapting to intermediate failures.

  • Framework Overview: TAB formulates 3D-VG as a Think–Act–Build loop in which a VLM agent invokes visual tools and directly reconstructs a 3D bounding box from sequential RGB-D streams.The input includes RGB images, aligned depth maps, camera intrinsics, and camera extrinsics.
  • Reference Target Localization: The agent parses free-form queries into target classes, attributes, spatial conditions, and global scene features for downstream tool execution.Query Analysis structures descriptions such as “the pillow on the left bed” into explicit execution arguments.
  • Reference Target Localization: Coarse-to-fine filtering retrieves candidate frames, after which Score&Rank selects a reference frame and Seg&Marker separates same-class instances.Foundation detectors identify frames containing the target class, while VLM verification checks the parsed scene constraints.
  • Semantic-Anchored Geometric Expansion: Semantic-Anchored Geometric Expansion projects the centroid into unobserved frames, checks visibility, segments verified projections, and filters the resulting geometry into a final 3D box.The visibility check uses image-domain, predicted-depth, and sensor-noise conditions before adding frames to the geometric expansion pool.
  • Semantic-Anchored Geometric Expansion: Semantic Temporal Expansion tracks the target bidirectionally from the reference frame, stopping in each direction when identity verification fails.Accepted frames contribute segmentation masks to a growing local video context used to construct initial geometry.
  • 2D to 3D Reconstruction: The framework inverse-projects masked depth pixels into world coordinates, aggregates them into an initial point cloud, and computes a viewpoint-invariant 3D centroid.Camera intrinsics recover local 3D points, while frame-specific extrinsics transform them into a common world coordinate system.
  • 2D to 3D Reconstruction: TAB uses statistical outlier removal and DBSCAN clustering to isolate the main object geometry from aggregated multi-view points.The cleaned cluster’s spatial extremes are used to estimate the axis-aligned 3D bounding box.
  • Robust Execution: TAB provides fault tolerance by relaxing thresholds or skipping non-critical steps when candidate retrieval or depth reconstruction encounters local failures.This dynamic adjustment is intended to maintain execution in noisy environments.

4 Benchmark Refinement

The paper finds non-negligible annotation noise in ScanRefer and Nr3D and manually refines affected queries for more rigorous evaluation.

  • The authors identify three primary annotation-error types in ScanRefer and Nr3D, beginning with ambiguous references that admit multiple valid candidates.They resolve ambiguous references by adding distinctive information.

5 Experiments

Experiments evaluate TAB on ScanRefer and Nr3D using open-source models, benchmark metrics, and ablations of its semantic and geometric expansion components. TAB achieves strong zero-shot results without pre-scanned point clouds, while the ablations show the value of multi-view context and geometric expansion.

  • 5.1 Settings: Experiments use ScanRefer and Nr3D, with ScanRefer reporting Acc@0.25 and Acc@0.5 and Nr3D reporting top-1 selection accuracy.Both benchmarks use ScanNet indoor scenes; ScanRefer distinguishes Unique and Multiple queries.
  • 5.1 Settings: TAB samples 300 frames per video and is built entirely with open-source models, including Qwen3-VL-32B, Grounding DINO, and SAM3.The framework uses Qwen3-VL-32B as its primary VLM agent, with Grounding DINO for coarse detection and SAM3 for instance segmentation.
  • 5.2 3D Visual Grounding Results: 71.2% Acc@0.25 and 46.4% Acc@0.5 are achieved on ScanRefer without point-cloud inputs, including 60.1% Acc@0.25 on the Multiple subset.Adding Mask3D proposals raises Acc@0.5 from 46.4% to 61.6% while Acc@0.25 changes from 71.2% to 71.6%.
  • 5.2 3D Visual Grounding Results: 68.0% overall accuracy is achieved on Nr3D without pre-scanned point clouds, surpassing SPAZER at 63.8% and SceneVerse at 64.9%.TAB reaches 63.2% on Hard and 62.5% on View-Dependent subsets.
  • 5.3 In-Depth Analysis: 41.6% Acc@0.25 on ScanRefer and 52.0% on Nr3D result when reconstruction uses only a single reference frame.The single-view baseline is susceptible to depth noise and self-occlusion because it lacks multi-view context.
  • 5.3 In-Depth Analysis: Removing semantic temporal expansion lowers performance to 48.9% on Nr3D Dep. and 41.1% Acc@0.25 and 22.2% Acc@0.5 on ScanRefer Multiple.The ablation attributes this decline to centroid bias from partial single-view surfaces.
  • 5.3 In-Depth Analysis: Removing multi-view geometric expansion causes semantic tracking to fail under extreme viewpoint changes or intermediate occlusions, reducing localization precision.The ablation isolates the coverage deficit of relying only on VLM-tracked frames.
  • 5.3 In-Depth Analysis: TAB produces a tight 3D box with IoU = 0.74, compared with 0.42 when Multi-View Geometric Expansion is removed.The qualitative comparison links the lower IoU to fragmented geometry from reduced view coverage.

6 Conclusion

The conclusion presents TAB as a dynamic reasoning-and-reconstruction framework that separates semantic understanding from multi-view geometry. Its Semantic-Anchored Geometric Expansion addresses coverage deficits by projecting a 3D centroid across unobserved frames to build a complete representation from RGB-D streams.

  • Figure 4 compares different TAB components qualitatively.
  • TAB reformulates zero-shot 3D visual grounding as a dynamic reasoning and reconstruction process using an iterative Think-and-Act loop with 2D VLMs.The framework explicitly decouples semantic understanding from multi-view geometry.
  • Semantic-Anchored Geometric Expansion projects a 3D centroid across unobserved frames to collect multi-view masks and build a complete 3D point cloud from RGB-D streams.The mechanism is introduced to overcome the coverage deficit of purely semantic tracking.

A.1 Expert Skill: 3D Visual Grounding

The expert skill defines a sequential 3D visual-grounding pipeline that parses queries, filters and ranks frames, isolates a target, expands observations temporally and geometrically, and computes a 3D bounding box.

  • The expert skill is a Markdown blueprint that defines the standard operating procedure for the agent’s 3D visual-grounding and reconstruction pipeline.It is designed to be directly read and parsed by the agent.
  • The pipeline parses the query into target class, visual attributes, spatial conditions, and global scene features.
  • Coarse mask filtering retains images containing the target class, while fine VLM filtering verifies scene semantics and falls back if it removes every image.
  • Remaining images are scored by alignment with query attributes and conditions, then sorted in descending order.
  • The highest-scoring image becomes the reference view, where the VLM identifies the matching object ID among same-class instances.Segmentation then isolates the selected object ID for downstream processing.
  • Temporal expansion tracks the target forward and backward with VLM identity verification and SAM masks, producing a temporally consistent candidate clip.
  • The verified images and masks are aggregated into an initial 3D point cloud before geometric expansion.
  • Geometric expansion projects the target’s absolute 3D centroid across scene frames using camera extrinsics and Z-Buffer checks.

A.2 Prompts

The framework provides prompts that govern the VLM agent’s reasoning, filtering, and tracking behavior.

  • A.2 Prompts: Prompts are supplied for different modules of the framework.They specify how the agent should behave across its modules.
  • A.2 Prompts: The prompts guide the agent’s reasoning behavior.
  • A.2 Prompts: The prompts guide the agent’s filtering and tracking behaviors.

### AVAILABLE TOOLS

The available tools are specified through strict parameter definitions for the agent’s tool calls.

  • ### AVAILABLE TOOLS: The framework lists the available tools for the agent.
  • ### AVAILABLE TOOLS: Each tool is governed by a strict parameter definition.
  • ### AVAILABLE TOOLS: The parameter definitions constrain how tool actions must be specified.

### RESPONSE FORMAT

The response format defines an iterative Think–Act loop and specialized visual verification, filtering, tracking, segmentation, and structured-output behaviors.

  • ### RESPONSE FORMAT: The agent must respond in a step-by-step loop containing a Thought followed by an Action.
  • ### RESPONSE FORMAT: The visual verification assistant checks objects, quantities, spatial relations, and attributes against the user’s description.
  • ### RESPONSE FORMAT: The visual filtering assistant assigns zero when the target class is not visible and lowers scores when reference objects are missing.
  • ### RESPONSE FORMAT: The tracking assistant verifies exact instance identity while allowing modest viewpoint, lighting, scale, and occlusion changes.
  • ### RESPONSE FORMAT: The segmentation marker prompt directs object identification from annotated images containing bounding boxes and numeric IDs.

A.3 Tool Library

The tool library supports a complete grounding trace from query parsing and image filtering through target tracking, geometric expansion, reconstruction, and final 3D bounding-box calculation.

  • A.3 Tool Library: The agent dynamically invokes specialized tools built on SAM, VLMs, and multi-view geometric projection functions.Tool calls use strict JSON parameter schemas.
  • A.4 Agent Execution Trace Example: The execution trace begins by parsing the query into a target class, attributes, spatial conditions, and scene features.The example query identifies a pillow, its top-pillow attribute, proximity to the table, and the two-bed scene context.
  • A.4 Agent Execution Trace Example: Coarse filtering retains 90 pillow-containing images, while fine filtering retains 24 images matching the scene feature.
  • A.4 Agent Execution Trace Example: The agent scores and ranks candidate images, selects the best image and target ID, and isolates the target with a mask and bounding box.The selected example is image 00085.jpg with target object ID 5.
  • A.4 Agent Execution Trace Example: Temporal expansion yields 10 images, which are segmented and used for an initial point-cloud reconstruction.
  • A.4 Agent Execution Trace Example: Geometric multi-view expansion projects the reconstructed centroid across other frames to increase view coverage before dense reconstruction.
  • A.4 Agent Execution Trace Example: The final point cloud is converted into an axis-aligned 3D bounding box for the target pillow.For the example, the output is [1.6515, 1.1065, 0.7770, 0.4687, 0.6466, 0.2580].
  • A.4 Agent Execution Trace Example: The framework uses fallback behavior when intermediate errors or invalid states occur.The trace describes returning to a previous stable state rather than simply crashing.
Loading 2604.00528v2…