Source-linked AI summary
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent
Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F. Fouhey, Joyce Chai
TL;DR
3D visual grounding matters for household robots, while existing methods require labeled data or struggle with complex language. LLM-Grounder uses an LLM agent with visual grounding tools and spatial reasoning, achieving state-of-the-art zero-shot accuracy on ScanRefer and stronger gains for complex queries. The approach is limited by GPT-based cost and latency for resource-constrained or real-time robotics.
Problem
Existing 3D grounding methods often require labeled data or exhibit “bag-of-words” behavior that misses compositional relations in complex queries.
Method
LLM-Grounder uses an LLM agent to decompose queries, orchestrate tools such as OpenScene and LERF, and reason over bounding boxes, volumes, distances, and spatial relations.
Results
The method achieves state-of-the-art zero-shot grounding accuracy on ScanRefer without labeled data, with LLM assistance improving zero-shot grounders by 5.0% and 17.1% on Accuracy@0.25 for LERF and OpenScene, respectively.
Takeaways & Limitations
LLM-Grounder provides a zero-shot, open-vocabulary approach that is particularly effective for complex text queries in 3D visual grounding.
Takeaways & Limitations
GPT-based reasoning can be computationally expensive and slow, limiting deployment in resource-constrained environments and real-time robotic applications.
Abstract
from arXiv · showhide
3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to decompose complex natural language queries into semantic constituents and employs a visual grounding tool, such as OpenScene or LERF, to identify objects in a 3D scene. The LLM then evaluates the spatial and commonsense relations among the proposed objects to make a final grounding decision. Our method does not require any labeled training data and can generalize to novel 3D scenes and arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and demonstrate state-of-the-art zero-shot grounding accuracy. Our findings indicate that LLMs significantly improve the grounding capability, especially for complex language queries, making LLM-Grounder an effective approach for 3D vision-language tasks in robotics. Videos and interactive demos can be found on the project website https://chat-with-nerf.github.io/ .
I. INTRODUCTION
3D visual grounding is essential for household robots, but existing methods require labeled 3D-text data or struggle with compositional relations. LLM-Grounder uses an LLM agent to decompose queries, use visual tools, and reason over feedback for zero-shot open-vocabulary grounding.
- 3D visual grounding supports household-robot navigation, manipulation, and question-answering.
- Existing trained approaches typically require large amounts of 3D-text training pairs that are difficult to obtain.
- CLIP-based open-vocabulary methods exhibit “bag-of-words” behavior, often ignoring textual order, attributions, and relations.
- LLMs can address complex grounding queries by decomposing language, using tools, collecting feedback, and applying spatial and commonsense reasoning.
- LLM-Grounder requires no labeled training data and generalizes zero-shot to novel 3D scenes and arbitrary text queries.
- The method achieves state-of-the-art zero-shot grounding accuracy on ScanRefer, with stronger LLM benefits for more complex queries.
II. RELATED WORK
Prior 3D grounding methods often use joint 3D-language training or CLIP-based open-vocabulary bridges, while LLM-Grounder uses an agent to handle compositional language and tool orchestration without training.
- 3D-language grounding benchmarks require understanding compositional semantics together with 3D scene structure, geometry, and object semantics.
- Jointly trained methods are limited to closed-vocabulary settings tied to object classes in ScanNet-derived benchmarks.
- CLIP-based open-vocabulary methods work well for simple noun phrases but lack compositional understanding of relations, attributions, and order.
- LLM-Grounder applies these agent capabilities to decompose grounding queries and orchestrate downstream visual grounders without training.
- LLM agents are characterized by goal-directed planning, tool use, and interaction with environmental feedback.
- In robotics, prior LLM-agent systems use planning and tools as high-level controllers for long-horizon, complex tasks.
III. METHOD
LLM-Grounder places an LLM agent at the center of grounding: it plans subtasks, invokes target and landmark tools, and reasons over their spatial feedback to select candidates.
- GPT-4 decomposes complex queries, orchestrates visual tools, and reasons over environmental feedback using spatial and commonsense knowledge.
- Planning: The agent first plans by describing observations and breaking the grounding problem into smaller subtasks.
- Tools: Target and landmark finders return candidate bounding boxes represented by centroids and sizes.
- Tools: The target finder computes candidate volumes, while the landmark finder computes Euclidean distances from target candidates to landmarks.
- Reasoning: Bounding boxes, volumes, and distances provide feedback for filtering false positives and rejecting candidates inconsistent with stated spatial relations.
- Tools: LERF and OpenScene implement the target and landmark finders as open-vocabulary CLIP-based 3D visual grounders.
IV. EXPERIMENTS
The experiments compare LLM-assisted grounding with CLIP-based grounders, evaluate closed-vocabulary performance against trained methods, and examine qualitative generalization in in-the-wild scenes.
- The evaluation first measures whether an LLM agent improves zero-shot open-vocabulary grounding over CLIP-based 3D visual grounders.
- The study also evaluates the method in a closed-vocabulary setting against closed-vocabulary and trained approaches.
- Qualitative examples on in-the-wild scenes assess the approach’s generalization.
A. Dataset
ScanRefer benchmarks 3D object localization in indoor scenes from natural-language descriptions. It provides standard accuracy metrics at two 3D bounding-box IoU thresholds.
- Dataset: ScanRefer contains 51,583 human-written descriptions of 11,046 objects from 800 ScanNet 3D scenes.The objects span 18 semantic categories.
- Dataset: The ScanRefer train, validation, and test splits contain 36,665, 9,508, and 5,410 descriptions, respectively.
- Dataset: The experiments use the first 14 validation scenes, comprising 998 text-and-3D-object pairs.
- Dataset: ScanRefer reports Accuracy@0.25 and Accuracy@0.5, using 0.25 and 0.5 as 3D bounding-box IoU thresholds.
B. Baseline Methods
ScanRefer uses an end-to-end 3D-text neural architecture to localize objects from natural-language input. Its pipeline extracts point-cloud features, proposes object boxes, and matches language to candidate boxes.
- Baseline Methods: ScanRefer processes the 3D point cloud into PointNet++ features before clustering points and proposing object bounding boxes.
- Baseline Methods: The pipeline fuses language features with point clusters and proposed boxes to determine which boxes the language refers to.
- Baseline Methods: Training uses supervision from text-and-bounding-box pairs together with ground-truth bounding boxes.
C. Results
LLM-Grounder is evaluated qualitatively and against baselines for zero-shot, open-vocabulary grounding. Adding an LLM agent improves both LERF- and OpenScene-based grounding on Accuracy@0.25.
- Results: LLM-Grounder produces qualitative examples, including demonstrations in in-the-wild scenes.
- Results: 5.0% and 17.1% Accuracy@0.25 improvements are achieved by adding an LLM agent to LERF and OpenScene, respectively.
- Results: The LLM agent improves zero-shot, open-vocabulary grounding compared with the baselines.
D. Ablation Study
The ablations examine visual and textual query difficulty. LLM agents help with some disambiguation and increasingly complex queries, but their advantage diminishes when visual cues or language complexity exceed their effective reasoning range.
- Visual difficulty: LLM agents improve grounding more for low visual-difficulty queries, where the referenced class has no distractor objects.Among 998 queries, 232 had low visual difficulty and 766 had high visual difficulty.
- Visual difficulty: A qualitative example shows the LLM agent using spatial reasoning to disambiguate the correct object instance.
- Text difficulty: Performance decreases as sentence complexity increases with or without an LLM agent, while LLM-equipped models perform better at higher complexities.
- Limitations: More nuanced visual cues are usually required for complex instance disambiguation, which the LLM agent lacks because it is blind.
- Text difficulty: The performance advantage of an LLM grows with query complexity but diminishes after a threshold.Query complexity is measured by counting nouns in the sentence.
V. CONCLUSION AND LIMITATIONS
LLM-Grounder uses LLMs as the central agent for 3D visual grounding and performs particularly well on complex text queries. The approach offers a zero-shot, open-vocabulary solution, but GPT-based reasoning introduces cost and latency constraints for deployment.
- LLM-Grounder leverages Large Language Models as the central agent for orchestrating the grounding process.
- The empirical evaluations show that LLM-Grounder excels particularly in handling complex text queries.
- LLM-Grounder offers a robust, zero-shot, open-vocabulary solution for 3D visual grounding tasks.
- Limitations: GPT-based reasoning can be computationally expensive, limiting deployment in resource-constrained environments.
- Limitations: GPT-model reasoning latency can bottleneck real-time robotic applications requiring rapid decision-making.