Source-linked AI summary
Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation
Ziyu Zhu, Xilin Wang, Yixuan Li, Zhuofan Zhang, Xiaojian Ma, Yixin Chen, Baoxiong Jia, Wei Liang, Qian Yu, Zhidong Deng, Siyuan Huang, Qing Li
TL;DR
Embodied agents must understand observed 3D scenes while deciding where to explore, a capability missing from static-reconstruction-based 3D-VL models. MTU3D unifies online RGB-D spatial memory, grounding and frontier selection, and trajectory pre-training, achieving state-of-the-art results across navigation and embodied question-answering benchmarks.
Problem
Existing 3D-VL models primarily ground objects in static reconstructed scenes and typically lack active perception and exploration capabilities.
Method
MTU3D learns online query representations from RGB-D frames, jointly optimizes object grounding and frontier exploration, and pre-trains decisions on over one million diverse trajectories.
Results
13.7%, 23.0%, and 9.1% SR improvements are reported on HM3D-OVON, GOAT-Bench, and SG3D, while embodied question answering also improves.
Takeaways & Limitations
Bridging visual grounding and exploration supports efficient, versatile embodied navigation across categories, language descriptions, reference images, and real-world deployment.
Abstract
from arXiv · showhide
Embodied scene understanding requires not only comprehending visual-spatial information that has been observed but also determining where to explore next in the 3D physical world. Existing 3D Vision-Language (3D-VL) models primarily focus on grounding objects in static observations from 3D reconstruction, such as meshes and point clouds, but lack the ability to actively perceive and explore their environment. To address this limitation, we introduce \underline{\textbf{M}}ove \underline{\textbf{t}}o \underline{\textbf{U}}nderstand (\textbf{\model}), a unified framework that integrates active perception with \underline{\textbf{3D}} vision-language learning, enabling embodied agents to effectively explore and understand their environment. This is achieved by three key innovations: 1) Online query-based representation learning, enabling direct spatial memory construction from RGB-D frames, eliminating the need for explicit 3D reconstruction. 2) A unified objective for grounding and exploring, which represents unexplored locations as frontier queries and jointly optimizes object grounding and frontier selection. 3) End-to-end trajectory learning that combines \textbf{V}ision-\textbf{L}anguage-\textbf{E}xploration pre-training over a million diverse trajectories collected from both simulated and real-world RGB-D sequences. Extensive evaluations across various embodied navigation and question-answering benchmarks show that MTU3D outperforms state-of-the-art reinforcement learning and modular navigation approaches by 14\%, 23\%, 9\%, and 2\% in success rate on HM3D-OVON, GOAT-Bench, SG3D, and A-EQA, respectively. \model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images. These findings highlight the importance of bridging visual grounding and exploration for embodied intelligence.
1. Introduction
Embodied scene understanding requires agents to combine visual grounding with active exploration, but existing 3D-VL models rely on static reconstructions and typically lack exploration capabilities. MTU3D addresses this gap with online spatial memory, joint grounding-exploration learning, and large-scale trajectory pre-training.
- Existing 3D-VL models use static 3D reconstructions for grounding, while embodied agents operate in partially observable environments and need active perception.
- The paper identifies online RGB-D representation learning, joint object grounding and exploration, and large-scale trajectory training as three central challenges.
- MTU3D builds global spatial memory from local queries extracted directly from RGB-D frames using semantic and spatial features from 2D foundation models.
- MTU3D represents unexplored regions as frontier queries and jointly optimizes object grounding with frontier selection.
- 13.7%, 23.0%, and 9.1% SR improvements are reported on HM3D-OVON, GOAT-Bench, and SG3D, respectively.The paper also reports 2.4%, 13.0%, and 6.3% SPL improvements on those benchmarks, plus LM-SR and LLM-SPL gains for embodied question answering.
- The framework supports open-vocabulary navigation, multimodal lifelong navigation, sequential navigation, and active embodied question answering.
2. Related work
3D-VL research has advanced spatial grounding, semantics, scene understanding, and multiple-task modeling. However, reliance on static meshes or point clouds limits applicability to real-time embodied agents that must explore and perceive environments online.
- 3D-VL models ground language in 3D scenes through spatial relationships, object semantics, and scene structures.
- Existing approaches include task-specific, pretrained, and unified models that address individual or multiple 3D-VL tasks.
- Static meshes and point clouds make many existing 3D-VL models unsuitable for embodied agents requiring real-time exploration and perception.
3. MTU3D
MTU3D processes partial RGB-D sequences into persistent object and frontier queries, jointly selects grounding or exploration targets, and trains decisions on diverse simulated and real trajectories. Its architecture forms a continuous perception-action loop through spatial memory, reasoning, planning, and new observations.
- Architecture: MTU3D extracts object and frontier queries from partial RGB-D sequences and dynamically stores them in a spatial memory bank.
- Online Query Representation Learning: Each RGB-D observation combines an RGB image, depth image, and camera pose to produce segment-level 2D and 3D representations.
- Online Query Representation Learning: Local queries encode global 3D boxes, instance masks, open-vocabulary embeddings, decoder features, and confidence scores.
- Online Query Representation Learning: Historical and current queries are merged using bounding-box IoU, exponential moving averages, and unioned instance masks.
- Online Query Representation Learning: Frontiers represent boundaries between explored and unexplored areas and provide 3D exploration targets that are recalculated during navigation.
- Unified Grounding and Exploration: A spatial reasoning transformer scores object and frontier candidates against language or image goals, after which a shortest-path planner guides navigation.
- Vision-Language-Exploration Training: Vision-Language-Exploration pre-training jointly trains grounding and exploration decisions on over one million trajectories from simulated and real-world data.
4. Experiment
MTU3D is evaluated across navigation, embodied question answering, ablations, qualitative scenarios, and real-robot deployment. Results show strong performance across tasks, with gains in success, efficiency, memory-supported lifelong navigation, and real-time operation.
- Open Vocabulary Navigation: MTU3D achieves the highest SR in HM3D-OVON Val Unseen at 40.8%, while its SPL trails Uni-Navid in some settings.The short trajectories favor video-based models that can navigate directly after recognizing a target.
- Task-oriented Sequential Navigation: MTU3D reaches 23.8% s-SR, 8.0% t-SR, and 16.5% SPL on SG3D, leading the reported sequential-navigation metrics.SG3D requires consistency across multiple task steps and is more difficult than GOAT-Bench and HM3D-OVON.
- Multi-modal Lifelong Navigation: MTU3D records the highest SR across GOAT-Bench settings, including 52.2% Val Seen, 48.4% Val Seen Synonyms, and 47.2% Val Unseen.It also achieves 30.5% SPL in Val Seen and 27.7% SPL in Val Unseen.
- Active Embodied Question Answering: MTU3D-enhanced GPT-4V improves LLM-SR from 41.8% to 44.2% and LLM-SPL from 7.5% to 37.0% on embodied question answering.GPT-4o with MTU3D reaches 51.1% LLM-SR and 42.6% LLM-SPL.
- Ablation Studies: VLE pre-training raises SR from 27.8% to 33.3% in OVON, 22.2% to 36.1% in GOAT, and 22.9% to 27.9% in SG3D.The improvement is reported across different task settings and distributions.
- Ablation Studies: At exploration step 6, MTU achieves 50.0% SR and 35.3% SPL versus frontier exploration’s 33.3% SR and 30.3% SPL.The model uses semantic guidance rather than selecting the nearest frontier blindly; spatial memory also improves SR across object, description, and image goals.
5. Conclusions
MTU3D bridges visual grounding and exploration for embodied scene understanding, with joint optimization supporting efficient navigation across diverse input modalities. Its VLE training and real-world deployment yield strong benchmark performance and evidence of generalization.
- MTU3D bridges visual grounding and exploration to advance embodied scene understanding.
- Jointly optimizing grounding and exploration supports efficient navigation across diverse input modalities.
- Vision-Language-Exploration training leverages large-scale trajectories and achieves state-of-the-art performance on multiple Embodied AI benchmarks.
- Real-world deployment demonstrates generalization to realistic 3D environments using mixed simulation and real-world training data.
A. More Implementation Details
The implementation refines local RGB-D-derived queries, matches them to historical global queries through geometric similarity, and updates a dynamic spatial memory. Spatial reasoning then combines visual features, language instructions, and query relationships for exploration decisions.
- Local Query Refinement: Local query refinement uses decoder layers to retrieve object-relevant information from input features.
- Spatial Reasoning: Masked attention focuses queries on localized relevant regions, improving fine-grained feature capture while reducing computational overhead.
- Query Matching and Fusion: A dynamic spatial memory bank matches current local and previous global queries using bounding-box IoU, filtering low-similarity pairs with threshold ϵ.
- Query Matching and Fusion: Unmatched local boxes enter the global representation, while matched queries fuse masks by union and other representations by weighted averaging.
- Spatial Reasoning: The spatial reasoning transformer concatenates global and frontier queries, attends to current features, incorporates language goals, and models spatial relationships.
B. Benchmarks and baseline
The evaluation covers open-vocabulary, multimodal lifelong, sequential, and question-directed exploration settings, comparing MTU3D with supervised, reinforcement-learning, modular, video-based, and language-model-based baselines.
- HM3D-OVON: HM3D-OVON evaluates open-vocabulary navigation, using supervised, corrective-action, reinforcement-learning, video-based, and training-free baselines.
- Goat-Bench: GOAT-Bench evaluates multimodal lifelong navigation with image, class, and description goals, comparing modular and reinforcement-learning approaches.
- SG3D: SG3D evaluates sequential navigation to objects in a specified order, including modular video-agent and unified reinforcement-learning baselines.
- A-EQA: A-EQA evaluates exploration for answering questions, with MTU3D generating trajectories and videos while GPT-4o/V handles question answering.
C. Trajectory collection
The trajectory collection strategy explores episodes by updating the global map, selecting goals and frontiers, and recording decisions until success or a terminal failure status. Randomized action selection is used because relying only on random or ground-truth actions can cause overfitting.
- Collection strategy: Trajectory collection randomly selects actions while updating explored maps, visited frontiers, visible identifiers, and episode status.
- Motivation: Using only random or ground-truth actions can lead to model overfitting, motivating the collection strategy’s broader action variation.
- Decision process: At each step, the strategy selects the closest goal and checks visibility, reachability, and whether a frontier offers a better route.
- Decision process: Visible and reachable goals produce successful trajectories, while closer frontiers trigger continued exploration when appropriate.
- Termination: The procedure labels unreachable, invisible, and failed outcomes before returning the decision list and terminal status.