Source-linked AI summary
Visual Language Maps for Robot Navigation
Chenguang Huang, Oier Mees, Andy Zeng, Wolfram Burgard
TL;DR
VLMaps address the lack of spatial precision in visual-language navigation by fusing pretrained visual-language features with a 3D reconstruction of the physical world. Built from robot video and paired with LLMs, they localize open-vocabulary spatial goals and generate embodiment-specific obstacle maps, enabling more complex language-guided navigation than existing methods.
Problem
Existing visual-language navigation methods match observations to language but remain disjoint from environmental mapping, limiting the spatial precision available from classic geometric maps.
Method
VLMaps fuse pretrained visual-language features from image observations with a 3D reconstruction and use LLMs to translate instructions into map-localized open-vocabulary goals.
Results
Experiments show VLMaps enable more complex language instructions, including spatial open-vocabulary navigation, than existing methods and support more effective long-horizon multi-object navigation.
Takeaways & Limitations
Spatially anchoring visual-language features enables natural-language indexing, zero-shot spatial goal navigation, and runtime obstacle-map generation for robots with different embodiments.
Takeaways & Limitations
VLMaps remain sensitive to 3D reconstruction noise and odometry drift and cannot resolve object ambiguities in cluttered scenes containing similar objects.
Abstract
from arXiv · showhide
Grounding language to the visual observations of a navigating agent can be performed using off-the-shelf visual-language models pretrained on Internet-scale data (e.g., image captions). While this is useful for matching images to natural language descriptions of object goals, it remains disjoint from the process of mapping the environment, so that it lacks the spatial precision of classic geometric maps. To address this problem, we propose VLMaps, a spatial map representation that directly fuses pretrained visual-language features with a 3D reconstruction of the physical world. VLMaps can be autonomously built from video feed on robots using standard exploration approaches and enables natural language indexing of the map without additional labeled data. Specifically, when combined with large language models (LLMs), VLMaps can be used to (i) translate natural language commands into a sequence of open-vocabulary navigation goals (which, beyond prior work, can be spatial by construction, e.g., "in between the sofa and TV" or "three meters to the right of the chair") directly localized in the map, and (ii) can be shared among multiple robots with different embodiments to generate new obstacle maps on-the-fly (by using a list of obstacle categories). Extensive experiments carried out in simulated and real world environments show that VLMaps enable navigation according to more complex language instructions than existing methods. Videos are available at https://vlmaps.github.io.
I. INTRODUCTION
VLMaps addresses the gap between language-grounded visual observations and spatially precise environmental maps by fusing pretrained visual-language features with a 3D reconstruction. The resulting maps support zero-shot spatial navigation, embodiment-specific obstacle mapping, and stronger long-horizon multi-object navigation than baseline alternatives.
- Classic geometric maps support path planning but struggle with unseen language instructions, while end-to-end language navigation methods require copious training data.
- Existing VLM-based navigation methods generalize to unseen object goals but remain disjoint from environmental mapping and lose the spatial precision of geometric maps.
- VLMaps fuse pretrained visual-language features from image observations directly with a 3D reconstruction, enabling natural-language indexing of spatially anchored maps.
- With LLMs, VLMaps localize open-vocabulary spatial goals such as positions between landmarks, to the right of landmarks, or kitchen areas.
- VLMaps can generate obstacle maps for different robot embodiments from natural-language landmark categories, such as treating tables as obstacles for a mobile robot but traversable for a drone.
- Extensive experiments show more effective long-horizon multi-object navigation than CoW and LM-Nav, with particular strength on spatial open-vocabulary navigation tasks.
II. RELATED WORK
Prior work combines geometric mapping, semantic understanding, and zero-shot visual-language grounding, but existing approaches remain limited in spatial precision and representation sharing. VLMaps fuses visual-language features with 3D reconstruction to support language-indexed spatial maps.
- Semantic mapping augments 3D reconstructions with dense semantic labels or object-oriented representations for object-level mapping.
- Vision-and-language navigation methods often follow route instructions on topological graphs, limiting low-level planning and real-world applicability.
- Zero-shot methods such as LM-Nav and CoW handle unseen object goals but are less capable of finer-grained spatial queries.
- VLMaps fuse pretrained visual-language features with 3D reconstruction by back-projecting dense pixel embeddings onto reconstructed surfaces.
- Each map cell can average embeddings from multiple views of the same object, producing a persistent spatial visual-language representation.
B. Localizing Open-Vocabulary Landmarks
VLMaps localize user-specified landmarks by comparing top-down map embeddings with free-form text-category embeddings. The highest-similarity category is assigned to each map cell.
- Users provide a free-form language list of categories such as chairs, sofas, tables, furniture, or floor.
- CLIP converts each category into an embedding, while flattened map-cell embeddings form a matrix for comparison.
- The pixel-to-category similarity matrix stores how likely each top-down map cell belongs to each text category.
- Argmax assignment across categories reshapes into a segmentation map whose cells contain the selected language-list label.
C. Generating Open-Vocabulary Obstacle Maps
VLMaps generate embodiment-specific obstacle maps by combining geometric occupancy with language-defined obstacle categories. Different robots can therefore select different obstacle subsets from the same map.
- A VLMap supports runtime localization of natural-language obstacle categories for collision avoidance or shortest-path planning.
- The geometric obstacle map marks projected depth points as occupied after filtering points by height.
- Users define a potential obstacle list in language, with lower and upper height thresholds controlling retained 3D points.
- For each embodiment, selected obstacle-category masks are unioned and intersected with geometric occupancy to form the final obstacle map.
D. Zero-Shot Spatial Goal Navigation from Language
VLMaps support long-horizon navigation from language by decomposing commands into spatial subgoals and executing generated robot programs. These subgoals can reference relations, distances, directions, and repeated movements.
- Long-horizon navigation accepts landmark descriptions specified by natural-language instructions.
- VLMaps represent spatial goals such as between landmarks or a measured distance from a landmark, extending beyond object-centric references.
- An LLM interprets commands, decomposes them into subgoals, and can synthesize new sequences of robot API calls through few-shot prompting.
- Generated programs can express chained movements, spatial relations, orientation changes, distances, and repeated visits.
- Navigation primitives localize open-vocabulary landmarks in a pre-generated VLMap and use embodiment-specific obstacle maps with an off-the-shelf navigation stack.
IV. EXPERIMENTS
The experiments evaluate VLMaps against open-vocabulary navigation baselines, spatial language goals, cross-embodiment efficiency, and map-construction choices.
- The experiments compare VLMaps with recent open-vocabulary baselines on multi-object goal navigation.They also examine spatial language goals, cross-embodiment navigation efficiency, and different map-construction approaches.
A. Simulation Setup
Experiments use Habitat with Matterport3D for multi-object and spatial navigation, and AI2THOR for multi-embodiment obstacle-map evaluation. VLMaps is compared with LM-Nav, CoW, and CLIP Map, using sequential object subgoals and success-rate evaluation.
- Simulation Setup: Habitat with Matterport3D evaluates multi-object and spatial goal navigation, while AI2THOR supports multi-agent embodiment evaluation.AI2THOR is used because it supports agent types including LoCoBot and drone.
- Simulation Setup: Baselines include LM-Nav’s image-observation graph, CoW’s CLIP-GradCAM saliency map, and CLIP Map’s projected CLIP feature map.A ground-truth semantic map is also reported as a systems-level upper bound.
- Simulation Setup: The multi-object benchmark uses 91 sequences, each requiring four sequential subgoals selected from 30 object categories.A subgoal succeeds when the stop position is within one meter of the correct object; long-horizon performance uses success rate over 364 subgoals.
- Simulation Setup: The multi-object results specifically show favorable VLMaps performance on longer-horizon tasks with multiple subgoals.
- Simulation Setup: VLMaps performs consistently better than all baselines in multi-object navigation.CoW and CLIP Map produce considerable false positives, which can send planning toward wrong nearest masked targets.
C. Zero-Shot Spatial Goal Navigation from Language
VLMaps are evaluated on language-specified spatial goals requiring precise localization relative to objects. They outperform open-vocabulary baselines and support embodiment-specific obstacle maps that improve navigation efficiency.
- C. Zero-Shot Spatial Goal Navigation from Language: VLMaps outperform other open-vocabulary zero-shot navigation baselines on spatial goals specified by natural language.
- C. Zero-Shot Spatial Goal Navigation from Language: Spatial language navigation is harder than object navigation because it requires accurately reaching a location relative to a landmark.The text connects weaker CoW and CLIP Map localization with poorer performance on these tasks.
- D. Cross-Embodiment Navigation: VLMaps generate embodiment-specific obstacle maps from obstacle-category lists for navigation with both a LoCoBot and a drone.The evaluation uses more than 100 sequences of subgoals in AI2THOR.
- D. Cross-Embodiment Navigation: The drone with a customized drone map achieves higher SPL than the drone using a ground map while maintaining a similar success rate.Comparable SPL values between the drone with its drone map and the LoCoBot with a ground map indicate generalized navigation efficiency across embodiments.
- D. Cross-Embodiment Navigation: With a more efficient obstacle map, the drone flies over the sofa directly to the laptop, whereas the LoCoBot moves aside to avoid it.
E. Real Robot Experiments
Real-world experiments evaluated VLMaps on an HSR robot using RGB-D SLAM and 20 language-based spatial goals. The robot completed half of the navigation goals, including spatial, relative-position, and repeated instructions, while failures were linked to reconstruction and localization issues.
- Experimental setup: 374 frames and RTAB-Map RGB-D SLAM supported VLMaps evaluation on an HSR robot in a semantically rich indoor scene.The scene contained more than ten object classes, and global localization initialized the robot pose during inference.
- Experimental setup: 20 language-based spatial goals were tested with the robot initialized at different locations across runs.
- Results: 10 of 20 navigation goals were completed, including six spatial goals, three relative-to-current-position goals, and one repeated instruction.Examples included moving between landmarks, moving relative to the robot, and repeating a movement between two objects.
- Limitations: VLMaps remains sensitive to 3D reconstruction noise and odometry drift during navigation.These limitations constrain reliability in the real-world setting.
APPENDIX
The appendix documents navigation primitives, semantic-segmentation evaluation, and prompt-processing details. VLMaps generally outperformed CoW Map across frequent categories, but similar-looking objects and feature averaging produced segmentation errors.
- Navigation primitives: The appendix lists navigation primitives for object-relative, spatial, metric, directional, rotational, and repeated movements.Examples include moving between two objects, moving three meters relative to an object, absolute turns, and repeated traversals.
- Prompting: CLIP text retrieval used ensembles of prompt templates rather than single category-label prompts.Templates included variants such as “A photo of label” and “A picture of label”.
- Segmentation evaluation: Semantic segmentation was evaluated on Habitat top-down maps using Matterport3D RGB-D frames and excluding several unsupported categories.The evaluation reported top-10 frequent-category per-class IOU values.
- Segmentation results: VLMaps performed better than CoW Map in most top-10 frequent categories because CoW’s GradCAM saliency maps introduced noise and over-segmentation.
- Segmentation limitations: VLMaps received 0 IOU for “seating,” which the authors attribute to LSeg encoding the visually unseen class as a similar seen object.
- Segmentation limitations: Similar sofa and chair appearance can cause category confusion, while averaging embeddings at top-down-map locations can create noisy predictions.The authors link such errors to wrong planning behaviors and predictions such as “sink” on a table.
E. Prompts for Obstacle Maps Generation
Obstacle maps are generated by indexing landmarks with open-vocabulary category lists and taking unions of selected category masks. The selected obstacle categories differ for ground and flying robots, reflecting embodiment-specific traversability.
- LoCoBot obstacle map: For a LoCoBot, the method indexes categories including chairs, walls, tables, windows, stairs, and other objects, then unions selected masks into the obstacle map.The selected union excludes the listed floor category and includes wall, chair, table, window, stairs, and other.
- Drone obstacle map: For a drone, the method indexes a broader list including sofa, counter, ceiling, and ceiling lights, then unions wall, window, stairs, ceiling lights, cabinet, and other masks.
- Category vocabularies: The appendix lists distinct landmark vocabularies for the evaluated navigation settings.The category lists include room structures, furnishings, appliances, and other indoor objects.