Source-linked AI summary
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation
Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, Bernadette Bucher
TL;DR
Robots need semantic knowledge to search efficiently for objects in unfamiliar environments, while task-specific approaches can be closed-set and simulation-bound. VLFM combines occupancy frontiers with BLIP-2-based value maps for zero-shot target-driven navigation. It achieves state-of-the-art ObjectNav results across Gibson, HM3D, and MP3D and is demonstrated on Spot in a novel office environment.
Problem
Task-specific semantic navigation methods are limited to trained object categories and are often trained only on simulated data, motivating zero-shot navigation for unseen objects in novel environments.
Method
VLFM builds occupancy maps from depth observations, uses BLIP-2 to score RGB images against target-containing text prompts, and selects frontiers through a language-grounded value map.
Results
VLFM achieves state-of-the-art SPL results across Gibson, HM3D, and MP3D, including +19.2% SPL over SemExp in Gibson and +5.4% over PONI in MP3D.
Takeaways & Limitations
VLFM demonstrates that pre-trained foundation models can support spatially grounded semantic reasoning without task-specific training and can transfer to real-world robotic navigation.
Takeaways & Limitations
VLFM assumes target objects are easily visible from the robot camera’s default height and its semantic map is task-specific rather than reusable across sequential target-navigation tasks.
Abstract
from arXiv · showhide
Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. We introduce a zero-shot navigation approach, Vision-Language Frontier Maps (VLFM), which is inspired by human reasoning and designed to navigate towards unseen semantic objects in novel environments. VLFM builds occupancy maps from depth observations to identify frontiers, and leverages RGB observations and a pre-trained vision-language model to generate a language-grounded value map. VLFM then uses this map to identify the most promising frontier to explore for finding an instance of a given target object category. We evaluate VLFM in photo-realistic environments from the Gibson, Habitat-Matterport 3D (HM3D), and Matterport 3D (MP3D) datasets within the Habitat simulator. Remarkably, VLFM achieves state-of-the-art results on all three datasets as measured by success weighted by path length (SPL) for the Object Goal Navigation task. Furthermore, we show that VLFM's zero-shot nature enables it to be readily deployed on real-world robots such as the Boston Dynamics Spot mobile manipulation platform. We deploy VLFM on Spot and demonstrate its capability to efficiently navigate to target objects within an office building in the real world, without any prior knowledge of the environment. The accomplishments of VLFM underscore the promising potential of vision-language models in advancing the field of semantic navigation. Videos of real-world deployment can be viewed at naoki.io/vlfm.
I. INTRODUCTION
VLFM addresses semantic navigation toward unseen objects in unfamiliar environments by combining frontier exploration with spatially grounded vision-language reasoning. It achieves strong benchmark results while retaining zero-shot applicability to open-set and real-world navigation.
- VLFM is a zero-shot approach for target-driven semantic navigation toward unseen objects in novel environments.
- The method uses a pre-trained vision-language model to derive image-grounded semantic values for selecting promising exploration frontiers.
- 12% Gibson, 5% MP3D, and 3% HM3D absolute SPL increases over prior state-of-the-art approaches were reported for ObjectNav.
- Unlike task-specific methods limited to trained object categories and often simulation-only data, VLFM accepts an open-set of categories and is demonstrated on a real robot.
III. PROBLEM FORMULATION
ObjectNav asks a robot to find an instance of a target object category in an unseen environment using egocentric RGB-D, odometry, and discrete motion actions. Success requires stopping within 1 m of a target within 500 steps.
- ObjectNav requires searching for an instance of a specified target object category in a previously unseen environment.
- The robot relies on an egocentric RGB-D camera and odometry providing relative distance and heading from its starting pose.
- The action space includes 0.25 m forward moves, 30° turns and vertical looks, plus STOP.
- An episode succeeds when STOP is called within 1 m of any target instance in 500 or fewer steps.
IV. VISION-LANGUAGE FRONTIER MAPS
VLFM organizes navigation into initialization, exploration, and goal navigation, maintaining maps that connect explored-space frontiers with target-oriented waypoint selection. Depth and odometry support iterative top-down obstacle and frontier mapping.
- Initialization: VLFM initializes its frontier and value maps by rotating in place for a complete turn.
- Exploration: During exploration, VLFM updates both maps, creates frontier waypoints, and selects the most valuable waypoint for locating the target category.
- Goal navigation: After detecting a target object, the robot transitions from frontier exploration to goal navigation.
- Occupancy and frontiers: The occupancy map uses depth and odometry observations to build a top-down 2D map of encountered obstacles.
- Occupancy and frontiers: Frontiers represent boundaries between explored and unexplored space and provide candidate waypoints for continued exploration.
B. Value map generation
VLFM builds a semantic value map from BLIP-2 scores computed on RGB observations and target-containing text prompts, while depth excludes occluded regions. Confidence-weighted updates stabilize values when views overlap.
- The value map assigns each explored-area pixel a semantic relevance value for locating the target and ranks frontiers by that value.
- The map contains separate semantic-value and confidence channels and is updated iteratively using depth and odometry observations.
- BLIP-2 computes cosine similarity directly between the current RGB image and a target-containing text prompt, then projects scores onto the top-down value map.
- Confidence depends on pixel position relative to the optical axis, with confidence 1 on-axis and 0 at the horizontal FOV edges.
- When the current FOV overlaps previously seen regions, semantic values and confidence scores are updated through confidence-weighted averaging.
- Depth-derived masking excludes regions occluded by obstacles before semantic scores update the value map.
C. Object detection
VLFM uses pre-trained detectors to determine whether a target object is visible, supporting open-vocabulary detection beyond COCO categories.
- Pre-trained object detectors infer bounding boxes with semantic labels to determine whether the target object is visible.
- YOLOv7 handles target objects within COCO categories, while Grounding-DINO detects categories outside COCO.
- When detection succeeds, Mobile-SAM extracts the object contour, and depth identifies the nearest point for waypoint navigation.
D. Waypoint navigation
VLFM navigates toward either an unexplored frontier or a detected target using a waypoint-navigation policy based on visual geometry and odometry.
- The robot receives a frontier waypoint before detection or a target-object waypoint after detection.
- A PointNav policy determines actions for reaching the current waypoint using visual observations and odometry.
- The trained PointNav policy uses egocentric depth, relative distance, and heading, without RGB images.
- VLFM can replace PointNav with another method that guides the robot to visually observed frontier or target waypoints.
V. EXPERIMENTAL SETUP
The evaluation uses Habitat validation splits across three real-world 3D-scan datasets, reports SR and SPL, and compares VLFM with zero-shot and supervised baselines.
- The validation benchmarks are Gibson, HM3D, and MP3D, containing 1000, 2000, and 2195 episodes respectively.HM3D spans 20 scenes and 6 categories; MP3D spans 11 scenes and 21 categories.
- Success rate and Success weighted by inverse Path Length measure task performance, with SPL rewarding shorter successful paths.
- Zero-shot baselines include CoW, ESC, SemUtil, and ZSON, which use CLIP, language models, object detections, or transferred ImageNav representations.
- Supervised comparisons include PONI, PIRLNav, RegQLearn, and SemExp, using task-specific training, demonstrations, or reinforcement learning.
- VLFM outperforms previous zero-shot methods and performs competitively against methods trained directly on ObjectNav.
VI. RESULTS
The experiments examine VLFM’s ObjectNav performance against trained and zero-shot methods, value-fusion choices, and real-world deployability.
- The experiments compare VLFM with trained and zero-shot methods across multiple datasets.
- They evaluate how fusing current and previously seen values affects VLFM performance.
- They test whether VLFM can be deployed successfully in the real world.
A. Benchmark results
VLFM achieves strong ObjectNav performance across Gibson, HM3D, and MP3D, including improvements over zero-shot and trained approaches. Its current limitation is single-floor navigation, which causes failures on stair-dependent episodes.
- VLFM exceeds prior zero-shot methods across all three benchmarks, with the largest reported gains in Gibson.Compared with SemUtil, VLFM gains +11.7% SPL and +14.7% success in Gibson; compared with ESC, it gains +8.1% SPL and +13.3% success in HM3D, and +3.3% SPL and +7.7% success in MP3D.
- VLFM surpasses trained ObjectNav methods in Gibson and MP3D, establishing new state-of-the-art metrics there.It improves over SemExp by +19.2% SPL and +19.0% success in Gibson, and over PONI by +5.4% SPL and +4.6% success in MP3D.
- In HM3D, VLFM trails PIRLNav in success by 11.6% but leads it by 3.3% SPL despite being entirely zero-shot.PIRLNav was trained on 77k human demonstrations from the HM3D training split.
- VLFM’s Gibson advantage is attributed partly to the absence of stair-required episodes, unlike HM3D and MP3D.PIRLNav can traverse stairs, while VLFM’s simulator setup supports only single-floor episodes.
- VLFM fails 14.6% of HM3D episodes and 9.6% of MP3D episodes requiring stair traversal to reach targets.
- Higher Gibson and HM3D performance than MP3D is attributed to differences in 3D-scan quality.MP3D has lower visual fidelity than HM3D, while Gibson scenes were repaired and verified to be free of holes and artifacts.
B. Ablations
The ablation section examines how VLFM updates semantic values when previously observed areas are revisited and demonstrates a real-world Spot deployment implementation.
- VLFM updates revisited areas by combining previous and current value-map estimates.The evaluated alternatives are Replacement, Unweighted avg., and the full Weighted avg. approach.
- Replacement overwrites the previous value, whereas Unweighted avg. averages the previous and current values.
- Weighted avg. combines values using confidence scores to weight the average.
- On Spot, VLFM uses the Boston Dynamics API for waypoint navigation and the gripper camera to provide inputs to BLIP-2.ZoeDepth is used because the gripper camera has limited depth-sensing range.
- BLIP-2, GroundingDINO, MobileSAM, and ZoeDepth run in real time on a laptop with an RTX 4090 MaxQ Mobile GPU and 16 GB of VRAM.
VII. CONCLUSION
VLFM is a modular zero-shot framework that spatially grounds vision-language reasoning for ObjectNav in novel environments. Simulated benchmarks and Spot demonstrations support its reported zero-shot navigation performance and real-world viability, while several scope limitations remain.
- VLFM selects frontier waypoints using semantic prompts and pre-trained models, then detects targets and navigates with modular components.The framework’s components can be replaced as improved models become available.
- VLFM achieves state-of-the-art zero-shot ObjectNav performance in simulated 3D home environments.
- Spot demonstrations in an office building support VLFM’s viability in real-world scenarios.
- VLFM assumes target objects are easily visible from the robot camera’s default height.Future work could direct the camera actively or use manipulation to search inside closed drawers.
- VLFM’s semantic value map is task-specific and cannot support sequential navigation to different objects or other language-specified navigation tasks.Future work could investigate alternative prompts, value maps, and semantic tracking for long-horizon and multitask execution.
- The results highlight zero-shot foundation-model reasoning as a way to provide spatially grounded semantics without task-specific training.