Source-linked AI summary
ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings
Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, Dhruv Batra
TL;DR
Open-world ObjectNav requires agents to find object instances from natural-language goals in unexplored environments without ObjectNav-specific rewards or demonstrations. ZSON trains semantic-goal navigation from ImageNav using multimodal image-text embeddings, then evaluates language-projected goals, achieving higher success than prior zero-shot methods across three datasets and showing room-aware behavior for compound instructions.
Problem
Open-world ObjectNav seeks scalable ways for agents to find object instances from free-form descriptions without requiring ObjectNav rewards, demonstrations, or a fixed category taxonomy.
Method
ZSON projects image goals and language descriptions into a shared multimodal semantic embedding space using CLIP, trains SemanticNav on ImageNav, and applies the learned navigator to open-world ObjectNav.
Results
ZSON outperforms previous zero-shot ObjectNav methods across Gibson, MP3D, and HM3D evaluations, with reported success gains of 4.2%–20.0%, and exhibits room-aware behavior for compound instructions.
Takeaways & Limitations
The approach supports scalable zero-shot open-world navigation and can respond to explicit or inferred room information in natural-language instructions.
Takeaways & Limitations
Agents may struggle when targets occur in unusual locations, and biases in training environments may affect deployment in non-traditional settings.
Abstract
from arXiv · showhide
We present a scalable approach for learning open-world object-goal navigation (ObjectNav) -- the task of asking a virtual robot (agent) to find any instance of an object in an unexplored environment (e.g., "find a sink"). Our approach is entirely zero-shot -- i.e., it does not require ObjectNav rewards or demonstrations of any kind. Instead, we train on the image-goal navigation (ImageNav) task, in which agents find the location where a picture (i.e., goal image) was captured. Specifically, we encode goal images into a multimodal, semantic embedding space to enable training semantic-goal navigation (SemanticNav) agents at scale in unannotated 3D environments (e.g., HM3D). After training, SemanticNav agents can be instructed to find objects described in free-form natural language (e.g., "sink", "bathroom sink", etc.) by projecting language goals into the same multimodal, semantic embedding space. As a result, our approach enables open-world ObjectNav. We extensively evaluate our agents on three ObjectNav datasets (Gibson, HM3D, and MP3D) and observe absolute improvements in success of 4.2% - 20.0% over existing zero-shot methods. For reference, these gains are similar or better than the 5% improvement in success between the Habitat 2020 and 2021 ObjectNav challenge winners. In an open-world setting, we discover that our agents can generalize to compound instructions with a room explicitly mentioned (e.g., "Find a kitchen sink") and when the target room can be inferred (e.g., "Find a sink and a stove").
1 Introduction
ZSON addresses the closed-world limits of ObjectNav with a zero-shot, open-world approach that transfers ImageNav training into language-conditioned object navigation. It uses a shared semantic embedding space and reports gains across three datasets, plus room-aware behavior for compound instructions.
- ObjectNav traditionally uses a closed vocabulary, motivating methods that find objects described in open-world language.
- ZSON projects image and language goals into a shared semantic space, separating semantic-goal representation from navigation-policy learning.CLIP maps image-goals and object-goals into representations intended to encode the same target semantics.
- Image-goal training avoids ObjectNav rewards, 3D semantic annotations, and potentially costly human demonstrations, reducing labeling requirements.Image-goals can be procedurally generated by sampling points in 3D environments.
- 31.3% success in Gibson is a 20.0% absolute improvement over previous zero-shot results, while MP3D reaches 15.3% with a 4.2% gain.On HM3D, zero-shot SPL matches a state-of-the-art method trained with 40k human demonstrations.
- Pretrained visual encoders improve zero-shot ObjectNav success by 9.4%–10.4%, while expanding training environments from 72 to 800 improves success by 6.6%.The encoder gains exceed the corresponding ImageNav improvements of 4.5%–5.8%.
- SemanticNav agents respond to explicit and inferred room information, including “bathroom sink,” “kitchen sink,” and “sink and stove” instructions.
2 Related Work
Related work applies image-text alignment and zero-shot techniques to visual navigation, but ZSON emphasizes semantic goal embeddings and learned navigation without explicit object localization.
- Image-Text Alignment Models: Multimodal alignment models learn shared image-text representations from large image-caption datasets, including CLIP, ALIGN, and BASIC.
- CLIP for Visual Navigation: Using CLIP embeddings directly in navigation still requires ObjectNav rewards or demonstrations, which are costly to collect at scale.
- Zero-Shot ObjectNav: ZER transfers ImageNav to zero-shot ObjectNav through independent modality encoders, but image-goal embeddings may lack semantic information.
- Zero-Shot ObjectNav: ZSON uses a goal embedding space designed to capture semantics and empirically demonstrates its benefits.
- Zero-Shot ObjectNav: Unlike CoW’s GradCAM localization and heuristic exploration, ZSON learns a navigation policy without explicit object localization.
3 Preliminaries: Image-Text Alignment and Image-Goal Navigation
The paper combines image-text alignment with ImageNav, whose procedurally generated episodes provide scalable training without scene annotation.
- Image-Text Alignment Models: Image-text alignment maps corresponding images and text into a shared embedding space using dual encoders and contrastive learning.
- Image-Goal Navigation: ImageNav asks agents to reach the viewpoint where an RGB goal image was captured, succeeding when STOP occurs within 1.0m.
- Image-Goal Navigation: ImageNav episodes can be procedurally generated from starting positions and reachable goal viewpoints without annotating objects or rooms.Dataset scale is therefore limited by available training environments rather than scene labels.
4 Approach
ZSON trains a SemanticNav policy on CLIP-encoded image-goals and deploys it on CLIP-encoded language goals, enabling zero-shot ObjectNav without direct task training.
- Semantic-Goal Navigation: Image-goals are encoded with CLIP’s visual encoder to train SemanticNav agents at scale, then deployed for zero-shot object-goal navigation.
- Semantic-Goal Navigation: CLIP visual embeddings represent image-goal semantics while discarding some low-level features that do not correlate with web-scraped captions.
- Agent Architecture: A pretrained ResNet-50 encodes egocentric RGB observations, which are combined with the goal representation and previous action before recurrent policy prediction.
- Training: SemanticNav is trained with reinforcement learning using DD-PPO, color jitter, random translation, and an ImageNav reward function.
- Training: The ImageNav reward encourages reaching the goal, aligning the camera with its heading, and efficient navigation through success, angle, distance, and slack terms.
- ObjectNav Deployment: Object-goals are encoded with CLIP’s text encoder so their embeddings should be close to the visual embeddings used during training.
- ObjectNav Deployment: The method uses object names directly as inputs without prompt engineering, such as “sofa” rather than a templated phrase.
5 Experimental Findings and Qualitative Results
Experiments evaluate ZSON across ImageNav and three ObjectNav datasets, compare it with zero-shot baselines, ablate training choices, and examine complex language instructions. ZSON improves zero-shot ObjectNav performance and shows room-aware behavior in qualitative trials.
- Experimental Setup: The evaluation covers one ImageNav dataset and three ObjectNav datasets, using standard success rate and SPL metrics across two agent configurations.The ObjectNav datasets are Gibson, HM3D, and MP3D; configuration A is used for ImageNav and Gibson ObjectNav, while configuration B is used for HM3D and MP3D.
- Zero-Shot Object-Goal Navigation: 4.0% absolute and 35% relative gains in zero-shot ObjectNav success over ZER occur under matched Gibson and ResNet-9 settings.The matched comparison reports an increase from 11.3% to 15.3% success and attributes the difference to learning navigation over semantic-goal embeddings rather than image-goal embeddings.
- Additional Ablations: 9.4%–10.4% ObjectNav success improvements from OVRL pretraining exceed its 4.5%–5.8% ImageNav gains, indicating stronger effects on zero-shot transfer.The ablation describes a substantially larger downstream impact than the improvement on the ImageNav training task.
- Additional Ablations: 6.6% higher ObjectNav success follows training on 800 HM3D environments instead of 72 Gibson environments, despite a 0.9% ImageNav decrease.The authors interpret this trend as evidence that training-environment diversity is particularly useful for zero-shot ObjectNav.
- Qualitative Analysis: Qualitative trials show room-aware navigation for explicit instructions such as “bathroom sink” and inferred-room instructions such as “sink and a stove.”Each instruction is evaluated with five stochastic rollouts; the agent avoids the wrong room and navigates directly when target rooms are visible.
6 Discussion
The paper presents ZSON as a scalable zero-shot approach to open-world ObjectNav and identifies both its practical strengths and deployment boundary.
- ZSON projects image-goals into a semantic-goal embedding space using CLIP, creating a SemanticNav task without annotated 3D environments or human demonstrations.The approach supports scalable training and zero-shot transfer to open-world ObjectNav.
- SemanticNav agents outperform previous zero-shot ObjectNav methods and show navigation patterns consistent with understanding compound instructions such as “Find a sink and a stove.”The compound-instruction behavior is reported as an observation in an open-world setting.
- Limitations and Impact: Agents may struggle when navigation targets appear in unusual locations, such as a stove in a bedroom.Biases in training environments may exaggerate this issue and affect deployment in non-traditional settings.
- Limitations and Impact: The authors suggest mitigating these biases and using SemanticNav’s natural-language interface to guide exploration in such scenarios.
Checklist
The checklist records affirmative reporting across claims, limitations, ethics, assets, experiments, and reproducibility, while marking theoretical-results items as not applicable.
- The paper states that its main claims accurately reflect its contributions and scope, and that its limitations are described.
- Potential negative societal impacts are discussed, and the paper states that it conforms to ethics review guidelines.
- Theoretical-results assumptions and complete proofs are marked not applicable.
- For experiments, the checklist states that error bars, compute resources, training details, and reproducibility materials are reported.
- For existing or newly curated assets, the checklist states that creators, licenses, URLs, consent, and data risks are addressed.
- Human-subject and crowdsourcing items are marked not applicable where indicated.
- Table 4 compares ObjectNav methods by whether they are open-world and zero-shot.
A Extended Discussion of Related Work
Related work distinguishes ObjectNav methods by open-world capability and use of ObjectNav supervision, positioning ZSON as combining both properties through shared semantic embeddings.
- Open-world methods accept objects beyond a closed category vocabulary, while zero-shot methods avoid ObjectNav annotations such as labeled environments or human demonstrations.
- EmbCLIP supports open-vocabulary object descriptions through CLIP text encoding but is trained with labeled 3D environments, so it is not zero-shot.
- ZER is zero-shot through ImageNav training but cannot perform open-world ObjectNav because it maps a closed object-category set into image-goal embeddings.
- CoW supports open-world ObjectNav with CLIP encoders but uses a heuristic policy that cannot learn indoor layout regularities such as stoves being found in kitchens.
- ZSON uses CLIP to place image-goals and object-goals in a common semantic-goal space, enabling ImageNav training followed by zero-shot transfer to open-world ObjectNav.
- The authors propose multi-task training with image- and object-derived semantic-goals as future work, noting that this would not remain zero-shot.
- Table 5 is identified as reporting finetuning results using ObjectNav annotations, with asterisks marking reproduced results.
B Results of ObjectNav Finetuning
Finetuning ZSON agents with ObjectNav annotations substantially improves ObjectNav performance, showing the effect of adding task-specific supervision after SemanticNav pretraining.
- Finetuning for 25M steps in MP3D raises success from 15.3% to 22.9% and SPL from 4.8% to 9.2%.The 9.2% SPL exceeds the cited RGB-only OVRL result of 7.0% SPL.
- Finetuning for 100M steps in HM3D raises success from 25.5% to 49.6% and SPL from 12.6% to 27.0%.
- The HM3D finetuned result exceeds the MP3D-trained comparison agent by 16.8% absolute in success and 14.7% absolute in SPL.
- The ablation section identifies additional visual-encoder comparisons for SemanticNav agents.
C Additional Ablation Experiments
The additional ablations compare visual encoders with and without OVRL pretraining and report associated ImageNav and ObjectNav success-rate changes. Default training hyperparameters are listed separately in Table 7.
- Without pretraining, switching from ResNet-9 to ResNet-50 drops ImageNav SR by 2.9% and ObjectNav SR by 4.3%.
- With OVRL pretraining, switching to the larger ResNet-50 improves performance on all tasks.
- Table 7 lists the default hyperparameters used to train SemanticNav agents.
D Additional Training Details
The training details specify implementation choices and reduce environments per GPU for Gibson ablations to accommodate encoder-dependent GPU memory constraints.
E Additional Qualitative Results
Additional qualitative trials show successful navigation across several object categories, alongside failures involving stopping short, poor exploration, and visually similar objects. Performance varies across targets and starting positions.
- The agent successfully navigates to stairs, tables, and televisions, demonstrating qualitative versatility across object categories.
- The agent finds a refrigerator in 4 of 5 trials but once stops before entering the kitchen despite viewing it.
- The agent finds a bathtub in 3 of 5 trials, with two failures caused by not entering the bathroom.
- The agent finds a sofa in 3 of 5 trials but stops in the dining area twice and never enters the room containing two sofas.
- The agent fails to find a desk in all five trials, often stopping near a table and never entering the room containing the desk.
- For dresser navigation, the agent succeeds in 4 of 5 trials from Start A but fails in all five from Start B after entering the kitchen near cabinets.