Source-linked AI summary
ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation
Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, Xin Eric Wang
TL;DR
Zero-shot object navigation must locate novel objects in unknown environments without costly navigation training that generalizes poorly under distribution shifts. ESC combines pre-trained vision-language grounding, commonsense room-object reasoning, and soft constraints integrated with frontier exploration. Across MP3D, HM3D, and RoboTHOR, it achieves state-of-the-art results and large gains over zero-shot baselines, including 288% relative SR improvement over CoW on MP3D.
Problem
Object navigation methods trained on limited objects and environments often perform poorly on novel objects or environments, motivating zero-shot navigation without navigation-data training.
Method
ESC uses pre-trained vision-language grounding and commonsense language models, then represents commonsense as soft constraints through PSL to guide frontier-based exploration.
Results
ESC achieves state-of-the-art zero-shot object navigation results across MP3D, HM3D, and RoboTHOR, including 288% relative SR improvement over CoW on MP3D.
Takeaways & Limitations
The results support using pre-trained commonsense knowledge with frontier-based exploration for generalized zero-shot object navigation.
Takeaways & Limitations
ESC uses a fixed strategy to combine commonsense knowledge, and future work considers learning frontier selection through limited finetuning.
Abstract
from arXiv · showhide
The ability to accurately locate and navigate to a specific object is a crucial capability for embodied agents that operate in the real world and interact with objects to complete tasks. Such object navigation tasks usually require large-scale training in visual environments with labeled objects, which generalizes poorly to novel objects in unknown environments. In this work, we present a novel zero-shot object navigation method, Exploration with Soft Commonsense constraints (ESC), that transfers commonsense knowledge in pre-trained models to open-world object navigation without any navigation experience nor any other training on the visual environments. First, ESC leverages a pre-trained vision and language model for open-world prompt-based grounding and a pre-trained commonsense language model for room and object reasoning. Then ESC converts commonsense knowledge into navigation actions by modeling it as soft logic predicates for efficient exploration. Extensive experiments on MP3D, HM3D, and RoboTHOR benchmarks show that our ESC method improves significantly over baselines, and achieves new state-of-the-art results for zero-shot object navigation (e.g., 288% relative Success Rate improvement than CoW on MP3D).
1. Introduction
ESC addresses zero-shot object navigation by combining open-world scene understanding with commonsense reasoning and soft constraints for exploration. It achieves state-of-the-art performance across three benchmarks, including large relative gains over comparable baselines.
- Motivation: Existing methods often generalize poorly to novel objects and environments because they rely on dataset-specific training or limited exploration strategies.This motivates zero-shot navigation without additional annotated trajectory data.
- Motivation: Object navigation requires semantic scene understanding and commonsense reasoning to infer where a goal object is likely located.For example, the agent can infer that a fireplace is likely in a living room and explore an unseen area there.
- Approach: ESC represents uncertain commonsense relationships as soft logic predicates and integrates them with frontier-based exploration to select navigation actions.Probabilistic Soft Logic assigns continuous-valued knowledge to frontiers, enabling training-free exploration.
- Results: ESC achieves state-of-the-art zero-shot object navigation results and outperforms baselines across MP3D, HM3D, and RoboTHOR.The reported improvements include 285% relative gains in SPL and SR over CoW on MP3D, plus strong improvements over ZSON.
- Approach: ESC uses pre-trained vision-language grounding and commonsense language models for open-world scene understanding and room-object reasoning.The framework first grounds scene content and then infers correspondences between rooms and objects.
2. Problem Definition
Zero-shot object navigation requires an agent to find unseen goal objects in new environments without object-goal labels or navigation-data training. This setting is more demanding than supervised navigation because both environments and target objects are novel.
- Task Definition: In conventional object navigation, an agent must navigate from a random location to any instance of a specified goal-object category.At each step, it receives an egocentric RGB-D observation and sometimes pose readings, then selects an action including STOP.
- Zero-Shot Setting: Zero-shot object navigation evaluates new environments and goal objects without training on object-goal labels or any navigation data.The targeted setting excludes navigation experience entirely, making it distinct from supervised object navigation.
3. Our ESC Approach
ESC combines open-world scene grounding, semantic mapping, commonsense reasoning, and frontier exploration to navigate toward unseen goal objects without navigation training.
- Framework: ESC converts RGB observations and prompts into semantic scene information while maintaining a map of rooms, objects, and frontiers.The mapping process projects scene information into a navigation map used to identify explorable frontiers.
- Open-World Semantic Scene Understanding: GLIP provides prompt-based grounding that can detect objects and rooms beyond fixed detector classes.Object and room prompts support open-world semantic scene understanding and can be extended to new test data.
- Commonsense Reasoning: A language model assigns real-valued likelihood scores to goal-object relations and goal-room relations using the detected semantic context.These scores represent whether the goal is likely to be near an object or located in a room.
- Frontier-Based Exploration: ESC selects frontiers using distance together with nearby-object and room information rather than distance alone.The intended policy favors frontiers near objects or in rooms where the goal is likely to appear.
- Soft Commonsense Constraints: Probabilistic Soft Logic represents uncertain commonsense relations as weighted rules that guide frontier selection.The formulation includes positive and negative object and room reasoning, a distance constraint, and a sum constraint that makes frontiers compete.
- Soft Commonsense Constraints: The sum constraint limits the combined frontier-selection scores to one, preventing all frontier variables from simultaneously taking value one.This encourages competition among candidate frontiers.
4. Experimental Setup
The evaluation covers three object-navigation benchmarks with varied environments and object types, using SR and SPL under a common 500-step limit and comparing zero-shot and supervised baselines.
- Metrics: All benchmarks impose a maximum of 500 navigation steps and report Success Rate and Success Rate weighted by inverse path Length.SPL is the primary challenge metric, while SoftSPL is additionally reported for ablations.
- Evaluation Protocol: ESC and CoW are identified as the only compared methods with no navigation training experience.ProcTHOR also has a zero-shot version, while its main approach uses large-scale synthesized-environment training and dataset-specific fine-tuning.
- Baselines: The comparison includes zero-shot methods ZSON and CoW alongside supervised methods PONI and ProcTHOR.CoW uses CLIP-based GradCAM localization with frontier exploration, whereas ZSON projects object and image goals into a shared embedding space.
- Implementation Details: The ESC experiments fix distance thresholds at d_f = 1.6m, d_o = 1.6m, and d_r = 0.6m.PSL rules use weight 1.0 when one commonsense level is active, with the shortest-distance rule weighted 2.0 when both levels are used.
5. Results and Analysis
ESC consistently outperforms zero-shot baselines across object-navigation benchmarks, narrowing the gap with supervised methods. Ablations indicate that open-world grounding, commonsense reasoning, and commonsense-guided exploration contribute to its performance.
- Result Comparison with SOTA Methods: 284% SPL and 288% SR relative improvements on MP3D over CoW establish ESC’s strongest reported benchmark gains.CoW uses the same evaluation setting as ESC.
- Result Comparison with SOTA Methods: ESC performs consistently well across MP3D, HM3D, and RoboTHOR, while other zero-shot methods show inconsistent cross-dataset performance.The authors characterize this as evidence of stronger generalizability.
- Result Comparison with SOTA Methods: ESC narrows the zero-shot–supervised performance gap on HM3D and RoboTHOR and outperforms supervised THDA on MP3D.This comparison is reported for object-goal navigation tasks.
- Result Comparison with SOTA Methods: 288% SR improvement over CoW on MP3D reflects ESC’s advantage on goal objects associated with specific rooms or nearby objects.Category-wise comparisons report consistent gains over CoW, including difficult cases such as toilets and beds.
- Ablation Study: GLIP-based grounding improves over CoW, while ESC’s commonsense reasoning further outperforms heuristic frontier exploration across datasets and metrics.GoW isolates open-world grounding with vanilla frontier exploration, using the same navigation policy as ESC.
- Ablation Study: Room and object reasoning generally improve performance, with room reasoning usually contributing more; object reasoning slightly hurts on RoboTHOR because objects are placed more randomly.Using both reasoning levels performs better on MP3D and HM3D.
6. Related Work
Related work spans supervised object navigation, zero-shot approaches, commonsense reasoning, and large pre-trained models for embodied agents. ESC transfers pre-trained commonsense knowledge into zero-shot object navigation through first-order rules encoded with Probabilistic Soft Logic.
- Supervised object-navigation methods encode egocentric images or semantic maps and train navigation policies with imitation learning or reinforcement learning.
- Zero-shot approaches address generalization to new objects and environments, but some train image-goal agents in target environments or rely on exploration heuristics.
- Commonsense reasoning has been used in embodied tasks through in-domain semantic maps, external object relations, semantic priors, and procedural planning.
- ESC transfers commonsense knowledge from large language models into object navigation by expressing it as first-order rules in Probabilistic Soft Logic for exploration.
- Large pre-trained models have been applied to object navigation, task planning, and vision-and-language navigation after demonstrating strong vision-language and reasoning abilities.
7. Conclusion and Future Work
ESC combines pre-trained language-image knowledge with soft commonsense constraints for zero-shot object navigation. The reported experiments support its efficacy and generalizability, while future work targets richer knowledge and a more flexible frontier-selection strategy.
- ESC leverages pre-trained knowledge from a language-image grounding model and a large language model for zero-shot object navigation.
- The framework introduces commonsense into frontier-based exploration as a soft constraint via Probabilistic Soft Logic.
- Experiments illustrate the efficacy and generalizability of ESC from different perspectives.
- Future work may acquire additional commonsense, including spatial relations between rooms, and other knowledge for embodied AI tasks.
- ESC uses a fixed strategy to combine commonsense knowledge, motivating improved frontier selection or limited fine-tuning.
A.1. GLIP Implementation Details
The implementation uses GLIP for semantic detection, language models for object and room reasoning, and PSL-based frontier selection within a mapped navigation loop. Dataset-specific policies address sensing and localization constraints.
- GLIP Implementation Details: GLIP-L is used for object and room detection with a 0.61 threshold and dataset-specific prompt vocabularies.MP3D and HM3D use a common indoor-object set based on MP3D goals, while RoboTHOR uses furniture and object categories plus its goal objects.
- PSL detailed explanation: PSL inference creates hinge-loss potentials over grounded logical rules using Łukasiewicz continuous-valued semantics.The potentials use PSL’s linear penalty function to help determine unobserved variables.
- Commonsense Reasoning: DeBERTa and ChatGPT provide commonsense scores for object-near and room-level reasoning questions.DeBERTa scores candidate rooms and objects, while ChatGPT generates scores usable as IsCooccur values.
- Navigation Policy: The navigation policy reasons about the goal at object and room levels before initializing semantic and navigation maps.GLIP updates semantic information during navigation, and PSL selects frontiers from commonsense and map information.
- Dataset-specific Policies: MP3D and HM3D use a long-distance goal policy because depth is limited to 5 meters.Detected goal objects remain targets until they are within the depth range; RoboTHOR instead estimates movement from successive depth inputs because GPS is unavailable.
B. Dataset Details
The evaluation covers three embodied-navigation benchmarks with different target-object inventories: MP3D, HM3D, and RoboTHOR.
- Dataset Details: MP3D contains 21 target objects, HM3D contains 6, and RoboTHOR contains 12 target objects.
C.1. Results per category
ESC performs well across HM3D goal categories and outperforms CoW on most RoboTHOR categories. An example illustrates how commonsense reasoning guides frontier selection toward a toilet.
- ESC performs well across all reported goal categories on HM3D.
- ESC outperforms CoW on most RoboTHOR object goals, except houseplant, basketball, and spray bottle.The authors attribute these exceptions partly to CLIP’s object localization ability and RoboTHOR’s smaller exploration space.
- In the illustrated navigation example, commonsense reasoning helps ESC choose frontiers that lead toward the toilet.The agent rejects kitchen, table, and chair contexts as unlikely toilet locations before selecting alternative frontiers.