Source-linked AI summary
Object Goal Navigation using Goal-Oriented Semantic Exploration
Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, Ruslan Salakhutdinov
TL;DR
Object goal navigation in unseen environments challenges end-to-end methods because exploration and long-term planning remain difficult. SemExp builds an episodic semantic map and uses goal-oriented semantic exploration, achieving state-of-the-art navigation, winning the CVPR 2020 Habitat ObjectNav Challenge, and transferring to a mobile robot.
Problem
Object goal navigation requires finding an instance of a specified object category in unseen environments, while end-to-end methods struggle with exploration and long-term planning.
Method
SemExp combines explicit episodic semantic mapping with a goal-oriented semantic policy that uses the target category to select exploration goals.
Results
SemExp achieves state-of-the-art object goal navigation, wins the CVPR 2020 Habitat ObjectNav Challenge, and transfers with similar performance to a mobile robot.
Takeaways & Limitations
Ablations indicate that semantic priors about relative object arrangements support more efficient goal-driven navigation.
Takeaways & Limitations
With ground-truth semantic segmentation, success rises from 54.4% to 73.1%, indicating substantial errors from segmentation inaccuracies; remaining failures mostly involve not finding the goal object.
Abstract
from arXiv · showhide
This work studies the problem of object goal navigation which involves navigating to an instance of the given object category in unseen environments. End-to-end learning-based navigation methods struggle at this task as they are ineffective at exploration and long-term planning. We propose a modular system called, `Goal-Oriented Semantic Exploration' which builds an episodic semantic map and uses it to explore the environment efficiently based on the goal object category. Empirical results in visually realistic simulation environments show that the proposed model outperforms a wide range of baselines including end-to-end learning-based methods as well as modular map-based methods and led to the winning entry of the CVPR-2020 Habitat ObjectNav Challenge. Ablation analysis indicates that the proposed model learns semantic priors of the relative arrangement of objects in a scene, and uses them to explore efficiently. Domain-agnostic module design allow us to transfer our model to a mobile robot platform and achieve similar performance for object goal navigation in the real-world.
1 Introduction
Object goal navigation requires more than recognizing target objects: agents must remember explored space and use scene-level semantic priors to choose where to explore. SemExp addresses these needs with explicit semantic maps and a goal-oriented exploration policy.
- Motivation: Object goal navigation requires detecting target objects and understanding where they are likely to occur in unseen scenes.The task therefore combines passive visual recognition with active scene understanding.
- Motivation: Long-term episodic memory tracks explored and unexplored areas, while semantic priors guide which regions to explore next.Together, these capabilities support efficient search for the target object.
- Limitations of Prior Approaches: End-to-end recurrent learning methods suffer from large sample complexity and poor generalization because they memorize training-environment object locations and appearances.
- Limitations of Prior Approaches: Active Neural SLAM improves exploration and sample efficiency with explicit obstacle maps but does not encode semantics or learn semantic priors.
- SemExp: SemExp extends explicit mapping with semantic categories and a semantically aware long-term policy for goal-oriented exploration.
- SemExp: SemExp outperforms prior methods in visually realistic simulation, won the CVPR 2020 Habitat ObjectNav Challenge, and transfers to a mobile robot with similar real-world performance.
2 Related Work
Prior navigation research includes geometric and semantic mapping, analytical planning, and end-to-end learning, but explicit maps have generally learned semantics only implicitly. This work combines structured semantic maps with a policy that selects exploration goals from the object category.
- Semantic Mapping: Prior mapping work builds geometric maps through structure from motion or SLAM, with some approaches incorporating semantics using graphical models or learned vision models.
- Semantic Mapping: The proposed semantic mapping instead uses differentiable projection operations with supervision in map space.
- Navigation: Classical navigation computes paths with explicit geometric maps and selects goals using heuristics such as Frontier-based Exploration.
- Learning-based Navigation: End-to-end navigation methods use recurrent, structured spatial, or topological representations, including recent approaches for object goal navigation.
- Positioning SemExp: Explicit maps improve performance and sample efficiency over end-to-end methods, but prior modular systems learn semantics implicitly.
- Positioning SemExp: SemExp uses an explicit structured semantic map to support semantically aware exploration policies for object-goal navigation.
3 Method
SemExp maintains an episodic semantic metric map, predicts semantic structure from RGB-D observations, and uses a goal-oriented policy to select long-term exploration targets. A deterministic planner then executes low-level actions toward those targets.
- Task Definition: The Object Goal task asks an agent to navigate to an instance of a specified category from a random starting location in an environment.
- Overview: SemExp contains learnable Semantic Mapping and Goal-Oriented Semantic Policy modules, followed by a deterministic local policy based on analytical planning.
- Semantic Map Representation: Its K × M × M metric map stores obstacle, explored-area, and object-category channels, with 5cm × 5cm physical cells.
- Semantic Mapping: The mapping pipeline combines RGB-based semantic predictions with depth-derived point clouds, differentiable 3D projection, and conversion to a top-down semantic map.
- Semantic Mapping: Map-space loss supplements first-person segmentation loss because small image-segmentation errors can become large projection errors.
- Goal-Oriented Semantic Policy: When the goal category is observed, the policy selects its mapped locations; otherwise, it selects regions where the category is likely to occur.
- Goal-Oriented Semantic Policy: The policy samples long-term goals every u = 25 steps, reducing the exploration horizon and reinforcement-learning sample complexity.
- Local Policy: The deterministic local policy uses the Fast Marching Method on the obstacle channel to follow paths toward selected long-term goals.
4 Experimental Setup
Experiments evaluate SemExp and baselines in Gibson and MP3D using RGB-D observations, pose, object goals, semantic mapping, and navigation metrics.
- Datasets: Experiments use Gibson and Matterport3D scenes in the Habitat simulator, with separate training and test splits.The training set contains 86 scenes and the test set contains 16 scenes.
- Task and Environment: The observation space includes 640 × 480 RGB-D images, base odometry, and an integer goal category, with four available actions.The action space is move_forward, turn_left, turn_right, and stop.
- Task and Environment: The six object-goal categories are chair, couch, potted plant, bed, toilet, and tv, while the semantic map encodes 15 categories.Mask-RCNN with a Feature Pyramid Network and ResNet50 backbone pretrained on MS-COCO provides object detection and instance segmentation.
- Implementation: SemExp uses a five-layer semantic-mapping denoising network and a convolutional Goal-driven Semantic Policy that receives the semantic map, agent orientation, and goal category.The policy is trained with PPO, while semantic mapping uses binary cross-entropy map prediction loss.
- Evaluation: Evaluation compares Success, SPL, and DTS across SemExp and recurrent RGB-D and semantic-RGB-D reinforcement-learning baselines.Success measures successful episodes, SPL measures success weighted by path length, and DTS measures distance from the success-threshold boundary at episode termination.
5 Results
SemExp outperforms baselines across simulated environments, and ablations attribute this advantage to goal-oriented semantic exploration. The model also wins the Habitat challenge and transfers successfully to real-world navigation, while semantic segmentation remains a major error source.
- Simulation Results: 54.4%/36.0% success on Gibson/MP3D versus 44.6%/32.1% for Active Neural SLAM, with SemExp outperforming all baselines.The evaluation used 10 million training frames and 1000 Gibson plus 2000 MP3D episodes.
- Ablations and Error Analysis: Removing the Goal-oriented Policy produces performance comparable to Active Neural SLAM, indicating that it learns semantic priors for more efficient exploration.Removing the Semantic Map also reduces performance, but less than removing the Goal-oriented Policy.
- Ablations and Error Analysis: 73.1% versus 54.4% success with ground-truth semantic segmentation indicates around 19% potential improvement from better semantic segmentation.Most remaining failures involve not finding the goal object and could improve with better semantic exploration.
- Habitat Challenge: 25.3% success versus 18.8% for the second-place entry made the SemExp submission the winner of the CVPR 2020 Habitat ObjectNav Challenge.The challenge used 21 goal-object categories, requiring segmentation over 40 MP3D categories.
- Real-World Transfer: 65% success in 20 real-world trials demonstrates successful transfer of SemExp to a Locobot navigating toward potted plants.The learned mapping and goal-policy modules operate on domain-agnostic top-down maps, while Mask RCNN uses real-world training data.
6 Conclusion
The paper concludes that SemExp combines explicit semantic episodic memory with goal-oriented exploration to achieve strong object-goal navigation performance. Its semantic priors support efficient navigation, challenge-winning results, and real-world transfer, while future improvements target semantic mapping and exploration.
- 6 Conclusion: SemExp incorporates semantics into explicit episodic memory and learns goal-oriented semantic exploration policies for object-goal navigation.The model achieves state-of-the-art performance and won the CVPR2020 Habitat ObjectNav challenge.
- 6 Conclusion: Ablations show that SemExp learns semantic priors that lead to more efficient goal-driven navigation.The episodic map can also support more efficient navigation for sequences of object goals.
- 6 Conclusion: Domain-agnostic module design enabled successful transfer of the model to the real world.The paper identifies semantic mapping and goal-oriented exploration as future improvement dimensions.