Source-linked AI summary
InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment
Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, Hao Dong
TL;DR
Robots need to follow diverse language instructions in unexplored environments, but prior systems are constrained by task-specific strategies and scarce varied navigation data. InstructNav unifies planning with Dynamic Chain-of-Navigation and converts it into robot trajectories through Multi-sourced Value Maps, achieving zero-shot performance across tasks and robustness in varied real scenes. The paper also identifies reliance on closed-source large models and occlusion-sensitive semantic maps as limitations.
Problem
Different navigation tasks require different strategies, while scarce instruction-navigation data leaves previous methods constrained to one instruction type.
Method
InstructNav uses Dynamic Chain-of-Navigation to unify instruction planning and Multi-sourced Value Maps to convert linguistic plans into actionable trajectories.
Results
InstructNav handles diverse instruction types without navigation training or pre-built maps, completing R2R-CE zero-shot and improving zero-shot Habitat ObjNav by 10.48% and DDN by 86.34%.
Takeaways & Limitations
Experiments in simulators and diverse real indoor scenes demonstrate the generalization and effectiveness of the training-free generic navigation system.
Takeaways & Limitations
The best-performing system relies on closed-source large models, and semantic value-map quality is affected by occlusion.
Abstract
from arXiv · showhide
Enabling robots to navigate following diverse language instructions in unexplored environments is an attractive goal for human-robot interaction. However, this goal is challenging because different navigation tasks require different strategies. The scarcity of instruction navigation data hinders training an instruction navigation model with varied strategies. Therefore, previous methods are all constrained to one specific type of navigation instruction. In this work, we propose InstructNav, a generic instruction navigation system. InstructNav makes the first endeavor to handle various instruction navigation tasks without any navigation training or pre-built maps. To reach this goal, we introduce Dynamic Chain-of-Navigation (DCoN) to unify the planning process for different types of navigation instructions. Furthermore, we propose Multi-sourced Value Maps to model key elements in instruction navigation so that linguistic DCoN planning can be converted into robot actionable trajectories. With InstructNav, we complete the R2R-CE task in a zero-shot way for the first time and outperform many task-training methods. Besides, InstructNav also surpasses the previous SOTA method by 10.48% on the zero-shot Habitat ObjNav and by 86.34% on demand-driven navigation DDN. Real robot experiments on diverse indoor scenes further demonstrate our method's robustness in coping with the environment and instruction variations.
1 Introduction
InstructNav targets generic instruction navigation by unifying diverse navigation strategies with Dynamic Chain-of-Navigation and Multi-sourced Value Maps. It achieves zero-shot results across simulated tasks and demonstrates robustness in varied real indoor scenes.
- Different navigation instruction types emphasize different strategies, limiting previous systems to specific instruction categories.Object goal navigation explores for objects, visual language navigation follows step-by-step directions, and demand-driven navigation searches for related objects.
- Dynamic Chain-of-Navigation unifies instruction planning around actions and landmarks, updating both from observed scene objects at each decision step.The dynamic process also supports semantic label alignment, commonsense reasoning, and exploration of unseen environments.
- Multi-sourced Value Maps represent actions, landmarks, history trajectories, and multimodal predictions to convert linguistic planning into actionable robot trajectories.The maps guide waypoint selection, landmark following, avoidance of repetitive movement, and multimodal reasoning about ambiguous instructions.
- InstructNav is presented as the first generic system to execute diverse instructions continuously without navigation training or pre-built maps.Its planning and value-map components connect language-level DCoN decisions to robot control.
- 10.48% improvement on zero-shot Habitat ObjNav and 86.34% improvement on demand-driven navigation DDN are reported against previous state-of-the-art methods.The system also completes R2R-CE zero-shot and outperforms many task-training methods, while real-world tests cover diverse indoor scenes and instructions.
2 Related Work
Instruction-guided navigation research has largely developed separate methods for object goals, step-by-step routes, and demand-driven searches. Large language and multimodal models provide capabilities relevant to instruction following, planning, and visual perception.
- Instruction-guided Navigation: Object goal navigation finds a specific object, visual language navigation follows step-by-step instructions to destinations, and demand-driven navigation searches for objects related to human needs.
- Instruction-guided Navigation: Existing methods generally execute one predefined instruction type and therefore have limited application scenarios across diverse navigation tasks.InstructNav is positioned as a system that follows multiple instruction types.
- Large Models for Navigation: Large language and multimodal models offer instruction following, task planning, and visual perception capabilities relevant to robotic navigation.Their development has motivated applications of large models in robotic navigation.
3.1 Problem Formulation and Method Overview
The generic task asks a robot to follow natural-language instructions to a target in an unexplored continuous environment without pre-built maps. InstructNav combines DCoN planning with Multi-sourced Value Maps to produce low-level movement trajectories.
- Problem Formulation: The robot observes egocentric RGB, depth, and camera-pose data at each step, then executes low-level actions toward the target without pre-built maps.The task operates in an unexplored continuous environment and permits no pre-built maps.
- Method Overview: Dynamic Chain-of-Navigation provides the unified planning paradigm for diverse instructions, while Multi-sourced Value Maps model key navigation elements.The value maps support planning actionable trajectories for low-level movement.
3.2 Dynamic Chain-of-Navigation Planning
Chain-of-Navigation expresses instructions as linked action–landmark steps, but generic planning must also resolve semantic mismatches, unseen landmarks, and abstract human demands. DCoN addresses these issues through stepwise re-inference from observed objects.
- Chain-of-Navigation: Chain-of-Navigation transforms instructions into an “Action 1 - Landmark 1 →Action 2 - Landmark 2 →...” sequence resembling language-model chain-of-thought reasoning.
- Challenges: Directly extracting actions and landmarks fails when instruction landmarks do not match semantic-segmentation labels.Such mismatches hinder retrieval of the intended landmarks on the map.
- Challenges: Generic planning must also handle landmarks absent from explored areas and abstract demands that cannot be decomposed into concrete actions and landmarks.
- Dynamic Chain-of-Navigation: DCoN re-infers the next action and landmarks at every decision step using observed scene objects rather than producing one fixed plan.Its prompt includes robot definition, navigation strategy, prediction format, and episode information.
3.3 Multi-sourced Value Maps
Multi-sourced Value Maps translate DCoN’s linguistic actions and landmarks into waypoint selection by combining action, semantic, trajectory, and intuition signals. These maps use scene observations and navigation history to guide actionable movement in unexplored environments.
- Semantic Value Map: RGB-D observations and 2D semantic segmentation produce a scene semantic point cloud for constructing the value maps.
- Semantic Value Map: The Semantic Value Map assigns higher values to navigable areas near DCoN-specified landmarks.It derives landmark locations by lifting 2D semantic masks into 3D using depth and camera pose.
- Action Value Map: The Action Value Map assigns higher values to regions compatible with DCoN actions, including directional sectors, exploration frontiers, and doorway approaches.Enter and Exit are represented through an Approach action with a Doorway landmark.
- Trajectory Value Map: The Trajectory Value Map favors navigable areas farther from the robot’s recorded history trajectory, encouraging diverse paths.
- Intuition Value Map: The Intuition Value Map projects the multimodal model’s predicted navigation area, assigning value to navigable positions and requesting re-prediction when the area is unusable.The model analyzes an equal-interval panorama together with the instruction, next action, and landmarks before selecting a movement direction.
3.4 Navigation Process with Multi-sourced Value Maps
InstructNav combines the four value maps into a decision-making map, avoids obstacles, and selects a highest-value navigation goal for trajectory planning. Its evaluation covers object goal navigation, visual language navigation, and demand-driven navigation comparisons.
- Decision-Making Map: The decision-making value map m is obtained by summing the intuition, action, trajectory, and semantic value maps.Obstacle regions are set to zero before selecting the navigation goal and planning a trajectory with A*.
- Trajectory Planning: The navigation goal is selected as the highest-value point on m, after which A* plans the robot trajectory.Simulation uses rotate-then-forward tracking, while real-world experiments directly control robot speed.
- Evaluation: Table 1 compares InstructNav with state-of-the-art methods on HM3D object goal navigation.
- Evaluation: Table 2 compares methods on R2R-CE visual language navigation while excluding MP3D-specific waypoint predictors for fairness.
- Evaluation: Table 3 compares state-of-the-art methods on DDN demand-driven navigation.
4 EXPERIMENTS
Experiments evaluate InstructNav across simulated navigation tasks, ablations, open-source model substitutions, and real-robot indoor scenes. Results support zero-shot performance across task types while showing the effects of visual-observation count, DCoN, and value-map components.
- Simulation Experiments: InstructNav outperforms all zero-shot HM3D object-goal navigation methods on success rate and is comparable to the best trained model, OVRL.
- Simulation Experiments: InstructNav is the first model to complete R2R-CE visual language navigation zero-shot and outperforms many task-trained models.
- Simulation Experiments: InstructNav outperforms all DDN baselines by a large margin on unseen scenes and unseen instructions.
- Ablation Study: N = 6 RGB observations consistently performs best across all three navigation tasks; smaller N can miss visual information, whereas larger N can increase misunderstanding.
- Ablation Study: Ablating DCoN significantly decreases success rate on all three tasks, while removing any one of the four value maps weakens performance across tasks.
- Open-source Large Models: Open-source Llama3 70B and LLaVA1.6 34B achieve performance comparable to GPT models on a portion of tasks, including ObjectNav.
- Real-robot Experiments: Real-robot experiments independently execute diverse instruction types without pre-built maps in offices, apartments, libraries, galleries, and teaching buildings.
5 Conclusion
The paper concludes that InstructNav unifies diverse navigation instructions through DCoN and converts linguistic plans into actionable trajectories with Multi-sourced Value Maps. Experiments in simulators and on a real robot demonstrate the training-free method’s generalization and effectiveness, while its best performance still depends on closed-source large models.
- InstructNav targets generic instruction navigation in continuous environments without navigation training or pre-built maps.
- DCoN unifies different navigation instructions, while Multi-sourced Value Maps convert linguistic planning into robot-actionable trajectories.
- Extensive simulator and real-robot experiments demonstrate the generalization and effectiveness of the training-free method.
- The current system relies on closed-source large models for its best performance, and semantic-map quality is influenced by occlusion.