Source-linked AI summary
OmniNav: A Unified Framework for Prospective Exploration and Visual-Language Navigation
Xinda Xue, Junjun Hu, Minghua Luo, Shichao Xie, Jintao Chen, Zixun Xie, Kuichen Quan, Wei Guo, Mu Xu, Zedong Chu
TL;DR
Embodied navigation must combine visual and language understanding with exploration, but existing approaches often lack a unified solution across navigation paradigms. OmniNav combines waypoint prediction, fast–slow planning, and general-purpose training in one framework; it reports improved benchmark performance and real-robot deployment, while identifying full slow-system deployment as future work.
Problem
Embodied navigation requires robots to understand visual environments and language while exploring, but existing methods are often task-specific and can struggle with unseen instructions, environments, or object recognition.
Source: §1 · pp. 1–2Method
OmniNav unifies four navigation modes with a fast waypoint policy and a slower, memory-supported planner, and trains with general-purpose vision-language data alongside navigation tasks.
Source: §1 · p. 2Results
OmniNav improves success rates over the current best on R2R-CE and RxR-CE, achieves the best performance on OVON, and demonstrates quadruped deployment with up to 5 Hz cloud inference.
Source: §6 · p. 10Takeaways & Limitations
The authors present OmniNav’s unified multimodal interface and joint training approach as a broadly applicable path toward generalization and robustness in embodied navigation.
Source: §6 · pp. 9–10The paper does not fully deploy the slow system on a physical robot or systematically optimize its performance under real-world constraints.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Embodied navigation presents a core challenge for intelligent robots, requiring the comprehension of visual environments, natural language instructions, and autonomous exploration. Existing models often fall short in offering a unified solution across diverse navigation paradigms, resulting in low success rates and limited generalization. We introduce OmniNav, a unified framework addressing instruct-goal, object-goal, point-goal navigation, and frontier-based exploration within a single architecture. Our approach features a lightweight, low-latency policy that accurately predicts continuous-space waypoints (coordinates and orientations). This policy surpasses action-chunk methods in precision and supports real-world deployment at control frequencies up to 5 Hz. Architecturally, OmniNav employs a fast-slow system design: a fast module generates waypoints using short-horizon visual context and subtasks, while a slow module performs deliberative planning with long-horizon observations and candidate frontiers to select subsequent subgoals and subtasks. This collaboration enhances path efficiency and maintains trajectory coherence, particularly in exploration and memory-intensive scenarios. Crucially, we identify that the primary bottleneck isn't merely navigation policy learning, but a robust understanding of general instructions and objects. To boost generalization, OmniNav integrates large-scale, general-purpose training datasets, including those for image captioning and visual recognition, into a joint multi-task regimen. This significantly improves success rates and robustness. Extensive experiments confirm OmniNav's state-of-the-art performance across various navigation benchmarks, with real-world deployment further validating its efficacy. OmniNav provides practical insights for embodied navigation, charting a scalable path towards versatile, highly generalizable robotic intelligence.
1 INTRODUCTION
OmniNav unifies several embodied-navigation tasks through fast waypoint generation and slower, memory-supported planning. It combines this architecture with general-purpose vision-language training, and the authors report state-of-the-art benchmark results and real-robot deployment.
1 INTRODUCTION
- Existing methods are often task-specific, while point-goal, instruction-goal, and object-goal navigation each present distinct practical or generalization challenges.Point-goal tasks rely on explicit coordinates, instruction-goal methods can generalize poorly, and object-goal navigation requires robust recognition and efficient planning.Source: §1 · p. 1
- OmniNav supports instruct-goal, object-goal, point-goal, and frontier-based exploration in one architecture.A fast module generates high-precision coordinate-and-orientation waypoints for low-latency control; a slow module uses long-horizon observations and frontier cues to select subgoals and subtasks, with central memory providing spatiotemporal context.Source: §1 · p. 2
- Joint training with large-scale generic vision-language data and navigation tasks is presented as a way to improve instruction following, open-vocabulary object perception, generalization, and robustness.The paper highlights data such as captioning and referring or grounding examples.Source: §1 · p. 2
- OmniNav reports state-of-the-art results across multiple navigation benchmarks, with real-robot deployments validating practicality.The introduction does not provide benchmark-specific values.Source: §1 · p. 2
2 RELATED WORKS
Related work spans VLM-based action prediction, dual-system control and planning, and frontier-based exploration. OmniNav combines a fast local policy with slower global planning and selects frontiers using semantic reasoning over associated images.
2 RELATED WORKS
- Prior VLM navigation methods decode low-level actions autoregressively, a paradigm described as prone to compounding errors and slow inference; OmniNav instead appends a flow-matching policy to a VLM backbone for non-autoregressive trajectory generation.The cited discussion associates this design with improved prediction accuracy and computational efficiency, especially in unseen environments.Source: §2 · p. 2
- OmniNav assigns local execution to a fast visual policy and global planning to a slow system using frontier exploration and images as long-term memory.The paper presents this as a more concise implementation than scene graphs or complex semantic maps, while noting those structures could also implement the slow system.Source: §2 · p. 3
- Unlike methods that typically select the closest frontier with simple heuristics, OmniNav links frontiers to egocentric images and uses chain-of-thought reasoning to select promising targets.The selection is conditioned on the current task.Source: §2 · p. 3
3 Approach
OmniNav pairs a fast visual waypoint policy with slow planning that uses memory and frontiers to choose subgoals, supporting both multi-task navigation and long-horizon exploration.
3 Approach
- Slow planning supplies high-level subgoals, while the fast system uses visual input to adapt its waypoint sequence and navigate around obstacles.It can also adjust its final pose based on the target's current appearance.Source: Figure 1 · p. 3, §3 · pp. 4–5
- The fast system predicts five continuous-space waypoints conditioned on visual, textual, and coordinate inputs, using a DiT policy with self- and cross-attention.Each waypoint includes 2D position, orientation, and an arrival flag.Source: §3 · p. 4
- At inference, waypoint sequences are refined from noise through five Euler integration steps, with the final denoised sequence serving as the predicted waypoints.The policy is trained to estimate the denoising residual.Source: §3 · p. 4
- The slow system localizes a target from current or historical views, or selects a semantically relevant exploration subgoal when the target is unseen.It draws on a memory bank of pose-stamped observations and evaluates frontiers using nearby historical images.Source: §3 · pp. 4–5
- The slow system reasons over frontier candidates using spatial and semantic context, and can return a target location already present in view or memory.For example, it prioritizes bathroom-related locations when searching for a toilet.Source: §3 · p. 5, Figure 2 · p. 5
4 DATA and Training
OmniNav trains on a mix of general web, simulation, and limited real-robot data, combining broad language-vision skills with navigation tasks and continuous waypoint prediction.
4 DATA and Training
- The training data follows a pyramid of general web, simulation, and a very small amount of real-robot data.The overview groups training examples into navigation, embodied QA, general MLLM, and grounding/referring data.Source: §4 · p. 5, Figure 3 · p. 6
4.1 General Dataset
- General-purpose QA, captioning, OCR, chart understanding, coding, and math data complement navigation training with language, visual, reasoning, and planning skills.Grounding and referring data targets precise language-to-instance mapping and spatial or relational cues.Source: §4.1 · p. 6
4.2 Multi-Task Navigation Dataset
- Navigation training draws on Citywalker point-goal videos, R2R and RxR instruction-path data, and OVON object-goal trajectories in Habitat scenes.The data also includes frontier-based exploration episodes and embodied QA from ScanQA and R2R-EnvDrop.Source: §4.2 · pp. 6–7
- Navigation trajectories are represented as local planar position, heading, and arrival targets, while simulator handling of vertical motion supports multi-floor paths.Agents still receive 3D visual observations that reflect stairs and floor changes.Source: §4.2 · p. 7
4.3 Discrete and Continuous Joint Training
- Training first uses an autoregressive objective for discrete action and semantic data, then adds flow matching for continuous waypoint prediction while retaining 20% of Stage 1 data.Waypoint coordinates are min-max normalized during Stage 2.Source: §4.3 · p. 7
- Stage 1 training uses 96 NVIDIA H20 GPUs for 120 hours, and Stage 2 uses 64 GPUs for 48 hours at a lower learning rate.The authors state that joint training substantially improves success rates in open environments.Source: §4.3 · p. 7
5 Experiments
OmniNav is evaluated across instruction-, object-, and point-goal navigation, with ablations and real-robot deployment examining its components and practicality.
Instruct goal
- 4.4% on R2R-CE and 4.3% on RxR-CE are OmniNav’s success-rate improvements over the previous leading model.The comparisons use only the fast system with pure RGB inputs.Source: §5 · p. 7, Table 1 · p. 8
Object goal
- On HM3D-OVON, OmniNav exceeds the strongest prior method by 2.7% with visual inputs and by 18.4% with slow-system frontier reasoning.The slow system uses depth and odometry to build and maintain an occupancy map.Source: §5 · p. 8, Table 2 · p. 9
Point goal
- On CityWalker’s open-set MAOE metric, OmniNav scores 11.53% versus CityWalker’s 15.23%.CityWalker supports outdoor point-goal navigation.Source: §5 · p. 8
Ablation Study
- On HM3D-OVON, all four components together perform best; the slow system, precise policy, general data, and CoT contribute complementary capabilities.The slow system supports long-horizon subgoals, while CoT enables auditable reasoning and self-correction.Source: §5 · pp. 8–9, Table 3 · p. 9
Real-world deployment
- The deployed fast system runs at over 5 Hz and sends waypoints to an onboard speed controller.Deploying the complete slow system still requires engineering for real-time LiDAR/depth integration and evaluation under real-world constraints.Source: §5 · p. 9
6 CONCLUSION & FUTURE WORKS
OmniNav combines continuous-waypoint control with long-horizon planning, memory, and unified multimodal task handling. Its benchmark results and quadruped deployment support its practical potential, while semantic subgoal selection and lifelong memory remain future directions.
Conclusion
- OmniNav’s fast system generates continuous waypoints for low-latency, high-precision control, while its slow system plans subgoals using long-horizon visual memory and frontiers.Explicit CoT supports interpretability and self-correction.Source: §6 · p. 9
- Unified multimodal tokenization handles instruction-, object-, and point-goal tasks within one model, and joint discrete-continuous training helps preserve base VLM capabilities.The training scheme also incorporates general-purpose and referring/grounding data to improve navigation generalization and robustness.Source: §6 · pp. 9–10
- OmniNav improves success rates over the current best on R2R-CE and RxR-CE, achieves the best OVON performance, and demonstrates quadruped deployment at up to 5 Hz.The deployment uses tri-view inputs and a 20-frame history buffer.Source: §6 · p. 10, Figure 4 · p. 10
Future Works
- Future work targets semantics-driven subgoal selection, purely visual memory, and retrievable lifelong spatiotemporal memory.Figure 4 depicts three navigation tasks in a zero-shot real-robot setting, with blue arrows tracing trajectories and yellow boxes marking targets.Source: §6 · p. 10, Figure 4 · p. 10
A.1 Analysis of Training Data and Model scale
Data ablations suggest that general vision-language training helps recognition of varied objects, while model scale matters less when additional data are included.
Analysis of Training Data and Model scale
- Removing Embodied Q&A or Grounding/referring degrades performance on small background objects, while removing General MLLM data leads to failures on irregular objects.The findings suggest these data components help recognition of small objects and that general vision-language data helps with irregular targets.Source: §A.1 · p. 15, Table 4 · p. 15
- The model still struggles with carpet and clothes, whose complex folds and textures motivate dedicated VQA-style training data.The proposed data would emphasize texture-heavy patterns to strengthen fine-grained visual reasoning.Source: §A.1 · p. 15
- With additional data, 3B and 7B models show nearly identical navigation performance; without it, the 7B model outperforms the 3B model.The authors attribute the data-rich result to data sufficiency rather than model capacity, while noting that complex-object recognition remains unstable across model sizes.Source: §A.1 · p. 15, Table 5 · p. 15
- A systematic study of scaling laws and optimal training configurations remains future work because of computational limits.The proposed study would examine how data quality, data composition, and model size jointly affect performance.Source: §A.1 · p. 15