Source-linked AI summary
Embodied Navigation Foundation Model
Jiazhao Zhang, Anqi Li, Yunpeng Qi, Minghan Li, Jiahang Liu, Shaoan Wang, Haoran Liu, Gengze Zhou, Yuze Wu, Xingxing Li, Yuxin Fan, Wenjun Li, Zhibo Chen, Fei Gao, Qi Wu, Zhizheng Zhang, He Wang
TL;DR
Embodied navigation still generalizes less broadly than general vision-language tasks across task settings, embodiments, camera configurations, and horizons. NavFoM addresses this gap with a unified cross-task, cross-embodiment model trained on eight million samples, using identifier tokens and budget-aware temporal sampling. It achieves state-of-the-art or competitive benchmark performance without task-specific fine-tuning and shows strong generalization in real-world experiments.
Problem
Embodied-navigation methods have limited generalization across tasks and embodiments compared with vision-language models’ zero-shot performance on general vision-language tasks.
Method
NavFoM unifies multimodal navigation across embodiments, tasks, camera configurations, and horizons using temporal-viewpoint indicator tokens and budget-aware temporal sampling.
Results
NavFoM achieves state-of-the-art or competitive performance across diverse benchmarks and embodiments without task-specific fine-tuning.
Takeaways & Limitations
The results support NavFoM as a generalist navigation model with cross-task and cross-embodiment generalization in public benchmarks and real-world robotic experiments.
Takeaways & Limitations
All methods perform poorly on the Unseen-Map split, which requires long traversals through complex neighborhoods and higher-quality UAV data.
Abstract
from arXiv · showhide
Navigation is a fundamental capability in embodied AI, representing the intelligence required to perceive and interact within physical environments following language instructions. Despite significant progress in large Vision-Language Models (VLMs), which exhibit remarkable zero-shot performance on general vision-language tasks, their generalization ability in embodied navigation remains largely confined to narrow task settings and embodiment-specific architectures. In this work, we introduce a cross-embodiment and cross-task Navigation Foundation Model (NavFoM), trained on eight million navigation samples that encompass quadrupeds, drones, wheeled robots, and vehicles, and spanning diverse tasks such as vision-and-language navigation, object searching, target tracking, and autonomous driving. NavFoM employs a unified architecture that processes multimodal navigation inputs from varying camera configurations and navigation horizons. To accommodate diverse camera setups and temporal horizons, NavFoM incorporates identifier tokens that embed camera view information of embodiments and the temporal context of tasks. Furthermore, to meet the demands of real-world deployment, NavFoM controls all observation tokens using a dynamically adjusted sampling strategy under a limited token length budget. Extensive evaluations on public benchmarks demonstrate that our model achieves state-of-the-art or highly competitive performance across multiple navigation tasks and embodiments without requiring task-specific fine-tuning. Additional real-world experiments further confirm the strong generalization capability and practical applicability of our approach.
1 INTRODUCTION
NavFoM addresses limited generalization in embodied navigation by unifying tasks and embodiments within one foundation model. It uses diverse navigation data and mechanisms for varying camera configurations, temporal horizons, and deployment budgets, achieving strong benchmark performance without task-specific fine-tuning.
- Motivation: Prior embodied-navigation methods remain constrained by narrow task settings, fixed camera configurations, or embodiment-specific designs.VLMs generalize strongly on several general vision-language tasks, but embodied-navigation generalization remains comparatively limited.
- Approach: NavFoM is trained on eight million navigation samples spanning quadrupeds, drones, wheeled robots, cars, and tasks including navigation, searching, tracking, and autonomous driving.The dataset combines public navigation datasets with pseudo web-video navigation data.
- Approach: Temporal-viewpoint indicator tokens encode camera viewpoints and navigation-horizon timing, enabling co-tuning across camera setups and with image- and video-QA data.The indicators organize multimodal inputs from different camera configurations and temporal contexts.
- Approach: A token budget-aware temporal sampling strategy dynamically controls observation tokens to address deployment constraints including hardware memory cost and inference speed.The strategy supports a unified approach to token sampling across diverse camera setups and task horizons.
- Results: 64.4% SR in multi-camera VLN-CE RxR improves on 56.3% SR, while zero-shot HM3D-OVON reaches 45.2% SR versus 43.6% for the prior fine-tuned SOTA.NavFoM also reports competitive results across object searching, tracking, autonomous driving, and real-world robotic platforms without task-specific fine-tuning.
2 RELATED WORKS
Related work develops large-model, cross-embodiment, and cross-task approaches for navigation, but existing methods address different parts of generalization. These studies motivate NavFoM’s unified treatment of tasks, embodiments, and multimodal inputs.
- Large models for Navigation: Large language and vision-language models have been integrated into navigation to provide open-world understanding and zero-shot reasoning capabilities.Prior approaches include off-the-shelf zero-shot use, chain-of-thought mechanisms, and structured reasoning frameworks.
- Cross-embodiments Navigation: Cross-embodiment navigation studies target generalization across differences in embodiment shape, size, and sensor configuration.Recent work uses transformer-based policies trained on large-scale cross-embodiment data, but the passage does not establish a single unified solution.
- NavFoM pipeline: NavFoM’s pipeline unifies Image QA, Video QA, and Navigation while using task-specific heads for language modeling or direct trajectory prediction.Temporal-viewpoint indicator tokens organize text and visual tokens across the unified pipeline.
- Cross-task Navigation: Cross-task navigation research shows that combining data from different navigation categories can improve performance across varied navigation scenarios.Examples include reinforcement-learning agents trained in simulators and generalist video-based vision-language models.
3 METHOD
NavFoM represents multimodal navigation observations with temporal-viewpoint-aware visual tokens and budget-aware sampling, allowing one architecture to process varied cameras, tasks, and temporal horizons. Its pipeline combines visual encoding, token organization, language conditioning, and waypoint prediction while preserving recent information under token constraints.
- General Navigation Setting: NavFoM maps multi-camera image sequences and language instructions to waypoint-based navigation trajectories.Visual observations are encoded, organized with navigation-specific tokens, combined with language tokens, and decoded into trajectories.
- Unified Architecture: NavFoM co-tunes navigation with image- and video-question-answering samples through a dual-branch vision-language architecture.Navigation predicts action tokens, whereas question answering predicts tokens autoregressively.
- Observation Encoding: Visual features combine DINOv2 and SigLIP representations, then use grid pooling to produce compact fine- and coarse-grained tokens.Fine-grained features represent the latest observation, while coarse-grained features represent navigation history and video data.
- Temporal-Viewpoint Indicator Tokens: TVI tokens encode camera viewpoint and temporal position, enabling flexible organization across arbitrary camera arrangements and navigation horizons.Navigation uses base, time, and angle embeddings; video QA uses base and time embeddings, while image QA uses the base embedding.
- Design Effectiveness: Additional indicator tokens show the most robust training performance and strong evaluation performance among the compared multi-view organization approaches.The authors attribute this advantage to preserving the off-the-shelf visual token space.
- Budget-Aware Temporal Sampling: BATS dynamically samples temporal observations under a token budget, retaining more recent information while adapting to different camera counts.The method is designed to constrain inference and memory costs while preserving historical context, and it maintains stable inference speed across navigation.
4 EXPERIMENTS
NavFoM is evaluated across navigation tasks, embodiments, camera configurations, and real-world settings using standard benchmark metrics and trajectory-based assessments. It achieves strong benchmark and real-world performance without task-specific camera fine-tuning, while performance remains weak on especially demanding unseen-map traversal.
- Experiment Setup: Evaluations cover vision-and-language navigation, object searching, tracking, and autonomous driving across egocentric, four-camera, six-camera, and eight-camera configurations.Standard metrics include SR, OS, SPL, nDTW, NE, TR, L2 distance, CR, and PDMS.
- Benchmark Results: NavFoM achieves state-of-the-art VLN-CE performance without camera-specific fine-tuning, reaching 64.4% SR in multi-camera RxR and 57.4% SR in single-camera RxR.The multi-camera result uses four RGB cameras and outperforms prior methods using RGB-D cameras and odometry.
- Benchmark Results: NavFoM achieves state-of-the-art performance on OpenUAV and performance comparable to SOTA on NAVSIM and nuScenes without explicitly modeling driving-specific scene information.OpenUAV trajectories average 200 meters, whereas all methods perform poorly on the Unseen-Map split requiring average 300-meter traversal through complex neighborhoods.
- Benchmark Results: 43.6% SR on the VAL UNSEEN object-navigation split exceeds the previous SOTA method's 40.8% SR, with consistent gains when moving from single-camera to four-camera evaluation.The model was trained only on single-camera search samples, so the camera-setting transfer is evaluated zero-shot.
- Benchmark Results: NavFoM reaches state-of-the-art tracking performance on EVT-Bench and improves modestly by 0.6% SR when evaluated zero-shot with four views instead of one.The smaller gain is attributed to the benchmark assumption that most targets appear in front of the robot.
- Real-World Results: Across 110 reproducible real-world test cases, NavFoM improves over Uni-NaVid and handles challenging long-horizon scenarios involving quadrupeds, humanoids, drones, and wheeled robots.The test set includes 50 VLN, 30 search, and 30 tracking samples in a recorded 5m × 5m environment.
5 DISSCUSION AND CONCLUSION
NavFoM is presented as a starting point for navigation foundation models, with experiments showing impressive performance and potential for further improvement.
- NavFoM explores intelligence learned from cross-embodiment and cross-task navigation data.
- Temporal-viewpoint indicator tokens address varying camera configurations and navigation horizons while enabling co-training with navigation and question-answering data.
- A token budget-aware temporal sampling strategy balances navigation performance and efficiency across camera setups and task horizons.
- Experiments on public benchmarks and real-world environments demonstrate impressive performance and potential for improvement with advanced techniques or higher-quality data.
- The authors characterize NavFoM as merely a starting point toward a navigation foundation model.
A.1 ACTION PLANNING MODEL
The action-planning procedure normalizes predicted trajectory scales across embodiments because different embodiments operate at different spatial scales.
- Different embodiments have distinct trajectory scales, with indoor robots moving meters and cars moving dozens of meters.
- Predicted trajectory dimensions are normalized to the range [-1,1] across embodiments by multiplying by a task-specific scaling factor α_task.
- The scaling factor uses the 99th percentile rather than each dimension’s absolute maximum value.
A.2 DETIALS OF USING BATS
BATS manages visual-token selection under a token budget, including a precomputed sampling procedure for efficient processing of long navigation histories.
- Table 7 reports scaling factors for different dimensions of predicted trajectories across embodiments.
- BATS samples visual tokens once their number exceeds the token budget B, while retaining all tokens when they remain within budget.
- The sampling process uses a forgetting curve and precomputes P(t, T) for a given budget to accelerate execution.
A.3 NAVIGATION DATA PREPARATION
The appendix presents trajectory visualization for VLN-CE RxR and describes how Habitat-based discrete actions are unified across navigation tasks.
- Figure 14 visualizes VLN-CE RxR trajectories for both training and evaluation.
- Habitat navigation uses low-level discrete actions including Move Forward, Turn Left, Turn Right, and Stop.
- Turn Left corresponds to 15 degrees in VLN-CE R2R but 30 degrees in VLN-CE RxR and HM3D-ObjNav.
- The appendix describes a simple strategy for unifying discrete actions across navigation tasks.
A.4 REAL-WORLD DEPLOYMENT SYSTEM
The real-world deployment system treats NavFoM as a visual-language-action model that predicts trajectories from camera observations and instructions, with embodiment-specific APIs executing them. The system uses remote-server inference to process compressed robot observations and return trajectories.
- NavFoM predicts robot trajectories from visual observations captured by one or more cameras together with language instructions.
- Off-the-shelf embodiment-specific APIs drive each robot along NavFoM’s predicted trajectory and may incorporate LiDAR or other sensors when necessary.
- The deployment architecture places NavFoM on a remote GeForce RTX 5090 server connected to robot clients through the Internet.
- Robots compress current observations and transmit them with the user instruction to the server, which outputs a trajectory for execution.
B.1 ADDITIONAL RESULTS ON AUTONOMOUS DRIVING
NavFoM is evaluated on the planning-oriented nuScenes autonomous-driving benchmark using open-loop metrics. It achieves comparable performance to autonomous-driving-specific baselines without explicitly modeling driving-related information.
- Table 8 reports the nuScenes comparison using open-loop metrics calculated according to DiffusionDrive.
- The table marks the best and second-best results with bold and underlining, respectively.
- NavFoM achieves comparable performance to strong autonomous-driving baselines on nuScenes without explicitly modeling driving-related information.