Source-linked AI summary
ViNT: A Foundation Model for Visual Navigation
Dhruv Shah, Ajay Sridhar, Nitish Dashora, Kyle Stachowicz, Kevin Black, Noriaki Hirose, Sergey Levine
TL;DR
Mobile robotics lacks a broadly adaptable foundation model because robots, environments, and task specifications vary widely. ViNT addresses this with image-goal pretraining on diverse robot data, then extends and adapts the model for navigation tasks. The paper reports cross-robot generalization, long-horizon exploration, and adaptation to modalities such as GPS and routing commands.
Problem
Foundation-model transfer is difficult in mobile robotics because environments, platforms, and applications are highly diverse.
Method
ViNT is a Transformer-based image-goal navigation model trained on heterogeneous robot trajectories and adaptable to downstream task specifications.
Results
ViNT generalizes across robots and environments, supports long-horizon navigation with diffusion-proposed subgoals, and adapts to GPS coordinates or routing commands.
Takeaways & Limitations
ViNT provides a broadly capable navigation policy that can support diverse mobile-robot applications through zero-shot deployment and downstream fine-tuning.
Takeaways & Limitations
ViNT requires more inference computation than simpler feedforward networks and assumes structural similarity in sensors and action representations.
Abstract
from arXiv · showhide
General-purpose pre-trained models ("foundation models") have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on singular datasets. ViNT can be augmented with diffusion-based subgoal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or routing commands) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establishes ViNT as an effective foundation model for mobile robotics. For videos, code, and model checkpoints, see our project page at https://visualnav-transformer.github.io.
1 Introduction
ViNT asks how foundation-model principles can extend to mobile robotics despite diverse environments, platforms, and applications. It proposes a cross-embodiment visual-navigation model intended for zero-shot deployment and downstream adaptation.
- Robotics complicates foundation-model transfer because environments, platforms, and applications are highly diverse.
- A robot foundation model should deploy zero-shot in novel settings and adapt to downstream objectives, goal specifications, behaviors, sensors, and embodiments.
- ViNT trains on camera-image goals using a general objective applicable to almost any mobile-robot dataset.
- ViNT supports zero-shot control of new robots, exploration, indoor mapping, kilometer-scale outdoor navigation, and fine-tuning to GPS or routing specifications.
- The authors release ViNT weights and training and deployment code through the project page.
2 Related Work
Prior mobile-robotics approaches commonly rely on platform-specific real-world data or simulation, whereas ViNT targets one generalist policy adaptable across robots and tasks.
- Existing mobile-robotics methods often learn from small real-world datasets tied to one platform or from simulation with paired robot and environment models.
- ViNT targets downstream navigation to GPS goals, goal images, and skill-conditioned driving with one generalist model rather than specialist solutions.
- Related work studies broad generalization, instruction following, or simulation-to-real transfer, while ViNT emphasizes small-data adaptation across embodiments and tasks.
- ViNT's architecture uses EfficientNet encoders and a Transformer decoder to predict goal distance and future actions.
3 The ViNT Model
ViNT learns image-goal navigation from heterogeneous trajectory data using a Transformer that predicts both dynamical distance and future actions. Its goal representation, embodiment-agnostic actions, and deployment design support cross-robot use.
- Image-goal navigation requires only videos and actions, avoiding ground-truth localization, semantic labels, and other metadata.
- ViNT receives current and past observations plus a subgoal image, then predicts time to the subgoal and H future actions.
- The 31M-parameter Transformer is trained end-to-end for efficient inference and downstream prompting or fine-tuning.
- ViNT jointly encodes current and goal observations because relative features help prevent the model from ignoring the goal.
- Training samples temporal context and a future observation as the subgoal, supervising both future actions and temporal distance.
- Relative waypoints normalized by robot top speed provide an embodiment-agnostic action space.
- The model trains on over 100 hours from 8 robotic platforms, while deployment runs at 4Hz with a robot-specific controller.
4 Long-Horizon Navigation with ViNT
ViNT extends short-horizon image-goal control into long-horizon navigation by combining its policy with an online topological graph, heuristics, and diffusion-generated subgoals. The resulting planner supports directed reaching and undirected exploration in unseen environments.
- Long-Horizon Navigation: A topological graph supplies short-horizon subgoals when tasks lack goal images or exceed ViNT's direct planning horizon.
- Long-Horizon Navigation: The robot builds an episodic-memory graph online, with nodes as subgoal observations and edges representing traversed or predicted-reachable paths.
- Long-Horizon Navigation: An A*-like planner generates, scores, selects, and executes subgoals while adding them to the graph for later shortcut discovery.
- Subgoal Generation with Diffusion: The subgoal-generation problem requires diverse, reachable samples from a high-dimensional, multimodal RGB-image distribution.
- Subgoal Generation with Diffusion: Diffusion generates diverse future-image candidates, which ViNT grounds using temporal distances and action rollouts before planning.
- Subgoal Generation with Diffusion: Latent-space subgoal sampling produced optimization challenges and poor performance in this framework, although it remains a future direction.
- Long-Horizon Navigation: Goal-directed heuristics support reaching distant goals, while removing them enables undirected exploration for workspace coverage.
5 ViNT: A Foundation Model For Downstream Tasks
ViNT’s learned navigational priors support adaptation beyond image-goal navigation through full-model fine-tuning or soft prompts for new goal modalities. These approaches transfer capabilities to new settings while requiring minimal task-specific data.
- ViNT adapts its navigational priors to downstream tasks beyond navigating to image goals by fine-tuning part or all of the model.
- Full model fine-tuning: Full-model fine-tuning improves on-task performance and transfers capabilities to new environments and embodiments without retraining from scratch.ViNT can master new environments and embodiments with as little as 1 hour of navigation data.
- Adapting to new modalities: A learned soft-prompt mapping converts subgoals such as 2D coordinates or routing directions into ViNT’s shared goal-token space.
- Adapting to new modalities: Adapting goal specifications with minimal data preserves ViNT’s performance and generalization while enabling new tasks.
6 Real-world Evaluation
ViNT is evaluated across multiple real-world robots, environments, navigation objectives, and adaptation settings. Results show strong exploration, goal-directed navigation, cross-embodiment transfer, fine-tuning, and emergent collision-avoidance behaviors.
- Evaluation Setup: Experiments deploy ViNT on five robotic platforms and address exploration, novel-robot generalization, fine-tuning, and new task specifications.The evaluation includes challenging indoor and outdoor environments previously unseen during training.
- 6.1 Navigation Performance: ViNT paired with physical search outperforms baselines in undirected goal-reaching across indoor and outdoor environments.Diffusion subgoal proposals support efficient paths, while other baselines struggle to explore large indoor environments.
- 6.1 Navigation Performance: ViNT uses goal-directed heuristics to reach 2D positions in unseen environments with higher success, greater collision-free distance, and fewer interventions than the prior state of the art.The evaluation uses wheel odometry, GPS coordinates, or satellite imagery, with improved outdoor SPL and crowded-space collision avoidance.
- 6.2 Zero-Shot Generalization: a Single Policy to Drive Any Robot: Policies trained across robot embodiments match or outperform single-robot specialists across studied platforms, with especially improved generalization to unseen robots such as Go 1.ViNT also shows positive transfer on the in-domain Vizbot, where it greatly outperforms a specialist model.
- 6.3 Broader Generalization via Fine-Tuning: ViNT achieves strong downstream performance after fine-tuning, including substantially higher success than the next-best baseline with as little as 1 hour of fine-tuning data.The results indicate adaptation to new environments with very little data.
- 6.5 Emergent Behaviors: Despite a simple self-supervised objective, ViNT exhibits emergent collision avoidance, robustness to pedestrians, and preferences for paved roads and hallway centers.With invalid random subgoals, ViNT reaches the goal 80% of the time and can ignore poor diffusion-generated subgoals while using valid ones.
7 Discussion
ViNT is presented as a generalist robot foundation model trained on diverse multi-robot data and supporting multiple navigation capabilities. Its limitations include inference cost and assumptions about robot structure, action representations, and sensors.
- Discussion: ViNT is trained for generic image-goal navigation on diverse data from many robots and supports long-horizon navigation, exploration, fine-tuning, and alternative task specifications.Supported adaptations include autonomous driving, GPS coordinates, and turn-by-turn routing commands.
- Discussion: ViNT generalizes across robots and environments, outperforms prior navigational models, adapts efficiently to new domains and tasks, and exhibits emergent behaviors.The paper gives dynamic-pedestrian navigation as an example of an emergent behavior.
- Limitations and Future Work: Inference is more computationally expensive than simpler feedforward convolutional networks, creating challenges for power-constrained platforms such as quadcopters.The authors note that the Transformer-based design remains significantly costlier at deployment time despite targeting efficient inference.
- Limitations and Future Work: ViNT assumes structural similarity across robots and cannot control quadcopter altitude, handle changed action representations, or accommodate sensors such as LIDAR.The authors suggest training on varied modalities and action spaces as future work.
A ViNT Model Architecture
ViNT uses visual observations and a subgoal image to predict both the time needed to reach the subgoal and a future action sequence. Its 31M-parameter Transformer is trained end-to-end from scratch.
- A ViNT Model Architecture: The architecture uses EfficientNet-B0 convolutional encoders to produce 512-dimensional embeddings for observation and subgoal images.All 18 EfficientNet-B0 layers are used and initialized from scratch.
- A ViNT Model Architecture: ViNT tokenizes current, past, and subgoal visual inputs, then predicts a sequence of H future actions and the temporal distance to the subgoal.The model receives current and past observations together with a subgoal image.
A.1 Goal-Conditioning Architectures
ViNT compares late fusion, early fusion, and FiLM goal-conditioning architectures. Its selected fusion design aims to retain early fusion’s navigation performance while preserving adaptation to new goal modalities.
- A.1 Goal-Conditioning Architectures: Late fusion encodes observations and goals independently before combining them in Transformer attention layers.This avoids channel-wise concatenation before the model input.
- A.1 Goal-Conditioning Architectures: Early fusion concatenates the goal image with every observation image before tokenization, enabling joint observation-goal features but reducing adaptation flexibility.The goal token is removed because goal information is already present in each observation token.
- A.1 Goal-Conditioning Architectures: FiLM conditions separately encoded observations using an EfficientNet goal encoder, but training was unstable for image-based navigation.The approach follows FiLM+EfficientNet conditioning while replacing the language encoder with an EfficientNet encoder.
- A.1 Goal-Conditioning Architectures: Late fusion performs poorly because relative observation-goal features matter, whereas early fusion is inflexible when adapting to new goal modalities.ViNT’s fusion architecture closely matches early fusion while supporting downstream adaptation.
B Implementation Details
The implementation combines image-to-image diffusion for short-horizon subgoals with topological-graph physical search and several long-range heuristics. It also specifies CARLA fine-tuning and GPS or command-based adaptation procedures.
- B.2 Subgoal Diffusion: The diffusion model maps a current image to candidate subgoal images sampled from g(o_si | o_t), using training pairs separated by 5–20 future timesteps.Image conditioning is implemented by channel-wise concatenation at the U-Net input.
- B.2 Subgoal Diffusion: Classifier-free guidance improves the visual fidelity of generated subgoals.The diffusion model uses a fixed linear noise schedule and an unweighted training objective.
- B.3 Long-Horizon Physical Search via Topological Graphs: Physical search maintains an open set of unvisited diffusion-generated subgoals and visits nodes using a cost combining graph distance, predicted distance, and a heuristic.The procedure follows a standard A∗-like search over a topological graph.
- B.3 Long-Horizon Physical Search via Topological Graphs: The search supports coverage exploration with h(s) = 0 and position-guided navigation with Euclidean distance h(s) = ∥s − G∥.A satellite-guided heuristic can instead estimate whether a subgoal lies on a trajectory toward the long-horizon goal.
- B.4 CARLA Fine-Tuning: CARLA fine-tuning uses 181 training trajectories in Town 01 and 52 held-out trajectories in Town 02, collected with a rule-based oracle agent.The sampled start and end locations were up to 900 meters apart.
- B.5 Adaptation: GPS adaptation removes ViNT’s goal encoder and learns a fixed tensor concatenated with ego-centric GPS coordinates before a two-layer MLP predicts the final Transformer token.Discrete-command adaptation similarly learns one latent per command and selects it by command index.
C Training Dataset
ViNT is trained and evaluated across diverse robotic platforms, datasets, environments, and navigation settings. The experiments compare generalist ViNT against specialist, model-based, and subgoal-generation baselines.
- Dataset composition: The training data spans over 100 hours of real-world navigation trajectories collected across 8 distinct robotic platforms.Platforms include commercially available robots and custom vehicles with widely varying dynamics and camera configurations.
- Dataset composition: Table 7 characterizes ViNT’s dataset as over 150 hours of navigation data across 8 robots in challenging indoor, outdoor, and off-road environments.
- Experimental settings: The coverage and guided exploration experiments use enclosed indoor settings, confined outdoor environments, goal images, GPS locations, satellite images, and learned heuristics.Experiments include LoCoBot, Vizbot, and Clearpath Jackal platforms, with failures defined by unreachable goals, time limits, or collisions.
- CARLA evaluation: CARLA experiments collect expert trajectories for fine-tuning and evaluate image-goal following by measuring average progress toward goals before collision.The setup uses 181 training trajectories, 52 held-out test trajectories, and goal distances up to 900 meters.
E.3.3 Adaptation Experiments
The adaptation experiments compare ViNT fine-tuning with models initialized from scratch, general-purpose visual representations, and another pre-trained navigation model. They also examine whether navigation-relevant features transfer when the encoder is frozen.
- Pre-training conditions: ImageNet, SimCLR, and VC-1 initialize ViNT’s visual encoder differently before fine-tuning on the CARLA on-task dataset.VC-1 is pre-trained on Ego4D, manipulation, navigation, and ImageNet images using masked auto-encoding.
- Encoder comparison: VC-1’s weak Section 6.4 performance may reflect its frozen encoder while the other visual encoders were fine-tuned.
- Encoder comparison: Table 8 compares ViNT fine-tuning with and without a frozen encoder against a general-purpose visual encoder.The table reports that frozen ViNT features appear to transfer more readily to out-of-distribution inputs than general-purpose features.