Source-linked AI summary
GNM: A General Navigation Model to Drive Any Robot
Dhruv Shah, Ajay Sridhar, Arjun Bhorkar, Noriaki Hirose, Sergey Levine
TL;DR
GNM addresses the limited data available for vision-based navigation by training a goal-conditioned policy across heterogeneous robot datasets. It standardizes action representations and uses temporal context, yielding cross-robot generalization, including deployment on new platforms and an underactuated quadrotor.
Problem
Vision-based navigation policies are constrained by limited single-robot data despite shared navigation objectives across differing embodiments, sensors, and dynamics.
Method
GNM aggregates heterogeneous navigation trajectories and trains an image-goal policy using normalized shared actions and robot-specific temporal context.
Results
The omnipolicy outperforms single-dataset policies, generalizes to unseen environments and robots, and controls an underactuated quadrotor.
Takeaways & Limitations
A single navigation policy trained on diverse robot data can support broad real-world deployment across environments and embodiments.
Takeaways & Limitations
The system assumes ground robots with forward-facing RGB cameras and does not explicitly handle broader sensing, actuation, or traversability differences.
Abstract
from arXiv · showhide
Learning provides a powerful tool for vision-based navigation, but the capabilities of learning-based policies are constrained by limited training data. If we could combine data from all available sources, including multiple kinds of robots, we could train more powerful navigation models. In this paper, we study how a general goal-conditioned model for vision-based navigation can be trained on data obtained from many distinct but structurally similar robots, and enable broad generalization across environments and embodiments. We analyze the necessary design decisions for effective data sharing across robots, including the use of temporal context and standardized action spaces, and demonstrate that an omnipolicy trained from heterogeneous datasets outperforms policies trained on any single dataset. We curate 60 hours of navigation trajectories from 6 distinct robots, and deploy the trained GNM on a range of new robots, including an underactuated quadrotor. We find that training on diverse data leads to robustness against degradation in sensing and actuation. Using a pre-trained navigation model with broad generalization capabilities can bootstrap applications on novel robots going forward, and we hope that the GNM represents a step in that direction. For more information on the datasets, code, and videos, please check out our project page https://sites.google.com/view/drive-any-robot.
I. INTRODUCTION
GNM asks whether heterogeneous navigation data from multiple robots can support a broadly generalizable, reusable vision-based navigation policy. The paper frames shared navigation objectives and reports an omnipolicy that transfers across robots, environments, and changing sensing or actuation conditions.
- GNM addresses whether cross-robot datasets can overcome the limited data available for learning-based navigation policies.
- Wheeled robots, quadrupeds, and drones share abstract objectives such as exploration, path planning, goal reaching, and collision avoidance despite differing embodiments.
- The authors aggregate navigation trajectories from 6 robots, train an embodiment-agnostic omnipolicy, and deploy it on 4 platforms, including 2 new robots.
- The same learned policy performs goal-reaching in challenging indoor and outdoor environments and outperforms policies trained with any single dataset.
- The work positions heterogeneous on-robot data as complementary to Internet-scale representation learning and domain-transfer methods.
III. MULTI-ROBOT TRAINING DATASET
GNM constructs a heterogeneous navigation dataset from multiple robot platforms, environments, sensing conditions, and behaviors. The released data combines more than 60 hours of real-world trajectories with RGB observations, actions, and odometry.
- The dataset aggregates navigation trajectories from 8 source datasets collected on platforms with varying dynamics, sensors, and behaviors.
- More than 60 hours of trajectories span 6 robotic platforms, including TurtleBot, Clearpath Jackal, Warthog, Spot, a Yamaha Viking ATV, and an RC Car.
- The trajectories combine tele-operated and autonomous navigation in diverse indoor and off-road environments, with top speeds ranging from 0.2 to 10m/s.
- Each example pairs forward-facing monocular RGB images with commanded actions and local odometry measurements.
- Different camera parameters create variation in camera pose and intrinsic parameters while preserving a common monocular RGB sensor type.
IV. TRAINING A GENERAL NAVIGATION MODEL
The GNM trains a common image-goal navigation policy by aligning heterogeneous robots through shared action representations and robot-specific temporal context. It predicts normalized waypoint-based actions and goal distance for integration with topological navigation.
- IV. TRAINING A GENERAL NAVIGATION MODEL: GNM formulates a common image-goal navigation task in which a policy navigates from egocentric observations toward a goal image.
- IV. TRAINING A GENERAL NAVIGATION MODEL: The framework emphasizes standardized action representations and a robot-capability summary so heterogeneous robots can share one policy.
- A. A Shared Abstraction Across Robots: Raw actions differ substantially across robots, including differential-drive velocities versus Ackermann throttle and steering, making direct joint learning difficult.
- A. A Shared Abstraction Across Robots: GNM uses relative waypoints, yaw change, and temporal distance to the goal as a shared mid-level action representation derived from local odometry.
- A. A Shared Abstraction Across Robots: Robot-specific controllers unnormalize the waypoints and track them to produce low-level velocities or motor commands.
B. Embodiment Context
GNM supplies robot embodiment information through context so one policy can adapt its behavior to differing configurations and dynamics. Experiments identify temporally consistent context from consecutive observations as the most effective representation.
- A capability summary can encode properties such as robot size and turning radius, helping the policy distinguish valid behaviors across embodiments.
- The most effective representation uses temporally consistent context from k consecutive past observations, enabling deployment on novel robot configurations.
C. Implementation Details
The GNM combines embodiment context and transformed actions with a goal-conditioned, temporally conditioned policy for heterogeneous robot data. Deployment uses a topological map and robot-specific low-level controllers to execute predicted waypoints.
- Embodiment context and action-space transformation allow a simple goal-reaching policy to learn from heterogeneous robot datasets.
- The policy takes current and goal observations, predicts normalized waypoints and distances, and stacks k = 5 past observations as temporal context.
- The GNM omnipolicy was evaluated on four robots in challenging indoor and outdoor environments.
- Training pairs image-goal observations from the same trajectory as positives and observations from different trajectories as negatives, with jointly trained distance and action heads.
- Deployment combines the goal-reaching policy with a topological map whose edges use temporal distance estimates, while a robot-specific controller tracks predicted waypoints.
V. DEPLOYING THE GNM ACROSS ROBOTS
The deployment study evaluates whether multi-robot training generalizes across robots and environments, improves over single-domain training, and remains robust under degradation. It also examines the importance of the model’s design choices.
- The experiments ask whether multi-robot training enables generalization to novel robots and environments.
- They test whether GNM policies outperform policies trained solely on single-domain data.
- The study analyzes how important the design choices in Section IV are for GNM performance.
- The experiments examine whether policies trained with multiple datasets are more robust to degradation than single-domain policies.
A. Meet the Robots
The GNM is deployed across four robot platforms, including unseen ground robots and a quadrotor, using one policy trained on heterogeneous data. Across LoCoBot and Jackal evaluations, diverse-data policies generalize better than single-domain baselines.
- Robot platforms: The evaluation uses four platforms, including a quadrotor and two other robots without corresponding training data.
- Robot platforms: Vizbot uses a fisheye camera and has no training data from Vizbot or another Roomba-like robot.
- Robot platforms: The DJI Tello is an unseen quadrotor restricted to horizontal-plane navigation 1m above the ground.
- Robot platforms: LoCoBot has no corresponding training data, while Jackal differs from related collection platforms in camera and mounting height.
- Zero-shot deployment: The same GNM controls four robots without per-robot fine-tuning, including a Tello unseen during training, and outperforms single-robot policies by up to 5x in some cases.
- Zero-shot deployment: On LoCoBot and Jackal, increasingly diverse training data improves navigation across unseen indoor and outdoor settings, outperforming single-domain policies.
- Zero-shot deployment: Table II reports mean success rates and states that GNM-Mid outperforms the best single-robot policy for each evaluated robot.
C. A Systematic Analysis of the Design Space
The design-space analysis evaluates GNM choices on an unseen LoCoBot across indoor environments with increasing navigation complexity. Complexity varies from open, smooth layouts to larger settings combining tight passages and multiple turns.
- Each design choice is evaluated on an unseen LoCoBot without corresponding training data.
- Easy environments have wide passages and smooth turns, whereas Moderate environments contain tight passages or sharp turns.
- Hard environments extend up to 50m and combine tight passages with multiple turns.
1) Shared Action Space:
Standardized action representations and embodiment context are central design choices for effective multi-robot navigation. Normalized waypoints perform best across varied environments, while temporal context improves harder navigation tasks.
- Shared Action Space: A normalized-waypoint policy significantly outperforms velocity- and waypoint-based policies, including in challenging environments.Velocity actions work well in easier environments, but velocity and waypoint policies struggle with dynamic maneuvers such as sharp turns.
- Embodiment Context: Adding either temporal or static embodiment context significantly boosts performance in harder environments requiring tight navigation.The temporal variant is superior, indicating that information such as speed and turning radius supports generalization.
- Design Choices: Action representation, goal-conditioned architecture, and embodiment context are important design choices for multi-robot learning.The systematic analysis in Table V identifies all three choices as important for facilitating multi-robot learning.
3) Policy Architecture:
The policy architecture is evaluated alongside robustness to changes in robot parameters. GNM compensates for steering limits and other degradations while reaching navigation goals.
- Policy Architecture: The study compares single-encoder stacking, Siamese encoding, and a conditional architecture for incorporating goal information.These are the three policy architectures evaluated for goal-conditioned navigation.
- Robustness to Degradation: Training across heterogeneous data encourages shared affordances that improve robustness to variation in sensor placement and mechanical properties.The paper tests this robustness under actuation, viewpoint, and physical-damage degradations.
- Robustness to Degradation: GNM compensates for clipped maximum angular velocity by taking a longer, smoother path without localization failures.The policy also reaches goals after camera-position perturbations and physical damage during navigation.
VI. DISCUSSION
The paper demonstrates that a general navigation policy trained on multiple robots can control novel platforms, including an underactuated quadrotor. It also identifies capability assumptions and dataset scale as important limitations.
- VI. DISCUSSION: A multi-robot goal-conditioned policy controls new robots in challenging environments, including an underactuated quadrotor.The demonstrated robots range from RC cars to ATVs, and the approach supports real-world navigation on platforms not seen during training.
- VI. DISCUSSION: GNM policies are more robust than single-domain policies to degraded actuation, perturbed sensor viewpoints, and physical damage.These degradation scenarios are summarized in Fig. 5.
- VI. DISCUSSION: The system assumes ground robots with forward-facing RGB cameras and does not explicitly model broader capability differences.Future work includes diverse sensing, actuation beyond speed and steering, and traversability.
- VI. DISCUSSION: The 60-hour dataset could be expanded because larger and broader data may enable better generalization.The paper presents this as a limitation of the current instantiation.
- VI. DISCUSSION: A broadly generalizing navigation model may serve as a pre-trained base for downstream navigation applications.The authors compare this prospective role to the use of pre-trained backbones in computer vision.