PaperlayerView exact revision on arXiv

Source-linked AI summary

X-MOBILITY: End-To-End Generalizable Navigation via World Modeling

Wei Liu, Huihua Zhao, Chenran Li, Joydeep Biswas, Billy Okal, Pulkit Goyal, Yan Chang, Soha Pouya

arXiv:2410.17491v3cs.RO

TL;DR

General-purpose robot navigation must handle diverse environments and platforms despite limitations in classical methods, learning-based generalization, and scarce supervisory data. X-MOBILITY uses an autoregressive latent world model, multi-task decoders, and decoupled world modeling and policy learning. It outperforms state-of-the-art navigation methods, transfers zero-shot from simulation to a real robot, and shows potential for cross-embodiment generalization.

  • Problem

    General-purpose navigation remains difficult because classical methods require extensive tuning and struggle with uncertainty, while learning-based methods face out-of-distribution generalization and data-scarcity challenges.

    Source: §I · p. 1

  • Method

    X-MOBILITY combines an autoregressive latent world model and multi-task decoders with decoupled world modeling and action-policy imitation trained from heterogeneous data sources.

    Source: §I · pp. 1–2

  • Results

    X-MOBILITY outperforms state-of-the-art navigation methods, achieves zero-shot Sim2Real transfer, and shows potential for cross-embodiment generalization.

    Source: §I · p. 2, A. Sim2Real Transfer · p. 6

  • Takeaways & Limitations

    The approach supports generalizable navigation in unseen environments while using photorealistic synthetic data and varied sources with or without supervisory control.

    Source: §I · p. 2, A. Sim2Real Transfer · p. 6

Abstract

from arXiv · show

General-purpose navigation in challenging environments remains a significant problem in robotics, with current state-of-the-art approaches facing myriad limitations. Classical approaches struggle with cluttered settings and require extensive tuning, while learning-based methods face difficulties generalizing to out-of-distribution environments. This paper introduces X-Mobility, an end-to-end generalizable navigation model that overcomes existing challenges by leveraging three key ideas. First, X-Mobility employs an auto-regressive world modeling architecture with a latent state space to capture world dynamics. Second, a diverse set of multi-head decoders enables the model to learn a rich state representation that correlates strongly with effective navigation skills. Third, by decoupling world modeling from action policy, our architecture can train effectively on a variety of data sources, both with and without expert policies: off-policy data allows the model to learn world dynamics, while on-policy data with supervisory control enables optimal action policy learning. Through extensive experiments, we demonstrate that X-Mobility not only generalizes effectively but also surpasses current state-of-the-art navigation approaches. Additionally, X-Mobility also achieves zero-shot Sim2Real transferability and shows strong potential for cross-embodiment generalization.

I. INTRODUCTION

X-MOBILITY addresses the difficulty of robust, generalizable robot navigation by combining world modeling, multi-task latent representations, and decoupled policy learning. It outperforms state-of-the-art methods and supports zero-shot Sim2Real transfer across challenging, unseen environments.

  • Existing classical methods require complex deployment-specific tuning and struggle with uncertainty, while learning-based methods face sampling, covariate-shift, and data-scarcity challenges.POMDP-based methods also demand substantial system-modeling effort and computational resources.Source: §I · p. 1
  • Its lightweight autoregressive architecture learns a latent representation that probabilistically captures world states and dynamics.Source: §I · pp. 1–2
  • A multi-stage pipeline uses large-scale photorealistic synthetic datasets generated with NVIDIA Isaac Sim to train the navigation model.Source: §I · p. 2
  • X-MOBILITY introduces an end-to-end world-model-based navigation model that outperforms state-of-the-art methods in challenging environments absent from training data.The model is designed to generalize across mobility applications and out-of-distribution scenarios.Source: §I · pp. 1–2
  • Decoupling world modeling from action-policy imitation enables training from heterogeneous data, including off-policy data without supervisory controls.On-policy data with supervisory control supports action-policy learning, while off-policy data supports world-dynamics learning.Source: §I · p. 2
  • X-MOBILITY achieves zero-shot Sim2Real transferability and shows potential for generalization across different robot embodiments.Source: §I · p. 2

II. X-MOBILITY

X-MOBILITY formulates navigation as solving a POMDP in a learned latent state space rather than a manually specified one. Recurrent state estimation maintains a belief over partially observed states for policy learning.

A. Problem Formulation

  • X-MOBILITY solves the navigation POMDP in a learned state representation using state estimation, state prediction, and action-policy networks.The policy network imitates a teacher policy assumed to approximate the optimal policy.Source: A. Problem Formulation · p. 2

B. Belief Representation In Learned State Space

  • The learned representation avoids manually defining an expressive yet computationally tractable state space for generalizable navigation.Source: B. Belief Representation In Learned… · p. 2
  • Partial observability is handled by maintaining a Normal-distribution belief whose mean and covariance are continuously estimated and updated from new observations.The belief is parameterized as s ∼ N(µθ, σθ).Source: B. Belief Representation In Learned… · p. 2

C. State Estimation Network

  • The state estimation network uses recurrent history to capture non-Markovian effects when motion and observation models are unknown.Observation embeddings and applied actions are used to model the belief state.Source: C. State Estimation Network · p. 2
  • The previous history and current state are concatenated into a latent state used by the multi-task decoders.Source: C. State Estimation Network · p. 2

1) Model Implementation:

X-MOBILITY implements recurrent belief estimation with neural networks and encodes visual and motion observations into compact embeddings. The architecture combines these inputs with learned history to estimate latent states.

1) Model Implementation:

  • The state-estimation transition uses a gated recurrent unit, while multilayer perceptrons parameterize the Normal belief distribution.The action command is embedded, combined with history and observation features, and sampled into a learned state.Source: 1) Model Implementation · p. 2

2) Observation Encoding:

  • The model encodes front-view images and robot motion, including linear forward velocity, into a compact observation representation.A pretrained DINOv2 image encoder produces a visual token, while fully connected layers encode speed before concatenation.Source: 2) Observation Encoding · pp. 2–3
  • The state estimator and predictor, together with multi-task decoders, generate a rich latent representation for action-policy learning.Source: 1) Model Implementation · p. 2, Figure 2 · p. 3

3) Multi-task Decoding:

Multi-task decoders train the latent state to retain visual and semantic information relevant to navigation. A separate predictor forecasts future states by matching its beliefs to those inferred from observations.

3) Multi-task Decoding:

  • RGB reconstruction and perspective-view semantic segmentation provide complementary decoding tasks that shape a navigation-relevant latent state.RGB reconstruction uses a latent diffusion model, while semantic segmentation uses a StyleGAN decoder and cross-entropy loss.Source: 3) Multi-task Decoding · p. 3
  • End-to-end training of encoders, decoders, and state estimation produces a rich latent state that supports action-policy learning.Source: 3) Multi-task Decoding · p. 3

D. State Prediction Network

  • The state prediction network forecasts consequences of future actions without observation embeddings, supporting long-horizon planning.Source: D. State Prediction Network · p. 3
  • A KL-divergence loss aligns predictor and estimator belief distributions so predicted future states match observed data.This design improves navigation in dynamic environments and broadens the world model’s applications.Source: D. State Prediction Network · p. 3

E. Action Policy Network

X-MOBILITY combines latent-state navigation with route-conditioned action prediction and separates world-model learning from policy imitation. This design supports training from both logged off-policy actions and on-policy supervisory data.

E. Action Policy Network

1) Route Encoding:

  • Route information is transformed into the robot frame and encoded as a regional route embedding, allowing map-free operation with simple lines or waypoint sequences.The route may be a straight line to the destination or waypoints from a topological graph.Source: 1) Route Encoding · p. 3

2) Action Decoding:

  • Self-attention fuses the latent state and route embedding before an MLP decodes six-dimensional velocity commands and an optional five-pose navigation path.The policy is trained with L1 imitation loss; the path output shapes latent states and supports safety checking.Source: 2) Action Decoding · pp. 3–4

F. Multi-stage Training

  • A two-stage pipeline first trains the world model with logged off-policy actions, then jointly trains the policy and world model using on-policy supervisory data.This uses datasets without high-quality teacher policies to learn dynamics before policy imitation.Source: F. Multi-stage Training · p. 4

III. EXPERIMENT SETTING

The experiments use photorealistic Isaac Sim warehouse data collected with random and Nav2 teacher policies. Training first uses random-action sequences for world modeling and then teacher-policy data for action-policy learning.

A. Dataset

  • Random-action data uses randomized starting positions, action sequences, and coverage logic, while teacher-policy data uses closed-loop Nav2 tasks with random start-goal pairs.The two collection procedures provide exploratory and supervisory action inputs.Source: A. Dataset · p. 4
  • The dataset contains 100K Nav2-policy frames and 160K random-action frames collected evenly across four warehouse scenarios.Data was recorded at 5Hz with varied textures, layouts, and realistic lighting in large-scale, semi-structured, cluttered settings.Source: A. Dataset · p. 4
  • Each frame records RGB observations, semantic labels, robot-frame route segments, speed, path poses, and velocity commands for navigation learning.Semantic classes include navigable space, forklifts, cones, signs, pallets, fences, and background.Source: A. Dataset · p. 4

B. Training

  • World-model training runs for 100 epochs on five-frame random-action sequences, followed by 100 epochs of joint policy training on teacher-policy data.Training uses batch size 32 across eight H100 GPUs, with AdamW at learning rate 10^-5.Source: B. Training · p. 4

C. Evaluation

Evaluation combines open-loop prediction metrics with closed-loop navigation tests spanning in-distribution and unseen warehouse scenarios. X-MOBILITY consistently outperforms the compared methods in closed-loop navigation, while random-action pretraining improves out-of-distribution performance.

C. Evaluation

  • The benchmark contains 10 warehouse scenarios with narrow corridors, clutter, and low obstacles, including five environments absent from training.Fifty additional randomly generated cluttered narrow-corridor scenes test generalization, with success rate, weighted trip time, and angular acceleration as metrics.Source: C. Evaluation · pp. 4–5

A. Navigation Performance

  • X-MOBILITY consistently outperforms Nav2, behavior cloning, and MILE across closed-loop navigation metrics.Compared with Nav2, it achieves higher mission success and smoother motion in every scenario; BC and MILE perform worse, especially in difficult or out-of-distribution settings.Source: A. Navigation Performance · p. 5

C. Evaluation

  • All learning-based methods achieve similar open-loop test metrics, whereas closed-loop evaluation separates their navigation performance.Open-loop evaluation measures linear-speed, angular-speed, and path-prediction MAE.Source: A. Navigation Performance · p. 5, Table II · p. 5

1) World Model Pretrain:

  • Pretraining the world model on 160K random-action frames improves navigation particularly in random-obstacle scenarios and improves semantic segmentation performance.The result supports using off-policy data to strengthen out-of-distribution behavior.Source: 1) World Model Pretrain · p. 5

2) Semantic Decoding:

Semantic decoding makes latent representations more useful for generalization, while recurrent history tracking produces smoother closed-loop behavior. The diffusion-policy variant improves path prediction but degrades action quality and navigation reliability.

2) Semantic Decoding:

  • Without semantic decoding, warehouse performance improves but random-obstacle performance declines, indicating reduced generalizability.Attention analysis shows semantic decoding directs attention toward objects such as signs, pallets, and fences.Source: 2) Semantic Decoding · p. 5
  • Recurrent history tracking produces smoother motion than resetting history and latent state at every cycle.Resetting frequently causes overshooting during turns, while recurrence integrates history over longer horizons than those seen during training.Source: 4) Diffusion Policy · p. 5

4) Diffusion Policy:

  • The diffusion policy improves path prediction but degrades linear action commands and struggles to reach destinations without collisions.It conditions on policy state to jointly decode commands and paths, but its overall closed-loop behavior is inconsistent.Source: 4) Diffusion Policy · p. 5, Table III · p. 6
  • The semantic-segmentation evaluation compares predicted IoU across five future steps and model variants with or without pretraining or learned-policy prediction.The state predictor decodes future latent states into interpretable semantic segmentation.Source: Table IV · p. 6, Figure 5 · p. 6

C. World State Prediction

X-MOBILITY predicts future latent world states and decodes them into semantic segmentation, while transferring to real-robot navigation without fine-tuning.

C. World State Prediction

  • X-MOBILITY consistently predicts navigable surfaces across five future steps, with pretraining improving IOU across semantic classes while policy-enabled prediction becomes less accurate at longer horizons.The evaluation compares pretrained and non-pretrained models, with and without learned-policy prediction.Source: C. World State Prediction · p. 6, Figure 6 · p. 6

A. Sim2Real Transfer

  • Zero-shot deployment on a Nova Carter robot enabled safe navigation through cluttered obstacles in an unseen laboratory environment without fine-tuning.The real-world setup used obstacle and lighting variations, with results reported as comparable to simulation benchmarks.Source: A. Sim2Real Transfer · p. 6, Figure 7 · p. 7, Table V · p. 7
  • DINOv2 features, a probabilistic world model, and photorealistic Isaac Sim data are identified as factors supporting robustness during zero-shot Sim2Real transfer.The probabilistic design is intended to handle dynamic real-robot noise that simulation cannot fully capture.Source: A. Sim2Real Transfer · pp. 6–7

B. Cross Embodiment

X-MOBILITY generalizes across four robot embodiments in simulation, while future work targets stronger platform adaptation and richer dynamic-scene training.

B. Cross Embodiment

  • X-MOBILITY successfully navigated Nova Carter, Forklift, Unitree Go2, and Unitree G1 when paired with appropriate controllers or locomotion policies.Performance varied with camera placement and robot dynamic constraints, but the model showed potential for cross-embodiment generalization.Source: B. Cross Embodiment · p. 7, Figure 8 · p. 7

VI. FUTURE WORK

  • The paper positions X-MOBILITY as a generalizable navigation model integrating world modeling with imitation learning for out-of-distribution scenarios.This conclusion frames cross-embodiment capability as an ongoing development target.Source: §VI · p. 7
  • Future work proposes detailed robot-specification encoding, reinforcement-learning fine-tuning, and datasets containing more diverse scenes with dynamic obstacles.These directions aim to improve adaptation across embodiments and clarify the world model’s role in action-policy learning.Source: §VI · p. 7

APPENDIX I MODEL DESCRIPTION

The appendix lists X-MOBILITY’s model parameters and training hyperparameters in dedicated tables.

APPENDIX I MODEL DESCRIPTION

Loading 2410.17491v3…