PaperlayerView exact revision on arXiv

Source-linked AI summary

FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

Lingfeng Zhang, Zeying Gong, Xiaoshuai Hao, Haoxiang Fu, Qiang Zhang, Mingliang Zhou, Hangjun Ye, Xiaojun Liang, Junwei Liang, Wenbo Ding

arXiv:2606.30367v1cs.RO

TL;DR

VLN agents must maintain spatial understanding across long action sequences, but many methods primarily generate actions without explicitly modeling world-state evolution. FutureNav jointly encodes text, visual, and spatial information and trains four complementary world-action objectives; with a 4B backbone, it achieves state-of-the-art performance across multiple VLN benchmarks and outperforms prior larger models.

  • Problem

    VLN requires long-horizon grounding and spatial reasoning, yet many existing policies lack explicit modeling of world states, action-induced transitions, and future states.

    Source: §1 · pp. 1–2, §2.1 · p. 3

  • Method

    FutureNav fuses text, visual, and spatial features in a VLM and jointly trains action policy, inverse dynamics, forward dynamics, and future spatial-state generation.

    Source: §1 · pp. 2–3

  • Results

    FutureNav achieves state-of-the-art performance on multiple VLN benchmarks with a 4B-scale backbone and outperforms prior larger VLN foundation models.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    Unified world-action modeling strengthens spatial and transition understanding during training while preserving an efficient policy-only inference pathway.

    Source: §1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.

1 Introduction

FutureNav addresses the limits of direct observation-to-action navigation by unifying spatially informed policy learning with world-state and transition modeling. Its four objectives improve benchmark performance while retaining an efficient policy-only inference pathway.

1 Introduction

  • Direct-action VLN methods generally lack explicit supervision for world-state representation, action-induced transitions, and future-state prediction, while many world-model systems add separate inference-time computation.This leaves action choice insufficiently coupled to how the environment changes and what state may follow.Source: §1 · pp. 1–2
  • The architecture fuses geometry-aware features from a frozen spatial encoder with visual tokens, giving the LLM explicit spatial-state information through residual addition.The model retains a single-pass autoregressive action-decoding pathway.Source: §1 · p. 2
  • The four training objectives cover direct action decoding, action inference from observation changes, action-conditioned next-state regression, and action-free short-horizon future-state prediction.At inference, only the action policy is required; dynamics and future prediction are optional.Source: §1 · pp. 2–3
  • FutureNav achieves state-of-the-art results on multiple VLN benchmarks with a 4B-scale backbone and outperforms prior larger VLN foundation models.The experiments also report strong zero-shot real-world navigation without real-world fine-tuning.Source: §1 · p. 3
  • FutureNav uses one VLM-based action-decoding policy to jointly learn action prediction, inverse dynamics, forward dynamics, and future spatial-state generation.The framework combines text, visual, and spatial features before action decoding.Source: §1 · pp. 2–3

2 Related Work

Related VLN systems increasingly use end-to-end VLM policies, while world-model approaches add imagined observations or latent foresight. FutureNav unifies policy, dynamics, and future-state prediction in a 3D-aware, instruction-conditioned latent-feature framework.

2.1 Vision-and-Language Navigation

  • Continuous VLN requires language grounding, observation-history retention, spatial understanding over long horizons, and executable low-level control in unseen environments.Earlier systems addressed these demands with waypoints, topological planning, semantic maps, and spatial memories.Source: §2.1 · p. 3
  • Most VLM-based VLN methods learn direct observation-to-action mappings without explicitly modeling world state or action-induced future spatial changes.This limits supervision for long-horizon spatial reasoning and decision robustness.Source: §2.1 · p. 3

2.2 World Modeling for Embodied Navigation

  • World-model navigation methods predict future observations or latent features for look-ahead planning, but may use separate acting policies, extra depth supervision, or only partial action-observation modeling.Latent prediction reduces the cost of pixel-level generation, but does not by itself unify all world-action relations.Source: §2.2 · p. 3
  • FutureNav instead combines action policy, forward dynamics, inverse dynamics, and future generation within one 3D-aware, instruction-conditioned policy operating in latent feature space.Its architecture uses a VLM backbone together with a frozen spatial encoder for spatial-state encoding.Source: §2.2 · p. 3, Figure 2 · p. 4

3 Methodology

FutureNav augments a VLM action policy with spatial features and auxiliary world-modeling objectives for continuous VLN. Training models action prediction, state transitions, and future spatial evolution, while inference can decode actions through only the policy branch.

3.1 Overview

  • The overview formulates continuous VLN as autoregressive generation of STOP, MOVE_FORWARD, TURN_LEFT, or TURN_RIGHT from instructions and egocentric observation history.The policy predicts the next low-level action as text.Source: §3.1 · p. 4

3.2 Input Encoding

  • A frozen spatial encoder processes the observation frames, and a trainable merger projects its features into the VLM hidden space for residual fusion with visual tokens.Spatial-aware and visual tokens are aligned before entering the language model.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 World-Action Modeling

  • FutureNav combines an autoregressive action policy with forward dynamics, inverse dynamics, and future generation heads for unified world-action modeling.The four modules are trained on top of the language-model hidden states.Source: §3.1 · p. 4, §3.4 · p. 6
  • The forward head predicts the next latent spatial feature from the current observation and teacher-forced action, representing the action-conditioned state transition.Its target is the pooled spatial-encoder feature of the next observation.Source: §3.3 · p. 5
  • The inverse head recovers the action from observed state changes, complementing forward dynamics with an action-state consistency constraint.It classifies the discrete action transforming the previous observation into the current one.Source: §3.3 · p. 5
  • The future-generation head predicts the next spatial feature without the action token, learning an action-agnostic prior over short-horizon scene evolution.It combines context and current visual hidden states and uses the next-observation spatial feature as its target.Source: §3.3 · pp. 5–6

3.4 Training Objective

  • Training combines policy, forward, inverse, and generation losses, while inference uses only the policy branch and therefore adds no world-modeling computation.The visual and spatial encoders remain frozen, while language components, the merger, and world-action heads are optimized.Source: §3.4 · p. 6

4 Experiments

FutureNav is evaluated on continuous VLN benchmarks using standard validation-unseen metrics, component and representation ablations, training-efficiency comparisons, and qualitative simulator and real-world analyses. It achieves strong benchmark performance with limited data, while its world-action objectives and spatial representations improve accuracy, convergence, and spatially consistent behavior.

4.1 Experimental Setup

  • Experiments use R2R-CE and RxR-CE validation-unseen splits with SR and SPL as primary metrics, plus NE, OS, and nDTW where applicable.The real-world deployment uses a single RGB camera and evaluates navigation without additional real-world fine-tuning.Source: §4.1 · p. 6

4.2 Main Results

  • With no external data, FutureNav-4B improves over JanusVLN by 4.4% in R2R SR and 6.0% in RxR SR, while FutureNav-8B reaches 55.5 SR and 51.4 SPL on R2R-CE.These models remain competitive on RxR-CE, indicating the gains do not simply come from additional training data.Source: §4.2 · p. 7
  • FutureNav-4B achieves the best overall R2R-CE results, improving over JanusVLN by 11.3% in NE, 8.1% in SR, and 7.9% in SPL despite slightly less external data.On RxR-CE, FutureNav-8B reduces NE by 29.7% over JanusVLN and improves SR/SPL/nDTW by 13.7%/15.4%/11.3%.Source: §4.2 · p. 8

4.3 Ablation Studies

  • Combining all three world-action objectives improves the policy-only baseline by 5.0 SR and 5.0 SPL while reducing NE from 6.00 to 5.13.Forward dynamics provides the largest individual gain, while future and inverse dynamics also improve performance, indicating complementary supervision beyond action imitation.Source: §4.3 · p. 8, Table 2 · p. 8
  • The proposed spatial latent representation reaches 55.1 SR and 50.1 SPL with the lowest NE of 5.13, outperforming VAE, VLM visual, and DINO semantic latents.The comparison supports explicitly modeling spatial-aware latent states for future world modeling.Source: §4.3 · p. 8, Table 3 · p. 8
  • At 42% of training, FutureNav reaches 42.3 SR and 37.8 SPL versus 33.2 SR and 27.9 SPL for the policy-only baseline.FutureNav reaches 50.2 SR after roughly 63% of training, matching the policy-only model’s final SR and demonstrating faster improvement before convergence.Source: §4.3 · pp. 8–10, Figure 4 · p. 8

4.4 Qualitative Analysis

  • In simulated VLN-CE comparisons, FutureNav better preserves spatial progress, follows landmark-conditioned turns, and stops nearer the intended target than JanusVLN [44].Zero-shot real-world examples likewise show landmark grounding, route maintenance, and appropriate stopping without additional real-world fine-tuning.Source: Figure 5 · p. 9, Figure 6 · p. 9, §4.4 · p. 10

5 Conclusion

FutureNav unifies action prediction with inverse dynamics, forward dynamics, and future spatial-state generation in a single multimodal action-decoding framework. It retains an efficient policy-only inference pathway, achieves state-of-the-art performance with a 4B-scale backbone, and shows complementary gains from its world-action objectives and spatial latent targets.

5 Conclusion

  • FutureNav jointly learns action prediction, inverse dynamics, forward dynamics, and future spatial-state generation instead of relying only on direct action imitation.The framework is designed for continuous vision-and-language navigation.Source: §5 · p. 10
  • FutureNav preserves an efficient policy-only inference pathway while strengthening spatial and transition understanding during training.Its unified design keeps action decoding single-path at inference.Source: §5 · p. 10
  • With only a 4B-scale backbone, FutureNav achieves state-of-the-art performance and improves over prior larger VLN foundation models.Ablations confirm complementary gains from the world-action objectives and spatial latent prediction targets.Source: §5 · p. 10

Related papers

Loading 2606.30367v1…