PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTAv2: World Imagination for Indoor Vision-and-Language Navigation

Yanjia Huang, Xianshun Jiang, Xiangbo Gao, Mingyang Wu, Zhengzhong Tu

arXiv:2512.00041v1cs.ROcs.CV

TL;DR

VLN agents need to follow language instructions while navigating continuous spaces, but prior methods struggle with reachability, risk, and brittle long-horizon prediction. VISTAv2 generates short-horizon futures conditioned on observations, actions, and instructions, projects them into an online value map, and fuses that guidance with a standard planner. It improves benchmark performance and path efficiency, while ablations support action-conditioned imagination, value fusion, and map-space planning.

  • Problem

    VLN methods can mistake visual relevance for reachability, while long-horizon world-model objectives are brittle and slow.

    Source: §1 · p. 1

  • Method

    VISTAv2 uses an action-aware diffusion world model to generate short-horizon futures, convert them into an egocentric value map, and fuse that map with the planner’s native score.

    Source: §1 · p. 1, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Compared with VISTA, VISTAv2 improves Val-Unseen SR and SPL by +3.6 and +5.4, reduces TL by 19%, and improves Test-Unseen SPL by +2.3.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Action-conditioned imagination expressed as map-space value and fused with the existing planner provides a practical, interpretable route to more efficient VLN.

    Source: §4.2 · p. 6, §5 · p. 8

    Rollouts are short-horizon and depend on monocular depth and pose estimates, with failures in low-texture, reflective, or cluttered scenes.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-and-Language Navigation (VLN) requires agents to follow language instructions while acting in continuous real-world spaces. Prior image imagination based VLN work shows benefits for discrete panoramas but lacks online, action-conditioned predictions and does not produce explicit planning values; moreover, many methods replace the planner with long-horizon objectives that are brittle and slow. To bridge this gap, we propose VISTAv2, a generative world model that rolls out egocentric future views conditioned on past observations, candidate action sequences, and instructions, and projects them into an online value map for planning. Unlike prior approaches, VISTAv2 does not replace the planner. The online value map is fused at score level with the base objective, providing reachability and risk-aware guidance. Concretely, we employ an action-aware Conditional Diffusion Transformer video predictor to synthesize short-horizon futures, align them with the natural language instruction via a vision-language scorer, and fuse multiple rollouts in a differentiable imagination-to-value head to output an imagined egocentric value map. For efficiency, rollouts occur in VAE latent space with a distilled sampler and sparse decoding, enabling inference on a single consumer GPU. Evaluated on MP3D and RoboTHOR, VISTAv2 improves over strong baselines, and ablations show that action-conditioned imagination, instruction-guided value fusion, and the online value-map planner are all critical, suggesting that VISTAv2 offers a practical and interpretable route to robust VLN.

1. Introduction

VISTAv2 addresses VLN limitations by combining short-horizon, action-conditioned imagination with map-space value guidance while preserving a standard planner. It targets reachability, collision risk, and long-rollout brittleness through score-level fusion and efficient deployment.

Motivation

  • Language–vision matching can favor visually relevant but unreachable targets, while long-horizon world-model optimization compounds pose, sampling, and appearance errors.These limitations can cause detours, backtracking, artifact chasing, and fewer candidates evaluated per step.Source: §1 · p. 1

Contributions

  • VISTAv2 rolls out short-horizon, action-conditioned futures and converts them into an egocentric value map for planner guidance.The system is designed as a test-time plug-in rather than a replacement for frontier-based planning.Source: §1 · p. 1

Approach

  • VISTAv2 fuses imagined value, language priors, and the planner’s native score to rank frontier-based candidate trajectories, then executes the first control.The pipeline combines local mapping, frontier proposals, imagined futures, and receding-horizon execution.Source: §1 · p. 1, Figure 1 · p. 2

Results

  • Compared with VISTA, VISTAv2 improves Val-Unseen SR and SPL by +3.6 and +5.4 while reducing TL from 13.26 to 10.73.It also improves Test-Unseen SPL by +2.3 while reducing TL from 14.20 to 12.44.Source: §1 · p. 2

2. Related Work

Related VLN work uses semantic frontier scoring or generative world models, but continuous navigation still requires stronger reachability and collision-risk reasoning. VISTAv2 combines action-conditioned short-horizon video imagination with map-based, score-level planning guidance.

Vision-and-Language Navigation

  • VLN research has progressed from panoramic navigation graphs to continuous 3D control, while foundation and VLM methods strengthen language grounding and zero-shot generalization.Continuous settings expose realistic collision and geometric challenges beyond graph-constrained navigation.Source: §2 · p. 2

Video Generation as World Models

  • Video generative models support controllable future prediction for planning, including action-conditioned navigation world models used to simulate trajectories and verify candidate plans.Conditional Diffusion Transformers provide scalable context processing, while deployment can use MPC and re-ranking without rewriting the planner.Source: §2 · pp. 2–3

VISTAv2

  • VISTAv2 conditions its video diffusion world model on recent egocentric frames, instructions, and candidate trajectories to generate short-horizon future observations.Pose integration produces the trajectory conditioning used by the action-aware CDiT in VAE latent space.Source: Figure 2 · p. 3

Language Guided Mapping and Frontier Exploration

  • Language-guided mapping ranks frontier candidates using vision–language similarity, but static semantic scores only indirectly capture reachability and collision risk.VISTAv2 adds an action-conditioned, short-horizon prior by projecting imagined futures into an egocentric value map and fusing it with the native objective.Source: §2 · p. 3

3. Method

VISTAv2 augments a standard frontier-based VLN planner with language-conditioned, action-aware imagination and an online egocentric value map. It rolls out candidate futures, scores instruction relevance and navigability, projects evidence into a map, and fuses the result with the planner’s native objective.

3. Method

  • VISTAv2 adds action-conditioned future imagination and score-level value fusion to a standard frontier-based planner without replacing its search procedure.The system executes only the first control of the highest-scoring candidate and repeats the process online.Source: §3 · p. 3, §3.1 · p. 4

3.1. Pipeline Overview

  • The pipeline expands candidates, imagines futures, scores them with a language–vision reasoner, fuses imagined and prior values with planner scores, and retains the top-K candidates.After ranking, the planner executes the best candidate’s first control before updating the map and repeating the loop.Source: §3.1 · pp. 3–4

3.2. World Model Formulation

  • For each candidate action sequence, the world model predicts short-horizon egocentric futures conditioned on recent observations, poses, instructions, and controls.The predictor operates in latent space with a Conditional Diffusion Transformer, decoding only a sparse subset of frames for downstream scoring.Source: §3.2 · p. 4
  • The continuous egocentric action space specifies planar displacement, yaw change, and step-duration scaling under platform limits, with STOP available when appropriate.Candidate trajectories are resampled to a fixed horizon and represented with per-timestep action tokens, time codes, and a learned turn-versus-straight mode bit.Source: §3.2 · p. 4

3.3. Imagination to Value (I2V)

  • The I2V head combines instruction alignment, traversability/free-space, and obstacle or uncertainty cues into normalized confidence maps projected onto an egocentric grid.Depth-aware splatting, bilinear accumulation, footprint smoothing, and field-of-view masking produce the spatial value representation.Source: §3.3 · pp. 4–5
  • Multiple imagined frames are aggregated with temporal discounting, while temperature-controlled log-sum-exp preserves promising peaks without the brittleness of hard maxima.The resulting value map is sampled along each candidate path and fused with the planner’s native score.Source: §3.3 · p. 5

2 V prior

The value-fusion implementation combines imagined and prior path scores with the base planner score, while using efficient latent rollouts and sparse decoding. Training uses indoor egocentric trajectories and a large-scale optimization setup.

2 V prior

  • The fused candidate score is defined as the base score plus λ1 times the imagined-path score and λ2 times the prior-path score.Source: §2 · p. 5
  • A fixed uncertainty gate with threshold θ=0.6 disables imagination when rollout uncertainty is high.The threshold is used across all experiments and is referenced with Figure 5.Source: §2 · p. 5

3.4. World Model Details

  • The world model uses a CDiT-L backbone with per-timestep action tokens and an instruction embedding, processing 224 × 224 inputs in bf16 precision.The encoder and decoder follow the Navigation World Model architecture.Source: §3.4 · p. 5
  • Training uses 489k indoor egocentric RGB frames from HM3D and MP3D organized into trajectories with poses and actions.The split follows navigation splits, and trajectories shorter than a fixed minimum length are discarded.Source: §3.4 · p. 5
  • Optimization runs for approximately 200k steps on 8 RTX 6000 Ada GPUs using AdamW, cosine decay with warmup, gradient clipping, EMA, mixed precision, and effective batch size 16.Training takes approximately 8 days under the reported schedule.Source: §3.4 · p. 5
  • Table 1 evaluates R2R methods under a shared 3 m success radius and identical stop rules.These evaluation conditions define the scope of the reported R2R comparison.Source: Table 1 · p. 6

4. Experiments

VISTAv2 is evaluated on continuous VLN benchmarks and shows stronger success and path-efficiency results than baselines, while ablations isolate the benefits of imagination, value fusion, and uncertainty gating.

4.1. Experiment Setup

  • Across R2R and RoboTHOR, VISTAv2 improves SR/SPL over strong baselines while maintaining competitive NE/TL.Evaluation uses continuous VLN-CE episodes and standard VLN metrics on a single GeForce RTX 4090.Source: §4.2 · p. 6, §4.1 · p. 6

4.2. Experiments Results

  • 73.7 Val-Unseen SPL and 69.0 Test-Unseen SPL accompany shorter trajectories, while SR/NE trade-offs remain modest against VISTA.Val-Unseen TL decreases from 13.26 to 10.73, and Test-Unseen TL decreases from 14.20 to 12.44.Source: §4.2 · p. 6
  • Against semantic-only or frontier-style methods, VISTAv2 raises SPL by approximately 5–10 points and SR by a similar margin.The result supports converting imagined futures into an egocentric value map rather than relying only on language–vision matching.Source: §4.2 · p. 6
  • Compared with goal-only imagination, VISTAv2 increases Val-Unseen SR by approximately 9–14 points and Test-Unseen SR by approximately 8 points.The action-conditioned short-horizon rollouts and score-level fusion incorporate near-term reachability and geometric risk during planning.Source: §4.2 · p. 6
  • 61.3 SR and 34.7 SWPL on RoboTHOR outperform VISTA by +18.2 SR and +5.9 SWPL.The method also surpasses CLIP-based, OWL, ESC, and VLTNet baselines by a large margin.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3. Ablations: From semantics to imagination to imagined value

  • Adding imagination to VLFM yields +25.3 Val-Unseen SR and +37.9 SPL, reaching 77.8/68.3 versus 52.5/30.4.Short-horizon rollouts provide reachability evidence without additional path supervision and complement language matching with a geometry-aware prior.Source: §4.3 · p. 7
  • Adding imagination-to-value fusion improves Val-Unseen by +1.6 SR, +5.4 SPL, and 19.1% lower TL, while Test-Unseen SPL rises +2.3 with 12.4% lower TL.The fusion preserves the planner and suppresses imagined frontiers and oscillations, though Test-Unseen SR decreases by 1.8 and NE rises by 0.28.Source: Table 3 · p. 7, §4.3 · p. 8
  • Performance peaks at uncertainty-gating threshold θ=0.6; lower thresholds approach no fusion, whereas higher thresholds admit noisy rollouts.The sweep reports SR/SPL against θ and fallback rate on R2R Val-Unseen.Source: §4.3 · p. 8, Figure 5 · p. 8

5. Conclusion

VISTAv2 uses a language-conditioned, action-aware world model to imagine short-horizon egocentric futures and express their guidance as an egocentric value map. It improves navigation performance with shorter paths while retaining a standard frontier-based planner.

  • VISTAv2 delivers consistent SR and SPL gains with shorter paths on R2R and RoboTHOR.Its value maps are interpretable and lightweight to deploy.Source: §5 · p. 8

6. Limitations and Future Work

VISTAv2’s short-horizon rollouts depend on monocular depth and pose estimates, with failures concentrated in visually difficult scenes. Future work targets uncertainty-aware longer-horizon planning, richer actions, memory, and sim-to-real transfer.

  • Failures occur in low-texture, reflective, or cluttered scenes where monocular-depth and pose uncertainty increases.Source: §6 · p. 8

Related papers

Loading 2512.00041v1…