PaperlayerView exact revision on arXiv

Source-linked AI summary

FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation

Jing Zuo, Lingzhou Mu, Fan Jiang, Chengcheng Ma, Mu Xu, Yonggang Qi

arXiv:2601.13976v2cs.CVcs.RO

TL;DR

VLN needs reasoning that jointly handles language, visual-spatial context, and long action sequences, while existing textual and multimodal CoTs have grounding, supervision, generalization, and token-efficiency limitations. FantasyVLN unifies textual, visual, and multimodal CoT training with VAR-based latent compression and direct instruction-to-action inference, improving LH-VLN navigation accuracy and efficiency while substantially reducing inference latency.

  • Problem

    VLN still lacks reasoning mechanisms that jointly address the semantic–spatial gap while remaining interpretable and sample-efficient, and existing CoT methods face grounding, supervision, generalization, or token-overhead limitations.

  • Method

    FantasyVLN jointly trains textual, visual, and multimodal CoT modes, compresses imagined visual observations with a pretrained VAR model, aligns CoT and direct predictions, and performs direct instruction-to-action inference.

  • Results

    FantasyVLN achieves superior LH-VLN navigation performance across SR, ISR, CSR, and CGT, with values of 2.44, 11.01, 9.64, and 8.99, respectively, while implicit reasoning reaches approximately one action per second versus 0.19 for explicit CoT reasoning.

  • Takeaways & Limitations

    The framework provides reasoning-aware navigation through direct action prediction, combining multimodal CoT benefits with real-time inference efficiency.

  • Takeaways & Limitations

    Multimodal CoT remains constrained by imagined-observation token inflation, with 5–7-action reasoning steps expanding beyond 3k–5k tokens and making real-time navigation infeasible.

Abstract

from arXiv · show

Achieving human-level performance in Vision-and-Language Navigation (VLN) requires an embodied agent to jointly understand multimodal instructions and visual-spatial context while reasoning over long action sequences. Recent works, such as NavCoT and NavGPT-2, demonstrate the potential of Chain-of-Thought (CoT) reasoning for improving interpretability and long-horizon planning. Moreover, multimodal extensions like OctoNav-R1 and CoT-VLA further validate CoT as a promising pathway toward human-like navigation reasoning. However, existing approaches face critical drawbacks: purely textual CoTs lack spatial grounding and easily overfit to sparse annotated reasoning steps, while multimodal CoTs incur severe token inflation by generating imagined visual observations, making real-time navigation impractical. In this work, we propose FantasyVLN, a unified implicit reasoning framework that preserves the benefits of CoT reasoning without explicit token overhead. Specifically, imagined visual tokens are encoded into a compact latent space using a pretrained Visual AutoRegressor (VAR) during CoT reasoning training, and the model jointly learns from textual, visual, and multimodal CoT modes under a unified multi-CoT strategy. At inference, our model performs direct instruction-to-action mapping while still enjoying reasoning-aware representations. Extensive experiments on LH-VLN show that our approach achieves reasoning-aware yet real-time navigation, improving success rates and efficiency while reducing inference latency by an order of magnitude compared to explicit CoT methods.

1 Introduction

VLN requires joint semantic and spatial understanding for long-horizon action planning, but existing textual and multimodal CoT approaches face grounding, supervision, token-efficiency, and generalization challenges. FantasyVLN addresses these issues with unified implicit multimodal reasoning and direct inference, achieving improved navigation performance and substantially lower latency.

  • VLN requires agents to combine language semantics, visual geometry, and long-horizon reasoning to plan action sequences.
  • Textual CoT methods decompose instructions or generate subgoals but limit joint semantic-spatial modeling, face difficult supervision, and may generalize poorly to unseen environments.
  • Multimodal CoT improves semantic-spatial coupling but expands 5–7-action reasoning steps to over 3k–5k tokens versus usually <500 for textual CoT.The resulting sequence-length increase raises training and inference latency and makes real-time navigation infeasible even on high-end GPUs.
  • FantasyVLN compresses imagined visual observation tokens into a compact VAR latent space during training and performs direct instruction-to-action mapping at inference.
  • Unified multi-CoT training jointly learns textual-only, visual-only, and textual–visual modes, aligns their action predictions with direct prediction, and supports mode switching through gating.
  • Experiments on LH-VLN report improved navigation success and efficiency while reducing inference latency by an order of magnitude compared with explicit CoT approaches.

2 Related Works

VLN research progressed from modular imitation- or reinforcement-learning systems toward CoT-based long-horizon reasoning. Existing approaches often restrict reasoning to one modality, whereas FantasyVLN unifies textual, visual, and multimodal CoT paradigms.

  • Early VLN systems separated perception, instruction understanding, and action planning, using imitation or reinforcement learning with auxiliary tasks.
  • These early methods made progress on short-term tasks but struggled with reasoning and planning for long-horizon, multi-stage navigation.
  • Recent VLN methods use CoT reasoning, but existing CoT-based approaches generally confine reasoning to a single modality.
  • CoT variants include textual reasoning, visual future-frame generation, and multimodal paired textual-visual reasoning steps.
  • FantasyVLN is presented as the first unified CoT framework integrating textual, visual, and multimodal reasoning paradigms.

3 Methods

FantasyVLN unifies textual, compressed visual, and multimodal reasoning within one agent while using latent visual tokens and cross-mode alignment for efficient inference. The framework trains multiple reasoning modes but performs navigation through a direct action-prediction mode.

  • 3.2 Problem Setup: FantasyVLN formulates VLN as an embodied agent navigating continuous 3D environments from language instructions and historical multi-view observations.The agent predicts actions until stopping or reaching the maximum step.
  • 3.3 Compact Visual Chain-of-Thought: 30 visual tokens can represent a 256 × 256 image through VAR latent representations with precise reconstruction and higher compression than comparable compressors.Table 1 compares compression ratio and reconstruction error (MSE) across visual compressors.
  • 3.4 Unified Multimodal Chain-of-Thought: UM-CoT integrates textual, compressed visual, and multimodal reasoning modes within a single agent using binary gates that activate textual and visual reasoning.The multimodal mode generates paired textual–visual reasoning steps, while the gating mechanism supports CoT, T-CoT, CompV-CoT, and MM-CoT modes.
  • 3.4 Unified Multimodal Chain-of-Thought: Textual CoT generates intermediate semantic planning steps, whereas MM-CoT combines those steps with imagined visual observations to guide action prediction.Textual steps can infer subgoals, assess progress from observations, and identify actionable cues.
  • 3.3 Compact Visual Chain-of-Thought: CompV-CoT predicts compressed future-observation tokens in VAR latent space before conditioning action prediction on the imagined latent trajectory.The VAR model is frozen during training, while the VLM learns latent future-observation prediction and subsequent action inference.
  • 3.5 Cross-Mode Alignment Constraint: Cross-mode alignment uses non-CoT action predictions as supervisory signals for CoT variants, embedding diverse reasoning behaviors into a shared latent policy.Alternating optimization aligns shared-parameter reasoning modes to identical supervisory signals; non-CoT reasoning is used during inference to meet real-time demands.

4 Experiments

Experiments on the unseen-task, unseen-scene LH-VLN test setting evaluate navigation accuracy, inference efficiency, reasoning-mode combinations, VAR-scale choices, alignment, training dynamics, and explicit versus implicit reasoning.

  • Navigation Accuracy: FantasyVLN achieves SR, ISR, CSR, and CGT of 2.44, 11.01, 9.64, and 8.99, respectively, surpassing all baselines on LH-VLN.The benchmark evaluates multi-stage navigation accuracy using these four metrics.
  • Inference Efficiency: Implicit reasoning models exhibit comparable efficiency and outperform explicit CoT-VLA, which produces only 0.19 actions per second.Implicit reasoning decodes a single action token, whereas explicit reasoning generates thousands of CoT tokens.
  • Contribution of Each Reasoning Mode: Combining any CoT mode with non-CoT reasoning consistently improves navigation performance, while integrating all four modes further enhances performance.The evaluated modes are non-CoT, T-CoT, V-CoT, and MM-CoT.
  • VAR Scale Selection: Scale 4 achieves the best ISR among VAR scales from 1 to 10, while smaller scales lack visual information and larger scales introduce redundancy.Reconstruction experiments using ground-truth latents through a selected scale and predicted remaining latents support this choice.
  • Effect of Cross-Mode Alignment Constraint: Cross-mode alignment raises SR from 0 to 2.44, ISR from 2.39 to 11.01, CSR from 1.19 to 9.64, and CGT from 1.28 to 8.99.The comparison evaluates FantasyVLN with and without the alignment constraint.
  • Training Efficiency: FantasyVLN converges within a few thousand iterations, whereas WorldVLA requires over 10k iterations to reach moderate token prediction accuracy.The paper attributes this difference to compact VAR-latent prediction replacing dense pixel reconstruction.
  • Explicit vs. Implicit Reasoning: In multimodal CoT, implicit inference achieves 2.44 SR and 11.01 ISR, surpassing explicit inference by a large margin.The comparison evaluates explicit and implicit inference under T-CoT, V-CoT, and MM-CoT.

5 Conclusion

FantasyVLN uses unified implicit reasoning to retain CoT benefits without explicit inference-time token explosion. On LH-VLN, it improves navigation accuracy and efficiency while reducing inference latency by an order of magnitude.

  • FantasyVLN compresses imagined visual observations into a compact latent space using a pretrained VAR model.It jointly trains textual, visual, and multimodal CoT modes under a unified multi-CoT strategy.
  • The framework performs direct instruction-to-action mapping at inference while retaining reasoning-aware behavior.
  • Experiments on LH-VLN show substantially improved navigation accuracy and efficiency with inference latency reduced by an order of magnitude versus explicit CoT baselines.
  • Implicit multimodal reasoning provides a practical pathway toward real-time embodied navigation.The framework uses compact latent reasoning signals to address the semantic–spatial gap in complex environments.

A Data Preparation

The data-preparation pipeline represents LH-VLN trajectories as instruction-conditioned sequences of historical observations and future action chunks. It uses non-overlapping trajectory slices and prompts tailored to single-stage and multi-stage navigation.

  • A.1 Preprocessing: Each LH-VLN expert trajectory contains visual–action pairs and a natural-language task instruction.The agent acts online using current and historical visual observations.
  • A.1 Preprocessing: Training samples are created by partitioning each navigation trajectory into non-overlapping slices containing the instruction, visual observations, and future actions.
  • A.1 Preprocessing: Future action targets are grouped into chunks of five actions during slice construction.The implementation sets k = 5.
  • A.1 Preprocessing: Single-stage prompts ask the agent to choose its next five actions from directional commands or stop after completing the mission.
  • A.1 Preprocessing: Multi-stage prompts require stopping after each completed subtask and tracking how many subtasks have been finished.Historical information is framed as progress up to the current subgoal.

A.3 T-CoT Data Annotation

The T-CoT annotation process generates intermediate reasoning labels for navigation slices using instructions, historical and current images, and ground-truth action sequences. Annotations organize reasoning into semantic planning, visual description, action planning, and visual imagination.

  • A.3 T-CoT Data Annotation: Qwen-VL-Max annotates all 18,554 LH-VLN training navigation slices with T-CoT reasoning.
  • A.3 T-CoT Data Annotation: Each annotation input includes the user instruction, up to 20 historical front-view images, three current views, and the ground-truth action sequence.
  • A.3 T-CoT Data Annotation: Semantic planning decomposes each mission into precise subtasks with clear spatial goals and stepwise navigation milestones.
  • A.3 T-CoT Data Annotation: Visual description records what historical and current images reveal about completed and upcoming subtasks.
  • A.3 T-CoT Data Annotation: The output format combines Semantic Plan, Visual Description, Action Planning, and Visual Imagination within a single reasoning block.

A.4 Data Augmentation

FantasyVLN augments training examples by perturbing historical image sequences while preserving the final three observation images. It uses stochastic subsampling and trimming to reduce redundancy and vary visual histories.

  • A.4 Data Augmentation: Training augmentation changes only historical images while keeping the final three observation frames unchanged.Each sample can receive up to two additional augmented variants.
  • A.4 Data Augmentation: Uniform subsampling replaces sufficiently long histories with a stride-2 subsequence at probability 0.5.The operation reduces temporal redundancy and encourages reliance on coarser state transitions.
  • A.4 Data Augmentation: Stochastic history trimming removes either the first two frames or a randomly selected pair of consecutive frames under stated length conditions.At least one trimming operation must be triggered for the augmented sample to be retained.

B Detail Formulations.

FantasyVLN unifies four reasoning modes within one framework, using direct action prediction for non-CoT reasoning and latent visual representations for efficient visual reasoning.

  • Unified reasoning modes: FantasyVLN unifies Non-CoT, T-CoT, V-CoT, and MM-CoT reasoning modes within a single framework.
  • Non-CoT reasoning: The non-CoT mode directly predicts actions from the instruction and visual observations using a pretrained VLM navigation agent.
  • T-CoT reasoning: Textual CoT decomposes instructions into subgoals, infers the current goal from observations, and identifies decision evidence from the current visual observation.
  • CompV-CoT reasoning: CompV-CoT predicts low-scale VAR latent representations instead of pixel-level images, which are then reconstructed into pixel observations for action prediction.
  • MM-CoT reasoning: MM-CoT generates paired textual-visual reasoning steps before predicting future actions, while the VAR remains frozen during training.
  • Unified training: The unified multi-CoT objective combines non-CoT, textual, visual, and multimodal reasoning modes under one framework.

C Implementation Details

Implementation uses Qwen2.5-VL with LoRA-based parameter-efficient tuning and online simulator evaluation for multi-stage navigation.

  • Training details: Qwen2.5-VL is used as the base model with LoRA-based parameter-efficient tuning applied to language layers and vision–language projection modules.
  • Training details: Training runs on 64 H20 GPUs with AdamW, a 1 × 10^-4 learning rate, 0.1 weight decay, cosine scheduling, and bfloat16 precision.
  • Evaluation: Online evaluation places the agent in a simulator where it repeatedly receives visual observations, predicts actions, and updates its state after execution.
  • Special tokens: The implementation adds action, VAR latent, system, and gating tokens to control navigation prediction and reasoning modes.
Loading 2601.13976v2…