Language-Conditioned World Modeling for Visual Navigation

Yifei Dong; Fengyi Wu; Yilong Dai; Lingdong Kong; Guangyu Chen; Xu Zhu; Qiyu Hu; Tianyu Wang; Johnalbert Garnica; Feng Liu; Siyu Huang; Qi Dai; Zhi-Qi Cheng

Source-linked AI summary

Language-Conditioned World Modeling for Visual Navigation

Yifei Dong, Fengyi Wu, Yilong Dai, Lingdong Kong, Guangyu Chen, Xu Zhu, Qiyu Hu, Tianyu Wang, Johnalbert Garnica, Feng Liu, Siyu Huang, Qi Dai, Zhi-Qi Cheng

arXiv:2603.26741v1Mar 23, 2026cs.CVcs.AIcs.RO

TL;DR

LCVN addresses language-conditioned visual navigation when an agent must follow an instruction from only an initial egocentric observation, without goal images or intermediate feedback. The paper introduces an open-loop benchmark and complementary world-modeling agents, finding that diffusion-based models produce more temporally coherent rollouts while autoregressive models generalize better to unseen environments.

Problem
Language-conditioned navigation requires agents to ground instructions in visual perception and control while generating trajectories without visual goals or intermediate environmental feedback.
Method
The paper introduces the LCVN task and dataset, then compares LCVN-WM with latent actor–critic LCVN-AC against autoregressive multimodal LCVN-Uni for joint language, prediction, and action modeling.
Results
The two framework families offer complementary advantages: diffusion-based world models produce more temporally coherent rollouts, whereas autoregressive agents generalize better to unseen environments.
Takeaways & Limitations
LCVN provides a shared benchmark for studying language grounding, imagination, and policy learning together in embodied navigation.

Abstract

from arXiv · show

We study language-conditioned visual navigation (LCVN), in which an embodied agent is asked to follow a natural language instruction based only on an initial egocentric observation. Without access to goal images, the agent must rely on language to shape its perception and continuous control, making the grounding problem particularly challenging. We formulate this problem as open-loop trajectory prediction conditioned on linguistic instructions and introduce the LCVN Dataset, a benchmark of 39,016 trajectories and 117,048 human-verified instructions that supports reproducible research across a range of environments and instruction styles. Using this dataset, we develop LCVN frameworks that link language grounding, future-state prediction, and action generation through two complementary model families. The first family combines LCVN-WM, a diffusion-based world model, with LCVN-AC, an actor-critic agent trained in the latent space of the world model. The second family, LCVN-Uni, adopts an autoregressive multimodal architecture that predicts both actions and future observations. Experiments show that these families offer different advantages: the former provides more temporally coherent rollouts, whereas the latter generalizes better to unseen environments. Taken together, these observations point to the value of jointly studying language grounding, imagination, and policy learning in a unified task setting, and LCVN provides a concrete basis for further investigation of language-conditioned world models. The code is available at https://github.com/F1y1113/LCVN.

1 Introduction

LCVN frames visual navigation as open-loop trajectory generation from one egocentric observation and language, addressing the challenge of coupling perception, prediction, and control without environmental feedback. It introduces a dataset and two complementary framework families whose experiments reveal distinct strengths in rollout coherence and generalization.

Existing visual-navigation systems often struggle to unify perception, prediction, and control, while many world models remain decoupled from policy learning.
LCVN requires an agent to generate an entire future trajectory from a single egocentric observation and natural-language instruction without intermediate environmental feedback.
The LCVN Dataset provides language-annotated trajectories spanning diverse environments and instruction styles for systematic evaluation.
The LCVN-WM and LCVN-AC family combines diffusion-based future-state imagination with latent actor–critic policy learning.
LCVN-Uni uses an autoregressive multimodal architecture to predict actions and observations in a shared representation.
Experiments show that LCVN agents outperform strong baselines, with diffusion models producing more temporally coherent rollouts and autoregressive agents generalizing better to unfamiliar environments.

2 Related Work

Related work spans goal-conditioned and free-exploration navigation, predictive world models, and policy-centric methods. LCVN uses open-loop language-conditioned trajectory generation to examine how perception, prediction, and control can be coupled without environmental feedback.

Visual-navigation research includes goal-conditioned and free-exploration settings, with goal-conditioned navigation requiring trajectories toward target images or locations.
LCVN instead asks agents to imagine a full future trajectory from a starting observation and instruction without environmental feedback.
World models have evolved from recurrent latent dynamics to Transformer and diffusion-based generators that support simulation and planning.
Policy-centric methods map observations to actions directly, whereas model-centric approaches predict environment dynamics and can be paired with latent policy learning.

3 LCVN: Language-Conditioned Visual Navigation

LCVN defines language-conditioned navigation as open-loop action generation from an initial RGB observation and instruction, then supports it with a multi-source, multi-style instruction dataset. The dataset combines standardized trajectories with expert-verified language annotations across diverse environments.

An agent receives an egocentric RGB observation and a word-sequence instruction, then generates sequential continuous navigation actions until a null action signals arrival.
The LCVN dataset sources trajectories from five robotics datasets covering complementary real-world navigation settings.
TartanDrive is reserved as an unseen split for zero-shot evaluation, while other sources support training and in-domain evaluation.
A coarse-to-fine annotation pipeline generates instructions automatically and then has experts verify and refine them against the corresponding videos.
The dataset contains 39,016 trajectories and 117,048 instructions in Concise, Intricate, and Landmark-based styles.

4 The LCVN Frameworks

The LCVN frameworks connect language-conditioned future-state prediction with policy learning through two complementary designs: a diffusion world model paired with a latent actor–critic agent, and a unified autoregressive model that predicts actions and observations together.

4.1 LCVN-WM: LCVN-WM predicts future observations from current latent context, navigation actions, instructions, time shifts, and diffusion conditioning.It uses Diffusion Forcing with independently noised context latents to strengthen temporal modeling.
4.1 LCVN-WM: LCVN-WM conditions latent-state prediction on language through instruction embeddings integrated with visual latents via cross-attention.Action, time-shift, and diffusion-step embeddings are also incorporated into the conditioning mechanism.
4.2 LCVN-AC Agent: LCVN-AC learns language-conditioned policies and value functions in LCVN-WM’s latent space using latent trajectory plans and intrinsic rollout rewards.Expert and learner plans are aligned with KL divergence, while rewards measure agreement between predicted and expert latent trajectories.
4.3 LCVN-Uni Agent: During LCVN-Uni inference, predicted observations from prior steps—not environmental observations—feed the next prediction cycle.The model therefore performs autoregressive open-loop rollout using its own predicted observation history.
4.3 LCVN-Uni Agent: LCVN-Uni combines navigation planning and world modeling in one autoregressive multimodal backbone that jointly predicts the next action and observation.Its unified objective balances discretized action prediction with future-observation reconstruction.

5 Experiments

Experiments compare LCVN agents and baselines across navigation, imagination, conditioning, instruction styles, representations, and model sizes. The results show complementary strengths between diffusion-based and unified autoregressive designs, while language, action, latent representations, and larger models improve performance.

Navigation performance: LCVN-WM + LCVN-AC performs better in known environments, whereas LCVN-Uni generalizes better to unseen environments.The former benefits from its LDiT design and intrinsic latent-space reward; the latter benefits from a unified architecture and shared representation.
Imagination performance: LCVN-WM (k = 4) achieves the highest one-step PSNR at 20.316, while LCVN-Uni (k = 2) achieves the best one-step DreamSim at 0.072.Both agents maintain stable open-loop rollouts and mitigate compounding errors over longer horizons.
Language guidance: Removing language consistently lowers navigation success rates and weakens imagination stability.The experiments characterize language as the goal specification that semantically aligns states and destinations.
Language, Action and Time Conditioning: Action conditioning contributes more than language conditioning, which contributes more than time conditioning, and removing action causes the largest performance drop.The authors attribute this ordering to explicit control signals being essential for grounding latent dynamics, while language alone cannot resolve fine-grained transitions.
Instruction Style and Context Size: Landmark-grounded instructions perform best, concise instructions slightly outperform intricate ones, and larger contexts generally improve performance.For LCVN-Uni, increasing context to k = 4 reduces per-frame resolution under the fixed token window, so k = 2 is selected; LCVN-WM + LCVN-AC uses k = 4.
Latent Space and Model Size: Latent-space training outperforms pixel-space training, while larger LCVN-WM models improve imagination quality and the XL variant performs best overall.Latent encoding also integrates with the latent-space LCVN-AC policy and value functions.

6 Conclusion

The paper introduces language-conditioned visual navigation as an open-loop trajectory-generation task and provides a dataset and frameworks that couple perception, imagination, and planning. Its experiments show strong navigation and imagination performance, with LCVN-Uni offering strong generalization in unseen environments.

The LCVN frameworks couple perception, imagination, and planning through diffusion-based LCVN-WM with latent actor–critic LCVN-AC and unified autoregressive LCVN-Uni.
LCVN agents outperform competitive baselines in navigation and imagination, while LCVN-Uni shows strong generalization in unseen environments.

A LCVN Dataset Details

The LCVN Dataset pairs each trajectory with three instruction styles that vary in specificity and grounding. These styles support comparisons of navigation behavior across directional, contextual, and landmark-based language.

Each LCVN trajectory is annotated with concise, intricate, and landmark-grounded instruction styles.Concise instructions emphasize essential directional cues; intricate instructions add scene context; landmark-grounded instructions anchor navigation to salient environmental features.
The dataset examples use purple text for directional guidance and blue text for landmark references.
The three styles enable evaluation across different levels of specificity and emphasis.The examples demonstrate minimal guidance, contextual enrichment, and explicit ties to recognizable environmental features.

B Additional Implementation Details

The supplementary implementation details describe how LCVN-WM and LCVN-AC are trained and coupled during inference. They also explain the multimodal inputs used to connect navigation planning and visualization in LCVN-Uni.

Training Phase 1 (LCVN-WM): LCVN-WM uses a diffusion transformer with CLIP instruction cross-attention, action and time-shift conditioning, and Diffusion Forcing across latent context states.These choices support prediction of future latent states conditioned on navigation actions and instructions.
Training Phase 2 (LCVN-AC): LCVN-AC learns in LCVN-WM’s latent space using KL-aligned latent plans, intrinsic rollout rewards, and plan-consistency and language-alignment losses.The training objective is designed to produce semantically grounded navigation behavior.
Inference Stage: During inference, LCVN-WM imagines future latent states while LCVN-AC generates actions from those states and instruction embeddings until stopping.
Prompting and Coupling: The supplementary prompt example shows navigation instructions and egocentric observations jointly guiding the planner and world model.

B.3 Training Loss Details for LCVN-Uni

LCVN-Uni jointly trains navigation planning and visual imagination, using a combined objective that predicts actions and future observations from language-conditioned context. Its training procedure combines discretized action prediction with a visual imagination loss.

LCVN-Uni predicts the next action and next first-person observation from a navigation instruction and current visual context.The task jointly evaluates navigation planning and visualization after executing the predicted action.
The joint objective is Ljoint = Lplan + λLimagine, balancing action-planning loss against future-observation fidelity.λ controls the relative importance of planning and imagination.
Actions are represented by x_t, y_t, and ϕ_t, then discretized into tokens from separate bins for each dimension.The action loss uses predicted token probabilities over these discretized action components.
The imagination loss compares predicted visual tokens with ground-truth embeddings using distances to a codebook and predicted token assignments.This loss is applied to visual tokens in the predicted next observation.
Training samples trajectory observations, actions, instructions, and time-shift information before recursively predicting future latent states with the world model.The procedure accumulates diffusion-based world-model loss over training steps.

C.1 Evaluation Metric Details

The evaluation measures navigation accuracy, pose consistency, visual fidelity, perceptual similarity, and long-horizon rollout stability. It combines trajectory metrics with one-step and multi-step imagination metrics.

Navigation Quality: Success Rate counts trajectories whose final distance to the target is smaller than the agent’s average step size.The metric evaluates whether the terminal predicted position meets the task’s success criterion.
Navigation Quality: Absolute Trajectory Error measures global pose accuracy, while Relative Pose Error measures local consistency across consecutive motions.ATE uses aligned predicted and reference poses; RPE compares relative motion deviations.
Imagination Quality: SSIM, PSNR, LPIPS, and DreamSim evaluate structural, pixel-level, perceptual, and semantic aspects of predicted visual rollouts.LPIPS and DreamSim use deep representations intended to better reflect perceptual or semantic similarity.
Imagination Quality: Rollout metrics SSIM@n, PSNR@n, LPIPS@n, and DreamSim@n extend imagination evaluation to long-horizon stability.Open-loop rollouts recursively feed predicted frames back into the model; experiments set n as 8.

C.3 More Experiments

Additional experiments compare inference efficiency, joint versus interleaved prediction, and generalization to unseen environments. LCVN-Uni is more efficient and generalizes better, while LCVN-WM provides stronger imagination performance.

Inference Speed: LCVN-WM achieves substantially lower inference time than NWM and LCVN-Uni while maintaining competitive performance.The reported per-step latency includes action prediction and imagination, and further acceleration is discussed through fewer diffusion steps or quantization.
Interleave vs. Predict Both: Predict Both achieves comparable navigation and imagination accuracy to Interleave while providing 1.3× higher computational efficiency.Predict Both jointly predicts actions and observations in one forward pass.
Generalization in Unseen Environments: LCVN-Uni consistently outperforms LCVN-WM + LCVN-AC in Success Rate on unseen environments while maintaining competitive ATE and RPE.The comparison is reported for navigation performance under distributional shift.
Generalization in Unseen Environments: LCVN-WM achieves superior imagination performance, whereas LCVN-Uni shows stronger navigation generalization in unseen environments.The results distinguish imagination quality from transfer performance across novel scenarios.