PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Models

Hongjin Chen, Shangyun Jiang, Tonghua Su, Chen Gao, Xinlei Chen, Yong Li, Zhibo Chen

arXiv:2604.07957v1cs.AIcs.CVcs.RO

TL;DR

Reliable trajectory prediction from a single egocentric observation remains challenging: VLMs can be unstable, and generated futures do not alone provide grounded supervision. WorldMAP turns generated futures into semantic-spatial memory and planning-derived labels for a lightweight student. On Target-Bench, it leads the compared methods on ADE and FDE, with the paper arguing that world models may be most useful as sources of structured supervision.

  • Problem

    VLMs remain unstable at grounded trajectory prediction from one egocentric observation, while generated views do not themselves provide structured supervision for navigation learning.

    Source: §I · pp. 1–2

  • Method

    WorldMAP grounds targets and obstacles across generated future views, plans trajectory pseudo-labels, and distills them into a lightweight student that predicts from vision-language inputs.

    Source: §I · p. 2, C. Student Trajectory Learning · p. 5

  • Results

    WorldMAP reduces ADE from 51.27 to 42.06 (18.0%) and FDE from 67.19 to 38.87 (42.1%) versus Gemini-3-Pro on Target-Bench.

    Source: B. Main Results · p. 6

  • Takeaways & Limitations

    The results suggest world models may be more valuable for synthesizing grounded supervision than for supplying action-ready imagined evidence in embodied navigation.

    Source: §V · p. 8, §VI · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) and generative world models are opening new opportunities for embodied navigation. VLMs are increasingly used as direct planners or trajectory predictors, while world models support look-ahead reasoning by imagining future views. Yet predicting a reliable trajectory from a single egocentric observation remains challenging. Current VLMs often generate unstable trajectories, and world models, though able to synthesize plausible futures, do not directly provide the grounded signals needed for navigation learning. This raises a central question: how can generated futures be turned into supervision for grounded trajectory prediction? We present WorldMAP, a teacher--student framework that converts world-model-generated futures into persistent semantic-spatial structure and planning-derived supervision. Its world-model-driven teacher builds semantic-spatial memory from generated videos, grounds task-relevant targets and obstacles, and produces trajectory pseudo-labels through explicit planning. A lightweight student with a multi-hypothesis trajectory head is then trained to predict navigation trajectories directly from vision-language inputs. On Target-Bench, WorldMAP achieves the best ADE and FDE among compared methods, reducing ADE by 18.0% and FDE by 42.1% relative to the best competing baseline, while lifting a small open-source VLM to DTW performance competitive with proprietary models. More broadly, the results suggest that, in embodied navigation, the value of world models may lie less in supplying action-ready imagined evidence than in synthesizing structured supervision for navigation learning.

I. INTRODUCTION

Single-observation navigation requires grounded, traversable trajectories, yet VLM predictions remain unstable and generated futures alone do not provide structured supervision. WorldMAP converts those futures into planning-derived teacher labels for a lightweight student, and performs strongly on Target-Bench.

I. INTRODUCTION

  • Reliable navigation from egocentric observations and language instructions requires grounding the target, inferring traversable space, and predicting a physically plausible path in continuous, unseen environments.Unlike discrete action selection, the task involves producing a path toward a semantic target,.Source: §I · p. 1
  • VLMs remain unstable as direct trajectory predictors, while plausible world-model views do not themselves supply structured supervision for grounded trajectory learning.The paper identifies conversion of generated futures into persistent semantic and geometric structure as the missing mechanism.Source: §I · p. 2

A New Teacher-Student Distillation Framework for Navigation Trajectory Prediction

  • WorldMAP uses a world-model-driven teacher to ground targets and obstacles across generated views, plan a trajectory, and distill pseudo-labels into a student predicting from vision-language inputs.The student does not rerun the teacher pipeline at test time.Source: §I · pp. 2–3

Strong Performance on Target-Bench

  • WorldMAP achieves the best ADE and FDE among compared methods on Target-Bench and makes a small open-source VLM competitive with stronger proprietary models.Target-Bench evaluates navigation toward semantic targets in unstructured real-world environments [28].Source: §I · p. 3

II. RELATED WORK

Prior navigation work uses structured spatial representations, VLM decision-making, and imagined futures, but single-observation trajectory prediction remains challenging. WorldMAP separates semantic grounding from geometric planning to turn generated futures into structured supervision.

A. Structured Spatial Representations for Navigation

  • Explicit localization, mapping, and planning over geometric representations provide interpretable intermediate state, reliable geometric control, and traversability-aware planning, but often depend on persistent map construction, repeated scene interaction, or online state estimation.Source: A. Structured Spatial Representations… · p. 3

B. World Models and Vision-Language Reasoning for Navigation

  • World-model and VLM navigation methods explore imagined views and language-guided decisions –, while benchmark evidence still finds errors in single-observation grounding and goal localization [23], [28].Generated futures have often served as transient test-time evidence; WorldMAP instead uses them to synthesize structured supervision, [26],.Source: B. World Models and Vision-Language… · p. 3

A. Overview

  • WorldMAP separates semantic grounding from geometric planning, expressing scene geometry in a shared navigation plane and BEV system before converting generated futures into supervision.Its teacher consolidates generated video into semantic-spatial memory, grounds targets and obstacles, and supplies supervision for direct student prediction.Source: A. Overview · p. 3

B. World-Model-Driven Trajectory Teacher

The teacher converts generated future videos into grounded trajectory supervision through world construction, task-aware perception, and explicit geometric planning. It builds shared semantic and spatial representations, then grounds targets and obstacles for downstream planning.

B. World-Model-Driven Trajectory Teacher

1) World Construction:

  • The teacher combines generated future videos, depth-based scene reconstruction, and semantic-spatial memory, using a shared ground plane as the reference for projection and trajectory generation.Multiple video streams provide complementary nearby viewpoints and scene hypotheses for grounding and map construction.Source: 1) World Construction · p. 4

2) Task-Aware Grounding:

  • Task-aware grounding retrieves instruction-relevant views, segments a target region, and identifies path-relevant obstacle masks for projection into the BEV coordinate system.Target grounding follows instruction semantics, while obstacle grounding focuses on objects that interfere with traversal; both use the same geometric frame.Source: 2) Task-Aware Grounding · p. 4

3) Geometric Planning and Supervision:

Grounded multi-view evidence is accumulated in a shared top-down BEV representation, converted into a conservative cost map, and planned into waypoint supervision. Explicit planning converts grounded scene structure into a teacher trajectory rather than serving as the final policy.

4) Cost BEV Construction:

  • The method projects grounded multi-view evidence onto a shared navigation plane and accumulates it in a top-down BEV grid.Depth points are retained within a ground-level height band, and repeated observations are fused with confidence-aware averaging.Source: 4) Cost BEV Construction · pp. 4–5
  • The RGB BEV aggregates coverage from multiple future views and overlays grounded target and obstacle regions in the same coordinate system.The representation preserves metric structure.Source: 4) Cost BEV Construction · p. 5

5) Cost-Aware Planning and Supervision:

  • Obstacle cells are blocked, unobserved regions are conservatively non-traversable, and safety margins raise costs near obstacles before FMM produces a smoothed, resampled path.The waypoints are mapped back onto the navigation plane as teacher trajectory supervision, not as the final navigation policy.Source: 5) Cost-Aware Planning and Supervision · p. 5

C. Student Trajectory Learning

The student distills teacher-generated trajectory supervision into direct vision-language prediction. It predicts multiple trajectory hypotheses and trains them with waypoint and direction-consistency losses.

C. Student Trajectory Learning

  • Teacher-generated trajectories provide supervision that already encodes target grounding, obstacle awareness, and explicit geometric planning.The student distills this structure rather than rediscovering it from scratch.Source: C. Student Trajectory Learning · p. 5, Figure 4 · p. 5
  • The lightweight student predicts K trajectory hypotheses with confidence scores from multimodal features, then outputs a trajectory directly from an observation and instruction.At inference, it does not rerun the teacher pipeline.Source: C. Student Trajectory Learning · p. 5
  • Each hypothesis combines waypoint regression with segment-direction consistency, and training uses the best-matching hypothesis.The direction term helps preserve local trajectory shape beyond pointwise regression; λd = 0.5 in the implementation.Source: C. Student Trajectory Learning · p. 5

IV. EXPERIMENTS

Experiments use Target-Bench examples pairing first-person observations and navigation instructions with SLAM-estimated robot trajectories. Evaluation uses projected 2D trajectories and reports ADE, FDE, and normalized DTW.

IV. EXPERIMENTS

  • Target-Bench samples pair a first-person RGB observation and a natural-language instruction with a SLAM-estimated 3D quadruped trajectory.The benchmark covers path planning toward semantic targets in unstructured indoor and outdoor environments.Source: A. Experimental Setup · p. 5
  • The SLAM-estimated trajectory is projected onto the real first-frame image to provide a 2D pixel-space ground-truth trajectory.Final evaluation uses trajectory error in 2D pixel space.Source: A. Experimental Setup · p. 5
  • The reported trajectory metrics are Average Displacement Error (ADE), Final Displacement Error (FDE), and normalized Dynamic Time Warping (DTW).Each metric is defined on the projected trajectories.Source: A. Experimental Setup · pp. 5–6

L DTW(ˆP, P), (5)

The section evaluates WorldMAP as a trained student against direct VLM and world-model-augmented systems, reporting the best ADE and FDE and examining trajectory quality and limits on attributing gains solely to supervision.

L DTW(ˆP, P), (5)

  • All three metrics use projected 2D trajectories in the real first-frame image, with lower values indicating better performance.The reported main results correspond to the final WorldMAP student.Source: L DTW(ˆP, P), (5) · p. 6

B. Main Results

  • The comparison tests direct trajectory prediction against training signals from world models, across proprietary VLMs, open-source VLMs, MindJourney, and WorldMAP.WorldMAP is evaluated as a trained student; the other systems are evaluated as direct or test-time reasoning systems under the same protocol.Source: B. Main Results · p. 6

C. Analysis

  • WorldMAP more consistently follows traversable floor geometry, avoids implausible shortcuts, and stops closer to the intended target than the compared baselines.The qualitative comparison is shown in Figure 5.Source: B. Main Results · p. 6

B. Main Results

  • WorldMAP reduces ADE from 51.27 to 42.06 (18.0%) and FDE from 67.19 to 38.87 (42.1%) versus Gemini-3-Pro, while normalized DTW is 31.95 vs. 31.63.The table reports projected 2D image-space trajectory metrics on Target-Bench; lower is better for all metrics.Source: Table I · p. 6, B. Main Results · p. 6

C. Analysis

  • The comparison does not isolate supervision as the only changed factor, because the teacher also contributes retrieval, grounding, and explicit planning.MindJourney also underperforms the direct o3 baseline on all three metrics, so additional imagined views are not automatically beneficial in this benchmark.Source: C. Analysis · p. 6

D. Ablation Studies

The ablations show that teacher-generated supervision works best when combined with ground-truth trajectories and quality-filtered. The discussion attributes the broader gains to consolidating generated futures into grounded supervision, while bounding the evidence to the tested setting.

D. Ablation Studies

  • WorldMAP more consistently follows traversable floor geometry and stops closer to the intended target than competing methods in the qualitative comparison.Baselines more often drift, overshoot, or pass through non-traversable regions.Source: Figure 5 · p. 7
  • Combining ground-truth trajectories with usable world-model pseudo-labels gives the strongest results; usable-only filtering yields the best ADE and FDE, while adding borderline labels slightly improves DTW.Ground-truth-only and pseudo-label-only training both perform poorly.Source: D. Ablation Studies · p. 8, Table III · p. 8

V. DISCUSSION

  • The evidence supports a combined teacher–student design: memory retrieval, semantic grounding, and explicit planning complement supervision rather than supervision alone explaining the gains.The contribution lies in converting generated futures into grounded supervision, not in any single module.Source: §V · p. 8
  • World models may be more useful as supervision sources than as action-ready evidence because current generations do not reliably provide long-horizon, cross-view consistency.The discussion notes that directly consuming imagined views can hurt precise trajectory prediction when their geometry is inconsistent.Source: §V · p. 8
  • The reported gains establish effectiveness on Target-Bench, but not transfer to highly dynamic scenes, multi-level environments, or long-horizon exploration.The pipeline also remains constrained by future-generation and geometric-reconstruction quality.Source: §V · p. 9

VI. CONCLUSION

WorldMAP turns world-model-generated futures into persistent semantic-spatial structure and planning-derived supervision for a lightweight student. On Target-Bench it achieves the best ADE and FDE among compared methods, and the conclusion favors world models as supervision engines over action-ready imagined evidence.

VI. CONCLUSION

  • WorldMAP converts generated futures into persistent semantic-spatial structure and planning-derived trajectory supervision, which a lightweight student uses for direct prediction from vision-language inputs.The teacher extracts grounded trajectory pseudo-labels from generated videos.Source: §VI · p. 9
  • WorldMAP achieves the best ADE and FDE among compared methods on Target-Bench and brings a small open-source VLM to DTW performance competitive with proprietary models.Source: §VI · p. 9
  • The conclusion suggests world models may be most valuable for synthesizing aligned navigation supervision, rather than supplying action-ready imagined evidence.Source: §VI · p. 9

Supplementary Material

  • The supplementary material provides implementation details and qualitative evidence, covering trajectory projection, homography alignment, teacher supervision, and student behavior.Source: Supplementary Material · p. 11

APPENDIX A DATA PROCESSING DETAILS

The appendix describes how benchmark trajectories are projected and aligned across real and world-model frames, then details how the teacher grounds targets and obstacles for geometric planning. The resulting smoothed paths serve as trajectory supervision.

APPENDIX A DATA PROCESSING DETAILS

  • Target-Bench’s SLAM-estimated 3D trajectories are projected onto real first frames as 2D pixel-space references, then scene-specific homographies transfer them to world-model frames.The mapping uses ORB feature matching and RANSAC, with feature, inlier, and stability checks.Source: Appendix A · p. 11, Figure 7 · p. 11, Figure 8 · p. 11
  • The accepted homography is used during preprocessing to transfer projected trajectories between real and world-model frames; downstream planning remains in the world-model frame.Source: Appendix A · p. 11

A. Teacher Pipeline

  • The teacher retrieves semantically relevant memory views, grounds targets and obstacles across views, and projects the fused masks into a shared BEV map.Grounding uses a VLM-based mask judge for targets and filters heavily over-segmented obstacle masks before 3D fusion.Source: A. Teacher Pipeline · pp. 11–12, Figure 10 · p. 12
  • FMM planning treats obstacles and background as blocked, adds safety margins and traversal costs near uncertain regions, then smooths and resamples the path as teacher supervision.The goal combines the nearest accessible target boundary with the target centroid.Source: A. Teacher Pipeline · p. 12

B. Student Distillation

The student is trained in two stages to encode vision-language inputs and decode teacher-supervised waypoint hypotheses, then runs without the teacher at inference. Qualitative examples show more instruction-specific stopping positions and approach routes than the compared baselines.

B. Student Distillation

  • Stage A adapts a Qwen3-VL-8B backbone with LoRA to encode first-person RGB observations and instructions into geometry-aware features; Stage B trains a lightweight trajectory decoder.Source: B. Student Distillation · p. 12
  • The decoder combines the normalized start point with pooled text, vision, and cross-modal features to predict multiple future waypoint hypotheses in image space using teacher-generated supervision.Source: B. Student Distillation · p. 12
  • At inference, the student runs without the teacher and selects one final trajectory from its hypotheses.The reported setup uses a Qwen3-VL-8B backbone and a lightweight decoder.Source: B. Student Distillation · p. 12

APPENDIX C ADDITIONAL QUALITATIVE RESULTS

  • Across representative indoor and outdoor examples, the predictor more often stops at the instructed curb, door-facing region, or stopping pose and approaches from a plausible direction.Baselines show more variance, including wrong-target drift, overshooting, offset stops, and implausible headings.Source: Appendix C · p. 12, Figure 11 · p. 13
Loading 2604.07957v1…