PaperlayerView exact revision on arXiv

Source-linked AI summary

See What I Need, Dream What I Want: A Navigation Foundation Model on a Self-Constructed Representation Space

Jiawei Chen, Yinxi Wang, Zihan Yang, Meijun Gao, Yaowei Liu, Shuming Zhang, Zecheng Li, Xiangju Mi, Yuxuan Liu, Runzhong Huang, Tianlin Li, Xun Zhang, Zhengyang Wang, Xu Miao, Tong Yang

arXiv:2610.31762v1cs.ROcs.CV

TL;DR

Navigation policies built on vision foundation models largely leave representation design to pretraining objectives. RepresentNav constructs a policy-objective-driven visual space and predicts future representations in it before acting. It achieves state-of-the-art results across several navigation benchmarks and shows benefits from both its representation and its dream.

  • Problem

    Navigation policies built on vision foundation models largely leave representation design to encoder pretraining objectives, although navigation needs representations shaped for the decisions at hand.

    Source: §1 · p. 2

  • Method

    RepresentNav constructs a purposive representation from the internal layers of vision foundation models and jointly trains a latent chain-of-thought to predict future representations in that space.

    Source: §1 · pp. 2–3

  • Results

    RepresentNav sets a new state of the art across VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, including 83.80% success on VLNVerse fine-grained (+20.05).

    Source: §1 · p. 3

  • Takeaways & Limitations

    The self-constructed space nearly doubles the gain over concatenating the same models’ outputs, while dreaming in that space removes a quarter of collisions.

    Source: §1 · p. 3

    The representation construction has been validated only on navigation; its transfer to manipulation, dexterous hands, and humanoid whole-body control remains untested.

    Source: §6 · p. 11

Abstract

from arXiv · show

What visual representation should a navigation policy act on? Navigation policies built on vision foundation models largely leave representation design to the encoders' pretraining objectives. We present RepresentNav, a navigation foundation model built on a self-constructed representation space: a visual representation constructed by the policy's own objectives from vision foundation models. RepresentNav builds this space with Representation Residuals, a compact three-layer transformer that attends over features spanning the full depth of multiple complementary vision foundation models, so that the policy learns to see what it needs for navigation. In the same space it dreams what it wants: before every action a latent chain-of-thought predicts the representation of the scene a moment ahead. The dream is trained end-to-end with the policy as a joint-embedding predictive architecture (JEPA), with a next-representation prediction loss and a regularizer on its targets. RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks, and its gains carry over to a physical robot amid clutter and moving people. On VLNVerse fine-grained, RepresentNav achieves 83.80% success, outperforming the previous state of the art by 20.05 percentage points, and on EVT-Bench distracted tracking it reaches 85.5% with a single camera (+10.1). With the same encoders, the self-constructed space clearly outperforms concatenating their outputs and learns to weight their layers unevenly, and a dream in it cuts collisions by a quarter. A single dreamed latent decodes into the depth, semantics, and 3D occupancy of the scene a moment ahead, more accurately than pretrained encoders applied to the real future frame, although occupancy is never supervised, and the rollout tracks the robot's own motion over the next second: the dream emerges as a compact latent world model.

1 Introduction

RepresentNav constructs a navigation-specific visual space from vision foundation models and predicts future representations in that space before acting. It achieves state-of-the-art results across navigation benchmarks, while its representation and dream each show distinct benefits.

1 Introduction

  • The motivating question is whether perception should provide a general-purpose scene description or make explicit only what the behavior at hand needs.The introduction frames the representation as a description that makes decision-relevant information explicit and leaves out the rest.Source: §1 · p. 2
  • Representation Residuals build a self-constructed space by attending across the full depth of three complementary vision foundation models.The space is constructed by the policy’s objectives, and what it makes explicit can differ across regions of a view.Source: §1 · p. 2
  • Before each action, a latent chain-of-thought predicts future observation representations in the same space, trained jointly with the policy as a JEPA.The prediction targets are future-view [CLS] tokens, with a next-representation loss and VISReg regularization.Source: §1 · p. 3
  • RepresentNav sets a new state of the art on VLNVerse, VLN-CE R2R, EVT-Bench, and two point-goal benchmarks.It reaches 83.80% success on VLNVerse fine-grained (+20.05) and 85.5% tracking rate on EVT-Bench distracted tracking (+10.1) with a single camera.Source: §1 · p. 3
  • Without a dream, the self-constructed space nearly doubles the gain of concatenating the same three models’ outputs by using layers concatenation never sees.The learned weighting of layers also differs across regions of a view.Source: §1 · p. 3
  • Dreaming in the self-constructed space removes a quarter of collisions.The comparison is against acting without the dream.Source: §1 · p. 3
  • A single dreamed latent decodes depth, semantics, and 3D occupancy more accurately than pretrained encoders on the real future frame, despite no occupancy supervision.The rollout also tracks robot and target motion up to 1.2 s ahead.Source: §1 · p. 3
  • A two-stage curriculum first trains the policy to act through its dream, then to act on what it sees and dreams together.Source: §1 · p. 3

2 Related Work

RepresentNav replaces fixed visual interfaces with a task-objective-built interface over foundation-model layers and predicts in the representation used by the policy. This distinguishes its approach from prior navigation systems that choose their visual or predictive spaces in advance.

2 Related Work

  • Representation Residuals build the policy’s visual interface over layers spanning several vision foundation models, rather than fixing it to final or selected layers, concatenation, routing, or distillation.Source: §2 · p. 3
  • Prior navigation world models predict in spaces chosen in advance, while RepresentNav trains a JEPA in the space the policy sees through.The related-work discussion notes that JEPA spaces are learned with a predictor and kept from collapsing by a target mechanism or regularizer.Source: §2 · p. 3
  • Navigation foundation models commonly use a visual interface fixed before the task exists; RepresentNav changes the underlying space while retaining their generality.Source: §2 · p. 3
  • Figure 2 depicts Representation Residuals constructing the representation and a latent chain-of-thought predicting future [CLS] tokens before the planning head acts.Decoded latents are for visualization only.Source: Figure 2 · p. 4

3 Method

RepresentNav constructs a navigation representation from vision-foundation-model features and predicts future representations in that same space before acting. It trains the space, prediction, navigation policy, and auxiliary question-answering path through a two-stage curriculum.

3.1 Representation Residuals: See What I Need

  • Representation Residuals attend across the full depth of multiple encoders, letting the policy’s own objective determine which visual structure the representation makes explicit.The method uses frozen semantic, geometric, and dynamic encoders; three transformer layers aggregate spatial features and a [CLS] token gathers each view.Source: §3.1 · pp. 4–5

3.2 Latent Chain-of-Thought: Dream What I Want

  • The latent chain-of-thought predicts future representations in the policy’s space, so the task objective determines what the dream anticipates.Future observations build targets but do not enter the policy input; the rollout predicts continuously and its final hidden state maps to eight waypoints.Source: §3.2 · p. 5
  • VISReg regularizes future targets toward zero mean, unit variance, and Gaussian shape while leaving their content to navigation and prediction.The regularizer is applied to targets gathered across the global batch and averaged over four future times.Source: §3.2 · p. 5

3.3 Navigation-Grounded Predictive Curriculum

  • Training proceeds in two stages: Stage I favors prediction while the policy acts through its dream, and Stage II uses full causal attention and shifts weight to trajectory prediction.The stages use disjoint, task-balanced halves of the training samples; the vision models and backbone visual encoder remain frozen.Source: §3.3 · p. 6

3.4 Training Data

  • Training uses about 5.08M samples, including 4.69M navigation samples across VLN, tracking, and point-goal tasks and 0.39M question-answering samples.Both stages draw evenly from every source and task.Source: §3.4 · p. 6

4 Experiments

RepresentNav is evaluated on vision-and-language navigation, active tracking, and point-goal navigation across benchmarks and real-world settings. It leads reported results on several benchmarks, and real-world tests show navigation across three tasks with scene and target readouts.

4.1 Experimental Setup

  • Evaluation covers VLN-CE, VLNVerse, EVT-Bench, InternVLA-N1, and ABotN-PointBench, using each benchmark’s task-specific metrics and settings.ABotN-PointBench success requires collision-free arrival.Source: §4.1 · p. 6

4.2 Benchmark Results

  • 85.5 tracking rate on EVT-Bench distracted tracking with one camera is 10.1 points above the best monocular baseline; with multiple views, RepresentNav reaches 86.8.TrackVLA and TrackVLA++ have lower collision rates.Source: §4.2 · p. 6, Table 1 · p. 7
  • 95.0% mean success on InternVLA-N1 is 4.0 points above SPAN-Nav; on ABotN-PointBench, collision-free success is 97.5% and SPL is 95.8, each 2.1 points above ABot-N1.ABotN-PointBench counts an episode only when the robot arrives without a collision.Source: §4.2 · p. 6, Table 1 · p. 7
  • 83.80% success on VLNVerse fine-grained instructions is 20.05 points above Qwen-RobotNav-8B; RepresentNav also leads reported Val-Unseen metrics and has the best multi-view R2R success, SPL, and navigation error.On RxR it ties Qwen-RobotNav-4B on nDTW, while Qwen-RobotNav-4B leads the other RxR metrics.Source: Table 1 · p. 8, §4.2 · p. 9

4.3 Real-World Results

  • In unseen real-world environments, RepresentNav performs point-goal navigation, person following, and language-directed navigation; occupancy readouts preserve planning-relevant spatial information.Bounding-box and semantic readouts reveal target and scene-layout perception; quantitative evaluations use Go2 and Bumi platforms.Source: §4.3 · p. 9, Figure 3 · p. 9

5 Ablation Study and Analysis

The analysis tests the policy on 4,000 point-goal trajectories, where success is near ceiling and differences appear mainly in collisions, then examines the constructed space, dream, and decoded latent state.

5.1 The Self-Constructed Space Exploits the Internal Structure of the Priors

  • 93.45% success and 18.05% collisions with the constructed space outperform concatenation’s 87.30% success and 28.70% collisions; the dynamic encoder contributes 3.65 versus 0.60 points.The constructed space draws 67% of dynamic-model attention from layer L30 and 30% from its final layer, and preferences vary by region.Source: §5.1 · p. 9, Table 2 · p. 10

5.2 The Dream Pays Off in the Self-Constructed Space

  • 13.45% collisions with self-constructed-space dream targets is a quarter fewer than 18.05% without a dream, while backbone-feature targets yield 22.00%.Unsupervised rollout gives 15.90% collisions; removing VISReg raises collisions to 16.40%.Source: §5.2 · p. 10

5.3 One Dreamed Latent Holds a Compact World State

  • 34.81% occupancy IoU from the +0.4 s latent exceeds the best pretrained-encoder reference’s 29.79%, despite occupancy receiving no supervision or encoder output.The four-latent rollout also recovers robot displacement up to +1.2 s and target heading during tracking.Source: §5.3 · p. 10, Figure 5 · p. 11

6 Conclusion and Limitations

RepresentNav constructs a purposive representation on top of vision foundation models and dreams in that space; its validation is currently limited to navigation.

6 Conclusion and Limitations

  • RepresentNav builds a purposive representation on general-purpose vision foundation model descriptions, then constructs and dreams in that space.The paper reports state-of-the-art results across vision-and-language, tracking, and point-goal navigation, transfer to a physical robot, and a compact world state in one dreamed latent.Source: §6 · p. 11
  • Whether the construction transfers beyond navigation—to manipulation, dexterous hands, or humanoid whole-body control—remains untested.The authors state that what the space makes explicit depends on the policy’s training objective.Source: §6 · p. 11

A Method Details

The architecture aligns multilevel encoder features before applying distinct spatial readouts and projectors for observations and future targets.

A.1 Architecture

  • DINOv3 and DA3 process single frames, while OpenD4RT processes same-view four-frame clips; independent adapters align multilevel features to a 16 × 16 grid and common channel dimension.Source, layer, and position embeddings are added after alignment.Source: §A.1 · p. 16
  • Current views produce 64 fine spatial tokens and historical views eight coarse tokens, with a normalized 1024-dimensional [CLS] representation for each view.Current views use concatenated 2 × 2 neighborhoods; historical views average-pool over a 2 × 4 partition.Source: §A.1 · p. 16
  • The target projector maps the 4096-dimensional concatenated view targets to 256 dimensions, matching the prediction head’s 256-dimensional output.Missing views are zero-filled, and the prediction head projects each 2,560-dimensional hidden state.Source: §A.1 · p. 16
  • VISReg is computed separately for each future time after gathering target representations across the global batch, then averaged across times.Source: §A.1 · p. 16

A.2 Training

Training first emphasizes navigation-relevant future representations, then shifts toward trajectory learning while retaining prediction and target-space regularization.

A.2 Training

  • Stage I trains four-step latent CoT with online future targets and isolates visual and language prefixes so the action position attends only to language and generated latents.Its loss weights are (λnav, λwm, λreg, λqa) = (0.01, 0.1, 0.9, 1); QA receives weight 1 despite representing one tenth of the mixture.Source: §A.2 · p. 16
  • Stage II inherits Stage I and restores standard causal attention, allowing actions to read observations, instructions, and predicted latents together.Its weights (λnav, λwm, λreg, λqa) = (1, 0.001, 0.009, 1) emphasize trajectory learning while retaining prediction and regularization.Source: §A.2 · p. 16
  • Both stages update the backbone, Representation Residuals, learnable [CLS], planner, and projectors, while the three source encoders and Qwen3-VL visual encoder remain frozen.QA uses backbone visual input and DA3/DINOv3 features without navigation latent rollout or future-target construction.Source: §A.2 · p. 16
  • The two stage datasets are disjoint by sample but not by episode, so samples from one episode can occur in both halves.Training data composition is shown in Figure 6.Source: §A.2 · p. 17, Figure 6 · p. 17

A.3 The Two-Stage Curriculum

The two-stage curriculum makes the policy more sensitive to its latent inputs than opening attention from the start, while the representation analysis examines how the learned space uses encoder layers and how much its gains depend on capacity.

A.3 The Two-Stage Curriculum

  • Cross-scene swaps shift the two-stage policy’s trajectory by 4.95 cm versus 1.38 cm for the single-stage model, and zeroing latent inputs shifts it by 8.24 cm versus 2.68 cm.The shifts are larger in 183 and 182 of 200 observations, respectively; ADE rises from 4.38 cm to 7.06 cm and 9.76 cm under the interventions. These are open-loop action responses, not closed-loop success or collision rates.Source: §A.3 · pp. 17–18, Table 4 · p. 17, Figure 7 · p. 18

B.1 Protocols

  • Readout decoders are trained after the fact on frozen latents to analyze recovery of geometry, semantics, target states, and motion.The same lightweight decoder is trained on each representation for a given comparison.Source: §B.1 · p. 18

B.2 The Self-Constructed Space (Section 5.1)

  • The attention analysis tracks Representation Residuals’ weights across encoder layers and over training, including a running state counted as geometric.Figure 8 averages attention shares over the residual layers and spatial positions; Figure 4(b)’s colors instead show contrast-enhanced relative preferences, not absolute weights or causal contributions.Source: §B.2 · p. 19, Figure 8 · p. 19
  • With three Representation Residuals layers, point-goal navigation reaches 93.45% SR and 84.2% SPL; layer-count variants lie within 2.2 points on both metrics.The comparison concerns point-goal navigation without the dream and reports collision rate of 18.05% for the three-layer default.Source: Table 5 · p. 19, §B.2 · p. 19

B.3 The Target of the Dream (Section 5.2)

The section tests the dream-target regularizer, analyzes what future-scene information can be read from dreamed latents, and compares target and motion readouts on Track.

B.3 The Target of the Dream (Section 5.2)

  • Without VISReg, success falls from 95.00% to 92.90% and collisions rise from 13.45% to 16.40%, worse than the unsupervised rollout’s 93.35% success and 15.90% collisions.SIGReg and VICReg recover less than VISReg; the comparison changes the regularizer on the dream targets.Source: §B.3 · p. 19, Table 6 · p. 20

B.4 What the Dream Holds (Section 5.3)

  • The future-scene readout comparison evaluates depth, semantics, occupancy, and ego-motion, with readouts from one vector per observation.Depth targets are normalized relative inverse depth rather than metric depth; occupancy uses voxel labels, and ego-motion ADE measures XY error at six future positions.Source: §B.4 · p. 20, Table 7 · p. 20, Figure 9 · p. 21

B.4.1 Target Localization and Motion on Track

  • RepresentNav’s front-view box IoU is 72.81%, versus 59.33% for two-stage SIGReg and 33.69% for the backbone ViT; future person-motion ADE is 0.44450 m versus 0.78507 m and 1.09701 m.Future ego-motion ADE is 0.15455 m for RepresentNav, versus 0.22126 m and 0.33404 m. The readouts use frozen models on Track samples, not closed-loop navigation.Source: §B.4.1 · p. 22, Table 8 · p. 22, Figure 10 · p. 22

C Qualitative Results

The section presents real-world demonstrations and a 40-case comparison across VLN and tracking, alongside platform, deployment, AI-use, and ethics information.

C Qualitative Results

  • RepresentNav completes 14 of 20 real-world VLN cases (70%) and 18 of 20 tracking cases (90%), versus Uni-NaVid’s 7 (35%) and 8 (40%).These correspond to success-rate improvements of 35 percentage points on VLN and 50 percentage points on tracking.Source: C Qualitative Results · p. 26, Figure 14 · p. 26
  • Real-world examples show point-goal navigation, instruction-following routes, and tracking a designated moving person.The demonstrations include passing indoor landmarks and following a person wearing green pants and a white top.Source: C Qualitative Results · pp. 26–27, Figure 15 · p. 27
  • The deployment sends RGB observations to an RTX 4090 server running RepresentNav and returns predicted waypoints to the onboard controller.The described platforms are a Go2 quadruped and a Bumi humanoid.Source: Figure 14 · p. 26, C Qualitative Results · p. 26

D AI Use Statement

  • The authors report using Claude to improve manuscript writing and take responsibility for the final content, analyses, and conclusions.Source: D AI Use Statement · p. 28

E Ethics Statement

  • The authors state that real-world recordings disclose no personal information and that appearing faces are blurred.Source: E Ethics Statement · p. 28

F Reproducibility Statement

The reproducibility statement maps the paper’s methods, evaluation, and supporting materials to sections and appendices.

F Reproducibility Statement

  • The paper assigns Representation Residuals, latent chain-of-thought targets, the two-stage curriculum, and training data to Sections 3.1–3.4.Source: F Reproducibility Statement · p. 28
  • Section 4 covers benchmarks, metrics, and implementation details, while Appendix A gives architecture and training details including loss weights and stage data splits.Source: F Reproducibility Statement · p. 28
  • Appendix B.1 provides protocols for ablations and representation analyses to support reproducibility.Source: F Reproducibility Statement · p. 28
Loading 2610.31762v1…