PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Representation and History Modeling for Navigation World Models

Guangfu Guo, Xiaoqian Lu, Rui Liu, Yutong Chen, Kunpeng Liu, Long Cheng

arXiv:2609.29555v1cs.CVcs.RO

TL;DR

Navigation world models need visual representations suited to prediction objectives and efficient ways to reuse history across candidate queries. This paper benchmarks five frozen representations and develops reusable-history architectures. Results vary by prediction objective, and the history designs trade prediction quality against query efficiency.

  • Problem

    Reconstruction quality alone may not identify a suitable predictive representation, while global attention repeatedly recomputes shared history for candidate actions.

    Source: §I · p. 1

  • Method

    The paper benchmarks five frozen visual representations in one dynamics framework and develops Cached-Linear and Balanced GDN for reusable history processing.

    Source: §I · pp. 1–2

  • Results

    Across RECON, SACSoN, and SCAND, the best representation differs by prediction objective, while Cached-Linear is more efficient for shared-history queries and Balanced GDN improves selected direct-prediction results.

    Source: §I · p. 2

  • Takeaways & Limitations

    Choose the visual representation for the prediction task, then choose history processing for the expected query workload.

    Source: §VI · pp. 9–10

Abstract

from arXiv · show

Navigation World Models (NWMs) predict action-conditioned visual futures for planning. Two practical challenges are central to their design: selecting a suitable visual representation and efficiently modeling observation history for repeated candidate queries. Standard Global-Softmax attention provides flexible interactions but repeatedly processes the same history, leading to increasing computation and memory costs for long contexts and multi-query planning. We study both problems within a unified conditional flow-transformer framework. We first compare five frozen visual representations under the same dynamics model and evaluation. To reduce redundant history computation, we design Cached-Linear, a hybrid architecture that combines local and shifted-window attention for target mixing with linear attention for reusable history access. We further develop Balanced Gated Delta Network (GDN), which augments this design with frame-wise recurrent memory for temporal history modeling. Experiments on RECON, SACSoN, and SCAND show that representation choice depends on the prediction objective: PAE-L performs best for reconstruction, RAE-B for direct prediction, and V-JEPA for long-horizon rollout. Under shared-history workloads, Cached-Linear substantially reduces computation and memory compared with Global-Softmax, while Balanced GDN improves selected direct-prediction endpoints with efficient context reuse. Overall, we systematically study visual representation and history modeling for NWMs and develop hybrid reusable-history architectures for efficient long-context and multi-query prediction.

I. INTRODUCTION

Navigation world models must choose representations for prediction objectives, not reconstruction alone, while efficiently reusing observation histories across candidate queries. This work benchmarks five frozen representations and compares two reusable-history designs with Global-Softmax.

I. INTRODUCTION

  • Autoregressive rollout can accumulate small prediction errors, so reconstruction, direct prediction, and long-horizon rollout may favor different visual representations.Selecting a representation by reconstruction quality alone can therefore be misleading for navigation world models.Source: §I · p. 1
  • Global attention recomputes history-dependent information for each candidate action, and this cost grows with history length and the number of queries.A useful history mechanism should retain prediction quality while reusing shared-history computation.Source: §I · p. 1
  • Cached-Linear reuses a linear summary of observation history across candidate queries, while Balanced GDN adds frame-wise recurrent memory for temporal modeling.Both designs are compared with the standard Global-Softmax baseline for prediction quality and repeated-query efficiency.Source: §I · pp. 1–2
  • The framework compares five frozen representations using the same dynamics model and evaluation pipeline.The representations are RAE-B, RAE-L, PAE-L, Cosmos, and V-JEPA.Source: Figure 1 · p. 2, §I · p. 2
  • Across RECON, SACSoN, and SCAND, different representations lead for reconstruction, direct prediction, and long-horizon rollout, while Cached-Linear is more efficient for shared-history queries and Balanced GDN improves selected direct-prediction results.Global-Softmax remains competitive for a single short query; Balanced GDN particularly improves RAE-B Direct@32.Source: §I · p. 2

B. Visual-Representation

Prior work motivates efficient attention and recurrent memory for long contexts, while this paper applies conditional flow matching to predict future visual states from observations and actions. Its architecture separates reusable history processing from target conditioning for repeated queries.

B. Visual-Representation

  • Representation autoencoders, compact representation-aligned latents, and video-prediction features motivate comparing visual state spaces beyond reconstruction quality alone.Related work also motivates separating visual fidelity, downstream utility, and representation quality.Source: B. Visual-Representation · p. 2

C. Efficient Attention and History Modeling

  • Standard Transformer attention has quadratic sequence-length cost; kernelized linear attention avoids explicitly constructing the full query–key matrix, while local windows with shifts allow cross-window communication.Gated DeltaNet offers a recurrent alternative with gated memory control and delta-rule updates; related world models combine recurrent memory with periodic or bidirectional attention.Source: C. Efficient Attention and History… · pp. 2–3

A. Navigation World Model

  • A frozen encoder maps recent camera observations to spatial latent grids, and conditional flow matching trains the model to move a noisy future latent toward the clean latent given history, action, horizon, and flow time.At inference, Euler integration starts from Gaussian noise and follows the learned field from t = 1 to t = 0 before a frozen decoder maps the final latent to an image.Source: A. Navigation World Model · p. 3
  • The history computation is condition-independent, allowing the same encoded observation window to serve multiple horizons, integration steps, or candidate actions.The target stream receives action, horizon, and flow-time modulation, while history latents enter a separate context stream.Source: A. Navigation World Model · p. 3
  • Direct prediction generates each horizon independently from real observations, whereas autoregressive rollout inserts each short-step prediction into history and tests stability after model-generated states enter later queries.Rollout does not decode and re-encode intermediate predictions.Source: A. Navigation World Model · p. 3

B. Frozen Visual Representation

The paper keeps visual representations frozen and compares their decoded predictions in image space, with history computation separated from action conditioning so it can be reused across candidate actions.

B. Frozen Visual Representation

  • All visual representations produce a 16 × 16 spatial grid and remain frozen during dynamics training, while differing in pretraining objective, channel count, and decoder design.The dense, compact-aligned, generative-compact, and predictive-video representations cover semantic, generative, and temporally predictive pretraining.Source: B. Frozen Visual Representation · p. 3
  • Table I is titled as a summary of history-mixing components and their roles.The supplied passage identifies the table's scope but not its entries.Source: Table I · p. 3
  • Decoded reconstruction measures information retained by the frozen representation, while decoded end-to-end prediction combines representation, dynamics, and decoder error.The reconstruction reference is useful context, but is not a mathematical lower bound on the end-to-end score.Source: B. Frozen Visual Representation · p. 3

C. Action Conditioning

  • The action encodes relative forward, lateral, and heading displacement; its features are fused with horizon and flow-time embeddings and injected into the target branch.Because the history branch depends only on the observation window, its computation can be reused across candidate actions.Source: C. Action Conditioning · p. 3

D. History Mixer Architectures

The section replaces repeated global history attention with reusable-history mixers, pairing local target mixing and cached linear history access; Balanced GDN adds frame-wise recurrent memory.

a) Local target mixing

  • Cached-Linear uses local window and shifted-window attention for target mixing, preserving local attention while allowing information to propagate across neighboring regions.Shifted-window attention cyclically shifts the partition by half a window and applies a boundary mask.Source: D. History Mixer Architectures · p. 4

b) Cached linear history access

  • Cached-Linear builds a history cache once at linear cost and reuses it for target queries, avoiding the full query–history score matrix.The same observation history can be shared across flow-integration steps and candidate actions.Source: D. History Mixer Architectures · p. 4

c) Balanced GDN recurrent history model

  • Balanced GDN augments Cached-Linear’s additive history summary with frame-wise recurrent memory, replacing one self-mixing block in each four-block group with a Gated Delta Network.The remaining blocks combine local attention, shifted-window attention, cached linear cross-attention, and a final global attention anchor.Source: D. History Mixer Architectures · pp. 4–5
  • GDN evaluates all P token contributions within a frame in parallel and scans frames in temporal order; its recurrent-state memory does not grow with context length.The stated per-head memory is O(d_h^2 + d_h), and the learned update can replace stale information as new frames arrive.Source: D. History Mixer Architectures · p. 4
  • The 12 mixer blocks form three four-block groups, combining GDN, window attention, cached linear mixing, shifted-window attention, and a final full-attention block.The schedule uses GDN for frame-wise temporal memory, local attention for spatial interactions, and a global communication path in the final block.Source: D. History Mixer Architectures · p. 5

d) Auxiliary iREPA spatial alignment

  • The auxiliary Balanced GDN + iREPA variant adds a training-time alignment loss from one intermediate dynamics feature to the clean visual-feature target, then removes that branch at inference.It keeps the tokenizer, decoder, and history-mixer schedule unchanged and uses one feature level without multi-block fusion.Source: D. History Mixer Architectures · p. 5

A. Evaluation Overview

The evaluation uses shared decoded-image quality comparisons while keeping systems timing separate. It specifies balanced multi-dataset sampling, direct and rollout tasks, and a common training setup.

A. Evaluation Overview

  • Quality comparisons use the same decoded-image protocol, while systems timing is kept separate from prediction quality.Source: A. Evaluation Overview · p. 5

B. Data and Evaluation Sets

  • Each epoch samples 62,813 observations per dataset; paired comparisons reuse sample identifiers and initial diffusion noise, and rollout returns predicted latents without intermediate decode/re-encode.Four future targets are supplied per observation; each optimizer step contains 96 observations and 384 conditional transitions.Source: B. Data and Evaluation Sets · p. 5
  • Direct@4 and Direct@16 endpoints are generated independently from real history; trajectory prediction is a two-second, eight-step open-loop test with 120 candidates, top-3 selection, and one CEM update.The nominal held-out sets contain 199 SACSoN, 598 RECON, and 42 SCAND trajectories.Source: B. Data and Evaluation Sets · p. 5

C. Training Configuration

  • Training uses a 12-block CDiT-B/2 backbone, AdamW, BF16, seed 42, deterministic sampling, and EMA checkpoints.The Global-Softmax/Cached-Linear RAE pair has 364.24M trainable parameters; cross-representation Balanced GDN bundles are kept near 350M.Source: C. Training Configuration · p. 5

D. Metrics, Aggregation, and Uncertainty

This section defines image-quality metrics and aggregation, conditioning and motion diagnostics, and the controlled selection protocol for comparing representation and history designs.

D. Metrics, Aggregation, and Uncertainty

  • DINO is the primary decoded endpoint metric; LPIPS, DreamSim, PSNR, and SSIM provide complementary image measures with their stated better-direction conventions.Lower is better for DINO, LPIPS, and DreamSim; higher is better for PSNR and SSIM.Source: D. Metrics, Aggregation, and Uncertainty · p. 5

E. Conditioning and Control Diagnostics

  • Conditioning sensitivity measures the DINO-distance increase after action or horizon corruption, while a frozen-state linear probe tests accessible relative planar-motion information.The action is zeroed, shuffled, or sign-flipped, or the requested horizon is mismatched.Source: E. Conditioning and Control Diagnostics · p. 5

F. Joint Selection Protocol

  • The selection protocol retains PAE-L for reconstruction, RAE-B for direct prediction, and V-JEPA for rollout; the 3×4 component matrix is exploratory.Table II is the reference for training budgets and evidence roles; the visual-representation study compares five frozen states under Global-Softmax.Source: F. Joint Selection Protocol · pp. 5–6, Table II · p. 6

V. RESULTS

Across the visual-representation study, the best representation changes with prediction objective, evaluation metric, and dataset. Conditioning diagnostics also show model responses to action and horizon inputs, while frozen visual features lack a clear linear encoding of relative motion.

1) Different Prediction Objectives Favor Different Visual Representations:

  • PAE-L has the lowest reconstruction DINO at 0.069; RAE-B leads direct prediction at 0.574 for Direct@16 and 0.608 for Direct@32, while V-JEPA leads long-horizon rollout at 0.761 and 0.771.RAE-B leads at shorter rollout horizons, but V-JEPA overtakes it after 3.25 s.Source: 1) Different Prediction Objectives… · p. 6

2) Representation Rankings Vary Across Metrics and Trajectory Prediction:

  • Representation rankings vary by metric and dataset: RAE-B leads several image metrics, while trajectory-prediction winners differ across SACSoN, RECON, and SCAND.RAE-L has the lowest RECON RPE, and V-JEPA performs best on SCAND; Table V covers two-second, eight-step open-loop trajectory prediction.Source: 2) Representation Rankings Vary Across… · p. 6, Table V · p. 7

3) Visual Representations Respond to Motion Conditions:

  • Corrupting or changing action and horizon inputs worsens prediction quality on average across all five representations; RAE-B is most sensitive and Cosmos least sensitive.The frozen-feature probe finds negative macro R2 for all five representations, with no clear linear encoding of relative SE(2) motion.Source: 3) Visual Representations Respond to… · pp. 6–7

B. Main Experiment: Representation–Mixer Trade-offs

The main experiment compares history mixers and selected representation–mixer configurations. Balanced GDN improves one direct-prediction endpoint over Global-Softmax but does not improve rollout, while preferred configurations vary across image metrics and datasets.

B. Main Experiment: Representation–Mixer Trade-offs

  • The comparison covers seven selected configurations built from retained visual representations and history-processing designs.The reported questions include mixer effects on direct prediction and rollout, ranking shifts across metrics and datasets, and whether image prediction aligns with open-loop trajectory prediction.Source: B. Main Experiment:… · p. 7, 2) Preferred Configurations Vary… · p. 8

1) History Mixers Affect Direct Prediction and Rollout Differently:

  • Direct@32 DINO improves from 0.554 with Global-Softmax to 0.540 with Balanced GDN, but Rollout@16 rises from 0.530 to 0.542 and Rollout@32 from 0.554 to 0.563.Avg-4 DINO is similar across Global-Softmax (0.5300), Cached-Linear (0.5310), and Balanced GDN (0.5314).Source: Figure 4 · p. 7

2) Preferred Configurations Vary Across Metrics and Datasets:

  • No configuration leads every image metric: RAE / Global-Softmax leads Avg-4 DINO, DreamSim, and PSNR, while other configurations lead LPIPS and SSIM.The results also vary by dataset: SCAND selects an RAE-B configuration at all four DINO endpoints, while PAE / Balanced GDN + iREPA leads several RECON and SACSoN endpoints.Source: 2) Preferred Configurations Vary… · p. 7, Table VII · p. 8, Figure 5 · p. 8

3) Image-Prediction Gains Do Not Necessarily Translate to Better Trajectory Prediction:

Open-loop trajectory results differ from decoded-image results, and ablations show that preferred configurations vary by representation, metric, and prediction horizon. Reusable history processing is more efficient under shared-query workloads, with Balanced GDN trading slightly more computation for selected prediction gains.

3) Image-Prediction Gains Do Not Necessarily Translate to Better Trajectory Prediction:

  • 3.259/0.777 on SACSoN, 1.468/0.401 on RECON, and 1.021/0.285 on SCAND are the ATE/RPE results for RAE / Global-Softmax, lowest on all three datasets.No Balanced GDN configuration wins a trajectory column, despite strong results for other configurations on some image metrics.Source: Table IX · p. 8, 3) Image-Prediction Gains Do Not… · p. 8

C. Ablation I: Component Bundles

  • Adding iREPA raises DINO in all 12 comparisons; SACSoN winners also vary by horizon and metric across RAE-B and V-JEPA configurations.RAE-B with Balanced GDN gives the lowest Direct@4 DINO, RAE-B with Cached-Linear the lowest Direct@16 DINO, and V-JEPA with Cached-Linear the lowest FID-199 at both horizons.Source: C. Ablation I: Component Bundles · p. 8, Table XI · p. 9
  • Balanced GDN gives the lowest Avg-4 DINO for RAE-B and PAE-L, while Cached-Linear is best for V-JEPA; PAE-L favors different mixers for direct prediction and rollout.No history mixer is consistently best across representations and prediction modes.Source: C. Ablation I: Component Bundles · p. 8, Table X · p. 9
  • Changed action and prediction-horizon conditions increase mean DINO by 0.0034 to 0.0318, although one PAE-L case does not follow the trend.Overall, the results indicate that evaluated models use conditioning information during prediction.Source: C. Ablation I: Component Bundles · p. 8, Table XII · p. 9

1) Shared-History Efficiency and Quality Trade-off:

  • 51.67 s to 7.15 s latency and 34.63 GiB to 5.35 GiB peak memory: Cached-Linear is more efficient than Global-Softmax for 64 history frames and 120 queries.The comparison reports H100 BF16 latency and peak memory.Source: Table XIII · p. 9, 1) Shared-History Efficiency and… · p. 9
  • At 128 frames Global-Softmax runs out of memory while Cached-Linear remains nearly unchanged; Balanced GDN costs slightly more computation but improves RAE-B Direct@32.Cached-Linear provides the strongest efficiency, while Balanced GDN offers a recurrent quality–efficiency trade-off.Source: 1) Shared-History Efficiency and… · p. 9

2) Architecture Efficiency Comparison:

Under a fixed GPU workload, Cached-Linear and Balanced GDN are faster than Global-Softmax. The conclusion recommends choosing visual representation by prediction task and history mixer by expected workload.

VI. CONCLUSION

  • The preferred visual representation changes with prediction objective, so reconstruction quality alone is not enough to choose the predictive state.The conclusion distinguishes representation choice from history-mixer choice.Source: §VI · pp. 9–10

2) Architecture Efficiency Comparison:

VI. CONCLUSION

  • Global attention remains effective for simple queries, while reusable history mechanisms are more suitable when the same context serves many candidate predictions.The paper's design principle is to choose the visual representation for the task, then the history-processing method for the expected workload.Source: §VI · p. 10

Related papers

Loading 2609.29555v1…