PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent-WAM: Latent World Action Modeling for End-to-End Autonomous Driving

Linbo Wang, Yupeng Zheng, Qiang Chen, Shiwei Li, Yichen Zhang, Zebin Xing, Qichao Zhang, Xiang Li, Deheng Qian, Pengxuan Yang, Yihang Dong, Ce Hao, Xiaoqing Ye, Junyu han, Yifeng Pan, Dongbin Zhao

arXiv:2603.24581v1cs.CVcs.RO

TL;DR

World-model planners have gaps in scene compression, spatial understanding, and use of temporal dynamics. Latent-WAM combines geometric distillation and compact scene tokens with causal prediction of future world status. It reports state-of-the-art results on NAVSIM v2 and HUGSIM, including a 3.2 EPDMS gain over prior perception-free methods.

  • Problem

    World-model-based planners have inadequately compressed representations, limited spatial understanding, and underused temporal dynamics.

    Source: §1 · p. 2

  • Method

    Latent-WAM combines geometric distillation and learnable-query scene compression with a causal Transformer that predicts future world status from historical visual and motion representations.

    Source: §1 · p. 3

  • Results

    89.3 EPDMS on NAVSIM v2 and 45.9 RC and 28.9 HD-Score on HUGSIM are reported as state-of-the-art, with a 3.2 EPDMS gain over prior perception-free methods.

    Source: §1 · p. 3, HUGSIM · p. 11, §5 · p. 15

  • Takeaways & Limitations

    The paper concludes that spatial-aware compression and dynamic latent modeling provide compact, planning-relevant world representations for autonomous driving.

    Source: §5 · p. 15

    Predicted poses use the ego vehicle’s local coordinate system and assume uniform intervals between consecutive poses.

    Source: §3.4 · p. 8

Abstract

from arXiv · show

We introduce Latent-WAM, an efficient end-to-end autonomous driving framework that achieves strong trajectory planning through spatially-aware and dynamics-informed latent world representations. Existing world-model-based planners suffer from inadequately compressed representations, limited spatial understanding, and underutilized temporal dynamics, resulting in sub-optimal planning under constrained data and compute budgets. Latent-WAM addresses these limitations with two core modules: a Spatial-Aware Compressive World Encoder (SCWE) that distills geometric knowledge from a foundation model and compresses multi-view images into compact scene tokens via learnable queries, and a Dynamic Latent World Model (DLWM) that employs a causal Transformer to autoregressively predict future world status conditioned on historical visual and motion representations. Extensive experiments on NAVSIM v2 and HUGSIM demonstrate new state-of-the-art results: 89.3 EPDMS on NAVSIM v2 and 28.9 HD-Score on HUGSIM, surpassing the best prior perception-free method by 3.2 EPDMS with significantly less training data and a compact 104M-parameter model.

1 Introduction

Latent-WAM targets gaps in world-model planners’ compression, spatial understanding, and use of temporal dynamics with a spatial-aware encoder and causal latent world model. It reports state-of-the-art results on NAVSIM v2 and HUGSIM, including a 3.2 EPDMS advantage over prior perception-free methods.

1 Introduction

  • World-model planners face inadequate compression, weak spatial understanding, and underused history when predicting only the next representation from the current frame.The paper identifies these limitations as associated with sub-optimal planning performance.Source: §1 · p. 2
  • SCWE distills geometric knowledge into the vision backbone and uses learnable queries to compress enriched visual features into spatially informed tokens.The contribution is described as a scene-compression method for spatially aware visual representations.Source: §1 · pp. 2–3
  • DLWM uses a causal Transformer to predict future world status from historical visual and motion representations, with self-supervised visual and supervised motion prediction.A lightweight trajectory decoder generates the planned trajectory from world-status representations.Source: §1 · p. 3
  • 89.3 EPDMS on NAVSIM v2 and 45.9 RC and 28.9 HD-Score on HUGSIM establish new state-of-the-art results; the perception-free method exceeds prior approaches by 3.2 EPDMS.The reported attention maps focus on spatial structures and driving intent.Source: §1 · p. 3

2 Related Works

Prior end-to-end driving work spans dense, sparse, and language-informed scene representations, while world-model approaches use video generation or latent prediction. Latent-WAM combines geometric distillation and compact tokens with ego-conditioned latent prediction and 3D-RoPE.

2.1 Scene Representation in End-to-End Autonomous Driving

  • End-to-end scene representations progressed from dense BEV supervision to lighter vectorized or sparse forms and VLM-based semantic representations; Latent-WAM distills geometric knowledge and compresses scenes into learnable-query tokens.The paper identifies weak 3D spatial understanding and cumbersome representations as issues in prior approaches.Source: §2.1 · p. 3

2.2 World Models for Autonomous Driving

  • World models for driving include pixel-level video generation and 3D occupancy or point-cloud models; later work unifies 2D and 3D modeling or uses models for simulation and planning.Latent-WAM uses a latent world model, incorporates ego status to guide future predictions, and adds 3D-RoPE for spatiotemporal tracking.Source: §2.2 · pp. 3–4

3 Method

Latent-WAM combines geometric distillation and learnable scene queries with causal prediction of unified scene-and-ego world states. Its trajectory decoder plans from these representations, and training combines trajectory, alignment, world-model, and ego-status losses.

3.1 Overall

  • Latent-WAM has three modules: SCWE compresses images and distills geometric perception, DLWM autoregressively predicts future world status from scene and ego history, and a trajectory decoder forecasts over four seconds.SCWE and DLWM jointly optimize the vision backbone; the design targets planning without extra inference cost from auxiliary modules.Source: §3.1 · p. 4

3.2 Spatial-Aware Compressive World Encoder

  • Learnable scene queries interact with image-patch tokens to compress multi-view visual information, while WorldMirror distills geometric awareness into the vision backbone.Geometric alignment compares normalized projected SCWE patch tokens with normalized foundation-model features using cosine similarity.Source: §3.2 · p. 5, Scene Compression · p. 5, Geometric Alignment · p. 5

3.3 Dynamic Latent World Model

  • Per-camera scene tokens are aggregated and concatenated with encoded ego status to form unified frame-wise world states for prediction and planning.The causal world model autoregressively predicts future unified states from historical scene and ego representations, with future targets supplied by a frozen, EMA-updated SCWE copy.Source: World latent status aggregation · p. 6, Causal world model prediction · p. 6
  • Frame-wise attention allows bidirectional attention within each frame and restricts cross-frame attention to earlier tokens, enabling parallel future prediction with temporal causality.Teacher forcing uses ground-truth tokens as context during training; 3D-RoPE encodes temporal, camera-view, and token positions.Source: Teacher Forcing Attention Mask · p. 7, 3D-RoPE · p. 7

3.4 Trajectory Planning

  • Learnable trajectory queries and current world status produce candidate trajectories, from which the current driving command selects the final trajectory.Each pose is represented by (x, y, θ) in the ego vehicle’s local coordinates, with uniform intervals between consecutive poses.Source: §3.4 · p. 8

3.5 Training and Inference

  • End-to-end training combines trajectory imitation, geometric alignment, future-world prediction, and ego-status supervision; inference requires only SCWE and the trajectory decoder.The objective weights alignment, world-model, and ego-status losses by α, β, and γ, respectively.Source: §3.5 · p. 8, Inference · p. 8

4 Experiment

Experiments report strong NAVSIM v2 and zero-shot HUGSIM results, then examine the contributions of compression, geometric distillation, dynamic prediction, and spatially aware representations. Ablations also compare backbone choices and temporal prediction density.

4.1 Implementation Details

  • The 104M-parameter inference model uses DINOv2-Base, 16 learnable scene queries, four frames, and three camera views; training ran for 100 epochs on 32 A100 GPUs.The training configuration took approximately two days.Source: Architecture · p. 9, Training Configuration · p. 9

4.2 Benchmarks & Main Results

  • Latent-WAM ranks first in NAVSIM v2 EPDMS and zero-shot HUGSIM RC (45.9), while matching the best HUGSIM HD-Score (28.9).The NAVSIM comparison includes methods using perception annotations; HUGSIM evaluation uses a model trained exclusively on NAVSIM without fine-tuning.Source: §4.2 · pp. 9–10, Table 1 · p. 10, Table 2 · p. 10

4.3 Ablation Study

  • Compressing image patches into scene tokens costs 0.2 EPDMS from the 87.9 direct-patch baseline, while dynamic modeling and added ego status raise scores from 87.7 to 88.3.Adding geometric information to the world model reaches 89.0, and the full model reaches 89.3.Source: Effectiveness of Each Component · p. 11
  • Geometric distillation reaches 89.3 EPDMS, versus 88.3 without geometry and 88.0 with frozen geometry features concatenated as key-value inputs.The authors attribute the weaker concatenation result likely to misalignment with the planning objective.Source: Impact of Geometric Information · pp. 11–12, Table 4 · p. 12
  • DINO-Base with full fine-tuning achieves 89.3 EPDMS, while LoRA variants score 84.7 for DINO-Small and 68.5 for DINO-Base.The authors conclude effective geometric distillation requires sufficient model capacity and full fine-tuning flexibility.Source: Vision Backbone for Geometric… · p. 12, Table 5 · p. 12
  • Prediction at stride 4 reaches 89.3 EPDMS versus 88.4 for final-frame-only prediction; denser stride 2 reaches 89.1 and adds computation without extending the horizon.The authors conclude moderate prediction density balances supervision effectiveness and optimization efficiency.Source: World Model Prediction Temporal Stride · pp. 12–13, Table 6 · p. 13
  • Predicted trajectories align better with human paths and maintain safer distances than Epona, while World4Drive produces acceptable but sub-optimal paths.Fig. 4 visualizes human trajectories in green and predicted trajectories in yellow.Source: Figure 4 · p. 13, Qualitative Analysis · p. 13
  • Geometric distillation concentrates attention on lane markings, scene structures, and drivable areas; attention also follows the intended direction for straight, right-turn, and left-turn maneuvers.The baseline has more scattered attention, including substantial weights on irrelevant background regions.Source: Qualitative Analysis · p. 14, Figure 5 · p. 14

5 Conclusion

Latent-WAM builds compact, planning-relevant world representations through spatial-aware compression and dynamic latent modeling, achieving state-of-the-art results on NAVSIM v2 and HUGSIM.

5 Conclusion

  • Latent-WAM achieves 89.3 EPDMS on NAVSIM v2 and 45.9 RC and 28.9 HD-Score on HUGSIM, with significantly less training data and fewer parameters than competing methods.Its framework combines a Spatial-Aware Compressive World Encoder with a Dynamic Latent World Model.Source: §5 · p. 15

A. Metrics

This section defines the NAVSIM v2 and HUGSIM evaluation metrics, including NAVSIM v2’s added rule-compliance and comfort measures and HUGSIM’s route-completion-scaled score.

A.1. NAVSIM

  • NAVSIM v2 EPDMS combines collision, drivable-area, direction, and traffic-light terms with a weighted average of progress, time-to-collision, lane-keeping, and comfort metrics.DDC, TLC, LK, HC, and EC are newly introduced in NAVSIM v2.Source: §A.1 · p. 19
  • NAVSIM v2 adds Driving Direction Compliance, Traffic Light Compliance, Lane Keeping, History Comfort, and Extended Comfort to the metric set.The named measures are DDC, TLC, LK, HC, and EC, respectively.Source: §A.1 · p. 19

A.2. HUGSIM

  • HUGSIM’s final HD-Score averages per-frame scores across timestamps and scales the result by route completion, Rc ∈[0, 1].Its component metrics are No at-fault Collision, Drivable Area Compliance, Time to Collision, and Ego Comfort, using NAVSIM v1 definitions.Source: §A.2 · p. 19
  • HUGSIM’s NC and TTC metrics also account for collisions with static background entities using semantic segmentation.Examples include buildings, fences, and vegetation.Source: §A.2 · p. 19

B. Data Processing Pipeline

The pipeline preprocesses multi-view images, adjusts camera geometry for resizing and cropping, transforms between camera and world coordinates, and extracts geometric features with a frozen foundation model.

B.1. Image Preprocessing

  • Three camera views are resized from 1920 × 1080 to 455 × 256, then center-cropped to 448 × 224 images with a 2:1 aspect ratio.The views are left, front, and right.Source: §B.1 · p. 20

B.2. Camera Intrinsic Adjustment

  • Camera intrinsics are adjusted for resize scale and center-crop offsets to preserve geometric consistency.The focal lengths and principal point are scaled, with crop offsets subtracted from the principal-point coordinates.Source: §B.2 · p. 20

B.3. Camera Extrinsic Transformation

  • The camera-to-world extrinsic matrix contains rotation and translation, and its inverse provides the world-to-camera transform used for 3D-to-2D projection.Source: §B.3 · p. 20

B.4. Geometric Feature Extraction

  • Frozen WorldMirror extracts patch-level geometric features and uses camera pose and intrinsic information as conditions while excluding depth supervision.The features are precomputed and cached offline, reducing training time and GPU memory overhead.Source: §B.4 · p. 21

C. Inference Latency

Inference latency is measured per module on a single A100 GPU for a single batch, with warm-up iterations and repeated forward passes.

C. Inference Latency

  • Per-module latency is measured on one A100 GPU for a single batch after three warm-up iterations, averaging the final result over 10 forward passes.The provided table is identified as reporting per-module inference latency, but no latency values are stated in the supplied passages.Source: C. Inference Latency · p. 21, Table 7 · p. 21

D.1. Trajectory Planning

The NAVSIM visualizations distinguish human driving trajectories from each method’s predicted trajectories.

  • Green marks the human driving trajectory, while yellow marks the corresponding method’s predicted trajectory.The figure provides additional trajectory visualizations on NAVSIM.Source: Figure 6 · p. 22

D.2. Attention Map

Geometric distillation yields more focused attention on task-relevant regions across four driving scenarios; HUGSIM planning visualizations cover several datasets.

D.2. Attention Map

C.3. HUGSIM

Related papers

Loading 2603.24581v1…