PaperlayerView exact revision on arXiv

Source-linked AI summary

Drive-HWM: Hierarchical World Models for Dynamic-Latent Guided Autonomous Driving

Zhaoxin Fan, Tianbao Zhang, Wenjun Wu, Xiaofeng Wang, Yeying Jin, Jian Zhao, Zheng Zhu, Shuicheng Yan

arXiv:2609.03572v1cs.CV

TL;DR

Existing driving world models either separate future prediction from action generation or predict both at the same temporal scale, limiting the balance between extended anticipation and immediate, observation-grounded decisions. Drive-HWM couples a slow model’s multi-step, dynamics-aware predictions with a fast model’s next-frame and immediate-action predictions. Experiments report strong driving performance and ablations validating the design.

  • Problem

    Existing approaches separate future prediction from action generation or jointly predict them at one temporal scale, leaving the organization of extended anticipation and responsive, observation-grounded action prediction unresolved.

    Source: §I · p. 1

  • Method

    Drive-HWM couples a slow model’s multi-step future representations, learned through optical-flow supervision, with a fast model that jointly predicts the next frame and immediate action.

    Source: §I · p. 2

  • Results

    Experiments report strong driving performance, and ablations validate the hierarchical design, dynamics-aware future representations, and joint next-frame and action prediction.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework organizes extended scene prediction and observation-grounded action generation at complementary temporal scales.

    Source: §I · p. 2

    The architecture adds computational and memory overhead, and it does not explicitly model multimodal futures or predictive uncertainty, which may limit performance in ambiguous and rare driving scenarios.

    Source: §V · p. 12

Abstract

from arXiv · show

World models offer a promising paradigm for autonomous driving by predicting how traffic scenes may evolve and using such predictions to support action generation. However, existing approaches either separate future prediction from action generation or jointly predict them at the same temporal scale, making it difficult to simultaneously achieve long-horizon anticipation and responsive, observation-grounded decision making. We present Drive-HWM, a hierarchical slow--fast world modeling framework that organizes future representation prediction and action generation at complementary temporal scales. The slow world model predicts multi-step future representations to capture extended scene evolution. To explicitly model the abundant motion dynamics in driving environments, we introduce Dynamic-Aware Latents learned through optical-flow prediction. Guided by these future representations, the fast model uses a lightweight multimodal backbone and an autoregressive expert to jointly predict the next frame and the immediate action from the latest observation. Next-frame prediction encourages the fast model to capture imminent scene evolution, while one-step action generation allows decisions to be continuously updated as new observations arrive. Extensive experiments on NAVSIM v1 and v2 demonstrate the strong driving performance of Drive-HWM. Comprehensive ablation studies further validate the effectiveness of the hierarchical slow--fast design, dynamics-aware future representations, and joint next-frame and action prediction.

I. INTRODUCTION

Existing driving world models either separate future prediction from action generation or jointly predict them at a scale that struggles to balance long-horizon anticipation with immediate, observation-grounded control. Drive-HWM couples multi-step, dynamics-aware future representations with frequent next-frame and one-step action prediction; experiments and ablations support its design.

Motivation

  • Joint future-state and action prediction can be limited to short horizons, while rollout uncertainty and errors may degrade future representations and their decision-making utility.The introduction contrasts this with approaches that keep future prediction and action generation in separate components or training stages.Source: §I · p. 1
  • Future representations should capture extended scene evolution, while actions should remain high-frequency and grounded in the latest observation.The introduction notes that long action sequences may accumulate execution errors and deviate from intended behavior.Source: §I · pp. 1–2

Proposed approach

  • Optical-flow-trained Dynamic-Aware Latents encode motion; conditioned on these latents and the latest observation, the fast model jointly predicts the next frame and immediate action.Next-frame prediction encourages consistency between local scene understanding and action generation, while one-step actions can be revised as observations arrive.Source: §I · p. 2
  • Drive-HWM couples a slow model predicting multi-step future representations with a fast model generating observation-grounded actions.The framework is intended to combine long-horizon anticipation with responsive action generation at complementary temporal scales.Source: §I · p. 2

Evaluation

  • Experiments report strong driving performance, and ablations validate the hierarchical design, dynamics-aware future representations, and joint next-frame and action prediction.The experiments cover NAVSIM v1 and v2.Source: §I · p. 2

II. RELATED WORK

Related work spans action-producing vision–language models, future-predicting world models, and models that jointly predict world states and actions. Drive-HWM distinguishes explicit future representation prediction from action generation by their temporal roles, then introduces its slow–fast approach.

II. RELATED WORK

  • Vision–language–action models map multimodal observations and instructions to trajectories or controls, while predictive world models primarily output future observations or representations.The paper uses “world model” more narrowly for models that explicitly predict future observations, states, or representations.Source: §II · pp. 2–3
  • World–action models jointly connect future-state prediction with action generation, but common rollouts may overlook their different temporal requirements.The related-work discussion contrasts extended scene-evolution prediction with actions that respond promptly to the latest observation.Source: §II · p. 3
  • Drive-HWM assigns multi-step, dynamics-aware future prediction to a slow model and next-frame plus immediate-action prediction to a lightweight fast model.The design uses long-horizon anticipation to guide actions while retaining high-frequency observation grounding.Source: §II · p. 3

III. METHOD

  • The method section presents the driving formulation and slow–fast overview, then describes the slow and fast models and their training objectives.Source: §III · p. 3

A. Problem Formulation and Framework Overview

Drive-HWM separates long-horizon scene prediction from per-timestep action generation to address the different temporal demands of anticipation and responsive decisions. Slow predictions provide persistent context, while the fast model incorporates new observations and can correct decisions when scenes change unexpectedly.

A. Problem Formulation and Framework Overview

  • Driving is formulated as sequential decision-making under partial observability, with actions conditioned on interaction history because a single observation does not fully characterize the environment state.Observations may include multi-view images and auxiliary sensory inputs.Source: A. Problem Formulation and Framework… · p. 3
  • The slow model updates every N timesteps to predict K-step future latents, while the fast model generates actions at every timestep using the latest observation history and aligned slow prediction.The slow prediction is reused within each update interval as the fast model incorporates newly observed evidence.Source: A. Problem Formulation and Framework… · pp. 3–4
  • The fast model injects temporally aligned dynamic latents via FiLM into an Emu3-8B-based model that jointly predicts the next action and next-frame visual tokens.The slow model predicts multi-horizon optical flow under dynamic supervision to produce Dynamic-Aware Latents.Source: A. Problem Formulation and Framework… · p. 4, Figure 2 · p. 4
  • Reusing slow predictions provides persistent long-range context without preventing the action model from correcting decisions when unexpected scene changes occur.The fast model continuously incorporates newly observed evidence.Source: A. Problem Formulation and Framework… · p. 4

B. Slow World Model: Dynamic-Aware Future Prediction

The slow world model predicts multi-step, motion-aware future representations rather than appearance-heavy future RGB features. It learns Dynamic-Aware Latents through optical-flow prediction, supplying future-oriented context to the fast model.

Motivation

  • Future RGB features may emphasize appearance and spatial content, while driving-relevant ego-motion, object displacement, and temporal evolution occupy only a small part of the representation.Directly predicting future RGB also requires modeling appearance details that are difficult to extrapolate and not necessarily useful for action generation.Source: B. Slow World Model: Dynamic-Aware… · p. 4

Dynamic-Aware Future Prediction

  • Dynamic-Aware Latents are learned by predicting future optical flow, emphasizing motion while suppressing static appearance details such as texture, illumination, and color.The latent preserves contextual information for interpreting displacement, rather than passing raw flow fields to the fast model.Source: B. Slow World Model: Dynamic-Aware… · pp. 4–5

Multi-step optical-flow prediction

  • The slow model predicts scene dynamics at K future offsets from the same observed history, decoding each offset’s flow from its corresponding latent.This parallel formulation avoids recursive flow warping and direct propagation of early-offset prediction errors to later offsets.Source: B. Slow World Model: Dynamic-Aware… · pp. 4–5

Flow prediction objective

  • The flow objective learns long-range scene dynamics without photorealistic reconstruction and shapes latents that the fast model reuses for action generation.When flow validity or visibility masks are available, the loss is evaluated only at valid locations.Source: B. Slow World Model: Dynamic-Aware… · p. 5

C. Fast World Model: Dynamics-Conditioned Action Prediction

The fast world model uses current visual evidence together with a temporally aligned Dynamic-Aware Latent to generate observation-grounded actions at every timestep. It jointly trains action generation with next-frame visual prediction, while only action tokens need to be decoded at inference.

Dynamics-conditioned action prediction

  • At every timestep, the fast model generates actions from the latest visual evidence while using the slow model’s Dynamic-Aware Latent as predictive context.The slow model updates less frequently and cannot incorporate observations arriving after its latest update.Source: C. Fast World Model:… · p. 5

FiLM-based dynamic conditioning

  • FiLM injects the aligned Dynamic-Aware Latent into the Emu3 hidden representation, adapting feature channels while leaving the current observation as the primary input.The fast model combines up-to-date visual evidence with anticipated scene dynamics rather than treating the slow prediction as an explicit action command.Source: C. Fast World Model:… · p. 6

Autoregressive action expert

  • A driving-specific autoregressive expert specializes the multimodal representation for action generation and models dependencies among action tokens.The fast model is instantiated with Emu3-8B as its action-prediction backbone.Source: C. Fast World Model:… · pp. 5–6

Next-frame visual supervision

  • Joint next-frame prediction and action generation provide supervision for immediate scene transitions alongside control decisions.Next-frame prediction supplies spatially dense supervision, regularizes against shortcut learning, and encourages use of the Dynamic-Aware Latent; future image generation can be omitted at inference.Source: C. Fast World Model:… · p. 6

D. Training Objective

Drive-HWM is trained with long-horizon latent prediction, short-horizon latent prediction, and action-generation supervision. Training shares each slow prediction across fast-model steps and uses predicted rather than ground-truth future representations.

Training objective

  • Drive-HWM is trained using supervision for long-horizon latent prediction, short-horizon latent prediction, and action generation.The fast-world-model objective balances action prediction loss against one-step latent prediction.Source: D. Training Objective · p. 7

Action and visual objectives

  • For autoregressively tokenized actions, the action loss is token-level negative log-likelihood; continuous actions can instead use regression loss.The visual-prediction contribution is controlled by λimg.Source: D. Training Objective · p. 7

Training and inference schedule

  • Each slow prediction is shared across the N fast-model steps in its update interval, and training conditions the fast model on predicted rather than ground-truth future representations.At inference, the slow model runs only at selected timesteps while cached latents are supplied to the fast model at every intermediate timestep.Source: D. Training Objective · p. 7

IV. EXPERIMENT

The experiment section outlines quantitative and qualitative evaluation of Drive-HWM, followed by ablations of its components and design choices.

Experimental evaluation

  • The evaluation introduces datasets, metrics, and implementation details before comparing Drive-HWM quantitatively with state-of-the-art methods.The quantitative comparison assesses overall driving performance.Source: §IV · p. 7

Qualitative evaluation

  • Qualitative visualizations analyze long-horizon anticipation and action-generation capabilities of the slow–fast hierarchy..Source: §IV · p. 7

Ablation studies

  • Ablation studies examine component contributions and validate the design choices..Source: §IV · p. 7

A. Implementation Details

Drive-HWM is pretrained on nuPlan, fine-tuned on NAVSIM, and evaluated under both NAVSIM v1 and v2 protocols.

A. Implementation Details

  • NAVSIM v1 reports NC, DAC, TTC, comfort, and EP, summarizing them with PDMS; NAVSIM v2 adds five metrics and uses EPDMS.The passage provides the formulas for both aggregate scores.Source: A. Implementation Details · pp. 7–8

B. Quantitative Comparison

  • The model is jointly pretrained on nuPlan for 10K steps, then fine-tuned on NAVSIM for 6K steps using action-prediction loss.The slow-model update interval and prediction horizon are both eight timesteps.Source: A. Implementation Details · p. 7

A. Implementation Details

  • Table II compares methods on NAVSIM v2 using the extended evaluation metrics.Best and second-best results are marked, and Drive-HWM is highlighted in light blue.Source: Table II · p. 8
  • Training uses AdamW with an initial learning rate of 2 × 10^-4 and a cosine learning-rate schedule.Source: A. Implementation Details · p. 8

B. Quantitative Comparison

Drive-HWM achieves the strongest aggregate scores on both NAVSIM protocols and improves on a same-backbone single-world-model baseline. Its slow–fast design also adds long-horizon reasoning with modest latency overhead.

B. Quantitative Comparison

  • PDMS 93.3 on NAVSIM v1 and EPDMS 86.2 on NAVSIM v2 are the highest overall scores reported for Drive-HWM.It also records the best or tied-best result on four v1 metrics and seven v2 metrics.Source: B. Quantitative Comparison · pp. 8–9
  • Against DriveVLA-W0 [46], Drive-HWM improves all six listed NAVSIM v1 metrics, including PDMS by 0.3 points.The reported gains are 0.2 NC, 1.0 DAC, 1.0 TTC, 0.1 comfort, 0.2 EP, and 0.3 PDMS points.Source: B. Quantitative Comparison · p. 8
  • Against DriveVLA-W0 [46], Drive-HWM improves all ten NAVSIM v2 metrics, with gains of 1.4 DDC, 1.7 EP, 1.5 LK, and 27.5 EC points.Its EP of 88.1 is 0.3 points below the best result; LK and EC are competitive but not best.Source: B. Quantitative Comparison · pp. 8–9
  • Average latency is 84.8 ms versus 117.8 ms for DriveVLA-W0 [46], a reduction of approximately 28%, while PDMS rises from 93.0 to 93.3.The slow model takes 25.6 ms and runs once every eight steps, amortizing to 3.2 ms per step.Source: B. Quantitative Comparison · p. 9

C. Qualitative Comparison

Across four challenging scenarios, Drive-HWM’s trajectories more closely match human maneuver intent and lane-level geometry than the competing methods. The qualitative comparison attributes this to long-horizon anticipation combined with observation-grounded local correction.

C. Qualitative Comparison

  • In pedestrian avoidance, Drive-HWM follows the human trajectory while clearing the crossing pedestrian; at an intersection, its trajectory nearly overlaps the human turning path.The compared baselines deviate spatially or toward alternative paths.Source: C. Qualitative Comparison · p. 9
  • At the traffic-light intersection, Drive-HWM follows the intended turning corridor, while competing methods continue toward the intersection center and miss the route transition.Across all four scenarios, it shows stronger agreement with human behavior in maneuver intention, lane geometry, obstacle avoidance, and traffic-rule compliance.Source: C. Qualitative Comparison · p. 9
  • The slow model anticipates long-horizon scene evolution and preserves route and interaction intentions, while the fast model grounds that context in the latest observation for timely local corrections.The passage contrasts this complementary operation with short-sighted or directionally inconsistent trajectories from single-timescale baselines.Source: C. Qualitative Comparison · p. 9

D. Ablation Study

Ablations support the slow–fast decomposition and show that the slow and fast stages benefit from different prediction targets. They also identify effective backbone and conditioning choices, while highlighting interactive-scene failures and favorable human ratings.

Impact of the Hierarchical Slow–Fast World Modeling Architecture

  • The combined slow–fast model performs best; K=8 balances long-horizon foresight against prediction uncertainty, unlike fast-only reactivity or slow-only delayed adaptation.Fast-only lowers NC, DAC, and PDMS from 99.5, 98.4, and 93.3 to 99.3, 97.4, and 93.0, respectively.Source: D. Ablation Study · p. 10

Impact of the Foundation World Model Backbone Choice

  • V-JEPA performs best among the slow-model backbones, while Emu3 performs best among the evaluated fast-model backbones.The authors attribute V-JEPA’s suitability to its focus on temporally meaningful dynamics without reconstructing low-level visual detail.Source: D. Ablation Study · p. 10

Impact of the Prediction Target of the Slow Model

  • Optical-flow supervision achieves the highest NC, DAC, and PDMS, while its dense displacement cues capture scene dynamics more explicitly than BEV, depth, or RGB targets.The slow-model comparison holds the remaining configurations unchanged; optical flow also provides more motion-discriminative cues than BEV and depth representations.Source: Figure 4 · p. 10, D. Ablation Study · p. 10, Figure 5 · p. 10

Motion-Aware Representation Probing

  • Optical-flow latents exceed the second-best RGB latents by 6.9 points on future ego-motion prediction and 6.7 points on motion consistency.Frozen representations are evaluated with lightweight linear probes on both tasks.Source: D. Ablation Study · p. 11, Table VI · p. 11

Impact of Slow-to-Fast Conditioning Strategies

  • FiLM achieves the best overall conditioning performance, reaching 99.5 NC, 98.4 DAC, and 93.3 PDMS.It converts the slow-model latent into feature-wise affine parameters that modulate the fast model’s intermediate features.Source: D. Ablation Study · p. 11, Table VII · p. 11

Impact of the Next-frame Visual Supervision in the Fast Model

  • Next-frame RGB supervision achieves the best overall fast-model performance, while optical flow better suits the slow model’s dynamics-oriented representations.RGB retains local semantic and appearance information, including lane markings, signals, layouts, and nearby-agent states.Source: D. Ablation Study · p. 11, Table VIII · p. 11

Failure Analysis

  • At an unsignalized Y-intersection, the predicted trajectory fails to yield sufficiently as an approaching vehicle enters the ego path, creating a potential collision.The ground-truth-trajectory-conditioned observations remain consistent with the safe maneuver in this example.Source: D. Ablation Study · p. 11, Figure 6 · p. 12

User Study

  • In 50 paired scenarios rated by 20 people, Drive-HWM is preferred in 68% of comparisons versus 21% for DriveVLA-W0, with 11% showing no clear preference.Drive-HWM also receives higher mean ratings for safety (4.58 vs. 4.21), comfort (4.29 vs. 4.06), human-likeness (4.44 vs. 4.05), and route correctness (4.51 vs. 4.17).Source: D. Ablation Study · p. 12, Figure 7 · p. 12

V. CONCLUSION

The conclusion presents Drive-HWM as a slow–fast framework that couples multi-step prediction with observation-grounded action generation. It reports effectiveness on NAVSIM v1 and v2, while noting computational overhead and unmodeled multimodal futures and uncertainty.

Conclusion

  • Drive-HWM couples multi-step future-representation prediction with immediate action generation grounded in the latest observation.The slow model learns motion- and interaction-aware latents through optical-flow supervision; the fast model jointly predicts the next RGB frame and immediate action.Source: §V · p. 12
  • Experiments on NAVSIM v1 and v2 demonstrate the framework’s effectiveness, and ablations validate dynamics-aware prediction, next-frame supervision, and slow-to-fast conditioning.Source: §V · p. 12

Limitations

  • The architecture adds computational and memory overhead, and the model does not explicitly capture multimodal futures or predictive uncertainty.The authors identify these gaps as possible constraints in ambiguous and rare driving scenarios.Source: §V · p. 12
Loading 2609.03572v1…