PaperlayerView exact revision on arXiv

Source-linked AI summary

PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving

Jinchang Xu, Hongda Yu, Fengwei Dong, Wenhui Huang, Xi Wei, Yongzhi Liu, Sunan Zhang, Jirao Wang, Chen Lv, Bingbing Li, Guodong Yin, Weichao Zhuang

arXiv:2610.11382v1cs.ROcs.LG

TL;DR

Existing driving world models less often ask which future representation is most useful for planning. PlanWAM shapes a future latent with planning objectives and distills it from privileged future observations into a history-only prediction for planning. It achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and 38.7 HD-Score in zero-shot closed-loop HUGSIM evaluation.

  • Problem

    Existing methods focus on predicting future information and using it, but less often ask which future representation best serves planning.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    PlanWAM shapes future latents with trajectory-planning objectives and trains a history-only prior to predict them as planning context.

    Source: §1 · p. 2

  • Results

    93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and 38.7 HD-Score in zero-shot closed-loop HUGSIM evaluation demonstrate PlanWAM’s results across both settings.

    Source: §5 · p. 10

  • Takeaways & Limitations

    PlanWAM provides deployable foresight for trajectory generation and selection through planning-shaped future representations.

    Source: §5 · p. 10

Abstract

from arXiv · show

World models in end-to-end autonomous driving predict future scene evolution to provide foresight for trajectory planning. Existing methods mainly study how to predict the future and how to use it, but less often ask which future representation is actually most useful for planning. To this end, we propose PlanWAM, a Planning-Shaped World Action Model. The key idea is to let the planning task shape the future-state representation, so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning. Specifically, we first use a Temporal Register Pyramid to compress multi-frame historical information in a recency-aware manner, learning a compact history representation oriented toward future reasoning and planning. We then introduce a privileged future posterior branch that observes ground-truth future frames, and shape its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation. Hindsight-to-Foresight Distillation trains a prior branch that depends only on history to predict this future latent representation. The predicted future latent representation serves as planning context and guides trajectory generation and selection. PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations. Extensive experiments further demonstrate that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.

1 INTRODUCTION

Reliable driving needs history and possible future scene evolution, but existing world models less often ask which future representation best serves planning. PlanWAM makes planning objectives shape the future latent and predicts it from history for trajectory planning.

1 INTRODUCTION

  • Existing methods progressed from pixel reconstruction to latent prediction and future-conditioned planning, yet less often study which future representation is most useful for planning.Representations based on reconstruction or generic image features may not retain information relevant to trajectory decisions.Source: §1 · pp. 1–2
  • PlanWAM frames future modeling as planning-oriented predictive representation learning, shaping future information around trajectory-planning objectives rather than scene reconstruction.The framework links planning objectives, future prediction, and which historical information is retained.Source: §1 · p. 2
  • PlanWAM compresses multi-frame, multi-view history, shapes a future latent using privileged ground-truth future observations and planning objectives, then distills a history-only prior to guide trajectory generation and selection.The privileged posterior is used during training; the predicted future latent supplies planning context at inference.Source: §1 · pp. 2–3
  • 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and 38.7 HD-Score in zero-shot closed-loop HUGSIM evaluation demonstrate PlanWAM’s performance across open- and closed-loop settings.The introduction also reports that analyses find improved trajectory generation and selection and fewer safety-critical failures, including collisions and low-TTC events.Source: §1 · p. 3

2 RELATED WORK

The related work describes end-to-end planners and driving world models that increasingly use generative, semantic, or future-state information. PlanWAM’s distinction is to shape the future representation according to planning objectives.

2.1 END-TO-END AUTONOMOUS DRIVING

  • End-to-end driving methods progress from structured multitask models to generative trajectory planners and vision-language systems, while still relying mainly on current or historical observations.PlanWAM instead predicts a future representation to guide trajectory generation and evaluation.Source: §2.1 · p. 3

2.2 WORLD MODELS FOR AUTONOMOUS DRIVING

  • Driving world models either generate future scenes or learn dynamic representations for driving tasks, but prior work less often asks what future information planning needs.PlanWAM directly shapes the future latent with planning objectives.Source: §2.2 · p. 3

3 METHOD

PlanWAM makes trajectory-planning objectives shape the future representation, then predicts that representation from history to guide planning. Its method combines recency-aware history compression, a privileged future posterior, and a history-only prior trained to predict the posterior’s planning-shaped latent.

3.1 PROBLEM FORMULATION AND FRAMEWORK OVERVIEW

  • PlanWAM inserts a future-state representation between history and planning, shaping it with planning objectives and predicting it from history for trajectory generation and selection.The privileged posterior uses ground-truth future observations during training; the prior supplies predicted future latents at inference.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 RECENCY-AWARE HISTORY REPRESENTATION

  • The Temporal Register Pyramid compresses historical scene registers by temporal level, assigning more queries to recent history and compressing distant history more strongly.The resulting history representation is jointly optimized for future-latent distillation and trajectory planning, without an additional historical reconstruction objective.Source: §3.2 · p. 5

3.3 PLANNING-SHAPED FUTURE REPRESENTATION

  • Planning losses back-propagate through the planner and scorer to shape the privileged posterior’s future latent, prioritizing information that affects trajectory generation and selection.The posterior uses ground-truth future observations alongside the compact history representation.Source: §3.3 · pp. 5–6

3.4 HINDSIGHT-TO-FORESIGHT PREDICTION AND PLANNING

  • Hindsight-to-Foresight Distillation aligns the history-only prior’s prediction with the posterior’s planning-shaped future latent.The predicted latent is combined with history as foresight context for the trajectory planner and scorer.Source: §3.4 · p. 6

3.5 TRAINING AND INFERENCE

  • Training first optimizes the privileged posterior, then freezes it and fine-tunes the prior; inference removes the posterior and relies only on history.At inference, the prior predicts the future representation, which is combined with the history representation for planning.Source: §3.5 · p. 6

4 EXPERIMENTS

PlanWAM is evaluated on NAVSIM open-loop benchmarks and zero-shot HUGSIM closed-loop driving, with ablations examining its components and failure reduction. The reported results favor planning-shaped foresight across these evaluations, while DAC failures slightly deteriorate and proposal-collapse issues remain.

4.1 EXPERIMENTAL SETUP

  • Models train on NAVSIM navtrain and are tested on navtest; PlanWAM uses four camera views across four historical frames and is evaluated zero-shot on HUGSIM.NAVSIM-v1 uses PDMS, NAVSIM-v2 uses EPDMS, and HUGSIM reports HD-Score and Route Completion.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 93.8 PDMS on NAVSIM-v1 improves on DrivoR by 0.7 points and DriveFuture by 3.1; 90.9 EPDMS on NAVSIM-v2 exceeds the strongest learned result by 1.0 point and Human Agent by 0.6.PlanWAM also obtains the best learned NC of 99.0 and TTC of 98.5 on NAVSIM-v2.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8
  • 53.7 average RC and 38.7 HD-Score on HUGSIM outperform the strongest baselines by 7.8 and 9.8 points, respectively, without simulator-specific fine-tuning.On Hard / Extreme, PlanWAM reaches 51.9 / 37.2 RC and 35.6 / 22.0 HD-Score.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • NAVSIM examples show safer interaction-aware trajectories than DrivoR, while HUGSIM rollouts transfer across KITTI-360, PandaSet, nuScenes, and Waymo.The examples qualitatively support the quantitative gains.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 ABLATION STUDIES

  • Adding TRP, H2F, and FCP in sequence raises PDMS from 93.1 to 93.4, 93.5, and 93.8, respectively.FCP uses the predicted future latent as context for trajectory generation and selection.Source: §4.3 · p. 8, Table 4 · p. 9
  • Planning-shaped future latents reach 93.8 PDMS versus 93.5 for generic future-feature supervision; conditioning both trajectory generation and selection also reaches 93.8.Using the latent only for selection reaches 93.7 PDMS, while auxiliary supervision or generation-only conditioning brings little gain.Source: §4.3 · p. 9, Table 5 · p. 9, Table 6 · p. 9
  • Under a 128-token budget, recency-aware allocation reaches 93.8 PDMS; latent distillation raises similarity from 0.22 to 0.92 and PDMS to 93.7, while posterior initialization yields 0.93 similarity and 93.8 PDMS.Increasing the token count to 256 yields only 93.6 PDMS; allocation matters more than enlarging the representation.Source: §4.3 · p. 9, Table 7 · p. 9, Table 8 · p. 9

4.4 ANALYSIS OF FAILURE REDUCTION

  • On NAVSIM-v1 PDMS=0 failures, NC and TTC failures drop by 62.3% and 52.4%, but DAC failures deteriorate slightly; total failures fall from 314 to 237, mainly through 87 fewer trajectory-selection errors.Proposal quality improves somewhat, while proposal-collapse issues remain.Source: §4.4 · p. 9, Figure 5 · p. 10

5 CONCLUSION

PlanWAM frames future modeling as planning-oriented predictive representation learning, distilling planning-shaped future latents from privileged future observations to a history-only prior. It reports strong results on NAVSIM and zero-shot closed-loop HUGSIM.

5 CONCLUSION

  • PlanWAM reshapes future modeling as planning-oriented representation learning and achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 and 38.7 HD-Score in zero-shot HUGSIM evaluation.Its history-only prior predicts future latents shaped by trajectory-planning objectives for trajectory generation and selection.Source: §5 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement describes an anonymized repository and identifies the public benchmarks used. It also maps supplementary appendices to implementation, training, evaluation, ablation, and visualization details.

REPRODUCIBILITY STATEMENT

  • An anonymized repository provides source code and supplementary material, and experiments use public NAVSIM-v1, NAVSIM-v2, and HUGSIM benchmarks.Appendices cover architecture and data flow, training objectives, scoring protocols, additional ablations, and qualitative results.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The authors report using generative AI for language support, formatting, and related-work discovery, while excluding it from research design and execution. The appendix contents list supplementary implementation, training, evaluation, ablation, and visualization material.

AI USE STATEMENT

  • Generative AI assisted translation, writing polish, formatting, and related-work discovery; authors verified suggested literature, reviewed the text, and report no AI use in research design or execution.They take responsibility for the final content.Source: AI USE STATEMENT · p. 10

The contents of this appendix are as follows:

A DETAILED ARCHITECTURE AND DATA FLOW

The architecture predicts a future latent from history, uses history and that latent to generate candidate trajectories, then scores candidates to select one. The appendix details token bookkeeping and the transformer roles in this flow.

A DETAILED ARCHITECTURE AND DATA FLOW

A.1 TOKEN BOOKKEEPING AND END-TO-END FLOW

  • The reported configuration compresses 256 historical registers into 128 history tokens and retains 256 future slots, yielding a 384-token planning memory.The prior and posterior use matching time–camera–register slot layouts.Source: §A.1 · p. 14, Table 9 · p. 15

A.2 WORLD-MODEL TRANSFORMER

  • The shared world-model block cross-attends posterior or prior queries to history context to produce the future latent; inference uses the prior queries.The prior queries follow the future-time, camera, and register-slot order of the posterior target.Source: §A.2 · p. 15

A.3 TRAJECTORY PLANNER

  • The planner transformer refines 64 candidate tokens against history–latent memory, then an MLP decodes each into eight poses.Each pose contains x, y, and ψ.Source: §A.3 · p. 15

A.4 TRAJECTORY SCORER AND SELECTION

  • The scorer cross-attends embedded candidates to the same memory, adds ego status, predicts six subscores, and ranks candidates for selection.The geometry supplied to the scorer is detached, so its scoring loss does not update the trajectory decoder through that path.Source: §A.4 · p. 15

A.5 ARCHITECTURE SUMMARY

  • Table 10 summarizes transformer configurations, including attention-head counts and FFN activation settings.The table caption states that FFNs use GELU unless an output head is explicitly described as an MLP.Source: Table 10 · p. 16

B.1 TWO-STAGE TRAINING PROTOCOL

Training first learns planning modules with a privileged future posterior, then distills its target into a history-only prior. Trajectory generation uses best-of-K supervision, while scoring and latent losses train proposal selection and foresight alignment.

B.1 TWO-STAGE TRAINING PROTOCOL

  • Stage 1 learns the privileged posterior and shared planning modules from history and future observations; Stage 2 freezes a separate posterior teacher and trains the history-only prior, with future images withheld from the student and removed at test time.The teacher output is detached before latent alignment.Source: §B.1 · p. 16, Table 11 · p. 16

B.2 TRAJECTORY GENERATION OBJECTIVE

  • Only the closest of K candidate trajectories receives direct regression supervision, while learned candidate tokens and self-attention preserve a multimodal proposal set.The loss compares candidate trajectories with the demonstrated ego trajectory.Source: §B.2 · p. 16

B.3 TRAJECTORY SCORING OBJECTIVE

  • Candidate scores are trained against official proposal-simulator targets for six planning subscores, using binary cross-entropy on predicted logits.Intermediate NC or DDC targets of 0.5 become conservative negatives, and undefined TTC targets are masked; the objective trains selection from planning-quality signals rather than trajectory distance.Source: §B.3 · pp. 16–17

B.4 LATENT REGULARIZATION AND HINDSIGHT-TO-FORESIGHT DISTILLATION

  • Latent regularization penalizes vanishing token norms and redundant slots, while token-aligned teacher distillation trains the history-only prior toward the privileged target.A frozen teacher and stop-gradient keep Stage 2 from moving the target toward the student; the latent loss is computed in FP32 under mixed precision.Source: §B.4 · p. 17

C.1 BENCHMARK SUMMARY

The evaluation combines NAVSIM proposal scoring with HUGSIM closed-loop simulation. The section defines how each benchmark scores safety, progress, comfort, route completion, and aggregate results.

C.1 BENCHMARK SUMMARY

  • NAVSIM uses deterministic, non-reactive rollouts of recorded actor trajectories, while HUGSIM provides closed-loop observations and interacting actors to probe compounding errors and distribution shift.The NAVSIM-trained model is evaluated in HUGSIM without simulator-specific adaptation.Source: §C.1 · pp. 17–18, Table 12 · p. 17

C.2 NAVSIM-V1: PDM SCORE

  • NAVSIM-v1 PDMS combines NC and DAC as multiplicative safety gates with weighted TTC, EP, and comfort quality terms, then averages scene-level scores.Component means need not reproduce aggregate PDMS when substituted into the score formula.Source: §C.2 · p. 18

C.3 NAVSIM-V2: EXTENDED PDM SCORE

  • NAVSIM-v2 adds DDC, TLC, lane keeping, history comfort, and extended comfort; its evaluator applies a human-penalty filter before scene-level aggregation.DDC, TLC, NC, and DAC are multiplicative penalties; EP and TTC have weights of 5, while LK, HC, and EC have weights of 2.Source: §C.3 · p. 18

C.4 HUGSIM: ROUTE COMPLETION AND HD-SCORE

  • HUGSIM’s HD-Score combines per-timestamp driving scores with route completion; each difficulty mean is weighted by its scenario count in the overall average.The public split contains 436 scenarios, and the average is not an equal mean of the four difficulty means.Source: §C.4 · pp. 18–19, Table 13 · p. 18

D ADDITIONAL CONTROLLED ABLATIONS

The ablations examine posterior query choices, future duration, distillation weight, and trainable modules on NAVSIM-v1 navtest. Their results show that the reported settings and downstream training choices matter for planning scores.

D ADDITIONAL CONTROLLED ABLATIONS

  • The ablations use NAVSIM-v1 navtest, and the highlighted row marks the reported setting.The posterior-query comparison reports posterior PDMS rather than deployed-prior PDMS.Source: D ADDITIONAL CONTROLLED ABLATIONS · p. 19

D.1 FUTURE REGISTERS IN THE POSTERIOR

  • Direct future-register queries reach 95.0 posterior PDMS, compared with 94.6 when registers serve as keys and values and 94.2 after MLP compression to 128 query tokens.These are posterior-only scores, not scores for the deployed prior.Source: Table 14 · p. 19, §D.1 · p. 19

D.2 OBSERVED-FUTURE DURATION

  • Observing future times {1, 2, 3, 4} seconds reaches 93.8 PDMS, versus 93.5 with only the 1 s future.The longer horizon uses 256 future tokens rather than 64.Source: Table 15 · p. 19, §D.2 · p. 19

D.3 LATENT-DISTILLATION WEIGHT

  • A latent-distillation weight of 0.05 yields 0.93 similarity and the highest reported PDMS, 93.8; at 0.10, similarity rises to 0.95 while PDMS falls to 93.7.The 0.01 setting reports similarity 0.88 and PDMS 93.6; without distillation, the values are 0.46 and 93.5.Source: Table 16 · p. 20, §D.3 · p. 20

D.4 DISTILLATION LOSS AND TRAINABLE MODULES

  • Training the remaining modules with the ViT base frozen and adapted by LoRA reaches 93.8 PDMS, versus 93.0–93.1 with the posterior planner and scorer held fixed.Training the prior world model with the scorer reaches 93.4, and training the planner as well reaches 93.5.Source: Table 17 · p. 20, §D.4 · p. 20
  • With only the prior world model trainable, latent alignment alone gives 93.0 PDMS; adding planning and scoring losses gives 93.1, while alignment alone leaves planning score low.Table 17 reports latent similarities of 0.96 and 0.94, respectively.Source: Table 17 · p. 20, §D.4 · p. 20

E ADDITIONAL QUALITATIVE RESULTS

The qualitative results offer complementary views of candidate generation and selection in NAVSIM and plan stability across closed-loop HUGSIM rollouts, while leaving quantitative conclusions to Sec. 4.2.

E ADDITIONAL QUALITATIVE RESULTS

  • The figures provide complementary qualitative views: NAVSIM shows candidate generation and selection in a frame, while HUGSIM rollouts show plans across changing observations.The examples have no metric annotations; quantitative conclusions remain in Sec. 4.2.Source: E ADDITIONAL QUALITATIVE RESULTS · p. 20

E.1 NAVSIM CANDIDATE GENERATION AND SELECTION

  • NAVSIM examples show all 64 candidates in gray, the selected plan in red, and the recorded trajectory in green across varied road and traffic scenes.Rows pair front-view images with ground-truth trajectories in the BEV scene and PlanWAM output.Source: §E.1 · pp. 20–21, Figure 7 · p. 21, Figure 8 · p. 22

E.2 ZERO-SHOT CLOSED-LOOP ROLLOUTS ON HUGSIM

  • HUGSIM figures show two six-step rollouts for each of four source domains, with the same model and no HUGSIM-specific training.Overlaid corridors, centerlines, and waypoints visualize the current plan.Source: §E.2 · p. 25, Figure 11 · p. 25, Figure 12 · p. 26

Related papers

Loading 2610.11382v1…