PaperlayerView exact revision on arXiv

Source-linked AI summary

SV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous Driving

Jinyang Wang, Shiwei Li, Junjian Wang, Zhiqiang Deng, Jianbin Gao, Yihang Zhao, Liu Liu, Yongjia Zhao, Jinlong Chen, Huirui Xu, Yifeng Pan, Kangwei Liu, Fan Ren, Ji Tao, Minghao Yang

arXiv:2609.03602v1cs.CVcs.RO

TL;DR

Many recent WM planners use single-front-view inputs at inference to reduce prediction and generation cost, despite side and rear context being important in safety-critical driving maneuvers. SV-WAM retains six-camera conditioning and uses future-video prediction as training-time supervision, enabling action-only inference; it reports state-of-the-art NAVSIMv2 planning and competitive zero-shot nuScenes transfer.

  • Problem

    Recent WM planners commonly use single-front-view inputs to reduce inference cost, although side and rear context is critical for lane changes, merges, turns, and nearby-vehicle interactions.

    Source: Introduction · p. 1

  • Method

    SV-WAM jointly trains future-action and surround-view future-video latents with an action-centered causal mask, then removes the video branch for full-surround-view action-only inference.

    Source: Introduction · p. 2, Conclusion · pp. 7–8

  • Results

    91.0 EPDMS on NAVSIMv2 accompanies state-of-the-art planning performance, while SV-WAM demonstrates competitive zero-shot transfer on nuScenes with low latency.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    Future-video co-training can serve as supervision for full-surround-view action planning without requiring video denoising and decoding at deployment.

    Source: Efficient Action-Only Inference · p. 5, Conclusion · pp. 7–8

    Its approximately 5B-parameter backbone remains relatively large for resource-constrained on-board platforms.

    Source: Limitations and Future Work · p. 14

Abstract

from arXiv · show

World models (WMs) have demonstrated strong potential for end-to-end autonomous driving by learning predictive representations of future scene dynamics. However, generating future videos during inference introduces substantial computational overhead, leading many recent driving WMs to adopt a single front camera as input for efficient deployment. This design restricts spatial coverage in safety-critical maneuvers such as lane changes, merges, and turns. To address this limitation, we propose SV-WAM, a surround-view world-action model (WAM) that preserves full six-camera observations while maintaining efficient inference. SV-WAM leverages future-video prediction as dense training supervision for action learning within a shared generative model, rather than as an inference-time output. At the core of this design is an action-centered causal mask that prevents action tokens from attending to future-video tokens during joint action-video denoising. Consequently, the video branch can be discarded at deployment, enabling efficient action-only planning. Furthermore, we introduce a differentiable drivable-area compliance regularizer that penalizes vehicle-footprint corners approaching or crossing drivable boundaries, improving planning safety and boundary awareness. Extensive experiments on the closed-loop NAVSIMv2 benchmark and the open-loop nuScenes benchmark demonstrate that SV-WAM achieves state-of-the-art planning performance with low inference latency and competitive zero-shot transfer capability.

Introduction

World models provide future-aware representations for end-to-end driving, but inference-time video generation makes retaining surround-view input computationally challenging. SV-WAM uses future-video training supervision to support action-only planning with six-camera context, and reports strong NAVSIMv2 and nuScenes results.

Introduction

  • World models forecast driving-scene evolution to provide future-aware representations beyond sparse action imitation.Such predictive supervision is valuable for closed-loop planning under constraints from surrounding agents and road structure.Source: Introduction · p. 1
  • Recent world-model planners often use a single front camera to reduce inference cost, despite the importance of side and rear context for maneuvers.Six-view future-video rollout would substantially increase visual tokens and inference latency.Source: Introduction · p. 1
  • SV-WAM jointly denoises future actions and six-view future-video latents, while an action-centered causal mask blocks action tokens from attending to future-video tokens.This allows the video branch to be removed at deployment while retaining full six-view conditioning.Source: Introduction · p. 2
  • 91.0 EPDMS on NAVSIMv2 and 342 ms latency on one NVIDIA H20 GPU; SV-WAM outperforms recent world-model-based planners.The latency runs from preprocessed inputs to trajectory output.Source: Introduction · p. 2
  • On nuScenes, SV-WAM transfers zero-shot with 0.89 m average L2 error and 0.16% average collision rate.The paper reports competitive zero-shot transfer on nuScenes.Source: Introduction · p. 2

Related Work

SV-WAM combines surround-view conditioning with training-time future-video supervision, while structuring action prediction so future-video generation is not required at inference. Its method also adds a differentiable drivable-area penalty and action-only inference.

Related Work

  • Recent driving world models commonly use front-view inputs, while some surround-view latent models retain pretrained geometric encoders and world-action models can incur video-generation latency at inference.Structured planners also often use task-specific intermediate representations and lack explicit future-scene modeling.Source: Related Work · p. 2

Input Tokenization

  • Six ordered camera views are concatenated by width and encoded into video latents split into historical conditioning and future-video targets, while action increments are projected into tokens.The future-video latents are used only during training.Source: Input Tokenization · p. 3

Surround-View World-Action Model

  • Joint flow-matching training denoises future actions and video latents, while the action-centered mask blocks action tokens from attending to future-video tokens.Future-video tokens can attend to action tokens, allowing video supervision without making action prediction depend on future-video tokens.Source: Surround-View World-Action Model · p. 3

Drivable Area Compliance Regularization

  • The drivable-area regularizer evaluates signed distances at rolled-out vehicle-footprint corners and uses log-mean-exp aggregation to emphasize high-risk boundary violations.Positive signed distance indicates a point inside the drivable area; negative distance indicates an off-road location.Source: Drivable Area Compliance Regularization · pp. 4–5

Efficient Action-Only Inference

  • At inference, SV-WAM omits future-video tokens and output, denoises actions iteratively, and reuses the cached conditioning prefix while retaining reference-video and state conditioning.The video branch is skipped during deployment.Source: Efficient Action-Only Inference · p. 5

Experiments

Experiments evaluate closed-loop planning on NAVSIMv2 and zero-shot transfer on nuScenes, alongside component, camera-coverage, inference-step, and latency analyses. SV-WAM leads the reported NAVSIMv2 results, transfers without target-domain fine-tuning, and achieves low trajectory-output latency.

Experimental Setup

  • Closed-loop evaluation uses NAVSIMv2 EPDMS, while nuScenes transfer is assessed without target-domain fine-tuning using L2 trajectory error and collision rate.Reported nuScenes metrics include values at 1s, 2s, 3s, and their averages.Source: Experimental Setup · p. 5, Main Results · p. 5

Main Results

  • 91.0 EPDMS gives SV-WAM the best performance on the NAVSIMv2 navtest split.The comparison uses the closed-loop metric suite with human penalty enabled.Source: Main Results · p. 5, Table 1 · p. 6
  • 0.89 m average L2 error and 0.16% collision rate are the lowest among zero-shot methods on nuScenes.SV-WAM remains competitive with several in-domain fine-tuned planners.Source: Main Results · p. 5, Table 2 · p. 6

Ablation Studies

  • Future-video co-training raises EPDMS from 83.1 to 87.7; DAC regularization raises EPDMS to 90.1 and DAC from 95.6 to 98.6; fine-tuning reaches 91.0.The ablation compares training components on NAVSIMv2 navtest.Source: Ablation Studies · pp. 5–6, Table 3 · p. 6
  • With the causal mask, two-step EPDMS is 91.0 versus 90.9 and 90.8 at 5 and 10 steps; removing the video branch leaves planning unchanged.Without the mask, increasing inference steps from 2 to 10 improves EPDMS from 87.3 to 88.2.Source: Ablation Studies · p. 6, Table 4 · p. 7
  • Increasing camera coverage consistently improves EPDMS, with six views achieving the best final score and improving DDC, LK, and EC.Side and rear views provide context unavailable from a single front camera.Source: Ablation Studies · pp. 6–7

Qualitative Analysis

  • In a challenging turn, SV-WAM’s predicted trajectory follows ground truth more closely and aligns better with road geometry than DriveLaW and DriveVLA-W0.The qualitative comparison includes current six-camera observations and corresponding BEV visualizations.Source: Figure 5 · p. 7, Qualitative Analysis · p. 7

Efficiency Analysis

  • 341.6 ± 2.1 ms is the lowest trajectory-output latency among compared world-model planners, despite using six cameras.Removing the causal mask raises latency to 848.0 ± 1.8 ms; optional video decoding is excluded from both timings.Source: Efficiency Analysis · p. 7, Table 5 · p. 7

Conclusion

SV-WAM retains six-camera context for planning while using future-video prediction as training-time supervision rather than an inference-time output. Its action-centered causal mask supports action-only inference, and a differentiable drivable-area regularizer is evaluated on NAVSIMv2 and nuScenes.

Conclusion

  • The action-centered causal mask enables joint action-video denoising while blocking action tokens from attending to future-video tokens.The video branch can therefore be removed at inference, yielding action-only planning with full surround-view coverage.Source: Conclusion · pp. 7–8
  • A differentiable drivable-area compliance regularizer is introduced to improve road compliance.Source: Conclusion · p. 8
  • Experiments on NAVSIMv2 and nuScenes validate the proposed design’s effectiveness and efficiency.Source: Conclusion · p. 8

Implementation Details

The implementation encodes six-camera histories and ego states alongside action tokens in a shared diffusion-transformer, and trains with future-video supervision. At deployment, it removes the video branch and denoises action tokens using cached observation states.

Implementation Details

  • The Wan2.2-5B diffusion-transformer encodes six-view mosaics into video tokens, while lightweight MLPs tokenize ego states and actions.The backbone retains 30 blocks and approximately 5B parameters; a linear head maps action outputs to flow velocity.Source: Implementation Details · p. 10
  • Inputs use four history frames from six cameras, with ego-state continuous components normalized to [−1, 1] and command entries kept as one-hot values.The six camera views are resized to 448×224; state statistics come from the official NAVSIM training split.Source: Implementation Details · p. 10
  • The model predicts 12 future action tokens using fixed action bounds, then denormalizes and accumulates them in SE(2).The first eight tokens form the standard 4 s NAVSIM trajectory; training also denoises 12 future six-view video latents.Source: Implementation Details · p. 10
  • Training uses 10k iterations followed by 1k fine-tuning iterations, with AdamW and rectified-flow training on 16 H800 GPUs.Each stage takes roughly one day; the flow-matching loss weights action and video equally.Source: Implementation Details · p. 10
  • The drivable-area penalty is computed on vehicle-corner samples along an LQR rollout and backpropagates through the rollout to predicted actions.The implementation reuses NAVSIM’s map semantics while using differentiable PyTorch rollout and footprint queries.Source: Implementation Details · p. 10
  • At deployment, the video branch is removed, observation key/value states are cached, and 12 action tokens are denoised in two flow steps.With the same inference steps, masked-model trajectories are identical up to numerical precision whether or not future-video tokens are present.Source: Implementation Details · pp. 10–11

Dataset and Metric Details

The evaluation uses NAVSIM splits for closed-loop planning and nuScenes for open-loop, cross-dataset evaluation without target-domain fine-tuning. NAVSIMv2 extends evaluation with pseudo-simulation and safety-oriented metrics, while nuScenes reports displacement and collision measures.

NAVSIM

  • NAVSIM is built on OpenScene driving logs; experiments train on trainval and report main results on the NAVSIMv2 navtest split.Six views are resized and concatenated into a mosaic, using four history frames sampled at 2 Hz.Source: NAVSIM · p. 11
  • NAVSIMv2 uses pseudo-simulation to evaluate shifted future states, while navhard adds a two-stage protocol and human-behavior penalty for harder reactive scenarios.Source: NAVSIM · p. 11
  • EPDMS combines multiplicative safety terms with weighted quality scores; a single violation of a safety gate can zero the scenario score.The safety terms include collision, drivable-area, driving-direction, and traffic-light compliance; quality terms include progress, time-to-collision, lane keeping, and comfort.Source: NAVSIM · p. 11
  • NAVSIMv1 results use PDMS, the predecessor to EPDMS; PDMS omits the additional v2 terms and does not split comfort into history and extended comfort.Source: NAVSIM · p. 11

nuScenes

  • nuScenes has 1,000 Boston and Singapore driving scenes and is used only for evaluation, with no training or fine-tuning on that dataset.Zero-shot planners are evaluated directly on the validation split using four recent keyframes from all six cameras.Source: nuScenes · p. 11
  • The nuScenes open-loop evaluation reports L2 displacement error and collision rate at 1 s, 2 s, and 3 s, plus averages across horizons.Source: nuScenes · p. 12

Additional Quantitative Results

On additional benchmarks, SV-WAM ties EponaV2 on the harder NAVSIMv2 navhard split, outperforms listed baselines on NAVSIMv1, and shows competitive zero-shot nuScenes results with an evaluation caveat.

NAVSIMv1

NAVSIMv2 navhard

  • 36.1 EPDMS ties EponaV2 and ranks behind RAP (39.6) on NAVSIMv2 navhard; SV-WAM also records 94.9 first-stage DAC.The split includes a two-stage evaluation and human-behavior penalty; EC scores are 84.4 and 80.1 across the two stages.Source: Additional Quantitative Results · p. 12

nuScenes

  • 0.89 m average L2 error and 0.16% collision rate are competitive with and on average better than several nuScenes-fine-tuned baselines despite zero-shot evaluation.The paper treats this comparison as a cross-dataset sanity check because open-loop nuScenes metrics are sensitive to ego-status shortcuts.Source: Additional Quantitative Results · p. 12

Additional Ablations

Ablations show stable performance across tested action-token counts and training settings, with 12 action tokens giving the highest EPDMS in the token-count analysis. Small drivable-area regularization weights and tested fine-tuning batch sizes also preserve performance, while the future-video loss-weight changes have little effect within the tested range.

Number of Action Tokens

  • Twelve action tokens yield the highest EPDMS in the sensitivity analysis, while performance remains stable across the tested token counts.Eight tokens shorten the prediction horizon and slightly reduce EPDMS; sixteen introduce additional long-horizon uncertainty.Source: Additional Ablations · p. 12

Training-Side Hyperparameters

  • Both λdac = 0.01 and 0.05 achieve 91.0 EPDMS, while λdac = 0.1 reduces the score to 90.5.The main experiments use λdac = 0.01.Source: Additional Ablations · p. 12
  • Performance varies only marginally across effective fine-tuning batch sizes of 320, 640, and 1280, with 640 yielding the highest score.Planning performance is largely insensitive to the future-video loss weight within the tested range; the experiments use λact : λvid = 1 : 1.Source: Additional Ablations · p. 12

Inference-Efficiency Details

The deployed action-only path omits future-video processing and fits roughly within a 2 Hz planner’s 0.5-second budget on an H800. On an H20, adding the future-video branch substantially increases trajectory-only latency, and decoding requested video adds further time.

Inference-Efficiency Details

  • On an H20, the action-only path is dominated by six-view VAE encoding at ∼175 ms and two-step denoising at ∼151 ms.Enabling the future-video branch raises trajectory-only latency to 848.0 ± 1.8 ms; optional future-video VAE decoding adds 1394.3 ± 3.3 ms when video outputs are requested.Source: Inference-Efficiency Details · p. 12
  • On a single H800, the deployed action-only pipeline runs at roughly 2 Hz, within the 0.5 s budget of a 2 Hz planner.Table 11 breaks down the H800 cost of the action-only path against the joint action–video path.Source: Inference-Efficiency Details · p. 12

Additional Qualitative Results

The additional examples examine camera coverage, zero-shot transfer, maneuver-level planning, comparisons with front-view methods, and remaining failure cases. They show where surround-view context is used and identify conditions where planning still fails.

Effect of Camera Coverage

  • Full six-camera coverage keeps predicted trajectories closer to the ground truth and lane center than reduced-view settings, while action tokens attend most strongly to front-left and rear views during a left lane change.Those views contain the neighboring vehicle constraining the maneuver.Source: Effect of Camera Coverage · pp. 12–13

Zero-shot In-house Transfer

  • In an in-house zero-shot left-turn waiting-lane scenario, SV-WAM remains closest to ground truth while clearing the rear-left vehicle; Epona collides, and PWM deviates substantially.None of the compared methods is fine-tuned on the target domain.Source: Zero-shot In-house Transfer · p. 13

Scenario-wise Planning Examples

  • Representative NAVSIMv2 trajectories show the action-only planner remaining stable across straight driving, turns, and intersection scenarios.The examples span common closed-loop maneuvers with different interaction and geometry patterns.Source: Scenario-wise Planning Examples · p. 13

Comparison with Other Methods

  • The qualitative comparison covers 15 NAVSIMv2 scenes, including dense intersections, turns, and nearby-traffic interactions, with front-view baselines alongside six-camera SV-WAM.Each row is one scene; columns show DriveLaW, DriveVLA-W0, and SV-WAM.Source: Comparison with Other Methods · p. 14

Failure Cases

  • Representative failures include incorrect paths in ambiguous two-branch turns, failures by all three methods in heavy rain, and SV-WAM being rear-ended after accelerating too conservatively on green.The examples point to interaction reasoning and uncertainty handling as future-work directions.Source: Failure Cases · p. 14

Limitations and Future Work

The approximately 5B-parameter backbone remains relatively large for resource-constrained onboard platforms. Planned work includes improving robustness and generalization, developing lighter variants, and evaluating the system in real-vehicle closed-loop testing.

Limitations and Future Work

  • The approximately 5B-parameter backbone remains relatively large for deployment on resource-constrained onboard platforms.This is the stated deployment boundary despite efficient action-only inference.Source: Limitations and Future Work · p. 14
  • Planned work includes scaling training with larger and more diverse real-world driving data to further improve robustness and generalization.Source: Limitations and Future Work · p. 14
  • The authors plan to extend benchmark-based evaluation to closed-loop testing on real vehicles to assess reliability and practical deployability under diverse conditions.They also plan to explore distillation, pruning, and quantization for lightweight edge-deployment variants.Source: Limitations and Future Work · p. 14
Loading 2609.03602v1…