PaperlayerView exact revision on arXiv

Source-linked AI summary

VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning

Bo Jiang, Shaoyu Chen, Hao Gao, Bencheng Liao, Qian Zhang, Wenyu Liu, Xinggang Wang

arXiv:2402.13243v2cs.CVcs.RO

TL;DR

Learning human-like driving from demonstrations is difficult because planning actions are uncertain and existing methods deterministically regress them. VADv2 instead learns a probabilistic action distribution over a tokenized planning vocabulary and samples actions for end-to-end control, achieving state-of-the-art performance across multiple benchmarks.

  • Problem

    Driving demonstrations contain stochastic, non-deterministic actions, while existing learning-based planners directly regress actions under a deterministic scene-to-action assumption.

  • Method

    VADv2 models planning with a probabilistic field, discretizes the continuous action space into a planning vocabulary, tokenizes actions and sensor data, and learns distributions from demonstrations.

  • Results

    VADv2 achieves state-of-the-art planning performance across closed- and open-loop settings and multiple benchmarks, including CARLA, Bench2Drive, NAVSIM, and a 3DGS-based benchmark.

  • Takeaways & Limitations

    Probabilistic planning provides a demonstrated end-to-end approach for modeling uncertain driving behavior and controlling vehicles through sampled actions.

  • Takeaways & Limitations

    Simulator and 3DGS-based closed-loop environments still have naive agent behaviors and insufficient scene realism, which may restrict VADv2’s performance.

Abstract

from arXiv · show

Learning a human-like driving policy from large-scale driving demonstrations is promising, but the uncertainty and non-deterministic nature of planning make it challenging. Existing learning-based planning methods follow a deterministic paradigm to directly regress the action, failing to cope with the uncertainty problem. In this work, we propose a probabilistic planning model for end-to-end autonomous driving, termed VADv2. We resort to a probabilistic field function to model the mapping from the action space to the probabilistic distribution. Since the planning action space is a high-dimensional continuous spatiotemporal space and hard to tackle, we first discretize the planning action space to a large planning vocabulary and then tokenize the planning vocabulary into planning tokens. Planning tokens interact with scene tokens and output the probabilistic distribution of action. Mass driving demonstrations are leveraged to supervise the distribution. VADv2 achieves state-of-the-art closed-loop performance on the CARLA Town05 benchmark, significantly outperforming existing methods, and also leads the recent Bench2Drive benchmark. We further provide comprehensive evaluations on NAVSIM and a large-scale 3DGS-based benchmark, demonstrating its effectiveness in real-world applications. Code is available at https://github.com/hustvl/VAD.

1 INTRODUCTION

VADv2 addresses the uncertainty and non-determinism of driving actions by replacing deterministic regression with probabilistic planning. It tokenizes the action space and learns scene-conditioned action distributions from demonstrations for end-to-end control.

  • Motivation: Human drivers can choose multiple reasonable maneuvers in the same scenario, making timing, speed, and action selection highly stochastic.Examples include lane keeping or overtaking when following another vehicle, and yielding or overtaking when facing oncoming traffic.
  • Motivation: Existing methods directly regress trajectories or control signals, assuming a deterministic scene-to-action relation that fails when feasible solutions are multiple and non-convex.Such regression can produce an intermediate action with potential safety risks.
  • Probabilistic planning: Probabilistic planning models a scene-conditioned stochastic policy p(a|o) to represent correlations between candidate actions and driving scenes.The approach ranks actions and samples a high-scoring one rather than regressing a single optimal action.
  • Method: VADv2 uses a probabilistic field, discretizes the continuous spatiotemporal action space into a planning vocabulary, tokenizes it, and interacts planning tokens with scene tokens.Large-scale driving demonstrations supervise the predicted action distribution.
  • System: VADv2 processes surround-view image sequences in a streaming manner, outputs an action distribution, and samples one action for vehicle control.The model is presented as an end-to-end driving system based on probabilistic planning.

2 RELATED WORK

Related work spans unified scene perception, motion prediction, learning-based planning, and language-model-based driving. VADv2 represents planning probabilistically while using tokenized sensor and action representations in an end-to-end framework.

  • Perception: BEV representations provide a unified way to encode driving scenes and fuse multimodal data for downstream autonomous-driving tasks.LSS explicitly predicts depth for perspective-to-BEV transformation, whereas BEVFormer uses spatial and temporal attention without explicit depth prediction.
  • Motion Prediction: Motion prediction forecasts other traffic participants’ trajectories, using historical trajectories and maps or jointly learning perception and prediction end to end.Some methods represent future motion with dense occupancy and flow fields.
  • Planning: Learning-based planning ranges from black-box control prediction to integrated perception, prediction, and vectorized-scene planning, but existing approaches use varied planning representations and assumptions.Examples include UniAD, VAD, diffusion-based planners, and predefined trajectory anchors.
  • Large Language Model in Autonomous Driving: LLM-based autonomous-driving methods support scene understanding, evaluation, or planning, but limited inference speed constrains real-time deployment.These approaches differ from VADv2’s tokenized probabilistic planning pipeline.
  • VADv2: VADv2 discretizes the action space into a planning vocabulary, approximates its distribution from demonstrations, and samples one action at each time step.Its framework tokenizes multi-view sensor data and planning actions before producing the action distribution.

3 VADV2

VADv2 encodes scene information and candidate planning trajectories as tokens, then predicts and supervises an action distribution for probabilistic planning. Its training combines demonstration-derived distribution learning with conflict and scene-token supervision.

  • Scene Encoder: VADv2 transforms multi-view sensor data into map, agent, traffic-element, and image tokens, supplemented by navigation and ego-state embeddings.These compact scene features provide the foundation for the planning module.
  • Probabilistic Planning: Probabilistic planning represents the policy as p(a|o), where actions are future waypoint sequences conditioned on observed scene information.Each action is represented by waypoint coordinates across future timestamps.
  • Probabilistic Planning: The continuous action space is discretized into a vocabulary of representative demonstration trajectories, with N=4096 by default and kinematically feasible actions.Furthest trajectory sampling selects representative actions from the demonstration action set.
  • Probabilistic Planning: Each candidate action is encoded as a planning token, and a Transformer decoder uses it with scene, navigation, and ego-state features to predict its probability.The action token serves as the query while scene tokens provide keys and values.
  • Training: Training uses distribution, conflict, and scene-token losses, learning demonstration frequencies with KL divergence while reducing probabilities for actions conflicting with agents or road boundaries.Map, agent, and traffic-element tokens receive corresponding supervision signals.

4 EXPERIMENTS

Experiments evaluate VADv2 across simulated, NAVSIM, and 3DGS-based settings using closed-loop, open-loop, safety, multimodal, ablation, and qualitative analyses. VADv2 shows strong benchmark performance, robust probabilistic planning, and multimodal trajectory quality.

  • Experimental settings: VADv2 is evaluated on CARLA, NAVSIM/NAVSIMv2, and a 3DGS-based benchmark, including closed-loop and planning metrics.CARLA uses Route Completion, Infraction Score, and Driving Score; the 3DGS benchmark evaluates collision and trajectory deviation.
  • Benchmark results: 85.1 Driving Score, 98.4 Route Completion, and 0.87 Infraction Score are achieved on Town05 Long.Relative to DriveMLM, Driving Score improves by 9.0; compared with the previous best camera-based method, it increases by up to 10.2.
  • Benchmark results: 76.15 Drive Score is achieved on Bench2Drive, the highest reported result in that benchmark.The model uses camera sensors as perception input in the reported end-to-end evaluation.
  • Benchmark results: 0.270 Collision Ratio is achieved on the 3DGS-based benchmark, a 15.6% improvement over TransFuser’s 0.320.VADv2 maintains a competitive Deviation Ratio of 0.243.
  • Multimodal planning: Top-1 and other top-5 trajectories have comparable performance, indicating high-quality multimodal planning outputs.The top-1 trajectory achieves the best overall performance among the evaluated candidates.
  • Ablation study: Removing Distribution Loss, Conflict Loss, or scene and traffic-token designs harms planning accuracy, while the complete design performs best.The ablation uses 50k driving-demonstration clips, and the best planning performance occurs when all listed designs are included.
  • Ablation study: Probabilistic planning consistently outperforms deterministic planning and remains stable across traffic densities.Deterministic planning degrades noticeably as density varies, while probabilistic planning maintains stable performance.
  • Visualization: Qualitative examples show multiple reasonable trajectories for speed variation, creeping, lane changes, and intersection decisions.The visualizations include alternative trajectories when a target lane contains another vehicle.

5 CONCLUSIONS AND LIMITATIONS

The paper introduces VADv2 as an end-to-end model based on probabilistic planning and reports strong performance across simulated and complex driving evaluations. It also identifies limitations in current simulator and 3DGS-based closed-loop environments.

  • Conclusions: VADv2 introduces an end-to-end driving model based on probabilistic planning and achieves state-of-the-art closed-loop performance.The conclusion reports effectiveness and robustness in complex driving scenarios.
  • Limitations: Simulator and 3DGS-based closed-loop environments have naive agent behaviors and insufficient scene realism that may restrict VADv2’s performance.The authors propose using larger-scale expert driving data to improve planning and bridge simulation with real-world deployment.

A APPENDIX

The appendix includes ablations on vocabulary size and planning manners, extending analysis of VADv2’s design choices.

  • Ablation studies: Vocabulary-size effects are examined through an ablation study.The appendix provides a dedicated vocabulary-size ablation table.
  • Ablation studies: Different planning manners are compared in an additional ablation study.The appendix provides a dedicated planning-manner ablation table.

A.1 CLOSED-LOOP RESULTS ON THE TOWN05 SHORT BENCHMARK

The Town05 Short benchmark tests targeted driving behaviors, including dense-traffic and pre-intersection lane changes. VADv2 achieves the highest Driving Score in this setting.

  • Benchmark setup: Town05 Short evaluates targeted behaviors including lane changes in dense traffic and before intersections.The benchmark focuses on challenging closed-loop driving scenarios.
  • Results: VADv2 achieves the highest Driving Score, while LeapVAD has higher Route Completion but lower Driving Score.The reported difference suggests more infractions for LeapVAD.

A.2 MORE ABLATION STUDY

Ablations show that finer action-space discretization and more driving demonstrations improve planning metrics, while probabilistic planning provides its main advantage in closed-loop stability. Vocabulary sampling choices have limited impact, with FTS performing best and transferring across scenarios.

  • Vocabulary Size: Larger planning vocabularies improve both L2 and collision metrics by reducing action-space discretization error.The ablation attributes the improvement to better representation of the action space.
  • Amount of Training Clips: Increasing the amount of driving-demonstration clips improves both L2 and collision metrics.
  • Probabilistic vs. Deterministic: Probabilistic and deterministic planning perform similarly open loop, but probabilistic planning achieves better closed-loop stability and performance.The deterministic variant struggles with planning uncertainty in closed-loop settings.
  • Vocabulary Sampling: Vocabulary sampling strategies differ only slightly; FTS provides the best action-space coverage and strongest results, while nuScenes-derived vocabularies retain comparable NAVSIM performance.The comparable NAVSIM results indicate generalization across scenarios.

A.3 MORE EVALUATION DETAILS OF THE 3DGS-BASED BENCHMARK

The 3DGS-based benchmark evaluates reconstruction across weather conditions, dataset diversity, and planning-system efficiency. VADv2 maintains comparable latency while improving the primary collision-rate metric.

  • Weather Conditions: 3DGS reconstruction achieves PSNR values of 29.5, 28.8, and 28.2 in sunny, rainy, and nighttime scenes, respectively.
  • Efficiency: VADv2 adds planning-vocabulary overhead but retains latency comparable to baselines while improving the primary collision-rate metric.The same perception backbone is used across methods, so latency differences mainly reflect planning-module design.
  • Dataset Statistics: The validation dataset emphasizes both scale and scene diversity compared with nuScenes and NAVSIM.nuScenes is dominated by straight-driving scenarios, whereas NAVSIM has more turns but limited data.
Loading 2402.13243v2…