Source-linked AI summary
VAD: Vectorized Scene Representation for Efficient Autonomous Driving
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, Xinggang Wang
TL;DR
Autonomous-driving planning needs scene understanding that is both safe and efficient, but rasterized representations are costly and omit instance-level structure. VAD replaces them with a fully vectorized end-to-end representation that uses map and agent information for planning constraints. On nuScenes, VAD achieves state-of-the-art planning with substantially lower error and collision rate than UniAD, while running faster; its multi-modality planning use and additional traffic information remain open issues.
Problem
Rasterized scene representations used for planning are computationally intensive and miss critical instance-level structure information.
Method
VAD models vectorized agent motion and map elements in a fully vectorized end-to-end autonomous-driving paradigm and uses them as planning guidance and constraints.
Results
VAD-Base reduces average planning displacement error by 30.1% and average collision rate by 29.0% versus UniAD while running 2.5× faster on nuScenes.
Takeaways & Limitations
VAD demonstrates that a fully vectorized driving-scene representation can provide high planning performance and efficiency relevant to autonomous-driving safety and deployment.
Takeaways & Limitations
VAD uses only the most confident multi-modality agent trajectory for its collision constraint, and incorporating the full predictions and additional traffic information remains for future exploration.
Abstract
from arXiv · showhide
Autonomous driving requires a comprehensive understanding of the surrounding environment for reliable trajectory planning. Previous works rely on dense rasterized scene representation (e.g., agent occupancy and semantic map) to perform planning, which is computationally intensive and misses the instance-level structure information. In this paper, we propose VAD, an end-to-end vectorized paradigm for autonomous driving, which models the driving scene as a fully vectorized representation. The proposed vectorized paradigm has two significant advantages. On one hand, VAD exploits the vectorized agent motion and map elements as explicit instance-level planning constraints which effectively improves planning safety. On the other hand, VAD runs much faster than previous end-to-end planning methods by getting rid of computation-intensive rasterized representation and hand-designed post-processing steps. VAD achieves state-of-the-art end-to-end planning performance on the nuScenes dataset, outperforming the previous best method by a large margin. Our base model, VAD-Base, greatly reduces the average collision rate by 29.0% and runs 2.5x faster. Besides, a lightweight variant, VAD-Tiny, greatly improves the inference speed (up to 9.3x) while achieving comparable planning performance. We believe the excellent performance and the high efficiency of VAD are critical for the real-world deployment of an autonomous driving system. Code and models are available at https://github.com/hustvl/VAD for facilitating future research.
1. Introduction
VAD addresses the safety and efficiency challenges of autonomous-driving planning with a fully vectorized scene representation. It uses instance-level map and agent information as planning guidance and constraints while reporting strong nuScenes performance and faster inference.
- Traditional modular systems decouple perception and planning, allowing perception errors to propagate into planning and creating safety concerns.
- Rasterized scene representations are computationally intensive and miss critical instance-level structure information.Examples include semantic, occupancy, flow, and cost maps.
- VAD models vectorized agent motion and map elements in an end-to-end representation, removing dense rasterization and hand-designed post-processing.
- Vectorized maps provide road structure for trajectory search, while agent motion vectors provide instance-level restrictions for collision avoidance.
- VAD combines implicit query-based scene learning with explicit instance-level planning constraints to guide planning.The constraints include an ego-agent collision constraint for maintaining safe lateral and longitudinal distances.
- VAD-Base reduces average planning displacement error by 30.1% and average collision rate by 29.0% versus UniAD while running 2.5× faster.The reported values are 0.72m versus 1.03m, 0.22% versus 0.31%, and 4.5 FPS versus 1.8 FPS, respectively.
2. Related Work
Prior autonomous-driving research spans perception, motion prediction, and planning, using both rasterized and vectorized scene representations. Related approaches include query-based perception, vectorized map prediction, learned motion forecasting, direct trajectory prediction, and reinforcement learning.
- Perception: Camera-based perception methods include 3D query detection, positional encoding, BEV feature modeling, dense map segmentation, and vectorized map prediction.
- Perception: MapTR and related query features and perception results are used in subsequent motion-prediction and planning stages.
- Motion Prediction: Motion-prediction methods use rasterized BEV images or vectorized representations with CNN, GNN, or Transformer architectures.
- Planning: Learning-based planning includes direct trajectory or control prediction, reinforcement learning, and explicit dense cost-map approaches.
3. Method
VAD learns a fully vectorized scene from BEV features and uses query interactions plus explicit vectorized constraints to produce safer, efficient planning trajectories.
- 3.1. Vectorized Scene Learning: VAD projects multi-frame, multi-view images into BEV features, then encodes map and agent information with dedicated queries.The scene is represented through map vectors and agent motion vectors.
- 3.1. Vectorized Scene Learning: Map queries predict lane dividers, road boundaries, and pedestrian crossings, whose vectors provide direction and drivable-area information for planning.Map vectors are predicted with class scores and used in planning.
- 3.1. Vectorized Scene Learning: Agent queries decode traffic-agent attributes and multi-modal future trajectories, which are passed to planning and used for collision avoidance.Agent-agent and agent-map attention enriches features before motion prediction.
- 3.2. Planning via Interaction: The ego query interacts first with agent queries and then map queries, after which a planning head combines the updated query, ego status, and driving command.The planning head outputs an ego trajectory and uses turn-left, turn-right, or go-straight commands.
- 3.3. Vectorized Planning Constraint: During training, VAD regularizes the ego trajectory with collision, boundary, and lane-direction constraints derived from vectorized agents and maps.The constraints address longitudinal and lateral agent safety, drivable-area adherence, and consistency with lane direction.
- 3.4. Experiments: Table 1 evaluates open-loop planning on nuScenes val and reports VAD as the best-performing and fastest method under the stated GPU and input conditions.The comparison deactivates ego-status information for fairness; FPS measurements use different GPUs for UniAD versus ST-P3 and VAD.
4. Experiments
Experiments evaluate VAD on nuScenes and CARLA, comparing planning performance, collision rates, map representations, design choices, qualitative behavior, and runtime. VAD achieves strong open- and closed-loop results while retaining high inference efficiency.
- Experimental Setup: nuScenes contains 1000 driving scenes with 1.4M 3D bounding boxes, and evaluation uses Displacement Error and Collision Rate.Scenes last roughly 20 seconds, use six cameras with 360° horizontal coverage, and keyframes are annotated at 2Hz.
- Closed-loop Planning: VAD outperforms prior vision-only methods in closed-loop planning on Town05, reaching 30.31 DS and improving RC from 56.36 to 75.20 on Town05 Long.On Town05 Short, VAD improves DS by 9.15 over ST-P3 and has a better RC; ST-P3 has better RC on Town05 Long but worse DS.
- Ablation Study: Ablations show that ego-agent and ego-map interactions and the three vectorized planning constraints improve safety and planning accuracy.Using all three constraints together yields the lowest collision rate and best planning accuracy, while removing ego-map interaction increases planning distance error.
- Representation and Efficiency: Rasterized map representation produces a much higher collision rate, while VAD-Tiny’s planning module requires only 3.4ms of runtime.The runtime profile attributes most computation to the backbone and BEV encoder; motion and map modules account for 34.6% of total runtime.
5. Conclusion
VAD explores fully vectorized scene representation and its use for planning, aiming to combine high performance with high efficiency. The conclusion also identifies open questions in using multimodal predictions and incorporating additional traffic information.
- VAD models driving scenes with fully vectorized representations to improve planning performance and efficiency.The paper presents vectorized scene information as relevant to autonomous-driving safety and deployment.
- VAD uses the most confident trajectory from multimodal agent predictions in its collision constraint, leaving broader multimodal planning for future work.
- Incorporating lane graphs, road signs, traffic lights, and speed limits remains an area for further exploration.