Source-linked AI summary
GenAD: Generative End-to-End Autonomous Driving
Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, Long Chen
TL;DR
End-to-end autonomous driving pipelines struggle to model future ego-agent interactions and structural trajectory priors. GenAD uses map-aware instance tokens, a learned structural latent space, and temporal generation to jointly produce motion predictions and planning results. On nuScenes, it achieves state-of-the-art vision-centric end-to-end performance with high efficiency.
Problem
Existing serial perception-prediction-planning pipelines do not comprehensively model future ego-agent interactions, trajectory structure, and planning uncertainty.
Method
GenAD uses map-aware instance-centric tokens, a variational autoencoder for structural trajectory latents, and a temporal model to jointly generate ego and agent futures.
Results
GenAD achieves state-of-the-art planning performance on nuScenes and the best L2 errors among compared methods with efficient inference speed.
Takeaways & Limitations
Generative latent-space modeling provides a unified framework for motion prediction and ego planning while capturing high-order ego-agent interactions and structural trajectory priors.
Abstract
from arXiv · showhide
Directly producing planning results from raw sensors has been a long-desired solution for autonomous driving and has attracted increasing attention recently. Most existing end-to-end autonomous driving methods factorize this problem into perception, motion prediction, and planning. However, we argue that the conventional progressive pipeline still cannot comprehensively model the entire traffic evolution process, e.g., the future interaction between the ego car and other traffic participants and the structural trajectory prior. In this paper, we explore a new paradigm for end-to-end autonomous driving, where the key is to predict how the ego car and the surroundings evolve given past scenes. We propose GenAD, a generative framework that casts autonomous driving into a generative modeling problem. We propose an instance-centric scene tokenizer that first transforms the surrounding scenes into map-aware instance tokens. We then employ a variational autoencoder to learn the future trajectory distribution in a structural latent space for trajectory prior modeling. We further adopt a temporal model to capture the agent and ego movements in the latent space to generate more effective future trajectories. GenAD finally simultaneously performs motion prediction and planning by sampling distributions in the learned structural latent space conditioned on the instance tokens and using the learned temporal model to generate futures. Extensive experiments on the widely used nuScenes benchmark show that the proposed GenAD achieves state-of-the-art performance on vision-centric end-to-end autonomous driving with high efficiency. Code: https://github.com/wzzheng/GenAD.
1. Introduction
GenAD reframes vision-centric end-to-end driving as trajectory generation to address missing ego-agent interactions and structural trajectory priors in serial pipelines.
- Motivation: Serial end-to-end pipelines separate perception, motion prediction, and planning, limiting their treatment of future traffic evolution.Existing systems can include additional stages such as map perception, detection, tracking, and occupancy prediction.
- Motivation: Ego actions can affect other agents, whose responses can subsequently alter ego planning.The paper gives an ego lane shift affecting rear-car actions as an example of this high-order interaction.
- Motivation: Future trajectories share structural regularities, such as continuity and straight-line motion, but existing methods often omit this prior.The paper links this omission to inaccurate predictions and planning.
- Proposed direction: GenAD models autonomous driving as trajectory generation in a structural latent space, jointly performing motion prediction and ego planning.Its scene tokenizer produces map-aware instance-centric representations, while a VAE models trajectory distributions and a GRU generates latent-space futures.
2. Related Work
Prior work develops vision-centric perception, prediction, and planning components, but common end-to-end systems retain serial prediction-planning designs. GenAD instead jointly generates ego and agent futures in a probabilistic latent space.
- Perception: Vision-centric perception methods transform image features into 3D space for tasks including detection, map reconstruction, and semantic occupancy prediction.Approaches use explicit depth projection or 3D queries with deformable cross-attention.
- Prediction: Motion prediction methods increasingly combine perception and prediction end to end, using attention and temporal networks to model agent and map information.Earlier approaches represented paths and maps as BEV images, vectors, or tokens.
- Planning: End-to-end planning methods commonly use serial perception, prediction, and planning pipelines, with systems such as ST-P3, UniAD, and VAD varying in module scope and efficiency.VAD simplifies UniAD with vectorized scene representation and fewer modules.
- GenAD: GenAD addresses serial-design limitations by simultaneously generating ego and other-agent trajectories in a learned probabilistic latent space.This formulation targets future ego effects on agent prediction and the uncertainty of motion prediction and planning.
3. Proposed Approach
GenAD represents driving scenes with compact, map-aware instance tokens and formulates prediction and planning as future trajectory generation in a structured latent space. It models ego-agent interactions, trajectory uncertainty, and temporal evolution to generate realistic futures.
- Scene Representation: The approach first extracts BEV features from camera inputs and builds semantic map and agent-token representations.Past-frame BEV features are aligned to the current coordinate system before map and agent features are refined.
- Scene Representation: GenAD adds an ego token to agent tokens, then applies self-attention to model high-order interactions between the ego vehicle and surrounding agents.This addresses interactions such as an ego lane shift affecting rear-car behavior.
- Scene Representation: Cross-attention between instance tokens and map tokens produces compact, map-aware representations for motion prediction and trajectory planning.The resulting tokens incorporate agent-ego interactions and semantic-map information.
- Future Generation: A GRU evolves latent states over time while an MLP decoder converts each state into a waypoint, instead of decoding the whole trajectory at once.This separates temporal movement modeling in latent space from waypoint decoding.
- Trajectory Prior Modeling: Rather than directly decoding trajectories, GenAD models future trajectories as samples from p(T|I) to represent their uncertainty.The instance representation I conditions the trajectory distribution.
- Trajectory Prior Modeling: A VAE maps ground-truth trajectories to Gaussian distributions in a latent space, capturing the structural prior of realistic continuous, straight, or curved trajectories.The learned distribution is intended to improve motion-prediction and planning authenticity.
- Generative End-to-End Autonomous Driving: At inference, GenAD samples a latent state from p(z|I), then uses the trajectory generator and waypoint decoder to perform future prediction and planning.The future trajectory encoder is discarded during inference, and the full model is trained end-to-end with auxiliary map and detection objectives.
4. Experiments
Experiments on nuScenes evaluate GenAD across perception, motion prediction, and planning, including efficiency, ablations, and qualitative comparisons with VAD-tiny.
- Main results: GenAD achieves the best L2 errors among evaluated end-to-end methods with efficient inference, while UniAD has lower collision rates using additional tracking and occupancy supervision.The comparison highlights a performance/speed trade-off and the difficulty of obtaining competitive collision performance with fewer 3D labels.
- Perception and prediction performance: GenAD outperforms VAD on perception, prediction, and planning tasks at similar inference speed.The reported metrics include 3D detection, map segmentation, end-to-end prediction accuracy, average L2 error, and collision rate.
- Effect of the instance-centric scene representation: Adding ego-to-agent interaction to VAD-tiny substantially improves L2 error and collision rate, while removing it from GenAD greatly worsens collision performance.The ablation attributes the degradation to difficulty learning trajectory distributions without high-order ego-agent interactions.
- Effect of the generative framework: Both trajectory prior modeling and latent future trajectory generation improve planning, and combining them yields a large additional gain.The result supports factorizing the joint distribution for latent trajectory prior modeling.
- Visualizations: GenAD produces better and safer trajectories than VAD-tiny across straight-driving, overtaking, turning, and complex multi-agent scenarios.In complex traffic scenes, GenAD continues to produce usable results while VAD cannot safely move through.
5. Conclusion
The paper presents GenAD as a generative vision-based end-to-end driving framework that models high-order ego-agent interactions and structural trajectory priors. Experiments on nuScenes demonstrate state-of-the-art planning performance, while future work considers other generative models.
- GenAD replaces the conventional serial perception-prediction-planning design with generative modeling of future trajectories.
- The framework enables high-order ego-agent interactions and generates more accurate future trajectories using a learned structural prior.
- Experiments on nuScenes demonstrate state-of-the-art planning performance for GenAD.
- Future work may explore generative adversarial networks and diffusion models for end-to-end autonomous driving.