Source-linked AI summary
End-to-End Driving with Online Trajectory Evaluation via BEV World Model
Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, Zhaoxiang Zhang
TL;DR
End-to-end driving needs effective online trajectory evaluation, but future-state supervision and efficient future modeling remain challenging. WoTE uses a BEV world model to forecast trajectory-conditioned future states and evaluate candidates. The framework achieves state-of-the-art performance on NAVSIM and Bench2Drive while maintaining real-time efficiency.
Problem
Online trajectory evaluation must assess candidate futures for safe end-to-end driving, but image-level world models are costly and real-world logs provide only one future state for multiple candidates.
Method
WoTE predicts trajectory-conditioned future BEV states and evaluates trajectories using a BEV world model supervised by semantic BEV maps and simulator rewards.
Results
WoTE achieves state-of-the-art performance on NAVSIM and closed-loop Bench2Drive while maintaining real-time efficiency.
Takeaways & Limitations
The results establish online trajectory evaluation with imagined future states as a research direction for end-to-end autonomous driving.
Takeaways & Limitations
Future-state prediction requires multiple imagined outcomes, whereas real-world driving datasets provide only a single future state.
Abstract
from arXiv · showhide
End-to-end autonomous driving has achieved remarkable progress by integrating perception, prediction, and planning into a fully differentiable framework. Yet, to fully realize its potential, an effective online trajectory evaluation is indispensable to ensure safety. By forecasting the future outcomes of a given trajectory, trajectory evaluation becomes much more effective. This goal can be achieved by employing a world model to capture environmental dynamics and predict future states. Therefore, we propose an end-to-end driving framework WoTE, which leverages a BEV World model to predict future BEV states for Trajectory Evaluation. The proposed BEV world model is latency-efficient compared to image-level world models and can be seamlessly supervised using off-the-shelf BEV-space traffic simulators. We validate our framework on both the NAVSIM benchmark and the closed-loop Bench2Drive benchmark based on the CARLA simulator, achieving state-of-the-art performance. Code is released at https://github.com/liyingyanUCAS/WoTE.
1. Introduction
End-to-end driving methods increasingly predict multiple trajectories, making reliable online evaluation important for safety. Traditional rule-based evaluation depends on perception results and is sensitive to perception inaccuracies.
- Multiple trajectory prediction better reflects driving’s multi-modal nature and improves performance.
- Effective evaluation is crucial for selecting among multiple possible trajectories safely and reliably.
- Traditional trajectory evaluation uses rules and perception outputs such as bounding boxes and maps.
- Perception inaccuracies make traditional evaluation sensitive and difficult to optimize directly end to end.
𝟎𝟎. 𝟏𝟏 Reward Model
WoTE evaluates multiple predicted trajectories by imagining their future BEV states and selecting the highest-reward option. BEV modeling addresses image-level latency and supervision challenges while supporting real-time evaluation.
- WoTE predicts multi-modal trajectories, forecasts each trajectory’s future BEV states, and selects the trajectory receiving the highest reward.
- Evaluating trajectories from current state alone is challenging because candidate trajectories can lead to different unknown futures.
- World models support trajectory evaluation by modeling environmental dynamics and reasoning about future scenarios.
- Image-level driving world models can be time-consuming, while real-time video-rendering approaches require off-board reconstruction and are unsuitable onboard.
- WoTE predicts future states in compact BEV space, enabling efficient feed-forward prediction and supervision from BEV traffic simulators.
- The BEV world model uses simulated semantic BEV maps and established simulator evaluation rules as future-state supervision and reward targets.
- WoTE achieves state-of-the-art performance on NAVSIM and closed-loop CARLA-based Bench2Drive benchmarks.
2. Related Works
Related work covers end-to-end driving and trajectory evaluation, contrasting model-free assessment with WoTE’s differentiable, model-based use of BEV features and trajectory embeddings.
- 2.1. End-to-end Autonomous Driving: End-to-end autonomous driving directly maps sensor inputs to trajectories or control signals instead of decomposing driving into traditional subtasks.
- 2.1. End-to-end Autonomous Driving: Existing end-to-end driving methods include imitation-learning and reinforcement-learning training paradigms.
- 2.2. Trajectory Evaluation: Trajectory evaluation methods are categorized as model-free or model-based according to whether they explicitly model environmental dynamics.
- 2.2. Trajectory Evaluation: Model-free approaches often use explicit trajectory representations and non-differentiable metrics, limiting end-to-end optimization.
- 2.2. Trajectory Evaluation: WoTE uses BEV features and encoded trajectory embeddings for fully differentiable end-to-end trajectory evaluation.
3. Method
WoTE jointly predicts multiple trajectories and evaluates them by forecasting trajectory-conditioned future BEV states, then selecting the highest-reward trajectory. Its BEV-space world model supports recurrent future prediction and simulator-based supervision.
- Framework overview: WoTE integrates a Trajectory Predictor, BEV World Model, Reward Model, and BEV-space supervision into one end-to-end framework.The predictor proposes trajectories, the world model predicts corresponding future BEV states, and the reward model evaluates them.
- Trajectory Prediction: The BEV encoder converts multi-modal LiDAR and multi-view RGB inputs into a unified BEV state used for trajectory prediction.The BEV state is represented as a feature map with spatial dimensions and channels.
- Trajectory Prediction: Trajectory anchors are refined using cross-attention between anchor embeddings and the current BEV state to produce multiple trajectory candidates.An MLP predicts offsets that are added to the anchors to obtain refined trajectories.
- BEV World Model: For each refined trajectory, the world model forms a state-action pair and recurrently predicts future BEV states over multiple time steps.The model processes BEV features and trajectory action embeddings with a transformer encoder in parallel across state-action pairs.
- Reward Model: The Reward Model uses current and predicted future BEV states to estimate imitation and simulator-defined rewards, selecting the trajectory with the highest final reward.Simulator rewards include criteria such as no collisions and drivable-area compliance; the final reward combines imitation and simulation rewards with weights.
- BEV Space Supervision: BEV semantic maps supervise predicted states with Focal Loss, while simulators provide future BEV maps and trajectory rewards for training.The semantic maps encode classes including roads, walkways, centerlines, static objects, vehicles, and pedestrians.
4. Experiments
Experiments evaluate WoTE on NAVSIM and closed-loop Bench2Drive, then examine trajectory evaluation, reward design, recurrent prediction, trajectory count, latency, and generalization. Across these studies, the BEV world model improves evaluation performance while remaining efficient for real-time driving.
- Benchmark: WoTE achieves a PDMS of 87.1 on the NAVSIM test set and outperforms the model-free Hydra-MDP approach.The comparison highlights model-based trajectory evaluation using future-scenario simulation.
- Benchmark: WoTE improves the Bench2Drive Driving Score by 1.81 points in closed-loop CARLA evaluation.Bench2Drive evaluates 220 short routes using Driving Score and route success rate.
- Trajectory evaluation and future-state prediction: Trajectory evaluation and future-state prediction jointly improve performance over trajectory prediction alone.The ablation compares a TransFuser baseline, trajectory evaluation without future states, and the full framework.
- Reward ablation: Imitation and simulation rewards are complementary: imitation performs better on NC and TTC, whereas simulation performs better on DAC and EP.Combining both reward types improves overall performance across key metrics.
- Prediction and trajectory-count ablations: Finer recurrent future-state prediction improves performance, while increasing trajectories from 64 to 128 yields a marked improvement and gains diminish from 128 to 256.The default configuration uses 256 trajectories; the world model predicts future states recurrently.
- Efficiency and generalization: 18.7 ms total latency on an NVIDIA L20 GPU keeps the framework within real-time requirements, and 1024 unseen trajectory anchors increase PDMS by 1.3.State-action pairs are processed in parallel, and the trajectory evaluation module generalizes across trajectory configurations.
5. Conclusion
WoTE uses a BEV world model to evaluate trajectories by modeling future driving-scene evolution. It provides dense simulator supervision, filters unsafe candidates, and achieves state-of-the-art performance with real-time efficiency.
- Framework: WoTE introduces a BEV world model for end-to-end trajectory evaluation in autonomous driving.The model supports evaluation informed by the dynamic evolution of driving scenarios.
- Trajectory evaluation: WoTE effectively filters low-quality trajectories and helps avoid collisions compared with trajectory prediction without an evaluation module.The comparison uses TransFuser as the trajectory prediction baseline.
- Trajectory evaluation: WoTE retains diverse multi-modal trajectories that comply with traffic rules and avoid unsafe outcomes, while unreasonable trajectories receive lower rewards.Reward visualizations use brighter colors for higher rewards and gray for very low rewards.
- Framework: Dense BEV-space simulator supervision supplies semantic future states and rule-based reward targets for trajectory evaluation.This supervision supports the framework's evaluation process.
- Results: WoTE achieves state-of-the-art performance on NAVSIM and Bench2Drive while maintaining real-time efficiency.Bench2Drive is a closed-loop benchmark based on the CARLA simulator.