Source-linked AI summary
Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, Jose M. Alvarez
TL;DR
The paper examines why ego-status inputs can dominate open-loop nuScenes planning and why existing evaluations may not reflect true planning quality. It analyzes these issues, introduces Curb Collision Rate, and proposes a simple BEV-Planner baseline, concluding that prevailing comparisons warrant caution.
Problem
Open-loop nuScenes planning can be strongly affected by ego status, while existing metrics incompletely capture planning behavior and the benchmark’s reliability remains in question.
Method
The paper conducts analyses of existing methods, redesigns a simple BEV-Planner baseline, and introduces Curb Collision Rate to evaluate trajectory adherence to road boundaries.
Results
Across simple driving scenarios, perception does not appear to improve L2 distance or collision rate, while map information can reduce collision rate in turning scenarios.
Takeaways & Limitations
Planning evaluations should use more diverse metrics because L2 distance and collision rate capture only partial aspects of planning capability.
Takeaways & Limitations
The study’s open-loop setting is constrained because historical ego trajectories come from human drivers, and methods using ego status are not directly comparable.
Abstract
from arXiv · showhide
End-to-end autonomous driving recently emerged as a promising research direction to target autonomy from a full-stack perspective. Along this line, many of the latest works follow an open-loop evaluation setting on nuScenes to study the planning behavior. In this paper, we delve deeper into the problem by conducting thorough analyses and demystifying more devils in the details. We initially observed that the nuScenes dataset, characterized by relatively simple driving scenarios, leads to an under-utilization of perception information in end-to-end models incorporating ego status, such as the ego vehicle's velocity. These models tend to rely predominantly on the ego vehicle's status for future path planning. Beyond the limitations of the dataset, we also note that current metrics do not comprehensively assess the planning quality, leading to potentially biased conclusions drawn from existing benchmarks. To address this issue, we introduce a new metric to evaluate whether the predicted trajectories adhere to the road. We further propose a simple baseline able to achieve competitive results without relying on perception annotations. Given the current limitations on the benchmark and metrics, we suggest the community reassess relevant prevailing research and be cautious whether the continued pursuit of state-of-the-art would yield convincing and universal conclusions. Code and models are available at \url{https://github.com/NVlabs/BEV-Planner}
1. Introduction
The paper examines why ego status can support strong open-loop planning results on nuScenes and argues that dataset simplicity and incomplete metrics distort conclusions about perception and planning.
- Open-loop evaluation uses real-world log replay, avoiding simulator domain gaps while treating human driving as planning ground truth.
- Ego-MLP matches state-of-the-art methods using only ego status, while planner-integrated ego status methods gain no demonstrated superiority from additional perception information.
- 73.9% of nuScenes data involve straightforward driving, where maintaining velocity, direction, or turning rate can often suffice.
- Current L2 distance and collision-rate metrics may under-penalize risky trajectories, including paths that leave the road.
- The paper introduces Curb Collision Rate to measure interactions between predicted trajectories and road boundaries.
- The authors conjecture that nuScenes’ simple scenarios and ego-status reliance may explain recent state-of-the-art scores, while existing metrics incompletely assess planning quality.
2. Related Work
Related work spans BEV-based perception and end-to-end driving, while highlighting a methodological concern about using human-provided historical ego trajectories in open-loop learning.
- BEV perception methods improve 3D understanding through latent depth estimation and temporal information.
- End-to-end driving learns from raw sensor data to trajectories or commands, reducing manual feature extraction and supporting diverse driving scenarios.
- Using historical ego trajectories in open-loop driving is debated because the model does not generate those trajectories; a human driver does.
3. Proposed BEV-Planner
BEV-Planner is a simple baseline that fuses BEV features with a learnable ego query and predicts trajectories without human-labeled perception annotations.
- BEV-Planner generates BEV features, concatenates historical BEV features without alignment, and applies cross-attention with a learnable ego query.
- The refined ego query is passed through MLPs to produce final predicted trajectories.
- The baseline can incorporate ego status either into BEV generation or by concatenating it with the planner’s ego query.
- BEV-Planner requires no human-labeled bounding boxes, tracking IDs, or HD maps, using only an L1 trajectory-supervision loss.
- The authors state that the baseline is not intended for real-world deployment because it lacks adequate constraints and interoperability.
4. Experiments
The experiments show that ego status can dominate open-loop planning results, while perception and map information matter more in challenging scenarios and under road-adherence metrics. The analyses also expose weaknesses in current evaluation practices and show that visual features become less informative when ego status is introduced.
- Dataset and metrics: CCR evaluates predicted-trajectory collisions with curbs, revealing road-adherence failures overlooked by L2 distance and collision rate.Ego-MLP performs worse than UniAD and VAD on CCR, while BEV-Planner performs worst because it uses no map information.
- Ego status dependence: Ego-MLP matches state-of-the-art methods on existing L2 distance and collision-rate metrics using only ego status.The reproduced model removes history trajectory ground truths to avoid potential label leakage.
- Ego status versus perception: Perception information benefits complex turning scenarios, but its average effect is obscured because turning scenes comprise only 13% of the evaluation data.For straightforward driving, perceptual information does not improve L2 distance or collision rate and can reduce average performance.
- Dataset and metrics: UniAD post-processing reduces collisions with other agents but can increase road-boundary risk because it lacks map-prior constraints.The paper therefore reports UniAD without post-processing by default.
- Limitations: Open-loop evaluation remains difficult for continuous cornering, where methods with varied settings produce suboptimal trajectories.The paper calls for a more diverse and stringent evaluation framework even after introducing CCR.
- Model analysis: Introducing ego status makes BEV-Planner++ converge rapidly and shifts feature activations toward the ego vehicle, often behind it.The authors speculate that ego status reduces the model’s need to extract information from BEV features.
5. Conclusion
The paper analyzes shortcomings in open-loop end-to-end autonomous driving and concludes that better datasets and more comprehensive metrics are more urgent than continued nuScenes state-of-the-art optimization.
- The paper presents an in-depth analysis of shortcomings in current open-loop, end-to-end autonomous driving methods.
- Existing nuScenes models are strongly affected by ego status, which can diminish their use of sensory information.Ego status includes velocity, acceleration, and yaw angle.
- Existing planning metrics can produce substantially different evaluations and fail to capture all aspects of planning performance.
- The authors advocate diverse, comprehensive metrics to reduce local optimization that may neglect other safety hazards.
- The limitation is a trade-off between planning metrics, while the baseline remains weaker in CCR because it uses no perception annotations such as HD maps.
A. Implementation Details
The implementation details identify incorrect tail-sample supervision in ST-P3 and describe how ego-status inputs are selectively disabled or retained in UniAD and VAD-Base.
- ST-P3 generates erroneous training and testing targets for tail samples by using samples from other scenes instead of masking incomplete future trajectories.NuScenes clips typically last 20 seconds, so samples from 17–20 seconds lack complete three-second futures.
- For UniAD and VAD-Base, the implementation disables ego status during BEV generation by setting the use can bus flag to False.
- For UniAD, ego status is incorporated into the planner by concatenating its vector with query features, following the VAD methodology.
B. Metrics Details.
The metrics discussion examines collision-rate implementation, trajectory smoothness, valid-sample handling, yaw-aware collision detection, and road-boundary behavior in open-loop planning.
- Collision Rate: Open-loop collision rates are biased because other agents follow predetermined trajectories rather than reacting to the ego vehicle.
- Collision Rate: N represents the number of steps at intervals of t seconds, while I_i indicates whether the ego car intersects at step i.
- Collision Detection: Current methods preserve a 0 yaw angle, increasing false negatives and false positives in collision detection; the paper estimates yaw from trajectory variation.
- Collision Rate: The original collision-rate implementation treats collisions at each moment as mutually independent, whereas the modified version measures whether a collision occurs along the predicted trajectory.
- Trajectory Smoothness: Trajectory smoothness is assessed from the squared deviation of multiple waypoints predicted for the same absolute time, but the resulting metric adds little information beyond L2 distance.
- Road Adherence: UniAD’s post-optimization can move trajectories toward the road boundary while avoiding another vehicle, creating a different accident risk.
- Ego Status: BEVFormer incorporates ego status when initializing BEV queries, a detail not addressed by current end-to-end approaches.
C. Neglected Ego Status in Perception Stage.
Ego status can enter end-to-end planning before the planner: BEVFormer projects it into hidden features and BEV queries, with substantial downstream planning effects despite marginal perception changes.
- UniAD and VAD use BEVFormer, which projects ego status onto hidden features and incorporates it into BEV queries during BEV generation.
- Removing ego-status input during the BEV stage causes a marked decline in VAD and UniAD planning performance.
- Ego-status integration has only a marginal effect on BEVFormer perception performance but can substantially influence end-to-end planning performance.
D. Post Optimization of UniAD.
Post-processing in UniAD can reduce collisions while increasing road-boundary intersections because its optimizer overlooks map priors. CCR identifies such low-quality trajectories, but metrics for high-quality trajectories remain an open direction.
- UniAD’s collision optimizer produces trajectories that intersect road boundaries more often because it overlooks map priors.The optimizer prioritizes collision avoidance while disregarding other safety risks.
- CCR identifies low-quality trajectories even when the next timestep still starts from the ground-truth trajectory after an extremely erroneous prediction.
- An appropriate metric for highlighting high-quality trajectories remains an open direction for future exploration.
E. Dropping Cameras
Camera removal affects planning differently depending on ego-status input and driving scenario. Aggregate metrics can obscure worse turning behavior because straight-driving cases dominate the evaluation set, while continuous cornering remains difficult for all methods.
- Removing camera inputs from VAD without ego status significantly worsens L2 distance and collision rate.
- With ego status, removing camera input does not markedly impair VAD performance, whereas without ego status camera removal causes a significant decline.
- Without ego status or camera input, the model relies solely on driving commands to guide future direction.
- Camera removal increases road-boundary interactions in turning scenarios, but conservative straight driving improves overall averages because straight cases comprise 87% of evaluation samples.Intersection-LR rises in turning scenarios, while Intersection-ST decreases in straight-driving scenarios.
- All evaluated methods predict suboptimal trajectories in scenarios requiring continuous turning, especially high-curvature bends.