Source-linked AI summary
Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes
Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, Jingdong Wang
TL;DR
The paper asks whether nuScenes open-loop L2 error and collision rate adequately distinguish autonomous-driving planning methods. It tests this with a physical-state-only MLP and finds similar planning performance to perception-based methods, while identifying collision-rate and distribution-related evaluation concerns.
Problem
The paper examines whether commonly used nuScenes L2-error and collision-rate metrics accurately measure the superiority of different end-to-end planning methods.
Method
A simple MLP directly predicts future ego trajectories from physical states and a high-level command, without visual or point-cloud perception information.
Results
The physical-state-only model achieves similar nuScenes planning performance to perception-based methods, while perception-based methods have lower collision rates.
Takeaways & Limitations
The findings suggest that current nuScenes evaluation metrics may not adequately capture the superiority of different autonomous-driving planning methods.
Takeaways & Limitations
The model performs well on nuScenes but is an impractical toy incapable of functioning in real-world scenarios without knowledge beyond ego states.
Abstract
from arXiv · showhide
Modern autonomous driving systems are typically divided into three main tasks: perception, prediction, and planning. The planning task involves predicting the trajectory of the ego vehicle based on inputs from both internal intention and the external environment, and manipulating the vehicle accordingly. Most existing works evaluate their performance on the nuScenes dataset using the L2 error and collision rate between the predicted trajectories and the ground truth. In this paper, we reevaluate these existing evaluation metrics and explore whether they accurately measure the superiority of different methods. Specifically, we design an MLP-based method that takes raw sensor data (e.g., past trajectory, velocity, etc.) as input and directly outputs the future trajectory of the ego vehicle, without using any perception or prediction information such as camera images or LiDAR. Our simple method achieves similar end-to-end planning performance on the nuScenes dataset with other perception-based methods, reducing the average L2 error by about 20%. Meanwhile, the perception-based methods have an advantage in terms of collision rate. We further conduct in-depth analysis and provide new insights into the factors that are critical for the success of the planning task on nuScenes dataset. Our observation also indicates that we need to rethink the current open-loop evaluation scheme of end-to-end autonomous driving in nuScenes. Codes are available at https://github.com/E2E-AD/AD-MLP.
1. Introduction
The paper questions whether nuScenes open-loop metrics reliably distinguish end-to-end planning methods. It tests this concern with an MLP that predicts future ego trajectories from physical states and commands, without perception or prediction features.
- Motivation: Existing autonomous-driving systems commonly separate perception, prediction, and planning into independent stages.This separation can cause information loss and error accumulation because optimization targets and model training are independent.
- Related work: Perception-based end-to-end methods connect environmental representation, prediction, and planning to produce future ego trajectories.Examples include vision-based, query-based, and vectorized scene representations.
- Evaluation question: The paper evaluates whether current nuScenes metrics accurately measure the superiority of different planning methods by withholding camera- and LiDAR-derived perception and prediction information.Only the ego vehicle’s physical state is used as input.
- Approach: The proposed model directly predicts future ego trajectories from concatenated physical-state vectors using an MLP trained with ground-truth trajectory supervision.Evaluation follows prior work using L2 error and collision rate.
- Main observation: Despite using no perception information, the simple model achieves similar trajectory-planning performance on nuScenes, suggesting that current metrics may inadequately distinguish methods.The paper attributes this result partly to information about future motion being reflected in past trajectory, velocity, acceleration, and temporal continuity.
2. Method
The method concatenates recent ego motion, instantaneous kinematics, and a high-level command, then uses a simple MLP to predict six future trajectory frames. Training uses L1 trajectory loss with reweighting for occupancy-grid coincidences.
- Model inputs: The model takes ego states and a high-level command as its only inputs and is trained end-to-end without visual or point-cloud feature extraction.The command represents the vehicle’s future short-term motion trend.
- Model inputs: The ego-state input includes four past frames of (x, y, θ), current (v_x, v_y, ω), and current (a_x, a_y, β) values.These quantities are flattened and concatenated into a one-dimensional vector.
- Model inputs: The high-level command is one-hot encoded as turn left, go straight, or turn right because the model does not use HD maps.Commands are assigned from future lateral displacement over 3 seconds.
- Network structure: The concatenated input vector has dimension 21, and the MLP outputs x-y coordinates and heading angles for six future frames.The network is summarized as Linear21-512-ReLU-Linear512-18.
- Loss function: The loss is an L1 distance between predicted and ground-truth ego trajectories, with a 0.5 reweighting for points coinciding in the same 0.5m occupancy-grid segment.This reweighting is used for hard-sample mining to reduce collision rate.
3. Experiments
Experiments use nuScenes and standard L2-error and collision-rate metrics across 1-, 2-, and 3-second horizons. Results show strong trajectory performance but expose sensitivity in collision evaluation and the dataset’s predominantly straight-driving distribution.
- Dataset and metrics: Experiments use nuScenes scenes and key frames containing camera, LiDAR, velocity, and acceleration data for training and testing.The dataset includes 1K scenes and approximately 40K key frames collected mainly in Boston and Singapore.
- Dataset and metrics: Evaluation reports L2 error in meters and collision rate in percentage for predicted trajectories over 1s, 2s, and 3s horizons.Collision testing projects scene objects into BEV occupancy representations and checks overlap with predicted waypoints.
- Main results: Adding acceleration, velocity, and high-level command information reduces average L2 error from 0.97m to 0.29m and collision rate from 0.49% to 0.19%.The ablation examines trajectories, velocity, acceleration, and command information.
- Main results: The method’s collision rate remains higher than some perception-based methods, with the reported 0.19% versus 0.14% difference representing about 2–3 samples among 4819.The authors associate this with insufficient decision-making from fitting motion information alone and call for more discriminative testing scenarios.
- Trajectory distribution: Nearly 70% of heading and curvature angles lie within −0.2 to 0.2 and −0.02 to 0.02 radians, respectively, indicating mostly straight, small-angle forward motion.Trajectory points are concentrated in the middle forward area, with few large-angle turns.
- Collision evaluation: A 0.5m occupancy-map grid can falsely classify a ground-truth trajectory as colliding when a 0.1m grid correctly identifies no collision.Coarse grids can also produce irregular object masks.
4. Analysis and Discussion
The nuScenes training data is dominated by short-horizon, near-straight ego motion, while occupancy-grid resolution can produce false collision detections. These distributions and metric artifacts complicate reliable planning evaluation.
- Trajectory Distribution of nuScenes: Future 3s trajectory points are concentrated around straight motion, with most paths being straight or only slightly curved.
- Trajectory Distribution of nuScenes: Nearly 70% of heading angles lie between −0.2 and 0.2 radians, while curvature angles lie between −0.02 and 0.02 radians.
- Trajectory Distribution of nuScenes: nuScenes ego vehicles tend to move forward along straight lines and at small angles over short time horizons.
- Collision in Ground Truth: About 2% of ground-truth trajectory samples overlap occupancy-grid obstacles and are falsely detected as collisions.
- Collision in Ground Truth: A 0.5m occupancy grid falsely marks a ground-truth trajectory as colliding, whereas a 0.1m grid correctly identifies no collision.
- Collision in Ground Truth: At 0.6m grid resolution, collision samples comprise 4.8% of training data and 3.0% of validation data, showing sensitivity to grid size.
5. Conclusion and Limitations
The paper argues that a physical-state-only MLP can match perception-based planning performance on nuScenes, exposing limitations in the current evaluation scheme. It also acknowledges that this toy model is impractical for real-world autonomous driving.
- A simple MLP using only ego physical states achieves similar nuScenes performance to methods using visual or point-cloud perception.
- The result suggests that current nuScenes evaluation metrics may not adequately capture the superiority of different end-to-end planning methods.
- The model performs well within nuScenes but is acknowledged to be an impractical toy incapable of functioning in real-world scenarios.
- The authors state that driving using only the ego vehicle’s states is an insurmountable challenge.
- The authors believe perception-based planning methods will ultimately produce much safer trajectories than their toy model.