Source-linked AI summary
Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Driving
Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, Junchi Yan
TL;DR
Existing E2E-AD evaluation relies heavily on open-loop metrics or limited closed-loop routes, while inconsistent training data hinders fair comparison. Bench2Drive introduces a closed-loop benchmark with diverse interactive scenarios, short skill-focused routes, and official training data, and its evaluations reveal limitations of both L2 error and Drive Score. The benchmark provides multi-ability assessment and standardized conditions for comparing E2E-AD systems.
Problem
Existing E2E-AD evaluations use open-loop metrics that insufficiently reflect planning, limited CARLA routes, and nonstandard training data that hinder fair algorithm-level comparison.
Method
Bench2Drive combines a closed-loop evaluation toolkit with 44 interactive scenarios across 220 short routes and a standardized dataset of 2 million annotated frames from 13,638 clips.
Results
Bench2Drive evaluations confirm that L2 error does not reflect actual driving performance, while Drive Score lacks detail and encourages over-conservative strategies.
Takeaways & Limitations
Bench2Drive enables granular, comprehensive, realistic, and fair closed-loop comparison of E2E-AD driving capabilities.
Takeaways & Limitations
CARLA simulation rendering has gaps compared with the real world, so real-world datasets could provide complementary evaluation.
Abstract
from arXiv · showhide
In an era marked by the rapid scaling of foundation models, autonomous driving technologies are approaching a transformative threshold where end-to-end autonomous driving (E2E-AD) emerges due to its potential of scaling up in the data-driven manner. However, existing E2E-AD methods are mostly evaluated under the open-loop log-replay manner with L2 errors and collision rate as metrics (e.g., in nuScenes), which could not fully reflect the driving performance of algorithms as recently acknowledged in the community. For those E2E-AD methods evaluated under the closed-loop protocol, they are tested in fixed routes (e.g., Town05Long and Longest6 in CARLA) with the driving score as metrics, which is known for high variance due to the unsmoothed metric function and large randomness in the long route. Besides, these methods usually collect their own data for training, which makes algorithm-level fair comparison infeasible. To fulfill the paramount need of comprehensive, realistic, and fair testing environments for Full Self-Driving (FSD), we present Bench2Drive, the first benchmark for evaluating E2E-AD systems' multiple abilities in a closed-loop manner. Bench2Drive's official training data consists of 2 million fully annotated frames, collected from 13638 short clips uniformly distributed under 44 interactive scenarios (cut-in, overtaking, detour, etc), 23 weathers (sunny, foggy, rainy, etc), and 12 towns (urban, village, university, etc) in CARLA v2. Its evaluation protocol requires E2E-AD models to pass 44 interactive scenarios under different locations and weathers which sums up to 220 routes and thus provides a comprehensive and disentangled assessment about their driving capability under different situations. We implement state-of-the-art E2E-AD models and evaluate them in Bench2Drive, providing insights regarding current status and future directions.
1 Introduction
Bench2Drive addresses limitations in existing autonomous-driving evaluation by combining closed-loop testing, diverse interactive scenarios, short routes, and standardized training data. It is designed to provide granular, realistic, and fair assessment of E2E-AD capabilities.
- Open-loop metrics such as L2 error and collision rate are insufficient for evaluating planning because of distribution shift and causal confusion.
- Existing CARLA benchmarks mainly test basic driving skills and fail to examine behavior in complicated, interactive traffic.
- Bench2Drive evaluates 44 interactive scenarios across 220 short routes, isolating individual skills for detailed comparison.Each route is around 150 meters and contains a single specific scenario.
- The benchmark uses a standardized training set of 2 million fully annotated frames from 13,638 clips across diverse scenarios, weathers, and towns.Shared training conditions support algorithm-level comparison.
- Closed-loop evaluation lets system actions influence the environment, while short routes provide more smoothed evaluations.
- Bench2Drive exposes shortcomings in existing evaluation: L2 error can misrepresent driving performance, while Drive Score lacks detail and can encourage over-conservative strategies.
2 Related Work
Prior planning benchmarks span specialized datasets, integrated platforms, and CARLA-based closed-loop evaluations, but leave gaps in multi-ability assessment and fair algorithm-level comparison. Bench2Drive combines closed-loop multi-ability analysis with official training data to address these shortcomings.
- Planning benchmarks evolved from specialized datasets to integrated platforms that evaluate multiple synergistic system components.
- nuScenes provides open-loop metrics but has been criticized for inadequately evaluating planning capabilities.
- Bench2Drive is presented as the only listed benchmark combining closed-loop evaluation with multi-ability analysis.
- CARLA Leaderboard V1 assesses basic skills, while Leaderboard V2 lacks expert demonstration data for standardized training.
- Existing E2E-AD methods are commonly evaluated either open-loop or in relatively simple CARLA scenes such as Town05Long and Longest6.
3 Bench2Drive
Bench2Drive combines an official expert dataset with a closed-loop, multi-ability evaluation framework for granular and fair assessment of E2E-AD systems. Its 220 short routes span interactive scenarios, towns, and weathers, while additional metrics assess goal achievement, efficiency, and smoothness.
- 3.1 Data Collection Agent: Bench2Drive provides a large-scale, fully annotated CARLA dataset, an evaluation toolkit, and implementations of state-of-the-art E2E-AD methods.The dataset includes expert data and annotations intended to support training and comparison across the benchmark.
- 3.1 Data Collection Agent: Think2Drive serves as the expert model for collecting student-training data because it can solve all 44 scenarios used to construct Bench2Drive.The expert model uses privileged simulation information to navigate CARLA and collect data.
- 3.2 Expert Dataset: The official training set includes annotations such as 3D bounding boxes, depth, semantic segmentation, HD-Map data, and Think2Drive value estimations and features.The sensor setup includes LiDAR, cameras, radar, IMU/GNSS, a BEV camera, and HD-Map information.
- 3.3 Multi-Ability Evaluation: Bench2Drive uses 44 scenarios and five short routes per scenario across different towns and weathers, producing 220 routes for isolated-skill assessment.The short-route design reduces variance and supports reporting five advanced urban-driving skills: merging, overtaking, give way, traffic sign, and emergency brake.
- 3.3 Multi-Ability Evaluation: The evaluation reports Success Rate and Driving Score, then adds efficiency and smoothness metrics to assess trajectory quality beyond route completion.Efficiency compares ego speed with nearby-vehicle speed, while smoothness is evaluated through trajectory segments rather than only the entire trajectory.
4 Experiments
Bench2Drive experiments compare implemented E2E-AD baselines under open-loop, closed-loop, and multi-ability evaluation. The results show that closed-loop assessment reveals differences obscured by L2 error, while expert feature distillation helps and strong interaction remains difficult.
- Baselines and training: The study implements TCP, ThinkTwice, DriveAdapter, UniAD, VAD, and AD-MLP as baseline E2E-AD methods for Bench2Drive.The implementations cover methods using camera and ego-state inputs, explicit perception and prediction, vectorized scene representations, coarse-to-fine planning, and expert-feature distillation.
- Baselines and training: The baselines are trained on 950 clips and evaluated with 50 held-out clips, with balanced weather and coverage of all 44 scenarios in validation.AD-MLP and TCP use 1 A6000, while ThinkTwice, DriveAdapter, UniAD, and VAD use 8 A100 GPUs.
- Open-loop and closed-loop results: Low L2 error does not guarantee strong closed-loop driving: UniAD-base has lower L2 error than VAD but worse closed-loop performance, while AD-MLP performs poorly in both settings.The experiments describe L2 error as useful for checking convergence and fitting, but insufficient for advanced model comparison because it overlooks distribution shift and causal confusion.
- Open-loop and closed-loop results: Expert feature distillation improves performance: TCP, ThinkTwice, and DriveAdapter outperform VAD and UniAD by a large margin.The paper attributes this comparison to expert features providing driving knowledge that can mitigate overfitting in the high-dimensional autonomous-driving input space.
- Multi-ability results: All models achieve unsatisfying scores on strong-interaction skills including merging, overtaking, and emergency braking.The paper links this difficulty to the long-tail scarcity of interactive frames and the limitations of directly supervised imitation learning for interaction reasoning.
- Data analysis: Bench2Drive contains more turning trajectories, increasing action diversity and reducing the gap between open-loop and closed-loop evaluation.The paper presents this trajectory distribution as a source of better training data.
5 Conclusion
Bench2Drive provides a closed-loop benchmark, training dataset, and evaluation toolkit for granular assessment of E2E-AD methods. The paper also identifies simulation-rendering gaps and positions real-world datasets as complementary.
- Bench2Drive is presented as a benchmark, fully annotated training set, and evaluation toolkit for closed-loop E2E-AD assessment.The benchmark evaluates state-of-the-art methods and examines their strengths and weaknesses.
- CARLA rendering gaps relative to the real world make real-world datasets complementary to Bench2Drive.
- The dataset comparison frames real-world datasets as realistic, while simulation offers reactive behavior but can have cartoon-style rendering.
- Diffusion models may provide realistic and reactive rendering, but their illusion and artifact issues require further exploration.
- Bench2Drive is intended to complement rather than replace real-world testing because simulation may omit rare edge cases, unexpected human behavior, and environmental variation.
Checklist
The checklist records affirmative responses for claims, limitations, societal impacts, reproducibility materials, training details, asset licensing, and new-asset release, with nonapplicable items marked accordingly.
- The authors report that their main claims reflect the paper’s contributions and scope and that the work’s limitations are described.
- The checklist states that potential negative societal impacts were discussed and that the ethics review guidelines were followed.
- The authors report providing code, data, instructions, and training details needed to reproduce the experimental results.
- The checklist marks error-bar reporting as not applicable and reports that compute resources were specified.
- Existing assets are cited and licensed, new assets are included, and consent and personally identifiable information questions are marked not applicable.
- Crowdsourcing and human-subjects questions, including participant instructions, risks, wages, and compensation, are marked not applicable.
A Details of Data Collecting
Bench2Drive data collection combines automatic pipelines with manual checking, expert-model route filtering, CARLA API annotations, object categorization, coordinate handling, map metadata, trigger volumes, and compressed storage.
- Data collection combines automatic pipelines with manual checking.
- Route: Think2Drive runs predefined routes, and clips with infractions are discarded while traversal and manual checks promote coverage across towns, weathers, and scenarios.
- Annotations: CARLA API annotation bugs are corrected by manually computing pedestrian speeds, padding missing Speedometer and IMU values with zero, and compensating for missing stop-sign boxes.
- Object Class: Objects are grouped into vehicles, traffic signs, traffic lights, and pedestrians, with vehicles further divided into static and dynamic categories.
- Coordinate System: CARLA uses a Z-up left-hand Unreal Engine coordinate system, requiring compass-yaw conversion and filtering of rare NaN values.
- Data Compression: Images use JPG compression, segmentation and depth use PNG, LiDAR uses laszip, and JSON uses GZIP; inference also applies in-memory JPG compression and decompression.
B Distribution of Scenarios, Towns and Weathers
Bench2Drive’s weather distribution is nearly uniform, whereas town data is dominated by Town12 and Town13 because those towns are larger and host many newer scenarios.
- The dataset’s weather distribution is nearly uniform, while its town distribution is dominated by Town12 and Town13.
- More data comes from Town12 and Town13 because CARLA designed these newer towns to be much larger, offering more diverse landscapes.
- The town imbalance also reflects the concentration of recently designed Leaderboard v2 scenarios in newer towns.
C Implementation Details of Baselines
Baseline implementations follow official open-source code, environments, and configurations, with targeted changes for CARLA3 compatibility, training scale, and fair sensor comparisons.
- Detection classes are changed to match CARLA3 for methods that include object-detection modules.
- Bench2Drive data are collected at 10Hz, producing approximately 10× more frames than nuScenes while also having higher redundancy.
- Computationally demanding methods receive training adjustments to account for the differing data scale and training-step budget.
- ThinkTwice and DriveAdapter use expert BEV features regenerated by the Think2Drive expert.
- For fair comparison, ThinkTwice and DriveAdapter are modified to use six cameras without LiDAR.
D Training and Evaluation Resource Requirements
The benchmark documents training and evaluation resources while specifying NPC behavior parameters and rule-based interactions that shape closed-loop traffic conditions.
- Resource requirements: Evaluation on 220 routes can be parallelized across more GPUs, yielding a linearly faster evaluation process.
- NPC behavior: NPC behavior modes are cautious, normal, and aggressive, controlling speed, responses to other vehicles, and safety protocols.
- NPC parameters: The behavior configuration includes maximum speed, speed-limit offset, deceleration behind slower vehicles, collision-time estimates, proximity thresholds, emergency braking distance, and tailgating counters.
- NPC interaction: NPCs interact with the ego vehicle through behavior designs that determine how they respond to its presence and actions.
- Behavior algorithm: The rule-based NPC algorithm updates surrounding information, stops for red lights, stop signs, pedestrians, or nearby vehicles, and otherwise performs car following or intersection control.
F Details about Infraction Score
Bench2Drive defines infractions and evaluates behavior across 44 corner scenarios spanning obstacle avoidance, interactions with traffic participants, junction negotiation, and highway maneuvers.
- Infraction-triggering interactions: The NPC decision procedure prioritizes emergency stops for red lights, stop signs, pedestrians, and vehicles within braking distance, otherwise using following or PID control.
- Scenario and scoring scope: Bench2Drive extends CARLA Leaderboard V2 with 44 corner scenarios, whose infractions are assigned penalties under the Leaderboard-based scoring design.
- Obstacle avoidance: Obstacle scenarios require lane changes or route deviations around parked vehicles, construction, accident vehicles, and slow-moving hazards in same- or opposite-direction traffic.
- Dynamic hazards: Other scenarios test prompt braking or maneuvering for opening doors, sudden pedestrian or bicycle crossings, stopped vehicles, emergency vehicles, and opposing traffic incursions.
- Junction interactions: Junction scenarios cover pedestrian priority, unprotected turns, signalized and non-signalized negotiation, red-light violations by other vehicles, and traffic-flow merging.
- Highway interactions: Highway and interurban scenarios require crossing or merging with traffic when entering or exiting roads, including responses to vehicles merging from ramps.
J Datasheet
The datasheet positions Bench2Drive as a publicly distributed, maintained benchmark for granular closed-loop E2E-AD assessment, built from automatically collected CARLA clips and reusable annotations.
- Purpose: Bench2Drive was created to provide comprehensive and realistic FSD testing with a closed-loop, granular assessment of E2E-AD driving skills.
- Creators: The dataset and benchmark were curated by ReThinkLab researchers at Shanghai Jiao Tong University.
- Distribution: The release includes data, code, and checkpoints through GitHub and Hugging Face.
- Maintenance: The authors and potential new ReThinkLab members maintain the dataset, including future updates and continued support for older versions.
- Composition: Each clip contains hundreds or thousands of 10Hz frames with raw sensor information and annotations, and Bench2Drive contains 13,638 clips.
- Organization: Clips are annotated with scenario types and locations, enabling relationships among clips sharing scenarios or nearby locations.
- Data provenance: The dataset is self-contained and collected automatically in CARLA using code written by the authors.
- Additional uses: Existing annotations support 3D detection, segmentation, depth estimation, tracking, and motion prediction in addition to end-to-end driving.