Source-linked AI summary
Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol
Thomas Schwarzfischer, Emad Aghajanzadeh, Moritz Thoma, Nael Fasfous, Manoj Rohit Vemparala, Pierpaolo Mori, Daniel Mueller-Gritschneder, Ulf Schlichtmann
TL;DR
Closed-loop evaluation is intended to assess E2E driving more faithfully than open-loop evaluation, but Bench2Drive’s protocol can distort model assessment. The paper identifies comfort and navigation-signal pitfalls, proposes corrections, and finds that the reported comfort tradeoff is no longer detectable at matched speed.
Problem
Bench2Drive’s evaluation protocol can misrepresent E2E driving models through its comfort metric and navigation-signal handling.
Source: §I · p. 1Method
The authors correct comfort scoring with distance-based segmentation and propose navigation fixes that preserve model steering and align target-point sampling with training.
Source: §I · pp. 1–2Results
The corrected comfort metric cuts correlation with stationary time from r = 0.97 to r = 0.06, and the driving-score/comfort tradeoff is no longer detectable at matched speed.
Source: §V · p. 7Takeaways & Limitations
Corrected comfort scores can be recomputed from logged trajectories without rerunning closed-loop simulation, and navigation fixes produce sharply different effects on UniAD and VAD.
Source: A. Evaluation Protocol · p. 6, §V · p. 8
Abstract
from arXiv · showhide
Closed-loop evaluation is widely regarded as a more faithful way to assess end-to-end (E2E) driving models than open-loop evaluation, and Bench2Drive, built on the CARLA simulator, has become the dominant benchmark for it. We show that its evaluation protocol itself contains pitfalls that can misrepresent the models scored on it. Bench2Drive's comfort metric, adapted from nuPlan, has several implementation errors and a segmentation scheme that rewards standing still, so it mostly measures how long a vehicle stays stationary, correlating with stationary time at r = 0.97 across eight models. We propose a corrected, distance-based metric that cuts that correlation to r = 0.06. Under it, the strong driving-score/comfort tradeoff the original metric reports is no longer detectable once models are compared at matched driving speed, which suggests that comfort trades against how fast a model drives rather than against how well it drives. We also uncover two pitfalls in how the harness supplies navigation signals: the default controller can replace a model's own steering with an angle toward the route target point, and the target point supplied at evaluation time can differ substantially from what a model saw in training. We demonstrate both on UniAD and VAD, where their effects differ sharply, and argue that future closed-loop evaluations should check for them. We will release all fixes as standalone code, so corrected comfort scores can be obtained from logged trajectories without re-running closed-loop simulation.
I. INTRODUCTION
Bench2Drive’s comfort score is distorted by implementation errors and time-based segmentation that rewards standing still, while its navigation harness can also override model steering and supply target points unlike those seen in training.
I. INTRODUCTION
- The original comfort score tracks stationary time at r = 0.97 across eight models; distance-based scoring cuts this correlation to r = 0.06.Fixed one-second chunks reward standing still, even after implementation errors are corrected.Source: §I · p. 1
- The default controller can replace model steering with an angle toward the route target point, and evaluation target points can differ substantially from training inputs.The paper proposes a controller that preserves predicted steering and a route planner matching training-time target sampling.Source: §I · p. 1
- The authors re-benchmark comfort across eight E2E models, demonstrate navigation fixes on UniAD and VAD, and plan standalone code for recomputing comfort from existing logs.Corrected comfort scores can be obtained without rerunning closed-loop simulation.Source: §I · p. 2
II. RELATED WORK
Prior work spans varied E2E architectures and evaluation protocols, while this paper examines whether closed-loop harness choices and comfort calculations faithfully assess driving.
A. End-to-end driving architectures
- E2E architectures range from shared-BEV perception, prediction, and planning systems to transformer sensor fusion, trajectory-vocabulary scoring, and language-model planners.UniAD and VAD use shared BEV representations, while VAD uses vectorized scenes instead of rasterized ones.Source: A. End-to-end driving architectures · p. 2
B. Evaluation of End-to-end models
- Closed-loop evaluation reacts to a policy’s actions at every step, unlike replay-based open-loop evaluation, but real-data approaches still score plans that are not revised.CARLA fully closes the loop and renders future sensor scenes; Bench2Drive uses it for edge-case routes scored on driving score, success, efficiency, and comfort.Source: B. Evaluation of End-to-end models · p. 2
- Navigation signals may be discrete commands or continuous target points, and harness-supplied target distributions and steering can differ from model training or override model control.Prior work studies navigation-signal integration and target-point bias; this paper focuses on how the evaluation harness supplies the signal.Source: B. Evaluation of End-to-end models · p. 2
- NuPlan-style comfort checks require six kinematic quantities to stay within expert-calibrated thresholds, but prior human-perception work does not test whether closed-loop ports preserve that calibration.The paper identifies implementation details and segmentation choice as dominant influences on the score.Source: B. Evaluation of End-to-end models · p. 2
III. EVALUATION PITFALLS AND CORRECTIONS
The paper examines navigation-related pitfalls during closed-loop decision-making and a comfort metric computed afterward; here it details steering overruling in Bench2Drive’s default controller.
A. Steering Overruling in B2D’s Controller (SO)
- Steering overruling selects the route-target angle when its angle is smaller or tracking error is large, replacing model steering in over half of frames in some scenarios.This can hide compounding prediction errors and suppress evasive maneuvers by substituting a shallower target-point angle.Source: A. Steering Overruling in B2D’s… · p. 3
- The authors test steering overruling by disabling it for UniAD and VAD on Dev10 and compare results with a constant-velocity route-following baseline.Dev10 is Bench2Drive’s 10-route development subset, with one route per scenario category.Source: A. Steering Overruling in B2D’s… · p. 3
- Removing steering overruling drops UniAD’s driving score from 53.42 to 19.76 and success rate to zero, while VAD remains essentially unchanged at 52.37 to 52.49.A constant-velocity policy scores 48.57, within five points of both models under the default protocol.Source: A. Steering Overruling in B2D’s… · p. 3
- Because the controller masks models by different amounts, default-controller scores are not comparable across models; the authors recommend reporting scores with steering overruling removed.A default score alone does not reveal how much of the driving the model performed itself.Source: A. Steering Overruling in B2D’s… · p. 3
B. Target Point Mismatch
The navigation harness determines both the target point and accompanying road command from route sampling; training and evaluation differ, and an aligned planner only partly resolves the mismatch for VAD.
B. Target Point Mismatch
- The harness reads target points and road commands from one polyline index, so route representation and index advancement govern both signals.The authors identify these as the two choices that differ between data collection and evaluation.Source: B. Target Point Mismatch · p. 3
- Training targets have a median distance of 4.94 m, while the original evaluation planner emits targets roughly an order of magnitude farther ahead with jumps of tens of meters.The mismatch affects both target points and the road commands attached to them, pushing models toward inputs absent from training.Source: B. Target Point Mismatch · p. 3
- The aligned planner follows the dense route polyline with a forward-only cursor and a 4.06 m standoff chosen from the fifth percentile of training target distances.Advancing the cursor in a loop makes the standoff independent of control frequency.Source: B. Target Point Mismatch · p. 3
- The aligned planner only partly resolves VAD’s mismatch: its target-distance p95 shifts from 48.8 m to 45.6 m, while behind-ego frames rise from 0.6% to 12.7%.Those frames carry targets 44 m to 53 m away because the ego has left the intended path and the route runs behind it.Source: B. Target Point Mismatch · p. 4
- On top of the steering-overruling fix, target-point alignment raises UniAD’s driving score from 19.76 to 26.06 and VAD’s from 52.49 to 59.32.VAD rises above its default score, while UniAD reaches roughly half of what the default protocol credits it with.Source: B. Target Point Mismatch · p. 4
C. Comfort
Bench2Drive reports comfort as the share of comfortable route segments, but implementation errors and fixed-time segmentation make the metric poorly reflect comfort while driving. The metric can instead reward standing still, and the section also illustrates pitfalls in how evaluation supplies navigation signals.
1) Bench2Drive’s comfort metric:
- Bench2Drive compares six kinematic signals with human-calibrated thresholds and averages route comfort scores based on one-second segments.A segment passes only when every frame stays within every defined threshold.Source: 1) Bench2Drive’s comfort metric · p. 4
C. Comfort
- Figure 1 depicts steering overruling, where the controller substitutes target-point steering, and target-point mismatch, where evaluation commands switch at a different route location than in training.The illustrated mismatch combines a straight command with a heading and road geometry supporting completion of a right turn, a combination absent from training.Source: Figure 1 · p. 4
1) Bench2Drive’s comfort metric:
- Four implementation errors distort comfort measurements: mismatched sampling rate, undersized smoothing, invalid yaw processing, and unconverted yaw units.These errors inflate jerk and yaw violations, including when motion is smooth or genuinely comfortable.Source: 1) Bench2Drive’s comfort metric · pp. 4–5
- Fixed one-second segments let stopped vehicles accumulate comfortable segments indefinitely, so the score can reward getting stuck rather than comfortable driving.A stuck route contributes as many comfort segments as a completed route despite covering less distance.Source: 1) Bench2Drive’s comfort metric · p. 5
2) Proposed Metric:
The proposed comfort metric retains the original thresholds while correcting calculation, filtering, and segmentation. It uses route-distance bins so stopped time does not add comfort segments.
2) Proposed Metric:
- The corrected metric keeps nuPlan’s physical smoothing spans at CARLA’s 20 Hz by using 16-frame acceleration and 30-frame jerk windows, converting yaw to radians, and differentiating yaw rate.The corrected time step makes the resulting signals comparable to the values used to calibrate nuPlan’s thresholds.Source: 2) Proposed Metric · p. 5
- A single Savitzky-Golay filter is applied across the whole episode before segmentation, avoiding chunk-boundary artifacts and using CARLA’s 0.05 s derivative step.The filter uses polynomial order 2, matching nuPlan’s.Source: 2) Proposed Metric · p. 5
- The metric divides each route into 100 equal-distance bins and scores comfort over distance actually driven, so standing still advances no bin.Changing the bin count changes absolute scores, but rankings remain identical for the tested counts 10, 20, 50, and 100 (Spearman ρ = 1.00).Source: 2) Proposed Metric · p. 5
A. Evaluation Protocol
The evaluation covers eight models on Bench2Drive’s 220-route benchmark, using officially released checkpoints and new closed-loop runs. The authors distinguish which models the two navigation pitfalls can affect and limit the mismatch demonstration to UniAD and VAD.
A. Evaluation Protocol
- Eight models are evaluated on Bench2Drive’s full 220-route benchmark, spanning language-based, generative, trajectory-scoring, reinforcement-learned, and rule-based systems.The models are UniAD, VAD, SparseDriveV2, TFv6, Orion, SimLingo, CaRL, and PDM-Lite.Source: A. Evaluation Protocol · p. 5
- The authors rerun officially released checkpoints because the comfort variants require per-frame ego-state logs that released results do not retain or report.The correction itself can be recomputed from retained logs without rerunning simulation.Source: A. Evaluation Protocol · p. 6
- Steering overruling applies only to models whose released code gives predicted waypoints to Bench2Drive’s default controller; target-point mismatch exposure for the other six models remains unaudited.The authors correct steering overruling for UniAD and VAD, and demonstrate target-point mismatch on those two models.Source: A. Evaluation Protocol · p. 6
B. Traditional Comfort Reveals a Measurement Bug
Bench2Drive’s traditional comfort score tracks stationary time closely, while corrected calculations allow comfortable driving to count. Distance-based segmentation removes the stationary-time incentive, but its reduced correlation with stationary time is a designed property, not independent validation of comfort.
B. Traditional Comfort Reveals a Measurement Bug
- r = 0.966 between traditional comfort and stationary time across eight models; UniAD and VAD have the highest traditional comfort scores and stationary times but the lowest driving scores.Their traditional comfort scores are 39.28 and 45.95, and their stationary times are 45.67% and 47.64%, respectively.Source: B. Traditional Comfort Reveals a… · p. 6
C. Corrected Comfort Metric
- Fixing implementation errors and smoothing before segmentation lets the corrected implementation recognize comfortable driving, raising scores across the models.Time-based segmentation still rewards stationary periods, regardless of whether the calculations are corrected.Source: C. Corrected Comfort Metric · p. 6
- UniAD, VAD, and SimLingo score lower with distance-based segmentation than with the fixed time-based implementation because stopped time no longer earns comfort credit.In the SimLingo example, the distance-based score stays flat when stopped, while the traditional score rises in stationary periods and falls during motion.Source: C. Corrected Comfort Metric · p. 6, Figure 3 · p. 6
- Distance-based segmentation reduces the stationary-time correlation from r = 0.97 traditionally and r = 0.66 after implementation fixes to r = 0.06 (p = 0.88).The authors treat this as confirmation that the correction behaves as designed, not independent evidence that it measures comfort better.Source: C. Corrected Comfort Metric · p. 6
D. A Sanity Check Against Efficiency
The distance-based comfort metric recovers an association with efficiency that the traditional metric misses. At matched driving speed, the corrected metric shows no driving-score relationship, unlike the traditional metric.
D. A Sanity Check Against Efficiency
- Efficiency measures speed relative to surrounding traffic independently of comfort, providing a low-cost sanity check; eight-model correlations are indicative rather than conclusive.Table II reports traditional, fixed-implementation, and distance-based comfort results across 220 Bench2Drive routes.Source: D. A Sanity Check Against Efficiency · p. 6, Table II · p. 7
- r = −0.69 (p = 0.054), distance-based comfort correlates negatively with efficiency, unlike traditional comfort (r = −0.41, p = 0.31).The distance-based correlation remains r = −0.68 when PDM-Lite is excluded; Spearman ρ = −0.76 (p = 0.03).Source: D. A Sanity Check Against Efficiency · p. 7
- At matched speed, corrected comfort shows no driving-score relationship (r = +0.45, p = 0.32), while traditional comfort remains strongly negative (r = −0.80, p = 0.03).The raw correlations are confounded because efficiency and driving score correlate at r = +0.85 (p = 0.007).Source: D. A Sanity Check Against Efficiency · p. 7
E. Cross-Model Insights
The corrected, distance-based metric changes the apparent comfort–driving-score tradeoff: at matched speed, the strong negative relationship reported by the traditional metric is no longer detectable. Cross-model patterns also distinguish planners with similar designs and show that navigation-signal pitfalls affect UniAD and VAD differently.
E. Cross-Model Insights
- The two language-model planners occupy opposite corrected-comfort extremes, pointing to trajectory decoders rather than language-model presence as the apparent determinant of ride quality.Orion is highest at 75.98 and SimLingo lowest at 25.86; the authors also report that CaRL has the highest speed-adjusted comfort (+17.9) despite no comfort term in its reward.Source: E. Cross-Model Insights · p. 7
- Fig. 4 shows corrected comfort’s negative relationship with efficiency, while at matched speed it leaves no relationship with driving score.For the speed-adjusted comparison, the traditional metric remains strongly negative, while the corrected metric’s slope is consistent with no relationship at n = 8.Source: Figure 4 · p. 7
- At matched speeds, SparseDriveV2 is 21.1 points more comfortable but scores 7.3 points lower than TFv6.The authors conclude that comfort is neither a byproduct of driving competence nor strictly opposed to it once speed is held fixed.Source: E. Cross-Model Insights · p. 7
V. CONCLUSIONS
- At matched speed, the strong driving-score/comfort tradeoff reported by the traditional metric is no longer detectable with the corrected metric.The traditional metric tracks stationary time at r = 0.97, compared with r = 0.06 for the corrected metric.Source: §V · p. 7, Figure 4 · p. 7
- The harness can overwrite a model’s steering and supply a target point outside its training distribution; these pitfalls affect UniAD and VAD differently.The authors argue that scores under the default protocol are therefore not comparable across models.Source: §V · p. 8