When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving

• Driving benchmarks provide a common basis for comparing progress toward safe, comfortable, efficient, and rule-following driving goals.Source: §1 · p. 1
• As benchmark scores become rewards or trajectory-selection objectives, the introduction asks whether gains still reflect improvements in the behaviors those scores intend to measure.The paper calls this property metric validity.Source: §1 · p. 1
• Prior findings establish score–behavior discrepancies and open- versus closed-loop gaps, but leave unclear where discrepancies arise in evaluation and how optimization exposes them.Source: §1 · p. 1

2 SCORE–BEHAVIOR ANALYSIS

• Metric validity asks whether an optimization score gain is accompanied by improvement in behavior-specific diagnostics, rather than treating one aggregate as a complete measure of driving quality.For measured quantity F, the section defines its change as the expectation under optimized outputs minus that under baseline outputs.Source: §2 · p. 2
• The scoring chain comprises execution, measurement, subscore mapping, and aggregation, creating blind spots through omitted behaviors, insensitive mappings, or transformations of requested motion.The section tests predicted insensitivity and checks whether the corresponding behavior deteriorates in optimized outputs despite a score gain.Source: §2 · p. 2
• Omission makes a component unable to change the aggregate when other inputs are fixed; matched trajectory probes test this invariance while controlling other scored properties.For included components, aggregate gains may also reflect trade-offs or changed gate pass rates, so component and conditional measurements distinguish those contributions.Source: §2 · p. 2
• Thresholds and clipping create flat score regions where behavior can change without a score response; continuous readouts reveal within-region changes, while rescoring tests resolution.Finite differences handle boundaries.Source: §2 · pp. 2–3
• Execution can attenuate requested-motion changes before measurement, while the subscore mapping can flatten the remaining response; the combined test does not isolate each stage’s contribution.The section compares requested and executed motion under smoothing and retiming interventions.Source: §2 · p. 3
• Closed-loop tests compare planners across execution settings and inspect later requests under replanning; a nonzero interaction alone does not establish a gain reversal.A reversal requires opposite signs of the score gains in the two settings.Source: §2 · p. 3
• Score insensitivity permits mismatch, but optimization objectives and planner constraints also shape behavior, so controlled interventions and optimized outputs are examined separately.Source: §2 · p. 3

3.1 EXPERIMENTAL SETUP

• The experiments use official NAVSIM scoring across NAVSIM and WorldEngine evaluations and assess Plan-R1 on nuPlan val14 under non-reactive and reactive traffic.The setup distinguishes fixed-output EPDMS rescoring from the NAVSIM-v2 two-stage protocol, which generates new predictions at original and synthetic states.Source: §3.1 · pp. 3–4

3.2 OMISSION: UNSCORED DIRECTION

• ReCogDrive’s RL checkpoints improve aggregate scores while direction compliance falls; TOAD’s highest-predicted-score candidates show the same deterioration, whereas NoRD is a counterexample.Under the released v2 objective, deployed TOAD compliance increases slightly on original and synthetic scenes.Source: §3.2 · p. 4

3.3 THRESHOLDING: MOTION WITHIN PASSING REGIONS

• Thresholded comfort leaves continuous motion differences unresolved, and Plan-R1’s mean absolute jerk peak increases after RL in both traffic regimes despite improvements in several other measures.Both jerk changes pass paired statistical checks; the binary comfort term has no supported mean change in non-reactive traffic and decreases slightly in reactive traffic.Source: §3.3 · pp. 4–5

3.4 EXECUTION TRANSFORMATION: REQUESTED VERSUS SCORED MOTION

• ReCogDrive-2B’s planned acceleration peak rises from 1.14 to 2.84 m/s2 after RL, while its executed longitudinal peak changes from 0.88 to 1.20 m/s2.Qwen-Drive RL’s planned acceleration/deceleration peaks are 1.65/1.88 m/s2, versus simulated peaks of 0.28/0.53; the different motion layers do not directly measure discarded requests.Source: §3.4 · p. 6
• Smoothing DiffusionDrive v2’s positions removes 70–74% of heading jitter on navhard and WorldEngine while mean PDMS changes by less than 0.01 in magnitude.The oscillatory illustrative plan scores 0.979, versus 0.874 for v1 and 0.878 for Human, although all three pass comfort after tracking.Source: §3.4 · p. 6, Figure 2 · p. 7
• Retiming Qwen-Drive’s RL plan reduces peak speed change per unit time by nearly half while mean PDMS changes by only −0.0002.The intervention preserves endpoint speeds and total path length; Qwen-Drive’s increased smoothing displacement is concentrated in longitudinal waypoint spacing, and its plan-adherence error improves.Source: §3.4 · p. 6
• NoRD improves PDMS while reducing jitter and smoothing displacement, whereas TOAD reduces jitter but slightly increases smoothing displacement and adherence error.Requested-motion diagnostics distinguish plan quality from controller accommodation; execution scores alone cannot establish both.Source: §3.4 · p. 6
• Under NAVSIM-v2’s two-stage evaluation, ReCogDrive and Qwen-Drive improve drivable-area compliance and progress in both stages while extended comfort falls, but all three combined-score changes have intervals containing zero.NoRD improves all displayed components and retains a positive combined gain; broader coverage exposes trade-offs rather than ensuring every behavior improves with the aggregate.Source: §3.4 · p. 7, Table 3 · p. 7

4 RL GAINS DEPEND ON THE EXECUTION INTERFACE

• The matched comparison holds traffic, scoring, rendering, and inference seeds fixed, but the contribution of specific filtered motion components to reversal remains unisolated.Initial predictions agree exactly across interfaces for each checkpoint.Source: §4 · pp. 8–9
• Changing playback to LQR reverses RL gain signs for both ReCogDrive sizes and Qwen-Drive; NoRD and TOAD retain positive gains under both interfaces.The interaction intervals exclude zero for the three reversing pairs; Qwen-Drive’s change in RL gain is +0.525.Source: §4 · p. 8, Table 4 · p. 8
• With Qwen-Drive’s first plan frozen, replacing LQR with playback changes the RL score by only −0.005, with a 95% interval spanning zero.This control distinguishes fixed-request execution from repeated replanning; the controls are not an additive causal decomposition.Source: §4 · p. 8
• In earlier playback rollouts, 82.6% of 2,296 RL plans predict slowing, yet all seven same-time revisions are positive across 196 RL scenes.Mean revisions are +2.184 m/s for RL and −0.262 m/s for IL, showing how within-plan slowing can coexist with upward revisions during execution.Source: §4 · p. 8
• On the full interface cohort, Qwen-Drive’s RL same-time speed revision averages +2.17 m/s under playback versus +0.58 under LQR.The paired change is −1.58 m/s, with a log-bootstrap 95% interval of [−1.71, −1.45]; this describes feedback response without isolating its causal contribution to reversal.Source: §4 · p. 9

5 CONCLUSIONS & TAKE-AWAYS

• Metric validity under optimization cannot be inferred from aggregate score gains alone because behavioral distinctions may be lost through omission, thresholding, or execution transformations.Gains can reverse across interfaces during repeated replanning, so their benefit depends on how requests are executed and returned as feedback.Source: §5 · p. 9
• Analyze the scoring chain before optimization by identifying omitted behaviors, score-insensitive differences, and execution changes, then testing blind spots with controlled interventions..Source: §5 · p. 9
• Preserve behavioral quantities behind aggregate scores by reporting continuous measurements, separating requested from executed motion, and pairing progress with safety constraints.Source: §5 · p. 9
• Test whether gains transfer when execution changes and planning repeatedly responds to resulting states, while tracking safety-critical failures.Source: §5 · p. 9
• Revalidate the complete scoring process because changing scoring rules alone does not address behavior altered or removed by execution.Check whether the revised process captures meaningful behavioral differences and whether optimizing it produces better behavior.Source: §5 · p. 9

A.1 SCENE SETS AND COMPARISONS

• Open-loop scoring tracks four-second plans with LQR and a kinematic bicycle model, while WorldEngine returns updated observations for replanning every 0.5 seconds.The primary execution-interface comparison uses 288 scenes per condition; earlier playback comparisons use separate historical cohorts.Source: §A.1 · p. 11

A.2 BEHAVIORAL MEASUREMENTS

• Geometry diagnostics include smoothing displacement, heading jitter, path length, and plan-adherence error; heading jitter is a sum in radians, not a time derivative.Plan adherence compares interpolated requested positions with tracked positions; nuPlan comparisons also include later execution after replanning.Source: §A.2 · p. 12
• Pace measurements distinguish planned from simulated motion, while outcome rates count at-fault collisions or road departures across matched scenes.Reported progress is decomposed into pass frequency and conditional progress; comparisons conditional on both arms passing use their intersecting passing set.Source: §A.2 · pp. 12–13
• Comfort evaluation covers six channels, with binary checks and continuous bound-utilization measures applied to executed motion.The channels include longitudinal and lateral acceleration, jerk magnitude, longitudinal jerk, yaw acceleration, and yaw rate.Source: §A.2 · pp. 12–13
• Direction compliance uses the maximum accumulated displacement in a one-second oncoming-traffic window, mapped to scores of 1, 0.5, or 0 at stated thresholds.The v2 implementation removes intersection positions from the mask, so v1 and v2 DDC values are reported separately.Source: §A.2 · p. 13

A.3 DESCRIPTIVE PLANNER PROFILES

• Planner profiles average configuration-level ratios equally, with Human = 1 denoting equality to the logged-human mean and separate axes for score-influencing and zero-weight quantities.Planner composition differs across groups, so the profiles are descriptive.Source: §A.3 · p. 13

A.4 PAIRED STATISTICAL ANALYSIS

• Paired mean changes use scene-bootstrap and log-bootstrap 95% intervals, with Holm correction applied across the original analysis families.For fixed released checkpoints, intervals characterize evaluation-scene and driving-log variability; † marks cases with uncertain or unsupported changes.Source: §A.4 · p. 14

A.5 PLANNER-SPECIFIC SETTINGS

• Planner-specific comparisons include PDMS-reward fine-tuning, Qwen-Drive reinforcement objectives, and TOAD search against predicted PDMS with comfort and anchoring penalties.The supplied settings also specify Plan-R1’s 15-second scenarios, 10 Hz replanning, and LQR execution.Source: §A.5 · pp. 14–15

B.1 COMFORT AND TRACKER EFFECTS

• Binary comfort failed on 410 of 291,504 navtest evaluations (0.14%), while the largest within-scene comfort utilization averaged 0.32–0.38 of its threshold for the three navhard trajectories.For ReCogDrive-2B, planned absolute acceleration peak rose from 1.14 to 2.84 m/s2 while executed peak rose from 0.88 to 1.20 m/s2.Source: Figure 4 · p. 15, §B.1 · p. 15

B.2 CONTINUOUS COMFORT RESCORING

• Continuous comfort rescoring preserved the DiffusionDrive, DiffusionDriveV2, and Human ranking; the worst-channel margin retained 65% and 75% of DiffusionDriveV2’s advantage across the two sets.The comparison used fixed outputs and left the other PDMS components unchanged.Source: §B.2 · pp. 15–16, Table 5 · p. 16

B.3 DRIVING-DIRECTION CONSTRUCTION

• Across 42 matched pairs, wrong-way travel averaged 8.80 m while paired PDMS differed by +0.000019; DDC reweighting or gating yielded negative differences.Re-aggregation with DDC weight 2 or 5 gave −0.157 or −0.278, and using DDC as a gate gave −0.797.Source: §B.3 · p. 16

B.4 POSITIONAL SMOOTHING AND RE-TIMING

• Smoothing DiffusionDriveV2 positions removed 70% and 74% of heading jitter across the two sets, while mean PDMS changed by −0.003 and +0.007, respectively.Mean waypoint displacements were 0.417 and 0.409 m; the edit preserved output headings.Source: §B.4 · p. 16
• Re-timing reduced the RL checkpoint’s largest planned speed change from 2.46 to 1.31 m/s2, with mean PDMS change −0.0002 and an interval including zero.Re-timing smoothed segment speeds along each trajectory’s own path while preserving endpoint speeds and total path length.Source: Table 7 · p. 17, §B.4 · p. 17

B.5 PROGRESS AND CANDIDATE TIES

• NoRD’s +0.091 EP change accompanied a pass-rate rise from 81.7% to 92.3%, while conditional EP moved from 0.846 to 0.847.In the fixed 800-candidate pool, 274 scenes had tied PDMS maxima, with tied-set heading jitter spanning 0.43–7.29 radians.Source: §B.5 · p. 17

C.1 SCENE-SET REPLICATIONS

• Across navhard and the WorldEngine test set, ReCogDrive’s geometric changes and NoRD’s reductions in jitter and smoothing displacement recurred.Qwen-Drive again increased smoothing displacement while reducing plan-adherence error; GTRS-Dense changes remained small on these quantities.Source: §C.1 · pp. 17–18, Table 8 · p. 18

C.2 LOG-LEVEL UNCERTAINTY

• The TOAD search arm’s log interval included zero for PDMS, while its geometric changes had mixed directions; DiffusionDrive IL-to-RL and GTRS-Dense EPDMS intervals also included zero.The original correction family included the augmentation-only arm, whose PDMS change was approximately zero.Source: §C.2 · p. 18, Table 9 · p. 19

C.3 QUALITATIVE CASES AND SEARCH RESPONSE

• Six selected scenes show oscillatory DiffusionDriveV2 plans scoring above both v1 and Human, although all three pass comfort after tracking.The figure pairs cropped front-camera views with bird’s-eye views and reports smoothing displacement and PDMS.Source: §C.3 · p. 19, Figure 6 · p. 20
• Under the v2 search objective, deployed DDC rose from 0.9933 to 0.9944 on original scenes and from 0.9369 to 0.9388 on synthetic scenes.DDC was included in the target; the comparison used the same released checkpoint without search and with proposal augmentation and CEM.Source: §C.3 · p. 21
• Within one CEM round, TOAD’s highest-predicted-score candidate gained 0.008 official PDMS as DDC fell from 0.94 to 0.91; EPDMS fell from 0.809 to 0.795.Subsequent rounds produced little additional change in this candidate sequence; TOAD deploys the converged CEM mean subject to its acceptance rule.Source: Figure 7 · p. 21, §C.3 · p. 21

D.1 FULL MATCHED EXECUTION-INTERFACE COMPARISON

• The full comparison uses 1,152 scores per family, retains failures, and changes only the execution interface while holding checkpoints, scenes, inference, horizon, traffic, and scoring fixed.There are 576 identical initial-plan pairs per family; seeds are fixed, so intervals describe scene/log variability rather than training or seed variability.Source: §D.1 · p. 21
• The full NR results are distinct from earlier playback cohorts, and their open-loop or reactive results are not combined into a new paired contrast.Each four-condition comparison is within one host; NoRD's full comparison disables prefix caching for repeatable initial predictions.Source: §D.1 · pp. 21–22
• Under playback versus LQR, mean speed revisions were −0.273 versus −0.058 for IL and +2.167 versus +0.584 for RL; paired differences were +0.215 and −1.584, respectively.These are descriptive behavioral responses to the interface intervention, not causal mediation estimates.Source: §D.1 · p. 22
• Fixed-state image replacement and same-first-plan contrasts do not explain the rollout reversal on their own, motivating joint examination of execution and subsequent feedback.The differential contrasts are +0.008 [−0.037, +0.050] for image replacement and −0.034 [−0.075, +0.003] for the same-first-plan comparison.Source: §D.1 · p. 22
• A score-based gain need not improve requested-motion diagnostics or transfer across execution interfaces, and the observed interaction does not identify deliberate causal selection of a discarded behavior.Qwen-Drive's plan-adherence error improves after RL while TOAD's increases slightly, so a universal increase in request–execution distance is unsupported.Source: §D.1 · p. 22

D.2 HISTORICAL PLAYBACK COMPARISONS

• Under historical playback, ReCogDrive and Qwen-Drive reverse their open-loop ordering, while NoRD retains a gain and TOAD improves despite a small negative open-loop change.Table 12 reports open-loop, non-reactive closed-loop, and reactive closed-loop changes; Figure 9 displays NR scene-level distributions and paired means.Source: §D.2 · p. 23, Table 12 · p. 23, Figure 9 · p. 23
• In NR evaluation, better/worse scene counts were 62/167 for ReCogDrive-2B, 67/164 for ReCogDrive-8B, 76/153 for Qwen-Drive, 134/83 for NoRD, and 138/51 for TOAD.Ties are omitted; Figure 8 circles show paired means on the same NR-matched scenes, with equal marker sizes.Source: §D.2 · p. 23, Figure 8 · p. 23
• The NR ReCogDrive changes have paired-test p-values 2 × 10−17 for 2B and 4 × 10−12 for 8B; paired contrasts between regimes have p-values 4×10−22 and 2×10−12.The section also reports paired intervals for Qwen-Drive and TOAD and significant NR and reactive gains for NoRD.Source: §D.2 · p. 23
• Qwen-Drive’s NR PDMS falls from 0.780 to 0.400, with at-fault collisions rising from 11 to 72 and drivable-area exits from 28 to 111.Under reactive traffic, PDMS falls from 0.766 to 0.408; collisions rise from 13 to 57 and drivable-area exits from 28 to 118.Source: §D.2 · pp. 23–24
• NoRD’s zero-score count falls from 114 to 94, while TOAD’s falls from 46 to 23 in NR and 45 to 30 in reactive traffic.TOAD's NR drivable-area exits fall from 41 to 19, while at-fault collisions change from 5 to 4.Source: §D.2 · p. 24

D.3 SUCCESSIVE PREDICTIONS AND ROLLOUT DIAGNOSTICS

• Qwen-Drive’s 1,897/2,296 RL plans (82.6%) predict slowing, yet all seven same-time revisions are positive in 196 RL scenes versus one IL scene.Mean revisions are +2.184 m/s for RL and −0.262 m/s for IL; a selected RL rollout rises from 4.939 to 17.121 m/s and leaves the road at 4 s.Source: §D.3 · p. 24
• In the selected continuation, IL remains within the official drivable area at all 29 saved frames through 14 s, while only one RL footprint corner remains inside at 4 s.The continuation compares different arrival times and speeds near the same junction; no standard 14-second PDMS is assigned.Source: §D.3 · p. 24
• The appendix reports ReCogDrive foldback counts using 0.05 m and 0.5 m displacement filters, with the filters treated as descriptive amplitude thresholds rather than safety boundaries.Table 14 includes configurations and both filters without outcome-based filtering or new model inference.Source: §D.3 · p. 25, Table 14 · p. 25

D.4 NUPLAN SCORE REPRODUCTION

• Plan-R1’s reinforcement scores reproduce published CLS values within 0.1 points in both regimes, and stored sub-metrics reproduce the official aggregate to 10−15.Table 15 reports nuPlan val14 CLS in score points multiplied by 100.Source: §D.4 · p. 25, Table 15 · p. 25

D.5 NUPLAN PAIRED SCORES AND MOTION

• Plan-R1’s score, progress, and plan-consistency improvements accompany increased absolute jerk peaks in both traffic settings; the binary comfort term shows no supported mean change in NR and a slight decrease in reactive traffic.The continuous jerk measurement identifies a motion change not described by the comfort pass rate alone.Source: §D.5 · p. 25, Table 16 · p. 26, Table 17 · p. 26

E.1 FIXED-OUTPUT EPDMS RESCORING

• On smaller fixed-output rescoring sets, NoRD improves revised scores and displayed components; ReCogDrive retains gains, while Qwen-Drive’s lane-keeping and history-comfort declines are not significant.The rescoring evaluates fixed outputs and is separate from optimizing the revised objective.Source: §E.1 · p. 26, Table 18 · p. 27

E.2 REINFORCEMENT PAIRS UNDER THE TWO-STAGE PROTOCOL

• The protocol re-infers plans for three planner pairs on 450 original and 5,462 synthetic scenes, then scores and aggregates them with the unmodified official code.Synthetic scenes start from counterfactual ego states and rendered observations; paired intervals resample the 450 original scenes or their mapping groups.Source: §E.2 · p. 28
• Under the two-stage protocol, ReCogDrive, Qwen-Drive, and NoRD combined changes are −0.008 [−0.038, +0.019], −0.008 [−0.035, +0.018], and +0.003 [−0.021, +0.027], respectively; all intervals include zero.NoRD gains +0.143 and +0.049 in stages one and two, and +0.079 [+0.048, +0.112] in combined EPDMS.Source: §E.2 · p. 28
• The three free-form pairs gain 0.105, 0.079, and 0.063 PDMS open-loop and 0.077, 0.060, and 0.062 EPDMS under fixed-output rescoring, but their open-loop advantage does not survive re-inference.Their stage-one changes are +0.000, −0.015, and −0.003; synthetic-stage changes are −0.012, −0.016, and −0.010, with intervals containing zero.Source: §E.2 · p. 28
• On the synthetic stage, two-frame extended comfort falls by 0.378, 0.527, and 0.296 for the ReCogDrive and Qwen-Drive pairs, while NoRD’s rises by 0.038.The declines complement closed-loop prediction-revision measurements; the official combined score is a mean of stage products rather than a product of stage means.Source: §E.2 · p. 28