Source-linked AI summary
From Proxy Learning to Driving Decisions: A Transfer-Based Framework for Evaluating Future-Aware Autonomous Driving Planners
Yikai Wu
TL;DR
Future-aware planners can improve predictive proxies without reliably improving selected driving decisions, creating a need to validate transfer across the full planning chain. PDT decomposes where transfer is lost and constrains positive claims through paired, scale-aware, safety-sensitive, and robust validation. In the NAVSIM-v1 case study, proxy gains coexist with unresolved or negative decision-level evidence, and none of 432 screened configurations passes the robustness gate.
Problem
Proxy improvements or restricted-subset gains are often interpreted as planning improvements without verifying proposal ordering, selected trajectories, full-scale utility, and critical components.
Method
PDT combines Decision-Transfer Decomposition with Reliability-Constrained Validation to audit transfer from representation and scoring through selection, support, and deployment.
Results
None of 432 two-seed configurations passes the predefined robustness rule, while the case study documents proxy, subset, aggregate, and selection reversals or unresolved effects.
Takeaways & Limitations
PDT supports reporting pass, fail, and indeterminate outcomes at specific transfer stages instead of treating proxy or aggregate gains as decision-level evidence.
Takeaways & Limitations
The evaluation covers one proposal-based future-aware planning family and NAVSIM-v1, so it does not establish cross-architecture or cross-benchmark invariance.
Abstract
from arXiv · showhide
Future-aware representations and world models are increasingly used in proposal-based autonomous-driving planners to improve trajectory selection. However, improvements in proxy objectives or restricted subsets are often interpreted as planning gains without verifying proposal ordering, selected trajectories, full-scale utility, and critical driving components. We propose the Proxy-to-Decision Transfer (PDT) Framework, an analysis framework that evaluates when learned future information supports a reliable driving-performance improvement claim. Its Decision-Transfer Decomposition Module localizes value loss through score margins, switch-conditioned utility, and support-versus-selection regret. Its Reliability-Constrained Validation Module requires exact pairing, a minimum meaningful effect, scale-expanded confirmation, safety non-compensation, sequential comparability, and family-level robustness. On a representative future-aware planner evaluated with NAVSIM-v1, component BCE decreases from 0.705 to 0.530 while held selected PDM decreases from 0.963 to 0.961. A separate candidate improves a 512-record prefix by 0.00909, with a scene-bootstrap 95% interval of [0.000744, 0.0177], but its 2048-record and complete-support intervals include zero. A proposal-level replay further confirms the switch-utility decomposition, yet none of 432 screened configurations passes the two-half, two-seed robustness gate. PDT therefore identifies where decision transfer fails or remains indeterminate across proxy, subset, aggregate, and selection evidence.
1 Introduction
Future-aware planning introduces a gap between improved internal proxies and reliable driving decisions. PDT makes that transfer auditable by locating failure points and validating claims across decision utility, scale, safety, and robustness.
- Motivation: Future-aware planners combine learned representations, proposal scoring, discrete top-1 selection, and trajectory utility within an integrated planning pipeline.These stages connect future prediction to the final selected trajectory.
- Motivation: Lower proxy loss, changed scores, or positive subset performance can fail to improve the selected trajectory or full-scale driving utility.Scores may not cross the top-1 boundary, and local gains may disappear under paired evaluation.
- Framework: PDT audits transfer across representation, scoring, selection, support, and deployment rather than treating these evidence levels as interchangeable.The framework turns a future-aware planning claim into an auditable chain of tests.
- Framework: The Decision-Transfer Decomposition separates decision-value loss using score margins, switch-conditioned utility, and support-versus-selection regret.Its diagnostics identify whether value is lost before scoring, at selection, or through candidate support.
2 Related Work
Related work advances integrated planners, future modeling, and decision-oriented evaluation, but final scores alone leave transfer mechanisms and evidence reliability underdetermined. PDT complements these strands by auditing how evidence moves from predictive proxies to downstream planning decisions.
- Planning-Oriented End-to-End Driving: Integrated and proposal-based planners combine perception, prediction, mapping, and trajectory generation, while generative methods enlarge feasible action support.Examples include UniAD, VAD, SparseDrive, and DiffusionDrive.
- Transfer Validity: The literature compares complete planners by benchmark score, leaving unclear whether gains arise from representation, ordering, candidate support, or evaluation artifacts.PDT tests these transfer locations separately instead of attributing the final score to a named module by default.
- Future-Aware Modeling: Future-aware methods predict spatial states, latent scene evolution, or controllable video futures to support planning and trajectory evaluation.These approaches include freespace forecasting, DriveWorld, OccWorld, Drive-WM, and related world-model planners.
- Future-Aware Modeling: Future-prediction accuracy or visual fidelity is insufficient unless generated futures are converted into scores that order feasible trajectories by driving utility.The decision criterion is the quality of trajectory ordering, not prediction quality alone.
- Planning Evaluation and Decision-Oriented Validation: Decision-oriented benchmarks include open-loop, closed-loop, and non-reactive protocols, but unmatched supports, metric conventions, safety compensation, and repeated search can still distort comparisons.PDT separates benchmark quality from the reliability of evidence produced with that benchmark.
3 Proposed Method
PDT evaluates future-aware planning through an auditable chain from representation and proposal scoring to selection, support, and deployment reliability. Its diagnostics separate decision-boundary activation, switch-conditioned utility, regret sources, and validation comparability.
- Framework scope: PDT evaluates improvement claims for proposal-based planners without prescribing a new planner architecture.
- Failure locations: The framework tests five transfer locations: representation, scoring, selection, support, and deployment.Each stage is marked pass, fail, or indeterminate; missing or non-comparable evidence yields indeterminate rather than a negative result.
- Score margins and decision-boundary activation: A changed proxy or score matters for decisions only when proposal-wise perturbations cross the baseline decision margin and activate a competing trajectory.PDT therefore requires proposal arrays, selected indices, margins, and exact candidate identities to reconstruct switching behavior.
- Switch-conditioned decision utility: ∆J = ρµ exactly decomposes mean decision-utility change into trajectory switch rate and conditional utility of a switch under common support and identical evaluation.A positive decision-transfer claim requires both ρ > 0 and µ > 0; no switches or non-positive conditional utility reject it.
- Support and selection regret: Rtotal = Rsupport + Rselection separates regret caused by unavailable trajectories from regret caused by ranking available candidates.A score-only intervention can reduce Rselection but cannot reduce Rsupport when the candidate set is fixed; attributing residual regret to support requires an external comparison space.
- Reliability-Constrained Validation: A reliable positive verdict requires exact pairing, scale-expanded confirmation, meaningful-effect and safety non-compensation checks, sequential comparability, and robustness gates.The protocol uses paired differences on matched supports and does not promote an unthresholded sign-only difference to a reliable improvement.
4 Experimental Results
The experiments instantiate PDT on a frozen future-aware planner and NAVSIM-v1 using locked, sequential protocols. The study provides a detailed case analysis rather than cross-architecture validation.
- Experimental setup: The evaluation uses a representative proposal-based future-aware planner with NAVSIM-v1 and real driving logs in non-reactive simulation.The model combines a joint-embedding predictive visual representation with multimodal trajectory proposals.
- Experimental setup: The protocol proceeds through locked development, prefix, and full-evaluation stages, with thresholds and safety gates recorded before relevant outcomes were opened.The complete PDT taxonomy was synthesized retrospectively rather than fully preregistered.
- Evaluation support: The full NAVSIM-v1 comparison contains 12,146 valid output records with zero failed predictions and exact baseline/candidate token support.Reported PDMS values come from the locked official evaluator without denominator reinterpretation.
- Evidence organization: Table 2 organizes the analysis around evidence units whose descriptive names indicate each model or experiment's methodological role.Internal run identifiers are omitted from the table.
4.2 Evaluation Metrics
PDT separates internal learning signals from proposal utility and official aggregate evaluation. Its validation uses paired, scale-aware, component-sensitive evidence while recognizing that bootstrap intervals describe recorded support rather than route- or log-level inference.
- Evidence levels: Future temporal loss and component BCE measure internal target learning, but lower proxy values are not treated as driving utility.When proposal arrays exist, PDT adds ranking, margin, selection, switch-count, ρ, and µ diagnostics.
- Proposal utility: Selected PDM is the utility of the chosen proposal, combining collision and drivable-area compliance with weighted progress, TTC, and comfort.The displayed weighting assigns 5 to progress, 5 to TTC, and 2 to comfort.
- Aggregate utility: PDMS is the official aggregation of scenario PDM scores, reported alongside available collision, drivable-area, TTC, driving-direction, and comfort components.This preserves both aggregate and critical-component views of performance.
- Reliability constraints: The protocols require exact pairing, fixed splits or quartiles, unseen suffixes, zero-failure checks, minimum expansion effects, and nonnegative critical-component changes.Paired bootstrap intervals use 10,000 deterministic resamples and summarize uncertainty on recorded support.
- Inference boundary: Bootstrap intervals summarize uncertainty on recorded support and do not replace route- or log-level inference.Main-text metrics use approximately three significant digits, with smaller deltas reported separately.
4.3 Implementation Details
Implementation freezes the evaluator, data joins, target generation, and checkpoint-selection procedures to support exact paired comparisons. The calibration stress test then evaluates 432 two-seed configurations under a family-level gate.
- Frozen evaluation: The backbone and evaluator were frozen, while candidate and baseline repositories were byte-compared for scoring implementation and configuration.Checkpoint selection used outcome-independent development support before official evaluation.
- Data integrity: Candidate and baseline CSVs were joined by token and required identical token sets, finite values, and zero failures.These checks enforce exact paired support for the comparison.
- Target and split control: Direct utility targets were regenerated in exact scoring batches, and mixed target-cache batching was rejected when it changed scores.Fixed validation halves were outcome-independent, while prefix expansion used locked checkpoints and immutable comparison CSVs.
- Calibration stress test: The calibration stress test trained 216 arms across calibrator forms, perturbation caps, regularization strengths, weight decays, learning rates, and two seeds.Four evaluation steps per arm produced 864 seed-level outcomes.
- Calibration stress test: The 864 seed-level outcomes were grouped into 432 two-seed configurations for the family-level gate.A CPU replay used the same frozen training and validation tokens.
4.4 Main Experiments
The experiments trace future-aware improvements from proxy learning through proposal selection and deployment-scale utility. Across these levels, local or proxy gains fail to establish reliable, robust decision transfer.
- Test 1: Proxy Improvement and Selected Utility: Component BCE decreased from 0.705 to 0.530, while held selected PDM decreased from 0.963 to 0.961.The result rejects sufficiency of better component BCE for better selected PDM in the tested scorer, without showing component supervision is generally harmful.
- Test 2: Decision-Boundary Activation: Future temporal loss decreased from 0.0527 to 0.0329, but validation selections and selected PDM remained unchanged.The four selections stayed [28] [21] [20] and selected PDM remained 0.978; oracle headroom coexisted with zero observed switching.
- Test 2: Decision-Boundary Activation: Four of 64 validation decisions switched in the repeatable linear configuration, yielding ∆J = 1.06 × 10−4; three switches had zero utility difference.The switch-utility decomposition was numerically confirmed, but the interval had a zero lower endpoint and the second fixed half had zero gain.
- Test 3: Deployment-Scale Stability of Local Gains: +0.00909 occurred on prefix512, but the prefix2048 and complete-support intervals included zero.On complete support, the candidate was lower than the frozen development reference by 6.64 × 10−4, with interval [−0.00248, 0.00121].
- Test 4: Aggregate Utility and Critical-Component Regressions: +4.44 × 10−4 in the exact paired full comparison had an interval crossing zero and coincided with a TTC regression of −0.00782.The candidate therefore failed the study’s zero-tolerance safety non-compensation rule; aggregate gains in progress and drivable-area compliance coexisted with critical-component declines.
- Family-Level Robustness: Zero of 432 two-seed configurations passed the robust two-half/two-seed gate.Only four configurations had positive paired averages, and none was strictly positive on both fixed validation halves across both seeds.
4.5 Evaluation-Rule Ablation and Conclusion Reversals
The evaluation-rule ablation shows that conclusion status depends on the complete PDT transfer chain, not numerical positivity alone. Removing individual rules can turn a positive shortcut conclusion into a rejected or bounded claim.
- Ablation design: The ablation removes one PDT claim rule at a time while holding the underlying artifacts fixed.This is an evaluation-rule ablation rather than a new model-training experiment.
- Conclusion reversals: Common evaluation shortcuts produce a positive conclusion that the restored transfer chain rejects or bounds.The ablation tests whether evidence is used to expand a run, claim a safety-consistent improvement, or continue score-only calibration.
- Interpretation: “Numerically positive” denotes a sign-only verdict and does not imply a practically meaningful effect.
4.6 Visualization of Transfer Attrition
Figure 8 tracks how locally positive observations are rejected at downstream transfer stages. It distinguishes attrition at selected utility, full-scale utility, critical components, and robustness replication.
- Observed attrition: Three paths begin with positive local observations but are rejected at selected utility, full-scale utility, or the critical component vector.The figure summarizes verdict transitions rather than plotting unavailable proposal-level arrays.
- Observed attrition: The calibration path begins with isolated nominal improvements but fails replication across fixed halves and seeds.
- Interpretation: Positive local evidence is assigned only to the transfer stage it supports and cannot license a stronger downstream claim.
5 Discussion
The discussion presents PDT as a staged evaluation method that separates learning, planning-improvement, and deployment claims. Its case study shows that transfer can remain unresolved because of selection, support, scale, reliability, and logging limitations.
- Evaluation method: PDT evaluates future-aware planning signals across representation, scoring, selection, support, and deployment rather than collapsing them into one aggregate score.Its decomposition uses proposal-wise activation surplus, while validation constrains positive claims through exact pairing, scale expansion, component checks, and robustness conditions.
- Selection diagnostics: The frozen-tensor replay changes four of 64 decisions with ρ = 0.0625 and µ = 0.00170, but three switches are utility-neutral.This separates decision coverage from the utility of switched trajectories.
- Implementation: PDT requires stable candidate identifiers, proposal-wise scores, selected indices, top-1/top-2 margins, utilities where feasible, outcomes, evaluator metadata, and support keys.Persisting these fields makes the transfer chain auditable without redesigning the planner.
- Limitations: The evidence is limited to one proposal-based planner family and NAVSIM-v1, with incomplete proposal arrays and no demonstration of real-world safety or closed-loop causal effects.The complete-run data cannot support log-clustered intervals, complete-run switch-conditioned utility, or external support-regret decomposition.
- Future work: Future work should prospectively evaluate diverse planners with preregistered thresholds, frozen supports, common logging, and controlled support-expansion experiments.Holding the scorer fixed while adding feasible candidates can distinguish selection from candidate-support limitations.
6 Conclusions
The paper concludes that PDT evaluates whether future-aware evidence transfers through proposal decisions into reliable planning claims. The NAVSIM-v1 case study shows failures and unresolved stages across utility, support, components, and robustness.
- Framework: PDT combines Decision-Transfer Decomposition with Reliability-Constrained Validation to analyze transfer across representation, scoring, selection, and candidate support.The framework limits positive claims through exact pairing, scale expansion, critical-component non-compensation, and sequential comparability.
- Case-study conclusion: The NAVSIM-v1 case study finds lower selected utility despite proxy improvement, unresolved larger-support intervals, declining critical components, and zero robust configurations among 432 screened settings.A proposal replay exactly closes ∆J = ρµ while revealing low decision coverage and mostly utility-neutral switches.
- Implication: PDT supports reporting pass, fail, and indeterminate outcomes at specific transfer stages for model selection, experiment design, and scoped conclusions.The findings do not establish that future-aware learning is ineffective; they establish the need for decision-level evaluation under explicit support, uncertainty, and reliability conditions.
Data and Code Availability
The paper provides minimal retained evidence and deterministic analysis materials through arXiv ancillary files, while directing readers to original repositories for excluded benchmark assets and model weights.
- ArXiv ancillary files contain paired scenario-level metrics, frozen proposal tensors, configuration locks, diagnostic outputs, derived tables, scripts, and SHA-256 hashes.
- NAVSIM is publicly available from its cited source.
- Large public benchmark assets and upstream model weights are excluded and should be obtained from their original repositories.