Source-linked AI summary
Evaluation Is All You Need for Multi-Modal Autonomous Driving
Zeyu He, Shiqi Liu, Ke Chen, Yun Yan, Jinzi Wu, Dianqiao Lei, Sirui Wang, ShuRui Peng, Tao Chen, Zhuo Huang, Yu Wu, Yadong Shao, Zhichao Li, Ke Sun, Yang Guan, Keqiang Li, Shengbo Eben Li
TL;DR
Multi-modal planners can generate strong candidate sets, but unreliable evaluation leaves some oracle potential unrealized. iDriveVLA combines safety-aware scoring, VLM-guided semantic modulation, and oracle-aligned training; it reports state-of-the-art performance on NAVSIM v1 and v2, including 94.95 PDMS on v1, above the human-expert reference.
Problem
In ambiguous and long-tail driving scenarios, planners may generate strong candidate trajectories but struggle to reliably select the best, leaving planning potential unrealized.
Source: §1 · pp. 1–2Method
iDriveVLA improves candidate quality and selection with a Safety-aware Scorer, a VLM-guided Modulator, and progressive oracle-aligned training.
Source: §1 · p. 2Results
Takeaways & Limitations
The findings support treating trajectory evaluation as an explicit research direction alongside candidate generation, while substantial selected-to-oracle potential remains.
Source: §5 · pp. 9–10A substantial gap between selected and oracle performance remains, so existing candidate sets are not fully exploited.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Multi-modal planning is promising for autonomous driving by representing multiple plausible behaviors in ambiguous and long-tail scenarios. Existing methods mainly focus on improving trajectory multi-modality, enhancing trajectory representations, or reshaping the candidate distribution. Nevertheless, we identify a pronounced generation-evaluation asymmetry in multi-modal planning: despite strong oracle performance, existing planners often fail to reliably select the best available candidate, leaving substantial planning potential unrealized. To address this challenge, we propose iDriveVLA, a multi-modal planning framework that improves the candidate trajectory space while enabling more reliable and context-aware trajectory evaluation. Specifically, iDriveVLA introduces a unified trajectory evaluator comprising a Safety-aware Scorer for quality and risk estimation, together with a VLM-guided Modulator for scene-adaptive criterion weighting. We further develop an oracle-aligned progressive training strategy consisting of candidate imitation pretraining, candidate space refinement, and semantic ranking alignment. On the public NAVSIM v1 leaderboard, iDriveVLA achieves a new state-of-the-art performance of 94.95 PDMS, surpassing the human-expert reference.
1 INTRODUCTION
Multi-modal planning addresses ambiguous driving scenes by generating and evaluating several plausible trajectories, but evaluation can lag candidate generation. iDriveVLA combines safety-aware and scene-adaptive evaluation with progressive training to improve candidate quality and selection.
1 INTRODUCTION
- Ambiguous and long-tail scenarios motivate multiple plausible trajectories, which can differ in safety, efficiency, comfort, and interaction with other agents.Examples include partially occupied lanes, unusual obstacles, temporary road structures, and ambiguous traffic interactions.Source: §1 · p. 1
- Multi-modal planning faces a generation–evaluation asymmetry: strong candidate sets contain unrealized oracle potential when evaluators cannot reliably distinguish trajectories.Increasing diversity adds competing behaviors for the evaluator, while predefined planning signals may lack safety awareness and scene-adaptive semantic reasoning.Source: §1 · p. 2, Figure 1 · p. 2
- iDriveVLA combines a Safety-aware Scorer for filtering low-quality candidates with a VLM-guided Modulator for scene-adaptive semantic evaluation.The framework also uses candidate imitation pretraining, candidate space refinement, and semantic ranking alignment to improve candidate generation and selection reliability.Source: §1 · p. 2
- 94.95 PDMS earns iDriveVLA 1st place on NAVSIM v1, surpassing the human-expert reference.Source: §1 · p. 2
2 RELATED WORK
Related work brings semantic reasoning, multi-modal candidate generation, and trajectory scoring to autonomous driving. The paper frames evaluator accuracy and adaptation to scene semantics as remaining challenges.
Vision-Language-Action Planning
- VLM-based planners use semantic priors and reasoning through trajectory outputs, but often trail strong end-to-end models in trajectory-level planning performance.Source: §2 · p. 2
Multi-modal Trajectory Generation
- Multi-modal generation spans vocabulary-based, generative, and proposal-based methods, improving candidate diversity and quality while downstream evaluation constrains their contribution to driving performance.Examples include VADv2, GenAD, DiffusionDrive, iPad (Guo et al., 2025), and DrivoR.Source: §2 · p. 3
Scorer-Based Autonomous Driving
- Trajectory scorers rank diverse hypotheses using learned functions, interpretable metrics, or demonstration-derived rules, but their criteria largely rely on geometric or predefined signals.These scorers have limited ability to adapt candidate preferences to high-level scene semantics.Source: §2 · p. 3
3 METHOD
The method diagnoses a gap between the quality of trajectories available in a candidate set and the evaluator’s ability to select among them. iDriveVLA combines safety-aware, scene-adaptive evaluation with three progressive training stages to improve candidate quality and selection reliability.
3.1 DIAGNOSING THE TRAJECTORY EVALUATION BOTTLENECK
- Oracle candidates cluster near high PDMS, but selected candidates have a heavier low-score tail, indicating that evaluation can leave available planning potential unrealized.The analysis compares the selected trajectory with the highest-PDMS candidate from the same set.Source: §3.1 · p. 3, Figure 2 · p. 3
3.2 SAFETY-AWARE SEMANTIC TRAJECTORY EVALUATION
- The evaluator predicts criterion-wise planning scores and safety risks, filters unsafe candidates, then aggregates scores to select the final trajectory.Its six criteria cover collision avoidance, drivable-area compliance, time to collision, ego progress, driving-direction compliance, and comfort.Source: §3.2 · pp. 4–5
- A VLM-guided Modulator uses scene semantics to adapt criterion weights, changing candidate evaluation without modifying the trajectories themselves.A frozen VLM encodes the front-view image and a predefined prompt; the Modulator conditions on that representation and candidate features.Source: §3.2 · p. 5
3.3 ORACLE-ALIGNED PROGRESSIVE TRAINING
- Candidate imitation pretraining jointly trains the generator and Scorer using a best-of-K imitation objective and candidate-level planning-metric and safety supervision.The objective combines trajectory imitation with structured evaluator supervision while preserving diverse motion hypotheses.Source: §3.3 · p. 5
- Candidate space refinement weights training toward expert trajectories with higher downstream PDMS quality while retaining structured evaluator supervision.This reward-weighted supervised refinement targets generated distributions that may still contain trajectories suboptimal for downstream planning.Source: §3.3 · pp. 5–6
- Semantic ranking alignment optimizes the Modulator with downstream planning rewards while freezing the generator, Scorer, and VLM backbone.STAPO stabilizes optimization by masking abnormal gradients from outlier policy updates, while the Modulator learns scene-dependent candidate preferences.Source: §3.3 · p. 6
4 EXPERIMENTS
On NAVSIM v1 and v2, iDriveVLA achieves strong benchmark results, while ablations and qualitative examples examine its components and trajectory choices. The experiments use NAVSIM data and compare the planner with DrivoR and other methods.
4.1 SETTINGS
- Training follows a three-stage procedure and uses NAVSIM’s approximately 103K training and 12K test samples, augmented with 134K simulated samples; evaluation reports PDMS on v1 and EPDMS on v2.The reported setup uses DrivoR for trajectory generation and a frozen InternVL3-2B for semantic modulation.Source: §4.1 · pp. 6–7
4.2 MAIN RESULTS
- 94.9 PDMS on NAVSIM v1 ranks iDriveVLA first among compared methods and exceeds the 94.8 human-driver reference.The paper reports the human-driver reference score as 94.8 PDMS.Source: Table 1 · p. 7, §4.2 · p. 8
- 87.4 EPDMS on NAVSIM v2 gives iDriveVLA state-of-the-art performance, with higher ego progress than LaST-VLA and CLOVER while maintaining competitive safety and compliance.Results use the original NAVSIM v2 evaluation code without the human-behavior filtering fix.Source: Table 2 · p. 7, §4.2 · p. 8
4.3 ABLATION STUDIES
- The Safety-aware Scorer raises PDMS from DrivoR’s 93.7 to 93.9, refinement raises it to 94.2, and adding the Modulator achieves the best overall performance.The scorer suppresses risky candidates without sacrificing driving efficiency; refinement improves candidate-set quality before selection.Source: Table 3 · p. 8, §4.3 · p. 8
4.4 QUALITATIVE RESULTS
- In Figure 4, iDriveVLA avoids DrivoR’s collisions and drivable-area departures while making smoother, more efficient progress in turning and lane-changing examples.The examples also show DrivoR behaving overly conservatively during some maneuvers.Source: §4.4 · p. 8, Figure 4 · p. 9
- With the candidate set fixed, the Modulator changes selection preferences; examples show closer lane following, greater obstacle clearance, and less overcaution without additional collisions or drivable-area violations.Appendix D.2 comparisons indicate iDriveVLA can identify optimal or near-optimal candidates.Source: §4.4 · pp. 8–9, Figure 5 · p. 9
5 CONCLUSION
The paper frames multi-modal planning around trajectory evaluation as well as candidate generation, and reports gains on NAVSIM v1 and v2. It also notes that selected performance remains substantially below oracle performance.
5 CONCLUSION
- Experiments on NAVSIM v1 and v2 show gains in overall planning quality, safety, compliance, and driving progress, with state-of-the-art performance on both benchmarks.The framework combines safety-aware scoring, semantic modulation, and oracle-aligned training to improve candidate quality and trajectory selection.Source: §5 · p. 9
- A substantial gap between selected and oracle performance remains, so the potential of existing candidate sets is not fully exploited.The authors identify more reliable, adaptive, and semantically informed trajectory evaluation as a direction for future work.Source: §5 · p. 10
- Generative AI tools supported manuscript clarity, routine code debugging and revisions, and illustrative figures, but not scientific content or substantive research decisions.The authors report inspecting and revising AI-assisted outputs and independently developing and verifying the scientific work.Source: AI USE STATEMENT · p. 10
A THEORETICAL ANALYSIS OF CANDIDATE COMPETITION
The analysis shows that imperfect evaluation can make oracle recovery worse as candidate competition increases; Stage 2 instead reweights demonstrations by downstream planning reward while retaining supervised targets.
A THEORETICAL ANALYSIS OF CANDIDATE COMPETITION
- The analysis models one oracle trajectory against K − 1 lower-scoring candidates, with evaluation errors drawn independently from a continuous distribution.It defines the evaluation gap as the oracle score minus the selected candidate's score.Source: A THEORETICAL ANALYSIS OF CANDIDATE… · p. 14
- When Pr(0 < F(ϵ⋆ + γ) < 1) > 0, adding competing trajectories strictly lowers oracle recovery and raises the expected evaluation gap.The result follows for a stylized candidate set with one oracle trajectory and K − 1 alternatives scoring r⋆ − γ.Source: A THEORETICAL ANALYSIS OF CANDIDATE… · pp. 14–15
- More candidates can improve candidate coverage and oracle potential, but imperfect evaluation means stronger generation alone does not guarantee better final selection.Source: A THEORETICAL ANALYSIS OF CANDIDATE… · p. 15
B INTERPRETATION OF CANDIDATE SPACE REFINEMENT
- Stage 2 exponentially tilts the demonstration distribution toward higher downstream planning rewards, reweighting scenes rather than changing their expert trajectory targets.With one expert trajectory per scene, refinement remains fully supervised and preferentially optimizes the shared generator toward higher-quality trajectory patterns.Source: B INTERPRETATION OF CANDIDATE SPACE… · p. 15
C.1 EVALUATION METRICS
Closed-loop planning is evaluated with benchmark-specific scores: PDMS on NAVSIM v1 and the extended EPDMS on NAVSIM v2.
C.1 EVALUATION METRICS
- PDMS is reported for NAVSIM v1, while EPDMS is reported for NAVSIM v2.Source: §C.1 · p. 15
- PDMS combines No At-Fault Collision, Drivable Area Compliance, Time-to-Collision, Comfort, and Ego Progress.Source: §C.1 · p. 16
- EPDMS extends PDMS with driving-direction and traffic-light compliance, lane keeping, and history and extended comfort components.Its listed components also include No At-Fault Collision, Drivable Area Compliance, Ego Progress, and Time-to-Collision.Source: §C.1 · p. 16
C.2 IMPLEMENTATION DETAILS
The implementation combines a four-view visual encoder and trajectory scorer with a safety head and a frozen vision-language model that conditions a DiT modulator.
C.2 IMPLEMENTATION DETAILS
- The generator takes four camera views and ego state, using DINOv2 ViT-S adapted with LoRA and 16 visual registers per view.Source: §C.2 · p. 16
- The trajectory decoder and scorer each have four layers, while a two-layer MLP predicts collision and drivable-area-departure risks.Source: §C.2 · p. 16
- Semantic modulation uses frozen InternVL3-2B features to condition an eight-block DiT Modulator.Source: §C.2 · p. 16
D.1 COMPUTATIONAL TIME AND MEMORY ANALYSIS
The complete iDriveVLA pipeline runs in 297.20 ms per sample on an A800 GPU with a single-element batch, 186.10 ms longer than DrivoR.
D.1 COMPUTATIONAL TIME AND MEMORY ANALYSIS
- 297.20 ms per sample is the complete pipeline latency on an A800 GPU with a single-element batch.This is 186.10 ms more than DrivoR; the authors state that the sub-second latency meets practical driving applications' real-time inference requirements.Source: §D.1 · p. 16
D.2 QUALITATIVE ANALYSIS OF TRAJECTORY SELECTION
iDriveVLA’s selected trajectories closely follow or overlap with oracle trajectories, while DrivoR can select unsafe candidates despite a safe oracle being available.
D.2 QUALITATIVE ANALYSIS OF TRAJECTORY SELECTION
- DrivoR selects an unsafe candidate with zero PDMS despite a safe oracle trajectory being available.The trajectories differ subtly in curvature or lateral offset, yet those differences determine collision risk and road compliance.Source: Figure 6 · p. 17, §D.2 · p. 17
- The iDriveVLA selection examples show trajectories that closely follow or overlap with the oracle trajectories.Figure 7’s comparison illustrates identification of high-quality candidates from a diverse trajectory set.Source: Figure 7 · p. 17, §D.2 · p. 18
- iDriveVLA closely follows the oracle in one example and identifies the oracle candidate in another.The oracle is the candidate with the highest ground-truth PDMS in the generated set.Source: §D.2 · pp. 17–18