PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Better Visual Representations Always Lead to Better End-to-End Autonomous Driving?

Zihao Zhang, Haochen Tian, Tianyu Li, Changhui Jing, Jingliang He, Naisheng Ye, Ziyuan Pu, Zhenjie Yang

arXiv:2610.09695v1cs.ROcs.CV

TL;DR

Systematic comparisons across planner–VFM pairings remain limited, leaving unclear when VFM representations improve driving. ViRA aligns planner features with a frozen VFM without changing inference; VFM-guided representations improve driving across planners, and ViRA-Diffusion reaches 92.3 EPDMS on NAVSIM v2 navtest, 1.9 points above a recent state-of-the-art method.

  • Problem

    Systematic comparisons across planner–VFM pairings remain limited, and it is unclear whether VFMs’ visual-task advantages translate into improved driving performance.

    Source: §1 · p. 1

  • Method

    ViRA aligns planner representations with a frozen VFM target while keeping planner architecture and inference cost unchanged.

    Source: §1 · p. 1, §2.1 · p. 3

  • Results

    VFM-guided representations improve driving across planners, including zero-shot closed-loop evaluation; ViRA-Diffusion scores 92.3 EPDMS on NAVSIM v2 navtest, 1.9 points above the recent state-of-the-art method.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The findings highlight the importance of considering VFM target selection and planner supervision jointly when integrating VFMs into end-to-end driving.

    Source: §1 · p. 3

    VFM pre-training biases may transfer to planners and degrade performance in underrepresented regions or rare safety-critical scenarios without careful validation.

    Source: E LIMITATIONS AND BROADER IMPACT · p. 21

Abstract

from arXiv · show

Visual foundation models (VFMs) are increasingly integrated into end-to-end autonomous driving for their powerful representations, yet it remains unclear when these representations improve driving performance. To investigate this question, we introduce ViRA, a planner-agnostic visual representation alignment framework that keeps the planner architecture and inference cost unchanged. Our study reveals three findings: (1) VFM-guided visual representations consistently improve driving performance across diverse end-to-end planners, with gains extending to zero-shot closed-loop evaluation. (2) The choice of VFM target matters for planning performance, and alignment to a different VFM can further benefit planners with pre-trained VFM encoders. (3) Auxiliary perception supervision reduces sensitivity to VFM target selection, narrowing the EPDMS spread across five targets from 2.7 to 0.5 points and potentially compensating for less effective VFM targets. Guided by these findings, we develop ViRA-Diffusion, a diffusion-based planner trained without auxiliary perception supervision, which achieves 92.3 EPDMS on NAVSIM v2 navtest, outperforming recent methods in our comparison by at least 1.9 points. The results motivate jointly considering target selection and planner supervision when integrating VFMs into end-to-end autonomous driving. The results and demo are available at https://github.com/OpenDriveLab/ViRA.

1 INTRODUCTION

Systematic comparisons across planner–VFM pairings remain limited, leaving unclear whether VFMs’ visual-task strengths translate into better driving. ViRA studies these pairings and finds consistent driving gains, with results shaped by both VFM target and planner supervision.

1 INTRODUCTION

  • ViRA compares planner–VFM pairings by aligning planner representations to a frozen VFM, preserving planner architecture and inference cost.The paper motivates this study because systematic comparisons across pairings remain limited.Source: §1 · pp. 1, 3
  • 7.9 and 8.6 EPDMS are the average gains over respective baselines on NAVSIM v2 navtest and navhard; zero-shot HUGSIM HD-Score improves by 4.0 points.Randomly initialized targets fail to reproduce the perception and planning gains, supporting the importance of pre-trained VFM representations.Source: §1 · p. 2
  • For Rap*, VFM targets yield gains from 6.6 to 12.1 EPDMS; VGGT alignment raises the DINOv3-backbone score from 83.8 to 86.3 (+2.5).Without auxiliary perception supervision, TransFuser also performs best with DINOv3; with supervision, TransFuser and DiffusionDrive perform similarly across targets.Source: §1 · p. 2
  • Auxiliary supervision narrows the five-target gain spread from 2.7 to 0.5 EPDMS points, chiefly compensating for less effective targets.The lowest score drops from 88.8 to 86.4 without supervision, while the highest stays nearly unchanged.Source: §1 · pp. 2–3
  • 92.3 EPDMS on NAVSIM v2 navtest lets ViRA-Diffusion outperform the recent state-of-the-art method by 1.9 points without auxiliary perception supervision.The planner uses DINOv3 alignment.Source: §1 · p. 3

2 PRELIMINARIES

ViRA provides a shared way to study visual representations across planner designs: it aligns planner features with frozen VFM representations during training while leaving inference unchanged. The paper evaluates NAVSIM v2 benchmarks and zero-shot HUGSIM closed-loop performance.

2.1 VFM-GUIDED END-TO-END PLANNER

  • ViRA uses spatial-attention matching and bidirectional contrastive alignment of global embeddings to transfer VFM spatial dependencies and scene-level semantics to the planner encoder.The frozen VFM processes the same observation for representation-level supervision; its features are not passed to the policy head, and no additional planner-training annotations are required.Source: §2.1 · p. 3

2.2 BENCHMARK AND EVALUATION

  • NAVSIM v2 navtest and navhard use EPDMS; HUGSIM evaluates NAVSIM-trained planners zero-shot in a reactive, photorealistic closed-loop simulator using HD-Score.HD-Score weights route completion by per-step safety and comfort; TransFuser perception is assessed with mIoU over seven BEV semantic classes.Source: §2.2 · p. 4
  • The t-SNE visualizations compare planner representations with and without ViRA against reference VFM representations for TransFuser and Rap*.The visualization suggests ViRA shifts planner representations toward the reference VFM in t-SNE space.Source: Figure 2 · p. 4

3 VFM-GUIDED VISUAL REPRESENTATIONS DO IMPROVE DRIVING

ViRA improves driving performance across planners and benchmarks, including zero-shot closed-loop evaluation, while improving perception and requiring a pre-trained target. Its alignment also preserves the deployed planner architecture and inference cost.

3.1 DRIVING PERFORMANCE ACROSS DIFFERENT BENCHMARKS

  • With ViRA, TransFuser, DiffusionDrive, and Rap∗ gain 5.4, 7.4, and 10.9 EPDMS points, respectively, on NAVSIM v2 navtest.All methods use camera input only, without LiDAR.Source: §3.1 · p. 4, Table 1 · p. 5
  • On NAVSIM v2 navhard, Rap∗ improves from 32.8 to 49.2 EPDMS, a gain of 16.4 points, while all three planners score higher with ViRA.DAC and DDC also improve, reflecting better drivable-area and driving-direction compliance in challenging scenarios.Source: Table 2 · p. 5, §3.1 · p. 5
  • In zero-shot HUGSIM evaluation, ViRA improves overall HD-Score by 2.7, 6.4, and 3.0 points for TransFuser, DiffusionDrive, and Rap∗, respectively.The NAVSIM-trained planners are evaluated without further training.Source: §3.1 · p. 5, Table 3 · p. 6

3.2 REPRESENTATION QUALITY IMPROVES MEASURABLY

  • Qualitative results show improved BEV predictions and planned trajectories, alongside changes in attention to surrounding road users.The paper presents these as evidence of improved perception alongside driving performance.Source: Figure 3 · p. 6, §3.2 · p. 6

3.3 THE GAINS REQUIRE A PRE-TRAINED TARGET

  • Random-target alignment lowers TransFuser BEV segmentation mIoU from 36.4 to 30.7, whereas pretrained DVGT alignment raises it to 39.1.Random-target alignment also lowers EPDMS by 1.6 points for TransFuser and 1.7 for Rap∗; a pretrained target improves both planners.Source: Table 4 · p. 6, §3.3 · pp. 6–7, Table 5 · p. 7

3.4 GAINS WITHOUT ADDITIONAL INFERENCE COST

  • DVGT alignment raises EPDMS from 72.8 to 83.7 while retaining 20 ms latency and 69M parameters, nearly matching DINOv3-L’s 83.8 EPDMS at 65 ms and 351M parameters.The VFM and alignment heads are used only during training; Figure 4 contrasts ViRA with backbone replacement and indicates parameter count by bubble size.Source: §3.4 · p. 7, Figure 4 · p. 7

4 PLANNING GAINS DEPEND ON BOTH VFMS AND PLANNERS

Planning gains vary with the VFM target and planner supervision: a different target can improve a planner with a pretrained VFM backbone, while auxiliary perception supervision reduces variation across targets. ViRA-Diffusion applies these findings without auxiliary perception supervision.

4.1 THE CHOICE OF VFM TARGET MATTERS

  • For Rap∗ with a DINOv3-L backbone, VGGT alignment raises EPDMS from 83.8 to 86.3, a 2.5-point improvement.Across targets, the perception-free Rap∗’s EPDMS ranges from 79.4 to 84.9, a 5.5-point spread; DINOv3 gives the largest gain in this setting.Source: Table 6 · p. 7, §4.1 · pp. 7–8

4.2 AUXILIARY SUPERVISION SHAPES SENSITIVITY TO VFM TARGETS

  • Across five targets, TransFuser’s EPDMS spread is 0.5 points with auxiliary supervision and 2.7 without it.Without supervision, the maximum is nearly unchanged (89.3 to 89.1), while the minimum drops from 88.8 to 86.4; the comparison indicates that supervision mainly compensates for less effective targets.Source: Figure 6 · p. 8, §4.2 · p. 8

4.3 PRACTICAL IMPLICATIONS

  • ViRA-Diffusion, trained with DINOv3 alignment and no auxiliary perception supervision, achieves 92.3 EPDMS on NAVSIM v2 navtest.Table 7 compares it with recent methods; the passage describes its result as comparing favorably with those included.Source: Table 7 · p. 9, §4.3 · p. 9

5 RELATED WORK

Related work improves end-to-end driving through policy-head design, auxiliary perception supervision, data scaling, and VFM-based representation methods. The paper identifies limited understanding of how planner design and supervision shape VFM-guided planning gains.

Related work

  • Prior end-to-end driving work has focused on policy heads, auxiliary perception supervision, and scaling training data through synthetic scenes and augmentation.Other work uses semantic and geometric priors from pretrained VFMs, but its benefits are demonstrated within specific systems and offer limited guidance across targets and planner designs.Source: §5 · p. 9
  • Driving studies use VFMs through backbone replacement, auxiliary supervision from VFM predictions, or feature integration, while representation alignment supervises intermediate features with frozen visual representations.How planner architecture and auxiliary perception supervision shape VFM-guided representation benefits remains unclear.Source: §5 · p. 9

6 CONCLUSION AND OUTLOOK

The paper concludes that VFM-guided alignment improves driving across planners, while target choice and auxiliary supervision shape the gains. ViRA-Diffusion achieves 92.3 EPDMS on NAVSIM v2 navtest, and the authors identify broader planner and task coverage as future work.

6 CONCLUSION AND OUTLOOK

  • 92.3 EPDMS on NAVSIM v2 navtest: ViRA-Diffusion uses DINOv3 alignment without auxiliary perception supervision.The conclusion identifies this as the planner developed from the study’s findings.Source: §6 · p. 10
  • VFM-guided representations yield consistent driving gains across planners and benchmarks, including zero-shot closed-loop evaluation.The paper concludes that target selection matters, while auxiliary perception supervision narrows performance differences across targets.Source: §6 · p. 10

A.1 VFM-GUIDED E2E PLANNER

  • ViRA aligns planner features with a frozen VFM target while leaving the planner architecture and inference cost unchanged.The VFM branch supervises alignment during training and is discarded before inference.Source: §A.1 · pp. 14–15, Figure S.1 · p. 14

A.2 LATENT REPRESENTATION ALIGNMENT

  • Spatial Dependency Alignment matches spatial dependencies, while Scene Semantic Alignment uses bidirectional contrastive learning to preserve global semantic consistency.Together, the objectives improve planner representations for trajectory planning.Source: §A.2 · p. 15

A.3 TRAINING OBJECTIVE

ViRA adds representation-alignment losses to a planner’s existing training objective without changing its architecture or training protocol. The original trajectory and native auxiliary supervision remain part of that objective when present.

A.3 TRAINING OBJECTIVE

  • ViRA adds representation-level alignment losses while keeping the planner architecture and training protocol unchanged.The original planner objective may include trajectory supervision and native auxiliary objectives.Source: §A.3 · p. 15
  • The total objective combines the planner objective with weighted spatial and semantic alignment losses.For planning-only planners, the original objective contains trajectory prediction; for planners with auxiliary objectives, those terms are retained unchanged.Source: §A.3 · p. 15
  • The alignment weights are λSDA = 1 and λSSA = 0.1, giving greater weight to spatial alignment.Table S.1 summarizes model and training hyperparameters.Source: §A.3 · p. 15, Table S.1 · p. 16

B.1 BENCHMARK AND METRICS

The evaluation uses NAVSIM v2 driving benchmarks, zero-shot HUGSIM closed-loop simulation, and BEV segmentation to assess planner performance and representations. Training and alignment implementation details cover the evaluated planners and frozen VFM targets.

B.1 BENCHMARK AND METRICS

  • NAVSIM v2 evaluation uses navtest and navhard, with EPDMS combining multiplicative penalties and weighted driving-quality scores.Navtest covers 12,146 real-world scenarios; navhard uses 244 challenging real-world and 4,164 synthetic follow-up scenarios.Source: §B.1 · p. 16
  • HUGSIM evaluates NAVSIM-trained planners zero-shot in closed loop across four difficulty levels, reporting HD-Score on a 0–100 scale.Surrounding vehicles react to the planner’s output during simulation.Source: §B.1 · p. 16
  • Representation semantics are measured with BEV segmentation mIoU averaged over seven classes, including static objects and pedestrians.Those two classes are omitted from the main-text class-wise table because their IoU remains below 0.5 for every evaluated target, but they remain in mIoU.Source: §B.1 · pp. 16–17

B.2 TRAINING DETAILS

  • Training follows the NAVSIM protocol on navtrain; TransFuser and DiffusionDrive use detection and segmentation auxiliaries, unlike Rap*.The planners use their corresponding official settings.Source: §B.2 · p. 17

C.1 LATENT ALIGNMENT IMPLEMENTATION

  • Frozen VFM targets provide intermediate representations before their task-specific prediction heads, avoiding reliance on explicit pseudo-labels.The implementation uses official public checkpoints for DVGT, VGGT, DINOv3, SAM3, and DA3.Source: §C.1 · p. 18
  • Planner and VFM features are resized and projected into a shared latent space; VFM and alignment heads are discarded after training.This introduces no additional inference cost.Source: §C.1 · p. 18

C.2 DIFFERENT ALIGNMENT OBJECTIVES ABLATION

The ablation favors jointly aligning spatial dependencies and semantic consistency, with KL divergence and contrastive loss performing best among the tested combinations. Supplementary feature correlations also become stronger and more structured after DVGT supervision.

C.2 DIFFERENT ALIGNMENT OBJECTIVES ABLATION

  • 89.0 EPDMS: KL divergence for spatial alignment plus contrastive loss for semantic alignment performs best among the tested objective combinations.Joint spatial and semantic alignment generally outperforms a single alignment objective.Source: §C.2 · p. 18

C.3 SUPPLEMENTARY ALIGNMENT ANALYSIS

  • After DVGT supervision, TransFuser features show stronger, more structured correlation with VFM features than the original encoder.The authors interpret this as feature-level evidence that ViRA narrows the representation gap.Source: Figure S.2 · p. 18, §C.3 · pp. 18–19

C.4 VFM TARGET DEPENDENCY ANALYSIS

ViRA’s planning gains vary by VFM target and persist across planner designs; random-target controls and BEV segmentation results further characterize the alignment effects.

C.4 VFM TARGET DEPENDENCY ANALYSIS

  • Across the five targets used in the main comparison, ViRA improves TransFuser by 5.2–5.7 EPDMS and Rap* by 6.6–12.1 EPDMS, showing gains across both planners.Source: §C.4 · p. 19
  • Removing auxiliary perception supervision widens the five-target EPDMS spread from 0.5 to 2.7 points, indicating greater sensitivity to target selection.The highest score stays nearly unchanged, while the lowest falls from 88.8 to 86.4.Source: Table S.4 · p. 19, §C.4 · p. 19
  • Random-target alignment lowers EPDMS by 1.6 points for TransFuser and 1.7 for Rap*, while pre-trained DVGT raises them by +5.4 and +10.9, respectively.Applying the same objectives to random features does not reproduce the gains from pre-trained VFM representations.Source: §C.4 · p. 19, Table S.3 · p. 19
  • All pre-trained VFM targets improve TransFuser BEV segmentation mIoU, with gains ranging from +0.5 for SAM3 to +3.0 for DINOv3.The reported results are class-wise BEV segmentation performance under different VFM targets.Source: §C.4 · p. 19

D.1 ADDITIONAL QUALITATIVE ANALYSIS

Additional visualizations compare ViRA across planners, VFM targets, and driving scenarios; failure cases show that improved representations do not ensure successful trajectories in challenging settings.

D.1 ADDITIONAL QUALITATIVE ANALYSIS

  • Additional visualizations show ViRA’s scene understanding and planning behavior across perception-based and perception-free planners, VFM targets, and scenarios.Examples include complex intersections, comparisons across targets, and settings where perception-free planning benefits vary by target and scenario.Source: §D.1 · p. 20

D.2 FAILURE CASES

  • Even with more complete BEV semantic maps, perception-based planners can still miss the human expert’s trajectory in challenging scenarios.The paper identifies trajectory decoding, temporal reasoning, interaction modeling, and decision-making under uncertainty as remaining needs for robust planning.Source: §D.2 · p. 20

E LIMITATIONS AND BROADER IMPACT

ViRA has added training-time costs and has not been evaluated across broader architecture and deployment settings; the authors also flag potential VFM bias risks and document asset licensing.

E LIMITATIONS AND BROADER IMPACT

  • Extracting and storing frozen VFM representations can add training cost, especially with large-scale VFMs or high-resolution multi-view inputs.Feature caching, lightweight VFM distillation, or selective alignment are proposed to reduce this cost and improve scalability.Source: E LIMITATIONS AND BROADER IMPACT · pp. 20–21
  • Evaluation covers three E2E planners, NAVSIM v2 navtest and navhard, and zero-shot HUGSIM, but not sensor fusion, vision-language-action models, or real-vehicle deployment.The authors call for validation on more planners and real-world data to assess generalization.Source: E LIMITATIONS AND BROADER IMPACT · p. 21
  • VFM pre-training may transfer geographical, weather, cultural, or traffic-distribution biases, potentially degrading performance in underrepresented regions or rare safety-critical scenarios.The authors call for evaluation across diverse environments, transparent failure analysis, and safety-oriented validation before deployment.Source: E LIMITATIONS AND BROADER IMPACT · p. 21

F LICENSE OF ASSETS

  • The study uses public datasets, benchmarks, VFMs, and planners under their associated licenses and terms, and does not redistribute third-party data or model assets.Released code will include attribution and license notices for the assets used.Source: F LICENSE OF ASSETS · pp. 21–22

Related papers

Loading 2610.09695v1…