PaperlayerView exact revision on arXiv

Source-linked AI summary

ConDenseDrive: Knowledge-Dense, Token-Sparse Visual Encoder for End-to-End Autonomous Driving

Shengzhen Li, Yongkang Li, Hao Gao, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang

arXiv:2610.16908v1cs.CVcs.RO

TL;DR

End-to-end driving needs spatiotemporal visual representations, but generic image pretraining and dense patch processing leave representation quality and inference efficiency jointly under-addressed. ConDenseDrive distills complementary visual knowledge, adapts register tokens across frames, and uses them to guide token compression. It reports improved planning with a 2.0× speedup and strong results across NAVSIM and zero-shot HUGSIM evaluation.

  • Problem

    Visual encoders are often initialized from generic image pretraining and densely process camera-frame patches, while representation quality and computational cost are rarely considered jointly.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    ConDenseDrive distills complementary visual, semantic, and geometric knowledge, adapts register tokens across frames for planning, and uses those registers to guide patch-token compression.

    Source: §1 · p. 2

  • Results

    94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2; zero-shot HUGSIM reaches 37.4 HD-Score, with a 2.0× speedup while improving planning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments and qualitative analyses support visual encoder design as a route to reliable and efficient end-to-end driving.

    Source: §6 · p. 10

Abstract

from arXiv · show

End-to-end driving requires rich spatiotemporal representations to support scene understanding and trajectory planning under strict latency constraints. However, most systems initialize their visual encoders from generic image pretraining and densely process all patch tokens for each camera view and input frame, limiting both representation quality and inference efficiency. To address these challenges, we present ConDenseDrive, a knowledge-dense and token-sparse visual encoder for end-to-end autonomous driving. Our three-stage framework first makes the encoder knowledge-dense by distilling complementary visual, semantic, and geometric knowledge from three vision foundation models. It then adapts the distilled encoder to multi-frame end-to-end driving and integrates spatiotemporal context into register tokens for planning. Finally, it makes the encoder token-sparse by applying register-guided token compression, reducing redundant patch computation and yielding a 2.0$\times$ speedup while improving planning performance. ConDenseDrive achieves 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2, and reaches 37.4 HD-Score in zero-shot evaluation on HUGSIM. Extensive experiments and qualitative analyses demonstrate that visual encoder design provides an effective route to reliable and efficient end-to-end driving.

1 INTRODUCTION

End-to-end driving relies on visual encoders for scene information, yet generic pretraining and dense processing leave representation quality and computational cost insufficiently addressed together. ConDenseDrive combines complementary knowledge distillation, multi-frame register adaptation, and register-guided token compression; reported results pair stronger planning with faster encoding.

1 INTRODUCTION

  • Visual encoders provide scene information to planners and can add substantial inference latency, but their representation quality and computational cost are rarely considered jointly.Scaling a backbone can enrich representations but increases visual-encoding latency; multi-camera and historical-frame context also brings a dense-processing cost.Source: §1 · pp. 1–2
  • ConDenseDrive distills complementary visual, semantic, and geometric knowledge, adapts the encoder across frames using register tokens, then compresses patches using those registers.Compression retains informative patches and fuses the rest without an auxiliary routing network.Source: §1 · p. 2
  • 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2; zero-shot HUGSIM reaches 37.4 HD-Score, with a 2.0× speedup while improving planning.The paper attributes the gains to encoder-centered optimization without redesigning the downstream planner.Source: §1 · p. 2

2 RELATED WORK

Prior end-to-end driving work has primarily developed scene representations and planning components, while visual encoders and token processing offer a distinct design target. ConDenseDrive positions its approach among visual distillation and token-compression methods by using driving-oriented registers to guide patch processing.

End-to-End Autonomous Driving

  • Prior systems emphasize structured scene representations such as BEV and vectorized elements, trajectory generation, trajectory selection, VLA reasoning, or closed-loop reinforcement learning.Examples include UniAD and VAD, DiffusionDrive and DriveSuprim, and RAD and RAD-2 [Gao et al., 2026a;b].Source: §2 · pp. 2–3

Visual Knowledge Distillation

  • Visual distillation methods combine complementary vision teachers or transfer foundation-model features to compact encoders, and DriveZero applies multi-teacher distillation to end-to-end driving.ConDenseDrive jointly addresses visual knowledge transfer and encoder efficiency for end-to-end driving.Source: §2 · p. 3

Token Compression

  • Compression methods prune, merge, fuse, or variably process tokens, while driving-specific approaches compress extracted features or temporal context.ConDenseDrive instead scores patch tokens by attention from driving-oriented register tokens, retaining informative patches and fusing lower-scoring ones within selected encoder blocks.Source: §2 · p. 3

3 PRELIMINARIES

The task takes synchronized multi-camera images over observed frames and auxiliary ego-state and navigation inputs, then predicts future ego poses. The stated objective is to learn driving-oriented spatiotemporal representations while reducing computation on less relevant patches without degrading trajectory prediction.

3 PRELIMINARIES

  • The input comprises synchronized images from V camera views over T observed frames, auxiliary ego-state and navigation information, and a predicted trajectory of Q future ego poses.The trajectory is represented as positions and heading for each future pose.Source: §3 · p. 3
  • The objective is to optimize the visual encoder while retaining the downstream planner interface, learning driving-oriented spatiotemporal representations and reducing computation on less relevant patch tokens without degrading trajectory prediction.Processing every view and frame contributes a patch-token sequence and makes visual encoding costly.Source: §3 · p. 3

4 METHOD

ConDenseDrive combines multi-teacher distillation, multi-frame register adaptation, and register-guided token compression. Together, the stages enrich a compact encoder, incorporate spatiotemporal context into planning registers, and reduce visual encoding cost.

4.1 MULTI-TEACHER VISUAL DISTILLATION

  • Three frozen teachers provide complementary visual, semantic, and geometric supervision to enrich the student encoder without increasing its size.DINOv3, SigLIP2, and Depth Anything 3 supervise dense patch features; adapters align student and teacher features, then are discarded before Stage 2.Source: §4.1 · pp. 3–4

4.2 MULTI-FRAME REGISTER ADAPTATION

  • Multi-frame register adaptation uses camera-specific registers to aggregate image evidence, then adds temporal and ego-motion embeddings to capture scene evolution across frames.Temporal attention runs independently within each camera stream; registers and ego state and navigation information are passed to the trajectory generator and scorer.Source: §4.2 · p. 5

4.3 REGISTER-GUIDED TOKEN COMPRESSION

  • Register-guided compression scores patch tokens by attention from planning registers, retains the highest-scored fraction, and merges lower-scored tokens into one fused token.The routing scores are shaped by the end-to-end driving objective, and the method introduces no auxiliary routing network or loss.Source: §4.3 · p. 5

5 EXPERIMENTS

Experiments evaluate ConDenseDrive on NAVSIM v1 and v2 and zero-shot HUGSIM, then isolate its stages and design choices. Results cover planning performance, inference efficiency, and visualized feature and token behavior.

5.1 EXPERIMENTAL SETUP

  • Training uses NAVSIM navtrain (103,288 scenes), evaluates on navtest (12,146), and tests zero-shot on 436 HUGSIM scenarios using PDMS, EPDMS, and HD-Score.The implementation uses front, rear, left, and right camera images and 16 register tokens per view per frame.Source: §5.1 · pp. 5–6

5.2 MAIN RESULTS

  • ConDenseDrive reaches 94.5 PDMS on NAVSIM v1 and 90.8 EPDMS on NAVSIM v2, outperforming strong scene- or planner-focused approaches.The NAVSIM v1 model uses T = 2 and ρ = 0.4; on NAVSIM v2 it exceeds CLOVER by 0.4 EPDMS.Source: §5.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6
  • In zero-shot evaluation across all 436 HUGSIM scenarios, ConDenseDrive scores 37.4 HD-Score, 4.9 points above DrivoR.The three-stage training pipeline uses no data from HUGSIM’s four source datasets.Source: §5.2 · p. 7, Table 3 · p. 7
  • At T = 2, ρ = 0.4 gives a 2.0× speedup while improving PDMS; at T = 1, compression cuts perception latency 30.0% at ρ = 0.7 and 46.6% at ρ = 0.3.The selected two-frame model achieves 94.5 PDMS at 26.6 FPS.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 ABLATION STUDIES

  • Multi-teacher distillation raises PDMS from 91.8 to 94.1 without throughput loss; temporal attention reaches 94.3, and compression restores throughput to 26.6 FPS at 94.5 PDMS.The two-frame temporal-attention model runs at 13.6 FPS before compression.Source: §5.3 · p. 8, Table 4 · p. 8
  • Register-token routing scores 94.3 PDMS versus 94.0 with CLS; dense-checkpoint initialization raises PDMS from 93.6 to 94.3.The routing comparison uses T = 1 and ρ = 0.7; the initialization comparison uses the same setting.Source: §5.3 · p. 8, Table 5 · p. 8, Table 6 · p. 8
  • At ρ = 0.4, encoder computation falls 59.0% and throughput reaches 26.6 FPS while PDMS rises from 94.3 to 94.5.At ρ = 0.3, throughput reaches 28.1 FPS while matching the ρ = 1.0 result; compressed rates use the same training hyperparameters.Source: §5.3 · p. 8, Table 7 · p. 8

5.4 QUALITATIVE ANALYSIS

  • Visualizations show student features preserving fine appearance and broad depth cues, while compression retains salient nearby objects and varies token allocation across camera views.The front view preserves more road-surface tokens near the driving path; side views prioritize nearby obstacles, while sky and distant-building tokens are fused.Source: §5.4 · pp. 8–9, Figure 4 · p. 9, Figure 5 · p. 9

6 CONCLUSION

ConDenseDrive treats the visual encoder as a central design target for reliable and efficient end-to-end driving. Its three-stage framework enriches a compact encoder and concentrates computation on planning-relevant patches.

6 CONCLUSION

  • ConDenseDrive enriches a compact visual encoder with complementary visual knowledge and concentrates computation on planning-relevant image patches.The stated goal is to improve end-to-end driving reliability and efficiency.Source: §6 · p. 9
  • The framework combines multi-teacher distillation, multi-frame register adaptation, and register-guided token compression.Source: §6 · p. 10
  • Experiments and qualitative analyses show improved planning performance and inference efficiency.The authors hope this work encourages further research on driving-oriented visual encoders.Source: §6 · p. 10

A IMPLEMENTATION DETAILS

The implementation appendix specifies evaluation metrics and protocols, training configurations, and efficiency profiling procedures for ConDenseDrive.

A IMPLEMENTATION DETAILS

  • NAVSIM v1 uses PDMS, which combines collision, drivable-area, time-to-collision, comfort, and ego-progress components.Higher PDMS values indicate better planning performance.Source: A IMPLEMENTATION DETAILS · p. 14
  • NAVSIM v2 uses EPDMS, adding direction, traffic-light, and lane-keeping compliance and replacing comfort with history comfort and extended comfort.The paper reports corrected official-evaluator scores as EPDMS and original-evaluator scores as EPDMS∗.Source: A IMPLEMENTATION DETAILS · p. 14
  • HUGSIM evaluation measures zero-shot closed-loop generalization across all 436 scenarios without HUGSIM-specific training or fine-tuning.Overall HD-Score weights difficulty levels according to their scenario counts.Source: A IMPLEMENTATION DETAILS · p. 14
  • All three training stages use AdamW; stages 2 and 3 sample frames at 2 Hz and adapt the frozen backbone with rank-32 LoRA updates.Stage 1 distills DeiT-S from DINOv3, SigLIP2, and Depth Anything 3; stage 3 enables register-guided token compression.Source: A IMPLEMENTATION DETAILS · p. 14
  • Profiling uses batch size one, four camera views, and an NVIDIA A100, measuring mean inference latency over 1,000 timed iterations after 100 warm-ups.The timed region excludes data loading and host-side preprocessing; throughput is 1,000 divided by mean latency in milliseconds.Source: A IMPLEMENTATION DETAILS · p. 14

B ADDITIONAL STAGE 1 STUDIES

A camera-only DiffusionDrive transfer study tests whether the distilled encoder can improve planning without token compression or changes to the remaining setup.

B ADDITIONAL STAGE 1 STUDIES

C ADDITIONAL STAGE 2 STUDIES

The additional stage 2 studies compare temporal length, attention variants, and which register tokens are passed to the planner.

C ADDITIONAL STAGE 2 STUDIES

D ADDITIONAL STAGE 3 STUDIES

Additional Stage 3 studies compare token fusion with discarding and examine how keep rate trades encoder computation and throughput against PDMS.

D ADDITIONAL STAGE 3 STUDIES

E ADDITIONAL QUALITATIVE VISUALIZATIONS

Additional visualizations compare student features with foundation-model features and show register-guided token compression across camera views and scenes.

E ADDITIONAL QUALITATIVE VISUALIZATIONS

Related papers

Loading 2610.16908v1…