Source-linked AI summary
Driving on Registers
Ellington Kirby, Alexandre Boulch, Yihong Xu, Yuan Yin, Gilles Puy, Éloi Zablocki, Andrei Bursuc, Spyros Gidaris, Renaud Marlet, Florent Bartoccioni, Anh-Quan Cao, Nermin Samet, Tuan-Hung VU, Matthieu Cord
TL;DR
End-to-end driving must process extensive sensor features while selecting among uncertain trajectory proposals. DrivoR compresses multi-camera ViT features into camera-aware registers, uses separate decoders for trajectory generation and scoring, and achieves state-of-the-art results across three benchmarks while supporting behavior modulation.
Problem
Trajectory-proposal planning requires reliable scoring of multiple possible futures, but backbone features produce thousands of tokens that create a major computational bottleneck.
Method
DrivoR fine-tunes per-camera ViT registers into compact scene tokens and uses disentangled trajectory-generation and scoring modules.
Results
DrivoR achieves state-of-the-art results on NAVSIM-v1, NAVSIM-v2, and the closed-loop HUGSIM benchmark.
Takeaways & Limitations
A simple full-transformer architecture with register-based compression and disentangled scoring can support accurate and behavior-controllable end-to-end driving.
Takeaways & Limitations
The evaluation required corrected implementations because official NAVSIM code contained anomalies that made its original results unreliable and not directly comparable.
Abstract
from arXiv · showhide
We present DrivoR, a simple and efficient transformer-based architecture for end-to-end autonomous driving. Our approach builds on pretrained Vision Transformers (ViTs) and introduces camera-aware register tokens that compress multi-camera features into a compact scene representation, significantly reducing downstream computation without sacrificing accuracy. These tokens drive two lightweight transformer decoders that generate and then score candidate trajectories. The scoring decoder learns to mimic an oracle and predicts interpretable sub-scores representing aspects such as safety, comfort, and efficiency, enabling behavior-conditioned driving at inference. Despite its minimal design, DrivoR outperforms or matches strong contemporary baselines across NAVSIM-v1, NAVSIM-v2, and the photorealistic closed-loop HUGSIM benchmark. Our results show that a pure-transformer architecture, combined with targeted token compression, is sufficient for accurate, efficient, and adaptive end-to-end driving. Code and checkpoints will be made available via the project page.
1. Introduction
DrivoR targets the computational bottleneck created by processing thousands of visual tokens across many trajectories. It replaces uniform pooling with camera-aware register compression, then generates and scores trajectory proposals for adaptive selection.
- E2E planning methods map raw sensor data and ego state directly to driving decisions, while trajectory proposals capture navigation uncertainty.
- Thousands of backbone tokens must be processed for hundreds of trajectories, making perception a major parameter and FLOP bottleneck that worsens with resolution or sensor count.
- Uniform spatial pooling imposes sensor-resolution requirements and averages all cameras equally, motivating compact scene representations that retain planning-relevant information.
- DrivoR uses fixed per-camera register tokens as compact scene descriptors, replacing uniform pooling while reducing visual representation length.
- The architecture generates and scores trajectory proposals with disentangled modules, selecting the final trajectory using predicted sub-scores for behavior modulation at inference.
- DrivoR relies on scoring annotations rather than explicit 3D supervision and achieves state-of-the-art results on NAVSIM-v1, NAVSIM-v2, and HUGSIM.
2. Related Work
Related work emphasizes token reduction for efficient vision transformers, direct end-to-end driving, and trajectory scoring. DrivoR builds on these directions by using sensor registers as downstream scene tokens.
- Vision-transformer attention costs grow quadratically with sequence length, motivating token-reduction methods ranging from pooling and matching to learned latent queries and register tokens.
- End-to-end driving has progressed from modular systems toward methods that predict planned trajectories or actions directly from sensor inputs.
- Many end-to-end methods use off-the-shelf CNN image encoders, while transformer innovation has often focused on mid-level sensor features rather than stronger pretrained backbones.
- Trajectory-proposal planning addresses multimodality by producing multiple futures, making reliable candidate scoring central to selecting the best trajectory.
- DrivoR introduces sensor registers in the encoder and uses the resulting registers as scene tokens in downstream decoders.
3. Method
DrivoR uses a simple transformer encoder-decoder pipeline that compresses per-camera ViT features into camera-aware scene tokens, then generates and scores candidate trajectories. Separate scoring representations and interpretable sub-scores support trajectory selection under adjustable driving preferences.
- Overall architecture: DrivoR uses a classical transformer encoder with two decoders, avoiding complex intermediate representations.The encoder handles perception, while separate decoders estimate trajectories and score them.
- Perception encoder: Per-camera ViT registers compress visual features into N × R camera-aware scene tokens.Registers are finetuned with the backbone and preserve camera identity, allowing the model to distinguish views such as front, left, and right.
- Trajectory decoder: Learnable trajectory queries attend to scene tokens through cross-attention and are decoded into candidate trajectories.Ego status inputs are added to the queries before decoding, and an MLP maps final trajectory tokens to trajectories.
- Trajectory learning: Winner-Takes-All training supervises only the closest predicted trajectory, encouraging diverse plausible pathways.An optional longer-horizon target can add a second regression target after resampling it to the prediction horizon.
- Scoring decoder: The scoring decoder embeds each decoded trajectory separately, preventing it from reusing latent details retained in trajectory-generation tokens.It attends to scene tokens, blocks scoring gradients from flowing into the trajectory decoder, and predicts six dedicated score components.
- Behavior-conditioned scoring: Separate sub-scores for aspects such as safety, comfort, and efficiency can be recombined at inference to select trajectories for different behavior profiles.The scoring network learns each component against an oracle scorer, and changing the component weights changes the reward used for trajectory selection.
4. Experiments
DrivoR achieves strong benchmark performance, efficient inference, and competitive scaling through a compact ViT-based architecture with register compression and separate trajectory and scoring modules. Ablations show that pretrained perception, camera-aware token behavior, disentangled scoring, and task-specific training choices materially affect results.
- Benchmark results: DrivoR outperforms all other methods on NAVSIM-v1 and approaches human-level performance.The comparison uses camera-only methods on the NAVSIM-v1 test set.
- Benchmark results: DrivoR outperforms existing works on NAVSIM-v2 navhard-two-stage after the official evaluation bug fix.The benchmark evaluates generalization to perturbed ego states with the extended EPDMS metric.
- Benchmark results: 49.8 RC and 35.7 HD-Score are achieved zero-shot on HUGSIM using a model trained only on NAVSIM-v1.These are the highest scores among the reported baselines on the pre-challenge HUGSIM test set.
- Efficiency: More than 3x throughput improvement, from 400ms/forward to 110ms/forward, accompanies 3x reductions in GFLOPS and peak memory usage.This comparison uses a ViT-L GTRS baseline on a single-element batch and an A100 GPU without quantization or acceleration.
- Perception and compression: Register-based compression improves performance over comparable pooling while reducing the visual representation used by downstream trajectory prediction and scoring.The architecture uses 16 registers per camera by default, while register similarity indicates specialized front-camera tokens and collapsed rear-camera tokens.
- Ablations: Increasing registers improves performance up to a plateau between 16 and 32 per camera, while increasing trajectory queries yields gains that plateau around 64.The default configuration selects 16 registers and 64 trajectory queries.
- Scoring and training: Separate trajectory-generation and scoring branches are supported by distinct camera attention patterns, while disentanglement and multiple sub-score prediction improve performance.Generation emphasizes the front view, whereas scoring uses left or rear views depending on trajectory sharpness or collision risk.
- Training objectives: The longer-trajectory objective improves NAVSIM-v1 but hurts NAVSIM-v2 warmup-two-stage, reflecting different preferences for progress versus cautious recovery under perturbed states.NAVSIM-v1 rewards progress over comfort, whereas NAVSIM-v2 requires more cautious driving to avoid collisions and recover safely.
5. Conclusion
DrivoR is an end-to-end driving method using register-based compression and disentangled scoring representations. The authors conclude that full-transformer architectures without complex intermediate states or large trajectory dictionaries can achieve state-of-the-art results.
- DrivoR combines register-based compression with disentangled scoring representations for end-to-end driving.
- Full-transformer architectures can achieve state-of-the-art results without complex intermediate states or large trajectory dictionaries.
- Future work may extend compression to historical frames, additional sensors, or map information.
A. Description of PDMS and EPDMS metrics
PDMS and EPDMS aggregate penalty and behavioral sub-metrics, with multiplicative rule-compliance penalties and additive behavioral terms. DrivoR predicts these sub-scores separately, enabling configurable inference weighting.
- Metric structure: PDMS and EPDMS combine penalty metrics measuring driving-rule compliance with behavioral metrics such as comfort and progress.
- Metric structure: Penalty metrics are multiplicative, so one failed targeted driving rule can reduce the overall score to zero.
- Metric structure: Behavioral metrics are additive, allowing trade-offs between terms such as comfort and progress.
- Training and inference weights: During training, each predicted sub-score uses an individual loss with all sub-score weights λc set to 1.
- Training and inference weights: At inference, NAVSIM-v1 retains standard PDMS weights, while NAVSIM-v2 adjusts weights except for Comfort.
E. Efficiency analysis
The efficiency analysis compares model computation and trajectory decoding choices. DrivoR uses compact register-based representations and single-token trajectory decoding to improve efficiency and produce smoother trajectories.
- Throughput analysis: Table 11 compares parameters, GFLOPs, peak memory, and throughput against NAVSIM-v2 EPDMS performance.
- Throughput analysis: FLOP counts exclude Scaled Dot Product Attention, so absolute counts are incomplete although relative performance should remain consistent.
- Trajectory decoding: Single-token trajectory decoding produces a large performance increase compared with decoding a set of tokens.
- Trajectory decoding: Single-token trajectories are much smoother and less noisy than multi-token trajectories.
G. Expanded NAVSIM-v1 results
The expanded NAVSIM-v1 comparison evaluates DrivoR against existing methods across potentially different sensor modalities and post-processing choices. The authors emphasize competitive performance without post-processing or additional sensor modalities.
- Benchmark comparison: RAP is trained on a dataset 10× larger than the default navtrain training set.
- Benchmark comparison: Table 13 compares NAVSIM-v1 test-set scores across existing methods using potentially different sensor modalities.
- Evaluation protocol: Best-of-N scoring uses ground truth to select among predicted trajectories, which is not aligned with the NAVSIM evaluation protocol.
- Evaluation protocol: DrivoR achieves competitive performance compared with methods using post-processing techniques or additional sensor modalities.
H. NAVSIM-v2 results before benchmark fix
DrivoR ranks among the strongest NAVSIM-v2 methods despite a much lighter design and no extra training data, ranking first after the benchmark bug fix.
- DrivoR ranks among the best-performing methods with a much lighter model design and without extra training data.The full comparison includes results from before the NAVSIM benchmark bug fix.
- DrivoR ranks first among state-of-the-art methods after the NAVSIM-v2 benchmark bug fix.The fix concerns filtering out human driver errors, which are not penalized after correction.
I. Limitations and additional visualizations
Visualizations show camera-dependent attention, a current-frame navigation failure mode, and behavior-conditioned trajectory changes that improve navigation in NAVSIM-v2 OOD scenes.
- Camera focus: DrivoR’s scoring and prediction pathways focus on different cameras across diverse NAVSIM-v1 scenarios.The visualizations include right-turn, intersection-crossing, and failure-case scenes.
- Camera focus: Adding Traffic Light Compliance scoring may redirect attention toward visible traffic lights, although DrivoR already achieves a very high TLC score without active TLC scoring.The intersection-crossing example shows scoring attention focused mainly on the rear camera despite the visible traffic light.
- Failure cases: Without historical frames, ambiguous current scenes can produce wrong-way trajectories when the front camera lacks interpretable objects.In the failure visualization, attention focuses entirely on the right camera and the trajectories undercut the turn.
- Behavior re-weighting: Tuned passive behavior produces shorter, less aggressive trajectories that better navigate NAVSIM-v2 out-of-distribution scenes than the default agent.The default NAVSIM-v1 agent travels faster with less spread, whereas the tuned passive agent is better suited to NAVSIM-v2 OOD scenes.
- Comparison details: RAP uses a dataset 10× larger than navtrain, while a comparison variant shares the ViT-S backbone and DrivoR registers with GTRS heads retained.
- Comparison details: Table 14 reports the NAVSIM-v2 navhard-two-stage comparison using the EPDMS metric.GTRS-A denotes GTRS-Aug, and GTRS-D denotes GTRS-Dense.