Source-linked AI summary
Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving
Kewei Zhang, Jin Wang, Sensen Gao, Chengyue Wu, Yulong Cao, Songyang Han, Boris Ivanovic, Langechuan Liu, Marco Pavone, Song Han, Daquan Zhou, Enze Xie
TL;DR
Driving VLAs must combine accurate, globally consistent trajectory planning with efficient inference, but autoregressive and full-sequence diffusion approaches have important limitations. Fast-dDrive uses structure-aware block diffusion and achieves state-of-the-art trajectory accuracy at 6× the throughput of full-sequence diffusion baselines.
Problem
Driving VLAs must jointly provide accurate, globally consistent trajectories and efficient inference, while autoregressive decoding compounds waypoint errors and is memory-bandwidth-bound.
Method
Fast-dDrive freezes deterministic JSON schema tokens as a scaffold, aligns diffusion blocks with semantic sections, prioritizes safety-critical tokens, and supports speculative decoding with shared-prefix rollout scaling.
Results
Fast-dDrive achieves state-of-the-art trajectory accuracy at 6× the throughput of full-sequence diffusion baselines, while maintaining the highest RFS among diffusion-based VLAs.
Takeaways & Limitations
The results support structured generation and efficient decoding as complementary objectives for block-diffusion driving VLAs.
Takeaways & Limitations
Evaluation focuses on open-loop benchmarks, so closed-loop validation of reactive capabilities in dynamic environments remains future work.
Abstract
from arXiv · showhide
End-to-end autonomous driving via Vision-Language-Action (VLA) models demands a precarious balance between high-fidelity trajectory planning and efficient inference. Existing paradigms typically fall short: autoregressive (AR) VLAs are memory-bandwidth-bound on edge hardware and prone to exposure-bias drift, while full-sequence diffusion models preclude KV-cache reuse and suffer from "logical leakage" that violates the fundamental perceive-then-plan causality. We present Fast-dDrive, a block-diffusion VLA that performs bidirectional refinement within semantic units while enforcing strict causal ordering across them. Leveraging the observation that driving VLAs often emit structured JSON-like outputs, Fast-dDrive freezes structural tokens into a section scaffold and employs a section-aware training recipe that prioritizes safety-critical planning. We further introduce Scaffold Speculative Decoding to achieve AR-equivalent quality at significantly higher throughput. Finally, we propose a low-overhead test-time scaling scheme: by forking $N$ stochastic trajectory rollouts from a single shared-prefix KV cache and averaging them, we effectively suppress prediction variance at a fractional computational cost. Empirical results demonstrate that Fast-dDrive redefines the speed-accuracy frontier for driving agents. On the WOD-E2E test set, Fast-dDrive achieves SOTA ADE@3s and ADE@5s, alongside the highest RFS among diffusion-based VLAs; on nuScenes, it reduces average L2 error to $0.32$m (a $22\%$ improvement). When integrated with SGLang, our framework delivers $12\times$ throughput speedup over the AR baseline, narrowing the gap between high-capacity VLAs and the efficiency demands of real-time on-vehicle deployment.
1. Introduction
Fast-dDrive addresses exposure bias and inefficient inference in driving VLAs with section-aligned block diffusion and frozen structural scaffolds. Its speculative decoding and shared-prefix rollout scaling preserve AR-equivalent outputs while improving throughput and accuracy.
- Motivation: AR driving VLAs decode reasoning and trajectory tokens sequentially, allowing early waypoint errors to compound into physically implausible 5 s maneuvers.Each waypoint conditions on previously emitted coordinates, which may be noisy.
- Method: Fast-dDrive freezes schema-determined JSON keys and syntax, denoising only value tokens to focus capacity on positional content.The scaffold covers deterministic structural tokens in structured outputs that bundle perception, chain-of-thought, and trajectory.
- Method: Section-Aware Structured Diffusion aligns block boundaries with semantic sections, guarantees 100% structural validity, and prioritizes safety-critical sections without inference overhead.It combines section-weighted cross-entropy with a section-adaptive Beta noise schedule.
- Inference: Scaffold Speculative Decoding auto-accepts scaffold tokens and verifies parallel masked-diffusion drafts, producing outputs identical to pure AR at lower latency.The AR head verifies the parallel MDM draft while deterministic scaffold tokens are accepted automatically.
- Results: 12× throughput is achieved by combining Fast-dDrive with SGLang, while shared-prefix scaling forks N trajectory rollouts from one KV cache and averages them for accuracy gains.Only the trajectory section is sampled at non-zero temperature, trading fractional extra inference compute for improved accuracy.
- Conclusion: Block-diffusion VLAs with structure-aware training and inference can match or exceed strong AR and full-sequence-diffusion accuracy at substantially higher throughput.The approach retains interpretable structured chain-of-thought outputs.
2. Related Work
Prior work spans autonomous-driving VLAs, diffusion language models, and efficient decoding or test-time scaling. These efforts establish the foundations for Fast-dDrive’s focus on sequential-decoding bottlenecks, diffusion modeling, speculative generation, and shared-prefix rollouts.
- Vision-Language-Action Models for Autonomous Driving: Autoregressive VLAs unify perception, reasoning, and planning, improve trajectory prediction in long-tail scenarios, and remain sequential and memory-bandwidth-bound at batch size 1.Recent work also incorporates chain-of-thought reasoning.
- Diffusion Large Language Models: Discrete diffusion for text has advanced from foundational formulations and masked diffusion objectives to large-scale models matching autoregressive performance.Post-training methods further align diffusion language models with human preferences.
- Efficient Decoding and Test-Time Scaling: Speculative decoding accelerates autoregressive generation by drafting multiple tokens for parallel verification, while self-speculative variants reuse one model for drafting and verification.Fast-dLLM extends speculative decoding to block diffusion.
- Efficient Decoding and Test-Time Scaling: Shared-prefix rollouts amortize deterministic prefix computation and apply stochasticity only to the trajectory section at a fractional per-rollout cost.The deterministic structure of the first three sections enables this computation sharing.
3. Methodology
Fast-dDrive combines block-causal diffusion with a frozen JSON scaffold, section-aware safety training, two inference modes, and shared-prefix stochastic rollout averaging. Its design preserves causal ordering across semantic sections while enabling parallel refinement within sections and reducing inference overhead.
- Block-Causal Diffusion: Fast-dDrive partitions outputs into left-to-right blocks with bidirectional attention within each block and causal attention across blocks, enabling KV-cache reuse.Each block attends to the prompt and preceding blocks, but not future blocks.
- Scaffold Construction and Section-Aligned Blocks: The model freezes JSON structural tokens into a scaffold and denoises only value tokens, guaranteeing 100% structural correctness while reducing denoising workload by ∼30%.Section-aligned blocks follow the causal order CO → Expl → FMB → Traj, with NULL padding for variable-length sections.
- Scaffold Construction and Section-Aligned Blocks: The structured output contains critical_objects, explanation, future_meta_behavior, and trajectory sections whose token counts, difficulty, and safety impacts differ substantially.The trajectory section contains five waypoint coordinates over five seconds, while critical_objects contains 12 binary detections.
- Safety-Prioritized Training: Safety-prioritized training uses section-weighted cross-entropy and section-specific Beta noise schedules, with both mechanisms adding zero inference overhead.Larger section weights emphasize hard, high-impact tokens, while Beta schedules adapt noise to each section’s difficulty profile.
- Scaffold Speculative Decoding: Scaffold Speculative Decoding auto-accepts scaffold tokens, drafts value tokens bidirectionally, and verifies them causally, requiring exactly 2 forward passes per block.Section-aligned blocks provide complete semantic context for drafting and improve acceptance compared with arbitrary fixed-size blocks.
- Test-Time Inference Scaling: Test-time scaling averages N stochastic trajectory rollouts forked from a shared deterministic prefix, and mean ADE@5s decreases monotonically with N.Each rollout is interpolated to 20 waypoints using Jerk-Minimizing Trajectory fitting before equal-weight averaging.
4. Experiments
Fast-dDrive is evaluated on nuScenes and WOD-E2E using camera, ego-state, and navigation inputs, with planning accuracy and single-H100 inference efficiency as principal metrics. It achieves strong accuracy and throughput through Scaffold Speculative Decoding, shared-prefix trajectory rollouts, and complementary SASD training components.
- Experimental Setup: Fast-dDrive uses RGB cameras, ego state, and a high-level navigation command without LiDAR, radar, or HD maps.It uses three past front-camera frames on nuScenes and three front-facing cameras on WOD-E2E.
- WOD-E2E Results: On WOD-E2E, Scaffold Spec attains the lowest ADE@3s and ADE@5s, with RFS above dVLM-AD and competitive with the strongest AR baseline.These results are achieved in a single inference run without GRPO post-training or a larger trajectory pool.
- WOD-E2E Results: 4×–6× decoding throughput over dVLM-AD and AR baselines is achieved while committing approximately 5 tokens per model forward pass.Shared-prefix multi-trajectory rollout further reduces both ADE values at sub-2× the wall-clock cost of one Scaffold-Spec pass.
- nuScenes Results: On nuScenes, Fast-dDrive has the lowest average L2 among listed reasoning VLM/VLA systems, consistently improving over diffusion and AR-with-CoT baselines across three horizons.It also matches or improves upon classical training-based driving policies without interpretable reasoning.
- Inference Efficiency: Scaffold Spec achieves roughly 6× dVLM-AD throughput by combining block-level KV-cache reuse with scaffold-aware speculative decoding.Scaffold auto-acceptance removes approximately 30% of tokens from draft verification while matching vanilla self-speculative accuracy.
- Ablations and Test-Time Scaling: IWL is the primary SASD contributor to RFS, while combining IWL and SNS produces the best RFS because their effects are complementary.ADE@5s decreases monotonically with trajectory rollout count N; N=4 is adopted as the default accuracy–latency trade-off.
5. Conclusion
Fast-dDrive is presented as a block-diffusion VLA that uses driving-output structure to advance planning accuracy and inference efficiency simultaneously. Its frozen scaffold, semantic diffusion blocks, and safety-focused training achieve state-of-the-art trajectory accuracy at 6× the throughput of full-sequence diffusion baselines.
- Fast-dDrive exploits the inherent structure of driving outputs to advance planning accuracy and inference efficiency simultaneously.
- Deterministic schema tokens form a frozen scaffold, while diffusion blocks align with semantic sections and training prioritizes safety-critical tokens.
- 6× throughput accompanies state-of-the-art trajectory accuracy versus full-sequence diffusion baselines, showing structured generation and efficient decoding are complementary.
A. Input Modalities: Full Details
Fast-dDrive uses dataset-specific forward-facing visual inputs: three chronological front-camera frames for nuScenes and three current-frame forward views for WOD-E2E. Inputs are resized without inference-time augmentation and combined with ego state and navigation commands, excluding auxiliary sensors.
- nuScenes inputs: nuScenes uses CAM_FRONT only, with three frames at t∈{−1.0, −0.5, 0} s presented chronologically relative to prediction time.Side and rear cameras are excluded to match the dVLM-AD evaluation protocol and preserve fair comparison.
- WOD-E2E inputs: WOD-E2E uses FRONT_LEFT, FRONT, and FRONT_RIGHT at t=0 s instead of all eight surround views.The three forward views were sufficient for open-loop planning while keeping the visual token budget tractable for the 3B-parameter backbone.
- WOD-E2E inputs: The WOD-E2E joint-view variant concatenates FRONT_LEFT|FRONT|FRONT_RIGHT horizontally, roughly halving visual tokens while trading per-view resolution for a wider panorama.This variant is resized as a single panoramic image before encoding.
- Preprocessing and conditioning: Images preserve aspect ratio with a maximum 512-pixel longer side; inference uses no cropping or photometric augmentation, while inputs also include ego state and navigation commands.The model excludes LiDAR, radar, HD maps, and other auxiliary sensor modalities.
B. Qualitative Case Studies
Fast-dDrive is qualitatively evaluated on five Waymo end-to-end driving scenes spanning diverse planning regimes. Its predictions match the ground-truth trajectory direction in every case.
- Qualitative Case Studies: Five examples cover nighttime left turns, lane-following behind a pickup, right turns, green-light cruising, and wet-weather left turns.The scenes span deliberate steering, longitudinal following, lateral commitment, traffic-element detection, and reasoning under adverse conditions.
- Qualitative Case Studies: Fast-dDrive’s predictions match the ground-truth trajectory direction in every qualitative case.The examples are drawn from five different Waymo end-to-end driving scenes.
C. Limitations
Fast-dDrive’s limitations concern its fixed JSON-schema scaffold, the extra compute required for shared-prefix scaling, and evaluation restricted to open-loop benchmarks. Future work should validate reactive behavior in closed-loop simulations.
- Scaffold Construction: The fixed JSON schema may require manual template adjustment when task definitions change fundamentally, such as object counts or reasoning granularity.The schema covers most current end-to-end driving tasks.
- Inference Scaling: Shared-prefix inference scaling improves accuracy but adds fractional compute that may remain prohibitive in extremely low-latency edge environments.Even a single additional forward pass can be constrained in such settings.
- Evaluation: Current evaluation focuses on open-loop benchmarks, motivating future closed-loop simulations to further validate the model’s reactive behavior.Open-loop benchmarks assess planning quality against human experts.