Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Driving

Motivation

• AR driving VLAs decode reasoning and trajectory tokens sequentially, allowing early waypoint errors to compound into physically implausible 5 s maneuvers.Each waypoint conditions on previously emitted coordinates, which may be noisy.Source: §1 · p. 1

Method

• Fast-dDrive freezes schema-determined JSON keys and syntax, denoising only value tokens to focus capacity on positional content.The scaffold covers deterministic structural tokens in structured outputs that bundle perception, chain-of-thought, and trajectory.Source: §1 · p. 2
• Section-Aware Structured Diffusion aligns block boundaries with semantic sections, guarantees 100% structural validity, and prioritizes safety-critical sections without inference overhead.It combines section-weighted cross-entropy with a section-adaptive Beta noise schedule.Source: §1 · p. 2

Inference

• Scaffold Speculative Decoding auto-accepts scaffold tokens and verifies parallel masked-diffusion drafts, producing outputs identical to pure AR at lower latency.The AR head verifies the parallel MDM draft while deterministic scaffold tokens are accepted automatically.Source: §1 · p. 3

Results

• 12× throughput is achieved by combining Fast-dDrive with SGLang, while shared-prefix scaling forks N trajectory rollouts from one KV cache and averages them for accuracy gains.Only the trajectory section is sampled at non-zero temperature, trading fractional extra inference compute for improved accuracy.Source: §1 · p. 3

Conclusion

• Block-diffusion VLAs with structure-aware training and inference can match or exceed strong AR and full-sequence-diffusion accuracy at substantially higher throughput.The approach retains interpretable structured chain-of-thought outputs.Source: §1 · p. 3

Vision-Language-Action Models for Autonomous Driving

• Autoregressive VLAs unify perception, reasoning, and planning, improve trajectory prediction in long-tail scenarios, and remain sequential and memory-bandwidth-bound at batch size 1.Recent work also incorporates chain-of-thought reasoning.Source: §2 · p. 3

Diffusion Large Language Models

• Discrete diffusion for text has advanced from foundational formulations and masked diffusion objectives to large-scale models matching autoregressive performance.Post-training methods further align diffusion language models with human preferences.Source: §2 · p. 3

Efficient Decoding and Test-Time Scaling

• Speculative decoding accelerates autoregressive generation by drafting multiple tokens for parallel verification, while self-speculative variants reuse one model for drafting and verification.Fast-dLLM extends speculative decoding to block diffusion.Source: §2 · p. 3
• Shared-prefix rollouts amortize deterministic prefix computation and apply stochasticity only to the trajectory section at a fractional per-rollout cost.The deterministic structure of the first three sections enables this computation sharing.Source: §2 · pp. 3–4

Block-Causal Diffusion

• Fast-dDrive partitions outputs into left-to-right blocks with bidirectional attention within each block and causal attention across blocks, enabling KV-cache reuse.Each block attends to the prompt and preceding blocks, but not future blocks.Source: §3.1 · p. 4

Scaffold Construction and Section-Aligned Blocks

• The model freezes JSON structural tokens into a scaffold and denoises only value tokens, guaranteeing 100% structural correctness while reducing denoising workload by ∼30%.Section-aligned blocks follow the causal order CO → Expl → FMB → Traj, with NULL padding for variable-length sections.Source: Table 1 · p. 4, §3.2 · p. 5
• The structured output contains critical_objects, explanation, future_meta_behavior, and trajectory sections whose token counts, difficulty, and safety impacts differ substantially.The trajectory section contains five waypoint coordinates over five seconds, while critical_objects contains 12 binary detections.Source: §3.2 · pp. 4–5

Safety-Prioritized Training

• Safety-prioritized training uses section-weighted cross-entropy and section-specific Beta noise schedules, with both mechanisms adding zero inference overhead.Larger section weights emphasize hard, high-impact tokens, while Beta schedules adapt noise to each section’s difficulty profile.Source: §3.2 · p. 5

Scaffold Speculative Decoding

• Scaffold Speculative Decoding auto-accepts scaffold tokens, drafts value tokens bidirectionally, and verifies them causally, requiring exactly 2 forward passes per block.Section-aligned blocks provide complete semantic context for drafting and improve acceptance compared with arbitrary fixed-size blocks.Source: §3.3 · pp. 6–7, Figure 3 · p. 6

Test-Time Inference Scaling

• Test-time scaling averages N stochastic trajectory rollouts forked from a shared deterministic prefix, and mean ADE@5s decreases monotonically with N.Each rollout is interpolated to 20 waypoints using Jerk-Minimizing Trajectory fitting before equal-weight averaging.Source: §3.4 · p. 7

Experimental Setup

• Fast-dDrive uses RGB cameras, ego state, and a high-level navigation command without LiDAR, radar, or HD maps.It uses three past front-camera frames on nuScenes and three front-facing cameras on WOD-E2E.Source: §4.1 · p. 8

WOD-E2E Results

• On WOD-E2E, Scaffold Spec attains the lowest ADE@3s and ADE@5s, with RFS above dVLM-AD and competitive with the strongest AR baseline.These results are achieved in a single inference run without GRPO post-training or a larger trajectory pool.Source: §4.2 · p. 8
• 4×–6× decoding throughput over dVLM-AD and AR baselines is achieved while committing approximately 5 tokens per model forward pass.Shared-prefix multi-trajectory rollout further reduces both ADE values at sub-2× the wall-clock cost of one Scaffold-Spec pass.Source: §4.2 · pp. 8–9

nuScenes Results

• On nuScenes, Fast-dDrive has the lowest average L2 among listed reasoning VLM/VLA systems, consistently improving over diffusion and AR-with-CoT baselines across three horizons.It also matches or improves upon classical training-based driving policies without interpretable reasoning.Source: §4.1 · p. 8, §4.2 · p. 9

Inference Efficiency

• Scaffold Spec achieves roughly 6× dVLM-AD throughput by combining block-level KV-cache reuse with scaffold-aware speculative decoding.Scaffold auto-acceptance removes approximately 30% of tokens from draft verification while matching vanilla self-speculative accuracy.Source: §4.3 · p. 9

Ablations and Test-Time Scaling

• IWL is the primary SASD contributor to RFS, while combining IWL and SNS produces the best RFS because their effects are complementary.ADE@5s decreases monotonically with trajectory rollout count N; N=4 is adopted as the default accuracy–latency trade-off.Source: §4.4 · pp. 9–10

• Fast-dDrive exploits the inherent structure of driving outputs to advance planning accuracy and inference efficiency simultaneously.Source: §5 · p. 10
• Deterministic schema tokens form a frozen scaffold, while diffusion blocks align with semantic sections and training prioritizes safety-critical tokens.Source: §5 · p. 10
• 6× throughput accompanies state-of-the-art trajectory accuracy versus full-sequence diffusion baselines, showing structured generation and efficient decoding are complementary.Source: §5 · p. 10

nuScenes inputs

• nuScenes uses CAM_FRONT only, with three frames at t∈{−1.0, −0.5, 0} s presented chronologically relative to prediction time.Side and rear cameras are excluded to match the dVLM-AD evaluation protocol and preserve fair comparison.Source: A. Input Modalities: Full Details · p. 11

WOD-E2E inputs

• WOD-E2E uses FRONT_LEFT, FRONT, and FRONT_RIGHT at t=0 s instead of all eight surround views.The three forward views were sufficient for open-loop planning while keeping the visual token budget tractable for the 3B-parameter backbone.Source: A. Input Modalities: Full Details · p. 11
• The WOD-E2E joint-view variant concatenates FRONT_LEFT|FRONT|FRONT_RIGHT horizontally, roughly halving visual tokens while trading per-view resolution for a wider panorama.This variant is resized as a single panoramic image before encoding.Source: A. Input Modalities: Full Details · p. 11

Preprocessing and conditioning

• Images preserve aspect ratio with a maximum 512-pixel longer side; inference uses no cropping or photometric augmentation, while inputs also include ego state and navigation commands.The model excludes LiDAR, radar, HD maps, and other auxiliary sensor modalities.Source: A. Input Modalities: Full Details · p. 11

Qualitative Case Studies

• Five examples cover nighttime left turns, lane-following behind a pickup, right turns, green-light cruising, and wet-weather left turns.The scenes span deliberate steering, longitudinal following, lateral commitment, traffic-element detection, and reasoning under adverse conditions.Source: B. Qualitative Case Studies · p. 11
• Fast-dDrive’s predictions match the ground-truth trajectory direction in every qualitative case.The examples are drawn from five different Waymo end-to-end driving scenes.Source: B. Qualitative Case Studies · p. 11

Scaffold Construction

• The fixed JSON schema may require manual template adjustment when task definitions change fundamentally, such as object counts or reasoning granularity.The schema covers most current end-to-end driving tasks.Source: C. Limitations · p. 12

Inference Scaling

• Shared-prefix inference scaling improves accuracy but adds fractional compute that may remain prohibitive in extremely low-latency edge environments.Even a single additional forward pass can be constrained in such settings.Source: C. Limitations · p. 12

Evaluation

• Current evaluation focuses on open-loop benchmarks, motivating future closed-loop simulations to further validate the model’s reactive behavior.Open-loop benchmarks assess planning quality against human experts.Source: C. Limitations · p. 12