DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

• Existing closed-loop platforms lack both controllable traffic scenarios and realistic observations that preserve policy-relevant scene features.Physics-based simulators support controllability but retain a sim-to-real appearance gap; generative and reconstruction-based simulators improve realism but often sacrifice controllability.Source: §1 · p. 1
• Its video model uses cached early-timestep latents for long-horizon consistency, traffic-layout guidance for state alignment, and diverse-scene distillation to broaden visual appearance.These designs target hundreds of consecutive rollout frames while preserving policy-relevant cues such as lane geometry and traffic-element state.Source: §1 · p. 2
• Under FDπ, DreamStream outperforms the strongest prior closed-loop simulator by 1.6× on nuScenes and 4.7× on NAVSIM.FDπ computes Fréchet distance over E2E-policy scene-context features for paired real and simulated scenes; it correlates more strongly with driving performance than FID and FVD.Source: §1 · pp. 2–3
• Navhard-CL converts static NAVSIM into reactive closed-loop testing with a log-replay base set and adversarial behavior and weather variations.Navhard-AdvBehavior adds adversarial agents, while Navhard-AdvWeather renders the same scenarios under varied weather, lighting, and road-surface conditions.Source: §1 · pp. 2–3
• Navhard-CL exposes scorer miscalibration, proposal-coverage failure, and vision-encoder brittleness under appearance shifts in current E2E policies.The benchmark is intended to complement existing benchmarks by diagnosing observation and behavior gaps under closed-loop deployment.Source: §1 · pp. 2–3
• DreamStream couples a physics simulator that rolls out interactive scenarios with an autoregressive video model that translates symbolic simulator state into visual observations.This design pairs visual realism for ego-agent observations with background traffic logged using real-world dynamics.Source: Figure 1 · p. 2, §1 · p. 3

Autoregressive Video Generation for Autonomous Driving

• Prior driving-video work predicts frames with generative world models [20] [21] [22] [23] [25] [26], synthesizes views from dynamic 3D scenes [27] [8] [9] [10], or re-renders simulator state with pretrained video models [28] [16] [12] [11].These pipelines are typically trained on small datasets such as nuScenes and inherit narrow visual styles, limiting weather, lighting, and road-condition diversity [29] [11] [12].Source: §2 · p. 3
• DreamStream uses diverse-scene distillation from a pretrained video foundation model in a multi-stage autoregressive pipeline to enable weather and lighting variations [30].Source: §2 · p. 3

End-to-End Driving Policies Evaluation

• Open-loop evaluation benchmarks policies efficiently on fixed observations [29] [31] [18] [19] [32] [33], but strong open-loop performance may not predict reliable closed-loop behavior [6] [3] [37].DreamStream provides generative observations grounded in closed-loop dynamics to support comprehensive E2E policy evaluation [38] [39] [18] [41].Source: §2 · p. 3

3.1 Overview

• DreamStream couples simulator state, autoregressive video generation, and an evaluated E2E policy, executing only Δr planned steps before regenerating layout-conditioned camera frames and repeating.Surrounding agents can use log-replay, IDM, or adversarial control, and the executed trajectory is scored with closed-loop metrics.Source: §3.1 · pp. 3–4
• Scenarios start from an initial world state and camera frame, supplied by real-world driving logs [42] or generated scenarios such as safety-critical variants [43].Source: §3.1 · pp. 3–4

3.2 Autoregressive Video Diffusion Model Distillation

• A three-stage distillation pipeline first learns conditional video generation, then few-step autoregressive prediction, and finally long-horizon rollout stability.Stage 1 trains from a pretrained video model using traffic-layout conditions and an initial-frame anchor; Stage 2 distills with ground-truth context, while Stage 3 trains on self-generated context to address the train-test gap [44] [45] [46].Source: §3.2 · p. 4
• Traffic-guided distillation uses classifier-free guidance [47] to steer generation toward traffic-layout conditions, improving layout alignment without additional data.During Stage 1, layout is removed with probability 0.1 for the unconditional condition; conditional and unconditional predictions are combined during generation.Source: §3.2 · p. 5
• Synthetic clips built from existing driving scenarios provide visually diverse distillation data, helping the autoregressive student retain adverse weather and lighting absent from limited driving logs.Stages 2 and 3 train on these clips to support rollouts outside the narrow visual domain of the driving data.Source: §3.2 · p. 5

4.1 Metric Design

• FDπ measures the Fréchet distance between policy-extracted scene-context features from real and generated frames, aggregating normalized distances across public E2E policies.Lower FDπ indicates better preservation of policy-relevant visual information, and zero means perfect alignment; the metric correlates more strongly with driving performance than FID or FVD.Source: §4.1 · pp. 5–6

4.2 Results

• FDπ is 7.27 on nuScenes val, 1.6× lower than the strongest baseline, and DreamStream consistently outperforms prior baselines on NAVSIM navtest.DreamStream leads in three of five per-policy columns; HUGSIM performs on par for RAP, whose spatial cross-attention relies on 3D geometry. Comparisons cover nuScenes val and NAVSIM navtest against MagicDrive, Panacea, Dreamland [16], DriveArena [11], DreamForge [12], and HUGSIM [8], using five E2E policy architectures [39] [41] [52] [48] [38] [49].Source: Table 1 · p. 6, §4.2 · pp. 6–7

4.3 Metric Ablation

• Held-out FDπ correlates with PDMS at r = −0.65, versus FID at r = +0.11 and FVD at r = +0.08, when each policy's features are excluded.The held-out setup correlates FD computed without a given policy against that policy's PDMS, indicating FDπ captures policy-independent scene-information corruption.Source: §4.3 · p. 7
• FDπ increased 20% more on 540 challenging Navhard scenes than on a random subset and responded 2.3× more strongly than FID to local corruptions in 1,517 nuScenes samples.The injected corruptions included 2m agent displacement and lane removal.Source: §4.3 · p. 7

5 Navhard-CL Closed-loop Benchmark

• Navhard-CL comprises Navhard-Base real-world logs, Navhard-AdvBehavior adversarial agents, and Navhard-AdvWeather adverse and scarce weather, lighting, and road-surface conditions.The benchmark is built on DreamStream and based on NAVSIM navhard [19].Source: §5 · p. 7

5.1 E2E Policies Performance

• DreamStream-rendered observations yield higher closed-loop scores across evaluated policies, while photorealistic DriveArena scores below game-engine BridgeSim.The comparison reports driving score (DS), extended PDM score (EPDMS), and route completion (RC).Source: §5.1 · p. 7, Table 2 · p. 8

5.2 Closed-loop Gap Analysis

• Under oracle scoring, scorer bias is already large on Navhard-Base and widens for some architectures on AdvBehavior, while proposal coverage fails and LTF localizes the gap to trajectory decoding [54].From Navhard-Base to Navhard-AdvBehavior, driving score decreases dramatically; lower TTC, LK, and HC subscores indicate inability to react safely.Source: §5.2 · p. 8, Table 2 · p. 8
• For high-driving-score (≥80) Navhard-AdvWeather scenarios, rain and snow degrade DrivoR more than night, especially in drivable-area compliance and lane keeping.This visual robustness gap is not exposed by closed-loop benchmarks with a narrow, distribution-matched visual domain.Source: §5.2 · p. 8

Traffic-layout guidance

• FDπ improves from 12.18 to 9.50 on nuScenes val with traffic-guided distillation.The guidance strengthens traffic-layout signals, aligning generated frames with simulator states and supporting more accurate E2E policy decisions.Source: §6 · p. 8

Diverse-scene distillation

• Diverse-scene distillation consistently improves CLIP similarity across weather and lighting conditions.Without it, the model quickly reverts to the training distribution’s narrow appearance instead of preserving weather and lighting from the first frame.Source: §6 · pp. 8–9, Table 4 · p. 9, Figure 6 · p. 9

Long-horizon rollout stability

• DreamStream’s FDπ drift remains below all compared baselines on nuScenes and NAVSIM, while early-timestep KV caching mitigates drift by 68% and 48%, respectively.Drift is measured as FDπ over rollout length.Source: §6 · p. 9, Figure 7 · p. 9

Efficiency

• DreamStream enables real-time generation, batch benchmarking, and linear multi-GPU scaling.Full-model training costs approximately 200 A100 GPU-days; runtime is reported for a single RTX PRO 6000 GPU.Source: §6 · p. 9, Table 5 · p. 9

7 Limitations

• The evaluation does not yet include on-road or hardware-in-the-loop deployment, and the video model accumulates drift over extensive long-horizon rollouts.The authors cite cost and safety as reasons for the evaluation scope and leave reducing long-horizon drift to future work.Source: §7 · p. 9

8 Conclusion

• DreamStream improves over the best prior closed-loop simulators by 1.6× on nuScenes and 4.7× on NAVSIM under FDπ.FDπ is a policy-oriented Fréchet distance for measuring visual alignment.Source: §8 · p. 9
• DreamStream pairs a physics simulator with an autoregressive video model, and Navhard-CL adds behavior and visual variations to create a closed-loop benchmark.The benchmark is built on DreamStream.Source: §8 · p. 9
• Navhard-CL exposes performance gaps and failure modes missed by prior closed-loop benchmarks, pointing to scorer calibration and proposal coverage as directions for stronger policies.Source: §8 · p. 9

A More Related Work

• Game-engine simulators and their closed-loop benchmarks provide layout control and reactive adversarial behavior, but hand-built 3D assets can create a sim-to-real visual gap that corrupts policy perception [55] [56] [60] [4] [5] [6] [7] [3].Source: A More Related Work · p. 15
• Reconstruction-based renderers produce photo-realistic rollouts but are limited to captured scenarios with fixed appearance and can artifact under off-trajectory views or dynamic-object insertion [27] [8] [62] [19] [9] [63] [10].Source: A More Related Work · p. 15
• Generative re-rendering offers photo-realistic, controllable rollouts, but small in-domain training datasets can yield narrow visual styles and weak 3D grounding [11] [12] [28] [16] [26] [22] [21] [25].Source: A More Related Work · p. 15
• DreamStream distills a video model from a large pretrained model to render diverse, photo-realistic observations while retaining grounded, controllable, reactive simulation [Tab. 6].It uniquely combines photorealism and visual diversity with grounded, reactive closed-loop simulation.Source: A More Related Work · p. 15, Table 6 · p. 15

B Methodology Details

• The appendix separates B.1’s three-stage distillation objectives from the training hyperparameters and implementation settings reported in B.2.Source: B Methodology Details · p. 15

B.1 Autoregressive Video Model Distillation Details

• Stage 1 trains a bidirectional latent-video teacher with flow matching to follow scene layout and generate coherent, continuous frames conditioned on layout and the initial camera frame.The condition combines scene layout with the initial frame’s CLIP embedding.Source: §B.1 · p. 15
• Stage 2 distills the teacher into a few-step causal student for chunk-level autoregressive prediction, using bidirectional attention within chunks and causal attention across chunks.Source: §B.1 · pp. 15–16
• For Stage 2, the student is trained by ODE regression against teacher reverse-time PF-ODE trajectories while retaining diffusion-forcing context noise.Source: §B.1 · p. 16
• Stage 3 uses Self Forcing [44] to close the ground-truth-context train-test mismatch by training with cached, self-generated history and a holistic rollout distribution-matching objective.The objective uses a frozen real-score network initialized from the teacher and a trainable critic on student rollouts.Source: §B.1 · p. 16
• Conditioning on early-timestep latents in the KV cache mitigates rapid quality drift from compounding high-frequency errors and empirically stabilizes rollout.Source: §B.1 · p. 16

B.2 Implementation Details

• Training uses Wan 2.1 (1.3B) at 832 × 480 and Wan 2.2 (5B) at 1280 × 704, initialized from Wan-Fun-Control [30] with shared settings otherwise.Stages 1–3 run for 10K, 20K, and 1K steps respectively; traffic-guided distillation uses guidance scale 7.5 to sample ODE trajectories.Source: §B.2 · p. 16

B.3 Synthetic Clip Construction for Diverse-Scene Distillation

• For diverse-scene distillation, 500 nuPlan-based layouts remain unchanged while Nano-Banana re-renders each initial frame along lighting, weather, or road-surface axes.The Stage-1 teacher rolls out videos from the edited frame and original layout; manual review retains 500 synthetic scenes for Stage-2 and Stage-3 training.Source: §B.3 · pp. 16–17

C.1 Training Dataset

• DreamStream training uses more than 20,000 nuPlan [31] videos spanning around 90 hours, with HD-map and box annotations projected into perspective and factual prompts generated by Qwen3-VL [65].Each video spans 15–20 seconds; layout conditions include map elements and traffic agents, while prompts emphasize scene objects, geometry, motion, and context.Source: §C.1 · p. 17

C.2 Metric Details

• Open-loop PDMS [18] and EPDMS [19] score planned trajectories against logged futures, while closed-loop Driving Score multiplies route completion by mean per-frame EPDMS.PDMS uses NC, DAC, and DDC constraints with EP, TTC, and comfort features; EPDMS adds TLC, LK, HC, and EC, and Driving Score uses route completion relative to the expert path or goal.Source: §C.2 · pp. 17–18

C.3 Closed-loop Simulation Setup

• Closed-loop evaluations use BridgeSim [3] for 8 seconds, replanning every 5 frames from one recent rendered frame and executing the first 5 planned steps.Surrounding agents start at ground-truth poses and follow IDM behavior, reacting to the ego vehicle while adhering to traffic rules.Source: §C.3 · p. 18

C.4 Navhard-CL Scenario Curation

• Navhard-CL combines a real-world log-replay base set with adversarial-behavior and appearance-shift variants to diagnose observation and behavior gaps in closed-loop E2E policy deployment.Navhard-Base comprises 421 NAVSIM navhard scenarios ported to MetaDrive; it serves as a distribution-matched reference.Source: §C.4 · p. 18
• Navhard-AdvBehavior retains 325 safety-critical adversarial variants across five pre-crash categories, pairing each with its base scenario on the same ego route and map.Adv-BMT introduces an agent that maneuvers to provoke a collision; the categories are rear-end, straight crossing paths, opposite direction, changing lanes, and left turn across path.Source: §C.4 · p. 18
• Navhard-AdvWeather produces more than 5k scenario-appearance combinations by varying lighting, weather, and road-surface conditions in rendered base scenarios.The axes include six lighting conditions, four weather conditions, and three road-surface conditions.Source: §C.4 · p. 18

C.5 Closed-loop Gap Analysis

• The Navhard-AdvBehavior analysis separates scorer bias from proposal coverage by comparing each learned scorer with an oracle that chooses the proposal with the highest simulator-ground-truth EPDMS.Scorer bias measures the score decrease from mis-ranking generated proposals; coverage measures proposal degradation when scoring is optimal.Source: §C.5 · pp. 18–19, Table 7 · p. 19
• Scorer bias varies across E2E policies: DrivoR has the largest gap, DiffusionDriveV2’s is effectively zero, and DrivoR lacks a positive-EPDMS proposal on 14.1% of adversarial cases versus 7.9% of base cases.Under safety-critical distributions, DiffusionDrive and DiffusionDriveV2 scorer bias doubles, while DrivoR’s narrows as its Oracle performance collapses.Source: §C.5 · p. 19, Table 7 · p. 19
• Navhard-CL extends open-loop scorer-mismatch analysis [54] into reactive closed-loop rollouts, where scorer mis-ranking compounds performance costs.The analysis exposes scorer bias that open-loop evaluation on logged trajectories cannot reveal.Source: §C.5 · pp. 19–20
• Oracle driving scores fall on Navhard-AdvBehavior, exposing proposal-coverage failures that remain even when scoring is optimal.For DrivoR, the best proposal’s average EPDMS falls from 70.6 on Navhard-Base to 66.7 on Navhard-AdvBehavior; LTF shows a similar-scale gap, locating the failure in trajectory decoding.Source: Table 7 · p. 19, §C.5 · p. 20
• Fog, snow, sand, and puddles degrade DrivoR’s performance more than lighting or common weather, mainly through route completion and drivable-area compliance rather than safety submetrics.NC and TTC remain stable across conditions, while lighting and common weather have little impact because they occur in training.Source: Table 8 · p. 19, §C.5 · p. 20

D More Visualization

• Additional DreamStream visualizations show qualitative examples of closed-loop policy evaluation and its visual realism and alignment.The examples appear in Figure 9.Source: D More Visualization · p. 20, Figure 9 · p. 20