NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

• Closed-loop evaluation tests policies by updating the simulator from each policy action and returning sensor observations rendered from the updated state.This interaction matters because policy actions influence how a scenario unfolds over time.Source: §1 · p. 4
• Reconstruction-based neural simulators enable photorealistic what-if testing in captured scenes but struggle to extend beyond captured corridors or generate consistent observations in substantially new conditions.Source: §1 · p. 4
• Generative world models use visual priors to synthesize extreme weather, unusual deformable objects, and pedestrian or articulated motion that changes with vehicle proximity (Ren et al., 2025).Source: §1 · p. 4
• OmniDreams is an action-conditioned generative world model mid- and post-trained on Cosmos (NVIDIA, 2025), generating photorealistic sensor observations interactively and responsively to driving policies.Source: §1 · p. 5
• Its autoregressive diffusion-video simulator conditions sensor generation on simulator state and driving actions, while attending to past-generation KV caches for long-rollout consistency.The closed-loop workflow uses Alpamayo 1 (NVIDIA, 2026) as policy and AlpaSim (NVIDIA, 2025) as orchestrator.Source: §1 · p. 5
• At 720p, the 2B single-camera version renders 68 FPS on one GB300, while the 4-camera version reaches up to 105 FPS on 16 GB300 GPUs.Source: §1 · p. 5
• On PAI, the OmniDreams-post-trained WAM reduces collision 6.9%→4.2%, collision_front 1.0%→0.9%, collision_lateral 0.6%→0.4%, and collision_rear 5.3%→3.0%.These preliminary results outperform VLA-based Alpamayo 1.5 despite using ∼2 B vs. ∼10 B parameters.Source: §1 · p. 5

2.1. Data Sources

• RDS provides 3M clips for mid-training, and RDS-HQ-1M provides 1.14M clips for finetuning and post-training, drawing from driving logs across 15 countries.Both datasets contain 1080p video from seven synchronized cameras at 30 fps; RDS clips are 20 s, while RDS-HQ-1M mixes 10 s and 20 s clips. RDS is used in (Agarwal et al., 2025), and RDS-HQ-1M expands RDS-HQ in (Ren et al., 2025).Source: §2.1 · p. 5

2.2. Conditioning Signal Extraction

• Conditioning pairs a rendered world-scenario map encoding scene geometry, actors, and driver actions with environmental text prompts and recent visual history.Maps combine route-matched HD annotations with tracked 3D actors, rendered as pixel-aligned camera overlays following (NVIDIA, 2025); Qwen2.5-VL-7B (Bai et al., 2023) generates captions for each camera and 10-s window, with short, medium, and long prompts sampled at 0.1/0.2/0.7.Source: §2.2 · p. 6

2.3. Data Curation and Quality Filtering

• The quality pipeline removes sequences with unreliable sensors or annotations, prediction disagreements, visual artifacts, and repetitive clips identified through trajectory and visual features.A VLM detects visual problems such as chromatic aberration, and deduplication downsamples repeated patterns such as straight highway driving.Source: §2.3 · pp. 6–7

2.4. Dataset Composition

• 16,600 h across 3M clips supply mid-training, while 4,944 h across 1,142,285 clips support finetuning and post-training; 5,000 balanced clips are held out, including 300 60-s clips.The held-out set covers rare road users, large vehicles, adverse weather and lighting, and unusual infrastructure or accident scenes; 504,488 training clips are 10 s and 637,797 are 20 s.Source: §2.4 · p. 7

2.5. Curation and Inspection with SIL-Wheel

• SIL-Wheel (NVIDIA, 2026) supports clip search and curation, enabling targeted post-training slices, scenario-based evaluation buckets, and visual inspection of labels and filtered clips.Its composable search uses captions, semantic and visual retrieval, ego trajectories, classifier scores, and perception filters; selected underrepresented conditions include rare weather, construction, vulnerable road users, and complex interactions.Source: §2.5 · p. 7

3. Model Architecture

• At each step, OmniDreams updates the world from the latest action and autoregressively generates a short sequence of future frames.This makes generation responsive to changes in driving-policy actions or human input.Source: §3 · p. 8
• OmniDreams-SV generates 8 frames per step from one front-facing view, while OmniDreams-MV generates 16 frames per step across four synchronized views.The multi-view variant covers front, cross-left, cross-right, and front-telescope cameras.Source: §3 · p. 8

3.1. Network Architecture

• The network combines first-frame RGB, text prompts, structured world scenarios, and cached tokens in a causal transformer based on Cosmos-Predict 2.5 (NVIDIA, 2025).World scenarios include static maps and dynamic-agent bounding boxes; policy or human actions render the world-scenario conditioning video.Source: §3.1 · pp. 8–9
• A lightweight MLP encodes simulator state into compact control tokens, which are concatenated with visual tokens to condition generation with minimal computational overhead.Separating control signals from visual content also stabilizes autoregressive generation while preserving real-time throughput.Source: §3.1 · p. 9

3.2. Consistent Multi-View Generation

• Factorized temporal and cross-view attention supports consistent generation across up to 7 synchronized views.Temporal attention operates within each view using a causal KV cache, while cross-view attention connects views at each time step to align shared scene elements.Source: §3.2 · p. 9

4. Training

• OmniDreams mid-trains Cosmos-Predict 2.5 on RDS, adds multi-view adaptation for OmniDreams-MV, then adds autoregressive generation, world-scenario control, and Self Forcing DMD distillation.The resulting models support minutes-long rollouts without significant quality loss, while distillation enables few-step generation and removes autoregressive error accumulation.Source: §4 · p. 10

4.1. World-Scenario Control and Multi-view adaptation

• OmniDreams-MV learns view embeddings and cross-view attention across four camera views, then trains world-scenario control on 93-frame clips before extending to 189 frames.A 1:1 text-to-video and image-to-video mix supports first-frame conditioning while preserving novel-content generation; controlled bidirectional models serve as distillation teachers.Source: §4.1 · p. 11

4.2. Mid-training for Autoregressive Generation

• Diffusion Forcing training converts bidirectional models to causal generation, first without world-scenario control on RDS, then with a control branch on RDS-HQ-1M.The additional optimization addresses the limited scale of world-scenario-labeled data.Source: §4.2 · p. 11

Causal Masking

• Causal masking limits each frame’s attention to preceding frames, enabling autoregressive or block-autoregressive generation initialized from a bidirectional video model.The implementation uses Flex-Attention (Dong et al., 2024).Source: Causal Masking · pp. 11–12

4.3. Distillation

• Self Forcing (Huang et al., 2025) trains autoregressive self-rollouts to close teacher-forcing exposure bias, while video-level DMD (Yin et al., 2024) matches generated clips to real data without paired or pixel-wise supervision.Training uses a 2-step diffusion process and a rolling KV cache; restricting attention to the matching rolling window reduces inference complexity from O(TL^2) to O(TL).Source: §4.3 · p. 12
• A longer-context bidirectional teacher reduces shifting artifacts in long autoregressive rollouts, and distillation uses a challenging subset of 58k videos spanning complex urban, highway, and rural conditions.The progressive teacher strategy continues fine-tuning the model previously distilled with a short-context teacher.Source: §4.3 · pp. 12–13

5.1. Model Optimization

• The model uses local-window attention, a static-shape streaming KV cache, compiled CUDA Graphs, lightweight encoders and decoders, and hoisted step-invariant operators.Attention windows are 6 latent frames for single-view and 8 for multi-view; LightVAE encodes single-view conditioning, pixel shuffle encodes multi-view conditioning, and LightTAE decodes both.Source: §5.1 · p. 13

5.3. End-to-End Performance

• 105 effective FPS per camera for four-view 16-frame chunks and 68 effective FPS for single-view 8-frame chunks; single-view exceeds the 30 FPS real-time threshold.The four-view result uses 16 GB300 GPUs; the single-view result uses one GB300.Source: §5 · p. 13, Table 3 · p. 14, Table 2 · p. 14

5.2. Multi-GPU Inference

• Four-view inference shards across camera and temporal axes as V=4 and T=4, saturating 16 GPUs while leaving spatial sharding for larger configurations.The hierarchy prioritizes camera then temporal sharding to preserve attention structure, using ring attention to overlap KV transfer with local computation.Source: §5.2 · p. 14

5.4. FlashDreams: Generalized Inference and Serving Infra

• 1.95× speed-up over the official implementation on one GB200 for Self Forcing (Huang et al., 2025), and 2.49× on 4 H100s for Lingbot-World (Team et al., 2026).FlashDreams (NVIDIA, 2026) packages the streaming optimizations and a gRPC/webRTC client-server stack for streaming control inputs and RGB frames.Source: §5.4 · p. 15

6. Closed-Loop Simulation Integration

• OmniDreams runs as a stateful gRPC video server, reusing its KV cache, captured CUDA Graphs, and renderer state across chunks while the client iteratively sends trajectories and receives generated frames.Only the first request includes an initial frame; later requests carry the new trajectory and prompt.Source: §6 · p. 15

6.1. AlpaSim

• AlpaSim’s modular, containerized microservice architecture connects renderers, policies, and physics through gRPC; OmniDreams replaces NuRec and requires new rendering-state protocols and chunk-aware runtime logic.Asynchronous RPCs allow concurrent rollouts to be serialized across services, which must retain state across potentially out-of-order requests.Source: §6.1 · p. 16

6.2. Networked integration in AlpaSim

• A rank-0 gRPC server forwards render requests to other video-model ranks via NCCL events, then gathers and returns frames, avoiding unified dependencies and rank-dispatch logic across OmniDreams and AlpaSim.The prototype serializes frames; future work proposes RDMA-based gRPC or using gRPC only for coordination while transferring frames over NCCL.Source: §6.2 · p. 16

6.3. Session-based state

• Each rollout begins a session with a seed frame and world map; the server associates these inputs with a pre-allocated KV cache and returns an ID used to identify later requests’ rollout state.The session abstraction represents the frame-history conditioning that guides subsequent video generation.Source: §6.3 · p. 17

6.4. Supporting chunked generation

• Chunked video generation requires ego and actor poses for every frame in a chunk, so policy and traffic trajectories must remain fixed during that chunk to avoid invalidating its generated video.The simulator considers post-fetch and pre-fetch strategies; future models aim to reduce chunk size toward frame-at-a-time generation.Source: §6.4 · p. 17

6.4.1. Post-fetch generation

• Post-fetch generation rolls out policy and traffic across a chunk without new visual input, then requests video from the resulting trajectories, inserting the returned frames earlier in the simulation timeline.This strategy requires policies that can act when the latest image lags behind the latest ego position, and its frame insertion makes the timeline out of order.Source: §6.4.1 · p. 17

6.4.2. Pre-fetch generation

• AlpaSim uses pre-fetch generation: policy and traffic produce multi-step trajectories at chunk boundaries, which are interpolated for video frames injected at their capture timestamps before the simulation timer advances.This preserves event ordering and also applies trajectory-level operation to physics and controller modules.Source: §6.4.2 · p. 17

7. Post-training OmniDreams as a World-Action Model (WAM)

• The 2 B-parameter OmniDreams-SV 120° front-wide checkpoint is fine-tuned from its causal pre-map checkpoint to predict 64 ego waypoints over 6.4 seconds at 10 Hz.Training jointly sums video and trajectory flow-matching losses with independently sampled times; inference uses one causal backbone pass and four trajectory flow-matching steps, without video denoising.Source: §7 · pp. 17–18, §7.1 · p. 18

7.1. Architecture and training

• The WAM retains OmniDreams’ causal DiT and adds projected DINOv2 and 30° front-telescope features plus an interleaved ego-motion history token to condition trajectory prediction.The history token embeds the previous 1.6 seconds of ego motion; its DiT output feeds a 12-layer U-Net-shaped MLP that predicts the future trajectory latent.Source: §7.1 · p. 18

7.2. Closed-loop comparison to Alpamayo 1.5

• Collision fell from 6.9% with Alpamayo 1.5 to 4.2% with OmniDreams WAM, which uses ∼2 B versus ∼10 B parameters.Under Alpamayo 1.5’s 10 Hz, 20-second protocol on training-excluded NuRec scenes, Front/Lateral/Rear collisions were 0.9%/0.4%/3.0% versus 1.0%/0.6%/5.3%. The authors propose joint world-model–policy training as a future direction for driving.Source: §7.2 · p. 18

8. Post-Training OmniDreams as a Diffusion Fixer

• Reconstruction-based simulators can produce blur, missing content, ghosting, floating geometry, and other view-dependent artifacts when requested camera poses depart substantially from capture trajectories.These artifacts can reduce closed-loop rollout realism and introduce undesirable distribution shifts for downstream policies.Source: §8 · pp. 18–19
• OmniDreams-corrected outputs remove blur, ghosting, missing regions, and spurious geometry while preserving scene layout, camera viewpoint, and driving-relevant structures.At inference, the model operates autoregressively on rendered frames and maintains temporal consistency through causal-history and KV-cache conditioning.Source: Figure 7 · p. 19, §8 · p. 19
• Post-training pairs degraded simulator renderings with clean target images and starts denoising from the degraded rendering to map artifacts onto the clean-image manifold.The learned correction preserves scene layout, camera viewpoint, and driving-relevant structure.Source: §8 · p. 19

9.1. Simulation Quality

• OmniDreams generates decision-relevant dynamics and preserves traffic and roadway cues; its final distilled model leads the reported training-stage comparison on FVD and conditioning-fidelity metrics.The reported measures include 3D detection, lane-line regression and detection, and Temporal Sampson consistency. Four-camera generation remains real-time on NVIDIA GB300, with compute increasing as cameras are added.Source: §9.1 · pp. 19–21, Table 4 · p. 20

9.2. Long Rollouts

• Causal generation, Self Forcing distillation with a progressive long-context teacher, and bounded local attention with attention-sink tokens support stable long rollouts.The paper reports improved visual quality and temporal stability after continued long-context distillation, with decent quality maintained even after minutes of rollout.Source: §9.2 · pp. 21–22, Figure 9 · p. 21, Table 6 · p. 22

9.3. Long-tail Coverage and Counterfactual Variations

• OmniDreams supports long-tail coverage through prompt and first-frame conditioning without retraining, plus targeted finetuning for scenarios too rare to arise naturally.The paper describes these as complementary handles for producing broad scenario coverage in balanced training and evaluation sets.Source: §9.3 · p. 22

9.3.1. Controllable Scenario Editing

• Text prompts control appearance, while the world-scenario map controls scene structure and agent layout; targeted edits leave untargeted scene attributes visually stable.Together with the first-frame RGB seed, these inputs enable systematic variations from a single scene.Source: §9.3.1 · p. 22, Figure 10 · p. 23

9.3.2. Out-of-Distribution Object Modeling

• Randomized dynamic-cuboid dropout lets OmniDreams propagate out-of-distribution objects inserted in the first frame without an explicit cuboid trajectory.This addresses the inconsistent conditioning that can result when an object appears in the RGB seed but has no corresponding position, extent, or trajectory in the world-scenario map.Source: §9.3.2 · pp. 22–23

9.4. Closed-Loop Evaluation

• Closed-loop evaluation pairs OmniDreams sensor synthesis with AlpaSim and Alpamayo 1.5, alternating generated observations and policy actions.The evaluation compares four policy classes over 20-second rollouts on 501 compatible NuRec dataset scenes, with policies replanning every 533 ms.Source: §9.4 · p. 23, §9.4.1 · p. 24

9.4.1. Closed-Loop Comparison to Reconstruction-based NuRec Simulator

• OmniDreams preserves the policy ranking under NuRec and OmniDreams, supporting its use as a faithful proxy for closed-loop policy evaluation.The comparison swaps only the sensor simulator while holding the orchestrator, traffic and physics services, and initial scene state fixed; NuRec serves as a reconstruction-based reference in this regime.Source: §9.4.1 · pp. 23, 25, Figure 13 · p. 25
• OmniDreams remains more stable than NuRec as closed-loop trajectories move farther from recorded paths, while requiring substantially more compute.The simulators are comparable near the recorded trajectory; NuRec’s FVD rises rapidly with greater deviation. OmniDreams also needs only a calibrated first-frame seed and world-scenario representation, whereas NuRec requires synchronized multi-camera capture and per-scene reconstruction.Source: §9.4.1 · pp. 25–26, Figure 14 · p. 26

10.1.1. Reconstruction-based World Models

• Reconstruction-based simulators scale from captured data and render faithfully within logged corridors, but extrapolate poorly; Figure 15 contrasts NuRec with OmniDreams’ more natural pedestrian motion.The comparison says OmniDreams preserves lane geometry, traffic participants, and scene layout while improving pedestrian quality.Source: §10.1.1 · pp. 26–27, Figure 15 · p. 27

10.1.2. Video models as world simulator

• General video models provide broad visual priors, while OmniDreams specializes Cosmos for driving with policy-action and scenario-map control, closed-loop policy interaction, and per-chunk latency targets.The general-purpose models discussed do not target OmniDreams’ per-chunk latency budget.Source: §10.1.2 · pp. 27–28

10.2. Closed-loop AV simulation infrastructure

• AV simulators range from abstract graphics through artist-generated assets to neural reconstruction; world models add a fourth perception-realism tier, supported by AlpaSim’s microservice architecture and chunk-based stateful semantics.AlpaSim’s architecture admits remote, distributed video-model rendering.Source: §10.2 · p. 28

11. Conclusion

• OmniDreams, Alpamayo 1, and AlpaSim form a generative, closed-loop framework for generating highly dynamic, unconstrained autonomous-driving test scenarios.The conclusion frames this as a transition from purely reconstruction-based environments to generative world models.Source: §11 · p. 29
• By synthesizing rare weather anomalies and unpredictable pedestrian movements, OmniDreams addresses long-tail edge cases standard simulators struggle to recreate.Source: §11 · p. 29
• The approach provides scalable tools to rigorously evaluate and potentially train advanced vision-language-action policies, bridging simulated testing and safe real-world deployment.Source: §11 · p. 29

B. Glossary

• The glossary spans autonomous-driving, video-diffusion, parallel-attention, and NVIDIA SIL terminology, with acronyms and product names expanded on first mention in the body.Examples include VLA for a multimodal model that maps images or video plus text to actions, and DiT and VAE for model architecture and latent-space tokenization.Source: B. Glossary · p. 36, Concepts and acronyms · p. 36

Concepts and acronyms

• The acronym reference defines VLA, WAM, BEV, HD map, VRU, and LiDAR, alongside modeling, attention, training, and inference terminology.It also describes streaming KV cache, local-window attention, cross-view attention, Diffusion Forcing, Self Forcing, and Distribution Matching Distillation.Source: Concepts and acronyms · p. 36

NVIDIA SIL platform components

• The glossary identifies Cosmos-Predict 2.5 as NVIDIA’s video-foundation-model platform (33), Alpamayo 1 as NVIDIA’s open-source AV policy stack (36), and AlpaSim as an open-source closed-loop AV simulator (34).AlpaSim is the orchestrator into which OmniDreams plugs; NuRec is a neural-reconstruction tool for AV simulation (30), and SIL-Wheel is NVIDIA SIL’s multimodal AV video search and curation platform (37).Source: NVIDIA SIL platform components · pp. 37–38

Datasets and benchmarks

• RDS is a multi-view AV mid-training corpus sourced from real-world driving logs at 1080p and 30 FPS across 15 countries, while RDS-HQ-1M is the post-training corpus with quality- and HD map-focused human verification.Source: Datasets and benchmarks · p. 38

Mathematical notation

• Notation defines x as a latent video output by the Cosmos VAE encoder; x1:T as a T-frame latent sequence; x_t as its noisy flow-matching latent; c as auxiliary conditioning; and θ as learnable parameters.The production multi-view configuration uses K=16 frames per generation chunk.Source: Mathematical notation · pp. 38–39
• Notation covers denoising-step variables and chunk-based temporal-attention windows, while ↑/↓ mark metric direction, → marks before-and-after values, and × denotes Cartesian products or multiplicative scaling.The denoising-step notation distinguishes a checkpoint’s total denoising steps from a sampled training step; arrows can report latency or quality under a stack-level optimization.Source: Mathematical notation · p. 39