PaperlayerView exact revision on arXiv

Source-linked AI summary

NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation

NVIDIA, :, Aarti Basant, Amlan Kar, Despoina Paschalidou, Fangyin Wei, Francesco Ferroni, Guillermo Garcia Cobo, Haithem Turki, Huan Ling, Jaewoo Seo, James Lucas, Jay Zhangjie Wu, Jialiang Wang, Jonathan Lorraine, Jun Gao, Kai He, Katarina Tothova, Kevin Xie, Michał Tyszkiewicz, Qi Wu, Riccardo de Lutio, Ruilong Li, Sanja Fidler, Seung Wook Kim, Tianchang Shen, Tianshi Cao, Tobias Pfaff, William Lew, Xindi Wu, Xuanchi Ren, Yifan Lu, Yuxuan Zhang, Zan Gojcic, Zian Wang

arXiv:2606.03159v2cs.CVcs.AIcs.RO

TL;DR

Safe closed-loop evaluation of autonomous-driving policies remains difficult because reconstruction-based simulators are tied to captured scenes and struggle with novel conditions. OmniDreams addresses this with a real-time, action-conditioned generative world model, whose downstream policy reduces collision from 6.9% to 4.2% versus Alpamayo 1.5 despite using roughly five times fewer parameters.

  • Problem

    Closed-loop autonomous-driving evaluation needs interactive, scalable simulation, but reconstruction-based methods remain tied to captured data and struggle with substantially new scenes or conditions.

  • Method

    OmniDreams is an action-conditioned generative world model mid- and post-trained on Cosmos to produce photorealistic sensor observations interactively and responsively.

  • Results

    Collision fell from 6.9% to 4.2% versus Alpamayo 1.5 despite OmniDreams’s policy using roughly five times fewer parameters, approximately 2B versus 10B.

  • Takeaways & Limitations

    OmniDreams preserves policy rankings under closed-loop comparison and can serve as a backbone for downstream autonomous-driving policy architectures.

  • Takeaways & Limitations

    Compared with reconstruction-based simulators, OmniDreams requires substantially more compute, creating a quality–compute tradeoff.

Abstract

from arXiv · show

As autonomous vehicle capabilities advance, the safe evaluation of driving policies in long-tail scenarios remains a critical bottleneck. In closed-loop simulation, the driving policy model actively interacts with the environment, where its actions dynamically update the simulator state and directly influence the next set of generated sensor observations. While recent reconstruction-based neural simulators offer photorealism, they are fundamentally constrained by their initial captured data and struggle to generalize to highly dynamic or novel scenes. To overcome these limitations, we introduce OmniDreams, a foundation generative world model mid- and post-trained from the Cosmos diffusion model to autoregressively generate action-conditioned videos in real time. By leveraging the rich visual priors of Cosmos and mid- and post-training on 21k hours of driving scenarios, OmniDreams synthesizes complex, unobserved phenomena that are hard for traditional simulators to capture, such as extreme weather and unpredictable dynamic agent behaviors. Crucially, it autoregressively conditions its photorealistic sensor generation on past frames, the current simulator state, and immediate driving actions. Deployed in a closed-loop system with the Alpamayo 1 policy model and AlpaSim orchestrator, OmniDreams acts as a highly responsive, reactive environment, providing a scalable and comprehensive solution for training and evaluating next-generation autonomous driving policies. We additionally show preliminary results indicating that a world-action model (WAM) post-trained from OmniDreams achieves strong performance on the Physical AI Autonomous Vehicles NuRec dataset, surpassing the VLA-based Alpamayo 1.5 research policy model while using only 1/5 the total parameters. These results highlight the potential for a real-time world model like OmniDreams to also serve as a backbone for policy architectures.

1. Introduction

OmniDreams is introduced as an action-conditioned generative world model for interactive, scalable closed-loop autonomous-vehicle simulation. It generates photorealistic sensor observations while supporting long-rollout consistency, real-time interaction, and downstream driving-policy improvement.

  • Motivation: Closed-loop evaluation requires an interactive, scalable simulator to test autonomous-driving policies in safety-critical, long-tail scenarios before deployment.The policy actively interacts with the environment, so its actions affect subsequent simulator states and observations.
  • Limitations of Existing Simulation: Reconstruction-based neural simulators provide photorealistic testing but remain anchored to captured data, limiting generalization beyond observed scenes.Their what-if testing is grounded in reconstructed real-world environments and controlled variations of recorded scenarios.
  • OmniDreams: OmniDreams is an action-conditioned generative world model mid- and post-trained on Cosmos that produces photorealistic, interactive sensor observations.Its generative visual priors support challenging phenomena, including dynamic weather, unusual deformable objects, and complex agent behaviors.
  • System Design: OmniDreams uses autoregressive diffusion video generation, past-generation KV-cache attention, simulator-state and driving-action conditioning, and real-time rendering.The closed-loop system uses Alpamayo 1 as the policy model and AlpaSim as the simulation orchestrator.
  • Downstream Policy Results: 6.9% to 4.2% collision reduction is achieved by a policy post-trained from OmniDreams on the PAI autonomous-driving dataset.Reported changes also include collision_front from 1.0% →0.9%, collision_lateral from 0.6% →0.4%, and collision_rear from 5.3% →3.0%.

2. Data

OmniDreams is trained on diverse real-world driving logs from two multi-camera datasets, with structured world-state, text, and visual-history conditioning extracted from the data. The training pipeline filters low-quality and repetitive sequences, then composes geographically and scenically varied mid-training, post-training, and evaluation splits.

  • Data sources: Two real-world driving datasets span 15 countries and provide 1080p clips from seven synchronized camera views.The datasets are Real Driving Scene (RDS) and RDS-HQ-1M, which contains 1.14M clips.
  • Conditioning inputs: Training examples pair each clip with a rendered world-scenario map, an environmental text prompt, and a cache of recent visual history.The world-scenario map encodes static HD-map elements, dynamic actors, and policy or driver actions; prompts describe scenery, weather, driving behavior, and traffic.
  • Data curation: A multi-stage quality pipeline removes unreliable sensor or annotation sequences, visual artifacts, and repetitive clips identified through trajectory and visual-feature deduplication.Examples include ego-trajectory jumps, uncertain auto-labeled objects, prediction disagreements, and chromatic aberration.
  • Dataset composition: 16,600 h and 3M clips from RDS support mid-training, while RDS-HQ-1M supplies 4,944 h and 1,142,285 clips for post-training and finetuning.The RDS-HQ-1M split contains 504,488 10 s clips and 637,797 20 s clips; 5,000 clips are held out for evaluation and testing.
  • Scenario balancing: SIL-Wheel constructs post-training mixtures and held-out evaluation slices that emphasize underrepresented rare weather, construction zones, vulnerable road users, and complex multi-agent interactions.The selection process considers geography, weather, time of day, ego behavior, agent density, and rare safety-critical scenarios.

3. Model Architecture

OmniDreams is an autoregressive, action-responsive world model that generates future frames from updated simulator states using causal diffusion and streaming KV caching. It supports single- and multi-view generation through structured conditioning, lightweight control tokens, and factorized attention for efficient cross-view consistency.

  • Autoregressive Generation: At each time step, OmniDreams updates the world state from the latest action and autoregressively generates a short sequence of future frames conditioned on that state.This design supports immediate responses to driving-policy or human inputs during closed-loop simulation.
  • Autoregressive Generation: OmniDreams uses causal diffusion, conditioning each prediction only on past observations and current inputs while maintaining temporal consistency through a streaming KV cache.The cache reuses previously computed attention keys and values.
  • Model Variants: OmniDreams-SV generates one front-facing view with 8 frames per step, whereas OmniDreams-MV jointly generates four synchronized views with 16 frames per step.The multi-view cameras are front, cross-left, cross-right, and front-telescope.
  • Conditioning and Control: The model combines first-frame RGB, text describing environmental attributes, structured maps and dynamic-agent states, and noisy latent tokens in a causal transformer backbone.Structured simulator state is encoded into compact control tokens by a small MLP and concatenated with visual tokens, adding minimal computational overhead.
  • Multi-View Attention: Factorized temporal and cross-view attention avoids full joint attention, improving efficiency while preserving shared-scene consistency across views and enabling up to 7 synchronized views.Temporal attention uses causal KV caching within each view, while cross-view attention operates across views at each time step.

4. Training

OmniDreams uses staged adaptation from Cosmos-Predict 2.5, adding driving, multi-view, world-scenario, and causal-generation training. Self Forcing and progressive-teacher distillation further target exposure bias, long-rollout consistency, and perceptual quality.

  • Multi-stage adaptation: Training begins by mid-training Cosmos-Predict 2.5 on RDS, then adapts OmniDreams-MV with view embeddings and cross-view attention for multi-view generation.The multi-view model uses front-wide, cross-left, cross-right, and front-telescope camera clips.
  • World-scenario map control: World-scenario control is added through a zero-initialized branch, progressing from 93-frame clips to 189-frame clips to learn longer-term temporal consistency.The branch is trained with the flow-matching objective.
  • Autoregressive training: Causal masking with Diffusion Forcing converts bidirectional models into causal models for autoregressive generation, with training continued on RDS and RDS-HQ-1M data.Causal masking restricts each frame’s attention to the current and preceding frames.
  • Self Forcing distillation: Self Forcing combines autoregressive self-rollouts with video-level distribution matching to reduce exposure bias and compounding errors during long rollouts.Its DMD formulation uses reverse-KL matching between self-rolled-out clips and real data, without paired data or pixel-wise supervision.
  • Progressive teacher: A progressive longer-teacher strategy addresses shifting artifacts caused by rolling KV caches extending beyond the training context window.A longer-context bidirectional model serves as teacher for continued finetuning after short-context distillation.
  • Distillation data: 58k high-quality videos are used for distillation to adapt OmniDreams to dense traffic, varied lighting, and intricate road geometry while retaining highway and rural generation quality.The subset comes from the full training corpus and emphasizes perceptually demanding urban conditions.

5. Training-free Model Inference Optimization

OmniDreams achieves real-time closed-loop inference through training-free systems optimization, combining static-shape execution, cache streaming, local attention, lightweight latent processing, and multi-axis parallelism. The deployed system reaches 105 effective FPS per camera for four-view generation and 68 effective FPS for single-view generation.

  • Runtime performance: 105 effective FPS per camera is achieved for 16-frame four-view chunks at 704×1280 in 151 ms on 16 NVIDIA GB300 GPUs.The four-view configuration generates all cameras jointly.
  • Runtime performance: 68 effective FPS is achieved for 8-frame single-view chunks at 704×1280 in 118 ms on a single NVIDIA GB300.Single-view inference already meets the defined 30 FPS real-time threshold on one GPU.
  • Inference optimizations: A pre-allocated static-shape KV cache, asynchronous cache updates, torch.compile, and reusable CUDA Graphs reduce rollout overhead without changing diffusion-model weights.The CUDA Graph is captured lazily on the first chunk and reused for subsequent chunks and same-shape rollouts.
  • Inference optimizations: Local-window attention limits temporal computation to 6 latent frames for single-view and 8 latent frames for multi-view generation.These windows correspond to 24 and 32 RGB frames, trading memory use against speed.
  • Inference optimizations: LightVAE, LightTAE, and pixel shuffle replace heavier latent processing components to accelerate conditioning and decoding, with pixel-shuffle encoding latency below 1 ms for multi-view inference.LightVAE provides the best reconstruction-quality and speed balance for single-view inference.
  • Generalization: FlashDreams packages the streaming cache, local attention, and CUDA-graph techniques into an open-source inference stack validated on additional Wan2.1-based backbones.The stack was also evaluated on Self Forcing, a Wan2.1-based autoregressive text-to-video model.

6. Closed-Loop Simulation Integration

OmniDreams is integrated into AlpaSim as a stateful, gRPC-served camera renderer that supports distributed inference and autoregressive video generation. The integration manages rollout sessions and aligns multi-step trajectories with video-chunk boundaries so closed-loop simulation can consume generated frames consistently.

  • System integration: OmniDreams replaces NuRec as AlpaSim’s camera renderer through new gRPC protocols that manage rendering state and video-model chunk semantics.AlpaSim uses a modular microservice architecture with specialized services communicating over gRPC.
  • Distributed inference: A rank-0 gRPC server receives rendering requests, forwards them to other video-model ranks via NCCL events, gathers completed frames, and returns serialized results to AlpaSim.This avoids unifying OmniDreams’ dependencies, code, and rank-dispatch logic with AlpaSim’s components.
  • State management: Each rollout starts a new session with a seed frame and map representation; the server associates these inputs with a pre-allocated KV cache and returns an ID reused across requests.The session ID identifies which rollout state each request corresponds to, supporting OmniDreams’ autoregressive conditioning on prior frames.
  • Chunk-aligned simulation: Post-fetch generation is an alternative in which policy and traffic models roll out an entire video chunk without receiving new visual inputs before the video-model request.This requires the policy to operate when its latest visual frame lags behind its latest ego position.
  • Chunk-aligned simulation: Because video chunks require ego and actor poses for all frames, AlpaSim generates multi-step trajectories and commits them at chunk boundaries instead of predicting only the next pose.Policy, traffic, physics, and controller services operate at the trajectory level, and trajectories are interpolated at video-frame timestamps before rendering requests.

7. Post-training OmniDreams as a World-Action Model (WAM)

OmniDreams’ single-view 2 B-parameter backbone is fine-tuned into a world-action model that predicts future ego trajectories using visual and motion-history conditioning. Under the Alpamayo 1.5 evaluation protocol, it lowers overall and directional collision rates on the NuRec dataset.

  • Model and training: The OmniDreams-SV checkpoint has 2 B parameters and uses a 120° front-wide camera for end-to-end trajectory prediction.It is compared with Alpamayo 1.5 under the same evaluation setting.
  • Model and training: The policy predicts a 6.4-second future ego trajectory at 10 Hz, producing 64 waypoints.Training begins from an OmniDreams causal autoregressive checkpoint before world-scenario map fine-tuning.
  • Model and training: DINOv2 patch features from a 30° front-telescope camera are projected and summed into noisy video latent tokens, while a history token encodes 1.6 seconds of ego motion.The history token is interleaved with video patches along the temporal axis while preserving video-to-video causality.
  • Evaluation results: 4.2% Collision is achieved by OmniDreams WAM versus 6.9% for Alpamayo 1.5 on 574 NuRec scenes.Evaluation uses 10 Hz replanning and 20-second rollouts, excluding scenes used to train OmniDreams WAM.
  • Evaluation results: 0.9% Front, 0.4% Lateral, and 3.0% Rear collisions are reported for OmniDreams WAM, versus 1.0%, 0.6%, and 5.3% for Alpamayo 1.5.These directional rates correspond to the overall collision comparison on the Physical AI Autonomous Vehicles NuRec dataset.

8. Post-Training OmniDreams as a Diffusion Fixer

OmniDreams is post-trained as an autoregressive correction module that removes novel-view artifacts from reconstruction-based simulator renderings while preserving scene structure and temporal consistency. This combines reconstruction systems’ scene-specific geometric grounding with OmniDreams’ learned visual priors.

  • Motivation: Reconstruction-based simulators can suffer novel-view artifacts that degrade closed-loop realism and introduce distribution shifts for downstream policies.Examples include blur, ghosting, missing regions, and spurious geometry.
  • Artifact correction: The correction module refines reconstructed frames by removing novel-view artifacts while preserving scene layout, camera viewpoint, and driving-relevant structures.Figure 7 shows corrected outputs that retain the original scene organization and viewpoint.
  • Post-training method: OmniDreams is post-trained on paired degraded reconstruction renderings and clean target images for reconstruction-artifact correction.The distilled autoregressive checkpoint uses a denoising procedure initialized from degraded renderings rather than random Gaussian noise.
  • Inference: Autoregressive correction maintains temporal consistency through causal history and KV-cache conditioning.The model operates on rendered frames from the reconstruction system during inference.
  • System integration: The approach lets reconstruction-based simulators retain scene-specific geometric grounding while benefiting from OmniDreams’ learned visual priors.The correction module is applied on top of pre-reconstructed scenes.

9. Experiments and Results

OmniDreams demonstrates realistic and controllable closed-loop simulation through physically plausible dynamics, preserved scene semantics, stable long-horizon rollouts, and targeted scenario variation. In policy evaluation, it preserves the ranking observed under NuRec while offering broader controllability at a higher compute cost.

  • Appearance and Dynamics: OmniDreams generates physically realistic dynamics and preserves traffic lights, lane markings, traffic signs, and roadway structures that affect downstream AV policy behavior.The reported dynamics include articulated vulnerable road users and windshield wipers, while errors in structural details can propagate into policy behavior.
  • Appearance and Dynamics: The final distilled OmniDreams-SV model achieves the best FVD score and strongest fidelity to conditioning signals across 3D detection, lane-line, and temporal-consistency metrics.These quantitative metrics are evaluated on 1,000 clips sampled from the 5,000-clip held-out RDS-HQ-1M evaluation split.
  • Real-Time Multi-View Generation: Real-time multi-view generation remains observable for up to four cameras on NVIDIA GB300, with synchronized views maintaining coherent road layout, actors, lighting, and scene context.Users can post-train the model for the desired camera count and type, although each additional camera increases inference compute.
  • Long-Horizon Stability: Stable long-horizon generation combines causal autoregression with streaming KV caching, Self Forcing distillation, a progressive long-context teacher, and bounded local attention with attention-sink tokens.These mechanisms let the simulator recondition on updated state and recent history at every step while suppressing exposure bias and shifting artifacts.
  • Long-Tail Scenario Control: Prompt, first-frame, and world-scenario-map conditioning enable targeted scenario variations while preserving untargeted road geometry, static structure, and distant background content.Lightweight targeted finetuning further steers generation toward rare distributions, while coordinated conditioning avoids artifacts and inconsistent dynamics from naive first-frame edits.
  • Closed-Loop Policy Evaluation: OmniDreams preserves the closed-loop policy ranking from NuRec, with OmniDreams WAM strongest, followed by Alpamayo 1.5 with 4, 2, and 1 camera.OmniDreams requires only a calibrated first-frame seed and abstract world-scenario representation, whereas NuRec requires synchronized multi-camera capture and per-scene offline 3DGS reconstruction; the tradeoff is higher compute.

10. Related Work

Related work spans world models and simulation infrastructure, including reconstruction-based neural simulators, general-purpose video generators, controllable driving-video models, and systems for interactive rollout. OmniDreams distinguishes itself through action- and scenario-conditioned closed-loop operation, real-time latency, and integration into a fourth tier of perception realism.

  • World Models and Simulation Infrastructure: Closed-loop autonomous-vehicle simulation combines a world model that synthesizes sensor observations with infrastructure that connects the policy, world model, and simulation loop.The paper organizes related work around these two components.
  • Reconstruction-Based Neural Simulators: NeRF- and Gaussian-based methods reconstruct captured driving environments, reducing simulation-to-reality gaps while enabling scalable, rapid conversion of real-world observations into simulation scenes.NeuRAD models automotive sensor characteristics, while EmerNeRF decomposes dynamic scenes into static, dynamic, and flow fields.
  • Generative Video World Models: General-purpose video world models such as Sora, Movie Gen, and Wan motivate broad visual simulation, whereas OmniDreams specializes Cosmos for autonomous driving with policy-trajectory and world-scenario-map control.OmniDreams also runs in a closed loop with a real policy and traffic simulator and targets per-chunk latency that general-purpose models do not.
  • Driving Video Generation: Controllable driving-video systems progress from DriveGAN to diffusion and structured-control models including DriveDreamer, GAIA-1, and GAIA-2.These methods synthesize driving footage from scene-level signals, including controls over weather and agents.
  • Efficient Autoregressive Video Generation: Interactive video-diffusion simulation requires algorithmic and systems advances, including self-rollout, asymmetric distillation, distribution-matching distillation, and methods targeting long-rollout stability.OmniDreams adopts the Self Forcing recipe and combines it with DMD.
  • Simulation Infrastructure and Realism: AV simulators range from abstract graphics and artist-generated assets to neural reconstructions, while OmniDreams positions world models as a fourth tier of perception realism through AlpaSim integration.The AlpaSim extension supports remote distributed rendering and chunk-based stateful semantics for a video-diffusion world model.

11. Conclusion

OmniDreams advances autonomous-driving validation by replacing purely reconstruction-based environments with a generative world model. Integrated with Alpamayo 1 and AlpaSim, it generates highly dynamic, unconstrained testing scenarios involving complex interactive elements.

  • OmniDreams marks a pivotal shift from purely reconstruction-based environments to generative world models for autonomous-driving validation.
  • Integrated with the Alpamayo 1 policy and AlpaSim orchestrator, the framework generates highly dynamic, unconstrained testing scenarios.
  • The model synthesizes complex interactive elements, including rare weather anomalies and unpredictable pedestrian movement.

A. Contributors and Acknowledgments … NVIDIA SIL platform components

The paper credits its research contributors and defines terminology spanning autonomous-driving research, video diffusion, parallel-attention systems, and NVIDIA’s SIL platform components. The glossary also identifies the model variants, simulator, policy stack, datasets, metrics, and implementation technologies used throughout.

  • A.1. Contributors: The research contributors are listed alphabetically, including Aarti Basant, Amlan Kar, Despoina Paschalidou, Fangyin Wei, and 29 additional contributors.The passage provides the complete contributor list through Zian Wa.
  • B. Glossary: The glossary states that acronyms and product names from autonomous driving, video diffusion, parallel-attention systems, and NVIDIA’s SIL platform are expanded at first mention.It presents the terms for reference rather than introducing a separate technical result.
  • Concepts and acronyms: VLA denotes a vision-language-action model that ingests images or video plus text and emits a planned trajectory or control signal.The definition is split across two consecutive glossary passages.
  • Concepts and acronyms: WAM denotes a World-Action Model, a video-conditioned policy that maps video input to actions without an explicitThe supplied passage ends after “explicit,” so the definition is reported only to that point.
  • Concepts and acronyms: The glossary defines BEV and HD maps as structured scene representations, while VRU and LiDAR refer to non-vehicle agents and active depth sensing.HD maps encode lanes, signs, crosswalks, and dynamic-agent bounding boxes; VRUs include pedestrians, cyclists, and scooter riders.
  • Concepts and acronyms: The model glossary identifies DiT, VAE, MLP, AdaLN, RoPE, and KV cache as architectural or inference components used by OmniDreams.It specifically associates DiT with OmniDreams-SV and OmniDreams-MV and KV caching with streaming inference.
  • NVIDIA SIL platform components: The SIL platform glossary links Cosmos-Predict 2.5 to OmniDreams-SV and OmniDreams-MV, Alpamayo 1 to closed-loop demos, and AlpaSim to simulation orchestration.It also identifies NuRec as a neural-reconstruction tool and SIL-Wheel as the platform used to construct training and evaluation slices.

Datasets and benchmarks · Mathematical notation

The paper uses RDS for mid-training and RDS-HQ-1M for post-training, while defining notation for latent videos, conditioning, diffusion objectives, autoregressive generation, and deployment dimensions. The notation also distinguishes overloaded symbols across training, inference, and parallelism contexts.

  • Datasets and benchmarks: RDS is the multi-view autonomous-driving training corpus for mid-training, sourced from real-world driving logs at 1080p and 30 FPS across 15 countries.RDS is identified as the mid-training corpus and its resolution, frame rate, and geographic coverage are specified.
  • Datasets and benchmarks: RDS-HQ-1M is the post-training corpus, verified for data quality and HD-map accuracy.Its verification process focuses on quality and HD maps.
  • Mathematical notation: The latent video x is produced by the Cosmos VAE encoder, while x1:𝑇 denotes a T-frame latent sequence and x<𝑖 its autoregressive prefix.The notation separates the full latent video, frame sequence, individual frame, and prefix used for autoregressive modeling.
  • Mathematical notation: The noisy latent is x𝑡= (1 −𝑡) x + 𝑡𝜖 for 𝑡∈[0, 1], with per-token timestep vector t sampled independently under Diffusion Forcing.The flow-matching notation defines the interpolation and distinguishes sequence-level and per-token timesteps.
  • Mathematical notation: Noise 𝜖 follows 𝒩(0, 𝐼), auxiliary conditioning c includes captions, first-frame images, HD-map control, and ego trajectories, and 𝜃 denotes learnable model parameters.These symbols cover stochastic input, conditioning modalities, and the parameters of OmniDreams DiT and distilled variants.
  • Mathematical notation: Training notation includes expectations E · [·], stop-gradient sg[·], the squared Euclidean norm ‖ · ‖2, and losses such as L𝐷𝐹.The paper also defines model and data distributions, with distillation minimizing a divergence between them.
  • Mathematical notation: The model uses u𝜃 for rectified-flow velocity prediction, v𝑡= 𝜖−x as the flow-matching target, and causal predictors conditioned on x<𝑖.The notation covers conditional velocity prediction and autoregressive latent-prefix prediction.
  • Mathematical notation: Deployment notation distinguishes 𝐾=16 frames per production generation chunk, 𝑁=4 camera views, and overloaded 𝑇 and 𝐾 meanings across inference and training sections.𝑇 can denote sequence length or temporal parallelism, while 𝐾 can denote chunk length or Self Forcing denoising steps with 𝐾=2.
Loading 2606.03159v2…