PaperlayerView exact revision on arXiv

Source-linked AI summary

Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim, Jun Gao, Laura Leal-Taixe, Mike Chen, Sanja Fidler, Huan Ling

arXiv:2506.09042v3cs.CV

TL;DR

Collecting large-scale, multi-view, temporally consistent driving data and rare safety-critical scenarios is costly and difficult. Cosmos-Drive-Dreams uses driving-specialized Cosmos models to generate synthetic AV data, which improves downstream perception and policy performance, especially in challenging scenarios.

  • Problem

    Creating large-scale, multi-view, temporally consistent driving datasets is expensive, and rare safety-critical scenarios are difficult to collect at scale.

    Source: §1 · p. 2

  • Method

    Cosmos-Drive-Dreams specializes Cosmos-1 models for driving and uses them in a scalable pipeline to generate synthetic data for downstream AV tasks.

    Source: §1 · p. 2

  • Results

    Synthetic data improves 3D lane detection, 3D object detection, and policy learning, especially in challenging scenarios, with measurable gains when augmenting a large real-world dataset.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors release the models, code, customization toolkit, and synthetic driving dataset through the Cosmos platform.

    Source: §1 · p. 3

    Generative simulators can struggle to produce rare events with high fidelity or frequency when those events are poorly represented in real-world training data.

    Source: §5.3 · p. 21

Abstract

from arXiv · show

Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams

1. Introduction

Cosmos-Drive-Dreams uses driving-specialized world foundation models to generate synthetic data that addresses scarce long-tail scenarios and supports downstream autonomous-driving tasks.

Introduction

  • Cosmos-Drive-Dreams targets rare, safety-critical driving scenarios that are costly to collect but important for robust AV perception and policy models.Naturally occurring data may leave downstream models struggling to generalize to corner cases.Source: §1 · p. 2
  • Prior 3D-scene approaches [10] [26] [46] [49] are costly to scale, while reconstruction methods [9] [43] [48] [51] [59] [67] remain tied to observed scenes.Their non-generative nature limits creation of novel challenging cases such as extreme weather or illumination changes.Source: §1 · p. 2
  • Cosmos-Drive specializes NVIDIA Cosmos-1 for driving, while Cosmos-Drive-Dreams uses the models to generate synthetic data for downstream AV learning.The pipeline amplifies recorded or authored scenes by modifying text descriptions and gathers generated results into synthetic datasets.Source: §1 · pp. 2–3
  • The model suite supports geometry-aware video control, consistent multi-view expansion, annotation of in-the-wild videos, and high-quality LiDAR generation.Structured conditioning can include HDMap, 3D cuboids, text, and optional LiDAR depth; multi-view expansion can produce up to six consistent views.Source: §1 · p. 2
  • The authors release Cosmos-Drive models, post-training and inference code, a customization toolkit, and a diverse synthetic driving dataset.The models are made available through the Cosmos platform.Source: §1 · pp. 2–3
  • Cosmos-Drive-Dreams synthetic data improves 3D lane detection, 3D object detection, and policy learning, especially in challenging scenarios and even alongside large real-world datasets.The introduction states that the pipeline demonstrates measurable downstream gains when augmenting a large-scale real-world dataset.Source: §1 · p. 3

2. Cosmos-Drive: Cosmos Model Suite for AV

Cosmos-Drive post-trains Cosmos world foundation models into a suite for controllable driving-video generation, multi-view expansion, in-the-wild annotation, and LiDAR generation. The suite combines driving-specific post-training with models for layout control and multimodal data generation.

2.1. Background: Cosmos WFM and RDS Dataset Series

  • Cosmos-1 [1] provides generalist world foundation models, while RDS and RDS-HQ provide driving video and annotation data for post-training.RDS contains approximately 3.6 million six-view, 20-second clips; RDS-HQ contains 750 hours of six-view video with HDMap, 3D cuboid, and timestamp-aligned LiDAR annotations.Source: §2.1 · pp. 3–4

2.2. Driving-Specific Post-Training from Cosmos WFM.

  • Cosmos-7B-Text2World is fine-tuned on frontal RDS videos, while its multi-view variant is post-trained on six-view videos.The frontal model is trained for 140k steps at batch size 64 with learning rate 1.5 × 10−5; later mechanisms add layout control and annotation using RDS-HQ data.Source: §2.2 · p. 4

2.3. Cosmos-Transfer1-7B-Sample-AV: Driving Video Generative with Precise Layout Control

  • Cosmos-Transfer1-7B-Sample-AV uses HDMap and LiDAR ControlNets to control scene layout and motion and generate rare or safety-critical scenarios.HDMap videos are rendered from 3D cuboids, lane lines, road boundaries, poles, crosswalks, road markings, traffic lights, and signs; separate HDMap and LiDAR branches can be fused at inference. Text prompts also vary generated scenes, including heavy rain and surreal fire scenarios.Source: §2.3 · p. 5

2.4. Cosmos-7B-Single2Multiview-Sample-AV: Single-View to Multi-View Expansion

  • Cosmos-7B-Single2Multiview-Sample-AV expands frontal video into fixed-rig multi-view video, optionally conditioned on multi-view prompts, HDMaps, or LiDAR.It is fine-tuned from Cosmos-7B-Multiview-Sample-AV using clean input-view tokens, shared self-attention across views, and per-view text cross-attention; all five output views can be generated simultaneously at inference.Source: §2.4 · p. 6

2.5. Cosmos-7B-Annotate-Sample-AV: In-the-Wild Video Annotation Model

  • Cosmos-7B-Annotate-Sample-AV converts in-the-wild RGB video into HDMaps and LiDAR-like depth maps, reducing reliance on costly sensors and manual labels.It predicts road geometry, lane lines, dynamic-object cuboids, and depth from video; modality-specific context embeddings support multiple outputs and mixed datasets that may lack some outputs.Source: §2.5 · pp. 6–8

2.6. Cosmos-7B-LiDAR-GEN-Sample-AV: Specializing Cosmos WFM for LiDAR Generation

  • Cosmos-7B-LiDAR-GEN-Sample-AV generates LiDAR from HDMaps or multi-view RGB, and Prompt Rewriter supports weather-varying outputs with unchanged HDMaps.HDMap conditioning aligns LiDAR with scene layout and bounding-box control; RGB conditioning uses front, rear-left, and rear-right cameras. The model captures weather-dependent LiDAR responses, which vary with conditions such as rain.Source: §2.6 · pp. 8, 10, 11
  • Accurate LiDAR range maps require motion compensation and sensor-specific elevation and azimuth profiles; these corrections remove projection artifacts such as ghost pixels.The pipeline also repeats each range-map row four times to address scanning-pattern and frame-rate sparsity, then clips and normalizes range values.Source: §2.6 · pp. 9–10

3. Cosmos-Drive-Dreams: Scalable Synthetic Data Generation Pipeline for AV

Cosmos-Drive-Dreams generates diverse, high-fidelity multi-view driving videos from structured labels or in-the-wild footage, then filters poor outputs for use as augmented training data. Its prompt rewriting and rejection-sampling modules support varied environmental conditions while preserving scene content and video realism.

3. Cosmos-Drive-Dreams: Scalable Synthetic Data Generation Pipeline for AV

  • The pipeline turns HDMap and 3D-cuboid labels or in-the-wild video into pixel-aligned controls, diversifies captions, generates single-view video, expands it to consistent multi-view perspectives, and filters poor outputs.The structured inputs and generated videos can be used together as augmented data for downstream tasks.Source: §3 · pp. 11–12

3.1. Prompt Rewriter

  • Qwen2.5-7B-Instruct rewrites captions for different times and weather while preserving the scene’s core content, actions, and spatial layout.The specified variations include golden hour, morning, night, and changes to weather or environment; rewritten prompts should avoid details inconsistent with the new conditions.Source: §3.1 · pp. 12–13

3.2. Rejection Sampling

  • Qwen2.5-VL-7B-Instruct [2] automatically rejects videos with artifacts that undermine realism or continuity, enabling scalable, human-free quality control.The filter checks for issues such as object disappearance, shape distortion, rendering glitches, background artifacts, and temporal discontinuity; minor imperfections are acceptable if realism is preserved.Source: §3.2 · p. 14

3.3. Cosmos-Drive-Dreams on the RDS-HQ and Waymo Open Datasets

  • 3% of generated samples are discarded when applying the pipeline to RDS-HQ and Waymo Open [50] videos conditioned on seven weather and time-of-day attributes.Frontal videos are 704 × 1280 pixels and 121 frames at 30 FPS; their first 57 frames are extended into multi-view videos at 576×1024 resolution, and the remaining videos serve as augmented training data.Source: §3.3 · p. 14

4. Cosmos-Drive-Dreams Pipeline Evaluation

Cosmos-Drive-Dreams improves downstream driving-task performance, with particularly strong gains in challenging or underrepresented scenarios. The evaluation covers 3D lane detection, camera-based and LiDAR-based 3D object detection, and policy learning.

4.1. 3D Lane Detection

  • Table 1 reports that Cosmos-Drive-Dreams significantly improves 3D lane-detection performance over baseline and Albumentations.Source: Table 1 · p. 15

4.2. 3D Object Detection

  • Adding Cosmos-Drive-Dreams data to 1K real clips yields +14.5% LET-AP, comparable to a 9.3× increase in real data.Performance gains also scale with the amount of synthetic data added and apply to both single-view and multi-view detection.Source: Figure 16 · p. 17, §4.2 · p. 17
  • HDMap-generated SDG bridges auto-labeled RDS-HQ data to human-labeled RDS-HQ-HL: 1K human-labeled clips plus SDG approximately matches 93K human-labeled clips.This comparison is linearly interpolated; SDG performance gains also increase with the amount of generated data.Source: Figure 16 · p. 17, §4.2 · p. 18

4.3. LiDAR-based 3D Object Detection

  • LiDAR-based 3D detection generally improves when SDG data is added across real-data subsets, with better recall for larger and more distant objects.The evaluation jointly trains Transfusion-LiDAR with generated LiDAR data and real RDS-HQ subsets.Source: Table 3 · p. 18, §4.3 · pp. 18–19, Figure 18 · p. 18

4.4. Policy Learning

  • To reach minADE = 1.35, policy training uses 60k real clips, versus 50k with Rs2r = 0.5 or 39k with Rs2r = 3.0.Adding SDG consistently improves trajectory prediction at each real-data scale, including 135k clips, although the SDG was generated from 50k source clips.Source: §4.4 · p. 19, Figure 20 · p. 20
  • Targeted SDG data comprising ≤2% of training improves policy performance on VRU/left corner cases without reducing general driving performance.The targeted set focuses on vulnerable road users and sharp left turns, using SDG generated from unlabeled RDS clips.Source: §4.4 · pp. 19–20, Figure 20 · p. 20

5. Related Work

Related work on synthetic driving data spans physics-based simulation, reconstruction-based rendering, and generative approaches, each with distinct capabilities and limitations. These trade-offs motivate scalable methods for producing realistic, diverse driving scenarios.

5.1. Physics-based Simulators

  • Physics-based simulators [10] [26] [46] [49] provide repeatable control over environments, weather, traffic, agent behavior, and sensor streams for perception and planning benchmarks.Examples include CARLA [10], MetaDrive [26], and AirSim [49].Source: §5.1 · p. 20
  • Despite fine-grained control, physics-based simulators often lack photorealism and require substantial engineering; covering rare edge cases remains labor-intensive.Examples include snowy nights and multi-agent collisions.Source: §5.1 · p. 21

5.2. Reconstruction-based Simulators

  • Reconstruction-based simulators turn real driving logs into photorealistic neural scenes using NeRF [34] or 3D Gaussian Splatting [22], enabling high-fidelity closed-loop simulation.Examples include UniSim [67], OmniRe [9], and 3DGRUT [35] [63].Source: §5.2 · p. 21
  • Reconstruction methods remain tied to logged viewpoints, can artifact under novel views, and offer limited scene-attribute editing for diverse scenarios.Difix3D [62] and UniSim [67] use generative priors or discriminators to address novel-view limitations; works [28] target relighting.Source: §5.2 · p. 21

5.3. Generation-based Simulators

  • Generative systems support controllable driving-scene synthesis: Drive-WM [56] models driving dynamics, Panacea structures video generation, and MagicDrive [12] [13] [14], Delphi [33], and GAIA [19] enable scene editing or goal-directed generation.These models can generalize beyond handcrafted or reconstructed environments, but rare events remain difficult when poorly represented in training data; GAIA-2 [47] scales training to 14,000 hours.Source: §5.3 · p. 21
  • Cosmos-Drive-Dreams builds on the Cosmos World Foundation Model [1], post-training Cosmos-1 on 20,000 hours of balanced driving footage to improve long-tail and region-specific coverage.The pretrained model draws on tens of millions of physical and AI-related videos, enabling realistic, diverse driving-video generation at scale.Source: §5.3 · pp. 21–22

6. Cosmos-Drive-Dreams Open-Source Summary

The section summarizes the released Cosmos-Drive-Dreams models, dataset, and toolkit, with further details in Table 5.

6. Cosmos-Drive-Dreams Open-Source Summary

  • Table 5 summarizes the released Cosmos-Drive-Dreams models, dataset, and toolkit.Source: §6 · p. 22

A Subset of RDS-HQ Dataset

The section releases a labeled 5,843-clip RDS-HQ subset and 81,802 synthetic front-view clips spanning weather and time-of-day variants. Its toolkit supports HD-map rendering, trajectory customization, and third-party dataset conversion for synthetic data generation.

A Subset of RDS-HQ Dataset

  • The released RDS-HQ subset contains 5,843 ten-second clips with HD-map, 3D-object, camera-intrinsic, and camera-pose labels.HD-map labels cover road and traffic features represented as polylines, polygons, or 3D cuboids; 3D-object labels span vehicles, people, riders, animals, and protruding objects.Source: A Subset of RDS-HQ Dataset · p. 22
  • The synthetic release contains 81,802 121-frame front-view clips with weather variants, divided into two chunks per approximately 300-frame real clip.Variants include golden-hour, morning, night, rainy, snowy, sunny, and foggy; chunk 0 covers frames 1–121 and chunk 1 covers frames 122–242.Source: A Subset of RDS-HQ Dataset · p. 23

6.2. Cosmos-Drive-Dreams Toolkits

  • HD-map video rendering projects 3D polylines, polygons, and cuboids using pinhole or f-theta camera models, producing condition videos for the model suite.The toolkit supports data parallelism.Source: §6.2 · p. 23
  • The trajectory-customization GUI interpolates driving trajectories from recorded camera poses, enabling fine-grained design of corner cases in 3D scenes.A bird's-eye-view visualization helps determine position, and the resulting trajectory can be used in HD-map video rendering.Source: §6.2 · p. 23, Figure 22 · p. 23
  • A conversion script brings third-party datasets such as Waymo Open Dataset into RDS-HQ format for training-data generation or fine-tuning on users' datasets.Users can build a Cosmos-Drive-Dreams pipeline with the pretrained models or fine-tune them on their own data.Source: §6.2 · p. 23

7. Conclusion

Cosmos-Drive-Dreams is a scalable synthetic-data pipeline built on Cosmos-Drive video-generation models, and its generated data enhances perception and policy learning, especially on long-tail cases. Diffusion-model compute makes large-scale generation resource-intensive, motivating future optimization.

7. Conclusion

  • Generated data enhances perception and policy learning, especially in long-tail cases.The pipeline is built on Cosmos-Drive video-generation models post-trained from Cosmos WFM.Source: §7 · p. 23
  • Cosmos-Drive-Dreams is a scalable synthetic-data pipeline built upon Cosmos-Drive video-generation models post-trained from Cosmos WFM.Source: §7 · p. 23
  • Computationally heavy diffusion models make large-scale data generation time- and resource-intensive, a limitation left for future optimization.Source: §7 · p. 23

A. Contributors and Acknowledgements · A.1. Core Contributors

The core contributors are credited across Cosmos-Drive model post-training, pipeline development, downstream evaluation, toolkit development, LiDAR generation, data curation, and architectural design.

A.1. Core Contributors

  • Cosmos-Drive world foundation model post-training is credited to Xuanchi Ren, Tianshi Cao, Amirmojtaba Sabour, Tianchang Shen, and Jun Gao.Ren, Cao, and Sabour are marked as equal contributors.Source: §A.1 · p. 24
  • Cosmos-Drive-Dreams pipeline development is credited to Xuanchi Ren, Yifan Lu, Tianshi Cao, and Jay Zhangjie Wu; data curation is credited to Yifan Lu, Xuanchi Ren, Tianchang Shen, and Mike Chen.Source: §A.1 · p. 24
  • Cosmos-Drive-Dreams downstream task evaluation is credited to Yifan Lu, Ruiyuan Gao, Tobias Pfaff, and Seung Wook Kim.Lu, Gao, and Pfaff are marked as equal contributors.Source: §A.1 · p. 24
  • Cosmos-Drive-Dreams toolkit development is credited to Yifan Lu, Xuanchi Ren, and Tianshi Cao, while architectural design is credited to Sanja Fidler and Huan Ling.Fidler and Ling are marked as corresponding authors.Source: §A.1 · p. 24
  • Cosmos-7B-LiDAR-GEN-Sample-AV post-training is credited to Shengyu Huang and Laura Leal-Taixe.Source: §A.1 · p. 24
  • Cosmos-7B-LiDAR-GEN-Sample-AV downstream task evaluation is credited to Runjian Chen and Shengyu Huang.Source: §A.1 · p. 24

Related papers

Loading 2506.09042v3…