Source-linked AI summary
Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models
Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, Seung Wook Kim, Jun Gao, Laura Leal-Taixe, Mike Chen, Sanja Fidler, Huan Ling
TL;DR
Collecting and annotating real-world AV data is costly, and rare edge cases are difficult to capture at scale. The paper introduces Cosmos-Drive-Dreams, a synthetic data pipeline powered by driving-specialized Cosmos world foundation models, and reports improved downstream performance across perception and policy-learning tasks, especially for challenging scenarios.
Problem
Real-world AV data collection and annotation are costly, while rare edge cases important for training and testing are difficult to capture at scale.
Method
Cosmos-Drive-Dreams specializes Cosmos-1 world foundation models for controllable, high-fidelity, multi-view driving-video generation and synthetic dataset creation.
Results
Synthetic data improves 3D lane detection, 3D object detection, and driving policy learning, especially in challenging scenarios, with gains also observed when augmenting large real-world datasets.
Takeaways & Limitations
The pipeline provides a way to scale driving-data quantity and diversity with challenging scenarios for downstream AV learning.
Takeaways & Limitations
Generation-based simulators remain reliant on their training distribution, so rare scenarios may be difficult to produce with high fidelity or frequency.
Abstract
from arXiv · showhide
Collecting and annotating real-world data for safety-critical physical AI systems, such as Autonomous Vehicle (AV), is time-consuming and costly. It is especially challenging to capture rare edge cases, which play a critical role in training and testing of an AV system. To address this challenge, we introduce the Cosmos-Drive-Dreams - a synthetic data generation (SDG) pipeline that aims to generate challenging scenarios to facilitate downstream tasks such as perception and driving policy training. Powering this pipeline is Cosmos-Drive, a suite of models specialized from NVIDIA Cosmos world foundation model for the driving domain and are capable of controllable, high-fidelity, multi-view, and spatiotemporally consistent driving video generation. We showcase the utility of these models by applying Cosmos-Drive-Dreams to scale the quantity and diversity of driving datasets with high-fidelity and challenging scenarios. Experimentally, we demonstrate that our generated data helps in mitigating long-tail distribution problems and enhances generalization in downstream tasks such as 3D lane detection, 3D object detection and driving policy learning. We open source our pipeline toolkit, dataset and model weights through the NVIDIA's Cosmos platform. Project page: https://research.nvidia.com/labs/toronto-ai/cosmos_drive_dreams
1. Introduction
Cosmos-Drive-Dreams addresses the cost and scarcity of large-scale, annotated driving data, especially for rare and safety-critical scenarios. It specializes Cosmos-1 world foundation models for controllable synthetic driving-data generation and reports gains across downstream AV tasks.
- Motivation: Real-world collection of large-scale, multi-view, temporally consistent driving datasets with fine-grained annotations is expensive, while rare safety-critical events are difficult to capture at scale.Examples include sudden pedestrian crossings, erratic vehicle behavior, unusual road layouts, and extreme weather.
- Approach: Cosmos-Drive specializes NVIDIA Cosmos-1 world foundation models for driving and powers the Cosmos-Drive-Dreams synthetic data pipeline.The pipeline is designed to generate synthetic data for downstream autonomous-driving task learning.
- Approach: The model suite supports geometry-aware video generation, consistent multi-view expansion, and automatic annotation from in-the-wild driving videos.These capabilities use structured inputs such as HDMaps, 3D cuboids, text, and optional LiDAR depth.
- Pipeline: Cosmos-Drive-Dreams creates variations of recorded or authored scenes by modifying text descriptions and gathers the results into synthetic datasets.The generated data is intended to increase scenario quantity and diversity while retaining high-fidelity driving content.
- Results: Synthetic data improves 3D lane detection, 3D object detection, and policy learning, especially in challenging scenarios, including when augmenting large real-world datasets.The paper also reports releasing the pipeline toolkit, dataset, and model weights.
2. Cosmos-Drive: Cosmos Model Suite for AV
Cosmos-Drive is a driving-specialized suite built from Cosmos world foundation models, combining controlled video synthesis, multi-view expansion, annotation, and LiDAR generation. Its components use structured scene inputs and in-the-wild videos to produce controllable, diverse driving representations.
- Driving-specific post-training: Cosmos-Drive post-trains Cosmos-1 world foundation models into driving-specific text-to-video and dense conditional models.The suite includes Cosmos-Transfer1-7B-Sample-AV, Cosmos-7B-Single2Multiview-Sample-AV, and Cosmos-7B-Annotate-Sample-AV.
- Precise layout control: Cosmos-Transfer1-7B-Sample-AV generates single-view driving videos from HDMaps and prompts, with optional LiDAR inputs for precise layout control.Its structured controls represent cuboids, lane lines, road boundaries, poles, crosswalks, markings, traffic lights, and signs.
- Precise layout control: HDMap conditioning preserves the relative pose between the ego camera and road geometry while enabling varied trajectories and rare or safety-critical scenarios.The paper gives emergency-vehicle encounters, sharp turns, and complex merging behaviors as examples.
- Multi-view expansion: The single-view-to-multi-view model expands frontal-view videos into fixed-rig, view-consistent multi-view videos using optional multi-view HDMap and LiDAR inputs.During inference, all five output views can be generated simultaneously.
- In-the-wild annotation: The annotation model converts in-the-wild RGB driving videos into HDMaps, 3D cuboids, and LiDAR-like depth representations.Its architecture uses modality-specific context embeddings for outputs such as HDMap and LiDAR.
- LiDAR generation: Cosmos WFM is extended with high-quality LiDAR generation conditioned on HDMaps or images for simulating complex driving scenarios.The contribution includes LiDAR preprocessing, tokenizer fine-tuning, and an HDMap- or image-conditioned LiDAR generative model.
3. Cosmos-Drive-Dreams: Scalable Synthetic Data Generation Pipeline for AV
Cosmos-Drive-Dreams generates diverse, high-fidelity multi-view driving data by controlling video synthesis with structured scene inputs and rewritten environmental prompts. Rejection sampling filters unrealistic outputs before the data is used for downstream tasks.
- Pipeline: The pipeline renders HDMap or 3D-cuboid controls, rewrites captions for environmental diversity, generates single-view videos, expands them to consistent multi-view videos, and filters unrealistic outputs.A VLM-based rejection-sampling stage removes poor or implausible generations before dataset construction.
- Prompt Rewriting: An LLM rewrites captions by changing time or weather while preserving the original scene content and spatial layout.Supported variations include golden hour, morning, night, rain, fog, and snow.
- Quality Filtering: A VLM evaluates generated clips for object disappearance, distortion, rendering glitches, consistency problems, background artifacts, and temporal discontinuity.The evaluator classifies clips as Clean or Artifacted, while allowing minor imperfections that preserve realism.
- Quality Filtering: Rejected samples from the rejection-sampling process are documented for the synthetic Waymo Open and RDS-HQ datasets.The paper shows these discarded samples in Figures 13 and 14.
- Data Scaling: Structured labels and in-the-wild driving videos can provide inputs for generating synthetic driving data across varied environmental conditions.The pipeline is applied to RDS-HQ and Waymo Open using attributes such as time of day and weather.
4. Cosmos-Drive-Dreams Pipeline Evaluation
Across 3D lane detection, 3D object detection, and policy learning, adding Cosmos-Drive-Dreams data consistently improves performance, with especially strong gains in challenging weather, smaller-data regimes, and targeted scenarios.
- Overall Evaluation: Synthetic data consistently improves performance across 3D lane detection, 3D object detection, and driving policy learning, especially in extreme weather and nighttime scenarios.Experiments sample generated clips using the synthetic-to-real ratio Rs2r, with “w/o SDG” denoting real-data-only training.
- 3D Lane Detection: Cosmos-Drive-Dreams improves 3D lane detection under both general and extreme weather conditions, including rainy, foggy, and nighttime cases.Visualizations report more accurate scene perception after adding synthetic data.
- 3D Object Detection: Adding SDG data to 1K human-labeled clips yields +14.5% LET-AP, comparable to a 9.3× increase in real data.The comparison is reported for RDS-HQ-HL in the stated setting B experiment.
- 3D Object Detection: Object detection improves across dataset scales in both monocular and multi-view settings, with the largest gains in rainy and foggy corner cases.The reported improvements extend beyond corner cases to general scenes.
- 3D Object Detection: 1K human-labeled clips plus Cosmos-Drive-Dreams data perform approximately comparably to 93K human-labeled clips when linearly interpolated.Performance also correlates positively with the amount of SDG data added.
- Policy Learning: Synthetic data improves policy predictive accuracy at every tested real-data size, including 135k clips, and can reduce data demand for a target minADE.For minADE = 1.35, the reported alternatives are 60k real clips, 50k real clips with Rs2r = 0.5, or 39k clips with Rs2r = 3.0.
- Policy Learning: Targeted SDG augmentation improves the RDS-Bench[VRU/Left] result even when it comprises no more than 2% of the training set without harming general driving performance.The result is reported for policies trained with targeted SDG data.
5. Related Work
Synthetic driving simulation has progressed from controllable physics-based and realistic reconstruction-based systems toward generative models that offer broader visual diversity. Cosmos-Drive builds on this generative direction by post-training a world foundation model on balanced driving data to improve long-tail coverage.
- Physics-based simulators: Physics-based simulators provide structured control over environments, sensors, weather, traffic, and agent behavior, but scaling rare scenarios remains labor-intensive.They also often lack photorealism and require extensive engineering for long-tail diversity.
- Reconstruction-based simulators: Reconstruction-based methods produce photorealistic, reusable closed-loop scenes from driving logs, yet remain constrained by captured viewpoints and limited scene-attribute modification.Novel perspectives can introduce artifacts or degraded quality, while changes to lighting, texture, and weather remain difficult.
- Generation-based simulators: Generative models synthesize realistic driving scenes beyond handcrafted or reconstructed environments, but their coverage of rare events depends on the underlying training distribution.GAIA-2 addresses this limitation by scaling its training data to 14,000 hours of driving video.
- Cosmos-Drive: Cosmos-Drive extends generative driving simulation by post-training Cosmos-1 on 20,000 hours of balanced driving footage to represent long-tail and region-specific conditions.The resulting models target realistic and diverse driving video generation at scale.
6. Cosmos-Drive-Dreams Open-Source Summary
The open-source summary points readers to a table containing details about the released Cosmos-Drive-Dreams models, dataset, and toolkit.
- Released resources: Table 5 summarizes the Cosmos-Drive-Dreams released models, dataset, and toolkit.The passage directs readers to the table for further details.
- Released resources: The section uses Table 5 as the detailed reference for the open-source release.No individual model, dataset, or toolkit specifications are given in the supplied passage.
- Released resources: The open-source summary is organized around three release categories: models, dataset, and toolkit.These categories are named as the contents to be summarized in Table 5.
A Subset of RDS-HQ Dataset
The released RDS-HQ subset provides annotated real driving clips, while the toolkit renders structured labels and supports trajectory customization for synthetic data generation. The synthetic release expands clips across multiple weather and time-of-day variants.
- RDS-HQ subset: The RDS-HQ release contains 5,843 10-second clips with HD maps, 3D objects, camera intrinsics, and camera poses.HD-map labels include road geometry, lane infrastructure, poles, markings, traffic controls, and signs.
- Synthetic dataset: The synthetic dataset contains 81,802 121-frame front-view clips spanning golden-hour, morning, night, rainy, snowy, sunny, and foggy variants.Real clips are divided into two 121-frame chunks for the synthetic release.
- Toolkit: The toolkit supports HD-map video rendering from 3D polylines, polygons, and cuboids using pinhole or f-theta camera models.These rendered videos serve as conditioning inputs for the model suite.
- Toolkit: Its trajectory customization GUI records camera poses at key frames and automatically interpolates a driving trajectory.The interface also provides bird’s-eye-view visualization and exports the full trajectory.
7. Conclusion
Cosmos-Drive-Dreams is presented as a scalable synthetic-data pipeline built on post-trained Cosmos world-foundation-model video generators. The generated data improves perception and policy learning in long-tail cases, while diffusion-based generation remains computationally expensive.
- Contribution: Cosmos-Drive-Dreams is a scalable synthetic-data pipeline built on Cosmos-Drive video generators post-trained from Cosmos WFM.The conclusion identifies this as the paper’s core system.
- Results: Generated data enhances perception and policy learning, especially for long-tail cases.The conclusion states this outcome without specifying individual task metrics.
- Limitation: Diffusion-based generation makes large-scale synthetic-data production time- and resource-intensive.The authors leave optimization of this computational burden to future work.
A.1. Core Contributors
The listed contributors cover Cosmos-Drive model post-training, pipeline development, downstream evaluation, toolkit creation, LiDAR generation, data curation, and architectural design.
- Cosmos-Drive World Foundation Model Post-training is attributed to Xuanchi Ren, Tianshi Cao, Amirmojtaba Sabour, Tianchang Shen, and Jun Gao.
- Cosmos-Drive-Dreams Pipeline Development is attributed to Xuanchi Ren, Yifan Lu, Tianshi Cao, and Jay Zhangjie Wu.
- Downstream Tasks Evaluation is attributed to Yifan Lu, Ruiyuan Gao, Tobias Pfaff, and Seung Wook Kim, while Toolkit work is attributed to Yifan Lu, Xuanchi Ren, and Tianshi Cao.
- LiDAR generation post-training is attributed to Shengyu Huang and Laura Leal-Taixe, with downstream evaluation by Runjian Chen and Shengyu Huang.
- Data Curation is attributed to Yifan Lu, Xuanchi Ren, Tianchang Shen, and Mike Chen; Architectural Design is attributed to Sanja Fidler and Huan Ling.
A.2. Contributors
The listed contributors provide data-pipeline, engineering, solution-architecture, and product-management support for the project.
- Data Pipeline Support is attributed to Yuchong Ye and Zhuohao (Chris) Zhang.
- Engineering Support is attributed to Lyne Tchapmi, Mohammad Harrim, and Pooya Jannaty.
- Solution Architect Partner Support is attributed to John Shao, Yu Chen, and Summer Xiao.
- Product Manager responsibilities are attributed to Aditya Mahajan and Matt Cragun.