Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control

Introduction

• Using depth and segmentation controls can make CG-simulator renderings more realistic while preserving scene structure and semantics, addressing the synthetic-to-real domain gap.These inputs are often freely available in CG-based simulators.Source: §1 · p. 1
• Cosmos-Transfer1 adds separately trained, modality-specific ControlNet branches to the Cosmos-Predict1 diffusion-transformer world model and fuses them at inference.The design uses one control branch per input modality.Source: §1 · p. 1
• A spatiotemporal control map weights each modality at each location and time, enabling control over scene geometry, foreground detail, and background diversity.The map can also be inferred by a separate neural module.Source: §1 · p. 1
• The paper evaluates generation quality and controllability on Physical AI world-generation tasks and highlights robotics Sim2Real and autonomous-vehicle data enrichment applications.It also demonstrates an inference-scaling strategy for real-time generation with an NVIDIA GB200 NVL72 rack.Source: §1 · pp. 1–2
• The authors open-source Cosmos-Transfer1 code, model weights, and example scripts to advance research in the field.The repository is https://github.com/nvidia-cosmos/cosmos-transfer1.Source: §1 · p. 2

• The denoiser predicts the noise added to noisy video tokens from those tokens and the noise deviation.Source: §2 · p. 2
• A DiT-based ControlNet pairs the base diffusion model with a control branch of three transformer blocks for conditional inputs.Three blocks were chosen as an empirical balance between control effectiveness and inference efficiency.Source: §2 · p. 2
• Control-branch blocks inherit corresponding base-model weights, and zero-initialized linear-layer outputs are added to matching base activations; training optimizes the added blocks while freezing the base.Source: §2 · pp. 2–3
• Cosmos-Transfer1 uses spatiotemporal control maps to weight modality-specific branch outputs by location, letting relevant inputs guide different regions.The branches extract control information from inputs such as segmentation, depth, and edge.Source: Figure 2 · p. 3
• ControlNet retains the base model's text-prompt input capability when the base model accepts text prompts.Source: §2 · p. 3

Method

• Cosmos-Transfer1 post-trains the Cosmos-Predict1 diffusion world model to generate world simulations from conditional videos across multiple modalities.Source: §3 · p. 3
• A spatiotemporal control map assigns modality weights across video width, height, and time, enabling fine-grained adaptive use of relevant modalities in different regions.Source: §3 · p. 3
• Each control branch’s block activation is multiplied element-wise by its corresponding control-weight slice before being added to the main branch.Source: §3 · p. 3
• Control maps may be manually designed, derived from heuristic rules, or predicted by a trained neural module; modality weights are normalized to sum to one when their total exceeds one at a site.Source: §3 · pp. 3–4
• Control branches are trained separately and fused at inference, reducing training memory and implementation burden while allowing different training data per modality and modality changes at inference.Separate training fits one control branch into memory at a time, and paired data may be difficult to obtain for some modalities.Source: §3 · p. 4

Training setup

• Each standard control branch trains on the high-quality finetuning dataset using 1024 NVIDIA H100 GPUs for 2 to 4 weeks, depending on modality.An inference call generates a 5-second 1280x704p video under 24 fps.Source: §4 · p. 4

General video modalities

• Vis, Edge, Depth, and Seg controls preserve color and composition, object boundaries, scene geometry, and scene semantics, respectively, while enabling other generated changes.The inputs are produced using bilateral blur, Canny edges, normalized depth maps, and object segmentation masks with randomized colors; blur parameters and edge thresholds are randomized during training.Source: §4 · pp. 4–5, Figure 3 · p. 5

Autonomous-driving data

• RDS-HQ provides 65K 20-second surrounding-view clips, approximately 360 hours of driving video, paired with 10 Hz LiDAR scans and dense captions.The dataset was curated for the autonomous-driving model and includes HD map and 3D bounding-box annotations.Source: §4 · pp. 5–6

Autonomous-driving modalities

• HDMap control preserves road layout and supports ego-trajectory simulation, while LiDAR control preserves driving-scene semantics as environmental conditions and lighting change.HD maps combine road annotations with tracked 3D boxes; LiDAR scans are temporally densified, projected onto camera frames, and interpolated to reduce holes.Source: §4 · p. 6

Video upscaling

• The 4KUpscaler converts generated or real 720p videos to 4K using overlapping patch-based generation with averaging at each denoising step for seamless boundaries.Training uses random crops of corrupted high-resolution videos to recover original high-quality patches.Source: §4 · pp. 6–7, Figure 5 · p. 7

5. Evaluations

• TransferBench contains 600 examples—200 each for robotics, driving, and egocentric scenes—and evaluation measures control adherence, diversity, and overall generation quality.Adherence metrics include Blur SSIM, Edge F1, Depth si-RMSE, and Mask mIoU; diversity uses Diversity-LPIPS, and quality uses DOVER-based Quality Score.Source: §5 · pp. 7–8

5.1. Unimodal versus Multimodal

• 0.96 Blur SSIM (Vis), 0.28 Edge F1 (Edge), and 8.54 Quality Score (uniform multimodal) show modality-specific strengths alongside high multimodal quality.Single-control models lead on their corresponding visual and edge metrics, while the uniform multimodal model achieves the best depth reconstruction and highest Quality Score.Source: §5.1 · p. 9, Table 1 · p. 10

5.2. Case Study for Spatiotemporal Control Maps

• 0.43 to 0.81 foreground Blur SSIM follows increased foreground Vis weight (Pearson r = 0.93), while background Depth si-RSME improves from 1.88 to 0.52 (r = −0.92).SalientObject uses a VLM to classify segmented phrases as foreground or background, then assigns Vis and Edge to the foreground and Depth and Segmentation to the background to trade precise foreground control for background variability.Source: §5.2 · pp. 9–11, Figure 7 · p. 11
• Swapping Vis and Edge conditioning to the background improves background Blur SSIM and Edge F1 but degrades foreground alignment; assigning sparser Depth and Segmentation there raises Diversity-LPIPS while maintaining on-par visual quality.Diversity-LPIPS rises from FG 0.01 to 0.12 and BG 0.03 to 0.33 when regions shift from Vis and Edge to Depth and Segmentation conditioning.Source: §5.2 · pp. 11–12, Table 2 · p. 11

5.3. Case Study for Robotics Sim2Real Data Generation

• In 120 robotics videos, spatiotemporal multimodal settings rank among the top three in Quality Score, Diversity-LPIPS, and foreground Mask mIoU, with balanced alignment and improved robot fidelity.They improve photorealism and diversity while preserving robot shape, reducing broken artifacts, and supporting plausible dynamics as lighting, textures, and scene details change.Source: §5.3 · pp. 12–14, Figure 8 · p. 13

5.4. Case Study for Autonomous Driving Data Enrichment

• Combining Depth and Segmentation restores a road line and produces more plausible vehicle orientations, while fusing HDMap and LiDAR improves lane layout realism over single-signal controls.The examples also use these controls to generate variations of safety-critical scenarios for AV testing and training.Source: §5.4 · pp. 15–16, Figure 10 · p. 15
• HDMap+LiDAR control achieves the highest lane mIoU while maintaining 3D-Bbox mAP, with reprojection error comparable to LiDAR-only and better than HDMap-only.The passage identifies lane accuracy as a key metric for AV developers.Source: §5.4 · pp. 17–18, Table 4 · p. 18
• LiDAR-only control achieves the best 3D-Bbox mAP and lowest reprojection error but worst lane mIoU, reflecting its lack of explicit lane layout information.Source: §5.4 · p. 17, Table 4 · p. 18
• Using Omniverse Blueprint Sensor RTX APIs to simulate LiDAR depth, Cosmos-Transfer1 generalizes from real to synthetic LiDAR and generates scenes and effects absent from its AV fine-tuning data.The authors present this as a way to enhance the visual diversity of simulated scenes.Source: §5.4 · p. 18

6. Real-time Inference

• GB200 NVL72's 72 Blackwell GPUs and any-to-any NVLink network support model parallelism, while each B200's 192GB HBM stores a full model copy.Unlike LLMs that generate one token at a time, Cosmos-Transfer1-7B generates tens of thousands of tokens in one shot.Source: §6 · p. 18
• The strategy uses data parallelism in non-attention layers and head-parallelism in attention, splitting a 56K-token sequence across GPUs.All-to-all communication occurs in attention; positive and negative conditioning workloads are split between two GPU groups, distributing attention across 64 GPUs so each handles 56K query, key, and value tokens.Source: §6 · p. 18
• 4.2 seconds generates 5 seconds of video on 64 B200 GPUs, with approximately 40X diffusion-runtime speedup over 1 GPU.Diffusion accounts for over 99% of the workload; end-to-end runtime falls below 5 seconds at 64 GPUs.Source: §6 · pp. 18–19, Table 5 · p. 19

Visual Domain Transfer

• Generative models transfer segmentation maps and sketches into photorealistic images and extend this approach to temporally coherent video across several application domains.The cited work spans static-image conversion (Dundar et al., 2020; Fontanini et al., 2025; Huang et al., 2022; Lv et al., 2022, 2024; Park et al., 2019; Shi et al., 2022; Sushko et al., 2020; Wang et al., 2018, 2022, 2021; Xue et al., 2023) and video synthesis (Chung et al., 2023; Esser et al., 2023; Mallya et al., 2020; Wang et al., 2018, 2019; Zhuo et al., 2022, 2024), with applications including content creation, robotics, autonomous driving, VR/AR, and gaming.Source: §7 · p. 19

Spatial Control for Diffusion Models

• Spatial-control methods divide into training-free and additional-training approaches, with the latter generally achieving superior results, and extensions address video generation.ControlNet (Zhang et al., 2023) adds an encoder branch initialized from a pre-trained model and trains only that branch; video methods adapt its encoder or use masked attention (Lin et al., 2024; Jain et al., 2024).Source: §7 · p. 19

Enhancing Simulation with Generative Models

• Generative models enhance Physical AI simulation through more realistic simulator outputs, scalable scenario generation, and video models explored as learnable simulators for planning and control.Rao et al. (2020) and Ho et al. (2021) use GANs for simulator-image refinement; Zhao et al. (2024) report diffusion with ControlNet outperforms GAN methods with fewer artifacts and better structural fidelity, while Pronovost et al. (2023) use latent diffusion for diverse, safety-critical traffic scenarios. NVIDIA (2025) presented an open-source pre-trained world-model platform for physical-AI tasks.Source: §7 · pp. 19–20

• Cosmos-Transfer1 adds multimodal control branches and adaptive weighting to Cosmos-Predict1 for controllable world generation and improved generation quality.The model is diffusion-based.Source: §8 · p. 20
• Extensive evaluations show that Cosmos-Transfer1 preserves scene structure from condition inputs while enabling fine-grained control for robotics Sim2Real and autonomous vehicle data enrichment.These applications use the model to help bridge the synthetic-to-real domain gap.Source: §8 · p. 20
• Inference scaling enables real-time throughput generation with an NVIDIA GB200 NVL72 rack, and the code and models are open-sourced to advance Physical AI research.Source: §8 · p. 20

A. Prompt Upsampler

• The upsampler addresses a domain shift between diverse user queries and the detailed descriptions used to train Cosmos-Transfer1, which can otherwise lead to suboptimal performance.Its objective is to accommodate requests that vary in style, format, and content.Source: A. Prompt Upsampler · p. 21
• Upsampled prompts should preserve fidelity to the original prompt and conditioned video, retain and elaborate on its elements, and match the structure and length of training prompts.These criteria are fidelity, completeness, and in-distribution consistency (NVIDIA, 2025).Source: A. Prompt Upsampler · p. 21
• A finetuned Pixtral-12B (Agrawal et al., 2024) takes a condition video and user prompt to produce a detailed prompt with consistent structure.The paired dataset contains 1M videos per modality; Gemma-2-9B-it (Gemma Team, 2024) generates diverse short prompts, and the model is jointly trained across modalities for one epoch with FSDP2 (Zhao et al., 2023).Source: A. Prompt Upsampler · p. 21
• With segmentation and depth inputs, the upsampler expands short requests using condition-video context, adding scene and object details beyond the original prompt.Examples expand a robot-task prompt with kitchen, robot, and object details, and a driving prompt with road surroundings and other vehicles.Source: A. Prompt Upsampler · pp. 21–22

B.1. Core Contributors

• The listed core contributors are Maciej Bala, Tiffany Cai, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Pooya Jannaty, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Shitao Tang, Ting-Chun Wang, Jiashu Xu, Xiaodong Yang, Xiaohui Zeng, and Yu Zeng.Source: §B.1 · p. 23
• MYL initiated the adaptive Multimodal Control design, while TCW, TC, and PJ implemented its inference.TCW and XZ also trained the individual ControlNets of Cosmos-Transfer1-7B.Source: §B.1 · p. 23
• The team curated Cosmos-Transfer1-7B training data, robotics simulation data, and the RDS-HQ dataset.ST, YG, QM, and HM curated training data; FR, DF, JG, and YG curated robotics simulation data; YL, XR, and TS curated RDS-HQ.Source: §B.1 · p. 23
• QM, FF, JG, YG, XY, and HM evaluated Cosmos-Transfer1-7B using a benchmark and framework built by FF and QM.QM, HM, and MYL designed the evaluation framework.Source: §B.1 · p. 23
• XR, TS, and HL trained ControlNets and conducted experiments for Cosmos-Transfer1-7B-Sample-AV.SF and HL supervised that model’s research, data curation, training, and evaluation.Source: §B.1 · p. 23
• AH and MB designed and implemented real-time inference on the NVIDIA GB200 NVL72 system.Source: §B.1 · p. 23
• JX and YG built the prompt upsampler.Source: §B.1 · p. 23
• HM supervised Cosmos-Transfer1-7B research, data curation, training, and evaluation; MYL supervised the project.HM and MYL organized paper writing, and all core authors contributed to writing.Source: §B.1 · p. 23