Source-linked AI summary
Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Control
NVIDIA, :, Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, Dieter Fox, Yunhao Ge, Jinwei Gu, Ali Hassani, Michael Isaev, Pooya Jannaty, Shiyi Lan, Tobias Lasser, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Fabio Ramos, Xuanchi Ren, Tianchang Shen, Xinglong Sun, Shitao Tang, Ting-Chun Wang, Jay Wu, Jiashu Xu, Stella Xu, Kevin Xie, Yuchong Ye, Xiaodong Yang, Xiaohui Zeng, Yu Zeng
TL;DR
Cosmos-Transfer1 addresses controllable world generation from multimodal spatial inputs, aiming to preserve scene structure while improving realism for world-to-world transfer. It adds modality-specific control branches with adaptive spatiotemporal weighting, and reports balanced multimodal quality, applications to Physical AI, and real-time generation at scale.
Problem
World simulation videos need controllable generation from multimodal inputs such as segmentation, depth, and edge to improve realism while preserving scene structure and semantics.
Method
Cosmos-Transfer1 post-trains a diffusion world model with separate control branches for each modality and spatiotemporal maps that weight their influence by location and time.
Results
The multimodal model achieves the highest Quality Score (8.54) in the reported unimodal-versus-multimodal comparison and generates 5 seconds of video in 4.2 seconds with 64 GPUs.
Takeaways & Limitations
Cosmos-Transfer1 supports fine-grained world generation for robotics Sim2Real and autonomous-vehicle data enrichment, with open-sourced models and code.
Takeaways & Limitations
The Edge F1 metric requires pixel-level alignment of thin extracted canny edges, producing relatively low F1 scores for all methods.
Abstract
from arXiv · showhide
We introduce Cosmos-Transfer, a conditional world generation model that can generate world simulations based on multiple spatial control inputs of various modalities such as segmentation, depth, and edge. In the design, the spatial conditional scheme is adaptive and customizable. It allows weighting different conditional inputs differently at different spatial locations. This enables highly controllable world generation and finds use in various world-to-world transfer use cases, including Sim2Real. We conduct extensive evaluations to analyze the proposed model and demonstrate its applications for Physical AI, including robotics Sim2Real and autonomous vehicle data enrichment. We further demonstrate an inference scaling strategy to achieve real-time world generation with an NVIDIA GB200 NVL72 rack. To help accelerate research development in the field, we open-source our models and code at https://github.com/nvidia-cosmos/cosmos-transfer1.
1. Introduction
The paper addresses multimodal controllable world generation for video simulations, using adaptive control to support realistic, structurally preserved world transfer and Physical AI applications.
- Motivation: Multimodal controllable world generation produces simulation videos from inputs such as segmentation, depth, and edge.These inputs specify target-world details across spatial locations and time instances.
- Approach: Cosmos-Transfer1 adds separately trained control branches for each modality to a diffusion-based Cosmos-Predict1 world model and fuses them at inference.The design uses one control branch per multimodal video input.
- Adaptive Control: Spatiotemporal control maps weight each modality at each location and time, enabling region-specific control such as emphasizing depth for geometry or edges for foreground detail.Higher weights give modalities greater influence on generation output.
- Evaluation and Applications: The model is evaluated on Physical AI world-generation tasks, with applications including robotics Sim2Real and autonomous-vehicle data enrichment.The paper also discusses practical deployment through inference scaling and open-sourced models and code.
2. Preliminary
The preliminary section explains diffusion denoising and the DiT-based ControlNet extension used to condition the base world model while preserving its frozen pretrained weights.
- Diffusion Denoising: A diffusion denoiser predicts noise from noisy video tokens and their noise deviation.The DiT architecture implements the denoiser with a sequence of transformer blocks.
- ControlNet: ControlNet extends a base diffusion model with a transformer-based control branch for conditional inputs.The control branch outputs are passed through zero-initialized linear layers and added to corresponding base-model activations.
- ControlNet: During ControlNet training, the added control blocks are optimized while the base model remains frozen.This design lets the conditional denoiser leverage conditional tokens while retaining the base model.
- Conditional Inputs: ControlNet inherits the base model's input capabilities, including text prompts when the base diffusion model accepts them.The conditional denoiser predicts noise from noisy tokens, noise deviation, and conditional tokens.
3. Method
Cosmos-Transfer1 uses modality-specific control branches and spatiotemporal weighting to adaptively combine conditional inputs during world generation.
- Model Construction: Cosmos-Transfer1 post-trains the Cosmos-Predict1 diffusion world model to generate simulations from N modality-specific conditional videos.Each conditional input represents a different modality.
- Adaptive Control: The spatiotemporal control map assigns modality weights over video width, height, and frame dimensions for fine-grained adaptive control.It allows the model to select relevant modalities in different spatiotemporal regions.
- Adaptive Control: A control branch activation is multiplied element-wise by its modality's control-map slice before being added back to the main branch.The weighted activation is formed for each control branch and transformer block.
- Control-Map Construction: Control maps may be manually designed, derived from heuristic rules, or predicted by a trained neural module.When modality weights at a site sum above one, they are normalized to sum to one.
- Training Strategy: Individual control branches are trained separately and fused only at inference to reduce memory use and support modality-specific training data.The approach is intended to ease large-scale training with memory-expensive video data.
4. Modality and Training
Cosmos-Transfer1 is instantiated in modality-specific and autonomous-driving variants, with separate control branches trained for each modality. It also includes a 4KUpscaler that converts 720p inputs into 4k outputs using patch-based generation.
- Modalities: Cosmos-Transfer1-7B is trained with separate control branches for modalities including blur visual, segmentation, depth, and edge.Each single-modality setting is named Cosmos-Transfer1-7B [modality name].
- Training and inference: The base model generates 5-second 1280x704p videos at under 24 fps per inference call.This throughput is inherited from Cosmos-Predict1-7B-Video2World.
- Modality behavior: Cosmos-Transfer1-7B [Vis] preserves colors and coarse composition, while [Edge], [Depth], and [Seg] preserve boundaries, geometry, and scene semantics respectively.The figure uses the same prompt across individual modality settings.
- Autonomous-driving variant: Cosmos-Transfer1-7B-Sample-AV uses HD maps and LiDAR-related semantic inputs for autonomous-driving scenes.Its training dataset contains 360 hours of high-quality driving data with HD map and 3D bounding-box annotations.
- Upscaling: Cosmos-Transfer1-7B-4KUpscaler upscales generated or real 720p videos to 4k while adding realistic reflections and sharpening textures.Inference divides the 4k output into overlapping 3 × 3 grids and averages overlapping regions for seamless boundaries.
5. Evaluations
Evaluations show that multimodal and spatially adaptive control balance alignment, quality, diversity, and scene preservation across robotics and autonomous-driving generation tasks. Results also demonstrate practical benefits for Sim2Real and simulation-data enrichment.
- 5.1. Unimodal versus Multimodal: 0.96 Blur SSIM is achieved by visual control, while edge control reaches the best Edge F1 at 0.28.Single modalities specialize in preserving coarse structure and color or capturing dense structural details.
- 5.1. Unimodal versus Multimodal: 0.68 Mask mIoU is achieved by segmentation control, while depth control records Depth si-RMSE of 0.49.Depth and segmentation provide sparser constraints, allowing more generation freedom but potentially less accurate isolated reconstructions.
- 5.1. Unimodal versus Multimodal: 0.37 and 0.31 diversity LPIPS result from excluding Blur visual and Edge, whereas excluding Depth and Segmentation reduces diversity LPIPS to 0.25 and 0.23.Dense controls favor precise alignment and high-fidelity transfer, while sparse controls support more variable scene synthesis.
- 5.1. Unimodal versus Multimodal: The full multimodal model achieves the best depth reconstruction and highest Quality Score of 8.54, despite ranking second in Vis and Edge Alignment.Combining complementary modalities produces more balanced output than relying on a single control input.
- 5.2. Case Study for Spatiotemporal Control Maps: 0.93 Pearson correlation accompanies foreground Blur SSIM improvement from 0.43 to 0.81 as foreground visual weight increases.Increasing background depth weight improves Depth si-RSME from 1.88 to 0.52 with Pearson correlation −0.92.
- 5.2. Case Study for Spatiotemporal Control Maps: Switching regions from visual-edge to depth-segmentation conditioning raises diversity LPIPS from 0.01 to 0.12 in foreground and from 0.03 to 0.33 in background.The swap maintains good depth and segmentation alignment with on-par visual quality.
- 5.3. Case Study for Robotics Sim2Real Data Generation: Spatiotemporal multimodal control improves robotics-video photorealism and diversity while better preserving robot shape and reducing broken artifacts.The approach is presented as a way to enhance synthetic data while preserving task-relevant properties for Sim2Real.
- 5.4. Case Study for Autonomous Driving Data Enrichment: Combining depth and segmentation restores road structure and produces more plausible vehicle orientations, improving generated-scene realism.The autonomous-driving example uses uniform weights of 0.5 for depth and segmentation.
6. Real-time Inference
Cosmos-Transfer1-7B is scaled across GPUs using a parallelism strategy designed for real-time generation. With 64 GPUs, it generates a 5-second 720p video faster than its duration.
- 6. Real-time Inference: The GB200 NVL72 architecture supports model parallelism through its network of 36 Grace CPUs and 72 Blackwell GPUs.The implementation targets Cosmos-Transfer1-7B inference on this system.
- 6. Real-time Inference: The strategy uses data parallelism in non-attention layers and head-parallelism in attention layers.Each B200 GPU can store an entire model copy, allowing the 56K-token sequence for a 5-second 720p video to be sharded across GPUs.
- 6. Real-time Inference: Approximately 40X speedup is achieved from 1 to 64 GPUs for diffusion runtime.Diffusion runtime accounts for over 99% of the workload and is the portion parallelized across GPUs.
- 6. Real-time Inference: At 64 GPUs, the system generates 5 seconds of video in 4.2 seconds, achieving real-time throughput.The reported end-to-end runtime also dips below 5 seconds at this scale.
7. Related Work
Related work spans visual domain transfer, spatial control for diffusion models, and generative methods for improving Physical AI simulation. These lines of research motivate controllable video generation for realistic and diverse simulation.
- 7. Related Work: Visual domain transfer research converts abstract representations such as segmentation maps or sketches into photorealistic images and extends this goal beyond static images.
- 7. Related Work: Spatial control methods for diffusion models include both training-free approaches and methods that require additional training on pretrained models.
- 7. Related Work: Generative models have enhanced simulation by improving realism, diversity, and utility for Physical AI tasks.
- 7. Related Work: Prior work includes simulator-output refinement, latent-diffusion scenario generation for autonomous-agent testing, and video-generation models used as learnable simulators.
8. Conclusion
Cosmos-Transfer1 combines multimodal control branches with adaptive weighting for controllable world generation. The paper reports applications in Physical AI and real-time generation at scale, and releases its code and models.
- 8. Conclusion: Cosmos-Transfer1 adds multimodal control branches and an adaptive weighting scheme to Cosmos-Predict1.
- 8. Conclusion: The model preserves scene structure from condition inputs while supporting fine-grained control for robotics Sim2Real and autonomous vehicle data enrichment.
- 8. Conclusion: An inference scaling strategy enables real-time throughput generation with an NVIDIA GB200 NVL72 rack.
- 8. Conclusion: The authors open-source the code and models to advance Physical AI research.
A. Prompt Upsampler
The prompt upsampler addresses a mismatch between short user queries and detailed training descriptions by expanding prompts using both text and conditioned videos. Its design targets fidelity, completeness, and in-distribution consistency across modalities.
- A. Prompt Upsampler: The prompt upsampler expands short user queries into detailed prompts because training descriptions may differ substantially from user inputs.
- A. Prompt Upsampler: Its three criteria are fidelity to the original prompt and video, completeness of retained elements, and consistency with the training prompt distribution.
- A. Prompt Upsampler: A finetuned Pixtral-12B receives a condition video and user prompt, then produces a more detailed prompt with consistent structure.The condition video may contain segmentation masks or depth.
- A. Prompt Upsampler: For segmentation inputs, the upsampler recognizes scene elements and enriches short descriptions with additional visual details.Examples expand a robot-and-bottle action into descriptions of the kitchen, robot appearance, objects, and subsequent actions.
- A. Prompt Upsampler: For depth inputs, the upsampler develops a holistic understanding of the scene and recognizes surrounding objects.
B.1. Core Contributors
This section lists the core contributors and summarizes their roles in designing, training, curating data for, and evaluating Cosmos-Transfer1-7B.
- MYL initiated the adaptive Multimodal Control design.
- TCW and XZ trained the individual ControlNets of Cosmos-Transfer1-7B.
- TCW, TC, and PJ implemented Multimodal Control inference.
- ST, YG, QM, and HM contributed to Cosmos-Transfer1-7B training-data curation.
- Multiple contributors curated robotics simulation data, the RDS-HQ dataset, and the evaluation benchmark and framework.
B.2. Contributors
This section lists additional contributors to the project.
- Additional contributors include Hassan Abu Alhaija, Jose Alvarez, Tianshi Cao, Liz Cha, Joshua Chen, and other named collaborators.
- The listed collaborators include researchers and engineers such as Mike Chen, Michael Isaev, Shiyi Lan, Tobias Lasser, and Alice Luo.
- The contributor list also includes Xinglong Sun, Jay Wu, Kevin Xie, Stella Xu, and Yuchong Ye.