PaperlayerView exact revision on arXiv

Source-linked AI summary

Cosmos 3: Omnimodal World Models for Physical AI

NVIDIA, :, Aditi, Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, Aarti Basant, Mukesh Beladiya, Mohammad Qazim Bhat, Zaid Pervaiz Bhat, Dan Blick, Vanni Brighella, Han Cai, Tiffany Cai, Eric Cameracci, Jiaxin Cao, Yulong Cao, Mark Carlson, Carlos Casanova, Ting-Yun Chang, Yan Chang, Yu-Wei Chao, Prithvijit Chattopadhyay, Roshan Chaudhari, Chieh-Yun Chen, Junyu Chen, Ke Chen, Qizhi Chen, Wenkai Chen, Xiaotong Chen, Yu Chen, An-Chieh Cheng, Click Cheng, Xiu Chia, Jeana Choi, Chaeyeon Chung, Wenyan Cong, Yin Cui, Magdalena Dadela, Nalin Dadhich, Wenliang Dai, Joyjit Daw, Alperen Degirmenci, Rodrigo Vieira Del Monte, Robert Denomme, Sameer Dharur, Marco Di Lucca, Ke Ding, Wenhao Ding, Yifan Ding, Yuzhu Dong, Nicole Drumheller, Yilun Du, Aigul Dzhumamuratova, Aleksandr Efitorov, Hamid Eghbalzadeh, Naomi Eigbe, Imad El Hanafi, Hassan Eslami, Benedikt Falk, Jiaojiao Fan, Jim Fan, Amol Fasale, Sergiy Fefilatyev, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Vikram Fugro, Prashant Gaikwad, TJ Galda, Katelyn Gao, Yihuai Gao, Wenhang Ge, Sreyan Ghosh, Arushi Goel, Vivek Goel, Akash Gokul, Rama Govindaraju, Jinwei Gu, Miguel Guerrero, Elfie Guo, Aryaman Gupta, Siddharth Gururani, Hugo Hadfield, Song Han, Ankur Handa, Zekun Hao, Mohammad Harrim, Ali Hassani, Nathan Hayes-Roth, Yufan He, Chris Helvig, Cyrus Hogg, Madison Huang, Michael Huang, Sophia Huang, Yufan Huang, Jacob Huffman, DeLesley Hutchins, Suneel Indupuru, Boris Ivanovic, Arihant Jain, Joel Jang, Ryan Ji, Yanan Jian, Dongfu Jiang, Jingyi Jin, Atharva Joshi, Nikhilesh Joshi, Pranjali Joshi, Andy Ju, Jaehun Jung, Weiwei Kang, Scott Kassekert, Jan Kautz, Ashna Khetan, Julia Kiczka, Slawek Kierat, Gwanghyun Kim, Kuno Kim, Sunny Kim, Kezhi Kong, Xin Kong, Zhifeng Kong, Tomasz Kornuta, Egor Krivov, Hui Kuang, Saurav Kumar, Chia-Wen Kuo, George Kurian, Wojciech Kutak, JF Lafleche, Himangshu Lahkar, Omar Laymoun, Jayjun Lee, Sanggil Lee, Gabriele Leone, Boyi Li, Freya Li, Jiajun Li, Jinfeng Li, Ling Li, Pengcheng Li, Shangru Li, Tingle Li, Xiaolong Li, Xuan Li, Zhaoshuo Li, Zhiqi Li, Hao Liang, Maosheng Liao, Chen-Hsuan Lin, Tsung-Yi Lin, Ming-Yu Liu, Sifei Liu, Zihan Liu, Hai Loc Lu, Xiangyu Lu, Alice Luo, Ruipu Luo, Wenjie Luo, Jiangran Lyu, Martin Ding Ma, Nic Ma, Qianli Ma, Dawid Majchrowski, Louis Marcoux, Miguel Martin, Qing Miao, Ashkan Mirzaei, Shreyas Misra, Kaichun Mo, Durra Mohsin, Hyejin Moon, Pawel Morkisz, Saeid Motiian, Kirill Motkov, Seungjun Nah, Yashraj Narang, Deepak Narayanan, Thabang Ngazimbi, Julian Ouyang, Shubham Pachori, David Page, Yatian Pang, Sehwi Park, Mahesh Patekar, Mostofa Patwary, Marco Pavone, Trung Pham, Wei Ping, Soha Pouya, Shrimai Prabhumoye, Varun Praveen, Delin Qu, Hesam Rabeti, Morteza Ramezanali, Marilyn Reeb, Xuanchi Ren, Kristen Rumley, Wojciech Rymer, Jun Saito, Yeongho Seol, John Shao, Piyush Shekdar, Tianwei Shen, Humphrey Shi, Min Shi, Stella Shi, Kevin Shih, Mohammad Shoeybi, Mateusz Sieniawski, Shuran Song, Alexander Sotelo, Amir Sotoodeh, Sunil Srinivasa, Vignesh Srinivasakumar, Bartosz Stefaniak, Rahul Heinrich Steiger, Shangkun Sun, Jiaxiang Tang, Shitao Tang, Yangyang Tang, Yue Tang, Tolou Tavakkoli, Kayley Ting, Krzysztof Tomala, Wei-Cheng Tseng, Jibin Varghese, Sergei Vasilev, Thomas Volk, Raju Wagwani, Roger Waleffe, Andrew Z. Wang, Boxiang Wang, Haoxiang Wang, Qiao Wang, Shihao Wang, Shijie Wang, Ting-Chun Wang, Yan Wang, Yu Wang, Rohit Watve, David Wehr, Fangyin Wei, Xinshuo Weng, Jay Zhangjie Wu, Kedi Wu, Hongchi Xia, Summer Xiao, Tianjun Xiao, Kevin Xie, Daguang Xu, Jiashu Xu, Mengyao Xu, Ruqing Xu, Xingqian Xu, Yao Xu, Dinghao Yang, Dong Yang, Hans Yang, Xiaodong Yang, Xuning Yang, Yichu Yang, Yurong You, Zhiding Yu, Hao Yuan, Simon Yuen, Xiaohui Zeng, Pengcuo Zeren, Cindy Zha, Haotian Zhang, Jenny Zhang, Jing Zhang, Liangkai Zhang, Paris Zhang, Shun Zhang, Xuanmeng Zhang, Zhizheng Zhang, Ann Zhao, Yilin Zhao, Yuliya Zhautouskaya, Charles Zhou, Fengzhe Zhou, Shilin Zhu, Yuke Zhu, Dima Zhylko, Artur Zolkowski

arXiv:2606.02800v4cs.CVcs.AIcs.LGcs.MMcs.RO

TL;DR

Physical AI training in the real world is slow, expensive, and potentially dangerous, while prior work has largely treated understanding and generation separately. Cosmos 3 jointly models language, image, video, audio, and action, and establishes a new state of the art across most capabilities while remaining competitive with or outperforming specialized models.

  • Problem

    Physical AI training in the real world is slow, expensive, and potentially dangerous, while prior work has largely treated understanding and generation separately.

    Source: §1 · p. 5

  • Method

    Cosmos 3 jointly models language, image, video, audio, and action for understanding and generation within a unified framework.

    Source: §1 · p. 5

  • Results

    Cosmos 3 establishes a new state of the art across most capabilities, remaining highly competitive with or outperforming specialized models.

    Source: §1 · p. 7

  • Takeaways & Limitations

    Cosmos 3 provides a foundation for developing Physical AI agents while reducing the need to compose separate models for understanding, generation, world modeling, and action.

    Source: §8 · p. 80

    Public PAIBench-G Image-to-Video leaderboard results were not reproducible, so the internal evaluation used a different domain-score judge.

    Source: §6.2.2 · p. 57

Abstract

from arXiv · show

We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-transformers architecture. By supporting highly flexible input-output configurations, Cosmos 3 seamlessly unifies critical modalities for Physical AI -- effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework. Our evaluation demonstrates that Cosmos 3 establishes a new state-of-the-art across a diverse suite of understanding and generation tasks, demonstrating omnimodal world models as scalable, general-purpose backbones for embodied agents. Our post-trained Cosmos 3 models were ranked as the best open-source Text-to-Image and Image-to-Video models by Artificial Analysis, and the best policy model by RoboArena at the time the technical report was written. To accelerate open research and deployment in Physical AI, we make our code, model checkpoints, curated synthetic datasets, and evaluation benchmark available under the Linux Foundation's OpenMDW-1.1 License at https://github.com/nvidia/cosmos and https://huggingface.co/collections/nvidia/cosmos3. The project website is available at https://research.nvidia.com/labs/cosmos-lab/cosmos3.

1. Introduction

Cosmos 3 unifies understanding and generation across language, image, video, audio, and action in one framework for Physical AI. The paper reports state-of-the-art results across most capabilities and positions the model as a foundation for scalable training and adaptation.

Motivation

  • Physical AI needs both understanding of world dynamics and generation of plausible futures, yet prior systems separate perception, simulation, and action prediction into fragmented models.The paper argues that this separation is limiting because understanding requires reasoning about future outcomes, while generation depends on structured representations of the world and agent behavior.Source: §1 · p. 5

Contribution

  • Cosmos 3 jointly models language, image, video, audio, and action for understanding and generation, unifying multiple model classes without architectural modifications.Input-output configurations support multimodal reasoning, image and video generation, future prediction, synchronous audio-video generation, and joint action prediction with environmental simulation.Source: §1 · p. 5, Figure 1 · p. 5

Training and adaptation

  • Cosmos 3 addresses training-data and environment bottlenecks through synthetic data generation, task- and embodiment-specific specialization, and a longer-term role generating training environments.It can be post-trained on target data without architectural changes while retaining a common world representation; the paper describes a DROID world-action model and synthetic data-generator variants.Source: §1 · p. 6

Release

  • The authors release code, model checkpoints, curated synthetic datasets, and an evaluation benchmark under the OpenMDW-1.1 License to support open research and deployment.The resources are provided at github.com/nvidia/cosmos and huggingface.co/collections/nvidia/cosmos3.Source: §1 · p. 6

Evaluation

  • Cosmos 3 establishes a new state-of-the-art across most evaluated capabilities, while remaining highly competitive with or outperforming specialized models.Table 1 summarizes the benchmark results; detailed evaluations appear in Section 6.Source: Table 1 · p. 6, §1 · p. 7

2. Model Architecture

Cosmos 3 unifies language, vision, audio, and action in a Mixture-of-Transformers architecture, using autoregressive prediction for language and iterative denoising for other modalities. Its design combines modality-specific encoders, domain-aware action representations, dual-stream attention, and temporal position embeddings.

2.1. Encoders

  • Cosmos 3 embeds multimodal inputs into a shared representation space, adds modality-specific embeddings to non-language tokens, and treats action tokens as links between reasoning, world modeling, and physical control.Language is generated by next-token prediction, while other modalities are generated through iterative denoising.Source: §2 · p. 7, §2.1 · p. 7

2.1.1. Image and Video; 2.1.2. Audio

  • Visual understanding uses a jointly trained ViT, while video generation uses a frozen VAE; audio generation uses a frozen VAE that encodes 48-kHz stereo audio into 25 tokens per second.The ViT uses 16 × 16 patches and merges 2 × 2 tokens; the video VAE compresses temporally by 4× and spatially by 32 × 32.Source: §2.1.1 · p. 7, §2.1.2 · p. 7

2.1.3. Action

  • Actions represent causal world-state transitions and share geometric structure across embodiments through relative-pose pseudo-actions, current grasp states, and domain-specific input and output projections.The representation can include ego poses, effector poses, and grasp states; rotations use a 6D representation and are converted to SO(3) matrices using SVD.Source: §2.1.3 · pp. 8–9, Figure 3 · p. 8

2.2.1. Token Arrangement

  • The unified token format places autoregressive reasoning tokens before diffusion tokens, with clean conditioning tokens preceding noisy targets and vision, audio, and action ordered within diffusion sequences.Autoregressive tokens include language and ViT-encoded media; diffusion tokens include VAE-encoded vision, audio, and action, denoised while jointly attending to autoregressive context.Source: §2.2.1 · p. 9

2.2.2. Generation Mode

  • Generation modes cover language, text-to-image, text-to-video with optional audio, image/video continuation, control-video transfer, and action prediction as forward dynamics, inverse dynamics, or joint video-action policy.For action, forward dynamics predicts visual states from context and clean actions, inverse dynamics infers actions from visual transitions, and policy mode predicts both actions and video.Source: §2.2.2 · pp. 9–10, Figure 4 · p. 10

2.3.1. Dual-Tower Layer Structure; 2.3.2. Dual-Stream Joint Attention

  • Each MoT layer routes autoregressive tokens to a pretrained-VLM-initialized reasoner tower and diffusion tokens to a separate generator tower, whose bidirectional attention reads both streams while autoregressive attention remains causal and self-contained.The towers have independent normalization, attention-projection, and feed-forward parameters, and autoregressive tokens are never updated based on diffusion tokens.Source: §2.3.1 · p. 11, Figure 5 · p. 11, §2.3.2 · p. 12

2.4. Multimodal Position Embedding; 2.4.2. Absolute Temporal Modulation

  • Cosmos 3 extends 3D MRoPE with absolute temporal indexing to align video, audio, and action on a shared physical-time axis, using FPS modulation to accommodate different sampling rates.A fixed temporal gap of 15000 separates autoregressive and diffusion positions to reduce initial-frame oversaturation and checkerboard artifacts.Source: §2.4 · p. 12, §2.4.2 · p. 13, §2.4.1 · p. 13

2.5. Model Variants

  • The Edge, Nano, and Super variants span 4B, 16B, and 64B parameters respectively; Nano and Super are released, while Edge is planned for a later release.Their underlying dense transformers have 2B, 8B, and 32B parameters, respectively.Source: §2.5 · p. 14

3. Data

The data section describes how Cosmos 3’s Reasoner and Generator use distinct curricula that progress from broad pre-training toward specialized capabilities. It covers curation of image, video, audio, and action data with structured annotations, including three action pillars and stricter audio-visual selection for mid-training.

3.1. Reasoner Data

  • The Reasoner curriculum comprises approximately 24.2M samples, shifting from broad image-text and text-only pre-training toward Physical AI specialization in supervised fine-tuning.The 2.2M-sample SFT mixture includes 50% video-text data to strengthen spatiotemporal understanding across robotics, smart infrastructure, and autonomous vehicles.Source: §3.1 · p. 15

3.1.1. Pre-Training

  • The pre-training filter removes 4.23% of data as near-duplicates; AI-judge filtering retains 78% at threshold 2 and 46% at threshold 5.The primary pre-training mixture uses threshold 2 to balance quality with capability coverage, while SFT uses threshold 5 for higher-confidence supervision.Source: §3.1.1 · pp. 16–17
  • The final pre-training mixture contains approximately 22M samples, led by OCR (9.44M, 42.9%), 2D grounding (3.62M, 16.5%), and visual QA (2.48M, 11.3%).It also includes reasoning, captioning, text QA, video QA, and instruction-following data, emphasizing image-text alignment and spatial grounding while retaining a lightweight video component.Source: Table 3 · p. 15, Figure 7 · p. 15, §3.1.1 · p. 17

3.1.2. Supervised Fine-Tuning

  • Reasoner SFT builds spatial, temporal, and physical-plausibility understanding alongside domain data for autonomous vehicles, robotics, and smart infrastructure.Supervision includes simulator-grounded spatial QA, timestamped event and motion data, video-based plausibility judgments, and structured scene-caption upsampling. Domain examples include driving decisions, robot action plans and embodied QA, and warehouse spatial intelligence.Source: §3.1.2 · pp. 17–19

3.2. Generator Data

  • The Generator progressively introduces images, videos, and audio during pre-training, then adds actions and interleaved multimodal content during mid-training.Mid-trained checkpoints are post-trained independently into domain specialists, including Text-to-Image, Image-to-Video, and Policy-DROID models.Source: §3.2 · p. 20, Figure 8 · p. 20

3.2.1. Image and Video

  • Image-and-video pre-training uses 767M images and 347.7M clips processed from 7.8B raw images and 3B source videos.Curation includes preprocessing, embedding-based deduplication, categorization and quality filtering; mid-training adds high-quality and synthetic media plus video-transfer data.Source: §3.2.1 · pp. 20–21
  • Image mid-training uses 60% real, 36% synthetic, and 4% text-rendering images; video mixes 46.0% filtered pre-training clips, 43.9% domain-specific clips, and 10.1% capability-oriented clips.Images undergo resolution and DOVER aesthetic filtering; video curation targets robotics, driving, human activity, and difficult motion or manipulation cases.Source: §3.2.1 · p. 21
  • Synthetic corpora target physical interactions, robot tasks, driving, digital humans, and warehouse operations; 3M transfer videos support control-conditioned RGB generation, including MADS scenario-map inputs.Transfer controls include edges, blur, depth, segmentation, and world-scenario maps; MADS provides 1.1M samples with seven synchronized camera views.Source: §3.2.1 · p. 22
  • Post-training image data uses curated synthetic, text-rendering, and high-quality real images, while video combines a pre-training regularizer with targeted SFT examples.SFT videos include synthetic, human-curated real, and embedding-retrieved real clips selected to address generation-quality gaps and failure cases.Source: §3.2.1 · p. 22
  • Structured JSON captions improve recall while maintaining high precision by systematically covering entities, attributes, relationships, scene content, and video dynamics.The specialized benchmark evaluates assertion-level precision against source media and recall against human-curated ground-truth assertions.Source: §3.2.1 · pp. 22–23

3.2.2. Audio

  • Audio pre-training uses 138.9M video clips with usable tracks, while mid-training filters these to 18.8M clips: 12.8M non-speech and 6M speech-synchronized.The curation aims to retain speech synchronized with visible faces, remove off-screen speech from non-speech examples, and suppress non-instrumental background music.Source: §3.2.2 · p. 23
  • Audio curation separates speech and residual stems, uses synchronization and event classifiers to select branches, and re-captions edited waveforms so text matches training audio.Speech edits retain lip-synchronized speech only when music removal passes speech, music, and non-silence checks; non-speech examples apply vocal or music removal as needed.Source: §3.2.2 · pp. 23–24

3.2.3. Action

  • Paired text-video-action data enables action-conditioned future prediction, trajectory-to-action inference, and joint action-video generation; the curated set contains 8.4M episodes and 61.3K hours.It spans egocentric motion, robotics, autonomous vehicles, and camera motion.Source: §3.2.3 · p. 24, Figure 9 · p. 25
  • Egocentric hand manipulation supplies 67.4% of action hours, alongside driving at 16.3%, robotics at 8.7%, and camera motion at 7.5%.Sources include head-camera poses and hand keypoints, diverse in-house driving logs, robot episodes using state-difference pseudo-actions including failures, and filtered video-derived camera poses.Source: §3.2.3 · pp. 24–25

4. Training

Cosmos 3 trains a Reasoner and a Generator in stages, transferring the Reasoner’s learned world knowledge to generation before progressively adding multimodal, Physical AI, and action-focused training.

4.1. Reasoner Training

  • Reasoner training combines large-scale image-text and video-text pre-training with supervised fine-tuning on curated Physical AI tasks, then initializes the Generator from its weights.The Reasoner jointly trains its language model, ViT, and projector from the start rather than using a separate projector-only alignment stage; SFT sampling is importance-aware and retains a fixed 1:4 pre-training-to-SFT budget stream.Source: §4 · pp. 25–26, §4.1 · p. 26, §4.1.1 · p. 26

4.2.1. Pre-Training

  • Generator pre-training uses rectified flow matching with a shared denoiser across modalities, and trains image, video, and audio generation across multiple tasks and resolutions.It covers 256p, 480p, and 720p and Text-to-Image, Text-to-Video, Image-to-Video, and Video-to-Video; sequences are packed into a 74,000-token budget.Source: §4.2 · p. 27, §4.2.1 · pp. 27–28

4.2.2. Mid-Training

  • Mid-training specializes visual data for Physical AI, using 15.6M image samples and 74.7M curated videos spanning robotics, driving, human activity, physics, and simulation.The mixture targets rare dynamics and embodied scenes while retaining broad visual priors; action losses are scaled by 10× to compensate for smaller per-element MSE.Source: §4.2.2 · pp. 29–30
  • Mid-training adds action and video-transfer supervision while retaining visual generation modes, unifying action, audio, control, and video tokens in a shared temporal and attention structure.Action data trains future-video prediction from actions, action inference from trajectories, and joint action and visual-future generation; transfer data conditions generation on edge, blur, depth, segmentation, or driving-scenario maps.Source: §4.2.2 · pp. 29–30

4.2.3. Text-to-Image Post-Training

  • Text-to-Image post-training uses two SFT stages, first for broad specialization and then for refinement on ultra-high-quality image-caption pairs.The stages run for 20k and 2k steps, respectively; both use a 70k-token context window and images above 720p.Source: §4.2.3 · pp. 30–31
  • 91.36 on UniGenBench, top-1 among open-weight models for Text-to-Image and Image-to-Video, and first on RoboLab, RoboArena, and MolmoSpaces.The UniGenBench score is for the full benchmark; leaderboard rankings were reported at the time of submissions.Source: §4.2.3 · p. 31, §4.2.4 · p. 31, §4.2.5 · p. 32

4.2.4. Image-to-Video Post-Training

  • Image-to-Video post-training uses filtered pre-training data, curated and synthetic clips, and 20% T2I image tokens to retain semantic alignment.The model is specialized for 480p and 189-frame generation through 10k iterations at a learning rate of 1 × 10−5.Source: §4.2.4 · p. 31

4.2.5. Robot Policy Post-Training

  • Robot policy post-training adds proprioception and action-specific modules to Cosmos3-Nano, producing 32 joint-position actions at 15Hz from robot state and three-view images.Inference uses four diffusion steps, CFG parallelism, and skips video-latent decoding to reduce overhead; the policy server runs on two NVIDIA RTX Pro 6000 GPUs.Source: §4.2.5 · pp. 31–32

5. Infrastructure

This section describes the infrastructure supporting Cosmos 3’s multimodal data curation, training, and inference lifecycle, including SILA, specialized and distributed computing techniques, and optimized serving paths. It also covers benchmark evaluation and how prompt upsampling composes Reasoner and Generator pathways.

5.1. Data Infrastructure

  • SILA consolidates storage, metadata management, distributed processing, semantic retrieval, and dataset visualization to curate training data from tens of billions of multimodal candidates.Its framework supports large-scale transformations, embedding-based retrieval and deduplication, and interactive dataset inspection and debugging.Source: §5.1 · p. 32

5.1.1. Large-Scale Data Processing

  • SILA organizes curation in a shared Lance dataset, coordinating fragment-level work and recovery while staging heterogeneous processing operations.Typed columns let researchers add or recompute curation signals incrementally; leased fragments, durable segment checkpoints, and atomic commits support recovery, while staged Ray actor pools use backpressure to control work across stages (Moritz et al., 2018).Source: §5.1.1 · pp. 33–34
  • SILA reduced job startup latency from 30–60 minutes to roughly 5 minutes and achieved a 10× throughput increase over the previous architecture.In peak production windows, individual stages processed billions of row-level annotations per day, shortening large curation campaigns from months to weeks.Source: §5.1.1 · p. 35

5.1.2. Embedding Storage and Semantic Retrieval

  • SILA stores embeddings alongside sample metadata in Lance and builds LanceDB vector indexes over the primary dataset for semantic retrieval, clustering, and deduplication.Production retrieval uses 4096-dimensional embeddings over tens of billions of rows, with 64K IVF partitions, PQ compression, and cosine similarity.Source: §5.1.2 · p. 35

5.1.3. Dataset Visualization, Inspection, and Debugging

  • SILA supports pipeline validation, progress analysis, sample-level inspection, debugging, and analytical querying directly over its shared curation data.Development Lance tables preserve production schema and representative characteristics; fragment metadata estimates column coverage, while interactive tools inspect media and associated curation signals.Source: §5.1.3 · p. 36

5.2. Training Infrastructure

  • A unified training framework supports both Reasoner and Generator, combining multimodal loading, distributed training, training loops, and asynchronous checkpoint saving.HSDP shards model state within replica groups and Context Parallelism shards sequences across devices; on-the-fly variational encoding avoids offline latent extraction.Source: §5.2 · p. 37

5.2.1. Data Loader

  • The Cosmos 3 data loader combines token-budgeted packed sequences, per-stream loaders, rank-synchronous stream selection, and bounded look-ahead packing to handle heterogeneous multimodal batches.A token budget bounds per-step workload without cross-sample padding; synchronized stream choices align ranks, while look-ahead fills residual capacity from smaller samples.Source: §5.2.1 · pp. 38–39
  • Rank-synchronous stream selection improves end-to-end training throughput by 54% over the unsynchronized baseline, while look-ahead packing increases effective sequence length by 8% over baseline.Look-ahead packing preserves skipped samples’ original order and is capped at ten attempts per iteration in production.Source: §5.2.1 · p. 39

5.2.2. Attention Implementation

  • Two-way flat attention improves Cosmos3-Nano end-to-end training throughput by 22% versus a FlexAttention-based baseline.Separate variable-length kernel calls support causal Reasoner attention and bidirectional Generator attention over each sample’s Reasoner-plus-Generator context, avoiding fixed-length padding overhead.Source: §5.2.2 · p. 40, Figure 14 · p. 41

5.2.1. Data Loader; 5.2.3. Distributed Training

  • HSDP shards model state while Ulysses context parallelism shards sequences, and the data loader prewarms every stream before training begins.Ulysses uses two all-to-all collectives per attention layer and reuses the same variable-length kernels; per-iteration loader monitoring reports sampling ratios, packing, buffer waits, and token-budget use. Prewarming primes workers and caches across ranks before the first forward pass.Source: §5.2.1 · pp. 39–40, §5.2.3 · pp. 41–42

5.2.4. Selective Activation Checkpointing

  • Selective Activation Checkpointing improves Cosmos3-Nano end-to-end training throughput by 13% at a 74,000-token per-batch budget, with no change in numerical results.It retains attention outputs, which have high recomputation cost but comparatively low activation-memory cost.Source: §5.2.4 · p. 42

5.2.5. Torch Compile for Transformer Blocks

  • torch.compile with fullgraph=True and dynamic=True improves Cosmos3-Nano Generator training throughput by 41%.Full-graph compilation reduces CPU overhead and enables operator fusion; dynamic shapes handle variable sequence lengths across mixed-modality batches.Source: §5.2.5 · p. 42

5.2.6. Video Tokenizer

  • Chunked Wan2.2 encoding uses 68 frames per call at 256p, 24 at 480p, and 12 at 720p; torch.compile cuts encode latency by a further 52%.Sharding compilation of 45 static-shape graphs across at least 45 ranks reduces tokenizer warm-up from roughly 15 minutes to under one minute; known frame counts can bypass padding and cropping.Source: §5.2.6 · p. 43, Figure 15 · p. 44

5.2.7. Checkpointing

  • Asynchronous checkpointing reduces end-to-end training time by 4% for Cosmos3-Nano and 9% for Cosmos3-Super versus synchronous saves every 30 minutes.Memorizing deterministic save plans further reduces checkpointing overhead by approximately 60%.Source: §5.2.7 · p. 44, Table 7 · p. 45

5.3. Serving Infrastructure; 5.3.1. Plain PyTorch

  • Serving supports Reasoner through TensorRT-LLM and vLLM, Generator through vLLM-Omni, and a native PyTorch reference path; CUDA graphs speed T2I generation by 30% to 60%.The PyTorch path captures repeated transformer-layer computation while leaving dynamic request and sampling logic outside the graph; context parallelism and classifier-free guidance parallelism further reduce inference latency.Source: §5.3 · pp. 45–46, §5.3.1 · pp. 46–47

5.2.8. Throughput Summary

  • Nano processes 507 iterations/hour, 4.56M image tokens and 16.23M video tokens per GPU-hour; Super processes 185 iterations/hour, 1.66M image tokens and 5.91M video tokens per GPU-hour.Per-GPU throughput rises from 520 to 673 TFLOPS and MFU from 0.23 to 0.30 from Nano to Super, despite Super's lower token throughput.Source: §5.2.8 · p. 45, Table 8 · p. 46

5.3.1. Plain PyTorch

  • At 256p, request batching raises text-to-video throughput by 8% to 55%; gains diminish at 480p, and 720p permits no batching under the 74k-token context limit.Batching targets throughput, while distributed inference primarily reduces latency; torch compile and reasoner-output caching benefit both. Prompt upsampling expands a short user prompt into a structured JSON description that conditions the Generator alongside user-provided inputs.Source: §5.3.1 · pp. 47–48, Table 9 · p. 48

5.3.2. Inference Frameworks for Reasoner: vLLM and TensorRT-LLM

  • Nano and Super Reasoners reuse Qwen3-VL (Bai et al., 2025b) support in vLLM and TensorRT-LLM, while Edge Reasoner requires a dedicated, upstreamable vLLM integration for its Nemotron backbone (NVIDIA, 2025).The reused support provides optimized attention kernels, paged KV-cache management, continuous batching, and multimodal input handling with minimal configuration changes.Source: §5.3.2 · p. 48

5.3.3. Inference Frameworks for Generator: vLLM-Omni

  • Cosmos 3 Generator integrates with vLLM-Omni for image, video, audio, and action-conditioned generation, with serving features aimed at reducing latency and memory while improving throughput.Cache-DiT and FP8 quantization reduce denoising cost; Ulysses context parallelism and CFG-Parallel distribute work, while HSDP, CPU offload, and VAE-Patch-Parallel reduce memory pressure. Figure 16 compares single-GPU backend performance and multi-GPU B200 latency scaling; no numerical outcomes are stated in these passages.Source: §5.3.3 · pp. 48–49, Figure 16 · p. 50

5.4. Benchmark Infrastructure

  • The benchmark system orchestrates generation, scoring, and endpoint evaluation across Lepton or Slurm clusters, recording model, code, benchmark, setting, and dataset metadata for score traceability.It supports heterogeneous video, audio, action, and text-response benchmarks using public integrations and custom evaluators, with measures including fidelity, synchronization, adherence, trajectory accuracy, physical plausibility, and reasoning correctness.Source: §5.4 · pp. 49–50
  • Generator evaluation separates output generation from scoring, storing artifacts so they can be rescored with new metrics or evaluators without rerunning generation.Scores and run metadata reside in a relational database, generated artifacts in object storage, and a benchmark portal exposes dashboards, leaderboards, example inspection, and model or checkpoint comparisons.Source: §5.4 · p. 50

6. Results

This section evaluates Cosmos 3 across multimodal understanding and generation, physical simulation, video and audiovisual tasks, and robot policy learning. It reports performance across these evaluations, including policy results, transfer and adaptation, and guidance for flexible generation and multimodal prompting.

6.1. Reasoner Evaluation

  • Cosmos 3 outperforms open- and closed-source models on robotics, smart infrastructure, and driving, while remaining competitive on general benchmarks and trailing Gemini 3.1 Pro (Google DeepMind, 2025a).It improves over Cosmos-Reason2 on general capabilities, while a small gap to Gemini 3.1 Pro remains in robotics; named domain comparisons include RynnBrain (Dang et al., 2026), Mimo-Embodied, and Gemma-4 (Google DeepMind, 2026b).Source: §6.1 · p. 53

6.2.1. Image Generation Evaluation

  • Image-generation evaluation covers prompt following, scene-text rendering, human preference, and aesthetics, including UniGenBench’s 1,170 prompts and CVTG’s character-level text-rendering measures.UniGenBench reports mean testpoint accuracy across original and Physical-AI prompt subsets; CVTG tests English long-prompt and Chinese character rendering, while HPSv3 and Aesthetic V2 provide complementary image-quality measures.Source: §6.2.1 · pp. 54–55
  • Cosmos3-Super-Text2Image ranks #1 among open-weight models and #4 among all models on the Artificial Analysis Text-to-Image leaderboard.The ranking comes from crowdsourced public voting.Source: §6.2.1 · p. 55, Figure 18 · p. 56

6.2.2. Video Generation Evaluation

  • Cosmos3-Super achieves the highest overall PAIBench-G scores on T2V and I2V across all models, while Cosmos 3 matches the second-best RBench result.RBench emphasizes task correctness and physical plausibility in embodied robotics scenarios; Cosmos3-Super also outperforms its Cosmos-Predict2.5 predecessor on both benchmarks.Source: Table 12 · p. 57, §6.2.2 · p. 57
  • 43.8 direct and 48.9 with WMReward+BoN are Cosmos3-Super’s I2V Physics-IQ scores; its V2V scores are 59.7 direct and 63.4 with WMReward+BoN.These results exceed the strongest listed baselines in the corresponding modes and settings.Source: Table 13 · p. 58, §6.2.2 · p. 58
  • Human evaluation ranks Cosmos3-Super highest among open-source models on HUE T2V and HWB, which assess broad Physical AI video generation and egocentric human-manipulation videos, respectively.HUE uses atomic binary verification across semantic alignment, physical laws, geometric reasoning, and visual integrity; HWB reports instruction-following and physical-plausibility pass rates.Source: Table 14 · p. 59, §6.2.2 · pp. 59–60
  • On HWB, Cosmos3-Super scores 71.9, the best overall; Cosmos3-Super also leads open-source models on HUE T2V (89.3) and is 0.1 points behind Veo-3.1 on HUE I2V (89.6 vs. 89.7).Cosmos3-Nano scores 66.9 on HWB, second among open-source models, and ranks second among open-source models on HUE I2V (88.5) and third on T2V (87.6).Source: §6.2.2 · p. 60

6.2.3. Audio Generation Evaluation

  • Seedance-1.5-Pro has the highest overall AVQ (7.64) and PQ (7.06), while Cosmos3-Nano leads SAV, SA, and AVAlign and Cosmos3-Super leads VisualSupport.A Cosmos3-Nano example shows audio transients coinciding with visible hammer strikes, illustrating temporal alignment between acoustic and visual events.Source: §6.2.3 · pp. 61–62, Table 15 · p. 62, Figure 20 · p. 62

6.2.4. Transfer Generation Evaluation

  • Cosmos3-Super matches or surpasses Cosmos-Transfer2.5 across all four PAIBench-C controls: Nano leads DOVER and segmentation, Super leads edge and depth, and all three models are on par for blur SSIM.Cosmos 3 provides these results with a unified backbone rather than dedicated per-modality ControlNet branches.Source: §6.2.4 · p. 63, Table 16 · p. 63
  • On AVBench-C, Cosmos 3 matches or surpasses Cosmos-Transfer2.5 on every driving-scene metric; human video-quality scores are 2.86 for Super and 2.82 for Nano versus 2.59 for the baseline.Lane-line fidelity is comparable across all three models, within ±0.05; Nano leads dynamic-object correspondence, while static-structure and environment scores are on par.Source: §6.2.4 · p. 64, Table 17 · p. 64

6.2.5. Action Generation Evaluation

  • For autonomous-vehicle inverse dynamics, both PT-init Cosmos 3 variants outperform VGGT and DepthAnything3, and MT-init further improves Nano; specialized driving data also enables better metric-scale translation estimation.The comparison uses six-second driving clips with accurate ego-vehicle trajectories at 10 FPS.Source: §6.2.5 · pp. 65–66, Table 18 · p. 65
  • With MT-init, Cosmos3-Super reaches 0.142° RRE, 0.026 m RTE, and 0.99 m ATE in camera-motion forward dynamics, outperforming Lingbot-World and HY-World 1.5 on all three metrics.PT-init already provides robust camera control; MT-init further improves it.Source: §6.2.5 · p. 66
  • On DROID robotics forward dynamics, MT-init reaches PSNR values of 25.52 dB for Cosmos3-Nano and 26.04 dB for Cosmos3-Super; on HWB egocentric motion, scores rise to 16.12 dB and 16.19 dB.On DROID, PT-init Cosmos3-Nano reaches 23.24 dB versus Ctrl-World at 22.99 dB; on HWB, PT-init scores are 15.22 dB for Nano and 15.34 dB for Super, compared with LOME at 9.36 dB.Source: Table 18 · p. 65, §6.2.5 · pp. 66–67
  • Cosmos3-Nano-Policy-DROID sets new state-of-the-art results on RoboLab, RoboArena, and MolmoSpaces; on RoboLab specific instructions, success rate is 39.7% versus π0.5 at 28.1% and DreamZero at 25.2%.The post-trained policy also outperforms Cosmos3-Nano (PT-init), supporting the value of incorporating action data from diverse sources during mid-training.Source: §6.2.5 · pp. 67–68, Table 19 · p. 68
  • Cosmos3-Nano-Policy-DROID ranked #1 on RoboArena and MolmoSpaces, achieving 39.0% oracle success on MolmoSpaces All Combined.The MolmoSpaces evaluation used the same model and hyperparameters as RoboArena, without benchmark-specific tuning.Source: Figure 26 · p. 69, §6.2.5 · p. 69, Figure 27 · p. 70
  • Physical-robot rollouts demonstrate successful language-conditioned manipulation, with the policy tolerating unseen tasks, retrying failures, and remaining robust to human interventions.Figure 25 shows snapshot frames from physical-robot rollouts.Source: §6.2.5 · p. 69, Figure 25 · p. 69
  • At 500 iterations, MT-init reached 24.6% success versus 0.0% for PT-init; by 2,000 iterations, MT-init reached 97.4% on LIBERO-10.Each checkpoint was evaluated with 500 rollouts; the results indicate faster adaptation to a new embodiment and environment with MT-init.Source: §6.2.5 · pp. 69–70, Table 20 · p. 70
  • Camera-motion FD PSNR rose from 11.96 to 12.82 (+0.86) with AV co-training, and several robot domains also improved camera FD PSNR.The reported robot-domain gains include Google Robot (+0.79), Franka Panda Single (+0.73), and Franka Panda Dual (+0.48); ID MSE improved for every shown co-training partner in the camera row.Source: §6.2.5 · p. 71, Figure 28 · p. 71
  • Egocentric motion improved AgiBot FD PSNR from 23.07 to 23.20 (+0.13), while an egocentric-warmed checkpoint improved AgiBot adaptation at every measured step.The warmed checkpoint gained +0.94 FD PSNR at 5K and roughly +1.3–1.6 later; robot co-training also benefited WidowX-250, which gained +1.39 FD PSNR and +2.29 policy PSNR from Google Robot.Source: §6.2.5 · p. 72, Figure 29 · p. 72
  • Across camera, vehicle, egocentric-hand, robot-dynamics, and policy tasks, Cosmos 3 serves as an adaptable action foundation model, with cross-domain co-training improving early adaptation.The conclusion also notes shared structure among forward-dynamics, inverse-dynamics, and policy objectives, and alignment of videos predicted alongside policy actions with simulator rollouts driven by those actions.Source: §6.2.5 · pp. 72–73

6.3.1. Generation Guide

  • The Generator supports 10–30 FPS, 5–400 frames, and 256p, 480p, or 720p outputs, using structured JSON prompts for scene and temporal descriptions.The prompt format also includes audio descriptions and explicit media controls; action modes use 50 denoising steps for forward/inverse dynamics and 4 for policy mode.Source: §6.3.1 · pp. 73–74

6.3.2. Cosmos 3 Reasoner as Prompt Upsampler

  • Cosmos 3 Reasoner expands text, image-plus-text, or video-plus-text requests into a schema-constrained JSON scene program spanning visual attributes, temporal plans, audio descriptions, and generation controls.For image-to-video, the conditioning image is treated as definitive evidence for the first frame, and the template specifies the output fields and grounding constraints.Source: §6.3.2 · pp. 74–75

7. Related Work

Related work traces Physical AI’s progression across world models, multimodal reasoning, generation, and embodied intelligence toward unified representations; Cosmos 3 jointly models language, image, video, audio, and action for understanding and generation.

7.1. World Models for Physical AI

  • Prior work distinguishes compact predictive latent models for planning and control from generative models that expose multimodal futures; Cosmos 3 combines state reasoning with future-token generation.Its reasoner interprets multimodal context and infers structured world state, while its generator synthesizes future image, video, audio, and action tokens.Source: §7.1 · p. 76

7.2. Multimodal Understanding and Embodied Reasoning

  • Most understanding models describe scenes or infer intent without generating future observations or executable actions in the same representation space; Cosmos 3 lets structured understanding condition world generation directly.Physical intelligence requires persistent, spatially grounded scene state and reasoning about affordances, object state, and task progress.Source: §7.2 · p. 77

7.3. Video Generation and Visual World Simulation

  • Realistic video alone is insufficient for world simulation, which requires consistent object identity, temporal causality, and controllable interventions; Cosmos 3 uses video as input, reasoning evidence, and output trajectory.Generated trajectories can be conditioned on text, observations, actions, and audio rather than prompts alone.Source: §7.3 · p. 77

7.4. Action Modeling, VLAs, and World-Action Models

  • Cosmos 3 treats forward dynamics, inverse dynamics, and policy mode as conditioning patterns of one multimodal sequence model over video, audio, text, and action tokens.Prior work organizes action modeling into forward dynamics, inverse dynamics, and policy mode, while many generative systems specialize in a domain, action space, or embodiment.Source: §7.4 · pp. 77–78

7.5. Audio and Audio-Visual Generation

  • Cosmos 3 models audio as synchronized evidence within the world-modeling interface, enabling joint video-audio generation and video-conditioned audio or audio-conditioned video generation.For simulation, sound timing and alignment with visible events matter; audio can provide causal evidence about contacts and partly occluded events.Source: §7.5 · pp. 78–79

7.6. Omnimodels for Understanding and Generation

  • Cosmos 3 pairs an autoregressive reasoner with a diffusion generator to unify multimodal understanding, world simulation, inverse dynamics, and action generation for Physical AI.Its interface supports video/text-to-text understanding, text/video-to-video generation, audio-video generation, and world-action modeling.Source: §7.6 · p. 79

8. Conclusion

Cosmos 3 unifies understanding and generation across language, image, video, audio, and action in one architecture for Physical AI. The paper presents it as a foundation for specialized models and agents that perceive, reason, simulate, and act in the real world.

8. Conclusion

  • Cosmos 3 unifies multimodal understanding and generation across language, image, video, audio, and action, reducing the need to compose separate models.Its modality-specific encoders and structured token arrangements work with a Mixture-of-Transformers backbone coupling autoregressive reasoning and diffusion-based generation.Source: §8 · p. 80
  • Across diverse understanding and generation benchmarks, Cosmos 3 demonstrates strong results and broad capability coverage, while remaining amenable to downstream specialization through post-training.The paper describes scalable data, training, serving, and evaluation infrastructure as a foundation for developing Physical AI agents.Source: §8 · p. 80
  • The authors expect Cosmos 3 to bridge synthetic and real worlds through better synthetic data, starting points for specialized models, and closed-loop Physical AI training environments.They release source code, pre-trained checkpoints, and curated benchmarks to accelerate research toward general-purpose Physical AI agents.Source: §8 · p. 80

A. Caption Details … B. Default Prompts and Prompt Upsampling Templates for Generator

This appendix describes structured captioning schemas and prompt templates for Cosmos 3 Generator across image, video, and action tasks, including negative prompts that specify visual and physical failures. It also covers agentic image-prompt upsampling that evaluates generated images and uses structured feedback to rewrite prompts, with the harness returning a highest-scoring image and supporting early stopping.

A. Caption Details

  • Custom JSON caption schemas provide consistent, detailed image and video annotations while giving the team control over output structure and reducing missing or hallucinated fields.The image schema covers static scene properties; the video schema adds temporal information, including actions, transitions, camera motion, and audio cues.Source: A. Caption Details · p. 81

A.1. Captioner Models

  • Image captioning independently describes four quadrants and the center; video receives a separate pass labeling temporal changes, while both final models use LoRA-fine-tuned Qwen3-VL-8B, with video sampled at 8 FPS and temperature 0.7.The default minimum and maximum pixel bounds are 131,072 and 25,165,824, respectively.Source: §A.1 · p. 81

A.2. Image Schema

  • Image annotations use predefined JSON fields for subjects, scene composition, background, lighting, aesthetics, style, viewpoint, and cinematographic properties, improving detail recall and consistency while maintaining precision.Subject records can include appearance, spatial placement, pose, and count-sensitive attributes.Source: §A.2 · pp. 81–82

A.3. Video Schema

  • The video schema inherits static image attributes and adds actions, state changes, timelines, segments, transitions, camera motion, and optional audio descriptions to capture temporal progression.The number of actions, segments, and transitions varies with video complexity.Source: §A.3 · p. 84

B. Default Prompts and Prompt Upsampling Templates for Generator

  • The generator appendix provides inference-time upsampler instructions for different conditions and points to Tab. 21 for their accompanying sampling configurations.Source: B. Default Prompts and Prompt… · p. 85

B.1. Upsampler Prompt Template for Cosmos 3 Reasoner

  • Reasoner templates convert prompts for text-to-video, text-to-image, image-to-video, and video-to-video tasks into structured JSON, with task-specific timing, resolution, and conditioning constraints.The image-to-video template treats the attached frame as visual ground truth; the video-to-video template treats the conditioning video as the observed prefix and text as future intent.Source: §B.1 · pp. 86–87

B.2. Upsampler Prompt Template for Cosmos3-Super-Text2Image

  • The Cosmos3-Super-Text2Image upsampler expands even brief requests into dense JSON, inferring plausible, internally consistent visual details while requiring every template key and nested field.Empty values are allowed only for truly inapplicable fields, and the template forbids adding or omitting keys.Source: §B.2 · p. 87
  • The Text2Image upsampler requires all top-level JSON keys and structures image descriptions around subjects, setting, lighting, aesthetics, and a comprehensive scene caption.Subject fields cover appearance and pose, while lists and unused fields follow specified empty-value conventions.Source: §B.2 · pp. 87–88

B.3. Upsampler Prompt Template for Cosmos3-Super-Image2Video

  • The Image2Video upsampler turns a starting image and motion instruction into a detailed chronological caption that respects visible content, physical cause and effect, and continuity.It also customizes the negative prompt by removing default terms that contradict the intended video.Source: §B.3 · pp. 88–90

B.4. Prompt Prefix for Video Transfer

  • Video-transfer prompts prepend a system instruction telling the model to condition generation on visual control signals alongside the user caption.The prefix is added via the chat template during both training and inference.Source: §B.4 · p. 90

B.5. Prompt Template for Action Generation

  • Action-generation prompts pair a natural-language behavior description with JSON metadata for viewpoint, action timing, idle frames, duration, frame rate, resolution, and aspect ratio.Multiview inputs are concatenated into one canvas with view-layout metadata; inverse-dynamics samples omit idle-frame metadata.Source: §B.5 · pp. 90–91, Figure 30 · p. 91

B.6. Cosmos 3 Generator Negative Prompt

  • The base-generator negative prompt targets low-quality or distorted visuals, poor spatial coherence, implausible anatomy, and discontinuous motion or state changes.Its listed failure modes include compression artifacts, subjects improperly grounded in scenes, malformed limbs, and abrupt visual flicker.Source: §B.6 · p. 91
  • The negative prompt describes background failures including incoherent depth, visible seams, texture drift, flicker, repeated patterns, and unstable detail across distances.It frames these defects as making the environment feel like a collage rather than a coherent three-dimensional scene.Source: §B.6 · p. 93
  • The negative prompt also targets unstable shadows and lighting, implausible camera motion, and violations of physical laws such as broken collisions, friction, and momentum.The listed shadow defects include temporal shimmer and crawling; camera defects include teleportation and motion disconnected from scene content.Source: §B.6 · pp. 93–94

B.7. Agentic Upsampling for Cosmos3-Super-Text2Image

  • The agentic harness allows up to 2 prompt rewrites, scores generated images, and returns the highest-scoring image, stopping early at a score of at least 9 with no severe issues.The first iteration uses the Appendix B.2 upsampler template and an empty negative prompt; the described submission used GPT-5.5 for upsampling and rewriting and Gemini3.1-Pro for critique.Source: §B.7 · pp. 94–95
  • The critic reports image-quality scores, prompt-element coverage, categorized defects, and specific rewrite directives, checking issues such as physics, anatomy, lighting, depth, text, and prompt following.Its JSON output includes prompt-adherence, visual-quality, aesthetics, physical-plausibility, category, and overall scores, plus issue severity and findings.Source: §B.7 · pp. 95–96
  • Apply relevant visual checks without first classifying the prompt, covering exact text and typography as well as human anatomy, poses, and physically possible interactions.Text checks address spelling, legibility, placement, layout, and commercial or UI clarity; human checks apply when people or body parts are present or required.Source: §B.7 · p. 96
  • For stylized or realistic imagery, assess clean style and geometry or material realism, lighting physics, lens plausibility, and real-world physical constraints.Stylized checks penalize messy geometry, inconsistent line language, broken vector shapes, muddy palettes, and unwanted photorealistic texture.Source: §B.7 · p. 96
  • The VLM reports 0–10 quality, adherence, plausibility, and category scores, applicable text-rendering and photorealism scores, issues, prompt-element findings, and improvement directives.Its structured feedback also includes category findings and a rationale for the assessment.Source: §B.7 · p. 96
  • Iteration history records each iteration’s overall, prompt-adherence, and category scores and whether it cleared the threshold.The history entries include an iteration number and a threshold-cleared boolean.Source: §B.7 · p. 96
  • The rewrite returns exactly two top-level keys, pairing a schema-preserving positive prompt with a concise negative prompt that suppresses wrong alternatives without negating required content.The positive prompt retains the prior resolution and aspect ratio; negative instructions suppress concrete artifacts rather than provide positive guidance.Source: §B.7 · p. 97

C. Synthetic Dataset for Generator Training … C.5. SDG-Warehouse

The appendix describes five synthetic datasets used for generator mid-training, their modalities and distributions, and an ablation study of their use in Cosmos 3 training.

C. Synthetic Dataset for Generator Training

  • The appendix documents five SDG datasets, summarizes their scale and modality coverage, and describes an ablation comparing their use in generator training.Table 22 identifies RGB, depth, segmentation, bounding boxes, physics state, camera parameters, and captions as covered modalities, with partial coverage noted where applicable.Source: C. Synthetic Dataset for Generator… · p. 97, Table 22 · p. 97

C.1. SDG-PhyxSim

  • SDG-PhyxSim provides 76,489 simulated interaction runs, captured from four synchronized views with dense depth, segmentation, and per-object physics annotations.The release spans ten scene families and totals approximately 57 million RGB frames; lossless depth video is the largest contributor to its ∼14.9 TiB storage.Source: §C.1 · pp. 97–98

C.2. SDG-RobotSim

  • SDG-RobotSim releases 386,270 RGB clips across collision, manipulation, and humanoid motion, covering mobile, quadruped, humanoid, manipulator, bimanual, and dexterous embodiments.Clips include task and generator metadata, embodiment, camera setup, and available simulator state; generator-specific annotations may include robot and object poses, contacts, and task success.Source: §C.2 · pp. 99–100

C.3. SDG-DriveSim

  • SDG-DriveSim contains 264,000 clips across seven scenario families, targeting rare driving events such as emergency interactions, obstacle nudging, cut-ins, degraded visibility, and unusual pedestrian crossings.The release totals approximately 1,467 hours at 4K and 24 fps; natural-language-authored scenarios are expanded into controlled permutations of environmental conditions and assets.Source: §C.3 · pp. 100–101

C.4. SDG-SynHuman

  • SDG-SynHuman provides 236,937 clips totaling 5,841 hours, pairing rendered RGB with metric depth and per-frame camera parameters generated deterministically from the scene graph.The release spans 4,050 digital human assets, 8,184 animations, and 398 indoor and outdoor environments, with clips containing one to nine humans and varied camera motion.Source: §C.4 · pp. 101–102

C.5. SDG-Warehouse

  • SDG-Warehouse comprises ∼123K clips across four simulated industrial-safety scenarios, with synchronized views and deterministic per-pixel and per-object annotations.The scenarios are forklift–human near-misses (23%), warehouse fires with evacuations (36%), forklift–shelf collisions (20%), and box pickup (21%); the release totals ∼412 hours at 1920×1080 and 30 fps.Source: §C.5 · p. 103
  • The SDG ablation compares models fine-tuned from the same pretrained baseline using one source at a time or a mix of all five sources on PAIBench-G T2V.The supplied passage describes the comparison setup and scoring annotations but provides no result values.Source: Table 26 · p. 104

C.6. Distribution of SDG Datasets

Synthetic data generation (SDG) occupies long-tail regions of the video embedding space that web-scale pre-training alone does not sufficiently cover.

C.6. Distribution of SDG Datasets

  • SDG embeddings occupy long-tail regions beyond sufficient pre-training coverage, as shown by joint embedding geometry and distances to the pre-training manifold against an in-distribution baseline.Fig. 36 visualizes the joint geometry, while Tab. 25 quantifies distances from each SDG source to the pre-training manifold.Source: §C.6 · p. 104

C.7. Ablation Study: Impact of SDG Datasets

Different synthetic data sources improve different evaluation domains, but individual sources can harm others, especially Human. Combining all sources produces broad, balanced gains while retaining a residual Human-domain decline.

C.7. Ablation Study: Impact of SDG Datasets

  • SDG-DriveSim gains Robot (+0.89) and Common Sense (+0.52); SDG-RobotSim raises Quality most (+0.26), while Warehouse improves AV (+0.29) and Physics (+0.21).Warehouse also improves Overall and Domain and maintains Robot (+0.60); RobotSim lifts Robot (+0.23), and PhyxSim gains Industry (+0.85) and Robot (+0.80).Source: §C.7 · pp. 104–105
  • SDG-SynHuman leads individual sources with the best overall score (79.79) and gains in Domain (+0.29), Common Sense (+0.66), AV (+0.47), and Robot (+1.23).The passage attributes its broad transfer to the breadth of human-centric synthetic content.Source: §C.7 · p. 105
  • Human declines for every source, from −0.38 with SDG-SynHuman to −0.69 with SDG-PhyxSim; RobotSim also reduces AV (−1.03) and Industry (−1.22), and PhyxSim reduces AV (−1.42).The passage attributes the Human-domain gap to simulators' limited fidelity in reproducing real human appearance, motion, and behavior, and argues for mixing sources.Source: §C.7 · p. 105
  • SDG-All improves eight of nine metrics, reaches the best Quality score (72.56), and leaves only Human lower (−0.47).Its sources form distinct, tightly clustered regions with narrow overlap against the pre-training distribution, supporting their role as complementary data rather than redundant subsets.Source: §C.7 · p. 105, Figure 36 · p. 105, Table 25 · p. 105

D. Cosmos3-Edge LLM Model Training

Cosmos3-Edge is a dense 2B model trained from scratch through base pre-training, long-context extension, and supervised fine-tuning. Its SFT model outperforms Qwen3.5-2B on math and science benchmarks, is comparable on instruction following and long context, and lags on general-domain evaluation.

Base pre-training

  • Base pre-training uses 15T Nemotron-corpus tokens at 8,192-token sequences, switching from broad to higher-quality data while preserving optimizer state and the learning-rate schedule.The data-mixture switch aligns with the WSD schedule’s transition from stable to decay; the tokenizer is shared with NVIDIA Nemotron-3 models (NVIDIA, 2025).Source: D. Cosmos3-Edge LLM Model Training · p. 106

Long-context extension

  • Long-context extension trains on 128K-token sequences to support a 128K context window, while targeting robustness and quality at the deployed 32K-token sequence length.It uses a RoPE base of 1e8, a constant learning rate of 1.2 × 10−5, and 90B tokens; the data blend adds long-document QA at 20%.Source: D. Cosmos3-Edge LLM Model Training · p. 106

Supervised fine-tuning

  • SFT uses Nemotron-Cascade-2 data (Yang et al., 2026b) across mathematics, coding, science, chat, instruction following, tool use, and code-agent tasks.Approximately 26M examples are packed into roughly 2.6M sequences of up to 128K tokens; capability peaks after approximately 1.7 epochs (140K steps).Source: D. Cosmos3-Edge LLM Model Training · p. 106

Evaluation

  • The SFT model substantially outperforms Qwen3.5-2B on math and science benchmarks, matches it on instruction following and long context, and lags on general-domain benchmarks.The comparison covers HMMT25 Feb, GPQA, MMLU-Pro, AA-LCR, IFBench, and Scale AI Multi-Challenge.Source: D. Cosmos3-Edge LLM Model Training · p. 106

E. Additional Ablation Study

The additional ablations examine architectural, data, and training choices behind Cosmos 3’s capabilities. The supplied text benchmark comparison reports stronger mathematical and scientific reasoning for Cosmos3-Edge than Qwen3.5-2B, with comparable instruction-following and long-context scores.

E. Additional Ablation Study

  • The ablations examine how the Reasoner and Generator interact in the Mixture-of-Transformers framework and assess the impact of audio and action training signals.Source: E. Additional Ablation Study · p. 106
  • The studies also investigate temporal conditioning and whether learned world and action representations transfer across domains.Source: E. Additional Ablation Study · pp. 106–107
  • Cosmos3-Edge substantially improves mathematical and scientific reasoning over Qwen3.5-2B on HMMT25 Feb and GPQA, while scoring comparably on IFBench and AA-LCR.The table caption provides no numerical scores.Source: Table 27 · p. 107

E.1. How the Reasoner Benefits the Generator

Replacing Qwen3-VL-8B with the Cosmos3-Nano Reasoner in the understanding tower improves Generator domain scores, especially in Physical AI domains, while quality scores remain comparable. Both variants use a Generator tower trained from scratch.

E.1. How the Reasoner Benefits the Generator

  • The ablation compares Cosmos3-Nano models with Qwen3-VL-8B or Cosmos3-Nano Reasoner understanding towers, training both Generator towers from scratch for 90K iterations on 256 GPUs.Training is restricted to 256p and 480p, 0–200-frame clips, and 25K sequence length, using joint image–video training; quality scores are comparable.Source: §E.1 · p. 107
  • 75.7 vs. 73.7 on T2V and 80.8 vs. 80.0 on I2V are the Domain scores for the Reasoner and Qwen3-VL variants, respectively.Table 28 reports PAIBench domain and quality scores; subject and background consistency apply only to I2V.Source: Table 28 · p. 107, §E.1 · p. 107
  • On T2V, the Reasoner raises Robot by +4.8 (66.5 →71.3), Physics by +0.5 (88.7 →89.2), AV by +2.3 (52.6 →54.9), and Industry by +1.0 (77.4 →78.4).Source: §E.1 · p. 107, Table 28 · p. 107
  • On I2V, the Reasoner improves Domain scores in Common sense (+0.7), Industry (+0.8), Human (+1.0), and Physics (+0.2).Source: §E.1 · p. 107

E.2. Choice of FPS Control

The ablation compares two ways to condition generation on a target frame rate and finds that combining them yields the best composite score, with gains concentrated in motion fidelity rather than video perceptual quality.

E.2. Choice of FPS Control

  • MRoPE FPS modulation scales the temporal axis of unified 3D MRoPE by target FPS, while Text Control inserts target duration and FPS as natural-language text in the structured JSON caption.The four variants are Base (no control), Text Control only, MRoPE FPS Modulation only, and both mechanisms together.Source: §E.2 · pp. 107–108
  • The ablation trains four models for 130K iterations on 128 GPUs each, then evaluates 5-second, 480p samples across three seeds per prompt using videos from four FPS bands.Training uses 256p and 480p, clips of 0–200 frames, 25K sequence length, and joint image–video training; the evaluation set has approximately 100 videos per 10, 15, 24, and 30 FPS band, with ±2 FPS tolerance.Source: §E.2 · pp. 107–108
  • The Composite Score averages FPS-band Video Quality multiplied by Motion Fidelity; Motion Fidelity incorporates control robustness, measured as per-prompt variability in three-seed Dynamic Degree scores.Video Quality uses DOVER perceptual quality, and Dynamic Degree measures motion presence on a 0–1 scale.Source: §E.2 · p. 108
  • Text Control + MRoPE FPS Modulation produces the best composite score (+1.30 over Base), versus +1.12 for MRoPE FPS Modulation and +0.77 for Text Control.VQ stays within a 0.2-point window across all four settings, and the combined configuration is adopted for Cosmos 3 Generator.Source: Table 29 · p. 108, §E.2 · p. 108

E.3. Audio Data in Pre-Training · E.4. Synergy Between Action Modes · E.5. Video-Action Consistency

Joint video-audio pre-training avoids degrading video-generation quality and may modestly improve video metrics; a joint action model improves inverse-dynamics accuracy and policy coverage with a modest forward-dynamics tradeoff; predicted videos remain aligned with executed actions on held-out RoboLab.

E.3. Audio Data in Pre-Training

  • For the audio pre-training ablation, two Cosmos3-Nano variants start from the same pre-trained checkpoint and train on video-only or joint video-audio data for 20k iterations on 128 GPUs at 256p and 480p.The evaluation examines the effect of including audio during continued pre-training on video metrics.Source: §E.3 · p. 108
  • Continued pre-training without audio lowers both T2V and I2V scores, while joint video-audio pre-training does not degrade video generation and may modestly benefit video-centric metrics.The compared generator variants differ only in whether audio is used during training, with shared video data.Source: §E.3 · p. 108, Table 30 · p. 109

E.4. Synergy Between Action Modes

  • The PushT action-mode ablation compares three single-mode checkpoints trained for 2K steps each with one joint FD/ID/policy checkpoint trained for 6K steps, matching optimization steps per mode.It uses Cosmos3-Edge and reports FD PSNR, ID MSE, and policy coverage across 50 initializations with 10 rollouts per initialization.Source: §E.4 · p. 109, Table 31 · p. 109
  • ID MSE falls from 1.11×10−3 to 3.09×10−4 (72% relative reduction) and policy coverage rises from 74.1% to 77.3%, while FD PSNR drops from 27.13 to 26.22.The joint FD/ID/policy checkpoint gives the best policy coverage and ID accuracy under the same per-mode optimization budget, with a modest reconstruction-fidelity tradeoff.Source: Table 31 · p. 109, §E.4 · p. 109

E.5. Video-Action Consistency

  • Cosmos3-Nano-Policy-DROID, trained only on DROID, is evaluated on held-out RoboLab by comparing predicted video with simulator rollouts from the same initial state.Scores are aggregated across action chunks for left and wrist camera views; the wrist view is more challenging because the moving camera reveals new content.Source: §E.5 · p. 109, Figure 37 · p. 110
  • On held-out RoboLab, predicted-video PSNR reaches 23.19 dB for the left view and 17.33 dB for the wrist view, indicating consistency with simulator rollouts.For each predicted action chunk, the same chunk is executed in RoboLab and PSNR is computed between predicted video and the resulting rollout; the wrist view is harder because its camera moves with the end-effector.Source: §E.5 · p. 109, Figure 37 · p. 110

F. Cosmos-HumanEval Benchmark (Cosmos-HUE) … Training Recipe

Cosmos-HUE evaluates generated videos with atomic questions across four dimensions and reports separate text-to-video and image-to-video leaderboards. Its appendix specifies the conservative scoring and annotation procedures used for those evaluations.

F. Cosmos-HumanEval Benchmark (Cosmos-HUE)

  • HUE counts Yes as success and treats Unclear as No, leaving it in the denominator but out of the Yes numerator.Scores aggregate answered video-question observations equally, with dimension scores computed over questions in that dimension.Source: F. Cosmos-HumanEval Benchmark… · p. 110
  • Each video receives up to 16 questions rated by two annotators, with disagreements decided by a third quality-control reviewer.Question refinement targets low score variance across repeated runs and real-video ground-truth scores near 100%; the question bank remains under revision.Source: F. Cosmos-HumanEval Benchmark… · pp. 110–111
  • 100 prompts, five random-seed generations per prompt, and up to 20 questions per video yield up to 10,000 binary observations per T2V model checkpoint.The stated 95% confidence-interval width is around ±0.6 points for top-tier scores; Real video GT scores 93.6 on T2V and 94.4 on I2V prompts.Source: F. Cosmos-HumanEval Benchmark… · p. 111
  • Veo-3.1 leads T2V overall (91.3), while Cosmos3-Super is the best open-source model (89.3) and leads open-source on 9 of 12 axes.Cosmos3-Super also beats every generator on AV (87.7) and Physics (91.5); Real video GT scores 93.6 overall.Source: F. Cosmos-HumanEval Benchmark… · p. 111, Table 32 · p. 112
  • Veo-3.1 leads I2V overall (89.7), with Cosmos3-Super just 0.1 points behind (89.6).Cosmos3-Super leads Visual Integrity (94.2), Robotics (91.1), and Miscellaneous (94.8), and ties Veo-3.1 on Semantic Alignment (90.3).Source: F. Cosmos-HumanEval Benchmark… · p. 111, Table 33 · p. 112

Generator Data – Audio

Training Recipe

  • The supplied Training Recipe passages list contributor names rather than describing a training procedure.The listed names span six passages in this subsection.Source: Training Recipe · p. 115

Related papers

Loading 2606.02800v4…