PaperlayerView exact revision on arXiv

Source-linked AI summary

Cosmos World Foundation Model Platform for Physical AI

NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Yi Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, Artur Zolkowski

arXiv:2501.03575v3cs.CVcs.AIcs.LGcs.RO

TL;DR

Scaling Physical AI training data is challenging because it requires interleaved observations and actions. Cosmos pre-trains visual world models on large-scale video data and fine-tunes them for specific Physical AI environments. The authors demonstrate adaptability to 3D world navigation, robotic manipulation, and autonomous vehicle systems.

  • Problem

    Scaling Physical AI training data is challenging because it must contain interleaved observations and actions from interactions with the physical world.

    Source: §1 · p. 1

  • Method

    Cosmos pre-trains general-purpose visual world models on large-scale video data, then fine-tunes them on environment-specific datasets for specialized Physical AI setups.

    Source: §1 · p. 1

  • Results

    Pre-trained Cosmos world models demonstrate adaptability to 3D world navigation, robotic manipulation, and autonomous vehicle systems.

    Source: §9 · p. 58

  • Takeaways & Limitations

    World foundation models could support more cost-effective, time-efficient policy evaluation in otherwise unavailable environments, reserving physical resources for promising policies.

    Source: §2.1 · p. 4

    Cosmos models remain unreliable physical simulators, with shortcomings in object permanence, contact-rich dynamics, instruction following, and adherence to physical principles.

    Source: §9 · p. 58

Abstract

from arXiv · show

Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model. In this paper, we present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups. We position a world foundation model as a general-purpose world model that can be fine-tuned into customized world models for downstream applications. Our platform covers a video curation pipeline, pre-trained world foundation models, examples of post-training of pre-trained world foundation models, and video tokenizers. To help Physical AI builders solve the most critical problems of our society, we make Cosmos open-source and our models open-weight with permissive licenses available via https://github.com/nvidia-cosmos/cosmos-predict1.

1. Introduction

Cosmos provides a platform for building visual world models through large-scale video pre-training followed by environment-specific fine-tuning. It combines video curation, diffusion and autoregressive models, learned tokenizers, and input/output guardrails.

1. Introduction

  • Cosmos pre-trains generalist visual world models on diverse videos, then fine-tunes them for specialized Physical AI environments.World models offer safe digital twins for interaction, addressing observation–action data collection whose exploratory actions can damage physical systems. Post-training typically uses smaller prompt–video datasets, with actions, trajectories, or instructions as prompts.Source: §1 · p. 1, Figure 2 · p. 3
  • About 100M video clips lasting 2 to 60 seconds are extracted from a 20M-hour video collection, selecting rich dynamics and high visual quality.A visual language model captions each clip per 256 frames. GPU H.264 hardware handles decoding and transcoding, while Ray-based orchestration coordinates models with different throughputs (Moritz et al., 2017).Source: §1 · p. 1
  • Cosmos pairs transformer-based diffusion and autoregressive world models with learned video tokenizers using continuous vectors and discrete integers, respectively.Diffusion generates videos through gradual denoising, while autoregression generates pieces conditioned on previous outputs. Attention-based encoder–decoder tokenizers compress videos while preserving content, limiting token counts that drive training complexity.Source: §1 · pp. 1, 3
  • Post-training adapts Cosmos world models for camera-controlled navigation, robotic action-conditioned prediction, and autonomous-driving tasks.Camera-pose conditioning of the diffusion model enables viewpoint exploration of generated worlds. Fine-tuning on robotic video–action sequences improves prediction of future world states given robot actions.Source: §1 · pp. 3–4
  • A two-stage guardrail system protects developers by blocking harmful inputs with pre-Guard and harmful outputs with post-Guard.Source: §1 · p. 4
  • Cosmos makes pre-trained world models and tokenizers available under the NVIDIA Open Model License, while acknowledging that world modeling remains unsolved.The platform targets Physical AI builders, and further research is needed despite the paper’s model-design improvements.Source: §1 · p. 4

2. World Foundation Model Platform

Cosmos combines video curation, causal tokenization, pre-trained world models, post-training examples, and guardrails to support predicting future observations from past video and perturbations. Proposed policy-development, planning, and synthetic-data applications are not empirically evaluated here.

2. World Foundation Model Platform

  • A world foundation model predicts future observations from past RGB video and a current perturbation.Perturbations can include Physical AI actions, random disturbances, or textual descriptions.Source: §2 · p. 4, Figure 3 · p. 4

2.1. Future Cosmos

  • WFMs could support cheaper, faster policy evaluation, initialize policies to address data scarcity, and provide reward-model feedback for reinforcement learning.Digital copies of Physical AI systems could test policies in otherwise unavailable environments, helping developers reject incapable policies before committing physical resources.Source: §2.1 · p. 4
  • Proposed planning, model-predictive control, and synthetic-data applications are not empirically evaluated in this paper.Simulated action sequences can be scored to choose plans or receding-horizon controls, with world-model accuracy upper-bounding performance. Conditioning synthetic-data generation on depth or semantic maps could support Sim2Real.Source: §2.1 · pp. 4–5

2.2. Current Cosmos

  • The Cosmos platform integrates video curation, tokenization, WFM pre-training, post-training examples, and guardrails.Camera-pose fine-tuning enables free navigation in generated worlds, alongside fine-tuning examples for humanoid and autonomous-driving tasks. Guardrails block harmful inputs and outputs.Source: §2.2 · pp. 5–6, Figure 4 · p. 5
  • The scalable video curator splits videos into shots, filters for quality and dynamic information, adds VLM annotations, and semantically deduplicates the dataset.The resulting training dataset is designed to be diverse but compact.Source: §2.2 · p. 5
  • Causal video tokenizers offer different compression ratios, enable joint image–video training, and align with Physical AI systems operating in a causal world.Current-frame tokens never depend on future observations; single-image inputs let the same tokenizers leverage image datasets containing rich appearance information and greater diversity.Source: §2.2 · p. 5
  • Transformer-based diffusion and autoregressive WFM families generate future video from past video and text prompts across different model capacities.Diffusion pre-training proceeds from Text2World to Video2World; autoregressive pre-training proceeds from next-token foresight generation to text-conditioned Video2World. A fine-tuned diffusion decoder enhances autoregressive outputs, while an LLM-based prompt upsampler supports better control.Source: §2.2 · pp. 5–6

3. Data Curation

Cosmos curates video and image data through splitting, filtering, annotation, semantic deduplication, and sharding to produce training datasets for tokenizers and world foundation models. GPU acceleration and streaming infrastructure support processing at scale.

3.1. Dataset / 3.6. Sharding

  • 20M hours of raw video, approximately, span 720p–4k resolutions and combine proprietary datasets with publicly available open-domain Internet videos.Coverage includes driving, manipulation, human activity, navigation, first-person views, nature dynamics, camera movements, synthetic scenes, and other videos; the modular pipeline also processes images. Processed clips are packaged into webdatasets by resolution, aspect ratio, and length, with stricter filtering producing higher-quality fine-tuning datasets.Source: §3.1 · pp. 6–7, §3.6 · p. 10

3.2.1. Shot Detection

  • Neural shot detectors outperform heuristic methods on ShotBench, with TransNetV2 preferred over AutoShot for more challenging shot changes and GPU-accelerated splitting.ShotBench combines existing datasets to evaluate clean shot cuts in heavily edited videos. Splitting produces clips without shot changes, discards clips shorter than 2s, and subdivides clips longer than 60s.Source: §3.2.1 · pp. 7–8, Table 1 · p. 8

3.2.2. Transcoding / 3.7. Infrastructure

  • ∼6.5× higher transcoding throughput results from combining hardware and software optimizations.Clips are standardized into high-quality MP4 using h264_nvenc; PyNvideoCodec replaces ffmpeg for video transcoding, while ffmpeg handles audio remixing. Infrastructure uses AnyScale Ray (Moritz et al., 2017), decouples transfers from computation, and extends Fragmentation Gradient Descent to balance streaming stages across complementary hardware resources.Source: Table 2 · p. 8, §3.2.2 · p. 8, §3.7 · p. 11

3.3.1. Motion Filtering / 3.3.2. Visual Quality Filtering

  • Motion and visual-quality filters remove static, erratically moving, distorted, and low-aesthetic clips while labeling camera-motion types.A lightweight ViT classifier performs best with NVIDIA TensorRT-accelerated optical flow among the tested motion inputs. Quality filtering removes the bottom 15% of perceptual scores from a DOVER-based model (Wu et al., 2023) and applies an aesthetic threshold of 3.5 using an image model.Source: §3.3.1 · p. 9, §3.3.2 · p. 9

3.3.3. Text Overlay Filtering

  • Text-overlay filtering targets excessive post-processed text rather than text originally present in scenes, such as street names.An MLP binary classifier uses embeddings from InternVideo2 and training labels generated by a proprietary VLM; the authors report high validation prediction accuracy.Source: §3.3.3 · p. 9

3.3.4. Video Type Filtering

  • Video-type filtering excludes unwanted content and rebalances the training distribution toward categories more relevant to world foundation models.A proprietary VLM labels eight uniformly sampled frames per clip to create taxonomy-specific training and evaluation data for an MLP on reused InternVideo2 embeddings. Filtering excludes abstract patterns, game footage, and animation, upsamples human actions and interactions, and downsamples less relevant nature or landscape videos.Source: §3.3.4 · p. 9

3.4. Annotation

  • 10× higher captioning throughput than a PyTorch half-precision baseline comes from an FP8-quantized TensorRT-LLM engine for the internal 13B-parameter VILA model.VILA (Lin et al., 2024; Xue et al., 2024) produced more accurate descriptions than the tested alternatives in a small-scale human evaluation. The video-captioning model receives eight uniformly sampled frames and generates consistent, factual descriptions averaging 559 characters or 97 words for training supervision and conditioning.Source: Table 3 · p. 10, §3.4 · p. 10

3.5. Deduplication

  • Semantic deduplication removes near-duplicate videos, while reused embeddings and clusters enable dataset-wide search with free-form text and video queries.Following SemDeDup (Abbas et al., 2023) and DataComp (Gadre et al., 2024), GPU k-means groups InternVideo2 embeddings into 10,000 clusters; within-cluster comparisons retain the highest-resolution duplicate and remove about 30% of training data. Blockwise distance computation limits GPU memory use, while search supports data debugging and identifying gaps between pre-training data and downstream applications.Source: §3.5 · p. 10

4. Tokenizer

Cosmos Tokenizer provides continuous and discrete image/video tokenizers using a temporally causal wavelet architecture. It achieves state-of-the-art reconstruction results at high compression rates while running faster than prior tokenizers.

4. Tokenizer

  • Cosmos Tokenizer offers continuous and discrete image/video tokenizers with configurable compression, flexible aspect ratios, and inference beyond training lengths.Compression balances reduced storage and computation against lost visual detail; training uses high-resolution images and long videos without category or aspect-ratio restrictions. Reconstruction quality remains strong at resolutions above those used for training.Source: §4 · pp. 11–12, §4.2 · p. 16

4.1. Architecture

  • A temporally causal wavelet encoder-decoder removes pixel redundancy before semantic compression, processing only current and past frames for downstream Physical AI applications.A two-level wavelet transform downsamples space and time fourfold before factorized convolutions and causal attention; the decoder mirrors the encoder with upsampling. Continuous latents use a vanilla autoencoder with dimension 16; discrete latents use Finite-Scalar-Quantization (Mentzer et al., 2023), dimension 6, levels (8, 8, 8, 5, 5, 5), and vocabulary size 64,000.Source: §4.1 · pp. 13–14

4.2. Training Strategy

  • Two-stage joint image/video training supervises only decoder outputs, avoiding latent commitment and KL losses while optimizing reconstruction, temporal smoothness, and visual detail.Alternating image/video minibatches initially use L1 reconstruction loss and VGG-19 perceptual loss (Simonyan and Zisserman, 2014). The second stage uses optical-flow loss for temporal smoothness and Gram-matrix loss for sharpness, with adversarial fine-tuning improving details particularly at high compression.Source: §4.2 · pp. 14–15

4.3. Results

  • 4 × 8 × 8 compression gives continuous and discrete Cosmos video tokenizers state-of-the-art results across all evaluated metrics on DAVIS and TokenBench.At 2× and 8× higher compression—8 × 8 × 8 and 8 × 16 × 16—reconstruction quality still exceeds prior art.Source: Table 5 · p. 16, Table 6 · p. 16, §4.3 · p. 17
  • 500 videos form TokenBench, a released benchmark standardizing high-resolution, long-duration tokenizer evaluation across robotic manipulation, driving, egocentric, and web domains.Videos retain their first 10 seconds and have their shorter side resized to 1080 pixels; additional video evaluation uses DAVIS at 1080p. Image evaluation uses MS-COCO 2017 (Lin et al., 2014) and ImageNet-1K, with PSNR, SSIM, image rFID (Heusel et al., 2017), and video rFVD as evaluation metrics.Source: §4.3 · pp. 16–17
  • 8 × 8 compression gives continuous and discrete Cosmos image tokenizers state-of-the-art results; 16 × 16 often matches or exceeds prior art at 8 × 8.The larger setting provides 4× greater compression.Source: Table 7 · p. 17, Table 8 · p. 17, §4.3 · pp. 17–18
  • 2×–12× faster encoding and decoding than prior state-of-the-art tokenizers accompanies the smallest model size for both Cosmos image and video tokenizers.Measurements use one A100 80GB GPU. One-shot video encoding supports up to 8 seconds at 1080p or 10 seconds at 720p without exhausting memory.Source: §4 · p. 13, Table 9 · p. 17, §4.3 · p. 18

5. World Foundation Model Pre-training

Cosmos pre-trains diffusion and autoregressive world foundation models to capture real-world physics and natural behaviors. These families support video generation and prediction, with prompt upsampling and a diffusion decoder addressing prompt-distribution gaps and token-compression distortions.

World Foundation Model Pre-training

  • 10,000 NVIDIA H100 GPUs supported training all reported WFMs over three months, using diffusion denoising and autoregressive next-token prediction.Both paradigms decompose generation into sequences of easier subproblems.Source: §5 · p. 18
  • 7B and 14B diffusion Text2World models generate videos from text; fine-tuning adds video observations for Video2World future prediction.Video2World conditions predictions on the current video observation and a text-described perturbation. These latent diffusion models consume continuous tokens produced by Cosmos-Tokenize1-CV8×8×8-720p.Source: §5 · p. 18
  • Prompt upsampling addresses the distribution gap between human prompts and VLM-generated video descriptions used to train diffusion WFMs.Cosmos-UpsamplePrompt1-12B-Text2World is based on Mistral-NeMo-12B-Instruct and converts human prompts into descriptions preferred by the diffusion models.Source: §5 · p. 18
  • 4B and 12B autoregressive base models predict future video from current observations, while text-conditioned variants incorporate T5 embeddings through transformer cross-attention.The base models are Llama3-style GPT models trained from scratch without language understanding. They use Cosmos-Tokenize1-DV8×16×16-720p; a fine-tuned 7B Text2World diffusion decoder maps discrete DV8×16×16 tokens to continuous CV8×8×8 tokens to address compression-induced distortions.Source: §5 · pp. 18–19

5.1. Diffusion-based World Foundation Model

Cosmos diffusion world foundation models denoise compact video latents and support text-, image-, and video-conditioned generation. Qualitative examples show finer details and more intricate motion from 14B Text2World models, and richer scenes and more stable motion from 14B Video2World models, compared with their 7B counterparts.

5.1.1. Formulation

  • EDM-based training (Karras et al., 2022, 2024) combines denoising score matching and preconditioning with learned uncertainty weighting to balance objectives across noise levels.A small MLP predicts continuous uncertainty at each noise level, downweighting uncertain denoising tasks while penalizing uncertainty itself. This addresses deterioration of the initial balance between noise-level contributions during training.Source: §5.1.1 · p. 19

5.1.2. Architecture

  • 5,000 training steps yield reasonable performance during resolution or video-length changes using NTK-RoPE (Peng and Quesnelle, 2023); learned absolute positions further reduce loss and morphing artifacts.The denoiser converts latent videos into sequences using nonoverlapping 1×2×2 patches. Its three-axis RoPE supports arbitrary sizes, aspect ratios, and lengths, with FPS-scaled temporal frequencies compatible with joint image/video training.Source: §5.1.2 · pp. 20–21
  • 36% fewer parameters—11B reduced to 7B—are achieved through low-rank adaptive-normalization projections, with performance parity across all evaluation metrics for Cosmos-Predict1-7B.The denoiser adapts DiT (Peebles and Xie, 2023), interleaving spatiotemporal self-attention, cross-attention using T5-XXL embeddings, and feedforward layers. Query/key RMSNorm addresses attention-logit instability, while LoRA (Hu et al., 2022) compresses adaptive-normalization projections.Source: §5.1.2 · pp. 20–21

5.1.3. Training Strategy

  • Joint image/video training aligns latent distributions, standardizes frames and channels, and scales video noise by the square root of frame count to address slower video convergence.Standardization maintains consistent signal-to-noise ratios across latent scales, supporting adaptation when the tokenizer changes. Training progresses from 57-frame, 512-pixel inputs to 121-frame, 720-pixel inputs and high-quality fine-tuning; five aspect-ratio buckets use longest-side resizing, reflection padding, and padding masks.Source: §5.1.3 · pp. 21–22
  • Video2World concatenates noise-augmented conditioning frames temporally, supplies a binary conditioning mask, and trains only on generated frames while varying conditioning length.Text2World retains text embeddings during training, uses negative prompts at inference, and requires higher guidance for satisfactory videos than for high-quality images. BF16 forward/backward computation, FP32 updates, tenfold denoising-loss scaling, and lower AdamW betas and eps coefficients improve training stability.Source: §5.1.3 · pp. 22–23

5.1.4. Scaling Up

  • FSDP and context parallelism achieve comparable Model FLOPs Utilization to HunyuanVideo (Kong et al., 2024) and MovieGen (Polyak et al., 2024), without tensor/sequence parallelism.For Cosmos-Predict1-14B, 64-way FSDP reduces estimated parameter/gradient/optimizer memory from 280 GB to approximately 4 GB per GPU, while eight-way context parallelism reduces activations from 310 GB to approximately 40 GB; these estimates exclude additional overhead. Context parallelism overlaps peer-to-peer key/value transfers with attention computation and is disabled for image iterations and cross-attention; quadratic self-attention costs also make the usual LLM FLOPs approximation inaccurate here.Source: §5.1.4 · pp. 24–25

5.1.6. Results

  • Cosmos-Predict1’s 14B models outperform their 7B counterparts qualitatively: Text2World improves visual detail and intricate motion, while Video2World improves scene richness and motion stability.Both sizes produce visually high-quality videos. Video2World supports image/video conditioning and autoregressive extension, including examples initialized from one image and continued using the previous nine frames.Source: Figure 12 · p. 25, Figure 13 · p. 26, §5.1.6 · pp. 26–27

5.1.5. Prompt Upsampler

  • Text2World fine-tunes Mistral-NeMo-12B-Instruct, whereas Video2World uses zero-shot Pixtral-12B (Agrawal et al., 2024) to enrich user prompts.The upsamplers target faithful, visually detailed descriptions matching training-prompt length, structure, and style; Video2World additionally considers the visual conditions. Cosmos-UpsamplePrompt1-12B-Text2World learns from VLM-generated short captions paired with original detailed prompts and videos, preserving authentic content and short-to-long prompt fidelity.Source: §5.1.5 · pp. 25–26

5.2. Autoregressive-based World Foundation Model

This section presents Cosmos autoregressive world foundation models trained to predict discrete video tokens, covering their architecture, staged training, scalable execution, inference acceleration, and diffusion decoding. It examines qualitative improvements and remaining failures, with larger models showing greater robustness under single-frame conditioning and all models remaining stable under 9-frame video conditioning.

5.2.1. Architecture

  • 3 × 10^-4 for z-loss (de Brébisson and Vincent, 2016) stabilizes large-scale training without adversely affecting model performance.Architectural additions include text cross-attention, 3D positional embeddings, and QK-Normalization (Wortsman et al., 2023). A 64,000-token FSQ (Mentzer et al., 2023) vocabulary accompanies relative 3D RoPE and sinusoidal absolute embeddings, whose combination reduces loss and morphing; temporal-only YaRN (Peng et al., 2023) extends context.Source: §5.2.1 · pp. 27–29

5.2.2. Scaling Up

  • 192 GB is required for the 12B model’s parameters, gradients, and optimizer states, motivating tensor and sequence parallelism across GPUs.Tensor parallelism (Shoeybi et al., 2019) partitions linear-layer weights, while sequence parallelism (Korthikanti et al., 2023) partitions sequence-independent activations, each reducing relevant per-GPU storage to 1/TP_SIZE. The model omits MQA and GQA; context and pipeline parallelism remain future work.Source: §5.2.2 · p. 29

5.2.3. Training Strategy

  • Training progresses from first-frame-conditioned prediction with 17-frame context to 34-frame context, then adds text conditioning through newly initialized cross-attention layers.Stage 2 jointly trains on images and videos; 4B and 12B base predictors become 5B and 13B Video2World models through additional text-conditioned training. Models train at 640 × 1024, followed by 30,000 cooling-down iterations on high-quality image-video pairs with learning rate decaying linearly to zero.Source: §5.2.3 · pp. 29–30

5.2.4. Inference Optimization Towards Real-Time Generation

  • 2.0× and 3.2× token-throughput speedups are achieved with Medusa for the 4B and 5B models, respectively; nine heads provide the best trade-off.Alongside KV caching, tensor parallelism, and torch.compile, Medusa (Cai et al., 2024) predicts multiple tokens with extra heads and verifies them through rejection sampling. On eight H100 GPUs and 50 unseen videos, forward passes decrease by up to 4.6× for 4B and 6.1× for 5B; selective fine-tuning of the last two transformer layers and unembedding layer performs best.Source: §5.2.4 · pp. 30–31, Table 15 · p. 31
  • 10 FPS real-time generation is achieved with low-resolution Cosmos-Predict1-4B adaptation at 320 × 512 on eight H100 GPUs.The tokenizer and WFM are fine-tuned on target-domain low-resolution videos before adding Medusa heads; benchmarking uses BF16 and torch.compile’s max-autotune mode. Separately, Medusa consistently accelerates both 4B and 5B models across GPU configurations at 640 × 1024.Source: §5.2.4 · pp. 31–32, Table 17 · p. 32, Table 16 · p. 32

5.2.5. Diffusion Decoder

  • The diffusion decoder addresses compression-induced blur by fine-tuning Cosmos-Predict1-7B-Text2Video to reconstruct continuous video tokens conditioned on discrete tokens.Training pairs discrete 8 × 16 × 16 tokens with less-compressed continuous 8 × 8 × 8 tokens; learned 16-dimensional discrete embeddings are spatially upsampled and concatenated with noisy continuous inputs for denoising. At inference, the conditional denoiser generates continuous tokens, which Cosmos-Tokenize1-CV8×8×8-720p decodes into RGB video.Source: §5.2.5 · pp. 32–33

5.2.6. Results

  • Cosmos-Predict1-12B improves motion and sharpness over Cosmos-Predict1-4B, while Cosmos-Predict1-13B-Video2World improves motion over Cosmos-Predict1-5B-Video2World.Diffusion decoding enhances details while preserving content. Upsampled prompts do not improve autoregressive Text2World outputs; the authors hypothesize that predominantly video-only pretraining insufficiently encourages text use.Source: §5.2.6 · pp. 33–34, Figure 17 · p. 34, Figure 18 · p. 35

5.2.7. Limitations

  • Generated videos can contain objects unexpectedly appearing from below.The authors investigate failures using 100 Physical AI inputs, generating videos with single-frame and nine-frame conditioning and manually inspecting the outputs; this supplied part reports no failure rates.Source: §5.2.7 · pp. 34–35
  • Below 2% failure rates are observed for all models with 9-frame video conditioning.Generation is stable across all evaluated models under this conditioning.Source: §5.2.7 · p. 35
  • Cosmos-Predict1-4B and Cosmos-Predict1-5B-Video2World exhibit higher corruption rates under single-frame conditioning than the larger models.Source: §5.2.7 · p. 35
  • Cosmos-Predict1-12B and Cosmos-Predict1-13B-Video2World are more robust under single-frame conditioning than the smaller models.Source: §5.2.7 · p. 35

5.3. Evaluation

Evaluation tests generated videos for 3D consistency and physics alignment, leaving broader assessment to future work. Cosmos WFMs improve geometric and novel-view consistency over the baseline, but physics adherence remains limited despite benefits from additional conditioning frames.

5.3.1. 3D Consistency

  • Geometric consistency uses square-root, diagonal-normalized keypoint errors and fundamental matrices estimated through OpenCV’s eight-point RANSAC.SuperPoint (DeTone et al., 2018) and LightGlue (Lindenberger et al., 2023) detect and match keypoints. The square-root error is expressed in pixel units and normalized by frame diagonal relative to a 960 × 540 canvas.Source: §5.3.1 · p. 36
  • Novel-view consistency evaluates every eighth held-out frame using 3D Gaussian splatting (Kerbl et al., 2023), measuring PSNR, SSIM, and LPIPS.The model is fitted to the remaining frames using default Nerfstudio settings.Source: §5.3.1 · p. 36
  • Cosmos WFMs outperform the baseline in geometric and novel-view consistency, with higher camera-pose estimation success and better held-out-view quality.Camera-pose estimation success reaches real-world-video levels. Among videos with successfully estimated poses, synthesized held-out views improve across all image-synthesis metrics.Source: §5.3.1 · pp. 36–37, Table 19 · p. 37

5.3.2. Physics Alignment

  • 800 1080p videos of 100 frames benchmark Newtonian physics and rigid-body dynamics across eight simulated scenarios.PhysX (NVIDIA, 2024) and Isaac Sim (NVIDIA, 2024) generate four static-camera views with randomized objects and backgrounds; all objects are visible initially. Scenarios cover free fall, tilted and U-shaped slopes, stable and unstable stacks, dominoes, seesaws, and gyroscopes.Source: §5.3.2 · p. 37
  • Diffusion WFMs outperform autoregressive WFMs in nine-frame pixel-level prediction; more conditioning frames improve kinematics, but larger models show no physics-alignment advantage.Additional conditioning frames help infer speed and acceleration. Higher visual quality in larger models does not establish better physics alignment.Source: Table 20 · p. 39, §5.3.2 · pp. 39–40
  • Physics evaluation conditions rollouts on one or nine frames, optionally adds kinematic text prompts, and compares predictions with simulated references using pixel-, feature-, and object-level metrics.Metrics include PSNR, SSIM, DreamSim (Fu et al., 2023), and object-mask IoU using SAMURAI (Yang et al., 2024) tracking to reduce background and visual-quality confounders. Evaluation spans 33 frames and averages across frames, videos, and four rollout seeds; PSNR and SSIM exclude conditioning frames.Source: §5.3.2 · p. 39, Table 20 · p. 39
  • All WFMs struggle with physics adherence, exhibiting object impermanence, deformation, implausible kinematics, and gravity violations.Stronger physics alignment requires better data curation and model design. Planned evaluations introduce more complex scenarios, greater photorealism to narrow the sim-to-real gap, and refined metrics.Source: §5.3.2 · pp. 37, 40

6. Post-trained World Foundation Model

Cosmos WFMs are fine-tuned for camera-controlled world generation, action-controlled robotic manipulation, and multi-view support for training autonomous driving agents. These are sample applications rather than complete or production systems; developers must fine-tune models on datasets specific to their Physical AI setups.

6. Post-trained World Foundation Model

  • Camera-control post-training enables 3D navigable visual world generation.Source: §6 · p. 40
  • Action-control post-training is demonstrated on two robotic setups for two different robotic manipulation tasks.Source: §6 · p. 40
  • Multi-view post-training supports training autonomous driving agents.Source: §6 · p. 40
  • Post-trained models carry the “-Sample” suffix because they demonstrate applications, not complete systems or production-ready models.Developers need to fine-tune the WFMs on custom datasets for their Physical AI setups and target applications.Source: §6 · p. 40

6.1. Post-training WFM for Camera Control

Camera-pose conditioning adapts Cosmos-Predict1-7B-Video2World to generate temporally coherent, 3D-consistent videos from a single image and specified camera trajectories. The post-trained model improves visual quality and camera control over CamCo (Xu et al., 2024), while generalizing across a training–test distribution shift.

6.1. Post-training WFM for Camera Control

  • Camera-pose conditioning turns Cosmos-Predict1-7B-Video2World into Cosmos-Predict1-7B-Video2World-Sample-CameraCond, generating temporally coherent, 3D-consistent videos from one reference image and specified camera trajectories.Perspective changes follow the underlying 3D structure of the scene.Source: §6.1 · p. 40

6.1.1. Dataset

  • 256-frame clips from DL3DV-10K (Ling et al., 2024) receive dense camera-pose annotations through GLOMAP (Pan et al., 2025) structure-from-motion.The first frame's pose is the identity transform, with subsequent poses expressed relative to it. A proprietary VLM supplies text prompts describing the videos as static scenes.Source: §6.1.1 · pp. 40–41

6.1.2. Fine-tuning

  • 8× temporal compression is accommodated by concatenating each latent representation with the fourth frame's Plücker embedding from its eight-frame group.Plücker embeddings match the latent embeddings' spatial dimensions, and camera poses are relative to the initial frame.Source: §6.1.2 · p. 41

6.1.3. Evaluation

  • Cosmos-Predict1-7B-Video2World-Sample-CameraCond outperforms CamCo (Xu et al., 2024) in FID/FVD, camera-pose estimation success, and agreement with input camera trajectories.Testing uses 500 RealEstate10K samples, measuring rotation/translation errors after camera-pose re-estimation and Procrustes alignment following Lin et al. (2021). CamCo's 14-frame outputs cover the same 57-frame trajectories using 4× temporal downsampling.Source: §6.1.3 · pp. 41, 43, Table 22 · p. 43
  • Despite DL3DV-10K-to-RealEstate10K distribution shift, Cosmos generates 3D-structured content beyond the reference image and generalizes to unseen camera trajectories, unlike struggling CamCo (Xu et al., 2024).Qualitative comparisons show Cosmos maintaining video quality and alignment with camera-control inputs, whereas CamCo can produce inaccurate trajectories or images with un-estimatable camera poses.Source: Figure 21 · p. 42, §6.1.3 · p. 43
  • Joystick-like controls navigate generated worlds, while different random seeds produce distinct, spatially and temporally coherent worlds from identical image-and-camera inputs.Controls include moving forward, moving backward, rotating left, and rotating right; these capabilities could support Physical AI prediction of alternative futures.Source: §6.1.3 · p. 43

6.2. Post-training WFM for Robotic Manipulation

Fine-tuned Cosmos WFMs support instruction-based video prediction and action-based next-frame generation for robotic manipulation. Both model sizes outperform VideoLDM-Instruction in human evaluation and IRASim-Action in action-based prediction.

6.2. Post-training WFM for Robotic Manipulation

  • Post-training adapts pre-trained WFMs to predict instruction-following videos from a robot frame and text, or next frames from a robot frame and action vector.Given an action sequence, the next-frame model can run autoregressively to predict a video of the robot executing those actions.Source: §6.2 · p. 43

6.2.1. Datasets

  • Approximately 200 hours of egocentric robot video supply roughly 12,000 instruction-labeled episodes for the internal Cosmos-1X dataset.Episodes last 1–9 seconds, with videos captured at 30 FPS and 512 × 512 resolution. Each episode receives a one-sentence instruction subsequently upsampled using a proprietary VLM.Source: §6.2.1 · p. 43
  • Approximately 20,000 Bridge (Ebert et al., 2022) episodes provide third-person kitchen robot-arm videos with seven-dimensional gripper-space actions.Videos have 320 × 256 resolution at 5 FPS, and actions encode translation, rotation, and gripper change as in OpenVLA (Kim et al., 2024). The comparison configuration follows prior work (Zhu et al., 2024).Source: §6.2.1 · pp. 43, 46

6.2.2. Fine-tuning

  • Fine-tuning conditions both model sizes on T5 instruction embeddings through cross-attention, while introducing separate action-conditioning modules.For Cosmos-Predict1-5B-Video2World-Sample-ActionCond, an MLP embeds the action vector for cross-attention. For Cosmos-Predict1-7B-Video2World-Sample-ActionCond, an MLP embeds the action vector and adds it to the DiT timestamp embedding.Source: §6.2.2 · p. 46

6.2.3. Evaluation

  • 78.3% overall preference went to Cosmos-Predict1-7B-Video2World-Sample-Instruction versus 13.0% for VideoLDM-Instruction in human evaluation on Cosmos-1X.Both Cosmos instruction models, including Cosmos-Predict1-5B-Video2World-Sample-Instruction, were preferred across instruction following, object permanence, verity, and overall planning reasonableness. Ten evaluators compared anonymous video pairs across 23 test episodes.Source: Figure 24 · p. 46, §6.2.3 · p. 47
  • Both Cosmos-Predict1-5B-Video2World-Sample-ActionCond and Cosmos-Predict1-7B-Video2World-Sample-ActionCond outperform IRASim-Action on Bridge action-based next-frame prediction.IRASim-Action was fine-tuned from IRASim (Zhu et al., 2024); evaluation compared autoregressively generated videos against ground truth over 100 randomly selected official Bridge test episodes. Metrics were PSNR, SSIM, Latent L2 (Zhu et al., 2024), and FVD.Source: Table 23 · p. 48, §6.2.3 · p. 48

6.3. Post-training WFM for Autonomous Driving

Fine-tuning Cosmos produces six-camera driving world models supporting text, trajectory, and video conditioning. These models outperform VideoLDM-MultiView in generation quality and multi-view consistency, while trajectory-conditioned generation closely follows prescribed paths.

6.3.1. Dataset

  • Approximately 3.6 million 20-second clips comprise the Real Driving Scene dataset, providing approximately 20,000 hours of synchronized six-camera video and ego-motion information.Selection targets distributions of traffic density, weather, illumination, speed, behavior, and road type, with additional mining for rare road structures. Each camera view receives a separate caption identifying its orientation.Source: §6.3.1 · p. 48

6.3.2. Fine-tuning

  • Fine-tuning Cosmos-Predict1-7B-Text2World produces simultaneous six-camera driving videos using view-independent positional embeddings, global view embeddings, and view-dependent textual cross-attention.Self-attention spans all six views during denoising, but each view's cross-attention accesses only its own textual description.Source: §6.3 · p. 48, §6.3.2 · p. 49
  • 64 three-dimensional translation points, spaced 0.1 seconds apart, encode the trajectory condition for controlling generated driving motion.Cosmos-Predict1-7B-Text2World-Sample-MultiView-TrajectoryCond combines text-described environments with future trajectory paths through an embedded denoiser condition. Finer per-interval action control remains future work.Source: §6.3.2 · p. 49

6.3.3. Evaluation

  • 201-frame six-view videos result from extending 57-frame text-conditioned generations with Cosmos-Predict1-7B-Video2World-Sample-MultiView; qualitative examples also demonstrate rare and out-of-domain scenes.All three model variants output six views of 57 frames at 848 × 480 resolution, with video conditioning incorporating previous frames for continuation. Generalization examples include driving toward an ice castle and driving on a river.Source: §6.3.3 · p. 49, §6.3.2 · p. 49, Figure 28 · p. 51
  • 1000 samples evaluate generation quality, while 800 samples—200 each for forward motion, left turns, right turns, and other trajectories—evaluate consistency.Quality scores average across views using 16 frames per video; Temporal Sampson Error measures temporal consistency, while Cross-view Sampson Error measures cross-view consistency. Human review of 20 randomly sampled eight-second videos found no physically impossible scenarios among 157 tracked objects.Source: §6.3.3 · pp. 50, 51, 53
  • Cosmos-Predict1-7B-Text2World-Sample-MultiView and Cosmos-Predict1-7B-Text2World-Sample-MultiView-TrajectoryCond outperform VideoLDM-MultiView in FID, FVD, and multi-view geometric consistency, with the trajectory-conditioned model achieving the best consistency.Explicit 3D trajectory guidance further improves geometric consistency.Source: §6.3.3 · pp. 50–52, Table 24 · p. 52
  • <7cm separates Cosmos post-trained world models from the ground-truth oracle in Trajectory Following Error (TFE), indicating accurate adherence to prescribed paths.Trajectory consistency also exceeds VideoLDM-MultiView and approaches real-world videos. Trajectory Agreement Error compares front-camera trajectories estimated from two camera pairings, while TFE compares estimated trajectories against the supplied condition.Source: Table 25 · p. 52, §6.3.3 · pp. 52–53

7. Guardrails

Cosmos uses two-stage guardrails: Pre-Guard blocks harmful prompts, while Post-Guard filters unsafe generated videos and obscures faces. Dedicated red teaming probes the system and individual guardrail components.

7.1. Pre-Guard

7.1.1. Keyword Blocking

  • Keyword blocking rejects an entire prompt if any lemmatized input word matches profanity in a hard-coded blocklist.WordNetLemmatizer uses an English lexical database to reduce word variants to their roots before comparison.Source: §7.1.1 · p. 54

7.1.2. Aegis Guardrail

  • Aegis (Ghosh et al., 2024) uses its stricter defensive variant to reject unsafe prompts, preventing video generation and displaying an error.It is fine-tuned from Llama-Guard (Inan et al., 2023) using NVIDIA’s Aegis Content Safety Dataset, which covers 13 critical safety-risk categories. Prompt filtering rejects violence, sexual content, criminal planning, weapons, substance abuse, suicide, child sexual abuse material, hatred, harassment, threat, and profanity; prompts outside these categories are considered safe for this filter.Source: §7.1.2 · p. 54

7.2. Post-Guard

  • Post-Guard combines a video content safety filter with a face blur filter to screen generated visual outputs.The safety filter blocks harmful outputs, while the face filter obscures detected faces.Source: §7 · p. 53, §7.2 · p. 54, Figure 30 · p. 54

7.2.1. Video Content Safety Filter

  • The video safety filter classifies individual frames and flags the entire video as unsafe if any frame is classified as unsafe.Training balances data to reduce false positives and false negatives, combining VLM-labeled dataset frames, synthetic corner cases, and human gold-standard annotations. A simple MLP classifies SigLIP frame embeddings during training and inference.Source: §7.2.1 · pp. 54–55

7.2.2. Face Blur Filter

  • The face blur filter pixelates high-confidence facial regions larger than 20 × 20 pixels detected by RetinaFace (Deng et al., 2020).Pixelation obscures faces while preserving overall scene composition for Physical AI applications.Source: §7.2 · p. 54, §7.2.2 · p. 55

7.3. Red Team Effort

  • Over 10,000 distinct prompt-video pairs had been tested and annotated by the red team at publication, covering diverse unsafe content.Standard and adversarial examples receive expert harm ratings on a 1–5 scale, with unsafe content localized by start and end frames. Targeted tests also probe each guardrail component independently to identify weaknesses and improve edge-case performance.Source: §7.3 · p. 55

8. Related Work

Cosmos connects neural world modeling and video generation with camera control, robotic control, and driving simulation. It provides both diffusion-based and autoregressive world foundation models, alongside tokenizers built on prior methods.

World models

  • Ha and Schmidhuber (2018) introduced neural world models for future-state prediction, whereas traditional system-specific physical models restrict generalization and reuse.Traditional planning and control employ physics laws and system identification, typically within low-dimensional state spaces. Deep learning enables world models learned directly from visual observations.Source: §8 · p. 55
  • World model architectures span recurrent latent dynamics, autoregressive or diffusion-based video generation, and generative simulation combining generative AI with physical simulators.Applications include verification (Hu et al., 2023), model predictive control (Bar et al., 2024; Hansen et al., 2024), and model-based reinforcement learning (Alonso et al., 2024; Ding et al., 2024; Robine et al., 2023; Yang et al., 2023). Demonstrated domains include games, real-world robots, and autonomous driving.Source: §8 · pp. 55–56

Video generative models

  • Cosmos provides diffusion-based and autoregressive world foundation models, reflecting diffusion’s visual-quality advantage and autoregression’s ability to leverage techniques developed for LLMs.Video generation has progressed from short, low-resolution clips to realistic, high-resolution videos. Common conditioning inputs include text, images, reference videos, and actions.Source: §8 · p. 56

Video generation with camera control

  • Camera-controlled generation has expanded from neural rendering of explicit 3D representations to direct camera-conditioned view synthesis and diffusion-based approaches.Direct synthesis bypasses explicit 3D priors, while scaling video training data suggests potential for full 3D consistency. Camera controllability is being investigated for robotics and autonomous navigation (He et al., 2024; Wang et al., 2024; Xu et al., 2024).Source: §8 · pp. 56–57

Generative models for robotic control

  • Generative robotic control includes diffusion policies for imitation learning, visual-subgoal motion planning, and generative pre-training using image and video data.Visual sub-goals generalize across environments better than task-specific action sequences (Black et al., 2023; Du et al., 2024; Finn and Levine, 2017; Ko et al., 2024; Zhou et al., 2024). Pre-training transfers diffusion-model features (Gupta et al., 2024) or predicts future frames before joint action-and-frame fine-tuning (Cheang et al., 2024; He et al., 2024).Source: §8 · p. 57

Generative models for autonomous driving

  • Cosmos uses a pre-trained world foundation model for a scalable driving simulator with high resolution, elevated frame rates, and multi-view consistency.Prior driving video generators support conditioning on text, images, trajectories, 3D data, or maps, but face limitations in data scale, resolution, and camera-view count.Source: §8 · p. 57

Tokenizer

  • Cosmos tokenizers build on FSQ (Mentzer et al., 2023) and causal architecture (Yu et al., 2023) to target efficient, high-quality visual compression.Continuous tokenizers compress visuals into latents for diffusion and decode generated latents back to RGB. Discrete tokenizers additionally quantize latents, enabling integration with text and audio in LLMs and VLMs.Source: §8 · pp. 57–58

9. Conclusions and Discussions

Cosmos combines data curation, tokenizers, and diffusion and autoregressive world foundation models for downstream Physical AI tasks. Diffusion models currently provide better generation quality in the reported evaluations, but physical reliability and evaluation remain unresolved challenges.

9. Conclusions and Discussions

  • Cosmos integrates data curation, continuous and discrete tokenizers, diffusion and autoregressive architectures, and fine-tuning for downstream Physical AI tasks.The pretrained models demonstrate adaptability to 3D world navigation, robotic manipulation, and autonomous vehicle systems, which require 3D consistency and action controllability.Source: §9 · p. 58

Limitations

  • Current world foundation models, including Cosmos, remain unreliable physical simulators, with failures in object permanence, contact-rich dynamics, and instruction following.Visually realistic videos do not necessarily obey gravity, light interactions, or fluid dynamics.Source: §9 · p. 58
  • Physical-fidelity evaluation remains difficult because human judgments are subjective and may not align positively with downstream Physical AI metrics.Proposed directions include multimodal LLM evaluators and existing physical simulators for reproducible, interactive evaluation with less dependence on human assessment.Source: §9 · p. 58

Autoregressive vs. Diffusion WFMs.

  • Diffusion-based world foundation models currently outperform autoregressive models in generation quality for 3D consistency and robotics video generation.Fine-tuning supports camera poses, end-effector positions, and autonomous vehicle trajectories as controls, as well as novel output formats such as multi-view videos.Source: §9 · p. 58
  • Autoregressive models could inherit world knowledge from pretrained LLMs and accelerate generation through inference optimizations for causal attention.If realized, these capabilities could support interactive control and real-time robotics planning and simulation.Source: §9 · p. 58
  • Hybrid approaches bridge diffusion and autoregressive models through causal-attention distillation (Yin et al., 2024) and locally bidirectional attention with diffusion heads (Zhou et al., 2024).The distilled causal-attention students support key-value caching during inference (Yin et al., 2024). The authors plan further investigation of hybrid formulations and their trade-offs.Source: §9 · p. 58

A. Contributors and Acknowledgements · A.1. Core Contributors

Core contributors are credited for data curation, tokenization, world-model pre-training, prompt upsampling, diffusion decoding, and evaluation. The credits also identify guardrail contributors and Ming-Yu Liu as platform architect.

A.1. Core Contributors

  • Data curation credits Jacob Huffman, Francesco Ferroni, Alice Luo, Niket Agarwal, and additional contributors.Ming-Yu Liu is credited as platform architect.Source: §A.1 · p. 59
  • Tokenizer contributors are Jinwei Gu, Xian Liu, Songwei Ge, Ting-Chun Wang, Haoxiang Wang, and Fitsum Reda.Source: §A.1 · p. 59
  • Diffusion-based world foundation model pre-training credits Qinsheng Zhang, Lin Yen-Chen, Xiaohui Zeng, and eight additional contributors.The other contributors are Huan Ling, Shitao Tang, Maciej Bala, Ting-Chun Wang, Yu Zeng, Seungjun Nah, Qianli Ma, and Hanzi Mao.Source: §A.1 · p. 59
  • Autoregressive-based world foundation model pre-training credits Haoxiang Wang, Yifan Ding, Xian Liu, Jiaojiao Fan, Xiaohui Zeng, and Yogesh Balaji.Source: §A.1 · p. 59
  • Prompt upsampler contributors are Yunhao Ge, Haoxiang Wang, Jiashu Xu, and Yin Cui.Guardrail contributors are Jibin Varghese, Arslan Ali, Grace Lam, and Pooya Jannaty.Source: §A.1 · p. 59
  • Diffusion decoder contributors are Huan Ling, Jiaojiao Fan, Fitsum Reda, Yogesh Balaji, Hanzi Mao, and Qinsheng Zhang.Source: §A.1 · p. 59
  • Evaluation credits cover 3D consistency and physics alignment during pre-training, plus camera control, robotics, and autonomous driving in post-training.Jiahui Huang and Chen-Hsuan Lin are credited for 3D consistency; physics alignment credits Francesco Ferroni, Prithvijit Chattopadhyay, Xinyue Wei, Qianli Ma, Gergely Klár, and Chen-Hsuan Lin. Post-training evaluation contributors include Xiaohui Zeng for camera control, Lin Yen-Chen for robotics, and Seung Wook Kim for autonomous driving.Source: §A.1 · p. 59

Related papers

Loading 2501.03575v3…