PaperlayerView exact revision on arXiv

Source-linked AI summary

Cosmos World Foundation Model Platform for Physical AI

NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Yi Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, Artur Zolkowski

arXiv:2501.03575v3cs.CVcs.AIcs.LGcs.RO

TL;DR

Physical AI needs safer and more scalable ways to learn from world-interacting data, while current world models remain incomplete as physical simulators. Cosmos presents an open world foundation model platform that curates video, trains tokenizers and diffusion or autoregressive WFMs, and post-trains them for specialized Physical AI applications. The paper reports strong tokenizer performance and better current generation quality for diffusion WFMs, while identifying reliability limitations in physical realism and model behavior.

  • Problem

    Physical AI training data is difficult to scale because action sequences perturb the physical world, motivating a digital twin that can be safely interacted with.

  • Method

    Cosmos combines video curation, causal video tokenizers, diffusion and autoregressive WFM pre-training, downstream post-training, and guardrails.

  • Results

    Diffusion-based WFMs currently deliver better generation quality than autoregressive WFMs, while Cosmos Tokenizer achieves state-of-the-art results across reported image and video tokenizer metrics.

  • Takeaways & Limitations

    The platform demonstrates adaptability to 3D navigation, robotic manipulation, and autonomous vehicle systems requiring 3D consistency and action controllability.

  • Takeaways & Limitations

    Current models are not reliable physical-world simulators and exhibit failures involving object permanence, contact-rich dynamics, instruction following, and physical principles.

Abstract

from arXiv · show

Physical AI needs to be trained digitally first. It needs a digital twin of itself, the policy model, and a digital twin of the world, the world model. In this paper, we present the Cosmos World Foundation Model Platform to help developers build customized world models for their Physical AI setups. We position a world foundation model as a general-purpose world model that can be fine-tuned into customized world models for downstream applications. Our platform covers a video curation pipeline, pre-trained world foundation models, examples of post-training of pre-trained world foundation models, and video tokenizers. To help Physical AI builders solve the most critical problems of our society, we make Cosmos open-source and our models open-weight with permissive licenses available via https://github.com/nvidia-cosmos/cosmos-predict1.

1. Introduction

Physical AI requires safe digital interaction with the world, but scaling action-observation data is difficult because exploratory actions can damage systems and environments. Cosmos addresses this with a pre-training-and-post-training platform combining curated video, world foundation models, tokenizers, and specialized downstream adaptations.

  • Motivation: Physical AI combines sensors and actuators, yet training data is difficult to scale because actions perturb the physical world and may cause damage.A world foundation model provides a digital twin that Physical AI can safely interact with.
  • Platform approach: Cosmos pre-trains generalist WFMs on large-scale video, then fine-tunes them with data from a target Physical AI environment.The post-training dataset is specific to the targeted setup, while pre-training exposes models to diverse visual experiences.
  • Platform approach: The platform develops a video curation pipeline that extracts about 100M clips from a 20M-hour video collection.Clips range from 2 to 60 seconds and receive VLM-generated captions per 256 frames.
  • Model design: Cosmos explores transformer-based diffusion and autoregressive models using continuous and discrete video tokens, respectively.Both approaches decompose video generation into more tractable sub-problems, while tokenization compresses video content for learning.
  • Downstream applications: Post-trained WFMs support camera control, robotic instruction following, and autonomous-driving scenarios, while a guardrail blocks harmful inputs and outputs.These examples illustrate adaptations for multiple Physical AI setups rather than a single downstream task.
  • Scope and release: The models and tokenizers are released under the NVIDIA Open Model License, although the world foundation model problem remains far from solved.The paper explicitly calls for additional research to advance the state of the art.

2. World Foundation Model Platform

The Cosmos platform models future visual observations from past observations and perturbations, then supports safe development through curation, tokenization, pre-training, post-training, and guardrails. Its pre-trained models are specialized for downstream Physical AI applications using controls such as actions, camera poses, and prompts.

  • World-model definition: A WFM predicts the future observation from past visual observations and a current perturbation, which may be an action, random input, or text description.In the visual setting, the observations are RGB video and the perturbation represents a control signal.
  • Platform components: The platform includes a video curator, video tokenizer, pre-trained WFM, post-training samples, and guardrail.The guardrail blocks harmful inputs and outputs for safer model usage.
  • Platform components: Causal video tokenizers compress video without using future observations, enabling joint image-video training and alignment with causal Physical AI systems.Using single-image inputs makes the causal tokenizer also function as an image tokenizer.
  • WFM pre-training: Cosmos pre-trains scalable transformer-based diffusion and autoregressive WFMs, using continuous and discrete token representations, respectively.Diffusion and autoregressive models are developed as alternative approaches for pre-trained world modeling.
  • WFM pre-training: Diffusion pre-training progresses from Text2World generation to Video2World generation conditioned on past video and a text prompt.The second stage predicts a future video world from past video and an input text prompt.
  • WFM pre-training: Autoregressive pre-training progresses from next-token generation to text-conditioned Video2World generation.The model first learns foresight from past video, then incorporates a text prompt for future-world generation.
  • World-model post-training: Post-training adapts pre-trained WFMs for camera-pose navigation, humanoid robotics, and autonomous driving.Camera-pose conditioning enables users to navigate freely through the generated world.

3. Data Curation

Cosmos curates heterogeneous video into training data through splitting, filtering, annotation, deduplication, and sharding. The pipeline targets useful physical dynamics, standardizes processing, and improves throughput for large-scale dataset construction.

  • Curation pipeline: Cosmos Video Curator uses five steps: splitting, filtering, annotation, deduplication, and sharding.Splitting creates shots, filtering removes low-value clips, annotation adds descriptions, deduplication removes semantic redundancy, and sharding organizes clips by resolution and aspect ratio.
  • Raw data: The curated data targets driving, manipulation, human activity, navigation, first-person views, nature dynamics, camera motion, synthetic renders, and other categories.The listed category proportions sum to the full target distribution, with nature dynamics at 20% and driving at 11%.
  • Raw data: The raw collection contains about 20M hours of 720p-to-4k video, much of which is redundant or uninformative for learning world physics.The pipeline selects valuable portions and also collects image data for joint image-video training.
  • Shot detection: Shot splitting segments arbitrary-length videos at shot changes, discards clips shorter than 2s, and caps clip length at 60s.Longer clips are further split so later filtering can assess their usefulness for learning world physics.
  • Shot detection: End-to-end neural shot detectors outperform hand-crafted or heuristic methods, with TransNetV2 performing better on challenging shot changes.TransNetV2 also supports GPU acceleration for higher splitting throughput.
  • Transcoding: Standardizing clips with high-quality H.264 encoding improves dataloader stability and training efficiency across heterogeneous source codecs.The pipeline uses h264_nvenc and stress-tests fast motion and high-frequency textures for visual degradation.
  • Transcoding: Combining transcoding improvements produces a ∼6.5× increase in throughput.The comparison evaluates hardware and software configurations for transcoding performance.
  • Filtering: Filtering combines motion, visual-quality, text, and video-type filters to remove poor clips, select fine-tuning data, and tailor the WFM training distribution.Motion filtering also tags camera-motion types such as pan, zoom, and tilt.

4. Tokenizer

Cosmos Tokenizer is an encoder-decoder suite of continuous and discrete visual tokenizers designed to compress images and videos while preserving reconstruction quality and improving efficiency. Its causal, wavelet-space architecture supports flexible resolutions and strong compression-quality trade-offs across image and video benchmarks.

  • Architecture: The tokenizer encodes an input video into compact tokens and decodes those tokens to reconstruct the video while preserving visual information.The encoder-decoder design maps videos to token representations and reconstructs them at the original resolution.
  • Architecture: Its temporally causal wavelet-space architecture processes current and past frames without future observations.A 2-level wavelet transform downsamples inputs along spatial and temporal dimensions before causal encoder processing.
  • Results: Cosmos Tokenizer maintains strong quality at higher resolutions and runs 2× ∼12× faster than prior tokenizers while using the smallest model size.Runtime measurements were performed on a single NVIDIA A100 80GB GPU.
  • Results: Cosmos Tokenizer achieves state-of-the-art video-tokenizer performance on DAVIS and TokenBench at 4 × 8 × 8 compression.It remains better than prior art at 8 × 8 × 8 and 8 × 16 × 16 compression, demonstrating an excellent compression-quality trade-off.
  • Results: For images, Cosmos Tokenizer achieves state-of-the-art results at 8 × 8 compression and often matches or exceeds prior-art quality at 16 × 16.The 16 × 16 setting uses a 4× larger compression ratio than 8 × 8.

5. World Foundation Model Pre-training

Cosmos pre-trains world foundation models as generalists that capture general knowledge of real-world physics and natural behaviors. It develops diffusion- and autoregressive-based model families, each with base and derivative models and supporting components.

  • Pre-trained WFMs are generalists designed to capture general knowledge of real-world physics and natural behaviors.
  • Cosmos builds two scalable WFM families: transformer-based diffusion models and transformer-based autoregressive models.Both paradigms decompose video generation into sequences of easier sub-problems.
  • Table 10 organizes two WFM families, each containing two base models and two derivative models, plus diffusion prompt upsampling and autoregressive diffusion decoding.
  • Diffusion-based WFMs: The diffusion family includes 7B and 14B Text2World models, later fine-tuned into Video2World models that predict future video from current observations and text perturbations.These models use continuous visual tokens from Cosmos-Tokenize1-CV8×8×8-720p.
  • Autoregressive-based WFMs: The autoregressive family includes 4B and 12B base models that predict future videos from current observations, with cross-attention added to incorporate text prompts.These models are Llama3-style GPT models trained from scratch for video prediction and initially have no language understanding.

5.1. Diffusion-based World Foundation Model

The diffusion-based WFM uses latent diffusion over compact video representations, with EDM-inspired denoising, DiT-based conditioning, and scalable parallelization. Its qualitative generations are photorealistic, while larger models show richer scenes and more stable motion.

  • Latent diffusion formulation: The diffusion WFM operates in a tokenizer’s learned latent space, reducing video dimensionality and computational costs during training and inference.Cosmos-Tokenize1-CV8×8×8-720p produces the latent representations.
  • Latent diffusion formulation: EDM-inspired training uses denoising score matching on clean samples corrupted with Gaussian noise and a weighted expectation over noise levels.The denoiser is noise-conditioned and follows EDM preconditioning.
  • Latent diffusion formulation: An uncertainty function reweights denoising losses across noise levels, reducing the contribution of uncertain tasks while penalizing excessive uncertainty.The function is parameterized by a simple MLP and optimized with the overall loss.
  • Denoiser architecture: The denoiser adapts DiT for controllable video generation using 3D patchification, self-attention, text cross-attention, and feed-forward layers.Latent video states are projected from non-overlapping 3D cubes into token inputs, while T5-XXL embeddings provide semantic context.
  • Denoiser architecture: FPS-aware 3D RoPE supports varying video sizes, aspect ratios, lengths, and frame rates, while learnable absolute embeddings reduce training loss and morphing artifacts.The architecture achieves reasonable performance within 5,000 training steps during progressive training.
  • Scaling up: Context parallelism distributes query and key-value sequences across GPUs, overlapping communication and computation to address FLOPs and activation-memory demands.For Cosmos-Predict1-14B, FSDP reduces approximately 280 GB to 4 GB per GPU, while CP reduces approximately 310 GB to 40 GB per GPU.
  • Scaling up: Without tensor or sequence parallelism, the setup achieves comparable Model FLOPs Utilization to approaches using those techniques.The paper leaves a detailed analysis of tradeoffs for future work.
  • Results: Both Text2World and Video2World models generate photorealistic videos with strong visual quality, motion dynamics, and text alignment; 14B models produce richer scenes and more stable or intricate motion.The Video2World models support image and video conditioning and extended autoregressive generation.

5.2. Autoregressive-based World Foundation Model

The autoregressive WFM treats video generation as next-token prediction over discrete video tokens, using transformer modifications for spatial, temporal, textual, and training-stability requirements. Experiments show scalable inference acceleration, real-time generation, improved quality with larger models and diffusion decoding, alongside text-conditioning and single-frame robustness limitations.

  • Model formulation: Autoregressive WFMs convert videos into discrete tokens and train a Transformer decoder to predict each next token from preceding tokens.The objective models the conditional probability of the next token given prior video tokens.
  • Architecture: The architecture adds 3D positional embeddings, cross-attention for text conditioning, and QK-Normalization to the standard Transformer.It combines 3D RoPE for relative temporal-spatial positions with 3D absolute positional embeddings for absolute coordinates.
  • Tokenization: The model uses Cosmos-Tokenize1-DV8×16×16-720p with FSQ quantization, producing a 64,000-token vocabulary for autoregressive generation.The six-dimensional latent space is quantized into levels (8, 8, 8, 5, 5, 5).
  • Training stability: Z-loss stabilizes large-scale training by penalizing squared logits, with λ = 3 × 10−4 found to maintain healthy gradient norms.The authors report this coefficient as optimal, especially when scaling training across many GPU nodes.
  • Inference optimization: Medusa accelerates inference, reaching up to 2.0× token throughput for 4B and 3.2× for 5B models, with 9 heads giving the best trade-off.The corresponding reductions in forward passes are up to 4.6× for 4B and 6.1× for 5B models; more heads can reduce passes while slowing throughput.
  • Results and limitations: The models generate 10 video frames in less than 1 second at 10 FPS, while larger models improve motion and detail and diffusion decoding restores details lost through discrete compression.The diffusion decoder preserves content while enhancing details; 9-frame video conditioning has a failure rate below 2%, unlike less robust single-frame conditioning in smaller models.

5.3. Evaluation

The evaluation measures whether pre-trained WFMs preserve 3D structure and physical dynamics in generated videos. Controlled physics simulations reveal stronger kinematic prediction with more conditioning frames, while physics adherence remains a major limitation.

  • Evaluation goals: WFMs are evaluated for 3D consistency and physics alignment because generated videos should remain geometrically plausible and obey physical laws over time.The authors note that these metrics cover only two aspects of a broader, nontrivial evaluation problem.
  • Controlled physics benchmark: The benchmark compares WFM rollouts against physically correct photorealistic simulations across scenarios targeting Newtonian physics and rigid-body dynamics.It includes eight 3D scenarios, such as free fall, slopes, stacks, and dominoes, with randomized objects and backgrounds.
  • Evaluation protocol: Rollouts are conditioned on either 1 or 9 initial ground-truth frames, optionally with captions focused on object kinematics, and evaluated using pixel-, feature-, and object-level metrics.PSNR and SSIM compare predicted and reference frames; object-level evaluation uses propagated instance masks and IoU.
  • Results: More conditioning frames improve prediction of object kinematics because they provide information about quantities such as speed and acceleration.The reported evaluation averages metrics across frames, videos, and four rollout seeds.
  • Results: Diffusion WFMs outperform autoregressive WFMs in pixel-level prediction under 9-frame conditioning, but larger models do not consistently improve physics alignment.The authors attribute the pixel-level difference to higher visual quality from diffusion models and report that all WFMs still struggle with physical adherence.
  • Limitations and future work: Rigid-body simulations expose failures including object impermanence, deformation, implausible kinematics, and gravity violations, motivating better data curation, model design, and metrics.The authors plan more complex and photorealistic scenarios to narrow the sim-to-real gap.

6. Post-trained World Foundation Model

The platform post-trains pre-trained WFMs for controllable 3D world generation. Camera-conditioned models generate coherent futures under specified trajectories and outperform CamCo on visual quality, pose estimation, and camera control.

  • Overview: Post-training examples cover camera control for navigable worlds, robotic action control, and multi-view generation for autonomous driving.The discussed models are sample applications rather than complete production systems.
  • Camera-conditioned world generation: Camera pose conditioning turns Cosmos-Predict1-7B-Video2World into a model that generates temporally coherent, 3D-consistent videos from a single image and camera trajectory.Camera information is encoded with Plücker embeddings concatenated with latent video representations.
  • Evaluation: The camera-control evaluation measures frame- and video-level quality with FID and FVD, and 3D consistency through camera-pose re-estimation errors and success rates.Predicted trajectories are aligned to reference trajectories using Procrustes analysis.
  • Quantitative comparison: Cosmos-Predict1-7B-Video2World-Sample-CameraCond achieves lower FID/FVD scores, higher pose-estimation success, and trajectory estimates closer to the original controls than CamCo.Both models were trained on DL3DV-10K and tested on RealEstate10K, creating a distribution shift.
  • Qualitative results: Joystick-like forward, backward, left-rotation, and right-rotation controls demonstrate navigation through simulated worlds and prediction of futures under different camera movements.Different random seeds also produce diverse worlds while maintaining 3D spatial and temporal coherence.

6.2. Post-training WFM for Robotic Manipulation

The paper fine-tunes WFMs for instruction-based robot video prediction and action-conditioned next-frame generation. Both applications produce stronger results than their respective baselines on human preference or video metrics.

  • Task formulation: Instruction-based prediction maps a robot’s current frame and text instruction to a video of the robot carrying out the instruction.Action-based prediction maps the current frame and a 7-dimensional action vector to the next frame and can be rolled out autoregressively.
  • Datasets: The robotic benchmarks use Cosmos-1X egocentric videos for instruction following and Bridge third-person robot-arm episodes for action-conditioned prediction.Cosmos-1X contains approximately 200 hours and 12,000 episodes; Bridge contains approximately 20,000 episodes.
  • Fine-tuning: The instruction models inject T5 instruction embeddings through cross-attention, while action models add action-embedding MLP modules because action is absent from pre-training.The 5B action model uses cross-attention, whereas the 7B model adds the action embedding to DiT timestamp embeddings.
  • Instruction evaluation: Instruction models are evaluated on instruction following, object permanence, verity, and overall usefulness for robot planning through anonymous pairwise human comparisons.Ten evaluators assessed 23 test episodes.
  • Instruction results: 78.3% overall preference versus 13.0% for VideoLDM-Instruction was achieved by the 7B Cosmos instruction model, while both Cosmos models outperformed the baseline across four dimensions.The 5B Cosmos instruction model also performed better than VideoLDM-Instruction.
  • Action-prediction results: Both Cosmos action-conditioned models outperform IRASim-Action on PSNR, SSIM, Latent L2, and FVD evaluations on Bridge.Qualitative samples show predicted frames closely matching ground-truth frames for both model sizes.

6.3. Post-training WFM for Autonomous Driving

The paper develops multi-view WFMs for autonomous-driving simulation using surround-view driving data and optional trajectory conditioning. Cosmos models improve multi-view geometry and trajectory consistency over VideoLDM-MultiView, with explicit trajectory guidance ranked best.

  • Motivation and data: An autonomous-driving world model should generate synchronized views matching a vehicle’s multiple cameras to support simulation and agent training.The paper targets six camera directions and uses ego-motion to construct trajectory data.
  • Motivation and data: The RDS dataset contains approximately 3.6 million 20-second surround-view clips, totaling about 20,000 hours across six cameras.Data mining targets attributes including traffic density, weather, illumination, speed, vehicle behavior, and road type.
  • Multi-view models: The fine-tuned models generate six views simultaneously, with text-, trajectory-, or previous-video-frame conditioning and outputs of 57 frames at 848 × 480 resolution.The video-conditioned model extends videos generated by the text-conditioned multi-view model.
  • Architecture: The architecture uses independent positional embeddings per view, global view embeddings, and view-dependent cross-attention for each view’s text description.These design choices represent view differences while allowing cross-view self-attention during denoising.
  • Trajectory control: Trajectory conditioning represents an agent path as 64 3D translation points and provides explicit control over generated driving scenarios.The trajectory embedding is supplied as a conditional input to the denoiser.
  • Results: Cosmos multi-view models achieve better geometric plausibility and trajectory consistency than VideoLDM-MultiView, with trajectory consistency close to real-world videos.The trajectory-conditioned model is ranked best for consistency, and its small margin indicates accurate path following.

7. Guardrails

Cosmos Guardrail uses complementary pre- and post-generation defenses to block harmful prompts and outputs. Its components combine text filtering, visual safety classification, face blurring, and adversarial evaluation.

  • Guardrail Architecture: The two-stage guardrail blocks harmful prompts before generation and harmful visual outputs afterward.Pre-Guard filters inputs, while post-Guard filters generated videos and obscures detected faces.
  • Pre-Guard: Pre-Guard combines an LLM-based semantic guardrail with a keyword blocklist for explicitly unsafe prompts.The blocklist is the first defense, followed by Aegis as a second line for semantically complex prompts.
  • Pre-Guard: Aegis classifies prompts across violence, sexual, criminal, weapons, substance-abuse, suicide, child-safety, hatred, harassment, threat, and profanity categories.Prompts outside these categories are considered safe from the prompt-filtering standpoint.
  • Post-Guard: Post-Guard combines a frame-level video content safety classifier with a face-blur filter for generated outputs.The classifier separates safe and unsafe classes, while face regions larger than 20 × 20 pixels are pixelated.
  • Post-Guard: The safety classifier is trained using dataset videos, synthetic WFM generations covering corner cases, and human-annotated gold-standard labels.Training data are balanced to mitigate false positives and false negatives.
  • Evaluation: A dedicated red team probes the system with standard and adversarial prompts, and expert annotators rate generated videos across harm categories.Annotations also identify the start and end frames containing unsafe content.

8. Related Work

Related work spans learned world models, video generation, robotic control, autonomous-driving simulation, and visual tokenization. Cosmos builds on these lines while developing efficient continuous and discrete tokenizers for world foundation models.

  • World Models: World models predict future states from current states and inputs, supporting prediction and decision-making in physical environments.Prior work includes latent recurrent models and visual-space conditional video-generation approaches.
  • World Models: World models have been applied to verification, planning-based model predictive control, model-based reinforcement learning, games, robots, and autonomous driving.These applications motivate foundational world models for physical-world tasks.
  • Video Generative Models: Video-generation research includes text-to-video, image-to-video, diffusion, autoregressive, and camera-controlled generation.Diffusion dominates existing work, while autoregressive models offer unified multimodal handling.
  • Generative Models for Robotic Control: Generative models are increasingly used for robotic control, including diffusion-based visuomotor policies and pretrained world models for downstream tasks.These approaches target imitation learning and robotic decision-making.
  • Generative Models for Autonomous Driving: Video generation for autonomous driving conditions on modalities such as text, images, trajectories, 3D data, and maps.This line of work aims to simulate diverse driving scenarios.
  • Tokenizers: Visual tokenizers compress visual data into continuous or discrete representations for efficient generative-model training and multimodal integration.Cosmos tokenizers build on FSQ and causal architectures to provide efficient, high-quality tokenizers.

9. Conclusions and Discussions

Cosmos presents a broad platform for general-purpose physical-world simulation and demonstrates adaptation to navigation, manipulation, and autonomous-driving tasks. The authors emphasize that current world foundation models remain unreliable physical simulators and that evaluation itself is difficult.

  • Conclusions: Cosmos integrates data curation, continuous and discrete tokenizers, diffusion and autoregressive models, and fine-tuning for Physical AI tasks.The platform targets world models requiring 3D consistency and action controllability.
  • Conclusions: The pretrained models adapt to 3D world navigation, robotic manipulation, and autonomous vehicle systems.These applications require 3D consistency and action controllability.
  • Limitations: Current models, including Cosmos, remain unreliable physical-world simulators because of imperfect object permanence, contact-rich dynamics, instruction following, and physical-law adherence.The cited examples include gravity, light interactions, and fluid dynamics.
  • Evaluation: Human evaluation of physical fidelity is difficult because judgments are subjective and may not align with downstream-task metrics.The authors suggest multimodal-LLM evaluators and physical simulators as promising directions.
  • Diffusion vs. Autoregressive WFMs: Diffusion-based WFMs currently provide better generation quality in 3D consistency and robotics video generation evaluations.They can incorporate camera pose, end-effector positions, and vehicle trajectories through fine-tuning.

A.1. Core Contributors

The core contributors are organized by platform component and evaluation area. Listed roles cover curation, tokenization, model pre-training, prompting, decoding, consistency and physics evaluation, post-training evaluation, guardrails, and platform architecture.

  • Core Contributors: Named contributors cover data curation, tokenization, diffusion and autoregressive WFM pre-training, prompt upsampling, and diffusion decoding.These roles span the platform’s data and model-development components.
  • Core Contributors: Evaluation contributors are listed for 3D consistency, physics alignment, camera control, robotics post-training, and autonomous-driving post-training.The roles correspond to both pre-training and downstream-task evaluation.
  • Core Contributors: Additional named roles cover guardrail development and platform architecture.These roles complement the data, modeling, and evaluation teams.

A.2. Contributors

This section lists the contributors to the Cosmos World Foundation Model Platform.

  • The contributors are Anqi Li, Arsalan Mousavian, Artur Zolkowski, Bartosz Stefaniak, Dieter Fox, and Ethan He.
  • Additional contributors are Kaichun Mo, Morteza Ramezanali, Przemek Tredak, Wei Yang, Xiaowei Ren, Yongxin Chen, and Zeeshan Patel.
  • The section credits thirteen contributors in total.
Loading 2501.03575v3…