PaperlayerView exact revision on arXiv

Source-linked AI summary

World Simulation with Video Foundation Models for Physical AI

NVIDIA, :, Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, Aaron Blakeman, Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, Yu-Wei Chao, Prithvijit Chattopadhyay, Mike Chen, Yongxin Chen, Yu Chen, Shuai Cheng, Yin Cui, Jenna Diamond, Yifan Ding, Jiaojiao Fan, Linxi Fan, Liang Feng, Francesco Ferroni, Sanja Fidler, Xiao Fu, Ruiyuan Gao, Yunhao Ge, Jinwei Gu, Aryaman Gupta, Siddharth Gururani, Imad El Hanafi, Ali Hassani, Zekun Hao, Jacob Huffman, Joel Jang, Pooya Jannaty, Jan Kautz, Grace Lam, Xuan Li, Zhaoshuo Li, Maosheng Liao, Chen-Hsuan Lin, Tsung-Yi Lin, Yen-Chen Lin, Huan Ling, Ming-Yu Liu, Xian Liu, Yifan Lu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Seungjun Nah, Yashraj Narang, Abhijeet Panaskar, Lindsey Pavao, Trung Pham, Morteza Ramezanali, Fitsum Reda, Scott Reed, Xuanchi Ren, Haonan Shao, Yue Shen, Stella Shi, Shuran Song, Bartosz Stefaniak, Shangkun Sun, Shitao Tang, Sameena Tasmeen, Lyne Tchapmi, Wei-Cheng Tseng, Jibin Varghese, Andrew Z. Wang, Hao Wang, Haoxiang Wang, Heng Wang, Ting-Chun Wang, Fangyin Wei, Jiashu Xu, Dinghao Yang, Xiaodong Yang, Haotian Ye, Seonghyeon Ye, Xiaohui Zeng, Jing Zhang, Qinsheng Zhang, Kaiwen Zheng, Andrew Zhu, Yuke Zhu

arXiv:2511.00062v2cs.CVcs.AIcs.LGcs.RO

TL;DR

Physical AI systems need safer and more scalable alternatives to real-world training and evaluation. The paper develops Cosmos-Predict2.5 and Cosmos-Transfer2.5 using curated video, flow-matching training, specialized post-training, and reinforcement learning, reporting leading benchmark results and demonstrated utility in robotics and autonomous driving.

  • Problem

    Training Physical AI directly in the real world is slow, costly, and risky, creating a need for safe visual simulation proxies.

  • Method

    The paper develops unified video world models and control-net world-translation models using curated data, flow matching, improved text embeddings, domain-specific post-training, and reinforcement learning.

  • Results

    The models achieve leading Physical AI benchmark results, with Transfer2.5 improving detection metrics by up to 60% over Transfer1-7B-Sample-AV.

  • Takeaways & Limitations

    The demonstrated capabilities support synthetic data generation, policy validation, autonomous-driving simulation, robotic manipulation, and closed-loop simulation.

Abstract

from arXiv · show

We introduce [Cosmos-Predict2.5], the latest generation of the Cosmos World Foundation Models for Physical AI. Built on a flow-based architecture, [Cosmos-Predict2.5] unifies Text2World, Image2World, and Video2World generation in a single model and leverages [Cosmos-Reason1], a Physical AI vision-language model, to provide richer text grounding and finer control of world simulation. Trained on 200M curated video clips and refined with reinforcement learning-based post-training, [Cosmos-Predict2.5] achieves substantial improvements over [Cosmos-Predict1] in video quality and instruction alignment, with models released at 2B and 14B scales. These capabilities enable more reliable synthetic data generation, policy evaluation, and closed-loop simulation for robotics and autonomous systems. We further extend the family with [Cosmos-Transfer2.5], a control-net style framework for Sim2Real and Real2Real world translation. Despite being 3.5$\times$ smaller than [Cosmos-Transfer1], it delivers higher fidelity and robust long-horizon video generation. Together, these advances establish [Cosmos-Predict2.5] and [Cosmos-Transfer2.5] as versatile tools for scaling embodied intelligence. To accelerate research and deployment in Physical AI, we release source code, pretrained checkpoints, and curated benchmarks under the NVIDIA Open Model License at https://github.com/nvidia-cosmos/cosmos-predict2.5 and https://github.com/nvidia-cosmos/cosmos-transfer2.5. We hope these open resources lower the barrier to adoption and foster innovation in building the next generation of embodied intelligence.

1. Introduction

The paper motivates video world models as safe proxies for training and evaluating Physical AI, then introduces Cosmos-Predict2.5 and Cosmos-Transfer2.5 to improve controllable generation and world translation across applications.

  • Motivation: Physical AI training in the real world is slow, costly, and risky, motivating high-quality visual simulators as safe proxies.Such simulators can support perception and control learning entirely in silicon before deployment.
  • Cosmos-Predict2.5: Cosmos-Predict2.5 unifies Text2World, Image2World, and Video2World generation while improving data filtering, architecture, text grounding, and post-training.Its training recipe combines curated Physical AI data, model merging, reinforcement learning, and Cosmos-Reason1 text representations.
  • Cosmos-Predict2.5: The model delivers substantial gains over Cosmos-Predict1 in output quality and prompt alignment.
  • Applications: Cosmos-Predict2.5 supports synthetic data generation, policy validation, autonomous-driving simulation, and camera-controllable robotic-manipulation worlds.
  • Cosmos-Transfer2.5: Cosmos-Transfer2.5 extends the family to visual world translation, offering higher quality than Cosmos-Transfer1 at 3.5× smaller size and supporting long-horizon translation and closed-loop simulation.
  • Open resources: The authors release source code, pretrained checkpoints, and curated post-training examples to help practitioners adapt models for Physical AI setups.

2. Data

The data pipeline scales and filters diverse video into structured training resources, combining general curation with domain-specific pipelines for Physical AI applications.

  • Released Models: The released model set maps model capabilities to their corresponding inputs for practical use.
  • Video Curation Pipeline: The pipeline processes diverse video through shot splitting, transcoding, cropping, filtering, captioning, deduplication, and sharding.These stages target high-throughput construction of large-scale, high-quality, semantically diverse data.
  • Video Curation Pipeline: Filtering combines aesthetic, motion, OCR, perceptual-quality, semantic-artifact, and VLM filters before excluding physically unrealistic content.The VLM is applied last because it is computationally more expensive.
  • Video Curation Pipeline: Captions are generated on 5-second windows at multiple lengths, emphasizing factual descriptions of objects, motion, and context for supervision and conditioning.
  • Video Curation Pipeline: Semantic deduplication retains the highest-resolution clip within similar-content clusters, while top-down sharding organizes data by content type, resolution, aspect ratio, and length.These structures support scalable sampling, curriculum training, and domain balancing.
  • Video Curation Pipeline: 35 million hours of raw video yield over 6 billion clips, with 200 million high-quality clips retained after stricter filtering.Survival decreases from 30% in Cosmos-Predict1 to 4% in Cosmos-Predict2.5.
  • Domain-specific Curation: Domain-specific pipelines cover robotics, autonomous driving, smart spaces, human dynamics, and physics, using adjusted filters and customized caption prompts.Robotics data is filtered for low resolution and near-static motion, with slow actions sped up for consistency.

3. Method

Cosmos-Predict2.5 uses flow matching with adaptive noise schedules and a latent denoising network conditioned by text, images, or videos. Its architecture combines relative positional encoding, Cosmos-Reason1 text embeddings, and frame replacement for coherent multimodal world generation.

  • 3.1. Flow Matching: Flow matching trains the model to predict ground-truth velocity with mean squared error over noisy interpolated latents and conditioning information.The conditioning may include text embeddings, reference frames, and other conditional inputs.
  • 3.1. Flow Matching: A shifted logit-normal timestep distribution biases training toward higher noise levels to help reconstruct signals with heavily disrupted correlations.The shift is controlled by β; β = 1 leaves the timestep distribution unchanged.
  • 3.2. Network Architecture: The denoising network removes absolute positional embeddings while retaining relative embeddings, improving flexibility for higher-resolution content and longer video sequences during post-training.The network largely reuses the latent-diffusion DiT design from Cosmos-Predict1.
  • 3.2. Network Architecture: Cosmos-Reason1 replaces the earlier T5 encoder, using concatenated multi-block token activations projected into 1024-dimensional embeddings for cross-attention conditioning.This representation is intended to capture both local and global linguistic context.
  • 3.2. Network Architecture: The model supports Text2World, Image2World, and Video2World generation within one architecture, with image and video modes grounding outputs in reference visual content.Frame replacement keeps conditioned initial frames faithful and propagates visual cues into later frames for more coherent generation.

4. Training

Training progressively increases resolution and task diversity, then specializes models through domain-specific SFT, model merging, and reinforcement learning. Human evaluations report stronger target-domain performance and preferences for RL-generated videos, while the 14B infrastructure has lower utilization than the 2B model.

  • 4.1. Pre-training: Progressive training increases pixel resolution and task diversity, followed by domain-specific SFT, model merging, and RL post-training.Pretraining moves from Text2Image to Image2World and Video2World, then adds Text2World as resolution advances.
  • 4.2.1. Supervised Fine-tuning: Each domain-specific SFT model achieves a significantly higher human-preference win rate than the pretrained baseline on its target domain.The five domains are object permanence, high motion, complex scenes, driving, and robotic manipulation.
  • 4.2.1. Supervised Fine-tuning: Simple hyperparameter grid search consistently outperforms heuristic selection based on individual fine-tuned models’ win rates, leading to selection of model soup as the final post-trained model.The comparison covers four merging approaches, with DARE-Linear the exception to otherwise comparable performance.
  • 4.2.2. Reinforcement Learning: RL increases reward scores in both Text2World and Image2World settings, and human voters prefer RL-generated videos on average.VideoAlign evaluates text alignment, motion quality, and visual quality during post-training.
  • 4.3. Distillation: 36.49% MFU is reported for Cosmos-Predict2.5-2B versus 33.08% for Cosmos-Predict2.5-14B, with the drop attributed to communication costs from large context parallelism.The infrastructure uses timestep distillation and supporting fused attention and sharding adaptations.

5. Results

Post-trained Cosmos-Predict2.5 models perform competitively with substantially larger Wan models on PAI-Bench and human preference evaluations. Qualitative examples further show realistic, physically coherent simulations across physical AI scenarios.

  • PAI-Bench: Cosmos-Predict2.5-2B and 14B perform similarly to Wan2.2 27B-A14B in Text2World and are the best-performing models in Image2World.PAI-Bench reports Domain and Quality scores separately, with the Overall Score averaging both.
  • Human Evaluation: 30.0% versus 26.2% human preference favors Cosmos-Predict2.5-2B over Wan2.2 5B, while 33.0% versus 34.8% is comparable with Wan2.1 14B.The 2B model is 60.0% and 85.7% smaller than the respective comparison models.
  • Human Evaluation: 48.6% versus 31.8% human preference favors the post-trained 14B over Wan2.1 14B, while 38.1% versus 35.9% is on par with Wan2.2 27B-A14B.The 14B model has only half the parameter count of Wan2.2 27B-A14B.
  • Human Evaluation: Increasing model size from 2B to 14B raises preference against Wan2.1 14B from 33.0% to 48.6%.This comparison uses the same Wan2.1 14B baseline.
  • Qualitative Examples: Qualitative samples show accurate driving behaviors, realistic industrial and robotics scenes, and physically coherent motion.The examples are presented as representative physical AI scenarios that complement quantitative and human evaluations.

6. Applications

The applications extend Cosmos-Transfer2.5 and Cosmos-Predict2.5 across transfer, robotics augmentation, long-video generation, and policy-oriented Physical AI workflows. Results include improved control adherence, reduced long-horizon error accumulation, and stronger robustness to novel visual conditions.

  • 6. Applications: The model family supports policy evaluation, camera-controllable multiview generation, and synthetic data generation for VLA training.These extensions cover action-conditioned world models and multiple Physical AI application settings.
  • 6.1. Cosmos-Transfer2.5: Cosmos-Transfer2.5 provides control-net generation from edges, blurred video, segmentation maps, and depth maps sourced from simulation or real-world video.Its control branches are distributed throughout the network to integrate conditioning more gradually.
  • 6.1. Cosmos-Transfer2.5: 3.5× smaller Cosmos-Transfer2.5-2B outperforms Cosmos-Transfer1-7B on control adherence and overall video quality.The evaluation uses PAIBench-Transfer across driving and robotics videos.
  • 6.1. Cosmos-Transfer2.5: Cosmos-Transfer2.5-2B accumulates fewer errors and maintains higher fidelity over long video sequences than Cosmos-Transfer1-7B.The RNDS curves show substantially less reduction over time for the smaller model.
  • 6.2. Real-Robot Data Augmentation: Text-controlled visual augmentation enables systematic simulation of challenging out-of-domain robot scenarios while preserving controllability.The approach supports semantic edits that standard image-based transformations cannot provide.
  • 6.2. Real-Robot Data Augmentation: 24 successes out of 30 trials were achieved by the Cosmos-Transfer2.5-2B-augmented policy, outperforming both baselines on novel object and environment changes.The base policy failed to generalize, while the standard image-augmentation baseline succeeded in just one case.

6.3. Cosmos-Transfer2.5 for Driving Simulation

Cosmos-Transfer2.5-2B/auto/multiview generates synchronized, controllable driving views from world scenario maps containing HD map elements and dynamic-object cues. It improves visual quality and control-signal adherence over the prior model.

  • Model: Cosmos-Transfer2.5-2B/auto/multiview generates consistent driving scenes across multiple viewpoints.It extends the multiview predictor with a control branch.
  • Control Representation: World scenario maps combine HD map elements with color-coded, motion-shaded 3D bounding boxes for vehicles and pedestrians.The maps provide semantic and motion cues across seven camera views.
  • Evaluation: Up to 2.3x gains in FVD/FID scores are reported while temporal and cross-camera Sampson error remains competitive.The evaluation uses disjoint multi-view clips from RDS-HQ-HL.
  • Evaluation: Up to 60% improvement in detection metrics is observed over Transfer1-7B-Sample-AV for 3D cuboids and lanes.Control adherence is evaluated by comparing detector outputs on generated videos with ground-truth labels.

6.4. Multi-view Generation with Camera Control

Cosmos-Transfer2.5 extends world generation to camera-controllable multiview robotics, synthesizing target viewpoints from a reference video and camera trajectories. The resulting videos are evaluated for camera accuracy and cross-view consistency.

  • Model: The model synthesizes multiple target-view videos from a reference-view robot video conditioned on camera trajectories.This supports mappings such as a humanoid’s head-camera view to its gripper views.
  • Method: Target camera poses are encoded as Plücker raymaps and injected into video tokens before self-attention.A projection layer aligns raymap representations with video-latent dimensionality.
  • Variants: Two variants generate synchronized gripper-view or transformed-view videos from robotic manipulation inputs.The variants are fine-tuned on Agibot, MultiCamVideo, and SynCamVideo data.
  • Evaluation: Evaluation measures camera trajectory error and cross-view consistency across 80 validation videos and 16 camera trajectories.Metrics use predicted camera poses and Sampson error between generated views.
  • Qualitative Comparison: The multiview model generates more coherent videos across viewpoints than a single-view targeted baseline.The comparison highlights inconsistencies in the baseline output.

6.5. Synthetic Data Generation for VLA training

The paper uses Cosmos-Predict2.5 to generate synthetic robot videos with language, vision, and pseudo-action annotations for VLA training and action-conditioned prediction. Evaluations report strong instruction following and improvements over a prior action-conditioned baseline.

  • VLA Data Generation: Synthetic videos paired with instructions and extracted pseudo-actions augment real demonstrations for VLA training.The goal is to improve generalization across object, behavior, and environment variations.
  • VLA Data Generation: Cosmos-Predict2.5-14B/robot/gr00tdream-gr1 achieves the highest instruction-following scores on the GR1 humanoid robot dataset.It particularly outperforms competing models on object and environment generalization.
  • Action-Conditioned Prediction: The action-conditioned model predicts future frames from a conditional image and robot-action sequence, autoregressively composing full trajectories.Each chunk is conditioned on the last generated frame.
  • Action-Conditioned Prediction: Action conditioning is incorporated through an action-embedder MLP added to DiT timestamp embeddings.Cross-attention and channel concatenation are evaluated as alternatives.
  • Evaluation: Cosmos-Predict2.5-2B/robot/action-cond outperforms the Cosmos-Predict1 baseline across PSNR, SSIM, Latent L2, and FVD on Bridge.The comparison uses 100 episodes from the official Bridge test set.
  • Evaluation: The action-conditioned model demonstrates better object permanence than the prior model in qualitative Bridge rollouts.The comparison is shown against ground-truth frames.

7. Related Work

Related work spans abstract and video-based world models, while Physical AI has motivated open-source models and benchmarks focused on embodied applications. The paper positions Cosmos-Predict2.5 and Cosmos-Transfer2.5 as enhanced successors supporting robotics and autonomous-driving simulation.

  • World Models: World-model research includes predictive models of future states from observations and actions, with approaches spanning abstract latent representations and other world-dynamics paradigms.
  • Video Generative Models: Video generation has advanced rapidly through powerful general-purpose systems, but many leading models remain proprietary.
  • Foundation World Model for Physical AI: Most existing world models target digital content, whereas Cosmos-Predict1 and Cosmos-Transfer1 introduced open-source world models for Physical AI.
  • Foundation World Model for Physical AI: Cosmos-Predict2.5 supports VLA training, robot policy training and validation, autonomous-driving simulation, and robotic manipulation, while Cosmos-Transfer2.5 improves long-horizon translation and closed-loop simulation.

8. Conclusion

The paper presents Cosmos-Predict2.5 and Cosmos-Transfer2.5 as video world foundation models for Physical AI, trained with curated data, flow matching, improved text embeddings, domain-specific post-training, and reinforcement learning. The models achieve leading benchmark results and are demonstrated in robotics and autonomous driving, with released models and code intended to support a simulation-first ecosystem.

  • Cosmos-Predict2.5 and Cosmos-Transfer2.5 are presented as video world foundation models for Physical AI.
  • The models use curated video datasets, flow-matching training, improved text embedding, domain-specific post-training, and reinforcement learning.
  • The models achieve leading results on Physical AI benchmarks and demonstrate effectiveness in robotics and autonomous driving, where high-fidelity synthetic video is essential.
  • Released models and code aim to establish Cosmos as a world foundation model platform for a simulation-first ecosystem that advances Physical AI and bridges simulation with real-world deployment.

A.1. Contributors

The contributors section lists the paper’s contributors across the supplied passages.

  • The supplied contributor list includes Arslan Ali, Junjie Bai, Maciej Bala, Yogesh Balaji, and Aaron Blakeman.
  • The supplied contributor list includes Tiffany Cai, Jiaxin Cao, Tianshi Cao, Elizabeth Cha, and Yu-Wei Chao.
  • The supplied contributor list continues with Sanja Fidler, Xiao Fu, Ruiyuan Gao, Yunhao Ge, and Jinwei Gu.
Loading 2511.00062v2…