Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

NVIDIA; :; Alisson Azzolini; Junjie Bai; Hannah Brandon; Jiaxin Cao; Prithvijit Chattopadhyay; Huayu Chen; Jinju Chu; Yin Cui; Jenna Diamond; Yifan Ding; Liang Feng; Francesco Ferroni; Rama Govindaraju; Jinwei Gu; Siddharth Gururani; Imad El Hanafi; Zekun Hao; Jacob Huffman; Jingyi Jin; Brendan Johnson; Rizwan Khan; George Kurian; Elena Lantz; Nayeon Lee; Zhaoshuo Li; Xuan Li; Maosheng Liao; Tsung-Yi Lin; Yen-Chen Lin; Ming-Yu Liu; Xiangyu Lu; Alice Luo; Andrew Mathau; Yun Ni; Lindsey Pavao; Wei Ping; David W. Romero; Misha Smelyanskiy; Shuran Song; Lyne Tchapmi; Andrew Z. Wang; Boxin Wang; Haoxiang Wang; Fangyin Wei; Jiashu Xu; Yao Xu; Dinghao Yang; Xiaodong Yang; Zhuolin Yang; Jingxu Zhang; Xiaohui Zeng; Zhe Zhang

Source-linked AI summary

Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

NVIDIA, :, Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, Liang Feng, Francesco Ferroni, Rama Govindaraju, Jinwei Gu, Siddharth Gururani, Imad El Hanafi, Zekun Hao, Jacob Huffman, Jingyi Jin, Brendan Johnson, Rizwan Khan, George Kurian, Elena Lantz, Nayeon Lee, Zhaoshuo Li, Xuan Li, Maosheng Liao, Tsung-Yi Lin, Yen-Chen Lin, Ming-Yu Liu, Xiangyu Lu, Alice Luo, Andrew Mathau, Yun Ni, Lindsey Pavao, Wei Ping, David W. Romero, Misha Smelyanskiy, Shuran Song, Lyne Tchapmi, Andrew Z. Wang, Boxin Wang, Haoxiang Wang, Fangyin Wei, Jiashu Xu, Yao Xu, Dinghao Yang, Xiaodong Yang, Zhuolin Yang, Jingxu Zhang, Xiaohui Zeng, Zhe Zhang

arXiv:2503.15558v3Mar 18, 2025cs.AIcs.CVcs.LGcs.RO

TL;DR

Physical AI systems must connect perception and reasoning to actions in dynamic physical environments, but existing language models struggle to ground textual knowledge in real-world interactions. Cosmos-Reason1 defines ontologies for physical common sense and embodied reasoning, develops 7B and 56B multimodal models, and trains them with supervised fine-tuning and reinforcement learning. Physical AI SFT improves backbone VLM performance by more than 10% on the proposed benchmarks, while RL improves results across all benchmarks.

Problem
Existing language models often struggle to connect knowledge acquired from text with real-world interactions and dynamics.
Method
The paper defines physical-common-sense and embodied-reasoning ontologies, develops multimodal models, and trains them with Physical AI SFT and RL using video-centered data and benchmarks.
Results
Physical AI SFT improves backbone VLM performance by more than 10% on the proposed physical common-sense and embodied-reasoning benchmarks, while RL improves all benchmarks.
Takeaways & Limitations
Cosmos-Reason1 provides models, ontologies, benchmarks, code, and pretrained weights intended to support progress in Physical AI reasoning.
Takeaways & Limitations
RoboFail performance remains stagnant, which the authors attribute primarily to insufficient representative training data and complex affordance, visual-detail, and out-of-distribution reasoning errors.

Abstract

from arXiv · show

Physical AI systems need to perceive, understand, and perform complex actions in the physical world. In this paper, we present the Cosmos-Reason1 models that can understand the physical world and generate appropriate embodied decisions (e.g., next step action) in natural language through long chain-of-thought reasoning processes. We begin by defining key capabilities for Physical AI reasoning, with a focus on physical common sense and embodied reasoning. To represent physical common sense, we use a hierarchical ontology that captures fundamental knowledge about space, time, and physics. For embodied reasoning, we rely on a two-dimensional ontology that generalizes across different physical embodiments. Building on these capabilities, we develop two multimodal large language models, Cosmos-Reason1-7B and Cosmos-Reason1-56B. We curate data and train our models in two stages: Physical AI supervised fine-tuning (SFT) and Physical AI reinforcement learning (RL). To evaluate our models, we build comprehensive benchmarks for physical common sense and embodied reasoning according to our ontologies. Evaluation results show that Physical AI SFT and RL bring significant improvements. To facilitate the development of Physical AI, we make our code and pre-trained models available under the NVIDIA Open Model License at https://github.com/nvidia-cosmos/cosmos-reason1.

1. Introduction

Cosmos-Reason1 addresses the difficulty multimodal models have grounding knowledge in real-world interactions by combining physical reasoning capabilities, specialized models, training stages, and benchmarks. The paper reports improvements from supervised fine-tuning and reinforcement learning across its evaluations.

LLMs often struggle to connect textual physical-world knowledge with real-world interactions and dynamics.
The paper defines physical common sense and embodied reasoning ontologies to establish capabilities and measure Physical AI progress.The common-sense ontology covers Space, Time, and Fundamental Physics; the embodied-reasoning ontology spans four capabilities and five agent types.
Cosmos-Reason1 introduces 7B and 56B multimodal models trained with Physical AI SFT and Physical AI RL.The models process video through a vision encoder and projector before the language model, generating explanations and embodied decisions in natural language.
The authors curate around 4M video-text annotations, including captions, multiple-choice questions, and long chain-of-thought reasoning traces.The data pipelines follow the proposed physical common-sense and embodied-reasoning ontologies and use human annotations and model distillation for SFT.
The evaluation suite includes three physical common-sense benchmarks with 604 questions from 426 videos and six embodied-reasoning benchmarks with 610 questions from 600 videos.The embodied benchmarks cover tasks across humans, robot arms, humanoid robots, and autonomous vehicles.
Physical AI SFT and rule-based, verifiable-reward RL improve model performance, with RL improving results on all benchmarks.

2. Physical AI Reasoning

The paper frames Physical AI reasoning around embodiment-agnostic physical common sense and action-grounded embodied reasoning. It organizes these capabilities into ontologies while focusing evaluation on perception, action effects, and future interaction decisions.

Common Sense Reasoning: Physical common sense is an embodiment-agnostic understanding of what is possible, impossible, or likely in the real world.It supports predicting plausible and implausible situations without requiring human-like mechanisms or embodiments.
Common Sense Reasoning: The physical common-sense ontology contains Space, Time, and Fundamental Physics, divided into 16 fine-grained subcategories.Its subcategories cover object relationships, event order and causality, object permanence, mechanics, and other physical principles.
Embodied Reasoning: Embodied reasoning requires processing incomplete sensory inputs, predicting action effects, and planning behavior in dynamic, uncertain environments.
Embodied Reasoning: The embodied-reasoning ontology spans capabilities and physical embodiments across humans, animals, robotic arms, humanoids, and autonomous vehicles.
Embodied Reasoning: The paper focuses on video-based sensory processing, task-completion verification, and next plausible action prediction, leaving learning from interactions for future work.

3. Cosmos-Reason1

Cosmos-Reason1 is a family of two multimodal models that process video and text through a decoder-only architecture to produce long chain-of-thought responses. The 56B model uses a hybrid Mamba-MLP-Transformer backbone alongside Transformer-based components.

Cosmos-Reason1 comprises the 7B and 56B multimodal models specialized for Physical AI reasoning.
Cosmos-Reason1-7B uses Qwen2.5-VL, while Cosmos-Reason1-56B uses InternViT-300M-V2.5 with Nemotron-H.
The models align video tokens with text tokens before feeding both into a decoder-only LLM backbone.A vision encoder and projector map video into the LLM token embedding space.
3.2. Hybrid Mamba-MLP-Transformer Backbone: The 56B backbone combines Mamba, MLP, and Transformer components to improve long-sequence efficiency while retaining Transformer-based long-context modeling.Mamba provides linear-time sequence modeling, while selected Transformer layers capture details that selective state spaces may miss.
The 7B model is trained with TP=4, whereas the 56B hybrid model uses TP=8 and PP=2.

4. Reinforcement Learning

Cosmos-Reason1 uses Physical AI SFT followed by reinforcement learning with GRPO and customized infrastructure. The asynchronous framework improves training efficiency and supports fault tolerance and dynamic resource scaling.

The training pipeline adapts pretrained visual-language models through Physical AI SFT and then Physical AI RL.RL post-training targets physical common sense and embodied reasoning abilities.
4.1. Algorithm: GRPO derives each response’s advantage by normalizing its reward within a group of responses, avoiding a separate critic model.
4.2. Training Framework: Approximately 160% improvement in training efficiency is reported over colocated frameworks.
4.2. Training Framework: The framework can continue training after node failures and supports dynamic scaling through fault-tolerant mesh management and dispatcher redundancy.
4.2. Training Framework: The RL framework separates dispatching, actor rollout, and policy training into a fully asynchronous pipeline.Actor rollout generates responses and computes rewards and advantages, while policy training optimizes the actor.

5. Data

The paper builds Physical AI SFT data from video-text annotations covering physical common sense and embodied reasoning. Its curation pipelines combine captions, questions, reasoning traces, and short-horizon action data across embodiments.

Physical common sense data use video VQA, while embodied reasoning data span humans, robot arms, humanoids, and autonomous vehicles.
5.1.1. Physical Common Sense SFT: The physical common sense pipeline selects videos, creates detailed captions, generates understanding and hypothetical reasoning questions, and cleans reasoning traces.
5.1.1. Physical Common Sense SFT: The common sense corpus includes ∼99k understanding and ∼59.4k reasoning free-form SFT samples from 9.9K videos.
5.1.1. Physical Common Sense SFT: The MCQ pipeline produces ∼2.4M understanding questions from ∼1.2M clips and ∼600k reasoning questions from ∼356k clips.
5.1.2. Embodied Reasoning SFT: Embodied reasoning curation targets task completion, action affordance, and next plausible action prediction using captions, questions, answers, and detailed reasoning traces.
5.1.2. Embodied Reasoning SFT: Embodied data are segmented into short-horizon clips centered on individual actions or distinct subtasks.

6. Benchmark

The benchmarks evaluate physical common sense and embodied reasoning from video using binary or multiple-choice questions. They organize common sense by the physical ontology and embodied tasks by completion, affordance, and next-action capabilities.

The evaluation asks binary yes/no or multiple-choice questions grounded in video context and designed to require reasoning.
6.1. Physical Common Sense: The physical common sense benchmark contains 604 questions from 426 video clips selected from an initial pool of 5737 questions.
6.1. Physical Common Sense: The 604 common sense questions cover Space, Time, and Fundamental Physics, with Time representing 49.33% of the benchmark.
6.2. Embodied Reasoning: The embodied reasoning benchmark targets task-completion verification, action affordance, and next plausible action prediction.
6.2. Embodied Reasoning: Unified templates, hierarchical action granularity, and manual choice refinement reduce textual cues and ambiguity in embodied reasoning questions.
6.2. Embodied Reasoning: Embodied evaluation includes held-out RoboVQA and RoboFail clips, BridgeData V2 next-action questions, AgiBot next-subtask questions, HoloAssist subtask questions, and autonomous-vehicle behaviors.

7. Experiments

The experiments evaluate Cosmos-Reason1 on physical common sense, embodied reasoning, and intuitive physics, comparing supervised fine-tuning with reinforcement learning. Physical AI SFT improves benchmark performance, while RL further improves most components but leaves specific challenges.

7.1.1. Physical Common Sense Results: Cosmos-Reason1-7B and Cosmos-Reason1-56B significantly improve physical common sense over their respective backbones, with 56B slightly outperforming OpenAI o1.
7.1.2. Embodied Reasoning Results: Both model variants achieve over a 10% improvement over their respective backbone VLMs on embodied reasoning.
7.1.3. Intuitive Physics Results: Existing models remain near chance on arrow-of-time and object-permanence tasks, whereas Cosmos-Reason1-7B improves significantly across all three intuitive-physics tasks.
7.2.2. Physical Common Sense and Embodied Reasoning Results: Physical AI RL improves performance across most physical common-sense and embodied-reasoning benchmark components, except RoboFail.
7.2.2. Physical Common Sense and Embodied Reasoning Results: RoboFail remains difficult because it requires highly observant perception, comprehensive temporal processing, and reasoning about complex physical constraints.
7.2.2. Physical Common Sense and Embodied Reasoning Results: RL trains the model to evaluate answer choices carefully, reject ambiguous options, and choose a conservative action outside the listed choices when necessary.
7.2.3. Intuitive Physics Results: Intuitive-physics data scale efficiently through video reversal, image-based spatial puzzles, and simulated object-permanence tasks with verifiable rewards.
7.2.3. Intuitive Physics Results: SFT improves all intuitive-physics tasks, RL further enhances spatial-puzzle and object-permanence capabilities, and arrow-of-time reasoning remains challenging.

8. Related Work

Related work spans foundation models, vision-language-action systems, and embodied chain-of-thought methods for physical AI. These approaches seek to connect multimodal perception, sequential reasoning, and action generation across embodiments.

Foundation Models for Physical AI: Early embodied-agent systems used pretrained LLMs and VLMs as static modules for zero-shot natural-language task planning.
Vision-Language-Action Models: End-to-end VLA models fine-tune pretrained vision-language models with robotics data to generate embodied actions and improve task generalization and robustness.
Reasoning for Physical AI: Embodied chain-of-thought methods enable sequential decisions and failure explanation, but most rely on manual prompting to structure reasoning.
Reasoning for Physical AI: CoT-VLA predicts a future image before predicting the action that should interact with the environment to achieve it.
Other Physical Embodiments: VLA models have also been applied to autonomous driving through large-scale vision-language-action datasets for multimodal decision-making.
Vision-Language Models: Vision-language models commonly use decoder-only or other established multimodal architectures, including model families such as Flamingo, LLaVA, InternVL, QwenVL, NVLM, and Llama-3.2-Vision.

9. Conclusion

Cosmos-Reason1 combines ontologies, supervised fine-tuning, and reinforcement learning to improve multimodal physical-world reasoning. The resulting models learn concepts such as the arrow of time and object permanence, supporting open development of Physical AI.

Physical AI SFT improves the backbone VLM’s performance by more than 10% on proposed physical common sense and embodied reasoning benchmarks.
Physical AI RL further boosts accuracy by over 5%.
Cosmos-Reason1 models can learn intuitive physics, including the arrow of time and object permanence, which existing models struggle with.
The authors plan to release code and open-weight models to expedite progress on Physical AI systems that understand and perform complex physical tasks.

A. Prompts Used for Data Curation

The appendix presents prompts for curating physical common-sense and embodied-reasoning data from videos. These prompts generate challenging questions, extract reasoning, predict robot actions, and test spatial and temporal understanding.

A.1. Physical Common Sense Question-Construction Prompt: The physical common-sense question-construction prompt asks for six concise, challenging questions requiring multi-step reasoning from video captions.
A.1. Physical Common Sense Question-Construction Prompt: A.1 questions cover common-sense and planning, spatial reasoning, and temporal reasoning, with two questions allocated to each category.
A.1. Physical Common Sense Question-Construction Prompt: The construction rules require both caption information and external knowledge while forbidding answers obtainable from either source alone.
A.2. Physical Common Sense Reasoning Extraction Prompt: The physical-common-sense extraction prompt supplies a video description and asks the model to answer a hypothetical question about the depicted physical process.
A.3. AgiBot Reasoning Extraction Prompt: The AgiBot prompt describes robot arms, objects, surroundings, and task state, then requests the robot’s next immediate action in a specified format.
A.4. Spatial Puzzle Reasoning Extraction Prompt: The spatial-puzzle prompt presents 32 frames from eight 2×2 image grids and asks which frames match the first image and which lies in a specified direction.
A.5. Arrow of Time Reasoning Extraction Prompt: The arrow-of-time prompt asks whether a frame sequence plays forward or backward and requests visual evidence for the temporal direction.
A.6. Object Permanence Reasoning Extraction Prompt: The appendix also includes an object-permanence reasoning extraction prompt as a distinct curation task.

B. Additional results

Additional examples examine spatial-puzzle reasoning and temporal or object-permanence understanding. The examples include a frame-selection response and reported improvements from reinforcement learning in distinguishing temporal and persistence cues.

Spatial Puzzle: The spatial-puzzle task asks which of 32 frames is most likely to appear to the right of the first frame, with four answer choices.
Spatial Puzzle: The example reasoning identifies the first frame as a weightlifter scene and evaluates candidate frames using their visual content and sequence position.
Spatial Puzzle: The example selects Frame 2 as the most logical right-adjacent choice based on the provided sequence.
Spatial Puzzle: Another reasoning example describes a different first frame as a shirtless, long-haired, bearded man doing pull-ups in a minimalist room.
Temporal Reasoning: RL enables the model to identify key features in the first frame and compare them with subsequent frames instead of associating spatial questions with temporal reasoning.
Object Permanence: RL enables the model to infer that an object’s disappearance is not caused by camera movement, using concise and direct reasoning.

C.1. Core Contributors

The paper’s core contributors span ontology design, data and benchmark curation, model training, reasoning-trace processing, and infrastructure development.

YC, SS, MYL, and TYL defined the physical common-sense and embodied-reasoning ontologies.
Contributors curated physical-common-sense, embodied-reasoning, self-supervised, and extracted-reasoning data and benchmarks.
The team designed captioning methods, trained base VLMs, and built infrastructure for the project.

C.2. Contributors

The Contributors section lists the paper’s authors and summarizes their roles in annotation, model development, training-data curation, infrastructure, ontology, and overall data curation.

The paper lists 16 authors: Hannah Brandon, Jinju Chu, Jenna Diamond, Francesco Ferroni, Rama Govindaraju, Jinwei Gu, Brendan Johnson, Rizwan Khan, Elena Lantz, Yen-Chen Lin, Alice Luo, Andrew Mathau, Yun Ni, Lindsey Pavao, Misha Smelyanskiy, and Yao Xu.
Several contributors collected human annotations, while Jinwei Gu supported the vision encoder and Francesco Ferroni curated training data.
Yun Ni, Rama Govindaraju, and Misha Smelyanskiy supported scaling model-training infrastructure, while Yen-Chen Lin contributed physical common sense ontology insights.
Alice Luo helped with overall data curation.