Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning

• The paper defines a hierarchical physical common-sense ontology spanning Space, Time, and Fundamental Physics, plus a two-dimensional embodied-reasoning ontology covering four capabilities across five agent types.The physical common-sense ontology further divides into 16 fine-grained subcategories.Source: §1 · p. 1
• Cosmos-Reason1-7B and Cosmos-Reason1-56B process video through a vision encoder and projector before multimodal reasoning, and the code and pretrained models are released under the NVIDIA Open Model License.The models use long chain-of-thought reasoning to produce natural-language explanations and embodied decisions, such as a next action.Source: §1 · pp. 2–3
• Physical AI reinforcement learning uses rule-based, verifiable multiple-choice-question rewards grounded in human annotations or automatically generated from video structure.Video-derived questions include solving puzzles with shuffled spatiotemporal patches and predicting whether a video plays forward or backward.Source: §1 · p. 2
• The team curates around 4M video-text annotations—including physical-understanding captions, multiple-choice questions, and long chain-of-thought traces—using ontology-based pipelines and human annotation or model distillation.The distillation source is DeepSeek-R1.Source: §1 · p. 2
• The evaluation suite includes three physical common-sense benchmarks with 604 questions from 426 videos and six embodied-reasoning benchmarks with 610 questions from 600 videos.The embodied benchmarks span tasks involving humans, robot arms, humanoid robots, and autonomous vehicles.Source: §1 · pp. 2–3
• Rule-based, verifiable reinforcement-learning rewards improve Cosmos-Reason1 on all evaluated benchmarks.The paper explores rewards from human-annotated multiple-choice questions and automatically generated video-structure questions, including shuffled-patch puzzles and forward-versus-backward prediction.Source: §1 · p. 3

2. Physical AI Reasoning

• Physical AI reasoning combines embodiment-agnostic physical common sense with embodied reasoning for planning future interactions, spanning fast intuitive and slower deliberate judgment.The section frames both capabilities as important for Physical AI models.Source: §2 · p. 3

2.1. Common Sense Reasoning

• Physical common sense captures what is possible, impossible, or likely in the world, helping AI learn skills with less trial and error and avoid critical mistakes.The paper motivates this capability by noting that real-world AI training is expensive and can pose risks to systems and their surroundings.Source: §2.1 · p. 3
• The common-sense ontology defines 16 capabilities across Space, Time, and Fundamental Physics without prescribing the mechanisms or embodiments used to achieve them.Its subcategories cover spatial relations and environment; actions, event order, causality, camera, and planning; and physical attributes, states, object permanence, mechanics, electromagnetism, thermodynamics, and anti-physics.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2. Embodied Reasoning

• Embodied reasoning grounds decisions in action and requires processing complex sensory inputs, predicting action effects, respecting physical constraints, and learning from interaction.It must handle raw, incomplete, and ambiguous inputs, anticipate physical consequences, plan feasible long-horizon actions, and update behavior from feedback.Source: §2.2 · p. 4
• The embodied-reasoning ontology spans humans, animals, and varied robot forms, organizing capabilities and physical embodiments in two dimensions.The shared skills include navigating, manipulating, and adapting decisions to environmental conditions and task goals.Source: §2.2 · p. 4
• The paper studies the first three embodied capabilities using video, task-completion verification, next-plausible-action prediction, and action affordance, while leaving learning from interaction for future work.Its videos cover humans, robot arms, humanoid robots, and autonomous vehicles.Source: §2.2 · p. 5

3.1. Multimodal Architecture

• Both models use a decoder-only multimodal architecture that aligns image or video tokens with text embeddings before the LLM backbone.The architecture uses a vision encoder and a projector with a downsampling two-layer MLP; Figure 3 illustrates the flow from video and text inputs to the LLM and its responses.Source: §3.1 · p. 6, Figure 3 · p. 7
• Cosmos-Reason1-7B uses Qwen2.5-VL, while Cosmos-Reason1-56B uses InternViT-300M-V2.5 with Nemotron-H.For the 56B model, images are divided into 1–12 448 × 448 tiles plus a thumbnail; videos use up to 32 frames at 2 fps, resized to 448 × 448, with each frame’s 1,024 visual tokens downsampled to 256.Source: §3.1 · p. 6

3.2. Hybrid Mamba-MLP-Transformer Backbone

• Cosmos-Reason1-56B uses a hybrid Mamba-MLP-Transformer backbone to combine linear-time sequence modeling with Transformer layers for long-context modeling.The 7B dense model was trained with TP=4, while the 56B hybrid model used TP=8 and PP=2; Figure 4 illustrates the hybrid backbone.Source: §3.2 · pp. 7–8, Figure 4 · p. 8

4. Reinforcement Learning

• Physical AI-focused RL follows SFT to further enhance the models’ physics common sense and embodied reasoning abilities.The paper describes RL as the second of two stages for adapting a pretrained visual language model to Physical AI reasoning.Source: §4 · p. 8

4.1. Algorithm

• The models use GRPO (Shao et al., 2024), which avoids a separate critic by normalizing rewards among a cohort of responses to each prompt to derive advantages.The paper selects GRPO for its simplicity and computational efficiency.Source: §4.1 · p. 8

4.2. Training Framework

• The asynchronous framework improves training efficiency by approximately 160% over colocated frameworks through heterogeneous policy-training and actor-rollout deployment and end-to-end asynchronous scheduling.Its dispatcher schedules prompts across the pipeline; the framework architecture also includes actor rollout and policy training components.Source: §4.2 · p. 8, Figure 5 · p. 9
• The framework can reconfigure and continue training after node failures without costly restarts, while dispatcher redundancy and dynamic scaling support robustness and flexible resource allocation.Scaling is supported both up and down according to workload demands.Source: §4.2 · pp. 8–9

5.1. Physical AI Supervised Fine-Tuning

• Physical AI SFT data targets physical common-sense and embodied reasoning, using curated VQA and cross-embodiment demonstrations with understanding annotations and long chain-of-thought traces.Some VQA pairs are generated through a model-in-the-loop pipeline; the SFT data also includes intuitive-physics tasks.Source: §5.1 · pp. 9–10

5.1.1. Physical Common Sense SFT

• Caption-grounded physical common-sense VQA pairs cover video understanding and hypothetical spatial, temporal, and common-sense reasoning, with DeepSeek-R1 traces cleaned into SFT samples.Free-form data uses 9.9K human-captioned videos to produce ∼99k understanding and ∼59.4k reasoning samples; VLM-captioned clips yield ∼2.4M understanding MCQs and ∼600k reasoning MCQs.Source: §5.1.1 · pp. 10–11

5.1.2. Embodied Reasoning SFT

• Embodied SFT targets task-completion verification, action affordance, and next-action prediction through short-horizon clips, state-action captions, QA pairs, and reasoning traces.The pipeline segments demonstrations, uses VLM captions (human captions for AV data), elicits traces from DeepSeek-R1, then cleans and rewrites them.Source: §5.1.2 · pp. 11–12
• Embodied data spans robot manipulation, egocentric human activity, and autonomous driving, with source-specific questions about plausible next actions or subtasks.RoboVQA yields ∼930k QA pairs with traces across all three target properties; AgiBot yields 19.8K clips for next-subtask prediction, while HoloAssist yields 139,653 clips and AV uses ∼12.4K human-captioned videos.Source: §5.1.2 · pp. 13–14

5.1.3. Intuitive Physics SFT: Spatial Puzzles, Arrow-of-Time and Object Permanence

• Intuitive-physics SFT trains spatial continuity, temporal order, and object permanence using spatial puzzles, forward/reversed videos, and simulated manipulation clips.The resulting datasets contain 11k spatial-puzzle videos, 30,000 short clips and their reversed versions, and 10K object-permanence clips; object-permanence traces come from an intermediate Cosmos-Reason1-7B model.Source: §5.1.3 · pp. 14–15

5.2. Physical AI Reinforcement Learning

• RL makes open-ended reasoning verifiable by converting SFT reasoning data into single-answer MCQs, manually checking quality, and balancing options where needed.Physical common-sense RL uses 5,133 questions from 1,989 videos, split into easy and hard subsets by four-model performance; embodied RL selects 200-250 samples per source, and intuitive-physics RL uses 24,079 samples.Source: §5.2 · pp. 15–16

6. Benchmark

• Questions require reasoning from video context, and evaluation measures only final-answer accuracy, not the quality of the reasoning trace.Questions use binary yes/no or multiple-choice formats.Source: §6 · p. 16

6.1. Physical Common Sense Reasoning

• The physical common-sense set contains 604 manually selected questions from 426 clips, covering Space (80), Time (298), and Fundamental Physics (226).The questions were curated from an initial pool of 5,737 questions: 2,828 binary and 2,909 multiple-choice.Source: §6.1 · pp. 16–17

6.2. Embodied Reasoning

• Embodied questions cover task-completion verification, action affordance, and next-action prediction, with unified templates, action granularity, and manually refined choices to promote visual reasoning.Actions are organized hierarchically as atomic actions, coarser subtasks, and dataset-specific goals; refinement addresses ambiguous options and encourages use of the full clip context.Source: §6.2 · p. 17
• The benchmark pairs 101 validation clips from RoboVQA with 100 harder curated RoboFail examples and 100-clip QA sets from BridgeData V2, AgiBot, and HoloAssist.RoboVQA clips came from the validation split excluded from SFT training. RoboFail tests temporal context, physical constraints, and nuanced questions; the other datasets ask about plausible next actions or subtasks.Source: §6.2 · pp. 17–18

7.1. Physical AI Supervised Fine-Tuning

• SFT uses balanced domain sampling and reports model accuracy averaged over five inferences, while evaluating other models with zero-shot chain-of-thought prompting.Cosmos-Reason1-7B is trained for 12.5K iterations; Cosmos-Reason1-56B is trained for 30K iterations followed by 20K at a decayed learning rate.Source: §7.1 · p. 18

7.1.1. Physical Common Sense Results

• Cosmos-Reason1-56B achieves the best physical common-sense accuracy, slightly outperforming OpenAI o1, while both Cosmos-Reason1 variants improve over their respective backbones.The authors attribute the gains to their curated common-sense dataset, which they describe as a foundation for further RL improvements.Source: §7.1.1 · pp. 18–19, Table 7 · p. 19

7.1.2. Embodied Reasoning Results

• Both Cosmos-Reason1 variants improve embodied-reasoning results by over 10% compared with their respective backbone VLMs.The authors attribute the gains to Physical AI SFT.Source: Table 8 · p. 19, §7.1.2 · p. 19

7.2.1. Experiment Setup

• RL uses accuracy and format rewards, balanced dataset sampling, and dynamically shuffled multiple-choice options to train Physical AI reasoning.Accuracy is checked by matching the answer tag to ground truth; format is checked for separate think and answer tags. Training uses nine sampled outputs per question and runs for 500 iterations.Source: §7.2.1 · p. 20

7.2.2. Physical Common Sense and Embodied Reasoning Results

• Physical AI RL improves most physical common-sense and embodied-reasoning benchmark components, but RoboFail remains consistently challenging through SFT and RL.RoboFail tests difficult real-world action affordance and task-completion verification scenarios, including cases requiring detailed perception, temporal context, and reasoning about complex physical constraints.Source: Table 9 · p. 20, §7.2.2 · p. 20

7.1.3. Intuitive Physics Results

• Existing models struggle above chance on arrow-of-time and object-permanence tasks, while GPT-4o and OpenAI o1 perform better on spatial puzzles; the 7B model improves across all three tasks.The authors argue that common benchmarks may not capture physical-world understanding despite models’ performance on standard benchmarks such as MMMU.Source: §7.1.3 · p. 19, Table 10 · p. 20

7.2.3. Intuitive Physics Results

• After RL, Cosmos-Reason1 further improves spatial-puzzle and object-permanence capabilities, while arrow-of-time reasoning remains challenging.The RL model can identify anti-physics motion despite stationary distractors; spatial-puzzle training supports systematic cross-frame comparison, and RL yields more direct object-permanence reasoning than the non-RL model.Source: Table 10 · p. 20, §7.2.3 · p. 22
• Intuitive-physics training data is readily scalable and adaptable to RL with verifiable rewards, enabling a larger dataset with minimal human effort.The paper describes video reversal for arrow-of-time data, image-based spatial puzzles, and object-permanence data from simulated environments.Source: §7.2.3 · p. 22

8.1. Foundation Models for Physical AI

• Pre-trained LLMs served as zero-shot task planners (Li et al., 2022; Song et al., 2023; Huang et al., 2022; Ahn et al., 2022) or code generators (Liang et al., 2022); pre-trained models, especially VLMs, often lacked physical common sense.Source: §8.1 · p. 22
• End-to-end VLA models fine-tuned on robotics data show promise for generalization and robustness, while hierarchical variants separate language-driven planning from low-level motor control (Shi et al., 2025; Li et al., 2025).The hierarchical architectures are described as improving task decomposition.Source: §8.1 · p. 22
• Embodied chain-of-thought methods support sequential decisions and failure analysis, but often rely on manual prompting; CoT-VLA predicts a future image before the action (Zawalski et al., 2024; Liu et al., 2023; Elhafsi et al., 2023; Zhao et al., 2025).The cited work frames manual prompting as limiting autonomous adaptation and generalization.Source: §8.1 · p. 24

8.2. Vision Language Models

• Decoder-only VLMs integrate image tokens into LLM self-attention, while cross-attention VLMs process them through cross-attention; Dai et al. (2024) found stronger visual-context reasoning for decoder-only models, which this work adopts.The comparison concerns college-level multidisciplinary knowledge and mathematical reasoning tasks in a visual context.Source: §8.2 · p. 24

8.3. LLMs and VLMs with Reasoning Capabilities

• Prior LLM reasoning work and multimodal reasoning largely focused on coding, mathematics, and STEM, while recent efforts bring R1-style reasoning to VLMs; this work explores reasoning for Physical AI (Jaech et al., 2024; DeepSeek-AI, 2025).The passage also cites chain-of-thought prompting as enhancing LLM reasoning (Wei et al., 2022).Source: §8.3 · p. 24

• Physical AI SFT improves backbone VLM performance by more than 10% on the proposed physical common-sense and embodied-reasoning benchmarks, while RL further boosts accuracy by over 5%.Source: §9 · p. 24
• The work defines foundational Physical AI ontologies and constructs supervised fine-tuning data and benchmarks for physical common sense and embodied reasoning.Source: §9 · p. 24
• With SFT and RL, Cosmos-Reason1 can learn intuitive physics, including the arrow of time and object permanence, which existing models struggle with.Source: §9 · p. 25
• The authors plan to release the code as open-source and the models as open-weight to advance Physical AI systems.Source: §9 · p. 25

A.1. Physical Common Sense Question-Construction Prompt

• Given a video caption, generate six challenging, multi-step questions: two each for common-sense and planning, spatial, and temporal reasoning, with each question limited to 10 words.The prompt supplies the caption as the input.Source: §A.1 · p. 26
• Questions should combine caption information with external knowledge, avoid answers stated directly or answerable from external knowledge alone, and draw on physical, spatial, and temporal reasoning.Suggested categories include physical attributes and state changes, spatial plausibility and affordances, and action understanding, event order, and planning.Source: §A.1 · p. 26

A.2. Physical Common Sense Reasoning Extraction Prompt

• For physical common-sense extraction, the example presents a boiling-noodle video and asks what would happen if the pot were not boiling.Source: §A.2 · p. 27

A.3. AgiBot Reasoning Extraction Prompt

• For AgiBot reasoning extraction, the prompt describes two robot arms handling grapes and an apple, then requests the robot’s next immediate action in a specified response format.The apple-holding arm releases the apple into a shopping-cart bag, while the other arm holds grapes above the fruit stand.Source: §A.3 · p. 28

A.4. Spatial Puzzle Reasoning Extraction Prompt

• For spatial-puzzle extraction, identify three of 31 frames that match the first frame’s image, then select which of those lies in a specified direction.The puzzle contains eight images, each divided into a 2×2 grid, and requires a prescribed answer format.Source: §A.4 · p. 29

A.5. Arrow of Time Reasoning Extraction Prompt

• For arrow-of-time extraction, the prompt asks for visual clues that show a forward-playing video is not reversed, requesting evidence without a verdict or bullet points.Source: §A.5 · p. 29

A.6. Object Permanence Reasoning Extraction Prompt

• For object-permanence extraction, the example asks whether an akita black bowl that becomes occluded and vanishes unexpectedly contradicts object permanence, with reasoning and answer in tagged fields.The prompt notes that both arm movement and camera movement can cause occlusion.Source: §A.6 · p. 29

B. Additional results

• The puzzle presents 32 frames and asks which of four options—Frames 2, 30, 9, or 11—is most likely to be right of the first frame.The prompt asks for a spatial relation to the first frame.Source: B. Additional results · p. 30
• The pre-RL rationale chooses Frame 2 as the right-side frame because it follows Frame 1 in sequence, despite noting that right-side position is not necessarily temporal order.It concludes that Frame 2 is the most logical choice based on sequence.Source: B. Additional results · p. 30
• RL enables the model to identify key features in the first frame and compare them with other frames, correcting its pre-RL confusion of spatial questions with temporal reasoning.Figure 11 contrasts the pre-RL error with the RL-enabled reasoning.Source: Figure 11 · p. 30, B. Additional results · p. 30

C.1. Core Contributors

• The core-contributor roster includes Alisson Azzolini, Junjie Bai, Jiaxin Cao, and many others spanning the credited project roles.The passage lists the core contributors by name; their contributions are detailed separately.Source: §C.1 · p. 32
• Core contributors defined the physical common sense and embodied reasoning ontologies, designed the overall system, and organized paper writing.YC, SS, MYL, and TYL defined the ontologies; MYL and TYL designed the system; YC and TYL organized writing.Source: §C.1 · p. 32
• Core contributors curated physical common sense, embodied reasoning, and self-supervised data and benchmarks, and designed a captioning method.YC, PC, JX, AZW, TYL, XY, FW, XL, SG, ZL, JJ, ZH, LT, and others are credited across these efforts.Source: §C.1 · p. 32
• Core contributors trained base VLMs and Physical AI supervised fine-tuned and reinforcement-learning models, and evaluated the reasoning models.The passage credits separate teams for base VLM training, supervised fine-tuning, reinforcement learning, and reasoning-model evaluation.Source: §C.1 · p. 32
• Core contributors built infrastructure for supervised fine-tuning and reinforcement learning.The passage identifies distinct contributors for the two infrastructure efforts.Source: §C.1 · p. 32

C.2. Contributors

• The additional-contributor roster names Hannah Brandon, Jinju Chu, Jenna Diamond, and other contributors acknowledged for project support.The passage lists the additional contributors by name; their contributions are detailed separately.Source: §C.2 · p. 32
• Additional contributors helped collect human annotations, supported vision-encoder work, curated training data, and assisted with overall data curation.The passage credits YX, LP, AM, RK, JC, BJ, EL, HB, and JD with annotations; JG with vision-encoder work; FF with training data; and AL with overall curation.Source: §C.2 · p. 32
• Additional contributors supported scaling model-training infrastructure and provided insights on the physical common sense ontology.YN, RG, and MS supported infrastructure scaling; YCL provided ontology insights.Source: §C.2 · p. 32