PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboScape: Physics-informed Embodied World Model

Yu Shang, Xin Zhang, Yinzhou Tang, Lei Jin, Chen Gao, Wei Wu, Yong Li

arXiv:2506.23135v1cs.CVcs.RO

TL;DR

Robotic learning needs scalable data, but existing embodied world models often lack the physical awareness needed for realistic contact-rich video. RoboScape jointly learns RGB generation with temporal depth and keypoint dynamics, achieving strong video-generation performance and supporting policy training and evaluation with generated data.

  • Problem

    Existing embodied world models often lack 3D geometric and motion-dynamic awareness, producing physically unrealistic videos in contact-rich robotic scenarios.

    Source: §1 · p. 1

  • Method

    RoboScape uses a unified multi-task framework combining RGB generation, temporal depth prediction, and adaptive keypoint dynamics learning.

    Source: §1 · p. 2

  • Results

    RoboScape achieves strong RGB and depth prediction performance, improves downstream policy training with synthetic data, and reliably evaluates policies against simulator outcomes.

    Source: §1 · pp. 2–3, §5 · p. 9

  • Takeaways & Limitations

    Physics-informed joint learning enables one world model to support realistic video generation, synthetic-data policy learning, and policy evaluation.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while addressing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly in modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical properties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-informed world models to advance embodied intelligence research. The code is available at: https://github.com/tsinghua-fib-lab/RoboScape.

1 Introduction

RoboScape addresses the physical limitations of embodied world models by jointly learning RGB generation with temporal depth and keypoint dynamics. The model targets scalable robotic learning through unified physical modeling and demonstrates strong video-generation and downstream utility.

1 Introduction

  • Existing embodied world models often optimize RGB pixels without physical awareness, causing implausible motion, spatial inconsistency, and deformation artifacts in contact-rich manipulation.These failures can compromise policies learned from generated data, especially for deformable objects such as cloth.Source: §1 · pp. 1–2
  • The unified framework avoids complex external model cascades by injecting temporal depth features into RGB prediction and enforcing temporal consistency on dynamically sampled keypoints.This design aims to combine global spatial understanding with fine-grained object-motion modeling in one efficient training framework.Source: §1 · p. 2
  • RoboScape achieves state-of-the-art performance in RGB and depth prediction while providing a superior balance between these metrics and baseline world models.The video-generation evaluation compares RoboScape with four advanced embodied and general world models across six metrics.Source: §1 · p. 2
  • Synthetic data from RoboScape improves classic robotic policies including Diffusion Policy [29] and pi0 [30], while the model also evaluates policies with strong correlation to ground-truth simulator outcomes.These experiments assess synthetic-data policy training and policy evaluation as downstream applications.Source: §1 · pp. 2–3
  • RoboScape jointly learns RGB video generation, temporal depth prediction, and adaptive keypoint tracking to produce visually faithful and physically plausible robotic videos.Temporal depth captures global 3D structure, while keypoint dynamics model local deformation, motion, and implicit material properties.Source: §1 · p. 2
  • The work includes an automated robotic data-processing pipeline with physical-prior labels and a curated large-scale, high-quality dataset supporting visual quality, geometric accuracy, and action controllability.The pipeline is illustrated in Figure 1 and supports the multimodal data required for physics-informed world-model learning.Source: §1 · p. 2, Figure 1 · p. 3

2 Methodology

RoboScape formulates action-conditioned embodied video prediction and builds a multimodal pipeline with physical-prior annotations. Its unified autoregressive model jointly predicts RGB, temporal depth, and keypoint dynamics to improve geometric and physical consistency.

2.1 Problem Formulation

  • The world model predicts the next visual observation from past observations and robot actions, with frames represented as RGB images and actions as continuous control vectors.Source: §2.1 · p. 3

2.2 Robotic Data Processing Pipeline with Physical Priors Annotation

  • Temporal depth maps and tracked keypoint trajectories provide physical priors extracted from videos using pretrained depth and tracking models.Source: §2.2 · p. 3
  • The data pipeline slices videos by camera boundaries and action semantics, filters low-quality clips, and categorizes scenes and difficulty for curriculum learning.Source: §2.2 · p. 3

2.3 RoboScape: A Physics-informed Embodied World Model

  • RoboScape uses an autoregressive Transformer with RGB and depth branches, jointly adding temporal depth prediction and adaptive keypoint dynamics learning to model global geometry and local motion.Source: §2.3 · p. 4
  • The branches process historical latent tokens conditioned on robot actions and positions, use additive action fusion, and inject projected depth features into RGB features.Source: §2.3 · pp. 4–5
  • Keypoint dynamics learning selects highly active trajectories, aligns sampled keypoint tokens across time, and applies trajectory-guided attention to emphasize difficult dynamic regions.Source: §2.3 · p. 5

3 Experiments

RoboScape is evaluated across video quality, synthetic-data policy learning, and policy evaluation. Results show strong video prediction, useful generated data, and high agreement with the ground-truth simulator.

3.1 Experimental Settings

  • Experiments use 50,000 AgiBotWorld-Beta videos spanning 147 tasks and 72 skills, with comparisons against embodied and general world-model baselines.The evaluation covers appearance fidelity, geometric consistency, and action controllability.Source: §3.1 · p. 6, §3.2 · p. 6

3.2 Video Quality Evaluation

  • RoboScape consistently outperforms four baselines across six video-generation metrics, combining visual fidelity with action controllability.The comparison includes IRASim, iVideoGPT, Genie, and CogVideoX.Source: §3.2 · p. 7, Table 1 · p. 7
  • Removing either temporal depth learning or keypoint dynamics learning degrades performance, respectively causing geometric distortions or unreal motion patterns.The ablation study identifies complementary contributions from both physics-informed components.Source: §3.2 · p. 7, Table 2 · p. 7, Figure 4 · p. 8

3.3 Robotic Policy Learning with Synthetic Data

  • Generated data improves policy learning as its quantity increases, with Diffusion Policy using only generated data nearly matching training with real data on Robomimic Lift.The trend also holds for π0 on the more demanding LIBERO tasks after a small real-data warm-up.Source: §3.3 · pp. 7–8, Table 3 · p. 8

3.4 Robotic Policy Evaluation

  • 0.953 Pearson correlation with the ground-truth simulator shows RoboScape can serve as a better robotic policy evaluator than the compared models.Policies were evaluated over 100 runs in both the simulator and world models.Source: §3.4 · p. 8, Figure 5 · p. 9

4 Related Work

Related work frames world models as predictors of future environmental states and distinguishes diffusion from autoregressive generation. Physics-aware video methods use explicit representations or implicit training objectives, motivating unified physical-information modeling.

4.1 World Model

  • World models predict future environmental states from current observations and actions, with diffusion and autoregressive models forming the main modeling categories.Applications include autonomous driving, embodied intelligence, and gaming.Source: §4.1 · p. 9

4.2 Physics-aware Generative Model

  • Physics-aware video generation includes explicit texture or material representations and implicit loss-based or joint-generation approaches.The paper positions unified RGB, temporal-depth, and keypoint-dynamics learning within this landscape.Source: §4.2 · p. 9

5 Conclusion and Future Work

RoboScape integrates physical knowledge into video generation through multi-task learning without cascaded external physics engines. Its evaluations show gains in video quality, downstream policy training, and policy evaluation.

5 Conclusion and Future Work

  • RoboScape uses temporal depth prediction for 3D scene structure and dynamic keypoint learning for object deformation and motion patterns.The unified framework is designed to improve both visual and physical realism.Source: §5 · p. 9
  • The model outperforms baselines in video generation, synthetic-data utility for manipulation-policy training, and policy-evaluation effectiveness.Future work will test the generative world model with real robots.Source: §5 · p. 9

A Broader Impacts

The broader-impact discussion links physics-plausible synthetic data to reduced real-world data demands and more efficient robotic training. It also identifies synthetic-data diversity and governance as deployment considerations.

A Broader Impacts

  • Physics-plausible synthetic data may reduce reliance on costly real-world collection and support safer, more efficient training for assistive, disaster-response, and industrial robots.The discussion also notes that computational efficiency could lower barriers to broader research participation.Source: A Broader Impacts · p. 14
  • Responsible deployment remains bounded by unresolved synthetic-data diversity and governance considerations in safety-critical domains.These are identified as future requirements rather than demonstrated capabilities.Source: A Broader Impacts · p. 14

B Baseline Details

  • The compared baselines include IRASim, iVideoGPT, Genie, and CogVideoX, spanning embodied and general video-generation models.IRASim and CogVideoX are described as DiT-based, while iVideoGPT and Genie are autoregressive or foundation world models.Source: B Baseline Details · p. 14

C Scaling Behavior of RoboScape

RoboScape improves as both model capacity and training-data scale increase, although geometric accuracy responds less consistently to additional data.

Model scaling

  • All six evaluation metrics improve significantly as model capacity increases across RoboScape-S (34M), RoboScape-M (131M), and RoboScape-L (544M).The scaling study evaluates three model variants and reports a clear model-capacity scaling law.Source: C Scaling Behavior of RoboScape · p. 14

Data scaling

  • In RoboScape-S, increasing training data from 1,000K to 6,000K clips consistently enhances visual quality, with the data-scale study evaluating 1,000K, 3,000K, and 6,000K clips.Source: C Scaling Behavior of RoboScape · p. 14
  • Geometric accuracy shows marginal improvement or slight degradation with more data because smaller datasets can overfit to the conditional input’s final frame.Such overfitting can artificially inflate geometric metrics without producing meaningful temporal dynamics.Source: C Scaling Behavior of RoboScape · p. 15

Scaling implications

D.1 Video Generation Results

RoboScape generates future robotic video frames from an initial condition and provides additional qualitative generation results.

Video generation

  • Additional visualizations present generated videos from RoboScape, with only the subsequent eight frames shown.Source: §D.1 · p. 15

D.2 Robotic Policy Learning

RoboScape-generated data is evaluated for robotic policy learning on Robomimic and LIBERO tasks.

Robotic policy learning

  • Generated-data visualizations are provided for Robomimic and LIBERO, supporting the downstream policy-learning experiments.Source: §D.2 · p. 15

D.3 Robotic Policy Evaluation (add visualization results of our model and baselines

RoboScape and baseline models are compared qualitatively in robotic policy evaluation, including both successful and failed rollouts.

Policy evaluation

Supplementary visualizations

  • Supplementary visualizations show model-generated sequences for comparison, including eight-frame rollouts on Robomimic and LIBERO.The Robomimic visualization displays every fifth frame from t=0 to t=40, while the LIBERO visualization displays every tenth frame from t=0 to t=80.Source: Figure 9 · p. 16, Figure 10 · p. 16, Figure 8 · p. 16
Loading 2506.23135v1…