Source-linked AI summary
Robot Learning from a Physical World Model
Jiageng Mao, Sicheng He, Hao-Ning Wu, Yang You, Shuyang Sun, Zhicheng Wang, Yanan Bao, Huizhong Chen, Leonidas Guibas, Vitor Guizilini, Howard Zhou, Yue Wang
TL;DR
Generated videos offer visual guidance for manipulation, but translating their motions into accurate robotic actions remains difficult when physical constraints are not modeled. PhysWorld reconstructs a physical world from generated videos and trains object-centric residual policies with that model’s feedback. On diverse real-world tasks, it reports the highest average success rate among compared methods, while noting that simulator fidelity may introduce sim-to-real gaps.
Problem
Generated videos provide pixel-level task guidance, but directly retargeting their motions can neglect physical constraints and produce inaccurate manipulation.
Source: §I · p. 1Method
PhysWorld reconstructs a physically interactable scene from generated videos and uses object-centric residual reinforcement learning to turn visual demonstrations into feasible actions.
Source: §I · p. 2, A. Physical World Modeling from Video… · p. 3, B. Object-Centric Learning from the… · p. 5Results
PhysWorld achieves the highest average success rate at 82%, compared with 67% for the second-best method [8], across the reported real-world task comparison.
Source: B. World Modeling Improves… · p. 6Takeaways & Limitations
The framework is presented as enabling zero-shot generalizable real-world manipulation without real-world data collection.
Source: §I · p. 2, §V · p. 7Physical world modeling is bounded by simulator fidelity and may introduce additional sim-to-real gaps.
Source: §V · p. 7
Abstract
from arXiv · showhide
We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic visual demonstrations from language commands and images, offering a powerful yet underexplored source of training signals for robotics. However, directly retargeting pixel motions from generated videos to robots neglects physics, often resulting in inaccurate manipulations. PhysWorld addresses this limitation by coupling video generation with physical world reconstruction. Given a single image and a task command, our method generates task-conditioned videos and reconstructs the underlying physical world from the videos, and the generated video motions are grounded into physically accurate actions through object-centric residual reinforcement learning with the physical world model. This synergy transforms implicit visual guidance into physically executable robotic trajectories, eliminating the need for real robot data collection and enabling zero-shot generalizable robotic manipulation. Experiments on diverse real-world tasks demonstrate that PhysWorld substantially improves manipulation accuracy compared to previous approaches. Visit \href{https://pointscoder.github.io/PhysWorld_Web/}{the project webpage} for details.
I. INTRODUCTION
Video generation can provide task-conditioned visual demonstrations, but translating their pixel motions into accurate robotic actions remains difficult. PhysWorld couples video generation with physical world reconstruction and object-centric residual reinforcement learning; it reports improved accuracy across real-world tasks.
I. INTRODUCTION
- Video-generation models can synthesize photorealistic task demonstrations from images and language, and their videos capture object dynamics and embodiment motions useful for policy learning.Their generalization across scenarios is attributed to training on large-scale Internet data.Source: §I · p. 1
- Directly retargeting generated video motions neglects physical constraints, while prior alignment methods generally require costly, labor-intensive real-world demonstrations –.Other approaches extract actions from visual cues such as flows, sparse tracks, or object poses –[8].Source: §I · p. 1
- PhysWorld identifies physical feasibility as the bottleneck: generated videos offer visual plausibility, whereas real-world robot interaction requires physically accurate actions.Its proxy physical world model provides feedback for learning to imitate generated motions consistently with physics.Source: §I · p. 1
- Given one RGB-D image and a task prompt, PhysWorld generates a task-conditioned video, reconstructs an interactable physical scene, and learns object-centric residual policies for executable actions.The approach is described as avoiding real-world data collection and enabling zero-shot generalizable manipulation.Source: §I · p. 2
- PhysWorld’s experiments show substantial accuracy improvements across diverse real-world manipulation tasks and report consistent outperformance over existing approaches.The framework is presented as enabling physically grounded, generalizable manipulation.Source: §I · p. 2
II. RELATED WORKS
Prior video-based robot learning either trains action models with real demonstrations or extracts actions directly from visual cues, while real-to-sim reconstruction often requires dedicated multi-view captures. PhysWorld instead uses generated videos and a proxy physical world model for physically informed action learning.
II. RELATED WORKS
- Action models trained on generated video frames require large amounts of real robotic demonstrations, whereas direct visual-cue approaches can neglect physical plausibility and yield inaccurate manipulation –[8],,.PhysWorld uses a proxy physical world model to provide feedback for imitation of generated motions.Source: §II · p. 2
- Video-based robot learning also uses transferable representations, embodiment-agnostic motion tracking, real-to-sim reconstruction, and reinforcement learning –.Related work shares ideas with PhysWorld in object-pose tracking, reconstruction, and reinforcement learning [8],,,,, [47].Source: §II · p. 2
- Unlike approaches relying on laboratory reconstruction and human demonstrations, PhysWorld uses a single generated video to learn physical actions despite motion blur or visual hallucinations in in-the-wild videos [36], [41]–[47].The cited related-work discussion contrasts those settings with PhysWorld’s use of generated videos.Source: §II · p. 2
- Many real-to-sim methods require dedicated multi-view captures to reconstruct complete textured meshes or Gaussian splats, limiting their fit to monocular generated videos –.PhysWorld uses generative priors to model physical scenes from single-view video without additional multi-view capture.Source: §II · p. 2
III. METHOD
PhysWorld targets open-world manipulation from an RGB-D image and language command. It combines video-generation guidance with physical-world feedback to produce physically feasible actions.
III. METHOD
- The system takes an RGB-D image and language-based task command as input and outputs physically feasible actions to complete the task.The paper frames this as open-world robotic manipulation.Source: §III · p. 2
- Video generation supplies pixel-level task guidance, while the physical world model supplies feedback for learning from that guidance.The method unifies these components rather than treating generated motion as directly executable.Source: §III · p. 2
- The method first models the physical world from generated videos, then learns robotic actions from that model.These components are described in Sections III-A and III-B, respectively.Source: §III · p. 2
A. Physical World Modeling from Video Generation
PhysWorld converts generated video into a physically interactable scene by reconstructing geometry and motion, generating meshes, and aligning them with physical constraints. The resulting digital twin supplies grounded feedback for learning executable actions.
A. Physical World Modeling from Video Generation
- Generated-video motions become targets for policy learning after the method estimates a 4D spatiotemporal representation, creates textured meshes, and constructs the physical scene.The pipeline addresses partial observations and visual artifacts using generative priors.Source: A. Physical World Modeling from Video… · p. 2
- The framework accepts image-to-video models with text control to generate future frames depicting task completion, and primarily uses Veo3 for output quality.Additional video-generation models are evaluated in ablation studies.Source: A. Physical World Modeling from Video… · pp. 2–3
- MegaSaM initializes dynamic-scene reconstruction, and the method calibrates its depth estimates to metric scale using the captured RGB-D image’s depth.A global scale and shift are fit over valid pixels with a robust regression objective, then applied across frames.Source: A. Physical World Modeling from Video… · p. 3
- Because depth and point clouds are not directly usable for physics simulation, the method generates complete object and background meshes from partially observed monocular videos.It separates object crops, inpaints missing background imagery, and uses an image-to-3D generator for textured object meshes.Source: A. Physical World Modeling from Video… · p. 3
- For occluded background geometry, the reconstruction assumes objects rest on the background and fills hidden regions using supporting planes or scene boundaries.Object and background meshes are then registered and resized to match the observed point cloud.Source: A. Physical World Modeling from Video… · p. 3
- The reconstructed meshes become physically interactable through physical-property estimation, gravity alignment, and collision optimization.A vision-language model estimates typical physical parameters such as mass and friction from object categories.Source: A. Physical World Modeling from Video… · p. 4
- Gravity alignment rotates the meshes from camera coordinates to the world frame, using the estimated ground-plane normal to align the scene with world up.The method estimates the plane normal with RANSAC and computes the minimal aligning rotation.Source: A. Physical World Modeling from Video… · p. 4
- Collision optimization adjusts object placement relative to the background to prevent initial collisions and preserve clearance for simulation.It uses a background signed distance field and vertical object translations; the resulting digital twin provides physical feedback for learning executable actions.Source: A. Physical World Modeling from Video… · p. 4
B. Object-Centric Learning from the Physical World Model
PhysWorld uses object-pose motion from generated videos as policy-learning targets, then combines grasping and planning baselines with residual reinforcement learning under physical-world feedback.
B. Object-Centric Learning from the Physical World Model
- Object-centric learning tracks object poses from generated videos, which are treated as more robust motion targets than optical flow or object tracks.The framework supports other forms of motion supervision, left for future exploration.Source: B. Object-Centric Learning from the… · p. 4
- Residual reinforcement learning adds policy-learned corrections to grasping and motion-planning baseline actions, enabling adaptation through physical-world feedback.The baseline itself need not succeed because learned residuals can rectify imperfect actions.Source: B. Object-Centric Learning from the… · pp. 4–5
- The policy is trained with rewards for tracking target object poses, maintaining stable grasping and movement, and discouraging actions that fail inverse kinematics or motion planning.PPO is the learning algorithm, and baseline actions accelerate convergence by limiting learning to residual corrections.Source: B. Object-Centric Learning from the… · p. 5
IV. EXPERIMENTS
The experiments evaluate PhysWorld’s video-generation and physical-world-modeling components, its ability to generalize without task-specific robot demonstrations, and key design choices and limitations. They are organized around questions about video generation, world modeling, and object-centric residual reinforcement learning.
IV. EXPERIMENTS
- The study asks whether video generation enables more generalizable robotic manipulation.This is its first empirical question.Source: §IV · p. 5
- The study asks whether physical world modeling improves robustness in manipulation tasks.This is its second empirical question.Source: §IV · p. 5
- The study asks whether object-centric residual reinforcement learning enhances policy effectiveness compared with other methods.This is its third empirical question.Source: §IV · p. 5
A. Video Generation Enables Generalizable Manipulation
PhysWorld is evaluated on diverse real-world tasks, where generated task-conditioned videos provide visual guidance for zero-shot manipulation. Across video generators, Veo3 has the highest overall usable-video ratio, and robotic-data finetuning tends to outperform generic generators.
A. Video Generation Enables Generalizable Manipulation
- PhysWorld is evaluated on ten real-world tasks spanning wiping, watering, placement, sweeping, and pouring, with task-conditioned videos grounded into executable actions.The evaluation requires no additional robot data and demonstrates zero-shot manipulation in the real world.Source: A. Video Generation Enables… · p. 5
- Veo3 achieves the highest overall usable-video ratio across the evaluated tasks, while robotic-data finetuning tends to outperform generic generators.The ratio counts videos from which object poses can be recovered robustly.Source: A. Video Generation Enables… · p. 6, Table I · p. 6
B. World Modeling Improves Manipulation Robustness
PhysWorld's physical world model improves manipulation success and reduces grasping and tracking failures compared with zero-shot approaches without world modeling. Its reconstruction still incurs errors when occluded geometry is recovered from monocular generated videos.
B. World Modeling Improves Manipulation Robustness
- The reconstructed scenes are geometry-consistent and physically interactable, providing physical feedback for robot learning.The method combines geometry-aligned 4D reconstruction with generative priors to recover physical scenes from monocular inputs.Source: B. World Modeling Improves… · p. 6
- PhysWorld achieved the highest average success rate (82%), outperforming second-best RIGVid [8] (67%) by a large margin among three zero-shot methods without physical world modeling.Source: B. World Modeling Improves… · p. 6
- Compared with RIGVid [8], PhysWorld reduces grasping failures from 18% to 3% and tracking failures from 5% to 0%.Figure 6 categorizes failures as grasping, tracking, dynamics, and reconstruction.Source: B. World Modeling Improves… · p. 6, Figure 6 · p. 7
- PhysWorld introduces 7% reconstruction errors, partly because completed geometry in occluded regions may misalign with real-world geometry.The authors suggest advance multiview reconstruction of the environment may mitigate this issue.Source: B. World Modeling Improves… · p. 7
C. Object-Centric Learning Enhances Policy Effectiveness
Object-centric learning outperforms embodiment-centric learning on the reported placement tasks, while residual reinforcement learning converges faster than learning from scratch under the same physical world model. Baseline grasp-and-plan actions constrain exploration to a feasible neighborhood that the residual policy refines.
C. Object-Centric Learning Enhances Policy Effectiveness
- Object-centric learning achieves 90% versus 30% for embodiment-centric learning on bookshelf placement, and 80% versus 10% on shoebox placement.Generated videos may hallucinate hands or show inconsistent hand kinematics, while object motion is more stable and easier to estimate under occlusion.Source: C. Object-Centric Learning Enhances… · p. 7, Table II · p. 7
- Residual RL converges within a few hundred iterations and obtains higher object-tracking rewards than RL from scratch under the same budget.The comparison uses the physical world model and the task of pouring a tomato from a pan onto a plate.Source: C. Object-Centric Learning Enhances… · p. 7
- Baseline grasp-and-plan actions constrain residual-policy exploration to a small feasible neighborhood, while world-model feedback refines trajectories.RL from scratch can also succeed, but requires longer training and more careful reward designs.Source: C. Object-Centric Learning Enhances… · p. 7
V. CONCLUSION
PhysWorld bridges video generation and robot learning by reconstructing physically interactable scenes and learning object-centric residual reinforcement learning policies. The authors note that simulator fidelity bounds physical world modeling and may introduce additional sim-to-real gaps.
V. CONCLUSION
- PhysWorld reconstructs physically interactable scenes from generated videos and learns object-centric residual RL policies to turn visual demonstrations into physically feasible actions for zero-shot generalizable real-world manipulation.The conclusion presents this as a bridge between video generation and robot learning through physical world modeling.Source: §V · p. 7
- Future work includes synthesizing physically accurate videos with this framework to train robotic video generation models.Source: §V · p. 7
- Physical world modeling is bounded by simulator fidelity and may introduce additional sim-to-real gaps.The authors still argue that a world model is necessary for reliable physical feedback and more robust learning.Source: §V · p. 7