Source-linked AI summary
RealWonder: Real-Time Physical Action-Conditioned Video Generation
Wei Liu, Ziyu Chen, Zizhang Li, Yue Wang, Hong-Xing Yu, Jiajun Wu
TL;DR
Existing video models lack the structural understanding needed to predict how true 3D physical actions affect scenes. RealWonder bridges actions and video through physics simulation, 3D reconstruction, and a distilled four-step generator, achieving up to 13.2 FPS at 480×832 resolution for interactive control across diverse materials.
Problem
Current video generation models cannot simulate the physical consequences of 3D actions such as forces and robotic manipulations.
Method
RealWonder reconstructs a 3D scene, simulates specified actions, and translates their consequences into optical flow and RGB representations for a distilled video generator.
Results
13.2 FPS at 480×832 resolution enables real-time action-conditioned video generation from a single image across rigid, deformable, fluid, and granular materials.
Takeaways & Limitations
RealWonder enables interactive exploration of forces, robot actions, and camera controls for robotics simulation and AR/VR applications.
Takeaways & Limitations
3D reconstruction errors, particularly in depth estimation, can lead to suboptimal simulation and video results.
Abstract
from arXiv · showhide
Current video generation models cannot simulate physical consequences of 3D actions like forces and robotic manipulations, as they lack structural understanding of how actions affect 3D scenes. We present RealWonder, the first real-time system for action-conditioned video generation from a single image. Our key insight is using physics simulation as an intermediate bridge: instead of directly encoding continuous actions, we translate them through physics simulation into visual representations (optical flow and RGB) that video models can process. RealWonder integrates three components: 3D reconstruction from single images, physics simulation, and a distilled video generator requiring only 4 diffusion steps. Our system achieves 13.2 FPS at 480x832 resolution, enabling interactive exploration of forces, robot actions, and camera controls on rigid objects, deformable bodies, fluids, and granular materials. We envision RealWonder opens new opportunities to apply video models in immersive experiences, AR/VR, and robot learning. Our code and model weights are publicly available in our project website: https://liuwei283.github.io/RealWonder/
1 Introduction
RealWonder addresses the lack of real-time video generation conditioned on true 3D physical actions by using physics simulation to bridge continuous actions and visual video representations. Its reconstructed, physics-aware, four-step system supports interactive generation at up to 13.2 FPS.
- Existing video generators primarily support passive generation or simple 2D controls, not physical consequences from forces, torques, or robot actions.
- Physics simulation converts 3D action consequences into optical flow and coarse RGB previews that video models can process.This preserves the causal relationship between actions and outcomes while avoiding direct tokenization of continuous, unbounded signals.
- The method avoids action-video pairs by training the video generator with 2D flow-video pairs, while flow conditioning reduces diffusion overhead.
- RealWonder reconstructs a simulatable 3D scene, simulates specified actions, and conditions a distilled four-step video generator on physics-grounded representations.The system accepts forces, robot actions, and camera motions from a single input image.
- 13.2 FPS at 480×832 resolution enables real-time action-conditioned video generation on a single GPU.
- RealWonder supports interactive visual consequences for rigid objects, deformable bodies, fluids, and granular materials.Users can apply forces, control robotic grippers, or move cameras and observe the resulting video.
2 Related Work
Related work has expanded video generation from passive synthesis toward user-controlled and real-time systems, but existing controls remain largely 2D or trajectory-based. Acceleration and streaming methods improve speed, yet trajectory-controlled approaches still do not provide the described 3D physical-action capability.
- Controllable Video Generation: Controllable video generation has explored depth, sketches, drag interactions, subject-specific control, optical flow, camera control, and motion trajectories.
- Real-time and Streaming Video Generation: Acceleration methods use adversarial post-training or distribution matching distillation to enable one- or few-step streaming video generation.
- Real-time and Streaming Video Generation: Causal streaming generators face temporal drifting, motivating follow-up methods designed to improve temporal consistency.
- Real-time and Streaming Video Generation: MotionStream provides real-time streaming generation with trajectory-based control, representing a related but distinct control setting.
3 RealWonder
RealWonder converts 3D physical actions into physics-derived visual representations that condition a real-time video generator. Its pipeline reconstructs a simulatable scene, predicts action consequences across materials, and synthesizes streaming video from flow and RGB cues.
- Formulation: RealWonder accepts a single RGB image and 3D actions comprising external forces, robot commands, and camera poses, producing an interactive video stream.Robot commands specify end-effector position, orientation, and gripper state; camera actions define viewpoint rotation and translation.
- Streaming Inference: 13.2 FPS at 480×832 resolution supports real-time generation of physically plausible videos from the simulated action consequences.The system’s intermediate representations are computed in real time and condition photorealistic video synthesis.
- Single-Image 3D Scene Reconstruction: The system reconstructs a simulatable scene from one image using lightweight point clouds for static backgrounds and dynamic objects.The representation balances geometric detail with rendering efficiency for real-time operation.
- Single-Image 3D Scene Reconstruction: Objects are classified into six material categories with estimated physical parameters, while users can override classifications and parameters for control or accuracy.Categories include rigid, elastic, cloth, smoke, liquid, and granular materials; parameters include density, friction, elasticity, and viscosity as applicable.
- Physics Simulation as Intermediate Bridge: Physics simulation converts actions into updated scene states, optical flow, and coarse RGB previews that preserve action consequences in a visual domain.Specialized solvers handle rigid, elastic, cloth, smoke, liquid, and granular materials; the RGB preview adds structural cues such as occlusion changes.
- Real-Time Conditional Video Generation: The video generator is adapted for flow conditioning and distilled from a pretrained image-to-video model into a causal four-step streaming generator.At inference, SDEdit incorporates the coarse RGB preview during the four-step denoising process alongside flow-based conditioning.
4 Experiments
RealWonder is evaluated against video-generation and physics-based baselines across physical-action fidelity, visual quality, runtime, and material diversity. Results show strong action adherence, realistic appearance, real-time streaming, and complementary conditioning signals.
- Comparison to Baselines: Baseline models often produce plausible visuals but fail to realize specified physical outcomes, including incorrect object motion, object merging, or missing dynamics.Tora and CogVideoX-I2V fail on persimmon hits and misinterpret the boat trajectory.
- Comparison to Baselines: RealWonder combines physics-based action adherence with video-model visual realism, including dynamic shadows, water splashes, collapsing sand, and soft clothing.PhysGaussian provides reasonable boat motion but lacks dynamic shading and shadows.
- Quantitative and User-Study Results: RealWonder achieves the best or second-best results across reported metrics and is significantly more favored than competing baselines in the 2AFC user study.The human study uses 400 participants.
- Controllability: Different actions applied to the same scene produce corresponding physical outcomes, such as a sand castle falling in the direction of the applied wind.The demonstrated winds come from opposite directions.
- Ablation Study: Removing the physics simulator prevents plausible action consequences, while removing RGB previews disrupts overall motion and removing flow can produce static videos.The ablation concludes that both flow and RGB preview conditioning signals are necessary.
5 Conclusion
RealWonder presents real-time physical action-conditioned video generation by integrating physics simulation with video synthesis. The system supports action inputs for motion planning and AR/VR while avoiding direct tokenization of continuous actions and action-video dataset curation, but its results depend on accurate 3D reconstruction.
- Conclusion: RealWonder integrates physics simulation into real-time physical action-conditioned video generation and supports applications in motion planning and AR/VR.The system is designed for action inputs and real-time feedback.
- Conclusion: The physics-simulation bridge avoids tokenizing continuous actions and curating action-video datasets.This is presented as a way to sidestep both obstacles.
- Limitations: 3D reconstruction errors, especially inaccurate depth estimation, can cause suboptimal simulation and video results.The authors identify more reliable large reconstruction models as future work.
A Additional Implementation Details
The implementation reconstructs a simulatable 3D scene, estimates materials, and streams physics-derived conditions into a causal video generator. Training uses flow conditioning and distillation, while inference renders optical flow and coarse RGB previews before four-step generation.
- 3D Scene Reconstruction: The reconstruction pipeline segments objects, fills occluded backgrounds, estimates monocular depth and camera intrinsics, and reconstructs object meshes in scene coordinates.SAM 2, FLUX-based inpainting, MoGE-2, and SAM3D provide these stages.
- Material Estimation: Material properties are estimated from the input image with GPT-4V and can be overridden by users.The experimental material parameters are summarized in Table S2.
- Video Generator: The video generator is trained for optical-flow conditioning, causal attention adaptation, and distribution-matching distillation using synthetic video trajectories.The implementation uses a 1.3B image-to-video model and trains the final distillation stage for 600 iterations with batch size 64.
- Streaming Inference: Algorithm 1 reconstructs the scene and materials, simulates actions, computes optical flow and coarse RGB previews, then streams causally generated frames.The streaming generation loop uses 4-step diffusion and reports 13.2 FPS.
B Additional Analysis
RealWonder supports extended physical-action video streaming across diverse materials and remains visually robust to reconstruction or material errors. Its outputs target physical plausibility rather than strict physical correctness, while quantitative evaluations show strong performance across metrics.
- Long video generation: RealWonder simulates sequences of physical actions over extended time windows, whereas video-generation baselines stop after their first window.Physics-based PhysGaussian also supports longer streaming, but its visual quality quickly degenerates from error accumulation.
- Additional scenes: RealWonder covers rigid, deformable, granular, fluid, and multi-material interactions in additional scenes.The examples include different materials and interaction types, with gravity applied in the supplementary visualizations.
- Sensitivity to Reconstruction: 20% depth perturbations and snow-to-sand material substitutions leave visual realism robust in stress tests.The video generator tolerates minor conditioning errors from the physics simulator.
- Physical Plausibility: The system aims for visually plausible consequences of input actions, not outputs that strictly obey all physical laws.Strict physical correctness remains substantially more challenging and is identified as an important direction for future research.