WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions

• Single-image action-conditioned generation targets dynamic 3D scenes under gravity, wind-like force fields, and point forces.Source: §1 · p. 2
• Physics-only methods require accurate solvers and complete physical states, restricting them mainly to rigid bodies and simple elasticity.Source: §1 · p. 2
• WonderPlay supports interactions with rigid bodies, cloth, liquids, gases, and granular substances from a single input image.Source: §1 · p. 2
• WonderPlay combines a physics solver’s coarse action-induced simulation with video generation and feeds the synthesized video back to update the simulation.Source: §1 · p. 2
• Motion–appearance bimodal control with spatially varying masking improves simulator conditioning while reducing hallucination in trustworthy regions such as static backgrounds.Source: §1 · p. 2
• The framework is reported to outperform pure physics-based methods and adapted video generators in visual quality and physical plausibility across varied interactions.Source: §1 · p. 2

Action-conditioned dynamic scene generation

• Modal-basis approaches represent motion as vibrations, but their basis functions struggle with motions beyond vibration-like behavior.Source: §2 · p. 2
• Physics-based 3D methods typically require complete geometry from complex multi-view captures, limiting their use when only a single image is available.Source: §2 · p. 2
• PhysMotion combines a single image, a 3D physics solver, and video generation, but restricts dynamics to rigid and elastic phenomena because physics computes the entire scene dynamics.Source: §2 · p. 3

Controllable video generation

• Video generators offer realistic physical phenomena but generally lack controllability for general 3D actions, especially actions such as fluids or wind that are difficult to encode as trajectories.Source: §2 · p. 3

World models

• Interactive world models have largely focused on video-game domains, while real-world systems provide limited interaction beyond camera control or text.Source: §2 · p. 3

Dynamic 3D scene generation

• Existing dynamic 3D generation methods model motion in scenes or objects but do not simulate dynamics in response to actions.Source: §2 · p. 3

3. WonderPlay

• WonderPlay formulates dynamic scene synthesis from a single image and forces including gravity, 3D force fields, and 3D point forces.The output is a time-indexed dynamic 3D scene resulting from applying those actions to the reconstructed initial scene.Source: §3 · p. 3

3.1. 3D Scene Reconstruction

• The initial scene separates a static Gaussian-surfel background from dynamic objects represented as topological Gaussian surfels with connectivity and velocities.Objects are initialized by generating a mesh from an image segment and binding Gaussian surfels to mesh vertices, with six material classes.Source: §3.1 · p. 4

3.2. Hybrid Generative Simulator

• The hybrid simulator first applies coupled physics solvers to estimate coarse dynamics, then conditions a video generator to synthesize finer motion consistent with the action consequence.The framework uses multiple solvers for different materials and couples them for multiphysics scenes such as fluid–rigid interactions.Source: §3.2 · p. 4
• The video generator receives motion flow, a rendered coarse video, and the input image through a bimodal control scheme.Flow provides motion control, while RGB conditioning uses SDEdit to preserve coarse scene content while synthesizing details.Source: §3.2 · p. 5
• Spatially varying responsibility assigns the video generator a lower modification level on static backgrounds than on dynamic objects to reduce hallucinated details.The binary mask marks dynamic-object pixels separately from background pixels during denoising.Source: §3.2 · p. 6
• The generated video closes the loop by supervising updates to the coarse dynamic scene through differentiable rendering and a photometric L1 loss.The resulting scene retains non-dynamics attributes while updating object motion and appearance to match the generated video.Source: §3.2 · pp. 4, 6

4.1. Results

• Video baselines produced plausible visuals but often violated actions, altered scenes, or generated ineffective dynamics in duck and boat examples.Tora submerged and reshaped the duck and altered the boat scene, while CogVideoX-I2V moved the duck incorrectly and failed to produce meaningful boat dynamics.Source: §4.1 · p. 6
• WonderPlay combines physics-based action handling with video-based realism, producing water waves, bubbles, reflections, and dynamics across diverse materials.The qualitative results include varied scenes and actions, while the physics simulator preserves action-consistent behavior such as wind-driven and elastic motion.Source: §4.1 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 7
• WonderPlay achieved the best or second-best performance across all quantitative metrics on 15 scenes.The metrics cover imaging, aesthetic, motion quality, consistency, and GPT-4o-based physical realism.Source: Table 2 · p. 6, §4 · p. 6, §4.1 · p. 8
• In a human study, about 70% to 80% of participants preferred WonderPlay over all baselines across physics plausibility, motion fidelity, and visual quality.Source: Table 1 · p. 6, §4.1 · p. 8

4.2. Ablation on Hybrid Generative Simulator

• The hybrid simulator refines coarse physics, replacing sticky smoke motion and grainy appearance with smoother swirls and cleaner visuals.The video generator improves both dynamics and appearance relative to physics-only simulation.Source: §4.2 · p. 8, Figure 7 · p. 8
• Using both motion and appearance signals gives the best video results, whereas either signal alone causes lost detail or hallucinated dynamics and background changes.Without RGB, motion conditioning can hallucinate unsupported content; without flow, appearance conditioning fails to preserve detailed sand dynamics.Source: §4.2 · p. 8, Figure 8 · p. 8

5. Conclusion

• WonderPlay provides action-conditioned dynamic 3D scene generation from a single image using a hybrid generative simulator.The framework is presented as delivering both simulation fidelity and visual quality across diverse scenes and interactions.Source: §5 · p. 8

A. Additional Evaluation

• WonderPlay significantly outperformed baselines in later staged evaluations involving complex physics and geometry, although baselines performed well in early simple stages.The stages progress from a rigid ball falling onto a desk to elastoplastic, multiphysics, and complex-shape scenes.Source: A. Additional Evaluation · p. 12
• Lower diffusion-control values underuse video priors, while higher values can weaken adherence to the physics simulation.The reported optimal configuration slightly outperformed alternative settings for s1, s2, and γ.Source: A. Additional Evaluation · p. 12

B.1. Additional Implementation Details

• The implementation uses H = 480, W = 720, T = 48, 49 output frames, DDIM sampling, S = 25 steps, and γ = 0.4.Appearance conditioning is applied at s1 = 21 and s2 = 18 diffusion steps.Source: §B.1 · p. 12

B.2. Reconstructing Background

• The background is reconstructed by decomposing the input image into layers, unprojecting pixels with estimated depth, and optimizing photometric agreement through differentiable rendering.The resulting background is used as the initial scene background.Source: §B.2 · p. 12

B.3. Reconstructing Topological Gaussian Surfels

• Objects are converted into topological Gaussian surfels by generating an InstantMesh mesh, binding one surfel to each mesh vertex, and optimizing surfel parameters against rendered multiview images.Initialization uses vertex normals and colors before differentiable-rendering optimization.Source: §B.3 · p. 12
• Object representations are registered to scene coordinates by estimating orientation with DUSt3R and solving for scale and translation from 3D correspondences.Correspondences are sampled from unprojected scene pixels and rendered object representations.Source: §B.3 · p. 12
• Internal filling is adopted to stabilize the physics simulation.The technique follows PhysGaussian.Source: §B.3 · p. 12

B.5. Simulation Parameters

• Material parameters are estimated by a vision-language model and may receive manual adjustment to keep simulation results physically plausible.The implementation considers homogeneous materials and records default simulation values in Table S2.Source: §B.4 · p. 12, §B.5 · p. 13

B.4. Material and Physics Solvers

• Six material classes are modeled with rigid-body, MPM, or PBD solvers: rigid, elastic, cloth, smoke, liquid, and granular.Rigid objects use shape matching; elastic, liquid, and granular materials use MPM, while cloth and smoke use PBD.Source: §B.4 · pp. 12–13
• Topological Gaussian surfels augment Gaussian representations with connectivity and velocity, allowing physics solvers to update dynamic scene attributes.The edge matrix encodes surfel connectivity, while velocity stores motion for simulation.Source: §B.4 · p. 12
• The solver suite handles progressively harder scenes, from a rigid ball falling onto a rigid plane to soft-body, fluid-contact, and complex-geometry interactions.The four stages incrementally replace the rigid ball with a soft ball, replace the plane with water, and add complex geometry.Source: Figure S1 · p. 13, Figure S2 · p. 13
• MPM transfers particle momentum to a spatial grid and back to particles, whereas PBD directly solves position constraints for incompressibility, cloth deformation, and bending.These solver-specific updates produce positions and velocities used to advance the simulated dynamics.Source: §B.4 · p. 13

B.6. Rendering Simulated Dynamics

• Rigid surfels directly inherit mesh-vertex positions, while particle-based surfels use averaged updates from their ten nearest sampled particles.This mapping transfers MPM and PBD particle motion into the Gaussian representation used for rendering.Source: §B.6 · p. 13
• Simulation parameters and their default values are tabulated for the physical simulation process.The parameters can be roughly estimated with a vision-language model and manually adjusted when needed.Source: Table S2 · p. 14