ContactGaussian-WM: Learning Physics-Grounded World Model from Videos

Meizhong Wang; Wanxin Jin; Kun Cao; Lihua Xie; Yiguang Hong

Source-linked AI summary

ContactGaussian-WM: Learning Physics-Grounded World Model from Videos

Meizhong Wang, Wanxin Jin, Kun Cao, Lihua Xie, Yiguang Hong

arXiv:2602.11021v1Feb 11, 2026cs.ROcs.AIcs.CV

TL;DR

World models struggle with sparse, contact-rich videos because existing approaches can lack physical grounding or differentiable contact handling. ContactGaussian-WM unifies visual and collision geometry with an end-to-end differentiable rigid-body model, and experiments show strong performance, generalization, and utility for synthesis and real-time MPC.

Problem
Sparse, contact-rich videos remain challenging because existing world models can be data-hungry, physically unconstrained, and difficult to optimize through contact.
Method
ContactGaussian-WM uses unified Gaussian visual and collision geometry with differentiable collision detection and closed-form contact dynamics to learn physical properties from sparse videos.
Results
Experiments show that ContactGaussian-WM learns physically consistent world models in complex contact-rich scenarios and supports long-horizon prediction, data synthesis, and real-time MPC.
Takeaways & Limitations
The framework provides a differentiable physics-grounded world model for prediction, simulation data synthesis, and real-time planning across simulation and real-world settings.
Takeaways & Limitations
The current framework is limited to rigid-body systems, and Gaussian-sphere collision detection may be inaccurate under deep interpenetration.

Abstract

from arXiv · show

Developing world models that understand complex physical interactions is essential for advancing robotic planning and simulation.However, existing methods often struggle to accurately model the environment under conditions of data scarcity and complex contact-rich dynamic motion.To address these challenges, we propose ContactGaussian-WM, a differentiable physics-grounded rigid-body world model capable of learning intricate physical laws directly from sparse and contact-rich video sequences.Our framework consists of two core components: (1) a unified Gaussian representation for both visual appearance and collision geometry, and (2) an end-to-end differentiable learning framework that differentiates through a closed-form physics engine to infer physical properties from sparse visual observations.Extensive simulations and real-world evaluations demonstrate that ContactGaussian-WM outperforms state-of-the-art methods in learning complex scenarios, exhibiting robust generalization capabilities.Furthermore, we showcase the practical utility of our framework in downstream applications, including data synthesis and real-time MPC.

I. INTRODUCTION

Existing world models struggle to learn physically reliable, long-horizon predictions from sparse, contact-rich videos. ContactGaussian-WM addresses this with unified visual-geometric Gaussians and differentiable physics, demonstrating broad performance and planning utility.

Motivation: Sparse, contact-rich videos remain difficult for world models, whose data hunger and missing physical constraints can produce interpenetration, disappearance, and poor long-horizon consistency.These limitations affect planning-oriented use of predictive world models.
Proposed framework: ContactGaussian-WM learns a physics-grounded world model from a small number of interactive videos using a unified geometric and visual representation.The framework uses isotropic Gaussian geometry, contact dynamics, and differentiable rendering.
Proposed framework: The framework enables end-to-end learning by combining differentiable collision detection, contact dynamics, and a 3DGS renderer.Its geometry is used for collision computation while the renderer produces predicted images.
Applications: The learned model supports long-horizon, accurate, physically consistent prediction, simulation data synthesis, and real-time MPC planning.These demonstrations include sim-to-real use and downstream control applications.
Evaluation: Experiments in simulation and real-world environments validate generalization across complex interaction scenarios and report performance exceeding existing methods[17, 7].The evaluation spans sparse-observation learning and diverse contact-rich settings.

2) Physics-grounded world models:

Physics-grounded world models improve physical plausibility but face differentiability and contact-resolution challenges. ContactGaussian-WM combines differentiable collision detection with a closed-form contact model to provide stable analytical gradients.

Physics-grounded world models: Explicit physics engines improve physical structure but commonly rely on gradient-free optimization, while black-box models lack strict non-penetration and friction guarantees.These limitations hinder efficient parameter identification and generalization in contact-rich settings.
Differentiable world models: Differentiable world models can suffer from contact-boundary discontinuities, numerical approximations, and non-differentiable collision detection that impede geometry optimization.These issues make stable learning difficult in contact-rich interactions.
Physics-grounded world models: Mesh-based engines use precise forward-simulation algorithms but are non-differentiable and ill-suited to gradient-based optimization.The limitation motivates continuous geometric representations for learning.
Contact dynamics: Traditional rigid-body contact is often posed as an NCP, with polyhedral friction-cone approximations enabling LCP solvers but creating computational challenges at high dimensions.The paper instead adopts a complementarity-free contact formulation.
ContactGaussian-WM: ContactGaussian-WM implements differentiable collision detection and a closed-form contact model for more efficient simulation and stable analytical gradients.Its framework integrates these components with a unified world-model pipeline.

B. Problem formulation

ContactGaussian-WM formulates future-image prediction as learning Gaussian scene parameters and physical properties through a differentiable simulator. Its two-stage initialization and refinement align visual appearance with collision geometry.

B. Problem formulation: The model predicts future images by transforming a unified Gaussian scene representation with the physical state and rendering the result.Geometry and appearance are represented separately as G_geo and G_vis within the Gaussian set.
B. Problem formulation: Learning jointly optimizes Gaussian parameters and physical parameters, but the framework separates this into Gaussian optimization followed by physical refinement with geometry fine-tuning.The staged design addresses the high nonlinearity and ambiguity of joint learning.
C. Stage I: Scene Initialization with Sphere Geometry Gaussian Splatting (SG-GS): Stage I obtains multi-view geometric observations with SAM2 and initializes the scene using Sphere Geometry Gaussian Splatting.The initialization uses calibrated-camera observations and extracted geometry features.
B. Problem formulation: The overview pipeline uses Gaussian geometry for differentiable collision detection, passes contact points to contact dynamics, and renders the next image from the resulting state.The full pipeline is differentiable for end-to-end learning.
C. Stage I: Scene Initialization with Sphere Geometry Gaussian Splatting (SG-GS): Isotropic spherical primitives keep Gaussian geometry compatible with closed-form differentiable collision detection in the refinement stage.Rotation updates are frozen so each primitive remains spherical.
C. Stage I: Scene Initialization with Sphere Geometry Gaussian Splatting (SG-GS): Decoupled optimization first fits geometric attributes against geometric maps, then refines visual attributes with L1 and D-SSIM losses while freezing collision geometry.This separates stable physical geometry from high-frequency visual detail.
C. Stage I: Scene Initialization with Sphere Geometry Gaussian Splatting (SG-GS): The resulting SG-GS representation provides visual accuracy and geometric robustness by turning Gaussian primitives into explicit collision geometries.This creates alignment between the rendered appearance and the physical model.

D. Stage II: Phys-Geo Refinement

Stage II refines physical parameters and collision geometry by differentiating through Gaussian-based collision detection. The method smooths distance computation and penetration handling to preserve gradients for contact-rich dynamics.

D. Stage II: Phys-Geo Refinement: Physical parameters θ and geometric attributes Ggeo are jointly optimized from visual-observation gradients through differentiable collision detection, contact dynamics, and 3DGS rendering.θ includes mass-inertia, friction, stiffness, and damping parameters.
1) Differentiable collision detection.: The collision geometry is modeled as a union of spherical primitives whose centers come from Gaussian positions and radii equal twice their scale.This gives a simple distance representation for collision queries.
1) Differentiable collision detection.: A minimum over sphere distances defines the collision field, while LogSumExp smoothing makes shortest-distance estimation differentiable.The smoothing factor β controls the approximation.
1) Differentiable collision detection.: Because nearest-sphere distance can misestimate penetration inside an object, a negative-distance penalty with a sigmoid transition robustly corrects interpenetration while preserving differentiability.The resulting distance supports projection to the nearest surface point and computation of the surface normal.

2) Differentiable complementarity-free contact dynamics.:

The contact model replaces iterative complementarity solving with a differentiable closed-form approximation of contact forces. Its impedance formulation uses penetration regimes to represent contact modes while satisfying Coulomb friction laws.

2) Differentiable complementarity-free contact dynamics.: The complementarity-free contact model approximates contact forces through an impedance mechanism on dual friction cones instead of iteratively solving contact constraints.Learnable stiffness K and damping D determine the closed-form contact-force solution.
2) Differentiable complementarity-free contact dynamics.: The dynamics combine generalized mass, non-contact forces, signed collision distances, friction-cone Jacobians, and learnable stiffness and damping.The next velocity is computed from these quantities, followed by rigid-body position integration.
2) Differentiable complementarity-free contact dynamics.: Different penetration regimes across dual-cone facets capture sticking, sliding, and separation while automatically satisfying Coulomb friction laws.This formulation supports contact-rich rigid-body transitions without iterative complementarity resolution.
3) 3DGS rendering: The differentiable simulator produces the next predicted state, which is transformed through forward kinematics and mapped to images by the camera pose and 3DGS renderer.This connects contact dynamics to image-space supervision.

4) End-to-End Gradient Flow:

ContactGaussian-WM propagates image-space gradients backward through rendering and simulation to optimize physical and geometric parameters. Geometry is updated through collision interactions rather than directly through the rendering loss because motion blur makes direct visual optimization ambiguous.

4) End-to-End Gradient Flow:: Image-loss gradients backpropagate through the renderer to the state and then through the differentiable simulator to optimize θ and Ggeo.The jointly optimized parameters are denoted Θ.
4) End-to-End Gradient Flow:: The gradient flow through time is recursively computed through the differentiable simulator, linking parameter updates across successive predicted states.This establishes the computational graph needed for end-to-end optimization.
4) End-to-End Gradient Flow:: Because motion blur makes initial Gaussian optimization ambiguous, image loss does not directly update Gaussian geometry through rendering; geometry is refined only through propagated physical collision interactions.This design separates direct visual alignment from collision-based geometric refinement.
IV. RESULTS: The evaluation tests sparse contact-rich physical-law learning, real-world generalization, sim-to-real control transfer, and downstream robotic utility.The framework is evaluated in simulation and real-world scenarios, with downstream details reported in the Appendix.

A. Evaluations in simulation

Simulation evaluates ContactGaussian-WM on quasi-dynamic continuous-contact and high-dynamic discrete-contact tasks against data-driven and physics-based baselines. The method is comparable on pushing and more effective on sparse, discontinuous collision scenarios, where it supports stable generalization.

Training performance: DreamerV3 reports higher training PSNR under closed-loop one-step prediction, whereas other methods use open-loop cumulative error, making the comparison asymmetric.
Training performance: In Push-slide-settle, physics-based models perform comparably because continuous contact produces smooth gradients and trajectories governed mainly by friction.
Training performance: Fall-and-rebound is harder because brief collisions provide limited information and discrete contacts create gradient discontinuities.
Training performance: ContactGaussian-WM maintains stable and accurate gradient flow by using sparse, discontinuous collision information for end-to-end learning.
Test performance: DreamerV3 fails to preserve scene consistency and physical plausibility on novel states and actions during long-horizon forecasting.

B. Evaluations in real world

Real-world experiments test free-fall and robotic manipulation across varied materials and use open-loop long-horizon prediction and trajectory transfer. Parameter learning improves prediction fidelity and narrows the sim-to-real gap.

Experimental setup: The real-world setup covers free-fall and high-impact LEAP Hand manipulation with sliding and flipping across rubber, PVC, and wood surfaces.
Experimental metrics: Evaluation uses cumulative PSNR of long-horizon open-loop predictions because reliable real-world state information is unavailable.
Experimental setup: The method is compared with an otherwise identical model without parameter learning to isolate the effect of learned physical parameters.
Real-world results: Parameter learning improves ContactGaussian-WM’s prediction fidelity while narrowing the sim-to-real gap in open-loop trajectory transfer.

C. Applications

ContactGaussian-WM supports physically accurate simulation-data synthesis and real-time MPC for in-hand manipulation. Its current scope is rigid-body systems, with known limitations in rendering quality and deep-interpenetration collision accuracy.

Simulation data synthesis: ContactGaussian-WM generates physically accurate and visually plausible simulation data for complex, contact-rich, high-dynamic interactions.The model is presented as an engine for simulation data synthesis after evaluations in simulated and real-world settings.
Real-time MPC control: Differentiability and explicit dynamics enable integration with MPC, allowing a LEAP Hand to plan and execute in-hand redirection in MuJoCo.The experiment trains the world model on MuJoCo interactive data before running real-time MPC.
Applications: The framework is presented as better suited to highly dynamic, contact-rich videos and is integrated with both simulation-data synthesis and real-time MPC.These applications are summarized as demonstrated uses of the proposed sparse-video world model.
Limitations: The unified scene representation can reduce rendering quality, while Gaussian-sphere collision detection may become inaccurate under deep interpenetration.The current framework is also limited to rigid-body systems and excludes deformable objects.

APPENDIX

The appendix details unified scene reconstruction, differentiable sphere collision handling, and simulation-data preparation. It emphasizes penetration handling as a stability modification for LogSumExp-based collision detection.

Unified Representation: The unified representation combines Gaussian splats from SG-GS with OBJ meshes extracted from an implicit SDF using Soft Distance Detection and Marching Cubes [59].The representation is visualized through Gaussian splats and extracted meshes.
Collision detection: LogSumExp collision detection is augmented with a sigmoid-based fixed-distance rule when penetration occurs, stabilizing penetration estimates.Without this handling, LogSumExp can produce incorrectly small penetration values inside objects and exacerbate penetration.
Differentiable contact dynamics: Collision information provides surface normals through normalized SDF gradients, from which the contact Jacobian is computed using rigid-body kinematics [53].The appendix describes the Jacobian and collision-distance quantities used for subsequent contact processing.
Differentiable contact dynamics: The contact Jacobian is projected onto normal and tangential directions, then stacked with collision distances for contact-dynamics processing.The tangential and normal components are indexed per contact point and direction.
Simulation data: Simulation trajectories are recorded with a fixed camera pose, including object initial states, camera parameters, and end-effector motion.The resulting simulation dataset is visualized in Fig. 9.

2) Implementation Details of Baselines:

The appendix standardizes baseline training and test conditions, supplements qualitative visualizations, and describes real-world LEAP Hand control and initialization procedures.

Implementation Details of Baselines: ContactGaussian-WM, PIN-WM[17], and CEM+MuJoCo+R share 15 training frames, while DreamerV3[7] uses 100 randomized 25-frame sequences; all methods share the same test set.This setup is intended to ensure a fair comparison across methods.
Supplement to Qualitative Results: The method outperforms other baselines on Fall-and-rebound tasks and shows superior performance on Push-slide-settle tasks in two simulation scenarios.Qualitative results for the camera object are provided in Fig. 11.
Real-world implementation: Real-world LEAP Hand experiments estimate camera pose with ArUco markers and recover hand pose by aligning rendered parts with real images.Position control is used to implement velocity-like control while limiting maximum joint velocity.
Initial-state estimation: Initial object pose is estimated by aligning the first frame, and initial velocity is estimated from the first three non-contact frames without an additional state-estimation module.Irrelevant parameters are frozen during these optimization steps.
Real-time MPC control: Real-time MPC is evaluated in MuJoCo on different objects for palm redirection with the LEAP Hand.Additional visualizations are provided for the redirection experiment.