PaperlayerView exact revision on arXiv

Source-linked AI summary

PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation

Wenxuan Li, Hang Zhao, Zhiyuan Yu, Yu Du, Qin Zou, Ruizhen Hu, Kai Xu

arXiv:2504.16693v2cs.LGcs.RO

TL;DR

Non-prehensile manipulation requires learning under complex 3D dynamics while avoiding costly demonstrations and simulation-to-reality gaps. PIN-WM identifies a physics-informed 3D world model from few-shot visual interactions and augments it with physics-aware digital cousins. The approach surpasses recent Real2Sim2Real methods and reaches 75% Push and 65% Flip success without real-world fine-tuning.

  • Problem

    Non-prehensile manipulation is difficult to learn because friction, restitution, inertia, and simulation-to-reality discrepancies complicate dynamics modeling and policy transfer.

    Source: §I · p. 1

  • Method

    PIN-WM differentiably identifies 3D rigid-body physics from few-shot, task-agnostic visual interactions and uses Gaussian-Splatting observational loss plus physics-aware digital cousins.

    Source: §I · p. 2

  • Results

    75% Push and 65% Flip success were achieved in real-world experiments without real-world fine-tuning, while overall performance surpassed recent Real2Sim2Real state-of-the-arts.

    Source: §I · p. 2

  • Takeaways & Limitations

    The learned world-model ensemble supports robust non-prehensile manipulation policies with Sim2Real transfer from minimal task-agnostic interaction data.

    Source: §I · p. 2

    Rendering alignment remains important because robot-generated shadows can distort rendering loss and compromise learned physical properties; the current framework focuses on rigid-body dynamics.

    Source: §V · p. 9

Abstract

from arXiv · show

While non-prehensile manipulation (e.g., controlled pushing/poking) constitutes a foundational robotic skill, its learning remains challenging due to the high sensitivity to complex physical interactions involving friction and restitution. To achieve robust policy learning and generalization, we opt to learn a world model of the 3D rigid body dynamics involved in non-prehensile manipulations and use it for model-based reinforcement learning. We propose PIN-WM, a Physics-INformed World Model that enables efficient end-to-end identification of a 3D rigid body dynamical system from visual observations. Adopting differentiable physics simulation, PIN-WM can be learned with only few-shot and task-agnostic physical interaction trajectories. Further, PIN-WM is learned with observational loss induced by Gaussian Splatting without needing state estimation. To bridge Sim2Real gaps, we turn the learned PIN-WM into a group of Digital Cousins via physics-aware randomizations which perturb physics and rendering parameters to generate diverse and meaningful variations of the PIN-WM. Extensive evaluations on both simulation and real-world tests demonstrate that PIN-WM, enhanced with physics-aware digital cousins, facilitates learning robust non-prehensile manipulation skills with Sim2Real transfer, surpassing the Real2Sim2Real state-of-the-arts.

I. INTRODUCTION

PIN-WM addresses difficult non-prehensile manipulation dynamics by identifying a 3D physics-informed world model from few-shot, task-agnostic visual interaction data. Physics-aware digital cousins then support robust policy learning and Sim2Real transfer.

Motivation

  • Non-prehensile manipulation is difficult because friction, restitution, inertia, and contact configurations complicate prediction, planning, and control.Source: §I · p. 1
  • Existing imitation and reinforcement-learning approaches face costly real-world demonstrations, simulation-to-reality discrepancies, or poor out-of-distribution generalization.Source: §I · p. 1

PIN-WM

  • PIN-WM identifies critical 3D rigid-body physics parameters end-to-end from few-shot, task-agnostic interaction trajectories and visual observations.Source: §I · p. 2
  • Gaussian-Splatting rendering loss enables learning from RGB images without additional state-estimation modules.Source: §I · p. 2

Digital Cousins

  • Physics-aware perturbations convert the identified digital twin into digital cousins that introduce meaningful variations around learned physics and rendering parameters.Source: §I · p. 2

Results

  • Real-world experiments achieve 75% success in Push and 65% in Flip while transferring policies without real-world fine-tuning.Source: §I · p. 2

A. Non-Prehensile Manipulation

Prior work spans analytical manipulation models, learned world models, differentiable physics, and domain randomization, but limitations remain in physical fidelity, data efficiency, and transfer realism. PIN-WM addresses these limitations with differentiable 3D identification and constrained physics-aware randomization.

A. Non-Prehensile Manipulation

  • Non-prehensile manipulation controls objects without fully grasping them, but prior analytical methods often assume known physics or idealized models.Source: A. Non-Prehensile Manipulation · p. 2
  • Imitation learning requires substantial real-world data, while simulated deep reinforcement learning faces major simulation-to-reality transfer challenges.Source: A. Non-Prehensile Manipulation · p. 2

B. World Models for Policy Learning

  • Data-driven world models provide interactive environments for policy training but depend heavily on training-data quantity and quality and struggle with out-of-distribution scenarios.Source: B. World Models for Policy Learning · pp. 2–3
  • Existing physics-informed approaches use gradient-free optimization or simplified 2D models, limiting identification quality and representation of complex 3D interactions.Source: B. World Models for Policy Learning · p. 3

C. Domain Randomization

A. Overall Framework

The framework first identifies a visual-physics world model from target-domain observations, then uses it and physics-aware digital cousins for reinforcement-learning policy transfer. Its differentiable components connect state dynamics, rendering, and observed images.

A. Overall Framework

  • The Real2Sim stage identifies target-domain physics parameters from visual observations using task-agnostic physical interactions.Source: A. Overall Framework · p. 3
  • PIN-WM composes a differentiable physics function with a differentiable renderer, while assuming geometry is known.Source: A. Overall Framework · pp. 3–4
  • The physics function predicts the next state from the current state and action, and the rendering function generates the corresponding next image; system identification optimizes both parameter sets to match observations.Source: A. Overall Framework · p. 4
  • After identification, the world model becomes an interactive reinforcement-learning environment for training policies intended to transfer without real-world fine-tuning.Source: A. Overall Framework · p. 4
  • Physics-aware digital cousins perturb physics and rendering parameters to model unobserved discrepancies while preserving similar underlying properties.Source: A. Overall Framework · p. 4, Figure 2 · p. 4

B. Physics-INformed World Model

PIN-WM identifies 3D rigid-body physics from visual observations by combining differentiable rendering with differentiable physics simulation. The method estimates physical parameters from task-agnostic interaction trajectories without requiring state estimation.

Physics modeling

  • PIN-WM models scene dynamics with rigid-body simulation, representing the object, end-effector, and floor through positions, orientations, and twist velocities.Its physical parameters include mass and inertia, restitution, and friction coefficients; elasticity and plasticity are outside the rigid-body scope.Source: B. Physics-INformed World Model · p. 4

Rendering alignment

  • Rendering alignment first optimizes 2D Gaussian Splatting parameters from multiple static views so object poses can be rendered differentiably.The learned representation updates Gaussian splats when the object pose changes, supporting visual comparison across states.Source: B. Physics-INformed World Model · p. 5

Physics identification

  • PIN-WM estimates physics parameters by minimizing rendered-versus-observed video discrepancy after applying task-agnostic end-effector translation actions.The differentiable physics model predicts next states using the identified parameters, while the rendering function aligns target-object motion in visual observations.Source: B. Physics-INformed World Model · pp. 4–5

Differentiable dynamics

  • A velocity-based Linear Complementarity Problem enforces joint, contact, and friction constraints, then semi-implicit Euler integration updates object pose from the resulting velocities.A primal-dual interior-point solver preserves gradient propagation to the physics parameters, and the estimated parameters remain compatible with mainstream simulators.Source: B. Physics-INformed World Model · pp. 5–6

Implementation assumptions

  • The method converts end-effector translation into velocity over action horizon H, assumes robot dynamics can be ignored because the robot is much heavier, and recursively propagates derivatives across H/h simulation steps.The floor remains stationary, while each action is divided into smaller sub-actions for accurate pose integration.Source: B. Physics-INformed World Model · p. 6

C. Physics-aware Digital Cousins

Physics-aware digital cousins address residual Sim2Real discrepancies by perturbing rendering and physics parameters around the identified world model. These variations preserve physics-informed structure while providing diversity for policy training and transfer.

Motivation

  • Digital cousins perturb rendering and physics parameters near their identified values to model discrepancies caused by inaccurate or partial observations.This constrains randomization around the learned world model rather than broadly sampling unconstrained source-domain parameters.Source: C. Physics-aware Digital Cousins · p. 6

Parameter randomization

  • The method randomizes spherical-harmonics rendering parameters and identified physics parameters, modeling lighting, material, and physical variations across digital cousins.Spherical-harmonics parameters are perturbed separately for each Gaussian splat.Source: C. Physics-aware Digital Cousins · p. 6

Policy transfer

  • Using perturbation magnitude δ = 0.1, PIN-WM generates digital cousins for zero-shot policy transfer and trains policies with PPO before direct deployment in the target domain.The world model is compatible with arbitrary reinforcement-learning methods, although PPO is used in the described implementation.Source: C. Physics-aware Digital Cousins · p. 6

IV. RESULTS AND EVALUATIONS

The evaluation examines system identification, policy learning, Sim2Real transfer, and real-world deployment across push and flip manipulation tasks. Simulation provides identification and policy evidence, while real-world tests assess deployment effectiveness.

Evaluation scope

  • Simulation and real-world experiments evaluate whether PIN-WM improves deployable non-prehensile manipulation policies and real-world performance.Simulation supplies ground truth for system-identification accuracy and addresses the first three evaluation questions; real-world tests assess policy deployment.Source: §IV · p. 6

Tasks

  • The benchmark uses push and flip tasks, requiring planar translation and rotation for pushing and full 3D translation and rotation for flipping.The push task moves a planar object to a target pose, whereas flipping turns a lying object upside down by poking.Source: §IV · pp. 6–7

Simulation results

  • Figure 3 shows simulation manipulation trajectories produced by the proposed method for both push and flip tasks.The supplied caption identifies the trajectory scope but does not state a numerical comparison or winner.Source: Figure 3 · p. 7

A. Evaluations in Simulation

Simulation evaluations compare policy performance and system-identification accuracy across data-driven, predefined-physics, and learned-physics baselines. PIN-WM achieves the strongest reported performance by combining differentiable 3D identification with physics-aware digital cousins.

Policy performance

  • PIN-WM achieves the best target-domain policy performance on both push and flip tasks, while removing PADC reduces performance but still outperforms the alternatives.The comparison uses 100 test episodes per method.Source: Table I · p. 7, A. Evaluations in Simulation · p. 8
  • Data-driven baselines perform poorly because they require substantially more data and generalize weakly out of distribution, reinforcing the value of physics priors.Dreamer V2 uses 100 task-agnostic trajectories, and Diffusion Policy imitates 100 task-completion trajectories.Source: A. Evaluations in Simulation · pp. 7–8
  • Physics-aware digital cousins improve transfer by perturbing physics and rendering parameters around identified values, whereas broad domain randomization introduces excessive randomness.RoboGSim randomizes rendering without identifying physics, while Domain Rand + I uses broad parameter variation.Source: A. Evaluations in Simulation · pp. 7–8

Robustness

  • PIN-WM remains effective under noisy geometry despite performing physical-parameter identification under a perfect-geometry assumption.The noise experiments are reported in Appendix B.Source: A. Evaluations in Simulation · p. 8

System identification

  • PIN-WM identifies 3D rigid-body dynamics more accurately than ASID and 2D Physics in both push and flip scenarios.ASID is limited by gradient-free search, while 2D Physics struggles with 3D dynamics.Source: A. Evaluations in Simulation · p. 8, Table II · p. 8
  • Dreamer V2 converges quickly on training data but degrades on test distributions, while PIN-WM maintains stable and efficient push-task learning.The evaluation measures one-step prediction error and reports transition and rotation errors.Source: A. Evaluations in Simulation · p. 8, Figure 4 · p. 8

B. Evaluations in Real-World

Real-world evaluations deploy policies learned from the identified simulator without real-world fine-tuning. PIN-WM transfers to pushing and flipping while outperforming alternative approaches in success and efficiency.

Experiment setup

  • The real-world setup combines a robot with eye-in-hand and eye-to-hand cameras for object and scene observation.The object pose and mesh are initialized using FoundationPose before visual and physics identification.Source: B. Evaluations in Real-World · p. 8, Figure 5 · p. 8

Evaluation scope

  • Real-world comparisons exclude data-driven baselines because collecting their required task-agnostic or task-completion trajectories is highly expensive.The comparison therefore focuses on methods requiring no or few-shot real-world data.Source: B. Evaluations in Real-World · p. 9

Real-world policy performance

  • PIN-WM achieves higher success rates and requires fewer steps than competing methods across real-world push and flip evaluations.The comparison uses identical initial conditions across 20 trials and reports results in Table III.Source: B. Evaluations in Real-World · p. 9, Table III · p. 9

Sim2Real transfer

  • Transfer performance improves when both physical parameters and rendering representations are learned through differentiable optimization and augmented with physics-aware digital cousins.Random-parameter simulators and rendering-only alignment remain physically or visually misaligned with deployment.Source: B. Evaluations in Real-World · p. 9

Push task

  • PIN-WM successfully pushes a T-shaped object to its target in a few steps, while alternatives require longer trajectories or fail.The method also transfers to a larger T-shaped object and additional slippery-surface objects.Source: B. Evaluations in Real-World · p. 9

V. CONCLUSIONS

PIN-WM learns physics-informed 3D rigid-body world models from visual observations and uses physics-aware digital cousins to support Sim2Real transfer. The method identifies parameters from few-shot, task-agnostic trajectories while addressing rendering alignment and rigid-body scope limitations.

  • PIN-WM identifies 3D rigid-body physics parameters from visual observations using few-shot, task-agnostic interaction trajectories.The identified digital twin is diversified through physics and rendering perturbations around learned mean values.Source: §V · p. 9
  • Physics-aware digital cousins perturb physics and rendering parameters around identified values to support Sim2Real transfer.Source: §V · p. 9
  • Visual alignment limits parameter accuracy because robot-generated shadows can distort rendering-loss estimation.The authors suggest differentiable relighting with Gaussian Splatting as a possible remedy.Source: §V · p. 9
  • The current framework focuses on rigid-body dynamics, leaving deformable-object manipulation for future extensions using advanced differentiable physics engines.Source: §V · p. 9

APPENDIX

The appendix details fair baseline implementation and shows that physics-aware randomization around learned parameters outperforms broad uniform randomization. It also reports that PIN-WM achieves the best physical-parameter identification performance among compared methods.

A. Implementation Details for Baselines

  • All baselines use careful implementation choices, including official implementations, shared RL architectures, rewards, hyperparameters, and stopping criteria.Diffusion Policy uses successful trajectories from expert policies trained with ground-truth physical parameters and no randomization.Source: A. Implementation Details for Baselines · p. 14

1) Superiority over uniform randomization:

  • PIN-WM uses a much smaller randomization range around learned parameters θ∗, whereas DR uses a broad range R covering target parameters θ†.The broad DR range reduces performance; shrinking it improves results, while using GT θ† performs best in Table IV.Source: 1) Superiority over uniform… · p. 14, Table IV · p. 14

2) Identified physical parameters:

  • PIN-WM consistently achieves the best physical-parameter identification performance in the controlled comparison reported in Table V.The comparison estimates one parameter at a time while fixing the others at ground-truth values; inertia is excluded because it is a 3 × 3 matrix.Source: 2) Identified physical parameters · p. 14, Table V · p. 14

3) Robustness to geometry noise:

PIN-WM remains robust to increasing geometry noise and performs strongly in further real-world pushing and flipping evaluations. The experiments compare one-step prediction errors under controlled noise and task success across real-world objects and actions.

3) Robustness to geometry noise:

  • At σ = 3.0 geometry noise, PIN-WM remains robust and outperforms ASID using perfect geometry.Prediction errors increase as noise rises, and performance is measured by one-step trajectory error under identical random actions.Source: 3) Robustness to geometry noise · p. 14, Table VI · p. 14

C. Further Real-World Evaluations

Loading 2504.16693v2…