Source-linked AI summary
gradSim: Differentiable simulation for system identification and visuomotor control
Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jerome Parent-Levesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, Sanja Fidler
TL;DR
Estimating physical properties from video is difficult because image formation discards 3D information, while existing solutions require costly precise 3D labels. gradSim combines differentiable multiphysics simulation with differentiable rendering to backpropagate from pixels to physical attributes and controls, achieving competitive or superior performance across evaluated settings and supporting visuomotor control without 3D supervision.
Problem
Video-based system identification is ill-posed, and precise 3D labels are labor-intensive and infeasible for systems such as deformable solids or cloth.
Method
gradSim jointly models differentiable scene dynamics and image formation, enabling gradients from video pixels to underlying physical and dynamical attributes.
Results
gradSim recovers physical properties from video across rigid, deformable, and thin-shell bodies and supports visuomotor control, with performance competitive to or sometimes superior to physics-only differentiable simulators.
Takeaways & Limitations
The unified computation graph enables physical estimation and challenging visuomotor control without relying on state-based 3D supervision.
Takeaways & Limitations
∇Sim has limited capability for contact-rich motion, does not currently implement articulated bodies, and is inept at handling tiny masses of 100g or less.
Abstract
from arXiv · showhide
We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph -- spanning from the dynamics and through the rendering process -- enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
1 INTRODUCTION
gradSim addresses video-based physical-property estimation by jointly differentiating scene dynamics and image formation, avoiding dependence on precise 3D supervision. The framework supports physical identification across body types and visuomotor control from image-space observations.
- Motivation: Video-based estimation of physical attributes is ill-posed because image formation loses information about 3D scene dynamics.The challenge includes inferring properties such as mass and elasticity from image sequences while accounting for the imaging process.
- Motivation: Precise 3D labels required by existing approaches are labor-intensive and infeasible to generate for deformable solids or cloth.
- Approach: gradSim couples differentiable scene dynamics with differentiable image formation, enabling backpropagation from video pixels to physical and dynamical attributes.Unlike differentiable physics engines that require state-space supervision, gradSim uses target information specified in image space.
- Contributions: The framework demonstrates recovery of friction, elasticity, deformable material parameters, and visuomotor controls exclusively from video observations.
- Evaluation: gradSim is evaluated on rigid, deformable, and thin-shell bodies, with performance competitive to or sometimes superior to physics-only differentiable simulators.It is also applied to visuomotor control tasks involving deformable solids and cloth.
2 ∇Sim: A UNIFIED DIFFERENTIABLE SIMULATION ENGINE
gradSim unifies differentiable physical dynamics and rendering into a simulation function that maps scene parameters and time to images. This computation graph supports gradient-based estimation from video while allowing differentiable rendering choices and adjustable simulation rates.
- Unified simulation: gradSim treats physics estimation and rendering as a unified simulation process rather than disjoint tasks.Its two main components are a differentiable physics engine for scene states and a differentiable renderer for 2D images.
- Simulation function: The simulation function maps state and parameter vector p plus time t to an image I, whose gradients support estimating physical parameters from video.A gradient-based optimizer uses the change in rendered output caused by infinitesimal parameter perturbations.
- Differentiable physics: The physical state uses generalized coordinates q and velocities u, while model parameters θ describe system-specific design or material properties.The combined state is represented as s(t) = [q(t), u(t)].
- Differentiable physics: System dynamics are represented through implicit time-step relations g(s−, s+, θ) = 0 connecting initial and updated states with model parameters.The formulation abstracts arbitrary discrete time-integration schemes while retaining differentiability through the physics operation.
- Differentiable rendering: SoftRas and DIB-R smooth rasterization discontinuities so rendering can provide gradients through triangle edges and texture sampling.
- System design: Physics simulation and rendering run at independent adjustable rates, trading computation for accuracy by rendering fewer frames than dynamics updates.
3 EXPERIMENTS
The experiments test ∇Sim for recovering physical parameters from video and for image-based visuomotor control without relying on 3D supervision. Across rigid, deformable, and cloth settings, ∇Sim achieves accurate estimation, smooth optimization landscapes, and effective control gradients.
- 3.1.1 RIGID BODIES (RIGID): ∇Sim estimates mass from video with an absolute relative error of 9.01e-5, nearly two orders of magnitude better than ConvLSTM and competitive with 3D-supervised diffphysics.The rigid-body evaluation uses videos of objects receiving known impulses and compares against PyBullet + REINFORCE, diff. physics only, ConvLSTM, Average, and Random baselines.
- 3.1.1 RIGID BODIES (RIGID): ∇Sim precisely recovers rigid-body friction and elasticity from video, while PyBullet + REINFORCE is sensitive to simulation parameters and performs worse.The contact parameters are estimated from video observations alone, whereas diffphysics requires accurate 3D ground truth.
- 3.1.2 DEFORMABLE BODIES: ∇Sim accurately recovers parameters for 100 deformable-object instances, including per-particle masses and Lamé material properties, as well as cloth velocities.The deformable experiments include cloth, balls, and beams; beam elasticity is recovered by matching deformation under gravity.
- 3.1.3 SMOOTHNESS OF THE LOSS LANDSCAPE IN ∇Sim: ∇Sim’s image-space loss is well-behaved across mass initializations from 0.1 kg to 5 kg, whereas PyBullet + REINFORCE exhibits multiple local minima and a narrow convergence region.Using MSE on the first and last predicted and true video frames provides the best gradients in the reported mass experiment.
- 3.2 VISUOMOTOR CONTROL: ∇Sim solves the 2D walker control environment within three gradient-descent iterations and reaches soft-body target states using a single image as the implicit goal.In the 3D FEM control task, diffphysics uses timestep-level 3D supervision, while ∇Sim reaches the goal with more iterations; for cloth, center-of-mass control is often unable to determine configuration accurately.
- 3.1.4 MODELING IMPERFECTIONS: Imperfect dynamics models harm parameter identification more than imperfect renderers, with incorrect rigid/deformable object modeling causing the most severe degradation.Modeling a deformable body as rigid can make deformation parameters irrecoverable, while accounting for rendering imperfections performs better than dynamics-imperfect settings.
4 RELATED WORK
Related work spans learned dynamical representations, physics-structured models, sensor-based reasoning, and differentiable 3D rendering. These approaches motivate combining scene dynamics with image formation for video-based physical inference.
- Learned dynamics methods model system states or observations as embeddings, while explicit dynamics models reduce learnable parameters and improve interpretability.
- Physics-based deep learning injects structure through Lagrangian or Hamiltonian operators, conserved quantities, or ground-truth supervision.
- Sensor-based and intuitive-physics approaches predict force effects and model multi-object interactions from observed signals.
- Neural scene-representation methods model 3D structure with voxels, meshes, or implicit shapes.
- Differentiable rendering computes image gradients with respect to scene geometry, camera, and lighting, using rasterization methods that smooth rendering discontinuities.
5 CONCLUSION
The paper presents a differentiable simulator integrating physical dynamics, rendering, and automatic differentiation across rigid, deformable, and thin-shell systems. It concludes that this unified approach supports video-based estimation and visuomotor control, while identifying extensions toward richer real-world simulation.
- ∇Sim enables system identification from videos by differentiating through the physical processes governing dynamics and image formation.
- Future work targets contact-rich motion, articulated bodies, and higher-fidelity physically based renderers to move closer to real-world operation.
- The simulator represents physical states with generalized coordinates and velocities, and propagates them through parameterized dynamics whose gradients can be computed with discrete adjoints.
- The framework uses source-code transformation and runtime-generated C++/CUDA kernels wrapped as PyTorch operations for automatic differentiation.
- Deformable solids use tetrahedral FEM with Neo-Hookean elasticity, while thin-shells use triangular elements with bending and lift/drag models.
- Its physical models cover rigid bodies, compliant contact and friction, deformable solids, thin-shells, pendula, mass-springs, and incompressible fluids.
B DISCRETE ADJOINT METHOD
The discrete adjoint method expresses time integration as an implicit relation between successive states and model parameters, enabling gradients through general time-stepping schemes. It balances explicit-method memory and stability issues against implicit-method computational overhead and dissipation.
- The method represents an arbitrary time-stepping scheme as an implicit relation g(s−, s+, θ) = 0 between initial state, final state, and model parameters.
- This formulation supports both explicit and implicit integration methods within the same differentiation framework.
- Explicit methods may require very small stable time steps and storing every input state, whereas implicit methods can add computational overhead or numerical dissipation.
- The paper illustrates the discrete adjoint procedure with semi-implicit Euler and then discusses how Jacobian structure affects adjoint computation for semi-implicit and fully implicit methods.
- The simulator uses triangular and tetrahedral FEM discretizations with angle-based and volumetric activation parameters, matching its triangle-based differentiable rasterization pipeline.
C.1 FINITE ELEMENT METHOD
The finite-element implementation models deformable solids and thin shells with mesh-based Neo-Hookean elasticity and differentiable physical parameters. Its implementation uses transformed Python kernels that generate CPU/GPU code and integrate with PyTorch.
- The deformable-body model uses a Neo-Hookean constitutive law with Lamé parameters and per-element actuation for tetrahedral finite elements.
- Lamé parameters λ and µ control resistance to shearing and volumetric strains and can vary by element to represent heterogeneous materials.
- Mesh-based discretization uses triangle meshes for thin shells and tetrahedral FEM for solids, enabling integration with the triangle-based differentiable rasterizer.
- Relaxed contact models replace hard non-penetration and friction constraints with differentiable penalty and hinge formulations.
- The framework also implements pendula, smoke, and other differentiable simulations, with parameter estimation demonstrated by matching rendered videos.
- Source-code transformation generates forward and reverse C++ kernels for CPU or GPU execution, wrapping each kernel as a PyTorch autograd operation.
E MPC CONTROLLER ARCHITECTURE
∇Sim’s optimization landscapes are generally smooth and well behaved, supporting physical-parameter estimation, while REINFORCE can introduce sensitive local optima for mass estimation. The MPC controller uses a compact three-layer network driven by phase-shifted sinusoidal signals.
- E MPC CONTROLLER ARCHITECTURE: The MPC controller uses simulation time as input, generates N phase-shifted sinusoidal signals, and maps them through a zero-bias fully connected layer and final activation layer.The output is a vector of per-element activation values.
- Loss-landscape analysis: ∇Sim uses pixelwise mean-squared error to estimate physical parameters and analyzes its loss landscape to assess gradient relevance.The framework contains multiple nonlinear functional blocks.
- Loss-landscape analysis: A unique, dominant minimum at λ = µ = 1000 supports precise elasticity-parameter estimation for deformable FEM solids.The analysis identifies the well-behaved loss landscape as a key contributor to ∇Sim’s estimation ability.
- Loss-landscape analysis: REINFORCE can produce poor local optima near the current mass estimate, requiring careful tuning of step size, sampling noise, and sampling range.Instability occurs for masses close to zero, reducing the method’s utility when these hyperparameters are unknown.
- Loss-landscape analysis: Video length and frame rate change loss-landscape steepness, but all tested cases retain a smooth landscape with the same unique minimum.Shorter videos tend to produce steeper landscapes than longer videos.
G DATASET DETAILS
The rigid-body experiments use a curated 14-mesh dataset with randomized physical attributes and compare differentiable simulation against gradient-estimated and ConvNet baselines. PyBullet-REINFORCE is configured to match the dataset’s initial scene and rendering conditions while using reward decay for optimization stability.
- G DATASET DETAILS: The rigid-body dataset contains 14 simplified meshes spanning primitive shapes, fruits, vegetables, animals, office objects, and airplanes.Meshes contain 250 or fewer vertices to accelerate collision detection.
- Baselines: The PyBullet baseline estimates gradients with REINFORCE because the simulator is otherwise non-differentiable.The implementation follows approaches inspired by Wu et al. and Rezende et al.
- Baselines: PyBullet uses the dataset’s starting pose and camera parameters so that the correct mass would produce frames aligned with those generated by ∇Sim.The mass is randomly initialized in [0, Nv], where Nv is the number of vertices.
- Baselines: Reward decay progressively reduces normalized frame rewards during PyBullet-REINFORCE optimization.The decay is updated after each optimization step using reward_decay = reward_decay ∗ decay_factor.
- Baselines: The ConvNet baseline builds on EfficientNet-B0 and predicts five physical parameters by minimizing mean-squared error.Its architecture includes two convolutional layers followed by linear layers and ReLU activations.
I COMPUTE AND TIMING DETAILS
∇Sim supports parameter-estimation experiments on laptop GPUs and summarizes a broad set of optimizable simulation parameters, including state, material, actuation, gravity, friction, elasticity, and external forces.
- I COMPUTE AND TIMING DETAILS: Rigid and nonrigid parameter-estimation experiments run in under 5–20 minutes per object on CPU and under 1 minute on a GTX 1060 GPU.These timings are reported for a laptop with an Intel i7 processor and GeForce GTX 1060 GPU.
- I COMPUTE AND TIMING DETAILS: Visuomotor control experiments take about 30 minutes per episode on CPU and under 5 minutes per episode on GPU.The supplied passage names control-fem and control-cloth among these experiments.
- I COMPUTE AND TIMING DETAILS: ∇Sim exposes optimizable parameters for particle positions and velocities, mass, orientation, spring properties, actuation, gravity, friction, elasticity, and external forces.These categories are listed in Table 6.
K LIMITATIONS
∇Sim has limitations involving numerical stability, unsupported articulated bodies, contact-rich discontinuities, and unmodeled real-world phenomena. These constraints define important boundaries for deployment beyond the demonstrated settings.
- K LIMITATIONS: ∇Sim and ∇PyBullet are inept at handling tiny masses of 100g or less, which may require physics-engine and numerical-stability improvements.The limitation is tied to optimizing physical parameters for such objects.
- K LIMITATIONS: Articulated bodies are not implemented, limiting systems that use multiple prismatic joints and their additional degrees of freedom.This is identified as a current implementation gap.
- K LIMITATIONS: ∇Sim has limited capability for contact-rich motion with many discontinuities, motivating more sophisticated contact detection and LCP-based solutions.The current system can model contacts involving simple geometries such as triangle meshes and planar surfaces.
- K LIMITATIONS: Real-world deployment remains constrained by physical phenomena that physics engines do not model and that must be resolved for in-the-wild use.The paper frames these shortcomings as avenues for subsequent research.
L BROADER IMPACT
∇Sim links physically grounded modeling with gradient-based optimization to improve interpretability, debuggability, and potentially training efficiency. Its practical use remains bounded by modeling bias, upfront simulation engineering, and possible misuse.
- Explicitly modeled visual dynamics provide a bridge between gradient-based and model-based optimization while supporting human explainability and debuggability.
- ∇Sim trades neural flexibility for physical interpretability, isolating bias in physically grounded variables that users can probe through parameter or pixelwise gradients.This supports debugging discrepancies between simulated and observed behavior.
- ∇Sim could help identify controller sensitivities, test altered physical conditions, and accelerate reinforcement-learning convergence with lower training energy use.The same differentiable machinery could also enable physically plausible adversarial scenes that induce unsafe actions.
- Operationalizing differentiable simulation requires substantial upfront engineering and effort to identify and replicate unmodeled real-world dynamics.Modeling choices also retain inherent bias and require active human participation to mitigate it.