One-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering

I. INTRODUCTION

• Existing differentiable-simulator methods optimize physical parameters but assume known geometry and appearance, while NeRF and Gaussian Splatting lack surfaces suited to rigid-body collision detection.The authors state that no existing method jointly identifies all three properties from sparse robot observations.Source: §I · p. 1
• The proposed representation couples differentiable point-based geometry with a grid-based appearance field, enabling joint optimization of rigid-object shape, appearance, and physical properties.It uses Shapeas-Points reconstruction and interpolates vertex textures from the appearance grid.Source: §I · pp. 1–2
• The method identifies world models online from sparse robot observations through end-to-end differentiable simulation and rendering.The world model takes the form of a rigid body simulator.Source: §I · pp. 1–2
• The authors report accurate world models from a single episode of robot interaction with the environment.The evaluation covers simulated and real-world identification problems.Source: §I · p. 2

A. Differentiable Rigid Body Simulator

• Differentiable simulators handle contact dynamics with smooth contact models, sub-gradients of linear complementarity problems, or implicit gradients of nonlinear optimization.The cited approaches include –.Source: A. Differentiable Rigid Body Simulator · p. 2
• Most reviewed simulators do not provide geometry gradients; the exceptions cited are,, and [17].Source: A. Differentiable Rigid Body Simulator · p. 2
• This work adopts Strecke et al. for physical realism, numerical stability, and GPU-accelerated computation.Source: A. Differentiable Rigid Body Simulator · p. 2

B. Learnable Geometry and Appearance Models

• NeRF and Gaussian Splatting learn detailed appearance from sparse images but lack clear rigid-object surfaces for collision detection and typically require many diverse views.Source: B. Learnable Geometry and Appearance… · p. 2
• The SaP-based representation combines point-cloud flexibility for large geometric and topological changes with mesh-based volumetric collision detection, enabling end-to-end image-based shape optimization.Source: B. Learnable Geometry and Appearance… · p. 2

C. World Models

• Existing world-model approaches either identify dynamics from full state information, lack simultaneous GAP optimization, or predict RGB frames with data-driven generative models.The paper contrasts these approaches with a physics-grounded general-purpose rigid simulator that can provide information such as contact forces.Source: C. World Models · p. 2
• The differentiable pipeline converts oriented surface points into an occupancy grid and mesh, then simulates and renders predicted scenes for comparison with observed RGB-D images.The caption describes gradient descent jointly optimizing shape, appearance, and physical parameters.Source: Figure 2 · p. 3

III. PROBLEM DEFINITION

• The identification problem estimates scene parameters θ from sparse observations of end-effector pose, sensed contact force, RGB-D images, camera pose, and segmentation masks.Source: §III · p. 3

A. Differentiable Object Representation

• The representation challenge is to support complex geometry, topology, and appearance while enabling collision detection; point clouds and Gaussian Splatting lack penetration-depth information, while meshes limit large geometric and topological changes.Source: A. Differentiable Object Representation · p. 3
• Shapeas-Points represents surfaces as oriented points, whose normal directions guide Poisson reconstruction of an implicit occupancy field.The field describes whether locations are inside or outside the solid geometry.Source: A. Differentiable Object Representation · p. 3
• Differentiable marching cubes converts the occupancy field into a triangle mesh for collision detection, while an appearance grid supplies vertex properties through trilinear interpolation.This work stores and renders color, though the same approach can incorporate other appearance properties.Source: A. Differentiable Object Representation · p. 4
• The object’s physical parameters include mass-inertial properties and frictional coefficients, while terrain is represented by an oriented, colored point cloud.The terrain point cloud is rendered without simulating interactions between two terrains.Source: A. Differentiable Object Representation · p. 4

B. Differentiable Simulator

• The simulator is limited to unconstrained rigid-body dynamics with dry-frictional contacts; soft-body and articulated-object constraints are only noted as potentially incorporable.The paper states that differentiation for such constraints has been studied, for example in.Source: B. Differentiable Simulator · p. 4
• At each step, a mixed linear complementarity problem computes normal and tangential contact forces before the configuration is integrated forward.The forces act at detected contact points, and the integration uses timestep δt.Source: B. Differentiable Simulator · p. 4
• Primal-dual solution and sensitivity analysis make the contact problem differentiable, propagating gradients through contact Jacobians to object geometry.The method adopts the GPU implementation of Strecke et al..Source: B. Differentiable Simulator · p. 4

C. World Model Identification

• Two-stage optimization and 3D foundation-model initial guesses address poor initial geometry, particularly in occluded regions where naive gradient descent may converge poorly.The foundation models generate initial guesses from partial visual observations.Source: C. World Model Identification · p. 4

1) Two-stage Optimization:

• The first stage fits only the first observation, using RGB, depth, visible-point-cloud, penetration, balance, and shape-regularization losses.The unilateral Chamfer term compares observed object points with mesh vertices; penetration and balance terms constrain physical settling, while regularizers constrain shape.Source: 1) Two-stage Optimization · pp. 4–5
• The balance loss sums object-position changes during three no-action simulation steps, providing gradients that can expand geometry into unseen regions as the object settles.The authors describe this as a cue for filling occluded geometry, such as an object's underside on a table.Source: 1) Two-stage Optimization · p. 5
• Both depth and point-cloud losses are needed: depth discourages geometry extending into background, while point-cloud matching helps expand meshes that are too small.Depth alone may provide no gradient to expand an undersized mesh; point-cloud matching alone does not prevent expansion beyond observed points.Source: 1) Two-stage Optimization · p. 5
• The second stage applies robot controls across timesteps, retains the first-frame fitting loss except penetration and balance, and adds point-cloud and end-effector-position losses.For computational efficiency, it renders two intermediate frames and the last frame rather than every timestep; gradients pass through rendering and simulation derivatives.Source: 1) Two-stage Optimization · p. 5

2) Geometry Prior:

• TripoSR generates an initial object mesh from the segmented RGB image, which is registered to the first-frame object point cloud using RANSAC and scale-aware iterative closest point.The reconstruction model is used because the optimization relies on a reasonable initial guess and unseen geometry may otherwise fail to make terrain contact.Source: 2) Geometry Prior · p. 5
• The experiments assume terrain hidden by the object is flat and optimize terrain colors but not point-cloud positions.The authors suggest jointly predicting and optimizing hidden terrain geometry may require more online data to resolve contact ambiguities.Source: 2) Geometry Prior · p. 5
• The optimized Drill geometry matches ground truth across three novel simulation views, but lacks fine details unseen from the top view.The rendered terrain checkers are less sharp than ground truth because the colored terrain point cloud is rendered as points.Source: Figure 6 · p. 6
• The shape optimization corrects a simulated Box's terrain-intersecting underside and thickens an initially too-thin Sugar mesh so it satisfies physics constraints.The examples contrast a simulated object and a physical object during stage one.Source: Figure 8 · p. 6

• The method is evaluated first with simulated data and then in real-world experiments.Source: §V · p. 6

A. Simulation Experiments

• In simulation, a training trajectory is selected from 24 pushes of each object, with the remaining trajectories reserved for evaluation.The PyBullet experiments include a box and eight YCB objects; pushes vary starting location and direction.Source: A. Simulation Experiments · p. 6
• The simulation comparison evaluates shape and physical-parameter identification for generalization to new physical interactions, using PhysGS as a baseline.For PhysGS, total mass and friction are grid-searched against partial Chamfer distance on the training trajectory’s final frame.Source: A. Simulation Experiments · p. 7
• The method identifies simulation dynamics accurately and has low trajectory-prediction errors, while PhysGS generalizes very poorly to test trajectories.Average rotation error reaches 28.0° for TunaCan because surface-point tracking cannot properly differentiate cylindrical-object rotation; novel-view synthesis achieves 0.00225 MSE, 0.965 SSIM, and 26.5 PSNR.Source: A. Simulation Experiments · p. 7

B. Physical Experiments

• Physical tests use a UR5e arm, force-torque sensor, and overhead RGB-D camera to push Drill, Sugar, and Mustard along six trajectories, with one for training.Each trajectory contains 48 time steps at 0.03 s intervals.Source: B. Physical Experiments · p. 7
• 0.186 kg mass identification error and 6.10 mm unilateral Chamfer distance were achieved on average across testing trajectories and objects in the physical experiments.The Chamfer distance compares observed object points with predicted geometry at the trajectory’s last frame.Source: B. Physical Experiments · p. 7

VI. LIMITATIONS

• The method assumes ground-truth object masks, lacks advanced appearance and lighting models, and can produce rendering artifacts; an optimization run takes under 15 minutes on the reported PC.Rotation tracking also struggles for objects such as cylinders when surface points do not identify rotation properly.Source: §VI · p. 7

VII. CONCLUSION

• The conclusion characterizes joint identification of object physical properties, geometry, and appearance as a crucial step in downstream robot manipulation.Source: §VII · p. 7
• Future directions include multi-body dynamics, richer appearance models, and physics-based perception to correct wrong object masks.Source: §VII · p. 8