Source-linked AI summary
RealSimLoop: Online Real-to-Sim Adaptation via Differentiable Reduced-Order Simulation with Vision Feedback
Zhihao Cen, Chuhua Xian, Hailin Sun, Yuliang Liufu, Zhen Zhang, Xiangyu Chu, Hongmin Cai, Yunbo Zhang, Guoxin Fang
TL;DR
Deformable-object observations are often insufficient to recover hidden physical quantities, while online real-to-sim adaptation is hindered by expensive optimization and changing material properties. RealSimLoop addresses this with differentiable reduced-order simulation, differentiable rendering, and sliding-window online adaptation. Experiments report stronger visual and physical tracking than offline or marker-based alternatives, while identifying limits for highly dynamic systems and fine-grained material distributions.
Problem
Sparse visual observations do not directly provide hidden deformable-object quantities, while offline real-to-sim adaptation struggles with modeling error, computational cost, drift, and time-varying properties.
Method
RealSimLoop combines neural reduced-order differentiable simulation, differentiable rendering, and a sliding-window objective to adapt material properties online from visual feedback.
Results
The method outperforms offline baselines and marker-based approaches while supporting force prediction, 3D stress reconstruction, and temperature-dependent stiffness tracking.
Takeaways & Limitations
The framework maintains physically meaningful deformable-object simulations while adapting to evolving behavior under quasi-real-time computational constraints.
Takeaways & Limitations
Reduced-order models can face convergence and extrapolation problems for highly dynamic systems or unseen deformation modes, motivating full-space simulation in such cases.
Abstract
from arXiv · showhide
Real-world observations of deformable objects are often sparse or surface-level, while downstream tasks require hidden physical quantities such as internal deformation, stress fields, and interaction forces. Physics-based simulation can recover these quantities, but online real-to-sim adaptation remains challenging due to costly full-space optimization, limited feedback, and time-varying material properties. To address these challenges, we propose RealSimLoop, a differentiable framework for online real-to-sim adaptation using vision data as physical feedback. Our approach achieves quasi-real-time performance by executing differentiable simulation within a reduced-order neural subspace, drastically accelerating the optimization loop. We couple this efficient dynamics model with differentiable rendering, enabling direct gradient backpropagation that leverages high-fidelity pixel data to refine physical parameters such as material stiffness. Furthermore, by employing a sliding-window objective function, RealSimLoop enables robust online adaptation, allowing the system to track time-varying material properties and effectively bridge the real-to-sim gap arising from model reduction or unmodeled dynamics. Extensive experiments demonstrate that our method outperforms conventional offline methods, and we validate the framework's versatility in downstream applications, including external force prediction and 3D stress field reconstruction with novel view synthesis.
1 Introduction
RealSimLoop targets the gap between sparse visual observations and the hidden physical quantities needed for deformable-object applications. It combines online visual feedback, differentiable reduced-order simulation, and sliding-window adaptation to maintain physically meaningful simulations as conditions change.
- 1 Introduction: Sparse or surface-level observations cannot directly provide internal deformation, stress distribution, interaction forces, or material properties needed by downstream applications.Physics-based simulation is used to infer these hidden quantities from limited observations.
- 1 Introduction: Offline adaptation can accumulate open-loop drift because fixed fitted parameters cannot readily track time-varying properties or sudden environmental changes.Conventional methods also risk modeling errors when idealized constitutive laws do not match complex real materials.
- 1 Introduction: RealSimLoop continuously synchronizes a differentiable physical simulator with vision feedback for online real-to-sim adaptation.The framework is designed for quasi-real-time operation while maintaining physical consistency.
- 1 Introduction: The method combines sliding-window optimization, neural reduced-order differentiable simulation, and differentiable rendering to adapt material properties from image observations.These components respectively address evolving behavior, computational cost, and physical feedback from vision.
- 1 Introduction: Compared with offline and purely vision-based pipelines, RealSimLoop is presented as continuously closing the real-to-sim gap while preserving physical consistency across robotic manipulation, force estimation, and structural analysis.The authors validate the framework in virtual and physical settings.
2 Related Work
Prior work combines differentiable simulation, reduced-order modeling, and image-based observation, but online real-to-sim adaptation remains constrained by computational cost and evolving physical properties. RealSimLoop addresses this gap with sliding-window adaptation integrated with reduced-order differentiable simulation.
- 2 Related Work: Image-based observations provide a scalable alternative to motion capture and specialized force-sensing devices, and differentiable rendering enables image-level optimization of simulation states and physical parameters.These approaches supply dense visual feedback but do not by themselves replace physically grounded simulation.
- 2 Related Work: Online real-to-sim adaptation is computationally expensive because it repeatedly simulates and differentiates physical dynamics.Existing MPM- and mesh-free approaches may remain unsuitable for real-time or long-sequence online adaptation.
- 2 Related Work: Most prior real-to-sim methods fit one parameter set to a pre-collected sequence and therefore struggle to track evolving physical properties during dynamic interactions.This limitation motivates an online objective that uses recent observations rather than the full history.
- 2 Related Work: RealSimLoop integrates a sliding-window adaptation objective with reduced-order differentiable simulation to improve efficiency while tracking evolving physical properties online.The supplied pipeline description also includes differentiable rendering and vision-based feedback for optimization.
- 2 Related Work: Differentiable simulation enables gradients of simulation outputs with respect to physical parameters and initial states, supporting gradient-based inverse problems in graphics and robotics.Prior work has applied this capability to rigid-body control, cloth manipulation, and offline soft-body real-to-sim transfer.
- 2 Related Work: Reduced-order simulation accelerates physics by mapping high-dimensional systems into lower-dimensional subspaces, while neural mappings improve representation of nonlinear deformations.Recent neural reduced-order methods provide the basis for efficient differentiable deformable simulation.
3 Preliminary and Overview
The paper formulates real-to-sim adaptation as constrained optimization of material parameters against observed real-world behavior, then improves online responsiveness with reduced-order simulation, sliding windows, and dense visual feedback.
- 3.1 Preliminary of Simulation and Problem of Real-to-Sim: Real-to-sim adaptation minimizes simulation–observation discrepancy by optimizing material parameters subject to the deformable system’s nonlinear state equations.The observations may come from motion capture or multi-view RGB data.
- 3.2 Short Discussion and Overview: 57% elongation? No—offline methods lose accuracy beyond 100% elongation, while fixed material parameters cannot compensate for long-horizon error accumulation.The cited discussion reports collapse at large elongations even with the Yeoh model.
- 3.2 Short Discussion and Overview: The online objective updates material properties over a recent window, limiting accumulated reality-gap error while retaining responsiveness to evolving behavior.The window spans the current steps {T_e−w+1,…,T_e}.
- 3.2 Short Discussion and Overview: A neural subspace mapping replaces full-space time integration with a low-dimensional state z∈R^r mapped to full-space configurations, accelerating online updates.The reduced formulation stacks window states as Z∈R^(w×r) and enforces reduced residual equations.
- 3.2 Short Discussion and Overview: Differentiable 3D Gaussian Splatting converts dense visual observations into a vision-based objective for matching complex deformation patterns and dynamic systems.This extends conventional geometry-only objectives based on marker positions.
4 Method and Details
RealSimLoop maps reduced-order simulation states to synchronized Gaussian rendering primitives, optimizes recent vision observations, and backpropagates gradients through the simulator to refine material parameters online.
- 4.1 GS-Based Vision-Geometry Mapping: Gaussian centers are barycentrically mapped to enclosing tetrahedral vertices and updated with the deformed mesh, preserving spatial alignment during rendering.The mapping uses Gaussian attributes including centers, covariance, spherical harmonics, and opacity.
- 4.2 Window-based Online R2S Adaptation objectives: A sliding-window objective balances material-characteristic capture, temporal responsiveness, computational efficiency, and adaptation to material variation.Smaller windows update faster but are noisier; larger windows are slower but more stable and accurate.
- 4.2 Window-based Online R2S Adaptation objectives: The differentiable renderer compares rendered observations with captured multi-view images, allowing the online loss to update material parameters over the current window.The renderer is based on 3DGS, and the images span K views.
- 4.3 Gradient-based Optimization: 7.56× speedup is reported for the subspace solution, supporting online efficiency in the multi-material cable-driven structure experiment.The figure also shows convergence of Young’s modulus estimates for two silicone materials.
- 4.3 Gradient-based Optimization: The adjoint method propagates material gradients backward through time without explicitly forming dense trajectory Jacobians, while exploiting system sparsity.State gradients are first computed at each timestep and then transferred to material parameters.
- 4.3.1 Vision-to-State Gradient Backpropagation: The adapted simulation directly transfers simulation-optimized control to the real dual-arm robot without additional tuning and reconstructs von Mises stress.The stress distribution is transferred to Gaussian attributes for real-time, multi-view visualization.
5 Results and Discussion
RealSimLoop demonstrates online real-to-sim adaptation across deformable manipulation, stiffness tracking, collisions, and stress-related downstream tasks, while reduced-order simulation accelerates optimization. Vision feedback and sliding-window adaptation improve reconstruction and track changing material behavior, though reduced-order models can struggle with highly dynamic or unseen deformations.
- Computation and Physical Results: In dual-arm bar manipulation, reduced-order adaptation cuts computation from 3.30 s to 0.53 s per frame while achieving 4.6 mm average reconstruction error.The error corresponds to 2.3% of the model size.
- Downstream Applications: Online adaptation supports shape control, stress reconstruction, spatially varying material identification, and external cable-force prediction.Updated material properties enable simulation-based control, while reconstructed stress fields support multi-view visualization and force prediction.
- Dynamic and Time-Varying Behavior: The method tracks increasing stiffness as temperature falls and matches transient deformation better than the offline baseline in high-speed collision tests.For the ball-bouncing case, PSNR increases from 18.0 to 22.6 for the first key frame with the online method.
- Comparison with Offline Methods: Online adaptation outperforms fixed offline models for long-horizon and aging-material scenarios, including PSNR of 25.91 versus 24.42 in bar manipulation.The offline baseline fails to track temporal stiffness changes and accumulates reconstruction errors over time.
- Ablation Study: Increasing subspace dimension or temporal-window size improves accuracy but increases update time, with marginal gains and approximately linear time growth at larger settings.The parameters r, w, and c are tuned per model to balance accuracy and computational efficiency.
- Vision-based Feedback: Vision feedback lowers average vertex L2 error from 0.052 m to 0.026 m and converges faster than marker-based input.The denser spatial information in images is associated with faster material-parameter convergence; the reported vertex error reduction is about 50%.
- Limitations and Future Work: Reduced-order models may fail to converge in highly dynamic cases and may extrapolate poorly to unseen deformations or out-of-range material parameters.The high-speed ball case therefore reports full-space results, while future work considers adaptive refinement and improved extrapolation.
6 Conclusion
RealSimLoop integrates differentiable reduced-order simulation with visual feedback for online real-to-sim adaptation, capturing time-varying dynamics while reducing computational bottlenecks. Experiments show effectiveness across manipulation and stiffness-tracking tasks, with support for force prediction, stress reconstruction, and robot manipulation.
- RealSimLoop consistently outperforms offline baselines and marker-based approaches across virtual and physical experiments.
- The framework combines a neural subspace solver with window-based optimization to reduce computation while capturing time-varying dynamics.
- Experiments cover elastic object manipulation, cable-driven multi-material manipulation, and temperature-dependent stiffness tracking.
- Downstream applications include force prediction, 3D stress reconstruction, and robot-based manipulation.