Source-linked AI summary
SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation
Mu Huang, Hui Wang, Kerui Ren, Linning Xu, Yunsong Zhou, Mulin Yu, Bo Dai, Jiangmiao Pang
TL;DR
Deformable-object real-to-sim manipulation needs accurate, stable modeling of dynamics driven jointly by robot actions and environmental interactions. SoMA uses a unified, robot-conditioned neural simulator over hierarchical Gaussian splats, and reports 20% improvement with stable long-horizon simulation and generalization to unseen interactions. Its current evaluation remains limited to four object categories and unseen actions within that scope.
Problem
Real-to-sim simulation of deformable objects remains challenging because existing approaches rely on predefined physics or lack robot-conditioned control for rich interactions.
Method
SoMA jointly represents robots, deformable objects, and environments in Gaussian-splat space, using robot-conditioned alignment, force-driven dynamics, and multi-resolution training.
Results
20% improvement in RGB and depth performance is reported, with accurate, stable long-horizon simulation and generalization to unseen interactions across complex manipulation tasks.
Takeaways & Limitations
SoMA supports action-conditioned real-to-sim manipulation without predefined physical models, including stable simulation of complex tasks such as long-horizon cloth folding.
Takeaways & Limitations
Evaluation covers four object categories and unseen actions within that scope, so broader validation across geometries, materials, and interaction patterns is still required.
Abstract
from arXiv · showhide
Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.
1. Introduction
SoMA addresses the difficulty of stable, accurate real-to-sim simulation for deformable-object manipulation by jointly modeling objects, robot actions, and environmental effects in a learned representation. Its robot-conditioned, force-driven, multi-resolution design supports long-horizon interaction and improves evaluation performance by 20%.
- Motivation: Real-to-sim manipulation requires scalable behavioral reproduction because real-world deformable-object data collection is costly and risky.The resulting simulations can support data synthesis, augmentation, and policy learning.
- Motivation: Practical simulators must preserve deformable geometry and dynamics under sustained robot–object interaction despite occlusion and complex contact.They must balance physical fidelity with long-horizon interaction consistency without introducing real-to-sim bias.
- Limitations of Existing Approaches: Existing physics-based methods provide consistent interactions but require predefined models, while differentiable and neural approaches remain limited by simplified assumptions or weak interaction modeling.The supplied related-work passage is truncated, so the comparison is stated only at the supported level.
- SoMA: SoMA unifies deformable objects, robot actions, and environmental effects in a learned Gaussian-splat representation for causal, stable long-horizon simulation without predefined physical rules.Its design includes robot-conditioned alignment, force-driven splat dynamics, and multi-resolution training.
- Evaluation: 20% improvement in RGB and depth performance is reported across public benchmarks and a new real-world dataset, alongside advantages for sustained interactive manipulation.The contribution statement identifies this as state-of-the-art performance and reports the improvement at the RGB-and-depth level.
2. Related Works
Prior work spans geometric reconstruction, physics-based simulation, and neural dynamics modeling. These approaches provide useful scene or motion representations, but the supplied passages emphasize limitations in parameter specification and interaction modeling.
- 3D Reconstruction and Scene Representation: Classical multi-view geometry and newer Gaussian-Splatting methods recover camera poses and scene structure for vision-based simulation pipelines.The passage names SLAM, COLMAP, VGGT, Pi3, and AnySplat as representative approaches.
- Physics-Based Simulators: Physics-based simulators model deformable objects under controlled settings but require carefully specified physical parameters and simulator configurations.The passage identifies FEM, MPM, and SPH as examples.
- Neural-Based Dynamics Modeling: Dynamic-scene Gaussian Splatting methods provide temporally consistent geometry but mainly reconstruct observed motions with limited interaction modeling.This distinguishes motion reconstruction from interaction-aware simulation.
- Neural-Based Dynamics Modeling: Neural simulators on Gaussian-Splat representations regress future states from past states, improving generalization over re...The supplied passage ends mid-sentence, so no stronger claim is added.
3. Preliminaries
SoMA represents deformable objects as hierarchical Gaussian splats and propagates motion through a multi-level graph. This structure supports coherent global motion while retaining local deformations.
- Hierarchical Gaussian-Splat Representation: Each Gaussian splat encodes spatial position, anisotropic covariance, mass, and additional physical attributes.The representation is intended to model deformable-object state at the splat level.
- Hierarchical Graph: Gaussian splats are organized into a hierarchical graph to improve efficiency and capture deformable-object dynamics across spatial scales.The hierarchy is constructed bottom-up from fine splats into coarser cluster levels.
- Dynamics Propagation: Graph neural networks propagate dynamics top-down by predicting latent motion and deformation for cluster nodes and transferring them to child nodes.This learned transformation connects coarse-level predictions to finer Gaussian-splat states.
- Dynamics Propagation: Hierarchical propagation combines coherent global motion with preserved local deformations at the Gaussian-splat level.The representation therefore links multi-scale structure to deformable motion modeling.
4. Method
SoMA builds a unified real-to-sim neural simulator that maps robot, object, and environmental states into a shared space and predicts force-driven deformable dynamics. Its hierarchical Gaussian-splat representation, robot conditioning, and blended multi-resolution supervision target stable long-horizon interaction under partial observability.
- SoMA Framework: SoMA jointly models robots, deformable objects, and environments in a shared simulation space, allowing robot joint-space actions to directly drive object dynamics.The simulator is designed for interaction-aware and numerically stable long-horizon simulation from visual observations.
- Scene Initialization via R2S Mapping: A scene-to-simulation mapping aligns reconstructed geometry, robot kinematics, camera poses, and physical reference frames across heterogeneous coordinate systems.The mapping recovers scale and rigid transformations, computes robot poses from joint configurations, and resolves gravity using the supporting-plane normal and camera direction.
- Force-Driven GS Dynamics Modeling: The simulator represents deformable dynamics with interaction forces: environmental effects are explicit, while robot-induced forces are inferred through an interaction graph and propagated hierarchically across Gaussian splats.Each node’s dynamics model predicts motion from historical states and aggregated interaction forces, rather than relying only on state-based regression.
- Training Strategy: Multi-resolution temporal training first captures long-range dynamics at a larger stride, then learns fine-grained dynamics from randomly sampled subsequences at the original resolution.The training design addresses accumulated prediction errors in Gaussian-splat simulation; super-resolution is used for reconstruction while dynamics training remains at original image resolution.
- Immigrated Supervision: Blended supervision applies image losses only to visible object regions and adds physics-inspired consistency for occluded splats.Masked reconstruction limits visual supervision to reliable observations, while hierarchical regularization propagates plausible dynamics and mitigates long-horizon drift.
5. Experiments
Experiments evaluate SoMA on real-world robot manipulation through resimulation, unseen-action generalization, challenging T-shirt folding, and component ablations. Across these settings, SoMA maintains stable long-horizon dynamics and outperforms the compared baselines.
- Datasets and tasks: The evaluation uses rope, doll, cloth, and T-shirt datasets collected with synchronized RGB images and robot joint states.Sequences contain diverse configurations and actions, with 7:3 training-test splits.
- Datasets and tasks: Resimulation evaluates reconstruction accuracy on training trajectories, while generalization evaluates unseen manipulation sequences using open-loop robot-action-conditioned simulation.Both tasks initialize models with reconstructed Gaussian splats and condition simulation on per-frame robot actions.
- Resimulation and generalization: SoMA achieves the best performance across all reported resimulation and generalization metrics against PhysTwin and GausSim.The evaluation includes RGB metrics and depth-based geometric metrics.
- Resimulation and generalization: SoMA maintains accurate and stable dynamics on unseen action trajectories and contact configurations, whereas competing simulators show larger deviations or incoherent deformations.The reported advantage is attributed to conditioning dynamics on robot actions and interaction cues.
- T-shirt folding: On T-shirt folding, SoMA stably simulates the full process with coherent geometry and realistic dynamics while outperforming baselines across all metrics.The task involves long-horizon dynamics, large deformations, and frequent self-contacts.
- Overall findings: These results support SoMA as a simulator for task-level soft-body manipulation with complex interactions.The experiments emphasize stable and accurate simulation across diverse objects and challenging manipulation settings.
- Ablations: The full model performs best overall; removing multi-resolution training consistently reduces performance, while image-only supervision causes the largest degradation, especially in generalization.Joint training slightly lowers resimulation accuracy but improves generalization.
6. Applications
SoMA provides a forward-simulation backend for deformable objects under rich robot–object interactions. Its stable long-horizon dynamics and direct robot-action conditioning support simulation-driven robot learning and policy transfer.
- Applications: SoMA predicts deformable-object dynamics under different manipulation strategies through stable long-horizon simulation conditioned directly on robot actions.It provides a practical virtual environment for analyzing complex soft-body behaviors, including self-contact-heavy T-shirt folding.
7. Conclusion
SoMA is a Gaussian-splat neural real-to-sim simulator that learns deformable dynamics and robot–object interactions from RGB observations without predefined physical models. Experiments show stable long-horizon simulation and strong generalization across unseen interactions and complex tasks.
- Conclusion: SoMA learns deformable-object dynamics end-to-end from RGB observations on Gaussian splat representations without predefined physical models.The simulator explicitly models robot–object interactions and conditions long-horizon simulation on robot actions.
- Conclusion: Experiments demonstrate strong generalization to unseen interactions and consistent improvements over prior methods across diverse objects and complex manipulation tasks.The conclusion specifically includes cloth folding among the evaluated complex tasks.
Impact Statement
SoMA is positioned as an interaction-aware, data-driven simulator that explicitly represents physical states while targeting real-to-sim robot manipulation. It distinguishes itself from passive reconstruction, appearance-focused video models, and rule-based simulation through a balance of physical consistency, controllability, and fidelity.
- Technical paradigms: Passive 4D reconstruction recovers temporally consistent geometry and appearance but does not explicitly model physical states or interactions for controllable manipulation.This limits its suitability for physical reasoning in the paper’s comparison.
- Technical paradigms: Video world models prioritize perceptually plausible future observations, whereas physics-oriented simulation prioritizes physical or geometric consistency under interventions.The paper notes that appearance-focused models can produce view-inconsistent or physically implausible behavior under external interventions.
- Technical paradigms: Rule-based simulators depend on predefined formulations and manually tuned parameters, while data-driven neural simulators learn physical properties and interaction dynamics from data.The comparison frames this as a trade-off between predefined physical structure and adaptability.
- Position of SoMA: SoMA occupies a distinct design point for real-to-sim robot manipulation by explicitly representing physical states and learning interaction-aware dynamics from data.Its stated balance includes physical consistency, controllability, and fidelity to real-world observations.
- Related R2S methods: Compared with related methods, SoMA differs in problem setting and supervision while targeting robot-conditioned interaction with Gaussian-splat rendering.PhysTwin focuses on hand manipulation, while GS-Dynamics uses additional 3D tracking or control-point supervision.
A.3. Limitations and Future Directions
The evaluation is limited in scale, and future work should test SoMA on more diverse objects, materials, and manipulation behaviors. The data pipeline uses synchronized multi-view observations, robot states, and reconstructed Gaussian splats to support simulation.
- A.3. Limitations and Future Directions: Evaluation covers four object categories and unseen actions within this scope, limiting conclusions about broader scalability and robustness.Broader validation is needed across more diverse object geometries, materials, and interaction patterns.
- A.3. Limitations and Future Directions: Future studies should use larger, more diverse datasets spanning hundreds of object instances and a wider range of manipulation behaviors.Such studies would help characterize the limits of data-driven interaction-aware simulation and whether generalization improves with data diversity.
- Dataset Collection & Preprocessing: The collection setup synchronously records multi-view RGB images and robot joint states at 30 FPS.One camera is mounted on the robot end-effector for robot-conditioned mapping, while two tabletop cameras provide additional views.
- Dataset Collection & Preprocessing: Object and robot masks provide visual supervision and identify regions occluded by the manipulator during interaction.GroundingDINO and Grounded-SAM2 generate the masks from text prompts.
- Dataset Collection & Preprocessing: Robot joint states are converted into end-effector pose and gripper parameters, then mapped into a unified simulation coordinate system.The mapping uses the robot-conditioned real-to-sim transformation.
- Dataset Collection & Preprocessing: The reconstruction pipeline estimates camera poses and an initial point cloud, then optimizes and converts it into a 3D Gaussian Splatting representation.The robot-to-simulation mapping uses scale and rigid transformation parameters derived from reference-object dimensions and mounted-camera poses.
B.2. Model Implementation
SoMA uses a hierarchical Gaussian-splat simulator with multi-resolution temporal training and image/depth evaluation focused on object regions. The implementation includes fixed clustering, graph-based message passing, and benchmark comparisons for cloth manipulation.
- Model Implementation: Training uses four NVIDIA H200 GPUs, while inference uses one GPU; first-stage training takes approximately 24 hours.The reported implementation uses separate multi-GPU training and single-GPU inference configurations.
- Model Implementation: Gaussian splats are organized into a fixed three-level hierarchy with approximately [n, n/2, 2] nodes and 30 control points.The hierarchy is constructed once using distance-based clustering and remains unchanged during training and inference.
- Model Implementation: The backbone uses shared-parameter mesh graph networks for multi-scale message passing and predicts per-splat velocity, deformation, and rotation updates.Each graph network uses 128-dimensional embeddings and 16 encoder layers.
- Model Implementation: Multi-resolution temporal training first uses subsampled sequences, then fine-tunes on full-resolution sequences for stable long-horizon rollouts.The T-shirt folding dataset uses k = 5 during the initial stage, and performance is relatively insensitive to k within a reasonable range.
- Model Implementation: Image super-resolution preprocessing improves Gaussian-splat reconstruction by sharpening object boundaries and masks.Selected frames are processed with 4× super-resolution before segmentation and reconstruction.
- Model Implementation: Image metrics are computed on visible object regions, while depth metrics use valid object regions and tabletop depth for masked-out pixels.The evaluation reports PSNR, SSIM, LPIPS, Absolute Relative Error, and RMSE.
- Model Implementation: SoMA maintains consistent multi-view accuracy and plausible dynamics, and more accurately simulates T-shirt folding than PhysTwin.The comparisons cover main and side views as well as folding motion against ground truth.
C.3. Results on the Datasets of PhysTwin
On PhysTwin datasets, SoMA is evaluated using tracked hand trajectories as control signals with a closed-gripper assumption. It produces visually and physically consistent dynamics and outperforms baselines across evaluated metrics.
- Results on the Datasets of PhysTwin: The PhysTwin datasets provide 3D-tracked hand trajectories, which SoMA treats as robot end-effector motions with a closed gripper.This setting evaluates SoMA beyond explicit robot-action-driven control.
- Results on the Datasets of PhysTwin: SoMA produces dynamics visually and physically consistent with ground-truth sequences under externally provided motion trajectories.The evaluation uses the PhysTwin dataset setting rather than explicit robot joint actions.
- Results on the Datasets of PhysTwin: SoMA consistently outperforms baseline methods across all evaluated metrics on the PhysTwin datasets.Qualitative results are reported in Fig. 4(c), with quantitative results reported in Table 8.