SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds

Yunsong Zhou; Hangxu Liu; Xuekun Jiang; Xing Shen; Yuanzhen Zhou; Hui Wang; Baole Fang; Yang Tian; Mulin Yu; Qiaojun Yu; Li Ma; Hengjie Li; Hanqing Wang; Jia Zeng; Jiangmiao Pang

Source-linked AI summary

SIM1: Physics-Aligned Simulator as Zero-Shot Data Scaler in Deformable Worlds

Yunsong Zhou, Hangxu Liu, Xuekun Jiang, Xing Shen, Yuanzhen Zhou, Hui Wang, Baole Fang, Yang Tian, Mulin Yu, Qiaojun Yu, Li Ma, Hengjie Li, Hanqing Wang, Jia Zeng, Jiangmiao Pang

arXiv:2604.08544v2Apr 9, 2026cs.ROcs.AIcs.CV

TL;DR

Deformable manipulation is data-intensive, while conventional simulation is poorly aligned with real geometry, soft dynamics, and cloth interaction. SIM1 builds a physics-aligned real-to-sim-to-real pipeline that digitizes scenes, calibrates deformable dynamics, and synthesizes filtered trajectories. Policies trained purely on synthetic data achieve comparable performance to real-data baselines and demonstrate strong zero-shot transfer and generalization.

Problem
Deformable manipulation requires broad data coverage, but existing simulation pipelines are poorly aligned with real scenes, soft dynamics, and cloth-suitable behaviors.
Method
SIM1 aligns geometry, dynamics, and motion through metric-accurate scene digitization, stabilized soft-body simulation with behavior calibration, and diffusion-based trajectory synthesis with filtering.
Results
90% and 76% zero-shot success rates are achieved on 𝜋0.5 and 𝜋0, with generalization gains of +50% and +56% over real-data baselines.
Takeaways & Limitations
High-fidelity simulation is a viable scalable source of supervision that complements real-data collection for deformable manipulation.
Takeaways & Limitations
Material calibration requires expert-guided parameter tuning for each asset, constraining full automation across arbitrary cloth types.

Abstract

from arXiv · show

Robotic manipulation with deformable objects represents a data-intensive regime in embodied learning, where shape, contact, and topology co-evolve in ways that far exceed the variability of rigids. Although simulation promises relief from the cost of real-world data acquisition, prevailing sim-to-real pipelines remain rooted in rigid-body abstractions, producing mismatched geometry, fragile soft dynamics, and motion primitives poorly suited for cloth interaction. We posit that simulation fails not for being synthetic, but for being ungrounded. To address this, we introduce SIM1, a physics-aligned real-to-sim-to-real data engine that grounds simulation in the physical world. Given limited demonstrations, the system digitizes scenes into metric-consistent twins, calibrates deformable dynamics through elastic modeling, and expands behaviors via diffusion-based trajectory generation with quality filtering. This pipeline transforms sparse observations into scaled synthetic supervision with near-demonstration fidelity. Experiments show that policies trained on purely synthetic data achieve parity with real-data baselines at a 1:15 equivalence ratio, while delivering 90% zero-shot success and 50% generalization gains in real-world deployment. These results validate physics-aligned simulation as scalable supervision for deformable manipulation and a practical pathway for data-efficient policy learning.

1. Introduction

Deformable manipulation needs substantially more varied data than rigid-object tasks, yet real-world collection is costly and conventional simulation is poorly grounded. SIM1 addresses this gap through physics-aligned real-to-sim-to-real modeling and reports strong zero-shot transfer from synthetic data.

Motivation: Deformable manipulation demands broader state and visual coverage because geometry and contact-rich dynamics evolve continuously.Real-world data acquisition at scale remains prohibitively expensive.
Motivation: Conventional sim-to-real pipelines break down for deformable manipulation because scenes, soft dynamics, and motion primitives are mismatched to reality.Synthetic data therefore mainly supports pre-training, with reliable performance still depending on real-world post-training.
SIM1: SIM1 uses alignment-first real-to-sim-to-real modeling to ground simulation in physical geometry, dynamics, and manipulation behavior.Its stages include metric-accurate scene reconstruction, stabilized soft-body modeling, and diffusion-based trajectory synthesis with filtering and randomization.
Results: 90% and 76% zero-shot success rates are achieved on 𝜋0.5 and 𝜋0, respectively, with generalization gains of +50% and +56% over real-data baselines.The experiments also report that 15 synthetic samples provide training value comparable to one real demonstration.

2. Related Work

Prior work shows that scaling real-robot data and synthetic data benefits rigid manipulation, but deformable manipulation remains underrepresented and more data-hungry. SIM1 targets this gap while addressing limitations in alignment and deformable simulation solvers.

Data scaling: Large-scale manipulation datasets expand coverage for rigid scenarios, whereas deformable manipulation remains underrepresented.Soft-body dynamics create continuous shape variation and contact-rich behaviors that are more data-hungry than rigid-object tasks.
Simulation-to-real: Simulation-based generation has used transformation, real-to-sim alignment, and physical-twin reconstruction, with strong results mainly in rigid manipulation.Emerging capabilities extend to humanoid and deformable tasks.
Scene digitization: Existing deformable-object digitization approaches use MPM, spring-mass models, and multi-engine platforms, but solver accuracy remains limited.VBD suffers unrealistic stretching, while PBD and FEM trade off accuracy against efficiency.
SIM1: SIM1’s solver and calibration infrastructure support online rigid–soft coupling with stable deformation dynamics for high-fidelity data generation.This extends solver improvements into a broader manipulation pipeline.

3. SIM1 Framework

SIM1 constructs metric-accurate scenes, stabilizes deformable physics, calibrates simulated behavior against real executions, and synthesizes manipulation trajectories for scalable training. Its framework targets the geometry, dynamics, and motion mismatches that hinder deformable sim-to-real transfer.

Framework: SIM1 converts real scenes into metric-accurate digital assets and uses aligned simulation to produce synthetic manipulation data.The pipeline proceeds from scene digitization to calibrated dynamics and diffusion-based trajectory synthesis.
Scene digitization: Metric-accurate geometry is treated as a prerequisite because discrepancies in shape, scale, or configuration can propagate into dynamic and contact errors.Garments are scanned and processed into geometric replicas, while robots and environments are imported and dimensionally aligned.
Physics simulation: Existing solvers inadequately support rigid–soft coupling, causing delayed strain updates, excessive stretching, unstable contact, and real-world misalignment.SIM1 addresses this with deformation-stable dynamics and calibration under identical control inputs.
Physics simulation: The deformation-stable solver activates virtual elastic constraints when mesh edges exceed a stretch threshold, injecting corrective forces during optimization.The constraint is defined using current vertex positions, rest length, and a maximum stretch ratio.
Physics simulation: Penalty stiffness and constraint variables are updated across Newton iterations to enforce strain progressively and prevent explosive stretching.The maximum stiffness and ramping rate govern this correction process.
Motion synthesis: Structured trajectory decomposition separates interaction from motion, reuses demonstrated grasp configurations, and supports diverse diffusion-based synthesis.This design is intended to preserve interaction fidelity for state-dependent deformable contacts.

4. Experiments

The experiments test whether physics-grounded simulation can replace or scale real data for deformable manipulation. Across controlled comparisons, synthetic data matches real-data supervision under equal budgets, improves domain-shift robustness, and scales more efficiently, while solver stability is important for policy generalization.

Experimental Setup: The experiments evaluate S2R transfer, cross-domain generalization, and data-scaling efficiency under purely simulation-trained policies.The study compares real demonstrations, collected simulation data, and simulator-generated data across in-domain and out-of-domain settings.
Simulation versus Real Data: 97% real-data success versus 87% simulation-data success under the representative π0.5 setting shows comparable performance under equal budgets.The reported gap is 10 percentage points, supporting substantial supervision fidelity from physics-aligned simulation.
Out-of-Domain Robustness: 50%, 13%, and 47% gains over real-data baselines occur under spatial shifts, texture variation, and lighting perturbations, respectively.These results indicate broader variation coverage and improved robustness beyond the training distribution.
Pretraining Confound Analysis: 76% synthetic-training success versus 0% for the real-data baseline under de novo initialization shows task acquisition without pretrained manipulation knowledge.The comparison attributes task success to generated synthetic supervision rather than preexisting manipulation priors.
Scaling Analysis: One real demonstration provides comparable benefit to approximately 15 synthetic samples for π0.5 under in-domain evaluation.In representative texture generalization, the equivalence shifts to roughly 5 synthetic samples per real sample.
Scaling Analysis: Synthetic data is weak in extremely low-data regimes but eventually surpasses real-data training as simulation volume increases.Real-data gains saturate with limited diversity, whereas simulation performance grows rapidly with additional samples.
Real-World Deployment: A synthetic-only policy reaches 70% success on an unseen polo shirt, compared with 20% for the real-data baseline.The deployment uses no real demonstrations and tests different material, texture, and geometry on a physical robot.
Ablation Study: A deformation-stable solver improves generated-data policy performance from 47% in-domain with diffusion generation to 67% in-domain and 76% average success.The ablation identifies solver stability as essential for translating scalable simulation data into policies that generalize physically.

5. Conclusion

SIM1 transforms limited real demonstrations into scalable synthetic data for deformable manipulation. The resulting policies achieve zero-shot sim-to-real transfer and support data-efficient learning while complementing real-robot data.

SIM1 transforms limited real demonstrations into scalable synthetic data for deformable manipulation.
SIM1 jointly addresses geometric accuracy, dynamic fidelity, and motion synthesis to achieve zero-shot sim-to-real transfer in garment folding.
Policies trained purely on synthetic data achieve comparable performance to real-data baselines, with consistent improvement as data scales.
Expert-guided material calibration remains necessary for each asset, constraining full automation across arbitrary cloth types.

B. SIM1 Implementation Details

SIM1 models cloth dynamics with an AVBD-based solver that combines elastic, bending, strain, and collision-related forces. Newton-type iterations update mesh vertices while enforcing physical constraints.

SIM1 uses the Augmented Vertex Block Descent solver to simulate cloth dynamics.
Newton-type iterations compute vertex displacement updates by minimizing total energy while satisfying stretch, bending, and collision-avoidance constraints.
The StVK hyperelastic model assigns higher energy to triangles that stretch or shear, producing restoring forces toward the undeformed configuration.
Dihedral-angle bending models assign restoring forces to deviations between current and preferred edge-sharing triangle angles.
Elastic, bending, constraint, and external forces are combined to determine vertex displacement updates during solver iterations.
Each vertex update is obtained from a local 3 × 3 linear system derived from Newton optimization.

B.5. Penetration Avoidance

SIM1 prevents unsafe cloth updates with a geometric safety filter and integrates synchronized robot–simulation infrastructure. Calibration and GPU execution support stable, physically consistent data collection.

B.5. Penetration Avoidance: The geometric safety filter prevents interpenetration with obstacles or self-collision.
B.5. Penetration Avoidance: A safe displacement bound uses collision radius and minimum primitive distances to clip raw Newton updates.
B.5. Penetration Avoidance: Clipped updates prevent overshooting the safety margin, reducing tunneling and numerical instability.
B.6. Position Updation: The clipped displacement is applied to the current iteration’s starting vertex position to produce the final updated position.
B.7. Simulation Infrastructure: The infrastructure synchronizes physical teleoperation with GPU-accelerated simulation through direct correspondence between robot joints and simulated actuators.
B.7. Simulation Infrastructure: Cloth properties are calibrated with elasticity moduli, a 5% strain limit, friction parameters, and β = 0.42 for numerical robustness.
B.7. Simulation Infrastructure: NVIDIA Warp runs rigid-body dynamics and the AVBD cloth solver entirely on the GPU at approximately 15 fps.

C. Synthetic Data Generation Algorithm

SIM1 generates synthetic manipulation trajectories by decomposing expert demonstrations into reusable interaction segments and synthesizing transitions with diffusion models.

The pipeline transforms teleoperated demonstrations into large-scale synthetic trajectories through structured decomposition and diffusion-based motion generation.
Reusable grasp-to-release interaction segments are extracted from expert demonstrations to form a library of manipulation phases.
Sampled segment sequences define high-level task skeletons for synthetic trajectory construction.
A diffusion model synthesizes feasible transition trajectories between adjacent segment endpoints, producing complete manipulation trajectories.

D.1. Additional Tasks

SIM1 generalizes beyond T-shirts to towels, shorts, and polo shirts, with synthetic-data-trained policies transferring across real-world garment tasks and conditions.

D.1. Additional Tasks: Polo-shirt folding transferred successfully despite differences in geometry, size, material, and friction from the training garments.No similar polo-shirt instance appeared in the training dataset, indicating transfer under a substantial distribution shift.
D.1. Additional Tasks: Real-world evaluations included in-domain T-shirt folding, randomized object positions, varying polo-shirt textures and lighting, shorts, and towels.The deployment results cover both in-domain and out-of-domain settings.
D.1. Additional Tasks: Synthetic demonstrations vary garments, tables, lighting, textures, and temporally sampled trajectories across T-shirts, towels, shorts, and polo shirts.The visualizations illustrate diversity and applicability to multiple garment types.

D.4. Cost-efficiency Analysis

SIM1's simulation pipeline reduces the cost of synthetic trajectory generation while increasing daily throughput relative to physical data collection.

D.4. Cost-efficiency Analysis: Physical data collection costs approximately $282 daily, yielding 104 trajectories and a unit cost of about $2.71 per trajectory.The daily estimate includes $200 for manual labor and $82 for hardware depreciation.
D.4. Cost-efficiency Analysis: 27× lower cost and 6.8× higher throughput were achieved compared with physical data collection.Simulation costs $0.10 per trajectory and generates approximately 710 trajectories daily, versus $2.71 per real trajectory and 104 trajectories daily.
D.4. Cost-efficiency Analysis: The simulation framework costs roughly $71 daily on a server with 8× NVIDIA RTX 4090 GPUs.The stated rendering time is 16.2 minutes per GPU, with an amortized cost of $0.37 per hour.

D.5. Failure Cases

SIM1's failure analysis shows that corrupted synthetic samples can cause systematic policy failures, while filtering substantially limits invalid trajectories.

D.5. Failure Cases: Low-quality or invalid generated samples can poison the training set and lead to systematic failures.The reported sensitivity makes generated-data quality a central failure boundary for policy training.
D.5. Failure Cases: Corrupted samples produce behaviors including overreaching, premature gripper closure before contact, and misaligned grasps.These failures differ from out-of-distribution failures such as missing the garment edge or mispositioning relative to the object.
D.5. Failure Cases: The synthetic-data visualizations show randomized garment, table, lighting, texture, and trajectory configurations across multiple garment types.These passages describe generated scenarios rather than failure outcomes.
D.5. Failure Cases: Over 99% filtering success was achieved for invalid trajectories using the discriminator, with simple rule-based checks offering further improvement.The filtering pipeline is intended to keep the synthetic dataset highly reliable and minimally corrupted.