PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboSimStudio: An Interactive Toolkit for Human and Agent Rapid Real2Sim of Robot Manipulation Tasks

Kaiyuan Chen, Shuangyu Xie, Uksang Yoo, Xinyue Zhu, Cael Magner, Alvin Zhu, Daehwa Kim, Eric Heiden, Erwin Coumans, Miles Macklin, Spencer Huang, Ken Goldberg

arXiv:2610.18835v2cs.ROcs.AIcs.GRcs.LG

TL;DR

Robot manipulation simulations require physical models that match contact, dynamics, and deformation, not just reconstructed appearance. RoboSimStudio makes physics choices editable and supports human and agent refinement; in a sponge-wiping pilot, simulation feedback improved the policy before real deployment, while a real-world failure prompted a simulation revision.

  • Problem

    Real2Sim must choose and refine physics, contact, and parameter settings to match observed manipulation behavior, and visual realism alone does not ensure physical accuracy for contact-rich or deformable interactions.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    RoboSimStudio couples solvers within Newton and external engines and exposes editable physics choices with visual and spatial-temporal feedback for human and agent refinement.

    Source: §1 · p. 2, §3 · pp. 4–5

  • Results

    RoboSimStudio was demonstrated on 18 contact-rich tasks, and in the sponge-wiping pilot GaP improved simulated wiped area from 0% to 100% over 14 iterations before real-robot deployment.

    Source: §4.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Real-world feedback exposed a fixed-support mismatch, and the agent revised the simulation to reproduce bar motion and tipping under contact.

    Source: §4.3 · p. 8

    The agentic study is a pilot, transfer was evaluated on one manipulation task, and simulated-real agreement was assessed primarily qualitatively.

    Source: §5 · p. 9

Abstract

from arXiv · show

It has been difficult to accurately model the physics of the interaction between a robot and the objects it manipulates, including contact mechanics, coupled contact dynamics, deformations, stiffness and friction. We present RoboSimStudio (RSS), an interactive toolkit that makes choices of solver, contact model, and physical parameters explicit and editable, allowing both humans and coding agents to quickly create and iteratively refine simulations based on observed physical behavior. We implement RSS with the open Newton physics simulation system and describe experiments with human-guided real2sim using (1) visual evaluation, (2) force-feedback and (3) agentic Real2Sim2Real. Code, video and additional information can be found on the project website: https://robosimstudio.github.io

1 Introduction

Robot manipulation Real2Sim requires choosing and refining physics, contact, and material models to match observed behavior. RoboSimStudio makes these choices editable for both humans and coding agents and demonstrates visual, force-feedback, and agentic workflows.

1 Introduction

  • Accurate Real2Sim requires iteratively selecting physics models, contact models, and physical parameters to match observed robot-manipulation behavior.The interaction includes robot kinematics, coupled contact dynamics, and deformation.Source: §1 · p. 2
  • Humans and coding agents share a Real2Sim loop of modifying a scene, executing interactions, observing discrepancies, and refining the model.Humans use direct manipulation and visual or force feedback; agents inspect simulator states and revise scene configurations and robot behavior.Source: §1 · p. 2
  • RoboSimStudio exposes solver, contact-model, and physical-parameter choices for interactive adjustment by humans and coding agents.Simulator state and spatial-temporal text and visual feedback let agents observe the consequences of changes and refine simulations.Source: §1 · p. 2
  • Evaluation includes 18 contact-rich simulations, force-feedback interaction, and a pilot agentic Real2Sim2Real sponge-wiping task.Coding agents built simulations from smartphone video, improved a policy with simulation feedback, and used a real-world failure to revise physical constraints.Source: §1 · p. 2

2 Related Work

Prior simulators offer increasingly varied physics capabilities, but constructing manipulation environments still involves selecting solvers, contact models, and parameters. RoboSimStudio focuses on editable physical behavior and feedback-driven refinement, especially where visual realism does not ensure physical accuracy.

2 Related Work

  • The related-work comparison table covers physics capabilities, interactive interfaces, and Real2Sim workflows, distinguishing built-in, partial, and custom support.The caption defines ✓ as built-in support, ∼ as partial support, and “Custom” as requiring additional implementation.Source: Table 1 · p. 3
  • Simulation frameworks increasingly support rigid, articulated, and deformable-body physics, yet environment construction still requires solver selection, contact configuration, and parameter tuning.RoboSimStudio builds on Newton and supports co-simulation with external engines for interactive refinement.Source: §2 · pp. 2–3
  • RoboSimStudio gives coding agents access to scene configurations, physics models, and simulation feedback for agentic Real2Sim construction and refinement.This builds on coding agents’ use of iterative reasoning and acting to modify code, execute programs, and refine solutions from feedback.Source: §2 · p. 3
  • Prior Real2Sim and generative approaches reduce scene-construction or policy-development effort, but often focus on specific reconstruction or generation pipelines.Visually plausible 3D reconstructions do not guarantee physical accuracy for contact-rich interactions or deformable objects.Source: §2 · pp. 3–4

3 RoboSimStudio

RoboSimStudio composes robots, objects, and multiple physics solvers in an editable scene. Its human and agent interfaces support direct physical probing, parameter tuning, and iterative simulation refinement.

3 RoboSimStudio

  • Scenes combine robots, objects, and physics while keeping solver, contact-model, and physical-parameter choices explicit and accessible through code, a human studio, or an agent interface.Newton and external engines exchange contact and force information to support co-simulation.Source: §3 · p. 4
  • A common declarative robot interface supports Franka Panda, UR5e, and single- and bimanual YAM, with kinematics, motion generation, and control interfaces.Robots can be commanded in joint or end-effector action spaces, while privileged object state is separated from robot observations.Source: §3 · p. 4
  • Different object types can use different solvers, coupled through contact forces or through poses and contact wrenches with external engines.By default, articulated robots use MuJoCo-Warp, cables and cloth use VBD, and soft bodies can use external engines; Figure 3 shows five tasks with different solver and coupling choices.Source: §3 · pp. 4–5, Figure 3 · p. 5
  • The browser studio lets users inspect and manipulate objects, control robot arms, and change physical parameters during simulation when supported.Figure 4 illustrates the same towel fold under four bending and stretch stiffness settings; parameters requiring a scene rebuild are marked.Source: §3 · p. 5, Figure 4 · p. 5
  • Agents iteratively modify scenes, execute robot motion primitives, and use spatial-temporal feedback on errors, object motion, contact, grasping, slipping, and task progress.Humans can also adjust parameters in the shared studio and guide an agent asynchronously without interrupting execution.Source: §3 · pp. 5–6

4 Case Studies

The case studies demonstrate visual and force-feedback Real2Sim workflows and a pilot agentic Real2Sim2Real task. Simulation feedback improved wiping performance, while a real-world support-displacement failure motivated a revised physical constraint.

4.1 Human Real2Sim with Visual Feedback

  • Visual refinement produced 18 manipulation simulations spanning rigid contact, articulated motion, cables, cloth, and soft-body deformation.Examples include sub-millimeter USB-C insertion, leaf pressing and recovery, and polishing modeled using contact forces and Preston’s equation [57].Source: §4.1 · pp. 6–7

4.2 Human Real2Sim with Force Feedback

  • Force-feedback teleoperation renders simulated contact through a low-cost Gello leader arm, helping operators perceive and adjust tool-hanging interactions.A speed-dependent deadband and pause handling avoid rendering ordinary simulation lag as contact.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3 Agentic Real2Sim2Real

  • Agents built and verified a multi-physics sponge-wiping simulation from smartphone video in about an hour, with one construction completed in 56 minutes.After a real rollout displaced a support fixed in the original simulation, the agent replaced the fixed constraint with a free rigid body; the revised model captured bar motion and tipping under suitable contact.Source: §4.3 · pp. 7–8
  • Across 14 iterations, GaP improved simulated wiped area from 0% to 100%, and the improved policy completed the real-world sponge-wiping task.The pilot simulation coupled MuJoCo-Warp within Newton with SuperDex for sponge deformation and evaluated 30 bar-and-sponge layouts.Source: §4.3 · p. 7, Figure 7 · p. 8

5 Conclusion

RoboSimStudio makes solver, contact-model, and physical-parameter choices editable for human- and agent-guided robot-manipulation simulation. The paper demonstrates 18 contact-rich tasks and a pilot in which an agent-built sponge-wiping simulation supported policy deployment, while noting limits in scope and evaluation.

5 Conclusion

  • A simulation-feedback-improved Graph-as-Policy was deployed on a real robot, with real-world failure feedback guiding further simulation refinement.Agents constructed the bimanual sponge-wiping simulation from smartphone video in approximately one hour.Source: §5 · p. 9
  • RoboSimStudio couples solvers within Newton and external physics engines, with interactive interfaces for humans and coding agents to construct and refine simulations.The toolkit makes solver, contact-model, and physical-parameter choices explicit and editable.Source: §5 · p. 9
  • The toolkit was demonstrated on 18 contact-rich tasks involving rigid bodies, articulated objects, cables, cloth, and soft-body deformation.Source: §5 · p. 9
  • The system remains limited by supported physics solvers and external-engine overhead, while sim-to-real transfer was evaluated on one task with primarily qualitative behavior comparisons.Future work calls for quantitative evaluation across diverse tasks.Source: §5 · p. 9

Related papers

Loading 2610.18835v2…