PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, Pulkit Agrawal

arXiv:2403.03949v3cs.ROcs.AIcs.LG

TL;DR

Robust manipulation policies require either extensive demonstrations or unsafe, impractical real-world exploration. RialTo constructs digital twins from small amounts of real-world data, transfers demonstrations into simulation for RL fine-tuning, and reports a 67% improvement in average success rate across eight tasks.

  • Problem

    Imitation learning needs substantial human supervision for robustness, whereas real-world reinforcement learning requires impractical unsafe data collection.

  • Method

    RialTo constructs realistic simulation environments on the fly and uses inverse distillation to transfer real-world demonstrations into simulation for reinforcement-learning robustification.

  • Results

    67% improvement in average success rate over baselines is reported across eight tasks with varying object poses, visual distractors, and physical perturbations.

  • Takeaways & Limitations

    RialTo enables non-expert users to program manipulation controllers that transfer to real-world scenes with enhanced robustness and generalization using minimal human effort.

  • Takeaways & Limitations

    RialTo currently depends on accurate depth sensing and is mainly limited to articulated rigid bodies and tasks that can be easily simulated.

Abstract

from arXiv · show

Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/

I. INTRODUCTION

RialTo targets robust manipulation under local scene and object variations by combining real-world imitation learning with reinforcement learning in quickly constructed simulations. Its pipeline uses digital twins, transferred demonstrations, and policy distillation to reduce human effort and unsafe real-world data collection while improving robustness.

  • Robust deployment requires handling scene configurations, object slips, physical disturbances, clutter, lighting changes, and other visual distractors without task-specific assumptions.
  • Imitation learning policies often lack robustness and recovery behaviors unless large numbers of demonstrations intentionally cover disturbances and mistakes.
  • RL can discover robust recovery behaviors autonomously, but direct real-world training is prohibitively slow, unsafe, and difficult to reset and reward.
  • RialTo constructs realistic simulation analogs on the fly and uses them for robust policy learning with limited human effort.
  • Inverse distillation transfers real-world demonstrations into reconstructed simulations, bootstrapping reinforcement learning and biasing policies toward realistic behavior.
  • 67% improvement in average success rate over baselines is reported across eight tasks involving varying object poses, visual distractors, and physical perturbations.

II. RELATED WORK

RialTo combines reconstructed physical scenes with simulation-based reinforcement learning and real-world demonstrations, extending prior work beyond visual-only transfer and simplifying scene construction. The approach uses separated geometry, articulations, and physical parameters to support policy training in reconstructed environments.

  • Prior behavior cloning methods can perform well on modest datasets but generally degrade under variations in scene layout and appearance.
  • Prior simulation-based RL work commonly relies on domain randomization, system identification, or improved simulator visuals to reduce the sim-to-real gap.
  • Earlier real-to-sim pipelines created digital twins using 3D reconstruction or inverse graphics, while RialTo uses these approaches as compatible scene-transfer components.
  • Visual-only reconstructed scenes cannot support physical interaction or adaptation to environmental changes beyond visual distractions.
  • Real-to-sim transfer of scenes: RialTo reconstructs textured geometry, separates objects into bodies or links, and specifies kinematic relations and physical parameters for simulation.
  • Real-to-sim transfer of scenes: A graphical interface lets users separate meshes and add joints, while the system uses uniform default mass and friction values to compensate for uncertain dynamics.

C. Robustifying Real-World Imitation Learning Policies in Simulation

RialTo constructs simulation environments from scanned real-world scenes, then transfers real demonstrations into simulation to initialize robust policy learning. Its inverse-distillation procedure recovers privileged simulation-state demonstrations from perception-based policies trained on real data.

  • Scene construction: RialTo’s GUI converts scanned environments into organized USD assets that can be imported directly into the simulator.Users can cut meshes, specify joints, and organize reconstructed scenes after scanning with tools such as NeRFStudio, ARCode, or Polycam.
  • Policy initialization: The method fine-tunes simulation policies with reinforcement learning rather than training them from scratch, reducing manual engineering requirements.The policy is initialized from a small number of expert demonstrations collected in the real world.
  • Policy initialization: The resulting simulation demonstrations bootstrap robust state-based reinforcement-learning fine-tuning while retaining behaviors initiated from real-world data.The approach is designed to improve beyond the original real-world imitation policy.
  • Inverse distillation: Inverse distillation converts real-world observation-action demonstrations into simulation trajectories paired with privileged low-level state information.This addresses the absence of Lagrangian state measurements in real-world demonstrations.
  • Inverse distillation: A perception-based policy trained on Dreal is executed on simulated sensor observations to collect successful privileged demonstrations Dsim.The procedure naturally pairs simulated perceptual observations with low-level Lagrangian state information.

2) Reinforcement Learning Fine-tuning in Simulation:

RialTo trains a privileged state-based policy in simulation with sparse-reward reinforcement learning augmented by demonstrations, then distills it into a sensory student co-trained on real data. This combination addresses exploration, transfer, and the simulation-to-reality perceptual gap.

  • Reinforcement learning fine-tuning: RialTo uses sparse goal-detection rewards and an imitation-learning loss within PPO to fine-tune policies from privileged simulation-state demonstrations.The demonstrations help address scalable exploration without extensive reward engineering.
  • Reinforcement learning fine-tuning: The imitation term biases simulation policies toward physically plausible, safe solutions while training robustness across randomized robot, object, and goal poses.The resulting policy operates from Lagrangian state across varied configurations and environmental conditions.
  • Teacher-student distillation: Because privileged state is unavailable during deployment, RialTo distills the simulation teacher into a policy operating directly on sensory observations.The student policy is trained using teacher-student distillation with interactive DAgger labeling.
  • Teacher-student distillation: Co-training the distillation objective on simulation data and original real-world demonstrations bridges the perceptual gap and improves real-world generalization.The paper reports that this significantly increases resulting real-world success rate.

IV. EXPERIMENTAL EVALUATION

RialTo is evaluated across eight manipulation tasks under randomized poses, visual distractors, and physical disturbances. Across tasks, it maintains substantially higher robustness than behavior cloning, including when behavior cloning uses more demonstrations.

  • Evaluation setup: The evaluation spans eight tasks involving free-object manipulation, articulated-object interaction, and less-controlled real-world scenes.Tasks include placing books, plates, and mugs, manipulating drawers and cabinets, opening a toaster, and putting a cup in the trash.
  • Evaluation setup: The experiments test randomized object or robot poses, cluttered visual distractors, and physical disturbances applied during rollouts.Disturbances include moving manipulated objects or targets, closing articulated objects, and moving the robot base when possible.
  • Baselines and ablations: With 15 demonstrations, RialTo is compared against imitation learning, while additional ablations examine real-data co-training, demonstration source, scene targeting, distractors, and demonstrations.The study also compares against behavior cloning using 50 demonstrations.
  • Robustness results: 91% average success is achieved for randomized object poses, 77% with distractors, and 75% with disturbances across tasks.These results are reported for RialTo in Figure 5.
  • Robustness results: Behavior cloning averages 25% success with randomized poses, 11% with distractors, and 5% with disturbances.The presence of distractors and disturbances severely reduces pure imitation-learning performance.
  • Data and effort: Collecting 50 behavior-cloning demonstrations takes 1 hour and 45 minutes, compared with 30 minutes for 15 RialTo demonstrations plus 15 minutes of active environment building.The paper reports that RialTo achieves approximately 2.5 times higher success despite fewer demonstrations and less time.

B. Impact of Co-Training with Real-World Data

Co-training with real-world demonstrations improves RialTo's real-world performance, especially on harder manipulation tasks, while target-scene reconstruction outperforms training on randomized synthetic assets.

  • 3.5x and 2x success rate increases occur for book on shelf and plate on rack, respectively, when co-training uses 15 real-world demonstrations.
  • Co-training with real-world demonstrations outperforms simulated-only or no-real-data training on harder tasks and matches them on easier tasks.
  • Real-world co-training improves grasping behavior by leaving more space before grasping and helps reduce visual and physical sim-to-real gaps.
  • 90% versus 10% real-world success is achieved by training on the target drawer reconstruction rather than four randomized Objaverse drawers.
  • Policies initialized from real-world or simulated demonstrations perform similarly when both use real-world co-training during teacher-student distillation.

D. Scaling RialTo to In-the-Wild Environments

RialTo extends to uncontrolled, cluttered, and in-the-wild scenes while improving robustness over imitation learning. Its training benefits from distractors, demonstration bootstrapping, and compact-state fine-tuning.

  • D. Scaling RialTo to In-the-Wild Environments: 57% average improvement over standard imitation learning is obtained across three tasks in uncontrolled and in-the-wild kitchen scenes.
  • Visual distractors: Adding visual distractors during training increases real-world success on the challenging mug-on-shelf task.
  • Demonstration bootstrapping: RL from scratch fails on three of five tasks and performs much worse on the other two, while demonstrations guide exploration and discourage exploiting simulator inaccuracies.
  • Compact-state fine-tuning: 96% success after 12 hours with compact-state RL contrasts with 1% after 35 hours using vision-only RL for drawer opening.
  • Scene construction: The GUI study reports 14 minutes 40 seconds of average active time and 25 minutes 12 seconds of total time to create a scene.

VII. LIMITATIONS AND CONCLUSION

RialTo combines real-world imitation learning with reinforcement learning in digital twins to obtain robust real-world policies, but its applicability depends on sensing, simulation, task, and training-time constraints.

  • Limitations: Accurate depth sensors are required, but they can struggle with thin, transparent, and reflective objects.
  • Limitations: The approach is limited to tasks that are easily simulated and objects that can be converted into digital assets, currently primarily articulated rigid bodies.
  • Limitations: RialTo currently requires around 2 days of end-to-end wall-clock training per task, making continual learning infeasible.
  • Conclusion: RialTo achieves robustness to environmental variations and disturbances by combining real-world imitation learning with large-scale RL in on-the-fly digital twins.
  • Method: The task success function selects successful inverse-distillation trajectories and supplies the sparse reward for RL fine-tuning.

A. Simulation details

RialTo uses Isaac Sim and reconstructed scene assets to train state-based and point-cloud policies, then transfers learned behavior through teacher-student distillation with simulated and real trajectories.

  • Simulation platform: Isaac Sim is used to simulate each task, with Orbit inspiring the reinforcement-learning and robot-learning implementation.
  • Simulation parameters: The GUI uses default collision-mesh settings of 64 hull vertices and 32 convex hulls for most objects, with task-specific exceptions.
  • State-based policy: The privileged state-based policy is a two-layer 256-unit MLP that outputs a 14-action categorical distribution for discrete end-effector control.
  • Point-cloud policy: Point-cloud policies encode observations into 128-dimensional embeddings and output categorical distributions over the same discrete actions.
  • Distillation: Teacher-student distillation mixes full-cloud, real-camera, distractor, and real-world trajectories, followed by DAgger relabeling with the simulator policy.

C. Simulated Assets Baseline Details

This section describes baseline implementations for multi-task simulation and imitation learning, including distractor augmentation and the hardware and point-cloud configurations used in evaluation.

  • The multi-task simulation baseline randomly selects a drawer each episode and adds the selected drawer index to the state observation.
  • The imitation learning baseline uses 15 real-world keyboard demonstrations with point-cloud preprocessing, arm-mesh completion, jitter, dropout, and translation randomization.
  • Adding point clouds from different object meshes did not improve distractor robustness, possibly because the added meshes produced no occlusions.
  • Experiments used two Panda Franka arms, with the fixed-table setup supporting shelf, rack, cabinet, and drawer tasks and the mobile-table setup supporting the toaster task.
  • Vision policies used Intel RealSense D455 and D435 depth cameras, while point-cloud generation and teacher-student distillation followed dedicated parameter configurations.

XI. FURTHER ANALYSIS

The further analysis examines computational costs, failure modes, demonstration requirements, synthetic assets, and multi-task extensions of RialTo.

  • A. RL from vision: Vision-based RL is substantially slower because it uses smaller GPU batches and point-cloud rendering is approximately 10x slower than compact-state processing.The section states that vision-policy batches are 100x smaller on the same GPU and that rendering point clouds adds roughly a 10x slowdown.
  • A policy trained from scratch without demonstrations can exploit simulation inaccuracies, producing behavior that achieves 62% simulated toaster-opening accuracy but does not transfer to reality.
  • C. RL from different amounts of real-world data: For book placement, PPO remains at 0% success below 15 real-world demonstrations because the policy does not transfer to simulation for inverse distillation.
  • Adding synthetic assets to standard RialTo produced no clear improvement on the real-world drawer-opening task.
  • E. RialTo Multi-Task: The multi-task procedure trains separate policies, distills their trajectories into a task-conditioned policy, and applies sequential DAgger; drawer performance matches single-task RialTo at 90%.On the mug-on-shelf task, performance decreases from 100% single-task success to 80% multi-task success, while remaining above the imitation learning baseline.

F. Sim-to-real gap

This section analyzes sim-to-real performance and the GUI-based process for constructing articulated scenes from scans.

  • F. Sim-to-real gap: The final point-cloud policy generally shows no sim-to-real performance gap, and simulation can be harder than the real world for some tasks.For the mug-on-shelf task, simulation performance is reported as worse than real-world performance.
  • XIII. GUI USER STUDY: The GUI pipeline combines scene scanning, mesh processing, object separation, articulation, and placement to create usable articulated simulation scenes.
  • XIII. GUI USER STUDY: The user study measured scene and object scanning, processing, uploading, cutting, and articulation-related work across six participants.The study used a GUI and recorded the active time required for constructing articulated scenes.
  • XIII. GUI USER STUDY: The GUI supports independent manipulation of scanned objects by allowing users to cut objects from connected scene meshes.
  • XIII. GUI USER STUDY: Faster users created smaller files and completed processing more quickly, but their scans had less precise background details than the most accurate scan.

A. Scaling laws of the RialTo GUI

The GUI’s scene-construction time scales with scene complexity, specifically the numbers of objects, cut objects, and joints, while policy training adds substantial computational time.

  • The time relation combines scene scanning, object scanning, object cutting, and joint-addition costs according to the relevant object, cut-object, and joint counts.
  • Total active scene-construction time increases linearly with the number of objects and joints in the scene.
  • Average measured times were 4:50 per object scan, 3:14 per scene scan, 2:54 per added joint, and 3:40 per cut object.
  • Training one task averages 2 days and 3 hours, including real-world demonstration learning, RL fine-tuning, and teacher-student distillation.The reported stages average 7 hours, 20 hours, and 24 hours respectively.
Loading 2403.03949v3…