PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation

Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, Pulkit Agrawal

arXiv:2403.03949v3cs.ROcs.AIcs.LG

TL;DR

Robust manipulation policies require either extensive demonstrations or unsafe, impractical real-world exploration. RialTo constructs digital twins from small amounts of real-world data, transfers demonstrations into simulation for RL fine-tuning, and reports a 67% improvement in average success rate across eight tasks.

  • Problem

    Imitation learning needs substantial human supervision for robustness, whereas real-world reinforcement learning requires impractical unsafe data collection.

    Source: §I · pp. 1–2

  • Method

    RialTo constructs realistic simulation environments on the fly and uses inverse distillation to transfer real-world demonstrations into simulation for reinforcement-learning robustification.

    Source: §I · pp. 2–3

  • Results

    67% improvement in average success rate over baselines is reported across eight tasks with varying object poses, visual distractors, and physical perturbations.

    Source: §I · p. 3

  • Takeaways & Limitations

    RialTo enables non-expert users to program manipulation controllers that transfer to real-world scenes with enhanced robustness and generalization using minimal human effort.

    Source: §VII · p. 11

    RialTo currently depends on accurate depth sensing and is mainly limited to articulated rigid bodies and tasks that can be easily simulated.

    Source: §VII · p. 11

Abstract

from arXiv · show

Imitation learning methods need significant human supervision to learn policies robust to changes in object poses, physical disturbances, and visual distractors. Reinforcement learning, on the other hand, can explore the environment autonomously to learn robust behaviors but may require impractical amounts of unsafe real-world data collection. To learn performant, robust policies without the burden of unsafe real-world data collection or extensive human supervision, we propose RialTo, a system for robustifying real-world imitation learning policies via reinforcement learning in "digital twin" simulation environments constructed on the fly from small amounts of real-world data. To enable this real-to-sim-to-real pipeline, RialTo proposes an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. We also introduce a novel "inverse distillation" procedure for bringing real-world demonstrations into simulated environments for efficient fine-tuning, with minimal human intervention and engineering required. We evaluate RialTo across a variety of robotic manipulation problems in the real world, such as robustly stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness without requiring extensive human data collection. Project website and videos at https://real-to-sim-to-real.github.io/RialTo/

I. INTRODUCTION

RialTo targets robust manipulation under local scene and object variations by combining real-world imitation learning with reinforcement learning in quickly constructed simulations. Its pipeline uses digital twins, transferred demonstrations, and policy distillation to reduce human effort and unsafe real-world data collection while improving robustness.

  • Robust deployment requires handling scene configurations, object slips, physical disturbances, clutter, lighting changes, and other visual distractors without task-specific assumptions.Source: §I · p. 1
  • Imitation learning policies often lack robustness and recovery behaviors unless large numbers of demonstrations intentionally cover disturbances and mistakes.Source: §I · p. 1
  • RL can discover robust recovery behaviors autonomously, but direct real-world training is prohibitively slow, unsafe, and difficult to reset and reward.Source: §I · p. 1
  • RialTo constructs realistic simulation analogs on the fly and uses them for robust policy learning with limited human effort.Source: §I · p. 3
  • Inverse distillation transfers real-world demonstrations into reconstructed simulations, bootstrapping reinforcement learning and biasing policies toward realistic behavior.Source: §I · p. 3
  • 67% improvement in average success rate over baselines is reported across eight tasks involving varying object poses, visual distractors, and physical perturbations.Source: §I · p. 3

II. RELATED WORK

RialTo combines reconstructed physical scenes with simulation-based reinforcement learning and real-world demonstrations, extending prior work beyond visual-only transfer and simplifying scene construction. The approach uses separated geometry, articulations, and physical parameters to support policy training in reconstructed environments.

  • Prior behavior cloning methods can perform well on modest datasets but generally degrade under variations in scene layout and appearance.Source: §II · p. 3
  • Prior simulation-based RL work commonly relies on domain randomization, system identification, or improved simulator visuals to reduce the sim-to-real gap.Source: §II · p. 3
  • Earlier real-to-sim pipelines created digital twins using 3D reconstruction or inverse graphics, while RialTo uses these approaches as compatible scene-transfer components.Source: §II · p. 3
  • Visual-only reconstructed scenes cannot support physical interaction or adaptation to environmental changes beyond visual distractions.Source: §II · pp. 3–4

Real-to-sim transfer of scenes

C. Robustifying Real-World Imitation Learning Policies in Simulation

RialTo constructs simulation environments from scanned real-world scenes, then transfers real demonstrations into simulation to initialize robust policy learning. Its inverse-distillation procedure recovers privileged simulation-state demonstrations from perception-based policies trained on real data.

Scene construction

  • RialTo’s GUI converts scanned environments into organized USD assets that can be imported directly into the simulator.Users can cut meshes, specify joints, and organize reconstructed scenes after scanning with tools such as NeRFStudio, ARCode, or Polycam.Source: Figure 2 · p. 5

Policy initialization

Inverse distillation

  • Inverse distillation converts real-world observation-action demonstrations into simulation trajectories paired with privileged low-level state information.This addresses the absence of Lagrangian state measurements in real-world demonstrations.Source: C. Robustifying Real-World Imitation… · p. 5, 1) Inverse-distillation from… · p. 6
  • A perception-based policy trained on Dreal is executed on simulated sensor observations to collect successful privileged demonstrations Dsim.The procedure naturally pairs simulated perceptual observations with low-level Lagrangian state information.Source: 1) Inverse-distillation from… · p. 6

2) Reinforcement Learning Fine-tuning in Simulation:

RialTo trains a privileged state-based policy in simulation with sparse-reward reinforcement learning augmented by demonstrations, then distills it into a sensory student co-trained on real data. This combination addresses exploration, transfer, and the simulation-to-reality perceptual gap.

Reinforcement learning fine-tuning

  • RialTo uses sparse goal-detection rewards and an imitation-learning loss within PPO to fine-tune policies from privileged simulation-state demonstrations.The demonstrations help address scalable exploration without extensive reward engineering.Source: 2) Reinforcement Learning Fine-tuning… · p. 6
  • The imitation term biases simulation policies toward physically plausible, safe solutions while training robustness across randomized robot, object, and goal poses.The resulting policy operates from Lagrangian state across varied configurations and environmental conditions.Source: 2) Reinforcement Learning Fine-tuning… · p. 6

Teacher-student distillation

  • Because privileged state is unavailable during deployment, RialTo distills the simulation teacher into a policy operating directly on sensory observations.The student policy is trained using teacher-student distillation with interactive DAgger labeling.Source: D. Teacher-Student Distillation with… · p. 6
  • Co-training the distillation objective on simulation data and original real-world demonstrations bridges the perceptual gap and improves real-world generalization.The paper reports that this significantly increases resulting real-world success rate.Source: D. Teacher-Student Distillation with… · p. 6

IV. EXPERIMENTAL EVALUATION

RialTo is evaluated across eight manipulation tasks under randomized poses, visual distractors, and physical disturbances. Across tasks, it maintains substantially higher robustness than behavior cloning, including when behavior cloning uses more demonstrations.

Evaluation setup

  • The evaluation spans eight tasks involving free-object manipulation, articulated-object interaction, and less-controlled real-world scenes.Tasks include placing books, plates, and mugs, manipulating drawers and cabinets, opening a toaster, and putting a cup in the trash.Source: §IV · p. 7
  • The experiments test randomized object or robot poses, cluttered visual distractors, and physical disturbances applied during rollouts.Disturbances include moving manipulated objects or targets, closing articulated objects, and moving the robot base when possible.Source: §IV · p. 7

Baselines and ablations

  • With 15 demonstrations, RialTo is compared against imitation learning, while additional ablations examine real-data co-training, demonstration source, scene targeting, distractors, and demonstrations.The study also compares against behavior cloning using 50 demonstrations.Source: §IV · p. 7, A. RialTo Learns Robust Policies via… · p. 8

Robustness results

  • 91% average success is achieved for randomized object poses, 77% with distractors, and 75% with disturbances across tasks.These results are reported for RialTo in Figure 5.Source: A. RialTo Learns Robust Policies via… · p. 8
  • Behavior cloning averages 25% success with randomized poses, 11% with distractors, and 5% with disturbances.The presence of distractors and disturbances severely reduces pure imitation-learning performance.Source: A. RialTo Learns Robust Policies via… · p. 8

Data and effort

  • Collecting 50 behavior-cloning demonstrations takes 1 hour and 45 minutes, compared with 30 minutes for 15 RialTo demonstrations plus 15 minutes of active environment building.The paper reports that RialTo achieves approximately 2.5 times higher success despite fewer demonstrations and less time.Source: A. RialTo Learns Robust Policies via… · p. 8

B. Impact of Co-Training with Real-World Data

Co-training with real-world demonstrations improves RialTo's real-world performance, especially on harder manipulation tasks, while target-scene reconstruction outperforms training on randomized synthetic assets.

D. Scaling RialTo to In-the-Wild Environments

RialTo extends to uncontrolled, cluttered, and in-the-wild scenes while improving robustness over imitation learning. Its training benefits from distractors, demonstration bootstrapping, and compact-state fine-tuning.

D. Scaling RialTo to In-the-Wild Environments

Visual distractors

Demonstration bootstrapping

Compact-state fine-tuning

  • 96% success after 12 hours with compact-state RL contrasts with 1% after 35 hours using vision-only RL for drawer opening.Source: C. RL from Vision · p. 10

Scene construction

VII. LIMITATIONS AND CONCLUSION

RialTo combines real-world imitation learning with reinforcement learning in digital twins to obtain robust real-world policies, but its applicability depends on sensing, simulation, task, and training-time constraints.

Limitations

  • Accurate depth sensors are required, but they can struggle with thin, transparent, and reflective objects.Source: §VII · p. 11
  • The approach is limited to tasks that are easily simulated and objects that can be converted into digital assets, currently primarily articulated rigid bodies.Source: §VII · p. 11
  • RialTo currently requires around 2 days of end-to-end wall-clock training per task, making continual learning infeasible.Source: §VII · p. 11

Conclusion

  • RialTo achieves robustness to environmental variations and disturbances by combining real-world imitation learning with large-scale RL in on-the-fly digital twins.Source: §VII · p. 11

Method

  • The task success function selects successful inverse-distillation trajectories and supplies the sparse reward for RL fine-tuning.Source: §VIII · p. 16

A. Simulation details

RialTo uses Isaac Sim and reconstructed scene assets to train state-based and point-cloud policies, then transfers learned behavior through teacher-student distillation with simulated and real trajectories.

Simulation platform

  • Isaac Sim is used to simulate each task, with Orbit inspiring the reinforcement-learning and robot-learning implementation.Source: A. Simulation details · p. 16

Simulation parameters

  • The GUI uses default collision-mesh settings of 64 hull vertices and 32 convex hulls for most objects, with task-specific exceptions.Source: A. Simulation details · p. 16

State-based policy

Point-cloud policy

Distillation

C. Simulated Assets Baseline Details

This section describes baseline implementations for multi-task simulation and imitation learning, including distractor augmentation and the hardware and point-cloud configurations used in evaluation.

XI. FURTHER ANALYSIS

The further analysis examines computational costs, failure modes, demonstration requirements, synthetic assets, and multi-task extensions of RialTo.

A. RL from vision

  • Vision-based RL is substantially slower because it uses smaller GPU batches and point-cloud rendering is approximately 10x slower than compact-state processing.The section states that vision-policy batches are 100x smaller on the same GPU and that rendering point clouds adds roughly a 10x slowdown.Source: A. RL from vision · p. 19, Figure 14 · p. 19
  • A policy trained from scratch without demonstrations can exploit simulation inaccuracies, producing behavior that achieves 62% simulated toaster-opening accuracy but does not transfer to reality.Source: B. RL from Scratch · p. 19, Figure 15 · p. 20

C. RL from different amounts of real-world data

E. RialTo Multi-Task

  • The multi-task procedure trains separate policies, distills their trajectories into a task-conditioned policy, and applies sequential DAgger; drawer performance matches single-task RialTo at 90%.On the mug-on-shelf task, performance decreases from 100% single-task success to 80% multi-task success, while remaining above the imitation learning baseline.Source: E. RialTo Multi-Task · p. 20, Table XII · p. 20

F. Sim-to-real gap

This section analyzes sim-to-real performance and the GUI-based process for constructing articulated scenes from scans.

F. Sim-to-real gap

  • The final point-cloud policy generally shows no sim-to-real performance gap, and simulation can be harder than the real world for some tasks.For the mug-on-shelf task, simulation performance is reported as worse than real-world performance.Source: F. Sim-to-real gap · p. 20, Table XIII · p. 21

XIII. GUI USER STUDY

  • The GUI pipeline combines scene scanning, mesh processing, object separation, articulation, and placement to create usable articulated simulation scenes.Source: 1) 3D reconstruction software used · pp. 20–21, §XIII · p. 21
  • The user study measured scene and object scanning, processing, uploading, cutting, and articulation-related work across six participants.The study used a GUI and recorded the active time required for constructing articulated scenes.Source: §XIII · p. 21, Figure 16 · p. 21, Table XIV · p. 22
  • The GUI supports independent manipulation of scanned objects by allowing users to cut objects from connected scene meshes.Source: §XIII · p. 21
  • Faster users created smaller files and completed processing more quickly, but their scans had less precise background details than the most accurate scan.Source: §XIII · p. 22

A. Scaling laws of the RialTo GUI

The GUI’s scene-construction time scales with scene complexity, specifically the numbers of objects, cut objects, and joints, while policy training adds substantial computational time.

  • The time relation combines scene scanning, object scanning, object cutting, and joint-addition costs according to the relevant object, cut-object, and joint counts.Source: A. Scaling laws of the RialTo GUI · p. 22
  • Total active scene-construction time increases linearly with the number of objects and joints in the scene.Source: A. Scaling laws of the RialTo GUI · p. 22
  • Average measured times were 4:50 per object scan, 3:14 per scene scan, 2:54 per added joint, and 3:40 per cut object.Source: A. Scaling laws of the RialTo GUI · p. 22
  • Training one task averages 2 days and 3 hours, including real-world demonstration learning, RL fine-tuning, and teacher-student distillation.The reported stages average 7 hours, 20 hours, and 24 hours respectively.Source: §XIV · pp. 22–23

Related papers

Loading 2403.03949v3…