PaperlayerView exact revision on arXiv

Source-linked AI summary

RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real

Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine, Julian Ibarz, Mohi Khansari

arXiv:2006.09001v1cs.ROcs.CVcs.LG

TL;DR

Real-world RL data collection is costly, and task-agnostic simulation-to-real translation may lose task-relevant features. RL-CycleGAN adds Q-value-based scene consistency to CycleGAN translation, achieving strong transfer on two robotic grasping tasks with limited real data. The method improves grasping performance while avoiding task-specific engineering, but addresses only the visual simulation-to-real gap.

  • Problem

    Real-world RL data is costly to collect, while task-agnostic translation may fail to preserve features relevant to the task.

  • Method

    RL-CycleGAN constrains CycleGAN image translation with an RL-scene consistency loss that preserves Q-values associated with task-relevant image semantics.

  • Results

    RL-CycleGAN substantially outperforms prior approaches on two vision-based robotic grasping tasks and achieves high real-world performance with modest real data.

  • Takeaways & Limitations

    The approach enables automated, task-aware visual simulation-to-real transfer without per-task manual engineering.

  • Takeaways & Limitations

    RL-CycleGAN addresses visual but not physics-based simulation-to-real differences, which would require adapting entire state-action trajectories.

Abstract

from arXiv · show

Deep neural network based reinforcement learning (RL) can learn appropriate visual representations for complex tasks like vision-based robotic grasping without the need for manually engineering or prior learning a perception system. However, data for RL is collected via running an agent in the desired environment, and for applications like robotics, running a robot in the real world may be extremely costly and time consuming. Simulated training offers an appealing alternative, but ensuring that policies trained in simulation can transfer effectively into the real world requires additional machinery. Simulations may not match reality, and typically bridging the simulation-to-reality gap requires domain knowledge and task-specific engineering. We can automate this process by employing generative models to translate simulated images into realistic ones. However, this sort of translation is typically task-agnostic, in that the translated images may not preserve all features that are relevant to the task. In this paper, we introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. This allows us to learn a task-aware translation. Incorporating this loss into unsupervised domain translation, we obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning. In evaluations of RL-CycleGAN on two vision-based robotics grasping tasks, we show that RL-CycleGAN offers a substantial improvement over a number of prior methods for sim-to-real transfer, attaining excellent real-world performance with only a modest number of real-world observations.

1. Introduction

RL can learn task-specific visual representations, but real-world data collection is costly and simulation-to-reality transfer remains difficult. RL-CycleGAN addresses this gap with automated, task-aware image translation using an RL-scene consistency loss.

  • Real-world reinforcement-learning data is costly and time-consuming to collect, motivating policy training in simulation.
  • Task-agnostic domain adaptation and simulation randomization often require manual modification for each task.
  • RL-CycleGAN automatically translates simulated observations into realistic images using a GAN and an off-policy real-experience dataset.
  • The method targets vision-based transfer and does not address physics-based simulation-to-real differences.
  • RL-CycleGAN uses RL-scene consistency with CycleGAN losses, avoids per-task engineering, and substantially outperforms prior approaches on two robotic grasping tasks.

2. Related Work

Prior simulation-to-real methods adapt features, pixels, or simulation variability, but often require manual task-specific design. CycleGAN improves unpaired translation while introducing risks of semantic alteration that RL-CycleGAN addresses with task-aware guidance.

  • Simulation data can be abundant but differs from reality visually and physically, while this work focuses on visual differences such as textures, lighting, colors, and objects.
  • Domain adaptation learns from many simulated source examples and few real target examples through feature-level or pixel-level adaptation.
  • The method extends CycleGAN-based unpaired pixel translation with additional reinforcement-learning-specific losses.
  • RCAN requires manually defining canonical task scenes and corresponding simulator randomization.
  • GraspGAN preserves simulated segmentation masks as an auxiliary task to constrain image translation for robotic grasping.
  • CycleGAN uses two reciprocal generators and cycle consistency to encourage preservation of source-image aspects during unpaired translation.
  • CycleGAN may hide information or deterministically alter semantics, motivating joint GAN training with an RL model.
  • The evaluation considers closed-loop, vision-based grasping with Q-learning and RGB image observations.

3. Preliminaries

The preliminaries introduce CycleGAN for unpaired simulation-to-real image translation and Q-learning for value-based control. These components provide the basis for adding RL-specific consistency constraints.

  • 3.1. CycleGAN: CycleGAN learns reciprocal Sim2Real and Real2Sim mappings between unpaired simulation and real image domains.
  • 3.1. CycleGAN: Adversarial objectives train both mappings, while cycle consistency encourages each image to be reconstructed after round-trip translation.
  • 3.1. CycleGAN: Cycle consistency prevents drastic departures because the original scene must remain recoverable, although scene alterations can still occur.
  • 3.2. Q-learning: Q-learning estimates total expected future reward with Q(s, a), where vision-based states are input images and actions are candidate actions.
  • 3.2. Q-learning: The policy selects argmax_a Q(s, a), and RL-CycleGAN jointly trains a Q-function with CycleGAN using Q-values for additional consistency losses.

4. RL-CycleGAN

RL-CycleGAN preserves task-relevant semantics by constraining CycleGAN translations to maintain Q-values across original, adapted, and cycled images. It jointly trains translation and task models using simulated and real image-action data.

  • 4. RL-CycleGAN: The method distinguishes visual style from RL-relevant semantics by requiring adapted images to preserve task-model outputs.
  • 4. RL-CycleGAN: A deep Q-learning task model processes simulated and real images through separate Qsim and Qreal networks.
  • 4. RL-CycleGAN: The six original and translated images are passed through the corresponding Q-networks to produce Q-values for consistency training.
  • 4. RL-CycleGAN: RL-scene consistency encourages equal Q-values within simulated and real image triples representing the same scene.
  • 4. RL-CycleGAN: Separate Q-networks are trained with TD-loss on original and generated images, while the full objective combines weighted losses.
  • 4. RL-CycleGAN: The networks are jointly trained from scratch using simulated data and off-policy real episodes; final performance was best after freezing Sim2Real and retraining Q.
  • 4. RL-CycleGAN: Figure 2 depicts Sim2Real translation, Real2Sim cycling, separate Q-network training, and shared Q-value constraints across simulated and real images.

5. Task Setup

The paper evaluates vision-based grasping on two real-robot setups with different robots, objects, bins, and simulation environments. Both tasks use simulated environments with substantial visual gaps from reality.

  • The evaluation uses two real-world robot grasping setups with different robots, objects, bins, and simulation environments.
  • Robot 1 Setup: Robot 1 uses a Kuka IIWA to grasp varied objects from a metal bin, with procedurally generated random geometric shapes in simulation.Models trained purely in simulation perform poorly on real robots because simulated images are unrealistic.
  • Robot 2 Setup: Robot 2 grasps trash-like items from three adjacent bins using a mobile base that remains fixed during each episode.The base is randomly positioned at the start of each episode.
  • Robot 2 Setup: Robot 2 policies must generalize across all three bins and varied camera angles while the base location changes between episodes.
  • Evaluation Protocol: Evaluation includes single-bin grasping from a central base position and multi-bin grasping with varied base locations and object-bin assignments.Each evaluation uses 6 robots, 6 grasps, and three repetitions.

6. Experiments

Experiments compare visual adaptation methods and assess whether RL-CycleGAN improves grasping when simulation is combined with limited off-policy real data. Across two robot setups, it substantially improves real-world grasp success and reaches strong performance with fewer on-policy trials.

  • The experiments evaluate simulation-to-real methods for robotic grasping with limited off-policy real data and relatively cheap simulated experience.The goal is to reduce the performance gap from using off-policy data alone and reduce required real-world data.
  • GANs For RL: Without adaptation, simulation-only training achieves 21% real-world grasp success versus 95% in simulation; visual randomization raises real success to 37%.
  • GANs For RL: RL-CycleGAN achieves 70% grasp success, exceeding regular GAN at 29%, CycleGAN at 61%, and GraspGAN at 63% when models train on adapted simulations alone.Its RL-scene consistency loss preserves Q-value-relevant task information without manually identifying task-salient properties.
  • Mixing Real Data And Simulation: With 5,000 real-world trials for Robot 1, RL-CycleGAN improves grasp success from 15% to 75%; with 580,000 trials, it improves success from 87% to 94%.The same real data trains both the GAN and the RL process.
  • Mixing Real Data And Simulation: With 3,000 real episodes for Robot 2, RL-CycleGAN improves performance from 13% to 72%; with 80,000 episodes, it reaches 95% grasp success.Multi-bin grasping with randomized base locations reaches 93% success.
  • On-robot Fine-tuning: Using 5,000 off-policy episodes, RL-CycleGAN reaches 75% success and, after 10,000 on-policy episodes, matches RCAN at 94%.RCAN required 28,000 on-policy episodes to reach 94% success.

7. Conclusion

RL-CycleGAN addresses the visual simulation-to-real gap for vision-based robotics and improves real-world grasping across two varied setups. It separates realistic style adaptation from RL-relevant semantics while leaving physics-based differences for future work.

  • RL-CycleGAN significantly improves real-world vision-based robotics across two varied grasping setups.
  • The RL scene consistency loss preserves RL-relevant semantics while CycleGAN losses adapt visual style, removing task-specific feature engineering.
  • RL-CycleGAN addresses visual, not physics-based, simulation-to-real differences; adapting entire state-action trajectories remains future work.
  • 28,000 real grasps with RL-CycleGAN attain Robot 1 performance matched by 580,000 real grasps.
  • 94% success with 580,000 real grasps makes RL-CycleGAN comparable to the 96% state-of-the-art result without costly on-robot training.
  • 72% grasp success on Robot 2 with 3,000 real grasps outperforms the 36% baseline trained with 80,000 grasps.

A. Model Training Details

RL-CycleGAN jointly trains CycleGAN components and QT-Opt using real and simulated data, with selective gradient application and data-dependent real-loss weighting. Training includes safeguards against instability and Q-network-generator shortcut behavior.

  • The U-net generators use spectral normalization, while images are resized and skip-connected features are combined during up-convolutions.
  • RL-CycleGAN jointly trains the CycleGAN with the QT-Opt Q-function using real and simulated image batches.The training setup uses separate replay buffers and Bellman updates for off-policy real and adapted simulation-to-real data.
  • A final Qreal is trained from scratch with the pretrained and fixed Sim2Real generator using only reinforcement learning loss.
  • End-to-end optimization can produce incorrect uniform Q-values and spuriously low RL-scene loss through mode collapse.
  • Selective gradients apply RL loss only to Q and RL-scene consistency loss only to generators G and F, discouraging poor encoding-decoding shortcuts.
  • The real-data loss weight is 0.1 with 3,000 episodes and 2.0 with 80,000 episodes, with the corresponding values reported as optimal.

B. Robot Simulated Objects

The two grasping setups use different simulation-object strategies: procedural random shapes for unseen-object evaluation and simulated versions of common real-world objects for trash-like grasping.

  • Robot 1: Robot 1 procedurally generates random-shaped objects by attaching rectangular prisms at random locations and orientations.The generated objects support evaluation on unseen objects.
  • Robot 2: Robot 2 simulates 51 common real-world objects, including bottles, cups, utensils, cans, mugs, and a wine glass.
  • Robot 2: The Robot 2 simulated-object set does not cover all real-world objects used during evaluation.
Loading 2006.09001v1…