Source-linked AI summary
Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, Konstantinos Bousmalis
TL;DR
Real-world robotics data is costly, and simulated vision does not transfer readily to reality while common adaptation can require unlabeled real images. RCAN learns a randomized-to-canonical image translation using simulation only, enabling a simulation-trained grasping policy to reach 70% zero-shot success and 91% after 5,000 real-world grasps.
Problem
Real-world data collection is costly, while simulation-trained vision models face a reality gap and visual domain adaptation often requires unlabeled real-world data.
Method
RCAN learns from randomized simulation images to produce equivalent canonical simulation images, then translates real-world images into that canonical domain for policy execution.
Results
70% zero-shot grasp success was achieved on unseen objects, rising to 91% with 5,000 real-world grasps versus 580,000 grasps for a from-scratch system.
Takeaways & Limitations
RCAN enables grasping policies trained in simulation to transfer to the real world while using over 99% fewer real-world samples than the stated from-scratch comparison.
Takeaways & Limitations
Translation artifacts can impair grasping, particularly when the agent must distinguish small objects from artifacts during regrasping decisions.
Abstract
from arXiv · showhide
Real world data, especially in the domain of robotics, is notoriously costly to collect. One way to circumvent this can be to leverage the power of simulation to produce large amounts of labelled data. However, training models on simulated images does not readily transfer to real-world ones. Using domain adaptation methods to cross this "reality gap" requires a large amount of unlabelled real-world data, whilst domain randomization alone can waste modeling power. In this paper, we present Randomized-to-Canonical Adaptation Networks (RCANs), a novel approach to crossing the visual reality gap that uses no real-world data. Our method learns to translate randomized rendered images into their equivalent non-randomized, canonical versions. This in turn allows for real images to also be translated into canonical sim images. We demonstrate the effectiveness of this sim-to-real approach by training a vision-based closed-loop grasping reinforcement learning agent in simulation, and then transferring it to the real world to attain 70% zero-shot grasp success on unseen objects, a result that almost doubles the success of learning the same task directly on domain randomization alone. Additionally, by joint finetuning in the real-world with only 5,000 real-world grasps, our method achieves 91%, attaining comparable performance to a state-of-the-art system trained with 580,000 real-world grasps, resulting in a reduction of real-world data by more than 99%.
1. Introduction
The paper addresses costly real-world data collection and simulation-to-real transfer by using RCAN to map randomized simulation and real images into a canonical simulation domain. This enables simulation-trained grasping to transfer zero-shot and with substantially less real-world data.
- Motivation: Real-robot data collection is expensive and cumbersome, while simulation-to-real transfer remains difficult because of large domain shifts.The shift includes simulated images and physics, as well as differences in object distributions.
- Existing approaches: Visual domain adaptation typically requires large amounts of unlabelled real-world images, whereas domain randomization trains across varied sensory inputs to improve robustness.Domain randomization targets superficial visual variation such as textures and shadows.
- Advantages: RCAN provides an interpretable intermediate canonical representation and does not require real-world data to learn its translation function.The authors contrast these properties with domain adaptation and direct policy training under domain randomization.
- RCAN: RCAN uses a cGAN to transform randomized simulation images into equivalent non-randomized canonical images, enabling real-world images to be mapped to the same domain.The grasping policy is trained in the predefined canonical simulator, while RCAN performs reality-to-simulation translation without real-world data.
- Results: 70% zero-shot grasp success was achieved on unseen real-world objects after training QT-Opt purely in simulation, almost doubling direct domain-randomization performance.The result demonstrates transfer without real-world training data for the policy.
- Results: 91% grasp performance was reached with only 5,000 additional real-world grasping episodes, compared with QT-Opt trained from scratch using 580,000 real-world grasps.This corresponds to a reduction of over 99% in required real-world samples.
2. Related Work
Prior work reduces simulation-to-real costs through domain adaptation, randomized simulation, and reality-to-simulation transfer, but RCAN differs by learning its adaptation entirely in simulation.
- Simulation-to-real transfer: Simulation-to-real transfer reduces expensive real-data collection but must address visual and dynamics differences between simulation and reality.Prior work includes rendered-object grasping models and other transfer methods.
- Domain randomization: Domain randomization trains policies directly across randomized simulated environments for transfer to real-world manipulation tasks.The approach extends data augmentation and varies simulated sensory properties.
- Visual domain adaptation: Visual domain adaptation learns source-to-target generalization using mostly unlabeled target data, through feature-level or pixel-level adaptation.Pixel-level adaptation restyles source images, while image-to-image translation usually learns from matched domain pairs.
- Related grasping methods: GraspGAN combines pixel- and feature-level adaptation but requires significant previously collected unlabeled real-world data.This distinguishes it from RCAN’s data-free reality-to-simulation training.
- Reality-to-simulation transfer: Reality-to-simulation transfer decouples policy training from adaptation, but prior work used unlabeled real-world data whereas RCAN learns from randomized-to-canonical simulation images alone.RCAN adapts real images to the canonical simulation domain at inference.
3. Background
The paper uses QT-Opt as its reinforcement-learning demonstration while grounding the method in Q-learning and Bellman-error minimization.
- Role in the paper: RCAN is compatible with reinforcement-learning or imitation-learning algorithms because it adapts only the input.QT-Opt is used as a state-of-the-art vision-based grasping baseline and demonstration algorithm.
- QT-Opt: QT-Opt is an off-policy, continuous-action generalization of Q-learning that learns a parameterized state-action value function.The objective is to maximize expected total reward through a learned policy.
- Q-learning objective: The Q-function is learned by minimizing a Bellman error between predicted values and target values.The target is defined as QT(s, a, s′) = r(s, a) + γV(s′), with D specified as cross-entropy.
- Action selection: QT-Opt selects actions by evaluating the argmax with the cross-entropy method rather than directly taking the Q-function argmax.Two target networks, Polyak averaging, and clipped double Q-learning improve stability.
4. Method
RCAN learns to translate randomized simulation images into canonical representations, enabling real-world images to be transformed for a policy trained entirely in simulation. Its generator combines paired reconstruction, auxiliary semantic and depth supervision, and adversarial training before feeding adapted images to QT-Opt.
- RCAN overview: RCAN uses a conditional GAN to translate randomized simulated images into canonical simulation images, then applies the trained generator to real-world images.The canonical representation uses uniform colors for the background, tray, and arm while preserving object textures.
- Training data: Training pairs randomized and canonical observations of the same simulated scene, with matching semantic content plus canonical segmentation and depth targets.The paired observations are collected by applying randomization, capturing an image, restoring the canonical scene, and capturing its counterpart.
- Training data: Simulation randomization varies textures, lighting, and aspects of the arm and tray to broaden the visual diversity of training scenes.Textures are sampled from over 5,000 images, and lighting position, direction, and color are randomized.
- RCAN training: The generator is trained with image, segmentation, and depth equality losses, while a GAN objective encourages sharp canonical-looking outputs.The image loss uses mean pairwise squared error, whereas the auxiliary mask and depth losses use L2 distance.
- Policy integration: QT-Opt receives the original image concatenated channel-wise with the generator’s adapted image, while generated depth and segmentation masks remain training auxiliaries.This adapted state is used for a policy that is trained in simulation and then deployed in the real world.
- Qualitative behavior: Qualitative outputs reconstruct canonical appearances in cluttered real scenes, but real-to-canonical reconstruction of the gripper is weaker than randomized-to-canonical reconstruction.The gripper discrepancy is identified as a source of failure cases discussed later in the paper.
5. Experiments
The experiments evaluate zero-shot and low-data transfer of QT-Opt grasping policies across multiple Kuka IIWA robots and unseen objects. RCAN substantially improves simulation-to-real transfer and reaches high performance with only 5,000 real-world grasps.
- Experimental setup: The evaluation tests grasping arbitrary unseen objects across multiple Kuka IIWA robots, including zero-shot simulation-trained policies and low-data real-world finetuning.Each robot performs 102 grasp attempts on 5 to 6 previously unseen test objects.
- Zero-shot and baseline results: 87% grasp success was achieved by QT-Opt after 580,000 off-policy real-world grasps.This result is the real-world training reference reported from Kalashnikov et al.
- Zero-shot and baseline results: 70% zero-shot grasp success was achieved by RCAN, compared with 33% to 37% for policies trained directly with domain randomization.QT-Opt trained with heavy randomization remained stable in simulation, but direct randomization transferred poorly; RCAN more than doubled real-world success.
- Joint finetuning: 5,000 real-world grasps raised direct-domain-randomization policies to 77% to 85% grasping success.The authors report a rapid increase of approximately 50 percentage points after joint finetuning.
- Joint finetuning: 91% grasp success was achieved by RCAN after joint finetuning with 5,000 real-world grasps, exceeding the 87% system trained with 580,000 grasps.With 28,000 online grasps, RCAN reached 94%, approximately matching the corresponding high-data performance.
- Data efficiency: Most success gains for both RCAN and mild randomization occurred within the first 2,000 on-policy grasps.The experiment re-evaluated both agents at every 1,000-grasp checkpoint through 5,000 grasps.
6. Conclusion
RCAN translates randomized simulation images into a canonical representation, enabling real-world images to be translated into the same representation for policies trained in simulation. The approach outperforms domain randomization and is presented as applicable beyond grasping where real-world data is expensive.
- RCAN learns randomized-to-canonical translation so real-world images can be converted into the canonical representation used by simulation-trained grasping policies.
- RCAN is reported as superior to the common domain-randomization approach for sim-to-real grasping.
- The transfer strategy is proposed as applicable beyond grasping, including producing segmentation masks for self-driving cars.
- Future work includes using unlabelled real-world data to improve real-to-canonical translation and testing auxiliary outputs as additional grasping-network inputs.
A. RCAN Architecture
The generator G uses a U-Net-style convolutional architecture to transform source images into generated outputs. Its implementation combines downsampling, upsampling, skip connections, normalization, and ReLU activations.
- G follows a U-Net-style architecture with convolutional downsampling, bilinear upsampling, and skip connections fused by channel-wise concatenation.
- The first two downsampling layers use 3×3 convolutions with stride 2, while later layers use average pooling with 3 × 3 stride-1 convolutions.
- Upsampling uses bilinear interpolation followed by 3 × 3 stride-1 convolutions.
- Skip connections are merged through channel-wise concatenation followed by a 1 × 1 convolution.
- The layers use instance normalization and ReLU activations.
B. QT-Opt Architecture
The grasping Q-function combines source and generated canonical images with action and state information to estimate action values. The generator also produces RGB, segmentation, and depth outputs for the adaptation pipeline.
- The action space contains Cartesian translation, sine-cosine rotation, a gripper open/close command, and a learned stopping criterion.
- The Q-function concatenates source and generated canonical images channel-wise before processing them with a convolutional network.
- Action and state variables are fused with image features to produce the scalar Q value Qθ(s, a).
- The reward is 1 for a successful grasp, 0 for an unsuccessful grasp, and −0.05 on other transitions.
- The generator processes source RGB images to produce generated RGB images together with segmentation masks and depth images.