Source-linked AI summary
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, Vincent Vanhoucke
TL;DR
The paper tackles the expense of labeled grasping data and the difficulty of transferring purely simulated training to real objects. It combines randomized simulation with pixel- and feature-level domain adaptation for monocular RGB grasping, reducing real-data requirements while maintaining real-world performance in the reported evaluations.
Problem
Labeled grasping datasets are costly, while models trained purely on simulated images often fail to generalize to real-world data.
Method
The paper combines randomized simulated grasping experience with pixel-level GraspGAN adaptation and feature-level DANN adaptation for monocular RGB grasping.
Results
The hybrid GraspGAN-DANN approach performs best in most cases, with the largest gains in lower real-data regimes.
Takeaways & Limitations
Synthetic data and domain adaptation can reduce the real-world samples needed for a given performance level by up to 50 times.
Takeaways & Limitations
The adaptation methods do not explicitly reason about physical discrepancies between simulation and the real world.
Abstract
from arXiv · showhide
Instrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative is to use off-the-shelf simulators to render synthetic data for which ground-truth annotations are generated automatically. Unfortunately, models trained purely on simulated data often fail to generalize to the real world. We study how randomized simulated environments and domain adaptation methods can be extended to train a grasping system to grasp novel objects from raw monocular RGB images. We extensively evaluate our approaches with a total of more than 25,000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN. We show that, by using synthetic data and domain adaptation, we are able to reduce the number of real-world samples needed to achieve a given level of performance by up to 50 times, using only randomly generated simulated objects. We also show that by using only unlabeled real-world data and our GraspGAN methodology, we obtain real-world grasping performance without any real-world labels that is similar to that achieved with 939,777 labeled real-world samples.
I. INTRODUCTION
The paper addresses the cost of labeled grasping data by combining synthetic simulation with domain adaptation for monocular RGB grasping. It evaluates randomized simulation and adaptation methods for transferring grasping to novel real-world objects.
- Learning-based grasping can generalize to real-world object statistics but requires large labeled datasets that are costly in time and money.Performance may be strongly influenced by the amount of available data.
- Simulation offers automatically annotated synthetic experience while preserving the same learning system, but simulated images and objects must adequately match real-world conditions.The paper treats simulation-to-real image differences and simulated-object generality as its principal challenges.
- The work proposes approaches for incorporating synthetic data into end-to-end monocular RGB grasping, especially in lower-data and no-data regimes.The system targets grasping previously unseen natural objects using monocular RGB images.
- The experiments include 25,704 real grasps of 36 diverse test objects across simulated objects, randomization strategies, and domain adaptation techniques.
- The paper presents a claimed first demonstration of effective simulation-to-real transfer for monocular vision-based grasping on previously unseen natural objects.
II. RELATED WORK
Related work establishes simulation as a cheap source of annotated data and situates this study among randomized simulation and domain adaptation approaches. The paper focuses on monocular RGB grasping, where appearance differences remain challenging.
- Simulation-to-real transfer is attractive because simulators can provide practically infinite cheap data with flawless annotations.
- Earlier randomized grasping methods varied textures, lighting, and camera position but were demonstrated mainly in simple geometric environments.
- Unlike those methods, this work evaluates transfer to diverse, novel real-world objects.
- Domain adaptation transfers models from a source domain to a target domain; here, simulation is the source and the real world is the target.
- Feature-level adaptation learns domain-invariant features, whereas pixel-level adaptation operates on image appearance.
III. BACKGROUND
The paper uses a tested data-driven monocular vision grasping approach as the base system and studies how simulation and domain adaptation affect its training. The focus is on combining simulated and real-world data.
- The study compares training regimes that use simulation and domain adaptation with an existing data-driven monocular vision-based grasping approach.
- The method is based on two main domain adaptation techniques discussed in the paper.
A. Deep Vision-Based Robotic Grasping
The grasping system predicts grasp success from paired monocular images and a motion command, then uses a servoing function to select robot actions. This work extends training of the prediction network with simulated data while leaving servoing unchanged.
- The grasp prediction CNN C(x_i,v_i) takes images before and during a grasp attempt plus a motion command and outputs predicted grasp-success probability.
- For top-down pinch grasps, each motion command has five dimensions: three for position and two for sine-cosine rotation encoding.
- A manually designed servoing function uses C's grasp probabilities to continuously control the robot's motor command.
- Training samples contain visual inputs, motion commands, and the success label of the entire grasp sequence.
- The experiments use self-supervised real-world episodes from six physical Kuka IIWA arms and compare simulated-real training regimes.
- 67.65% successful grasps was achieved by the best CNN trained on the entire Kuka real dataset.
B. Domain Adaptation
The approach combines domain-adversarial training with pixel-level adaptation to transfer grasping from simulation to real images. Pixel-level adaptation learns a generator that makes simulated images resemble real-world samples, while DANN learns domain-invariant features.
- Domain-adversarial training uses shared features for task prediction and domain classification, encouraging representations that remain expressive while becoming domain-invariant.A gradient reversal layer implements the adversarial optimization, while the task loss keeps shared features relevant to grasping.
- The DANN domain loss is a cross-entropy objective over source and target samples using binary domain labels.The domain classifier predicts each input’s domain, with d_i ∈{0,1} denoting the ground-truth domain label.
- Pixel-level domain adaptation learns a generator that maps source images into adapted images resembling the target domain before task-specific prediction.The generator is trained with a discriminator that estimates whether images are real-world samples.
- The proposed pixel-level method addresses transfer without real-world object models and targets previously unseen real-world objects.It operates on 472 × 472 images and modifies the generator, discriminator, adversarial objective, and auxiliary losses relative to earlier implementations.
IV. OUR APPROACH
The approach studies simulation-to-real transfer by varying object models, simulated scene appearance and dynamics, and how simulated and real experience are combined.
- The study evaluates three transfer factors: simulated 3D object models, scene appearance and dynamics, and integration of simulated and real experience.
A. Grasping in Simulation
The simulation environment uses Bullet-based virtual grasping with diverse object sources and large-scale parallel data collection. Evaluation includes challenging real-world objects excluded from training.
- A Bullet-based virtual environment was constructed to generate simulation datasets for evaluating simulation-to-real transfer.The environment emulates the Kuka grasping setup and uses a simple renderer.
- Two object sources were evaluated: 1,000 procedurally generated random geometric shapes and realistic objects from the ShapeNet repository.Procedural objects were built by attaching rectangular prisms at random locations and orientations, then converting and smoothing the resulting meshes.
- The real-world evaluation used 36 objects that were excluded from training and simulation-dataset construction.The test set was designed to vary in shapes, sizes, and material properties.
- 70%-90% simulated grasp success was achieved after training grasp success predictors from scratch on simulated datasets.The collected samples included diversity from grasp prediction networks with varying performance.
B. Virtual Scene Randomization
The study examines whether randomizing visual appearance and scene dynamics changes grasp performance. Adding Gaussian noise to simulated motor commands improved early real-world grasp success, whereas adding it to real data did not.
- Adding ε cm of Gaussian noise, with ε ∼N (0,1), to horizontal motor-command components improved real grasp success in early experiments.This randomization was therefore added to all simulated samples.
- Adding the same motor-command noise to real data did not improve grasp performance.
C. Domain Adaptation for Vision-Based Grasping
The paper combines feature-level and pixel-level domain adaptation to transfer synthetic grasping inputs toward the real world while preserving task-relevant semantics. Its GraspGAN uses adversarial realism objectives together with grasp-task and content constraints, while DANN and domain-specific batch normalization address feature and domain-statistics differences.
- GraspGAN adapts synthetic images toward realistic appearances while preserving the semantics needed by the grasping task.The adapted generator is subsequently fixed, while feature-level adaptation operates on features from transferred images and synthetic motor commands.
- DANN applies domain-adversarial adaptation to the last convolutional layer of the grasp-success prediction model.This placement performed better in preliminary experiments than applying the DANN loss to other layers.
- Domain-specific batch-normalization mixing computes separate simulated and real-world batch statistics while sharing the same model parameters.The authors report DBN mixing as useful for domain adaptation even without a DANN loss.
- GraspGAN combines a U-Net generator with a multi-scale, patch-based discriminator that evaluates realism at global and local levels.The discriminator receives paired synthetic images as a six-channel input, encouraging consistent relationships between corresponding images while modeling local textures and global consistency.
- The generator maps synthetic images xs to adapted images xf using alternating optimization with adversarial, task, and content-similarity losses.The discriminator uses an LSGAN objective, while the task loss measures grasp-success prediction with binomial cross-entropy.
V. EVALUATION
The evaluation tests simulation, randomization, object realism, and domain adaptation across varying real-data regimes, using 25,704 physical grasps and unseen objects. Simulated data consistently improves real-world grasping, while GraspGAN-based unsupervised adaptation outperforms simulated-only and a large labeled real-only baseline.
- Evaluation setup: 25,704 physical grasps evaluated transfer across 36 diverse test objects, with 8 million simulated samples used whenever simulation was included.The test objects were excluded from real-world training and simulation dataset construction.
- Simulation benefits: Simulated data significantly and consistently improves real-world grasp performance regardless of the number of real-world samples.This conclusion is based on procedurally generated objects and the gains shown in Fig. 5.
- Simulation conditions: Procedural objects outperform ShapeNet objects across the evaluated randomization scenarios when combined with 10% of the real-world data.The authors therefore use procedural objects for the remaining experiments.
- Simulation conditions: Visual randomization with DBN mixing improves upon naive mixing without randomization across the evaluated experiments, although isolated visual-randomization gains are generally inconclusive.The broader comparison uses Table III, while Table I reports the procedural-object result under 10% real data.
- Domain adaptation: The GraspGAN-DANN hybrid performs best in most cases and yields the largest gains in lower real-data regimes.DANN with DBN mixing generally outperforms naive mixing, but its effect on randomized data is inconclusive in 3 of 5 cases.
- Domain adaptation: Using only unlabeled real-world data, GraspGAN outperforms sim-only models with or without randomization and a real-only model trained on 939,777 labeled samples.The GraspGAN was trained using all 9 million real samples without their labels, and the grasping model used data refined by the generator.
- Scope and caveats: The evaluation targets a challenging monocular-RGB setting with diverse objects, while absolute grasp success is not claimed to be state of the art.The authors emphasize equal-real-data comparisons rather than the best possible grasping system.
VI. CONCLUSION
The study finds that simulated data and domain adaptation can substantially reduce real-world data requirements for monocular RGB robotic grasping. However, the transfer methods do not explicitly reason about physical discrepancies between simulation and reality, and the reported success rate leaves room for improvement.
- 50 times fewer real-world samples achieved comparable or better grasping performance when simulated data was included.The system operated from over-the-shoulder monocular RGB images, without depth information or analytic 3D models.
- Realistic 3D models were not essential for simulated training to improve grasping performance.
- Domain adaptation substantially improved performance in most cases by producing plausible transformations of synthetic images.
- The adaptation methods addressed cross-domain invariance but did not explicitly model physical discrepancies between simulation and the real world.The experiments did find that randomizing dynamics properties was important.
- The reported grasp success rate still has room for improvement, motivating further research.