PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Randomization and Generative Models for Robotic Grasping

Joshua Tobin, Lukas Biewald, Rocky Duan, Marcin Andrychowicz, Ankur Handa, Vikash Kumar, Bob McGrew, Jonas Schneider, Peter Welinder, Wojciech Zaremba, Pieter Abbeel

arXiv:1710.06425v2cs.ROcs.LG

TL;DR

The paper addresses limited object diversity in robotic grasping datasets and tests whether unrealistic procedural objects can support generalization. It combines domain-randomized object synthesis with autoregressive grasp-distribution modeling, achieving over 90% simulated success on unseen realistic objects and 80% real-world success despite simulation-only training.

  • Problem

    Grasping models often train on only hundreds or thousands of unique object instances, creating a data-availability and generalization challenge.

  • Method

    The paper generates diverse unrealistic procedural objects using domain randomization and trains an autoregressive model that maps sensor inputs to a grasp probability distribution.

  • Results

    Greater than 90% success was achieved on previously unseen realistic objects in simulation, and real-world grasp attempts reached 80% despite training entirely in simulation.

  • Takeaways & Limitations

    Procedurally generated non-realistic objects supported high success on realistic objects, while autoregressive modeling enabled efficient high-likelihood grasp sampling.

Abstract

from arXiv · show

Deep learning-based robotic grasping has made significant progress thanks to algorithmic improvements and increased data availability. However, state-of-the-art models are often trained on as few as hundreds or thousands of unique object instances, and as a result generalization can be a challenge. In this work, we explore a novel data generation pipeline for training a deep neural network to perform grasp planning that applies the idea of domain randomization to object synthesis. We generate millions of unique, unrealistic procedurally generated objects, and train a deep neural network to perform grasp planning on these objects. Since the distribution of successful grasps for a given object can be highly multimodal, we propose an autoregressive grasp planning model that maps sensor inputs of a scene to a probability distribution over possible grasps. This model allows us to sample grasps efficiently at test time (or avoid sampling entirely). We evaluate our model architecture and data generation pipeline in simulation and the real world. We find we can achieve a $>$90% success rate on previously unseen realistic objects at test time in simulation despite having only been trained on random objects. We also demonstrate an 80% success rate on real-world grasp attempts despite having only been trained on random simulated objects.

I. INTRODUCTION

The paper addresses limited object diversity in grasping data and proposes procedurally generated objects plus learned grasp sampling to improve generalization and efficiency. Simulation and real-world evaluations show strong performance despite training without realistic objects.

  • Motivation: Current grasping methods may use only hundreds or thousands of unique object instances, limiting generalization.The paper contrasts this with ImageNet’s approximately 15M unique images across 22K categories.
  • Motivation: Domain randomization trains models for difficult test domains by randomizing non-essential simulator aspects, motivating unrealistic procedural object generation.The paper hypothesizes that sufficient variety in generated meshes can generalize to realistic objects.
  • Motivation: Learned grasp sampling complements grasp-quality estimation by selecting promising grasps efficiently for challenging objects.Prior methods commonly use random sampling or small online optimization problems.
  • Approach: The autoregressive model maps sensor inputs to a probability distribution over grasps representing estimated success likelihood.A second hand-camera model scores sampled grasps to choose which one to execute.
  • Results: 92% success was achieved on test objects after scoring sampled grasps, while the 20 highest-probability grasps contained a successful grasp for 96% of test objects.The highest-probability grasp alone succeeded on 89% of test objects.
  • Results: 84% success on random objects and 92% on previously unseen real-world objects were achieved despite training only on non-realistic randomly generated objects; real-world attempts reached 80%.These results evaluate the data-generation and grasp-planning approach across simulation and physical experiments.

II. RELATED WORK

The related work situates domain randomization within simulation-to-real robotics and contrasts this paper with concurrent procedural-object grasping research. It also motivates autoregressive modeling through direct likelihood computation.

  • Domain randomization: Domain randomization randomizes simulator aspects that lack essential relevance to improve generalization from low-fidelity simulation to difficult test domains.The approach has roots in robotics work dating back to the Radical Envelope of Noise Hypothesis.
  • Domain randomization: A concurrent study similarly found procedural objects useful for robotic grasping but focused on combining simulation with real grasping data.This paper instead emphasizes generalization to novel objects without real-world training data.
  • Generative models: Autoregressive models factor an N-dimensional distribution into conditional one-dimensional distributions modeled sequentially.Each conditional distribution depends on preceding variables in the chosen ordering.
  • Generative models: Unlike GANs, autoregressive models can directly compute sample likelihoods, which is advantageous when selecting high-likelihood grasps.This property connects the model family to grasp selection rather than only sample generation.

C. Robotic grasping

Robotic grasping research spans analytical and empirical methods, with deep learning increasingly used to estimate grasp quality or control actions. The paper distinguishes its joint learning of grasp scoring and sampling from prior strategies.

  • Analytical and empirical methods: Analytical methods use contact models and 3D shape knowledge to optimize grasp-quality metrics, often under simplified physical assumptions.Examples include resisting external wrenches and constraining object motion.
  • Analytical and empirical methods: Empirical methods maximize grasp-quality metrics through sampling, using simulation, human labels, or self-supervised learning to measure success.These methods evaluate or learn grasp quality rather than relying solely on analytical models.
  • Deep learning for robotic grasping: Deep-learning grasping work varies by how training data are collected and how noisy observations become grasp candidates.Synthetic depth, combined synthetic and real images, and automated data collection are among the surveyed approaches.
  • Deep learning for robotic grasping: Visuomotor policies iteratively refine grasp targets by predicting success probabilities for observations and motor commands.The cited approach uses the cross-entropy method to select commands predicted to maximize success.
  • Deep learning for robotic grasping: A common alternative samples grasps and scores them with a function f(I, g) → s, often optimizing grasp parameters with the cross-entropy method.The paper instead jointly learns a scoring function and sampling distribution.
  • Deep learning for robotic grasping: Autoregressive modeling directly represents a high-dimensional, multimodal grasp distribution rather than relying on coarse sampling followed by separate refinement.The cited grasp representation has 204 or 206 dimensions.

III. METHOD

The method generates and evaluates diverse procedural objects, comparing them with realistic datasets and a ShapeNet-based training baseline. The object pipeline combines random primitives into varied meshes and rescales them to realistic size distributions.

  • Method overview: The method maps one or more scene observations I = {I_j} to a grasp g to attempt.The method section covers data generation, model architecture, and training procedure.
  • Data collection: Experiments use procedurally generated random objects alongside ShapeNet and YCB objects.Figure 3 presents these three object groups from left to right.
  • Data collection: The object-generation pipeline samples 1–15 primitive meshes, randomly scales them, sequentially intersects them, and rescales the result to match real-object sizes.The primitive dataset was built from more than 40,000 ShapeNet meshes.
  • Evaluation: The generated-object procedure is compared against training on rescaled ShapeNet objects.This provides a realistic-object baseline for evaluating procedural training data.

2) Grasp sampling and evaluation:

The model learns a multimodal grasp distribution from depth observations, samples likely grasps autoregressively, and uses a hand-camera evaluator to select one for execution.

  • Sampling and evaluation: Candidate grasps are uniformly sampled from a discretized 4- or 6-dimensional space before invalid grasps are rejected or evaluated in simulation.The 4-dimensional setting applies when attention is restricted to upright grasps.
  • Model architecture: The image representation module maps one or more depth images to an embedding s for grasp planning.Each image is processed by a separate convolutional neural network before the representations are combined into s.
  • Sampling and evaluation: The system samples k high-likelihood grasps with beam search, captures aligned hand-camera observations, scores them, and executes the highest-scoring grasp.The grasp evaluation model maps each aligned image to a scalar success-likelihood score.
  • Model architecture: The system combines a grasp planning module with a grasp evaluation model using depth observations and close-up hand-camera images.The planning module samples likely grasps, while the evaluator estimates each sampled grasp’s success likelihood.
  • Model architecture: The autoregressive module models a probability distribution over grasps by conditioning each grasp dimension gi on preceding dimensions.Each βi outputs a softmax over 20 possible values for the next grasp dimension.

C. Training methodology

Training separately optimizes the grasp planning and evaluation models because hand-camera capture makes the full procedure nondifferentiable.

  • Training methodology: The planning and evaluation models are trained independently because capturing hand-camera images makes the complete evaluation procedure nondifferentiable.The grasp evaluation function is trained with supervised learning from hand-camera images labeled by grasp success.
  • Training methodology: The grasp planning module minimizes the negative log-likelihood of successful grasps conditioned on object observations.The training data comprise object datasets, observations, and sets of successful grasps.
  • Training methodology: The objective is decomposed across grasp dimensions using the autoregressive model’s conditional distributions.Each conditional distribution models the next grasp dimension given the observation and preceding dimensions.
  • Training methodology: Successful-grasp matrices are padded to a common size m = max({mi}), with an indicator identifying valid successful-grasp entries.The indicator prevents arbitrary padding values from contributing as successful grasps.
  • Training methodology: Reusing the observation representation and its gradients across successful grasps increases training speed by more than a factor of 10.The optimization reuses ∇θα(Ii) once for each observation Ii.

IV. EXPERIMENTS

The experiments test generalization from unrealistic random objects, grasp-sampling efficiency, the value of object diversity, and real-world performance.

  • Experimental questions: The experiments ask whether models trained on unrealistic random objects match realistic-object training on novel realistic objects.This is the first of four questions covering generalization, sampling efficiency, object-count effects, and real-world transfer.
  • Experimental questions: The study evaluates how efficiently the autoregressive architecture samples grasps and how important many unique training objects are.These questions assess the proposed sampling method and the relationship between object diversity and grasping performance.
  • Experimental questions: The experiments also measure how well models trained with the proposed methodology work in the real world.The evaluation includes a direct real-world transfer question in addition to simulation-based tests.

A. Experimental setup

The evaluation uses million-scene realistic, random, and mixed datasets, multiple training seeds, ShapeNet and YCB test scenes, and MuJoCo execution with beam-search sampling.

  • Datasets: The study trains on ShapeNet-1M, Random-1M, and ShapeNet-Random-1M datasets containing realistic, procedurally random, or mixed single-object scenes.The mixed dataset contains 500,000 scenes from each of the other two datasets.
  • Datasets: Each object set records 2,000 grasp attempts per object, yielding at least one successful grasp for more than 95% of sampled objects.A small fraction of random training objects are ungraspable with the Fetch gripper.
  • Training setup: Models are trained with Adam at a learning rate of 10^-4, using three random seeds and reporting their average unless otherwise noted.The reported setup averages results across the three seeds.
  • Evaluation scenes: Evaluation uses 300 training and hold-out scenes from ShapeNet-1M and Random-1M, plus 300 scenes from 75 graspable YCB objects.The YCB meshes are capable of being grasped by the robot’s gripper.
  • Execution protocol: In MuJoCo, the system samples k = 20 likely grasps with beam width 20, executes the highest-scoring one, and defines success as lifting the object 30cm.Only one grasp is attempted for each evaluation scene.

B. Performance using randomly generated training data

Training on procedurally generated random objects yielded strong generalization to unseen realistic objects, while the full method outperformed autoregressive-only sampling. Remaining failures concentrated on objects near gripper limits and highly irregular shapes.

  • Generalization to unseen objects: >90% success on previously unseen YCB objects was achieved when training entirely on randomly generated objects.The result held across the synthetic training sets evaluated in Figure 5.
  • Generalization to unseen objects: Training on 1M random objects performed comparably to training on 1M realistic object instances.
  • Comparison with baselines: The full method performed significantly better than sampling only the highest-likelihood grasp from the autoregressive model.The autoregressive-only baseline executes one sampled grasp without evaluation by model f.
  • Failure cases: The learned models failed when grasps collided with near-maximum-size objects, slipped on highly curved objects, or lacked candidates for highly irregular objects.Objects close to the gripper's 10cm maximum size were a recurring boundary case.
  • Failure cases: Additional edge-of-graspability training objects or visual servoing could alleviate failures near the gripper's size limit.The paper identifies these as possible remedies for the primary size-related failures.
  • Sampling efficiency: The top 20 autoregressive grasps contained a successful grasp for more objects than the complete method successfully executed.The gain in valid candidates saturated between 10 and 20 samples, suggesting grasp evaluation may bottleneck overall performance.

D. Effect of amount of training data

With enough training data, models trained on randomly generated objects can match realistic-data models, but realistic distributions perform better with smaller training sets. The physical evaluation transferred synthetic-data training to previously unseen real objects, while sensor and object properties produced failure modes.

  • With enough data, randomly generated-object models perform as well as models trained using realistic data.
  • With smaller training sets, realistic object distributions significantly outperform the unrealistic random-object distribution on the test set.
  • Real-world depth input came from an Intel RealSense D435, with simulated noise modeled using Gamma multiplicative and zero-mean Gaussian additive components.
  • 80% success was achieved on 30 previously unseen YCB objects in physical-robot experiments.The result was 24 successful grasps out of 30.
  • Highly translucent objects caused failures because the RealSense could not provide accurate readings for them.
  • Objects with highly nonuniform densities also caused failures because the training pipeline did not generate components with different densities.

V. CONCLUSION

The paper concludes that non-realistic procedurally generated objects can train models that grasp realistic objects successfully without realistic-object training. It also identifies efficient grasp sampling and broader robotic-task applications as future directions.

  • A model trained entirely on non-realistic procedurally generated objects achieved high success on realistic objects without realistic-object training.
  • The autoregressive architecture found a successful grasp for 96% of objects within its first 20 samples and achieved 92% overall success on the first attempt.
  • Future work includes larger training sets, failed-grasp feedback, visual servoing, and additional sensor modalities such as haptic feedback.
  • Domain randomization may benefit pick-and-place, grasping in clutter, and tool use by enabling random generation of hundreds of thousands or millions of 3D scenes.
Loading 1710.06425v2…