PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, Pieter Abbeel

arXiv:1703.06907v1cs.ROcs.LG

TL;DR

Robotic learning is limited by the reality gap and the difficulty of collecting precise labeled data on real hardware. This paper trains an object detector with domain-randomized simulation and achieves about 1.5 cm real-world accuracy, sufficient for grasping in clutter.

  • Problem

    The reality gap and scarce precise real-world labels limit learning camera-based robotic manipulation, despite labeled data being easy to generate in simulation.

  • Method

    The paper trains a deep neural-network object detector in simulation while randomizing simulated environments to promote generalization to real-world camera data.

  • Results

    Around 1.5 cm accuracy was achieved for geometric-object localization in the real world using only simulated data with simple algorithmically generated textures.

  • Takeaways & Limitations

    A detector trained only in simulation can achieve sufficient real-world accuracy for grasping in clutter.

Abstract

from arXiv · show

Bridging the 'reality gap' that separates simulated robotics from experiments on hardware could accelerate robotic research through improved data availability. This paper explores domain randomization, a simple technique for training models on simulated images that transfer to real images by randomizing rendering in the simulator. With enough variability in the simulator, the real world may appear to the model as just another variation. We focus on the task of object localization, which is a stepping stone to general robotic manipulation skills. We find that it is possible to train a real-world object detector that is accurate to $1.5$cm and robust to distractors and partial occlusions using only data from a simulator with non-realistic random textures. To demonstrate the capabilities of our detectors, we show they can be used to perform grasping in a cluttered environment. To our knowledge, this is the first successful transfer of a deep neural network trained only on simulated RGB images (without pre-training on real images) to the real world for the purpose of robotic control.

I. INTRODUCTION · II. RELATED WORK

The paper investigates domain randomization as a way to bridge simulation-to-real transfer for robotic vision, focusing on object localization as a step toward manipulation. It reports real-world detector accuracy of around 1.5 cm using only simulated images with randomized rendering and no additional training.

  • I. INTRODUCTION: Simulation could make robotic learning faster, more scalable, and lower-cost than collecting data with physical robots.The motivation is strengthened by recent deep reinforcement learning results in games and robotic control.
  • I. INTRODUCTION: Physics discrepancies create a reality gap because system identification is time-consuming and error-prone, while simulators omit effects such as backlash and wear.The supplied passage also identifies nonrigidity and fluid dynamics as unmodeled physical effects.
  • I. INTRODUCTION: Hundreds of thousands of low-fidelity rendered images vary camera positions, lighting, object positions, and non-realistic textures before real-world testing without additional training.This figure passage illustrates the domain-randomization setup and its sim-to-real evaluation protocol.
  • I. INTRODUCTION: Domain randomization exposes models to widely varying simulated environments so they may generalize to the real world with no additional training.The approach randomizes the simulator rather than training on one fixed simulated environment.
  • I. INTRODUCTION: The study targets transfer from low-fidelity simulated camera images, whose high-dimensional inputs are attractive for their low cost and rich data but difficult to process.The passage frames supervised deep neural networks as a tool for learning generalizable representations from pixels.
  • I. INTRODUCTION: Object localization is chosen as a first step toward replacing complex hand-engineered detection pipelines and transferring general-purpose manipulation behaviors.The task is described as detecting an object’s location from pixels.
  • I. INTRODUCTION: 1.5 cm is the reported real-world detector accuracy for a range of geometric objects trained only on simulated data with simple algorithmically generated textures.The result concerns object localization and uses simulated rather than real training data.

A. Object detection and pose estimation for robotics · B. Domain adaptation · C. Bridging the reality gap

The paper situates its approach against established 3D object-pose methods and domain-adaptation strategies, then distinguishes its reality-gap solution by transferring from simulation without real-world adaptation. Unlike methods requiring realistic rendering, calibration, or real-data training, it uses fast, low-quality rendering while preserving coherent 3D scene information.

  • A. Object detection and pose estimation for robotics: Traditional object-detection methods construct or learn 3D object models and match test-image features against them to estimate object pose.Examples include full 3D meshes, 3D metric features, SIFT features, and color co-occurrence histograms.
  • A. Object detection and pose estimation for robotics: Compared with traditional methods, the paper avoids 3D reconstruction through a simple deep-learning pipeline that may scale better to challenging problems.Traditional approaches can use richer sensory data and estimate full 3D pose without assumptions about object placement surfaces.
  • B. Domain adaptation: Domain adaptation commonly transfers models through target-domain retraining, source-target statistical weight adaptation, invariant features, or mappings between domains.Related reinforcement-learning work also studies invariant feature representations.
  • B. Domain adaptation: The paper studies transfer from simulation to the real world without performing domain adaptation.This frames the paper’s central departure from methods that modify models or representations for a target domain.
  • C. Bridging the reality gap: Prior reality-gap methods have pursued realistic simulation, depth rendering, fine-tuning, domain adaptation, or iterative learning control using real-world data.Realistic RGB rendering alone has had limited transfer success, whereas realistic depth simulation can enable reasonable transfer.
  • C. Bridging the reality gap: In contrast, the proposed method requires no additional real-world training and permits low-quality, fast rendering that need not match real textures, lighting, or scene configurations.The method remains compatible with most domain-adaptation techniques.
  • C. Bridging the reality gap: The approach differs from many synthetic-data vision methods by rendering coherent 3D scenes rather than placing realistically textured objects on solid or photographic backgrounds.This design is intended to preserve the 3D spatial information needed for rich physical interactions.
  • C. Bridging the reality gap: Related work has transferred policies from varied 3D scenes and textures to quadrotor flight, whereas this paper targets high-precision tasks without relying on precise camera information or calibration.The cited quadrotor experiments involved collision avoidance in hallways and open spaces.

III. METHOD

The method trains a deep neural-network object detector in simulation using domain randomization. The detector maps a single monocular camera frame to Cartesian coordinates for objects of interest while ignoring distractors.

  • Object detection objective: The detector maps one monocular camera frame to each object’s Cartesian coordinates (x_i, y_i, z_i).The target objects are denoted {s_i}_i.
  • Object detection objective: Scenes may include distractor objects that the network must ignore.
  • Training approach: The approach trains the deep neural network in simulation using domain randomization.The section then specifies the domain-randomization procedure and neural-network architecture.

A. Domain randomization

Domain randomization trains generalization by varying simulated scene, camera, lighting, and image properties so real-world data becomes another domain variation. The setup fixes table height for monocular 2D pose estimation and uses non-photorealistic rendering with randomized textures.

  • A. Domain randomization: The MuJoCo renderer is non-photorealistic, so physically plausible textures and lighting are unnecessary for training.The method’s goal is sufficient simulated variability for real-world generalization rather than realistic image synthesis.
  • A. Domain randomization: Training randomizes distractor number and shape, object positions and textures, environmental textures, camera parameters, lighting, and image noise.The randomized camera variables include position, orientation, and field of view; lighting variables include number, position, orientation, and specular characteristics.
  • A. Domain randomization: The fixed table height converts monocular uncalibrated-camera object localization into a 2D pose estimation task.Image noise type and amount are also randomized during training.
  • A. Domain randomization: Object textures are sampled uniformly from random RGB values, gradients, or checker patterns, withholding target-object color information during training.The detector therefore receives only the objects’ size and shape as target-related visual information.
  • A. Domain randomization: Camera randomization avoids real-world calibration and precise placement by sampling each training camera within a (10 × 5 × 10) cm box around an approximately matched viewpoint.The simulated camera is manually initialized to approximately match the real camera’s viewpoint and field of view.

B. Model architecture and training

The detector uses a modified VGG-16 convolutional architecture to predict object coordinates from downsized webcam images. It is trained with Adam on an L2 position loss, using a reduced learning rate to improve convergence and avoid a center-mapping local optimum.

  • Architecture: 224 × 224 webcam images are input to the network, which predicts the (x, y, z) coordinates of objects of interest.ReLU nonlinearities are used throughout, with max pooling between convolutional-layer groupings.
  • Architecture: The detector is a modified VGG-16 network with standard convolutional layers, 256- and 64-unit fully connected layers, and no dropout.The architecture was chosen for strong performance across computer vision tasks and broad availability of pretrained weights.
  • Training: The detector is trained with stochastic gradient descent on L2 loss between estimated and true object positions using Adam.The loss directly measures the discrepancy between predicted and ground-truth object coordinates.
  • Training: 1e−4 learning rate improved convergence and avoided a local optimum that mapped all objects to the table center.This rate was lower than Adam’s standard 1e−3 learning rate.

IV. EXPERIMENTS · A. Experimental Setup

The experiments trained simulated-image object detectors for eight geometric objects and evaluated their real-world localization and manipulation potential. The setup used rendered tabletop scenes with distractors and labels giving each target’s world-frame center of mass.

  • A. Experimental Setup: Eight object detectors were trained, with each geometric object represented by a mesh for simulator rendering.The experiments used mesh representations for each of the eight objects.
  • A. Experimental Setup: Each training sample included a rendered image of an object and one or more distractors on a simulated tabletop.Distractors were selected from the geometric object set.
  • A. Experimental Setup: The training label specified the target object’s center-of-mass Cartesian coordinates in the world frame.The labels corresponded to the object’s Cartesian center-of-mass location.
  • A. Experimental Setup: The experiments evaluated localization accuracy in the real world, including scenes with distractor objects and partial occlusions.Robustness was assessed under both distractors and partial occlusions.
  • A. Experimental Setup: The study assessed which elements of the approach were critical for transferring detectors from simulation to the real world.This goal focused on identifying the approach components most important for successful transfer.
  • A. Experimental Setup: The experiments determined whether the learned detectors were accurate enough to support robotic manipulation tasks.The setup explicitly connected detector accuracy with manipulation capability.

B. Localization accuracy

Real-world evaluation used 480 webcam images of geometric objects under uncontrolled scene conditions, showing that the detectors localized objects within 1.5 cm on average despite clutter and partial occlusions.

  • Localization accuracy: 480 webcam images tested detectors across varying geometric-object scenes at camera distances of 70 cm to 105 cm.The camera position remained constant, while lighting and surrounding scene content were uncontrolled.
  • Localization accuracy: Each of eight geometric objects had 60 labeled images spanning isolated, distractor-present, and partially occluded conditions.Ground-truth positions were measured by aligning one object per image on a tabletop grid.
  • Localization accuracy: 1.5 cm average localization error was achieved for objects in the real world.This real-world error exceeded the 0.3 cm to 0.5 cm error on simulated training data, indicating over-fitting to simulation.
  • Localization accuracy: The detectors performed well in the presence of clutter and partial occlusions.These conditions were included in the real-world test images alongside images containing the object alone.

C. Ablation study

The ablation study found that transfer was sensitive to training-set scale, texture diversity, distractors, camera randomization, and pre-training, but not meaningfully to random noise. Performance improved with more simulated examples and required substantial texture variation, while distractors were critical for real-world resilience.

  • Overall sensitivity: The method was at least somewhat sensitive to every tested factor except random noise.The evaluated factors included training-image count, unique textures, preprocessing noise, distractors, camera position, and pre-trained weights.
  • Training data: 5,000 training samples yielded relatively accurate real-world detection with pre-trained models, while performance improved up to around 50,000 samples.Figure 4 evaluated sensitivity to the number of simulated training examples for pre-trained and from-scratch models.
  • Pre-training: With large amounts of training data, random initialization achieved nearly the same real-world transfer performance as pre-trained weight initialization.The hypothesis that pre-training was essential for generalization to the real world was false in this setting.
  • Texture diversity: Fewer than 1,000 unique textures caused significant performance degradation, showing that extensive texture randomization was necessary for transfer.The experiment fixed the training set at 10,000 examples while varying unique scene texturizations.
  • Scene randomization: Training with distractors appeared critical for resilience to distractors in real-world scenes.The ablation also found that camera randomization consistently provided a slight accuracy boost, although reasonably high accuracy remained possible without it.

D. Robotics experiments

Robotics experiments showed that detectors trained with domain randomization could localize and grasp objects on a Fetch robot in clutter, under occlusion, and amid previously unseen distractors. The pipeline succeeded in 38 of 40 general trials and 9 of 10 trials involving a Spam can with non-uniform textures.

  • General cluttered-scene grasping: Two consistently accurate detectors were evaluated for picking up detected objects across 20 increasingly cluttered scenes using estimated positions and off-the-shelf motion planning.The evaluation also tested robustness to discrepancies in object distributions between training and testing.
  • General cluttered-scene grasping: 38 out of 40 trials succeeded on a Fetch robot, including highly cluttered scenes with significant target occlusion.The detectors used only target shape and size, not target color, and detected objects placed close to same-colored objects.
  • Non-uniform-texture object grasping: 9 of 10 trials succeeded when a Spam can was tested among previously unseen food-item distractors.Training used geometric distractors, while testing used other YCB food items, and the detector ignored the changed distractors.

V. CONCLUSION

The paper demonstrates that simulation-only object detectors can support real-world grasping in clutter, while identifying improvements needed for more precise and contact-rich manipulation. Domain randomization is presented as a promising approach for transferring increasingly complex simulated robotic behaviors to real robots.

  • V. CONCLUSION: Simulation-only object detectors achieved sufficient real-world accuracy to perform grasping in clutter.Future work targets reliability for contact-rich manipulation and higher-precision tasks.
  • V. CONCLUSION: Accuracy could improve through higher-resolution frames and optimized model architectures.These are listed as future directions for object detectors trained with domain randomization.
  • V. CONCLUSION: The detector estimates object positions, enabling a motion planner to execute grasping motions at those locations.Figure 6 shows two representative executions using vision learned only in simulation.
  • V. CONCLUSION: Further improvements include broader rendering randomization, more data, additional viewpoints, stereo or depth, and domain adaptation.The proposed changes target detector accuracy in simulation-to-real transfer.
  • V. CONCLUSION: Domain randomization may help make complex policies learned through deep reinforcement learning useful on real robots.The conclusion frames this as a promising direction for bridging the reality gap in robotic behaviors.

APPENDIX · A. Randomly generated samples from our method

The appendix presents randomly generated, randomly textured scenes used to train the paper’s object detectors. Figure 7 provides a selection of these training images.

  • A. Randomly generated samples from our method: Figure 7 displays images used during training for the paper’s object detectors.The images are presented as a selection from the training data.
  • A. Randomly generated samples from our method: The appendix includes a selection rather than an exhaustive collection of training images.Both the paragraph and figure caption describe the displayed material as a selection.
  • A. Randomly generated samples from our method: The object detectors detailed in the paper were trained using the displayed images.The paragraph directly connects the images to training for those detectors.
  • A. Randomly generated samples from our method: Figure 7 is the appendix’s visual record of the method’s randomly generated training samples.The figure presents randomly textured scenes used in training.
  • A. Randomly generated samples from our method: The training images consist of randomly textured scenes.The scenes are identified as samples used during the training phase of the method.
  • A. Randomly generated samples from our method: The displayed samples belong to the training phase of the method.The figure caption explicitly identifies the scenes as used during training.
Loading 1703.06907v1…