PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulation

Kuan Fang, Yunfei Bai, Stefan Hinterstoisser, Silvio Savarese, Mrinal Kalakrishnan

arXiv:1710.06422v2cs.LGcs.AIcs.CVcs.RO

TL;DR

Real-world manipulation data is difficult to collect and label, and instance grasping has no labeled real-world training data in this setting. The paper trains instance-grasp prediction in simulation and adapts it using real and simulated indiscriminate-grasp data; its framework achieves a 60.8% success rate for instance grasping in real-world trials.

  • Problem

    Real-world robot data collection is time-consuming, and instance grasping lacks labeled real-world data because target identification and tracking make labeling challenging.

    Source: §I · p. 1

  • Method

    A multi-task domain-adaptation framework trains instance-grasp prediction in simulation and uses a domain-adversarial loss with real and simulated indiscriminate-grasp data to transfer the model.

    Source: §I · p. 2

  • Results

    60.8% success rate for instance grasping in real-world robot trials.

    Source: C. Evaluation of Instance Grasping · p. 6

  • Takeaways & Limitations

    The method was demonstrated grasping household dishware objects unseen during instance-grasp training in simulation.

    Source: §VI · p. 7

    Mask R-CNN struggles with objects occluded by the gripper and is too computationally expensive to run at every time step, so masks are predicted only from the initial image.

    Source: §VI · p. 7

Abstract

from arXiv · show

Learning-based approaches to robotic manipulation are limited by the scalability of data collection and accessibility of labels. In this paper, we present a multi-task domain adaptation framework for instance grasping in cluttered scenes by utilizing simulated robot experiments. Our neural network takes monocular RGB images and the instance segmentation mask of a specified target object as inputs, and predicts the probability of successfully grasping the specified object for each candidate motor command. The proposed transfer learning framework trains a model for instance grasping in simulation and uses a domain-adversarial loss to transfer the trained model to real robots using indiscriminate grasping data, which is available both in simulation and the real world. We evaluate our model in real-world robot experiments, comparing it with alternative model architectures as well as an indiscriminate grasping baseline.

I. INTRODUCTION

Real-world manipulation data can be costly to collect and label, while simulation alone faces a reality gap. The paper addresses sim-to-real instance grasping with simulated instance labels and a multi-task domain-adaptation framework using indiscriminate grasping data from both domains.

I. INTRODUCTION

  • Real-world data collection is time-consuming, and success-label evaluation can be expensive or infeasible; simulation supplies scalable experiments but does not by itself ensure robust real-world performance.The paper focuses on instance grasping without labeled real-world data, where identifying and tracking the target object complicates labeling.Source: §I · p. 1
  • The robot uses a moving monocular RGB camera and a target mask obtained once at the beginning of grasping, rather than a new mask at every time step.Mask R-CNN produces instance masks, and one object is sampled as the target.Source: §I · p. 2
  • The framework trains instance-grasp prediction in simulation and uses real and simulated indiscriminate-grasp data with domain-adversarial adaptation to transfer the model to real robots.The tasks share model parameters and are trained simultaneously; the network predicts success probability for each candidate motor command from monocular RGB and a target-instance mask.Source: §I · p. 2
  • The authors demonstrate grasping household dishware and generalization to objects unseen during instance-grasp training in simulation.The evaluation uses a real robot platform with a mounted moving monocular RGB camera.Source: §I · p. 2

II. RELATED WORK

Related work spans data-driven grasping, simulation-to-real transfer, domain adaptation, and instance segmentation. This paper distinguishes its approach through instance-level grasping of naturalistic objects using monocular RGB and a target mask.

Robotic grasping

  • Earlier deep-learning grasping methods predict grasp success from visual inputs, while this work targets a specified object instance and generates labeled instance-grasp data in simulation.Related work also includes geometry-based grasping and semantic grasping of an object class.Source: §II · p. 2

Transfer learning from simulation to real robots

  • Prior sim-to-real grasping work uses depth images and calibrated cameras for indiscriminate grasping, whereas this work uses monocular RGB without camera calibration or object geometry.Prior domain-randomization studies address simpler geometries or free-space motions; this work considers naturalistic household objects and unseen-object generalization.Source: §II · p. 2

Domain adaptation

  • Feature-level domain adaptation learns domain-invariant representations; this work applies a multi-task variant to address the lack of labeled real-world instance-grasp data.Prior work also combines feature- and pixel-level adaptation for grasping everyday objects, using a small amount of labeled real-world data [30].Source: §II · pp. 2–3

Instance segmentation and object detection

  • Mask R-CNN [12] segments objects in real-world images using synthetic-image training, allowing the target to be specified without manually labeling objects in the image.Bounding boxes or a few object pixels are noted as possible alternative inputs, left for future work.Source: §II · p. 3

A. End-to-End Grasp Prediction

The grasp predictor scores candidate robot actions from camera images and commands, and the training data come from repeated grasp attempts. Domain-adversarial training uses a classifier to encourage features transferable between simulation and the real world.

A. End-to-End Grasp Prediction

  • At each time step, the network scores sampled actions from the initial and current RGB images and a robot command, and the highest-scoring action is selected.The cross-entropy method runs for three iterations to center action-sampling distributions on high-scoring regions.Source: A. End-to-End Grasp Prediction · p. 3
  • Training data come from repeated grasp attempts, with success labels determined automatically in real-world trials and by object position in simulation.The network is trained with log loss against the success labels.Source: A. End-to-End Grasp Prediction · p. 3

B. Adversarial Loss for Domain Adaptation

  • A domain classifier predicts whether intermediate features come from simulation or the real world, while the network minimizes adversarial loss to penalize domain shift.The classifier is trained by maximizing binomial cross-entropy with respect to its parameters; the network parameters are updated to minimize the adversarial loss.Source: B. Adversarial Loss for Domain… · p. 3
  • The framework jointly trains three grasp-prediction towers on real-world indiscriminate, simulated indiscriminate, and simulated instance-grasping data.The towers share parameters and minimize four losses to predict instance-grasp success with features transferable between simulation and the real world.Source: Figure 3 · p. 4

IV. MULTI-TASK TRANSFER LEARNING FROM SIMULATION FOR INSTANCE GRASPING

Instance grasp prediction estimates the success probability of candidate commands for a specified target object. Training uses simulated instance-grasping data and a single mask sampled at the start of an episode.

A. End-to-End Instance Grasp Prediction

  • The instance network predicts the probability of successfully grasping a specified object for each candidate motor command, using the target mask and RGB images as inputs.The mask specifies the target before grasping begins, and the network fuses it with the images and command to estimate success.Source: A. End-to-End Instance Grasp Prediction · p. 4
  • Training uses simulated instance-grasping data, with hindsight collection producing successful and failed target-mask examples from each successful indiscriminate grasp.About 20% of the resulting training trials are labeled as instance-grasp successes.Source: A. End-to-End Instance Grasp Prediction · p. 4
  • At test time, instance-grasping outcomes are evaluated manually by human annotators.Source: A. End-to-End Instance Grasp Prediction · p. 4
  • The target mask is sampled from an instance segmentation of the initial image and remains the network's mask input as the camera viewpoint changes during arm motion.Although it no longer aligns with the target in later images, the mask still provides location information when fused with RGB input.Source: A. End-to-End Instance Grasp Prediction · p. 4

B. Multi-Task Domain Adaptation

The multi-task framework shares a network across three grasping domains and trains the towers jointly. Indiscriminate-grasping data support feature learning in both domains, while adversarial training encourages simulation-to-real transfer.

B. Multi-Task Domain Adaptation

  • Three shared-parameter towers use real-world indiscriminate, simulated indiscriminate, and simulated instance-grasping data to train the framework jointly.The instance tower predicts simulated instance-grasp success, while the two indiscriminate towers learn grasp-prediction features from their respective domains.Source: B. Multi-Task Domain Adaptation · pp. 4–5
  • The prediction network takes initial and current images, a candidate motor command, and a target mask, and outputs instance-grasp success probability.Source: Figure 4 · p. 5
  • The objective combines instance-grasp, two indiscriminate-grasp, and adversarial losses, with the adversarial term encouraging features transferable between simulation and the real world.The weights α, β, and λ scale loss terms.Source: B. Multi-Task Domain Adaptation · p. 5
  • All towers use the same instance-grasp network; the indiscriminate towers receive a constant mask marking every pixel as a candidate object.This mask interpretation is consistent across indiscriminate and instance grasping, allowing the tasks to use the same input modalities without extra layers for simulated instance data.Source: B. Multi-Task Domain Adaptation · p. 5
  • The Mask R-CNN training example combines a real background image with rendered objects and overlays ground-truth bounding boxes and segmentation masks.Source: Figure 5 · p. 5

C. Neural Network Architecture

The network uses separate image and mask feature streams, then combines their features with the motor-command representation to predict instance-grasp success. Its normalization layers are selected to avoid cross-domain batch-normalization issues during testing.

C. Neural Network Architecture

  • A separate convolutional stream maps the target mask to features that are concatenated with RGB features before fully connected layers predict grasp success.The input images and masks are cropped to 472×472, and the motor command has five dimensions: three for translation and two for orientation encoding.Source: C. Neural Network Architecture · p. 5
  • The network replaces batch normalization with instance normalization in convolutional layers and layer normalization in fully connected layers.The authors cite inconsistent batch-normalization behavior across training and testing domains as the reason for removing it; the replacements are described as supporting consistent behavior and fast convergence.Source: C. Neural Network Architecture · p. 5
  • The simulation uses 130 scanned 3D models of real-world dishware, with object textures rendered in simulation.Examples include bowls, mugs, cups, and teapots.Source: Figure 6 · p. 6

A. Instance Segmentation Prediction with Mask R-CNN

The evaluation combines Mask R-CNN instance segmentation with simulated and real-world indiscriminate-grasp data, then tests instance grasping on real robots. The framework achieved a 60.8% instance-grasp success rate in those trials.

A. Instance Segmentation Prediction with Mask R-CNN

  • Mask R-CNN, trained on synthetic images, segments object instances in clutter despite occlusion not being explicitly covered in its training set.The synthetic training images combine real background images with rendered objects and cover the pose space of interest.Source: A. Instance Segmentation Prediction… · p. 6

B. Data Collection and Training

  • Data collection produced 100,000 real-world indiscriminate trials and 1 million simulated trials for dishware grasping.Real data came from nine Jaco arms and around 100 objects; simulation used 130 scanned models and episodes with 1–6 objects. Training ran for 300,000 iterations, with adversarial loss added after 50,000.Source: B. Data Collection and Training · p. 6

C. Evaluation of Instance Grasping

D. Ablative Analysis

The ablations compare the proposed framework with indiscriminate grasping and alternative domain-adaptation and network designs. The proposed method achieved the highest instance-grasp success rate among the four methods in the real-world comparison.

D. Ablative Analysis

  • Indiscriminate grasping succeeded in 81.2% of trials, but only 13.6% grasped the sampled target object.The indiscriminate policy does not distinguish objects, unlike instance grasp prediction, which targets a specified object.Source: D. Ablative Analysis · p. 7
  • The two-tower adaptation model achieved 34.8% instance-grasp success, while the proposed framework transferred the policy better and had the highest success rate among four methods.The two-tower setup used real-world indiscriminate and simulated instance-grasping data, but not simulated indiscriminate-grasping data.Source: D. Ablative Analysis · p. 7, Table I · p. 7
  • The alternative network with new fully connected layers for instance grasping achieved 58.4% success, and the authors report gains from using constant masks with shared architectures.This alternative shared only the RGB stream's convolutional parameters between the instance and indiscriminate networks.Source: D. Ablative Analysis · p. 7

VI. CONCLUSION

The paper presents multi-task domain adaptation for instance grasping, using simulated labeled instance data and indiscriminate-grasping data from simulation and the real world. It demonstrates real-robot grasping of dishware unseen during simulated instance training and identifies constraints around segmentation masks.

VI. CONCLUSION

  • The framework trains instance grasping in simulation and uses domain-adversarial loss with indiscriminate-grasping data from simulation and the real world for transfer.The network takes monocular RGB images and an instance mask, then predicts grasp success probability for each candidate motor command.Source: §VI · p. 7
  • The real-robot demonstrations used household dishware unseen during simulated instance training, and the authors report outperforming alternative architectures and an indiscriminate-grasping baseline.Source: §VI · p. 7
  • Mask R-CNN performs poorly when the robot gripper occludes objects, and computational expense limits segmentation to the initial RGB image.The authors note that updating masks at later time steps could further improve instance-grasping performance.Source: §VI · p. 7
  • The method uses segmentation masks to specify targets, while bounding boxes and user-specified pixels are alternatives and other tasks are potential extensions.Source: §VI · p. 7
Loading 1710.06422v2…