PaperlayerView exact revision on arXiv

Source-linked AI summary

Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours

Lerrel Pinto, Abhinav Gupta

arXiv:1509.06825v1cs.LGcs.CVcs.RO

TL;DR

Human-labeled grasp datasets are difficult to exhaustively annotate, while prior robot trial-and-error datasets were too small for high-capacity models. The paper collects 50K robot trials over 700 hours, trains an angle-specific CNN classifier, and uses multi-stage hard-negative collection. The resulting system reaches 66% success on novel objects and 79.5% accuracy on the fixed test set.

  • Problem

    Human labeling is difficult because objects have multiple valid grasps, while few-hundred-example trial-and-error datasets can overfit high-capacity models and provide limited evidence of generalization.

  • Method

    The paper collects robot self-supervision at scale, formulates grasp prediction as 18-way binary classification over image patches, and uses staged learning to collect hard negatives.

  • Results

    The method achieves 66% grasp success on novel objects, 79.5% accuracy on the fixed test set, and improves test accuracy from 76.9% to 79.3% after one reinforcement stage.

  • Takeaways & Limitations

    Large-scale robot interaction data and multi-stage learning support grasp prediction that generalizes to unseen objects in real-robot testing.

Abstract

from arXiv · show

Current learning-based robot grasping approaches exploit human-labeled datasets for training the models. However, there are two problems with such a methodology: (a) since each object can be grasped in multiple ways, manually labeling grasp locations is not a trivial task; (b) human labeling is biased by semantics. While there have been attempts to train robots using trial-and-error experiments, the amount of data used in such experiments remains substantially low and hence makes the learner prone to over-fitting. In this paper, we take the leap of increasing the available training data to 40 times more than prior work, leading to a dataset size of 50K data points collected over 700 hours of robot grasping attempts. This allows us to train a Convolutional Neural Network (CNN) for the task of predicting grasp locations without severe overfitting. In our formulation, we recast the regression problem to an 18-way binary classification over image patches. We also present a multi-stage learning approach where a CNN trained in one stage is used to collect hard negatives in subsequent stages. Our experiments clearly show the benefit of using large-scale datasets (and multi-stage training) for the task of grasping. We also compare to several baselines and show state-of-the-art performance on generalization to unseen objects for grasping.

I. INTRODUCTION

The paper replaces difficult, semantically biased human grasp labeling with large-scale robot self-supervision, addressing limited trial-and-error data that can overfit high-capacity models. It collects 50K trials over 700 hours and combines an 18-way CNN formulation with multi-stage hard-negative learning.

  • Motivation: Human labeling is difficult because objects admit multiple grasps, making exhaustive positive and negative annotation impractical.The paper also notes that the largest vision-based grasping dataset had only about 1K isolated-object images.
  • Motivation: Prior trial-and-error approaches used only a few hundred datapoints for CNNs with hundreds of thousands of parameters, risking overfitting and poor generalization.The paper motivates substantially scaling robot interaction data for learning representations that generalize to unseen objects.
  • Contributions: Grasp prediction is formulated as 18-way binary classification over angle bins rather than regression, while staged learning collects hard negatives with the latest model.The staged curriculum alternates learning and data collection to improve the grasping model.
  • Contributions: The experiments report benefits from larger datasets and multi-stage learning, including generalization to grasping unseen objects.The paper compares against baselines, performs ablations, and validates the method with real-robot testing.

II. RELATED WORK

Prior grasping systems use analytical geometry, databases, simulation, visual learning, or small-scale robot experience, but each faces modeling, realism, generalization, annotation, or scalability constraints. This paper instead develops a self-supervising staged curriculum using thousands of trial-and-error runs and a CNN classifier.

  • Analytical and database methods: Analytical approaches require complete object geometry, friction, and mass information, while inferring these properties from cameras is extremely difficult.These requirements make direct 3D reasoning challenging for practical grasp prediction.
  • Analytical and database methods: Grasp databases rank candidates by similarity to stored instances, but they do not generalize well to objects outside the database.Database retrieval therefore remains limited by the coverage of its prior grasp instances.
  • Simulation: Simulated grasping samples and ranks candidates analytically, but simulated environments and analytic metrics may not mirror the highly unstructured real world.The related work raises realism concerns for simulation-based evaluation and learning.
  • Visual learning: Visual-learning methods avoid explicit object modeling, yet human annotation and large-scale training-data collection remain difficult and prevent scalable big-data grasp prediction.The paper identifies these constraints across RGB/RGB-D and synthetic-data approaches.
  • Robot self-supervision: Recent robot trial-and-error methods used only a few hundred runs for high-capacity networks, whereas this paper scales self-supervised curriculum learning to thousands of runs.The approach trains a CNN classifier from collected datapoints and uses staged learning to find hard negatives.

A. Trial and Error Experiments

The robot collects grasping data through repeated randomized interactions in cluttered scenes. Each trial samples a region, grasp point, and gripper angle, executes the grasp, and labels success from force-sensor readings.

  • Data collection: The workspace contains multiple objects placed haphazardly, after which the system executes multiple random trials in succession.The collection setup uses objects with varying graspability on a table with a dull white background.
  • Prediction: The test-time system samples patches at different positions and selects the top graspable location with its corresponding gripper angle.The patch is 1.5 times the gripper size to support graspability and angle prediction.
  • Region sampling: The system samples a random region of interest from a Kinect image after background subtraction reduces trials in empty areas.The mixture-of-Gaussians procedure focuses random sampling near detected objects.
  • Grasp configuration: For each region, the robot samples a grasp point uniformly and chooses a random angle in [0, π] for the symmetric two-fingered gripper.The arm first moves to 25cm above the object before sampling the grasp configuration.
  • Execution and annotation: The robot executes the grasp, raises the object by 20cm, and labels success or failure using gripper force-sensor readings.This produces trial outcomes directly from robot interaction rather than manual grasp-location annotation.

B. Problem Formulation

The paper predicts graspability from image patches centered on candidate grasp points. Because graspability depends on angle and objects admit multiple grasps, the CNN outputs 18 angle-specific binary predictions instead of a single structured regression.

  • Task formulation: The grasp synthesis task seeks a successful configuration (xS, yS, θS) from an object image I.The paper represents a grasp with a rectangle encoding location and gripper angle.
  • Task formulation: Regression is problematic because each object has multiple grasp locations and CNNs perform better at classification than structured-output regression.A two-step graspable/non-graspable classifier is also unsuitable because patch graspability varies with gripper angle.
  • CNN input: The CNN input is a 380x380 image patch centered at the grasp point and resized to 227x227 for the ImageNet-trained AlexNet input.The patch is 1.5 times the projected gripper-fingertip size to include contextual information.
  • CNN output: The network estimates an 18-dimensional likelihood vector for graspability at 0°, 10°, through 170°, forming an 18-way binary classification problem.At test time, sampled image patches produce 18 angle-specific graspability values.

C. Training Approach

The training approach represents grasping with an 18-way binary output over image patches and uses staged data collection to refine the model with targeted experience.

  • Network and loss design: The CNN uses rotated 380x380 patches, AlexNet convolutional layers pretrained on ImageNet, and fully connected layers with 4096 and 1024 neurons.The patches are resized to 227x227 for the ImageNet-trained AlexNet input.
  • Network and loss design: Grasp prediction is formulated as 18 binary angle-bin classifiers rather than one multiclass output, allowing multiple grasp angles for a patch.Only the loss for the executed trial’s angle bin is backpropagated.
  • Staged data collection: The trained network becomes a grasping prior for subsequent robot data collection, where previously seen and novel objects provide corrective and reinforcing experiences.The learned model’s highly ranked patches focus more on important image regions than random patches.
  • Staged data collection: Each later trial evaluates 800 random patches across 18 angle bins and selects grasps by importance sampling over the resulting grasp-ability prior matrix.The matrix has dimensions 800 × 18, with entries given by network activations.
  • Staged data collection: At iteration k, the dataset combines the previous dataset with weighted newly collected data, and the next network is fine-tuned from the preceding model.The importance factor Γ is fixed at 3; later iterations use a learning rate of 0.001 for 5 epochs.

IV. RESULTS

The training data were collected through random grasp trials on cluttered tables containing objects with varying graspability, producing 50K grasp interactions.

  • Training data collection: The training dataset contains objects with varying graspability collected during robot interaction experiments.The dataset was collected over 150 objects.
  • Training data collection: The robot collected data from cluttered tables rather than objects presented in isolation.This collection setting is intended to support generalization to clutter-related tasks.
  • Training data collection: 50K grasp experience interactions were collected through the large-scale data collection and learning approach.The dataset statistics are summarized in Table I.

B. Testing and evaluation setting

Evaluation uses a balanced held-out set of physical robot interactions on unseen objects, enabling fixed comparisons across methods and testing performance in real grasping conditions.

  • Test set: The held-out test set contains 3K physical robot interactions on 15 novel and diverse objects in multiple poses.The test set is balanced by random sampling from the collected robot interactions.
  • Metric: Accuracy is measured as binary classification of whether the object was grasped for a given image patch and executed grasp angle.The prediction target is grasp success or failure.
  • Evaluation rationale: The evaluation preserves identical test data for comparisons while retaining data collected from a real robot.Methods performing well on this test set should work well on the real robot, according to the paper.
  • Results: 79.5% accuracy was achieved by the deep learning approach followed by multi-stage reinforcement on the unseen-object test set.The paper compares this result with baseline methods in Table II.
  • Real-robot evaluation: The paper also evaluates real-robot grasping on isolated objects and demonstrates clearing cluttered objects.These experiments extend evaluation beyond the held-out test set.
  • Baselines: The common-sense heuristic baseline encodes rules including grasping near the patch center and along the smallest object width.Its variants also include rejecting objects that are too thin for the gripper.
  • Baselines: The heuristic’s best parameter sweep reached 62.11% accuracy, substantially below the method’s reported accuracy.The paper attributes the heuristic’s low accuracy to poor performance on cluttered objects.

D. Comparison with learning based baselines

The CNN-based grasping method outperforms the reported kNN and linear SVM baselines, while accuracy improves as training data increases for both seen and unseen objects.

  • The baselines use HoG features because preserving rotational variance is important for grasping.
  • 69.4% maximal accuracy is achieved by optimistic kNN, compared with 73.3% for the linear SVM baseline.
  • Increasing training-set size improves accuracy on both seen and unseen objects.

Effects of data:

More data, pretrained visual features, and staged hard-negative mining each improve grasping accuracy, while aggregating data across stages is important for retaining performance.

  • Effects of data: Accuracy gains from additional data are strongest through about 20K datapoints, after which improvements become small.
  • Effects of pretraining: 76.9% accuracy with pretrained networks exceeds the 64.6% achieved by a network trained from scratch.
  • Effects of multi-staged learning: 79.3% testing accuracy after one reinforcement stage rises from 76.9%, showing the effect of hard negatives; performance reaches 79.5% after three stages.
  • Effects of data aggregation: Without aggregating data, accuracy falls from 76.9% to 72.3% when training uses only the current stage’s data.

F. Robot testing results

The robot evaluates grasping on novel and previously seen objects under varied conditions, using neighborhood-based re-ranking to address arm imprecision and testing clutter removal.

  • Testing setup: The system tests novel objects never previously seen by the robot, as well as previously seen objects under different conditions.
  • Grasp re-ranking: The re-ranking procedure samples 10 neighboring patches, averages their best angle scores, and executes the configuration with the largest resulting score.This is intended to tolerate grasp-execution errors of a few millimeters.
  • Failure cases: Failures include slipping, gripper-size incompatibility, and imprecision during precision grasps.
  • Grasp results: The robot achieves a 66% success rate on 150 novel-object tries and 73% on previously seen objects in different conditions.
  • Clutter removal: Across five attempts, the robot clears a clutter of 10 objects in an average of 26 interactions.

V. CONCLUSION

The paper demonstrates self-supervised grasp learning at substantially larger scale than prior experiments, enabling high-capacity CNN training and generalization to novel objects.

  • The framework collects 50K tries over 700 robot hours, increasing training data 40x relative to traditional experiments using a few hundred examples.
  • The large dataset supports training a high-capacity convolutional network with 18M new parameters after ImageNet pretraining.
  • The grasp network achieves a 66% grasp rate for novel objects and is presented as a step toward larger public robot-interaction datasets.
Loading 1509.06825v1…