Source-linked AI summary
Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection
Sergey Levine, Peter Pastor, Alex Krizhevsky, Deirdre Quillen
TL;DR
Robotic visual feedback control is difficult because conventional approaches often require specified features and camera calibration. This paper learns a CNN grasp-success predictor from monocular images and uses it for continuous gripper servoing. Trained on large-scale, varied grasp data, the method achieves high success across diverse and novel objects while correcting errors through feedback.
Problem
Robotic visual feedback control often requires manually specified features and precise camera-to-end-effector calibration.
Method
A CNN predicts whether task-space gripper motions will produce successful grasps, and continuous servoing repeatedly updates motor commands using those predictions.
Results
The controller achieves high success rates across diverse and novel objects and attains the highest success rates in experiments against open-loop and hand-engineered baselines.
Takeaways & Limitations
Continuous visual feedback lets the system adjust to object motion, perturbations, inaccurate actuation, and calibration discrepancies during grasping.
Takeaways & Limitations
Generalization is limited when test-time robots or environments differ substantially from training, including non-flat settings such as shelves and narrow cubbies.
Abstract
from arXiv · showhide
We describe a learning-based approach to hand-eye coordination for robotic grasping from monocular images. To learn hand-eye coordination for grasping, we trained a large convolutional neural network to predict the probability that task-space motion of the gripper will result in successful grasps, using only monocular camera images and independently of camera calibration or the current robot pose. This requires the network to observe the spatial relationship between the gripper and objects in the scene, thus learning hand-eye coordination. We then use this network to servo the gripper in real time to achieve successful grasps. To train our network, we collected over 800,000 grasp attempts over the course of two months, using between 6 and 14 robotic manipulators at any given time, with differences in camera placement and hardware. Our experimental evaluation demonstrates that our method achieves effective real-time control, can successfully grasp novel objects, and corrects mistakes by continuous servoing.
1. Introduction
The paper addresses the difficulty of incorporating vision into robotic feedback control by learning hand-eye coordination directly from monocular images. A CNN predicts grasp-success probabilities, and continuous servoing uses those predictions to adjust task-space gripper motion without precise camera calibration.
- Motivation: Robotic manipulation commonly relies on advance planning and relatively simple feedback, whereas visual feedback control is difficult to incorporate directly.Visual servoing typically requires manually specified features and calibration of the camera-to-end-effector geometry.
- Approach: The method learns hand-eye coordination for grasping by mapping image pixels directly to task-space gripper motions likely to produce successful grasps.The model uses visual cues to infer the spatial relationship between the gripper and graspable objects.
- Approach: The system combines a CNN grasp-success predictor with continuous servoing that repeatedly updates motor commands.Choosing the best predicted path provides feedback to perturbations, object motion, and inaccurate actuation.
- Data Collection: Over 800,000 grasp attempts were collected using similar but nonidentical manipulators over several months to train the CNN.Variation in wear, interacted objects, and camera pose provided a diverse training dataset.
- Evaluation: The controller achieved high success rates across a wide range of objects and outperformed open-loop and hand-engineered calibrated baselines in the experiments.The evaluation included large, small, hard, soft, deformable, translucent, and novel objects.
2. Related Work
Prior grasping work includes geometric and data-driven methods, while visual servoing generally targets object-relative poses using specified visual features. This approach instead emphasizes self-supervised, continuous hand-eye coordination learned from large-scale grasp attempts.
- Grasping Methods: Grasping methods are broadly categorized as geometrically driven or data-driven.Geometric methods analyze object shape using criteria such as force closure or caging, while data-driven methods learn grasp configurations or finger placement.
- Grasping Methods: Prior data-driven grasp selection methods include human-supervised prediction, offline geometric criteria, and deep-learning-based approaches.Feedback has also been used for force-related grasping criteria.
- Self-Supervised Learning: Compared with prior self-supervised learning of grasp poses, this method observes the gripper and chooses motor commands continuously rather than making open-loop predictions.The prior method predicted optimal grasp orientation for an image patch using data collected by a heuristic grasping system.
- Data Scale: The training set contained over 800,000 grasp attempts, more than an order of magnitude larger than prior direct self-supervised datasets and more than twice the size of prior synthetic-grasp datasets.These comparisons are made with Pinto and Gupta (2015) and Kappler et al. (2015), respectively.
- Visual Servoing: Unlike typical visual servoing, the approach learns grasp-oriented motor commands and uses the gripper-object relationship rather than relying on manually designed visual features.Visual servoing commonly moves a camera or end-effector toward a desired object-relative pose.
- CNN-Based Control: The work also differs from prior CNN applications that mostly address passive perception or simpler control tasks by applying CNN grasp-success prediction to robotic control.Related CNN work includes object recognition, localization, segmentation, reinforcement learning, visual servoing, and simulated robotics.
3. Overview
The system separates grasp-success prediction from continuous control: a CNN evaluates candidate task-space motions from visual input, and a servoing function uses those predictions to control the gripper. Training uses varied robots and camera poses, while grasp candidates are visualized with calibrated projections only for display.
- Prediction Network: The prediction network g(I_t, v_t) takes visual input and a task-space motion command, then outputs the predicted probability of a successful grasp.It must locate the gripper and reason about scene geometry, material properties, and object relationships from monocular images.
- Servoing Function: The servoing function f(I_t) uses the prediction network to continuously control the robot toward a successful grasp.Separating prediction from servoing permits supervised CNN training while the controller optimizes grasp performance.
- Training Data: The training data came from over 800,000 grasp attempts collected across similar but nonidentical robots with varied camera poses and gripper wear.The variation was intended to provide independence from precise camera calibration and support generalization.
- Network Inputs: Figure 2 visualizes sampled target grasp positions with colors encoding predicted success probabilities from green at 1.0 to red at 0.0.The calibrated projection is used only for visualization; the network receives offsets from the current gripper position in the robot frame.
4. Grasping with Convolutional Networks and Continuous Servoing
The method learns a CNN grasp-success predictor from visual observations and task-space motions, then continuously servoes the gripper using predicted success probabilities. Its formulation supports feedback-based correction without camera calibration, while relying on an approximate transitive-dynamics assumption for its reinforcement-learning interpretation.
- 4.1. Grasp Success Prediction with Convolutional Neural Networks: The prediction network estimates whether a task-space motion will produce a successful grasp from the current camera observation.The network must locate the gripper, interpret scene geometry and material properties, and reason about the motion's outcome without camera-frame transformation.
- 4.1. Grasp Success Prediction with Convolutional Neural Networks: Each grasp generates T training samples pairing an image and motion vector with the final grasp-success label.The vector runs from the current pose to the pose eventually reached before closing the gripper.
- 4.1. Grasp Success Prediction with Convolutional Neural Networks: The CNN combines current and pregrasp images with a motor command and outputs grasp-success probability through a sigmoid.The motor command is fused into the convolutional representation before later convolutions, pooling, and fully connected layers.
- 4.2. Continuous Servoing: The servoing mechanism samples candidate motor commands, evaluates them with the predictor, and selects commands expected to maximize grasp success.Sampling also permits workspace, joint-limit, user-location, and gripper-height constraints.
- 4.2. Continuous Servoing: The controller uses stopping and lifting heuristics based on predicted success, including a 90% threshold for closing and a 50% threshold for raising the gripper.These heuristics were reported effective for grasping varied objects in cluttered scenes.
- 4.3. Interpretation as Reinforcement Learning: The reinforcement-learning interpretation treats the predictor as a Q-function for T = 2, while longer-horizon use assumes that motion effects are transitive across intermediate states.That assumption can fail when intermediate motions move objects, but the authors report it as a practical approximation.
5. Large-Scale Data Collection
The authors collected grasp data in parallel across varied robotic hardware and environments. The process produced about 800,000 attempts with minimal human intervention and used diverse objects and two success tests.
- 5.1. Hardware Setup: Camera viewpoints, bin locations, lighting, objects, and gripper wear varied across robots during data collection.These differences produced substantial variation in appearance and geometry across the training setup.
- 5.2. Data Collection: About 800,000 grasp attempts were collected over two months without manual annotation or supervision.Human intervention was limited to replacing bin objects and turning on the system.
- 5.2. Data Collection: Random motor commands succeeded on 10% - 30% of attempts, and the dataset combined random grasps with grasps generated by successively updated networks.The network was updated four times while the number of steps increased from T = 2 to T = 10.
- 5.2. Data Collection: Training objects were common household and office items spanning 4 to 20 cm along their longest axis and were periodically replaced.They were placed in sloped-sided bins to reduce wedging and increase data diversity.
- 5.2. Data Collection: Grasp success was measured using gripper-position readings and a drop test comparing images before and after dropping the object.The drop test supplemented position readings because thin objects were often missed.
6. Experiments
Experiments evaluated continuous servoing on previously unseen, diverse objects under replacement and no-replacement protocols, compared against open-loop, random, and hand-engineered baselines, and examined data scaling and qualitative grasp strategies.
- Experimental setup: The test set comprised previously unseen objects spanning heavy, light, flat, large, small, rigid, soft, and translucent categories.Objects were evaluated in a bin-based grasping setup, including a no-replacement condition.
- Comparisons: The evaluation compared continuous servoing with open-loop, random, and hand-engineered grasping methods.The open-loop method used the same network architecture and training set but required known camera calibration and lacked continuous visual feedback.
- Experimental protocols: 100 replacement attempts tested grasping in clutter, while no-replacement trials reported failure rates over the first 10, 20, and 30 attempts across four repetitions.The replacement protocol allowed grasped objects to be returned to the bin, whereas no-replacement trials removed objects progressively.
- Quantitative results: Continuous servoing exceeded baseline and prior methods in all reported cases and nearly cleared the bin within 30 grasps under no replacement.The bin was cleared after 30 grasps in one of four attempts; the other three had one object left, later cleared in two cases on the 31st attempt.
- Data requirements: Performance continued improving as the training dataset grew, while the authors reported that additional data could further improve accuracy.Table 2 evaluated roughly the first eighth, quarter, and half of the full dataset.
- Qualitative results: The system selected different grasp strategies for soft and hard objects and grasped challenging objects including translucent, awkwardly shaped, and heavy items.Soft objects were grasped by embedding a finger into the center, whereas hard objects were grasped from either side.
7. Discussion and Future Work
The discussion presents continuous, calibration-independent grasping learned from large multi-robot data and highlights both unconventional strategies and limits on generalization beyond the training distribution.
- Discussion: The method combines a deep grasp-success predictor with continuous servoing trained on over 800,000 attempts from 14 manipulators with camera and hardware variation.The authors associate this training setup with invariance to camera calibration and small hardware variations.
- Discussion: The system grasped novel objects and used continuous feedback to correct mistakes and reposition the gripper after perturbations or object movement.The conclusion describes these capabilities as demonstrated experimentally.
- Limitations: Generalization assumes similar training and test distributions and may fail for substantially different robot platforms or non-flat settings such as shelves and narrow cubbies.The authors suggest increasing training-setup diversity as a mitigation and future direction.
- Qualitative strategies: The method discovered distinct strategies for soft and hard objects, including pinching into soft objects rather than placing fingers on opposite sides.The authors observed this behavior for paper tissues and sponges.
- Future work: The controlled laboratory experiments motivate future deployment-oriented study of large-scale data collection across multiple robotic platforms.The broader implication is presented as a long-term prospect rather than a demonstrated real-world result.
A. Servoing Implementation Details
The servoing implementation searches over candidate task-space motions, constrains them to the workspace, and adapts their evaluation to the table-based training setup.
- Inference: Inference uses three cross-entropy-method iterations, sampling 64 grasp directions, retaining the six best, and refitting the Gaussian distribution.The first iteration is centered at the gripper’s current pose.
- Motion constraints: Candidate motions are rejection-sampled to remain within the workspace and avoid rotations exceeding 180° about the vertical axis.These constraints can also impose user-defined grasping restrictions.
- Network inputs: Because the CNN was trained on trajectories ending at table height, grasp directions are projected to that height before network evaluation.The executed motion may nevertheless place the gripper above the table.
- Workspace setup: The prototype workspace dimensions and position were set manually by moving the arm to each workspace corner.The authors note that workspace and table geometry could instead be obtained automatically or supplied by a higher-level mechanism.
B. Determining Grasp Success
Grasp success is inferred using both gripper-state inspection and image subtraction to supplement the limitations of either signal alone.
- Success detection: The system checks whether the gripper fingers close completely and supplements this test with image subtraction after the arm lifts away.The gripper-state test detects large objects effectively but can miss small or thin objects.
C. Details of Hand-Engineered Grasping System Baseline
The hand-engineered baseline uses a depth-sensor perception pipeline with camera-to-arm extrinsic calibration. It computes grasp configurations from accumulated point clouds represented as a segmented 3D graph.
- The baseline replaces monocular imagery with depth sensing and requires extrinsic camera calibration relative to the arm base.
- Its grasp pipeline accumulates depth-sensor point clouds into a voxel map, converts that map into a 3D graph, and segments it.
- The resulting segmented representation is used to compute grasp configurations.