Source-linked AI summary
Learning a visuomotor controller for real world robotic grasping using simulated depth images
Ulrich Viereck, Andreas ten Pas, Kate Saenko, Robert Platt
TL;DR
Robotic grasping must remain reliable despite object motion, perceptual noise, and kinematic inaccuracies, while large-scale real-robot training is costly. The paper learns a closed-loop controller from simulated wrist-mounted depth images using a CNN distance function, and reports stronger robustness than one-shot grasp detection under disturbances, with successful transfer to real sensor images.
Problem
Robotic grasping remains challenging under perceptual noise, object movement, and kinematic inaccuracies, while obtaining large quantities of real-robot training data is often impractical.
Method
A wrist-mounted depth sensor feeds a CNN that learns the distance to the nearest viable grasp for candidate pose offsets, with training data generated in simulation.
Results
The controller outperforms one-shot GPD under kinematic noise, perceptual errors, and object shifts, while simulated-depth training transfers well to real noisy sensor images.
Takeaways & Limitations
Closed-loop visual feedback enables dynamic correction during grasping, extending robustness beyond one-shot grasp pose prediction.
Takeaways & Limitations
The noisy-motion evaluation is not entirely realistic because real manipulator errors typically appear as constant nonlinear displacement offsets rather than Gaussian noise.
Abstract
from arXiv · showhide
We want to build robots that are useful in unstructured real world applications, such as doing work in the household. Grasping in particular is an important skill in this domain, yet it remains a challenge. One of the key hurdles is handling unexpected changes or motion in the objects being grasped and kinematic noise or other errors in the robot. This paper proposes an approach to learning a closed-loop controller for robotic grasping that dynamically guides the gripper to the object. We use a wrist-mounted sensor to acquire depth images in front of the gripper and train a convolutional neural network to learn a distance function to true grasps for grasp configurations over an image. The training sensor data is generated in simulation, a major advantage over previous work that uses real robot experience, which is costly to obtain. Despite being trained in simulation, our approach works well on real noisy sensor images. We compare our controller in simulated and real robot experiments to a strong baseline for grasp pose detection, and find that our approach significantly outperforms the baseline in the presence of kinematic noise, perceptual errors and disturbances of the object during grasping.
1 Introduction
Robust robotic grasping requires closed-loop correction for perceptual noise, object motion, and kinematic inaccuracies. The paper addresses these challenges with wrist-mounted depth sensing and simulation-trained control.
- Existing deep-learning grasping systems reach approximately 85% to 93% success in ideal conditions but must handle realistic perceptual and kinematic errors.
- One-shot grasp detection cannot dynamically correct when objects move or the environment changes during grasping.
- Real-world robotic training data is often impractical to collect at the scale required by supervised learning approaches.
- The proposed system mounts a depth sensor near the end-effector, trains entirely in simulation, and learns a distance-to-nearest-grasp function with a CNN.
- Wrist-mounted sensing fixes the camera-to-gripper configuration, allowing the same controller to grasp objects from different directions.
2 Related Work
Prior grasping methods commonly rely on one-shot perception or manually designed visual features, limiting correction under noise and changing conditions. The paper instead combines learned visual feedback with a wrist-mounted sensor and simulated training.
- Deep grasp perception methods estimate viable grasp poses from noisy sensor data but typically suffer from perceptual errors and inaccurate robot kinematics.
- Visual servoing for grasping has received limited attention, and existing methods require manual feature design or specification.
- Levine et al.’s closed-loop approach required months of training on multiple physical robots, a 17-layer CNN, and a static camera.
- The proposed approach differs by using simulated depth data, a shallower CNN, and a camera mounted to the robot wrist.
3 Approach
The approach learns a distance-to-nearest-grasp function from simulated depth images and uses local descent to guide a wrist-mounted gripper toward viable poses. Restricting candidate actions near the current pose supports stable, iterative corrections.
- 3.1 CNN Model: At each control step, a wrist-mounted depth image and candidate planar offset a = (x,y,θ) are mapped to d(I,a), the distance to the nearest viable grasp.The distance is measured in meters in (x,y,θ) pose space, with angular distance weighted by 0.001 meter/degree.
- 3.1 CNN Model: The CNN uses convolutional and inner-product layers to predict the distance-to-go associated with each candidate grasp pose action.
- 3.2 Generating training data: OpenRAVE generates simulated image-action pairs with distance-to-nearest-grasp labels from collision-free, force-closure grasp candidates.The dataset includes 12.5k scenes and 500k depth images, with 10 sampled offset poses per image.
- 3.3 Controller: For each image, the controller samples candidate actions near the origin and selects the action with minimum predicted distance before moving the gripper.
- 3.3 Controller: Local gradient following can select a closer object even when another object has the globally best predicted grasp pose.
4 Simulation Experiments
Simulation experiments compare CTR with one-shot GPD under ideal and noisy motion, and test whether closed-loop correction reduces perceptual errors. CTR matches GPD without noise but is more resilient to motion noise and single-image errors.
- Comparison setup: The evaluation averages results over 400 trials comparing CTR and GPD with exact or Gaussian-corrupted manipulator motions.The noisy-motion scenario adds uncorrelated zero-mean Gaussian noise to each hand-displacement dimension at every control step.
- Comparison setup: The simulated comparison includes an initial-position condition, CTR with and without noise, and GPD with and without noise.These scenarios measure both baseline grasp localization and resilience to kinematic noise.
- Limitations: The noisy-motion simulation is only an approximation because real manipulator errors typically appear as constant nonlinear displacement offsets rather than Gaussian noise.The authors nevertheless use it to test controller resilience to kinematic errors.
- Kinematic noise: Without motion noise, CTR and GPD move the hand approximately equally well toward a grasp pose.The comparison uses distances in (x,y,θ), excluding z because object distance is assumed recoverable from depth.
- Kinematic noise: With motion noise, CTR achieves substantially smaller distances to detected true grasps than GPD because it can compensate for motion errors.GPD cannot compensate after its one-shot prediction, whereas CTR updates its motion through control iterations.
- Perceptual correction: Closed-loop correction improves CNN predictions even without kinematic noise by compensating for perceptual errors in the initial depth image.The CNN-only condition moves directly to a global prediction, while CNN + CTR adds the controller afterward.
5 Robot Experiments
Robot experiments evaluate CTR against GPD on an UR5 in isolation, dense clutter, and clutter disturbed during grasping. CTR reaches 97.5% success in isolation and strongly outperforms GPD in the dynamic disturbance scenario, while GPD performs better in dense clutter.
- Experimental design: The UR5 experiments cover isolated tabletop objects, dense clutter, and dense clutter whose position shifts after several controller iterations.CTR is compared with GPD, a strong grasp pose detection baseline.
- Experimental setup: The setup uses a 6-DOF UR5, a Robotiq 85 parallel-jaw gripper, and wrist-mounted RealSense SR300 and Structure IO depth sensors.RealSense supplies controller depth images, while Structure IO supplies the point cloud required by GPD.
- Objects in isolation: 97.5% grasp success was achieved by CTR on 40 isolated-object attempts, matching GPD’s success rate in that scenario.CTR failed once, on an upright bottle where it did not converge to the correct pose.
- Dense clutter: CTR achieved 88.9% success on 74 dense-clutter attempts, while GPD achieved 94.8% on 96 attempts.CTR failures mainly involved objects slipping from the fingers, whereas GPD failures included perceptual errors and slipping.
- Changing object positions: After a manual object-pile shift, CTR achieved 77.3% success, compared with 22.5% for GPD.CTR could react to the shift, whereas GPD proceeded using the unchanged original grasp prediction.
6 Discussion
The controller uses gripper-mounted depth feedback and simulated training data to dynamically correct grasp misalignment. It transfers to real sensor images and handles disturbances and errors that challenge one-shot grasping.
- The visuomotor controller uses depth feedback from a gripper-mounted sensor to dynamically correct misalignment during grasping.
- A deep CNN learns a distance function from depth images and grasp-pose actions using simulated sensor data.
- The simulated-depth CNN transfers well to real sensor images after invalid depth readings are corrected.
- The controller reacts to shifting objects and inaccurate gripper placement, while fine-grained adjustments exceed the GPD baseline especially under kinematic noise.
- Figure 7 illustrates cluttered scenes alongside depth images and grasps predicted to have low distance to the nearest grasp.