Source-linked AI summary
Planning Multi-Fingered Grasps as Probabilistic Inference in a Learned Deep Network
Qingkai Lu, Kautilya Chenna, Balakumar Sundaralingam, Tucker Hermans
TL;DR
Multi-fingered grasp planning faces challenges in representing grasp configurations and avoiding reliance on external planners. This paper optimizes grasp configurations directly within a learned network that predicts success probability. The approach generally achieves higher grasp success rates than sampling-based neural-network grasp planners and runs fast enough for robotic deployment.
Problem
Deep-learning approaches have focused relatively little on multi-fingered grasping, with grasp-configuration representation and reliance on external candidate-grasp planners identified as key difficulties.
Source: §1 · p. 2Method
A CNN predicts grasp-success probability from object visual information and grasp configuration, and constrained gradient ascent optimizes the configuration within the network.
Source: §1 · p. 2Results
The method generally achieves higher grasp success rates than sampling-based approaches used for neural-network grasping and runs fast enough for deployment in a robotic system.
Source: §5 · p. 14Takeaways & Limitations
The learned network can plan directly over grasp configurations without an external planner supplying candidate grasps.
Source: §1 · p. 2The formulation leaves priors over grasp parameters, collision avoidance, full arm kinematics, and tactile or haptic inputs for future extensions.
Source: §2 · p. 4
Abstract
from arXiv · showhide
We propose a novel approach to multi-fingered grasp planning leveraging learned deep neural network models. We train a convolutional neural network to predict grasp success as a function of both visual information of an object and grasp configuration. We can then formulate grasp planning as inferring the grasp configuration which maximizes the probability of grasp success. We efficiently perform this inference using a gradient-ascent optimization inside the neural network using the backpropagation algorithm. Our work is the first to directly plan high quality multifingered grasps in configuration space using a deep neural network without the need of an external planner. We validate our inference method performing both multifinger and two-finger grasps on real robots. Our experimental results show that our planning method outperforms existing planning methods for neural networks; while offering several other benefits including being data-efficient in learning and fast enough to be deployed in real robotic applications.
1 Introduction and Motivation
The paper addresses limited deep-learning work on multi-fingered grasping and the difficulties of representing grasp configurations and relying on external planners. It proposes planning directly with a learned grasp-success network and reports data-efficient learning, multiple high-quality grasps, and real-robot validation.
1 Introduction and Motivation
- Deep-learning methods had shown strong performance for parallel-jaw grippers, while relatively little work addressed multi-fingered grasping and its configuration-representation and external-planner challenges.The paper identifies those two challenges as restricting deep learning for multi-fingered grasping.Source: §1 · pp. 1–2
- The network takes an object image patch and grasp parameters as input, then constrained gradient ascent searches for configurations with higher predicted success probability.Joint angles are kept within their limits during inference.Source: §1 · p. 2
- Different initial configurations can produce multiple high-quality grasps, and classification-based learning is more data-efficient than direct grasp regression.The authors note that multiple grasps can let a robot select one suited to its current task, while classification can use negative grasp examples.Source: §1 · pp. 2–3
- The paper proposes two CNN architectures, provides a simulated multi-finger grasp dataset, and validates inference for multi-finger and two-finger planning on real robots.The dataset is described as more realistic than existing simulation datasets.Source: §1 · p. 3
- The method directly optimizes grasp configurations inside a learned network, rather than relying on an external planner to generate candidate grasps.The authors describe this as the first work to directly optimize grasp configurations inside a deep network.Source: §1 · pp. 2–3
2 Grasp Planning as Probabilistic Inference
The paper formulates grasp planning as finding a hand pre-shape that can successfully grasp and lift an isolated object using sensor information. It optimizes the grasp-success probability predicted by a neural network, using constrained gradient ascent over grasp parameters.
2 Grasp Planning as Probabilistic Inference
- Grasp planning seeks a hand pre-shape—joint angles and palm pose—that can grasp and lift an isolated object without dropping it, using sensor readings without explicit object knowledge.The robot moves to the planned pose and executes a controller to close the hand.Source: §2 · p. 3
- The planner searches for the grasp configuration that maximizes the neural network's predicted probability of success given sensor readings and hand configuration.The network classifier has a logistic output trained to predict grasp success as a Bernoulli distribution.Source: §2 · pp. 3–4
- Backpropagation computes gradients with respect to grasp parameters, and projected updates with a backtracking line search continue until convergence.The algorithm takes an object image and initial grasp configuration; joint limits and reachable workspace are encoded as constraints.Source: §2 · p. 4
- The formulation does not consider priors over grasp parameters, collision-avoidance or full arm-kinematics constraints, or tactile and haptic inputs.The authors identify these as possible extensions to the optimization or network inputs.Source: §2 · p. 4
3 Deep Networks for Multi-fingered Grasp Learning
The section presents two CNN architectures that predict grasp success from an object’s RGBD representation and grasp configuration. It also describes their training procedure and the gradients used to support inference when image patches depend on the grasp configuration.
3 Deep Networks for Multi-fingered Grasp Learning
- Both networks predict grasp-success probability from an object image and grasp configuration, using an 8-channel representation of RGB, depth, surface normals, and curvature.The channels are RGB (3), depth (1), normals (3), and curvature (1).Source: §3 · p. 4
3.1 A Multi-channel Grasp Configuration and Image CNN
- The config-CNN combines convolutional image-patch features with grasp-configuration features tiled across a pooled image feature map, then predicts grasp success.The configuration features and image features pass through further convolutional, pooling, and fully connected layers before a logistic-regression output.Source: §3.1 · pp. 4–5
3.2 An Image Patch Based CNN for Grasping
- The patches-CNN extracts palm and fingertip image patches for a grasp, processes each patch, and concatenates their features before predicting success.The architecture is shown for a Baxter parallel-jaw gripper and can be extended to a multi-finger hand.Source: §3.2 · p. 5
3.3 Training
- The classifiers are trained with cross-entropy and stochastic gradient descent using Adam, with dropout and positive-grasp oversampling for multi-finger training.For two-finger training, the authors instead mirror each grasp patch and do not oversample.Source: §3.3 · p. 6
3.4 Computing Gradients for Image Patches in Inference
- Inference accounts for configuration-dependent image patches by combining backpropagation with finite differences to estimate patch gradients with respect to hand configuration.The patch-gradient terms cover the fingers and palm; the config-CNN is decomposed into image-patch and configuration branches feeding a combined network.Source: §3.4 · pp. 6–7
4 Experimental Validation
The evaluation uses simulated multi-finger grasp data and tests inference for multi-finger grasps in simulation and on a real robot, then applies it to two-finger grasps on Baxter. Across these experiments, the authors compare inference with heuristic and learned-planning alternatives.
4.1 Multi-finger Grasping Data Collection
- The simulated dataset contains 1,507 trials across 125 Bigbird objects, with 159 successful grasps.Success was labeled when the robot lifted an object 0.15 m without dropping it.Source: §4.1 · p. 8
4.2 Grasp Classification CNN Evaluation
- The ROC evaluation reported AUC means of 0.83 for seen objects and 0.82 for unseen objects, versus 0.5 for random guessing.The patches-CNN predicted every test grasp as failure; the config-CNN’s comparative success supports the value of directly providing grasp configuration parameters.Source: §4.2 · pp. 8–9, Figure 4 · p. 9
4.3 Multi-finger Grasp Inference in Simulation
- Inference significantly outperformed alternative approaches for both seen and unseen objects in simulation.Gradient-based optimization also refined failing initial grasps into successful ones, including a side grasp on an unseen spray-adhesive object.Source: §4.3 · p. 11, Figure 5 · p. 11
4.4 Real Robot Multi-finger Grasp Inference
- Real-robot multi-finger inference outperformed the comparison methods on all tested objects.Pringles was present in training as a different flavor; the other tested objects were unseen.Source: §4.4 · p. 12, Figure 7 · p. 13
4.5 Experiments for Two-finger Gripper Grasp Inference
- 84% average success rate across two-finger objects was achieved with four inference initializations, comparable to.The network predicted at least 96% confidence, while grasps succeeded 84% of the time.Source: §4.5 · p. 14
5 Conclusions
The planner generally achieves higher grasp success rates than sampling-based approaches used with neural networks while remaining fast enough for deployed robotic systems. The authors also identify room to improve multi-finger results and describe directions for better calibration and broader grasp variety.
5 Conclusions
- The planner generally achieves higher grasp success rates than sampling-based approaches used for grasping with neural networks, while remaining fast enough for deployment.Source: §5 · p. 14
- CNN classification of grasp success allows more data-efficient learning than directly predicting grasps as neural-network outputs.Source: §5 · p. 14
- Multi-finger grasping results leave room for significant improvement, and improvements to the underlying classifier would immediately benefit the planner.Source: §5 · p. 14
- The authors are examining priors for better-calibrated success probabilities and wish to explore active data collection for greater efficiency and wider grasp variety.Source: §5 · p. 15