Source-linked AI summary
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?
Roberto Calandra, Andrew Owens, Manu Upadhyaya, Wenzhen Yuan, Justin Lin, Edward H. Adelson, Sergey Levine
TL;DR
Predicting grasp success from vision alone is difficult because contact forces and compliance are hard to infer indirectly. This paper trains end-to-end deep models on visual, tactile, and combined inputs using more than 9,000 grasps, finding that tactile information improves outcome prediction and grasping performance. The approach evaluates grasps after closure and can require repeated proposals, limiting its practicality.
Problem
The paper asks whether integrating touch sensing improves grasp-outcome prediction beyond purely visual perception.
Method
It trains end-to-end deep neural networks on raw visual and GelSight tactile inputs, comparing vision-only, touch-only, and combined models.
Results
The multimodal visual-tactile model performed best in outcome prediction and achieved a 14% improvement over vision-only grasp selection.
Takeaways & Limitations
Tactile sensing is important for real-world grasping, and deep networks can learn directly from raw visuo-tactile inputs.
Takeaways & Limitations
The rejection-based method evaluates grasps only after execution and may require an arbitrarily high number of attempts under the random proposal mechanism.
Abstract
from arXiv · showhide
A successful grasp requires careful balancing of the contact forces. Deducing whether a particular grasp will be successful from indirect measurements, such as vision, is therefore quite challenging, and direct sensing of contacts through touch sensing provides an appealing avenue toward more successful and consistent robotic grasping. However, in order to fully evaluate the value of touch sensing for grasp outcome prediction, we must understand how touch sensing can influence outcome prediction accuracy when combined with other modalities. Doing so using conventional model-based techniques is exceptionally difficult. In this work, we investigate the question of whether touch sensing aids in predicting grasp outcomes within a multimodal sensing framework that combines vision and touch. To that end, we collected more than 9,000 grasping trials using a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger, and evaluated visuo-tactile deep neural network models to directly predict grasp outcomes from either modality individually, and from both modalities together. Our experimental results indicate that incorporating tactile readings substantially improve grasping performance.
1 Introduction
Tactile sensing offers contact information that complements vision but remains difficult to integrate and model. The paper addresses this gap by evaluating end-to-end models that predict grasp outcomes from vision, touch, or both.
- Vision localizes objects quickly, whereas touch measures compliance and contact force after contact, including when the grasp is difficult to see.
- Tactile sensing has seen limited adoption because of hardware constraints and the difficulty of integrating tactile readings into standard control schemes.
- End-to-end learning has been used for vision-based grasp prediction, but learning directly from tactile readings has received less study.
- The paper evaluates vision-only, touch-only, and combined vision-and-touch deep neural networks for grasp outcome prediction.
- GelSight sensors provide high-resolution images of contact-induced deformation for the tactile component of the evaluation.
2 Related Work
Prior grasping research includes analytic, data-driven, tactile, and multimodal approaches, but the paper targets end-to-end learned prediction from rich visual and tactile inputs. It also provides a controlled test of whether touch improves outcome prediction.
- Analytic grasping models rely on object, environment, and gripper models plus manually defined grasp metrics, making outcomes vulnerable to misspecification and unmodeled effects.
- Tactile sensors have been used to estimate grasp stability, model object dynamics, and detect slip so that grasping force can be adjusted.
- Model-based systems have combined visual and tactile information and improved grasping performance over single-modality inputs.
- The paper presents end-to-end models that process rich visual and tactile inputs to predict whether a grasp will succeed or fail.
- GelSight records high-resolution contact-surface topography as a 2D image that can be processed with convolutional neural networks.
3 Predicting Successful Grasps from Vision and Touch
The paper formulates grasp-success prediction as a multimodal function of RGB and two fingertip GelSight images. Separate convolutional processing followed by feature fusion supports comparisons among visual, tactile, and combined inputs.
- The model outputs the probability of grasp success from a frontal RGB image and left- and right-fingertip GelSight images.
- Experiments compare input configurations including vision-only, tactile-only, combined inputs, and depth substituted for RGB.
- Each modality is processed independently by a convolutional network before its features are concatenated and passed to a fully connected network.
- The model uses ResNet-50 convolutional processing with late fusion of the visual and tactile modalities.
- Training initializes visual and tactile CNNs from ImageNet-pretrained weights and shares parameters between networks for the same modality.
4 Grasping with Vision and Touch
The learned predictor is used to select grasp configurations by evaluating randomly proposed gripper closures before lift-off. A grasp is accepted when its predicted success probability exceeds the experimental threshold.
- The procedure randomly proposes gripper closures near the object, evaluates each prediction, and accepts a pose whose predicted success probability exceeds 0.9.
- Candidate configurations vary end-effector position, gripper angle, and applied gripping force.
- The prediction is evaluated after closing the fingers around the object and before attempting to lift it.
5 Experimental Setting & Data Collection
The study used a robotic setup with two GelSight-equipped gripper fingers and automated collection of varied grasping trials across many objects.
- The setup combined a 7-DoF Sawyer arm, Weiss WSG-50 parallel gripper, two GelSight sensors, and a front-mounted Microsoft Kinect 2.Each GelSight sensor recorded raw pixels at 1280x960 resolution and 30 Hz over a 24x18 mm area.
- Grasp positions, heights, orientations, and gripping forces were randomized to produce stable grasps, slips, and failures.The robot lifted each object and waited two seconds before recording whether it remained in the gripper.
- 9269 grasping trials were collected from 106 unique objects, with labels automatically generated from GelSight contact detection and manually corrected in ambiguous cases.
6 Experimental Results
The experiments compared tactile, visual, and combined models for predicting grasp outcomes and selecting grasps. Tactile sensing improved prediction accuracy, while the combined model performed best overall and improved real-world grasp selection over vision alone.
- Evaluation: Models were evaluated on unseen test data and in a real-world robotic grasping experiment.
- Evaluation: Training and test sets were split by object, and results were averaged across three random object splits.
- Models: The experiments compared tactile, RGB-vision, depth-vision, combined vision+tactile, indentation-feature, and one-sensor variants.The indentation model used features capturing indentation area and force magnitude; one-sensor variants addressed GelSight sensor variability.
- Outcome Prediction Accuracy: The tactile model significantly outperformed the visual model, and the multimodal visual-tactile model achieved the best performance in outcome prediction.
- Outcome Prediction Accuracy: The small dataset limited large expressive models, so subsequent grasp-performance experiments used only end-to-end trained models.The paper leaves further comparison between hand-designed and learned features for future work with larger datasets.
- Grasping Performance: 14% improvement over vision alone was achieved by the multimodal tactile+vision model in grasp selection on 12 previously unseen objects.The visuo-tactile model also avoided empty-grasp lifts that occurred with the purely visual model.
7 Discussion and Future Work
The visuo-tactile model substantially improved grasp-outcome prediction and demonstrated the importance of tactile sensing for real-world robot grasping. However, the proposed rejection-and-search approach remains limited for practical deployment and does not account for post-liftoff events such as slipping.
- More than 9,000 grasps trained deep neural networks to predict outcomes from visual, tactile, and combined inputs.The models used raw visuo-tactile data and required no tactile-sensor characterization or robot or object model.
- The visuo-tactile model substantially improved grasp-outcome prediction over models using only one sensing modality.A real-world active grasp-selection evaluation further tested the different models.
- The rejection mechanism evaluates a grasp only after execution and before liftoff, so the search may require arbitrarily many attempts.The experiments used basic random search for proposing new grasp locations; most good solutions were accepted within 2 or 3 grasps, but some took substantially longer.
- The approach does not explicitly consider post-liftoff events such as slipping, limiting its use of tactile information during grasping.Future work should develop more practical visuo-tactile solutions for real-world robot grasping.
- The results demonstrate tactile sensing’s importance for real-world robot grasping and deep networks’ effectiveness with raw visuo-tactile inputs.The findings motivate studying how tactile sensors can be integrated efficiently to select successful grasp configurations.