Source-linked AI summary
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward H. Adelson, Sergey Levine
TL;DR
The paper addresses the difficulty of using tactile feedback for interactive robotic grasp adjustment, where vision alone cannot readily react to contact. It learns an end-to-end action-conditional model from raw visuo-tactile data, and reports effective grasp adjustment, reduced force, and interpretable behaviors, while remaining limited to single-step predictions, coarse actions, and uncluttered settings.
Problem
Vision-based grasping cannot easily measure or react to ongoing contact forces, limiting interactive grasp adjustment after contact.
Method
An end-to-end multimodal action-conditional model predicts candidate grasp-adjustment outcomes from raw visual and GelSight tactile inputs.
Results
The approach outperforms baselines across grasp-outcome estimation, efficient adjustment, and force reduction while maintaining competitive performance.
Takeaways & Limitations
Raw visuo-tactile action-conditioned modeling supports efficient grasping, gentler grasps, and useful interpretable grasping behaviors.
Takeaways & Limitations
The model makes only single-step predictions, uses relatively coarse actions, and is not yet extended to realistic cluttered environments.
Abstract
from arXiv · showhide
For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus cannot easily benefit from feedback after initiating contact. In this paper, we investigate how a robot can learn to use tactile information to iteratively and efficiently adjust its grasp. To this end, we propose an end-to-end action-conditional model that learns regrasping policies from raw visuo-tactile data. This model -- a deep, multimodal convolutional network -- predicts the outcome of a candidate grasp adjustment, and then executes a grasp by iteratively selecting the most promising actions. Our approach requires neither calibration of the tactile sensors, nor any analytical modeling of contact forces, thus reducing the engineering effort required to obtain efficient grasping policies. We train our model with data from about 6,450 grasping trials on a two-finger gripper equipped with GelSight high-resolution tactile sensors on each finger. Across extensive experiments, our approach outperforms a variety of baselines at (i) estimating grasp adjustment outcomes, (ii) selecting efficient grasp adjustments for quick grasping, and (iii) reducing the amount of force applied at the fingers, while maintaining competitive performance. Finally, we study the choices made by our model and show that it has successfully acquired useful and interpretable grasping behaviors.
I. INTRODUCTION
The paper studies interactive grasping with tactile feedback, proposing a learned visuo-tactile model that adjusts grasps and supports reduced-force operation without sensor calibration or analytical contact-force models.
- Vision-based grasping cannot readily measure or react to ongoing contact forces, so grasp configurations are typically selected before contact.
- The proposed model learns to probe objects and reactively adjust grasps using raw inputs from vision and paired GelSight sensors.
- The paper introduces a multimodal action-conditional model for grasping with vision and touch, including experiments on novel objects and analysis of learned behaviors.
- The approach permits explicit contact-force constraints, enabling gentler grasps with significantly reduced force.
II. RELATED WORK
The paper positions its contribution against analytic, tactile-classification, and policy-optimization approaches by learning action-conditioned grasp outcomes directly from raw visuo-tactile inputs for efficient regrasping.
- Analytic grasping methods depend on assumptions about object geometry, environments, grippers, and manually defined grasp metrics that may be misspecified.
- Unlike prior tactile work focused on evaluating ongoing grasp stability, this approach selects adjustments intended to produce a stable new grasp.
- The model requires no prior model or transition function because it learns entirely end-to-end from raw inputs.
- Compared with directly optimized policies, the action-conditioned model can use any collected data and change the evaluation objective, including reducing grasping force.
- The practical contribution is efficient grasping with as few regrasps as possible, using predictive models of candidate-action outcomes.
III. HARDWARE SETUP
Experiments use a Sawyer arm and parallel gripper instrumented with two high-resolution GelSight tactile sensors, alongside a frontal Kinect2 RGB camera.
- The hardware consists of a 7-DoF Sawyer arm, a Weiss WSG-50 parallel gripper, and one GelSight sensor on each finger.
- Each GelSight provides 1280x960 raw pixel measurements at 30 Hz over a 24 mm × 18 mm sensing area.
- A frontal Microsoft Kinect2 supplies visual data, while GelSight measures high-resolution contact-surface topography through a deformable illuminated gel.
IV. DEEP VISUO-TACTILE MODELS FOR GRASPING
The grasping problem is formulated as greedy action selection using a model that predicts whether a candidate action will lead to a successful next grasp state.
- The robot greedily selects gripper actions that maximize the probability of successfully grasping an object.
- Given current visuo-tactile observations and an action, the model predicts the probability that the resulting configuration will lead to success at the next time step.
- Actions move the gripper to a new pose relative to its current pose, while grasp outcomes are binary success labels used during training and estimated during evaluation.
A. End-to-End Outcome Prediction
The model learns future grasp success from raw RGB, tactile, and action inputs using an end-to-end multimodal network. Its design combines separate image-processing streams with an action-processing network trained jointly.
- Model formulation: The network predicts future grasp success from the current state s and candidate action a.The state includes raw RGB and two GelSight tactile inputs.
- Network design: Three convolutional stacks process the frontal RGB image and two GelSight tactile images as multimodal state inputs.
- Training: The model is jointly optimized with cross-entropy loss after ImageNet pretraining and tied tactile-network weights.Training uses 18,070 examples for 9,000 iterations with batch size 16.
- Network design: GelSight deformations are emphasized by passing differences between images before and after contact to the tactile networks.
- Network design: The action network uses a two-layer multilayer perceptron with 1024 hidden units per layer to process action and pose vectors.
B. Regrasp Optimization
The learned action-conditional model supports stochastic search over grasp adjustments, using predicted success to select actions. Training data come from self-supervised robot trials with automatically generated tactile-contact labels and augmentation.
- Regrasp optimization: The robot selects the action maximizing predicted future grasp success, a*_t = arg max_a f(s_t, a).
- Regrasp optimization: Stochastic search samples candidate actions, evaluates their predicted success probabilities, and executes the highest-scoring action.Evaluating 5,000 samples takes approximately 0.6 s in the reported experiments.
- Data collection: Self-supervised trials use depth to initialize object-centered gripper poses with randomized position and height settings.
- Data collection: Grasp-success labels are generated automatically by tactile-contact classifiers, with manual labeling for borderline or rare ambiguous cases.
- Data collection: 6,450 grasping trials were collected from more than 65 training objects.
- Data collection: Dataset augmentation adds stationary and intermediate state-action pairs observed while the gripper moves or holds an object.After augmentation, the dataset contains 18,070 examples.
- Data collection: Replacing worn GelSight gels initially degraded performance, but performance stabilized after data were collected with several different gels.
VI. EXPERIMENTAL RESULTS
The experiments evaluate outcome prediction and action dependence using cross-validation on the collected grasp dataset. They compare multimodal, vision-only, and action-ablated model variants.
- Model evaluation: The evaluation asks whether the model predicts future grasp success for novel objects conditional on relative grasp adjustments.
- Model evaluation: Table I reports cross-validation accuracy for models trained with 18,070 data points.
- Model evaluation: K-fold cross-validation with K = 3 partitions the grasp data by object instance for predictive evaluation.
- Model evaluation: The action-ablated Tactile + Vision model represents an unconditional baseline that averages over possible actions.
B. Robot Grasp Evaluation
Robot experiments test iterative regrasping on unseen Easy and Hard objects after training on collected data. The visuo-tactile model reaches 94% accuracy on Easy objects and outperforms comparison models, with larger gains reported for compliant or visually ambiguous objects.
- Robot grasp evaluation: Robot trials begin from randomly sampled end-effector positions and angles without initially closing the gripper, followed by a series of regrasp actions.
- Robot grasp evaluation: 18,070 training examples were used to evaluate performance on 11 previously unseen Easy objects.
- Robot grasp evaluation: 94% accuracy was achieved by the visuo-tactile model on Easy objects, significantly outperforming vision-only and cylinder-fitting models.
- Robot grasp evaluation: On Hard objects, on-policy data collection produced 25,404 additional datapoints for retraining Vision and Tactile+Vision models.
- Robot grasp evaluation: The visuo-tactile model again outperformed the vision-only and cylinder-fitting models on the Hard test set.
- Robot grasp evaluation: The largest performance improvements appeared for compliant objects and objects whose good grasp was visually difficult to determine.
C. Understanding the Learned Visuo-Tactile Model
The model learned context-dependent grasp adjustments involving force, finger height, and downward motion. Its predictions favor lower forces for unstable contacts and downward movement when the fingers hold an object near its top.
- Grasping Force: The model learned that increasing force can improve success for stable contacts but reduce success when large forces displace unstable objects.For a cube contacted only at a corner, the model predicted that lower forces were preferable because large forces could cause slipping.
- Height and Center-of-Mass: Decreasing the fingers’ height generally increases the predicted grasp success rate.
- Height and Center-of-Mass: Downward motion is predicted to help when the fingers hold an object near its top, but not when they hold it by the bottom.The model used GelSight intensity changes to visualize fingertip contact in these examples.
- Height and Center-of-Mass: Almost 50% of successful-policy z-translation actions used the maximum allowed downward motion of 2cm.This indicates a strong learned preference for downward movement in the successful grasps analyzed.
D. Minimum Force Grasp
The action-conditional model can optimize grasps for minimum force rather than only maximum success. On the Green tea cup, this substantially reduced applied force while preserving nearly the same success rate for the tactile-vision policy.
- Minimum Force Grasp: The action-conditional model supports optimizing grasp adjustments for minimum contact force rather than only grasp success.This contrasts with static grasp classification because the model predicts outcomes after modifying grasp strength.
- Minimum Force Grasp: 95/100 versus 94/100 successful grasps were obtained for maximum-success and minimum-force optimization on the Green tea cup.
- Minimum Force Grasp: 10 vs 20 N mean force was measured for minimum-force versus maximum-success optimization on successful Green tea cup grasps.
- Minimum Force Grasp: 76% success was achieved by both force objectives in the Vision-only evaluation, with force reduced from 18 to 6 N under minimum-force optimization.These results were reported for 50 trials and were lower in success than the tactile-vision model.
VII. CONCLUSIONS
The learned action-conditional visuo-tactile model grasps unseen objects with high success and can reduce grasping force while preserving a similar success chance. Its current limitations include single-step predictions, coarse actions, and evaluation outside cluttered environments.
- The model grasps a wide range of unseen objects with a high success rate after training on over 6,000 trials from 65 objects.
- The action-conditioned policy reduces exerted grasping force while preserving a similar chance of success.
- The model makes only single-step predictions and does not perform information-gathering actions.
- Its actions are relatively coarse, limiting delicate pre-grasp manipulation and potential reaction to slippage during lift-off.
- Extending the approach to realistic cluttered environments would require more continuous feedback control, potentially using torque control.
- 10 vs 20 N: the tactile-plus-vision minimum-force policy substantially lowers mean force while maintaining comparable success to the maximum-success policy.