Source-linked AI summary
Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations
Francois R. Hogan, Maria Bauza, Oleguer Canal, Elliott Donlon, Alberto Rodriguez
TL;DR
Robotic grasping still lacks tactile reasoning, and open-loop execution is limiting in settings such as occlusions. This paper learns a tactile grasp-quality metric and searches simulated local tactile transformations to choose regrasp adjustments. The policy reports a 70% average relative increase in grasping accuracy on 8 test objects.
Problem
Robotic grasping has long lacked tactile reasoning, and open-loop execution is especially limiting in occluded settings.
Source: §I · p. 1Method
The approach learns a tactile grasp-quality metric from data and uses it to select local regrasp adjustments by simulating tactile transformations.
Source: §I · pp. 1–2Results
70% average relative increase in grasping accuracy is reported on a test set of 8 objects.
Source: §I · p. 2Takeaways & Limitations
The results support using tactile feedback to improve grasping performance through local adjustments.
Source: §IX · p. 7The policy considers only local regrasps; future work would account for object geometry and more global tactile transformations.
Source: §IX · p. 8
Abstract
from arXiv · showhide
This paper presents a novel regrasp control policy that makes use of tactile sensing to plan local grasp adjustments. Our approach determines regrasp actions by virtually searching for local transformations of tactile measurements that improve the quality of the grasp. First, we construct a tactile-based grasp quality metric using a deep convolutional neural network trained on over 2800 grasps. The quality of each grasp, a continuous value between 0 and 1, is determined experimentally by measuring its resistance to external perturbations. Second, we simulate the tactile imprints associated with robot motions relative to the initial grasp by performing rigid-body transformations of the given tactile measurements. The newly generated tactile imprints are evaluated with the learned grasp quality network and the regrasp action is chosen to maximize the grasp quality. Results show that the grasp quality network can predict the outcome of grasps with an average accuracy of 85% on known objects and 75% on a cross validation set of 12 objects. The regrasp control policy improves the success rate of grasp actions by an average relative increase of 70% on a test set of 8 objects.
I. INTRODUCTION
The paper addresses the limits of open-loop grasping by using tactile information to assess and adjust grasps locally. It combines a learned grasp-quality metric with simulated tactile transformations to select adjustments.
I. INTRODUCTION
- The proposed system uses tactile information to assess grasp quality in real time, predict failures, and search for quality-improving local adjustments.Source: §I · p. 1
- Robotic grasping remains limited by a lack of tactile reasoning, while open-loop execution is especially limiting when the scene is occluded.The paper motivates tactile adjustment with the example of a human securing a grasp without visual information.Source: §I · p. 1, Figure 1 · p. 1
- The regrasp policy simulates local tactile transformations and uses the grasp-quality metric to select adjustments expected to better secure the object.The metric evaluates how likely a grasp is to resist external forces emulated by shaking the gripper, on a continuous scale from 0 to 1.Source: §I · p. 1
- The tactile grasp-quality metric is learned from more than 2,800 grasps and is tailored to the geometry, compliance, friction, and sensing capabilities of the gripper.The authors describe the approach as a hybrid of learning the metric from data and planning with that model.Source: §I · pp. 1–2
- 70% average relative increase in grasping accuracy on a test set of 8 objects is reported for the model-based regrasp policy.The policy is guided by the data-driven grasp-quality metric.Source: §I · p. 2
II. RELATED WORK
Prior grasp planning includes geometric and image-based methods, as well as closed-loop approaches using visual and tactile sensing. This work is situated among tactile methods while focusing on pure tactile feedback from local contact information.
II. RELATED WORK
- Classical grasp planning seeks geometric closure, but its success depends on accurate object-pose estimates and its metrics can weakly predict physical grasp quality.Source: §II · p. 2
- Object-agnostic point-cloud and RGBD methods can handle varied objects and clutter, but are often open-loop and cannot update motion after the initial grasp location is set.Perception errors, clutter, and unaccounted object motion can produce unexpected contacts or imperfect grasp locations.Source: §II · p. 2
- Closed-loop grasping has used RGB video, RGBD with infrared sensing, and simulated depth-image policies, while tactile sensing supports reactive control based on contact.Source: §II · p. 2
- Earlier tactile work includes predicting grasp success, learning adjustments for cylindrical objects, and adapting trajectories; this paper focuses on pure tactile feedback from local contact information.Source: §II · p. 2
III. APPROACH
The approach learns a tactile-based grasp-quality model and uses it to plan reactive local adjustments after initial contact. It evaluates the resulting regrasp by shaking the gripper.
III. APPROACH
- A data-driven grasp-quality model captures physical properties of the gripper and tactile sensor, then guides regrasp planning instead of relying on classical grasp metrics.Source: §III · p. 3
- The system starts from an antipodal grasp affordance computed from a perceived point cloud, then perturbs the proposal to simulate disturbances.The baseline affordance planner can be any grasp planner; this work uses the MIT-Princeton ARC 2017 algorithm [4].Source: §III · p. 3
- After closing on the object, the robot records tactile imprints, plans and performs a local adjustment from them, then shakes the gripper to evaluate the regrasp.Source: §III · p. 3
IV. ROBOTIC SYSTEM
The robotic platform combines an industrial arm, parallel-jaw gripper, tactile fingers, cameras, and load cells. GelSlim sensors capture high-resolution images of contact-surface geometry through membrane deformation.
IV. ROBOTIC SYSTEM
- The platform uses a 6DOF industrial arm, a parallel-jaw gripper with two image-based tactile fingers, RGBD cameras, and load cells beneath each storage bin.The load cells detect contact between the end-effector and bin, supporting safe autonomous data collection with a position-controlled arm.Source: §IV · p. 3
- Each GelSlim sensor uses an internal camera to capture membrane deformation when an object presses against it, rendering high-resolution contact-surface geometry.Source: §IV · pp. 3–4
V. DATA COLLECTION
The data-collection process creates varied grasp examples and labels them by how long they resist shaking. It combines vision-based initial grasp proposals, deliberately perturbed grasps, and physical evaluation.
V. DATA COLLECTION
- Initial grasp proposals come from antipodal regions in point clouds, then lateral noise tailored to object dimensions creates more challenging grasps.The noise encourages grasps near object borders while aiming to maintain a similar ratio of successes and failures.Source: §V · p. 4
- Grasp quality is scored by shaking the gripper after a 30 N grasp: failures score from 0 to 0.5, while no detected failure scores 1.The score scales with resistance time; the shaking duration is four seconds.Source: §V · p. 4
- The dataset covers grasps on 20 objects spanning varied shapes, textures, and hardness, producing diverse tactile imprints.The collection uses the shaking-based quality scores as labels.Source: §V · p. 4
VI. TACTILE-BASED GRASP QUALITY
The paper learns grasp quality from tactile images, then uses that metric to rank simulated local regrasp motions. The policy relies on translated tactile imprints and assumes the object pose remains stable and the new imprint is a rigid transformation of the original.
VI. TACTILE-BASED GRASP QUALITY
- Two tactile images captured before lifting are processed by a shared, ImageNet-pretrained ResNet50, whose concatenated features feed a sigmoid quality output.The network is trained with Adam and cross-entropy loss.Source: Figure 4 · p. 4, §VI · p. 4
- The tactile-only grasp metric predicts resistance to external perturbations from labeled tactile measurements, enabling real-time assessment and planning of regrasp actions.It is learned as a regressor using a deep convolutional network.Source: §VI · p. 4
VII. GRASP ADJUSTMENT POLICY
- The policy captures the initial imprint, simulates candidate regrasp imprints, scores them with the learned metric, and performs the highest-ranked motion.Candidate motions are in the parallel-jaw gripper plane.Source: §VII · p. 4, Figure 5 · p. 5
- Candidate imprints are generated by rigid-body transformation, with this implementation considering translations and mirroring newly introduced pixels.The mirror is applied iteratively to fill the image; the approach is best suited to objects wider than the tactile sensor.Source: §VII · p. 5
- The policy uses local contact geometry without requiring object identity or global geometry, but assumes stable object pose and a rigidly transformed post-regrasp imprint.The rigid-imprint assumption is more likely to be violated because regrasping can introduce an unknown contact area; the authors report good performance even when assumptions are violated.Source: §VII · p. 5
VIII. EXPERIMENTAL RESULTS
The experiments assess the learned grasp-quality metric through cross-validation on 12 objects and evaluate the regrasp policy on eight new objects.
VIII. EXPERIMENTAL RESULTS
- The experimental section evaluates the learned grasp-quality metric and the regrasp policy.The metric is assessed through cross-validation on 12 objects, while policy performance is analyzed on eight new objects.Source: §VIII · p. 5
- The grasp-quality metric is evaluated using cross-validation on 12 objects.Source: §VIII · p. 5
- The regrasp policy is analyzed on eight new objects.Source: §VIII · p. 5
A. Grasp quality metric
The learned metric achieves 85% testing accuracy on known objects and 75% average accuracy on unknown objects in cross-validation. Its predictions are more challenging for oddly shaped or heavy objects, while tactile information often suffices to infer grasp quality.
A. Grasp quality metric
- 85% testing accuracy was achieved for grasps from known objects, using a 0.5 decision boundary for successful grasps.The training accuracy was 90%.Source: A. Grasp quality metric · p. 5
- 75% average accuracy on unknown objects suggests the learned metric generalizes across objects, although performance varies by object.The evaluation withholds one object for validation and trains on the remaining 11.Source: A. Grasp quality metric · p. 5
- Oddly shaped and heavy objects are most challenging to predict; small grasp-position differences can change quality for heavy objects despite similar imprints.Source: A. Grasp quality metric · pp. 5–6
- Examples of high- and low-quality grasps indicate that tactile information often suffices to infer grasp quality.Source: A. Grasp quality metric · p. 6
B. Grasp adjustment based on tactile information
The policy searches for tactile-informed grasp adjustments in settings where vision alone may be insufficient. On eight challenging new objects, it improves grasping accuracy using small motions normal to the gripper.
B. Grasp adjustment based on tactile information
- The policy is evaluated on eight new objects chosen for heavy weight or complex geometry, with gripper motions from 0 to 3 cm in 0.6 cm steps.Motions are along the gripper’s normal direction.Source: B. Grasp adjustment based on tactile… · p. 6
- Relative accuracy increases by an average of 70%; overall accuracy rises by at least 14% and approximately 24% on average with grasp adjustment.The evaluation compares grasps with and without adjustments.Source: B. Grasp adjustment based on tactile… · p. 6
- In visual occlusion or large clutter, tactile information can help a grasping system decide whether to keep a grasp or regrasp.Figure 7 illustrates cases in which the policy improves initial grasp affordances.Source: B. Grasp adjustment based on tactile… · p. 6
C. Comparison with heuristic baseline: centroid-centering
Centroid-centering improves open-loop accuracy on two objects, but the grasp-quality-based policy performs better. The paper suggests that fine tactile contact details may explain this advantage.
C. Comparison with heuristic baseline: centroid-centering
- For the glucose bottle and tomato can, centroid-centering reaches 65% and 53% accuracy versus open-loop’s 58% and 36%, below the policy’s 83% and 72%.Each accuracy is averaged over 100 grasps.Source: C. Comparison with heuristic baseline:… · p. 6
- The learned metric recovers the distribution of grasp-quality labels and discriminates scores better than a simple 0-1 classification task.The qualitative examples show predicted and labeled qualities for test-set grasps.Source: Figure 6 · p. 6
- The authors hypothesize that the policy’s advantage over hand-crafted heuristics reflects fine contact details available in the full tactile image.They propose that a holistic use of the image can exploit those details.Source: C. Comparison with heuristic baseline:… · p. 7
IX. DISCUSSION
The paper combines a learned tactile grasp-quality metric with simulated local tactile transformations to select adjustments. Tests report a 70% average relative accuracy improvement on eight objects, while the approach remains local and discontinuous.
IX. DISCUSSION
- On eight test objects, tactile feedback yields an average relative accuracy improvement of 70% over open-loop grasping.Figure 7 shows examples of the gripper repositioning after an initial grasp.Source: §IX · p. 7, Figure 7 · p. 7
- A self-supervised convolutional network trained on 2,800 tactile images from 12 objects predicts grasp quality, and the policy selects simulated nearby imprints with the highest predicted score.The grasp-quality metric is used to plan local adjustments.Source: §IX · p. 7
- CAM highlights tactile-image regions correlated with high-intensity contact areas; the authors suggest grasp quality is strongly related to pressure.The examples show tactile imprints alongside their corresponding CAM regions.Source: §IX · p. 7, Figure 8 · p. 7
- The considered regrasp actions are local and require leaving the object; future work would explore global transformations using 3D geometry and continuous readjustments.The paper also proposes combining tactile and visual information to further improve grasping.Source: §IX · p. 8