PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer

Yunhai Han, Kelin Yu, Rahul Batra, Nathan Boyd, Chaitanya Mehta, Tuo Zhao, Yu She, Seth Hutchinson, Ye Zhao

arXiv:2112.06374v6cs.RO

TL;DR

Deformable-object grasping is difficult because safe force depends on uncertain contact interactions and geometry. The paper uses tactile and visual feedback from pinching and sliding with Transformer models to estimate safe grasping force. The models outperform CNN+LSTM on slip detection, and fruit-dataset pretraining improves banana-dataset training speed and accuracy.

  • Problem

    Safe grasping of deformable objects is challenging because contact interactions, object dynamics, and geometry are uncertain, and grasp force must avoid both slipping and damage.

    Source: §I · p. 1

  • Method

    The framework encodes visual and tactile feedback from pinching and sliding with Transformers, predicts grasp outcomes at candidate force thresholds, and infers a safe grasping force.

    Source: §I · p. 2

  • Results

    On public slip detection, TimeSformer and ViViT outperformed CNN+LSTM by 3.1% and 2.0%, respectively; banana pretraining cut epoch time from 23 to 8 seconds and improved training accuracy by 10%.

    Source: §I · p. 2, 1) Experiment Setup · p. 10

  • Takeaways & Limitations

    The framework selects grasping strength for fruits with varying hardness and surface texture, and its model can serve as a pre-trained model for new datasets.

    Source: §VI · p. 11

Abstract

from arXiv · show

Reliable robotic grasping, especially with deformable objects such as fruits, remains a challenging task due to underactuated contact interactions with a gripper, unknown object dynamics and geometries. In this study, we propose a Transformer-based robotic grasping framework for rigid grippers that leverage tactile and visual information for safe object grasping. Specifically, the Transformer models learn physical feature embeddings with sensor feedback through performing two pre-defined explorative actions (pinching and sliding) and predict a grasping outcome through a multilayer perceptron (MLP) with a given grasping strength. Using these predictions, the gripper predicts a safe grasping strength via inference. Compared with convolutional recurrent networks, the Transformer models can capture the long-term dependencies across the image sequences and process spatial-temporal features simultaneously. We first benchmark the Transformer models on a public dataset for slip detection. Following that, we show that the Transformer models outperform a CNN+LSTM model in terms of grasping accuracy and computational efficiency. We also collect a new fruit grasping dataset and conduct online grasping experiments using the proposed framework for both seen and unseen fruits. {In addition, we extend our model to objects with different shapes and demonstrate the effectiveness of our pre-trained model trained on our large-scale fruit dataset. Our codes and dataset are public on GitHub.

I. INTRODUCTION

Deformable-object grasping must balance slipping from insufficient force against damage from excessive force, while contact geometry and friction vary. The paper proposes a vision-tactile Transformer framework that uses exploratory actions to estimate safe grasping force and evaluates it on fruit grasping tasks.

I. INTRODUCTION

  • The work frames safe fruit grasping as a force-selection problem because weak grips can slip, while strong grips can cause damage amid variable contact geometry and friction.Visual-sequence methods may miss dense local deformation near contact; tactile sensors can capture deformation and local contact geometry.Source: §I · p. 1
  • TimeSformer and ViViT encode sensor-image sequences into fused physical features used to predict grasp outcomes at a given force threshold.A separate MLP classifies fruit types for automatic bin sorting.Source: §I · pp. 1–2
  • On public slip detection, TimeSformer and ViViT outperformed CNN+LSTM by 3.1% and 2.0%, respectively; banana pretraining cut training time fourfold and improved training accuracy by 10%.The paper also reports online grasping experiments on seen and unseen fruits.Source: §I · p. 2
  • The framework estimates safe grasping force from visual and tactile feedback gathered during pinching and sliding, without prior contact or geometric models.Its control parameter is the depth value read from tactile feedback, without external force-torque sensors.Source: §I · p. 2
  • Attention analysis indicates that the models use contact-area information and capture dependencies between initial and final grasp states.The analysis also examines sensitivity to visual and tactile image quality.Source: §I · p. 2

II. RELATED WORK

Prior grasping work spans sensorized hands, vision-based learning, tactile feedback, and slip control, but leaves challenges in force selection for deformable objects. The paper positions visuo-tactile Transformers as a framework for estimating safe grasping force through exploratory interactions.

A. Robotic Grasping

  • Robotic grasping studies include multimodal sensing, compliant hands, learned grasp policies, empirical force estimation, and slip-based stabilization, but several assume objects are already stably grasped or known in weight.The paper targets safe-force estimation for deformable objects.Source: A. Robotic Grasping · p. 2

B. Vision-tactile Sensor Fusion

  • Vision-tactile fusion has improved grasping performance, but cited experiments primarily use rigid objects, while CNN+LSTM classification methods do not learn safe-manipulation control policies.The cited classification tasks include slip detection, object recognition, and physical-property perception.Source: B. Vision-tactile Sensor Fusion · p. 2

C. Transformers for Robotics

  • Prior Transformer robotics studies address tabletop actions, throwing, and locomotion; the authors identify tactile-and-visual-image robotic grasping as unexplored in the cited work.Those studies report improvements over baselines in task performance and training efficiency.Source: C. Transformers for Robotics · pp. 2–3

III. METHODS

  • The proposed method first gathers object information through low-interaction pinching and sliding, then infers a safe force threshold for execution.The robot records these interactions with two different sensors.Source: §III · p. 3

1) Tactile:

The system pairs tactile contact sensing with RGB vision and uses Transformer encoders to process image sequences. Its models differ in how they factor spatial and temporal attention.

1) Tactile:

  • GelSight sensors use elastomer deformation to provide high-resolution information about fingertip contact and object compliance.Although the gripper has two fingertip sensors, the described experiments use one for a minimum system setup.Source: 1) Tactile · p. 3

2) Vision:

  • A wrist-mounted RealSense D435 supplies RGB images, angled 15 degrees to center the view on grasped objects.Source: 2) Vision · p. 3

B. Transformer Model

  • Transformer encoders embed image patches with positional information; TimeSformer attends to temporal then spatial dimensions, while ViViT processes both dimensions in parallel.TimeSformer uses a CLS token for task-level output; ViViT averages its final patch outputs instead. Transformer layers combine self-attention and an MLP with normalization and residual connections.Source: B. Transformer Model · pp. 3–4, Figure 2 · p. 3

C. Grasping Framework for Safe Force Estimation

The framework estimates a safe grasping threshold from visual and tactile feedback gathered during pinching and sliding. It combines the resulting features with candidate thresholds to predict grasping outcomes.

1) Grasping Outcome Prediction:

  • GelSight’s maximum depth value approximates grasping force; exceeding the selected threshold for three continuous frames triggers grasping.The threshold is also passed to the prediction model, and its unit is pixels.Source: 1) Grasping Outcome Prediction · p. 4
  • Two exploratory actions across two sensor modalities produce four vectors, which are concatenated by action and projected into a low-dimensional fused physical-feature embedding.The linear mapping uses a learnable weight matrix and bias.Source: 1) Grasping Outcome Prediction · pp. 4–5

C. Grasping Framework for Safe Force Estimation

1) Grasping Outcome Prediction:

  • The framework also includes a fruit-classification model that uses the fused physical-feature embedding.The overview also shows classification and outcome prediction as separate branches.Source: Figure 4 · p. 5

2) Safe Force Threshold Estimation:

The safe threshold is selected by testing candidate values against predicted grasping outcomes.

2) Safe Force Threshold Estimation:

  • Uniformly sample force thresholds, predict the outcome for each using the fused embedding, and select a threshold predicted to grasp safely.If multiple candidates are viable, select their average.Source: 2) Safe Force Threshold Estimation · p. 5

D. Grasping Framework for Fruit Classification

The framework includes a fruit-type classifier based on fused physical features, and the experiments benchmark Transformer models for slip detection. The slip-detection evaluation uses test objects unseen during training and averages results across five randomized dataset splits.

D. Grasping Framework for Fruit Classification

  • A separate MLP classifies fruit type from the fused physical-feature embedding, with the Transformer weights frozen during classifier training.Only the MLP is trained in a supervised scheme.Source: D. Grasping Framework for Fruit… · p. 5

IV. EXPERIMENTS

  • The experiments use the robot setup shown in Fig. 1.Source: §IV · p. 5

A. Transformers for Slip Detection

  • On the slip-detection dataset, TimeSformer and ViViT outperform CNN+LSTM by 3.1% and 2.0%, respectively.Reported values average five dataset splits, with variances in parentheses.Source: Table I · p. 5

1) Experiment Setup:

  • Slip detection is evaluated on five randomized train-validation-test splits, with test data drawn from objects unseen during training.The study compares vision-only, tactile-only, and combined inputs; the Transformer models use embedding size 256, eight layers, and 16 heads.Source: 1) Experiment Setup · pp. 5–6
  • The fruit experiments include six fruit types, and GelSight images show differing sensed deformation during pinching across their hardness and surface textures.Source: Figure 5 · p. 6

2) Experiment Analysis:

The Transformer models outperform CNN+LSTM on grasp-outcome classification and run substantially faster during robotic deployment. Their multi-sensor performance is stronger than single-sensor performance, motivating evaluation for safe fruit grasping.

2) Experiment Analysis:

  • 92.3% for TimeSformer and 90.0% for ViViT, versus 88.0% reported in on the same split with both sensor inputs.The passage attributes the advantage potentially to modeling long-term temporal dependencies and extracting spatial and temporal information simultaneously through self-attention.Source: 2) Experiment Analysis · p. 6
  • 2.46 s (25.6%) for TimeSformer and 2.43 s (25.3%) for ViViT, versus 9.61 s for ResNet18 and 21.39 s for VGG16 in feed-forward computation.The comparison uses both vision and tactile inputs, the same test dataset split and batch size, and an NVIDIA GeForce RTX 2070.Source: 2) Experiment Analysis · p. 6
  • The Transformer models perform similarly with either single sensor and better with multiple sensors, while CNN+LSTM performs better with tactile-only than vision-only input.The authors state that multi-sensor input provides better cues for slip detection.Source: 2) Experiment Analysis · p. 6

1) Experiment Setup:

  • The fruit-grasping training dataset contains 782 trials across six fruit types, with force thresholds varied by fruit to balance the three grasping outcomes.Vision and tactile data were collected at 30 Hz; each action sequence uses eight frames sampled at three-frame intervals.Source: 1) Experiment Setup · p. 6

2) Experimental Evaluation:

3) Grasping Outcome Prediction on Unseen Fruit:

The Transformers predict grasp outcomes on unseen fruit and are reported to outperform CNN+LSTM in online grasping of both seen and unseen fruit. Online deployment also selects different safe force thresholds as fruit properties vary.

3) Grasping Outcome Prediction on Unseen Fruit:

4) Online Fruit Grasping Evaluation:

5) Attention Analysis:

Attention visualizations show that the model attends across time and to contact-relevant image regions. The section also describes a noise-sensitivity evaluation using fruit images at several Gaussian noise levels.

5) Attention Analysis:

  • The first two contact frames receive larger temporal attention weights than intermediate frames, while final-frame patches also attend to preceding neighbors.The authors conjecture that initial and ending contact information is more inferable to physical status and grasping outcome.Source: 5) Attention Analysis · p. 8
  • At the final frame, tactile attention concentrates on local contact regions, while visual attention focuses near the gripper’s fingertips on fruit surfaces.Source: 5) Attention Analysis · p. 8
  • Attention interpretability may offer an alternative way to analyze physical deformation properties captured by tactile and visual sensors during contact-rich tasks.Source: 5) Attention Analysis · p. 8

A. Sensitivity Analysis of Visual and Tactile Feedback

  • The sensitivity setup adds Gaussian noise at σ = 20, 50, and 100 to apple and tomato images, then compares predictions with unnoised inputs.It tests tactile and visual images across force thresholds, averaging ten predictions for each noise level and threshold.Source: 1) Experiment Setup · p. 8

2) Experiment Evaluation:

The evaluation finds greater prediction sensitivity to tactile than visual image noise. Tests on unseen irregular objects show stronger grasping results for corn than banana, alongside an attention comparison under different noise levels.

2) Experiment Evaluation:

  • At σ = 100, visual noise causes a 6.5% outcome difference, while tactile noise causes an approximately 3.5 difference, or 20% of outcomes.Results for apples and tomatoes support greater robustness to visual than tactile image noise.Source: Figure 12 · p. 9, 2) Experiment Evaluation · p. 9

3) Attention Analysis:

B. Fruit Grasping Evaluation for Unseen Irregular Objects

2) Attention Analysis:

The paper visualizes spatial attention for corn, banana, apple, and banana after pre-training, then evaluates pre-training for banana grasping. The pre-trained model reduces training time and improves reported training accuracy.

2) Attention Analysis:

C. Fine-tuning the Pre-trained Model for an Object with New Geometries

1) Experiment Setup:

  • Pre-training reduces banana-model training time per epoch from approximately 23 seconds to 8 seconds.Source: 1) Experiment Setup · p. 10
  • With pre-training, banana training accuracy reaches 75%, 10% above training without pre-training, and reported validation oscillations are mitigated.The authors attribute the oscillations without pre-training to the banana dataset’s limited size and single object type.Source: 1) Experiment Setup · p. 10

3) Experiment Evaluation:

Banana experiments report higher grasping success with pretraining, including online trials, while attention analysis finds focus on local contact regions across banana and apple. The conclusion highlights grasping across varying fruit properties and irregular shapes, while noting the model-free approach relies on learned attention from rich data.

3) Experiment Evaluation:

  • 84% online grasping success with the pre-trained model compares with 76% without pretraining; offline success reaches 84% versus 52% for the baseline.The banana evaluation compares the original model, a model trained solely on banana data, and a banana-trained model using pretraining.Source: 3) Experiment Evaluation · p. 10

4) Attention Analysis:

  • After banana training, TimeSformer attends to local visual contact regions and fruit surfaces in tactile images for both banana and apple.The analysis reports this despite differences in fruit shape and contact-surface texture.Source: 4) Attention Analysis · p. 10

VI. CONCLUSIONS AND DISCUSSIONS

  • The framework selects grasping strength for fruits with varying hardness and surface texture, and can be pretrained on fruits with complex, irregular shapes.The conclusion also reports better performance than CNN+LSTM for slip detection and grasping-outcome prediction.Source: §VI · p. 11
  • The approach remains model-free and relies on learned attention from rich data; physical contact models are proposed as future work.Source: §VI · p. 11

Related papers

Loading 2112.06374v6…