Source-linked AI summary
Slip Detection with Combined Tactile and Visual Information
Jianhua Li, Siyuan Dong, Edward Adelson
TL;DR
Slip detection is challenging in robotic manipulation, where contact-state feedback supports grasp adjustment. The paper combines GelSight tactile and external-camera sequences in a DNN, achieving 88.03% test accuracy on unseen objects; the authors anticipate improvement with more data.
Problem
Slip is a common loss-of-contact state caused by insufficient force or improper grasping strategy, and existing tactile detection can face calibration and weak-signal limitations.
Method
A DNN combines eight-frame GelSight and external-camera sequences using CNN feature extraction and LSTM temporal modeling to classify stable grasps or slip.
Results
88.03% test accuracy was achieved by the Inception-V3 model using raw tactile and vision image sequences.
Takeaways & Limitations
Tactile and vision information are complementary for slip detection, supporting a multisensor-fusion approach to robotic grasp stability.
Abstract
from arXiv · showhide
Slip detection plays a vital role in robotic manipulation and it has long been a challenging problem in the robotic community. In this paper, we propose a new method based on deep neural network (DNN) to detect slip. The training data is acquired by a GelSight tactile sensor and a camera mounted on a gripper when we use a robot arm to grasp and lift 94 daily objects with different grasping forces and grasping positions. The DNN is trained to classify whether a slip occurred or not. To evaluate the performance of the DNN, we test 10 unseen objects in 152 grasps. A detection accuracy as high as 88.03% is achieved. It is anticipated that the accuracy can be further improved with a larger dataset. This method is beneficial for robots to make stable grasps, which can be widely applied to automatic force control, grasping strategy selection and fine manipulation.
I. INTRODUCTION
Slip is a difficult but important contact-state problem in robotic manipulation because detecting it can support force adjustment and grasp-planning decisions. The paper combines GelSight tactile images with external-camera images and trains a DNN to classify grasp stability.
- Slip occurs when a grasp uses insufficient force or an improper grasping strategy, making contact-state feedback important for manipulation.
- GelSight detects slip through object-texture motion or marker motion in the contact area, with 0.024 mm/pixel spatial resolution.
- The earlier threshold-based GelSight method achieved 71% accuracy but required threshold tuning across objects and struggled with weak contact signals.
- The proposed method trains a DNN on paired GelSight and external-camera images collected during grasping and lifting experiments.
- Training used 84 daily objects across 1102 grasping and lifting experiments, while 10 new objects were used for testing.
A. Slip detection
Prior slip-detection approaches sense vibration, force, thermal change, acceleration, or relative motion, but several require advance calibration or have difficulty separating slip from contact changes. Vision-tactile learning has been explored, while the paper targets slip detection during lifting.
- Tactile slip sensors have used vibration, force ratios, thermal change, and relative motion as physical signals.
- Force-ratio methods require the contact surface’s friction coefficient to be measured in advance, while thermal sensors may confuse slip and contact signals.
- A prior vision-tactile neural network produced a primitive slip representation but could not distinguish slip from pressure changes and lacked a real grasping experiment.
- A GelSight-and-RGB-camera model achieved 94% accuracy for slip prediction at initial grasping, which differs from slip detection during lifting.
- A previous GelSight detection method achieved 71% accuracy but faced threshold-tuning problems and weak signals for smooth objects with small contact areas.
B. GelSight sensor
GelSight is a camera-based tactile sensor that captures fine contact geometry through a deformable, illuminated gel. Surface markers provide motion cues related to contact forces and slip.
- GelSight uses a soft silicone gel with a reflective membrane, three directional RGB LEDs, and a webcam to capture gel deformation.
- The captured RGB values can be used to infer the depth map of the contact surface.
- Markers on the gel surface indicate normal, shear, and torsional forces and can detect incipient slip.
- The sensor was designed to measure the fine geometry profile of the contact surface.
III. MODEL DESCRIPTION
The model processes synchronized eight-frame sequences from GelSight and external cameras using pretrained CNN feature extraction followed by recurrent temporal modeling. It classifies the sequence as a stable grasp or slip.
- The DNN takes eight paired GelSight and external-camera frames as input and outputs the probability of slip.
- A pretrained CNN extracts features from each image, after which paired tactile and visual features are concatenated and reduced to 64 features.
- Two 64-unit LSTM layers process the eight feature rows to capture temporal information before a final classifier predicts stable grasp or slip.
- Dropout uses 50% keep-probability after the fully connected layer and 80% after each LSTM layer to reduce overfitting.
- Because the dataset was relatively small, pretrained-network parameters were fixed and only the remaining DNN layers were trained.
A. Experimental setup and data collection
The experimental setup combines a GelSight sensor and external camera on a gripper mounted to a UR5 robot arm. Training data come from grasping and lifting varied daily objects, with image sequences selected around the lifting onset and expanded using sliding windows.
- A. Experimental setup and data collection: A 6-DOFs UR5 arm uses a WSG 50 parallel gripper with one GelSight finger and a side-mounted webcam.The gripper opening is reduced to 80.0 mm by the GelSight sensor, and the lifting speed is 40.0 mm/s.
- A. Experimental setup and data collection: Training experiments cover 84 daily objects differing in size, shape, surface texture, material, and weight.Gripping distances are selected around each object's automatically measured barely touching distance.
- A. Experimental setup and data collection: Each DNN input contains eight successive image frames, including one frame captured two frames before lifting begins.The remaining seven frames start at the lifting reference time T(0).
- A. Experimental setup and data collection: A sliding window expands 1,102 grasping experiments into 5,510 data pieces, with 85% for training and 15% for validation.The dataset includes translational and rotational slip, while incipient slip is categorized as slip.
B. Slip detection results
The evaluation tests models on unseen objects while varying input format, pretrained CNN, and sensor combination. Combined tactile and vision inputs achieve the strongest test accuracy, while image-difference processing highlights motion information.
- B. Slip detection results: Ten previously unseen objects are evaluated in 152 grasps, producing 760 test image sequences with balanced successful and failed cases.The test objects differ from training objects in shapes, materials, and weights.
- B. Slip detection results: The study varies input data format, pretrained CNN type, and data-source combination to identify effective DNN parameters.The compared data sources are tactile-only, vision-only, and tactile-vision.
- B. Slip detection results: Image-difference sequences are computed relative to T(−2), with an offset of 128 ensuring positive pixel values and highlighting object motion.The resulting tactile and external-camera sequences are used to train multiple models.
- B. Slip detection results: Three pretrained CNN architectures—VGG-16, VGG-19, and Inception-V3—are compared using ImageNet pretrained weights.Features are extracted from the CNNs for the slip-detection task.
SEQUENCE USING TACTILE AND VISION IMAGES
The results compare sequence lengths for tactile and vision image inputs and against the earlier GelSight-only method. Eight frames perform best, while six frames remain comparable and can detect slip earlier than the previous method.
- SEQUENCE USING TACTILE AND VISION IMAGES: The 8-frame input gives the highest test accuracy, while the 6-frame results are also comparable.This finding holds for both model parameter settings reported in the experiment.
- SEQUENCE USING TACTILE AND VISION IMAGES: The proposed models outperform the earlier GelSight-only method for every tested input length.The largest difference at six frames indicates earlier slip-signal detection by the proposed model.
C. Experimental analysis
The experiments show why combining GelSight tactile images with external-camera images improves slip classification: tactile cues can be ambiguous, while the fused model distinguishes slip from gel deformation and stable grasps.
- The combined GelSight and external-camera model achieves much better accuracy than either single-source model.
- In a smooth-surfaced rheostat slip case, GelSight images repeat similar shapes and image differences change little after lifting begins.These tactile patterns can make the grasp appear stable despite slipping.
- External-camera sequences provide complementary visual information when GelSight observations are difficult to interpret.The paper identifies smooth objects and repeated shapes as circumstances where tactile-only judgment is challenging.
- For a heavy metal object, apparent downward motion reflects gel-surface stretching rather than relative motion between the object texture and sensor markers.The neural network classifies this grasp as stable, indicating that it distinguishes gel deformation from slip.
V. CONCLUSIONS
The paper presents a DNN slip detector that fuses GelSight tactile sensing with external-camera images during object lifting. Across grasp experiments, the method supports slip classification and illustrates sensor fusion for robotic manipulation.
- The method uses GelSight tactile sensing and an external camera mounted on the gripper without prior knowledge of object physical parameters.
- More than 1200 grasp experiments with 94 objects use 84 objects for training and test the remaining 10 unseen objects.
- Tactile and vision information are complementary for slip detection, with vision providing more cues for slippery, smooth surfaces.
- The work provides an example of a learning method for multiple sensor fusion.