Source-linked AI summary
Improved GelSight Tactile Sensor for Measuring Geometry and Slip
Siyuan Dong, Wenzhen Yuan, Edward Adelson
TL;DR
The paper addresses limited geometric accuracy and difficult fabrication in an earlier compact GelSight fingertip sensor, alongside the challenge of robust slip detection across varied objects. It introduces a Lambertian-membrane sensor with redesigned illumination and evaluates geometry reconstruction and slip detection during robotic grasping. The new sensor reports improved reconstruction accuracy, detects translational and rotational slip across 37 objects without prior object knowledge, and supports grasp control.
Problem
The earlier Fingertip GelSight had limited 3D reconstruction accuracy and complicated fabrication, while robust slip detection across varied object properties remained challenging.
Method
The paper redesigns GelSight with a Lambertian membrane and more uniform illumination, then uses calibrated photometric reconstruction and relative marker-texture motion for slip detection.
Results
The new sensor has lower reconstruction errors and smaller spatial variances, and detects translational and rotational slip for 37 objects without prior object knowledge.
Takeaways & Limitations
The sensor assists safe manipulation tasks and can support applications including safe grasping, object recognition, and hardness estimation.
Abstract
from arXiv · showhide
A GelSight sensor uses an elastomeric slab covered with a reflective membrane to measure tactile signals. It measures the 3D geometry and contact force information with high spacial resolution, and successfully helped many challenging robot tasks. A previous sensor, based on a semi-specular membrane, produces high resolution but with limited geometry accuracy. In this paper, we describe a new design of GelSight for robot gripper, using a Lambertian membrane and new illumination system, which gives greatly improved geometric accuracy while retaining the compact size. We demonstrate its use in measuring surface normals and reconstructing height maps using photometric stereo. We also use it for the task of slip detection, using a combination of information about relative motions on the membrane surface and the shear distortions. Using a robotic arm and a set of 37 everyday objects with varied properties, we find that the sensor can detect translational and rotational slip in general cases, and can be used to improve the stability of the grasp.
I. INTRODUCTION
The paper introduces a redesigned Fingertip GelSight sensor to improve geometric accuracy and simplify fabrication while supporting slip detection for robotic grasping.
- Tactile sensing helps robots assess object holding and gripping force during interaction.
- Robust slip detection remains challenging because objects vary in weight and geometry.Slip occurs when gripping force is insufficient, and prior systems use signals such as force, vibration, acceleration, or skin stretch.
- The existing Fingertip GelSight provides 640×480-pixel resolution at 0.024 mm/pixel but has limited surface-normal precision and fabrication difficulty.
- The new design uses a redesigned illumination system suited to a Lambertian surface, improving reconstructed 3D height maps.
- The sensor uses relative membrane motions and shear-related distortions to detect slip and enhance grasping.
II. RELATED WORK
Related work establishes GelSight as a high-resolution optical tactile sensor while identifying accuracy and fabrication limitations in the earlier fingertip design.
- Optical tactile sensors infer contact geometry and forces from camera images of deforming surfaces.
- GelSight combines silicone gel, colored LEDs, and a camera to reconstruct depth from calibrated color-to-surface-normal mappings.
- GelSight has supported texture recognition, soft-material inspection, and compact robotic fingertip sensing.
- Li’s fingertip sensor produced coarse 3D maps because non-uniform illumination and a semi-specular coating impaired surface-normal estimation and photometric stereo.
C. Slip detection
The paper develops GelSight-based slip detection for varied grasping situations and presents a redesigned sensor intended to improve geometry reconstruction and manufacturability.
- Related slip detection: Earlier slip-detection work used force, vibration, acceleration, or shear-to-normal-force ratios to identify grasp instability.
- Related slip detection: Yuan et al. detected slip from peripheral contact displacement but did not test real-time grasping or varied textured objects.
- New sensor design: The proposed redesign uses a Lambertian surface, more uniform illumination, and a standardized 3D-printable framework.
- New sensor design: Three-color LED arrays, a hexagonal tray, and tilted mounts provide illumination across the sensing surface.
- New sensor design: The sensor replaces semi-specular coating with an easier-to-make, non-toxic Lambertian membrane and uses a dome-shaped reflective surface for improved uniformity.
- New sensor design: The compact sensor integrates a webcam, protective cover, and gripper-compatible frame using standardized fabrication.
- Geometry reconstruction: A calibrated RGB-to-surface-normal lookup table is averaged across positions to reduce illumination-variance noise in height reconstruction.
- Geometry reconstruction: Figure 3 compares reconstructed depth for ball arrays, a finger, a watch chain, and a quarter using the new and previous sensors.
IV. EVALUATION OF GEOMETRY MEASUREMENT WITH GELSIGHT
The evaluation compares the new and previous GelSight sensors across color response, lookup-table accuracy, illumination variance, and reconstructed 3D shape quality.
- The evaluation covers gradient-versus-color change, lookup-table mapping accuracy, spatial illumination variance, and 3D shape reconstruction.
A. Gradient vs. Color Change
The new sensor provides a more linear relationship between surface-normal slope and image color change across a wider angular range than Li’s sensor.
- The new sensor improves the linear mapping between surface normals and color change.
- Li’s sensor shows larger color changes near contact, but color no longer represents slope changes well around 30 degrees.
- The new sensor’s color change is less obvious but remains linear across a larger range of slope angles.
B. Mapping accuracy of the lookup table
The new sensor more accurately maps image colors to surface-normal pitch and yaw angles, with improved pitch accuracy and fewer position-dependent failures.
- The new sensor’s measured surface-normal angles more closely follow ground truth for pixels within the contact area.
- R2 for measured pitch angles improves from 0.557 with Li’s sensor to 0.818 with the new sensor.
- The new sensor also exceeds Li’s sensor in reconstructing yaw angles, although both achieve very high R2 values.
- The new sensor has higher and more tightly bounded R2 distributions than Li’s sensor across calibration positions.Li’s sensor has a low-R2 tail, whereas the new sensor lacks this tail and shows no blind spot in the sensing surface.
D. Quality of 3D shape reconstruction
The new sensor reconstructs smoother, less noisy 3D shapes and supports slip detection by combining object–marker motion with marker-distribution and contact-area cues.
- D. Quality of 3D shape reconstruction: The new sensor produces smooth 3D reconstructions that preserve object shapes, while Li’s sensor produces grainier, noisier, and sometimes deformed maps.Fine features, such as the eagle’s feather on a quarter coin, are clearer with the new sensor.
- V. SLIP DETECTION WITH NEW GELSIGHT SENSOR: For highly curved, smooth objects, slip is inferred from smaller peripheral marker displacement relative to central motion.A significant decrease in contact area indicates severe slip.
- V. SLIP DETECTION WITH NEW GELSIGHT SENSOR: For textured objects, slip is detected by comparing translational and rotational motion of object texture against the sensor’s markers.A sufficiently large relative displacement or accumulated translation or rotation indicates slip.
- V. SLIP DETECTION WITH NEW GELSIGHT SENSOR: The marker-motion method also detects rotational slip, though less perfectly and with a simplified norm-based computation.
- V. SLIP DETECTION WITH NEW GELSIGHT SENSOR: The experiment evaluates slip cues on 37 everyday objects varying in size, shape, material, and surface texture.
A. Experimental setup
The grasp experiments use a UR5 arm and WSG 50 gripper equipped with a GelSight finger to lift 37 varied everyday objects and respond to detected slip.
- The system combines a UR5 6DOF arm with a WSG 50 parallel gripper carrying the GelSight sensor on one finger.The arm has 850 mm reach and ±0.1 mm repeatability; mounting the sensor reduces gripper opening to 80 mm.
- Grasp trials use 37 everyday objects with varied sizes, shapes, materials, and surface textures.The robot approaches, grasps, and slowly lifts each object by 3 cm while using GelSight feedback.
- When slip is detected during lifting, the robot releases and re-grasps with a larger contact-detection threshold until the grasp is considered safe.
B. Slip prediction
The study evaluates slip prediction during grasping across 37 objects, distinguishing successful, failed, and borderline cases. Prediction failures commonly arise with flat, smooth objects under very small gripping forces and with significant rotational slip.
- 37 objects were grasped and lifted 7 to 10 times each using different contact thresholds, producing varied gripping forces.Most objects had approximately balanced successful-grasp and slip cases; some very light or smooth objects had only successful or slip cases.
- Slip outcomes were grouped as successful, failure, or border cases according to relative motion, grasp firmness, and lifting behavior.Failure included significant translational or rotational slip, while border cases involved barely lifting the object without a tight grasp.
- Flat, smooth objects under very small gripping forces caused 28% of failures because weak shear signals and insufficient texture hindered slip detection.Noise interfered with marker-motion analysis, while the smooth surface provided too little texture for detection.
- Significant rotational slip while grasping a flat object caused another 28% of failures in marker-based measurement.The passage states that more thorough measurement of marker rotational movement could prevent this failure mode.
C. Grasp control with slip detection
The paper uses GelSight slip feedback to increase gripping force after detected slip and evaluates this closed-loop strategy on 33 objects. Across 99 grasp experiments, the robot achieved an 89% success rate, although very small gripping forces sometimes left insufficient contact information for effective slip measurement.
- The closed-loop controller suspends and releases an object after detected slip, then re-grasps it at the same position with a higher contact threshold.The experiment tested 33 objects, excluding four the robot could not lift.
- GelSight controls gripping force using changes in surface geometry or marker displacement, with each trial increasing the threshold by 1.2 times.The initial threshold was identical across objects and represented bare contact, while stable-grasp thresholds differed by object.
- 88 of 99 grasp experiments succeeded, yielding an 89% success rate, with an average of 2.3 grasp attempts per experiment.Individual grasp experiments used between 1 and 6 grasp attempts.
- In failure cases, gripping forces were so small that the contact area provided insufficient information for effective slip measurement.Humans could clearly see most slips detected by GelSight, indicating that sensor measurement was less effective in these low-force cases.
- The new sensor detected translational and rotational slip during grasping without prior knowledge of object properties across 37 tested objects.The authors report that its implementation assists safe manipulation tasks.