Source-linked AI summary
Tactile Mapping and Localization from High-Resolution Tactile Imprints
Maria Bauza, Oleguer Canal, Alberto Rodriguez
TL;DR
The paper studies how tactile imprints can support object shape reconstruction and in-hand localization. It combines learned local-shape estimates with robot kinematics to build tactile maps, reporting accurate localization with coarse imprint matching followed by ICP refinement.
Problem
The paper studies how to reconstruct object shape from tactile contact and use it to identify and localize objects in-hand.
Source: §I · p. 1Method
It learns local contact shape from tactile imprints, combines multiple imprints with robot kinematics to build tactile maps, and uses CTI followed by ICP for localization.
Source: §I · pp. 1–2Results
Combining CTI for coarse localization with ICP for fine refinement improves localization accuracy, while estimates depend closely on tactile-shape resolution.
Source: §VIII · p. 6Takeaways & Limitations
The approach yields tactile feedback that can be used in real manipulation tasks.
Source: §VIII · p. 6
Abstract
from arXiv · showhide
This work studies the problem of shape reconstruction and object localization using a vision-based tactile sensor, GelSlim. The main contributions are the recovery of local shapes from contact, an approach to reconstruct the tactile shape of objects from tactile imprints, and an accurate method for object localization of previously reconstructed objects. The algorithms can be applied to a large variety of 3D objects and provide accurate tactile feedback for in-hand manipulation. Results show that by exploiting the dense tactile information we can reconstruct the shape of objects with high accuracy and do on-line object identification and localization, opening the door to reactive manipulation guided by tactile sensing. We provide videos and supplemental information in the project's website http://web.mit.edu/mcube/research/tactile_localization.html.
I. INTRODUCTION
The paper uses high-resolution tactile imprints as dense contact descriptors and combines them with robot kinematics to build tactile maps for object localization. It presents local shape estimation, global tactile mapping, and in-hand object identification and localization.
I. INTRODUCTION
- High-resolution tactile imprints act as dense local descriptors, while recognizing previously touched objects supports data association in estimation and reconstruction.The paper also links fine tactile differentiation and contact-force distribution to part comparison and precise manipulation.Source: §I · p. 1
- The framework combines tactile imprints with robot kinematics to build an object tactile map for identification and localization.It uses imprint similarity for coarse data association and contact shape for fine pose refinement.Source: §I · pp. 1–2
- CNNs estimate contact-patch shape from tactile imprints, achieving sub-millimeter accuracy on known contact geometries.The paper validates this local-shape estimate using controlled geometries.Source: §I · p. 1
- Robot kinematics and multiple grasps are fused to reconstruct an object's global tactile shape and discrete tactile-imprint representation.The reported reconstruction recovers the main dimensions of known objects with less than 5% error.Source: §I · pp. 1–2
II. RELATED WORK
Prior tactile estimation and localization work often relies on vision, planar-object assumptions, or low-resolution and bulky sensors. This work uses high-resolution GelSlim imprints to reconstruct local contact shape and reuse tactile maps for 3D object identification and localization.
II. RELATED WORK
- Many tactile estimation methods rely heavily on vision, while manipulation often creates occlusions; some prior approaches use touch mainly for contact detection.The cited work notes that prior methods may also focus on planar objects or use bulky sensors that impede dexterous manipulation.Source: §II · p. 2
- Prior shape-reconstruction methods explore regions with low-resolution sensors, making their reconstructions difficult to reuse for other applications.The authors contrast this with high-resolution sensing that reconstructs local contact shape and reuses tactile maps for localization.Source: §II · p. 2
- The approach uses a slim GelSlim finger and dense offline tactile maps to identify and locate 3D objects without prior shape knowledge.The paper frames tactile mapping as a way to support data association for manipulation of 3D objects.Source: §II · p. 2
III. APPROACH: TACTILE MAPPING AND LOCALIZATION
The framework reconstructs an object's tactile map offline and uses a new tactile imprint to identify and localize the object in-hand. Its three stages estimate local contact shape, assemble a global map using robot kinematics, and match online imprints to that map.
III. APPROACH: TACTILE MAPPING AND LOCALIZATION
- The framework first estimates the local shape from each tactile imprint using a CNN.The local shape is represented as a heightmap of the object geometry at contact.Source: §III · p. 2
- A self-supervised CNN maps tactile imprints to local contact shapes, while robot kinematics fuse multiple imprints and shapes into a global tactile map.The global map represents both object shape and its tactile imprints.Source: §III · p. 2
- For localization, an online imprint is compared with reconstructed tactile shapes for object identification and ICP-based pose estimation within the global map.The approach uses tactile imprints for coarse data association and contact shape for fine refinement.Source: §III · p. 2
IV. LOCAL SHAPE ESTIMATION
A CNN maps tactile images to local contact heightmaps, trained from controlled geometries with automatically collected imprint–heightmap pairs. Reported accuracy reaches 0.060±0.016 mm with 2,000 training datapoints, and held-out shapes are reconstructed at roughly 0.1 mm error.
IV. LOCAL SHAPE ESTIMATION
- Five controlled geometries provide identifiable local shapes for automatically labeling tactile-imprint and heightmap pairs.The collection includes 600 pairs per object, with 100 held out for testing; the CNN uses 10 convolutional layers and x/y pixel-position channels.Source: §IV · p. 3, Figure 2 · p. 3
- 0.060±0.016 mm average reconstruction accuracy is reached with 2,000 training datapoints.With 500 datapoints, reported accuracy is 0.1 mm; RMSE decreases as training data increases.Source: §IV · p. 3
- Held-out objects are reconstructed with error on the order of 0.1 mm in cross-validation.The evaluation trains with 2,000 datapoints while removing each training object in turn.Source: §IV · p. 3, Table I · p. 3
- Examples on novel complex objects show CNN-computed heightmaps recovering local geometry from tactile imprints.The figure pairs tactile imprints with the corresponding computed local-shape heightmaps.Source: §IV · p. 3, Figure 3 · p. 3
V. GLOBAL TACTILE AND SHAPE MAPPING
The tactile map combines contact imprints with robot kinematics to reconstruct an object’s tactile shape. Its accuracy depends on kinematics and contact-patch heightmaps, and reconstructed dimensions are mostly within a few millimeters.
V. GLOBAL TACTILE AND SHAPE MAPPING
- Each heightmap becomes a calibrated sensor-frame point cloud, transformed into the world frame and combined with other clouds from the same object.The method assumes accurate robot kinematics and a rigid, calibrated transformation among sensor, gripper, and robot arm.Source: §V · pp. 3–4
- Maps were built from grasps on a 10 mm grid at three orientations: 0 and ±20° in the vertical axis.The fixed objects were grasped at multiple locations and orientations to collect tactile imprints.Source: §V · p. 4
- Most reconstructed object parameters have errors of less than a few millimeters.A denser imprint grid improves accuracy for some objects but requires longer exploration; exploration was kept equal across maps for fair comparison.Source: §V · p. 4, Table II · p. 4
VI. TACTILE LOCALIZATION
The localization method uses tactile-image similarity for a coarse pose and ICP to refine it against a map assembled from similar touches. Across tested objects, ICP refinement can reduce error, especially with more map touches and a good initial estimate.
VI. TACTILE LOCALIZATION
- CTI-ICP-N initializes from similar tactile imprints, then refines the local cloud with ICP against an auxiliary cloud containing N nearby map touches.Using all map imprints for ICP is more time-consuming and does not improve results over small N; cropped maps focus ICP’s search and may reduce local-minimum traps.Source: §VI · p. 4
- Localization is evaluated by removing one touch from each of five object maps and measuring the RMSE after estimating its pose.The objects vary in weight, dimensions, symmetry, and texture.Source: §VI · p. 4
- CTI’s median RMSE is about 10 mm for three of five objects and 20 mm for the other two; gum-box symmetry makes errors below 20 mm difficult.The gum-box imprints can look alike when separated by 20 mm.Source: §VI · p. 5, Table III · p. 5
- For objects with varied contact patches, CTI-ICP-5 improves on CTI; on the hair brush, its RMSE distribution has more density to the left despite similar medians.Both CTI methods improve over RANDOM, while outliers keep CTI and CTI-ICP-5 medians nearly the same; refinement is stronger when the initial guess is good.Source: §VI · p. 5, Figure 5 · p. 5
- With 25% or more of the scissors’ point clouds, CTI’s median RMSE stays at 10 mm, while CTI-ICP-N reduces error by almost half.At 10% (around 50) of the point clouds, the median is around 20 mm; the figure compares medians for CTI, CTI-ICP-1, and CTI-ICP-5.Source: §VI · p. 6, Figure 6 · p. 6
VII. IN-HAND IDENTIFICATION AND LOCALIZATION
The approach identifies a grasped object by matching its tactile imprint to explored object maps, then localizes the imprint on the selected map. In three examples it correctly identifies objects and estimates their in-hand locations in real time.
VII. IN-HAND IDENTIFICATION AND LOCALIZATION
- Object identity is assigned from the most similar map imprint, after which the new imprint is stitched to that object’s tactile shape to estimate pose.The resulting pose is expressed relative to the tactile sensor.Source: §VII · p. 6
- The approach correctly identifies each of three grasped objects and estimates its in-hand location, with pose estimation fast enough for real time.The reported steps are one CNN pass, feature-space similarity comparison, and ICP with small point clouds.Source: §VII · p. 6
VIII. DISCUSSION AND FUTURE WORK
The paper combines tactile imprints with robot kinematics to build tactile maps and localize objects in-hand. The discussion identifies practical extensions and limitations spanning sensing, mapping, and localization.
VIII. DISCUSSION AND FUTURE WORK
- Combining tactile imprints with robot kinematics supports object identification and localization, with CTI providing coarse localization and ICP fine refinement.The approach builds a tactile map from multiple imprints and reports accurate tactile feedback for real manipulation tasks.Source: §VIII · p. 6
- The current approach is purely tactile, while the authors propose visual feedback to create more complete maps from tactile point-cloud estimates of global shape.They frame the tactile-only design as a way to demonstrate tactile sensing as an enabler of manipulation.Source: §VIII · p. 6
- Shape priors and object properties such as mass or texture could help produce a more accurate tactile map.Source: §VIII · p. 6
- Symmetry or lack of texture can make an object hard or impossible to localize from one imprint, leaving a set of possible poses; multiple imprints could help estimate the actual pose.Source: §VIII · p. 6
- The method uses one tactile sensor; for a two-jaw gripper, two sensors would likely improve discrimination and location accuracy.Source: §VIII · p. 6
- The current solution ignores non-contact information, although adding constraints from where contact does not occur could greatly enhance performance.Source: §VIII · p. 6
- More tactile imprints could support training a better feature space for distinguishing imprints.Source: §VIII · p. 6
- ICP stitching of tactile-imprint point clouds is challenging, and the authors suggest choosing metrics that address their sparsity.Source: §VIII · p. 6