Source-linked AI summary
Extended Tactile Perception: Vibration Sensing through Tools and Grasped Objects
Tasbolat Taunyazov, Luar Shui Song, Eugene Lim, Hian Hian See, David Lee, Benjamin C. K. Tee, Harold Soh
TL;DR
The paper asks whether robots can infer contact properties through held objects, rather than only detect contact events. It uses finger-mounted tactile sensing and learning to decode vibrations, demonstrating localization and classification tasks as well as grasp-stability prediction.
Problem
The paper asks whether robots can infer contact properties, such as location and contact type, on held rigid objects from tactile signals.
Source: §III · p. 3Method
The approach combines robot-finger tactile sensors with learning models that map tactile sequences to contact properties on held objects.
Source: §II · p. 2, §III · p. 3Results
Vibro-tactile sensing extended perception across rod localization, handover grasp-stability prediction, and fork-based food classification; best grasp-stability models reached around 60–80% accuracy across three objects.
Source: §I · pp. 1–2, §VI · p. 6Takeaways & Limitations
Vibro-tactile perception can extend robot sensing beyond its body, like having virtual sensors on held objects.
Source: §I · p. 2, §VII · p. 7
Abstract
from arXiv · showhide
Humans display the remarkable ability to sense the world through tools and other held objects. For example, we are able to pinpoint impact locations on a held rod and tell apart different textures using a rigid probe. In this work, we consider how we can enable robots to have a similar capacity, i.e., to embody tools and extend perception using standard grasped objects. We propose that vibro-tactile sensing using dynamic tactile sensors on the robot fingers, along with machine learning models, enables robots to decipher contact information that is transmitted as vibrations along rigid objects. This paper reports on extensive experiments using the BioTac micro-vibration sensor and a new event dynamic sensor, the NUSkin, capable of multi-taxel sensing at 4~kHz. We demonstrate that fine localization on a held rod is possible using our approach (with errors less than 1 cm on a 20 cm rod). Next, we show that vibro-tactile perception can lead to reasonable grasp stability prediction during object handover, and accurate food identification using a standard fork. We find that multi-taxel vibro-tactile sensing at sufficiently high sampling rate led to the best performance across the various tasks and objects. Taken together, our results provides both evidence and guidelines for using vibro-tactile perception to extend tactile perception, which we believe will lead to enhanced competency with tools and better physical human-robot-interaction.
I. INTRODUCTION
The introduction asks whether robots can infer contact properties through held objects, beyond detecting contact events. It proposes finger-mounted vibro-tactile sensing and learning to decode vibrations transmitted along standard objects.
I. INTRODUCTION
- The approach addresses whether robots can locate contact on held tools or identify contacted objects, extending beyond prior detection of whether contact occurred.The introduction connects this capacity to sensing objects out of reach or too difficult or dangerous to touch directly.Source: §I · p. 1
- The system uses compact tactile sensors on the robot gripper, rather than specially instrumented tools, to sense vibrations transmitted along standard rigid objects.Examples include rods, forks, spoons, and handled items such as plates.Source: §I · pp. 1–2
- NUSkin is a 40-taxel event tactile sensor modified for 4 kHz sampling, with asynchronous positive or negative spikes responding to contact-pressure changes.The BioTac hydrophone is also tested, but has one dynamic sensor.Source: §I · p. 2
- Learning models decode the tactile signals, although high-rate sequences and sparse event spikes are difficult to learn from in data-constrained tactile settings.The authors report that neural models performed best in their tests, while binned or smoothed spike counts and FFT features with classical learning also worked well.Source: §I · p. 2
- On a 20 cm rod, localization error was less than 1 cm; the approach also supports grasp-stability prediction during handover and food classification through a fork.The introduction presents these as demonstrations of sensing through held objects.Source: §I · pp. 1–2
- The work reports real-world robot experiments and design findings, and makes its code and annotated tactile-signal datasets publicly available.The authors identify multi-sensor, multimodal sensing as a plausible next step.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
Earlier robotic tactile work largely examined objects directly touching the sensor, while this paper focuses on decoding contact properties transmitted through held objects. It compares finger-mounted dynamic sensors and extends prior work with NUSkin and additional tasks.
II. BACKGROUND AND RELATED WORK
- Dynamic tactile sensing measures rapidly changing tactile information and includes accelerometers and whiskers whose base motion can reveal object presence or shape.Whiskers are notable because their sensing elements are at the base, not in the whiskers themselves.Source: §II · p. 2
- Earlier vibration-based approaches used mounted accelerometers, microphones, or force sensors on containers and specially designed tools for content or texture recognition.A sensorized “Forque” was also used for food classification; this paper instead uses sensors on robot fingers with standard objects.Source: §II · p. 2
- BioTac senses vibrations through a hydro-acoustic pressure sensor, while NeuTouch uses 39 piezoresistive taxels that transmit pressure changes as spikes.The modified NeuTouch, NUSkin, samples at 4 kHz; comparing these sensors examines sensing modes, taxel count, and transmitted data.Source: §II · p. 2
- Prior robotics research mainly measured properties of objects in direct sensor contact; interpreting vibrations through grasped tools remained relatively unexplored.A closely related study used BioTac and machine learning for contact localization, while this work emphasizes dynamic sensing, NUSkin, and additional tasks.Source: §II · pp. 2–3
III. PROBLEM STATEMENT: EXTENDED TACTILE PERCEPTION
Extended tactile perception is the inference of contact properties on a held or touched rigid object from signals transmitted through it. The paper uses learning to avoid difficult analytical modeling and addresses the sensing and sequence-learning demands of vibration capture.
III. PROBLEM STATEMENT: EXTENDED TACTILE PERCEPTION
- Extended tactile perception means accurately perceiving properties of contacts on a held or touched rigid object.The target properties include contact location and properties of the contacted object.Source: §III · p. 3
- Mechanical transients carry contact locations and object properties along a rod, but analytical solutions become difficult with unknown materials or complex and non-uniform tools.The paper notes that Euler-Bernoulli beam equations are difficult to solve generally and require precise material properties.Source: §III · p. 3
- The learning task maps tactile signal sequences to contact properties such as location, accommodating irregular observation times and variable sequence lengths.A learned parameterized function associates each observed signal with its target property.Source: §III · p. 3
- Because Pacinian and Meissner’s corpuscles detect vibration frequencies as high as 400 Hz, sampling should be performed at least above the 800 Hz Nyquist rate; these rates produce long tactile time series that may be difficult to interpret.Source: §III · p. 3
IV. EVENT-BASED TACTILE SENSOR
NUSkin is a modified NeuTouch event tactile sensor designed for vibration sensing at higher sampling rates. Its revisions increase temporal sampling and arrange taxels in a regular lattice, while softer skin amplifies stimuli and supports tool grasping.
IV. EVENT-BASED TACTILE SENSOR
- NeuTouch uses a 39-taxel graphene pressure transducer and asynchronously transmits pressure changes as positive and negative spikes through ACES.The transducer captures vibrations up to 1500 Hz; unique electrical pulse signatures allow asynchronous taxel transmission and deconvolution.Source: §IV · p. 3
- NUSkin modifies NeuTouch by increasing its sampling rate from 1 kHz to 4 kHz to better accommodate the graphene pressure transducer’s frequency response.The increase is enabled by shortening the pulse signature duration from 1 ms to 0.25 ms.Source: §IV · p. 3
- NUSkin uses a regular lattice of 40 taxels, producing matrix-structured data that facilitates deep-learning methods.The prior NeuTouch arrangement has 39 taxels.Source: §IV · p. 4
- NUSkin’s softer Ecoflex 00-10 skin amplifies stimuli and increases deformation during contact, aiding signal collection and stable grasping.The passage links amplification to more spatial-temporal features and deformation to more stable grasping of tools and objects.Source: §IV · p. 4
V. EXPERIMENT SETUP: TASKS AND ROBOTS
The experiments test vibration-based perception across rod-tap localization, handover grasp stability, and fork-based food identification. They compare sensors, features, and learning models under repeated train-and-test evaluation.
V. EXPERIMENT SETUP: TASKS AND ROBOTS
- The study tests whether robots can interpret vibrations through held objects, how fast sensing must be, and whether multiple taxels improve performance.Its three tasks are tap localization on rods, grasp-stability prediction during handover, and food identification with a standard fork.Source: §V · p. 4
- Tap localization predicts a human tap’s position on 20, 30, or 50 cm acrylic rods; stability classification distinguishes stable from unstable human grasps on a rod, plate, or box.Food identification classifies seven fork-contact classes: a control, water, and five foods.Source: §V · p. 4
- The robot used BioTac or NUSkin sensors on gripper fingers; data collection included 1,000 taps per rod and 50 examples per object and class for the other tasks.Tap locations were motion-capture annotated, while grasp stability was manually labeled; objects were not released during handover trials.Source: §V · p. 4
- The evaluated features included raw signals, binned spike counts, FFT features, neural autoencoder representations, and Event Spike Tensor features.NUSkin raw spikes performed poorly in preliminary tests, so binned counts were used, with 5 ms bins for localization and stability and 50 ms bins for food identification.Source: §V · p. 5
- SVM, MLP, and GRU-based RNN models performed regression for localization or classification for stability and food type, using repeated random 90/10 train-test splits.Training used grid search and four-fold cross-validation; results were averaged over five or 20 repetitions depending on task and model.Source: §V · p. 5
VI. RESULTS
Across the tested tasks, fast vibro-tactile sensing supported contact localization, grasp-stability prediction, and food identification. Higher sampling rates and spatially distributed taxels improved localization performance.
Tap localization
- Best localization errors were approximately 1 cm, 1.7 cm, and 3.1 cm on 20, 30, and 50 cm rods; neural models performed best among learned models.NUSkin generally outperformed BioTac PAC, especially on the 50 cm rod, where it was approximately 1 cm better.Source: §VI · pp. 5–6
Grasp stability prediction
- Grasp-stability classification reached around 60–80% accuracy across three objects, with mixed sensor performance.BioTac PAC scored higher for the plate, while NUSkin performed better with the box; static tactile sensing and other modalities could improve results.Source: §VI · p. 6, Table II · p. 6
Food identification
- Food identification accuracies were close to 90% with NUSkin and FFT/EST features, with errors concentrated among similar items.NUSkin distinguished the empty-plate control from water at approximately 80%, despite the small forces involved.Source: §VI · p. 6, Table III · p. 6, Figure 4 · p. 6
Sampling frequency
- Localization errors fell substantially as sampling increased from 5 Hz to 1 kHz, with smaller gains thereafter and best performance around 2–3 kHz.The reported results indicate that 2 kHz was sufficient for good performance across the examined tasks.Source: Figure 5 · p. 6, §VI · p. 7
Number of taxels
- Using 80 taxels reduced localization errors compared with one 40-taxel finger or BioTac PAC, and outperformed every single-taxel model.The advantage of 80 taxels was consistent across the tested machine-learning methods and features.Source: §VI · p. 7, Figure 6 · p. 7, Figure 7 · p. 7
VII. DISCUSSION AND CONCLUSION
The experiments demonstrate tactile perception beyond the robot’s boundary through vibro-tactile sensing. The evidence is bounded by rigid objects, while non-rigid or multi-link objects and multimodal sensing remain important open directions.
Conclusion
- Vibro-tactile sensing extended tactile perception beyond the robot’s boundary and supported contact localization and contact-type classification.The paper identifies rod localization, handover grasp-stability classification, and food classification through a fork as demonstrated tasks.Source: §VII · p. 7
Limitations and future work
- Experiments used rigid objects; non-rigid or multi-link objects may transform signals nonlinearly or introduce aliasing, and remain future work.The authors consider extension possible if sufficient information travels along the object to the tactile sensors.Source: §VII · p. 7
- Vibration is only one sensing mode; a coherent world representation requires other modalities, and combined sensing could improve grasp-stability prediction.The paper points to combining static and dynamic tactile sensing with multimodal sequence-learning models as a potential approach.Source: §VII · p. 7