SonicSense: Object Perception from In-Hand Acoustic Vibration

1 Introduction

• Acoustic vibration sensing can capture object properties such as material and geometry, while air microphones are susceptible to ambient noise.Contact microphones sense vibrations caused by physical contact.Source: §1 · p. 1
• Prior systems commonly tested fewer than five primitive objects, homogeneous materials, single-finger sensing, or different contacts on the same objects.The introduction also notes limited training data and manual or fixed-pose interaction procedures.Source: §1 · pp. 1–2
• SonicSense addresses container inventory differentiation, material prediction, 3D shape reconstruction, and object re-identification across 83 diverse real-world objects.The system combines hardware and algorithm advances with heuristic exploration and learning-based models.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Works

• Tactile sensing includes vision-based, barometric, capacitive, piezoresistive, magnetic, and thermal approaches, while SonicSense explores acoustic vibration as another modality.The paper describes acoustic vibration as offering low-power detection of high-frequency object vibrations.Source: §2 · p. 2
• Prior acoustic sensing studies estimate selected object states, whereas SonicSense targets inventory status, material classification, shape reconstruction, and re-identification.The cited prior tasks include liquid amount, materials, classes, shapes, and poses [7] [36] [15].Source: §2 · p. 2
• Existing robot acoustic-sensing work often uses few primitive objects, homogeneous materials, small models, limited data, or manual and fixed-pose collection.SonicSense instead explores larger, more diverse real-world objects with complex geometry and heterogeneous materials.Source: §2 · pp. 2–3
• Real-world acoustic datasets have used human interactions or impact hammers in noise-controlled settings, complicating their use in real-world robotic perception.The paper contrasts this with direct investigation in real-world robotic systems.Source: §2 · p. 3

3.1 Robot Hand Design

• The four-finger hand uses one-degree-of-freedom fingers and supports tapping, grasping, and shaking for object perception.Each fingertip embeds a piezoelectric contact microphone and carries a counterweight that increases finger momentum and supports strong tapping vibrations; estimated total cost is $215.26.Source: §3.1 · p. 3

3.2 Real-World Object Dataset on Acoustics, Material, Shape, and Category

• The dataset contains 83 objects across nine material categories, including 22.9% with multiple materials and varied geometries and sizes.It includes 54 everyday objects and 29 3D-printed primitive objects, plus scanned meshes, point clouds, and per-point material annotations.Source: §3.2 · p. 3, Figure 3 · p. 3

3.3 Heuristic-Based Interaction Policy

• The heuristic policy probes from top-down and side directions until contact, estimates object height and radius, then grid-samples contact locations.Contact is detected using motor voltage feedback; the policy assumes a maximum object height, hand-graspable size, and objects fixed on the table.Source: §3.3 · pp. 3–4

4.1 Material Classification Model and Training

• The material classifier maps a Mel-spectrogram from an impact sound to the material label for that contact location.Its architecture uses convolutional and multilayer perceptron layers and is trained with cross-entropy loss.Source: §4.1 · p. 4

4.2 Shape Reconstruction Model and Training

• The shape model maps sparse contact points to a dense, complete object point cloud, using learned shape priors to address noisy and ambiguous contact locations.It uses a Point Completion Network-inspired encoder and decoder, optimizes Chamfer Distance, and pre-trains on synthetic data before increasing the real-data share during training.Source: §4.2 · pp. 4–5

4.3 Object Re-identification Model and Training

• The re-identification model fuses Mel-spectrograms and corresponding contact points to predict an object label among 82 objects.The model encodes audio and contact points separately, then fuses their features; training uses cross-entropy loss.Source: §4.3 · p. 5

4.4 Implementation Details

• All models are trained on one NVIDIA GeForce RTX 3090 GPU, with training lasting less than an hour to a few hours depending on the task.Hyperparameters are selected by grid search using validation performance.Source: §4.4 · p. 5

5.1 Characterizing Basic Sensing Capabilities of SonicSense

• Under increased ambient noise, fingertip signal amplitude barely changed while the external air microphone’s amplitude rose by a few thousand units.The comparison suggests the hardware focuses on vibration signals from physical contact.Source: §5.1 · p. 5, Figure 6 · p. 6
• Container vibration features distinguished solid-object numbers and geometries as well as subtle, continuous liquid states.The study used container rotation with dice and horizontal shaking with bottles of different water amounts; t-SNE showed clear clusters from twelve acoustic features.Source: §5.1 · p. 6, Figure 7 · p. 6

5.2 Material Classification

• 0.523 F1 was the initial material-classification result, prompting iterative filtering of low-occurrence predictions under a local uniformity and smoothness assumption.Data were split by object, with 60, 11, and 11 objects for training, validation, and testing to avoid overlapping acoustic events.Source: §5.2 · p. 6

5.3 Shape Reconstruction

• Primitive objects generally had near-perfect shape reconstruction, and the method also reconstructed objects with concave geometries.The figure describes reconstructed point clouds from sparse, nonuniform, and noisy contact positions.Source: §5.3 · p. 7, Figure 10 · p. 7
• Spray-nozzle and bottle-cap predictions failed amid few spray objects and complex geometry in small regions.The authors also note that the wine-glass result is likely achieved through simulation augmentation because only one wine glass appears in the real-object dataset.Source: §5.3 · pp. 7–8

5.4 Object Re-identification

• 92.52% test accuracy re-identified objects from fifteen random contact locations, versus 1.33% for the random baseline and 43.45% for the nearest neighbor baseline.The evaluation used fifteen spectrogram channels from random contact locations, analogous to a few robot taps.Source: §5.4 · p. 8, Figure 11 · p. 8
• 84.11% acoustic vibration-only accuracy exceeded 48.89% tapping point-only accuracy in the re-identification ablations.The authors report that adding rough contact positions further improves performance over acoustic input alone.Source: §5.4 · p. 8

6 Conclusion

• SonicSense combines hardware and software for in-hand acoustic vibration perception on a multi-finger robot hand.The authors report inventory estimation for solid and liquid objects, material classification, 3D shape reconstruction, and object re-recognition.Source: §6 · p. 8
• The evaluation uses diverse real-world objects with complex geometry and heterogeneous materials, and tests on unseen objects rather than different contact events on the same objects.A simple exploration policy avoids manual interaction and fixed-pose replay.Source: §6 · p. 8
• Current work assumes objects are fixed on the table, excludes multiple objects, and does not consider complex dexterous manipulation.The authors also identify integrating multiple sensing modalities as future work.Source: §6 · p. 8

A. Hardware Configuration of Acoustic Robot Hand

• The hand uses a 3D-printed PLA structure, an LX-224 servo motor with 20kg·cm torque and accurate position and voltage feedback, contact microphones, and two counterweights totaling 40g on each fingertip.The motor controller and audio jack are inside the palm.Source: A. Hardware Configuration of Acoustic… · p. 13

Contact Position Calculation

• Fingertip positions are calculated from detected contact events, recorded joint angles, and link lengths from the hand's CAD model.The fingertip is represented at the center of the fingerprint.Source: Contact Position Calculation · p. 13
• The hand coordinate frame uses the palm center as its origin, with the y-axis parallel to the finger joints and the z-axis perpendicular to the palm.The finger-joint coordinate frames share the hand frame's orientation, and each joint-frame origin is at a motor center.Source: Contact Position Calculation · p. 13
• The hand is mounted to the Franka Emika Panda arm with aligned hand and end-effector coordinates, enabling contact positions to be transformed into the arm frame.The hand and end-effector fingertip coordinates are treated as equal before applying the end-effector-to-arm transformation.Source: Contact Position Calculation · p. 14

B. Interaction Policy for Data Collection

• Two side taps at different heights provide valid contacts from which the robot roughly estimates object height.The object is fixed at the center of a black base, and the highest detected contact point informs the estimate.Source: B. Interaction Policy for Data… · p. 14, Figure 13 · p. 14
• The third finger searches for the highest edge by moving toward the base centerline and stepping lower after a no-contact pass.The contact point's distance from the centerline estimates radius, while its distance from the base plane estimates height.Source: B. Interaction Policy for Data… · pp. 14–15, Figure 14 · p. 15
• After dimension estimation, sequential taps cover the top, left, and back surfaces, with 2 cm radius and 20 cm height thresholds guiding coverage choices.The radius threshold determines whether top tapping is useful, and object height determines the starting scale for left- and back-side tapping.Source: B. Interaction Policy for Data… · pp. 15–16, Figure 15 · p. 15, Figure 16 · p. 16
• Motor voltage feedback detects contact more reliably than acoustic signals, after which forward kinematics localizes the contact point in 3D.External force increases motor current and decreases the voltage reading; the collected tapping data use valid contact events from 83 objects.Source: B. Interaction Policy for Data… · p. 16

C. Acoustic Vibration Recording and Processing

• Each tap has a five-second recording, from which a 20,000-sample segment is selected around a local peak in 1,000-sample average absolute amplitude.The extracted signal is converted into a 64 × 64 Mel spectrogram; recordings without clear impacts, such as foam, may instead yield motor noise.Source: B. Interaction Policy for Data… · p. 16, C. Acoustic Vibration Recording and… · pp. 16–17

Dataset Construction

• Two depth-camera point clouds are fused and aligned with annotated object point clouds to assign each contact the nearest point's material label.These assigned labels serve as ground truth for training material classification.Source: Dataset Construction · p. 17
• Material classification uses tapping data from 82 objects, excluding a disinfecting wipe whose paper-and-plastic signal was considered too unique and out of distribution.Object-based splits contain 60 training, 11 validation, and 11 testing objects; training and validation classes are balanced by duplication.Source: Dataset Construction · p. 17

Material Classification Model Architecture

• The material-classification network architecture is specified in Tab. 3.The supplied passage identifies the table but does not provide its architecture details.Source: Material Classification Model… · p. 17, Table 3 · p. 18

Material Classification Hyperparameters and Results

• Across three random splits, the model is selected by validation accuracy and evaluated with average F1 because the testing dataset is unbalanced.Tab. 4 lists hyperparameter selections and results, while Fig. 17 presents initial and refined predictions as confusion matrices.Source: Material Classification… · pp. 17–18, Table 4 · p. 18, Figure 17 · p. 18

Dataset Construction

• Shape reconstruction uses sparse contact points from 15 shape categories, with category-based training, validation, and testing splits.Training data are duplicated and augmented, and the study adds 629 simulated models while keeping validation and test objects real-world.Source: Dataset Construction · pp. 18–19, Table 5 · p. 19

Shape Reconstruction Model Architecture

• The shape-reconstruction network architecture is specified in Tab. 6.Tab. 7 lists hyperparameters and results across three random data splits.Source: Shape Reconstruction Model Architecture · p. 19, Table 6 · p. 19, Shape Reconstruction Hyperparameters… · p. 19

Dataset Construction

• Object re-recognition uses vibration signals and contact points from 82 objects, splitting each object's taps into training, validation, and testing sets.The augmented dataset contains 410,000 training, 4,100 validation, and 4,100 testing data points.Source: Dataset Construction · p. 20

Object Re-recognition Model Architecture

• The object re-recognition model architecture is specified in Tab. 8.Tab. 9 reports results across three random splits and identifies acoustic vibration and contact-point location as input modalities.Source: Object Re-recognition Model Architecture · p. 20, Table 8 · p. 20, Object Re-recognition Hyperparameters… · p. 20

G. Implementation Details for Resistance Test Against Ambient Noise

• The ambient-noise resistance test places an air microphone beside the fourth finger and plays Gaussian white noise toward the robot hand.An iPhone sound-level-meter app monitors noise while its microphone, the fingertip surface, and the air microphone are aligned in one plane.Source: G. Implementation Details for… · p. 21

• Tab. 10 lists parameters for converting waveforms into spectrograms and extracting twelve descriptors.Source: H. Parameters of Acoustic Vibration… · p. 21, Table 10 · p. 21
• Liquid shaking and pouring use a larger hop length because their acoustic clips are longer.Source: H. Parameters of Acoustic Vibration… · p. 21
• Rigid-object collisions use a smaller hop length and a larger highest frequency to capture brief, higher-frequency vibrations.Source: H. Parameters of Acoustic Vibration… · p. 21

• Twelve interpretable descriptors summarize acoustic signals, including RMS, spectral centroid, bandwidth, MFCCs, chroma, and tonnetz.Source: §I · pp. 21–22
• Each container subtask used 30 trials, with descriptor features averaged across four fingers and rescaled to the 0–1 range.Source: §I · p. 22
• The rescaled descriptors were reduced to 2D to test separation of events triggered by different object states; Tab. 11 reports means and SEM.Source: §I · p. 22, Table 11 · p. 22