Source-linked AI summary
The Boombox: Visual Reconstruction from Acoustic Vibrations
Boyuan Chen, Mia Chiquier, Hod Lipson, Carl Vondrick
TL;DR
Estimating object states inside containers is important for robotics, but camera-based perception can be limited by occlusion and poor illumination. The Boombox uses a convolutional network to reconstruct contents from contact-microphone vibrations, and the authors report reconstruction within centimeters even under total occlusion and poor illumination.
Problem
Estimating object states inside containers is important for robotic grasping and retrieval, but camera-based state estimation can be impractical when objects are occluded.
Source: §1 · p. 1Method
The Boombox uses contact-microphone vibrations and a convolutional network to reconstruct the visual scene inside a container.
Source: §1 · p. 1Results
The Boombox reconstructs the visual scene within centimeters, even under total occlusion and poor illumination.
Source: §1 · p. 1Takeaways & Limitations
The results suggest sound is a promising sensing modality when cameras are unavailable, including under occlusion or poor illumination.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Interacting with bins and containers is a fundamental task in robotics, making state estimation of the objects inside the bin critical. While robots often use cameras for state estimation, the visual modality is not always ideal due to occlusions and poor illumination. We introduce The Boombox, a container that uses sound to estimate the state of the contents inside a box. Based on the observation that the collision between objects and its containers will cause an acoustic vibration, we present a convolutional network for learning to reconstruct visual scenes. Although we use low-cost and low-power contact microphones to detect the vibrations, our results show that learning from multimodal data enables state estimation from affordable audio sensors. Due to the many ways that robots use containers, we believe the box will have a number of applications in robotics. Our project website is at: boombox.cs.columbia.edu
1 Introduction
Cameras can be unreliable for estimating objects inside containers because of occlusion, poor lighting, and fragile calibration. The Boombox uses contact-microphone vibrations and a convolutional network to reconstruct the contents’ visual scene.
1 Introduction
- Accurately localizing and estimating object states inside containers matters for robotic grasping and retrieval tasks.Source: §1 · p. 1
- Camera-based state estimation can be impractical inside containers because objects may be occluded, lighting may be poor, and 3D calibration can be fragile during contact and collisions.Source: §1 · p. 1
- The Boombox uses a convolutional network to reconstruct the visual scene inside a container from its vibrations, including under total occlusion and poor illumination.The box uses contact microphones to detect vibrations produced when objects interact with it.Source: §1 · p. 1, Figure 1 · p. 2
2 Related Works
Prior work integrates sound and vision for sound generation, property recognition, and representation learning, while this paper predicts visual structure inside containers from sound. It also addresses a broader reconstruction target than prior robot audio work described here.
2 Related Works
- This work predicts the 3D visual structure inside a container from sound, unlike prior audiovisual work on sound generation, property recognition, or representation learning.Source: §2 · p. 2
- Prior robot audio work includes predicting actions that caused sounds, calibrating bouncing dynamics, and estimating a falling cube’s trajectory; this paper predicts the scene under full occlusion.The paper outputs RGB and depth images for three different blocks.Source: §2 · p. 2
- Analytical sound-localization features such as arrival-time and amplitude differences are difficult to extract from moving, narrow-band signals, and they localize objects rather than recover shape and orientation.Source: §2 · p. 2
3 The Boombox
The Boombox combines a container fitted with contact microphones and a learning approach that uses vibrations to recover object structure. Its multimodal dataset pairs recordings of objects settling in the box with camera images.
3 The Boombox
- The Boombox is a hardware-and-software platform for reconstructing visual structure inside containers from sound.Source: §3 · p. 2, §3.2 · p. 3
3.1 Detecting Vibrations
- A standard open-top plastic storage container uses contact microphones on its walls to detect solid-box vibrations caused by object collisions.The microphones detect vibration in solid objects rather than air; four audio channels are recorded.Source: §3.1 · p. 2, Figure 2 · p. 3
3.2 Vibration Characteristics
- Analytical extraction of arrival-time characteristics is difficult because objects bounce, signals interfere, echoes occur, and close microphones make timing differences noise-sensitive.The model is trained to identify robust signal features for reconstructing shape, orientation, and position.Source: §3.2 · p. 3
3.3 Multimodal Training Dataset
- The training data pairs synchronized microphone recordings with RGB and depth images captured by a top-down camera during object drops.Three wooden blocks of different shapes produced 1,575 sequences across diverse falling trajectories.Source: §3.3 · pp. 3–4, Figure 2 · p. 3
3.2 Vibration Characteristics
- Vibrations captured by two microphones contain characteristics that must be combined over time to reconstruct object position, orientation, and shape.Source: §3.2 · p. 3, Figure 3 · p. 4
3.3 Multimodal Training Dataset
- Objects bounce around before settling, producing moving sound sources and repeated, potentially interfering vibrations.Source: §3.2 · p. 3, Figure 4 · p. 4
4 Predicting the Visual Scene from Vibration
The section describes a convolutional model that maps four-microphone vibration spectrograms to top-down RGB and depth images, trained to predict the contents’ final stable state.
4.1 Model
- The model predicts top-down RGB and depth images from spectrograms of vibrations recorded by four microphones.Four audio clips are concatenated along the time and frequency dimensions after Fourier transformation and conversion to Mel scale.Source: §4.1 · p. 4
- A fully convolutional encoder-decoder gives each output pixel access to the full spectrogram, while a multi-scale decoder combines two upsampling branches.The branches’ outputs are concatenated at each decoder layer, except the final layer, which uses one transposed convolution.Source: §4.1 · p. 4
4.2 Learning
- Training targets the final stable image; RGB uses expected squared error, while depth uses expected L1 distance to address noisy, outlier-prone depth ground truth.After learning, both image types are predicted from vibration alone, with visual data serving only as supervision during training.Source: §4.2 · pp. 4–5
4.3 Implementation Details
- The networks are trained for 500 epochs with Adam, batch size 32, and learning-rate reductions at epochs 20, 50, and 100.Training uses one NVIDIA RTX 2080 Ti GPU, with the learning rate halved at each reduction.Source: §4.3 · p. 5
5 Experiments
The experiments test whether acoustic vibrations can recover object state across mixed-shape and known-shape settings. The model outperforms baselines in mixed-shape reconstruction, while ablations show that microphone layout, amplitude, and timing information matter to its predictions.
5 Experiments
- Because each drop yields a unique resting pose, the dataset is split 80/10/10 for training, validation, and testing, with means and standard errors reported across three random seeds.The results use various dataset splits.Source: §5 · p. 5
5.1 Evaluation Metrics
- IoU compares predicted and ground-truth segmentation masks, while localization score counts predictions whose center error is below half the ground-truth box diagonal.The localization metric is designed to assess whether the block is in the right spatial position.Source: §5.1 · p. 5
5.2 Baselines
- The TDoA baseline uses GCC-PHAT and selects the box location closest to the estimated arrival-time difference; random sampling uses a training-set image as prediction.The mixed-shape comparison also includes an average-bounding-box baseline.Source: §5.2 · pp. 5–6, §5.3 · p. 6
5.3 Reconstruction with Mixed Shapes
- With mixed shapes, the model outperforms TDoA, random-sampling, and average-bounding-box baselines at localization and shape prediction.Qualitative examples show predictions of object shape, position, and orientation.Source: Figure 6 · p. 6, §5.3 · p. 6, Figure 5 · p. 6
5.4 Reconstruction with Known Shape
- With shape known, the convolutional network achieves the best performance for most shapes on both evaluation metrics while reconstructing position and orientation.Figure 7 reports means and standard errors across three random seeds.Source: §5.4 · p. 6, Figure 7 · p. 7
- Training with multiple shapes improves block and stick performance by a large margin, but cube performance drops amid confusion with smaller-surface-area shapes.The reported confusion is between the cube and the stick or block.Source: Figure 9 · p. 7, §5.4 · p. 7
5.5 Ablations and Analysis
- Test-time microphone flips nearly eliminate performance; removing amplitude and shifting timing also worsen it, with timing shifts moving predictions spatially.The authors interpret these patterns as evidence that the model uses microphone layout, amplitude, and time-difference-of-arrival information.Source: Figure 8 · p. 7, §5.5 · p. 8
- t-SNE embeddings often cluster by position and orientation, while gradual color transitions suggest smooth spatial interpolation.The visualization colors points by ground-truth position and orientation.Source: §5.5 · p. 8
6 Conclusion
The Boombox combines a low-cost container with a convolutional network to reconstruct images of its contents from sound. The authors describe sound as a promising sensing modality when cameras are unavailable, including under occlusion or poor illumination.
6 Conclusion
- The low-cost Boombox integrates a convolutional network that reconstructs images of container contents from sound, which the authors present as promising when cameras are unavailable.The conclusion identifies occlusion and poor illumination as examples of those conditions.Source: §6 · p. 8
- Future work aims to use acoustic signatures to infer objects’ 3D geometry and materials and to expand the object collection.Source: §6 · p. 8
- The authors also identify audio as a potential signal for understanding object dynamics during multi-object interactions such as collisions.Source: §6 · p. 8