Source-linked AI summary
Visually Indicated Sounds
Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, William T. Freeman
TL;DR
The paper asks whether sounds caused by visible object interactions can be predicted from silent video as a way to study materials and actions. It uses a recurrent neural network to predict sound features from drumstick-interaction videos and converts them into waveforms with example-based synthesis. The generated sounds were realistic enough to fool participants and carried information about materials and physical interactions.
Problem
The paper asks whether predicting sounds directly caused by visible interactions can study material properties and physical actions without explicitly supplying material labels.
Method
A recurrent neural network predicts sound features from silent videos of people hitting, scratching, and probing objects, followed by example-based waveform synthesis.
Results
The full system significantly outperformed image-matching and random-sound baselines in a real-or-fake experiment, while predicted sounds supported material, action, and reaction classification.
Takeaways & Limitations
Predicted sound can encode information about material properties and physical interactions even when the model is trained to predict sound rather than material labels.
Takeaways & Limitations
The onset detector assumes short impacts contain a single amplitude peak, an assumption that often fails for splash sounds.
Abstract
from arXiv · showhide
Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.
1. Introduction
The paper introduces visually indicated sounds: predicting sounds caused by depicted physical interactions to study materials and actions without explicitly providing material labels. It presents a drumstick-interaction dataset and a recurrent-neural-network-based synthesis pipeline evaluated through perceptual and learned-representation analyses.
- Visually indicated sounds are sounds directly caused by physical interactions shown in video, rather than merely associated with scene content.
- Predicting held-out soundtracks requires models to learn about visible materials and actions from raw audio-visual regularities without explicit material labels.
- The dataset mimics physical exploration by recording people hitting, scratching, and prodding objects with a drumstick.
- The synthesis algorithm maps videos to audio features with a recurrent neural network, then generates waveforms by exemplar matching or parametric inversion.
- The study evaluates predicted sounds psychophysically and analyzes whether the learned predictions encode action and material information.
2. Related work
Related work connects the paper to Foley sound synthesis, sound-based material perception, impact-sound recognition, neural waveform generation, and cross-modal representation learning. The paper distinguishes its approach by predicting sound from vision and examining the predicted sound itself as an informative signal.
- Foley: The algorithm performs automatic Foley by synthesizing plausible sound effects for silent videos without a human sound-effects performer.
- Sound and materials: Prior research shows that impact sounds can reveal material properties and support object or material recognition when visual cues are ambiguous.
- Sound synthesis: The technical approach resembles neural speech-synthesis systems that predict sound features and then generate waveforms, while related work also uses physical simulations for impact sounds.
- Learning visual representations from natural signals: Unlike approaches using temporal proximity or other cross-modal signals, this work learns visual representations by predicting sound from vision.
- Learning visual representations from natural signals: The paper argues that predicted sound itself, rather than only an internal representation, can serve as an informative signal for multiple tasks.
3. The Greatest Hits dataset
The Greatest Hits dataset records close-up human interactions with materials, pairing visual evidence of impacts and reactions with synchronized sound. Its annotations and onset detection support analysis of material, action, reaction, and impact events.
- Dataset collection: The dataset contains videos of people probing environments with a drumstick by hitting, scratching, and poking objects.
- Dataset collection: The drumstick provides a consistent sound-generating action while leaving the object visible enough to observe post-impact reactions relevant to material inference.
- Dataset design: Close-up, observer-focused views capture fine-grained texture and object reactions, unlike full-scene object- or scene-centric datasets.
- Dataset statistics: The collection includes 977 videos from indoor and outdoor scenes, averaging 48 actions per video, with approximately 69% hits and 31% scratches.
- Onset detection: Impact onsets are detected from denoised-audio amplitude peaks using gradient thresholding, mean-shift merging, and non-maximal suppression.
- Semantic annotations: Annotations for approximately 62% of impacts include material, action, reaction, and pixel-location labels, with manually removed or merged categories to reduce errors.
4. Sound representation
The paper represents sounds as compressed subband envelopes derived from filtered waveforms, producing a cochleagram that captures material-related spectral structure. These features support both sound analysis and material classification.
- Sound features: The sound representation filters waveforms into subband envelopes, applies a nonlinearity, downsamples them to 90Hz, and compresses them.The filter bank contains 40 ERB-spaced band-pass filters plus low- and high-pass filters, with Hilbert envelopes.
- Limitation: The onset-detection assumption usually fits scratches and hits but can fail for splash sounds.The paper notes that more sophisticated onset-detection methods could address this case.
- Cochleagram: The resulting representation is a cochleagram whose blocks correspond to sound features used by the prediction model.The paper visualizes mean cochleagrams across material and reaction classes.
- Material information: 45.8% class-averaged accuracy (chance = 5.9%) was achieved when predicting material classes from impact sounds.A linear SVM used subband envelopes as feature vectors, with 260 impacts per class after resampling.
5. Predicting visually indicated sounds
The model predicts audio features from visual appearance and motion, then synthesizes waveforms using either parametric conversion or nearest-neighbor examples. Its recurrent architecture aligns video and audio sequences while modeling temporal dynamics.
- Task formulation: The task maps video-frame sequences to audio-feature sequences and then generates a waveform from the predicted features.
- Image representation: Motion is represented with spacetime images formed from grayscale versions of the previous, current, and next frames.This avoids relying on optical flow, which is difficult to estimate for fast, non-rigid motion.
- Model architecture: A CNN extracts image features, while an RNN processes the resulting time series to predict sound features.Figure 4 depicts this visual-to-audio pipeline and shows images corresponding to one impact.
- Image representation: The input concatenates CNN features from each spacetime image with CNN features from the first color frame.The CNN features come from AlexNet’s penultimate fc7 layer; the first color image reduces computational cost.
- Sound prediction model: An LSTM aligns video and audio rates by replicating each CNN feature vector k = 3 times before updating hidden states over audio timesteps.The hidden states are transformed affinely into predicted sound features.
- Training: Training minimizes a robust timestep-wise loss, while PCA reduces each 42-dimensional sound-feature vector to 10 dimensions before prediction.The robust loss is ρ(r) = log(ϵ + r^2), with ϵ = 1/252; PCA is inverted during evaluation.
- Waveform synthesis: Waveforms are generated either by imposing predicted envelopes on white noise or by transferring the waveform of the nearest training exemplar under L1 distance.The example-based method imposes a stronger natural-sound prior and is more effective for plausible sounds to human listeners.
6. Experiments
The experiments evaluate sound prediction with human judgments, acoustic metrics, material and action recognition, and impact detection. Results show realistic predictions, informative sound representations, and task-specific strengths and failures.
- Psychophysical evaluation: The full system significantly outperformed image-matching methods and random sound sampling in the psychophysical experiment.The full system combined RGB and spacetime input, RNN connections, ImageNet pretraining, and example-based waveform generation; the comparison with random sampling had p < 0.001.
- Acoustic metrics: The network was more accurate than image-matching methods for loudness and spectral-centroid prediction on both mean squared error and correlation coefficients.Loudness was measured from maximum compressed-subband-envelope energy, while spectral centroid was measured around the impact center.
- Material controls: Material information alone was insufficient: material-label oracle models did not outperform the model, which significantly exceeded the random-sampling oracle at p < 10^-4.The oracle study restricted sound examples to videos sharing the input material label.
- Detection and failure modes: Impact detection achieved APs of 43.6% with spacetime images and 21.6% with RGB-only input, while performance varied by material and synthesis method.Parametric synthesis worked better for noisy materials such as dirt than hard materials such as metal, whereas example-based synthesis struggled with textural sounds like splashing water.
- Learned information: Predicted sounds supported class-averaged recognition of materials at 22.7%, actions at 68.6% versus 50% chance, and reactions at 53.5% versus 20% chance.ImageNet pretraining increased material recognition to 28.8% and reaction recognition to 55.2%, but slightly decreased action recognition to 66.5%.
7. Discussion
The work frames visually indicated sound synthesis as a problem requiring models to learn material properties and physical interactions. It introduces a dataset and recurrent-neural-network algorithm, and reports better performance than baselines while identifying future research directions.
- The proposed task requires algorithms to learn about material properties and physical interactions.
- The authors introduce a drumstick-probing video dataset and an algorithm based on recurrent neural networks.
- Psychophysical experiments and automated metrics show that the algorithm significantly outperforms baselines.
- Future work may pursue realistic video-to-sound generation or use sound and material interactions for physical scene understanding.
A1. Model implementation
The implementation uses recurrent models, temporal alignment, feature transformations, and overlapping windows for sound prediction and detection. Supplementary analyses describe feature settings, nearest-neighbor dataset walks, and material-class confusion patterns.
- Detection aligns predicted and ground-truth sound features with small shifts and a smoothness term for neighboring shifts.The maximum shift is 8 samples, approximately 0.09 seconds.
- A weighting variable reduces the importance of silent video portions, while a two-frame output delay gives the RNN additional temporal context.The implementation uses α = 3 and τ = 2.2 for the weighting variable.
- A coloring transformation maps predicted sound-feature statistics into the real-feature space before L1 nearest-neighbor search.The transformation addresses differences such as smaller predicted waveform peak amplitudes.
- Figure A1 evaluates material recognition across frequency-channel counts and shows confusion patterns for sounds predicted by the pretrained model.
- Long videos are processed as overlapping 10-second subsequences, with transitions chosen by least sum-of-squares difference in the overlap.
- The model uses AlexNet features and LSTM layers, with two 256-dimensional hidden-state layers for centered videos.Detection uses three LSTM layers, and sampling-rate differences are handled by upsampling inputs to the final LSTM layer.
- Figure A2 walks through the dataset using AlexNet fc7 nearest-neighbor matches, while varying hit location, material, and action across some sequences.
- Some material confusions occur between leaf and grass, grass and dirt, and cloth and cushion, at 5%, 8%, 8%, and 9% respectively.
A2. Dataset details
The dataset was collected with wooden drumsticks, an SLR camera, external audio recording equipment, and silent-video impact annotations from online workers.
- Data collection used two wooden hickory drumsticks, an SLR camera at 29.97 Hz, and separate audio recording equipment.The audio equipment comprised a ZOOM H1 recorder and Rode VideoMic Pro microphone.
- Online workers labeled impacts by visually examining silent videos without sound.The annotation process used finer-grained categories before merging categories that workers labeled inconsistently.
- Figure A2 provides a dataset walk based on AlexNet fc7 features.