Source-linked AI summary
Ambient Sound Provides Supervision for Visual Learning
Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio Torralba
TL;DR
Learning visual representations from unlabeled images alone can miss information available in sound, while human annotations are costly and audio-visual correspondence is imperfect. The paper trains a CNN to predict statistical summaries of ambient sound from video frames, then evaluates the resulting representation. It finds information about objects and scenes, recognition performance comparable to state-of-the-art unsupervised methods, and object-selective internal units.
Problem
Human annotations are costly, and image-only learning may lack sound-based information about semantics, events, mechanics, objects, and scenes.
Method
A convolutional neural network predicts a statistical summary of ambient sound associated with a video frame.
Results
The learned representation contains information about objects and scenes, supports recognition tasks with performance comparable to state-of-the-art unsupervised methods, and develops object-selective units.
Takeaways & Limitations
Ambient sound available in consumer videos provides a useful training signal for visual learning without explicit human annotation.
Takeaways & Limitations
The study considers one sound-texture representation and calls for systematic study of when sound does and does not reveal visual objects.
Abstract
from arXiv · showhide
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds.
1 Introduction
The paper proposes ambient sound as an abundant, complementary supervisory signal for learning visual representations, despite loose audio-visual correspondence. A CNN predicts statistical summaries of sound from video frames, and its features support recognition while developing object-selective units.
- Sound prediction may reveal objects and scenes associated with distinctive noises, including people, cars, flowing water, wind, and refrigerators.
- Ambient sound is ubiquitous and free compared with human annotations, while providing information about semantics, events, and mechanics beyond images alone.
- Audio-visual correspondence is imperfect because sound sources may be outside the visual field and characteristic sounds may be absent.
- The model classifies a video frame’s statistical summary of the sound recorded at that time, then validates the learned representation on object and scene information.
- The learned features achieve similar performance to state-of-the-art unsupervised and self-supervised methods on object and scene recognition.
- Intermediate CNN layers develop object-selective detectors from unlabeled audio-visual signals, including objects associated with characteristic sounds.
2 Related Work
The work extends natural- or self-supervised visual learning by using sound, rather than imagery alone, as a complementary signal. It frames sound prediction as a classification task and distinguishes natural videos from settings where actions visibly indicate sound.
- Self-supervised methods learn visual representations by predicting targets derived from natural signals rather than human annotations.
- Prior video methods use camera motion or cross-frame tracking, while image methods predict patch locations or transformation invariance.
- This work differs by deriving supervision from sound, a modality complementary to vision and informative about objects and scenes.
- The sound-prediction task uses straightforward classification, enabling a network design resembling object- and scene-recognition models.
- Related audio-visual work predicts sounds visually indicated by actions, whereas the natural videos here often contain sound sources outside the frame.
3 Learning to predict ambient audio
The method represents ambient audio with time-averaged sound textures and trains CNNs to classify sound-derived labels from single video frames. Audio clusters often align with recurring objects and scene attributes, motivating the learned visual supervision.
- The model predicts a frame’s corresponding sound from a single image, using the task to learn transferable visual features.
- 3.1 Statistical sound summaries: Sound summaries average audio over 3.75 seconds using 32 cochlear-like filters, Hilbert envelopes, compression, and texture statistics.
- 3.1 Statistical sound summaries: Each image receives a 502-dimensional sound-texture vector normalized for gain and rescaled across feature types.
- 3.2 Predicting sound from images: Instead of direct regression, the method defines explicit sound categories and formulates prediction as classification for easier network analysis.
- 3.2 Predicting sound from images: The Clustering model uses k-means centroids as image labels, and removes training examples farther than the median distance from their cluster centroid.
- 3.2 Predicting sound from images: Audio clusters often contain common objects and scene attributes, including children indoors, crowds, water scenes, and outdoor wind.
- 3.2 Predicting sound from images: The Binary model projects sound textures onto 30 principal components, thresholds them into binary codes, and predicts them with sigmoid outputs and cross-entropy loss.
- 3.2 Predicting sound from images: The Spectrum model approximately predicts the frequency spectrum in a 33.3-millisecond interval centered on the input frame.
4 Results
The sound-trained CNN learns object-selective units and image representations that support object and scene recognition. Its recognition performance is comparable to other annotation-free methods, while its learned detectors differ from those of competing self-supervised models.
- Object-selective units: 91 of 256 units were object-selective under the study’s detector criterion.A unit counted as selective when at least 60% of its top 60 activations contained the same object.
- Comparisons with other learned representations: 72 of 256 units in the tracking-based model were object-selective, often responding to textural stuff such as grass, ground, and water.The egomotion-based model had 27 such units, while patch-position units were often selective for image position.
- Object-selective units: The sound-trained model contained more units selective for objects with characteristic sounds than the Places scene-recognition network.A significant fraction of the sound-trained units were selective to people, including adults, babies, and crowds.
- Object-selective units: 43.7% of the top 30 video clips per unit contained the sound made by the object to which the neuron was selective.This analysis was performed for the object-selective units’ top responses.
- Recognition performance: The binary-coding sound model obtained comparable performance to other unsupervised learning methods on the evaluated recognition tasks.The reported recognition evaluations used learned CNN features for object and scene recognition.
- Recognition performance: Sound-texture models outperformed the model predicting only the frequency spectrum, and time-averaged sound statistics were reported as helpful.The sound-based models significantly outperformed other methods when conv5 features were globally pooled.
5 Discussion
Sound is proposed as an abundant, annotation-free supervisory signal for learning visual representations. The resulting CNN representation contains information about objects and scenes, while the choice of sound representation remains an open direction.
- Sound is abundantly available without human annotations and conveys information about objects and scenes, making it a potential supervisory signal for visual learning.
- A CNN trained to predict a statistical sound summary from a video frame learns image representations containing information about objects and scenes.
- The paper leaves open whether alternative audio representations would enable the model to learn about additional object types.
A1 Sound label space
The authors examine how learned image-feature quality changes with the number of audio-label clusters. Recognition performance on PASCAL VOC2007 increases as the cluster count grows, with 30 clusters used in the experiments.
- Recognition performance on PASCAL VOC2007 increases with the number of clusters defining the audio label space.The experiments used 30 clusters.
A2 Additional unit visualizations
The supplementary experiments extend object-selective-unit analyses beyond the Flickr evaluation setting. They compare selectivity across datasets and provide additional neuron visualizations, while noting fewer total detectors for SUN and ImageNet.
- Additional object-detector histograms and neuron visualizations extend the analysis to methods and units beyond the main Flickr-based figures.
- 67 object-selective units were found using SUN and ImageNet images, compared with 91 for the Flickr dataset.Some categories, including baby, appeared relatively less often in SUN and ImageNet.
- The SUN and ImageNet visualization sample was drawn from 200,000 images sampled equally from the two datasets.
A3 Sound textures
The paper represents ambient audio with sound-texture statistics derived from cochlear and modulation analyses. Supplementary experiments vary the audio-label cluster count and selectivity threshold, and visualize object-selective units across datasets and methods.
- Sound-texture construction: Sound textures are computed by filtering the waveform with bandpass filters, applying a Hilbert transform, resampling to 400Hz, and compressing samples with a 0.3 power.
- Sound-texture construction: The representation includes correlations between cochlear channels whose index differences are 1, 2, 3, or 5.
- Supplementary analyses: Object recognition performance on PASCAL VOC2007 increases with the number of audio-label clusters, while the experiments use 30 clusters.
- Supplementary analyses: Object-selectivity analysis uses a 60% threshold: an object detector requires the object to appear in at least 60% of its top 60 activations.
- Sound-texture construction: Modulation responses use 10 logarithmically spaced bandpass filters with center frequencies from 0.5 to 200Hz.
- Sound-texture construction: Marginal statistics include each cochlear channel’s mean and standard deviation, plus sequence loudness measured as the median timestep energy.
- Supplementary analyses: The supplementary figures count object-selective units across SUN and ImageNet, visualize their top five activations, and quantify units for an additional method.