Source-linked AI summary
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
Andrew Owens, Jiajun Wu, Josh H. McDermott, William T. Freeman, Antonio Torralba
TL;DR
The paper asks whether ambient sound can supervise visual learning despite loose correspondence between what is heard and what is seen. It trains a CNN to predict sound summaries from video frames, then evaluates the resulting representation. The representation contains information about objects and scenes, supports competitive recognition performance, and yields object-selective internal units.
Problem
Ambient sounds are ubiquitous and informative, but their loose association with visible content makes their usefulness for visual representation learning uncertain.
Method
A convolutional neural network predicts a statistical summary of held-out sound from each video frame, using ambient audio as supervision.
Results
The learned representation contains information about objects and scenes, performs competitively on object and scene recognition, and develops object-selective units.
Takeaways & Limitations
Ambient sound can provide supervision for learning visual representations from unlabeled audio-visual data.
Takeaways & Limitations
Audio-visual correspondence is loose because sound-producing objects may be outside the visual field or may not produce characteristic sounds.
Abstract
from arXiv · showhide
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds. This paper extends an earlier conference paper, Owens et al. 2016, with additional experiments and discussion.
1 Introduction
The paper proposes using ubiquitous ambient sound as supervision for visual representation learning, despite loose audio-visual correspondence. A CNN predicts sound summaries from video frames and learns representations informative about objects and scenes.
- Motivation: Ambient sound and vision often describe the same world structures, providing a rich signal for learning visual representations.Sound can convey semantics, events, and mechanics that may be largely orthogonal to information available from images alone.
- Motivation: Sound prediction must detect structures shared across modalities while ignoring visual changes that do not affect the sound.Figure 1 highlights invariance to illumination, camera angle, and people appearing in the frame.
- Approach: The method trains a CNN to predict a statistical summary of held-out sound associated with each video frame.The task is formulated as classification from a visual frame to the sound summary recorded at the same time.
- Results: The learned features support object and scene recognition with performance competitive with state-of-the-art unsupervised and self-supervised methods.The paper also evaluates the representation through visualizations of its internal units.
- Results: Intermediate CNN layers learn object-selective detectors from unlabeled audio-visual signals without explicit human annotation.The paper presents this as an audio-visual analogue of object detectors emerging in scene-recognition networks.
2 Related Work
The work builds on self-supervised visual learning, audio-visual learning, and computational sound representations. It differs from imagery-only supervision by using complementary sound, while extending earlier audio-visual approaches and comparing with newer self-supervised methods.
- Audio representation: The approach uses a sound representation previously applied to sound recognition and synthesis tasks.The representation is based on sound textures.
- Audio-visual learning: Earlier audio-visual models jointly trained audio and visual networks by minimizing disagreement between modality-specific codebooks.Those methods assigned shared labels to paired audio and visual examples and iteratively refined the codebooks.
- Self-supervised visual learning: Prior self-supervised methods learn visual representations from natural signals such as camera motion, frame tracking, image patches, or transformations.These approaches derive prediction targets from video or static images rather than human annotations.
- Relation to prior work: This work uses sound as a complementary supervisory modality that provides information about objects and scenes and is largely invariant to visual transformations.The paper contrasts sound-based supervision with methods whose signals come entirely from imagery.
- Audio-visual learning: The work is related to an earlier model predicting soundtracks for videos showing people striking objects, where actions visually indicate the sounds.The present work instead concerns ambient sound supervision in less explicitly sound-producing visual settings.
- Expanded paper: The expanded paper adds comparisons with human-annotated audio labels, class activation map visualizations, and a broader comparison of learned image features.It also situates the method among newer self-supervised approaches, including colorization and motion prediction.
3 Learning to predict ambient audio
The method predicts an image’s ambient sound summary rather than precise instantaneous audio, using sound-texture categories or binary codes as visual supervision. These labels are derived from several-second audio summaries and trained with a CNN on video frames.
- 3.1 Statistical sound summaries: The model predicts a sound representation from a single video frame because precise sound timing is difficult to infer from visual information.The target is averaged over several seconds to better match the time scale of visual objects.
- 3.1 Statistical sound summaries: The sound representation is a 502-dimensional texture vector containing normalized marginal moments, correlations, modulation power, and energy.The texture is computed from audio over a 3.75-second window using cochlear and modulation statistics.
- 3.2 Predicting ambient sound from images: The Clustering model converts sound textures into image labels by assigning each texture to its nearest k-means centroid.Examples far from their cluster centroid are removed during training.
- 3.2 Predicting ambient sound from images: Audio clusters often contain shared visual content, including indoor scenes with children, crowds, water scenes, and outdoor scenes with light wind.The clusters were defined from audio, but many associated images share objects or broader scene attributes.
- 3.2 Predicting ambient sound from images: The Binary model projects each sound texture onto 30 principal components, thresholds the projections, and predicts the resulting code with sigmoid outputs.Training uses cross-entropy loss for the resulting multi-label classification problem.
- 3.2 Predicting ambient sound from images: The Spectrum model instead predicts the mean cochlear-channel values from a 33.3 ms interval centered on the video frame.This provides a comparison target based on an approximately instantaneous frequency spectrum rather than a full sound texture.
4 What does the network learn to detect?
The network learns many object-selective units from ambient-sound supervision, including detectors for objects associated with characteristic sounds. These detectors differ from those learned through other supervision and remain broadly similar when evaluated on still-image datasets.
- Comparing supervision types: The sound-supervised model was compared with CNNs trained on Places, egomotion, and tracking tasks to assess how supervision shapes object selectivity.The comparison used the same neuron-visualization methodology across networks.
- Identifying object-selective units: 91 of 256 units were object-selective under the paper’s 60% activation criterion.Units were labeled using masked top activations and human annotation.
- Comparing supervision types: Compared with tracking supervision, ambient-sound supervision produced more high-level object detectors and more detectors for characteristic-sound objects such as people, babies, and waterfalls.The Places model had more total object-selective units, but fewer detectors for some characteristic-sound categories than the sound-supervised model.
- Linking detectors to sound: 43.7% of videos containing characteristic-sound object detections also contained the corresponding object sounds.The estimate was based on listening to the top 30 video clips for each unit.
- Cross-dataset evaluation: The model yielded 67 object-selective detectors on SUN and ImageNet, compared with 91 on YFCC100m, while preserving a similar object distribution.Some categories, including baby, appeared less often among detectors on the still-image datasets.
- Visualizing sound predictions: Class activation maps linked predicted speech to faces and flowing-water sounds to waterfalls, connecting visual structures with sound categories.The visualizations used heat maps for images assigned high probability to selected audio classes.
5 Evaluating the image representation
The learned sound-supervised image representation was evaluated on object, scene, and detection tasks. It achieved competitive recognition performance, while results depended on feature design, audio representation, and fine-tuning.
- Evaluation setup: The evaluation used CNN activations with linear SVMs for object and scene recognition, plus CNN initialization for Fast R-CNN detection.Object recognition used PASCAL VOC 2007, while scene recognition used SUN397.
- Recognition results: The binary-coding sound model achieved performance comparable to other unsupervised learning methods, although colorization was the overall best-performing model.The comparison covered object recognition on PASCAL VOC 2007 and scene recognition on SUN397.
- Recognition results: Sound-texture models outperformed the frequency-spectrum model, and global pooling of convolutional features revealed substantial semantic information.The authors attribute the sound-texture advantage to additional time-averaged statistics.
- Object detection: The sound-trained model outperformed random initialization and Agrawal et al. for detection, but Doersch et al. and other models performed better.Fine-tuning substantially changed the network, making performance sensitive to training parameters.
- Audio prediction: The sound-prediction model classified the correct sound cluster 15.8% of the time, versus 3.3% chance and 6.6% for the most-common-label baseline.The clustering-based model used 30 audio clusters.
- Audio representation: Increasing the number of audio clusters produced a small improvement beyond 30, whereas using two clusters substantially reduced visual-feature performance.Fewer clusters also reduced the effective training set because poorly represented examples were removed.
6 Studying the role of audio supervision
Audio supervision was examined by replacing sound-derived labels with visual features and by using human-annotated AudioSet categories. The results favor semantically informative audio labels, while exposing limits of clustering and annotation purity.
- Visual-feature control: Replacing sound textures with visual texton histograms produced significantly worse object- and scene-recognition performance.The resulting convolutional units were rarely selective for objects and instead often responded to visual “stuff” such as grass and water.
- Visual-feature control: The authors attribute the audio-based advantage to audio labels being largely invariant to visual transformations, which requires greater visual generalization.The visual-texton model could approximate its labels without achieving comparable high-level generalization.
- Audio representation: Audio clusters do not map cleanly one-to-one onto sound sources, making cluster membership a messy proxy for semantic audio categories.The clustering procedure nevertheless produced groups containing common objects and scene attributes.
- Human-annotated sounds: The AudioSet result suggests substantial room to improve by choosing audio representations that better capture semantics.AudioSet included 527 categories, including music, speech, vehicles, animals, and explosions.
- Human-annotated sounds: Human-annotated AudioSet supervision performed significantly better than state-of-the-art unsupervised methods but still lagged behind ImageNet-based training.It also substantially outperformed the authors’ unsupervised models trained on the same dataset.
- Human-annotated sounds: AudioSet labels may incorporate visual information because annotators watched videos and visual classifiers contributed during collection.The authors therefore treat these labels as an upper bound on purely audio-derived supervision.
- Learned units: AudioSet-trained and Flickr-video-trained networks shared many face-selective units but differed in their abundance of musical-instrument-selective units.The instrument units likely emerged from AudioSet’s many music- and instrument-related categories.
7 Discussion
The discussion presents ambient sound as an abundant, complementary supervisory signal for learning visual representations. The approach demonstrates semantic information in the learned features but remains limited by the chosen audio representation.
- Motivation: Sound is abundant without human annotations, conveys information about objects and scenes, and may provide information unavailable from unlabeled images alone.Its complementarity with vision is a central motivation for using sound as supervision.
- Contribution: The proposed CNN predicts a statistical sound summary from a video frame, and its learned representation contains information about objects and scenes.The evidence comes from visualizations and recognition experiments.
- Limitations and future work: The sound-texture representation detects some objects, including people and waterfalls, but a better audio representation may enable learning about other objects.The authors identify jointly learning audio and visual representations as a direction for improvement.
A Sound textures
The paper represents several-second audio clips with sound-texture statistics derived from cochlear-like filtering, correlations, modulation responses, marginal moments, and normalization.
- Sound-texture construction: Sound textures are computed from audio clips using a perceptually inspired model that assumes stationarity within a 3.75-second temporal window.The method follows McDermott and Simoncelli (2011).
- Subband envelopes: The input waveform is filtered with a bank of bandpass filters to obtain cochleagram features, whose envelopes are resampled to 400Hz and compressed with a 0.3 power transform.The envelope computation uses the Hilbert transform after cross-correlation with the filter bank.
- Correlations: The representation includes correlations between selected cochlear channels whose index separations are 1, 2, 3, or 5.These correlations form a vector of correlation values.
- Modulation filters: It also includes responses from 10 logarithmically spaced modulation bandpass filters spanning 0.5 to 200Hz.The modulation responses are computed separately for each band.
- Marginal statistics and normalization: Marginal statistics summarize each cochlear channel with its mean and standard deviation, while loudness is the median energy across timesteps; features are normalized for gain differences.Normalization divides cochlear features by loudness and scales modulation responses by variance.