Ambient Sound Provides Supervision for Visual Learning

• Sound prediction may reveal objects and scenes associated with distinctive noises, including people, cars, flowing water, wind, and refrigerators.Source: §1 · p. 1
• Ambient sound is ubiquitous and free compared with human annotations, while providing information about semantics, events, and mechanics beyond images alone.Source: §1 · pp. 1–2
• Audio-visual correspondence is imperfect because sound sources may be outside the visual field and characteristic sounds may be absent.Source: §1 · p. 2
• The model classifies a video frame’s statistical summary of the sound recorded at that time, then validates the learned representation on object and scene information.Source: §1 · p. 2
• The learned features achieve similar performance to state-of-the-art unsupervised and self-supervised methods on object and scene recognition.Source: §1 · pp. 2–3
• Intermediate CNN layers develop object-selective detectors from unlabeled audio-visual signals, including objects associated with characteristic sounds.Source: §1 · p. 3

• Self-supervised methods learn visual representations by predicting targets derived from natural signals rather than human annotations.Source: §2 · p. 3
• Prior video methods use camera motion or cross-frame tracking, while image methods predict patch locations or transformation invariance.Source: §2 · p. 3
• This work differs by deriving supervision from sound, a modality complementary to vision and informative about objects and scenes.Source: §2 · p. 3
• The sound-prediction task uses straightforward classification, enabling a network design resembling object- and scene-recognition models.Source: §2 · p. 4
• Related audio-visual work predicts sounds visually indicated by actions, whereas the natural videos here often contain sound sources outside the frame.Source: §2 · p. 4

• The model predicts a frame’s corresponding sound from a single image, using the task to learn transferable visual features.Source: §3 · p. 5, §3.2 · p. 5

3.1 Statistical sound summaries

• Sound summaries average audio over 3.75 seconds using 32 cochlear-like filters, Hilbert envelopes, compression, and texture statistics.Source: §3.1 · p. 5
• Each image receives a 502-dimensional sound-texture vector normalized for gain and rescaled across feature types.Source: §3.1 · p. 5

3.2 Predicting sound from images

• Instead of direct regression, the method defines explicit sound categories and formulates prediction as classification for easier network analysis.Source: §3.2 · p. 6
• The Clustering model uses k-means centroids as image labels, and removes training examples farther than the median distance from their cluster centroid.Source: §3.2 · p. 6
• Audio clusters often contain common objects and scene attributes, including children indoors, crowds, water scenes, and outdoor wind.Source: §3.2 · p. 6
• The Binary model projects sound textures onto 30 principal components, thresholds them into binary codes, and predicts them with sigmoid outputs and cross-entropy loss.Source: §3.2 · p. 6
• The Spectrum model approximately predicts the frequency spectrum in a 33.3-millisecond interval centered on the input frame.Source: §3.2 · p. 6

Object-selective units

• 91 of 256 units were object-selective under the study’s detector criterion.A unit counted as selective when at least 60% of its top 60 activations contained the same object.Source: §4.1 · p. 9

Comparisons with other learned representations

• 72 of 256 units in the tracking-based model were object-selective, often responding to textural stuff such as grass, ground, and water.The egomotion-based model had 27 such units, while patch-position units were often selective for image position.Source: §4.1 · p. 10

Object-selective units

• The sound-trained model contained more units selective for objects with characteristic sounds than the Places scene-recognition network.A significant fraction of the sound-trained units were selective to people, including adults, babies, and crowds.Source: §4.1 · pp. 9, 11
• 43.7% of the top 30 video clips per unit contained the sound made by the object to which the neuron was selective.This analysis was performed for the object-selective units’ top responses.Source: §4.1 · p. 11

Recognition performance

• The binary-coding sound model obtained comparable performance to other unsupervised learning methods on the evaluated recognition tasks.The reported recognition evaluations used learned CNN features for object and scene recognition.Source: §4 · p. 9, §4.2 · pp. 11–12
• Sound-texture models outperformed the model predicting only the frequency spectrum, and time-averaged sound statistics were reported as helpful.The sound-based models significantly outperformed other methods when conv5 features were globally pooled.Source: §4.2 · pp. 12–13

• Sound is abundantly available without human annotations and conveys information about objects and scenes, making it a potential supervisory signal for visual learning.Source: §5 · p. 14
• A CNN trained to predict a statistical sound summary from a video frame learns image representations containing information about objects and scenes.Source: §5 · p. 14
• The paper leaves open whether alternative audio representations would enable the model to learn about additional object types.Source: §5 · p. 14

• Recognition performance on PASCAL VOC2007 increases with the number of clusters defining the audio label space.The experiments used 30 clusters.Source: A1 Sound label space · p. 17

• Additional object-detector histograms and neuron visualizations extend the analysis to methods and units beyond the main Flickr-based figures.Source: A2 Additional unit visualizations · p. 17
• 67 object-selective units were found using SUN and ImageNet images, compared with 91 for the Flickr dataset.Some categories, including baby, appeared relatively less often in SUN and ImageNet.Source: A2 Additional unit visualizations · p. 17
• The SUN and ImageNet visualization sample was drawn from 200,000 images sampled equally from the two datasets.Source: A2 Additional unit visualizations · p. 17

Sound-texture construction

• Sound textures are computed by filtering the waveform with bandpass filters, applying a Hilbert transform, resampling to 400Hz, and compressing samples with a 0.3 power.Source: A3 Sound textures · p. 17
• The representation includes correlations between cochlear channels whose index differences are 1, 2, 3, or 5.Source: A3 Sound textures · p. 17

Supplementary analyses

• Object recognition performance on PASCAL VOC2007 increases with the number of audio-label clusters, while the experiments use 30 clusters.Source: Figure A1 · p. 18
• Object-selectivity analysis uses a 60% threshold: an object detector requires the object to appear in at least 60% of its top 60 activations.Source: Figure A1 · p. 18

Sound-texture construction

• Modulation responses use 10 logarithmically spaced bandpass filters with center frequencies from 0.5 to 200Hz.Source: A3 Sound textures · p. 18
• Marginal statistics include each cochlear channel’s mean and standard deviation, plus sequence loudness measured as the median timestep energy.Source: A3 Sound textures · p. 18

Supplementary analyses

• The supplementary figures count object-selective units across SUN and ImageNet, visualize their top five activations, and quantify units for an additional method.Source: Figure A2 · p. 19, Figure A3 · p. 19, Figure A4 · p. 19