Source-linked AI summary
Learning to Localize Sound Source in Visual Scenes
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, In So Kweon
TL;DR
The paper asks whether paired sounds and visual scenes can support sound-source localization without labels. It introduces a two-stream, attention-based network with a unified supervised and semi-supervised extension, and finds that small amounts of supervision correct some unsupervised false conclusions while enabling effective localization.
Problem
The paper investigates whether a model can learn correspondence between sound and visual scenes from paired observations alone and use it to localize sound sources.
Method
It uses separate sound and visual networks with sound-guided attention, and adds supervised loss to support unsupervised, semi-supervised, and fully supervised learning.
Results
The unsupervised model localizes sources across varied categories, including cluttered scenes, while semi-supervised learning achieves the best performance and corrects some false conclusions.
Takeaways & Limitations
Small amounts of human supervision can discipline the learned sound–visual correspondence and capture more semantically meaningful source relationships.
Takeaways & Limitations
Unsupervised learning can produce semantically unmatched localizations, such as identifying railway tracks instead of a train, because correlations may not reveal true causality.
Abstract
from arXiv · showhide
Visual events are usually accompanied by sounds in our daily lives. We pose the question: Can the machine learn the correspondence between visual scene and the sound, and localize the sound source only by observing sound and visual scene pairs like human? In this paper, we propose a novel unsupervised algorithm to address the problem of localizing the sound source in visual scenes. A two-stream network structure which handles each modality, with attention mechanism is developed for sound source localization. Moreover, although our network is formulated within the unsupervised learning framework, it can be extended to a unified architecture with a simple modification for the supervised and semi-supervised learning settings as well. Meanwhile, a new sound source dataset is developed for performance evaluation. Our empirical evaluation shows that the unsupervised method eventually go through false conclusion in some cases. We show that even with a few supervision, false conclusion is able to be corrected and the source of sound in a visual scene can be localized effectively.
1. Introduction
The paper asks whether sound–visual pairs can support unsupervised localization of sound sources, motivated by their natural correspondence. It proposes an attention-based multimodal framework that can also incorporate supervision and introduces a dataset for evaluation.
- Motivation: Visual events and sounds are correlated, providing spatial and temporal cues about the source within a scene.The paper notes that sound complements visual information while remaining correlated with visual events.
- Problem: The central problem is learning sound-source localization from synchronized video and audio without supervision, analogous to human learning.The challenge is to associate sound with visual scenes and reveal the source location from paired clips.
- Approach: The proposed framework uses separate sound and visual streams whose correlated representations feed an attention mechanism for localization.The architecture processes each modality independently before integrating sound context with visual activations.
- Problem: Unconstrained videos may contain audio unrelated to visible content or sources located off-screen, making unlabeled localization difficult.Narration and commentary are examples of audio that may not correspond to visible sources.
- Contributions: Adding supervised loss converts the framework into a unified architecture supporting unsupervised, semi-supervised, and fully supervised learning.The paper also introduces a dataset with supervised annotations for evaluation and analysis.
2. Related Work and Problem Context
Prior work exploits audio–visual synchrony, cross-modal correlation, and attention, while earlier localization methods often rely on engineered cues or specialized hardware. This paper instead learns vision-based localization from ordinary video and microphone audio.
- Joint sound-visual models: Recent joint audio–visual methods use one modality as supervision for learning representations of the other or learn aligned cross-modal representations.The paper situates its audio module relative to SoundNet and related cross-modal approaches.
- Human-inspired localization: Human sound-source localization is guided by visual information, motivating models that learn sound–visual relationships from everyday events.The cited cognitive-science work emphasizes close correlation between the modalities in human perception.
- Computational localization: Earlier computational approaches rely on synchrony, spatial sparsity, low-dimensionality, handcrafted motion cues, or segmentation.These methods precede the proposed unsupervised deep-learning formulation.
- Sensing setup: Acoustic localization commonly uses microphone arrays and phase differences, whereas this work localizes sources in the visual domain using only a microphone.The contrast concerns the sensing setup rather than the learning objective.
- Attention: The proposed attention mechanism lets sound context interact with spatial visual representations across image axes.This extends the attention philosophy used in computational visual attention models to sound-guided localization.
3. Proposed Algorithm
The proposed algorithm uses separate sound and visual networks connected by an attention-based localization module. Sound–visual interactions produce a confidence map and visual representation for learning sound-source locations.
- Network architecture: The architecture combines sound, visual, and attention modules in a two-stream network for vision-based sound localization.The sound and visual streams process their respective modalities, while the attention model connects them.
- Sound network: The sound network uses a 1-D convolutional architecture with ReLU and pooling to encode high-level concepts from variable-length raw waveforms.Average pooling over sliding windows produces a fixed 1000-D representation, while later fully connected layers produce a 512-D embedding.
- Visual network: The visual network extracts spatially organized local features and uses sound-conditioned attention to produce a confidence map and representative source feature.The localization module combines visual activation V with sound embedding h, then transforms the selected visual feature into the final visual embedding.
- Localization network: For each spatial location, attention generates a positive weight from the interaction between the sound embedding and local visual feature.The attention weight is interpreted as the probability that the location corresponds to the sound context and is normalized with softmax.
- Localization network: The attention mechanisms use cosine similarity, including an alternative designed to suppress negative correlations, rather than a learned multilayer perceptron.Both mechanisms operate on normalized vectors and therefore use cosine-similarity values in the range [−1, 1].
- Learning objective: The model learns localization by making visual and sound embeddings comparable, allowing error backpropagation to train the attention measure through the sound context.The representative visual feature is sampled according to the attention map and transformed into a visual embedding aligned with sound features.
4. Localizing Sound Source via Listening
The method learns sound localization by matching visual-frame and audio representations, using positive same-video pairs and negative pairs from other videos. Unsupervised training can produce semantically mismatched localizations, while supervisory signals and a combined loss support correction.
- Unsupervised learning: The model compares visual-frame and sound representations at spatial locations to produce sound-source localization predictions.The two-stream setup uses visual and sound subnetworks, with an attention-based localization module linking the modalities.
- Unsupervised learning: Unsupervised training uses sound from a nearby window in the same video as a positive pair and sound from another video as a negative pair.A triplet loss maps positive samples closer to the visual query than negative samples.
- Unsupervised learning: The unsupervised objective creates a cyclic shared representation in which attention selects locations correlated with the sound embedding.The positive-pair constraint makes the visual feature resemble the sound feature, encouraging shared audio-visual embeddings for localization.
- Unsupervised learning: Unsupervised localization can reach false conclusions, such as localizing railway tracks when given a train sound and train image.The model may reinforce an initially correlated but incorrect activation, especially when unrelated regions are visually simpler or larger.
- Semi-supervised learning: Semi-supervised training adds a supervised loss when reference attention maps are available, and the unified loss also supports fully supervised training.The loss combines unsupervised and supervised terms; the supervised term can use mean square error or cross entropy, with cross entropy performing slightly better empirically.
5. Experimental Results
The experiments introduce an annotated sound-source localization dataset and evaluate unsupervised, semi-supervised, and supervised learning. Unsupervised localization works across varied scenes but can produce semantically mismatched results, while supervision improves localization.
- 5.1. Dataset: A new dataset uses human annotations of sound-source locations and object-versus-ambient source types for evaluation and supervision.The dataset samples 5k Flickr-SoundNet frames and sounds; annotators draw bounding boxes and tag them as object or ambient.
- 5.1. Dataset: The evaluation uses consensus intersection over union (cIoU) to account for multiple annotations, with a cIoU threshold of 0.5.Three subjects annotate each example, and consensus scores are assigned to pixels according to annotation agreement.
- 5.2. Results and Analysis: The unsupervised network localizes sound sources across varied categories and can isolate relevant regions despite visual clutter.Qualitative results include distinguishing water from people and separating stage performers from audiences in concert scenes.
- 5.2. Results and Analysis: Localization responses change when different sounds are paired with the same frame, showing interactive sound-conditioned behavior.In scenes containing water and people, water sounds highlight water while human sounds highlight people.
- 5.2. Results and Analysis: The unsupervised model sometimes draws semantically mismatched conclusions, whereas supervised and semi-supervised learning provide more precise localization.The semi-supervised model achieves promising results by combining supervised and unsupervised data, and near 1k supervised samples are reported as sufficient to learn well.
- 5.2. Results and Analysis: Training the unsupervised network with 144k samples yields significant improvement over training with 10k samples.The quantitative evaluation compares learning schemes and sample counts, with random prediction results reported for reference.
6. Discussion and Conclusion
The work introduces sound source localization in visual scenes and shows that unsupervised learning can work plausibly but may produce false conclusions. Small amounts of human supervision can correct these errors and capture semantically meaningful relationships.
- The paper introduces sound source localization in visual scenes and builds a benchmark dataset for the task.
- The unsupervised network localizes sound sources across varied categories but can falsely match sounds with visual content without prior knowledge.
- Small amounts of human supervision can discipline the model and correct false conclusions toward semantically meaningful relationships.
- The findings suggest that unsupervised sound-video representations may require at least limited supervision.