Source-linked AI summary
Localizing Visual Sounds the Easy Way
Shentong Mo, Pedro Morgado
TL;DR
Unsupervised audio-visual source localization seeks visible sound sources without manual location annotations, while prior approaches rely on identifying likely sounding and non-sounding regions. EZ-VSL learns correspondence without constructing those regions and adds object-guided localization. The authors report state-of-the-art performance on Flickr SoundNet and VGG-Sound Source.
Problem
Unsupervised sound localization lacks manual source-location annotations, and prior methods must distinguish likely sounding from non-sounding regions to train audio-visual similarity.
Source: §1 · pp. 1–2Method
EZ-VSL uses multiple-instance learning to match audio with at least one location in its paired image, then combines audio-visual similarity and object maps at inference.
Source: §1 · pp. 2–3Results
EZ-VSL achieves state-of-the-art performance on Flickr SoundNet and VGG-Sound Source, with combined object priors and audio-visual maps surpassing current methods by large margins.
Source: §1 · p. 3, §5 · p. 14Takeaways & Limitations
The authors report that object priors alone surpass prior work on both benchmarks, while combining them with audio-visual similarity yields more accurate predictions.
Source: §1 · p. 3
Abstract
from arXiv · showhide
Unsupervised audio-visual source localization aims at localizing visible sound sources in a video without relying on ground-truth localization for training. Previous works often seek high audio-visual similarities for likely positive (sounding) regions and low similarities for likely negative regions. However, accurately distinguishing between sounding and non-sounding regions is challenging without manual annotations. In this work, we propose a simple yet effective approach for Easy Visual Sound Localization, namely EZ-VSL, without relying on the construction of positive and/or negative regions during training. Instead, we align audio and visual spaces by seeking audio-visual representations that are aligned in, at least, one location of the associated image, while not matching other images, at any location. We also introduce a novel object guided localization scheme at inference time for improved precision. Our simple and effective framework achieves state-of-the-art performance on two popular benchmarks, Flickr SoundNet and VGG-Sound Source. In particular, we improve the CIoU of the Flickr SoundNet test set from 76.80% to 83.94%, and on the VGG-Sound Source dataset from 34.60% to 38.85%. The code is available at https://github.com/stoneMo/EZ-VSL.
1 Introduction
The paper targets unsupervised localization of visible sound sources, where prior approaches struggle to identify sounding and non-sounding regions and can depend on localization maps to train the representations that produce them. EZ-VSL replaces explicit region construction with multiple-instance audio-visual learning and adds object-guided localization; the authors report state-of-the-art results on both benchmarks.
1 Introduction
- Prior methods face a circular training challenge: accurate sounding regions guide representation learning, but accurate representations are needed to identify those regions.The authors say this can produce sub-optimal local minima as models bootstrap from their own localization ability.Source: §1 · p. 2
- Audio-visual similarity alone overlooks visual priors that make many image regions, such as floors, skies, tables, and walls, unlikely sound sources.EZ-VSL adds an object-guided inference scheme that combines an audio-visual similarity map with an object localization map from a lightweight pretrained visual model.Source: §1 · pp. 2–3
- EZ-VSL learns audio-visual correspondence by matching audio to at least one location in its paired image, without explicitly constructing sounding or non-sounding regions.The objective also requires no match to locations in other images.Source: §1 · pp. 2–3
- EZ-VSL surpasses prior methods by large margins on Flickr SoundNet and VGG-Sound Source when combining object priors with audio-visual similarity maps.The authors also report that the object prior alone surpasses prior work on both benchmarks.Source: Figure 1 · p. 2, §1 · p. 3
2 Related Work
Related work spans joint audio-visual representation learning and methods that localize sound sources from video. EZ-VSL is positioned as a simplified localized-correspondence objective supplemented by object-guided localization.
2.1 Audio-Visual Joint Learning
- Audio-visual joint learning includes unsupervised correspondence and synchronization tasks, while this work focuses on learning sound-source localization from unlabeled videos.Prior work also applies weakly supervised multiple-instance learning to action localization and hybrid attention to video parsing.Source: §2.1 · p. 3
2.2 Audio-Visual Source Localization
- Localization methods range from shallow probabilistic models and global correspondences to localized attention and contrastive approaches; EZ-VSL instead uses simplified multiple-instance learning and object guidance.The cited work notes that global-correspondence methods show localization qualitatively but do not match the localization performance of methods trained on localized correspondences.Source: §2.2 · p. 4
3 Method
EZ-VSL learns audio-visual correspondence without explicit sounding or non-sounding regions, aligning audio with at least one location in its associated image while using object maps to refine localization at inference.
3 Method
- EZ-VSL targets unsupervised localization of sounding objects in video without using manual location annotations for training.Source: §3 · p. 4
3.1 Overview
- A global audio representation and location-specific visual features are mapped into a shared space, where multiple-instance contrastive learning aligns audio with its associated image.At inference, object-localization maps from an object-recognition encoder are combined with audio-visual localization maps.Source: §3.1 · p. 5, Figure 2 · p. 5
3.2 Audio-visual matching by multiple-instance contrastive learning
- The multiple-instance objective projects audio and visual representations into a shared feature space, then matches audio to its most similar location in the associated image while contrasting locations from other images.Negative bags come from other images in the same mini-batch, and the training loss sums audio-to-visual and visual-to-audio directions.Source: §3.2 · p. 6
3.3 Object-Guided Localization
- At inference, an ImageNet-pretrained object-recognition model supplies object priors, using either class-posterior maps or feature-activation magnitudes, which are linearly combined with audio-visual similarities.The object encoder is not trained for audio-visual similarity; its prior favors objects whether or not they produce the sound, and α balances the maps.Source: §3.3 · p. 7
4 Experiments
EZ-VSL is evaluated on Flickr SoundNet and VGG-Sound Source, with experiments covering benchmark comparisons, generalization, component ablations, and qualitative localization. The results report gains over prior work, while analyses identify effective localization choices and cases the model still struggles with.
4.2 Comparison to prior work
- 7.93% CIoU and 3.36% AUC gains on Flickr, and 4.25% CIoU and 1.34% AUC gains on VGG-Sound Sources, establish EZ-VSL’s state-of-the-art results across settings.The Flickr gains are for models trained on Flickr 144k; the VGG-Sound Sources gains are on the more challenging test set. The evaluation uses Flickr SoundNet and VGG-Sound datasets, with CIoU and AUC metrics.Source: §4 · pp. 7–8, Table 1 · p. 9, Table 2 · p. 9
4.3 Open Set Audio-Visual Localization
- On VGG-Sound, EZ-VSL outperforms LVS on heard and unheard categories, with slightly better performance on unheard categories.The evaluation trains on 110 sampled VGGSound categories and tests on samples from those 110 heard categories and another 110 unheard categories.Source: §4.3 · p. 10, Table 3 · p. 10
4.4 Cross Dataset Generalization
- Across datasets, EZ-VSL outperforms the best previous method when a VGG-Sound-trained model is tested on Flickr SoundNet and a Flickr-trained model on VGG-SS.The cross-dataset experiments include models trained on VGG-Sound 10k, 144k, or the full set, and Flickr 10k or 144k.Source: §4.4 · p. 10, Table 4 · p. 11
4.5 Experimental analysis
- Both audio-visual localization and object-guided localization surpass LVS when evaluated alone, and combining them can improve over object-guided localization alone.L1-OGL, which uses no audio information, already performs strongly; the authors attribute this partly to object regions being likely sound sources and most test scenes containing a single sounding object.Source: §4.5 · p. 11, Table 5 · p. 12
- The best OGL balance is α = 0.4 for Flickr and α = 0.5 for VGG-SS; α = 0.4 is the default, and the output-size optima are 1024 and 512, respectively.The α = 0 and α = 1 settings perform worse, indicating both OGL and AVL components matter for accurate localization.Source: §4.5 · p. 11, Figure 3 · p. 12, Figure 4 · p. 12
- The shared audio-visual latent dimensionality affects performance: substantially reducing or increasing it can hurt, with optima of 1024 for Flickr and 512 for VGG-SS.Both unimodal encoders output 512-dimensional representations.Source: §4.5 · p. 12, Figure 4 · p. 12
- The proposed max-pooling match of the global audio feature with localized visual features localizes more accurately than average pooling or matching global features.This experiment sets α = 1 to evaluate audio-visual localization maps without merging them with object-guided maps; the method assumes at least one sounding object is visible in the image.Source: §4.5 · pp. 12–13, Table 6 · p. 13
4.6 Qualitative Results
- Qualitative predictions include tight overlap with ground-truth boxes, but EZ-VSL struggles with tight maps for small objects and with crowd sounds.Examples include identifying musical instruments rather than the people playing them, and a dog rather than a man.Source: §4.6 · p. 13, Figure 5 · p. 14, Figure 6 · p. 14
5 Conclusion
EZ-VSL combines audio-visual correspondence learning with object-guided localization and achieves state-of-the-art performance on two benchmarks. It also shows advantages in open-set localization and cross-dataset generalization.
5 Conclusion
- EZ-VSL achieves state-of-the-art performance on Flickr SoundNet and VGG-Sound Source compared with previous contrastive and non-contrastive baselines.The conclusion also reports comprehensive ablation studies of the method's components.Source: §5 · p. 14
- EZ-VSL learns relative correspondence between visual and audio instances without explicitly computing negative regions.The conclusion describes using a simple cross-entropy loss.Source: §5 · p. 14
- Its object-guided localization scheme mixes the audio-visual joint map with an object map from a lightweight pre-trained visual model.The stated goal is to improve the localization of sound sources in an image.Source: §5 · p. 14
- The conclusion reports advantages in open-set visual sound source localization and cross-dataset generalization.Qualitative maps show object-guided maps covering scene objects, while the final EZ-VSL map tends to focus better on the sounding object.Source: §5 · pp. 14–15, Figure 7 · p. 15