PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Sound Sources Localization from Coarse to Fine

Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, Weiyao Lin

arXiv:2007.06355v2cs.CV

TL;DR

The paper addresses visual localization of multiple sound sources in unconstrained videos without pairwise sound-object annotations. It introduces a two-stage framework that disentangles category-specific audio and visual representations and aligns them coarse-to-fine, achieving strong localization and sound-separation results in complex scenes.

  • Problem

    Existing methods struggle to distinguish multiple sound sources and precisely align sounds with visual objects in unconstrained videos without pairwise annotations.

  • Method

    A two-stage audiovisual framework uses classification and correspondence learning, then class-specific representations for coarse-to-fine cross-modal alignment.

  • Results

    The model achieves state-of-the-art results on a public localization dataset and considerable performance for multi-source localization in complex scenes.

  • Takeaways & Limitations

    Localization outputs provide valuable visual references for sound separation, where the model obtains comparable performance to existing methods.

  • Takeaways & Limitations

    The approach is constrained by its category set; adding more categories is proposed to establish more precise sound-object associations.

Abstract

from arXiv · show

How to visually localize multiple sound sources in unconstrained videos is a formidable problem, especially when lack of the pairwise sound-object annotations. To solve this problem, we develop a two-stage audiovisual learning framework that disentangles audio and visual representations of different categories from complex scenes, then performs cross-modal feature alignment in a coarse-to-fine manner. Our model achieves state-of-the-art results on public dataset of localization, as well as considerable performance on multi-source sound localization in complex scenes. We then employ the localization results for sound separation and obtain comparable performance to existing methods. These outcomes demonstrate our model's ability in effectively aligning sounds with specific visual sources. Code is available at https://github.com/shvdiwnkozbw/Multi-Source-Sound-Localization

1 Introduction

The paper targets audiovisual localization in unconstrained multi-source scenes, where video-level correspondence is too coarse and pairwise sound-object annotations are unavailable. It addresses these challenges with two-stage, coarse-to-fine alignment that disentangles category-specific audiovisual representations.

  • Multi-source videos mix sounds, making video-level supervision insufficient for precise sound-to-visual-source alignment.
  • Existing localization methods mainly handle simple scenes or rely on difficult-to-obtain stereo audio, while multi-source evaluation remains limited.
  • The paper identifies two challenges: distinguishing sound sources and establishing satisfactory sound-object alignment without one-to-one annotations.
  • The framework uses classification predictions and CAM to extract class-specific sound-object representations, then aligns them from category-level correspondence to fine-grained video- and category-level matching.
  • The model achieves state-of-the-art public-dataset localization results, considerable multi-source localization performance, and valuable visual references for sound separation.

2 Related Work

Prior audiovisual localization work models joint audio-visual relationships, but earlier approaches often use coarse correspondence or limited localization mechanisms. This paper positions its method as directly handling unconstrained complex scenes through class-specific fine-grained alignment.

  • Earlier audiovisual methods use natural correspondence, temporal synchronization, or teacher-student supervision to learn cross-modal representations.
  • Sound-localization methods include audiovisual correspondence, attention, audio-guided alignment, clustering, and self-supervised approaches.
  • The paper reports direct training on unconstrained videos while precisely localizing visual sources for different sounds in complex scenes.
  • CAM generates class-specific saliency maps from holistic image labels, while gradient-based variants extend weakly supervised localization to arbitrary CNN architectures.
  • Unlike prior work using CAM only to measure final cross-modal relationships, this method uses it to disentangle features before fine-grained audiovisual alignment.

3 Approach

The approach combines audio-visual classification and video-level correspondence, then uses class-specific representations for fine-grained alignment. Its correspondence network learns whether audio and visual inputs match, producing coarse category-level correspondence before finer localization.

  • The framework first performs classification and video-level audiovisual correspondence, then uses Grad-CAM to disentangle class-specific features for alignment.
  • Audio and visual backbones, classification, and correspondence networks jointly exploit audiovisual pairs and their natural correspondence.
  • The classification task uses the same categories for both modalities, with video tags or pretrained-model predictions providing supervision.
  • Because videos contain multiple sound sources, classification uses multi-label binary cross entropy with class labels and predicted probabilities.
  • The correspondence network extracts 512-D audio and visual features, concatenates them into a 1024-D vector, and classifies corresponding versus mismatched pairs.
  • The multi-task objective combines classification and audiovisual correspondence losses, yielding coarse-grained category-level correspondence after training.

3.2 Audiovisual Feature Alignment

The framework uses classification-driven Grad-CAM maps to disentangle category-specific audio and visual representations, then aligns corresponding features through shared embeddings and contrastive learning.

  • Feature Disentanglement: Grad-CAM produces class-specific maps that disentangle audio and visual feature representations by category.The maps are computed from last-convolutional-layer activations and classification outputs, then used for category-aware pooling.
  • Feature Disentanglement: Weighted global pooling over class-specific maps converts feature maps into category-aware representations.The resulting representations are used instead of directly aligning the original feature maps.
  • Feature Disentanglement: The method produces C 512-dimensional vectors representing all categories for each audio and visual modality.These class-specific vectors are passed to the subsequent fine-grained alignment stage.
  • Fine-Grained Audiovisual Alignment: Audio and visual class-specific features are projected into a shared embedding space using modality-specific fully connected layers.The projected features are compared with Euclidean distance before alignment training.
  • Fine-Grained Audiovisual Alignment: Contrastive loss aligns sound-object representations, using a margin hyper-parameter to distinguish matching from nonmatching pairs.The approach considers category and video context when selecting positive pairs because one category can contain varied entities.

3.3 Sound Localization and Its Application in Separation

The aligned audio-visual representations generate source-aware localization maps by comparing visual regions with class-specific audio features. These localized sources are then pooled into guidance features for mix-and-separate sound separation.

  • Sound Localization: Localization projects a testing image's visual feature map into the shared embedding space and compares it with disentangled class-specific audio features.This comparison produces source-aware maps for each sound class.
  • Sound Localization: Kc_i assigns each visual region a likelihood of being the visual source for the c-th class of sound.The map is normalized and resized to the original image size to form the final localization map.
  • Sound Localization: Class-labeled localization results can evaluate sound localization under multi-source conditions.The maps associate localized visual sources with their sound-class labels.
  • Sound Separation: Weighted global pooling over localized source maps produces visual guidance for sounds belonging to each class.The pooling operation follows the same general approach used for class-aware representations.
  • Sound Separation: A U-Net processes mixed audio spectrograms using automatically determined visual-source guidance, and inverse short-time Fourier transform reconstructs waveforms.The output masked spectrogram corresponding to each visual source is converted into an audio waveform.

4 Experiments

Experiments evaluate the two-stage audiovisual model on unconstrained videos, SoundNet-Flickr, and multi-source AudioSet scenes, with qualitative, quantitative, ablation, and sound-separation analyses. The results show category-specific localization in mixed audio, strong localization performance, and comparable separation results when visual representations guide separation.

  • SoundNet-Flickr: SoundNet-Flickr uses unconstrained audiovisual pairs, with 10k training pairs and 250 annotated test pairs for localization evaluation.The experiments use seven first-level AudioSet categories as classification targets and no extra supervision for training.
  • Sound Localization on SoundNet-Flickr: Mixed-source visualizations show the model separating class-specific responses and highlighting corresponding sound-producing regions.Examples include simultaneously localizing a person and a gun area from mixed human-speaking and gunfire audio.
  • Sound Localization on SoundNet-Flickr: Compared with CAM, the model distinguishes sounding from silent objects within the same category and more precisely localizes human yelling.The violin example separates a sounding violin from a silent piano, while CAM provides only category-based localization.
  • Multi-Source Localization on AudioSet: AudioSet evaluation introduces a multi-source pipeline over 15 musical instruments and partitions testing into 4,273 single-source, 211 two-source, and 19 three-source pairs.The ablation compares AVC, multi-task learning, and the full two-stage method across difficulty levels.
  • Multi-Source Localization on AudioSet: In level-2 AudioSet scenes, the full method localizes specific instruments more accurately than AVC and multi-task baselines, including a flute missed by both alternatives.The study also reports comparable SDR, SIR, and SAR when visual representations guide sound separation, whereas Grad-CAM masks are too low-resolution.

5 Conclusions

The framework disentangles category-specific audio and visual representations from complex scenes and aligns them coarse-to-fine. It shows promising performance for multi-source localization and sound separation, while future work targets more precise associations through additional categories.

  • The audiovisual framework disentangles representations from complex scenes and performs cross-modal alignment in a coarse-to-fine manner.
  • The model shows promising performance on multi-source sound localization in complex scenes and on sound source separation.
  • Adding more categories to the classification task is proposed to better distinguish sounds and objects and establish more precise sound-object associations.

1 Generating Pseudo Labels for Unlabelled Videos

For unlabelled videos, the method generates pseudo labels as classification supervision. It uses pretrained audio and visual classifiers and organizes general target categories using AudioSet’s hierarchical ontology.

  • Pseudo labels are generated as classification supervision when training on unlabelled videos such as SonudNet-Flickr and AVE.
  • CRNN pretrained on AudioSet and ResNet-18 pretrained on ImageNet predict classification probabilities for audio and visual inputs.
  • The method selects first-level labels from AudioSet’s hierarchical ontology to organize general categories for coarse-grained audiovisual correspondence.
  • AudioSet is described as having a four-level label hierarchy used to choose general classification targets.

2 Experiments on AVE Dataset

On AVE, the study evaluates unsupervised cross-modality temporal localization with A2V and V2A subtasks using sliding-window predictions. Fine-grained alignment improves performance, especially for V2A, although overlapping-event scarcity limits its effect on A2V.

  • Dataset and task: The AVE dataset contains 4143 10-second clips across 28 event categories, split into 3339 training, 402 validation, and 402 test videos.
  • Dataset and task: Training uses audiovisual pairs without labels, while temporally annotated event boundaries are used for evaluation.
  • Dataset and task: The two subtasks localize synchronized content across modalities: visual localization from audio (A2V) and audio localization from video (V2A).
  • Method: Sliding windows compare audio and visual contexts through a correspondence score to predict the synchronized segment’s temporal position.
  • Results: Fine-grained alignment further improves performance and produces a large-margin advantage on V2A, but remains not best on A2V.The passage attributes the limited A2V effect to few overlapping events, which restricts fine-grained alignment efficacy.
  • Results: Visualization shows localization changing over time and identifying specific sound makers, including guitar, accordion, dog barking, and other sounds.Frames are sampled at 1 fps, with heatmaps corresponding to one-second audio clips.

3 Comparison with CAM

The two-stage framework is compared with category-level CAM outputs using an AVE-trained model and seven general categories. On a human-annotated SoundNet-Flickr subset, it outperforms CAM by a large margin, supporting fine-grained sound-object alignment.

  • Comparison setup: The first stage establishes category-level coarse correspondence, while the second stage performs fine-grained sound-object alignment.
  • Comparison setup: The comparison uses an AVE-trained model with seven general categories and contrasts the framework’s localization procedure with CAM maps weighted by audio probabilities.
  • Results: On a human-annotated SoundNet-Flickr subset, the two-stage framework outperforms CAM by a large margin.
  • Results: The comparison is presented as evidence for the efficacy of fine-grained sound-object alignment in the second stage.

4 Additional Results

Additional results show that the model distinguishes multiple sound sources, including sources within the same category, and supports cross-modal retrieval. Three-source examples also reveal a case where visual-audio correlation links an unseen engine source with a boat.

  • The model generally distinguishes different sound sources in two-source scenes.
  • Compared with CAM, the model can localize the sounding object when multiple objects share the same category.
  • Two-source localization maps qualitatively demonstrate the model’s multi-source sound localization performance.
  • In one three-source example, the model associates an unseen engine source with a boat, likely because boats and engine sounds are highly correlated.
  • Aligned audiovisual features support retrieval of top-5 gallery examples using either an image or an audio clip as the query.
  • Three-source localization maps provide one response for each potential sound source.
Loading 2007.06355v2…