Source-linked AI summary
Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, Dejing Dou
TL;DR
The paper tackles class-aware localization of sounding objects in mixed scenes, where machines must distinguish multiple sounding objects from silent ones without object-category annotations. It learns object representations from single-source localization, then uses category-aware maps and audiovisual distribution matching for cocktail-party scenes. Experiments on realistic and synthesized data show improved localization and silent-object filtering, while the method still requires rough partitioning of single- and multiple-source videos.
Problem
Mixed sound scenes require localizing sounding objects by category and filtering silent objects without manual object-category annotations.
Method
A two-stage framework learns a representation dictionary from single-source localization, then matches audio and visual object-category distributions to localize sounding objects in multi-source scenes.
Results
The method shows excellent performance on synthesized and realistic data, including improved class-aware localization and filtering of silent objects.
Takeaways & Limitations
Audiovisual consistency can provide self-supervision for discriminative sounding-object localization across different object classes.
Takeaways & Limitations
The learned object semantics still require rough partitioning of single-source and multiple-source videos.
Abstract
from arXiv · showhide
Discriminatively localizing sounding objects in cocktail-party, i.e., mixed sound scenes, is commonplace for humans, but still challenging for machines. In this paper, we propose a two-stage learning framework to perform self-supervised class-aware sounding object localization. First, we propose to learn robust object representations by aggregating the candidate sound localization results in the single source scenes. Then, class-aware object localization maps are generated in the cocktail-party scenarios by referring the pre-learned object knowledge, and the sounding objects are accordingly selected by matching audio and visual object category distributions, where the audiovisual consistency is viewed as the self-supervised signal. Experimental results in both realistic and synthesized cocktail-party videos demonstrate that our model is superior in filtering out silent objects and pointing out the location of sounding objects of different classes. Code is available at https://github.com/DTaoo/Discriminative-Sounding-Objects-Localization.
1 Introduction
The paper addresses class-aware localization of sounding objects in mixed scenes, where machines must distinguish multiple sounding categories from silent objects without object annotations. It proposes a two-stage framework that learns object knowledge from simpler single-source scenes and extends it to cocktail-party localization.
- Existing audiovisual similarity strategies work in single-sound scenes but mostly fail to discriminate sounding objects in complex mixed scenes.
- Cocktail-party localization must identify sounding objects by category while filtering out visually present silent objects.
- The proposed framework first learns potential object knowledge from single-source sound localization and aggregates it into a category representation dictionary.
- It then generates class-aware localization maps and selects sounding objects in multi-source scenes through self-supervised audiovisual category-distribution matching.
- The method is evaluated on synthesized cocktail-party videos with annotated sounding-object boxes and on realistic data, showing excellent performance on both.
2 Related work
Prior work uses visual cues, audiovisual correspondence, and attention to localize sound sources, but existing approaches do not fully address class-aware localization in mixed scenes.
- Weakly supervised methods use holistic image labels, while self-supervised visual methods can extract class-agnostic object heat maps.
- Self-supervised audiovisual learning exploits natural sound–vision correspondence through cross-modal representation learning, clip correspondence, or temporal synchronization.
- Sound-source localization methods commonly correlate audio and visual features spatially or use attention to identify sounding regions.
- The paper distinguishes its goal by targeting discriminative localization of multiple sounding objects from mixed sound without manual object-category annotations.
3 The proposed method
The method uses curriculum-style two-stage learning: it builds object representations from single-source localization, then performs class-aware multi-source localization by matching visual and audio category distributions.
- Framework: The framework separates audiovisual data into simple single-source and complex multi-source scenarios, then progresses from the former to the latter.
- Stage 1: Object representation: In the first stage, audiovisual correspondence localizes single-source sounding regions without extra semantic annotations.
- Stage 1: Object representation: Candidate object representations are extracted from masked localization features and organized into a learned dictionary whose entries represent object-category characteristics.
- Stage 2: Discriminative localization: In the second stage, the dictionary produces K category-specific localization maps for objects appearing in multi-source visual scenarios.
- Stage 2: Discriminative localization: The inferred sounding area filters category maps so visually present objects without corresponding sound receive low sounding responses.
- Stage 2: Discriminative localization: Audiovisual consistency is enforced by matching visual and audio sounding-object category distributions with KL divergence, alongside category-agnostic sounding-area detection.
4 Experiments
Experiments evaluate single-source and multiple-source sounding-object localization across MUSIC and AudioSet settings, using localization and silent-object filtering metrics. The model improves class-aware localization and qualitatively preserves low responses for silent objects.
- Datasets and annotation: The experiments use MUSIC and AudioSet-instrument data, including solo, duet, synthetic, and multi-source settings.MUSIC contains instrument videos, while AudioSet-instrument includes 63,989 clips across 15 instrument categories.
- Evaluation metrics: CIoU measures average class-specific localization overlap, while NSA measures activation area for silent categories below threshold τ.Together, the metrics evaluate discriminative localization and silent-object filtering.
- Single sounding object localization: Alternative learning produces more category-discriminative visual features and improves single-source localization over compared methods.The feature distributions are visualized with t-SNE, comparing global and masked object features.
- Multiple sounding objects localization: The model shows significant CIoU improvements over compared methods in realistic and synthetic cocktail-party scenarios.The authors attribute this to category-based audiovisual alignment and visual knowledge learned during the first stage.
- Multiple sounding objects localization: Visualizations show class-specific localization of sounding guitar and cello while maintaining low responses for silent saxophone and other regions.The comparison includes realistic and synthetic cocktail-party videos.
5 Discussion
The proposed method uses object semantics learned from simple cases to localize sounding objects without category annotations. Its current training still requires rough partitioning of single- and multiple-source videos.
- Discussion: Object localization from single-source videos builds discriminative object representations, while audiovisual consistency supervises category distribution alignment.The framework expands from simple single-source cases to cocktail-party localization.
- Discussion: The approach still requires rough partitioning of single- and multiple-source videos, which the authors identify as a future-study issue.This is the stated scope limitation of the learned object semantics.
7 Broader Impact
The work extends machine audiovisual perception toward human-like understanding of multi-source scenes, using self-supervision to associate sounds with visual objects without elaborate manual annotation. It may support applications such as deaf-accessible scene understanding and parsing mixed auditory input.
- The approach is fully self-supervised while providing discrimination for visual objects and correlation across audio and visual modalities.
- The method may facilitate unimodal and multimodal learning and modeling of complex scenes without elaborate manual annotation.
- Potential applications include audio-assistant scene understanding for deaf people by identifying which objects produce sound.
- The work also addresses the cocktail-party effect by targeting pertinent content from mixed auditory input.
8 Appendix
The appendix details the audiovisual architecture, training procedures, ablations, qualitative results, and failure cases. Results show benefits from alternating optimization, audiovisual consistency, silent-area suppression, and pretrained visual features, while several scene configurations remain difficult.
- Architecture Details: The shared ResNet-18-S5 backbone increases feature resolution by removing a late-stage stride and omits global average pooling and the final classifier.
- Architecture Details: The audio network adapts ResNet-18-S5 to one-channel spectrograms and replaces global average pooling with global max pooling to suppress nonsounding areas.
- Architecture Details: Audiovisual localization projects and normalizes audio and image features, broadcasts audio spatially, and uses cosine similarity followed by convolution, sigmoid activation, and pooling.
- Learning Procedure: Alternating classification and localization objectives improves localization because categorization provides reference for generalized object localization.
- Qualitative Results: The class-aware maps aim to localize sounding objects of different classes while assigning low activation to silent objects.
- Ablation Study: The loss weight λ has limited effect in [0.5, 1.0], whereas extreme values respectively encourage localization overfitting or hinder sounding-area detection.
- Ablation Study: The method remains robust to cluster count and mask threshold, achieving comparable performance without knowing the dataset’s category count.
- Ablation Study: The second-stage product operation is crucial on synthetic data, while the audiovisual consistency objective improves localization on synthetic and real-world duet data.