PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Visual Sound Separation Using Minus-Plus Net

Xudong Xu, Bo Dai, Dahua Lin

arXiv:1908.11602v2cs.CVcs.SDeess.AS

TL;DR

Mixed sounds can obscure the complementary information audio provides for visual tasks, motivating visual sound separation. MP-Net recursively separates and removes sounds in average-energy order; on two datasets, it consistently outperforms state-of-the-art methods and maintains steady performance as mixture size increases.

  • Problem

    Sounds from multiple sources are often mixed, so they must be separated and associated with visual sources before their complementary information can be used.

    Source: §1 · p. 1

  • Method

    MP-Net recursively separates sounds in descending average energy, removes each prediction from the mixture, and refines sounds with a plus stage.

    Source: §1 · p. 1

  • Results

    On two datasets, MP-Net consistently outperforms state-of-the-art methods and maintains steady performance as the number of sounds in mixtures increases.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Once trained, MP-Net can handle mixtures with arbitrary numbers of sounds and may associate separated sounds with possible sources in the corresponding video.

    Source: §5 · p. 8

    The ordering criterion uses a sound's average spectrogram energy.

    Source: §1 · p. 1

Abstract

from arXiv · show

Sounds provide rich semantics, complementary to visual data, for many tasks. However, in practice, sounds from multiple sources are often mixed together. In this paper we propose a novel framework, referred to as MinusPlus Network (MP-Net), for the task of visual sound separation. MP-Net separates sounds recursively in the order of average energy, removing the separated sound from the mixture at the end of each prediction, until the mixture becomes empty or contains only noise. In this way, MP-Net could be applied to sound mixtures with arbitrary numbers and types of sounds. Moreover, while MP-Net keeps removing sounds with large energy from the mixture, sounds with small energy could emerge and become clearer, so that the separation is more accurate. Compared to previous methods, MP-Net obtains state-of-the-art results on two large scale datasets, across mixtures with different types and numbers of sounds.

1. Introduction

Visual sound can complement images, but mixed sources must be separated and associated with the scene before audio can be used. MP-Net recursively removes salient sounds and is reported to generalize across sound counts and types while outperforming state-of-the-art methods on two datasets.

1. Introduction

  • Sound can complement visual cues for object detection [17] [18] and description generation [6] [7] [26] [5], but mixed sources must first be separated and associated with the scene.Source: §1 · p. 1
  • Visual sound separation is challenging because visible sources may be silent, mixtures vary in source count and type, and high-energy sounds can obscure weaker ones.Source: §1 · p. 1
  • Earlier independent-separation methods assume fixed sound types or counts, can produce outputs inconsistent with the mixture, and may miss quieter sounds.Source: §1 · p. 1
  • MP-Net identifies sounds recursively in descending average energy, removing each salient sound until the mixture is empty or contains only noise.The plus stage refines sounds using a residual based on the sound and preceding separated sounds; average energy means average spectrogram energy.Source: §1 · p. 1
  • Removing higher-energy sounds lets lower-energy sounds emerge more clearly and allows MP-Net to handle mixtures with arbitrary numbers and types of sounds.The framework dynamically determines the number of sounds without knowing their number or types in advance.Source: §1 · pp. 1–2
  • On two large-scale datasets, MP-Net obtains more accurate results and generalizes better than the state-of-the-art method.The introduction attributes this to separating salient sounds from the mixture after prediction.Source: §1 · p. 2

2. Related Work

Related work spans audio-visual representation learning, sound localization, and visual sound separation. The closest separation methods make fixed assumptions and predict sounds independently, unlike the paper's recursive approach.

2. Related Work

  • Audio-visual embedding methods learn shared representations or correspondence, whereas visual sound separation must isolate each independent sound using the corresponding video.The passage cites Aytar et al., Arandjelovic et al. [2], and extensions [21] [20] [16] [29].Source: §2 · p. 2
  • Sound localization identifies the video region generating a sound through synchrony, event detection [24] [3], or feature-map activations [23] [3]; separation must first isolate sounds.Source: §2 · p. 2
  • Visual speech separation targets mixtures of human speech [1] [8], while this work addresses sounds of different types with more diverse characteristics.Source: §2 · p. 2
  • The closest methods [10] [28] predict sounds independently under fixed type or count assumptions, risking mixture inconsistency and weaker predictions for low-energy sounds.MP-Net instead predicts recursively by average energy and removes each prediction from the mixture.Source: §2 · p. 2

3. Visual Sound Separation

MP-Net frames visual sound separation as recovering independent sounds from a mixture using video, then recursively removes salient sounds and refines outputs. The section also introduces AMID to evaluate separation quality.

3. Visual Sound Separation

  • MP-Net has a minus stage that recursively removes salient sounds and a plus stage that refines each separated sound using a residual.The framework associates separated sounds with sources in the video and does not require the number of sounds in advance.Source: §3 · p. 2, §3.1 · pp. 2–3, Figure 1 · p. 3

3.2. M-Net

  • M-Net predicts sub-spectrograms with a U-Net, uses video associations to form a mask, and removes each prediction until residual energy falls below a threshold.This procedure lets less salient sounds emerge in later recursive steps.Source: §3.2 · pp. 3–4

3.3. P-Net

  • P-Net uses a residual based on preceding predictions to refine M-Net sounds that may have missed shared content.The final sound spectrogram combines the M-Net prediction and its residual.Source: §3.3 · p. 4

3.4. Mutual Distortion Measurement

  • AMID measures average SSIM similarity between separated sounds and annotations of other sounds; lower scores indicate more distinct separation.Because it uses SSIM over spectrograms, AMID is insensitive to frequency locations of mismatches.Source: §3.4 · p. 4
  • MP-Net obtains the best performance across metrics and settings on VEGAS and MUSIC.The reported table captions identify these two datasets and the scope of the comparisons.Source: Table 1 · p. 5, Table 2 · p. 5

4. Experiments

Experiments on MUSIC and VEGAS compare MP-Net with PixelPlayer and MIML across datasets and mixture settings. The results report strong benchmark performance, robustness to mixtures with more sounds than seen in training, and qualitative separation and localization advantages.

4.1. Datasets

  • MUSIC covers 11 instrument categories, while VEGAS covers 10 natural-sound types; the experiments use the MUSIC validation set as test data and 2,000 VEGAS test samples.MUSIC's test set contains duets without mixture ground truth, so the validation set is used for testing.Source: §4.1 · p. 5

4.2. Training and Testing Details

  • Because real mixed-video ground truth is unavailable, the authors construct mixed data from solo clips, each containing at most one sound.They follow the Mix-and-Separate strategy to combine n videos into a mix-n sample.Source: §4.2 · p. 5
  • For fair evaluation, MP-Net is given the mixture's sound count, although it can determine the count using its termination criterion and exceeds 90% correct-count accuracy on MUSIC.Prediction order is determined by Eq. (7).Source: §4.2 · p. 8

4.3. Experimental Results

  • MP-Net obtains the best results in most settings on MUSIC and VEGAS, outperforming PixelPlayer and MIML by large margins.The comparison covers mix-2 and mix-3 training and evaluation, with ratio and binary masks for MP-Net and PixelPlayer.Source: §4.3 · p. 8
  • MP-Net with P-Net outperforms the version without P-Net on all metrics, with P-Net compensating for shared patterns lost during sound removal.The authors connect this compensation to filling spectrogram blanks and keeping separated sounds consistent with the actual mixture.Source: §4.3 · p. 8
  • When trained on fixed-count mixtures, MP-Net steadily outperforms baselines on mixtures of up to five sounds.The tests vary the number of sounds in mixtures on both MUSIC and VEGAS.Source: §4.3 · p. 8
  • In qualitative examples, MP-Net separates violin and guitar sounds that PixelPlayer misses under accordion masking, while PixelPlayer also confuses natural-sound sources.For the natural-sound example, PixelPlayer predicts the same sound for rail transport and water flowing and fails to separate a dog sound.Source: §4.3 · p. 8
  • MP-Net produces more precise associations between separated sounds and possible visual sources than PixelPlayer in the shown localization samples.The paper reports that these associations use Eq. (7).Source: §4.3 · p. 8

5. Conclusion

MP-Net recursively separates sounds from video-corresponding mixtures, and on two datasets it outperforms state-of-the-art methods while maintaining steady performance as mixture sound counts increase.

  • On two datasets, MP-Net consistently outperforms state-of-the-art methods and maintains steady performance as mixture sound counts increase.Source: §5 · p. 8
  • MP-Net separates larger-energy sounds first and removes them from the mixture, allowing it to handle mixtures with arbitrary numbers of sounds once trained.Source: §5 · p. 8
  • MP-Net can associate separated sounds with possible sound sources in the corresponding video, potentially linking audio and visual data.Source: §5 · p. 8

Related papers

Loading 1908.11602v2…