PaperlayerView exact revision on arXiv

Source-linked AI summary

Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy

Yin Cao, Qiuqiang Kong, Turab Iqbal, Fengyan An, Wenwu Wang, Mark D. Plumbley

arXiv:1905.00268v4cs.SDeess.AS

TL;DR

Sound event detection and localization involve identifying events and estimating their temporal and spatial locations, but their interaction in learning-based systems is not fully understood. This paper trains SED first, transfers its feature layers to DOAE, and uses SED ground truth as a training mask. The proposed method improves both tasks and significantly outperforms the baseline.

  • Problem

    The relationship between SED and DOAE remains insufficiently understood, while the SELDnet baseline jointly trains them using phase spectrograms that are difficult for neural networks to learn from.

    Source: §1 · pp. 1–2

  • Method

    The method trains an SED branch first, transfers its learned feature layers to DOAE, and uses SED ground truth as a mask when training DOAE.

    Source: §1 · p. 2

  • Results

    The proposed method improves both SED and DOAE and significantly outperforms the baseline, whereas joint training degrades both tasks.

    Source: §1 · p. 2, §4 · pp. 4–5

  • Takeaways & Limitations

    SED predictions can contribute to DOAE while separate task training improves performance on both tasks.

    Source: §1 · p. 2, §4 · p. 4

Abstract

from arXiv · show

Sound event detection (SED) and localization refer to recognizing sound events and estimating their spatial and temporal locations. Using neural networks has become the prevailing method for SED. In the area of sound localization, which is usually performed by estimating the direction of arrival (DOA), learning-based methods have recently been developed. In this paper, it is experimentally shown that the trained SED model is able to contribute to the direction of arrival estimation (DOAE). However, joint training of SED and DOAE degrades the performance of both. Based on these results, a two-stage polyphonic sound event detection and localization method is proposed. The method learns SED first, after which the learned feature layers are transferred for DOAE. It then uses the SED ground truth as a mask to train DOAE. The proposed method is evaluated on the DCASE 2019 Task 3 dataset, which contains different overlapping sound events in different environments. Experimental results show that the proposed method is able to improve the performance of both SED and DOAE, and also performs significantly better than the baseline method.

1. INTRODUCTION

Sound event detection and localization both matter in real-world sound analysis, but their interaction in learning-based systems remains insufficiently understood. The paper proposes a two-stage method that transfers SED features to DOAE and reports improved performance on both tasks over the baseline.

1. INTRODUCTION

  • SED predicts when events occur, unlike audio tagging, which only identifies whether events are present in a clip.Neural approaches include CNNs, RNNs, and CRNNs, which combine convolutional and recurrent layers.Source: §1 · p. 1
  • Parametric GCC methods estimate time differences using cross-spectrum phase, but their primary limitation is handling multiple sources.Learning-based DOAE methods have also been studied, though prior neural approaches mainly considered static sources.Source: §1 · p. 1
  • Real-world sound events have spatial directions as well as event identities and temporal information, motivating investigation of SED and localization together.DCASE 2019 Task 3 introduced sound event localization and detection for overlapping sound sources.Source: §1 · p. 1
  • The SELDnet baseline jointly trains SED and DOAE using magnitude and phase spectrograms, while the paper notes that phase spectrograms are difficult for neural networks to learn from.Relationships between SED and DOAE had not been revealed in that setup.Source: §1 · p. 2
  • The proposed two-stage method improves both SED and DOAE and significantly outperforms the baseline, while preserving SED's contribution to DOAE.SED is trained first; its learned feature layers are transferred to the DOAE branch, which uses the SED ground truth as a training mask.Source: §1 · p. 2
  • Joint training of SED and DOAE degrades both tasks, despite evidence that SED can contribute to DOAE performance.The paper responds by separating task training while transferring learned SED features to DOAE.Source: §1 · p. 2

2. LEARNING METHOD

The method pairs log-mel and GCC-PHAT features with a two-stage network: it trains SED first, then transfers and fine-tunes its CNN feature layers for DOAE. The DOAE branch uses SED masks during training and SED predictions during inference.

2. LEARNING METHOD

  • The proposed two-stage network is intended to exploit the tasks’ mutual strength, while the baseline jointly trains them with a weighted combination of SED and DOAE losses.The baseline’s weighted loss entails a trade-off between the tasks according to the selected loss weight.Source: §2 · p. 2

2.1.1. Ambisonics

  • Ambisonics encodes plane-wave source signals with spherical-harmonic vectors that carry DOA information; higher order provides higher directional resolution.Order L requires at least (L + 1)^2 microphones.Source: §2.1.1 · p. 2

2.1.2. Generalized Cross-Correlation

  • GCC-PHAT whitens the cross-power spectrum to retain phase, but GCC-based TDOA estimates can be unstable in high reverberation or low-SNR settings and do not directly handle multiple sources.The input’s GCC-PHAT dimension is set using the largest microphone spacing and maximum delay; the mel-band count should be at least twice the maximum delayed samples plus one.Source: §2.1.2 · p. 2

2.1. Features

  • Log-mel spectrograms and GCC-PHAT, which carries phase-difference information between microphone pairs, are stacked as input features.The feature formats are paired with First-Order Ambisonics or a tetrahedral microphone array.Source: §2.1.2 · p. 2, §2.1 · p. 2

2.2. Network architecture

  • The DOAE branch transfers and fine-tunes the SED CNN layers, using ground-truth SED masks for training and SED predictions for inference.The SED branch predicts event activity; the DOAE branch outputs azimuth and elevation for each event.Source: §2.2 · p. 3, Figure 1 · p. 3

3. EXPERIMENTAL STUDY

The experiments compare SED and DOAE methods across input features, network architectures, and data formats. The proposed two-stage method achieves the best metric scores among the compared methods, while results also show that trained SED features contribute to DOAE.

3.1. Evaluation metrics

  • SED is evaluated with segment-based error rate, F-score, and frame-level mAP; DOAE uses DOA error and frame recall.Lower ER and DOA error, and higher F-score, mAP, and frame recall, indicate better performance.Source: §3.1 · p. 3

3.2. Methods for comparison

  • Comparisons include the magnitude-and-phase baseline, SELDnet with log-mel and GCC-PHAT, DOA with transferred CNN layers, and DOA-NT without transfer.Methods are evaluated using CNNs and CRNNs with both microphone-array and FOA data.Source: §3.2 · pp. 3–4

3.3. Hyper-parameters

  • Training uses two-second clips with one-second overlap, 32 kHz audio, and 50 epochs.The learning rate starts at 0.001 for 30 epochs, then decays by 10% each epoch; features use a 1024-point Hanning window with a 320-point hop.Source: §3.3 · p. 4

3.4. Results

  • The proposed two-stage method has the best metric scores among the compared methods; joint training degrades both tasks relative to it.SELDnet with log-mel and GCC-PHAT inputs also outperforms the baseline using magnitude and phase spectrograms.Source: §3.4 · p. 4, Table 1 · p. 4
  • FOA is not shown to outperform four-channel microphone-array data, while CRNNs perform better than CNNs in most cases.For one clip, most SED and DOAE predictions are accurate in temporal and spatial dimensions.Source: §3.4 · p. 4, Figure 2 · p. 4

4. CONCLUSION

The paper proposes a two-stage method for polyphonic sound event detection and localization, using SED information to support DOAE. It reports improved branch representations and significantly better performance than the baseline.

  • The method trains the SED branch first, transfers its learned feature layers to the DOAE branch, and uses SED ground truth as a mask for DOAE training.Source: §4 · p. 4
  • The paper reports that SED information can improve DOAE, whereas joint training of SED and DOAE degrades both tasks.Source: §4 · p. 4
  • The proposed method significantly outperforms the baseline while enhancing each branch's network representation and retaining SED contributions to DOAE.Source: §4 · p. 5
Loading 1905.00268v4…