Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

Sharath Adavanne; Archontis Politis; Joonas Nikunen; Tuomas Virtanen

Source-linked AI summary

Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

Sharath Adavanne, Archontis Politis, Joonas Nikunen, Tuomas Virtanen

arXiv:1807.00129v3Jun 30, 2018cs.SDeess.AS

TL;DR

Existing SELD methods are limited in handling multiple overlapping sources and often depend on array-specific features. This paper proposes SELDnet, a convolutional recurrent network using continuous 3D DOA regression and SED confidence, and reports higher DOA recall with robustness across array formats and acoustic conditions.

Problem
Existing SELD methods localize at most two overlapping events, rely on method-specific features, and lack comprehensive evaluation across competitive baselines and acoustic conditions.
Method
SELDnet jointly recognizes, localizes, and tracks overlapping sound events using SED outputs and continuous 3D Cartesian DOA regression from phase and magnitude spectrograms.
Results
The proposed method achieves higher DOA recall than baseline methods and remains robust to reverberation, low SNR, unseen DOAs, and moderately mismatched rooms across Ambisonic and circular arrays.
Takeaways & Limitations
SELDnet provides a generic approach for simultaneously recognizing, localizing, and tracking multiple overlapping sound events across array structures.

Abstract

from arXiv · show

In this paper, we propose a convolutional recurrent neural network for joint sound event localization and detection (SELD) of multiple overlapping sound events in three-dimensional (3D) space. The proposed network takes a sequence of consecutive spectrogram time-frames as input and maps it to two outputs in parallel. As the first output, the sound event detection (SED) is performed as a multi-label classification task on each time-frame producing temporal activity for all the sound event classes. As the second output, localization is performed by estimating the 3D Cartesian coordinates of the direction-of-arrival (DOA) for each sound event class using multi-output regression. The proposed method is able to associate multiple DOAs with respective sound event labels and further track this association with respect to time. The proposed method uses separately the phase and magnitude component of the spectrogram calculated on each audio channel as the feature, thereby avoiding any method- and array-specific feature extraction. The method is evaluated on five Ambisonic and two circular array format datasets with different overlapping sound events in anechoic, reverberant and real-life scenarios. The proposed method is compared with two SED, three DOA estimation, and one SELD baselines. The results show that the proposed method is generic and applicable to any array structures, robust to unseen DOA values, reverberation, and low SNR scenarios. The proposed method achieved a consistently higher recall of the estimated number of DOAs across datasets in comparison to the best baseline. Additionally, this recall was observed to be significantly better than the best baseline method for a higher number of overlapping sound events.

I. INTRODUCTION · A. Sound event detection · B. Sound source localization

SELD jointly detects sound events, estimates their spatial trajectories, and associates them with labels. The introduction frames overlapping-event detection and DOA estimation as complementary challenges shaped by microphone-array geometry and feature design.

I. INTRODUCTION: SELD combines temporal sound-event activity detection, spatial-location trajectory estimation, and association of textual labels with active events.
I. INTRODUCTION: SELD can support automatic activity description, hearing-impaired sound visualization, robot navigation, natural interaction, and audio surveillance.
A. Sound event detection: SED detects event onsets, offsets, and labels, while polyphonic SED recognizes sound events that overlap in time.
A. Sound event detection: Multichannel SED methods ranked among the top performers in DCASE 2016 and 2017 real-life sound-event-detection challenges.
A. Sound event detection: Recognition of overlapping sound events improved as spatial sampling increased, with the best performance obtained using first-order Ambisonics.First-order Ambisonics (FOA) corresponds to four channels.
B. Sound source localization: Sound-source localization determines a source direction or position relative to a microphone; this paper focuses on direction-of-arrival estimation.DOA methods include parametric and deep-neural-network approaches.
B. Sound source localization: DNN-based localization was introduced to address drawbacks of parametric methods, including limited robustness to reverberation and low SNR, while integrating with SELD.Existing methods also simultaneously detect DOAs of overlapping sound events.
B. Sound source localization: Prior DOA studies used differing array geometries and generally estimated full azimuth, whereas fewer methods jointly estimated azimuth and elevation.These differences make direct performance comparison difficult.

C. Joint localization and detection · D. Contributions of this paper · II. METHOD

The paper addresses polyphonic SELD by jointly associating sound events with multiple DOAs over time, overcoming limitations of existing methods in source count, direction coverage, and feature generality. Its CRNN uses separate phase and magnitude spectrogram features to predict frame-level event activity and corresponding spatial trajectories in parallel.

C. Joint localization and detection: Overlapping-event DOA estimation requires associating multiple estimates with their respective sources, while polyphonic SELD additionally requires association between recognized events and estimated DOAs.When SED and DOA estimation are performed separately, this becomes a data association problem.
D. Contributions of this paper: Existing SELD research is limited, with no comprehensive study of DNN-based SED, DOA, and SELD design choices across competitive baselines and diverse acoustic conditions.Only one published DNN-based SELD approach is identified in the passage.
D. Contributions of this paper: Prior SELD methods localize at most two overlapping events, while the only DNN-based method uses predefined direction grids and requires many output classes for more labels or finer spatial resolution.Existing approaches also use method-specific features, according to the passage.
D. Contributions of this paper: The proposed method is the first to localize and recognize more than two overlapping sound events simultaneously while tracking their activity over time.It also localizes sources at any azimuth and elevation angles, as stated in the passage.
D. Contributions of this paper: The proposed method and all datasets are publicly available, along with real-life impulse responses for experiments using custom sound events.The stated purpose is to facilitate reproducibility and enable custom dataset experimentation.
II. METHOD: The method takes multichannel audio, extracts separate phase and magnitude spectrogram features from each channel, and processes consecutive spectrogram frames.These features are supplied as a sequence to the proposed SELD model.
II. METHOD: A CRNN maps each feature sequence to parallel outputs that identify active sound-event classes and predict their respective spatial locations for every input frame.The outputs produce temporal activity and a DOA trajectory for each sound-event class.

A. Feature extraction · B. Neural network architecture · C. Training procedure

The method extracts separate magnitude and phase spectrogram features, processes them with convolutional and recurrent layers, and jointly predicts sound-event activity and Cartesian DOAs. Training combines binary cross-entropy and MSE objectives, with Cartesian regression avoiding angular wrap-around discontinuities.

A. Feature extraction: The feature extractor applies an M-point DFT to each channel using Hamming windows with 50% overlap, retaining M/2 positive-frequency bins without the zeroth bin.Magnitude and phase are extracted as separate features, producing T × M/2 × 2C feature sequences.
B. Neural network architecture: Multiple 2D CNN layers learn local shift-invariant spectrogram features with P filters using 3 × 3 × 2C receptive fields and ReLU activations.Kernels spanning all channels enable learning of relevant inter-channel information.
B. Neural network architecture: CNN activations are reshaped into T feature vectors of length 2P and passed to bidirectional GRU layers that learn temporal context.Each GRU layer uses Q nodes with tanh activations.
B. Neural network architecture: Parallel fully connected branches perform SED and DOA estimation, with weights shared across time steps.The architecture is called SELDnet.
B. Neural network architecture: SELDnet outputs SED confidences in [0, 1] and DOA coordinates in [−1, 1] for each axis, selecting an event and its DOA when SED confidence exceeds 0.5.Network hyperparameters are optimized using cross-validation.
C. Training procedure: Training uses binary cross-entropy for SED predictions and MSE for DOA estimates, with active-event coordinates as targets and zero coordinates for inactive events.The MSE is a scaled version of squared 3D Euclidean distance, so reducing it reduces the distance between predicted and reference points.
C. Training procedure: Cartesian coordinates provide continuous regression targets across angular wrap-around boundaries, unlike azimuth and elevation, which are discontinuous there.The method trains with a weighted MSE and binary cross-entropy combination for 1000 epochs using Adam, early stopping, and a 100-epoch non-improvement criterion.

III. EVALUATION … 3) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response (REAL) dataset:

The evaluation uses seven datasets spanning artificial and real-life impulse responses, array formats, acoustic conditions, and varying source overlap. The REAL dataset specifically uses Eigenmike4 measurements and isolated UrbanSound8K events to synthesize spatialized FOA scenes.

III. EVALUATION: The proposed SELDnet is evaluated on seven datasets covering anechoic and reverberant scenarios, circular and Ambisonics arrays, and real-life impulse responses.Four datasets use artificial impulse responses, while three use real-life impulse responses recorded with a spherical array and encoded into Ambisonics.
A. Datasets: The datasets consist of stationary point sources, with each source associated with a sound event class.
1) TUT Sound Events 2018 - Ambisonic, Anechoic and Synthetic Impulse Response (ANSYN) dataset:: The ANSYN dataset uses artificial impulse responses in an anechoic environment and includes subsets with no overlap, up to two overlaps, and up to three overlaps.Each subset has three cross-validation splits containing 240 training and 60 testing FOA recordings of 30 s at 44100 Hz.
2) TUT Sound Events 2018 - Ambisonic, Reverberant and Synthetic Impulse Response (RESYN) dataset:: The RESYN dataset matches the ANSYN setup but places sound events within a room using the image source method.The microphone is centered in the room, and source distances range from 1 m to the room boundary at 0.5 m resolution.
2) TUT Sound Events 2018 - Ambisonic, Reverberant and Synthetic Impulse Response (RESYN) dataset:: RESYN uses a moderately reverberant 10 × 8 × 4 m room and mismatched test rooms sized at 80% and 125% of the reference room volume.Room 1 has reverberation times from 1.0 to 0.4 s across octave bands with center frequencies from 125 Hz to 4 kHz.
3) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response (REAL) dataset:: The REAL dataset collects impulse responses in a real environment using an Eigenmike4 spherical microphone array and a continuously moving Genelec G Two loudspeaker.The measurement follows a circular trajectory at one elevation at a time.
3) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response (REAL) dataset:: REAL impulse responses are estimated independently across all 32 Eigenmike channels, using MUSIC-analyzed directions and azimuths sampled at 10° resolution.Measurements cover elevations from −40° to 40° at 1 m and from −20° to 20° at 2 m, both in 10° increments.
3) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response (REAL) dataset:: REAL scenes use isolated UrbanSound8K events, randomly assigned collected distances and directions, convolved with corresponding impulse responses, and converted to FOA format.The source classes include car horn, children playing, dog barking, drilling, engine idling, gunshot, jackhammer, siren, and street music; children playing and air conditioner were excluded from synthesis.

4) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response big (REALBIG) dataset: … B. Baseline methods

The evaluation expands SELDnet across larger ambisonic REALBIG subsets and circular-array CANSYN/CRESYN datasets, while comparing it with six SED, DOA, and SELD baselines. REALBIGAMB further introduces separate ambient-noise mixtures at 0, 10, and 20 dB SNRs.

4) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response big (REALBIG) dataset:: REALBIG subsets contain 750 recordings of 30 s each, split into 600 training and 150 testing recordings.The subsets were generated for three ambisonic REAL conditions to study performance versus dataset size.
4) TUT Sound Events 2018 - Ambisonic, Reverberant and Real-life Impulse Response big (REALBIG) dataset:: REALBIGAMB mixes separately recorded ambient sound into each ambisonic REALBIG subset at 0, 10, and 20 dB SNRs.Testing ambiance was kept separate from training ambiance.
6) TUT Sound Events 2018 - Circular array, Anechoic and Synthetic Impulse Response (CANSYN) dataset:: CANSYN replicates ANSYN sound events using a circular array with radius 5 cm and eight omnidirectional microphones.Microphones are positioned every 45° from 0° through 315°, with the array plane parallel to the ground.
7) TUT Sound Events 2018 - Circular array, Reverberant and Synthetic Impulse Response (CRESYN) dataset:: CRESYN synthesizes a circular-array version of the ambisonic RESYN room 1 dataset, placing the array at the room center with its plane parallel to the floor.The circular microphone array configuration parallels the CANSYN setup.
B. Baseline methods: SELDnet is compared with six baselines comprising two SED, three DOA, and one SELD method.The SED baselines are single- and multichannel, while the DOA baselines are parametric and DNN-based.
B. Baseline methods: The baseline set spans detection, localization, and joint localization-detection tasks.The six methods are summarized in Table III.

1) SED baseline: · 2) DOA baseline: · 3) SELD baseline (HIRnet):

The paper benchmarks SELDnet against established SED, DOA, and joint SELD systems. The baselines span multichannel SED, parametric and DNN-based localization, and a circular-array CNN for joint detection and localization.

1) SED baseline:: MSEDnet is the state-of-the-art multichannel SED comparator and scales to any number of input audio channels.It won the top two positions among 34 submissions in DCASE 2017, using single-channel and multichannel modes.
1) SED baseline:: For one-to-one comparison with SELDnet, MSEDnet uses phase and magnitude STFT components instead of its original 40-band log mel-band energy inputs.The paper reports that SED metrics changed little when using STFT phase and magnitude.
2) DOA baseline:: SELDnet’s DOA estimation is compared with MUSIC, DOAnet, and AZInet, covering parametric, 3D DNN-based, and azimuth-only DNN-based approaches.MUSIC is parametric, whereas DOAnet and AZInet are DNN-based.
2) DOA baseline:: MUSIC estimates multiple narrowband source DOAs through spatial-covariance subspace decomposition and supports generic array setups.Broadband estimation combines covariance matrices over three frames and frequency bins from 50 to 8000 Hz.
2) DOA baseline:: MUSIC uses the dataset reference for the number of active sources, providing a best-possible benchmark when source-count knowledge is available.This also enables comparison with DOA estimation without knowledge of the active-source count.
2) DOA baseline:: DOAnet is the suitable DNN baseline because it attempts 3D DOA estimation for multiple overlapping sources using a CRNN and multichannel phase-magnitude spectra.The passage identifies it as the only listed method attempting this combination.
2) DOA baseline:: AZInet performs multi-label azimuth classification from 0° to 180° at 5° resolution using each channel’s phase spectrogram.Its directional-information rationale assumes omnidirectional microphones, compact arrays, and far-field sources.
3) SELD baseline (HIRnet):: HIRnet is the joint SED-and-DOA baseline, evaluated only on CANSYN and CRESYN because it targets circular arrays of omnidirectional microphones.It uses per-channel log-spectral power and predicts eight full-azimuth angles for each speech and music class.

C. Evaluation metrics

The evaluation separates sound event detection and direction-of-arrival estimation, using complementary metrics for activity classification, localization accuracy, and matching the predicted and reference numbers of DOAs. A combined SELD score supports early stopping and integrates detection with localization-dependent performance.

Sound event detection: SED is evaluated with one-second, non-overlapping segment-wise error rate (ER) and F-score derived from frame-level predictions.An event is considered active for the entire segment if active in any frame within it.
Joint evaluation: SED and DOA estimation are jointly assessed, with ideal SED performance defined as F-score one and ER zero, and ideal DOA performance as frame recall one and DOA error zero.The F-score and frame recall values are reported as percentages in the tables.
DOA estimation: DOA accuracy is measured by the central-angle error σ between each estimated and reference 3D Cartesian direction.The central angle ranges from 0 to 180 degrees, and dataset-level DOA error averages these angular differences across all estimates.
DOA estimation: Frame recall is TP/(TP + FN), where TP counts frames with matching predicted and reference DOA counts and FN counts unequal frames.It accounts for frames in which the numbers of estimated and reference DOAs differ and is reported as a percentage.
Joint evaluation: Early stopping uses a combined SELD score, which is ideal at zero and weights SED performance more than DOA because localization depends on detection.The score includes frame recall alongside the SED metrics.

D. Experiments

The experiments evaluate SELDnet across different dimensions using multiple datasets to characterize its potential and drawbacks.

D. Experiments: SELDnet is evaluated across different dimensions using experiments on different datasets.These experiments are intended to assess the method’s potential and drawbacks.

1) SELDnet architecture and model parameter tuning: … 6) Performance with ambiance at different SNR:

The paper evaluates SELDnet across architecture and output-format choices, continuous DOA generalization, reverberation mismatch, dataset size, and ambient-noise SNR conditions. These experiments cover simulated, real-life, reverberant, and ambient-noise settings.

1) SELDnet architecture and model parameter tuning:: SELDnet architectures and parameters are explored on ANSYN O2 using CNN, RNN, and FC layer counts from 0 to 4.The study also tunes nodes over [16, 32, 64, 128, 256, 512], sequence lengths over [32, 64, 128, 256, 512], and branch loss weights over [1, 5, 50, 500].
2) Selecting SELDnet output format:: Polyphonic SED uses frame-wise binary activity for each sound class, while DOA estimation compares azimuth/elevation and unit-sphere x, y, z regression formats.The passage describes frame-wise binary outputs as standardized for polyphonic SED and notes that DOA output formats remain under experimentation.
3) Continuous DOA estimation and performance on unseen DOA values:: Regression-based DOA estimation is evaluated for continuous-space generalization by testing on azimuth and elevation values shifted 5° from 10° training grids.The temporal locations remain unchanged, making the testing DOAs unseen relative to training.
4) Performance on mismatched reverberant dataset:: SELDnet is evaluated on RESYN, REAL, REALBIG, and REALBIGAMB reverberant datasets and compared with the parametric MUSIC baseline.The experiment addresses reverberant conditions, where parametric DOA methods are described as sensitive to reverberation.
4) Performance on mismatched reverberant dataset:: The study examines domain mismatch because DNN performance can degrade when trained on anechoic data and tested on reverberant data.The passage notes that training on a similar reverberant dataset improves performance but would be impractical for every room and reverberation condition.
5) Performance on the size of the dataset:: SELDnet performance is compared on REAL and REALBIG, which have similar content but different dataset sizes.This experiment isolates the effect of dataset size across the two real-life datasets.
6) Performance with ambiance at different SNR:: SELDnet performance with ambient sound is studied on REALBIGAMB at 0, 10, and 20 dB sound-event SNRs.The experiment evaluates performance across three specified signal-to-noise ratios.

7) Generic to array structure: · IV. RESULTS AND DISCUSSION · 1) SELDnet architecture and model parameter tuning:

The paper evaluates SELDnet’s generality across array structures and tunes its CRNN architecture and input settings. Results support retraining for changed array configurations and identify a compact configuration that performs best across datasets.

7) Generic to array structure:: SELDnet is designed to localize and recognize sound events from any array structure when microphone response, spacing, and count remain unchanged between training and testing.A changed array configuration requires retraining for the new array.
7) Generic to array structure:: A circular-array evaluation tests whether SELDnet generalizes beyond Ambisonics using different microphone count, planar placement, and omnidirectional responses.The evaluation compares against dataset-compatible baselines.
7) Generic to array structure:: The circular-array comparison includes SEDnet, MSEDnet, HIRnet, AZInet, and the azimuth-only SELDnet-azi version.SELDnet’s x-, y-, and z-axis DOA outputs are also reported on CANSYN and CRESYN.
7) Generic to array structure:: Across most experiments, training and testing share 468 spatial locations at 10° azimuth-and-elevation resolution, while sound-example sets remain mutually exclusive.Distance is an additional variable for events at those spatial locations.
1) SELDnet architecture and model parameter tuning:: On ANSYN O2, hyper-parameter tuning compares CNN-only, RNN-only, and CRNN SELDnet variants by SELD score and model-parameter count.The comparison is visualized in Figure 3.
1) SELDnet architecture and model parameter tuning:: The best tuned architecture uses three 64-node CNN layers, two 128-node GRU layers, one 128-node fully connected layer, and frequency pooling of (8, 8, 2).This configuration has about 513,000 parameters.
1) SELDnet architecture and model parameter tuning:: The best input settings use M = 512 with 256 frames, while tuning sequence length at fixed M = 512 favors 512 frames, equivalent to 2.97 s.Sequences longer than 512 frames were not studied because of hardware restrictions.
1) SELDnet architecture and model parameter tuning:: For RESYN, changing sequence length from 512 to 256 frames improves performance without changing the 513,000 network parameters; similar settings work best for CANSYN, CRESYN, and REAL subsets.The REALBIG and REALBIGAMB studies use the same parameters as ANSYN.

2) Selecting SELDnet output format: · 3) Continuous DOA estimation and performance on unseen DOA values: · 4) Performance on mismatched reverberant dataset:

The study selects Cartesian x, y, z outputs because angle wrap-around discontinuities hinder DOA estimation and SELD performance. SELDnet also recognizes, localizes, and tracks overlapping events, while remaining robust to reverberation and moderately mismatched rooms.

2) Selecting SELDnet output format:: Cartesian x, y, z output improves learning across datasets compared with azimuth/elevation angles.The paper attributes this to avoiding discontinuity at the angle wrap-around boundary, which reduces DOA-estimation and SELD performance.
2) Selecting SELDnet output format:: Table IV reports SED and DOA-estimation metrics for ANSYN and RESYN datasets, including RESYN room 2 and 3 tests trained on room 1.Best scores for subsets are shown in bold.
3) Continuous DOA estimation and performance on unseen DOA values:: Figure 7 visualizes SELDnet inputs and outputs for ANSYN O1 and O2 over respective 1000-frame test sequences.The plots align spectrogram, SED, and DOA reference and prediction time frames.
3) Continuous DOA estimation and performance on unseen DOA values:: SELDnet successfully recognizes, localizes in time and space, and tracks multiple overlapping sound events simultaneously.The corresponding SED predictions are reported as unaffected in the visualization.
4) Performance on mismatched reverberant dataset:: On RESYN room 1 subsets, SELDnet performs significantly better than DOAnet in frame recall, although DOAnet has lower DOA error for O1 and O2.MUSIC performs poorly in reverberant conditions, while SELDnet’s SED metrics are comparable to or better than MSEDnet’s best baseline performance.
4) Performance on mismatched reverberant dataset:: SELDnet remains robust to reverberation and performs seamlessly on moderately mismatched room configurations.In the reported comparison, DOAnet has better DOA error, MUSIC performs poorly, and SELDnet’s SED metrics are comparable to MSEDnet.
4) Performance on mismatched reverberant dataset:: SELDnet correctly estimated two sources in 76% of frames whose reference contained two sources in Figure 8c.The paper relates frame recall for DOA estimation to the confusion matrix’s source-count representation.

5) Performance on the size of the dataset: … V. CONCLUSION

SELDnet’s performance declines on real and highly complex acoustic scenes, but it remains robust across array structures and achieves stronger DOA frame recall than competing baselines. The proposed regression-based CRNN jointly recognizes, localizes, and tracks events using method-independent spectrogram features, trading lower DOA error for higher recall.

5) Performance on the size of the dataset:: On REAL, SELDnet’s overall performance declines versus ANSYN and RESYN, while MSEDnet performs better on SED metrics.SELDnet retains significantly better DOA frame recall than DOAnet, although DOAnet has lower DOA error.
6) Performance with ambiance at different SNR:: SELDnet remains robust at 10 and 20 dB SNR with ambiance, shows small drops on O2 and O3 subsets, and drops considerably at 0 dB SNR.Across datasets, SELDnet has lower DOA error than MUSIC but higher error than DOAnet, while providing significantly higher frame recall than DOAnet.
7) Generic to array structure:: For circular arrays, SELDnet-azi outperforms MSEDnet on SED metrics across all CRESYN subsets but not CANSYN O2 and O3.SELDnet-azi has better frame recall than AZInet across datasets except CANSYN O1, whereas AZInet has lower DOA error.
7) Generic to array structure:: Using SED output as a confidence measure improves SELDnet’s DOA frame recall significantly and consistently across evaluated datasets.Its DOA error remains consistently higher than the classification-based baseline, potentially because regression does not fully learn the continuous DOA mapping.
V. CONCLUSION: SELDnet simultaneously recognizes, localizes, and tracks sound events over time by estimating each class’s DOA with three regressors for 3D Cartesian coordinates on a unit sphere.Regression enables DOA estimation in a continuous space.
V. CONCLUSION: The SED-derived DOA confidence measure extends SED performance to SELD and yields higher DOA recall, while classification baselines achieve lower DOA error despite poorer recall.SELDnet uses phase and magnitude spectrograms as non-method-specific input features.