PaperlayerView exact revision on arXiv

Source-linked AI summary

Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019

Archontis Politis, Annamaria Mesaros, Sharath Adavanne, Toni Heittola, Tuomas Virtanen

arXiv:2009.02792v2eess.AScs.SD

TL;DR

SELD needed evaluation methods and data that jointly captured what sounds occurred, when they occurred, and where they were located. The paper reviews the first DCASE 2019 SELD challenge, its dataset and systems, and reevaluates submissions with joint metrics. The analysis shows that separate detection and localization rankings could favor systems that performed worse on the combined task.

  • Problem

    SELD combines event detection with localization, but challenge evaluation separately measured detection and localization, motivating joint measurement.

  • Method

    The paper overviews the DCASE 2019 challenge, analyzes submitted systems, and applies joint localization-detection metrics with confidence intervals and rank-correlation analysis.

  • Results

    17 of 22 submitted systems ranked above the baseline; the best reported ER was 0.06, F1-score 96.7%, LE 2.7°, and ECR 96.8%.

  • Takeaways & Limitations

    Joint reevaluation changed the standing of systems that were strong on detection or localization separately but weaker when both had to be correct together.

  • Takeaways & Limitations

    DoA-classification systems exploited the dataset’s 324 fixed DoAs and may require many more classes for dense, continuous, or moving-source settings.

Abstract

from arXiv · show

Sound event localization and detection is a novel area of research that emerged from the combined interest of analyzing the acoustic scene in terms of the spatial and temporal activity of sounds of interest. This paper presents an overview of the first international evaluation on sound event localization and detection, organized as a task of the DCASE 2019 Challenge. A large-scale realistic dataset of spatialized sound events was generated for the challenge, to be used for training of learning-based approaches, and for evaluation of the submissions in an unlabeled subset. The overview presents in detail how the systems were evaluated and ranked and the characteristics of the best-performing systems. Common strategies in terms of input features, model architectures, training approaches, exploitation of prior knowledge, and data augmentation are discussed. Since ranking in the challenge was based on individually evaluating localization and event classification performance, part of the overview focuses on presenting metrics for the joint measurement of the two, together with a reevaluation of submissions using these new metrics. The new analysis reveals submissions that performed better on the joint task of detecting the correct type of event close to its original location than some of the submissions that were ranked higher in the challenge. Consequently, ranking of submissions which performed strongly when evaluated separately on detection or localization, but not jointly on both, was affected negatively.

I. INTRODUCTION

SELD combines sound event detection with source localization to characterize acoustic scenes across time and space. The paper reviews the first DCASE 2019 SELD challenge, its systems, and joint evaluation of detection and localization.

  • SELD combines event recognition and temporal occurrence detection with spatial source localization for fuller acoustic-scene characterization.The spatial dimension supports applications including environment inference, robotic mapping, navigation, and audio surveillance.
  • Before DCASE 2019, only a handful of SELD approaches existed, and earlier work generally treated detection and localization separately.Prior methods included Gaussian mixture models, hidden Markov models, and support vector machines.
  • Deep neural networks became promising candidates for joint SELD because they had established strong performance in detection and shown promising localization results.Early joint approaches used generic multichannel power, magnitude, and phase spectrogram features.
  • DCASE 2019 introduced SELD as a challenge task with a new spatialized-event dataset and a SELDnet baseline.The challenge attracted a remarkable number of submissions for a novel topic, followed by further work addressing its limitations.
  • The paper overviews challenge systems and reevaluates them using joint localization-detection metrics alongside separate performance analyses.It also calculates confidence intervals and examines correlations between metrics and system rankings.

II. SOUND EVENT DETECTION AND LOCALIZATION IN DCASE 2019 CHALLENGE

The DCASE 2019 challenge defined SELD around identifying sound events, their timing, and their spatial trajectories, using a realistic spatialized dataset and a SELDnet baseline. Dataset construction combined real room measurements and ambient noise with simulated mixtures, while systems used several architectural approaches.

  • Task definition: SELD identifies sound-event classes, onset and offset times, and spatial trajectories from multichannel recordings.The 2019 challenge considered static scenes with fixed DoAs in azimuth and elevation.
  • Baseline and system approaches: Common SELD architectures included joint class-location DNNs, separate classification and localization DNNs, and DNN classification paired with parametric localization.The SELDnet baseline used stacked multichannel magnitude and phase spectrograms with jointly weighted detection and localization losses.
  • Dataset: The dataset challenge required variability across events, overlaps, positions, rooms, acoustic conditions, and ambient noise that real-only recordings could not practically cover.Acoustic simulation improved control but could deviate from real spatial room impulse responses.
  • Dataset: The challenge used a hybrid recording-simulation strategy with real impulse responses from five indoor locations and 504 azimuth-elevation-distance combinations.The recorded domain covered 360° azimuth, -40° to 40° elevation, and 1 to 2 m distance.
  • Dataset: Mixtures were one-minute multichannel recordings of spatialized events with randomized onsets and positions, up to two simultaneous events, and recorded ambient noise.Events were convolved with room impulse responses from the same space before mixture synthesis.

C. Evaluation and ranking

The challenge initially evaluated sound event detection and localization separately, then formulated joint metrics that assess class correctness together with spatial proximity. These metrics expose cases where separate rankings favor systems that do not jointly detect events near their reference locations.

  • The initial evaluation measured detection and localization performance individually.
  • Separate detection and localization scores can both be perfect when event locations are swapped, because standalone detection ignores spatial identity.The example motivates evaluating whether the correct event type is detected close to its original location.
  • Localization error uses angular distances between associated predictions and references, with associations computed by the Hungarian algorithm.Only associated predictions contribute to the localization error; the association count is K = min(M, N).
  • Localization recall complements localization error by accounting for the number of localized events detected relative to the references.Event count recall measures the proportion of frames or segments in which the predicted and reference event counts match.
  • Joint evaluation matches predictions and references by class, then uses spatial thresholds to count true positives, false positives, and false negatives.Class-aware localization evaluates spatial error only for correctly classified events, while location-aware detection imposes a spatial constraint on correct classifications.
  • The joint framework supports class-dependent localization metrics and optional class-specific spatial thresholds, although the study uses non-thresholded localization metrics for joint results.

B. Segment-based measurement

Segment-based metrics extend frame-level sound event detection and localization measurements to one-second segments. The paper presents segment-level localization alternatives and uses one-second evaluation except for additional frame-level analysis in the reevaluation.

  • Segment-based detection treats an event as active when it occurs in at least one frame within the segment.
  • Localization can be generalized to segments using a mean direction-of-arrival or Cartesian position before measuring error.Alternatively, localization error can be averaged from frame-level reference-prediction pairs within the segment.
  • The study evaluates results in one-second segments, adding frame-level localization results during the Section V reevaluation.

IV. CHALLENGE RESULTS

The first DCASE 2019 SELD task attracted substantial participation for a newly introduced challenge. It received 58 system submissions from 22 teams comprising 65 members affiliated with universities and companies.

  • 58 systems were submitted by 22 teams comprising 65 members.
  • Participants were affiliated with 16 universities and 8 companies.

A. Overall challenge results

The challenge results show broad gains over the baseline, while separate detection and localization rankings can disagree and system designs vary substantially.

  • 17 of 22 submitted systems ranked above the baseline under the official ranking method.
  • 17 systems surpassed the baseline on both ER and F1-score, with best values of ER 0.06 and F1-score 96.7%.
  • 18 systems exceeded the baseline on LE and 14 on ECR, with best values of LE 2.7° and ECR 96.8%.
  • Detection and localization metrics produced different ranking orders among the top systems.He THU led detection but had relatively high LE, whereas Chang HYU achieved the best LE among the top systems.
  • All submissions used at least one deep-learning component, with most using CRNN-based architectures and some combining separate SED and DoA networks.Nine systems used multitask learning; others used engineered data association, and no significant performance difference was found between the two audio formats.

B. Analysis of individual systems

The strongest systems combined specialized features, training strategies, post-processing, or task-specific priors, producing different trade-offs between detection and localization.

  • Kapka SRPOL ranked first with very high localization and detection performance using four dedicated CRNN subtasks.Its pipeline estimated source count and directions before classifying events, with post-processing coupling outputs across stages.
  • Kapka SRPOL’s prior knowledge enabled resolving two same-class simultaneous events but reduced generality.The architecture assumed at most two simultaneous sources and directions quantized at 10° intervals.
  • Cao Surrey improved the baseline with domain-specific features, separated detection and localization losses, two-stage training, and input ensembling.Ablation studies attributed a drastic performance effect to the improved features and two-stage training architecture.
  • Xue JDAI achieved stronger detection than the top two systems but lower localization, combining spectral and spatial features with elaborate post-processing.The system used SRP-based DoA estimation, refined by the CRNN only for simultaneous events, and separate CNN branches before recurrent layers.
  • He THU achieved the best overall SED performance, while localization was only marginally better than the baseline.SpecAugment may have supported detection, whereas basic localization spectrogram features may have limited localization accuracy.
  • Some systems exploited separate detection and localization evaluation, including random DoA association or engineered task-specific processing.Nguyen NTU randomly associated DoAs with overlapping detections, while MazzonYasuda NTT used spatial augmentation and GCC-PHAT features.
  • Chang HYU added sound-activity models to select event outputs and used a CNN for DoA estimation.
  • Other submissions used staged transfer learning, TrellisNet layers, microphone-pair CRNNs, intermediate TDOAs, or model-based DoA estimation.These alternatives illustrate the range of architectural and signal-processing choices across the submitted systems.

C. Discussion on submitted systems

Submitted systems commonly improved SELD by using task-focused features, separating detection and localization, and exploiting task-specific architectural choices. However, these strategies introduced limitations in event association and generalization beyond the challenge setting.

  • Input features: Task-focused features such as log-mel spectra, GCC-PHAT arrays, and active intensity vectors were popular among top submissions and often improved detection or localization.The top system nevertheless succeeded with raw multichannel spectrograms by tightly coupling its architecture to the task and exploiting prior knowledge.
  • Architecture and training: Two-stage systems separating SED and DoA estimation were popular and showed improved results over joint modeling in one comparison.Randomly associating detections and localizations exploited the challenge’s separate evaluation but would not be suitable in practice.
  • Event association: Most submissions tied one DoA output to each class, limiting systems to one localized event per class when same-class instances occurred simultaneously.Localize-before-detect systems instead produced class-independent event locations before classification, avoiding this association problem.
  • Localization: DoA classification benefited from the dataset’s 324 fixed directions but may require many more classes for dense, continuous, or moving-source scenarios.Fully classification-based joint models would require no. of DoA classes × no. of event classes outputs.
  • Localization: Parametric DoA estimation required no training and could generalize to unseen environments, but was more susceptible to reverberation without additional processing.The third-best system used this approach, and another submission achieved one of the smallest DoA errors.

V. REEVALUATION OF CHALLENGE ENTRIES USING JOINT METRICS

The paper reevaluates all DCASE 2019 Task 2 systems with joint localization-and-detection measures. It uses segment-level evaluation, multiple angular thresholds, confidence intervals, and new cumulative rankings for comparison.

  • Joint evaluation: All DCASE 2019 Task 2 submissions were evaluated with joint measures combining localization and detection performance.The analysis sought a single metric encompassing all aspects of system performance.
  • Evaluation setup: The joint metrics were computed in one-second segments using 10° and 30° angular error thresholds for location-aware detection.The results were presented in a table ordered by official challenge rank.
  • Ranking analysis: Jackknife confidence intervals were calculated by leaving out one file at a time, and cumulative ranks were estimated from the proposed joint measures.The top ten systems were also visualized using the joint evaluation results.

A. Analysis of systems

Joint metrics changed the interpretation of challenge performance by penalizing poor association between detections and locations and reducing the official ranking’s detection bias. Systems with strong localization and proper association could therefore rise substantially.

  • Joint metrics: The joint metrics were stricter than independent localization and detection metrics, using location-aware measures such as LRCD, LECD, F10° and ER10°.A 10° threshold was selected as relatively strict relative to the systems’ average localization errors.
  • Ranking effects: Systems that separated localization and detection without proper association tended to fall in rank, especially those assigning DoAs randomly to detections.Their association problems appeared as reduced location-aware detection scores and larger localization-related errors.
  • Ranking effects: Systems with stronger detection than localization also dropped because the original metrics favored detection performance.The new metrics reduced this imbalance by evaluating detection conditional on spatial accuracy.
  • Ranking effects: Properly associated systems with stronger localization improved their ranks despite not having the best detection scores.Park ETRI reached fourth place, while PerezLopez UPF rose from nineteenth place below the baseline to seventh under strict location-dependent detection.
  • Threshold sensitivity: A 30° threshold produced rankings closer to the official ordering because relaxed localization criteria approached independent detection metrics.The threshold also exceeded the systems’ average LECD of about 20°.

B. Metrics analysis

The metric analysis shows that official rankings were driven more by detection than localization, while joint metrics differed in how strictly they integrated spatial accuracy. Error-rate metrics offered the strongest single-metric compromise across ranking, detection, and localization criteria.

  • Correlation with official ranking: Detection metrics F1 and ER were highly correlated with the official ranking, whereas localization error was less correlated.This indicates that strong detection performance was important for achieving a high official rank.
  • Joint metrics: Among joint metrics, segment-based LRCD correlated most strongly with the official ranking, more than frame-based LRCD(f).More permissive metrics generally correlated more strongly with the official rank than stricter alternatives.
  • Metric relationships: Data association made metrics less permissive, paralleling the lower correlations observed for stricter 10° thresholds than for 30° thresholds.Localization-only and detection-only metric pairs showed stronger within-family correlations than detection metrics paired with location-aware counterparts.
  • Metric relationships: LRCD correlated 0.93 with the official ranking, while its correlation with LECD was only 0.50.Thus, selecting a model using LRCD alone may not guarantee the best LECD.
  • Metric selection: Error-rate metrics ER10° and ER30° were recommended when only one metric could be used because they aligned with official ranking, location-aware F1, and LECD.They also showed moderately high correlation of 0.71–0.81 with the official rank for the corresponding location-aware detection metrics.

C. Discussion

The challenge systems achieved very high localization and detection performance, but the dataset represented only a restricted subset of real spatial scenes. The discussion therefore motivates more realistic evaluation conditions and notes that the proposed joint-metric analysis remains empirical.

  • The dataset used static, well-separated events, large open spaces, close microphones, high direct-to-reverberant ratios, and high-SNR ambient noise.These conditions limited corruption of spatial and spectral event characteristics.
  • DCASE2020 focused on more challenging evaluation conditions closer to reality after DCASE2019 confirmed successful performance under restricted dataset conditions.
  • The proposed metrics remain an open research topic because their accuracy, consistency, and general behavior have not yet been rigorously analyzed.The reported findings should therefore be interpreted as empirical, based on observations and correlations with expected system behavior.
  • Future SELD challenges still need to address moving receivers and sources as well as directional interferences of unknown types.These spatial properties had not yet been addressed in the discussed challenge progression.
Loading 2009.02792v2…