Source-linked AI summary
Sound Event Detection in Multichannel Audio Using Spatial and Harmonic Features
Sharath Adavanne, Giambattista Parascandolo, Pasi Pertilä, Toni Heittola, Tuomas Virtanen
TL;DR
Polyphonic sound event detection must identify overlapping events, while existing systems have used mono-channel audio. This paper combines spatial and harmonic features with an RNN-LSTM; the reported results are promising, though the limited dataset constrains conclusions about superiority.
Problem
Polyphonic SED must recognize overlapping sound events, while state-of-the-art systems described here use single-channel audio.
Source: §1 · p. 1Method
The system combines stereo-derived spatial TDOA features and harmonic pitch features with an RNN-LSTM for multi-label sound event detection.
Source: §1 · p. 1, §2.2 · p. 2, §2.3 · p. 2Results
Most evaluated feature combinations performed equal to or better than the mono-channel baseline in average F-scores, with marginally similar average ER.
Source: §4.3 · p. 4Takeaways & Limitations
The proposed multichannel features look promising for SED, but the reported evaluation does not conclusively establish their superiority over monaural features.
Source: §4.3 · p. 4, §5 · p. 4Pitch features were limited to the top three dominant values per frame because training examples had no more than three overlapping events.
Source: §2.2 · p. 2
Abstract
from arXiv · showhide
In this paper, we propose the use of spatial and harmonic features in combination with long short term memory (LSTM) recurrent neural network (RNN) for automatic sound event detection (SED) task. Real life sound recordings typically have many overlapping sound events, making it hard to recognize with just mono channel audio. Human listeners have been successfully recognizing the mixture of overlapping sound events using pitch cues and exploiting the stereo (multichannel) audio signal available at their ears to spatially localize these events. Traditionally SED systems have only been using mono channel audio, motivated by the human listener we propose to extend them to use multichannel audio. The proposed SED system is compared against the state of the art mono channel method on the development subset of TUT sound events detection 2016 database. The usage of spatial and harmonic features are shown to improve the performance of SED.
1. INTRODUCTION
Polyphonic sound event detection must recognize overlapping events, whereas monophonic systems focus on the dominant event. This paper extends mono-channel SED with spatial and harmonic features alongside an RNN-LSTM.
1. INTRODUCTION
- Automatic SED applications include environmental monitoring, smart homes, and multimedia annotation for content-based retrieval.A sound event is an audio segment that listeners can consistently label and distinguish.Source: §1 · p. 1
- Polyphonic SED recognizes all overlapping events, unlike monophonic systems, which target only the most dominant event.The paper focuses on polyphonic soundscapes as a closer match to real-life scenarios.Source: §1 · p. 1
- Prior polyphonic SED work used MFCCs with HMMs, NMF-based approaches, or generalized Hough transforms with local spectral features.One NMF approach retained a constraint on the number of overlapping events, later addressed using coupled NMF.Source: §1 · p. 1
- Recent state-of-the-art SED systems use log mel-band energies with DNNs or multi-label RNN-LSTMs; this paper continues with RNN-LSTM.The choice is motivated by reported RNN-LSTM performance over DNN in.Source: §1 · p. 1
- The proposed spatial and harmonic features are reported to outperform mono-channel features even with a dataset of around 60 minutes.The paper frames multichannel audio as a way to use spatial information for sound event detection.Source: §1 · p. 1
2. SOUND EVENT DETECTION
Sound event detection locates events in time and assigns known labels, including when events overlap. The proposed system uses stereo-derived log mel-band energy, pitch and periodicity, and band-specific TDOA features.
2. SOUND EVENT DETECTION
- Sound event detection identifies event locations over time and assigns them to known labels, including for isolated or overlapping events.The system receives stereo input and aims to detect and classify both isolated and overlapping events.Source: §2 · pp. 1–2
2.1. Log mel-band energy
- The system extracts log mel-band energies from both stereo channels, motivated by the potential to learn information similar to interaural intensity difference.Each channel is framed into 40 ms windows with 50% overlap, and 40 mel-bands span the spectrum.Source: §2.1 · p. 2
2.2. Harmonic features
- The harmonic features are absolute pitch and periodicity, which provides a confidence measure for extracted pitch values.Because training overlaps contained at most three events at a time, the system uses up to three dominant pitch values per frame.Source: §2.2 · p. 2
2.3. Time difference of arrival (TDOA) features
- Overlapping events combine in a frame’s feature vector, while spatial localization through interaural time delay is a human cue for identifying them.The system divides the spectrum into mel-bands and estimates TDOA in each band, giving the classifier spatial features across frequency.Source: §2.3 · p. 2
- TDOA is estimated separately in five mel-bands; different bandwise delays can represent overlapping sources with different frequency distributions.The correlation peak gives the TDOA for each mel-band and frame. The system also concatenates estimates from 120, 240, and 480 ms windows, and uses median filtering to address noisy values.Source: §2.3 · pp. 2–3
3. MULTI-LABEL RECURRENT NEURAL NETWORK BASED SOUND EVENT DETECTION
The section uses RNN-LSTM networks for multi-label sound event detection, with recurrent context and an output that can mark several sound classes active in the same frame. The network is trained and evaluated on normalized feature sequences using frame-level class targets.
3. MULTI-LABEL RECURRENT NEURAL NETWORK BASED SOUND EVENT DETECTION
- RNNs incorporate information from previous timesteps, while LSTM architectures are used to attenuate the vanishing-gradient problem.The paper uses LSTM as its RNN architecture.Source: §3 · p. 3
- A sigmoid output predicts each class's activity probability, and thresholding those predictions yields a binary activity matrix.This supports simultaneous active-class predictions in a frame.Source: §3 · p. 3
3.1. Neural network configurations
- Feature sequences are normalized, split into non-overlapping 25-frame sequences, and thresholded at 0.5 during testing.Training targets are binary class-presence vectors for each frame; test features use scaling parameters estimated on training data.Source: §3.1 · p. 3
- The system uses two hidden layers of 32 LSTM units, with one output neuron per class and binary cross-entropy training.Training uses back propagation through time, Adam, block mixing, and early stopping based on validation ER.Source: §3.1 · p. 3
4. EVALUATION AND RESULTS
The evaluation uses four-fold development data from two TUT sound-event contexts and compares segment-based ER and F-score for mono and feature-combination systems. The full spatial and harmonic feature evaluation generally matches or exceeds the mono baseline in average F-score, though the small dataset limits firm conclusions about binaural superiority.
4.1. Dataset
- The development subset contains home recordings with 11 classes and residential-area recordings with 7 classes, evaluated using four folds per context.The contexts contain 10 and 12 recordings, respectively; each fold uses 20% of training recordings for validation.Source: §4.1 · p. 3
4.2. Metrics
- ER and F-scores are computed on one-second segments, with fold results combined to reduce bias from data imbalance.The evaluation follows established SED metrics.Source: §4.2 · p. 3
4.3. Results
- With two exceptions, full spatial and harmonic feature combinations matched or exceeded the baseline in average F-score, with marginally similar average ER.The exceptions were mel2; pitch2 and mel2; tdoa3; pitch32; the authors caution that the roughly 60-minute dataset does not establish binaural superiority.Source: Table 2 · p. 4, §4.3 · p. 4
- The mono log mel-band RNN-LSTM baseline had better F-score than the GMM baseline across contexts, with comparable ER.The GMM baseline uses mono MFCC features and reports context-average ER 0.91 and F-score 23.%.Source: §4.3 · pp. 3–4
- Among the tested hybrid combinations, all except mel1; tdoa improved F-score over the baseline across contexts.Table 2 reports feature-combination results by home and residential-area context.Source: §4.3 · p. 4, Table 2 · p. 4
- Binaural mel2 and mel2; tdoa; pitch2 were submitted to DCASE 2016 and evaluated among the top-performing systems.Table 3 compares segment-based ER and F-score for development and evaluation datasets.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
The paper proposes spatial and harmonic features with RNN-LSTM for multi-label sound event detection. Its evaluation on a limited dataset found the proposed multichannel features performed substantially better than the mono-channel baseline.
5. CONCLUSION
- The proposed spatial and harmonic features are used with RNN-LSTM networks for multi-label sound event detection.The evaluation covered 60 minutes of data across four cross-validation folds and two contexts.Source: §5 · p. 4
- Future work will focus on novel data augmentation, including the unexplored challenge of augmenting spatial features.The authors also identify bidirectional RNNs and coupling RNNs with convolutional neural networks as directions for further study.Source: §5 · p. 4