Source-linked AI summary
Sound Event Detection Using Spatial Features and Convolutional Recurrent Neural Network
Sharath Adavanne, Pasi Pertilä, Tuomas Virtanen
TL;DR
Sound event detection in multichannel audio needs representations that exploit spatial information and can distinguish overlapping events, while prior binaural evidence was limited by a small dataset. The paper uses low-level binaural features in an extended CBRNN, learning separate feature volumes before recurrent processing. Binaural features improve absolute F-score over monaural features by 6.1% on TUT-SED 2016 and 2.7% on the larger TUT-SED 2009 dataset.
Problem
Prior binaural spatial-feature evidence for sound event detection was limited because it used a dataset of only around one hour.
Method
The paper extends CRNNs with separate CNN processing for multiple low-level binaural feature volumes, followed by bi-directional LSTMs.
Results
Binaural features improve absolute F-score over monaural features by 6.1% on TUT-SED 2016 and 2.7% on TUT-SED 2009.
Takeaways & Limitations
The network learns sound events better when multichannel features are presented as separate volume layers rather than concatenated channel vectors.
Takeaways & Limitations
The work trains a single context-independent system to learn sound-event classes across contexts.
Abstract
from arXiv · showhide
This paper proposes to use low-level spatial features extracted from multichannel audio for sound event detection. We extend the convolutional recurrent neural network to handle more than one type of these multichannel features by learning from each of them separately in the initial stages. We show that instead of concatenating the features of each channel into a single feature vector the network learns sound events in multichannel audio better when they are presented as separate layers of a volume. Using the proposed spatial features over monaural features on the same network gives an absolute F-score improvement of 6.1% on the publicly available TUT-SED 2016 dataset and 2.7% on the TUT-SED 2009 dataset that is fifteen times larger.
1. INTRODUCTION
The paper targets sound event detection in multichannel audio by exploiting spatial features and extending CRNNs to learn multiple feature types. It evaluates binaural features against monaural features and channel concatenation.
- Motivation: Sound event detection identifies event onsets and offsets in acoustic scenes and labels the detected events.Applications include biodiversity assessment, surveillance, and content-based retrieval.
- Motivation: Multichannel features such as TDOA and mel-band energies may help distinguish overlapping sound events, beyond combining classification likelihoods across channels.Earlier multichannel work did not exploit the full potential of multichannel features.
- Research gap: Prior binaural spatial features improved over monaural features, but evidence was limited because the dataset contained only around one hour of audio.The paper motivates evaluation on larger datasets to test whether the observed advantage is reliable.
- Contributions: The paper compares low-level features such as GCC-PHAT with high-level features such as TDOA and arranges channel features as separate layers rather than concatenating them.It extends CRNNs to process multiple feature types and uses a bi-directional LSTM.
2. BINAURAL FEATURES FOR POLYPHONIC SED
The paper extracts binaural features for polyphonic SED, where overlapping events must be recognized together. These features encode channel intensity, spatial delay, dominant frequency, and pitch-related information.
- Task and feature extraction: Polyphonic SED recognizes overlapping sound events together with isolated events, using feature extraction followed by a neural network that predicts activity for each class.Features are extracted at a constant hop length of 20 ms.
- Log mel-band energies: Binaural channel intensity and frequency differences can help distinguish overlapping events, motivated by human interaural intensity difference processing.The combination of intensity differences across frequency bands provides spatial information.
- Log mel-band energies: Log mel-band energies use 40 mel-bands in 40 ms Hamming windows from both binaural channels, allowing the network to learn channel differences and other features.The network receives channel-wise energies instead of only their direct difference.
- TDOA and GCC-PHAT: TDOA features are estimated with GCC-PHAT across five mel-bands because spatial delay and spectral differences can help separate overlapping events.Multi-resolution windows of 120 ms, 240 ms, and 480 ms produce 15 TDOA values per frame.
- TDOA and GCC-PHAT: GCC-PHAT is also tested as a low-level representation against TDOA, which is selected from GCC-PHAT peaks.The formulation uses FFT coefficients from the two binaural channels and constrains delays to the microphone spacing range.
- Dominant frequencies and ACR: Dominant frequencies and auto-correlation features provide complementary spectral and pitch-related cues for distinguishing overlapping sound events.ACR uses 40 ms binaural-channel windows and 400 correlation values spanning 107.5 Hz to 4410 Hz.
3. CONVOLUTIONAL RECURRENT NEURAL NETWORK
The proposed CBRNN represents each channel and feature type as structured volumes, learns them with separate CNNs, and models their temporal interactions with bi-directional LSTMs.
- Architecture: The architecture extends CRNN by combining convolutional, recurrent, and fully connected layers for multichannel audio features.The recurrent component is extended with bi-directional LSTMs.
- Multichannel representation: Channel features are layered into volumes so two-dimensional CNNs can learn channel-wise filters and inter-channel information.For M frames of length L from two channels, the input becomes an M × L × 2 volume instead of an M × 2L vector.
- Multichannel representation: Separate feature volumes are formed for mel, dominant frequency, ACR, TDOA, and GCC-PHAT with feature-specific dimensions.Examples include T × 40 × 2 for mel, T × 400 × 2 for ACR, and T × 60 × 3 for GCC-PHAT.
- Network processing: Separate CNNs learn local shift-invariant features from each volume before the feature maps are concatenated and passed to two bi-directional LSTMs.Pooling reduces feature-map dimensions while preserving the time axis for onset and offset detection.
- Output and training: A time-distributed fully connected layer with sigmoid activation predicts multiple simultaneously active sound-event classes.Batch normalization, dropout, Adam optimization, binary cross-entropy, and early stopping are used during training.
4. EVALUATION AND RESULTS
The system is evaluated on two real-life binaural datasets using segment-based error rate and F-score, with binaural features, low-level spatial representations, and multi-layered inputs compared against monaural baselines. Binaural features improve performance across datasets, while context influences which feature combinations work best.
- Datasets: The evaluation uses TUT-SED 2009 and TUT-SED 2016, with manually annotated binaural recordings spanning different contexts and event classes.TUT-SED 2009 contains 1133 minutes across 10 contexts and 61 event classes, while TUT-SED 2016 contains 78 minutes across home and residential-area contexts.
- Evaluation metrics: Performance is measured with one-second segment-based error rate and F-score, averaged across contexts.The metrics use true positives, false positives, false negatives, insertions, deletions, and substitutions.
- Input representation: Multi-layered binaural feature inputs perform relatively better than simple feature concatenation, with similar improvements observed for TDOA, dominant frequency, GCC-PHAT, and ACR.The comparison is reported for mel features on TUT-SED 2009, while the other feature comparisons are not tabulated.
- Overall results: 2.7% and 6.1% absolute F-score improvements are obtained over monaural features on TUT-SED 2009 and TUT-SED 2016, respectively.Binaural features improve both error rate and F-score across datasets; the proposed CBRNN uses the same network for the comparison.
- Feature comparison: Low-level GCC-PHAT and ACR features achieve performance comparable to the corresponding high-level TDOA and dominant-frequency features.The paper reports that this lets the network learn equivalent information without tuning the number of TDOA or dominant-frequency values.
- Context effects: Feature combinations vary by context: dominant frequency with mel helps indoor and intense environments, whereas TDOA or GCC-PHAT with mel helps outdoor contexts.This context pattern is consistent with dominant frequency and mel producing the best results on the larger TUT-SED 2009 dataset.
5. CONCLUSION
The paper concludes that an extended CRNN can learn sound events from multiple low-level binaural feature classes arranged as separate input layers. Across two datasets, binaural spatial features outperform monaural features, supporting the use of spatial information for SED.
- The extended CRNN handles multiple feature classes and processes feature maps with bidirectional LSTMs.
- Separate layers for multichannel features enable better sound-event learning than monaural features alone.
- Low-level features can provide information equivalent to higher-level representations learned by the network.
- The evaluation uses a larger dataset to test spatial-feature learning and a public dataset for benchmarking.
- Table 2 reports context-wise F-scores for the TUT-SED 2009 dataset.