Source-linked AI summary
Recurrent Neural Networks for Polyphonic Sound Event Detection in Real Life Recordings
Giambattista Parascandolo, Heikki Huttunen, Tuomas Virtanen
TL;DR
Polyphonic sound event detection must identify categories and boundaries for overlapping events in real-life recordings. The paper uses a multilabel BLSTM RNN to map acoustic mixture features to class activity indicators, achieving higher performance than the previous state-of-the-art FNN, with further gains from data augmentation. The approach also uses about half as many parameters as that FNN.
Problem
Polyphonic sound event detection must recognize each overlapping sound event’s category and beginning and ending in mixtures with unknown, potentially large polyphony.
Method
A multilabel BLSTM RNN maps acoustic features of mixture recordings to independent binary activity indicators for each event class.
Results
15.1% relative improvement in average framewise F1 and 6.8% in 1 second block F1 over the previous state-of-the-art FNN, with data augmentation producing the best results.
Takeaways & Limitations
The BLSTM approach outperforms the previous state-of-the-art FNN on the same large real-life-recording database while using half as many parameters.
Takeaways & Limitations
Future work is needed to develop novel data augmentation techniques and study attention mechanisms and convolutionally extended RNNs.
Abstract
from arXiv · showhide
In this paper we present an approach to polyphonic sound event detection in real life recordings based on bi-directional long short term memory (BLSTM) recurrent neural networks (RNNs). A single multilabel BLSTM RNN is trained to map acoustic features of a mixture signal consisting of sounds from multiple classes, to binary activity indicators of each event class. Our method is tested on a large database of real-life recordings, with 61 classes (e.g. music, car, speech) from 10 different everyday contexts. The proposed method outperforms previous approaches by a large margin, and the results are further improved using data augmentation techniques. Overall, our system reports an average F1-score of 65.5% on 1 second blocks and 64.7% on single frames, a relative improvement over previous state-of-the-art approach of 6.8% and 15.1% respectively.
1. INTRODUCTION
Polyphonic sound event detection targets overlapping sound sources in real-life recordings, where identifying each event’s category and boundaries is harder than monophonic detection. The paper proposes a multilabel BLSTM RNN that models sequential audio context for this task.
- The problem: Polyphonic detection identifies each overlapping sound event’s category, beginning, and ending when multiple sources emit simultaneously.The number of simultaneously active sources is unknown and potentially large.
- The problem: Overlapping sounds make mixture features differ from features extracted from isolated sounds, increasing the task’s difficulty.
- Previous approaches: Earlier approaches used MFCC features with GMM-HMM models or template matching supported by sound-source separation techniques such as NMF.
- Previous approaches: Feedforward neural networks achieved the best previous results by applying multilabel time-windowed MLPs to spectral mixture features and smoothing outputs temporally.
- Proposed approach: The proposed multilabel BLSTM RNN directly models sequential audio information and can avoid tailored postprocessing or smoothing steps.
2. RECURRENT NEURAL NETWORKS
Recurrent neural networks use feedback connections to carry information across time, while bidirectional and LSTM designs extend context and preserve longer-term information. BLSTM replaces bidirectional recurrent neurons with LSTM units.
- Recurrent neural networks: RNNs compute hidden activations and outputs sequentially, with feedback connections allowing previous information to circulate through hidden-layer memory.
- Recurrent neural networks: Bidirectional RNNs split each hidden layer into forward and backward components to provide symmetrical past and future context.
- Long short-term memory: Standard RNNs can be difficult to train because vanishing gradients make the influence of past inputs decay exponentially over time.
- Long short-term memory: LSTM addresses this problem with memory blocks containing a self-connected cell and input, forget, and output gates.
- Long short-term memory: The LSTM cell state and gates support long-term storage through multiplicative operations, with gate activations defined by logistic functions.
- Long short-term memory: A BLSTM network is formed by replacing the simple recurrent neurons in a bidirectional RNN with LSTM units.
3. METHOD
The system extracts normalized log-mel spectral features from raw audio and uses a multilabel BLSTM to produce independent event-activity indicators. Training includes regularization and feature-domain augmentation, while thresholding converts continuous outputs into binary detections.
- System pipeline: The proposed system maps spectral features from raw audio to binary activity indicators for each sound-event class using a BLSTM RNN.
- Feature extraction: Audio is divided into 50 millisecond frames with 50% overlap, and each frame is represented by log magnitudes in 40 mel bands.
- Feature extraction: Sequences are split into non-overlapping lengths of 10, 25, and 100 frames, corresponding to 0.25, 0.62, and 2.5 seconds.
- Targets and outputs: Each frame has a binary target component for every class, indicating whether that sound event is present.
- Targets and outputs: The multilabel network uses one logistic output neuron per class, and its outputs need not sum to one because multiple classes may be active simultaneously.
- Targets and outputs: Continuous class outputs are thresholded into binary indicators, with no post-processing because the RNN outputs are already smooth.
- Data augmentation: Feature-domain augmentation uses time stretching, sub-frame time shifting, and blocks mixing to reduce overfitting.
- Data augmentation: The experiments expanded the dataset by approximately 16 times using the combined augmentation techniques.
4. EVALUATION
The proposed BLSTM system was evaluated through five-fold cross-validation on real-life recordings spanning ten contexts, using framewise and one-second-block F1-scores. It outperformed the FNN baseline, with further gains from data augmentation, and showed similar F1-scores across polyphony levels.
- Experimental setup: The evaluation used 103 recordings totaling 1133 minutes from ten real-life contexts, split into approximately 60%, 20%, and 20% training, validation, and test data in five folds.Results were averaged across the five cross-validation folds using the same partitions as earlier experiments.
- Experimental setup: The evaluated networks used 40-dimensional inputs and four hidden layers of 200 LSTM units, with bidirectional layers reading sequences forwards and backwards.Experiments compared BLSTM with and without augmentation against a standard unidirectional LSTM and the prior FNN.
- Evaluation metrics: Performance was measured using average framewise F1-score and F1-score over non-overlapping one-second blocks, averaged across contexts.A class was marked active for a one-second block when it was active in at least one frame.
- Results: 15.1% and 6.8% were the BLSTM with data augmentation’s relative improvements over the FNN for average framewise F1 and one-second block F1, respectively.Without augmentation, BLSTM improved over the same-data FNN by 13.5% for average framewise F1 and 4.3% for one-second block F1.
- Results: The proposed RNN outperformed the FNN in most contexts even without data augmentation, while F1-scores remained approximately constant across polyphony levels.This indicates robustness when several events are combined.
5. CONCLUSIONS
The paper concludes that multilabel BLSTM RNNs improve polyphonic sound event detection on a large real-life recording database. Data augmentation further improves performance, while future work targets new augmentation methods, attention, and convolutional extensions.
- Conclusions: The proposed multilabel BLSTM RNN outperformed the previous state-of-the-art FNN on the same large database of real-life recordings.The RNN used half as many parameters as the FNN.
- Conclusions: 15.1% and 6.8% were the average improvements over the whole dataset for framewise F1 and one-second block F1, respectively.Data augmentation was reported to reduce overfitting and further improve performance.
- Future work: Future work will investigate novel data augmentation techniques, attention mechanisms, and coupling recurrent networks with convolutional neural networks.These directions are stated as extensions for both the augmentation strategy and the model.