PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection

Emre Çakır, Giambattista Parascandolo, Toni Heittola, Heikki Huttunen, Tuomas Virtanen

arXiv:1702.06286v1cs.LGcs.SD

TL;DR

Polyphonic sound event detection must handle overlapping events and substantial variation in real-life recordings. The paper combines convolutional feature extraction with recurrent temporal modeling in a CRNN and compares it with CNN, RNN, and established baselines. The CRNN shows clear performance improvements across four datasets, but its performance depends strongly on the amount of annotated data.

  • Problem

    Polyphonic SED must detect overlapping sound events in unstructured real-life recordings with noise, multiple sources, and within-class acoustic variation.

  • Method

    The paper combines convolutional layers for feature extraction and frequency invariance with recurrent layers for temporal context in a jointly trained CRNN.

  • Results

    The CRNN consistently outperforms CNNs, RNNs, and baseline methods on the main metric across three datasets, with evaluation results showing improvement across four datasets overall.

  • Takeaways & Limitations

    Combining CNN and RNN capabilities is a supported effective approach for polyphonic SED across the evaluated everyday-sound datasets.

  • Takeaways & Limitations

    CRNN performance depends strongly on annotated data volume, with lower performance on the smaller TUT-SED 2016 dataset than on larger datasets.

Abstract

from arXiv · show

Sound events often occur in unstructured environments where they exhibit wide variations in their frequency content and temporal structure. Convolutional neural networks (CNN) are able to extract higher level features that are invariant to local spectral and temporal variations. Recurrent neural networks (RNNs) are powerful in learning the longer term temporal context in the audio signals. CNNs and RNNs as classifiers have recently shown improved performances over established methods in various sound recognition tasks. We combine these two approaches in a Convolutional Recurrent Neural Network (CRNN) and apply it on a polyphonic sound event detection task. We compare the performance of the proposed CRNN method with CNN, RNN, and other established methods, and observe a considerable improvement for four different datasets consisting of everyday sound events.

I. INTRODUCTION

Polyphonic sound event detection targets overlapping events in varied real-life recordings, where CNNs and RNNs address complementary weaknesses. The proposed CRNN combines their strengths and outperforms the compared baselines across real-life datasets.

  • Problem: Real-life SED is challenged by environmental noise, overlapping sources, and substantial variation among events within the same class.Different sound sources, such as dog breeds, can produce acoustically different instances of one class.
  • Problem: Polyphonic SED detects multiple overlapping sound events and their onset and offset times in audio recordings.This addresses scenarios such as busy streets, where several event classes may occur simultaneously.
  • Prior limitations: Feedforward networks improve over established methods but lack time and frequency invariance and restrict temporal context to short windows.Their fixed input-to-hidden connections limit modeling of local event variations and longer dependencies.
  • Prior limitations: CNNs provide time-frequency invariance but lack longer temporal context, whereas RNNs integrate earlier windows but do not easily capture frequency-domain invariance.These complementary limitations motivate combining the two architectures.
  • Contribution: The proposed multi-label CRNN performs scene-independent polyphonic SED by integrating CNN and RNN capabilities and is evaluated against FNN, CNN, RNN, and GMM baselines.The evaluation covers three datasets of real-life recordings, and the method is reported to outperform previous approaches.

II. METHOD

Polyphonic SED is formulated as multi-label classification over consecutive feature frames. The system estimates class activity probabilities frame by frame and combines them over time to recover event boundaries.

  • Problem formulation: Polyphonic SED temporally locates and labels sound event classes present in a polyphonic audio signal.The task uses frame-level sound features and class activity targets derived from onset/offset annotations.
  • Problem formulation: The classifier estimates p(y_t(k) | x_t, θ) for each event class and frame, then thresholds probabilities to obtain binary activity predictions.The feature vector x_t contains frame-level sound features, while θ denotes classifier parameters.
  • Training: Supervised training uses onset/offset annotations to set each class target to 1 when present in a frame and 0 otherwise.The trained model predicts event activity when annotations are unavailable in real-life use.
  • Problem formulation: Multiple non-zero target elements are allowed in one frame, making polyphonic SED a multi-label classification problem.Combining predictions over consecutive frames determines onset and offset times for each class.
  • Temporal context: Temporal context is represented by a sequence of feature frames and corresponding target outputs over a context window of T frames.This formulation preserves sequential information for methods that model temporal context.

B. Proposed Method

The proposed CRNN transforms a time-frequency context window through convolutional feature extraction, recurrent temporal integration, and feedforward probability estimation. Frequency pooling preserves time alignment while increasing frequency invariance.

  • 3) Output layer: The recurrent layers integrate extracted features over time, and a sigmoid feedforward layer estimates event activity probabilities for each frame and class.The full convolutional, recurrent, and feedforward stack is trained jointly through backpropagation.
  • 1) Convolutional layers: A context window of F log mel bands over T frames enters multiple convolutional layers with non-overlapping frequency-axis pooling.Two-dimensional filters extract features while pooling reduces frequency dimensionality and provides frequency invariance.
  • 1) Convolutional layers: Zero-padding and frequency-only pooling preserve the time dimension so target frames remain aligned with hidden activations.The convolutional layers use ReLU activations before max pooling.
  • 2) Recurrent layers: After convolution, the feature maps are stacked over frequency and fed as a sequence of frames into multiple stacked recurrent layers.The recurrent component processes the CNN output tensor after its remaining frequency bands are combined.

3) Feedforward layer:

The feedforward output layer converts recurrent activations into per-class event probabilities, which are thresholded into binary activity predictions. The architecture also uses regularization, batch normalization, and dataset-specific temporal pooling.

  • 3) Feedforward layer:: A sigmoid feedforward layer reads the final recurrent activations to estimate event activity probabilities for each sound-event class.The same weight set is applied to features extracted from each frame.
  • 4) Binarization:: Predicted event activity is obtained by thresholding each class probability at a constant C between 0 and 1.The threshold produces binary predictions for the event classes.
  • Regularization:: Dropout regularization is used during training to reduce overfitting by randomly removing hidden units at each iteration.The approach is described as reducing unit co-adaptation and approximating model averaging.
  • Regularization:: Batch normalization follows every convolutional or fully connected layer to speed training by normalizing activations using mini-batch statistics.Layer outputs are normalized toward zero mean and unit variance.
  • Comparison to other CRNN architectures:: For CHiME-Home, a temporal max-pooling layer is added before the output layer to accommodate chunk-level rather than frame-level annotations.Pooling combines predictions across the context window.
  • Comparison to other CRNN architectures:: CNNs and RNNs are evaluated as CRNN special cases by removing the recurrent or convolutional layers, respectively.This directly tests the benefit of combining both types of layers.

III. EVALUATION

The evaluation uses four datasets and scene-independent training, including synthetic mixtures and recordings from real-life environments. CHiME-Home additionally requires temporal pooling because its annotations are provided per audio chunk.

  • III. EVALUATION: Four datasets are evaluated, including one synthetic mixture dataset and three datasets recorded in real-life environments.The experiments train a single model per dataset regardless of the number of acoustic scenes.
  • Datasets: TUT-SED Synthetic 2016 contains mixtures from 16 sound-event classes, with 100 mixtures split into 60% training, 20% testing, and 20% validation.The dataset contains 566 minutes of audio and uses different event instances across partitions.
  • Datasets: TUT-SED 2009 contains 1,133 minutes of binaural recordings from 10 real-life scenes, with 61 defined sound-event classes.Recordings are 10–30 minutes long and include environments such as streets, offices, restaurants, and stadiums.
  • Datasets: TUT-SED 2016 development contains 78 minutes of manually annotated recordings from residential-area and home scenes captured at different locations.Location variation produces variability in the active sound-event classes between recordings.
  • Datasets: CHiME-Home provides 4-second home-environment chunks labeled with one or more of seven sound classes.The refined setup contains 1,946 development chunks and 846 evaluation chunks.
  • Dataset adaptation: Because CHiME-Home lacks frame-level annotations, the architecture adds temporal max-pooling before the output layer to pool predictions over time.The other datasets use frame-level event annotations.

B. Evaluation Metrics

Evaluation uses segment-based metrics computed from event-detection statistics, with frame and one-second segment lengths. The paper also reports a legacy F1 calculation for comparison on TUT-SED 2009.

  • B. Evaluation Metrics: The evaluation uses segment-based metrics at two resolutions: one frame of 40 ms and one-second segments.Metric subscripts identify the segment length, such as F1frm and F11sec.
  • F1 score: Segment-based F1frm is the primary metric, using true positives, false positives, and false negatives accumulated across evaluated segments.A detected event matching an annotated event is a true positive; missed and spurious events are false negatives and false positives.
  • F1 score: Micro-averaging gives each active instance per evaluated segment equal influence by accumulating intermediate statistics over test data and folds.For multiple scenes, metrics are calculated per scene and then averaged across scenes.
  • F1 score: Precision and recall are combined as their harmonic mean to produce the F1 score.The paper also references detailed visual explanations for segment-based F1 in the multilabel setting.
  • Error rate: Segment-based error rate uses substitutions, insertions, deletions, and annotated active classes as its intermediate statistics.These statistics are calculated per segment before computing the total error rate.
  • Comparison with prior work: The legacy TUT-SED 2009 F1 calculation averages segment scores before scene averaging, which can introduce measurement bias under class imbalance.The paper reports this legacy score alongside its own accumulated-statistics calculation for comprehensive comparison.

C. Baselines

The study compares CRNN performance with established GMM and deep FNN baselines. The FNN baseline uses a short spectrogram context and a multilayer multi-label classifier.

  • C. Baselines: GMM and FNN are used as comparison baselines, representing established generative modeling and straightforward deep learning approaches.The FNN provides a baseline for the more complex CNN, RNN, and CRNN architectures.
  • GMM: The GMM baseline trains separate positive and negative models for each sound-event class and makes decisions using likelihood ratios.Models are trained with expectation-maximization after k-means initialization and use diagonal covariance matrices.
  • FNN: The FNN baseline extracts 40 log-mel-band energies from each 40 ms frame and stacks five vectors to create a 100 ms context window.Its output estimates event activity probabilities for all classes using sigmoid activation.
  • FNN: The FNN contains two 1,600-unit maxout hidden layers and an output layer with K sigmoid units, totaling approximately 1.6 million parameters.Sliding-window post-processing from the original FNN work is omitted for fair architectural comparison.

D. Experiments set-up

The experiments standardize log-mel preprocessing and compare tuned CNN, RNN, and CRNN architectures under shared training and evaluation procedures.

  • Preprocessing: All neural networks use normalized log-mel band energies extracted from 40 ms STFT frames with 50% overlap and 40 mel bands.Training uses overlapping subsequences, while validation and test data use no overlap.
  • Network configurations: Architectures are selected through validation-based hyperparameter searches over layer counts, hidden units, feature maps, pooling arrangements, dropout, and sequence length.The search covers CNN, RNN, and CRNN configurations across the evaluated datasets.
  • Training setup: The networks use batch normalization, dropout rate 0.25, sigmoid outputs, 5x5 convolutional filters, GRU recurrent layers, binary cross-entropy, and Adam optimization.GRUs were chosen after preliminary experiments found similar results with LSTMs and fewer parameters for GRUs.
  • Comparative design: CNN, RNN, and CRNN comparisons use matched optimization procedures to support a fair architectural comparison.CNNs replace recurrent layers with feedforward layers, while RNNs remove convolutional layers and add feedforward layers before the output.
  • Training and evaluation: Event probabilities are thresholded at C = 0.5, and training uses early stopping when validation performance fails to improve for more than 100 epochs.The best-performing validation weights are restored after stopping.

IV. RESULTS

Across the reported datasets, CRNNs achieve the strongest main-metric results against CNN, RNN, GMM, and FNN comparisons, with additional experiments examining architecture and parameter effects.

  • Main results: CRNNs consistently outperform CNNs, RNNs, GMM, and FNN baselines on the main metric across all three reported datasets.Results are computed on test sets using the best configurations selected from validation data.
  • Main results: 6.6% and 13.6% are the absolute frame-based F1 improvements of CRNN over CNN and RNN, respectively, on TUT-SED Synthetic 2016.All four deep learning methods outperform the GMM baseline.
  • Additional analyses: The study also examines convolution filter shapes and parameter count versus accuracy in the synthetic dataset experiments.These analyses complement the primary CNN, RNN, and CRNN test comparisons.

1) Class-wise performance:

CRNN generally performs best across sound-event classes and shows robustness patterns linked to frequency variation and overlapping events, while some short events remain difficult.

  • Class-wise performance: CRNN outperforms CNN and RNN on almost all classes in the polyphonic TUT-SED Synthetic 2016 setting.The class-wise results account for events occurring alongside other classes rather than in isolation.
  • Class-wise performance: Cat meowing is the worst-performing class for all three networks, often confused with acoustically similar baby crying.Short, non-impulsive events are more likely to be masked by overlapping sounds.
  • Filter shapes: Small convolution kernels, including (5,5) and (3,3), perform best in the filter-shape experiments.The very low result for (1,5) highlights the importance of including multiple frequency bands in convolutional features.
  • Model size: For identical parameter counts, CRNN has a clear accuracy advantage over CNN and RNN in the reported comparison.The authors interpret this pattern as an architectural rather than model-size advantage.
  • Frequency shift invariance: CNN and CRNN are more robust than RNN to small pitch shifts, although their accuracy-decrease difference diminishes for negative shifts.RNN shows a significant absolute accuracy drop when test mixtures are shifted by ±2 quartertones.
  • Network outputs: In overlapping examples, CRNN detects gun shot alongside baby crying and tracks both crowd applause and masked baby crying more accurately than CNN or RNN.The sequence illustrates CRNN predictions closely matching the ground truth across changing polyphonic conditions.

B. TUT-SED 2009

On TUT-SED 2009, CRNN performance is compared with scene-dependent and scene-independent methods, with deep-learning approaches showing strong results. CRNN outperforms previous methods and other neural architectures in the reported comparisons.

  • The comparison includes earlier HMM- and NMF-based systems, alongside scene-dependent and scene-independent methods.
  • CRNN achieves significantly higher performance than previous methods on the dataset and improves over other neural-network approaches.
  • CRNN and RNN obtain the best framewise F1 results, while CRNN leads CNN and RNN on framewise and 1-second-block error rate.
  • FNN achieves better 1-second-block F1, but this advantage comes with a very large 1-second-block error rate.

E. Visualization of convolutional layers

The paper visualizes input patterns that strongly activate neurons in successive CRNN convolutional layers. Its conclusion describes convolutional feature extraction followed by recurrent processing, while noting data dependence as a limitation.

  • E. Visualization of convolutional layers: The visualization examines patterns that strongly activate neurons in different CRNN convolutional layers.A random input is optimized using the gradient of a selected neuron’s activation with respect to the input.
  • E. Visualization of convolutional layers: Figure 7 arranges two columns of crops with time on the horizontal axis and mel bands on the vertical axis.
  • V. CONCLUSIONS: CRNN first extracts higher-level features using convolution and frequency pooling, then feeds them to recurrent layers for event-activity probability estimation.
  • V. CONCLUSIONS: The authors identify dependence on large amounts of annotated data as a limitation, especially for the small TUT-SED 2016 dataset.
Loading 1702.06286v1…