PaperlayerView exact revision on arXiv

Source-linked AI summary

Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimization

Qiuqiang Kong, Yong Xu, Wenwu Wang, Mark D. Plumbley

arXiv:1912.04761v2cs.SDeess.AS

TL;DR

Weakly labelled SED data provide clip tags without event onset and offset times, and prior work had not compared segment-wise and clip-wise training. The paper compares those training approaches, proposes a CNN-Transformer and automatic threshold optimization, and reports improved AT F1 and error rate with threshold optimization.

  • Problem

    Weakly labelled SED data provide clip-level event tags without onset and offset times, and research had not compared segment-wise and clip-wise training.

    Source: §I · p. 1

  • Method

    The paper compares segment-wise and clip-wise training, proposes a CNN-Transformer for AT and SED, and introduces automatic threshold optimization.

    Source: §I · p. 2

  • Results

    AT F1 improves from 0.629 to 0.646, and error rate falls from 0.78 to 0.68 with automatic threshold optimization.

    Source: §VII · pp. 8–9

  • Takeaways & Limitations

    The CNN-Transformer performs similarly to the CRNN system and has the advantage of parallel computation.

    Source: §VII · p. 9

Abstract

from arXiv · show

Sound event detection (SED) is a task to detect sound events in an audio recording. One challenge of the SED task is that many datasets such as the Detection and Classification of Acoustic Scenes and Events (DCASE) datasets are weakly labelled. That is, there are only audio tags for each audio clip without the onset and offset times of sound events. \qk{We compare segment-wise and clip-wise training for SED that is lacking in previous works. We propose a convolutional neural network transformer (CNN-Transfomer) for audio tagging and SED, and show that CNN-Transformer performs similarly to a convolutional recurrent neural network (CRNN)}. Another challenge of SED is that thresholds are required for detecting sound events. Previous works set thresholds empirically, and are not an optimal approaches. To solve this problem, we propose an automatic threshold optimization method. The first stage is to optimize the system with respect to metrics that do not depend on thresholds, such as mean average precision (mAP). The second stage is to optimize the thresholds with respect to metrics that depends on those thresholds. Our proposed automatic threshold optimization system achieves a state-of-the-art audio tagging F1 of 0.646, outperforming that without threshold optimization of 0.629, and a sound event detection F1 of 0.584, outperforming that without threshold optimization of 0.564.

I. INTRODUCTION

The paper addresses weakly labelled SED, where training clips lack event onset and offset times, and the limited research comparing segment-wise with clip-wise training. It also targets empirical threshold selection and the difficulty CNNs have capturing long-term temporal dependencies.

I. INTRODUCTION

  • The paper targets weakly labelled SED, where training data provide event presence or absence but not onset and offset times.It focuses on the DCASE 2017 Task 4 smart-cars dataset, whose training clips are weakly labelled.Source: §I · p. 1
  • Segment-wise and clip-wise training had not been compared for SED, while frame-level labels are time-consuming to obtain.Segment-wise methods assign clip tags to segments; clip-wise methods train on entire clips.Source: §I · p. 1

B. Convolutional recurrent neural network (CRNNs)

  • CNN receptive fields can limit their ability to capture long-term dependencies, while recurrent networks retain sequential history in hidden states.The paper describes LSTM and GRU variants; its AT and SED systems use bidirectional GRU.Source: §I · p. 1, B. Convolutional recurrent neural… · p. 2

I. INTRODUCTION

  • Thresholds for AT and SED post-processing are usually selected empirically, and may not be optimal.The paper proposes automatic threshold optimization to address this selection problem.Source: §I · p. 2

A. Conventional CNNs

  • A conventional CNN applies convolution and pooling to learn local time-frequency patterns, then predicts event presence over time for AT and SED.Time and frequency pooling are applied, and the temporal predictions can be aggregated into clip-level presence probabilities.Source: A. Conventional CNNs · p. 2

III. CNN-TRANSFORMER

The CNN-Transformer combines CNN-derived audio features with transformer self-attention to model temporal relationships while allowing parallel computation. Its output supports timewise sound-class predictions and clip-level audio tagging.

III. CNN-TRANSFORMER

  • Transformers model relationships across sequence time steps with self-attention and can parallelize computation, unlike recurrent networks that compute hidden states sequentially.The paper motivates this architecture for temporal dependencies in audio clips.Source: §III · p. 2, A. Transformer · p. 2

A. Transformer

  • A transformer encoder uses query, key, and value representations to weight information across time steps.The query-key inner product represents correlations between time steps; softmax converts these into attention weights on the values.Source: A. Transformer · p. 3

B. CNN-Transformer

  • The CNN-Transformer converts log mel spectrograms into CNN feature embeddings, transforms them, and predicts sound-class probabilities over time.An aggregation such as averaging those probabilities over time yields audio-tagging results.Source: B. CNN-Transformer · p. 3

IV. SEGMENT-WISE V.S. CLIP-WISE SED

The section compares segment-wise and clip-wise training for AT and SED with weakly labelled data. Segment-wise training assigns clip tags to each segment, whereas clip-wise training learns clip tags without explicitly assigning segment labels.

A. Segment-wise training

  • Segment-wise training splits a clip into segments, gives each segment the clip tags, and trains a classifier to predict those tags.At inference, segment predictions provide SED outputs, and aggregating them can provide clip-level AT predictions.Source: A. Segment-wise training · p. 3

B. Clip-wise training

  • Because weak labels omit event timing, a segment may not contain an event despite inheriting its clip tag, making segment-wise labels potentially inaccurate.The approach converts SED to audio tagging on segments.Source: B. Clip-wise training · p. 3
  • Clip-wise training avoids explicitly assigning tags to segments and learns clip tags from latent segment predictions aggregated into a clip prediction.The clip-level output can be trained end-to-end against the weak clip label.Source: B. Clip-wise training · pp. 3–4

V. AUTOMATIC THRESHOLD OPTIMIZATION

The method separates system training from threshold selection, then optimizes thresholds for threshold-dependent metrics. It uses numerical gradients because thresholding and the evaluation metrics are non-differentiable.

V. AUTOMATIC THRESHOLD OPTIMIZATION

  • For audio tagging, a class is present when its output exceeds its corresponding threshold.The thresholds are denoted ΘAT = {µ1, ..., µK}.Source: §V · p. 4
  • For sound event detection, clip-level thresholds first determine which classes are present, then segment-level thresholds locate events in time.A lower segment threshold is also applied to neighbouring segments of active ones to reduce missed detections.Source: §V · p. 4
  • Thresholds are optimized in a second stage for a trained system, using a chosen metric such as F1 score or error rate (ER).The first stage evaluates systems with threshold-independent metrics such as mean average precision (mAP).Source: §V · p. 4
  • Numerical gradients estimate how the objective changes when an individual threshold parameter is perturbed, avoiding analytical differentiation of non-differentiable thresholding and metrics.The parameter is perturbed by a small constant, with the other threshold values unchanged.Source: §V · p. 4
  • The resulting threshold gradients are optimized iteratively with Adam to minimize the selected loss.The loss can be ER or negative F1; minimizing negative F1 is equivalent to maximizing F1.Source: §V · pp. 4–5

A. Experimental setup

The experiments use the DCASE 2017 Task 4 weakly supervised sound event detection dataset for smart cars, drawn from a subset of AudioSet.

A. Experimental setup

  • The DCASE 2017 Task 4 dataset contains over 140 hours of weakly labelled training audio and 17 sound events in warning and vehicle categories.The clips come from a subset of AudioSet and are extracted from YouTube videos.Source: A. Experimental setup · p. 5
  • Training uses 51,172 weakly labelled clips, while validation and evaluation use 488 and 1,103 clips, respectively.Both validation and evaluation subsets include weak and strong labels for evaluation.Source: A. Experimental setup · p. 5

B. Feature

Audio is converted to log mel spectrograms, using 64 mel filter banks over a 50 Hz–14 kHz frequency range.

B. Feature

  • Audio is converted to mono and resampled to 32 kHz before log mel spectrogram extraction.A 1,024-sample Hanning window and 320-sample hop produce 100 frames per second; 64 mel filters span 50 Hz to 14 kHz.Source: B. Feature · p. 5

C. Model

The model section describes a CNN for segment-wise training, while evaluation uses threshold-independent average precision and threshold-dependent F1 and error rate metrics.

C. Model

  • The segment-wise training system uses a 9-layer CNN with four convolutional blocks, each containing two 3 × 3 convolutional layers.The blocks have 64, 128, 256, and 512 feature maps, followed by frequency averaging and time-distributed sigmoid predictions.Source: C. Model · pp. 5–6, Table II · p. 6

D. Evaluation metrics

  • Mean average precision (mAP) averages per-class average precision, which is calculated across thresholds and does not depend on a selected threshold.A random guess has mAP 0.06 on DCASE 2017 Task 4.Source: D. Evaluation metrics · p. 6
  • Audio tagging systems are ranked by macro-averaged F1, while sound event detection systems are ranked by micro-averaged F1 and error rate (ER) on 1-second segments.Lower ER is better; it counts insertions, deletions, and substitutions relative to ground-truth events.Source: D. Evaluation metrics · p. 6

E. Segment-wise AT and SED

Segment-wise training assigns clip-level tags to each segment even though event timing is unknown, and performance varies with segment duration across audio tagging and SED.

E. Segment-wise AT and SED

F. Clip-wise AT and SED

Clip-wise systems perform differently across audio tagging and sound event detection, and automatic threshold optimization improves the reported AT and SED results.

F. Clip-wise AT and SED

  • CNN-Transformer and CNN-biGRU systems reach AT mAP of 0.65; CNN-Transformer leads SED mAP at 0.45 versus 0.44 for CNN-biGRU.CNN-biGRU achieves SED ER of 0.66, compared with 0.69 to 0.86 for other systems.Source: F. Clip-wise AT and SED · p. 7, Figure 2 · p. 8
  • SED mAP favors segment-wise training, whereas SED ER favors clip-wise training; the paper notes these metrics use frame-wise and one-second-segment evaluation, respectively.The clip-wise ER comparison applies the same thresholds as the segment-wise systems.Source: F. Clip-wise AT and SED · p. 7
  • Class-wise results vary: screaming has the highest AT mAP at 0.94, car passing by the lowest at 0.20, and civil defense siren the highest SED mAP at 0.80.Civil defense siren has SED ER of 0.26, while bicycle has ER over 1.Source: F. Clip-wise AT and SED · p. 7, Figure 3 · p. 8

G. Automatic thresholds optimization

  • Validation and evaluation precision-recall curves have similar but nonoverlapping trends, and different sound classes have different thresholds for optimal F1.Some classes retain high precision across thresholds, while precision drops rapidly with recall for others.Source: G. Automatic thresholds optimization · pp. 7–8
  • CNN-Transformer AT F1 rises from 0.557 to 0.599 on validation and from 0.629 to 0.646 on evaluation after threshold optimization.CNN-biGRU SED ER falls from 0.80 to 0.65 on validation and from 0.78 to 0.68 on evaluation.Source: G. Automatic thresholds optimization · p. 8
  • 0.646 AT F1 and 0.584 SED F1 on evaluation are reported for the proposed systems with automatic threshold optimization.The CNN-biGRU-Att SED system also achieves an ER of 0.68, comparable with the ensemble-based CNN system.Source: G. Automatic thresholds optimization · p. 8

VII. CONCLUSION

The conclusion reports improved AT and SED results with automatic threshold optimization and similar CNN-Transformer and CRNN performance. It also identifies parallel computation as a CNN-Transformer advantage.

VII. CONCLUSION

  • 0.629 to 0.646 AT F1 and 0.78 to 0.68 SED ER are the reported changes with automatic threshold optimization.The conclusion says the improvements in AT and SED are mainly from threshold optimization.Source: §VII · pp. 8–9
  • The clip-wise systems achieve AT mAP of 0.650 and SED ER of 0.68.Source: §VII · p. 8
  • CNN-Transformer performs similarly to the CRNN system, with the advantage of being computed in parallel.Source: §VII · p. 9
  • The stated future direction is extending sound event detection systems to large-scale training data such as AudioSet.Source: §VII · p. 10
Loading 1912.04761v2…