PaperlayerView exact revision on arXiv

Source-linked AI summary

Sound Event Detection and Time-Frequency Segmentation from Weakly Labelled Data

Qiuqiang Kong, Yong Xu, Iwona Sobieraj, Wenwu Wang, Mark D. Plumbley

arXiv:1804.04715v3cs.SDeess.AS

TL;DR

Strongly labelled SED data require time-consuming onset and offset annotation, whereas weakly labelled data provide event presence alone. This paper learns time-frequency masks from weak labels and uses them for detection and separation; its experiments report that GWRP outperforms comparison systems in SED and time-frequency segmentation.

  • Problem

    Strongly labelled SED requires onset and offset annotations that take more time to collect than clip-level labels, while weak labels omit event timing.

    Source: §I · p. 1

  • Method

    The framework jointly trains a segmentation mapping to produce event time-frequency masks and a classification mapping to predict event presence from weak labels.

    Source: §III · p. 3, A. Training from weakly labelled data · p. 3

  • Results

    GWRP outperforms global max pooling, global average pooling, and previously proposed systems in time-frequency segmentation and sound event detection.

    Source: §VI · p. 10

  • Takeaways & Limitations

    The learned masks can provide event detections and, as a byproduct, separated sound-event waveforms.

    Source: §I · p. 2

Abstract

from arXiv · show

Sound event detection (SED) aims to detect when and recognize what sound events happen in an audio clip. Many supervised SED algorithms rely on strongly labelled data which contains the onset and offset annotations of sound events. However, many audio tagging datasets are weakly labelled, that is, only the presence of the sound events is known, without knowing their onset and offset annotations. In this paper, we propose a time-frequency (T-F) segmentation framework trained on weakly labelled data to tackle the sound event detection and separation problem. In training, a segmentation mapping is applied on a T-F representation, such as log mel spectrogram of an audio clip to obtain T-F segmentation masks of sound events. The T-F segmentation masks can be used for separating the sound events from the background scenes in the time-frequency domain. Then a classification mapping is applied on the T-F segmentation masks to estimate the presence probabilities of the sound events. We model the segmentation mapping using a convolutional neural network and the classification mapping using a global weighted rank pooling (GWRP). In SED, predicted onset and offset times can be obtained from the T-F segmentation masks. As a byproduct, separated waveforms of sound events can be obtained from the T-F segmentation masks. We remixed the DCASE 2018 Task 1 acoustic scene data with the DCASE 2018 Task 2 sound events data. When mixing under 0 dB, the proposed method achieved F1 scores of 0.534, 0.398 and 0.167 in audio tagging, frame-wise SED and event-wise SED, outperforming the fully connected deep neural network baseline of 0.331, 0.237 and 0.120, respectively. In T-F segmentation, we achieved an F1 score of 0.218, where previous methods were not able to do T-F segmentation.

I. INTRODUCTION

Strong sound-event labels require onset and offset annotations, which are time-consuming to collect, while weakly labelled data records event presence alone. The paper proposes learning time-frequency masks from weak labels to support sound-event detection and separation.

I. INTRODUCTION

  • Strongly labelled datasets are often limited to minutes or hours because onset and offset annotation takes more time than clip-level classification.Weakly labelled datasets provide event presence without onset, offset, or event-order annotations.Source: §I · p. 1
  • The framework learns time-frequency masks from weak labels, then uses them to detect events and separate sound-event waveforms from mixtures.The masks are predicted intermediate outputs rather than known training targets.Source: §I · p. 2

II. WEAKLY SUPERVISED SOUND EVENT DETECTION

  • Weakly supervised SED has been approached with multiple instance learning and convolutional neural networks.The introduction frames weakly labelled data as providing event presence without temporal annotations.Source: §II · p. 2

A. Multi-instance learning method

Multiple instance learning treats each clip as a bag of frames with clip-level labels, while CNN audio tagging predicts event presence without locating event times. Weakly supervised SED methods seek temporal localization from these limited labels.

A. Multi-instance learning method

  • In multiple instance learning, an audio clip is a bag of frame features, and its weak label indicates whether an event occurs at least once.The clip-level label is the maximum over frame-level event labels.Source: A. Multi-instance learning method · p. 2
  • MIL methods can underestimate positive instances and cannot predict time-frequency segmentations from weak labels.The section describes neural-network, SVM, negative-mining, and CRANE approaches to MIL.Source: A. Multi-instance learning method · p. 2

B. Convolutional neural networks for audio tagging and weakly supervised sound event detection

  • CNN audio tagging predicts clip-level event-presence probabilities, but not event onset and offset times.Typical systems apply global pooling and fully connected layers after convolutional processing.Source: B. Convolutional neural networks for… · pp. 2–3
  • Global max pooling can focus on the dominant time-frequency unit and underestimate event duration; attention pooling localizes in time but not in time-frequency.Segment-based localization also assumes that parts of segments correspond to clip labels, which may be unreasonable when events occur only in certain frames.Source: B. Convolutional neural networks for… · p. 3

III. TIME-FREQUENCY SEGMENTATION, SOUND EVENT DETECTION AND SEPARATION FROM WEAKLY LABELLED

The proposed framework learns event-specific time-frequency segmentation masks from weakly labelled audio. A segmentation mapping produces masks, and a jointly trained classification mapping predicts event presence.

III. TIME-FREQUENCY SEGMENTATION, SOUND EVENT DETECTION AND SEPARATION FROM WEAKLY LABELLED

  • A CNN-based segmentation mapping learns time-frequency masks from weakly labelled audio, unlike audio-tagging CNNs that predict only clip-level presence.The masks are learned without known mask targets, using clip-level audio tags during training.Source: §III · p. 3, A. Training from weakly labelled data · p. 3

A. Training from weakly labelled data

  • The segmentation mapping g1 converts an input time-frequency representation into one mask per sound event.Each mask ideally corresponds to the event's ideal ratio mask.Source: A. Training from weakly labelled data · p. 3
  • A classification mapping g2 converts each event mask into that event's presence probability, and binary cross-entropy trains both mappings end-to-end against weak labels.The weak labels are binary representations of event presence.Source: A. Training from weakly labelled data · p. 3

B. Time-frequency segmentation

  • At inference, the trained segmentation mapping produces masks that indicate which time-frequency units contribute to event presence.The learned masks are affected by the classification mapping.Source: B. Time-frequency segmentation · p. 3

C. Sound event detection

The framework converts time-frequency masks into frame- and event-wise detections by averaging across frequency and post-processing. The same masks can also separate event waveforms from a mixture.

C. Sound event detection

  • Averaging each time-frequency mask across frequency bins produces a frame-wise event score, which can be post-processed into event-wise predictions.The inference pipeline begins with the input representation and segmentation mapping to obtain masks.Source: C. Sound event detection · p. 3, Figure 4 · p. 4

D. Sound event separation

  • Upsampled masks are multiplied element-wise by the mixture-spectrum magnitude to form segmented spectrograms, then inverse Fourier transform and overlap-add produce separated waveforms.Interpolation aligns the masks with the complex spectrum when their time-frequency sizes differ; the spectrum phase is used for waveform reconstruction.Source: D. Sound event separation · p. 4

IV. PROPOSED SEGMENTATION MAPPING AND CLASSIFICATION MAPPING

The section describes a CNN segmentation mapping that produces time-frequency masks and classification mappings that estimate sound-event presence from those masks. The framework also uses the masks for sound-event separation.

A. Segmentation mapping

  • The segmentation mapping g1 takes a log mel spectrogram and uses a CNN with sigmoid outputs to produce T-F masks constrained between 0 and 1 as valid ideal ratio masks.Convolutional layers omit downsampling to retain mask resolution; batch normalization is used to stabilize and speed up training.Source: A. Segmentation mapping · p. 4

B. Classification mapping

  • The classification mapping g2 maps each sound-event mask to its presence probability, with global pooling methods explored for this mapping..Source: B. Classification mapping · p. 4
  • In the separation stage, upsampled masks multiply the input magnitude spectrum to form segmented spectrograms, which are inverse-transformed into separated sounds.Source: Figure 5 · p. 5

IV. PROPOSED SEGMENTATION MAPPING AND CLASSIFICATION MAPPING

  • Figure 6 organizes the framework into weak-label training, sound-event detection, and sound-event separation stages.Source: Figure 6 · p. 5

1) Global max pooling:

The section compares global max pooling (GMP), global average pooling (GAP), and global weighted rank pooling (GWRP) for classifying T-F masks. GMP and GAP can respectively underestimate and overestimate events, while GWRP weights mask values by rank.

1) Global max pooling:

  • GMP assumes one active T-F unit suffices for an event, but its maximum-only backpropagation encourages sparse masks and can underestimate events.Source: 1) Global max pooling · p. 5

2) Global average pooling:

  • GAP treats all T-F units as contributing equally to the clip label, which can overestimate events that last only a short time.Source: 2) Global average pooling · p. 5

3) Global weighted rank pooling:

  • GWRP weights mask values in descending rank order, interpolating between GMP at r = 0 and GAP at r = 1 while emphasizing high-valued T-F units.Figure 7 compares masks learned with the three pooling methods against an ideal ratio mask; it plots 6 of 41 event masks.Source: 3) Global weighted rank pooling · pp. 5–6, Figure 7 · p. 6

C. Post-processing for sound event detection

The experiments mix foreground sound events with acoustic scenes at several SNRs and evaluate tagging, detection, and segmentation. Event-wise predictions use thresholding and duration-based post-processing.

C. Post-processing for sound event detection

  • Event-wise SED uses positive audio-tagging predictions to select classes, thresholds frame scores at 0.2, extends boundaries to 0.1, and removes or joins events using a 320 ms duration threshold.Source: C. Post-processing for sound event… · p. 6

A. Dataset

  • The dataset combines 41 classes of foreground events with 10-second clips from 10 acoustic scenes, creating mixtures with three non-overlapping events at 20, 10, and 0 dB SNR.The study uses 8,000 mixed clips per SNR in four cross-validation folds; sound events are truncated to at most 2 seconds.Source: A. Dataset · p. 6

B. Evaluation metrics

  • Evaluation uses F-score, AUC, and mAP for audio tagging, frame-wise SED, and T-F segmentation, and error rate for event-wise SED.Source: B. Evaluation metrics · p. 6

1) Basic statistics:

This section defines the basic event-count terms and describes the evaluation metrics used to measure classification and detection performance. AUC and average precision summarize performance without requiring a manually selected threshold.

1) Basic statistics:

  • True positives match active reference and predicted events, false negatives are missed active reference events, and false positives are predicted events absent from the reference.Source: 1) Basic statistics · p. 6

2) Precision, recall and F-score:

3) Area under the curve (AUC):

  • AUC is the area under the ROC curve plotting true positive rate against false positive rate; it needs no manual threshold, and random guessing scores 0.5.Source: 3) Area under the curve (AUC) · p. 7

4) Average precision:

  • Average precision averages precision across recall values, avoids threshold selection, and does not count true negatives.Source: 4) Average precision · p. 7

5) Error rate:

This section defines event-wise error evaluation, then describes the feature representation and CNN configuration used by the system.

5) Error rate:

  • Event-wise ER counts insertions, deletions, and substitutions; detections allow 200 ms onset and 200 ms / 50% offset collars.The paper uses the toolbox for event-based SED evaluation.Source: 5) Error rate · p. 7

C. Feature extraction

  • Audio clips are represented by log mel spectrograms using a 2048-sample FFT window, 1024-sample overlap, and 64 mel bands.The setup is described as offering good time and frequency resolution.Source: C. Feature extraction · p. 7

D. Model

  • The model uses convolutional blocks and a sigmoid 1×1 convolution to produce T-F masks, followed by global pooling for clip-level presence probabilities.The CNN configuration is designed to fit a single GPU with 12 GB RAM.Source: D. Model · p. 7, Table I · p. 7

E. Audio tagging

The results compare audio tagging and frame-wise SED across pooling methods and SNRs. GWRP performs strongly, although the leading method varies by metric and SNR.

E. Audio tagging

  • At 0 dB, attention achieves the best audio-tagging F1-score of 0.612, followed by GWRP at 0.534.At 20 dB, attention leads F1-score (0.714) and mAP (0.755), while GWRP leads AUC (0.955).Source: E. Audio tagging · pp. 7–8, Table III · p. 8

F. Frame-wise sound event detection

G. Event-wise sound event detection

GWRP gives the strongest reported event-wise SED and T-F segmentation results among the evaluated pooling methods. The conclusion notes that the learned masks do not yet perfectly match the IRM.

G. Event-wise sound event detection

H. Time-frequency segmentation

  • At 0 dB, GWRP achieves the best average T-F segmentation F1-score of 0.218; previous systems could not produce T-F segmentation results.At 20 dB, GWRP achieves F1-score 0.324, AUC 0.849, and mAP 0.268; its masks look closer to the IRM than those from GMP and GAP.Source: H. Time-frequency segmentation · pp. 9–10

VI. CONCLUSION

  • The framework jointly trains segmentation and classification mappings on weak labels; its masks support event detection and can yield separated waveforms, but do not perfectly match the IRM.The paper identifies improving mask-to-IRM matching as future work for event separation.Source: §VI · p. 10
Loading 1804.04715v3…