Source-linked AI summary
Large-scale weakly supervised audio classification using gated convolutional neural network
Yong Xu, Qiuqiang Kong, Wenwu Wang, Mark D. Plumbley
TL;DR
The paper targets audio tagging and weakly supervised sound event detection from clip-level labels without event timestamps. It uses a gated CRNN with temporal attention to localize events, achieving top DCASE 2017 challenge rankings in both subtasks. The approach remains weakly supervised because event locations are inferred as intermediate variables rather than trained from frame-level labels.
Problem
Weakly labelled audio provides event presence or absence without timestamps, making sound event detection difficult when frame-level strong labels are unavailable.
Method
A unified CRNN applies learnable GLUs to time-frequency representations and uses temporal attention to localize audio events from weak labels.
Results
The system ranked 1st in audio tagging and 2nd as a team in weakly supervised SED in the DCASE 2017 challenge.
Takeaways & Limitations
The unified model supports both clip-level audio tagging and event localization while learning from weakly labelled recordings.
Takeaways & Limitations
SED event locations are only weakly supervised intermediate variables because the task lacks frame-level strong labels.
Abstract
from arXiv · showhide
In this paper, we present a gated convolutional neural network and a temporal attention-based localization method for audio classification, which won the 1st place in the large-scale weakly supervised sound event detection task of Detection and Classification of Acoustic Scenes and Events (DCASE) 2017 challenge. The audio clips in this task, which are extracted from YouTube videos, are manually labeled with one or a few audio tags but without timestamps of the audio events, which is called as weakly labeled data. Two sub-tasks are defined in this challenge including audio tagging and sound event detection using this weakly labeled data. A convolutional recurrent neural network (CRNN) with learnable gated linear units (GLUs) non-linearity applied on the log Mel spectrogram is proposed. In addition, a temporal attention method is proposed along the frames to predicate the locations of each audio event in a chunk from the weakly labeled data. We ranked the 1st and the 2nd as a team in these two sub-tasks of DCASE 2017 challenge with F value 55.6\% and Equal error 0.73, respectively.
1. INTRODUCTION
The paper addresses audio tagging and weakly supervised sound event detection when recordings have clip-level labels but no event timestamps. It proposes one unified model using gated convolutional processing and temporal attention for localization.
- Task setting: Weakly labelled audio records indicate event presence or absence without timestamps, creating a setting for both audio tagging and sound event detection.The DCASE 2017 task uses AudioSet-derived clips and defines audio tagging and weakly supervised SED subtasks.
- Motivation: Bag-of-frames labeling is inaccurate because some audio events occur only briefly within a clip.The paper motivates time-frequency processing and localization to address this mismatch.
- Proposed approach: The proposed unified neural network simultaneously supports audio tagging and weakly supervised sound event detection.Its outputs include final audio-tagging predictions and SED predictions extracted from an intermediate localization module.
- Proposed approach: Learnable gated linear units replace convolutional ReLU activations to control information flow and emphasize relevant time-frequency units.The recurrent layer captures temporal information after convolutional processing.
- Proposed approach: Temporal attention attends to event-containing frames and ignores unrelated audio segments, enabling localization from weak labels.The method therefore infers event locations without timestamp annotations.
2. PROPOSED GATED LINEAR UNITS IN CRNN FOR AUDIO TAGGING
The model combines a CRNN baseline with gated linear units, weak-label training, mini-batch balancing, and system fusion for audio tagging. GLUs modulate convolutional features so the network can emphasize relevant time-frequency units.
- CRNN baseline: The CRNN converts recordings to log Mel time-frequency representations, extracts convolutional features, models temporal context with a bidirectional recurrent network, and predicts audio-tag posteriors.The baseline uses a feed-forward output layer whose size matches the audio classes.
- CRNN baseline: Binary cross-entropy trains predicted audio-tag probabilities against recording-level reference labels.Weights are updated by backpropagation, with Adam used for stochastic optimization.
- Gated linear units: GLUs replace ReLU activations and learn gates that attend to relevant time-frequency units while ignoring unrelated sounds.A gate near 1 attends to a unit, whereas a gate near 0 ignores it.
- Gated linear units: Each GLU output is a convolutional linear projection modulated by a sigmoid gate computed from the same input representation or feature maps.The element-wise product combines the linear and gated branches.
- Training strategy: Mini-batch balancing limits the average frequency ratio between the most and least frequent classes to at most 5:1.This addresses the dataset imbalance, such as 25,744 car examples versus 273 car-alarm examples.
- Training strategy: Two-level fusion averages systems across epochs and across configurations to improve stability and robustness.The second level averages posteriors from systems using different configurations.
3. PROPOSED LOCALIZATION FOR WEAKLY SUPERVISED SOUND EVENT DETECTION
The paper addresses weakly supervised sound event detection, where event timestamps must be inferred without frame-level labels. Its localization method preserves temporal resolution, uses softmax attention to weight salient frames, and trains through clip-level labels.
- Weakly supervised SED must infer event timestamps from clip-level labels without frame-level strong labels.The task is harder than audio tagging because temporal locations are unknown and strong labels are unavailable.
- An additional softmax feed-forward network produces localization vectors that attend to the most salient frames for each class.Pooling is applied only along the spectral axis, preserving time resolution for frame-level classification and localization.
- The localization vector is multiplied element-wise with each frame’s classification output before temporal aggregation.This combines class predictions with attention-derived frame weights.
- The final acoustic event tag prediction is a weighted average of frame-level outputs across the audio chunk.T denotes the final frame-level resolution along the spectrogram; without temporal pooling, it matches the whole spectrogram’s frame count.
- Training remains weakly supervised because the loss compares reference clip labels with the final output rather than frame-level annotations.Temporal event locations are inferred as intermediate variables during training.
4. EXPERIMENTS AND RESULTS
Experiments evaluate the system on a 17-class Google AudioSet subset using log-Mel and MFCC features with gated CRNN configurations. The system ranks first in audio tagging and second in sound event detection, while an example shows mostly accurate temporal localization with a small false alarm.
- Experimental setup: The experiments use a 17-class Google AudioSet subset of 10-second YouTube clips, with log-Mel filter banks and MFCCs as features.AudioSet contains 632 sound event classes and 2 million human-labeled clips; each input chunk has 240 frames and 64 mel channels.
- Experimental setup: The model uses three gated convolutional blocks, a bidirectional gated recurrent network, and 17 output nodes corresponding to the event classes.Pooling differs by task: 2*2 for audio tagging and 1*2 for sound event detection.
- Audio tagging: The final system ranks 1st in audio tagging on the blinded evaluation set by F1, with a notable absolute 3% improvement over the second system.The fusion system combines models trained on log-Mel and MFCC features.
- Weakly supervised sound event detection (SED): The fusion system ranks 2nd as a team in sound event detection, while the first-place team achieves 0.66 Error rate and 55.5% F1 score.The paper notes that the first-place system used separate segment inputs and assumed events occurred everywhere along the chunk, unlike the unified method.
- Weakly supervised sound event detection (SED): An example over 240 frames localizes ‘train’ and ‘train horn’ events almost accurately, with a small segment false alarm for ‘train horn’.The example illustrates the proposed temporal localization method on a single audio clip.
5. CONCLUSIONS
The paper presents a unified method for audio tagging and weakly supervised sound event detection, combining gated CRNN features with temporal attention-based localization. The final system achieved 57.7% F1 and ranked second in the SED sub-task.
- 57.7% F1 placed the final system first in DCASE2017 audio tagging.
- The proposed gated CRNN uses learnable gated linear units to select features related to final labels.
- Temporal attention-based localization identifies occurred events along the audio chunk under weak supervision.
- The team ranked second in the DCASE2017 sound event detection sub-task.