Source-linked AI summary
Knowledge Transfer from Weakly Labeled Audio using Convolutional Neural Network for Sound Events and Scenes
Anurag Kumar, Maksim Khadkevich, Christian Fugen
TL;DR
Sound-event learning is constrained by limited large-scale datasets, costly manual weak labeling, and the difficulty of collecting labels for rare events. The paper trains a variable-length CNN on weakly labeled AudioSet audio and uses its representations through direct extraction or target-task adaptation. Transferred representations support acoustic-scene classification and capture higher-level semantic information; on AudioSet’s 527 classes, NS yields absolute improvements of 1.2 MAUC and 4.6 MAP.
Problem
Sound-event learning is constrained by limited large-scale datasets, costly manual weak labeling, and the difficulty of collecting labels for rare events.
Source: §1 · p. 1Method
The paper trains a variable-length CNN on weakly labeled AudioSet audio and uses its representations through direct extraction or target-task adaptation.
Source: §1 · p. 1, §2.1 · p. 2, §3.2 · p. 3Results
Transferred representations support acoustic-scene classification and capture higher-level semantic information; on AudioSet’s 527 classes, NS yields absolute improvements of 1.2 MAUC and 4.6 MAP.
Source: §1 · p. 1, §4.1 · p. 3Takeaways & Limitations
The reported findings support using representations learned from weakly labeled sound-event data for both domain and task adaptation in sound events and acoustic scenes.
Source: §1 · p. 1
Abstract
from arXiv · showhide
In this work we propose approaches to effectively transfer knowledge from weakly labeled web audio data. We first describe a convolutional neural network (CNN) based framework for sound event detection and classification using weakly labeled audio data. Our model trains efficiently from audios of variable lengths; hence, it is well suited for transfer learning. We then propose methods to learn representations using this model which can be effectively used for solving the target task. We study both transductive and inductive transfer learning tasks, showing the effectiveness of our methods for both domain and task adaptation. We show that the learned representations using the proposed CNN model generalizes well enough to reach human level accuracy on ESC-50 sound events dataset and set state of art results on this dataset. We further use them for acoustic scene classification task and once again show that our proposed approaches suit well for this task as well. We also show that our methods are helpful in capturing semantic meanings and relations as well. Moreover, in this process we also set state-of-art results on Audioset dataset, relying on balanced training set.
1. INTRODUCTION
Large-scale sound-event learning is constrained by the difficulty and cost of labeling audio, while knowledge transfer from audio models remains relatively unexplored. The paper trains a weak-label CNN on AudioSet and proposes transferring its learned representations to sound-event and acoustic-scene tasks.
1. INTRODUCTION
- Sound-event detection and classification are constrained by limited large-scale datasets, while manual weak labeling remains resource-intensive and some events are rare.The paper motivates transfer from large datasets as one response to cases where collecting labeled data is difficult.Source: §1 · p. 1
- Audio-to-audio transfer is presented as relatively unexplored for sound events and scenes, despite prior audio transfer work and visual-model transfer to audio.The paper distinguishes its approach from Soundnet’s transfer from visual models to sound recognition.Source: §1 · p. 1
- The proposed AudioSet-trained CNN supports transfer to sound-event and acoustic-scene tasks, and the authors report that its representations capture semantic information and scene–event relationships.They describe this as an extensive exploration of transferring knowledge from a CNN trained on a large-scale sound-event dataset.Source: §1 · p. 1
2. DEEP CNN FOR WEAKLY LABELED AUDIO
The paper introduces a CNN framework that trains on weak recording-level labels and handles variable-length audio. Its segment outputs are pooled for recording-level prediction, with a multi-label loss for recordings that may contain multiple events.
2. DEEP CNN FOR WEAKLY LABELED AUDIO
- Earlier CNN work largely used strongly labeled data or weakly labeled datasets too small to indicate large-scale generalization and transfer potential.The paper contrasts this with an approach that treats weak labels as weak during training.Source: §2 · p. 2
2.1. Network Architecture
- The fully convolutional network scans variable-length audio in overlapping 128-frame segments, pools segment outputs for recording-level predictions, and can retain segment outputs for temporal localization.For the described network, segments advance by 64 log-mel frames; recording-level loss is computed after pooling.Source: §2.1 · p. 2
2.2. Multi-Label Training Loss
- For multi-label recordings, the model uses sigmoid class posteriors and averages binary cross-entropy losses across classes.The setup accommodates the possibility that multiple sound events occur in one web-audio recording.Source: §2.2 · p. 2
3. TRANSFER AND REPRESENTATION LEARNING
The paper extracts recording-level representations by aggregating segment outputs from the source-trained network. It proposes both direct off-the-shelf features and target-task adaptation, including direct adaptation of F1.
3. TRANSFER AND REPRESENTATION LEARNING
- Segment-level outputs from F1 or F2 are aggregated by max or average pooling into recording-level representations of 1024 or source-class dimensionality, respectively.F1 encodes source-audio knowledge, while F2 outputs a distribution over source labels.Source: §3 · p. 3
3.1. Direct Off-the-shelf Representations
- The off-the-shelf method feeds target-task log-mel audio into the source network and aggregates F1 and F2 outputs without adapting the network.It yields 1024-dimensional F1 and source-class-dimensional F2 representations.Source: §3.1 · p. 3
3.2. Transfer and Adapt for Learning Representations
- The adaptation methods transfer B1–B6 without updating them; Method I replaces F2 with a target-class convolutional layer and updates F1 and that new layer.Adapted representations are intended to be more discriminative for the target classification task.Source: §3.2 · p. 3
T . Method II (N II
Methods II and III preserve a source-label stage while adding target-task mappings at segment or recording level. Adaptation updates F1, F2, and the new target layer; the methods also accommodate different target labels and audio lengths.
T . Method II (N II
- Method II adds a target-class convolutional layer after F2 and updates F1, F2, and the new layer to map through source labels to target labels.The mapping is performed at segment level, and F2 uses ReLU during adaptation.Source: T . Method II (N II · p. 3
- Method III adds a target-class fully connected layer after global pooling and updates F1, F2, and that layer to learn the mapping at recording level.Its motivation is otherwise the same as Method II.Source: T . Method II (N II · p. 3
- The adaptation approaches can handle variable-length recordings and strongly or weakly labeled target data; target losses differ for multi-label and single-label tasks.The paper describes sigmoid with a multi-label loss for multi-label targets and softmax with categorical cross-entropy for single-label targets.Source: T . Method II (N II · p. 3
4. EXPERIMENTS AND RESULTS
The experiments report transfer results on Audioset, ESC-50, and DCASE 2016, alongside evidence that learned representations capture semantic structure and scene–event relationships. They evaluate both domain adaptation for sound-event classification and task adaptation for acoustic scene classification.
4.1. Audioset Results
- 1.2 absolute (1.3% relative) MAUC and 4.6 absolute (27.5% relative) MAP improvements over all 527 Audioset classes accompany 33% faster inference and 39% faster training.The table compares MAUC and MAP over all events and relative training and inference times.Source: Table 1 · p. 3, §4.1 · p. 3
4. EXPERIMENTS AND RESULTS
- The experiments use target-task training data to adapt NS, then train linear SVMs on representations; ESC-50 and DCASE 2016 use their respective cross-validation folds.The reported settings use 50 adaptation epochs and average accuracy across five ESC-50 runs or four DCASE runs.Source: §4 · p. 3, §4.2 · p. 4, §4.3 · p. 4
4.2. Domain Adaptation: Sound Event Classification
- On ESC-50, the proposed method outperforms the best method by 9.3%, setting state-of-the-art results; the comparison uses mean accuracy over all 50 classes.Source: §4.2 · p. 4
4.3. Task Adaptation: Acoustic Scene Classification
- 4.1% absolute improvement across 15 DCASE 2016 scenes includes gains of 33.0% for Park and 18.7% for Train.Task-adapted representations perform much better than representations taken directly from NS for this task.Source: §4.3 · p. 4, Table 3 · p. 4
- The ESC-50 t-SNE embeddings group representations by five broad semantic categories, while Vacuum Cleaner lies closer to several Exterior sounds than to its Domestic category.The DCASE panel visualizes representations for 15 acoustic scenes.Source: Figure 2 · p. 4
- Frequently top-activated F2 neurons correspond to sound events expected in their scenes, yielding semantically meaningful scene–event relations.For each scene, the analysis counts events appearing among the top five activations per segment and reports frequently occurring events.Source: §4.3 · p. 4, Figure 2 · p. 4
5. CONCLUSIONS
The paper presents a weak-label CNN trained on Audioset and transfer methods for target-task representations. Its experiments report strong benchmark performance and semantic scene–event relationships.
5. CONCLUSIONS
- The CNN model for weakly labeled sound events achieves state-of-art Audioset results while remaining computationally efficient and suited to transfer learning.The paper reports 83.5% ESC-50 accuracy, above stated human accuracy, and describes automatically discovering scene–event relationships.Source: §5 · p. 4
- Target-task representations learned using the CNN achieve state-of-art ESC-50 results, with 83.5% accuracy surpassing human accuracy on the dataset.Source: §5 · p. 4
- The transfer methods also support higher-level semantic understanding, including automatic discovery of relationships between acoustic scenes and sound events.Source: §5 · p. 4