Source-linked AI summary
General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline
Eduardo Fonseca, Manoj Plakal, Frederic Font, Daniel P. W. Ellis, Xavier Favory, Jordi Pons, Xavier Serra
TL;DR
The paper addresses general-purpose audio tagging across 41 diverse AudioSet categories using heterogeneous user-generated Freesound audio and annotations of varying reliability. It describes the task, constructs the FSDKaggle2018 dataset, and provides a baseline achieving 0.70 mAP@3 on the entire test set.
Problem
The task targets robust recognition of a broad range of sound events in realistic user-generated audio, including 41 diverse categories and training annotations with varying reliability.
Method
The paper defines a multiclass audio-tagging task, prepares the FSDKaggle2018 Freesound dataset, and separates manually verified from non-verified annotations for baseline development.
Results
0.70 mAP@3 was achieved on the entire test set after five training epochs.
Takeaways & Limitations
The task and dataset provide a benchmark for broadly applicable sound event classifiers using heterogeneous Freesound content and annotations with different reliability levels.
Takeaways & Limitations
Automatically generated candidate annotations depend on user tags and AudioSet mapping, so their reliability depends on subsequent validation.
Abstract
from arXiv · showhide
This paper describes Task 2 of the DCASE 2018 Challenge, titled "General-purpose audio tagging of Freesound content with AudioSet labels". This task was hosted on the Kaggle platform as "Freesound General-Purpose Audio Tagging Challenge". The goal of the task is to build an audio tagging system that can recognize the category of an audio clip from a subset of 41 diverse categories drawn from the AudioSet Ontology. We present the task, the dataset prepared for the competition, and a baseline system.
1. INTRODUCTION
The paper introduces a general-purpose audio-tagging task spanning 41 diverse AudioSet categories and user-generated Freesound audio. It targets both broad sound-event recognition and training with annotations of varying reliability.
- Earlier DCASE audio-tagging tasks focused on domestic sounds and smart cars, whereas this task targets general-purpose audio.
- The task classifies audio clips into one of 41 diverse categories drawn from the AudioSet Ontology.
- Freesound user-generated audio offers a large research resource but introduces annotation challenges not addressed by earlier DCASE tasks.
- The task addresses recognizing more diverse sound events and leveraging training subsets with annotations of varying reliability.
- The resulting systems may support automatic multimedia description and acoustic monitoring applications.
2. TASK SETUP
DCASE 2018 Task 2 is a Kaggle multiclass, single-tag audio-tagging competition that predicts one category per test clip. Evaluation uses ranked predictions and separate public and private leaderboard divisions.
- The task is a multiclass problem requiring one predicted category for each audio clip in the test set.
- The competition ran on Kaggle from March 30th to July 31st 2018 under the name “Freesound General-Purpose Audio Tagging Challenge”.
- The unreleased-label test set was divided into 19% for the public leaderboard and 81% for the private leaderboard.
- mAP@3 rewards correct labels most when ranked first and accepts up to three predicted labels per clip.
- Participants submitted predicted labels in CSV files, with at most two submissions daily and two final submissions for ranking.
- A complementary Judges’ Award recognized novel, problem-specific, and efficient approaches beyond leaderboard performance.
3. DATASET
FSDKaggle2018 is a curated Freesound dataset organized around AudioSet categories, combining manually verified and non-verified annotations. Its construction preserves variable clip durations and annotation reliability while producing a 41-category benchmark.
- Dataset source: FSDKaggle2018 is a reduced subset of FSD, prepared for the task from Freesound audio content.
- Dataset source: Freesound provides heterogeneous user-uploaded recordings with tags, titles, descriptions, and Creative Commons licensing.
- Annotation creation: 268,261 Freesound clips were mapped to AudioSet categories through user-tag mappings, generating weak clip-level candidate annotations.
- Annotation creation: Candidate annotations were manually validated through user ratings, training examples, category descriptions, quality controls, and agreement-based prioritization.
- Dataset curation: Manually-verified annotations contain only PP ratings, while non-verified annotations mainly comprise unrated candidates plus a small amount of rated data.
- Dataset curation: Curation filtered categories and clips by AudioSet hierarchy, duration, licensing, annotation counts, and annotation quality before forming the final dataset.
- Dataset description: The dataset contains 11,073 mono 44.1 kHz PCM files across 41 categories, with clips lasting 300ms to 30s and nearly 18h of training audio.
- Dataset limitations: Non-verified clips may contain multiple sound sources despite having one ground-truth label, and their reliability depends on user tags and ontology mapping.
4. BASELINE SYSTEM
The baseline applies a shallow CNN to logmel spectrograms and aggregates overlapping window predictions into clip-level outputs. It reaches an mAP@3 of 0.70 on the entire test set after five training epochs.
- The baseline uses a relatively shallow 3-layer CNN with logmel spectrogram inputs and a 41-way softmax classifier.
- Audio is processed as 0.25s overlapping windows with 0.125s hops, using 25ms STFT windows every 10ms and 64 mel bins.
- 0.70 mAP@3 is achieved on the entire test set after training for 5 epochs.
- The baseline scores 0.70 on the public leaderboard split and 0.69 on the private split.
5. CONCLUSION
The paper describes the DCASE 2018 Task 2 setup, FSDKaggle2018 dataset, and baseline for general-purpose sound event recognition. The dataset combines varying annotation reliability and variable-length clips to support open research in sound event recognition.
- The paper presents the task setup, FSDKaggle2018 dataset, and baseline for DCASE 2018 Task 2.
- FSDKaggle2018 includes training subsets with annotations of varying reliability and variable-length audio clips.
- The competition and baseline are intended to foster open research in sound event recognition.