Source-linked AI summary
A multi-device dataset for urban acoustic scene classification
Annamaria Mesaros, Toni Heittola, Tuomas Virtanen
TL;DR
The paper addresses acoustic scene classification with a larger, more variable multi-city and multi-device dataset for DCASE 2018. It presents three task settings and a CNN baseline; the baseline reaches 61% on subtask A evaluation data but drops to 47.6% on devices B and C under mismatch.
Problem
Existing acoustic scene classification datasets and evaluations offered limited geographic and recording-device variability, restricting study of mismatched conditions.
Method
The paper constructs the TUT Urban Acoustic Scenes 2018 datasets across six European cities and defines three subtasks, evaluating a convolutional neural network baseline.
Results
61% accuracy was obtained on subtask A evaluation data, while average accuracy on devices B and C was 47.6% in the mismatched-device setting.
Takeaways & Limitations
The dataset supports research on larger-scale urban acoustic variability, recording-device mismatch, and the use of external data or transfer learning.
Abstract
from arXiv · showhide
This paper introduces the acoustic scene classification task of DCASE 2018 Challenge and the TUT Urban Acoustic Scenes 2018 dataset provided for the task, and evaluates the performance of a baseline system in the task. As in previous years of the challenge, the task is defined for classification of short audio samples into one of predefined acoustic scene classes, using a supervised, closed-set classification setup. The newly recorded TUT Urban Acoustic Scenes 2018 dataset consists of ten different acoustic scenes and was recorded in six large European cities, therefore it has a higher acoustic variability than the previous datasets used for this task, and in addition to high-quality binaural recordings, it also includes data recorded with mobile devices. We also present the baseline system consisting of a convolutional neural network and its performance in the subtasks using the recommended cross-validation setup.
1. INTRODUCTION
DCASE acoustic scene classification has grown through increasingly larger datasets, while DCASE 2018 targets greater geographic and channel variability. The paper introduces a ten-class, multi-city dataset and three subtasks addressing standard and mismatched-device conditions.
- Motivation: DCASE acoustic scene classification is a recurring supervised multiclass task that attracted 17, 48, and 97 submissions in 2013, 2016, and 2017.Successive challenge editions also introduced larger datasets for training modern machine-learning systems.
- Motivation: DCASE 2017 exposed an acoustic-condition mismatch between development and evaluation recordings, causing a significant performance drop across systems.The mismatch followed a roughly one-year temporal gap between recordings.
- Dataset contribution: Parallel recordings from different devices introduce channel variability and enable large-scale study of classification under mismatched recording conditions.Previous public evaluations generally used one recording device, limiting mismatched-device evaluation.
- Paper scope: The paper presents the DCASE 2018 subtasks, recording procedure, experimental setup, CNN baseline, results, and conclusions.The sections cover dataset construction, task variants, baseline architecture, and performance evaluation.
2. DATA RECORDING PROCEDURE
The TUT Urban Acoustic Scenes 2018 dataset records ten urban scenes across six European cities using multiple locations, sessions, and synchronized recording devices. Two development versions expose high-quality-only data or additional mobile-device recordings.
- Recording design: Recordings cover ten predefined urban scenes across Barcelona, Helsinki, London, Paris, Stockholm, and Vienna.The scenes include transport, commercial, public, traffic, and park environments.
- Recording design: Each city and scene uses multiple locations, with 5–6 minutes recorded per location across 2–3 sessions and split into 10-second files.Recording locations are numbered to associate audio material from the same location.
- Devices: Device A is a binaural in-ear microphone and recorder setup sampling at 48 kHz with 24-bit resolution.The microphones were worn in the ears to mimic sound reaching a listener’s auditory system.
- Devices: Three mobile devices recorded simultaneously with device A, producing different-quality audio that was synchronized using Panako acoustic fingerprinting.The mobile devices were designated B, C, and D and included smartphones and a camera.
- Dataset versions: The standard development dataset contains device A recordings, while the Mobile version adds recordings from devices B and C.Both dataset versions are freely available.
3. TASK DEFINITION
The task labels short audio samples with predefined acoustic-scene classes in a supervised, closed-set setting. DCASE 2018 extends this setup through standard, mismatched-device, and external-data subtasks.
- Task definition: Acoustic scene classification assigns each audio sample to one predefined acoustic-scene class using labeled training examples.The setup is supervised and uses a closed set of categories.
- Subtask A: Subtask A evaluates the typical classification problem when development and evaluation data use the same high-quality recording device.Natural variation such as weather and people remains uncontrolled.
- Subtask B: Subtask B studies classification when high-quality training audio and multiple lower-quality evaluation devices create a recording-channel mismatch.The system must account for differing audio channels between development and evaluation data.
- Subtask C: Subtask C permits external data and transfer learning while keeping subtasks A and B restricted to common development data.This design supports studying possible improvement from additional data sources.
4. EXPERIMENTAL SETUP
The experiments use related classification setups across three subtasks with location-based development splits and organizer-evaluated hidden metadata. They quantify dataset scale, device coverage, and evaluation conditions for matched and mismatched recordings.
- Experimental design: Subtask A uses only device A, subtask B includes parallel devices B and C, and subtask C allows external data and transfer learning.The three subtasks retain the same basic classification problem while changing available data or conditions.
- Evaluation protocol: Evaluation metadata is hidden from participants, who submit predicted scene labels for organizer-performed scoring.The development set includes full metadata, whereas the evaluation set is published as audio only.
- Development data: The device-A development dataset contains 8,640 ten-second segments totaling 24 hours, with training locations covering approximately 70% of each city and scene class.The split includes 6,122 training segments and 2,518 test segments.
- Mobile development data: The Mobile development dataset adds two hours each from devices B and C to the 24 hours from device A, totaling 28 hours after channel alignment.Device A audio was resampled and converted to single channel for alignment.
- Evaluation data: The device-A evaluation dataset contains 3,600 segments totaling 10 hours, with locations different from those in development.Evaluation scoring uses class-wise average accuracy on balanced scene classes.
- Mobile evaluation data: The Mobile evaluation dataset contains 42 hours from devices A–D, including parallel and non-parallel recordings plus 1,440 segments from unseen device D.Ranking uses devices B and C, while device D assesses generalization to an unseen device without disclosed device identities.
5. BASELINE SYSTEM RESULTS
The baseline is a CNN system evaluated on subtasks A and B using repeated train/test trials and device-specific results. It performs reasonably in matched conditions but shows substantial degradation under device mismatch, especially for compressed device D.
- System architecture: The baseline uses a CNN derived from a top-ranked DCASE 2016 submission, with batch normalization and modified layer sizes.It extracts 40-band log mel energies from each 10-second file and feeds a 40x500 representation to two CNN layers and one fully connected layer.
- Experimental setup: Model selection uses a validation split with no shared recording locations between training and validation data.The split contains approximately 30% of the original training data and includes material from each city.
- Subtask A: 61% evaluation accuracy in subtask A slightly exceeds the 59.7% development accuracy, while public square is the hardest class at 33.9% evaluation accuracy.Street with traffic exceeds 80%, and most listed scene classes perform similarly across development and evaluation sets.
- Subtask B: Subtask B trains only on 6,122 high-quality segments from device A and uses average performance on devices B and C for official ranking.No additional mismatch-handling techniques are included, and results are reported separately by device.
- Subtask B: Device differences persist across development and evaluation sets, including metro accuracy changes from 20% to 45% for devices B and C and from 46% to 61% for device A.The passage reports that class-wise performance is mostly similar between development and evaluation sets while the device gap remains.
6. CONCLUSIONS
The DCASE 2018 acoustic scene classification task is organized around three subtasks using freely available datasets. They address larger and more variable data, device mismatch, and the use of external resources or transfer learning.
- Subtask A applies acoustic scene classification to a dataset larger and more acoustically variable than earlier challenge data.
- Subtask B focuses on solutions to the device mismatch problem.
- Subtask C permits external data and transfer learning to increase classification performance.
- The datasets are freely available beyond the challenge and may be extended with more cities and scene classes.The authors identify future directions including unbalanced-data training and open-set classification.