Source-linked AI summary
Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutions
Toni Heittola, Annamaria Mesaros, Tuomas Virtanen
TL;DR
The paper addresses acoustic scene classification under device mismatch and computational constraints, both important for real-world deployment. It presents the DCASE 2020 datasets, baselines, challenge results, and analysis, with top accuracies of 76.5% for multi-device classification and 96.5% for low-complexity classification. The paper concludes that data augmentation dominated multi-device solutions, while quantization was common among low-complexity systems.
Problem
Acoustic scene classifiers must generalize across many devices, including devices unavailable during training, while some applications require limited model complexity.
Method
The paper defines two DCASE 2020 subtasks, introduces their datasets and baseline systems, and analyzes submitted systems for multi-device robustness and low complexity.
Results
76.5% was the highest accuracy for Subtask A versus 51.4% for its baseline, while Subtask B reached 96.5% versus 89.5% for its baseline.
Takeaways & Limitations
Data augmentation was the main tool for multi-device generalization, while most low-complexity solutions used model-parameter quantization.
Takeaways & Limitations
Feature-extraction computational complexity was excluded from the system-complexity estimate because no established comparison method was used.
Abstract
from arXiv · showhide
This paper presents the details of Task 1: Acoustic Scene Classification in the DCASE 2020 Challenge. The task consists of two subtasks: classification of data from multiple devices, requiring good generalization properties, and classification using low-complexity solutions. Here we describe the datasets and baseline systems. After the challenge submission deadline, challenge results and analysis of the submissions will be added.
1. INTRODUCTION
DCASE 2020 frames acoustic scene classification around two real-world challenges: generalizing across many recording devices and operating under limited computational capacity. The task therefore introduces separate subtasks for multi-device robustness and low-complexity classification.
- Acoustic scene classification assigns a test recording to a predefined class describing its recording environment.
- Deep-learning systems are increasingly dominant, but their performance is sensitive to mismatches between training and testing devices.
- Realistic applications may involve many devices unavailable during training, making generalization across unseen devices a central challenge.
- Subtask A targets classification across real and simulated devices with strong generalization properties.
- Subtask B targets single-device classification under a maximum model-size constraint as a proxy for test-time complexity.
2. ACOUSTIC SCENE CLASSIFICATION WITH MULTIPLE DEVICES
Subtask A evaluates acoustic scene classifiers across real and simulated mobile devices, including devices unseen during training. It defines a multi-device dataset, standard metrics, an OpenL3-based baseline, and baseline evidence of device mismatch.
- Dataset and evaluation: The task uses ten acoustic scene classes and emphasizes generalization across multiple recording devices.
- Dataset and evaluation: TAU Urban Acoustic Scenes 2020 Mobile combines four real devices with synthetic devices simulated from device A recordings.
- Dataset and evaluation: The evaluation set contains 33 hours from 11 devices, including five devices unseen during training.
- Dataset and evaluation: Submissions are evaluated by macro-averaged accuracy and multi-class cross-entropy, with ranking based on accuracy.
- Baseline system: The OpenL3 baseline uses embeddings followed by two fully connected layers and is trained with Adam for 200 epochs.
- Baseline results: 54.1% is the baseline development-set accuracy, with performance falling to 39–48% on unseen devices.
3. LOW-COMPLEXITY ACOUSTIC SCENE CLASSIFICATION
Subtask B studies three-class acoustic scene classification on one device under a 500 KB non-zero-parameter limit. Its baseline uses a compact CNN, while the specification excludes feature-extraction complexity from the estimate.
- Dataset and evaluation: The task groups ten scenes into indoor, outdoor, and transportation classes and uses recordings from device A.
- System complexity requirements: The model-size limit is 500 KB for non-zero parameters, corresponding to 128,000 parameters in 32-bit floating-point format.
- System complexity requirements: The limit permits sparsity or parameter quantization as alternative ways to satisfy the complexity requirement.
- System complexity requirements: Feature-extraction computational complexity is excluded from system-size estimation, while embedding-generation networks count toward the limit.
- Baseline system: The baseline is a CNN with two convolutional layers, one fully connected layer, and a 450 KB model size.
4. CHALLENGE RESULTS
The challenge attracted substantial participation in both subtasks, with submissions addressing device generalization and low-complexity acoustic scene classification. Top systems substantially exceeded their respective baselines, using augmentation for Subtask A and compression-oriented model designs for Subtask B.
- 92 systems from 28 teams entered Subtask A, while 86 systems from 30 teams entered Subtask B, producing the largest participation in Acoustic Scene Classification to date.Interest was balanced between the multiple-device and low-complexity problems.
- Subtask A: 76.5% was the highest Subtask A accuracy, compared with 51.4% for the baseline system.Most submissions used mel-energy features and deep learning, while 71 submissions used at least mixup augmentation.
- Subtask A: Subtask A systems generalized well, with only a small development–evaluation performance difference, although performance remained highest on the heavily represented device A.Accuracy on training-seen devices was somewhat higher than on unseen devices, but the gap was smaller than for the baseline.
- Subtask A: The lowest log-loss among 90 systems was 0.755 for Liu UESTC, whose accuracy-optimized system was different from that lowest-log-loss system.Hu GT had much smaller log-loss than the top-accuracy system despite comparable accuracy.
- Subtask B: 96.5% was the highest Subtask B accuracy, compared with 89.5% for the baseline system.Submissions commonly used slim models, depth-wise separable CNNs, pruning, quantization, or related restrictions.
- Subtask B: Subtask B model sizes ranged from 8.8 KB to 499.5 KB, while McDonnell USA used a 3M-parameter single-bit-weight model.Most models were close to the allowed maximum size; the single-bit representation enabled a comparatively large model.
5. CONCLUSIONS AND FUTURE WORK
DCASE 2020 Task 1 targeted real-world acoustic scene classification through device robustness and low-complexity requirements. Data augmentation was the main approach for generalization, while quantization dominated low-complexity solutions; future work should improve complexity measurement.
- The two subtasks focused on robustness across multiple devices and low-complexity solutions for limited-resource devices.
- Data augmentation was the main tool for the multiple-device generalization problem, while most low-complexity solutions used quantized model parameters.
- Future challenge editions should develop a more accurate and universal methodology for measuring model or computational complexity.The goal is a widely accepted way to compare methods intended for limited-resource devices.