Source-linked AI summary
Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
Justin Salamon, Juan Pablo Bello
TL;DR
Environmental sound classification has limited labeled data, constraining the use of high-capacity CNNs. The paper proposes a deep CNN with audio augmentation, achieving state-of-the-art performance and showing that augmentation affects sound classes differently.
Problem
Relative scarcity of labeled environmental-sound data limits the effective use of high-capacity CNNs and has hindered improvement over simpler models.
Method
The paper combines a deep CNN architecture with audio data augmentation and examines how different audio deformations influence classification performance.
Results
The augmented SB-CNN achieves a mean accuracy of 0.79, exceeding its unaugmented version and the augmented shallow SKM baseline.
Takeaways & Limitations
Different augmentation sets affect sound classes differently, suggesting that class-conditional augmentation could further improve performance.
Abstract
from arXiv · showhide
The ability of deep convolutional neural networks (CNN) to learn discriminative spectro-temporal patterns makes them well suited to environmental sound classification. However, the relative scarcity of labeled data has impeded the exploitation of this family of high-capacity models. This study has two primary contributions: first, we propose a deep convolutional neural network architecture for environmental sound classification. Second, we propose the use of audio data augmentation for overcoming the problem of data scarcity and explore the influence of different augmentations on the performance of the proposed CNN architecture. Combined with data augmentation, the proposed model produces state-of-the-art results for environmental sound classification. We show that the improved performance stems from the combination of a deep, high-capacity model and an augmented training set: this combination outperforms both the proposed CNN without augmentation and a "shallow" dictionary learning model with augmentation. Finally, we examine the influence of each augmentation on the model's classification accuracy for each class, and observe that the accuracy for each class is influenced differently by each augmentation, suggesting that the performance of the model could be improved further by applying class-conditional data augmentation.
I. INTRODUCTION
Environmental sound classification has broad applications, but CNNs have been difficult to exploit because labeled datasets are relatively scarce. The paper addresses this limitation with a deep CNN and audio data augmentation.
- CNNs can learn spectro-temporal patterns from spectrogram-like inputs, making them well suited to environmental sound classification.
- Earlier CNN results were comparable to dictionary learning but did not improve upon the shallower approach.
- High-capacity neural networks require substantial training data to learn functions that generalize well and achieve high accuracy on unseen sounds.
- Environmental sound datasets remain considerably smaller than datasets used for image-classification research.
- Data augmentation creates additional labeled training examples through deformations that preserve semantic labels and may improve invariance to those deformations.
- The paper proposes a localized-kernel CNN, audio augmentation, and analysis of how different deformations affect performance.
A. Deep Convolutional Neural Network
The proposed model is a five-layer CNN that processes log-mel-spectrogram time-frequency patches through convolutional, pooling, and dense layers. Its localized kernels are intended to combine small patterns into larger sound signatures despite spectro-temporal masking.
- The architecture contains three convolutional layers, two pooling operations, and two fully connected layers.
- The network maps an input X to prediction Z through a composite nonlinear function F parameterized by Θ.
- Convolutional layers use filters and valid convolution, followed by point-wise activation functions.
- Strided max-pooling after the first two convolutional layers reduces feature-map dimensions, speeds training, and builds some scale invariance.
- The final two layers are dense matrix-product layers operating on flattened representations.
- Small 5×5 receptive fields are designed to learn localized patterns that later combine into larger time-frequency signatures under masking.
B. Data Augmentation
The study augments audio before log-mel-spectrogram conversion using four deformation types. Each deformation is parameterized to preserve label validity while expanding training variation.
- Four audio deformations produce five augmentation sets, and each is applied before conversion to the log-mel-spectrogram representation.
- Augmentation parameters are chosen so that each deformation preserves the semantic validity of the sound label.
- Time stretching changes playback speed while preserving pitch, using factors 0.81, 0.93, 1.07, and 1.23.
- Pitch shifting changes pitch while preserving duration, using small shifts of −2, −1, 1, and 2 semitones or larger shifts of −3.5, −2.5, 2.5, and 3.5 semitones.
- Dynamic range compression uses four parameterizations, while background-noise augmentation mixes samples with recordings from four acoustic scenes.
C. Evaluation
Evaluation uses UrbanSound8K, a ten-class dataset of 8732 field-recorded clips, with ten-fold cross-validation. The figures compare accuracy and class-confusion changes across models and augmentation settings.
- UrbanSound8K contains 8732 clips of up to 4 seconds spanning 10 environmental sound classes.
- Figure 1 compares unaugmented accuracy for SKM, PiczakCNN, and SB-CNN with augmented accuracy for SKM and SB-CNN.
- Figure 2 shows the augmented SB-CNN confusion matrix and changes in confusion between augmented and unaugmented training.
- The evaluation contrasts the proposed model with prior CNN and dictionary-learning approaches on the same environmental sound classification task.
- Models are evaluated with 10-fold cross-validation, reporting the accuracy scores from the ten folds as a box plot.
III. RESULTS
The proposed SB-CNN performs comparably to established models without augmentation but substantially improves with augmented training. Augmentation benefits most classes, although its effects vary by class and augmentation type.
- The augmented SB-CNN significantly outperformed the shallow SKM model, with p = 0.0003 in a paired two-sided t-test.
- Augmentation improved overall classification accuracy for all classes, while increasing confusion between air conditioner and engine idling.Confusion between air conditioner and drilling decreased with augmentation.
- Pitch augmentations had the greatest positive impact overall and were the only augmentation sets without a negative effect on any class.The air conditioner class was negatively affected by dynamic range compression and background-noise augmentations.
- Only half of the classes benefited more from combining all augmentations than from applying a subset.
IV. CONCLUSION
The study concludes that strong environmental sound classification performance comes from combining a deep, high-capacity CNN with augmented training data. Because augmentation effects differ across sound classes, class-conditional augmentation may improve performance further.
- The augmented deep CNN outperformed both the same CNN without augmentation and a shallow dictionary learning model with augmentation.
- Different augmentation sets influenced classification accuracy differently across sound classes.
- Class-conditional data augmentation could further improve the model’s performance.