Source-linked AI summary
Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks
M. Huzaifah
TL;DR
Environmental sound classification needs effective inputs because environmental audio is non-stationary and lacks speech’s phonetic structure. The paper compares several time-frequency transforms and MFCCs with CNNs on two datasets, finding that transform choice, windowing, and convolutional design affect accuracy.
Problem
Environmental sound classification requires suitable input representations because environmental sounds are non-stationary, dynamic, and lack speech’s underlying phonetic structure.
Method
The paper compares linear- and Mel-scaled STFT, CQT, CWT, and MFCC representations using CNN variants on ESC-50 and UrbanSound8K.
Results
Mel-STFT was consistently strong, time-frequency representations generally outperformed MFCCs by up to 15–20% in some cases, and 2D convolution generally outperformed 1D convolution.
Takeaways & Limitations
The preferable representation and window size depend on the audio characteristics and sound classes, while 2D convolution generally provides better results.
Abstract
from arXiv · showhide
Recent successful applications of convolutional neural networks (CNNs) to audio classification and speech recognition have motivated the search for better input representations for more efficient training. Visual displays of an audio signal, through various time-frequency representations such as spectrograms offer a rich representation of the temporal and spectral structure of the original signal. In this letter, we compare various popular signal processing methods to obtain this representation, such as short-time Fourier transform (STFT) with linear and Mel scales, constant-Q transform (CQT) and continuous Wavelet transform (CWT), and assess their impact on the classification performance of two environmental sound datasets using CNNs. This study supports the hypothesis that time-frequency representations are valuable in learning useful features for sound classification. Moreover, the actual transformation used is shown to impact the classification accuracy, with Mel-scaled STFT outperforming the other discussed methods slightly and baseline MFCC features to a large degree. Additionally, we observe that the optimal window size during transformation is dependent on the characteristics of the audio signal and architecturally, 2D convolution yielded better results in most cases compared to 1D.
I. INTRODUCTION
Environmental sound recognition supports applications including surveillance, robotics, and home automation, but environmental audio is more chaotic than speech. This study compares time-frequency representations as CNN inputs across two public datasets.
- Environmental sound recognition contributes to surveillance, robotics, and home automation applications.
- Environmental sounds are often more chaotic and noise-like than speech and lack its underlying phonetic structure.
- CNNs and spectrogram-like time-frequency representations have become distinctive developments in environmental sound classification.
- Time-frequency representations are considered useful learning features because environmental sounds are non-stationary and dynamic.
- The study compares linear- and Mel-scaled STFT, CQT, and CWT against MFCC features using several CNN variants on ESC-50 and UrbanSound8K.
A. Datasets
The paper evaluates environmental sound classification on ESC-50 and UrbanSound8K, which differ in size, class structure, and balance.
- ESC-50 contains 2000 five-second recordings evenly divided among 50 classes spanning five major sound groups.
- UrbanSound8K contains 8732 field-recorded clips of four seconds or less across 10 urban-sound classes.
- ESC-50 has relatively few samples for its many classes, while UrbanSound8K is imbalanced, with fewer car horn, gun shot, and siren examples.
- Prior reported accuracies were 64.5% for an earlier ESC-50 CNN, 74.2% for a deeper pretrained network, and a mean classification result from prior work on UrbanSound8K.
B. Pre-processing
The preprocessing pipeline constructs comparable image-like inputs from multiple transforms, while varying window resolution and preserving equalized input sizes across representations.
- Five input types were extracted: linear-STFT, Mel-STFT, CQT, CWT, and MFCC cepstrograms.
- Audio was standardized to four seconds, resampled at 22050 Hz, and transformed using whole clips without augmentation.
- Wideband and narrowband transforms used window lengths L = 2048 and L = 512, respectively, with hop size fixed at L/2.
- STFT spectrograms represent sound power as squared STFT magnitude, convert values to decibels, normalize them to [-1,1], and use linear or Mel frequency spacing.
- CQT uses tonal-spaced filters with constant Q, while CWT uses 256 frequency bins and a Morlet mother function.
- Inputs were downscaled to 37×50 or 154×12 pixels to equalize feature-map area, speeding training without sacrificing much accuracy.
C. Network Architecture and Evaluation
The evaluation compares shallower and deeper CNNs, square and frequency-spanning filters, and cross-validated training under controlled optimization settings.
- The study implemented shallower Conv-3 and deeper Conv-5 CNN variants based on image-recognition and prior environmental-sound models.
- Convolutional filters were either 3×3 squares or M×3 rectangles spanning all frequency bins and therefore operating one-dimensionally over time.
- Conv-3 used more aggressive max pooling, while both architectures used ReLU, dropout, and L2 regularization to reduce overfitting.
- Models were trained with Adam, batch size 100, cross-entropy loss, and 200 epochs for ESC-50 or 100 epochs for UrbanSound8K.
- Evaluation used 5-fold cross-validation for ESC-50 and 10-fold cross-validation for UrbanSound8K, with separate repeated runs and held-out test folds.
A. Impact of time-frequency representation
Spectral time-frequency representations generally outperformed MFCC features, while linear-STFT, Mel-STFT, and CQT performed comparably across both datasets.
- Time-frequency representations outperformed baseline MFCC features across experimental cases, by up to 15–20% in some cases.
- Median classification accuracies and median absolute deviations were reported for all experimental cases.
- Linear-STFT, Mel-STFT, and CQT performed comparably on both datasets, whereas CWT results were lower and closer to MFCC.
- Top-performing model variations were identified using ANOVA and post-hoc Tukey tests.Multiple transformations could be top performers when pairwise differences were not statistically significant.
B. Effect of CNN architecture and filter size
CNN depth and convolutional filter dimensionality affected performance: shallower models and 2D convolution generally performed better, with dataset- and model-specific exceptions.
- The shallower Conv-3 model tended to yield better accuracies than Conv-5 regardless of input.The authors associate this pattern with possible overfitting in the deeper model.
- Conv-3 training and test accuracy curves tended to be closer together than Conv-5 curves on ESC-50.The comparison is shown for narrowband Mel-STFT input.
- 2D convolution generally gave better results than 1D, except for Conv-3 on ESC-50.
- The authors suggest that variable-sized filters could trade off pitch invariance against discriminative power.They also note that Mel-STFT and CQT were expected to be more tolerant to pitch variation.
C. Wideband vs Narrowband
Wideband and narrowband transformations showed dataset- and class-dependent strengths rather than a consistent overall winner.
- The benefit of wideband over narrowband transforms was not consistent across both datasets.
- Wideband Mel-STFT performed better for classes with high frequency variation, while narrowband favored short temporal or droning sounds.Wideband performed poorly for “drilling,” “jackhammer,” and “air conditioner,” while excelling for “children playing.”
IV. CONCLUSION
The study compared time-frequency representations and CNN configurations for environmental sound classification. Mel-STFT was consistently strong, time-frequency inputs generally beat MFCCs, and 2D convolution usually outperformed 1D.
- The study compared linear- and Mel-scaled STFT, CQT, and CWT representations against MFCC features on two environmental sound datasets.
- Mel-STFT spectrograms were consistently good performers, while Linear-STFT and CQT also performed well on some models.
- All time-frequency representations generally produced better accuracies than baseline MFCC features.
- The preferred transformation window depended on the sound class, and 2D convolution generally worked better than 1D.The exception was the shallower model on ESC-50.