Source-linked AI summary
Data augmentation approaches for improving animal audio classification
Loris Nanni, Gianluca Maguolo, Michelangelo Paci
TL;DR
Animal audio classification is constrained by limited, costly training data, motivating improved augmentation and CNN-based recognition. The paper evaluates augmented audio and spectrogram representations with ensembles of adapted CNNs, reporting that fine-tuned CNN ensembles outperform prior state-of-the-art approaches in two audio classification problems. It also cautions that standard image augmentation protocols may be ineffective or harmful for audio.
Problem
Animal audio datasets are often limited because sample collection and labeling are expensive, while accurate classification supports biodiversity monitoring.
Method
The study evaluates data augmentation approaches with CNN ensembles using audio representations and adapted pretrained CNNs for animal audio classification.
Results
Fine-tuned CNN ensembles maximize performance on the two tested audio classification problems and outperform previous state-of-the-art approaches.
Takeaways & Limitations
The paper presents a large study of data augmentation for CNN-based animal audio classification using multiple approaches and classifiers.
Takeaways & Limitations
Standard image-oriented augmentation protocols may be useless or, in the worst case, unsuitable for audio classification.
Abstract
from arXiv · showhide
In this paper we present ensembles of classifiers for automated animal audio classification, exploiting different data augmentation techniques for training Convolutional Neural Networks (CNNs). The specific animal audio classification problems are i) birds and ii) cat sounds, whose datasets are freely available. We train five different CNNs on the original datasets and on their versions augmented by four augmentation protocols, working on the raw audio signals or their representations as spectrograms. We compared our best approaches with the state of the art, showing that we obtain the best recognition rate on the same datasets, without ad hoc parameter optimization. Our study shows that different CNNs can be trained for the purpose of animal audio classification and that their fusion works better than the stand-alone classifiers. To the best of our knowledge this is the largest study on data augmentation for CNNs in animal audio classification audio datasets using the same set of classifiers and parameters. Our MATLAB code is available at https://github.com/LorisNanni.
1. Introduction
The introduction motivates animal audio classification for biodiversity monitoring and examines data augmentation with CNN ensembles to address limited training data. It situates the work within audio representations, deep learning, and prior fusion-based recognition systems.
- Prior studies reported gains from combining CNNs with handcrafted or canonical approaches across audio and animal-recognition tasks.
- Deep learning shifted audio recognition toward visual representations such as spectrograms, enabling CNNs to learn informative features directly.
- Animal sound classification supports biodiversity assessment and minimally invasive monitoring of species such as birds, whales, frogs, bats, and cats.
- Animal audio datasets are often small because collecting and labeling samples is expensive, making data augmentation important for CNN training.
- This work compares augmentation approaches and CNN combinations across two datasets, using ensembles whose classifiers are combined by sum rule.
- The experiments found that augmentation techniques improved classification accuracy.
2. Audio Image Representation
The paper converts audio signals into image representations using the Discrete Gabor Transform, a Gaussian-windowed form of the Short-Time Fourier Transform. Its output represents frequency content over time for CNN processing.
- The Discrete Gabor Transform converts audio signals into image representations and is a particular case of the Short-Time Fourier Transform.
- The continuous Gabor transform convolves a Gaussian window with the signal multiplied by a complex exponential.
- In the transform, x(t) is the signal, ω is frequency, i is the imaginary unit, and σ^2 controls Gaussian-window width.
- The discrete transform uses discrete convolution and produces a matrix whose columns represent signal frequencies at fixed times.
3. Convolutional Neural Networks
The study adapts pretrained CNNs for animal audio classification by replacing their final layers and retraining them. It evaluates GoogleNet and VGGNet architectures as complementary classifiers.
- CNNs extract features through convolutional layers and perform final classification through fully connected and classification layers.
- Pooling layers downsample representations while reducing parameters, overfitting risk, and computational complexity.
- The study adapts networks pretrained on ImageNet or Places365 by replacing their final three layers and retraining the full CNNs.
- Two CNN families are tested and combined: GoogleNet and VGGNet.
- GoogleNet uses parallel convolutional filters in Inception modules, while VGGNet uses homogeneous 3x3 filters with pooling after groups of convolutional layers.
4. Data Augmentation approaches
The paper evaluates four augmentation protocols spanning image-like spectrogram transformations, standard audio processing, direct spectrogram manipulation, and raw-signal transformations. These protocols generate varied training examples through geometric, acoustic, mixing, masking, shifting, warping, and distortion operations.
- Protocol overview: Four augmentation protocols operate on spectrograms, raw signals, or both, using standard MATLAB methods and specialized transformations.The protocols are StandardIMG, StandardSGN, Spectro, and Signal.
- StandardIMG: StandardIMG applies image-style reflections, scaling, rotation, and translation to spectrogram representations.Reflections occur independently with 50% probability; scaling, rotation, and translation use random ranges specified in the protocol.
- Spectro: Spectro applies six spectrogram transformations, including same-class mixing, pitch and time shifts, VTLN, masking, and thin-spline warping.EMDA creates new images by weighted averaging of two randomly selected same-label spectrograms; other operations alter timing, frequency, or image structure.
- Signal: Signal applies 11 transformations directly to raw audio, including resampling, noise, clipping, speed changes, distortion, gain, time shifts, mixing, compression, and pitch shifts.The protocol includes both signal-level perturbations and synthetic-signal construction, such as swapping time segments or summing same-class signals.
5. Experimental results
The experiments evaluate augmentation protocols and CNN fusions on cat and BIRDZ animal-audio datasets using stratified ten-fold cross-validation. Fusion methods generally outperform stand-alone CNNs, while the best augmentation depends on the dataset and standard image augmentation performs poorly.
- Datasets and protocol: Stratified ten-fold cross-validation assessed augmentation effects on two animal-audio datasets, CAT and BIRDZ, using recognition rate as the performance indicator.The datasets contain cat sounds and bird recordings, respectively.
- Fusion methods: Five fusion approaches combined CNN outputs using sum rules, including fusions across augmentation protocols and across raw-signal and spectrogram representations.The ensembles were motivated by the assumption that collective decisions are less likely to err than individual-network decisions.
- Augmentation results: Spectro performed best on CAT, whereas Signal performed best on BIRDZ; Signal outperformed the NoAUG baseline on both datasets.No single augmentation protocol outperformed all others across every test.
- Classifier results: The best stand-alone CNN was VGG16 coupled with Signal, but its performance was clearly below that of the ensembles.VGG16 also exhibited convergence problems that required rerunning training, with problematic scores treated as zero in fusion calculations.
- Classifier results: StandardIMG produced the worst results, even relative to NoAUG, because generic image transformations such as reflection can reverse time-frequency patterns in spectrograms.The experiments therefore support augmentation techniques designed specifically for audio signals or their spectrograms.
- Comparison with prior work: Fusion Si+Sp outperformed the state-of-the-art comparisons on both datasets, while the study also reports a clear testing-protocol mismatch for one comparison.The authors used ten-fold cross-validation, whereas the cited work used repeated 60% training and 40% test splits.
Conclusion
The study evaluates data augmentation for CNN-based animal audio classification using raw signals and spectrograms, finding that fusion of fine-tuned CNNs performs best across the two tested problems. It also cautions that augmentation must reflect dataset characteristics because image-derived protocols can harm performance.
- Conclusion: Different data augmentation approaches were tested and compared for automated classification of bird and cat sounds.The approaches were applied to raw audio signals and spectrogram representations.
- Conclusion: CNNs were trained with four organized augmentation protocols and then combined using the sum rule.The protocols were applied to audio signals and their spectrogram representations.
- Conclusion: An ensemble of different fine-tuned CNNs maximized performance on the two tested audio classification problems and outperformed previous state-of-the-art approaches.
- Conclusion: The study is presented as the largest investigation of data augmentation for CNNs in audio classification, with further validation planned across additional datasets, tasks, CNN topologies, parameters, and augmentation methods.The MATLAB code is freely available for comparison.
- Conclusion: Image-specific augmentation techniques can be useless or detrimental for some datasets; StandardIMG performed worse than classification without augmentation.The authors therefore state that the dataset's nature must be considered when selecting an augmentation protocol.