PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking CNN Models for Audio Classification

Kamalesh Palanisamy, Dipika Singhania, Angela Yao

arXiv:2007.11154v2cs.CVcs.SDeess.AS

TL;DR

The paper asks whether standard ImageNet-pretrained CNNs can transfer effectively to audio spectrogram classification, despite the difference between spectrograms and natural images. It fine-tunes these models, analyzes which pretrained weights remain useful, visualizes learned spectrogram features, and evaluates run-to-run ensembles. The approach improves over random initialization and reaches reported state-of-the-art validation accuracy on ESC-50 and UrbanSound8K.

  • Problem

    The paper investigates whether ImageNet-pretrained image CNNs provide useful transfer learning for audio spectrograms, an area where prior work focused mainly on audio-pretrained or more complex models.

  • Method

    The paper fine-tunes pretrained Inception, ResNet, and DenseNet models on Melspectrograms, analyzes pretrained-weight changes and contributions, visualizes learned features, and ensembles independently trained runs.

  • Results

    92.8% validation accuracy on ESC-50 and 87.42% on UrbanSound8K were achieved by the ensemble DenseNet, reported as state-of-the-art on both datasets.

  • Takeaways & Limitations

    ImageNet-pretrained CNNs can serve as strong audio-classification baselines with a single input-feature set, while pretrained knowledge is especially important in early network layers.

Abstract

from arXiv · show

In this paper, we show that ImageNet-Pretrained standard deep CNN models can be used as strong baseline networks for audio classification. Even though there is a significant difference between audio Spectrogram and standard ImageNet image samples, transfer learning assumptions still hold firmly. To understand what enables the ImageNet pretrained models to learn useful audio representations, we systematically study how much of pretrained weights is useful for learning spectrograms. We show (1) that for a given standard model using pretrained weights is better than using randomly initialized weights (2) qualitative results of what the CNNs learn from the spectrograms by visualizing the gradients. Besides, we show that even though we use the pretrained model weights for initialization, there is variance in performance in various output runs of the same model. This variance in performance is due to the random initialization of linear classification layer and random mini-batch orderings in multiple runs. This brings significant diversity to build stronger ensemble models with an overall improvement in accuracy. An ensemble of ImageNet pretrained DenseNet achieves 92.89% validation accuracy on the ESC-50 dataset and 87.42% validation accuracy on the UrbanSound8K dataset which is the current state-of-the-art on both of these datasets.

I. INTRODUCTION

The paper examines whether standard ImageNet-pretrained CNNs can provide strong, minimally designed baselines for audio classification despite differences between spectrograms and natural images. It studies transfer learning, pretrained representations, and CNN behavior on spectrograms.

  • Spectrograms are popular audio representations because they work well with convolutional neural networks, although they differ from natural images by containing sequential temporal information.
  • AlexNet pretrained on ImageNet achieved 78% on GTZAN, demonstrating that spectrograms can be treated as images for audio classification.
  • Later advances from AlexNet to ResNet, Inception, and DenseNet had not been systematically applied as ImageNet-pretrained models for audio tasks.
  • Prior work increasingly used multiple input representations, custom CNNs, RNNs, CRNNs, or models pretrained on large audio datasets.
  • Using ImageNet-pretrained Inception, ResNet, and DenseNet with a single Melspectrogram input achieved state-of-the-art results on ESC-50 and UrbanSound8K and above 90% accuracy on GTZAN.
  • The paper analyzes transfer learning between images and spectrograms and uses Integrated Gradients to examine what CNNs learn from spectrograms.

A. Audio Classification

Audio classification research has used CNNs on spectrograms and other representations, with transfer learning increasingly relying on large audio datasets or complex multi-model systems. ImageNet-pretrained models provide an alternative transfer-learning route.

  • CNNs have been applied to music genre classification, environmental sound classification, and audio generation.
  • Raw-waveform audio classification uses 1-D convolutional models such as EnvNet and Sample-CNN, while many state-of-the-art systems use CNNs on spectrograms.
  • ImageNet-pretrained models have supported transfer learning for image segmentation, medical image analysis, and video classification.
  • Audio transfer learning has mainly pretrained models on large audio datasets such as AudioSet and the Million Song Dataset.

D. From Image Classification to Audio Classification

The paper contrasts increasingly complex audio-classification designs with a simpler approach using one model and one input feature set. It evaluates this approach on ESC-50, UrbanSound8K, and GTZAN.

  • Recent audio-transfer-learning systems often use large models and increasingly complex features, while earlier ImageNet-based approaches modified the design substantially.
  • A single model and a single input-feature set are reported to achieve state-of-the-art performance across multiple audio-classification tasks.
  • The stated benefit of this approach is reduced time and space complexity for developing audio-classification models.
  • The evaluated datasets are ESC-50, UrbanSound8K, and GTZAN.
  • ESC-50 contains 2,000 five-second clips from 50 classes and uses five-fold cross-validation for accuracy evaluation.

2) UrbanSound8k:

The section describes dataset settings, representation choices, three-channel spectrogram preprocessing, and the DenseNet architecture used in the experiments. It also identifies pretrained-versus-random weight accuracy as a comparison.

  • UrbanSound8k: UrbanSound8K contains 8,732 clips across 10 urban-sound classes, with clips no longer than four seconds and evaluation by 10-fold cross-validation.
  • GTZAN: GTZAN contains 1,000 30-second music clips across 10 genres, with 20% reserved for validation because no official split exists.
  • Feature representation: Log-Melspectrograms performed best among the tested ESC-50 representations, including log-spectrograms, MFCCs, and Gammatone-spectrograms.
  • Input construction: The CNN inputs use three channels formed either by replicating one Mel-spectrogram or by computing channels with different window sizes and hop lengths.
  • DenseNet architecture: DenseNet 201 uses four dense blocks containing 6, 12, 48, and 32 convolution layers, respectively.
  • Weight initialization: Table I compares accuracy when using pretrained weights versus randomly initialized weights.
  • Input representation: For ESC-50 and UrbanSound8K, the reshaped log-scaled inputs use 128 mel bins and size (128, 250), while GTZAN uses (128, 1500).

C. Models

The paper uses three standard ImageNet-trained CNN architectures for audio classification: Inception, ResNet, and DenseNet.

  • C. Models: The models are Inception, ResNet, and DenseNet pretrained on ImageNet.Inception combines multiple convolution sizes, ResNet uses residual blocks with skip connections, and DenseNet connects each layer to all preceding layers.

D. Deep Ensemble

The deep ensemble trains identical models independently and averages their softmax outputs, using initialization and mini-batch differences to create prediction diversity.

  • D. Deep Ensemble: Five independent models use the same architecture, hyperparameters, and training procedure, then average their softmax outputs at test time.Different linear-layer initializations and mini-batch orderings produce diversity among the models.
  • D. Deep Ensemble: The ensemble approach is evaluated for its effectiveness and compared with state-of-the-art models.The experiments examine pretrained weights, deep ensembling, and comparisons with existing methods.

A. Training the Models

The models are trained with shared hyperparameters, while pretrained and randomly initialized variants use different training schedules for comparison.

  • A. Training the Models: A learning rate of 1e−4 and weight decay of 1e−3 were selected for all models using grid search.The models used Adam with batch size 32 and were trained on one Nvidia RTX 2080 GPU.
  • A. Training the Models: ImageNet-pretrained ResNet, Inception, and DenseNet models were fine-tuned for 70 epochs on ESC-50, GTZAN, and UrbanSound8K.The learning rate decreased by a factor of 10 every 30 epochs.
  • A. Training the Models: Randomly initialized ResNet, Inception, and DenseNet models were trained from scratch on the same three datasets for 450 epochs.Their learning rate decreased by a factor of 10 at epochs 300 and 350.

2) Results:

Pretrained weights improve performance over random initialization, and transfer-learning analyses identify the early network—especially Block3—as particularly important for spectrogram classification.

  • 2) Results:: Pretrained weights improve accuracy by 20% on ESC-50, 10% on UrbanSound8K, and over 3% on GTZAN.The paper attributes these differences to insufficient data samples in the small datasets.
  • 3) Analysis of Pre-Trained Weights:: SVCCA measures feature similarity before and after fine-tuning, while weight fusion and freezing test which portions of DenseNet transfer effectively.These experiments use DenseNet on ESC-50.
  • 3) Analysis of Pre-Trained Weights:: Initializing through Block3 with pretrained weights raises validation accuracy to up to 90%, whereas later pretrained blocks add no improvement.The results indicate that pretrained knowledge is especially useful in the network’s initial portions.
  • 3) Analysis of Pre-Trained Weights:: Freezing Block3 reduces accuracy by nearly 10%, while freezing the first two blocks reduces it by only 2−3%.Removing Block4 preserves about 90% accuracy, but removing both Block3 and Block4 lowers validation accuracy to about 85%.

1) Setup:

Ensembling five independently trained models improves predictions on ESC-50 and UrbanSound8K, while the limited GTZAN validation set makes a small performance drop correspond to few samples.

  • Five models with different linear-layer initializations and mini-batch orderings are ensembled by averaging their softmax outputs.The ensemble accuracy is calculated from the averaged outputs.
  • Ensembling improves individual-model predictions on ESC-50 and UrbanSound8K.
  • A 1% GTZAN performance drop corresponds to two incorrectly predicted samples because validation uses only 200 samples.

1) Comparing Methods:

The proposed pretrained CNN models are compared with existing state-of-the-art systems across ESC-50, UrbanSound8K, and GTZAN, achieving leading reported results on the first two datasets.

  • The prior ESC-50 and UrbanSound8K state-of-the-art model uses a modified ResNet with ImageNet weights, self-attention layers, and three-channel spectrogram inputs.
  • GTZAN’s reported state-of-the-art accuracy is 94.5%, with noise cited as limiting higher accuracy.
  • 92.8% validation accuracy is achieved by the DenseNet ensemble on ESC-50, and 87.42% by the same model on UrbanSound8K.The paper identifies these as current state-of-the-art results on both datasets.
  • 91.99% accuracy is achieved by the ResNet ensemble on GTZAN.
  • The conclusion reports retained pretrained knowledge in initial layers, substantial intermediate-layer changes, and learned spectrogram energy-distribution boundaries.
Loading 2007.11154v2…