PaperlayerView exact revision on arXiv

Source-linked AI summary

PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, Mark D. Plumbley

arXiv:1912.10211v5cs.SDeess.AS

TL;DR

Audio pattern recognition lacks extensive large-scale pretraining despite the diversity of tasks and the success of pretraining in other fields. This paper trains PANNs on AudioSet, transfers them across six audio tasks, and evaluates multiple CNN architectures, including Wavegram-Logmel-CNN. PANNs achieve state-of-the-art AudioSet tagging and state-of-the-art performance on several transferred tasks.

  • Problem

    Earlier audio pattern recognition systems were built on specific datasets with limited durations, while large-scale pretraining remained limited.

  • Method

    The authors train PANNs on raw AudioSet recordings, investigate diverse CNN architectures, and transfer the pretrained models to other audio pattern recognition tasks.

  • Results

    PANNs achieve state-of-the-art AudioSet tagging and state-of-the-art performance on several transferred tasks, with Wavegram-Logmel-CNN reaching 0.439 mAP.

  • Takeaways & Limitations

    PANNs can generalize to other audio pattern recognition tasks and can be fine-tuned with limited data on new tasks.

Abstract

from arXiv · show

Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification, music classification, speech emotion classification and sound event detection. Recently, neural networks have been applied to tackle audio pattern recognition problems. However, previous systems are built on specific datasets with limited durations. Recently, in computer vision and natural language processing, systems pretrained on large-scale datasets have generalized well to several tasks. However, there is limited research on pretraining systems on large-scale datasets for audio pattern recognition. In this paper, we propose pretrained audio neural networks (PANNs) trained on the large-scale AudioSet dataset. These PANNs are transferred to other audio related tasks. We investigate the performance and computational complexity of PANNs modeled by a variety of convolutional neural networks. We propose an architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and waveform as input feature. Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392. We transfer PANNs to six audio pattern recognition tasks, and demonstrate state-of-the-art performance in several of those tasks. We have released the source code and pretrained models of PANNs: https://github.com/qiuqiangkong/audioset_tagging_cnn.

I. INTRODUCTION

The paper addresses limited large-scale pretraining for audio pattern recognition by training PANNs on AudioSet and transferring them across tasks. It evaluates varied CNN architectures and introduces Wavegram-Logmel-CNN for broad audio tagging.

  • Audio pattern recognition spans audio tagging, acoustic scene classification, music classification, speech emotion classification, and sound event detection.
  • Large-scale pretraining remains an open question because earlier systems often used private or task-specific datasets with limited durations or sound classes.
  • AudioSet provides over 5,000 hours of recordings across 527 sound classes, enabling large-scale training for general audio tagging.
  • The authors train PANNs on raw AudioSet recordings, investigate performance and computational complexity across neural architectures, and propose Wavegram-Logmel-CNN.
  • The work also transfers PANNs to other audio pattern recognition tasks and releases source code and pretrained models.
  • Wavegram-Logmel-CNN reaches 0.439 mAP on AudioSet tagging, exceeding the previous state-of-the-art system at 0.392 and Google’s system at 0.314.

2) Adapting CNNs for AudioSet tagging:

The paper adapts convolutional architectures for AudioSet tagging, using spectrogram-based CNNs with pooling, learned embeddings, task-specific outputs, and binary cross-entropy training.

  • The PANNs build on cross-task CNN systems and add a fully connected layer to the penultimate layer to increase representation ability.
  • The investigated CNN family includes 6-, 10-, and 14-layer networks inspired by AlexNet and VGG-like architectures.
  • Global pooling combines averaged and maximized feature vectors into fixed-length representations, followed by an added fully connected layer for embedding extraction.
  • For downstream tasks, linear classifiers use softmax for classification or sigmoid for tagging, while dropout is applied after downsampling and fully connected layers.
  • PANN outputs represent presence probabilities for K sound classes, and binary cross-entropy is minimized with gradient descent over AudioSet training clips.

2) Adapting ResNets for AudioSet tagging:

The paper adapts ResNets and lightweight MobileNets for AudioSet tagging while also examining waveform-based one-dimensional CNN alternatives to log-mel spectrogram systems.

  • Adapted ResNets first reduce log-mel spectrogram size with convolutional and downsampling layers, then use 22-, 38-, or 54-layer residual architectures.
  • MobileNets target lower computational cost by reducing parameters and multiply-add operations through depthwise separable convolutions.
  • The paper adapts MobileNetV1 and MobileNetV2, whose convolutional blocks contain two and three convolutional layers, respectively.
  • Waveform-based alternatives include DaiNet with large first-layer kernels and LeeNet with small waveform kernels replacing spectrogram extraction.

3) Adapting one-dimensional CNNs for AudioSet tagging:

The paper adapts one-dimensional CNNs by learning frequency-aware Wavegrams from waveforms and combining them with log-mel spectrograms. The resulting Wavegram-Logmel-CNN uses complementary time-domain and spectrogram information for AudioSet tagging.

  • Wavegrams learn time-frequency representations from waveforms, addressing the missing frequency axis in earlier one-dimensional CNNs.Frequency information helps represent sound events with different pitch shifts as the same class.
  • The Wavegram-CNN replaces hand-crafted log-mel spectrogram inputs with learned Wavegram features.The Wavegram is designed as a neural-network-learned modification of the Fourier transform.
  • The waveform CNN downsamples 32 kHz audio to 100 feature frames per second before reshaping its output into time-frequency channels.The stated calculation is 32,000/5/4/4/4 = 100 frames per second.
  • Wavegram-Logmel-CNN concatenates Wavegram and log-mel representations along the channel dimension.The combination is intended to use information from both time-domain waveforms and log-mel spectrograms.
  • The section also introduces data balancing and augmentation for training AudioSet tagging systems.Data balancing addresses highly imbalanced data, while augmentation is used to prevent overfitting.

A. Data balancing

The training procedure addresses AudioSet’s long-tailed class distribution and limited examples through data balancing and augmentation. The paper compares transfer strategies including frozen feature extraction and full fine-tuning.

  • A. Data balancing: AudioSet has a long-tailed class distribution, ranging from over 900,000 clips for Speech and Music to tens for Toothbrush.Without balancing, clips are uniformly sampled during mini-batch training.
  • A. Data balancing: Mixup and SpecAugment augment training data because some sound classes contain only hundreds of clips.The paper describes augmentation as a way to reduce overfitting.
  • A. Data balancing: Mixup interpolates both inputs and targets from two audio clips using a coefficient sampled from a Beta distribution.The default application is on the log-mel spectrogram.
  • Transfer learning: PANN transfer uses either frozen embedding features with a new classifier or initialization followed by fine-tuning of all parameters.The paper also compares both strategies with training a task-specific system from scratch.

VI. EXPERIMENTS

Experiments evaluate PANNs on AudioSet tagging and transfer them across audio pattern-recognition tasks. AudioSet experiments use downloaded waveforms, standardized preprocessing, multiple metrics, and comparisons with prior systems.

  • PANNs are evaluated first on AudioSet tagging and then transferred to acoustic scene classification, general audio tagging, music classification, and speech emotion classification.
  • AudioSet contains 527 sound classes and 2,063,839 training clips, with 1,934,187 training clips successfully downloaded for the experiments.Audio clips are padded or trimmed to 10 seconds, converted to mono, and resampled to 32 kHz.
  • The log-mel CNN configuration uses an STFT with a 1024-sample Hamming window, 320-sample hop, and 64 mel filter banks.The configuration produces 100 frames per second and uses 50 Hz to 14 kHz cutoff frequencies.
  • AudioSet tagging is evaluated with mAP, mAUC, and d-prime, calculated per class and then averaged.Average precision is the area under the recall–precision curve, whereas AUC incorporates true negatives.

2) Class-wise performance:

Class-wise results vary substantially across AudioSet sound classes, and performance is not consistently correlated with training-set size. Data balancing, full-data training, and mixup affect convergence and tagging performance.

  • 2) Class-wise performance: Class-wise AP varies widely: Music and Speech exceed 0.80, whereas Inside, small reaches only 0.19.
  • 2) Class-wise performance: Audio tagging performance is not always correlated with the number of training clips.The paper attributes differences partly to varying difficulty across sound classes.
  • A. Data balancing: Data balancing enables faster convergence within limited training iterations.Systems trained with the full 1.9 million clips outperform systems trained with the balanced 20k subset.
  • A. Data balancing: 0.431 mAP is obtained with mixup, compared with 0.416 without mixup.On the 20k balanced subset, mixup yields 0.278 mAP versus 0.221 without mixup.

5) Hop sizes:

Smaller hop sizes improve CNN14 AudioSet tagging performance, while training-data volume and frequency resolution also affect mAP. The selected 64 mel bins balance performance and computational cost.

  • Hop sizes:: 0.431 mAP: CNN14 with a hop size of 320 outperforms hop sizes of 500, 640, and 1000.These hop sizes correspond to 10.00, 15.63, 20.00, and 31.25 ms between adjacent frames, respectively.
  • Embedding dimension increases improve CNN14 mAP across the investigated dimensions of 32, 128, 512, and 2048.
  • 0.406 mAP: training CNN14 with 50% of the downloaded AudioSet data causes a 5.8% drop from full-data performance.Using 80% of the full data decreases mAP from 0.431 to 0.426, a 1.2% drop.
  • 0.406 mAP: 8 kHz audio performs 5.8% below 32 kHz, while 16 kHz reaches 0.427, within 1.0% of 32 kHz.The result indicates that information in the 4 kHz–8 kHz range is useful for audio tagging.
  • 0.442 mAP: 128 mel bins outperform 64 and 32 mel bins, which achieve 0.431 and 0.413, respectively.Computation complexity increases linearly with the number of mel bins; the paper adopts 64 as a trade-off.

10) Number of CNN layers:

Deeper convolutional PANNs generally improve AudioSet tagging, with ResNet38 and Wavegram-Logmel-CNN among the strongest systems. MobileNets substantially reduce computation and model size at lower mAP.

  • Number of CNN layers:: 0.431 mAP: the 14-layer CNN outperforms the 10-layer and 6-layer CNNs, which achieve 0.380 and 0.343.The paper contrasts this with earlier results on smaller datasets, where shallower CNNs performed better.
  • 0.434 mAP: ResNet38 slightly outperforms the other investigated systems, while ResNet54 reaches 0.429 without further improvement.ResNet22 achieves 0.430, similar to CNN14.
  • 0.389 mAP: MobileNetV1 is 9.7% below CNN14's 0.431 while using only 8.6% of its multi-adds and 5.9% of its parameters.The lightweight architecture is evaluated as a lower-complexity alternative.
  • 0.365 mAP: Res1dNet31 leads the proposed one-dimensional CNN systems, ahead of Res1dNet51 at 0.355 and LeeNet11 at 0.266.The improved 24-layer LeeNet reaches 0.336, while DaiNet reaches 0.295.
  • 0.439 mAP: Wavegram-Logmel-CNN achieves state-of-the-art performance among all PANNs.Wavegram-CNN reaches 0.389 and outperforms the best previous one-dimensional CNN system, indicating that the learned Wavegram feature is effective.

15) Complexity analysis:

PANN complexity is evaluated using multi-adds and parameter counts, revealing a trade-off between AudioSet tagging performance and inference efficiency. The paper then transfers PANNs to several downstream tasks, including few-shot settings.

  • Complexity analysis:: 42.2 × 10^9 multi-adds and 80.8 million parameters: CNN14 is substantially larger than CNN6 and CNN10 for 10-second inference.
  • 3.6 × 10^9 and 2.8 × 10^9 multi-adds: MobileNetV1 and MobileNetV2 reduce computational cost and system size.They use around 4.8 million and 4.1 million parameters, respectively.
  • Transfer learning:: PANN transfer is investigated across other audio pattern-recognition tasks, including settings with limited labelled training clips.The paper frames this as an application of PANNs to few-shot learning.
  • ESC-50:: 0.947 accuracy: fine-tuned CNN14 outperforms the previous ESC-50 state of the art at 0.865.Freeze_L1 and Freeze_L3 achieve 0.918 and 0.908, respectively.

2) DCASE 2019 Task 1:

On DCASE 2019 Task 1, fine-tuning PANNs improves over training CNN14 from scratch, although feature extraction alone does not. The supplied passages also report transfer results for DCASE 2018 Task 2, MSoS, and GTZAN.

  • DCASE 2019 Task 1:: 0.764 accuracy: fine-tuned CNN14 outperforms CNN14 trained from scratch at 0.691 on DCASE 2019 Task 1.Freeze_L1 and Freeze_L3 achieve 0.689 and 0.607, respectively, and do not outperform the from-scratch system.
  • 0.851 accuracy: the cited DCASE 2019 state-of-the-art system exceeds the reported fine-tuned CNN14 result.The cited system uses a combination described in the passage but not fully included here.
  • DCASE 2018 Task 2:: Fine-tuned CNN14 outperforms systems trained from scratch and systems using PANN as a feature extractor on DCASE 2018 Task 2.Its mAP@3 is comparable to the cited state-of-the-art system.
  • MSoS:: 0.960 accuracy: fine-tuned CNN14 outperforms the previous MSoS state of the art at 0.930.CNN14 trained from scratch reaches 0.760, while fine-tuning and feature extraction outperform that baseline.

5) GTZAN:

GTZAN evaluates music genre classification, where prior state-of-the-art performance reached 0.939 accuracy; the paper also reports that pretrained PANNs approach state-of-the-art on GTZAN.

  • GTZAN contains 1,000 30-second clips spanning 10 music genres, evaluated using 10-fold cross validation.
  • 0.939 accuracy is achieved by the previous state-of-the-art GTZAN system using a bottom-up broadcast neural network.
  • Fine-tuned PANNs approach state-of-the-art performance on the GTZAN classification task.
  • PANNs generalize to new audio pattern recognition tasks, including GTZAN, where training data are limited.
Loading 1912.10211v5…