PaperlayerView exact revision on arXiv

Source-linked AI summary

PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation

Yuan Gong, Yu-An Chung, James Glass

arXiv:2102.01243v3cs.SDcs.LGeess.AS

TL;DR

AudioSet audio tagging has benefited from architecture innovations, but the importance of training techniques has received less attention. PSLA combines ImageNet pretraining, balanced sampling, augmentation, label enhancement, and aggregation, achieving strong results with EfficientNet models on AudioSet and FSD50K. The findings present a training recipe whose components are quantified through extensive experimentation.

  • Problem

    Audio tagging research has emphasized model architectures and attention modules, while the contribution of training techniques for AudioSet has received less attention.

  • Method

    PSLA combines ImageNet pretraining, balanced sampling, data augmentation, label enhancement, weight averaging, and ensemble methods for EfficientNet audio-tagging models.

  • Results

    0.474 mAP was achieved by the PSLA ensemble on AudioSet, while the single 13.6M-parameter model reached 0.444 and the model achieved 0.567 mAP on FSD50K.

  • Takeaways & Limitations

    The proposed training strategies improve EfficientNet performance without modifying its model architecture and provide a recipe for AudioSet training.

Abstract

from arXiv · show

Audio tagging is an active research area and has a wide range of applications. Since the release of AudioSet, great progress has been made in advancing model performance, which mostly comes from the development of novel model architectures and attention modules. However, we find that appropriate training techniques are equally important for building audio tagging models with AudioSet, but have not received the attention they deserve. To fill the gap, in this work, we present PSLA, a collection of training techniques that can noticeably boost the model accuracy including ImageNet pretraining, balanced sampling, data augmentation, label enhancement, model aggregation and their design choices. By training an EfficientNet with these techniques, we obtain a single model (with 13.6M parameters) and an ensemble model that achieve mean average precision (mAP) scores of 0.444 and 0.474 on AudioSet, respectively, outperforming the previous best system of 0.439 with 81M parameters. In addition, our model also achieves a new state-of-the-art mAP of 0.567 on FSD50K.

I. INTRODUCTION

Audio tagging has advanced with AudioSet and model innovations, but the paper argues that training techniques are equally important. PSLA combines pretraining, sampling, labeling, augmentation, and aggregation strategies to improve performance on AudioSet and FSD50K.

  • I. INTRODUCTION: AudioSet contains over 2 million 10-second clips with 527 event labels, making it a large and comprehensive audio-tagging benchmark.The dataset includes weak, utterance-level labels and multiple labels per clip.
  • I. INTRODUCTION: 0.439 mAP was achieved by the previous best spectrogram- and waveform-based CNN system on AudioSet.Earlier systems progressed from 0.314 with shallow fully connected networks to 0.392 with residual networks and attention.
  • I. INTRODUCTION: 0.474 mAP was obtained by an EfficientNet ensemble on AudioSet, while a 13.6M-parameter single model reached 0.444.Both results are reported as outperforming the previous best system using 81M parameters.
  • I. INTRODUCTION: 0.567 mAP was achieved on the FSD50K benchmark, where the framework was evaluated to assess generalization.FSD50K contains 200 classes drawn from the AudioSet ontology.
  • I. INTRODUCTION: PSLA investigates four training categories: ImageNet pretraining, balanced sampling and augmentation, label enhancement, and model aggregation.The paper studies these techniques and their design choices, many of which had previously appeared separately.

B. Training and Evaluation Details

The experiments use EfficientNet-based audio tagging models trained and evaluated on AudioSet with mAP as the main metric. The model converts fixed-length audio into log Mel features, applies convolutional processing and multi-head attention, and produces class predictions.

  • B. Training and Evaluation Details: mAP is the main evaluation metric because it is informative for imbalanced datasets such as AudioSet and FSD50K.The discussion also reports ROC AUC and d-prime for comparison with prior work using those metrics.
  • B. Training and Evaluation Details: Each 10-second waveform is converted into a 1056 × 128 log Mel filterbank feature vector before entering the CNN.The features use a 25ms Hamming window every 10ms, with zero padding to 1056 frames.
  • B. Training and Evaluation Details: EfficientNet-B2 produces a 33 × 4 × 1408 tensor, which is mean-pooled over frequency into a 33 × 1408 representation.This representation is passed to the multi-head attention module.
  • B. Training and Evaluation Details: A four-headed attention module combines attention and classification branches through element-wise multiplication and temporal mean pooling.The weighted outputs of the heads are summed after scaling by learnable weights.
  • B. Training and Evaluation Details: 13.64M parameters comprise the EfficientNet-B2 audio-tagging model after adding the attention module and adjusting the classification layer.The paper retains this architecture while improving performance through training techniques.

III. NETWORK PRETRAINING

The paper evaluates ImageNet pretraining for audio tagging, finding consistent gains even with large in-domain training data and practical implementation advantages for spectrogram-based models.

  • Pretraining effectiveness: ImageNet pretraining improves audio tagging performance even when training uses the full AudioSet.The EfficientNet comparison initializes intermediate layers with ImageNet-pretrained or random weights while keeping subsequent training identical.
  • Pretraining effectiveness: 51.9% and 5.8% relative mAP improvements occur on the balanced-set and full-set experiments, respectively.
  • Data-volume dependence: ImageNet-pretraining gains decrease as audio-tagging training volume increases but remain noticeable across evaluated volumes.The evaluated volumes are 100k, 200k, 300k, and 500k clips.
  • Pretraining effectiveness: With balanced AudioSet, 60 epochs with ImageNet pretraining achieves 0.2385 mAP, versus 0.1694 mAP after 120 epochs from random initialization.The reported comparison indicates that additional training iterations alone produce a smaller improvement than ImageNet initialization.
  • Practical considerations: ImageNet pretraining requires no additional in-domain labeled or unlabeled dataset and is straightforward to adapt to audio models.For image-like inputs, adaptation modifies the first convolution and final classification layer.
  • Practical considerations: The approach is limited to models that accept 2D image-like inputs, such as spectrograms.The paper notes that many deep learning models for audio tasks meet this condition.

IV. BALANCED SAMPLING AND DATA AUGMENTATION

AudioSet’s extreme class imbalance makes simple upsampling or downsampling impractical, motivating random balanced sampling and augmentation. The proposed sampling reduces imbalance but retains multi-label and data-coverage limitations that augmentation addresses.

  • Annotation quality: AudioSet’s potential miss annotations are suggested by parent-class counts substantially exceeding the summed counts of their more specific subclasses.The caption gives “Speech” versus gender/age speech classes and “Music” versus happy/sad music classes as examples.
  • Balanced sampling: Random balanced sampling assigns higher weights to samples containing rare events and draws N samples per epoch with replacement.Sample weights account for all events appearing in each multi-label sample.
  • Balanced sampling: Balanced sampling remains imperfect because minority events are often paired with majority events, so sampled class frequencies stay imbalanced.
  • Balanced sampling: 40.9% of data is unseen after 30 epochs under balanced sampling, while low-frequency samples are repeated, increasing overfitting risk.

B. Time and Frequency Masking

Time and frequency masking augment audio during training by masking contiguous regions of the spectrogram. These masks improve audio-tagging performance without increasing the number of training samples per epoch.

  • Time and frequency masking: 18.2% and 14.6%: relative improvements from time and frequency masking on the balanced-set and full-set experiments, respectively.The overall number of training samples per epoch remains unchanged.
  • Time and frequency masking: Frequency masking removes f consecutive frequency channels, while time masking removes t consecutive time regions during training.Mask parameters are sampled on-the-fly for each audio sample.

C. Mix-up Training

Mix-up training creates convex combinations of audio examples and their labels, with design choices intended to produce more even mixtures and reduce mismatch with evaluation data. Larger α and a medium mix-up rate perform better in the reported experiments.

  • Mix-up construction: Mix-up constructs new waveform-level training examples by combining two audio samples and their corresponding labels with weight λ.The resulting example and label are convex combinations of the paired inputs and labels.
  • Mix-up design choices: A λ near 0.5 produces more evenly mixed samples and lets the model see both constituent samples.
  • Mix-up design choices: A 0.5 mix-up rate limits training–evaluation mismatch because evaluation samples are not mixed up.
  • Mix-up results: 0.3108 mAP: larger α with a medium mix-up rate outperforms the plain α=mix-up rate=1 setting at 0.3079 mAP.The authors use α = 10 and mix-up rate = 0.5 thereafter.

D. Summary

Training improvements include augmentation and balanced sampling, while label enhancement addresses prevalent AudioSet annotation errors. These techniques improve performance, although evaluation-set noise can mask gains.

  • Training techniques: 11.6% relative improvement and 0.4397 mAP result from combining balanced sampling with masking and mixup on the full set.Balanced sampling alone lowers performance, but helps when combined with augmentation.
  • Training techniques: 30.3% relative improvement and 0.3108 mAP result from data augmentation on the balanced set.The authors use balanced sampling for full AudioSet and augmentation as defaults thereafter.
  • Label quality: AudioSet contains substantial missing-label errors, including severe mismatches between parent and child speech-class annotations.Only 3.7% of full-set speech samples are labeled as male, female, or child speech, while 3,201 clips have child labels without the parent Speech label.
  • Label quality: Type I error omits a child label despite its event being present, whereas Type II error omits corresponding parent labels from child-labeled clips.These errors are formalized relative to the AudioSet ontology.
  • Label enhancement: Enhanced labels improve impacted and non-impacted classes, with larger relative gains for impacted classes, but AudioSet evaluation noise limits observed improvement.Enhanced-label models outperform original-label models on all reported evaluation sets except the original AudioSet evaluation set.

A. Model Weight Averaging

Weight averaging averages checkpoints after learning-rate reduction and consistently improves performance without additional training or inference cost. The best single models reach 0.3192 and 0.4435 mAP on balanced and full AudioSet experiments.

  • Method: Weight averaging equal-averages model weights across checkpoints, placing the solution near the center of a wide flat low-loss region.The paper motivates this as a route to better generalization than conventional training.
  • Results: 0.9% improvement results from weight averaging in both balanced-set and full-set experiments.Averaging begins after the learning rate decreases to one-quarter of its initial value.
  • Design choice: Model-averaging improvement is not sensitive to the exact starting epoch, with the weight-averaging optimum around epoch 15.The figure relates performance to the epoch at which averaging begins.
  • Results: 0.3192 and 0.4435 mAP are achieved by the best single models on balanced and full AudioSet experiments, respectively.The paper characterizes weight averaging as easy to implement, cost-free at training and inference, and consistently beneficial.

B. Ensemble

The ensemble section compares prediction-combination strategies, using simple voting while avoiding iterative ensembles because AudioSet training is computationally expensive.

  • Evaluation: Every reported ensemble has higher mAP than the best individual model in its committee.Table XI reports committee size, average mAP, best-model mAP, and ensemble mAP.
  • Ensemble design: Simple voting is used to compare multiple ways of building an ensemble committee.Iterative methods such as boosting are excluded because AudioSet training is computationally unreasonable for this work.

1) Checkpoint Averaging:

Checkpoint averaging combines predictions from multiple training epochs, while averaging models from different seeds or settings increases committee diversity. The reported ensembles consistently outperform their individual models.

  • Checkpoint Averaging: Checkpoint averaging averages outputs from models saved at multiple epochs, operating in model space rather than weight space.For full-set training, it is equivalent to bootstrap aggregating because sampling is performed with replacement.
  • Checkpoint Averaging: Averaging all checkpoints from the first epoch produces the highest mAP, while averaging from any epoch can outperform the best single checkpoint.The approach substantially increases inference computational overhead, reducing deployment practicality.
  • Model diversity: Three models trained with different random seeds yield a larger improvement than checkpoint averaging with only three models.The authors connect this result to the importance of ensemble diversity.
  • Model diversity: Ensembling models across pretraining, mixup, augmentation, and label-enhancement settings always improves performance.The best model uses all available models in the committee.

A. Ablation Study

The study evaluates PSLA across model architectures and datasets, finding that its training techniques consistently improve performance and generalize beyond AudioSet.

  • Ablation Study: Removing any PSLA technique causes a performance drop, indicating that all proposed techniques contribute to the best model.Balanced sampling is especially important: removing it reduces AudioSet performance below the pretraining-only model's 0.3939 mAP.
  • Various Audio Tagging Models: PSLA is evaluated with six architectures, including MobileNet V2, EfficientNet variants, and ResNet50, using the same input and training setting.Checkpoint averaging is used for PSLA models to reduce computational cost.
  • Various Audio Tagging Models: 0.4435 mAP is achieved by the single EfficientNet-B2 model with 4-headed attention, while single-headed attention and mean pooling perform slightly worse.The EfficientNet-B0 single-headed model has 5.36M parameters and achieves comparable performance.
  • Various Audio Tagging Models: Model choice depends on application: attention supports frame-level tagging, mean pooling supports streaming, and smaller models suit resource-constrained devices.Class-wise AP correlations exceed 0.95 across models, suggesting similar class-level behavior.
  • Generalization to FSD50K: 0.5671 mAP is achieved by the PSLA ensemble on FSD50K, outperforming the single and weight-averaged models and the best existing model.Removing any PSLA technique also reduces FSD50K performance, supporting generalization of the techniques.

D. Learning Curve of PSLA models

The learning-curve experiments assess training stability across balanced and full AudioSet settings, while the conclusion reports substantial gains from combining PSLA techniques.

  • D. Learning Curve of PSLA models: The best single EfficientNet-B2 with 4-headed attention is trained three times for both balanced and full AudioSet experiments.The plotted variability is represented by standard deviation across random seeds.
  • D. Learning Curve of PSLA models: The three repeated runs converge and achieve almost the same result, indicating little performance variation with the random seed.Figure 9 presents the learning curve and shaded standard deviation.
  • VIII. Conclusion: 130.6% and 28.2% performance improvements are obtained over a normal EfficientNet on the balanced and full AudioSet experiments, respectively.These gains come from combining the training techniques without modifying the model architecture.
  • VIII. Conclusion: 0.444 and 0.474 mAP are achieved by the single and ensemble EfficientNet models on AudioSet, respectively, surpassing the previous best system's 0.439 with 81M parameters.The reported best model uses 13.6M parameters.
  • VIII. Conclusion: The PSLA pipeline is presented as a recipe for AudioSet training because most methods are model agnostic and can combine with different architectures and attention modules.The authors intend the documented techniques to facilitate future audio tagging research.
Loading 2102.01243v3…