Source-linked AI summary
CNN Architectures for Large-Scale Audio Classification
Shawn Hershey, Sourish Chaudhuri, Daniel P. W. Ellis, Jort F. Gemmeke, Aren Jansen, R. Channing Moore, Manoj Plakal, Devin Platt, Rif A. Saurous, Bryan Seybold, Malcolm Slaney, Ron J. Weiss, Kevin Wilson
TL;DR
The paper asks whether image-classification CNNs and large weakly labeled datasets can support large-scale audio classification. It compares architectures and dataset scales on YouTube-100M, finding strong CNN performance and substantially better Audio Set AED results from learned embeddings than raw log-mel features. The study also identifies imperfect, unevenly relevant machine-generated labels as a dataset limitation.
Problem
The paper investigates whether large datasets and CNN architectures successful in image classification can provide effective evidence for audio soundtrack classification and AED.
Method
The study trains fully connected DNN, AlexNet, VGG, Inception, and ResNet models on YouTube-100M log-mel inputs while varying training-set and label-vocabulary sizes, then evaluates embeddings on Audio Set AED.
Results
CNNs outperform the fully connected baseline, larger label and video sets generally improve performance up to a point, and ResNet embeddings substantially outperform raw log-mel features on Audio Set AED.
Takeaways & Limitations
Image-classification CNN analogs can achieve excellent audio-classification results, while embeddings from the best classifier provide a stronger AED representation than raw features.
Takeaways & Limitations
YouTube-100M labels are machine-generated, not 100% accurate, and vary in acoustic relevance.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) have proven very effective in image classification and show promise for audio. We use various CNN architectures to classify the soundtracks of a dataset of 70M training videos (5.24 million hours) with 30,871 video-level labels. We examine fully connected Deep Neural Networks (DNNs), AlexNet [1], VGG [2], Inception [3], and ResNet [4]. We investigate varying the size of both training set and label vocabulary, finding that analogs of the CNNs used in image classification do well on our audio classification task, and larger training and label sets help up to a point. A model using embeddings from these classifiers does much better than raw features on the Audio Set [5] Acoustic Event Detection (AED) classification task.
1. INTRODUCTION
The paper tests whether large-scale datasets and CNN architectures developed for image classification transfer effectively to audio soundtrack classification and AED. It compares architectures, training and label-set sizes, and weak-label learning at YouTube-100M scale.
- Motivation: YouTube-100M contains 70 million training videos totaling 5.24 million hours, tagged with 30,871 video-level labels for soundtrack classification.The broader goal is to help categorize, organize, and index video through machine hearing.
- Research questions: The study compares fully connected DNNs with AlexNet, VGG, Inception, and ResNet on video soundtrack classification.It also asks whether trained models transfer to Acoustic Event Detection.
- Related work: Prior AED datasets were much smaller than YouTube-100M, motivating evaluation of models with large capacity on a very large dataset.Earlier AED systems used MFCCs and classifiers including GMMs, HMMs, NMF, SVMs, CNNs, and RNNs.
- Contribution: The paper compares several architectures and reports first published applications of Inception and ResNet networks to audio, to the authors’ knowledge.This extends comparisons beyond the VGG-based system used in prior acoustic-scene classification work.
- Method: Whole-soundtrack decisions are formed by applying a classifier to non-overlapping segments and averaging the segment-level output sets.This follows the finding that simple averaging of frame-level CNN outputs performed nearly as well as more complex temporal-combination models.
- Weak supervision: The study investigates weak video-level labels despite many segments being uninformative, while acknowledging that label weakness and segment relevance are difficult to quantify.For some classes, such as Beach, background ambience may itself provide informative evidence.
- Representation: Audio is represented as image-like log-mel spectrogram patches and processed with minimally altered image-classification networks including Inception-V3 and ResNet-50.Training subsets span 23K to 70M videos to assess training-set size.
2. DATASET
YouTube-100M is a large, automatically labeled collection of YouTube videos whose labels cover broad and specific topics but are not perfectly accurate. The dataset supports soundtrack classification at substantial scale.
- Dataset composition: YouTube-100M consists of 100 million videos, including 70M training videos, 10M evaluation videos, and 20M validation videos.The training split totals 5.4M hours, with videos averaging 4.6 minutes.
- Examples: Table 1 provides example labels drawn from the 30K label set.The examples illustrate the dataset’s label vocabulary rather than a model result.
- Labels: The 30K label vocabulary contains 30,871 Knowledge Graph topic identifiers, with videos averaging around five labels each.Labels range from generic topics such as Song to specific topics such as Cormorant.
- Label limitations: Machine-generated labels are not 100% accurate and vary in acoustic relevance, including acoustically relevant labels such as Trumpet and less relevant labels such as Web Page.Videos may also receive multiple labels with different degrees of specificity.
3.1. Training
The training pipeline converts audio into log-mel spectrogram patches and trains classifiers on inherited video-level labels assigned to short, non-overlapping frames.
- Framing: Audio is divided into non-overlapping 960 ms frames, producing approximately 20 billion training examples from the 70M videos.Each frame inherits all labels assigned to its parent video.
- Spectrogram construction: Each frame is transformed with a short-time Fourier transform using 25 ms windows every 10 ms, then integrated into 64 mel-spaced frequency bins.A small offset is added before log transformation to avoid numerical issues.
- Model input: The resulting 96 × 64 log-mel spectrogram patches are the inputs to all classifiers.These patches provide the common representation for the architecture comparisons.
3.2. Evaluation
Evaluation aggregates frame-level predictions into video-level scores and reports balanced AUC, equivalent d-prime, and mAP across classes. Multiple balanced evaluation sets vary label-vocabulary size.
- Evaluation sets: Three balanced evaluation sets contain roughly 33 examples per class for 30K, 3K, and 400 labels.Their sizes are 1M, 100K, and 12K videos, respectively.
- Score aggregation: Each 960 ms frame is passed through the classifier, and classifier output scores are averaged across all segments in a video.This produces video-level predictions from frame-level classification outputs.
- Metrics: Performance is measured with balanced average AUC, equivalent d-prime class separation, and mean Average Precision across classes.AUC summarizes ROC discrimination, while mAP averages Average Precision across classes.
- Metric interpretation: Perfect classification corresponds to AUC 1.0 and infinite d-prime, whereas random guessing corresponds to AUC 0.5 and d-prime zero.mAP is affected by class priors, which are below 10^-4 for most classes.
3.3. Architectures
The paper compares a fully connected DNN with audio-adapted versions of AlexNet, VGG, Inception, and ResNet, using architectures modeled on successful image classifiers. The models differ in their audio-specific input adaptations, computational sizes, and training configurations.
- Architectures: The baseline experiments compare a fully connected DNN with several networks closely modeled on successful image classifiers.Baseline experiments use the 3K most frequent labels from the original 30K-label vocabulary.
- AlexNet: The AlexNet audio variant changes the initial convolution stride, replaces local response normalization with batch normalization, and uses a 3087-unit final layer.Its input is 96 × 64, compared with the original 224 × 224 × 3 input.
- VGG: The VGG audio variant changes the final layer to 3087 sigmoid units and uses batch normalization instead of local response normalization.Reducing the initial strides did not improve performance in this setup, and parallelizing beyond 10 GPUs did not help significantly.
- Inception: The Inception V3 audio variant truncates the stem, removes the auxiliary network, and changes average pooling to 10 × 6.The audio variant has 28M weights and 4.7B multiplies, versus 27M weights and 5.6B multiplies for the original network.
- ResNet: The ResNet-50 audio variant removes the first convolution's stride of 2 and changes average pooling to 6 × 4.The audio variant has 30M weights and 1.9B multiplies, versus 26M weights and 3.8B multiplies for the original network.
4. EXPERIMENTS
The experiments compare audio classification architectures and vary label vocabulary and training-set size. Inception and ResNet perform best, broader label sets offer modest gains, and very large datasets provide limited additional improvement beyond 700K–7M videos, while ResNet embeddings substantially improve Audio Set AED.
- 4.1. Architecture Comparison: Inception and ResNet achieve the best performance among the evaluated architectures, and all CNNs outperform the fully connected baseline.Their capacity and convolutional units capture structures across the audio representation.
- 4.1. Architecture Comparison: d-prime remains centered near 2.0 across label prior, while its variance increases for less-common classes.The analysis covers a random 20% subset of the 30K classes.
- 4.2. Label Set Size: Performance improves slightly with larger training label vocabularies, although models without the 128-unit bottleneck perform better overall.The comparison evaluates 400 labels using ResNet-50 variants trained on 70M videos.
- 4.3. Training Set Size: 700K, 7M, and 70M training-video models are mostly close in performance, whereas 70K and 23K models perform worse and likely overfit.The experiment uses the same ResNet-50 architecture with a 3K-label set.
- 4.4. AED with the Audio Set Dataset: ResNet embeddings raise Audio Set AED performance from 0.137 to 0.314 mAP and from 0.904 to 0.959 AUC relative to log-mel inputs.The corresponding d-prime values are 1.846 for the log-mel baseline and 2.452 for embeddings.
5. CONCLUSIONS
The paper finds that image-classification CNNs work well for audio, with larger label vocabularies yielding modest gains and larger training sets helping up to a point. ResNet embeddings substantially improve AED over the baseline on Audio Set, while segment-level outputs also distinguish changing sound sources.
- 5. CONCLUSIONS: State-of-the-art image networks achieve excellent audio-classification results compared with a fully connected network and earlier image architectures.
- 5. CONCLUSIONS: Larger label vocabularies modestly improve performance when evaluating smaller label sets.
- 5. CONCLUSIONS: Increasing training videos improves performance up to 7M, while regularization might reduce the gap for smaller datasets.
- 5. CONCLUSIONS: ResNet embeddings produce a significant increase over the baseline for AED on Audio Set.
- 5. CONCLUSIONS: Frame-by-frame classifier outputs clearly distinguish different sound sources at different points in a video.