PaperlayerView exact revision on arXiv

Source-linked AI summary

ESResNet: Environmental Sound Classification Based on Visual Domain Models

Andrey Guzhov, Federico Raue, Jörn Hees, Andreas Dengel

arXiv:2004.07301v1cs.CVcs.LGcs.SDeess.AS

TL;DR

ESC approaches often rely on domain-specific features and architectures, and several reported UrbanSound8K results use splits that are not directly comparable. The paper combines log-power STFT spectrograms with a visual-domain CNN, extending it with stereo processing and attention. ESResNet-Attention achieved 91.50%, 97.00%, and 84.21% on ESC-50, ESC-10, and US8K, respectively; stereo input plus attention raised US8K accuracy to 85.42%.

  • Problem

    ESC approaches often rely on domain-specific features and architectures, and several reported UrbanSound8K results use splits that are not directly comparable.

    Source: §I · p. 1, E. Comparability of Results on the… · p. 2

  • Method

    The paper combines log-power STFT spectrograms with a visual-domain CNN, extending it with stereo processing and attention.

    Source: C. Spectrogram · p. 3, §VI · p. 7

  • Results

    ESResNet-Attention achieved 91.50%, 97.00%, and 84.21% on ESC-50, ESC-10, and US8K, respectively; stereo input plus attention raised US8K accuracy to 85.42%.

    Source: C. Attention-boosted vs. Vanilla · p. 7

  • Takeaways & Limitations

    The authors distinguish prior US8K results by official versus unofficial splits and provide code, including re-implementations, for reproducibility.

    Source: §VI · p. 7

    The model uses fixed feature extraction; learning time-frequency representations remains future work.

    Source: §VI · p. 7

Abstract

from arXiv · show

Environmental Sound Classification (ESC) is an active research area in the audio domain and has seen a lot of progress in the past years. However, many of the existing approaches achieve high accuracy by relying on domain-specific features and architectures, making it harder to benefit from advances in other fields (e.g., the image domain). Additionally, some of the past successes have been attributed to a discrepancy of how results are evaluated (i.e., on unofficial splits of the UrbanSound8K (US8K) dataset), distorting the overall progression of the field. The contribution of this paper is twofold. First, we present a model that is inherently compatible with mono and stereo sound inputs. Our model is based on simple log-power Short-Time Fourier Transform (STFT) spectrograms and combines them with several well-known approaches from the image domain (i.e., ResNet, Siamese-like networks and attention). We investigate the influence of cross-domain pre-training, architectural changes, and evaluate our model on standard datasets. We find that our model out-performs all previously known approaches in a fair comparison by achieving accuracies of 97.0 % (ESC-10), 91.5 % (ESC-50) and 84.2 % / 85.4 % (US8K mono / stereo). Second, we provide a comprehensive overview of the actual state of the field, by differentiating several previously reported results on the US8K dataset between official or unofficial splits. For better reproducibility, our code (including any re-implementations) is made available.

I. INTRODUCTION

Environmental sound classification (ESC) distinguishes everyday sound classes and has applications including multimedia retrieval and automated urban-sound analysis. The paper examines whether an image-domain model can address ESC amid audio-specific architectures and evaluation comparability concerns.

I. INTRODUCTION

  • ESC classifies everyday sounds, with potential applications in multimedia retrieval and automated analysis of urban sounds.The cited examples include improving video retrieval through audio and offering more detailed insights about high noise levels.Source: §I · p. 1
  • Audio-specific architectures and engineered features make it harder to benefit from advances in fields such as computer vision.The authors investigate how a state-of-the-art image-domain approach performs on ESC.Source: §I · p. 1

A. Raw Waveform and 1D-CNN

  • Raw-waveform 1D-CNNs process audio without preprocessing, while gammatone-filterbank initialization was reported to improve results over random initialization.EnvNet v1/v2 and later multiscale 1D-CNNs are examples of this model group, [6],.Source: A. Raw Waveform and 1D-CNN · p. 2
  • The paper instead uses a fixed, wide-spectrum time-frequency transform to reduce complexity and trainable parameters with limited training data.The authors identify learning time-frequency representations as a possible future direction.Source: A. Raw Waveform and 1D-CNN · p. 2

B. Learnable Filterbanks and 2D-CNN

Prior ESC work includes learnable feature extraction and fixed time-frequency inputs for 2D-CNNs. This paper uses a single log-power spectrogram feature and discusses data augmentation as a way to increase training-data variability.

B. Learnable Filterbanks and 2D-CNN

  • ConvRBM feature extraction paired with a CNN classifier is presented as an alternative to fixed feature extraction,.The cited work splits the model into feature extraction and classification components.Source: B. Learnable Filterbanks and 2D-CNN · p. 2

C. Pre-computed Time-Frequency Representation and 2D-CNN

  • Earlier 2D-CNN work spans spectrogram and other precomputed representations, including MFCC, CRP, TEO, filterbank energies, gammatone spectrograms, chromagrams, spectral contrast, and Tonnetz –.The paper points to Table I for a detailed description of features used in those studies.Source: C. Pre-computed Time-Frequency… · p. 2
  • The paper’s model uses a single-feature input: log-power spectrograms.The authors note that earlier time-frequency features were developed with computational complexity or compression in mind, while increased computational capacity may permit a full-range feature.Source: C. Pre-computed Time-Frequency… · p. 2

D. Data Augmentation

  • Data augmentation increases training-data variability and acts as a regularizer that prevents overfitting.The paper introduces audio transformations used to augment training data.Source: D. Data Augmentation · p. 2

1) Time Stretching:

The supplied passages describe time stretching, pitch shift, and time inversion as augmentation approaches, and separately flag UrbanSound8K split comparability as an evaluation concern. The proposed model combines log-power spectrograms with a visual-domain CNN.

1) Time Stretching:

2) Pitch Shift:

  • Pitch shift manipulates spectral characteristics while preserving track duration.Source: 2) Pitch Shift · p. 2

3) Time Inversion:

  • Time inversion is described as an effective augmentation related to randomly flipping images during visual-classification training [6].Source: 3) Time Inversion · p. 2

E. Comparability of Results on the UrbanSound8K (US8K) Dataset

  • UrbanSound8K results from several papers are not directly comparable because they used unofficial, non-standard, or custom data splits or snippet-generation strategies.The authors identify at least five such papers and direct readers to Table I and Section V-D for details.Source: E. Comparability of Results on the… · p. 2

III. MODEL

  • The proposed approach uses log-power spectrograms with a visual-domain convolutional neural network for environmental sound classification.The model section also describes attention, stereo audio, weight initialization, and spectrogram computation.Source: §III · p. 2

A. Residual Networks

ESResNet uses a ResNet-50 structure with skip connections, and an optional attention extension adds parallel branches that weight input features. The attention blocks target information across time and frequency.

A. Residual Networks

  • ResNet skip connections bypass layers and merge their inputs and outputs, with the stated motivation of preventing vanishing gradients.ESResNet is based on vanilla ResNet-50; Figure 1 depicts its main branch and residual-layer structure.Source: A. Residual Networks · p. 2, Figure 1 · p. 3

B. Attention

  • The attention branch processes feature maps alongside the main branch and applies logistic-function outputs; the first four blocks use depth-wise separable convolution.Figure 1 shows the optional attention blocks as a parallel extension of the main ESResNet branch.Source: B. Attention · p. 3, Figure 1 · p. 3
  • Attention blocks are added in parallel to the residual layers or average pooling, producing weights intended to focus on important time- and frequency-domain information.The first four blocks handle either frequency- or time-related information; the last handles a joint time-frequency representation.Source: B. Attention · p. 3, Figure 1 · p. 3

C. Spectrogram

The model represents audio with log-power spectrograms computed from the STFT. Its framing and windowing choices balance time and frequency resolution while reducing spectral perturbations.

C. Spectrogram

2) Window Function:

  • The minimum 4-term Blackman-Harris window reduces framing-related spectrum perturbations and offers reasonable bandwidth with very low spectral leakage.Its coefficients are a0 = 0.35875, a1 = 0.48829, a2 = 0.14128, and a3 = 0.01168.Source: 2) Window Function · p. 3
  • 37.5 ms frames with 66.1% overlap (24.8 ms) set the time-frequency resolution trade-off for the Blackman-Harris window.The paper uses the recommended overlap for this window.Source: 2) Window Function · p. 3

D. Input Channel Transformation

ESResNet adapts image-domain approaches to audio by transforming spectrograms into three channels and using a shared-weight architecture for stereo input. The paper also uses ImageNet pre-training as a weight initializer.

D. Input Channel Transformation

  • The spectrogram is split into lower, middle, and upper frequency bands to fill three channels instead of replicating grayscale input or adding zeros.The channels align respectively with the visual red, green, and blue channels.Source: D. Input Channel Transformation · p. 4, Figure 2 · p. 4

E. Handling Stereo Audio Using Siamese-like Architecture

F. ImageNet Training as Weight Initializer

  • The model uses weights from a network trained on ImageNet, which the authors report is beneficial for environmental sound classification.ImageNet training uses more than 1 million samples across 1000 classes.Source: F. ImageNet Training as Weight… · p. 4

IV. EXPERIMENTAL SETUP

The experimental setup evaluates the model on ESC-50, its ESC-10 subset, and UrbanSound8K using the datasets’ author-provided folds. For UrbanSound8K, the split must keep overlapping snippets from an original recording on the same side of evaluation.

1) ESC-50 / -10:

  • ESC-50 contains 2,000 mono recordings across 50 classes, and ESC-10 is a 10-class subset with the other dataset characteristics unchanged.ESC-50 is divided into five author-provided folds used for evaluation.Source: 1) ESC-50 / -10 · p. 4

2) UrbanSound8K:

  • UrbanSound8K contains 8,732 mono and stereo samples across 10 classes, and evaluation uses its 10 author-provided folds.Track lengths vary up to 4 seconds and native sample rates range from 16 kHz to 48 kHz.Source: 2) UrbanSound8K · p. 5
  • Because UrbanSound8K snippets overlap by 50%, official folds keep subsequent snippets from an original track out of both training and evaluation sets.Random shuffling can put shared portions of an original recording in both sets.Source: 2) UrbanSound8K · p. 5

3) Data Pre-processing:

The preprocessing and training setup standardizes audio for spectrogram computation, trains with Adam and a decaying learning rate, and applies time-based augmentations.

3) Data Pre-processing:

  • Audio samples are resampled to 44.1 kHz, framed at 1,654 samples, and reflection-padded to 2,048 samples for the FFT.The frame length follows the chosen 37.5 ms window.Source: 3) Data Pre-processing · p. 5

B. Model Training

  • Training uses Adam for 300 epochs with batch size 16, exponential learning-rate decay after warm-up, weight decay, and categorical cross-entropy.The learning rate starts at 0.00025, is ten times lower for five epochs, then warms up linearly for ten epochs before decaying with γ = 0.985.Source: B. Model Training · p. 5
  • Training augmentation applies time inversion with probability 0.5 and time scaling sampled uniformly from [1.25^-1, 1.25].The paper describes time scaling as combining time stretching and pitch shift.Source: B. Model Training · p. 5

C. Re-implementation

The section reports that the proposed approaches outperform prior approaches in a fair comparison, and that ImageNet pretraining improves ESResNet accuracy across the reported datasets.

C. Re-implementation

  • The authors re-implemented LMCNet because had no published code and TFNet because its temporarily available code enabled reproduction with minor changes.The TFNet repository later disappeared.Source: C. Re-implementation · p. 5
  • The unofficial US8K evaluation used a 10-fold StratifiedKFold split with one shared random seed across experiments.Source: C. Re-implementation · p. 5

V. RESULTS

  • The proposed approaches outperform previous approaches in a fair comparison.Table I presents evaluation accuracy results.Source: §V · p. 5, Table I · p. 6

A. ImageNet Weights vs. Random Weights

  • 90.80 % versus 81.15 % on ESC-50, 96.75 % versus 92.50 % on ESC-10, and 83.59 % versus 79.91 % on US8K for ESResNet after ImageNet pretraining.The largest relative change is on ESC-50, which the authors describe as challenging with restricted training samples.Source: A. ImageNet Weights vs. Random Weights · pp. 5–6

B. Stereo vs. Mono

Stereo input and attention improve the reported US8K and cross-dataset results, while reproductions of earlier results raise concerns about evaluation splits and reproducibility.

B. Stereo vs. Mono

  • 84.90 % on US8K with ImageNet-pretrained ESResNet stereo input exceeds 83.59 % for vanilla mono and 84.21 % for attention mono.From-scratch ESResNet reaches 81.31 % with stereo versus 79.91 % with mono; mono attention reaches 81.00 %.Source: B. Stereo vs. Mono · p. 7

C. Attention-boosted vs. Vanilla

  • 91.50 % on ESC-50, 97.00 % on ESC-10, and 84.21 % on US8K are reported for ESResNet-Attention; stereo plus attention reaches 85.42 % on US8K.The authors report that attention outperforms prior state-of-the-art results on all three datasets.Source: C. Attention-boosted vs. Vanilla · p. 7

D. Official and Unofficial Splits and Reproducibility Problems

  • Re-implemented LMCNet results lead the authors to attribute the reported results to evaluation without the official US8K split.Source: D. Official and Unofficial Splits and… · p. 7
  • TFNet reproduction produced lower accuracy than reported on ESC-50 and US8K, but higher results on a random unofficial US8K split, leaving the reported setup unresolved.The authors suggest either the shared code lacks crucial steps or the original experiments used neither the official nor a completely random unofficial split.Source: D. Official and Unofficial Splits and… · p. 7
  • Random-split US8K results are reported in parentheses because they are not a basis for fair comparison.The experiments roughly quantify the influence of random splitting.Source: D. Official and Unofficial Splits and… · p. 7

VI. CONCLUSION

ESResNet shows that a visual-domain model using log-power spectrograms can perform strongly on environmental sound classification, with attention and stereo processing improving reported configurations. The paper also emphasizes evaluation protocol and identifies learned representations and domain-transfer effects as future work.

VI. CONCLUSION

  • Attention-augmented ESResNet reached the highest accuracy and significantly outperformed prior state-of-the-art models in fair comparisons on ESC-50 and UrbanSound8K.Source: §VI · p. 7
  • With regular log-power spectrograms, ESResNet performed competitively with humans on ESC-50, while ImageNet pre-training allowed it to outperform current state-of-the-art methods.Source: §VI · p. 7
  • On UrbanSound8K, multiple input channels gave an additional performance gain with only minor Siamese-like architectural changes.Source: §VI · p. 7
  • The authors demonstrate that random splitting influences UrbanSound8K evaluation, distinguish official from unofficial results, and release code including re-implementations.Source: §VI · p. 7
  • Future work will investigate learned time-frequency representations and which classes benefit more or less from ImageNet transfer.Source: §VI · p. 7

Related papers

Loading 2004.07301v1…