Source-linked AI summary
Attention based Convolutional Recurrent Neural Network for Environmental Sound Classification
Zhichao Zhang, Shugong Xu, Tianhao Qiao, Shunqing Zhang, Shan Cao
TL;DR
Environmental sound classification must contend with silent and semantically irrelevant frames among the frames that carry sound-class information. The paper combines a convolutional recurrent network with frame-level attention, and reports improved classification accuracy on ESC-10 and ESC-50 compared with PiczakCNN.
Problem
ESC signals can contain silent or irrelevant frames, while only some frames contribute to sound-class representations.
Source: §1 · p. 2, §2.3 · p. 4Method
Results
ACRNN outperforms PiczakCNN by 13.2% on ESC-10 and 21.2% on ESC-50, and is reported to achieve state-of-the-art classification accuracy.
Source: §3.2 · p. 7, §4 · p. 10Takeaways & Limitations
The paper reports that attention on RNN layers obtains the highest accuracy, while attention on CNN layers does not always improve performance.
Source: §4 · p. 10CNN-layer attention does not always improve performance, a result the authors identify for future exploration.
Source: §4 · p. 10
Abstract
from arXiv · showhide
Environmental sound classification (ESC) is a challenging problem due to the complexity of sounds. The ESC performance is heavily dependent on the effectiveness of representative features extracted from the environmental sounds. However, ESC often suffers from the semantically irrelevant frames and silent frames. In order to deal with this, we employ a frame-level attention model to focus on the semantically relevant frames and salient frames. Specifically, we first propose an convolutional recurrent neural network to learn spectro-temporal features and temporal correlations. Then, we extend our convolutional RNN model with a frame-level attention mechanism to learn discriminative feature representations for ESC. Experiments were conducted on ESC-50 and ESC-10 datasets. Experimental results demonstrated the effectiveness of the proposed method and achieved the state-of-the-art performance in terms of classification accuracy.
1 Introduction
The introduction motivates frame-level attention for environmental sound classification, where silent and irrelevant frames complicate learning useful representations. It presents an attention-based convolutional RNN and reports experiments on ESC-10 and ESC-50.
1 Introduction
- ESC research spans dictionary learning, matrix factorization, GMMs, deep networks, and CNN–RNN combinations.Source: §1 · pp. 1–2
- Silent periods and semantically irrelevant frames matter because only a few intermittent frames may carry class-related sound information.The introduction argues that selecting relevant frames is important for ESC.Source: §1 · p. 2
- ACRNN uses frame-level attention to focus on semantically relevant frames and produce discriminative features for ESC.The paper explores attention for both CNN and RNN layers.Source: §1 · p. 2
- The proposed architecture extracts high-level features with a CNN, passes them to bidirectional GRUs, and combines the convolutional RNN with attention.Source: §1 · p. 2
- Experiments are conducted on the ESC-10 and ESC-50 datasets.Source: §1 · p. 3
2 Methods
The method combines Log-GT input features, a convolutional RNN, and frame-level attention for environmental sound classification. It also augments training data using time and pitch transformations and mixup.
2 Methods
- The ACRNN pipeline combines Log-GT features, a convolutional RNN, frame-level attention, and data augmentation for environmental sound classification.The method section introduces these stages as the proposed approach.Source: §2 · p. 3
2.1 Feature Extraction and Preprocessing
- Each input is a 128×128×2 Log-GT representation formed by concatenating a log-scaled 128-band Gammatone spectrogram with its temporal derivative.The spectrogram uses 23 ms STFT windows with 50% overlap; 128-frame segments also overlap by 50%.Source: §2.1 · p. 3
2.2 Architecture of Convolutional RNN
- CNN layers learn high-level features from Log-GTs, then bidirectional GRU layers model temporal correlations before a softmax classifier outputs class probabilities.The convolutional RNN has eight convolutional layers and two bidirectional GRU layers; the GRUs summarize temporal information.Source: §2.2 · p. 4
2.3 Frame-level Attention Mechanism
- Frame-level attention targets semantically relevant frames because silent or noisy frames may reduce robustness and increase misclassification.The figure illustrates relevant and silent or noisy frames for dog bark, baby cry, and clock tick.Source: §2.3 · p. 4, Figure 2 · p. 5
Attention for CNN layers:
- A 3×3 convolution and frequency average-pooling produce a softmax attention map that weights CNN features frame by frame.The attention vector is applied across frequency and channel dimensions of the CNN features.Source: Attention for CNN layers · p. 5
2.3 Frame-level Attention Mechanism
- For RNN attention, a one-layer MLP and softmax produce frame importance weights, which form a weighted feature vector for final classification.The weighted sum aggregates frame-level convolutional RNN features.Source: Attention for RNN layers · p. 5
2.4 Data Augmentation
- Training augmentation uses time stretch, pitch shift, and mixup, which combines two training features and their one-hot labels using a beta-distributed mix factor.Time-stretch factors are sampled from [0.8, 1.3], and pitch shifts from [-3.5, 3.5].Source: §2.4 · pp. 5–6
3 Experiments and Results
Experiments on ESC-10 and ESC-50 compare ACRNN with existing methods and examine the effects and placement of frame-level attention. ACRNN achieves the highest reported classification accuracy on both datasets, while attention results vary by layer.
3.1 Experiment Setup
- ESC-50 contains 2,000 recordings across 50 classes; ESC-10 is a 400-sample, 10-class subset, and models train for 300 epochs.The evaluation uses official-fold 5-fold cross-validation; the reported results are averages across folds.Source: §3.1 · p. 7, Table 1 · p. 7
3.2 Experiment Results
- ACRNN improves absolute accuracy over PiczakCNN by 13.2% on ESC-10 and 21.2% on ESC-50, and has the highest accuracy on both datasets.WaveMsNet and Multi-Stream CNN tie ACRNN on ESC-10 while using fused raw and spectrogram features; ACRNN uses spectrogram features alone.Source: Table 1 · p. 7, §3.2 · pp. 7–8
- The ESC-50 confusion matrix reports 86.1% average accuracy; church bells reach 100% recognition, while helicopter recognition is 52.5%, with 17.5% misclassified as airplane.The authors attribute the helicopter–airplane errors to similar sound characteristics.Source: Figure 4 · p. 8, §3.2 · p. 8
3.3 Effects of attention mechanism
- Attention significantly improves accuracy with or without augmentation, while augmentation alone adds 2.0% on ESC-10 and 4.8% on ESC-50.The augmentation combines time stretch, pitch shift, and mixup.Source: Table 2 · p. 9, §3.3 · p. 9
3.4 Where to apply attention
- Applying attention at RNN layer l10 gives the highest accuracy, with absolute gains of 0.7% on ESC-10 and 1.5% on ESC-50.For CNN layers, l2 improves accuracy on both datasets; other CNN layers can decrease ESC-50 accuracy.Source: Table 3 · p. 9, §3.4 · pp. 9–10
4 Conclusion
The paper proposes ACRNN for environmental sound classification and reports state-of-the-art classification accuracy on ESC-10 and ESC-50. Its results also show that attention placement matters: RNN-layer attention performs best, while CNN-layer attention does not always improve performance.
4 Conclusion
- ACRNN combines a convolutional recurrent neural network with frame-level attention for environmental sound classification.Source: §4 · p. 10
- Experiments on ESC-10 and ESC-50 demonstrate state-of-the-art classification accuracy for the proposed method.Source: §4 · p. 10
- Attention on RNN layers yields the highest accuracy, whereas applying it to CNN layers does not always improve performance.The authors identify CNN-layer attention as a topic for future work.Source: §4 · p. 10