Source-linked AI summary
Very Deep Convolutional Neural Networks for Raw Waveforms
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, Samarjit Das
TL;DR
Raw-waveform acoustic modeling is challenging because prior waveform CNNs were generally shallow and may not learn sufficiently discriminative high-level features. The paper proposes very deep fully convolutional CNNs with tailored receptive fields, batch normalization, residual learning, and down-sampling. On environmental sound recognition, the 18-layer model improves absolute accuracy over a 3-layer model by 15.56% and reaches performance competitive with log-mel spectrogram CNNs.
Problem
Prior raw-waveform CNNs generally used only a few convolutional layers, while learning discriminative acoustic representations directly from long waveforms remained challenging.
Method
The paper proposes fully convolutional waveform CNNs with up to 34 weight layers, using a broad first receptive field, narrow later receptive fields, batch normalization, residual learning, and down-sampling.
Results
15.56% absolute accuracy improvement over 2-convolutional-layer networks was achieved, while the 18-layer model reached 71.8% accuracy competitive with log-mel spectrogram CNNs.
Takeaways & Limitations
Deeper fully convolutional architectures substantially improve raw-waveform acoustic modeling and can achieve performance competitive with CNNs using log-mel spectrogram inputs.
Takeaways & Limitations
The dataset was too small to train M34-res without further regularization, leading to overfitting despite 99.21% training accuracy and 63.47% test accuracy.
Abstract
from arXiv · showhide
Learning acoustic models directly from the raw waveform data with minimal processing is challenging. Current waveform-based models have generally used very few (~2) convolutional layers, which might be insufficient for building high-level discriminative features. In this work, we propose very deep convolutional neural networks (CNNs) that directly use time-domain waveforms as inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e.g., vector of size 32000), necessary for processing acoustic waveforms. This is achieved through batch normalization, residual learning, and a careful design of down-sampling in the initial layers. Our networks are fully convolutional, without the use of fully connected layers and dropout, to maximize representation learning. We use a large receptive field in the first convolutional layer to mimic bandpass filters, but very small receptive fields subsequently to control the model capacity. We demonstrate the performance gains with the deeper models. Our evaluation shows that the CNN with 18 weight layers outperform the CNN with 3 weight layers by over 15% in absolute accuracy for an environmental sound recognition task and matches the performance of models using log-mel features.
1. INTRODUCTION
Deep CNNs can jointly learn feature representations and predictions, but applying them directly to raw audio remains challenging because prior waveform models were usually shallow. This work addresses that gap with very deep waveform CNNs and reports substantial gains on environmental sound recognition.
- CNNs learn temporally invariant features directly from time-domain waveforms while jointly optimizing representation and classification.
- Prior waveform-based audio CNNs generally used only about two convolutional layers, limiting their depth relative to successful vision CNNs.
- The proposed architectures use up to 34 weight layers and process long raw-waveform sequences efficiently.
- 15% absolute accuracy improvement was achieved over 2-convolutional-layer networks on environmental sound recognition.
- Raw-signal deep models achieved performance competitive with log-mel-feature models, including a reported parity result for environmental sound recognition.
2. VERY DEEP CONVOLUTIONAL NETWORKS
The proposed waveform CNNs combine deep stacks with small later receptive fields, a broad first receptive field, batch normalization, residual learning, and aggressive down-sampling. These choices target efficient computation and trainability for long raw-audio inputs.
- Deep architectures: The architectures use receptive field 3 in later 1D convolutional layers to control parameters and computation as depth increases.
- Deep architectures: The first two layers reduce temporal resolution by 16x through large convolutional and max-pooling strides, limiting computation in subsequent layers.
- Deep architectures: Feature-map counts double after temporal resolution decreases, providing increased channel capacity at lower resolutions.
- Batch Normalization: Batch normalization normalizes convolution outputs before ReLU and alleviates exploding or vanishing gradients during deep-network optimization.
- Residual Learning: Residual learning trains blocks to approximate F(x) = H(x) − x through skip connections, and is applied in M34-res.
3. EXPERIMENT DETAILS
The experiments evaluate raw-waveform CNNs on UrbanSound8k using standardized 8 kHz audio, with a held-out official fold and no data augmentation. Models are trained from scratch with Adam and regularization.
- Dataset and preprocessing: UrbanSound8k contains 8,732 clips of 10 urban environmental sounds, totaling 9.7 hours, with official fold 10 used for testing.
- Dataset and preprocessing: Audio waveforms are down-sampled to 8 kHz, standardized to mean 0 and variance 1, and used without data augmentation.
- Architectures: The proposed architectures are fully convolutional and omit fully connected layers and dropout; convolutional layers are followed by batch normalization.
- Training: The models are trained for 100–400 epochs with Adam, scratch initialization, Glorot initialization, and ℓ2 regularization coefficient 0.0001.
4. RESULTS AND ANALYSES
On UrbanSound8k, deeper raw-waveform CNNs substantially outperform shallow models, while batch normalization, receptive-field design, and fully convolutional architectures affect optimization and generalization.
- Network depth: M3 performs poorly, and increasing its filters by 50% does not significantly improve accuracy, indicating limited shallow-model capacity.The comparison contrasts M3 with M3-big, which has 2.5x more parameters.
- Network depth: M18 reaches 71.68% test accuracy, outperforming M3 by 15.56% in absolute accuracy.Performance increases with depth through M18, while M34-res reaches 63.47% because of overfitting.
- Training efficiency: Using stride 1 instead of 4 in M11 increases training time 3.5x and lowers test accuracy to 67.37%, whereas M18 reaches 68.42% in 2 hours.Aggressive initial down-sampling makes very deep networks more economical to train.
- Fully convolutional design: Adding fully connected layers increases parameters and training time by 2–95% without improving test accuracy, and can reduce generalization.The fully convolutional design therefore avoids the added fully connected layers in the tested variants.
- Receptive-field analysis: A first-layer receptive field of 8 or 320 degrades performance by up to 6.6% relative to receptive field 80.Small fields provide insufficient frequency resolution, while large fields smooth local structures and miss local impulse patterns.
- Batch normalization: Batch normalization supports optimization of deep networks: M34-no-bn remains near random-guess accuracy after 159 epochs, while M18-no-bn has lower test accuracy.The results also indicate a regularization effect from batch normalization.
- Receptive-field analysis: All tested first-layer variants learn bandpass-filter banks, but small receptive fields disperse frequency bands and large ones lack sufficient high-frequency filters.The standard M18 variant has well-distributed filters across frequencies.
5. CONCLUSION
Very deep fully convolutional CNNs can process raw acoustic waveforms efficiently and achieve strong environmental sound recognition performance without conventional input features.
- 18 weight layers achieve 71.8% accuracy and outperform 2-convolutional-layer networks by 15.56% absolute accuracy.The result is competitive with CNNs using log-mel spectrogram inputs.
- The proposed architectures operate directly on acoustic waveform inputs and scale to networks with up to 34 weight layers.They combine batch normalization, residual learning, and down-sampling for efficient optimization.
- The networks use a broad first-layer receptive field and narrow receptive fields thereafter.The first layer is intended to mimic bandpass filters while later layers control model capacity.
- Fully convolutional networks compare favorably with architectures containing fully connected layers.