PaperlayerView exact revision on arXiv

Source-linked AI summary

Very Deep Convolutional Neural Networks for Raw Waveforms

Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, Samarjit Das

arXiv:1610.00087v1cs.SDcs.LGcs.NE

TL;DR

Raw-waveform acoustic modeling has typically used few convolutional layers, leaving the effectiveness of deeper CNNs for learning discriminative features uncertain. This paper studies very deep CNNs that operate directly on waveforms and reports a 15.56% absolute accuracy gain over networks with two convolutional layers on environmental sound recognition.

  • Problem

    Raw-waveform acoustic modeling has typically used few convolutional layers, leaving the effectiveness of deeper CNNs for learning discriminative features uncertain.

    Source: §4 · p. 3

  • Method

    The paper studies fully convolutional CNNs up to 34 weight layers that take time-domain waveforms directly, using batch normalization, residual learning, and designed down-sampling.

    Source: §1 · p. 1

  • Results

    15.56% in absolute accuracy over networks with 2 convolutional layers, while the 18-weight-layer model reaches 71.8% accuracy and is competitive with CNNs using log-mel spectrogram inputs.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The results show that deeper architectures substantially improve acoustic modeling using waveforms for environmental sound recognition.

    Source: §4 · p. 3

    The authors believe their dataset is too small to train the M34-res model without further regularization.

    Source: §4 · p. 4

Abstract

from arXiv · show

Learning acoustic models directly from the raw waveform data with minimal processing is challenging. Current waveform-based models have generally used very few (~2) convolutional layers, which might be insufficient for building high-level discriminative features. In this work, we propose very deep convolutional neural networks (CNNs) that directly use time-domain waveforms as inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e.g., vector of size 32000), necessary for processing acoustic waveforms. This is achieved through batch normalization, residual learning, and a careful design of down-sampling in the initial layers. Our networks are fully convolutional, without the use of fully connected layers and dropout, to maximize representation learning. We use a large receptive field in the first convolutional layer to mimic bandpass filters, but very small receptive fields subsequently to control the model capacity. We demonstrate the performance gains with the deeper models. Our evaluation shows that the CNN with 18 weight layers outperform the CNN with 3 weight layers by over 15% in absolute accuracy for an environmental sound recognition task and matches the performance of models using log-mel features.

1. INTRODUCTION

The introduction presents very deep, fully convolutional networks that learn directly from raw audio waveforms, using design and optimization choices for long sequences. It reports improved environmental sound accuracy over shallow waveform CNNs and parity with log-mel systems.

1. INTRODUCTION

  • Jointly learning representations and predictions has been effective in speech recognition [2] and image classification, while designing and training deep CNNs for new domains remains challenging.CNNs can learn spatially or temporally invariant features, and successful vision models commonly have tens or more than a hundred layers.Source: §1 · p. 1
  • Prior waveform CNNs performed well, including matching log-mel features, but generally used only a few convolutional layers, such as two.Source: §1 · p. 1
  • The proposed raw-waveform CNNs have up to 34 weight layers and use batch normalization, residual learning, and careful down-sampling to train efficiently on long sequences.They use a large first-layer receptive field to mimic bandpass filters, smaller receptive fields later, and a fully convolutional design without fully connected layers or dropout; they can process varying audio lengths.Source: §1 · p. 1
  • Over 15% in absolute accuracy, deep networks improve on networks with 2 convolutional layers for environmental sound recognition.The paper also reports parity between raw time-signal and log-mel models, describing it as the first such report for environmental sound recognition.Source: §1 · pp. 1–2

2. VERY DEEP CONVOLUTIONAL NETWORKS

The paper describes five waveform-input architectures designed to support deep, efficient learning through small later-layer filters, downsampling, and fully convolutional classification. Batch normalization and residual connections enable very deep models, while the first-layer receptive field is set to 10 ms.

2. VERY DEEP CONVOLUTIONAL NETWORKS

  • The networks take long 1D waveform vectors as input and use 3-sample receptive fields after the first convolution, reducing temporal resolution 16× in the first two layers while doubling feature maps thereafter.ReLU activations are used to reduce computation cost [15].Source: §2 · p. 2
  • Fully convolutional models replace high-dimensional fully connected layers with global average pooling, and perform comparably or better than counterparts with fully connected layers.The design is intended to make the convolutional layers learn the representation; the paper reports this comparison in its evaluation.Source: §2 · p. 2
  • The first convolution spans 10 milliseconds, and the authors report that substantially smaller or larger first-layer receptive fields perform poorly.The duration-based choice is intended to remain meaningful across sampling rates and is similar to the window size used for many MFCC computations.Source: §2 · p. 2
  • Batch normalization before ReLU keeps gradients well-behaved, enabling training of the very deep M18 and M34-res networks.The method normalizes each convolutional layer's output to alleviate exploding and vanishing gradients [12].Source: §2 · p. 2
  • M34-res uses residual blocks with skip connections, training layers to approximate the residual mapping F(x) = H(x) − x instead of directly fitting H(x).Residual learning is presented as a framework for easing the training of very deep networks [6].Source: §2 · p. 2

3. EXPERIMENT DETAILS

Experiments use UrbanSound8k with a fixed test fold and standardized, down-sampled waveforms; CNNs are trained from scratch, with additional variants used for analysis.

EXPERIMENT DETAILS

  • UrbanSound8k contains 8,732 clips across 10 urban sounds; fold 10 is the test set, and the remaining folds are used for training and validation.Waveforms are down-sampled to 8 kHz and standardized to mean 0 and variance 1; training data are shuffled without augmentation.Source: §3 · p. 2
  • Models are trained with Adam [20] for 100–400 epochs until convergence, using scratch initialization, Glorot initialization, and ℓ2 regularization at 0.0001.The models are implemented in TensorFlow [22] and trained on Titan X GPU machines.Source: §3 · pp. 2–3
  • The proposed waveform architectures use batch normalization after every convolution and omit fully connected layers and dropout.Table 1 describes their convolutional and pooling configurations, including residual blocks only in M34-res.Source: Table 1 · p. 3
  • Analysis variants add two 1,000-dimensional fully connected layers with dropout, or increase M3 and M5 filter counts by 50% and 100%, respectively.The fully connected variants also add batch normalization after each fully connected layer and have substantially more parameters than the original models.Source: §3 · p. 3

4. RESULTS AND ANALYSES

Very deep waveform CNNs improve sound-recognition accuracy over shallow models, but gains depend on suitable receptive fields, batch normalization, and avoiding overfitting. The results also show that fully connected layers do not improve test accuracy.

4. RESULTS AND ANALYSES

  • M3 performed poorly, and adding 50% more filters and 2.5× more parameters in M3-big did not significantly improve performance.The results indicate that shallow two-layer CNNs have limited capacity to extract discriminative features from raw waveforms, even with a larger model.Source: §4 · p. 3, Table 4 · p. 4
  • 71.68% test accuracy was achieved by M18, which outperformed M3 by as much as 15.56 percentage points.M18 was reported as competitive with spectrogram-based CNNs, though the evaluations differed: this work used the 10th fold and 8 kHz audio, while used 10-fold evaluation and 44.1 kHz audio.Source: Table 2 · p. 3, §4 · p. 3
  • M11 with stride 1 in its first convolution achieved 67.37% test accuracy, versus M18’s 68.42%, while taking 3.5× longer to train.The stride-1 M11 took 10 hours, compared with 2 hours for M18, illustrating the training efficiency of aggressive initial down-sampling.Source: §4 · pp. 3–4
  • M34-res reached 99.21% training accuracy but only 63.47% test accuracy, which the authors attribute to overfitting.The authors suggest the dataset is too small to train M34-res without further regularization; it nevertheless outperformed M3 and M5.Source: §4 · p. 4
  • Fully connected layers increased parameter counts and training time by 2–95% but did not improve test accuracy, and reduced it for M3-fc and M11-fc.The authors suggest that omitting these layers pushes learning into the convolutional layers, improving representation and generalization.Source: §4 · p. 4, Table 5 · p. 4
  • First-layer receptive fields of 8 or 320 degraded performance by up to 6.6% relative to receptive field 80.The learned kernels were bandpass filters: small receptive fields produced more dispersed bands, while large ones lacked sufficient high-frequency filters to respond to local impulses.Source: §4 · p. 4, Figure 2 · p. 4
  • Without batch normalization, M34 could not be optimized and remained near the 10% random-guess accuracy after 159 epochs.M18 without batch normalization reached high training accuracy but lower test accuracy, suggesting batch normalization also has a regularizing effect.Source: §4 · p. 4, Table 6 · p. 4

5. CONCLUSION

The conclusion presents very deep waveform-input CNNs as efficient to optimize and reports improved environmental sound recognition accuracy, competitive with log-mel CNNs. It also highlights fully convolutional architectures as favorable to fully connected alternatives and promising for speech and time-series modeling.

5. CONCLUSION

  • 71.8% accuracy and a 15.56% absolute accuracy gain over networks with 2 convolutional layers make the 18-layer network competitive with log-mel CNNs.The networks compare favorably with models using fully connected layers.Source: §5 · p. 4
  • Networks with up to 34 weight layers operate directly on acoustic waveforms and are efficiently optimized using batch normalization, residual learning, and down-sampling.The architecture uses a broad receptive field in the first convolutional layer and narrow receptive fields in the rest.Source: §5 · p. 4
  • The fully convolutional networks compare favorably with networks using fully connected layers and may improve CNNs for speech recognition and other time-series modeling.Source: §5 · p. 4

Related papers

Loading 1610.00087v1…