Source-linked AI summary
Very Deep Multilingual Convolutional Neural Networks for LVCSR
Tom Sercu, Christian Puhrsch, Brian Kingsbury, Yann LeCun
TL;DR
The paper addresses the limited depth and architectural development of CNNs for LVCSR. It proposes very deep, multilingual, and multi-scale CNNs, reporting improvements on low-resource Babel and Switchboard benchmarks, while identifying further gains expected from additional training and architectural techniques.
Problem
CNNs for LVCSR had not kept pace with advances in other domains where deeper neural networks performed better.
Method
The paper introduces very deep CNNs with small kernels, multilingual CNNs with multiple untied layers, and multi-scale input features for LVCSR.
Results
The proposed architectures improved WER over DNN baselines on Babel and achieved 11.8% WER on Switchboard 300 after cross-entropy training.
Takeaways & Limitations
Very deep and multilingual CNN architectures proved valuable for low-resource speech recognition and improved performance across the evaluated LVCSR tasks.
Takeaways & Limitations
The authors expect additional gains from sequence training, joint DNN training, annealed dropout, and maxout nonlinearities.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) are a standard component of many current state-of-the-art Large Vocabulary Continuous Speech Recognition (LVCSR) systems. However, CNNs in LVCSR have not kept pace with recent advances in other domains where deeper neural networks provide superior performance. In this paper we propose a number of architectural advances in CNNs for LVCSR. First, we introduce a very deep convolutional network architecture with up to 14 weight layers. There are multiple convolutional layers before each pooling layer, with small 3x3 kernels, inspired by the VGG Imagenet 2014 architecture. Then, we introduce multilingual CNNs with multiple untied layers. Finally, we introduce multi-scale input features aimed at exploiting more context at negligible computational cost. We evaluate the improvements first on a Babel task for low resource speech recognition, obtaining an absolute 5.77% WER improvement over the baseline PLP DNN by training our CNN on the combined data of six different languages. We then evaluate the very deep CNNs on the Hub5'00 benchmark (using the 262 hours of SWB-1 training data) achieving a word error rate of 11.8% after cross-entropy training, a 1.4% WER improvement (10.6% relative) over the best published CNN result so far.
1. INTRODUCTION
The paper addresses the limited depth of LVCSR CNNs by adapting advances from vision and extending CNNs to multilingual and multi-scale speech modeling. It frames multilingual training as especially valuable for low-resource recognition.
- Motivation: LVCSR CNNs had not kept pace with deeper architectures that improved performance in other domains.The work is motivated by the contrast between CNN progress in natural-data tasks and the shallower CNNs traditionally used in LVCSR.
- Related architectural advances: VGG-inspired stacks of small 3×3 kernels provide the architectural basis for very deep LVCSR CNNs.The VGG design replaces large kernels with stacked 3×3 convolutions and ReLU nonlinearities, enabling networks with many weight layers.
- Multilingual modeling: Multilingual CNNs aim to leverage training data from languages other than the target language in low-resource speech recognition.The paper extends multilingual neural-network ideas to CNNs, an extension the authors identify as previously unpublished.
- Multi-scale modeling: Multi-scale features aim to exploit more acoustic context at very low computational cost.The approach is connected to multi-scale modeling in vision and to earlier multi-scale explorations in tandem and CLDNN systems.
2. ARCHITECTURAL AND TRAINING NOVELTIES
The proposed system combines very deep 3×3-kernel CNNs, multilingual parameter sharing with selective untied layers, and multi-scale inputs. Training uses adaptive gradient methods and data balancing, while multi-scale training yields only small gains.
- Very Deep Convolutional Networks: The deepest WDX configuration has 14 weight layers, comprising 10 convolutional and 4 fully connected layers.The architectures increase in depth across configurations, and non-classic networks use 3×3 convolutional kernels.
- Multilingual Convolutional Networks: Multilingual VBX networks share convolutional lower layers while using language-specific weights and biases in multiple higher fully connected layers.The lowest fully connected layer remains shared because it contains most parameters and functions with the shared convolutional stages as a multilingual feature extractor.
- Multilingual Convolutional Networks: Untying all fully connected layers except the lowest one gave optimal performance across architectures, whereas untying the first fully connected layer caused strong degradation.The higher fully connected layers act as language-specific classifiers.
- Multi-scale feature maps: Multi-scale feature maps add larger contextual views through downsampling with different strides, while requiring only a small increase in computation and parameters.The first convolution combines information across scales because the modified configuration mainly adds input feature maps.
- Multi-scale feature maps: Multi-scale training produced small gains, while increasing context size had a stronger positive impact at higher computational cost.The result distinguishes the modest effect of multi-scale training from the larger but more expensive effect of broader context.
- Training: Adadelta and Adam were used for initial training, and Babel data balancing progressed from balanced sampling toward natural target frequencies.For Switchboard, the balancing exponent was varied between 0.4 and 0.8, with decoding priors adjusted to the final sampling distribution.
3. EXPERIMENTAL RESULTS
Experiments on Babel and Switchboard show that deeper, multilingual, and multi-scale CNN designs improve LVCSR performance, with optimization strategy affecting convergence and final WER.
- Babel: 2.5 WER improvement was achieved by the VC architecture over the baseline DNN with one language and 3 hours of Babel data.
- Babel: Multi-scale features provided a modest gain, while larger context improved WER at computational cost proportional to convolutional context size.
- Switchboard 300: 14.5% relative improvement over the classical CNN baseline and 10.6% relative improvement over the published CNN were reported after cross-entropy training.
- Switchboard 300: 11.8% WER was achieved on Hub5’00 SWB after training on 262 hours of SWB-1 data with SGD from random initialization.
- Switchboard 300: Adadelta or Adam followed by SGD reached 12.2% WER after 140M frames, whereas SGD alone reached 11.8% WER in 3.5 passes.
4. DISCUSSION
The paper presents architectural advances for LVCSR CNNs, including very deep, multilingual, and multi-scale designs, and reports gains across low-resource and Switchboard evaluations.
- The paper introduces very deep CNNs with small 3×3 kernels, multilingual CNNs, and multi-scale input features for LVCSR.
- The best model has 14 weight layers and multi-scale features target more acoustic context with minimal computational increase.
- The paper reports 2.50% WER improvement with 3 hours of data, 5.77% with six languages and 18 hours, and 11.8% WER on Hub5’00.
- The authors expect additional gains from sequence training, joint DNN training, annealed dropout, and maxout nonlinearities.