Source-linked AI summary
Deep-FSMN for Large Vocabulary Continuous Speech Recognition
Shiliang Zhang, Ming Lei, Zhijie Yan, Lirong Dai
TL;DR
RNN training for sequential speech modeling can be computationally complex and vulnerable to gradient vanishing or exploding. The paper introduces Deep-FSMN, adding skip connections between memory blocks and combining the architecture with lower frame rate. Across large English and Mandarin tasks, DFSMN outperforms BLSTM-based systems, with over 20% relative improvement over LFR-LCBLSTM on the 20000-hour Mandarin task.
Abstract
from arXiv · showhide
In this paper, we present an improved feedforward sequential memory networks (FSMN) architecture, namely Deep-FSMN (DFSMN), by introducing skip connections between memory blocks in adjacent layers. These skip connections enable the information flow across different layers and thus alleviate the gradient vanishing problem when building very deep structure. As a result, DFSMN significantly benefits from these skip connections and deep structure. We have compared the performance of DFSMN to BLSTM both with and without lower frame rate (LFR) on several large speech recognition tasks, including English and Mandarin. Experimental results shown that DFSMN can consistently outperform BLSTM with dramatic gain, especially trained with LFR using CD-Phone as modeling units. In the 2000 hours Fisher (FSH) task, the proposed DFSMN can achieve a word error rate of 9.4% by purely using the cross-entropy criterion and decoding with a 3-gram language model, which achieves a 1.5% absolute improvement compared to the BLSTM. In a 20000 hours Mandarin recognition task, the LFR trained DFSMN can achieve more than 20% relative improvement compared to the LFR trained BLSTM. Moreover, we can easily design the lookahead filter order of the memory blocks in DFSMN to control the latency for real-time applications.
1. INTRODUCTION
RNNs model sequential data and LSTMs have outperformed FNNs in acoustic modeling, but recurrent training can be computationally demanding and vulnerable to gradient problems. The paper extends feedforward FSMNs with skip connections and evaluates Deep-FSMN on larger English and Mandarin speech-recognition tasks.
1. INTRODUCTION
- RNNs model sequential data over extended periods, and LSTM variants have been observed to outperform FNNs on acoustic-modeling tasks.FNNs map fixed-size inputs to fixed-size outputs, whereas RNNs can store memory in network weights.Source: §1 · p. 1
- RNN training commonly relies on BPTT, which increases computational complexity and may cause vanishing or exploding gradients.Source: §1 · p. 1
- Unfolded RNNs and TDNNs are feedforward alternatives for modeling temporal context; TDNNs efficiently model long temporal contexts.The unfolded RNN uses a fixed number of time steps and requires training time comparable to standard FNNs, while outperforming FNNs.Source: §1 · p. 1
- Earlier FSMNs modeled long-term dependencies without recurrent feedback, and the paper introduces DFSMN skip connections to support deeper networks.The authors report that DFSMN consistently outperformed BLSTM, particularly with LFR, and that lookahead order can be designed to meet latency demands.Source: §1 · p. 1
- DFSMN achieved 9.4% WER on the 2000-hour Fisher task and over 20% relative improvement over LFR-LCBLSTM on a 20000-hour Mandarin task.The Fisher result was a 1.5% absolute WER reduction versus the authors’ BLSTM baseline; the Mandarin comparison used LFR-trained systems.Source: §1 · p. 1
2. FROM FSMN TO CFSMN
FSMN adds FIR-like memory blocks to feedforward networks to encode long context. Compact FSMN places these blocks on linear projection layers to simplify the architecture and speed learning.
2. FROM FSMN TO CFSMN
- FSMN adds tapped-delay memory blocks to hidden layers, using learnable FIR-like filters to encode long context in a fixed-size representation.The paper motivates this design by viewing recurrent layers as conceptually approximable by high-order finite impulse response filters.Source: §2 · pp. 1–2
- The memory block’s look-back and lookahead orders specify the historical and future context windows represented at each time instance.Its output can be passed to the next hidden layer like a standard hidden activation.Source: §2 · p. 2
- cFSMN simplifies FSMN by adding a smaller linear projection after nonlinear hidden layers and placing memory blocks on the projection layers.The stated goals are to address the extra parameters introduced by memory blocks and speed learning.Source: §2 · p. 2
- Both FSMN and cFSMN remain feedforward architectures and can be trained with mini-batch stochastic gradient descent using standard backpropagation.Source: §2 · p. 2
3. DEEP-FSMN
DFSMN adds skip connections between cFSMN memory blocks to address gradient vanishing in deeper networks. Stride and lookahead settings also shape redundancy and latency.
3. DEEP-FSMN
- Skip connections route lower-layer memory-block outputs to higher layers, helping gradients flow directly to lower layers and addressing gradient vanishing in deeper cFSMN.Deeper cFSMN built by adding layers is described as susceptible to gradient vanishing.Source: §3 · p. 2
- DFSMN uses identity mappings for the within-block skip connection in all experiments.The connection may be linear or nonlinear; identity is an option when memory-block dimensions match.Source: §3 · p. 2
- Stride factors are added to memory blocks to reduce redundancy from overlap between adjacent speech frames.Source: §3 · p. 2
- Lookahead-filter order and stride in each memory block determine total latency, which DFSMN can be configured to meet real-time demands.The paper gives total latency as a function of those settings.Source: §3 · p. 3
4. EXPERIMENTS
Across English and Mandarin speech-recognition experiments, DFSMN and LFR-DFSMN outperform BLSTM baselines, with results also depending on model depth, modeling units, and lookahead settings.
4.1. English Recognition Task
- The English task uses about 2,000 hours of Switchboard and Fisher training data and evaluates WER on the Switchboard portion of Hub5e00.Experiments use a tri-gram language model.Source: §4.1 · p. 3
- Stride benefits memory-block performance, and deeper DFSMN architectures consistently improve results from exp2 through exp5.Table 2 presents WER for DFSMN architectures with different configurations.Source: §4.1 · p. 3, Table 2 · p. 3
- 9.4% WER was achieved by DFSMN on the 2,000-hour Fisher task, an absolute 1.5% WER reduction versus BLSTM with a smaller model.The result used cross-entropy without feature-space or speaker-space adaptation.Source: §4.1 · p. 3
4.2. Mandarin Recognition Task
- The Mandarin evaluation uses 5,000- and 20,000-hour training sets, about 30 hours of test data, and character error rate as its metric.The 5,000-hour set is a subset of the full 20,000-hour set.Source: §4.2 · p. 3
4.2.1. 5000-hour-task
- About 20% relative CER reduction versus LFR-LCBLSTM and about 3 times speedup in training and decoding RTF were reported for LFR-DFSMN.The comparison also reports that deeper cFSMN can degrade while DFSMN consistently improves with depth.Source: §4.2.1 · p. 4, Table 3 · p. 4, Table 4 · p. 4
4.2.2. 20000-hour-task
- More than 20% relative improvement over LFR-LCBLSTM was achieved by DFSMN with 20 frames of latency versus 40; reducing delay from 20 to 5 frames loses about 5% performance (150 ms).Table 5 compares models on the 20,000-hour task; the reported latency at five 30-ms frames is suitable for real-time applications.Source: §4.2.2 · p. 4, Table 5 · p. 4
5. CONCLUSIONS
DFSMN combines skip connections between adjacent memory blocks with deeper architecture, and the reported experiments show gains over BLSTM, especially with lower frame rate. The paper reports 9.4% WER on the 2,000-hour Fisher task and more than 20% relative improvement on the 20,000-hour Mandarin task.
5. CONCLUSIONS
- DFSMN consistently outperforms BLSTM, with especially strong gains when trained with lower frame rate.The reported results include 9.4% WER on the 2,000-hour Fisher task and more than 20% relative improvement on the 20,000-hour Mandarin task versus LFR-trained LCBLSTM, with smaller model size and lower latency.Source: §5 · p. 4