Source-linked AI summary
Highway Long Short-Term Memory RNNs for Distant Speech Recognition
Yu Zhang, Guoguo Chen, Dong Yu, Kaisheng Yao, Sanjeev Khudanpur, James Glass
TL;DR
The paper addresses difficult distant speech recognition and the training challenges of deep LSTM networks. It introduces highway connections, latency-controlled bidirectional LSTMs, and efficient frame- and sequence-discriminative training. On AMI SDM, the proposed models achieve 43.9/47.7% WER and improve over DNN and DLSTM baselines.
Problem
Distant speech recognition is challenging because of reverberation and overlapping acoustic signals, while deep LSTMs are harder to train and slower to converge.
Method
The paper adds gated cross-layer highway connections, introduces latency-controlled bidirectional LSTMs, and develops efficient training using frame and sequence discriminative criteria.
Results
43.9/47.7% WER on AMI SDM dev/eval sets was reported, with relative improvements of 15.7% over DNNs and 5.3% over DLSTM RNNs.
Takeaways & Limitations
Highway connections consistently outperform normal (B)LSTMP networks and support deeper models with greater benefit from sequence discriminative training.
Takeaways & Limitations
CSC-BPTT limits exploitable history to chunk contexts and adds decoding computation because contexts must be recomputed for each chunk.
Abstract
from arXiv · showhide
In this paper, we extend the deep long short-term memory (DLSTM) recurrent neural networks by introducing gated direct connections between memory cells in adjacent layers. These direct links, called highway connections, enable unimpeded information flow across different layers and thus alleviate the gradient vanishing problem when building deeper LSTMs. We further introduce the latency-controlled bidirectional LSTMs (BLSTMs) which can exploit the whole history while keeping the latency under control. Efficient algorithms are proposed to train these novel networks using both frame and sequence discriminative criteria. Experiments on the AMI distant speech recognition (DSR) task indicate that we can train deeper LSTMs and achieve better improvement from sequence training with highway LSTMs (HLSTMs). Our novel model obtains $43.9/47.7\%$ WER on AMI (SDM) dev and eval sets, outperforming all previous works. It beats the strong DNN and DLSTM baselines with $15.7\%$ and $5.3\%$ relative improvement respectively.
1. INTRODUCTION
Distant speech recognition remains difficult because reverberation and overlapping signals persist despite advanced processing. The paper addresses this challenge with highway LSTMs and latency-controlled bidirectional LSTMs, reporting strong AMI SDM results.
- DSR remains challenging because reverberation and overlapping acoustic signals persist despite sophisticated front-end processing and multi-pass decoding.
- Highway connections add gated direct links between adjacent LSTM layers, enabling deeper networks by providing a more direct path for information flow.
- LC-BLSTMs exploit full past history while limiting lookahead to a fixed number of frames, controlling latency compared with standard BLSTMs.
- 43.9/47.7% WER on AMI SDM dev/eval sets was reported as the best result on this task, with relative improvements of 15.7% over DNNs and 5.3% over DLSTMs.
- The paper evaluates standard and highway (B)LSTM RNNs at 3 and 8 layers on the AMI single distant microphone setup.
2. RELATED WORK
The related work connects highway LSTMs to prior gated cross-layer architectures while distinguishing their formulation and speech-recognition focus.
- Prior highway-network studies also used gated linear connections between layers, but one focused on DNNs and another on text tasks.
- The paper states that its highway LSTM formulation differs from the earlier DNN-focused highway networks.
- The paper notes that the tools and scripts for reproducing its results are publicly available through the CNTK toolkit.
3. HIGHWAY LONG SHORT-TERM MEMORY RNNS
The paper extends LSTMs with gated connections between memory cells in adjacent layers, while also developing bidirectional variants that use future context. These designs target deeper networks and controlled use of speech context.
- 3.1. Long short-term memory RNNs: LSTM cells use gated information flow and a linear dependence between successive memory-cell states.The paper defines input, forget, and output gates, along with cell and cell-output activations.
- 3.2. Deep LSTM RNNs: Deep LSTMs stack layers, but conventional cross-layer interaction must pass through each lower layer’s output-input connection.The lower layer’s output is fed to the upper layer as input.
- 3.3. HLSTM RNNs: Highway LSTMs add a direct gated connection between memory cells in adjacent layers.A carry gate controls how much information flows directly from lower-layer cells to upper-layer cells.
- 3.3. HLSTM RNNs: The highway connection can vary between plain LSTM behavior and directly passing the previous layer’s cell memory.This more direct cross-layer influence can alleviate gradient vanishing when training deeper LSTM RNNs.
- 3.4. Bidirectional HLSTM RNNs: Bidirectional extensions process past and future contexts with separate forward and backward layers whose outputs are concatenated for the next layer.The backward layer operates from t = T to 1, replacing t −1 with t + 1 in the forward-layer equations.
4. EFFICIENT NETWORK TRAINING
The paper addresses training and decoding constraints in bidirectional recurrent networks by carrying past history while limiting future context. It also proposes a two-forward-pass approach for sequence discriminative training.
- 4.2. Sequence discriminative training: Sequence discriminative training packs multiple short chunks from multiple sequences into one minibatch and uses two forward passes.The described packaging example uses 40 sequences with 20 frames per chunk.
- 4.1. Latency-controlled bi-directional model training: CSC-BPTT splits sequences into chunks and adds left and right contextual frames so chunks can be independently trained in large minibatches.The appended context frames provide information but do not generate error signals during training.
- 4.1. Latency-controlled bi-directional model training: CSC-BPTT limits exploitable history and requires recomputing both contextual sides during decoding.The resulting model is no longer a true bidirectional RNN under the stated context limitation.
- 4.1. Latency-controlled bi-directional model training: Latency-controlled bidirectional RNNs carry the whole past history while retaining truncated future context.Left-context information is carried from the previous chunk instead of recomputing left contextual frames for each chunk.
5. EXPERIMENT SETUP
Experiments use the AMI meeting corpus under the single distant microphone condition, with conventional feature extraction, alignment, and neural-network training pipelines. The recurrent models use projected LSTM architectures and specified training schedules.
- 5.1. Data: The experiments use the AMI meeting corpus’s single distant microphone condition, with an 80-hour training split from the recommended corpus partition.The corpus contains around 100 hours of meeting recordings from instrumented rooms with multiple microphone types.
- 5.2. Baseline systems: Kaldi performs feature extraction, early triphone training, and decoding, while a GMM-HMM triphone system supplies forced-alignment labels.The alignments support subsequent neural-network training.
- 5.2. Baseline systems: The DNN pipeline uses 40-dimensional filterbank features with delta and delta-delta features, concatenated over 15 frames into 1,800 dimensions.CNTK trains a six-layer DNN with 2,048 sigmoid units per layer.
- 5.3. Neural network models: Projected LSTM models use 1,024 memory cells per hidden layer and a 512-node projection, while projected bidirectional models use a 300-node projection.The highway companions share these structures and add highway connections.
- 5.4. Training details: Models are randomly initialized, and validation controls learning-rate reduction when no gain is observed.Unidirectional models use truncated BPTT with 20-frame segments and 40 simultaneous utterances.
6. RESULTS
Highway LSTMP networks consistently outperform standard counterparts, with dropout further reducing WER. Highway connections also support deeper networks and larger gains from sequence discriminative training.
- 6. RESULTS: The experiments evaluate WER on the AMI SDM1 eval set, reporting both the full set and a non-overlapping-speech subset.
- 6.1. 3-layer Highway (B)LSTMP: Highway LSTM and BLSTM networks consistently outperform their non-highway counterparts, although the initial margin is small.
- 6.2. Highway (B)LSTMP with dropout: Dropout on highway connections further lowers WER, with a 0.1 rate early in training increased to 0.8 after five epochs.
- 6.3. Deeper highway LSTMP: Highway connections preserve recognition performance when expanding from 3 to 8 layers, unlike normal LSTMP networks, whose performance degrades dramatically.
- 6.4. Highway LSTMP with sequence training: 5% relative improvement comes from sequence training with highway connections and dropout, compared with 3% without the highway connection.On the non-overlapping segment subset, the relative improvement is roughly 7%.
7. CONCLUSION
The paper presents highway LSTM networks for far-field speech recognition. Experiments suggest that highway connections, especially with dropout, improve performance, support deeper networks, and increase the benefit of sequence discriminative training.
- 7. CONCLUSION: Highway LSTM networks consistently outperform normal (B)LSTMP networks, especially when dropout controls the highway connection’s on/off state.
- 7. CONCLUSION: Highway connections allow the network to become much deeper and obtain larger benefits from sequence discriminative training.