Source-linked AI summary
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition
Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee
TL;DR
Deep recurrent LSTM networks become difficult to train as depth adds spatial complexity to their existing temporal structure. The paper introduces residual LSTM, which separates spatial and temporal shortcuts and reuses LSTM components to reduce parameters. On AMI SDM distant-speech recognition, residual LSTM achieved the strongest reported deep-model WER while plain and highway LSTMs degraded with depth.
Problem
Increasing depth in recurrent architectures is difficult because LSTM and GRU networks already have temporal depth, making vanishing or exploding gradients a central training challenge.
Method
Residual LSTM adds a spatial shortcut between output layers, separates it from the temporal memory path, and reuses the projection matrix and output gate instead of dedicated shortcut gates.
Results
10-layer residual LSTM achieved the lowest AMI SDM WER at 41.0%, while 10-layer plain and highway LSTMs increased WER by 13.7% and 6.2% over 3-layer baselines.
Takeaways & Limitations
Residual LSTM improved with increasing depth on the AMI corpus, while 10-layer plain and highway LSTMs suffered severe training loss.
Takeaways & Limitations
WER comparisons are intended to measure relative architecture performance because forced-alignment labels differed from prior work and can change absolute WER by 1–2%.
Abstract
from arXiv · showhide
In this paper, a novel architecture for a deep recurrent neural network, residual LSTM is introduced. A plain LSTM has an internal memory cell that can learn long term dependencies of sequential data. It also provides a temporal shortcut path to avoid vanishing or exploding gradients in the temporal domain. The residual LSTM provides an additional spatial shortcut path from lower layers for efficient training of deep networks with multiple LSTM layers. Compared with the previous work, highway LSTM, residual LSTM separates a spatial shortcut path with temporal one by using output layers, which can help to avoid a conflict between spatial and temporal-domain gradient flows. Furthermore, residual LSTM reuses the output projection matrix and the output gate of LSTM to control the spatial information flow instead of additional gate networks, which effectively reduces more than 10% of network parameters. An experiment for distant speech recognition on the AMI SDM corpus shows that 10-layer plain and highway LSTM networks presented 13.7% and 6.2% increase in WER over 3-layer aselines, respectively. On the contrary, 10-layer residual LSTM networks provided the lowest WER 41.0%, which corresponds to 3.3% and 2.8% WER reduction over plain and highway LSTM networks, respectively.
1. Introduction
Deep recurrent networks are difficult to train because increasing depth compounds vanishing or exploding gradients, especially in LSTM and GRU architectures. Residual LSTM addresses this with separate spatial shortcuts, residual mappings, and parameter reuse, while improving deep-network speech-recognition performance.
- Deep recurrent architectures are difficult to train because they already have depth in the temporal domain, intensifying vanishing or exploding gradient problems as layers increase.
- Residual LSTM introduces a highway connection between adjacent output layers, separating the spatial shortcut from LSTM’s temporal memory path.Using output layers instead of internal memory cells can reduce interference with temporal gradient flow.
- Each residual LSTM output layer learns residual mappings rather than reproducing outputs already available through the highway path.This avoids spending computation or capacity regenerating similar outputs from prior layers.
- Residual LSTM reuses the LSTM projection matrix as a gate network, saving more than 10% of learnable parameters relative to highway LSTM.
- On AMI SDM, 10-layer plain and highway LSTMs increased WER by 13.7% and 6.2% over 3-layer baselines, whereas 10-layer residual LSTM achieved the lowest WER of 41.0%.
2. Revisiting Highway Networks
Residual and highway networks provide shortcut paths for deep architectures, while LSTM uses internal memory cells and gates to manage temporal dependencies. Highway LSTM extends this idea by reusing internal memory cells for spatial connections, but deeper networks can suffer WER degradation.
- 2.1. Residual Network: Residual networks use identity shortcut mappings so stacked layers learn residual functions rather than reproduce existing outputs.With no new residual mapping needed, layers can bypass identity mappings without additional training.
- 2.2. Highway Network: Highway networks combine a transformed layer output with a gated shortcut path from the input.The transform gate controls how much of each path contributes to the next layer.
- 2.2. Highway Network: Highway paths are optional: a transform gate of 1 ignores the shortcut, whereas a gate of 0 bypasses the output layer.Unlike residual connections, highway shortcuts are not always active.
- 2.3. Long Short-Term Memory (LSTM): LSTM uses internal memory cells with forget and input gates to represent long- and short-term sequential dependencies.The forget gate controls prior memory passed to the next time step, while the input gate scales new input.
- 2.4. Highway LSTM: Highway LSTM reuses LSTM internal memory cells for spatial highway connections between stacked layers.It improved far-field speech recognition over plain LSTM, but WER degraded as depth increased from 3 to 8 layers.
3. Residual LSTM
Residual LSTM separates the spatial shortcut from the temporal memory-cell update, using output-layer connections to reduce gradient interference in deep recurrent networks. It reuses existing LSTM components to control shortcut scaling and reduce parameters.
- Architecture: Residual LSTM adds a shortcut from the previous output layer to the current projection output, rather than accumulating the shortcut on the internal memory cell.The previous output layer is used as the lower-layer shortcut, while the existing LSTM update equations remain unchanged.
- Gradient flow: Separating spatial and temporal paths reduces the impact of the temporal cell update on subsequent layers.The temporal gradient can flow through the forget gate, while residual LSTM limits its direct leakage into the next layer through path separation.
- Parameterization: The projection matrix is reused to scale the LSTM output, so residual LSTM does not increase parameters relative to plain LSTM.The shortcut is always active, making scaling of the main path necessary.
- Parameterization: Approximately 10% fewer parameters are obtained when N is 1024, the projection dimension is N/2, and the network has more than five layers.The stated reduction is relative to a highway LSTM with the same N and a projection matrix.
- Output scaling: The output gate is reused to scale shortcut outputs and control variance without adding a separate normalization layer or parameter.A trainable output gate can adapt the output range and better handle variable output variance than a fixed scaling factor.
4. Experiments
Experiments on AMI meeting speech evaluate training behavior and WER as LSTM depth increases, including SDM-only and combined SDM/IHM settings. Residual LSTM avoids the depth-related degradation observed in plain and highway LSTMs and achieves the strongest reported WERs.
- Experimental setup: AMI experiments use meeting recordings with distant-microphone speech, and compare plain, highway, and residual LSTM acoustic models.The models use 1024 memory cells and 512 output nodes; decoding reports overlapped and non-overlapped WER.
- Training performance with increasing depth: 15% training and 3.6% CV cross-entropy increases occurred for 10-layer highway LSTMs relative to 3-layer models.The increased CV loss was not attributed to overfitting because training cross-entropy also increased.
- Training performance with increasing depth: 10-layer residual LSTMs improved CV cross-entropy with depth, although training cross-entropy increased 6.7%.The authors associate the training-loss increase with better generalization, possibly from improved deep feature representations.
- WER evaluation with SDM corpus: 13.7% and 6.2% WER increases affected 10-layer plain and highway LSTMs, respectively, versus their 3-layer networks.The comparison uses non-overlapped WER and shows degradation as depth increases for both baselines.
- WER evaluation with SDM corpus: 41.0% WER was the lowest result for the 10-layer residual LSTM, reducing WER by 3.3% and 2.8% versus 3-layer plain and highway LSTMs.Within residual LSTMs, 5-layer and 10-layer models reduced WER by 1.2% and 2.2% versus the 3-layer network.
- Evaluation caveat: WER comparisons may differ from prior results because the forced-alignment labels were not identical.The authors state that 1–2% WER variation can result from aligned-label quality, while the evaluation focuses on relative architecture performance.
- WER evaluation with SDM and IHM corpora: With combined SDM and IHM training data, the 10-layer residual LSTM achieved 39.3% non-overlapped WER, 3.1% below the 5-layer model.The 10-layer highway LSTM still incurred a 6.6% WER increase with depth, while increased data provided larger gains for deeper residual LSTMs.
5. Conclusion
The paper concludes that residual LSTM is a deep recurrent architecture using shortcut connections between adjacent layer outputs. On AMI speech recognition experiments, it improves with increasing depth while 10-layer plain and highway LSTMs suffer training loss.
- Conclusion: Residual LSTM provides shortcut connections between adjacent layer outputs without dedicated shortcut gate networks.It reuses the LSTM projection matrix and output gate, reducing network parameters by roughly 10% compared with highway LSTMs.
- Conclusion: AMI experiments showed residual LSTMs improving significantly with depth, whereas 10-layer plain and highway LSTMs suffered severe training loss.The conclusion contrasts residual LSTM's depth behavior with both baseline architectures.