Source-linked AI summary
Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition
Yangyang Shi, Yongqiang Wang, Chunyang Wu, Ching-Feng Yeh, Julian Chan, Frank Zhang, Duc Le, Mike Seltzer
TL;DR
Low-latency streaming ASR requires transformers to operate on partially available speech without the delays and inefficient training associated with existing augmented-memory approaches. Emformer addresses these issues with cached attention states, redesigned memory carryover, and parallelized block training, achieving improved LibriSpeech accuracy alongside faster training and decoding. Under 80 ms average latency, it achieves WER 3.01% on test-clean and 7.09% on test-other.
Problem
Streaming ASR must produce outputs from partially available speech, while AM-TRF duplicates left-context computation and sequentializes block processing during training.
Method
Emformer caches left-context keys and values, carries memory from lower layers, disables summary-vector attention to the memory bank, and uses parallelized block processing in training.
Results
Emformer outperformed baselines in hybrid and transducer systems, with WER 3.01% on test-clean and 7.09% on test-other under 80 ms average latency.
Takeaways & Limitations
Emformer supports low-latency streaming recognition with improved LibriSpeech accuracy, 4.6-fold training speedup, and 18% decoding RTF reduction versus AM-TRF.
Takeaways & Limitations
For large datasets, the caching strategy does not give WER reduction.
Abstract
from arXiv · showhide
This paper proposes an efficient memory transformer Emformer for low latency streaming speech recognition. In Emformer, the long-range history context is distilled into an augmented memory bank to reduce self-attention's computation complexity. A cache mechanism saves the computation for the key and value in self-attention for the left context. Emformer applies a parallelized block processing in training to support low latency models. We carry out experiments on benchmark LibriSpeech data. Under average latency of 960 ms, Emformer gets WER $2.50\%$ on test-clean and $5.62\%$ on test-other. Comparing with a strong baseline augmented memory transformer (AM-TRF), Emformer gets $4.6$ folds training speedup and $18\%$ relative real-time factor (RTF) reduction in decoding with relative WER reduction $17\%$ on test-clean and $9\%$ on test-other. For a low latency scenario with an average latency of 80 ms, Emformer achieves WER $3.01\%$ on test-clean and $7.09\%$ on test-other. Comparing with the LSTM baseline with the same latency and model size, Emformer gets relative WER reduction $9\%$ and $16\%$ on test-clean and test-other, respectively.
1. INTRODUCTION
Streaming speech recognition must generate outputs from partially available utterances, making vanilla transformers difficult to use because look-ahead context can introduce substantial delay. Emformer improves the augmented memory transformer with cached attention states and parallelized training, achieving lower latency, faster training, and improved LibriSpeech recognition results.
- Motivation: Streaming ASR must produce outputs from partially available speech, while stacked look-ahead context can create significant delay.The delay arises from look-ahead context leaking across transformer layers.
- Prior approach: AM-TRF reduces long-range self-attention computation with a memory bank, but duplicates left-context computation and relies on sequential block processing during training.Its memory bank carries context from previous segments in an autoregressive manner, limiting training parallelization.
- Emformer: Emformer caches previous segments’ self-attention keys and values to remove duplicated left-context computation.It also changes memory carryover and disables summary-vector attention to the memory bank.
- Emformer: Emformer applies parallelized block processing in training to support low-latency speech recognition models.This addresses the sequential training limitation of AM-TRF.
- Results: 17% relative WER reduction on test-clean and 9% on test-other were obtained against AM-TRF under a 640 ms average-latency constraint.Emformer also reduced training time by almost 80% and decoding RTF by 18%.
- Results: At 80 ms average latency, Emformer achieved WER 3.01% on test-clean and 7.09% on test-other.It saved more than 91% computation from AM-TRF in this low-latency scenario.
2. EMFORMER
Emformer improves AM-TRF for low-latency streaming recognition by caching left-context projections, changing memory-bank flow, suppressing summary-vector attention to memory, and enabling parallel training. These changes address redundant computation, inefficient sequential training, and context-weighting issues while reducing computation substantially in low-latency settings.
- AM-TRF memory mechanism: The memory bank stores prior context information and produces a new memory vector for each segment through modified attention.In AM-TRF, the memory vector from one segment is required by the next segment in the same layer.
- Cache key and value from previous segments: Emformer caches key and value projections from previous segments, avoiding repeated left-context computation and omitting unnecessary left-context query projection.The left context overlaps prior center blocks, and left-context outputs are not passed to the next layer.
- Motivation: Emformer addresses AM-TRF’s sequential low-latency training limitation, which produces small computation loops and extremely low GPU usage.AM-TRF’s autoregressive memory dependency makes block processing sequential during training.
- Parallelized block processing in training: Emformer carries the memory bank from a lower layer rather than within the current layer, allowing each layer to train the whole sequence in parallel.This design replaces AM-TRF’s same-layer dependency that forces sequential block processing during training.
- Disallow attention between the summary vector and memory bank: Emformer disables summary-vector attention to the memory bank because including it can overweight the most-left context and destabilize long-form recognition.The paper reports that setting this attention weight to zero stabilizes training and improves recognition accuracy for long-form speech.
3.1. Data and Setup
The experiments evaluate Emformer on LibriSpeech using hybrid and transducer setups, with specified feature processing, model configurations, and training procedures.
- Dataset: LibriSpeech provides 1000 hours of book-reading utterances, with clean and noisy “other” subsets for development and evaluation.Models are selected using development-set WER and evaluated on the test sets.
- Hybrid model: The hybrid setup uses 80-dimensional log Mel filter bank features sampled every 10 ms, with speed perturbation and SpecAugment.
- Hybrid model: Emformer maps features into 512-dimensional inputs, using eight-head self-attention, 512-node layers, and 2048-dimensional feed-forward inner layers.Dropout is 0.1; medium-latency experiments use memory-bank length 4, while low-latency experiments use length 0.
- Transducer model: The transducer setup uses 1024 BPE sentence pieces, a two-layer LSTM predictor, and shallow fusion with a neural network language model during beam search.Predictor embeddings have 256 dimensions, LSTM layers have 512 hidden nodes, and the linear projection has 640 dimensions.
3.2. Results
Emformer improves streaming speech recognition efficiency and accuracy over AM-TRF and LSTM baselines across medium- and low-latency hybrid settings, while also performing consistently in transducer systems.
- From AM-TRF to Emformer: 4.6-fold training speedup and 18% relative RTF reduction were achieved over AM-TRF at 960 ms latency.Caching key and value computations removes duplicated left-context work; training decreased from 1.14 to 0.5 hours per epoch.
- From AM-TRF to Emformer: 17% and 9% relative WER reductions over AM-TRF were obtained on test-clean and test-other, respectively.These results use hybrid systems with 960 ms latency.
- Results from hybrid systems: 2.50% test-clean and 5.62% test-other WER were achieved at 960 ms EIL with the 120M-parameter Emformer and sMBR training.At 640 ms EIL, the same system achieved 2.62% and 5.97% WER on test-clean and test-other.
- Results from hybrid systems: 9% and 15% relative WER reductions over similarly sized and constrained LSTM systems were obtained on test-clean and test-other at low latency.The 36-layer Emformer achieved 3.01% and 7.09% WER under an 80 ms latency constraint.
- Results from transducer systems: At the same 640 ms EIL, Emformer consistently outperforms LCBLSTM on WER in transducer systems.With an external NNLM, transducer systems achieve WER similar to hybrid systems.
4. CONCLUSIONS
The paper concludes that Emformer combines cached left-context computation, modified memory handling, and parallelized block training for efficient low-latency recognition. On LibriSpeech, it outperforms baselines in hybrid and transducer systems while achieving strong WER at 960 ms and 80 ms latency.
- CONCLUSIONS: Emformer removes duplicated AM-TRF left-context computation with caching and disables summary-vector attention to stabilize training.It also changes memory carryover to support parallelized block processing during training.
- CONCLUSIONS: 4.6-fold training speedup and 18% decoding RTF reduction were obtained over AM-TRF.LibriSpeech experiments showed improvements over baselines in both hybrid and transducer systems.
- CONCLUSIONS: 2.50% and 5.62% WER were achieved on test-clean and test-other at 960 ms EIL, while 80 ms latency yielded 3.01% and 7.09%.The 960 ms results also used decoding RTF 0.13.