Source-linked AI summary
Transformer-based Acoustic Modeling for Hybrid Speech Recognition
Yongqiang Wang, Abdelrahman Mohamed, Duc Le, Chunxi Liu, Alex Xiao, Jay Mahadeokar, Hongzhao Huang, Andros Tjandra, Xiaohui Zhang, Frank Zhang, Christian Fuegen, Geoffrey Zweig, Michael L. Seltzer
TL;DR
Hybrid speech recognition needs effective acoustic models despite recurrent networks’ limitations in long-term dependency modeling and parallel processing. This paper evaluates transformer-based acoustic models with positional encodings, iterated loss, and limited right context. On Librispeech, the models achieve 19% to 26% relative WER reductions over the previous best hybrid result with a standard 4-gram LM, while neural-LM rescoring reaches state-of-the-art results.
Problem
RNN acoustic models have difficulty modeling long-term temporal dependencies and processing speech in parallel, motivating alternatives for hybrid ASR.
Method
The paper proposes transformer-based acoustic models for hybrid ASR and evaluates positional embeddings, iterated loss for deep transformers, and limited right context.
Results
19% to 26% relative WER reduction is achieved on Librispeech over the previous best hybrid result with a standard 4-gram LM; neural-LM rescoring yields state-of-the-art results.
Takeaways & Limitations
Transformer acoustic models significantly outperform BLSTM baselines on Librispeech, with findings also confirmed on a much larger internal dataset.
Takeaways & Limitations
The transformer’s quadratic cost with speech-signal length remains a major blocker to practical use, and limited per-layer right context can accumulate into a large look-ahead window.
Abstract
from arXiv · showhide
We propose and evaluate transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. We also present a preliminary study of using limited right context in transformer models, which makes it possible for streaming applications. We demonstrate that on the widely used Librispeech benchmark, our transformer-based AM outperforms the best published hybrid result by 19% to 26% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, our proposed approach achieves state-of-the-art results on Librispeech. Our findings are also confirmed on a much larger internal dataset.
1 Introduction
The paper proposes transformer-based acoustic models for hybrid speech recognition to address limitations of RNN acoustic models. On Librispeech, the approach improves substantially over strong BLSTM and prior hybrid baselines.
- RNN acoustic models struggle with long-term temporal dependencies because of vanishing or exploding gradients.
- RNN recurrence also makes parallel processing of speech signals difficult.
- Self-attention directly connects arbitrary input positions and supports parallel computation, motivating transformer acoustic modeling.
- The work proposes and evaluates transformer-based acoustic models with positional encodings, iterated loss, and an initial study of streamable transformers.
- 19% to 26% relative WER improvement is obtained over the previous best hybrid result on Librispeech with a standard 4-gram LM.
2 Hybrid Architecture
Hybrid ASR uses an acoustic encoder to produce frame-level representations and HMM-state posteriors, which are combined with lexicons and language models during decoding. The paper applies transformers to improve this practical hybrid architecture.
- An acoustic encoder maps input frames x_1, · · ·, x_T to high-level embeddings z_1, · · ·, z_T.
- The embeddings produce posterior distributions over tied HMM states for each frame.
- Posterior distributions are combined with lexicons and language models to construct a search graph, then decoded to find the best hypothesis.
- Hybrid ASR remains useful for the authors’ practical problems because it integrates readily with knowledge sources such as personalized lexicons.
3 Acoustic Modeling Using Transformer
The paper describes self-attention-based transformer acoustic encoders, positional information mechanisms, and architectural components for deep hybrid ASR models. It also explains iterated loss and the use of self-attention in place of recurrent acoustic encoders.
- 3.1 Self-Attention and Multi-Head Attention: Self-attention computes attention distributions over input positions and combines value projections to produce output embeddings.
- 3.1 Self-Attention and Multi-Head Attention: Limited right-context masking restricts attention beyond t + R, enabling streaming processing of speech signals.
- 3.1 Self-Attention and Multi-Head Attention: Multi-head attention applies multiple attention heads independently, concatenates their outputs, and linearly transforms them into a common space.
- 3.2 Architecture of Transformer: Each transformer layer combines multi-head attention with a position-wise feed-forward network, residual connections, and dropout regularization.
- 3.3 Positional Embedding: Because transformer outputs are permutation-invariant, positional information is added through sinusoid embeddings, frame stacking, or convolutional embeddings.
- 3.4 Training Deep Transformer: Iterated loss adds auxiliary cross-entropy losses from intermediate transformer layers to enable training deeper networks.
- 3.5 Relation to Other Works: For hybrid speech recognition, self-attention replaces recurrent layers in the acoustic encoder without using the transformer’s sequence-to-sequence cross-attention.
4 Experiments
Experiments evaluate transformer acoustic models against BLSTM baselines on Librispeech and a large internal English video ASR task, while studying positional embeddings, depth, iterated loss, and limited right context. Transformers consistently improve recognition results, including under standard 4-gram decoding and on the internal task.
- Experiment setup: Librispeech experiments use 960 hours of read speech, four clean/other development and test sets, and a standard 4-gram LM for first-pass decoding.The evaluation corpus contains acoustically more challenging “other” sets, and test utterances are not segmented.
- Positional embeddings: Four positional-embedding methods are evaluated, including no positional embedding, sinusoid absolute positions, frame stacking, and convolutional embedding.Frame stacking uses the current frame plus eight future frames before stride-2 sampling; convolutional embedding implicitly encodes relative position.
- Transformer vs. BLSTM: 2–4% lower WER on test-clean and 7–11% lower WER on test-other are obtained by transformer models versus similarly sized BLSTM-based models.The comparison includes models with similar parameter counts and a larger approximately 163M-parameter BLSTM baseline.
- Iterated loss and depth: 5.5% relative WER reduction is achieved by increasing transformer depth to 20 layers, from 6.46 to 6.10.Deeper models can become difficult to train, motivating auxiliary cross-entropy losses from intermediate transformer layers.
- Librispeech results: 19% and 26% WER reductions on test-clean and test-other are achieved over the previous best 4-gram-only hybrid system, while neural-LM rescoring yields 2.26/4.85 WERs.The reported rescored results are described as state-of-the-art on Librispeech.
- Large-scale experiment: On a 13.7K-hour internal English video ASR task, transformer acoustic models outperform vggBLSTM by 4.0–7.6%.The task includes curated clean, clean, and noisy test sets, with training and test data initially segmented into maximum 10-second segments.
5 Discussions And Conclusions
The work finds that transformer-based acoustic models outperform BLSTM systems on Librispeech, while identifying unresolved questions about their sources of gains and practical deployment costs.
- Transformer-based acoustic models significantly outperform BLSTM systems and provide the best acoustic models on the Librispeech benchmark.
- Initial experiments on a much larger and more challenging dataset confirm the Librispeech findings.
- The experiments do not establish how much of the transformer’s superior performance comes from replacing recurrence with self-attention.
- The quadratic cost with respect to speech-signal length remains a major blocker to practical use of transformer-based acoustic models.