Source-linked AI summary
A Comparative Study on Transformer vs RNN in Speech Applications
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto, Xiaofei Wang, Shinji Watanabe, Takenori Yoshimura, Wangyou Zhang
TL;DR
Speech applications use sequence-to-sequence models, but Transformer systems require practical guidance and reproducible evaluation alongside conventional RNNs. This paper compares Transformer and RNN across ASR, ST, and TTS benchmarks, finding Transformer outperformed RNN on 13/15 ASR corpora while exposing decoding-speed limitations.
Problem
Transformer speech applications require more complex configurations, and existing systems lacked open-source toolkits and reproducible experiments.
Method
The paper conducts comparative experiments between Transformer and RNN across 15 monolingual ASR, one multilingual ASR, one ST, and two TTS corpora, providing training guidance and reproducible recipes.
Results
Transformer outperformed RNN on 13/15 ASR corpora and produced performance benefits across the examined speech tasks.
Takeaways & Limitations
The study provides training tips, pretrained models, and reproducible open-source recipes intended to support further Transformer research in speech applications.
Takeaways & Limitations
Transformer decoding is much slower than Kaldi because naive self-attention requires O(n^2) computation in speech length, motivating faster decoding algorithms.
Abstract
from arXiv · showhide
Sequence-to-sequence models have been widely used in end-to-end speech processing, for example, automatic speech recognition (ASR), speech translation (ST), and text-to-speech (TTS). This paper focuses on an emergent sequence-to-sequence model called Transformer, which achieves state-of-the-art performance in neural machine translation and other natural language processing applications. We undertook intensive studies in which we experimentally compared and analyzed Transformer and conventional recurrent neural networks (RNN) in a total of 15 ASR, one multilingual ASR, one ST, and two TTS benchmarks. Our experiments revealed various training tips and significant performance benefits obtained with Transformer for each task including the surprising superiority of Transformer in 13/15 ASR benchmarks in comparison with RNN. We are preparing to release Kaldi-style reproducible recipes using open source and publicly available datasets for all the ASR, ST, and TTS tasks for the community to succeed our exciting outcomes.
1. INTRODUCTION
This paper intensively compares Transformer and RNN models for ASR, ST, and TTS, addressing the practical difficulty of configuring Transformer speech systems and the lack of reproducible open-source experiments.
- The study compares Transformer and RNN performance across ASR, ST, and TTS speech applications.
- Transformer speech systems require more complex choices involving optimization, network structure, and data augmentation than conventional RNN systems.
- Existing Transformer-based speech applications lacked an open-source toolkit and reproducible experiments, motivating a community-driven project.
- The experiments evaluate error rates, training curves, and multi-GPU scalability across speech tasks.
- The work contributes a large-scale comparison, Transformer training guidance, and reproducible ESPnet recipes and pretrained models using public datasets.
2. SEQUENCE-TO-SEQUENCE RNN
Sequence-to-sequence models transform source sequences into target sequences using encoder and decoder networks. In the RNN formulation, recurrent encoder and decoder modules generate targets using encoded inputs and target prefixes.
- Sequence-to-sequence models learn to transform a source sequence X into a target sequence Y.
- The source sequence may contain speech features for ASR and ST or characters for TTS.
- During training, the decoder receives a ground-truth target prefix; during decoding, it receives a generated prefix.
- The model minimizes a scalar loss between the generated sequence Ypost and target sequence Y.
- RNN encoders typically use bidirectional LSTMs, while RNN decoders commonly use unidirectional LSTMs with encoder-decoder attention.
3. TRANSFORMER
Transformer replaces recurrent connections with self-attention, using multi-head attention, feedforward layers, and positional encodings for sequence modeling. Its decoder masks future target frames, enabling parallel computation while preserving autoregressive generation.
- Transformer architecture: Transformer learns sequential information through self-attention rather than the recurrent connections used in RNNs.
- Multi-head attention: The attention matrix is formed from query-key interactions, while values provide the representations combined by attention.
- Multi-head attention: Multi-head attention runs multiple attention mechanisms in parallel and combines their outputs with learnable projections.
- Self-attention encoder: Transformer encoder layers use self-attention followed by a two-layer feedforward network with learnable weights and biases.
- Self-attention decoder: The decoder uses encoder-decoder attention and masked self-attention so target frame t cannot connect to future frames, allowing parallel masking without sequential operations.
- Positional encoding: Sinusoidal positional encodings are concatenated with encoder and decoder inputs to represent time locations in the non-recurrent model.
4. ASR EXTENSIONS
The ASR system converts log-mel speech features into encoded representations and predicts token sequences with a Transformer sequence-to-sequence decoder. Training combines sequence-to-sequence and CTC objectives, while decoding combines S2S, CTC, and language-model scores through beam search.
- ASR architecture: ASR predicts character or SentencePiece token sequences from log-mel filterbank speech features.
- ASR encoder architecture: An encoder preprocessing CNN subsamples 83-dimensional log-mel filterbank frames with pitch features before the encoder body produces encoded features.
- ASR decoder architecture: The decoder uses encoded speech features and previous target tokens to predict the posterior distribution of the next token.
- ASR training and decoding: ASR training minimizes a weighted sum of sequence-to-sequence and CTC negative log-likelihoods controlled by hyperparameter α.
- ASR training and decoding: Beam-search decoding combines scores from the sequence-to-sequence decoder, CTC module, and RNN language model using hyperparameters γ and λ.
5. ST EXTENSIONS
Speech translation uses the same sequence-to-sequence modules and speech inputs as ASR, but source and target languages differ. Unlike ASR, it cannot use CTC because translation does not guarantee monotonic source-target alignment.
- ST extensions: Speech translation receives ASR-style speech features and target token sequences, but the source and target languages are different.
- ST extensions: CTC is unavailable for speech translation because the task does not guarantee monotonic alignment between source and target sequences.
6. TTS EXTENSIONS
The TTS system generates log-mel filterbank features and end-of-sequence probabilities from character sequences using Transformer encoder-decoder modules. Its training and inference procedures differ through teacher forcing during training and autoregressive prediction during inference.
- TTS architecture: TTS generates log-mel filterbank features and predicts end-of-sequence probabilities from input character sequences.
- TTS encoder architecture: The encoder embeds character IDs, adds learnably scaled positional encoding, and transforms them into encoded features for the decoder.
- TTS decoder architecture: Training uses ground-truth acoustic features with teacher forcing, whereas inference feeds predicted features autoregressively.
- TTS decoder architecture: The Prenet converts 80-dimensional log-mel features into hidden features, while the decoder predicts target features and EOS probabilities before Postnet refinement.
- TTS training: TTS training combines L1 feature loss, weighted BCE EOS loss, and guided attention loss without additional balancing hyperparameters.
- TTS inference: During inference, autoregressive prediction stops when the EOS probability exceeds a threshold such as 0.5.
7. ASR EXPERIMENTS
Across 15 ASR corpora, the study compares Transformer and RNN systems under varied datasets and training configurations, finding broad Transformer advantages alongside distinct training requirements and decoding costs.
- Settings: Transformer used a common architecture across most corpora, whereas RNN followed corpus-specific existing best architectures.The largest LibriSpeech corpus used a larger Transformer attention configuration.
- Settings: Transformer required different optimization settings because its training iteration was eight times faster and its updates were more fine-grained than RNN.RNN used Adadelta with early stopping, while Transformer followed prior settings including dropout, learning rate, and warmup steps.
- Results: 13/15 corpora favored Transformer over RNN in character/word error rate, including low-resource, large-resource, noisy, and far-field tasks.Transformer results were also comparable to Kaldi’s HMM-based system on 7/12 corpora.
- Results: Transformer reached RNN’s best accuracy about eight times faster on LibriSpeech with one GPU, while larger minibatches improved Transformer accuracy but not RNN accuracy.Smaller Transformer minibatches typically caused under-fitting after warmup steps.
- Discussion: Increasing Transformer minibatch size addressed under-fitting while simultaneously improving training speed and accuracy; accumulated gradients could emulate large minibatches without multiple GPUs.Dropout was essential for Transformer against over-fitting, whereas it did not improve RNN results.
- Discussion: Transformer decoding was much slower than Kaldi’s system because naive self-attention requires O(n^2) computation for speech length n.The authors identify faster Transformer decoding as necessary for direct comparison with DNN-HMM systems.
8. MULTILINGUAL ASR EXPERIMENTS
The multilingual ASR experiment evaluates shared Transformer and RNN models across ten languages. Transformer significantly outperformed RNN in nine languages, including comparisons against a deeper external BLSTM system.
- Experimental setup: 10 languages were evaluated with one shared multilingual model whose output units covered 5,297 graphemes and special symbols.The languages included English, Japanese, Mandarin Chinese, German, Spanish, French, Italian, Dutch, Portuguese, and Russian.
- Results: Transformer significantly outperformed the authors’ RNN in 9 languages.The same superiority held against the deeper 7-layer BLSTM and RNNLM system reported by Watanabe et al.
- Results: 28.0% relative improvement was the largest reported gain, occurring for VoxForge Italian; 8 languages exceeded 10% relative improvement.These results support Transformer’s advantage over RNN in multilingual end-to-end ASR.
- Results: Figure 3 compares multilingual end-to-end ASR results among Watanabe et al.’s RNN, ESPnet RNN, and ESPnet Transformer systems.The figure’s central comparison is whether Transformer maintains its advantage across languages and against both RNN baselines.
9. SPEECH TRANSLATION EXPERIMENTS
The speech translation experiment compares Transformer and RNN on Fisher-CALLHOME English–Spanish, finding higher Transformer BLEU but more serious under-fitting during training.
- Transformer improved BLEU from 16.5 with the RNN baseline to 17.2 on CALLHOME evltest.
- Transformer showed more serious under-fitting than RNN during speech translation training.
- Using a pretrained encoder from the ASR experiment was the proposed solution for Transformer’s under-fitting.
10. TTS EXPERIMENTS
The TTS experiments compare Transformer with Tacotron 2 on M-AILABS and LJSpeech, examining convergence, training practices, attention behavior, decoding speed, and future trade-offs.
- 10. TTS EXPERIMENTS: Transformer and Tacotron 2 showed similar L1 loss convergence on M-AILABS and LJSpeech.
- 10. TTS EXPERIMENTS: Larger minibatches improved Transformer validation L1 loss and training speed but harmed RNN L1 loss.
- 10.3. Discussion: Transformer TTS training can be accelerated with large minibatches, while validation loss, especially BCE loss, can overfit more easily.
- 10.3. Discussion: Attention maps should be monitored for convergence, and guided attention loss must be selectively applied because some Transformer heads are not diagonal.
- 10.3. Discussion: 78.5 ms per frame versus 6.5 ms for RNN was reported for Transformer filterbank decoding on a single-thread CPU.
- 10.3. Discussion: FastSpeech reduced decoding to 0.6 ms per frame and produced comparable speech quality to autoregressive Transformer TTS.
- 10.3. Discussion: Further work should investigate the trade-off between training speed and quality and introduce ASR techniques for TTS.
11. SUMMARY
The paper presents a broad Transformer-versus-RNN study across ASR, speech translation, and TTS, reporting promising results and practical guidance.
- The study covers 15 monolingual and one multilingual ASR corpus, one speech translation corpus, and two TTS corpora.
- The experiments report huge improvements in many ASR tasks and explain how the models were improved.
- Reproducible recipes, pretrained models, and training tips are intended to accelerate Transformer research in speech applications.