Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer

Kanishka Rao; Haşim Sak; Rohit Prabhavalkar

Source-linked AI summary

Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer

Kanishka Rao, Haşim Sak, Rohit Prabhavalkar

arXiv:1801.00841v1Jan 2, 2018cs.CLcs.SDeess.AS

TL;DR

The paper asks whether RNN-T can provide competitive streaming end-to-end ASR while benefiting from additional text, pronunciation data, and subword units. It combines pretrained acoustic and language components within an RNN-T and finds that a 30k-wordpiece system achieves near-baseline WER across voice-search and voice-dictation tasks.

Problem
End-to-end ASR systems were simpler but had limited evidence of matching conventional systems, especially on voice-search, while relying mainly on expensive transcribed acoustic data.
Method
The paper trains streaming RNN-T models with CTC- and language-model-pretrained components, incorporates pronunciation and text data, and compares grapheme with wordpiece output units.
Results
A 30k-wordpiece RNN-T achieves 8.5% WER on voice-search and 5.2% on voice-dictation, compared with baseline WERs of 8.3% and 5.4%, respectively.
Takeaways & Limitations
A single end-to-end neural model can perform state-of-the-art streaming speech recognition while using wordpieces and additional data to improve performance.
Takeaways & Limitations
The paper notes that CTC assumes model outputs at a frame are conditionally independent of previous output labels.

Abstract

from arXiv · show

We investigate training end-to-end speech recognition models with the recurrent neural network transducer (RNN-T): a streaming, all-neural, sequence-to-sequence architecture which jointly learns acoustic and language model components from transcribed acoustic data. We explore various model architectures and demonstrate how the model can be improved further if additional text or pronunciation data are available. The model consists of an `encoder', which is initialized from a connectionist temporal classification-based (CTC) acoustic model, and a `decoder' which is partially initialized from a recurrent neural network language model trained on text data alone. The entire neural network is trained with the RNN-T loss and directly outputs the recognized transcript as a sequence of graphemes, thus performing end-to-end speech recognition. We find that performance can be improved further through the use of sub-word units (`wordpieces') which capture longer context and significantly reduce substitution errors. The best RNN-T system, a twelve-layer LSTM encoder with a two-layer LSTM decoder trained with 30,000 wordpieces as output targets achieves a word error rate of 8.5\% on voice-search and 5.2\% on voice-dictation tasks and is comparable to a state-of-the-art baseline at 8.3\% on voice-search and 5.4\% voice-dictation.

1. INTRODUCTION

The paper examines RNN-T as a streaming, end-to-end alternative to conventional ASR pipelines, while addressing performance gaps and limited use of text-only data. It explores incorporating text and pronunciation data and using wordpieces for recognition.

1. INTRODUCTION: Conventional ASR separates acoustic, pronunciation, and language modeling, whereas end-to-end systems use one neural network to output transcripts directly from audio.End-to-end sequence-to-sequence models contain encoder and decoder components corresponding to acoustic and language models.
1. INTRODUCTION: Typical encoder-decoder systems encode the entire input before decoding, preventing real-time streaming recognition.Streaming architectures such as RNN-T can decode after the first input is encoded.
1. INTRODUCTION: RNN-T is selected because it supports streaming recognition while directly producing transcript outputs.The work considers streaming recognition architectures specifically through the RNN-T model.
1. INTRODUCTION: End-to-end models remained worse than conventional systems on voice-search despite being comparable on dictation in earlier evaluations.The cited earlier work trained end-to-end models on approximately 12,500 hours of transcribed data.
1. INTRODUCTION: The paper explores RNN-T architectures and shows how additional text and pronunciation data can improve end-to-end ASR.It also investigates wordpieces as sub-word output units for speech recognition.

2. RNN-TRANSDUCER

RNN-T combines an acoustic encoder with prediction and joint networks to model output history while processing speech incrementally. Its blank-versus-label transitions enable streaming inference, and its prediction network can be pretrained from text-only language data.

2. RNN-TRANSDUCER: The encoder maps acoustic frames to higher-level representations, and the prediction and joint networks together function as the decoder.The encoder is analogous to a CTC-based acoustic model.
2. RNN-TRANSDUCER: RNN-T consists of an encoder, prediction network, and joint network trained together with the RNN-T loss.The joint network combines encoder and prediction outputs to produce probabilities over blanks and target symbols.
2. RNN-TRANSDUCER: The prediction network removes CTC’s conditional-independence limitation by conditioning on previous non-blank predictions.It receives the last non-blank label and produces a decoder representation.
2. RNN-TRANSDUCER: During inference, a non-blank prediction updates the prediction network, while a blank advances the encoder to the next acoustic frame.Alternating these updates allows recognition to stream, with termination after a blank at the final frame.
2. RNN-TRANSDUCER: Unlike related streaming architectures, RNN-T’s prediction network is not conditioned on encoder output, enabling language-model pretraining on text-only data.The model uses beam search for inference, with a computational simplification that skips some prefix summations.

3. UNITS, ARCHITECTURES AND TRAINING

The paper studies grapheme and wordpiece output units alongside deep LSTM architectures and staged pretraining. Encoder CTC pretraining, language-model initialization, hierarchical multitask losses, and time reduction are used to initialize and accelerate RNN-T training.

3. UNITS, ARCHITECTURES AND TRAINING: Grapheme RNN-Ts use letters, digits, special symbols, and a space symbol to produce word-segmentable transcripts.The space symbol segments recognized grapheme sequences into words.
3. UNITS, ARCHITECTURES AND TRAINING: Wordpieces provide subword output units that make large-vocabulary RNN-T inference more practical than using millions of whole-word labels.Wordpiece vocabularies include an additional space symbol and are learned from text-derived word counts.
3. UNITS, ARCHITECTURES AND TRAINING: The explored architectures use 5-to-12-layer LSTM encoders and two-layer LSTM decoders, with optional initialization from CTC acoustic models and language models.Pretraining is used to initialize encoder and decoder parameters before joint RNN-T optimization.
3. UNITS, ARCHITECTURES AND TRAINING: Hierarchical CTC pretraining jointly predicts phonemes, graphemes, and wordpieces at different encoder depths.The wordpiece architecture uses these targets at depths 5, 10, and 12, respectively.
3. UNITS, ARCHITECTURES AND TRAINING: A time-convolutional layer reduces the encoder sequence length by a factor of three for wordpiece models.The reduction lowers the number of encoder features processed by the decoder during training and inference.

4. EXPERIMENTAL SETUP

The experiments compare RNN-T recognizers with a conventional ASR baseline using large-scale acoustic, pronunciation, and text data. Evaluation uses voice-search and voice-dictation test sets, with RNN-T components initialized from CTC and language-model pre-training.

Baseline system: The conventional baseline uses separate acoustic, pronunciation, and language models trained on approximately 18,000 hours of transcribed voice data.Its acoustic model uses CTC-trained LSTMs, while pronunciation resources include expert dictionaries and learned pronunciations.
Baseline system: The baseline language model draws on 150 million voice-search sentences, 150 million voice-dictation sentences, and tens of billions of typed search sentences.The model is pruned to 100 million n-grams with a 4-million-word vocabulary.
RNN-T training: RNN-T models use the same acoustic data as the baseline, with a CTC-pre-trained encoder and a decoder initialized from an LSTM language model trained on text-only data.Approximately half a billion text sentences are sampled for decoder language-model pre-training.
Evaluation: Word error rate is reported on voice-search and voice-dictation test sets containing roughly 15,000 utterances each.The experiments incrementally measure the impact of RNN-T architecture and pre-training choices.

5. RESULTS

The results show incremental gains from CTC pre-training, deeper and hierarchical-CTC encoders, text pre-training, and wordpiece outputs. The strongest configuration uses a deep wordpiece RNN-T and substantially improves over earlier grapheme systems.

Architecture and pre-training: CTC pre-training improves grapheme RNN-T WER from 13.9% to 13.2% on voice-search and from 8.4% to 8.0% on voice-dictation.The comparison uses otherwise identical five-layer encoder and two-layer decoder architectures.
Architecture and pre-training: The deeper eight-layer encoder improves WER from 13.2% to 12.0% on voice-search and from 8.4% to 6.9% on voice-dictation.Multi-CTC losses at depths 5 and 8 were used because training without the multi-loss setup was unstable.
Architecture and pre-training: Pronunciation-aware hierarchical CTC improves voice-search WER from 12.0% to 11.4%, with little improvement on voice-dictation.Phoneme targets at depth 5 expose the encoder to pronunciation variants before grapheme targets at depth 8.
Architecture and pre-training: Text-only language-model pre-training improves WER from 11.4% to 10.8% on voice-search and from 6.8% to 6.4% on voice-dictation.A two-layer, 700-cell LSTM language model is pre-trained with grapheme targets.
Wordpiece models: The 30k-wordpiece RNN-T achieves 8.5% voice-search WER and 5.2% voice-dictation WER, comparable to the state-of-the-art baseline.A time convolution reduces encoder sequence length by a factor of 3 without affecting WER, reducing training and inference processing.

6. ANALYSIS

Most reported improvements arise from fewer substitution errors, especially when wordpieces, pronunciation data, and text data are added. The examples indicate gains in context-sensitive recognition of similar-sounding words and names.

Error analysis: Wordpieces yield a 2.3% absolute WER improvement over graphemes, with 1.5% attributable to fixing substitution errors.Pronunciation and text data each improve voice-search WER by an absolute 0.6%, also through substitution-error reductions.
Error analysis: The corrected substitutions include proper names and context-dependent words, such as “barbara stanwick” becoming “barbara stanwyck.”The analysis also reports improvements involving pronunciation variants and longer-range language context.

7. CONCLUSION

The paper demonstrates streaming end-to-end speech recognition with RNN-T while incorporating acoustic, pronunciation, and text data through pre-training. A 30k-wordpiece model reaches near-baseline WER on voice-search and matches or slightly improves the baseline on voice-dictation.

Contributions: CTC encoder pre-training gives a 5% relative WER improvement, while replacing a five-layer encoder with an eight-layer encoder gives a further 10% relative improvement.These improvements are reported as part of the paper’s incremental RNN-T development.
Contributions: Hierarchical CTC incorporating phoneme targets improves voice-search WER by 5% relative, with little impact on voice-dictation.Decoder pre-training with LSTM language models incorporates text-only data and produces an overall 5% relative improvement.
Language-model analysis: Table 2 reports parameter counts and held-out-set word perplexity for LSTM language models trained with different output units.The conclusion identifies wordpieces as significantly outperforming grapheme-based RNN-Ts.
Final comparison: 30k-wordpiece RNN-Ts achieve 8.5% WER on voice-search and 5.2% on voice-dictation, versus baseline WERs of 8.3% and 5.4%.The comparison is between a single end-to-end model and a conventional system with separate acoustic, pronunciation, and language models.