End-to-end Speech Recognition with Word-based RNN Language Models

1. INTRODUCTION

• End-to-end ASR seeks to simplify module-based systems by training a single network from paired acoustic and language data without linguistic knowledge.The paper describes this as making it feasible to build ASR systems without expert knowledge.Source: §1 · p. 1
• External LMs can be combined with an end-to-end decoder through shallow fusion, adding LM log probabilities during decoding.Earlier character-based LMs were used to predict the next character in Japanese and Mandarin systems.Source: §1 · p. 1
• English character-based LMs can underperform word LMs because they must model linguistic constraints across long character sequences.Character-based models allow open-vocabulary recognition, whereas word-based models face an out-of-vocabulary problem.Source: §1 · p. 1
• Multi-level decoding combines character-based scoring within words with word-based rescoring at word boundaries, while retaining character-level scores for out-of-vocabulary words.Character scores can help hypotheses survive until a word boundary, when word probabilities become available.Source: §1 · p. 1
• Two-level character-and-word RNN-LMs increase training time, decoding computation, and memory use, motivating a simpler language-modeling strategy.The paper notes that building both LMs takes almost twice the training time.Source: §1 · pp. 1–2
• The proposed look-ahead word-based RNN-LM uses word probabilities to predict next characters, replacing the character LM while targeting competitive accuracy with less decoding computation.The paper evaluates the approach on WSJ and LibriSpeech.Source: §1 · p. 2

2. END-TO-END ASR ARCHITECTURE

• The hybrid ASR architecture combines an encoder-decoder with external language-model scoring, and the proposed LMs can also be applied to standard attention-based encoder-decoders.The evaluation uses a hybrid CTC/attention network.Source: §2 · p. 2

2.1. Network architecture

• A VGG-style CNN and stacked BLSTM layers encode speech, while the decoder jointly uses CTC, attention, and an RNN-LM to predict labels.Two CNN max-pooling steps reduce the hidden sequence length to T/4; labels are distinct characters or alphabet symbols.Source: §2.1 · p. 2
• Joint CTC/attention training maximizes a log-linear combination of their posterior probabilities, with CTC's forward-backward algorithm helping estimate monotonic alignments.The shared CNN/BLSTM encoder serves both the CTC and attention decoder networks.Source: §2.1 · p. 2

2.2. Decoding with external language models

• Output-label-synchronous beam search scores partial hypotheses using CTC, attention, and LM probabilities, adding the LM probability with scaling factor γ in the log-probability domain.The search limits partial hypotheses at each length to a predefined beam width to improve efficiency.Source: §2.2 · pp. 2–3
• Attention and LM scores are accumulated recursively as each character extends a partial hypothesis, whereas CTC uses the cumulative probability of label sequences sharing that prefix.The CTC prefix probability is defined as the cumulative probability of all label sequences having the hypothesis as their prefix.Source: §2.2 · pp. 2–3

3.1. Basic approach

• Character sequences map deterministically to word sequences using spaces, and the word-level RNN-LM scores the last word when a space or end-of-sentence label is hypothesized.Homonyms, alternative spellings, and ambiguous segmentations are handled as distinct context-dependent or beam-search hypotheses.Source: §3.1 · p. 3

3.2. Multi-level RNN-LM

• The multi-level LM combines character- and word-level RNN-LMs, applying word probabilities at boundaries while retaining character predictions for within-word beam survival and OOV words.Character probabilities for known words are canceled at word ends; character predictions help keep hypotheses alive until then.Source: §3.2 · p. 3

3.3. Look-ahead word-based RNN-LM

• The look-ahead LM replaces the character LM with next-character probabilities derived from anticipated-word masses on a prefix tree, with cumulative sums enabling efficient computation.The model uses alphabetically ordered vocabulary IDs for the cumulative sums.Source: §3.3 · p. 4

• Acoustic-to-word CTC maps acoustic features directly to words but requires substantial paired acoustic and text data; one cited model used 125,000 hours of transcribed audio.Source: §4 · p. 4
• The proposed architecture pairs a character- or subword-based encoder-decoder with an external RNN-LM trained on large text, making it more suitable for low-resource languages with limited parallel data.Source: §4 · p. 4
• Subword methods require selecting a unit inventory: more units can improve acoustic expressiveness but make training harder with limited data, and encoder-decoder and LM units need not match.Source: §4 · pp. 4–5

5. EXPERIMENTS

• WSJ evaluation used si284 for training, dev93 for validation, and eval92 for evaluation; the study also tested on LibriSpeech, a 1,000-hour audiobook corpus.WSJ contains approximately 80 hours of data.Source: §5.1 · p. 5, §5 · p. 5, Table 1 · p. 5

5.1. Evaluation with WSJ

• The WSJ RNN-LMs were trained on 37M words from 1.6M sentences, with word-LM vocabularies of 20K, 40K, and 65K.Source: §5.1 · p. 5
• 5.1% WER on WSJ Eval'92 was the best reported among end-to-end ASR systems on the WSJ benchmark.The paper cautions that differing network architectures make direct comparison difficult.Source: §5.1 · pp. 5–6, Table 3 · p. 6
• The 20K look-ahead LM raised WSJ Eval'92 WER from 5.6% to 6.1%, with errors mainly from OOV words; increasing vocabulary size to 40K and 65K yielded a large improvement.The authors suggest that look-ahead scores consistent with word-LM probabilities helped the 65K model in beam search.Source: §5.1 · p. 5, Table 2 · p. 5
• The 20K-word look-ahead LM increased decoding time by 38%; with 65K-word look-ahead LMs, decoding time was 55% of the No LM baseline and remained below the 20K multi-level LM.Source: §5.1 · pp. 5–6, Figure 3 · p. 5

5.2. Evaluation with LibriSpeech

• On LibriSpeech, the 65K look-ahead LM had competitive WER to the multi-level LM while retaining a speed benefit and eliminating character-LM training.The experiments used 9.4M words of audio transcriptions for language-model training.Source: §5.2 · p. 6, Table 5 · p. 6

6. CONCLUSION

• 5.1 %WER on WSJ Eval’92 was achieved with increased vocabulary size, the best WER reported for end-to-end ASR on this benchmark.Source: §6 · p. 6
• The look-ahead word-based RNN-LM supports decoding with only one LM, with competitive accuracy and less beam-search computation than the multi-level LM.It also eliminates the need to train a character-level LM.Source: §6 · p. 6
• The proposed method was evaluated on WSJ and LibriSpeech for computational cost and recognition accuracy.Source: §6 · p. 6