Source-linked AI summary
On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, Patrick Nguyen
TL;DR
Conventional ASR favors phonemes over graphemes for English, while evidence for attention-based models with less training data was limited. This paper compares phoneme, grapheme, and word-piece units across LibriSpeech tasks and finds grapheme and word-piece models consistently outperform phonemic ones.
Problem
For attention-based encoder-decoder ASR, it was not established whether grapheme models outperform phoneme models when labeled training data is limited.
Source: §1 · p. 1Method
The authors evaluate phoneme, grapheme, and word-piece LAS models across LibriSpeech 100hr, 460hr, and 960hr tasks, then examine rescoring and N-best hypotheses.
Source: §1 · p. 1Results
Across LibriSpeech subsets, grapheme and word-piece models consistently outperform phonemic models; rescoring their 960hr word-piece N-best lists yields 9% relative improvements on test-clean.
Source: §6.1 · p. 3, §7 · p. 4Takeaways & Limitations
Word-piece attention models can achieve relatively low oracle WER with only 8-best hypotheses, and rescoring those hypotheses with graphemic or phonemic models gives good improvements.
Source: §7 · p. 4Phoneme-output models give up standalone attention-based recognition, and their decoder language model may be weaker than one based on character units.
Source: §4.1 · p. 2
Abstract
from arXiv · showhide
In conventional speech recognition, phoneme-based models outperform grapheme-based models for non-phonetic languages such as English. The performance gap between the two typically reduces as the amount of training data is increased. In this work, we examine the impact of the choice of modeling unit for attention-based encoder-decoder models. We conduct experiments on the LibriSpeech 100hr, 460hr, and 960hr tasks, using various target units (phoneme, grapheme, and word-piece); across all tasks, we find that grapheme or word-piece models consistently outperform phoneme-based models, even though they are evaluated without a lexicon or an external language model. We also investigate model complementarity: we find that we can improve WERs by up to 9% relative by rescoring N-best lists generated from a strong word-piece based baseline with either the phoneme or the grapheme model. Rescoring an N-best list generated by the phonemic system, however, provides limited improvements. Further analysis shows that the word-piece-based models produce more diverse N-best hypotheses, and thus lower oracle WERs, than phonemic models.
1. Introduction
This work tests whether grapheme and word-piece units retain their advantage over phonemes in attention-based speech recognition with less training data. Across LibriSpeech tasks, grapheme and word-piece models consistently outperform phonemic models.
1. Introduction
- Character-based sequence-to-sequence models map audio features directly to words and jointly learn acoustic, pronunciation, and language modeling without a hand-crafted lexicon.The passage names graphemes, BPEs, and word-pieces as character-based output units.Source: §1 · p. 1
- The paper asks whether grapheme-based encoder-decoder models also approach or exceed phoneme-based models when less labeled training data is available.Earlier empirical comparisons in this setting had used large amounts of labeled data.Source: §1 · p. 1
- For English, conventional ASR studies found phoneme-based models outperform grapheme-based models, with the gap typically narrowing as training data increases.The question is whether this pattern carries over to attention-based encoder-decoder models.Source: §1 · p. 1
- Across LibriSpeech 100hr, 460hr, and 960hr tasks, grapheme and word-piece models consistently outperform phonemic models, including with limited training data.The paper also studies rescoring and analyzes the quality and oracle error rates of models’ N-best hypotheses.Source: §1 · p. 1
2. Sequence-to-Sequence Speech Models
The paper uses Listen, Attend, and Spell (LAS) models, with an encoder, attention module, and decoder. The decoder predicts output labels conditioned on previously predicted labels.
2. Sequence-to-Sequence Speech Models
- LAS combines an encoder, attention module, and decoder to map frame-level audio features to output labels.Figure 1 depicts the LAS model.Source: §2 · p. 1, Figure 1 · p. 1
- The encoder transforms input audio frames into hidden activations, and attention summarizes those activations into a vector for each prediction step.Both encoder and decoder use recurrent neural networks, allowing joint optimization of the model.Source: §2 · pp. 1–2
- Standard LAS models use grapheme, word-piece, or BPE output units, while the decoder conditions on the history of previously predicted labels.The cited passage identifies grapheme, word-piece, and BPE as standard character-based units.Source: §2 · p. 2
3. Phonemic Sequence-to-Sequence Model
The phonemic LAS model uses phonemes as output units, with pronunciation and decoding procedures to handle pronunciation variants, unknown words, word boundaries, and homophones.
3. Phonemic Sequence-to-Sequence Model
- Phonemic LAS uses phonemes as output units; a pronunciation lexicon can ease integration of new words but gives up end-to-end recognition and complicates training and decoding.The passage notes that phonemes are natural acoustic-modeling labels for non-phonetic languages.Source: §3 · p. 2
- To create a unique training mapping when words have multiple pronunciations, the model randomly selects one variant and uses UNK for words absent from the lexicon.It adds an end-of-word token, EOW, to model word boundaries; the authors report that this improves performance.Source: §3 · p. 2
- For decoding, a word-based n-gram language model and lexicon are composed as a WFST search network to constrain and score partial hypotheses.This addresses homophone ambiguity in decoding.Source: §3 · p. 2
4. LibriSpeech Experimental Setup
The experiments use LibriSpeech subsets and compare model units under distinct evaluation setups. The paper describes the acoustic features, model sizes, and training procedure.
4.1. Dataset
- LibriSpeech provides 100hr, 460hr, and 960hr training subsets, with clean and other development and test sets; Table 1 summarizes OOV rates across data scenarios and the lexicon.About 800M words of extra text and an official 3-gram word LM are available; phonemic models use that LM, while grapheme and word-piece models are evaluated without a lexicon or LM unless noted.Source: §4.1 · p. 2, Table 1 · p. 2
- The word-piece models use 16K units trained on each subset, and the official LibriSpeech pronunciation lexicon contains 200K words.The passage defines OOV as words unseen in training data.Source: §4.1 · p. 2
- Using phonemes as output units gives up standalone recognition with the attention-based model, and the phoneme-level decoder language model may be weaker than one based on character units.The passage frames the latter as a possibility, relating it to a subword-level class-based language model.Source: §4.1 · p. 2
4.2. Models and training
- The models use 80-dimensional log-mel features with deltas and accelerations, and two strided convolution layers reduce the encoder time resolution by a factor of four.Small, medium, and large configurations vary in encoder and decoder sizes.Source: §4.2 · p. 2
- All models are trained from random initialization for about 80 epochs using asynchronous stochastic gradient descent with Adam across 16 GPUs.The experiments use the open-source TensorFlow Lingvo toolkit.Source: §4.2 · p. 2
5. Standalone Performance Results
On 960hr, larger models improve WER and the large word-piece model has the best grapheme/word-piece baseline results. Grapheme and word-piece models also outperform phonemic models on 100hr and 460hr tasks.
5.1. Baseline model performance on 960hr
- The large word-piece model achieves the best WERs among grapheme and word-piece baselines, and models benefit from additional parameters.Table 2 summarizes WERs for the grapheme and word-piece models.Source: Table 2 · p. 2, §5.1 · p. 2
5.2. Phonemic model performance on 960hr
- Despite using a lexicon and 3-gram word LM, the phonemic system performs worse than the best graphemic model.The phonemic models’ PERs improve as model size increases from 7M to 130M; removing EOW causes about 2% absolute WER degradation.Source: Table 3 · p. 3, §5.2 · p. 3
- With shallow-fusion LSTM LM, the word-piece model reaches WERs of 3.6% on test-clean and 10.3% on test-other, narrowing the gap to the best reported hybrid system.The model is trained with cross-entropy, without minimum word error rate training.Source: §5.2 · p. 3
5.3. Results on 100hr and 460hr tasks
- At 460hr, large models perform best, while at 100hr, medium models perform best; even at higher OOV rates, grapheme and word-piece models outperform phonemic models.Attention-based model performance also degrades dramatically as training data is reduced.Source: Table 5 · p. 3, §5.3 · p. 3
6. Rescoring Experiments
The section tests cross-unit rescoring and dual-decoder approaches to exploit complementarity among speech-recognition models. Rescoring word-piece hypotheses produces the clearest gains, while phonemic N-best lists have weaker oracle coverage and union decoding adds little.
6.1. N-best Rescoring results
- When the word-piece model rescores N-best lists generated by the phonemic system, improvements are limited to only up to 4% relative; its 30-best list has much higher oracle WERs than the word-piece model’s 8-best list.Source: §6.1 · p. 3
- Rescoring word-piece hypotheses improves test-clean WER by 9% relative with either model, and test-other WER by 8% with phonemes or 9% with graphemes.Combining all models adds only up to 0.1 absolute improvement.Source: §6.1 · p. 3, Table 6 · p. 4
6.2. Union of N-best lists with cross-rescoring results
- Union cross-rescoring adds only marginal test-other gains over rescoring 8-best word-piece hypotheses; phoneme-rescored 16-best word-piece lists perform slightly better than the union.The approach independently decodes lists from both models, cross-rescores them, and selects the best hypothesis from the union.Source: §6 · p. 3, Table 8 · p. 4, §6.2 · p. 4
6.3. Why is Oracle WER So High for Phonemic System?
- Phonemic N-best diversity centers on homophones, whereas word-piece hypotheses vary spellings of unseen words such as “bozzle”; phonemic LAS assigns homophones the same score.The paper notes that one parameter weights the external language model across the search, and says addressing this issue may be important for improving the phonemic system.Source: §6.3 · p. 4
6.4. Rescoring with an auxiliary decoder
- A shared-encoder model with a word-piece main decoder and phoneme auxiliary decoder improves rescoring, but an independent phoneme model gives larger improvements.The auxiliary decoder predicts the next word in phonemes, with its recurrent states initialized from the main decoder at each word boundary; it adds 30M parameters. Figure 2 depicts this arrangement.Source: Table 9 · p. 4, §6.4 · p. 4, Figure 2 · p. 4
7. Conclusion
Across LibriSpeech subsets, grapheme- and word-piece-based models consistently outperform phoneme-based models. The authors argue this pattern is not explained by training-data amount and report useful word-piece N-best rescoring, while identifying streaming end-to-end models as a future test.
7. Conclusion
- Across the LibriSpeech subsets, word-piece and grapheme models consistently outperform phoneme-based models.The authors conclude that character-based units’ dominance is not due to the amount of training data.Source: §7 · p. 4
- The authors suggest the pattern is more likely related to the model itself, including the decoder’s conditioning on all preceding labels.They contrast this explanation with training-data amount.Source: §7 · p. 4
- Word-piece attention models achieve relatively low oracle WER with 8-best hypotheses, and graphemic or phonemic rescoring of those hypotheses gives good improvements.Source: §7 · p. 4
- Whether streaming end-to-end approaches such as RNN-T show similar trends remains a future question.Source: §7 · p. 4