Source-linked AI summary
Multi-Dialect Speech Recognition With A Single Sequence-To-Sequence Model
Bo Li, Tara N. Sainath, Khe Chai Sim, Michiel Bacchiani, Eugene Weinstein, Patrick Nguyen, Zhifeng Chen, Yonghui Wu, Kanishka Rao
TL;DR
Multi-dialect ASR systems can generalize poorly to individual dialects, while conventional systems require dialect-specific pronunciation and language models. This paper uses dialect information in a single LAS model, which outperforms dialect-dependent models across seven English dialects, including a reported 16.5% relative WER reduction on AU.
Problem
ASR systems trained for one dialect perform poorly on another, and pooled multi-dialect systems can generalize poorly to individual dialects; conventional systems also require separate pronunciation and language models.
Source: §1 · p. 1Method
Results
The all-layer 1-hot system outperformed dialect-dependent models on every test set, with a 16.5% relative WER reduction on AU.
Source: §4.2.2 · p. 4, §5 · p. 4Takeaways & Limitations
The results support modeling seven English dialects in one LAS system, with shared parameters and improved recognition quality reported for low-resource dialects.
Source: §1 · p. 2, §5 · p. 4With the 8D vectors tested, the authors observed no large performance differences between 1-hot and learned embeddings, so those experiments do not establish a preference.
Source: §4.2.2 · p. 3
Abstract
from arXiv · showhide
Sequence-to-sequence models provide a simple and elegant solution for building speech recognition systems by folding separate components of a typical system, namely acoustic (AM), pronunciation (PM) and language (LM) models into a single neural network. In this work, we look at one such sequence-to-sequence model, namely listen, attend and spell (LAS), and explore the possibility of training a single model to serve different English dialects, which simplifies the process of training multi-dialect systems without the need for separate AM, PM and LMs for each dialect. We show that simply pooling the data from all dialects into one LAS model falls behind the performance of a model fine-tuned on each dialect. We then look at incorporating dialect-specific information into the model, both by modifying the training targets by inserting the dialect symbol at the end of the original grapheme sequence and also feeding a 1-hot representation of the dialect information into all layers of the model. Experimental results on seven English dialects show that our proposed system is effective in modeling dialect variations within a single LAS model, outperforming a LAS model trained individually on each of the seven dialects by 3.1 ~ 16.5% relative.
1. INTRODUCTION
The paper investigates a single LAS model for seven English dialects, motivated by the difficulty of building conventional multi-dialect ASR and the weak generalization of pooled systems. Adding dialect information lets the model outperform dialect-dependent systems and improves recognition quality for low-resource dialects.
1. INTRODUCTION
- ASR systems tuned for one dialect perform poorly on another, while systems trained on many dialects can generalize poorly to each individual dialect.Conventional systems also require separate pronunciation and language models per dialect, where wrong-dialect sub-word predictions can propagate errors.Source: §1 · p. 1
- For neural-network systems, dialect adaptation by continued training on dialect-specific data works well, unlike the MLLR and MAP techniques commonly used for GMM systems.Source: §1 · p. 1
- Sequence-to-sequence models jointly learn acoustic, pronunciation, and language modeling in one neural network, making them a candidate for simpler multi-dialect systems.Pooling dialect graphemes lets the model jointly represent variations across dialects.Source: §1 · p. 1
- Explicit dialect information enables multi-dialect LAS to outperform dialect-dependent models, with reported recognition-quality improvement for low-resource dialects.The model shares most parameters across dialects, which the paper says forces cross-dialect generalization during training.Source: §1 · pp. 1–2
2. MULTI-DIALECT LAS MODEL
The section describes the pooled LAS baseline and three ways to supply dialect information: as output targets, input vectors, or cluster coefficients. The approaches assume dialect information is known in advance or readily obtainable.
2. MULTI-DIALECT LAS MODEL
- LAS combines an encoder, decoder, and attention module; the pooled baseline shares English graphemes across dialects and assumes dialect information is available.The encoder is acoustic-model-like, the decoder language-model-like, and attention learns their alignment.Source: §2 · p. 2
2.1. Dialect Information as Output Targets
- Appending a dialect symbol to the grapheme targets trains dialect classification alongside grapheme prediction; placing it last avoids making grapheme prediction depend on an earlier dialect decision.The paper contrasts this with multi-task learning, which requires balancing separate objective functions.Source: §2.1 · p. 2
2.2. Dialect Information as Input Vectors
- Dialect 1-hot or learned-embedding vectors can be appended to encoder layers, decoder layers, or both, enabling dialect-dependent biases for acoustic or language-model variations.Source: §2.2 · p. 2
2.3. Dialect Information as Cluster Coefficients
- The tested CAT setup uses dialect vectors to weight cluster outputs that compensate activation offsets at the encoder’s fourth LSTM layer, while limiting added complexity.CAT adds network layers and parameters; this study tests it only in the encoder.Source: §2.3 · p. 2
3. EXPERIMENTAL DETAILS
The experiments use noisy, anonymized voice-search speech from seven English dialects, with dialect-specific test sets and a shared LAS architecture. The reported table covers utterance counts for training and testing by dialect.
3. EXPERIMENTAL DETAILS
- Training data comprises about 40K hours and 35M utterances across seven English dialects, representing Google voice-search traffic.The utterances are anonymized and hand-transcribed; the training data includes artificially corrupted clean speech.Source: §3 · pp. 2–3
- Each dialect has a non-overlapping test set of roughly 10K anonymized, hand-transcribed utterances, or about 11 hours of voice-search speech.The table reports training and testing utterance counts per dialect.Source: §3 · p. 3, Table 1 · p. 3
- The shared LAS uses 80-dimensional log-mel features, five 1024D unidirectional LSTM encoder layers, a two-layer 1024D LSTM decoder, and 75 grapheme symbols.The model has 60.6M parameters and is trained with cross-entropy and asynchronous stochastic gradient descent.Source: §3 · p. 3
4. RESULTS
The results compare pooled and dialect-specific LAS models, then test ways of supplying dialect information to a shared model. Across the reported comparisons, dialect-aware systems improve on pooling alone, and the best system outperforms the dialect-specific models across dialects.
4.1. Pooling All Data
- Fine-tuning the pooled LAS model for each dialect achieves better recognition performance than pooling alone.Keeping the baseline training for a similar extra number of steps did not improve WER.Source: §4.1 · p. 3, Table 2 · p. 3
4.2.1. Results using Dialect Information as Output Targets
- Adding the dialect symbol at the end of the grapheme sequence performs better than adding it at the beginning, and S4 beats S2 except on IN and ZA.The end position avoids making grapheme prediction depend on potentially erroneous dialect classification.Source: §4.2.1 · p. 3, Table 3 · p. 3
4.2.2. Results using Dialect Information as Input Vectors
- S6, with dialect vectors fed to the decoder, outperforms S2 on every dialect except IN, where performance is the same.Encoder input (S5) gains on GB, ZA, AU, NG, and KE, is comparable on US, and remains worse on IN than S2; 1-hot and learned 8D embeddings show no big differences.Source: §4.2.2 · p. 3, Table 4 · p. 3
- Wrong dialect vectors degrade encoder results more than decoder results; IN is most distinguishable, while the smallest dialects vary little with the vector.The results suggest stronger acoustic than language-model variation across dialects; NG and KE also benefit more from parameter sharing, and ZA vectors slightly improve their performance over their own vectors.Source: §4.2.2 · p. 4, Figure 1 · p. 4
- Feeding 1-hot dialect vectors into all LAS layers (S7) beats dialect-dependent models on every test set, with the largest gain on AU at 16.5% relative WER reduction.The 8D 1-hot and learned embedding representations show no big differences in the encoder and decoder experiments; the authors suggest the small vector dimensionality may not distinguish them.Source: §4.2.2 · pp. 3–4, Table 4 · p. 3
4.2.3. Results using Information as Cluster Coefficients
- CAT with learned embeddings is more effective on US and IN and similar on other dialects than encoder input using 1-hot vectors, but adds around 3M parameters versus 160K.The authors leave a more thorough study of CAT for future work.Source: §4.2.3 · p. 4, Table 5 · p. 4
4.3. Combining Adaptation Strategies
- The combined results report that S7(1hot) outperforms dialect-specific models, with 3.1~16.5% WER reductions across dialects.Combining dialect identification with dialect vectors performs much better than S4 alone, but similarly to S7(1hot).Source: Table 6 · p. 4, §4.3 · p. 4
5. CONCLUSIONS
A seven-dialect English LAS system combines 1-hot dialect vectors at each encoder and decoder layer with grapheme targets ending in the dialect name. The authors report that this approach outperforms individually trained LAS models, while CAT may model dialect variation more powerfully at increased parameter cost.
5. CONCLUSIONS
- The 1-hot dialect-vector LAS system outperforms models trained on each individual language.The system uses dialect vectors at every encoder and decoder layer and appends the dialect name to the grapheme target.Source: §5 · p. 4
- Training on seven English dialects combines shared hidden representations for grapheme prediction and dialect classification.The model appends the dialect name as the last target symbol and feeds a 1-hot dialect vector into each LAS layer.Source: §5 · p. 4
- CAT could potentially model dialect variation more powerfully, but at the cost of increased parameters.The authors leave addressing that cost to future work.Source: §5 · p. 4