Rnn-transducer with language bias for end-to-end Mandarin-English code-switching speech recognition

1. INTRODUCTION

• Prior language-identity approaches use an auxiliary LID model, while limited code-switching training data poses a problem for end-to-end methods.Earlier DNN-HMM systems also use separately trained acoustic, language, and lexicon models, and complex lexicons require substantial human effort.Source: §1 · p. 1
• 16.2% and 12.9% relative error reduction on two test sets are reported for the proposed method over the RNN-T baseline.The method uses language identities to bias predictions at code-switching points.Source: §1 · p. 1
• The proposed RNN-T learns language identities from transcription by adding language IDs at code-switching points, eliminating the need for an additional LID model.The introduction also describes constraining word embeddings by language ID and using predicted IDs to adjust output posteriors.Source: §1 · p. 1

2. REVIEW OF RNN-T

• Unlike CTC, RNN-T conditions outputs on previous predictions through a prediction network.The prediction network is explicitly conditioned on previous outputs.Source: §2 · p. 2
• Because frame-level target alignments are unknown and T > U, RNN-T adds a blank symbol φ; function B removes consecutive repeats and blanks from an alignment.X is the acoustic input sequence with T frames, and Y is the target sequence with U outputs.Source: §2 · p. 2
• RNN-T computes P(Y ∈ Y*|X) by marginalizing over all possible alignments.The alignments are mapped to output sequences through B.Source: §2 · p. 2
• The encoder maps acoustic features to representations, the prediction network processes previous non-blank labels, and the joint network produces logits for a softmax distribution.The model is trained by maximizing the log-likelihood of the target-sequence probability.Source: §2 · p. 2

3. RNN-T WITH LANGUAGE BIAS

• The output symbol set is augmented with Mandarin and English language IDs, intended to model the distribution of code-switching in transcripts.Tagged data can bias the model to predict language IDs at code-switching points.Source: §3 · p. 2
• RNN-T can predict language IDs as output symbols and jointly use acoustic and linguistic information to model code-switching.Its prediction network acts like a language model over the history of labels.Source: §3 · p. 2
• Language-specific vectors are concatenated to English and Mandarin word and tag embeddings to strengthen their association with the corresponding language ID.The paper says this constraint helps the model learn distinctions between the two languages.Source: §3 · p. 2
• At inference, predicted language IDs adjust output posteriors to bias the next-step decode toward words in a particular language.The method handles speech recognition and LID simultaneously without additional training or inference burden.Source: §3 · p. 2
• The proposed model's training and inference algorithms are similar to those of standard RNN-T.Source: §3 · p. 3

4.1. Dataset

• SEAME provides spontaneous conversational Mandarin-English speech, split into train, testsge (Southeast Asian-accent English-biased), and testman (Mandarin-biased) sets.Speech-speed perturbation expands the training data threefold using rates of 0.9, 1, and 1.1.Source: §4.1 · p. 3

4.2. DNN-HMM Baseline System

• The DNN-HMM baseline uses a seven-layer TDNN trained with LF-MMI, a 4-gram language model, and a lexicon combining English and Mandarin entries.Its acoustic input consists of 13-dimensional MFCCs.Source: §4.2 · p. 3

4.3. RNN-T System

• The RNN-T baseline uses a four-layer, 512-unit LSTM encoder, a two-layer, 512-unit LSTM prediction network, and a 512-unit feed-forward joint network.It uses 80-dimensional log Mel-filterbank features, 512-dimensional word embeddings, dropout of 0.2, and beam search with size 35.Source: §4.3 · p. 3

4.4. Wordpieces

• BPE English wordpieces balance modeling-unit duration and unit counts against Chinese characters; the baseline uses 3,090 English wordpieces and 3,643 Chinese characters.The passage says Chinese-character acoustic counterparts are much longer than English letters, motivating the balance.Source: §4.4 · p. 3

4.5. Evaluation Metrics

• Mixed error rate (MER) combines English word error rate and Mandarin character error rate to balance errors across the two languages.The passage contrasts MER with using WER or CER alone.Source: §4.5 · p. 3

5.1. Results of RNN-T Model

• Under standard decoding, end-to-end systems have higher MER than LF-MMI TDNN, while data augmentation is more effective for end-to-end systems; testsge results are worse than testman.The authors suggest the gap to TDNN may shrink with more data and attribute the test-set difference probably to more difficult accented English in testsge.Source: §5.1 · p. 3
• 9.3% and 7.6% relative MER reductions on the two test sets are reported for tagged-transcription RNN-T versus baseline; word-embedding constraints add a smaller improvement.The following experiments use augmented training data; the authors identify tagged text as the main factor behind the MER reduction and report no added training or inference burden.Source: §5.1 · pp. 3–4

5.2. Effect of Language IDs Re-weighted Decode

• Language-ID reweighting outperforms standard decoding, with similar MER for the two tested λ values; the authors use λ = 0.2 thereafter.Reweighting is more effective on testman than testsge, though the authors say language-ID accuracy is hard to compute and offer higher accuracy only as a possible explanation.Source: §5.2 · p. 4

5.3. Results of Language Model Re-score

• Language-model rescoring with N-best lists (N=35) further improves performance; both N-gram and neural language models use tagged training transcriptions.The authors say the result leaves room to further optimize the RNN-T prediction network.Source: §5.3 · p. 4
• 25.9% and 21.3% relative MER reduction versus unaugmented RNN-T, and 16.2% and 12.9% versus augmented RNN-T, are reported for the best method on two dev sets.The paper reports that RNN-T methods outperform their baselines in both scenarios.Source: Table 4 · p. 4, §5.3 · p. 4

6. CONCLUSIONS AND FUTURE WORK

• On SEAME, MER on the two dev sets falls from 33.3% and 44.9% to 27.9% and 39.1%, respectively.The conclusion says the method handles speech recognition and LID simultaneously without an additional LID system or added training or inference burden.Source: §6 · p. 4
• Future work will pre-train the RNN-T prediction network on a large text corpus, then fine-tune with labeled speech while freezing that network.Source: §6 · p. 4

• The passage identifies “fig3_4.jpg” as an available JPG figure file, without describing its contents.Source: http://arxiv.org/ps/2002.08126v1 · p. 7