PaperlayerView exact revision on arXiv

Source-linked AI summary

On the End-to-End Solution to Mandarin-English Code-switching Speech Recognition

Zhiping Zeng, Yerbolat Khassanov, Van Tung Pham, Haihua Xu, Eng Siong Chng, Haizhou Li

arXiv:1811.00241v2cs.CL

TL;DR

Conventional DNN-HMM recognition requires a mixed-language lexicon and separately optimized acoustic, language, and lexicon components, which can yield suboptimal performance. The paper combines E2E Mandarin-English recognition with data augmentation, BPE units, auxiliary LID multitask learning, and vocabulary expansion for neural language-model rescoring. LID-based multitask learning helps reduce Token Error Rate, and BPE subword units yield better recognition results than characters.

  • Problem

    Conventional DNN-HMM recognition requires a mixed-language lexicon and separately optimized acoustic, language, and lexicon components, which can yield suboptimal performance.

    Source: §1 · p. 1

  • Method

    The paper combines E2E Mandarin-English recognition with data augmentation, BPE units, auxiliary LID multitask learning, and vocabulary expansion for neural language-model rescoring.

    Source: §1 · p. 1

  • Results

    LID-based multitask learning helps reduce Token Error Rate, and BPE subword units yield better recognition results than characters.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The E2E strategy avoids lexicon modeling and jointly learns the recognition system's connected neural networks.

    Source: §1 · p. 1

    λ1 is a hyper-parameter controlling each model’s contribution and is in [0, 1].

    Source: §3.2 · p. 2

Abstract

from arXiv · show

Code-switching (CS) refers to a linguistic phenomenon where a speaker uses different languages in an utterance or between alternating utterances. In this work, we study end-to-end (E2E) approaches to the Mandarin-English code-switching speech recognition (CSSR) task. We first examine the effectiveness of using data augmentation and byte-pair encoding (BPE) subword units. More importantly, we propose a multitask learning recipe, where a language identification task is explicitly learned in addition to the E2E speech recognition task. Furthermore, we introduce an efficient word vocabulary expansion method for language modeling to alleviate data sparsity issues under the code-switching scenario. Experimental results on the SEAME data, a Mandarin-English CS corpus, demonstrate the effectiveness of the proposed methods.

1. Introduction

The paper frames Mandarin-English code-switching recognition as a challenge for conventional systems and pursues an end-to-end alternative. It combines baseline improvements with language-identification multitask learning and vocabulary expansion for language-model rescoring.

1. Introduction

  • Conventional DNN-HMM systems require a mixed-language lexicon and separately optimized components, motivating an end-to-end approach.The paper says separate optimization can lead to suboptimal performance; its E2E system avoids lexicon modeling and jointly learns connected neural networks.Source: §1 · p. 1
  • Data augmentation and BPE subword units form competitive E2E baselines, with BPE outperforming character units.The authors report BPE results are better than characters and that data augmentation is more effective for E2E than DNN-HMM systems.Source: §1 · p. 1
  • Language identification is added as an auxiliary multitask objective, and the authors report reduced Token Error Rate.The method explicitly learns LID alongside E2E speech recognition.Source: §1 · p. 1
  • A word-vocabulary expansion method inspired by targets cross-lingual data sparsity in code-switching language-model rescoring.Source: §1 · p. 1

2. Related work

Earlier code-switching speech recognition relied on conventional HMM-based systems, while end-to-end work explored CTC and attention approaches. The paper distinguishes its code-switching setting from prior monolingual E2E and multilingual recognition work.

2. Related work

  • Earlier code-switching speech recognition relied on GMM-HMM or DNN-HMM frameworks.Source: §2 · p. 1
  • E2E speech recognition developed along CTC and attention-based directions, with hybrid CTC/attention systems reported to outperform either method alone.Source: §2 · p. 1
  • The paper applies E2E to code-switching, unlike prior E2E work described as monolingual; simultaneous multilingual recognition is explicitly distinguished from code-switching.Source: §2 · pp. 1–2

3. Approaches to end-to-end CSSR

The approach section develops E2E code-switching baselines, adds language identification to hybrid CTC/attention recognition, and modifies neural language-model rescoring to address cross-lingual sparsity.

3.1. Approaches to develop E2E CSSR baselines

  • Speech-speed perturbation expands the training data threefold, while English BPE units paired with Mandarin characters address the imbalance in output-unit counts.The augmented data use 90%, 100%, and 110% speaking rates; the BPE baseline retains Mandarin characters.Source: §3.1 · p. 2

3.2. Multitask Learning of CSSR with LID

  • The hybrid E2E recognition objective combines attention and CTC losses, with λ1 controlling their contributions.Source: §3.2 · p. 2
  • LID is added as an auxiliary training task, using either the recognition attention model or an independent attention model.The framework illustrates shared attention on the left and separate attention on the right.Source: §3.2 · p. 2, Figure 1 · p. 2

3.3. Vocabulary expansion for neural language model

  • Vocabulary expansion treats words at cross-lingual transitions as infrequent and borrows probability mass from semantically close shortlist words.The method avoids learning a larger neural language model but requires prior semantic clustering.Source: §3.3 · p. 2

4. Experimental setup

Experiments use the SEAME spontaneous conversational Mandarin-English code-switching corpus, with separate Mandarin-biased and Southeast Asian accent English-biased test sets. The setup also specifies a DNN-HMM comparison baseline and the E2E model configuration.

4.1. Data

  • SEAME training utterances contain code-switching about 68% of the time; devman is Mandarin-biased and devsge is Southeast Asian accent English-biased, each with 10 balanced-gender speakers.Results are reported as token error rate (TER), counting Chinese characters and English words respectively.Source: §4.1 · pp. 2–3

4.2. DNN-HMM baseline system

  • The DNN-HMM comparison baseline is a Kaldi lattice-free MMI TDNN with six 1024-unit hidden layers, MFCCs plus i-vectors, and a training-transcription 4-gram LM.Its 40-dimensional MFCC features are combined with 100-dimensional i-vectors; outputs are language-independent senones.Source: §4.2 · p. 3

4.3. E2E ASR system setup

  • The ESPnet E2E system uses a one-layer CNN, six-layer 320-unit BLSTM encoder, and one-layer 320-unit LSTM decoder, with CTC weight 0.2.Four BPE vocabularies contain 1.9k, 2k, 3k, and 4k units; character coverage is 0.9995, yielding a minimum of 1806 mixed Mandarin-English symbols.Source: §4.3 · p. 3

5. Experimental results and analysis

The experiments compare E2E CSSR baselines, LID multitask variants, and N-best language-model rescoring. The combined approaches lower TER on both development sets, while analyses report baseline, substitution, and LID effects.

5.1. Results of the E2E CSSR system

  • E2E systems generally trail LF-MMI TDNN, while data augmentation reduces TER and 3k BPE outperforms characters.The authors note that further effort is needed for E2E on limited training data; 3k BPE gives the best results among the reported units.Source: Table 2 · p. 3, §5.1 · p. 3
  • Cross-lingual substitutions are about 10% on each test set, and SE→M is approximately 5% higher than SM→E.The paper interprets the direction difference as more English words being substituted by Mandarin characters than the reverse.Source: Table 3 · p. 3, §5.1 · p. 3
  • Most reported cross-lingual substitution examples involve short non-content words or acoustically similar modal words.Table 4 presents the top five examples on both test sets.Source: Table 4 · p. 3, §5.1 · p. 3

5.2. Effect of MTL with LID

  • Both LID-MTL methods improve TER in most cases and achieve their best results around λ2=0.2, with no obvious difference; gains are not significant and seem stronger for character systems.Figure 2 plots TER against λ2, with devman on the left axis and devsge on the right.Source: §5.2 · pp. 3–4, Figure 2 · p. 4, Table 5 · p. 4

5.3. Effect of the NLM vocabulary expansion

  • TER fell from 34.5% to 25.0% and from 46.5% to 34.5% on the two dev sets, approaching the strong LF-MMI TDNN system.The final results use N-best rescoring with N=30 and an NLM implemented as an RNN-LM.Source: Table 6 · p. 4, §5.3 · p. 4

6. Conclusion and future work

The paper proposes several methods to improve end-to-end Mandarin-English code-switching speech recognition, including changes to training data, subword units, multitask learning, and language-model rescoring. Future work will explore external monolingual data and incorporating a language model into the existing model.

6. Conclusion and future work

  • The proposed approaches include data augmentation, byte-pair encoding subword units for English, and language-identification-based multitask learning.Source: §6 · p. 4
  • Vocabulary expansion is used for neural language models to rescore N-best results.Source: §6 · p. 4
  • Future work will study leveraging external monolingual data and incorporating a language model into the existing model.Source: §6 · p. 4

Related papers

Loading 1811.00241v2…