Source-linked AI summary
Monolingual Data Selection Analysis for English-Mandarin Hybrid Code-switching Speech Recognition
Haobo Zhang, Haihua Xu, Van Tung Pham, Hao Huang, Eng Siong Chng
TL;DR
CSSR has limited code-switching training data, and simply merging languages can fail on within-utterance switches. This paper analyzes monolingual-data selection for English-Mandarin CSSR, including training without code-switching data. Its results show that effects depend on data matching and code-switching pattern, while a DNN-HMM system can recognize within-utterance switches when trained on monolingual data.
Problem
CSSR often has limited code-switching training data despite the availability of more monolingual data, and simple language merging can fail on within-utterance code-switching.
Source: §1 · p. 1Method
The paper analyzes how adding Mandarin or English monolingual data affects English-Mandarin CSSR, including a DNN-HMM system trained without code-switching data.
Source: §1 · p. 1Results
Mandarin additions improve Mandarin recognition but can hurt overall performance beyond a balance point; English-data effects vary by utterance pattern, while merging the two monolingual sets consistently improves performance.
Source: §5.1 · p. 3, §5.2 · p. 3, §6 · p. 4Takeaways & Limitations
The study concludes that monolingual-data benefits depend on data matching and the code-switching environment, and that HMM-DNN systems perform within-utterance code-switch recognition.
Source: §7 · p. 4The proposed explanations for Mandarin-data gains and the balance point are hypotheses that need further support.
Source: §5.1 · p. 3
Abstract
from arXiv · showhide
In this paper, we conduct data selection analysis in building an English-Mandarin code-switching (CS) speech recognition (CSSR) system, which is aimed for a real CSSR contest in China. The overall training sets have three subsets, i.e., a code-switching data set, an English (LibriSpeech) and a Mandarin data set respectively. The code-switching data are Mandarin dominated. First of all, it is found using the overall data yields worse results, and hence data selection study is necessary. Then to exploit monolingual data, we find data matching is crucial. Mandarin data is closely matched with the Mandarin part in the code-switching data, while English data is not. However, Mandarin data only helps on those utterances that are significantly Mandarin-dominated. Besides, there is a balance point, over which more monolingual data will divert the CSSR system, degrading results. Finally, we analyze the effectiveness of combining monolingual data to train a CSSR system with the HMM-DNN hybrid framework. The CSSR system can perform within-utterance code-switch recognition, but it still has a margin with the one trained on code-switching data.
1. Introduction
Because code-switching speech recognition has limited training data and simple language merging can fail on within-utterance switches, the paper studies how to select monolingual data for an English-Mandarin contest system.
1. Introduction
- The contest provides 200 hours of English-Mandarin code-switching data, 500 hours of Mandarin data, and permitted use of 960 hours of LibriSpeech English.The work uses a DNN-HMM system and focuses on exploiting the monolingual data for the contest setting.Source: §1 · p. 1
- Using all three data sets worsened results, motivating data selection; adding Mandarin helped Mandarin recognition with insignificant English degradation, but gains reversed beyond a limit.Adding LibriSpeech English consistently degraded results in the initial experiments.Source: §1 · p. 1
- The study also tests training without code-switching data, finding that merging pure Mandarin and English data consistently benefits both languages in the DNN-HMM system.The authors contrast this with end-to-end systems that fail on utterances containing code-switching words when trained with monolingual data.Source: §1 · p. 1
- Here, data selection means reasonably exploiting monolingual data for a CSSR system with a specific code-switching pattern.Source: §1 · p. 1
2. Data description
The data description covers the released training and test sets and the English-word distribution within the code-switching sets.
2. Data description
- The code-switching data are Mandarin-dominated, and most utterances contain only one English word.Source: §2 · p. 2
- English-word distributions are similar between training set T1 and dev set E1, while E2 and E3 are more alike.Source: §2 · p. 2, Figure 1 · p. 2
- Table 1 describes the data sets and their lengths; the released resources include three training sets, three test sets, two development sets, and one evaluation set.Source: Table 1 · p. 2, §2 · p. 2
3. Experimental setup
The experiments use Kaldi with LF-MMI-trained TDNNf acoustic models, specified acoustic features, and a language-dependent phone set.
3. Experimental setup
- Acoustic models use the LF-MMI criterion over a 15-layer TDNNf, with 40-dimensional MFCCs concatenated with 100-dimensional i-vectors.Each layer is decomposed as 1536x256 and 256x1536, with a 256-dimensional bottleneck; ReLU and data augmentation are used.Source: §3 · p. 2
- The vocabulary is approximately 200k words, including about 121k Mandarin and 79k English words.Source: §3 · p. 2
- The language-dependent phone set has 210 Mandarin initials and finals and 42 English phones; the released trigram language model covers approximately 813k Mandarin and 116k English words.Source: §3 · p. 2
4. Baseline
The baseline uses CS200 code-switching training data and compares lexicon choices and training-data combinations using MER.
4. Baseline
- MER is a token error rate combining Chinese character error rate and English word error rate.Source: §4 · p. 2
- Manual pronunciation checking improved the Chinese entries in lexicon L0 to produce L1, which gave the best results and was used in later experiments.The initial lexicon decomposed out-of-vocabulary Chinese words into characters to address a mismatch with language-model segmentation.Source: §4 · p. 2, Table 2 · p. 2
- Table 3 results show that adding Mandarin data significantly reduces MER, whereas adding Libri English data degrades results.These opposing outcomes motivate examining data-selection effects.Source: Table 3 · p. 2, §4 · p. 2
5. Analysis of adding monolingual data
The analysis finds that monolingual data can help some code-switching recognition outcomes, but its effects depend on data matching and utterance code-switch patterns. Mandarin data helps mainly on Mandarin-dominated utterances, while adding LibriSpeech English can degrade results.
5.1. Mandarin data
- The Mandarin analysis incrementally added Man500 data while keeping CS200 fixed, selecting utterances to cover speakers before setting the amount and randomizing the final selection.Source: §5.1 · pp. 2–3
- About 400 hours of Mandarin data produced the best eval MER, while adding more beyond a balance point could hurt overall performance.The authors attribute gains to similarity between the Mandarin portions of Man500 and CS200, but say the proposed explanations need further support.Source: Figure 2 · p. 3, §5.1 · p. 3
- Mandarin-data gains declined as utterances contained more English words; gains were unstable for groups with at least three English words, and the four-or-more group degraded on eval.Across the category analysis, gains were mainly from utterances with a single English word, which dominate the data.Source: §5.1 · p. 3, Figure 3 · p. 3
5.2. English data
- For Mandarin-dominated utterances, adding English data did not help and more could degrade results, whereas English-rich utterances did not necessarily worsen as English data increased.The authors identify both data mismatch and code-switching pattern as possible factors, and say Mainland Chinese English data is needed for a solid conclusion.Source: §5.2 · p. 3
- As LibriSpeech English data increased, recognition degraded on both the Mandarin and English parts of the code-switching utterances.The authors attribute this pattern to mismatch with English spoken by Chinese Mainland speakers, potentially enlarged when English words are mixed with Chinese.Source: §5.2 · p. 3, Figure 4 · p. 4
6. Pure monolingual data
The section tests whether merging pure monolingual data can train an HMM-DNN code-switching system. Performance improves as the two monolingual sets are merged, but the evaluation-set gap from a system trained on code-switching data remains notable.
6. Pure monolingual data
- The study removes code-switching utterances from Man500, then incrementally adds LibriSpeech English data to the fixed Mandarin subset for HMM-DNN training.The section contrasts this test with prior E2E results, where monolingual-trained systems almost completely fail on within-utterance code-switching.Source: §6 · p. 4
- Merging Mandarin and English monolingual data consistently improves performance, with Mandarin CER and English WER both decreasing.The English WER decrease stops above 50%, while overall WER continues to decrease; the balance point depends on the code-switching pattern.Source: §6 · p. 4
- The monolingual-trained system improves recognition, but its eval-set performance remains notably behind the system trained with CS200 code-switching data.Figure 6 compares MER, CER, and WER across test sets and compares eval MER between the two systems.Source: Figure 6 · p. 4, §6 · p. 4
7. Conclusion
The conclusion finds that monolingual data can help code-switching recognition, but matching and the amount used matter. It also reports that HMM-DNN systems trained without real code-switching data can recognize within-utterance switching, though their behavior differs from E2E systems.
7. Conclusion
- HMM-DNN systems trained without real code-switching data still perform well on within-utterance code-switching recognition.The conclusion says this observation differs from what was seen under the End-to-end ASR framework.Source: §7 · p. 4
- Monolingual data helps, but data matching is crucial and the useful amount depends on the code-switching environment.The study also characterizes the training and testing code-switching data as Mandarin-dominated.Source: §7 · p. 4