Source-linked AI summary
Towards End-to-End Code-Switching Speech Recognition
Ne Luo, Dongwei Jiang, Shuaijiang Zhao, Caixia Gong, Wei Zou, Xiangang Li
TL;DR
Code-switching ASR faces a need for expert linguistic knowledge, while earlier end-to-end work used artificially concatenated monolingual utterances rather than spontaneous code-switching speech. This paper applies a hybrid CTC-Attention system to Mandarin-English code-switching and studies modeling units, language identification, and decoding on SEAME. The system outperforms most traditional code-switching ASR systems on that corpus.
Problem
Code-switching ASR requires expert linguistic knowledge, and the only prior end-to-end system used artificially concatenated monolingual utterances rather than spontaneous code-switching speech.
Source: §1 · p. 1Method
The paper applies a joint CTC-Attention framework to Mandarin-English code-switching and studies modeling units, language identification, and decoding strategies on SEAME.
Source: §1 · p. 1Results
The system outperforms most traditional code-switching speech recognition systems on SEAME.
Source: §5 · p. 4Takeaways & Limitations
The authors report that Mandarin-character/English-subword units, an added LID objective, and word-aware beam search reduce MER in code-switching ASR.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Code-switching speech recognition has attracted an increasing interest recently, but the need for expert linguistic knowledge has always been a big issue. End-to-end automatic speech recognition (ASR) simplifies the building of ASR systems considerably by predicting graphemes or characters directly from acoustic input. In the mean time, the need of expert linguistic knowledge is also eliminated, which makes it an attractive choice for code-switching ASR. This paper presents a hybrid CTC-Attention based end-to-end Mandarin-English code-switching (CS) speech recognition system and studies the effect of hybrid CTC-Attention based models, different modeling units, the inclusion of language identification and different decoding strategies on the task of code-switching ASR. On the SEAME corpus, our system achieves a mixed error rate (MER) of 34.24%.
1. INTRODUCTION
Code-switching ASR must handle multiple languages within an utterance, while system building faces data, co-articulation, and linguistic-knowledge challenges. This paper applies joint CTC-Attention to Mandarin-English speech on SEAME and studies modeling units, language identification, and decoding strategies.
1. INTRODUCTION
- Code-switching speech contains more than one language within an utterance, and its ASR poses training-data, co-articulation, and expert-knowledge challenges.The paper says earlier work mainly focused on the first two challenges.Source: §1 · p. 1
- End-to-end ASR predicts graphemes or characters directly from acoustics, reducing system-building effort and avoiding the need for expert linguistic knowledge.The paper presents this as an attractive approach for code-switching ASR.Source: §1 · p. 1
- The only prior end-to-end code-switching system used artificially concatenated monolingual utterances rather than spontaneous code-switching speech.Earlier work also addressed language modeling and co-articulation challenges using other approaches.Source: §1 · p. 1
- This study applies a joint CTC-Attention framework to Mandarin-English code-switching on SEAME and examines modeling units, language identification, and decoding strategies.Source: §1 · p. 1
2. END-TO-END FRAMEWORK
The section describes CTC and attention as complementary end-to-end recognition approaches. CTC marginalizes frame-level alignments, while attention encodes acoustics and predicts output symbols from the acoustic sequence and previous predictions.
2.1. Connectionist temporal classification (CTC)
- CTC treats output predictions at different time steps as conditionally independent given aligned inputs.Source: §2.1 · p. 1
- CTC computes a label sequence probability by marginalizing over frame-length paths that collapse to that sequence after repeated labels and blanks are removed.The blank label allows frames and labels to share a length, and the method does not require prior input-output alignment.Source: §2.1 · pp. 1–2
2.2. Attention based models
- The attention encoder maps acoustic input to a higher-level representation, and its decoder predicts the next symbol from the acoustics and previous outputs.Source: §2.2 · p. 2
- Attention selects or weights input frames for each output element; content-based and location-based mechanisms are two main types.Location-awareness is added to the latter to better fit speech recognition.Source: §2.2 · p. 2
3. METHODS
The methods combine CTC and attention, compare character and subword modeling units, add language identification, and constrain decoding with word information. The section specifies how each component is incorporated into the recognition system.
3.1. Hybrid CTC-Attention based models
- The hybrid objective combines attention and CTC losses; CTC encourages monotonic alignment and attention learns label dependencies.The paper describes CTC as an auxiliary encoder-training task that helps attention converge.Source: §3.1 · p. 2
3.2. Acoustic modeling units
- The study compares character units for both languages with Mandarin characters plus English subwords, using BPE to build the English subword set.BPE repeatedly replaces the most frequent symbol pair with a new symbol until the chosen subword-set size is reached; English words are segmented greedily using the longest matching subword.Source: §3.2 · p. 2
3.3. Joint language identification (LID)
- LID-Label adds CH and EN output symbols, which the decoder predicts at code-switching points before subsequent characters or subwords.Source: §3.3 · pp. 2–3
- LID-MTL adds an encoder-sharing network trained on aligned language-ID sequences with cross-entropy loss, alongside attention and CTC.The proposed framework shares the encoder across the attention decoder, CTC, and LID components.Source: §3.3 · p. 3, Figure 2 · p. 3
3.4. Decoding Strategy
- Because some high-ranked subword sequences did not form valid words, the authors built a dictionary of frequent English words and words in the SEAME training set.Source: §3.4 · p. 3
- Decode1 filters for valid-word candidates after beam search, whereas Decode2 discards invalid-word candidates during beam search.Source: §3.4 · p. 3
4. EXPERIMENTS
Experiments on SEAME evaluated a hybrid CTC-Attention system across data, training, modeling units, language identification, and decoding choices. Mixed Chinese-character and English-subword units performed better than character-only units, while LID-MTL improved over baseline and Decode2 had lower final MER than Decode1.
4.1. Data
- SEAME contains 66.8 hours of spontaneous Mandarin-English conversation and interview speech from 155 Singaporean and Malaysian speakers.Only 12% of transcribed segments were Mandarin monolingual and 6% were English monolingual; the data were divided into train, development, and test sets using several criteria.Source: §4.1 · p. 3
4.2. Training
- The hybrid model uses a shared two-convolution, four-layer bidirectional GRU encoder with two time-pooling layers and a one-layer GRU decoder; all experiments also use an RNNLM.The encoder reduces input sequence length fourfold; the RNNLM has two 800-unit LSTM layers and is trained on SEAME's training set.Source: §4.2 · p. 3
4.3. Choice of MTL weight
- The Character-Character model achieved its lowest MER with λ = 0.8, the weight chosen for subsequent experiments.The authors report this as consistent with multi-task training performing better than the attention objective alone.Source: §4.3 · p. 3, Table 2 · p. 4
4.4. Effect of using different modeling units
- Character-Character models performed worse than both Character-Subword models, and the 500-subword model performed worse than the 200-subword model.The authors attribute the mixed-unit advantage to closer correspondence between output-unit and audio-segment length, and the 500-subword result to sparser distributions on this small corpus.Source: §4.4 · p. 4, Table 3 · p. 4
4.5. Joint language identification
- LID-MTL significantly improved over baseline, but LID-Label showed little improvement; LID weights above 0.1 made performance much worse.The authors suspect frequent language switching makes language-ID prediction harder; they used a LID weight of 0.1 in subsequent experiments.Source: §4.5 · p. 4, Table 4 · p. 4
4.6. Effect of different decoding strategy
- Decode2 had lower final MER than Decode1, although its stronger beam-search restriction could remove correct candidates after early decoding mistakes.The authors suggest the model may struggle to relate subword parts and that SEAME's small size is one possible explanation.Source: §4.6 · p. 4, Table 5 · p. 4
5. CONCLUSIONS
The hybrid CTC-Attention end-to-end Mandarin-English system outperforms most traditional code-switching speech recognition systems on SEAME. The authors report gains from Chinese-character/English-subword units, adding language identification, and word-informed beam search, and plan broader investigation.
5. CONCLUSIONS
- The hybrid CTC-Attention end-to-end Mandarin-English system outperforms most traditional code-switching speech recognition systems on SEAME.Source: §5 · p. 4
- The combination of Chinese characters and English subwords is the optimal acoustic modeling unit choice.Source: §5 · p. 4
- Adding a third cross-entropy objective for language identification significantly reduces mixed error rate.Source: §5 · p. 4
- A decoding strategy that considers word information during beam search also helps reduce mixed error rate.Source: §5 · p. 4
- The authors plan a more thorough investigation of their strategies and evaluation on other code-switching tasks, potentially involving more than two languages.Source: §5 · p. 4