Source-linked AI summary
Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition
Xinyuan Zhou, Emre Yılmaz, Yanhua Long, Yijie Li, Haizhou Li
TL;DR
Code-switching ASR must recognize speech containing multiple languages, but CS training resources are scarce. The paper proposes a Transformer with language-specific encoder and decoder attention modules, pre-trained on monolingual speech. On SEAME, it reports relative TER reductions of 10.2% and 10.8% on the two evaluation sets.
Problem
CS ASR must handle multiple languages in speech, and it is typically low-resource because code-switching acoustic and text resources are scarce.
Source: §1 · p. 1Method
Results
10.2% and 10.8% relative TER reductions on the evalman and evalsge sets, respectively, compared with the baseline system.
Source: §5 · p. 4Takeaways & Limitations
On SEAME, the proposed architecture outperforms the baseline Transformer with a single encoder and a state-of-the-art Kaldi-based hybrid ASR system.
Source: §6 · p. 4Evaluation of the MED Transformer on a recently released large-scale code-switching corpus remains future work.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Code-switching (CS) occurs when a speaker alternates words of two or more languages within a single sentence or across sentences. Automatic speech recognition (ASR) of CS speech has to deal with two or more languages at the same time. In this study, we propose a Transformer-based architecture with two symmetric language-specific encoders to capture the individual language attributes, that improve the acoustic representation of each language. These representations are combined using a language-specific multi-head attention mechanism in the decoder module. Each encoder and its corresponding attention module in the decoder are pre-trained using a large monolingual corpus aiming to alleviate the impact of limited CS training data. We call such a network a multi-encoder-decoder (MED) architecture. Experiments on the SEAME corpus show that the proposed MED architecture achieves 10.2% and 10.8% relative error rate reduction on the CS evaluation sets with Mandarin and English as the matrix language respectively.
1. Introduction
Code-switching ASR is a low-resource challenge, while existing system designs trade language-specific modeling against unified bilingual training. The paper proposes a pre-trained MED Transformer with language-specific encoder and decoder attention branches and unified decoding.
1. Introduction
- Transformer-based sequence-to-sequence acoustic modeling jointly encapsulates acoustic and language information in one network.Its multi-head attention supports parallel computation and long-context modeling without recurrence.Source: §1 · p. 1
- CS ASR is typically low-resource because acoustic and text resources containing code-switching are scarce.Prior approaches adapt high-resource acoustic models or use multi-task learning, while language-specific information was captured only in the decoder or output layer.Source: §1 · p. 1
- Independent language-dependent models offer precise language-specific modeling but require integrating different decoder distributions, whereas a single bilingual model is optimized to the average of two languages.The paper positions its architecture as combining aspects of both approaches.Source: §1 · p. 1
- The MED Transformer uses two symmetric language-specific encoder branches and corresponding decoder attention modules, pre-trained on monolingual data before optimization on limited code-switching data.The complete model is trained with CTC and Kullback-Leibler divergence criteria.Source: §1 · p. 1
2. Transformer Architecture
The Transformer encodes input sequences into high-level representations and autoregressively generates output tokens. Multi-head attention relates queries, keys, and values across subspaces, while positional encodings add sequence-position information.
2.1. Encoder-Decoder with Attention
- The encoder transforms input vectors into high-level representations, and the decoder generates output tokens autoregressively using the previous token as input.Encoder layers include self-attention and feed-forward sublayers; decoder layers add source-target attention over encoder outputs.Source: §2.1 · p. 2
2.2. Multi-Head Attention
- Multi-head attention projects queries, keys, and values into multiple subspaces, computes attention across them, then concatenates and projects the outputs.Scaling dot products by 1/√dk is used to avoid extremely small softmax gradients from large dimensions.Source: §2.2 · p. 2
2.3. Positional Encoding
- Positional encodings add position-dependent values to input dimensions because attention lacks recurrence and convolution for modeling input order.The model can thereby learn relative positions between sequence elements.Source: §2.3 · p. 2
3. Multi-Encoder-Decoder Transformer
The MED Transformer uses separate language-specific encoders and decoder attention modules to represent and combine the two languages. Monolingual pretraining initializes these components, while multi-objective learning trains the encoders with CTC alongside attention.
3.1. Architecture
- Two symmetric language-specific encoders learn individual language attributes, and language-specific multi-head attention in the decoder combines their representations for code-switching speech.The design aims to enhance discrimination between encoder representations while integrating acoustic and code-switching context.Source: §3.1 · p. 2
- Each language-specific encoder and its corresponding attention module is pretrained on monolingual resources, then the MED model is fine-tuned on code-switching speech.Fine-tuning jointly uses CTC and KLD criteria.Source: §3.1 · p. 3
- Two 3 × 3 CNN layers with stride 2 downsample the input, reducing GPU memory use and the length difference from the label sequence.Source: §3.1 · p. 3
3.2. Multi-Objective Learning
- The CTC loss is included alongside the attention loss to jointly train the encoder in a multi-objective learning setup.Source: §3.2 · p. 3
- The multi-objective learning setup uses a tuning parameter λ ∈ [0, 1] and trains both language-specific encoders simultaneously using the CTC loss on their summed outputs.The paper calls this CTC-based multi-objective learning MOL.Source: §3.2 · p. 3
4. Experiments
Experiments use monolingual Mandarin and English corpora to pre-train components and SEAME for code-switching evaluation. The comparative study selects Transformer+MOL+LM as its baseline and tests language-specific encoder and decoder components.
4.1. Corpora
- SEAME provides 112 hours of Mandarin-English code-switching speech; AISHELL-2 and Librispeech supply 1K and 960 hours of monolingual speech for pre-training.The two evaluation sets use Mandarin and Singapore English as their respective matrix languages; 5% of training utterances form the development set.Source: §4.1 · p. 3
4.2. Experimental Setup
- The 12-layer encoder and 6-layer decoder use 256-dimensional model states, 4 attention heads, CTC weight 0.3, and 30 training epochs.The setup uses 9,230 combined Character-BPE units, 80-dimensional log Mel-filterbank features plus pitch, and a 10-hypothesis beam with CTC one-pass decoding and LM shallow fusion.Source: §4.2 · p. 3
4.3. Comparative Study
- Transformer+MOL outperforms Speech-Transformer and the LF-MMI hybrid, LM shallow fusion further improves it, and combined-corpus pre-training adds no improvement.Transformer+MOL+LM is therefore used as the baseline.Source: §4.3 · pp. 3–4
- The comparative study tests language-specific encoders with a standard decoder and language-specific decoder attention with a standard encoder.These systems isolate the contributions of the two types of language-specific modules.Source: §4.3 · p. 4
5. Results and Discussion
On SEAME, MED reduces TER relative to the baseline on both evaluation sets, with reductions across English and Mandarin tokens. Ablations and encoder-output distributions examine the role of language-specific components.
Results
- The proposed MED Transformer-based ASR system achieves relative TER reductions of 10.2% on evalman and 10.8% on evalsge, compared with the baseline system.Source: §5 · p. 4
- English-word TER reductions are 12.8% and 12.1%, while Mandarin-character reductions are 8.6% and 6.2% on evalman and evalsge, respectively.The authors report that English recognition is more challenging than Mandarin recognition.Source: §5 · p. 4
Ablation study
- MED reaches 16.7% TER on evalman and 23.1% on evalsge; the encoder-only M-En system performs similarly, while decoder-only M-De does not improve CS ASR.The authors describe the MED scores as the best SEAME results in the literature.Source: §5 · p. 4
Encoder responses
- For monolingual inputs, the corresponding language encoder shows greater output variance; both encoders show comparable activation for code-switching input.Figure 2 plots normalized final-layer values, with attention-vector indices horizontally and down-sampled frame indices vertically.Source: §5 · p. 4, Figure 2 · p. 4
6. Conclusion
The paper proposes MED for Mandarin-English code-switching ASR, using language-specific Transformer components pre-trained on monolingual speech. On SEAME, it outperforms the single-encoder Transformer baseline and a Kaldi-based hybrid system.
Conclusion
- MED combines two symmetric language-specific encoders with corresponding decoder attention blocks, pre-trained on large monolingual speech corpora.The architecture is proposed for Mandarin-English code-switching ASR.Source: §6 · p. 4
- On SEAME, MED outperforms the single-encoder Transformer baseline and a state-of-the-art Kaldi-based hybrid ASR system.Evaluation on a recently released large-scale code-switching corpus remains future work.Source: §6 · p. 4