Source-linked AI summary
Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairs
Gio Paik, Hyunseo Shin, Soungmin Lee
TL;DR
CS-ASR must handle scarce multilingual code-switching data whose language-pair requirements expand rapidly. The paper tests whether fine-tuning, model merging, and domain generalization transfer capabilities from seen to unseen pairs, finding partial transfer but limited overall generalization.
Problem
CS-ASR lacks resources across diverse language pairs, making pair-specific support difficult as the number of possible pairs grows with the supported languages.
Method
Using WHISPER-MEDIUM, the paper evaluates fine-tuning, model merging, and domain generalization across English, Korean, Japanese, and German, including unseen-pair evaluation datasets.
Results
Bilingual fine-tuning partially transfers to unseen pairs, while existing merging and domain-generalization methods remain insufficient to fully close the seen–unseen performance gap.
Takeaways & Limitations
Generalization to unseen language pairs requires complex task-level adaptations beyond simple domain-level transfer.
Takeaways & Limitations
The training and evaluation data are limited in quantity and diversity, and the unseen-pair experiments do not test entirely unseen languages.
Abstract
from arXiv · showhide
Automatic Speech Recognition (ASR) has become a key technology for human--AI interaction. However, code-switching ASR (CS-ASR) remains particularly challenging due to the severe scarcity of multilingual CS speech resources across diverse language pairs. Existing approaches primarily improve CS-ASR performance through synthetic CS speech generation or pair-specific fine-tuning on limited bilingual datasets. Nevertheless, these approaches face an inherent scalability limitation, as support for CS must be developed separately for language pairs whose number grows combinatorially with the number of supported languages. In this work, we investigate whether CS capabilities learned from a limited set of seen language pairs can generalize to unseen language pairs through model merging and domain generalization methods. Our experiments show that merged bilingual CS-ASR models modestly generalize to unseen language pairs, suggesting limited transfer of bilingual CS capabilities across language pairs.
1. Introduction
The paper asks whether code-switching capabilities learned from accessible language pairs can generalize to unseen pairs, addressing the scarcity and quadratic growth of pairwise CS data. It explores model merging and domain generalization, but finds only limited gains from existing methods.
- CS-ASR remains difficult because multilingual speech resources are scarce, while possible language pairs grow quadratically with the number of supported languages.
- The study investigates transfer from KO-EN, JA-EN, and DE-EN to unseen KO-JA and KO-DE pairs using model merging and domain generalization.
- Fine-tuning on one language pair yields slight gains on other pairs, while model merging and domain generalization can further improve unseen-pair recognition.
- The gains remain limited, motivating methods tailored to CS-ASR rather than naive applications of existing merging or domain-generalization techniques.
- The experiments systematically test whether CS-ASR capabilities learned from specific language pairs generalize to unseen language pairs.
- The work constructs Korean-Japanese and Korean-German CS speech evaluation datasets, including an open-source Korean-Japanese dataset.
2. Related Works
Prior CS-ASR work largely targets individual seen language pairs because non-English pair resources are scarce. This paper builds on model merging and domain generalization, whose use in multilingual CS-ASR remains limited or unexplored.
- Public datasets for non-English pairs such as Korean-Japanese and Korean-German remain virtually nonexistent, motivating alternative data-generation approaches.
- Synthetic speech generation and concatenated monolingual segments have been explored, but the related passage truncates before stating their full limitations.
- Existing CS-ASR studies primarily improve recognition for individual language pairs rather than generalizing capabilities to unseen pairs.
- Model merging combines independently fine-tuned models in weight space, with Task Arithmetic, TIES-Merging, and DARE representing progressively developed strategies.
- Model merging has been applied to multilingual language modeling and other ASR robustness settings, but its application to multilingual CS-ASR remains unexplored.
- Domain generalization methods use optimization strategies such as meta-learning, gradient agreement, gradient-variance alignment, and gradient-guided annealing.
3. Experiments
The experiments evaluate transfer from English-centric seen pairs to unseen Korean-Japanese and Korean-German pairs using fine-tuning, model merging, and domain generalization. Transfer is modest: TIES is stable, while DG generally provides little improvement and unseen-pair error remains high.
- The study uses WHISPER-MEDIUM to test whether capabilities learned from seen English-centric pairs improve recognition on unseen non-English-centric pairs.
- Evaluation combines three seen bilingual datasets with newly constructed Korean-Japanese and Korean-German evaluation sets, using Mixed Error Rate as the metric.
- Fine-Tuning: Fine-tuning on one CS dataset improves the corresponding pair and sometimes other pairs, but different-pair adaptation does not consistently produce large MER reductions.
- Merging: TIES achieves the most stable merging behavior, with the KO-EN + JA-EN merge reaching average MER 0.14 on seen bilingual tasks while remaining competitive on unseen pairs.
- Merging: Task Arithmetic and DARE are unstable in three-model merges, and naive parameter arithmetic can cause severe degradation.
- Domain Generalization: Fishr improves average unseen-pair MER by 0.08 over fine-tuning on all seen pairs, but absolute MER remains above 0.3.
- Domain Generalization: The limited DG gains may reflect a mismatch between shared-mechanism assumptions and pair-dependent changes in the target language distribution.
4. Fine-Tuned Parameter Analysis
Layer-wise MAV analysis measures how extensively parameter rows change during KO-EN CS fine-tuning. Higher encoder and decoder layers show larger modifications, suggesting adaptation concentrates in deeper representations.
- Figure 1 reports the percentage of rows whose parameter-delta MAV exceeds 5 × 10^-5 for each encoder and decoder projection matrix.
- Higher MAV ratios indicate that a larger portion of parameters in the corresponding module changed substantially during code-switching fine-tuning.
- Both encoder and decoder modifications increase progressively in higher layers, while lower layers remain relatively stable.
- The pattern suggests that code-switching adaptation primarily occurs in deeper semantic and linguistic representations rather than low-level acoustic processing.
5. Limitations
The study’s unseen-pair performance remains limited, and its data, language coverage, and model scope constrain the conclusions. The authors call for broader resources and methods specifically designed for code-switching generalization.
- Average MER on unseen pairs reaches 0.32 after model merging and domain generalization, remaining far from practical deployment.Fine-tuning on seen pairs achieves sub-0.2 MER.
- The training and evaluation data have limited quantity and diversity, including 582 JA-EN utterances from one speaker and two evaluation speakers per unseen pair.These constraints restrict linguistic and speaker diversity.
- Unseen-pair experiments combine languages observed during training and therefore exclude generalization to entirely unseen languages such as French or Chinese.
- Experiments use only WHISPER-MEDIUM, so broader understanding requires evaluation on larger Whisper variants and recent audio language models.
- Future work should improve multilingual CS data and develop objectives and adaptation methods that explicitly address language-pair shifts.The authors also propose analyzing code-switching-related model components and expanding language-pair resources.
6. Conclusion
The paper evaluates whether CS-ASR learned from selected language pairs can generalize to unseen pairs without pair-specific data. Bilingual fine-tuning transfers partially, but existing merging and domain-generalization methods do not fully close the seen–unseen performance gap.
- The study evaluates fine-tuning and model merging with WHISPER-MEDIUM across English, Korean, Japanese, and German code-switching settings.
- Bilingual CS-ASR fine-tuning partially transfers to unseen language pairs, while existing merging and domain-generalization methods remain insufficient.
- Layer-wise MAV analysis finds code-switching adaptation concentrated in higher encoder and decoder layers.The authors suggest that unseen-pair generalization requires task-level adaptations beyond simple domain-level transfer.
- Robust CS-ASR will require architectures and adaptation strategies specifically designed for transferable code-switching capability.
A. Experimental Details
The experiments use WHISPER-MEDIUM with fixed fine-tuning configurations, AdamW optimization, cosine decay, and linear warmup. Training schedules differ between single-pair and multi-pair or domain-generalization experiments.
- WHISPER-MEDIUM is used as the backbone model for all experiments.
- Single-language-pair fine-tuning uses batch size 8 for 73 training steps.
- Multi-pair fine-tuning and domain-generalization experiments use batch size 9 for 195 training steps.
- Optimization uses AdamW with cosine learning-rate decay and a linear warmup covering 10% of total training steps.
B. Parameter Analysis
Figures 2 and 3 compare layer-wise row-level MAV threshold ratios between pretrained WHISPER-MEDIUM and JA-EN or DE-EN code-switching models.
- Figure 2 reports layer-wise row-level MAV threshold ratios between pretrained WHISPER-MEDIUM and JA-EN code-switching.
- Figure 3 reports layer-wise row-level MAV threshold ratios between pretrained WHISPER-MEDIUM and DE-EN code-switching.