Source-linked AI summary
SyRA-CS: Synthetic-to-Real Adaptation for Code-Switched ASR with Meta-Learning
Fadel Thior, Ahmer Tabassum, Haania Siddiqui, Hanan Aldarmaki, Thamar Solorio
TL;DR
Real code-switched speech is scarce, and synthetic speech does not fully match it. SyRA-CS uses pair-specific Reptile meta-learning on synthetic speech before adapting to one hour of labeled real target speech. It lowers WER versus continual fine-tuning in all 16 model–pair comparisons, with gains varying across models and metrics.
Problem
Real code-switched speech is scarce, while synthetic speech does not fully match real speech, motivating study of adaptation under limited labeled target data.
Source: §1 · p. 1Method
SyRA-CS learns a pair-specific Reptile initialization for each pretrained model from the corresponding synthetic CS-FLEURS subset, then adapts it on one hour of labeled real target speech.
Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4Results
SyRA-CS lowers WER versus continual fine-tuning in all 16 model–pair comparisons and CER in 15, with relative macro WER reductions of 5.3% to 17.7%.
Source: §7 · p. 8Takeaways & Limitations
The gains extend across the four tested architectural settings, with the largest gains for XLS-R and smaller but consistent gains for the other three models.
Source: §7 · p. 8Each architectural setting uses a single pretrained model, so differences in Reptile’s effectiveness cannot be attributed to architecture alone.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Code-switched (CS) automatic speech recognition (ASR) remains difficult because real CS speech is scarce and synthetic speech does not fully match it. We investigate whether synthetic CS speech is more useful for learning an adaptation-friendly initialization than for ordinary continual fine-tuning. We propose SyRA-CS, a pair-specific synthetic-to-real adaptation framework based on Reptile meta-learning. For each language pair and pretrained model, we meta-train on the corresponding synthetic CS-FLEURS subset and then adapt using one hour of labeled real target speech. We evaluate four architectural settings: Encoder-CTC, FastConformer-RNN-T, Attention Encoder-Decoder (AED) and Speech-LLM. Experiments cover on Arabic-, French-, German-, and Hindi-English, against synthetic-only transfer, real-only fine-tuning, and continual fine-tuning. SyRA-CS improves over continual fine-tuning on WER in all 16 model-pair comparisons and on CER in 15 of 16, with relative macro WER reductions of 17.7% for XLS-R and 5.3-5.8% for the other three models. The gains hold across the tested models but vary in size across architectures, language pairs and metrics.
1 Introduction
Real code-switched speech is scarce, and synthetic speech does not fully match it. The paper tests whether pair-specific Reptile meta-learning can produce an initialization that adapts better to real speech under a limited data budget.
1 Introduction
- Real CS corpora are costly to collect and transcribe, and models trained for one condition often degrade on another as switching patterns vary.These differences include speaker community, accent, discourse function, topic, and orthographic convention.Source: §1 · p. 1
- Synthetic CS speech can expand supervision cheaply, but its cleaner acoustics and lower speaker and accent variability may limit transfer to real speech.The paper asks whether meta-learning on synthetic speech yields an initialization that adapts better with limited labeled real speech.Source: §1 · p. 1
- SyRA-CS learns a separate Reptile initialization for each language-pair and pretrained-model combination, then adapts it using one hour of labeled real target speech.The study frames this as data-efficient synthetic-to-real adaptation, not a new optimizer or broad benchmark of meta-learning algorithms.Source: §1 · pp. 1–2
- The evaluation spans Encoder–CTC, FastConformer–RNN-T, attention encoder–decoder, and Speech-LLM systems, comparing synthetic-only transfer, real-only fine-tuning, and continual fine-tuning.The study also characterizes target-set switching ratios and checks robustness on the two smallest evaluation sets.Source: §1 · p. 2
2 Related Work
Related work covers multilingual speech pretraining, synthetic data for code-switched ASR, and meta-learning for low-resource adaptation. SyRA-CS combines same-pair synthetic code-switched episodes with first-order Reptile.
2.1 Multilingual self-supervised
- Self-supervised speech models such as wav2vec 2.0, HuBERT, and XLS-R, alongside multilingual speech corpora, have strengthened multilingual transfer.The cited resources include FLEURS, Common Voice, VoxPopuli, and MMS.Source: §2.1 · p. 2
2.2 Code-switched ASR and synthetic data
- Synthetic code-switched data is used to address scarce real speech, but its value depends on data quality and downstream task (Hamed et al., 2025).Prior work explores synthetic phrase-level switching, text-first adaptation, and benchmarks including CS-FLEURS (Yan et al., 2025).Source: §2.2 · p. 2
2.3 Meta-learning for ASR
- Prior meta-learning work adapts ASR models to low-resource or code-switched speech, while SyRA-CS samples episodes from synthetic speech of the same language pair and uses first-order Reptile.Earlier code-switched meta-transfer work drew episodes from monolingual corpora (Winata et al., 2020); related approaches include Meta-Adapter and SMILE (Hsu and Lee, 2024).Source: §2.3 · pp. 2–3
3 Data, Model, and Evaluation
The evaluation applies Reptile across four ASR architectures, using pair-specific synthetic CS-FLEURS sources and one-hour real target adaptation sets. It scores normalized hypotheses with WER and CER.
3.1 Backbones and architecture scope
- Four pretrained systems span Encoder–CTC, FastConformer–RNN-T, attention encoder–decoder, and Speech-LLM architectures, with Reptile operating on each model’s trainable parameters and transcription objective.The study cautions that differences in pretraining, tokenization, parameter scope, and decoding prevent attributing cross-model differences to architecture alone.Source: §3.1 · p. 3
3.2 Source tasks: CS-FLEURS data
- Each model–pair run samples synthetic episodes only from that pair’s CS-FLEURS subset, learning a separate initialization rather than a joint multilingual one.The synthetic subsets contain 4.51 hours for Arabic-English, 10.28 hours for French-English, 9.37 hours for German-English, and 6 hours for Hindi-English.Source: §3.2 · p. 4
3.3 Target tasks: MUCS and Switchlingua data
- Each target pair uses a randomly selected one-hour labeled adaptation subset and a corresponding real code-switched test set.SwitchLingua’s human recordings use generated code-switched text prompts; MUCS Hindi-English comes from human-spoken tutorials.Source: §3.3 · p. 4
3.4 Metrics and Text normalization
- WER and CER are computed after shared normalization of references and hypotheses, without system-specific post-processing.The shared steps include Unicode, markup, punctuation, whitespace, and Latin-script case normalization.Source: §3.4 · p. 4
4 Meta Learning Adaptation
SyRA-CS learns pair- and model-specific initializations through Reptile episodes on synthetic speech, then adapts them to one-hour real subsets. Its first-order update avoids differentiating through the inner optimization trajectory.
4 Meta Learning Adaptation
- For each model and language pair, synthetic minibatch episodes start from that pair’s current initialization and perform model-specific inner gradient updates.The resulting initialization is adapted to the one-hour real subset and evaluated on held-out real examples, which are excluded from training and adaptation.Source: §4 · pp. 4–5
4.1 Why First-Order Meta-Learning?
- Reptile updates the initialization using parameter displacements across episodes, avoiding differentiation through inner-loop optimization.This reduces the derivative overhead of standard second-order MAML, but repeated inner-loop training still has a computational cost.Source: §4 · p. 5, §4.1 · p. 5
5 Experiments and Results
SyRA-CS is compared with three alternative training regimes under controlled, pair-specific conditions. It outperforms continual fine-tuning across nearly all model–pair–metric comparisons, with gains varying by model and evaluation set.
5.1 Baselines and Reproducibility
- All regimes share a pretrained checkpoint, held-out evaluation set, and—when using real adaptation—the same one-hour target subset; SyRA-CS uses Reptile episodes from synthetic data.The four regimes are synthetic-only transfer, real-only fine-tuning, CFT, and SyRA-CS.Source: §5.1 · p. 5
5.2 Main results
- SyRA-CS has the lowest error in 31 of 32 model–pair–metric cells, improving on CFT in all 16 WER and 15 of 16 CER comparisons.Table 2 compares the four training regimes within each pretrained model.Source: §5.2 · p. 5, Table 2 · p. 6
- 17.7% macro WER and 15.6% macro CER reductions give XLS-R the largest gains; its CFT baseline is also the weakest of the four models.XLS-R macro WER falls from 0.4740 to 0.3900, and macro CER from 0.1885 to 0.1590.Source: §5.2 · p. 6
- Nemotron 3.5 and Audex-2B reduce macro WER/CER by 5.3%/4.4% and 5.8%/7.5%; Whisper-small reduces them by 5.5% and 8.3%.Whisper-small’s Arabic-English WER decreases from 0.352 to 0.348 while CER increases from 0.162 to 0.181.Source: §5.2 · p. 6
- Synthetic-only transfer has the highest error in 28 of 32 cells, while CFT beats real-only fine-tuning in 29 of 32.CFT and SyRA-CS use the same synthetic subset and one-hour real subset, so their comparison isolates training strategy rather than additional data.Source: §5.2 · p. 6
5.3 Code-switching profile and Robustness checks
- Code-switched utterances make up 76.9%–88.5% of the target test sets; Hindi-English has 2.996 switches per utterance, but these statistics do not identify what drives gains.The code-switched shares are 88.5% for Arabic-English, 88.4% for French-English, 76.9% for German-English, and 80.8% for Hindi-English.Source: §5.3 · p. 6, Figure 5 · p. 7
- Across test-set subsets excluding German-English and then both German- and Arabic-English, all four models retain positive macro WER and CER gains; these are descriptive checks, not significance tests.Variability across training seeds and adaptation subsets is not assessed.Source: §5.3 · p. 7
6 Discussion
The results support using synthetic CS speech to learn an adaptation-friendly initialization rather than relying on ordinary sequential fine-tuning. The largest Reptile benefit appears for XLS-R, though the study cannot attribute that difference causally to its pretraining setup.
6 Discussion
- SyRA-CS consistently improves over continual fine-tuning, supporting the hypothesis that synthetic CS speech is more effective for learning an adaptation-friendly initialization than for ordinary sequential fine-tuning.The closest matched baseline uses the same synthetic source set and real target subset.Source: §6 · p. 7
- XLS-R has the largest gain: real-only fine-tuning nearly fails on Hindi-English at WER 0.980, while SyRA-CS reduces macro WER by 17.7%.The authors suggest the meta-learned initialization matters most when the backbone lacks task-specific supervision, but note that one model per architecture prevents a causal claim.Source: §6 · p. 8
- SyRA-CS uses no more data than continual fine-tuning, making it a low-cost alternative under a one-hour real-data budget if additional meta-training compute is acceptable.Whisper’s Arabic-English WER improves while CER worsens, which the authors attribute to Arabic orthographic variation not collapsed by shared normalization.Source: §6 · p. 8
7 Conclusion
SyRA-CS is a pair-specific Reptile framework for synthetic-to-real code-switched ASR adaptation. Across four pretrained models, it improves WER over continual fine-tuning in every model–pair comparison, with gains remaining positive when the two smallest evaluation sets are excluded.
7 Conclusion
- SyRA-CS lowers WER versus continual fine-tuning in all 16 model–pair comparisons and CER in 15, with relative macro WER reductions of 5.3% to 17.7%.The evaluation spans Encoder–CTC, FastConformer–RNN-T, AED, and Speech-LLM models using the same synthetic source data and one-hour real adaptation budget within each model.Source: §7 · p. 8
- Gains are largest for XLS-R, smaller but consistent for the other three models, and remain positive when the two smallest evaluation sets are excluded.The authors describe this as consistent with Reptile’s intended role of learning an initialization that specializes readily to real CS speech.Source: §7 · p. 8
- Future work will add compute-matched controls, multi-seed significance testing, CS diagnostics for all models, additional language pairs, and more spontaneous speech corpora.Source: §7 · p. 8
Limitations
The study’s evidence is bounded by pair-specific adaptation, fixed data budgets, limited model coverage, and single-run evaluation. These constraints limit claims about multilingual transfer, data requirements, architecture effects, Reptile-specific benefits, and statistical reliability.
Limitations
- The experiments evaluate pair-specific synthetic-to-real adaptation, not a shared multilingual meta-initialization or universal cross-pair transfer.Reptile is applied within each language pair rather than jointly over pairs.Source: Limitations · p. 8
- Because synthetic-source datasets and the one-hour real adaptation budget are fixed, the study does not establish minimum data requirements or how the two sources trade off.Controlled data-scaling experiments are needed to characterize requirements and diminishing returns.Source: Limitations · p. 8
- Each architectural setting has one pretrained model, preventing attribution of gain differences to architecture alone; alternative meta-learning algorithms are also not compared.The study therefore cannot determine whether the benefit is specific to Reptile.Source: Limitations · p. 8
- Results use one training seed and one random one-hour adaptation subset per pair, while two evaluation sets retain only 104 and 218 test utterances.The subset analysis is descriptive rather than a significance test; multi-seed runs and paired utterance-level statistics are needed before smaller gains can be considered reliable.Source: Limitations · pp. 8–9