Preserve to Expand: Source-Aware Adaptation and Model Merging for Multilingual LLM-Based ASR

1 Introduction

• Expansion is difficult with limited data and computation because new-language adaptation must preserve previously supported languages, while replay-based retention requires storing and reusing source-language data.Fine-tuning on the new language can cause catastrophic forgetting, including in connector-based Speech LLMs for ASR.Source: §1 · p. 1
• Model merging combines learned updates without additional training, but independently learned updates can interfere; how they behave in connector-based ASR expansion without source replay remains unclear.TIES-Merging trims small parameter changes and resolves sign conflicts between task vectors.Source: §1 · p. 1
• The study asks how adaptation and merging balance target acquisition with source retention, and how independent target updates can be merged for multiple new languages.It focuses on connector-based Speech LLMs without source-language replay or gradient-based source adaptation during expansion.Source: §1 · pp. 1–2
• SSU-Wanda+TIES gives the strongest single-target adaptation–retention trade-off; joint update merging retains source performance stably, while recursive adaptation can improve target performance under favorable orders but is order-sensitive.Target-only fine-tuning and standard TIES merging learn new languages but often overwrite source-language performance.Source: §1 · p. 2
• The experiments compare direct target fine-tuning, source-aware selective adaptation, and TIES-based merging, extending the analysis from single-target to multi-target expansion.They use three multilingual base connectors and six target languages under the same target-language adaptation budget.Source: §1 · p. 2

2.1 Multilingual ASR Expansion

• Prior multilingual ASR expansion work addresses forgetting through replay-based retention, parameter-regularized adaptation, and decoder-specific optimization; this work instead studies target-update construction and merging without source replay or gradient-based source adaptation.The analysis covers single- and multi-target connector expansion with lightweight source-language calibration.Source: §2.1 · p. 2

2.2 Model Merging for Adaptation

• Weight averaging and task arithmetic suggest separately learned capabilities can sometimes be combined without additional training, but update interference remains a concern in source-free connector-based multilingual ASR expansion.TIES-Merging trims small changes and resolves sign conflicts; prior multilingual, speech, and task-specific LLM merging work has not fully examined this ASR setting.Source: §2.2 · pp. 2–3

2.3 Parameter-Selective Adaptation

• Parameter-selective adaptation protects parameters important to prior capabilities, and Source-Shielded Updates estimate source-language importance to freeze units during target-language fine-tuning.Here, SSU-style masking uses a lightweight source-language calibration set only to construct connector-adaptation masks.Source: §2.3 · p. 3

3.1 Connector-Based ASR Backbone

• A frozen speech encoder produces acoustic representations that a trainable connector maps into the frozen LLM’s embedding space for autoregressive transcription.The method applies adaptation and merging only to connector parameters.Source: §3.1 · p. 3

3.2 Target Adaptation with Source Calibration

• SSU uses source-language calibration data to identify important connector columns and freezes selected columns during target-language adaptation.The protected set comprises the top-ρ proportion of columns; SSU-RAND instead selects columns uniformly at random.Source: §3.2 · pp. 3–4
• SSU+FT first obtains an SSU-adapted connector, then applies standard unmasked target-language fine-tuning to test the value of that initialization.The method also evaluates SSU-MAG, which scores parameters by weight magnitude, and SSU-Wanda, whose scores use source-calibration activations.Source: §3.2 · p. 4

3.3 Target-Update Model Merging

• TIES trims small update entries, elects a sign per parameter dimension, and merges only updates agreeing with that sign before adding the scaled result to the base connector.For a single target update, there are no cross-update sign conflicts, so TIES reduces to trimming before scaling; for multiple targets, updates are independently obtained from the same base connector.Source: §3.3 · p. 5
• Multi-target TIES tests whether independently learned target updates can be combined without recursively fine-tuning across multiple target languages.In this setting, TIES can resolve sign conflicts among target-language updates before aggregation.Source: §3.3 · p. 5

4.1 Experimental Setting

• The experiments use Common Voice, three multilingual base connectors spanning Germanic, Romance, and Slavic, and six target languages; base training and target adaptation each use five hours per language.SSU-Wanda additionally uses five minutes per source language from validation data for activation collection and mask construction, without source-language replay or gradient-based source adaptation.Source: §4.1 · p. 5

4.2.1 Target Adaptation and Source Retention

• On the Germanic base, Target FT reduces target WER from 102.70% to 27.86% but raises source WER from 22.57% to 69.73%; source WER also rises by +55.54% and +54.49% for Romance and Slavic.Standard TIES retains most of this source degradation, and SSU-Wanda+FT alone also leaves substantial source degradation.Source: §4.2.1 · p. 6
• SSU-Wanda+TIES gives the strongest single-target trade-off, keeping Germanic and Romance source-WER increases to +0.24% and +0.69% while reducing target WER from 102.70% to 35.82% and 111.16% to 37.36%, respectively.For Slavic, it reduces source degradation from +53.95% under TIES to +6.68% while improving target WER from 110.15% to 51.69%.Source: §4.2.1 · p. 6, Table 3 · p. 6

4.2.2 Effect of Source-Aware Protection

• SSU-Wanda+TIES achieves the lowest source WER and overall WER across all three base groups, while sometimes giving up target-language gains achieved by SSU-Rand or SSU-Mag.For Slavic, SSU-Rand+TIES has 41.69% target WER and +22.54% source degradation, versus 51.69% and +6.68% for SSU-Wanda+TIES; source activations give Wanda a more conservative mask than random selection or magnitude alone.Source: §4.2.2 · pp. 6–7

4.2.3 Base–Target Compatibility

• Serbian reaches 27.63% target WER with +4.17% source degradation, but same-group Latin-script Polish and Slovenian are harder, and Slovenian causes greater source degradation.Kyrgyz has lower source degradation than Slovenian (+2.26% versus +21.11%) despite remaining difficult in target WER, so group or script match alone does not rank expansion difficulty.Source: §4.2.3 · p. 7, Table 4 · p. 7

4.3.1 Joint Update Merging versus Recursive Adaptation

• Joint TIES merging keeps source WER close to the base connector while reducing target WER from 102.70% to 64.93% for Germanic, 111.16% to 67.22% for Romance, and 110.15% to 97.41% for Slavic.Recursive adaptation can learn targets more aggressively but is more order-sensitive; source-aware joint variants can protect source languages further while retaining less target improvement than TIES.Source: §4.3.1 · pp. 7–8, Table 5 · p. 8
• Recursive FT then SSU's configured-order results vary by base: it achieves the best target WER for Germanic and Slavic, but its Romance result is much worse.Recursive rebased SSU improves target WER in some settings but causes large source degradation for Romance and Slavic.Source: §4.3 · p. 7, §4.3.1 · p. 8

4.3.2 Recursive Order Sensitivity

• Changing target order improves Romance overall WER from 57.81% to 31.82% and Germanic from 39.61% to 35.05%; Slavic recursive results vary from 60.06% to 80.86%.Recursive adaptation can outperform update merging under favorable target sequences, but is higher-variance and can accumulate interference under poorly matched orders.Source: §4.3.2 · p. 8, Table 6 · p. 8

5 Conclusion

• SSU-Wanda+TIES gives the strongest single-target adaptation–retention trade-off, while target fine-tuning and standard TIES often overwrite source-language performance.The conclusion reports that SSU-Wanda+TIES preserves source performance while substantially improving target performance.Source: §5 · p. 8
• For multi-target expansion, update merging is stable and source-preserving, while recursive adaptation can improve target performance under favorable orders but is highly order-sensitive.The conclusion says effective expansion requires careful coordination among target-language updates.Source: §5 · p. 8
• Effective multilingual ASR expansion requires balancing target acquisition, source retention, and interactions among target-language updates.Source: §5 · p. 8

6 Limitations

• Experiments adapt only the connector while keeping the speech encoder and LLM frozen, so conclusions may not directly generalize to systems updating those components or the full ASR model.Source: §6 · p. 9
• Multi-target experiments cover a limited number of target languages and target orders, restricting characterization of recursive expansion.Source: §6 · p. 9
• The paper does not propose an automatic target-order selection method; more languages and principled order-selection strategies are needed to fully characterize recursive expansion.Source: §6 · p. 9

7 Ethical Considerations

• Experiments use publicly available Common Voice speech data, without collecting new recordings or additional speaker metadata.Source: §7 · p. 9
• The paper discloses no personally identifiable information about speakers.Source: §7 · p. 9
• New-language improvements can come at the expense of previously supported languages, and average WER may obscure language and speaker-group disparities.Source: §7 · p. 9
• Per-language performance is reported for source and target languages to make differences across languages visible.Source: §7 · p. 9

A.1 Data Allocation and Calibration

• Each base has five source languages with five training hours per language, and each target adaptation run uses five hours.The Germanic and Romance bases expand to Norwegian, Italian, and Serbian; the Slavic base expands to Italian, Serbian, Polish, Slovenian, and Kyrgyz.Source: §A.1 · p. 11
• SSU-Wanda calibration uses five minutes from each source language, balanced by duration, for activation collection and mask construction—not replay or gradient-based source adaptation.Random and magnitude-based masking do not require activation collection.Source: §A.1 · p. 11

A.2 Expansion Protocols

• Single-target experiments independently add each target to a base connector and compare direct fine-tuning, unprotected TIES, and random, magnitude-based, or Wanda-based SSU variants.Source: §A.2 · p. 11
• Joint update merging adapts target-specific connectors independently from one shared base and merges their updates without further gradient training; recursive adaptation trains targets sequentially from the preceding connector.Each recursive stage uses only its current target language for gradient-based training.Source: §A.2 · p. 11
• Evaluated recursive orders are only a subset of possible orders, rather than an exhaustive enumeration.The configured orders differ across the Germanic/Romance and Slavic bases.Source: §A.2 · p. 11

• Tables 7–9 report full per-language WER for independent single-target expansion across Germanic, Romance, and Slavic languages.These tables complement the macro-averaged summary in Table 3.Source: §B.1 · p. 11, Table 7 · p. 12, Table 8 · p. 12
• Table 7 presents Germanic single-target expansion results in WER (%).Source: Table 7 · p. 12
• Table 8 presents Romance single-target expansion results in WER (%).Source: Table 8 · p. 12
• Table 9 presents Slavic single-target expansion results in WER (%).Source: Table 9 · p. 13

B.2 Full Multi-Target Expansion Results

• Tables 10–12 report WER (%) for Germanic, Romance, and Slavic multi-target expansion under evaluated target orders.They provide per-language details behind the final-stage summary in Table 5.Source: §B.2 · p. 14, Table 10 · p. 15, Table 11 · p. 16

B.3 Recursive Multi-Target Order

• Table 13 reports WER (%) trajectories for Recursive FT then SSU under additional target orders.Target mean is computed over accepted targets at each stage.Source: §B.3 · p. 18, Table 13 · p. 19

B.4 Target-Update Conflict

• Table 14 reports pairwise target-update conflict after TIES trimming; low cosine similarity and high sign disagreement indicate often-conflicting update directions.Source: §B.4 · p. 20, Table 14 · p. 21