PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates

Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, Nikolaos Aletras

arXiv:2512.04844v2cs.CLcs.AI

TL;DR

Adapting instruct LLMs to underrepresented languages is constrained by costly specialized instruction data and the risk of forgetting source abilities during unlabeled-data adaptation. SSU uses source-informed parameter scoring and column-wise freezing before target-language continual pre-training. Across five languages and two model scales, it preserves source capabilities while reaching target-language performance that matches or surpasses full fine-tuning.

  • Problem

    Specialized instruction-tuning data is often unavailable or costly for underrepresented languages, and unlabeled target-data adaptation risks degrading source capabilities.

    Source: §1 · pp. 1–2

  • Method

    SSU uses a small source-data sample to identify important parameters, freezes selected columns, and applies the mask during continual pre-training on unlabeled target-language data.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Across five languages and 7B and 13B models, SSU best preserves core source capabilities over strong baselines while target-language proficiency matches or surpasses full fine-tuning.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    SSU offers a pathway to expand instruct models' linguistic reach without costly specialized target-language data.

    Source: §1 · p. 2, §7 · p. 9

    The study does not exhaustively tune hyperparameters for all methods, so reported performance may not be each method's global optimum across languages.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce Source-Shielded Updates (SSU), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.3% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.

1 Introduction

The paper addresses adapting instruct models to underrepresented languages with unlabeled target text while preserving source-language capabilities. It introduces SSU, which uses source-informed parameter selection and column-wise freezing; experiments report strong target-language performance with source capabilities largely preserved.

1 Introduction

  • Specialized instruction-tuning data is often unavailable or costly for underrepresented languages, while unlabeled-target adaptation can trigger forgetting that harms general-purpose abilities.Prior post-hoc approaches described here largely fail to prevent substantial degradation of these capabilities.Source: §1 · pp. 1–2
  • SSU identifies source-critical parameters from a small source-data sample, then freezes selected columns during continual pre-training on unlabeled target-language data.Its column-wise design is intended to preserve feature transformations while adaptation proceeds.Source: §1 · p. 2
  • The experiments span five typologically diverse languages and 7B and 13B models, evaluating both English source-language abilities and target-language performance.Evaluated source abilities include chat, instruction-following, safety, general generation, and classification.Source: §1 · p. 2
  • At two model scales, SSU outperforms baselines on core instruction-following and safety tasks while achieving leading target-language proficiency and nearly preserving source-language performance.The analysis also reports that SSU avoids linguistic code-mixing associated with other methods.Source: §1 · p. 2
  • The authors report that analysis supports column-wise freezing and source-data-driven scoring, and that SSU avoids code-mixing seen in other methods.They connect avoiding code-mixing with stronger source chat and instruction-following abilities.Source: §1 · p. 2

2 Related Work

The section reviews approaches to catastrophic forgetting and selective parameter updates, distinguishing dynamic from static selection. SSU is presented as a source-focused static approach that selects parameters using source data rather than random or target-data signals.

Language Adaptation

  • The section frames language adaptation as target-language continual pre-training that can degrade original capabilities, especially chat and instruction-following.These abilities are described as vital to instruct models' general-purpose functionality.Source: §2 · p. 2

Catastrophic Forgetting

  • Catastrophic-forgetting methods include regularization, replay, model merging or post-hoc pruning, architecture-based methods, and selective parameter updates.Selective updates restrict which existing weights change during training.Source: §2 · pp. 2–3

Selective Parameter Updates

  • Selective updates can mitigate forgetting; dynamic methods change trainable parameters during training, whereas static methods define them beforehand.The section notes that static selection can integrate readily with existing pipelines and combine with methods such as regularization and replay.Source: §2 · p. 3
  • SSU uses a small source-data sample to identify and freeze source-critical parameters, unlike approaches that select randomly or rely on target data.This source-focused static strategy is presented as a distinct selection criterion for language adaptation.Source: §2 · p. 3

3 SSU: Selective Parameter Updates via Importance Freezing

SSU uses source calibration data to score important parameters, aggregates scores by input-feature column, freezes selected columns, and applies the resulting mask during continual pre-training on unlabeled target text.

3 SSU: Selective Parameter Updates via Importance Freezing

  • SSU adapts an instruct model during continual pre-training on raw, unlabeled target-language data, aiming to limit forgetting while retaining target-task gains comparable to full fine-tuning.The framework takes a calibration set and a freezing ratio as inputs and proceeds in three stages.Source: §3 · p. 3

3.1 Parameter Importance Scoring

  • Wanda scores each weight using its magnitude multiplied by the L2 norm of its input activations, calculated from source calibration data.The scoring excludes embeddings and the language-modeling head, which are updated during training.Source: §3.1 · p. 3

3.2 Column-wise Masking

  • SSU aggregates parameter-importance scores by input-feature column and freezes the top k% of columns, using 50% by default.The method uses column-wise freezing to preserve complete feature pathways; it also describes each bias element as its own column.Source: §3.2 · pp. 3–4

3.3 Continual Pre-training

  • During causal-language-modeling CPT, the static mask zeros gradients for frozen columns while other parameters update on unlabeled target data.The authors state that this preserves knowledge in critical input-feature pathways and mitigates catastrophic forgetting.Source: §3.3 · p. 4

4 Experimental Setup

Experiments use OLMo 2 Instruct at two model scales and five underrepresented target languages. The setup specifies source calibration and target CPT data, compares multiple adaptation baselines, and evaluates source and target tasks across several benchmarks.

4.1 Source Models

  • The experiments use OLMo 2 Instruct models at 7B and 13B scales, chosen for instruction-following ability and documented training data.The paper also reports that findings generalize to OLMo 3.Source: §4.1 · p. 4, §4.2 · p. 4

4.2 Target Languages

  • Five typologically diverse target languages are underrepresented in source-model training data, appearing at least 840 times less frequently than English in Common Crawl.The languages were selected in part for available datasets with consistent task formulations, though data variations prevent direct cross-language performance comparisons.Source: §4.2 · p. 4

4.3 Calibration and Training Data

  • Calibration uses 500 randomly selected samples of length 2,048 from OLMo 2's original instruction-tuning data, while CPT uses 200M MADLAD-400 tokens per language.The CPT subset is described as clean and sampled following prior work.Source: §4.3 · p. 4

4.4 Baselines

  • Baselines include the unadapted source model, full fine-tuning, AdaLoRA, HFT, GMT, and calibration-free SSU-Rand and SSU-Mag variants.The paper also compares SSU with LoTA and S2FT in an appendix; it reports that only SSU consistently combines source preservation with target gains among those methods.Source: §4.4 · pp. 4–5

4.5 Evaluation Benchmarks and Metrics

  • Evaluation covers chat, instruction-following, safety, source-language generation and reasoning, and target-language translation, summarization, reading comprehension, and reasoning.Metrics include instruction-following accuracy, win rate, Likert score, exact match, chrF++, and accuracy across the specified benchmarks.Source: §4.5 · p. 5

5 Results

SSU-Wanda performs strongly across target-language, chat, safety, and source-language tasks while limiting the forgetting associated with full fine-tuning. Its target-language results are highly competitive with full fine-tuning, and source-data-driven scoring outperforms random or magnitude-only freezing.

5 Results

  • 5.9% (7B) and 4.7% (13B) are SSU-Wanda's average relative performance drops from Source on chat and instruction-following benchmarks; FFT's overall average performance loss is 34.1% (7B) and 32.3% (13B).The chat and instruction-following tasks measure core instruct-model capabilities, including multi-step reasoning and following complex constraints.Source: §5 · p. 5
  • Across aggregated task averages, SSU-Wanda delivers full fine-tuning benefits without associated catastrophic forgetting and consistently outperforms other evaluated methods.Table 2 aggregates average performance across all languages per task.Source: Table 2 · p. 6, §5 · p. 7
  • SSU-Rand is 18.2% (7B) and 16.0% (13B) below Source, while SSU-Mag drops 23.0% (7B) and 21.7% (13B), underscoring the importance of source-data-driven scoring.Both variants outperform FFT but substantially underperform SSU-Wanda.Source: §5 · pp. 5–6
  • 0.1% (7B) and 2.0% (13B) are SSU-Wanda's safety performance drops from Source, while FFT and target-data-driven GMT drop safety scores by up to 10.2%.Other selective methods partially preserve source performance but lag behind SSU-Wanda.Source: §5 · p. 6
  • Monolingual SUM, MRC, and MMLU relative drops never exceed 2.0% (7B) and 1.0% (13B), while target-to-English MT improves up to 52.3% relative to Source.SSU-Wanda ranks first across all source-language benchmarks for 7B and leads in MT and MMLU for 13B.Source: §5 · p. 6
  • SSU-Wanda's target-language performance is highly competitive with FFT, exceeding it on every benchmark for 7B models and half of benchmarks for 13B models.The reported performance difference from FFT is consistently minimal.Source: §5 · p. 6

6 Analysis

The analysis probes how freezing ratios, freezing structure, parameter scoring, and calibration data affect source retention and target-language adaptation. Across coding, low-resource, and response-quality analyses, SSU-Wanda retains source capabilities while remaining competitive on target tasks, with trade-offs that vary by task and setting.

Parameter Freezing Ratio

  • Higher freezing ratios generally improve source chat and safety but often reduce target-task performance; the paper recommends 50% for balance, at least 60% for source priorities, and at most 40% for target priorities.MT is an exception: its performance declines as the freezing ratio increases, particularly beyond 37.5%, because it requires knowledge of both source and target languages.Source: §6 · p. 7

Alternative Freezing Methods

  • Column-wise freezing leads on chat, safety, and source-language tasks, while remaining competitive on target tasks with only a 1.2-point MT drop versus element-wise freezing.The analysis attributes the advantage to preserving entire feature pathways; row-wise freezing does not protect a complete output feature.Source: §6 · p. 7, Table 3 · p. 8

Alternative Importance Scoring Methods

  • 4.0% (SSU-Wanda), 4.3% (SSU-SparseGPT), and 3.5% (SSU-FIM) are the average drops on monolingual source tasks, versus 13.5% (SSU-Rand) and 17.9% (SSU-Mag).The comparison supports using source-data-driven importance scores rather than calibration-data-free variants.Source: §6 · p. 8

Calibration Data for Parameter Importance Scoring

  • Alpaca calibration performs comparably to the original calibration data, with a maximum difference of 1.0; reducing calibration samples from 500 to 128 yields negligible differences.SSU-Wanda requires source calibration data for its importance scoring, but the analysis finds robustness to these calibration-data choices.Source: §6 · p. 8

Universality of Shielded Parameters

  • 10.4% (SSU-Wanda, 7B) and 4.2% (SSU-Wanda, 13B) are the reported average relative HumanEval performance drops; FFT drops 49.7% for 7B.SSU-Wanda maintains coding proficiency near Source, suggesting retention of capabilities shared across languages.Source: §6 · pp. 8–9, Table 4 · p. 8

Ultra-low-resource Settings

  • At 20M target-language tokens, SSU-Wanda ranks best or second-best in 10 of 13 tasks and exceeds FFT on SUM (26.6), MRC (.357), and MMLU (.352).In this 10%-of-default-data setting, it also retains core capabilities better than baselines, though AdaLoRA retains source capabilities strongly while acquiring target language more weakly.Source: §6 · p. 9, Table 5 · p. 9

Qualitative Analysis

  • 1.0% of SSU 7B responses are code-mixed on average, compared with 6.4% for HFT and 16.9% for GMT.The analysis links the lower code-mixing rate with more robust source-language retention and stronger chat performance.Source: §6 · p. 9

7 Conclusion

The paper introduces Source-Shielded Updates (SSU) for adapting instruct models with unlabeled target-language data. Across five languages and two model scales, it reports stronger retention of source capabilities than strong baselines alongside target-language performance matching or surpassing full fine-tuning.

7 Conclusion

  • The conclusion frames SSU as a scalable approach to language adaptation using only unlabeled target-language data.Its scope is instruct-model adaptation under that data setting.Source: §7 · p. 9
  • SSU adapts instruct models using unlabeled target-language data and shields source knowledge before adaptation with importance scoring and column-wise freezing.It identifies critical source parameters using a small source calibration set, then freezes the corresponding structural units during adaptation.Source: §7 · p. 9
  • Across five languages and 7B and 13B models, SSU best preserves source capabilities such as instruction-following and safety while achieving target-language proficiency matching or surpassing full fine-tuning.The conclusion describes this as a pathway to expand instruct models’ linguistic reach without costly specialized data.Source: §7 · p. 9

Limitations

The evaluation prioritizes scalable, LLM-specific baselines and does not exhaustively tune every method. The authors note that reported results may not reflect each method’s global optimum.

Limitations

  • The comparison focuses primarily on selective parameter update methods HFT and GMT; orthogonal approaches such as model merging or replay are not explored in combination with SSU.The paper identifies synergy with orthogonal strategies as a future direction and says some foundational continual-learning methods are computationally prohibitive at billion-parameter scale.Source: Limitations · p. 10
  • Because exhaustive hyperparameter searches across 100+ adapted models were not performed, reported performance may not be each method’s global optimum.The study uses an established protocol and a 50% freezing ratio for comparison, with sensitivity analysis provided for freezing ratios.Source: Limitations · p. 10

Ethical Considerations

The authors flag possible bias and safety risks when adapted instruct models are deployed in underrepresented languages. They call for further scrutiny, responsible deployment, and continued cross-lingual safety-alignment research.

Ethical Considerations

  • Deployment of adapted instruct models in underrepresented languages requires further scrutiny because they may reinforce harmful biases or introduce safety vulnerabilities that standard benchmarks fail to detect.The authors identify responsible deployment and continued cross-lingual safety-alignment research as priorities.Source: Ethical Considerations · p. 10

A Evaluation Details

Evaluation uses LLM judges for two benchmarks and language-specific prompt templates for each task.

A Evaluation Details

B.1 General Setup

Across supplementary comparisons, SSU-Wanda balances source preservation and target-language gains, while its static mask adds little training overhead and its calibration results are robust to sample size.

B.1 General Setup

  • SSU’s static mask adds zero backward-pass overhead, and pre-computation is less than 0.3% of total training time.In OLMo 2 7B experiments, SSU training took 34,156s versus 34,979s for full fine-tuning.Source: §B.1 · p. 17

B.2 Alternative Scoring Method Implementations

  • SSU-SparseGPT scores weights using the average squared activation of the corresponding input neuron, while SSU-FIM uses average squared loss gradients as a Fisher-score approximation.Source: §B.2 · pp. 17–18

C Supplementary Results

  • Supplementary results cover English chat and instruction-following, English safety, general English, and target-language benchmarks; selected tasks are averaged across three runs.IFEval, AE2, MTB, GSM8K, MT, and SUM use three runs; the other results are single-run deterministic evaluations.Source: C Supplementary Results · p. 18, Table 10 · p. 20, Table 9 · p. 20

D.1 Impact of Freezing Ratio on Baselines

  • Across freezing ratios, HFT follows SSU’s trend but does not surpass it; GMT often performs strongly on target tasks above 60% but remains low on monolingual source tasks.Source: §D.1 · p. 22, Figure 3 · p. 23

D.2 Calibration Data Size for Parameter Importance Scoring

  • Reducing calibration data from 500 examples to 128 changes results minimally, with a maximum difference of 1.2 points on IFEval.Source: §D.2 · p. 22, Table 13 · p. 23

D.3 Comparison to Additional Baselines

  • SSU-Wanda combines a 4.0% average source drop with 30.7% target gains, while LoTA at 90% sparsity has a 7.6% drop and 23.9% gains.S2FT preserves source capabilities with a 3.3% drop but yields 2.3% target gains.Source: §D.3 · p. 22
  • LoTA at 50% sparsity reaches 31.7% target gains but incurs a 19.9% monolingual source drop; at 37.5% sparsity, the drop reaches 25.4%.Expanding S2FT’s parameter budget still does not match SSU; at rank 64 it has an 8.2% source drop and 15.0% target gains versus SSU-Wanda’s 4.0% drop and 30.7% gains.Source: §D.3 · p. 22

D.4 Generalization to OLMo 3 Architecture

On OLMo-3-7B-Instruct, SSU preserves source performance while improving target-language performance. The paper also analyzes its structural protection and evaluates target-language instruction-following with a translation proxy.

D.4 Generalization to OLMo 3 Architecture

  • On OLMo-3-7B-Instruct, SSU has 1.1% average source-task degradation and 17.3% target-task gains; GMT has 4.5% degradation and 20.5% gains.S2FT’s source degradation is -0.1%, but its target-language performance is 1.4% below the original model.Source: §D.4 · p. 23, Table 16 · p. 24

D.6 Proxy Evaluation on Target-Language Instruction-following

  • The target-language instruction-following proxy repurposes machine translation with target-language instructions and scores monolingual output, reference sentence count, and language-appropriate full stops.Source: §D.6 · pp. 24–25
  • SSU-Wanda has the highest strict accuracy across evaluated languages and model scales, but models perform better translating target-to-English than English-to-target.Source: §D.6 · p. 25, Table 17 · p. 25
  • SSU-Wanda’s best models achieve only 1.8% (7B) and 3.5% (13B) adherence to the full-stop criterion.The authors report that unlabeled target-language corpora provide weak formatting signals and the prompts do not explicitly request punctuation.Source: §D.6 · pp. 25–26

E Extended Related Work

  • The authors situate SSU among continual-learning methods and distinguish soft regularization, hard structural protection, and adaptive or hybrid approaches.They note that task identifiers required by some foundational methods do not fit unified general-purpose instruct models, while one-shot static scoring is viable at LLM scale.Source: E Extended Related Work · pp. 26–27
  • SSU uses source-data importance scores to identify critical parameters and column-wise structural freezing to protect them during adaptation.The authors frame this as static hard parameter protection for instruct-LLM language adaptation.Source: §D.5 · p. 23, E Extended Related Work · p. 26

F License

  • The listed models and datasets permit academic research; OLMo 2 and Olmo-3 models use Apache License 2.0, while the listed datasets have several licenses.Source: F License · p. 27

G Use of Generative AI Tools

Loading 2512.04844v2…