Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages

1 Introduction

• Translate-then-SFT can yield limited multilingual gains, may degrade original-language capabilities, and can train on noisy machine translations.The introduction identifies translation accuracy, especially for low-resource languages, as a constraint on translated training text.Source: §1 · p. 1
• The introduction asks whether LLMs’ relatively strong resource-rich-language capabilities can improve abilities in other languages beyond translating training pairs.Figure 1 contrasts vanilla SFT, translate-then-SFT, and SDRRL, which additionally leverages a generated answer.Source: Figure 1 · p. 1, §1 · p. 2
• SDRRL uses model-generated resource-rich-language responses and translated target-language counterparts as supervision for sentence-level knowledge self-distillation.It also adds a small amount of machine-translated parallel data to align linguistic representation spaces and mitigate translation-noise effects on generation.Source: §1 · p. 2
• SDRRL improves target-language performance across comprehension and generation tasks while preserving capabilities in resource-rich languages.Experiments use LLaMA-2-7B and SeaLLM-7B, with English as the resource-rich language, and distill into 14 other languages.Source: §1 · p. 2

Multilingual Language Models

• Despite multilingual pretraining and instruction tuning, popular LLMs’ capabilities lag in languages with fewer resources amid imbalanced training data.The related-work discussion notes that these models nevertheless retain multilingual understanding and generation capabilities.Source: §2 · p. 2

Cross-Lingual Transfer

• Cross-lingual transfer work moves skills from source languages through prompting, machine translation, synthetic-data distillation, and language-similarity approaches.The paper distinguishes its focus on leveraging proficiency in the resource-rich language to improve other languages.Source: §2 · p. 2

Cross-Lingual Alignment

• Cross-lingual alignment methods construct multilingual supervision, while the paper argues that a model’s own high-resource-language responses can also provide effective signals.The paper reports that its self-distillation improves multilingual performance while helping maintain performance in the original resource-rich languages.Source: §2 · p. 2

Knowledge Distillation

• This work applies self-distillation between linguistic representation spaces within the same LLM.The related-work discussion describes self-distillation as transferring knowledge between model components or stages.Source: §2 · p. 3

3 Method

• The method section frames SDRRL as sentence-level self-distillation followed by parallel translation-based instruction data for multilingual generation.Its two-part overview contrasts constructing a transfer set from resource-rich-language responses with using parallel translation-based data.Source: §3 · p. 3

3.1 SFT and Translate-then-SFT Paradigm

• SFT trains model outputs toward ground-truth responses using cross-entropy, while translate-then-SFT translates each question-answer pair into the target language before training.The SFT dataset loss averages sample losses.Source: §3.1 · p. 3

3.2 Self-Distillation from Resource-Rich Languages (SDRRL)

• SDRRL treats resource-rich-language model responses as learning references and adds their translations to a transfer set for sentence-level knowledge distillation.The method also incorporates parallel translation-based instruction data to improve multilingual generation.Source: §3.2 · p. 3, §3 · p. 3

3.2.1 Transfer Set Construction

• The transfer set combines original instruction examples with model-generated responses, sampled equally from the original and generated datasets.The model generates a response for each original question before the two datasets are sampled.Source: §3.2.1 · p. 3

3.2.2 Transfer Set Translation

• Semantically identical but linguistically diverse samples support implicit alignment of language representations, while DTL and DLT support cross-linguistic generation and help mitigate off-target target-language generation.The four combinations vary whether questions and responses remain in the resource-rich language or are translated.Source: §3.2.2 · p. 4

3.2.3 Applying Code-Switching

• Code-switching adds token-level alignment by applying a bilingual tool only to questions, increasing language diversity without changing responses.A dictionary-matched question token is replaced with probability 0.15.Source: §3.2.3 · p. 4

3.2.4 Incorporating External Parallel Corpus

• A tiny external parallel corpus supplies machine-translation and sentence-completion instructions, adding non-synthetic natural target-language text to improve generation quality.The completion examples contain target-language sentences split at random positions, while translation examples include both directions.Source: §3.2.4 · p. 4, Table 1 · p. 4

3.2.5 Training Objective

• The final training dataset combines six subsets: DLL, DTL, DLT, DTT, Dmt, and Dcomp.The objective groups these subsets as U = {LL, TL, LT, TT, mt, comp}, with each D_d corresponding to its respective subset.Source: §3.2.5 · p. 5

4.1 Setup

• Experiments use LLaMA-2-7B with English as the resource-rich source and 14 target languages, evaluating comprehension, translation, summarization, and question answering.The benchmarks are BELEBELE, FLORES, XL-SUM, and MKQA.Source: §4.1 · p. 5

4.2 Main Results

• Across comprehension and generation tasks, SDRRL beats baselines in almost all target languages; FLORES gains reach about +6.0 BLEU in both directions and +4.0 COMET.The experiments also report approximately +1.5 BLEU on comprehension tasks.Source: Table 3 · p. 6, §4.2 · p. 6
• About +0.55 ROUGE-L F1 improvement is reported on XL-SUM, while T-SFT and CIT fall by -1.36 and -2.08 BLEU, respectively, on FLORES.XL-SUM requires longer-text generation; SDRRL still improves its average performance where CIT and XCOT decline amid translation quality and pipeline noise.Source: §4.2 · pp. 6–7, Table 4 · p. 7
• Only SDRRL preserves original English capabilities while the other methods show a performance drop relative to SFT.The paper notes that retention is more challenging for languages with unique alphabets, such as Japanese and Korean.Source: §4.2 · p. 7

4.3 Ablation Study

• Removing any single SDRRL component degrades performance; model-generated resource-rich-language responses and semantically equivalent cross-language sentences both contribute to gains.Removing model-generated responses particularly reduces performance in both target languages and English.Source: §4.3 · p. 7
• Translated and code-switched data provide limited ground truth that benefits generation, while parallel data alone severely degrades performance and regularizes against noisy translation.The authors describe these data as mitigating deterioration of the multilingual representation space caused by noisy machine-translated text.Source: §4.3 · pp. 7–8

4.4 Visualization of Representation Space for Source and Target Langauges

• After SDRRL, representations of semantically equivalent source- and target-language instructions move closer together in the t-SNE visualization.The analysis represents each sentence with the last token’s last hidden state, reduced from 4096 to 2 dimensions for visualization.Source: §4.4 · p. 8, Figure 2 · p. 8

4.5 SeaLLM as Different Backbone Model

• On SeaLLM-7B, SDRRL improves target-language average scores by +2.39 while maintaining English performance; baselines drop at least -2.02 versus vanilla SFT.The authors present this result as evidence of generalizability to another LLM backbone.Source: §4.5 · p. 8, Table 7 · p. 8

5 Conclusion and Future Work

• SDRRL uses the model's resource-rich-language responses and target-language counterparts as supervision for knowledge distillation.The method aims to align additional target languages with resource-rich languages.Source: §5 · p. 8
• Experiments across 16 languages on LLaMA-2 and SeaLLM compare SDRRL with various baselines.Source: §5 · p. 8
• SDRRL significantly enhances target-language performance while preserving source-language capabilities.The authors present this as highlighting LLMs' multilingual potential and informing further multilingual LLM research.Source: §5 · pp. 8–9

Limitations

• Experiments use only a single source language and a single target language.The authors suggest investigating mixtures of source and target languages and their mutual influences.Source: Limitations · p. 9
• The method relies on external machine translation and a small amount of machine-translated parallel data for its transfer set.Self-translation and target-language monolingual text are proposed as future directions.Source: Limitations · p. 9
• The method does not modify the LLM architecture; for extremely low-resource languages, architecture changes or additional data might be beneficial.The authors mention vocabulary expansion and continued pre-training as examples.Source: Limitations · p. 9

A Implementation Details

• SacreBLEU uses the signature “nrefs:1 | case:mixed | eff:no | tok:flores200 | smooth:exp | version:2.0.0”.Source: A Implementation Details · p. 12
• The Stanford Alpaca dataset has 52,002 entries and is licensed under CC BY-NC 4.0.For each target language, 1,000 machine-translation parallel-corpus entries are sampled from Opus100.Source: A Implementation Details · p. 12
• NLLB translation uses beam size 4, with other settings defaulting to Huggingface Transformers; baseline reimplementations use the same MT system for multilingual alignment data.Source: A Implementation Details · p. 12
• XL-SUM and MKQA do not cover all 16 experimental languages, and MKQA questions without ground truth are skipped.Source: A Implementation Details · p. 12

B More Detailed Results on SeaLLM

• The reported language set comprises three common South-east Asian languages: Indonesian, Thai, and Khmer.Source: B More Detailed Results on SeaLLM · p. 12
• SDRRL outperforms baselines on Indonesian, Thai, and Khmer, demonstrating generalizability across different LLMs.Source: B More Detailed Results on SeaLLM · p. 12
• The SeaLLM results include the BELEBELE benchmark.Source: Table 8 · p. 12
• The SeaLLM results include XL-SUM on the target language.Source: Table 9 · p. 13
• The SeaLLM results include the FLORES benchmark.Source: Table 10 · p. 13

• The comparison reports average SDRRL gains over SFT across NLU, including BELEBELE, and NLG, including FLORES, XL-SUM ROUGE-L, and MKQA.Source: Table 12 · p. 13, C Experiments with Non-English… · p. 13
• SDRRL’s distillation gains were smaller with French than English as source, attributed to lower LLaMA-2-7B French performance and reduced NLLB effectiveness.Source: C Experiments with Non-English… · p. 13
• The French-source experiment generated responses in French and replicated the Indonesian, Japanese, and Korean experiments.Source: C Experiments with Non-English… · p. 13

• After SDRRL, the Hungarian example’s severe off-target responses in English were effectively mitigated.Source: D Case Study · p. 13, Table 13 · p. 14
• In the Slovenian history example, SDRRL alleviated hallucination and produced more detailed, refined, and fluid content.Source: D Case Study · p. 14, Table 13 · p. 14
• In the Bulgarian example, SDRRL improved clarity and natural flow while eliminating the listed grammatical errors.Source: D Case Study · p. 14, Table 13 · p. 14

E Off-Target Issue Analysis

• The analysis evaluates LLaMA-2 responses on Dolly and Bactrain-X, tracking off-target occurrence rates across target languages.Source: E Off-Target Issue Analysis · p. 14, Figure 3 · p. 15
• SDRRL constructively mitigates off-target issues, supporting consistency between input and response languages.Source: E Off-Target Issue Analysis · pp. 14–15, Figure 3 · p. 15

F Potential Risks of Our Method

• Aligning African-language responses to English may lead them to adhere to English cultural practices and social norms.Source: F Potential Risks of Our Method · p. 15