LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoning

1 Introduction

• Translation-enhanced LLMs face reasoning challenges, including on LiveCodeBench-V5 and AIME2025.Tower-Plus-9B improves multilingual instruction-following but underperforms on these reasoning tasks.Source: §1 · p. 1, Figure 1 · p. 1
• Qwen3-XPlus starts from Qwen3 instruct models and applies layer-selective tuning to balance translation quality and reasoning without extra parameters.The approach tunes four layers closest to the embedding layer and fifteen layers further away.Source: §1 · p. 1, Figure 2 · p. 2
• Qwen3-XPlus uses 0.8 billion tokens of preprocessed parallel data, versus hundreds of billions or more for several compared translation models.The data are standardized and quality-controlled using NLLB and OPUS-100 sources.Source: §1 · p. 2
• Qwen3-XPlus gains 15+ spBLEU points and 40+ xComet points in low-resource Swahili, and improves average performance by over 1 point across 7 multilingual tasks.The paper also reports maintaining reasoning capabilities comparable to Qwen3 instruct models.Source: §1 · p. 2
• The proposed recipe is presented as reducing training complexity and making translation enhancement more accessible across languages while preserving reasoning capabilities.The paper introduces open-sourced Qwen3-XPlus-8B and Qwen3-XPlus-14B models.Source: §1 · p. 2

2.1 Massively Multilingual Translation with Large Language Models

• Massively multilingual translation uses one machine-translation model to handle many language directions.LLaMAX and Tower apply continued pretraining and instruction tuning with massive parallel and monolingual data.Source: §2.1 · pp. 2–3
• Large-data translation approaches can weaken general instruction-following and reasoning capabilities after training.The paper describes its approach as a parameter-efficient way to train a translation LLM while keeping or improving general ability.Source: §2.1 · p. 3

2.2 Parameter-Efficient Finetuning

• Parameter-efficient fine-tuning methods are grouped into additive, selective, reparameterized, and hybrid types; the proposed method is selective PEFT.The cited examples include adapters and soft prompts, selective methods, LoRA-family methods, and hybrids.Source: §2.2 · p. 3

3 Qwen3-XPlus Training Recipe

• The recipe trains an instruct model with instruction-formatted parallel data using layer-selective tuning after data construction.The overview presents these stages as the Qwen3-XPlus training pipeline.Source: §3 · p. 3

3.1 Training Data Construction

• NLLB and OPUS-100 parallel data are unified, cleaned, deduplicated, language-checked, quality-filtered, and converted to instruction format.Filtering includes SimHash deduplication and translation-quality scoring with NLLB-200-Distilled-600M.Source: §3.1 · p. 3

3.2 Layer-Selective Tuning

• Single-layer experiments find that intermediate-layer tuning, especially around layer 20, significantly reduces translation performance.Layerwise nuclear norms measure Q, K, and V parameter sensitivity on en-zh FLORES-101 data, with similar trends in other directions.Source: Figure 4 · p. 3, §3.2 · p. 4, Figure 5 · p. 4

3.3 Training Algorithm

• Training proceeds in two stages: tune the bottom k layers, then the top m layers, while middle-layer parameters remain frozen.Both stages use the same instruction-formatted multilingual parallel data, and stage two starts from stage one's parameters.Source: §3.3 · p. 4

4.1 Setting

• Experiments evaluate translation on FLORES-101, seven multilingual benchmarks, and 15 reasoning datasets, using xComet and spBLEU for translation.The setup compares general instruction models, tuning strategies, multilingual models, and domain-specialized models.Source: §4.1 · pp. 4–5

4.2.2 Performance Comparison

• Qwen3-XPlus-14B achieves the highest xComet scores in 6 of 7 reported languages and beats Aya-Expanse-32B across evaluated languages despite using fewer than half its parameters.Qwen3-XPlus-8B surpasses Aya-Expanse-32B in 12 of 14 translation directions; the low-resource advantages over LLaMAX3-8B-Alpaca are 4.51, 8.04, and 10.67 xComet scores for x →sw, x →th, and x →bn.Source: Table 1 · p. 5, §4.2.2 · pp. 6–7
• Across reasoning tasks, Qwen3-XPlus performs on par with Qwen3 and clearly surpasses Tower-Plus-9B.Figure 7 compares the models on 15 reasoning datasets.Source: Figure 7 · p. 6, §4.2.2 · p. 7

4.2.1 Effectiveness of layer-selective tuning

• Across 28 experimental settings, Qwen3-XPlus achieves higher xComet scores than Qwen3 in 27 cases.The reported gains are especially pronounced for weaker languages such as Swahili; Figure 6 compares Qwen3-XPlus with Qwen3, FFT, and LoRA.Source: Figure 6 · p. 5, §4.2.1 · p. 6

4.2.2 Performance Comparison

• Qwen3-XPlus-8B beats Qwen3-8B on 6 of 7 multilingual datasets, and Qwen3-XPlus-14B beats Qwen3-14B on 5 of 7.The models use general parallel corpora without task-specific multilingual data and rank among the best evaluated multilingual LLMs.Source: Table 2 · p. 6, §4.2.2 · p. 7

4.2.3 Core Findings

• Qwen3-XPlus starts from Qwen3 instruct models and uses layer-selective tuning to extend multilingual capability while inheriting the starting model’s strengths.The paper contrasts this with multilingual approaches that start from base models; it notes that Sailor2-8B-Chat has weaker translation performance than Qwen2.5-7B-Instruct.Source: §4.2.3 · p. 7
• With 0.8B training tokens, Qwen3-XPlus achieves competitive multilingual and general-task performance compared with models trained on substantially more data.It is trained solely on general parallel corpora, but does not surpass its starting model on knowledge-intensive XGPQA without task-specific domain knowledge.Source: §4.2.3 · pp. 7–8
• The training procedure uses the SFT stage without the more demanding CPT or RL phases.The authors say denser training stages could offer further improvements.Source: §4.2.3 · p. 8

5.1 Layer Combination Analysis

• Training the bottom four and top fifteen layers together delivers the best translation performance and is the configuration used in the main experiments.Training the top fifteen layers gives the largest improvement among higher-layer choices; layer 20 is skipped after negatively impacting performance.Source: §5.1 · p. 8

5.2 Effect of Two-Stage Training

• Sequentially fine-tuning lower layers and then higher layers performs better than training both groups simultaneously.The authors attribute the advantage possibly to smoother adaptation; even single-stage training outperforms Qwen3-8B.Source: §5.2 · p. 8

5.3 Generalization to Unseen Languages

• On 12 languages unseen during tuning, Qwen3-XPlus-14B consistently outperforms Qwen3-8B.The training corpora cover 17 languages.Source: §5.3 · p. 8

5.4 Adaptability to Different Backbones

• Using the same training setup on Llama3.1-8B substantially improves performance across multiple languages, with particularly notable gains on low-resource languages.Figure 9 reports the different-backbone experiment.Source: §5.4 · p. 8, Figure 9 · p. 8

5.5 Adaptability to Different Tasks

• On OpenThoughts, layer-selective tuning improves accuracy over FFT by 1.22%–2.86%; on WebSyn, it improves accuracy by 0.61%–4.00%.Across four WebSyn evaluation sets, layer-selective tuning gains 0.68%–2.29%, while FFT decreases performance on three; Table 5 compares the methods.Source: §5.5 · p. 9, Table 5 · p. 9

6 Conclusion

• Qwen3-XPlus significantly enhances translation performance across diverse languages, particularly in low-resource contexts.The approach uses instruct models trained on limited parallel datasets.Source: §6 · p. 9
• The authors report substantial gains in multilingual tasks alongside competitive reasoning capabilities.The conclusion attributes these results to leveraging instruct models and limited parallel datasets.Source: §6 · p. 9
• The work presents its approach as a basis for future developments in multilingual enhancement.Source: §6 · p. 9

A Appendix

• Low-rank LoRA performs significantly better than high-rank configurations.Source: A Appendix · p. 15
• The authors suggest stronger mitigation of catastrophic forgetting as a possible explanation for low-rank LoRA’s advantage.Source: A Appendix · p. 15
• The main experiments use LoRA with rank = 8.Source: A Appendix · p. 15

• Table 9 lists the models evaluated in the study.Source: §A.1 · p. 15

• NLLB contains parallel text for 148 English-centric and 1465 non-English-centric language pairs and has a complete size of 450GB.Source: §A.2 · p. 15
• OPUS-100 is an English-centric corpus covering 100 languages, with approximately 55M sentence pairs.Of its 99 language pairs, 44 have 1M training pairs, 73 have at least 100k, and 95 have at least 10k.Source: §A.2 · p. 15

• Table 10 lists the benchmarks used in the evaluation.Source: §A.3 · p. 15

A.4 Hyperparameter Settings

• Training ran for 1 epoch with a 1e-5 learning rate, cosine scheduling to 2e-6, a 0.03 warmup ratio, bf16 precision, and 8 NVIDIA H800 GPUs.The per-device batch size was 1, with gradient accumulation over 2 steps.Source: §A.4 · p. 15

A.6 Analysis of LoRA Variants and Hyperparameters

• Vanilla LoRA showed a clear translation-performance advantage over the other tested variants and hyperparameter settings.Source: §A.6 · p. 15, Table 11 · p. 19
• Tables 6–9 cover model comparisons, translation results, reasoning and multilingual tasks, and evaluated-model information, respectively.Table 6 compares Qwen3-XPlus with Qwen3 and FFT with LoRA on 17 FLORES-101 languages; Table 7 reports translation results on 17 languages.Source: Table 6 · p. 16, Table 7 · p. 17, Table 8 · p. 18