PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Get Smarter from Targeted Synthetic Multilingual Data

Ishika Agarwal, Arkajyoti Charaborty, Tanner Sorensen, Neha Gupta, Andreas Stolcke

arXiv:2608.15964v1cs.CLcs.AI

TL;DR

Multilingual LLMs must balance English performance, cross-language consistency, and overall accuracy. HOTFIXR targets multilingual weaknesses with synthetic post-training data, achieving a 6.2% average improvement and 5.6% average generalization improvement across training-based baselines.

  • Problem

    Multilingual LLMs face a tradeoff between optimizing English performance and maintaining cross-language consistency without reducing overall accuracy.

  • Method

    HOTFIXR trains a data generator using student-model feedback to produce synthetic samples targeting difficulty and multilingual representation gaps.

  • Results

    Across training-based baselines, HOTFIXR achieves the best performance, with 6.2% average improvement and 5.6% average generalization improvement.

  • Takeaways & Limitations

    HOTFIXR improves in-distribution performance while remaining reliable on out-of-distribution tasks and languages.

  • Takeaways & Limitations

    The study focuses mainly on high-resource languages, relies on a strong label-generation model, and may not extrapolate to unverifiable domains.

Abstract

from arXiv · show

Language-specific competency (LSC) is the phenomenon of a language model performing better or worse depending on the language of the prompt. In other words, a language model outputs different (and potentially incorrect) responses to the same semantic query when prompted in different languages. Prior work attributes this to an internal misalignment of semantic representation across languages. Currently, there are two main approaches to address LSC in the literature: (1) routing all queries through English, improving performance, but limiting language expressivity to English; or (2) training on language-balanced data, equalizing model performance across languages, but reducing overall performance. In this work, we take a data centric perspective and introduce HOTFIXR: Hardness Optimized Training data For Improving X-Lingual Reasoning. It is a data generation framework that uses models to probe and learn a student model's multilingual weaknesses, and generates data to mitigate them. HOTFIXR can generate multilingual synthetic training data that can improve multilingual performance. We evaluate on three in-distribution tasks, three out-of-distribution tasks, and four out-of-distribution languages. On average, HOTFIXR (1) improves in-distribution performance by 6.2%, (2) reduces catastrophic forgetting (induced by fine-tuning) on OOD tasks by 3.7%, and (3) on OOD languages by 7.1%. Overall, as many real-world applications requires multilingual LLMs, our work contributes to the efforts of making LLMs multilingually proficient. We will release code upon acceptance.

1 Introduction

Multilingual LLMs are needed for non-English applications, but English-dominant training creates representational gaps and a tradeoff between English performance, cross-language consistency, and overall accuracy. HOTFIXR addresses this tradeoff by generating post-training data that probes a student model’s multilingual weaknesses without degrading general abilities.

  • Many LLM business applications require reliable conversation in languages other than English, motivating multilingual model training with non-English data.
  • English-dominant training causes LLMs to anchor non-English processing in English, potentially limiting non-English understanding through cross-language representational gaps.
  • Language-balanced pretraining improves consistency across languages but can reduce English performance and overall model accuracy, illustrating the multilingual tradeoff.
  • HOTFIXR is a post-training synthetic data framework designed to improve multilingual ability without degrading general abilities.
  • HOTFIXR trains a question-generation model on student-model feedback signals to probe and target the student’s weaknesses.

2 Related Works

Related work attributes multilingual LLM misalignment to language-dependent representations, often shaped by English-dominant pretraining. Proposed mitigations include prompting, contrastive and architectural methods, multilingual reasoning fine-tuning, and data curation.

  • Language Misalignment: Multilingual LLMs may produce different answers to the same factual query across languages because their representations are misaligned.Related behaviors include language-specific knowledge and non-isomorphic word or phrase mappings.
  • Language Misalignment: English-dominant pretraining data gears models toward internally representing language in English.Models reportedly route latent thinking through English spaces in early-to-middle layers before returning to non-English spaces later.
  • Mitigating Misalignment: Smart prompting elicits language-dependent knowledge but relies on hand-crafted prompts that may not generalize and can be sensitive to prompt style.These limitations make prompting potentially unreliable across language models.
  • Mitigating Misalignment: Architectural and fine-tuning methods address multilingual reasoning by combining specialized model capabilities or aligning reasoning chains and traces across languages.Examples include multilingual embeddings provided to a reasoning model, aligning low-resource chains with high-resource chains, and crafting correct reasoning traces in other languages.
  • Multilingual Data Curation: Most multilingual data-curation work filters and cleans existing datasets through deduplication, multilingual-document removal, and model-based sample selection.The described selection strategy keeps training samples within the distribution of an existing reference set.

3 HOTFIXR Methodology

HOTFIXR trains a question generation model using student feedback to generate multilingual samples that target language-agnostic and language-specific weaknesses. The generated 5,000-sample dataset is labeled and used for student-model supervised fine-tuning.

  • Framework: HOTFIXR generates samples, scores them with an acquisition function for lingual deficits, and trains the question generator to produce more informative data.The framework uses student-model signals to guide question generation, with higher-reward samples targeting weaknesses useful for downstream learning.
  • Weakness Signals: The acquisition function captures both language-agnostic incompetency and language-specific incompetency in student models.Language-agnostic incompetency concerns questions the model cannot answer correctly in any language, whereas language-specific incompetency concerns failures in particular languages.
  • Weakness Signals: Language-agnostic incompetency is measured from uncertainty when the student answers using English-centric reasoning.A higher reward is assigned when the student model is uncertain in its strongest language, assumed to be English because most pre-training data is English.
  • Weakness Signals: Language-specific incompetency is measured by cosine distance between hidden states from English reasoning and reasoning in French, Spanish, Arabic, Portuguese, or Italian.The compared states are taken from the final reasoning token, and larger distances receive higher rewards because reasoning for the same answer should align across languages.
  • Training Pipeline: The trained question generator creates a 5,000-sample multilingual dataset, labeled with Qwen/Qwen2.5-32B-Instruct, for student-model SFT.Samples are generated in English, French, Spanish, Arabic, Portuguese, and Italian before supervised fine-tuning.

4 Experiments

HOTFIXR is evaluated across multiple model families, multilingual tasks, languages, and baselines, with results averaged over three runs. It improves in-distribution performance while limiting degradation on out-of-distribution tasks and languages, at the cost of additional data-generator optimization.

  • Evaluation setup: The evaluation uses Qwen 7B, Qwen 14B, and Llama 8B instruction-tuned models across six in-distribution and two out-of-distribution languages.Training languages are English, French, Spanish, Arabic, Portuguese, and Italian; German and Japanese are out-of-distribution.
  • Evaluation setup: HOTFIXR is evaluated on four tasks spanning agentic reasoning, factual knowledge, RAG reading comprehension, and translation.The tasks are Nemotron, MMMLU, mHotPotQA, and OPUS-100.
  • In-distribution results: 4.3%: HOTFIXR improves Nemotron STEM, MATH, and CHAT performance over the base model, while outperforming the best selection and synthesis methods by 6.0% and 5.8%.The improvements over the base model are 5.5% on Qwen 7B, 3.1% on Qwen 14B, and 4.4% on Llama 8B.
  • Out-of-distribution results: 0.9%: HOTFIXR’s average degradation on three out-of-distribution tasks is limited relative to the base model, while it is 5.6% better than other training-based baselines.Compared to SelectionGT and SelectionGEN, HOTFIXR achieves a 6.4% improvement.
  • Out-of-distribution results: 7.1%: HOTFIXR performs better than other training-based baselines on out-of-distribution languages despite an overall post-training performance decline.For OOD languages, the evaluation includes German and Japanese for Nemotron and Russian and Chinese for RAG.
  • Cost and limitations: 0.96min/training data sample: HOTFIXR incurs this data-generator training cost using roughly 40 RL steps and 6 A100 GPUs.The reported cost corresponds to roughly 8 hours for |DQG| = 500, while the generator-training cost is described as a one-time expense.

5 Conclusion

The paper presents HOTFIXR, a principled data-synthesis pipeline for improving multilingual abilities without degrading overall model performance. It targets multilingual consistency and agentic abilities by generating samples that are difficult and exhibit larger multilingual representation gaps.

  • Conclusion: HOTFIXR adapts models to improve multilingual abilities without degrading overall performance.The conclusion frames this as the paper’s central objective.
  • Conclusion: HOTFIXR is a principled data-synthesis pipeline that trains data-generation models to target agentic abilities and multilingual consistency.The generated data is designed to address both capabilities in multilingual settings.
  • Conclusion: Its reward identifies samples that are difficult according to LAI and have larger multilingual representation gaps according to LSI.These criteria guide the data generator toward targeted multilingual training examples.

6 Limitations

HOTFIXR primarily addresses high-resource languages, while low-resource language coverage requires further consideration. The authors also note that heavily biased training data may produce harmful synthetic data.

  • 6 Limitations: The study focuses on seven high-resource languages, while low-resource language coverage remains future work and biased training data may generate harmful data.The evaluated high-resource languages are English, French, Spanish, German, Japanese, Portuguese, and Italian; Arabic is also tested, but broader low-resource coverage requires special consideration.

A Reward Hacking

The question-generator training dataset DQG contains only 500 examples because reward hacking causes the generator to repeatedly produce the same locally optimal sample. This behavior reflects failure to explore beyond a local maximum lingual deficit reward, even when the in-context sample changes.

  • A Reward Hacking: DQG is limited to 500 examples because the question generator reward hacks during training.The paper identifies reward hacking as the reason for the dataset’s small size.
  • A Reward Hacking: Reward hacking traps the generator at a local maximum lingual deficit reward, causing it to regenerate the same data point repeatedly.Changing the in-context sample does not alter the generated output, which remains the exact same data point.

B Error Bars for Performance on ID and OOD tasks, and Per-Language Performance on OOD tasks.

This section reports error bars for Nemotron tasks and per-language performance with error bars for factual, translation, and RAG tasks. Minimum and maximum standard deviations across three runs are documented for each task family, explaining the figures’ small error bars.

  • Error bars and per-language performance: Error bars cover Nemotron tasks and per-language results for factual, translation, and RAG tasks.Figures 8–11 provide these visualizations across the evaluated models and methods.
  • Error bars and per-language performance: Minimum and maximum standard deviations are recorded across 3 runs for Nemotron, Translation, Factual, and RAG tasks.Tables 6–9 report these ranges by model and method, explaining the small error bars shown in the figures.
  • Error bars and per-language performance: Per-language performance is shown for factual MMMLU, translation OPUS-100, and RAG multilingual HotPotQA queries.Figures 9, 10, and 11 organize the results by language for each task.

C Knowledge Distillation

The distillation control uses Qwen 32B to generate both questions and labels, isolating gains from question generation versus label generation. Results indicate that HOTFIXR’s trained question generation better identifies model weaknesses, so much of its success comes from targeted questions rather than distilled labels alone.

  • Distillation Control: The Distillation (32b) baseline uses Qwen 32B to generate both questions and labels, separating question-generation gains from label-generation gains.This setup is similar to the Untrained baseline but replaces it with Qwen 32B.
  • Results: Distillation with a larger model improves performance, but HOTFIXR’s trained question generation understands model weaknesses better.The comparison is reported in Table 10 against the Distillation (32b) baseline.
  • Results: HOTFIXR’s empirical success comes largely from targeted question generation rather than correct distilled labels from a large model.Table 10 adds the Distillation method to support this attribution.

D An Experiment on Continual Learning · E Prompts

A second HOTFIXR round uses a freshly trained question generator with a continuously trained student and produces larger gains, including on unseen tasks and languages. The section also documents prompts for generating data across Nemotron subsets and multiple languages.

  • D An Experiment on Continual Learning: HOTFIXR’s continual-learning experiment trains the Round 1 question generator from scratch while continuing the student from Round 0.Round 0 denotes the first iteration, and Round 1 denotes the second iteration with a fresh generator and continuously trained student.
  • D An Experiment on Continual Learning: Two HOTFIXR rounds improve student performance much more than one round.Figure 12 reports the delta in student-model performance for two rounds versus one round.
  • D An Experiment on Continual Learning: Each HOTFIXR iteration improves the student substantially, even on unseen out-of-distribution tasks and languages.The reported improvements extend beyond cross-linguality to overall model response quality.
  • D An Experiment on Continual Learning: Iterative HOTFIXR is expected to yield higher gains in cross-linguality and model response quality.This conclusion follows from the larger improvements observed across iterations, including on OOD tasks and languages.
  • E Prompts: Figures 13–15 provide three prompts for HOTFIXR question-generator training and data generation on Nemotron’s MATH, STEM, and CHAT subsets.The prompts correspond to the three named Nemotron subsets.
  • E Prompts: The Nemotron MATH prompt takes a question, answer, reasoning, and an alternating language selected from English, French, Spanish, Arabic, Portuguese, or Italian.The prompt uses inputs from the Nemotron MATH training dataset.

F LLM Usage

The authors primarily used LLMs to create figures from experimental results, with all outputs heavily reviewed. Other uses were minimal, and LLMs were not used for code files, literature reviews, or methodology/evaluation descriptions.

  • LLM Usage: The main LLM use was Claude Opus 4.8 generating matplotlib figure code from experimental results.The authors provided the results and prompted Claude Opus 4.8 to write the figure code.
  • LLM Usage: Other LLM uses were limited to writing enhancement and small code suggestions.
  • LLM Usage: The authors did not use LLMs to write code files, conduct literature reviews, or describe methodology or evaluation.All LLM use was heavily reviewed by the authors.
  • LLM Usage: HOTFIXR prompts take Nemotron STEM or CHAT questions, answers, and reasoning as input while alternating among six languages.The languages are English, French, Spanish, Arabic, Portuguese, and Italian.
Loading 2608.15964v1…