PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Imbalance Driven Rewarding for Multilingual Self-improving

Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, Jiajun Zhang

arXiv:2410.08964v3cs.CLcs.AI

TL;DR

Many languages remain underrepresented in LLM capabilities, while dominant and non-dominant language differences provide a potential self-generated preference signal. The paper uses iterative DPO with self-translated multilingual responses, reporting improved performance in both language groups and across instruction-following and arithmetic reasoning.

  • Problem

    LLMs have disproportionately benefited English and Chinese, leaving many other languages underrepresented despite their worldwide use.

    Source: §1 · p. 1

  • Method

    Language Imbalance Driven Rewarding uses a model’s inherent language imbalance to construct self-generated multilingual preference pairs for iterative DPO training.

    Source: §1 · p. 2, §2 · p. 4

  • Results

    Experiments on Llama-3-8B-Instruct report improved performance in non-dominant and dominant languages, with gains in instruction-following and arithmetic reasoning.

    Source: §1 · p. 2, §5 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    The results support bootstrapping multilingual performance across languages without human-authored datasets, with improvements that can saturate as language performance gaps narrow.

    Source: §1 · p. 2

    The approach assumes that translation does not alter language biases and that self-translation largely preserves preference rankings.

    Source: §2 · p. 3, §3.2 · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner. Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's capacity, thereby yielding an iterative reward signal. Fine-tuning Meta-Llama-3-8B-Instruct over two iterations of this approach results in continuous improvements in multilingual performance across instruction-following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46% win rate on the X-AlpacaEval leaderboard and 13.9% accuracy on the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvement of LLMs. The code is available at https://github.com/ZNLP/Language-Imbalance-Driven-Rewarding

1 INTRODUCTION

LLMs disproportionately benefit English and Chinese, while many other languages remain underrepresented. Language Imbalance Driven Rewarding turns this imbalance into a self-generated preference signal, then iteratively optimizes multilingual performance.

1 INTRODUCTION

  • Language imbalance is a concern because LLMs have favored English and Chinese, leaving many languages underrepresented despite worldwide use.The introduction links the imbalance to risks for users operating in less dominant languages.Source: §1 · p. 1
  • Language imbalance creates a natural preference ranking between dominant and non-dominant languages that can serve as a reward signal for their simultaneous optimization.The approach also aims to eliminate reliance on human-authored data and surpass the dominant-language performance ceiling.Source: §1 · p. 2
  • The method generates multilingual responses, self-translates them across dominant and non-dominant languages, and trains iteratively with a DPO variant that adds an NLL term.The translated responses are used to construct preference data, and each trained model can provide reward signals in subsequent iterations.Source: §1 · p. 2, Figure 1 · p. 2
  • Two iterations starting from Meta-Llama-3-8B-Instruct improve non-dominant-language instruction following and dominant-language performance, with effects expected to saturate as the performance gap narrows.The authors describe this as an opportunity to bootstrap multilingual performance without human-authored datasets.Source: §1 · p. 2

2 LANGUAGE IMBALANCE DRIVEN REWARDING

At each iteration, the current model generates and translates multilingual responses to construct preference pairs, then trains on those pairs to produce the next model. The updated model repeats this process.

2 LANGUAGE IMBALANCE DRIVEN REWARDING

  • For multilingual prompts, the model generates responses and self-translates dominant-language and non-dominant-language responses into the counterpart language.The setup assumes an instruction-following model with basic multilingual generation capabilities and prompts covering participating languages.Source: §2 · pp. 2–3
  • The method assumes translation does not alter language biases, so the dominant-language response is preferred over its non-dominant-language counterpart.This preference ranking is used to construct multilingual pairs for languages supported by the model.Source: §2 · p. 3
  • The multilingual preference dataset combines dominant-language and non-dominant-language pairs, including responses translated between the two language groups.The model’s self-translation supplies the counterparts used in the pair construction.Source: §2 · p. 3
  • Training uses a DPO variant that maximizes the chosen response probability, minimizes the rejected response probability, and adds an NLL term for chosen labels.The NLL term is normalized by response length, whereas the DPO loss is not; the trained model becomes the next iteration’s model.Source: §2 · p. 3
  • Each iteration uses the current model to generate multilingual preference data, optimizes on that data, and uses the resulting model to generate data for the next iteration.The procedure defines M0 as the base instruction-following model, then generates D0 for M1 and D1 for M2.Source: §2 · pp. 3–4

3 DISCUSSION

The discussion tests whether Llama-3 has language-specific response-quality differences and whether self-translation preserves the preference ranking. The authors report both an initial language gap and preservation of the ranking after translation.

3 DISCUSSION

  • The discussion identifies language capability differences as the source of the reward signal used for DPO training.It frames preserved response-preference rankings under self-translation as a key question for the method.Source: §3 · p. 4, §3.2 · p. 4

3.1 DO LLMS EXHIBIT SIGNIFICANT DIFFERENCES IN MULTILINGUAL CAPABILITIES?

  • GPT-4 evaluation of 100 multilingual Alpagasus instructions finds a significant response-quality difference between English and non-dominant languages in Llama-3-8B-Instruct.English is designated the dominant language based on the LLaMA 3 technical report; Table 1 summarizes average response quality across languages.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 DOES TRANSLATION PRESERVE THE RANKING OF RESPONSE PREFERENCES?

  • Self-translation largely preserves the preference ranking: translated dominant-language responses score below their originals, while a substantial gap remains over non-dominant-language responses.The authors report the same ranking in English when non-dominant-language responses are translated into English.Source: §3.2 · p. 4, Table 2 · p. 5
  • Reward accuracy for multilingual preference pairs exceeds 0.50, ranging from 0.57 in Spanish to 0.79 in Japanese.The authors report that weaker-performing languages tend to exhibit stronger reward signals; rewards of 0.60 are used to classify low- and high-reward languages.Source: §3.2 · pp. 4–5, Table 3 · p. 5

4 GENERAL INSTRUCTION FOLLOWING

Across two training iterations, Language Imbalance Driven Rewarding improves multilingual instruction-following results, including on an unseen language. The evaluations also examine leaderboard performance, conversational ability, alignment tax, and how reward signals change across iterations.

4.1 EXPERIMENTAL SETUP

  • Experiments use Llama-3-8B-Instruct, English as the dominant language, four non-dominant training languages, and Chinese as an unseen language.The study samples 1,000 Alpagasus prompts and translates them into other languages; evaluation includes head-to-head comparisons, a multilingual leaderboard and MT-Bench, plus multilingual NLP benchmarks.Source: §4.1 · p. 5, Evaluation and Metrics · p. 5

4.2 HEAD-TO-HEAD PERFORMANCE

  • Head-to-head results show gains across training and unseen languages, with high-reward languages gaining more initially and later training-language gains converging.English gains 15.3% in M1 vs. M0 and 14.6% in M2 vs. M1; Chinese gains are 32.2% and 22.4%, respectively. The figure reports multilingual instruction-following win rates across iterations.Source: §4.2 · pp. 6–7, Figure 2 · p. 6

4.3 X-ALPACAEVAL LEADERBOARD

  • After two rounds, Llama-3-8B-Instruct improves average win rate by 7.46% over GPT-4 Turbo across five languages and performs comparably to 70B-level models.The leaderboard reports multilingual instruction-following results evaluated by GPT-4.Source: Table 4 · p. 7, §4.3 · p. 7
  • Compared with multilingual alignment SFT, the method reports higher performance on non-dominant languages and improves English rather than degrading it.The reported non-dominant-language comparison is 22.18% vs. 18.32% in Llama3; English changes by +9.19% with this method and -3.02% with alignment SFT.Source: §4.3 · p. 7

4.4 PERFORMANCE ON MULTILINGUAL MT-BENCH

  • Llama3's multilingual MT-Bench score rises from 6.80 in M0 to 7.51 in M2, while its reward signal weakens over iterations.MT-Bench scores are reported on a scale of 10.Source: §4.4 · pp. 7–8, Table 5 · p. 8

4.5 ALIGNMENT TAX ON MULTILINGUAL NLP BENCHMARKS

  • Across four multilingual NLP benchmarks, iteration shows no significant degradation from base models and slight improvement, which the authors interpret as no alignment tax.The evaluation averages results over five training languages and unseen Chinese; it is motivated by prior findings of forgetting after instruction tuning and RLHF.Source: §4.5 · p. 8, Table 6 · p. 8

4.6 THE REWARD SIGNAL CHANGES OVER ITERATIONS, GETTING STRONGER OR WEAKER?

  • Reward accuracy changes across iterations: most initially high-reward training languages shift lower, while English improves and sustains lower-reward languages' opportunity for further iteration.On unseen languages, Italian reward accuracy increases as English improves, while Japanese reward accuracy drops after Iteration 1 alongside reduced off-target responses.Source: §4.6 · p. 8, Figure 3 · p. 8

4.7 SCALING AND GENERALIZING

  • The paper also tests scaling to Qwen2-7B and extensions involving a weaker base model, lower-resource languages, and external translation.These extensions are described as experiments in the appendices.Source: §4.7 · pp. 8–9

5 ARITHMETIC REASONING

On MGSM, Language Imbalance Driven Rewarding improves arithmetic reasoning across training and unseen languages and outperforms multilingual alignment and MAPO variants. The section evaluates reasoning accuracy across ten languages using Llama-3-8B-Instruct.

5 ARITHMETIC REASONING

  • Arithmetic reasoning is evaluated because existing LLMs show inconsistent reasoning capabilities across languages despite reasoning being considered language-agnostic.The section scales the method to this task to enhance reasoning performance across languages.Source: §5 · p. 9

5.1 EXPERIMENTAL SETUP

  • The evaluation uses MGSM, with 250 manually translated GSM8K problems in ten languages, and reports reasoning accuracy across five training and five unseen languages.Off-target rates are also assessed using LangDetect.Source: §5.1 · p. 9, Table 7 · p. 9
  • Comparisons include multilingual alignment and MAPO, with MAPO variants controlling for sampling count or training-data size.All preference pairs were trained under identical conditions, and MAPO's best results after two iterations were reported.Source: §5.1 · p. 9

5.2 RESULTS

  • 16.2% average improvement in unseen languages with M1 exceeds the 11.6% gain in training languages; multilingual alignment gains 4.9% in unseen languages.In training languages, M2 improves 11.6% on average, compared with 7.8% for multilingual alignment.Source: §5.2 · p. 9, Table 7 · p. 9
  • With the same sampling effort, MAPO† underperforms across all languages, and MAPO‡ performs worse in most languages using the same training data.The authors attribute MAPO†'s result likely to limited preference data.Source: §5.2 · p. 10

6 RELATED WORK

Prior work improves multilingual capabilities through multilingual training or cross-lingual alignment, while self-improvement methods have generally focused on overall rather than cross-language capability. The paper distinguishes its approach from MAPO by using the LLM itself to translate and form preferences across dominant and non-dominant languages.

LLM Self-Improving

  • Existing self-improvement methods use self-generated data or model-based judgments but primarily target overall capability, not self-improvement across languages.The passage describes self-synthetic, self-critical, and self-rewarding approaches.Source: §6 · p. 10

Multilingual LLMs

  • Multilingual training and cross-lingual alignment are established approaches, but high-quality multilingual data, especially for low-resource languages, remains scarce and expensive.The approaches include adding multilingual data and aligning performance between dominant and non-dominant languages.Source: §6 · p. 10

MAPO comparison

  • Unlike MAPO, which uses an off-the-shelf translation model to assess cross-language consistency, this approach uses the LLM itself to translate and constructs preference pairs from language imbalance.The paper says its approach supports both dominant and non-dominant languages and general tasks.Source: §6 · p. 10

7 CONCLUSION

The paper uses inherent language imbalance as a reward signal, generating and self-translating responses between dominant and non-dominant languages to form preferences for Iterative DPO training. Experiments on Llama-3-8B-Instruct show improvements in instruction-following and arithmetic reasoning, and the paper frames multilingual self-improvement as an area for further development.

7 CONCLUSION

  • Language Imbalance Driven Rewarding uses the imbalance between dominant and non-dominant languages as a reward signal for multilingual self-improvement.The approach starts from an instruction-following model with basic multilingual capabilities.Source: §7 · p. 10
  • The method generates and self-translates responses across dominant and non-dominant languages, constructs preference rankings, and trains with Iterative DPO.These steps bootstrap multilingual capabilities within the model.Source: §7 · p. 10
  • Experiments on Llama-3-8B-Instruct show improved instruction-following and arithmetic reasoning, and the paper presents autonomous multilingual improvement as a direction for future models.The conclusion states that much remains to be explored.Source: §7 · p. 10

A LIMITATIONS AND FUTURE WORK

On Qwen2-7B-Instruct, two iterations improve reported leaderboard and multilingual benchmark results, and the head-to-head evaluation reports gains in training languages and English. The paper also notes coarse reward granularity, possible artifacts in self-translated responses, and evaluation reproducibility details.

A LIMITATIONS AND FUTURE WORK

  • The reward is coarse-grained, and self-translated responses may contain artifacts that hinder response quality.The paper identifies more refined reward signals as future work.Source: A LIMITATIONS AND FUTURE WORK · p. 17

B REPRODUCIBILITY STATEMENT

  • Evaluations primarily use greedy decoding; AlpacaEval and MT-Bench rely on OpenAI's API, whose response randomness may affect reproducibility.The paper says codes and model weights were made public after review.Source: B REPRODUCIBILITY STATEMENT · p. 17

C SCALING ON MODEL: QWEN2

  • Qwen2-7B-Instruct was chosen as the base model to test generalizability because it has stronger multilingual capabilities and seldom produces off-target responses.Source: C SCALING ON MODEL: QWEN2 · p. 17

C.1 HEAD-TO-HEAD PERFORMANCE

  • English in M1 gains 23.5% ∆W-L over M0, and English in M2 gains 7.3% ∆W-L over M1; training-language M1 gains range from 21.0% to 31.2%.Figure 4 presents Qwen2 head-to-head performance; the passage says the results are consistent with Llama3.Source: §C.1 · p. 17, Figure 4 · p. 17

C.2 X-ALPACAEVAL LEADERBOARD

  • 5.84% average win-rate improvement over GPT-4 Turbo across five languages follows two iterations; against multilingual alignment, win rate is 21.85% versus 17.68%.The paper reports English performance increasing 10.54% with its method, while multilingual alignment shows a 3.6% decline.Source: §C.2 · pp. 17–18, Table 8 · p. 18

C.3 MULTILINGUAL MT-BENCH

  • Qwen2-7B-Instruct's average multilingual MT-Bench score rises from 8.05 to 8.20 after two training iterations.The passage notes that Qwen2's high initial score reduces the effectiveness of the language-imbalance reward signal.Source: §C.3 · p. 18, Table 9 · p. 18

C.4 MULTILINGUAL NLP BENCHMARKS

Across multilingual NLP benchmarks, iterative optimization shows slight improvements without significant degradation or alignment tax. The authors also use within-language comparisons and GPT-4o evaluations to address concerns about evaluator bias.

C.4 MULTILINGUAL NLP BENCHMARKS

  • Slight multilingual benchmark improvements over the base models show no alignment tax across five training languages and Chinese.The reported averages cover four benchmarks; Table 10 presents Qwen2 results.Source: §C.4 · p. 18, Table 10 · p. 18

D.1 HOW TO AVOID LANGUAGE BIAS IN LLM-AS-A-JUDGE

  • Within-language pairwise comparisons in Tables 2 and 3 avoid cross-lingual scoring bias, unlike individual GPT scores that may vary by language.GPT-4o evaluations in Table 11 align with GPT-4 results, supporting the robustness of the findings despite potential evaluation bias.Source: §D.1 · p. 19, §D.2 · p. 19, Table 11 · p. 19

D.2 ALIGNING WITH ADVANCED MODEL: USING GPT-4O AS A JUDGE

  • GPT-4o X-AlpacaEval results remain consistent with Table 4, although using a GPT-4 Turbo-calibrated configuration may introduce bias.Table 12 reports the GPT-4o evaluation.Source: §D.2 · p. 19, Table 12 · p. 19

D.3 HOW TO AVOID TRANSLATIONESE BIAS IN MULTILINGUAL BENCHMARKS EVALUATION

The paper addresses possible translationese bias and tests the method on a weaker model, lower-resource languages, and an external-translation variant. It also describes the multilingual arithmetic-reasoning data preparation.

D.3 HOW TO AVOID TRANSLATIONESE BIAS IN MULTILINGUAL BENCHMARKS EVALUATION

  • Self-translation uses a different data distribution from predominantly English-to-other-language machine-translated benchmarks, reducing the risk of translationese bias.The authors also argue that LLM self-translations reduce translationese artifacts and produce fluent, natural translations.Source: §D.3 · p. 20

E.1 PERFORMANCE ON WEAKER MODEL: LLAMA2

  • Two iterations of language imbalance-driven rewarding improve Llama2-7B-Chat across training languages, especially languages where its initial performance was weaker.Table 13 presents the X-AlpacaEval results.Source: §E.1 · p. 20, Table 13 · p. 20

E.2 PERFORMANCE ON LOWER-RESOURCE LANGUAGES: bn, sw, th

  • Post-training gains in Bengali, Swahili, and Thai are incremental and cannot fully overcome limitations in the pre-training data.The approach nevertheless iteratively improves performance across all languages in the reported low-resource setting.Source: §E.2 · p. 20

E.3 RELAXATION OF THE SELF-IMPROVEMENT PARADIGM UNDER EXTREME SCENARIOS

  • External Google Translate can provide higher-quality low-resource-language data, while self-translation better improves English by preserving a consistent generation space.The comparison is reported in Tables 14 and 15.Source: §E.3 · p. 21, Table 14 · p. 21, Table 15 · p. 21

F.2 EXPERIMENTAL DETAILS ON ARITHMETIC REASONING

  • For arithmetic reasoning, multilingual GSM8K instructions are translated from 7,473 English training examples, and responses with correct English reasoning are filtered before optimization.Prompts explicitly constrain the response language during training and inference.Source: §F.2 · p. 22

F.3 EXPERIMENTS ENVIRONMENTS

Experiments use a specified GPU and software stack, with AdamW optimization and fixed DPO+NLL settings.

F.3 EXPERIMENTS ENVIRONMENTS

  • Experiments ran on Ubuntu 22.04 with eight NVIDIA A100 GPUs and used LLaMAFactory, vLLM, DeepSpeed ZeRO Stage 2, and Flash-Attention 2.The implementation uses Python 3.10, PyTorch 2.3.0, and accelerate.Source: §F.3 · p. 22

F.4 HYPERPARAMETERS

  • Training uses AdamW with a cosine schedule and 3% warm-up; DPO+NLL uses KL-penalty β = 0.1 and α = 1 throughout.Table 16 lists the experiment hyperparameters.Source: §F.4 · p. 22, Table 16 · p. 22

G.1 MULTILINGUAL NLP BENCHMARKS

The appendix describes multilingual benchmark coverage and the prompts used for scoring, head-to-head comparisons, leaderboard evaluation, translation, and multilingual reasoning. It reports slight benchmark improvements during iterative training.

H.1 GPT-4 SCORE PROMPT

  • The evaluation uses MMLU in a 5-shot setting, HellaSwag and ARC Challenge in zero-shot settings, and TruthfulQA in a zero-shot setting to assess whether models generate truthful answers.Source: §G.1 · p. 23

G.1 MULTILINGUAL NLP BENCHMARKS

  • With 1,000 Alpagasus instructions per language, iterative training yields slight improvements across multiple multilingual NLP benchmarks without alignment tax.Table 17 contains detailed results.Source: §G.1 · p. 23, Table 17 · p. 23

H.2 HEAD-TO-HEAD COMPARISON PROMPT

  • The head-to-head prompt compares two answers to an instruction and requests justifications with integer scores for each answer.The evaluation description also swaps response order and counts a win when one answer wins twice or wins once and draws once.Source: §H.2 · p. 24

H.3 X-ALPACAEVAL PROMPT

  • The X-AlpacaEval prompt selects the best of two unordered model outputs based on quality and relevance, requiring the output language to match the instruction unless specified otherwise.Its response is limited to one model identifier.Source: §H.3 · p. 25

H.4 SELF TRANSLATION PROMPT

  • The self-translation prompt asks the model to translate input sentences into a specified language and wrap the result in designated tags.Source: §H.4 · p. 25

H.5 MULTILINGUAL REASONING PROMPT

  • The multilingual reasoning prompt instructs the model to complete a task and respond in the specified language.Source: §H.5 · p. 26

Related papers

Loading 2410.08964v3…