RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

John Dang; Arash Ahmadian; Kelly Marchisio; Julia Kreutzer; Ahmet Üstün; Sara Hooker

Source-linked AI summary

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, Sara Hooker

arXiv:2407.02552v1Jul 2, 2024cs.CLcs.AIcs.LG

TL;DR

Preference optimization research has focused mainly on English and a few other languages, leaving multilingual transfer and high-quality multilingual preference data insufficiently studied. The study constructs multilingual preference data and systematically compares language coverage, dataset size, DPO, and online RLOO starting from Aya 23 8B across 23 languages. Multilingual data improves multilingual and unseen-language performance, while online preference optimization outperforms offline and best RLOO reaches 69.5%, 72.4%, and 77.5% against Gemma, Llama-3, and Mistral.

Problem
Preference optimization research has focused mainly on English and a few other languages, leaving multilingual transfer and high-quality multilingual preference data insufficiently studied.
Method
The study constructs multilingual preference data and systematically compares language coverage, dataset size, DPO, and online RLOO starting from Aya 23 8B across 23 languages.
Results
Multilingual data improves multilingual and unseen-language performance, while online preference optimization outperforms offline and best RLOO reaches 69.5%, 72.4%, and 77.5% against Gemma, Llama-3, and Mistral.
Takeaways & Limitations
Preference optimization can transfer across languages, and multilingual coverage, larger data, and online samples improve alignment within the studied 23-language scope.
Takeaways & Limitations
The study covers only 23 of roughly 7000 languages, does not distinguish dialects, and is limited in scaling experiments to larger models by compute constraints.

Abstract

from arXiv · show

Preference optimization techniques have become a standard final stage for training state-of-art large language models (LLMs). However, despite widespread adoption, the vast majority of work to-date has focused on first-class citizen languages like English and Chinese. This captures a small fraction of the languages in the world, but also makes it unclear which aspects of current state-of-the-art research transfer to a multilingual setting. In this work, we perform an exhaustive study to achieve a new state-of-the-art in aligning multilingual LLMs. We introduce a novel, scalable method for generating high-quality multilingual feedback data to balance data coverage. We establish the benefits of cross-lingual transfer and increased dataset size in preference training. Our preference-trained model achieves a 54.4% win-rate against Aya 23 8B, the current state-of-the-art multilingual LLM in its parameter class, and a 69.5% win-rate or higher against widely used models like Gemma-1.1-7B-it, Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3. As a result of our study, we expand the frontier of alignment techniques to 23 languages covering half of the world's population.

1 Introduction

The paper studies why multilingual preference optimization has lagged behind English-focused work and tests whether data quality, multilingual coverage, and online training resolve that gap. It finds cross-lingual transfer, benefits from multilingual preference data, and strong results against open-weight baselines.

Motivation: Multilingual preference optimization is difficult because high-quality preference data is scarce, translation can introduce artifacts, and heterogeneous languages can produce negative transfer.These challenges motivate the paper’s systematic study of data source, data amount, language coverage, and online versus offline optimization.
Main findings: 54.9% win-rate on unseen languages follows training on five languages, compared with 46.3% from English-only training.The result demonstrates cross-lingual transfer from preference optimization.
Main findings: Multilingual preference data raises win-rates from 46.4% to 53.4% when all languages are included, a gain of up to 7.0%.The study reports that increasing the number of training languages consistently improves multilingual performance over English-only data.
Main findings: RLOO outperforms DPO by up to 10.6% in average win-rate and improves unseen-language win-rates by up to 8.3%.The reported average win-rates are 54.4% for RLOO and 43.8% for DPO.
Main findings: 54.4% win-rate against Aya 23 8B and at least 69.5% against Gemma-1.1-7B-it, Meta-Llama3-8B-Instruct, and Mistral-7B-Instruct-v0.3 across 23 languages.The reported win-rates are 69.5%, 72.4%, and 77.5% against Gemma, Llama-3, and Mistral, respectively.

2 Methodology

The methodology constructs diverse multilingual preference data while comparing language coverage, data scale, and offline versus online optimization. It uses translated prompts but generates multilingual completions from multiple models to reduce translation artifacts and increase preference diversity.

Addressing Data Scarcity: Prior translation-based multilingual preference training can suffer from translationese artifacts and reduced preference-pair diversity.The paper hypothesizes that low-quality translated data may explain poor results from earlier multilingual preference-training efforts.
Addressing Data Scarcity: Approximately 50K English prompts are translated into 22 additional Aya-supported languages, then paired with multilingual completions from Command and Command R+.Using multiple multilingual-capability models increases completion diversity relative to translating English preference completions.
Addressing Data Scarcity: Translated completions are ranked as bad completions 91% of the time, providing a proxy label that steers the model away from translation artifacts.The method compares translated completions with high-quality direct multilingual generations.
Offline vs Online Preference Training: The study compares DPO with RLOO to test how offline and online generations affect multilingual preference optimization.RLOO is selected as a simpler REINFORCE-style alternative whose prior experiments outperformed PPO and DPO.
Offline vs Online Preference Training: The preference mixtures vary language coverage and budget: English-only 50K, five languages with 50K, all 23 languages with 50K, and all 23 languages with 230K prompts.These variants isolate cross-lingual transfer, multilingual coverage, and the effect of increasing the total preference-data budget.

3 Experimental Set-up

Experiments start from Aya 23 8B, a multilingual 23-language base model, and evaluate preference-optimized variants against open-weight models on multilingual generation and summarization. Evaluation uses GPT-4-Turbo-simulated win-rates across held-out multilingual datasets.

3 Experimental Set-up: Aya 23 8B is used because it supports 23 languages, achieves state-of-the-art performance in its class, and is not preference-trained.This isolates the impact of multilingual preference optimization.
Model Comparisons: The comparison set includes Meta-Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-1.1-7B-it, which are widely used open-source instruction models.These models differ in pretraining data, language coverage, and preference-training procedures.
Model Comparisons: Figure 2 reports DPO and RLOO win-rates for the 23-language average, English, and unseen languages as training-language coverage increases.These views assess overall performance, English transfer, and cross-lingual transfer.
Evaluation: The study evaluates open-ended generation on a 200-instance Dolly test set translated into 101 languages and summarization on XLSum across 15 Aya-supported languages.Human-edited Dolly data is used where available to correct translation issues.
Evaluation: Win-rates are simulated with GPT-4-Turbo, using randomized output order to reduce evaluation bias.The paper notes that LLM-simulated win-rates correlate highly with human evaluation in English and multilingual settings.

4 Results and Discussion

Preference optimization improves multilingual open-ended generation, transfers to unseen languages, and can preserve standard multilingual benchmark performance. Online RLOO generally outperforms offline DPO, while multilingual data coverage and size affect gains and optimization stability.

Win-rates Against Open-Weights Models: 69.5%, 72.4%, and 77.5% win-rates against Gemma-1.1, Llama-3, and Mistral respectively are achieved by the best RLOO variant across 23 languages.The comparisons improve over the base Aya 23 8B by 7.4, 5.8, and 8.5 percentage points, respectively.
Increasing multilinguality in preference data improves winrates: Multilinguality improves some open-ended results: DPO rises from 43.3% to 47.0%, while RLOO reaches 54.0% with five-language training and English rises from 47.5% to 53.0% with 23-language training.For summarization, the 23-language mixture improves over the five-language subset from 65.1% to 70.7%; however, DPO English performance declines as languages increase under a fixed budget.
Cross-lingual transfer to unseen languages: 54.9% versus 46.3% unseen-language win-rates show that five-language training transfers better than English-only training.The corresponding gains over loss rates are 19.4% and 7.3% for RLOO, while online training transfers more strongly than offline training.
Online optimization vs offline: 54.0% versus 47.0% win-rates show that online RLOO outperforms offline DPO for the 23-language mixture.Across multilingual data mixtures, RLOO also achieves higher cross-lingual transfer, including 54.9% versus 43.3% on unseen languages with five-language training.
Role of data size and reward over-optimization: 50.2% versus 47.0% win-rates show that increasing DPO data from approximately 2K to 10K examples per language helps, whereas the same increase does not help RLOO.The 230K-example RLOO run degraded after 0.5 epochs, possibly because online optimization enabled reward-model overoptimization.
Is there a multilingual alignment tax?: Multilingual preference optimization largely matches base Aya 23 8B performance on multilingual benchmarks, with mMMLU changing by +0.3 for DPO and -0.2 for RLOO.MGSM accuracy changes by -0.5 for DPO and -1.5 for RLOO, supporting a minimal alignment tax under the reported setting.

5 Related Work

Prior multilingual preference-training efforts were limited, often relying on translated preference data and RLHF-based instruction tuning. Synthetic data generation and LLM-based evaluation provide related approaches for reducing the cost of collecting human feedback.

Reinforcement Learning from Human Feedback (RLHF): RLHF aligns language models by training a reward model and using online reinforcement learning, while recent work also studies offline alternatives such as DPO.Canonical RLHF commonly uses PPO or RLOO to optimize policies on online samples.
Synthetic Data: Synthetic data generation reduces the expense of obtaining human-written completions or feedback by using LLMs to produce training signals.Distillation commonly uses a stronger model to generate completions for fine-tuning a weaker model.
Preference Optimization for Multilingual LLMs: Multilingual preference optimization has received limited attention, with prior work training models such as BLOOMZ on non-English languages using RLHF and translated preference data.Okapi generated preference data by translating an extended Alpaca dataset and generating responses from the target model.
Preference Optimization for Multilingual LLMs: Translated preference data can improve language coverage but may introduce translation artifacts and reduce preference diversity.Repeatedly translating the same preference pairs can make it difficult to separate data quantity from data quality.

6 Conclusion

The study finds that multilingual preference optimization improves alignment across languages, with gains scaling with language and example coverage and transferring to unseen languages. The resulting preference-trained Aya 23 8B expands alignment to 23 languages and outperforms the original model and widely used open-weight models.

6 Conclusion: Multilingual preference data improves performance over English-only training, with gains scaling with the number of languages and total training examples.The study also reports cross-lingual transfer to languages absent from preference-training data and better performance from online than offline optimization.
6 Conclusion: The study identifies cross-lingual transfer, multilingual data coverage, dataset size, and online sampling as important variables for multilingual alignment.These findings summarize the study’s reported mechanisms and training-design comparisons.
6 Conclusion: Preference-trained Aya 23 8B covers 23 languages and outperforms the original Aya 23 8B and widely used Gemma, Mistral, and Llama 3 models.The 23 covered languages represent about half of the world’s population.

7 Limitations

The paper’s limitations include cultural-bias risks from synthetic and translated data, incomplete linguistic coverage and dialect treatment, and compute constraints that limit experiments with larger models. The supplied background passages also describe preference-optimization stages and methods rather than additional limitations.

7 Limitations: Synthetic and translated datasets may introduce cultural bias because their source prompts over-represent Global North or Western contributions.This may skew model behavior toward a narrower selection of cultural viewpoints.
7 Limitations: The model covers only 23 of roughly 7,000 languages and does not distinguish dialects within the languages it supports.Future work is directed toward broader population and language coverage.
7 Limitations: Compute constraints prevented extensive preference-optimization experiments with larger models.Some individual runs required five days on a single 8 × H100 80GB GPU instance.
Background: The paper’s background describes preference optimization as using feedback data to fine-tune an instruction-tuned policy toward human preferences.Online methods may train a reward model, whereas DPO avoids reward modeling and uses offline preference pairs.
Background: The study uses RLOO as its online method because it is described as more efficient than PPO and reduces variance through multiple samples without an additional network.RLOO estimates each sample’s return using the remaining samples for the same prompt.

B Additional Win-Rate Results

Direct Dolly comparisons show that RLOO models consistently outperform their DPO counterparts across dataset splits. The evaluation reports win rates for models trained with the two methods, while related results account for generation-length differences across models.

Additional Win-Rate Results: RLOO models consistently outperform DPO models in direct Dolly win-rate comparisons across all dataset splits.Table 5 provides the head-to-head comparison between models trained with RLOO and DPO.
Additional Win-Rate Results: The multilingual evaluation compares preference-trained models on 15 covered languages against the original Aya 23 8B model.Average generation lengths were 138 tokens for the base model, 234 for the best DPO model, and 171 for the best RLOO model.

C NLP Benchmark Results

The evaluation reports multilingual benchmark results for base and preference-trained Aya 23 8B models across generative and discriminative tasks. Results cover XLSum, held-out discriminative benchmarks, multilingual MMLU, and MGSM.

C NLP Benchmark Results: Benchmark results compare the base Aya 23 8B model with DPO- and RLOO-trained Aya 23 models across multilingual tasks.The reported evaluations include XLSum, held-out XCOPA, XStoryCloze, and XWinoGrad, multilingual MMLU, and MGSM.
C NLP Benchmark Results: Table 6 organizes 15-language XLSum win rates by training mixture, contrasting English-only, 5-language, and 23-language data for DPO and RLOO.Most runs use 50K total examples; ML-23* uses 230K, with the best checkpoint selected across two epochs.
C NLP Benchmark Results: Held-out discriminative evaluation averages zero-shot performance across XCOPA, XStoryCloze, and XWinoGrad, while MMLU and MGSM provide additional multilingual benchmark results.MGSM uses native-language chain-of-thought prompts and strict-match scoring.

E Full Language Set Win-Rates

The paper reports full win-rate results for the 23-language evaluation set against Aya 23 8B and additional comparisons against three widely used instruction-tuned models.

E Full Language Set Win-Rates: Full 23-language win-rate results are reported for ML-23-230K DPO and RLOO runs against the base Aya 23 8B model.The DPO and RLOO results are provided in Tables 13 and 14, respectively.
E Full Language Set Win-Rates: The ML-23-230K RLOO run is additionally evaluated against Gemma-1.1-7B-it, Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3.These comparisons are reported in Tables 10, 11, and 12, respectively.

F Language List

The paper defines the 23-language evaluation coverage used by Aya 23 and supplies supporting results organized by training method and comparison model.

F Language List: Table 15 lists the 23 languages supported by Aya 23, together with each language’s script, family, and subgrouping.This table describes the language inventory used for multilingual evaluation.
F Language List: Additional tables organize all-language DPO and RLOO results and RLOO comparisons against Gemma, Llama, and Mistral instruction-tuned models.Tables 10–14 cover these model comparisons and training runs.