PaperlayerView exact revision on arXiv

Source-linked AI summary

Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment

Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, Ahmad Beirami

arXiv:2404.12318v2cs.CL

TL;DR

Because multilingual human preference data are difficult to obtain at scale, the paper tests whether a reward model trained in one language can align models in others. Across summarization and open-ended dialog, cross-lingual transfer is consistently successful and sometimes outperforms same-language reward models, including in settings without target-language supervised-finetuning data.

  • Problem

    Human preference data are costly to gather across languages, leaving open how to align a language model in a target language without preference data in that language.

    Source: §1 · p. 1

  • Method

    The paper applies a reward model trained on source-language preference data directly to optimize a model in a target language.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Across summarization and open-ended dialog, reward-model transfer is consistently useful across evaluation settings and sometimes outperforms same-language reward models.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The findings point toward lower training and deployment costs for language models serving users worldwide.

    Source: §1 · p. 2

    Free-form generation evaluation is imperfect, and the many reward-optimization methods cannot all be studied; some may not benefit from cross-lingual reward-model transfer.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Aligning language models (LMs) based on human-annotated preference data is a crucial step in obtaining practical and performant LM-based systems. However, multilingual human preference data are difficult to obtain at scale, making it challenging to extend this framework to diverse languages. In this work, we evaluate a simple approach for zero-shot cross-lingual alignment, where a reward model is trained on preference data in one source language and directly applied to other target languages. On summarization and open-ended dialog generation, we show that this method is consistently successful under comprehensive evaluation settings, including human evaluation: cross-lingually aligned models are preferred by humans over unaligned models on up to >70% of evaluation instances. We moreover find that a different-language reward model sometimes yields better aligned models than a same-language reward model. We also identify best practices when there is no language-specific data for even supervised finetuning, another component in alignment.

1 Introduction

Because human preference data are costly to gather across languages, the paper asks how to align a target-language model without target-language preference data. It proposes transferring a source-language reward model and reports consistent cross-lingual gains, sometimes exceeding same-language reward-model alignment.

1 Introduction

  • The approach repurposes a reward model trained for a source language to align a model in a different target language.Figure 1 illustrates English-to-Spanish transfer in place of using a Spanish reward model.Source: §1 · p. 1, Figure 1 · p. 1
  • Human win rates against the target-language unaligned SFT model show that source-language reward models can improve target-language alignment, sometimes more than target-language reward models.Figure 2 reports human judgments and 95% confidence intervals across validation instances.Source: Figure 2 · p. 1
  • Cross-lingual reward-model alignment consistently helps across summarization and open-ended dialog, reward-optimization methods, and evaluation settings, sometimes outperforming same-language reward models.The reported pattern holds under both human and language-model judgments.Source: §1 · p. 2
  • The findings suggest reward signals generalize across input distributions and may help lower costs for training and deploying language models for more users worldwide.Source: §1 · p. 2

2 Background: Alignment From Human Feedback

The background describes alignment as supervised finetuning, reward modeling, and reward optimization. It explains how these stages use task examples and human judgments to improve model outputs.

2 Background: Alignment From Human Feedback

  • Alignment commonly combines supervised finetuning, reward modeling, and reward optimization after pretraining a base model.The base model is assumed to have been trained with a next-token prediction objective.Source: §2 · p. 2
  • Supervised finetuning trains the base model on task inputs and example outputs by maximizing the conditional likelihood of each output given its input.The resulting model is denoted πSFT.Source: §2 · p. 2
  • Reward modeling trains a proxy for human-judged output quality, initialized from πSFT and fitted to human judgments.The background distinguishes pointwise and pairwise feedback for this training.Source: §2 · p. 2
  • Pointwise feedback labels a single generation good or bad for binary classification, while pairwise feedback selects a preferred generation and can train a reward model with the Bradley-Terry model.The pairwise formulation can also generalize to more than two outputs.Source: §2 · p. 2
  • Reward optimization either reranks multiple πSFT samples with the reward model at inference time or changes the model through reinforcement learning to favor high-scoring samples.Source: §2 · p. 2
  • The RL objective uses a dataset of inputs DRO and regularization hyperparameter β, is typically optimized with PPO (Schulman et al., 2017), and best-of-n provides a cleaner testbed in some analyses.Source: §2 · p. 2

3 Reward Model Transfer for Cross-Lingual Alignment

The paper transfers a reward model trained in one language to align a model in another, addressing the expense of collecting target-language reward judgments. The proposed approach relies on cross-lingual generalization and is compared with automatically translated reward data.

3 Reward Model Transfer for Cross-Lingual Alignment

  • Target-language alignment ordinarily requires both supervised-finetuning data and human-judgment reward-model data, with reward-model data particularly costly and ideally recollected periodically to avoid over-optimization.The paper therefore focuses mainly on alignment without target-language reward-model data, while also examining settings without target-language supervised-finetuning data.Source: §3 · p. 2
  • The proposed setup uses a reward model trained for a source language to optimize a model in a target language, assuming multilingual training enables the reward model to understand both languages.The intuition is that generation quality can transfer across languages; the paper notes this as an assumption.Source: §3 · p. 3
  • The paper compares reward-model transfer with a simple baseline that automatically translates reward-model training data.Source: §3 · p. 3

4 Experimental Setup

The experiments test summarization and open-ended dialog generation using multilingual datasets and language models. Model quality is assessed with reward models, prompted language models, and human pairwise judgments.

4 Experimental Setup

  • Experiments cover summarization and open-ended dialog generation, tasks with substantial real-world relevance.Source: §4 · p. 3
  • Summarization uses Seahorse human pointwise ratings in six languages, SFT data from four multilingual summarization sources, and the 3.7B-parameter mT5-XL base model.The six languages are German, English, Spanish, Russian, Turkish, and Vietnamese; the SFT sources are XSum, XL-Sum, MLSum, and WikiLingua.Source: §4 · p. 3
  • Dialog uses multilingual pairwise human ratings from OpenAssistant and finetunes on each pair's human-preferred response, focusing on English, Spanish, and Russian because other languages have limited data.Source: §4 · p. 3
  • The transfer assumption may be weak for tasks or instances shaped more strongly by culture-specific factors, where generations may be judged differently across languages.Source: §4 · p. 3
  • Dialog experiments use PaLM-2-XXS and focus on best-of-n because the authors found reinforcement learning ineffective on OpenAssistant.They report confirming the ineffectiveness in their own experiments.Source: §4 · p. 3
  • Evaluation compares aligned-model responses with SFT responses using target-language reward models, zero-shot-prompted GPT-4 and PaLM-2-L, and native or advanced speakers.Human evaluation covers only a subset of language pairs and 250 RL or 100 best-of-n instances per pair; win rate is the fraction of pairwise judgments preferring the aligned model.Source: §4 · p. 3

5 Results

Cross-lingual reward optimization generally improved over target-language SFT and sometimes outperformed monolingual alignment. It remained useful without target-language SFT data, while domain match and source-language choice shaped practical recommendations.

5.1 Cross-Lingual Alignment Is Effective

  • Cross-lingual reward optimization improved over SFT in all cases under target-language RM evaluation and generally improved quality under PaLM-2-L and GPT-4.Human evaluation showed the same trend, with larger confidence intervals; PaLM-2-L and GPT-4 also agreed with humans at rates similar to reported human-human agreement.Source: §5.1 · pp. 3–4, Figure 3 · p. 4, Figure 4 · p. 5
  • Averaged across 30 cross-lingual pairs, RM transfer beat translate-train under best-of-n by win rate 58.8 vs. 57.5.RM transfer also requires training one source-language RM rather than separate RMs for each target language.Source: §5.1 · p. 4

5.2 Cross-Lingual Alignment Sometimes Outperforms Monolingual Alignment

  • Cross-lingual optimization often outperformed target-language-RM alignment, possibly because language mismatch reduces policy overfitting to lexical artifacts.Monolingual RM use was more bag-of-words-like (R2 0.65 vs. 0.63); translate-train underperformed, though translation quality confounds that evidence.Source: §5.2 · pp. 4–5

5.3 Cross-Lingual Alignment Without Target-Language SFT Data

  • Without target-language SFT, translated SFT substantially hurt SFT and best-of-n quality, while domain matching narrowed gaps and translated SFT often preserved RL performance.Round-trip back-translation bridged most of the gap attributed to translation errors; translated SFT sometimes outperformed the original under domain control, and RL results had degeneration exceptions.Source: §5.3 · pp. 5–6, Figure 5 · p. 6

5.4 Practical Recommendations

  • Organic target-language SFT data are preferred when available; otherwise, translation may help, with attention to domain match or greater reliance on RL.Source-language rankings were generally stable across targets, especially for best-of-n; English RMs were often highly ranked and most frequently best.Source: §5.4 · pp. 6–7, Figure 6 · p. 7

6 Analysis

Cross-lingual alignment depends on reward models generalizing across languages, but that generalizability alone does not explain downstream success. Simple language features and KL-divergence likewise do not fully account for alignment outcomes.

6 Analysis

  • RM generalizability within reward modeling is a prerequisite for cross-lingual alignment, but other factors also affect downstream success.The ranking of source languages by reward-model generalizability has low agreement with alignment utility; for summarization, Kendall’s τ = 0.1.Source: §6 · p. 7, §6.1 · pp. 7–8

6.1 Impact of RM Generalizability Within Reward Modeling

  • Cross-lingual RMs often exceed majority-baseline accuracy for summarization and random performance for dialog, though improvements are modest and dialog sometimes falls below the length baseline.The RMs’ accuracies are near human-agreement levels; even apparently weak RMs can achieve high cross-lingual alignment effectiveness.Source: §6.1 · p. 7
  • Source-language RM accuracy and generalizability rankings do not match downstream transfer utility; for summarization, source-language rankings have Kendall’s τ = 0.1 with alignment win rates.Turkish has the highest in-language validation accuracy in Figure 7 but is not best in Figure 6; the ranking result is the same for best-of-n and RL.Source: §6.1 · pp. 7–8

6.2 Impact of Language Features

  • Transfer utility rankings do not match language frequency or SFT and RM data quantities, and WALS features show no significant association after Bonferroni correction.English benefits most from Vietnamese for summarization RL despite their disparate language families; Russian’s weak transfer and non-Latin script offer no clear general pattern.Source: §6.2 · p. 8

6.3 Impact of Policy Divergence

  • Win rates show no clear correlation with KL-divergence from the SFT model, so the authors report no observed reward over-optimization.Four of 123 WALS features are significant without correction; the authors regard these as likely spurious correlations.Source: §6.3 · pp. 8–9, Figure 8 · p. 8

7 Related Work

The paper situates reward-model transfer within zero-shot cross-lingual transfer and multilingual alignment research. It distinguishes its downstream alignment study from prior work focused on supervised fine-tuning or translated data.

Zero-shot cross-lingual transfer

  • Zero-shot cross-lingual transfer commonly fine-tunes a multilingual LM on a source-language task and evaluates it in another language; this paper extends the framework to downstream alignment.Prior alignment studies cited here focus on supervised fine-tuning and translated data.Source: §7 · p. 9
  • The paper’s transfer setup follows the broader cross-lingual generalizability tradition but tests whether that generalizability is useful for alignment.The cited nearby alignment studies examine SFT using translated data, rather than reward-model transfer.Source: §7 · p. 9

Multilingual Alignment

  • Prior multilingual SFT work collects, assembles, or translates instruction data, while some prior work also translates reward-model data.The paper reports that cross-lingual transfer outperforms translation for reward-model data.Source: §7 · p. 9

8 Conclusion

The paper concludes that reward models trained in one language can support alignment in another across two tasks. It also reports cases where a different-language reward model is more effective and identifies issues and remedies when target-language SFT data are unavailable.

8 Conclusion

  • Experiments across two tasks show that alignment can use a reward model trained in a different language.Source: §8 · p. 9
  • A different-language reward model sometimes aligns more effectively than a same-language reward model.Source: §8 · p. 9
  • The paper identifies issues and remedies for alignment settings without target-language SFT data and suggests domain generalization as future work.Source: §8 · p. 9

Limitations

The authors qualify their findings by noting limitations in evaluation, methods, language coverage, and assumptions about generation quality. The accompanying details describe the summarization data and training settings.

Limitations

  • Cross-lingual free-form generation is difficult to evaluate, and the study covers only best-of-n and PPO, uses a single reward score, and tests relatively high-resource languages.The authors note that results may differ for lower-resource source languages or reward-optimization methods not studied.Source: Limitations · p. 9
  • The assumption that generation quality is language-agnostic may fail for culture-specific tasks or instances, where the authors expect reduced cross-lingual generalizability.Source: Limitations · p. 9

A Dataset Details and Statistics

B Training Details

  • SFT learning rates were 10^-3 for summarization and 10^-5 for dialog; RM used 10^-4, PPO ran 3,000 and 2,500 steps respectively, and best-of-n used n = 64.Source: B Training Details · p. 15

C LM Judge Accuracy on Ground-truth Reward Modeling Data

The authors check whether an LM judge can assess generation quality by comparing its reward-model validation accuracy with human agreement. They report that these checks support using an LM judge in this setup.

C LM Judge Accuracy on Ground-truth Reward Modeling Data

D GPT-4 as a Judge Results

GPT-4 evaluation shows the same general pattern as earlier results: cross-lingual reward optimization is effective and can sometimes outperform monolingual optimization. The section also examines reward-model transfer, bilingual reward models, and the prompts used for tasks and evaluation.

D GPT-4 as a Judge Results

  • Cross-lingual reward optimization is generally effective and sometimes outperforms monolingual optimization; GPT-4 and PaLM-2-L agree on 72% and 71% of summarization instances in English and Spanish, and 75% and 73% for dialog.GPT-4 evaluations were capped at 1,000 instances per dataset row; the reported judge agreement exceeds baseline human-human agreement.Source: D GPT-4 as a Judge Results · pp. 16–17

E Verifying RM Transfer for Reward Modeling

  • In most tested language pairs, source-language reward models assign higher scores to monolingually aligned target-language models than to unaligned SFT models.The score difference also increases during RL training; the OpenAssistant RL case is omitted because RL training is difficult there.Source: E Verifying RM Transfer for Reward… · p. 17, Figure 10 · p. 17

F Alignment Using Bilingual RMs

  • The bilingual reward model, trained from pooled German and English SFT and reward data, shows no noticeable gain over the better English reward model.The authors leave systematic validation of whether bilingual models match the best monolingual model to future work.Source: F Alignment Using Bilingual RMs · p. 17, Figure 11 · p. 18

G.1 Task Instructions

  • Task instructions substitute the target language for [LANGUAGE], while reward modeling concatenates texts with delimiters and uses no task prompt.Source: §G.1 · p. 18

G.2 Evaluation Prompts

  • Human and GPT-4 comparisons randomize output order; PaLM-2-L is prompted with both orderings and its probabilities for “1” and “2” are averaged.Source: §G.2 · p. 18
  • Evaluation prompts ask raters or models to choose between two outputs, with summary judgments based on coherence, accuracy, coverage, and overall quality.The dialog-generation prompt is adapted from Li et al. (2023b), while the summarization prompt is adapted from Lee et al. (2023).Source: §G.2 · pp. 18–19

H Raw Results

This section points readers to raw numerical results corresponding to the paper’s plots, covering alignment, reward-model generalizability, and supervised-finetuning strategies.

H Raw Results

  • Tables 10 and 11 report cross-lingual alignment win rates for summarization and dialog generation, respectively, measured against the target-language SFT model and judged by GPT-4 using best-of-n with n = 64.Source: Table 10 · p. 20, Table 11 · p. 20
  • Tables 18–20 give reward-model validation accuracy for summarization and dialog, plus RL-model KL divergence from target-language SFT models.Source: Table 18 · p. 21, Table 19 · p. 21, Table 20 · p. 21
  • Tables 21–25 report ROUGE-L and alignment performance under alternative SFT strategies, and compare monolingual with bilingual reward models.The alignment comparisons are measured as win rates against the monolingual target-language SFT model, as specified in the captions.Source: Table 21 · p. 21, Table 22 · p. 21, Table 23 · p. 22
Loading 2404.12318v2…