PaperlayerView exact revision on arXiv

Source-linked AI summary

CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations

Mike Zhang, Ali Basirat, Desmond Elliott

arXiv:2605.26293v1cs.CLcs.AI

TL;DR

Multilingual preference tuning lacks evidence that reward-distribution-based pair construction transfers without language-specific annotation. CroCo applies DPO to multilingual self-generations scored by an English-preference-trained reward model, finding broad gains, reduced forgetting, and a strong dependence on on-policy data.

  • Problem

    The paper asks whether reward-distribution-based contrastive preference tuning transfers across languages without language-specific preference annotation.

  • Method

    CroCo constructs contrastive preference pairs from multilingual policy self-generations scored by an English-only reward model atop a multilingual base, then applies DPO.

  • Results

    Paired DPO improves over the respective bases in most structured settings, beats EuroLLM-9B in 10/14 and aya-3B in 11/14 EuroEval settings, and wins across all 11 evaluated open-ended languages.

  • Takeaways & Limitations

    Cross-lingual self-generation contrastive tuning can provide multilingual alignment without per-language preference annotation while avoiding the forgetting seen with SFT.

  • Takeaways & Limitations

    The evidence covers fourteen European, Latin-script languages, while transfer to typologically distant, non-Latin, or genuinely low-resource languages remains open.

Abstract

from arXiv · show

Prior work establishes that controlled contrastiveness between self-generated responses from large language models, set via reward scores, improves downstream preference tuning in English. We extend this method to multiple languages and evaluate two models across a total of 14 high and low-resource languages on a diverse set of tasks. Our central finding is that cross-lingual contrastive preference tuning on self-generations (CroCo) transfers without language-specific preference annotation. A reward model trained on English preferences (atop a multilingual base) produces useful within-language rankings across most languages, and pairing in either a monolingual or multilingual setting improves over each model on the majority of setups while preventing the catastrophic forgetting of supervised fine-tuning. We observe that the gains require on-policy data. Off-policy responses reduce the benefit and online preference optimization fails to improve over the offline variant. Specifically, on structured tasks, our method matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for Aya-3B. On open-ended generation, both tuned models win against their respective base across 11 evaluated languages. Overall, we show promising directions for multilingual preference tuning.

1 Introduction

Prior English-only findings motivate testing whether reward-distribution-based contrastive preference tuning transfers across languages without language-specific preference annotation. CroCo examines this question across monolingual and multilingual regimes and models of different scales.

  • The approach pairs a policy model with a reward model that scores on-policy responses, extending self-generated preference tuning previously studied in English.
  • Multilingual transfer remains unresolved because prior approaches use translation-based signals, capability gaps, or loss reweighting rather than testing reward-distribution-based pair construction.
  • CroCo asks whether contrastive preference tuning on self-generations transfers across languages without language-specific preference annotation.
  • The hypothesis is that relative within-language reward rankings can suffice despite cross-lingual miscalibration, allowing an English-only reward model atop a multilingual base.
  • CroCo reports gains across models and languages while standard supervised fine-tuning causes catastrophic forgetting in both models.
  • On structured evaluation, multilingual Paired DPO matches or exceeds the base in 6/7 languages for EuroLLM-9B and 4/7 settings for aya-3B.

2 Problem Formulation

The method constructs contrastive preference pairs from policy self-generations scored by an external reward model, then applies DPO in monolingual or multilingual settings. Pair selection targets a controlled reward contrast rather than the lowest-scoring response.

  • DPO optimizes a policy using preference pairs containing a chosen completion and a rejected completion relative to a frozen reference model.
  • The dataset quality is central to downstream performance, motivating controlled construction of preference pairs.
  • For each prompt, the policy generates K candidates and an external reward model scores every candidate.
  • The rejected response is selected near µ_x − 2σ_x in the reward distribution, creating controlled contrastiveness instead of choosing the minimum-reward candidate.
  • The multilingual extension translates English prompts into target languages, generates language-specific responses, and tunes either each language independently or all languages jointly.
  • The study tests robustness across two models with different scales, 3B and 9B parameters.

3 Experimental Setup

The experiments translate a stratified English instruction sample into European languages, generate and reward-score on-policy responses, and compare supervised, reward-filtered, and paired-DPO constructions. Evaluation spans multilingual structured tasks and held-out-language generalization.

  • 3.1 Data: The data comprise a stratified 20K sample from Dolci-Instruct-SFT translated into Danish, Dutch, French, German, Italian, and Spanish.
  • 3.2 Pair construction: EuroLLM-9B and aya-3B generate 64 responses per instance, which are scored by Skywork-Reward-V2-Qwen3-8B, an English-preference-trained reward model with a multilingual base.
  • 3.1 Data: The sample covers nine task domains, with coding, reasoning, chat, and math accounting for most instances.
  • 3.2 Pair construction: The four data strategies are in-language or all-language SFT, Max-R SFT retaining only the highest-scoring response, and Paired DPO using contrastive pairs.
  • 3.3 Evaluation: EuroEval covers 32 datasets across seven target languages and multiple task types, with Norwegian, Portuguese, and Swedish reserved for cross-lingual generalization.
  • 3.3 Evaluation: Table 1 reports language- and model-level EuroEval aggregates, with base scores and tuned-model differences summarized over three seeds.
  • 3.3 Evaluation: Table 2 evaluates held-out-language transfer using dataset-averaged absolute differences from the EuroLLM-9B baseline.

4 Results

Paired DPO generally improves multilingual performance over the base while avoiding the substantial degradation caused by translated-data SFT. Gains extend across structured and open-ended tasks, high- and low-resource languages, and comparisons with larger models.

  • Structured-task results: SFT loses 3.8–9.9 points for EuroLLM-9B in 6/7 languages and 3.8–11.3 points for aya-3B in all 7, demonstrating catastrophic forgetting.Monolingual and multilingual SFT both degrade performance relative to baseline across nearly all languages and both models.
  • Structured-task results: Paired DPO beats the EuroLLM-9B baseline in 10 of 14 settings and is non-negative for aya-3B in 12 of 14, with no drop exceeding 1.3 points.The largest EuroLLM-9B gain is +3.6 on Italian under monolingual training; aya-3B has one meaningful drop, −0.3 on French multilingual training.
  • Structured-task results: Multilingual preference baselines degrade EuroLLM-9B in most applicable languages, while Paired DPO is the only method non-negative on average across all evaluated languages.ICR and MAPO lose as much as 7–10 points on Spanish, whereas Paired DPO avoids consistent degradation.
  • Open-ended generation: Paired DPO wins against both bases in all seven open-ended-generation languages, with LC win rates of 54.5%–58.4% for EuroLLM-9B and 55.5%–66.3% for aya-3B.The pattern holds across models differing by 3× in parameter count.
  • Open-ended generation: DPO narrows the gap to the larger Gemma model in 5/7 languages for EuroLLM-9B and all 7 for aya-3B.The method moves models trained on their own outputs closer to larger references they never observed.
  • Low-resource languages: Paired DPO improves EuroLLM-9B over the base in all four low-resource languages and outperforms all baselines against Gemma3-12B-it in Galician and Maltese.LC win rates reach 60.7% in Galician and Welsh, 58.6% in Maltese, and 54.0% in Irish.

5 Discussion

Translation is unnecessary for most CroCo setups, while prompt-language alignment and on-policy data materially affect performance. Offline DPO benefits from on-policy generation, whereas online DPO fails to improve over the baseline in the tested setting.

  • Does Translation of the Data Help?: In-language translation improves paired DPO in four of six EuroLLM-9B languages but worsens standard SFT, indicating that relative reward gaps tolerate translation noise better than absolute targets.Paired DPO benefits in Danish, German, French, and Italian, with the largest Italian difference at +3.6 versus −0.7 for English-only data.
  • Does Prompt Language Matter?: Matching the prompt language to the chosen response produces gains or ties in every language except Italian, while alternative prompt assignments degrade performance in most languages.The alternative variants lose up to 4.7 points on French.
  • Does Off-policy Data Work?: Off-policy paired DPO reduces forgetting but yields gains of at most +1.7 points, far below on-policy wins in 10/14 EuroLLM-9B and 11/14 aya-3B settings.The effect appears regardless of which model supplies the off-policy data, supporting the importance of on-policy provenance rather than generator quality.
  • Offline versus Online DPO: Offline DPO reaches roughly +0.6 over baseline by step 200 on Danish EuroLLM-9B tasks, while online DPO remains within ±0.2 and has higher variance.The comparison uses 16 responses scored by the same reward model.
  • Offline versus Online DPO: Online DPO underperforms with an external reward model because live optimization can amplify reward-model biases, whereas offline DPO fixes the labels during dataset construction.This explanation is presented as a feedback-loop account of the observed offline–online difference.

6 Related Work

Related work has studied preference tuning, multilingual alignment, and self-distillation, but CroCo targets reward-distribution-based pair construction across languages. The paper extends this setup using one English-trained reward model across seven European languages and two model scales.

  • Preference Tuning and Data Construction: Prior DPO research increasingly emphasizes chosen-response quality and relative reward gaps, including reward-distribution choices for rejected responses.These findings were established in English-only settings.
  • Multilingual Preference Alignment: Multilingual preference alignment has used systematic multilingual evaluations, translation-based signals, English–non-English capability gaps, and loss reweighting for noisy pairs.The cited approaches provide alternatives to reward-distribution-based multilingual pair construction.
  • Self-distillation: Self-distillation transfers high-resource ability through model-generated supervision, while newer on-policy variants teach weaker selves from privileged context over their own rollouts.These methods are closely related through self-generated or on-policy supervision.
  • Reward Models: Prior evidence indicates that English-trained reward models can transfer cross-lingually, although substantial multilingual reward-model quality gaps remain.This motivates testing whether within-language rankings suffice for CroCo.
  • This Work: CroCo extends contrastive preference tuning from English to seven European languages and two models, scoring on-policy samples with a single reward model.The evaluated models are EuroLLM-9B and aya-3B.

7 Conclusion

CroCo extends contrastive preference tuning across languages, models, datasets, and task types. Paired self-generations improve multilingual performance while avoiding the forgetting seen with SFT, and relative reward gaps remain useful under translation noise.

  • Conclusion: CroCo extends contrastive preference tuning from English to multiple languages across EuroLLM-9B, aya-3B, 32 language-specific datasets, and m-ArenaHard 2.1.The study covers 14 European languages.
  • Conclusion: 10 of 14 EuroEval settings improve for EuroLLM-9B and 11 of 14 for aya-3B, with no loss exceeding 1.3 points.CroCo also wins across all tested high- and low-resource languages on m-ArenaHard 2.1.
  • Conclusion: SFT on translated or reward-filtered data causes substantial forgetting, whereas paired self-generation DPO improves performance while preserving the baseline more reliably.The relative reward gap remains informative under translation noise where an absolute SFT target does not.

Limitations

The findings are bounded by language coverage, machine-translated data, a single reward model, and parameter-efficient training on models no larger than 9B parameters.

  • Scope: The study covers 14 European languages written in Latin script, leaving transfer to typologically distant, non-Latin, and genuinely low-resource languages open.Only four tested languages are described as genuinely low-resource within the study’s broader coverage.
  • Data: Machine translation from one English instruction corpus may confound results through translation artifacts and source-domain effects, particularly for noisy Italian data.The translations were produced with TranslateGemma-27B.
  • Reward Model: The reward signal comes from one off-the-shelf model, and the study does not directly measure within-language ranking quality or test whether another reward model changes results.This limits conclusions about reward-model generality.
  • Training and Scale: All fine-tuning uses LoRA and the largest model has 9B parameters, so applicability to full-parameter training or larger models remains uncertain.The passage identifies this as a scope boundary for the conclusions.

Ethics Statement

The method can align multilingual models with reward-model preferences, including harmful preferences, without introducing capabilities that meaningfully exceed the base models.

  • The alignment pipeline could be used with reward signals encoding harmful preferences.
  • The authors state that the released method does not introduce capabilities meaningfully exceeding the underlying base models.

A Representative Samples from the Reward Distribution

Reward scores provide meaningful within-language rankings across models and languages, while remaining uncalibrated across languages. The resulting distributions support using an English-trained reward model for multilingual preference construction.

  • Within-language ranking: Representative samples show that low-reward responses are less coherent or on-task than maximum-reward responses while remaining attempts at the same task.This contrastiveness condition is illustrated for benign, safety-relevant, and social-media prompts.
  • Cross-lingual reward distributions: Per-language reward means and spreads are similar enough within each model to support English-preference-trained reward models as within-language rankers.The models differ in overall reward level, but aya-3B has comparable spread and the method requires within-language ranking rather than cross-language calibration.
  • Pair composition: The multilingual paired dataset includes each language as both chosen and rejected, rather than defaulting to English as the chosen response.
  • Experimental controls: The reported pipeline fixes LoRA settings across SFT and DPO runs so differences are attributable to data construction and loss rather than adapter configuration.The configuration uses rank r = 16, α = 32, and dropout 0.05 across attention and MLP projection matrices.
  • Evaluation: The evaluation suite covers multilingual datasets and reports absolute differences from the EuroLLM-9B baseline for prompt-language ablations.

G Offline vs Online DPO

Offline DPO outperforms online DPO in the Danish EuroLLM-9B comparison. Offline training reaches a higher, stable improvement, whereas online training remains near baseline with greater variance.

  • +0.6 improvement over baseline is reached by offline DPO around step 200 and then maintained.
  • With an external reward model, online training can create a feedback loop that amplifies reward-model biases and encourages exploitation of over-weighted features.Offline DPO instead uses the reward model as a fixed labeler during dataset construction.

I Per-Dataset Results

Per-dataset and subcategory results show that paired DPO improves or preserves performance more reliably than SFT and narrows gaps to larger instruction models in most evaluated settings.

  • Detailed results: The appendix provides per-dataset results for both models, prompt-language ablations, English-only versus translated post-training, and off-policy data.
  • Open-ended generation: The EuroLLM-9B base loses across all m-ArenaHard subcategories and languages, with the largest deficits on math.
  • Open-ended generation: After DPO, EuroLLM-9B win rates rise across most language-subcategory cells, with creative writing showing the most consistent improvement.
  • Open-ended generation: Paired DPO improves all aya-3B language-subcategory cells, with the largest gains on German and French.
  • Held-out languages: Paired DPO generalizes positively to 7/11 held-out EuroEval datasets, while multilingual SFT degrades performance on all held-out datasets.
Loading 2605.26293v1…