PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Chain in Alignment: Cross-Lingual Ranking Preference Optimization

Seungyoon Lee, Minhyuk Kim, Jungseob Lee, Heuiseok Lim

arXiv:2608.23149v1cs.CLcs.AI

TL;DR

English-centric preference alignment leaves performance in other languages suboptimal, motivating methods that transfer English preference knowledge across languages. CRPO uses hierarchical ranking over parallel preference pairs to align target-language language consistency and response quality, and experiments report consistent gains in instruction-following and knowledge-intensive tasks.

  • Problem

    English-centric preference data can leave other languages with inconsistent and degraded alignment, while existing approaches do not effectively leverage internalized English preference knowledge.

  • Method

    CRPO transfers English preference knowledge through hierarchical ranking of parallel English–target-language preference pairs, jointly modeling language and response quality beyond binary comparisons.

  • Results

    CRPO consistently improves instruction-following and knowledge-intensive task performance and enhances the likelihood preference for desirable responses.

  • Takeaways & Limitations

    Cross-lingual hierarchical ranking provides a preference-alignment approach that supports high-fidelity, precise responses in the target language.

  • Takeaways & Limitations

    Processing four response candidates in one step requires more training computation than standard binary comparisons.

Abstract

from arXiv · show

The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-Lingual Ranking Preference Optimization (CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment.

1 Introduction

English-centric alignment data can leave other languages with inconsistent or degraded responses. CRPO addresses this by transferring English preference knowledge through hierarchical cross-lingual ranking, improving target-language alignment and task performance.

  • English-centric alignment gains remain limited outside English, where models can produce inconsistent languages and degraded responses.
  • Existing multilingual approaches often require costly annotation or target-language data but fail to connect internalized English preferences with target-language alignment.
  • CRPO leverages inherent English preference knowledge to align the target-language preference distribution.
  • CRPO uses parallel English–target-language preference pairs and hierarchical ranking to jointly represent language consistency and response quality.
  • CRPO achieves substantial improvements in instruction-following and knowledge-intensive tasks, while producing more accurate likelihood preferences for winning and losing responses.
  • The cross-lingual preference hierarchy itself induces significant performance gains in target-language alignment.

2 Related Work

Multilingual alignment research has often depended on translated or heterogeneous data and can retain language bias. CRPO extends ranking-based alignment by jointly modeling language and quality across languages.

  • Multilingual capability research commonly translates English data or mixes multilingual corpora, but these approaches may require large datasets or retain language bias.
  • Prior multilingual alignment methods extend English-proven alignment algorithms, use zero-shot reward transfer, or derive signals from external translation models.
  • Earlier list-based ranking studies mainly target quality or safety and remain English-centric.
  • CRPO jointly models linguistic and quality hierarchies in a cross-lingual ranking structure, yielding meaningful target-language capability improvements.

3 CRPO Framework

CRPO extends DPO from binary comparisons to hierarchical ranking across parallel English and target-language responses. It jointly optimizes language consistency and response quality using LambdaLoss-based signals and an NLL term.

  • CRPO extends DPO’s pairwise alignment capabilities into a ranking paradigm for cross-lingual contexts.
  • LambdaLoss jointly optimizes multiple candidate responses across a global ranking structure rather than relying only on binary comparisons.The ranking objective can weight pairwise transitions according to their impact on overall list quality.
  • CRPO constructs parallel English and target-language preference pairs and assigns hierarchical relevance labels based on response quality and language type.
  • The hierarchy aligns intra-lingual and cross-lingual preferences while prioritizing the preferred target-language response to preserve input-output language consistency.
  • CRPO combines ranking-based alignment with an NLL objective through LCRPO = (1 − α) · LNLL + α · Llambda(πθ; πref, β).The hyperparameter α balances the ranking-based alignment and language-modeling objectives.
  • CRPO explores LambdaRank, nDCG2, and nDCG2++ weighting schemes, using nDCG2 as the primary scheme because simpler schemes can sufficiently align multilingual preference distributions.nDCG2 provides localized gradients for rank inversion, while nDCG2++ combines nDCG2 with LambdaRank.

4 Experimental Setup

The experiments compare CRPO with established baselines across five languages, three model families, parallel preference data, and multilingual instruction-following and knowledge-understanding evaluations. The setup also examines reward dynamics and generated-response quality.

  • Models and Training: Experiments use Llama-2-7B, Llama-3-8B, and Mistral-7B-v0.1 across Chinese, Indonesian, Korean, Swahili, and Bengali.The languages span high-, medium-, and low-resource settings with typological and script diversity.
  • Models and Training: Each language dataset contains 3,000 UltraFeedback instances translated into parallel target-language preference pairs corresponding to the original English pairs.
  • Models and Training: The parallel dataset, including English pairs, is split into 90% training and 10% test data.
  • Baselines: CRPO is compared with SFT+DPO, which trains on same-language response pairs, and CLO, which uses binary English-target-language comparisons.CLO designates the target-language response as chosen and restricts NLL loss to target-language responses.
  • Evaluation: Multilingual AlpacaEval evaluates conversational ability on 805 questions using win rate and length-controlled win rate against the SFT model.
  • Evaluation: MMMLU and Bele assess target-language knowledge utilization, linguistic comprehension, literacy, and contextual understanding.MMMLU covers 57 subjects, while Bele uses four-choice reading-comprehension questions scored by answer log-likelihood.
  • In-depth Analysis: Intrinsic analyses examine reward distributions, preference-pair log probabilities, and externally modeled response-quality distributions.

5 Results and Analysis

CRPO achieves the strongest overall results across languages, including greater robustness in low-resource settings and improved knowledge utilization. Its hierarchical ranking signals also produce stronger reward and chosen-response likelihood shifts than comparison methods.

  • Overall Performance: CRPO achieves the best results across languages, while other methods show notable degradation in low-resource settings.The degradation is particularly pronounced for Llama-3-8B in Bengali, whereas CRPO remains consistently robust.
  • Robustness in Low-Resource Languages: CRPO scores 62.17 WR in Swahili with Llama-3, using relatively well-aligned English pairs as a logical anchor for target-language alignment.The authors attribute this gain to realigning language quality and hierarchy.
  • Knowledge Utilization: CRPO exceeds the strongest baseline by over 4 MMMLU points in Indonesian and reaches 68.66 on Korean Belebele.These results extend CRPO’s gains beyond conversational evaluation to knowledge utilization and linguistic comprehension.
  • Reward and Probability Shifts: CRPO consistently produces larger positive reward differences than SFT, while SFT+DPO shifts less and CLO remains near SFT or degrades in Indonesian.Figure 2 compares reward-difference distributions across languages for Llama-3-8B models trained with SFT, CLO, SFT+DPO, and CRPO.
  • Reward and Probability Shifts: CRPO increases both chosen-response log-likelihood and reward difference across languages, unlike methods whose chosen-response likelihoods remain largely aligned with SFT.The analysis frames this as increasing the likelihood of favored responses rather than primarily suppressing rejected responses.
  • Weighting-Scheme Analysis: All three non-uniform weighting schemes outperform SFT+DPO and uniform weighting, supporting hierarchical ranking signals over pairwise comparisons.Uniform weighting causes a notable performance drop, while LambdaRank and nDCG2 perform strongly across most settings.

6 Conclusion

The paper presents CRPO as a hierarchical ranking framework for transferring preference knowledge along a language chain. It reports consistent superiority over standard pairwise approaches and improved likelihood for preferred target-language responses, while identifying wider language-pair coverage as future work.

  • Conclusion: CRPO establishes preference manifolds within and across languages by reformulating target-language alignment as a hierarchical ranking problem.The framework transfers preference knowledge along the language chain rather than relying on isolated binary comparisons.
  • Conclusion: CRPO consistently surpasses standard pairwise approaches and enhances the generative likelihood of preferred responses in the target language.The conclusion links these improvements to high-fidelity, precise target-language responses.
  • Future Work: Future work will investigate dynamic gain adaptation based on linguistic similarities and training stages to support a wider array of language pairs.The proposed direction aims to mitigate optimization complexity in expansive ranking spaces.

Limitations

CRPO incurs higher training cost because it ranks four response candidates in one step, while the evaluated benchmarks may miss language-specific and cultural capabilities.

  • Computational cost: CRPO processes four response candidates per training step, demanding more computational resources than binary-comparison methods.The paper characterizes this ranking overhead as a trade-off for jointly optimizing quality and consistency.
  • Evaluation scope: AlpacaEval, MMMLU, and Belebele measure general performance but may not fully capture language-specific characteristics, cultural contexts, or localized content.The limitation is especially relevant to evaluating cultural nuances and localized content.

Ethics Statement

The paper frames its ranking design around balancing language adaptation with response quality, while testing representative gain configurations under computational constraints.

  • Ethics and data: The paper uses publicly available resources but acknowledges that residual biases or harmful concepts in English source data may propagate across languages.It reports adherence to copyright guidelines and terms of use for original, translated, and language resources.
  • Preference hierarchy: CRPO’s hierarchy assigns higher priority to high-quality English responses than to lower-quality target-language responses, while preserving language consistency and quality optimization.The design aims to encourage desirable responses regardless of language without allowing language matching to dominate quality.
  • Gain formulation: The standard exponential gain setting can create biased local gradients, motivating manually selected gains for a more balanced optimization manifold.The paper notes that the theoretical gain space is infinite, so experiments use representative fixed configurations.
  • Empirical validation: The standard exponential setting (7, 3, 1, 0) generally yields suboptimal win rates, while the hierarchy-violating configuration (9, 5, 7, 4) performs slightly worse across most evaluated languages.The latter configuration penalizes the English winner below the target-language loser and compromises overall performance.

B Implementation Details

The experiments tune common training settings, analyze reward distributions and multilingual benchmarks, and separately assess whether target-language alignment harms English performance.

  • Training: All preference-optimization methods receive thorough hyperparameter tuning, with a learning rate of 8e-6 and two training epochs selected for stable results.The search covered learning rates [5e-7, 8e-6, 1e-5] and one or two epochs.
  • Reward analysis: Figure 4 compares chosen and rejected response reward distributions for SFT, CLO, SFT+DPO, and CRPO across languages.The figure separates favored-response rewards from rejected-response rewards.
  • Training and decoding: Training uses maximum sequence length 3072, a linear schedule with 10% warmup, α = 0.2 for CRPO and SFT+DPO, α = 0.5 for CLO, and β = 0.1 for DPO.Generation uses temperature 0.8, maintaining the training sequence-length limit.
  • Evaluation: m-ArenaHard translates Arena-Hard prompts into 22 languages, and the study reports win rates against SFT for overlapping languages.Llama-2 is excluded because some queries exceed its relatively short context length.
  • English evaluation: CRPO generally achieves the highest English win rate across target-language configurations for Llama-2-7B and Llama-3-8B, indicating reduced alignment tax.The English evaluation uses AlpacaEval and includes English in every method’s training-data composition.

E Reward Distribution Analysis

CRPO shifts reward distributions toward favored responses more strongly than the other methods, while rejected-response shifts remain comparable to SFT+DPO.

  • Chosen responses: CRPO consistently assigns higher rewards to favored responses than the other evaluated methods, whereas the alternatives show little shift relative to SFT.This comparison is shown in Figure 4(a) across languages.
  • Rejected responses: SFT+DPO and CRPO produce comparable distributional shifts for rejected responses, consistent with DPO increasing rewards for both response types.The reported reward difference for CRPO is therefore primarily associated with elevating desirable-response rewards rather than only penalizing unfavored responses.

F Comparison with Another Ranked Optimization

Under the same training budget, CRPO substantially outperforms PRO despite both using ranking-based preference optimization. The comparison attributes this gap to CRPO’s use of full ranked-list positions and ranking-aware emphasis on top-ranking preference violations.

  • Experimental comparison: CRPO substantially outperforms PRO in AlpacaEval under the same training budget.Table 7 reports results across three target languages and two backbone models.
  • Experimental comparison: PRO preserves responses in the input language but does not achieve comparable improvements in response quality.The authors interpret this as language adaptation without equivalent gains in generated-content quality.
  • Experimental comparison: CRPO improves both target-language and English AlpacaEval performance by a large margin.
  • Optimization design: PRO mainly forms pairwise comparisons, whereas CRPO uses relative positions across the ranked list and ranking-aware objectives.CRPO places greater emphasis on preference violations affecting the top of the ranking, providing a more fine-grained learning signal.
  • Optimization design: The ranking-aware objective complements NLL language adaptation by preserving and promoting high-quality responses according to their ranked positions.NLL can encourage target-language generation without necessarily improving content quality.
Loading 2608.23149v1…