PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, Young Jin Kim

arXiv:2401.08417v4cs.CL

TL;DR

Smaller 7B–13B LLMs still trail conventional translation models, so the paper introduces Contrastive Preference Optimization to move beyond reference-mimicking fine-tuning. Applied to ALMA, CPO produces ALMA-R, which matches or surpasses GPT-4 and WMT competition winners on reported translation benchmarks.

  • Problem

    Smaller 7B–13B LLMs still underperform conventional machine translation models, limiting their translation performance relative to larger LLMs.

  • Method

    Contrastive Preference Optimization trains ALMA with preference data to avoid near-perfect but flawed translations and learn superior ones beyond supervised fine-tuning.

  • Results

    ALMA-13B-R matches or surpasses GPT-4 and WMT competition winners, scoring 85.74 KIWI-XXL and 94.05 XCOMET for en→xx translations.

  • Takeaways & Limitations

    CPO pushes beyond the performance bottleneck of supervised fine-tuning for moderate-sized LLM translation models.

  • Takeaways & Limitations

    Comparisons on WMT’22 exclude TowerInstruct because it used WMT’22 test data for training.

Abstract

from arXiv · show

Moderate-sized large language models (LLMs) -- those with 7B or 13B parameters -- exhibit promising machine translation (MT) performance. However, even the top-performing 13B LLM-based translation models, like ALMA, does not match the performance of state-of-the-art conventional encoder-decoder translation models or larger-scale LLMs such as GPT-4. In this study, we bridge this performance gap. We first assess the shortcomings of supervised fine-tuning for LLMs in the MT task, emphasizing the quality issues present in the reference data, despite being human-generated. Then, in contrast to SFT which mimics reference translations, we introduce Contrastive Preference Optimization (CPO), a novel approach that trains models to avoid generating adequate but not perfect translations. Applying CPO to ALMA models with only 22K parallel sentences and 12M parameters yields significant improvements. The resulting model, called ALMA-R, can match or exceed the performance of the WMT competition winners and GPT-4 on WMT'21, WMT'22 and WMT'23 test datasets.

1. Introduction

Decoder-only LLMs show strong translation ability, but smaller models face limited linguistic diversity and modest improvements. The paper identifies reference-quality and SFT limitations, then introduces CPO and a preference dataset to improve translation quality beyond reference imitation.

  • Larger LLMs such as GPT-3.5 (175B) and GPT-4 exhibit strong translation abilities, motivating MT with decoder-only LLMs.
  • Smaller LLM translation improvements remain relatively modest, primarily because pre-training is English-centric and linguistically limited.
  • CPO addresses SFT’s reference-quality ceiling and lack of safeguards against near-perfect but flawed translations, breaking the bottleneck of reference-mimicking training.SFT can cap performance at training-data quality and does not prevent occasional minor errors such as omitted translation parts.
  • Human-written parallel data can be inferior to strong system-generated translations, making exclusive reference replication and reference-based evaluation potentially flawed.
  • The authors build and release a high-quality preference dataset for machine translation.

2. Gold or Gilded? Scrutinizing Gold Reference Quality

Gold references are central to MT training and reference-based evaluation, yet human-written references can be imperfect and sometimes surpassed by advanced model translations. The section therefore compares references with model outputs using reference-free evaluation to motivate learning from superior translations.

  • Why Reference Quality Matters: Reference quality directly affects both model training and reference-based evaluation, including BLEU and COMET-22.Training losses minimize differences from gold references, while substandard references compromise evaluation precision.
  • Gold References Can Be Surpassed: Human-written gold references may omit information that advanced translation models provide, as illustrated by the incomplete expansion of “CEP.”Figure 2 shows model translations adding the abbreviation’s full name where the reference retains only “CEP.”
  • Evaluation Setup: Reference-free evaluation compares FLORES-200 references with ALMA-13B-LoRA and GPT-4 outputs across five English-centric language pairs.The comparison uses 2009 development and test samples per language direction and two 10B-parameter evaluators, KIWI-XXL and XCOMET-XXL.
  • Motivation for Rejection Learning: Advanced translation outputs can sometimes surpass gold references, motivating methods that teach models to reject adequate but imperfect translations rather than merely imitate references.Table 1 reports average evaluator performance and Win Ratio for gold references, ALMA-13B-LoRA outputs, and GPT-4 outputs.

3. Contrastive Preference Optimization

CPO addresses scarce machine-translation preference data by constructing scored translation triplets and replacing DPO’s inefficient reference-policy computation with a uniform-prior approximation. Its loss combines preference learning with a preferred-data behavior-cloning term.

  • Preference data construction: CPO constructs preference data from FLORES-200 development and test sets, using 2009 parallel sentences per language pair.The dataset covers the same language pairs discussed in Section 2.
  • Preference data construction: For each source, GPT-4, ALMA-13B-LoRA, and the original reference form a translation triplet scored by KIWI-XXL and XCOMET.The resulting scores are used to compare the three translation outputs.
  • Preference data construction: The highest-scoring triplet translation is preferred, the lowest-scoring one is dis-preferred, and the middle-scoring one is discarded.This selection rule is based on reference-free model scores.
  • CPO objective: CPO removes DPO’s need to store and sequentially execute both policy and reference models by setting the reference model to a uniform prior.The reference-policy terms cancel, leaving no additional computation or storage beyond the policy model.
  • CPO objective: The CPO loss contains one preference-learning term and one negative-log-likelihood term from a behavior-cloning regularizer on preferred data.The regularizer is equivalent to adding an SFT term on preferred examples.

4. Experiments

Experiments evaluate ALMA-13B-R against strong translation baselines across ten directions on WMT’21–WMT’23, using compact preference training and reference-free metrics. CPO consistently improves performance, matching or surpassing GPT-4, WMT winners, and competing moderate-sized LLM translators.

  • Experimental setup: Preference training uses 2K × 10 dir from FLORES-200 across ten translation directions.The directions are cs↔en, de↔en, is↔en, zh↔en, and ru↔en.
  • Evaluation: Evaluation focuses on WMT’21 for Icelandic and WMT’22 for other languages, with auxiliary WMT’23 tests covering six directions.The WMT’23 directions are de↔en, zh↔en, and ru↔en.
  • Experimental setup: ALMA-13B-R updates only rank-16 LoRA weights, adding 12M parameters to the 13B model, with β=0.1.Training is many-to-many multilingual translation initialized from ALMA-13B-LoRA.
  • Results: ALMA-13B-R achieves average scores of 85.74 on KIWI-XXL and 94.05 on XCOMET for en→xx translations.These results bring ALMA-13B-R to a level comparable to or above GPT-4 and WMT competition winners.
  • Training-objective comparison: SFT marginally improves xx→en but slightly worsens en→xx, while DPO slightly decreases performance and CPO significantly improves all translation directions.All objectives use ALMA-13B-LoRA as the base model.
  • Results: Across six WMT’23 directions, ALMA-13B-R surpasses ALMA-13B-LoRA and TowerInstruct and matches or exceeds WMT winners.The WMT’23 results are reported as averages across all six directions.

5. Analyses

The analyses find that CPO’s gains are not substantially tied to the metric used to select preference data and are supported by human evaluation. Ablations further show that both loss components and high-quality dis-preferred data are important for translation performance.

  • Metric and Method Bias: Using KIWI-XXL, XCOMET, or their ensemble to select preference data produces minimal final performance disparities across metrics.The study retrains ALMA-13B-LoRA with CPO under each preference-data construction.
  • Metric and Method Bias: DPO or SFT on metric-preferred data can lower performance on the metric used for selection, making metric-alignment gains difficult to obtain.This supports the interpretation that CPO improvements are not simply translations optimized to match evaluator preferences.
  • Human Evaluation: Human evaluation of 400 zh→en WMT’22 samples shows ALMA-13-R outperforming ALMA-13B-LoRA, supporting reference-free models for preference-data construction and evaluation.Four bilingual speakers rated randomized translations on a 0–6 scale; Table 7 reports mean scores, ranks, and win ratios.
  • CPO Ablations: Removing either Lprefer or LNLL reduces the benefit relative to using both CPO loss components.Training only with LNLL is equivalent to SFT on preferred data.
  • Preference Data Ablations: Excluding ALMA data harms en→xx performance, whereas excluding GPT-4 data harms xx→en performance; manually noised dis-preferred data substantially lowers all three metrics in both directions.The noised data uses random word deletions with probability 0.15 and swaps within range 1 with probability 0.3.

6. Conclusion … Appendix H Evaluation on Non-Comet Metrics

The study identifies weaknesses in gold-reference quality and reference-based evaluation, then introduces CPO and ALMA-13B-R as a stronger translation approach. The appendices provide additional results, theoretical analysis, data details, winner information, WMT’23 experiments, and non-Comet metric evaluations.

  • 6. Conclusion: CPO addresses quality issues in gold references that challenge both SFT training and reference-based MT evaluation.The method is presented as a more efficient variant of DPO.
  • 6. Conclusion: ALMA-13B-R is a moderate-size LLM translation model that matches or sometimes surpasses GPT-4 and WMT competition winners.The conclusion describes this as a significant advancement in machine translation.
  • Appendix Contents: The appendices include comprehensive WMT’21 and WMT’22 results.These results are listed under Appendix A.
  • Appendix Sections Contents: The paper includes a theoretical treatment of the CPO loss.This material appears in Appendix C.
  • Appendix B Prompts Used for Translations: The appendices document prompts used for translations and the details and influence of human-labeled data.The human-labeled-data discussion appears in Appendix D.
  • Appendix E Information of WMT winners: Additional appendices report information about WMT winners and estimated accuracy with human agreements.Estimated accuracy with human agreements is listed in Appendix F.
  • Appendix H Evaluation on Non-Comet Metrics: The paper also includes experimental results on WMT’23 and evaluates translation quality with non-Comet metrics.Appendix I examines the effectiveness of the BC regularizer for DPO.

A. Comprehensive Results of WMT’21 and WMT’22 · B. Prompts for Translations · C. Theory

The paper reports comprehensive WMT’21–WMT’22 comparisons showing ALMA-13B-R performs comparably to GPT-4 and WMT winners while surpassing several leading translation systems. It also finds reference-free metrics more consistent than BLEU and specifies the prompts used for GPT-4 and ALMA translation evaluation.

  • A. Comprehensive Results of WMT’21 and WMT’22: CPO significantly enhances ALMA-7B-LoRA performance when trained with the same preference data as ALMA-13B-R.The study extends the ALMA-13B-R training methodology to the 7B model size.
  • A. Comprehensive Results of WMT’21 and WMT’22: ALMA-13B-R demonstrates performance comparable to GPT-4 and WMT winners on comprehensive WMT’21–WMT’22 evaluations.The comparisons include recently released LLM-based translators and advanced conventional translation models.
  • A. Comprehensive Results of WMT’21 and WMT’22: ALMA-13B-R surpasses Google Translate, NLLB, MADLAD-10B, and GPT-3.5 in many cases.These comparisons cover leading commercial, multilingual, and general-purpose translation systems.
  • A. Comprehensive Results of WMT’21 and WMT’22: BLEU often diverges from neural-based and reference-free metrics, so the paper argues against relying on BLEU alone.WMT winners can score highly on BLEU without receiving corroborating support from reference-free models.
  • A. Comprehensive Results of WMT’21 and WMT’22: 87.74 (Ours) vs. 87.68 (GPT-4) on COMET-22 for en→xx demonstrates comparable performance under a reference-free-oriented evaluation perspective.The passage states that COMET-22 is more consistent with reference-free metrics and more robust than BLEU.
  • A. Comprehensive Results of WMT’21 and WMT’22: Tables 9 and 10 provide full en→xx and xx→en WMT’21–WMT’22 results using both reference-free and reference-based metrics.The tables compare systems including WMT winners, GPT models, commercial translators, and multilingual translation models.
  • B. Prompts for Translations: The evaluation uses the prompt format from Hendy et al. for GPT-4 and the prompt from Xu et al. for ALMA models.The prompts are depicted in Figure 5.
  • B. Prompts for Translations: The translation prompt instructs the system to output only the result and specifies source and target languages around the source sentence.Its system message begins, “You are a helpful translator and only output the result.”

C.1. Proof of The Upper Boundary · C.2. BC Regularizer Simplification

The appendix proves that DPO with an ideal preferred-data reference has an additive-constant upper bound by the uniform-reference loss. It then simplifies CPO’s constrained objective into a final loss by applying Lagrangian duality, setting λ to 1, and expanding the KL divergence.

  • C.1. Proof of The Upper Boundary: C.1: When πref = πw, L(πθ; πw) + C is upper bounded by L(πθ; U), with C constant.πw is defined as an ideal policy precisely aligned with the true preferred-data distribution.
  • C.1. Proof of The Upper Boundary: C.1: For each preference example, πw(yw|x) = 1 and 0 ≤ πw(yl|x) ≤ 1, so preferred predictions need no reference-model reweighting.Under these conditions, the DPO loss can be reformulated using the ideal reference policy.
  • C.1. Proof of The Upper Boundary: C.1: Expanding the sigmoid function yields the reformulated loss used in the upper-bound derivation.The proof proceeds by transforming the DPO objective after substituting the ideal reference-policy conditions.
  • C.1. Proof of The Upper Boundary: C.1: Because πw is fixed, log πw(yl|x)^β contributes no gradients, so omitting it preserves the optimization objective.The resulting objective is denoted L′(πθ; πw), and the omitted term is identified with the theorem’s constant C.
  • C.1. Proof of The Upper Boundary: C.1: Since 0 ≤ πw(yl|x) ≤ 1, the reformulated loss can be upper bounded, establishing the theorem’s inequality.The proof concludes that L(πθ; πw) + C is upper bounded by L(πθ; U), with C defined from the data expectation.
  • C.2. BC Regularizer Simplification: C.2: CPO minimizes L(πθ; U) while constraining the difference between the preferred-data distribution and the learnable policy’s outputs.This is the original constrained formulation of contrastive preference optimization.
  • C.2. BC Regularizer Simplification: C.2: Lagrangian duality converts the constrained CPO objective into an equivalent unconstrained objective with hyperparameter λ set to 1.The resulting optimization is then expanded through the KL divergence.
  • C.2. BC Regularizer Simplification: C.2: Expanding the KL divergence produces the final formulation of the CPO loss function.The KL expansion completes the simplification from the constrained objective to the operational loss.

D. Details And Influence of Human-Labeled Preference Data … E.2. Systems For WMT’23

The paper finds that human-labeled preference data has limited influence on translation performance, likely because many translations are tied and human evaluation may be biased. It also documents how WMT’21–’23 winner systems were selected for comparison.

  • D. Details And Influence of Human-Labeled Preference Data: Human-labeled preference data has relatively minimal impact, probably because of tied translations and potential human bias.
  • D.1. Data Construction Details: 2K additional pair-wise sentences cover only en→de and en→zh, with human evaluators choosing between Google Translate and GPT-4 outputs.English sources came from Wikipedia and were filtered to remove timestamps and URLs.
  • D.2. Influence on Performance: The comparison fine-tuned ALMA-13B-LoRA models on triplet data alone versus triplet plus human-labeled data.Because the added data covers only de and zh, changes were expected primarily for translations involving those languages.
  • D.2. Influence on Performance: Human-labeled data produced minimal en→zh gains, a slight en→de decline, no substantial en→xx difference, and a minor xx→en decrease.
  • E.1. Systems For WMT’21 And WMT’22: The WMT’21 and WMT’22 competition winners were taken from the systems used by Hendy et al. (2023).The paper directs readers to Hendy et al. (2023) for further details.
  • E.2. Systems For WMT’23: For WMT’23, systems for de↔en and zh↔en were selected by highest human rankings using DA+SQM, while de↔ru used the highest reported COMET-22 system.The selected systems are listed in Table 14.

F. Estimated Accuracy with Human Agreements

The paper replaces static metric-improvement thresholds with dynamic thresholds calibrated to human-perceived distinctions between systems. At an 80% human-concordance rate, the required improvement margin is at least 1.24 for the cited metric.

  • Threshold calibration: Dynamic thresholds are calibrated to the minimal metric difference humans perceive as distinguishing two systems.This replaces specifying a fixed improvement threshold for each metric.
  • Human agreement: ≥1.24 improvement aligns with an 80% concordance rate between estimated metric differences and human judgments.The passage presents this as an example of the human-calibrated threshold approach.
  • Metric estimates: Table 15 reports thresholds and estimated accuracies for each metric used in the paper.The supplied passage provides the table’s purpose but not its metric-specific entries.

G. Full Results of WMT’23

Table 16 presents the comprehensive WMT’23 results, where ALMA-13B-R performs best on average among state-of-the-art translation models. The table also marks strong improvements over original ALMA using estimated human-judgment accuracy.

  • Overall performance: ALMA-13B-R performs best on average among the state-of-the-art translation models in WMT’23.This performance is consistent with its results in WMT’21 and WMT’22.
  • Overall performance: Table 16 reports the comprehensive results of WMT’23 across all evaluated systems.The highest score among all systems is shown in bold.
  • Result annotations: Dark blue boxes indicate improvements over original ALMA achieving at least 80% estimated accuracy with human judgment.Lesser improvements are highlighted in shallow blue boxes.

H. Evaluation on ALMA-R with Non-Comet Metric · I. The Effectiveness of The BC Regularizer for DPO

The paper validates ALMA-R with the non-COMET BLEURT-20 metric and examines whether adding a BC regularizer to DPO improves translation performance. The analyses compare ALMA-13B-LoRA with ALMA-13B-R and evaluate the effect of adding LNLL to the original DPO objective.

  • H. Evaluation on ALMA-R with Non-Comet Metric: BLEURT-20 provides a non-COMET, neural, reference-based evaluation of ALMA-13B-LoRA and ALMA-13B-R.The metric is introduced to address concerns that COMET models may be highly correlated because of similar training procedures.
  • H. Evaluation on ALMA-R with Non-Comet Metric: The non-COMET evaluation tests whether ALMA-R’s gains remain evident under a metric different from COMET.This analysis is motivated by concerns about correlation among COMET models.
  • H. Evaluation on ALMA-R with Non-Comet Metric: Table 17 reports the BLEURT-20 score comparison between ALMA-13B-LoRA and ALMA-13B-R.The preference data used for training was constructed with COMET-based evaluations.
  • I. The Effectiveness of The BC Regularizer for DPO: DPO loss LDPO = L(πθ, πref) can be augmented with an additional BC regularizer.The paper presents this regularizer as an addition to the original DPO formulation.
  • I. The Effectiveness of The BC Regularizer for DPO: Adding LNLL to the DPO objective produces notable translation improvements in both directions involving English.The observation is reported from Table 18 and concerns translations to and from English.
  • I. The Effectiveness of The BC Regularizer for DPO: The results suggest that original DPO lacks the BC regularizer that steers the model toward the preferred data distribution.Combining DPO with the BC regularizer could approach CPO performance but incurs additional cost.
Loading 2401.08417v4…