PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

Zhuotong Chen, Fang Liu, Xuan Zhu, Yanjun Qi, Mohammad Ghavamzadeh

arXiv:2502.04567v1cs.AI

TL;DR

Existing PO heuristics for choosing dispreferred completions lack full theoretical justification (Tunstall et al., 2023; Lambert et al., 2024). This paper formulates PO as NLL estimation, uses contrastive-divergence sampling to select hard negatives, and reports that MC-PO beats existing baselines while OnMC-PO improves further.

  • Problem

    Existing PO methods mostly choose dispreferred completions using simple heuristics such as maximizing the human- or AI-rated score gap, without full theoretical justification (Tunstall et al., 2023; Lambert et al., 2024).

    Source: §1 · p. 1

  • Method

    The paper formulates PO as NLL minimization and estimates the normalization constant with contrastive-divergence sampling to select hard negatives.

    Source: §1 · pp. 1–2

  • Results

    In five of six studies, MC-PO outperforms existing baselines, and OnMC-PO yields further improvement in the online setting.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    Treating PO as NLL estimation connects it to normalization-constant sampling strategies for generating dispreferred completions.

    Source: §1 · p. 1

    MC-PO adds approximately 30% training time over DPO, and its single-step MCMC kernel is typically suboptimal.

    Source: §6 · p. 8

Abstract

from arXiv · show

Existing studies on preference optimization (PO) have centered on constructing pairwise preference data following simple heuristics, such as maximizing the margin between preferred and dispreferred completions based on human (or AI) ranked scores. However, none of these heuristics has a full theoretical justification. In this work, we develop a novel PO framework that provides theoretical guidance to effectively sample dispreferred completions. To achieve this, we formulate PO as minimizing the negative log-likelihood (NLL) of a probability model and propose to estimate its normalization constant via a sampling strategy. As we will demonstrate, these estimative samples can act as dispreferred completions in PO. We then select contrastive divergence (CD) as the sampling strategy, and propose a novel MC-PO algorithm that applies the Monte Carlo (MC) kernel from CD to sample hard negatives w.r.t. the parameterized reward model. Finally, we propose the OnMC-PO algorithm, an extension of MC-PO to the online setting. On popular alignment benchmarks, MC-PO outperforms existing SOTA baselines, and OnMC-PO leads to further improvement.

1. Introduction

Existing preference optimization (PO) selects dispreferred completions with heuristics such as score-gap maximization, without full theoretical justification (Tunstall et al., 2023; Lambert et al., 2024). The paper formulates PO as NLL estimation and uses sampling, including contrastive divergence, to guide dispreferred-completion selection.

1. Introduction

  • Existing PO heuristics choose dispreferred completions by maximizing their human- or AI-rated score gap from preferred completions, but lack full theoretical justification (Tunstall et al., 2023; Lambert et al., 2024).The paper asks how to choose dispreferred completions for PO.Source: §1 · p. 1
  • The framework formulates PO as NLL minimization and estimates its intractable normalization constant by sampling, whose samples can serve as dispreferred completions.This connects NLL estimation to PO and enables use of advanced normalization-constant sampling strategies.Source: §1 · pp. 1–2
  • Contrastive divergence uses MCMC to estimate the log-normalization gradient, while its MC kernel samples high-probability completions that act as hard negatives.The probability model is proportional to the target policy's log-likelihood, so sampled completions closely resemble preferred ones and challenge the model to distinguish them.Source: §1 · pp. 1–2
  • Offline MC-PO applies a CD Monte Carlo kernel to select target-policy-likelihood-proportional hard negatives, while OnMC-PO extends the framework online and uses target-policy preferred samples for unbiased normalization-gradient estimation.MC-PO uses preference datasets with multiple dispreferred completions per prompt.Source: §1 · p. 2
  • MC-PO outperforms existing SOTA baselines, and OnMC-PO leads to further improvement.The paper also reports numerical validation of sampling strategies and proves that the MC kernel leads to optimal model performance.Source: §1 · p. 2

2. Preference Optimization as NLL Estimation

This section connects preference optimization to negative log-likelihood estimation, treating dispreferred completions as samples that help estimate an otherwise intractable normalization constant. It uses this connection to motivate sampling strategies for selecting those completions.

2.1. Background: Preference Optimization

  • The KL-constrained optimum includes a prompt-specific partition function Z(x), which is intractable over the combinatorial output space.Source: §2.1 · p. 2
  • DPO reparameterizes the reward in terms of the target policy and optimizes it from pairwise preferences to distinguish preferred from dispreferred completions.Source: §2.1 · p. 2

2.2. Preference Optimization as NLL Estimation

  • The proposed formulation casts preference optimization as NLL fitting of a probability model to target-distribution observations, with a normalization constant complicating optimization.Source: §2.2 · p. 3

2.3. Preference Optimization via Sampling-Based Solution for Its NLL Estimation Formulation

  • With one noise sample, the sampling-based NLL objective is equivalent to DPO, and dispreferred completions can be interpreted as importance samples estimating the normalization constant.Source: §2.3 · pp. 3–4
  • RNCE uses a target observation and proposal-distribution noise samples; broader NLL sampling strategies offer guidance for choosing dispreferred completions.Source: §2.3 · pp. 3–4

3. Novel Preference Optimization via CD

The paper applies contrastive divergence (CD) to preference optimization, using an MCMC kernel to select high-reward hard negatives. Its online extension has an unbiased gradient estimator under the stated sampling condition.

3.1. Preference Optimization with MCMC Sampling

  • MC-PO applies a CD Monte Carlo kernel to sample hard negatives in proportion to the estimated reward model.Source: §3.1 · pp. 4–5
  • A single-step CD objective coincides with RNCE, while CD's reward-proportional sampling is intended to improve normalization-constant estimation.Source: §3.1 · p. 4
  • When a dispreferred completion's estimated reward exceeds the preferred completion's, the gradient magnitude increases, supporting a stronger policy update.Source: §3.1 · p. 4
  • MC-PO is an offline algorithm that runs the MCMC chain for one step and selects among candidate completions using policy-dependent categorical weights.Source: §3.1 · p. 5

3.2. MCMC for Online Preference Optimization

  • When the online true observation is sampled from the probability model, OnMC-PO's CD estimate of the log-normalization gradient is unbiased.Source: §3.2 · p. 5

4. Related Works

The related-work section distinguishes preference-alignment methods by their objectives and positions the paper's NLL-estimation perspective as guidance for selecting dispreferred completions.

4. Related Works

  • Existing direct preference optimization variants are categorized as contrastive-based or classification-based according to their objectives.Source: §4 · p. 5
  • Examples include IPO's squared loss, SimPO's length regularization, RPO's supervised regularization, and classification-based methods KTO, BCO, and NCA.Source: §4 · p. 5
  • The paper interprets dispreferred completions as estimative samples for the normalization constant, then develops MC-PO to select high-quality negatives with an MCMC kernel.Source: §4 · p. 5

5. Numerical Experiments

Experiments evaluate MC-PO and OnMC-PO on open-ended instruction-following benchmarks and examine how sampling strategy, candidate quality, and training setup affect performance. MC-PO generally outperforms offline baselines, while OnMC-PO improves further in the reported online setting.

5.1. Experimental Setup

  • Evaluations use AlpacaEval 2 and Arena-Hard winrates judged by Mistral-Large-Instruct-2407, across base, instruct, and batched-online training setups.Base models train on Nectar; the instruct setup regenerates completions from UltraFeedback prompts, while the batched-online setup refreshes preference data across three training batches.Source: §5.1 · p. 6

5.2. Main Results: Comparing with SOTA PO

  • MC-PO outperforms existing baselines in five of six studies, and OnMC-PO reaches 72.63% winrate on Alpaca with Llama-3.1-8B-Instruct, outperforming baselines.In the base setup, MC-PO beats DPO by 4.5% on Alpaca-Eval and 9% on Arena with Mistral-7B-SFT; Llama-3.1-8B-SFT reaches 35.84% on AlpacaEval and 63.77% on Arena.Source: §5.2 · p. 6, Table 1 · p. 7

5.3. Analysis of Sampling Strategies in MC-PO

  • The MCMC-kernel sampler balances categorical-distribution exploitation with alternative-candidate exploration, which the authors connect to accurate normalization-gradient estimation and improved performance.The paper identifies this sampling strategy as the best-performing option among those examined.Source: §5.3 · p. 7
  • Min sampling yields lower performance and higher variance, while MC-PO and Max perform best with five high-quality candidates; performance declines as candidate quality falls.With only one candidate per prompt, all three sampling strategies are equivalent.Source: §5.3 · p. 7, Figure 2 · p. 8

5.4. Data and Ablation Analysis of MC-PO

  • With injected noisy candidates, MC-PO samples using candidate log-probabilities and selects semantically hard negatives rather than negatives that are merely syntactically similar.The noise sample is created by randomly switching two tokens in the preferred completion.Source: §5.4 · pp. 7–8, Table 2 · p. 8
  • Increasing the number of dispreferred samples consistently improves MC-PO, whereas random-sampling RNCE shows no notable improvement.The comparison examines models trained with multiple dispreferred samples.Source: §5.4 · p. 8, Table 3 · p. 8
  • Pairing each preferred completion with every dispreferred completion increases training time fourfold and yields lower reported winrates than MC-PO on both benchmarks.The augmented-data alternative reports 34.18% on Alpaca-Eval and 59.62% on Arena; MC-PO reports 35.84% and 63.77%, respectively.Source: §5.4 · p. 8
  • In the batched-online Llama-3.1-8B-Instruct comparison, OnMC-PO reaches 77.71% winrate on Arena versus online DPO’s 73.28%; online DPO achieves 72.63% on Alpaca-eval.Source: §5.4 · p. 8

6. Conclusion, Limitations and Future Work.

The paper frames alignment as NLL estimation and proposes sampling-based preference optimization. Its CD-based MC-PO adds training cost, and the authors identify single-step MCMC as a typically suboptimal choice.

6. Conclusion, Limitations and Future Work.

  • CD-based MC-PO adds approximately 30% training time compared with DPO because of the MCMC-kernel computations.The authors propose more efficient training algorithms as future work.Source: §6 · p. 8
  • MC-PO runs its MCMC kernel for only one step, which the authors describe as typically suboptimal for acquiring high-quality samples.They state that multiple chain steps are essential for high-quality samples and would add further computational overhead.Source: §6 · p. 8
  • The paper formulates alignment as NLL estimation and proposes sampling-based solutions for preference optimization.The authors identify multi-step MCMC-based PO as a future research direction.Source: §6 · p. 8

A. Theoretical Derivations

This theoretical background presents RLHF’s KL-constrained reward-maximization formulation and explains how DPO derives a preference-data objective. The normalization partition function is difficult to compute over the large completion space.

A. Theoretical Derivations

  • The KL constraint limits deviation from the reference policy and is described as helping prevent reward-model overfitting, preserve generation diversity, and avoid mode collapse.The reward model is estimated from observed data and may not accurately represent real-world human preference.Source: §A.1 · p. 11
  • The partition function normalizes the probability, but its exact computation is often infeasible because the completion space is combinatorially large..Source: §A.1 · p. 11
  • DPO derives its objective by expressing the reward through the optimal policy, reference policy, and partition function, then substituting into the Bradley-Terry preference model.The resulting objective is optimized over preference data containing preferred and dispreferred completions.Source: §A.1 · p. 11

A.2. Proof of Proposition 2.1

The proof frames RNCE as a multi-class identification problem: among one target observation and proposal-distributed noise samples, the model should identify the target index. It assumes equal prior probability for each index.

A.2. Proof of Proposition 2.1

  • RNCE treats one target observation and multiple proposal-distributed noise samples as a multi-class identification problem.The target observation is the one sampled from π∗.Source: §A.2 · p. 12
  • The index z marks the target observation among the samples, with each possible index assigned equal prior probability.The proof sets P(z = i) = 1/(M + 1).Source: §A.2 · p. 12
  • The probability model is trained to maximize the posterior probability that z identifies the target observation.Source: §A.2 · p. 12

A.3. Proof of Lemma 3.1

The lemma connects one-step contrastive divergence to RNCE and interprets the sampling step as importance sampling for estimating the log-normalization gradient.

A.3. Proof of Lemma 3.1

  • With a single MCMC step, CD and RNCE share the same objective function.Source: §A.3 · p. 13
  • The sampling-based estimate targets the intractable gradient of the log-normalization constant.Source: §A.3 · p. 13
  • CD uses importance sampling from proposal distribution μ to estimate that gradient.Source: §A.3 · p. 13

A.4. Proof of Lemma 3.2

The proof specializes the sampling-based objective to one noise sample and identifies the resulting pair as preferred and dispreferred completions.

A.4. Proof of Lemma 3.2

  • For M = 1, the proof derives the gradient of the sampling-based objective.Source: §A.4 · p. 13
  • In this one-noise-sample case, y0 and y1 are the preferred and dispreferred completions, respectively.Source: §A.4 · p. 13
  • The derivation uses the sigmoid identity 1 − σ(x) = σ(−x).Source: §A.4 · p. 13

A.5. Proof of Proposition 3.3

The proposition establishes unbiasedness for the estimated log-normalization gradient when the initial sample comes from pθ. The marginal-distribution derivation assigns equal prior probability to the index z.

Then the marginal distribution

A.5. Proof of Proposition 3.3

B.1. Experimental Setup

The experiments use base, instruct, and batched-online setups, with preference data generated from ranked completions or refreshed model outputs. They evaluate preference-optimized models with judged winrates and specify training and baseline settings.

B.1. Experimental Setup

  • The base setup fine-tunes Llama-3.1-8B on Nectar, treating rank-1 as preferred, while the instruct setup initializes Llama-3.1-8B-Instruct and regenerates completions from UltraFeedback prompts.Nectar prompts are described as having 5 ranked completions after rank-2 is removed; a dispreferred completion is randomly selected from the remaining 5 candidates. Instruct generates two completions at each temperature—0.6, 0.8, and 1.0—then uses iterative pairwise ranking by Llama-3.1-70B-Instruct.Source: §B.1 · p. 16
  • The batched-online setup divides training into three segments and regenerates preference data from the model checkpoint at each segment’s start.It is presented as a middle ground between offline and purely online training, and as more efficient than a purely online approach because online inference-kernel startup can be costly.Source: §B.1 · p. 16
  • The MC kernel uses the reference policy as its proposal in the instruct setup and the Nectar completion-generating LLMs in the base setup.The listed base proposals include GPT-4, GPT-3.5-turbo, GPT-3.5-turbo-instruct, Llama-2-7B-chat, and Mistral-7B-Instruct.Source: §B.1 · p. 16
  • Training uses full-parameter tuning, effective batch size 128, and two epochs; final-checkpoint results average three seeds, and Mistral-Large-Instruct-2407 judges winrates.Hyperparameter optimization tests seven learning rates; the evaluation prompt asks the judge to compare correctness, helpfulness, completeness, and clarity without treating answer length as a criterion.Source: §B.1 · p. 17
  • MC-PO improves substantially from epoch 1 to 2, while extending training to epoch 3 adds no further performance improvement.Table 4 reports MC-PO performance by training epoch.Source: §B.1 · p. 17, Table 4 · p. 17

B.2. Details on Baseline Preference Optimization Algorithms

  • Baseline settings use β=0.01 where applicable, λ=0.1 for supervised next-word prediction, and γ=10 in SimPO and CPO.Table 5 summarizes baseline objective implementations and identifies λ as the RPO/CPO supervised-loss regularization parameter and γ as a SimPO hyperparameter.Source: §B.2 · p. 17, Table 5 · p. 18
Loading 2502.04567v1…