PaperlayerView exact revision on arXiv

Source-linked AI summary

Noise Contrastive Alignment of Language Models with Explicit Rewards

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, Jun Zhu

arXiv:2402.05369v3cs.LGcs.CL

TL;DR

DPO is mainly designed for pairwise preferences, while scalar-reward datasets with multiple responses cannot be directly used and may lose information during preprocessing. The paper introduces NCA and InfoNCA for reward and preference alignment; its experiments report gains over preference baselines with reward data and NCA gains over DPO on complex reasoning tasks.

  • Problem

    DPO is tailored to pairwise preference data, whereas scalar-reward datasets with multiple responses cannot be directly used and preprocessing can discard reward values and responses.

    Source: §1 · pp. 1–2

  • Method

    InfoNCA and NCA use contrastive-learning frameworks to extract LM policies from reward or preference data, with InfoNCA including DPO as a pairwise special case.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Reward-based InfoNCA/NCA outperform preference baselines, and pairwise NCA outperforms DPO on complex reasoning tasks such as math and coding.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The reported results support using suboptimal responses in reward datasets and identify NCA as a preference-setting alternative that prevents likelihood decline.

    Source: §1 · p. 2, §7 · p. 10

    The reward-alignment setup assumes one optimal response sampled from the optimal policy and remaining responses sampled independently from the pretrained LM.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

User intentions are typically formalized as evaluation rewards to be maximized when fine-tuning language models (LMs). Existing alignment methods, such as Direct Preference Optimization (DPO), are mainly tailored for pairwise preference data where rewards are implicitly defined rather than explicitly given. In this paper, we introduce a general framework for LM alignment, leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling reward datasets explicitly annotated with scalar evaluations. Our framework comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct extraction of an LM policy from reward data as well as preference data. Notably, we show that the DPO loss is a special case of our proposed InfoNCA objective under pairwise preference settings, thereby integrating and extending current alignment theories. By comparing NCA and InfoNCA, we demonstrate that the well-observed decreasing-likelihood trend of DPO/InfoNCA is caused by their focus on adjusting relative likelihood across different responses. In contrast, NCA optimizes the absolute likelihood for each response, thereby effectively preventing the chosen likelihood from decreasing. We evaluate our methods in both reward and preference settings with Mistral-8*7B and 7B models. Experiments suggest that InfoNCA/NCA surpasses various preference baselines when reward datasets are available. We also find NCA significantly outperforms DPO in complex reasoning tasks like math and coding.

1 Introduction

DPO is tailored to pairwise preferences, leaving scalar-reward datasets with multiple responses difficult to use without discarding reward values and responses. The paper introduces InfoNCA and NCA as contrastive alignment methods for reward and preference data, with NCA addressing the likelihood decline associated with relative-response adjustment.

1 Introduction

  • DPO is designed for pairwise preference labels, while scalar reward datasets can retain multiple responses and their individual reward values.Selecting the best response and pairing it with a random alternative discards other responses and reward values.Source: §1 · pp. 1–2
  • InfoNCA directly extracts policies from reward or preference datasets with arbitrary response counts and includes DPO as a special case for pairwise preferences.The paper frames InfoNCA as a multi-category extension of DPO’s binary classification loss.Source: §1 · p. 2
  • NCA targets absolute response likelihood, mitigating the decline associated with DPO and InfoNCA adjusting relative likelihoods across responses.The introduction reports that pairwise NCA prevents chosen likelihood from decreasing.Source: §1 · p. 2, Figure 2 · p. 2
  • Reward-based alignment with InfoNCA/NCA outperforms preference baselines, while pairwise NCA outperforms DPO on complex reasoning tasks such as math and coding.Reward-based methods achieve higher GPT-4 evaluation rewards; the paper attributes their improvement to using additional suboptimal responses.Source: §1 · p. 2

2 Background: Direct Preference Optimization

The background formulates LM alignment as constrained policy optimization using a pretrained model, instructions, responses, rewards, and a temperature coefficient. DPO learns from pairwise preferences by turning a reward model into a likelihood-ratio parameterization and optimizing a binary classification loss.

2 Background: Direct Preference Optimization

  • LM alignment is framed as constrained policy optimization, with the pretrained LM as reference and a reward function reflecting human intentions.The formulation identifies instructions and responses as x and y, and α as a temperature coefficient.Source: §2 · p. 2
  • DPO assumes pairwise preference data and models annotator preference probabilities with a learnable implicit reward under Bradley-Terry theory.The preference data label a preferred response over a dispreferred response for each instruction.Source: §2 · p. 3
  • DPO parameterizes the implicit reward as the log-likelihood ratio between the policy and pretrained LM, enabling policy extraction through binary classification.When the learned reward matches the true reward and β equals α, the passage states that the learned policy equals the optimal policy.Source: §2 · p. 3

3 InfoNCA: Extending DPO from Preference to Explicit Rewards

InfoNCA extends alignment from pairwise preferences to reward datasets with arbitrary numbers of responses. It formulates reward alignment as classification and recovers DPO in the pairwise, low-temperature limit.

3 InfoNCA: Extending DPO from Preference to Explicit Rewards

  • InfoNCA handles reward and preference datasets with arbitrary response counts, addressing DPO’s limitation to pairwise preference data.Reward annotations also allow K responses to be evaluated with K ratings rather than C2 K pairwise comparisons.Source: §3 · p. 3

3.1 Reward Alignment through Multi-Class Classification

  • For K candidates, one optimal response is sampled from π* and K−1 noise responses from μ; the task is to identify the optimal candidate from reward labels.The optimal response is sampled according to π*(y|x) ∝ μ(y|x)e^{r(x,y)/α}.Source: §3.1 · p. 3
  • InfoNCA uses a K-category cross-entropy loss, with reward-softmax labels and predictions from learned rewards rθ.The loss reaches zero when rθ*(x, y) = r(x, y)/α + C(x).Source: §3.1 · p. 4

3.2 InfoNCA Subsumes DPO as A Special Case

  • InfoNCA recovers the DPO objective when K = 2 and α → 0.The paper characterizes this as InfoNCA’s pairwise-preference special case.Source: §3.2 · p. 4
  • Larger K can improve performance, while α trades off diversity and optimality; as α → 0, InfoNCA increases likelihood for the optimal response and decreases it for others.At this limit, reward data becomes preference data.Source: §3.2 · p. 5

4 NCA: Fixing Decreased Response Likelihood Issue for InfoNCA

NCA is proposed to address decreasing response likelihood by learning absolute rather than relative rewards. Its single-response classification formulation predicts model source, while its absolute-reward focus distinguishes it from InfoNCA and DPO.

4 NCA: Fixing Decreased Response Likelihood Issue for InfoNCA

  • NCA focuses on absolute rewards, counteracting the likelihood-decrease trend shared by InfoNCA and DPO, which adjust relative rewards across responses.The paper states that NCA and InfoNCA can converge to the optimal LM policy under ideal conditions.Source: §4 · p. 5

4.1 Reward Alignment through Absolute Reward Prediction

  • NCA constructs a binary classification task for a single response, asking whether it came from the optimal LM or the pretrained LM.The response is sampled from one of those two sources, and the task uses its reward to predict the source.Source: §4.1 · p. 5
  • NCA’s loss has opposing terms: the first tends to increase rθ(x, y), while the second tends to decrease it.At training start, the direction of the combined force depends on the difference in their weights.Source: §4.1 · p. 6

4.2 Connection between NCA and InfoNCA/DPO

  • Unlike InfoNCA and DPO’s relative reward calibration, NCA optimizes absolute reward values and effectively prevents preferred-response likelihood from decreasing, particularly helping math and coding tasks.Its policy definition requires Eμ(y|x)e^{rθ(x,y)} = 1.Source: §4.2 · p. 6

5 Experiments

The experiments compare reward-based alignment on UltraFeedback with preference-based alignment on UltraInteract, then examine when NCA differs from DPO. Results favor using all reward data and show NCA’s advantages on reasoning tasks and in preserving chosen-response likelihood.

5.1 Aligning Language Models with Explicit Rewards

  • InfoNCA and NCA outperform the DPO baseline when fine-tuned on UltraFeedback, exploiting all information in the reward dataset.The comparison uses a Mistral-7B model; Table 2 reports the results.Source: §5.1 · p. 7, Table 2 · p. 7
  • Increasing responses from K = 2 to K = 4 consistently improves InfoNCA and NCA performance across various hyperparameters.Figure 4 varies the number of suboptimal responses while fixing the highest-reward response.Source: Figure 4 · p. 7, §5.1 · p. 8
  • Combinatorial DPO underperforms InfoNCA and NCA; DPO×3 helps somewhat, while DPO×C2 4 performs worse than simple pruning.The authors attribute the difference to convergence guarantees for InfoNCA and NCA that combinatorial preference methods lack.Source: §5.1 · p. 8

5.2 NCA vs. DPO in Aligning Language Models with Pairwise Preference

  • Across benchmarks, NCA consistently outperforms DPO, while DPO hurts performance on most reasoning tasks for Mixtral-8×7B-SFT.NCA improves performance on both the 7B and 8×7B models; Table 3 presents UltraInteract results and marks decreases after training.Source: §5.2 · p. 8, Table 3 · p. 8
  • Preferred-response likelihood decreases after DPO training but increases with NCA, a pattern reported across preference and reward learning.NCA adjusts absolute data likelihood, whereas DPO/InfoNCA adjust relative likelihood across responses.Source: §5.2 · pp. 8–9, Figure 5 · p. 8
  • NCA is more suitable for reasoning tasks with sparse high-quality answers, while DPO may suit instruction-following tasks without dataset “golden” answers.The authors characterize NCA as benefiting from dataset regularization and DPO as relying more on LLM generalization.Source: §5.2 · p. 9
  • NCA tolerates greater divergence from the initial SFT policy and is more robust to hyperparameter changes than DPO.The reported grid search finds DPO can fail to improve alignment or collapse under some hyperparameters; NCA does not show similar issues.Source: §5.2 · p. 9, Figure 6 · p. 9

6 Related Work

Prior alignment approaches tend to handle either explicit rewards or pairwise preferences, and often do not support both. The paper positions NCA and InfoNCA alongside NCE and InfoNCE, established contrastive optimization methods.

6 Related Work

  • Reinforcement learning handles explicit rewards but requires reward-model training followed by policy extraction, while preference-based methods align through one loss but are confined to pairwise data.Other approaches are also not tailored to reward datasets; one DPO extension considers only binary rewards, whereas this paper’s methods handle continuous rewards and preference data.Source: §6 · p. 9
  • NCE and InfoNCE are established optimization methods, respectively using binary classification and multiclass cross-entropy losses, with applications across representation learning and generative modeling.The passage also notes applications in language modeling, diffusion modeling, and reinforcement learning.Source: §6 · p. 9

7 Conclusion

The conclusion presents NCA and InfoNCA as contrastive alignment methods for reward and preference data, with InfoNCA including DPO as a special case. The appendix gives objective assumptions and pseudocode for applying the losses.

7 Conclusion

  • NCA and InfoNCA apply NCE and InfoNCE theories to both reward and preference data, with InfoNCA including DPO as a special case.The conclusion reports that reward-based alignment outperforms preference baselines by using suboptimal responses, and pairwise NCA outperforms DPO on complex reasoning tasks while preventing data likelihood from decreasing.Source: §7 · p. 10

A.1 InfoNCA Objective

  • The InfoNCA objective considers a batch with one reward-tilted optimal response and K−1 noise responses sampled independently from the pretrained model.The theorem passage states an optimal solution under unlimited model capacity and data samples.Source: §A.1 · pp. 14–15

A.2 NCA Objective

  • The NCA setup classifies a response as sampled either from the optimal policy or from the pretrained model, with the binary variable ν indicating the source.The appendix states an optimal-solution result under unlimited model capacity and data samples.Source: §A.2 · pp. 15–16

B Pseudocode

  • The pseudocode takes policy and reference log-probabilities, rewards, α, and β; its NCA branch combines an optimization term with a regularization term.The implementation also identifies InfoNCA and NCA as the selectable reward-loss types.Source: B Pseudocode · p. 17

C Experimental Details

The UltraFeedback experiments fine-tune models for one epoch with QLoRA, while the UltraInteract recipe fine-tunes all model parameters. The appendix also identifies the rating-system source.

C Experimental Details

  • UltraFeedback experiments use QLoRA with rank 16, train for one epoch, and set default α and β to 0.01.The setup fine-tunes from the Mistral-7B-SFT-beta model and uses AdamW with a 5e-6 learning rate.Source: C Experimental Details · p. 18
  • UltraInteract experiments fine-tune all parameters; the 7B model uses the Eurus recipe and the Mixtral-8x7B model is first supervised-fine-tuned with the Eurus-70B-SFT recipe.Both preference-learning recipes train for one epoch with β = 0.1.Source: C Experimental Details · p. 18
  • The rating system is adapted from [47].Source: Figure 7 · p. 18

D Additional Experiment Results

The appendix reports hyperparameter sensitivity results for InfoNCA and NCA, with NCA showing greater tolerance to variation.

D Additional Experiment Results

  • NCA shows greater tolerance to hyperparameter variations than InfoNCA.Figure 8 reports MT-bench scores across α and β, averaged over K = 2 and K = 4 experiments.Source: Figure 8 · p. 19
  • Performance is relatively more sensitive to β variations than to α variations.The comparison covers MT-bench scores for InfoNCA and NCA across both hyperparameters.Source: Figure 8 · p. 19
  • Table 4 compares different functional forms of rθ on UltraFeedback datasets.The caption identifies the table’s scope but does not report its results.Source: Table 4 · p. 19

E Appendix F: Additional Empirical Results

The appendix compares sampled generations from DPO and NCA policies, reporting that NCA generations are generally more similar to dataset-preferred responses.

Appendix F: Additional Empirical Results

  • NCA policy generations are generally more similar to the preferred response in the dataset than DPO generations.The appendix provides sampled responses to compare the policies’ data regularization features.Source: E Appendix F: Additional Empirical… · p. 20
  • The reported qualitative comparison concerns how policy generations resemble preferred dataset responses.The text frames this comparison as evidence about data regularization.Source: E Appendix F: Additional Empirical… · p. 20
  • Tables 5–8 present sampled generations and log-likelihood ratios for NCA and DPO/InfoNCA policies.The captions identify the table contents but do not establish specific numerical comparisons.Source: Table 5 · p. 20, Table 6 · p. 20, Table 7 · p. 21

Appendix H: Comparing NCA with DPO on Data Regularization.

Appendix H contrasts DPO’s relative-likelihood adjustment with NCA’s focus on absolute data likelihood, while Appendix G compares the proposed method with SLiC-HF on framework, regularization, and learning target.

Appendix H: Comparing NCA with DPO on Data Regularization.

  • The example assumes four possible responses with rewards, but the preference dataset contains only ya > yb, leaving yc and yd unobserved.The discussion notes that an unseen response such as yd may be low quality.Source: Appendix H · p. 22
  • NCA prevents the winning response likelihood from decreasing by optimizing absolute data likelihood rather than only the relative margin.The comparison says DPO can satisfy its loss while both observed response likelihoods fall, potentially increasing likelihood for an unseen, unreliable response.Source: Appendix H · p. 22

Appendix G: Comparision with Related Works

  • SLiC is adapted from LM calibration methods, whereas the proposed method is based on noise contrastive estimation.The passage also contrasts their policy-regularization approaches.Source: Appendix G · p. 22
  • SLiC’s training includes an additional SFT loss controlled by λ; the proposed method uses parameterization for regularization.The proposed parameterization is stated as rθ = β log πθ/µ, with β controlling it.Source: Appendix G · p. 22
  • SLiC directly optimizes policy model πθ, while the proposed method directly optimizes residual model rθ.Source: Appendix G · p. 22

Related papers

Loading 2402.05369v3…