Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

1 Introduction

• Traditional rankings compare two responses under an identical fixed context, while human preferences can span non-identical contexts.The paper illustrates cross-context preference with preferring an accurate camera review over an incoherent movie review.Source: §1 · p. 1
• The authors propose eliciting rankings jointly over instruction-response pairs to capture preferences that fixed-context protocols may obscure.The example contrasts a helpful response with one that ignores its instruction's context.Source: §1 · pp. 1–2, Figure 1 · p. 2
• JPO upweights the joint probability of the chosen instruction-response pair over the rejected pair, and conditional rankings are a special case when instructions match.The paper contrasts this with DPO-style methods that rely on conditional rankings; ratings can also leave equal-scored responses indecisive.Source: §1 · p. 2
• JPO exceeds DPO by 5.2% and 3.3% win-rate points on summarization and dialogue datasets, respectively.The introduction also reports higher win rates against gold responses on unseen instructions and better AlpacaEval2 results against DPO and KTO.Source: §1 · p. 2

2 Related Work

• RLHF first fits a reward model on preference data, then uses reinforcement learning to align the language model to maximize that reward, sometimes with a divergence penalty.RLAIF uses language-model feedback as an alternative feedback source.Source: §2 · p. 3
• DPO optimizes directly in model-parameter space, bypassing reward modeling, and subsequent methods extend or modify its objective.The passage names listwise alignment, statistical rejection sampling, margin offsets, and reference-policy-free SimPO.Source: §2 · p. 3
• Unlike prior methods described here, JPO compares joint instruction-response distributions rather than only conditional response preferences.The related-work discussion contrasts JPO with RPO, which compares conditional likelihoods across prompts.Source: §2 · p. 3

3 Background

• A pretrained model's alignment workflow comprises instruction-response data collection, supervised fine-tuning, preference-data acquisition, and deployment of an alignment algorithm.The instruction-response data may be human-crafted or AI-generated.Source: §3 · p. 3
• After supervised fine-tuning, feedback such as rankings can support reward-model training and RLHF alignment.The passage also notes RLAIF as an alternative that uses language-model feedback.Source: §3 · p. 3
• DPO directly optimizes model parameters and bypasses reward modeling, addressing instability associated with PPO-based reward maximization.Source: §3 · p. 3

4 Joint Preference Optimization (JPO)

• The paper motivates joint comparisons as a way to model preferences across contexts, such as choosing the summary that better represents its corresponding article.Source: §4 · pp. 3–4
• Annotators select a preferred or equal instruction-response pair, with equal indicating that both pairs are perceived as equally good or bad.When the instructions are identical, this joint setup is equivalent to the original ranking protocol.Source: §4 · p. 4
• Joint feedback can be collected across combinations of existing instruction-response pairs without gathering additional instruction-response data.It also avoids requiring multiple responses for every instruction, which can be difficult for low-resource languages.Source: §4 · p. 4
• JPO upweights the preferred instruction-response pair's joint probability relative to the rejected pair, and reduces to DPO when instructions match.The objective includes a correction factor based on the prior probability of the instructions under the language model.Source: §4 · p. 4

4.1 Comparison of Joint Preferences with Prior Preference Protocols

• JPO learns from rankings over non-identical instructions, a capability the passage says prior preference work had not explored.The authors report that JPO outperforms both DPO and KTO, without giving a numeric margin here.Source: §4.1 · p. 4

5.1 Instruction-Response Acquisition

• The authors collect conditional rankings for responses to the same instruction and joint preferences for instruction-response pairs matched across different instances.They use TL;DR summarization and single-turn Anthropic-Helpful dialogues, collect AI and human feedback, and construct joint data from randomly selected responses.Source: §5.1 · pp. 4–5

5.2 Feedback from AI and Humans

• ChatGPT supplies conditional and joint preferences, while two human annotators label comparisons as chosen, rejected, or equal.The authors report approximately 50K AI comparisons and human annotations for 2,000 comparisons; average agreement is 67.5% between humans and 66.5% between human and AI annotators.Source: §5.2 · p. 5, §5.3 · p. 5

5.4 Interplay Analysis

• 71% of joint comparisons between instruction-response pairs whose responses were conditionally chosen or rejected received a decisive preference.The study also found 29% ties in comparisons involving one conditionally chosen and one rejected pair.Source: Figure 2 · p. 6, §5.4 · p. 7
• In 48% of comparisons between a conditionally chosen and rejected pair, annotators did not prefer the chosen pair: 19% preferred the rejected pair and 28% tied.The authors report that these judgments vary with the context of the comparisons.Source: Figure 3 · p. 6, §5.4 · p. 7
• Joint judgments could be decisive even when both responses had been rejected conditionally, with annotators comparing the responses’ accuracy.In the example, humans preferred response B for Instruction 1 over response D for Instruction 2, although both were conditionally rejected.Source: Table 3 · p. 6, §5.4 · p. 7

6.1 Setup

• The alignment experiments use TL;DR summarization and Anthropic-Helpful dialogue data, with win rates averaged across three sampling temperatures.Figure 4 reports the gold-response comparisons for the alignment objectives; its AlpacaEval2 panel uses temperature 0.7.Source: §6.1 · p. 7, Figure 4 · p. 8
• The base LLM is supervised-finetuned, then trained with JPO on merged conditional and joint preferences; DPO and KTO are applied to the SFT model for comparison.ChatGPT compares model and gold responses on 500 unseen test instructions, with win rates averaged across sampling temperatures 0.001, 0.5, and 1.0.Source: §6.1 · p. 7

6.2 Results

• JPO outperforms DPO by 5.2% and 3.3% win-rate points on the summarization and helpfulness datasets, respectively.JPO performs better than DPO across sampling temperatures, with similar trends relative to KTO.Source: §6.2 · p. 8, Figure 4 · p. 8

6.3 Extending the Results to AlpacaEval

• JPO-aligned LLM outperforms DPO-aligned LLM by 1.8 percentage points on AlpacaEval2 using the length-controlled win-rate metric.The authors describe this as an evaluation on a broad set of instructions.Source: §6.3 · p. 8, Figure 4 · p. 8

Impact of Joint Preferences over Non-Identical Instructions

• Training with joint preferences alone achieves a 71.7% win rate on summarization, demonstrating alignment without conditional preferences.The reported result is averaged across three sampling temperatures.Source: §7 · p. 8

Impact of Dataset Size

• JPO reaches a 71.9% win rate on TL;DR, beating same-size conditional-only DPO by 4.2 percentage points under a 50:50 conditional/joint mix.The total dataset size was fixed to match the conditional-preference dataset.Source: §7 · pp. 8–9, Figure 6 · p. 9

Data Scaling

• 42.4% to 71.7% win rate as joint comparisons rise from 100 to 9,000, with under 1% change from 4,000 to 9,000, showing nonlinear scaling.These measurements use the TL;DR dataset at sampling temperature 0.001.Source: Figure 7 · p. 9, §7 · p. 9

8 Conclusion

• JPO outperforms DPO and KTO on summarization and dialogue datasets, and also on AlpacaEval2.The paper also reports that JPO uncovers human decision-making heuristics obscured by the traditional approach.Source: §8 · p. 9
• The number of possible joint preferences grows quadratically with instruction-response dataset size, making informative-comparison selection a future research direction.The paper also identifies evaluation through joint rankings as future work.Source: §8 · p. 9

9 Limitations

• Ranking feedback does not quantify task-specific strengths or weaknesses, and different forms of feedback data can disagree.The paper notes these constraints as evidence of the complex, multidimensional nature of human preferences.Source: §9 · p. 9
• Joint comparisons between highly dissimilar instructions may be difficult and bias annotators toward preferring helpful or harmful responses.The paper suggests instruction similarity as a possible basis for deciding which pairs to compare.Source: §9 · pp. 9–10
• Most AMT annotators were from the U.S. or Canada, so the dataset does not represent the world’s diverse demographics.The paper calls for studying how demographic diversity affects feedback and subsequent LLM alignment.Source: §9 · p. 10

A Ranking Feedback Acquisition Protocol

• The ranking protocol collects preferences between responses for a fixed instruction, with annotators judging dimensions such as helpfulness, coherence, and harmlessness.The protocol starts with instructions and corresponding responses from a supervised fine-tuned model.Source: A Ranking Feedback Acquisition Protocol · p. 13

B Alignment Algorithms

• DPO aligns a model from conditional preferences between two responses for one instruction without an external reward model.The paper identifies the supervised fine-tuned model as the usual reference model.Source: B Alignment Algorithms · p. 13

C Dataset Statistics

• The reported instruction counts exclude instances with repeated instructions.The passage introduces these statistics as Table 4’s dataset summary.Source: C Dataset Statistics · p. 13
• Each dataset instance contains one instruction and a pair of responses, and the reported decisive-preference counts exclude ties.The passage specifies decisive preferences from ChatGPT.Source: C Dataset Statistics · p. 13

• JPO reduces to DPO when the prompts for preferred and rejected generations are the same, as shown in Lemma E.1.Source: D Proof for JPO subsuming DPO · p. 13

E JPO on AlpacaEval2 Leaderboard

• For the experiment, DPO aligned an SFT model using Ultrafeedback-binarized (60K instances), while JPO paired chosen and rejected instruction-response tuples as a proxy for true joint-preference distribution.JPO training used one epoch, β = 0.01, and learning rates {1e-7, 3e-7, 5e-7}; AlpacaEval2 responses were sampled at temperature 0.7.Source: E JPO on AlpacaEval2 Leaderboard · pp. 13–14

F Qualitative Examples

• The qualitative examples examine how human joint rankings and annotators’ reasoning relate to conditional rankings.Source: F Qualitative Examples · p. 14

G.1 Supervised Finetuning Details

• The Mistral-7B SFT details are presented in Table 6, and the model is full-finetuned using the cited source code.Source: §G.1 · p. 14, Table 6 · p. 16

G.2 JPO

• JPO training details are presented in Table 7; the learning rate is selected from {1e-5, 5e-5, 5e-4}, and DPO uses identical settings.The DPO source code uses the TRL library.Source: §G.2 · p. 14, Table 7 · p. 20

E JPO on AlpacaEval2 Leaderboard

• The evaluation compares SFT-, DPO-, and JPO-aligned models against gold responses using ChatGPT, with results averaged over three runs and responses sampled at three temperatures.Source: Table 5 · p. 15

F.1 Anthropic-Helpful Examples

• On Anthropic-Helpful examples, joint comparisons distinguish between responses both accepted conditionally and preserve the conditional preference when comparing an accepted with a rejected response.Source: Figure 8 · p. 15, Figure 9 · p. 16

H ChatGPT Prompts

• The paper provides separate ChatGPT prompts for acquiring conditional rankings and joint preferences over instruction-response pairs.Source: H ChatGPT Prompts · p. 15

F.2 TL;DR Summarization Examples

• TL;DR examples show joint rankings can reverse conditional preferences, distinguish two conditionally accepted responses, or prefer a conditionally rejected response over a non-rejected one.Source: Figure 10 · p. 17, Figure 11 · p. 18, Figure 12 · p. 19