PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

arXiv:2402.19085v3cs.CLcs.AIcs.SYeess.SY

TL;DR

Conflicting human preferences create an alignment tax, and existing alignment methods are mostly unidirectional. CPO conditions training on explicit preference scores; the paper reports that it outperforms several baselines across the 3H objectives.

  • Problem

    Alignment objectives can conflict, while most prior techniques optimize a scalar reward or heuristically mix data or reward models rather than address tensions across objectives in principle.

    Source: §1 · p. 1

  • Method

    CPO uses explicit preference conditions, with supervised fine-tuning on preference-token inputs followed by direct preference optimization under those conditions.

    Source: §1 · p. 2

  • Results

    CPO maintains alignment performance versus DPO in single-objective controllability and surpasses SFT, PPO, DPO, and Curry-DPO across helpfulness, honesty, and harmlessness.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Explicit conditioning is essential to alleviating alignment trade-offs, and the reported results show flexibility across the three 3H objectives.

    Source: §1 · p. 2, §6 · p. 9

    The study focuses on the 3H principles, and adversarial users may intentionally guide the model to generate harmful content.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the "alignment tax" -a compromise where enhancements in alignment within one objective (e.g.,harmlessness) can diminish performance in others (e.g.,helpfulness). However, existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. To navigate this challenge, we argue the prominence of grounding LLMs with evident preferences. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives, thereby guiding the model to generate responses that meet the requirements. Our experimental analysis reveals that the aligned models can provide responses that match various preferences among the "3H" (helpfulness, honesty, harmlessness) desiderata. Furthermore, by introducing diverse data and alignment goals, we surpass baseline methods in aligning with single objectives, hence mitigating the impact of the alignment tax and achieving improvements in multi-objective alignment.

1 Introduction

Conflicting 3H objectives create an alignment tax, while prior methods are mostly unidirectional and heuristic mixing does not address objective tension in principle. CPO conditions optimization on explicit preferences and reports improved alignment across helpfulness, honesty, and harmlessness.

1 Introduction

  • The “3H” objectives can conflict: improving helpfulness may undermine harmlessness, a trade-off known as the alignment tax (Ouyang et al., 2022).Source: §1 · p. 1
  • Prior methods mostly optimize scalar rewards or heuristically mix alignment data or reward models, leaving objective tension unresolved in principle (Schulman et al., 2017; Ouyang et al., 2022; Rafailov et al., 2023; Bianchi et al., 2023; Touvron et al., 2023).Source: §1 · p. 2
  • CPO makes user preferences explicit, guiding desirable behavior while optimizing other objectives as much as feasible.The approach narrows focus to fewer maximizing objectives to alleviate conflicts among alignment objectives.Source: §1 · p. 2
  • CPO combines supervised fine-tuning with preference learning: CPSFT adds preference tokens, and CDPO compares responses under the given value conditions.CPSFT trains generation conditioned on tokens such as <Helpfulness:5> and <Harmlessness:1>; CDPO increases the probability of the preferred response and decreases that of the other.Source: §1 · p. 2
  • CPO achieves single-objective controllability while maintaining alignment performance versus DPO, and surpasses SFT, PPO, DPO, and Curry-DPO across helpfulness, honesty, and harmlessness.The comparisons are reported for Mistral-7B and attribute the multi-objective results to explicitly grounding preference conditions.Source: §1 · p. 2

2 Method

CPO turns multi-objective alignment into a conditional problem guided by explicit preference conditions. It implements this through controllable preference SFT and controllable direct preference optimization.

2 Method

  • The method has two stages: controllable preference supervised fine-tuning (CPSFT) and controllable direct preference optimization (CDPO).Source: §2 · p. 2

2.1 Controllable Multi-objective Alignment

  • CPO makes value alignment conditional: controlled objectives track supplied preference conditions, steering behavior toward contextually relevant preferences.Source: §2.1 · pp. 2–3

2.2 Controllable Preference Optimization

  • CPO extends SFT and DPO into controllable CPSFT and CDPO to address conditional multi-objective optimization.Source: §2.2 · p. 3

2.2.1 Controllable Preference Supervised Fine-tuning

  • CPSFT adds preference conditions to the input, using prompt tokens such as <Helpfulness: 5> to train generations under those conditions.Source: §2.2.1 · p. 3

2.2.2 Controllable Preference Direct Optimization

  • CDPO compares conditional preference pairs using a weighted value that penalizes controlled objectives' deviation from conditions while maximizing uncontrolled ones; without controls, it reduces to DPO.Source: §2.2.2 · pp. 3–4

3 Experiments

Experiments assess the 3H metrics through controllability tests and multi-objective alignment evaluation.

3.1 Settings

  • UltraFeedback supplies fine-grained helpfulness and honesty scores, while UltraSafety adds 3,000 harmful instructions with jailbreak prompts and four security-level completions.HH-RLHF contributes chosen harmless and rejected harmful responses.Source: §3.1 · p. 4

3.2 Controllability on Different Aspects

  • GPT-4 scored helpfulness, honesty, and harmlessness on a 1–10 scale using MT-Bench, HaluEval 2.0, and 200 HackaPrompt jailbreak samples.The jailbreak attempts were grouped into three difficulty levels.Source: §3.2 · p. 4
  • CPSFT is more controllable than SFT; DPO improves harmlessness but lacks controllability, while CPO improves a single objective while preserving controllability.The paper attributes CPO’s approach to combining CPSFT with CDPO.Source: §3.2 · p. 4, Figure 3 · p. 5

3.3 Multi-Objective Alignment Evaluation

  • CPO achieves the best overall performance, particularly high safety scores while preserving helpfulness and honesty.Compared with SFT, PPO, DPO, and Curry-DPO using the same alignment data, it shows strong results across the three objectives.Source: §3.3 · p. 5

4 Analysis

Section 4 evaluates CPO through performance trade-offs, sensitivity analysis, case studies, and human evaluation.

4.1 Performance Trade-off Evaluation

  • CPO achieves a more favorable trade-off across helpfulness, honesty, and harmlessness than 3H-Best and DPO, while DPO shows cross-objective declines.The comparison uses MT-Bench, HaluEval 2.0, and Hackaprompt.Source: §4.1 · pp. 6–7, Figure 4 · p. 6

4.2 Sensitivity Analysis

  • At ω = 0.4, helpfulness and honesty reach a favorable balance; as λ increases, controllability rises while performance first improves and then deteriorates.The authors attribute the initial improvement to partial conflict mitigation and the later decline to effects on generative capacity.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 Case Study

  • With Helpfulness:5 and Honesty:5, the model may give harmful illegal guidance, while adding Harmlessness:5 prompts legal and ethical alternatives.The case study also reports more specific, higher-quality answers with higher helpfulness preference ratings.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 Human Evaluation

  • GPT-4 evaluations and human assessments show similarity exceeding 90% across MT-Bench, HaluEval 2.0, and HackaPrompt.The paper presents this agreement as evidence of the accuracy and effectiveness of GPT-4 evaluations.Source: §4.4 · p. 7, Table 3 · p. 8

5 Related Work

Related work covers alignment for helpfulness, harmlessness, and honesty, as well as approaches to alignment trade-offs and controllable generation. The paper distinguishes CPO by using controllable generation to address conflicts among objectives.

LLM Alignment

  • Prior alignment work emphasizes helpfulness and harmlessness, while honesty receives relatively less attention and is often studied through calibration and hallucination mitigation.The section also describes supervised fine-tuning approaches that teach models to refuse or express uncertainty beyond their knowledge boundary.Source: §5 · p. 8

Alignment Tax

  • Trade-off approaches include diverse rewards, SFT or DPO preference modeling, and Curry-DPO, but the paper says multi-objective learning can still involve mutual influence.Training multiple reward models is described as costly and unstable to fine-tune large foundation models.Source: §5 · p. 9

Controllable Alignment During Inference

  • Unlike controllable-generation work aimed at customizing user needs, this paper uses preference tokens to mitigate conflicts among multiple alignment objectives.The section also notes inference-time approaches using control tokens, data, model interpolation, or aligned-model logits.Source: §5 · p. 9

6 Conclusion

The conclusion presents explicit conditioning as essential to alleviating alignment trade-offs and identifies CPO as the proposed technique. It reports flexibility and performance across helpfulness, honesty, and harmlessness.

6 Conclusion

  • The paper concludes that explicit conditioning is essential to alleviating performance trade-offs in LLM alignment.Source: §6 · p. 9
  • CPO combines supervised fine-tuning with preference learning.Source: §6 · p. 9
  • Evaluation validates CPO's flexibility and performance in aligning helpfulness, honesty, and harmlessness.Source: §6 · p. 9

Limitations

The study is limited to the three established alignment principles of helpfulness, honesty, and harmlessness, and notes risks from adversarial users steering models toward harmful content.

Limitations

  • The study covers only helpfulness, honesty, and harmlessness; adversarial users may still steer models toward harmful content, so developers should consider access carefully.The authors describe their operational criteria as enabling controllability, while warning that full access may require careful consideration.Source: Limitations · p. 9

A.1 Introduction of Direct Preference Optimization

DPO starts from a reward-based RL objective but avoids explicitly modeling rewards by linking reward functions to optimal policies. Preference data then supports direct policy optimization without explicit reward estimation or reinforcement learning optimization.

Derivation of the DPO Objective

  • DPO avoids explicit reward modeling by using an analytical relationship between reward functions and optimal policies.The relationship expresses the optimal policy for a given reward function.Source: §A.1 · p. 13
  • Given the intractable partition function, the reward can be reparameterized using the optimal policy and reference policy.The derivation takes logarithms and rearranges the policy relationship.Source: §A.1 · p. 13

Preference-Based Optimization

  • Under models such as Bradley–Terry, preference probabilities depend on reward differences, allowing the partition function to be eliminated and a policy maximum-likelihood objective formed.The resulting objective does not require explicit reward estimation or reinforcement learning optimization.Source: §A.1 · p. 13

Gradient Analysis and DPO Update

  • DPO updates increase the likelihood of preferred outcomes and decrease that of less-preferred outcomes, weighted by relative estimated rewards.The gradient expectation is over preference data D.Source: §A.1 · p. 13

DPO Pipeline

  • The pipeline optimizes a policy on an offline preference dataset against the DPO loss, using a reference policy and predefined β.The authors say this permits reuse of existing preference datasets and mitigates distribution shift by appropriately initializing the reference policy.Source: §A.1 · p. 13

A.2 UltraSafety Dataset Construction

UltraSafety combines seed safety instructions, generated additional instructions, and screened jailbreak prompts, with model completions rated for harmlessness.

UltraSafety Dataset Construction

  • UltraSafety starts with 1,000 safety instructions from AdvBench and MaliciousInstruct, then bootstraps another 2,000 using Self-Instruct.The seed instructions are drawn from the two named datasets.Source: §A.2 · p. 13
  • The authors manually screen jailbreak prompts from AutoDAN and Shen et al., selecting 830 high-quality prompts.These prompts contribute to the dataset's jailbreak-safety coverage.Source: §A.2 · p. 13
  • Each harmful instruction is paired with completions from models at varying security levels and GPT4 ratings, where 1 denotes harmlessness and 0 denotes harmfulness.The passage lists a pool of 16 commercial, LLaMA-series, and non-LLaMA models.Source: §A.2 · p. 13

A.3 Open Source Models

The appendix describes model resources, training-data construction, single-objective controllability results, and additional evaluation materials. CPO is reported to match DPO on helpfulness and honesty while performing better on harmlessness.

A.3 Open Source Models

  • The appendix provides download links for WizardLM-7B, Zephyr-7B-beta, Mistral-7B-Instruct-v0.2, and LLaMA2-7B-chat.These are the four open-source models listed in this subsection.Source: §A.3 · p. 14

A.4 Construction of Training Data

  • CPSFT data balances single-, two-, and three-objective control; CDPO data attaches preference conditions and balances their selection across single- and multi-objective preferences.Preference-token values are based on UltraFeedback and UltraSafety ratings, with equal distribution across the three objectives and a 1:1 balance between controllability and performance maximization.Source: §A.4 · p. 14

A.5 Controllability on Single Objective

  • CPO performs comparably to DPO on helpfulness and honesty, exceeds DPO on harmlessness, and enables response-quality controllability.The passage characterizes these results as occurring without trade-offs within individual dimensions.Source: §A.5 · p. 14

A.6 Additional Results

  • Figure 6 compares DPO and CPO helpfulness controllability across eight MT-Bench subcategories.The passage identifies the breakdown as an additional helpfulness comparison.Source: §A.6 · p. 14, Figure 6 · p. 15

A.7 Case Studies

A.8 Evaluation prompts

  • Figures 7 and 8 provide the evaluation templates for harmlessness and honesty, respectively.The appendix identifies these as the prompts used in the experiments.Source: §A.8 · p. 14, Figure 7 · p. 16, Figure 8 · p. 18
  • Table 4 compares single-objective controllability using MT-Bench for helpfulness, HaluEval 2.0 for honesty, and HackaPrompt jailbreak prompts for harmlessness.For harmlessness, Condition 0 corresponds to table Condition 1, and Condition 1 corresponds to table Condition 5.Source: Table 4 · p. 15
Loading 2402.19085v3…