Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment

1 Introduction

• MOA jointly considers objectives such as safety, helpfulness, factuality, and diversity, producing models across preference weights that form a (close-to) Pareto Front.The introduction identifies preference heterogeneity as the motivation for moving from single-objective to multi-objective alignment.Source: §1 · p. 1
• Conflicting preferences create contradictory optimization targets for DPO-based MOA, potentially disrupting alignment with individual objectives and hindering a superior Pareto Front.Because different objectives can favor different responses, simply discarding conflict instances is not considered viable given their high prevalence in existing datasets.Source: §1 · pp. 1–2
• On HelpSteer and BeaverTails, SIPO significantly improves over baseline methods; on BeaverTails, it achieves 2.1 and 3.0 average improvement on helpful and harmless rewards.The paper reports the two reward improvements for BeaverTails and significant improvement over baselines on both datasets.Source: §1 · p. 2
• SIPO self-generates and selects Pareto-optimal responses, then pairs them with original responses for non-conflicting DPO fine-tuning.After initial alignment, it samples high-quality responses using self-refinement and filters them for Pareto-optimality over the original responses.Source: §1 · p. 2

Results on Different Methods and Objectives

• At 90% conflicts, DPO soups approach unaligned SFT performance, while rising conflict ratios move Pareto fronts downward across datasets and methods.The downward trend appears for both DPO soups and MODPO.Source: Figure 2 · p. 3, Figure 3 · p. 3, Results on Different Methods and… · p. 3

2 Preliminary Experiments

• Conflicting labels create opposing optimization targets; in DPO Loss Weighting with two objectives, the losses are opposites and can produce conflicting gradient updates.The paper states that this issue extends to other DPO-based MOA approaches and larger objective counts.Source: §2 · p. 3
• The controlled experiments vary conflict ratios from 0% to 90% across MODPO and DPO soups, two objective pairs, and Alpaca-7B and LLaMA-2-7B backbones.The objective pairs are correctness and verbosity in HelpSteer, and helpfulness and harmlessness in BeaverTails.Source: §2 · pp. 2–3

Results on Different Methods and Objectives

• Helpful and harmless show larger average performance decreases than correctness and verbosity across both methods.The authors suggest this may relate to the more conflicting nature of helpfulness and harmlessness.Source: Results on Different Methods and… · p. 3
• LLaMA-2-7B performs better than Alpaca-7B on both BeaverTails objectives, yet its Pareto Front remains affected by conflict ratio.Figure 4 reports BeaverTails results for DPO soups with LLaMA-2-7B.Source: Figure 2 · p. 3, Figure 4 · p. 4, Results on Different Methods and… · p. 4
• Reducing the conflict ratio generally tightens Pareto-front performance ranges across preference weights, indicating lower steerability.The authors conjecture that reduced conflict may also reduce data versatility.Source: Results on Different Methods and… · p. 4
• More than 50% of BeaverTails examples have conflicting helpfulness and harmlessness preferences; HelpSteer conflicts exceed 10% and increase with more objectives.The statistics indicate that preference conflicts are prevalent in both datasets.Source: Results on Different Methods and… · p. 4, Table 1 · p. 4

3.1 Pareto-Optimal Responses

• Pareto-optimal responses outperform both original responses across all objectives, avoiding preference conflicts and supporting a better Pareto Front.The method uses the relationships between each constructed response and the original pair to form non-conflicting preferences.Source: §3.1 · p. 4

3.2 SIPO Framework: Responses Generation

• SIPO first aligns objective-specific policy LLMs with DPO, then samples responses under varied preference weights using MOD decoding.The varied weights are intended to increase sampling diversity, while MOD is used to ensure sampling quality.Source: §3.2 · pp. 4–5
• A self-refinement stage has an evaluator review sampled responses from different objective perspectives and revise them using the original sampling policy.The evaluator mixes objectives with a preference weight; review prompts include instructions and in-context examples.Source: §3.2 · p. 5
• Filtering estimates rewards with the implicit reward functions of the DPO policies and retains responses that outperform both originals across the evaluated objectives.When multiple candidates satisfy the constraints, SIPO selects the one with the largest average reward.Source: §3.2 · p. 5
• The filtering stage also evaluates candidates with mixed-objective policy LLMs formed by merging model weights under preference weights.These merged policies provide additional reward estimates for mixed objectives.Source: §3.2 · p. 5

3.3 SIPO Framework: Fine-Tuning

• Fine-tuning pairs each selected response with either original response as the preferred option, then applies DPO with an NLL term to reduce forgetting.The method describes the resulting preferences as non-conflicting and says they prevent forgetting on the original responses.Source: §3.3 · pp. 5–6

4.1 Results

• SIPO’s Pareto front outperforms all baselines on HelpSteer and BeaverTails, with larger gains over MOD on BeaverTails.The authors attribute the difference potentially to a higher proportion of preference conflicts in BeaverTails; BeaverTails gains are mostly on helpfulness, while both HelpSteer objectives improve.Source: §4.1 · pp. 6–7, Figure 6 · p. 6

4.2 In-depth Analysis

• Removing either refinement or filtering lowers the Pareto front, and removing filtering causes the larger decrease on both datasets.On HelpSteer, SIPO - filter falls below MOD under some preference weights, highlighting the importance of filtering for response quality.Source: §4.2 · p. 7, Figure 7 · p. 7
• Among alternative preference designs, learning only yc ≻ yw is comparable to SIPO on BeaverTails but lower on HelpSteer; the authors use yc ≻ yl to avoid forgetting.The sequential design and yw ≻ yl alone perform worse than SIPO, with conflicting-preference fine-tuning offered as a possible explanation.Source: §4.2 · p. 7, Figure 8 · p. 7
• Combining SIPO with DPO soups improves performance on both datasets.This supports the framework’s applicability alongside another DPO-based approach.Source: §4.2 · p. 7, Figure 9 · p. 7
• On HelpSteer, sampled yc has better rewards than the original preferred and dispreferred responses on both objectives; on BeaverTails, it improves harmlessness but lowers helpfulness by 0.4.SIPO nevertheless improves the BeaverTails Pareto front in helpfulness, while the imbalanced gains indicate a need for further optimization.Source: §4.2 · pp. 7–8, Table 2 · p. 8
• Replacing conflicting preferences with non-conflicting ones does not improve performance and can underperform non-training MOD.This result is presented as confirming SIPO’s suitability for resolving conflicts.Source: §4.2 · p. 8, Figure 10 · p. 8
• A second SIPO round on BeaverTails improves performance and average response rewards over the first round; helpfulness rises 85.29% (3.4 → 6.3).The first-round helpfulness change was -11.75%.Source: §4.2 · p. 8, Figure 11 · p. 8

Generalization to Different Backbone LLMs

• With Alpaca-7B, SIPO - refine consistently outperforms all baselines on BeaverTails despite the backbone’s context-length limitation.The authors use SIPO - refine, the closest variation of SIPO, for this evaluation.Source: Generalization to Different Backbone… · p. 8

Learning from Human Feedback

• RLHF uses reward models and reinforcement learning, while DPO learns directly from preference data; AI-generated feedback has also been explored to reduce human labeling.The paper describes DPO-based methods as aiming to improve efficiency and stability over RL-based methods.Source: §5 · p. 8

Multi-Objective Alignment of LLM

• MOA methods include RL-based proxy rewards, DPO-based approaches such as MODPO, decoding-time logit manipulation, and prompt-based techniques.Other work explores constraints among objectives and conditional generation, which the paper describes as orthogonal to its setting.Source: §5 · pp. 8–9

Self-Improvement

• Prior work applies self-improvement through self-generated data and feedback, including with DPO; SIPO differs from Wang et al. (2024b) by focusing on sampling Pareto-optimal responses.Source: §5 · p. 9

• SIPO automatically generates and uses Pareto-optimal responses to resolve preference conflicts, and outperformed baseline methods in achieving a superior Pareto front.Future work will extend experiments to more objectives and DPO-based methods and improve the efficiency of obtaining Pareto-optimal responses.Source: §6 · p. 9

Experiments on More Objectives and DPO-based Approaches

• Experiments primarily cover two objectives per dataset and combine SIPO with MOD and DPO soups; broader exploration of objectives, methods, and backbone sizes remains future work.The paper notes experiments with three objectives and a 3B model in Appendix B.Source: Limitation · p. 9

More Validation on the Effectiveness of Pareto-optimal Response

• The alternative of sampling separate responses for each objective has not yet been compared with SIPO, though the authors consider it potentially less effective.Source: Limitation · p. 9

Generating Pareto-optimal Responses with Additional Stronger LLMs

• This work uses self-improvement without human-labeled or stronger-LLM-labeled data; distilling Pareto-optimal responses from stronger LLMs remains future work.Source: Limitation · p. 9

Additional Computational Overhead

• Sampling, refinement, filtering, and fine-tuning could make SIPO more computationally costly than simpler methods.The authors plan to explore ways to reduce this overhead while maintaining the benefits of self-improvement.Source: Limitation · p. 9

Ethical Consideration

• Examples in Appendices C and E may contain harmful or offensive content.Source: Ethical Consideration · p. 9

A.1.1 BeaverTails

• BeaverTails uses 9,000 training examples, 1,000 validation examples, and 500 non-overlapping test questions.The training and validation data come from BeaverTails-10k; test questions come from BeaverTails-30k.Source: §A.1.1 · p. 13

A.1.2 HelpSteer

• HelpSteer experiments use correctness and verbosity preferences, with 970 training instances, 216 validation instances, and 188 test questions.Pairs with identical scores on either selected dimension are excluded.Source: §A.1.2 · p. 13

A.2 Details of External Reward Models

• BeaverTails evaluation uses reward and cost models, while HelpSteer evaluation uses correctness and verbosity scores from Wang et al. (2024b).Cost is treated as a negative value to represent harmlessness reward.Source: §A.2 · p. 13

A.3 Implementation Details

• Supervised fine-tuning uses all training-split responses, with three epochs on BeaverTails and two on HelpSteer.Both use max_length 2048, batch size 1, and two gradient-accumulation steps; learning rates are 1e-4 and 1e-5, respectively.Source: §A.3 · p. 13
• SIPO uses one training epoch, β=0.1 and α=0.1, with a 5e-6 fine-tuning learning rate for HelpSteer and objective-specific rates for BeaverTails.The BeaverTails rates are 5e-6 for helpfulness and 5e-5 for harmlessness; other training and sequence-length settings are also specified.Source: §A.3 · p. 13
• Refinement generates and rewrites responses using preference weights, then applies six reward models at weights {0.0, 0.2, 0.4, 0.6, 0.8, 1.0} to filter for Pareto-optimality.The process uses 2,500 BeaverTails samples and 582 HelpSteer samples.Source: §A.3 · p. 13
• The refinement process yields 2,102 Pareto-optimal BeaverTails responses and 369 HelpSteer responses for fine-tuning.Source: §A.3 · p. 14

B Supplementary Experimental Results

• Figure 12 reports HelpSteer performance across six preference weights, while Figure 13 summarizes average SIPO improvement over MOD across weights.Source: Figure 12 · p. 14, Figure 13 · p. 14
• SIPO combined with DPO soups achieves Pareto improvement at four of six points on BeaverTails with Qwen2.5-3B-Instruct, with competitive results at the others.The experiment uses 400 sampled responses without extensive parameter tuning.Source: B Supplementary Experimental Results · p. 14, Table 3 · p. 15
• On three-objective Anthropic-HH tasks, SIPO exceeds MOD on 14 of 18 reward scores, with the remaining scores comparable.The objectives are helpfulness, harmlessness, and humor; extensive parameter tuning was not performed due to time constraints.Source: B Supplementary Experimental Results · p. 14, Table 4 · p. 15
• Using fewer sampling weights can bias responses toward one objective, so a sufficiently diverse set supports balanced multi-objective optimization.Sampling-weight choices significantly affect response rewards.Source: B Supplementary Experimental Results · p. 14, Table 5 · p. 15
• Filtering weights have less impact on rewards; {0, 1.0} is generally sufficient, while a single intermediate weight such as 0.6 can introduce bias.Reducing filtering weights can improve efficiency while preserving response quality.Source: B Supplementary Experimental Results · p. 14, Table 5 · p. 15

Tabular Results for Performance Comparison

• Tables 6 and 7 provide detailed numerical performance comparisons for BeaverTails and HelpSteer, respectively.Source: Tabular Results for Performance… · p. 14, Table 6 · p. 15, Table 7 · p. 15

C Prompts for Review Generation and Rewriting

• Review-generation and rewriting prompts use one-shot examples on HelpSteer; BeaverTails uses two-shot review generation and one-shot rewriting.Source: C Prompts for Review Generation and… · p. 16

D Discussion on Efficiency and Computation Overhead

• SIPO adds computational overhead through sampling, refinement, filtering, and fine-tuning.Source: D Discussion on Efficiency and… · p. 16
• Applying SIPO to a subset of conflicting examples, processing weights in parallel, and using a single round reduce computation needs.The implementation applies the stages to a subset such as 2,500 BeaverTails examples; one round can produce substantial improvements.Source: D Discussion on Efficiency and… · p. 16
• SIPO + soups achieves better performance than MODPO with fewer GPU hours, while SIPO + MOD adds overhead.The discussion attributes most of the latter cost to MOD’s expensive decoding process.Source: D Discussion on Efficiency and… · p. 16
• Future efficiency directions include multi-objective decoding, early discarding before refinement, sequential filtering, and active-learning prioritization.Sequential filtering can reduce the number of responses filtered for each objective; active learning prioritizes ambiguous or highly conflicting examples.Source: D Discussion on Efficiency and… · p. 16

E Case Study

• The BeaverTails-10K rewrite adopts nonjudgmental support, making the response less harmful while preserving helpfulness.Table 12 is identified as an example from SIPO's BeaverTails-10K refinement stage.Source: E Case Study · p. 16, Table 12 · p. 20
• The HelpSteer rewrite becomes significantly longer by adding detail while maintaining factual accuracy and relevance.Table 13 is identified as an example from SIPO's HelpSteer refinement stage.Source: E Case Study · p. 16, Table 13 · p. 22
• A sample response describes magazine features including fact files, comic strips, activities, puzzles, and articles.Its examples elaborate on fact files and comic strips, including their informational and educational aims.Source: E Case Study · p. 21