PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Objectives Need a Dial? Predicting Objective Conflict and Covering Trade-offs in Steerable Pluralistic Alignment

David Tsoi, Esra Dönmez

arXiv:2609.26929v1cs.AIcs.CLcs.CY

TL;DR

Diverse, sometimes conflicting values make a single averaged preference an incomplete way to align models. The paper tests MODPO compatibility predictors and training-free ways to cover trade-offs; predictors work on human-annotated data but not AI-annotated data, and reuse methods do not reliably match direct training.

  • Problem

    The paper asks when one MODPO model can improve two objectives together and how to cover many trade-offs without training a separate model for each.

    Source: §1 · p. 1

  • Method

    The study tests two pre-training compatibility measurements across objective pairs and compares nearest-model selection and parameter merging with direct training.

    Source: §1 · p. 2

  • Results

    The measurements predicted compatibility on human-annotated HelpSteer but failed on AI-annotated UltraFeedback, while nearest-model selection and merging did not reliably match direct training.

    Source: §1 · p. 2, §7 · pp. 7–8

  • Takeaways & Limitations

    Length and repetition confounded UltraFeedback reward-model scores, so objective compatibility there requires independent evaluation to confirm.

    Source: §7 · p. 7

    The study steers only two objectives at a time, and all trained-model results use a single seed.

    Source: Limitations · p. 9

Abstract

from arXiv · show

People hold diverse, sometimes conflicting values, so no single aligned model can satisfy everyone. Pluralistic alignment therefore calls for steerable models that can balance competing objectives differently. Multi-Objective Direct Preference Optimization (MODPO) does this by using an objective weight to span a continuum of trade-offs. We study two questions: when can one model improve two objectives simultaneously, and how can many trade-offs be covered without training a separate model for each? Across seven objective pairs from HelpSteer and UltraFeedback, two pre-training measurements predict whether objectives align or conflict for human-annotated data, but not for AI-annotated data, where response length and repetition confound reward-model scores. For broader trade-off coverage, selecting the nearest trained model and merging model parameters both help, but neither consistently matches direct training. These findings yield practical guidance for building steerable models that serve diverse preferences.

1 Introduction

People value different, sometimes competing qualities in model responses, so a single averaged preference can hide whose values are served. The paper studies when MODPO can improve two objectives together and how trained models can cover new trade-offs without retraining.

1 Introduction

  • MODPO uses an objective weight to steer a model between two objectives, with models trained at different weights spanning a continuum of trade-offs.Stakeholders can select a compromise suited to their context.Source: §1 · p. 1

RQ1

  • The paper asks whether conflict rate and SFT reference-model score correlation can predict compatibility before MODPO training.Conflict rate measures how often objectives prefer different responses; correlation measures whether the reference model’s higher-scoring responses for one objective also score higher for the other.Source: §1 · p. 1

RQ2

  • Nearest-model selection and parameter merging reuse trained models to cover new trade-offs, but neither reliably matches direct training.Selection uses the trained model closest in weight; merging blends adapters from neighboring trained models.Source: §1 · pp. 1–2

Contributions

  • Across seven objective pairs, two pre-training measurements predicted joint improvement or trade-offs on human-annotated HelpSteer but failed on AI-annotated UltraFeedback.The UltraFeedback failure was strongly confounded by response length and repetition in reward-model scores.Source: §1 · p. 2

2 Related Work

Prior work addresses pluralistic alignment, multi-objective steering, preference disagreement, model reuse, and reward-model evaluation. This paper uses MODPO to test whether objective compatibility can be predicted before training and compares reuse methods with direct training.

Pluralistic alignment

  • Prior pluralistic-alignment approaches include population-level agreement, max-min optimization over diverse preferences, and coordination among multiple LLMs.This paper focuses on steerable pluralism, adjusting one model’s objectives for different stakeholders.Source: §2 · p. 2

Steering multiple objectives

  • Existing steering methods set objective balance in the DPO loss, a reward direction, the prompt, or generation-time decoding; this work asks whether outcomes can be predicted before training.MODPO is one example of incorporating balance into the DPO loss (Zhou et al., 2024).Source: §2 · p. 2

Disagreement in preference data

  • Preference conflict measures when two objectives prefer different responses for the same pair; this paper measures conflict rather than trying to remove it.Li et al. (2025) describe conflict as common and reduce it by rewriting responses.Source: §2 · p. 2

Reusing and merging models

  • For new preference balances, this work blends LoRA adapters from neighboring trained MODPO models and compares merging with nearest-model selection and direct training.Prior weight-space methods include Rewarded Soups (Rame et al., 2023) and Personalized Soups (Jang et al., 2024).Source: §2 · p. 2

Evaluating with reward models

  • Because reward-model scores can reflect response length and repetition, the paper reports these alongside reward scores and re-decodes with repetition control.These checks help interpret whether objectives are genuinely compatible.Source: §2 · p. 2

3 Background

MODPO trains a policy for a weighted mix of two objectives, using helpfulness preference pairs and a second-objective reward margin. The setup relies on an SFT reference model and can store objective-weight variants as small adapters.

Preference data and conflict

  • The preference-pair conflict rate is the fraction of jointly labeled response pairs on which the two objectives prefer different responses.It is computed before training and measures disagreement in the data.Source: §3 · p. 3

SFT reference model

  • The SFT reference model initializes each MODPO run, anchors the policy, and supplies pre-training generations for scoring.It is trained on helpfulness demonstrations without preference signal or the second objective.Source: §3 · p. 3

MODPO

  • MODPO combines two objectives in one policy, with weight w setting their balance and the second objective entering through a reward-model margin.The preference pairs come from helpfulness; flipping the margin sign reverses which direction of the second objective is preferred.Source: §3 · p. 3

LoRA

  • LoRA freezes pretrained weights and learns a low-rank adapter, making it cheap to store objective-weight variants and enabling adapter merging.The trained model consists of the base model plus its adapter.Source: §3 · p. 3

4 Method

The method evaluates compatibility predictors and training-free reuse against direct MODPO training. It measures two pre-training signals, probes conflict-rate behavior descriptively, and tests nearest-model selection and adapter merging at target weights.

4 Method

  • The studies test whether pre-training measurements predict joint improvement or trade-offs, and whether trained models can be reused to reach new trade-offs without retraining.The method distinguishes dataset scores used to build preference pairs from evaluator reward-model scores on generated responses.Source: §4 · p. 3

4.1 Measuring objective compatibility before training

  • Conflict rate counts jointly labeled response pairs where objectives disagree, while reference-model correlation compares their evaluator scores on generations from the SFT model.A high correlation and low conflict rate are expected to indicate objectives that move together.Source: §4.1 · pp. 3–4

4.2 Probing the conflict rate

  • To probe conflict rate, the study retrains only the second-objective margin reward model on conflict-only and non-conflict-only subsets while holding helpfulness pairs fixed.Because the subsets differ in size and composition, the results are descriptive rather than causal.Source: §4.2 · p. 4

4.3 Covering the trade-off continuum at inference time

  • For target weight w⋆, nearest-model selection uses the closest trained weight, while merging interpolates the LoRA adapters of the closest lower and upper models without new training.The merge is a heuristic, not an optimization at w⋆, so its behavior is not guaranteed to match a directly trained model.Source: §4.3 · p. 4

5 Experimental Setup

The experiments compare seven objective pairs across human-scored HelpSteer and AI-scored UltraFeedback, then evaluate trade-off coverage on PKU-SafeRLHF-10K. Models use LoRA, and evaluation measures objective scores alongside response length and repetition.

Datasets

  • HelpSteer supplies human scores for five objectives across 10,459 prompts, while UltraFeedback supplies GPT-4 scores for four objectives across 63,967 prompts.The coverage study uses PKU-SafeRLHF-10K with the helpfulness/harmlessness pair.Source: §5 · p. 4

Objectives

  • The seven pairs fix helpfulness as the first objective and test four HelpSteer and three UltraFeedback second objectives, with both verbosity margin signs creating eight settings.The selected objectives span expected reinforcement, independence, and competition with helpfulness.Source: §5 · p. 4

Models and training

  • LLAMA-2-7B is used for HelpSteer and ALPACA-7B for UltraFeedback and PKU-SafeRLHF; training uses LoRA and MODPO models at multiple objective weights.The trade-off study uses an SFT reference; coverage uses the ALPACA-7B checkpoint without further SFT and compares fine, coarse, and target weight sets.Source: §5 · p. 4
  • At the 2048-token limit, Figure 1 plots both pre-training measurements for the objective pairs; HelpSteer correctness and coherence show the clearest high-correlation, low-conflict pattern.HelpSteer verbosity toward concision has the highest conflict rate, while the measurements part company for verbosity.Source: Figure 1 · p. 5
  • The primary training limit is 2048 tokens, which retains almost all preference pairs; a 512-token limit retains far fewer.Training uses learning rate 10^-4, β = 0.1, AdamW, a cosine schedule, and one seed per model.Source: §5 · p. 5

Evaluation

  • Evaluation uses one greedy response per held-out prompt or response pair, scores objectives with ArmoRM or Beaver models, and reports scores, length, repetition, and paired-bootstrap intervals.The 2048-token models are also re-decoded with repetition controls to test whether score changes track length and repetition.Source: §5 · p. 5

6 Results

Across HelpSteer, pre-training measurements track whether MODPO can improve two objectives together, but they fail to predict UltraFeedback outcomes, where length and repetition confound reward-model scores. For broader trade-off coverage, model selection and parameter merging reduce the need for retraining, yet neither reliably matches direct training.

6.1 When can one model satisfy two objectives? (RQ1)

  • Under the 2048-token limit, HelpSteer correctness and coherence had high correlations (0.98, 0.91) and low conflict rates (0.03, 0.07), and MODPO improved both objectives at every weight.Complexity showed a partial match, while verbosity steered toward concision produced a trade-off consistent with its high conflict rate (0.774).Source: §6.1 · p. 5, Table 1 · p. 5
  • Despite high correlations (0.81–0.88) and moderate conflict, all three UltraFeedback pairs degraded the second objective at every weight and never improved both.Every UltraFeedback model scored below the reference on truthfulness, so the measurements that worked for HelpSteer did not predict joint improvement here.Source: §6.1 · p. 5
  • With repetition control, UltraFeedback objective scores rose by +0.10 to +0.15, while HelpSteer scores barely moved; the apparent conflict is therefore strongly confounded by length and repetition.Re-decoding changes length, repetition, wording, and content together, so it does not isolate the cause; an independent evaluator would be needed to settle it.Source: §6.1 · p. 6, Table 2 · p. 6
  • Flipping the margin sign reverses the preferred direction of the second objective, making direction a control knob orthogonal to objective weight.For verbosity, the positive sign raises verbosity at every weight but improves helpfulness only at w=0.9; the negative sign steers toward concision and yields a trade-off.Source: §6.1 · p. 6, Figure 3 · p. 7
  • Training the margin reward model only on conflicting pairs shifts outcomes but does not decide them; correlation still matters alongside conflict.For HelpSteer verbosity toward concision, the conflict-set split improved helpfulness at every weight while worsening concision at higher weights.Source: §6.1 · pp. 6–7

6.2 Covering the continuum without retraining (RQ2)

  • Merging has the smallest combined mean gap from direct training (1.14, versus 1.69 for fine selection and 2.40 for coarse), but neither reuse method reliably reproduces a directly trained model.Merging is not consistently closest at individual weights; its 95% confidence interval for the paired difference includes zero in 5 of 10 comparisons, which indicates no detected difference, not equivalence.Source: Table 3 · p. 7, §6.2 · p. 7

7 Discussion

The compatibility measurements work on human-annotated HelpSteer but are confounded on AI-annotated UltraFeedback. Reusing trained models covers new trade-offs approximately; whether objectives need a dial depends on genuine conflict, and model behavior should be checked empirically.

7 Discussion

  • The conflict rate and reference-model correlation predict MODPO behavior on human-annotated HelpSteer, but not AI-annotated UltraFeedback, where length and repetition confound reward scores.Controlling for those confounds makes results more consistent with the measurements, but an independent evaluator is needed to confirm compatibility.Source: §7 · p. 7
  • Use high correlation and low conflict to identify candidate objective pairs for joint service, then check generated length, repetition, and decoding sensitivity.For candidate trade-offs, the recipe is low correlation, high conflict, and a margin sign chosen for the desired steering direction.Source: §7 · p. 8
  • Parameter merging has the smallest combined gap from direct training, but neither merging nor selection consistently matches direct training at individual target weights.Merging is an interpolation heuristic, not an optimization procedure for the target weight.Source: §7 · p. 8, Figure 5 · p. 8
  • For new target weights, nearest-model selection is a simple fallback and merging can fill gaps, but train directly when exact behavior matters.The reuse methods offer cheap, approximate coverage rather than matching direct training reliably.Source: §7 · p. 8
  • A trade-off dial is a false choice when objectives reinforce each other; expose one only when objectives genuinely compete, after accounting for length and repetition confounds.Measurements should flag candidate trade-offs or jointly served pairs, while affected people make the normative choice of which trade-offs to expose.Source: §7 · p. 8
  • Characterize MODPO empirically by generated length, repetition, and scored behavior because the same objective weight can behave differently with different data and decoding.The objective weight is a steering dial, not a guarantee of a particular behavior.Source: §7 · p. 8

8 Conclusion

The paper studies when MODPO can jointly satisfy two objectives and how to cover trade-offs cheaply. Across seven objective pairs, compatibility measurements predict behavior on human-annotated HelpSteer but not AI-annotated UltraFeedback, while model reuse helps cover new trade-offs without reliably matching direct training.

8 Conclusion

  • Across seven objective pairs, two pre-training measurements predicted MODPO behavior for human-annotated HelpSteer but not AI-annotated UltraFeedback, where length and repetition confounded reward-model scores.Controlling for these confounds made results more consistent with the measurements, though an independent evaluator is needed to confirm compatibility.Source: §8 · pp. 8–9
  • Whether objectives can be served jointly cannot be read from one number; it depends on the data, decoding, and evaluator together.An independent evaluator would be needed to confirm compatibility after controlling for length and repetition confounds.Source: §8 · p. 9
  • Nearest-model selection and parameter merging help cover new trade-offs without retraining, but neither reliably matches direct training.The paper presents these findings as practical guidance for building models that serve diverse stakeholder preferences.Source: §8 · p. 9

Limitations

The study’s conclusions are bounded by its two-objective, model, evaluator, and training settings; its conflict-split comparison is suggestive rather than causal.

Limitations

  • Results cover two objectives at a time and one 7B base model per dataset, so other objective pairings, larger or different models, and multi-objective behavior may differ.The authors also note that conflict-split training changes dataset size and composition, making the results suggestive of a conflict-rate effect rather than causal.Source: Limitations · p. 9
  • Coverage tests only a fixed set of trained models and target weights, with parameter merging evaluated only at λ = 0.5.The authors propose testing other targets and merge positions, and a selection rule using reward and cost scores rather than weight distance alone.Source: Limitations · p. 9
  • A single seed per trained model leaves seed-to-seed variation unquantified.Source: Limitations · p. 9
  • Reward-model scores are proxies for human judgment; within-evaluator comparisons do not establish absolute score levels or transfer.The authors note evaluator training-data overlap with the datasets and recommend an independent human or LLM-judge evaluation, especially for UltraFeedback’s length and repetition confounds.Source: Limitations · p. 9

Ethics Statement

Repetition control changes the per-pair scores more on UltraFeedback than HelpSteer, while the paper flags governance and reward-model risks and documents its evaluation setup.

Ethics Statement

  • Exposing steerability can permit weights favoring helpfulness over harmlessness, and choosing which trade-offs to expose is a governance question beyond the study’s scope.The authors caution that reward-model scores reflect training-data biases and response length and repetition, not ground truth for human values.Source: Ethics Statement · p. 9

A Training and Evaluation Details

  • HelpSteer uses its official 95/5 split, UltraFeedback a seeded random 90/10 split, and PKU-SafeRLHF-10K a 9,000/1,000 split.Evaluation uses 400 unique prompts for trade-offs and 700 response pairs for coverage.Source: A Training and Evaluation Details · p. 12
  • Training uses a single seed per model, greedy primary decoding, and paired bootstrap confidence intervals over prompts or response pairs.Training settings, LoRA configurations, and generation limits differ by study and dataset; intervals use 10,000 resamples at 95%.Source: A Training and Evaluation Details · p. 12

B Objective Compatibility Measurements

C Per-Model Trade-off Scores

  • Table 5 lists per-model helpfulness and second-objective scores across eight settings, training-length limits, and decoding conditions.Table 6 supplies corresponding length and repetition diagnostics.Source: C Per-Model Trade-off Scores · p. 12

D Repetition Control, Per Pair

  • No HelpSteer pair shifts by more than 0.036 on either objective, while every UltraFeedback pair shifts by at least 0.103 under repetition control.Length and repetition drops are comparable across datasets.Source: D Repetition Control, Per Pair · p. 12

E Conflict-Split Results

Conflict-split scores move most on the conflict set; in coverage tests, nearest-model selection and merging have limits relative to direct training.

E Conflict-Split Results

  • The joint and non-conflict sets stay close to the main results, while scores move on the conflict set.Table 8 reports helpfulness and second-objective scores for joint, non-conflict, and conflict sets; the split is based on dataset score direction.Source: E Conflict-Split Results · p. 12, Table 8 · p. 14

F Coverage Study

  • Coarse selection returns the directly trained model at some target weights; Table 10 reports each target’s absolute gap from direct training.Table 3 averages these per-target gaps.Source: F Coverage Study · p. 12, Table 10 · p. 16
  • Every merged model falls between its neighboring models, so it cannot reach direct-target scores outside that interval.The caption identifies such targets as reward at w⋆=0.7 and 0.9, and cost at w⋆=0.1.Source: Table 11 · p. 16

Related papers

Loading 2609.26929v1…