PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee, Kyunghyun Cho, Sangdon Park

arXiv:2610.04845v1cs.LG

TL;DR

Multi-objective alignment must accommodate conflicting human values and varied user trade-offs, but existing approaches either train separate preference models, combine experts after training, or use fixed preference sampling. MAESTRO trains one prompt-conditioned policy against an adaptive Dirichlet preference distribution and reports the highest hypervolume on all three objective pairs across its evaluated tasks, with the lowest compared training cost.

  • Problem

    Existing methods leave hard preference regions under-trained because which regions are difficult depends on the objectives, while separate models or post-hoc expert combinations have other limitations.

    Source: §1 · p. 1, §2 · p. 4

  • Method

    MAESTRO trains a prompt-conditioned policy end-to-end with RL while online mirror descent updates a Dirichlet preference distribution toward preferences the current policy serves worst.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    MAESTRO attains the highest hypervolume on all three objective pairs across HH-RLHF, BeaverTails, and summarization, matching or exceeding multi-model methods at the lowest training cost among compared methods.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Adapting where it trains raises the trade-off a single conditioned policy realizes, and the reported ranking transfers to held-out reward models and an unoptimized objective pair.

    Source: §5 · p. 10

    The approximate Pareto-optimality guarantee assumes simultaneous ε-realizability: one conditioned policy must be uniformly within ε of every per-preference optimum.

    Source: §2 · p. 4

Abstract

from arXiv · show

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

1 INTRODUCTION

Multi-objective alignment seeks a preference-conditioned policy that can trace trade-offs among conflicting objectives. MAESTRO addresses the challenge of choosing which preferences to train on by adapting their distribution during training.

1 INTRODUCTION

  • Per-preference scalarization requires a separate model for each preference, while post-hoc methods combine a smaller set of experts rather than optimizing intermediate trade-offs during training.Post-hoc approaches rely on assumed linearity in weight or logit space; single-model methods such as RiC and SteerLM instead write reward values into prompts and presuppose reward ranges.Source: §1 · p. 2, §1.1 · pp. 2–3
  • Fixed preference distributions can leave hard regions under-trained, and which regions are hard varies with the objectives and as the policy improves.The paper’s Figure 1 contrasts preference-sampling concentrations that work differently on Harmless–Helpful and Humor–Helpful.Source: §1 · p. 2, Figure 1 · p. 2
  • MAESTRO trains one prompt-conditioned policy end-to-end against a Dirichlet preference distribution updated toward preferences the policy serves worst.The paper reports the highest hypervolume on all three objective pairs across HH-RLHF, BeaverTails, and summarization, at the lowest training cost among compared methods.Source: §1 · p. 2, §1.1 · p. 3

1.1 RELATED WORK

  • Existing single-model methods also use preferences drawn from distributions fixed before training, whereas MAESTRO updates its training distribution jointly with the policy.The related-work discussion distinguishes fixed sampling from MAESTRO’s policy-dependent preference distribution.Source: §1 · p. 2, §1.1 · p. 3

2 MULTI-OBJECTIVE ALIGNMENT WITH A PREFERENCE-CONDITIONED POLICY

Multi-objective alignment trains a language model to respond to user preferences over several objectives, with the goal of covering their Pareto trade-offs. The section contrasts per-preference and post-hoc approaches with a worst-case objective over preference distributions.

2 MULTI-OBJECTIVE ALIGNMENT WITH A PREFERENCE-CONDITIONED POLICY

  • The preference simplex represents relative weights over objectives, and a conditioned policy’s return is the expected reward vector weighted by the specified preference.The section models responses as token-level generation and uses linear scalarization to define preference-weighted return.Source: §2 · pp. 3–4
  • Training a separate policy for every preference is costly, while post-hoc methods train objective experts and combine them at inference rather than optimizing those trade-offs directly.The cited approaches include parameter interpolation and mixing next-token distributions.Source: §2 · p. 4
  • Maximizing expected return under a fixed preference distribution can leave hard regions under-trained, and the hard regions depend on the objectives.The section motivates a worst-case objective over candidate preference distributions instead of a fixed average.Source: §2 · p. 4
  • Under simultaneous ε-realizability, a conditioned policy uniformly within ε of every per-preference optimum is approximately supported Pareto-optimal at strictly positive weights.The guarantee applies to the inner minimax lower bound for any fixed distribution in the ambiguity set.Source: §2 · p. 4

3 MAESTRO: MULTI-OBJECTIVE ALIGNMENT VIA END-TO-END STEERING AND ROBUST OPTIMIZATION

MAESTRO trains a single preference-conditioned policy end-to-end, pairing prompt-based preference inputs with an adaptive Dirichlet adversary. The policy and adversary alternate updates so training focuses on preferences the current policy serves worst.

3.1 PROMPT-BASED CONDITIONING FOR MULTI-OBJECTIVE ALIGNMENT

  • The model receives preference weights as a natural-language prompt prefix, allowing users to specify a trade-off at deployment without changing the model.A system prefix also describes the objectives and instructs the model to follow the supplied weights.Source: §3.1 · p. 5

3.2 MINIMAX PREFERENCE OPTIMIZATION WITH A DIRICHLET ADVERSARY

  • The adversary parameterizes preference sampling as Dir(α) and updates its concentration with an unnormalized exponentiated-gradient step that preserves positivity.The bounded Dirichlet family keeps worst-case preferences spread over a neighborhood, enabling stochastic exploration of hard regions.Source: §3.2 · p. 6

3 MAESTRO: MULTI-OBJECTIVE ALIGNMENT VIA END-TO-END STEERING AND ROBUST OPTIMIZATION

  • MAESTRO alternates an online mirror-descent update of the preference distribution toward poorly served preferences with a PPO update of the policy.The procedure uses a Dirichlet adversary rather than fixing the preference distribution in advance.Source: §3 · p. 4, §3.2 · p. 6

3.2 MINIMAX PREFERENCE OPTIMIZATION WITH A DIRICHLET ADVERSARY

  • The adversary scores reward ranks within each batch to balance objective scales without requiring their numeric ranges.The rank statistic is bounded in [0, 1] per objective; finite τ smooths the rank and retains information about response separation.Source: §3.2 · pp. 6–7
  • After updating the distribution, MAESTRO forms a weighted combination of per-objective advantages and applies the clipped PPO loss to update the policy.The per-objective advantages are estimated with generalized advantage estimation.Source: §3.2 · p. 7

4 EXPERIMENTS

The experiments evaluate MAESTRO across assistant and summarization trade-offs, comparing its Pareto fronts with per-preference, post-hoc, and single-model baselines. Results cover benchmark performance, training cost, generalization, and ablations of the adversary and conditioning choices.

4 EXPERIMENTS

  • The experiments use HH-RLHF and BeaverTails for Harmless–Helpful–Humor trade-offs, plus Reddit TL;DR for Summary–Faithful, evaluating Pareto fronts with four metrics.The metrics are hypervolume, inner product, minimum scalarized return, and preference gap.Source: §4 · p. 7

4.1 MAIN RESULTS

  • MAESTRO traces the outermost assistant-task fronts on BeaverTails and HH-RLHF, matching or exceeding MORLHF while dominating RiC and Panacea across the simplex.On BeaverTails, it covers the widest three-objective region among Soup-based methods and RiC; on Humor–Helpful it is comparable to Bone Soups.Source: §4.1 · pp. 7–8, Figure 2 · p. 8, Figure 3 · p. 9

4.2 FURTHER ANALYSIS

  • A trained policy spans 4–5× the range of the base model when evaluated on another objective pair, and MAESTRO remains ahead under reward models never used in training.RiC collapses under the unseen reward models despite training well under the given ones.Source: §4.2 · p. 8
  • 8.95 GPU hours makes MAESTRO the cheapest method on BeaverTails Harmless–Helpful, less than half of any baseline and a tenth of MORLHF, while producing the best fronts.It trains one model in one epoch; Table 3 reports training cost on an A100-80GB.Source: Table 3 · p. 8, §4.2 · p. 8

4.1 MAIN RESULTS

  • MAESTRO reaches a better Summary–Faithful trade-off than baselines and leads on hypervolume, though nearly independent reward scores leave little trade-off to trace.Many preference points are dominated by others on this task.Source: §4.1 · p. 8, Figure 3 · p. 9

4.3 ABLATION STUDY

  • Across three tested fixed Dirichlet choices, no single choice works best on both assistant objective pairs; MAESTRO matches or beats all of them on either pair.The center-concentrated (5, 5) choice is best on Harmless–Helpful but worst on Humor–Helpful.Source: Figure 4 · p. 9, §4.3 · p. 9
  • Replacing the reward surrogate with min–max normalization or changing its sharpness changes the fronts little, while removing the system prefix costs coverage at the extremes.MAESTRO traces the better front at each of the three tested actor learning rates.Source: §4.3 · p. 9

5 CONCLUSION

MAESTRO trains one preference-conditioned policy end-to-end and adapts its training preferences with an adversary. Across three task settings, it achieves the highest hypervolume on all three objective pairs at the lowest compared training cost.

5 CONCLUSION

  • MAESTRO trains one preference-conditioned policy end-to-end and chooses which preferences to train on instead of fixing them in advance.Its Dirichlet concentration is updated by online mirror descent, alternating with PPO, toward preferences the current policy serves worst.Source: §5 · p. 9
  • A per-objective within-batch rank statistic balances objective scales in the adversary update without requiring reward ranges.The policy itself is still trained on the raw scalarized reward.Source: §5 · p. 9
  • Across HH-RLHF, BeaverTails, and summarization, MAESTRO achieves the highest hypervolume on all three objective pairs at the lowest compared training cost.It matches or exceeds methods that train several models.Source: §5 · p. 10

A ADDITIONAL RELATED WORK

  • Preference conditioning is established in multi-objective reinforcement learning, where it amortizes preferences in one model, while C-MORL instead trains a population of policies.The paper presents prompt prefixes as the language-model form of preference input conditioning.Source: A ADDITIONAL RELATED WORK · p. 15
  • CPO conditions on target objective levels, CLP combines parameter replicas, and HoE routes among frozen experts derived from single-objective models.HoE trains only per-preference routers, with prior alignment runs carrying the cost.Source: A ADDITIONAL RELATED WORK · p. 15
  • MODPO retains a separate policy per preference, whereas SIPO mixes per-objective policies at decoding time and fine-tunes on Pareto-dominating responses.These methods avoid the RL step, not reward modeling, and fix preferences before training.Source: A ADDITIONAL RELATED WORK · p. 15

B PROMPT TEMPLATE

  • The preference prefix serializes the sampled vector as a percentage-style natural-language instruction prepended to each user query.For example, weights (0.8, 0.2) become “80% harmless, 20% helpful.”Source: B PROMPT TEMPLATE · p. 16
  • System prefixes explain each objective and tell the model to prioritize higher-weight objectives when they conflict.Assistant and summarization tasks use different system-prefix wording.Source: B PROMPT TEMPLATE · p. 16

C IMPLEMENTATION DETAILS

Implementation details describe baseline reimplementations and tuning, an evaluation choice for Bone Soups, and PPO value estimation across objectives.

C IMPLEMENTATION DETAILS

  • Bone Soups was evaluated without extrapolation because its released implementation tunes extrapolation coefficients on data overlapping the training set.The paper treats that tuning as an additional model-selection step that would make the comparison less clean.Source: C IMPLEMENTATION DETAILS · p. 17
  • Panacea and MO-ODPO were reimplemented in a shared training stack and tuned over learning rates, with additional MO-ODPO sweeps over Dirichlet parameters.Settings were selected on a validation portion split from the dataset’s test split, disjoint from the evaluation portion.Source: C IMPLEMENTATION DETAILS · p. 17
  • PPO uses one value head per objective over a shared LLM backbone, with objective-specific GAE advantages and a combined clipped-policy and value loss.Each value head is trained with a clipped MSE loss.Source: C IMPLEMENTATION DETAILS · p. 17

C.1 EVALUATION METRICS

The evaluation uses Pareto-front metrics and operational reward-model objectives across assistant and summarization tasks. It also reports implementation and dataset details for the experiments.

C.1 EVALUATION METRICS

  • The four Pareto-front metrics are hypervolume, inner product, minimum scalarized return, and preference gap.Hypervolume measures dominated objective-space volume; inner product measures preference-weighted reward alignment.Source: §C.1 · p. 17
  • The hypervolume reference point is set below the component-wise minimum across methods and evaluated preferences by 0.1 times each objective’s range.The offset is specified for numerical stability.Source: §C.1 · p. 17
  • Minimum scalarized return is the evaluated grid’s worst-served preference score, while preference gap is the spread between its best- and worst-served preferences.A small preference gap can also result when the policy ignores preferences, so it is interpreted with minimum scalarized return and the front.Source: §C.1 · pp. 17–18
  • Table 4 summarizes model architecture, training, and evaluation details for the text-generation experiments.The listed implementation details include offline and online learning rates, training steps, and online generation sample size.Source: Table 4 · p. 18, §C.1 · p. 18
  • HH-RLHF and BeaverTails evaluate helpful and harmless responses, while the summary task summarizes Reddit posts.The listed test sets are 8.5K examples for HH-RLHF, 3K for BeaverTails, and 2K for summarization.Source: §C.1 · p. 18

C.2 OBJECTIVES AND THEIR REWARD MODELS

  • Objectives are defined by frozen reward models used identically by every method.The set includes harmlessness, helpfulness, humor, summary quality, and faithfulness, with the humor and faithfulness objectives using classifier logits.Source: §C.2 · p. 19

D.1 QUANTITATIVE RESULTS

MAESTRO leads the reported front-quality comparisons on most BeaverTails and summarization metrics, while reproductions and sensitivity tests examine MO-ODPO, reward normalization, and surrogate sharpness.

D.1 QUANTITATIVE RESULTS

  • MAESTRO ranks first in all four BeaverTails front-quality columns and summarization hypervolume, and third on summarization inner product.It is best on both worst-case metrics for the two BeaverTails pairs; Bone Soups has the best summarization minimum scalarized return and RiC the smallest preference gap.Source: Table 6 · p. 19, Table 5 · p. 19

D.2 ON REPRODUCING MO-ODPO

  • With LoRA on LLaMA-2-7B and reward rescaling, the reproduced MO-ODPO fronts remain collapsed; more epochs advance them slightly without spreading them.The authors attribute the collapse to their setting, noting that the original work reports a spread front with longer training and a larger, fully fine-tuned backbone.Source: §D.2 · p. 20, Figure 5 · p. 20

D.3 MIN–MAX NORMALIZATION IN PLACE OF THE SURROGATE

  • Using min–max reward normalization instead of the surrogate yields close BeaverTails fronts, indicating the surrogate is not what carries the result.The normalization requires each reward’s range to be known in advance.Source: Figure 6 · p. 20

D.4 SENSITIVITY TO THE SURROGATE’S SHARPNESS τ

  • Across four τ settings and three objective pairs, the fronts differ little, with no consistent ordering for two pairs.The authors use τ = 1 throughout, noting that moderate settings space the harmless–helpful trade-off somewhat more evenly than the largest setting.Source: Figure 7 · p. 21, §D.4 · p. 21

D.5 EVALUATION UNDER HELD-OUT REWARD MODELS

MAESTRO retains the outermost Pareto front under held-out reward models on BeaverTails and HH-RLHF, though HH-RLHF’s harmlessness-heavy results are affected by a training-reward pathology. The appendix also examines system-prefix and KL-scaling choices.

D.5 EVALUATION UNDER HELD-OUT REWARD MODELS

  • Under held-out reward scorers, MAESTRO traces the outermost front on both datasets, while RiC falls inside the RL-based methods.The figure compares Harmless–Helpful fronts on BeaverTails and HH-RLHF.Source: §D.5 · p. 21, Figure 8 · p. 21
  • On HH-RLHF, the training reward penalizes refusals of harmless questions while the held-out scorer rewards them, degrading both axes at harmlessness-heavy preferences.This pathology affects every method equally; BeaverTails is the clean held-out comparison because its prompts are genuinely harmful.Source: §D.5 · p. 21

D.6 EFFECT OF PROMPT TEMPLATE

  • The system prefix improves front quality on both BeaverTails objective pairs, especially Humor–Helpful, where the objectives conflict more.Figure 9 compares prompts with and without the objective-describing system message.Source: Figure 9 · p. 22, §D.6 · p. 22

D.7 ON KL REGULARIZATION IN MAESTRO

  • Figure 10 compares MAESTRO fronts with and without KL scaling on HH-RLHF and BeaverTails; scaled runs also multiply the adaptive controller’s target KL by M.Source: Figure 10 · p. 22

D.8 LIMITATION OF BONE SOUPS

The appendix reports Bone Soups’ sensitivity to its interpolation hyperparameter, compares MAESTRO and Bone Soups across actor learning rates, and tests cross-pair transfer by changing the prompt prefix alone.

D.8 LIMITATION OF BONE SOUPS

  • Bone Soups’ Pareto front collapses for some β values, especially β = 0.6, which its original paper suggests as optimal.The authors describe this hyperparameter sensitivity as insufficiently characterized in the original paper.Source: §D.8 · p. 23, Figure 11 · p. 23

D.9 SENSITIVITY TO THE ACTOR LR

  • For each tested actor learning rate, MAESTRO traces a better Pareto front than Bone Soups, although learning rate changes front position and length.The comparison tests 5 × 10^-6, 10^-5 and 5 × 10^-5.Source: Figure 12 · p. 23, §D.9 · p. 23

D.10 GENERALIZATION TO UNSEEN OBJECTIVE PAIRS

  • The cross-pair transfer test evaluates policies trained on one objective pair with the other pair’s prompts, against the base SFT model and an in-domain policy.The figure covers Harmless–Helpful to Humor–Helpful and the reverse on BeaverTails and HH-RLHF.Source: §D.10 · p. 24, Figure 13 · p. 24

D.11 WHERE THE OBJECTIVES BARELY CONFLICT

The appendix links weak objective conflict to short Pareto fronts and reports that prompt-conditioned MAESTRO’s front is comparable to MORLHF’s larger per-preference policy space. It also defines Pareto optimality and the Pareto front.

D.11 WHERE THE OBJECTIVES BARELY CONFLICT

  • Table 7 measures reward-model rank agreement and Spearman correlation on SFT generations; the dialogue objectives disagree more systematically than the summarization pair.Agreement is the fraction of response pairs ordered alike, with 50% as chance.Source: §D.11 · p. 25, Table 7 · p. 25
  • When objectives barely conflict, MAESTRO’s adversary has little preference-dependent weakness to target, and the DeBERTa–Faithfulness front collapses while MORLHF spreads only slightly.Figure 14 shows the extreme case; reward-model ordering agreement is near chance for this pair.Source: §D.11 · p. 25, Figure 14 · p. 25

E.1 PARETO OPTIMALITY

  • A policy is Pareto-optimal if no other policy is at least as good on every objective and strictly better on one; the Pareto front is the objective-space image of the Pareto set.Source: §E.1 · p. 26
  • MAESTRO’s front within one shared policy space is empirically comparable to MORLHF’s front across its larger per-preference policy space.The authors say this suggests prompt-based conditioning does not substantially reduce Pareto-front expressiveness.Source: §E.1 · p. 26

E.2 MINIMAX OPTIMIZATION

The section connects minimax optimality to per-preference performance under uniform realizability, then examines KL scaling, the Dirichlet ambiguity set, and the adversary’s feedback signal.

E.2 MINIMAX OPTIMIZATION

  • Uniform ε-realizability makes a conditioned policy ε-optimal for any ambiguity set and approximately supported Pareto-optimal at strictly positive preferences.When ε = 0, the induced policy is weakly Pareto-optimal.Source: §E.2 · p. 26
  • The converse need not hold, and the guarantee covers only supported Pareto points; matching the full front requires an idealized convex achievable return set.The cited condition is policy-class closure under mixing with strictly positive supporting weights.Source: §E.2 · pp. 26–27

E.3 KL REGULARIZATION IN MAESTRO

  • For symmetric preference sampling, matching each objective’s baseline reward tilt requires βours = βb/M; conditioned-policy scaling is a compromise, and reported gains may partly reflect a higher target KL.The scaling correction relies on incomplete conditioning and is not a bound.Source: §E.3 · p. 28

E.4 THE AMBIGUITY SET AND HOW MUCH IT SMOOTHS

  • Bounded-concentration Dirichlet distributions smooth the inner objective, while increasing αmax makes it approach the pointwise worst preference under continuity.Restricting the adversary to this family replaces the pointwise worst case with a distributional one.Source: §E.4 · pp. 28–29
  • The adversary concentrates on conflicting interior trade-offs rather than simplex vertices, where scalarized return is higher.This is the reported pattern in Figure 2(a).Source: §E.4 · p. 29

E.5 WHAT THE ADVERSARY IS SCORED ON

  • The policy is trained on scalarized reward, but the adversary uses a batch-relative surrogate; its update is a stochastic semi-gradient heuristic, not an unbiased gradient of the minimax objective.Outside exact simultaneous realizability, the heuristic shifts sampling toward preferences receiving lower batch-relative feedback.Source: §E.5 · pp. 29–30
Loading 2610.04845v1…