PaperlayerView exact revision on arXiv

Source-linked AI summary

MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment

Tony Tu, Sayan Chakraborty, Ruomeng Xu, Tony Qin, Austin Tian

arXiv:2608.14828v1cs.AIcs.CLcs.LG

TL;DR

Multi-objective preference training can sacrifice weaker objectives when rewards are combined additively. MINT instead selects candidates by their weakest objective before unchanged DPO distillation, improving both objectives and reducing imbalance across two dialogue domains.

  • Problem

    Additive scalarization can let gains on one objective offset losses on another, producing lopsided responses despite improved aggregate reward.

  • Method

    MINT selects the best-of-K candidate with the strongest weakest objective and distills the resulting preference pair with unchanged Direct Preference Optimization.

  • Results

    Across emotional support and adversarial negotiation, MINT improves both objectives while reducing imbalance; on emotional support, weaker-axis reward min(AQ, EQ) rises from 0.37 to 0.64.

  • Takeaways & Limitations

    MINT’s balancing benefit tracks reference-policy imbalance and persists through interaction while that imbalance remains.

Abstract

from arXiv · show

Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p -> negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p < 10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.

1 Introduction

MINT addresses multi-objective alignment failures caused by additive aggregation by selecting candidates according to their weakest objective while leaving preference distillation unchanged. Across emotional support and price negotiation, it improves both objectives and reduces imbalance.

  • Conflicting objectives are conventionally combined into a weighted sum, which can favor aggregate improvement without ensuring balanced behavior.
  • MINT selects the candidate whose weakest objective is strongest and distills that preference through the standard best-of-K DPO pipeline.The rejected candidate is the most lopsided response.
  • The method changes candidate selection rather than the DPO objective, reward model, or sampling procedure, enabling composition with preference-optimization variants.
  • MINT is evaluated in cooperative emotional support and adversarial price negotiation using action quotient and emotional quotient objectives scored by a frozen language-model judge.
  • Across both dialogue domains, min-selection improves both objectives while substantially reducing their imbalance.

2 Related Work

Prior work established preference optimization through DPO and developed methods for multi-objective scalarization and best-of-K distillation. MINT builds on best-of-K preference distillation while replacing single-reward candidate ranking with min-selection.

  • Preference optimization: DPO converts reward-model-plus-RLHF alignment into a single supervised objective on preference pairs, avoiding explicit reward modeling and reinforcement learning.DPO treats the language model as its own implicit reward model and has motivated variants such as IPO and KTO.
  • Multi-objective alignment: Multi-objective alignment commonly scalarizes conflicting objectives with weighted sums, including multi-objective RLHF, MODPO, and Rewarded Soups.The cited objectives include helpfulness versus harmlessness and informativeness versus conciseness.
  • Rejection sampling and best-of-K distillation: Rejection-sampling methods improve policies by sampling candidates, scoring them, and fine-tuning on reward-ranked best-of-K selections.RAFT and ReST distill selected candidates back into the policy, while statistical rejection sampling connects this selection to preference optimization.
  • Rejection sampling and best-of-K distillation: Best-of-N sampling is a strong inference-time baseline, while BOND distills its behavior into one model to remove inference-time cost.MINT is positioned within this broader best-of-K preference-distillation paradigm.
  • Rejection sampling and best-of-K distillation: MINT generates K candidates, scores them with a frozen judge, and distills the result through DPO, differing from prior single-scalar-reward ranking.The passage identifies MINT as an instance of best-of-K preference distillation; its ranking rule is introduced as the contrast with prior work.

3 Method

MINT trains a multi-objective dialogue policy by selecting preference pairs according to the weakest objective rather than an additive reward, then distilling those pairs with the unchanged DPO objective. It uses the p → −∞ limit of generalized-mean scalarization, so the chosen response maximizes min(AQ, EQ) and the rejected response minimizes it.

  • Pipeline: MINT samples candidate responses, scores each objective with a frozen judge, selects a chosen/rejected pair using a balance-seeking rule, and applies DPO.The pipeline’s only departure from standard best-of-K DPO is its preference-pair selection rule.
  • Objective representation: The method handles m interpretable objectives scored on [0, 1], using AQ for task progress and EQ for relational quality in its two-objective instantiation.MINT is agnostic to both the number and semantics of objectives.
  • Motivation: Naive additive optimization can improve aggregate reward while degrading the weaker objective, so MINT addresses objective imbalance during preference selection.A surplus on one axis can compensate for a deficit on the other under summation, allowing lopsided responses to be selected.
  • Scalarization: Generalized-mean scalarization spans additive selection to worst-case selection, with p → −∞ yielding M−∞(a, b) = min(a, b).As p decreases, the scalarization becomes increasingly dominated by the smaller objective coordinate.
  • Min-selection: Under min-selection, the chosen response has the strongest weaker objective and the rejected response is the most lopsided candidate; ties are broken by total reward.Candidates are ranked by the ordered key min(a, b), a + b.
  • Training setup: Experiments use p → −∞(min), a single offline round R=1, and m=2 objectives, with candidates generated, scored, and distilled once.The reference policy is also used for candidate generation and DPO comparison.

4 Results

Across held-out emotional-support and negotiation evaluations, MINT improves the weaker objective while reducing objective imbalance. Its advantage persists throughout rollouts when the reference remains imbalanced, but attenuates when the reference recovers or is balanced.

  • Static held-out results: 0.37 to 0.64: on ESConv, MINT raises min(AQ, EQ) relative to SFT while improving AQ from 0.53 to 0.72 and EQ from 0.47 to 0.71.The weaker-axis improvement is +0.27, with Wilcoxon signed-rank p ≈3 × 10^-43 and higher scores on 79% of matched turns.
  • Static held-out results: 0.583 to 0.657: in CraigslistBargain, MINT improves min(AQ, EQ) over the base model while tightening the objective gap from 0.177 to 0.135.This DPO-only negotiation result has Wilcoxon p = 0.0002 across n = 103 turns and shows the balancing effect without an intermediate SFT stage.
  • Interactive rollout results: +0.235 to +0.145: against the ESConv SFT reference, MINT’s min(AQ, EQ) advantage persists from the opening turn through full ten-turn rollouts.The advantage is significant throughout, with Wilcoxon p < 10^-4, across approximately 280 turns and 30 rollouts per policy.
  • Interactive rollout results: +0.077, +0.035, . . . , +0.001: in CraigslistBargain rollouts, MINT’s advantage decays with depth and ends indistinguishable from the base at ∆min = +0.001, p=0.43 pooled.The reference seller is already balanced under the prompt, so it recovers or maintains the weaker axis without sustained MINT gains.
  • Interactive rollout results: MINT’s weaker-objective improvement is proportional to reference-policy imbalance and persists only while that imbalance remains along the trajectory.The ESConv SFT reference is badly and increasingly imbalanced, whereas the ESConv base regains achievement and the CraigslistBargain base is balanced from the outset.
  • Method and evaluation: MINT evaluates candidates by min(AQ, EQ) rather than a weighted sum, favoring stronger and more balanced policies.The minimum-selection rule scores each pair on its weaker axis, preventing gains on an easy objective from offsetting losses on a hard one.

Appendix Contents · A Training Details · A.1 Emotional Support DPO Training

The ESConv DPO appendix describes a merge-then-adapt training setup with min-selected preference pairs and tracks training dynamics through early stopping, preference accuracy, and reward margins. Validation loss flattens near step 130, preference accuracy plateaus around 0.84, and reward margins grow steadily without divergence.

  • A.1 Emotional Support DPO Training: Roughly step 130 marks where validation DPO loss decreases monotonically and flattens.The checkpoint with the lowest validation loss is selected under early stopping.
  • A.1 Emotional Support DPO Training: About 0.84 is the plateau reached by held-out-pair preference accuracy after rising from near-chance.The mean reward margin between chosen and rejected responses grows steadily without diverging.
  • A.1 Emotional Support DPO Training: Min(AQ, EQ) selection with a sum tie-break forms the ESConv preference pairs.The selection follows the procedure specified in Section 3.4.
  • A.1 Emotional Support DPO Training: Llama-3.1-8B-Instruct provides the ESConv policy backbone.A supervised SFT checkpoint serves as the reference policy πref.
  • A.1 Emotional Support DPO Training: 41.9M trainable parameters comprise 0.52% of the model in the fresh low-rank DPO adapter.The adapter uses rank 16, α=32, and dropout 0.05.
  • A.1 Emotional Support DPO Training: Disabling the adapter exactly recovers the SFT reference policy.This property follows from merging the SFT adapter into the base weights before training a fresh DPO adapter.

A.2 Price-Negotiation DPO Training

Price-Negotiation DPO training uses a Gemma-3-27B policy with a 4-bit QLoRA adapter and the adapter-disabled quantized model as the exact reference. Training shows stable validation optimization, rising held-out preference accuracy, and a steadily increasing reward margin.

  • Training setup: The policy is trained from Gemma-3-27B-Instruct with 4-bit QLoRA, using the adapter-disabled quantized base model as πref.The DPO adapter covers all attention and MLP projections, with rank 8, α=16, dropout 0.05, and 56.8M trainable parameters.
  • Training dynamics: Held-out preference accuracy rises from about 0.76 to a plateau near 0.84.Validation loss falls sharply during the first ∼50 steps before flattening near its minimum.
  • Training dynamics: The mean reward margin grows steadily to roughly 5.8, while smoothly decreasing validation loss indicates stable optimization rather than reward over-optimization.The noisier training loss is attributed to 4-bit quantization and a small effective batch, consistent with adversarial, sharply separable price-negotiation preferences.

B Reward Rubrics and Judge Configuration

A single frozen Llama-3.1-8B-Instruct judge produces independently scored achievement and emotional rewards using domain-specific rubrics. Probe-turn results support treating min(AQ, EQ) and |AQ−EQ| as balance measures rather than proxies for overall quality.

  • B.1 Frozen judge model: A single frozen Llama-3.1-8B-Instruct judge supplies every reward, remaining unchanged during sampling, pair construction, training, and evaluation.The fixed judge keeps the reward signal identical across training and evaluation conditions.
  • B.1 Frozen judge model: Each axis reward is a weighted average of criterion scores, with scores clamped to [0, 1] and the same prompts and weights used throughout.Within comparisons, candidate scoring and both evaluation modes use an identical reward model.
  • B.1 Frozen judge model: The judge scores the two axes independently, using divergent worked examples so pleasantness or effectiveness alone cannot inflate both scores.This orthogonality makes min(AQ, EQ) a balance objective rather than a proxy for overall quality.
  • B.2 Emotional support (ESConv): For ESConv, AQ measures actionable, specific support, whereas EQ measures validation, warmth, and timing; premature advice is penalized only under EQ.A concrete but ill-timed suggestion can therefore be high-AQ/low-EQ rather than globally downgraded.
  • B.3 Negotiation (CraigslistBargain): For CraigslistBargain, AQ measures seller-favorable price and agreement progress, while EQ measures language warmth independently of deal success.The adversarial rubric requires curt successful closes to score low EQ and warm concessions to score high AQ.
  • B.4 Axis independence in practice: AQ ≈0.65 and EQ ≈0.25 for a curt, effective negotiation close demonstrate the judge assigns high AQ and low EQ to maximally divergent turns.Warm capitulation receives the reverse pattern, and weak cross-axis correlation supports min(AQ, EQ) and |AQ−EQ| as balance measures.
Loading 2608.14828v1…