Source-linked AI summary
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang
TL;DR
Scalar-reward RLHF struggles to represent diverse and conflicting user preferences. DPA combines multi-objective rewards with unit-vector preference conditioning and preference-conditioned RSF, achieving arithmetic helpfulness–verbosity control on Mistral-7B while remaining competitive with DPO.
Problem
Scalar-reward RLHF may inadequately capture complex, conflicting, and user-dependent human preferences, often favoring an average-user preference.
Method
DPA trains multi-objective rewards and encodes user preferences as unit vectors for preference-conditioned alignment using rejection-sampling fine-tuning.
Results
DPA achieves arithmetic control over helpfulness–verbosity preferences on Mistral-7B while maintaining competitive performance with DPO.
Takeaways & Limitations
A single DPA-aligned LLM can accommodate varying user trade-offs through numerically specified preference directions.
Takeaways & Limitations
DPA relies on a robust multi-objective reward model; reward-model bias or missed harmful content can propagate through fine-tuning and inference.
Abstract
from arXiv · showhide
Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).
1. Introduction
LLMs must align with diverse human preferences, but scalar-reward RLHF can miss conflicting objectives and user-dependent needs. DPA addresses this by representing multiple rewards and encoding preferences as directions, enabling arithmetic control over helpfulness and verbosity.
- LLM assistants must align with preferences including helpfulness, honesty, harmlessness, and verbosity.
- Scalar-reward RLHF can produce verbose helpful responses and converge toward average-user preferences, overlooking conflicting user needs.
- Multi-objective reward models represent preference dimensions such as helpfulness, verbosity, and harmlessness, but linear combinations may remain user-dependent and inference-unfriendly.
- DPA lets users specify a unit preference vector that controls a single LLM’s reward trade-off during generation.
- DPA evaluations on Mistral-7B show arithmetic control over helpfulness–verbosity trade-offs while maintaining competitive performance with DPO.
2. Directional Preference Alignment
DPA extends RLHF with multi-objective rewards and direction-based user preferences, enabling one LLM to optimize different arithmetic trade-offs. Its preference-conditioned RSF procedure samples and fine-tunes responses for specified directions while avoiding the storage overhead of maintaining separate models.
- Directional Preference Alignment: DPA represents each response with multiple objective rewards and models user preferences as unit-vector directions in that reward space.The reward model rates prompt-response pairs across attributes such as helpfulness, correctness, and verbosity.
- Directional Preference Alignment: For a preference direction v, the LLM conditions generation on both the prompt x and v, producing responses optimized for that user-dependent trade-off.The targeted user population is represented by a distribution over preference directions, and training maximizes expected reward under that distribution.
- Comparison with Existing Methods: The comparison framework evaluates RLHF algorithms by multi-objective rewards, preference arithmetic, single-model support, and feasibility guarantee.These criteria describe whether methods support multiple objectives, arithmetic preference control, one model for different preferences, and reachable control vectors.
- Reward Optimization via Rejection Sampling: DPA uses preference-conditioned Rejection Sampling Fine-tuning: it samples responses, ranks them by direction-weighted reward, selects the best, and fine-tunes iteratively.RSF is used instead of PPO because the paper characterizes it as simpler, more stable, and comparably rewarding for LLM alignment.
- Comparison with Existing Methods: Unlike SteerLM’s direct reward-vector prompting, DPA avoids uncontrolled responses when a requested reward vector is infeasible for a particular prompt.The cited example is jointly demanding high helpfulness and very low verbosity for a Romeo and Juliet summary.
- Comparison with Existing Methods: Unlike reward-soup methods, DPA uses a single LLM rather than maintaining k objective-specific LLMs and interpolating them at inference.Reward soups combine the weights of k models for each newly assigned combination vector, creating storage and computation overhead.
3. Empirical Results
Experiments on Mistral-7B evaluate DPA’s arithmetic control over helpfulness–verbosity preferences. DPA Pareto-dominates strong baselines on validation rewards and remains competitive with DPO on AlpacaEval-2.0.
- Experimental setup: DPA targets arithmetic control over helpfulness and verbosity preferences on Mistral-7B.The experiments use these two reward objectives to assess preference-conditioned generation.
- Reward modeling: The multi-objective reward model uses ten output dimensions from HelpSteer’s five attributes and UltraFeedback.It is trained on the union of HelpSteer and UltraFeedback, initialized from Mistral-7B.
- Directional preferences: The helpfulness–verbosity directions range from pure helpfulness toward less verbosity on the unit circle.The preference-conditioned reward is v1r1 + v2r2, with the directional distribution designed around the positive correlation between helpfulness and verbosity.
- Alignment procedure: DPA samples directional preferences and selects the highest-reward response among 16 generated responses per preference during iterative rejection sampling.The selected responses are used for preference-conditioned fine-tuning.
- Validation rewards: For t ≥1, DPA Pareto-dominates SFT, DPO, and SteerLM, while later DPA iterations Pareto-dominate earlier ones.The expanding empirical front indicates improved rewards across the considered user preferences.
- External evaluation: On AlpacaEval-2.0, DPA outperforms SteerLM and achieves competitive performance against DPO while retaining arithmetic preference control.The authors attribute discrepancies from validation rewards to differences between their reward model and GPT-4-turbo.
4. Related Works
Related work spans general LLM development, RLHF algorithm design, and fine-grained preference representation. These studies motivate multi-objective alignment while leaving room for DPA’s preference-conditioned approach.
- Large Language Models: LLMs have advanced across classification, generation, and complex reasoning through extensive pre-training on large datasets.The related-work discussion distinguishes closed-source and other model settings.
- RLHF Algorithmic Designs: PPO is predominant in RLHF but is less efficient and stable than supervised fine-tuning and sensitive to implementation choices.These properties make achieving strong PPO performance challenging in practice.
- RLHF Algorithmic Designs: DPO has attracted attention for its simplicity, stability, and effectiveness, while the paper leaves algorithmic designs beyond RSF to future work.The authors note that such algorithmic ideas could also be incorporated into DPA.
- Fine-grained Preference Representation: Prior multi-objective methods represent different preference aspects but commonly combine rewards in a fixed way.The paper positions DPA as addressing this limitation through preference-conditioned alignment.
5. Limitations
DPA depends on the quality of its multi-objective reward model. Errors or bias in that model can propagate through fine-tuning and affect generated content.
- Reward-model dependence: DPA’s efficacy is intrinsically linked to the precision and discriminative capability of its multi-objective reward model.The framework may amplify shortcomings when the reward model misses preference subtleties or has biased reward distributions.
- Safety boundary: If the reward model fails to recognize harmful content, DPA could lead the aligned model to produce it during inference.This limitation makes harmful-content recognition an explicit dependency of the framework.
6. Conclusion
DPA represents multidimensional user preferences with high-dimensional preference vectors and explores the Pareto front. On Mistral-7B, it yields a more effective helpfulness–verbosity trade-off than strong baselines such as DPO.
- Conclusion: DPA incorporates multidimensional user preferences through a high-dimensional preference vector.This addresses limitations of conventional scalar reward models by representing conflicting preferences in a multidimensional space.
- Conclusion: DPA efficiently explores the Pareto front and achieves a more effective helpfulness–verbosity trade-off on Mistral-7B than strong baselines such as DPO.The conclusion frames this as the paper’s demonstrated alignment outcome.