PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning to Thousands of Preferences via System Message Generalization

Seongyun Lee, Sue Hyun Park, Seungone Kim, Minjoon Seo

arXiv:2405.17977v2cs.CL

TL;DR

The paper addresses the difficulty of scaling individualized LLM alignment beyond generic public preferences and costly per-user retraining. It trains a model on diverse preference-bearing system messages, then reports gains on unseen preference prompts and general helpfulness benchmarks.

  • Problem

    Generic alignment often centers on broadly shared values, while individualized preference alignment can require repeated data collection and model retraining.

    Source: §1 · pp. 1–2

  • Method

    The authors construct the Multifaceted Collection of preference-bearing system messages and train JANUS to generalize across unseen preference instructions.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    JANUS achieves 75.2%, 72.4%, and 66.4% tie+win rates against Mistral 7B Instruct v0.2, GPT-3.5-Turbo-0125, and GPT-4-0613, respectively, and the authors report strong general helpfulness and harmlessness.

    Source: §5.1 · p. 6, §7 · p. 10

  • Takeaways & Limitations

    The authors conclude that system-message generalization can support majority and individual preferences without per-user fine-tuning.

    Source: §7 · p. 10

    The synthetic training data cannot be fully evaluated by humans, and flexible preference specification may increase vulnerability to malicious attacks such as jailbreaking.

    Source: §7 · p. 10

Abstract

from arXiv · show

Although humans inherently have diverse values, current large language model (LLM) alignment methods often assume that aligning LLMs with the general public's preferences is optimal. A major challenge in adopting a more individualized approach to LLM alignment is its lack of scalability, as it involves repeatedly acquiring preference data and training new reward models and LLMs for each individual's preferences. To address these challenges, we propose a new paradigm where users specify what they value most within the system message, steering the LLM's generation behavior to better align with the user's intentions. However, a naive application of such an approach is non-trivial since LLMs are typically trained on a uniform system message (e.g., "You are a helpful assistant") which limits their ability to generalize to diverse, unseen system messages. To improve this generalization, we create the Multifaceted Collection, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions. Using this dataset, we train a 7B LLM called Janus and test it on 921 prompts from 5 benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, and Self-Instruct) by adding various unseen system messages that reflect user preferences. Janus achieves tie+win rate of 75.2%, 72.4%, and 66.4% against Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4, respectively. Unexpectedly, on three benchmarks focused on response helpfulness (AlpacaEval 2.0, MT-Bench, Arena Hard Auto v0.1), Janus also outperforms LLaMA 3 8B Instruct by a +4.0%, +0.1%, +3.0% margin, underscoring that training with a vast array of system messages could also enhance alignment to the general public's preference as well. Our code, dataset, benchmark, and models are available at https://github.com/kaistAI/Janus.

1 Introduction

The paper argues that alignment to generic helpfulness and harmlessness misses diverse, context-dependent preferences, while per-user retraining is costly. It proposes training on diverse preference-bearing system messages and reports strong performance on unseen preferences and general-helpfulness benchmarks.

1 Introduction

  • Generic preference data can overlook individual conflicts, contribute to annotation disagreement, and inadvertently encourage undesirable traits such as verbosity.The paper contrasts diverse, context-dependent human preferences with alignment data centered on broadly shared values such as helpfulness and harmlessness.Source: §1 · p. 1
  • Prior personalized RLHF approaches require repeated preference-data collection and reward-model and LLM retraining, which may not scale in time, storage, and training cost.The paper asks whether diverse preferences can be supported without retraining an LLM for each user.Source: §1 · p. 2
  • The authors propose training on diverse system messages that express individual preferences so models can adapt to novel preferences at test time.This targets a limitation they associate with open-source models commonly trained using a generic system message.Source: §1 · p. 2
  • The Multifaceted Collection contains 197k system messages and three nonoverlapping-value variants per instruction, with responses for each variant.The dataset is described as representing individual multifaceted preferences; the paper reports quantitative and qualitative analyses of its preference variety.Source: §1 · p. 2
  • 74.8%, 70.8%, and 57.9% win rates against Mistral 7B Instruct v0.2, GPT-3.5-Turbo-0125, and GPT4-Turbo-0125, respectively, were reported in human evaluation.Evaluation used 315 instructions from five benchmarks, augmented with three human-verified, unseen context-specific system messages per instruction.Source: §1 · p. 2
  • JANUS reportedly averages 4.24 out of 5.0 in GPT-4 evaluation and exceeds LLaMA 3 8B Instruct, Mixtral 8x7B Instruct v0.1, and GPT-3.5-Turbo-0125 by 5.7% on average.The authors also report higher helpfulness than LLaMA 3 8B Instruct on three benchmarks and say diverse-preference training remains robust without test-time system messages.Source: §1 · p. 2

2 Related work

Related work addresses individualized alignment through preference-aware reward modeling and policy training, and uses system messages or instruction tuning to steer model behavior. The paper distinguishes its approach by scaling system messages to encode multifaceted preferences separately from task instructions.

Personalized RLHF for aligning to diverse preferences

  • Personalized RLHF work models preference distributions, trains preference-specific reward or user models, or merges policies, but typically retrains models to adapt to new preferences.The paper proposes instead an LLM that conforms to preferences when they are explicitly stated at test time.Source: §2 · p. 3

Utilizing system messages for context

  • System messages can encode roles or personalization, but prior studies examined relatively few message types and domains, limiting coverage of human preferences.The paper scales system messages into multifaceted variants for the same instruction.Source: §2 · p. 3

Instruction tuning

  • Unlike instruction tuning that embeds task instructions in user prompts, this approach uses system messages as meta-instructions encoding user values separately from tasks.The authors state that they know of no publicly released training dataset with this motivation and meta-instruction content.Source: §2 · p. 3

3 MULTIFACETED COLLECTION for scalable individualized alignment

The Multifaceted Collection represents preference as combinations of explicit, multidimensional values expressed in system messages. Its synthesis pipeline scales these messages and paired responses across many instructions.

3.1 Mapping a preference to multidimensional values

  • The design addresses multifacetedness by covering many possible facets and explicitness by stating the rationale for desirable responses.The paper frames these as requirements for representing diverse preferences and helping models discriminate between chosen and rejected responses.Source: §3.1 · p. 4
  • A preference set combines values from different dimensions, while verbalized value descriptions make desirable response qualities explicit in a system-message meta-instruction.The value hierarchy branches from general dimensions to specific subdimensions and values, supporting many facets and combinations.Source: §3.1 · p. 4

3.2 Instruction data construction

  • The pipeline starts from 66k sampled and filtered instructions, then generates three task-aligned preference sets per instruction using GPT-4-Turbo-0125.Seed values cover style, background knowledge, informativeness, and harmlessness, organized into 18 subdimensions and 107 values.Source: §3.2 · pp. 4–5
  • GPT-4-Turbo-0125 converts each preference set into a system message and generates its gold response; the dataset contains 197k responses and 66k chosen/rejected pairs.The rejected response is selected from the other two responses for the instruction.Source: §3.2 · p. 5
  • The paper reports low similarity among preference sets for each instruction and describes further analyses of system-message quality and safety.The stated quality and safety analyses are in Appendix E.Source: §3.2 · p. 5

4 Experimental setup

The experimental setup evaluates trained JANUS variants on preference-conditioned benchmarks, general helpfulness, and harmlessness. Assessments use human and model judges, with comparisons against open-source and proprietary baselines.

Training

  • JANUS 7B uses all 197k training instances; JANUS* 7B uses one third, with DPO and ORPO variants and a reward model for Best-of-N sampling.All models are based on Mistral 7B v0.2 and use its chat template to incorporate system messages.Source: §4 · p. 5

Benchmarks

  • The Multifaceted Bench contains 921 human-validated instances drawn from five benchmarks, with preference-specific system messages, reference answers, and scoring rubrics.The initial 945 instances were reduced after filtering 2.5% substandard samples.Source: §4 · p. 5
  • Helpfulness evaluation uses AlpacaEval 2.0, MT-Bench, and Arena Hard Auto v0.1, while harmlessness evaluation includes toxicity, fluency, diversity, and social-bias benchmarks.The harmlessness tests include RealToxicityPrompts, Winogender, CrowS-Pairs, and BBQ.Source: §4 · pp. 5–6

Baselines

  • Baselines include pretrained and instruction-tuned open-source models as well as GPT-3.5-Turbo-0125, GPT-4-0613, and GPT-4-Turbo-0125.For helpfulness and harmlessness, the authors also retrieve results from official leaderboards or prior papers.Source: §4 · p. 6

Evaluation settings

  • Human pairwise judgments and GPT-4-Turbo-0125 absolute scores evaluate the Multifaceted Bench; other benchmarks use model scoring, Elo comparisons, or the Perspective API.Helpfulness scores use 0–10 scales on AlpacaEval 2.0 and MT-Bench; Arena Hard Auto v0.1 uses Elo scores.Source: §4 · p. 6

5 Experimental results

Experiments evaluate JANUS on unseen preference instructions, general helpfulness, harmlessness, and response diversity. JANUS performs competitively on preference-conditioned evaluation and retains strong general-purpose performance.

5.1 Unseen multifaceted preference

  • 75.2%, 72.4%, and 66.4% tie+win rates against Mistral 7B Instruct v0.2, GPT-3.5-Turbo-0125, and GPT-4-0613, respectively.Reported rates include ties only when both responses were judged good.Source: §5.1 · p. 6
  • On MULTIFACETED BENCH, JANUS scores +33% over Mistral 7B v0.2 and +4.6% over Mistral 7B Instruct v0, while slightly trailing LLaMA 3 Instruct 70B.These are LLM-as-a-Judge evaluation comparisons.Source: §5.1 · p. 7
  • Best-of-64 sampling raises average scores from 4.14 to 4.28 for JANUS* 7B and from 4.24 to 4.31 for JANUS+DPO 7B.JANUS-RM 7B supplies the sampling scores.Source: §5.1 · p. 7

5.2 General helpfulness

  • JANUS beats LLaMA 3 8B Instruct by +4%p in AlpacaEval 2.0 LC win rate, scores 7.7 on MT-Bench, and 20.9 on Arena Hard Auto v0.1.The paper reports strong performance across its general helpfulness benchmarks.Source: §5.2 · pp. 7–8

5.3 Diversity and harmlessness

  • JANUS has lower average maximum toxicity and toxicity probability than other instruction-tuned models while retaining high fluency and diversity.The paper also reports moderate performance on social-bias benchmarks.Source: §5.3 · p. 8

6 Analysis

The analysis examines response length and diversity, robustness without preference messages at test time, and the training contribution of multifaceted data.

Verbosity of personalized responses

  • JANUS responses are longer on average than several baselines, but their length distribution closely matches reference answers.JANUS also outperforms Mistral 7B Instruct v0.2 and GPT-3.5 Turbo on verbosity-debiased AlpacaEval 2.0.Source: §6 · p. 9

Diversity of personalized responses

  • JANUS has the lowest average ROUGE-L similarity among its responses to the same instruction under three distinct system messages.The authors interpret this as evidence that it generates varying responses for different contexts.Source: §6 · p. 9

Effect of excluding multifaceted system messages at test time

  • Removing multifaceted system messages at test time causes only a minor performance drop for JANUS, while Mistral models decline substantially.GPT-4-Turbo-0125 remains the highest-scoring model with or without the message.Source: §6 · p. 9

Benefit of incorporating multifacetedness during training

  • Training with multifaceted system messages improves both multifacetedness and helpfulness, whereas default messages improve helpfulness at the cost of reduced multifacetedness.Multifaceted responses without explicit system-message supervision improve some but underperform models trained with those messages.Source: §6 · pp. 9–10

7 Discussion

The discussion presents system-message preference guidance as a route to personalized responses without per-user fine-tuning, while identifying unresolved safety and application challenges.

Discussion

  • JANUS models adapt to unseen multifaceted preferences while retaining general helpfulness and harmlessness, supporting alignment to individual and majority preferences without per-user fine-tuning.The approach uses system messages to guide responses tailored to nuanced user preferences.Source: §7 · p. 10

Broader impact of generalizing system messages instead of user messages

  • System-message preferences could apply across relevant queries, reducing repeated user effort and allowing common needs to be reflected automatically.The discussion also identifies implicit preference handling as a potential route to a more seamless chat experience.Source: §7 · p. 10

Limitations and future work

  • Synthetic training data cannot be fully human-evaluated, preference flexibility may increase jailbreak vulnerability, and managing implicit preferences remains application-dependent and out of scope.The authors include harmlessness in every system message and use data-generation safeguards.Source: §7 · p. 10

A Additional results on helpfulness

JANUS remains competitive on MT-Bench multi-turn questions against the two reported baselines.

A Additional results on helpfulness

  • 6.8 on multi-turn MT-Bench questions, compared with 6 for Mistral 7B Instruct v0.2 and 6.6 for LLaMA-3 8B Instruct.The authors report JANUS consistently outperforming both baselines in this setting.Source: A Additional results on helpfulness · p. 18

B Additional results on harmlessness

JANUS shows no striking social-bias issues relative to the compared models, and its answer distributions remain closer to human and pre-trained distributions than Mistral 7B Instruct v0.2’s. Fine-tuning still increases distance from human distributions, but JANUS’s similarity drop is smaller.

B Additional results on harmlessness

  • JANUS shows bias similar to LLaMA 3 8B Instruct and less than Mistral 7B Instruct v0.2 except on BBQ.The authors report no striking bias issues compared with the other models.Source: B Additional results on harmlessness · p. 18

C.1 Representativeness of the human population

  • The analysis compares model and human answer-choice distributions on GlobalOpinionQA and MPI using Jensen-Shannon distance averaged over five runs per prompt.The reported human populations are from the US and Japan.Source: §C.1 · p. 18, Figure 7 · p. 19
  • JANUS has a 2x smaller drop in similarity to human distributions than Mistral 7B Instruct v0.2, although aligned models become more distant from humans after fine-tuning.Its entropy is also described as closer to the pre-trained and human distributions than Mistral 7B Instruct v0.2’s.Source: §C.1 · pp. 18–19, Figure 8 · p. 19

C.2 Additional ablation studies

The ablations indicate that base-model capability and training-data volume affect performance, while hierarchical preference generation offers control over instruction-relevant values. JANUS responses are also reported to follow specific preferences in qualitative examples.

C.2 Additional ablation studies

  • On ten samples, freely generated preferences were sometimes irrelevant to the instruction or gave away solutions, whereas hierarchical generation provides control over user-side expectations.The authors base their dimensions on style, background knowledge, informativeness, and harmlessness, with seed examples helping produce individualized preferences.Source: §C.2 · pp. 19–20
  • Increasing the collection from 66k to 132k to 197k instances improves scores across five benchmarks.The 197k version pairs each prompt with three systems and responses; the smaller versions use two or one.Source: §C.2 · p. 20, Table 10 · p. 20
  • LLaMA 3 8B outperforms LLaMA 2 13B, while scaling LLaMA 2 from 7B to 13B improves benchmark performance.LLaMA 2 7B performs similarly to or slightly worse than Mistral 7B v0.2; the authors conclude that base-model size and capability influence performance.Source: §C.2 · p. 20, Table 11 · p. 20

C.3 Response quality

  • JANUS 7B’s qualitative examples show closer adherence than GPT-4 to a system message requesting a straightforward, logic-based response.The paper says GPT-4’s response was not entirely incorrect but did not adhere as closely to the preference.Source: §C.3 · p. 20

D Details of MULTIFACETED COLLECTION construction

  • The collection draws from five preference datasets, filters existing preference-bearing system messages, and retains 65,653 unique instructions.The sources are Chatbot Arena Conversations, DSP, UltraFeedback-binarized-clean, Nectar, and OpenHermesPreferences.Source: D Details of MULTIFACETED COLLECTION… · pp. 20–21
  • The seed values comprise four dimensions, 18 subdimensions, and 107 value descriptions.The full keyword list is reported in Table 13.Source: D Details of MULTIFACETED COLLECTION… · p. 21

E Analysis of MULTIFACETED COLLECTION

The collection’s preference descriptions show low overlap, and system messages receive favorable ratings for relevance, specificity, coherence, and naturalness. The benchmark construction includes human quality checks and filtering, while its final set retains 921 samples.

E Analysis of MULTIFACETED COLLECTION

F Ablation data: default system messages and generally helpful responses

  • For a generally helpful-response ablation, the authors replace each prompt’s three multifaceted system messages with randomly sampled default messages and generate responses with GPT-4-Turbo-0125.The default messages are sourced from FastChat.Source: F Ablation data: default system… · p. 24, Table 15 · p. 24

G Details of MULTIFACETED BENCH construction

  • The benchmark starts from 315 unique instructions drawn from five benchmarks, with three generated system messages and aligned reference answers per instruction.These messages use preferences absent from the collection to reduce potential data contamination.Source: G Details of MULTIFACETED BENCH… · pp. 24–25
  • Human evaluators initially rated 70% of reference answers high quality, while 23.7% followed the user instruction but not the system message; 77.2% of rubrics were considered well-constructed.After excluding 24 bad samples from 945, the final MULTIFACETED BENCH contains 921 samples.Source: G Details of MULTIFACETED BENCH… · p. 25, Figure 10 · p. 25

H Analysis of MULTIFACETED BENCH

MULTIFACETED BENCH has low train-test similarity in its system messages, supporting its use to assess unseen preferences. The appendices describe the training, evaluation, and annotation procedures used to analyze the benchmark and model responses.

H Analysis of MULTIFACETED BENCH

J.1 Instruction tuning and preference tuning

  • JANUS* uses one-third of the collection for instruction tuning, while JANUS uses all of it; DPO and ORPO training use aligned responses as chosen and non-aligned responses as rejected.The authors also train a reward model for Best-of-N generation; it is trained first on multifaceted preference data and then on general helpfulness data, which they report prevents reward hacking that rewards paraphrasing the system message.Source: §J.1 · p. 26, §J.2 · p. 27

J.2 Reward modeling

  • The reward model is initialized from a full-collection JANUS 7B checkpoint trained for one epoch, then replaces its next-token head with a scalar-reward regression head.Source: §J.2 · pp. 26–27

L.1 Multifacetedness evaluation

  • Human evaluation compares model responses pairwise against rubrics and reference answers, while GPT-4-Turbo-0125 scores each of four preference dimensions from 1 to 5.The LLM scores are averaged over dimensions and 921 items, with three evaluations per model.Source: §L.1 · pp. 27–28

L.2 Helpfulness evaluation

  • Helpfulness evaluation uses GPT-4-Turbo as the AlpacaEval 2.0 win-rate baseline and as the MT-Bench evaluator; Arena Hard Auto compares responses against GPT-4-Turbo.MT-Bench uses absolute scores from 0 to 10; Arena Hard Auto scores from 0 to 100.Source: §L.2 · p. 28

L.3 Harmlessness evaluation

  • Harmlessness evaluations include toxicity, fluency, and diversity measures, plus zero-shot social-bias benchmarks such as Winogender, CrowS-Pairs, and BBQ.Winogender measures pronoun prediction accuracy, including stereotype-challenging “gotcha” cases; BBQ tests ambiguous and disambiguated contexts.Source: §L.3 · pp. 28–29

M Prompts

  • The preference-generation prompt asks for instruction-relevant preferences across style, background knowledge, informativeness, and harmlessness, with descriptions limited to two sentences.The system-message prompt turns provided preferences into a message guiding model behavior; the rubric prompt evaluates how well responses reflect a specific preference.Source: M Prompts · pp. 31–33

N Qualitative examples

The examples illustrate how system-message preferences shape responses across physics tutoring, social guidance, and AI explanations. The paper also reports qualitative advantages for Janus over GPT-4 and distributional comparisons across models.

N Qualitative examples

  • Janus alternates between different expertise better than GPT-4 in two MULTIFACETED BENCH instances sharing one instruction.The examples contrast expertise-specific responses under the same instruction.Source: Table 16 · p. 35
  • A physics-tutor system message prompts explanations linking opened curtains to sunlight through light propagation and reduced obstruction.The example emphasizes cause-and-effect explanation while avoiding assumptions about the user's living conditions.Source: N Qualitative examples · p. 36
  • Janus addresses a sensitive issue with greater care and detailed advising, and provides more practical, digestible information than GPT-4.These comparisons are described for separate MULTIFACETED BENCH instances.Source: Table 17 · p. 37, Table 18 · p. 38
  • For a proposed all-girls club closure, the social-conscience response considers belonging and empowerment, suggests alternatives, and recommends community dialogue.The advice also raises possible effects on students and club members.Source: N Qualitative examples · pp. 37–38
  • The AI-framework explanation distinguishes focused algorithm libraries from frameworks spanning broader AI development and deployment workflows.The examples characterize libraries as suited to specific algorithms or prototyping and frameworks as more integrated for complex projects.Source: N Qualitative examples · p. 39
  • Janus diverges less from the pretrained distribution than Mistral 7B Instruct v0.2, while Winogender performance is described as competitive across models.The entropy figure likewise characterizes Janus as closer to human and pretrained distributions than Mistral 7B Instruct v0.2.Source: Figure 1 · p. 41, Figure 2 · p. 41, Figure 3 · p. 41
Loading 2405.17977v2…