PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Preference Optimization: Few-Shot Alignment of Large Language Models

Siyan Zhao, John Dang, Aditya Grover

arXiv:2310.11523v2cs.LGcs.AIcs.CL

TL;DR

Aligning an LLM to a particular group's preferences is difficult when many groups have little preference data. GPO adds a meta-learned transformer preference module for few-shot adaptation; it outperforms baselines on OpinionQA, with a reported 7.1% average increase over In-context Finetune.

  • Problem

    Existing alignment approaches are challenging to apply when there are many target groups and preference supervision for each group is limited.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    GPO meta-learns an independent transformer preference module that predicts group preferences from context examples and LLM representations of prompts and responses.

    Source: §1 · p. 2, §2.3 · p. 4

  • Results

    GPO achieves a 7.1% average increase over In-context Finetune across two base models and three group-split settings on OpinionQA.

    Source: §3.1 · p. 7

  • Takeaways & Limitations

    GPO adapts to specific group opinion distributions, including unseen test groups, using the same number of context samples as the compared methods.

    Source: §3.1 · pp. 7–8

    GPO may reflect harmful preferences in its outputs when such preferences occur in the meta-training or meta-test data.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Many applications of large language models (LLMs), ranging from chatbots to creative writing, require nuanced subjective judgments that can differ significantly across different groups. Existing alignment algorithms can be expensive to align for each group, requiring prohibitive amounts of group-specific preference data and computation for real-world use cases. We introduce Group Preference Optimization (GPO), an alignment framework that steers language models to preferences of individual groups in a few-shot manner. In GPO, we augment the base LLM with an independent transformer module trained to predict the preferences of a group for the LLM generations. For few-shot learning, we parameterize this module as an in-context autoregressive transformer and train it via meta-learning on several groups. We empirically validate the efficacy of GPO through rigorous evaluations using LLMs with varied sizes on three human opinion adaptation tasks. These tasks involve adapting to the preferences of US demographic groups, global countries, and individual users. Our results demonstrate that GPO not only aligns models more accurately but also requires fewer group-specific preferences, and less training and inference computing resources, outperforming existing strategies such as in-context steering and fine-tuning methods.

1 INTRODUCTION

Group alignment must accommodate subjective preferences that vary across demographics, cultures, and individuals, while existing approaches face trade-offs between prompting effort and group-specific supervision. GPO learns a few-shot preference module to adapt base LLMs to groups, including unseen groups.

1 INTRODUCTION

  • LLMs may represent diverse opinions, but their generations must be steered to match the requirements of a target group.The motivating applications include chatbots and creative writing, where judgments can vary across demographics, cultural and societal norms, and individual preferences.Source: §1 · p. 1
  • Prompting avoids model training but can be tedious and empirically poor for complex behaviors, while reward-model alignment requires substantial group-specific supervision and computation.Existing PPO, DPO, and Best-Of-N approaches are not designed for many target groups with limited per-group supervision.Source: §1 · pp. 1–2
  • GPO augments a base LLM with an independent transformer that learns in-context preference prediction from embedded prompts and corresponding LLM responses.Meta-learning on training groups enables few-shot adaptation to unseen groups without fine-tuning the base LLM.Source: §1 · p. 2
  • The learned module can replace a reward or preference function in policy-optimization and re-ranking algorithms.Source: §1 · p. 3
  • GPO surpasses the top-performing baselines by 7.1% on OpinionQA and by 8.4% on GlobalOpinionQA, and performs most effectively for individual preferences.The evaluations cover 22 US demographic groups and 14 countries.Source: §1 · p. 3

2 GROUP PREFERENCE OPTIMIZATION

The paper frames group alignment as few-shot adaptation because many groups may each have little preference data. GPO meta-learns a transformer preference model that uses labeled context examples to predict preferences for new inputs and groups.

2.1 PROBLEM SETUP

  • Group alignment targets a distribution of LLM outputs suited to a chosen group, while practical settings may involve many groups and little preference data for each.A group can be any collection of agents, including demographic groups or individual personas.Source: §2.1 · p. 3

2.2 RELATED WORK

  • Prompt engineering is training-free but can require heuristic design and has limited gains on complex groups; gradient-based alignment may require extensive group data or difficult tuning and stability management.These trade-offs motivate GPO's focus on explicit alignment under limited supervision.Source: §2.2 · pp. 3–4

2.3 PROPOSED METHOD

  • GPO meta-learns a transformer to predict target preferences from context preferences and target inputs, sampling training groups and context sizes uniformly.For decoding, target preferences are assumed conditionally independent given the context samples and target inputs; preliminary experiments found no noticeable improvement from modeling dependencies.Source: §2.3 · p. 4
  • GPO differs from in-context prompting a base LLM: it learns an augmenting few-shot model from preference labels rather than desired text generations.Prompt engineering can still be used with GPO by replacing its default input prompt.Source: §2.3 · p. 5
  • To handle long prompt-response inputs, GPO uses joint text embeddings; the authors found average token embeddings worked best among the tested extraction methods.The transformer drops positional encodings, concatenates each context input-preference pair, and masks target points from attending to one another.Source: §2.3 · pp. 4–5

3 EXPERIMENTS

The experiments evaluate group alignment on OpinionQA and GlobalOpinionQA using survey-derived preference distributions and multiple language models. GPO outperforms baselines on group and individual adaptation, with reported gains across the two datasets and improved alignment using few context samples.

3 EXPERIMENTS

  • OpinionQA covers 22 US demographic groups and 500 questions, while GlobalOpinionQA covers 14 countries and 2,554 questions.The benchmarks use multiple-choice survey responses to represent diverse opinions.Source: §3 · p. 5
  • Survey responses become normalized, question-specific group preference distributions, with all viewpoints from a question kept in either context or target data.The evaluation uses Wasserstein similarity for ordinal OpinionQA answers and Jensen-Shannon distance for often non-ordinal GlobalOpinionQA answers.Source: §3 · p. 6

3.1 RESULTS AND DISCUSSION

  • Base-model and steering results vary: Alpaca is near uniform on OpinionQA, Llama2 is lower, few-shot prompting declines, and SFT beats other gradient-free steering.Reward Model underperforms SFT, which the authors say may suggest overfitting with limited samples.Source: §3.1 · pp. 6–7
  • GPO improves alignment over In-context Finetune by 7.1% on OpinionQA and over the second-best baseline by 8.4% on GlobalOpinionQA.The first average covers two base models and three group-split settings; the second covers both base models and three group-split scenarios.Source: Figure 3 · p. 7, §3.1 · pp. 7–8
  • With 15 context samples, GPO adjusts predicted distributions for groups such as Hindus while steered LMs can overrepresent an option.The qualitative example also shows GPO maintaining a more balanced distribution for college graduates or some postgraduates.Source: §3.1 · pp. 7–8, Figure 4 · p. 8
  • On GlobalOpinionQA, In-context Finetune ranks second for Llama2 and third for Alpaca but requires about 4.7 times GPO's training time.The training-time comparison was measured on an NVIDIA RTX A6000.Source: §3.1 · p. 8
  • For Nigeria in GlobalOpinionQA, GPO improves alignment with fewer than 10 preference context samples, while Few-shot Prompt plateaus with greater variance.In-context Finetune adapts better than Few-shot Prompt after meta-training, but its context samples are limited by the LM's context window.Source: §3.1 · p. 8, Figure 5 · p. 9
  • Across 15 OpinionQA topics, GPO performs better than other baselines for individual preferences, evaluated with Alpaca-7b and a 40%/60% individual split.The study sampled 100 participants per survey and used their responses to 30 topic-related questions; SFT and Reward Model comparisons were limited to one survey by computational constraints.Source: Figure 6 · p. 9, §3.1 · p. 9

4 CONCLUSION

GPO is a few-shot approach for aligning LLM outputs to individual and group preferences with little preference data. It adapts to new test groups from a few examples, significantly outperforming prior methods on alignment score without gradient updates to the base LLM.

4 CONCLUSION

  • GPO significantly outperforms prior methods on group alignment score without gradient updates to the base LLM.It is also effective across popular open-source LLMs with varied parameter and pre-training dataset scales.Source: §4 · p. 10
  • GPO adapts to new test groups by predicting aligned preferences from a few examples of group-wise preference data.The method is trained on a meta-train dataset containing group-wise preference data.Source: §4 · pp. 9–10
  • GPO improves alignment score significantly more than baseline methods while using fewer samples.The conclusion describes this as greater sample efficiency.Source: §4 · p. 10

ETHICS STATEMENT

GPO may support more positive, useful, and inclusive experiences by aligning models to diverse interest-group preferences. The paper also warns that this capability may be misused and can reflect harmful preferences in model outputs.

ETHICS STATEMENT

  • Aligning models to diverse interest-group preferences may provide more positive, useful, and inclusive experiences for end users.Source: ETHICS STATEMENT · p. 11
  • Personalized responses could be misused to convince or show group members how to perform unethical actions.Source: ETHICS STATEMENT · p. 11
  • Biased, offensive, or harmful preferences in meta-training or meta-test data may be reflected in GPO outputs.The paper calls for future work on aligning outputs without amplifying harmful outputs.Source: ETHICS STATEMENT · p. 11

A LIMITATIONS

The paper identifies limits in its opinion data, multiple-choice evaluation, alignment objectives, and model initialization. These constrain the demonstrated scope and leave several effects and settings for future study.

A LIMITATIONS

  • Survey data may not represent entire groups, and the datasets are English-only, with language choice potentially affecting alignment metrics.The concern is especially noted when inputs or outputs differ from group members’ native languages.Source: A LIMITATIONS · p. 15
  • Evaluation focuses on multiple-choice questions, so GPO’s effectiveness on longer-form responses remains to be validated.The framework extends more broadly to generation formats, but long-form feedback representation and evaluation metrics also need consideration.Source: A LIMITATIONS · p. 15
  • Aligning to group preferences may worsen harmlessness or helpfulness, especially when preference data contradicts those values.It may also amplify biased or harmful model behaviors.Source: A LIMITATIONS · p. 15
  • A pretrained language-model transformer backbone might improve GPO performance or out-of-distribution generalization, but these benefits remain for future investigation.The proposed mechanism is potentially stronger encoding of world knowledge.Source: A LIMITATIONS · p. 15

B.1 OPINIONQA DATASET

The appendix describes the OpinionQA and GlobalOpinionQA datasets and the alignment metrics used for their different answer structures. It also reports the group and country samples used in the study.

B.1 OPINIONQA DATASET

  • OpinionQA uses 1-D Wasserstein Distance because its survey answers are ordinal, unlike KL divergence, which does not capture their ordinal relationships.Answer options are mapped to positive integers; the score is normalized by N − 1, where N excludes refusal options, and ranges from 0 to 1.Source: §B.1 · p. 15
  • The OpinionQA analysis covers 22 US demographic groups and 500 contentious questions with frequent subgroup disagreements.Table 1 identifies the demographic groups considered in the analysis.Source: §B.1 · pp. 15–16, Table 1 · p. 16

B.2 GLOBALOPINIONQA DATASET

  • GlobalOpinionQA questions generally lack ordinal answer structures, so the study uses Jensen-Shannon Distance rather than the OpinionQA metric.The questions come from Pew Research Center’s Global Attitudes surveys and the World Values Survey.Source: §B.2 · p. 16
  • The GlobalOpinionQA study selects 14 of the original 138 countries due to computational constraints, spanning several continents.Table 2 lists the countries considered.Source: §B.2 · p. 16, Table 2 · p. 16

C ABLATION ON THE GPO’S TRANSFORMER ARCHITECTURE

The architecture ablation compares GPO’s transformer biases with a standard autoregressive transformer. GPO’s context and target masking choices support permutation properties and outperform that baseline in alignment.

C ABLATION ON THE GPO’S TRANSFORMER ARCHITECTURE

  • Context invariance omits positional encodings and concatenates each preference pair into one token, preserving pair relations while making predictions insensitive to context order.Context pairs interact with each other under an alternative masking strategy.Source: C ABLATION ON THE GPO’S TRANSFORMER… · p. 17
  • Target equivariance masks query pairs so they attend only to context samples, preventing one query’s prediction from depending on other queries.Context samples self-attend, consistent with the stated conditional-independence principle.Source: C ABLATION ON THE GPO’S TRANSFORMER… · p. 17
  • GPO achieves superior alignment performance to a standard autoregressive transformer in the reported comparison.The comparison retains concatenated (x, y) tokens for the standard transformer; Table 3 reports the comparison on GlobalOpinionQA alignment tasks.Source: C ABLATION ON THE GPO’S TRANSFORMER… · p. 17, Table 3 · p. 17

D ABLATION ON GETTING EMBEDDINGS FROM THE LLM.

The embedding ablation compares three ways to represent a sentence using token embeddings. Averaging the token embeddings is most effective, while using only the last token is less adept at capturing sentence-level information.

D ABLATION ON GETTING EMBEDDINGS FROM THE LLM.

E ABLATION ON ADDING GROUP META-CONTEXT FOR GPO

Adding group meta-context embeddings does not improve GPO’s performance across the three group-split scenarios and instead performs worse. The section also describes baseline approaches spanning unsteered, prompted, and trained models.

E ABLATION ON ADDING GROUP META-CONTEXT FOR GPO

  • Adding group meta-context embeddings worsens GPO performance across the three group-split scenarios.The authors hypothesize that GPO lacks comprehensive world knowledge of diverse group attributes, while it excels at deducing preference distributions from context pairs.Source: E ABLATION ON ADDING GROUP… · p. 18, Table 5 · p. 18
  • The ablation embeds group context and concatenates it with each prompt-preference pair as an input token.The primary experiments embed viewpoints without group identity or attribute metadata.Source: E ABLATION ON ADDING GROUP… · pp. 17–18

F BASELINES DETAILS

  • The baselines include uniform answer probabilities, an unsteered language model, explicit group prompting, and few-shot preference examples.Source: F BASELINES DETAILS · p. 18
  • Other baselines fine-tune models per group, train a reward model, or fine-tune across groups for in-context adaptation to unseen groups.Source: F BASELINES DETAILS · pp. 18–19

G TRAINING SETTINGS

The training settings specify separate configurations for the trained baselines and GPO, including hardware, optimization, and model dimensions.

G TRAINING SETTINGS

  • Baseline fine-tuning methods use 8-bit quantization on one RTX A6000 GPU with 48GB VRAM, LoRA, bf16, and AdamW.Validation alignment score is used for early stopping.Source: G TRAINING SETTINGS · p. 19
  • Baseline effective batch size is 8 for SFT and in-context fine-tuning, and 4 for reward-model training.Learning rates are 1e-4 for Alpaca baselines and 2e-5 for Llama2-13B-chat baselines.Source: G TRAINING SETTINGS · p. 19
  • GPO uses a 6-layer transformer with embedding depth 4, 4 heads, and feedforward dimension 128.Training tasks sample 10–100 context examples and use a 3e-4 learning rate.Source: G TRAINING SETTINGS · p. 19

H EXTENDING GPO BEYOND MULTIPLE-CHOICE QUESTIONS

GPO extends beyond multiple-choice tasks to language generation with scalar feedback, using relative comparisons or absolute ratings. Its preference module can also serve as a reward model for PPO when separate-model inference is undesirable.

H EXTENDING GPO BEYOND MULTIPLE-CHOICE QUESTIONS

I ADDITIONAL RELATED WORK

  • Prompting avoids gradient updates but is context-limited and requires prompt engineering; gradient methods can be compute-intensive and typically yield specialized models for each subgroup.Source: I ADDITIONAL RELATED WORK · p. 20

J QUALITATIVE EXAMPLES OF GPO.

K CONTEXTUAL PROMPT EXAMPLES

Loading 2310.11523v2…