PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Robust Preference Optimization in Reward-free RLHF

Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, Ilija Bogunovic

arXiv:2405.20304v1cs.CLcs.LG

TL;DR

Preference-based RLHF often treats feedback from distinct groups as one distribution, although group preferences can diverge. GRPO adds group information and optimizes worst-case group performance within reward-free preference optimization. The paper reports improved worst-group performance, reduced loss imbalances, and higher probability accuracies than non-robust baselines.

  • Problem

    Preference-based alignment often uses a one-size-fits-all model despite distinct groups having divergent preferences, which can favor majority groups over minorities.

    Source: §1 · pp. 1–2

  • Method

    GRPO adds group information to the LLM context and optimizes worst-case alignment across groups within reward-free preference optimization.

    Source: §1 · pp. 2–3

  • Results

    GRPO improves performance for the worst-performing groups, reduces loss imbalances, and increases probability accuracies compared with non-robust baselines.

    Source: §1 · p. 3, §6 · p. 12

  • Takeaways & Limitations

    The authors see GRPO as promising for future tailored fine-tuning aimed at diverse teams and user groups.

    Source: §6 · p. 12

    When group data are balanced and difficulty is comparable, GRPO offers no significant advantage over standard reward-free RLHF; tuning the worst-case versus average-performance trade-off remains future work.

    Source: §6 · p. 12

Abstract

from arXiv · show

Adapting large language models (LLMs) for specific tasks usually involves fine-tuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers' groups (e.g., different demographics, ethnicities, company teams, etc.), traditional RLHF approaches adopt a "one-size-fits-all" approach, i.e., they indiscriminately assume and optimize a single preference model, thus not being robust to unique characteristics and needs of the various groups. To address this limitation, we propose a novel Group Robust Preference Optimization (GRPO) method to align LLMs to individual groups' preferences robustly. Our approach builds upon reward-free direct preference optimization methods, but unlike previous approaches, it seeks a robust policy which maximizes the worst-case group performance. To achieve this, GRPO adaptively and sequentially weights the importance of different groups, prioritizing groups with worse cumulative loss. We theoretically study the feasibility of GRPO and analyze its convergence for the log-linear policy class. By fine-tuning LLMs with GRPO using diverse group-based global opinion data, we significantly improved performance for the worst-performing groups, reduced loss imbalances across groups, and improved probability accuracies compared to non-robust baselines.

1 Introduction

LLM alignment is important because pretraining on internet-scale text carries risks of bias, inaccuracy, and controversial content. Since preference data can reflect distinct groups, the paper introduces GRPO to optimize worst-case group alignment rather than a single average preference.

1 Introduction

  • Distinct groups can have preferences that diverge, while conventional alignment treats individual feedback as samples from one broader preference distribution.The paper identifies demographics, ethnicities, and company teams as examples of such groups.Source: §1 · p. 1
  • One-size-fits-all alignment can favor majority preferences at the expense of minority groups.Figure 1 contrasts majority preferences G1 (R1 > R2) with minority preferences G2 (R2 > R1).Source: §1 · p. 2, Figure 1 · p. 2
  • GRPO adds group information to the LLM context and optimizes worst-case alignment performance across groups.The approach is presented as a group-robust formulation of DPO [35].Source: §1 · pp. 2–3
  • GRPO embeds group robustness in reward-free tuning, adaptively weighting group losses and using a gradient estimator tailored to group-robust DPO.The related-work discussion distinguishes this from robustness work focused on noise or out-of-preference data [19, 24].Source: §1 · p. 3
  • The paper studies GRPO's feasibility and convergence in the log-linear policy class and reports better worst-group performance, smaller loss imbalances, and higher probability accuracies than non-robust baselines.It also demonstrates use with Identity Preference Optimization (IPO) [1].Source: §1 · p. 3

2 Background

The background describes RLHF as a pipeline from supervised fine-tuning through reward learning to policy optimization. DPO uses preference data to optimize the policy directly, avoiding a separate reward model.

2 Background

  • RLHF comprises supervised fine-tuning of a pretrained model, reward learning, and reinforcement-learning fine-tuning.The supervised phase produces a reference policy, while reward learning uses prompts and preferred-versus-less-preferred responses.Source: §2 · p. 3
  • Preference modeling commonly assumes an unknown reward function, with the Bradley-Terry model providing one form for preference probabilities.The dataset consists of prompts and response pairs where human labelers prefer yw over yl.Source: §2 · pp. 3–4
  • DPO [35] exploits the closed-form reward-maximizing policy to optimize directly from preferences without explicitly modeling rewards.For a parameterized policy, minimizing DPO loss uses backpropagation and completion log-probabilities from both the policy and reference policy.Source: §2 · p. 4

3 Group Robust Preference Optimization

GRPO formulates preference optimization across groups as a worst-case loss problem, accommodating varied group prompts and preferences. The discussion also establishes an equilibrium result for a log-linear policy class and an equivalence between two routes to the robust DPO loss.

3 Group Robust Preference Optimization

  • GRPO incorporates group information into prompts and defines a group-indexed latent reward, allowing the same prompt to appear for groups that may prefer different responses.Group information can be an index or textual description; prompt distributions may overlap or be distinct.Source: §3 · pp. 4–5
  • The objective measures alignment by worst-case group loss and casts optimization as a policy-versus-group-weighting problem over the probability simplex.The logistic DPO loss can be replaced by hinge or squared loss; the squared-loss variant is explored in Section 4.1.Source: §3 · p. 5
  • GRPO applies beyond human groups to task or topic categories and to shared prompts with diverse user preferences, unlike average-preference DPO, which may overlook minority groups.Examples include helpful versus harmful, truthful versus unanswerable, and domain-specific categories such as math or chemistry.Source: §3 · p. 5
  • The max-min objective may be overly conservative and degrade average performance, motivating a balanced objective that trades off worst-case and standard preference optimization.The corresponding objective and algorithmic changes are discussed in Appendix B.4.Source: §3 · p. 5

3.1 Further Discussion and Insights

  • Proposition 1 establishes that the group-robust DPO problem has a Nash equilibrium under log-linear policy parameterization.The objective is convex in policy parameters and concave in group weights, enabling application of the minimax theorem.Source: §3.1 · p. 6
  • Proposition 2 shows that the same group-robust DPO loss results whether robustness is introduced in reward modeling or in KL-regularized policy optimization.The equivalence follows by substituting the robust KL-regularized policy solution into the non-robust objective.Source: §3.1 · p. 6

4 Algorithm

Algorithm 1 solves group-robust DPO through alternating policy and group-weight updates, increasing weight on groups with higher cumulative losses. The section also gives a convergence guarantee under stated assumptions and extends the approach to group-robust IPO.

4 Algorithm

  • Algorithm 1 alternates policy gradient descent with group-weight mirror ascent, increasing emphasis on groups with higher cumulative losses.The group weights use multiplicative weights with simplex renormalization; the policy update is weighted by those group weights.Source: §4 · pp. 6–7
  • With DPO loss, the gradient increases preferred-response likelihood and decreases rejected-response likelihood in proportion to reward disparity; group weights scale updates by cumulative loss, while N_g accounts for group sample volume.The text says the N_g factor ensures smaller-volume groups are not disadvantaged.Source: §4 · p. 7
  • Proposition 3 establishes global convergence for Algorithm 1’s average iterate under the stated convexity, boundedness, and Lipschitz assumptions.The text notes that these assumptions hold for log-linear policies when fine-tuning only the final LLM layer.Source: §4 · p. 7

4.1 Group Robust Identity Preference Optimization

  • Because standard DPO can learn deterministic policies, GR-IPO adopts IPO’s approach to randomized policies and replaces the policy update with weighted regression.The cited motivation is that IPO is more likely to assign appropriate preference probabilities and prevent overfitting.Source: §4.1 · p. 8
  • In the GR-IPO update, each row of S encodes the preferred-minus-less-preferred response feature-mapping difference for one prompt.The closed-form update also uses each sample’s group assignment and that group’s sample count.Source: §4.1 · p. 8

5 Experiments

The experiments evaluate group-robust preference optimization on synthetic group disparities and country-based opinion data. Across both settings, the robust methods improve worst-group outcomes, with GR-IPO also reducing loss imbalance on the real-data task.

5.1 Synthetic Experiments

  • Across synthetic scenarios, robust methods surpass vanilla and importance-sampling baselines; importance sampling falls short when disparities involve more than group size.The comparison covers worst-case validation loss and reward error.Source: Figure 2 · p. 9, Figure 3 · p. 9, Figure 4 · p. 10
  • Synthetic tests vary group-size balance and response distributions; the two-group setup uses coordinate-flipped feature parameters to create distinct group preferences.The scenarios include imbalanced sizes with shared response distributions, balanced sizes with different distributions, and both disparities together.Source: §5.1 · pp. 9–10

5.2 Global Opinion Experiments

  • The real-data evaluation uses GlobalOpinionQA responses from five countries and trains IPO and GR-IPO preference pairs from each question’s highest-probability option.The dataset contains 2,554 questions; the study evaluates worst-group loss and accuracy on a fine-tuned Gemma-2B model.Source: §5.2 · pp. 10–11
  • GR-IPO outperforms IPO on maximum group loss and minimum group reward accuracy, while reducing loss imbalance across countries.The comparison uses GlobalOpinionQA data and is averaged over five seeds.Source: Figure 5 · p. 11, §5.2 · p. 11
  • Groups 2 and 5 start with weaker log-probability accuracy; GR-IPO assigns them more weight and mitigates the model’s performance imbalance.The paper reports that the base LLM struggles to classify responses for these groups.Source: §5.2 · pp. 11–12

6 Conclusions

The paper presents GRPO for robustly aligning LLMs to diverse group preferences and reports theoretical analysis and experiments. It identifies balanced groups with comparable difficulty as a setting where GRPO offers no significant advantage over standard reward-free RLHF.

6 Conclusions

  • GRPO formalizes robust alignment to diverse group preferences and aims to minimize worst-case group loss; the paper reports theoretical analysis and experimental evaluation.The conclusion describes the approach as promising for tailored fine-tuning and mitigating group discrepancies.Source: §6 · p. 12
  • When group data are balanced and difficulty levels comparable, GRPO offers no significant advantage over standard reward-free RLHF algorithms.Tuning the trade-off parameter for applications remains future work.Source: §6 · p. 12

Appendix

  • The appendix collects related work, theoretical proofs and a weighted DPO gradient expression, IPO material, experimental details, and convergence proofs..Source: Appendix · p. 16

A Additional Related Work

  • The related-work discussion covers RLHF’s reward-model and PPO pipeline, reward-free alternatives including DPO, and prior group-robust work in language modeling and RLHF.DPO directly optimizes policies from preferences without a separate reward model [35]; earlier group-DRO language-model work focuses on pre-training [31] [49].Source: A Additional Related Work · pp. 16–17

B Theoretical Insights

  • The theoretical section specializes the general robust objective to a log-linear policy class.The derivation uses the log-linear policy form and a uniform reference policy.Source: §B.1 · p. 18

B.1 Robust Objective for the Log-linear Policy Class

  • For the log-linear policy class, the derivation substitutes a uniform reference policy and the policy’s log-linear parameterization into the robust objective..Source: §B.1 · p. 18

B.2 Robust KL-Regularized Policy Maximization

  • The robust KL-regularized policy maximization yields the same group-robust DPO loss, and its optimal policy has the standard non-robust KL-regularized form.The proposition connects the robust reward-maximization formulation to the group-robust DPO loss.Source: §B.2 · p. 19

B.3 Analysis of the Gradient

This material presents a gradient derivation for the group-robust loss and describes an objective that trades off worst-case and average group performance. The trade-off uses group weights and a parameter controlling the balance.

B.3 Analysis of the Gradient

  • The section simplifies the group-robust loss gradient using derivatives of the logarithm and sigmoid and the definition of the policy preference term.The appendix states that these steps yield the gradient expression.Source: §B.3 · pp. 20–21

B.4 Trading off worst-case vs. average performance

  • The trade-off objective combines fixed group weights with worst-case weighting, allowing performance to balance average and worst-case outcomes.The parameter χ controls the balance, and the maximizing group-weight player’s influence depends on χ and the input weights.Source: §B.4 · p. 21

C IPO Simplification

The section motivates reward-free preference optimization by describing a finite-data overfitting concern, then presents IPO as a regression-form objective with a closed-form solution in the linear bandit setting. Group IPO extends the setup to group-specific preferences, where unequal data counts can advantage larger groups.

C IPO Simplification

  • In finite data, observing only two responses per prompt can lead to inferring deterministic preferences and an unbounded reward difference under Bradley–Terry.The passage notes that this can drive the DPO policy probability for the less-preferred response toward zero.Source: C IPO Simplification · p. 22
  • IPO minimizes squared differences between true and policy preferences, yielding a linear-regression formulation and a closed-form solution in the linear bandit setting.The feature matrix contains differences between preferred and less-preferred response features.Source: C IPO Simplification · p. 22
  • In the linear bandit setting, IPO loss stability depends on the rank of the feature matrix.The robust IPO algorithm can use weighted regression with a closed-form solution.Source: C IPO Simplification · p. 23
  • Group IPO uses group-dependent features and aims to balance groups, while unequal preference-data counts can give the larger group an unfair advantage.The robust variant is introduced to address this imbalance.Source: C IPO Simplification · p. 23

D Additional Experiments and Details

The supplementary material gives synthetic-experiment hyperparameter information and describes how preference data were generated under different group imbalance scenarios.

D Additional Experiments and Details

D.1 Synthetic Experiments - Preferences data generation

  • Synthetic preference data cover group-size imbalance, differing preference distributions, and their combination; the size-imbalance ratio is 0.2:0.8.For differing preferences, the groups use distinct rules to select the second response before rewards determine which response is preferred.Source: §D.1 · p. 24

D.2 Synthetic Experiments - Additional feature parametrizations

The additional synthetic experiments vary feature parametrizations and group-size balance to test GR-DPO and GR-IPO against vanilla and importance-sampling methods. GR-DPO generally performs best on worst-case reward errors, while GR-IPO can overfit in uneven-group settings.

D.2 Synthetic Experiments - Additional feature parametrizations

  • The experiments use action space n = 7 and group space K = 2, comparing same-feature and flipped-feature configurations with coordinate-flipped reward vectors.The same-feature configuration uses coordinate-flipped reward vectors; the flipped configuration swaps the order of alternating sine and cosine feature terms across groups.Source: §D.2 · p. 24
  • With same feature vectors and even groups imbalanced in size, GR-DPO slightly improves over importance-sampling DPO, while GR-IPO matches importance-sampling IPO.The passage attributes this match to equal group difficulty, with groups differing only in data size.Source: Figure 6 · p. 25
  • In Figure 7, GR-DPO/GR-IPO outperform corresponding importance-sampling methods on worst-case validation loss, but GR-IPO tends to overfit.Figure 8 reports GR-IPO overfitting more clearly in reward errors, where it performs worse than IPO and importance-sampling IPO.Source: Figure 7 · p. 25, Figure 8 · p. 25
  • With the same feature vectors and uneven groups, GR-DPO/GR-IPO outperform corresponding importance-sampling methods on worst-case validation loss in the balanced-size setting.For imbalanced group sizes, GR-DPO/GR-IPO also outperform vanilla and importance-sampling methods on worst-case validation loss.Source: Figure 7 · p. 25, Figure 8 · p. 25
  • Across balanced and imbalanced group settings with same or flipped feature vectors, GR-DPO outperforms all other methods in worst-case reward errors (Figures 7, 8, 10, and 11).GR-IPO tends to overfit; its reward errors are worse than IPO in the balanced setting and worse than IPO and importance-sampling IPO in the imbalanced setting, where overfitting is more evident in reward errors.Source: Figure 7 · p. 25, Figure 8 · p. 25, Figure 10 · p. 26

D.3 Global Opinion Data Experiments - Setup and Additional Plots

The section describes the global-opinion-data training setup and additional plots, then introduces convergence-rate analysis for different sampling strategies. In the plotted IPO comparison, both methods improve accuracy, while GR-IPO achieves better worst-case alignment performance.

D.3 Global Opinion Data Experiments - Setup and Additional Plots

  • The study uses opinion prompts for five countries, with per-country data sizes of Nigeria 572, Egypt 570, India 376, China 309, and Japan 712.The data are split 80% for training, 10% for validation, and 10% for testing; SFT uses Gemma-2B for one epoch at learning rate 10^-4.Source: §D.3 · p. 26
  • IPO and GR-IPO use the SFT-trained model as reference; GR-IPO uses β = 0.01, learning rate 6 × 10^-5, and exponential step size 5 × 10^-7.The passage gives IPO learning rate 3 × 10^-5 and β = 0.01; both methods use AdamW and adaptive learning-rate reductions based on validation loss.Source: §D.3 · pp. 26–27
  • The experiments evaluate worst-group loss and accuracy, where accuracy is the percentage of prompt, winning-response, and losing-response triples correctly ordered by the learned preference function.GR-IPO reduces its exponential step size when its learning rate is reduced; the passage says these adaptive schedules ensure training stability and lead to convergence.Source: §D.3 · p. 27
  • GR-IPO achieves better worst-case alignment performance than IPO, although both methods improve accuracy during training.Figure 12 also plots end-of-training log-probability accuracies for GR-IPO.Source: Figure 12 · p. 27

E Convergence Proofs for Different Sampling Strategies

  • The convergence section studies convergence rates for different sampling strategies, including the proposed Algorithm 1.The supplied passage introduces the study but does not state the resulting rates.Source: E Convergence Proofs for Different… · p. 28

E.1 Convergence Proof for Algorithm 1

The section analyzes convergence of Algorithm 1's average policy iterate and an alternate uniformly sampled-group algorithm. Its stated guarantees rely on convexity, Lipschitz continuity, bounded loss, and bounded policy parameters.

E.1 Convergence Proof for Algorithm 1

  • The convergence analysis defines its target as the error of the average iterate relative to the minimum loss over θ ∈ Θ.The proof bounds this approximation error using a mirror-descent convergence result for max-min problems.Source: §E.1 · pp. 28, 30
  • Proposition 3 bounds the error of Algorithm 1's average iterate under nonnegative, convex, Lipschitz-continuous, bounded loss and bounded policy parameters.For the log-linear policy class, the DPO loss is convex and Lipschitz continuous, so Proposition 3 applies.Source: §E.1 · pp. 28, 31
  • The proof maps Algorithm 1's policy and group-weight updates to mirror descent, using gradient descent for policy parameters and exponentiated gradient ascent for group weights.The combined update treats θ and α as the joint variable ζ; α is maximized while θ is optimized as policy parameters.Source: §E.1 · p. 29

E.2 Alternate Sampling Strategy

  • Proposition 6 states that Algorithm 3’s average iterate achieves an error rate under the stated loss and parameter assumptions; the algorithm samples groups uniformly rather than proportionally to group sizes.Using the alternate Mθ and Mα for this sampling rule, the authors obtain the overall error bound.Source: §E.2 · pp. 32–33
Loading 2405.20304v1…