PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach

Xuying Li, Zhuo Li, Yuji Kosuga, Victor Bian

arXiv:2503.21819v1cs.CL

TL;DR

Conflicting alignment objectives and the complexity of RLHF make safe language generation challenging. The paper combines GRPO with a multi-label reward model that scores separate alignment aspects. It reports improved alignment metrics across model sizes, while noting that its scalarized reward does not directly support hard constraints.

  • Problem

    Alignment must balance objectives such as helpfulness and safety, while RLHF is complex and can be unstable and single scalar rewards make those trade-offs difficult to represent.

    Source: §1 · pp. 1–2

  • Method

    The method trains a multi-label reward model to predict separate alignment scores, aggregates them into a reward, and uses that reward with GRPO.

    Source: §1 · pp. 2–3, §2.4 · p. 4

  • Results

    The method improved safety, politeness, and overall alignment scores on adversarial prompts across model sizes, and improved all four evaluated metrics over base models.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    The results support multi-aspect reward optimization as an approach to more nuanced alignment across the evaluated model sizes and adversarial prompts.

    Source: §7 · p. 13

    The method uses equal weights for four aspects and scalarizes rewards, so it does not directly support hard constraints such as requiring safety to exceed a threshold.

    Source: §6 · p. 12

Abstract

from arXiv · show

Aligning large language models (LLMs) with human values and safety constraints is challenging, especially when objectives like helpfulness, truthfulness, and avoidance of harm conflict. Reinforcement Learning from Human Feedback (RLHF) has achieved notable success in steering models, but is complex and can be unstable. Recent approaches such as Direct Preference Optimization (DPO) simplify preference-based fine-tuning but may introduce bias or trade-off certain objectives~\cite{dpo}. In this work, we propose a Group Relative Policy Optimization (GRPO) framework with a multi-label reward regression model to achieve safe and aligned language generation. The GRPO algorithm optimizes a policy by comparing groups of sampled responses, eliminating the need for a separate value critic and improving training efficiency~\cite{grpo}. We train a reward model to predict multiple alignment scores (e.g., safety, helpfulness, etc.), which are combined into a single reward signal. We provide a theoretical derivation for using this learned multi-aspect reward within GRPO and discuss its advantages and limitations. Empirically, our approach improves all the safety and quality metrics evaluated in language generation tasks on model scales (0.5B, 7B, and 14B parameters), demonstrating a robust balance of objectives. We compare GRPO to PPO-based RLHF and DPO, highlighting that GRPO achieves alignment with significantly lower computational cost and explicit multi-objective handling. \textbf{We will open-source all trained models at https://huggingface.co/hydroxai.

1 Introduction

The paper addresses the difficulty of balancing conflicting alignment objectives and the complexity of RLHF. It proposes GRPO with a multi-label reward model, provides a theoretical analysis, and reports improvements across four metrics and three model sizes.

Motivation

  • Conflicting alignment goals make it difficult for a single scalar reward to represent trade-offs such as helpfulness versus safety.The paper also identifies standard RLHF as complex, resource-intensive, and potentially unstable to tune.Source: §1 · pp. 1–2

Approach

  • The approach combines GRPO, which normalizes rewards within groups of sampled outputs, with separate aspect scores aggregated into one policy-training reward.The reward model evaluates multiple response aspects, while GRPO avoids a learned value baseline; the paper presents this as an efficient way to handle multi-faceted alignment.Source: §1 · pp. 2–3

Theoretical Analysis

  • The theoretical analysis argues that learned rewards can yield a similar policy-gradient direction when they preserve candidate-output ordering, while systematic errors or poor weights can cause undesirable trade-offs.The paper points to relative-advantage normalization as mitigating some issues from an imperfect reward model.Source: §1 · pp. 2–3

Empirical Evaluation

  • The evaluation uses a RoBERTa-based judge scoring politeness, meaningfulness, actionability, and safety on an adversarial-prompt generation task.The task uses 7,000 attack prompts and fine-tunes three Qwen-2.5 model sizes with LoRA.Source: §1 · p. 2
  • GRPO with multi-label rewards improved all four evaluated metrics over base models across 0.5B, 7B, and 14B variants.The tuned models were also reported to respond more politely and refuse unsafe requests more effectively without sacrificing coherence or usefulness.Source: §1 · pp. 2–3

2 Background and Related Work

The background contrasts preference-optimization methods and multi-objective alignment approaches, then introduces GRPO as group-relative policy optimization without a separate value critic.

2.1 Preference Optimization for Language Model Alignment

  • PPO-based RLHF uses a scalar reward and can be unstable, vulnerable to reward hacking, and difficult to tune; scalar rewards may also entangle alignment criteria.Its tuning requires balancing learning rate, KL penalty, and critic accuracy.Source: §2.1 · p. 3
  • DPO [3] avoids explicit reward modeling and reinforcement learning by training on preferred and dispreferred output pairs, but still uses a scalar preference when objectives conflict.The described loss is derived from a Bradley-Terry model.Source: §2.1 · pp. 3–4

2.2 Multi-Objective Alignment

  • Multi-objective alignment work includes separate weighted reward models and Pareto-frontier policies, while the proposed approach predicts distinct alignment scores and aggregates them for GRPO.The paper presents the latter as a practical approach intended to improve interpretability and robustness while enabling efficient policy optimization.Source: §2.2 · p. 4

2.3 Multi-Objective Alignment

  • The paper again frames multi-objective alignment through weighted reward models and Pareto trade-offs, and proposes separate alignment scores aggregated into a scalar for GRPO.This approach is described as improving interpretability and robustness while enabling efficient policy optimization.Source: §2.3 · p. 4

2.4 Group Relative Policy Optimization (GRPO)

  • GRPO samples a group of outputs, scores them with a reward function, and uses group-relative normalized advantages for policy updates without a separate value critic.The background states that normalization reduces the impact of reward scale and bias and supports stable updates with learned or noisy rewards.Source: §2.4 · pp. 4–5

3 Methodology

The framework pairs GRPO with a learned reward model that scores multiple response qualities, then uses the aggregated reward to train the policy. Its justification rests on preserving candidate rankings and using group-relative advantages; implementation details describe the model and training setup.

3 Methodology

  • The framework combines GRPO with a learned multi-aspect reward and derives a training objective for using that reward in policy optimization.Source: §3 · p. 5

3.1 Learned Multi-Aspect Reward Model

  • The reward model predicts scores for multiple response qualities from prompt-response examples, fits annotated ratings with mean-squared error, and aggregates aspect scores into one reward.The experiments use an unweighted average of the aspect scores.Source: §3.1 · pp. 5–6

3.2 GRPO Training with the Learned Reward

  • GRPO samples and scores a group of candidate responses, then uses their relative advantages to update the policy with the learned reward.Theoretical justification emphasizes that preserving candidate rankings keeps the update direction helpful, while group normalization reduces the impact of shared bias and noisy rewards.Source: §3.2 · p. 6

3.3 Implementation Details

  • Implementation initializes Qwen-2.5 policies, fine-tunes LoRA adapters, and uses a frozen RoBERTa-base reward model that predicts four scores.The reward model achieved average validation R2 of 0.85; no KL penalty was used, with β = 0.Source: §3.3 · pp. 6–7

4 Experiments

Experiments evaluate GRPO on adversarial prompts using three Qwen-2.5 model sizes and four alignment dimensions. The reported results show improvements across the evaluated metrics, with a single-score ablation illustrating a safety–meaningfulness trade-off.

4.1 Setup and Datasets

  • The setup uses 0.5B, 7B, and 14B Qwen-2.5 models and 7,000 adversarial prompts scored for politeness, meaningfulness, actionability, and safety.The overall reward averages the four normalized scores.Source: §4.1 · p. 8

4.2 Training Procedure

  • Training uses the base models’ own generations, samples four responses per prompt, and selects checkpoints by validation reward after roughly two epochs.The training curves show reward gains across all sizes, with the 14B model highest and the 0.5B model plateauing earlier.Source: §4.2 · pp. 8–9, Figure 1 · p. 8

4.3 Results

  • Safety improved most, followed by politeness and actionability; all three model sizes improved on the evaluated reward metrics after GRPO.Table 1 presents normalized scores on a 0–1 scale for the alignment metrics and combined reward.Source: Table 1 · p. 9, §4.3 · p. 9
  • For the facial-recognition attack prompt, the base response described bypass techniques, while the GRPO response refused guidance and redirected to general cybersecurity principles.Source: §4.3 · pp. 9–10
  • In a blind comparison on 100 prompts, three annotators preferred the aligned 7B model in 72% of cases, the base model in 10%, and neither in 18%.Source: §4.3 · p. 10

4.4 Ablation: Single-Aspect vs. Multi-Aspect Reward

  • In the 0.5B single-score ablation, safety rose to 0.72 while meaningfulness fell by 0.02; the authors report that multi-aspect rewards maintained better balance.Source: §4.4 · p. 10

5 Discussion

GRPO with multi-aspect rewards produced aligned responses that sometimes offered help rather than simply refusing unsafe requests. The discussion reports gains across four metrics while noting reward-model and scaling considerations and no observed mode collapse or excessive hesitation.

5 Discussion

  • The aligned model sometimes answered self-harm requests with encouragement to seek help and counseling resources, although such alternatives were not explicitly in the data.The authors attribute this behavior to RL generalization from responses that received high scores.Source: §5 · p. 11
  • The reward model covers four aspects but omits others such as truthfulness and bias, and equal weighting may not suit settings where one objective is critical.The authors note that differing scales or noise could cause an aspect to dominate or be ignored.Source: §5 · p. 11
  • The approach showed no sign of mode collapse or excessive hesitation; the authors suggest relative ranking and usefulness-aware rewards may temper over-refusal.The passage contrasts this with reports of PPO-based RLHF becoming too refusal-prone when unsafe content is strongly penalized.Source: §5 · p. 11

6 Limitations and Future Work

The authors identify limitations in reward-model quality, dialogue scope, theoretical guarantees, objective weighting, and generalization across prompts and cultures. They propose follow-up work including human review, adversarial reward-model training, and broader evaluations.

6 Limitations and Future Work

  • Biased or incomplete reward labels can be inherited by the policy, and the scalar-reward ablation showed slight over-refusal.The reward model does not explicitly distinguish good compliance from good refusal; actionability only covers this indirectly.Source: §6 · p. 11
  • Adversarial training could make the reward model more robust to exploitation by showing it policy outputs that try to trick it.Source: §6 · p. 12
  • Experiments used single-turn prompts, leaving safety over long conversations with repeated user attempts untested.Sampling multiple full dialogues per context could be expensive, and the authors suggest iterative alignment as a possible direction.Source: §6 · p. 12
  • The paper provides no formal convergence proof for GRPO with function approximation and learned rewards, and does not guarantee a global optimum of true reward.The authors also note that advantage normalization may disrupt monotonic improvement guarantees, especially when sampled rewards have very small variance.Source: §6 · p. 12
  • Equal weights for the four reward aspects were somewhat arbitrary, and scalarization does not directly support hard constraints such as a minimum safety threshold.The authors propose constrained optimization or Pareto solutions as alternatives to explore.Source: §6 · p. 12
  • Training prompts may not cover unfamiliar request types or languages, and the primarily English and some Chinese data may encode cultural or dialect biases.The authors did not implement reward-model uncertainty estimation, which they suggest could support conservative fallback responses.Source: §6 · pp. 12–13
  • The authors propose human review of high-scoring outputs that seem problematic, using the resulting data to update the reward model.They also suggest exploring code generation, personalized instruction following, and controlled comparisons with PPO.Source: §6 · p. 13

7 Conclusion

The paper combines GRPO with a multi-label reward model and argues that learned rewards can guide updates when they preserve outputs’ relative ordering. It reports improvements in safety, politeness, and overall alignment across model sizes, while presenting the approach as a complementary route to multi-objective alignment.

7 Conclusion

  • The framework combines GRPO with a multi-label reward regression model, whose learned reward is argued to be sound when it preserves outputs’ relative ordering.The authors say GRPO’s normalization somewhat enforces this condition.Source: §7 · p. 13
  • Safety, politeness, and overall alignment scores improved on adversarial test prompts across various model sizes.The authors report that the resulting models remained useful while behaving more respectfully and safely.Source: §7 · p. 13
  • GRPO with learned rewards is presented as a complementary approach to PPO-based RLHF and DPO, with potentially lower resource use and natural support for multi-objective goals.The authors describe these benefits as potential and cite their findings alongside prior work.Source: §7 · pp. 13–14
  • The authors recommend future work on larger-scale applications, human oversight, and a broader set of alignment criteria.They frame the paper as one step toward methods that address the multifaceted nature of aligning AI systems with human values.Source: §7 · p. 14
Loading 2503.21819v1…