PaperlayerView exact revision on arXiv

Source-linked AI summary

f-GRPO and Beyond: Divergence-Based Reinforcement Learning Algorithms for General LLM Alignment

Rajdeep Haldar, Lantao Mei, Guang Lin, Yue Xing, Qifan Song

arXiv:2602.05946v3cs.LGstat.ML

TL;DR

Divergence-based alignment had been limited to preference supervision, leaving scalar-reward RLVR without a unified divergence foundation. The paper introduces f-GRPO and f-HAL to extend divergence estimation to reward-induced distributions and hybrid supervision, reporting improved math-reasoning RLVR performance and reduced reward hacking in safety alignment.

  • Problem

    Divergence-based alignment had been restricted to preference data, while RLVR provides only scalar rewards rather than explicitly aligned and unaligned response pairs.

  • Method

    The paper introduces f-GRPO for on-policy reward alignment and f-HAL, which combines reward optimization with off-policy preference supervision.

  • Results

    f-GRPO improves over GRPO on math reasoning, while f-HAL mitigates reward hacking in safety alignment with learned reward models.

  • Takeaways & Limitations

    Divergence estimation offers a unified foundation for designing reward-based and preference-based LLM alignment algorithms.

  • Takeaways & Limitations

    Experiments are limited to models up to 7B parameters and are intended to validate framework feasibility rather than identify a universally best divergence.

Abstract

from arXiv · show

Recent work shows that preference alignment objectives can be interpreted as divergence estimators between aligned (preferred) & unaligned (less-preferred) distributions, yielding a principled recipe for designing alignment losses. However, this view has so far been limited to preference-based supervision. We extend it to general LLM alignment, including reinforcement learning with verifiable rewards (RLVR), where alignment feedback is given only as scalar rewards. We introduce $f$-Group Relative Policy Optimization ($f$-GRPO), a class of on-policy RL objectives, and $f$-Hybrid Alignment Loss ($f$-HAL), which combines on-policy reward optimization with off-policy preference supervision. We show that these objectives estimate $f$-divergences between reward-aligned & reward-unaligned distributions induced by above- & below-average reward responses, and prove expected reward improvement after alignment. Empirically, $f$-GRPO improves over GRPO on math-reasoning RLVR tasks, while hybrid $f$-HAL mitigates reward hacking in on-policy safety alignment when verifiable rewards are unavailable and learned reward models must be used.

1 Introduction

LLM alignment can be viewed as separating desirable from undesirable responses, but divergence-based methods had been limited to preference data. The paper extends this view to scalar-reward RLVR and combines reward-based and preference-based alignment.

  • Alignment adapts language models toward desired behaviors, including mathematical reasoning, code generation, safety, and preference-based constraints.
  • Divergence-based preference alignment estimates divergences between chosen and rejected response distributions, providing interpretability, guarantees, and a recipe for designing losses.
  • The prior divergence view did not cover RLVR, where only scalar rewards are observed instead of explicitly chosen and rejected responses.
  • f-GRPO induces aligned distributions from above-average rewards and unaligned distributions from below-average rewards, then optimizes a variational f-divergence on-policy.
  • f-HAL combines on-policy reward exploration with off-policy preference supervision, using preferences as an anchor to mitigate reward hacking from imperfect learned rewards.
  • f-GRPO improves over GRPO on math-reasoning RLVR, while f-HAL mitigates reward hacking in safety alignment with learned reward models.

2 Preliminaries and Background

The preliminaries distinguish KL-regularized alignment, reward-based RLVR, and preference alignment, then frame direct preference objectives as f-divergence estimation. This background motivates an on-policy divergence estimator for reward-labeled samples.

  • The standard alignment objective maximizes expected reward while constraining the policy to remain close to a reference model.
  • RLVR: In RLVR, scalar rewards label sampled responses, so above-average responses are encouraged and below-average responses discouraged through on-policy updates.
  • RLVR: The clipped GRPO variant performs advantage-weighted policy updates over sampled responses and is used for empirical stability.
  • Preference alignment: Preference alignment directly observes preferred and less-preferred responses, enabling reward-model training or off-policy objectives such as DPO, BCO, and KTO.
  • Divergence view: FDO interprets direct preference alignment as estimating an f-divergence between aligned and unaligned distributions, extending a prior divergence perspective.

3 Method: From Divergence Estimation to On-Policy RL

The method extends divergence estimation to RLVR by using reward-labeled on-policy samples to construct aligned and unaligned distributions, then combines this objective with preference supervision in f-HAL.

  • Reward-induced distributions: Reward-labeled policy samples induce aligned and unaligned distributions through above- and below-average responses, respectively.The construction uses group responses sampled from the current policy and reward-based importance weights.
  • Reward-induced distributions: Reward-induced importance weights use softmax scores and truncated indicators to emphasize responses that distinguish aligned from unaligned behavior.Above-average responses contribute to the aligned term, while below-average responses contribute to the unaligned term, with correction for sampling-policy likelihood.
  • Divergence-based objective: The group-level objective plugs these reward-induced weights into an importance-sampled FDO formulation for divergence estimation.Its two branches estimate aligned and unaligned contributions from above- and below-average reward responses.
  • f-GRPO objective: The practical f-GRPO loss replaces sign-based weighting with scaled advantages while preserving update direction and matching standard on-policy gradient scale.The advantage magnitude is used to accelerate learning and align the gradient scale with standard on-policy reinforcement learning objectives.
  • f-HAL objective: f-HAL combines on-policy reward optimization with off-policy preference alignment through an interpolation parameter λ.λ = 1 yields FDO, λ = 0 yields f-GRPO, and intermediate values combine reward-driven exploration with preference guidance.

4 Main Results

The proposed objectives estimate divergences between reward- or mixture-aligned and unaligned distributions, yielding alignment-consistency and reward-improvement guarantees. f-HAL additionally combines preference and reward information, while f-GRPO provides sharper separation than GRPO under canonical links.

  • Divergence Estimation: At each on-policy iteration, f-GRPO and f-HAL estimate divergences between well-defined aligned and unaligned distributions, supporting reward-improvement guarantees.The aligned and unaligned distributions are induced by rewards for f-GRPO and by mixtures for f-HAL.
  • Reward Improvement: The fixed-point analysis characterizes iterates of the proposed on-policy and hybrid objectives, with f-HAL additionally requiring weak reward-density correspondence for its hybrid reward-improvement result.The assumption requires aligned density to be non-decreasing and unaligned density to be non-increasing in reward.
  • Divergence Estimation: f-GRPO uses above-average reward responses as aligned and below-average responses as unaligned under the current policy.This reward-induced construction replaces explicitly observed chosen and rejected samples in preference alignment.
  • Divergence Estimation: f-HAL estimates a divergence between mixture distributions combining preference-aligned samples with reward-aligned samples and preference-unaligned samples with reward-unaligned samples.The mixture lets f-HAL use reward feedback for exploration while retaining preference data as an anchor.
  • Alignment Consistency: Responses with larger aligned-to-unaligned density ratios are upweighted in f-HAL, while f-GRPO upweights responses above the previous policy’s average reward.These updates establish alignment consistency for the two objectives.
  • Reward Improvement: Under canonical links satisfying g^-1(f′∞)=∞, f-GRPO assigns mass only to above-average responses and monotonically improves reward until concentrating on maximum-reward responses.GRPO also increases above-average responses but retains nonzero probability on below-average responses after reweighting.

5 Experiments

Experiments evaluate f-GRPO on math-reasoning RLVR and f-HAL on safety alignment with learned rewards, comparing them against GRPO and direct-alignment baselines. f-GRPO generally improves over GRPO, while hybrid f-HAL mitigates reward hacking and benefits larger models more strongly.

  • Math Reasoning (RLVR Task): f-GRPO is evaluated on GSM8K, LIMR, and OpenRS using 1.5B and 7B Qwen2.5-Math models with binary verifiable correctness rewards.Pass@1 is measured across GSM8K(test), MATH500, AMC23, AIME24, and AIME25 using 16 samples per problem.
  • Math Reasoning (RLVR Task): f-GRPO improves over GRPO on most math benchmarks and achieves better aggregate Relative Overall Score and Average Rank.Across model sizes and datasets, it consistently matches or improves over GRPO, with strongest gains on LIMR and OpenRS.
  • Safety Alignment (Preference Alignment Task): Safety experiments compare direct aligners, GRPO, f-GRPO, and hybrid f-HAL on Qwen2.5-Base models with 1.5B and 7B parameters.The evaluation measures robustness through separation, attack success rate, and ToxiGen, and utility through MMLU-Pro, IFEval, and MuSR.
  • Safety Alignment (Preference Alignment Task): Within alignment classes, FDO variants of f-HAL generally outperform DPO and KTO, while f-GRPO consistently improves over GRPO across both model sizes.Safety results summarize average robustness, average utility, and their mean Total Score.
  • Safety Alignment (Preference Alignment Task): Hybrid f-HAL mitigates reward hacking by combining reward-based exploration with off-policy preference supervision when learned rewards are imperfect.Pure on-policy methods, especially GRPO, can converge to degenerate refusal behavior; f-HAL remains comparatively context-aware.
  • Safety Alignment (Preference Alignment Task): For Qwen2.5-7B, hybrid f-HAL often outperforms pure direct alignment, whereas direct aligners remain slightly stronger overall for Qwen2.5-1.5B.Hybrid f-HAL still substantially improves over pure on-policy reinforcement learning for the smaller model.

6 Conclusions

The paper presents a divergence-based framework extending preference-alignment divergence estimation to RLVR and proposes f-GRPO and f-HAL. It reports theoretical alignment and reward-improvement properties, alongside empirical gains in math reasoning and reduced reward hacking in safety alignment.

  • Conclusions: The framework extends divergence estimation from preference alignment to general LLM alignment, including reward-based and preference-based settings.It targets alignment as increasing desirable behavior and decreasing undesirable behavior.
  • Conclusions: f-GRPO and f-HAL are derived from variational representations of f-divergences as on-policy and hybrid alignment objectives.f-HAL combines on-policy reward optimization with off-policy preference supervision.
  • Conclusions: The objectives estimate divergences between reward- or mixture-aligned distributions, satisfy alignment-consistency properties, and improve expected reward after alignment.The framework interprets above- and below-average reward responses as reward-aligned and reward-unaligned behavior in RLVR.
  • Conclusions: Empirically, f-GRPO improves over GRPO on math reasoning, while hybrid f-HAL mitigates reward hacking in safety alignment with learned reward models.The two objectives are evaluated in the complementary regimes of verifiable rewards and preference-based safety alignment.
  • Limitations: The work is foundational rather than a search for a universally best divergence, and finite-sample training dynamics may differ across divergences.Experiments are limited by resource constraints to models up to 7B parameters.

A.4 Derivation of the FDO Objective

The FDO objective applies the variational representation of f-divergences to aligned and unaligned response distributions. At convergence, it recovers the negative f-divergence, providing the starting point for extension from preference alignment to RLVR.

  • FDO construction: FDO applies a variational representation of f-divergences to the aligned distribution D+ and unaligned distribution D−.The representation uses a convex function f and a variational function parameterized as T = g(rθ).
  • FDO construction: At convergence, the FDO objective recovers the negative f-divergence between D+ and D−.This follows from the optimality of the f-divergence variational representation.
  • Extension to RLVR: This variational FDO formulation serves as the starting point for extending divergence-based alignment from preference supervision to RLVR.The extension addresses settings where preference-aligned and unaligned samples are not explicitly observed.
  • Measure-theoretic foundation: The construction handles general positive measures through absolute-continuous and singular components in the Lebesgue decomposition.The appendix defines the divergence for both absolutely continuous and arbitrary P, Q cases.

C Proofs and Additional Theoretical Details

The appendix proves that monotone reward-based reweighting raises mean reward and establishes the divergence-estimation, alignment-consistency, and reward-improvement properties of f-GRPO and f-HAL. It also derives their sampled importance weights and connects the resulting objective to GRPO.

  • Auxiliary inequalities: The comonotone covariance inequality implies that non-decreasing reweighting functions produce nonnegative reward-weight covariance.The resulting mean-reward inequality is strict unless the reward or weight function is almost surely constant.
  • Auxiliary inequalities: Monotone reweighting increases the mean reward under the tilted distribution.This lemma supplies the key mechanism for the expected-reward improvement proof.
  • f-GRPO proof: As group size grows, empirical reward statistics converge to the previous policy’s mean and the induced empirical distribution converges to the reward-weighted form.The proof uses the strong law of large numbers and continuous mapping arguments.
  • Alignment consistency: The f-GRPO and f-HAL fixed-point updates are alignment-consistent because their density ratios are monotone in reward and upweight reward-aligned responses.For f-GRPO, responses above the previous policy’s average reward are upweighted; f-HAL uses mixture-aligned density.
  • Average reward improvement: Under the canonical-link conditions, each iteration increases average reward until the fixed point reaches the maximum reward for some prompt.The proof uses monotone reweighting and shows progressive support concentration toward higher-reward responses.
  • Reward-induced distributions: Reward-induced aligned and unaligned densities use higher rewards for aligned scoring and lower rewards for unaligned scoring, with truncation retaining above- or below-average responses.The construction motivates the f-GRPO weighting scheme and is not assumed to equal the true D+ and D− distributions.
  • Importance weighting: The sampled f-GRPO weights are softmax-normalized importance weights with a policy-sampling correction and advantage-sign truncation.The ±r term favors high- or low-reward responses, while −ln πθold(yj|x) corrects sampling from the old policy.
  • Resulting objective: The resulting loss generalizes GRPO by retaining standardized-advantage weighting while shaping the update through the chosen f-divergence.This preserves the divergence-estimator interpretation and the practical role of advantage-weighted on-policy optimization.

D Experiment Details

The experiments use the f-HAL training algorithm and evaluate math-reasoning models across multiple datasets, scales, and Pass@1 accuracy tables.

  • Training setup: f-HAL training combines on-policy rollouts with optional pairwise or binary off-policy supervision.The algorithm takes a policy, reference policy, divergence, link function, rollout settings, regularization, and optional supervision dataset.
  • Training setup: The training procedure samples prompt minibatches, generates responses with the behavior policy, computes rewards and advantages, and forms the on-policy update.The implementation initializes the behavior policy and on-policy gradient accumulator before processing rollouts.
  • Evaluation: Math-reasoning chat templates encourage standardized final-answer formatting with \boxed{} for automated evaluation.The template description applies to the math-reasoning side of the reported chat-template table.
  • Evaluation: The experiments report Pass@1 accuracy for Qwen2.5-Math-1.5B and Qwen2.5-Math-7B on LIMR, GSM8k, and OpenRS.Separate tables cover each model scale and dataset combination.

D.2 Separation Metric for Safety Alignment

Safety alignment is evaluated by measuring latent-space separation between safe and harmful prompt clusters before and after alignment. All aligned variants increase separation, with hybrid and off-policy objectives exceeding the on-policy method.

  • Separation metric: Latent-space separation represents the distance between safe and harmful prompt clusters using the Bhattacharyya distance DB.Prompt representations use the final-token hidden state, are reduced with PCA, and are visualized through the first two principal components.
  • Results: All aligned variants increase safe–harmful cluster separation relative to the base model.The separation is quantified after alignment with f-HAL using Jensen–Shannon divergence on Qwen-7B-Base.
  • Results: The on-policy fGRPO setting yields weaker separation than hybrid f-HAL and off-policy FDO.Figure 2 compares fGRPO at λ=0, f-HAL at λ=0.5, and FDO at λ=1.
  • Results: The post-safety-alignment robustness and utility evaluation reports AvgRob, AvgUtil, and their mean Total score, with pre-alignment Base separation equal to 1.16.AvgRob combines normalized separation scores and 100−ASR for AdvBench and SALAD.

D.4 Sample Generations Pre vs Post-Alignment (Safety Task)

The safety-generation examples compare base and aligned Qwen-7B responses under JS-based f-HAL across purely on-policy, hybrid, and off-policy settings. The examples emphasize safer aligned behavior on harmful prompts and stronger coherence in the hybrid setting.

  • Compared settings: JS-based f-HAL examples compare λ=0 f-GRPO, λ=0.5 hybrid alignment, and λ=1 FDO.These settings correspond respectively to purely on-policy, combined on/off-policy, and off-policy direct alignment.
  • Benign prompt: On a benign factual prompt, the hybrid setting often produces the most coherent answer.It leverages off-policy factual supervision while retaining on-policy reward-driven optimization.
  • Harmful prompt: On a harmful prompt, the base model produces unsafe procedural content, whereas aligned models provide robust safety-aligned refusals.The comparison is qualitative and uses JS-based f-HAL on Qwen-7B Base.

D.5 Sample Generations Pre vs Post-Alignment (Math Reasoning Task)

The math-reasoning example compares a base response with aligned-method responses on a prompt where the base model fails and aligned methods succeed.

  • Qualitative comparison: The base model receives score=0, while aligned methods receive score=1 on the qualitative math example.Responses are lightly truncated for space.
Loading 2602.05946v3…