PaperlayerView exact revision on arXiv

Source-linked AI summary

REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization

Jian Hu, Jason Klein Liu, Haotian Xu, Wei Shen

arXiv:2501.03262v9cs.CLcs.LG

TL;DR

PPO’s critic network makes RLHF costly, while critic-free methods face biased and unstable local advantage normalization. REINFORCE++ uses global advantage normalization through variants tailored to general RLHF and complex reasoning, with reported gains in stability, efficiency, generalization, and agentic-task performance.

  • Problem

    PPO’s critic network creates substantial computational and memory overhead, while local normalization in critic-free methods is biased and unstable.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    REINFORCE++ is a critic-free framework using Global Advantage Normalization, with a general k ≥ 1 variant and a k > 1 baseline variant for complex tasks.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    REINFORCE++ shows superior token-efficiency and OOD generalization, while REINFORCE++w/ Baseline prevents overfitting and outperforms GRPO and PPO in complex agentic reasoning tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Global normalization provides an effectively unbiased estimate as batch size increases, retains critic-free efficiency, and supports stronger generalization.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Reinforcement Learning from Human Feedback~(RLHF) plays a crucial role in aligning Large Language Models~(LLMs). The dominant algorithm, Proximal Policy Optimization~(PPO), employs a critic network to estimate advantages, which introduces significant computational and memory overhead. To address this, a family of critic-free algorithms (e.g., GRPO, RLOO) has emerged. However, these methods typically rely on \textit{prompt-level (local)} advantage normalization, which suffers from inaccurate advantage estimation, a tendency to overfit, and, as we show, is a theoretically biased estimator. To solve these challenges, we introduce REINFORCE++, a critic-free framework centered on \textbf{Global Advantage Normalization}. By normalizing advantages across the entire global batch rather than small, prompt-specific groups, our method provides a more stable and theoretically sound, \textit{effectively unbiased} estimate (whose bias vanishes as batch size increases). We introduce two variants: REINFORCE++, a highly efficient and general algorithm ($k \ge 1$) for general-domain RLHF, and REINFORCE++ /w baseline, a robust group-sampling variant ($k > 1$) for complex reasoning tasks. Our empirical evaluation demonstrates that each variant shows superior stability and performance in its respective domain, outperforming existing methods and even PPO in complex agentic settings.

1 Introduction

RLHF alignment commonly uses PPO, whose critic network adds substantial computational and memory costs. REINFORCE++ addresses critic-free advantage-estimation issues with global normalization and two variants for general RLHF and complex agentic tasks.

  • PPO uses a critic network to estimate advantages, creating substantial computational and memory overhead for LLM alignment.Source: §1 · p. 1
  • Critic-free methods avoid the critic by estimating advantages from multiple responses to the same prompt, but local normalization is mathematically biased.Source: §1 · p. 1
  • Small local groups can produce high-variance advantages and instability, especially when sampled responses receive similar rewards.Source: §1 · pp. 1–2
  • REINFORCE++ proposes a critic-free framework centered on Global Advantage Normalization as a stable, efficient, and theoretically sound alternative.Source: §1 · p. 2
  • REINFORCE++ supports k ≥ 1 for general-purpose RLHF and reasoning, while REINFORCE++w/ Baseline supports k > 1 for complex agentic tasks.Source: §1 · p. 2
  • REINFORCE++ achieves superior token-efficiency and OOD generalization, while its baseline variant prevents overfitting and outperforms GRPO and PPO in complex agentic reasoning tasks.Source: §1 · p. 2

2 Background and Related Work

PPO estimates advantages with a critic, while critic-free alternatives use reward statistics from sampled responses. Local normalization in methods such as GRPO introduces theoretical bias, instability, and task overfitting.

  • PPO optimizes a surrogate objective using a probability ratio and typically calculates advantages with Generalized Advantage Estimation.Source: §2.1 · p. 2
  • The temporal-difference error used by GAE depends on V(·), the critic network.Source: §2.1 · p. 2
  • Critic-free methods remove V(·) and compute advantages directly from rewards, using alternatives such as greedy, leave-one-out, or local-group baselines.Source: §2.1 · p. 2
  • Prompt-level normalization is theoretically biased because centered rewards and the local standard deviation are correlated within small groups.Source: §2.2 · pp. 2–3
  • Figure 1 compares PPO, ReMax, GRPO, RLOO, and REINFORCE++, including the critic-free and normalization differences among them.Source: Figure 1 · p. 3
  • Small group sizes can make the local standard deviation approach zero, causing exploding advantages, high variance, and unstable training.Source: §2.2 · p. 3
  • Local normalization rewards responses for outperforming same-prompt samples rather than being globally good, which can cause task overfitting.Source: §2.2 · p. 3

3 Method

REINFORCE++ replaces biased or unavailable local normalization with global batch normalization in a critic-free PPO-style framework. It provides efficient single-sample training and a group-sampling variant for complex tasks.

REINFORCE++

  • REINFORCE++ uses k = 1 for efficient, prompt-diverse general-purpose RLHF and can also operate with k > 1.The primary algorithm retains a critic-free architecture and maximizes prompt diversity in its k = 1 configuration.Source: §3.1 · p. 3, §3.4 · p. 4

Global Advantage Normalization

  • Global normalization makes the advantage estimator effectively less biased as batch size grows and improves training stability.The method is described as effectively unbiased as N →∞ and robust to outliers.Source: §3.1 · p. 3, §3.4 · p. 4

REINFORCE++w/ Baseline

  • REINFORCE++w/ Baseline combines group sampling with group-mean reward subtraction and global batch normalization for complex tasks.The two-step design reshapes rewards with a local baseline, then uses global statistics for stability.Source: §3.2 · pp. 3–4
  • The baseline variant uses a separate KL loss with the k2 estimator, which is described as stable and unbiased for Reverse KL gradients.The paper contrasts k2 with GRPO’s unstable k3 approximation.Source: §3.2 · p. 4

Global Advantage Normalization

  • REINFORCE++ normalizes advantages across the entire global training batch instead of within prompt-specific groups.The global batch mean and standard deviation converge toward stable constants as batch size increases.Source: §3.1 · p. 3, §3.4 · p. 4

Critic-Free Design

  • Both variants remove the critic network while retaining critic-free efficiency and reducing computational and memory overhead compared with PPO.The baseline variant is characterized as formally equivalent to PPO with the critic removed and global batch normalization replacing a learned value function.Source: §3.4 · p. 4, §3.3 · p. 4

4 Experiments

The experiments evaluate REINFORCE++ variants in general-domain RLHF, reasoning, and agentic settings. Results show token-efficient performance, more stable training, stronger OOD generalization, and reduced overfitting relative to local-normalization methods.

General-Domain RLHF

  • A score of 46.7 for REINFORCE++ (k = 1) statistically ties GRPO’s 46.8 while using 832 tokens versus 860 and achieving a per-token score of 0.0561.The result indicates that group sampling is not necessary for the general-task setting evaluated.Source: §4.1 · p. 5
  • REINFORCE++ increases reward more stably than GRPO while maintaining much smaller KL divergence, avoiding the reward-hacking behavior observed with GRPO.The analysis links local-normalization reward hacking to length exploitation.Source: §4.1 · p. 5, Figure 2 · p. 5

Agentic Reasoning

  • REINFORCE++w/ Baseline is evaluated in complex reasoning settings where group sampling and reward reshaping are useful, including sparse-reward tasks.The evaluation directly compares it with GRPO in complex reasoning.Source: §4.2 · p. 5, §4.3 · p. 5

Complex Reasoning

  • On 30 AIME-24 training questions, GRPO reaches 95.0% training accuracy but 0.0 Pass@1 on AIME-25, whereas REINFORCE++ reaches 71.0 training accuracy, 2.5 Pass@1, and 40.0 Pass@16.The comparison demonstrates catastrophic overfitting for GRPO and better held-out generalization for REINFORCE++.Source: §4.2.1 · p. 5
  • On Knights and Knaves puzzles, REINFORCE++ outperforms GRPO on tasks with four or more people and averages 62.1 versus 55.7.GRPO remains competitive on easier 2–3-person tasks but collapses on harder OOD tasks with 8 people.Source: §4.2.1 · p. 5

RL from Zero

  • REINFORCE++ shows better OOD generalization on AIME-24 and AMC-23 while remaining competitive on the in-distribution MATH-500 test set.This evaluation trains a Qwen2.5-Math-Base model from zero on MATH dataset splits.Source: §4.2.1 · p. 5

Appendix Section 5).

The reported tables and figures summarize REINFORCE++’s efficiency, generalization, and robustness across instruction-following, small-data, RL-from-zero, and logic-benchmark evaluations.

Agentic Evaluation

  • The agentic evaluation uses average@32 across AIME 2024, AIME-2025, HMMT FEB-2024/2025, and CMIMC benchmarks.The backbone is Qwen 2.5 Base 7B, trained with OpenRLHF on ORZ and DAPO datasets.Source: Appendix Section 5) · pp. 5–6
  • REINFORCE++w/ Baseline achieves the highest average accuracy of 24.10, exceeding GRPO at 22.58 and full-critic PPO at 21.85.The result is reported across the agentic benchmarks.Source: Appendix Section 5) · pp. 6–7

5 Best Practices

The paper recommends choosing between the two REINFORCE++ variants according to prompt diversity, sampling constraints, sample quality, and task complexity. Subsequent studies report supporting evidence for global normalization’s stability and generalization.

Variant Selection

  • REINFORCE++w/ Baseline is recommended for sample filtering and complex settings with many void or non-informative samples.Subtracting the intra-group mean reward is described as improving stability by filtering void samples.Source: §5.1 · p. 7
  • REINFORCE++ (k = 1) is recommended when prompt diversity and efficiency matter or multiple reward signals are difficult to obtain.The paper specifically mentions PRMs and online realtime sampling as such settings.Source: §5.1 · p. 7

Subsequent Evidence

  • The paper states that global advantage normalization has been independently validated and adopted in subsequent large-scale reasoning systems.This claim is presented as evidence of the principle’s stability and effectiveness.Source: §5.2 · p. 7
  • LitePPO reports more stable training and better generalization with global rather than GRPO’s local standard deviation.LitePPO combines REINFORCE++w/ Baseline with a token-level loss.Source: §5.2 · p. 7
  • ScaleRL finds batch-level normalization slightly superior in compute efficiency and final performance to prompt-level normalization in 16,000 GPU-hour experiments.The comparison directly evaluates REINFORCE++ against GRPO-style normalization at scale.Source: §5.2 · p. 7
  • DLER reports stable batch-wise normalization under output truncation while group-wise normalization shows declining accuracy.The study supports robustness to changing training conditions and reward landscapes.Source: §5.2 · p. 7

6 Conclusion

REINFORCE++ addresses bias and instability in critic-free RLHF by replacing local advantage normalization with global normalization and offering two use-case-specific variants. The reported results show strong generalization and improved performance on complex agentic tasks.

  • Prompt-level advantage normalization is theoretically biased and practically unstable in prior critic-free methods.Source: §6 · p. 7
  • Global Advantage Normalization provides a stable and effectively unbiased estimator by normalizing across the entire batch.Its bias vanishes as batch size increases.Source: §6 · p. 7
  • REINFORCE++ supports k ≥ 1 for general-domain RLHF, including efficient single-sample training and robust k > 1 reasoning applications.Source: §6 · p. 7
  • REINFORCE++w/ Baseline combines group sampling with global normalization for complex reasoning and agent tasks.The variant also uses group-mean reshaping and a k > 1 KL estimator.Source: §6 · pp. 7–8
  • REINFORCE++w/ Baseline outperforms both GRPO and critic-based PPO on complex tool-use benchmarks.Source: Table 4 · p. 8
  • REINFORCE++ achieves state-of-the-art generalization, while its baseline variant prevents low-data overfitting and outperforms GRPO and PPO on long-horizon tool-use tasks.Source: §6 · p. 8

A.1 Assumptions and Settings

The proof models rewards as a true baseline plus independent Gaussian noise and establishes that the finite-sample prompt-level advantage estimator is biased. As sample size grows, its denominator converges to a constant and the bias vanishes, motivating global batch normalization.

  • Rewards are modeled as ri = θ + ϵi, with independent Gaussian noise ϵi ∼ N(0, σ2).Source: §A.1 · p. 11
  • For any finite N ≥ 2, the prompt-level advantage estimator Ai is biased.Source: §A.1 · p. 11
  • The proof analyzes numerator bias and shows that the estimator denominator depends on the same noise term ϵi.Source: §A.1 · pp. 11–12
  • Because the resulting conditional expectation is not constant, Ai cannot be an unbiased estimator.Source: §A.1 · p. 12
  • As N → ∞, the denominator converges to σ and the numerator bias vanishes.Source: §A.2 · p. 13
  • Global-batch statistics become effectively constant as Nglobal increases, making the estimator effectively unbiased in the large-batch limit.The paper contrasts typical global batches such as 1024 with group batches such as 4 or 8.Source: §A.2 · p. 13

B.1 KL Penalty Design

The section evaluates KL estimators for a separate regularization loss under sampling from πθ, concluding that k2 correctly and stably estimates the Reverse KL gradient while k1 and k3 are unsuitable.

  • Sampling from πθ means the separate KL loss targets the Reverse KL, DKL(πθ||πref).Source: §B.1 · p. 14
  • k1 provides no constraining effect because its gradient does not depend on πref.The paper notes that k1 is used inside the REINFORCE++ reward, but not as a separate loss term.Source: §B.1 · p. 14
  • k2 provides a gradient equivalent to the theoretical Reverse KL gradient and matches the practical RKL gradient.The paper therefore adopts k2 for the KL loss in REINFORCE++-Baseline.Source: §B.1 · p. 14, Conclusion · p. 15
  • k3 estimates the Forward KL rather than the Reverse KL, despite samples being drawn from πθ.The Forward KL gradient requires sampling from πref, creating a mismatch.Source: §B.1 · p. 15
  • The k3-based method has high variance and numerical instability, motivating frequent resets of πref to prevent divergence.The cited issues arise from exploding importance weights and overflow-prone exp(log p − log q) calculations.Source: §B.1 · p. 15

B.2 Implementation Tricks

REINFORCE++ uses token-level advantages with the normalized reward assigned to the final token, alongside global-batch processing and reward transformations intended to stabilize training.

  • For t < T, the token-level advantage is 0; at t = T, it equals the normalized reward A_norm.This is described as standard practice in RLHF for LLMs.Source: §B.2 · p. 15
  • Global normalization computes mean and standard deviation over all N samples in the batch, including group-sampled batches.For REINFORCE++-Baseline with k = 4, N = 1024 can represent 256 unique prompts.Source: §B.2 · p. 15
  • Mini-batch updates process data in smaller chunks, permit multiple parameter updates, and introduce stochasticity during optimization.Source: §B.2 · p. 15
  • Reward processing combines z-score normalization, clipping, and scaling to mitigate outliers, avoid instability, and support numerical stability.Source: §B.2 · p. 15

Related papers

Loading 2501.03262v9…