PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Entropy-Controlled Policy Optimization

Guangran Cheng, Chengqi Lyu, Songyang Gao, Wenwei Zhang, Kai Chen

arXiv:2607.16850v1cs.CL

TL;DR

Heterogeneous task mixtures create different entropy regimes that make standard entropy regulation and GRPO advantages poorly suited to balancing exploration and exploitation across tasks. GEPO addresses this with group-entropy-conditioned asymmetric advantage shaping and achieves the best average performance across two models and thirteen benchmarks while maintaining balanced task improvements.

  • Problem

    Heterogeneous task mixtures create distinct entropy regimes and structurally biased normalized advantages, making cross-task optimization imbalanced.

  • Method

    GEPO uses rollout-derived group entropy and adaptive thresholds to asymmetrically shape advantages, moderating low-entropy reinforcement and high-entropy suppression.

  • Results

    Across two base models and thirteen benchmarks, GEPO achieves the best average performance among evaluated methods, including a highest average score of 71.2 on Qwen3.5-9B.

  • Takeaways & Limitations

    GEPO delivers balanced improvements across diverse task categories while preserving differentiated task-specific exploration levels without additional sampling.

  • Takeaways & Limitations

    The formal analysis assumes bounded advantage moments, limiting its scope to settings satisfying those assumptions.

Abstract

from arXiv · show

Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.

1. Introduction

LLM reinforcement learning must balance exploration and exploitation across heterogeneous task mixtures, where global or token-level entropy control overlooks task-specific optimization imbalance. GEPO addresses this through group-entropy-conditioned asymmetric advantage shaping and improves performance and inter-task balance without additional sampling cost.

  • Motivation: Heterogeneous mixtures of instruction following, coding, writing, and mathematical reasoning create task-specific differences in exploration, making cross-task exploration–exploitation balance a central challenge.The tasks differ in structure, solution diversity, and uncertainty of policy exploration.
  • Problem: Global- and token-level entropy-control methods overlook optimization imbalance among heterogeneous tasks, while GRPO’s reward standardization can produce biased normalized advantages.The passage identifies policy-level and token-level entropy mechanisms as insufficient for heterogeneous-task optimization.
  • Problem: Group entropy varies with task characteristics and can induce structural bias in GRPO’s normalized advantages, causing imbalanced optimization across tasks.Group entropy is calculated from the response groups of each task.
  • Method: GEPO is a lightweight, model-agnostic extension that uses group entropy as a diagnostic signal for entropy-conditioned asymmetric advantage shaping at the group level.The method is designed as an extension to related group-based policy optimization methods.
  • Results: Zero additional sampling cost accompanies GEPO’s improvements in average performance and inter-task balance across thirteen benchmarks using Intern-S1-mini and Qwen3.5-9B.The experiments also report state-of-the-art results among GRPO and entropy-aware RL methods and more stable optimization trajectories.

2. Method

GEPO extends GRPO with task-level group entropy, an unbiased Monte Carlo exploration signal normalized by response length. The method addresses entropy-dependent advantage distortion through adaptive, distribution-relative entropy control.

  • GRPO: GRPO estimates advantages from grouped responses and rewards, eliminating the need for a separate value function.For each prompt, GRPO samples K responses from the current policy and computes normalized advantages.
  • Group Entropy: Group entropy estimates policy uncertainty for each prompt, with low values indicating concentrated responses and high values indicating diverse exploration.The estimator is unbiased, has variance decreasing as O(K^-1), and is normalized by average response length for comparability.
  • Empirical Observations: Physics prompts center near ℋg ≈0.49, whereas general knowledge prompts center near ℋg ≈0.81, showing that a global entropy statistic cannot characterize all tasks.These domain-specific regimes occur under the same base policy.
  • Empirical Observations: Entropy regimes coincide with distinct advantage profiles, including Physics at ℋ̄g=0.49 and Instruction Following at ℋ̄g=0.78 with skewness = 2.32 and kurtosis = 4.84.The observed asymmetry also depends mechanically on lower group accuracy, which concentrates positive advantages on rare correct responses.
  • Optimization Consequences: Per-group centering and scaling do not create a common statistical reference frame, leaving advantage signals entropy dependent across prompts.Low-entropy groups can have larger policy-induced reward-distribution distortion, while batch optimization may become dominated by converging low-entropy tasks.
  • GEPO Design: GEPO uses task-aware group entropy compatible with GRPO’s grouped sampling and sets thresholds relative to the empirical entropy distribution rather than using fixed values.This accommodates variation across task mixtures, base models, and training stages without additional sampling procedures or computational overhead beyond GRPO.

3. Experiments

Experiments on Intern-S1-mini and Qwen3.5-9B across thirteen diverse benchmarks show that GEPO improves cross-task performance while maintaining smoother training and task-specific entropy regimes. Its gains are linked to stable, bounded entropy dynamics and differentiated exploration across heterogeneous tasks.

  • Performance Across Multiple Benchmarks: GEPO achieves the best performance on 7 of 13 benchmarks for Intern-S1-mini and the best overall average, spanning reasoning, physics, instruction following, and scientific understanding.GEPO improves challenging benchmarks including AIME25, IMO-Bench, and GPQA while also improving instruction following and scientific reasoning.
  • Performance Across Multiple Benchmarks: 71.2 average score: GEPO achieves the highest average on Qwen3.5-9B, ahead of Clip-Cov at 70.9 and KL-Cov at 69.9.Clip-Cov leads on AIME25, Physics, and instruction following, while KL-Cov leads on MMLU-Pro and ChartQAPro; GEPO achieves the best results on IMO-Bench and other benchmarks.
  • Training Stability: GEPO improves smoothly and monotonically across both models and all benchmarks, with notably less variance than the baselines.On Intern-S1-mini, GEPO shows clear upward trends with minimal fluctuation, while GRPO oscillates and AEPO has intermittent performance drops.
  • Training Stability: ∼85% to ∼40%: GRPO’s Qwen3.5-9B AIME25 accuracy collapses around step 170, alongside degradation on GPQA, MMMU-Pro, and MathVista.The collapse is attributed to unconstrained entropy dynamics under GRPO, whereas AEPO avoids complete collapse but continues to oscillate.
  • Training Dynamics of Entropy: GEPO maintains healthy, bounded policy entropy while preserving differentiated group entropy regimes suited to task-specific exploration.Unlike uniform entropy control, GEPO allows tasks requiring broader exploration to retain higher entropy and avoids forcing a common target across heterogeneous tasks.

4. Related Work

Related work frames policy entropy as a longstanding reinforcement-learning mechanism for exploration and highlights growing interest in entropy dynamics during LLM post-training. Recent studies examine how entropy relates to exploration, training stability, and reasoning performance.

  • Entropy in reinforcement learning: Policy entropy measures action uncertainty and exploration in reinforcement learning.Entropy regularization and maximum-entropy objectives encourage exploration in stochastic policies.
  • Entropy in reinforcement learning: Entropy regularization and maximum-entropy objectives are established mechanisms for encouraging exploration in stochastic policies.
  • Entropy in LLM post-training: Recent LLM post-training research investigates entropy dynamics in relation to exploration, training stability, and reasoning performance.

5. Conclusion

GEPO is a robust multi-task RL method for LLMs that uses rollout-derived group entropy and asymmetric advantage shaping to regulate exploration and exploitation without task annotations or extra sampling.

  • Method: GEPO applies group entropy estimated from existing rollouts to asymmetric advantage shaping in multi-task LLM reinforcement learning.The method is presented for robust multi-task reinforcement learning of large language models.
  • Method: Adaptive entropy boundaries moderate reinforcement in low-entropy groups and suppression in high-entropy groups.This asymmetric shaping design targets different entropy regimes across groups.
  • Benefits: GEPO regulates the exploration-exploitation trade-off without explicit task annotations or additional sampling.Its group entropy estimates come from existing rollouts rather than separately collected samples.

A. Advantage Asymmetry under Per-Group Normalization

Per-group normalization produces accuracy-dependent, asymmetric advantage distributions: as group accuracy falls, positive signals concentrate on few correct responses while negative signals spread across many incorrect responses. This asymmetry increases skewness, and heterogeneous prompt accuracies amplify pooled skewness beyond single-group predictions.

  • Advantage Asymmetry under Per-Group Normalization: Per-group normalization yields structurally different advantage distributions as group accuracy p changes.The analysis considers K responses with binary rewards, n+ correct responses, n− incorrect responses, and p = n+/K.
  • Advantage Asymmetry under Per-Group Normalization: As p→0, |A+|→∞ while |A−|→0, concentrating positive signal on few correct responses and distributing negative signal across many incorrect responses.Low-accuracy groups are empirically associated with higher group-entropy regimes in multi-task rollouts.
  • Advantage Asymmetry under Per-Group Normalization: Physics (p≈0.48) yields skewness ≈0.08, Math (p≈0.39) yields ≈0.45, and Instruction Following (p≈0.16) yields ≈1.86.Skewness is zero at p = 0.5, positive when p < 0.5, and grows without bound as p→0.
  • Advantage Asymmetry under Per-Group Normalization: Pooled task skewness differs from single-group predictions because prompts have varying accuracies p_j.Empirical pooled values are Physics ≈0, Math ≈0.75, and IF ≈2.32; heterogeneous accuracies amplify asymmetry beyond the single-group analysis.

B. Structural Bias from Entropy Heterogeneity

The appendix formalizes the setup underlying Propositions 2.1 and 2.2 and decomposes finite-sample error in group-based advantage computation.

  • Formal setup: The appendix provides the formal setup for Propositions 2.1 and 2.2.These propositions are stated in the main text.
  • Formal setup: It includes complete proofs of Propositions 2.1 and 2.2.
  • Error decomposition: A finite-sample error decomposition distinguishes the sources of estimation error in group-based advantage computation.

B.1. Formal Setup

The formal setup defines a finite response space for each prompt, samples grouped responses independently from the policy, and introduces a reference measure for comparing reward distributions. It characterizes this comparison through reference and policy-weighted reward CDFs, whose discrepancy depends on policy deviation from the reference measure.

  • Response space: For each prompt x, the response space 𝒜_x is finite, induced by the vocabulary and maximum generation length, with size N_x.The setup denotes this space by 𝒜_x and specifies |𝒜_x| = N_x.
  • Sampling: During training, group responses are sampled independently from the policy.
  • Reference measure: A reference measure ν_x is defined over 𝒜_x and may be uniform, with ν_x(a) = 1/N_x when explicitly stated.
  • Reward distributions: The setup compares a reference reward CDF F_νx(t) with a policy-weighted reward CDF F_πx(t), and their discrepancy is governed by policy deviation from the reference measure.F_νx(t) uses ν_x over responses with reward at most t, while F_πx(t) uses π_x over the same reward condition.

B.2. Proof of Proposition 2.1 (Distribution Distortion Bound)

The proof establishes the distribution distortion bound by analyzing threshold-defined level sets and then taking the supremum over those sets. The key inequality follows because these level sets are a subset of all measurable sets.

  • Level-set construction: For any threshold t, the proof defines the level set B_t := {a ∈ A_x: μ_x(a) ≤ t}.This threshold construction is the starting point of the argument.
  • Supremum step: Taking the supremum over all thresholds t is equivalent to taking it over the corresponding level sets B_t.The proof explicitly identifies the two supremum formulations.
  • Inequality justification: The first inequality holds because the family of level sets {B_t} is contained within the family of all measurable sets.Thus, optimizing over level sets cannot exceed the supremum over all measurable sets.

B.3. Proof of Proposition 2.2 (Advantage Bias Bound)

This section restates the advantage bias bound under bounded means and a positive policy standard deviation, beginning the proof with the mean difference.

  • Definitions: The proof defines policy-weighted and reference moments for the advantage-bias analysis.The supplied passage introduces these moments before the standardized advantages.
  • Definitions: It then introduces the corresponding standardized advantages.These standardized quantities follow the moment definitions in the proof setup.
  • Proof: The proof begins with Step 1, which analyzes the mean difference.This is the first explicitly listed proof step in the supplied passage.
Loading 2607.16850v1…