PaperlayerView exact revision on arXiv

Source-linked AI summary

Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving

Yiming Wang, Yikang Liu, Qingyuan Tian, Xingyu Chen, Zhuosheng Zhang, Zhaopeng Tu, Rui Wang

arXiv:2609.36750v1cs.LGcs.AIcs.CL

TL;DR

A self-rewarding response’s reward representation depends on its randomly sampled group context, so using a single realization may miss the desired signal. GMAE aggregates rewards over possible contexts to estimate expected advantages; across benchmarks it outperforms existing baselines and generalizes beyond mathematical training data.

  • Problem

    Self-rewarding methods construct references from rollout groups, making a response’s reward context-dependent; a single realization may miss the desired signal and make policy updates less reliable.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    GMAE aggregates reward realizations across possible group contexts into a response-level distribution and uses it to estimate expected advantage.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Across 30 of 32 model-benchmark combinations, GMAE outperforms existing self-rewarding baselines, with 2.0–2.8-point gains over the strongest baselines.

    Source: §5.2 · p. 7

  • Takeaways & Limitations

    Experiments support GMAE as a reliable general framework for zero-label self-evolution, with stable training and applicability across models and benchmarks.

    Source: §1 · p. 2, §8 · p. 9

    Self-rewarding rewards remain dependent on the sampled rollout group because the pseudo-label is constructed from that group.

    Source: §3 · p. 3

Abstract

from arXiv · show

Self-rewarding reinforcement learning (RL) enables large language models (LLMs) to self-evolve without human labels. Existing ensemble-based methods construct reward references from rollout groups and assign rewards accordingly. However, a response's reward representation also depends on its randomly sampled group context, i.e., the other responses in its group. Using only one group-context realization may miss desired reward signals and provide unreliable guidance for policy optimization. To address this issue, we propose Group-Marginalized Advantage Estimation (GMAE), which aggregates reward realizations across possible contexts into a response-level distribution and estimates expected advantages. Experiments across eight benchmarks and four base models demonstrate strong performance and cross-domain generalization. GMAE also exhibits stable learning, low extra cost, and good applicability across training datasets and RL backbones.

1 INTRODUCTION

Self-rewarding RL uses self-generated rewards for policy optimization without human labels, but existing approaches rely on signals whose reliability can be limited. GMAE aggregates reward realizations across group contexts, and experiments report strong performance and cross-domain generalization.

1 INTRODUCTION

  • Self-evolution without human labels uses environmental and data-derived signals; self-rewarding RL optimizes policies with self-generated rewards rather than ground-truth labels.The introduction frames zero-label learning as a fundamental setting for autonomous model evolution.Source: §1 · p. 1
  • Probability-based methods estimate response quality from token probabilities or entropy, while ensemble-based methods use rollout groups to construct reward references such as majority-vote pseudo-labels [Wang et al., 2023a; Zuo et al., 2026].Subsequent ensemble methods refine reference construction or reward functions [Zhang et al., 2026a; Wang et al., 2026a; Wu et al., 2026].Source: §1 · pp. 1–2
  • A response’s reward can depend on the other responses in its randomly sampled rollout group, so one context may miss the desired signal and reduce policy-update reliability.The missed signal cannot guide subsequent policy optimization.Source: §1 · p. 2
  • GMAE aggregates reward realizations across possible group contexts into a response-level distribution and uses it to estimate expected advantage.It is presented as a general strategy compatible with different reward-reference constructions and reward functions.Source: §1 · p. 2
  • Across four base models and eight benchmarks, GMAE generalizes from mathematical training data to unseen domains and shows stable learning with low variance across seeds.Further experiments support applicability across training datasets and RL backbones.Source: §1 · p. 2

2 PRELIMINARIES: RLVR AND SELF-REWARDING RL

The preliminaries contrast RLVR, which uses ground-truth answers as reward references, with self-rewarding RL, which infers references from sampled rollout groups. They also outline GRPO’s group-relative advantage setup.

2 PRELIMINARIES: RLVR AND SELF-REWARDING RL

  • GRPO samples G responses for a prompt, assigns each a scalar reward, and obtains advantages through group-wise normalization.The response-level advantage is shared across tokens, and the objective uses clipping and KL regularization.Source: §2 · p. 2
  • In standard RLVR, each response’s reward is computed against the ground-truth answer as a fixed external reference.This reference is denoted a_true in the passage.Source: §2 · p. 2
  • Figure 2 depicts reward realizations across group contexts for fixed correct and incorrect responses, plus self-rewarding and RLVR advantage distributions for the same responses.The caption identifies four self-rewarding methods and results from Qwen3-8B-Base on a DAPO-14K prompt.Source: Figure 2 · p. 3
  • Self-rewarding RL constructs a pseudo-label from the sampled rollout group and evaluates responses against that reference; majority voting is a common choice [Wang et al., 2023a; Zuo et al., 2026].Other methods refine reference construction or the reward function while retaining this pipeline.Source: §2 · p. 3

3 ON THE LIMITATION OF SELF-REWARD UNCERTAINTY

Self-rewarding rewards and advantages can vary with the randomly sampled group context, even for a fixed response. The section examines this variation and its implications for optimization reliability.

3 ON THE LIMITATION OF SELF-REWARD UNCERTAINTY

  • A response’s reward depends on its rollout group because the pseudo-label is constructed from the entire group, making the assigned reward context-dependent.The group is decomposed into a fixed response and its accompanying context to make that dependence explicit.Source: §3 · p. 3
  • For a fixed response, the accompanying group context is randomly sampled, so its reward—and resulting learning signal—depends on both the response and context.The passage defines reward over a generic context drawn from the group-context distribution.Source: §3 · p. 3
  • The empirical comparison fixes one correct and one incorrect response, samples 100 independent contexts of 15 other responses each, and compares self-rewarding with RLVR advantages.The context size follows G = 16, with the focal response excluded from the newly sampled group members.Source: §3 · p. 3
  • For fixed responses, self-rewarding rewards vary across contexts and can take the opposite of the desired signal; self-rewarding advantages often change sign, unlike RLVR advantages.RLVR advantages preserve the desired direction: correct responses remain non-negative and incorrect responses non-positive.Source: §3 · pp. 3–4

4 GMAE: GROUP-MARGINALIZED ADVANTAGE ESTIMATION

GMAE addresses the risk that a response’s reward signal depends on its sampled group context by aggregating reward realizations into a response-level distribution. Its core algorithm estimates advantages from these distributions, calibrates their scale, and can be paired with different self-rewarding rules.

4 GMAE: GROUP-MARGINALIZED ADVANTAGE ESTIMATION

  • GMAE aggregates reward realizations across group contexts into a response-level representation to improve the chance of retaining desired reward signals.The suitable context depends on the response and is unknown in advance.Source: §4 · p. 4

4.1 CORE ALGORITHM

  • For each response, GMAE forms a reward distribution over possible group contexts, then estimates advantage by drawing reward realizations from response-level distributions.The algorithm takes the expectation of group-normalized advantages under the product distribution.Source: §4.1 · p. 4
  • A shared candidate pool and sampled distinct contexts approximate marginalization; the error bound decreases with K and G′ but increases with the reward-value space size |Vi|.The paper sets τ = 10^4 after increasing evaluated contexts tenfold from 10 produced negligible changes in the empirical reward distribution.Source: §4.1 · p. 4
  • Because marginalization often reduces advantage scale, GMAE uses batch-level calibration to restore overall scale while preserving relative differences across prompts.Prompt-specific scale differences can reflect reward uncertainty, so calibrating each prompt separately would erase those differences.Source: §4.1 · p. 5

4.2 INSTANTIATING GMAE WITH SELF-REWARDING RULES

  • GMAE is agnostic to reward-reference construction and reward function; its binary-reward instantiations use standard voting, co-refinement, or self-refinement references.Binary rewards also restrict the reward-value space, reducing enumeration size and tightening the approximation bound.Source: §4.2 · p. 5

5 EXPERIMENTS

Experiments compare GMAE with self-rewarding baselines across models and benchmarks, then examine training stability, seed variance, and cost. GMAE improves benchmark results and shows steadier, lower-variance training with limited overhead.

5.1 SETUP

  • Experiments use four base models, eight benchmarks, DAPO-14K for training, and GRPO with 16 main and 16 auxiliary responses per prompt.The evaluation includes mathematics, science, knowledge, and coding; checkpoints are selected using DAPO-500 validation MEAN@16.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • GMAE outperforms self-rewarding baselines in 30 of 32 model-benchmark combinations, with 2.0–2.8-point gains and strong out-of-domain performance after math-only training.On Qwen3-1.7B-Base, the gap to RLVR narrows to 1.0 point.Source: §5.2 · p. 7
  • GMAE consistently improves its paired base methods, with average relative gains of 8.2%, 10.3%, and 9.5% for GMAE0, GMAECR, and GMAESR, respectively.Even the weakest GMAE variant outperforms the strongest non-GMAE baseline across evaluated models.Source: §5.2 · p. 7

5.3 DETAILED TRAINING BEHAVIOR

  • GMAE improves steadily without collapse on DAPO-500, whereas several baselines, including both probability-based methods, show a rise-then-collapse pattern.Figure 3 tracks validation performance over training steps.Source: §5.3 · p. 7, Figure 3 · p. 7
  • Across five seeds, GMAE0 has the lowest average MEAN@16 standard deviation at 0.48, and GMAE variants reduce variance in each reported paired comparison.The reported reductions are TTRL → GMAE0 (0.69 → 0.48), Co-Reward → GMAECR (0.76 → 0.55), and SR-TTRL → GMAESR (0.72 → 0.57).Source: Figure 4 · p. 7, §5.3 · pp. 7–8
  • GMAE adds little overhead over corresponding base methods because it reuses cached responses and uses lightweight scalar computation.GMAE0 adds G(K −1) voting operations and at most 2G scalar normalizations over TTRL; more elaborate variants incur additional computation while delivering stronger performance.Source: §5.3 · p. 8

6 EXTENDED ANALYSIS

Extended analyses examine how GMAE relates to correct-response frequencies, auxiliary response size, advantage calibration, and training-setting scalability. They report differences in response-count distributions, a performance–rollout-cost balance, and retained gains across settings.

6.1 WHY DOES GMAE WORK? A CORRECT-RESPONSE FREQUENCY ARGUMENT

  • At steps 50 and 200, GMAE0 has more prompts with small or moderate correct-response counts than TTRL, while both have almost the same mass at 16 correct responses.TTRL has more prompts with no correct responses; the authors interpret the pattern as preserving rare correct responses while maintaining consistently solved prompts.Source: §6.1 · p. 8, Figure 6 · p. 8

6.2 ABLATION STUDY

  • Increasing auxiliary response size G′ consistently improves performance, but the marginal benefit drops markedly beyond G′ = 16.The authors identify G′ = 16 as a balance between performance and rollout cost; the ablation compares average MEAN@16 across eight benchmarks.Source: §6.2 · p. 8, Figure 7 · p. 8
  • Batch-level advantage calibration performs best on both tested models, while prompt-level calibration performs worst and underperforms no calibration.Table 2 reports the GMAE0 calibration ablation on Qwen3-8B-Base.Source: §6.2 · p. 8, Table 2 · p. 9
  • The calibration factor λ stays between 1.15 and 1.35 and gradually decreases during training, as the non-marginalized and marginalized advantage scales narrow.Figure 8 tracks λ over training.Source: §6.2 · p. 9

6.3 SCALABILITY STUDY

  • Across RL-backbone settings, replacing GRPO with GSPO or REINFORCE++ while holding other settings fixed, and across training on Open-RS, MATH-8K, and DAPO-14K, GMAE consistently retains advantages over representative strong baselines.Source: §6.3 · p. 9

7 RELATED WORK

Related work covers self-evolving language models and self-rewarding RL, including approaches that use generated data, reasoning, policies, and rollout-based supervision. The paper situates GMAE in zero-label evolution and as a strategy addressing ensemble methods’ dependence on sampled rollout groups.

Self-Evolving LLMs

  • Self-evolving LLMs improve through self-generated experience, including data evolution, iterative reasoning, and policy or task evolution, but many approaches require demonstrations, labels, environmental feedback, or verifiers.The paper studies zero-label evolution from prompts without ground-truth answers or external reward models.Source: §7 · p. 9

Self-Rewarding RL

  • Self-rewarding RL uses self-generated supervision, with probability-based methods scoring individual rollouts and ensemble-based methods deriving pseudosupervision from multiple rollouts.The cited ensemble approaches include answer-frequency consensus [Zuo et al., 2026], trajectory consistency [Zhang et al., 2026a], and other voting, agreement, and reflection methods.Source: §7 · p. 9
  • GMAE addresses self-rewarding methods’ shared dependence on sampled rollout groups while supporting different self-reward rules.The related-work discussion contrasts group-dependent ensemble approaches with probability-based confidence, self-certainty, and entropy methods.Source: §7 · p. 9

8 CONCLUSION

The conclusion presents GMAE as a general advantage-estimation strategy for self-rewarding RL and reports consistent improvements and stable training across models and benchmarks.

8 CONCLUSION

  • GMAE marginalizes rewards over possible group contexts to provide more reliable optimization signals while remaining compatible with different self-rewarding rules.The conclusion reports consistent improvements and stable training across multiple models and benchmarks.Source: §8 · p. 9
  • The authors present GMAE as an effective approach to zero-label self-evolution.This conclusion is based on experiments across multiple models and benchmarks.Source: §8 · p. 9
  • The conclusion characterizes GMAE as a general strategy for self-rewarding RL rather than a method tied to one reward rule.It reports compatibility with different self-rewarding rules.Source: §8 · p. 9

AI USE STATEMENT

The appendix documents AI-assisted writing and retrieval, illustrates additional self-reward uncertainty cases, and describes several baseline reward constructions.

AI USE STATEMENT

  • The authors used generative AI for writing polishing and related-work retrieval, verified the generated content, and take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A ADDITIONAL DEMONSTRATIONS OF SELF-REWARD UNCERTAINTY

  • Four additional uncertainty cases use one correct and one incorrect Qwen3-8B-Base response per DAPO-14K prompt, each tested under 100 contexts of 16 responses.The responses’ rewards and group-normalized advantages are compared across four self-rewarding methods and with RLVR; Figures 9–12 provide the additional cases.Source: A ADDITIONAL DEMONSTRATIONS OF… · p. 13, Figure 9 · p. 13, Figure 10 · p. 13

B.1 TTRL

  • TTRL rewards answers matching the group’s majority, while CoVo derives intrinsic rewards from aggregated reasoning-state signals and adds a curiosity reward with a KL penalty.CoVo also defines response consistency and volatility; its intrinsic reward is the normalized magnitude of the aggregated vector for the response’s answer group.Source: §B.1 · p. 14, §B.2 · p. 15

B.3 SCOPE

  • SCOPE partitions responses into subgroups, uses confidence-weighted voting on bootstrap samples, and assigns each response its subgroup’s reference.The subgroup size is dynamically selected using a Pareto criterion balancing agreement and label diversity.Source: §B.3 · pp. 15–16

B.4 SELF-HARMONY

  • Self-Harmony selects a shared reference using answer frequencies across original and reframed prompts, whereas RESTRAIN retains and weights distinct candidate answers.Self-Harmony rewards agreement with its reference; RESTRAIN evaluates each response against every candidate answer.Source: §B.4 · p. 16, §B.5 · p. 16

B.6 CO-REWARDING

  • Co-Rewarding uses a slowly updated teacher group’s majority answer as the reference, while SR-TTRL explores answers, summarizes representative reasoning, and reflects to select a pseudo-label.SR-TTRL then assigns binary rewards by agreement with that selected reference.Source: §B.6 · p. 17, §B.7 · p. 17

C.1 APPROXIMATION ERROR OF SHARED-POOL ESTIMATION

The section analyzes shared-pool estimation error and relates context-induced reward uncertainty to the scale of GMAE’s advantages.

C.1 APPROXIMATION ERROR OF SHARED-POOL ESTIMATION

  • The approximation analysis assumes the ideal reward distribution has finite support and separates error into context subsampling and finite-pool sampling components.The theorem addresses the approximation between the ideal group-marginalized distribution and its shared-pool estimate.Source: §C.1 · p. 18
  • The proof bounds the context-subsampling and finite-pool errors separately, then combines them to obtain a probability guarantee for the total approximation error.Each component exceeds its corresponding bound with probability at most δ/2.Source: §C.1 · pp. 18, 20

C.2 UNCERTAINTY-AWARE ADVANTAGE SCALE

  • Context-induced reward uncertainty includes realizations with no relative reward information and disagreement between relative reward assignments across realizations.The theorem relates this uncertainty to the prompt-wise GMAE scale.Source: §C.2 · p. 20
  • Greater context-induced reward uncertainty yields a smaller prompt-wise GMAE scale, making updates more conservative when a response’s normalized advantage changes sign across realizations.Marginalization strengthens cancellation between realizations that favor and disfavor the same response.Source: §C.2 · pp. 20–21

D DETAILED IMPLEMENTATION OF GMAE INSTANTIATION

The implementation instantiates GMAE with three reward-reference strategies, reusing candidate responses and estimating advantages across sampled contexts.

D DETAILED IMPLEMENTATION OF GMAE INSTANTIATION

  • All three instantiations use binary agreement with the reward reference, estimate marginalized advantages from multiple reward realizations, and apply batch-level calibration.Each requires G′ auxiliary rollouts beyond its base method.Source: D DETAILED IMPLEMENTATION OF GMAE… · p. 21

D.1 GMAE0: STANDARD VOTING

  • GMAE0 uses majority voting and samples additional contexts without replacement from a shared candidate pool for each focal response.Compared with TTRL, its added operations require neither additional model generation nor backpropagation.Source: §D.1 · p. 22

D.2 GMAECR: CO-REFINEMENT

  • GMAECR uses majority voting over co-teacher response groups, and each context’s reference is reused across all main responses.The co-teacher update is inherited from Co-Rewarding and adds no GMAE-specific computation.Source: §D.2 · p. 22

D.3 GMAESR: SELF-REFINEMENT

  • GMAESR summarizes representative responses for candidate answers and performs self-reflection once to rank them, then uses the highest-ranked answer present in each context.It reuses that ranking across contexts rather than repeating reflection; compared with SR-TTRL, it adds at most G′ summary operations and no additional reflection operations.Source: §D.3 · p. 23

E FULL SCALABILITY ANALYSIS RESULTS

The scalability section provides benchmark-wise MEAN@16 tables for two RL backbones and two training datasets.

E FULL SCALABILITY ANALYSIS RESULTS

  • Table 4 reports benchmark-wise MEAN@16 results for Qwen3-8B-Base using GSPO as the RL backbone.Source: Table 4 · p. 24
  • Table 5 reports benchmark-wise MEAN@16 results for Qwen3-8B-Base using REINFORCE++ as the RL backbone.Source: Table 5 · p. 25
  • Tables 6 and 7 report MEAN@16 results for Qwen3-8B-Base trained with Open-RS and MATH-8K, respectively.Source: Table 6 · p. 25, Table 7 · p. 25

Related papers

Loading 2609.36750v1…