PaperlayerView exact revision on arXiv

Source-linked AI summary

iGRPO: Self-Feedback-Driven LLM Reasoning

Ali Hatamizadeh, Shrimai Prabhumoye, Igor Gitman, Ximing Lu, Seungju Han, Wei Ping, Yejin Choi, Jan Kautz

arXiv:2602.09000v1cs.AI

TL;DR

Complex mathematical reasoning remains difficult despite reinforcement learning’s benefits, and existing RL frameworks generally lack iterative feedback on their own outputs. The paper introduces iGRPO, which selects a high-reward draft and conditions a second GRPO-style refinement stage on it. Under matched rollout budgets, iGRPO consistently outperforms GRPO across models and benchmarks, including 85.62% on AIME24 and 79.64% on AIME25 for OpenReasoning-Nemotron-7B trained on AceReason-Math.

  • Problem

    Existing RL frameworks do not capitalize on iterative self-feedback and refinement, limiting how they train models for complex reasoning.

  • Method

    iGRPO samples multiple drafts, selects the highest-reward one, appends it to the prompt, and applies a GRPO-style update to conditioned refinements.

  • Results

    iGRPO consistently outperforms vanilla GRPO across model families and scales; OpenReasoning-Nemotron-7B reaches 85.62% on AIME24 and 79.64% on AIME25.

  • Takeaways & Limitations

    The results support iterative self-conditioning as a refinement wrapper for verifiable mathematical reasoning and other group-based policy optimization variants.

  • Takeaways & Limitations

    The GRPO and iGRPO setup uses one scalar reward per sampled completion, giving all tokens in an output the same normalized advantage.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions. Reinforcement Learning (RL) is a framework for aligning these models with task-specific rewards, improving overall quality and reliability. Group Relative Policy Optimization (GRPO) is an efficient, value-function-free alternative to Proximal Policy Optimization (PPO) that leverages group-relative reward normalization. We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts. In Stage 1, iGRPO samples multiple exploratory drafts and selects the highest-reward draft using the same scalar reward signal used for optimization. In Stage 2, it appends this best draft to the original prompt and applies a GRPO-style update on draft-conditioned refinements, training the policy to improve beyond its strongest prior attempt. Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models (e.g., Nemotron-H-8B-Base-8K and DeepSeek-R1 Distilled), validating its effectiveness on diverse reasoning benchmarks. Moreover, applying iGRPO to OpenReasoning-Nemotron-7B trained on AceReason-Math achieves new state-of-the-art results of 85.62\% and 79.64\% on AIME24 and AIME25, respectively. Ablations further show that the refinement wrapper generalizes beyond GRPO variants, benefits from a generative judge, and alters learning dynamics by delaying entropy collapse. These results underscore the potential of iterative, self-feedback-based RL for advancing verifiable mathematical reasoning.

1. Introduction

The paper identifies a gap in reasoning RL: existing frameworks do not use iterative self-feedback, despite evidence that reflection can improve multistep reasoning. It proposes iGRPO, which selects a rewarded draft and conditions refinement on it, and evaluates the approach against GRPO across models and benchmarks.

  • Motivation: Existing RL frameworks typically omit feedback or reflection on the model’s own outputs, leaving iterative refinement underused in complex reasoning.The paper contrasts this with human drafting and growing evidence that self-feedback supports multistep reasoning and error correction.
  • Method: iGRPO samples multiple candidate completions, scores them with GRPO’s group-based mechanism, and selects the highest-scoring draft as self-feedback.The selected draft becomes the model’s first-draft output and guide for improving the final response.
  • Method: The selected draft is appended to the original prompt, enabling a conditioned refinement stage that trains the model to surpass its best prior attempt.This preserves GRPO’s group-based reward signals while adding minimal extra overhead.
  • Evaluation: Controlled experiments compare iGRPO and GRPO under identical training conditions across DeepSeek-R1 Distilled, OpenMath-Nemotron, and multiple mathematical reasoning benchmarks.The evaluation includes AIME24, AIME25, MATH500, AMC23, GSM8K, and Minerva Math.
  • Evaluation: For 7B and 14B models, iGRPO consistently outperforms standard GRPO across the evaluated reasoning benchmarks.The experiments use models trained on the MATH dataset and assess performance across several model families and scales.

2. Related Work

Related work positions iGRPO within RL-based reasoning, GRPO optimization extensions, and model self-learning. Existing approaches improve optimization stability, sampling, verification, critique, or self-evaluation, whereas iGRPO centers on refining model-generated drafts through self-conditioning.

  • RL for reasoning: RL-based reasoning methods use task or outcome signals to improve mathematical and logical behavior, including STaR-style bootstrapping and large-scale outcome-driven training.The related work also situates open-weight efforts such as DeepSeek-R1 within this broader RL progress.
  • GRPO extensions: Dr. GRPO, DAPO, and GSPO primarily modify GRPO objectives, sampling, clipping, or reward shaping to address bias, instability, and reward noise.Their mechanisms include removing normalization terms, dynamic sampling, decoupled clipping, reward shaping, and sequence-level clipping.
  • Self-learning: Self-learning methods leverage feedback from the model or related agents through internal evaluation, self-play, verification, proof-oriented signals, or adversarial scenarios.The literature notes that unreliable rewards can hinder complex reasoning.
  • Verification and critique: Self-Verification uses the model’s verification scores to reweight or aggregate sampled solutions, while Critique-GRPO augments GRPO with critique behavior.These approaches differ from draft-conditioned refinement as described in the paper’s contribution.

3. Methodology

iGRPO extends GRPO with dynamic self-conditioning: it selects a reward-maximizing draft, appends it to the prompt, and optimizes conditioned refinements. This two-stage design uses Stage 1 for adaptive exploration and Stage 2 for gradient updates while preserving comparable generation cost under fixed sampling budgets.

  • GRPO background: GRPO estimates advantages from group-relative rewards without training a separate value function or critic.Sampled completions receive scalar rewards that are normalized within their group before the clipped policy update.
  • Motivation: iGRPO introduces dynamic self-conditioning by generating the conditioning signal with the evolving policy rather than using a fixed example.The selected draft changes as the policy changes, creating a coupled learning dynamic.
  • Theoretical analysis: For binary rewards, the expected reward of the selected best draft increases monotonically with the policy’s success probability.This supports progressively better conditioning as optimization improves the policy.
  • Stage 1: Draft Selection: Stage 1 samples N drafts, evaluates them with the reward function, and selects the highest-scoring draft as self-feedback.The selected draft is treated as an adaptive exploratory signal and is not differentiated through.
  • Stage 2: Conditioned Refinement: Stage 2 appends the selected draft to the original prompt, samples G completions, and applies GRPO-style advantages and updates only to these refinements.The augmented prompt encourages refinement beyond the strongest initial attempt.
  • Computational analysis: Under a fixed sampling budget, iGRPO has essentially the same dominant generation cost as GRPO despite its two stages.Its computation is controlled primarily by the total number of sampled completions per prompt.

4. Experiments

Under a matched sampling budget, iGRPO consistently improves over GRPO across model families and scales, with gains especially visible on difficult multi-step benchmarks. Its two-stage self-feedback design remains effective for both weaker and already-strong mathematical reasoners.

  • Controlled study with matched sampling budget: Eight completions per prompt are fixed across methods, with iGRPO redistributing them between exploration and refinement stages.All methods use the same training protocol and rule-based reward for fair compute comparison.
  • Generalist 8B model: the largest gains from self-feedback: 45.04% is iGRPO’s macro-average for Nemotron-H-8B-Base-8K, exceeding GRPO by 3.96 points and the strongest self-improvement baseline by 1.65 points.iGRPO also reaches 91.26% on GSM8K, with visible gains on AIME25 and Minerva.
  • Stronger 7B distilled reasoner: consistent gains concentrated on multi-step tasks: 69.87% is iGRPO’s overall average for DeepSeek-R1-Distill-Qwen-7B, above GRPO’s 68.29% and concentrated on multi-step tasks.Reported examples include 56.30% on AIME24 and 95.00% on AMC.
  • Math-specialized 7B model: improvements persist when the base is already strong: 76.07% is iGRPO’s average for OpenMath-Nemotron-7B, compared with 75.02% for GRPO, with gains on AIME24 and AMC.The base model starts at 74.83%, indicating improvements persist when the initial model is already strong.
  • Scaling to 14B parameters: benefits persist on complex reasoning: 73.02% versus 71.29% is the 14B macro-average comparison for DeepSeek-R1-Distill-Qwen-14B, while OpenMath-Nemotron-14B reaches 78.00% versus 76.73%.The improvement pattern remains stable at larger scale, including gains on AIME24 and AIME25.
  • Competitiveness against critique-style objectives: iGRPO keeps optimization aligned with the verifiable outcome reward while using a selected Stage 1 draft to condition Stage 2 refinement.This distinguishes it from critique-style methods that allocate capacity to auxiliary verification or critique behaviors.

5. Ablation

The ablations show that iGRPO’s refinement interface transfers beyond GRPO, accommodates generative judging, and changes entropy dynamics by sustaining exploration during training.

  • Beyond GRPO: iGRPO’s self-feedback refinement layer improves DAPO and GSPO by +1.1 to +1.2 macro-average points under matched rollout budgets.The gains primarily stem from the refinement interface rather than GRPO-specific details.
  • Interpretation: The refinement wrapper preserves the outcome-driven objective while adding self-conditioned improvement without requiring separate verification or critique behaviors.Stage 1 selects a high-reward draft, and Stage 2 directly optimizes refinements.
  • Generative judge: GPT-5 scoring raises average Pass@1 from 69.87 to 70.81 (+0.94) across six benchmarks when replacing the binary outcome checker.Pass@1 improves on all six benchmarks, with larger gains on AIME24/25 and Minerva.
  • Entropy dynamics: iGRPO maintains higher mid-training policy entropy than GRPO, delaying premature mode collapse before both methods converge near similar final entropy.The result supports sustained exploration during training rather than greater randomness at convergence.

6. Conclusion

iGRPO adds explicit self-feedback to outcome-driven reinforcement learning through a two-stage draft-selection and refinement loop. It consistently improves verifiable reasoning under matched rollout budgets, including new best AIME24/AIME25 results and delayed entropy collapse.

  • Conclusion: iGRPO selects the highest-reward Stage 1 draft, conditions Stage 2 on it, and applies a GRPO-style update to refinements.This creates dynamic self-conditioning absent from conventional group-based objectives.
  • Conclusion: iGRPO consistently improves verifiable math reasoning across 7B, 8B, and 14B backbones under matched rollout budgets.It outperforms vanilla GRPO and self-improvement baselines relying on critique or verification behaviors.
  • Conclusion: 85.62% and 79.64% are achieved on AIME24 and AIME25, respectively, when OpenReasoning-Nemotron-7B is trained on AceReason-Math with iGRPO.The same setting transfers gains beyond math to GPQA and MMLU-Pro.
  • Conclusion: The refinement wrapper generalizes to other group-based PPO variants, supports generative judges, and delays premature entropy collapse.These ablations indicate benefits beyond GRPO-specific details and higher randomness at convergence.

A. Policy Gradient Derivation for iGRPO

iGRPO constructs a self-conditioned context by selecting the highest-reward Stage 1 draft, then applies a GRPO/PPO-style update to Stage 2 refinements. The derivation treats Stage 1 as nondifferentiable context construction and uses group-normalized, token-averaged advantages with clipped importance weighting.

  • Stage 2 refinement: Stage 2 samples completion sequences conditioned on the augmented prompt and supplies the tokens receiving policy-gradient updates.The two-stage update applies gradients only to Stage 2 tokens.
  • Stage 1 self-conditioning: Stage 1 samples drafts from the old policy and selects the draft with maximum scalar reward to form the augmented prompt.The selected context induces a sampling distribution over augmented prompts, with no differentiation through draft generation or arg max selection within an iteration.
  • Group-relative advantage: iGRPO estimates each completion’s advantage from its reward relative to the group mean and standard deviation.The baseline is estimated per prompt from a group of G completions and their scalar rewards.
  • Group-relative advantage: The resulting scalar advantage is shared across completion tokens and averaged over token positions so different-length completions contribute comparably.This token-averaged form uses the 1/|o_j| factor.
  • PPO-style stabilization: The Stage 2 objective uses importance-ratio clipping, whose clipped branch contributes zero gradient when the ratio reaches the relevant bound.The clipping rule follows the standard PPO piecewise behavior.

A.5. Including the per-token KL penalty

iGRPO adds a per-token KL penalty that regularizes the Stage 2 policy toward a reference policy. The penalty is expressed through a non-negative likelihood-ratio estimator.

  • KL regularization: iGRPO includes a per-token KL penalty to the reference policy as part of its stabilized objective.The penalty is applied alongside the PPO/GRPO-style policy update.
  • KL estimator: The estimator uses the reference-to-current token-probability ratio ρ_j,t and computes KL as ρ_j,t − log ρ_j,t − 1.This form provides the non-negative per-token estimator used in the objective.

A.6. Final iGRPO surrogate objective and resulting policy gradient

The final iGRPO surrogate combines the two-stage sampling structure with clipped policy optimization and KL regularization. Stage 1 changes the self-conditioned context, while Stage 2 supplies the token-level gradient.

  • Final surrogate objective: The final surrogate reinstates two-stage sampling, with Stage 1 affecting the augmented-context distribution but not being differentiated through within an iteration.The objective combines the components defined by the clipped ratio and KL terms.
  • Resulting policy gradient: Importance-ratio clipping stabilizes the Stage 2 update, while a KL penalty regularizes the policy toward the reference policy.These mechanisms are included in the assembled objective and its differentiated gradient.
  • Resulting policy gradient: Stage 2 applies a standard GRPO/PPO-style token-level gradient weighted by group-normalized completion advantages.The same completion-level advantage is shared across tokens.
  • Additional validation: Training OpenMath-Nemotron-14B with iGRPO on 94,000 OpenR1-Math-220k examples achieves 66.04% on AIME25.The study uses reasoning traces from DeepSeek-R1 and reports the result in Table S.1.

B.1. Analysis of Pass@N on AIME Benchmarks

Pass@N accuracy increases as more solutions are sampled, but the saturation point differs between AIME24 and AIME25. The supplementary analysis evaluates this behavior alongside the training and resource-use setup.

  • Pass@N trends: Both AIME benchmarks improve with higher N, while AIME24 stabilizes earlier than AIME25.Figure S.1 compares iOpenMath-Nemotron-14B across multiple pass@N settings.
  • Saturation behavior: 93.33% on AIME24 is reached by N=16, whereas AIME25 continues to 96.67% at N=256.AIME25 briefly plateaus at 90.00% between N=32 and N=128 before increasing further.
  • Experimental setup: The ablation setup spans 7B and 14B models trained for one epoch on MATH and AceReason-Math.The reported setup uses multiple model architectures and datasets.
  • Experimental setup: The supplementary training configuration includes two nodes of eight NVIDIA A100 GPUs for 7B models, with one node reserved for vLLM.Table S.2 summarizes these hyperparameters and hardware settings.
  • Prompting and evaluation: The training prompt instructs the model to treat prior drafts or feedback as guidance and produce an improved answer rather than repeat it verbatim.The prompt emphasizes fixing mistakes, filling gaps, strengthening justification, and improving clarity.
  • Resource evaluation: Resource evaluation monitors peak GPU memory and throughput to compare training configurations and identify bottlenecks.Throughput is expressed as processed samples per second.

D.2. Measurements

iGRPO preserves nearly the same peak memory usage as GRPO while incurring modest throughput and total training-time overhead from its second refinement stage.

  • 54.9349 GB peak memory for iGRPO closely matches GRPO’s 54.9286 GB, a difference of roughly 0.0063 GB.The measured gap is described as practically negligible.
  • 0.34 samples/s throughput for iGRPO versus 0.41 samples/s for GRPO reflects a mild slowdown from the additional generation round.The passage characterizes this as a modest computational cost rather than a large-factor reduction.
  • 94.1 GPU hours for iGRPO versus 83.3 GPU hours for GRPO corresponds to roughly a 13% increase in wall-clock training time.The overhead comes from sequential Stage 1 and Stage 2 decoding, without requiring more GPUs or memory capacity.
  • iGRPO’s two-stage approach yields near-identical peak memory usage, a minor throughput decrease, and a modest 13% full-run time overhead on an 80 GB A100.The table summarizes memory, throughput, and total GPU-hour measurements under the same setup.

E. Additional Ablation Studies

Additional ablations examine training dynamics, KL regularization, and completion budgets. iGRPO maintains higher rewards with similar response lengths, while moderate completion budgets and simplified KL settings offer practical trade-offs.

  • Training Dynamics and Response Length: iGRPO maintains higher average rewards across training checkpoints, while both methods produce nearly identical response lengths.GRPO produces slightly longer outputs on average, and iGRPO’s two-stage process does not lead to lengthy responses.
  • Effect of KL Divergence Term: 70.23% is the highest overall score among the tested KL coefficients, achieved with β=0.0001.The differences across settings are relatively small, so β=0 is presented as a simpler option without significant performance sacrifice.
  • Effect of Number of Completions: Increasing from 4 to 8 total completions clearly improves performance, whereas gains beyond 8 are modest.Larger completion budgets increase training time and inference latency for minimal additional returns.
Loading 2602.09000v1…