PaperlayerView exact revision on arXiv

Source-linked AI summary

Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

Xiaoying Zhang, Yipeng Zhang, Hao Sun, Kaituo Feng, Chaochao Lu, Chao Yang, Helen Meng

arXiv:2506.03106v7cs.CLcs.AI

TL;DR

Numerical-only RL can plateau, provide ineffective self-reflection, and leave persistent failures. Critique-GRPO integrates natural-language critiques with numerical feedback so online RL learns from initial responses and guided refinements, achieving broad gains across eight reasoning tasks.

  • Problem

    Numerical-only RL suffers from performance plateaus, ineffective spontaneous self-reflection, and persistent failures.

  • Method

    Critique-GRPO is an online RL framework that jointly learns from initial responses and critique-guided refinements using natural-language and numerical feedback.

  • Results

    +15.0-21.6% average Pass@1 improvements are reported on Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight reasoning tasks.

  • Takeaways & Limitations

    Critique-GRPO consistently outperforms existing SFT and numerical-feedback-only RL methods across eight reasoning tasks.

  • Takeaways & Limitations

    The theoretical analysis assumes a reasoning problem with a hidden optimal solution sequence and action sets defined over L steps.

Abstract

from arXiv · show

Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs). However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures. We show that plateaued RL models can successfully refine failed solutions when given natural language critiques. Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization. This approach enables LLMs to learn simultaneously from initial responses and critique-guided refinements, effectively internalizing the exploration benefits of both stages. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024. The code and models are released at: https://github.com/zhangxy-2019/critique-GRPO

1. Introduction

Numerical-feedback RL improves LLM reasoning but can plateau, fail to support effective self-reflection, and leave persistent failures. Critique-GRPO integrates critiques with numerical rewards so models learn from both initial responses and refinements, yielding broad gains and self-improvement.

  • Motivation: Numerical-only RL exhibits performance plateaus, ineffective spontaneous self-reflection, and persistent failures.These limitations motivate richer feedback for scaling RL.
  • Motivation: Offline critique methods mainly imitate static critiques, leaving critique integration into online RL unexplored.The paper frames online exploration and real-time adaptation as the missing capabilities.
  • Approach: Critique-GRPO combines standard generation with critique-guided self-refinement, integrating natural-language and numerical feedback in online RL.Critiques come from model-based or rule-based reward systems, while refinements are learned alongside initial responses.
  • Results: +15.0-21.6% average Pass@1 improvements are reported on Qwen models, and +7.3% on Llama-3.2-3B-Instruct across eight reasoning tasks.The framework outperforms compared supervised and RL-based fine-tuning methods.
  • Results: +16.7% Pass@1 over GRPO is achieved on AIME 2024 when using self-critiques.Figure 1 also reports 66.7% pass@1 and consistent pass@k gains for k=1-256.
  • Contributions: The paper validates Critique-GRPO through extensive experiments across eight challenging reasoning benchmarks.The stated contributions include identifying numerical-feedback limitations, proposing the framework, and demonstrating superior benchmark performance.

2. Related Work

Numerical feedback has improved LLM reasoning, but sparse scalar rewards provide limited diagnostic information about where or why reasoning errors occur.

  • Enhancing LLM Reasoning with RL: Sparse numerical rewards make it difficult for models to identify and correct specific reasoning failures.The limitation arises because numerical feedback provides little information about error location or cause.

3. Limitations of RL and the Promise of Natural Language Guidance

Experiments identify saturation, weak spontaneous self-reflection, and persistent failures under numerical-only RL. Deliberate natural-language critiques enable refinements of previously unsolvable problems, with chain-of-thought critiques producing the strongest reported refinement results.

  • Three Limitations: Qwen2.5-7B-Base saturates after 120 steps despite increasing training prompts from 4k to 32k or extending training.Neither intervention yields significant gains.
  • Three Limitations: Spontaneous self-reflection behaviors contribute minimally to problem-solving success during RL fine-tuning.The analysis examines verification, backtracking, backward chaining, anticipation, and related behaviors.
  • Three Limitations: Approximately 29% of training questions remain unsolved by the best RL-finetuned model, with Pass@4 = 0.This persistent failure rate indicates limitations of numerical-only feedback.
  • Natural-Language Guidance: Three critique variants are evaluated on questions where RL-finetuned models persistently fail: indicative, indicative with ground truth, and chain-of-thought critiques.The critiques are used to prompt self-refinement on the Pass@4 = 0 subset.
  • Natural-Language Guidance: All three critique types enable successful refinement of previously unsolvable problems, unlike spontaneous self-reflection.The result is presented as evidence that deliberate critiques can succeed where spontaneous reflection fails.
  • Natural-Language Guidance: 36.47% valid refinement and 55.37% persistent-failure refinement rates are achieved by CoT critiques for Qwen2.5-7B-Base.Their effectiveness is attributed to richer, step-by-step explanations.

4. Critique-GRPO

Critique-GRPO extends GRPO with critique-guided refinements alongside initial responses, then optimizes the combined trajectories using scalar rewards, selection, weighted advantages, and policy shaping.

  • Framework Overview: Critique-GRPO optimizes the policy using trajectories from both standard generation and critique-guided refinement.The framework is built on GRPO and combines natural-language and numerical feedback.
  • From GRPO to Critique-GRPO: GRPO estimates advantages from the relative rewards of grouped responses for each query without a value-function approximation.Its objective uses sampled responses from an old policy and clipped probability ratios.
  • Online Data Construction: Initial responses are sampled online, evaluated by a reward system, and paired with generated critiques.Critique-GRPO supports rule-based and model-based reward systems, including indicative and CoT critiques.
  • Critique-Guided Self-Refinement: Refinement starts when the initial response set contains no correct solution, using in-context learning conditioned on the question, response, and critique.Refined responses are evaluated and combined with initial responses for training.
  • Critique-Guided Self-Refinement: A subset of refinements is selected, prioritizing correct solutions, to control entropy growth from refinement-induced distribution shifts.If no correct refinements exist, sampling is random.
  • Theoretical Motivation: Critique-guided exploration can reduce effective complexity from O(|S|^L) toward O(L) when critiques localize errors and provide corrections.Under a fixed computational budget, the analysis states that this yields a higher probability of sampling the optimal solution.
  • Online Policy Optimization: The mixed initial-and-refined set is optimized with scalar rewards, shared group baselines, and policy shaping for refined responses.The shaping function emphasizes currently low-probability tokens in valid refinements, while KL regularization is removed to permit substantial updates.

5. Experiments

Experiments evaluate Critique-GRPO across models, tasks, feedback types, and ablations. Results consistently favor online learning from natural-language critiques alongside numerical rewards, including self-critiquing and weak-to-strong settings.

  • Experimental Setup: Critique-GRPO is evaluated on mathematical, scientific, and general reasoning tasks using multiple policy and critique models.The setup includes five mathematical benchmarks and comparisons with supervised and reinforcement-learning baselines.
  • Main Results: +4.4 points and +3.8 points average Pass@1 gains over R1-GRPO and R1-Dr.GRPO are reported for Qwen2.5-7B-Base and Qwen3-8B, respectively.The corresponding averages are 42.66% →47.08% and 64.46% →68.26%.
  • Main Results: +11.9 and +8.81 points over Refinement FT, plus +11.4 and +12.4 points over CITL-FT, are reported on Qwen2.5-7B-Base and Qwen3-8B.These comparisons support online self-refinement over the cited offline approaches.
  • Math-Centric Backbone Models: 21.6% average Pass@1 improvement is achieved on Qwen2.5-Math-7B-Base using 4k prompts, outperforming numerical-feedback approaches using 46k prompts.Critique-GRPO also outperforms PRIME-Zero despite PRIME-Zero using dense intermediate rewards.
  • Self-Improvement: 4.5% and 12.0% average Pass@1 improvements over R1-GRPO and SFT demonstrate self-improvement through self-critiquing.Pass@k results on AIME24 and AIME25 remain consistently higher for k ∈[1, 256].
  • Exploration and Model Generalization: Critique-GRPO gains 5.1 and 4.0 points over GRPO on Llama-3.2-3B-Instruct and Qwen3-32B, while alternative critique models yield 5.6–6.8-point gains.The method also maintains higher policy entropy, and policy shaping contributes a +3.1% ablation gain.
  • Generalization: +12.3% average Pass@1 improvement is obtained from weaker-model refinements, while MATH-500 reaches 77.80% and AMC23 reaches 62.50%.On GPQA-Diamond, Critique-GRPO reaches 37.88% versus 34.34% for the sequential baseline.

6. Conclusion

The paper identifies limitations of numerical-only RL and proposes Critique-GRPO, which jointly uses natural-language and numerical feedback. Across eight reasoning tasks, it consistently outperforms supervised and numerical-feedback-only RL methods.

  • 6. Conclusion: Critique-GRPO jointly learns from initial responses and critique-guided self-refinements through natural-language and numerical feedback.The framework is presented as an online RL approach for scaling reasoning improvements.
  • 6. Conclusion: Across eight reasoning tasks, Critique-GRPO consistently outperforms existing supervised fine-tuning and numerical-feedback-only RL methods.The conclusion frames the method as relevant to scaling RL with diverse real-time feedback.

Impact Statement

The paper reports ethical safeguards, limited computational overhead, and performance gains from Critique-GRPO across larger models and critique settings. It also examines critique length and adversarial feedback to characterize when the approach succeeds or degrades.

  • The study reports privacy, fairness, and well-being safeguards, including research-only datasets without personally identifiable information and checks for offensive content and misinformation.
  • Natural language feedback provides granular diagnostic signals for refining responses, complementing reinforcement learning methods that improve large language model reasoning.
  • 5 and 19 GPU-hours add approximately 2% and 6.5% over GRPO for Qwen2.5-7B-Base and Qwen3-8B, while average gains reach +6 and +4.5 percentage points across eight tasks.
  • 47.08 to 46.00 performance declines when critique length increases from approximately 683 to around 2079 tokens, suggesting highly detailed critiques can hinder extraction of useful guidance.
  • Inverting adversarial critique conclusions causes entropy explosion and performance degradation, supporting the role of valid guidance in the observed improvements.
  • Critique-GRPO consistently surpasses GRPO across all eight reasoning tasks when scaled to the Qwen3-32B model.

E.1. Three Limitations of Learning with Numerical Feedback

The analysis identifies performance plateaus, ineffective spontaneous self-reflection, and persistent failures as limitations of numerical-feedback RL, then shows that deliberate critiques can enable refinement.

  • The investigation examines scaling, cognitive behaviors, and previously failed problems to characterize numerical-feedback RL limitations.
  • Performance Plateaus: 45-46% validation accuracy is Qwen2.5-7B-Base’s peak after 120 steps, while Qwen3-8B plateaus at 65-67% after 200 steps.Peak accuracy changes negligibly despite 8× more training prompts or training extended to 420 steps.
  • Ineffective Spontaneous Self-Reflection: Self-reflection behaviors contribute minimally across mathematical and STEM reasoning tasks after RL fine-tuning.The analyzed behaviors include verification, backtracking, backward chaining, and anticipation.
  • Persistent Failures: Approximately 29% of Qwen2.5-7B-Base problems and 3.75% of Qwen3-8B problems remain persistently unsolved with pass@4 = 0.These failures persist despite trial-and-error fine-tuning with numerical rewards and penalties.
  • Numerical feedback lacks the expressivity to explain why responses fail or how to correct multi-step reasoning.The paper therefore motivates richer feedback mechanisms for more effective learning.

F.4. Theoretical Analysis of Sample Efficiency

The theoretical analysis argues that language critiques provide more informative exploration than scalar rewards, reducing effective search complexity and improving policy optimization.

  • The analysis uses Transfer Eluder Dimension to quantify information gains from reward-informative language feedback.It formalizes critiques as feedback that lets policies distinguish hypotheses more efficiently than scalar rewards.
  • Reward-Only Learning: Binary reward-only exploration has exponential complexity in sequence length because failed sequences provide no information about other sequences.The resulting signal behaves like a sparse needle in a haystack.
  • Sample Efficiency: For L|S| ≪ M ≪ |S|^L, critique-guided exploration has a significantly higher success probability than standard reward-based exploration.The conclusion states that critique-guided refinement is strictly more sample-efficient.
  • Effective Dimension: First-error feedback reduces exploration to L sequential classification problems of size |S|, yielding effective dimension L|S|.Partitioning actions by the first incorrect position enables transfer across actions sharing error structure.
  • Effective Dimension: Correction suggestions further reduce the effective dimension to d_corr ≈ O(L).The correction makes the sequence of error indices strictly monotonic.
  • Policy Optimization: Critique-guided refinements increase the effective probability of observing optimal trajectories from p* toward P_crit, improving policy updates.The theory states P_crit ≫ p* and derives faster convergence by O(P_crit/p*), under stated assumptions.
  • Policy Optimization: Off-policy refinements can be orthogonal to initial-response updates, so clipping alone cannot prevent conflicting directions or entropy collapse.The proposed update combines initial and refinement gradients while controlling off-distribution refinement risk through regularization and sampling balance.

G. Implementation Details

The implementation compares Critique-GRPO with supervised and numerical-feedback RL baselines across mathematical reasoning models and standardized evaluation settings.

  • Evaluation: Evaluation covers MATH-500, Minerva-Math, OlympiadBench, AIME, AMC, and additional reasoning benchmarks for out-of-distribution generalization.Pass@1 uses greedy sampling with temperature 0; pass@k uses temperature 0.6 and top-p settings.
  • Reward Design: Binary scalar rewards assign +1 to correct final answers and 0 to incorrect answers using rule-based Math-Verify evaluation.These rewards serve as proxies for response accuracy during fine-tuning.
  • Compared Methods: The experiments compare Critique-GRPO with SFT, RAFT, Refinement FT, Critique FT, CITL-FT, R1-GRPO, R1-Dr.GRPO, and math-focused RL baselines.The compared methods span supervised fine-tuning and reinforcement-learning-based fine-tuning.
  • Models and Training: Experiments use Qwen2.5-7B-Base, Qwen2.5-Math-7B-Base, and Qwen3-8B, with GPT-4o serving as the reward model.For RL approaches, models are fine-tuned for 400 steps and the best performance is reported.

H. Detailed Results and Analysis of Self-Critiquing Mechanisms

Self-critiquing with Critique-GRPO improves self-improvement across pass@k metrics, outperforming external numerical-feedback RL and supervised fine-tuning baselines.

  • Mechanism: Self-critiquing uses the model as a reasoning-based reward model to evaluate its own responses with CoT critiques.The paper studies self-critique with ground-truth references and self-critique with self-evaluation.
  • Overall Self-Improvement: Critique-GRPO self-critiquing improves average pass@1 by 4.5% over R1-GRPO and 12.0% over SFT.The self-critique and self-evaluation variant improves average pass@1 by 2.3% over R1-GRPO without external supervision.
  • Pass@k Analysis: Critique-GRPO self-critiquing consistently outperforms R1-GRPO across pass@k values from 1 to 256.The pattern indicates gains beyond a single-sample evaluation.
  • Pass@k Analysis: 10-16.7% improvements over R1-GRPO occur on AIME24 for pass@k with k = 1 to 4.These gains are reported for Critique-GRPO self-critiquing.

I. Detailed Investigation of Policy Exploration During RL Finetuning

Critique-GRPO sustains higher policy entropy than comparison RL methods and avoids entropy collapse during self-critiquing, while achieving strong performance with limited response-length growth. However, higher entropy alone does not guarantee effective exploration.

  • Critique-GRPO generally maintains higher policy entropy than R1-GRPO and R1-Dr.GRPO, suggesting more consistent exploration.Entropy peaks before step 200 may reflect self-generated refinements that differ substantially from initial responses.
  • Critique-GRPO self-critiquing avoids entropy collapse and maintains higher entropy than R1-GRPO.The passage links increased exploration with improved performance.
  • 65.55% vs. 68.13% average Pass@1: weaker critique refinements achieve higher entropy but lower performance than self-critiquing.The comparison suggests that exploration-signal quality matters more than entropy magnitude alone.
  • Critique-GRPO achieves superior performance while minimally increasing response length on Qwen2.5-7B-Base and reducing response length on Qwen3-8B.The passage attributes this trend to more precise error identification and refinement.

J. Qualitative Analysis

The qualitative analysis evaluates generated reasoning across factuality, conciseness, correctness, logicality, and average performance. Critique-GRPO produces stronger responses than the base model in the presented case and achieves the best fine-grained scores reported on Minerva-MATH.

  • Qualitative Analysis: Critique-GRPO on Qwen2.5-7B-Base achieves the best performance across factuality, conciseness, correctness, logicality, and average performance.The analysis uses 100 generated Minerva-MATH responses and rates four dimensions plus the average.
  • Qualitative Analysis: Critique-GRPO outperforms R1-GRPO in factuality and correctness, indicating that CoT critiques help identify and improve errors.
  • Case Study: The case study evaluates the expression sin(arcsin 0.4 + arcsin 0.5) multiplied by sin(arcsin 0.5 − arcsin 0.4).
  • Case Study: The base Qwen3-8B response is incorrect because it formulates the expression as sin(A + B) · sin(A − B), whereas Critique-GRPO uses sin(α + β) · sin(β − α) correctly.The example contrasts redundant self-reflection in the base model with a correct formulation after Critique-GRPO fine-tuning.

K. Limitations

The paper identifies failed refinements as a remaining performance limitation and notes that critique detail, model choice, and training for self-refinement constrain effectiveness. It also describes prompts and analyses used to study reasoning behaviors and qualitative response quality.

  • Limitations: Policy models sometimes fail to follow CoT critiques when refining responses because self-refinement is not deliberately trained.The authors suggest improving refinement capabilities or training a specialized refinement model.
  • Limitations: The effectiveness of expert demonstrations as critiques is limited because models often produce conclusions and correct answers instead of detailed step-by-step refinements.The paper uses GPT-4o as a reasoning-based reward model without expert knowledge distillation.
  • Limitations: Future work should investigate which critique types provide the greatest refinement benefits, particularly for reasoning-intensive tasks.
  • Prompts: The reasoning-behavior analysis examines behaviors including verification, backtracking, subgoal setting, backward chaining, anticipation, and summarization.Generated responses count each behavior once, even when it appears multiple times in one response.
  • Prompts: The qualitative-analysis prompt rates final-answer correctness, verbosity, factual accuracy, and logical coherence on a 1–5 scale.

O. Responses Utilized in Qualitative Analysis

The qualitative examples compare an initial Qwen3-8B response with a Critique-GRPO refinement on a trigonometric expression. The initial response concludes −9/100, while the refined response concludes 9/100.

  • Response setup: The expression is rewritten using α = arcsin 0.4 and β = arcsin 0.5, with sine addition and subtraction formulas applied separately.The response identifies sin α = 0.4 and sin β = 0.5 before computing the two terms.
  • Refined response: The refined response concludes that the original expression equals 9/100, or 0.09, and confirms this with approximate multiplication.The reported approximations are sin(α + β) ≈ 0.805 and sin(β − α) ≈ 0.1118.
  • Failure case: A separate example is described as a failed refinement because the Critique-GRPO model does not follow the chain-of-thought critique when revising its response.The example concerns maximizing an expression under a constraint on positive real numbers.
Loading 2506.03106v7…