PaperlayerView exact revision on arXiv

Source-linked AI summary

Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning

Zhiheng Xi, Jixuan Huang, Xin Guo, Boyang Hong, Dingwen Yang, Xiaoran Fan, Shuo Li, Zehui Chen, Junjie Ye, Siyu Yuan, Zhengyin Du, Xuesong Yao, Yufei Xu, Jiecao Chen, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2510.24320v1cs.CLcs.AI

TL;DR

Critique training often depends on stronger supervisors or oracle verification, leaving a need for approaches that can train critics without those supports. Critique-RL uses two-stage reinforcement learning to optimize critique discriminability and helpfulness, and reports gains across in-domain and out-of-domain tasks.

  • Problem

    Existing critique training often relies on costly stronger-supervisor annotations or oracle verification, which can let critics avoid judging response quality.

    Source: §1 · pp. 1–2

  • Method

    Critique-RL trains a critic through two-player actor-critic interaction, using direct rule-based rewards for discriminability before refinement-based rewards for helpfulness with regularization.

    Source: §1 · pp. 2–3

  • Results

    For Qwen2.5-7B, Critique-RL improves performance by 9.02% on in-domain tasks and 5.70% on out-of-domain tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors report that Critique-RL generalizes to unseen tasks, indicating promise for scalable oversight.

    Source: §1 · p. 3

Abstract

from arXiv · show

Training critiquing language models to assess and provide feedback on model outputs is a promising way to improve LLMs for complex reasoning tasks. However, existing approaches typically rely on stronger supervisors for annotating critique data. To address this, we propose Critique-RL, an online RL approach for developing critiquing language models without stronger supervision. Our approach operates on a two-player paradigm: the actor generates a response, the critic provides feedback, and the actor refines the response accordingly. We first reveal that relying solely on indirect reward signals from the actor's outputs for RL optimization often leads to unsatisfactory critics: while their helpfulness (i.e., providing constructive feedback) improves, the discriminability (i.e., determining whether a response is high-quality or not) remains poor, resulting in marginal performance gains. To overcome this, Critique-RL adopts a two-stage optimization strategy. In stage I, it reinforces the discriminability of the critic with direct rule-based reward signals; in stage II, it introduces indirect rewards based on actor refinement to improve the critic's helpfulness, while maintaining its discriminability via appropriate regularization. Extensive experiments across various tasks and models show that Critique-RL delivers substantial performance improvements. For example, it achieves a 9.02% gain on in-domain tasks and a 5.70% gain on out-of-domain tasks for Qwen2.5-7B, highlighting its potential.

1. Introduction

Critique-RL addresses the limits of critique training that depends on stronger supervisors or oracle verification by training a critic through two-player actor-critic interaction. Its two-stage RL approach targets discriminability and helpfulness, after indirect rewards alone improved helpfulness without adequately optimizing discriminability.

1. Introduction

  • Prior critique-training approaches rely on costly, hard-to-scale stronger supervisors or on oracle verification that lets critics avoid judging response quality.Without an oracle verifier, these prompt-based approaches often reach a performance bottleneck.Source: §1 · p. 2
  • Critique-RL trains a critic to assess actor outputs and provide natural-language feedback, which the actor uses to refine its response.The paper frames this as an approach to developing critique models without stronger supervision or an oracle reward function at test time.Source: §1 · pp. 2–3
  • Indirect rewards based on actor-attempt correctness improve critique helpfulness but leave discriminability poorly optimized, creating an optimization bottleneck and possible RL collapse.The authors describe this as a failure of relying on indirect rewards alone.Source: §1 · p. 3
  • For Qwen2.5-7B, Critique-RL improves performance by 9.02% on in-domain tasks and 5.70% on out-of-domain tasks.The authors also report stable optimization of both discriminability and helpfulness and generalization to unseen tasks.Source: §1 · p. 3
  • Stage I uses direct rule-based rewards to optimize discriminability; Stage II uses actor-refinement rewards to improve helpfulness while regularization maintains discriminability.The method is presented as an online RL approach based on two-player actor-critic interaction.Source: §1 · pp. 2–3

2. Related Work

Prior critique approaches use prompting or fine-tuning, often with oracle verification or stronger-supervisor annotations. Critique-RL instead trains a separate critique model with RL to optimize both discriminability and helpfulness.

2. Related Work

  • Prompt-based critique methods often rely on an oracle verifier at test time, while Critique-RL trains both discrimination and helpful feedback without assuming one.Without external verification, even state-of-the-art models face significant challenges.Source: Prompt engineering for eliciting… · p. 3

Fine-tuning language models for critiquing.

  • Supervisor-annotated fine-tuning relies on stronger annotators, which the paper describes as costly and difficult to scale.The cited passage begins contrasting these methods with self-improvement approaches.Source: Fine-tuning language models for… · p. 3
  • Unlike self-critiquing approaches, Critique-RL uses a two-player paradigm and trains a separate critique model through RL.The passage positions this approach against methods that rely on self-improvement.Source: Fine-tuning language models for… · p. 4
  • Retroformer and CTRL use indirect rewards to improve critique helpfulness, whereas Critique-RL jointly optimizes helpfulness and discriminability in two stages.The cited passage describes the earlier RL phase as overlooking joint optimization of both capabilities.Source: Reinforcement learning for language… · p. 4

3. Preliminaries

The preliminaries describe a response-critique-refinement interaction, policy-gradient optimization of the critic, and metrics for actor refinement and critique discrimination.

3.1. The Two-Player Interaction Framework

  • In the two-player interaction, the actor answers a question, the critic assesses correctness and provides constructive feedback, and the actor generates a refinement.The actor’s original response is checked by an oracle verifier before critique and refinement.Source: §3.1 · p. 4

3.2. Policy Gradient for LLMs

  • Policy-gradient optimization samples a critique and an actor refinement, then increases the probability of a positive critique through gradient ascent.The objective maximizes the critique model’s reward function.Source: §3.2 · p. 4

3.3. Evaluation Metrics

  • Evaluation includes refinement accuracy and change rates, plus Acc@Dis for agreement between the critic’s correctness assessment and true response correctness.A lower Δc→i is better, while a higher Δi→c is better.Source: §3.3 · pp. 4–5

4. Methodology

Critique-RL addresses the weakness of indirect actor-outcome rewards, which can improve feedback helpfulness without adequately training critics to distinguish correct from incorrect responses. It uses direct discriminability rewards in Stage I, then optimizes helpfulness while maintaining discriminability in Stage II.

4.1. Motivating Findings: RL with Indirect Reward Signals Is Insufficient for Training Satisfactory Critique Models

  • The motivating experiments explore reward signals derived from the actor’s original and refined responses in a two-player actor-critic framework.The study investigates why these reward-shaping approaches fail to produce satisfactory critiquing ability.Source: §4.1 · p. 5

Analysis setups: data, models, and training methods.

  • The preliminary experiments use GSM8K and Qwen2.5-3B, with an actor trained to generate responses and refine them according to critiques.The base critique model is initialized using a supervised fine-tuning dataset built from prompted Qwen2.5-3B-Instruct critiques.Source: §4.1 · p. 5

Empirical findings and behavior analysis.

  • Indirect rewards based on actor attempts can produce conservative or aggressive behavior: refinement-based rewards reduce incorrect changes, while correction rewards risk introducing them.Refinement-based rewards do not significantly improve error correction, whereas correction rewards fail to effectively reduce incorrect changes.Source: Empirical findings and behavior analysis · p. 6
  • The three tested indirect reward functions fail to optimize discriminability effectively, improving judgments for one response type while weakening judgments for the other.The analysis suggests that actor-response-based rewards target helpfulness while overlooking discriminability.Source: Empirical findings and behavior analysis · p. 6

Key challenges.

  • The method targets accurate judgments of both correct and incorrect responses while improving feedback quality without encouraging overly conservative or aggressive behavior.These are the two challenges identified for reinforcement learning of critique models.Source: Key challenges · pp. 6–7

4.2. Two-Stage Critique-RL

Stage I: optimizing discriminability through direct reward signals.

Stage II: optimizing helpfulness while maintaining discriminability.

5. Experiments

Across the reported experiments, Critique-RL outperforms comparison methods on accuracy and discriminability, while iterative critique-refinement and training further improve results. The evaluation covers mathematical reasoning tasks, multiple model families, and both in-domain and out-of-domain test sets.

Datasets.

  • The evaluation uses five mathematical reasoning tasks, with MATH, GSM8K, and AQUA as in-domain tests and SVAMP and TheoremQA as out-of-domain tests.Training uses the train splits of MATH, GSM8K, and AQUA.Source: Datasets · p. 8

Models and baselines.

  • Experiments mainly use Qwen2.5-3B and Qwen2.5-7B, with additional models including Qwen2.5-72B, Llama3.2, and DeepSeek-R1-Distill-Qwen-7B.Baselines include SFT, STaR, Retroformer, and CTRL.Source: Models and baselines · p. 8

Generally, critique models can significantly improve actor’s reasoning performance.

  • In MATH, SFT with a critique model exceeds the no-critic actor by 7.34 points on 3B and 6.10 points on 7B.The authors interpret the findings as evidence that critique models can support scalable oversight.Source: Generally, critique models can… · p. 9

RL-based methods outperforms fine-tuning-based ones.

  • On Qwen2.5-3B, Critique-RL surpasses SFT by an average of 7.11 accuracy points across three datasets.The paper notes that SFT and STaR may reduce performance on AQuA, while Critique-RL provides positive improvements there.Source: RL-based methods outperforms… · p. 9
  • On Qwen2.5-7B, Critique-RL exceeds Retroformer by an average of 5.11 points on accuracy and 12.69 points on discriminability across three datasets.The paper also reports that it surpasses CTRL on GSM8K discrimination by 5.31 points for 3B and 6.36 points for 7B.Source: RL-based methods outperforms… · p. 10

5.3. Iterative Improvement of Critique-RL

  • The study tests iterative critique-refinement and alternating two-stage training, reporting consistent accuracy gains across refinement iterations on Qwen2.5-3B.On MATH, the second training iteration improves accuracy by 2.40 points and discriminability by 3.68 points over the first.Source: §5.3 · p. 10

6. Discussion and Analysis

The analysis tests which training stages and reward choices matter, how helpfulness can be assessed separately, and how Critique-RL scales and generalizes.

Ablation on different stages

Ablation on reward design for Stage II

  • Removing Stage II’s discrimination reward and KL regularization drops accuracy and discriminability; replacing r_refine with r_Δ or r_correction causes a slight performance drop.The authors suggest r_refine’s direct alignment with test-time Acc@Refine may explain its advantage.Source: Ablation on reward design for Stage II · pp. 10–11

Analysis of helpfulness when the oracle verifier Is available

  • With an oracle verifier isolating helpfulness, Critique-RL still outperforms other methods across datasets and models.All baselines improve in this setting, and the authors report that discriminability optimization also implicitly contributes to helpfulness.Source: Ablation on reward design for Stage II · p. 11, Figure 5 · p. 11

Evaluation of test-time inference compute scaling for Critique-RL

  • Critique-RL raises the performance ceiling and is more compute-efficient as inference compute scales, including versus 3K× parallel response sampling.The paper reports that K× response-critique-refinement sampling is more compute-efficient than 3K× parallel response sampling.Source: Evaluation of test-time inference… · p. 12

Generalization to OOD tasks

7. Conclusion

The paper concludes that Critique-RL optimizes discriminability and helpfulness, with experiments reporting gains across different actor models and model architectures.

7. Conclusion

  • The conclusion presents Critique-RL as a two-stage approach that optimizes both discriminability and helpfulness, with experiments supporting its stability and superiority.The paper identifies ablations and analyses as evidence about the method’s working mechanism.Source: §7 · p. 12

A. Performance on Varying Base Models

  • On TheoremQA, Critique-RL outperforms SFT, Retroformer, and CTRL by 26.75, 28.75, and 29.88 points in Acc@Dis, respectively, on the diverse-domain dataset.Source: A. Performance on Varying Base Models · p. 18
  • With a Qwen2.5-72B-Instruct actor, Critique-RL still outperforms baselines on in-domain and out-of-domain tasks and achieves higher discrimination.The reported actor-performance gains are less pronounced than in smaller-actor settings.Source: A. Performance on Varying Base Models · p. 18

B. Performance on Varying Model Series

  • On GSM8K, Critique-RL is effective with Llama3.2-3B as well as Qwen2.5 models, particularly improving critic discriminability.The authors describe these results as evidence of adaptability across model architectures.Source: B. Performance on Varying Model Series · p. 18

C. Comparison with Other Important Refinement Methods

Critique-RL outperforms several refinement methods on GSM8K and shows sampling-efficiency gains in test-time scaling experiments.

C. Comparison with Other Important Refinement Methods

D. More Test-time Scaling Results

  • Across MATH, GSM8K, and AQUA, Critique-RL consistently achieves approximately twice SFT’s sampling efficiency; on GSM8K with the 7B model, Critique-RL’s Pass@1 even surpasses SFT’s Pass@64.For inference-compute scaling on GSM8K, Critique-RL improves the performance ceiling and is more compute-efficient.Source: D. More Test-time Scaling Results · p. 19, Figure 6 · p. 20

E. Performance on Summarization Task

On summarization and math tasks, the experiments report improvements from Critique-RL, including gains over actor-only RL and SCoRe and efficient test-time refinement.

E. Performance on Summarization Task

  • For summarization, a reward model scores the original and revised summaries; the critic supplies a quality score and improvement suggestions.The method uses CNN/DailyMail and a reward model because rule-based verifiers do not directly apply to open-ended tasks.Source: E. Performance on Summarization Task · p. 20
  • On summarization, Critique-RL improves over baseline by 0.87 points in Score, 7.87 points in MSE@Dis, and 1.79 points in MAE@Dis.The paper says these results improve summary quality and demonstrate generalization to open-ended tasks.Source: E. Performance on Summarization Task · p. 21, Table 9 · p. 21

F. Validating the Effectiveness of Critique Model

G. Sensitivity Analysis

The sensitivity analysis finds the tested reward parameters are not sensitive and reports how the two stages change discriminability and helpfulness across training steps. Further parameter refinement could still yield additional gains.

G. Sensitivity Analysis

  • The tested β, β1, and β2 values were not sensitive on Qwen2.5-3B GSM8K; the experiments used β=0.01, β1=0.9, and β2=0.95.Source: G. Sensitivity Analysis · p. 22

Experiments on different training steps per stage.

H. Qualitative Analysis

Qualitative cases contrast error detection alone with constructive critique: Stage I can identify an error without improving the actor’s answer, whereas Stage II pairs detection with suggestions and correct refinement. A separate evaluation found that most sampled successful critiques made correct judgments and were rated high-quality.

  • 96.2% of 600 sampled critiques that successfully refined incorrect answers made correct discriminative judgments, and 93.3% were rated high-quality by GPT-4o.GPT-4o evaluated the critiques using ground-truth answers and solutions as references.Source: H. Qualitative Analysis · p. 23
  • Stage II detected the error, gave a constructive suggestion, and led to a correct refinement, unlike SFT, whose critique failed to detect errors and whose actor retained them.Source: H. Qualitative Analysis · p. 23, Figure 8 · p. 24
  • Stage I detected the error but gave a low-quality suggestion and led to an incorrect refinement; Stage II’s constructive suggestion led to a correct refinement.Source: H. Qualitative Analysis · p. 23, Figure 9 · p. 25
Loading 2510.24320v1…