Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning

Chi Ruan; Dongfu Jiang; Yubo Wang; Wenhu Chen

Source-linked AI summary

Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning

Chi Ruan, Dongfu Jiang, Yubo Wang, Wenhu Chen

arXiv:2509.22824v2Sep 26, 2025cs.CL

TL;DR

Standard RLVR rewards solution generation but does not explicitly elicit critique; this paper introduces CRL and trains Critique-Coder with both critique learning and standard RLVR. Critique-Coder outperforms RL-only models across evaluated benchmarks and also shows gains on logical reasoning, although it does not reliably critique final answers when reasoning traces are absent.

Problem
Standard RLVR improves problem solving but provides little mechanism to elicit critique or reflection on existing solutions.
Source: §1 · p. 2
Method
CRL rewards a model for correctly judging a question–solution pair, and Critique-Coder combines CRL with standard RLVR.
Source: §1 · p. 2
Results
Critique-Coder outperforms RL-trained models across evaluated benchmarks; on BBEH, it exceeds baseline and RL-trained models by +6.1 and +4.0 points on average across four subtasks.
Source: §3.3 · p. 7, §5 · p. 10
Takeaways & Limitations
The results support CRL as a complement to RL, with coding-trained Critique-Coder also showing transferable general reasoning on logic benchmarks.
Source: §1 · p. 2, §3.3 · p. 7
CRITIQUE-CODER does not reliably self-critique completed answers when only the final answer is available without intermediate reasoning trajectories or external signals.
Source: §3.7 · p. 9

Abstract

from arXiv · show

Reinforcement Learning (RL) has emerged as a popular training paradigm, particularly when paired with reasoning models. While effective, it primarily focuses on generating responses and lacks mechanisms to explicitly foster critique or reflection. Several recent studies, like Critique-Fine-Tuning (CFT) and Critique-Guided-Distillation (CGD) have shown the benefits of explicitly teaching LLMs how to critique. Motivated by them, we propose Critique Reinforcement Learning (CRL), where the model is tasked with generating a critique for a given (question, solution) pair. The reward is determined solely by whether the final judgment label $c \in \{\texttt{True}, \texttt{False}\}$ of the generated critique aligns with the ground-truth judgment $c^*$. Building on this point, we introduce Critique-Coder, which is trained on a hybrid of RL and CRL by substituting 20% of the standard RL data with CRL data. We fine-tune multiple models (Critique-Coder) and evaluate them on different benchmarks to show their advantages over RL-only models. We show that Critique-Coder consistently outperforms RL-only baselines on all the evaluated benchmarks. Notably, our Critique-Coder-8B can reach over 60% on LiveCodeBench (v5), outperforming other reasoning models like DeepCoder-14B and GPT-o1. Beyond code generation, Critique-Coder also demonstrates enhanced general reasoning abilities, as evidenced by its better performance on logic reasoning tasks from the BBEH dataset. This indicates that the application of CRL on coding datasets enhances general reasoning and critique abilities, which are transferable across a broad range of tasks. Hence, we believe that CRL works as a great complement to standard RL for LLM reasoning.

1 INTRODUCTION

The introduction motivates CRL as an explicit critique signal missing from standard RLVR, then presents Critique-Coder’s hybrid training and reported coding and logic gains.

The introduction situates code-reasoning gains in RL paired with chain-of-thought, alongside code-data pipelines and rewards based on tests or rules.Source: §1 · pp. 1–2
CRL rewards correctly judging a supplied question–solution pair, rather than generating a solution, and compares the predicted True/False label with the annotated judgment.Source: §1 · p. 2
Critique-Coder combines CRL with standard RLVR for QWEN3-4B and QWEN3-8B, iteratively extending training context from 16k to 32k tokens.Source: §1 · p. 2
60.8 points on LiveCodeBench (v5), Critique-Coder-8B outperforms DeepCoder-14B and GPT-o1; on BBEH, Critique-Coder improves +6.1 over baseline.Source: §1 · p. 2

2 METHOD

CRITIQUE-CODER unifies standard RL and CRL for code generation, training on both solution-generation and critique-judgment examples. The method covers dataset construction and a shared GRPO training procedure.

2.1 PRELIMINARY

Standard RL scores sampled solutions by test-case pass rate, while CRL rewards whether a critique’s judgment matches its labeled question–solution pair; GRPO combines both signals.Source: §2.1 · pp. 3–4

2.2 DATASET CONSTRUCTION

Filtering reduces average test input characters from 96,208 to 40 and cases from 87 to 24, shortening verification time.Source: §2.2 · p. 4
Candidate programs are generated and evaluated on filtered tests; a pass rate above 80% labels a solution True, otherwise False.The generating model’s 32.20% pass@1 is below QWEN3-4B (Thinking)’s 43.72%, so the authors say this is not distillation.Source: §2.2 · pp. 4–5, Figure 3 · p. 4
The hybrid dataset assigns 20% of samples to CRL and 80% to standard RL, balancing critique judgments with direct solution generation.The paper says this configuration mitigates format shift from overexposure to critique-style supervision.Source: §2.2 · p. 5

2.3 TRAINING

For CRL, the model extracts a judgment from its critique and earns 1 for matching ground truth or 0 otherwise; for RL, reward is test-case pass rate.Rewards become advantage estimates that jointly update the policy through GRPO.Source: §2.3 · pp. 5–6

3 EXPERIMENT

Experiments compare QWEN3-4B and QWEN3-8B against base and RL-trained models across coding benchmarks, then examine reasoning transfer, test-time scaling, CRL data ratios, and self-critique. CRITIQUE-CODER improves results over base and RL-only models, with gains also reported on BBEH logic tasks.

3.1 TRAINING SETUP

Training uses GRPO on QWEN3-4B and QWEN3-8B, with CRL rewards based on matching the ground-truth judgment and RL rewards based on test-case pass rates.The setup uses two response-length phases, 16k then 32k, and selects the checkpoint using LiveCodeBench (v5) validation performance.Source: §3.1 · p. 6

3.2 EVALUATION SETUP

Evaluation covers EvalPlus, BigCodeBench-Instruct, Aider-Polyglot, and LiveCodeBench (v5), using consistent sampling settings across tasks.The comparison includes coding baselines such as DeepSeek-R1-distill-14B, DeepCoder, DeepSeek-V2.5, and GPT-o1.Source: §3.2 · p. 6

3.3 MAIN RESULTS

59.0 LiveCodeBench score for CRITIQUE-CODER-4B is +4.8 over its base model and +2.4 over its RL-only variant; the 8B model improves over RL-only models across benchmarks.The 8B model gains +2.7 on BigCodeBench-Hard and +1.7 on the overall benchmark average over its RL-trained counterpart; 4B outperforms DeepCoder-14B while using 28% of its parameters. CRITIQUE-CODER generations also show longer reasoning traces and more explanatory code comments.Source: Table 3 · p. 7, §3.3 · p. 7

3.5 TEST TIME SCALING

62.0 LiveCodeBench (v5) accuracy after four sequence-scaling iterations is a +3.0 absolute gain over the 59.0 no-scaling baseline.Further reasoning-budget increases yield marginal but stable improvements; parallel test-time scaling does not yield gains.Source: Figure 5 · p. 8, §3.5 · p. 8

3.4 TRANSFERABLE REASONING ABILITY

CRITIQUE-CODER outperforms QWEN3-4B and its RL-trained variant on all four BBEH logic subtasks, averaging +6.1 points over the base model.It gains +8.0 on Time Arithmetic and +7.5 on Zebra Puzzles; all models use the same inference configuration.Source: Table 4 · p. 8, §3.4 · p. 8

3.6 ABLATION STUDY

47.2 average score with 20% CRL data is +1.5 over the pure RL baseline and is the best overall result among tested ratios.Performance gains diminish at higher CRL ratios; full replacement yields 44.5, below baseline, and the 50% setting degrades on BigCodeBench-I (Hard).Source: Table 5 · p. 9, §3.6 · p. 9

3.7 LIMITATIONS IN SELF-CRITIQUE

CRITIQUE-CODER does not enable reliable post-hoc self-critique when only the final answer is available, and critique-based parallel scaling did not improve performance.The authors distinguish this limitation from training-time critique, which supports reasoning and refinement before the final answer is produced.Source: §3.7 · p. 9

4 RELATED WORKS

Prior critique research includes self-correction, learned evaluators, and critique fine-tuning, while related work also develops RL for code generation and chain-of-thought reasoning. CRL differs from critique imitation by learning from feedback on final judgments.

4.1 CRITIQUE LEARNING

Prior critique work includes iterative self-correction, reward models that score outputs or reasoning steps, and CFT for critiquing candidate solutions.The passage notes that subsequent studies questioned the robustness and consistency of self-correction methods.Source: §4.1 · p. 10
CRL encourages active learning from final-judgment correctness, unlike CFT’s direct imitation of the critique reasoning process.The authors describe their approach as combining critique reasoning with reinforcement feedback.Source: §4.1 · p. 10

4.2 REINFORCEMENT LEARNING FOR CODE GENERATION

Code-generation RL work progressed from CodeRL’s actor–critic framework through PPOCoder and execution feedback in RLEF to AceCoder’s scalable question–test construction pipeline.Source: §4.2 · p. 10

4.3 CHAIN-OF-THOUGHT REASONING.

Reasoning-model work uses inference-time scaling and reflection, while Qwen3 combines thinking and non-thinking modes to balance latency and performance.The passage also describes KIMI K1.5 as incorporating long-context scaling and enhanced policy optimization.Source: §4.3 · p. 10

5 CONCLUSION

The conclusion presents CRL as a complement to standard RL, reporting gains in coding and logical reasoning while noting that critique-only training performs worse than RL alone.

59.0 on LiveCodeBench v5, CRITIQUE-CODER-4B exceeds the baseline by +4.8 points and the RL-only model by +2.4 points.Source: §5 · p. 10
+6.1 and +4.0 points on average across four BBEH subtasks, CRITIQUE-CODER surpasses the baseline and RL-trained models, respectively.The conclusion presents this as evidence of gains beyond coding tasks.Source: §5 · p. 10
Training exclusively on CRL data performs worse than RL alone because critique generation does not optimize task-oriented solutions, so the authors position CRL as a complement to RL.Source: §5 · p. 10
CRL incorporates feedback on the correctness of the critiques the model predicts, and the resulting capabilities are reported to extend beyond coding.Source: §5 · p. 10

A CRL TRAINING PROMPT

The section provides a prompt template for constructing CRL training data, asking a model to assess a submitted solution against a problem specification.

The section supplies the prompt template used to construct CRL training data.Source: A CRL TRAINING PROMPT · p. 15
The CRL training prompt asks the model to determine whether a submitted solution is correct and fully satisfies the specification.Source: Table 6 · p. 15
The prompt takes a question, described as a problem specification, and a submitted solution as its inputs.Source: Table 6 · p. 15