PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching Language Models to Critique via Reinforcement Learning

Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong

arXiv:2502.03492v2cs.LGcs.AIcs.CL

TL;DR

Self-improvement loops need feedback that can both judge correctness and offer actionable fixes. CTRL trains a decoupled critic with execution-guided supervision and reinforcement learning; it improves code-generation results across benchmarks and supports iterative scaling with base and stronger generators.

  • Problem

    Self-improvement can degrade without suitable external feedback, while reward models and verification tools often fail to provide actionable high-level guidance.

    Source: §1 · p. 1

  • Method

    CTRL decouples the critic from the task-performing model, first fine-tunes it on execution-feedback critiques, then optimizes it with GRPO for correction performance.

    Source: §1 · p. 1

  • Results

    Across four code benchmarks CTRL outperforms self-critique and stronger-critic methods; on CodeContests it achieves up to 106.1% relative Pass@1 improvement with its base model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CTRL reduces revision iterations and token consumption while increasing success rates, with its efficiency attributed to reduced error compounding.

    Source: §1 · p. 2

    The work focuses on pass rates; optimizing efficiency and safety and extending training to multi-turn critique revision remain future directions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Teaching large language models (LLMs) to critique and refine their outputs is crucial for building systems that can iteratively improve, yet it is fundamentally limited by the ability to provide accurate judgments and actionable suggestions. In this work, we study LLM critics for code generation and propose $\texttt{CTRL}$, a framework for $\texttt{C}$ritic $\texttt{T}$raining via $\texttt{R}$einforcement $\texttt{L}$earning, which trains a critic model to generate feedback that maximizes correction performance for a fixed generator model without human supervision. Our results demonstrate that critics trained with $\texttt{CTRL}$ significantly enhance pass rates and mitigate compounding errors across both base and stronger generator models. Furthermore, we show that these critic models act as accurate generative reward models and enable test-time scaling through iterative critique-revision, achieving up to 106.1% relative improvements across challenging code generation benchmarks.

1. Introduction

Self-improvement loops need feedback that both discriminates correctness and gives actionable suggestions. CTRL trains a decoupled critic to guide code revision, outperforming prior critique approaches and helping reduce error compounding during iterative refinement.

1. Introduction

  • Feedback remains a bottleneck because reward models offer simplified numerical signals and verification tools provide low-level traces rather than high-level fixes.Effective feedback must discriminate solution correctness and offer informative, actionable suggestions.Source: §1 · p. 1
  • CTRL decouples the critic from the task-performing model so a specialized critic can guide that model through iterative critique and revision.The task-performing model may be a stronger generator such as GPT-4o.Source: §1 · p. 1
  • Execution feedback first supplies critiques for supervised fine-tuning; GRPO then optimizes the critic against correction performance despite the large space of possible critiques.The training signal measures whether critiques help the task-performing model reach correct outputs.Source: §1 · p. 2
  • CTRL reduces revision iterations and token use while increasing success rates, with reduced error compounding attributed to early mistake correction.The critic guides the generator toward more direct solution paths without compromising solution quality.Source: §1 · p. 2
  • 106.1% and 23.5% relative Pass@1 improvements on CodeContests pair CTRL with its base and a stronger model, respectively; across four benchmarks it outperforms self-critique and stronger-critic methods.The stronger generator is GPT-4o.Source: §1 · p. 2

2. Preliminaries and Motivation

Iterative refinement depends on both evaluating candidate solutions and providing feedback that can improve them. Simulations suggest that stronger critique compounds gains across attempts and can offset weaker discrimination.

2. Preliminaries and Motivation

  • Iterative refinement starts from an initial solution proposal and uses feedback on each candidate to produce the next solution.The evaluation function assigns each candidate a binary correct-or-incorrect outcome.Source: §2 · p. 3
  • The framework distinguishes discrimination, which ranks solutions, from critiquing, which supplies actionable feedback; critique quality is modeled through correctness-transition dynamics.The transition parameters describe avoiding damage to correct solutions and improving incorrect ones.Source: §2 · p. 3
  • The simulations compare no critique, weak critique, and strong critique from an initial correctness probability of 0.1, then select the best of n generated solutions.No critique corresponds to independent sampling, or best-of-n sampling.Source: §2 · p. 3
  • Strong critique improves success over no critique, with benefits compounding across attempts while no-critique approaches plateau quickly.The simulations plot final correctness probability against the number of attempts after selecting among generated solutions.Source: §2 · p. 3, Figure 3 · p. 3
  • Strong critique can compensate for weaker discrimination: weak discrimination with strong feedback can outperform stronger discrimination without critique.The findings emphasize improving both capabilities, particularly actionable critique.Source: §2 · p. 3

3. Method

CTRL trains a specialized critic separately from the task-performing model, using improved solution quality as a proxy for critique helpfulness. Its two-stage approach synthesizes execution-guided critiques and then refines the critic through reinforcement learning for iterative code improvement.

3.1. Problem Statement

  • The setup pairs a solution-generating model with a critic, which is trained to maximize the revised solution’s expected quality as a proxy for critique helpfulness.The sandbox evaluates code against unit tests during training, but does not assist critique generation at test time.Source: §3.1 · pp. 3–4

3.2. Stage I: Execution-guided Critique Synthesis

  • Self-critique without additional feedback yielded only 7.88% → 8.36% and rarely corrected incorrect solutions.These results illustrate the difficulty of learning effective critiques from the large space of possible feedback.Source: §3.2 · p. 4
  • 11.76% vs. 8.97% Pass@1: reasoning over execution feedback to generate critiques outperformed directly using raw execution feedback for revisions.The comparison is on CodeContests with Qwen2.5-Coder as the fixed generator.Source: Table 1 · p. 4, §3.2 · p. 4
  • Stage I uses sandbox-derived hints—positive feedback for passing solutions, restart suggestions for total failures, and error details for partial failures—and supervised finetuning to internalize these strategies.Finetuning also supplies ground-truth correctness to improve discrimination; predefined templates may miss nuanced feedback needed for complex programming tasks.Source: §3.2 · p. 4, §3.3 · p. 4

3.3. Stage II: Reinforced Critique Generation

  • GRPO avoids value networks and uses group-relative reward advantages; this normalizes across problem types and yields zero advantages on problems that are too easy or too hard.The authors report noisy critique-quality estimates with PPO’s learned value networks.Source: §3.3 · p. 5

4. Experiments

Experiments assess critic discrimination and revision quality across three programming benchmarks and JudgeBench, using 18,820 filtered TACO problems and varied generator models. CTRL improves iterative critique-revision performance, generalizes across generators, and shows trade-offs in revision behavior and execution time.

4.2. Evaluating Critics for Iterative Critique-revisions

  • 11.76% pass@1 with the RL-trained critic exceeded 7.88% with zero-shot generation, alongside a 0.85% vs. 3.03% regression rate against its SFT counterpart.The regression rate measures initially correct solutions revised into incorrect ones.Source: §4.2 · p. 6
  • Three CTRL critique rounds raised Pass@1 from 11.76% to 15.15% while ∆↓ remained 0.85%; Figure 4 reports lower compounding than Qwen2.5-Coder and GPT-4o.The critic was trained on single-turn critiquing tasks but generalized to multi-turn settings.Source: §4.2 · p. 6, Figure 4 · p. 7
  • A critic trained with Qwen2.5-Coder generalized across programming tasks and helped the stronger GPT-4o generator.Table 3 compares performance across generators and benchmarks.Source: §4.2 · p. 6, Table 3 · p. 6

4.3. Evaluating Critics as Generative Reward Models

  • CTRL’s relative pass@1 gains grew with problem difficulty and iteration on CodeContests.Figure 5 compares pass@1 by problem difficulty; results are averaged over five seeds.Source: §4.2 · p. 6, Figure 5 · p. 7
  • 64.3% overall accuracy was comparable to Claude-3.5-Sonnet, with superior coding-specific performance despite programming-only training.Figure 6 presents the JudgeBench model-performance comparison.Source: Figure 6 · p. 7, §4.3 · p. 7

4.4. Analysis

  • In inference-only tests, 32B critics achieved up to 50% improvement with similarly sized generators, while 7B critics often degraded performance.Critics less capable than their generators typically degraded results, motivating subsequent CTRL finetuning experiments with 32B models.Source: §4.4 · pp. 7–8
  • CTRL revisions had lower mean code similarity than self-critique (0.313 vs. 0.482), indicating more substantial changes.Figure 7 displays similarity-score distributions; lower scores indicate more substantial revisions.Source: Figure 7 · p. 8, §4.4 · p. 8
  • On LiveCodeBench, CTRL’s 16.61% timeout rate exceeded zero-shot’s 10.54% and GPT-4o critic’s 8.93%, despite higher overall Pass@1 accuracy.The authors associate the higher timeout rate with more comprehensive solutions that take longer to execute.Source: §4.4 · p. 8

5. Related Work

Related work studies autonomous self-improvement, trained LLM critics, and test-time scaling. CTRL is distinguished by execution-feedback-based critique training and its use of a decoupled critic for iterative refinement.

Self-Improvement of LLMs

LLM Critics

Scaling Test-Time Compute

  • Test-time scaling commonly uses repeated sampling or modular frameworks; CTRL instead uses a trained decoupled critic to guide revisions, while a cited modular approach struggles with code generation (Saad-Falcon et al., 2024).The passage identifies repeated sampling with selection mechanisms (Brown et al., 2024) as another approach.Source: §5 · p. 8

6. Conclusion

The conclusion presents CTRL as reinforcement-learning-based critic training for iterative refinement, reporting improvements across benchmarks and test-time scaling, including guidance of stronger generators. It identifies efficiency, safety, and multi-turn critique-revision as future directions.

6. Conclusion

  • CTRL is presented as a reinforcement-learning framework for training critics to provide effective iterative-refinement feedback.The conclusion describes improvements over baselines across multiple benchmarks, including when guiding stronger generators.Source: §6 · p. 8
  • The work focuses on improving pass rates; optimizing efficiency and safety remains future work.The conclusion also names extending the pipeline toward multi-turn critique-revision as a future direction.Source: §6 · p. 9
  • The authors aim to extend the training pipeline toward multi-turn critique-revision.Source: §6 · p. 9

Impact Statement

The impact statement reports no immediate societal concerns requiring specific attention. The supplementary material outlines the training pipeline, related-method categories, simulation setup, and critique-generation prompts.

Impact Statement

  • The authors report no immediate societal concerns requiring specific attention, while noting that broader deployment and potential misuse merit consideration.Source: Impact Statement · p. 9

A. Pipeline

  • Figure 8 summarizes SFT critique training from execution feedback, followed by RL training that validates critic-guided revisions in a sandbox.The SFT stage trains the final critic using supervised finetuning; the RL stage uses its feedback to guide solution improvement.Source: Figure 8 · p. 12, A. Pipeline · p. 12

B. Supplementary Discussion of Related Work

  • CTRL contrasts with scalar reward models, generative reward models, and human-annotated critic models by combining actionable unsupervised critiques with execution feedback and reinforcement learning.The passage characterizes the first two categories as discrimination-focused and the latter as reliant on human-annotated critique data.Source: B. Supplementary Discussion of Related… · p. 12

C.1. Simulation

  • Figure 9 diagrams the compared selection processes; simulations vary critiquing (pcc, pcw) and discrimination (TPR/FPR) over 50,000 refinement trials.The simulation uses a Markov chain with initial correctness and probabilities for maintaining or improving correctness.Source: §C.1 · pp. 12–13, Figure 9 · p. 13

C.2. Prompt Templates

  • Execution results map to predefined hints that condition critiques; critiques explicitly referencing hints are filtered to prevent hallucination.The critique prompt includes the problem, initial solution, and corresponding hint.Source: §C.2 · p. 13

C.3. Training

CTRL training uses curated code problems and synthesized critiques for supervised finetuning, then trains the critic with reinforcement learning over multiple solutions per problem. The critique prompts request analysis, improvement suggestions, and an overall correctness judgment, while the execution-guided template also directs the model not to give direct solutions or mention the hint.

C.3. Training

  • The curated TACO dataset contains 18,820 deduplicated problems after malformed cases, problematic tests, and 47 benchmark-overlapping problems are excluded.It covers function-based and input-output-based problems, with their unit tests processed into assertion statements or sandbox stdin-stdout pairs, respectively.Source: §C.3 · p. 13
  • SFT trains on one initial solution and one synthesized critique per problem to enable the model to generate improved solutions.The training examples are problem-solution-critique pairs.Source: §C.3 · p. 13
  • The execution-guided critique prompt asks for analysis of an answer and constructive feedback, guided by a supplied hint, while forbidding direct solutions and any mention of the hint.Source: §C.3 · p. 14
  • RL training samples four initial solutions per problem and uses the corresponding problem-solution pairs to expose the critic to solution diversity and mitigate overfitting.Training uses VeRL and the hyperparameters listed in Table 9.Source: §C.3 · p. 14

C.4. Evaluation.

Evaluation uses specified decoding settings, majority-vote critic judgments, and a normalized-code similarity measure. The reported majority-vote accuracy improves as the number of votes increases.

C.4. Evaluation.

  • Inference uses temperature 0.7 for initial solutions and critiques, greedy decoding for revised solutions, and a 1,024-token maximum at every stage.RL hyperparameters are listed in Table 9.Source: §C.4 · p. 15, Table 9 · p. 15
  • Code similarity is measured after AST-based variable canonicalization and formatting normalization, using difflib.SequenceMatcher to score matching characters from 0 to 1.This follows Zheng et al. (2024) and is intended to account for semantically equivalent code with different variable names.Source: §C.4 · p. 15
  • Majority-vote accuracy improves as votes increase when critic-generated critiques are aggregated to compare solution pairs by their frequency of “Correct” labels.Figure 10(a) presents this accuracy improvement with increasing votes.Source: §C.4 · p. 15, Figure 10 · p. 16

D. The Credit Assignment Issue for Critic Training

PPO training encountered unstable value predictions because holistic critique quality made credit assignment difficult. The authors switched to GRPO, which uses Monte Carlo sampling to estimate advantages without a value network.

D. The Credit Assignment Issue for Critic Training

  • Unstable value predictions in PPO prompted the switch to GRPO, which estimates advantages through Monte Carlo sampling without a value network.Figure 10(b) shows the value-network training curve.Source: D. The Credit Assignment Issue for… · p. 16
  • Critique credit assignment is difficult because quality depends holistically on interdependent aspects such as accuracy, completeness, and constructiveness.The value network struggles to learn which parts of a generated critique contributed to its final reward.Source: D. The Credit Assignment Issue for… · p. 16
  • The authors report that GRPO provides more reliable credit assignment by avoiding the potentially unstable value-prediction problem.Source: D. The Credit Assignment Issue for… · p. 16

E. CTRL Samples

The appendix presents CTRL-generated critiques for Qwen2.5-Coder solutions across CodeContests, LiveCodeBench, and MBPP+.

E. CTRL Samples

Loading 2502.03492v2…