PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving Critique Abilities in Large Language Models

Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, Junyang Lin

arXiv:2501.05727v2cs.CLcs.AIcs.LG

TL;DR

LLMs need scalable feedback for tasks that are difficult for humans to evaluate, while existing critique-improvement methods rely on human annotations or stronger models. SCRIT generates self-training data using reference-guided contrastive critique and self-validation, then improves critique abilities without references at inference. Across mathematical and scientific benchmarks, it delivers gains in critique-correction accuracy and error-identification F1.

  • Problem

    Effective and scalable feedback remains difficult for tasks where humans struggle to evaluate outputs and LLMs may exceed human capability, while current methods depend on human annotations or stronger supervisors.

    Source: §1 · pp. 1–2

  • Method

    SCRIT synthesizes self-training data by using reference solutions for contrastive critique and validating whether proposed corrections yield valid solutions.

    Source: §1 · p. 2

  • Results

    SCRIT consistently improves critique-correction and error-identification performance across mathematical and scientific reasoning benchmarks, including a 10.0% relative gain and a 19.0% relative F1 improvement.

    Source: §1 · p. 3, §3.3 · p. 7

  • Takeaways & Limitations

    Self-generated, reference-guided critique data can improve a strong LLM’s critique abilities without external supervision during critique writing or reference use at inference.

    Source: §5 · p. 10

    The current framework uses all available data rather than a curriculum that selects examples by difficulty, leaving optimal difficulty-based training selection unresolved.

    Source: §4.7 · p. 10

Abstract

from arXiv · show

Despite their remarkable performance, Large Language Models (LLMs) face a critical challenge: providing feedback for tasks where human evaluation is difficult or where LLMs potentially outperform humans. In such scenarios, leveraging the critique ability of LLMs themselves - identifying and correcting flaws - shows considerable promise. This paper explores enhancing critique abilities of LLMs, noting that current approaches rely on human annotations or more powerful models, leaving the challenge of improving critique abilities without external supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework that trains LLMs with self-generated data to evolve their critique abilities. To address the low quality of naively generated data, we propose a contrastive-critic approach that uses reference solutions during data synthesis to enhance the model's understanding of key concepts, and incorporates a self-validation scheme to ensure data quality. The final trained model operates without any reference solutions at inference time. Implemented with Qwen2.5-72B-Instruct, a leading LLM, SCRIT demonstrates consistent improvements across a wide range of benchmarks spanning both mathematical and scientific reasoning: achieving a 10.0\% relative gain in critique-correction accuracy and a 19.0\% relative improvement in error identification F1-score. Our analysis reveals that SCRIT's performance scales positively with data and model size and enables continuous improvement through multi-round iterations.

1 Introduction

SCRIT addresses the scalable-oversight challenge by improving LLM critique abilities without human-written critiques or stronger supervisors. It synthesizes self-training data with reference-guided contrastive critique and self-validation, producing substantial gains across mathematical and scientific reasoning benchmarks.

1 Introduction

  • SCRIT targets the unresolved problem of improving critique abilities when human evaluation is difficult and stronger supervisors are unavailable.Existing approaches depend on human annotations or stronger LLMs, creating quality, cost, and scalability limitations.Source: §1 · pp. 1–2
  • SCRIT uses reference solutions during data synthesis so models can identify and correct errors rather than rubber-stamping flawed solutions.The final model does not require reference solutions at inference time.Source: §1 · p. 2, Figure 1 · p. 2
  • Self-validation filters generated critiques by checking whether their proposed corrections produce valid solutions.Together with contrastive critique, this generates high-quality training data without human supervision for writing critiques.Source: §1 · pp. 2–3
  • 10.0% relative gain in critique-correction accuracy: SCRIT improves Qwen2.5-72B-Instruct from 39.7% to 50.0% on deliberately incorrect solutions and also improves balanced and self-generated solutions.Across three scenarios, performance reaches 50.0%, 62.1%, and 62.9%, respectively.Source: §1 · p. 3
  • 19.0% relative improvement in error-identification F1: SCRIT raises the average from 37.8% to 45.0% across PRM800K and ProcessBench.The reported gains span mathematical and scientific reasoning benchmarks.Source: §1 · p. 3
  • SCRIT is implemented with Qwen2.5-72B-Instruct and improves performance across evaluation protocols, including mathematical reasoning datasets and error-identification benchmarks.On eight mathematical reasoning datasets across three scenarios, SCRIT improves critique-correction accuracy from 39.7% to 50.0%, 57.7% to 62.1%, and 61.7% to 62.9%; average F1 rises from 37.8% to 45.0%.Source: §1 · p. 3

2 SCRIT: Self-Evolving Critic

SCRIT builds a self-evolving critique pipeline for verifiable mathematical and scientific problems, using reference-guided critique generation, validation, and self-training without external supervision.

2.1 Problem Formulation and Overview

  • SCRIT maps a problem and solution to a structured critique containing step-wise analyses, a correctness and first-error conclusion, and a correction.The framework learns this critique function from problem–solution data and completes a self-evolving cycle without external supervision.Source: §2.1 · p. 3

2.2 Solution Collection

  • 452K validated problem-answer pairs are collected from NuminaMath, with diverse solutions generated by seven models and retained problems requiring both correct and incorrect solutions.This filtering enables later contrastive comparison between solution types.Source: §2.2 · pp. 3–4

2.3 Self-Critic Generation

  • Contrastive Critic uses correct reference solutions to ground concept understanding before step-wise critique, addressing rubber-stamping in difficult mathematical reasoning.It compares correct–incorrect or correct–correct solution pairs and proceeds through reference analysis, critique, conclusion, and correction stages.Source: §2.3 · p. 4

2.4 Self-Validation

  • Self-validation directly checks each generated correction and retains only critiques whose corrections are judged fully correct with no errors.This filtering targets cases that approve intermediate steps but reject the final answer only after detecting a discrepancy.Source: §2.4 · pp. 4–5

2.5 Self-Training

  • Validated critique triplets train Qwen2.5-72B-Instruct using only the problem and student solution, so reference solutions are absent from training inputs and inference.The training set consists of validated problem–solution–critique triplets, while reference solutions are used only during data generation.Source: §2.5 · p. 5

3 Experiments

SCRIT experiments track data construction and self-validation, evaluate critique across mathematical and scientific benchmarks, and report gains over the Qwen2.5-72B-Instruct baseline. The results include improvements in critique-correction accuracy, error-identification F1, and continued self-evolution across iterations.

3.1 Statistics of SCRIT

  • 342K of 665K problem-solution pairs pass self-critique and self-validation, producing high-quality critique triplets for self-training.Validation rates decline with problem difficulty, from 91.8% on GSM8K and 77.6% on ORCA Math to 27.1% on Olympiads, and from 91.7% for single-answer problems to 15.5% for seven-answer problems.Source: §3.1 · p. 5
  • Self-validation is more sensitive to problem difficulty than to solution-model source, while detected error positions align with human-labeled ProcessBench distributions.Validation rates remain relatively consistent across solution models, ranging from 48.9% to 57.4%.Source: §3.1 · pp. 5–6

3.2 Evaluation

  • SCRIT evaluates critique-and-correction across eight mathematical and scientific benchmarks and three scenarios, alongside error identification on PRM800K and ProcessBench.The protocols cover incorrect, balanced, and self-generated solutions, with mathematical and scientific reasoning datasets in the first protocol.Source: §3.2 · p. 6
  • Figure 3 examines data-size scaling, model-size scaling from 1.5B to 72B, and three rounds of multi-round self-evolution.The supplied figure description identifies these scaling dimensions, while the experiments use Contrastive Critic, Direct Critic, and Bug-Injection Critic comparisons.Source: Figure 3 · p. 7

3.3 Main Results

  • 10.0% relative gain in critique-correction accuracy brings SCRIT close to o1-mini across incorrect, balanced, and self-generated solutions.SCRIT reaches 50.0% versus 39.7% on deliberately incorrect solutions, retains a 4.4% advantage on balanced solutions, and reaches 62.9% versus 61.7% on the base model’s own solutions.Source: Table 1 · p. 6, §3.3 · p. 7
  • 19.0% relative improvement in error-identification F1 raises the average from 37.8% to 45.0% across PRM800K and ProcessBench.SCRIT improves consistently across datasets, including gains of 11.3% on GSM8K and 9.1% on MATH, while remaining below o1-mini.Source: Table 2 · p. 6, §3.3 · p. 7

4 Analysis

SCRIT improves critique across scientific and mathematical reasoning, scales with data and model size, and benefits from contrastive criticism, self-validation, and iterative self-training. Controlled analyses also identify domain diversity and solution-ratio effects, while difficulty and solution-source choices delimit performance.

4.1 Generalization to Scientific Reasoning

  • SCRIT generalizes to scientific reasoning, while scientific-domain training reaches a further +10.6% on balanced solutions and remains competitive on mathematics.The evaluation includes ARC-C, GPQA, and MMLU-STEM, and the scientific-data model captures domain-specific error patterns.Source: §4.1 · p. 7

4.2 Scaling Behavior of SCRIT

  • CC-Acc rises from 53.0% to 58.3% and EI-F1 from 37.8% to 45.1% as training data increases from 10K to 170K examples.The steepest gains occur early, followed by continued but more gradual improvement.Source: §4.2 · p. 8
  • CC-Acc increases from 41.7% to 58.3% and EI-F1 from 12.5% to 45.1% across 1.5B, 7B, and 72B models, with stronger scaling for error identification.SCRIT also improves Qwen2.5-32B-Instruct from 53.9% to 56.5% CC-Acc and 35.8% to 41.5% EI-F1.Source: §4.2 · p. 8

4.3 Which Critic Mechanism is Most Effective?

  • 58.3% CC-Acc and 45.1% EI-F1 at 170K examples show Contrastive Critic outperforming Direct and Bug-Injection Critics, which plateau or degrade.At 10K examples, Contrastive Critic already reaches 56.8% CC-Acc and 40.2% EI-F1; Bug-Injection Critic falls to 49.0% CC-Acc at 170K.Source: §4.3 · p. 8

4.4 Does Multi-round Iteration Foster Improvement?

  • Three SCRIT rounds produce additional gains in both metrics, indicating positive multi-round scaling and continued self-evolution without external supervision.Each round uses the enhanced model from the preceding round as its new base.Source: §4.4 · p. 9

4.5 How Important is Self-Validation?

  • Removing self-validation decreases CC-Acc by 0.8% and EI-F1 by 3.0%, while restricting training to GSM8K and MATH decreases both metrics by 1.4%.The analyses attribute these effects to filtering low-quality critiques and exposing the model to diverse problem-solving patterns and error types.Source: §4.5 · p. 9, §4.6 · p. 9

4.7 How Does Problem Difficulty Impact Performance?

  • Training on less complex problems yields better EI-F1, leaving optimal difficulty-based example selection as an open question.The authors suggest that simpler problems provide more structured concepts and strategies for reliable critique patterns.Source: §4.7 · p. 10

4.8 Does the Choice of Solution Model Matter?

  • Solution-source models have limited impact, whereas a higher bad-to-good solution ratio improves performance by increasing exposure to mistakes and corrections.The authors emphasize constructing diverse, informative contrastive pairs rather than relying on particular solution generators.Source: §4.8 · p. 10, §4.9 · p. 10

5 Conclusion

The paper positions SCRIT as a self-supervised alternative to human- or stronger-model-supervised critique training. It builds reliable mathematical ground truth from NuminaMath and uses reference solutions during synthesis while targeting critique improvement in verifiable domains.

5 Conclusion

  • The paper’s conclusion reports gains in critique-correction accuracy and error detection across mathematical and scientific reasoning, alongside positive scaling with data and model size.It also identifies self-validation and cross-domain generalization as strengths, while proposing extensions to coding and logic.Source: §5 · p. 10

A Related Work

  • Existing critique methods rely on human annotations or stronger models, making critique quality dependent on supervisor capability and supervision cost.Related work includes human-supervised critique, human-injected errors, and GPT-4 or o1-mini feedback.Source: A Related Work · pp. 13–14

5 Conclusion

  • SCRIT creates a closed-loop critique system that analyzes reference solutions, validates critiques through correction outcomes, and self-trains without external supervision.Its stated scope is domains with verifiable solutions, especially mathematical and scientific reasoning.Source: §5 · p. 10, A Related Work · p. 14

B Computing Ground Truth Answers for NuminaMath

  • NuminaMath supplies 860K diverse problems, but the paper constructs reliable ground truth because original solution correctness is not guaranteed.Solutions are generated with Qwen2.5-Math-72B-Instruct, evaluated by Qwen2.5-Math-RM-72B, and filtered using reward scores.Source: B Computing Ground Truth Answers for… · p. 14, §B.1 · p. 14

B.2 Threshold Selection and Validation

The ground-truth pipeline selects answers with a validated reward threshold, yielding a filtered dataset for subsequent training. The prompting templates distinguish direct, bug-injection, and contrastive critics, with contrastive analysis supporting more precise error localization.

B.2 Threshold Selection and Validation

  • A reward threshold of 1.0 achieves approximately 75% benchmark accuracy and approximately 85% accuracy in human evaluation.Majority voting among alternative solution sources performs worse than reward-based selection.Source: §B.2 · p. 14
  • The validated pipeline produces 452K problem-answer pairs for the paper’s foundation.The dataset follows threshold-based answer generation and validation.Source: §B.2 · p. 14

C Prompting Templates for Direct Critic, Bug-Injection Critic and Contrastive Critic

  • Direct Critic analyzes correctness directly, Bug-Injection Critic injects bugs before critique, and Contrastive Critic first analyzes a reference solution before step-wise critique.The three mechanisms differ in the contextual information supplied before critique generation.Source: Figure 4 · p. 15

D More Comparison between Direct Critic and Contrastive Critic

  • Contrastive Critic avoids blind approval by grounding critique in reference-solution concepts, enabling precise error localization and correction toward the correct answer.The comparison contrasts misleading approval from Direct Critic with effective critique from Contrastive Critic.Source: Figure 5 · p. 15

E Self-Validation Cases

The self-validation cases show how inconsistent corrections are rejected while logically correct critiques are accepted. The bug-injection cases illustrate that injected errors can be overly simple, limiting their usefulness for complex reasoning.

E Self-Validation Cases

  • Self-validation rejects a critique that misses the first trigonometric-equation error and produces a conflicting final answer.The rejected critique had access to a reference solution and used contrastive learning but still failed the consistency check.Source: Figure 6 · p. 16
  • Self-validation accepts a critique that identifies the continuity error, applies logical correction, and reaches the correct final answer.The verified critique is retained for training.Source: Figure 7 · p. 16

F Bug-Injection Case Study

G Adaptations to ProcessBench’s Evaluation Protocol

The evaluation protocol is adapted to test whether models genuinely understand mathematical errors rather than merely matching annotated steps. It therefore requires valid correction and recognizes that some errors span multiple consecutive steps.

G Adaptations to ProcessBench’s Evaluation Protocol

G.1 Requiring Effective Correction

  • Figures 10–12 show critics matching human-labeled error steps while misunderstanding graph theory, perpendicular medians, or finite geometric series.These cases produce incorrect corrections despite apparently correct error localization.Source: §G.1 · p. 17, Figure 10 · p. 18, Figure 11 · p. 18
  • Models must identify the error step and provide a correction leading to a mathematically valid solution.This stricter requirement addresses cases where matching human annotations does not demonstrate understanding.Source: §G.1 · pp. 18–19

G.2 Allowing Step-Level Flexibility

The protocol allows flexibility because mathematical errors may reasonably be assigned to multiple consecutive steps. A ±1-step matching tolerance preserves evaluation rigor while better reflecting this ambiguity.

G.2 Allowing Step-Level Flexibility

  • Mathematical errors can span multiple consecutive steps, making strict step-level matching overly rigid.Figures 13–15 illustrate cases involving different plausible error locations and continuous chains of incorrect reasoning.Source: §G.2 · p. 19, Figure 13 · p. 19, Figure 14 · p. 20
  • A ±1-step tolerance is used when matching model predictions with human annotations.This modification accounts for ambiguity while maintaining rigor in evaluation.Source: §G.2 · p. 20
  • The resulting protocol better captures mathematical-error understanding and the ability to guide effective corrections.The adjustment is intended to make evaluation more meaningful than exact step matching alone.Source: §G.2 · p. 20

H Distribution of First Error Step identified by Self-Critic

This section uses Qwen2.5-72B-Instruct to classify solutions as correct or incorrect and presents the associated system prompt.

H Distribution of First Error Step identified by Self-Critic

J Self-Training Details

Self-training uses distributed continued supervised fine-tuning with open-instruct and memory-saving training techniques. The configuration uses Qwen2.5-72B-Instruct across 32 NVIDIA A100 GPUs.

J Self-Training Details

  • Continued supervised fine-tuning is implemented with open-instruct on 4 servers containing 32 NVIDIA A100 GPUs.Training takes several hours, with exact time varying by hardware configuration and system load.Source: J Self-Training Details · p. 21
  • The training configuration uses 1 epoch and model parallel size 8.Source: J Self-Training Details · p. 22
  • Gradient checkpointing, FP16 mixed precision, and DeepSpeed ZeRO-3 are used to reduce memory usage and support distributed training.Source: J Self-Training Details · p. 22
Loading 2501.05727v2…