PaperlayerView exact revision on arXiv

Source-linked AI summary

Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate

Yubo Wang, Xiang Yue, Wenhu Chen

arXiv:2501.17703v4cs.CL

TL;DR

SFT trains language models to imitate annotated responses, motivating the question of whether reasoning can be improved more effectively through critique-based learning. The paper introduces CFT, which trains models to critique noisy responses, and reports consistent gains over SFT, strong data and compute efficiency, and broader instruction-following improvements, while noting limitations in critique quality and self-critique.

  • Problem

    SFT trains models to imitate annotated responses, while the paper investigates a more efficient approach for improving reasoning than scaling instruction data.

  • Method

    CFT trains models to generate critiques for questions paired with noisy responses, using critique datasets synthesized across WebInstruct, MetaMathQA, and NuminaMath.

  • Results

    CFT consistently outperforms SFT by 4–10% on mathematical reasoning benchmarks, matches resource-intensive RL performance with 50K examples and 8 H100 GPU-hours, and generalizes to broader STEM and instruction-following evaluations.

  • Takeaways & Limitations

    CFT provides a more data- and compute-efficient alternative to imitation-based training while improving reasoning and general instruction-following capabilities.

  • Takeaways & Limitations

    About 20% of manually inspected GPT-4o-generated critiques contain inaccuracies, and self-critique underperforms direct inference because of inconsistent standards and instability.

Abstract

from arXiv · show

Supervised Fine-Tuning (SFT) is commonly used to train language models to imitate annotated responses for given instructions. In this paper, we propose Critique Fine-Tuning (CFT), a method more effective than SFT for reasoning tasks. Instead of simply imitating correct responses, CFT trains models to critique noisy responses, inspired by human learning processes that emphasize critical thinking, deeper analysis, and nuanced understanding - traits often overlooked by standard SFT. To validate the effectiveness of CFT, we construct multiple critique datasets (e.g., WebInstruct, MetaMath, NuminaMath), where GPT-4o serves as the teacher to generate critiques in the form of ([query; noisy response], critique). Experiments on these datasets demonstrate that CFT consistently outperforms SFT by 4-10% across six mathematical reasoning benchmarks, and is effective across different base models including Qwen2.5, Qwen2.5-Math, and DeepSeek-Math. Notably, our model Qwen2.5-Math-CFT only requires 1 hour of training on 8 x H100 over the 50K examples, yet matches or outperforms strong competitors like Qwen2.5-Math-Instruct on most benchmarks, which use over 2M samples. Moreover, it matches the performance of SimpleRL, which is a DeepSeek-r1 replication trained with 140 x more compute. Experiments on IF_Eval and MT-Bench further demonstrate that CFT can significantly enhance the model's general generation and instruction-following capabilities, outperforming the Qwen2.5-Math-Instruct by a large margin. Ablation studies show that CFT is robust to noisy response sources and teacher critique models. These findings highlight that CFT offers a more effective alternative to advance the reasoning of language models.

1 Introduction

The paper proposes Critique Fine-Tuning (CFT), which trains models to analyze noisy responses rather than imitate annotated answers. Across reasoning and instruction-following evaluations, CFT improves performance and remains effective across datasets, teachers, and model bases.

  • Critique Fine-Tuning: CFT trains models to critique noisy query-response pairs, identifying flaws, suggesting improvements, and verifying correctness instead of imitating answers.The objective maximizes P(c|[x; y]) for an annotated critique c.
  • Experimental validation: CFT experiments use multiple critique datasets and 7B base models including DeepSeekMath-base, Qwen2.5, and Qwen2.5-Math.The evaluation compares CFT with strong SFT-trained variants.
  • Experimental validation: CFT outperforms strong competitors on broader STEM benchmarks while using 50K examples, compared with competitors trained on over 2M examples.The comparison includes AceMath and Qwen2.5-Math-Instruct.
  • General instruction following: CFT outperforms its SFT version and official Qwen2.5-Math-Instruct on MT-Bench and IF Eval.These evaluations test general instruction-following abilities.
  • Ablation findings: Controlling token length, shorter critique examples outperform standard SFT at 55.2% versus 50.4%.This comparison is presented as evidence that the gains stem from critique-based training rather than longer sequences.
  • Overall implication: CFT shows strong reasoning and general instruction-following improvements, supporting its potential as an alternative to SFT.The passage describes this as a potential replacement, not an established requirement for all language-model training.

2 Method & Dataset

The method constructs compact critique datasets from noisy instructional responses and trains models to generate critiques conditioned on questions and responses. These datasets span broad topics and support CFT’s data-efficient reasoning gains.

  • Dataset construction: WebInstruct is collected from educational and quiz resources and synthetically processed with language models for solution quality and format consistency.Its responses may remain noisy because verification and quality control are limited.
  • Dataset construction: WebInstruct covers mathematics, physics, chemistry, business, humanities, and other topics, providing broader coverage than primarily competition-derived datasets.Mathematics accounts for 65% of the dataset.
  • Dataset variants: The curated WebInstruct subsets include 50K direct SFT examples, verified SFT data, GPT-4o-generated answers, and GPT-4o critiques of original responses.The CFT subset uses the original responses and adds detailed critiques.
  • Dataset comparison: CFT datasets cover broader topics while being significantly smaller than existing SFT datasets, highlighting their training efficiency.Table 1 compares the CFT and SFT dataset designs.
  • Dataset generalization: Critiques are also synthesized for randomly sampled 50K examples from MetaMathQA and NuminaMath to test generalizability across datasets.GPT-4o critiques the original responses before CFT training.
  • Training objective: The training objective concatenates question x and noisy response y as input and optimizes the model to generate critique c.The objective is argmaxθ log P(c|[x; y]; θ).

3 Experiments

The experiments compare Critique Fine-Tuning with supervised fine-tuning, existing reasoning models, reinforcement-learning methods, and ablated training choices. Across these evaluations, CFT achieves strong reasoning and instruction-following performance with comparatively small data and compute budgets.

  • 3.2 Main Results (CFT vs. SFT): CFT consistently outperforms SFT across three 7B base models, with gains of 3.5%, 10.4%, and 6.7% over reported SFT baselines.The improvements are reported for DeepSeek-Math-7B, Qwen2.5-7B, and Qwen2.5-Math-7B, respectively.
  • 3.3 More Results (CFT Models vs. Existing Models): 48.1% average performance is achieved by Qwen2.5-Math-7B-CFT among 7B-scale models using 50K training samples.The model outperforms several specialized math models while using substantially less training data than models trained on hundreds of thousands or millions of samples.
  • 3.4 Comparison with RL-based Method: CFT matches SimpleRL overall and significantly outperforms it on AMC23 and Minerva-Math, despite using 8×H100 for 1 hour instead of substantially more compute.The AIME24 comparison is especially sensitive to randomness because its test set contains only 30 questions.
  • 3.5 Ablation Studies: 57.1% average performance makes WebInstruct the strongest CFT dataset, although SFT performs better on MetaMathQA and NuminaMath.The results suggest that CFT effectiveness is not determined solely by the quality of solution data.
  • 3.5 Ablation Studies: 52.0% versus 40.4% average performance shows that CFT remains stronger than SFT-verified with GPT-4o-mini critiques, while GPT-4o-1120 reaches 57.1%.The teacher-model comparison supports improvements with both critique models, with stronger results from GPT-4o-1120.
  • 3.5 Ablation Studies: 6.49 on MT-Bench exceeds 4.79 for the base model and 5.49 for Qwen2.5-Math-7B-Instruct, showing broader instruction-following gains.The paper also evaluates IF Eval and reports improvement over the instruction-tuned comparison model.

4 Limitations

CFT’s effectiveness depends on critique quality, and self-critique mechanisms underperformed direct inference because of inconsistent standards and instability.

  • Critique Data Quality: Roughly 20% of 50 manually inspected GPT-4o-1120 critiques contained inaccuracies, including misjudged steps, missed errors, or imprecise explanations.The authors identify higher-quality critique data, automated verification, and human-verified datasets as future directions.
  • Self-Critique: Self-critique consistently underperformed direct inference because models overlooked genuine errors or incorrectly flagged correct solutions.Higher sampling temperatures introduced instability during iterative refinement, so the final implementation uses direct inference without self-critique.

5 Related Work

Related work spans instruction tuning, mathematics-focused datasets, critique-oriented research, self-correction, and critique models. CFT differs by making critique feedback the training objective while using direct response generation at inference.

  • Instruction Tuning: Instruction-tuning datasets draw on human annotation or model synthesis to align pretrained language models with human expectations.Examples include FLAN, T0, SuperNI, Self-Instruct, WizardLM, and WildChat.
  • Math Instruction Tuning: Math instruction methods increasingly scale datasets to millions of examples, including MAmmoTH2, Open-MathInstruct, AceMath, and Qwen2.5-Math-Instruct.
  • Self-Correction: Self-correction research uses model-generated feedback, but later work reports that self-correction in reasoning is not quite reliable.
  • Critique Models: Critique models provide feedback to another model and include outcome-based and process-based reward models for improving reasoning.
  • CFT Distinction: CFT uses critique feedback explicitly as a training objective, then generates responses directly without explicit critique or iterative refinement at inference.

6 Conclusion

The paper introduces CFT, which trains models to critique and analyze responses rather than imitate them. Across experiments, CFT improves mathematical reasoning, matches resource-intensive reinforcement-learning performance with fewer resources, and generalizes to instruction following and broader STEM domains.

  • Main Findings: CFT consistently outperforms SFT by 4–10% on mathematical reasoning benchmarks.
  • Efficiency: CFT achieves comparable performance to resource-intensive RL methods using 50K versus 2M+ training examples and 8 H100 GPU-hours.
  • Generalization: CFT-trained models generalize effectively to broader STEM domains and exhibit strong instruction-following capabilities without traditional SFT or RL.
  • Future Directions: The paper identifies improving critique-data quality and coverage, enabling self-critique, combining CFT with RL, and extending the approach as future directions.

A.1 GPT-4o Case Study for Critique Data Generation

The case study illustrates how GPT-4o generates structured critiques by checking a student’s solution, identifying errors, and issuing a final correctness judgment. In the mitosis example, the critique catches the confusion between mitosis and meiosis while preserving the answer’s correct components.

  • Critique Generation: GPT-4o is prompted to judge whether a student’s answer is correct and conclude with a structured right-or-wrong label.
  • Student Solution: The student solution correctly describes mitosis in growth, tissue repair, cell replacement, and some asexual reproduction, but incorrectly assigns gamete production to mitosis.
  • Generated Critique: The critique identifies meiosis, not mitosis, as responsible for gamete production and concludes that the answer is wrong.
  • Case Study Implication: The example demonstrates that the critique both identifies the conceptual error and acknowledges the accurate parts of the response.

A.2 GPT-4o Case Study for WebInstruct-GPT-4o-SFT dataset Generation

The case study illustrates how GPT-4o generates structured, step-by-step answers for WebInstruct and how those outputs support the SFT dataset.

  • Prompt and response generation: The example prompt asks GPT-4o to answer a science or mathematics question clearly and conclude with a specified answer format.The displayed example asks for the derivative of a quadratic function using the Power Rule.
  • Step-by-step solution: The worked example concludes that the derivative is f′(x) = 2ax + b.The response combines the three differentiated terms into the final expression.
  • Dataset construction: The authors characterize these responses as clear, structured explanations with mathematical notation, step-by-step reasoning, and concise final answers.They state that such responses formed the foundation of the WebInstruct-GPT-4o-SFT dataset.

A.3 Case Study: Noisy Critique Data

The section introduces examples of noisy critique data and states that their effects on model performance are analyzed.

  • Noisy critique examples: The authors present examples of noisy critique data.These examples are used as material for examining critique quality and its consequences.
  • Impact analysis: The section analyzes how noisy critique data affects model performance.The supplied passage frames the examples as part of an impact analysis.
  • Noisy critique examples: The examples are presented as case studies rather than as a general description of critique generation.The passage explicitly introduces them for analysis of their impact.

A.3.1 Case 1: Incorrect Critique of Valid Solution

The case study examines errors in self-critique, contrasting flawed evaluations with the direct-inference process and identifying instability, regeneration, and computational costs.

  • Case-study failures: The case examples show that self-critique can incorrectly judge valid reasoning as flawed or accept reasoning containing errors.The supplied examples include false error detection and missed calculation errors.
  • Self-critique setup: A self-critique process evaluates generated solutions and can regenerate them through single-pass or two-stage procedures.The two-stage method separates solution generation from critique and permits up to eight attempts.
  • Observed issues: Self-critique applies inconsistent evaluation standards, producing both missed errors and false positives on valid solutions.The authors identify over-lenient evaluation and incorrect error flags as recurring patterns.
  • Observed issues: 77.2% falls to 73.5% on MATH as temperature rises from 0.1 to 0.6 for the single-pass method.The passage reports similar temperature trends on Minerva-Math.
  • Observed issues: Iterative self-critique also struggles to fix identified errors without introducing new ones and increases inference time and computational cost.The authors report diminishing performance returns from the iterative process.
  • Comparative performance: The two-stage method performs slightly better than single-pass self-critique but remains worse than direct inference.The authors attribute the gap to the added complexity of applying critique during inference.
Loading 2501.17703v4…