Source-linked AI summary
Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, Bo Zheng
TL;DR
The paper addresses the limited evaluation of long CoT quality and models’ ability to detect errors within such reasoning. It introduces DeltaBench, analyzes long CoTs across tasks, and evaluates critic models and PRMs. The benchmark reveals fundamental errors, ineffective reflection, redundancy, and weak critique performance, with no demonstrated critique advantage for o1-like models.
Problem
Systematic evaluation of long CoT quality and existing LLMs’ ability to critique long CoT reasoning had not been explored.
Method
DeltaBench collects and annotates long CoTs from multiple o1-like models across Math, Programming, PCB, and General Reasoning, then evaluates critic models and PRMs for error detection.
Results
Existing critique models have limited error-detection ability, while long CoTs contain fundamental errors, 67.8% useless reflections, and 27% redundant reasoning sections; o1-like models show no critique advantage over non-o1-like models.
Takeaways & Limitations
DeltaBench provides a benchmark for analyzing long CoT limitations and measuring critique quality in existing LLMs, including critic models and PRMs.
Takeaways & Limitations
Dataset construction and annotation are costly, human annotation may introduce subjective bias, and the static benchmark may not capture real-time advances.
Abstract
from arXiv · showhide
Recently, o1-like models have drawn significant attention, where these models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). In this paper, to understand the qualities of these long CoTs and measure the critique abilities of existing LLMs on these long CoTs, we introduce the DeltaBench, including the generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the ability to detect errors in long CoT reasoning. Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models. Then, we conduct extensive evaluations of existing process reward models (PRMs) and critic models to detect the errors of each annotated process, which aims to investigate the boundaries and limitations of existing PRMs and critic models. Finally, we hope that DeltaBench could guide developers to better understand the long CoT reasoning abilities of their models.
1 Introduction
The paper introduces DeltaBench to analyze long CoT quality across reasoning tasks and evaluate how well critic models and PRMs detect errors. Its analyses find substantial redundancy and fundamental errors, while existing critique abilities remain limited, including for o1-like models.
- Benchmark and analysis: DeltaBench collects long CoTs from QwQ, DeepSeek-R1, and Gemini 2.0 Flash Thinking across Math, Programming, PCB, and General Reasoning.Each CoT is divided into independent subtasks and annotated for reasoning usefulness, correctness, and reflection.
- Long CoT findings: Approximately 25% of QwQ-32B-Preview errors and 23% of Gemini 2.0 Flash Thinking errors are fundamental calculation, syntax, or format errors.
- Long CoT findings: Approximately 67.8% of reflections in collected long CoT responses are useless because they do not lead to the right answer.The paper defines effective reflection as reflection that leads to the right answer.
- Long CoT findings: On average, 27% of reasoning sections in collected long CoT responses are redundant.
- Critique evaluation: GPT-4-turbo-128k is the top-performing DeltaBench model but achieves an F1-score of only 40.8% for identifying errors in long CoT reasoning.
- Critique evaluation: o1-like models show no critique advantage over non-o1-like models, and DeepSeek-R1 has a 36% reduction in self-critique performance versus critiquing other o1-like models.
2 DeltaBench
DeltaBench constructs a section-level benchmark from diverse long CoT responses, with human annotations covering reasoning quality, correctness, strategy shifts, and reflections. It evaluates error detection using balanced data, PRM outlier detection, and macro-F1 for imbalanced error sections.
- Dataset Construction: DeltaBench contains 1,236 samples spanning five major domains and 48 subcategories, with balanced question types, difficulties, and long-CoT characteristics.The benchmark includes Math, Programming, PCB, and General Reasoning among its domains and balances long-CoT lengths and section counts.
- Correctness Assessment: Each long CoT is divided into independent subtasks and annotated for strategy shift, reasoning usefulness, reasoning correctness, and reflection efficiency.Correctness annotations record the first error step, explanation, and correction when an error is present.
- Dataset Construction: Long CoTs are generated from QwQ-32B-Preview, DeepSeek-R1, and Gemini 2.0 Flash Thinking using random sampling to capture diverse reasoning processes and potential errors.The generated responses cover math, programming, physics, chemistry, biology, and general reasoning tasks.
- Evaluation Metrics: PRMs use Z-Score outlier detection with threshold t = µ −σ, while critic models identify all erroneous sections and are evaluated with macro-F1.The threshold avoids relying on step-level dataset distributions, and macro-F1 addresses the smaller proportion of error sections.
- Correctness Assessment: Section-level evaluation targets all errors in complex reasoning processes rather than only the first error or sample-level correctness.The section granularity is designed to make long CoT evaluation more fine-grained and suitable for human annotation.
3 Analysis
DeltaBench analysis finds persistent reasoning and detail errors in long CoTs, low effective reflection and substantial redundancy. Critic and PRM performance is limited, especially for longer contexts and strategy errors.
- Error Analysis: Reasoning errors dominate Math, Programming, PCB, and General Reasoning, with General Reasoning reaching 43%.
- Error Analysis: DeepSeek-R1 has 22.7% reasoning errors but only 3.1% calculation errors, whereas QwQ-32B-Preview has 17.9% calculation errors.
- Error Analysis: Current models’ primary bottleneck remains reasoning ability, while calculation and formal mistakes also contribute substantially.
- Reflection Analysis: Only 30%–40% of reflections are valid overall, although DeepSeek-R1 performs best and math has higher effective-reflection rates.
- Reasoning Efficiency: Only 73% of collected long-CoT reasoning is useful on average, leaving 27% redundant; DeepSeek-R1 is most efficient and QwQ-32B-Preview reaches 70% usefulness.
- Critique Evaluation: Models recognize calculation errors best but strategy errors worst, indicating limited generalization across error types.
- Critique Evaluation: HitRate@k results align with the main evaluation, with Qwen2.5-Math-PRM-7B best identifying the first incorrect section.
4 Related Works
Related work situates DeltaBench among research on test-time scaling, process reward modeling, and LLM critique. These lines of work address reasoning improvement, process evaluation, and critique quality.
- Test-time Scaling: Test-time scaling improves reasoning by increasing generated tokens, with tree-search methods and o1-like models representing major approaches.
- Process Reward Modeling: Process reward models evaluate intermediate reasoning steps and complement outcome-level reward models for multi-step reasoning.
- Process Reward Modeling: Human-annotated process-level datasets provide resources for studying long chain-of-thought generation.
- LLM Critic: LLM-critique research evaluates verdict accuracy, feedback quality, correction quality, and self-critique for reasoning improvement.
5 Conclusion
The paper presents DeltaBench as a comprehensive benchmark for studying long-CoT limitations and evaluating LLM critique quality. It also uses the benchmark to analyze errors and guide future improvement of reasoning models.
- DeltaBench evaluates limitations of o1-like models and the critique quality of existing LLMs using generated long CoTs.
- The benchmark supports specific error analysis of o1-like models and detailed analysis of critic models and PRMs.
- The authors intend DeltaBench to guide improvements to reasoning models.
6 Limitations
DeltaBench is constrained by costly construction and annotation, possible subjectivity in human labels, and its static nature. The authors identify scaling, bias, and update coverage as future-work concerns.
- Dataset construction and annotation are costly, limiting scalability to larger data volumes.
- Despite rigorous human annotation, the benchmark may still contain subjective biases.
- As a static benchmark, DeltaBench may not fully capture real-time advances in models.
A.1 Data Sources
DeltaBench combines long CoTs from multiple o1-like models and domains with section-level annotations and validation procedures. The resulting data show slightly stronger generated-CoT accuracy in mathematics and programming than in general reasoning and PCB.
- Annotation procedure: Annotation uses three initial annotators, two junior reviewers, and five spot-checkers, with training and consistency checks supporting quality control.
- Domain-specific validation: Math correctness combines rule-based numerical checks with LLM evaluation, while programming uses sandbox execution, test cases, and LLM judgment.
- Annotation procedure: Sections exceeding 50 steps are iteratively divided to reduce annotation complexity and make long-CoT review more comprehensible.
- Dataset construction: DeltaBench covers long CoTs from QwQ-32B-Preview, DeepSeek-R1, and Gemini 2.0 Flash Thinking across Math, Programming, PCB, and General Reasoning.
- Generated-CoT quality: o1-like models enhanced with reinforcement learning perform slightly better on generated long-CoT accuracy in math and programming than in general reasoning and PCB.
A.8 Analysis of Other Evaluation Metrics
The evaluation compares F1-Score, First Error Accuracy, and Any Error Accuracy, finding aligned model rankings across metrics. GPT-4-turbo performs best, while quantile thresholding can overestimate PRM performance.
- First Error Accuracy measures identifying the first error, whereas Any Error Accuracy measures detecting any error in the sequence.
- GPT-4-turbo achieves 69% Any Error Accuracy, consistently outperforming other compared models across the reported metrics.
- Model rankings for First Error Accuracy and Any Error Accuracy closely align with F1-Score, supporting F1-Score as a comprehensive indicator.
- Using the ascending 5% quantile of all DeltaBench rewards as a PRM threshold overestimates performance relative to outlier detection.
B Error Classification
Human annotations classify errors in model answers into instruction-following failures, overconfidence in correctness, and other recognition errors.
- Human annotations classify underperforming-model errors into three categories: instruction failures, overconfidence in correctness, and other errors.
C Analysis of Underperforming Model
Underperforming models exhibit instruction-following deficiencies and may answer questions instead of critiquing response correctness. Error patterns differ across model families and include false negatives and error misidentification.
- Error patterns: The error taxonomy includes instruction failures, overconfidence that no error exists, false negatives, and misidentification of error type or location.
- Underperforming-model analysis: DeepSeek-R1-Distill models often answer questions directly instead of critically evaluating response correctness, indicating instruction-following deficiencies.
- Underperforming-model analysis: DeepSeek-R1 and o1-preview-0912 also show notable instruction-following challenges, though less than GPT-4o.