Source-linked AI summary
An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
Mingzhong Sun, Teresa Yeo, Armando Solar-Lezama, Tan Zhi-Xuan
TL;DR
The paper asks whether LRMs can evaluate reasoning independently of producing correct answers. Using the VAIR dataset and interpretability analyses, it finds a substantial production-evaluation gap driven by answer confirmation bias, while identifying limits in the evidence and scope of its conclusions.
Problem
The paper investigates whether LRMs can evaluate reasoning independently of producing answers, since production and evaluation can support each other and reasoning-production training creates an important confound.
Method
The paper constructs VAIR math question-solution pairs with invalid reasoning but valid answers, then combines behavioral evaluation with CoT analysis, linear probes, and causal patching.
Results
Frontier LRMs achieve at least 94.7% solution accuracy but fall to 47.9% evaluation accuracy on VAIR, while causal and representational analyses implicate answer confirmation bias.
Takeaways & Limitations
The findings indicate that reasoning production and evaluation are distinct capabilities, and that outcome-focused training may leave LRMs unable to robustly verify individual reasoning steps.
Takeaways & Limitations
The paper does not directly test whether LRM training objectives cause the failures, and its evaluation is limited to mathematical reasoning tasks.
Abstract
from arXiv · showhide
Studies of human reasoning have shown that people are typically stronger at evaluating reasoning than producing it from scratch. In contrast, large reasoning models (LRMs) are trained to excel at producing long chains of reasoning to solve complex problems. How then do LRMs perform at evaluating reasons? We investigate this with the Valid-Answer-Invalid-Reasoning (VAIR) dataset: math problems and solutions with trivial reasoning flaws but valid answers, designed to isolate reasoning evaluation from the confound of reasoning production. Unlike humans, who we find are only 6% worse at grading than solving such problems, we find a substantial production-evaluation gap in LRMs: frontier models score as low as 48% when evaluating VAIR solutions, despite near-perfect solution production. Why this enigma? Through chain-of-thought (CoT) analysis, we find evidence of an answer confirmation bias: LRMs often produce then check for the correct answer instead of carefully verifying each step, fabricating rationalizations even when noticing anomalous reasoning. Linear probes corroborate this, showing that while LRM activations encode some representation of valid reasoning, they fail to robustly represent VAIR solutions as invalid. Causal patching of the final answer's representations causes LRM verdicts and activations to flip, demonstrating that answer validity is responsible for models' confirmation biases. These findings indicate an outstanding limitation in dominant approaches to reasoning training, which incentivize LRMs to produce and confirm reasoning towards correct answers, but not to robustly evaluate the underlying reasons.
1 Introduction
The paper asks whether LRMs can evaluate reasoning as well as they produce it, and introduces VAIR to separate reasoning validity from answer correctness. It finds a large LRM production-evaluation gap and attributes it to answer confirmation bias linked to outcome-focused training.
- VAIR perturbs math solutions to add trivial reasoning flaws while preserving valid answers, isolating reasoning evaluation from answer validity.The dataset targets flaws such as missing premises and circular reasoning.
- LRMs struggle to evaluate VAIR solutions despite strong performance on direct problem solving and matched-validity evaluations.Frontier models incorrectly score invalid solutions as flawless up to 50% of the time, while achieving at least 90% accuracy on direct solving or matched controls.
- CoT analyses indicate that LRMs often reproduce the correct answer first, overlook flaws, or fabricate justifications for invalid reasoning.
- Linear probes and causal patching link the bias to internal representations that underrepresent invalid reasoning and shift with final-answer validity.Patching final-answer representations flips both model verdicts and activations.
2 Evaluating the Evaluation of Reasoning
The study evaluates reasoning production and grading with VAIR and matched controls across LRMs and humans. LRMs solve problems accurately but fail sharply when valid answers accompany flawed reasoning, whereas humans show a much smaller gap.
- 2.1 Dataset Construction: VAIR contains invalid reasoning with valid answers, while VAVR and IAIR provide controls in which reasoning and answer validity match.The dataset is built by perturbing seed problems and gold solutions with distinct reasoning flaws.
- 2.2 Experimental Design: The evaluation task asks participants to grade solutions from 3 for flawless reasoning to 0 for completely incorrect solutions.Production instead requires solving unperturbed problems and generating a step-by-step solution with a final answer.
- 2.3 The Production-Evaluation Gap in LRMs: 94.7% or above production accuracy contrasts with VAIR evaluation accuracy as low as 47.9% for GPT 5.4 and 52.5% for GPT 5.Control evaluation remains high at ≥91.9% on VAVR and ≥95.8% on IAIR, where answer validity tracks reasoning validity.
- 2.3 The Production-Evaluation Gap in LRMs: LRMs detect Circular Reasoning relatively well but struggle especially with Shuffled Reasoning and Missing Reasoning, with lower performance on harder MATH subsets than GSM8K.
- 2.3 The Production-Evaluation Gap in LRMs: Process reward models exhibit similar VAIR evaluation failures despite being trained explicitly to assess step-by-step validity.
- 2.4 The Reduction of the Gap in Human Reasoners: Humans achieve 80.8% on solving and 74.5% on VAIR evaluation, a maximum production-evaluation gap of 6.3%.Their VAIR performance is close to chance on Missing Reasoning cases.
- 2.4 The Reduction of the Gap in Human Reasoners: Humans spend less time grading than solving, whereas LRMs use more chain-of-thought tokens evaluating VAIR solutions than solving problems.The effort measures are human response time and model chain-of-thought token count.
3 Answer Confirmation Bias Explains the Production-Evaluation Gap
LRMs exhibit answer confirmation bias across behavioral, representational, and causal analyses: valid answers steer evaluation toward endorsement despite flawed reasoning. Causal patching reverses these effects, implicating answer-associated activations in the production-evaluation gap.
- LRMs’ answer confirmation bias distorts reasoning evaluation behaviorally and representationally, with valid answers steering judgments toward validity.The paper examines verbalized reasoning, internal activations, and causal effects of answer-associated representations.
- On VAIR solutions, evaluators often independently solve problems to confirm valid answers, then overlook flaws through Blind Endorsement or Forced Rationalization.These justification behaviors target flawed steps after the evaluator has reproduced the correct answer.
- CoT analysis is suggestive but not conclusive because verbalized reasoning may fail to reflect or cause the model’s underlying verdict.This motivates representation-level and causal analyses.
- 89% probe accuracy on held-out concordant cases falls below chance on Group B VAIR cases, while an all-group oracle probe remains near chance.These results indicate that VAIR activations are difficult to linearly separate from valid reasoning even with training exposure.
- Group B validity representations rise from P ≈0.5 to P > 0.8 before the verdict, converging with valid cases as answer validity overrides reasoning validity.The trajectory is consistent with the observed endorsement and rationalization behaviors.
- Causal patching invalid-answer activations produces verdict flip rates exceeding 50% across models and shifts evaluation toward step tracing and strict rejection.At one peak probe layer, GPT-oss-20B still shows a 14.2% flip rate; across-layer patching also sharply reduces Blind Endorsement.
4 Discussion
The paper finds that LRMs can produce reasoning effectively yet fail to evaluate flawed reasoning, with answer confirmation bias appearing behaviorally and representationally. These findings expose limits of outcome-focused reasoning training and motivate improved evaluation-oriented training, while leaving important scope questions open.
- LRMs exhibit answer confirmation biases at both behavioral and representational levels when evaluating reasoning with valid answers.
- Frontier LRMs can solve open research-mathematics problems yet fail to robustly evaluate grade-school mathematical reasoning.
- The production-evaluation gap resembles failures of epistemic vigilance found in sycophantic endorsement and multi-agent debate.
- The paper does not directly test whether LRM training objectives cause the observed failures, and its evaluation covers only mathematical reasoning.
- Mechanistic analyses were restricted to open-weight models under 20B parameters, leaving scaling to larger closed models unresolved.
A.1 Dataset Construction
The datasets are built by perturbing valid mathematical solutions to create controlled reasoning failures while preserving or altering answer correctness. Construction uses standardized prompting, reviewed examples, and explicit grading-task materials.
- VAIR starts from valid seed problems and injects four reasoning-failure categories while preserving the correct final answer.
- The four VAIR categories are Missing Premises, Missing Reasoning, Shuffled Reasoning, and Circular Reasoning.
- Circular-reasoning prompts require logically empty or tautological steps that still produce the correct numerical answer.
- VAVR pairs extracted seed problems with ground-truth solutions, producing 861 rows.
- The IAIR dataset uses generated flawed reasoning with incorrect answers and contains 861 manually reviewed rows.
- The grading task instructs models to assess reasoning step by step using a 0–3 rubric with examples for each score.
Extended Statistical Analysis
The extended analysis compares human accuracy and response times across solving and reasoning-evaluation conditions. It reports statistical testing procedures and presents condition-level results with uncertainty and significance annotations.
- The analysis uses Fisher’s exact test for accuracy and the Mann–Whitney U test for response time.
- Human performance is compared across SOLVING, VAVR, VAIR, and IAIR conditions using accuracy and response time.
- Accuracy is reported as mean percentage with binomial standard-error bars, while response time is reported in seconds with standard-error bars.
- Pairwise significance brackets appear only for statistically significant comparisons, using ∗p < 0.05, ∗∗p < 0.01, and ∗∗∗p < 0.001.
A.4 Reasoning Evaluation Task for Process Reward Models (PRMs)
The PRM evaluates reasoning token by token, but its performance declines sharply when flawed reasoning ends in a valid answer. The failure is especially severe for missing inferential steps and may reflect insufficient context use or answer-biased training labels.
- The PRM outputs token-by-token probability scores for the mathematical validity of each concluded reasoning step.
- Qwen2.5-Math-PRM-7B scores 93.8% on IAIR, 79.3% on VAVR, and 67.8% on VAIR.
- VAIR accuracy varies by error type: Circular Reasoning reaches 91.0%, whereas Missing Reasoning falls to 49.1%.
- Running-minimum score trajectories show early error detection on IAIR, while VAIR errors are not robustly identified.
- Step-level scoring may underweight prior context, allowing internally valid but contextually invalid steps to receive positive evaluations.
- Monte Carlo labels may reinforce answer confirmation because later rollouts can self-correct earlier invalid or missing steps.
B.1 Open-Source Model Behavioral Evaluation
The evaluation tested three open-source models on VAIR and VAVR using a standardized prompt and 0–3 grading rubric, followed by simplified binary grading trials. All three models still exhibited a production-evaluation gap.
- Three open-source models—Qwen3-0.6B, Qwen3-4B, and GPT-oss-20B—were evaluated on the VAIR and VAVR datasets.
- The initial evaluations used a standardized system prompt and the same 0–3 grading rubric as the LRM evaluations.
- The subsequent trials used a simplified binary grading scheme with greedy decoding at T = 0.
- All three models still suffered from the production-evaluation gap.Detailed model metrics are presented in Figures B5, B6, and B7.
B.2 Chain of Thought Analysis
The analysis classified evaluator chain-of-thought along workflow and justification-pathology dimensions. Examples show failures involving independent re-solving with blind endorsement and step tracing with forced rationalization, while causal patching could restore strict rejection in some cases.
- Evaluator chain-of-thought was classified across two dimensions: evaluation workflow and justification pathology.
- Evaluation Workflow: Independent_Solving describes re-solving the problem, matching the final answer, and glossing over the student’s reasoning steps.
- Evaluation Workflow: Step_Tracing describes verifying the student’s solution line-by-line without first solving the whole problem independently.
- Justification Pathology: The pathology labels distinguish Blind_Endorsement, Forced_Rationalization, and Strict_Rejection according to how evaluators handle logical flaws.
- Examples: Table B2 illustrates Independent Solving plus Blind Endorsement and Step Tracing plus Forced Rationalization as two workflow–pathology failure modes.
- Causal Patching: After causal patching, GPT-oss-20B sometimes restored epistemic vigilance and produced Strict Rejection, while other cases remained unresolved.
B.3 Linear Probe Analysis
The appendix reports static probe results and static probe ablation studies as part of the linear probe analysis.
- Figure B8 presents the static probe results.
- Together, Figures B8 and B9 document the reported static-probe analysis and its ablations.
- Figure B9 presents static probe ablation studies.
B.4 Dynamic Probe Analysis
The appendix reports dynamic probe accuracy across layers as part of the dynamic probe analysis.
- Figure B10 presents dynamic probe accuracy across layers.
- The figure organizes the dynamic probe analysis by model-layer accuracy.
- Layer-wise accuracy is the reported comparison for the dynamic probe results.
B.5 Causal Patching Analysis
This section presents causal patching analyses across all layers of Qwen-3-0.6B, Qwen-3-4B, and GPT-oss-20B, including flip-rate results for GPT-oss-20B.
- Causal patching results are reported across all layers of Qwen-3-0.6B.
- Causal patching results are reported across all layers of Qwen-3-4B.
- For GPT-oss-20B, the analysis reports causal patching at each layer alongside flip-rate results.