Source-linked AI summary
Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs
Deep Mehta
TL;DR
The paper asks whether inference scaling improves reasoning faithfulness, not merely answer accuracy. It studies four frontier models on GSM8K using statistical comparisons and an early-answering faithfulness probe, finding strongly model-dependent tradeoffs. The authors conclude that practitioners should evaluate self-consistency for their specific models because scaling can improve, reduce, or leave faithfulness largely unchanged.
Problem
The paper addresses the unstudied question of whether inference scaling improves reasoning faithfulness alongside accuracy.
Method
The study evaluates four frontier models on 100 GSM8K problems using statistical comparisons and the early-answering faithfulness probe.
Results
Inference scaling has model-dependent effects: GPT-5.2 gains accuracy with stable faithfulness, Claude Opus 4.5 loses accuracy while faithfulness rises sharply, and other models show modest or ceiling effects.
Takeaways & Limitations
Self-consistency is not universally beneficial, so teams should test specific models and weigh its accuracy-faithfulness tradeoffs before deployment.
Takeaways & Limitations
The study uses only the early-answering probe, while alternative probes may reveal different faithfulness patterns.
Abstract
from arXiv · showhide
Self-consistency has emerged as a popular technique for improving large language model accuracy on reasoning tasks. The approach is straightforward: generate multiple reasoning paths and select the most common answer through majority voting. While this reliably boosts accuracy, it remains unclear whether these gains reflect genuine improvements in reasoning quality. We investigate a fundamental question that has not been studied before: does inference scaling improve reasoning faithfulness? We conduct a comprehensive empirical study across four frontier models (GPT-5.2, Claude Opus 4.5, Gemini-3-flash-preview, and DeepSeek-v3.2) on 100 GSM8K mathematical reasoning problems. Our analysis employs bootstrap confidence intervals, McNemar's tests for paired comparisons, and Cohen's d effect sizes to quantify the effects rigorously. The results reveal striking differences across models that challenge common assumptions about self-consistency. GPT-5.2 shows the expected pattern: accuracy improves from 78% to 90% at N=5, with faithfulness remaining relatively stable (0.540 to 0.510). Claude Opus 4.5 tells a completely different story. Its accuracy actually drops from 78% to 74.3% while faithfulness jumps dramatically from 0.270 to 0.891 at N=5. DeepSeek-v3.2, already at 98% accuracy, shows ceiling effects with modest faithfulness gains (0.440 to 0.541). Gemini-3-flash improves from 81% to 86% accuracy with a slight faithfulness decrease (0.260 to 0.212). Problem difficulty analysis reveals that GPT-5.2 solves 82% of hard problems while breaking only 13% of easy ones. Claude, in contrast, breaks 23% of easy problems, explaining its accuracy decrease. These findings matter for practitioners: self-consistency is not universally beneficial, and teams should test their specific models before deployment. We release our code and provide practical recommendations for navigating these tradeoffs.
A Comprehensive Multi-Model Analysis of Self-Consistency Tradeoffs
The paper concerns chain-of-thought reasoning and self-consistency in large language models.
- Chain-of-thought reasoning is identified as a central topic alongside self-consistency and faithfulness.
1 Introduction
The paper asks whether self-consistency improves reasoning quality as well as accuracy, and studies this question across models, difficulty levels, and practical deployment concerns. It reports model-dependent effects and offers recommendations for practitioners.
- Motivation and Background: Self-consistency improves benchmark accuracy, but it remains unclear whether higher accuracy reflects better reasoning quality.The paper frames this as the question of whether models improve reasoning or merely aggregate answers more effectively.
- Motivation and Background: Unfaithful reasoning can produce correct answers with explanations that do not reflect the model’s actual computational process.The paper connects this concern to oversight, interpretability, debugging, and deployment decisions.
- Hypotheses: The paper finds strong support for a model-dependent relationship between inference scaling and faithfulness.
- Research Questions: The study tests whether inference scaling affects faithfulness across frontier models, how accuracy and faithfulness relate, and how effects vary with problem difficulty.
- Contributions: The paper contributes the first study of inference scaling and reasoning faithfulness, using four frontier models and rigorous statistical analysis.Its contributions also include model-dependent findings, difficulty analysis, practical recommendations, and released tools.
2 Related Work
Related work establishes the effectiveness of chain-of-thought and self-consistency while highlighting that reasoning faithfulness has received less attention. The paper bridges these previously separate literatures by studying faithfulness under inference scaling.
- Chain-of-Thought Reasoning: Chain-of-thought prompting elicits step-by-step reasoning and improves performance across mathematical, commonsense, and symbolic reasoning tasks.Zero-shot and least-to-most prompting are presented as related approaches for eliciting or structuring reasoning.
- Self-Consistency: Self-consistency samples multiple reasoning paths and selects the most frequent answer through majority voting to improve accuracy.
- Inference Scaling: Inference-scaling research studies how additional samples and compute affect performance, including compute allocation by problem difficulty.
- Reasoning Faithfulness: Faithfulness asks whether stated reasoning reflects the model’s actual computational process, a concern relevant to AI safety and interpretability.
- Faithfulness Measurement: Prior probes measure faithfulness through early answering, inserted mistakes, paraphrasing, and filler-token replacements.
- Prior Findings: Prior research found varying faithfulness across models and tasks, while showing that biased prompts can alter outputs without changing stated reasoning.These findings support the possibility that explanations can function as post-hoc rationalizations.
- Gap in the Literature: Self-consistency studies have focused on accuracy, whereas faithfulness studies have usually examined single-sample settings.This paper presents the first empirical study of faithfulness under inference scaling.
3 Methodology
The methodology evaluates inference scaling across four models and 100 GSM8K problems, measuring accuracy and faithfulness with paired statistical analyses. Faithfulness is operationalized through early answering, whose limitations are explicitly acknowledged.
- Experimental Design: The study compares faithfulness across scaling conditions, applies confidence intervals and statistical tests, and analyzes model and difficulty differences.
- Dataset Selection: The dataset contains 100 randomly sampled GSM8K test problems selected for statistical power and efficient multi-model experimentation.GSM8K provides clear ground truth, multi-step reasoning, benchmark comparability, and a range of difficulties.
- Model Selection and Scaling: The experiment evaluates four diverse frontier models and generates reasoning paths at multiple sample sizes, including N=1 as the single-sample baseline.
- Self-Consistency: For N>1, majority voting selects the answer receiving the most votes, with ties resolved by choosing the first answer encountered.
- Faithfulness Measurement: Faithfulness is measured with early answering: matching answers with and without reasoning score 0, while changed answers score 1.The resulting score is the proportion of paths in which reasoning changed the model’s answer.
- Limitations of the Probe: The early-answering probe tests necessity rather than causality, so a path can score as faithful while containing logical errors.
- Statistical Analysis: Accuracy comparisons use paired McNemar tests, while faithfulness comparisons use paired t-tests and Cohen’s d effect sizes.Accuracy is paired and binary; faithfulness is treated as continuous.
- Problem Difficulty Analysis: Difficulty analysis tracks hard problems solved and easy problems broken at higher sample sizes to distinguish benefits from costs of scaling.
4 Results
Inference scaling produces strongly model-dependent accuracy and faithfulness outcomes: GPT-5.2 improves accuracy, Claude’s accuracy falls as faithfulness rises, DeepSeek encounters ceiling effects, and Gemini changes modestly.
- Model comparisons: GPT-5.2 accuracy rises from 78% at N=1 to 90% at N=5, while faithfulness decreases slightly from 0.540 to 0.499.The N=5 accuracy gain is statistically significant, but faithfulness changes are not significant.
- Model comparisons: Claude Opus 4.5 accuracy decreases from 78% to 74.3%, while faithfulness rises from 0.270 to 0.891 at N=5 and then drops to 0.661 at N=20.The faithfulness increases are highly significant, with effect sizes d = 2.73 at N=5 and d = 1.82 at N=20.
- Model comparisons: DeepSeek-v3.2 remains at 98–99% accuracy because of ceiling effects, while faithfulness increases modestly from 0.440 to 0.541.The N=20 faithfulness increase is significant with medium effect size d = 0.50.
- Model comparisons: Gemini-3-flash improves accuracy from 81% to 86% at N=5 while faithfulness decreases from 0.260 to 0.212.Neither change is statistically significant.
- Problem difficulty: GPT-5.2 solves 82% of hard problems while breaking 13% of easy problems, whereas Claude breaks 23% of easy problems and solves 64% of hard problems.These differing difficulty-level dynamics explain why aggregate accuracy improves for GPT-5.2 but decreases for Claude.
- Scaling efficiency: N=5 is Pareto-optimal for GPT-5.2, delivering the best accuracy gain per unit cost at 0.024, while Claude has negative returns at any scaling level.Beyond N=5, GPT-5.2’s marginal returns decrease sharply.
5 Discussion
The discussion interprets inference scaling as a model-dependent tradeoff rather than a universal improvement: faithfulness and accuracy can move together, independently, or in opposite directions.
- Model-dependent mechanisms: GPT-5.2 gains 12% accuracy at N=5 while faithfulness changes from 0.540 to 0.510, with hard-problem solving driving gains and few easy problems broken.The authors suggest that confident but potentially unfaithful reasoning may be amplified by voting.
- Model-dependent mechanisms: Claude’s accuracy decreases 3.7% while faithfulness increases from 0.270 to 0.891 at N=5, and 23% of easy problems are broken.The paper hypothesizes that explicit multiple-path reasoning causes Claude to second-guess accurate initial intuitions.
- Model-dependent mechanisms: Claude’s faithfulness falls from 0.891 at N=5 to 0.661 at N=20, suggesting that additional sampling may restore some answers matching its early baseline.The authors state that single-sample chain-of-thought may therefore be preferable for Claude.
- Model-dependent mechanisms: DeepSeek’s 98% baseline accuracy leaves minimal room for improvement, but faithfulness increases from 0.440 to 0.541 at N=20.The increase is significant with p = 0.018 and d = 0.50.
- Model-dependent mechanisms: Gemini improves accuracy from 81% to 86% while faithfulness decreases from 0.260 to 0.212, with neither change statistically significant.The discussion characterizes Gemini’s behavior as relatively stable across sampling conditions.
- Cross-model interpretation: The findings support a model-dependent account of self-consistency and challenge the assumption that more diverse reasoning paths universally improve reasoning.The paper emphasizes that answer aggregation and improved reasoning are distinct mechanisms with different AI-safety implications.
- Limitations: The study uses one early-answering faithfulness probe, GSM8K mathematical reasoning problems, 100 problems, and API access with limited control over provider-side factors.Alternative probes, domains, larger samples, and different model versions may produce different patterns.
6 Practical Recommendations
The paper recommends evaluating self-consistency empirically for each model and domain, while explicitly tracking accuracy–faithfulness tradeoffs and problem-difficulty effects.
- Deployment testing: Self-consistency is not universally beneficial; practitioners should test their specific model and domain before deployment.Claude Opus 4.5 experiences net harm from inference scaling.
- Scaling budgets: For GPT-5.2 and Gemini, most scaling gains occur by N=5, while N=20 adds marginal improvement at four times the additional cost.The recommendation favors measuring returns before increasing the sampling budget.
- Faithfulness requirements: When interpretability matters, GPT-5.2 trades accuracy gains for slightly lower faithfulness, Claude trades faithfulness gains for lower accuracy, and DeepSeek gains faithfulness at ceiling accuracy.The appropriate scaling choice depends on whether accuracy or faithful reasoning is the primary requirement.
- Problem difficulty: Monitor whether scaling solves hard problems or breaks easy ones; Claude’s accuracy loss includes breaking 23% of easy problems.Difficulty-level tracking can reveal dynamics hidden by aggregate accuracy.
- Ceiling models: For near-ceiling models such as DeepSeek-v3.2 at 98% accuracy, scaling provides minimal benefit.Scaling should be skipped when its cost is not justified by the intended objective.
7 Future Work
Future work should broaden the evidence base, deepen faithfulness measurement, explain model differences, and explore adaptive inference scaling.
- Adding models such as Llama, Mistral, Qwen, and Gemma would test the generality of the observed scaling patterns.
- Perturbation analysis, attention visualization, and causal interventions could provide a more complete picture of faithfulness under scaling.
- Evaluating commonsense, scientific, and legal reasoning would assess whether the findings generalize beyond GSM8K mathematics.
- Deeper mechanistic analysis is needed to explain why Claude and GPT differ so dramatically in scaling behavior.
- Adaptive scaling based on problem difficulty or model uncertainty could improve efficiency over fixed N.
8 Conclusion
Inference scaling affects reasoning faithfulness differently across models rather than producing a universal improvement. The study recommends model-specific testing and reports released code and data for further research.
- Inference scaling effects vary dramatically: models may gain accuracy with stable faithfulness, lose accuracy while gaining faithfulness, or show ceiling effects.
- +12% accuracy at N=5 for GPT-5.2 accompanies roughly constant faithfulness, matching the expected self-consistency pattern.
- 3.7% lower accuracy for Claude Opus 4.5 accompanies a 230% faithfulness increase, consistent with overthinking across multiple samples.
- DeepSeek-v3.2 was already 98% accurate, while Gemini-3-flash showed modest effects in both directions.
- GPT-5.2 solves 82% of hard problems and breaks 13% of easy ones, whereas Claude breaks 23% of easy problems.
- Teams should test their specific models because self-consistency is not a universal improvement, and single-sample reasoning may work better for some models.
- The authors release code and data to facilitate further research.
Code Availability
The supplied code-availability passages provide implementation prompts but do not state a code or data release.
- The implementation examples instruct models to show work or return only a final numeric answer.
B Complete Results Table
The complete-results appendix includes experimental tables, statistical-analysis components, worked GSM8K examples, and an N=5 majority-vote failure case.
- The appendix lists tables for experimental hyperparameters, complete verified results, and estimated API costs.
- Paired binary outcomes distinguish cases correct at baseline but incorrect after scaling from cases incorrect at baseline but correct after scaling.
- Bootstrap analysis uses percentile intervals based on 1,000 resample estimates.
- The appendix provides an effect-size interpretation threshold of |d| < 0.2 for small effects.
- In one N=1 example, the reasoning incorrectly computes 48 + 24 as 62 and returns 62.
- In another N=1 example, converting 50 minutes to 5/6 hours yields correct earnings of $10.
- At N=5, majority voting selects 12 from responses 10, 12, 12, 12, 10, producing an incorrect answer through rounding errors across paths.