PaperlayerView exact revision on arXiv

Source-linked AI summary

Does More Thinking Pay Off? Reasoning Cost and Correctness of LLMs in Code Generation and Bug Fixing

Andie Huang, Ying Pang, Qian Zhang, Yihang Cheng, Lin Ma

arXiv:2610.36952v1cs.SE

TL;DR

Correctness-only evaluations leave unclear whether reasoning traces improve code generation and bug fixing enough to justify their token cost. The paper evaluates reasoning across Python benchmarks, analyzes trace behavior, and tests regeneration from shortened stored traces. Reasoning does not consistently improve accuracy, while retaining half a trace yields no significant correctness drop in the evaluated comparisons.

  • Problem

    Code-task evaluations usually report solution correctness, leaving unclear whether explicit reasoning changes outcomes, what traces contain, or whether their full length is needed.

    Source: §1 · p. 2

  • Method

    The study compares reasoning on and off, analyzes trace behaviors, and tests executable solutions regenerated from prefixes of stored traces across code-generation and bug-fixing tasks.

    Source: §1 · p. 3, §3.3 · p. 7, §3.5 · p. 10

  • Results

    Reasoning does not consistently help across benchmarks, and Fixed-50 removes about half of stored traces without a significant accuracy decrease in the evaluated comparisons.

    Source: §7 · p. 18

  • Takeaways & Limitations

    Correctness should be reported alongside thinking-token overhead, and shortened-trace evaluations should compare regenerated answers with the same model's full-trace answers.

    Source: §1 · p. 3, §7 · p. 18

    The findings cover Python code generation and bug fixing with open models of at most 70B parameters, and may not extend to other languages, tasks, or larger or closed models.

    Source: §6.2 · p. 17

Abstract

from arXiv · show

Many LLMs now emit an explicit reasoning trace before they write code, but benchmark results for code generation and bug fixing usually report correctness alone. It is therefore unclear how much the extra reasoning contributes relative to the tokens it consumes. We examine this question with 866 Python tasks drawn from HumanEval Pro, MBPP Pro, and BugsInPy under five model settings. We look at the accuracy gain and token overhead of reasoning, the behaviors that appear inside reasoning traces, and whether a shortened trace still leads to a correct solution. Turning reasoning on in Qwen3-32B over five runs improves mean pass@1 on HumanEval Pro from 71.1% to 76.5% and mean tests pass on BugsInPy from 3.5% to 7.9%. On MBPP Pro the improvement from 68.1% to 69.5% is not statistically significant, even though the model uses a median of over 2,000 thinking tokens per task. Across models, a longer median trace does not reliably mean higher accuracy. A qualitative analysis shows that traces mostly choose and check an implementation in code generation, while in bug fixing they mostly locate the fault and plan the repair. We then truncate stored traces offline and ask Qwen3-32B with reasoning off to produce a solution from what remains. Retaining the first half of a trace yields no statistically significant drop in correctness. Stopping when the trace becomes repetitive seldom triggers, and stopping once solution code first appears benefits some models but harms others. Neither rule beats plain fixed truncation. These results suggest that more reasoning is not always better and that studies should report reasoning cost alongside correctness.

1 Introduction

The introduction asks whether explicit reasoning improves code generation and bug fixing enough to justify its token overhead, since standard correctness metrics do not show what traces contain or whether they matter. It outlines a study comparing reasoning settings, analyzing trace behavior, and testing whether shortened stored traces can support correct regenerated solutions.

1 Introduction

  • Prior evaluations usually report executable correctness, leaving unclear whether reasoning changes code-task outcomes or whether later trace content is needed.A motivating example generated 7,430 thinking tokens before a 103-token solution, illustrating overhead and trace behavior not captured by correctness alone.Source: §1 · p. 2
  • The study compares Qwen3-32B with reasoning on and off across five runs, annotates traces from code generation and bug fixing, and tests correctness from early trace prefixes.It covers HumanEval Pro, MBPP Pro, and BugsInPy tasks.Source: §1 · p. 3
  • Reasoning increased mean accuracy on all three benchmarks, but gains were significant in only two of five HumanEval Pro runs, four of five BugsInPy runs, and none of the MBPP Pro runs.Across four reasoning models, longer median traces did not consistently correspond to higher accuracy.Source: §1 · p. 3
  • In code generation, reasoning traces mostly plan and check implementations, whereas bug-fixing traces mostly locate faults and plan repairs.The study derives a six-category codebook and tests it on traces not used during codebook development.Source: §1 · p. 3
  • For all four reasoning models and three benchmarks, retaining the first half of stored traces produced no significant correctness difference against the full-trace baseline; RedStop and NRP did not consistently improve on fixed truncation.The baseline compares answers written by the same model from full versus shortened traces, accounting for correctness changes introduced by regeneration.Source: §1 · p. 3

2 Related Work

Related work examines reasoning methods and code-task correctness, but typically does not establish whether explicit reasoning changes executable outcomes or whether later trace content contributes to solutions. This study treats generated traces as artifacts for qualitative coding and tests prefix sufficiency by regenerating executable answers from stored traces.

2.1 Reasoning in Large Language Models

  • Reasoning-mode models make thinking-token overhead observable, while prior chain-of-thought evaluations mainly focus on final answers and benefits vary by task type.Chain-of-thought extensions include generic reasoning instructions, multiple sampled paths, and searches over intermediate states.Source: §2.1 · pp. 3–4

2.2 LLMs for Code Generation and Bug Fixing

  • Code-generation and repair benchmarks test executable outcomes, but do not reveal whether explicit reasoning changed results or later trace content contributed to solutions.HumanEval Pro and MBPP Pro add harder self-invoking problems to their base problems.Source: §2.2 · p. 4

2.3 Qualitative Analysis in Software Engineering

  • The RQ2 codebook is a taxonomy of model reasoning-trace behavior, not evidence about human cognition.Its annotation approach follows qualitative coding practices for textual artifacts.Source: §2.3 · p. 4

2.4 Reasoning Efficiency and Reasoning Redundancy

  • The study tests whether prefixes of stored traces can regenerate executable solutions, comparing fixed-fraction, repetition-based RedStop, and first-usable-code NRP cutoffs on the same traces.Prior redundancy analyses focus on math, commonsense, and multiple-choice reasoning, leaving software engineering tasks less explored.Source: §2.4 · p. 4

3 Experiment Setup

The experiment evaluates code generation and bug fixing across three Python benchmarks, five model configurations, and repeated generations. It defines correctness and reasoning-cost measures, then uses trace annotation and offline prefix truncation to study reasoning behavior and sufficiency.

3.1 Tasks and Benchmarks

  • The study retains 164 HumanEval Pro tasks, 374 MBPP Pro tasks, and 328 BugsInPy bugs for executable evaluation.The BugsInPy set excludes bugs with unsuitable patches or benchmark fixes, timeouts, compile failures, or already-passing buggy versions.Source: §3.1 · pp. 5–6

3.2 Models

  • Five configurations are evaluated on the same 866 tasks, with Qwen3-32B providing the paired reasoning-on/off comparison.Each task-model combination is run five times at temperature 0.6 and top-p 0.95; other reasoning models are evaluated only with reasoning enabled.Source: §3.2 · p. 6

3.3 Evaluation Metrics

  • Correctness uses pass@1 for code generation and tests pass for bug fixing, while TTC counts reasoning-trace tokens using each source model’s tokenizer.Paired comparisons report accuracy deltas and confidence intervals across runs, with exact McNemar tests comparing changed task outcomes within each run.Source: §3.3 · pp. 6–7

3.4 Generation and Evaluation Pipeline

  • Generation prompts pair each code-generation base problem with its self-invoking problem and provide bug-fixing prompts with affected code, test commands, and available failing-test source.The five RQ1 runs produce 21,650 records; a separate run stores 3,464 reasoning traces for RQ2 and RQ3.Source: §3.4 · p. 8

3.5 Reasoning Analysis Design

  • The RQ2 analysis segments sampled traces for inductive behavior coding, with separate phases for codebook development, agreement validation, and application.Phase C samples code-generation and bug-fixing traces from each model; traces longer than 3,000 TTC are excluded to limit annotation workload.Source: §3.5 · pp. 8–9
  • RQ3 appends retained prefixes of stored traces to the original prompts and uses Qwen3-OFF to regenerate solutions against a full-trace control.Strategies include fixed 25%, 50%, or 75% prefixes, lexical-repetition stopping, and NRP, which detects an early segment containing code that could be submitted as the answer.Source: §3.5 · pp. 10–11

4 Results

Across the three benchmarks, reasoning’s accuracy gains vary, while its token cost is substantial and longer traces do not reliably correspond to higher accuracy. Trace behaviors differ between code generation and bug fixing, and retaining half a trace preserves correctness without a significant change in the tested code-generation tasks.

4.1 RQ1. How does enabling reasoning affect task correctness and thinking token overhead?

  • Qwen3-32B reasoning raised mean pass@1 from 71.1% to 76.5% on HumanEval Pro and mean tests pass from 3.5% to 7.9% on BugsInPy; MBPP Pro’s 68.1% to 69.5% gain was nonsignificant.Across five runs, gains were significant in two HumanEval Pro runs and four BugsInPy runs; none were significant on MBPP Pro.Source: §4.1 · pp. 11–12
  • Qwen3-ON’s median TTC ranged from 2,221 tokens on MBPP Pro to 3,469 on BugsInPy, and longer median traces did not consistently correspond to higher accuracy.On HumanEval Pro, Qwen3-ON used 3,318 tokens for 76.5% accuracy, while models with shorter traces had accuracy between 76.0% and 77.1%.Source: §4.1 · p. 12

4.2 RQ2. What recurring behaviors appear in reasoning traces, and how do they differ between code generation and bug fixing?

  • Code generation segments were 43.0% Strategy Formation, 34.3% Verification, and 17.1% Sensemaking; bug-fixing segments were 46.1% Diagnosis and Localization and 37.1% Strategy Formation.Together, the three leading code-generation categories accounted for 94.4%; Verification was 4.8% in bug fixing.Source: Figure 4 · p. 13, §4.2 · p. 13
  • The six behavior categories remained usable in 104 Phase C traces, but this sample does not establish that they cover behaviors in other models, languages, or software-engineering tasks.The analysis tests codebook usability on an unseen sample rather than estimating population proportions.Source: §4.2 · pp. 12–13
  • Code generation often transitions from Sensemaking to Strategy Formation to Verification and back; bug-fixing transitions move from Sensemaking to Diagnosis and Localization in 88% of cases, then from Diagnosis and Localization to Strategy Formation in 80%.Table 5 reports transitions between different consecutive behavior labels, omitting self-transitions.Source: §4.2 · pp. 13–14, Table 5 · p. 14

4.3 RQ3. How much correctness is retained when a new answer is generated from a shorter reasoning prefix?

  • Full-trace regeneration closely reproduced source-model correctness on HumanEval Pro and MBPP Pro, differing by at most 0.6 percentage points, and served as the truncation baseline.On BugsInPy, the maximum difference was 1.0 percentage point.Source: Table 6 · p. 14, §4.3 · p. 15
  • Fixed-50 removed 50.3% to 50.7% of traces on HumanEval Pro and MBPP Pro without a statistically significant accuracy change after Holm correction.Its accuracy was slightly lower in every case except R1-Distill-Qwen-32B on HumanEval Pro.Source: Table 6 · p. 14, §4.3 · p. 15
  • RedStop removed only 2.2% to 6.4% of code-generation traces, while NRP removed 30.7% to 46.2% with model-varying accuracy; BugsInPy results did not support a strong truncation claim.NRP was more accurate than Fixed-50 for some models and less accurate for others; all regenerated BugsInPy accuracy values remained at or below 8.7%.Source: §4.3 · pp. 15–16

5 Discussion

The discussion recommends evaluating reasoning by task and model, since trace length does not reliably indicate accuracy and similar token counts can represent different work. It also identifies conditions for using shortened traces and comparing stopping strategies.

5.1 Implications for Developers

  • Qwen3-32B’s MBPP Pro mean pass@1 rises from 68.1% to 69.5% without statistical significance despite a median of over 2,000 thinking tokens per task.The discussion cautions that an accuracy gain may still leave absolute correctness low for practical use.Source: §5.1 · p. 16
  • Keeping the first half of a trace causes no significant accuracy change on HumanEval Pro and MBPP Pro when Qwen3-OFF writes the answer.The comparison uses the same model to generate answers from full and shortened traces; input tokens, output tokens, and extra-call time still matter for efficiency estimates.Source: §5.1 · p. 16

5.2 Implications for Researchers

  • Researchers should report paired correctness and thinking-token overhead for each benchmark across repeated runs.Source: §5.2 · p. 16
  • In code generation, labeled segments mostly plan implementations or check candidate behavior, while bug-fixing segments mostly locate faults and plan repairs.Token counts alone do not show what reasoning is used for, so researchers should analyze trace content as well as length.Source: §5.2 · p. 16
  • About half of stored traces can be removed without a significant accuracy change on HumanEval Pro and MBPP Pro, suggesting shorter traces can inform training or efficiency targets.This discussion connects the result to prior work training models on compressed reasoning traces.Source: §5.2 · p. 16
  • Repetition rarely triggers an early stop, while stopping at the first solution code is not stable across models; neither signal beats plain fixed truncation.Offline screening applies candidate rules to the same stored traces, avoiding differences from separate generations.Source: §5.2 · pp. 16–17
  • A fair truncation baseline has the same model regenerate an answer from the full trace, isolating the amount of reasoning retained.Comparing shortened-trace answers with original answers would mix trace removal with the effect of writing a new answer.Source: §5.2 · p. 17

6 Threats to Validity

The validity discussion bounds the findings by the study’s stochastic, quantized setup, its Python task and model coverage, and the operational measures used for reasoning and truncation.

6.1 Internal Validity

  • Five stochastic runs capture variation across outputs but cannot cover every possible output, and Qwen3-OFF may reason implicitly while generating solutions.The paired Qwen3 conditions use the same checkpoint and evaluation procedure, so differences concern emitted traces rather than all reasoning.Source: §6.1 · p. 17
  • All model settings use 4-bit AWQ quantization, which may change reasoning length, behavior, or accuracy; results were not compared with higher-precision inference.Quantization is shared across comparison conditions, but absolute accuracy and Thinking Token Count may differ from unquantized models.Source: §6.1 · p. 17
  • Behavior labels rely on human judgment and characterize 104 annotated traces rather than population proportions; offline truncation uses Qwen3-OFF to regenerate answers from cuts chosen after full traces.Results may differ if source models themselves answer from shorter traces.Source: §6.1 · p. 17

6.2 External Validity

  • Findings cover Python code generation and bug fixing with open models of at most 70B parameters, not all languages, task types, model sizes, or decoding settings.The reproducible BugsInPy subset is limited to bugs compatible with the prompt and Docker evaluator and is not a uniform sample of all BugsInPy bugs.Source: §6.2 · p. 17

6.3 Construct Validity

  • Reasoning labels describe what trace text does, not how the model internally computes its answer; cross-model token counts are approximate because tokenizers differ.Prefix reduction excludes solution tokens and additional-call generation time, while BugsInPy truncation results are weak evidence because accuracy is at or below 8.7% in every condition.Source: §6.3 · pp. 17–18

7 Conclusion

The conclusion finds that explicit reasoning does not consistently improve Python code generation and bug fixing enough to justify its token overhead. Trace behavior differs by task, and fixed truncation can remove substantial stored reasoning without significant accuracy loss in evaluated comparisons.

7 Conclusion

  • Reasoning does not consistently help: Qwen3-32B gains more on HumanEval Pro and BugsInPy, while MBPP Pro’s gain is indistinguishable from chance despite over 2,000 thinking tokens per task.Across four reasoning models, longer traces do not reliably correspond to higher accuracy, and bug-fixing correctness remains far from practical use even with reasoning.Source: §7 · p. 18
  • Code-generation traces mostly develop and check implementations, whereas bug-fixing traces focus more on locating faults and planning repairs.The six behavior categories cover the held-out Phase C sample, but their distributions differ between task families; token count alone cannot explain reasoning use.Source: §7 · p. 18
  • Against answers regenerated from the full trace by the same model, Fixed-50 removes about half of stored reasoning without a significant accuracy decrease in evaluated comparisons.RedStop rarely finds a stopping point, and NRP varies across source models; the conclusion calls for reporting correctness alongside thinking-token overhead.Source: §7 · p. 18

Related papers

Loading 2610.36952v1…