A Study on Thinking Patterns of Large Reasoning Models in Code Generation

• Existing LLM code generation can produce plausible code but still violate requirements, hallucinate, misalign with user intent, and lack robust reasoning.Source: §1 · p. 1
• Prior reasoning taxonomies largely target general reasoning or traditional LLMs, leaving LRM reasoning traces in code generation insufficiently studied.Source: §1 · p. 2
• The study manually analyzes 1,150 reasoning traces from five open-source Qwen-series LRMs on Python CoderEval tasks, producing 15 reasoning actions across four phases.Source: §1 · p. 2
• LRMs generally follow a human-like coding workflow, while complex tasks add scaffolding, flaw detection, and style checks.Source: §1 · p. 2
• Qwen3 models show iterative reasoning, whereas DeepSeek-R1-7B follows a more linear, waterfall-like approach.Source: §1 · p. 2
• Unit-test creation and complete code generation strongly support functional correctness, while context- and reasoning-oriented prompts show improvement potential.Source: §1 · p. 2
• The paper releases 1,150 annotated reasoning traces to support research on LRM reasoning capabilities.Source: §1 · p. 2

2 Study Setup

• The study prompts five LRMs on 230 code-generation tasks, yielding 1,150 traces for analysis and four research questions covering patterns, model differences, correctness, and prompting.Source: §2 · pp. 2–3

2.1 LRM Selection

• The models are selected because their reasoning traces are accessible and they span sizes from DeepSeek-R1-7B and Qwen3-1.7B to QwQ-32B.Source: §2.1 · p. 3

2.2 Reasoning Data Collection

• CoderEval is chosen for realistic, dependency-aware Python and Java tasks spanning standalone snippets through project-level dependencies, but analysis focuses on Python because Java dependency categories are sparse.Source: §2.2 · pp. 3–4
• Each task combines a function signature and docstring in a code-generation prompt, producing 1,150 reasoning traces across 230 tasks and five models.Source: §2.2 · p. 4

2.3 Manual Taxonomy Construction

• Open coding samples 145 tasks and 725 traces, with two experienced annotators identifying and iteratively grouping fine-grained reasoning actions before consensus resolution.Source: §2.3 · p. 4
• Validation on the remaining 425 traces yields Cohen’s Kappa of 0.7054, and arbitration adds the Style Check action before final approval.Source: §2.3 · p. 4

Phase 1: Requirements Gathering

• Requirements Gathering identifies task context and constraints from function signatures and docstrings, including variables, parameters, return values, and functional requirements.The models also recognize ambiguities or missing information that may require assumptions, such as unclear parameter meanings or incomplete project context.Source: §3 · pp. 4–5

Phase 2: Solution Planning

• Solution Planning uses recalled programming knowledge, control- and data-flow construction, solution comparison, and ambiguity recognition to structure an approach before coding.The taxonomy distinguishes pre-implementation Solution Comparison from later Post-Hoc Alternative Exploration.Source: §3 · pp. 4–6

Phase 3: Implementation Generation

• Implementation Generation may proceed from scaffold code that outlines partial logic to complete code containing the executable function and its required components.Scaffolding lets models roughly verify logic, identify missing pieces, and iteratively refine an implementation before producing the final candidate.Source: §3 · pp. 5–6

Phase 4: Reflection

• Reflection evaluates generated code through unit tests, alternative exploration, edge-case identification, flaw correction, style checks, and final self-assertion.Unit tests are tied to prompt requirements, but model-generated execution and validation do not fully guarantee reliability.Source: §3 · p. 6

Taxonomy Overview

• 15 reasoning actions span four phases: Requirements Gathering, Solution Planning, Implementation Generation, and Reflection.Nearly all traces include all four phases, while only 10% omit Implementation Generation.Source: §3 · pp. 4, 6

Observed Action Usage

• Most actions occur frequently, whereas Scaffold Code Generation, Unit Test Creation, Flaw Identification, and Style Check each occur in fewer than 50% of traces.The taxonomy therefore captures both common workflow stages and less frequent verification or refinement behaviors.Source: §3 · p. 6

4.1 RQ1: Common Reasoning Patterns

• LRMs most often follow a human-like workflow of requirements analysis, ambiguity clarification, solution comparison, implementation, and defect review, with extra actions for harder tasks.The most frequent pattern accounts for approximately 17% of traces; simpler tasks often omit scaffolding or ambiguity recognition.Source: §4.1 · p. 7

4.2 RQ2: Comparison Across Models

• Qwen3 models show highly similar iterative reasoning across parameter sizes, whereas DeepSeek-R1-7B uses a more linear, waterfall-like style and often omits implementation generation.DeepSeek-R1-7B has lower frequencies of solution comparison, ambiguity recognition, edge-case identification, and alternative exploration than Qwen3 and QWQ-32B.Source: §4.2 · p. 7

4.3.1 Individual Reasoning Actions

• Unit Test Creation has the strongest individual association with correctness, while Constraint Identification, Ambiguity Recognition, and Solution Comparison show weak negative correlations.The negative associations are linked to incorrect assumptions, prematurely narrowed solution spaces, and reasoning loops.Source: §4.3.1 · p. 8, Figure 5 · p. 9

4.3.2 Combined Reasoning Actions

• Unit Test Creation, Self-Assertion, and Task Identification co-occur with higher success rates, with their combination achieving the highest conversion ratio from attempts to correct outputs.Apriori treats reasoning actions as items and traces as transactions, but mined combinations are subsequences rather than complete reasoning patterns.Source: §4.3.2 · p. 8, Table 4 · p. 9
• Reasoning adapts to dependency level: simpler tasks elicit focused unit-test verification, public-library tasks elicit exploratory library reasoning, and file-level tasks elicit more complete-code and edge-case generation.Project-runnable tasks produce no common combinations because passing results are sparse, while higher-complexity class- and file-runnable tasks show confidence drops of around 0.2–0.3.Source: §4.3.2 · pp. 8–9, Table 5 · p. 10

4.3.3 Key Observations and Insights

• LRMs recognize ambiguities and generate tests, but these behaviors are unreliable: ambiguity can cause repetitive loops, and generated tests can contain incorrect expected outputs.Ambiguity Recognition occurs on 82.18% of class-runnable and 83.24% of file-runnable tasks, while test creation appears in under 11% of higher-dependency tasks.Source: §4.3.3 · p. 10, Figure 7 · p. 11
• Self-assertion does not guarantee genuine confidence, and implementation reasoning appears in only 30% to 55% of traces, which may not fully correspond to the final code output.Models sometimes identify flaws yet output flawed code, while documentation-like reasoning may be omitted because prompts require code-only answers.Source: §4.3.3 · p. 10, Figure 8 · p. 11

4.4 RQ4: Prompting-based Improvements

• Context- or reasoning-guided prompts yield slight improvements across most LRMs, with GUIDE benefiting from unit-test guidance, but randomness prevents concluding that gains are universal.Additional context helps especially on higher-dependency project-runnable tasks, although Qwen3-14B declines under CONT prompting.Source: §4.4 · p. 11, Figure 9 · p. 11

5.1 Implications for Researchers

• Iterative Qwen3-style reasoning and the positive correlation between unit-test creation and test-pass rates suggest directions for improving LRM reasoning and fine-tuning.The authors propose emulating iterative programming practices and incorporating test-driven development paradigms.Source: §5.1 · p. 11
• Taxonomy-grounded visualizations could make LRM reasoning traces more interpretable and help developers trust model outputs.Source: §5.1 · p. 11

5.2 Implication for Software Developers

• Prompt engineering remains important: concise instructions should accompany complete problem statements and explicit contextual information such as docstrings, libraries, classes, and attributes.Incomplete, unclear, or ambiguous prompts can hinder reasoning and negatively affect performance.Source: §5.2 · p. 12
• Developers should review generated code and inspect reasoning traces because LRMs may generate unreliable tests or maintain unsound assumptions.Source: §5.2 · p. 12

• The findings’ cross-language generalizability is limited because the analysis focuses on Python, while reasoning traces for other languages remain future work.Source: §6 · p. 12
• Manual annotation and taxonomy construction threaten internal validity, although conflict-resolution strategies were used to improve reliability.Source: §6 · p. 12
• Non-zero temperature makes outputs nondeterministic, while temperature zero produced timeouts and incoherent repetitive reasoning; responses were documented for reproducibility.Source: §6 · p. 12

• General reasoning taxonomies address problem solving, critique, mathematics, and structured reasoning, but they are not specific to code-generation behavior.Source: §7 · p. 12
• Code-generation studies examine flaws, control flow, and the distinction between code-generation ability and code reasoning.Source: §7 · p. 12
• Existing work also investigates prompt variation and ambiguity clarification as mechanisms for influencing LLM reasoning in mathematics and code generation.Source: §7 · p. 12

• The study develops a taxonomy of 15 reasoning actions across four phases and reveals common and model-specific reasoning patterns.Source: §8 · p. 12
• The analysis shows how reasoning behaviors affect functional correctness and evaluates lightweight prompting strategies for improving code generation.Source: §8 · p. 13
• The findings provide insights into LRM reasoning and practical guidance for automated software development.Source: §8 · p. 13