PaperlayerView exact revision on arXiv

Source-linked AI summary

Enforcing Code Reasoning in Agentic Scaffold for More Effective Issue Resolution

Changshu Liu, Ji young Kim, Alireza Ghazanfari, Saanvi Nallu, Reyhaneh Jabbarvand

arXiv:2610.23782v1cs.SE

TL;DR

Whether and how programming agents reason about code remains unclear, and their reasoning is often unguided. DeskCheck systematically guides code reasoning across issue-resolution phases; on SWE-bench-Pro, it improves resolution rates over existing baselines.

  • Problem

    Agents use code reasoning spontaneously rather than systematically, despite its association with resolution outcomes and increasing prevalence on harder problems.

    Source: §1 · p. 2, §2.2.3 · p. 7

  • Method

    DeskCheck is an agentic scaffold that guides code reasoning across localization, patch repair and refinement, patch auditing, and validation.

    Source: §1 · p. 4, §3.1.4 · p. 10, §4.4 · p. 17

  • Results

    DeskCheck improves issue resolution on SWE-bench-Pro over mini-SWE-Agent and Claude Code by 6.29%–10.53%, with average token usage increasing 7.6%.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The findings motivate systematically guiding code reasoning across issue-resolution phases as a design principle for programming-agent scaffolds.

    Source: §2.2.3 · p. 7

    The findings come from six scaffolds and three model families on the SWE-bench family and may not transfer to other scaffolds or more recent LLMs.

    Source: §7 · p. 18

Abstract

from arXiv · show

Reasoning is central to ReAct-based programming agents, yet whether and how they reason about code remains unclear. We study 7,955 issue-resolution trajectories from six single- and multi-agent scaffolds across four programming languages on SWE-bench-Verified, SWE-bench-Pro, and SWE-bench-Lite. We identify 12 forms of code reasoning in 66.95% of trajectories. Code reasoning is associated with higher resolution rates, and its intensity increases with task complexity. However, it remains largely unguided: agents may focus on surface-level symptoms during localization, fail to refine near-correct patches, overlook issue requirements, and generate tests with limited behavioral diversity. To address these limitations, we propose DeskCheck, an agentic scaffold that systematically guides code reasoning throughout repository-level issue resolution. DeskCheck simulates execution paths to localize root causes, simulates candidate patches to refine near-misses, audits patches against issue requirements, and reasons about corner-case inputs and expected outputs before generating tests. On 731 SWE-bench-Pro instances, DeskCheck improves resolution rates over mini-SWE-Agent and Claude Code by 6.29% to 10.53%, at an additional cost of $0.02-$0.28 per instance (average: $0.14). Per-phase analyses show higher method-level localization recall, more successful repairs of imprecisely localized issues, and better detection of missed corner cases in near-correct patches. Exact McNemar tests show statistically significant improvements across all subject LLMs (p < 10^-10), while repeated experiments support robustness to agent nondeterminism. These findings motivate systematic code reasoning as a means of improving programming agents.

1 Introduction

The introduction frames code reasoning in programming agents as under-studied and often unguided, then presents DeskCheck as a scaffold that applies it across issue-resolution stages. Evaluation on SWE-bench-Pro reports improved resolution over both baselines.

1 Introduction

  • Prior trajectory analyses focused on agents’ actions and tool use, leaving the reasoning behind those actions—and code reasoning in particular—under-studied.The paper studies 7,955 issue-resolution trajectories from six scaffolds across three SWE-bench datasets and four programming languages.Source: §1 · p. 2
  • Code reasoning appears in 66.95% of trajectories and correlates with resolution, but agents use it spontaneously rather than systematically.The authors also report that agents use code reasoning more frequently as problems become harder.Source: §1 · p. 2
  • The examples expose four recurring gaps: symptom-level localization, difficult test setup, overlooked issue requirements, and validation tests with limited behavioral diversity.Examples include missing a crashing helper, failing to configure a test environment, patching only some requirements, and omitting case-insensitive or multi-word inputs.Source: §1 · pp. 2–4
  • DeskCheck guides reasoning through execution-path localization, patch simulation and refinement, requirement auditing, and input–expected-output reasoning for validation tests.The scaffold applies these mechanisms across localization, repair and refinement, audit, and validation.Source: §1 · p. 4
  • On 731 SWE-bench-Pro problems, DeskCheck improves issue resolution over mini-SWE-Agent by 6.29%–10.53% and over Claude Code by 8.34%.The reported gains hold across the studied MiniMax-M3, GPT-5.6-Luna, and Claude Haiku-4.5 backbones.Source: §1 · p. 4
  • The paper’s contributions include a large-scale study of agent reasoning, the DeskCheck scaffold, and its quantitative and qualitative evaluation.The study examines how agents incorporate code reasoning while resolving SWE-bench-family instances.Source: §1 · pp. 4–5

2 Code Reasoning in Issue Resolution Agents

The study examines code reasoning across programming-agent trajectories, how it relates to resolution, and how its use varies by task difficulty and scaffold. Code reasoning is common and associated with higher resolution, but its use varies across phases and motivates systematic guidance.

2 Code Reasoning in Issue Resolution Agents

  • The analysis selects six architecturally diverse scaffolds and evaluates their trajectories using Claude, GPT, and MiniMax backbone models.The selection includes mini-SWE-Agent, OpenHands, ExpeRepair, iCAT-Agent, Sonar Foundation Agent, and Claude Code.Source: §2 · p. 5

2.1 Code Reasoning Patterns

  • Code reasoning concerns an agent’s reasoning about a program fragment’s runtime behavior under a concrete or symbolic execution context before taking an environment-interaction action.The study categorizes reasoning steps using code elements and patterns including variable state, calls, loops, conditions, and returns or exceptions.Source: §2.1 · p. 5

2.1.1 Manual Audit.

  • A human audit of 120 trajectories found substantial agreement between majority labels and the automated categorization, with Krippendorff’s α=0.87.The audited sample included 60 trajectories classified with and 60 without code reasoning.Source: §2.1.1 · p. 6

2.2.1 RQ1: Prevalence of Code Reasoning in Program-

  • 66.95% of trajectories incorporated code reasoning on average across 14 configurations of six agent scaffolds.Incorporation varied by scaffold; ExpeRepair prompts require explicit analysis at every stage, while mini-SWE-Agent and OpenHands request reasoning less consistently.Source: §2.2.1 · p. 6, Figure 6 · p. 6

2.2.2 RQ2: Correlation with resolution.

  • Resolution was 69.84% when code reasoning was incorporated and 50.81% when it was not, and scaffold-level tests found an association with higher resolution.The chi-square analysis excluded Sonar Foundation Agent and ExpeRepair because they had too few instances without detected code reasoning.Source: Table 1 · p. 6, §2.2.2 · p. 7

2.2.3 RQ3: Influencing factors.

  • Code reasoning rose on average from 63.9% for easy issues to 67.6% for medium and 70.5% for difficult issues, though the trend was not monotonic for every configuration.Difficulty was based on edited-file counts for SWE-bench-Pro and benchmark difficulty levels for SWE-bench-Verified and SWE-bench-Lite.Source: §2.2.3 · p. 7, Figure 7 · p. 7
  • The phase with the most code reasoning varied by scaffold, while overall reasoning became more prevalent with harder issues, motivating systematic guidance across issue-resolution phases.Some scaffolds reasoned most during localization, others during validation, and ExpeRepair during patching.Source: §2.2.3 · p. 7, Figure 8 · p. 7

3 DeskCheck: Code Reasoning Augmented Issue Resolution Agent

Execution Simulation is a form of code reasoning in which an agent mentally steps through a program slice under an input rather than executing it.

3 DeskCheck: Code Reasoning Augmented Issue Resolution Agent

  • Execution Simulation means reasoning through a program slice step by step under a concrete or symbolic input until it terminates, without executing it.The slice may be an execution path or a patch; a simulation may exhibit any subset of the 12 identified code-reasoning patterns.Source: §3 · p. 8
  • The simulated program slice can be an execution path or a patch.Both are examples given for the program slice in the definition.Source: §3 · p. 8
  • The simulation uses a concrete or symbolic input and reasons until the slice terminates.The definition allows any subset of the 12 code-reasoning patterns to appear in a simulation.Source: §3 · p. 8

3.1 DeskCheck

DeskCheck divides issue resolution into five stages, using dedicated code-reasoning components after repository exploration. It simulates paths and patches, checks requirements, and uses test feedback to refine a patch before submission.

3.1 DeskCheck

  • DeskCheck uses five stages, with dedicated code-reasoning components in every stage except Explore, and assigns each stage a sub-agent to manage its task context.Explore is read-only: it reviews the task, searches the repository, and reads likely relevant source files.Source: §3.1 · p. 8, §3.1.1 · p. 8
  • The workflow iteratively refines patches using audit or test feedback and submits when validation reports no error.The overall workflow routes audit violations to lightweight repair before validation.Source: §3.1 · p. 8

3.1.2 Stage 2: Localize.

  • Localization builds a call graph, selects paths covering predicted anchors and other likely bug paths, then simulates them to identify root causes and edit sites.For each selected path, the agent predicts triggering inputs and reasons through execution step by step; localization can also identify where new functionality or structural changes belong.Source: §3.1.2 · p. 9

3.1.3 Stage 3: Repair & Refine.

  • Repair & Refine simulates an initial patch on predicted input–expected-output pairs and uses mismatch feedback to refine it when violations appear.The pairs are intended to fulfill both the implementation and issue requirements.Source: §3.1.3 · p. 10, §3.1.2 · p. 10

3.1.4 Stage 4: Patch Audit.

  • Patch Audit extracts issue requirements, simulates the patch against a predicted input–output pair for each, and routes detected violations to a lightweight repair.It runs in a clean context after Repair & Refine and can also refute false alarms.Source: §3.1.4 · p. 10, §3.1.2 · p. 10

3.1.5 Stage 5: Validation.

  • Validation combines repository regression tests with generated tests based on corner cases and issue-reproduction input–output pairs, then refines the patch if tests fail.If validation tests pass, the validation sub-agent submits the patch.Source: §3.1.3 · p. 10, §3.1.5 · p. 11

3.2 Experiment Setup

DeskCheck is implemented on mini-SWE-Agent and Claude Code and evaluated on SWE-bench-Pro. The supplied setup describes the benchmark and safeguards, but does not provide performance values.

3.2 Experiment Setup

  • The evaluation uses 731 SWE-bench-Pro instances spanning Python, JavaScript, Go, and TypeScript.The passage characterizes its multi-file tasks as more challenging than those of SWE-bench-Verified and SWE-bench-Lite.Source: §3.2 · p. 11
  • DeskCheck stages are implemented on mini-SWE-Agent and Claude Code by modifying or injecting prompts, with localization expansion budget k=1 to save tokens.The prompts and implementation are publicly available.Source: §3.2 · p. 11
  • Table 2 is identified as a comparison of DeskCheck’s overall performance with baseline scaffolds on SWE-bench-Pro.The supplied table passage gives its scope but no cell values.Source: Table 2 · p. 11

3.3 RQ4. Effectiveness of DeskCheck in Issue Resolution

On SWE-bench-Pro, DeskCheck outperforms mini-SWE-Agent and Claude Code across difficulty levels and programming languages. The authors report statistically significant gains across all three subject LLMs, while manual analysis identifies hidden-test cases that can make benchmark scores underestimate actual capability.

3.3.1 Overall Performance of DeskCheck.

  • DeskCheck resolves 65.25%, 59.37%, and 51.16% of instances with MiniMax-M3, GPT-5.6-Luna, and Claude Haiku-4.5, outperforming the corresponding baselines by 10.53%, 6.29%, and 8.34%.It costs $0.02–$0.28 more per instance; DeskCheck produces unique fixes compared with both baselines.Source: §3.3.1 · p. 11, Figure 11 · p. 11

3.3.2 Breakdown by Difficulty Levels.

  • DeskCheck resolves more easy, medium, and difficult instances than mini-SWE-Agent by 9.4%, 8.7%, and 8.0%, and than Claude Code by 11.4%, 8.0%, and 7.8%.Difficulty is proxied by golden-patch characteristics.Source: §3.3.2 · pp. 11–12, Figure 12 · p. 12

3.3.3 Breakdown by Programming Languages.

  • DeskCheck improves resolution rates across all programming languages, but Go remains the hardest language and yields its smallest gains.The authors attribute this largely to grading tests that require names found only in golden patches, not issue descriptions.Source: §3.3.3 · p. 12, Figure 13 · p. 12

3.3.4 Impact of Non-determinism.

  • Exact McNemar tests find significant gains for MiniMax-M3, GPT-5.6-Luna, and Claude Haiku-4.5 (p = 2.6 × 10^-13, 1.0 × 10^-11, and 1.6 × 10^-11).Baseline retries resolve only 16, 7, and 14 of DeskCheck’s 97, 49, and 74 unique fixes, respectively; the authors estimate 83.18% of gains likely stem from DeskCheck rather than run-to-run nondeterminism.Source: Figure 11 · p. 11, §3.3.4 · p. 12

3.3.5 Manual Analysis of Generated Patches.

  • Some Go hidden tests call five private helpers by golden-patch names although the issue names only one new function, causing DeskCheck’s differently named helpers to fail compilation.None of the 33 hidden tests run in this example, so the authors caution that SWE-bench-Pro resolution rates can underestimate capability and small score margins warrant caution.Source: Figure 14 · p. 12, §3.3.5 · p. 13

3.4 RQ5: Contribution of Guided Code Reasoning to Each Stage of DeskCheck

DeskCheck’s guided code reasoning contributes across localization, repair and refinement, audit, and validation. Analyses associate stage-triggered edits with rescuing previously failed instances, while localization achieves higher method-level recall.

3.4.1 Code Reasoning in the Localization.

  • DeskCheck-L achieves the highest method-level recall, and that advantage persists in final patches; its file-level recall is also highest or near-highest, with lower precision than some baselines.The comparison uses golden-patch edits as a proxy for ground truth and includes a 107-instance Python subset for SWE-Doctor comparisons; the proxy may include false positives or miss alternative valid fixes.Source: §3.4.1 · pp. 13–14, Table 3 · p. 13

3.4.2 Code Reasoning in Repair & Refine.

  • On baseline-failed instances, refinement is associated with higher rescue rates than no refinement: 57% vs. 27%, 19% vs. 10%, and 32% vs. 15% across the three models.Chi-square tests find a significant correlation between refinement and rescue for all three models (p < 0.05).Source: §3.4.2 · p. 14, Figure 16 · p. 14

3.4.3 Code Reasoning in Audit.

  • The audit flags requirement violations in 294, 287, and 223 instances, and lightweight repairs resolve 160, 91, and 76 of the changed patches across the three models.The reported counts distinguish flagged instances, changed patches, and resolved patches.Source: §3.4.3 · p. 14, Figure 17 · p. 15

3.4.4 Code Reasoning in Validation.

  • Validation-stage edits are associated with higher rescue rates than unedited patches on baseline-failed instances: 47% vs. 26%, 19% vs. 11%, and 31% vs. 16%.The authors attribute revisions to tests built from reasoned input–expected-output pairs that cover more diverse scenarios and edge cases; chi-square tests find significant correlations for all models (p < 0.05).Source: §3.4.4 · p. 15

4 Qualitative Analysis

The examples show how DeskCheck’s simulations expose root causes, near-miss behavior, and unmet requirements, while reasoned test inputs reveal corner cases that baseline tests miss.

4.1 Execution path simulation improves localization

  • Execution-path simulation traces invalid settings to _remove_empty_patterns, a hidden-test-called function omitted from the issue description and missed by Claude Code.The concrete input reveals that the function’s in and del operations crash on the invalid settings.Source: §4.1 · pp. 15–16, Figure 19 · p. 16

4.2 Initial patch simulation saves near-hits

  • Patch simulation detects that set_queue raises RuntimeError under the wrong condition, prompting refinement that passes hidden tests where mini-SWE-Agent submits an untested near-hit.The simulated parent-process call tests a condition difficult to reproduce with real tests requiring multiple processes.Source: §4.2 · p. 16, Figure 19 · p. 16

4.3 Auditing against the requirements completes the patch

  • Requirement-by-requirement audit catches two unmet requirements in a patch satisfying only six of eight, guiding repair that changes the fail-to-pass rate from 0/1 to 1/1.The examples identify missing behavior for active-context subscription handling and required return fields.Source: §4.3 · p. 16

4.4 Reasoning about inputs and expected outputs improves validation test quality

  • Reasoned validation covers case-insensitive and multi-word inputs missed by baseline tests, catches a failing patch, and prompts refinement; it also adds empty and None inputs absent from hidden tests.The added inputs extend behavioral coverage beyond both the baseline and hidden tests.Source: §4.4 · p. 17, Table 5 · p. 17

5 Ablation Study

The ablation progressively restores explicit code reasoning across DeskCheck’s stages on a stratified 100-instance subset. Resolution improves as stages are restored, with validation producing the largest reported increment.

5 Ablation Study

  • With MiniMax-M3, resolution rises from 53% without explicit reasoning to 56%, 58%, and 61% as localization, repair, and audit reasoning are restored, then reaches 68% with the full system.Restoring validation reasoning raises resolution from 61% to 68%, the largest reported stage increment.Source: §5 · p. 17, Table 6 · p. 17
  • Method-level localization recall rises from 0.59 to 0.65 when localization reasoning is restored and reaches 0.67 thereafter.The authors suggest that simulating execution paths identifies more bug-relevant locations.Source: §5 · p. 17, Table 6 · p. 17

6 Related Work

Prior work evaluates code reasoning in standalone programs or analyzes agent actions and tool use, while this work studies code reasoning within programming agents’ problem-solving processes. DeskCheck also differs from prior reproduction-test and patch-refinement approaches by using code reasoning to trace root causes and simulate patches.

Code Reasoning Evaluations

  • Existing code-reasoning benchmarks target standalone programs, while agent-trajectory analyses focus on actions and tool use rather than reasoning.The paper instead studies code reasoning in programming agents’ problem-solving processes at scale.Source: §6 · pp. 17–18

Reproduction Tests for Bug Localization; Patch Refinement

  • DeskCheck traces execution paths to reveal root causes when reproduction tests capture only surface symptoms, and simulates patches on issue-inferred inputs even without runnable tests.Prior refinement methods use external signals such as reproduction or regression tests, repository context, or patch conciseness; these signals can be unavailable or limited by test quality.Source: §6 · p. 18

Reproduction Tests for Bug Localization

  • Prior reproduction-test work includes generation from issue reports, heterogeneous prompting and execution feedback, test refinement, assertion inversion, and using test evidence for localization and repair.The passage names AEGIS [60], Otter [2], e-Otter++ [3], Issue2Test [53], AssertFlip [28], MarsCode Agent [45], and SWE-Doctor [18].Source: §6 · p. 18

Patch Refinement

  • Patch-refinement approaches use failing tests, enriched repository context, or conciseness objectives, whereas DeskCheck derives a refinement signal by simulating the patch on issue-inferred inputs.The passage names PatchPilot [34], REFINE [55], and Luo et al. [46].Source: §6 · p. 18

7 Threats to Validity

The study’s conclusions are bounded by its SWE-bench-family trajectories and by possible nondeterminism and hidden-test false negatives. The authors use diverse scaffolds, multiple validity checks, and manual review to address these threats.

External Validity

  • The code-reasoning findings come from 7,955 trajectories across six scaffolds and three model families on SWE-bench, so may not transfer to other scaffolds or newer LLMs.The authors include diverse scaffold architectures and SWE-bench-Pro’s Python, JavaScript, Go, and TypeScript coverage to mitigate this external-validity concern.Source: §7 · p. 18

Internal Validity

  • Because LLM-based systems and DeskCheck are nondeterministic, exact McNemar tests and baseline reruns checked whether improvements reflected code reasoning rather than run-to-run variation.On instances only DeskCheck resolved, 83.18% were not explained by baseline run-to-run nondeterminism.Source: §7 · p. 18

Construct Validity

  • SWE-bench-Pro hidden tests may reject correct patches, so the authors manually verify false negatives for DeskCheck and mini-SWE-Agent.Source: §7 · p. 18

8 Conclusion & Future Work

The paper categorizes programming agents’ code reasoning and studies 7,955 trajectories, then introduces DeskCheck, a five-stage framework for localization, repair, and validation. On SWE-bench-Pro, DeskCheck yields up to a 10.53% performance gain over mini-SWE-Agent and Claude Code, and the authors identify two directions for future work.

Conclusion

  • DeskCheck brings up to a 10.53% performance gain compared with mini-SWE-Agent and Claude Code on SWE-bench-Pro, with a marginal increase in token consumption and cost.Source: §8 · p. 18
  • The authors derive a systematic categorization of programming agents’ reasoning behaviors and analyze 7,955 trajectories to study how code reasoning is integrated into issue resolution.Source: §8 · p. 18
  • DeskCheck is a five-stage agent framework that uses different levels of code reasoning to improve performance in localization, repair, and validation.Source: §8 · p. 18

Future Work

  • Future work will extend DeskCheck beyond bug-fixing and certify the faithfulness of code reasoning, including execution-simulation correctness.Source: §8 · p. 18
Loading 2610.23782v1…