Source-linked AI summary
To Run or Not to Run: Analyzing the Cost-Effectiveness of Code Execution in LLM-Based Program Repair
Zhihao Lin, Junhua Zhu, Mingyi Zhou, Xin Wang, Zhensu Sun, Renyu Yang, David Lo, Li Li
TL;DR
The contribution of costly code execution to LLM-based program repair remains unclear. This paper studies execution behavior and finds that restricting execution barely changes repair success while reducing resource use.
Problem
The contribution of costly code execution to LLM-based program-repair performance, and whether its benefits justify its cost, remains unclear.
Method
The study analyzes public agent traces and controlled repairs with fixed agent scaffolds while varying execution access across four paradigms.
Results
1.25 percentage points is the average Prohibited–Unrestricted resolve-rate gap on commercial agents, and it is not statistically significant.
Takeaways & Limitations
Execution should be treated as a resource with an explicit cost-benefit tradeoff rather than a default capability.
Takeaways & Limitations
The conclusions are scoped to SWE-bench-style repository-level bug fixing with three current CLI agents and 200 instances.
Abstract
from arXiv · showhide
LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based approach has become standard practice in state-of-the-art systems. However, executions can be time-consuming and expensive, yet their impact on these agents remains underexplored. In this paper, we conduct a two-stage empirical study over execution behavior in LLM-based program repair. To characterize execution behavior at scale, we first analyze 7,745 agent traces from SWE-bench leaderboard submissions. Second, we evaluate 3,000 end-to-end repair attempts across 200 SWE-bench instances and three agents (Claude Code, Codex, and the open-source OpenCode) under four execution paradigms, which allows for a fine-grained comparison of performance and cost. Our analysis reveals three key observations: (1) Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task. Execution behavior varies substantially across agents and models, with frequency ranging from 2 to 19 per task, and late-stage executions consistently achieve higher success rates than early-stage ones. (2) Execution restrictions have little effect on repair success: on commercial agents with SOTA models the resolve-rate gap between Prohibited and Unrestricted is only 1.25 percentage points and not statistically significant, while Prohibited saves substantial token and wall-clock cost. (3) Execution benefit is concentrated rather than uniform. These patterns suggest that current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit. Execution, therefore, should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.
1 INTRODUCTION
The paper studies code execution as a costly but insufficiently understood component of iterative LLM-based program repair. It isolates execution access experimentally and combines large-scale trace analysis with controlled SWE-bench repair attempts.
- Motivation: Code agents use execution to reproduce bugs, localize faults, validate patches, and guide subsequent repair through test outcomes, errors, and logs.Execution feedback supports iterative improvement, but requires agents to generate commands, parse outputs, and wait for compilation, runtime, and test results.
- Research gap: The role of execution remains unclear because prior work largely treats it as an implicit pipeline component rather than systematically investigating its contribution or cost justification.Existing studies primarily emphasize architectures, prompting, search algorithms, or benchmark performance.
- RQ2: What is the impact of code execution on code agents’ performance?: The study varies execution access across four settings while holding three agent scaffolds fixed, enabling measurement of execution’s marginal effect on repair performance.The evaluated agents are Claude Code, Codex CLI, and OpenCode, with experiments spanning restricted to unrestricted execution access.
- RQ1: How do code agents conduct code execution?: 7,745 public traces show execution across all agent-model combinations, averaging 8.8 runs per task with frequencies ranging from 2 to 19.The study examines execution frequency, timing distribution, and outcomes across four agents and twelve LLMs.
- Findings and contributions: 1.25pp is the Prohibited–Unrestricted resolve-rate gap on commercial agents, while the open-source gap is ≈0pp; none is significant at p > 0.05.The paper also investigates conditions behind limited benefits by analyzing stable Pass→Pass and Fail→Fail outcomes.
2 BACKGROUND
Code agents combine persistent LLM reasoning with file exploration, code editing, and execution in iterative software-engineering workflows. Execution provides runtime feedback for patch revision but introduces time, token, and environment costs, motivating fine-grained analysis of when it helps.
- Code agents are LLM-based systems that combine reasoning with tool use in persistent control loops for complex software-engineering tasks.They support tasks including bug fixing, feature implementation, and code refactoring.
- Typical agents combine file exploration, code editing, and code execution to traverse code, apply patches, and run builds, linters, or tests.Representative systems include SWE-agent, OpenHands, Claude Code, and Codex.
- The repair loop inspects code, proposes a patch, runs tests, and revises it, using test outputs as the primary feedback channel.Agentless instead uses localization followed by repair without iterative execution while achieving competitive SWE-bench results.
- Code execution writes and runs a shell command, exposes test outputs or errors, and informs whether the agent revises the patch or submits it.A representative command is pytest tests/test_foo.py, executed in an environment with project dependencies.
- 24–68% cost reductions from limiting interaction turns have minimal impact on solve rates, but turn budgets obscure the higher cost of test execution.The study therefore focuses specifically on when execution helps versus when it adds overhead.
3 EXPERIMENTAL SETUP
The study combines broad trace analysis with a controlled evaluation of code-execution policies in automated program repair. It uses five execution conditions across 200 SWE-bench instances and three agents, measuring behavior, repair effectiveness, and cost.
- Research questions: The study addresses how agents execute code, how execution affects performance, and when execution provides benefits.These correspond to RQ1, RQ2, and RQ3.
- Execution definition: Code execution is defined as running code artifacts that produce runtime feedback, including test frameworks and Python scripts.Exploratory commands such as ls, cat, grep, and find are excluded from execution restrictions.
- Execution paradigms: Five conditions span Prohibited, Quota-Limited K=1, Quota-Limited K=3, Budget-Guided, and Unrestricted execution.Quota-Limited modes use point budgets, while Budget-Guided mode tests whether cost awareness can reduce unnecessary execution.
- Experimental scale: 3,000 end-to-end repair attempts cover 200 SWE-bench instances, three agents, and five execution modes.The agents are Claude Code, Codex, and OpenCode; the instances comprise 100 SWE-bench Lite and 100 SWE-bench Verified cases.
- Study design: The two-stage design analyzes 7,745-scale leaderboard traces for RQ1 and conducts controlled experiments for RQ2–RQ3 across four agents and twelve LLMs.The supplied setup identifies the trace-analysis agents and models, while the controlled study uses fixed Claude Code, Codex, and OpenCode configurations.
- Evaluation: Metrics include execution frequency, timing distribution, execution outcomes, resolve rate, post-execution modification rate, and actionable feedback ratio.Execution timing is divided into Early, Middle, and Late conversation stages; resolve rate is the primary effectiveness outcome.
4 EVALUATION
The evaluation reports an empirical study organized around three research questions: how agents use execution, how execution affects repair effectiveness and cost, and why execution feedback has limited benefits.
- Research questions: The study examines how agents use execution.This is the first research question guiding the evaluation.
- Research questions: The study evaluates execution’s impact on repair effectiveness and cost.This is the second research question guiding the evaluation.
- Research questions: The study investigates why execution feedback has limited benefits.This is the third research question guiding the evaluation.
4.1 RQ1: How Do Code Agents Conduct Code Execution?
Code execution is widely used across coding agents, averaging 8.8 test runs per task but varying substantially by agent and model. Late-stage executions are consistently more successful than early-stage executions, while overall execution success averages 57.9%.
- Execution frequency: 7,745 agent traces show code execution across all analyzed agent-model combinations, averaging 8.8 test executions per task.The traces cover four agents and twelve LLMs.
- Execution frequency: 18.7 versus 2.0 executions per task marks a 9× difference between OpenHands with Claude-4-Sonnet and Mini-SWE-agent with GPT-5.2.Execution frequency ranges from 2 to 19 per task across configurations.
- Execution timing: Late-stage execution is most common in many configurations, whereas SWE-agent with GPT-4 performs 42.4% of executions early and 29.6% late.OpenHands with Qwen3-480B concentrates 55.9% of executions late.
- Execution outcomes: 57.9% is the average execution success rate across configurations, ranging from 30.4% for SWE-agent with GPT-4o to 79.3% for LiveSWEAgent with Claude-Opus-4.5.TestFailure, TestError, and Python exceptions are the most common failure types.
- Execution timing: Late-stage tests outperform early-stage tests across all configurations, including OpenHands with Claude-3.5-Sonnet improving from 42% to 72%.Mini-SWE-agent with GPT-5.2 improves from 25% early to 67% late.
4.2 RQ2: Effectiveness and Cost Analysis
Across three agents and four execution paradigms, restricting execution preserves repair effectiveness while substantially reducing token and wall-clock costs. The benefit of execution is uneven across instances, making unrestricted execution an inefficient default.
- Effectiveness: 1.25pp is the average Prohibited–Unrestricted resolve-rate gap across four commercial-agent cells, and the difference is not statistically significant.Across all six cells including OpenCode, the average difference is −0.83pp.
- Effectiveness: 74.0% is Codex’s Prohibited resolve rate on Lite, exceeding Unrestricted’s 73% and showing no monotonic relationship between execution access and success.Quota-1 and Quota-3 instead reach 68.0% and 69.0%, respectively.
- Execution behavior: 0.36–0.39 completed runs per instance yield a concrete pass/fail signal under Prohibited’s unintended executions, insufficient to explain its resolve rate.Environment errors absorb about a quarter of attempted runs before test output is produced.
- Cost-effectiveness: 21–25% token savings make Codex’s Quota-1 the best cost-effectiveness tradeoff among its execution modes.For Codex, Quota-1 saves 21% on Lite and 25% on Verified relative to Unrestricted.
- Cost-effectiveness: 56–62% token savings and 48–54% wall-clock savings are achieved by Claude Code in Prohibited mode, while resolve-rate sacrifice remains 1–3 percentage points.Prohibited completes Claude Code tasks in 531–573 seconds, compared with 1,028–1,234 seconds for Unrestricted.
4.3 RQ3: When and Why Execution Has Limited Impact
Execution rarely changes repair outcomes: 547 of 600 agent-instance pairs are stable, while execution helps and hurts in roughly equal proportion. Its limited impact reflects little localization benefit, weak corrective feedback, and no consistent advantage on more complex bugs.
- Outcome transitions: 547 of 600 outcome cells are stable, with 24 Prohibited→Unrestricted failures versus 29 Unrestricted→Prohibited failures.The net benefit is 5 cells, indicating that execution helps and hurts in roughly equal proportion.
- Reproduction execution: Over 95% hit rate and over 93% recall are achieved by both commercial agents in Pass→Pass cases under both execution modes.OpenCode reaches 100% hit and 95.8% recall in both modes on its 12 Pass→Pass instances.
- Reproduction execution: 48.8% of Claude Code’s 164 reproduction executions are actionable, leaving 51.2% non-actionable.Non-actionable outputs include generic success messages, timeout errors without stack traces, or test names without source-file cues.
- Validation execution: 54–66% of commercial-agent cases involve a single code edit with no subsequent modifications, regardless of execution availability.This suggests many patches are either correct initially or contain issues that execution feedback does not enable agents to fix.
- Validation execution: 81.2% of Claude Code’s and 100% of Codex’s Fail→Fail cases achieved validation success while still failing official SWE-bench evaluation.For bugs solvable through reasoning alone, execution can confirm an already-correct solution rather than enable its discovery.
- Complexity stratification: Claude Code’s Prohibited–Unrestricted gap changes from −6.5pp for 1 hunk to +23.1pp for ≥4 hunks, while larger complexity does not consistently favor execution.At the largest bucket, Prohibited resolves nearly twice as many instances as Unrestricted; Codex and OpenCode also show non-monotonic patterns.
5 DISCUSSION
The findings motivate cost-sensitive repair agents that strategically restrict execution while maintaining repair effectiveness. Their implications extend to adaptive execution allocation, but conclusions remain scoped to SWE-bench-style repository-level bug fixing and the evaluated agents and instances.
- Practical implications: Restricting execution can preserve comparable resolve rates at significantly lower cost, while Prohibited also eliminates per-repository testbed setup for industrial deployments.This supports cost-sensitive agent design when execution cost matters.
- Broader principle: Agents may benefit more from deeper reasoning than from more frequent environment interaction.Execution feedback can validate correct hypotheses but also trigger unproductive search loops or mislead agents away from correct solutions.
- Threats to validity: 7–9% of Prohibited attempts occasionally run scripts that fail with environment errors under prompt-level budget enforcement.These attempts are counted as executions under an intention-to-treat framework.
- Threats to validity: The zero-execution and env-error-free subsets retain the ±5pp Prohibited–Unrestricted band, while a tool-enforced Claude Code Verified re-run reproduces equivalence at −4pp.These checks support that the result is not an artefact of soft enforcement.
- Threats to validity: The conclusions are scoped to SWE-bench-style repository-level bug fixing with Claude Code, Codex, and open-source OpenCode+Qwen2.5-Coder-32B.Evaluation covered the first 100 instances from each dataset, Lite and Verified, totaling 200 instances across repositories including Django, Flask, Requests, and Sympy.
6 RELATED WORK
Program repair spans search-based, learning-guided, template-based, neural, and code-specialized LLM methods. Modern agentic approaches iteratively refine patches using execution feedback, whereas Agentless follows a fixed pipeline.
- Program repair: Traditional automated program repair includes search-based, learning-guided, template-based, and neural methods.
- Program repair: Code-specialized LLMs underpin modern automated program repair agents.
- Program repair: Agentic repair approaches emphasize iterative refinement with execution feedback.Examples include SWE-agent, AutoCodeRover, ChatRepair, RepairAgent, InspectCoder, and PracAPR.
- Program repair: Agentless uses a fixed pipeline of localization, synthesis, and test selection.
7 CONCLUSION
The paper presents an empirical study of execution behavior in LLM-based program repair, combining large-scale trace analysis with controlled experiments. It examines execution across agents and paradigms, finding that test execution frequency varies widely across agents.
- Study scope: The study analyzes 7,745 agent traces from public SWE-bench submissions and 3,000 end-to-end repair attempts on 200 SWE-bench instances.The controlled experiments cover Claude Code, Codex, and open-source OpenCode+Qwen2.5-Coder-32B under four execution paradigms.
- Study scope: The experiments span three agents: Claude Code, Codex, and open-source OpenCode+Qwen2.5-Coder-32B.These agents are evaluated under four execution paradigms across 200 SWE-bench instances.
- Execution behavior: Test execution frequency varies from 2 to 19 runs per task across agents.The conclusion identifies substantial variation in execution frequency across agents.