PaperlayerView exact revision on arXiv

Source-linked AI summary

REDO: Execution-Free Runtime Error Detection for COding Agents

Shou Li, Andrey Kan, Laurent Callot, Bhavana Bhasker, Muhammad Shihab Rashid, Timothy B Esler

arXiv:2410.09117v1cs.SEcs.AI

TL;DR

Coding agents can produce runtime errors that are difficult for static analysis to detect, a concern as they tackle complex repository tasks. REDO combines static analysis with an LLM for execution-free detection and is reported to outperform prior methods across diverse scenarios. SWEDE evaluates detection on repository-level patches with external dependencies.

  • Problem

    Coding agents produce runtime errors, while static analysis tools can have difficulty detecting them, limiting evidence about execution-free detection for complex repository patches.

    Source: §1 · p. 1, §4 · p. 5

  • Method

    REDO combines differential static analysis with LLM-based detection, and SWEDE evaluates execution-free runtime-error detection on repository-level patches.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    REDO outperforms prior methods across diverse scenarios and achieves state-of-the-art results on all six evaluated STA metrics.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Combining static analysis and LLMs can leverage their distinct strengths across tasks, making the framework more versatile across a broader range of tasks.

    Source: §5.1 · p. 7

    The current LLM-based detection step is limited to one API call per data point.

    Source: §7 · p. 9

Abstract

from arXiv · show

As LLM-based agents exhibit exceptional capabilities in addressing complex problems, there is a growing focus on developing coding agents to tackle increasingly sophisticated tasks. Despite their promising performance, these coding agents often produce programs or modifications that contain runtime errors, which can cause code failures and are difficult for static analysis tools to detect. Enhancing the ability of coding agents to statically identify such errors could significantly improve their overall performance. In this work, we introduce Execution-free Runtime Error Detection for COding Agents (REDO), a method that integrates LLMs with static analysis tools to detect runtime errors for coding agents, without code execution. Additionally, we propose a benchmark task, SWE-Bench-Error-Detection (SWEDE), based on SWE-Bench (lite), to evaluate error detection in repository-level problems with complex external dependencies. Finally, through both quantitative and qualitative analyses across various error detection tasks, we demonstrate that REDO outperforms current state-of-the-art methods by achieving a 11.0% higher accuracy and 9.1% higher weighted F1 score; and provide insights into the advantages of incorporating LLMs for error detection.

1 INTRODUCTION

Coding agents can produce runtime errors that are difficult for static analysis to detect, even as they tackle increasingly complex repository-level tasks. REDO combines static analysis with an LLM, and SWEDE evaluates execution-free detection on repository-level patches.

1 INTRODUCTION

  • On SWE-Bench-lite, six leading agents averaged 1.8 SyntaxErrors, 25.8 TypeErrors, 5.2 NameErrors, and 11.3 AttributeErrors; the authors suggest full SWE-Bench may have more.The lite benchmark is described as deliberately limited to straightforward problems.Source: §1 · p. 1
  • Runtime errors can cause modified code to fail regardless of an agent’s comprehension and reasoning, while early detection can help agents iterate and potentially reduce time and cost.The paper argues runtime-error detection is as critical as comprehension, retrieval, and reasoning.Source: §1 · pp. 1–2
  • REDO achieves state-of-the-art performance across diverse scenarios on accuracy and weighted F1.Figure 1 compares REDO with current state-of-the-art methods across tasks.Source: Figure 1 · p. 2, §1 · p. 2
  • REDO combines static analysis tools with an LLM to detect runtime errors without code execution, and the paper introduces the repository-level SWEDE benchmark.The method is designed to balance reliability with the breadth of detectable errors.Source: §1 · p. 2

2 RELATED WORK

Prior work develops coding agents for repository-level tasks and applies static analysis to code quality, errors, and security. The use of LLMs with static analysis for runtime-error detection remains limited.

2.1 LLM-BASED CODE GENERATION, CODING AGENTS, AND SWE-BENCH

2.2 STATIC ANALYSIS TOOLS AND RUNTIME ERROR PREDICTION

  • Python tools include Pylint and Pyflakes for errors, Bandit for security vulnerabilities, and PyRight and MyPy for type checking.Prior LLM-plus-static-analysis work targets Linux-kernel bugs and security vulnerabilities; runtime-error detection remains less explored, alongside Bieber et al. (2022)’s GNN-based predictor and STA dataset.Source: §2.2 · p. 3

3 EXECUTION-FREE RUNTIME ERROR DETECTION FOR CODING AGENTS

REDO checks modified code in two phases: differential static analysis followed, when needed, by LLM-based detection. The design uses static analysis for dependable detection and LLM reasoning to address input-dependent errors it may miss.

3 EXECUTION-FREE RUNTIME ERROR DETECTION FOR CODING AGENTS

  • REDO checks patch safety through differential analysis followed by LLM-based detection, balancing reliability with the breadth of detectable errors.“Unsafe” patches might crash from runtime errors; “Safe” patches can run successfully.Source: §3 · p. 4

3.1 DIFFERENTIAL ANALYSIS

  • PyRight compares errors in the original and modified implementations so new errors can flag a patch, filtering almost all positives from the original code.Applied to original scripts alone, PyRight falsely classified an average of 267 instances—89% of testing instances—as unsafe; the framework can integrate other static-analysis tools.Source: §3.1 · p. 4
  • Static analysis often misses errors triggered by specific inputs because it cannot reason about runtime-dependent variable types in Python.The method adds LLM detection to address this input-context limitation.Source: §3.1 · p. 4

3.2 LLM-BASED DETECTION

  • REDO sends patches deemed safe by static analysis to an LLM, which receives the problem statement and original functions to reason about missed runtime errors.The prompt asks the LLM to enumerate potential errors caused by the modification and assess patch safety.Source: §3.2 · pp. 4–5
  • The study limits LLM detection to one API invocation per instance because of call cost and latency.It uses Claude-SONNET-3-5 with temperature fixed at zero.Source: §3.2 · p. 5

4 SWE-BENCH-ERROR-DETECTION (SWEDE)

SWEDE adapts SWE-Bench-lite for execution-free detection of runtime errors in generated repository patches. Its external dependencies and indirect tests make the task challenging because relevant code context can be difficult to infer.

4 SWE-BENCH-ERROR-DETECTION (SWEDE)

  • SWEDE extends SWE-Bench-lite’s repository-level problems with leaderboard patches and evaluation logs to detect runtime errors without executing code.The benchmark is based on SWE-Bench-lite and its evaluations using different coding agents.Source: §4 · p. 5
  • Repository dependencies and tests that do not directly interact with modified files make variable and function contexts harder to infer.Table 1 reports average external dependencies in parent folders across coding agents; the passage gives no specific values.Source: §4 · p. 5, Table 1 · p. 5
  • SWEDE labels patches positive when they contain runtime errors and negative when they pass unit tests or fail only because of wrong functionality.The paper evaluates detection using precision, recall, and F-1 score.Source: §4 · p. 5

5 EXPERIMENTS

On SWEDE, REDO and REDO-Pyflakes outperform static-analysis and LLM baselines; on STA, REDO-PyRight achieves state-of-the-art results across six metrics. The experiments also compare the methods’ strengths across tasks and examine REDO’s design factors qualitatively.

5.1 QUANTITATIVE RESULTS

  • The SWEDE evaluation compares REDO on six leaderboard coding agents against static-analysis tools, an LLM-only method, and REDO-Pyflakes; LLM responses are generated three times.The authors report mean and standard deviation because generation is stochastic even at temperature zero.Source: §5.1 · pp. 5–6
  • On SWEDE, REDO and REDO-Pyflakes outperform static-analysis and LLM baselines, with a stronger F1 trade-off and consistent results across LLM configurations.Static analysis usually has higher precision, while the LLM has better recall; REDO uses fewer API calls than the LLM-only method.Source: §5.1 · p. 6, Table 2 · p. 6
  • STA uses about 27,000 CodeNet-derived Python submissions and evaluates runtime-error detection with and without input context, using accuracy and weighted F1 metrics.REDO uses modified prompt templates for STA, and the authors sample error-free submissions with three random seeds for comparison.Source: §5.1 · p. 7
  • Static analysis tools and LLMs excel on distinct tasks across SWEDE and STA, while their integration makes REDO more versatile across tasks.Source: §5.1 · p. 7

5.2 QUALITATIVE ANALYSIS

  • The qualitative analysis examines REDO design factors and includes a figure captioned as a successful example.Source: §5.2 · p. 7, Figure 4 · p. 8

5.1 QUANTITATIVE RESULTS

  • REDO-PyRight achieves state-of-the-art performance across all six evaluated STA metrics; PyRight alone performs well, and REDO adds benefits over PyRight.The authors note discrepancies between annotated error types and their run results, and report bracketed evaluations based on error types identified in their runs.Source: §5.1 · p. 7, Table 3 · p. 8

2. How could the detected errors help?

REDO’s examples show that LLM reasoning can catch runtime errors static analysis misses, while limited input context can leave errors undetected; detected errors can also guide patch repair.

2. How could the detected errors help?

  • The LLM identified an AttributeError that PyRight missed because static analysis could not infer the type of user.The example involves an invalid email attribute accessed on a CustomEmailField instance.Source: §2 · p. 8
  • PyRight and the LLM both missed a query error because the available context made the query’s contents nearly impossible to infer.The example concerns an invalid column name; the authors identify limited input-context inference as a REDO limitation.Source: §2 · pp. 8–9, Figure 5 · p. 9
  • The patch-fixing algorithm uses REDO-detected errors to refine modified code chunks and generate a fixed patch.In a Django-12308 example, it avoids a risky attribute call that could cause an AttributeError.Source: §2 · p. 9

6 CONCLUSION

The paper presents REDO as a two-step runtime-error detection framework, proposes SWEDE for repository-level error detection, and reports quantitative and qualitative evaluation across tasks.

6 CONCLUSION

  • REDO combines differential analysis with LLM-based detection to balance reliability and the scope of detectable errors.The conclusion describes the two-step framework as the paper’s central method.Source: §6 · p. 9
  • SWEDE is proposed as a challenging runtime-error detection task for repository-level modifications by autonomous coding agents.Source: §6 · p. 9
  • The paper evaluates REDO quantitatively across tasks and qualitatively examines when LLM integration helps or falls short and how detected errors could help fix flawed patches.Source: §6 · p. 9

7 LIMITATION AND FUTURE WORK

The authors identify constraints in REDO’s current detection setup and SWEDE’s label scheme, and suggest directions for extending both detection and patch fixing.

7 LIMITATION AND FUTURE WORK

  • The current LLM-based detection step makes only one API call per data point; the authors suggest more calls, potentially using agentic AI, could improve detection.Source: §7 · p. 9
  • SWEDE currently classifies patches only as Safe or Unsafe, and the authors call for a wider range of error types.They point to error types addressed in STA as a comparison.Source: §7 · pp. 9–10
  • The authors identify more refined ensemble and patch-fixing algorithms as potential ways to improve REDO’s efficacy in supporting coding agents.Source: §7 · p. 10

ETHICS STATEMENT.

The authors report relying on prior methods, public sources, and SWE-Bench materials, and state that they identify no risks in the listed ethics areas.

ETHICS STATEMENT.

REPRODUCIBILITY STATEMENT.

The appendix documents REDO’s implementation and evaluation materials, ensemble results, and task-specific prompts for error detection and patch fixing.

REPRODUCIBILITY STATEMENT.

  • Experiments use three random seeds and report means and standard deviations, while the authors say some code files were not ready before submission.They offered to provide the remaining files during rebuttal.Source: REPRODUCIBILITY STATEMENT · p. 10

A REDO PSEUDOCODE

  • REDO compares static-analysis findings before and after applying a patch, marking changed findings unsafe and otherwise asking an LLM to assess the problem, original script, and patch.The pseudocode reverts the applied patch after static analysis and returns safe only if the LLM detects no runtime errors.Source: A REDO PSEUDOCODE · p. 13

C ADDITIONAL QUANTITATIVE RESULTS

  • Appendix tables cover performance metrics for Claude-3 OPUS, Claude-3 SONNET at temperature 0.5, and PyRight, LLM, and REDO on STA.The PyRight, LLM, and REDO table reports means and standard deviations.Source: Table 4 · p. 13, Table 5 · p. 15, Table 6 · p. 15

D ENSEMBLE PATCHES

  • Ensembling yields positive enhancements in most settings, averaging 3.1 for runtime-error instances and 4.6 when all instances are considered.Enhancements are especially significant when a base agent is paired with a more powerful auxiliary agent; CodeStory as the base agent instead shows negative enhancements.Source: D ENSEMBLE PATCHES · p. 14

E.1 PROMPT TEMPLATE: SYSTEM PROMPT

  • The system prompt casts the LLM as an execution-free runtime-error analyzer, while the code-editing prompt supplies the issue, original implementation, and proposed patch for a Safe-or-Unsafe judgment.The code-editing prompt asks for error types and reasoning when potential errors are found.Source: §E.1 · p. 15, §E.2 · pp. 15–17

E.3 PROMPT TEMPLATE: ERROR DETECTION ON CODENET WITH INPUT CONTEXT

  • The CodeNet prompts ask the LLM to assess runtime errors under valid, common inputs, with one variant supplying an input description and the other omitting it.Both variants include an implementation and a list of possible error classes.Source: §E.3 · p. 17, §E.4 · p. 18

E.5 PROMPT TEMPLATE: ERROR DETECTION ON SWE-BENCH-LITE

  • The SWE-Bench-lite prompt provides the problem statement, original implementation, and patch, then asks the LLM to identify runtime errors missed by static analysis and prune errors unlikely to matter to the issue.The remaining relevant errors are requested in a separate section.Source: §E.5 · p. 19

E.6 PROMPT TEMPLATE: PATCH FIXING

  • The patch-fixing prompt gives the detected errors and relevant code chunks, requests a smoke test and full rewritten chunks, and asks for a Complete or Incomplete outcome.Its example flags a missing geometric_mean import and marks the patch incomplete.Source: §E.6 · pp. 19–20

F ADDITIONAL QUALITATIVE RESULTS

The examples show REDO identifying runtime failures that static analysis misses, but also flagging an untriggered corner case and leaving possible later errors unresolved after patch fixing.

F.1 ANOTHER SUCCESSFUL EXAMPLE

  • On Django-10924, the LLM identifies a runtime-only failure caused when a non-callable path value reaches the init function, which static analysis misses.The example uses a Lingma-generated patch.Source: §F.1 · pp. 21–22, Figure 10 · p. 24

F.2 ANOTHER FAILED EXAMPLE

  • On Sympy-13471, the LLM flags a non-string num corner case as an error even though the instance passes its unit test because that case is not triggered.This is presented as a false-positive alarm.Source: §F.2 · p. 22, Figure 11 · p. 25

G PATCH FIXING

  • In the patch-fixing example, REDO identifies an AttributeError involving JSONField.prepare_value, and the corrected patch avoids JSONField and mitigates that crash.The evaluation log confirms the AttributeError caused a crash.Source: G PATCH FIXING · p. 22, Figure 12 · p. 26, Figure 13 · p. 27
  • Fixing the identified AttributeError may still leave AssertionError or other runtime errors, so further iterations may be needed to eliminate them.REDO is stated to be unable to detect the AssertionError in this example.Source: G PATCH FIXING · p. 22
Loading 2410.09117v1…