PaperlayerView exact revision on arXiv

Source-linked AI summary

SpecRover: Code Intent Extraction via LLMs

Haifeng Ruan, Yuntong Zhang, Abhik Roychoudhury

arXiv:2408.02232v4cs.SEcs.AI

TL;DR

Program improvement needs specifications of intended behavior, yet developer intent is often not formally specified at the component level. SpecRover iteratively infers intent through code search and review; it reports solving 19.3% of full SWE-bench issues and 31% of SWE-bench lite issues.

Abstract

from arXiv · show

Autonomous program improvement typically involves automatically producing bug fixes and feature additions. Such program improvement can be accomplished by a combination of large language model (LLM) and program analysis capabilities, in the form of an LLM agent. Since program repair or program improvement typically requires a specification of intended behavior - specification inference can be useful for producing high quality program patches. In this work, we examine efficient and low-cost workflows for iterative specification inference within an LLM agent. Given a GitHub issue to be resolved in a software project, our goal is to conduct iterative code search accompanied by specification inference - thereby inferring intent from both the project structure and behavior. The intent thus captured is examined by a reviewer agent with the goal of vetting the patches as well as providing a measure of confidence in the vetted patches. Our approach SpecRover (AutoCodeRover-v2) is built on the open-source LLM agent AutoCodeRover. In an evaluation on the full SWE-Bench consisting of 2294 GitHub issues, it shows more than 50% improvement in efficacy over AutoCodeRover. Compared to the open-source agents available, our work shows modest cost ($0.65 per issue) in resolving an average GitHub issue in SWE-Bench lite. The production of explanation by SpecRover allows for a better "signal" to be given to the developer, on when the suggested patches can be accepted with confidence. SpecRover also seeks to demonstrate the continued importance of specification inference in automated program repair, even as program repair technologies enter the LLM era.

I. INTRODUCTION

Autonomous program improvement seeks to fix bugs and add features, but component-level repair needs intent specifications that are often unavailable. SpecRover iteratively infers specifications during code search and uses reviewer feedback to vet patches; the paper reports results on SWE-bench and SWE-bench lite.

Motivation

  • Program repair needs developer intent at the component level, but intent is usually stated only for the whole system and may lack an accompanying test suite.Symbolic test-based approaches can address overfitting, but have a high entry barrier for developers.Source: §I · p. 1

Approach

  • SpecRover builds on AutoCodeRover, adding function summaries and test cases as specification sources that are distilled into a patch.Its code search captures intended behavior as it visits classes and methods, then sends specifications and generated tests to a reviewer agent.Source: §I · pp. 1–2

Specification inference

  • Iterative specification inference is proposed as a way to guide patching, with patch construction treated as a by-product of understanding developer intent.The paper describes this as a contribution to LLM-guided autonomous software engineering.Source: §I · p. 2

Patch review

  • A reviewer agent reconciles specifications, tests, and natural-language requirements, and can provide explanations and correctness evidence alongside patches.The evidence may be retained with patches to help track future regressions.Source: §I · p. 2

Experimental evidence

  • 19.3% of full SWE-bench issues and 31% of SWE-bench lite issues were solved by SpecRover.The reported cost was $0.65 per issue on SWE-bench lite.Source: §I · p. 2

II. MOTIVATING EXAMPLE

A scikit-learn issue illustrates how inferred method intent and reviewer feedback can expose and correct an incomplete patch. SpecRover’s broader workflow iteratively retrieves code, infers specifications, patches, and reviews candidates.

II. MOTIVATING EXAMPLE

  • In the scikit-learn example, check_array crashes when a non-numeric string is converted to float, and SpecRover infers intended behavior for the method.The inferred behavior serves as a specification for modifying the method.Source: §II · p. 2
  • The reviewer identifies that the initial patch handles only one of two failing string conversions, prompting a corrected patch that catches the other exception too.It compares the patch and reproducer-test behavior against the error information and issue statement.Source: §II · pp. 2–3

A. Overview

  • SpecRover starts from an issue and codebase, then iteratively infers specifications until a patch is generated and vetted as correct.The workflow presents inferred specifications and LLM agents as distinct parts of the process.Source: A. Overview · p. 3
  • Context retrieval produces issue-relative function summaries; reviewer feedback can send patch and reproducer-test corrections back to their agents.Approved patches are checked against an available regression suite, and a selection agent chooses among candidates after retries.Source: A. Overview · p. 3

B. Function Summary: Specification from Program

SpecRover extracts a concise, issue-relative specification for each newly retrieved code snippet during context retrieval. These function summaries localize repository-level issue solving into smaller function-level modification tasks.

Context retrieval

  • Context retrieval gathers code snippets through APIs and stops when the agent judges the context sufficient or a retrieval threshold is reached.The selected buggy locations are sent to the patching agent.Source: B. Function Summary: Specification… · pp. 3–4

Function summaries

  • Function summaries describe how a function should behave to meet the issue’s requirements and serve as local specifications for patch construction.The agent is prompted to infer intended behavior whenever a new snippet is retrieved.Source: B. Function Summary: Specification… · p. 4
  • The summaries decompose repository-level issue solving into function-level tasks, which the authors describe as smaller and more manageable for the patching agent.The paper relates this decomposition to promising LLM performance on function-level coding benchmarks.Source: B. Function Summary: Specification… · p. 4

C. Reviewer Feedback: Reconciling Specifications

SpecRover’s reviewer provides decisions and explanations about patches and reproducer tests, then feeds feedback back into the agents. The review considers both execution evidence and the issue statement because tests can be incorrect or incomplete.

Reviewer feedback

  • Reviewer feedback gives binary correctness decisions and explanations for both the patch and reproducer test.It is a meta-specification reflecting on specifications inferred in earlier steps.Source: C. Reviewer Feedback: Reconciling… · p. 4

Reconciling specifications

  • Feedback is returned to the patching and reproducer agents, while jointly considering the patch and test can reveal errors not apparent when examining either alone.The issue-specific test suite is usually unavailable when an issue occurs despite passing regression tests.Source: C. Reviewer Feedback: Reconciling… · pp. 4–5
  • A reproducer test may be incorrect or may incompletely express intent, so SpecRover’s reviewer also considers the natural-language issue statement.A patch may fail a correct implementation through a faulty test, or pass a test without fully resolving the issue.Source: C. Reviewer Feedback: Reconciling… · p. 5

Reviewer feedback

  • Reviewer explanations can help patch agents rectify incorrect patches and help users understand and accept correct ones.The feedback may be retained with the patch for future software maintenance.Source: C. Reviewer Feedback: Reconciling… · p. 5

D. Patch Selection

SpecRover combines test-based review with natural-language patch selection to handle cases where regression tests reject a correct fix. It also produces evidence artifacts intended to help developers assess and maintain generated patches.

D. Patch Selection

  • When regression tests may reject a correct patch, a selection agent compares rejected candidates with the issue description and can recover a promising patch.This addresses cases where resolving an issue necessarily changes existing behavior, such as changing a function signature.Source: D. Patch Selection · p. 5

E. Evidence

  • SpecRover can output buggy locations and intended behaviors, reproducer tests, and reasons for patch approval or selection.Approval or selection reasons can become part of a commit message, while reproducer tests can be integrated into the project test suite.Source: E. Evidence · pp. 5–6
  • These natural-language artifacts can help developers understand generated patches and can evolve with the software repository.The authors also describe the artifacts as potentially guiding LLM agents toward higher-quality patches.Source: E. Evidence · pp. 5–6

IV. EXPERIMENTAL SETUP

The experiments assess issue-resolution efficacy, developer confidence, and specification quality, using SWE-bench and comparisons with open-source systems. The setup also specifies evaluation procedures, agent implementation choices, and manual checks.

Research Questions

  • The evaluation asks about issue-resolution efficacy, developer confidence in patches and specifications, and the quality of specifications as evidence..Source: §IV · p. 6

Benchmark

  • SpecRover is evaluated on SWE-bench, a benchmark of 2,294 real-world GitHub issues, using only each issue statement and buggy codebase as input.The regression suite belongs to the buggy program; code and tests added in the developer-fixed version are not accessed.Source: §IV · p. 6

Baselines and Evaluation Metrics

  • For efficacy comparisons, the study includes open-source repository-level issue-solving systems that reported SWE-bench results.The baselines described include Agentless, SWE-agent, AppMap Navie, OpenDevin, Aider, and Moatless Tools.Source: §IV · p. 6
  • An issue counts as resolved when the tool's single generated patch passes the SWE-bench acceptance tests, which are not used during patch generation.The study reports pass@1 efficacy.Source: §IV · p. 6

Implementation and Parameters

  • SpecRover reuses AutoCodeRover's context-retrieval APIs and adds function-summary extraction, patch review, and patch selection agents.Experiments primarily use Claude 3.5 Sonnet, switching to GPT-4o for a task only when a Claude API error occurs; regression-test retries are capped at three.Source: §IV · p. 6

Randomness of LLMs and Manual Inspection

  • The study sets model temperature to zero to make output more deterministic and has two authors independently inspect results before cross-checking them.Disagreements were resolved in consultation with a third colleague.Source: §IV · p. 6

V. EVALUATION

SpecRover achieves the highest reported efficacy among the compared open-source tools on both SWE-bench datasets. The evaluation also examines cost, patch correctness, data memorization, and unresolved-task failures.

A. RQ1: Overall Efficacy of Task Resolving

  • 31% on SWE-bench Lite versus approximately 26%–27% for the previous top-performing group, and 19.31% versus 14.60% on full SWE-bench; SpecRover leads open-source tools on both.Table I compares efficacy, efficiency, and cost on both datasets.Source: A. RQ1: Overall Efficacy of Task… · p. 7, Table I · p. 7
  • 9 Lite patches were syntactically identical to ground-truth patches, accounting for 10% of the 93 resolved issues.The authors report that data memorization occurs very infrequently for SpecRover.Source: A. RQ1: Overall Efficacy of Task… · p. 7
  • $0.65 USD per issue is SpecRover's average SWE-bench Lite patch-generation cost; it spends an average of 309 seconds per issue.For the 93 resolved Lite issues, the reported cost is $0.36 USD per issue to generate the correct patch.Source: A. RQ1: Overall Efficacy of Task… · p. 7
  • 60.2% (56/93) of test-passing patches were semantically equivalent to the ground-truth patches; over 90% had either equivalence or the correct fix location.Of the 37 overfitting patches, 29 modified the same methods as the ground-truth patch.Source: A. RQ1: Overall Efficacy of Task… · p. 7
  • 207 SWE-bench Lite tasks were unresolved, and the paper examines these failures in three categories.Source: A. RQ1: Overall Efficacy of Task… · p. 7
  • Among unresolved Lite tasks, 61 (29.5%) involved incorrect fix locations and 39 (18.8%) involved incorrect code modifications.For 26 of the 39 incorrect-modification cases, the authors observed that the patch's gist was correct but a code detail was wrong.Source: A. RQ1: Overall Efficacy of Task… · p. 8

B. RQ2: Utility of autonomous SE, confidence in results

This section evaluates whether SpecRover’s reviewer can reduce the effort of assessing generated patches through correctness decisions and supporting evidence. Its results report reviewer accuracy, recall, and patch precision, including comparisons with other tools.

B. RQ2: Utility of autonomous SE, confidence in results

  • SpecRover frames reviewer validation and patch evidence as ways to reduce user effort, including the number of incorrect patches users examine.Users can examine a generated patch when the reviewer deems both the patch and reproducer test correct.Source: B. RQ2: Utility of autonomous SE,… · p. 8
  • Among 119 SWE-bench Lite tasks with reproducer tests, reviewer decisions had 64.7% accuracy, with 26 true positives and 51 true negatives; recall exceeded 60%.The table reports reviewer-decision scenarios and accuracy, precision, and recall.Source: B. RQ2: Utility of autonomous SE,… · p. 8, Table II · p. 8
  • SpecRover’s patch precision is 50.0% and more than 1.8x Agentless’s; without regression-test validation, precision falls only slightly to 48.3%.The authors relate higher precision to lower cognitive load for users examining proposed patches.Source: B. RQ2: Utility of autonomous SE,… · p. 8, Figure 6 · p. 8

C. RQ3: Quality of Specifications produced

RQ3 examines the quality of SpecRover’s generated specifications, including function summaries compared with developer-written pull-request titles. In the inspected cases, most summaries covered similar intent.

C. RQ3: Quality of Specifications produced

  • RQ3 investigates whether SpecRover’s generated specifications provide evidence that can help developers integrate automatically generated patches.The authors describe high-quality evidence as helping developers integrate patches into an existing codebase.Source: C. RQ3: Quality of Specifications… · p. 8

1) Function Summaries:

  • For 72/101 inspected issues (71%), generated function specifications covered intent similar to the developer-written pull-request title.The comparison included cases where SpecRover patched the same methods as the developer and excluded titles considered too brief.Source: 1) Function Summaries · pp. 8–9

2) Reviewer Feedback:

The reviewer-feedback section illustrates how feedback can summarize a patch and assess a patch separately from its reproducer test. Feedback quality is shown through examples because the authors report no ground truth for quantitative evaluation.

2) Reviewer Feedback:

  • The authors could not quantitatively evaluate reviewer-feedback quality because no ground truth was available.They illustrate feedback quality with two examples instead.Source: 2) Reviewer Feedback · p. 9
  • In the pytest example, reviewer feedback summarized a patch that moved line-number calculation to a branch unaffected by the runxfail option.The patch changed 51 lines, mostly indentation; the feedback was intended to help developers understand it before examining its details.Source: 2) Reviewer Feedback · pp. 9–10
  • In the SymPy example, the reviewer rejected an incorrect reproducer test while approving the correct patch.The test compared an object with a string, but the reviewer assessed the test and patch separately.Source: 2) Reviewer Feedback · p. 10

VI. CASE STUDY: SECURITY VULNERABILITY REPAIR

The case study demonstrates SpecRover’s application to a Linux-kernel security vulnerability, extending beyond its initial focus on GitHub issues in Python repositories. From a vulnerability report and PoC, it inferred local intended behavior and generated a patch that the reviewer approved.

VI. CASE STUDY: SECURITY VULNERABILITY REPAIR

  • Although initially designed for GitHub issues in Python repositories, SpecRover can be adapted to other application domains and programming languages.The case study applies it to a C vulnerability challenge from DARPA’s AI Cyber Challenge.Source: §VI · p. 10
  • The Linux-kernel case concerns a TIPC buffer overflow triggered by invalid user-supplied message-body sizes.The challenge includes a PoC input, which triggers the vulnerability report used in the case study.Source: §VI · pp. 10–11
  • From a report containing a call trace and minimal bug description, SpecRover inferred function-level intent and generated a patch adding checks before the dangerous memory operation.The reviewer approved the patch and provided comments to help developers understand it.Source: §VI · p. 11

VII. RELATED WORK

SpecRover extends LLM-based program repair with specification inference from multiple sources and a reviewer designed for natural-language requirements and unreliable tests. The perspectives section emphasizes patch vetting and explanations alongside efficacy, cost, and output quality.

VII. RELATED WORK

  • Automated program repair modifies a buggy program to pass its test suite, using approaches such as search, semantic analysis, machine learning, or combinations.The paper situates LLM agents as combining LLM capabilities with program analysis and test execution reasoning.Source: §VII · p. 11
  • Unlike AutoCodeRover’s structure-only intent inference, SpecRover can infer specifications from program structure, behavior, tests, and other sources, and explains produced patches.The paper positions its technical innovations as specification inference to guide patching and rigorous patch vetting by a reviewer agent.Source: §VII · p. 11, §VIII · p. 11
  • Unlike prior LLM specification-inference work focused on small programs or a supplied target function, SpecRover targets large programs without assuming target functions are known.The comparison covers works focused on one-function or one-class programs and work assuming a target function is provided.Source: §VII · p. 11
  • The reviewer judges patches using natural-language issue descriptions alongside tests and patches, which the authors note can both be unreliable.Unlike approaches relying on interactive user feedback or reviewing generated code alone, it is designed for natural-language instructions and unreliable tests.Source: §VII · p. 11

VIII. PERSPECTIVES

  • The perspectives section calls for patch vetting and sophisticated test generation, with efficacy, cost, and signal-to-noise ratio as developer concerns.The authors identify improving precision and recall as a future need for LLM agents.Source: §VIII · p. 11
  • At paper acceptance, AutoCodeRover-v2 reported 37.3% efficacy on SWE-bench Lite and 46.2% on SWE-bench Verified, and was packaged as a one-click issue-resolution GitHub bot.The authors also share source code and experimental artifacts publicly.Source: DATA AVAILABILITY · p. 11

Related papers

Loading 2408.02232v4…