PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Agent-based Program Repair at Google

Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, Satish Chandra

arXiv:2501.07531v1cs.SEcs.AI

TL;DR

The paper asks whether agent-based program repair, largely evaluated on open-source bugs, is viable for enterprise software. It curates GITS-Eval and evaluates Passerine in Google's development environment, finding higher repair rates for machine-reported than human-reported bugs and distinct distributions from SWE-Bench. These results establish an industrially relevant baseline while leaving broader generalization and method improvements open.

  • Problem

    Agent-based repair has been studied mainly on open-source SWE-Bench bugs, so its viability on diverse enterprise bugs remains unclear.

  • Method

    The paper curates a 178-bug GITS-Eval benchmark and evaluates Passerine, a SWE-Agent-inspired repair agent adapted to Google's internal development environment.

  • Results

    Passerine produced plausible patches for 73% of machine-reported and 25.6% of human-reported bugs, with valid patches for 43% and 17.9%, respectively.

  • Takeaways & Limitations

    GITS-Eval establishes a baseline for agent-based repair in an industrially relevant setting whose bugs differ distributionally from SWE-Bench.

  • Takeaways & Limitations

    The evaluation focuses on Google's internal repository, while other industrial settings may contain bugs with different characteristics.

Abstract

from arXiv · show

Agent-based program repair offers to automatically resolve complex bugs end-to-end by combining the planning, tool use, and code generation abilities of modern LLMs. Recent work has explored the use of agent-based repair approaches on the popular open-source SWE-Bench, a collection of bugs from highly-rated GitHub Python projects. In addition, various agentic approaches such as SWE-Agent have been proposed to solve bugs in this benchmark. This paper explores the viability of using an agentic approach to address bugs in an enterprise context. To investigate this, we curate an evaluation set of 178 bugs drawn from Google's issue tracking system. This dataset spans both human-reported (78) and machine-reported bugs (100). To establish a repair performance baseline on this benchmark, we implement Passerine, an agent similar in spirit to SWE-Agent that can work within Google's development environment. We show that with 20 trajectory samples and Gemini 1.5 Pro, Passerine can produce a patch that passes bug tests (i.e., plausible) for 73% of machine-reported and 25.6% of human-reported bugs in our evaluation set. After manual examination, we found that 43% of machine-reported bugs and 17.9% of human-reported bugs have at least one patch that is semantically equivalent to the ground-truth patch. These results establish a baseline on an industrially relevant benchmark, which as we show, contains bugs drawn from a different distribution -- in terms of language diversity, size, and spread of changes, etc. -- compared to those in the popular SWE-Bench dataset.

I. INTRODUCTION

This paper examines whether agent-based program repair can transfer from open-source benchmarks to Google's diverse enterprise software environment. It curates GITS-Eval and establishes a Passerine baseline, finding substantially different bug distributions and stronger results on machine-reported than human-reported bugs.

  • Motivation: Agent-based repair has primarily been evaluated on open-source SWE-Bench bugs, leaving its effectiveness in broader enterprise settings uncertain.SWE-Bench contains 2,294 Python bugs and fixes from popular GitHub repositories.
  • Benchmark: 178 bugs comprise GITS-Eval: 78 human-reported and 100 machine-reported, including 50 SAN and 50 TOD bugs.The set spans multiple projects and programming languages while remaining tractable for automated repair.
  • Benchmark: GITS bugs differ from SWE-Bench in language diversity, change size and spread, and code-term prevalence in bug descriptions.These differences mean performance on one benchmark may not indicate performance on the other.
  • Results: With 20 trajectories per bug, Passerine generated plausible patches for 25.6% of human-reported and 73% of machine-reported bugs.Machine-reported results were 68% for TOD bugs and 78% for SAN bugs.
  • Results: Manual annotation found semantically ground-truth-equivalent patches for 17.9% of human-reported and 43% of machine-reported bugs.The corresponding machine-reported rates were 24% for TOD and 62% for SAN.
  • Agent behavior: Passerine adapts its command sequences to bug type and can exploit rich machine-reported bug reports containing reproduction information.Trajectory analysis also reveals opportunities such as pruning degenerate trajectories.

A. Phase 0: Fixed Bugs Population

Phase 0 establishes a broad, diverse population of fixed GITS bugs, while subsequent selection emphasizes bugs that are assessable by agentic repair and separates human- from machine-reported cases.

  • Phase 0: Fixed Bugs Population: Phase 0 applies minimal filters to define a broad population of accessible fixed bugs associated clearly with their fixes.The resulting population is large and diverse within a specified timeframe.
  • Phase 1: Assessable Bugs: Phase 1 selects bugs that agentic APR could conceptually address and whose plausible fixes can be assessed through testable code changes.The criteria distinguish human-reported from machine-reported bugs based on report nature and available fix information.
  • Phase 1: Assessable Bugs: Machine-reported candidates include SAN bugs involving memory or thread errors and TOD bugs involving test-order dependence.SAN can report out-of-bounds accesses, uninitialized values, and data races.
  • Phase 2: Practical Evaluation Filters: Phase 2 excludes evaluation challenges such as long-running integration tests and bugs requiring multimodal understanding.Patch size is limited to fewer than 150 lines, corresponding to the 90th percentile of internal bug-fix patch sizes.

D. Phase 3: Heuristic Curation

Phase 3 validates reproducibility and applies human heuristic curation before finalizing GITS-Eval, then compares the resulting GITS bug distribution with SWE-Bench.

  • Phase 3: Heuristic Curation: Phase 3 executes associated tests to verify failure before the ground-truth patch and resolution after applying it.Flaky bugs are removed to support consistent reproducibility.
  • Phase 3: Heuristic Curation: Human review filters bugs unsuitable for automatic execution-based evaluation, including fixes involving non-derivable magic constants.At least one author labels each data point for magic-constant changes.
  • Final Evaluation Set: The final GITS-Eval set contains 178 bugs: 78 human-reported and 100 machine-reported, split evenly between SAN and TOD reports.The multi-stage process is intended to preserve representativeness while ensuring evaluation suitability.
  • GITS vs SWE-Bench: The GITS-versus-SWE-Bench comparison samples 2,000 Phase 1 GITS bugs to measure intrinsic distribution differences without later curation confounds.The comparison targets dimensions related to localization and editing difficulty.

1) Localization:

GITS bugs differ from SWE-Bench in localization and editing difficulty: their descriptions offer fewer code-search terms, while fixes are more dispersed, multi-file, multi-hunk, and larger.

  • Localization:: Only 18% of GITS bugs have at least 2 possible code terms, compared to approximately 60% in SWE-Bench.The paper treats likely identifier terms as a proxy for searchability during fault localization.
  • Localization:: GITS patches modify up to twice as many files as SWE-Bench patches and are more widely separated within files.Patch spread measures the number of unmodified lines between consecutive changes.
  • Editing:: GITS patches produce much larger hunk counts than SWE-Bench patches, indicating more fragmented changes.The comparison evaluates hunks formed by segmenting patch changes.
  • Editing:: Approximately 40% of GITS patches are under 100 lines, whereas almost all SWE-Bench patches are under 100 lines.The paper measures changed lines in ground-truth patches as an editing-complexity indicator.
  • Editing:: GITS patches span Java, C++, TypeScript, Kotlin, and Python, whereas SWE-Bench considers only Python files.This language distribution adds syntax and semantic variation to the repair setting.
  • Editing:: Overall, GITS differs from SWE-Bench in code-related terms, modified files, changed lines, and patch dispersion.These differences concern both localization and code modification.

F. GITS-Eval vs. SWE-Bench-Lite

GITS-Eval is a curated 178-bug benchmark for enterprise agentic repair, and its human- versus machine-reported subsets differ in complexity relative to SWE-Bench-Lite.

  • F. GITS-Eval vs. SWE-Bench-Lite: GITS-Eval is a curated subset of 178 bugs used to evaluate an agentic repair system, analogous to SWE-Bench-Lite in the Google context.The benchmark categorizes bugs as human-reported or machine-reported.
  • F. GITS-Eval vs. SWE-Bench-Lite: Machine-reported GITS-Eval bugs are comparable to SWE-Bench-Lite, while human-reported bugs display increased complexity.The human-reported subset resembles the earlier GITS-versus-SWE-Bench differences.
  • Agent design: Passerine uses a ReAct loop to produce thoughts, execute commands, observe results, and modify the workspace.Its tools include file operations, testing, compilation, and Google monorepo search.
  • Agent design: Passerine uses a small APR-focused command set rather than a full Linux command line or explicitly separated repair phases.The design relies on Google’s existing test-running infrastructure and dynamic agent behavior.
  • Agent design: Passerine includes the entire agent history in each LLM query, but some runs can exceed context limits.The paper identifies more sophisticated history management as a potential improvement.

B. Commands

Passerine exposes a compact, uniform command interface that lets the agent inspect, search, edit, test, and terminate execution through Google-internal APIs.

  • B. Commands: Passerine provides only 5 commands and 1 commonly referenced alias for interacting with Google internal APIs.The limited interface avoids requiring a full virtual machine for isolation and improves scalability.
  • B. Commands: Every command accepts positional arguments and optional text fragments, then returns an exit code and observable output text.This uniform interface makes command results available through the agent’s history.
  • B. Commands: The cat command displays workspace-file contents with line numbers, while code search queries Google’s internal Code Search API.These commands support inspection and localization.
  • B. Commands: The edit command replaces a specified file-line range and returns modified content with surrounding context and resulting line numbers.Edits operate through an internal API connected to the isolated codebase container.
  • B. Commands: The bazel command runs a target’s test suite and exposes parsed outcomes and relevant log excerpts to the agent.The finish command terminates execution with a success or failure argument.

C. Evaluation Framework for Historic Bugs

The evaluation framework reconstructs historical bug states, runs Passerine without future-state leakage, and verifies its final repository changes against bug tests.

  • C. Evaluation Framework for Historic Bugs: The framework supports evaluating agent configurations on Google infrastructure using historical bugs that have already been fixed.It also records structured execution logs and reproduction information.
  • C. Evaluation Framework for Historic Bugs: Before execution, the framework loads bug information, checks out the pre-fix repository state, and reproduces the bug to validate setup.Expected failing and passing test targets are specified for benchmark tasks.
  • C. Evaluation Framework for Historic Bugs: After confirming the bug, the framework reverts the repository to contain only information available before the ground-truth fix.This prevents the evaluation setup from retaining post-fix state.
  • C. Evaluation Framework for Historic Bugs: During execution, structured logs record commands, outputs, and files or lines entering the agent history, while command execution prevents future-state data leakage.These records support offline analysis and localization analysis.
  • C. Evaluation Framework for Historic Bugs: After termination or step-budget exhaustion, the framework restores necessary test files and runs reproduction targets to verify the agent’s changes.Agent termination can occur through a finish command or budget exhaustion.

IV. EVALUATING AGENT-GENERATED REPAIRS

The evaluation measures Passerine’s patch correctness and trajectory behavior on 178 Google bugs, using 20 sampled repair trajectories per bug. Passerine produces plausible and manually validated patches for both machine-reported and human-reported bugs.

  • Patch correctness: Plausible patches are those that pass bug-reproducing tests, whereas valid patches are semantically equivalent to the ground-truth fix.Validity is assessed manually, with evaluation focused on bug-fixing logic rather than regression-test generation.
  • Evaluation setup: The evaluation covers 178 GITS-Eval bugs: 50 TOD, 50 SAN, and 78 human-reported bugs.
  • Patch correctness: Plausibility and validity diverge by bug type: the gap is largest for TOD bugs and small for human-reported bugs.The paper attributes these differences to the testing criteria used for each bug type.
  • Patch correctness: 20 sampled trajectories per bug yield at least one plausible patch for 68% of TOD, 78% of SAN, and 25.6% of human-reported bugs.
  • Patch correctness: Manual annotation finds at least one valid patch for 24% of TOD, 62% of SAN, and 17.9% of human-reported bugs.
  • Overall finding: Passerine produces plausible and valid patches for both human-reported and machine-reported bugs in Google’s enterprise-scale setting.

B. Observations

Passerine adapts its commands and localization behavior to bug type and available report information. Trajectory analysis also identifies failure patterns that could support optimization and reviewer-burden reduction.

  • Agent strategies: Passerine adopts different command strategies for different bug reports despite having no high-level strategy restrictions.
  • Agent strategies: Human-bug trajectories typically begin with code search and file reading, whereas machine-bug trajectories often begin by running Bazel tests before editing.
  • Agent strategies: SAN trajectories often begin with an invalid unnecessary command caused by report assumptions about broader Google tooling, but Passerine recovers by step 2.
  • Localization: Among trajectories without plausible patches, 53.8% of machine-reported cases edit the correct file, compared with 3.5% of human-reported cases.
  • Localization: Richer machine-generated reports support better file-level localization and higher fix rates than less informative human-reported bugs.
  • Trajectory analysis: Trajectory analysis identifies no-testing and redundant-file-reading patterns, with the latter correlating with failures and suggesting pruning opportunities.

VI. DISCUSSION

The discussion identifies practical constraints in Passerine’s context handling, tools, localization, and sampling strategy. It also connects richer bug reports and trajectory analysis to possible future improvements.

  • Challenges and mitigations: Verbose outputs and append-only histories can push Passerine’s prompts beyond even large context windows such as 2M tokens.
  • Challenges and mitigations: Human-reported bugs provide reproduction information less often than machine-reported bugs, making bug exposure and subsequent localization/editing more difficult.
  • Challenges and mitigations: Passerine may attempt unavailable commands or linked documents, especially on human bugs, because its current tool set lacks those capabilities.
  • Challenges and mitigations: Fault localization is challenging for human-reported bugs, and Passerine currently neither explicitly tasks localization nor exploits rich development histories.
  • Challenges and mitigations: Passerine samples trajectories independently, leaving guided sampling or beam search as possible ways to improve patch diversity and performance.

B. Deployment Considerations

Deployment beyond the evaluated setting is constrained by Passerine’s limited repair reach and by the scope and manual construction of GITS-Eval. The authors frame Passerine as an industrial proof of concept rather than a validated solution for other benchmarks or environments.

  • Deployment scope: Passerine may encounter bugs beyond its repair capabilities, making deployment-time abstention important for avoiding unnecessary compute.The authors explore predicting when Passerine is unlikely to fix a bug, but applying dataset filters to unsolved bugs requires auxiliary prediction tasks and prior knowledge.
  • Deployment scope: GITS-Eval focuses on bugs from Google’s internal repository, so other industrial settings may exhibit different characteristics.The authors note that Google’s codebase may share properties with other large software providers, but this remains a scope boundary.
  • Evidence boundaries: Passerine is presented as a proof-of-concept for agent-based APR in an industrial setting, not as evidence of performance on other benchmarks.The paper does not claim how Passerine would perform on benchmarks such as SWE-Bench.
  • Evidence boundaries: Patch validity was assessed manually, and the authors identify manual assessment as subject to standard evaluation challenges.Multiple annotators were used for ambiguous cases and to mitigate risks from manually assessed dataset properties.

VIII. RELATED WORK

Related work spans traditional and agent-based APR systems, while Passerine targets program repair inside Google’s development environment. Its distinct contribution is an industrial proof-of-concept and systematic study using Google-specific tools and an internal benchmark.

  • Agent-based APR: SWE-Agent directly inspired Passerine’s ReAct-style loop and bash-like tool access, while other systems constrain actions or expose program-structure tools.RepairAgent uses a state machine, whereas AutoCodeRover uses tools such as class- and method-based definition search.
  • Agent-based APR: Passerine is an agent-based APR framework focused exclusively on program repair and uses ReAct-style tools with test-suite-based evaluation.Its current implementation exposes a subset of SWE-Agent’s commands.
  • Industrial setting: Passerine differs from prior systems by using custom tools for Google’s development environment and studying repair in a large industrial codebase.The paper describes this as the first systematic study of an agent-based APR system in such a codebase.
  • Non-agent APR: Non-agent APR work includes zero-shot LLM infilling and fixed repair pipelines with fault localization, patch generation, filtering, and ranking.Passerine instead uses an agent-based approach tailored to Google’s environment and does not rely on a repair-fine-tuned model or meaningful few-shot examples.
  • Positioning: The broader study contributes a Google-internal benchmark and reports plausible and valid repair rates for human- and machine-reported bugs.With 20 trajectories, Passerine produced plausible patches for 73% and 25.6% of machine- and human-reported bugs, while manual inspection found valid patches for 43% and 17.9%.
Loading 2501.07531v1…