PaperlayerView exact revision on arXiv

Source-linked AI summary

Critical Review of BugSwarm for Fault Localization and Program Repair

Thomas Durieux, Rui Abreu

arXiv:1905.09375v1cs.SE

TL;DR

APR and FL benchmarks need behavioral bugs, while BugSwarm consists of failing and passing builds that may fail for other reasons. This paper critically reviews BugSwarm against those fields’ requirements and finds that only 112 of its 3,091 pairs are compatible with current approaches.

  • Problem

    BugSwarm is a benchmark of failing and passing builds rather than necessarily behavioral bugs, although APR and FL rely on bugs for evaluation.

    Source: §I · p. 1

  • Method

    The paper critically reviews BugSwarm against APR and FL requirements, characterizing its content and assessing which build pairs meet those requirements.

    Source: §I · p. 1, §VI · p. 10

  • Results

    Only 112/3,091 (3.6%) BugSwarm pairs are compatible with current APR and FL approaches.

    Source: §VI · p. 10

  • Takeaways & Limitations

    BugSwarm’s name can suggest a bug benchmark, but it is conceptually a benchmark of build pairs, so suitability depends on examining its contents.

    Source: A. Failing builds vs. bugs · p. 8, §VI · p. 10

    BugSwarm may be updated, so observed results can differ in the future.

    Source: D. Threats to Validity · p. 9

Abstract

from arXiv · show

Benchmarks play an important role in evaluating the efficiency and effectiveness of solutions to automate several phases of the software development lifecycle. Moreover, if well designed, they also serve us well as an important artifact to compare different approaches amongst themselves. BugSwarm is a benchmark that has been recently published, which contains 3,091 pairs of failing and passing continuous integration builds. According to the authors, the benchmark has been designed with the automatic program repair and fault localization communities in mind. Given that a benchmark targeting these communities ought to have several characteristics (e.g., a buggy statement needs to be present), we have dissected the benchmark to fully understand whether the benchmark suits these communities well. Our critical analysis has found several limitations in the benchmark: only 112/3,091 (3.6%) are suitable to evaluate techniques for automatic fault localization or program repair.

I. INTRODUCTION

Benchmarks provide evidence for evaluating automated fault localization and program repair, but BugSwarm’s failing and passing build pairs do not necessarily represent behavioral bugs. This review characterizes BugSwarm and finds that only 112 of 3,091 pairs meet the stated requirements for those fields.

I. INTRODUCTION

  • As the number of APR and FL benchmarks grows, understanding their characteristics helps researchers assess evaluation quality and choose benchmarks suited to their needs.The paper situates BugSwarm among four recently presented benchmarks and several existing ones.Source: §I · p. 1
  • The review characterizes BugSwarm’s human patches, failures, and usage, then assesses its applicability to state-of-the-art APR and FL research.It aims to guide researchers on using BugSwarm and preventing misuse or incorrect recommendations.Source: §I · p. 1
  • Only 112/3,091 (3.6%) BugSwarm pairs meet the requirements of automatic fault localization and program repair.The review notes duplicate commits and builds that fail for non-behavioral reasons.Source: §I · p. 1
  • APR and FL require behavioral bugs, but BugSwarm is a benchmark of failing and passing builds, which can fail for other reasons.Using all its builds would produce bad precision rates and misleading analyses.Source: §I · p. 1
  • The paper contributes a characterization of BugSwarm’s build execution, diffs, failure types, and patch types, plus lessons for future APR and FL benchmarks.It also provides the source code, TravisCI logs, and diffs for 3,091 build pairs, and an interactive site to browse and filter them.Source: §I · p. 1

II. BACKGROUND

BugSwarm pairs failing and passing continuous-integration builds, with the passing build serving as the human-fix oracle. The pairs are packaged in Docker images, and the benchmark’s creation process includes repeated execution to check behavioral consistency.

A. What is BugSwarm?

  • BugSwarm contains 3,091 pairs of failing and passing builds from TravisCI; the passing build contains the human modification that makes the failing build pass.This build-pair approach is also used by Bears, unlike traditional APR and FL benchmarks such as Defects4J and QuixBugs.Source: A. What is BugSwarm? · p. 2
  • Each pair is encapsulated in a Docker image containing source code and scripts to reproduce the failing and passing executions.BugSwarm also provides a command-line interface to download and start images and run scripts inside them.Source: A. What is BugSwarm? · p. 2
  • BugSwarm’s creation process mines TravisCI projects, matches Git and TravisCI histories, filters for Docker-based builds, and creates images containing both builds.Each image is executed five times to check behavioral consistency; matching failing-test counts are treated as reproduction.Source: A. What is BugSwarm? · p. 2
  • BugSwarm’s 3,091 pairs comprise 1,827 Java and 1,264 Python pairs from 160 GitHub projects.The project totals are 108 Java projects and 52 Python projects.Source: A. What is BugSwarm? · p. 2

B. APR and FL Requirements

APR and FL require bugs and program context that support test-based evaluation and fair comparison. The paper organizes its analysis around BugSwarm’s build characteristics, execution and storage costs, and the pairs that meet those requirements.

B. APR and FL Requirements

  • APR and FL target behavioral bugs in program source, using passing tests as correct behavior and failing tests as evidence of incorrect behavior.Configuration or external-file bugs are outside the stated target.Source: B. APR and FL Requirements · p. 2
  • Evaluation also requires a known program setup, a unique bug, and a human patch that supplies an oracle for the fix and bug location.These requirements are intended to avoid overfitting one bug and bias in evaluation results.Source: B. APR and FL Requirements · p. 3

A. Research Questions

  • The analysis asks about BugSwarm’s build characteristics relative to APR and FL, its execution and storage costs, and which pairs meet field requirements.The first question examines failing builds, human patches, and failure reasons; the second estimates running experiments on Amazon cloud.Source: A. Research Questions · p. 3

B. Protocol

The authors collect source code, diffs, TravisCI build information, and Docker metadata to analyze BugSwarm. Their reproduction-based count differs from the benchmark’s reported build-pair total.

B. Protocol

  • The collection protocol obtains build-pair information through BugSwarm’s API, runs Docker images, reconstructs buggy and passing versions, and retrieves logs and Docker manifests.The collected source, diffs, build information, and image information support the analysis.Source: B. Protocol · p. 3
  • The authors download failing-to-passing diffs through GitHub’s API and compute metrics such as changed files, changed lines, file types, and commit interval.The artifacts and analysis scripts are public, and a website presents diffs and metrics for each artifact.Source: B. Protocol · p. 3

C. RQ1. Characteristics of BugSwarm’s Pairs of Builds

  • The authors report 2,949 pairs reproduced five times, differing from BugSwarm’s reported 3,091 pairs.The passage notes that they applied the reproduction criterion described in BugSwarm’s paper.Source: C. RQ1. Characteristics of BugSwarm’s… · p. 3

1) Buggy Builds:

The review finds inconsistencies in BugSwarm’s reported and usable build counts, alongside human patches with duplicate and multilocation changes. Patch timing and file changes also show that build fixes can differ from conventional bug fixes.

1) Buggy Builds:

  • The review observed that the selected build-pair count ranged from 2,042 to 2,949 over time; the API may return a different number in the future.The authors said a database property had been overwritten by mistake and that they fixed it manually.Source: 1) Buggy Builds · p. 4
  • 40.08% (2,949 − 1,767)/2,949 of the builds have duplicate failing commit ids, and eight Docker images are unavailable.The authors say source-only approaches should exclude duplicate-commit pairs to avoid misleading results.Source: 1) Buggy Builds · p. 4
  • After availability, five successful reproductions, and unique-commit filters, 1,759 pairs from 156 repositories remain for analysis.The builds average 2.56 years old.Source: 1) Buggy Builds · p. 4

2) Human Patches:

  • 198 duplicate diffs and 332 repeated commit messages suggest similar changes that could lead to APR and FL overfitting; 73.63% (1,301) of builds change a non-source file.Techniques evaluated on BugSwarm therefore need to support multiple file types.Source: 2) Human Patches · p. 4
  • 33 minutes is the median build-fix time, while 58% (1,024/1,767) of builds were fixed within an hour.Some fixes merely ignore or comment out tests, leaving the application regression unfixed while restoring a passing build.Source: 2) Human Patches · pp. 4–5
  • Java diffs contain more added than removed lines, while Python diffs show the reverse; most current approaches handle only one location, unlike most BugSwarm pairs.The review also notes that Docker-image diffs do not always match GitHub diffs, with no explanation found for the discrepancy.Source: 2) Human Patches · p. 5

3) Reasons for Failures:

The review classifies failures into ten categories and finds test failures to be the most common, followed by checkstyle and compilation errors.

3) Reasons for Failures:

  • The authors classify build failures into ten categories, including test failures, checkstyle, compilation errors, dependency errors, regression detection, and unknown causes.They derive failure reasons by analyzing failing execution logs.Source: 3) Reasons for Failures · p. 5
  • Documentation-generation failures occur in only one project, terasolunaorgguideline, which contains all documentation for the TERASOLUNA Server Framework.Source: 3) Reasons for Failures · p. 6
  • Test failures represent 62% of cases, making them the most frequent failure cause in the review’s BugSwarm summary.Checkstyle and compilation errors are the next most frequent categories.Source: 3) Reasons for Failures · p. 6

D. RQ2. BugSwarm Execution and Storage Cost

BugSwarm’s Docker images dominate its storage and execution burden. The authors estimate substantial AWS costs, while noting that selecting builds can reduce the time and cost.

D. RQ2. BugSwarm Execution and Storage Cost

  • 3,680.45 GB is the estimated disk space required for BugSwarm’s Docker images, with image storage and download contributing to the cost.The storage estimate uses Docker 18.09.2 on OSX; the ratio can differ by operating system and Docker version.Source: D. RQ2. BugSwarm Execution and Storage… · p. 7
  • At 80 Mbits/s, downloading BugSwarm takes between 2 and 11 days, depending on how many Docker layers must be downloaded.The authors report that retaining shared layers reduces download volume, but storing more than 350 GB of images can slow the computer dramatically.Source: D. RQ2. BugSwarm Execution and Storage… · p. 7
  • 711.30 USD is the estimated starting cost of one 20-minute-per-build experiment on AWS, excluding image decompression time.The estimate includes 666.06 USD for 983 hours of execution, in addition to download and storage costs.Source: D. RQ2. BugSwarm Execution and Storage… · p. 7
  • Selecting which build pairs to execute can significantly reduce the costs and execution time.Downloading and extracting the Docker image is the workflow step that most affects execution time and required storage.Source: D. RQ2. BugSwarm Execution and Storage… · pp. 6–7
  • Downloading BugSwarm on an AWS instance is estimated to cost 45.24 USD, with the estimate based on 3,680.45 GB of required storage and 2 days and 18.80 hours to download.The estimated cost includes 11.11 USD for machine rental and 34.13 USD for storage during download.Source: D. RQ2. BugSwarm Execution and Storage… · p. 7

E. RQ3. BugSwarm for APR and FL

The authors filter BugSwarm pairs against APR and FL requirements, then manually categorize the remaining patches. Their analysis finds that only a small fraction are bug fixes.

E. RQ3. BugSwarm for APR and FL

  • The filtering criteria address reproducibility, available Docker images, non-empty and unique diffs and commits, failing tests, source-file-only changes, and unchanged tests.The authors report reducing the set to 163 pairs: 65 Java and 98 Python.Source: E. RQ3. BugSwarm for APR and FL · pp. 7–8
  • 112 of the 163 filtered build pairs were manually identified as bug fixes; 40 were not bug fixes and 11 were unknown.The 112 identified bug fixes comprise 50 Java and 62 Python pairs.Source: E. RQ3. BugSwarm for APR and FL · p. 8
  • The RQ3 answer reports 146 compatible pairs—81 Java and 66 Python—representing 4.72% of BugSwarm, with 99 manually identified bug fixes.The RQ3 answer reports 48 Java and 51 Python bug fixes.Source: E. RQ3. BugSwarm for APR and FL · p. 8

IV. DISCUSSION

BugSwarm’s failing builds arise for ten different reasons, but only test failures expose bugs; its name can therefore mislead users about what the benchmark contains.

A. Failing builds vs. bugs

  • Only test failures among BugSwarm’s ten failing-build causes expose bugs; checkstyle errors and compilation failures do not produce incorrect behavior.The section gives checkstyle errors and inability to clone a submodule as examples of non-bug failures.Source: A. Failing builds vs. bugs · p. 8
  • BugSwarm’s name suggests a benchmark of bugs, although its contents are pairs of builds with varied failure causes.Source: A. Failing builds vs. bugs · p. 8
  • Using BugSwarm without analyzing its contents can lead to invalid reviewer recommendations or incorrect empirical evaluations.Source: A. Failing builds vs. bugs · p. 8

B. Comparison between BugSwarm and existing benchmarks for APR and FL

BugSwarm differs from bug-focused benchmarks in its build-based collection, Docker infrastructure, and large diffs. The authors recommend benchmark practices and note that no benchmark they examined meets all of them.

B. Comparison between BugSwarm and existing benchmarks for APR and FL

  • Docker images improve reproducibility, but BugSwarm images omit application dependencies, leaving builds vulnerable to missing or invalid dependencies.The authors identify dependencies as the most frequent source of unreproducibility according to BugSwarm’s authors; Docker images also add size and execution overhead.Source: B. Comparison between BugSwarm and… · pp. 8–9
  • BugSwarm’s average patch size is 1,026 lines, compared with 4 lines in Defects4J and 8 lines in Bears.The authors interpret this difference as reflecting BugSwarm’s different nature from the other benchmarks.Source: B. Comparison between BugSwarm and… · p. 9

C. Lessons learned

  • APR and FL benchmarks should target field requirements, including bugs exposed by failing tests and metadata such as bug locations or binaries.Source: C. Lessons learned · p. 9
  • The authors recommend artifact diffs, field-specific artifact selections, included dependencies, full benchmark and metadata access, open-source creation tools, documentation, and versioning.They report that none of the benchmarks compared meets all recommendations; Defects4J and Bears meet 5/6.Source: C. Lessons learned · p. 9

D. Threats to Validity

The authors identify threats from potentially faulty analysis scripts and future changes to BugSwarm or to fault-localization and repair requirements. They also explain BugSwarm’s selection and describe feedback that led to fixes in the benchmark.

D. Threats to Validity

  • The analysis scripts may contain bugs that affect reported results, although the scripts and raw output are publicly available for checking.BugSwarm may also change, so the observed result could differ; the authors provide scripts to redo the analysis and update the website and repository.Source: D. Threats to Validity · p. 9
  • The bug selection may become inadequate if fault-localization and repair requirements evolve beyond those considered in the analysis.The authors say the analysis reflects current requirements.Source: D. Threats to Validity · pp. 9–10

E. Discussion with BugSwarm’s Authors

  • Feedback from BugSwarm’s authors helped clarify the paper, and the review also helped identify fixes for duplicate artifact IDs, inconsistent pair counts, and unavailable Docker images.Source: E. Discussion with BugSwarm’s Authors · p. 10

A. Benchmarks

  • The authors chose BugSwarm because it was new, expected to have strong ICSE visibility, and differed from other benchmarks in creation process and content.Source: A. Benchmarks · p. 10

B. Benchmark Analysis

The paper characterizes BugSwarm’s contents and evaluates its suitability for fault localization and program repair. It finds that only 112 of 3,091 build pairs meet the current approaches’ requirements, while identifying selection criteria and benchmark characteristics relevant to researchers.

B. Benchmark Analysis

  • The analysis covers fix time, benchmark size, execution time, cost, failure type, and content suitability for the benchmark’s target research fields.Source: B. Benchmark Analysis · p. 10

VI. CONCLUSION

  • BugSwarm’s pairs include 142 that fail its five-reproduction inclusion criterion, 1,182 duplicate commits, and 8 unavailable pairs.Human patches modify an average of 1,026 lines across 7.71 files; failures include tests (62.32%), checkstyle errors (11.12%), and compilation errors (10.03%).Source: §VI · p. 10
  • The review identifies six requirements for using build pairs in repair and localization: availability, unique commit and diff, source-code change, test failure, and an unchanged test suite.Of 163 potentially relevant pairs, 112 were bug fixes, 40 non-bug fixes, and 11 unknowns.Source: §VI · p. 10
  • The authors report infrastructure overhead of 45.24 USD, 2d, 18.8h, and 3,680.45 GB, describing it as the cost of improved reproducibility.Source: §VI · p. 10
  • Only 112/3,091 (3.6%) BugSwarm build pairs are compatible with current automatic program repair and fault-localization approaches, fewer than in related benchmarks.Source: §VI · p. 10
  • The review gathers findings on requirements for benchmarks suited to fault localization and program repair, intended to inform upcoming benchmark sets.Source: §VI · p. 10
Loading 1905.09375v1…