PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissection of a Bug Dataset: Anatomy of 395 Patches from Defects4J

Victor Sobreira, Thomas Durieux, Fernanda Madeiral, Martin Monperrus, Marcelo A. Maia

arXiv:1801.06393v3cs.SE

TL;DR

Researchers need detailed bug and patch properties because existing datasets often omit them, limiting informed analysis of technique applicability and effectiveness. This paper combines automated extraction of quantitative properties with manual thematic analysis of 395 Defects4J patches. It reports concentrated patch sizes, spreading, repair actions, and patterns, while identifying limitations in manual analysis and independent pattern counting.

  • Problem

    Bug datasets often lack fine-grained properties needed to select suitable bugs and analyze technique performance by bug or patch characteristics.

  • Method

    The study analyzes Defects4J using automated extraction of patch size and spreading and manual thematic analysis of repair actions and patterns.

  • Results

    95% of Defects4J patches involve at most 22 lines, 77.21% include additions of method calls, conditional branches, or assignments, and nine patterns occur in 94.43% of patches, led by Conditional Block at 42.78%.

  • Takeaways & Limitations

    The extracted properties support informed Defects4J use, advanced analysis of repair results, and comparison with other bug datasets.

  • Takeaways & Limitations

    Manual taxonomy and annotation may contain small mistakes or misinterpretations, and independently counted patterns do not capture their co-occurrences or full patch context.

Abstract

from arXiv · show

Well-designed and publicly available datasets of bugs are an invaluable asset to advance research fields such as fault localization and program repair as they allow directly and fairly comparison between competing techniques and also the replication of experiments. These datasets need to be deeply understood by researchers: the answer for questions like "which bugs can my technique handle?" and "for which bugs is my technique effective?" depends on the comprehension of properties related to bugs and their patches. However, such properties are usually not included in the datasets, and there is still no widely adopted methodology for characterizing bugs and patches. In this work, we deeply study 395 patches of the Defects4J dataset. Quantitative properties (patch size and spreading) were automatically extracted, whereas qualitative ones (repair actions and patterns) were manually extracted using a thematic analysis-based approach. We found that 1) the median size of Defects4J patches is four lines, and almost 30% of the patches contain only addition of lines; 2) 92% of the patches change only one file, and 38% has no spreading at all; 3) the top-3 most applied repair actions are addition of method calls, conditionals, and assignments, occurring in 77% of the patches; and 4) nine repair patterns were found for 95% of the patches, where the most prevalent, appearing in 43% of the patches, is on conditional blocks. These results are useful for researchers to perform advanced analysis on their techniques' results based on Defects4J. Moreover, our set of properties can be used to characterize and compare different bug datasets.

I. INTRODUCTION

The paper argues that bug datasets need fine-grained characterization to support fair evaluation, informed technique selection, and analysis of which bugs techniques handle. It studies Defects4J through quantitative patch properties and qualitative repair actions and patterns.

  • Publicly available bug datasets enable fair comparison between techniques and replication of experiments.
  • Fine-grained dataset information supports selecting bugs with required properties and analyzing technique performance by bug or patch characteristics.
  • The study analyzes Defects4J's 395 real bugs using patch size, spreading, repair actions, and repair patterns.
  • The study highlights a need for techniques that leverage addition-only patches and for repair algorithms targeting specific repair patterns.
  • The methodology combines automated diff and patch analysis with manual thematic analysis to extract quantitative and qualitative properties.
  • Repair patterns are recurring abstractions that combine repair actions, and identifying them can support techniques such as template-based code synthesis.

A. Subject Dataset: Defects4J

Defects4J provides reproducible, isolated source-code bugs with paired buggy and fixed program versions. The study derives patch properties from diffs and manually identifies repair actions and recurring patterns, publishing the resulting data for exploration.

  • A. Subject Dataset: Defects4J: Defects4J 1.1 contains 395 real source-code bugs from six open-source Java projects, each with buggy and fixed program versions.Its bugs are reproducible and isolated from unrelated features or refactorings.
  • Data collection: The analysis begins by producing diff views between each buggy program version and its associated fixed version.These views serve as the source for data extraction and analysis.
  • 1) Patch Size:: Patch-size scripts count added, removed, and modified source lines, including consecutive or sparse changes and partial statements.
  • 2) Patch Spreading:: Patch spreading is measured through chunks, inter-chunk lines, and the numbers of modified files, classes, and methods.A single chunk has no spreading; additional chunks increase the measured spread, while empty and comment lines are excluded from chunk spreading.
  • 3) Repair Actions:: Repair actions are fine-grained changes over code elements, such as modifying a method-call parameter or removing a conditional branch.
  • Repair Patterns: Thematic analysis identifies recurring combinations of repair actions as repair patterns through coding, theme review, naming, and reporting.
  • Validation: One author manually identified and annotated repair actions and patterns, while two other authors validated all annotations.
  • Data Availability: The collected Defects4J data is consolidated in a publicly available JSON file and supported by a web interface for browsing, filtering, and understanding patches.The interface also includes runtime information and repair-tool results.

III. RESULTS AND ANALYSIS

The results analyze Defects4J patch size through line additions, removals, modifications, and total changed lines. Most patches are small, and addition-only patches form a substantial subset.

  • Patch composition: 29.87% of patches contain only added code, while 59.24% are exclusive to one change type.Nine patches contain only removals, showing that some fixes remove code without adding or modifying lines.
  • Added lines: 25% of patches have no added lines, half add at most two lines, and 95% add between zero and 19 lines.Additions range from zero to 48 lines, with larger additions occurring in outlier patches.
  • Removed lines: For 95% of patches, no more than six lines are removed.Removed-line counts are considerably lower than added-line counts.
  • Modified lines: 25% of patches have no modified lines, half have at most one modified line, and 95% have at most four.
  • Total patch size: The median patch size is four lines, and only 5% of patches involve more than 22 lines.Patch size ranges from one to 54 lines; at most 22 lines cover 95% of patches.

B. Spreading of the Defects4J Patches (RQ #2)

The spreading analysis measures chunks, separating lines, and modified files, classes, and methods. Most patches are localized, although many contain multiple chunks and some modify multiple methods.

  • Chunk spreading: 25% of patches have no chunk spreading, and half have spreading of no more than one line.A spreading of 214 lines covers 95% of patches.
  • Files: 92.41% of patches modify only one file, while 7.09% modify two files.Only two patches affect more than two files.
  • Classes: Only eight patches modify more classes than files.
  • Methods: 27% of patches change more than one method, and 47% of those patches involve Copy/Paste or Missing Null-Check patterns.Two patches change only class and field declarations without modifying methods.
  • Chunks: 38.23% of patches consist of a single continuous chunk, while 52.41% have only one line separating chunks.Patches contain between one and 20 chunks, with 95% containing at most eight.

1) Assignment:

The repair-action taxonomy distinguishes additions, removals, and modifications across assignments, conditionals, and loops. It also captures changes to expressions, branches, tests, and initialization variables.

  • Assignment: Assignment Addition occurs when a variable assignment appears in added lines without an assignment for that variable among removed lines.
  • Assignment: Assignment Removal occurs when a variable assignment appears in removed lines without a corresponding assignment among added lines.
  • Assignment: Assignment Modification changes the expression assigned to a variable, which appears in both removed and added lines.
  • Conditionals: Conditional actions include adding or removing branches and modifying, expanding, or reducing conditional expressions.The taxonomy covers if, if-else, else, and switch-case constructions.
  • Loops: Loop actions include adding or removing loops and modifying their conditional tests or initialization variables.The considered loop constructions are for, while, and do-while.

4) Method Call:

The taxonomy covers method-call changes, method-definition and signature changes, object instantiation, exception handling, and return statements. Actions are classified as additions, removals, or modifications where applicable.

  • Method Call: Method Call Addition includes adding a call or adding parameters by replacing it with an overloaded version.
  • Method Call: Method Call Removal includes removing a call or parameters by replacing it with an overloaded version having fewer parameters.
  • Method Call: Method Call Modification includes replacing or moving calls, modifying parameter values, or swapping parameter values.
  • Method Definition: Method-definition actions include adding or removing complete definitions or parameters and modifying names, parameter types, return types, modifiers, or overriding methods.
  • Other actions: Object-instantiation actions include adding, removing, or modifying expressions involving the keyword new.Exception actions include adding or removing try-catch blocks or throw statements.

8) Return: •

The study characterizes Defects4J patches through grouped repair actions and their distribution across patches. Additions dominate the most prevalent actions, while most patches contain no more than three action types, with a small number of complex outliers.

  • The analysis groups related repair actions to reduce graph fragmentation and distinguishes additions, removals, and modifications.
  • Method Call Addition occurs in 243 patches, followed by Conditional Branch Addition in 206 and Assignment Addition in 136.
  • Together, the three most prevalent repair actions cover 77.21% of patches after discounting co-occurrences.Adding structures surpasses removing or modifying existing ones in all three cases.
  • 50% of patches contain no more than three types of repair actions.
  • Some outlier patches contain 10–15 repair actions, with a maximum of 19.

D. Repair Patterns in the Defects4J Patches (RQ #4)

The manual analysis identified nine recurring repair patterns in Defects4J patches. These patterns include additions, expression fixes, wrapping or unwrapping code, reference corrections, null checks, repeated changes, and code movement.

  • Nine repair patterns were found in Defects4J patches and presented in descending order of prevalence.Examples are provided through code snippets in Figure 4.
  • Conditional Block is centered on adding missing conditional logic, including variants with return statements or exception throwing.
  • Expression Fix: Expression Fix covers modifications or expansions of logic and arithmetic expressions in conditions, assignments, and return statements.
  • Other patterns include Single Line, Wrong Reference, Missing Null-Check, Code Moving, and repeated copy-paste-like changes.
  • Wraps-with: Wraps-with surrounds existing code with structures such as conditionals, try-catch blocks, method calls, or loops.

3) Wraps-with:

The Wraps-with pattern encloses existing code in a new structure, with variants spanning conditional expressions, exception handling, method calls, and loops. Its inverse, Unwraps-from, removes such enclosing structures.

  • The if variant wraps existing code with conditional logic, while the if-else variant places it in one branch of a conditional expression.
  • Other Wraps-with variants add an else complement, a try-catch block, a method call, or a loop around existing code.
  • Unwraps-from is the inverse pattern, removing an enclosing method call or conditional structure.
  • Single Line: Single Line patches contain one line addition, one line removal, both as a modification, or special multi-line single-statement and moving cases.

4) Single Line:

The Wrong Reference pattern replaces an incorrectly referenced variable or method call, while Missing Null-Check adds conditional expressions that test nullity. Some patches repeat the same change at multiple locations.

  • Wrong Reference: Wrong Reference replaces a mistakenly used variable or method call with another reference.
  • Missing Null-Check: Missing Null-Check adds positive or negative nullity checks around existing logic.
  • Some patches repeat the same change at different locations, resembling a copy-paste operation across methods or files.

7) Copy/Paste:

The study identifies recurring repair patterns in Defects4J patches and examines their prevalence, composition, and implications for repair techniques. It also notes limitations from manual annotation, pattern independence, dataset scope, and isolated patches.

  • Repair patterns: Conditional Block is the most prevalent repair pattern, appearing in 169 patches (42.78%), followed by Expression Fix and Wraps-with.Constant Change and Code Moving are the least prevalent patterns.
  • Pattern distribution: Most patches have no more than two repair patterns, while outlier patches contain between four and seven.The median and upper quartile of the number of patterns per patch are the same.
  • Repair patterns: Nine repair patterns span 373 patches (94.43%), indicating recurring techniques across Defects4J patches.The patterns provide a basis for categorizing patches and developing repair tools for specific bug types.
  • Implications: Most Defects4J patches add more code than they remove or rewrite, motivating repair techniques that synthesize additions.The implication extends to fault localization, which should identify locations of missing code as well as wrong code.
  • Scope and validity: Defects4J may underrepresent bugs spread across many files, and its isolated patches may not reflect noisy real-world commits.The dataset contains source-code bugs by design and includes two duplicated bug pairs that were retained for analysis.
  • Scope and validity: The taxonomy and annotations rely on manual analysis, which can introduce small mistakes or misinterpretations and is difficult to scale.The authors use the manual process to gain insights into automating property collection.
  • Scope and validity: Treating repair patterns independently means that identifying one pattern does not fully describe the patch's other patterns or repair actions.More detailed pattern composition requires studying correlations and co-occurrences.

VI. RELATED WORKS

Prior bug datasets provide varying levels of patch size, spreading, syntactic, and manually evaluated information, but Defects4J lacked an in-depth characterization. This work analyzes Defects4J patches across four properties and draws implications for program-repair research and dataset use.

  • Dataset landscape: Several datasets omit detailed bug or patch information, while others provide only simple annotations such as bug type.The related work positions patch-focused datasets as particularly close to this study.
  • Dataset landscape: iBugs, ManyBugs, and Codeflaws annotate patch syntax or changes, but their taxonomies and available metrics differ from this work.iBugs provides size, spreading, and syntactic fingerprints; ManyBugs offers manually evaluated patch changes; Codeflaws provides AST-level differences but lacks size, spreading, and repair-pattern information.
  • Prior pattern studies: Pan et al. found 27 bug-fix patterns covering 46–64% of fixes, with method-call and if-condition patterns each appearing in about 20%, consistent with this study’s prevalent repair actions.The cited consistency concerns Method Call Addition and Conditional Branch Addition.
  • Prior pattern studies: Soto et al. found that fewer than 15% of 4,590,679 bug-fix commits contained the examined patterns, using a different dataset and collecting fewer patch-characterization metrics.This work additionally characterizes patch size and spreading.
  • This study’s contribution: The study addresses Defects4J’s missing in-depth characterization by analyzing patch size, spreading, repair actions, and repair patterns.Its contribution is framed as informing potential dataset users about Defects4J’s contents and supporting dataset comparison.
  • Implications: The findings support attention to addition-based repair, multi-point repair, and single-file repair when designing or evaluating program-repair techniques.The authors also state that Defects4J is appropriate for program-repair research and that automatic patch generation can rely on prevalent repair actions.
Loading 1801.06393v3…