PaperlayerView exact revision on arXiv

Source-linked AI summary

RunBugRun -- An Executable Dataset for Automated Program Repair

Julian Aron Prenner, Romain Robbes

arXiv:2304.01102v1cs.SEcs.LG

TL;DR

Large-scale NPR often relies on non-executable code, limiting evaluation to static comparisons even though execution can assess candidate fixes semantically. RunBugRun provides 450,000 executable, testable program pairs across eight languages, and baseline results show substantial repair challenges remain.

  • Problem

    NPR datasets often contain non-executable fragments and rely on static metrics, although lexical comparisons can miss semantically equivalent fixes.

    Source: §1 · pp. 1–2

  • Method

    RunBugRun provides over 450,000 buggy/fixed contest-program pairs with tests, execution infrastructure, and rule-based fine-grained labels.

    Source: §4 · pp. 10–11, §5 · p. 12

  • Results

    More than 40% of bugs (35% with 10 candidates) could not be repaired by the straightforward NPR baseline, and repair performance correlated with fix size.

    Source: §6 · p. 22

  • Takeaways & Limitations

    Short algorithmic programs allow efficient execution and facilitate a large range of experiments.

    Source: §7 · p. 23

    The dataset’s short algorithmic solutions differ substantially from the code of larger software projects.

    Source: §7 · p. 23

Abstract

from arXiv · show

Recently, we can notice a transition to data-driven techniques in Automated Program Repair (APR), in particular towards deep neural networks. This entails training on hundreds of thousands or even millions of non-executable code fragments. We would like to bring more attention to an aspect of code often neglected in Neural Program Repair (NPR), namely its execution. Code execution has several significant advantages. It allows for test-based evaluation of candidate fixes and can provide valuable information to aid repair. In this work we present a fully executable dataset of 450,000 small buggy/fixed program pairs originally submitted to programming competition websites written in eight different programming languages. Along with the dataset we provide infrastructure to compile, safely execute and test programs as well as fine-grained bug-type labels. To give a point of reference, we provide basic evaluation results for two baselines, one based on a generate-and-validate approach and one on deep learning. With this dataset we follow several goals: we want to lift Neural Program Repair beyond fully static code representations, foster the use of execution-based features and, by including several different languages, counterbalance the predominance of Java in the current landscape of APR datasets and benchmarks.

1 Introduction

RunBugRun addresses the divide between executable APR benchmarks and large, mostly static NPR datasets by providing a curated, multilingual executable dataset. Its baseline evaluation surfaces distinct challenges for repair approaches and leaves substantial room for improvement.

1 Introduction

  • NPR commonly trains on static textual, tree, or graph features, while execution can test candidate fixes semantically and avoid lexical false negatives from equivalent code forms.Existing APR benchmarks are often small, Java-focused, and manually curated, while large NPR datasets commonly contain non-executable fragments.Source: §1 · pp. 1–2
  • 450,000 contest-program bug pairs are fully compilable and executable, with tests and infrastructure to compile, safely execute, and test candidate fixes.The dataset also includes exception traces and error messages that may help localize and fix bugs [83].Source: §1 · p. 3
  • RunBugRun covers eight languages, four statically typed and four dynamically typed, and differences in dataset sizes create opportunities to investigate transfer learning.The languages are C, C++, Java, Python, JavaScript, Ruby, Go, and PHP.Source: §1 · p. 3
  • Careful curation checks that bugs fail tests, fixes pass, bugs are semantic rather than syntactic, and duplication, flakiness, and difficulty distribution are addressed.Fine-grained bug labels support diagnosis of APR systems’ weak spots.Source: §1 · p. 3
  • Baseline evaluation finds a strong correlation between repair performance and fixing-change count, identifies strengths and weaknesses for both approaches, and provides initial evidence of cross-language knowledge transfer.The authors conclude that both generate-and-validate and NPR approaches have ample room for improvement.Source: §1 · p. 3
  • RunBugRun’s programs are short algorithmic solutions, unlike large software projects, and its bugs were extracted from Project CodeNet through filtering, preprocessing, execution, and labeling.The motivation also includes improving dataset curation and performance insight, including analysis of model performance across bug categories.Source: §1 · pp. 2–3

2 Related Work

Prior APR resources divide between executable evaluation benchmarks and large non-executable datasets used mainly for NPR training. RunBugRun is situated among contest-derived datasets, with related work differing in scale, languages, curation, and infrastructure.

APR Benchmarks

  • Other executable benchmarks include Defects4J, Bugs.jar, Bears, QuixBugs, BugsInPy, Bugs.js, Defexts, and BugSwarm, with differing languages and project or contest origins.Defects4J has 365 bugs in version 1 and 835 in version 2; QuixBugs has 40 Python and Java algorithm implementations.Source: §2 · pp. 3, 5
  • Existing APR benchmarks are executable and provide tests, but generally do not provide training data; RunBugRun combines a large training set with a test split usable as a benchmark.The test split is described as considerably larger than existing benchmarks.Source: §2 · pp. 3, 5

NPR Datasets

  • NPR datasets typically provide non-executable code fragments without tests, including large mined buggy/fixed corpora across Java, Python, JavaScript, and other languages.Examples range from ManySStuBs4J’s over 150,000 Java bugs to TSSB’s 9 million Python pairs.Source: §2 · p. 5

Works based on contest submissions

  • Contest-derived work includes Codeforces-based datasets and Codeflaws; RunBugRun builds on Project CodeNet submissions from AIZU and AtCoder.Codeflaws is C-only, has 7,436 bugs, and categorizes defects into 40 classes; RunBugRun’s test set has over 30,000 bugs.Source: §2 · p. 5
  • Compared with Majd et al.’s contest dataset, RunBugRun executes all submissions against tests, filters syntax errors, checks cross-set duplicates, applies fine-grained hierarchical labels, and provides baselines and execution infrastructure.The authors also describe Python 2-to-3 conversion and occasional fixes for deprecations between minor Python versions.Source: §2 · p. 5

3 Motivation

The paper motivates RunBugRun as a bridge between executable but small traditional APR benchmarks and large but non-executable NPR datasets. It argues for scale, multilingual coverage, and stronger curation so repair can be evaluated and studied across more programs and bug types.

3.1 APR needs a large-scale and executable dataset

  • Large-scale executable datasets can combine test-based evaluation with broader coverage, reducing manual assessment and benchmark overfitting concerns.Small programs make compile-run-test cycles faster, while larger evaluation sets can reduce reliance on individual datapoints; strong tests offer a reasonable, though imperfect, alternative to manual assessment.Source: §3.1 · pp. 6–7
  • Execution-based evaluation recognizes functionally equivalent fixes that lexical metrics can miss, while execution traces and errors can also inform repair.The paper cites prior work showing that static metrics can mislead and that execution-derived feedback, stack traces, and error messages may help repair.Source: §3.1 · pp. 6–8, Figure 2 · p. 7

3.2 APR needs a multilingual dataset to overcome language bias

  • Existing APR datasets are Java-centric, while several popular languages, including Go, PHP, and Ruby, remain underrepresented.The paper argues that greater language coverage could support polyglot or language-agnostic repair, including work on bugs involving multiple languages.Source: §3.2 · p. 8

3.3 APR needs a curated and labelled dataset—and execution helps

  • Large mined datasets raise questions about whether instances are genuine, sufficiently difficult bugs and whether bug types are overrepresented; fine-grained labels support analysis by bug type.Execution can help check whether buggy versions fail tests while fixed versions pass, but the paper describes this as only a starting point for curation.Source: §3.3 · p. 9

4 Methodology

RunBugRun is built from programming-contest submissions by filtering and pairing likely buggy and fixed programs, collecting and generating tests, and executing submissions. Further curation, labeling, deduplication, and predefined splits produce a benchmark with reference baselines.

4 Methodology

  • Rejected and accepted submissions from the same problem and user are paired when their token-level difference is fewer than six tokens, targeting bugs fixable with few changes.The authors chose a low threshold because current APR systems struggle with bugs requiring multiple change actions.Source: §4 · p. 10
  • Tests combine extracted problem examples, CodeContests cases, and semi-automatically generated cases; the dataset adds 331,834 CodeContests tests to 9,082 extracted examples.For generated inputs, 10–15 accepted submissions are sampled per problem, and an input is added when their outputs agree.Source: §4 · pp. 10–11
  • Pairs with syntax errors are removed, and execution filters discard pairs whose buggy version passes all tests or times out, or whose fixed version fails or times out.Test cases are also filtered using overall pass-rate and timeout-rate thresholds of 45% and 12%, respectively.Source: §4 · pp. 10–11
  • A rule-based labeling system matches changed parse-tree nodes to manually written rules, yielding a taxonomy of 136 leaf labels and 16 stem labels.About 9% of submissions match no rule; each bug receives 2.3 labels on average.Source: §4 · p. 11
  • The test and validation sets are checked for flaky fixed submissions, and predefined splits balance test-set language representation while keeping problem-user instances together.The split permits submissions for the same problem to occur across sets, a choice intended to test learning and application of fix patterns; the authors considered but deferred a test set of only unseen problems.Source: §4 · pp. 11–12
  • The dataset removes exact duplicates and cross-set near duplicates, and supplies CodeT5 and Cardumen as neural and generate-and-validate reference baselines.Near duplicates are identified using trigram similarity; Cardumen is selected as a G&V baseline because of its results on QuixBugs.Source: §4 · p. 12

5 Results & Discussion

The results section frames the dataset’s contents and baseline performance as its central questions. The supplied passages describe the dataset’s scale and the intended analyses, but do not report their findings.

5 Results & Discussion

  • RunBugRun contains over 450,000 buggy/fixed program pairs from programming-contest submissions, with language identifiers, optional labels, test cases, and possible error messages.The passage notes that 9% of bugs do not fall into an existing label.Source: §5 · p. 12
  • The dataset-statistics questions examine language distribution, change counts, common bug labels, and errors or exceptions observed during test execution.The passages state these as research questions, not as findings.Source: §5 · p. 12
  • The baseline questions cover performance across languages, change counts, and labels, as well as cross-language transfer and the effect of candidate counts.The stated goal includes providing future users a yardstick for measuring tool performance.Source: §5 · p. 12

5.1 Dataset Statistics (RQ1)

RunBugRun’s dataset is skewed toward C/C++ and Python, with fixes generally requiring few token changes. Bug patterns and runtime errors vary across languages, including clear differences between statically and dynamically typed languages.

5.1.1 Language Distribution (RQ1.1)

  • C++ and Python dominate the language distribution, while JavaScript, Go, and PHP are least frequent; the test set raises the least-to-most frequent language ratio to roughly 10x.The ratio across the overall data is approximately 125.Source: §5.1.1 · pp. 12–13, Figure 4 · p. 13

5.1.2 Number of Changes (RQ1.2)

  • Almost 30% of bugs require one token change, while only 8.6% require six; the median is two changes except in JavaScript and Python, where it is three.About 75% of bugs are single-hunk changes, 20% have two hunks, and 5% have more than two.Source: §5.1.2 · p. 13, Figure 5 · p. 14

5.1.3 Bug Labels (RQ1.3)

  • Function or method call-argument changes dominate in six of eight languages, while if-condition changes lead in C++ and Go; output and literal changes are also frequent.Comparison operators are the most commonly changed operators.Source: §5.1.3 · p. 14, Table 2 · p. 15

5.1.4 Errors & Exceptions (RQ1.4)

  • Error identification uses language-dependent heuristics, and PHP error frequencies cannot be quantified reliably.The paper notes that PHP reports warnings and errors to standard output, complicating error detection.Source: §5.1.4 · pp. 14, 16
  • Dynamic-language errors most often involve variable or method name resolution, whereas static-language errors mainly involve invalid bounds; C and C++ also show frequent memory errors.For Go, 71% of error messages matched “index out of range” and 6% matched “slice bounds out of range.”Source: §5.1.4 · p. 16

5.2 Baselines (RQ2)

The section establishes CodeT5 and Cardumen as reference baselines, then examines performance by bug complexity, bug label, language, and candidate count.

5.2.1 Evaluation

  • Cardumen evaluation was limited to Java and a 10-minute, 100-generation budget; its search also faced low redundancy in these small programs.The Java test cases were automatically translated into JUnit4 tests.Source: §5.2.1 · p. 18, §5.2.2 · p. 18

5.2.2 Baseline Performance (RQ2.1)

  • CodeT5 plausibly repaired 59.8% of submissions across languages with five candidates, while Cardumen plausibly fixed 157 of roughly 3,300 Java bugs.The authors characterize Cardumen’s result as barely 5%.Source: §5.2.2 · p. 18
  • CodeT5 plausibility fell from 68.7% for single-change fixes to 42.8% for six-token fixes, and ranged from 69% for Python to 47.6% for Ruby.The change-size figures use five candidates per bug; language performance also reached 64% for C and 62% for Java.Source: §5.2.2 · p. 18

5.2.3 Performance across Labels – Strengths & Weaknesses (RQ2.2)

  • CodeT5 performance varied substantially by bug label, while label performance correlated only 0.09 with data amount; control-flow insertions and some variable changes were weak spots.Cardumen mainly fixed simple expression changes and failed on some labels, including added function calls and type conversions.Source: §5.2.3 · pp. 19–20

5.2.4 Knowledge Transfer between Languages (RQ2.3)

  • All-language training improved plausibility across test languages, with the largest gains for small-data languages; gains for C++ and Python were only 1–2%.The authors describe these results as preliminary evidence that low-data languages can benefit from cross-language training.Source: §5.2.4 · p. 21, Table 5 · p. 21

5.2.5 Number of Candidates in NPR (RQ 2.4)

  • With one candidate, plausibility was 46.13%; with ten, it was 64.33%, but doubling candidates from five to ten yielded only 7.5% relative improvement.Figure 7 plots plausibility rate against the number of candidates per bug for the all-language model.Source: Figure 7 · p. 21, §5.2.5 · p. 22

6 Discussion

The discussion identifies distinct challenges and opportunities for generate-and-validate (G&V) and neural program repair (NPR). It also outlines ways to use execution data, transfer learning, and benchmark updates.

6 Discussion

  • Cardumen’s constrained time budget and limited ingredient search likely depressed performance, while slow search and Java-only support constrain G&V at scale.The authors argue that more efficient search is needed for G&V to be competitive with NPR.Source: §6 · p. 22
  • More than 40% of bugs remained unrepaired by the NPR baseline, and performance tracked fix size; its stronger cases were simple expression fixes, weaker ones control-flow bugs.The discussion notes that the remaining bugs may be more challenging.Source: §6 · p. 22
  • Transfer learning results suggest small training sets need not be as limiting when models can learn from similar languages, though complex bugs or exotic language features may benefit less.The authors call for further research on models that exploit cross-language data more efficiently.Source: §6 · p. 22
  • The collected execution data was not used as a learning signal in the reported NPR baselines; runtime-error types and failing-test information are proposed as useful signals.The discussion also proposes execution feedback for repair attempts and possible G&V–NPR hybrids.Source: §6 · pp. 22–23
  • If benchmark performance begins to saturate, proposed updates include larger or reweighted fixes, harder bug categories, more languages, less training data, or problem-disjoint splits.The authors give performance exceeding 95% as an example of saturation.Source: §6 · p. 23

7 Limitations, Biases and Threats to Validity

RunBugRun uses short algorithmic solutions, with limited code and bug diversity, and its curation has several stated boundaries. These include incomplete language-specification tokenization, possible gaps in bug labeling, and the difficulty of assessing patch correctness manually at scale.

Limited scope

  • The dataset covers short algorithmic solutions, which differ from larger projects with complex hierarchies, dependencies, and multi-file code.The authors note that its bug-type distribution resembles that of open-source projects, while simple programs permit efficient execution and a broad range of experiments.Source: §7 · p. 23

Lack of diversity

  • Solutions to the same problem share similarities, limiting code and bug variety across the 2,693 problems in the test set.Bug labels are provided to help users understand bug types and per-type performance.Source: §7 · p. 23

Tokenization

  • The eight language tokenizers cover a large portion of their specifications, but perfect specification-level tokenization was out of scope.The authors describe perfect tokenization across all eight languages as requiring substantial engineering effort.Source: §7 · p. 23

Labeling

  • Bug labels come from manually written, often language-specific parse-tree rules, and some important change classes may not be adequately captured.Existing tools such as GumTree did not then support every dataset language.Source: §7 · p. 23

Plausibility vs correctness

  • Manual correctness assessment is too time-consuming to apply feasibly across a test set numbering in the thousands.In a random sample of 200 plausible CodeT5 fixes, 7 (3.5%) were judged incorrect; 46.5% matched the ground-truth patch, and 19 cases were difficult to assess.Source: §7 · pp. 23–24

8 Concluding Remarks

RunBugRun is a large, multilingual dataset of executable bugs, and baseline performance varies by language and number of required changes. The authors point to runtime information as a future direction and plan to expand the dataset's language coverage.

8 Concluding Remarks

  • RunBugRun results show NPR-generated single-change fixes were plausible in 68.7% of cases; plausibility fell to 49% for bugs requiring four or more changes and below 43% for six changes.CodeT5 plausibility was 69% for Python and 44.5–47.8% for JavaScript, Go, and Ruby. Mixing training data from popular languages boosted performance in all tested combinations.Source: §8 · p. 24
  • RunBugRun comprises executable bugs across eight languages, and the authors plan to add further languages.They also plan a parallel dataset of execution traces and identify runtime information as a potential input to machine-learning models.Source: §8 · p. 24
  • The authors identify execution traces as a potential information source for repair, particularly for fault localization.They cite prior work using neural execution-trace embeddings to guide APR search and execution traces to repair incorrectly synthesized programs.Source: §8 · p. 24

A Further Tables and Figures

The appendix presents distributions, example traces and submissions, a label hierarchy, and a candidate-evaluation visualization. Its captions describe what each visual represents, without reporting specific outcomes.

A Further Tables and Figures

  • Figure 8 covers test-case pass, fail, error, and timeout rates before filtering, while Figure 9 shows tests per bug and problem.Figure 10 plots submission line counts, omitting the longest 1.3% as outliers.Source: Figure 8 · p. 30, Figure 9 · p. 30, Figure 10 · p. 31
  • Figure 11 contrasts execution traces for a buggy and fixed test-set program; the traces diverge after the faulty if-condition on line 5.The divergence is highlighted in gray.Source: Figure 11 · p. 31
  • The Ruby training sample has incorrect loop bounds and a wrong variable that cause an exception, while the Java test sample has an if-condition with the wrong operator.The Ruby sample removes empty lines and updates error-message line numbers.Source: Figure 12 · p. 32, Figure 13 · p. 32
  • Figure 14 organizes dataset labels hierarchically, with actual labels at the leaves and label counts in parentheses.Table 6 presents evaluations by bug and candidate, using color for evaluation results.Source: Table 6 · p. 31, Figure 14 · p. 33

Related papers

Loading 2304.01102v1…