Source-linked AI summary
How to Design a Program Repair Bot? Insights from the Repairnator Project
Simon Urli, Zhongxing Yu, Lionel Seinturier, Martin Monperrus
TL;DR
Prior repair evaluations generally assessed algorithms, although practical use also needs failure detection, reproduction, and patch reporting. This paper presents Repairnator, an operating bot that monitors CI failures and attempts repairs; it generated patches for 15 bugs, none proposed because they overfit. The paper shares operational experience and recommendations for future repair bots.
Problem
Prior program-repair evaluations generally focused on repair algorithms, while practical use also needs failure detection, bug reproduction, and patch reporting.
Source: §1 · p. 1Method
Repairnator monitors CI test failures, attempts to reproduce them, runs public repair tools, and collects and reports attempt information.
Source: §1 · p. 1Results
15 bugs had generated patches, but none were proposed to developers because they overfit.
Source: §1 · p. 1Takeaways & Limitations
The authors share Repairnator’s operating experience through seven recommendations to help future developers design and operate repair bots.
Source: §1 · p. 2, §7 · p. 10GitHub exposes only the last merge commit of a pull request, which can create a discrepancy between the CI failure and the build failure considered.
Source: §4.3.1 · p. 7
Abstract
from arXiv · showhide
Program repair research has made tremendous progress over the last few years, and software development bots are now being invented to help developers gain productivity. In this paper, we investigate the concept of a " program repair bot " and present Repairnator. The Repairnator bot is an autonomous agent that constantly monitors test failures, reproduces bugs, and runs program repair tools against each reproduced bug. If a patch is found, Repairnator bot reports it to the developers. At the time of writing, Repairnator uses three different program repair systems and has been operating since February 2017. In total, it has studied 11 317 test failures over 1 609 open-source software projects hosted on GitHub, and has generated patches for 17 different bugs. Over months, we hit a number of hard technical challenges and had to make various design and engineering decisions. This gives us a unique experience in this area. In this paper, we reflect upon Repairnator in order to share this knowledge with the automatic program repair community.
1 INTRODUCTION
Prior program-repair evaluations focused mainly on repair algorithms, while practical use also requires failure detection, bug reproduction, and patch reporting. The paper presents Repairnator as an operating repair bot and shares its design experience and results.
1 INTRODUCTION
- Practical program repair requires failure detection, bug reproduction, and patch reporting in addition to evaluating repair algorithms.The authors argue that an end-to-end toolchain suited to mainstream development practices is desirable for industrial use.Source: §1 · p. 1
- A program repair bot monitors test failures, reproduces bugs, runs repair tools, and reports any found patch to developers.The authors note that, to their knowledge, no such bot’s design and operation had previously been reported.Source: §1 · p. 1
- Repairnator monitors continuous-integration failures, attempts reproduction, runs public repair tools, and collects and reports information about each attempt.The project designs, implements, and operates a repair bot for Java programs.Source: §1 · p. 1
- 11,523 test failures across 1,609 GitHub projects yielded patches for 15 bugs, but none were proposed because they overfit.Repairnator used three repair systems and had operated since February 2017.Source: §1 · p. 1
- The authors share design and engineering lessons from months of technical challenges operating Repairnator with the automatic program repair community.The paper reports results from 11 months of operation and makes its discussed data available.Source: §1 · pp. 1–2
- The paper contributes a bot blueprint, empirical facts about repair and bug reproduction, and seven recommendations for future repair bots.The reported contribution covers 11,523 CI failures across 1,609 software projects.Source: §1 · p. 2
2 OVERVIEW OF REPAIRNATOR
Repairnator is designed to propose patches for open-source bugs before developers fix them. Its workflow takes CI builds through analysis, failure reproduction, and patch synthesis, with design choices focused on GitHub, Java, and test failures.
2 OVERVIEW OF REPAIRNATOR
- Repairnator aims to propose patches for open-source bugs before developers fix them.The project is presented as a way to make scientific progress in program repair.Source: §2 · p. 2
2.1 Terminology
- CI builds record the source snapshot, execution outcome, and log; pull requests propose external changes for project maintainers to review.A merge commit combines branches and is automatically created when a pull request is accepted.Source: §2.1 · p. 2
2.2 Repairnator Workflow
- The workflow monitors configured projects, analyzes GitHub CI builds, reproduces failures, and uses reproduction information to search for patches.A patch analyst reviews generated patches before any correct patch is proposed to the original developers; the bot also collects research data.Source: §2.2 · pp. 2–3
2.3 Main Design Choices
- Repairnator targets GitHub and TravisCI, Java programs, and failed CI builds caused by test failures.The authors cite access to development information and available Java repair tools as design reasons; build-script and compilation failures are outside its stated focus.Source: §2.3 · p. 3
3 CI BUILD ANALYSIS
Repairnator selects projects and filters recent CI builds to identify test failures for repair. Its results show that many collected builds are not useful repair candidates, while the recommendations favor current service APIs and CI metadata over log parsing.
3.1 Approach
- The analysis stage selects projects once, then analyzes their builds at the start of each Repairnator run to produce candidates for repair.The weekly build chart distinguishes collected builds, Java CI failures, and builds with identified JUnit test failures.Source: §3.1 · p. 3, Figure 3 · p. 4, §3.2 · p. 4
3.1.1 List of Considered GitHub Projects.
- The project list combines TravisTorrent and GHTorrent queries, then filters candidates using Java/Maven and TravisCI information.TravisTorrent contains fewer repository entries than GHTorrent, while CI and hosting-service APIs can provide more current activity data than database metadata.Source: §3.1.1 · p. 3, §3.3.1 · p. 4
3.1.2 Analyzing Build Information.
- Repairnator considers recent CI-failing builds with at least one failing test; it checks CI status through the API and uses logs to identify test failures.Runs were usually launched every four hours and considered builds completed within the previous four hours.Source: §3.1.2 · p. 3
3.2 Results
- TravisCI was used by 22.14% of the 14,188 Java projects, while 306 projects (2.16%) had no passing TravisCI build.The authors say this often indicates an incorrectly configured TravisCI setup.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
- Among active TravisCI projects, 1,439 (10.14%) used Maven and 1,038 (7.32%) used Gradle; the authors justify focusing repair on Maven.For 358 projects (2.52%), the build tool could not be identified automatically.Source: §3.2 · p. 4
- Most collected builds were successful, and only about half of CI-failing builds were identified as containing test failures.The authors say this filtering is needed before attempting reproduction.Source: §3.2 · p. 4
- Around 50% of detected builds came from pull requests; Table 1 reports the ten most active projects by build count and their test-failing and pull-request builds.The authors note that prestodb/presto had 1,000 test-failing builds.Source: §3.2 · p. 4, Table 1 · p. 5
3.3 Recommendations
- The recommendations are to use CI and code-hosting APIs for an up-to-date project list and rely on CI metadata rather than error-prone log parsing where possible.The authors state that these recommendations will be applied in a future Repairnator version.Source: §3.3.1 · p. 4, §3.3 · p. 4, §3.3.2 · p. 5
4 LOCAL BUG REPRODUCTION
Local reproduction starts from an interesting failing build and aims to recover test outcomes. The process checks out the relevant code, compiles it, runs tests, and observes their results.
4.1 Approach
- The reproduction stage takes interesting failing builds as input and produces test information, including test counts, names, elapsed time, and exceptions.Its four steps are checking out the code, compiling it, executing tests, and observing their outcomes.Source: §4.1 · p. 5
4.1.1 Checking out the code.
- For pull-request builds, Repairnator recreates TravisCI’s merge commit to obtain the same source; reproduction stops if the commit is unavailable.Unavailable commits can result from amended commits or force-pushed branch updates.Source: §4.1.1 · p. 5
4.1.2 Compiling the code.
- Compilation uses mvn install, with a separate Maven cache for each reproduction attempt to avoid conflicting dependency versions.Repairnator skips additional checks such as checkstyle and code coverage because they are not relevant to its test-suite-based repair tools.Source: §4.1.2 · p. 5
4.1.3 Executing the tests.
- Tests run with mvn test, while Maven Failsafe integration tests are skipped because they may require resources unavailable in Repairnator’s environment.Source: §4.1.3 · p. 5
4.1.4 Observing test outcomes.
- Test outcomes are observed through Maven and JUnit APIs, using standard test drivers such as JUnit or TestNG.Source: §4.1.4 · p. 5
4.2 Results
Repairnator locally reproduced 3,551 of 11,523 failing builds, while reproduction success and failure types varied across projects. The results identify recurring obstacles to reproduction and show that a small set of failure types accounts for most observed cases.
4.2 Results
- 30.82% of 11,523 failing builds were reproduced locally by Repairnator.The authors attribute the difficulty to interacting build tools and numerous version differences.Source: §4.2 · p. 6
- 4,305 builds (37.36%) stopped at compilation; 2,130 (18.48%) lacked a locally reproduced test failure, and 1,172 (10.17%) hit test-harness errors.Compilation issues included manually installed dependencies or Java-version requirements; differing JDKs and flaky tests could prevent reproducing failures.Source: §4.2 · p. 6
- 334 builds (2.90%) failed at checkout, often because a commit was deleted, while 31 (0.27%) failed to clone, sometimes due to disk-space or network issues.Source: §4.2 · p. 6
- Reproduction varied by project: Druid had 359 of 579 failures reproduced (62%), versus 19.40% for Presto and 94% for jedis.Table 2 covers the ten projects with the most reproduced failing builds.Source: Table 2 · p. 6, §4.2 · p. 6
- AssertionError was the most common failure type, with more than 2,100 occurrences, and NullPointerException was also extremely frequent.Table 3 presents the most common collected test failure types.Source: Table 3 · p. 6, §4.2 · p. 6
- Ten failure types made up almost 70% of observed types (4,452 of 6,380), suggesting that targeting specific failure types may be an effective repair-tool strategy.Source: §4.2 · p. 6
4.3 Recommendations
The recommendations focus on reproducing CI failures faithfully, especially by matching the code state and isolating dependencies.
4.3.1 The problem of merge commits.
- For pull requests, GitHub exposes only the latest merge commit, so the reconstructed build can differ from the CI failure under study.Source: §4.3.1 · p. 7
- Recreate the pull-request merge commit to obtain the same code state as CI.Source: §4.3.1 · p. 7
4.3.2 Managing the dependencies.
- Isolate dependency managers and caches between reproduction attempts; reused dependencies can differ from those in the original bug and produce spurious behavior.For Maven, the authors recommend a separate local cache per attempt.Source: §4.3.2 · p. 7
5 PATCH SYNTHESIS
Patch synthesis is Repairnator's final stage and its core program-repair algorithm. The section describes the approach, reports results, and discusses findings.
5 PATCH SYNTHESIS
- Patch synthesis is the final stage of the Repairnator bot and corresponds to its core program-repair algorithm.Source: §5 · p. 7
- The section first describes Repairnator's patch-synthesis approach.Source: §5 · p. 7
- The section then presents synthesis results and discusses the findings.Source: §5 · p. 7
5.1 Approach
Repairnator gathers project and failure information, runs suitable repair tools, and routes candidate patches through analyst review before reporting them to developers.
5.1 Approach
- Patch synthesis gathers project information, launches the repair tools, and reports resulting patches, if any.Source: §5.1 · p. 7
5.1.1 Gathering project information.
- Repair tools receive the likely source location, failing-test names, failure type, and compile-time and run-time dependencies.Source locations target patch synthesis; failing-test names avoid rerunning the full suite; failure type informs tool selection; dependencies support dynamic analysis.Source: §5.1.1 · p. 7
5.1.2 Launching automatic repair tools.
- NPEFix is used for NullPointerException bugs, while Astor and Nopol are used in all repair attempts.Astor is a generate-and-validate tool; Nopol repairs conditional bugs, and Repairnator uses its dynamic synthesizer.Source: §5.1.2 · p. 7
5.1.3 Reporting patches to developer.
- A patch analyst checks whether a patch fixes the failure and avoids overfitting before proposing it to developers; Repairnator also archives patches and execution logs for research.Overfitting patches pass the test suite but are too specific to correct the whole buggy behavior.Source: §5.1.3 · p. 7
5.2 Results
Repairnator’s Nopol and NPEFix runs produced many test-suite-adequate patches, but the patches were considered overfitting. The results illustrate the gap between passing tests and producing a suitable fix.
5.2 Results
- Nopol, NPEFix, and Astor entered Repairnator at different times, but the reported results cover only Nopol and NPEFix.Source: §5.2 · p. 7
- All of those patches were considered overfitting, reinforcing overfitting as a major barrier to using program repair in industry.Source: §5.2 · pp. 7–8
- 15 builds yielded 1 307 test-suite-adequate patches among 3 551 successfully reproduced bugs, with two Nopol executions producing almost all patches.Table 4 covers projects with at least one Nopol or NPEFix patch; the results exclude Astor because its data was insufficient.Source: §5.2 · p. 7, Table 4 · p. 8
- One test-suite-adequate patch used a condition that always evaluates true, whereas the human fix checked for allowed value types.Source: §5.2 · p. 8
5.3 Recommendations
The recommendations address three practical constraints: reproducing the original failure, repairing bugs spanning Maven modules, and reducing bot response time. Each points to a specific engineering boundary or design choice.
5.3.1 The problem of spurious bugs.
- Because Repairnator does not reproduce developer build scripts or TravisCI environments, it cannot ensure a local failure matches the original; replicating both is recommended.The authors note that environment differences can cause spurious reproduction results.Source: §5.3.1 · p. 8
5.3.2 About multi-module projects.
- Current repair tools cannot fix failures spanning Maven modules, so the authors recommend accounting for multi-module projects from the start of tool design.Source: §5.3.2 · p. 8
5.3.3 About Response Time.
- Response time sums commit-to-run delay, Repairnator execution, and analyst validation; CI hooks could reduce the first component, but require project-owner activation.Repairnator’s periodic runs can begin up to four hours after a commit.Source: §5.3.3 · p. 9
6 RELATED WORK
The related work surveys software development bots and test-suite-based repair. It covers developer-facing automation and trust, repair strategies, and the risk that patches satisfy incomplete test suites without preserving desired behavior.
6.1 Software Development Bots
- Prior work discusses bot integration into developer workflows and trust, and reports higher confidence in answers from a human-presented bot than a disclosed programmatic bot.Source: §6.1 · p. 9
- Other development bots propose code-contract fixes [5], automate code review or defect fixes, and envision bots submitting bug-fix or documentation pull requests.Source: §6.1 · p. 9
6.2 Program Repair
- Test-suite-based repair uses a buggy program and tests to generate patches; approaches include generate-and-validate search and synthesis from execution-derived constraints [31].Source: §6.2 · p. 9
- Because test suites are rarely exhaustive, test-suite-based repair can produce patches that pass the tests but break untested desired behavior.Several studies have explored test-case generation to alleviate overfitting [34].Source: §6.2 · pp. 9–10
7 CONCLUSION
The authors present Repairnator as an operational repair bot and share seven recommendations from its use. Although it had not proposed an effective patch to a developer, it generated empirical data on repair challenges.
7 CONCLUSION
- The authors envision software repair bots becoming a standard tool for maintaining large code bases.Source: §7 · p. 10
- Repairnator ran for 11 months, reproduced many failing CI builds, and generated patches for 15 builds; the paper shares seven recommendations for future repair bots.Source: §7 · p. 10
- Repairnator had not yet proposed an effective patch to a human developer, but its operation enabled collection of empirical data on program-repair challenges.Source: §7 · p. 10