Source-linked AI summary
BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies
Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, Eng Lieh Ouh
TL;DR
Python’s growing use and distinctive project characteristics exposed a need for more testing and debugging research supported by reproducible bug data. BugsInPy addresses this need with a hand-curated, extensible benchmark of real Python bugs and supporting tools. It contains 493 bugs from 17 real-world projects, with artifacts and execution capabilities designed for controlled studies.
Problem
Python research lacks curated, reproducible bug data for empirically evaluating automated testing and debugging solutions.
Method
BugsInPy constructs a hand-curated benchmark and tool containing real, reproducible, isolated Python bugs with exposing tests and framework support.
Results
BugsInPy contains 493 real bugs from 17 real-world Python projects and supports controlled testing and debugging studies.
Takeaways & Limitations
BugsInPy provides an extensible foundation for reproducible research on testing and debugging tools for Python programs.
Takeaways & Limitations
The dataset may still contain mislabeled bugs, and the benchmark’s supporting programs may contain undetected bugs.
Abstract
from arXiv · showhide
The 2019 edition of Stack Overflow developer survey highlights that, for the first time, Python outperformed Java in terms of popularity. The gap between Python and Java further widened in the 2020 edition of the survey. Unfortunately, despite the rapid increase in Python's popularity, there are not many testing and debugging tools that are designed for Python. This is in stark contrast with the abundance of testing and debugging tools for Java. Thus, there is a need to push research on tools that can help Python developers. One factor that contributed to the rapid growth of Java testing and debugging tools is the availability of benchmarks. A popular benchmark is the Defects4J benchmark; its initial version contained 357 real bugs from 5 real-world Java programs. Each bug comes with a test suite that can expose the bug. Defects4J has been used by hundreds of testing and debugging studies and has helped to push the frontier of research in these directions. In this project, inspired by Defects4J, we create another benchmark database and tool that contain 493 real bugs from 17 real-world Python programs. We hope our benchmark can help catalyze future work on testing and debugging tools that work on Python programs.
1 INTRODUCTION
Python’s popularity and distinctive software characteristics motivate more Python-focused software engineering research, while the lack of curated bug datasets makes empirical testing and debugging studies costly. BugsInPy addresses this gap with a reproducible, extensible benchmark framework modeled after Defects4J.
- Python’s diverse domains and features, including duck typing and heterogeneous collections, may produce bug characteristics that differ from other languages.
- Curated collections of known bugs are needed for empirical, reproducible evaluation of automated testing and debugging solutions.
- BugsInPy adapts Defects4J’s benchmark model for Python, emphasizing real-world, reproducible, isolated bugs with tests that pass after fixes.
- The framework is intended to reduce entry barriers for developing and evaluating Python testing and debugging tools.
- 493 bugs from 17 real-world Python projects span machine learning, developer tools, scientific computing, and web frameworks.The projects are open-source GitHub repositories with more than 10,000 stars; manual construction and validation took an estimated 831 man-hours.
- BugsInPy combines a bug database, database abstraction layer, and test execution framework supporting testing, input generation, mutation analysis, and coverage analysis.
2 DETECTING BUGS FROM VERSION CONTROL HISTORY
BugsInPy is constructed by mining version-control history for developer-fixed Python source bugs, then validating reproducibility and isolation through tests, source inspection, and manual review.
- Each database bug must involve a source-code change, be reproducible by a failing test, and be isolated from unrelated changes.
- Identify Real Bugs: Researchers prioritize commits modifying test files, inspect commit messages and source changes, and consult linked GitHub issues when available.
- Identify Real Bugs: Only non-test Python source changes are considered, excluding fixes to configurations, build scripts, documentation, and test cases.
- Reproduce Real Bugs: A fixing commit is retained only when at least one test fails on the faulty version and succeeds on the fixed version.
- Isolate Real Bugs: Candidate bug-fixing commits are manually checked for isolation so unrelated refactoring or feature additions do not confound failing tests.
3 DATABASE OF REAL PYTHON BUGS
The BugsInPy database contains 493 real Python bugs from 17 popular open-source projects and records the revisions, isolated patches, and tests needed to expose them.
- 493 real Python bugs were collected from 17 open-source Python 3 projects on GitHub with more than 10K stars and PyPI availability.
- The project set was examined through commits on each repository’s master branch.
- Each bug records version-control revisions, the original isolated source patch, and test cases that expose the bug.
4 DATABASE ABSTRACTION LAYER
BugsInPy’s database abstraction layer hides repository and build details, allowing users to retrieve faulty or fixed code, compile projects, and run tests through uniform interfaces.
- The abstraction layer provides interfaces for accessing source code, build systems, and bug artifacts without exposing underlying commands or technologies.
- Source code access: A bug’s unique BugsInPy identifier is linked internally to its original Git revision hash for checking out faulty or fixed versions.
- Build systems: Build automation records project-specific dependency installation and setup procedures so users need not configure projects manually.
- Build and testing: The build process compiles projects and runs tests using project dependencies and the project’s unittest or pytest framework.
5 TEST EXECUTION FRAMEWORK
BugsInPy provides a test execution framework that simplifies common testing and debugging tasks by integrating existing tools and accessing faulty or fixed code through its abstraction layer.
- The framework supports test set selection, test input generation, mutation analysis, and code coverage computation.It is designed to minimize the effort required for these common testing and debugging tasks.
- Test Set Selection: Users can execute a single test case, all test cases, or any subset against faulty or fixed source-code versions.
- Test Input Generation: BugsInPy supports generating new test inputs through fuzzing.
- The framework runs on the database abstraction layer to access source versions and test cases while managing external testing tools and their generated data.
6 THREATS TO VALIDITY
BugsInPy uses independent manual review to reduce bug-labeling errors, but bugs may remain in the benchmark infrastructure despite repeated checks.
- Two authors must independently agree and be confident when deciding whether a commit fixes a bug or whether a bug is isolated.Bugs are discarded when consensus is not reached.
- The dataset includes only bugs on which the authors are highly confident.
- The programs supporting BugsInPy, including its test execution framework, may still contain bugs that the authors did not encounter.The authors tried to ensure these programs were bug-free and checked them multiple times.
7 RELATED WORK
Related benchmarks differ in language coverage, bug provenance, project scale, and infrastructure, while BugsInPy targets real bugs in popular Python projects.
- SIR contains 81 bugs across Java, C, C++, and C#, but only 35 are real bugs and the remainder come from mutation analysis.
- Defects4J contains 357 real bugs from 5 real-world Java programs and is the closest related benchmark to BugsInPy.
- Bugswarm automatically mines failing and subsequently passing Travis builds but requires downloading many Docker containers, whereas BugsInPy provides one container.
- BugsJS provides 453 real bugs from 10 JavaScript programs, while Defexts provides 225 Kotlin and 301 Groovy bugs.
- QuixBugs contains small, synthetically created Java and Python programs with seeded bugs, unlike BugsInPy’s 493 real bugs from 17 popular Python projects.
8 CONCLUSION AND FUTURE WORK
BugsInPy is an extensible, hand-curated framework for reproducible studies on real bugs in Python projects, currently covering 493 bugs from 17 real-world projects.
- BugsInPy enables controlled testing and debugging studies requiring experiments on real bugs in Python projects.
- 493 bugs from 17 real-world projects make BugsInPy the largest Python bug dataset to date.
- Hand curation is used to ensure that the bugs are reproducible and isolated.
- Future work includes adding more projects and bugs and integrating BugsInPy with more testing and debugging tools.Adding new projects and bugs requires manual effort, but the effort is described as one-time before bugs can be reproduced easily.