Source-linked AI summary
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
Dominik Sobania, Martin Briesch, Carol Hanna, Justyna Petke
TL;DR
The paper asks whether ChatGPT can reliably fix software bugs, since its bug-fixing quality was unclear. It evaluates ChatGPT on QuixBugs and compares it with Codex, CoCoNut, and standard APR methods, finding comparable deep-learning performance and stronger results than standard APR, with dialogue hints improving success further.
Problem
ChatGPT’s quality for automatic software bug fixing was unclear despite the need for automated program repair.
Method
The study evaluates ChatGPT on the 40-problem QuixBugs benchmark and compares it with Codex, CoCoNut, and reported dedicated APR results.
Results
19 of 40 bugs were fixed without additional information, versus 7 for standard APR; follow-up hints increased success to 77.5%.
Takeaways & Limitations
ChatGPT’s dialogue option lets users provide bug information, and human input can substantially help automated program repair.
Abstract
from arXiv · showhide
To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT's bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its success rate can be further increased, fixing 31 out of 40 bugs, outperforming state-of-the-art.
I. INTRODUCTION
The paper motivates evaluating ChatGPT for automated program repair because software bugs can have costly consequences, while existing approaches have important limitations. It evaluates ChatGPT on QuixBugs and compares it with Codex, CoCoNut, and standard repair methods.
- Uncorrected software bugs can cause failures in essential systems and high economic costs.
- Standard APR methods mutate code or synthesize repairs, but their test-suite and constraint-solver validation strategies can take hours.
- Deep-learning repair methods learn bug-fixing patterns and generate ranked patches, but their outputs may not compile because they are usually not automatically verified.
- ChatGPT extends source-code assistance with conversational context, allowing users to provide additional information about bugs.
- 19 of 40 ChatGPT repairs succeeded, compared with 19 for CoCoNut, 21 for Codex, and 7 for standard APR methods.
II. CHATGPT FOR AUTOMATED PROGRAM REPAIR
This section presents the methodology used to assess ChatGPT’s program repair performance.
- The section introduces the methodology for assessing ChatGPT’s program repair performance.
- The assessment focuses on ChatGPT’s performance in program repair.
- The stated methodology concerns automatic evaluation of ChatGPT for program repair.
A. Benchamrk
The benchmark study uses QuixBugs to evaluate ChatGPT on small programs, repeatedly requesting and manually checking proposed fixes.
- A. Benchamrk: QuixBugs provides 40 relatively small benchmark problems suitable for use in a dialogue system.
- A. Benchamrk: For each problem, the researchers remove comments and ask ChatGPT whether the Python code contains a bug and how to fix it.
- A. Benchamrk: The researchers make several independent requests for each problem and manually check the answers.
B. Comparison Study
The comparison study evaluates ChatGPT against standard APR methods and deep-learning-based approaches using reported benchmark results.
- B. Comparison Study: The study runs four independent ChatGPT requests for each QuixBugs problem.
- B. Comparison Study: Standard APR results come from a comprehensive literature study covering ten methods on QuixBugs.
- B. Comparison Study: CoCoNut results are taken from recent reported results for dedicated deep-learning APR approaches.
C. Dialogue Study
The dialogue study gives ChatGPT one standardized hint after an initial incorrect response, using an example input and expected output to clarify the bug.
- Hint design: After an incorrect first response, the researchers tell ChatGPT that the function fails and provide an input example demonstrating the problem.When ChatGPT incorrectly claims the program is correct, the reply explicitly states that the function does not work and supplies the relevant input.
- Hint design: For more complex inputs, the standardized hint includes a code snippet and the output that should result.The experiment provides only one such hint for each applicable case.
- Experimental setting: The hint experiment used the ChatGPT version available on January 9, 2023.
III. RESULTS AND DISCUSSION
The results section compares ChatGPT with Codex, CoCoNut, and standard automated program repair approaches, while also analyzing ChatGPT’s response behavior.
- Comparison: The study compares ChatGPT, Codex, CoCoNut, and standard automated program repair approaches.
- Response analysis: The researchers classify ChatGPT’s answers and discuss observations from working with the model.
A. Automatic Bug Fixing Performance
ChatGPT performs similarly to Codex and CoCoNut while outperforming standard APR approaches on QuixBugs. Its results also show run-to-run variance and require strict bug-fix validation.
- Evaluation: The comparison counts a ChatGPT fix when at least one of four runs gives a correct answer, while literature results report whether a correct fix is reported.
- Performance comparison: 19 benchmark problems are fixed by ChatGPT, compared with 21 by Codex, 19 by CoCoNut, and 7 by standard APR approaches.
- Evaluation: Standard APR approaches would solve 16 problems under test-suite-only evaluation, but only 7 generalizing problems are counted as correct.The stricter count reflects the requirement that fixes work on unseen inputs, not merely the available test suite.
- Variability: ChatGPT often solves problems in only one or two runs, and only BUCKETSORT and FLATTEN are fixed in all four runs.The authors therefore describe ChatGPT’s bug-fixing behavior as having relatively high variance.
- Discussion: ChatGPT’s responses are often close to the correct solution, indicating potential for improvement despite solving about as many problems as Codex.
- Evaluation: The evaluation counts only patches that identify and correct the QuixBugs bug, excluding unlocalized reimplementations and fixes requiring additional changes.
B. A Classification of ChatGPT’s Answers
The researchers classify ChatGPT’s responses into recurring answer types and find that requests for more information and claims that no bug exists are most common.
- Response classes: The researchers identify six response classes: more information required, no bug found, correct fix provided, tries to fix something else, new bug introduced, and alternative implementation.
- Response classes: Figure 2 reports how often each identified ChatGPT answer class occurs across the QuixBugs problems.
- Observed frequencies: Requests for more information are most frequent, while claims that the code has no bug are the second most common response.
- Dialogue use: Providing further information in the dialogue may lead to a correct bug fix in both response situations.
- Failure modes: Less frequent responses fix the target bug while introducing new errors or replace the intended repair with a working reimplementation.
C. A Discussion with ChatGPT
The section examines ChatGPT conversations for two QuixBugs problems and presents benchmark results for ChatGPT with additional follow-up information. The study also notes that the evaluation involved manual classification and a standardized Python benchmark.
- Benchmark comparison: Table I reports results for ChatGPT, Codex, CoCoNut, and standard APR approaches on QuixBugs, including ChatGPT’s successful-run counts.The table covers the benchmark problems and reports successful runs for ChatGPT in brackets.
- Answer categorization: Figure 2 counts the occurrences of identified classes of ChatGPT answers across the QuixBugs problems.The figure summarizes how frequently each answer class appeared.
- Conversational examples: Figures 3 and 4 show ChatGPT discussions for the GCD and BITCOUNT problems, respectively, using request and response markers for readability.The BITCOUNT example also shortens some responses and marks omitted large passages.
- Conversational examples: The conversational study investigates whether adding detail through dialogue can help resolve ambiguity in textual descriptions or test-suite specifications.The examples focus on the GCD and BITCOUNT benchmark problems.
- Evaluation considerations: The evaluation used a standard benchmark set and Python, while manually classified results remain subjective; the authors made conversations available for verification.These choices were intended to mitigate threats concerning benchmark and language selection and enable result checking.
V. CONCLUSIONS AND FUTURE WORK
The paper evaluates ChatGPT’s bug-fixing quality against Codex and dedicated APR approaches. ChatGPT performs similarly to deep-learning systems, exceeds standard APR results, and improves further when users provide additional bug information, although answer verification may impose mental costs.
- Conclusions: ChatGPT’s bug-fixing quality was compared with Codex and several dedicated APR approaches because its effectiveness had been unclear.The comparison addresses ChatGPT’s use as a dialogue-based system for improving erroneous source code.
- Conclusions: ChatGPT achieved similar performance to Codex and dedicated deep-learning APR, fixed 19 versus 7 out of 40 bugs compared with standard APR, and reached 77.5% with dialogue.The dialogue condition supplied more information about unsolved bugs in follow-up requests.
- Future work: The authors identify the mental cost of verifying ChatGPT’s answers as an unresolved consideration for practical use.They suggest automated hints and response verification as possible future directions.