Source-linked AI summary
A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair
Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, Zhenyu Chen
TL;DR
Popular APR benchmarks may overlap with data encountered by black-box LLMs during training, complicating assessment of their repair ability. The paper constructs a post-cutoff benchmark and evaluates ChatGPT with different prompts and dialogues. ChatGPT fixes 109 of 151 bugs with a basic prompt, with further fixes from additional prompts and dialogue.
Problem
Because black-box LLM training data are unknown, popular APR benchmarks may have been seen during training, complicating evaluation of repair ability.
Source: §1 · p. 2Method
The authors construct EvalGPTFix from post-cutoff AtCoder submissions and study ChatGPT repair with a basic prompt, added prompt information, and dialogue.
Source: §1 · p. 2Results
109 of 151 bugs are fixed with the basic prompt, and problem descriptions, error messages, bug locations, and dialogues yield additional fixes.
Source: §1 · p. 3, §9 · p. 11Takeaways & Limitations
The results indicate that ChatGPT’s bug-fixing ability can benefit from more detailed prompts and deeper dialogues.
Source: §1 · p. 3Because experiments cover only ChatGPT repairing software bugs, the findings may not generalize to other LLMs or tasks.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Large Language Models (LLMs) have been gaining increasing attention and demonstrated promising performance across a variety of Software Engineering (SE) tasks, such as Automated Program Repair (APR), code summarization, and code completion. For example, ChatGPT, the latest black-box LLM, has been investigated by numerous recent research studies and has shown impressive performance in various tasks. However, there exists a potential risk of data leakage since these LLMs are usually close-sourced with unknown specific training details, e.g., pre-training datasets. In this paper, we seek to review the bug-fixing capabilities of ChatGPT on a clean APR benchmark with different research objectives. We first introduce {\benchmark}, a new benchmark with buggy and the corresponding fixed programs from competitive programming problems starting from 2023, after the training cutoff point of ChatGPT. The results on {\benchmark} show that ChatGPT is able to fix 109 out of 151 buggy programs using the basic prompt within 35 independent rounds, outperforming state-of-the-art LLMs CodeT5 and PLBART by 27.5\% and 62.4\% prediction accuracy. We also investigate the impact of three types of prompts, i.e., problem description, error feedback, and bug localization, leading to additional 34 fixed bugs. Besides, we provide additional discussion from the interactive nature of ChatGPT to illustrate the capacity of a dialog-based repair workflow with 9 additional fixed bugs. Inspired by the findings, we further pinpoint various challenges and opportunities for advanced SE study equipped with such LLMs (e.g.,~ChatGPT) in the near future. More importantly, our work calls for more research on the reevaluation of the achievements obtained by existing black-box LLMs across various SE tasks, not limited to ChatGPT on APR.
1 INTRODUCTION
The paper examines whether data leakage compromises ChatGPT’s apparent bug-fixing ability on familiar APR benchmarks, then evaluates it on a benchmark drawn from post-cutoff programming submissions. It also studies prompt variations and dialogue-based repair.
1 INTRODUCTION
- APR aims to generate correct patches automatically to reduce the time and human resources spent on manual debugging.Learning-based APR can learn repair patterns from code repositories, but its performance depends on the quality and quantity of training data.Source: §1 · p. 2
- ChatGPT may have encountered bugs or patches from popular APR benchmarks during training, making their use an uncertain measure of its repair ability.The authors cite ChatGPT’s awareness of Defects4J and examples of memorization in other models as evidence of the concern.Source: §1 · p. 2
- EvalGPTFix contains 151 Java bug-fix pairs from AtCoder submissions in 2023, after ChatGPT’s stated September 2021 knowledge cutoff.The paper uses this benchmark to study ChatGPT’s repair ability while reducing the risk that benchmark code appeared in training.Source: §1 · pp. 2–3
- 109 of 151 bugs are fixed with the basic prompt; problem descriptions, error messages, and bug locations add 18, 25, and 10 fixes, while dialogues add 9 more.The dialogue fixes apply to bugs not fixed by the basic prompt or the prompt including error information.Source: §1 · p. 3
- The study compares ChatGPT with CodeT5 and PLBART, evaluates prompts containing problem descriptions, error feedback, or bug locations, and explores repair through continued dialogue.The authors also raise data leakage as a concern for black-box LLMs across code-related tasks.Source: §1 · pp. 2–3
2 BACKGROUND
The background contrasts traditional and learning-based automated program repair, then introduces large language models and their common architectures and training objectives.
2.1 Automated Program Repair
- Traditional APR includes heuristic-, constraint-, and template-based techniques; templates can generate patches for specific bugs but cannot cover bugs beyond their patterns or available donor code.Learning-based APR treats repair as translation from buggy to correct code, but its effectiveness depends on training-data quality.Source: §2.1 · pp. 3–4
2.2 Large Language Model
- LLMs use billions of parameters and large training corpora, with encoder-only, decoder-only, and encoder-decoder architectures distinguished by their components and objectives.The passage describes encoder-only models using masked language modeling, decoder-only models predicting the next token, and encoder-decoder models recovering corrupted input.Source: §2.2 · p. 4
3 STUDY DESIGN
The study tests ChatGPT’s repair performance, prompt variations, and interaction with dynamic execution feedback using EvalGPTFix. Its design specifies benchmark construction, repeated model queries, correctness checks, and two comparison models.
3.1 Research Questions
- The research questions ask about ChatGPT’s repair performance on EvalGPTFix, the effects of additional prompt information, and the effect of interaction with dynamic execution feedback.The three questions frame the study’s evaluation objectives.Source: §3.1 · p. 4
3.2 EvalGPTFix Construction
- EvalGPTFix is built from Java contest submissions beginning in 2023, pairing unaccepted and accepted submissions with fewer than six token differences and filtering them using test cases and static and dynamic checks.The resulting benchmark contains 151 bug-fix pairs from 15 problems in AtCoder Beginner Contests 297 and 298.Source: §3.2 · pp. 4–5
3.3 ChatGPT Setup
- The study queries ChatGPT through the gpt-3.5-turbo API repeatedly for each prompt to account for variation in its responses.The passage does not specify a fixed number of repetitions in this study-design description.Source: §3.3 · p. 5
3.4 Evaluation Metrics
- A patch counts as correct if it passes every test case, and a bug counts as fixed when ChatGPT produces a correct patch in any round.The benchmark test cases are used to assess patch correctness.Source: §3.4 · p. 5
3.5 Compared Techniques
- CodeT5 and PLBART are the two state-of-the-art LLM baselines used for comparison.The cited descriptions characterize CodeT5 as a T5-based pretrained model and PLBART as a BERT-based pretrained encoder-decoder model.Source: §3.5 · p. 5
4 RESULTS AND ANALYSIS
The results evaluate ChatGPT on basic bug-fixing prompts, compare it with other repair models, and test additional prompt information and dialogue. ChatGPT fixed bugs across all three study questions, with outcomes depending on repeated queries and prompt design.
4.1 RQ1: Effectiveness of ChatGPT
- 35 query rounds were needed for stable results, with no newly fixed bugs in rounds 33–35.The authors report that additional fixes declined over successive rounds, illustrating response randomness.Source: §4.1 · pp. 5–6, Figure 2 · p. 5
- ChatGPT fixed 96% of CE, 100% of TLE, 50% of RE, and 71% of WA bugs, and also repaired some logic errors.The paper attributes easier CE repairs to recognizable Java syntax errors, while its logic-error example prevents an array-index exception.Source: §4.1 · p. 6
- 109 of 151 bugs were fixed by ChatGPT, 30 more than CodeT5 and 68 more than PLBART; Figure 3 shows their overlap.The reported recall was 72.19%.Source: Figure 3 · p. 5, §4.1 · p. 6
4.2 RQ2: The Impact of Prompt
- Prompt information supplied different repair cues: problem descriptions expose intended behavior, error feedback reports failures, and bug locations narrow the code to inspect.The authors suggest error feedback can yield more variable patches because an error may originate in different code locations; a localized prompt also helped avoid changing an initially correct line.Source: §4.2 · pp. 6–8
- 34 additional bugs were fixed using problem descriptions, error information, or bug locations; the respective gains were 18, 25, and 10 bugs.Only five bugs were fixed by all three prompt types, and each type uniquely fixed some bugs.Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8
4.3 RQ3: Dialogue Study
- Dialogue fixed 9 of the 17 bugs left unfixed by single prompts, using feedback about a patch’s remaining failure to guide subsequent repair attempts.The paper describes cases where feedback reveals another bug after an initial fix or helps ChatGPT correct an earlier mistaken edit.Source: §4.3 · p. 9
5 DISCUSSION
The paper tests whether ChatGPT can repair bugs in code it generated itself, using dialogue-based feedback on AtCoder problems. It fixed only 2 of 13 initially buggy solutions after up to 30 dialogue rounds, and the authors suggest larger repair distances may help explain the contrast with developer-submitted bugs.
5 DISCUSSION
- For each of two AtCoder contests, the study generated 10 solutions per problem and selected the one passing the most test cases for possible repair.Solutions that passed all test cases were excluded; dialogue rounds used the procedure from the earlier dialogue study.Source: §5 · p. 9
- 2 of 13 self-generated buggy solutions were fixed after up to 30 dialogue rounds.ChatGPT initially generated correct solutions for 3 of 16 problems; the remaining 13 were considered for repair.Source: §5 · p. 9
- Among 16 contest problems, ChatGPT solved 3 initially and fixed only 2 of the remaining 13 bugs after 30 dialogue rounds, suggesting limited self-repair; its initial programs may require more edits than developer-submitted programs, whose edit actions differed by less than 6 tokens.Source: §5 · p. 9
6 CHALLENGE AND OPPORTUNITY
The discussion identifies prompt design, model specialization, and IDE integration as directions for future work. It notes that advanced prompts can further boost repair performance, while real-world assistance and the trade-offs between general-purpose and code-specialized LLMs remain open areas to explore.
6 CHALLENGE AND OPPORTUNITY
- Advanced prompts can further boost repair performance, and future work should explore prompts that combine clear task instructions, useful context, and examples.The paper describes these as elements of a well-crafted bug-fixing prompt for guiding accurate and helpful patches.Source: §6 · pp. 9–10
- General-purpose LLMs offer broad language understanding, while domain-specific LLMs may provide more specialized code guidance and potentially more accurate patches.The paper identifies exploring the advantages and disadvantages of both types, and combining general language understanding with specialized domain knowledge, as future directions.Source: §6 · p. 10
- It remains unclear how ChatGPT performs in real-world development environments; the paper proposes exploring iterative IDE workflows that accept natural-language bug descriptions and reproduction details.The discussion contrasts this potential workflow with APR techniques that accept buggy code and may produce numerous ineffective candidate patches.Source: §6 · p. 10
7 THREATS TO VALIDITY
The threats to validity concern EvalGPTFix’s benchmark scope, the selected baselines, and the focus on ChatGPT and APR. The authors note that contest algorithms may not reflect professional repair, and findings may not generalize beyond the chosen model and task.
7 THREATS TO VALIDITY
- EvalGPTFix contains small contest algorithms that may not accurately reflect real-world professional software repair capabilities.The authors also state that its programs vary in difficulty and type and come from programmers with diverse backgrounds.Source: §7 · p. 10
- The comparison uses CodeT5 and PLBART, but excludes other LLMs and existing APR techniques because the study focuses on LLMs in software engineering.The authors selected the two LLM baselines as state of the art in bug-fixing.Source: §7 · p. 10
- Because the experiments evaluate only ChatGPT on APR, the findings may not generalize to other LLMs or software engineering tasks.The authors regard ChatGPT and APR as representative examples, not as the full scope of their experiments.Source: §7 · p. 10
8 RELATED WORK
Related work spans template-based and learning-based automated program repair, followed by research on LLMs in software engineering and ChatGPT for repair. The paper places its study within work on ChatGPT while focusing on data leakage in black-box LLM evaluation.
8.1 Automated Program Repair
- Template-based repair applies predefined fix patterns, while learning-based approaches learn transformations from code and bug-fix data.Examples include PAR’s manually written patterns, TBar’s literature-derived templates, AVATAR’s static-analysis violations, and FixMiner’s clustering-based template mining.Source: §8.1 · p. 10
- Learning-based APR examples include DLFix’s two-layer model and CURE’s pretrained code model with search and tokenization strategies.The paper excludes these APR systems as baselines, citing prior evidence that LLMs can outperform APR approaches and its focus on comparing LLMs.Source: §8.1 · p. 10
8.2 Large Language Models in SE
- Prior LLM research spans program understanding and generation, including studies of pretrained models, AlphaRepair’s cloze-style repair, and Codex’s bug-fixing ability.The cited work reports Codex as especially effective on Python bugs in QuixBugs.Source: §8.2 · p. 11
- Earlier ChatGPT repair studies examined popular datasets, conversational repair behavior, deep-learning programs, and feedback-driven repair approaches.This paper instead examines data leakage in black-box LLMs using ChatGPT and APR as examples.Source: §8.2 · p. 11
9 CONCLUSION
The paper evaluates ChatGPT’s repair ability on EvalGPTFix to examine data leakage using a clean dataset. It reports 109 correct patches from the basic prompt, with further fixes from added prompt information and dialogue.
9 CONCLUSION
- ChatGPT produces 109 correct patches over 151 bugs on EvalGPTFix with only the basic prompt.The paper presents the clean benchmark as a way to examine the data-leakage issue for black-box LLMs.Source: §9 · p. 11
- Problem descriptions, error messages, and bug localizations yield 18, 25, and 10 additional fixes, respectively.Dialogue fixes nine more bugs.Source: §9 · p. 11
- The authors conclude that ChatGPT has promising bug-fixing ability that can be enhanced by detailed prompts and more dialogue.They call for reevaluating black-box LLM achievements across software engineering tasks, beyond ChatGPT on APR.Source: §9 · p. 11