Source-linked AI summary
Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT
Chunqiu Steven Xia, Lingming Zhang
TL;DR
Existing LLM-based APR generates and validates patches independently, missing useful test-failure information and lessons from earlier attempts. ChatRepair interleaves generation with automated conversational feedback from failing and plausible patches, achieving 114 and 48 correct fixes on Defects4j 1.2 and 2.0. The paper reports that this conversational approach reaches state-of-the-art repair performance, while plausible patches still require inspection because tests may be incomplete.
Problem
Existing LLM-based APR overlooks test-failure information and earlier incorrect or plausible patches while repeatedly sampling independent candidates.
Method
ChatRepair interleaves patch generation with feedback from relevant test failures and earlier failing or plausible patches in an automated ChatGPT conversation.
Results
114 and 48 correct bug fixes were achieved on Defects4j 1.2 and 2.0, respectively, 15 and 17 more than the best-performing baselines.
Takeaways & Limitations
ChatRepair demonstrates that dialogue-based LLMs can be directly applied to fully automated conversation-driven APR and outperform existing APR techniques.
Takeaways & Limitations
Plausible patches can pass incomplete test suites without correctly fixing the underlying bug, so developers must manually inspect them.
Abstract
from arXiv · showhide
Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then queries the LLM to generate patches. While the LLM-based APR tools are able to achieve state-of-the-art results, it still follows the classic Generate and Validate repair paradigm of first generating lots of patches and then validating each one afterwards. This not only leads to many repeated patches that are incorrect but also miss the crucial information in test failures as well as in plausible patches. To address these limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests, we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM -- ChatGPT. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for \$0.42 each!
1 INTRODUCTION
APR traditionally generates many candidate patches and validates them afterward, while ChatRepair introduces a conversation-driven approach that uses test failures and earlier patches during repair. The approach achieves state-of-the-art results on Defects4j datasets.
- APR tools traditionally follow Generate and Validate by producing many candidate patches before testing them.
- Current LLM-based APR can miss informative test failures, repeated patches, and useful plausible patches from earlier attempts.Test failures may contain concrete code snippets, while plausible patches can share locations or ingredients with correct fixes.
- ChatRepair uses a fully automated conversational APR paradigm that incorporates relevant test information and earlier patch attempts.The system is implemented with ChatGPT and prompts the model using dynamically collected repair information.
- 114 and 48 correct fixes were obtained on Defects4j 1.2 and 2.0, respectively, exceeding the best-performing baselines by 15 and 17.
2 BACKGROUND & RELATED WORK
APR research spans traditional, NMT-based, and LLM-based approaches, with LLMs increasingly used to synthesize patches directly from code context. ChatRepair extends this direction with automated conversational feedback and learning from both failing and plausible patches.
- Large Language Models: ChatGPT and InstructGPT are reinforcement-learning-based LLMs initialized from autoregressive models and fine-tuned using human feedback.
- Automated Program Repair: NMT-based APR translates buggy code into correct code using historical bug-fix datasets, but relies heavily on those training data.
- Large Language Models: LLMs are increasingly applied to APR because they can synthesize code from surrounding context after large-scale pre-training.Examples include cloze-style infilling and direct generation of fixed functions or lines.
- Automated Program Repair: AlphaRepair introduced cloze-style APR, masking buggy lines and using an LLM to fill them from context.
- Automated Program Repair: ChatRepair is presented as the first APR approach to use detailed feedback for every validated patch in a fully automated conversational process.It uses relevant test code and error messages and applies LLMs to extract and interpret test feedback.
3 APPROACH
ChatRepair turns APR into a conversation-driven process: it initializes ChatGPT with buggy code and test-failure information, then interleaves patch generation with validation feedback and learns from failed and plausible patches.
- 3.1 Initial Input: ChatRepair initializes ChatGPT for APR with the buggy function, relevant failing-test information, and task-specific prompting.The initial prompt can include historical fixes, the buggy function and location, the original buggy line, and failing-test details.
- 3.1 Initial Input: Failing-test information includes the test name, relevant triggering code lines, and the produced error message.ChatRepair supplies these details through natural-language prompting rather than custom encodings or handcrafted heuristics.
- 3.2 Conversational Repair: Each generated patch is immediately compiled and tested, and the resulting feedback is appended to the conversation before the next patch.If a patch fails the original test, ChatRepair refers to the initial failure; if another test fails, it provides that test’s name, code, and error message.
- 3.2 Conversational Repair: ChatRepair concatenates the initial prompt with prior patches and validation feedback so ChatGPT can avoid earlier failures.The algorithm limits both the number of queries and the amount of conversation history used for future generation.
- 3.3 Plausible Patch Generation: After finding plausible patches, ChatRepair asks ChatGPT for alternative variations instead of restarting from the original buggy code.This uses information in patches that pass the test suite to generate additional candidates, although developers must still inspect plausibility manually.
4 EXPERIMENTAL DESIGN
The evaluation compares ChatRepair with established APR techniques on Defects4j and QuixBugs across three repair scenarios, while measuring fixes, tries, and API cost.
- Evaluation Scope: ChatRepair is evaluated against state-of-the-art APR tools on the Defects4j and QuixBugs repair datasets.The comparison includes traditional, NMT-based, and LLM-based baselines.
- Repair Scenarios: The study examines single-line, single-hunk, and single-function repair scenarios.These scenarios correspond to replacing or adding one line, a continuous code hunk, or an entire function.
- Datasets: Defects4j 1.2 contains 391 bugs from six Java projects, categorized into 255 single-function, 154 single-hunk, and 80 single-line bugs.Single-hunk bugs are a subset of single-function bugs, and single-line bugs are a subset of single-hunk bugs.
- Baselines: The comparison covers 21 prior APR tools and an additional baseline that directly samples ChatGPT without conversation.The prior tools include eight learning-based or LLM-based systems and twelve traditional APR techniques.
- Metrics: The evaluation measures plausible and correct patches, tries, and dollar cost.ChatGPT access was priced at $0.002 per 1000 processed or generated tokens, with cost depending on query count and input size.
5 EVALUATION
ChatRepair improves automated program repair by combining test-failure information, conversational feedback, and learning from earlier incorrect and plausible patches. Across Defects4j and QuixBugs evaluations, it achieves stronger repair performance, while prompt and conversation design create measurable accuracy–cost trade-offs.
- RQ1: State-of-the-art Comparison: 114 and 48 correct bug fixes on Defects4j 1.2 and 2.0 respectively, with 15 and 17 more than the current state-of-the-art tools.ChatRepair also improves over BaseChatGPT by 34 and 23 fixes on the two Defects4j versions.
- RQ2: Repair Scenarios: ChatRepair achieves the highest number of bugs fixed in each single-line, single-hunk, and single-function repair setting against BaseChatGPT and CodexRepair.The comparison covers Defects4j 1.2 and two QuixBugs datasets.
- RQ2: Repair Scenarios: 4, 7, and 2 additional correctly fixed bugs result from plausible-patch generation in single-line, single-hunk, and single-function repairs on Defects4j 1.2.The step adds averages of 9.4, 16.6, and 5.5 plausible patches in those settings.
- 5.3.1 Initial Prompt: Failing test names, error messages, and exact failure lines improve repair performance over prompts containing only buggy code.The entire failing test body also performs well but has the highest average cost among the evaluated initial prompts.
- 5.3.2 Feedback Response: Dynamic feedback improves performance by supplying new failure information only when a generated patch produces different failing information.Concise feedback when no additional progress occurs can also reduce cost.
- 5.3.2 Feedback Response: Longer conversation histories retain performance but cost more than a conversation length of 3 while increasing the number of plausible patches over repeated independent sampling.Conversation length 1 is equivalent to sampling from the initial prompt without feedback; lengths 5 and 6 incur substantially higher cost than length 3.
6 THREATS TO VALIDITY
The evaluation faces threats from manually judging plausible-patch correctness and possible leakage of reference fixes into ChatGPT’s training data.
- Manual validation determines whether plausible patches are correct relative to the reference developer patch.The authors carefully examined and discussed each patch, following prior work.
- 77 of 212 correct patches matched reference developer fixes, indicating potential training-data leakage.These matching patches represented 36% of the correct patches across the three repair scenarios.
- After removing the 77 matching patches, ChatRepair still fixed 12 unique bugs missed by prior approaches.
- ChatRepair produced 34 more correct fixes than the base ChatGPT repair baseline using the same underlying model.The authors use this comparison to argue that the gains are not simply due to memorizing training data.
7 CONCLUSION
ChatRepair is a fully automated conversation-driven APR tool using ChatGPT, learning from incorrect and plausible patches alongside test-failure information. It achieves state-of-the-art performance on Defects4j 1.2 and 2.0.
- ChatRepair uses conversational repair to learn from incorrect and plausible patches while providing immediate, dynamic test-failure feedback.
- 114 and 48 bugs were fixed on Defects4j 1.2 and 2.0, respectively, achieving state-of-the-art performance.These results were 15 and 17 bugs above the best-performing baseline on the two versions.