Source-linked AI summary
Conversational Automated Program Repair
Chunqiu Steven Xia, Lingming Zhang
TL;DR
Prior LLM-based program repair repeatedly samples from the same input rather than using testcase validation feedback. Conversational APR alternates patch generation and validation, and its evaluation reports improved results over baseline sampling across 10 LLMs.
Problem
Prior LLM-based APR repeatedly samples from the same input rather than using testcase validation feedback to guide later patch generation.
Source: §1 · p. 2Method
Results
Conversational APR improves correct and plausible patch counts across unsupervisedly trained LLMs, reduces tries to a first plausible or correct patch, and improves over baseline sampling across 10 LLMs.
Source: §6.1 · p. 7, §7 · p. 9Takeaways & Limitations
The evaluation supports conversational APR as a promising direction for applying LLMs to program repair.
Source: §7 · p. 9The authors assume ChatGPT has a parameter count similar to GPT-3.5 because it is fine-tuned on GPT-3.5.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Automated Program Repair (APR) can help developers automatically generate patches for bugs. Due to the impressive performance obtained using Large Pre-Trained Language Models (LLMs) on many code related tasks, researchers have started to directly use LLMs for APR. However, prior approaches simply repeatedly sample the LLM given the same constructed input/prompt created from the original buggy code, which not only leads to generating the same incorrect patches repeatedly but also miss the critical information in testcases. To address these limitations, we propose conversational APR, a new paradigm for program repair that alternates between patch generation and validation in a conversational manner. In conversational APR, we iteratively build the input to the model by combining previously generated patches with validation feedback. As such, we leverage the long-term context window of LLMs to not only avoid generating previously incorrect patches but also incorporate validation feedback to help the model understand the semantic meaning of the program under test. We evaluate 10 different LLM including the newly developed ChatGPT model to demonstrate the improvement of conversational APR over the prior LLM for APR approach.
1 INTRODUCTION
Conversational APR alternates patch generation with validation feedback, using prior incorrect patches and their test results to guide later generations. The paper reports that this approach improves bug-fixing outcomes across 10 LLMs compared with sampling-based baselines.
1 INTRODUCTION
- APR seeks to reduce the manual cost of bug fixing, which can help address software-related financial losses and health and safety problems.The passage grounds these harms in Matteson (2018) and Hanbury (2019), and the manual cost in O’Dell (2017).Source: §1 · p. 1
- Traditional APR uses generate-and-validate; template tools match predefined patterns, while NMT-based tools are limited to training-data bug types and depend on difficult-to-construct, potentially noisy fix datasets.Template-based tools include Ghanbari et al. (2019); Liu et al. (2019); Lou et al. (2020); learning-based tools include Ye et al. (2022); Zhu et al. (2021); Jiang et al. (2021).Source: §1 · p. 1
- Prior LLM-based APR repeatedly samples from a common buggy-code prompt, risking duplicate incorrect patches while omitting testcase examples and failure outputs that could guide later repairs.Validation feedback can reveal incorrect versus expected outputs or compilation/runtime errors; prior incorrect patches are also included in conversational APR’s later prompts.Source: §1 · pp. 1–2
- Conversational APR interleaves patch generation and testcase validation, feeding each turn’s incorrect patches and feedback into the next prompt until a patch passes or the chain reaches its turn limit.The paper considers input/output/error feedback and notes that other feedback, including human evaluation, could also be used.Source: §1 · p. 2
- Across 10 popular LLMs, conversational APR fixed more bugs and reached correct patches faster than sampling-based baselines; it was also evaluated with ChatGPT (Schulman et al. (2022)1).The authors highlight its performance with a dialogue-focused LLM trained using reinforcement learning.Source: §1 · p. 2
2 BACKGROUND & RELATED WORK
Prior LLM-based APR can perform comparably to or better than earlier APR tools, but repeated sampling from the original buggy function misses test-case information. Conversational APR addresses these limitations by using prior patches and validation feedback, distinguishing it from related multi-step reasoning and synthesis approaches.
2.1 LLMS FOR APR
- Direct LLM-based APR included AlphaRepair’s zero-shot cloze-style repair, Codex prompting for patched functions, model-scaling studies, and few-shot, cloze-style, and single-line settings.AlphaRepair masks the buggy line and uses CodeBERT to fill in tokens; the cited approaches avoid fine-tuning on bug-fixing datasets.Source: §2.1 · pp. 2–3
- Prior LLM-based APR achieved comparable or better performance than earlier tools, but repeated sampling from the same original input produced repeated patches and omitted testcase information.Conversational APR builds new inputs from incorrect patches and validation feedback to address both limitations.Source: §2.1 · p. 3
2.2 MULTI-STEP PROGRAM REASONING AND SYNTHESIS USING LLMS
- Scratchpad and program-of-thoughts approaches expose intermediate program states or steps, but still generate the complete execution result or code in one shot.Conversational APR instead samples multiple times with different inputs to obtain one plausible patch.Source: §2.2 · p. 3
- Prior conversational synthesis used human feedback to correct generated code, while multi-step synthesis completed smaller program steps iteratively; conversational APR creates feedback through testcase execution without humans.Source: §2.2 · p. 3
3 CONVERSATIONAL APR
Conversational APR alternates patch generation and validation, feeding prior failed patches and testcase feedback into later prompts. Its turns run automatically until a patch passes all tests or the chain reaches its maximum length.
3 CONVERSATIONAL APR
- Conversational APR interleaves patch generation and validation, using each failed patch and its failing-testcase information as context for the next generation rather than repeatedly sampling the same prompt.Source: §3 · p. 3
- Each automated turn constructs a prompt from prior feedback, samples a patch, and validates it; a chain ends when a patch passes all tests or reaches its maximum length.Source: §3 · p. 3
3.1 PIPELINE & EXAMPLE
- In the sieve example, successive feedback-guided turns move from a patch that fails sieve(4), through an off-by-one error, to a candidate that passes all tests.The chain begins with sieve(2) exposing the bug; the final candidate switches any to all and passes the testcases.Source: §3.1 · pp. 4–5, Figure 1 · p. 4
- Testcase feedback both signals that earlier patches are incorrect, encouraging unique patches, and supplies input-output examples that can help the LLM understand function behavior.Source: §3.1 · p. 5
3.2 DESIGN DECISIONS
- The approach uses a short, open-ended prompt and functional validation feedback that invokes the function with failing-test inputs and highlights the output discrepancy.The paper compares this feedback style with alternatives, including omitting testcase information, in Section 6.2.Source: §3.2 · p. 5
- Maximum chain length bounds prompt history: length 1 repeats the initial prompt without prior patches, while longer chains expose more failures but can cause repetition or stagnation.When the preset maximum is reached, the approach restarts with a new chain; the limit also reflects the LLM's finite context window.Source: §3.2 · p. 5
4 DATASETS
The evaluation uses 10 LLMs to study model scale and design, and a filtered QuixBugs benchmark of Java and Python bugs with testcases.
4.1 LLMS
- The study evaluates 10 LLMs to examine scaling behavior and how pretraining and model design contribute to conversational APR effectiveness.Table 1 overviews the models, parameter counts, context windows, and training strategies; the evaluated models include CODEGEN variants, prefix-only Codex, and conversational ChatGPT fine-tuned with RLHF.Source: §4.1 · p. 5, Table 1 · p. 6, Figure 2 · p. 6
4.2 BENCHMARKS
- QuixBugs provides 40 algorithmic bugs in both Python and Java, each with failing testcases, but excluding 10 difficult cases leaves 30 bugs per language (60 total).The bugs are self-contained functions that usually require a single-line fix; the excluded detect-cycle testcases use graph inputs that are difficult to represent in the validation-feedback prompt.Source: §4.2 · pp. 6–7
5 EXPERIMENTAL SETUP
The evaluation is organized around two research questions, APR outcome and sampling-efficiency metrics, and implementation and generation details.
5.1 RESEARCH QUESTIONS
- RQ1 compares conversational APR with repeated sampling from the same initial prompt, while also examining model scaling and pre-training strategies; RQ2 tests chain length and validation-feedback prompts.The pre-training comparison includes ChatGPT and Codex.Source: §5.1 · p. 7
5.2 EVALUATION METRICS
- The evaluation counts plausible patches that pass all test cases and correct patches semantically equivalent to the developer patch, plus tries needed to obtain either outcome.When approaches or models fix a similar number of bugs, fewer tries are preferred to limit LLM sampling.Source: §5.2 · p. 7
5.3 IMPLEMENTATION
- The generation pipeline uses Hugging Face CODEGEN, Codex via the code-davinci-002 API, and manually prompted ChatGPT.For models other than ChatGPT, settings were top p = 0.95, temperature = 1, 50 samples per bug, and maximum chain length 3; experiments ran on a 32-core workstation with 256 GB RAM and 3 NVIDIA GeForce RTX 3090 GPUs.Source: §5.3 · p. 7
6 RESULTS
Conversational APR improves repair outcomes over repeated sampling from a fixed prompt by using validation feedback and prior patches. Its effectiveness depends on the model’s capacity for longer context and the feedback-prompting style.
6.1 RQ1: CONVERSATIONAL APR EFFECTIVENESS
- Conversational APR improves correct and plausible patch counts across unsupervisedly trained LLMs and model sizes, while reducing tries versus fixed-prompt sampling.Validation feedback can guide later patches toward passing failed testcases, while retaining incorrect patches as context helps avoid repeated outputs.Source: §6.1 · pp. 7–8, Table 2 · p. 8
- With a maximum chain length of three, ChatGPT outperformed Codex across both repair datasets under the same setting.The authors attribute the improvement partly to larger model size and suggest dialogue-oriented fine-tuning may also help.Source: §6.1 · p. 8, Table 3 · p. 8
6.2 RQ2: COMPONENT ANALYSIS
- Plausible-patch performance peaks around chain lengths of three or four, then declines; the decline is greater for smaller CODEGEN models.Long chains may overload models with failed-patch context, so the optimal length depends on the LLM.Source: §6.2 · pp. 8–9, Figure 3 · p. 8
- Adding failing-testcase details improves on feedback that only says a patch is incorrect, while functional prompting performs best among the tested styles.Functional prompting expresses the testcase input and expected output as a function call.Source: §6.2 · p. 9, Table 4 · p. 9
7 CONCLUSION
Conversational APR alternates patch generation with validation, using prior incorrect patches and feedback to guide future generations. Evaluation on 10 LLMs shows improvement over prior baseline sampling, and demonstrates the promise of ChatGPT for APR.
7 CONCLUSION
- Conversational APR improves over prior LLM-for-APR baseline sampling in an evaluation of 10 different LLMs.The conclusion also presents ChatGPT, a conversational/dialogue-driven LLM, as promising for conversational APR and APR more generally.Source: §7 · p. 9
- The approach interleaves patch generation and validation, appending previously incorrect patches and validation feedback to the input for future generations.This differs from prior approaches that repeatedly sample using the same input constructed from the original buggy code.Source: §7 · p. 9
- Using prior patches and validation feedback helps the model avoid repeating incorrect patches and understand the function's semantics.Source: §7 · p. 9