Source-linked AI summary
Enhancing Automated Program Repair with Solution Design
Jiuang Zhao, Donghao Yang, Li Zhang, Xiaoli Lian, Zitian Yang, Fang Liu
TL;DR
APR remains suboptimal on real-world issues, and design rationale in issue discussions is not yet effectively incorporated. DRCodePilot uses rationale-informed GPT-4 patching followed by feedback-based refinement. On Flink and Solr, it produces 4.7 and 3.6 times more full-match patches than GPT-4, respectively.
Problem
Real-world APR remains suboptimal, and the paper asks how design rationales scattered across issue discussions can be incorporated effectively.
Source: §1 · pp. 1–2Method
DRCodePilot prompts GPT-4 with extracted design rationales, then refines its draft using reference patches and identifier suggestions.
Source: §1 · p. 2Results
DRCodePilot produced 4.7 and 3.6 times more full-match patches than GPT-4 on Flink and Solr, respectively.
Source: §1 · p. 2Takeaways & Limitations
Ablations indicate that applying extracted design rationales alone can improve full-match ratios across the baseline models.
Source: §1 · p. 2The benchmark was scoped primarily to relatively simpler issues.
Source: §6.3 · p. 11
Abstract
from arXiv · showhide
Automatic Program Repair (APR) endeavors to autonomously rectify issues within specific projects, which generally encompasses three categories of tasks: bug resolution, new feature development, and feature enhancement. Despite extensive research proposing various methodologies, their efficacy in addressing real issues remains unsatisfactory. It's worth noting that, typically, engineers have design rationales (DR) on solution-planed solutions and a set of underlying reasons-before they start patching code. In open-source projects, these DRs are frequently captured in issue logs through project management tools like Jira. This raises a compelling question: How can we leverage DR scattered across the issue logs to efficiently enhance APR? To investigate this premise, we introduce DRCodePilot, an approach designed to augment GPT-4-Turbo's APR capabilities by incorporating DR into the prompt instruction. Furthermore, given GPT-4's constraints in fully grasping the broader project context and occasional shortcomings in generating precise identifiers, we have devised a feedback-based self-reflective framework, in which we prompt GPT-4 to reconsider and refine its outputs by referencing a provided patch and suggested identifiers. We have established a benchmark comprising 938 issue-patch pairs sourced from two open-source repositories hosted on GitHub and Jira. Our experimental results are impressive: DRCodePilot achieves a full-match ratio that is a remarkable 4.7x higher than when GPT-4 is utilized directly. Additionally, the CodeBLEU scores also exhibit promising enhancements. Moreover, our findings reveal that the standalone application of DR can yield promising increase in the full-match ratio across CodeLlama, GPT-3.5, and GPT-4 within our benchmark suite. We believe that our DRCodePilot initiative heralds a novel human-in-the-loop avenue for advancing the field of APR.
1 INTRODUCTION
Real-world APR remains suboptimal, while engineers rely on solutions and arguments discussed before coding. DRCodePilot brings that design rationale into GPT-4 patch generation and uses feedback to refine the draft.
1 INTRODUCTION
- In open-source maintenance, issue discussions record solution proposals and arguments before assignees implement them and submit patches through GitHub pull requests.The paper calls solutions and their accompanying arguments design rationale, which guides engineers and can reduce flawed fixes.Source: §1 · pp. 1–2
- Prior APR approaches have made limited use of discussion-derived rationale, while extracting it is difficult because discussions are noisy and complex and applying it requires reasoning and project knowledge.The authors introduce feedback-based self-reflection to help LLMs apply DR and address reasoning failures and code hallucinations.Source: §1 · p. 2
- DRCodePilot combines rationale-informed initial patching with feedback from reference patches and identifier suggestions to refine GPT-4’s output.DRs are extracted by pairing issue solutions with their arguments; reference patches come from fine-tuned CodeT5P.Source: §1 · p. 2
- 4.7 and 3.6 times more full-match patches than GPT-4 on Flink and Solr, respectively; CodeBLEU scores improved by 5.4% and 3.9%.The benchmark contains 714 Flink and 224 Solr samples, drawn from 938 matched issue-patch pairs.Source: §1 · p. 2
- The benchmark comprises 938 issue-patch pairs matched from Jira issues and GitHub commits in Flink and Solr.The study also reports that DR alone improves full-match ratios across its baseline models.Source: §1 · p. 2
2 MOTIVATING EXAMPLE
The motivating example shows that issue discussions can establish a solution before coding, yet translating that rationale into a robust patch still requires project understanding. Longer, noisy discussions and limited contextual knowledge make that translation harder.
2 MOTIVATING EXAMPLE
- In FLINK-32976, a missing TARGET-parameter check caused a null pointer exception, and commenters agreed on adding a null check.The assignee implemented the rationale by setting an empty string as the default value to improve robustness.Source: §2 · p. 3
- Although the issue’s solution was settled, the maintainer took 20 days to commit changes, motivating automation of rationale-guided patch generation.The paper says existing APR techniques do not support this capability.Source: §2 · p. 3
- Complex issues such as FLINK-340073 can involve 82 detailed comments debating three solutions, while long inputs and noisy logs challenge accurate design-information capture.LLMs may also produce erroneous outputs when project-specific background knowledge is lacking.Source: §2 · p. 3
- Design rationales can be abstract or conflicting, offer only high-level recommendations, and leave engineers to decide whether and how to apply them.The authors frame automatic and effective use of these rationales as the central challenge.Source: §2 · p. 3
3 APPROACH
DRCodePilot follows a software-maintenance workflow: it mines design rationale, drafts a GPT-4 patch, generates project-specific feedback, recommends identifiers, and asks GPT-4 to refine the draft.
3 APPROACH
- DRCodePilot models a maintenance lifecycle in which reports are discussed before maintainers craft patches, with five phases from rationale acquisition through final patch generation.The phases include defective-segment location and draft generation, reference-draft generation, and identifier recommendation.Source: §3 · p. 3
3.1 Design Rationale Acquisition
- DRMiner extracts design rationales from issue logs by identifying design-related text and aligning solution-argument pairs using prompts and heuristics.Its performance was validated on 2,092 sentences from 30 issues across three open-source systems.Source: §3.1 · p. 3
3.2 Defective Segment Location and Draft Patch Generation
- GPT-4 locates the defective segment and generates an initial patch using the extracted rationales in a structured prompt.The prompt includes an instruction, issue summary, buggy code, design rationale, and output instruction.Source: §3.2 · pp. 3–4
3.3 Reference Patch Generation
- A fine-tuned CodeT5P model supplies reference patches so GPT-4 can reassess drafts with project-specific knowledge.The 220M model is fine-tuned on masked Java methods extracted from the project repository.Source: §3.3 · pp. 4–5
3.4 Identifier Recommendation
- Identifier recommendations target patch identifiers absent from the project, using similar code snippets and other identifiers from the defective Java file.The method offers up to six replacement candidates for each flagged identifier.Source: §3.4 · p. 5
3.5 Final Patch Generation
- GPT-4 produces the final patch by revisiting its draft with reference-patch and identifier feedback.The final prompt presents the draft, reference patch, and suggestions together; reference patches provide solution exemplars and identifier suggestions target fine-grained errors.Source: §3.5 · p. 5
4 EXPERIMENTAL EVALUATION
The evaluation tests DRCodePilot on Jira issues linked to merged GitHub patches, comparing it with code LLM baselines and examining design rationale, feedback, and rationale quality. It reports gains over direct-generation baselines and explores how results vary with project knowledge and model choice.
4.1 Research Questions
- The evaluation asks whether DRCodePilot repairs bugs, whether its design rationale and feedback components help, and whether rationale benefits other LLMs.The study also examines the effect of rationale quality and whether general-purpose models can extract useful rationale from comments.Source: §4.1 · p. 5, §4.6 · p. 8
4.2 Data Preparation
- The benchmark links Jira issues to GitHub pull requests in Flink and Solr, retaining merged, closed issues with single-file Java logic changes of at most 22 lines.The authors note that these patches are, on average, less complex than SWE-bench issues, and report that fewer than 1% of merged patches could be evaluated executably because of legacy dependencies.Source: §4.2 · p. 6
4.3 Experiment Setting
- Full-match evaluation requires generated patches to be identical to human patches; CodeBLEU compares repaired functions with gold functions using syntactic, semantic, and structural signals.The authors omit pass@k because compiling legacy projects is difficult and passing tests alone does not establish that a patch is suitable for integration.Source: §4.3 · p. 6
4.4 Baseline Comparison (RQ1)
- DRCodePilot beats direct-generation baselines on full matches and CodeBLEU across Flink and Solr, with 109 Flink full matches—about 4.7 times GPT-4—and up to 3.6 times GPT-4’s Solr count.Among direct-generation baselines, GPT-4 performs best; DRCodePilot’s CodeBLEU scores also lead in both systems.Source: §4.4 · p. 7, Table 3 · p. 8
4.5 Ablation Study (RQ2)
- Removing design rationale causes the largest ablation drops: full-match counts fall by 81.65% and 94.44%, while CodeBLEU falls by 9.17% and 4.87% across the two systems, respectively.Removing identifier recommendations also lowers performance; removing reference-patch generation leaves Flink’s exact-match count unchanged and marginally increases Solr’s, while CodeBLEU declines in both.Source: §4.5 · p. 7, Table 4 · p. 8
4.6 General Impact of Design Rationale (RQ3)
- Across the 61 extensively discussed issues, design rationale improves results for the tested models, and manually annotated rationale yields more full matches than automatically mined rationale.GPT-3.5’s CodeBLEU decreases with manual annotations; the authors also report that original developer comments improve full matches over no comments.Source: §4.6 · p. 9, Table 5 · p. 9
5 RELATED WORKS
Prior work addresses design-rationale extraction and several approaches to automated program repair; DRCodePilot combines extracted rationale with feedback to guide LLM-based patch generation.
5.1 Design Rationale Extraction
- Design rationale comprises alternative designs and the reasons for accepting or rejecting them, supporting engineers’ development and design decisions.The paper also describes its documentation as important for preserving project decisions for future maintenance.Source: §5.1 · p. 10
- Discussions are noisy, interleaved, and diverse; DRMiner identifies solutions and associated arguments in Jira issue logs [3] [71].Earlier solution-description work received an average informativeness score of 3.3 on a 1–5 scale [41].Source: §5.1 · p. 10
5.2 Automatic Program Repair
- APR includes semantic-search, semantic-constraint [31], pattern-based [23], and learning-based methods [41] [64].Source: §5.2 · p. 10
- LLM-based APR commonly frames repair as code generation, while approaches also use fine-tuning, external feedback, bug reports, or retrieved API context [27] [34] [45] [59] [66].Code hallucination is a cited challenge for LLM repair [55].Source: §5.2 · p. 10
- DRCodePilot extracts solutions and arguments to guide zero-shot patch generation, then uses feedback to address technical-information gaps.Source: §5.2 · p. 10
6 DISCUSSION
The discussion describes conditions in which DRCodePilot may struggle and identifies benchmark, platform, and evaluation constraints that bound the study’s findings.
6.1 Failure Case Analysis
- Analysis of 72 patches identified two scenarios where DRCodePilot may underperform: weak or missing rationale and insufficient contextual knowledge.The sample comprised 36 full-match and 36 lower-quality patches.Source: §6.1 · p. 10
- Missing code snippets, external references, and unclear intended behavior can leave extracted rationale without context needed for accurate patches.The examples include a rationale referring to another issue and discussions that identify an error without specifying correct behavior.Source: §6.1 · pp. 10–11
6.2 Threats to Validity
- The custom benchmark was chosen because Defects4J and SWE-bench lack developer discussions or discussion metadata needed to extract rationale [19].Source: §6.2 · p. 11
- Rationale extraction was evaluated only on Jira; whether the approach transfers to GitHub issues remains to be assessed.The authors anticipate compatibility based on analogous metadata structures.Source: §6.2 · p. 11
- Independent annotations, consensus discussions, majority voting, and masking code in comments were used to mitigate annotation and comment-content validity threats.Source: §6.2 · p. 11
6.3 Limitations
- The study uses general-purpose LLM baselines and primarily covers simpler issues; specialized repair models were excluded because their required inputs were unavailable.The dataset was scoped similarly to Defects4J, where 95% of patches have fewer than 22 changed lines.Source: §6.3 · p. 11
- The study does not isolate the effects of solutions versus arguments in design rationale on patch quality.Source: §6.3 · p. 11
- Without execution-based evaluation, static full-match and CodeBLEU metrics may miss functional correctness; CodeBLEU can overestimate quality by scoring unchanged function portions.Source: §6.3 · p. 11
7 CONCLUSION AND FUTURE WORK
DRCodePilot combines design rationale with feedback to refine GPT-4 patches. On a 938-pair benchmark, it produced up to 4.7x and 3.6x more full-match patches than GPT-4.
7 CONCLUSION AND FUTURE WORK
- Up to 4.7x and 3.6x more full-match patches than GPT-4 were achieved by DRCodePilot across the 938-pair benchmark.The approach uses design rationale, reference patches, and identifier-replacement suggestions to guide patch generation and refinement.Source: §7 · p. 11