Source-linked AI summary
Revisiting Feedback-Driven LLM Code Repair: A Replication and Exploratory Java Extension
Louis Lalonde, Wassim Keddache, Thomas Perron Touchette, Leuson Da Silva, Foutse Khomh
TL;DR
The study examines reproducibility and language sensitivity in feedback-driven LLM code repair by partially replicating FeedbackEval in Python and extending it to Java. Python test feedback remained strongest, while Java feedback rankings varied by mutation type; lightweight prompting offered better cost-effectiveness without statistically supported gains from Chain-of-Thought.
Problem
Existing evidence on feedback-driven LLM code repair has limited evaluation of reproducibility and how programming-language differences affect feedback use.
Method
The study partially replicates FeedbackEval on Python benchmarks and evaluates 100 Java faulty instances derived from 50 tasks across feedback types and prompting strategies.
Results
Python test-based feedback achieved the highest repair performance, while Java rankings differed by mutation type and lightweight prompting had more favorable observed cost-effectiveness.
Takeaways & Limitations
The findings support replication and more controlled multilingual benchmarks with consistent mutation strategies and richer feedback representations.
Takeaways & Limitations
Java experiments used exclusively LLM-generated mutations, which may reflect training-distribution errors and inflate Repair@1 scores.
Abstract
from arXiv · showhide
Since the advent of Large Language Models (LLMs), practitioners have increasingly leveraged them to support their software engineering tasks, including automated code repair, showing promising results. Yet, concerns regarding reproducibility and generalizability remain largely unexplored. To further evaluate these concerns and associated impacts, we partially reproduce and conduct an exploratory Java extension of the FeedbackEval benchmark [1], which evaluates how LLMs leverage different feedback types for Python code repair. First, we partially replicate the original study on 394 repair tasks using GPT-4o and Claude 3.5 Sonnet, reproducing and observing the main qualitative trends reported in the original work. Second, we conduct an exploratory Java extension by constructing 100 erroneous repair instances from 50 Java tasks and evaluating feedback effectiveness. Our results show that previous conclusions from Python may be sensitive to benchmark construction, feedback representation, and tooling ecosystem, motivating more controlled multilingual benchmarks. Specifically, while test feedback remains the strongest feedback type in our Python replication, the same ranking is not observed in our Java extension, as simple and JUnit-based test feedback do not differ significantly. We hypothesize that differences in feedback informativeness and tooling ecosystems, such as the verbosity of test frameworks, may partly explain such a difference. Finally, lighter prompts reduce cost without significant differences in repair effectiveness. Overall, our findings confirm key trends under a partially controlled replication and highlight the need for more rigorous multilingual evaluation and careful feedback design in LLMbased repair systems.
I. INTRODUCTION
This study revisits FeedbackEval to assess reproducibility and whether feedback-effectiveness patterns generalize beyond Python. It partially replicates the original evaluation and extends it to Java while examining prompt cost-efficiency.
- LLM-based code repair is an iterative process in which models use feedback to identify and fix faults.
- FeedbackEval found that Python repair performance varies across test, compiler, human, and simple feedback, with test feedback reaching 37.8% Repair@1 for GPT-4o on CoderEval.
- The study investigates whether Python feedback patterns reproduce in Java despite differences in typing, compilation, testing, and diagnostic tooling.
- The authors partially replicate FeedbackEval on CoderEval and HumanEval, then construct 100 Java faulty instances from 50 tasks for feedback and prompting experiments.
- Python replication preserves test feedback as strongest, Java shows different feedback patterns, and lightweight prompts offer better observed cost-effectiveness without statistically supported gains from Chain-of-Thought.
B. Study Design
The study follows the original FeedbackEval protocol where feasible, evaluating repair across feedback types, prompt configurations, models, and established benchmarks. Its replication uses shared artifacts and Repair@1 while narrowing the model set for cost reasons.
- The replication reuses FeedbackEval datasets, buggy snippets, feedback artifacts, and evaluation scripts, although exact replication of RQ3 was infeasible.
- Each of 394 repair tasks is tested with test, compiler, human, and simple feedback, with single-attempt success measured by Repair@1.
- The study compares baseline, Chain-of-Thought, Few-shot, and ablation prompts on the same tasks, model configuration, and feedback type.
- The replication focuses on GPT-4o and Claude 3.5 Sonnet rather than the original five models because large-scale evaluation requires substantial API calls and cost.
C. Results
The replication largely preserves FeedbackEval’s qualitative findings, with test feedback strongest overall and performance higher on HumanEval than CoderEval. Prompt results remain broadly consistent, while individual configurations show quantitative variation.
- Replication Results: GPT-4o achieves 55.55% average Repair@1 versus 56.4% originally, while Claude 3.5 Sonnet achieves 60.2% versus 60.8%.
- Replication Results: Test feedback has the highest overall repair success, human feedback the lowest, and simple feedback remains competitive across models and datasets.
- Feedback Effectiveness Across Datasets: Repair success is consistently higher on HumanEval than CoderEval, with test feedback best on CoderEval and simple feedback slightly exceeding test feedback on GPT-4o HumanEval.
- Prompting Techniques: The replication’s average Repair@1 across prompt configurations is 50.83%, close to the original 50.09%, despite differences in individual prompting techniques.
III. EXTENSION STUDY
The Java extension tests whether Python feedback rankings persist under different compilation, testing, and feedback-generation characteristics. It constructs a controlled Java repair set while avoiding claims of an isolated language effect.
- The extension asks whether Python feedback-effectiveness trends reproduce in Java’s different compilation, testing, and feedback-generation setting.
- The authors construct controlled Java repair tasks and reproduce the original benchmark’s experimental setup for the extension.
2) Approach:
The Java extension constructs 100 faulty instances from 50 validated CoderEval tasks, using LLM-driven logical and compiler mutations and category-specific feedback modalities. It evaluates repair performance with Repair@1 and paired statistical tests in a controlled execution environment.
- The study samples 50 Java CoderEval tasks and filters reference implementations to ensure compilation and successful original validation before mutation injection.
- The Java mutations are generated exclusively by LLMs, limiting direct comparability with the original Python benchmark.The authors therefore treat the Java study as exploratory rather than as a direct cross-language comparison.
- 100 erroneous instances are created by generating one logical mutation and one compiler mutation for each task.Logical mutations include incorrect conditions, off-by-one errors, and wrong return values; compiler mutations prevent successful compilation.
- Logical mutations are evaluated with test, simulated human, and simple feedback, while compiler mutations use compiler, simulated human, and simple feedback.Test feedback comes from JUnit failures, and compiler feedback comes from javac errors.
- Repair@1 measures the percentage of first patches that compile when applicable and pass the associated test suite, with Wilson 95% confidence intervals.Because tasks are paired across feedback conditions, the analysis uses Cochran’s Q test and follow-up comparisons.
3) Results:
In the Java extension, feedback effectiveness varies by mutation category: no individual modality is statistically superior for logical mutations, while compiler and simple feedback outperform human feedback for compiler mutations. Logical repairs can also introduce compilation errors.
- Logical mutations: 86% Repair@1 is the highest observed result for logical mutations with simple feedback, followed by test feedback at 82% and human feedback at 74%.After Holm correction, no pairwise comparison is statistically significant, including simple versus test feedback (pHolm = 0.5000).
- Compiler mutations: 94% Repair@1 is the highest observed result for compiler mutations with compiler feedback, followed by simple feedback at 92% and human feedback at 78%.Compiler and simple feedback significantly outperform human feedback, but do not differ significantly from each other.
- Cross-category behavior: Logical repair attempts sometimes introduce compilation errors, so the resulting failures can cross from semantic defects into syntactic or type-level problems.The authors suggest that test feedback followed by compiler validation may support more robust repair pipelines.
1) Motivation:
The study examines whether prompting strategies improve Java repair and whether their costs justify any effectiveness differences. Across seven conditions, Repair@1 does not differ significantly, while lightweight prompts cost less and Chain-of-Thought costs more without significant improvement.
- Prompting setup: Seven prompting conditions are evaluated on the same 50 logical-mutation tasks with one repair attempt per task using GPT-4o-mini.Conditions include a baseline, Chain-of-Thought, Few-shot, and four single-component ablations.
- Evaluation: Cost efficiency is measured by total API cost, Repair@1, and CAD cost per successful repair.Cost per successful repair divides total API cost by the number of successful repairs; lower values indicate greater efficiency.
- Effectiveness: Repair@1 ranges from 80% to 88%, with no statistically significant overall difference or baseline comparison after Holm correction.The authors caution that nonsignificance does not establish equivalence among prompting conditions.
- Cost efficiency: CAD 0.000575 is the observed Chain-of-Thought cost per repair, compared with CAD 0.000236 for the baseline and CAD 0.000215 for No-Context.Chain-of-Thought uses more tokens without significantly improving Repair@1.
- Cost efficiency: Lightweight ablations reduce cost relative to the baseline without significant Repair@1 differences, indicating a more favorable observed cost-effectiveness trade-off.The broader exploratory summary reports the same pattern: prompting variants show no significant Repair@1 differences while lightweight prompts reduce cost relative to Chain-of-Thought.
IV. DISCUSSION
The discussion finds that feedback effectiveness varies across mutation types and settings, while lighter prompting offers better observed cost-effectiveness. It cautions that Java–Python differences cannot support direct cross-language conclusions because benchmark and mutation designs differ.
- Feedback patterns: Compiler feedback achieves the highest observed Repair@1 for Java compiler mutations, but does not differ significantly from simple feedback.The authors suggest javac’s localized locations, symbols, and type information may support repairs.
- Feedback patterns: Simple feedback achieves the highest observed Repair@1 for Java logical mutations, but its difference from test feedback is not statistically significant.The Java results differ from Python CoderEval, where test feedback performs best.
- Prompting trade-offs: Chain-of-Thought is least cost-efficient, while Few-shot adds input overhead without statistically supported Repair@1 gains.Ablation variants reduce cost without statistically supported Repair@1 differences from the baseline.
- Interpretation limits: The Java extension should not be interpreted as a direct Java–Python comparison because mutation strategies, datasets, feedback formats, and evaluation settings differ.The Java study uses LLM-generated mutations, whereas FeedbackEval combines tool-based mutations, LLM-generated errors, and incorrect generated solutions.
- Interpretation limits: LLM-generated mutations may be easier for the repair model to recognize and fix when they reflect familiar error patterns.The paper identifies mutation strategy as an important confound and recommends consistent, independent mutation generation for multilingual benchmarks.
- Implications: These limitations motivate controlled multilingual benchmarks before strong conclusions about cross-language generalization are drawn.The Java extension is presented as exploratory evidence that feedback effectiveness can vary across benchmark, mutation, and ecosystem settings.
C. Replication Challenges of LLM-based Studies
The replication exposed practical and methodological challenges involving API variability, computational cost, missing details, model availability, and statistical interpretation. These challenges complicate reproducibility and motivate more rigorous reporting and experimental controls.
- Replication challenges: API variability, computational cost, missing experimental details, and changing model availability created challenges during the FeedbackEval replication.The paper characterizes these issues as distinct from challenges in traditional software engineering experiments.
- Resource constraints: More than 3,000 API requests are required for 394 tasks across four feedback modalities and two models in RQ1 and RQ2.Additional prompting experiments increase the request count, financial cost, and execution latency.
- Statistical reporting: Repair@1 differences of only a few percentage points can make descriptive percentages prone to overinterpretation.The authors report that simple versus test feedback for Java logical mutations was not statistically significant after correction.
- Model availability: Claude 3.5 Sonnet was deprecated during the study, requiring all replication experiments involving it to run before the deprecation date.Model evolution can directly limit long-term reproducibility.
- Positioning: The study extends prior work by examining reproducibility and feedback behavior in an exploratory Java repair setting.The related-work discussion distinguishes this focus from prior approaches centered on improving repair in specific scenarios.
VI. THREATS TO VALIDITY
The threats to validity concern mutation generation, test translation, repair-quality measurement, model and dataset scope, single-iteration evaluation, and stochastic variability. Together, they limit how broadly the reported feedback effects should be generalized.
- Internal validity: Java mutations were generated exclusively with GPT-4o-mini, unlike the mixed mutation sources in the original Python dataset.This may make Java errors closer to the repair model’s training distribution and easier to fix.
- Construct validity: Repair success is defined as compilation and passing the associated test suite, excluding readability, maintainability, and uncovered subtle semantic bugs.Translated JUnit tests were manually inspected and validated against reference implementations and injected mutations.
- Mutation validity: LLM-generated mutations may be easier for similar models to repair and may inflate Repair@1 scores.The extension includes 13 logical mutation types, but the mutations may still differ from real-world or tool-generated bugs.
- External validity: The evaluated models, datasets, real-world settings, and single-iteration repair setup limit generalization to other models, larger datasets, complex systems, and iterative workflows.Mutation, test, simulated feedback, and repair generation all used the same LLM.
- Conclusion validity: Each condition was executed once, so stochastic variability across repeated generations was not quantified.Python replication comparisons with published results remain descriptive, and small or nonsignificant differences require caution.
VII. CONCLUSION
The study partially replicates FeedbackEval and extends it exploratorily to Java, confirming Python trends while finding mutation-dependent feedback patterns in Java. It concludes that controlled multilingual benchmarks, consistent mutation strategies, richer feedback, and broader evaluations are needed.
- Conclusion: The Python replication largely confirms FeedbackEval’s reported trends, particularly the effectiveness of test-based feedback over alternatives.This supports the robustness of the original findings under the replication’s experimental setup.
- Conclusion: Simple feedback ranks highest for Java logical mutations, but no pairwise difference remains significant after correction.For compiler mutations, compiler and simple feedback outperform simulated human feedback but do not differ significantly from each other.
- Interpretation: Differences in feedback types and tooling characteristics, including test-framework verbosity, may influence the observed divergence between Java and Python results.The conclusion cautions that mutation generation and dataset construction may also affect repair difficulty.
- Future work: Future work should develop controlled multilingual benchmarks, consistent mutation strategies, richer feedback representations, and evaluations across languages, models, and real-world scenarios.The study also provides replication and Java-extension artifacts for reproducing its experiments.