PaperlayerView exact revision on arXiv

Source-linked AI summary

FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks

Dekun Dai, MingWei Liu, Anji Li, Jialun Cao, Yanlin Wang, Chong Wang, Xin Peng, Zibin Zheng

arXiv:2504.06939v2cs.SE

TL;DR

LLMs’ ability to interpret and use diverse feedback for code repair remains insufficiently explored, particularly in complex scenarios. FeedbackEval evaluates this capability across tasks and feedback settings; mixed feedback yields the highest repair success, and iterative gains plateau after two or three iterations.

  • Problem

    LLMs’ ability to interpret and effectively use diverse feedback in code repair, especially in complex scenarios, remains insufficiently explored and is often assessed only implicitly at the system level.

    Source: §1 · p. 2

  • Method

    FeedbackEval benchmarks feedback-driven repair using HumanEval, CoderEval, and SWE-Bench-verified tasks and empirically evaluates LLMs across feedback settings.

    Source: §1 · pp. 2–3, §2.2.1 · p. 6

  • Results

    Mixed feedback achieves the highest repair success, iterative gains plateau after two or three iterations, and Deepseek-R1 averages 63.4% repair accuracy.

    Source: §1 · p. 3, §10 · p. 19

  • Takeaways & Limitations

    Task complexity magnifies differences between feedback-driven repair outcomes, particularly in repository-level scenarios.

    Source: §10 · p. 19

    Multi-iteration repair experiments were run only once per setting, and the study compares feedback effects rather than absolute repair success rates.

    Source: §8 · p. 18

Abstract

from arXiv · show

Code repair is a fundamental task in software development, facilitating efficient bug resolution and software maintenance. Although large language models (LLMs) have demonstrated considerable potential in automated code repair, their ability to comprehend and leverage diverse types of feedback, which is crucial for iterative self-correction in authentic debugging scenarios, remains insufficiently understood. To bridge this gap, we introduce FeedbackEval, a systematic benchmark constructed from three heterogeneous sources (HumanEval, CoderEval, and SWE-Bench-verified), to evaluate LLMs' feedback comprehension and code repair performance. We conduct a comprehensive empirical study on five state-of-the-art LLMs, including GPT-4o, Claude-3.5, Deepseek-R1, GLM-4, and Qwen2.5, to evaluate their behavior under both single-iteration and iterative code repair settings. Our results show that mixed feedback yields the highest repair success (63.6%), with LLM-Expert and test feedback providing strong targeted gains (62.9% and 57.9%, respectively), while minimal (53.1%) and compiler feedback (49.2%) offer moderate benefits and LLM-Skilled proves least effective (48.8%). Iterative feedback further enhances repair performance, though the marginal benefit diminishes after two or three iterations. Moreover, prompt structure is shown to be critical: structured reasoning (RR, CoT) and dynamic example selection deliver notable improvements, whereas removing semantic cues such as docstrings or role-play causes severe degradation. This work introduces a robust benchmark and delivers practical insights to advance the understanding and development of feedback-driven code repair using LLMs.

1 Introduction

FeedbackEval targets the limited understanding of how LLMs interpret diverse feedback during code repair, introducing a benchmark and empirical study of single-iteration and iterative repair. Results distinguish model performance, feedback effectiveness, iteration gains, and prompting strategies.

Motivation

  • Prior work often examined repair accuracy under narrow feedback conditions or single-iteration settings, leaving diverse modalities and prompting strategies undercompared.The paper also identifies a need for an extensible benchmark covering structured and unstructured feedback and iterative repair.Source: §1 · p. 2

FeedbackSynth and FeedbackEval

  • FeedbackSynth constructs FeedbackEval from diverse programming tasks and erroneous code, supplying structured, human-like, and composite feedback for reproducible evaluation.The benchmark is designed to support analysis of how LLMs interpret, prioritize, and act on heterogeneous feedback.Source: §1 · pp. 2–3

Empirical Study

  • The study evaluates five state-of-the-art LLMs across single-iteration repair, feedback types, iterative repair, and prompting strategies.Its research questions cover model performance, feedback effects, iteration, and prompt engineering.Source: §1 · p. 3

Main Findings

  • Deepseek-R1 achieved 63.4% average repair accuracy, ahead of Claude-3.5 (59.2%), GPT-4o (54.5%), Qwen2.5 (51.4%), and GLM-4 (51.1%).Deepseek-R1's lead over other models ranged from 4.2 to 12.3 percentage points.Source: §1 · p. 3
  • Mixed feedback produced the highest repair success, while LLM-Expert and test feedback gave strong targeted gains; LLM-Skilled performed worst.Compiler and minimal feedback provided moderate benefits.Source: §1 · p. 3
  • Iterative feedback improved repair, but gains typically stabilized after two to three cycles, with test and LLM feedback showing the most significant gains.The benefit of additional iterations diminished over successive cycles.Source: §1 · p. 3
  • Structured reasoning (RR, CoT) and dynamic example selection improved performance, while removing docstrings or role-play caused the sharpest degradation.The reported findings identify semantic understanding of intended functionality as more critical than environmental context.Source: §1 · p. 3

Contributions

  • The paper presents a benchmark, an empirical evaluation of LLM feedback use, and recommendations concerning structured feedback and prompting.These are stated as the paper's main contributions.Source: §1 · p. 3

2 Benchmark Construction

FeedbackEval is built with FeedbackSynth, an automated and extensible pipeline for assembling repair tasks, generating faulty code and feedback, and supporting systematic evaluation. The resulting benchmark combines varied tasks, errors, and feedback modalities.

2.1 Benchmark Format

  • Each benchmark task pairs a repair instance with feedback; its core materials include erroneous code, a docstring, project context, a reference implementation, and tests.These components separate repair context from the guidance signals used for evaluation.Source: §2.1 · pp. 4–5

2.2 Pipeline Construction

  • FeedbackSynth automates task collection, erroneous-code generation, and feedback generation to support rapid, reproducible, extensible benchmark construction.The pipeline is also presented as a potential source of training data.Source: §2.2 · p. 6

2.2.1 Code Generation Task Collection.

  • The task collection combines HumanEval, CoderEval, and 178 SWE-Bench-verified instances, spanning algorithmic, project-level, and repository settings.SWE-Bench-verified instances are filtered so repairs can be localized to a single function.Source: §2.2.1 · p. 6

2.2.2 Erroneous Code Generation.

  • Faulty code is generated through seven rule-based mutation operator types, LLM-induced subtle mistakes, and incorrect LLM-generated solutions that fail tests.SWE-Bench-verified contributes real-world repair instances rather than generated faulty code.Source: §2.2.2 · pp. 6–7

2.2.3 Automated Feedback Generation.

  • Six feedback forms include compiler and test diagnostics, minimal guidance, LLM-Skilled and LLM-Expert reviews, and a composite form combining four sources.LLM-Expert generation uses both faulty and reference implementations, while LLM-Skilled generation omits access to the ground truth.Source: §2.2.3 · p. 7
  • Compiler and test feedback are derived from static analysis and test execution, while the benchmark preserves the execution environment and tests for repeatable validation.Natural-language feedback is simulated with LLMs rather than costly human annotations.Source: §2.2.3 · p. 7

2.3 Resulting Benchmark

  • FeedbackEval contains 572 tasks and 3,914 erroneous code instances, with structured and non-structured feedback and a broad range of error types.Reported errors include logic, type, scope, object-oriented, system-level, runtime, and syntax errors.Source: §2.3 · p. 8
  • Its automated, extensible design is intended to support expansion to new languages, tasks, error categories, and feedback modalities.The pipeline is presented for both evaluation and use as a training resource.Source: §2.3 · p. 8

3 Experimental Setup

The experiments compare five selected LLMs under shared evaluation settings, using a subset of FeedbackEval for focused analysis. Repair@k measures pass rates across iterative repair rounds.

LLM Selection

  • The study evaluates GPT-4o, Claude-3.5, Deepseek-R1, GLM-4, and Qwen2.5, including open- and closed-source models.Deepseek-R1 is included as a reasoning-enhanced model for examining structured reasoning in feedback-driven repair.Source: §3 · p. 8

Implementation Procedure

  • Due to computational constraints, experiments use one randomly selected erroneous instance per task, totaling 572 instances paired with six feedback types.All models use the same sampling strategy and temperature of 0.3, and are accessed through their APIs.Source: §3 · p. 8

Evaluation Metric

  • Repair@k measures the code pass rate after k rounds of iterative repair to assess refinement using feedback.The same experimental settings and evaluation metrics are applied across models.Source: §3 · p. 8

4 RQ1: Single-Iteration Code Repair

The study evaluates single-iteration repair across feedback types and finds Deepseek-R1 has the highest overall Repair@1. Performance varies across task sets, with substantial drops on more complex tasks.

4.1 Design

  • The experiment applies one of six feedback types to 572 tasks and measures pass rate after one repair iteration, averaging two identical runs.The same prompt templates are used across models and feedback types, with docstrings and contextual details supplied.Source: §4.1 · p. 9

4.2 Results

  • 63.4% overall Repair@1 makes Deepseek-R1 the top model, ahead of the other evaluated LLMs by 4.2–12.3 percentage points.Claude-3.5 scores 59.2%, GPT-4o 54.5%, Qwen2.5 51.4%, and GLM-4 51.1%.Source: §4.2 · p. 9
  • HumanEval scores of 79.2%–86.2% fall to 31.7%–44.8% on CoderEval and 37.9%–59.2% on SWE-Bench-Verified.CoderEval comprises more complex repository-level tasks, while the SWE-Bench-verified subset contains 178 single-function repairs from real-world GitHub issues.Source: §4.2 · pp. 10–11, Figure 5 · p. 10

5 RQ2: Impact of Feedback Types

The study compares feedback modalities across models and task sets, finding that mixed feedback leads overall, while feedback utility varies with task complexity and guidance quality.

5.1 Design

  • The experiment compares Repair@1 by feedback type across models and task sets to examine how task complexity relates to feedback utility.The design assesses structured, unstructured, and mixed feedback on errors of differing complexity.Source: §5.1 · p. 11, §5 · p. 11

5.2 Results

  • 63.6% Repair@1 for mixed feedback leads the aggregate results, followed by LLM-Expert at 62.9% and test feedback at 57.9%.Minimal feedback reaches 53.1%, while compiler feedback reaches 49.2% and LLM-Skilled is least effective at 48.8%.Source: §5.2 · pp. 11, 13
  • Test feedback scores 85.9% on HumanEval, 40.4% on CoderEval, and 47.6% on SWE-Bench-verified, with weaker performance on repository-level tasks.Its single-function SWE-Bench subset partly recovers relative to CoderEval; the authors describe test feedback as explicit and useful for semantic or requirement-driven errors.Source: §5.2 · pp. 11, 13
  • Compiler feedback drops from 81.4% on HumanEval to 31.2% on CoderEval and 34.9% on SWE-Bench-verified, as precise diagnostics may not translate into semantic repairs.In the case study, compiler feedback flags unused arguments but leaves the core argument error unresolved.Source: §5.2 · pp. 12–13
  • LLM-Expert exceeds LLM-Skilled on HumanEval (89.0% vs. 72.1%), CoderEval (41.1% vs. 30.9%), and SWE-Bench-verified (58.4% vs. 43.4%).The passage attributes the advantage to targeted, high-quality guidance informed by solution knowledge.Source: §5.2 · p. 13
  • LLM-Skilled feedback ranks lowest on HumanEval and CoderEval but reaches 43.4% on SWE-Bench-verified, comparable to Minimal at 43.5%.The case study describes detailed suggestions being misinterpreted into an overly complex revision that misses the central argument error.Source: §5.2 · pp. 12–13
  • Minimal feedback scores 79.6% on HumanEval, 36.1% on CoderEval, and 43.5% on SWE-Bench-verified, consistently surpassing LLM-Skilled feedback.The authors suggest concise, clear directives can activate intrinsic debugging, while verbose or misleading suggestions may hinder repair.Source: §5.2 · p. 13
  • Mixed feedback leads overall on HumanEval (90.7%) and CoderEval (43.7%), but LLM-Expert beats it on SWE-Bench-verified (58.4% vs. 56.4%).Mixed feedback combines test and compiler diagnostics with LLM-Expert and minimal guidance; the authors suggest longer, heterogeneous signals may dilute key information in realistic projects.Source: §5.2 · p. 13

6 RQ3: Multiple-Iteration Repair

The study evaluates iterative repair over three feedback-driven rounds and reports improved performance, with gains generally leveling off after two or three iterations.

6.1 Design

  • For three iterations, each repair is reevaluated and new feedback of the same type is generated from remaining errors; Repair@k measures pass rate after k iterations.The study uses the same erroneous code set across models and pairs each task with one of six feedback types.Source: §6.1 · p. 14

6.2 Results

  • Test feedback outperforms compiler feedback by 2.8–13.3 points on HumanEval, 9.5–15.9 on CoderEval, and 9.4–18.9 on SWE-Bench-verified.The reported gap widens with task complexity, favoring execution-based validation over static error diagnostics in repository-level repairs.Source: §6.2 · p. 15
  • LLM-Expert nearly matches mixed feedback across datasets, while LLM-Skilled performs worst among unstructured feedback types.Reported ranges are 93–99% vs. 72–86% on HumanEval, 47–57% vs. 30–40% on CoderEval, and 55–78% vs. 34–65% on SWE-Bench-verified.Source: §6.2 · p. 15
  • Iterative feedback improves repair, with mixed and LLM-Expert yielding the largest gains; performance gains typically stabilize after two to three iterations.Most models show sharp gains from repair@1 to repair@2, while performance often plateaus by repair@3, especially with compiler, LLM-Skilled, and minimal feedback.Source: §6.2 · p. 15

7 RQ4: Prompting Techniques Impact

The prompt study tests enhancements and component removals on Claude-3.5 using single-iteration CoderEval repairs. Structured reasoning and relevant examples improve performance, while removing semantic cues produces marked declines.

7.1 Design

  • Claude-3.5 is evaluated on 100 randomly sampled CoderEval errors in single-iteration repair, with each prompting technique tested independently against the RQ1 baseline.The authors select Claude-3.5 based on prior performance and avoid combining prompt techniques to isolate their individual effects.Source: §7.1 · p. 16
  • The tested enhancements include CoT, fixed few-shot examples, BM25-retrieved ES-shot examples, Self-Ask, SG-ICL, Step-Back Prompting, and Rephrase and Respond.ES-shot retrieves one example per task; the strategies are evaluated individually.Source: §7.1 · p. 16
  • The ablation removes role-play, coding guidelines, docstrings, or project context to assess the contribution of each baseline prompt component.The removed elements respectively frame the model as an expert assistant, supply repair guidance, provide function/class descriptions, or supply project-level information.Source: §7.1 · p. 16

7.2 Results

  • Rephrase and Respond and Chain-of-Thought each improve Claude-3.5 by 3.1% over the 49.5% baseline; example selection reaches 51.6%.The authors report that contextually relevant examples outperform static demonstrations.Source: §7.2 · p. 16
  • Few-shot learning yields only marginal gains at 50.5%, while Step-Back Prompting and Self-Ask do not exceed the 48.5% baseline; SG-ICL scores 46.4%.The authors suggest automatically generated examples may introduce noise that undermines repair accuracy.Source: Figure 8 · p. 17
  • Removing role-play or docstrings reduces performance to 47.4%, while removing context or guidelines yields 48.5% against the 49.5% baseline.The authors describe docstrings as semantic guidance about intended functionality; context supplies project information such as related APIs and class definitions.Source: §7.2 · p. 17
  • The authors identify structured reasoning and example relevance as improvement principles, but note that reasoning-oriented strategies do not all generalize effectively to code repair.They associate RR and CoT with improved comprehension and accurate correction, while citing SBP, SA, and SG-ICL as counterexamples.Source: §7.2 · p. 17

8 Threats to validity

The validity discussion identifies boundaries in benchmark scope, error realism, feedback coverage, and empirical repeatability. It also describes measures taken to address some of these concerns.

Threats in Benchmark Construction

  • The benchmark covers Python tasks from HumanEval, CoderEval, and SWE-Bench-verified, limiting generalizability to other languages and coding scenarios.The authors note that FeedbackSynth is language-agnostic and can be extended.Source: §8 · p. 18

Threats in Erroneous Code

  • Error realism is addressed through rule-based mutations, LLM-induced perturbations, and naturally incorrect LLM outputs.Together, these strategies cover systematic mistakes and authentic failure cases commonly observed in practice.Source: §8 · p. 18

Threats in Feedback Types

  • Feedback coverage excludes interactive reviewer dialogue and domain-specific logs, and some feedback is LLM-simulated, raising fidelity concerns.The authors note that noisy or imperfect feedback occurs naturally in practice.Source: §8 · p. 18

Threats in Empirical Study

  • Multi-iteration experiments ran once per setting, while RQ1 was repeated twice with consistent trends; possible training-data leakage is also acknowledged.The study aims to compare feedback effects rather than measure absolute repair success rates.Source: §8 · p. 18

9 Related Work

The related work surveys LLM-based repair approaches and code benchmarks, noting that prior benchmarks largely emphasize single-iteration repair with limited feedback types. FeedbackEval is positioned as covering diverse errors, feedback modalities, and iterative repair.

9.1 LLM-based Code Repair

  • LLM-based repair approaches include cloze-style completion, static-analysis and retrieval-augmented patching, vulnerability repair, template-guided fine-tuning, and iterative conversation.ChatRepair is reported to resolve 162 of 337 bugs cost-effectively.Source: §9.1 · p. 18

9.2 Code Benchmark for LLMs

  • Code benchmarks span generation, translation, summarization, completion, and repair, including Java-focused CoderUJB, artificial-bug MuBench, and real-world Python SWE-Bench.SWE-Bench includes linked issue descriptions, failing tests, and ground-truth patches.Source: §9.2 · p. 19
  • Existing benchmarks mainly assess single-iteration repairs with limited feedback types, whereas FeedbackEval adds diverse error types, six feedback modalities, and iterative repair evaluation.The section introduces related work on LLM-based code repair and code benchmarks.Source: §9 · p. 18, §9.2 · p. 19

10 Conclusion

The conclusion presents FeedbackEval as a benchmark for feedback-driven code repair and summarizes findings on feedback types, iteration, and prompting. It also identifies hybrid feedback and broader language and error coverage as future directions.

10 Conclusion

  • Mixed feedback yields the highest repair success, with LLM-Expert and test feedback providing strong targeted guidance, while LLM-Skilled is least effective.Compiler and minimal feedback offer moderate gains; task complexity magnifies feedback differences, especially in repository-level scenarios.Source: §10 · p. 19
  • Iterative feedback improves repair performance, but gains plateau after two to three iterations.Source: §10 · p. 19
  • Structured reasoning techniques RR and CoT yield the largest prompt improvements, and dynamic ES-Shot selection surpasses traditional few-shot learning.Removing docstrings or role-play produces the most severe degradation, with semantic understanding of intended functionality more essential than environmental context.Source: §10 · p. 19
  • The authors plan to explore hybrid feedback strategies and extend FeedbackEval to other programming languages and broader error categories.Source: §10 · p. 19
Loading 2504.06939v2…