PaperlayerView exact revision on arXiv

Source-linked AI summary

RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidance

Haolin Jin, Zechao Sun, Huaming Chen

arXiv:2410.01242v2cs.SEcs.AIcs.CL

TL;DR

LLM code generation depends on clear, complete prompts, yet planning-first approaches can be inflexible when refinement is needed. RGD coordinates guide, debug, and feedback agents with memory retrieval and iterative failure analysis; it reports state-of-the-art performance across benchmarks, including a 9.8% improvement over direct generation on HumanEval with GPT-4o.

  • Problem

    LLM code generation depends on clear, complete task descriptions, while planning-first approaches can be inflexible when later refinements are needed.

    Source: §I · p. 1

  • Method

    RGD uses Guide, Debug, and Feedback LLM agents, with retrieved guides and task information informing code generation and failure analysis.

    Source: §I · p. 1, B. Guide LLM Agent · p. 3

  • Results

    RGD achieved state-of-the-art performance across datasets, including a 9.8% improvement over the direct-generation baseline on HumanEval with GPT-4o.

    Source: §I · p. 2, A. Main Result · p. 5

  • Takeaways & Limitations

    RGD combines stored memory and dynamic failure analysis to improve code-generation accuracy across multiple benchmarks.

    Source: §VI · p. 6

    APPS experiments used 100 samples, and LDB was not tested there because its program required significant modifications.

    Source: A. Main Result · p. 5

Abstract

from arXiv · show

Large Language Models (LLMs) have shown incredible potential in code generation tasks, and recent research in prompt engineering have enhanced LLMs' understanding of textual information. However, ensuring the accuracy of generated code often requires extensive testing and validation by programmers. While LLMs can typically generate code based on task descriptions, their accuracy remains limited, especially for complex tasks that require a deeper understanding of both the problem statement and the code generation process. This limitation is primarily due to the LLMs' need to simultaneously comprehend text and generate syntactically and semantically correct code, without having the capability to automatically refine the code. In real-world software development, programmers rarely produce flawless code in a single attempt based on the task description alone, they rely on iterative feedback and debugging to refine their programs. Inspired by this process, we introduce a novel architecture of LLM-based agents for code generation and automatic debugging: Refinement and Guidance Debugging (RGD). The RGD framework is a multi-LLM-based agent debugger that leverages three distinct LLM agents-Guide Agent, Debug Agent, and Feedback Agent. RGD decomposes the code generation task into multiple steps, ensuring a clearer workflow and enabling iterative code refinement based on self-reflection and feedback. Experimental results demonstrate that RGD exhibits remarkable code generation capabilities, achieving state-of-the-art performance with a 9.8% improvement on the HumanEval dataset and a 16.2% improvement on the MBPP dataset compared to the state-of-the-art approaches and traditional direct prompting approaches. We highlight the effectiveness of the RGD framework in enhancing LLMs' ability to generate and refine code autonomously.

I. INTRODUCTION

LLM code generation often relies on prompt engineering, but single-pass generation can miss edge cases and iterative repairs may not use failures effectively. RGD coordinates specialized LLM agents, test feedback, and retrieved guides to refine code.

I. INTRODUCTION

  • Single-pass generation can miss edge cases, while iterative reflection may fail to prompt effective changes and planning-first approaches can resist later refinements.LLMs may consequently repeat the same code across iterations or overlook requirements when task descriptions are incomplete.Source: §I · p. 1
  • RGD assigns guide generation, initial code generation, and failure analysis to specialized LLMs that collaborate on code repair.The Guide LLM creates a guide from the task description; the Debug LLM uses it for initial code generation, and the Feedback LLM analyzes failures and potential fixes.Source: §I · p. 1
  • RGD tests generated code against visible and hidden cases, iterating until all pass or the maximum iteration limit is reached.Failure analysis also considers passing cases to help prevent later generations from breaking previously passing tests.Source: Figure 1 · p. 2, §I · p. 2
  • The Guide LLM retrieves similar past tasks and successful generation guides from a memory pool to inform guidance for new tasks.Unlike code-focused retrieval, the pool stores relevant guides and task descriptions.Source: §I · p. 2
  • RGD reportedly achieved state-of-the-art performance across the evaluated benchmarks, surpassing frameworks including LDB.The evaluation covered HumanEval, HumanEval-ET, MBPP, MBPP-ET, and APPS using GPT-4o and GPT-4o-mini.Source: §I · p. 2

II. RELATED WORK

Related work covers LLM code-generation and debugging approaches alongside information retrieval methods that supply retrieved context to generation systems.

II. RELATED WORK

  • Prior LLM code-generation work includes prompt engineering, chain-of-thought and tree-of-thought reasoning, and self-repair from execution results.Other approaches use external tools for dynamic code correction.Source: §II · p. 2
  • LLM reasoning approaches include tree-of-thought, which represents possible reasoning states as nodes connected by transitions.The related work also notes chain-of-thought prompting and code debugging from execution results.Source: §II · p. 2
  • Information retrieval supplies helpful information from local or cloud storage to generation tasks, with RAG and GraphRAG moving zero-shot settings toward few-shot generation.Retrieval ranking can increase reliability and reduce hallucinations; RAG has also been applied to code generation.Source: §II · pp. 2–3

III. REFINMENT AND GUIDANCE DEBUGGING

RGD defines benchmark tasks through their descriptions, tests, and entry point, then uses a Guide Agent to create and refine generation guides. During debugging, it retrieves similar successful task guides from memory and combines them with the current guide and failure analysis.

A. Definitions

  • RGD represents each HumanEval and MBPP task as Q, Tv, Th, and E: its description, visible tests, hidden tests, and entry point.Source: A. Definitions · p. 3

B. Guide LLM Agent

  • The Guide Agent creates an initial guide from the task description and entry point, then uses task-similar memory guides during debugging.Memory matching is omitted during initial generation to avoid unnecessary token use and context overhead.Source: B. Guide LLM Agent · p. 3
  • Only tasks passing both visible and hidden tests contribute keyword sets to memory, which stores task descriptions, guides, and keywords.GPT-4o-mini extracts keywords from the task description and generated code.Source: B. Guide LLM Agent · p. 3
  • For a new task, hybrid retrieval uses SBERT description matching and BM25 keyword matching to find similar tasks and their guides.The retrieval algorithm returns the top three guides; the final guide augments a new initial guide with a retrieved guide and Feedback Agent failure analysis.Source: B. Guide LLM Agent · p. 3
  • The memory pool is intended to let the Guide Agent build on successful prior guides rather than start from scratch for each task.The authors state this enrichment contributes to more accurate, context-aware guides and better code generation.Source: B. Guide LLM Agent · pp. 3–4

C. Debug LLM Agent

The Debug Agent generates code from the task description, entry point, and Guide Agent’s guide. RGD filters simpler tasks in the first round so debugging resources can focus on harder problems.

C. Debug LLM Agent

  • Initial code generation uses the task description, entry point, and generation guide without memory matching or failure analysis.Prompts for initial generation and code fixing differ; the authors note that running seed executions for MBPP can be time-consuming, especially for MBPP-ET.Source: C. Debug LLM Agent · p. 4
  • RGD filters simpler tasks in the first round, directing computational resources toward more complex problems.The authors say this streamlines debugging and enhances its speed and effectiveness.Source: C. Debug LLM Agent · p. 4

D. Feedback LLM Agent

The Feedback Agent examines execution outcomes and produces failure analysis and modification suggestions. Its analysis is kept separate from code regeneration and supplied as auxiliary information to the Guide and Debug Agents.

D. Feedback LLM Agent

  • The Feedback Agent records passed and failed visible tests, failed code, task description, and any execution exception for analysis.The analysis consolidates execution results, including exception type when one occurs.Source: D. Feedback LLM Agent · p. 4
  • Failure analysis and proposed modifications are handled separately because direct regeneration from failed code and execution results can underperform single-task handling.The resulting failure analysis supports both the Guide Agent and Debug Agent.Source: D. Feedback LLM Agent · p. 4
  • Separating analysis from proposed corrections is intended to help LLMs handle complex tasks requiring nuanced debugging and iteration.Source: D. Feedback LLM Agent · p. 4

A. Dataset

The evaluation uses HumanEval and MBPP alongside extended-test variants and a 100-sample APPS subset. Test visibility and validation differ by dataset.

A. Dataset

  • HumanEval uses task-description tests as visible tests and dataset tests as hidden tests, while MBPP uses its first test as the visible test.Source: A. Dataset · p. 4
  • HumanEval-ET and MBPP-ET use a 6:4 visible-to-hidden test ratio, while APPS evaluation uses 100 samples balanced 5:3:2 across introductory, interview, and competition difficulty.Unlike the other datasets, APPS validation uses input-output results and requires a distinct handling approach.Source: A. Dataset · p. 4

B. Method

The evaluation compares RGD with five alternative approaches and reports Pass@1 accuracy over ten iterations. Table I reports results across benchmarks, with Δ denoting improvement over direct generation.

B. Method

  • RGD is compared with Direct, Chain-of-Thought, Self-Planning, Self-Debugging, and LDB.Direct generates code from the task description; the other approaches use prompting, planning, execution feedback, or the LDB framework.Source: B. Method · p. 4
  • Table I reports Pass@1 results for six approaches, with Δ measuring improvement over the Direct baseline.LDB is evaluated on HumanEval/ET and MBPP/ET; the other listed comparison approaches are tested on all five benchmarks.Source: B. Method · p. 4, Table I · p. 5
  • Pass@1 is evaluated over k=10 iterations, counting a problem as solved if it is solved successfully at least once.The evaluation primarily tests GPT-4o and GPT-4o-mini.Source: B. Method · p. 5

V. EXPERIMENT RESULT AND ANALYSIS

RGD outperforms the compared approaches across the reported benchmarks. The results include gains on HumanEval and MBPP-ET, as well as an APPS evaluation using 100 samples.

A. Main Result

  • RGD improves HumanEval Pass@1 by 9.8% over the direct code-generation baseline and 3.1% over LDB with GPT-4o; gains are particularly significant on HumanEval-ET and MBPP-ET, reaching 25.1% on MBPP-ET.The paper attributes the boost mainly to limited visible test cases in the original datasets, which can let code pass visible tests yet fail hidden tests.Source: A. Main Result · p. 5
  • 8.6% is the maximum reported improvement on APPS, using GPT-4o-mini; the APPS evaluation covers 100 samples.LDB was not tested on APPS because doing so would require significant program modifications.Source: A. Main Result · p. 5

B. Ablations Studies

The ablations remove the Memory Pool, Guide Agent, and Failure Feedback in turn. Each removal lowers performance, with the authors reporting larger ablation effects on MBPP than on HumanEval.

B. Ablations Studies

  • The ablations use GPT-4o on HumanEval and MBPP, comparing against original RGD scores of 97.6 Pass@1 and 83.4 Pass@1, respectively.Table II presents the GPT-4o ablation study on those two benchmarks.Source: Table II · p. 5, B. Ablations Studies · p. 5
  • Removing the Memory Pool results in drops of 1.9 percentage points on HumanEval and 5 percentage points on MBPP.Without the pool, the Guide Agent no longer retrieves cases to generate and refine guides.Source: B. Ablations Studies · p. 5
  • Removing the Guide Agent leads to drops of 4.4% on HumanEval and 6.4% on MBPP.The authors identify the Guide Agent as having a significant impact on overall RGD performance.Source: B. Ablations Studies · p. 5
  • Removing Failure Feedback lowers performance on both benchmarks, with the MBPP decrease reaching 9.8%.In this ablation, the Feedback LLM does not generate Failure Analysis for debugging and refinement.Source: B. Ablations Studies · p. 5
  • The authors report that removing any of the three components significantly reduces overall RGD performance, with larger drops tending to occur on datasets with more samples.They cite MBPP’s 500 samples versus HumanEval’s 163 and the accumulation of Memory Pool information during execution as context for the pattern.Source: B. Ablations Studies · pp. 5–6

VI. CONCLUSION

The conclusion presents RGD as a code-generation framework that combines Memory Pool-refined Generation Guides with Feedback Agent analysis. It reports state-of-the-art performance across multiple benchmarks.

VI. CONCLUSION

  • RGD refines Generation Guides using a Memory Pool and incorporates feedback through a dedicated Feedback Agent.The framework uses previously stored memory and dynamic failure analysis to iteratively refine generated code.Source: §VI · p. 6
  • The authors report that selectively extracting relevant guides and refining them based on task similarity boosts code-generation accuracy across multiple benchmarks.They characterize RGD’s experimental performance as state of the art in code generation and debugging.Source: §VI · p. 6
Loading 2410.01242v2…