AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

Contribution

• AgentCoder uses three agents—programmer, test designer, and test executor—to generate code, create tests, execute them, and iteratively refine failures.The loop ends when all tests pass or the iteration budget is exhausted.Source: §1 · p. 2

Design

• The test designer agent generates accurate, diverse, and comprehensive tests independently of the code, covering basic, edge, and large-scale cases.This separation is intended to preserve test objectivity and avoid bias from seeing the generated code.Source: §1 · p. 2

Results

• 96.3% and 91.8% pass@1 on HumanEval and MBPP with GPT-4 use 56.9K and 66.3K tokens, versus 90.2% and 78.9% with 138.2K and 206.5K tokens for state-of-the-art methods.The reported comparison combines performance and token overhead across the two benchmarks.Source: §1 · p. 2
• AgentCoder outperforms all evaluated baselines across code-generation experiments with 14 LLMs and 16 optimization approaches.With GPT-3.5, it reaches 77.4% and 89.1% pass@1, compared with 69.5% and 63.0% for the state of the art.Source: §1 · p. 3

Analysis

• Ablation analyses examine the contributions of different agents, test quality, and the need to separate code generation from test-case design.The reported studies compare agent configurations and test-generation strategies.Source: §1 · p. 3

2.1 Large Language Model for Code Generation

• Code-generation models include CodeBERT [13], PLBART, CodeGPT [40], GraphCodeBERT [15], CodeT5+ [36], Codex [6], and CodeGen [28].These models use pretrained code corpora, structured representations, or large-scale foundation-model architectures.Source: §2.1 · p. 3

2.2 Enhancing Code Generation through Prompt Engineering

• Prompt-based methods include self-refinement [26], Self-Debugging [9], CodeCoT, and Self-Collaboration [11], but generated tests or model-only debugging can miss functional or execution-detectable errors.CodeCoT generates code and tests with one model, while Self-Collaboration predicts bugs without executing code locally.Source: §2.2 · p. 3

2.3 Multi-agent Collaboration

• Existing multi-agent frameworks address varied tasks, whereas AgentCoder specifically addresses code generation through specialized collaboration.The paper positions AgentCoder as distinct from non-code multi-agent systems.Source: §2.3 · pp. 3–4
• AgentCoder responds to multi-agent limitations with independent test generation, execution-based feedback, and three agents instead of larger teams such as MetaGPT [17] and ChatDev [31].The stated goals are more effective feedback and lower communication token overhead.Source: §2.3 · p. 4

3 Methodology

• The pipeline sends task requirements to the programmer and test designer, then executes their code and tests locally until all tests pass.Failures return error messages to the programmer for regeneration; successful code is returned to the user.Source: §3 · p. 4

3.1 Programmer agent: code generation with Chain-of-Thought instruction

• The programmer agent uses four Chain-of-Thought stages: task clarification, algorithm selection, pseudocode creation, and executable code generation.It subsequently refines incorrect code using feedback from the other agents.Source: §3.1 · pp. 4–5

3.2 Test designer agent: generating basic, edge, and large scale tests

• The test designer generates basic, edge, and large-scale tests independently of the full code snippet to improve objectivity and expose diverse failures.The tests cover normal functionality, boundary or unusual inputs, and high-load performance or scalability conditions.Source: §3.2 · p. 5

3.3 Test executor agent: code validation and feedback Integration

• The test executor is a Python script that validates code and tests in a local environment, returns passing code, or sends execution errors to the programmer.It monitors terminal output to determine whether all tests pass and to report errors such as syntax failures.Source: §3.3 · p. 5
• Table 1 reports end-to-end results for HumanEval, HumanEval-ET, MBPP, and MBPP-ET, including baseline comparisons and improvement rates over zero-shot models.Missing baseline results are marked with “-”, and the final three rows report AgentCoder alone where no baseline optimization results exist.Source: Table 1 · p. 6

4.2 RQ1: How does AgentCoder perform?

• AgentCoder outperforms all compared base LLMs and optimization approaches across the evaluated datasets, with GPT-4 reaching 91.5% mean pass@1 and GPT-3.5-turbo reaching 84.1%.The evaluation compares 12 LLMs and 16 optimization methods across HumanEval, MBPP, and enhanced variants.Source: §4.1 · pp. 6–7, §4.2 · p. 7
• On enhanced datasets with more comprehensive tests, AgentCoder maintains performance comparable to the original datasets, while other approaches perform significantly worse.The paper attributes this robustness largely to rigorous tests generated by the test designer agent.Source: §4.2 · p. 7

4.3 RQ2: How do different agents contribute to the effectiveness of AgentCoder?

• Adding the test designer and test executor improves pass@1, while omitting either agent remains below the full three-agent system.With GPT-3.5-turbo, programmer-only scores 61.0% on HumanEval and 47.9% on MBPP; programmer plus executor scores 64.6% and 69.3%.Source: §4.3 · pp. 7–8

4.4 RQ3: How do code refinement iterations affect AgentCoder’s effectiveness?

• Increasing refinement iterations from 1 to 5 raises pass@1 on HumanEval, HumanEval-ET, MBPP, and MBPP-ET.The reported increases are 74.4% to 79.9%, 73.2% to 77.4%, 84.1% to 89.9%, and 80.3% to 89.1%, respectively.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 RQ4: How accurate are the tests generated by the test designer agent?

• AgentCoder’s generated tests are more accurate than the compared single-agent and multi-agent alternatives.With GPT-3.5-turbo, accuracy reaches 87.8% on HumanEval and 89.9% on MBPP, versus 47.0% and 57.2% for GPT-3.5-turbo alone; GPT-4 AgentCoder reaches 89.6% on HumanEval versus 79.3% for MetaGPT.Source: §4.5 · p. 8, Table 4 · p. 8

4.6 RQ5: How adequate are AgentCoder’s test cases in code coverage?

• AgentCoder’s tests achieve the highest reported code coverage, reaching 87.5% and 89.5% with GPT-3.5-turbo versus 77.2% and 82.9% for CodeCoT.With GPT-4, AgentCoder reaches 91.7% HumanEval coverage versus 81.7% for MetaGPT.Source: §4.6 · p. 9, Table 5 · p. 9

4.7 RQ6: Should programmer and test designer be separated to different agents?

• Separating programming and test design across agents improves pass@1, test accuracy, and code coverage over a single-agent setup.For HumanEval and MBPP, multi-agent pass@1 is 79.9% and 89.9%, versus 71.3% and 79.4% for one agent; the paper links this to distraction and test bias.Source: §4.7 · p. 9, Table 7 · p. 9, Table 6 · p. 9

• 96.3% and 91.8% pass@1 are achieved by GPT-4 AgentCoder on HumanEval and MBPP, respectively.Source: §5 · p. 10
• AgentCoder uses programmer, test designer, and test executor agents to improve code generation through automated test generation.Source: §5 · p. 10
• AgentCoder’s HumanEval and MBPP token overheads are 56.9K and 66.3K, compared with 138.2K and 206.5K for current state-of-the-art models.Source: §5 · p. 10

• Each agent must format its output as separate py[Code] and py[TestCases] blocks.Source: §A.1 · p. 14
• The executor extracts code and test cases by removing surrounding sentences before analysis.Source: §A.1 · p. 14
• This response format enables focused and accurate execution by the test executor agent.Source: §A.1 · p. 14

• AgentCoder requires fewer tokens and less execution time than baseline multi-agent frameworks.Source: §A.2 · p. 14
• Table 9 reports pass@1 and token usage for AgentCoder and baselines using GPT-4.Source: Table 9 · p. 14, §A.2 · p. 14
• Token and overhead measurements are reported separately for HumanEval and MBPP.Source: Table 9 · p. 14

• AgentCoder generates more fine-grained tests than CodeCoT in the illustrated HumanEval and MBPP examples.Source: §A.3 · p. 14
• The examples include tests for inputs whose lists contain no elements.Source: §A.3 · p. 14
• Considering such execution outcomes can improve the reliability of generated code for edge behaviors.Source: §A.3 · p. 14

A.4 Case Illustration on HumanEval dataset using AgentCoder

• Figures 6–10 provide prompt and response examples for AgentCoder’s agents and test executor.Figures 6–7 show the programmer prompt and response, Figures 8–9 show the test designer prompt and response, and Figure 10 shows test-executor source code.Source: §A.4 · p. 14
• The programmer agent is illustrated through paired prompt and response examples.Source: §A.4 · p. 14
• The test designer agent and test executor are illustrated through their response example and source code, respectively.Source: §A.4 · p. 14

A.5 Case Illustration of the programmer + test executor agent

• AgentCoder’s programmer and test executor pipeline is illustrated in Figure 11.Source: §A.5 · p. 14

A.6 Case Illustration of the programmer + test designer

• The programmer and test designer pipelines are illustrated in Figures 12 and 11, respectively.Source: §A.6 · p. 14, §A.5 · p. 14
• AgentCoder uses abs() to handle negative values, while CodeCoT omits this absolute-value check in the illustrated HumanEval example.Source: Figure 2 · p. 15
• AgentCoder’s generated HumanEval tests cover both left-lower and left-larger-than-right scenarios, whereas CodeCoT tests only the left-lower case.Source: Figure 3 · p. 16
• For the MBPP example, AgentCoder’s tests cover the empty-list edge case that CodeCoT overlooks, although both generated code samples are correct.Source: Figure 4 · p. 17, Figure 5 · p. 18