PaperlayerView exact revision on arXiv

Source-linked AI summary

CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging

Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez

arXiv:2502.05664v1cs.CLcs.AIcs.SE

TL;DR

LLM code-generation approaches that rely on external iterative debuggers depend heavily on the quality of the initial program. CODESIM uses planning, coding, and debugging agents with step-by-step simulation to verify plans and debug code. It achieves state-of-the-art results across benchmarks, while its token use remains higher than direct prompting.

  • Problem

    Tool-based iterative refinement depends heavily on initial-code quality, while generating correct first-pass programs for complex tasks remains challenging.

    Source: §1 · p. 1

  • Method

    CODESIM uses planning, coding, and debugging agents, applying step-by-step input/output simulation to verify plans and internally debug code.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 4

  • Results

    CODESIM achieves state-of-the-art performance across seven benchmarks, including 95.1% on HumanEval with GPT-4o.

    Source: §1 · p. 2, §5.1 · p. 6

  • Takeaways & Limitations

    The authors report potential further performance improvement when CODESIM is integrated with external debugging tools.

    Source: §1 · p. 2

    CODESIM uses around 13,640 tokens on average, compared with 560 tokens for direct prompting.

    Source: §8 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have made significant strides in code generation and problem solving. Current approaches employ external tool-based iterative debuggers that use compiler or other tool-based runtime feedback to refine coarse programs generated by various methods. However, the effectiveness of these approaches heavily relies on the quality of the initial code generation, which remains an open challenge. In this paper, we introduce CodeSim, a novel multi-agent code generation framework that comprehensively addresses the stages of program synthesis-planning, coding, and debugging-through a human-like perception approach. As human verifies their understanding of any algorithms through visual simulation, CodeSim uniquely features a method of plan verification and internal debugging through the step-by-step simulation of input/output. Extensive experiments across seven challenging competitive problem-solving and program synthesis benchmarks demonstrate CodeSim's remarkable code generation capabilities. Our framework achieves new state-of-the-art (pass@1) results-(HumanEval 95.1%, MBPP 90.7%, APPS 22%, and CodeContests 29.1%). Furthermore, our method shows potential for even greater enhancement when cascaded with external debuggers. To facilitate further research and development in this area, we have open-sourced our framework in this link (https://kagnlp.github.io/codesim.github.io/).

1 Introduction

Complex programming tasks remain difficult for LLMs, and existing tool-based refinement depends heavily on the quality of the initial code. CODESIM combines planning, coding, and debugging agents with step-by-step simulation to verify plans and debug code.

1 Introduction

  • Existing dual-pass methods refine initial programs using runtime or diagnostic feedback, but their effectiveness depends heavily on initial-code quality and they require numerous LLM-tool interactions.The passage cites Shi et al. (2024), Jin et al. (2024b), Zhong et al. (2024), and Levin et al. (2024).Source: §1 · p. 1
  • Complex tasks require algorithmic, data-structure, and problem-solving knowledge, while competitive programming also demands constraint adherence and passing unit tests.The paper identifies these demands as challenges for first-pass program synthesis.Source: §1 · p. 1
  • CODESIM uses planning, coding, and debugging agents, with simulation verifying plans and supporting internal debugging.The approach is inspired by how people visualize and refine algorithms; Figure 1 depicts the stages and iterative process.Source: §1 · p. 2, Figure 1 · p. 2
  • CODESIM is evaluated across seven programming-synthesis benchmarks and is reported to achieve state-of-the-art performance, with potential gains from external debugging.The experiments include multiple LLMs and ablation studies; the framework is to be open-sourced.Source: §1 · p. 2

2 Related Work

Related work spans code-generation techniques, LLMs, and multi-agent systems. CODESIM combines established paradigms and distinguishes itself through simulation-driven planning and LLM-based debugging.

2 Related Work

3 CODESIM

CODESIM organizes code generation into planning, coding, and debugging agents. Step-by-step input/output simulation verifies plans and helps locate code errors, with adaptive cycles linking the agents.

3 CODESIM

  • CODESIM addresses complex problem solving with planning, coding, and debugging agents, using simulation to verify plans and perform internal debugging.The approach is presented as mirroring how people visualize and refine algorithms.Source: §3 · p. 4

3.1 Planning Agent

  • The Planning Agent generates one relevant problem-solution exemplar, devises a plan, and simulates it step by step on a sample input to check the expected output.If the simulation output does not match, the agent revises the plan; otherwise, the plan is deemed valid.Source: §3.1 · p. 4

3.2 Coding Agent

  • The Coding Agent takes the problem description and Planning Agent's plan as input and translates them into executable code.Sample input/output tests determine whether the code is returned or passed to debugging.Source: §3.2 · p. 4

3.3 Debugging Agent

  • The Debugging Agent receives the problem, plan, code, and execution log, then simulates failing inputs step by step to locate bugs and revise the code.Unlike the cited LATS, AgentCoder, and Reflexion approaches, it does not require additional test-case generation.Source: §3.3 · p. 4

3.4 Adaptive Iteration

  • The adaptive process sends code that fails sample tests to debugging for up to d attempts, then returns to planning if needed, for at most p cycles overall.It returns code that passes all sample I/Os as the final solution for evaluation on hidden tests; the stated complexity is O(pd).Source: §3.4 · pp. 4–5

4 Experimental Setup

The experiments cover basic programming and competitive-programming benchmarks, compare CODESIM with code-generation baselines across several LLMs, and report pass@1. The paper also describes its reproducibility resources and iteration settings.

4.1 Datasets

  • Evaluation uses five basic programming benchmarks and two competitive-programming datasets: APPS and CodeContest.The basic benchmarks are HumanEval, HumanEval-ET, EvalPlus, MBPP, and MBPP-ET; datasets were collected from the MapCoder repository for fair comparison.Source: §4.1 · p. 5

4.2 Baselines and Metric

  • CODESIM is compared with MapCoder and other code-generation methods using proprietary and open-source LLMs, with pass@1 as the reported metric.The baselines include Direct, Chain of Thought, Self-Planning, Analogical Reasoning, Self-collaboration, Reflexion, and LATS; AgentCoder is excluded for reproducibility issues.Source: §4.2 · pp. 5–6

4.3 Reproducibility

  • The authors report open-sourcing code and result logs, with default planning and debugging tries set to p = 5 and d = 5 for simple tasks and p = 3 and d = 3 for competitive tasks.For CodeContest with GPT-4, the competitive-task setting uses p = 3 and d = 5.Source: §4.3 · p. 6

5 Results

CODESIM outperforms baselines on simple and contest-level code-generation tasks, and it also performs strongly across open-source LLMs. The results span foundational benchmarks, competitive programming datasets, and multiple model architectures.

5.1 Basic Code Generation

  • With GPT-4o, CODESIM scores 95.1% on HumanEval, 87.2% on EvalPlus, and 90.7% on MBPP, for an 82.7% average versus MapCoder’s 79.0%.CODESIM also averages 75.1% with ChatGPT and 81.3% with GPT-4.Source: §5.1 · p. 6

5.2 Competitive Problem Solving

  • With GPT-4, CODESIM reaches 29.1% on CodeContests and 22.0% on APPS, versus MapCoder’s 25.3% average on contest-level tasks.Table 3 reports Pass@1 results for CodeContests and APPS.Source: §5.2 · p. 6, Table 3 · p. 6
  • With ChatGPT, CODESIM scores 16.4% on CodeContests and 12.0% on APPS, averaging 14.2% versus MapCoder’s 12.0%.Table 3 reports Pass@1 results for CodeContests and APPS.Source: §5.2 · p. 6, Table 3 · p. 6

5.3 Performance Across Open-source LLMs

  • On LLaMa3.1-70B, CODESIM reaches 90.2% on HumanEval and 76.2% on EvalPlus, averaging 80.1% and closely matching GPT-4o.CODESIM averages 75.8% with Gemma2-9B and consistently outperforms other methods across the evaluated open-source models; MapCoder is excluded because those models often struggle with its output format.Source: §5.3 · pp. 6–7, Table 4 · p. 6

6 Ablation Studies and Analyses

The analyses examine which CODESIM components and settings matter, how performance varies across tasks, and whether external debugging can improve results. They also identify efficiency trade-offs and difficulty-specific challenges.

6.1 Impact of Different Agents

  • Both simulation-guided plan verification and simulation-based debugging contribute significantly to CODESIM’s performance.The ablation uses GPT-4o on HumanEval and reports Pass@1 for different CODESIM versions.Source: §6.1 · p. 7, Table 5 · p. 7

6.2 Fine-grained Analysis of the Impact of Simulation

  • CODESIM outperforms other approaches across simple and multi-agent settings, with both open-source and proprietary LLMs.The authors attribute these results to the impact of incorporating simulation.Source: §6.2 · p. 7, Table 6 · p. 7

6.3 Impact of Varying Programming Languages

  • CODESIM maintains strong performance across programming languages on xCodeEval.The evaluation uses ChatGPT; Table 7 reports Pass@1 results by language.Source: §6.3 · p. 7, Table 7 · p. 7

6.4 Use of External Debugger

  • With GPT-4o and LDB, CODESIM achieves 97.6% second-pass accuracy, setting a new state-of-the-art result for a dual-pass approach; its GPT-4o first pass also surpasses Reflexion’s 94.5% second-pass performance.Source: §6.4 · p. 8

6.5 Qualitative Example

  • Figure 2 illustrates simulation enhancing the plan and helping debug code through the same technique.A complete example, including LLM output, is provided in Appendix 12.Source: §6.5 · p. 8

6.6 Impact of p and d

  • Increasing the maximum planning and debugging steps proportionately improves performance, but also raises API calls and token consumption.The authors describe p and d as adjustable parameters for balancing accuracy and cost.Source: Figure 3 · p. 8, §6.6 · p. 8

6.7 Impact of Number of Sample I/Os

  • Adding five sample I/Os from HumanEval-ET raises ChatGPT accuracy to 89%, a 3.5% improvement over the previous 86%.HumanEval averages 2.82 sample I/Os per example in this analysis.Source: §6.7 · p. 8

6.9 API Call and Token Analysis

  • 4.13 thousand fewer tokens on average and a 7.1% accuracy increase distinguish CODESIM from MapCoder.Table 9 compares average API calls and token consumption; the reported token reduction is relative to MapCoder.Source: §6.9 · p. 8, Table 9 · p. 9

7 Conclusion and Future Work

CODESIM combines planning, coding, and debugging agents with simulation for plan verification and debugging. The paper concludes that it outperforms existing approaches and identifies mathematical reasoning and question answering as future domains.

7 Conclusion and Future Work

  • CODESIM integrates planning, coding, and debugging agents for programming problem solving.The framework uses multi-agent prompting capabilities of LLMs.Source: §7 · p. 9
  • Simulation supports plan verification and debugging, and the authors report that CODESIM outperforms existing state-of-the-art approaches.The conclusion characterizes the performance advantage as substantial.Source: §7 · p. 9
  • Future work will extend the approach to mathematical reasoning and question answering.Source: §7 · p. 9

8 Limitations

The paper identifies token cost, tool dependence, and unstudied code optimization as limitations or open directions. It also notes that more sample I/Os could help, while accurate generation of those inputs remains future work.

8 Limitations

  • The authors have not explored using external tools to assist during simulation, and aim to achieve the best performance without external tools.Source: §8 · p. 9
  • CODESIM uses around 13,640 tokens on average, compared with 560 for direct prompting, leaving room to improve efficiency.The authors report lower token consumption than MapCoder but higher consumption than direct prompting.Source: §8 · p. 9
  • The study evaluates generated-code correctness but does not study optimization, including test-time or memory optimization.Source: §8 · p. 9
  • More sample I/Os could potentially improve performance, but methods for generating accurate additional I/Os remain future work.Source: §8 · p. 9
  • The authors note that machine-generated code should be run inside a sandbox to avoid potential risks.Source: §8 · p. 9

9 Algorithm of CODESIM

Algorithm 1 describes CODESIM’s prompting process: planning and validation precede coding, followed by debugging when tests fail. The process is bounded by maximum planning and debugging steps.

9 Algorithm of CODESIM

  • The algorithm repeats plan generation and validation for up to p planning steps, refining a plan when validation feedback is negative.Source: §9 · p. 14
  • The Coding Agent generates code from the plan and tests it on sample I/Os; failed tests send code to the Debugging Agent for up to d attempts.Source: §9 · p. 14
  • The algorithm returns code when it passes the sample I/O test, or returns the current code after the planning loop ends.Source: §9 · p. 14

10 Exclusion of AgentCoder

The section explains why AgentCoder is excluded from comparison and outlines CODESIM’s agent prompting and plan simulation. A HumanEval example shows how simulating a failed test case can reveal a mismatch with the problem requirement and guide code revision.

10 Exclusion of AgentCoder

  • On HumanEval, AgentCoder reproduction attempts achieved 56.7% with ChatGPT after four iterations and 17.7% with GPT-4 after two, versus its reported 79.9% and 96.3% accuracies, respectively.The attempts consumed 11.9 million tokens with ChatGPT and 10.4 million with GPT-4.Source: §10 · p. 14
  • AgentCoder’s MBPP results were also excluded because it used all test cases as public cases, deviating from standard practice.The authors additionally report unaddressed reproducibility issues on AgentCoder’s GitHub page.Source: §10 · p. 14

11 Details Promptings of CODESIM

  • CODESIM’s Planning Agent generates a plan, verifies it through simulation, and refines it if needed; the Coding Agent implements it, and the Debugging Agent activates if sample tests fail.The prompts include rationales for each sentence.Source: §11 · p. 14

12 Example Problem

  • The problem asks for an ascending list of even integers between positive inputs a and b, inclusive, regardless of their order; the reported output is [2, 4, 6, 8].Source: §12 · pp. 19, 23
  • The simulation returns [10, 12, 14] for generate_integers(10, 14), although the expected output is [].The prompt’s examples define the task as returning even digits, not even numbers in the interval.Source: §12 · pp. 17, 25, 26
  • The debugging notes identify the plan as correct but the code generation as wrong: it selects even numbers rather than digits between a and b.The simulation is proposed as a way to distinguish a faulty plan from a faulty translation of the plan into code.Source: §12 · pp. 25–27
  • The revised code checks digits of numbers in the interval, appends even digits, then removes duplicates and sorts the result.The correction notes describe digit checking, uniqueness via a set, and ascending order via sorting.Source: §12 · p. 27
Loading 2502.05664v1…