Source-linked AI summary
Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
Nazmus Ashrafi, Salah Bouktif, Mohammed Mediani
TL;DR
The study examines how multi-agent collaboration and runtime debugging affect LLM code generation, separately and in combination. It chains collaboration before execution-informed debugging and evaluates the approaches across models and benchmarks. A simple two-agent workflow combined with debugging modestly improves mean accuracy while preserving code rigor and similar latency to debugging alone.
Problem
The study examines how multi-agent collaboration and runtime debugging, separately and together, affect LLM code generation across models.
Source: §1 · pp. 1–2Method
Results
Debugging generally outperforms agentic workflows; combining debugging with a two-agent Analyst-Coder workflow improves mean accuracy by 0.68% across 19 LLMs and two datasets while preserving code rigor and similar latency.
Source: §5 · p. 17Takeaways & Limitations
A straightforward agentic workflow combined with debugging offers a balance of functional accuracy and latency across a range of LLMs.
Source: §1 · p. 3Very poor initial code from the ACT phase can complicate debugging, making the integrated technique dependent on strong ACT performance.
Source: §3 · p. 4
Abstract
from arXiv · showhide
The use of large language models (LLMs) for automated code generation has emerged as a significant focus within AI research. As these pretrained models continue to evolve, their ability to understand and generate complex code structures has opened new possibilities for automating intricate programming tasks for the sake of accurate code generation. Although contemporary foundational models demonstrate promoting results, researchers continue to explore optimal post-training strategies to enhance code quality. These include supervised fine-tuning, retrieval-augmented generation (RAG), debugging, and many others. In this paper, we combine two widely used approaches namely multi-agent collaboration and runtime execution information-based debugging, for improving code generation functionality, reliability, and practical applicability. We perform an empirical study in order to extend the evaluation of the individual strategies as well as the proposed composition of the activities of both strategies. Our study use 19 LLMs to examines the performance of individual and the proposed strategies, offering comprehensive insights into how different programming activities compositions and training paradigms influence code generation effectiveness. In particular, we implement a chained system that combines both strategies to assess their combined impact on functional accuracy, code reliability, and generation latency using two benchmark datasets commonly used for code generation. Our findings provide valuable insights for organizations seeking robust AI-driven coding solutions by guiding them in selecting models that can better adapt to complex post-training strategies, ultimately fostering the adoption of more effective and reliable code generation technologies.
1 Introduction
The introduction frames multi-agent collaboration and runtime execution-based debugging as complementary ways to improve LLM code generation. It proposes chaining them and evaluates their effects across models and benchmarks.
1 Introduction
- LLM code generation is presented as a promising application for automating complex programming tasks, with post-training strategies intended to improve generated-code quality and reliability.The introduction names supervised fine-tuning, preference optimization, and retrieval-augmented generation among post-training approaches; some approaches do not require costly fine-tuning.Source: §1 · p. 1
- The study evaluates individual and combined multi-agent collaboration and runtime-debugging strategies across a diverse set of models.The authors aim to identify which model characteristics respond best to the combination and the conditions that optimize its impact.Source: §1 · pp. 1–2
- The paper chains multi-agent code generation with runtime execution-informed debugging, treating collaboration as process-oriented and debugging as product-oriented refinement.The sequence mirrors a workflow in which an analyst, coder, and tester collaborate before code is run and refined using execution feedback.Source: §1 · p. 2
- The evaluation compares functional accuracy, code rigor, and generation latency across system segments and the complete chain.The combined approach is reported to improve functional accuracy across a range of LLMs while maintaining latency comparable to standalone debugging.Source: §1 · pp. 2–3
- A two-agent Analyst-Coder workflow combined with debugging improves mean accuracy by 0.68% across 19 LLMs and two datasets, while a three-agent workflow reduces accuracy for most models.The combined approach significantly outperforms agentic workflow alone at α = 0.15, but the study does not establish a significant improvement over debugging alone.Source: §1 · p. 2
2 Literature Review
The literature review surveys multi-agent collaboration and runtime execution-based debugging as approaches for improving code generation with pretrained LLMs.
2 Literature Review
- The review covers two post-training approaches for code generation: multi-agent collaboration and debugging based on program runtime execution information.It treats them as established approaches whose combination is examined in the paper.Source: §2 · p. 3
2.1 Multi-agent collaboration
- Multi-agent frameworks divide coding work among specialized LLM agents, with examples assigning planning, coding, testing, retrieval, or debugging roles.AgentCoder uses test-design and execution feedback, MapCoder sequences four agents, and self-collaboration uses analyst, coder, and tester roles.Source: §2.1 · p. 3
2.2 Runtime execution information-based debugging
- Runtime-debugging frameworks use execution information such as failures, traces, and variable values to guide code refinement.Examples include print-based debugging, block-level analysis with execution traces, reflection from compiler feedback, and iterative self-debugging.Source: §2.2 · pp. 3–4
3 Methodology
The methodology chains multi-agent code generation with runtime-informed debugging. It uses visible tests to trigger refinement and hidden tests to evaluate the final output.
3 Methodology
- The framework first uses collaboration among specialized agents, then applies runtime execution information to debug and refine the generated code.The authors chose this order to mirror planning, coding, and debugging in a software-engineering workflow.Source: §3 · p. 4
- The workflow may miss errors not covered by its limited visible tests, and very poor initial ACT code can complicate debugging.The experiments also use the same prompts for all models for consistency, although the authors note that this may not be ideal.Source: §3 · p. 4
3.1 Multi-Agent Collaboration Phase
- The Analyst decomposes requirements and plans, the Coder implements the plan, and the Tester reviews functionality, readability, and maintainability for up to three interactions.The three-agent setup is modeled on a modified waterfall workflow and limits tester-coder interactions to retriesCT = 3.Source: §3.1 · p. 4
3.2 Debugging Phase
- When code fails visible tests, block-level CFG analysis and runtime variable logs inform a debugger, whose explanations guide the coder through up to four refinement cycles.The system uses the visible test that caused the failure and adopts block-level decomposition because it achieved the highest accuracy in the cited LDB evaluation.Source: §3.2 · p. 5
4.1 Experiment Setup
The experiment evaluates four research questions about accuracy, code rigor, and latency, using pass@1 across two benchmarks and a range of model configurations. It compares basic generation, agent-collaboration variants, debugging, and chained approaches.
4.1 Experiment Setup
- The study asks how combined strategies affect accuracy, how segments compare, whether code remains accurate under stricter tests, and what latency–quality trade-offs arise.It evaluates 19 models across two code-generation benchmarks.Source: §4.1 · p. 5
4.1.1 Metric
- With one sample per problem, pass@k reduces to pass@1, a binary measure of whether the first generated solution is correct.The single-sample setup is intended to reflect reliance on a model’s first suggestion.Source: §4.1.1 · p. 5
4.1.2 Datasets
- HumanEval+ adds 80 times more tests than HumanEval; a smaller score decline indicates more rigorous code, while a larger drop suggests fragility.HumanEval contains 164 tasks, each with an average of 7.7 tests.Source: §4.1.2 · p. 6
4.1.3 Baseline
- The baselines span basic prompting, analyst–coder collaboration, analyst–coder–tester collaboration, debugging alone, and AC+Debugger; ACT allows up to three coder–tester iterations.The proposed ACT+Debugger approach combines collaboration with debugging.Source: §4.1.3 · p. 6
4.1.4 Models
- The 19-model selection spans open- and closed-source systems, model sizes, training mixtures, and training styles, including models from 3.8B to 671B parameters.The study also includes generalist, instruction-trained, and code-specific models.Source: §4.1.4 · p. 8
4.2 RQ1: Chained system vs. Individual techniques
Across 19 models, ACT+Debug improves mean HumanEval accuracy over ACT, but its advantage over Debug alone is small and not statistically significant under the tested conditions. Further comparisons show that gains vary by model and are more pronounced when ACT and debugging perform similarly.
4.2 RQ1: Chained system vs. Individual techniques
- 64.82% mean accuracy for ACT+Debug exceeded ACT’s 57.16%, while Debug alone reached 63.86%; the gain over ACT was significant, but the 0.96% gain over Debug was not.The comparison used a one-tailed paired t-test at α=0.15.Source: §4.2 · p. 8, §4.2.1 · pp. 8–9, §4.2.2 · p. 9
4.2.1 Comparing the chained approach (ACT + Debugger) with ACT
- ACT+Debug significantly improved accuracy over ACT at α=0.15, and nearly all LLMs improved on HumanEval when debugging was added to ACT.Figure 2 compares the ACT and ACT+Debug accuracy distributions on HumanEval.Source: §4.2.1 · pp. 8–9, Figure 2 · p. 9
4.2.2 Comparing the chained approach (ACT + Debugger) with Debugger
- ACT+Debug’s 0.96% accuracy gain over Debug alone was not statistically significant; debugging may already capture much of the benefit by checking intermediate values and refining code.The authors suggest this may leave less scope for ACT to add significant gains.Source: §4.2.2 · p. 9, Figure 3 · p. 9
4.2.3 Further exploration of the chained approach (ACT + Debugger) vs. Debugger on HumanEval and HumanEval+
- ACT’s added value varied by model: GPT-4o and Claude 3.5 Sonnet improved on HumanEval but not HumanEval+, while several smaller models improved on both benchmarks.Reported gains on HumanEval and HumanEval+ were +11.62 and +7.93 for GPT-4o-mini, +3.04 and +3.04 for Mixtral 8x7B, and +4.26 and +2.44 for Mistral-Instruct v0.2 (7B).Source: §4.2.3 · p. 9, Table 2 · p. 10
- OpenAI models generally showed stable gains from combined approaches, and Claude 3.5 Sonnet reached 95.73% on HumanEval with ACT+Debug, 1.83% above Debug.Claude Haiku models were less responsive; some open-source models also showed little or no benefit, with Gemini 1.0 Pro an exception.Source: §4.2.3 · p. 11
- The ACT+Debug advantage tends to diminish as Debugger’s accuracy lead over ACT widens; gains are more pronounced when their standalone performance is closer.Figure 6 plots the Debugger–ACT accuracy gap against the ACT+Debugger improvement over Debugger across 38 data points.Source: §4.2.3 · pp. 11–12, Figure 6 · p. 12
- Reducing collaboration to AC while retaining debugging showed a similar relationship between the standalone accuracy gap and the combined approach’s improvement.Figure 7 compares the AC+Debugger–Debugger difference with the Debugger–AC difference across 38 data points.Source: §4.2.3 · p. 12, Figure 7 · p. 12
4.3 RQ2: Segment-wise performance
The section compares six code-generation approaches, from a single coder to chained collaboration and debugging. Across the benchmarks, the strongest approach differs by dataset, while debugging improves on agent-only generation and added agent complexity can sometimes hurt performance.
4.3.1 Segment Details
- The six approaches are Basic, AC, ACT, Debugger, AC + Debug, and ACT + Debug; ACT adds a tester who can interact with the coder for up to three iterations.Basic uses one coder; AC adds an analyst who plans before coding.Source: §4.3.1 · pp. 12–13
- Standalone debugging allows up to 10 cycles, whereas the chained AC + Debug and ACT + Debug approaches cap debugging at four cycles to save time and cost.The chained approaches also cap tester interactions at three.Source: §4.3.1 · p. 13
4.3.2 Segment Performance
- QwQ-Preview (32B) performs worse with AC and ACT than with Basic on both datasets, and ACT also underperforms AC for several other models.The authors suggest analyst-coder interaction may introduce misleading information for QwQ-Preview, while adding a tester can sometimes add complexity or inefficiency.Source: §4.3.2 · pp. 13–14
- ACT + Debug leads on HumanEval by 0.21% over AC + Debug, while AC + Debug leads on HumanEval+ and ACT + Debug performs 1.22% worse.The reported HumanEval+ result aligns with the pattern that the most complex approach is not always best under more rigorous evaluation.Source: §4.3.2 · pp. 13–14, Figure 9 · p. 14
- The Debugger raises mean accuracy over ACT by 6.7% on HumanEval and 7.36% on HumanEval+ through execution-based error detection and iterative refinement.It runs generated code against test cases, supplies failure context, and refines the output.Source: §4.3.2 · p. 14
- AC + Debug improves mean accuracy by 0.75% on HumanEval and 0.61% on HumanEval+.The Analyst-Coder output is debugged when it fails visible test cases.Source: §4.3.2 · p. 14
4.4 RQ3: Code Rigorousness
The section assesses code rigor by comparing accuracy on HumanEval with accuracy on HumanEval+, which uses 80 times more tests. It reports trade-offs across approaches and models: higher accuracy can coincide with larger drops under stricter testing.
4.4 RQ3: Code Rigorousness
- HumanEval+ applies 80 times more tests than HumanEval, and the section uses the accuracy differential between them as its measure of code rigor.A smaller differential indicates greater rigor in the paper's analysis.Source: §4.4 · p. 14, §4.4.1 · p. 14, Figure 11 · p. 15
4.4.1 Approach Consideration
- Basic has the smallest accuracy drop (90.83), while AC and ACT have drops of 129.27 and 118.51, respectively.The section describes Basic as the most rigorous despite comparatively lower overall accuracy, and reports a 137.74 drop for ACT + Debugger.Source: Figure 10 · p. 15, §4.4.1 · pp. 15–16
- AC + Debug combines a 0.68% increase in mean accuracy over Debugger Only with an accuracy drop of 110.41, versus 107.84 for Debugger Only.The authors characterize AC + Debug as balancing accuracy and robustness, while ACT + Debug has the highest accuracy drop at 137.74.Source: §4.4.1 · p. 15, Figure 11 · p. 15
4.4.2 Model Consideration
- Claude 3.5 Haiku is reported as an exception to the tendency for high-performing models to show larger accuracy drops, with stable performance under stricter testing.The authors describe it as offering the most favorable balance between accuracy and reliability among the models considered.Source: Figure 12 · p. 16, §4.4.2 · p. 16
4.5 RQ4: Latency Comparism
The latency comparison finds that more complex approaches generally take longer, but their quality gains do not rise proportionally. AC + Debug has the highest average accuracy across both datasets with moderate latency, while the full chain takes longest and performs slightly worse.
4.5 RQ4: Latency Comparism
- AC takes 20.89 minutes for 53.34% accuracy, while ACT takes 48.47 minutes for 54.04%, with no proportional quality gain from the additional time.The paper also reports substantial latency variance across models, especially in more complex configurations.Source: §4.5 · p. 16
- The standalone Debugger averages 61.02% accuracy in 31.11 minutes, a 6.93% average accuracy gain over Basic for 23.43 additional minutes.Basic averages 7.68 minutes and 54.09% accuracy; these are reported averages across both datasets and 19 LLMs.Source: §4.5 · p. 16, Figure 13 · p. 17
- AC + Debugger averages 61.7% accuracy across both datasets in 38.42 minutes, compared with ACT + Debugger at 61.195% in 68.42 minutes.The authors identify AC + Debugger as the optimal balance of efficiency and code quality.Source: §4.5 · p. 16, Table 3 · p. 17
5 Conclusion
The conclusion finds that debugging generally outperforms agentic workflows, while the benefit of combining them depends on their relative performance. Across the tested models, a simple Analyst-Coder workflow with debugging offers a modest accuracy gain while preserving rigor and similar latency.
5 Conclusion
- When debugging and agentic techniques perform similarly for a model, combining them yields better results; when debugging is substantially stronger, adding agentic steps diminishes benefits.The authors attribute the difference to hit-or-miss agentic conversations and the context-rich information provided by debugging.Source: §5 · p. 17
- Analyst-Coder plus debugging improves mean accuracy by 0.68% over debugging alone while preserving code rigor and similar latency across 19 LLMs.More complex agentic configurations combined with debugging often add latency without substantial gains in functional accuracy or code rigor.Source: §5 · p. 17
- The paper identifies balancing functional accuracy, code rigor, and generation latency as important for downstream applications such as code completion.It frames its findings as guidance for organizations selecting approaches for reliable, high-performance LLM-based systems.Source: §5 · pp. 17–18