CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation

1 Introduction

• Existing multi-agent frameworks sequence task analysis, code generation, and testing or repair, with CodeCoT and MapCoder presented as representative examples.CodeCoT uses task-understanding, test-generation, and code-generation agents; MapCoder generates step-by-step descriptions before code and repair.Source: §1 · p. 1
• An early misunderstanding can propagate: code may remove duplicates yet violate first-occurrence order while generated tests fail to check that constraint.The example motivates pruning low-quality outputs during generation rather than only at the end.Source: §1 · pp. 1–2
• CodeCoR uses four agents and prunes multiple candidate prompts, test cases, code snippets, and repair advice before downstream use.Failed code is tested locally, sent for repair, and regenerated; the highest-ranked code is returned.Source: §1 · p. 2
• 77.13% average Pass@1, versus MapCoder’s 72.8%, is reported for CodeCoR across HumanEval, HumanEval-ET, MBPP, and MBPP-ET.The paper also reports outperforming representative prompting and multi-agent baselines.Source: §1 · p. 2

2.1 Automatic Code Generation

• Prior code-generation work filters outputs [7], ranks samples or generates tests [23] [3], and uses iterative revision, feedback, or structured prompt planning [52] [48] [21] [31] [19].The section notes that Self-Planning can struggle to balance syntax and semantics on complex tasks.Source: §2.1 · p. 2

2.2 Multi-Agent Coding Models

• Multi-agent coding work includes role-based collaboration [10], tool interaction [49] [53], and frameworks such as MetaGPT [15], ChatDev [36], CodeCoT [16], and MapCoder [18].The section identifies dependence on individual agents in sequential workflows as a robustness challenge.Source: §2.2 · p. 2
• CodeCoR addresses that challenge by checking and pruning intermediate outputs before they are propagated to downstream agents.Source: §2.2 · p. 3

3.1 Overview

• The overview assigns prompt generation, test-case generation, code generation, result checking, and repair to four agents, with candidate pools pruned along the workflow.Selected prompts guide both test and code generation; failed code can enter repair rounds, and the highest-ranked code is returned.Source: §3.1 · p. 3

3.2 Pruning Methods

• Prompt and repair candidates are retained only when they meet clarity, relevance, conciseness, and context criteria; prompts require all four binary scores to be positive.If repair advice is pruned, failed test cases and the code are passed to the Coding Agent instead.Source: §3.2 · pp. 3–4
• Test pruning removes empty, incomplete, or invalid cases, while code pruning removes snippets that cannot compile because of syntax errors.Source: §3.2 · pp. 3–4
• CodeCoR uses stage-specific pruning to check intermediate outputs and reduce low-quality output propagation across workflow stages.Source: §3.2 · pp. 3–4

3.3 Overall Algorithm

• The overall algorithm generates and prunes prompts, tests, and code; if pruning removes every candidate at a stage, it retains the highest-scored one.Candidates are execution-tested; failed snippets receive pruned repair suggestions and are retested until the failed set empties or the repair-round limit is reached.Source: §3.3 · p. 4
• After the repair loop, remaining failed snippets enter the ranked set, and CodeCoR selects the highest-ranked code snippet as its output.Source: §3.3 · p. 5

4.1 Research Questions

• The evaluation asks about function-level effectiveness and the effects of CodeCoR's major components.The listed research questions are RQ1 and RQ2.Source: §4.1 · p. 5

4.2 Datasets

• HumanEval and HumanEval-ET cover diverse programming challenges; MBPP and MBPP-ET contain around 1,000 crowd-sourced Python problems focused on syntax and coding scenarios.HumanEval is described as a hand-written evaluation dataset with 164 Python programming problems.Source: §4.2 · pp. 5–6

4.3 Baselines

• The baseline set spans code-generation models and prompting or multi-agent methods, with prior-study results used when benchmark settings match.The listed comparisons include CodeCoT and MapCoder, among other methods.Source: §4.3 · p. 6

4.4 Evaluation Metrics

• Pass@1 measures first-attempt functional correctness, while CodeBLEU adds syntactic and semantic similarity as a complementary metric.CodeBLEU incorporates AST-based syntax and data-flow features; RAL-Bench uses Fun. for system-test pass rates.Source: §4.4 · p. 6

4.5 Implementation Details

• Function-level experiments use GPT-3.5-turbo with temperature 0.5; generated tests rank and repair candidates, while official tests determine reported Pass@1.The implementation sets K_p=3, K_t=5, K_c=3, K_r=3, and R_max=3.Source: §4.5 · p. 6

4.6 How effective is CodeCoR for function-level code generation? (RQ1)

• 86.6% Pass@1 on HumanEval and 79.2% on MBPP put CodeCoR ahead of MapCoder and CodeCoT on both datasets.On HumanEval, MapCoder scores 80.5% and CodeCoT 79.3%; on MBPP, they score 78.9% and 67.7%, respectively.Source: §4.6 · p. 7, Table 1 · p. 8
• CodeCoR's mean CodeBLEU is 52.12 across HumanEval and MBPP, above MapCoder at 49.22 and CodeChain at 47.56.Its dataset averages are 52.15 on HumanEval and 52.08 on MBPP.Source: §4.6 · p. 8, Table 2 · p. 8

4.7 How Effective Are the Major Components of CodeCoR? (RQ2)

• Without the Test Agent, Pass@1 reductions are reported at 45.1% on HumanEval, 43.3% on HumanEval-ET, 52.1% on MBPP, and 40.6% on MBPP-ET.Removing the Repair Agent or Pruning Method also lowers reported Pass@1, and the authors conclude that CodeCoR outperforms its component variants.Source: §4.7 · pp. 8–9, Table 3 · p. 9

5 What are the cost implications of CodeCoR? (RQ3)

• The full HumanEval cost comparison records execution time, CPU usage, disk writes, and network I/O on a dedicated server.The study used psutil to monitor these system-level costs.Source: §5 · p. 9, Table 4 · p. 9
• 91.61 seconds puts CodeCoR's HumanEval runtime ahead of MapCoder, CodeTree, DebateCoder, and CodeChain, but behind Self-Planning and SCoT.The authors characterize it as efficient among multi-agent and structured code-generation frameworks, though not the fastest overall.Source: §5 · p. 9, Table 4 · p. 9
• CodeCoR's disk write volume is 0.83 MB, the lowest among compared methods, while its 0.06% CPU usage is close to other methods.Its network receive is relatively high, attributed to receiving intermediate information during candidate generation, pruning, and repair.Source: §5 · p. 9

6 How does CodeCoR perform on repository-level code generation? (RQ4)

• DevEval tests code completion within existing projects, while RAL-Bench evaluates application-level repository generation using functional system tests.The comparison uses seven representative baselines under the same backbone model and execution protocol.Source: §6 · p. 9
• 8.88% Pass@1 on DevEval and 20.05% Fun. on RAL-Bench are the best reported results, exceeding the strongest baselines by 3.18 and 1.20 percentage points, respectively.The authors describe these results as preliminary evidence of generalization beyond standalone function-level tasks.Source: §6 · p. 9, Table 5 · p. 9

7.1 Can CodeCoR Work with Other LLMs?

• 43.9% Pass@1 on HumanEval with CodeLlama, versus Code-CoT’s 34.1%; with GPT-4, CodeCoR reaches 94.5% on HumanEval and 83.5% on HumanEval-ET.The CodeLlama results also include 37.8% on HumanEval-ET; Table 6 compares methods on HumanEval and HumanEval-ET.Source: §7.1 · p. 10, Table 6 · p. 10

7.2 How Many Tokens Are Generated by LLMs for Different Prompt Techniques?

• CodeCoR achieves strong performance with comparable or lower token usage than multi-agent baselines, while costing slightly more than some single-agent methods and less than other multi-agent frameworks.The authors describe this as a better balance between effectiveness and efficiency.Source: §7.2 · p. 10

7.3 How does the number of repair rounds affect the performance of the agents?

• Three repair rounds yield the best overall performance of the four agents in the HumanEval experiments.The repair process stops when code cannot be repaired to pass more test cases.Source: §7.3 · p. 10, Figure 6 · p. 10

7.4 Hyperparameter Analysis of Candidate Budget

• Pass@1 rises from 87.80% in cfg1 to 89.63% in cfg3, then drops to 80.49% in cfg4, which also has the highest cost.Prompt, code, and repair candidate counts increase together; test candidates are slightly more numerous because they are used for code generation, scoring, and pruning. The authors recommend cfg3 for HumanEval-like tasks.Source: §7.4 · p. 10

7.5 Search Stability and Candidate Diversity Analysis

• Across three HumanEval runs, final pass/fail agreement is 0.9350 despite seed-sensitive intermediate candidates; prompt diversity is around 0.86, test diversity 0.55–0.57, and code diversity 0.47–0.49.The authors interpret the candidate diversity as providing meaningful alternatives for pruning and selection.Source: §7.5 · pp. 10–11, Table 10 · p. 11

• LLM-based generation and pruning may remain stochastic despite ten experimental rounds and averaged results, and minor execution-environment fluctuations may add variability.The study does not claim every retained candidate is better than every pruned candidate; fine-grained retained-versus-pruned analysis is left for future work.Source: §8 · p. 11
• Pass@1 measures whether generated code passes benchmark test suites, while CodeBLEU is only a complementary measure of structural and semantic similarity to reference implementations.Source: §8 · p. 11
• The chosen datasets and settings may limit broader applicability, and automated benchmarks cannot fully replace controlled user studies of productivity, usability, trust, and cognitive load.The authors plan evaluation on wider datasets and in different environments; developer-centered evaluation remains future work.Source: §8 · p. 11

• CodeCoR uses four agents and pruning methods to check intermediate outputs and reduce the propagation of low-quality outputs.Source: §9 · p. 11
• 77.13% average Pass@1 across four public datasets, outperforming representative prompting and multi-agent baselines under the evaluation setting.Source: §9 · p. 11
• The experiments report improved code-generation accuracy while maintaining manageable system-level overhead.Source: §9 · p. 11