PaperlayerView exact revision on arXiv

Source-linked AI summary

Cogito, ergo sum: A Neurobiologically-Inspired Cognition-Memory-Growth System for Code Generation

Yanlong Li, Jindong Li, Qi Wang, Menglin Yang, He Kong, Shengsheng Wang

arXiv:2501.18653v1cs.SEcs.AI

TL;DR

Multi-agent code-generation methods commonly follow a plan-code-debug sequence, while agent communication adds computational costs. Cogito reverses that sequence and uses stage-specific memory; across eight tasks, it reduced token consumption by up to 66.29% and improved performance by an average of 12.2% over MapCoder.

  • Problem

    Existing multi-agent code-generation methods commonly follow a plan-code-debug workflow, while agent communication adds computational and operational expense.

    Source: §1 · pp. 1–2

  • Method

    Cogito reverses the workflow from Debugger to Coder to Planner and pairs this growth-based Super-Role with hippocampus-inspired memory organized by learning stage.

    Source: §1 · p. 2

  • Results

    12.2% average performance improvement over MapCoder and up to 66.29% lower token consumption across eight code-generation tasks using GPT-3.5-turbo and GPT-4.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluations highlight the potential of biologically inspired design principles to advance intelligent systems.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language models based Multi Agent Systems (MAS) have demonstrated promising performance for enhancing the efficiency and accuracy of code generation tasks. However,most existing methods follow a conventional sequence of planning, coding, and debugging,which contradicts the growth-driven nature of human learning process. Additionally,the frequent information interaction between multiple agents inevitably involves high computational costs. In this paper,we propose Cogito,a neurobiologically inspired multi-agent framework to enhance the problem-solving capabilities in code generation tasks with lower cost. Specifically,Cogito adopts a reverse sequence: it first undergoes debugging, then coding,and finally planning. This approach mimics human learning and development,where knowledge is acquired progressively. Accordingly,a hippocampus-like memory module with different functions is designed to work with the pipeline to provide quick retrieval in similar tasks. Through this growth-based learning model,Cogito accumulates knowledge and cognitive skills at each stage,ultimately forming a Super Role an all capable agent to perform the code generation task. Extensive experiments against representative baselines demonstrate the superior performance and efficiency of Cogito. The code is publicly available at https://anonymous.4open.science/r/Cogito-0083.

1 Introduction

Cogito reverses the conventional planning–coding–debugging workflow to mirror progressive human learning, combining evolving agent roles with hippocampus-inspired memory to support code generation with lower costs.

1 Introduction

  • The approach responds to conventional planning–coding–debugging pipelines, rising multi-agent communication costs, and external storage methods that trade excessive space for summarization-related information loss.The introduction contrasts the conventional sequence with human development from learning and practice toward independent planning.Source: §1 · pp. 1–2
  • Cogito progresses from debugging and code correction to coding initial solutions and finally planning, accumulating experience into a Super-Role with lower token consumption.Roles redundantly defined across agents are intended to make information exchange more efficient and reduce communication costs.Source: §1 · p. 2
  • Its hippocampus-inspired memory stores different content by learning stage and connects memory parts to preserve completeness and support adaptive programming workflows.The design draws on the hippocampus’s role in memory storage and retrieval.Source: §1 · p. 2
  • Cogito reduces token consumption by up to 66.29% and improves performance by an average of 12.2% over MapCoder across eight tasks using GPT-3.5-turbo and GPT-4.The paper reports that Cogito also excels in resource consumption.Source: §1 · p. 2

2 Related work

Related work frames LLM agents around planning and memory, describes role-based collaboration for software development, and organizes code generation into planning, coding, and debugging stages.

2.1 LLM Agents

  • LLM agents combine planning and memory as cognitive components with perception and action, and effective memory management matters for performance [Xi et al., 2023] [Wang et al., 2023; Zhang et al., 2024b].Planning decomposes tasks and schedules execution, while memory stores past actions and observations; this setup has motivated code-generation work [Wang et al., 2024a; Liu et al., 2024].Source: §2.1 · p. 2

2.2 Multi-Agent Collaboration for Software Development

  • Role-based collaboration divides complex work across specialized agents; MetaGPT uses five roles, MapCoder four, and Self-Organized a hierarchical parent-child structure [Hong et al., 2023] [Islam et al., 2024] [Ishibashi and Nishimura, 2024].Typical workflows include task refinement, execution, validation, and optimization [Lei et al., 2024b; Lei et al., 2024a].Source: §2.2 · p. 2

2.3 Prompt Engineering

  • Code-generation prompting commonly follows planning, coding, and debugging stages; AgentCoder produces pseudocode, while LLM4CBI stores and selects prompts to guide variations [Huang et al., 2023] [Tu et al., 2023].The three stages are associated with prior work on planning [Talebirad and Nadiri, 2023; Zhang et al., 2024a; Lin et al., 2024], coding [Rasheed et al., 2024a; Zan et al., 2024; Tao et al., 2024], and debugging [Li et al., 2023; Qin et al., 2024; Rasheed et al., 2024b].Source: §2.3 · pp. 2–3

3 Cogito

Cogito combines Planner, Coder, and Debugger roles with a memory module and a shared Super-Role that accumulates experience across tasks. Role scoring and collaboration settings guide refinement and reuse of solutions.

3.1 Agent Roles

  • Planner outlines a stepwise strategy, Coder implements it, and Debugger uses traceback feedback to fix code that fails sample-input and expected-output tests.The process builds on Chain of Thought [Wei et al., 2022], and code that passes the tests is finalized [Islam et al., 2024].Source: §3.1 · p. 3

3.2 Super-Role

  • A Super-Role rotates through Debugger, Coder, and Planner groups, carries prior experience across groups, then solves independently with up to five error-correction opportunities.The final answer is produced by the Super-Role using expertise accumulated through the three tasks.Source: §3.2 · p. 3

3.3 The Hippocampus-like Memory Module

  • The hippocampus-like module assigns CA1 to initial responses, optional CA2 to user-specific code preferences, CA3 to code versions and error traces, and CA4 to final correct results.The Dentate Gyrus is associated with forming new memories from large volumes of information; a tokenizer reduces stored design content while retaining essential details.Source: §3.3 · pp. 3–4

3.4 Agent Collaboration Settings

  • Role answers are weighted by initial weights of 0.4, 0.4, and 0.3 multiplied by quality-based importance scores, while two other roles are randomly reintroduced per group.Each group’s Debugger modifies code once, and expert-phase debugging allows five attempts.Source: §3.4 · p. 4

4 EXPERIMENTS

Cogito is evaluated on eight benchmarks using unbiased Pass@k against representative code-generation baselines. Across the experiments, the paper reports strong performance, lower resource use, and effects from its role sequence and configuration.

4.2 Overall Performance

  • Cogito achieves the highest scores across all datasets; GPT-4 yields the best results observed, supporting the effectiveness of reverse agent evolution.The comparison covers simple and contest-level problems.Source: Table 1 · p. 5, §4.2 · p. 5

Performance on Simple Code Generation Tasks

Performance on Complex Code Generation Tasks

  • Against MapCoder, Cogito improves on APPS, xCodeEval, and CodeContests by 59.29%, 30.19%, and 5.0% with GPT-3.5-turbo, and 24.1%, 4.2%, and 4.2% with GPT-4.A unified function-based validation approach reduces performance on some platforms, notably xCodeEval and CodeContests.Source: Performance on Complex Code Generation… · p. 5, Table 1 · p. 5

Performance Under Different Difficulty Levels

Consumption of API and Tokens

  • Average token consumption falls by 32.81% and API calls by 28.46%; on HumanEval, reductions reach 66.29% for tokens and 70% for calls.GPT-4 consumes more resources than GPT-3.5-turbo, possibly because it generates more detailed explanations.Source: Consumption of API and Tokens · p. 6, Table 2 · p. 6

4.3 Ablation Study

  • Removing planning, implementation, or debugging reduces performance by up to 14.02%, 12.19%, or 10.36%, respectively; changing the role order causes a 17.07% drop.The Super-Role’s final answer is described as indispensable, while adding diverse sample I/O yields a modest 0.6% improvement.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Hyper-parameter Analysis

  • More self-debugging attempts improve performance but require more time and tokens, while increasing experience-learning groups from three to six lowers performance and raises token use.The authors conclude that one experience-learning cycle is sufficient and reasonable.Source: §4.4 · p. 7, Table 4 · p. 7, Table 5 · p. 7

New Random Roles vs. Same Roles

  • Introducing two new random members during role transitions helps prevent an incorrect initial coding direction from compromising later debugging and redesign.The paper illustrates this rationale with an example on HumanEval.Source: New Random Roles vs. Same Roles · p. 7, Figure 6 · p. 7

5 Conclusion · Appendix · A Agent Prompt Details

Cogito reverses the planning–coding–debugging workflow and reports state-of-the-art code-generation performance with greater efficiency and lower computational cost. The appendix gives role-specific prompt templates and notes a testing limitation for xCodeEval and CodeContest.

5 Conclusion

  • Cogito reverses the traditional workflow, progressing through Debugger, Coder, and Planner roles before forming a Super-Role for complex code-generation tasks.The paper frames this progression as mimicking human growth and learning.Source: §5 · pp. 7–8
  • Cogito reports state-of-the-art performance with higher efficiency and lower computational cost than existing methods.The authors say future work will optimize the architecture and explore broader software-engineering tasks.Source: §5 · p. 8

A Agent Prompt Details

  • Planning prompts request concise guided solution steps and potential challenges, while explicitly instructing the model not to provide code.This template is shown for HumanEval and MBPP and appears again in the prompt details.Source: A Agent Prompt Details · pp. 11–12
  • Debugging prompts tell the model to fix the supplied code using its traceback and return code without analysis or annotations.The xCodeEval and CodeContest variant also requires a single-string function with a return statement.Source: A Agent Prompt Details · pp. 11, 14
  • Coding prompts ask for a Python solution guided by the plan and examples, with the required function name and code-only output.For xCodeEval and CodeContest, the prompt additionally requires a function taking one string parameter, containing all code, and ending with a return.Source: A Agent Prompt Details · pp. 12–13
  • Super-Role prompts use prior experience to produce code, while refinement prompts require a fundamentally different solution that fixes the error and passes the tests.Refinement prompts also provide examples and suggest exploring different logic or structures; dataset-specific variants retain the single-string function constraint.Source: A Agent Prompt Details · p. 13
  • For xCodeEval and CodeContest, unified single-string input and output tests may misdefine functions, so the authors recommend standard tests without forced content.The paper notes that function-definition discrepancies can make test results differ from reality.Source: A Agent Prompt Details · p. 13

B Supplementary Experimental Details · B.1 Datasets

The datasets use selected test inputs or subsets for evaluation, with dataset-specific handling for HumanEval, MBPP, CodeContest, APPS, and xCodeEval.

B.1 Datasets

  • HumanEval comes from Mapcoder [Islam et al., 2024], whose sample column extracts prompt-provided execution examples for easier execution and result return.Source: §B.1 · p. 16
  • MBPP uses some test-set data as inputs while preserving test-set independence and exclusivity between MBPP and MBPP-ET.Source: §B.1 · p. 16
  • CodeContest evaluation uses only the test section, which contains 165 problems.Source: §B.1 · p. 16
  • APPS uses a subset of problems extracted from the raw data by MapCoder.Source: §B.1 · p. 16
  • xCodeEval uses a subset of problems extracted from the raw data by MapCoder.Source: §B.1 · p. 16
  • For convenience, HumanEval was selected in a form that makes execution examples provided in prompts easier to execute and return results.Source: §B.1 · p. 16
  • In MBPP, selected test-set data serve as inputs, without compromising test-set independence or exclusivity with MBPP-ET.Source: §B.1 · p. 16

B.2 Evaluation Metric · B.3 Baselines

The evaluation metric describes Pass@k as the probability of including at least one relevant item among the top-k recommendations. The baselines compare direct and structured reasoning, retrieval-based, and state-of-the-art methods using two GPT models.

B.2 Evaluation Metric

  • The unbiased Pass@k version aims to correct recommendation biases that can distort performance assessments.The passage particularly notes biases arising from uneven distributions of relevant items or recommendation behaviors.Source: §B.2 · p. 16
  • Pass@k assesses whether at least one relevant item appears within the system’s top-k recommendations, with n denoting all items, c relevant items, and k the recommendation-set size.The metric is described as a probability over the top-k recommendations.Source: §B.2 · p. 16
  • The metric computes the probability of selecting no relevant items from the n − c non-relevant items, then subtracts that probability from 1.The no-relevant-items probability is based on the number of ways to choose k items from the non-relevant items, normalized by the number of ways to choose k from all n items.Source: §B.2 · p. 16
  • The metric’s expectation averages performance across users or test cases to account for different scenarios and variability in preferences and outcomes.This provides a broader measure than evaluating a single instance.Source: §B.2 · p. 16

B.3 Baselines

  • The baselines include Direct Method, Chain-of-Thought, Self-Planning, Analogical Reasoning, and Mapcoder, with comparisons spanning direct, structured, retrieval-based, and state-of-the-art approaches.Direct Method submits the prompt without decomposition; CoT solves step-by-step; Self-Planning separates planning from implementation; Analogical Reasoning uses retrieval; and Mapcoder is identified as state-of-the-art.Source: §B.3 · p. 16
  • All baseline tests use GPT-3.5-turbo (GPT-3.5-turbo-0125) and GPT-4 (GPT-4-0613) from OpenAI.Source: §B.3 · p. 16

B.4 An example answer for 5 methods of HumanEval #92

Figure 7 compares five methods’ solutions to HumanEval problem 92, highlighting errors and showing Cogito’s responses at different stages.

B.4 An example answer for 5 methods of HumanEval #92

  • All methods except Cogito fail HumanEval problem 92’s first test, which expects candidate('TEST') == 'tgst'.The example shows Cogito’s responses at different stages and highlights each method’s errors with explanations.Source: §B.4 · p. 16

B.5 The comparison between GPT-3.5-TURBO and GPT-4 responses.

Examples from HumanEval and APPS show GPT-4 succeeding where GPT-3.5 fails or struggles, particularly on complex problems.

B.5 The comparison between GPT-3.5-TURBO and GPT-4 responses.

  • Across HumanEval and APPS examples, GPT-4 generates correct code or handles problems effectively where GPT-3.5 fails or struggles.The comparison attributes GPT-4’s advantage on more complex and challenging problems to improved reasoning and code generation capabilities.Source: §B.5 · p. 16

B.6 The complete response process of Cogito in APPS #1628

The section presents Cogito’s full response process for APPS Task 1628, tracing role interactions and explaining how errors and modifications shape the final outcome.

B.6 The complete response process of Cogito in APPS #1628

  • Figure 10 presents Cogito’s complete solution process for APPS Task 1628, including responses from the roles involved throughout the task.The section also includes responses from adjacent roles in each group to provide context for the process.Source: §B.6 · p. 16, Figure 10 · p. 19
  • The role interactions show how different stages contribute to the final outcome and provide context for cause and effect.Adjacent-role responses are included within each group.Source: §B.6 · p. 16
  • The section identifies errors, changes, and correct parts in responses to explain why code passes or fails and where improvements or missteps occur.It breaks down the reasons behind each modification.Source: §B.6 · p. 16
Loading 2501.18653v1…