PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-collaboration Code Generation via ChatGPT

Yihong Dong, Xue Jiang, Zhi Jin, Ge Li

arXiv:2304.07590v3cs.SE

TL;DR

LLMs remain challenged by complex code-generation requirements, despite strong general code-generation ability. This paper proposes self-collaboration through role instructions and a software-development workflow, using analyst, coder, and tester agents. The approach improves performance over a single LLM agent and is shown on complex repository-level tasks.

  • Problem

    LLMs struggle with complex code-generation requirements, motivating methods that support collaborative decomposition, interaction, and quality control.

  • Method

    The framework uses role instructions to divide labor and coordinate LLM agents organized into analysis, coding, and testing stages.

  • Results

    The ChatGPT-based virtual team achieves significant improvements over a single LLM agent on multiple code-generation benchmarks and shows effectiveness on complex repository-level tasks.

  • Takeaways & Limitations

    Self-collaboration provides an approach to automatic code generation that may improve generated-code quality and reduce human intervention.

  • Takeaways & Limitations

    The evaluation benchmarks do not fully reflect actual software-development scenarios because suitable benchmarks and metrics remain lacking.

Abstract

from arXiv · show

Although Large Language Models (LLMs) have demonstrated remarkable code-generation ability, they still struggle with complex tasks. In real-world software development, humans usually tackle complex tasks through collaborative teamwork, a strategy that significantly controls development complexity and enhances software quality. Inspired by this, we present a self-collaboration framework for code generation employing LLMs, exemplified by ChatGPT. Specifically, through role instructions, 1) Multiple LLM agents act as distinct `experts', each responsible for a specific subtask within a complex task; 2) Specify the way to collaborate and interact, so that different roles form a virtual team to facilitate each other's work, ultimately the virtual team addresses code generation tasks collaboratively without the need for human intervention. To effectively organize and manage this virtual team, we incorporate software-development methodology into the framework. Thus, we assemble an elementary team consisting of three LLM roles (i.e., analyst, coder, and tester) responsible for software development's analysis, coding, and testing stages. We conduct comprehensive experiments on various code-generation benchmarks. Experimental results indicate that self-collaboration code generation relatively improves 29.9%-47.1% Pass@1 compared to the base LLM agent. Moreover, we showcase that self-collaboration could potentially enable LLMs to efficiently handle complex repository-level tasks that are not readily solved by the single LLM agent.

1 INTRODUCTION

Complex code-generation requirements remain difficult for LLMs, motivating a self-collaboration framework that assigns specialized roles and coordinates their interaction. The framework instantiates an analyst, coder, and tester team and reports improvements over a single LLM agent, including on complex repository-level tasks.

  • Motivation: Collaborative teamwork decomposes complex tasks into smaller subtasks and supports error detection and quality control.The motivation draws on teamwork theory and software-engineering practice.
  • Motivation: Training separate models for collaborative teamwork is costly, while limited collaborative-code-generation data makes that approach difficult.The paper identifies ChatGPT and related LLMs as a possible basis for division of labor without this training approach.
  • Approach: Role instructions divide labor by assigning specific responsibilities and coordinate interaction so LLM roles form a virtual team.Roles are prompted to act as domain experts and facilitate one another’s work.
  • Approach: The framework incorporates software-development methodology and instantiates three roles—analyst, coder, and tester—for analysis, coding, and testing stages.The elementary team follows software-development stages to organize the collaborative process.
  • Results: The ChatGPT-based virtual team achieves significant improvements over a single LLM agent on multiple code-generation benchmarks.The paper also evaluates self-collaboration on more complex code-generation scenarios.
  • Results: Self-collaboration shows notable effectiveness on complex tasks such as repository-level code generation that challenge a single LLM agent.This result is presented as evidence from real-world scenarios.

2 SELF-COLLABORATION FRAMEWORK

The self-collaboration framework separates complex-task solving into stages and roles, then coordinates their natural-language outputs through shared state. Role instructions define responsibilities and interactions while prior-stage outputs inform subsequent stages.

  • Framework: The framework has division-of-labor and collaboration components that map a requirement x to an output y.Division of labor decomposes the task into stages and assigns roles to process them.
  • Division of Labor: Role instructions assign each LLM a correlated role and detailed responsibilities to guide behavior throughout the task.The paper motivates role instructions by LLM sensitivity to context and reports better specialized-task performance with role-playing.
  • Collaboration: Each stage produces an output that later roles use, allowing roles to refine prior work through collaboration.The collaboration process uses prerequisite-stage outputs and updates the final output as stages progress.
  • Execution: Roles within one stage may operate in parallel, and independent stages can also be parallelized when they do not depend on one another.The execution order is determined by stage dependencies.
  • Collaboration: A shared blackboard stores exchanged information, and the framework iteratively samples stage outputs until an end condition returns the final output.The pseudocode initializes stages, roles, and the blackboard before repeatedly obtaining prior outputs and updating y.

3 INSTANCE

The paper instantiates self-collaboration with a simplified waterfall workflow for analysis, coding, and testing. An analyst plans, a coder writes and repairs code, and a tester produces quality-oriented feedback without external execution.

  • Workflow: The instance adopts a simplified waterfall model with sequential analysis, coding, and testing stages that can return to the previous stage when issues are found.The workflow uses feedback to refine earlier work.
  • Analyst: The analyst decomposes requirements into solvable subtasks and develops a high-level plan for the coder.The analyst focuses on abstraction and task decomposition rather than implementation details.
  • Coder: The coder writes code according to the analyst’s plan and repairs or refines it using tester feedback.The coder is the central implementation role but works under analyst and tester assistance.
  • Tester: The tester inspects functionality, readability, and maintainability and generates reports to help improve code quality.Testing is simulated by the model instead of using an external compiler and test cases.
  • Role Instructions: Role instructions include the team description, role responsibilities, and user requirements to initialize the ChatGPT agents.Interactions occur between successive-stage roles, with a maximum interaction limit n.

4 EVALUATION

The evaluation tests self-collaboration against prompting and model baselines across code-generation benchmarks, role configurations, LLMs, interaction settings, error categories, cost, and case studies. Results generally show substantial gains over single-agent generation, while revealing boundaries related to model expertise, requirement ambiguity, exception handling, and computational cost.

  • RQ1: Self-collaboration vs. Baselines: Self-collaboration achieves state-of-the-art performance across four code-generation benchmarks, improving over ChatGPT by 29.9%–34.6% relative.Gains are larger on HumanEval-ET and MBPP-ET, which use extended test cases.
  • RQ1: Self-collaboration vs. Baselines: Self-collaboration substantially outperforms zero-shot CoT, Iter-improving, self-planning, and self-debugging prompting baselines.The comparison indicates that the gains are not limited to repeatedly improving a single model output.
  • RQ1: Self-collaboration vs. Baselines: On APPS, self-collaboration enhances ChatGPT substantially and exceeds the previous SOTA approach CodeX (175B) + CodeT.The evaluation uses 1000 interview- and competition-level tasks from this more algorithmically intricate benchmark.
  • RQ1: Self-collaboration vs. Baselines: On CoderEval, self-collaboration substantially outperforms single ChatGPT, achieving a 47.1% relative improvement on Pass@1.CoderEval is presented as a more challenging and realistic benchmark.
  • RQ2: The Effect of Roles in Self-collaboration: The analyst–coder–tester team performs best among tested role configurations, with relative improvements of 40.8% on HumanEval and 47.1% on HumanEval-ET.Adding a compiler is comparable to the three-role team, while pair programming remains slightly worse.
  • RQ3: Self-collaboration on Different LLMs: Self-collaboration improves coding across different LLMs, with domain-specific expertise affecting the magnitude of improvement.The experiments include seven open-source LLMs, GPT-4, CodeLlama, Llama2, and instruction-tuned variants; role-playing also substantially outperforms baselines without role-playing.
  • RQ5: Error Analysis: Error analysis finds that self-collaboration corrects 18.4% of tasks that the coder alone fails, while remaining errors mainly involve coder limitations, ambiguous requirements, or exception handling.The additional analyst and tester introduce requirement deviations in less than 1% of cases.
  • RQ6: Cost and Case Studies: Self-collaboration delivers significant improvement with moderate token usage, but its cost-effectiveness is constrained by the high labor expenses of software development teams.The study also includes function-level and complex real-world case studies, including project, game, and website development scenarios.

5 RELATED WORK

Prior work explores multi-agent LLM collaboration, prompting, and software-development applications, while this paper combines role-playing, inter-role feedback, and software-development methodology for code generation.

  • Multi-agent Collaboration: Multi-agent collaboration coordinates multiple AI systems toward a shared objective, with prior LLM work exploring model orchestration and interactions between LLMs.
  • Self-Collaboration Code Generation: The paper introduces software-development methodology into LLM collaboration and evaluates self-collaboration through quantitative and qualitative analysis.
  • Prompting: Prompt engineering guides LLM behavior through designed inputs, including chain-of-thought and related methods for complex reasoning tasks.
  • Self-Collaboration Code Generation: This work uses role-playing instructions to differentiate one LLM into distinct roles that provide diverse and collaborative contributions.
  • LLMs in Software Development: Existing LLM applications often address individual software-development stages rather than coordinating stages synergistically for broader automation.
  • Self-Collaboration Code Generation: Self-collaboration assigns roles to software-development stages, enabling experts to provide feedback to one another while covering different process stages.

6 DISCUSSION AND FUTURE WORK

The discussion identifies gaps between benchmark evaluation and real software development, the possible value of human oversight, and the need to adapt team structures across problems.

  • Current benchmarks and metrics do not fully reflect actual software-development tasks, so the authors include complex real-world case studies.
  • Although the virtual team is autonomous and roles can monitor and correct one another, limited human guidance could improve real-world utility.
  • The three-role team targets code generation, but other problems may require correspondingly adjusted team structures.

7 CONCLUSION

The paper proposes an interactive self-collaboration framework in which three LLM agents collaboratively address code-generation tasks. It reports effectiveness and generalizability, while identifying potential for improved code quality and reduced human intervention.

  • The framework uses three distinct LLM agents to address code-generation tasks collaboratively within software-development processes.
  • Extensive experiments demonstrate the effectiveness and generalizability of the self-collaboration framework.
  • The approach has potential to improve generated-code quality, reduce human intervention, and accelerate development of complex software systems.

A.1 Code Generation

Code generation automatically produces source code from user requirements, but existing approaches are more proficient on minor tasks than on complex requirements and software-system design. Software development therefore uses staged processes and adaptable team structures, with methodology providing an organizing framework.

  • Code Generation: Code generation automatically produces source code from user requirements to support machine programming and software-development automation.
  • Code Generation: Existing code-generation approaches handle minor requirements such as function completion and line-level generation more effectively than complex requirements and system design.
  • Software Development: The software development life cycle divides system creation into stages including analysis, planning, coding, testing, deployment, and maintenance.
  • Software Development: Software-development teams include specialized roles, and their structure varies with project type, complexity, and methodology.
  • Software Development: The waterfall model organizes development linearly with defined phases, deliverables, and review processes.

B DETAILED SETTINGS AND BASELINES

The paper evaluates code generation with either natural-language descriptions plus signatures and use cases, or natural-language descriptions alone. The NL-only setting is explored because it better reflects real-world development and extends beyond function-level inputs.

  • The first setting supplies a natural-language description, function signature, and use cases as the input prompt.
  • The NL-only setting uses only the natural-language description as the input prompt.
  • The signature-and-use-case setting is mainly suited to function-level generation and is difficult to extend to file- or repository-level generation.
  • Some benchmarks, including MBPP, do not provide function signatures and use cases, as in many real-world scenarios.
  • The paper explores NL-only prompting because it is more consistent with real-world development scenarios.

C ABLATION STUDY WITH ROLES IN SELF-COLLABORATION CODE GENERATION.

Role ablations show that self-collaboration generally outperforms a single coding agent, while the most effective role combination depends on the benchmark and input setting. Analysts are especially influential in the signature-and-use-case setting, whereas MBPP under NL-only prompting favors coder-testers.

  • Self-collaboration is far more effective than a single LLM agent focused only on coding in the NL-plus-signature-and-use-cases setting.
  • Analyst-Coder nearly matches the full team in numerous cases, indicating a particularly pronounced influence of analysts in this experimental setup.
  • 36.7% and 39.4% relative improvements were achieved by the coder-tester team on MBPP and MBPP-ET, respectively, under NL-only prompting.
  • Analysts may underperform on MBPP because some tasks contain unconventional inputs and outputs without requirement hints, and the authors suspect possible data contamination.
  • The analyst role is significant when the LLM receives more comprehensive input-output information in the NL-plus-signature-and-use-cases setting.
  • Additional MBPP and HumanEval case studies illustrate self-collaboration on challenges difficult for a single LLM agent.

D EXTENDED CASE STUDY

The extended case study uses GPT-4 as three autonomous role sessions with fixed instructions. It demonstrates a pair-programming-style organization in which driver and observer roles divide work and cooperate.

  • GPT-4 serves as the base model, with three sessions assigning distinct roles to an autonomous virtual team.
  • The case studies use consistent role instructions without customized alterations.
  • The setup is described as being conducted completely autonomously by teams of models.
  • The extended setup configures a pair-programming team consisting of a driver and an observer.
  • The driver writes and explains code, then incorporates suggestions from the observer to fix or improve it.
  • The observer reviews the driver's code, assesses requirement coverage, predicts problems, and instructs corrections.

F THE PROMPT AND INSTRUCTION OF BASELINES WITHOUT ROLE-PLAYING

The baselines remove role-playing through zero-shot instruction or few-shot prompting, while the self-collaboration prompts separate analysis, coding, repairing, and testing responsibilities. The examples show testers identifying concrete implementation errors for subsequent repair.

  • Zero-shot instruction removes role-playing from the self-collaboration role instructions while preserving separate coding and repair instructions.
  • The analysis instruction decomposes requirements into subproblems and produces a high-level plan for the developer.
  • The coding instruction asks for efficient, readable, best-practice Python code that follows the plan without explanation.
  • The repairing instruction directs code fixes based on the test report while avoiding new bugs or performance regressions.
  • The testing instruction checks functionality, reports bugs, and concludes “Code Test Passed” when the code passes testing.
  • Few-shot prompting conveys each stage's task through four sampled examples excluded from evaluation for fairness.
  • Testing identifies that sort_even sorts values at even indices rather than even numbers, requiring correction while preserving odd positions.
  • Testing also detects that prime_fib treats 1 as prime and may consider 0 a prime Fibonacci number because of its initialization.
Loading 2304.07590v3…