PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolving Multi-Agent Collaboration Networks for Software Development

Yue Hu, Yuzhu Cai, Yaxin Du, Xinyu Zhu, Xiangrui Liu, Zijie Yu, Yuchen Hou, Shuo Tang, Siheng Chen

arXiv:2410.16946v1cs.SEcs.AIcs.MA

TL;DR

Human-designed multi-agent workflows have limited adaptability, while existing benchmarks leave complex software requirements difficult to evaluate automatically. The paper proposes EvoMAC, which evolves agent networks through feedback, and rSDE-Bench, which pairs requirements with tests; experiments report improvements on software- and function-level benchmarks and close agreement between automated and human evaluation.

  • Problem

    Current MAC systems rely on human-designed workflows that are difficult to adapt to diverse software tasks, and existing benchmarks inadequately cover comprehensive software requirements.

    Source: §1 · pp. 1–2

  • Method

    EvoMAC updates agents and their connections at test time using textual feedback from target-proxy verification, while rSDE-Bench pairs software requirements with black-box tests.

    Source: §1 · p. 2

  • Results

    EvoMAC outperforms previous SOTAs on Website Basic, Game Basic, and HumanEval, while rSDE-Bench evaluation reaches 99.22% coherence with human evaluation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    rSDE-Bench provides an automatically evaluated, requirement-oriented setting for assessing software-level coding beyond function-level tasks.

    Source: §1 · p. 2, §4.2 · p. 7

    The paradigm assumes target-proxy generation is significantly simpler than the original task and that outputs can be correctly verified in an objective environment.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

LLM-driven multi-agent collaboration (MAC) systems have demonstrated impressive capabilities in automatic software development at the function level. However, their heavy reliance on human design limits their adaptability to the diverse demands of real-world software development. To address this limitation, we introduce EvoMAC, a novel self-evolving paradigm for MAC networks. Inspired by traditional neural network training, EvoMAC obtains text-based environmental feedback by verifying the MAC network's output against a target proxy and leverages a novel textual backpropagation to update the network. To extend coding capabilities beyond function-level tasks to more challenging software-level development, we further propose rSDE-Bench, a requirement-oriented software development benchmark, which features complex and diverse software requirements along with automatic evaluation of requirement correctness. Our experiments show that: i) The automatic requirement-aware evaluation in rSDE-Bench closely aligns with human evaluations, validating its reliability as a software-level coding benchmark. ii) EvoMAC outperforms previous SOTA methods on both the software-level rSDE-Bench and the function-level HumanEval benchmarks, reflecting its superior coding capabilities. The benchmark can be downloaded at https://yuzhu-cai.github.io/rSDE-Bench/.

1 INTRODUCTION

Human-designed multi-agent workflows have advanced coding beyond line-level tasks but remain difficult to adapt to diverse software requirements. The paper introduces EvoMAC, which updates agents and connections through textual feedback, and rSDE-Bench, which evaluates software requirements automatically; reported results show gains across software- and function-level benchmarks.

1 INTRODUCTION

  • Prior MAC workflows advance coding from line-level to function-level tasks but remain human-designed, limiting task-specific performance and adaptation to diverse coding demands.The paper identifies manual task decomposition and prompt creation as obstacles to adapting these workflows to new challenges.Source: §1 · p. 1
  • EvoMAC adapts agents and connections per task at test time, using target-proxy verification, textual feedback, and textual back-propagation to update the coding team.The proposed cycle includes a coding team, a testing team that creates unit tests, and an updating team.Source: §1 · p. 2
  • Existing benchmarks leave comprehensive software requirements underrepresented, making it difficult to assess software-level coding capabilities.The paper notes that benchmarks commonly focus on individual functions or bug fixing rather than broader software requirements.Source: §1 · p. 2
  • rSDE-Bench pairs measurable requirements with black-box tests across 53 tasks, 616 requirements, Website and Game types, and Basic and Advanced difficulty levels.The synchronized requirement-test pairs enable automatic evaluation of requirement correctness.Source: §1 · p. 2
  • EvoMAC exceeds previous SOTAs by 26.48% on Website Basic, 34.78% on Game Basic, and 6.10% on HumanEval; rSDE-Bench achieves 99.22% coherence with human evaluation.The authors also report that performance consistently improves with more evolving times and that gains hold across two driving LLMs.Source: §1 · p. 2

2 RELATED WORKS

Related work covers LLM-driven multi-agent workflows and software-development benchmarks. The paper positions EvoMAC as an alternative to manually designed workflows and rSDE-Bench as addressing gaps in software-level benchmark coverage and evaluation.

LLM-based multi-agent collaboration

  • LLM-driven MAC systems coordinate agents through workflows across mathematics, software development, embodied tasks, and social simulation.The cited work includes Islam et al. (2024) for mathematics, Qian et al. (2023) and Hong et al. (2023) for software development, Mandi et al. (2024b) for embodied tasks, and Ziems et al. (2024), Pang et al. (2024), and Li et al. (2024b) for social simulation.Source: §2 · pp. 2–3

Software development benchmarks

  • Existing software benchmarks cover function completion, bug repair, and software generation, but function-completion and bug-repair tasks remain function-level.The paper notes that software-generation benchmarks often rely on costly human evaluation or indirect similarity measures rather than automatic verification of requirement correctness.Source: §2 · p. 3
  • rSDE-Bench is presented as the first benchmark combining diverse software requirements with automatic evaluation of requirement correctness.The authors propose it to support more realistic software-level coding capabilities.Source: §2 · p. 3

3 EVOMAC: SELF-EVOLVING MULTI-AGENT COLLABORATION NETWORK

EvoMAC treats a multi-agent collaboration network as an adaptable workflow and uses objective environmental feedback to update it at test time. For software development, coding, testing, and updating teams cycle through code generation, execution feedback, and workflow revision.

3.1 A GENERAL SELF-EVOLVING PARADIGM VIA TEXTUAL BACKPROPAGATION

  • EvoMAC models a collaboration network as a directed acyclic graph of agents, where prompts specify subtasks and edges encode workflow dependencies.Each agent receives the task requirement and the preceding agent’s output; the final agent’s generation integrates the subtasks.Source: §3.1 · p. 3
  • At test time, an objective executor compares generated output with a fixed target proxy and returns textual feedback about task-completion failures.The approach assumes proxy generation is simpler than the original task and that an objective environment can verify outputs against it.Source: §3.1 · p. 4
  • Textual backpropagation analyzes each agent’s contribution, then updates prompts and workflow by removing completed agents, correcting errors, and adding agents for missed subtasks.A gradient agent identifies fulfilled or missed subtasks and errors; an updating agent applies the resulting changes while preserving workflow dependencies.Source: §3.1 · pp. 4–5

3.2 SELF-EVOLUTION FOR SOFTWARE DEVELOPMENT

  • For software development, a coding team generates code, a testing team creates unit tests as the target proxy, and a compiler produces bug logs for feedback.The coding team is initialized through an organizer that decomposes requirements into subtasks and assembles a task-dependent number of agents.Source: §3.2 · pp. 5–6
  • The updating team uses execution logs to identify completed, unmet, and erroneous subtasks, then revises coding agents and their workflow.The cycle repeats to refine the coding team toward closer alignment with the test cases.Source: §3.2 · pp. 5–6

4 RSDE-BENCH: REQUIREMENT-ORIENTED SOFTWARE DEVELOPMENT ENGINEERING BENCHMARK

rSDE-Bench evaluates software-level coding against detailed requirements paired with tests, rather than relying only on brief instructions or indirect metrics. It covers website and game tasks at two difficulty levels and provides automatic requirement-level correctness evaluation.

4 RSDE-BENCH: REQUIREMENT-ORIENTED SOFTWARE DEVELOPMENT ENGINEERING BENCHMARK

  • rSDE-Bench evaluates software-level coding against comprehensive requirements paired with unit tests, aligning its task format more closely with real-world development practices.It includes 53 coding tasks and 616 test cases across website and game software.Source: §4 · p. 6, §4.1 · p. 6

4.1 BENCHMARK CONSTRUCTION

  • The benchmark covers website and game tasks, with basic requirements for fundamental functionality and advanced requirements for more complex features.Examples of advanced requirements include game logic and dynamic web content management.Source: §4.1 · pp. 6–7

4.2 AUTOMATIC EVALUATION

  • Requirement-level accuracy is the proportion of paired black-box test cases passed, enabling fully automated correctness evaluation.The authors contrast this with indirect consistency, completeness, and quality metrics, which they say do not verify whether generated code fulfills requirements.Source: §4.2 · p. 7

4.3 FEATURES

  • Requirements average 507 words for game tasks and 1,011 for website tasks, while evaluation uses automated unit-test pass rates.The figure caption reports that automatic accuracy is highly aligned with human evaluation across four dataset settings.Source: §4.3 · p. 7, Figure 4 · p. 8

5 EXPERIMENTS

Experiments assess the benchmark’s automatic evaluation, EvoMAC’s coding performance, and how evolution and system components affect results. They report close alignment with human evaluation and improvements across software- and function-level tasks.

5.1 EXPERIMENTAL SETUP

  • The experiments compare three single-agent models and five multi-agent baselines, with multi-agent systems powered by GPT-4o-Mini; they cover rSDE-Bench and HumanEval’s 164 Python problems.Two EvoMAC variants use GPT-4o-Mini and Claude-3.5-Sonnet.Source: §5.1 · p. 8

5.2 EFFECTIVENESS OF RSDE-BENCH’S EVALUATION AND EVOMAC

  • The automatic evaluation correlates 0.9922 with human evaluation, while consistency and quality correlate 0.2583 and 0.3041, respectively.The comparison spans two software types, four methods, and two requirement difficulties.Source: §5.2 · pp. 8–9
  • EvoMAC outperforms prior SOTA methods on rSDE-Bench and HumanEval, including by 26.48% on Website Basic and 34.78% on Game Basic versus single-agent methods.It also surpasses existing multi-agent methods by over 20%.Source: §5.2 · p. 9

5.3 EFFECTIVENESS OF EVOLVING

  • Across five dataset settings, EvoMAC improves with more iterations under both driving LLMs, and the more powerful model’s curve remains ahead.The settings span two software types, two difficulty levels, and function-completion tasks.Source: §5.3 · p. 9, Figure 6 · p. 10
  • Website page-display errors resolve more quickly than Game logic errors, which persist; the reported pattern is an early gain followed by a plateau.The analysis associates the plateau with more complex unresolved issues.Source: §5.3 · p. 9, Figure 5 · p. 9

5.4 ABLATION STUDY

  • Removing objective environment feedback reduces performance by 12.67% and 14.97% on Website Basic and Advanced, and 21.74% and 18.28% on Game Basic and Advanced.The comparison is between Variant g) and Variant f), which replaces the environment tool with agent critique.Source: §5.4 · pp. 9–10
  • Across task token lengths, EvoMAC outperforms ChatDev and GPT-4o-Mini; reducing either coding or testing teams to one agent lowers Website performance.The reported task-length degradation is smaller for modular Website tasks than for Game tasks requiring more coordination.Source: §5.4 · p. 10, Figure 7 · p. 10

5.5 CASE STUDY

  • In the RecipeHub case study, EvoMAC after one evolution iteration corrects earlier issues and fulfills requirements; multi-agent systems produce more structured code.The comparison includes GPT-4o-Mini, ChatDev, and EvoMAC before and after evolving.Source: Figure 8 · p. 10, §5.5 · p. 11

6 CONCLUSION

The paper proposes EvoMAC, a self-evolving MAC paradigm, and rSDE-Bench, a benchmark for requirement-oriented software development. Experiments report alignment between the benchmark’s automatic evaluation and human judgments, and EvoMAC’s advantage over prior SOTAs on both benchmark levels.

6 CONCLUSION

  • EvoMAC adapts agents and their connections during each task’s test phase using textual back-propagation.The paper presents this as a way to extend coding beyond function-level tasks to software-level development.Source: §6 · p. 11
  • rSDE-Bench combines complex, diverse software requirements with automatic evaluation of requirement correctness.The supplied conclusion describes it as a requirement-oriented software development benchmark.Source: §6 · p. 11
  • The authors plan to add a reward model and extend rSDE-Bench to more software types.Source: §6 · p. 11

7 BENCHMARK DETAILS

The benchmark details describe requirement formulation and testing tailored to Website and Game tasks. Test cases correspond to requirements and are organized by difficulty.

7 BENCHMARK DETAILS

  • Game requirements are manually crafted around GUI layout, interactions, rules, and state logging, while website requirements are expanded from a concise name using an LLM and predefined patterns.Source: §7 · p. 14
  • Black-box unit tests correspond directly to software requirements, with basic and advanced difficulty levels and test construction tailored to each software type.Game tests are manually created to track state changes after GUI interactions; website tests are LLM-generated Selenium tests followed by manual correction.Source: §7 · p. 14
  • Basic cases assess straightforward interactions or required page elements, while advanced cases test complex game logic or website functions such as authentication and dynamic content.Source: §7 · p. 14

8 ALGORITHM

The algorithm section points to EvoMAC’s self-evolving procedure. Its listed inputs and steps connect task generation, environment feedback, and updates to the MAC network.

8 ALGORITHM

  • The procedure takes a task input, an initialized MAC network, a target-proxy network, gradient and update functions, and an environment tool.Source: §8 · p. 16
  • It generates the target proxy from the task, then uses the current network to generate an output for each iteration.Source: §8 · p. 16
  • The environment evaluates generated output against the proxy to produce textual loss; textual back-propagation summarizes a gradient for updating prompts and pipeline.Source: §8 · p. 16

9 PROMPTS

The section presents the prompts used by EvoMAC’s coding, testing, gradient, and update agents.

9 PROMPTS

  • Coding prompts are provided for the coding organizer and coding agent.They appear in Tables 6 and 7.Source: §9 · p. 16
  • Testing prompts are provided for the testing organizer and testing agent.They appear in Tables 8 and 9.Source: §9 · p. 16
  • Gradient-agent and update-agent prompts are also provided.They appear in Tables 10 and 11, respectively.Source: §9 · p. 16

10 SOFTWARE PRESENTATION

EvoMAC generates games and websites with the described interfaces and behavior, and handles varied requirements in both domains.

10 SOFTWARE PRESENTATION

  • EvoMAC handles different GUI and game-rule requirements across games, as well as requirements for different websites.Source: §10 · pp. 16–17
  • EvoMAC outputs games with well-written GUIs and game rules.Figure 12 presents generated games.Source: §10 · p. 16, Figure 12 · p. 24
  • EvoMAC produces websites with beautified, user-friendly pages and correct transition logic.Figure 13 presents generated websites.Source: §10 · p. 17, Figure 13 · p. 25
Loading 2410.16946v1…