PaperlayerView exact revision on arXiv

Source-linked AI summary

Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents

Kexun Zhang, Weiran Yao, Zuxin Liu, Yihao Feng, Zhiwei Liu, Rithesh Murthy, Tian Lan, Lei Li, Renze Lou, Jiacheng Xu, Bo Pang, Yingbo Zhou, Shelby Heinecke, Silvio Savarese, Huan Wang, Caiming Xiong

arXiv:2408.07060v1cs.SEcs.AIcs.CLcs.LG

TL;DR

Real-world GitHub issue resolution is challenging, and SWE agents with similar resolve rates often solve different issues. DEI coordinates specialized agents through rating and re-ranking; on SWE-Bench Lite, it improves a group's 26.6% average resolve rate to 34.3% (↑25%).

  • Problem

    SWE agents with similar overall resolve rates can solve different sets of GitHub issues, while resolving issues from descriptions is a challenging task.

    Source: §1 · p. 1

  • Method

    DEI is a meta-policy module that integrates with existing SWE-agent frameworks and uses multi-stage rating and re-ranking to coordinate specialized agents.

    Source: §1 · p. 2, §5 · p. 11

  • Results

    On SWE-Bench Lite, a group with an average resolve rate of 26.6% can solve 54.3% of issues with an oracle reviewer, while DEI improves the group's resolve rate to 34.3% (↑25%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    DEI offers a horizontal approach to integrating existing diverse agents without requiring refactoring of engineering work.

    Source: §5 · p. 11

    As the candidate group grows, DEIBASE's improvement eventually plateaus, and some increases in candidate count slightly reduce performance.

    Source: §4.2.2 · p. 8

Abstract

from arXiv · show

Large language model (LLM) agents have shown great potential in solving real-world software engineering (SWE) problems. The most advanced open-source SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite. However, these sophisticated agent frameworks exhibit varying strengths, excelling in certain tasks while underperforming in others. To fully harness the diversity of these agents, we propose DEI (Diversity Empowered Intelligence), a framework that leverages their unique expertise. DEI functions as a meta-module atop existing SWE agent frameworks, managing agent collectives for enhanced problem-solving. Experimental results show that a DEI-guided committee of agents is able to surpass the best individual agent's performance by a large margin. For instance, a group of open-source SWE agents, with a maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3% resolve rate with DEI, making a 25% improvement and beating most closed-source solutions. Our best-performing group excels with a 55% resolve rate, securing the highest ranking on SWE-Bench Lite. Our findings contribute to the growing body of research on collaborative AI systems and their potential to solve complex software engineering challenges.

1 INTRODUCTION

Resolving real-world GitHub issues is challenging, and SWE agents with similar resolve rates often solve different issues. DEI coordinates agents with different specialties, improving the group's issue-resolution performance.

1 INTRODUCTION

  • Different SWE agents resolve different sets of GitHub issues despite similar overall resolve rates, likely reflecting their different skill sets.For example, OpenDevin executes issue-specific bug replications, while Moatless Tools and Agentless do not execute code in the repository.Source: §1 · p. 1
  • On SWE-Bench Lite, a group averaging 26.6% resolve rate can solve 54.3% of issues with an oracle reviewer; DEI improves the group to 34.3% (↑25%).The oracle consistently selects the best candidate patch.Source: §1 · p. 2
  • DEI is a meta-policy module that coordinates agents with different specialties through a multi-stage rating and re-ranking pipeline.The framework is designed to integrate with existing SWE-agent frameworks.Source: §1 · p. 2

2 RELATED WORK

Related work covers language-agent architectures, software-engineering agents, and multi-agent or ensemble methods. It describes both agent capabilities and ways of organizing collaboration.

Fundamental Language Agent

  • Language-agent architectures include ReAct, Reflexion, and CodeAct, which differ in tool use, memory for failed trials, and action representation.ReAct generates functional API calls and receives tool outputs; Reflexion appends failed-trial experience to memory; CodeAct uses code generation for a unified action space.Source: §2 · p. 3

Software Engineering Agent

  • SWE-agent approaches include repository exploration and patch generation, advanced code search and fault localization, predefined task graphs, and simplified localization-and-repair workflows.The section also describes OpenDevin's community-contributed agents and SWE-agent's LM-friendly commands and environment feedback.Source: §2 · p. 3

Multi and Ensemble Agents

  • Prior multi-agent frameworks organize collaboration through static workflows, group-chat ensembles, or hierarchical task assignment.Static workflows offer robustness but lose flexibility for unseen states or conditions; hierarchical organization supports top-down task decomposition.Source: §2 · p. 3

3 INTEGRATING EXPERTISE OF SWE AGENTS

This section frames issue-solving agents as diverse systems and models their objectives across repository contexts. DEI uses a context-aware meta-policy to select among agents, while DEIBASE reviews and scores their candidate patches.

3.1 BACKGROUND

  • SWE-Bench instances pair an issue description and pre-resolution repository with hidden tests; SWE agents generate patches using tools to inspect code, edit files, and run tests.The paper uses “SWE agents” for LLM-based systems that generate patches to solve code-base issues.Source: §3.1 · p. 3

3.2 DIVERSITY OF SWE AGENTS

  • Intra-agent diversity describes different runs solving different instances, while inter-agent diversity describes different agents solving different instances.Run-to-run differences may stem from LLM non-determinism and propagate through multi-step workflows; agent differences also reflect tools, workflows, and prompts.Source: §3.2 · p. 4

3.3.1 SWE AGENT PROBLEM FORMULATION

  • The CMDP formulation represents repository states, issue and repository context, available actions, context-dependent rewards and transitions, and an initial-state distribution.An SWE agent follows a policy over states and context, receiving rewards as the environment transitions through a trajectory; its objective is cumulative reward.Source: §3.3.1 · p. 4

3.3.2 OUR FRAMEWORK: DIVERSITY EMPOWERED INTELLIGENCE (DEI)

  • DEI addresses agents’ uneven performance across contexts with a meta-policy that selects among agent policies based on the observed context.The policies may specialize in subsets of the full context space, and DEI seeks to maximize expected cumulative reward across contexts.Source: §3.3.2 · pp. 4–5

3.3.3 DEIBASE: A SIMPLE YET EFFECTIVE IMPLEMENTATION

  • DEIBASE gives a review committee candidate patches and repository context, then scores patches and selects the highest-scoring candidates as more likely to work.Candidates can come from repeated runs of one agent or from multiple agents; the committee examines code before and after the patch and generates explanations to support its review.Source: §3.3.3 · p. 5, Figure 2 · p. 5
  • DEIBASE supplies the issue, relevant code context, and separate before-and-after code, then generates ordered explanations and scores each patch from 1 to 10.The explanations cover the issue, context relevance, modification location, patch effect, and conflicts; the scoring rubric distinguishes serious mistakes from minor violations.Source: §3.3.3 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate agent diversity and DEI on SWE-Bench Lite, finding substantial differences in issue coverage and improved candidate selection across settings. Committee results and ablations examine how agent composition, candidate count, votes, and explanations relate to resolve rates.

4.1.1 BENCHMARK AND AGENTS

  • On the 300-instance SWE-Bench Lite subset, experiments compare repeated runs of three agents with 10 leaderboard agents and evaluate reranking using n@k.Union@k minus Intersect@k is used to assess diversity, while n@k minus Average@k measures DEI's selection gains.Source: §4.1.1 · p. 6, §4.1.2 · p. 7

4.2.1 RESEARCH QUESTION 1: HOW DIVERSE ARE LLM-BASED SWE AGENTS?

  • At k=10, Union@k is more than 2x Average@k in two settings and more than 1.5x in the other two.The Union@k and Average@k gap grows as more candidates are added.Source: Figure 3 · p. 7, §4.2.1 · p. 8
  • Different-agent candidates cover more distinct issues than runs of one agent; near k=10, the 10-agent setting resolves 2× the group's average issues.The reported comparison is between distinct issues resolved and the average number resolved by a single agent in the group.Source: §4.2.1 · p. 8

4.2.2 RESEARCH QUESTION 2: HOW MUCH DOES DEI HELP?

  • DEIBASE's n@k significantly improves over Average@k for most k in every setting, with larger gains for different-agent candidates that plateau as k grows.The paper notes occasional slight performance drops as k increases and says current DEIBASE is not ideal for large groups.Source: §4.2.2 · p. 8
  • 7%: DEIBASE-Open increases resolve rates and beats most closed-source systems; all three DEI committees significantly outperform their group's best agent.DEIBASE-Open consists of four open-source agents.Source: Table 1 · p. 8, §4.2.2 · p. 9

4.3 ABLATION AND ANALYSES

  • With score averaging, more votes generally improve resolve rates; explanations slightly improve resolve rates in all four evaluated settings.In three of four settings DEIBASE performs much better than the average candidate with one vote; in the remaining setting, improvement appears with three votes.Source: §4.3 · p. 10

5 CONCLUSION

DEI integrates existing SWE agent frameworks to manage collaboration among specialized agents. On SWE-Bench Lite, an oracle-selected group averages a 26.6% resolve rate yet can solve 54.3% of issues, while DEI improves the group’s resolve rate to 34.3% (+7%).

5 CONCLUSION

  • An oracle-selected group with an average resolve rate of 26.6% can solve 54.3% of issues; DEI improves the group’s resolve rate to 34.3% (+7%).The paper presents DEI as an initial step toward harnessing agent diversity.Source: §5 · p. 11
  • DEI is a meta-policy module for integrating with existing SWE agent frameworks and managing collaboration among specialized agents.The approach is intended to scale without refactoring engineering work.Source: §5 · p. 11
  • DEI offers a horizontal scaling-out approach toward future AI-driven organizational management.The paper describes DEI as an initial step toward realizing a fully automated organizational AI.Source: §5 · p. 11

A.1 AGENTS EVALUATED

The DEI Committee additions are listed in an order generated by randomly shuffling chronological order with Python’s random shuffle function and seed 42.

A.1 AGENTS EVALUATED

  • The committee additions are ordered by a random shuffle of chronological order using Python’s random shuffle function and seed 42.Source: §A.1 · p. 13
  • Listed additions include IBM Research Agent101, agentless gpt4o, and moatless gpt4o.Source: §A.1 · p. 13
  • Other listed additions include opendevin codeact v1.8 claude35sonnet, autocoderover-v20240620, and amazon-q-developer-agent-20240430-dev.Source: §A.1 · p. 13

Related papers

Loading 2408.07060v1…