PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, Chi Wang

arXiv:2308.08155v2cs.AIcs.CL

TL;DR

LLM applications increasingly need methods for handling growing task complexity and diverse capabilities. AutoGen addresses this gap with customizable, conversable agents and programmable multi-agent conversations, and reports strong performance, reduced development effort, and broader application flexibility. The authors position the framework as promising but still requiring further study of safety and responsible use.

  • Problem

    As LLM-enabled tasks grow in range and complexity, the paper asks how to facilitate development of capable LLM applications using cooperating agents.

  • Method

    AutoGen combines customizable conversable agents backed by LLMs, human inputs, and tools with conversation programming for flexible multi-agent workflows.

  • Results

    AutoGen supports diverse applications and reports outstanding performance, reduced development code, and decreased manual burden across evaluations and pilot studies.

  • Takeaways & Limitations

    AutoGen provides a general framework for reusing, customizing, extending, and coordinating agents across practical multi-agent systems.

  • Takeaways & Limitations

    The work remains in an early experimental stage, and expanding agent numbers and degrees of freedom may introduce safety challenges requiring further study.

Abstract

from arXiv · show

AutoGen is an open-source framework that allows developers to build LLM applications via multiple agents that can converse with each other to accomplish tasks. AutoGen agents are customizable, conversable, and can operate in various modes that employ combinations of LLMs, human inputs, and tools. Using AutoGen, developers can also flexibly define agent interaction behaviors. Both natural language and computer code can be used to program flexible conversation patterns for different applications. AutoGen serves as a generic infrastructure to build diverse applications of various complexities and LLM capacities. Empirical studies demonstrate the effectiveness of the framework in many example applications, with domains ranging from mathematics, coding, question answering, operations research, online decision-making, entertainment, etc.

1 Introduction

AutoGen proposes a generalized multi-agent conversation framework for building LLM applications. It combines customizable, conversable agents with programmable interaction patterns and reports strong performance across applications.

  • AutoGen addresses how to facilitate development of LLM applications as task complexity and the range of LLM-enabled tasks expand.
  • AutoGen introduces customizable, conversable agents that combine LLMs, human inputs, tools, or combinations of these capabilities.Agents can be configured for roles such as coding, code execution, human feedback, and validation.
  • Conversation programming simplifies complex LLM workflows by expressing agent computations and interaction control as multi-agent conversations.
  • AutoGen provides applications across varied complexities and LLM capabilities, supported by benchmark evaluations and pilot studies.

2 The AutoGen Framework

The AutoGen framework represents agents as configurable conversational entities and programs their computations and control flow. Its unified interfaces, auto-reply mechanisms, and dynamic conversation support enable diverse interaction patterns.

  • Conversable Agents: A conversable agent maintains context from messages and can use LLMs, tools, human input, or other configured capabilities.
  • Conversable Agents: AutoGen supports specialized agents by reusing or extending built-in agents, including LLM-backed assistants and human proxies.
  • Conversation Programming: Figure 2 illustrates built-in agents, a two-agent system with a custom reply function, and the resulting automated chat.
  • Conversation Programming: Conversation programming separates agent computation from control flow to specify what agents do and when those computations occur.
  • Conversation Programming: Unified send, receive, and reply functions, together with automatic replies, support automated agent chat.
  • Conversation Programming: Custom reply functions and function calls enable static or dynamic multi-agent conversation flows.

3 Applications of AutoGen

AutoGen is demonstrated in applications spanning mathematics, retrieval-augmented question answering and code generation, interactive decision-making, coding, group chat, and conversational games. These examples show flexible conversation patterns and reported performance or productivity benefits.

  • Six applications were selected to cover real-world relevance, problem difficulty, and innovative potential.
  • A1: Math Problem Solving: AutoGen supports autonomous math problem solving by reusing two built-in agents and comparing the resulting system with multiple alternative approaches.
  • A2: Retrieval-Augmented Code Generation and Question Answering: Retrieval-augmented Chat uses two extended AutoGen agents for question answering and code generation, including an interactive retrieval feature.
  • A3: Interactive Decision Making: The ALFWorld application uses an assistant for planning and an executor for actions, integrating ReAct prompting in a two-agent system.
  • A4: Multi-Agent Coding: OptiGuide coordinates Commander, Writer, and Safeguard agents to generate, check, execute, and interpret code for optimization questions.
  • A4: Multi-Agent Coding: The OptiGuide workflow shrank from over 430 lines to 100 lines, while multi-agent unsafe-code F-1 improved 8% with GPT-4 and 35% with GPT-3.5-turbo.
  • Conversational Chess: Conversational Chess combines human or LLM players with a board agent that provides information and validates moves.

4 Discussion

The discussion presents AutoGen as a reusable and extensible framework that can improve performance and reduce development burden across complex multi-agent applications. It also identifies unresolved safety, privacy, bias, and accountability concerns as the framework expands.

  • AutoGen provides unified agent conversations and auto-reply mechanisms while supporting reuse, customization, extension, and conversation programming.
  • Experiments report improved performance, reduced development code, lower manual burden, dynamic multi-agent chats, and simpler code management.
  • Increasing agent numbers and other degrees of freedom may introduce safety challenges requiring additional study and careful consideration.
  • The discussion calls for continued work on integration, automation–human-control balance, conversation strategies, efficiency, privacy, bias, fairness, accountability, and transparency.

Ethics statement

The paper identifies privacy, bias, fairness, accountability, transparency, trust, and reliance as ethical considerations in developing and using AutoGen.

  • AutoGen’s human participation in agent conversations requires measures to protect user data and conversation privacy.
  • Because LLMs can reproduce training-data biases, developers should address potential bias and promote fairness and inclusivity.
  • Multi-agent cooperation creates a need for clear accountability and transparency mechanisms that let users trace agent decision-making.
  • AutoGen’s combination of human intelligence and conversational automation raises considerations about trust and reliance.

A Related Work

The related-work discussion positions AutoGen as a generic multi-agent infrastructure that differs from single-agent tools and specialized or less flexible multi-agent systems.

  • AutoGen supports multi-agent LLM applications, unlike the single-agent systems discussed in the comparison.
  • AutoGen supports both static and dynamic conversation patterns, tool usage, and human involvement, extending beyond BabyAGI’s static pattern.
  • AutoGen differs from CAMEL by natively supporting tool usage such as code execution and more flexible conversation patterns.
  • MetaGPT is specialized for automatic software development, whereas AutoGen is generic infrastructure for applications across scenarios.
  • Table 1 compares systems by infrastructure, conversation pattern, code execution, and human involvement, while related-system descriptions may change as projects evolve.

B Expanded Discussion

The applications show that AutoGen can support new and renovated applications with dynamic, multi-agent, and human-participatory workflows while keeping implementations relatively simple.

  • AutoGen enables dynamic multi-agent conversations rather than only fixed back-and-forth exchanges.
  • Humans can participate conversationally alongside multiple AI agents in applications A5 and A6.
  • Applications A1–A4 illustrate how AutoGen can quickly renovate existing applications, while A5 and A6 demonstrate new application opportunities.
  • Despite involving more than two agents or dynamic multi-turn cooperation, the AutoGen implementations remain simple.
  • Built-in agents provide ease of use, task division promotes modularity, and agent extension supports programmability.
  • The paper recommends starting with built-in agents and notes that unidirectional pipelines may suit tasks without back-and-forth troubleshooting or multi-agent interaction.

B.2 Future Work

The future-work discussion highlights unresolved questions about designing capable and cost-effective workflows, scaling safely, and maintaining human oversight of autonomous conversations.

  • The optimal number, roles, capabilities, interaction pattern, and automation level may depend on the specific application.
  • A central open question is which multi-agent workflow is most cost-effective for a given task.
  • Effective multi-agent workflows require highly capable agents that combine LLMs, tools, and human strengths.
  • As workflows scale and become more complex, tracking and debugging mechanisms will be needed to avoid incomprehensible agent chatter.
  • Fully autonomous conversations can be useful but require care, especially in high-risk applications, with fail-safes and effective human oversight.

C Default System Message for Assistant Agent

AutoGen’s built-in assistant uses a natural-language system message to program complex conversations through role, control-flow, output, automation, and grounding instructions.

  • The default assistant system message instructs the agent to solve tasks using coding and language skills.
  • It directs the assistant to suggest Python or shell code for users to execute in specified cases.
  • When information must be collected, the assistant uses executable code to browse, search, read, or print information before solving the task itself.
  • Figure 5 organizes the system message’s instructions into role play, control flow, output confinement, automation, and grounding.
  • Combining these prompting techniques programs a fairly complex conversation even with a simple two-agent topology, while requiring mechanisms for instruction exceptions and faults.

A1: Math Problem Solving

AutoGen applies reusable agents and human or expert interaction to mathematical problem solving, with evaluations covering qualitative trials and large MATH-dataset benchmarks.

  • Scenario 1: Autonomous Problem Solving: AutoGen’s autonomous math-solving evaluation uses GPT-4 with SymPy and compares against AutoGPT, ChatGPT+Plugin, and ChatGPT+Code Interpreter.
  • Scenario 1: Autonomous Problem Solving: 69.48% overall accuracy was achieved by AutoGen on the 5,000-problem MATH test set, compared with 55.18% for GPT-4.
  • Scenario 1: Autonomous Problem Solving: Quantitative evaluations reported AutoGen as having the highest problem-solving success rate among the compared methods.
  • Scenario 3: Multi-User Problem Solving: A multi-user workflow allows a student’s assistant to call an expert and the expert’s assistant when the initial solution is unsatisfactory.
  • Scenario 2: Human-in-the-loop Problem Solving: Human-in-the-loop workflows let users provide hints when systems cannot solve challenging problems autonomously.The described process supplies progressively specific hints, including distance equations, case handling, and point-based selection.
  • Scenario 1: Autonomous Problem Solving: AutoGen consistently solved the challenging plane-equation problem across all three trials, whereas competing systems solved it less consistently or failed.ChatGPT+Code Interpreter and ChatGPT+Plugin solved it in two of three trials, while AutoGPT failed in all three.

A2: Retrieval-Augmented Code Generation and Question Answering

Retrieval-Augmented Chat combines document retrieval with conversational generation for question answering and code generation. Interactive context updates help answer questions, while retrieved references enable code using APIs absent from an LLM’s training data.

  • Workflow: The Retrieval-Augmented User Proxy processes documents into chunks and embeddings, retrieves relevant context, and sends it with user questions to the Assistant.The Assistant generates code or text and can request updated context when its response is unsatisfactory.
  • Applications: Retrieval-Augmented Chat supports both question answering over document collections and code generation from current or private codebases.The latter addresses APIs that are unavailable or outdated in the LLM’s training data.
  • Question answering: Interactive retrieval enables the User Proxy to update context and resend it when the Assistant cannot initially find the required information.In the described example, this process leads the Assistant to produce the correct answer.
  • Evaluation: 23.40% F1 and 62.60% Recall were obtained for the first 500 Natural Questions questions without interactive retrieval.The authors report that this result aligns closely with the corresponding comparison and attribute the difference from DPR to the retrievers used.
  • Evaluation: Approximately 19.4% of Natural Questions questions triggered an “Update Context” operation, producing additional LLM calls.These operations reflect the interactive retrieval process used when more context is needed.
  • Code generation: With retrieved references, GPT-4 generated correct code using Spark-related APIs that were absent from its training data.Without those references, it produced a nonexistent spark parameter instead.

A3: Decision Making in Text World Environments

AutoGen is applied to ALFWorld through conversational agents that plan, execute, provide feedback, and supply commonsense grounding. Adding a grounding agent substantially improves performance across task types by addressing failures in physical-world reasoning.

  • A3: Decision Making in Text World Environments: The ALFWorld system uses an Assistant to propose plans and an Executor to perform actions and return environment feedback.A three-agent variant adds a grounding agent that supplies commonsense facts to the Executor when needed.
  • A3: Decision Making in Text World Environments: ALFWorld requires agents to explore textual household environments and complete high-level tasks through a textual interface.Tasks can involve varied locations and multiple interaction steps.
  • A3: Decision Making in Text World Environments: The Assistant often neglects basic household knowledge, creating a commonsense-reasoning challenge for the two-agent system.The grounding agent is introduced to provide that missing knowledge during task solving.
  • Results: Adding a grounding agent significantly improves ALFChat task success rates across the evaluated ALFWorld tasks.Table 3 reports success rates as the number of tasks completed divided by the total, based on three attempts per task.
  • Results: The two-agent design matches ReAct, whereas the three-agent design significantly outperforms ReAct.The authors attribute the discrepancy partly to differences between dialogue-completion and text-completion tasks, while linking the broader improvement to grounding.
  • Case study: Without grounding, the Assistant can conflate finding an object with taking it, contributing to failures on pick and look tasks.The grounding agent is described as eliminating this root cause in the illustrated failure cases.
  • Takeaways: Providing commonsense facts helps the decision-making agent make more informed decisions and boosts task success while preserving a simple, modular extension.The grounding agent serves as an external commonsense knowledge source.

A4: Multi-Agent Coding

AutoGen structures coding and group-chat workflows around specialized agents that coordinate code writing, safety checks, execution, critique, and feedback. These designs reduce implementation effort and support collaborative problem solving across coding applications.

  • A4: Multi-Agent Coding: OptiGuide’s Commander coordinates a Writer, Safeguard, and execution process for answering user questions with generated code.The Writer combines coding and interpretation, while the Safeguard screens code before execution.
  • A4: Multi-Agent Coding: Exceptions and execution failures are routed back to the Writer with logs, allowing the workflow to repeat until resolution or timeout.This provides iterative handling of security flags and code failures.
  • A4: Multi-Agent Coding: Over 430 lines of OptiGuide workflow code were reduced to 100 lines using AutoGen.The consolidation merges coder and interpreter roles into a single Writer agent and produces a more maintainable implementation.
  • A4: Multi-Agent Coding: ChatGPT plus Code Interpreter required manual code handling and could not execute private or customized dependencies such as Gurobi.These requirements disrupted the workflow and increased the chance of mistakes.
  • A4: Multi-Agent Coding: Both ChatGPT plus Code Interpreter and AutoGen-based OptiGuide answered 8 of 10 user-study questions correctly.The study used randomly selected coffee supply-chain questions and measured time and accuracy with an expert Python programmer.
  • A5: Dynamic Group Chat: The Dynamic Group Chat Manager selects one speaker, requests a response, and broadcasts that message to the other agents.The pilot study evaluated a four-agent system against a two-agent system and a task-based speaker-selection alternative across 12 complex tasks.
  • Applications: Conversational Chess uses agents for players and a board agent that supplies board information and checks move legality.AutoGen’s composition and agent design simplify implementation of these interactions.

A7: Online Decision Making for Browser interactions

AutoGen supports online browser decision-making by separating action planning from execution and feedback in a two-agent conversational system. On MiniWoB++, this design achieves competitive performance with a general framework while remaining modular and programmable.

  • Workflow: MiniWobChat uses an assistant agent for action decisions and an executor agent that interacts with the browser and returns feedback.The executor executes suggested actions, reports rewards or feedback, and the assistant continues until task success or failure.
  • Task setting: MiniWobChat addresses browser tasks requiring sequential mouse and keyboard actions, with decisions conditioned on online HTML status.The benchmark tasks require checking updated web state between actions rather than issuing a single manipulation.
  • Evaluation: 52.8% success rate across 49 MiniWoB++ tasks was only 3.6% lower than RCI, a method designed specifically for the benchmark.The comparison used all available tasks in RCI’s official code and default RCI settings.
  • Comparison: AutoGPT struggled with complex MiniWoB++ rules because its interface was less extensible and difficult to instruct for benchmark-specific use.The paper contrasts this with the clearer extension path offered by the two-agent chat system.
  • Takeaways: AutoGen’s modularity and programmability streamline assistant–executor conversations and support reusable agent-environment interaction components.The built-in AssistantAgent can be reused directly, while execution and assistant roles can be modified independently.
Loading 2308.08155v2…