PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Design of Agentic Systems

Shengran Hu, Cong Lu, Jeff Clune

arXiv:2408.08435v2cs.AI

TL;DR

The paper addresses the gap between hand-designed agent systems and automatically learned designs. It introduces ADAS and Meta Agent Search, which uses a meta agent to program agents in code and iteratively build on archived discoveries. Across domains, the discovered agents outperform hand-designed baselines and transfer well across models and domains, while current evaluations remain limited to single-step question answering.

  • Problem

    Existing ADAS methods mostly design prompts, limiting their ability to discover flexible agent design patterns beyond predefined workflows.

  • Method

    Meta Agent Search uses a Foundation Model as a meta agent that iteratively programs, evaluates, and archives new agents in code.

  • Results

    The discovered agents substantially outperform state-of-the-art hand-designed baselines and transfer strongly across models and domains, including 13.6/100 higher DROP F1 and 14.4% higher MGSM accuracy.

  • Takeaways & Limitations

    ADAS demonstrates the potential to automate agent-system design rather than relying solely on manual construction.

  • Takeaways & Limitations

    The paper evaluates Meta Agent Search only on single-step question-answering tasks and suggests extending it to more complex domains with multi-step interaction.

Abstract

from arXiv · show

Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

1 INTRODUCTION

The paper proposes Automated Design of Agentic Systems (ADAS) to replace hand-designed agent architectures with automatically discovered systems. Meta Agent Search programs agents in code, and experiments show strong performance and transfer across domains and models.

  • ADAS aims to automatically invent novel agentic building blocks and combine them into powerful agentic systems.
  • Code-based search can theoretically represent prompts, tool use, workflows, and their combinations, unlike approaches restricted mainly to prompt design.
  • Meta Agent Search iteratively programs new agents, evaluates them, archives discoveries, and uses the archive to guide later designs.
  • 13.6/100 F1 points on DROP and 14.4% accuracy on MGSM separate the discovered agents from hand-designed baselines.
  • 25.9% accuracy on GSM8K and 13.2% on GSM-Hard are reported after transfer across domains.

2 AUTOMATED DESIGN OF AGENTIC SYSTEMS (ADAS)

ADAS formulates automated agent design as optimization over a search space using a search algorithm and an evaluation function. The paper emphasizes code as a broad search space that can represent diverse agent components and workflows.

  • Search Space: The search space determines which agentic systems can be represented and discovered.
  • Search Algorithm and Evaluation: The search algorithm specifies how ADAS explores its search space, while the evaluation function measures candidate agents against objectives such as performance.
  • ADAS uses a search algorithm to discover agentic systems across a search space that optimize an evaluation function.
  • Search Space: Prompt-only search spaces cannot represent agents with workflows different from the predefined design.
  • Code Space: Code-space search theoretically enables discovery of arbitrary building blocks, including prompts, tool use, workflows, and systems combining them.

3 OUR ALGORITHM: META AGENT SEARCH

Meta Agent Search uses Foundation Models as meta agents that iteratively program new agents from an archive of prior discoveries. A lightweight framework supplies basic functions, while evaluation and refinement guide continued search.

  • Meta Agent Search adopts Foundation Models as meta agents that program increasingly novel agents using an archive of previous discoveries.
  • The framework provides essential functions such as Foundation Model queries and prompt formatting, leaving the meta agent to program a forward function.
  • The archive may begin with baseline agents, after which the meta agent generates a high-level idea and implements it in code.
  • Generated designs undergo self-reflection refinement, can be repeated up to five times, and are then added to the archive with evaluation metrics.
  • Performance metrics such as success rate or F1 score guide the meta agent toward interestingly new agents.

4 EXPERIMENTS

The experiments evaluate Meta Agent Search across ARC, reading comprehension, math, science, and multi-task benchmarks, including transfer across tasks and foundation models. Across these settings, searched agents outperform hand-designed baselines and develop increasingly sophisticated feedback and reasoning patterns.

  • Experimental scope: Meta Agent Search is evaluated on ARC, DROP, MGSM, MMLU, and GPQA, with additional transfer tests on GSM8K, GSM-Hard, MMLU, and DROP.The same algorithm is used across tasks, with task-specific descriptive text varied for each setting.
  • 4.1 Case Study: ARC Challenge: The ARC comparison uses five hand-designed agents, including Chain-of-Thought, Self-Consistency, Self-Refine, LLM-Debate, and Quality-Diversity.These baselines represent commonly used agent design patterns such as intermediate reasoning, reflection, debate, and answer diversity.
  • 4.1 Case Study: ARC Challenge: On ARC, the search progressively discovers higher-performing agents by extending an archive of prior discoveries.An iteration-3 design used multiple Chain-of-Thoughts, refinement, and ensembling; later designs incorporated diverse and trait-specific feedback.
  • 4.2 Reasoning and Problem-Solving Domains: Meta Agent Search outperforms the hand-designed baselines in every evaluated domain, with improvements of 13.6/100 F1 in reading comprehension and 14.4% accuracy in math.The reported gaps are smaller in the Multi-task and Science domains.
  • 4.3 Generalization and Transferability: Agents transferred from MGSM outperform baselines on held-out math and non-math domains, improving accuracy by 25.9% on GSM8K and 13.2% on GSM-Hard.Transferred math agents outperform hand-designed baselines on MMLU and DROP, although they do not fully match agents designed specifically for those target domains.
  • 4.3 Generalization and Transferability: Agents discovered with GPT-3.5 on ARC consistently outperform hand-designed agents when transferred to Claude-Haiku, GPT-4, and Claude-Sonnet.The best agent reaches nearly 50% accuracy on ARC with Claude-Sonnet.

5 RELATED WORK

ADAS extends automated AI design to agentic systems by learning prompts, tools, workflows, and other components rather than only optimizing manually specified designs. This paper positions code-space search as a way to cover broader agent designs than prior approaches.

  • Agentic Systems: Agentic systems combine Foundation Models with planning, tool use, reflection, and other building blocks to solve tasks through multi-step workflows.These components support application-specific agent designs beyond a monolithic model query.
  • Cross-Model Evaluation: Table 3 reports ARC test accuracy with 95% bootstrap confidence intervals when top agents discovered on GPT-3.5 are transferred to other Foundation Models.The caption states that discovered agents consistently outperform baselines across different models.
  • Automated Design of Agentic Systems: ADAS aims to automatically invent novel building blocks and powerful agentic systems, extending AI-generating and AutoML ideas to agent design.The paper frames ADAS as a newly forming research area related to AI-GAs and AutoML.
  • Existing Attempts to ADAS: Most existing ADAS attempts optimize prompts, while other methods learn selected components but leave some designs fixed or cover only part of the design space.Prior work includes prompt optimization, tool learning, workflow optimization, and joint learning of prompts, tools, and workflows.
  • Code-Space Design: Representing all agent components in code allows the search space to include prompts, tools, workflows, and arbitrary combinations of these components.The authors contrast this with approaches that use narrower or harder-to-search design spaces.

6 DISCUSSION AND CONCLUSION

The discussion presents ADAS as a potentially accessible and safety-relevant direction while emphasizing that current evaluations and objectives remain limited. The paper concludes that code-based Meta Agent Search can automate the discovery of powerful agentic systems and transfer across models and domains.

  • Safety Considerations: Generated code may act destructively despite low perceived likelihood, so the authors use containerized execution, manual inspection, and warnings.These measures are intended to control risks from limitations in model capability or alignment.
  • Broader Implications: ADAS may accelerate agent development and improve controllability and auditability by automating explicit, interpretable workflows.The authors also state that ADAS can be programmed using FM API access without expensive GPU hardware.
  • Broader Implications: The paper argues that ADAS could streamline the creation of safe, reliable agents as self-improving AI systems become more prominent.The stated application scope includes domains such as health and economics.
  • Future Work: The paper proposes higher-order ADAS, online continual learning, and multi-objective search over performance, cost, latency, and robustness as future directions.These directions would extend the meta agent, incorporate post-deployment feedback, and optimize multiple objectives.
  • Future Work: Current evaluation is limited to single-step question-answering tasks, motivating extensions to multi-step interaction with complex real-world environments.The authors identify more complex domains as a future research direction.
  • Conclusion: Meta Agent Search consistently outperforms hand-designed agents across domains, with discovered agents transferring well across models and domains.The conclusion presents this as evidence for automating powerful agentic-system design from the bottom up.

A GENERALIZATION AND TRANSFERABILITY

Meta Agent Search transfers agents discovered in MGSM to other math and non-math domains, where they outperform or match hand-designed baselines. The experiments use domain-specific prompts and iterative self-reflection during agent generation.

  • Transfer across math domains: Top agents transferred from MGSM consistently outperform baselines across GSM8K, GSM-Hard, SVAMP, and ASDiv.Table 4 reports test accuracy with 95% bootstrap confidence intervals.
  • Transfer beyond math: MGSM-discovered agents outperform hand-designed baselines in Reading Comprehension and Multi-task, and match them in Science after transfer beyond math.Their performance does not fully match agents searched specifically for the target domains.
  • Search and refinement: The search process uses domain descriptions, archive-based prompts, complete code generation, self-reflection, and runtime-error refinement to improve candidate agents.Runtime errors trigger reflection and reruns for up to five attempts.

D EXPERIMENT DETAILS FOR ARC CHALLENGE

The ARC experiments represent grid transformations as code-generation tasks and evaluate discovered agents by exact-match accuracy on predicted output grids. The best discovered ARC agent combines multiple candidate solutions, structured feedback, expert review, iterative refinement, and ensemble decision-making.

  • Task setup: The framework asks agents to generate solution code and supplies tools to test code on example grids and obtain task outputs.This makes code execution part of the ARC agent workflow.
  • Task setup: ARC tasks provide input-output grid examples and a test grid whose output must be predicted from the learned transformation rule.Each grid is represented as a rectangular matrix of integers corresponding to colors.
  • Evaluation: ARC accuracy is measured by Exact Match between the reference solution and the predicted answer, with discovered agents evaluated using gpt-3.5-turbo-0125.The meta agent uses gpt-4o-2024-05-13 to reduce search cost.
  • Discovered agent: The best ARC agent generates multiple initial candidates, retains solutions passing at least one example, and adds human-like and expert feedback.Its expert roles include efficiency and readability evaluation.
  • Discovered agent: Candidates undergo up to three refinement iterations before the top three solutions are combined in a final ensemble decision.The final module reasons over candidate thinking, code, and feedback before producing code for the answer.

E EXPERIMENT DETAILS FOR REASONING AND PROBLEM-SOLVING DOMAINS

The experiments evaluate agents on reasoning and problem-solving benchmarks spanning science, multilingual math, multitask knowledge, and reading comprehension. The domains differ in task format, sampling, and evaluation frequency.

  • Evaluation protocol: For GPQA, validation contains 32 questions and test contains 166; other domains use 128 validation and 800 test questions, with five GPQA evaluations and one for others.DROP uses one-shot questions, while the other domains use zeroshot questions.
  • DROP (Reading Comprehension): DROP tests reading comprehension and discrete reasoning over paragraphs, using passage-question-answer tasks.The example asks for two nationalities with equal population counts.
  • GPQA (Science): GPQA measures performance on expert-designed multiple-choice questions in biology, physics, and chemistry.The benchmark is described as challenging and Google-proof.
  • MGSM (Math): MGSM evaluates mathematical problem solving across multiple languages to test broad multilingual performance.The supplied example is a Japanese word problem.
  • MMLU (Multi-task): MMLU assesses questions across STEM, social sciences, humanities, and other subjects with varied difficulty.Its example uses a multiple-choice astronomy question.

F BASELINES

The paper compares Meta Agent Search with hand-designed agents and illustrates discovered designs that use peer review, decomposition, specialization, integration, and verified visual reasoning. Its algorithm iteratively designs, evaluates, refines, and archives agents.

  • Hand-designed baselines: The ARC baselines are Chain-of-Thought, Self-Consistency, Self-Refine, LLM-Debate, and Quality-Diversity.These represent stepwise reasoning, answer ensembling, iterative correction, role-based debate, and diversity search.
  • Hand-designed baselines: Reasoning and problem-solving experiments additionally use Step-back Abstraction and Role Assignment baselines.Self-Refine permits up to five refinement iterations, while LLM-Debate runs for two rounds.
  • Discovered agents: The Multi-Step Peer Review Agent uses multiple modules to produce answers, critique other answers, refine them, and make a final decision.It was discovered in the GPQA Reading Comprehension domain.
  • Discovered agents: The Divide and Conquer Agent decomposes a problem, assigns sub-problems to specialized experts, and integrates their solutions into a final answer.Its experts include physics, chemistry, biology, and general roles.
  • Discovered agents: The Verified Multimodal Agent generates a visual representation, verifies it, and uses the verified visual aid for step-by-step problem solving.This agent was discovered during the MGSM Math search.
  • Meta Agent Search: Meta Agent Search initializes an archive, generates agents in code, performs two self-reflection steps, evaluates them, repairs runtime errors up to five times, and archives discoveries.The archive supports later design iterations.

I IMPACT OF INITIALIZATION

Meta Agent Search remains stronger than hand-crafted baselines without initial designs, although initialization generally improves performance except in math, where starting from scratch performs better. These findings motivate studying how initialization choices affect search across domains.

  • Without initial agent designs, Meta Agent Search outperforms all hand-crafted baselines across every evaluated domain.
  • Initial solutions generally improve performance, except in math, where starting from scratch yields superior results.
  • The math-domain advantage of starting from scratch is hypothesized to reflect broader and more diverse exploration of reasoning strategies.
  • Future work should examine how the choice and quality of initialization affect search effectiveness across domains.

J COST OF EXPERIMENTS

Experiments are costly because evaluation repeatedly queries language models, though cheaper and better-performing models and more sophisticated evaluation functions may reduce ADAS costs.

  • $500 USD is the approximate cost of one ARC search-and-evaluation run, compared with about $300 USD for reasoning and problem-solving domains.
  • Evaluation of discovered agents is the primary expense because it queries the “gpt-3.5-turbo-0125” model.
  • The “gpt-4o-mini” model costs less than one-third as much as “gpt-3.5-turbo-0125” and offers better performance.
  • More sophisticated evaluation functions could significantly reduce ADAS costs because the current naive function is expensive and overlooks valuable information.
Loading 2408.08435v2…