PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Design of Agentic Systems

Shengran Hu, Cong Lu, Jeff Clune

arXiv:2408.08435v2cs.AI

TL;DR

Automated agent-system design remains underexplored beyond prompt-focused approaches. This paper presents Meta Agent Search, which iteratively programs agents in code from an archive of discoveries, and reports consistent outperformance of hand-designed agents across domains with transfer across models and domains.

  • Problem

    Most existing automated agent-design work focuses on prompts, while broader agent-system design remains underexplored.

    Source: §1 · p. 2

  • Method

    Meta Agent Search uses a meta agent to iteratively program novel agents in code, evaluate them, and add discoveries to an archive for later iterations.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    Across domains, Meta Agent Search consistently outperforms state-of-the-art hand-designed agents and transfers well across models and domains; DROP reading-comprehension F1 improves by 13.6/100.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    The results illustrate the potential of automated agent-system design as a research direction.

    Source: §6 · p. 11

    Generated code could still act destructively due to model capability or alignment limitations, despite overt malicious actions being highly unlikely in current settings.

    Source: §6 · p. 10

Abstract

from arXiv · show

Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

1 INTRODUCTION

The paper proposes Automated Design of Agentic Systems (ADAS), in which a meta agent programs and evaluates new agent designs in code. The authors report improvements over hand-designed baselines and strong transfer across domains.

Motivation

  • ADAS addresses costly manual tuning and the challenge of combining agent components, motivated by the broader shift from hand-designed to learned AI systems.The introduction presents automation as a possible way to reduce human effort and reach effective agent designs faster.Source: §1 · pp. 1–2

ADAS

  • The proposed code-space approach uses a meta agent to program entire agent systems, theoretically enabling designs of prompts, tool use, workflows, and other components.The paper argues that Turing-complete programming languages make this broad search space possible and that capable coding models can serve as meta agents.Source: §1 · pp. 2–3

Meta Agent Search

  • Meta Agent Search iteratively creates and evaluates novel agents, archives discoveries, and uses that archive to guide later search across several task benchmarks and held-out domains and models.Evaluations include ARC, reading comprehension, math, science, multitask problem solving, and transfer tests.Source: §1 · p. 3

Results

  • F1 on DROP improved by 13.6/100, MGSM accuracy by 14.4%, and transferred accuracy over baselines by 25.9% on GSM8K and 13.2% on GSM-Hard.The GSM8K and GSM-Hard improvements are reported after transfer across domains.Source: §1 · p. 3
  • Discovered agents showed strong transfer across similar and dissimilar domains, including from mathematics to reading comprehension.The paper presents this transfer as evidence of the agents’ robustness and generality.Source: §1 · p. 3

2 AUTOMATED DESIGN OF AGENTIC SYSTEMS (ADAS)

ADAS frames agentic-system design as an optimization process that searches for systems maximizing an evaluation objective. The paper highlights code-based search as a promising way to discover flexible agent designs.

2 AUTOMATED DESIGN OF AGENTIC SYSTEMS (ADAS)

  • ADAS has three components: a search space defining representable systems, an algorithm exploring that space, and an evaluation function scoring candidates.For example, prompt-only search cannot represent agents with workflows different from a predefined one; the space may also be graphs or feed-forward networks.Source: Figure 2 · p. 3, §2 · p. 4
  • ADAS uses a search algorithm to discover agentic systems within a search space that optimize an evaluation function.The evaluation objective may concern performance, cost, latency, or safety.Source: §2 · pp. 3–4
  • Searching agent designs in code could theoretically enable discovery of any combination of building blocks, including prompts, tool use, and workflows.Readable code may aid interpretation, debugging, and safety; code search can build on open-source frameworks and use foundation models’ coding expertise.Source: §2 · p. 4

3 OUR ALGORITHM: META AGENT SEARCH

Meta Agent Search uses a foundation model as a meta agent to iteratively program new agents from an expanding archive of discoveries. A lightweight framework supplies essential functions, leaving the meta agent to implement each agent’s forward function.

3 OUR ALGORITHM: META AGENT SEARCH

  • A framework of fewer than 100 lines provides essential functions such as FM querying and prompt formatting, so the meta agent only needs to program each system’s forward function.The forward function takes task information as input and returns the agent’s response.Source: §3 · p. 4
  • The meta agent iteratively proposes, implements, evaluates, and archives new agents, using reflection to check novelty and refine designs when evaluation errors occur.Error-driven refinement can repeat up to five times before the agent and its evaluation metrics are added to the archive.Source: §3 · pp. 4–5
  • The meta agent is encouraged to explore novel or worthwhile agents from the archive while maximizing task performance, measured with metrics such as success rate or F1 score.The archive can optionally start with baseline agents.Source: §3 · pp. 4–5

4 EXPERIMENTS

Across ARC, reasoning benchmarks, and transfer tests, Meta Agent Search discovers agents that outperform hand-designed baselines and retain advantages across tasks and models.

4.1 CASE STUDY: ARC CHALLENGE

  • On ARC, Meta Agent Search progressively discovers agents that outperform hand-designed baselines by building on and combining design-pattern stepping stones.An early pattern combined multiple Chain-of-Thought answers, refinement, and ensembling; the best agent later combined diverse, expert, and human-like feedback in a complex refinement mechanism.Source: Figure 3 · p. 5, §4.1 · p. 6

4.2 REASONING AND PROBLEM-SOLVING DOMAINS

  • Meta Agent Search outperforms hand-designed agents across all four domains, improving Reading Comprehension by 13.6/100 F1 and Math by 14.4% accuracy.It also outperforms prompt-optimization methods across domains; the reported gap is smaller in Multi-task and Science, which the authors hypothesize reflects limits in foundation-model knowledge.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 GENERALIZATION AND TRANSFERABILITY

  • On ARC, searched agents transferred to Claude-Haiku, GPT-4, and Claude-Sonnet consistently outperform hand-designed agents; the best agent reached nearly 50% accuracy with Claude-Sonnet.The top three agents were selected by GPT-3.5 test accuracy before transfer.Source: §4.3 · p. 8
  • Transferred from MGSM, the discovered agents improve accuracy by 25.9% on GSM8K and 13.2% on GSM-Hard versus baselines, and outperform hand-designed baselines on non-math domains.Their non-math performance does not fully match agents designed specifically for those target domains.Source: Table 2 · p. 8, §4.3 · p. 8

5 RELATED WORK

Related work frames ADAS alongside agentic-system building blocks and AI-generating/AutoML research. It distinguishes code-based agent discovery from prior approaches that optimize prompts, roles, tools, or workflows.

Agentic Systems

  • Agentic systems combine building blocks such as prompting, reasoning, reflection, memory, tools, and role-based collaboration, while ADAS seeks to automate discovery of new blocks and system designs.Source: §5 · pp. 8–9

AI-Generating Algorithms and AutoML

  • ADAS occupies the architecture- and learning-algorithm meta-learning pillars, and uses Foundation Models to program new agents in code.Source: §5 · p. 9

Existing Attempts to ADAS

  • Prior ADAS attempts mostly optimize prompts or roles, which can be domain-specific, while less-explored approaches optimize agent workflows as graphs.Source: §5 · p. 9
  • Prior methods that learn tools or workflows may miss agent designs or create harder search spaces; this work represents all components in code for FM-guided search.Source: §5 · p. 10

6 DISCUSSION AND CONCLUSION

The paper presents ADAS and Meta Agent Search as a route to automatically designing powerful agents, while highlighting safety considerations and directions for future research.

Safety Considerations

  • Generated code could still be destructive because of capability or alignment limits, so the authors use isolated container execution, manual inspections, and user warnings.They describe these practices as consistent with safety standards used in SWE-Bench and Voyager.Source: §6 · p. 10

Discussion

  • ADAS is presented as accessible through Foundation Model API access and potentially safer through explicit, interpretable workflows that improve controllability and auditability.The authors argue that publishing the work is net beneficial and call for safe-ADAS research to help ensure powerful systems remain aligned with human values.Source: §6 · p. 10

Future Work

  • Future directions include recursively improving meta-agents, learning from post-deployment feedback, and optimizing cost, latency, or robustness alongside performance.The paper considers higher-order meta-learning, online continual learning, and multi-objective search as promising directions.Source: §6 · p. 10
  • Results suggest that the best feedback and refinement design may vary by model: GPT-3.5 benefits from complex feedback, while other advanced models favor simpler feedback with more refinement.The authors interpret this difference as evidence that GPT-3.5 may be less capable at evaluating and refining answers.Source: §6 · p. 11
  • The authors propose extending evaluation to multi-step real-world environments and seeding ADAS with tools, retrieval, agent frameworks, multimodal capabilities, or multiple Foundation Models.Flexible model choice could account for instruction difficulty and data-privacy priorities.Source: §6 · p. 11
  • Further work could improve novelty search and evaluation by using richer search methods, execution logs, subjective-task metrics, and searches for agents that generalize across domains.The current approach evaluates on a single domain and uses numerical performance results, which the authors say can be expensive and miss useful information.Source: §6 · p. 11
  • ADAS may also offer a way to study how simple conditions can bootstrap increasingly complex systems analogous to human organizations and society.The authors connect this possibility to agentic systems’ use of natural language and prior work modeling organizations and towns with agents.Source: §6 · p. 11

Conclusion

  • Meta Agent Search builds agents from prior discoveries, outperforming hand-designed agents across domains and transferring well across models and domains.The authors describe this as a step toward automating the development of powerful agentic systems from the bottom up.Source: §6 · p. 11

SUPPLEMENTARY MATERIAL … C FRAMEWORK CODE

The supplement reports cross-domain transfer results and describes the prompts and code framework used to generate and implement agents. It sketches a chain-of-thought agent paired with a critic that evaluates answers and feeds feedback into iterative refinement.

A GENERALIZATION AND TRANSFERABILITY

B PROMPTS

  • The meta-agent prompt directs the model to use the archive to propose creative agent designs, then refine them through two rounds of reflection on novelty, implementation errors, and improvements.If generated code encounters a runtime error, the system reflects and reruns it up to five times.Source: B PROMPTS · pp. 21–22

C FRAMEWORK CODE

  • The framework is under 100 lines and uses namedtuple Info objects to pass information between modules and construct prompts from metadata.It also lets agents be defined with a forward function, illustrated by a self-reflection implementation.Source: C FRAMEWORK CODE · pp. 23, 25
  • The Chain-of-Thought module produces thinking and an answer, with its reflection instruction asking it to use prior attempts and feedback to avoid errors.Source: C FRAMEWORK CODE · p. 25
  • The Critic module reviews the answer, provides feedback, and marks whether it is correct.Source: C FRAMEWORK CODE · p. 25
  • The code first generates an answer, then loops up to N_max, obtains critic feedback, appends the prior thinking, answer, and feedback, and reruns reflection.Source: C FRAMEWORK CODE · p. 25

D EXPERIMENT DETAILS FOR ARC CHALLENGE

The ARC experiments represent grids as integer arrays and ask agents to generate code that learns transformations from examples and predicts test outputs. The discovered ARC agent combines candidate generation, feedback-driven refinement, and an ensemble decision.

ARC evaluation setup

  • Generated code is checked against example grids and applied to the test grid, with accuracy measured by exact match; GPT-4o powers the meta agent and GPT-3.5 Turbo evaluates discovered agents and baselines.The model split is used to reduce compute cost.Source: D EXPERIMENT DETAILS FOR ARC CHALLENGE · pp. 25–26

ARC task and representation

  • Figure 4 illustrates the ARC task format: infer transformation rules from input-output grid examples and apply them to a test grid.The grids are represented as two-dimensional arrays, with each color encoded by an integer.Source: D EXPERIMENT DETAILS FOR ARC CHALLENGE · p. 25, Figure 4 · p. 26
  • ARC tasks provide three demonstration input-output pairs and one test input; the agent must infer a transformation and produce the test output grid.A transformation need only be unambiguous and applicable to the examples and test input, not to all possible inputs.Source: D EXPERIMENT DETAILS FOR ARC CHALLENGE · p. 26

ARC agent implementation

  • The discovered ARC agent generates multiple coded candidates, tests them on examples, gathers human-like and expert feedback, refines solutions, and selects a final answer from the top three.Candidates are retained for refinement only if they solve at least one example; feedback is structured before iterative refinement.Source: D EXPERIMENT DETAILS FOR ARC CHALLENGE · pp. 27–28

E EXPERIMENT DETAILS FOR REASONING AND PROBLEM-SOLVING DOMAINS

The experiments use sampled validation and test sets, with repeated GPQA evaluations and domain-specific prompting styles. The evaluated tasks span science, reading comprehension, multilingual math, and broad subject knowledge.

Experiment setup

  • GPQA uses 32 validation and 166 test questions, with five evaluations; other domains use 128 validation and 800 test questions, with one evaluation each.Questions are zero-shot except for one-shot DROP; the meta agent uses gpt-4o-2024-05-13, while discovered agents and baselines use gpt-3.5-turbo-0125.Source: E EXPERIMENT DETAILS FOR REASONING AND… · p. 29

DROP (Reading Comprehension)

  • DROP assesses discrete reasoning and detailed comprehension across multiple paragraphs.The benchmark description provided to the meta agent frames the task as reading comprehension requiring discrete reasoning.Source: E EXPERIMENT DETAILS FOR REASONING AND… · p. 29

GPQA (Science)

  • GPQA is a challenging expert-designed multiple-choice benchmark covering biology, physics, and chemistry.The GPQA description provided to the meta agent identifies it as the Graduate-Level Google-Proof Q&A Benchmark.Source: E EXPERIMENT DETAILS FOR REASONING AND… · p. 29

MGSM (Math)

MMLU (Mult-task)

  • MMLU assesses question answering across a wide range of subjects and difficulty levels, including STEM, social sciences, and humanities.The benchmark is the Massive Multitask Language Understanding benchmark.Source: E EXPERIMENT DETAILS FOR REASONING AND… · p. 30

F BASELINES · G EXAMPLE AGENTS · H PSEUDOCODE OF THE META AGENT SEARCH

The appendices specify the hand-designed baselines used in the experiments, illustrate three agents discovered by Meta Agent Search, and give pseudocode for the search workflow.

F BASELINES

  • Seven hand-designed baselines are implemented: five for ARC and two additional baselines for Reasoning and Problem-Solving domains.The ARC baselines are Chain-of-Thought, Self-Consistency with Chain-of-Thought, Self-Refine, LLM-Debate, and Quality-Diversity; the additional baselines are Step-back Abstraction and Role Assignment.Source: F BASELINES · p. 30
  • The baseline implementations use step-by-step prompting, five-answer sampling for COT-SC, up to five Self-Refine iterations, two debate rounds, and three Quality-Diversity iterations.Role Assignment first selects a role from a predefined set and then answers in that role; COT-SC uses majority voting or an FM query to ensemble answers.Source: F BASELINES · p. 30

G EXAMPLE AGENTS

  • The Multi-Step Peer Review Agent, discovered on GPQA, has multiple modules generate answers, review one another, refine their attempts, and feed a final decision module.The example uses separate science expert and critic roles for module assignments.Source: G EXAMPLE AGENTS · p. 31
  • The Divide and Conquer Agent decomposes a problem, assigns sub-problems to specialized experts, and integrates their solutions into a final answer.It was discovered during search in the GPQA Reading Comprehension domain.Source: G EXAMPLE AGENTS · pp. 31–32
  • The Verified Multimodal Agent, discovered on MGSM, creates a visual representation, verifies it, and uses the verified visual aid in a Chain-of-Thought solution.Its example implementation uses separate visual-representation, verification, and Chain-of-Thought modules.Source: G EXAMPLE AGENTS · pp. 31–33

H PSEUDOCODE OF THE META AGENT SEARCH

  • Meta Agent Search initializes an archive with baseline agents, then iteratively designs and codes new agents, uses two self-reflection steps, evaluates them, and adds them with their metrics to the archive.When evaluation produces errors, the design can be refined for up to five iterations, with evaluation rerun after each refinement.Source: H PSEUDOCODE OF THE META AGENT SEARCH · p. 33

I IMPACT OF INITIALIZATION

Meta Agent Search discovers agents that outperform hand-crafted baselines across evaluated domains even without initialization. Initial solutions generally improve performance, but math is an exception where starting from scratch performs better.

I IMPACT OF INITIALIZATION

  • Without initial agent designs, Meta Agent Search discovers agents that outperform hand-crafted baselines across all evaluated domains.Table 6 compares Meta Agent Search with and without initial designs across multiple domains.Source: I IMPACT OF INITIALIZATION · p. 33, Table 6 · p. 34
  • Including good initial solutions generally improves performance, but starting from scratch performs better in math.The authors hypothesize that scratch initialization encouraged broader, more diverse reasoning strategies within limited iterations, benefiting math tasks.Source: I IMPACT OF INITIALIZATION · p. 34, Table 6 · p. 34
  • The authors identify initialization choice and quality as future research questions, including when scratch search may help and how to combine initialization with broad exploration.These are proposed directions for exploring search effectiveness across domains.Source: I IMPACT OF INITIALIZATION · p. 34

J COST OF EXPERIMENTS

A search-and-evaluation run costs approximately $500 USD on ARC and about $300 USD in reasoning and problem-solving domains. Evaluation queries drive most of the expense, while a cheaper, better-performing model and more sophisticated evaluation functions could reduce costs.

J COST OF EXPERIMENTS

  • Approximately $500 USD per ARC search-and-evaluation run and about $300 USD in reasoning and problem-solving domains are reported costs.These estimates cover a single run of search and evaluation.Source: J COST OF EXPERIMENTS · p. 34
  • Evaluation queries to gpt-3.5-turbo-0125 are the primary expense; gpt-4o-mini costs less than one-third as much and offers better performance.The authors suggest Meta Agent Search could achieve improved results at one-third of the cost with gpt-4o-mini. They also expect more sophisticated evaluation functions to reduce costs because the current function is expensive and overlooks valuable information.Source: J COST OF EXPERIMENTS · p. 34

Related papers

Loading 2408.08435v2…