Source-linked AI summary
EvoFlow: Evolving Diverse Agentic Workflows On The Fly
Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang, Hong Cheng, Kun Wang, Shuyue Hu, Lei Bai
TL;DR
Existing agentic automation often lacks LLM heterogeneity and optimizes for performance alone, despite variation in model capabilities and query complexity. EvoFlow uses niching-based evolutionary search to build a population of heterogeneous, complexity-adaptive workflows. Across reported evaluations, it outperforms prior workflows and can surpass o1-preview at a fraction of its inference cost.
Problem
Agentic automation pipelines often lack heterogeneous LLMs and focus on single-objective performance, while model capabilities and query difficulty vary.
Source: §1 · p. 2Method
EvoFlow applies niching-based evolutionary search to a population of heterogeneous, complexity-adaptive workflows, using tag-based retrieval, crossover, mutation, and niching selection.
Source: §1 · p. 2Results
Across seven benchmarks, EvoFlow workflows range from simple I/O to complex multi-turn interactions and outperform prior handcrafted and automated workflows by 1.23%∼29.86%; it also surpasses o1-preview at 12.4% of its inference cost using weaker open-source models.
Source: §1 · p. 2Takeaways & Limitations
EvoFlow’s reported results support cost-effectiveness-driven optimization of diverse workflows, with superior performance and significantly lower token costs across six benchmarks.
Source: §6 · p. 8The initial operator set is finite, though practitioners can customize templates and crossover or mutation can generate novel operators.
Source: §4.1 · p. 4
Abstract
from arXiv · showhide
The past two years have witnessed the evolution of large language model (LLM)-based multi-agent systems from labor-intensive manual design to partial automation (\textit{e.g.}, prompt engineering, communication topology) and eventually to fully automated design. However, existing agentic automation pipelines often lack LLM heterogeneity and focus on single-objective performance optimization, limiting their potential to combine weaker models for more customized and cost-effective solutions. To address this challenge, we propose EvoFlow, a niching evolutionary algorithm-based framework to automatically search a population of heterogeneous and complexity-adaptive agentic workflows, rather than a single homogeneous, complex workflow. Technically, EvoFlow performs \textit{(1) tag-based retrieval} to extract parent workflows from an agentic population, evolves new workflows through \textit{(2) crossover} and \textit{(3) mutation}, and employs \textit{(4) niching-based selection} to maintain population diversity and quality. Extensive evaluations across seven benchmarks demonstrate that EvoFlow is: \textbf{(I) diverse}, evolving a population of workflows ranging from simple I/O tasks to complex multi-turn interactions; \textbf{(II) high-performing}, outperforming previous handcrafted and automated workflows by $1.23\%\sim29.86\%$; \textbf{(III) economical}, surpassing powerful \llmname{o1-preview} at $12.4\%$ of its inference cost using weaker open-source models.
1. Introduction
Existing automated workflows often lack model heterogeneity and complexity diversity, motivating a search for workflows suited to varied queries. EvoFlow evolves a diverse workflow population and reports strong performance and cost results.
1. Introduction
- Existing automated pipelines often use homogeneous models and favor single, complex workflows, although models have complementary capabilities and queries vary in difficulty.The authors argue for heterogeneous model ensembles and simpler workflows for straightforward queries, reserving complex workflows for intricate challenges.Source: §1 · p. 2
- EvoFlow formulates workflow search as cost-performance multi-objective optimization and evolves a population of heterogeneous, complexity-adaptive workflows rather than one homogeneous complex workflow.Its workflow units are operator nodes assembled from LLM-agent invoking nodes.Source: §1 · p. 2
- For each incoming query, EvoFlow retrieves relevant parent workflows by tags, applies crossover and LLM, prompt, or operator mutation, then uses niching selection to maintain population diversity and quality.The contribution summary characterizes the resulting population as spanning simple I/O tasks through complex multi-turn interactions.Source: §1 · p. 2
- At 12.4% of o1-preview’s inference cost, EvoFlow using weaker open-source models surpasses it; homogeneous EvoFlow exceeds state-of-the-art workflows by 1.23%∼29.86% on six benchmarks.The contribution summary separately reports evaluations across seven benchmarks and a workflow range from simple I/O to complex multi-turn interactions.Source: §1 · p. 2
2. Related Work
Prior agentic-system work spans manually designed multi-agent frameworks, several automation strategies, and evolutionary optimization. The authors distinguish EvoFlow by evolving workflows rather than limiting evolutionary search to single-agent prompts or profiles.
LLM-based Autonomous Agents
- Early multi-agent frameworks relied heavily on manual configuration, motivating efforts to automate agentic workflow design.The related-work discussion names CAMEL, AutoGen, BabyAGI, and LLM-Debate as early frameworks.Source: §2 · pp. 2–3
Automated Agentic Workflows
- Automated workflow methods optimize prompts, inter-agent topology, or agent personas, while many remain homogeneous and produce fixed workflows.The paper contrasts these limits with integration of heterogeneous models and dynamic resource allocation across tasks of different complexity.Source: §2 · p. 3
Evolutionary Algorithm
- EvoAgent and EvoPrompt use genetic algorithms for agent profiles or prompts, but the paper characterizes their automation as constrained to single-agent optimization rather than workflow-level evolution.The comparison is presented in Table 1.Source: §2 · p. 3, Table 1 · p. 3
3. Preliminary
EvoFlow represents workflows hierarchically, from LLM-invoking nodes through composite operators to complete workflows. Its formulation contrasts single-objective workflow optimization with a cost-performance Pareto objective.
Search Space
- EvoFlow’s search space is hierarchical: LLM-invoking nodes form composite operator nodes, which are combined into complete workflows.An invoking node includes a prompt, temperature, and an LLM instance characterized by model size, token cost, and inference delay.Source: §3 · p. 3
Problem Formulation
- Traditional agentic automation optimizes a workflow using a task domain and performance evaluator, whereas EvoFlow formulates optimization as multi-objective.The multi-objective formulation balances system cost and performance through a Pareto-optimal set of non-dominated workflows distributed near the Pareto front.Source: §3 · pp. 3–4
Overall Framework
- The framework combines operator nodes into workflows, then retrieves parents and generates offspring through crossover and mutation before updating the population with niching-based selection.Figure 3 presents this overall sequence.Source: Figure 3 · p. 4
Problem Formulation
- EvoFlow shifts from invoking-node search to operator-node search because the former is excessively large and omits existing composite structures such as CoT, ToT, and Multiagent Debate.Feasible operator nodes define the reformulated search space.Source: §3 · p. 4
4. Methodology
EvoFlow initializes a varied workflow population and evolves it as queries arrive through relevant-parent retrieval, crossover, mutation, and feedback-based selection. Selection within cost- and tag-based niches aims to preserve diversity across domains and complexity.
4.3. Niching-based Selection
- Environmental feedback evaluates new workflows in a multi-objective space; Pareto-dominance fitness within each niche supports removal of the worst-performing workflow.Fitness is computed for workflows in the niche and the new individual, with the highest-fitness workflow eliminated.Source: §4 · p. 4, §4.3 · p. 6
4.1. Population Initialization
- Initialization seeds operator templates with established single- and multi-agent structures; workflows sample operators, instantiate models and prompts, and receive utility tags for domain matching.The authors state that finite initial templates do not prevent broader exploration because practitioners can customize templates and crossover or mutation can generate novel operators.Source: §4.1 · pp. 4–5
4.2. Retrieval, Crossover, and Mutation
- For each query, EvoFlow retrieves relevant tagged workflows as parents, crosses them, then mutates LLMs, prompts, or operators to produce offspring.Operator mutation can remove or add operators or alter their topological connections.Source: §4.2 · p. 5
4.3. Niching-based Selection
- Cost- and tag-based niching guides selection across domains and complexity, addressing the tendency of single-objective methods to produce increasingly complex workflows and higher API costs.The selection mechanism identifies niches using cost and tag similarity, then evaluates workflows within each niche.Source: §4.3 · pp. 5–6
4.4. Discussion
- Query-by-query evolution gradually produces a Pareto set of agentic workflows with varying complexity and superior performance.EvoFlow continuously evolves, mutates, and selects workflows in response to incoming queries.Source: §4.4 · p. 6
5. Experiments
The experiments compare EvoFlow across homogeneous, heterogeneous, and cross-domain settings, then assess workflow costs and framework behavior.
5.1. Experiment Setup
- The evaluation covers six public benchmarks across math reasoning, code generation, and embodied tasks, using homogeneous, heterogeneous, and cross-domain settings.The setup uses one closed-source and four open-source LLM backbones.Source: §5.1 · p. 6, §5.2 · p. 6
5.2. Performance Analysis
- Across six benchmarks, EvoFlow outperforms handcrafted and automated workflows; on ALFWorld it scores 68.57%, 2.15% above AgentSquare.On MATH, it exceeds vanilla gpt-4o-mini by 11.41% and AFlow by 6.42%.Source: §5.2 · p. 6, Table 2 · p. 7
- In heterogeneous MATH experiments, EvoFlow’s four open-source models surpass o1-preview by 2.7% at 12.4% of its overall cost.Table 3 compares heterogeneous experiments on MATH and MBPP and includes o1-preview, while EvoFlow uses four open-source LLMs.Source: §5.2 · p. 7, Table 3 · p. 7
- Cross-domain training improves EvoFlow’s MBPP performance from 87.62% to 88.35%, while joint MATH+MBPP training reduces another method’s performance to 39.18%.The paper attributes EvoFlow’s MBPP gain to optimizing the workflow population rather than a single individual.Source: §5.2 · p. 7
5.3. Cost Analysis & Case Study
- On MATH, EvoFlow uses 37.5% of AFlowQwen’s training cost and 19.5% of its inference cost, selecting cheaper models when appropriate.AFlowQwen’s stated training and inference costs are $1.22 and $2.62, respectively.Source: §5.3 · p. 7
5.4. Framework Analysis
- Removing tag-based retrieval or LLM mutation degrades performance and increases variance; removing operator mutation lowers performance by 3.5%–7.3%.Increasing population size from N = 5 to N = 25 improves performance by 3.1% but raises per-query cost from 8e-4 to 2e-3; the experiments use N = 15.Source: §5.4 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8
6. Conclusion
EvoFlow reframes autonomous workflow search as cost-effectiveness-driven multi-objective optimization and evolves heterogeneous, complexity-adapted workflow populations.
6. Conclusion
- Experiments across six benchmarks show superior EvoFlow performance with significantly lower token costs.The framework uses niching-based evolutionary algorithms to evolve a population of heterogeneous, complexity-adapted workflows.Source: §6 · p. 8
Impact Statement
The statement reports no ethical concerns and describes expected implications for adaptive, scalable multi-agent systems.
Impact Statement
- The authors report no ethical concerns regarding EvoFlow’s motivation, design, experiments, or data use.They describe expected implications including automated heterogeneous workflows and task-specific solutions across practical domains.Source: Impact Statement · p. 9
- EvoFlow is presented as promoting performance- and cost-optimized heterogeneous workflows for more adaptive and scalable systems.The statement identifies practical domains broadly but does not specify particular deployments.Source: Impact Statement · p. 9
A. Notations
This section defines EvoFlow’s workflow notation and experience pools, and describes the procedures, objectives, repositories, prompts, and implementation guidance used to generate and evolve workflows.
A. Notations
- The notation defines LLM, prompt, and operator mutation functions, and tracks mutated workflows, niche membership, cumulative cost and performance, dominance, and fitness.The review directs readers to Table 4 for the notation definitions.Source: A. Notations · p. 13, Table 4 · p. 13
C. Optimization Objective
- A workflow dominates another when it is at least as good on performance and negative cost in every objective, and strictly better on at least one.Pareto-optimal workflows form a set whose objective-space front represents performance–cost trade-offs, offering alternatives for different operational constraints.Source: C. Optimization Objective · pp. 13–15
B. Algorithm Table
- EvoFlow retrieves parent workflows by tag similarity, creates offspring through crossover and mutation, then identifies a niche for selection and elimination.The algorithm executes parents, offspring, and niche workflows, updates their cumulative cost and performance, and uses fitness to identify the worst individual for population updates.Source: B. Algorithm Table · p. 13, C. Optimization Objective · p. 14
D. Operator Repository
- The operator repository includes Chain-of-Thought reasoning, debate (Du et al., 2023), taking a step back (Zheng et al., 2023b), and self-consistency (Wang et al., 2023a).The debate setup uses three debaters for up to two rounds; self-consistency aggregates five reasoning paths by majority vote.Source: D. Operator Repository · p. 15
- Other operators include iterative self-refinement, ensemble response ranking (Jiang et al., 2023), tool-using ReAct (Yao et al., 2023b), and dynamically routed ExpertPrompt (Richards & et al., 2023; Xu et al., 2023).Users can customize the repository by incorporating operators of their choice.Source: D. Operator Repository · p. 15
E.1. Tag Generation Prompt
- The tag-generation prompt asks for five tags reflecting a workflow’s relevant academic disciplines and complexity, intended to match similar future tasks by cosine similarity.It instructs the model to avoid broad tags and return exactly five task-relevant tags.Source: §E.1 · pp. 15–16
E.2. Offspring Generation Prompt
- The offspring prompt asks for an improved, efficient, cost-effective workflow suited to a query, using parent designs and favoring simple, composable communication patterns while allowing novel structures.The prompt also requests JSON output following a supplied structure.Source: §E.2 · pp. 16–17
E.3.1. LLM MUTATION LLM Mutation
- LLM mutation replaces the operator-initializing backbone from four listed models, while prompt mutation changes invoking-node prompts, for example by adding few-shot examples or clarifying instructions.Prompt changes can guide an operator’s reasoning or communication; generated Custom-method output can be incorporated elsewhere in the workflow.Source: §E.3.1 · p. 17, §E.3.2 · p. 18
- In the custom method, input and instruction are concatenated directly, so placeholders are unsupported and concatenation must be handled externally.The implementation note also advises avoiding single quotes in code and adding comments where necessary.Source: §E.3.2 · p. 18
- Operator mutation adds or removes operators from reference workflows, considering task compatibility and performance and organizing collaboration with appropriate prompts.Operators should be initialized in __init__ before being called.Source: §E.3.3 · p. 18
F.1. LLM Experience Pool
- The LLM experience pool records each model’s workflow-level correctness as positive, negative, or not used, alongside textual feedback on its contribution to solving the task.The pool aggregates this quantitative and qualitative experience across workflows and tasks to inform LLM selection and adaptive workflow optimization.Source: §F.1 · p. 19
F.2. Workflow Experience Pool
- The workflow experience pool stores workflow–query–evaluation triplets, with positive or negative outcome labels and qualitative feedback on effectiveness, efficiency, and limitations.It aggregates records across workflows and queries to support analysis of capabilities and limitations and iterative design refinement.Source: §F.2 · p. 19
G.1. Dataset Statistics and Splits
The section specifies dataset partitioning and a benchmark-specific MATH sample.
G.1. Dataset Statistics and Splits
- MATH uses 617 difficulty-level-5 problems from four types; datasets are generally split 1:4 for training and testing, except ALFWorld follows prior settings.The four MATH types are Combinatorics & Probability, Number Theory, Pre-algebra, and Pre-calculus.Source: §G.1 · p. 19
G.2. Baseline Setups
The baseline setups include reasoning and multi-agent methods alongside automated workflow systems, with implementation and configuration choices specified for several comparators.
G.2. Baseline Setups
- Table 5 is identified as the source of dataset statistics.The passage gives the table caption but no dataset values.Source: Table 5 · p. 20
- Self-consistency ensembles five CoT solutions, while LLM-Debate uses five same-model agents, up to two rounds, and majority voting.CoT prompts step-by-step reasoning; ComplexCoT uses the cited code implementation.Source: §G.2 · pp. 19–20
- LLM-Blender combines two gpt-4o-mini models with one Qwen-2.5-72b and one llama-3.1-70b; MacNet uses its densely connected MESH variant.DyLAN and AgentVerse use their respective published implementations.Source: §G.2 · p. 20
- GPTSwarm and AutoAgents follow their original settings, while ADAS adopts its original implementation.AgentSquare uses gpt-4o-mini with early-stopping patience 5; AFlow is restricted to gpt-4o-mini with MAX ITERATION=20 for homogeneous comparisons.Source: §G.2 · p. 20
H. Supplementary Results
Table 6 compares methods across LLM backbones and training-dataset settings.
H. Supplementary Results
- Table 6 covers MATH, MBPP, and combined MATH+MBPP training; its paired combined-setting values report MATH and MBPP performance, respectively.The caption supplies scope and value ordering, but not the table's numerical results.Source: Table 6 · p. 21