Source-linked AI summary
AFlow: Automating Agentic Workflow Generation
Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, Chenglin Wu
TL;DR
Agentic workflow construction requires substantial human effort, and existing automation remains incomplete or inefficient. AFlow formulates workflow optimization as MCTS over code-represented workflows with LLM-invoking nodes, reusable operators, and execution feedback. Across six benchmarks, it outperformed manually designed and existing automated methods, while enabling cost-efficient smaller-model workflows.
Problem
Manual workflow construction limits LLM scalability and adaptability, while prior automation requires setup or struggles with workflow diversity and efficient search.
Method
AFlow uses LLMs within a variant of MCTS to search code-represented workflows composed of configurable nodes, edges, and reusable operators.
Results
Across six benchmark datasets, AFLOW outperformed manually designed methods by 5.7% and existing automated approaches by 19.5%.
Takeaways & Limitations
AFLOW workflows enabled smaller language models to outperform larger models on the Pareto front of cost-effectiveness.
Takeaways & Limitations
The paper focuses on reasoning tasks with numerical evaluation functions, and its formulation adopts code as the primary edge structure.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable potential in solving complex tasks across diverse domains, typically by employing agentic workflows that follow detailed instructions and operational sequences. However, constructing these workflows requires significant human effort, limiting scalability and generalizability. Recent research has sought to automate the generation and optimization of these workflows, but existing methods still rely on initial manual setup and fall short of achieving fully automated and effective workflow generation. To address this challenge, we reformulate workflow optimization as a search problem over code-represented workflows, where LLM-invoking nodes are connected by edges. We introduce AFlow, an automated framework that efficiently explores this space using Monte Carlo Tree Search, iteratively refining workflows through code modification, tree-structured experience, and execution feedback. Empirical evaluations across six benchmark datasets demonstrate AFlow's efficacy, yielding a 5.7% average improvement over state-of-the-art baselines. Furthermore, AFlow enables smaller models to outperform GPT-4o on specific tasks at 4.55% of its inference cost in dollars. The code is available at https://github.com/FoundationAgents/AFlow.
1 INTRODUCTION
AFLOW addresses the human effort and incomplete automation required to design agentic workflows by searching over code-represented workflows. It uses MCTS and reusable operators to discover workflows, outperforming prior methods across six benchmarks.
- Manual design and refinement of agentic workflows require substantial human effort, limiting LLM scalability and adaptability across domains.
- Existing automation methods still require manual setup, lack workflow diversity, or struggle to find effective workflows efficiently.
- AFLOW models workflows as interconnected LLM-invoking nodes whose code-based edges define logic, dependencies, and execution flow.
- Reusable operators, including Ensemble and Review & Revise, provide predefined node combinations that support workflow construction during search.
- AFLOW uses MCTS with soft mixed-probability selection, LLM-driven expansion, execution evaluation, and experience backpropagation to explore workflows.
- 5.7% improvement over manually designed methods and 19.5% over existing automated approaches were reported across six benchmark datasets.
2 RELATED WORK
Related work distinguishes agentic workflows from autonomous agents and categorizes automated optimization by what is optimized. AFLOW extends workflow optimization with named nodes, operators, and tree-structured MCTS experience.
- Agentic workflows use predefined sequences of multiple LLM invocations, whereas autonomous agents make dynamic decisions based on their environment.
- Workflows include general approaches and domain-specific processes for tasks such as code generation, data analysis, mathematics, and question answering.
- Automated agentic optimization covers prompt optimization within fixed workflows, hyperparameter optimization, and optimization of entire workflow structures.
- AFLOW extends code-based workflow representation with named nodes, predefined node-combination operators, tree-structured experience, and execution feedback.
3 PRELIMINARY
AFLOW formulates agentic workflow optimization as search over configurable LLM nodes and edge structures, selecting code for expressive workflow relationships. Its MCTS framework narrows and explores this space with operators and execution feedback.
- 3.1 PROBLEM FORMULATION: An agentic workflow is a series of LLM-invoking nodes connected by edges that define execution order.
- 3.1 PROBLEM FORMULATION: Nodes can vary by model, prompt, temperature, and output format, while edges represent relationships among node operations.
- 3.1 PROBLEM FORMULATION: Code is adopted as the primary edge structure because standard programming constructs express sequences, conditions, loops, and parallel relationships.
- 3.1 PROBLEM FORMULATION: Workflow optimization searches configurations of node parameters and edge structures to maximize an evaluation function for a task.
- 3.2 AFLOW OVERVIEW: Previous methods were limited by restricted search spaces, while ADAS used a larger space but inefficient linear heuristic search.
- 3.2 AFLOW OVERVIEW: AFLOW uses LLMs as optimizers within a variant of MCTS to explore code-represented nodes and edges and refine workflows through experience backpropagation.
- 3.2 AFLOW OVERVIEW: Fixing model, temperature, and format focuses the search on prompts and code-based edges, while operators package common agentic operations.
- 3.2 AFLOW OVERVIEW: AFLOW targets reasoning tasks with numerical evaluation functions and uses operators such as Generate, Format, Review and Revise, Ensemble, Test, Programmer, and Custom.
4 THE DESIGN DETAILS OF AFLOW
AFLOW treats complete workflows as nodes in an MCTS search tree and iteratively improves them through LLM-based expansion, execution feedback, and experience propagation. Its selection strategy balances exploiting high-scoring workflows with continued exploration, while early stopping limits unnecessary cost.
- Each MCTS tree node represents a complete workflow, enabling AFLOW to search for solutions across classes of problems.
- AFLOW preserves successful and failed workflow-modification experiences in the tree, reusing them when workflows are revisited while retaining blank-template exploration to avoid local optima.
- AFLOW initializes a template workflow, partitions data into validation and test sets, and evaluates candidate workflows during iterative optimization.
- The search selects workflows using a soft mixture of uniform and score-based probabilities, including the initial workflow to balance exploration and exploitation.
- LLM-based expansion modifies prompts or node connections using prior experience, after which workflows are executed repeatedly to obtain more reliable feedback.
- Backpropagation records performance and modification success, while early stopping ends the search after stagnation or a fixed iteration limit.
5 EXPERIMENTS
AFLOW is evaluated on six benchmarks spanning QA, code, and math against manual and automated workflow baselines. It improves average performance, transfers across executor models, identifies cost-effective workflows, and benefits from reusable operators while remaining capable without them.
- Datasets and metrics: AFLOW is evaluated on six public benchmarks using validation and test splits, with task-specific metrics for math, code, and question answering.
- Experimental setup: AFLOW is compared with direct invocation, reasoning, debate, refinement, MedPrompt, and ADAS workflow baselines using multiple executor models.
- Main results: 5.7% average improvement over manually designed methods and 19.5% over contemporary automatic workflow optimization methods are reported across six datasets.
- Transferability: Most workflows transfer strongly across executor models, although the workflow found with DeepSeek-V2.5 performs notably weaker on GPT-4o-mini than one found with GPT-4o-mini.
- Cost analysis: AFLOW identifies workflows that let weaker models outperform stronger models on the cost-effectiveness Pareto front.
- Ablation study: Operators improve search efficiency and workflow quality, while AFLOW without operators still reaches 93.1% on GSM8K and autonomously develops ensemble-like structures.
- Case study: On GSM8K, AFLOW evolves from a blank template through single-step operator additions and prompt modifications, retaining known paths while exploring alternatives.
6 CONCLUSION
AFLOW formulates automated workflow optimization as search over code-represented workflows and uses MCTS to explore and refine them. Across six benchmarks, it outperformed manually designed and existing automated methods while reducing reliance on human intervention.
- Conclusion: Across six benchmarks, AFLOW outperformed manually designed methods and existing automated optimization approaches.The conclusion attributes this performance to navigating the large workflow search space with code-represented workflows and MCTS.
- Conclusion: AFLOW autonomously discovered effective workflow structures even when predefined operators were removed.The ablation study found that it independently developed an ensemble-like operator.
- Conclusion: AFLOW placed weaker models ahead of stronger ones on the cost-effectiveness Pareto front.The result connects automated workflow optimization with computational-cost optimization.
- Conclusion: AFLOW represents workflows as interconnected LLM-invoking nodes with code-based edges that control inputs, outputs, logic, and dependencies.This graph-like representation supports flexible workflow structures beyond a fixed sequence of operations.
B.1 CASE STUDY OF AFLOW
AFLOW evolves workflows from minimal initial settings into task-specific structures with limited human intervention. Case studies show distinct workflows for code generation, mathematical reasoning, programming, and question answering.
- Case studies: AFLOW evolved from an empty workflow into a solution highly similar to a manually designed code-generation workflow.The case demonstrates comparable workflow efficiency with minimal human intervention.
- MATH: For MATH, AFLOW combined programmatic solutions, multiple reasoning strategies, answer refinement, and ensemble selection.The resulting workflow adapted a task-agnostic starting point into a complex, task-specific solution process.
- MBPP: For MBPP, AFLOW found a simple workflow combining operators with a FIX-CODE PROMPT by the fourteenth iteration.Its score was described as extremely high and stable, indicating a favorable cost-performance balance.
- HotpotQA: For HotpotQA, AFLOW used execution feedback to learn both reasoning and answer-formatting behavior.The case study attributes improved formatting to identifying the correct format through feedback.
- Ablation: Without predefined operators, AFLOW independently developed an ensemble-like operator that generated multiple solutions and selected among them.This ablation result indicates that the framework can discover effective structures under reduced guidance.
C.1 TREE-STRUCTURED EXPERIENCE.
AFLOW stores workflow-optimization experience in a tree structure and uses feedback to guide subsequent search. The case analysis links successful improvements to executable computation, ensemble selection, detailed solutions, and review steps.
- Less effective optimization steps: Removing the Programmer operator reduced a score from 0.4873 to 0.4336.The passage associates this decrease with losing the computational precision of programmatic solutions.
- Less effective optimization steps: Removing the review step reduced a score from 0.5378 to 0.5210.The passage links the decrease to losing solution-quality checks during refinement.
- Successful optimization steps: The Programmer operator improved a score from 0.4874 to 0.5244 by adding executable Python computation alongside result refinement.The reported change associated concrete computational capabilities with numerical accuracy and explanation quality.
- Successful optimization steps: Adding ScEnsemble improved a score from 0.5244 to 0.5277 by selecting among multiple solution approaches.The passage describes increased solution diversity and reliability through ensemble selection.
- Successful optimization steps: Adding detailed step-by-step solution generation improved a score from 0.52773 to 0.5513.The reported benefit was greater clarity and comprehensiveness while maintaining accuracy.
- Tree-structured experience: Tree-structured experience lets AFLOW explore multiple optimization directions while pruning less successful branches.Each tree node represents a workflow configuration, and scores guide later optimization decisions.
D PARETO FRONT: DETAILED COST-PERFORMANCE DATA
AFLOW is positioned as an automated workflow approach that reduces reliance on manually designed initial workflows while improving scalability and generalizability. The surrounding discussion situates this goal within broader workflow and multi-agent-system research.
- Workflow, agentic workflow, and multi-agent-system research uses related but distinct terminology for structured task-solving systems.
- Dynamic workflow adjustment based on execution feedback is presented as complementary related work to AFLOW’s automated generation and optimization focus.
- AFLOW targets automated workflow generation and optimization to reduce dependence on manually designed initial workflows.
F.1 ADAPTING AFLOW FOR OPEN-ENDED TASKS
AFLOW is extended to open-ended tasks by replacing numerical evaluation with LLM-as-a-judge assessment and adapting the workflow-optimization prompt. The extension is evaluated through long-form novel-generation and academic-idea-generation scenarios.
- Motivation: Open-ended tasks lack fixed numerical feedback because performance is typically judged by humans, limiting direct use of AFLOW’s original evaluation approach.
- Prompt adaptation: The workflow-optimization prompt is modified by removing reasoning-specific instructions for open-ended tasks.
- Evaluation adaptation: AFLOW’s open-ended-task extension uses an LLM-as-a-judge evaluation prompt to score responses while retaining the core workflow architecture.
- Evaluation: The adapted AFLOW is demonstrated on long-form novel generation and academic idea generation, with three human annotators scoring and ranking iteration outputs.
F.2.1 LONG-FORM NOVEL GENERATION
The long-form novel-generation case study compares a baseline Claude-3.5-sonnet response with an AFLOW-optimized workflow. AFLOW iteratively constructs a multi-stage workflow for outlining, character development, and chapter generation.
- Setup: The case study uses Claude-3.5-sonnet for execution and optimization and GPT-4o for evaluation on a question without reference answers.
- Workflow: The optimized workflow generates an outline, character profiles, and chapters through separate prompts and repeatedly appends generated chapters to the novel.
- Workflow: The workflow implementation generates 10 chapters in a loop and returns the novel together with the execution cost.
- Comparison: The baseline model declined to generate a novel-length narrative of exactly 20,000 words in one response.
- Comparison: The baseline response cited the difficulty of maintaining an exact word count alongside a cohesive narrative with developed characters and themes.
- AFLOW output: The AFLOW example begins with a chapter titled “The Ticking of Regret” and describes Emma Sinclair confronting a deadline while perceiving time unusually slowly.
F.2.2 ACADEMIC IDEA GENERATION
The academic-idea-generation case study applies AFLOW to a task requiring one feasible research proposal. Its optimized workflow decomposes generation, prioritization, elaboration, evaluation, and refinement into successive stages.
- Setup: The case study tests AFLOW with GPT-4o-mini for execution, Claude-3.5-sonnet for optimization, and GPT-4o for evaluation across 10 questions.
- Task: The task requires one concrete research idea that addresses significant challenges and demonstrates feasibility with current technology.
- Optimization: The optimized workflow was obtained after six AFLOW iterations.
- Workflow: The workflow generates a research idea, prioritizes the most promising option, elaborates it, evaluates it, and refines weaknesses.
- Evaluation: The evaluation criteria include novelty and originality, feasibility with current technology, potential impact, and clarity and coherence.
- Search assumptions: AFLOW’s search-space completeness assumes code-represented edges can express valid node relationships and LLM expansion can generate valid modifications with non-zero probability.
- Search assumptions: AFLOW’s convergence argument additionally assumes a bounded evaluation function, valid workflows, and a non-zero probability of generating improvements.