Source-linked AI summary
AgentFactory: Towards Automated Agentic System Design and Optimization
Enci Zhang, Haofeng Wang, Yuesheng Zhu, Xiaole Cui, Guibo Luo
TL;DR
Agentic-system design remains labor-intensive, while workflow-only automation overlooks foundation-model capabilities and deployment objectives. AgentFactory jointly optimizes fine-tuned models and workflows through a three-stage, multi-objective pipeline. Across eight benchmarks in five domains, it reports a 9.1% average performance gain, with larger specialized-domain improvements and lower inference costs.
Problem
Manual agentic-system design requires substantial effort, while workflow-only automation keeps models fixed and often focuses on single objectives.
Method
AgentFactory uses LLMs as optimizers in a three-stage pipeline that jointly searches foundation-model fine-tuning and workflow structures under performance, cost, and efficiency objectives.
Results
Across eight benchmarks spanning five domains, AgentFactory achieves a 9.1% average performance gain, including 19.6% on MedQA and 18.7% on FinEval, while maintaining lower inference costs.
Takeaways & Limitations
AgentFactory is presented as a promising approach for developing more capable and efficient agentic systems through automated optimization.
Takeaways & Limitations
When local fine-tuning is not feasible, AgentFactory uses the original model instead.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated remarkable capabilities as powerful components in agentic systems, enabling sophisticated reasoning and complex task execution. However, current approaches to manually designing and optimizing agentic systems heavily rely on manual effort, limiting their adaptability and scalability. Recent work has explored the automated optimization of workflow designs. However, these approaches often overlook the crucial role of model capabilities and focus on single performance metrics, failing to address real-world deployment constraints. In this paper, we present AgentFactory, a framework that jointly optimizes both foundation models and workflow structures in agentic systems while considering multiple objectives including performance, cost, and efficiency. AgentFactory leverages advanced LLMs as optimizers to navigate the vast search space of possible configurations, employing a three-stage optimization pipeline to automatically discover effective combinations of fine-tuned models and optimized workflows. Through an iterative optimization process, our framework systematically explores and evaluates different agentic system designs, adapting to task-specific requirements while maintaining operational efficiency. We evaluate AgentFactory across eight benchmarks spanning five domains, including general reasoning, coding, mathematics, medicine, and finance. Our experiments demonstrate that AgentFactory consistently outperforms both manually designed methods and existing automated approaches, achieving an average improvement of 9.1% across all benchmarks, with particularly significant gains in domain-specific tasks (19.6% on MedQA and 18.7% on FinEval). These results establish AgentFactory as a promising approach for developing more capable and efficient agentic systems through automated optimization.
1 Introduction
AgentFactory addresses the manual and workflow-only limitations of agentic system design by jointly optimizing models and workflows under multiple objectives. Across diverse benchmarks, it improves performance over existing approaches, especially in specialized domains.
- Motivation: Manual agentic-system design is labor-intensive, experience-dependent, and prone to sub-optimal configurations.Existing frameworks manually assign agent roles, workflow structures, and responsibilities, making the large design space difficult to explore.
- Motivation: Prompt optimization and workflow optimization automate parts of system design but do not fully address coordinated multi-component tasks.Prompt-focused methods may be insufficient for complex coordination, motivating broader workflow-level automation.
- AgentFactory: AgentFactory jointly optimizes foundation-model fine-tuning and workflow representations while considering performance, cost, and efficiency.Its optimization process uses powerful LLMs as optimizers to search beyond workflow-only designs.
- Results: 9.1% average performance improvement over existing methods is reported across eight benchmarks, with 19.6% on MedQA and 18.7% on FinEval.The reported gains are particularly pronounced in specialized domains, while inference costs remain lower.
- AgentFactory: AgentFactory uses a comprehensive automated agentic-system design and optimization framework.The contribution extends automated design beyond manually specified systems and fixed workflows.
2 Related Work
Related work has progressed from manually structured and single-component optimization toward automated workflow design, but existing approaches generally keep foundation models fixed. AgentFactory addresses this model-capability constraint by searching jointly over models and workflows.
- Agent-system evolution: Early agent systems used symbolic architectures or reinforcement learning, but faced limitations in flexibility or long-term planning.These limitations motivated the development of more sophisticated agent architectures.
- LLM-based agents: LLMs enabled flexible components for complex problem-solving, while agent architectures support deliberate reasoning and task coordination.Subsequent systems introduced reasoning structures, multi-agent discussion, role-playing, decomposition, and autonomous execution.
- Automated design: Automated agentic-system design includes prompt optimization, hyperparameter tuning, and workflow-structure optimization.Prompt and hyperparameter methods still face adaptation challenges and require substantial task-specific manual intervention.
- Automated design: Workflow-optimization methods use graph-based or code-based representations but encounter limitations with conditional logic or search constraints.These approaches broaden structural automation without changing the underlying foundation models.
- Research gap: Existing approaches commonly keep underlying models fixed, limiting gains because model capabilities influence agent behavior.AgentFactory instead performs iterative search over both model parameters and workflow representations.
3 Method
AgentFactory formulates agentic-system design as a multi-objective search over foundation-model configurations and workflow structures. Its LLM-guided pipeline iteratively plans, tunes models, designs workflows, evaluates candidates, and selects systems that satisfy targets or offer preferred trade-offs.
- Problem formulation: The optimization problem evaluates candidate systems on multiple objectives, including accuracy, cost, and latency, using task-specific evaluation functions.The evaluation functions produce a vector of metrics for task T.
- Problem formulation: AgentFactory searches jointly over LLM-based model configurations and workflow representations rather than optimizing workflows while keeping models fixed.The model space includes foundation models, instruction-tuning datasets, fine-tuning hyperparameters, and tuning methods such as LoRA, QLoRA, or full-parameter tuning.
- Workflow design: Workflow candidates can be represented as graphs, neural architectures, or executable code, enabling generated workflows to encode prompting, tool use, and control flow.The framework adopts code-based workflow representations in the ADAS style for complex agent behaviors.
- Optimization procedure: The LLM optimizer generates successive candidate systems from the previous optimization trajectory, model space, and meta-prompt, then records each system and its evaluation.Algorithm 1 initializes an empty trajectory, generates and evaluates an initial system, and repeats generation and evaluation for up to Kmax steps.
- Optimization procedure: Optimization stops when a candidate meets all target thresholds or when the maximum number of steps is reached; otherwise, the final system maximizes a scalarized objective over the trajectory.The scalarization function aggregates multiple objectives according to user preferences, yielding a Pareto-optimal trade-off.
- Optimization pipeline: AgentFactory decomposes optimization into planning, tuning, and workflow design before deploying a system that meets specified performance, cost, and efficiency requirements.Planning synthesizes strategies from task and system information; tuning specifies datasets and fine-tuning configurations, which are executed by a tuning backend.
4 Experiment Setup
AgentFactory was evaluated across five domains and eight benchmarks using comparisons with manual and automated agent-design methods under two foundation-model settings. The study also curated domain-relevant fine-tuning data to support model optimization.
- Evaluation Domains: Evaluation covered general reasoning, coding, mathematics, medicine, and finance across eight public benchmarks.The benchmarks included MMLU, DROP, HumanEval, MBPP, GSM8K, MATH, MedQA, and FinEval.
- Baselines: Comparisons included direct invocation, chain-of-thought, self-consistency, Reflexion, debate, ADAS, and AFlow.The manually designed baselines included 5-sample Self-Consistency CoT and 3-round Reflexion and Debate settings.
- Foundation Models: Experiments used Llama-3.1-8B-Instruct and GPT-4o-mini as foundation models, while AgentFactory could dynamically select between them.This setup evaluated performance and cost trade-offs across fixed-model and dynamic-selection settings.
- Fine-Tuning Data: Fine-tuning data combined benchmark training splits with CodeBagel, MathInstruct, and IndustryInstruction domain subsets.The curated data targeted programming, mathematical reasoning, health-medicine, and finance-economics tasks.
5 Experiment Results
AgentFactory outperformed manual and automated baselines across nearly all benchmarks while improving domain-specific performance and controlling inference cost. Results also show benefits from stronger optimizers, iterative joint optimization, and combining workflow design with fine-tuning.
- Main Results: 9.1% average performance improvement over current automated design methods was achieved across the benchmarks.The results were reported across eight benchmarks, with Avg denoting mean performance and Cost denoting total inference cost.
- Domain-Specific Results: 19.6% on MedQA and 18.7% on FinEval were the notable domain-specific gains over workflow-only optimization methods.The paper attributes these gains to jointly optimizing foundation models and workflows with targeted fine-tuning.
- Cost Efficiency: Up to 68% average cost reduction was achieved with GPT-4o-mini while maintaining the reported performance improvements.The multi-objective strategy balanced performance requirements against computational expenses by incorporating cost-effective models when possible.
- Optimizer Analysis: More capable optimizer LLMs produced better optimization outcomes, while GPT-4o and Claude-3.5-Sonnet produced similar results.These findings indicate robustness across optimizer implementations with comparable capability levels.
- Optimization Process: 74.6% FinEval accuracy followed mathematical-data integration, after workflow optimization raised accuracy from 62.3% to 66.7% and financial fine-tuning to 72.5%.The system later exceeded the 75% target and reduced cost from $1.14 to $0.60 by iteration 17.
- Ablation Study: The full pipeline surpassed fine-tuning-only optimization by 12.7% and workflow-design-only optimization by 8.2% on average.Both individual components improved over direct invocation, while their combination achieved the best overall performance across benchmarks.
6 Conclusion
AgentFactory formulates automated agentic-system design as LLM-guided multi-objective optimization over foundation models and workflow structures. Across eight benchmarks in five domains, it improved performance over existing methods while maintaining lower inference costs.
- Contribution: AgentFactory addresses prior methods’ focus on fixed models, workflow optimization, and single objectives.The framework jointly searches foundation-model parameters and workflow representations.
- Conclusion: 9.1% average performance gain was achieved across eight benchmarks, including 19.6% on MedQA and 18.7% on FinEval.The reported improvements were accompanied by lower inference costs.