Source-linked AI summary
OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, Guohao Li
TL;DR
Existing multi-agent systems are difficult to transfer across domains because they depend on domain-specific redesign and component-wide retraining. The paper introduces WORKFORCE, which decouples domain-agnostic planning from specialized execution, and OWL, which trains the planner for generalization. WORKFORCE reaches 69.70% accuracy on GAIA, while OWL improves Qwen2.5-32B-Instruct by 16.37% to 52.73%.
Problem
Existing multi-agent systems commonly require domain-specific redesign and retraining of all agents when applied to new domains, limiting cross-domain transferability.
Method
WORKFORCE combines a domain-agnostic Planner, Coordinator, and specialized Worker Nodes, while OWL uses supervised initialization followed by reinforcement learning to optimize the planner.
Results
69.70% accuracy was achieved on GAIA by WORKFORCE, while OWL improved Qwen2.5-32B-Instruct by 16.37% to 52.73%.
Takeaways & Limitations
The results support modular domain transfer through plug-and-play workers and a generalizable planning core for general-purpose AI assistants.
Takeaways & Limitations
Performance depends on reliable domain-specific tools, and reinforcement learning from real-world feedback can be time-consuming because of latency in online operations.
Abstract
from arXiv · showhide
Large Language Model (LLM)-based multi-agent systems show promise for automating real-world tasks but struggle to transfer across domains due to their domain-specific nature. Current approaches face two critical shortcomings: they require complete architectural redesign and full retraining of all components when applied to new domains. We introduce Workforce, a hierarchical multi-agent framework that decouples strategic planning from specialized execution through a modular architecture comprising: (i) a domain-agnostic Planner for task decomposition, (ii) a Coordinator for subtask management, and (iii) specialized Workers with domain-specific tool-calling capabilities. This decoupling enables cross-domain transferability during both inference and training phases: During inference, Workforce seamlessly adapts to new domains by adding or modifying worker agents; For training, we introduce Optimized Workforce Learning (OWL), which improves generalization across domains by optimizing a domain-agnostic planner with reinforcement learning from real-world feedback. To validate our approach, we evaluate Workforce on the GAIA benchmark, covering various realistic, multi-domain agentic tasks. Experimental results demonstrate Workforce achieves open-source state-of-the-art performance (69.70%), outperforming commercial systems like OpenAI's Deep Research by 2.34%. More notably, our OWL-trained 32B model achieves 52.73% accuracy (+16.37%) and demonstrates performance comparable to GPT-4o on challenging tasks. To summarize, by enabling scalable generalization and modular domain transfer, our work establishes a foundation for the next generation of general-purpose AI assistants.
1 Introduction
WORKFORCE addresses limited cross-domain transfer in multi-agent systems by separating general planning from specialized execution, while OWL trains the planner for broader generalization. On GAIA, the framework and training approach achieve strong reported performance with modular, open-source components.
- Motivation: Existing multi-agent systems often require domain-specific redesign for inference and retraining of every component when transferred to new domains.These constraints limit flexibility across domains.
- WORKFORCE: WORKFORCE separates a domain-agnostic Planner, a Coordinator, and specialized Worker Nodes to support modular cross-domain adaptation.Workers can be added or modified while the core planning architecture remains reusable.
- OWL: OWL combines supervised initialization with reinforcement learning to improve the generalization of a domain-agnostic planner.The training strategy targets cross-domain transfer while minimizing retraining requirements.
- Results: 69.70% accuracy was achieved by WORKFORCE on GAIA, exceeding the reported 55.15% accuracy of OpenAI’s Deep Research.GAIA evaluates multimodal reasoning, code execution, live web search, and diverse domains.
- Results: 52.73% score (+16.37%) was achieved by the OWL-trained Qwen2.5-32B-Instruct model without using GAIA data for training.The model outperformed GPT-4o-mini (47.27%) and Qwen2.5-72B-Instruct (49.09%).
- Contributions: The work releases its code, models, and data to support open research.
2 Preliminary
LLM agents perceive, reason, and act, while multi-agent systems coordinate multiple agents for complex tasks requiring diverse expertise. The paper frames WORKFORCE as a scalable approach that separates strategic planning from task execution across domains.
- LLM Agents: LLM-based agents operate in a perception-reasoning-action loop to pursue goals in their environments.They observe information, process it through a language model, choose actions, and execute them.
- Multi-Agent Systems: Multi-agent systems enable multiple LLM-based agents to collaborate on complex tasks requiring diverse expertise.Collaborative frameworks such as CAMEL and MetaGPT are presented as examples.
- WORKFORCE: WORKFORCE uses a Planner, Coordinator, and specialized Worker Nodes to coordinate tasks while adapting the worker layer across applications.The architecture preserves its core planning and coordination mechanisms when worker nodes change.
- Generalist Assistants: Generalist AI assistants target complicated tasks spanning multiple domains and modalities, including multimodal understanding, web browsing, reasoning, and problem-solving.GAIA is introduced as a benchmark for these capabilities.
3 Multi-Agent Inference: WORKFORCE
WORKFORCE addresses domain-specific rigidity in multi-agent systems by separating domain-agnostic planning from specialized execution. Its Planner, Coordinator, and Worker Nodes support modular task processing and strong GAIA performance.
- Motivation: WORKFORCE separates domain-agnostic planning from domain-specific execution to improve cross-domain transferability.The framework is designed to avoid complete redesigns for new application domains.
- Architecture: The Planner decomposes tasks, the Coordinator assigns subtasks and manages dependencies, and specialized Worker Nodes execute them with domain-specific tools.Workers return results through a shared task channel, while the Coordinator integrates intermediate outputs.
- Task Flow: WORKFORCE supports replanning when workers report failed subtasks, allowing the Planner to generate new subtasks from failure feedback.This mechanism provides test-time scaling for increasingly complex tasks.
- Experiments: 69.70% accuracy makes WORKFORCE the strongest open-source framework on GAIA, outperforming previous open-source systems across difficulty levels.The evaluation uses GAIA accuracy scores and compares WORKFORCE with multiple proprietary and open-source baselines.
- Experiments: 2.34% improvement over OpenAI’s Deep Research shows WORKFORCE can exceed a commercial proprietary framework on GAIA.The paper also reports a 1.89% advantage over Langfun Agent v2.1 on Level 1.
4 Multi-Agent Training: OPTIMIZED WORKFORCE LEARNING
OWL trains a generalizable Planner through supervised fine-tuning followed by preference-based reinforcement learning, using diverse task datasets and filtered trajectories. The resulting planner improves performance, especially on difficult tasks, while trajectory filtering further benefits training.
- Training Strategy: OWL uses a two-phase training paradigm that initializes the Planner with supervised fine-tuning and then optimizes it with DPO.DPO is used to improve decomposition strategies beyond imitation of expert demonstrations.
- Task Curriculum: The training curriculum spans web reasoning, tabular operations, mathematics, coding, document processing, and multimodal information handling.The selected datasets include HotpotQA, WikiTableQuestions, Math-related Problems, and Infinity-MM.
- Experiments: 52.73% performance and a 16.37% gain show that OWL-trained Qwen2.5-32B-Instruct surpasses GPT-4o-mini and Qwen2.5-72B-Instruct.GPT-4o remains higher overall at 60.61%, while the OWL-trained model is comparable on challenging Level 3 tasks.
- Experiments: DPO improves Level 3 performance by 7.69% over the base model, whereas supervised fine-tuning alone decreases Level 3 performance by 3.85%.The combined approach exceeds the base model across all difficulty levels.
- Ablation Study: Filtered trajectories consistently outperform unfiltered data, indicating that data quality matters more than quantity for planner training.This comparison is reported in the trajectory-filtering ablation study.
5 Analysis
WORKFORCE outperforms alternative agent frameworks across capability types, improves with OWL training and replanning, and remains robust on multi-capability tasks. Planner optimization yields larger gains than worker-only training, while foundation-model and tool limitations account for about half of errors.
- Performance Across Capability Types: WORKFORCE consistently outperforms Role Playing and Single Agent approaches across different capability types.
- Performance Across Capability Types: OWL training produces consistent improvements across all capability types.
- Test-time Scaling: Both GPT-4o and Claude-3.7-Sonnet workforces improve as replanning iterations increase, without access to ground truth.The authors interpret this as evidence of inherent self-correction and self-evolving capabilities during test time.
- Robustness Across Capability Requirements: WORKFORCE maintains consistent performance across tasks requiring one, two, or at least three capabilities.Role Playing drops from 62.3% to 34.6% on multi-capability tasks, whereas specialized workers handle focused subtasks.
- Planner vs. Worker Training: 45.45% from training only the planner exceeds 31.51% from training only workers, while joint training reaches 46.68% at substantially higher computational cost.The ablation supports prioritizing planner optimization because effective task decomposition is more crucial than enhancing individual workers.
- Error Analysis: Approximately half of WORKFORCE errors arise from foundation-model limitations or tool-related issues, with planner failures comprising 21.15% of agent-specific errors.The error analysis examines Claude-3.7-Sonnet results.
6 Related Work
Prior multi-agent systems use specialized roles and agent interactions for complex tasks, but many remain tied to domain-specific designs or fixed tool suites. OWL instead trains a single generalizable domain-agnostic planner to improve transferability and training efficiency.
- Earlier multi-agent systems assign specialized roles and interactions to tackle complex tasks beyond a single model’s capabilities.
- Supervised fine-tuning and reinforcement learning improve tool-augmented agents, but full-trajectory training scales poorly and can overfit fixed tool suites.
- MALT trains generator, verifier, and refiner agents through off-policy reward propagation across specialized roles.
- Unlike domain-specific multi-agent training, OWL trains a single generalizable domain-agnostic planner for domain transferability and training efficiency.
7 Conclusion
The paper concludes that WORKFORCE separates strategic planning from domain-specific execution to support cross-domain transfer without system redesign. Its GAIA results and OWL training gains motivate a scalable foundation for general-purpose assistants.
- WORKFORCE decouples strategic planning from domain-specific execution, enabling cross-domain transferability without system redesign.
- 69.70% accuracy on GAIA surpasses open-source alternatives and OpenAI’s Deep Research.
- OWL applies reinforcement learning and improves Qwen2.5-32B-Instruct by 16.37% on GAIA.
- The paper presents plug-and-play worker nodes with a generalizable planning core as a scalable foundation for general-purpose AI assistants.
- The work includes experiments, error analysis, dataset construction, code implementation, open-source maintenance, and methodology development by multiple contributors.
B Limitations
The system’s documented limitation is dependence on high-quality domain-specific tools and the time cost of reinforcement learning from real-world feedback. The supplied material also describes its processing workflow and task-prompt examples.
- Limitations: Performance depends on high-quality domain-specific tools, creating execution bottlenecks in domains without reliable toolkits.Reinforcement learning from real-world feedback is also time-consuming because of latency, including online-search delays.
- Workforce processing flow: WORKFORCE processes tasks through a Planner, Coordinator, and Worker registry connected by a shared task channel.Workers post results or failure information to the channel, while the coordinator manages assignments and processing.
- Prompt Examples: The planner prompt decomposes a task into numbered subtasks enclosed within <tasks> tags.The examples include extracting files, reviewing a PDF, executing code, and reporting the final answer.
- Prompt Examples: The example task asks how many applicants are missing exactly one qualification, using a ZIP file containing job-application data.The data includes 50 applicants and 15 missing Second Language values, which likely indicate no second language.
- Prompt Examples: The coordinator assigns subtasks to existing or newly created workers according to task requirements and available capabilities.The supplied workflow includes worker descriptions, task assignment, and a selected worker identifier.
D.3 Worker Implementation and Toolkit Design
WORKFORCE uses specialized worker agents and domain-aligned toolkits to execute diverse subtasks within a modular architecture. Its synthesis dataset spans multiple task types, while benchmark errors concentrate in tool and planning limitations.
- Worker Agents: WORKFORCE instantiates Web, Document Processing, and Reasoning/Coding agents with non-overlapping specialized tools.The agents handle web interaction, multimodal documents, and analytical reasoning or code execution, respectively.
- Toolkit Design: The toolkit supports browser simulation, document processing, multimodal analysis, and code execution through specialized capabilities.These capabilities are extended with additional agents and tools as task requirements evolve.
- Synthesis Dataset: 3,466 trajectories were collected, with 1,599 retained after cleaning; trajectories average 3.41 subtasks and 22.75 execution steps.Math tasks require the most subtasks at 3.87, while HotpotQA tasks require the most execution steps at 26.92.
- Failure Distribution: 32.69% of GAIA failures involve limited tool capability, while planner errors account for 21.15%.Limited model capability contributes 19.23% of errors, with worker, ambiguity, and response errors contributing smaller shares.
- Difficulty-Level Errors: Level 2 tasks have the most errors overall, whereas Level 3 tasks show more tool-selection and context-exceed failures.Level 2 includes six web-toolkit and six multimodal-toolkit failures; Level 3 includes four tool-selection and three context-exceed errors.
E.3 Error Cases Examples
The error examples expose failures from incorrect planning, incomplete subtask specifications, tool-selection mistakes, and conflicting intermediate results. These cases show how execution errors can arise even when individual subtasks produce plausible information.
- Subtask Scope: Subtask design can restrict an agent to matching tables with references without extracting the referenced papers’ contents.This limitation is explicitly identified in the table-to-reference analysis.
- Incorrect Plan: Incorrect planning can misinterpret the requested quantity, producing a station count that differs from the question’s intended traversal measure.The metro example distinguishes stations between endpoints from stations that must be passed to reach the destination.
- Example Outcome: The metro example identifies Shaw-Howard University and National Airport as Yellow Line stations, with seven stations listed between them.The listed intermediate stations exclude the starting station.
- Tool Selection Error: Tool-selection failure occurs when an agent searches metadata but does not open and inspect the video needed for the requested CFM values.The corrected response reports Cheater: 101.376 and Cheater Beater: 84.348.
- Result Verification: Conflicting subtask results can leave the final answer with incompatible values, as in the two reported Yoshi Circuit world-record times.The final agent response contrasts 1:23.964 with 1:41.614 and includes differing source details.
F.1 Differences in Information Sources
The paper highlights that real-world information sources can disagree, change over time, or become inaccessible. Such variation creates ambiguity in both answer selection and task execution.
- Evaluation Challenge: Real-world tasks may have several valid answers when information sources disagree, depending on which reference source is consulted.This source dependence is presented as a challenge for task evaluation.
- Information Sources: Different authoritative sources can produce divergent yet defensible answers, as homepage and Google Scholar publication orders differ.The homepage identifies “Mapping Human-Oriented Information to Software Agents for Online Systems Usage,” while Google Scholar lists “A new software agent ’learning’ algorithm” first.
- Outdated Information: Outdated or modified web content can prevent agents from reproducing an annotated answer.The Eva Draconis example became inaccessible because the YouTube channel and linked personal webpage no longer exposed the required information.
- Language Ambiguity: Ambiguous wording can support multiple interpretations, such as whether “2019 game” refers to an award year or release year.The two interpretations can lead to inconsistent results for the same benchmark question.
- Access Constraints: Network instability and permission restrictions can interrupt execution even after an agent locates the relevant resource.Authentication, human verification, and regional restrictions are named access barriers.
G.1 Data Pollution
Data pollution occurs when an agent retrieves pre-existing benchmark answers instead of performing the required reasoning. The paper also notes that tool use can create safety and system-stability risks.
- Data Pollution: Data pollution occurs when agents retrieve answers directly from sources such as GAIA’s Hugging Face repository instead of reasoning independently.The paper characterizes this as relying on pre-existing answers from training data.
- Mitigation: Filtering sources such as URLs containing “huggingface” is suggested as a mitigation for answer leakage.The goal is to prevent agents from accessing pre-existing benchmark answers.
- Safety Risks: Tool-enabled agents may access harmful content, generate inappropriate interactions, or damage local environments through code execution.These behaviors are presented as threats to user safety and system stability.
H Statistical Significance Analysis
Wilcoxon signed-rank tests assess whether Workforce and OWL-related improvements exceed random variability. The evaluation also reports capability-level performance and robustness indicators in Table 9.
- Wilcoxon signed-rank tests compare system-level performance across key baselines and planner configurations.The analysis covers both system comparisons and planner configurations.
- WORKFORCE significantly outperforms Single Agent and Role Playing baselines, with p-values of < 0.0001 and 0.0203, respectively.These results are reported as statistically significant rather than attributable to random variability.
- OWL-trained Qwen2.5-32B-Instruct significantly improves task planning effectiveness over the base model, with a Wilcoxon p-value of 0.0018.The improvement follows OWL-based reinforcement learning.
- Table 9 reports performance across capability requirements alongside robustness indicators including standard deviation and interquartile range (IQR).The complete capability-level performance table is identified as Table 9.
I.2 Planner vs. Worker Training
Training the planner alone outperforms training only workers and approaches the performance of training both components. The results support prioritizing planner optimization over worker-only specialization.
- Planner vs. Worker Training: 45.45% average accuracy from training only the planner exceeds 31.51% from training only workers.The comparison directly favors planner-only training.
- Planner vs. Worker Training: 46.68% average accuracy from training both components is only marginally above planner-only training at 45.45%.The reported difference between the two configurations is +1.23%.
- Planner vs. Worker Training: Training only workers degrades performance below the 36.36% baseline, indicating worker specialization without proper task decomposition can be counterproductive.The passage attributes the degradation to the absence of effective task planning and decomposition.
- Planner vs. Worker Training: The table compares planner-only, worker-only, and joint training configurations, with planner-only training nearly matching joint training while worker-only training performs worse.The caption identifies the central configuration-level comparison.