Source-linked AI summary
Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang, Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi
TL;DR
Complex agentic tasks require systems that can plan, act, observe, and recover across diverse environments. Magentic-One addresses this with an Orchestrator-led team of specialized agents and evaluates it with AutoGenBench, achieving statistically competitive results on three challenging benchmarks. Its main practical boundaries are evaluation focused on final accuracy and potentially high cost and latency.
Problem
Agentic systems need to generalize beyond specific domains to diverse complex tasks requiring multi-step action, observation, reasoning, and error recovery.
Method
Magentic-One uses an Orchestrator to plan, track progress, direct specialized web, file, coding, and terminal agents, and revise plans after stalls.
Results
Magentic-One is statistically competitive with state-of-the-art systems on GAIA, WebArena, and AssistantBench, with results of 38%, 32.8%, and 27.7%, respectively.
Takeaways & Limitations
The modular team and open-source implementation support a generalist agentic system spanning diverse open-ended tasks.
Takeaways & Limitations
Magentic-One requires dozens of iterations and LLM calls, potentially costing several US dollars and tens of minutes per task.
Abstract
from arXiv · showhide
Modern AI agents, driven by advances in large foundation models, promise to enhance our productivity and transform our lives by augmenting our knowledge and capabilities. To achieve this vision, AI agents must effectively plan, perform multi-step reasoning and actions, respond to novel observations, and recover from errors, to successfully complete complex tasks across a wide range of scenarios. In this work, we introduce Magentic-One, a high-performing open-source agentic system for solving such tasks. Magentic-One uses a multi-agent architecture where a lead agent, the Orchestrator, plans, tracks progress, and re-plans to recover from errors. Throughout task execution, the Orchestrator directs other specialized agents to perform tasks as needed, such as operating a web browser, navigating local files, or writing and executing Python code. We show that Magentic-One achieves statistically competitive performance to the state-of-the-art on three diverse and challenging agentic benchmarks: GAIA, AssistantBench, and WebArena. Magentic-One achieves these results without modification to core agent capabilities or to how they collaborate, demonstrating progress towards generalist agentic systems. Moreover, Magentic-One's modular design allows agents to be added or removed from the team without additional prompt tuning or training, easing development and making it extensible to future scenarios. We provide an open-source implementation of Magentic-One, and we include AutoGenBench, a standalone tool for agentic evaluation. AutoGenBench provides built-in controls for repetition and isolation to run agentic benchmarks in a rigorous and contained manner -- which is important when agents' actions have side-effects. Magentic-One, AutoGenBench and detailed empirical performance evaluations of Magentic-One, including ablations and error analysis are available at https://aka.ms/magentic-one
1 Introduction
Magentic-One is introduced as a generalist, modular multi-agent system with an Orchestrator that coordinates specialized agents and supports rigorous evaluation. Across GAIA, WebArena, and AssistantBench, it achieves strong task-completion or accuracy results that are statistically competitive with state-of-the-art systems.
- Magentic-One: Its modular design separates generally useful skills into reusable agents and supports adaptation and extensibility.
- AutoGenBench: AutoGenBench provides standalone, open-source evaluation with repetition, isolation, and controlled initial conditions for stochastic agents with side effects.
- Results: 38% task completion on GAIA, 32.8% on WebArena, and 27.7% accuracy on AssistantBench were statistically competitive with other state-of-the-art systems.Ablations and error analyses examine individual-agent contributions and identify opportunities for improvement.
- Magentic-One: Magentic-One combines five specialized agents with an Orchestrator that plans, maintains structured memory, assigns tasks, resets when stalled, and determines completion.Its agents include a Coder, Computer Terminal, File Surfer, Web Surfer, and Orchestrator.
2 Related Work
Related work covers single-agent systems using tools and prompting, multi-agent systems with modular roles, and benchmarks for general multi-step agent evaluation. Existing benchmarks often emphasize browser-based tasks.
- Single-Agent Approaches: Single-agent systems use LLM tools such as code execution and web browsing alongside prompting strategies for reasoning and planning.
- Multi-Agent Approaches: Multi-agent systems divide capabilities or roles across agents to provide a modular and flexible approach to complex tasks.
- Multi-Agent Approaches: Prior multi-agent systems include debate-based, hierarchical, and planning-oriented architectures, but differ from Magentic-One’s architecture.
- Agentic Evaluation: Agentic benchmarks evaluate general multi-step tasks using real websites, synthetic websites, browser interaction, and other settings.
3 Problem Setup
The paper defines complex tasks and agentic systems around iterative planning, action, observation, and reflection in partially observed computer environments. These tasks accept textual descriptions and optional file attachments, then return answers or environment states for evaluation.
- Complex Tasks: A complex task requires or significantly benefits from repeated planning, acting, observing, and reflecting across multiple steps.
- Complex Tasks: Task inputs contain a textual description and may include arbitrary file attachments such as images, datasets, audio clips, or PDFs.
- Agentic Systems: An agentic system operates a partially observed computer environment, using actions such as Python execution, web navigation, and local downloads within a time budget.
- Agentic Systems: After task completion, the system returns a text answer and action trace, while the environment’s final state is captured for evaluation.
4 Magentic-One Overview
Magentic-One coordinates specialized tool-using agents through an Orchestrator that maintains task and progress ledgers, plans actions, monitors progress, and revises plans after stalls. Its hierarchical workflow supports web, file, coding, and terminal operations across open-ended tasks.
- 4 Magentic-One Overview: The Orchestrator decomposes high-level tasks, plans work, directs specialized agents, tracks progress, and takes corrective actions.
- Outer Loop: The initial task ledger stores facts, facts to derive or look up, educated guesses, and a natural-language plan assigning steps to agents.
- Inner Loop: The inner loop creates a progress ledger by checking completion, repetition, forward progress, the next agent, and its instruction.
- Recovery: When stalling exceeds a threshold of ≤2 in experiments, the Orchestrator reflects, updates the task ledger, revises the plan, and starts another inner-loop cycle.
- Specialized Agents: WebSurfer, FileSurfer, Coder, and ComputerTerminal provide browser, file, coding, and shell capabilities for dynamic web and filesystem tasks.Capability decomposition gives the Orchestrator a smaller choice among broad skills while individual agents select lower-level actions.
- Specialized Agents: The team combines specialized tools with autonomous adaptation in dynamic web and file-system environments.
5 Experiments
Magentic-One is statistically competitive with state-of-the-art systems across GAIA, AssistantBench, and WebArena, while evaluation and ablation analyses expose important methodological boundaries and failure modes.
- 5.1 AutoGenBench and Setup: AutoGenBench controls repetition, isolation, and initial conditions to reduce stochastic and side-effect-related evaluation distortions.These controls address stateful-environment effects, such as later runs benefiting from artifacts created by earlier runs.
- 5.2 Results: Magentic-One achieves statistically comparable performance to state-of-the-art methods on GAIA and AssistantBench, while its GPT-4o variant is comparable to most WebArena systems.On WebArena, WebPilot and Jace.AI achieve statistically higher scores.
- 5.2 Results: 35.1% of WebArena validation tasks versus 30.5% of test tasks were completed correctly, suggesting mild overfitting after validation-based debugging.WebArena lacks a hidden test set, complicating fair comparison with leaderboard results.
- 5.2 Results: Magentic-One appears to compete better on hard, long, multi-step tasks than on easier tasks, consistent with hypothesized fixed overhead and added error opportunities.The authors identify improving performance on simpler tasks as an opportunity for future work.
- 5.3 Ablations: Removing the full Orchestrator ledgers reduces performance by 31%, while removing individual worker agents reduces performance by 21% to 39%.FileSurfer is especially important for attachment-heavy level 2 GAIA tasks, whereas WebSurfer is most important for level 1 tasks.
- 5.4 Error Analysis: The most common failure modes are persistent inefficient actions and insufficient verification, with inefficient navigation especially prevalent on WebArena.Agents may repeat unproductive behavior, complete tasks without adequate validation, or take inefficient paths through interfaces.
6 Discussion
Magentic-One’s multi-agent design supports performance, modular development, and extensibility, while its current evaluation, capabilities, learning, team composition, and safety controls impose important boundaries.
- 6.1 The Multi-Agent Paradigm: The centralized Orchestrator coordinates specialized agents, while the multi-agent design separates skills and supports development and reuse.The paper identifies multi-agent design as a principal contributor to performance and contrasts centralized orchestration with possible alternatives.
- 6.2 Limitations: Magentic-One’s evaluation focuses on final-answer accuracy, omitting cost, latency, user preference, and user value.The paper notes that partially correct trajectories can be valuable, while accurate answers may have little or negative value if delivered too late or at excessive cost.
- 6.2 Limitations: Most tasks require dozens of LLM iterations and calls, potentially costing several US dollars and tens of minutes per task.The authors identify smaller local models as one possible route to reducing these costs.
- 6.2 Limitations: Limited modality and action support prevents agents from handling some videos, document layouts, audio content, webpage interactions, and document types.Examples include inability to watch videos, inspect figures or layout, process non-speech audio, hover, or drag webpage elements.
- 6.2 Limitations: The fixed five-agent team can distract the Orchestrator when agents are unnecessary and lacks expertise when additional agents are needed.The paper suggests dynamically adding or removing agents according to task needs.
- 6.2 Limitations: Magentic-One discards trial-and-error insights between tasks, forcing repeated discovery of shared subtasks and increasing errors and costs.The authors identify long-term memory as a key direction for addressing this limitation.
- 6.3 Risks and Risk Mitigation: Autonomous computer operation introduces risks including phishing, social engineering, misinformation, alignment failures, and potential worker displacement.Mitigations include containers, synthetic environments, aligned models, filtering, monitoring, least privilege, and human oversight.
7 Conclusions
The conclusion presents Magentic-One as a generalist multi-agent system for open-ended file- and web-based tasks, combining Orchestrator-led planning with specialized tool-operating agents. It reports statistically competitive performance and releases both the system and an evaluation framework while acknowledging limitations and risks.
- 7 Conclusions: Magentic-One targets ad-hoc, open-ended, file- and web-based tasks through an Orchestrator directing four specialized agents.The Orchestrator plans, tracks progress, and recovers from errors using ledger-based orchestration.
- 7 Conclusions: Magentic-One is statistically competitive with state-of-the-art systems on three challenging benchmarks, demonstrating strong performance and generalization.The conclusion also summarizes the system’s open-source implementation and AutoGen-based event-driven framework.
- 7 Conclusions: The work discusses Magentic-One’s limitations and risks from generalist AI agents together with possible mitigation strategies.This frames the contribution alongside unresolved practical and safety boundaries.
A Statistical Methodology
The paper estimates uncertainty around benchmark accuracies with Wald 95% confidence intervals and compares Magentic-One with baselines using proportion z-tests.
- A Statistical Methodology: Table 1 reports each method’s mean and a Wald 95% confidence interval for its benchmark accuracy.The interval uses a normal approximation for the sample mean and is justified by the evaluated sample sizes and observed proportions.
- A Statistical Methodology: The authors use proportion z-tests to compare Magentic-One’s accuracy with each Table 1 baseline.McNemar’s test and pairwise t-tests are unavailable because the benchmarks do not expose task-level results.
B Capability to Category Mapping
The capability analysis reorganizes human annotations from GAIA into categories aligned with Magentic-One’s agent roles, including web browsing, coding, file handling, and no-tool capabilities.
- B Capability to Category Mapping: Figure 3 groups GAIA task performance by capability categories derived from and recoded from human annotator reports.The categories are intended to reflect the roles of Magentic-One’s agents.
- B Capability to Category Mapping: Web browsing covers searching and browsing the web, including browsers, search engines, maps, and internet archives.
- B Capability to Category Mapping: Coding covers coding and execution capabilities, including Python, calculators, audio/video processing, text processing, and natural language processing.
- B Capability to Category Mapping: File handling covers access to diverse document formats, including PDF, Word, Excel, PowerPoint, CSV, and XML files.
- B Capability to Category Mapping: No-tool capabilities are performed inherently by multimodal agents, including image recognition, OCR, computer vision, and color recognition.
C Error Analysis Code Book
The error analysis code book catalogs recurring execution failures, including unproductive persistence, weak verification, underused resources, navigation mistakes, communication problems, and security barriers, alongside successful completions.
- Code book overview: The validation-log analysis groups recurring failures into named codes covering verification, resource use, navigation, adaptation, communication, error handling, technical logic, access, and feedback.The code book defines codes and summarizes examples from validation logs across all benchmarks.
- Insufficient verification: Insufficient verification caused outputs and tasks to be accepted without checking data correctness or necessary criteria, producing unreliable outcomes.Examples include skipped dataset checks, unvalidated document scans, and unverified data interpretation.
- Resource use and persistence: Agents often underused available resources and persisted with manual or ineffective strategies, extending tasks despite APIs, search tools, or specialized agents.The logs describe bypassed APIs, overlooked advanced search, manual downloads despite FileSurfer, and repeated flawed paths.
- Navigation and technical execution: Navigation and technical errors led to missed targets, delays, syntax failures, misformatted queries, and incorrect parameter mappings.The examples include cycling through tabs, clicking incorrect navigation bars, indentation errors, and nonresponsive web functions.
- Coordination and feedback: Communication and delayed feedback produced duplicated work, unclear role boundaries, incomplete information, repeated restarts, and user uncertainty.Examples include simultaneous access to order histories, failed information handoffs, vague instructions, and delayed completion notifications.
- Security and successful completion: The logs also record security and access barriers, ignored warnings, and successful cases involving coordinated resource use, verification, and correct processing.Successful examples include verified data migration, matching financial reconciliation, improved processing speeds, smooth account creation, and compliant manuscript revisions.