MMAC-Copilot: Multi-modal Agent Collaboration Operating Copilot

1 Introduction

• LLM agents’ reliance on a single text modality limits the environmental information they can acquire, while knowledge-domain hallucinations can lead to flawed planning.The introduction says these challenges persist despite prior scripting and planning approaches.Source: §1 · p. 1
• MMAC-Copilot uses a team collaboration chain in which agents refine an initial plan using their domain expertise.The Planner forms the initial plan, which is then iteratively refined during execution.Source: §1 · p. 1
• The framework combines agents for planning, information retrieval, coding, visual processing, video analysis, and interaction supervision.The Mentor reviews the interface after each interaction to check whether the desired outcome was achieved.Source: §1 · pp. 1–2
• 6.8% average improvement over previous methods on GAIA, with scores of 45.16, 20.75, and 6.12 on level-1, level-2, and level-3 tasks, respectively.VIBench evaluates non-API-interactable applications in 3D gaming, recreation, and office environments.Source: §1 · p. 2
• VIBench assesses interaction with non-API-interactable applications, including 3D gaming, recreation, and office scenarios.It complements existing benchmarks by focusing on visually complex user interfaces.Source: §1 · p. 2

2 Related work

• The review is organized around LLM-based agents and autonomous agents.These are the two domains named as the section’s focus.Source: §2 · p. 2

2.1 LLM-based agents

• LLM-based agents decompose complex tasks into subtasks, sequence actions, and use collaboration or competition among agents across application domains.Examples include AutoGPT and TaskWeaver (Qiao et al., 2023), LangChain Agent, and multi-agent systems such as AutoGen (Wu et al., 2023), MetaGPT (Hong et al., 2023a), and AutoAgents (Chen et al., 2023).Source: §2.1 · pp. 2–3

2.2 Autonomous agents

• Prior autonomous GUI agents use screenshots, OCR, or system inspection tools to support mobile or desktop application interaction.Examples include Wonderland (Yan et al., 2023), AppAgent (Yang et al., 2023a), MobileAgent, CogAgent (Hong et al., 2023b), and UFO (Zhang et al., 2024).Source: §2.2 · p. 3

3.1 Specialization of agents

• Six agents divide work across planning, retrieval, coding, visual interface interpretation, video analysis, and screenshot-based oversight.The Planner may leave visually dependent subtasks coarse for specialist agents to refine.Source: §3.1 · pp. 4–5

3.2 Communication Protocols

• Role-specific JSON schemas expose only required keys to each agent, with visual observations reserved for the Mentor and Viewer and shared with the Planner.The authors state that this standardization is designed to reduce ambiguity and miscommunication while supporting integration of new agents.Source: §3.2 · p. 5

3.3 Team Collaboration Chain

• The team collaboration chain coordinates dynamic planning and iterative task refinement to support execution of difficult tasks.Figure 2 illustrates the chain using a Discord messaging task.Source: §3.3 · p. 5, Figure 2 · p. 6

3.3.1 Dynamic Planning and Subtask Refinement

• The Planner starts with a coarse plan, then Viewer and Video Analyst use current visual context to refine subtasks into detailed, atomic actions.This refinement adapts execution to real-time changes in the visual interface.Source: §3.3.1 · pp. 5–6

3.3.2 Continuous Feedback and Iterative Improvement

• After each subtask, the Mentor analyzes screenshots and feeds structured feedback into the next planning round when discrepancies or incomplete tasks appear.When no adjustment is needed, the system proceeds to the next subtask or terminates after the last one.Source: §3.1 · p. 5, §3.3.2 · p. 6

4.1 Benchmark and setting

• VIBench evaluates GUI interaction in 3D gaming, recreation, and office environments, with tasks judged by whether the final state satisfies the user request.Unlike GAIA’s text and API-based interactions, VIBench focuses on applications that cannot typically be controlled via standard APIs.Source: §4.1 · pp. 6–7

4.2 Evaluation Method

• GAIA answers are evaluated by exact match, while VIBench task success is judged from recorded videos by application experts and fails if unfinished within 30 rounds.VIBench judges are instructed to assess achievement of the stated objective, disregarding irrelevant actions and intermediate steps.Source: §4.2 · p. 7

4.4 Results

• MMAC-Copilot scored 45.16 on GAIA Level 1, 20.75 on Level 2, and 6.12 on Level 3, outperforming closest competitor FRIDAY by 6.8% overall.It had the highest scores on Levels 1 and 2 and matched the highest Level 3 score.Source: Table 1 · p. 8, §4.4 · p. 8
• MMAC-Copilot outperformed other systems across VIBench categories, averaging 63.16% in 3D gaming, 69.23% in recreation, 78.57% in office, and 70.32% overall.VIBench targets complex GUI environments where direct API interaction is not possible.Source: Table 2 · p. 8, §4.4 · p. 8

4.3 Baseline

• For VIBench comparisons, the study reports UFO and a Windows-adapted version of FRIDAY; the GAIA performance table reports private-test-set results.The study also uses the same Vision model for UFO as for MMAC-Copilot.Source: §4.3 · p. 8, Table 2 · p. 8, Table 1 · p. 8

• MMAC-Copilot is presented as a framework for enhancing agent interactions within complex digital environments.Source: §5 · p. 8
• The conclusion states that MMAC-Copilot effectively addresses agent hallucination and supports reliable responses to environmental changes.Source: §5 · p. 8
• The paper reports superior performance over existing systems on GAIA and VIBench.Source: §5 · p. 8

Limitation

• MMAC-Copilot struggles with complex UI interpretation, 3D spatial tasks such as navigation and aiming, and tasks with higher temporal sensitivity.The authors attribute these limits to GUI-model understanding, gaps in spatial understanding, and longer inference time, respectively.Source: Limitation · p. 8
• The authors suggest online-search knowledge databases to add decision-making context for complex UI layouts and real-time game cues.Source: Limitation · p. 8

A Demo on Mac OS

• A macOS example follows a Spotify request from the Planner’s initial plan through Programmer launch and Viewer search and playback actions.Source: A Demo on Mac OS · p. 12

B Agent prompts

• The paper provides the Programmer’s complete prompts in Figures 9–13 and 15–18.Source: B Agent prompts · p. 12

C GAIA Detail Experiment Setting

• GAIA covers real-world scenarios involving reasoning, multimodality, coding, and web browsing, using text questions sometimes accompanied by files.Questions range from everyday tasks to scientific queries and require concise, verifiable answers.Source: C GAIA Detail Experiment Setting · p. 12

C.1 Levels of Difficulty

• GAIA’s three difficulty levels progress from tasks requiring minimal tool use and up to five steps to complex problems requiring extensive actions and multiple tools.Level 2 involves integrating multiple sources and about 5 to 10 steps.Source: §C.1 · p. 12
• Figure 5 presents sample GAIA questions spanning Levels 1 through 3.Source: §C.1 · p. 12

C.2 Evaluation Methodology

• GAIA requires answers in a specified format and verifies correctness by exact match against predefined ground truth.The paper describes this evaluation as automated, precise, factual, reliable, and efficient.Source: §C.2 · p. 12

C.3 Model Experimental Setup

• The comparison includes GPT-3.5, GPT-4, AutoGPT, human annotators, and traditional web search.Source: §C.3 · p. 12
• GPT-4 is evaluated for core linguistic and reasoning capabilities without external tools or plugins; before asking a question, a prefix prompt specifies the answer format to ease extraction.Source: §C.3 · p. 12
• GPT-4 with plugins has tools for data analysis, web browsing, and third-party APIs.Source: §C.3 · p. 12
• AutoGPT uses GPT-4 and selects plugins based on the query context, with tests of tool selection and adaptation to complex requirements.Source: §C.3 · p. 12
• Models are tested on structured tasks from simple to highly complex, with three replicated sessions; the Spotify example shows the Planner, Programmer, and Viewer action sequence.The search baseline reviews the first page of conventional search-engine results, while human annotators provide responses based on expertise.Source: §C.3 · pp. 12–13, Figure 4 · p. 13

D.1 VIBench Results List

• Table 3 lists VIBench requests alongside MMAC-Copilot’s performance across eight popular applications.The requests cover commonly used functions.Source: §D.1 · p. 13
• Follow-up requests are numbered sequentially, and the Success column marks completion with ✓ and failure with ✗.Source: §D.1 · p. 13

D.2 Visualization

• Figures 7 and 8 visualize VIBench with its requests.Source: §D.2 · p. 13

D.3 Adapt for Windows

• Windows compatibility required changes to tool integration and platform-specific libraries.Source: §D.3 · p. 13
• The adaptation added a Windows-configured Python interpreter and Shell support, incorporated Windows path handling, and replaced unsupported database libraries with alternatives.The authors also adapted for differences in concurrent-request management across Windows Python libraries.Source: §D.3 · p. 13
• Figure 5 presents sample GAIA questions, and Figure 6 is labeled as the system prompt for answering GAIA.The sample-question caption describes reasoning, multimodal, and tool-use competencies, with some questions accompanied by visual evidence.Source: Figure 5 · p. 14, Figure 6 · p. 14
• Figures 7 and 8 are labeled as visualizations of how VIBench works, Parts I and II.Source: Figure 7 · p. 16, Figure 8 · p. 17
• Figures 9–12 are labeled as Programmer Part prompts.Source: Figure 9 · p. 18, Figure 10 · p. 19, Figure 11 · p. 20
• The supplied captions identify Figures 14, 15, 16, 17, and 18 individually as prompts for Programmer Part, with each caption explicitly naming the corresponding figure and prompt type.Source: Figure 14 · p. 23, Figure 15 · p. 24, Figure 16 · p. 25