PaperlayerView exact revision on arXiv

Source-linked AI summary

CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents

Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, Guohao Li

arXiv:2407.01511v4cs.AI

TL;DR

Interactive GUI-agent benchmarks commonly focus on single platforms and often evaluate only final task success, despite tasks that span devices and require intermediate-progress assessment. CRAB introduces graph-based task construction and evaluation for cross-environment tasks; on its benchmark, a single GPT-4o agent achieved the best overall completion ratio of 38.01%.

  • Problem

    Interactive GUI-agent benchmarks typically focus on a single platform, and final-state-only evaluation can miss agents’ incremental progress.

    Source: §1 · p. 3, §4.3 · p. 8

  • Method

    CRAB constructs cross-environment tasks from graph-structured subtasks and evaluates progress through a graph of intermediate checkpoints.

    Source: §1 · p. 3, §3.2 · p. 6, §4.3 · p. 9

  • Results

    38.01% was the best overall completion ratio, achieved by the single-agent GPT-4o configuration on CRAB Benchmark-v0.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Cross-environment tasks reflect multi-device scenarios and require agents to process messages and transfer information between environments.

    Source: §4.1 · p. 6

    Cross-platform scenarios intensify context-length constraints as screenshots accumulate, while coordinate-grounding tools may miss screen elements in complex GUIs.

    Source: §6.3 · p. 15

Abstract

from arXiv · show

The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and the complexities of constructing tasks and evaluators. To overcome these limitations, we introduce Crab, the first agent benchmark framework designed to support cross-environment tasks, incorporating a graph-based fine-grained evaluation method and an efficient mechanism for task and evaluator construction. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging Crab, we developed a cross-platform Crab Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated four advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%. All framework code, agent code, and task datasets are publicly available at https://github.com/camel-ai/crab.

1 Introduction

Existing interactive GUI-agent benchmarks typically focus on one platform, while cross-platform tasks and evaluation of intermediate progress remain limited. CRAB provides graph-based task construction and evaluation, and its 120-task benchmark’s best completion ratio was achieved by a single GPT-4o agent.

1 Introduction

  • Existing task-completion benchmarks typically evaluate one platform, limiting coverage of tasks that span devices such as a smartphone and PC.The comparison framework organizes benchmark features around interaction, multimodal observation, cross-platform support, evaluation, and task construction.Source: §1 · p. 3, Table 1 · p. 4
  • CRAB uses a graph-based approach to construct diverse tasks and evaluate both final outcomes and intermediate milestones.The figure describes selecting subtasks to form a graph, then using it to construct an evaluator and generate instructions.Source: Figure 1 · p. 2, §1 · p. 3
  • 38.01% was the best overall completion ratio, achieved by the single-agent GPT-4o configuration.CRAB Benchmark-v0 contains 120 real-world tasks across Android and Ubuntu desktop environments.Source: §1 · p. 3

2 Related Work

Prior GUI-agent work spans operating systems, web pages, and Android, alongside benchmarks for web, mobile, and desktop tasks. CRAB’s related-work discussion highlights the need for cross-environment evaluation and more flexible treatment of task-step relationships.

GUI agents

Benchmarks for GUI agents

  • GUI-agent benchmarks cover web, mobile, and computer operating systems, with goal-based and trajectory-based evaluation among the approaches described.The passage also identifies intermediate-reward evaluation in Mobile-env.Source: §2 · pp. 4–5
  • Existing benchmarks often overlook that sequential task steps can sometimes be reordered without changing success, while CRAB emphasizes cross-environment evaluation.The paper notes that this scenario is more complex and underexplored, and that graph evaluation requires greater implementation effort while offering flexibility and extensibility.Source: §2 · p. 5

3 Definitions

The definitions formalize devices as environments and tasks across them, then introduce a graph representation for decomposed subtasks. Together, they specify the task setting and the structure used to express subtasks and their dependencies.

3.1 Problem Formulation

  • A cross-environment task combines a set of reward-free POMDP environments with a natural-language instruction and reward function.An agent system may use one agent for planning and action-taking or multiple agents connected through a communication workflow.Source: §3.1 · p. 5
  • An agent system is modeled as a policy choosing an action and environment from the instruction, action history, and observations across environments.Each agent uses a fixed backend MLM and system prompt and retains its chat history.Source: §3.1 · p. 5

3.2 Graph of Task Decomposition

  • The Graph of Decomposed Tasks (GDT) represents subtasks as a DAG whose nodes specify an environment, instruction, and reward function, and whose edges encode sequential relationships.The decomposition can include subtasks with sequential and parallel connections.Source: §3.2 · p. 6

4 The Crab Framework

CRAB represents cross-environment tasks and their dependencies explicitly, then evaluates progress through checkpoints rather than requiring one fixed action sequence. Its framework also composes tasks from subtasks and tracks completion and efficiency.

4.1 Cross-environment Task

  • Cross-environment tasks require operations across multiple GUI environments, where agents plan actions, produce outputs, and retain information to transfer.The task is formalized over a set of environments, natural-language instructions, and a reward function; systems may use one agent or communicating agents.Source: §4.1 · pp. 6–7

4.2 Task Generation

  • GDT represents environment-specific subtasks as a DAG, with defined inputs and outputs supporting dependency links and task composition.A subtask output can serve as another subtask's input, establishing a prerequisite edge; GPT-4 drafts task descriptions from prompts for human review.Source: §4.2 · pp. 7–8

4.3 Graph Evaluator

  • Graph Evaluator checks ordered intermediate-state checkpoints, retaining required steps while allowing multiple valid execution paths.Each checkpoint has a binary verification function, and dependent checks activate after prerequisites are verified.Source: §4.3 · pp. 8–9
  • Completion Ratio measures completed subtask nodes, while Execution Efficiency and Cost Efficiency relate that ratio to actions and model tokens, respectively.The metrics are defined as CR = C / N, EE = CR / A, and CE = CR / T.Source: §4.3 · p. 9

5 The Crab Benchmark

CRAB Benchmark-v0 combines Android and Ubuntu environments with tasks spanning individual and cross-platform use. Its benchmark includes task-specific evaluators that assess intermediate or final environment states.

Environments

  • The benchmark uses an Android emulator and Ubuntu virtual machine, with screen-image observations and distinct action spaces for each environment.The action-space table groups actions by Ubuntu, Android, and actions relevant across environments.Source: §5 · pp. 9–10, Table 2 · p. 10

Tasks

  • 120 tasks include 29 Android, 73 Ubuntu, and 18 cross-platform tasks; 104 were composed from subtasks and 16 were made by hand.The benchmark uses 17 Android and 19 Ubuntu subtask templates, each linked to a graph evaluator and verified by at least two field experts.Source: §5 · pp. 10–11

Evaluators

  • Its 59 evaluator functions use platform-specific state checks, including Android UI-layout XML and Ubuntu image matching or OCR; tasks average 4.2 evaluators.For time-varying tasks, crawler scripts capture values at evaluation time for comparison with the agent's result.Source: §5 · p. 11

6 Experiments

The experiments introduce a 120-task benchmark spanning Android, Ubuntu, and cross-platform tasks, then compare six MLMs across single- and multi-agent systems. GPT-4-series models achieve the highest average success rates and completion ratios, while multi-agent systems perform slightly worse than single-agent systems.

6.1 The CRAB Benchmark

  • CRAB Benchmark-v0 contains 120 tasks—104 generated by sub-task composition and 16 hand-built for more complex scenarios—spanning 29 Android, 73 Ubuntu, and 18 cross-platform tasks.Each subtask template is linked to a graph evaluator with 1 to 4 nodes and verified by at least two related field experts.Source: §6.1 · p. 12

6.2 Baseline Agent System

  • The baselines combine six MLMs with single-agent, functionality-split, and environment-split systems that divide planning and action execution differently.The backend models include four commercial and two open-source MLMs.Source: §6.2 · pp. 12, 14

6.3 Result

  • GPT-4o and GPT-4 Turbo achieved the highest average success rates and completion ratios among the tested models.GPT-4o had higher cost efficiency than GPT-4 Turbo; disabling tool calling reduced GPT-4o performance, primarily through more invalid actions.Source: Table 3 · p. 13, §6.3 · pp. 14–15
  • Multi-agent systems performed slightly worse than single-agent systems, with communication-related information loss linked to misunderstandings and more invalid actions.Multi-agent settings also used more chat rounds and had lower cost efficiency.Source: §6.3 · p. 15
  • A 4.67% completion-ratio difference separated GPT-4o single-agent and environment-split systems despite equal success rates.The authors present completion ratio as a way to distinguish methods that success rate alone does not separate.Source: §6.3 · p. 15
  • Cross-environment challenges include action-space discrepancies, context limits as screenshots accumulate, imperfect element grounding, and icon-recognition failures.Platform-specific agents may mitigate action-space confusion, while complex GUI observations can still challenge GroundingDINO and OCR.Source: §6.3 · p. 15

7 Conclusion

CRAB provides graph-based task generation and evaluation for cross-environment tasks in smartphone and desktop settings. Its benchmark module structure links environment definitions with tasks composed from subtasks and evaluator subgraphs.

7 Conclusion

  • CRAB introduces graph-based task generation and evaluation intended to reduce task-design effort and provide more dynamic, accurate agent assessments.The authors present CRAB Benchmark-v0 as a set of cross-environment tasks in smartphone and desktop environments.Source: §7 · p. 16

A Benchmark Detail

  • The appendix provides documentation for setting up experiment environments and reproducing results.It also directs readers to Figure 4 for the benchmark module structure.Source: A Benchmark Detail · p. 24
  • Figure 4 depicts two environments and multiple tasks; tasks compose subtasks and evaluator subgraphs, while environments specify attributes including observation and action spaces.Arrows show compositional relationships between tasks and subtasks.Source: Figure 4 · p. 24

A.1 Dataset statistics

The dataset spans applications and tasks across platforms, with individual tasks often involving multiple applications and graph evaluators sized according to task complexity.

A.1 Dataset statistics

  • Each task uses 1.84 applications on average, and a single task often involves two or more applications.The dataset covers daily-life, programming, and office-work scenarios across two platforms.Source: §A.1 · p. 25
  • Task evaluator graphs range from 1 to 11 nodes, with more complex tasks involving larger graphs..Source: §A.1 · p. 25
  • Table 4 lists applications and their task counts for the Ubuntu environment.Source: Table 4 · p. 25

A.2 Environment Implementation Detail

The appendix describes CRAB’s virtualized environments, modular action-and-evaluator framework, module configuration, and JSON representation for composed tasks.

A.2 Environment Implementation Detail

  • Ubuntu runs in a QEMU/KVM virtual machine and Android in the Google Android Emulator, with platform-specific tools providing input and screen capture.Ubuntu interaction uses PyAutoGUI and MSS; Android interaction uses ADB.Source: §A.2 · p. 25
  • Android evaluators inspect XML UI layouts, while Ubuntu evaluators use image matching and OCR; crawlers capture changing values for evaluation.The dataset has 59 evaluator functions, averaging 4.2 evaluators per task.Source: §A.2 · pp. 25–26

A.3 Framework Design

  • CRAB defines actions as executable Python functions and evaluators as boolean-returning actions that can be linked into graphs for complex tasks.Actions, environments, tasks, and evaluations are decoupled to support a plug-and-play architecture.Source: §A.3 · pp. 26–27

A.4 Configuration by Modules

  • Decorators configure action names, descriptions, parameters, and implementations; observation actions can return screenshots, and task evaluators combine checks in a graph.Subtask templates specify attributes, expected output types, and evaluator generators for composing compatible subtasks.Source: §A.4 · pp. 28–31

A.5 Composed Task Format

  • Composed tasks are saved in JSON with task IDs, descriptions, subtask attribute values, and an adjacency-list graph structure.The task dataset combines a Python-defined subtask pool with platform-categorized composition files.Source: §A.5 · p. 31

B.1 Agent Implementation

The appendix describes a shared, API-tailored agent architecture, then specifies communication strategies and prompt structures for single-agent and multi-agent setups. It also previews platform, configuration, and case-study analyses.

B.1 Agent Implementation

  • Agents share a common architecture tailored to each provider API, with initialization parameters, environment-specific action schemas, system messages, and chat handling for requests and responses.Shared utilities support parsing, prompt generation, and schema conversion; multi-environment agents associate actions with their intended environments.Source: §B.1 · p. 32

B.2 Inter-agent Communication Strategies

  • Functionality-based communication divides work between a main agent that instructs and a tool agent that selects an action, parameters, and target environment.Environment-based communication uses a main agent and three tool agents for Android, Ubuntu, and Root, each with its own action space.Source: §B.2 · p. 33

B.3.1 Single Agent

  • The single-agent prompt asks one agent to inspect screenshots, explain its plan, and execute the task through the supplied action space.The prompt requires action function calls and, in its fuller form, one JSON object per action.Source: §B.3.1 · pp. 33–34

B.3.2 Multi-Agent by Functionality

  • The functionality-based prompt has the main agent specify actions, parameters, and environment, while a tool agent translates instructions into function calls.The tool agent is prompted to return function calls without additional text.Source: §B.3.2 · p. 34

B.3.3 Multi-Agent by Environment

  • The environment-based prompt assigns the main agent to plan and direct sub-agents, while each sub-agent acts only in its designated environment or skips its turn.Root and sub-environment agents receive distinct prompts and action responsibilities.Source: §B.3.3 · p. 35

C Further Result Analysis

  • Further analysis groups results by Ubuntu, Android, and cross-platform tasks, and examines three individual tasks across agent settings.Source: C Further Result Analysis · p. 35

C.1 Result by Platforms

Results vary by platform and agent configuration: cross-platform tasks highlight the value of completion ratio, while multi-agent systems face communication challenges alongside some environment-specific advantages. The case study illustrates both successful handoffs and premature completion.

C.1 Result by Platforms

  • GPT-4o, Gemini, and Claude perform better on Android, whereas GPT-4 Turbo performs better on Ubuntu; false completion is prevalent on Android and step limits on cross-platform tasks.Claude has a high invalid-action ratio on cross-platform tasks, while GPT-4o with JSON mode has a high invalid-action ratio on Android.Source: §C.1 · p. 36

C.2 Comparison between Single Agent and Multi-agent

  • Multi-agent systems perform slightly worse than single agents, with the authors attributing lower SR partly to false completion after sub-agents misinterpret instructions amid communication information loss.The authors say this occurs less frequently in the multi-environment configuration.Source: §C.2 · pp. 36–37
  • On cross-platform GPT-4o tasks, the single agent has a higher invalid-action rate than the environment-based multi-agent, which can assign agents limited action spaces for different environments.Single agents may select a correct action for the wrong environment, or use an incorrect action or format.Source: §C.2 · p. 37, Table 9 · p. 38
  • The authors propose reducing communication information loss and adding a critical agent to correct errors, while noting that both approaches increase token costs.The framework's error logs can help users analyze agent bottlenecks.Source: §C.2 · p. 37

C.1 Result by Platforms

  • GPT-4 Turbo (Single) achieves a CR of 52.61% on cross-platform tasks, while Gemini, Claude, and open-source model agents have an SR of 0.0.The authors report that CR better distinguishes models on these tasks than SR.Source: §C.1 · p. 36, Table 9 · p. 38

C.3.1 Cross-platform Task

  • In the cross-platform case, the agent must retrieve an incomplete task on Android and apply it in Ubuntu; some GPT-4o multi-agent configurations stop after incorrectly completing the final step.GPT-4o and GPT-4 Turbo single agents, and GPT-4 Turbo by functionality, complete the task; Gemini and Claude agents fail at earlier stages.Source: §C.3.1 · p. 39

C.3.2 Ubuntu Task

The Ubuntu case study asks agents to create a directory and copy specified text files. GPT-4o and GPT-4 Turbo use Terminal commands successfully, while Gemini and Claude fail after taking different routes.

C.3.2 Ubuntu Task

  • GPT-4o and GPT-4 Turbo interpret the copy task and use Terminal commands to create the directory and copy the specified files.The authors report that both agents select a simpler solution and accurately use Terminal to meet the task requirements.Source: §C.3.2 · pp. 39–40
  • Gemini and Claude both fail, struggling to click and select the intended icons despite using the same visual prompting mechanism as the successful agents.Source: §C.3.2 · p. 40
  • Claude opens the Ubuntu Desktop Guide instead of Terminal, while Gemini opens Firefox before reaching Terminal and continues exploring unrelated applications and icons.Gemini tries alternative methods in Files, but neither agent achieves the task goal.Source: §C.3.2 · p. 40

C.3.3 Android Task

The Android case study follows agents tasked with finding a contact’s email in Contacts and sending a message through Gmail. Results show repeated app-navigation errors, misunderstanding, and step-limit terminations, alongside partial progress by some configurations.

C.3.3 Android Task

  • GPT-4o by functionality eventually finds the contact information but returns home before the operation-step limit ends the process.Source: §C.3.3 · p. 40
  • GPT-4 Turbo by functionality obtains the contact information, then repeatedly opens Chrome and Messages instead of Gmail.Source: §C.3.3 · p. 40
  • Gemini 1.5 Pro by functionality enters Contacts but misunderstands the instruction, gets lost creating a new contact, and never retrieves the requested information.Source: §C.3.3 · p. 40
  • Claude 3 Opus by functionality repeatedly opens Messages or Assistant before finding the contact, then the process ends at the operation-step limit as it plans to open Gmail.The authors’ analysis also notes that agents often tap the wrong app and run out of operations before sending the message.Source: §C.3.3 · pp. 40–41
  • GPT-4o by environment reaches Gmail and attempts to enter the retrieved address, while several functionality-based agents terminate before completing the message task.The task requires retrieving the contact information in one app and using it in another.Source: §C.3.3 · pp. 40–41

Related papers

Loading 2407.01511v4…