PaperlayerView exact revision on arXiv

Source-linked AI summary

Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction

Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, Kai Yu

arXiv:2305.08144v4cs.AI

TL;DR

GUI benchmarks often lack trustworthy, reproducible evaluation and support for realistic interaction. Mobile-Env provides a controlled Android benchmarking toolkit with intermediate rewards and instructions, and its evaluations find that even advanced models struggle on tasks relatively simple for humans.

  • Problem

    GUI benchmarks need more trustworthy and reproducible evaluation, while existing interactive environments lack intermediate rewards and instructions for interactive learning and realistic user intent.

    Source: §1 · p. 2

  • Method

    Mobile-Env is an Android toolkit for controlled GUI evaluation that supports intermediate rewards and instructions, and supplies open-world and fixed WikiHow task sets.

    Source: §1 · p. 2

  • Results

    Across the benchmark evaluations, even advanced models struggle with tasks relatively simple for humans.

    Source: §1 · p. 2, §5.2 · p. 7, §5.3 · p. 8

  • Takeaways & Limitations

    The results underscore the need for more capable foundation models and more effective, robust GUI agent frameworks augmented with GUI understanding and operation knowledge.

    Source: §6 · p. 9

Abstract

from arXiv · show

The Graphical User Interface (GUI) is pivotal for human interaction with the digital world, enabling efficient device control and the completion of complex tasks. Recent progress in Large Language Models (LLMs) and Vision Language Models (VLMs) offers the chance to create advanced GUI agents. To ensure their effectiveness, there's a pressing need for qualified benchmarks that provide trustworthy and reproducible evaluations -- a challenge current benchmarks often fail to address. To tackle this issue, we introduce Mobile-Env, a comprehensive toolkit tailored for creating GUI benchmarks in the Android mobile environment. Mobile-Env offers an isolated and controllable setting for reliable evaluations, and accommodates intermediate instructions and rewards to reflect real-world usage more naturally. Utilizing Mobile-Env, we collect an open-world task set across various real-world apps and a fixed world set, WikiHow, which captures a significant amount of dynamic online contents for fully controllable and reproducible evaluation. We conduct comprehensive evaluations of LLM agents using these benchmarks. Our findings reveal that even advanced models (e.g., GPT-4V and LLaMA-3) struggle with tasks that are relatively simple for humans. This highlights a crucial gap in current models and underscores the importance of developing more capable foundation models and more effective GUI agent frameworks.

1 Introduction

Recent LLM/VLM progress motivates GUI agents, but trustworthy evaluation requires benchmarks that handle alternative task trajectories and uncontrolled environments. Mobile-Env provides an Android toolkit and task sets for controlled evaluation, where experiments show that advanced models still struggle with tasks relatively simple for humans.

Benchmark gap

  • Existing interactive benchmarks can misjudge successful alternative trajectories, while open evaluations lack isolation and controls, making results unreproducible and incomparable.Static datasets and interactive benchmarks may match action sequences rather than task outcomes; open environments also expose real-world devices and changing online content.Source: §1 · p. 2
  • Qualified GUI benchmarks need intermediate rewards for interactive learning and progressive instructions that reflect how user intent develops during a task.A single comprehensive instruction can be unnatural, and users may clarify their intent as new information emerges.Source: §1 · p. 2, Figure 2 · p. 3

Mobile-Env

  • Mobile-Env is an Android toolkit for isolated, controllable evaluation that supports intermediate rewards and instructions for reinforcement learning and conversational agents.Its framework is designed to make reliable benchmark construction easier.Source: §1 · p. 2

Evaluation

  • Experiments on 74 real-app tasks and a 150-task fixed WikiHow set find that advanced models struggle with tasks relatively simple for humans.The fixed set uses dynamic online content in a controlled and reproducible setting.Source: §1 · p. 2

2 Background

GUI interaction is modeled as an agent acting on a system with hidden states and intents, using observations and optional instructions to select actions. Qualified benchmarks should judge outcomes reliably while providing controlled environments and support for intermediate rewards and instructions.

2.1 Problem formulation

  • The GUI interaction formulation treats the user and GUI system as an environment with hidden intents and states, observed through screens and optional instructions as the agent acts.Actions change the system state, and optional rewards can assess transitions until task success or failure.Source: §2.1 · p. 3

2.2 Qualified benchmarks

  • Reliable evaluation should assess whether the task goal state is reached, using diverse signals because no single signal captures every relevant transition.Mobile-Env combines screen text, View Hierarchy, system logs, and the agent’s response to the user.Source: §2.2 · p. 4
  • Controllable environments fix and replay dynamic online content to support reproducible, fair comparisons and reduce risks to real devices and data.The passage also describes reverse-engineered solutions for replay servers and app certificate unpinning.Source: §2.2 · p. 4
  • Mobile-Env produces intermediate rewards with a listener-style evaluator to address reward sparsity in GUI tasks with complex observations and large action spaces.The rewards are designed to be provided safely during interaction.Source: §2.2 · p. 4
  • Progressive intermediate instructions let users clarify intent as tasks unfold and support evaluation of conversational GUI agents.The paper describes Mobile-Env as the first interactive environment to support intermediate instructions.Source: §2.2 · p. 4

3 Mobile-Env platform

Mobile-Env provides a uniform framework for hosting GUI environments and building benchmarks. It combines flexible Android interaction with controlled setup and signal-based state estimation to support reliable evaluation, intermediate rewards, and instructions.

3 Mobile-Env platform

  • Mobile-Env provides a uniform framework for hosting GUI environments and building qualified GUI benchmarks.The section focuses on its use features, with internal implementation details left to supplementary materials.Source: §3 · p. 4

3.1 Universal hosting of GUI environments

  • The platform hosts real-world Android apps with screenshots, View Hierarchy observations, and fine-grained touch, lift, and text actions.Wrappers can modify the action space, and environments can run open-world or controlled with multi-turn instructions.Source: §3.1 · p. 5

3.2 Creation of qualified GUI benchmarks

  • Benchmark setup, evaluation methods, and reward mechanisms are configured through an external Protocol Buffer task file.The platform’s benchmark methodology includes isolated, controllable environments and task-state estimation for evaluation, rewards, and instructions.Source: §3.2 · p. 5

3.2.1 Setup of isolated & controllable environments

  • Mobile-Env prepares tasks with a predefined Android OS image and configured ADB commands, launching a replay server when controlled online content is needed.Certificate pinning can block replay, so the authors implemented and verified three reverse-engineering-based solutions.Source: §3.2.1 · p. 5

3.2.2 Accurate task state estimation

  • Mobile-Env estimates task state from screen text, View Hierarchy, system logs, and user-response signals, combining them conjunctively, disjunctively, or with order constraints.A listener-style evaluator captures signals without disrupting runtime state or agent actions, enabling safe evaluation and intermediate rewards and instructions.Source: §3.2.2 · pp. 5–6, Figure 3 · p. 5

4 Task sets

Mobile-Env supports an open-world set of tasks in real apps and a fixed WikiHow set for controlled, reproducible evaluation. The sets cover app-based and cross-app tasks as well as WikiHow navigation, in-page operations, and article-based question answering.

4 Task sets

  • The task collection comprises an open-world set and a fixed controlled-world set.The two sets are used to evaluate GUI interaction tasks under different environment conditions.Source: §4 · p. 6

4.1 Open world set

  • The open-world set contains 74 tasks in real apps on the open Internet, covering system settings, four popular apps, and cross-app navigation.The apps include WikiHow, Ele.me, Douban, and IMDB; the set is intended to evaluate agents’ raw performance.Source: §4.1 · p. 6

4.2 Fixed world set (WikiHow)

  • The fixed WikiHow set uses 107,448 crawled pages and 856,045 resources to create 150 tasks from 13 templates for controlled evaluation.Template instructions are rewritten with ChatGPT, with multiple candidates randomly substituted at runtime.Source: §4.2 · pp. 6–7
  • WikiHow tasks comprise 59 cross-page navigation tasks, 51 in-page operation tasks, and 40 article-based QA tasks.QA tasks ask agents to summarize answers from article contents.Source: §4.2 · p. 7

5 Experiments

Across open- and fixed-world evaluations, current LLMs and VLMs achieve low success, with especially weak VLM results and task-dependent challenges. Experiments also compare prompting formats and test screenshot grounding with Set-of-Marks.

5.1 Experiment settings

  • Experiments covered closed-source and open-source LLMs and VLMs; text agents received simplified HTML, while vision agents received screenshots with Set-of-Marks.Average episode reward and success rate were reported as metrics.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 Results on open world set

  • 2.70% for Claude-3-Opus and 3.04% for GPT-4V; open-world results are low across models, while LLaMA-3-70B reaches 40.54% versus text GPT-4 at 43.24%.The open-world comparison appears in Table 4.Source: §5.2 · p. 7, Table 4 · p. 7
  • Text LLMs often outperformed VLMs on open-world tasks, which the authors link to VLM difficulty interpreting screen text, icons, and functions.Information-intensive screens were especially challenging for VLMs.Source: §5.2 · p. 7
  • Over half of GPT-4 and Claude-3-Opus predictions contained redundant newlines, while 26.40% of GPT-4V and 10.87% of Claude-3-Opus outputs contained thoughts without actions.The authors identify strict output-format following as a performance bottleneck.Source: §5.2 · p. 8

5.3 Results of text LLMs on fixed world set

  • 47.33% was the highest success rate for text LLMs on the fixed WikiHow set, leaving performance limited even in a reproducible setting.Table 5 presents results on the WikiHow task set.Source: §5.3 · p. 8, Table 5 · p. 8
  • QA tasks were harder than other task types because agents had to extract article content from HTML, while in-page tasks often demanded long-range scrolling or dropdown checks.The article's HTML interleaves text and markup, as illustrated in Figure 4.Source: §5.3 · p. 8, Figure 4 · p. 8
  • Single-turn prompting dramatically degraded GPT-3.5 performance relative to multi-turn prompting, which separates observation and action into user and assistant roles.The authors speculate this role distinction clarifies the input-output relation for the model.Source: §5.3 · pp. 8–9

5.4 Results of VLMs on fixed world set

  • 10.00% success was the best VLM result on 40 fixed-world tasks, and Set-of-Marks mitigated screenshot-grounding difficulty.The method asks VLMs to predict the index of a tagged screen element instead of precise coordinates.Source: §5.4 · p. 9

6 Conclusion

The conclusion presents Mobile-Env as a toolkit for building qualified GUI benchmarks and reports that current models remain deficient as GUI agents. It argues for more capable foundation models and more effective, robust agent frameworks.

6 Conclusion

  • Mobile-Env provides an isolated interactive environment and a flexible framework for reliable intermediate rewards and instructions.The conclusion frames it as a toolkit to simplify construction of qualified benchmarks.Source: §6 · p. 9
  • Two task sets built with Mobile-Env support insights into LLM-agent performance and reproducible, comparable evaluation.The cited VLM results table covers WikiHow tasks and varies Set-of-Marks and exemplar counts.Source: §6 · p. 9, Table 6 · p. 9
  • Experiments reveal current models' deficiencies as GUI agents and motivate more capable foundation models and more effective, robust agent frameworks augmented with GUI knowledge.Source: §6 · p. 9

Checklist

The checklist points readers to supplementary material for limitations and societal impacts, reports that experiments were not duplicated because of budget limits, and identifies released software and data assets.

Checklist

  • The authors direct readers to the supplementary material for limitations and potential negative societal impacts.Source: Checklist · p. 13
  • The paper reports that existing and newly curated assets are cited and their licenses mentioned, with links to code, data, and reproduction prompts.Source: Checklist · p. 13
  • Duplicated experiments were not conducted because of budget limits.Source: Checklist · p. 13

A Distribution & License

  • Mobile-Env and the WikiHow task set are open-sourced under Apache-2.0, with repositories and distribution links identified in the paper.The experiment code is also open-sourced.Source: A Distribution & License · p. 13

B Implementation details of Mobile-Env

Mobile-Env combines an Android simulator with task managers that provide task-specific instructions, rewards, and completion checks. Its implementation also supports flexible state estimation and reproducible WikiHow data replay.

B Implementation details of Mobile-Env

  • Task managers supply intermediate instructions and rewards, then signal success or failure; one simulator can host multiple managers while only one is active per episode.Agents receive simulator observations and task events, then send actions to the OS through the simulator.Source: B Implementation details of Mobile-Env · p. 14

B.1 Mobile-Env simulator

  • The simulator wraps Android Emulator, exposing screenshots and View Hierarchy XML plus TOUCH, LIFT, and TEXT actions; gRPC handles screenshots and touch actions, while ADB handles VH and text.The paper notes that ADB operations have much longer delay than the gRPC operations.Source: B Implementation details of Mobile-Env · p. 14, §B.1 · p. 15

B.2 Certificate unpinning

  • Certificate pinning is addressed through three reverse-engineering solutions: installing proxy certificates in a writable system image, runtime verifier replacement, or APK repackaging.The authors report that most tested common-use apps can be handled by at least one solution.Source: §B.2 · p. 15

B.3 Task state estimation

  • State estimation detects screen text and icons with external recognition systems and checks expected View Hierarchy nodes or system-log patterns.Text patterns use OCR, icons use a class or reference image, and VH and logs are acquired through ADB.Source: §B.3 · p. 15
  • A tree of AND/OR nodes expresses conjunctive or disjunctive signal checks, while skip connections encode event-order constraints for task state estimation and reward generation.The strategies are represented in task configuration files.Source: Figure 6 · p. 15, §B.3 · p. 16

B.4 Miscellaneous auxiliary tools

  • Auxiliary tools collect human demonstrations for behavior cloning and generate task configurations from templates, including sequential multi-stage tasks.The WikiHow fixed-world task set was built with the template toolkit.Source: §B.4 · p. 17

C.1 Instructions of open world set

  • The open-world instructions include varied app tasks, such as installing apps from F-Droid and recording bills in OpenMoneyBox.Examples include replacing Google Authenticator with Aegis Authenticator and installing VLC or MPV.Source: §C.1 · p. 18

C.2 App data crawling and replay for WikiHow task set

  • WikiHow replay stores 856,045 resources covering 107,448 distinct pages; Pyserini generates search-result pages dynamically because keyword combinations cannot feasibly be crawled exhaustively.Static resources are replayed from saved responses, while search pages use an app-derived template and an index of article titles.Source: §C.2 · pp. 19–20

C.3 Automatic task generation for WikiHow task set

The WikiHow task set is generated by combining operation templates with article and QA data into multi-stage tasks. Rewritten instructions and runtime variation diversify how those tasks are phrased.

C.3 Automatic task generation for WikiHow task set

  • Ten WikiHow operations are encoded as single-stage templates, then concatenated into 150 multi-stage tasks averaging 2.68 single-stage subtasks.The tasks include navigation, rating, sharing, bookmarking, and article-based question answering.Source: §C.3 · pp. 20–21
  • Task templates sample article details or QA answers into slots and concatenate stages only when the next target is on or referenced by the current page.QA tasks draw on wikihow-lists question types and WikiHowNFQA summarization questions.Source: §C.3 · pp. 20–21

C.4 Instruction rewriting & dynamic replacement for WikiHow task set

  • Colleagues and ChatGPT rewrite stereotyped instructions, and public expression patterns are randomly sampled at runtime with pattern branches and slot words adjusted for part of speech.The authors inspected rewritten instructions before summarizing transferable patterns.Source: §C.4 · p. 21

D.1 LLM access and compute resources

The appendix specifies the tested agents, their observations and action interfaces, prompt settings, and additional open-world analyses. It also describes performance difficulties when agents must launch apps from the Android homepage.

D.1 LLM access and compute resources

  • The evaluated agents include closed- and open-source LLMs and VLMs, deployed through provider APIs or on A10 GPUs; the Android Emulator runs on a Linux host with KVM.The listed open-source families include LLaMA 2, AgentLM, and LLaMA 3.Source: §D.1 · pp. 21–22

D.2 Observations for LLM/VLM-based agents

  • Text LLMs receive visible-leaf View Hierarchy elements converted to simplified HTML, while VLMs receive screenshots marked with detected text and icons.EasyOCR detects screen text, and vision-ui detects icons for the VLM inputs.Source: §D.2 · pp. 22–23

D.3 Action spaces for LLM/VLM-based agents

  • Agent actions such as CLICK, TYPE, INPUT, SCROLL, and SLIDE are translated into TOUCH, LIFT, and TEXT operations for Mobile-Env.CLICK and TAP use three TOUCH actions, long-click variants use ten, and sliding uses consecutive TOUCH actions followed by LIFT.Source: §D.3 · p. 23, Figure 10 · p. 24

D.4 Hyperparameter selection

  • Open-source models use two-shot in-context learning because two exemplars fit their 4K-token context; the sampling temperature is 0.1, and episodes exceeding 15 steps count as failures.The temperature balances structured action generation with free-form reasoning and reproducibility with exploration.Source: §D.4 · pp. 23–24

D.5 Prompts for LLM/VLM-based agents

  • The prompts require a THINK line followed by an ACTION line and define available VLM actions such as TAP, LONG_TAP, SCROLL, TYPE, ANSWER, and GOBACK.The appendix provides separate prompt tables for text LLMs and VLMs.Source: Table 12 · p. 24, §D.5 · pp. 25, 27, Table 13 · p. 27

D.6 More analyses on open world set

  • When tasks start from the Android homepage, most agents struggle to open the correct app, often failing to predict the upward swipe that reveals the app menu.The paper also notes that agents can confuse the Google icon with Gmail.Source: §D.6 · p. 28, Table 14 · p. 28
  • Other reported difficulties include unavailable View Hierarchy on some sensitive pages, model safety refusals, and GPT-4V exiting WikiHow unexpectedly.The paper notes that tasks on pages without VH may be completed only by visual models such as GPT-4V.Source: §D.6 · p. 28

D.7 More analyses & case study on fixed world set

The fixed-world analyses expose distinct weaknesses in output formatting, long-page navigation, and visual grounding. AgentLMs improve over LLaMA-2 models on WikiHow, while GPT-3.5 chat and instruct models show different task-type strengths.

Failure of GPT-4 on QA tasks

  • GPT-4 often inserts an extra newline between thought and ANSWER, violating the required output format and becoming a significant performance bottleneck.The behavior is illustrated in Figure 11.Source: §D.7 · p. 28

Capability difference of chat and instruct models

  • GPT-3.5 chat and instruct models have similar average success rates, 37.33% vs. 36.67%, but favor different task types: chat leads on in-page tasks and instruct on cross-page tasks.The reported cross-page rates are 38.98% for GPT-3.5 and 44.07% for GPT-3.5-instruct; in-page rates are 49.02% and 35.29%, respectively.Source: §D.7 · p. 28, Figure 12 · p. 29

Performances of AgentLMs

  • AgentLM models improve markedly over their LLaMA-2 counterparts on WikiHow, with the authors attributing their strength mainly to better output-format following and fewer invalid actions.The comparison is reported in Table 15, and Table 16 reports lower invalid-action ratios than LLaMA 2 models.Source: §D.7 · p. 29, Table 15 · p. 29, Table 16 · p. 30

Failure of AgentLM-7B on QA tasks

  • AgentLM-7B fails completely on QA tasks, often producing invalid actions or repeating the question as its answer despite having a suitable summary in its thought.The examples are shown in Figure 13.Source: §D.7 · p. 29, Figure 13 · p. 30

Results on single-stage sub-tasks

  • Nearly all models fail on WikiHow rating and author/category-to-article subtasks, which require sustained scrolling and often exceed the 15-step episode limit.Shortcuts exist, including opening the reference list near rating buttons or searching for an article, but few models use them.Source: §D.7 · pp. 29–30, Figure 14 · p. 30

Failure cases of VLMs

  • VLM agents can predict the plan but miss target coordinates, and may be distracted by notification popups absent from the View Hierarchy.In one example, a screen-pinning popup redirects the agent from the task goal.Source: Figure 16 · p. 30, §D.7 · p. 31

E Limitations

The authors identify unresolved contamination and SoM-preprocessor concerns, and call for broader benchmarks. They also describe the paper’s impact and revisions made in response to reviewers.

E Limitations

  • WikiHow QA may be answered from parametric memory rather than the underlying article, and the authors lack an efficient way to detect this contamination.They also report that SoM detector performance was not a bottleneck in their experiments and call for more environments and task sets.Source: E Limitations · p. 31

F Impact statement

  • Mobile-Env’s isolated environments prevent agents from directly performing vicious actions through the platform, to the authors’ knowledge.The fixed-world set uses publicly available WikiHow data, and the authors report no privacy data in it.Source: F Impact statement · p. 31

G Reviews from previous venue & revision

H Datasheet of fixed world task set

The fixed-world WikiHow task set is intended to make GUI evaluation reproducible and comparable. It comprises sampled public app data and task definitions, was collected by the authors, and is released openly for reuse and contribution.

H.1 Motivation

  • WikiHow provides a fixed-world benchmark intended to make GUI evaluation reproducible and comparable.Source: §H.1 · p. 32

H.2 Composition

  • The task set contains 150 task definitions, uses raw HTTP flows, and samples about 100,000 of WikiHow’s more than 340,000 articles.The task configurations specify goals, intermediate instructions, and rewards; URLs may duplicate resources, and article authors are identifiable from public pages.Source: §H.2 · pp. 32–33

H.3 Collection Process

  • The authors generated tasks from templates and crawled public WikiHow pages in breadth-first order from July 6 to August 18, 2022.They state that authors were not notified; they consider use of publicly available material under WikiHow’s terms and robots.txt to constitute consent, and provide no revocation mechanism, though authors can be contacted.Source: §H.3 · pp. 33–34

H.4 Uses

  • The paper evaluates agents based on different LLMs on the WikiHow task set.Source: §H.4 · p. 34

H.5 Distribution

  • Mobile-Env and the task set are publicly released under Apache-2.0, while copyright in WikiHow app data remains with the original authors and WikiHow.Source: §H.5 · p. 35

H.6 Maintenance

  • The authors welcome bug reports, feature contributions, and notification of new environments or task sets to update the available-task index.Source: §H.6 · p. 36

Related papers

Loading 2305.08144v4…