PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Mobile Device Control Agents across Diverse Configurations

Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, Kimin Lee

arXiv:2404.16660v3cs.HCcs.AIcs.LG

TL;DR

Mobile device control research lacks a unified benchmark covering practical tasks and varied configurations. B-MoCA provides an Android-based benchmark with 131 daily tasks and randomized environments, and evaluates foundation-model and demonstration-trained agents. These agents handle straightforward tasks but show limitations on complex tasks and generalization.

  • Problem

    Prior work often omits generalization across device configurations and practical daily tasks, and the field lacks a unified benchmark for these properties.

    Source: §1 · p. 1

  • Method

    B-MoCA is an Android benchmark with 131 realistic tasks, randomized device configurations, and evaluations of foundation-model and human-demonstration-trained agents.

    Source: §1 · pp. 1–2

  • Results

    Tested agents handle straightforward tasks, but foundation-model agents struggle with complex multi-interaction scenarios and behavior-cloned agents show limited generalization to unseen configurations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    B-MoCA provides a standardized platform for developing different types of mobile-device agents in a unified setting.

    Source: §6 · p. 9

    Foundation-model agents can hallucinate task completion and have difficulty with long-horizon tasks requiring multiple interactions.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Mobile device control agents can largely enhance user interactions and productivity by automating daily tasks. However, despite growing interest in developing practical agents, the absence of a commonly adopted benchmark in this area makes it challenging to quantify scientific progress. In this work, we introduce B-MoCA: a novel benchmark with interactive environments for evaluating and developing mobile device control agents. To create a realistic benchmark, we develop B-MoCA based on the Android operating system and define 131 common daily tasks. Importantly, we incorporate a randomization feature that changes the configurations of mobile devices, including user interface layouts and language settings, to assess generalization performance. We benchmark diverse agents, including agents employing large language models (LLMs) or multi-modal LLMs as well as agents trained with imitation learning using human expert demonstrations. While these agents demonstrate proficiency in executing straightforward tasks, their poor performance on complex tasks highlights significant opportunities for future research to improve effectiveness. Our source code is publicly available at https://b-moca.github.io.

1 INTRODUCTION

Mobile device control agents could support accessibility and productivity, but prior work lacks a unified benchmark spanning practical tasks and diverse device configurations. B-MoCA addresses this gap and evaluates agents that handle straightforward tasks better than complex ones.

1 INTRODUCTION

  • Existing benchmarks often overlook generalization across device configurations and practical daily tasks, leaving no unified basis for measuring progress.This gap matters because such tasks and generalization are important for deployment on real devices.Source: §1 · p. 1
  • B-MoCA is an Android-based interactive benchmark with customizable device configurations, including icon placement, size, wallpaper, language, and device type.Its open-ended environments include web access through Chrome, Wikipedia, Walmart, and Instagram.Source: §1 · p. 1, Figure 1 · p. 2
  • The benchmark defines 131 realistic tasks and uses rule-based success detectors to signal task completion during agent interactions.Tasks include opening applications, starting web searches, and changing device settings.Source: §1 · p. 1
  • The evaluation includes LLM- and MLLM-based agents and agents trained through behavior cloning on human demonstrations.The contributions also examine foundation-model design choices and the effect of data diversity on demonstration-trained agents.Source: §1 · p. 2
  • Tested agents handle straightforward tasks but struggle with complex scenarios; behavior-cloned agents imitate experts yet fail to generalize to unseen configurations.The authors present these shortcomings as opportunities for future research.Source: §1 · p. 2

2 B-MOCA

B-MoCA is an Android benchmark for common mobile-device tasks, designed to evaluate agents across realistic interactions and varied device setups. Its formulation supports multiple observation and action formats, while randomized training and test environments assess generalization.

2 B-MOCA

  • B-MoCA builds on Android and includes 131 tasks across default and third-party apps, each paired with a success detector.The tasks are selected for everyday prevalence and utility.Source: §2.1 · p. 2

2.1 DESIGN FACTORS

  • Device randomization varies setups such as icon placement, wallpaper, language, and device type to assess agents’ generalization.The benchmark also supports changes to icon size and dark mode.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 PROBLEM FORMULATION

  • The task is formulated as sequential decision-making: given an instruction, an agent observes the environment, acts, receives a success signal, and gets the next observation.Observations may be screen pixels, text descriptions of the UI, or both; supported actions include continuous and discrete forms.Source: §2.2 · p. 3

2.3 DAILY TASKS

  • The 131 realistic tasks exercise everyday skills, including setting alarms and enabling airplane mode, by requiring navigation and UI manipulation.Rule-based detectors use system information, app data, and UI attributes to check predefined completion criteria.Source: §2.3 · p. 3, Figure 3 · p. 4

2.4 ENVIRONMENT RANDOMIZATION

  • B-MoCA provides 45 randomized environments, split into 35 training and 10 test setups, with test environments containing unseen device configurations.Domain randomization is used to train agents across diverse configurations.Source: §2.4 · p. 4

3 BASELINE AGENTS

B-MoCA benchmarks foundation-model agents and custom agents trained with human demonstrations. The compared systems vary in their text and visual inputs, prompting, and training approaches.

3 BASELINE AGENTS

  • The benchmark covers LLM and MLLM agents as well as custom agents using fine-tuned open-source LLMs or VLM encoders trained with expert demonstrations.The LLM agents use text descriptions of screen layouts, while MLLM agents use text and visual inputs.Source: §3 · p. 4, §3.1 · p. 4

3.1 CLOSED-SOURCE MODELS

  • Text-based agents receive prompts with their role, action-space definition, goal, current observation, and previous actions, with optional few-shot examples.The prompts also use a Chain-of-Thought technique to enforce an output format.Source: §3.1 · p. 4

3.2 OPEN-SOURCE MODELS

  • Custom agents fine-tune open-source LLMs on expert actions or train VLM-encoder policies with behavior cloning to map task instructions and screen images to discrete actions.The VLM-based policies are optimized to imitate human expert demonstrations.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments evaluate foundation-model and custom agents across B-MoCA’s tasks and device configurations. Agents handle straightforward tasks better than complex ones, while randomization exposes generalization challenges.

4.1 EXPERIMENTAL SETUP

  • The experiments evaluate zero-shot closed-source LLMs on all 131 tasks and compare few-shot LLM/MLLM agents and custom agents on six long-horizon tasks.The six tasks require navigating multiple pages and manipulating diverse UI elements; results report means and standard errors across three runs.Source: §4.1 · pp. 5–6

4.2 MAIN RESULTS

  • Closed-source LLMs exceed 80% success on Alarm(simple), but performance declines as tasks become more complex.VLM-encoder custom agents reach at least 40% average success across the six challenging tasks, yet their generalization remains below 60%.Source: §4.2 · pp. 6–7, Figure 6 · p. 7
  • Closed-source foundation-model agents can hallucinate task completion and make errors in long-horizon interactions, while fine-tuned Llama-3 agents fail on Language and Wikipedia in most test environments.Fine-tuned Llama-3 completes Alarm tasks in test environments with high similarity to training environments; custom agents also struggle to generalize to Korean and tablet settings.Source: §4.2 · pp. 7–8
  • Additional image inputs yield marginal or detrimental gains for MLLM agents, while few-shot GPT-4o agents outperform open-source LLMs in the evaluated setting.The authors hypothesize that visual-domain gaps and longer input sequences may explain the limited benefit of image inputs.Source: §4.2 · p. 7

4.3 FURTHER ANALYSES

  • Randomized icon locations significantly reduce GPT-4o performance, while larger DPI changes are more disruptive than smaller ones in these environments.Language changes have limited impact on GPT-4o, which the authors attribute presumptively to multilingual training.Source: §4.3 · p. 8
  • Three few-shot examples improve success on some tasks, including Alarm(complex) and Call, but can also reduce performance, including on Wikipedia.Agents sometimes select the final action from an example trajectory instead of adapting it to the current UI.Source: §4.3 · p. 9, Table 2 · p. 9
  • On Language, custom-agent success rises from 23% with 7 training environments to 53% with 35.The reported results show a general increase in custom-agent performance as training environments are added.Source: Figure 8 · p. 8, §4.3 · p. 9

5 RELATED WORK

Related work covers open-ended digital environments and benchmarks for sequential decision-making, including device-control settings. B-MoCA adds randomized configurations to benchmark generalization.

Open-ended digital environment for autonomous agents

  • Digital interfaces differ from static games because their layouts, applications, and user interfaces change over time, requiring agents to adapt.Prior open-ended environments include games, websites, operating systems, and mobile devices.Source: §5 · p. 9

Benchmark for decision-making agents

6 CONCLUSION

B-MoCA provides everyday mobile tasks and environments simulating diverse device configurations. The authors present it as a unified platform for developing and evaluating different agent types.

6 CONCLUSION

  • B-MoCA combines everyday tasks with environments simulating numerous device configurations.The conclusion points readers to Appendix I for future directions and limitations.Source: §6 · p. 9
  • The authors report extensive experiments and describe B-MoCA as a standardized platform for developing different agent types in a unified setting.They hope the platform and its resources will support future breakthroughs in practical mobile device control.Source: §6 · p. 9

Appendix:

  • The cited title describes benchmarking a mobile device control agent across diverse configurations.Source: Appendix · p. 14

A.1 ENVIRONMENT IMPLEMENTATION AND INTERFACE

B-MoCA runs interactive Android environments from configured emulator snapshots and provides observation and action interfaces for agents. Its environments vary across device configurations, while the interface supports image or text observations and continuous or discrete actions.

A.1 ENVIRONMENT IMPLEMENTATION AND INTERFACE

  • B-MoCA environments run on Android emulators, with each environment stored as a snapshot after app installation and configuration.Randomized features include icon locations, DPI, wallpapers, and language.Source: §A.1 · p. 14
  • Observations can provide a screen image, an XML-derived text description of UI elements, or both.The screen translator parses the Android view hierarchy into tagged element descriptions; bounding-box coordinates can be included or excluded.Source: §A.1 · p. 14, Table 3 · p. 15
  • Agents act through continuous dual-gesture coordinates or discrete taps, swipes, and navigation-button actions.The discrete action vector for agents trained from scratch has 378 tap locations, four swipe directions, and three navigation buttons.Source: §A.1 · pp. 14–15

A.2 TRAINING AND TEST ENVIRONMENTS CONFIGURATIONS

  • Training uses Pixel 3 environments, while evaluation spans Pixel 3, Pixel 4, Pixel 4 XL, Pixel 6, and WGXA Tablet.The benchmark prepares 35 training and 10 test environments with varied device and display configurations.Source: §A.2 · p. 16, Table 4 · p. 17, Table 5 · p. 18

A.3 EXEMPLES OF UI ELEMENTS CHANGES IN THE RANDOMIZED ENVIRONMENTS

  • Randomized DPI changes UI shape and interaction, while language changes app-list ordering and text-based UI descriptions.The examples show how randomized features alter representative UI elements and challenge agents to manipulate them.Source: §A.3 · p. 20, Figure 9 · p. 20, Table 7 · p. 20

B.1 LIST OF DAILY TASKS

B-MoCA covers 131 everyday tasks across common application categories and uses task-specific criteria to evaluate completion. The appendix also illustrates successful demonstrations on six representative tasks.

B.1 LIST OF DAILY TASKS

  • B-MoCA presents 131 daily tasks that exercise common mobile-control skills, including manipulating icons, checkboxes, and sliders.The tasks are designed for diverse environments and cover commonly used applications.Source: §B.1 · p. 21
  • Tasks are grouped into System, Web/Shopping, Communication, Utility, and Event categories according to their applications.Multi-application tasks are categorized by their major subtask.Source: §B.1 · p. 21
  • Task completion is evaluated against criteria derived from significant environment changes observed in human demonstrations.The criteria draw on system information, app data, and UI-element attributes; detectors check conditions such as specified system logs, stored values, and UI states.Source: §B.1 · p. 21

B.2 EXEMPLARY DEMONSTRATIONS ON REPRESENTATIVE TASKS

  • Six representative tasks illustrate navigation and manipulation of elements such as checkboxes, dial pads, and time pickers.The demonstrations distinguish swiping actions from tapping actions.Source: §B.2 · p. 30, Figure 10 · p. 31

C GUIDELINES OF USING B-MOCA FOR CONTINUAL LEARNING

B-MoCA is presented as a testbed for open-ended and continual learning, including transfer across tasks and configurations. An example shows knowledge transfer on some tasks but continued difficulty with a more complex scenario.

C GUIDELINES OF USING B-MOCA FOR CONTINUAL LEARNING

  • B-MoCA supports study of open-ended learning and knowledge transfer across tasks and device configurations.Web access introduces changing contexts, and the benchmark can evaluate transfer from agents trained on multiple tasks.Source: C GUIDELINES OF USING B-MOCA FOR… · p. 32
  • Gemini-1.5-pro demonstrates meaningful knowledge transfer on tasks 4 and 5, but struggles with the more complex task 6.The comparison includes prior task experience in the prompt versus no experience.Source: C GUIDELINES OF USING B-MOCA FOR… · p. 32, Table 16 · p. 32

D.1 OVERVIEW OF AGENTS EMPLOYING FOUNDATION MODELS

  • The foundation-model workflow prompts an LLM with task instructions and text observations, then converts its selected text action into an emulator action.The prompt offers actions from a defined set, and the environment processes the converted action.Source: §D.1 · p. 33, Figure 11 · p. 33

D.2 PROMPT DETAILS FOR AGENTS WITH FOUNDATION MODELS

  • Foundation-model prompts combine general instructions with task-specific goals, observations, action history, and optional few-shot examples.Llama-3 examples may use sampled observation-action pairs rather than full trajectories because of context-length limits.Source: §D.2 · pp. 33–34

D.3 ARCHITECTURE DESIGN FOR CUSTOM AGENTS TRAINED WITH VLM ENCODER

Custom VLM-encoder agents fuse visual screen features with text task features to predict device actions. The appendix describes how demonstrations are collected and used for training and few-shot prompting.

D.3 ARCHITECTURE DESIGN FOR CUSTOM AGENTS TRAINED WITH VLM ENCODER

  • Custom agents encode screen images and task instructions separately, fuse their features with cross-attention, and predict a screen-touching action.The action vector comprises 378 tap locations, four swipe directions, and three navigation buttons.Source: §D.3 · p. 34

E.1 DEMONSTRATION DETAILS

  • Human experts collected demonstrations after familiarizing themselves with the emulator, completing six representative tasks in each training environment.Failed demonstrations are excluded, and few-shot examples use discrete actions unless continuous actions are necessary.Source: §E.1 · p. 36
  • The experiments use 210 demonstration trajectories from 35 training environments, while data-diversity experiments compare 42 and 210 trajectories from 7 and 35 environments.Closed-source foundation-model agents receive whole expert trajectories as few-shot examples, whereas Llama-3 receives sampled observation-action pairs.Source: §E.1 · p. 36, §E.2 · p. 36

E.2 INFERENCE CONFIGURATION DETAILS FOR AGENTS WITH FOUNDATION MODELS

  • Foundation-model inference uses three runs with fixed settings for temperature and top-p, and closed-source runs use fixed seeds for consistent output generation.The closed-source models do not provide deterministic generation by default.Source: §E.2 · p. 36

E.3 TRAINING CONFIGURATION DETAILS FOR CUSTOM AGENTS

Custom agents were trained from 210 human demonstrations using either fine-tuned Llama-3 or a VLM encoder, with distinct training schedules and hardware.

E.3 TRAINING CONFIGURATION DETAILS FOR CUSTOM AGENTS

  • Fine-tuned Llama-3-8B-Instruct used 15 epochs, batch size 8 with 8-step gradient accumulation, and 210 human demonstrations.Training used AdamW at 1e-6 with cosine annealing, LoRA rank 8 and alpha 16, and took about three hours on 8 RTX 3090 GPUs.Source: §E.3 · pp. 36–37
  • VLM-encoder policies trained for 10K steps with batch size 32 on 210 human demonstrations, taking about one hour on one RTX A6000 GPU.Training used Adam at 3e-4 with cosine annealing.Source: §E.3 · p. 37

F.1 ANALYSIS OF FAILURE CASES OF LLM AGENTS

The appendix examines failures and performance across LLM, MLLM, imitation-trained, and reinforcement-learning agents, highlighting planning, interaction, and generalization challenges.

F.1 ANALYSIS OF FAILURE CASES OF LLM AGENTS

  • GPT-4o failures include inaccurate plans, forgotten task history, hallucinated UI elements, repeated ineffective taps, and incorrectly formatted actions.Examples span multi-app alarm setup, Wikipedia feed customization, calling a number, navigating around an occluding navigation bar, and Calculator input.Source: §F.1 · pp. 38–41

F.2 PERFORMANCES OF LLM AGENTS USING LLAMA-3-70B ON ALL TASKS

  • Llama-3-70B shows reasonable zero-shot performance across tasks, but performance degrades as test environments apply more randomizations.The authors also report less detailed task descriptions than GPT-4o agents.Source: §F.2 · p. 42, Table 24 · p. 42, Table 25 · p. 42

F.3 PERFORMANCES OF CUSTOM AGENTS USING LLAMA-3 IN INTERACTIVE ENVIRONMENTS

  • Fine-tuned Llama-3 agents achieve high training-dataset action accuracy but limited interactive success and fail to recover from errors.The authors use this contrast to argue for real-system evaluation rather than relying on behavior-cloning accuracy alone.Source: §F.3 · p. 42, Figure 17 · p. 42

F.4 PERFORMANCES OF CUSTOM AGENTS USING VLM ENCODER IN TRAINING ENVIRONMENTS

  • VLM-encoder agents’ Language-task success falls from above 90% in training environments to below 60% in test environments.The authors interpret the training-test gap as remaining headroom for generalization across device configurations.Source: §F.4 · p. 43, Figure 18 · p. 43

F.5 PERFORMANCES OF MLLM AGENTS WITH AND WITHOUT FEW-SHOT EXAMPLES

  • Few-shot examples help GPT-4o MLLM agents on some tasks, such as Alarm(complex), but can reduce performance on others, such as Alarm(Simple).The comparison includes GPT-4o LLM agents with and without few-shot examples.Source: §F.5 · p. 43, Table 26 · p. 43

G AGENTS TRAINED WITH REINFORCEMENT LEARNING

  • The reinforcement-learning analysis trains policies using a success signal and examines whether reward shaping improves efficiency.The passage introduces the analysis but does not report its outcomes here.Source: G AGENTS TRAINED WITH REINFORCEMENT… · p. 44

G.1 EXPERIMENTAL SETUP

The study trains DDQN and PPO agents on selected mobile tasks, evaluates sparse success rewards, and explores dense reward shaping for longer-horizon behavior.

G.1 EXPERIMENTAL SETUP

  • DDQN uses a double-Q temporal-difference objective with target-network updates and ϵ-greedy exploration, while PPO uses a clipped surrogate objective and generalized advantage estimation.The agents share a VLM-based visual and text encoder backbone; evaluations include Gmail and Language tasks under sparse rewards and Call 911 under dense rewards.Source: §G.1 · p. 44

G.2 RESULTS WITH RL TRAINING

  • RL agents learn to open Gmail in training environments and improve test performance, but fail on longer tasks and cannot complete the Language task even during training.The authors attribute the Language-task difficulty mainly to sparse success feedback arriving only after completion, which does not guide exploration from scratch.Source: §G.2 · p. 44, Figure 19 · p. 45

G.3 EFFECT OF REWARD SHAPING

  • Dense Call 911 rewards let DDQN progress through task steps to full return in training environment 000, but it fails in test environment 100.The six-step reward is based on human-expert task completion; the authors note that designing such dense rewards is effortful because it requires task-step UI understanding.Source: §G.3 · pp. 44–45, Figure 20 · p. 45

H ADDITIONAL RELATED WORK

  • Related work spans foundation models for decision-making and approaches to assistive device control using structural UI information, vision-based learning, prompting, and fine-tuning.The paper positions its analysis as a study of foundation-model abilities for mobile-device control and methods for understanding and manipulating UI elements.Source: H ADDITIONAL RELATED WORK · p. 46

I LIMITATIONS AND FUTURE DIRECTIONS

The authors identify limits in text-entry tasks, evaluating open-ended tasks, and the benchmark’s exploration of foundation-model agents.

I LIMITATIONS AND FUTURE DIRECTIONS

  • Text-entry tasks are constrained because entering text through the on-screen keyboard requires excessively long interactions.The authors plan to add web search and email tasks using advanced interfaces.Source: I LIMITATIONS AND FUTURE DIRECTIONS · p. 46
  • ADB-based success detection does not capture behavior semantics, making tasks with ambiguous success criteria hard to evaluate.The authors suggest learned reward models from demonstrations as a possible way to integrate open-ended tasks.Source: I LIMITATIONS AND FUTURE DIRECTIONS · p. 46
  • The benchmark leaves unexplored agentic workflows and engineered prompts for foundation-model agents.The authors note that LLMs could serve as high-level planners using APIs or as low-level actors implemented with neural policies.Source: I LIMITATIONS AND FUTURE DIRECTIONS · p. 46

Related papers

Loading 2404.16660v3…