Source-linked AI summary
LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation
Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, Mengwei Xu
TL;DR
Static action-sequence evaluation can miss valid mobile-agent behavior, while human validation is difficult to scale. LlamaTouch evaluates on-device execution traces against annotated essential states and achieves nearly 80% accuracy in detecting completed tasks in real-world environments.
Problem
Static action-sequence matching misses valid paths and functionally correct actions, while human validation is difficult to reproduce and costly to scale.
Source: §1 · p. 2, §2.2 · p. 4Method
LlamaTouch runs mobile agents on realistic devices and checks their execution traces against manually annotated essential states.
Source: §1 · p. 2Results
Nearly 80% evaluation accuracy in detecting completed tasks in real-world environments, while prior action-based evaluation methods fail to do so.
Source: §1 · p. 2Takeaways & Limitations
Real-device testing also reveals the limited capabilities of current mobile agents in mobile UI automation tasks.
Source: §7 · p. 12Essential-state annotation can be biased or inaccurate when annotators have limited application knowledge or encounter unseen circumstances.
Source: §6 · p. 11
Abstract
from arXiv · showhide
The emergent large language/multimodal models facilitate the evolution of mobile agents, especially in mobile UI task automation. However, existing evaluation approaches, which rely on human validation or established datasets to compare agent-predicted actions with predefined action sequences, are unscalable and unfaithful. To overcome these limitations, this paper presents LlamaTouch, a testbed for on-device mobile UI task execution and faithful, scalable task evaluation. By observing that the task execution process only transfers UI states, LlamaTouch employs a novel evaluation approach that only assesses whether an agent traverses all manually annotated, essential application/system states. LlamaTouch comprises three key techniques: (1) On-device task execution that enables mobile agents to interact with realistic mobile environments for task execution. (2) Fine-grained UI component annotation that merges pixel-level screenshots and textual screen hierarchies to explicitly identify and precisely annotate essential UI components with a rich set of designed annotation primitives. (3) A multi-level application state matching algorithm that utilizes exact and fuzzy matching to accurately detect critical information in each screen, even with unpredictable UI layout/content dynamics. LlamaTouch currently incorporates four mobile agents and 496 tasks, encompassing both tasks in the widely-used datasets and our self-constructed ones to cover more diverse mobile applications. Evaluation results demonstrate LlamaTouch's high faithfulness of evaluation in real-world mobile environments and its better scalability than human validation. LlamaTouch also enables easy task annotation and integration of new mobile agents. Code and dataset are publicly available at https://github.com/LlamaTouch/LlamaTouch.
1 INTRODUCTION
Static action-sequence evaluation can reject valid mobile-agent behavior, while human validation is hard to reproduce and costly to scale. LlamaTouch evaluates on real devices by checking annotated essential states, with nearly 80% accuracy against human validation.
Motivation
- Static action matching misses valid alternative paths and self-correction, while human validation is difficult to reproduce and costly to scale.A single reference action sequence cannot cover all possible paths to task completion.Source: §1 · p. 2
LlamaTouch
- LlamaTouch runs agents on realistic mobile devices and evaluates recorded execution traces against manually annotated essential application states, ignoring non-essential operations.For example, opening Excel and reaching its login page are essential; installing the app is not.Source: §1 · p. 2
- Fine-grained annotation combines screenshots and textual view hierarchies, while multi-level matching applies exact and fuzzy checks to essential UI states.Approximate screen matching accommodates dynamic content, and mixed UI-state matching detects critical on-screen information.Source: §1 · p. 2
Dataset and testbed
- The dataset contains 496 tasks across popular Android applications, and its testbed supports integration of mobile agents into realistic environments.The testbed provides concise APIs intended to be compatible with most mobile agents.Source: §1 · p. 2
Evaluation
- Nearly 80% evaluation accuracy in detecting completed tasks in real-world environments, while prior action-based evaluation methods fail to do so.Human validation results serve as the ground truth.Source: §1 · p. 2
2 BACKGROUND AND MOTIVATION
Mobile UI agents interpret natural-language tasks using screen representations and produce interface actions. Existing benchmarks struggle to balance faithful evaluation with scalability because static action labels miss valid behavior, while human review is costly.
2.1 Agents for Mobile UI Task Automation
- Mobile UI agents use controllers to interpret instructions and UI context, then produce actions such as clicks, swipes, or text input.Their inputs can include screenshots or textual view hierarchies; action targets may be coordinates, icon markers, or HTML indices.Source: §2.1 · p. 3
2.2 Mobile UI Task Automation Benchmarks
- Existing mobile-agent benchmarks do not jointly achieve faithfulness and scalability, and human validation scales poorly across tasks, agents, and devices.Real-world execution environments alone do not inherently support essential-state matching.Source: §2.2 · pp. 3–4
- Static action datasets can misjudge functionally correct actions because parameters differ and can miss valid execution paths beyond their single reference sequence.The cited examples include wider clickable areas, equivalent search inputs, and tasks completed through different paths.Source: §2.2 · pp. 3–4
3 LLAMATOUCH DESIGN
LlamaTouch evaluates mobile agents in realistic mobile environments by checking annotated essential application states rather than requiring predefined action sequences. Its design combines on-device execution, fine-grained state annotation, and multi-level matching.
3.1 On-device Mobile UI Task Execution
- AgentEnv connects agents to realistic mobile environments, executes predicted actions, and records UI representations, actions, and system states for evaluation.The recorded trace includes screenshots, view hierarchies, foreground activity names, and actions.Source: §3.1 · pp. 4–5
3.2 Essential Application State Annotation
- Action-sequence matching can reject functionally correct parameter variations and cannot accommodate alternative task paths.LlamaTouch instead identifies overlapping essential application states across execution paths as task milestones.Source: §3.2 · p. 5
- LlamaTouch combines screenshots with textual view hierarchies to locate UI components and annotate essential states using primitives that specify what information to match.The primitives cover UI, system, and action states; annotations identify relevant elements and whether matching should be exact or fuzzy.Source: §3.2 · pp. 5–6
3.3 Faithful and Scalable Task Evaluation
- The evaluator processes essential states in sequence and marks a task complete only when every annotated state has a matching counterpart.It first approximately matches screens, then checks annotated states against components on the matched screens.Source: §3.3 · p. 6
- Activity matching filters for application screens, while fuzzy screen-info matching distinguishes functional pages when activity names are insufficient.Screen-info matching compares simplified HTML representations using sentence-embedding cosine similarity and a predefined threshold.Source: §3.3 · pp. 6–7
- On matched screens, LlamaTouch checks exact UI components, fuzzy textbox content, annotated actions, and supported system states.Click-action parameters use XPath targets, and system-state checking currently includes whether an application is installed.Source: §3.3 · pp. 7–8
- The evaluator reports task metrics from execution traces while aiming to retain the accuracy of real-world evaluation and the scalability of static-dataset evaluation.Agents need only define how to load task traces for evaluation.Source: §3.3 · p. 8
4 LLAMATOUCH DATASET
LlamaTouch combines sampled AITW tasks with newly generated tasks, recording interaction traces and annotating essential states. The resulting dataset spans diverse applications, task lengths, actions, and state-matching types.
4.1 Dataset Construction
- The dataset combines 102 sampled AITW tasks from 26 apps with 394 newly generated tasks from 46 apps.The generated tasks span categories including utilities, social media, and web shopping.Source: §4.1 · p. 8
- The dataset contains 496 tasks across 57 unique Android applications, with diverse task complexities.Uncertain annotation decisions were resolved by consensus among at least three annotators.Source: §4.1 · p. 8
4.2 Dataset Statistics
- AITW tasks average 7.35 completion steps versus 5.67 for LlamaTouch-generated tasks; overall, tasks average nearly 7 steps and range from 2 to 42.Table 3 reports dataset statistics and task complexity by average completion steps.Source: Table 3 · p. 8, §4.2 · p. 8
- All 496 tasks involve clicks; 292 involve swipes and 147 involve input, with more than 4 click actions per sample on average.The average number of each other action type is less than 2.Source: §4.2 · p. 8, Table 4 · p. 9
- Essential states for 441 tasks appear on one UI representation, compared with 53 on two and 2 on three.This indicates that most tasks check only the final state after completion.Source: §4.2 · p. 8, Table 5 · p. 9
- Exact UI-component matches account for 51% of essential states and exact activity matches for 36%; 418 of 496 tasks have at least one of these types.Table 5 reports the annotated essential-state statistics.Source: §4.2 · pp. 8–9, Table 5 · p. 9
- Fuzzy UI-component matching occurs in 44 tasks, with 51 such matches; fuzzy screen-info matching occurs in 31 tasks.The dataset includes all types of essential states described by the authors.Source: §4.2 · p. 9
5 EVALUATION
The evaluation compares task completion and evaluation accuracy against human validation, finding that LlamaTouch better recognizes successfully completed tasks than static action matching. It also examines matching primitives and agent performance across task sources and complexity levels.
5.1 Experiment Setup
- Experiments use an Android emulator and a Pixel 5 for incompatible apps, with four agents and human validation as the ground truth for comparing evaluation approaches.The four evaluation approaches include step-wise action match, LCS-based action match, LlamaTouch, and human validation.Source: §5.1 · p. 9
5.2 Metrics
- Task completion rate is the fraction of dataset tasks evaluated as completed; evaluation accuracy is the fraction whose method outcome matches human validation.LlamaTouch marks a task complete when its execution trace passes through all annotated essential states.Source: §5.2 · pp. 9–10
5.3 Task Completion Rate and Accuracy
- LlamaTouch achieves 79% average accuracy on human-validated successful traces, while step-wise and LCS action matching reach no more than 0.1%; its task completion rate is 8.67%.Across all tasks, the action-matching approaches have nearly 0% TCR, while LlamaTouch’s TCR closely aligns with human validation.Source: Table 7 · p. 10, §5.3 · p. 10, Table 6 · p. 10
5.4 Ablation Study
- Without exact-match primitives, LlamaTouch’s accuracy on all tasks falls from 95% to 19%; fuzzy matching shows no notable accuracy improvement with the evaluated agents.Activity matching contributes substantially, while exact matching for action and system state improves accuracy only slightly because few tasks use those primitives.Source: §5.4 · p. 10
5.5 Absolute Capabilities of Mobile Agents
- AutoUI and CoCo-Agent each reach 12.75% TCR on AITW, but only 2.29% and 2.31%, respectively, on generated tasks, indicating limited adaptation to unseen tasks.The authors attribute the gap mainly to differences in task complexity between datasets.Source: Table 9 · p. 11, §5.5 · p. 11
- TCR drops as tasks require more steps, and AutoDroid outperforms the other agents at every difficulty level; mobile agents still have substantial room to improve.AppAgent achieves a TCR similar to AutoDroid across all tasks.Source: §5.5 · p. 11
6 LIMITATIONS OF LLAMATOUCH
LlamaTouch’s evaluation is limited by its reliance on Android’s View Hierarchy and by possible biases in annotating essential states. The paper identifies expert review and diverse execution paths as ways to refine those annotations.
6 LIMITATIONS OF LLAMATOUCH
- Some WebView-based apps cannot be evaluated because they cannot provide Android’s View Hierarchy, which LlamaTouch requires for UI matching.The authors suggest incorporating screen-similarity detection or OCR/model-powered screen-element recognition when WebView app evaluation is required.Source: §6 · p. 11
- Human or autonomous annotators may miss application states or circumstances, making essential-state annotations inaccurate.For example, a task to delete YouTube may already be complete when the app is not installed, even without an uninstall-button action.Source: §6 · p. 11
- Expert review and diverse essential states across potential execution paths could refine annotations and improve evaluation robustness.The authors note that annotation biases might lower task-completion accuracy.Source: §6 · p. 11
7 CONCLUSION
LlamaTouch evaluates mobile agents on realistic devices by matching execution traces to annotated essential states. It tolerates different task paths and dynamic environments, with evaluation accuracy comparable to human validation while retaining static-dataset scalability.
7 CONCLUSION
- Matching execution traces to annotated essential states lets LlamaTouch tolerate different task paths and dynamic environments, reducing false negatives from previous evaluation approaches.LlamaTouch enables testing mobile agents in realistic mobile environments.Source: §7 · p. 11
- LlamaTouch achieves high evaluation accuracy comparable to human validation while preserving the scalability of evaluation on static datasets.The authors also report that real-device task execution reveals current mobile agents’ limited mobile UI automation capabilities.Source: §7 · p. 12
- Real-world mobile-device task execution reveals the limited mobile UI automation capabilities of current agents.Source: §7 · p. 12
A.1 APIs Provided by AgentEnv
AgentEnv provides APIs for mobile-agent integration and UI automation task execution on real devices. Table 11 lists the APIs, their parameters, and their return values.
A.1 APIs Provided by AgentEnv
- AgentEnv provides three categories of APIs for mobile-agent integration and UI automation task execution.Source: §A.1 · p. 13
- The APIs support mobile-agent integration and task execution on real mobile devices.Source: §A.1 · p. 13
- Table 11 lists the APIs together with their parameters and return values.Source: Table 11 · p. 13, §A.1 · p. 13