LearnAct: Few-Shot Mobile GUI Agent with a Unified Demonstration Benchmark

1 INTRODUCTION

• Mobile GUI agents struggle in long-tail real-world scenarios because app interfaces and task requirements vary too widely for general training datasets to cover.The introduction cites 1.68 million Google Play applications as of 2025 and notes that unseen scenarios hinder adoption.Source: §1 · p. 2
• Few-shot demonstrations can bridge the personalization gap by teaching agents stable task patterns and adaptation strategies from user-specific examples.The approach targets repetitive tasks with variable elements, including smart-home control, health monitoring, and enterprise software.Source: §1 · p. 2
• LearnGUI is introduced as a dataset for studying few-shot demonstration learning, with 2,252 offline tasks and 101 online tasks paired with high-quality human demonstrations.It is built upon AMEX [5] and AndroidWorld [23].Source: §1 · pp. 2–3
• LearnAct combines DemoParser for extracting knowledge, KnowSeeker for retrieving relevant knowledge, and ActExecutor for using it during task execution.ActExecutor combines user instructions, the real-time GUI environment, and retrieved demonstration knowledge to perform tasks.Source: §1 · pp. 2–3
• Gemini-1.5-Pro [26] accuracy rose from 19.3% to 51.7% with one demonstration, while UI-TARS-7B-SFT [22] online task success rose from 18.1% to 32.8% with LearnAct.The Gemini-1.5-Pro improvement is reported as 198.9% relative; accuracy also rose from 14.1% to 69.4% in CityMapper and from 17.4% to 69.2% in To-Do apps.Source: §1 · pp. 2–3

Mobile GUI Datasets and Environments

• LearnGUI unifies few-shot offline and online evaluation with 2,252 offline tasks and 101 online tasks built on AMEX [5] and AndroidWorld [23].It supports varying k-shot configurations and controlled similarity conditions between support and query tasks.Source: §2 · p. 3
• LearnGUI adapts AMEX [5] for few-shot evaluation through standardized actions, similar-task k-shot combinations, and UI and action similarity measurements.For online evaluation, it adds 101 human demonstrations to AndroidWorld [23], which originally lacked human demonstration trajectories.Source: §2 · p. 3

Mobile GUI Agents

• Existing mobile GUI agents use prompting or GUI-specific fine-tuning, but both approaches struggle to generalize to unseen applications, novel layouts, and unexpected task variations.Fine-tuned models require substantial training data, while prompting approaches often struggle with complex tasks.Source: §2 · p. 4
• LearnAct learns and reasons from screenshots without UI trees, using specialized components to extract, retrieve, and apply demonstration knowledge in new scenarios.The framework comprises DemoParser, KnowSeeker, and ActExecutor.Source: §2 · p. 4

3.1 Task Definition

• LearnGUI frames mobile GUI interaction as a POMDP and studies demonstration-conditioned policies using within-application k-shot tasks with controlled support-query similarity.Its design examines k=1, 2, or 3 demonstrations and similarity across instruction, UI, and action dimensions.Source: §3.1 · p. 4

3.2 Data Collection

• LearnGUI combines LearnGUI-Offline for systematic few-shot evaluation with LearnGUI-Online for real-time interactive assessment, and both use a unified action space.The shared action space is intended to keep evaluation consistent across the two components.Source: §3.2 · p. 4

3.2.1 LearnGUI-Offline.

• LearnGUI-Offline restructures AMEX [5] for few-shot evaluation, producing 2,252 tasks with valid k-shot combinations by selecting the most similar same-application tasks as demonstrations, with average similarity above 0.6.UI similarity uses TF-IDF vectorization and cosine similarity; action similarity uses embedding-based cosine similarity between task pairs.Source: §3.2.1 · pp. 4–5

3.2.2 LearnGUI-Online.

• LearnGUI-Online adapts AndroidWorld [23], which has 116 task templates, by collecting human demonstrations for 101 tasks suitable for human completion and excluding 15 challenging tasks.For dynamic tasks, the authors generated specific instances and recorded corresponding demonstrations, creating a realistic testbed for few-shot evaluation.Source: §3.2.2 · p. 5

3.3 Dataset Statistics

• LearnGUI contains 2,353 instructions across 73 applications, averaging 13.2 steps per task, with k-shot combinations and instruction, UI, and action similarity measures.The range of similarity levels supports analysis of how demonstration quantity and similarity affect learning outcomes.Source: §3.3 · p. 5

3.4 Dataset Splits

• LearnGUI-Offline is split into 2,001 training tasks per k-shot configuration across 44 apps and 251 test tasks per configuration across 9 apps.LearnGUI-Online adds 101 real-time tasks across 20 applications; the offline similarity profiles enable analysis of knowledge transfer under differing UI and action similarities.Source: §3.4 · pp. 5–6

4.1 DemoParser

• DemoParser converts instructions, screenshots, and raw actions into semantically descriptive knowledge, adding task-relevant memory annotations for information needed in later steps.It preserves the original instruction and action sequence while describing each action’s purpose and context.Source: §4.1 · pp. 6–7

4.2 KnowSeeker

• KnowSeeker selects top-k knowledge entries relevant to the current instruction and precomputes knowledge-base embeddings to make retrieval more efficient as the base grows.It bridges DemoParser’s knowledge base and ActExecutor’s execution environment.Source: §4.2 · pp. 7–9

4.3 ActExecutor

• ActExecutor uses a vision-language model to choose actions from the instruction, current screenshot, action history, and retrieved demonstrations, adapting analogous examples to novel UI states.Its execution loop integrates demonstration knowledge into sequential decision-making and continues until task completion or the step limit.Source: §4.3 · pp. 9–10

5.1 Experiment Setup

• Evaluations covered offline step-by-step execution and online real-world task completion, using commercial and open-source models; online LearnAct models used one-shot retrieval.Gemini-1.5-Pro was tested directly offline, while UI-TARS-7B-SFT and Qwen2-VL-7B were fine-tuned on the offline training set; Gemini was excluded online after limited preliminary task completion.Source: §5 · p. 10, §5.1 · p. 10

5.2.1 Offline Agent Capability Evaluation.

• With one demonstration, Gemini-1.5-Pro rose from 19.3% to 51.7%, UI-TARS-7B-SFT from 77.5% to 82.8%, and Qwen2-VL-7B from 71.8% to 77.3%.Gemini-1.5-Pro reached 57.7% with three demonstrations; Qwen2-VL-7B reached 79.4% with three.Source: Table 4 · p. 11, §5.2.1 · p. 11
• Gemini-1.5-Pro improved monotonically as demonstrations increased, whereas UI-TARS-7B-SFT peaked with one demonstration.The results suggest additional examples benefit the less specialized model, while the GUI-fine-tuned model extracts value from minimal demonstrations.Source: §5.2.1 · p. 11
• Application-specific gains included Gemini-1.5-Pro rising from 14.1% to 69.4% on CityMapper and from 17.4% to 69.2% on To-Do apps.The paper reports particularly notable gains in these complex or nonstandard application scenarios.Source: §5.2.1 · p. 11
• Across similarity profiles, Gemini-1.5-Pro’s action-match gains ranged from +29.3% to +39.6%, UI-TARS-7B-SFT gained +13.9% in UISHActSH at 3-shot, and Qwen2-VL-7B gained +67.4% action-type accuracy in UISHActSH at 2-shot.Gemini-1.5-Pro’s +38.4% match-accuracy gain in UISLActSH at 3-shot indicates action similarity can sometimes compensate for UI differences; gains were greatest when perceptual and procedural knowledge were relevant.Source: §5.2.1 · pp. 11–12, Table 5 · p. 12

5.2.2 Online Agent Capability Evaluation.

• Online task success rose from 9.9% to 21.1% for Qwen2-VL-7B and from 18.1% to 32.8% for UI-TARS-7B-SFT with LearnAct.UI-TARS-7B-SFT’s 32.8% approached GPT-4o’s 34.5%.Source: Table 6 · p. 12, §5.2.2 · p. 12

5.3 Ablation Study

• The full framework reached 51.7% accuracy, versus 40.6% without DemoParser and 41.6% without KnowSeeker, showing that both components contribute.Relevant demonstrations improved performance by 21.3% over baseline without action descriptions; adding DemoParser’s structured action descriptions added 11.1%, while relevant retrieval added 10.1%.Source: §5.3 · pp. 12–13, Table 7 · p. 13

• 198.9% relative improvement for Gemini-1.5-Pro [26] accompanies gains across evaluated models; UI and action similarity both influence transfer, and action similarity can compensate for UI differences.The authors identify both similarity dimensions as important for successful knowledge transfer.Source: §6 · p. 13

Data Collection and Dataset Expansion

• LearnGUI’s 2,252 offline tasks and 101 online tasks remain limited relative to mobile-app diversity, motivating broader coverage of applications with complex interactions and specialized domains.The discussion describes the dataset as a significant step forward but says it needs greater diversity and representativeness.Source: §6 · p. 13

K-shot Learning Analysis

• Because the current k-shot study covers only k=1, 2, and 3, future work should examine how demonstration counts affect gains and identify optimal counts across scenarios and model architectures.Source: §6 · p. 13

Enhanced Learning and Execution Strategies

• Future strategies could better transfer knowledge between support and query tasks through pattern abstraction, critical-decision identification, and adaptation of demonstrated strategies to novel scenarios.The current approach retrieves relevant demonstrations, but the authors see opportunities to extract and transfer their knowledge more efficiently.Source: §6 · p. 13

Agent Self-Learning

• Agents could learn from their own successful task completions by adding those executions to the knowledge base, enabling continuous self-improvement beyond reliance on human demonstrations.Source: §6 · p. 13

• A single demonstration raised Gemini-1.5-Pro [26] accuracy from 19.3% to 51.7% offline and UI-TARS-7B-SFT [22] task success from 18.1% to 32.8% online.Source: §7 · p. 14
• LearnGUI, the first comprehensive dataset for demonstration-based learning in mobile GUI agents, comprises 2,252 offline tasks and 101 online tasks with high-quality human demonstrations.Source: §7 · p. 14
• LearnAct extracts demonstration knowledge with DemoParser, retrieves relevant knowledge with KnowSeeker, and uses ActExecutor for demonstration-enhanced task execution.Source: §7 · p. 14
• The findings position demonstration-based learning as a promising direction for more adaptable, personalized, and practically deployable mobile GUI agents.Source: §7 · p. 14

• Figure 8 presents LearnGUI-Offline similarity-score distributions for instructions, UI, and actions to support analysis of their effects on learning efficacy.Source: Figure 8 · p. 16

B LEARNACT FRAMEWORK DETAILS

• The framework-details section expands on the LearnAct methods presented in Section 4.Source: B LEARNACT FRAMEWORK DETAILS · p. 16

B.1 DemoParser Prompts

• DemoParser prompts guide a vision-language model to turn demonstrations into semantically descriptive action descriptions and structured knowledge.The templates are shown in Figures 9 and 10.Source: §B.1 · p. 16
• Figure 9’s intermediate-action template standardizes descriptions and specifies memory annotations for important information observed during task execution.Source: Figure 9 · p. 17

B.2 ActExecutor Prompts

• ActExecutor prompts use current observations, action history, and demonstration knowledge to guide task-execution decisions.The prompts are shown in Figure 11.Source: §B.2 · p. 17
• ActExecutor’s prompts guide the vision-language model to select appropriate actions for task execution.Source: §B.2 · p. 17

B.3 Algorithm Details

• The appendix provides detailed algorithms for DemoParser’s knowledge extraction and ActExecutor’s task execution, the framework’s core computational processes.Source: §B.3 · p. 17

C ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSES

• The section provides additional experimental results and analyses that supplement Section 5.Source: C ADDITIONAL EXPERIMENTAL RESULTS AND… · p. 17

C.1 Online Performance Comparisons

• Figures 12 and 13 compare model performance with and without LearnAct enhancement in online evaluation scenarios.Source: §C.1 · p. 17

C.2 Case Studies of LearnAct Online Experiments

• The three case studies are presented as examples of demonstration knowledge benefiting task execution in unseen mobile applications.They are drawn from online experiments and cover different scenarios.Source: §C.2 · p. 17
• DemoParser generates action descriptions using before-and-after screenshots and prior descriptions, while ActExecutor uses observations, action history, and retrieved demonstrations to generate actions.The action-description format is “[On/In] [Screen], [Action], to [Purpose]” with optional memory.Source: §C.2 · p. 18, Figure 11 · p. 19
• Figures 12–13 present Qwen2-VL-7B and UI-TARS-7B-SFT task-success-rate comparisons with and without LearnAct across LearnGUI-Online task dimensions.The supplied captions identify the comparisons but do not state the rates or which system performs better.Source: Figure 12 · p. 20, Figure 13 · p. 20
• Figures 14–16 compare UI-TARS-7B-SFT with LearnAct versus its baseline on Joplin recipe-ingredient quantity and Simple Calendar event deletion, and Qwen2-VL-7B with LearnAct versus baseline on Arduia expense deletion.Source: Figure 14 · p. 21, Figure 15 · p. 22, Figure 16 · p. 23