Source-linked AI summary
GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices
Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Ping Luo
TL;DR
Prior mobile GUI datasets predominantly emphasize single-app tasks, leaving cross-app navigation with longer, context-spanning workflows insufficiently covered. The paper introduces GUIOdyssey and OdysseyAgent, combining richly annotated cross-app episodes with efficient historical-information processing. Experiments in in-domain and out-of-domain settings show that richer historical information substantially improves performance, while security concerns remain for sensitive operations.
Problem
Prior mobile GUI navigation datasets predominantly contain single-app tasks and mainly annotate operational actions, limiting coverage of complex cross-app navigation.
Method
The paper builds GUIOdyssey with multi-granularity instructions and semantic step annotations, then develops OdysseyAgent with a history resampler for efficient long-history processing.
Results
8,334 episodes average 15.3 steps across 6 devices, 212 apps, and 1,357 app combinations; experiments show richer historical information substantially improves performance.
Takeaways & Limitations
Cross-app GUI navigation benefits from modeling historical actions, screenshots, and context, with semantic annotations providing further gains in many scenarios.
Takeaways & Limitations
Sensitive operations remain vulnerable without robust safeguards, and malicious actors could exploit GUI agents to bypass security protocols or manipulate applications.
Abstract
from arXiv · showhide
Autonomous Graphical User Interface (GUI) navigation agents can enhance user experience in communication, entertainment, and productivity by streamlining workflows and reducing manual intervention. However, prior GUI agents often trained with datasets comprising tasks that can be completed within a single app, leading to poor performance in cross-app navigation. To address this problem, we present GUIOdyssey, a comprehensive dataset for cross-app mobile GUI navigation. GUIOdyssey comprises 8,334 episodes with an average of 15.3 steps per episode, covering 6 mobile devices, 212 distinct apps, and 1,357 app combinations. Each step is enriched with detailed semantic reasoning annotations, which aid the model in building cognitive processes and enhancing its reasoning abilities for complex cross-app tasks. Building on GUIOdyssey, we develop OdysseyAgent, an exploratory multimodal agent for long-step cross-app navigation equipped with a history resampler module that efficiently attends to historical screenshot tokens, balancing performance and inference speed. Extensive experiments conducted in both in-domain and out-of-domain scenarios validate the effectiveness of our approach. Moreover, we demonstrate that historial information involving actions, screenshots and context in our dataset can significantly enhances OdysseyAgent's performance on complex cross-app tasks.
1. Introduction
GUIOdyssey addresses the limited cross-app coverage of prior mobile GUI datasets by combining broad cross-app episodes with fine-grained and semantic annotations. OdysseyAgent uses a history resampler to process long histories efficiently, and experiments show that richer historical information improves navigation performance.
- Motivation: Existing mobile GUI datasets mainly focus on single-app tasks and operational actions such as clicking and scrolling.This leaves cross-app capabilities insufficiently represented.
- Motivation: Cross-app navigation requires integrating multiple apps and transferring context and data across longer, more complex workflows.These tasks increase working-memory, functional-knowledge, and error-propagation demands.
- Dataset: GUIOdyssey provides high- and low-level instructions, while annotators and GPT-4o supply action, screenshot, contextual, and reasoning annotations.A quality-check pipeline evaluates screenshot integrity, action accuracy, and instruction alignment.
- Dataset: GUIOdyssey contains 8,334 episodes averaging 15.3 steps, spanning 6 devices, 212 apps, and 1,357 app combinations.The dataset covers six cross-app task types and enriches steps with semantic reasoning annotations.
- OdysseyAgent: OdysseyAgent uses a history resampler that selectively attends to historical screenshot tokens to balance navigation performance and inference speed.The agent is evaluated in both in-domain and out-of-domain settings.
- Evaluation: Experiments show that comprehensively using historical actions, screenshots, and contextual information substantially improves cross-app navigation performance.Semantic annotations provide additional gains, particularly for high-level instructions and unseen domains.
2. Related Work
Prior GUI benchmarks and datasets support navigation, perception, and comprehension but predominantly confine tasks to single apps. GUIOdyssey targets the resulting cross-app challenges of longer sequences, working-memory management, and heterogeneous interactions.
- Existing GUI resources: Existing GUI research includes interactive environments for navigation and datasets focused on UI perception and comprehension.Recent GUI datasets nevertheless predominantly involve tasks confined to a single app.
- Cross-app challenges: Cross-app tasks use longer action sequences, increasing error-propagation risks compared with single-app tasks.Their longer horizons make reliable navigation more difficult.
- Cross-app challenges: Cross-app interactions require managing working memory because UI elements and contextual information span multiple apps.Agents must retain relevant information across app boundaries.
- Cross-app challenges: Cross-app tasks demand broader functional knowledge to integrate interactions such as file sharing, email composition, and messaging.These interactions involve distinct app capabilities and action types.
3. GUIOdyssey Dataset
GUIOdyssey is a cross-app mobile GUI navigation dataset built from diverse task instructions, human demonstrations, fine-grained semantic annotations, and quality checks. It contains long, varied episodes spanning multiple task types, apps, combinations, and devices.
- Metadata Definition: GUIOdyssey defines a GUI episode as screenshots and actions recorded while completing a user’s high-level instruction.Each episode contains the instruction, screenshots, actions, and total step count.
- Task Design: The dataset covers six cross-app task types, including general tools, information management, web shopping, media entertainment, social sharing, and multi-app operations.These categories span system operations, information use, purchases, streaming, social platforms, and complex cross-domain workflows.
- Task Design: 91 high-level instruction templates generate diverse episodes by varying items and selected apps.Templates are created by human participants and GPT-4, with item substitutions supporting instruction diversity.
- Data Collection: Human annotators record step-by-step demonstrations on Android emulators, saving screenshots before actions and documenting completed or impossible tasks.Recorded episodes include instructions, screenshots, actions, used apps, and additional notes.
- Episode Annotation: GPT-4o adds low-level instructions and semantic annotations covering screen descriptions, contextual information, and decision rationale.These annotations describe the current screen, preceding steps, and reasoning behind the next action.
- Dataset Statistics: 8,334 validated episodes average 15.3 steps and span 6 devices, 6 task types, 212 apps, and 1,357 app combinations.The dataset emphasizes longer, broader cross-app workflows than single-app datasets and includes device types such as foldable phones and tablets.
4. Method: OdysseyAgent
OdysseyAgent is a multimodal cross-app navigation agent designed to use long interaction histories without imposing the full inference cost of stacking historical screenshots. Its history resampler compresses past screenshot tokens before combining them with current observations, instructions, and actions for next-action prediction.
- Agent Design: OdysseyAgent fine-tunes Qwen-VL on GUIOdyssey for cross-app navigation with long screenshot and action histories.The design targets the tension between retaining historical context and maintaining fast inference during frequent user interactions.
- History Resampler: The history resampler compresses historical screenshot tokens before they reach the language model.It is introduced to reduce the overhead of stacking all past screenshots while preserving essential contextual information.
- Design Goal: The architecture is intended to balance navigation performance with inference efficiency when processing long histories.The paper reports a more favorable performance-efficiency balance than straightforward multi-image concatenation.
- History Resampler: The resampler uses single-layer cross-attention with learnable queries and historical screenshot tokens as keys and values.The resulting compressed history is concatenated with the current screen token, user instruction, and previous actions.
- Action Prediction: The fused representation is fed to the language model to predict the next action using a next-word prediction objective.The trainable parameters include the vision-language adapter, history resampler, and language model.
5. Experiment
The experiments evaluate OdysseyAgent across in-domain and out-of-domain app, task, and device settings, while examining instruction granularity, semantic annotations, and historical information. Results indicate that high-level out-of-domain navigation remains more difficult, whereas semantic annotations and richer historical context improve performance in relevant scenarios.
- Experimental setup: Four evaluation setups test generalization across random, app, task, and device splits.The first is in-domain; the remaining three are out-of-domain.
- Evaluation metrics: Action Matching Score evaluates individual actions, while Success Rate requires every action in an episode to be correct.Success Rate is stricter for longer tasks because each action must succeed.
- Out-of-domain performance: 16.26 and 5.92 are the high- and low-level performance declines out of domain without semantic annotations.With semantic annotations, the corresponding declines are 15.34 and 6.95.
- Out-of-domain performance: Semantic annotations improve performance in most out-of-domain scenarios, with especially notable gains for high-level instructions.The results also indicate that high-level instructions generalize less effectively than low-level instructions.
- Historical information: Historical actions, screenshots, and contextual summaries are examined as information sources for long-step cross-app navigation.A Chrome-to-Google-Docs task illustrates the need to remember and transfer information across apps.
- Additional analyses: Additional experiments investigate screenshot-history strategies, semantic annotation components, device transfer, instruction granularity, and cross-app versus single-app tasks.These analyses are reported in the appendix.
6. Conclusion
The paper introduces GUIOdyssey as a semantic-annotation-rich cross-app mobile GUI dataset and develops OdysseyAgent with a history resampler for efficient historical-image processing. Experiments cover in-domain and out-of-domain settings and indicate that richer historical information can substantially improve performance.
- Conclusion: GUIOdyssey is presented as the first comprehensive cross-app mobile GUI navigation dataset enriched with semantic annotations.The dataset is intended to address limitations of existing GUI navigation agents on cross-app tasks.
- Conclusion: OdysseyAgent is a multimodal cross-app navigation agent that uses a history resampler to balance historical-image processing performance and inference speed.The module efficiently processes historical image tokens.
- Conclusion: Extensive experiments evaluate OdysseyAgent in both in-domain and out-of-domain scenarios.The evaluation is conducted using GUIOdyssey.
- Conclusion: Richer utilization of historical information can substantially enhance OdysseyAgent’s performance.The conclusion connects historical information use with performance on the evaluated cross-app tasks.
7. Ethical Discussion
The dataset collection uses temporary accounts and virtual usernames rather than authentic personal information, and participant data is collected with informed consent. The paper also identifies security and ethical risks associated with deploying GUI agents.
- Privacy: Temporary accounts and virtual usernames are used, and the dataset contains no authentic personal information.This privacy measure applies to app registration and data collection.
- Ethical consent: Participants explicitly consent to including their human-annotated data in the dataset.The paper states that collection complies with ethical guidelines.
- Security concerns: Sensitive operations and malicious misuse create security risks that require robust safeguards, privacy-preserving techniques, and ethical guidelines.Examples include financial transactions, privacy management, bypassing security protocols, and manipulating applications.
8. Details of GUIOdyssey
GUIOdyssey organizes cross-app navigation into six task categories and records episodes with device, task, action, screenshot, instruction, and semantic information. Its annotation pipeline uses GPT-4o iteratively to generate low-level instructions and step-level reasoning, with quality checks and segmentation-assisted action evaluation.
- Task categories: Six task categories cover general tools, information management, web shopping, media entertainment, social sharing, and multi-app workflows.Multi-app tasks require three or more apps, while the other categories represent distinct cross-app activity types.
- Task categories: Multi-app tasks require three or more apps, such as finding a recipe, recording it in notes, and buying ingredients online.This category represents more complex operations spanning several applications.
- Data collection: The recording system uses Android Studio to virtualize devices and ADB to retrieve device status and monitor interaction events.Recorded actions include click coordinates and functional-key activity.
- Annotations: Fine-grained episode annotations contain low-level instructions and semantic annotations.Examples are provided in the paper’s annotation figures.
- Annotation pipeline: GPT-4o generates low-level instructions from episode instructions, screenshots, and actions, using bounding-box images for click-related actions.The image fidelity parameter is set to high.
- Annotation pipeline: Semantic annotations are generated iteratively in episode order using prior decision rationales and contextual information before producing each step’s screen description and reasoning.The process continues sequentially until all episode steps are annotated.
- Action evaluation: SAM2 assists evaluation by producing bounding boxes for clicked elements through segmentation.These boxes support assessment of whether model output actions are correct.
- Episode schema: Each episode records device and task metadata plus step-level screenshots, actions, instructions, semantic annotations, segmentation boxes, scroll trajectories, and annotator notes.The episode schema also includes a unique identifier and total step length.
9. Experiment Details
GUIOdyssey experiments use four train/test splits to assess in-domain performance and generalization across tasks, devices, and apps. OdysseyAgent is trained with standard optimization and evaluated alongside several GUI-agent baselines.
- Experimental setups: Four experimental splits assess in-domain performance and generalization across tasks, devices, and apps.The setups include random, task, device, and app/app-combination partitions.
- Experimental setups: 1,381 Tablet episodes form the test set while 6,953 episodes from other devices form the training set in the unseen-device split.The Tablet differs significantly from the other devices.
- Experimental setups: The unseen-app split selects low-frequency apps from 25 classes and uses an approximately 85% to 15% training-to-test ratio.The test set targets unseen apps and app combinations.
- Training and evaluation: OdysseyAgent training uses AdamW with a 2e−5 learning rate, cosine scheduling, global batch size 128, and DeepSpeed ZERO2-style data parallelism.Each action step is treated as an individual training sample.
- Training and evaluation: Evaluation compares proprietary LVLMs using a dedicated prompt and evaluates SphAgent and CogAgent with their officially recommended methods.The proprietary models include GPT-4V, GPT-4o, Claude3.5-sonnet, and InternVL2-Pro.
10. More Experiments
Additional experiments examine history compression, semantic annotations, instruction granularity, device transfer, and training-data composition. They report efficiency gains from the history resampler and broader benefits from semantic and cross-app training data.
- History processing: 256 history-resampler tokens replace 1,024 tokens for four historical steps while achieving comparable performance and improving training and inference efficiency.The comparison evaluates multi-image input against compressed historical screenshot representations.
- Semantic annotations: Using all semantic annotations improves AMS by 3.14 and SR by 35% compared with training without semantic annotations.The annotations comprise screen description, contextual information, and decision rationale.
- Semantic annotations: Detailed decision rationale contributes more than screen understanding or historical summarization, while combining at least two annotation types generally outperforms using one.This pattern is reported across in-domain and out-of-domain settings.
- Instruction granularity: High-level-instruction training transfers better across instruction granularities than low-level-instruction training, and training on both granularities outperforms training on one.The comparison is conducted on Train-Random/Test-Random using AMS.
- Device generalization: OdysseyAgent shows its weakest transferability on tablet devices when trained on five devices and tested on the remaining device.The cross-device evaluation tests adaptability to one held-out device at a time.
- Training-data composition: Cross-app GUIOdyssey data consistently improves performance in most single-app scenarios, whereas AITW data provides limited or negative improvements.The controlled comparison samples 50k training examples from GUIOdyssey, AITW, and AndroidControl and evaluates on AndroidControl.