Source-linked AI summary
MobileViews: A Million-scale and Diverse Mobile GUI Dataset
Longxi Gao, Li Zhang, Shihe Wang, Pengzhi Gao, Wei Liu, Jian Luan, Shangguang Wang, Yuanchun Li, Mengwei Xu
TL;DR
Mobile GUI datasets remain limited in scale, comprehensiveness, and fidelity, even as mobile screens are underrepresented in general-purpose VLM training. MobileViews combines VLM-enhanced traversal with native SoC-cluster environments to construct a million-scale screenshot-view hierarchy dataset. Training with MobileViews improves grounding across models by up to 6.1%.
Problem
Mobile GUI screens remain underrepresented in VLM training, and existing datasets are limited in scale, comprehensiveness, and fidelity.
Source: §1 · pp. 1–2Method
MobileViews combines a VLM-enhanced traversal framework with two SoC clusters to collect over 1.2M screenshot-view hierarchy pairs from more than 30K apps.
Source: §1 · p. 2Results
MobileViews improves GUI grounding across four VLMs by up to 6.1%.
Source: §1 · p. 3Takeaways & Limitations
MobileViews can serve as a reliable foundation for advancing mobile GUI agents.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Visual language models (VLMs) empower mobile GUI agents to interpret complex mobile screens and respond to user requests. Training such capable agents requires large-scale, high-quality mobile GUI data. However, existing mobile GUI datasets are limited in scale, data comprehensiveness, and fidelity. To overcome this, we utilize two mobile SoC clusters to provide over 200 native, high-fidelity mobile environments, along with a VLM-enhanced automatic application traversal framework for highly parallel, automated dataset collection with minimal human intervention. With this system, we propose MobileViews, a million-scale mobile GUI dataset comprising over 1.2 million unique screenshot-view hierarchy pairs from more than 30K modern Android applications. We assess the effectiveness of MobileViews by training four VLMs using the reinforcement learning-based GUI grounding task and evaluating them on two representative GUI grounding benchmarks. Results show that MobileViews significantly enhances grounding accuracy by up to 6.1%. Further analysis of data scale and quality underscores the critical role of large, high-quality datasets as reliable sources for training mobile GUI agents. The MobileViews dataset is publicly available at https://huggingface.co/datasets/mllmTeam/MobileViews.
1. Introduction
Mobile GUI data remains limited in scale, comprehensiveness, and fidelity, while mobile screens are underrepresented in general-purpose VLM training. MobileViews combines automated traversal with native SoC environments to build a large, diverse dataset, and experiments report improved grounding across models.
Motivation
- Existing mobile GUI datasets trade off scale and comprehensiveness, including missing view hierarchies, limited app diversity, and narrow resolution or language coverage.Manual collection also limits scalability, while natural-image-heavy training leaves dense, text-rich mobile screens underrepresented.Source: §1 · p. 1
Collection framework
- The crawler combines predefined interaction rules with VLM assistance for difficult states, minimizing manual intervention when traversal stalls.The VLM is triggered by conditions such as prolonged idleness, login pages, or pop-ups.Source: §1 · p. 2
- MobileViews pairs a VLM-enhanced crawler with two SoC clusters running over 200 native Android instances to collect data in parallel.The setup gathers over 1.2M unique screens in approximately 151,600 device-hours.Source: §1 · p. 2
The new dataset: MobileViews
- MobileViews contains over 1.2M unique screenshot-view hierarchy pairs from more than 30K apps, spanning six resolutions, English and Chinese GUIs, and complete trajectories.The dataset also includes 13K screen summaries, 26K QA pairs, and 26K GUI component captions.Source: §1 · p. 2
Experiments
- Training with MobileViews improves grounding across four VLMs by up to 6.1%.UI-Tars-1.5 gains 2.2% on ScreenSpot-v2 and 6.1% on ScreenSpot-Pro.Source: §1 · pp. 2–3
2. Related Work
Prior mobile GUI datasets vary in scale, informational coverage, and collection method, leaving gaps in the combination of visual and structural data, app diversity, and scalable high-fidelity collection.
Data scale
- Dataset scale ranges widely, but many grounding and navigation datasets cover relatively few apps and screens; UI-Venus selects 107K samples from 627K raw data to improve grounding performance [12].Rico contains about 63K screens, while the large AITW dataset has about 2.3M screens from 357 apps.Source: §2 · p. 3
Data comprehensiveness
Data collection methods
- Physical-device collection offers fidelity but faces cost and space constraints, while x86 emulators can limit compatibility with ARM apps; manual collection is also costly.AndroidControl [18] took one year to collect 14,548 episodes across 833 apps; MobileViews combines VLM-enhanced traversal with SoC clusters.Source: §2 · p. 3
3. MobileViews Dataset Construction
MobileViews combines a VLM-enhanced app crawler with SoC-cluster infrastructure to construct a large dataset of diverse mobile screens. Its traversal uses rules, VLM support, and limited human intervention, then scales across native Android instances.
3. MobileViews Dataset Construction
- MobileViews comprises over 1.2M screenshot-view hierarchy pairs from more than 30K modern apps, across six resolutions, English and Chinese GUIs, and complete trajectories.Its construction combines fixed interaction rules, VLM assistance, and minimal human intervention, scaled across two SoC clusters.Source: §3 · pp. 3–4
3.1. App Traversal Workflow
- The crawler collects app metadata, coordinates installation and launch, then follows predefined rules to traverse interactable components and collect screenshots and view hierarchies.When traversal gets stuck, the VLM interprets complex layouts and guides subsequent interactions into deeper interfaces.Source: §3.1 · p. 4
3.2. App Interaction with MobileViews Crawler
- Rule-based traversal identifies interactable components from view-hierarchy attributes and performs actions sequentially; GPT-4o assistance raises successful traversal from 18.6% to 44.2%.VLM support activates on predefined triggers or prolonged idleness, with human operators later resolving states that remain blocked.Source: §3.2 · pp. 4–5
3.3. SoC Cluster-Powered Scalable App Interaction
- Two SoC clusters with 120 Snapdragon processors provide native ARM execution, while SoC-level virtualization runs more than 200 Android instances in parallel.Fine-grained scheduling completes 151,600 device-hours of traversal within two months.Source: §3.3 · p. 5
4. MobileViews Dataset
MobileViews combines more than 1.2M unique mobile screens from over 30K Android apps with structural metadata, diverse configurations, and VLM-assisted annotations. Its app-coverage and screen-similarity analyses indicate broad coverage and low redundancy.
4. MobileViews Dataset
- MobileViews contains over 1.2M unique mobile screens from more than 30K Android apps, with clear advantages in scale, comprehensiveness, and collection scalability.Source: §4 · p. 5
- Screenshots, view hierarchies, multiple resolutions, English and Chinese GUIs, and traversal trajectories provide visual, structural, and contextual data for training mobile GUI agents.The dataset includes 50K unique Chinese screens and multiple English resolution settings.Source: §4 · p. 5
- From 20K candidate screens, multi-stage filtering produced 13K annotated screens, 26K question-answer pairs, and 26K GUI component captions.Annotations also include a screen summary and two functionality QA pairs per screen.Source: §4 · pp. 5–6
- Most apps contribute 10–20 unique screens, while 6.8% yield over 100, indicating broad app coverage and within-app GUI coverage.Source: Figure 5 · p. 6, §4 · p. 6
- Over 95% of MobileViews and Rico screens have fewer than 5 similar counterparts, while 30% of AITW screens have more than 30 similar duplicates.Similarity is measured using ImageHash, with identical perceptual hashes treated as similar; Figure 5b plots the CDF against the number of similar screens.Source: Figure 5 · p. 6, §4 · p. 6
5. Experiments
The experiments train and evaluate four VLMs for GUI grounding, comparing MobileViews training, data filtering, and random scaling. MobileViews improves benchmark performance across model types, while model-specific filtering yields further gains.
5.1. Training Algorithm
- GRPO aligns training with GUI interaction by rewarding predicted points inside target bounding boxes and omitting a format reward when models already produce the required output format.Its binary reward is 1 for a point inside the ground-truth box and 0 otherwise.Source: §5.1 · pp. 6–7
5.2. Experimental Setup
- Four VLMs were trained on the 26K annotated set with random scaling; filtering retained intermediate-reward samples for three models, and evaluation used ScreenSpot-v2 and ScreenSpot-Pro center-point accuracy.Qwen2.5-VL-7B was excluded from filtering because all samples received reward scores of 0 or 8.Source: §5.2 · p. 7
5.3. Results and Analysis
- 2.0–4.5% gains on ScreenSpot-Pro were consistent across all four models; UI-Tars-1.5 rose from 89.0% to 90.3% on ScreenSpot-v2 and from 41.6% to 46.1% on ScreenSpot-Pro.Source: §5.3 · pp. 7–8, Table 3 · p. 8
- 1.3–3.4% gains on ScreenSpot-v2 and 1.4–3.3% gains on ScreenSpot-Pro followed model-specific filtering; unfiltered MobileViews data already improved every model over baseline.Filtering also improved UI-Tars-1.5 on both benchmarks despite its 254-sample training set.Source: Table 4 · p. 8, §5.3 · p. 8
- 0.9–3.3% improvements on both benchmarks came from model-specific filtering, while random scaling sometimes fluctuated or declined for general-purpose models.UI-Tars-1.5 with 254 filtered samples performed comparably on ScreenSpot-v2 to training with 26K random samples.Source: Figure 6 · p. 8, §5.3 · p. 8
6. Conclusion
The paper presents MobileViews as a large-scale dataset and scalable collection framework for mobile GUI research. Its experiments report consistent grounding improvements across general-purpose and GUI-specific VLMs.
6. Conclusion
- MobileViews Crawler is a scalable framework for large-scale mobile app traversal and data collection.Source: §6 · p. 9
- The open-source dataset contains 1.2M unique screenshot-view hierarchy pairs from over 30K modern apps, with comprehensive metadata and bilingual support.Source: §6 · p. 9
- Experiments show consistent improvements for general-purpose and GUI-specific VLMs, and the paper presents MobileViews as a reliable foundation for advancing mobile GUI agents.Source: §6 · p. 9