GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

Sunqi Fan; Lingshan Chen; Runqi Yin; Qingle Liu; Yongming Rao; Meng-Hao Guo; Shi-Min Hu

Source-linked AI summary

GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

Sunqi Fan, Lingshan Chen, Runqi Yin, Qingle Liu, Yongming Rao, Meng-Hao Guo, Shi-Min Hu

arXiv:2606.29705v1Jun 29, 2026cs.AIcs.CLcs.CV

TL;DR

GUI agents need costly, fine-grained annotations that are difficult to collect at scale. GUICrafter trains with massive unannotated screenshots and a small amount of high-quality reinforcement data, achieving competitive or superior performance to UI-TARS and GUI-R1 with far less data.

Problem
Fine-grained GUI annotations are labor-intensive, while automatic labels are often unreliable, limiting scalable data collection for GUI agents.
Method
GUICrafter converts interactive signals from massive unannotated webpages and device screenshots into meta-tasks for weakly supervised pretraining, followed by reinforcement fine-tuning.
Results
Using about one-thousandth of UI-TARS’s data, GUICrafter achieves comparable or superior performance across six benchmarks and surpasses GUI-R1 by 4%-5% on ScreenSpot-Pro.
Takeaways & Limitations
GUICrafter demonstrates that weakly supervised pretraining can improve GUI agents’ data efficiency, visual grounding, and domain generalization.
Takeaways & Limitations
GUICrafter still relies on a small amount of human-annotated data during Stage 2.

Abstract

from arXiv · show

Data, as the fundamental substrate of modern intelligence, has greatly driven the development of current foundation models. Naturally, researchers aim to extend this paradigm to the domain of GUI agents, hoping to build strong GUI agents through a similar paradigm. However, GUI agent data cannot be directly harvested from the internet, making it costly and difficult to collect at scale. As a result, current GUI agents suffer from poor cross-device generalization and limited visual grounding ability for fine-grained GUI elements. As an attempt to address data challenge in GUI agents, we propose GUICrafter, a weakly-supervised GUI agent leveraging massive unannotated screenshots to substantially reduce the reliance on expensive human annotations. GUICrafter explores a curriculum learning framework for training GUI agents through two progressive stages. First, the model learns visual grounding from large-scale unannotated screenshots and webpages, leveraging the rich contextual signals inherent in GUI interactions without human annotations. Then, in Stage 2, we leverage a small amount of high-quality data to calibrate the model via reinforcement learning. Experiments show that GUICrafter achieves competitive, or even superior, performance to advanced systems like UI-TARS while using only 0.1% of its data. Furthermore, under the same amount of annotated data, GUICrafter surpasses all previous methods such as GUI-R1. Code, data, and models are available at https://github.com/fansunqi/GUICrafter.

1 Introduction

GUICrafter addresses costly GUI data annotation and resulting grounding and generalization limitations by converting massive unannotated screenshots and interaction signals into trainable data. Its two-stage curriculum combines weakly supervised pretraining with high-quality supervised calibration, improving performance across diverse GUI benchmarks.

Data challenge: Fine-grained GUI annotation is labor-intensive, while automatic MLLM labeling is often unreliable, limiting scalable data collection.Insufficient and incomplete training data leave agents vulnerable to subtle visual details and restrict cross-GUI and cross-domain generalization.
Weakly supervised data: GUICrafter converts massive unannotated screenshots and webpages into trainable data by extracting interactive signals automatically.It uses web scraping and abundant mobile data to generate corresponding GUI meta-tasks and actions without labor-intensive manual annotation.
Two-stage training: The curriculum uses two progressive RLVR stages: Stage 1 learns grounding from meta-task interaction rewards, while Stage 2 incorporates high-quality manually annotated data.The two-stage design integrates weakly supervised and supervised training to improve data utilization and learning efficiency.
Results: Across six benchmarks spanning different platforms, GUICrafter significantly improves visual grounding and generalization, with both stages increasing grounding accuracy on Mind2Web and ScreenSpot-Pro.The reported results show significant gains from Stage 1 and Stage 2, and superior performance to advanced systems including UI-TARS and GUI-R1.

2 Related Work

GUI agents have progressed from text-based representations to screenshot-driven vision systems, while reinforcement learning has improved grounding and enabled online, self-evolving interaction. GUICrafter differs from these approaches by adding a weakly supervised, annotation-free GUI pretraining stage.

Text-based computer-use agents rely on HTML and accessibility trees, but these representations suffer from inconsistency, volatility, and limited scalability.
Vision-based GUI agents use screenshots instead of textual descriptions, with systems such as Show-UI and UI-TARS improving efficiency, accuracy, perception, action modeling, and reasoning.Show-UI uses UI-guided token selection to reduce computational costs, while UI-TARS integrates enhanced perception, unified action modeling, and system-2 reasoning.
Reinforcement learning has been applied to GUI agents through workflow-guided exploration and reinforcement-based visual grounding with refined reward modeling.These directions include collecting valid web interaction trajectories and improving data efficiency through methods such as GUI-R1 and UI-R1.
GUICrafter distinguishes itself from GUI-R1 and UI-R1 by introducing a weakly supervised, annotation-free GUI pretraining stage that enhances GUI knowledge and comprehension.This pretraining stage is identified as Stage 1.
Online, multi-turn, self-evolving RL frameworks use reward models and interactive environments but typically require task instruction annotations and carefully constructed reward or world models.Examples include ZeroGUI, WebEvolver, MobileGUI-RL, SEAgent, and WebAgent-R1; these designs can add human-annotation or LLM-generation costs and environment-specific assumptions.

3 Method

GUICrafter trains GUI agents in two stages: weakly supervised pretraining on massive screenshots and interaction signals, followed by reinforcement fine-tuning on filtered high-quality data. Stage 1 constructs meta-tasks and verifiable rewards from webpages and mobile screenshots, while Stage 2 improves data reliability through automated filtering and human verification.

Stage 1 Weakly-Supervised GUI Pretraining: Stage 1 weakly supervised GUI pretraining teaches agents from abundant screenshots, mirroring how language models learn from large text corpora.The stage is designed to improve visual grounding and generalization without extensive human annotations.
Collecting Real-World Webpages & Screenshots: GUICrafter collects real-world webpages and screenshots, using recursive webpage crawling and mixed AndroidControl and AITZ screenshots for mobile training data.Webpages are stored in MHTML format, while mobile data uses automatically collected interactive elements rather than human-annotated trajectories.
Crafting Interactive Signals & Meta-Tasks: Interactive signals are extracted from webpage elements and mobile accessibility trees, then converted into action-specific meta-tasks for clicks, typing, selection, and mobile interactions.Playwright identifies web GUI elements from MHTML files, while mobile rules filter misannotated pages before meta-task construction.
Training Algorithm and Reward Design: RLVR combines format, action-type, position, and text rewards, and GRPO optimizes the base language model from the resulting feedback.Outputs must use JSON fields for action type, predicted position, and optional input text; text actions additionally require token-level F1 above a threshold.

4 Experiments

GUICrafter is evaluated across six GUI benchmarks using grounding, operation, action-type, step-success, and episode-success metrics. Across these evaluations, its two-stage curriculum delivers strong data-efficient performance, improves generalization, and benefits from GUI pretraining and Gaussian rewards.

Evaluation Setup: The evaluation covers Mind2Web, ScreenSpot-Pro, OmniACT, AndroidControl, AITW, and AndroidWorld across web, desktop, mobile, and real Android environments.Metrics include Element Accuracy, Operation F1, Step Success Rate, grounding accuracy, Type, GR, and Episode Success Rate.
Mind2Web: After Stage 1, GUICrafter improves average Mind2Web accuracy by over 10% versus Qwen2.5-VL-3B without human labor.The improvement holds across all Mind2Web subcategories.
Mind2Web: After both stages, GUICrafter achieves the best average Mind2Web grounding accuracy and comparable performance to UI-TARS using about one-thousandth of its data.GUICrafter uses 6,795 high-quality samples and 20,000 weakly supervised samples, compared with UI-TARS’s approximately 18.4M samples.
Mind2Web: GUICrafter outperforms UI-TARS more substantially on Mind2Web Cross-Website and Cross-Domain splits, indicating stronger generalization to distribution shifts.The result is attributed primarily to Stage 1 exposure to many real-world webpages.
ScreenSpot-Pro: On ScreenSpot-Pro, GUICrafter surpasses GUI-R1-3B by 4%-5% in average accuracy and achieves a 3% improvement over Stage 2-only training.It performs best among models of comparable size and wins in most subcategories.
AndroidControl, AITW, and OmniACT: On AndroidControl and AITW, Stage 1 reaches 62.35% Step SR on AndroidControl-Low and 44.65% on AndroidControl-High without annotated data, while the full curriculum reaches 50.89% overall zero-shot performance on AITW.The full model outperforms comparable-size models on AndroidControl and other zero-shot models on AITW; GUICrafter also achieves leading performance on OmniACT.
Ablations: The full curriculum significantly outperforms Stage 2 alone, with 4% improvement on Mind2Web and 3% improvement on ScreenSpot, while Gaussian rewards outperform binary rewards on all reported metrics.Meta-tasks are sufficiently expressive in Stage 1, and their simplification has almost no impact after two-stage training without human annotation or LLM APIs.

5 Analysis

The analysis shows that Stage 1 weak supervision improves visual grounding with few samples and continues scaling, while Stage 2 helps recover from failures and makes training more robust to noisy supervision.

Stage-wise analysis: Stage 1 can fail on a screenshot that Stage 2 training enables the agent to solve.Figure 3 illustrates three meta-tasks with different target regions and compares thoughts and actions across training stages.
Weakly-supervised data scaling: 10 weakly-supervised samples improve the base Qwen2.5-VL-3B model by 1.7% on Mind2Web and 2.6% on ScreenSpot-Pro.The ablation evaluates datasets of 10, 100, 1000, 10000, and 50000 samples, averaging three runs per setting.
Weakly-supervised data scaling: Performance improves as Stage 1 data increases, with no saturation observed up to 50k samples and convergence at about 350k samples.The results indicate that noisy weakly-supervised data remains scalable, with a relatively high noise ceiling.
Noise robustness: 84.9% of 1,000 randomly sampled Stage 1 examples are fully correct, while Stage 2 significantly reduces performance gaps across noise levels.Noise is defined as missing, overlapping, or disordered visual interactable elements; the two-stage framework is therefore robust to noisy Stage 1 supervision.

6 Conclusion

GUICrafter addresses GUI-agent data scarcity by learning from massive unannotated screenshots, reducing reliance on costly manual annotations. It achieves competitive or superior performance against advanced systems while demonstrating strong data efficiency and domain generalization.

6 Conclusion: GUICrafter addresses data scarcity in GUI agents by substantially reducing dependence on costly manual annotations.It enables GUI agents to evolve by observing massive unannotated screenshots.
6 Conclusion: GUICrafter achieves competitive or superior performance compared with advanced systems such as UI-TARS and GUI-R1.The conclusion presents these results as evidence of the method’s effectiveness despite reduced annotation dependence.
6 Conclusion: GUICrafter demonstrates strong data efficiency and domain generalization ability.These properties are highlighted by the experimental comparison with advanced GUI-agent systems.

Appendix · A Preliminaries · A.1 GUI Agent Formulation

The GUI agent formulation represents interaction as a sequence conditioned on the task instruction, observations, and action history. At each step, the agent produces a thought and a low-level action, yielding a multi-step interaction chain.

A.1 GUI Agent Formulation: The agent receives the task instruction I, the current observation o, and the action history h as input.The current observation is usually a screenshot.
A.1 GUI Agent Formulation: For a single-step task, the agent produces a thought t and a low-level action a.The action is part of the agent’s output alongside its reasoning state.
A.1 GUI Agent Formulation: Each low-level action contains an action type, a position, and optional text input.These fields define the executable GUI interaction.
A.1 GUI Agent Formulation: In a multi-step task, every step i has its own observation o_i, thought t_i, and action a_i.The formulation assigns a distinct interaction tuple to each step.
A.1 GUI Agent Formulation: The formulation therefore distinguishes single-step outputs from a temporally ordered multi-step interaction chain.Single-step behavior uses one observation-thought-action instance, whereas multi-step behavior repeats the tuple across steps.
A.1 GUI Agent Formulation: The full interaction is modeled as the instruction I followed by the ordered tuples (o_1, t_1, a_1), …, (o_n, t_n, a_n).This sequence captures the environment-agent interaction across n steps.

A.2 RLVR & GRPO Algorithm

This section describes RLVR and GRPO as verifier-guided policy optimization methods, with GRPO estimating advantages from grouped rewards and constraining updates through KL divergence. The resulting objective balances policy improvement against keeping updates within a reasonable range.

RLVR: RLVR models act as policies that map states s to actions a and receive answer-correctness feedback from deterministic verifiers.For each response, the verifier assigns a reward r_i.
GRPO: GRPO estimates advantages from verifier-assigned rewards and updates the policy while adhering to KL divergence constraints.GRPO is identified as an RLVR variant initially proposed in DeepSeekMath [35].
GRPO: The GRPO loss combines the policy probability ratio weighted by advantage with a KL-divergence penalty scaled by α.The objective uses π_θ(a|s)/π_θold(a|s) and D_KL(π_θold ∥ π_θ).
GRPO: The KL term keeps policy updates within a reasonable range, with α serving as the balancing hyperparameter.Here, s denotes the current state.

B Results on OmniACT

On OmniACT, GUICrafter’s Stage 1 GUI pretraining improves grounding over Qwen2.5-VL-3B, while the full curriculum outperforms comparable models and Stage 2-only training. Its final gains are larger on desktop than web.

Results on OmniACT: 18.96% on the web domain and 30.84% on the desktop domain: Stage 1 improves grounding accuracy over Qwen2.5-VL-3B.These results highlight the effectiveness of GUI pretraining.
Results on OmniACT: 2.1% on the web domain and 4.5% on the desktop domain: GUICrafter exceeds the second-best GUI-R1 after Stages 1 and 2.UI-TARS-2B is excluded because it has not released official OmniACT results.
Results on OmniACT: 3.1% on the web domain and 6.1% on the desktop domain: the full curriculum improves grounding accuracy over training with Stage 2 alone.This confirms that GUI pretraining strengthens the model.