Source-linked AI summary
Step-GUI Technical Report
Haolong Yan, Jia Wang, Xin Huang, Yeqing Shen, Ziyang Meng, Zhimin Fan, Kaijun Tan, Jin Gao, Lieyu Shi, Mi Yang, Shiliang Yang, Zhirui Wang, Brian Li, Kang An, Chenyang Li, Lei Lei, Mengmeng Duan, Danxun Liang, Guodong Liu, Hang Cheng, Hao Wu, Jie Dong, Junhao Huang, Mei Chen, Renjie Yu, Shunshan Li, Xu Zhou, Yiting Dai, Yineng Deng, Yingdan Liang, Zelin Chen, Wen Sun, Chengxu Yan, Chunqin Xu, Dong Li, Fengqiong Xiao, Guanghao Fan, Guopeng Li, Guozhen Peng, Hongbing Li, Hang Li, Hongming Chen, Jingjing Xie, Jianyong Li, Jingyang Zhang, Jiaju Ren, Jiayu Yuan, Jianpeng Yin, Kai Cao, Liang Zhao, Liguo Tan, Liying Shi, Mengqiang Ren, Min Xu, Manjiao Liu, Mao Luo, Mingxin Wan, Na Wang, Nan Wu, Ning Wang, Peiyao Ma, Qingzhou Zhang, Qiao Wang, Qinlin Zeng, Qiong Gao, Qiongyao Li, Shangwu Zhong, Shuli Gao, Shaofan Liu, Shisi Gao, Shuang Luo, Xingbin Liu, Xiaojia Liu, Xiaojie Hou, Xin Liu, Xuanti Feng, Xuedan Cai, Xuan Wen, Xianwei Zhu, Xin Liang, Xin Liu, Xin Zhou, Yifan Sui, Yingxiu Zhao, Yukang Shi, Yunfang Xu, Yuqing Zeng, Yixun Zhang, Zejia Weng, Zhonghao Yan, Zhiguo Huang, Zhuoyu Wang, Zihan Yan, Zheng Ge, Jing Li, Yibo Zhu, Binxing Jiao, Xiangyu Zhang, Daxin Jiang
TL;DR
GUI automation needs scalable, reliable trajectory annotation and practical solutions for deployment across heterogeneous devices and real-world usage. This work introduces CSRS and Step-GUI, GUI-MCP, and AndroidDaily, achieving strong benchmark performance while reducing annotation costs and supporting privacy-centric deployment and everyday evaluation.
Problem
GUI agents need high-quality trajectory and knowledge data, but traditional annotation is subjective and prohibitively expensive, while deployment also lacks standardized cross-platform interfaces and privacy protection.
Method
The paper develops CSRS for self-evolving trajectory calibration, trains Step-GUI models, proposes hierarchical GUI-MCP, and introduces AndroidDaily for real-world evaluation.
Results
Step-GUI-8B achieves 80.2% on AndroidWorld, 48.5% on OSWorld, 62.6% on ScreenShot-Pro, and 89.91%/52.50% on AndroidDaily static/end-to-end evaluation, while CSRS reduces annotation cost by 10-100×.
Takeaways & Limitations
The combined framework advances practical GUI agents across data acquisition, privacy-centric deployment, and evaluation grounded in authentic everyday usage.
Takeaways & Limitations
The GRPO objective has inherent instability for tokens with low baseline values because gradient modulation depends on the importance sampling ratio.
Abstract
from arXiv · showhide
Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving >90% annotation accuracy with 10-100x lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily, a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions.
1. Introduction
The paper addresses scalable, reliable GUI-agent training, deployment, and evaluation through CSRS, Step-GUI, GUI-MCP, and AndroidDaily. Together, these contributions span training data quality, cross-device privacy-preserving execution, and authentic daily-use assessment.
- Training pipeline: >90% annotation accuracy and 10-100× cost reduction are achieved by calibrating model-generated GUI trajectories with trajectory-level evaluation signals.CSRS combines automated verification scripts or human annotations with LLM-powered knowledge extraction to produce training data.
- Step-GUI models: 80.2% AndroidWorld, 48.5% OSWorld, 65% ScreenShot-Pro, and 89.91%/52.50% AndroidDaily static/end-to-end are achieved by Step-GUI-8B.The Step-GUI family includes 4B and 8B models, while the 4B variant is described as deployable on consumer-grade hardware.
- Deployment: GUI-MCP standardizes GUI automation across heterogeneous devices through low-level atomic operations and high-level delegation to locally deployed specialist models.Its privacy mode keeps raw screenshots and sensitive states on-device while sending only semantic summaries to external LLMs.
- Evaluation: AndroidDaily evaluates everyday mobile usage through static and interactive tasks grounded in empirical usage patterns rather than synthetic setups alone.The benchmark targets high-frequency daily scenarios and addresses gaps in single-step and insufficiently representative existing evaluations.
- Overall contribution: The paper presents a complete practical-GUI-agent pipeline spanning model training, standardized deployment interfaces, and authentic evaluation.The stated goal is to support GUI agents assisting users in daily mobile interactions.
2. Step-GUI
Step-GUI develops GUI agency through progressive training, error-driven refinement, grounding-data cleaning, and CSRS-based trajectory processing. The approach combines general multimodal capabilities with execution-oriented supervision and iterative data improvement.
- Mid-training data: The mid-train mixture balances general multimodal and knowledge data with agent-oriented data for visual grounding, action alignment, and cross-platform interaction.The supplied composition includes 1.9M high-quality text and multimodal samples, 2M knowledge-intensive samples, and 2.7M grounding samples.
- Cold-start data: ~1.67M cold-start samples are used versus ~11.2M mid-train samples, with cold-start assigning 52% to knowledge data for execution-failure correction.Cold-start emphasizes error-driven VQA knowledge injection and curated trajectories rather than broad exposure.
- Grounding data: GUI grounding requires functional semantics, latent interface state, and HCI world knowledge beyond appearance-based text-region alignment.These requirements motivate training that supports structured virtual-environment understanding and action.
- Grounding data: Iterative grounding cleaning uses pass-rate labeling, complexity scoring, filtering, curriculum learning, and hard-case refinement to improve supervision quality.The loop moves beyond sparse click supervision toward explicit modeling of interface structure, dynamics, and affordances.
- Calibrated Step Reward System: CSRS converts model-generated trajectories into high-quality, multidimensional training data through trajectory-level calibration and knowledge extraction.Its closed-loop role is Rollout → CSRS → Training, while successful and failed trajectories receive different data treatment.
2.2. Training
Step-GUI trains Qwen3-VL-based GUI agents through progressive stages and a self-evolving closed-loop pipeline that generates, verifies, refines, and reuses interaction data. Its reinforcement-learning design combines calibrated rewards for spatial accuracy, action semantics, and broader trajectory quality with exploration and efficiency mechanisms.
- Progressive Training: The training paradigm uses Mid-Training, Cold-Start Fine-Tuning, and RLVR to specialize 4B/8B Qwen3-VL models for GUI agency.The formulation treats GUI automation as sequential vision-based decision making over tasks, actions, states, and observations.
- Self-Evolving Data Engine: The closed-loop engine alternates policy-generated trajectories verified by CSRS with refinement of existing data through Self-Distillation and Rejection Sampling.Generated data supplies knowledge and verified trajectories for later training, while accepted and rejected samples are routed for reinforcement or weakness-focused refinement.
- Reinforcement Learning: GRPO specializes the policy for precise GUI reasoning and execution using group-normalized advantages, importance-sampling ratios, clipping, and KL regularization.The objective uses trajectory groups and token-level ratios relative to a reference policy, with β controlling KL regularization.
- Reward Design: CSRS-based training signals combine spatial-geometric rewards, action-semantic validity, and LLM-judged capability assessments for GUI trajectories.Spatial rewards cover coordinate and bounding-box precision, while semantic rewards assess action types, parameter values, and retrieved content.
- Stability and Efficiency: Semi-online hindsight, dynamic clipping, and importance-sampling reuse expand exploration and improve data efficiency, while low-probability tokens remain vulnerable to ratio explosion and clipped gradients.Failed rollouts receive ground-truth hints for a second pass, and the method identifies instability when reference probabilities are minimal.
3. GUI-MCP: An Efficient Protocol for Secure LLM-Device Interaction
GUI-MCP standardizes GUI automation through a hierarchical protocol that combines atomic device control with delegation to local specialist models. Its dual-layer design supports efficient execution while offering configurable privacy protections that keep sensitive processing on-device.
- Overall Design: The protocol addresses GUI automation across efficiency and privacy dimensions through a standardized dual-layer interface for LLM-device interaction.This design is presented as a path toward powerful, privacy-preserving assistants.
- Architecture: GUI-MCP uses Low-level MCP for atomic device operations and High-level MCP for abstract task execution through a local GUI specialist model.The main LLM can choose between fine-grained control and delegated execution according to task complexity and model capability boundaries.
- Low-level MCP: The low-level layer exposes device discovery, screenshot-based state perception, and basic interaction primitives for fine-grained planning and control.These interfaces provide flexible access to connected devices and current screen state.
- High-level MCP: The high-level layer accepts natural-language task descriptions and delegates complete tasks to a locally deployed GUI specialist model such as Step-GUI-4B.Delegation is suited to clear, independent tasks within the specialist model’s capability scope and can reduce main-LLM inference overhead.
- Task Allocation: Appropriate task allocation lets the main LLM delegate simple repetitive operations while retaining high-level planning and decision-making.Low-level control remains available for tasks requiring detailed planning, clarification, or capabilities beyond the specialist model.
- Privacy Protection: GUI-MCP’s High Privacy Mode keeps screenshot analysis and sensitive information on-device while sending external models only processed state summaries.Users can configure privacy from open screenshot transmission to fully private text-summary communication.
4. AndroidDaily: A Dynamic Benchmark for Agentic Tasks in Daily Life
AndroidDaily evaluates GUI agents on everyday mobile usage rather than primarily synthetic or application-catalog tasks. It combines static action prediction with end-to-end workflows spanning frequent daily-life scenarios and supports detailed analysis of task difficulty.
- Motivation: AndroidDaily addresses the gap between benchmarked GUI tasks and actual daily usage by selecting tasks from empirical mobile usage patterns.The benchmark prioritizes frequent activities such as food delivery, ride-hailing, short-form video, and mobile payments.
- Static Benchmark: The static benchmark contains 3146 actions covering eight Android automation types with parameterized ground-truth annotations and multiple valid solutions.Its action taxonomy includes AWAKE, CLICK, COMPLETE, INFO, LONGPRESS, SLIDE, TYPE, and WAIT.
- End-to-End Benchmark: The end-to-end benchmark evaluates 235 complete workflows using an LLM-based judger and overall task success rates.This protocol is more infrastructure-demanding but more closely mirrors real-world task execution.
- Scenario Coverage: The 235 end-to-end tasks span Transportation, Shopping, Social Media, Entertainment, and Local Services, with Transportation and Shopping comprising the largest categories.Transportation has 78 tasks and Shopping 61, followed by Social Media, Entertainment, and Local Services.
- Task Analysis: Tasks are additionally organized by cognitive operation, structural complexity, and instruction ambiguity to support targeted capability and failure-mode analysis.These dimensions distinguish information filtering, querying, analysis, atomic, composite, conditional/loop, and ambiguity levels.
5. Experiments
Step-GUI performs strongly across GUI grounding, mobile, desktop, and everyday-use evaluations, with compact 4B and 8B models often matching or exceeding larger systems. Its strongest results are in static action prediction and broad grounding benchmarks, while end-to-end workflows remain more difficult.
- Grounding Benchmarks: 62.6 is Step-GUI-8B’s highest reported ScreenSpot-Pro score, ahead of UI-TARS-1.5 at 61.6 and SeedVL-1.5 at 60.9.Step-GUI-4B reaches 60.0 and surpasses Qwen2.5-VL-72B at 43.6.
- Analysis: The reported results support strong parameter efficiency, with compact 4B and 8B models consistently matching or outperforming substantially larger 30B–72B models.The authors attribute the gains to multi-stage training involving visual grounding, knowledge injection, and iterative grounding cleaning.
- Desktop GUI Results: 48.5 is Step-GUI-8B’s OSWorld-Verified score, second only to Claude-4.5-sonnet at 61.4 and ahead of OpenAI CUA o3 at 23.0.The model also improves over Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct by 22.3 and 14.6 points.
- Mobile GUI Results: 80.2 is Step-GUI-8B’s AndroidWorld score, tying MobileRL-9B and exceeding UI-TARS-2 and Mobile-Agent-v3 at 73.3.Step-GUI-4B reaches 75.8, the second-best result among tested models.
- AndroidDaily Results: 89.91% average static action accuracy is achieved by Step-GUI-8B, while Step-GUI-4B reaches 87.02%, exceeding all reported baselines.UI-TARS-1.5 reaches 67.69%, while GPT-4o and Claude-4.5-sonnet score 17.73% and 10.90%.
- AndroidDaily Results: 52.50% end-to-end success for Step-GUI-8B trails UI-TARS-1.5 at 56.64% by 4.14 points despite its 89.91% static-action accuracy.Composite tasks remain challenging across models, with success rates of 14–20%.
6. Related Work
Related work develops automated reasoning, self-improvement, GUI data pipelines, and static or interactive benchmarks. Step-GUI distinguishes itself by emphasizing scalable reasoning-trajectory synthesis and efficient agent architecture for real-world GUI automation.
- Reasoning and Self-Improvement: Prior reasoning research uses chain-of-thought, synthetic prompting, and STaR to generate or bootstrap intermediate reasoning demonstrations.These methods target automated construction of reasoning exemplars from prompts or minimal seed data.
- Reasoning and Self-Improvement: Verification-oriented methods such as self-consistency, self-verification, and Self-Refine improve reliability through multiple paths or iterative critique.Other approaches address hallucinations or filter low-quality self-generated samples through external knowledge and offline reinforcement learning.
- GUI Agents: Recent GUI systems investigate data curation, cleaning, pretraining, post-training, and native GUI capabilities in both specialized and foundation models.The cited work includes UI-TARS, UI-TARS-2, OpenCUA, GUI-Owl, UITron, Claude, Seed, and Qwen series.
- GUI Benchmarks: GUI evaluation includes static benchmarks for grounding and planning alongside interactive benchmarks that measure task completion in realistic or virtualized environments.These benchmark families provide complementary views of GUI-agent capability.
- Positioning: Step-GUI focuses on scalable reasoning trajectory synthesis and efficient agent architecture design for real-world GUI automation.This focus contrasts with prior efforts centered on broader benchmark and training developments.
7. Conclusion
The work advances practical GUI agents across data, deployment, and evaluation through CSRS, Step-GUI, GUI-MCP, and AndroidDaily.
- The framework addresses data, deployment, and evaluation to bridge research capabilities and reliable daily assistance.It combines annotation-cost reduction, privacy-centric deployment, and real-world utility evaluation.
Appendix
Representative trajectories show the agent perceiving GUI elements, reasoning about requirements, and executing actions in sequence across multiple benchmarks.
- Successful trajectories span OSWorld, AndroidWorld, and AndroidDaily end-to-end tasks.The examples cover diverse graphical user interfaces and show visual states alongside agent actions.
- Each trajectory demonstrates ordered GUI interaction through perception, task reasoning, and action execution.
A. Trajectories on OSWorld
The OSWorld trajectories illustrate GUI agents completing varied desktop tasks involving applications, documents, spreadsheets, and settings.
- The trajectories cover installing a VS Code extension, restoring a closed tab, exporting a document, and changing application settings.
- Additional tasks include changing a recording folder, styling a slide, removing highlights, and constructing merged spreadsheet headers.
B. Trajectories on AndroidWorld
The AndroidWorld trajectories demonstrate mobile agents handling expense deletion, calendar scheduling, and browser-based numerical interaction tasks.
- The trajectories cover deleting specified expenses, creating a timed calendar event, and completing a multi-step browser form task.
C. Trajectories on AndroidDaily (End-to-End)
AndroidDaily presents realistic end-to-end mobile tasks spanning travel, shopping, and transportation scenarios. The trajectories require agents to combine information retrieval, cross-application interaction, and constraint-based choices.
- C. Trajectories on AndroidDaily (End-to-End): AndroidDaily trajectories cover cross-platform shopping, railway-and-map lookup, and flight-search tasks with explicit timing, location, and preference constraints.Examples include comparing delivery platforms, checking trains and distances, and filtering flights by airline or departure conditions.
- C. Trajectories on AndroidDaily (End-to-End): The shopping trajectory requires comparing Meituan Delivery and JD FlashEx before purchasing a specified Xianyu product with customized drink specifications.
- C. Trajectories on AndroidDaily (End-to-End): The railway-and-map trajectory combines a next-day train search through Railway 12306 with a distance lookup between Hangzhou East Railway Station and West Lake.
- C. Trajectories on AndroidDaily (End-to-End): The flight trajectories require finding round-trip or one-way options while applying date, route, airline, and red-eye-exclusion constraints.
D. GUI-MCP Example
The GUI-MCP example decomposes a high-level price-comparison request into parallel platform-specific GUI tasks and synthesizes their outputs. Local specialist agents execute application interactions, including a five-step JD.com search that extracts a product price.
- D. GUI-MCP Example: The example compares protein-powder prices across PDD, Taobao, and JD.com using MCP’s standardized interface for unified multi-platform control.
- D. GUI-MCP Example: The locally deployed JD.com specialist completes five steps—launching the app, searching, submitting the query, and extracting the ¥379 price—without further external-LLM interaction.
- D. GUI-MCP Example: GUI-MCP decomposes a price-comparison query into parallel platform-specific searches, executes them through on-device GUI agents, and synthesizes a structured response.