PaperlayerView exact revision on arXiv

Source-linked AI summary

Xiaomi-GUI-0 Technical Report

Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu, Hui Liu, Heng Qu, Qinzhuo Wu, Zhehao Yu, Tongbo Chen, Shiqi Cui, Anan Du, Shukai Jia, Yuanfa Li, Wei Liu, Yike Liu, Wenchao Lu, Zhenbo Luo, Haoyuan Sun, Jiatong Sun, Cheng Tan, Yajie Wang, Changqiao Wu, Tao Xiong, Jiahui Yang, Yuxuan Yuan, Ruoceng Zhang, Shaojie Zhang, Jian Zhu, Jian Luan, Cong Zou

arXiv:2606.31410v2cs.AI

TL;DR

Mobile GUI agents are often evaluated in offline, simulated, or standardized settings that do not fully reflect real application conditions. Xiaomi-GUI-0 uses a real-device-centered closed loop with error-driven training and achieves 72.0% success on RealMobile and 78.9% on AndroidWorld.

  • Problem

    Existing GUI-agent training and evaluation rely heavily on offline trajectories, static benchmarks, and simulated environments that incompletely capture real-device interactions and rare edge cases.

  • Method

    Xiaomi-GUI-0 combines real-device-dominant training and evaluation with an error-driven data flywheel that generates correction, reflection, and recovery supervision.

  • Results

    72.0% success on RealMobile and 78.9% on AndroidWorld, with improved success and robustness on real devices.

  • Takeaways & Limitations

    The report supports training and validating mobile GUI agents in real application environments to target stability, robustness, and deployability.

Abstract

from arXiv · show

Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation. However, existing GUI agents are trained and evaluated largely on offline trajectories, simulated environments, and standardized benchmarks. These differ substantially from real applications in interface layout, interaction logic, and abnormal-state distribution, and cannot faithfully characterize execution stability in real-world use, where account states, permission dialogs, payment authentication, and risk control continually reshape the state distribution and open a persistent gap between benchmark scores and real usability. To close this gap, we propose Xiaomi-GUI-0, a native multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop. At its core is a real-device-dominant hybrid infrastructure, where physical devices are the primary execution environment and sandboxes provide auxiliary support, so that data collection, training, rollout, and evaluation share an execution distribution close to real deployment. We construct multi-source training data spanning high-frequency head tasks, high-generalization data for long-tail intents, and capability-enhancement data for reflection and memory, and introduce an error-driven data flywheel that turns failure trajectories into corrected actions, reflective explanations, and recovery demonstrations. The model is trained through a progressive three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning. Evaluated on public benchmarks and our in-house RealMobile, Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.

1 Introduction

Xiaomi-GUI-0 addresses the gap between benchmark performance and reliable real-world mobile GUI operation through a real-device training and evaluation closed loop. Its infrastructure, error-driven data flywheel, progressive training pipeline, and RealMobile benchmark target execution stability, abnormal-state handling, and practical usability.

  • Motivation: Mobile GUI agents’ strong results on public datasets, static benchmarks, and simulated environments do not necessarily indicate robust operation on real mobile devices.Real applications continually alter interface states through sequential actions and expose conditions absent from clean offline trajectories and simulated evaluation.
  • Core approach: Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent developed within a real-device training and evaluation closed loop.The approach centers practical usability on training and validation in real application environments.
  • Infrastructure and data: The real-device-dominant hybrid infrastructure uses physical devices as the primary execution environment and sandboxes as auxiliary support for trajectory collection and task execution.It spans smartphones, tablets, and in-vehicle cockpits and supports multi-source data from real user requests and abnormal states.
  • Infrastructure and data: The error-driven data flywheel repairs failure states from real rollouts by annotating erroneous steps and generating correction, reflection, and recovery data.Interactive annotation identifies the critical error and correct action, while teacher-model scoring and takeover produce recovery trajectories.
  • Training: Training progresses through supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning to build task-execution capability.SFT establishes common app functions, UI operations, and canonical paths before later reinforcement-learning stages.
  • Evaluation: 72.0% success rate on RealMobile and 78.9% on AndroidWorld demonstrate Xiaomi-GUI-0’s evaluated performance across real-device and public simulated task execution.RealMobile evaluates end-to-end completion, exception recognition, and robustness under realistic devices, applications, account states, and abnormal conditions.

2 Infrastructure: Real Devices and Sandboxes

Xiaomi-GUI-0 uses a real-device-dominant hybrid infrastructure that combines physical devices with auxiliary sandboxes to balance real-world fidelity, scalability, and reproducibility. Its three-layer design manages resources, schedules tasks to suitable device and account states, and executes and archives complete interaction trajectories.

  • Hybrid infrastructure: Hundreds of physical phones and dozens of physical tablets provide the primary execution substrate, complemented by hundreds of sandbox instances for scalable and reproducible collection.Physical devices remain primary, while sandboxes provide auxiliary support.
  • Hybrid infrastructure: The hybrid environment routes virtualization-stable applications to sandboxes and applications dependent on real devices, networks, or native client states to physical devices.This routing addresses emulator incompatibility with anti-emulator detection and real-world abnormal states.
  • Three-layer design: The infrastructure separates responsibilities into resource, scheduling, and execution-and-collection layers.These layers respectively manage devices and sandboxes, match tasks to resources, and run tasks while archiving trajectories.
  • Resource layer: The resource layer covers nearly ten mainstream brands across smartphones, tablets, and in-vehicle cockpits, with phones covering the 100 most trafficked commercial applications.Coverage is maintained separately for each device type because interfaces vary by brand, operating system version, screen size, and resolution.
  • Scheduling layer: The scheduling layer dynamically matches tasks, devices, and account states using readiness profiles and continuously updated heartbeat and health reports.Profiles include device identity, type, resolution, installed applications, account availability, and risk-control level.
  • Execution and collection layer: The execution-and-collection layer initializes scheduled devices, captures initial observations, runs an observe–decide–act loop, and records complete interaction trajectories.Actions can come from a model service, human annotation interface, or maintenance tool.

3 Training Data Collection and Construction

Xiaomi-GUI-0 constructs training data through three progressive tiers covering frequent tasks, long-tail intents, abnormal states, and higher-level agent capabilities. An error-driven flywheel adds supervision aligned with the model’s execution errors by diagnosing failures, demonstrating recovery, and retraining on verified trajectories.

  • Data framework: Three progressive data tiers cover precise high-frequency execution, abnormal-state handling, long-tail generalization, and planning, reflection, and memory.The framework jointly addresses five supervision requirements through high-frequency, high-generalization, and capability-enhancement data.
  • High-frequency task data: High-frequency data uses expert-annotated real-user tasks across devices and starting pages, adding approximately 5,000 samples spanning 14 abnormal-state types.Examples include expired sessions, captchas, payment authentication, permission prompts, and network errors, with differentiated handling strategies.
  • High-generalization data: High-generalization data scales long-tail coverage through a five-level function tree, behavior-bucket query synthesis, hybrid-infrastructure rollouts, and trajectory- and step-level cleaning.Function-point back-tagging closes the coverage loop, while the automated pipeline expands coverage across applications, functions, and task complexity.
  • Capability-enhancement data: Capability-enhancement data explicitly annotates intermediate reasoning to improve planning, reflection, memory, and summarization in long-horizon GUI tasks.Structured CoT annotations guide goal decomposition, context tracking, intermediate observation summaries, and strategy adjustment.
  • Error-driven data flywheel: The error-driven flywheel executes, diagnoses, corrects, verifies, and retrains on failures, combining interactive first-error correction with teacher-model recovery demonstrations.Each step records screenshots, history, actions, scores, rationales, execution sources, and takeover flags, exposing both incorrect behavior and its recovery rationale.

4 Model Training

Model training uses a three-stage curriculum that progresses from dense supervised signals to step-level and trajectory-level reinforcement learning. The stages initialize a stable policy, correct local response errors, and optimize long-horizon GUI behavior in asynchronous online rollouts.

  • Training curriculum: Three phases—SFT, Step RL, and Agentic RL—progress from dense supervision to increasingly sparse feedback while moving from policy initialization to complete-trajectory optimization.SFT initializes the policy, Step RL corrects individual responses, and Agentic RL optimizes long-horizon trajectories.
  • Supervised fine-tuning: SFT jointly trains on four data sources to cover high-frequency tasks, long-tail intents, structured reasoning, and recovery from erroneous states.The sources are high-frequency task, high-generalization, agent-capability enhancement, and error-flywheel corrected-action data.
  • Step-level reinforcement learning: Step RL targets local response errors, including malformed actions, incorrect parameters, incomplete reasoning, missing memory updates, and reasoning-action inconsistencies.GSPO operates at the sequence level with group-relative advantages because correctness depends on the complete structured response rather than individual tokens.
  • Step-level reinforcement learning: A hierarchy-triggered cascade reward evaluates responses in fixed order, using low-cost rule checks first and LLM-as-judge checks for more expensive requirements.Evaluation exits at the first failing level, which determines the assigned reward.
  • Agentic reinforcement learning: Agentic RL optimizes multi-step trajectories to improve state tracking, error recovery, cross-application consistency, and final task completion.Its online framework decouples GPU-bound inference, device-bound execution, and data transfer through independent asynchronous processing paths.

5 The RealMobile Benchmark

RealMobile is a real-device benchmark built from real user traffic for evaluating GUI agents on live applications and realistic task distributions. It combines broad capability coverage, cross-application tasks, fine-grained partial-credit scoring, and dual verification.

  • Benchmark construction: RealMobile runs 100 hand-crafted tasks across 14 live applications on physical Android devices, using real user traffic and reproducible evaluation procedures.Its construction includes expert task design, golden trajectories, and additional successful and failed trajectories collected with multiple agents.
  • Task distribution: 57% of tasks span multiple applications, including 38% across two, 9% across three, and 10% across four or more applications.Across the 100 tasks, Figure 7 counts 193 application instances and shows coverage across capability domains.
  • Capability coverage: The benchmark organizes tasks into four capability domains: Foundation, Safety & Reflection, Memory & Knowledge, and Complex Reasoning & Planning.These domains cover basic interaction, safety boundaries and self-correction, information retention, and long-horizon reasoning and planning.
  • Scoring protocol: Tasks are decomposed into typically 3 to 6 verifiable sub-goals, with equal credit for each completed sub-goal instead of binary pass/fail scoring.This protocol distinguishes partial progress in multi-step tasks and provides more diagnostic evaluation signal.
  • Verification: The dual verification framework combines XML structure matching with logical semantic rules to verify sub-goals while accepting semantically equivalent actions.It is designed to balance brittle XPath matching against application-specific programmatic state checking.

6 Experiments

Experiments evaluate Xiaomi-GUI-0 on RealMobile and AndroidWorld, with strong overall success and detailed domain-level results. The model reaches perfect Foundation-domain performance, while Safety & Reflection remains its weakest area and Complex Reasoning & Planning is comparatively strong.

  • Overall results: 72.0% success on RealMobile and 78.9% on AndroidWorld were achieved by Xiaomi-GUI-0-30B-A3B, averaged across four runs.The four-run mean accounts for relatively large evaluation variance on these benchmarks.
  • Overall results: Xiaomi-GUI-0 attained the best AndroidWorld result among evaluated models, exceeding UI-TARS, GUI-Owl, UI-Venus, and several closed-source systems.It exceeded Gemini 3.1 Flash at 58%, Claude Opus 4.7 at 60%, and Claude Opus 4.6 at 33%.
  • RealMobile domains: 100.0% success on Foundation matched the strongest proprietary models, indicating that basic UI operation is approaching saturation.Foundation no longer discriminates strongly between capable agents according to the reported comparison.
  • RealMobile domains: 80.5% success on Complex Reasoning & Planning approached Gemini 3.1 Pro at 82.9% and exceeded Seed 2.0 Pro at 73.2%.The strongest open-source baseline reached only 31.7%.

7 Related Work

Related GUI-agent work spans modular VLM-based orchestration, emulator or sandbox benchmarks, and emerging reinforcement-learning approaches. These directions address scalable evaluation and improved long-horizon execution, but differ in execution environments and system design.

  • VLM-based GUI agents: Early GUI agents assemble planner, executor, reflector, and memory modules around general-purpose VLMs such as GPT-4o, Gemini, or Qwen-VL.Their performance is largely bounded by the underlying VLM, while deeper modules and more interaction rounds increase inference latency.
  • GUI benchmarks: Mobile GUI benchmarks divide broadly into emulator or sandbox environments and real-device evaluations.AndroidWorld exemplifies reproducible emulator or sandbox tasks verified with accessibility trees, UI trees, or VLM-based screenshot judges, but these settings bias state distributions toward simplified environments.
  • Reinforcement learning for GUI agents: Asynchronous online reinforcement learning and related methods establish reinforcement learning as a central direction for improving long-horizon execution, error recovery, and environmental adaptability.OpenClaw-RL improves performance through asynchronous online RL, while the broader line of work includes model fusion and large-scale virtual-machine rollouts.

8 Conclusion

The report introduces Xiaomi-GUI-0, a native end-to-end multimodal GUI agent for real mobile environments. It addresses the gap between benchmark performance and real-world usability through training and evaluation in a real-device-centered closed loop.

  • Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent designed for real mobile environments.
  • The system targets the persistent gap between high benchmark scores and real-world usability.
  • Training and evaluation occur in a closed loop centered on real-device execution rather than static benchmarks or simulated environments.The report states that static benchmarks and simulated environments only partially reflect deployment conditions.

Contributions and Acknowledgments · Appendix

The contributions and acknowledgments list names all contributors in alphabetical order by last name. The listed contributors include Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao, Niu Lian, Fazhan Liu, Hui Liu, Heng Qu†, Qinzhuo Wu, and Zhehao Yu, alongside additional contributors.

  • Contributions and Acknowledgments: Contributors are listed in alphabetical order by their last names.
  • Contributions and Acknowledgments: The list includes Wanxia Cao, Chengzhen Duan, and Pei Fu.
  • Contributions and Acknowledgments: The listed contributors also include Pengzhi Gao, Niu Lian, Fazhan Liu, and Hui Liu.
  • Contributions and Acknowledgments: Heng Qu† appears among the named contributors.
  • Contributions and Acknowledgments: Qinzhuo Wu and Zhehao Yu are included in the contributor list.
  • Contributions and Acknowledgments: Additional contributors include Tongbo Chen, Shiqi Cui, Anan Du, Shukai Jia, Yuanfa Li, Wei Liu, and Yike Liu.
  • Contributions and Acknowledgments: The remaining listed contributors are Wenchao Lu, Zhenbo Luo, Haoyuan Sun, Jiatong Sun, Cheng Tan, and Yajie Wang.
  • Appendix: The list concludes with Changqiao Wu, Tao Xiong, Jiahui Yang, Yuxuan Yuan, Ruoceng Zhang, Shaojie Zhang, and Jian Zhu.

A Data Collection Applications

Data collection spans 140 applications across three subsets: commercial mobile, tablet and cockpit, and AndroidWorld applications. These subsets cover everyday mobile use, large-screen interaction patterns, and stable benchmark evaluation.

  • Application subsets: 100 high-frequency commercial mobile applications target everyday mobile usage.This subset forms the largest application group in data collection.
  • Application subsets: 20 tablet and cockpit applications target large-screen interaction patterns.The subset extends coverage beyond standard mobile interfaces.
  • Application subsets: 20 AndroidWorld applications provide a stable benchmark environment.AndroidWorld is included as a distinct application subset for data collection.

B Action Space

Xiaomi-GUI-0 uses a compact unified action space in which each step selects one structured action spanning device interaction, communication, and task completion.

  • Action categories: The unified action space covers touch operations, text input, system navigation, user communication, and task termination.These action categories are summarized in Table 7.
  • Action execution: At each interaction step, the model selects exactly one action and emits its arguments as a JSON object.This enforces a single structured action per step.

B.1 Anomaly Semantics · C Function Tree Example · D Behavior-Bucket Examples

The paper defines fourteen anomaly handoff semantics for reliable GUI control, illustrates function-tree navigation in Bilibili, and uses behavior buckets to synthesize intent-level queries. Together, these structures represent abnormal states, application access paths, and coherent user motivations.

  • B.1 Anomaly Semantics: More than fifty online abnormal-state causes are consolidated into fourteen handoff semantics mapped to valid Fail.type values.These semantics cover states that block reliable automation.
  • B.1 Anomaly Semantics: When abnormal states prevent reliable automation, the model should stop, pause, or hand control back to the user instead of continuing UI actions.The prescribed response is state-dependent handoff rather than unrestrained action execution.
  • C Function Tree Example: The Bilibili function-tree example enumerates sibling functions at each level while expanding one representative branch toward a concrete function point.This simplified presentation preserves both breadth at each node and depth along one access path.
  • C Function Tree Example: The function tree captures which functions can be sampled at each node and how a top-level application entry leads to a concrete function point.The example makes application structure and access paths explicit.
  • D Behavior-Bucket Examples: Behavior buckets abstract user intent by combining exposed functions into coherent requests rather than merely recording how functions are reached.Each bucket represents a stable usage motivation and contains several concrete behavioral phrases.
  • D Behavior-Bucket Examples: Query synthesis first samples a behavior bucket and then binds concrete entities, filters, pages, or functions.This procedure converts an intent-level abstraction into a specific behavioral query.

D.1 Single-Application Example: Xiaohongshu · D.2 Cross-Application Example: Weibo and Bilibili · E System Prompt

The examples define single-application behavior buckets for Xiaohongshu and cross-application relations for Weibo and Bilibili, covering diverse consumption, creation, social, shopping, and relay or parallel tasks. The system prompt constrains Xiaomi-GUI-0 to perceive context, reason over plans and memory, and emit exactly one tool call in a strict XML-like format per step.

  • D.1 Single-Application Example: Xiaohongshu: Xiaohongshu behavior buckets span content consumption, local exploration, shopping, creation, social interaction, creator growth, and personal settings.All behaviors are completed within Xiaohongshu while covering these user motivations.
  • D.1 Single-Application Example: Xiaohongshu: Local exploration includes city, distance, and whole-city filtering plus discovery of nearby check-in spots, trips, exhibitions, performances, and CityWalk routes.The example also includes local offline activity discovery.
  • D.1 Single-Application Example: Xiaohongshu: Shopping and live interaction cover product-category browsing, image-based product search or recognition, wishlist and cart management, livestream channels, night-event tasks, lotteries, and reward-value browsing.These buckets combine e-commerce and market shopping with themed livestream-plaza viewing and interaction.
  • D.1 Single-Application Example: Xiaohongshu: Creation and social buckets support media selection, captions, publishing, creator tools, group creation, friend adding, interaction notifications, and direct-message checking.The listed creation tools include cover templates, AI effects, and filters.
  • D.2 Cross-Application Example: Weibo and Bilibili: Cross-application buckets require relay, contrast, or parallel relations rather than concatenating independent single-application tasks.The Weibo–Bilibili example assigns real-time trends, social discussion, images, and fan activity to Weibo, with longer-form video activity on Bilibili.
  • D.2 Cross-Application Example: Weibo and Bilibili: Weibo and Bilibili support parallel idol-support and leisure tasks, including fan-community actions, music-video viewing, text-image browsing, long-form videos, and travel-guide consumption.These examples may also relay local popularity signals or fragmented event updates into detailed video exploration.
  • E System Prompt: The agent prompt requires screen perception, interaction-history review, device and foreground-app context, reasoning with observation, reflection, planning, decision, and memory, followed by exactly one tool call per step.Available actions include Tap, LongPress, Swipe, Type, Search, Complete, and Speak, with tool calls encoded as JSON inside a fixed three-span XML-like response.
  • E System Prompt: Operational constraints use normalized [x, y] coordinates with 3-decimal precision, require dismissing unrelated pop-ups, and trigger self-correction after three unchanged or looping steps.The output must contain only <think>, <action>, and <tool_call> spans in that order, with no extra text.

F Case Study

The case studies demonstrate Xiaomi-GUI-0’s real-device capabilities in end-to-end task completion and mid-trajectory error recovery. The agent decomposes instructions into GUI actions and uses explicit reflection to revise plans when observed states deviate from expectations.

  • End-to-end task completion: The complete trajectory shows the agent observing screens, decomposing instructions into sub-goals, and executing GUI actions through task completion on a physical device.This demonstrates a unified action space spanning intent understanding to task completion.
  • Mid-trajectory error recovery: When the observed state deviates from the expected outcome, the agent records the discrepancy, revises its plan, and selects a corrective action.Explicit reflection enables recovery from intermediate uncertainty instead of continuing the original trajectory.
Loading 2606.31410v2…