PaperlayerView exact revision on arXiv

Source-linked AI summary

MAI-UI Technical Report: Real-World Centric Foundation GUI Agents

Hanzhang Zhou, Xu Zhang, Panrong Tong, Jianan Zhang, Liangyu Chen, Quyu Kong, Chenglin Cai, Chen Liu, Yue Wang, Jingren Zhou, Steven Hoi

arXiv:2512.22047v1cs.CV

TL;DR

GUI agents still face gaps in user interaction, UI-only operation, deployment architecture, and robustness in dynamic environments. MAI-UI combines self-evolving data, device-cloud collaboration, and online RL, achieving state-of-the-art results across grounding and mobile-navigation evaluations.

  • Problem

    Reliable, robust, and secure real-world GUI-agent deployment is limited by missing agent-user interaction, UI-only operation, practical deployment architecture, and brittleness in dynamic environments.

  • Method

    MAI-UI uses a self-evolving pipeline with user interaction and MCP calls, task- and sensitivity-aware device-cloud collaboration, and online RL with system optimizations.

  • Results

    MAI-UI sets state-of-the-art results across GUI grounding, offline and online mobile navigation, and realistic evaluation settings.

  • Takeaways & Limitations

    The full-spectrum 2B, 8B, 32B, and 235B-A22B family targets deployment constraints while consistently outperforming strong comparable-scale baselines.

Abstract

from arXiv · show

The development of GUI agents could revolutionize the next generation of human-computer interaction. Motivated by this vision, we present MAI-UI, a family of foundation GUI agents spanning the full spectrum of sizes, including 2B, 8B, 32B, and 235B-A22B variants. We identify four key challenges to realistic deployment: the lack of native agent-user interaction, the limits of UI-only operation, the absence of a practical deployment architecture, and brittleness in dynamic environments. MAI-UI addresses these issues with a unified methodology: a self-evolving data pipeline that expands the navigation data to include user interaction and MCP tool calls, a native device-cloud collaboration system routes execution by task state, and an online RL framework with advanced optimizations to scale parallel environments and context length. MAI-UI establishes new state-of-the-art across GUI grounding and mobile navigation. On grounding benchmarks, it reaches 73.5% on ScreenSpot-Pro, 91.3% on MMBench GUI L2, 70.9% on OSWorld-G, and 49.2% on UI-Vision, surpassing Gemini-3-Pro and Seed1.8 on ScreenSpot-Pro. On mobile GUI navigation, it sets a new SOTA of 76.7% on AndroidWorld, surpassing UI-Tars-2, Gemini-2.5-Pro and Seed1.8. On MobileWorld, MAI-UI obtains 41.7% success rate, significantly outperforming end-to-end GUI models and competitive with Gemini-3-Pro based agentic frameworks. Our online RL experiments show significant gains from scaling parallel environments from 32 to 512 (+5.2 points) and increasing environment step budget from 15 to 50 (+4.3 points). Finally, the native device-cloud collaboration system improves on-device performance by 33%, reduces cloud model calls by over 40%, and preserves user privacy.

1 Introduction

MAI-UI targets reliable real-world GUI-agent deployment by adding agent-user interaction, MCP tool use, device-cloud collaboration, and online RL. Across grounding, offline and online navigation, and realistic evaluation, it reports state-of-the-art results across model sizes.

  • Challenges: MAI-UI addresses deployment gaps involving ambiguous user instructions, UI-only brittleness, practical execution architecture, and dynamic-environment generalization.The paper motivates proactive user interaction, expanded action spaces, device-cloud execution, and training in dynamic environments.
  • Method: A self-evolving data pipeline adds user interaction and MCP tool use to navigation training data and lets the agent select among UI actions, user engagement, and MCP calls.The pipeline iteratively updates the model and corpus using rejection-sampled, manually annotated, and automatic rollout trajectories.
  • Method: A native device-cloud system routes execution according to task state and data sensitivity using local and cloud agents plus shared trajectory memory.The local agent also monitors trajectories, while unified memory maintains information exchange between execution locations.
  • Method: Online RL scales training to 500+ parallel GUI environments and long-horizon tasks with up to 50 interactive steps, improving navigation accuracy and robustness.The framework uses asynchronous rollout and hybrid parallelism for dynamic environments.
  • Results: 73.5% on ScreenSpot-Pro with zoom-in, 91.3% on MMBench GUI L2, 70.9% on OSWorld-G, and 49.2% on UI-Vision with zoom-in demonstrate strong grounding performance.The reported grounding results also include 75.0% on OSWorld-G-Refine and 96.5% on ScreenSpot-V2.
  • Results: 76.7% on AndroidWorld and 41.7% on MobileWorld establish strong online and realistic-navigation results, while GUI Odyssey reaches 83.4% offline success.MobileWorld performance surpasses end-to-end GUI baselines by +20.8 and is competitive with agentic frameworks using GPT-5 or Gemini-3-Pro planners.

2 MAI-UI

The methodology integrates GUI grounding, self-evolving trajectory data, agent-user interaction, MCP augmentation, online RL, and native device-cloud collaboration. Figure 2 provides a demonstration trajectory of MAI-UI.

  • Method: MAI-UI combines GUI grounding training, self-evolving trajectory data, agent-user interaction, MCP tool augmentation, online RL, and device-cloud collaboration.These components constitute the paper’s integrated methodology.

2.1 System Overview

MAI-UI formulates GUI agents for grounding and mobile navigation, extending standard UI control with user interaction and MCP tool use. Its architecture combines broad action support, multiple model sizes, and device–cloud execution routed by task state and data sensitivity.

  • Task formulation: MAI-UI covers general GUI grounding and mobile GUI navigation as its two task categories.
  • Task formulation: Mobile navigation uses instructions, screenshots, action history, and standard operations such as clicking, swiping, or typing to select the next action.
  • Action space: MAI-UI extends mobile control with answer, ask_user, and MCP tool-use actions beyond pure GUI operations.
  • Model architecture: The model family spans 2B, 8B, 32B, and 235B-A22B variants, jointly trained on grounding, perception, and mobile-use navigation data.
  • Device–cloud collaboration: The native device–cloud system routes computation by task state and data sensitivity using local and cloud agents linked through shared trajectory memory.

2.2 GUI Grounding & Perception

MAI-UI builds grounding and perception capabilities from real and open-source GUI data, diverse task generation, and instruction perspectives. It trains these capabilities with instruction-as-reasoning, format and point-in-box rewards, and optional two-pass zoomed inference.

  • Data collection: The pipeline combines open-source datasets with screenshots collected from real GUI environments to produce multi-task perception and multi-perspective grounding data.
  • Data collection: MLLM-guided exploration of containerized operating systems continuously changes interface states and produces screenshots for diverse GUI data.
  • Perception data generation: For each screenshot, randomly selected UI elements support generated question-answering, captioning, and state-prediction tasks covering semantic, relation, and layout understanding.
  • Grounding data generation: Instructions are generated from appearance, function, location, and intent perspectives, which serve as both model inputs and explicit reasoning pathways.
  • Training paradigm: Supervised fine-tuning first instills instruction-as-reasoning, while reinforcement learning uses format and point-in-box rewards for grounding.
  • Inference: An optional zoom-in strategy predicts a coarse coordinate, crops and resizes a centered window, then refines the coordinate in a second pass.

2.3 Mobile GUI Navigation

MAI-UI trains mobile GUI navigation through a self-evolving trajectory pipeline and online reinforcement learning for dynamic, long-horizon tasks. The pipeline expands task and trajectory diversity, adds user interaction and MCP tool use, and alternates environment rollouts with trajectory-level policy updates.

  • Mobile GUI Navigation: MAI-UI’s navigation training combines supervised fine-tuning on diverse multi-step trajectories with online reinforcement learning in dynamic environments.The two-stage pipeline targets both initial navigation capability and real-world generalization.
  • Navigation Task Generation: The data pipeline generates tasks from application manuals, expert-designed tasks, and filtered open-source datasets.These sources expand task diversity and scale while preserving reachability and quality constraints.
  • Trajectory Synthesis: Task expansion varies critical parameters or replaces core task objects while preserving the original scenario and applications.L1 changes parameters such as dates or thresholds; L2 changes the core objects within the same scenario.
  • Trajectory Synthesis: Trajectories combine human annotations with model-based rollouts, followed by manual review and fine-grained correctness judgments.Human reviewers verify demonstrations, while an MLLM judge assesses agent trajectories at trajectory and step levels, retaining useful partial segments.
  • Iterative Rejection Sampling: The self-evolving loop uses each model iteration to generate new trajectories, mixes rejection-sampled and synthesized data, and fine-tunes the next model.Rejection sampling narrows the pass@1–pass@N gap, while novel trajectories raise the pass@N ceiling and align data with evolving capabilities.
  • Agent–User Interaction and MCP Augmentation: The action space includes UI operations, user engagement, and MCP tool calls, teaching the model when to request missing information or use external tools.User-interaction trajectories use ask_user actions, while MCP trajectories record tool schemas, arguments, results, and subsequent UI actions.

2.4 Device-Cloud Collaboration

MAI-UI uses native device–cloud collaboration to combine lightweight on-device execution with higher-capacity cloud recovery. A local agent monitors alignment, shared trajectory memory preserves context, and cloud execution is triggered selectively according to deviation and data sensitivity.

  • Motivation: On-device execution is limited by model size, while cloud execution introduces latency, privacy risks, and network dependence.The architecture is designed to address the limitations of relying exclusively on either deployment mode.
  • Architecture: The system contains a Local GUI Agent, Cloud GUI Agent, and Local Unified Trajectory Memory for consistent information exchange.The local agent acts both as executor and trajectory monitor.
  • Local Agent: The Local Agent observes screens, generates actions, and monitors whether execution remains aligned with the user instruction.Monitoring checks failures, repeated actions, incorrect inputs, and broader task deviations.
  • Cloud Agent: The Cloud Agent is called when monitoring detects trajectory deviation and receives the trajectory history plus an error summary.It uses its higher capacity to execute subsequent recovery steps.
  • Trajectory Memory: Unified trajectory memory records instructions, screenshots, thoughts, and actions, allowing either model to resume from any state.The memory projects shared history into the action spaces expected by device and cloud models.
  • Training: The on-device model is jointly trained for GUI navigation and alignment monitoring, including error-summary generation for cloud handoffs.This multi-task training avoids separate models and fragile prompt engineering.

2.5 MobileWorld Benchmark

MobileWorld evaluates mobile agents under more realistic conditions than standard GUI benchmarks. It adds long-horizon, cross-application tasks requiring agents to clarify ambiguous requests and choose between GUI actions and MCP operations.

  • Motivation: Standard mobile GUI benchmarks often use simple applications, limited app categories, and perfectly clear instructions with GUI-only interaction.These assumptions leave a gap between benchmark performance and real-world mobile utility.
  • Agent–User Interaction: Agent–user interaction tasks require detecting ambiguity and proactively seeking clarification rather than making incorrect assumptions.This capability directly tests interaction beyond idealized instructions.
  • MCP Tool Integration: MCP integration tasks require choosing intelligently between manual GUI navigation and API-based operations through MCP.The benchmark treats tool selection as an efficiency-relevant decision beyond pure UI manipulation.

3 Experiments

MAI-UI is evaluated across grounding, mobile-use, and real-world-oriented benchmarks using multiple model sizes and a four-stage training process. It achieves state-of-the-art or competitive results across desktop, mobile, cross-application, agent–user interaction, and MCP evaluations.

  • Experimental Setup: MAI-UI uses 2B, 8B, 32B, and 235B-A22B models trained through grounding SFT, navigation SFT, grounding RL, and online navigation RL.The final stage targets robustness and generalization in dynamic environments.
  • Benchmarks: The evaluation covers grounding, offline and online mobile use, and real-world-oriented tasks involving agent–user interaction and MCP tool use.AndroidWorld provides live-emulator evaluation, while MobileWorld tests practical deployment capabilities.
  • Grounding Capability: 73.5% on ScreenSpot-Pro, 91.3% on MMBench-GUI L2, 70.9% on OSWorld-G, and 49.2% on UI-Vision establish strong grounding performance.The 73.5% ScreenSpot-Pro and 49.2% UI-Vision results use adaptive zoom-in.
  • Grounding Capability: 96.5% on ScreenSpot-V2 is a new state-of-the-art result, while the 2B variant reaches 92.5% on the same benchmark.The reported results span mobile, desktop, and web interfaces.
  • Mobile GUI Navigation: 76.7% on AndroidWorld is a new state of the art, surpassing UI-TARS-2 at 73.3% and Gemini-2.5-Pro at 69.7%.The 32B variant also establishes state of the art at its scale with 73.3%.
  • MobileWorld: 41.7% overall success on MobileWorld remains competitive with agentic frameworks, while MAI-UI reaches 51.1% on User-Int. and 37.5% on MCP.The end-to-end model is compared with Gemini-3-Pro+UI-Ins at 46.3% and Claude-4.5-Sonnet+UI-Ins at 43.8%.

3.3 MCP Augmentation and Agent-User Interaction

MAI-UI extends GUI agents with MCP tool use and proactive agent-user interaction, while device-cloud collaboration dynamically balances execution, efficiency, and privacy. These capabilities support compressed workflows, on-device completion, trajectory recovery, and privacy-preserving execution.

  • MCP Augmentation: MCP tool calls compress multi-step GUI operations and enable desktop-oriented workflows on mobile.The examples include route comparison across applications and structured retrieval of GitHub commit metadata for email summarization.
  • Device-Cloud Collaboration: 42.7% of steps execute locally and 40.5% of tasks complete entirely on-device, reducing cloud calls.The device-cloud system dynamically selects execution based on task state and data sensitivity.
  • Agent-User Interaction: MAI-UI pauses underspecified file-sharing tasks to request missing recipient and email-body details from the user.The task requires proactive clarification before execution can safely continue.
  • Device-Cloud Collaboration: +6.9 percentage points in task success results from providing monitor-generated error summaries during switching.The summaries support the cloud agent’s trajectory recovery process.
  • Device-Cloud Collaboration: Privacy-sensitive password entry remains on-device when trajectory deviation is detected, and the local agent ultimately completes the task.The privacy monitor blocks cloud switching, so no sensitive content is transmitted.

3.5 Online RL Analysis

Online RL improves MAI-UI across model scales and benefits from enhanced optimization, larger interaction budgets, more parallel environments, and suitable image resolution. Qualitative cases further show recovery from unexpected interface interruptions and failures.

  • Performance Gains: Online RL improves performance across all tested model scales: +4.0 points for 2B, +6.0 for 8B, and +3.5 for 32B.The corresponding relative improvements are 8.9%, 9.3%, and 5.0%.
  • Ablations: +6.0 percentage points from enhanced GRPO exceeds standard GRPO’s +1.8-point gain by an additional +4.2 points.The enhanced method adds data curriculum, repetition penalty, and experience replay.
  • Ablations: +6.0 percentage points results when the interaction budget increases from 15 to 50 steps.Budgets of 15, 30, and 50 steps yield progressive improvements of +1.7, +3.8, and +6.0 points.
  • Ablations: 720p matches 1080p performance while providing a ∼50.1% speedup per step, whereas 540p substantially degrades performance.Higher resolutions add visual tokens and slow training and inference.
  • Ablations: 70.7% final performance follows scaling parallel environments from 32 to 512, compared with 65.5% using fewer environments.Fewer environments show early saturation, while broader parallel exploration accelerates learning.
  • Enhanced Robustness: RL-trained MAI-UI dismisses unexpected permission dialogs and pop-ups, resumes execution, and recovers from initially selecting the wrong application.These qualitative cases demonstrate robustness in dynamic task execution.

4 Related Works

Related work frames GUI grounding as mapping instructions to screenshot locations and GUI navigation as executing multi-step actions. It also identifies shortcomings in pure UI manipulation and highlights SFT+RL and MCP as routes toward more capable agents.

  • GUI Grounding: GUI grounding maps natural-language instructions to target-element locations in screenshots.It is presented as a foundational capability for GUI agents.
  • GUI Grounding: Pure RL grounding methods can surpass SFT-based methods, but pure RL overlooks benefits from an initial SFT stage and SFT+RL can suffer policy collapse.MAI-UI uses SFT to teach diverse reasoning perspectives before RL selects appropriate pathways.
  • GUI Navigation: GUI navigation extends grounding to multi-step action sequences, while current agents often remain copilots rather than standalone executors.Agent-user interaction is therefore identified as an important capability for practical deployment.
  • Tool-Augmented Agents: MCP tools can compress long UI sequences into a few API calls and unlock desktop workflows previously infeasible on mobile.This addresses limitations of pure UI manipulation.

5 Conclusion

MAI-UI combines self-evolving data, device-cloud collaboration, and online RL to advance GUI grounding and mobile navigation toward practical deployment. Results and case studies cover benchmark performance, agent-user interaction, MCP tool use, efficiency, and privacy.

  • 5 Conclusion: MAI-UI introduces a family of foundation GUI agents with self-evolving data, native device-cloud collaboration, and online RL.The data pipeline includes user interaction and MCP tool calls, while execution is routed by task state and data sensitivity.
  • 5 Conclusion: MAI-UI achieves state-of-the-art results on grounding benchmarks and online and offline mobile-use navigation evaluations.MobileWorld results and qualitative cases also verify agent-user interaction and MCP-enabled tool use.
  • 5 Conclusion: Device-cloud collaboration improves on-device performance while reducing cloud model calls and providing performance, privacy, and cost benefits.The conclusion positions these advances as progress toward practical foundation GUI agents for mobile use.

A Additional Grounding Results

Additional grounding results are presented for ScreenSpot-V2 and OSWorld-G-Refine, with comparisons organized in Tables 10 and 11.

  • ScreenSpot-V2 results are reported in Table 10.
  • OSWorld-G-Refine results are reported in Table 11.
  • Table 10 compares performance on ScreenSpot-V2, marking the best and second-best results distinctly.
  • Table 11 compares state-of-the-art models on OSWorld-G-Refine, identifying the best and second-best results.
Loading 2512.22047v1…