PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai, Pengxiang Li, Liangyu Chen, Yue Wang, Steven Hoi

arXiv:2607.28227v1cs.AIcs.CV

TL;DR

GUI agents remain largely optimized for simulated benchmarks, leaving a gap between benchmark performance and real-world utility. Qwen-UI-Agent addresses this with a real-world-centric agent spanning mobile, computer-use, web, and DeepSearch, achieving state-of-the-art mobile-use performance and competitive results on computer- and browser-use tasks.

  • Problem

    GUI agents remain largely optimized for simulated benchmarks, limiting evidence of reliable real-device execution and broader real-world utility.

  • Method

    Qwen-UI-Agent combines real-device and sandbox environments with a unified GUI+CLI action space, agent-driven data flywheel, online reinforcement learning, and a cross-platform harness.

  • Results

    92.2% on MobileWorld-Real, with state-of-the-art mobile-use performance and competitive results on computer-use, browser-use, and GUI-grounding benchmarks.

  • Takeaways & Limitations

    Qwen-UI-Agent provides a foundation GUI agent evaluated across real-device mobile, computer-use, browser-use, and DeepSearch environments.

  • Takeaways & Limitations

    Real-device mobile results rely on AutoJudge, whose remaining errors may introduce minor uncertainty into the reported results.

Abstract

from arXiv · show

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.

1 Introduction

Qwen-UI-Agent is introduced as a real-world-centric foundation GUI agent designed to close the gap between simulated benchmark performance and practical utility through reliable real-device operation, cross-platform workflows, hybrid GUI–CLI execution, long-horizon planning, proactive services, and reduced human effort. It combines these capabilities across mobile, computer-use, web, and DeepSearch settings, achieving strong results across real-device, benchmark, browser-use, DeepSearch, and GUI-grounding evaluations.

  • Motivation and system scope: Qwen-UI-Agent targets real-world utility by operating reliably on real devices, completing long-horizon tasks, crossing platforms, initiating services proactively, combining GUI with CLI, and evolving with less human effort.The report frames these capabilities as the transitions needed to bridge the gap between simulated benchmarks and real-world use.
  • Multi-domain capabilities: Qwen-UI-Agent supports integrated workflows across mobile, web, computer-use, and DeepSearch environments while preserving context and task state across devices.Its harness can invoke DeepSearch for external information retrieval and verification before resuming GUI interaction.
  • Hybrid action space: Qwen-UI-Agent uses a unified action space combining GUI operations, bash-based CLI commands, and batched actions within a single trajectory.The design selects GUI or CLI actions according to suitability and batches compatible actions to reduce unnecessary steps.
  • Evaluation results: 92.2% on MobileWorld-Real, 97.5% on AndroidDaily, and 82.1% on MobileWorld demonstrate Qwen-UI-Agent’s strong mobile-use performance.On MobileWorld-Real, it exceeds Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro by 6.0, 7.5, 6.8, and 3.5 percentage points, respectively.

2 Qwen-UI-Agent · 2.1 System Overview · 2.2 Environment Infrastructure

Qwen-UI-Agent is a real-world-centric foundation GUI agent spanning mobile, desktop, web, and information-seeking environments through unified observations and actions. Its infrastructure combines scalable sandboxes, real-device execution, hybrid GUI+CLI control, and a common interface for heterogeneous environments.

  • 2 Qwen-UI-Agent: Qwen-UI-Agent integrates sandboxed and real-world environments for mobile-use, computer-use, browser-use, and DeepSearch tasks under a unified action space.The system methodology also includes automated data generation and training infrastructure.
  • 2.1 System Overview: Tasks may use mobile devices, web browsers, computer systems, DeepSearch systems, or combinations spanning multiple platforms.The formulation covers both single-environment tasks and cross-platform workflows.
  • 2.1 System Overview: Batched actions execute ordered operations consecutively within one decision step, reducing unnecessary inference and observation steps when further feedback is unnecessary.A decision may produce either one action or an ordered sequence, with K_t > 1 denoting batched execution.
  • 2.1 System Overview: The action space combines platform-specific GUI operations with cli_command, extending execution beyond graphical manipulation while preserving user control over consequential operations.GUI actions cover mobile, web, and desktop environments, while CLI complements them for programmatic execution.
  • 2.1 System Overview: Real-device mobile execution, multi-domain task execution, and broader real-world interaction form complementary capabilities enabled by the unified formulation.The real-device environment is used throughout model capability development and evaluation to reduce the simulation-to-reality gap.
  • 2.2 Environment Infrastructure: 10,000 isolated sandbox environments can run in parallel across mobile-use, computer-use, browser-use, and DeepSearch tasks.These environments provide controlled, resettable states for training, evaluation, and ablation while retaining domain-specific applications, context, and verification.
  • 2.2 Environment Infrastructure: Over 100 physical devices and over 150 applications comprise the real-device mobile runtime, managed by a health-aware scheduler.The scheduler tracks device, application, account, network, and display health, leasing eligible targets and rerouting execution when failures occur.
  • 2.2 Environment Infrastructure: A unified environment interface standardizes lifecycle and agent-facing inputs and outputs while delegating execution, reset, and verification to environment-specific adapters.Its asynchronous lifecycle includes acquire, reset, step, evaluate, tear_down, and release, enabling one agent and training pipeline to operate across heterogeneous environments.

2.3 Agent-Driven Data Flywheel

Qwen-UI-Agent uses an AutoResearch-style, agent-driven data flywheel to automate task, environment, verification, training, evaluation, and failure-analysis stages. The flywheel bootstraps domain coverage, filters trajectory data for supervision, and iteratively targets diagnosed capability weaknesses.

  • Overall Procedure: The flywheel has two stages: domain capability bootstrapping and an iterative refinement loop that generates data for subsequent training iterations.Bootstrapping uses strong foundation models to analyze mobile, desktop, and web capabilities, then generate initial tasks, environments, and candidate trajectories.
  • Knowledge- and Capability-Aware Task Synthesis: Task synthesis separates knowledge coverage from capability demand, specifying what the agent must know and how it must reason and act.Knowledge coverage concerns domain-specific operational knowledge, while capability demand concerns task completion behavior.
  • Agent-Driven Environment and Verifier Synthesis: Agents construct executable environments and verifiers for synthesized tasks, including application states, files, records, accounts, and cross-application context.Environment State Synthesis provides the reproducible interactive conditions required for training.
  • Step-Level Judging for Scalable SFT and Executable Verification for Online RL: Step-level data filtering yields SFT performance comparable to or better than complete trajectories selected by executable verifiers.The procedure retains correct contiguous steps, initial reflection or exploration steps, and recovery segments from erroneous states to valid paths.
  • Failure Analysis Driven Iteration: Failure analysis distinguishes model failures from environment, task, and verifier failures, then maps model errors to structured causes that guide the next optimization iteration.Example causes include missing application knowledge, constraint violations, and state-tracking errors.

2.4 Training

Qwen-UI-Agent training combines domain-conditioned expert specialization with capability-preserving in-distribution data and efficient long-trajectory SFT. It adds action-level and online reinforcement learning to improve local reliability and end-to-end task success through scalable environment interaction.

  • Supervised Fine-Tuning: Domain-conditioned expert training specializes models for mobile, desktop, and web demands, mixes cross-domain data, and merges expert checkpoints into one deployable model.Cross-domain mixing preserves transferable capabilities and reduces overfitting to each target domain.
  • Supervised Fine-Tuning: In-distribution examples sampled from the starting model and verified as correct are mixed with GUI trajectories to preserve general reasoning and agentic capabilities.The paper reports these capable examples are substantially more effective for capability preservation than challenging examples the model fails to solve.
  • Supervised Fine-Tuning: n = 5-step sliding windows advanced by n −1 = 4 steps reduce repeated processing of overlapping visual and textual context during long-trajectory SFT.Adjacent windows retain a one-step overlap.
  • Action RL: Action RL targets recurring local failures—including quantity incompleteness, premature completion, repetitive loops, and long-tail action-selection errors—with action-aware rewards.The structured reward accounts for format validity, action-type correctness, argument quality, sensitive-action penalties, and repetition penalties.
  • Online RL: Approximately 10,000 validated task–verifier pairs support Online RL, which uses unified infrastructure to optimize end-to-end success through interaction across heterogeneous GUI environments.Online RL addresses delayed trajectory-level consequences that action-level optimization does not fully capture.

2.5 Harness Layer: Proactive Service and Cross-Platform Task Execution

The harness layer extends GUI agents beyond reactive, isolated tasks by initiating assistance from actionable digital signals and executing stateful workflows across phones and computers. It represents ongoing matters as affairs, forms dependency-aware tasks, and coordinates appropriate executors while preserving shared context and task state.

  • Harness Layer: The harness adds proactive service and cross-platform execution, shifting initiation from explicit instructions to actionable events and carrying workflows across phones and computers.These capabilities preserve shared context, dependencies, and task state across platforms.
  • Mobile Notifications as a High-Value Signal Source: Notifications provide timely, broadly available, user-controlled signals, but the harness treats them as partial observations and gathers more information when needed.Mobile notifications aggregate communication, travel, finance, logistics, and scheduling events, with access governed by per-application permissions.
  • From a Notification to a Proactive Workflow: The stateful pipeline parses notifications into events, associates them with persistent affairs, derives executable tasks, and uses outcomes to update memory and future behavior.An event records a time-specific occurrence, while an affair represents an ongoing matter across events, applications, and days.
  • From a Notification to a Proactive Workflow: The harness reasons over urgency, missing information, unfinished actions, and cross-affair dependencies, then prepares decision-ready options with appropriate executors before controlled execution.Preparatory actions can include retrieving alternative trains, comparing travel times, or checking refund status; operations with external consequences remain controlled.
  • Cross-Platform Task Execution: Cross-platform execution jointly grounds mobile and desktop states, decomposes objectives into dependency-aware subtasks, selects GUI, search, file, CLI, or API executors, and runs independent subtasks concurrently.The shared state includes device-tagged observations, artifacts, tool outputs, and subtask results.

3 Experiments

Qwen-UI-Agent achieves leading mobile-use results and competitive performance across computer use, browser use, DeepSearch, and GUI grounding. Evaluations also test transfer to live devices, hybrid GUI-CLI execution, and broad capabilities beyond specialized GUI tasks.

  • Mobile Use: MobileWorld evaluates long-horizon, cross-application Android workflows, while MobileWorld-Real tests transfer to held-out live devices with real Apps, accounts, content, and networks.MobileWorld-Real reports environment errors separately and retains complete trajectories for auditing and reproducibility.
  • Mobile Use: 92.2% on MobileWorld-Real and 97.5% on AndroidDaily establish first-place mobile results, surpassing Seed 2.1 Pro at 88.7% and 95.2%, respectively.The 35B-A3B variant reaches 87.4% on MobileWorld-Real and 93.9% on AndroidDaily while activating only 3B parameters per token.
  • Computer Use: 79.5% on OSWorld-Verified ranks second behind Claude Opus 4.8, while OSWorld-v2 reaches 40.0% partial score and 13.9% binary completion.On OSWorld-v2, Qwen-UI-Agent exceeds GPT-5.5 by 0.9 percentage points in binary completion but trails it by 9.5 points in partial score.
  • Browser Use and DeepSearch: 73.6% on WebArena ranks first, exceeding Claude Opus 4.8 at 71.9%, while BrowseComp and BrowseComp-ZH reach 64.1% and 75.0%, respectively.DeepSearch complements GUI interaction by acquiring, comparing, and verifying external knowledge before subsequent application actions.
  • GUI Grounding: 81.5% on ScreenSpot-Pro with zoom-in leads all compared models; the 27B variant also scores 97.5% on ScreenSpot-V2 and 92.6% on MMBench-GUI L2.Without zoom-in, ScreenSpot-Pro reaches 76.6%, demonstrating accurate localization across common interfaces and complex professional software.
  • General Capability: Qwen-UI-Agent retains general reasoning and agentic capabilities while substantially outperforming specialized GUI models on general and agentic benchmarks.Qualitative computer-use examples illustrate hybrid execution combining CLI commands with GUI actions in multi-application workflows.

4 Behavioral Analysis of GUI Agent Execution

The behavioral analysis identifies distinct real-device failure modes arising from limited physical-device exposure, while showing that Qwen-UI-Agent combines GUI, CLI, and batched execution to improve workflow efficiency. Action RL further corrects erroneous decisions at step level, reducing reasoning tokens while improving task success.

  • Real-device failures: 52.0% of failures involve real-world scenario challenges, while 40.3% reflect execution capability limitations rooted in scarce real-device interaction experience.The two dimensions cover interface conditions underrepresented in simulators and deficits in exploration, recovery, and state tracking.
  • Execution capability limitations: 19.5% of failures are exploration failures, 14.3% erroneous action loops, and 6.5% lost execution state.The patterns involve shallow-page revisiting, repeated failed actions, and losing progress across app switching, interruptions, and subtasks.
  • Real-world scenario challenges: 24.7% of failures are UI misreading, 18.2% pop-up interference, and 9.1% physical widget control.These involve ambiguous stateful semantics, non-deterministic disruptions, and fine-grained closed-loop manipulation such as sliders and date pickers.
  • GUI–CLI execution: 55.1% of OSWorld-v2 actions are CLI operations, appearing in 98.2% of tasks; on OSWorld-Verified, the corresponding figures are 40.7% and 92.0%.The near-balanced GUI–CLI distribution indicates both channels are primary execution mechanisms in computer-use trajectories.
  • Batched execution: 3.1 primitive actions per batch reduce observation–reasoning–execution cycles, with batched execution appearing in 62.1% and 88.9% of tasks across the two benchmarks.GUI-only batches dominate, while CLI-only and mixed GUI+CLI batches demonstrate cross-modal execution support.
  • Action RL: 21.3% fewer reasoning tokens accompany more than 7% higher task SR after action RL training.The analysis attributes this shift to fine-grained correction of erroneous action decisions rather than aggregate success improvement alone.

5 Related Work

Recent GUI-agent research is shifting from prompt-only orchestration and benchmark optimization toward native multimodal policies designed for realistic deployment. Work across mobile, browser, and computer use emphasizes memory, reflection, reinforcement learning, structured information, and reliable long-horizon execution.

  • Cross-cutting trends: GUI-agent research increasingly favors native multimodal policies, self-evolving data engines, online reinforcement learning, and structured memory over prompt-only orchestration.The field is also moving beyond benchmark-only optimization toward real-device execution and other realistic deployment concerns.
  • Mobile Use: Mobile agents are advancing beyond screenshot-grounded action prediction through memory, reflection, reinforcement learning, agentic harnesses, unified GUI models, data flywheels, and multi-turn training.Examples include Mobile-Agent-v3.5 and UI-Venus-1.5 for cross-platform scaling, Step-GUI and UI-TARS-2 for iterative improvement, and MAI-UI for hybrid GUI-MCP operations and device–cloud collaboration.
  • Browser Use: Browser-use agents leverage structured information such as DOM trees, accessibility trees, and browser APIs alongside visual understanding and browser actions for end-to-end web navigation.WebVoyager and SeeAct represent early multimodal approaches, while OpenAI Operator, Google Project Mariner, and UI-TARS are cited as recent projects.
  • Computer Use: Computer-use agents target general interaction with desktop environments spanning diverse software and productivity applications.Compared with mobile and web agents, they face larger action spaces, more diverse interaction states, and longer horizons, making reliable grounding and execution verification critical.

6 Conclusion

Qwen-UI-Agent is presented as a real-world-centric foundation GUI agent designed for reliable operation across devices and platforms, combining GUI and CLI execution with proactive services. Its system integrates scalable environments, real-device mobile runtime support, agent-driven data improvement, reinforcement-learning training, and a harness layer.

  • Qwen-UI-Agent targets real-device operation, cross-platform workflows, GUI-plus-CLI execution, and proactive service initiation.
  • Its architecture integrates scalable sandbox environments with a robust real-device mobile runtime.
  • The agent uses a unified GUI+CLI action space, an agent-driven data flywheel, and training that combines SFT, action RL, and online RL.
  • A harness layer supports the agent’s real-world operation and workflow execution.

7 Limitations and Future Directions

The report’s main limitation is that real-device mobile trajectories are evaluated with AutoJudge rather than deterministic verification or manual expert examination. Future work targets more efficient and reliable long-horizon execution, cross-domain training, scalable high-fidelity environments, and stronger safety and user control.

  • Limitations: AutoJudge evaluates real-device mobile trajectories instead of deterministic verifiers or manual expert examination.Deterministic state-based verification is generally infeasible on physical phones, while manual adjudication would require substantial expert effort and introduce inter-annotator variation.
  • Efficient GUI Execution: Interaction latency remains a major obstacle because each GUI step typically requires observation, model inference, and an environment transition.Latency accumulates over long trajectories, motivating more efficient GUI execution mechanisms.
  • Harness-Assisted Long-Horizon Workflows: The harness could provide long-horizon support through context compression, task decomposition, memory management, progress tracking, and structured tool use.This complements hybrid GUI+CLI actions, batched actions, and online RL for improving execution efficiency and reliability.
  • Large-Scale Cross-Domain Online RL: Long-horizon online RL should become more efficient and stable while jointly training across browser-use, mobile-use, and computer-use environments.Slow, variable-length rollouts and sparse or delayed rewards make these settings costly and motivate improved data efficiency and cross-domain generalization.
  • Scalable Synthesis of High-Fidelity Environments: High-fidelity synthetic environments could complement expensive real-device environments by approximating application behavior while remaining scalable, controllable, and open-source in methodology.The report says the constructed environments were not integrated into model training in time, with the synthesis methodology planned for future release.
  • Safety, User Control, and Personalization: More systematic safety evaluations and safety-oriented training are needed to preserve user control as agents operate across applications and devices.Current safety-sensitive scenarios teach the model to invoke call_user and return control when additional input or authorization is required.

8 Contributions

The supplied passage identifies the paper’s authors but does not describe specific contributions.

  • Authorship: The paper is authored by Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, and coauthors.The passage lists 16 authors, including Yue Wang and Steven Hoi.

A Appendix · A.1 Validating AutoJudge for Real-Device Evaluation

Because real-device tasks lack deterministic verifiers and manual review is difficult to scale, the authors validate AutoJudge against independent human annotations. AutoJudge achieves high agreement, receives favorable expert adjudication, and is adopted as the primary evaluator for real-device experiments.

  • A.1 Validating AutoJudge for Real-Device Evaluation: Real-device tasks often lack deterministic, state-based verification because third-party Apps hide internal states and outcomes depend on changing online conditions.Outcomes may also depend on account status and interactions across multiple Apps.
  • A.1 Validating AutoJudge for Real-Device Evaluation: Manual review of every trajectory is difficult to scale and can introduce variation between annotators.
  • A.1 Validating AutoJudge for Real-Device Evaluation: 92.8% overall agreement: AutoJudge matches human annotations on 618 of 666 trajectories with conclusive human labels.Agreement excludes human annotations marked as unclear.
  • A.1 Validating AutoJudge for Real-Device Evaluation: 96.2% agreement for successful trajectories shows stronger alignment between AutoJudge and human annotations on completed tasks.
  • A.1 Validating AutoJudge for Real-Device Evaluation: Model failures and environment errors are harder to distinguish because similar observations can arise from incorrect Agent actions or live execution problems.
  • A.1 Validating AutoJudge for Real-Device Evaluation: Expert adjudication more often supports AutoJudge than the initial human annotation in disagreement cases, making the reported agreement a conservative accuracy estimate.
  • A.1 Validating AutoJudge for Real-Device Evaluation: AutoJudge is used as the primary evaluator for real-device experiments after validation.The authors describe it as a more accurate and scalable protocol that separately identifies model failures and live-environment errors.

A.2 AutoJudge Decisions on Real-Device GUI Agent Trajectories

AutoJudge evaluates real-device GUI trajectories by distinguishing successful task completion and model failures from failures caused by the live execution environment. Representative cases combine task descriptions, translated queries, selected screenshot actions, and evidence supporting the final decision.

  • Decision outcomes: AutoJudge separates successful execution and model failures from errors caused by the live execution environment.Figures 22–24 illustrate the three decision outcomes with trajectory evidence.
  • Decision outcomes: Representative decisions show the original Chinese task, its English translation, selected trajectory steps, rendered screenshot actions, and final-decision evidence.The examples are presented across Figures 22–24.
  • Pass decisions: AutoJudge labels a trajectory as pass when the evidence shows the agent successfully completes the requested task.Figure 22 provides representative pass decisions.
  • Model-failure decisions: Repeated clicks on the CNY 94.53 lobster open unrelated products, leaving the required cart and favorite actions incomplete.This evidence supports a model-failure decision for the Pupu Supermarket task.

A.3 Additional Qualitative Examples

The qualitative examples demonstrate Qwen-UI-Agent’s long-horizon real-device execution, hybrid GUI+CLI workflows, on-demand research during interaction, proactive services, and cross-platform task completion.

  • Long-horizon real-device execution: A 23-step shopping task sequentially empties the Taobao cart, selects three Xiaomi products, and maintains a 1,000-yuan budget and item checklist.The task selects a smart speaker, smart plug, and smart bulb from the Xiaomi flagship store.
  • Hybrid GUI+CLI execution: The agent uses CLI inspection alongside batched GUI hotkeys and Paste Special to transfer formatted spreadsheet data into a Writer document.The result is saved as price.docx on the desktop while preserving Calc formatting.
  • On-demand DeepSearch: During a Douyin workflow, DeepSearch verifies a weight-loss video’s claims against research papers and authoritative health sources, directly determining subsequent GUI actions.The agent extracts the claims through GUI actions, performs on-demand verification, and returns to Douyin.
  • Proactive service: At 08:00, the harness proactively combines commitments with live weather and commute conditions, compares transit and taxi options, and delivers actionable guidance.The brief includes umbrella advice, departure timing, and key reminders before the user asks.
  • Cross-platform service: The agent locates receipt images on a mobile virtual screen, transfers them to a designated PC directory, and consolidates their details into an expense spreadsheet.Mobile execution does not block the user’s own actions, and computer use combines GUI and CLI operations.
Loading 2607.28227v1…