PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

Hanzhang Zhou, Panrong Tong, Xu Zhang, Quyu Kong, Chenglin Cai, Tianyu Xia, Gongjie Zhang, Jianan Zhang, Long Li, Long Chen, Lei Wang, Gaole Dai, Pengxiang Li, Liangyu Chen, Yue Wang, Steven Hoi

arXiv:2607.28227v1cs.AIcs.CV

TL;DR

GUI agents remain largely optimized for simulated benchmarks, leaving a gap between benchmark performance and real-world utility. Qwen-UI-Agent combines sandbox and real-device environments with unified GUI/CLI actions and reports competitive results across mobile, computer, and browser benchmarks.

  • Problem

    GUI agents remain largely optimized for simulated benchmarks, leaving a substantial gap between benchmark performance and real-world utility.

    Source: §1 · p. 3

  • Method

    Qwen-UI-Agent combines sandbox and real-device environments with a unified GUI/CLI action space, agent-driven data flywheel, online RL, and a harness for proactive and cross-platform workflows.

    Source: §6 · p. 43

  • Results

    Across mobile, computer, and browser benchmarks, Qwen-UI-Agent reports 92.2% on MobileWorld-Real and competitive computer- and browser-use results.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The report presents Qwen-UI-Agent as a step toward GUI agents with real-world utility.

    Source: §1 · p. 3

    Real-device mobile results rely on AutoJudge rather than deterministic verification or expert review, and its remaining errors may introduce minor uncertainty.

    Source: §7 · p. 44

Abstract

from arXiv · show

GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.

1 Introduction

Qwen-UI-Agent targets real-world GUI utility through reliable real-device execution, cross-platform workflows, hybrid GUI and CLI actions, long-horizon completion, automated capability development, and proactive services. It combines these capabilities in a foundation agent spanning mobile, computer-use, web, and DeepSearch environments, with strong results across mobile and other GUI benchmarks.

Vision and Contributions

  • The report frames six transitions for real-world GUI agents: simulation to real devices, isolated domains to cross-platform workflows, GUI-only to hybrid and batched actions, short to long horizons, manual to AutoResearch-style development, and reactive to proactive service.Qwen-UI-Agent is introduced as a real-world-centric foundation agent developed around these transitions.Source: §1 · p. 3

Real-Device and Cross-Domain Capabilities

  • Qwen-UI-Agent uses more than 100 physical devices and supports over 150 apps for mobile task design, trajectory collection, online RL, and evaluation, alongside workflows spanning mobile, web, computer-use, and DeepSearch.Its harness preserves context and task state across environments and can invoke DeepSearch to retrieve and verify external information before resuming GUI interaction.Source: §1 · pp. 3–4

Hybrid Actions and Long-Horizon RL

  • A unified action space combines GUI operations, bash CLI commands, and batched actions, while verifier-guided online RL trains on trajectories exceeding 100 steps with approximately 10,000 concurrent simulated environments.A model-adaptive curriculum prioritizes tasks with intermediate success rates and replaces mastered tasks with harder ones; over 40% of computer-use action outputs are batched.Source: §1 · p. 4

Automated Development and Proactive Service

  • An agent-driven AutoResearch loop generates and assesses tasks, processes rollouts, analyzes failures, and plans iterations, while a proactive mobile harness turns notifications into personalized tasks and approved actions.Humans mainly supervise and make targeted revisions; the proactive harness also learns from user responses and long-term interaction history.Source: §1 · p. 4

Mobile Use

  • MobileWorld-Real success is 92.2%, AndroidDaily success is 97.5%, and MobileWorld score is 82.1%, with the reported advantages over frontier models on both MobileWorld benchmarks.On MobileWorld-Real, the reported margins over Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro are 6.0, 7.5, 6.8, and 3.5 percentage points; on MobileWorld, margins over Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro are 14.6, 12.0, and 8.9 points.Source: §1 · p. 4

Computer Use

  • OSWorld-Verified score is 79.5%, ranking second overall, while OSWorld-v2 partial-progress score is 40.0%, the third-best reported result.On OSWorld-v2, its binary success rate is 13.9%; its partial-progress score exceeds MiniMax M3 and Qwen 3.7 Plus by 17.7 and 18.5 percentage points while requiring 58.4% and 21.7% fewer steps per task.Source: §1 · p. 4

Browser Use and DeepSearch

  • WebArena score is 73.6%, while DeepSearch scores are 64.1% on BrowseComp and 75.0% on BrowseComp-ZH.On WebArena, Qwen-UI-Agent exceeds Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro by 1.7, 4.1, and 8.3 points; on the two BrowseComp benchmarks it surpasses Qwen3.5-397B-A17B and UI-TARS-2.Source: §1 · pp. 4–5

GUI Grounding and General Capabilities

  • ScreenSpot-Pro score is 81.5% under zoom-in, and Qwen-UI-Agent sets baseline-best results on UI-Vision, OSWorld-G-Refined, MMBench-GUI L2, and ScreenSpot-V2.The respective scores are 70.0%, 78.5%, 92.6%, and 97.5%; the report also finds it outperforms its Qwen base model on agentic tasks, remains comparable on general reasoning, and significantly exceeds GUI-specialized models across both categories.Source: §1 · p. 5

2 Qwen-UI-Agent · 2.1 System Overview

Qwen-UI-Agent is presented as a real-world-centric foundation GUI agent for tasks across mobile, desktop, web, and information-seeking systems. Its integrated design combines diverse environments, a unified action space, automated data iteration, supervised and reinforcement learning, and a harness for proactive, cross-platform workflows.

2 Qwen-UI-Agent

  • The system integrates sandboxed and real-world environments, an automated data flywheel, supervised fine-tuning with reinforcement learning, and a harness connecting the agent to user context.The flywheel links data synthesis, training, evaluation, failure analysis, and targeted data generation for the next iteration.Source: §2 · p. 5

2.1 System Overview

  • Qwen-UI-Agent targets interactive tasks across mobile, desktop, web, and information-seeking systems, with a formulation spanning both single-environment tasks and cross-platform workflows.At each decision step, multichannel observations can combine screenshots, command-execution results, and structured external-service responses, allowing joint reasoning over visual and programmatic feedback.Source: §2.1 · p. 6, §2.1.1 · p. 6

2.1.1 Task Formulation

  • A model decision can emit one action or an ordered batch executed consecutively, reducing extra inference and observation steps when no further environment feedback is needed.The agent’s interaction history includes prior observations, reasoning outputs, and actions.Source: §2.1.1 · p. 6

2.1.2 Action Space

  • The unified action space combines platform-native GUI actions with bash commands and structured API calls, while ask_user supports missing-information requests and confirmation for consequential operations.The action design preserves user control around sensitive data and payments.Source: §2.1.2 · p. 6, Table 1 · p. 7

2.1.3 Capability Landscape

  • Real-device mobile development and evaluation are intended to reduce the simulation-to-reality gap, while the same agent spans mobile, desktop, web, and DeepSearch tasks.The formulation also enables GUI and CLI operations to be interleaved in one trajectory, with batched actions supporting execution efficiency.Source: §2.1.3 · p. 7
  • A lightweight harness enables proactive service initiation and workflows spanning mobile and computer systems, extending the agent beyond isolated task execution.Source: §2.1.3 · p. 7

2.2 Environment Infrastructure

Qwen-UI-Agent’s environment infrastructure combines scalable sandboxes, real-device execution, hybrid GUI+CLI actions, and a unified interface across mobile, computer, browser, and DeepSearch tasks. Together, these components support controlled training while accommodating realistic device conditions and platform-specific interaction.

2.2.1 Scaling Sandbox Environments

  • Up to 10,000 parallel sandboxes span mobile, computer, browser, and DeepSearch tasks for scalable training and evaluation.The mobile sandbox uses containerized redroid devices and reproducible application backends; computer tasks extend OSWorld with bash execution; browser episodes use isolated Playwright contexts and state-based verification; DeepSearch combines ranked search with document retrieval.Source: §2.2.1 · pp. 8–9

2.2.2 Real-Device Mobile Environment: From Simulated Sandboxes to Real-World Learning

  • Over 100 physical devices and 150 applications support real-device training and evaluation under changing states, permissions, pop-ups, and network disruptions.A health-aware scheduler leases eligible device, application, account, network, and display resources and reroutes or blacklists unhealthy targets; virtual displays increase concurrent sessions, while a User Agent handles missing information, approvals, and user handoffs. A VLM-based judge distinguishes task success, model failure, and environment failure, feeding confirmed environment issues into maintenance and retaining validated trajectories for training.Source: §2.2.2 · pp. 9–10, Figure 4 · p. 9

2.2.3 Hybrid GUI+CLI Action Space

  • The hybrid GUI+CLI action space combines visually grounded application interaction with direct programmatic execution through one interface.In computer-use tasks, commands run in a non-interactive shell and return stdout, stderr, and exit status alongside screenshots; timeouts and command failures become recoverable observations, and batched sub-actions execute in order.Source: §2.2.3 · p. 10

2.2.4 Unified Interface for Heterogeneous Environments

  • A unified asynchronous environment interface standardizes lifecycle and agent inputs and outputs while preserving each platform’s native execution and verification semantics.Backend and task adapters handle resource setup, observations, actions, reset, and evaluation; leased sessions route interactions to isolated contexts, and a canonical representation translates supported GUI actions when policy and environment action spaces differ.Source: §2.2.4 · p. 11

2.3 Agent-Driven Data Flywheel

Qwen-UI-Agent uses an agent-driven flywheel to bootstrap broad training coverage and iteratively target weaknesses, with humans providing system-level oversight. Its design combines capability-aware task synthesis, reproducible environments, trajectory judging, and failure analysis to improve training data.

Overall Procedure

  • Bootstrapping builds a broad initial policy from generated tasks, environments, and rejection-sampled trajectories; subsequent cycles analyze failures, create targeted tasks and verifiers, and add accepted trajectories to training.Strong foundation models generate tasks and environment configurations, synthesize verifiers, evaluate trajectories, analyze failures, and plan iterations, while humans oversee and revise the workflow.Source: §2.3 · p. 11, Figure 5 · p. 12

Knowledge- and Capability-Aware Task Synthesis

  • Task synthesis combines knowledge coverage, such as application functions and workflows, with capability demands such as long-horizon tracking, constraint following, numerical and temporal reasoning, and error recovery.A reusable synthesis skill encodes composition rules, difficulty controls, environment prerequisites, and examples to generate tasks aligned with each iteration’s objectives.Source: §2.3 · p. 12

Agent-Driven Environment and Verifier Synthesis

  • Agents construct the application state and cross-application context needed for each task, then synthesize state-based verifiers for outcomes that can be reliably checked.These verifiers inspect application, file, database, or system state to form executable task–verifier pairs.Source: §2.3 · p. 12

Step-Level Judging for Scalable SFT and Executable Verification for Online RL

  • Fine-grained VLM step filtering yields SFT performance comparable to or better than full trajectories selected by executable verifiers, while executable verifiers provide high-precision outcome signals for online RL.The judge identifies contiguous task-advancing steps, reflection or exploration starts, and recovery segments from successful and failed trajectories.Source: §2.3 · pp. 12–13

Failure Analysis Driven Iteration

  • For each failed task, an analysis agent separates model errors from environment, task, or verifier problems, maps model failures to structured causes, and prioritizes weaknesses for targeted next-cycle tasks.Non-model issues are routed to environment maintenance or task and verifier revision.Source: §2.3 · p. 13

2.4 Training

Qwen-UI-Agent training combines domain-specialized SFT and capability-preservation data with action-level and trajectory-level reinforcement learning. Online RL is supported by scalable rollouts, automatically synthesized task–verifier pairs, and a curriculum that adapts to policy learning progress.

2.4.1 Supervised Fine-Tuning (SFT)

  • SFT trains domain experts on target-domain data mixed with other domains, then merges their checkpoints; verified in-distribution examples rehearse capabilities the starting model can already perform.The approach addresses cross-domain capability and general reasoning needs, while using correct starting-model responses across question answering, math, coding, visual understanding, search, and agentic tool use.Source: §2.4.1 · pp. 13–14
  • Sliding-window SFT processes five-step windows with a four-step stride, masking the shared boundary action to supervise four new actions per subsequent window.Earlier screenshots are omitted while textual history remains, and one-step overlap lets each new action condition on at least two screenshots while reducing repeated context processing.Source: §2.4.1 · p. 14

2.4.2 Action RL: Correcting Recurring Action Errors

  • Action RL targets six recurring errors: confusable-element grounding, sorting or ranking mistakes, incomplete multi-target work, premature completion, repetitive loops, and long-tail action failures.The errors range from selecting a nearby distractor or wrong ordering to omitting a final state-changing action or substituting frequent actions for critical infrequent ones.Source: §2.4.2 · p. 14
  • Action RL combines mined and actively collected failure examples with per-action rewards for format, action type, argument quality, sensitive actions, and repetition.A visual judge helps distinguish unproductive loops from purposeful exploration; entropy regularization and reasoning-length bounds counter declining token entropy and increasingly short reasoning traces.Source: §2.4.2 · p. 15

2.4.3 Online RL: Learning Long-Horizon Decision Making

  • Up to 10,000 concurrent rollouts support Online RL, while virtual screens increase real-device cluster throughput by approximately 20×.A unified infrastructure manages environment allocation, reset, interaction, and evaluation across mobile, computer-use, web, and DeepSearch sandboxes; virtual screens run multiple app sessions on separate displays of one device.Source: §2.4.3 · p. 15
  • Approximately 10,000 validated task–verifier pairs support Online RL, built from synthesized environment states, feasible tasks, and validated code-based verifiers.Coding agents synthesize coherent multi-application states and verifiers; LLM-generated tasks are filtered for feasibility and consistency, and verifier outputs are checked against rollout evidence.Source: §2.4.3 · pp. 15–16
  • Online RL uses active and monitoring pools to promote newly learnable tasks, reactivate regressed mastered tasks, and focus rollout budgets where rewards are informative.Tasks with intermediate success rates receive full rollout budgets; unsolvable or mastered tasks are monitored with smaller budgets, while all-fail or all-success groups provide little immediate learning signal.Source: §2.4.3 · p. 16

2.5 Harness Layer: Proactive Service and Cross-Platform Task Execution

The shared harness extends GUI agents beyond reactive, isolated tasks: it uses digital context to initiate assistance and coordinate workflows across mobile and computer environments. Proactive service and cross-platform execution preserve context, dependencies, and task state as work begins and moves between devices.

2.5 Harness Layer: Proactive Service and Cross-Platform Task Execution

  • The harness connects digital signals, ongoing-activity context, and GUI agents to initiate assistance proactively and coordinate tasks across mobile and computer environments.These capabilities address when a task can begin and where it can continue.Source: §2.5 · pp. 16–17

2.5.1 Proactive Service Based on Mobile Notifications

  • Mobile notifications provide timely, permission-controlled signals for actionable events, but the harness treats them as partial, noisy observations rather than complete world state.It gathers additional information only when needed.Source: §2.5.1 · pp. 17–18
  • A stateful pipeline parses notifications into structured events, links them to persistent affairs, derives tasks, and updates memory from execution outcomes.Events capture what happened; affairs represent ongoing matters across events, applications, and days; tasks specify what to do next.Source: §2.5.1 · p. 18
  • The harness weighs urgency, consequences, evidence, and saved user effort to time interventions, prepare decision-ready options, and keep consequential actions subject to confirmation.User responses to proposed tasks refine profile and feedback memory, calibrating future intervention value, timing, and automation.Source: §2.5.1 · pp. 18–19

2.5.2 Cross-Platform Task Execution

  • Cross-platform execution uses a hierarchical planner–executor with device-grounded observations and shared state to decompose workflows into dependent subtasks and replan from updated results.The shared state preserves instructions, device mappings, observations, artifacts, tool outputs, and subtask results.Source: §2.5.2 · p. 19
  • Independent subtasks can run concurrently on separate virtual Android displays, while each GUI subtask combines visual interaction with CLI or API operations and transfers artifacts across devices.Desktop commands use shell execution and Android commands use ADB; outputs, errors, and exit status return as structured observations.Source: §2.5.2 · p. 19

3 Experiments

This section evaluates Qwen-UI-Agent across mobile, computer, browser, DeepSearch, GUI-grounding, and general and agentic benchmarks. It also examines grounding across model scales and the harness’s support for proactive assistance and stateful mobile and computer workflows.

3.1 Experimental Setup

  • The 27B model is the primary variant, evaluated alongside 35B-A3B and 4B models across mobile, computer-use, browser, DeepSearch, GUI-grounding, and general and agentic benchmarks.The evaluation compares against proprietary frontier models, open-weight models, and specialized GUI agents; for general and agentic benchmarks, it also compares the 27B model with its base checkpoint.Source: §3.1 · pp. 20–21

3.3.1 Mobile-Use Evaluation.

  • 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily are achieved by Qwen-UI-Agent-27B, with first-place results on the real-device benchmarks.On MobileWorld, it sets a new state of the art under the 50-step budget, exceeding GPT 5.6 Sol by 12.0 percentage points; on MobileWorld-Real it surpasses Seed 2.1 Pro (88.7%), Gemini 3.1 Pro (86.2%), and GPT 5.6 Sol (85.4%), and on AndroidDaily it exceeds Seed 2.1 Pro (95.2%) and Gemini 3.1 Pro (93.8%).Source: Table 2 · p. 22, §3.3.1 · pp. 23–24, Table 3 · p. 23

3.2 MobileWorld-Real: Real-world Mobile Use Evaluation Benchmark

  • MobileWorld-Real comprises 409 human-derived tasks across 104 apps, executed on live Android devices with real apps, accounts, content, and networks.The benchmark covers seven daily-use domains and realistic interruptions, while its AutoJudge uses five independent VLM judges and majority voting to label trajectories as pass, failed, or env_error; environment errors are excluded from the success-rate denominator, and exact-match accuracy is 92.8% on 666 trajectories. Tasks and trajectories are held out from training, and the benchmark avoids state-dependent tasks that cannot be reliably reset.Source: §3.2 · pp. 22–23

3.3.2 Computer-Use Evaluation

  • On OSWorld-Verified, Qwen-UI-Agent scores 79.5%, ranking second; on OSWorld-v2, it records 40.0% partial progress and 13.9% binary completion.It trails Claude Opus 4.8 on OSWorld-Verified, while on OSWorld-v2 it exceeds GPT-5.5 in binary completion by 0.9 points and MiniMax M3 in partial progress and completion by 17.7 and 9.3 points. CLI actions make up 40.7% and 55.1% of actions on OSWorld-Verified and OSWorld-v2, respectively, and batched actions account for 39.6% and 41.6%.Source: Table 4 · p. 25, §3.3.2 · pp. 25–26, Table 5 · p. 25

3.3.3 Browser-Use and DeepSearch Evaluation

  • 73.6% on WebArena is the highest result among compared models; Qwen-UI-Agent also achieves 64.1% on BrowseComp and 75.0% on BrowseComp-ZH.Its WebArena score exceeds Claude Opus 4.8 (71.9%) by 1.7 points, while BrowseComp-ZH is second-best among compared systems; DeepSearch uses API-based retrieval and evidence synthesis to support information acquisition before GUI interaction.Source: Table 6 · p. 27, §3.3.3 · pp. 27–28, Table 7 · p. 27

3.3.4 GUI Grounding

  • Qwen-UI-Agent-27B scores 76.6% on ScreenSpot-Pro without zoom-in and 81.5% with zoom-in, ranking first among compared models on that benchmark.It also achieves 97.5% on ScreenSpot-V2, 92.6% on MMBench-GUI L2, 78.5% on OSWorld-G-Refined, and 70.0% on UI-Vision.Source: Table 8 · p. 29, §3.3.4 · p. 29
  • 76.1% (35B-A3B) and 67.8% (4B) on ScreenSpot-Pro without zoom-in rise to 80.2% and 74.0% with zoom-in.The 35B-A3B scores are 96.7% on ScreenSpot-V2, 92.0% on MMBench-GUI L2, 74.6% on OSWorld-G-Refined, and 65.9% on UI-Vision, activating only 3B parameters per token; the 4B scores are 94.9%, 87.9%, 70.5%, and 51.6%, respectively.Source: §3.3.4 · p. 30

3.3.5 General and Agentic Capabilities

  • Qwen-UI-Agent retains general reasoning and agentic capabilities from its base model and substantially outperforms specialized GUI models on general and agentic benchmarks.The report characterizes it as broadly capable rather than GUI-only, potentially supporting unusual tasks requiring knowledge, reasoning, and tool use beyond routine GUI interaction.Source: §3.3.5 · p. 30

3.4 Harness-Enabled Workflows

  • The harness turns notifications into proactive assistance, delivering contextualized morning and travel plans while leaving consequential decisions, such as booking, to the user.The examples include a scheduled morning brief using commitments, weather, and commute conditions, and recovery planning after a flight cancellation that threatens a next-day demo.Source: §3.4 · p. 31, Figure 11 · p. 31
  • The harness carries task state, artifacts, and dependencies across mobile and computer stages, enabling workflows to resume from current context rather than restart.Examples include restaurant discovery on mobile followed by spreadsheet creation and approval on desktop, receipt-to-expense-report handoff, and parallel restaurant search; mobile subtasks can run on virtual screens without blocking device use.Source: §3.4 · pp. 31–32, Figure 12 · p. 32

4 Behavioral Analysis of GUI Agent Execution

The section examines execution behavior to explain real-device failures and characterize how GUI, CLI, batched actions, and RL shape task execution. It complements aggregate benchmark scores with analyses of failure patterns, action choices, and trajectory-level changes.

4.1 How Does a Model’s Limited Real-Device Experience Affect Its Execution Behavior?

  • Real-device failures divide into execution-capability limitations (40.3%) and real-world scenario challenges (52.0%).Execution failures involve exploration (19.5%), erroneous action loops (14.3%), and lost execution state (6.5%); these reflect difficulty finding functions, recovering from failed actions, and retaining progress across long workflows.Source: §4.1 · pp. 32–34, Table 10 · p. 33
  • Real-world interface challenges include UI misreading (24.7%), pop-up interference (18.2%), and physical widget control (9.1%).These failures arise from ambiguous stateful pages, disruptions such as CAPTCHAs or paywalls, and controls needing incremental feedback-driven manipulation; simulated interfaces often omit these conditions.Source: Figure 13 · p. 33, §4.1 · p. 34

4.2 A Closer Look at GUI–CLI Coordination and Batched Execution in Qwen-UI-Agent

  • CLI operations comprise 40.7% of OSWorld-Verified actions and 55.1% of OSWorld-v2 actions, while batched actions comprise 39.6% and 41.6%, respectively.CLI appears in 92.0% of OSWorld-Verified tasks and 98.2% of OSWorld-v2 tasks; batches appear in 62.1% and 88.9%, averaging 3.1 primitive actions per batch. GUI is used for visible state and native or spatial controls, while CLI handles structured operations and verification.Source: §4.2 · pp. 34–37, Table 11 · p. 35
  • A GUI–CLI controller in the dinosaur-game case survived 60 seconds, executed 50 jumps, and reached a visible score of 156.GUI observations informed CLI controller revisions, including stateful jump control and background-relative obstacle detection; batching groups predictable local actions and stops when a new observation is needed.Source: §4.2 · pp. 37–39

4.3 Step-Level Behavioral Corrections after Action RL

  • Action RL improves task success by more than 7%, reduces reasoning tokens by 21.3%, and increases interaction steps by 8.4%.It raises success rates by 6.3% on grounding, 3.8% on Sorting and Ranking, 4.4% on Quantity and Multi-Target Completeness, 5.2% on Premature Completion, and 9.5% on short-range action-loop tasks. In the Mastodon example, the policy abandons an ineffective path and finds the invite-link settings.Source: §4.3 · pp. 39–40, Table 12 · p. 39, Figure 18 · p. 39
  • Action RL raises the share of long-tail action examples to about 40% and improves their reward by 6.4%.Long-tail actions account for 19.9% of primitive actions and have rewards 16.8% below frequent actions, despite often enabling specialized controls or states.Source: §4.3 · p. 40, Table 13 · p. 40

4.4 Trajectory-Level Behavioral Shifts after Online RL

  • Online RL increases verified completion and cross-modal execution: verification appears in 14.7% more OSWorld trajectories, while false stops fall by 11.2%.GUI action share rises by 6%, trajectories using both GUI and Bash rise by 10.6%, and execution–verification transitions increase from 40.2% to 52.4%. The emerging pattern uses Bash for state changes and GUI for inspection.Source: §4.4 · pp. 41–42
  • Instruction-constraint satisfaction rises by 8.6% on OSWorld and 7.5% on BrowseComp-ZH after online RL.The BrowseComp-ZH example shifts from satisfying one of four constraints to jointly applying all four and reaching the correct answer.Source: §4.4 · p. 42, Figure 21 · p. 42

5 Related Work

Related work increasingly favors native multimodal policies, self-evolving data, online reinforcement learning, and structured memory, while shifting from benchmark optimization toward real-device execution, privacy, reliability, safety, and latency (Seed, 2025; Zhou et al., 2025a; Gao et al., 2026; He et al., 2024; Wang et al., 2025b; Huang et al., 2026; Cao et al., 2026; Qin et al., 2025). The section reviews advances and evaluation trends in mobile, browser, and computer-use agents.

Mobile Use

  • Mobile agents are moving beyond screenshot-grounded action prediction toward systems using memory, reflection, reinforcement learning, and agentic harnesses.Examples include unified models across platforms, data flywheels and multi-turn RL, hybrid GUI-MCP operations, and real-device closed-loop training. Evaluations span grounding, controlled task completion, and realistic daily-use scenarios.Source: §5 · pp. 42–43

Browser Use

  • Browser agents leverage DOM trees, accessibility trees, and browser APIs, with evaluation evolving from short navigation to realistic long-horizon, multi-site workflows.Prior systems combine visual understanding with browser actions; more recent work explores autonomous interaction with complex websites and world-model-based approaches.Source: §5 · p. 43

Computer Use

  • Computer-use agents face larger action spaces, more diverse states, and longer horizons than mobile and web agents, making reliable grounding and execution verification critical.Recent foundation-model efforts use multimodal pretraining, large-scale interaction data, and agentic reasoning; benchmarks emphasize robustness, scalability, verification reliability, and complex real-world workflows.Source: §5 · p. 43

6 Conclusion

Qwen-UI-Agent is a real-world-centric foundation GUI agent that integrates environments, actions, data generation, training, and a harness around real-world use. Its reported results span mobile, computer, browser, DeepSearch, and GUI-grounding tasks.

6 Conclusion

  • Qwen-UI-Agent achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, 97.5% on AndroidDaily, 79.5% on OSWorld-Verified, and 40.0% on OSWorld-v2.The report also describes its performance across browser use, DeepSearch, and GUI grounding as leading or competitive.Source: §6 · p. 43
  • The system combines scalable sandbox environments, a real-device mobile runtime, a unified GUI+CLI action space, an agent-driven data flywheel, and SFT, action RL, and online RL.A harness layer supports proactive service initiation and cross-platform task execution.Source: §6 · p. 43
  • The reported results support co-designing models, environments, the data flywheel, training, and the harness around real-world usage.The report also notes strong general reasoning and agentic capabilities.Source: §6 · p. 43

7 Limitations and Future Directions

The report identifies limitations in evaluation, training coverage, synthetic-environment integration, and human oversight, and outlines directions for improving GUI agents’ efficiency, long-horizon execution, cross-domain learning, scalable environments, safety, and personalization.

Limitations

  • The report’s results remain subject to AutoJudge uncertainty, incomplete training coverage, and substantial human oversight.AutoJudge achieved 92.8% exact-match accuracy on 666 expert-examined examples, but may introduce minor uncertainty; 35B-A3B CUA and DeepSearch results and training with constructed synthetic environments were not included. The GUI capability pipeline remains agent-driven and requires human monitoring and intervention.Source: §7 · p. 44

Efficient GUI Execution

  • Reducing per-step interaction latency through faster inference, adaptive observation, asynchronous execution, and fewer interaction rounds remains essential for practical GUI use.Virtual displays can enable background or parallel execution, but do not eliminate the per-step perception–reasoning–action cost.Source: §7 · p. 44

Harness-Assisted Long-Horizon Workflows

  • Harness-level context compression, task decomposition, memory management, progress tracking, and structured tool use could improve long-horizon workflow efficiency and reliability.These approaches would complement the hybrid GUI+CLI action space, batched actions, and online RL.Source: §7 · p. 44

Large-Scale Cross-Domain Online RL

  • More efficient, stable online RL for costly, variable-length tasks with sparse or delayed rewards should jointly train across browser-, mobile-, and computer-use environments (Wang et al., 2025a; Hou et al., 2026).Joint training is proposed to improve data efficiency and cross-domain generalization.Source: §7 · p. 44

Scalable Synthesis of High-Fidelity Environments

  • Scalable synthesis of high-fidelity synthetic environments could complement costly real-device environments and help narrow the simulation-to-real gap.The environments were constructed but not integrated into training for this report; the synthesis methodology is planned for open release.Source: §7 · pp. 44–45

Safety, User Control, and Personalization

  • Future work should strengthen safety evaluation, safety-oriented training, interpretability, user control, and personalization for agents operating across applications and devices.Safety-sensitive training already teaches the model to invoke call_user and return control when input or authorization is needed; further work includes reliable user memories and profiles and user control over retained information.Source: §7 · p. 45

A Appendix · A.1 Validating AutoJudge for Real-Device Evaluation

Because real-device tasks lack reliable deterministic verifiers and manual trajectory review is difficult to scale consistently, the authors validate AutoJudge against independent human annotations. Expert adjudication more often supports AutoJudge in disagreements, so it is used as the primary evaluator for real-device experiments.

A.1 Validating AutoJudge for Real-Device Evaluation

  • Real-device evaluation motivates an independent validation because third-party apps hide internal state, outcomes can depend on changing online content or account status, and manual review is hard to scale consistently.Tasks can also involve interactions across multiple apps.Source: §A.1 · p. 52
  • 92.8% overall agreement: AutoJudge matched human annotations on 618 of 666 conclusively labeled trajectories, with 96.2% agreement for successful trajectories.Agreement excludes human annotations marked unclear; the comparison is reported in Table 14.Source: Table 14 · p. 52, §A.1 · p. 52
  • Expert adjudication more often supported AutoJudge than the initial human annotation in disagreement cases, making the reported agreement a conservative estimate of its accuracy.Following validation, AutoJudge became the primary evaluator for real-device experiments.Source: §A.1 · p. 52
  • AutoJudge separately identifies model failures and live-environment errors, which can be difficult to distinguish because similar observations may arise from either cause.The validation describes the resulting protocol as more accurate and scalable for real-device evaluation.Source: §A.1 · p. 52

A.2 AutoJudge Decisions on Real-Device GUI Agent Trajectories

AutoJudge uses trajectory evidence to distinguish successful execution and model failures from errors caused by the live execution environment, illustrated through representative outcomes.

A.2 AutoJudge Decisions on Real-Device GUI Agent Trajectories

  • Representative examples pair translated task instructions with selected trajectory steps, screenshot-rendered actions, and evidence supporting AutoJudge’s final decision.The examples span Figures 22–24 and illustrate successful execution, model failures, and live-environment errors.Source: §A.2 · p. 52
  • The Figure 22 trajectory evidence shows the Agent successfully completing the requested task.Source: Figure 22 · p. 53
  • The Figure 23 trajectory shows a model failure: the Agent follows an incorrect or incomplete path and does not satisfy the task requirements.Source: Figure 23 · p. 53

A.3 Additional Qualitative Examples

The examples illustrate Qwen-UI-Agent’s long-horizon device execution and its integration of GUI actions with CLI tools, on-demand research, and proactive services.

A.3 Additional Qualitative Examples

  • In a 23-step shopping task, the agent emptied a Taobao cart, selected three Xiaomi-store devices, and maintained a 1,000-yuan budget and item checklist.The sequential sub-goals were a smart speaker, smart plug, and smart bulb.Source: Figure 25 · p. 54
  • The agent combined CLI spreadsheet inspection with batched GUI hotkeys and Paste Special to transfer formatted data into a Writer document.Source: Figure 26 · p. 55
  • During GUI execution, the agent used DeepSearch to verify a Douyin video’s weight-loss claims against research papers and authoritative health sources, then posted an evidence-grounded comment.The verification conclusions informed subsequent GUI actions, completing an observe–research–decide–act loop.Source: Figure 27 · p. 55
  • At 08:00, the harness used a mobile notification to proactively deliver a morning brief comparing transit and taxi options with weather-based umbrella advice, departure timing, and reminders.Source: Figure 28 · p. 56

Related papers

Loading 2607.28227v1…