Source-linked AI summary
UI-Venus-1.5 Technical Report
Venus Team, Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, Beitong Zhou, Xingran Zhou, Weizhi Chen, Sunhao Dai, Jingya Dou, Yichen Gong, Yuan Guo, Zhenlin Guo, Feng Li, Qian Li, Jinzhen Lin, Yuqi Zhou, Linchao Zhu, Liang Chen, Zhenyu Guo, Changhua Meng, Weiqiang Wang
TL;DR
GUI agents must achieve both broad generality and consistently strong performance for reliable digital interaction. UI-Venus-1.5 combines Mid-Training, full-trajectory Online-RL, and model merging in one end-to-end agent, achieving state-of-the-art results across grounding and navigation benchmarks and practical coverage of Chinese mobile apps.
Problem
GUI agents face difficulty combining universal capability, strong performance, and easy deployment, with step-trace mismatch and domain shift limiting training.
Method
UI-Venus-1.5 uses 10B-token Mid-Training across 30+ datasets, full-trajectory Online-RL, and model merging of grounding, web, and mobile specialists.
Results
UI-Venus-1.5 establishes state-of-the-art performance across GUI grounding and navigation benchmarks, including 69.6% on ScreenSpot-Pro and 77.6% on AndroidWorld.
Takeaways & Limitations
The unified agent supports practical automation across 40+ Chinese third-party mobile applications, including ticket booking and shopping.
Takeaways & Limitations
Refusal prompts may cause a marginal performance trade-off on benchmarks lacking refusal examples, such as ScreenSpot-Pro.
Abstract
from arXiv · showhide
GUI agents have emerged as a powerful paradigm for automating interactions in digital environments, yet achieving both broad generality and consistently strong task performance remains challenging. In this report, we present UI-Venus-1.5, a unified, end-to-end GUI Agent designed for robust real-world applications. The proposed model family comprises two dense variants (2B and 8B) and one mixture-of-experts variant (30B-A3B) to meet various downstream application scenarios. Compared to our previous version, UI-Venus-1.5 introduces three key technical advances: (1) a comprehensive Mid-Training stage leveraging 10 billion tokens across 30+ datasets to establish foundational GUI semantics; (2) Online Reinforcement Learning with full-trajectory rollouts, aligning training objectives with long-horizon, dynamic navigation in large-scale environments; and (3) a single unified GUI Agent constructed via Model Merging, which synthesizes domain-specific models (grounding, web, and mobile) into one cohesive checkpoint. Extensive evaluations demonstrate that UI-Venus-1.5 establishes new state-of-the-art performance on benchmarks such as ScreenSpot-Pro (69.6%), VenusBench-GD (75.0%), and AndroidWorld (77.6%), significantly outperforming previous strong baselines. In addition, UI-Venus-1.5 demonstrates robust navigation capabilities across a variety of Chinese mobile apps, effectively executing user instructions in real-world scenarios. Code: https://github.com/inclusionAI/UI-Venus; Model: https://huggingface.co/collections/inclusionAI/ui-venus
1 Introduction
UI-Venus-1.5 addresses the challenge of building GUI agents that combine broad capability, strong task performance, and simple deployment. It adds Mid-Training, scaled Online-RL, and model merging, achieving strong benchmark and real-world results.
- Motivation: UI-Venus-1.5 targets the gap between universal capability, strong performance, and easy deployment in GUI agents.The report identifies step-trace accuracy mismatch and domain shift as additional challenges during training.
- Technical advances: 10B tokens from 30+ GUI datasets provide Mid-Training for GUI-VQA, grounding, and simple navigation before reinforcement learning.This stage is intended to establish robust inherent GUI knowledge.
- Technical advances: Full-trajectory Online-RL with expanded interaction devices improves navigation in complex mobile and web environments.The approach calculates rewards across complete trajectories rather than isolated steps.
- Technical advances: Model merging combines specialized grounding, web, and mobile models into one end-to-end agent with minimal performance loss across domains.The unified checkpoint simplifies deployment for users.
- Results: 69.6% on ScreenSpot-Pro, 75.0% on VenusBench-GD, 77.6% on AndroidWorld, and 76.0% on WebVoyager demonstrate strong grounding and navigation performance.The model is also optimized for complex tasks across 40+ Chinese mobile applications, including ticket booking, purchasing, and conversation management.
2 Methodology
UI-Venus-1.5 is an end-to-end multimodal agent that closes the loop from natural-language instructions and screenshots to executable GUI actions. Its training combines large-scale GUI data refinement, staged reinforcement learning, and model merging.
- System overview: The agent interprets user instructions, perceives screenshots, grounds intentions into actions, and interacts iteratively across mobile and web interfaces.This closed-loop design avoids handcrafted intermediate representations and API integrations.
- Training pipeline: The four-stage pipeline comprises Mid-Training, task-specific Offline-RL, Online-RL for complex navigation, and model merging.The stages progress from GUI knowledge injection to unified deployment.
- Data refinement: Teacher scoring, recursive trace rewriting, and manual verification increase the proportion of high-fidelity samples from 69.7% to 89.7%.The refinement process is designed to keep training signals dense and accurate.
- Real-device data: A DaaS generation loop uses candidate prompts, cloud-device execution, trajectory scraping, and multi-annotator verification to collect realistic interaction traces.Verified trajectories are reused as in-context examples for later task generation.
2.2 Offline Reinforcement Learning
Offline-RL uses structured rewards and domain-specific action constraints to optimize GUI interaction. However, rising step-level success can coexist with declining trace-level success, motivating later Online-RL.
- Grounding rewards: The grounding reward combines format correctness and point-in-box localization, with weights controlling their relative importance.The model must produce a syntactically valid answer and predict a center point inside the target element.
- Grounding: Refusal prompts train the model to return [−1, −1] when an instructed element is absent, reducing hallucinated coordinate predictions.This capability is especially relevant to refusal-specific grounding tasks.
- Navigation rewards: A decoupled Offline-RL reward system combines format reward with action reward for valid, contextually appropriate actions.Action reward includes action-type correctness and content- or coordinate-related components.
- Action spaces: Hover and Hotkey extend web interaction, while scrolling uses precise start and end coordinates on mobile but only direction on web.These constraints align action execution with platform-specific operating logic.
- Training observations: Step-level success rises during Offline-RL while trace-level success eventually peaks and declines, because individual rewards do not ensure successful multi-step composition.This discrepancy motivates appending Online-RL to improve deployable performance.
2.3 Online Reinforcement Learning
Online reinforcement learning extends GUI-agent training beyond static data by using direct environmental interaction, trajectory-level feedback, and a unified device infrastructure for large-scale execution. The resulting framework combines diverse task generation, stratified sampling, composite rewards, and group-relative advantages for dynamic GUI navigation.
- Motivation: Online-RL addresses static-training limits by collecting deployment-like trajectories through direct interaction with dynamic environments and refining the policy from observed feedback.The framework is supported by a Device-as-a-Service execution infrastructure and task-generation pipeline.
- Device-as-a-Service: The DaaS layer unifies heterogeneous device protocols and securely exposes Chrome, mobile, and desktop resources to upstream training and inference tasks.Its gateway uses secondary hash routing, zero-copy I/O, and a multi-protocol reverse proxy.
- Device-as-a-Service: Thousands of heterogeneous devices and millions of daily operation requests demonstrate the DaaS layer’s scale, while hundreds to thousands of concurrent devices support reinforcement-learning workloads.Device allocation responds at millisecond-level latency, and the system maintains stability under high-load conditions.
- Task Formulation: The task pool combines LLM-generated static workflows with MLLM-generated long-tail tasks from offline trajectories, filters duplicates by semantic similarity, and samples tasks by difficulty.Difficulty buckets are Easy (Nsteps ≤10), Medium (10 < Nsteps ≤20), and Hard (Nsteps > 20).
- Reward Design: The composite reward combines task completion, action-validity penalties, and trace-length decay, penalizing unparseable actions and discouraging redundant trajectories.The invalid-action penalty reduces invalid attempts during online exploration and improves sample efficiency.
- Task Formulation: Task success is verified either deterministically through system APIs or semantically by an MLLM judging the initial query against the final keyframe screenshot.The dual-track mechanism covers both explicit system outcomes and visually ambiguous task intent.
- Training Algorithm: Trajectory-level advantages normalize composite rewards within sampled groups and assign the resulting signal across each trajectory’s action steps for stable long-horizon policy updates.Group-relative competition is intended to reduce environmental stochasticity in credit assignment.
2.4 Model Merge
Model merging consolidates grounding, web, and mobile specialists into one GUI-agent checkpoint. TIES-Merge preserves cross-domain performance better than linear merging, although some tasks can decline relative to domain-specific models.
- Model Merge: Specialized grounding, web, and mobile checkpoints are merged into a single global model using parameters derived from a shared foundational model.The goal is to consolidate domain expertise into one unified GUI Agent.
- Model Merge: TIES-Merge prunes low-magnitude task updates and resolves parameter sign conflicts before aggregating aligned changes.These operations are designed to reduce parameter interference during fusion.
- Performance Comparison: TIES-Merge outperforms Linear Merge in cross-task fusion: UI-Venus-1.5-30B-A3B reaches 69.6% on ScreenSpot-Pro and 77.6% on AndroidWorld, versus Linear Merge drops of 2.9% and 2.3%.Before merging, the same model achieves 71.0% and 75.5% on those benchmarks, respectively.
- Performance Comparison: Model merging may reduce performance on some tasks relative to domain-specific models, but the merged checkpoint balances performance across all three domains without training a multi-task model from scratch.The paper characterizes this as a trade-off between specialized peak performance and unified deployment.
3 Experiments
UI-Venus-1.5 achieves strong and scalable performance across grounding and navigation benchmarks, while showing broad cross-platform generalization and practical GUI capabilities. Ablations indicate that mid-training, offline reinforcement learning, online reinforcement learning, and model merging contribute differently to grounding and long-horizon navigation.
- Grounding Benchmarks: 75.0% on VenusBench-GD and 69.6% on ScreenSpot-Pro establish state-of-the-art grounding performance for UI-Venus-1.5-30B-A3B.The model also leads OSWorld-G-R, OSWorld-G, and UI-Vision, while reaching 96.2% on ScreenSpot-V2.
- Navigation Benchmarks: 77.6% on AndroidWorld gives UI-Venus-1.5-30B-A3B a 4.3% absolute margin over MAI-UI-32B.The 2B, 8B, and 30B-A3B variants reach 55.6%, 73.7%, and 77.6%, respectively.
- Navigation Benchmarks: 21.5% on VenusBench-Mobile and 76.0% on WebVoyager show strong end-to-end performance across mobile and web navigation tasks.The model family achieves success rates of 8.7%, 16.1%, and 21.5% on VenusBench-Mobile and 56.4%, 70.8%, and 76.0% on WebVoyager across scales.
- Representation Analysis: 34.0% relative growth in Silhouette Score after mid-training indicates more separable GUI-specific feature clusters.The score increases from 0.235 to 0.315, while Intra-class Consistency decreases by 11.6%, indicating finer-grained GUI feature sensitivity.
- Ablation Studies: Offline-RL raises ScreenSpot-Pro scores by approximately 6–7% and AndroidWorld performance by up to 6.5% for the 8B model.The reported result attributes this improvement to GRPO on diverse, task-specific offline data aligning visual perception with GUI-specific action spaces.
- Ablation Studies: +14.5% absolute AndroidWorld gain for the 2B model marks online-RL as a major improvement stage for complex navigation.The passage links dynamic-environment interaction and exploration with improved long-horizon task handling and error recovery.
4 Related Works
Related work on GUI agents spans grounding, end-to-end navigation, and multi-agent frameworks. The field has progressed from rule-based or supervised approaches toward reinforcement learning, richer data pipelines, and improved alignment with real-world environments.
- GUI Agents: Early GUI agents relied on predefined rules, while LLMs enabled a single model to handle diverse GUI tasks.Rule-based systems exhibited limited scalability.
- GUI Grounding: GUI grounding maps natural-language instructions to precise GUI-element positions, commonly using supervised fine-tuning on labeled data.As benchmark accuracy ceilings emerged, newer evaluations introduced more complex grounding scenarios and post-processing methods.
- End-to-End GUI Agent: End-to-end GUI agents initially trained directly from foundation models but later incorporated DPO, GRPO, improved data generation, and greater computation.These developments aimed to narrow the gap between benchmark performance and practical deployment.
- End-to-End GUI Agent: Greater alignment between real-world and training environments has accompanied more robust practical deployment capabilities in end-to-end GUI agents.The related work frames this alignment as advancing feasibility in real-life scenarios.
- GUI Agent Framework: GUI agent frameworks distribute context across specialized sub-agents that analyze task progress and screens before selecting actions.Agent S is cited as using experience-augmented hierarchical planning with role-specific agents.
5 Conclusion
UI-Venus-1.5 combines multi-stage training with model merging to form a practical, unified GUI agent. It achieves state-of-the-art results across grounding and navigation benchmarks and supports automation in over 40 Chinese third-party apps.
- UI-Venus-1.5 uses Mid-Training, task-specific Offline-RL with model merging, and scaled Online-RL to develop robust GUI capabilities.The unified agent consolidates grounding, web, and mobile navigation expertise into one end-to-end model.
- UI-Venus-1.5 consolidates grounding, web, and mobile navigation expertise into a single end-to-end agent while preserving robust performance across tasks.
- UI-Venus-1.5 establishes state-of-the-art performance across GUI grounding and navigation benchmarks.
- The system is optimized for real-world utility across the 40+ Chinese third-party app ecosystem, including ticket booking and shopping.
6 Contributions
The listed UI-Venus-1.5 contributors are presented in alphabetical order by last name.
- The contributor list is ordered alphabetically by last name.
- Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen†, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, Beitong Zhou, and Xingran Zhou are listed as contributors.
- Weizhi Chen, Sunhao Dai, Jingya Dou, Yichen Gong, Yuan Guo, Zhenlin Guo, Feng Li, Qian Li, Jinzhen Lin, Yuqi Zhou, Linchao Zhu are listed as contributors.
- Liang Chen, Zhenyu Guo, Changhua Meng†, and Weiqiang Wang are listed as contributors.
A Action Space and Prompt Templates
The appendix defines a unified action space and prompt templates for grounding, mobile, web, and Chinese-app GUI-agent interactions. These templates specify available actions, output formats, and navigation instructions.
- A.1 Action Space: The action space unifies all actions and maps actions from existing open-source datasets into this shared space.
- A.2 Grounding: Grounding prompts request the center coordinates of the position matching an instruction, using [-1,-1] when the task is infeasible.
- A.3 Mobile: Mobile prompts require the agent to inspect the task, current screenshot, and previous actions before determining the next action.
- A.3 Mobile: Mobile agents can click, drag, scroll, type, launch apps, wait, finish, call the user, long-press, and use system navigation actions.
- A.3 Mobile: Mobile templates use structured think, action, and conclusion fields and instruct agents to verify the task, reply explicitly to questions, and explore by scrolling.
- A.4 Web: Web prompts use the same screenshot-and-history reasoning format, with web actions including URL launch, directional scrolling, hovering, double-clicking, and keyboard shortcuts.
- A.5 Chinese APPs Prompt: Chinese-app prompts describe executing operations from historical actions and device state, with guidance for activating inputs, searching or scrolling, backing out, and using app and system controls.
- A.5 Chinese APPs Prompt: The Chinese-app action definitions cover clicking, dragging, scrolling, launching apps, waiting, long-pressing, going back, and returning home.
B Experiment Details of All Grounding Benchmarks
For OSWorld-G, the authors recalculated UI-Venus-1.0’s performance after including the refusal task, so its results differ from the previous report.
- UI-Venus-1.0’s OSWorld-G performance was recalculated with the refusal task, producing results different from the previous report.
C Experiment Details of All Navigation Benchmarks
This section reports performance comparisons across GUI grounding and navigation benchmarks, identifying best and second-best models where specified. The benchmarks span VenusBench-GD, ScreenSpot variants, MMbench-GUI-L2, OS-World variants, UI-Vision, and VenusBench-Mobile.
- VenusBench-GD results are compared with best and second-best models explicitly marked.
- ScreenSpot-Pro and ScreenSpot-V2 provide benchmark-specific performance comparisons with best and second-best models indicated.
- MMbench-GUI-L2, OS-World-G, and OS-World-G-Refine report comparative benchmark results with best and second-best models marked.
- UI-Vision reports comparative results using best and second-best indicators, while VenusBench-Mobile marks the best-performing model.