Source-linked AI summary
UI-Venus Technical Report: Building High-performance UI Agents with RFT
Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, Weiqiang Wang
TL;DR
UI-Venus targets limitations in UI agents’ online navigation, data quality, and scale by applying GRPO-based reinforcement finetuning with curated data and self-evolving trajectory improvements. Its 7B and 72B variants achieve state-of-the-art results on grounding and navigation benchmarks, while hallucination between reasoning and final answers remains an open issue.
Problem
Existing UI agents have insufficient online planning, noisy training data, and limited exploration of large-scale reinforcement-finetuned models.
Method
UI-Venus combines GRPO training on Qwen2.5-VL, data cleaning and trace reconstruction, designed grounding and navigation rewards, and self-evolving trajectory alignment with sparse-action enhancement.
Results
UI-Venus achieves state-of-the-art performance across UI grounding and navigation benchmarks, including 65.9% AndroidWorld success for its 72B variant.
Takeaways & Limitations
The released UI-Venus models, data protocols, and evaluation resources provide open-source UI agents for further research on grounding and navigation.
Takeaways & Limitations
The hallucination gap between internal reasoning and final answers remains an open problem that can produce incorrect or inconsistent navigation behavior.
Abstract
from arXiv · showhide
We present UI-Venus, a native UI agent that takes only screenshots as input based on a multimodal large language model. UI-Venus achieves SOTA performance on both UI grounding and navigation tasks using only several hundred thousand high-quality training samples through reinforcement finetune (RFT) based on Qwen2.5-VL. Specifically, the 7B and 72B variants of UI-Venus obtain 94.1% / 50.8% and 95.3% / 61.9% on the standard grounding benchmarks, i.e., Screenspot-V2 / Pro, surpassing the previous SOTA baselines including open-source GTA1 and closed-source UI-TARS-1.5. To show UI-Venus's summary and planing ability, we also evaluate it on the AndroidWorld, an online UI navigation arena, on which our 7B and 72B variants achieve 49.1% and 65.9% success rate, also beating existing models. To achieve this, we introduce carefully designed reward functions for both UI grounding and navigation tasks and corresponding efficient data cleaning strategies. To further boost navigation performance, we propose Self-Evolving Trajectory History Alignment & Sparse Action Enhancement that refine historical reasoning traces and balances the distribution of sparse but critical actions, leading to more coherent planning and better generalization in complex UI tasks. Our contributions include the publish of SOTA open-source UI agents, comprehensive data cleaning protocols and a novel self-evolving framework for improving navigation performance, which encourage further research and development in the community. Code is available at https://github.com/inclusionAI/UI-Venus.
1 Introduction
UI-Venus addresses limitations in R1-like UI agents by combining reinforcement finetuning, high-quality data processing, and self-evolving navigation improvements. Its released models achieve state-of-the-art grounding and navigation results across reported benchmarks.
- Motivation: Current R1-like UI agents remain limited in online navigation planning, UI-data quality, and large-scale RFT exploration.The paper identifies insufficient summary, memory, and planning ability, noisy open-source data, and limited use of 72B models as three critical limitations.
- Approach: Self-Evolving Trajectory History Alignment & Sparse Action Enhancement refines historical reasoning and increases exposure to rare but pivotal actions.The framework aligns thought–action histories across training epochs and balances sparse-action sampling to improve navigation in complex UI scenarios.
- Approach: A three-stage pipeline filters, reconstructs, and iteratively generates UI traces, using about 107k and 350k high-quality samples for grounding and navigation.The stages include data filtering, trace reconstruction, and iterative trace generation with quality filtering.
- Approach: UI-Venus applies GRPO-based reinforcement finetuning to Qwen2.5-VL and releases separate grounding and navigation model variants.The series includes UI-Venus-Ground-7B/72B and UI-Venus-Navi-7B/72B.
- Results: 94.1% / 50.8% and 95.3% / 61.9% are achieved by the 7B and 72B variants on Screenspot-V2 / Pro, while AndroidWorld success rates reach 49.1% and 65.9%.The report states that these results surpass previous state-of-the-art baselines and demonstrate the effectiveness of scaling for GRPO UI-agent training.
2 Related Works
Related work progresses from specialized and collaborative GUI automation systems toward native UI agents that learn unified, end-to-end interaction. These approaches differ in system composition and training requirements.
- UI Grounding: UI grounding methods localize UI elements from natural-language instructions but traditional supervised approaches face limited out-of-distribution generalization.The passage describes UI grounding as foundational to automated UI interaction and identifies generalization as a primary limitation.
- UI Agent Framework: Collaborative UI-agent frameworks decompose GUI automation across specialized agents for planning, decision-making, reflection, and execution.Mobile-Agent is presented as an example of a planning-decision-reflection architecture.
- UI Agent Framework: This collaborative specialization improves task handling at the cost of increased system complexity and computational overhead.The cited discussion connects coordination among multiple agents with the added overhead.
- Native UI Agent: Native UI agents pursue unified end-to-end interaction with graphical interfaces without multiple specialized components.CogAgent and UI-TARS are cited as pioneering examples trained on diverse UI interaction data.
3 Methodology
The methodology extends GRPO-based reinforcement finetuning from UI grounding to UI navigation, using relative rollout rewards to train UI agents.
- GRPO for UI Agents: GRPO is extended from UI grounding to UI navigation and evaluated against supervised fine-tuning for UI-agent training.The paper presents GRPO preliminaries before discussing its application to navigation.
3.1 Preliminaries
GRPO trains UI agents by comparing multiple sampled rollouts for each question and optimizing the policy using normalized relative rewards. In navigation, the model uses evolving historical thought–action context to generate subsequent reasoning and actions.
- GRPO: For each question, GRPO samples multiple rollouts from the current policy and computes a reward for each rollout.The rollouts are grouped per training question before relative reward normalization.
- GRPO: Relative rewards within each rollout group are normalized to obtain advantages without requiring a separate critic model.This relative-baseline mechanism is described as improving training stability.
- GRPO: The clipping mechanism limits excessive policy updates, while the KL term constrains divergence from the reference model.The passage identifies epsilon as controlling the clipping range and beta as weighting the KL constraint.
- UI Navigation: UI-Venus-Navi generates thinking content and model actions from historical thought–action pairs, updating that context after each completed step.The figure description contrasts this with grounding, which uses an instruction and screenshot to output coordinates.
3.2 UI Grounding
UI-Venus constructs a cleaned UI-grounding dataset from diverse sources and trains with rewards that jointly enforce executable output format and localization correctness. Manual cleaning reduces a noisy, redundant corpus to a smaller high-quality training set, while automatic alternatives failed to work.
- Data Collection: 627k open-source grounding samples were collected from five datasets across mobile, desktop, and web platforms, with 107k retained for training after cleaning.The sources include Widget Captioning, UI RefExp, SeeClick, ShowUI, and OmniAct.
- Data Quality: Approximately 40% of current open-source grounding data contain significant noise, including ambiguous prompts, shifted boxes, and mismatched nested-element annotations.Boxes may shift across an entire image, while source-code-derived elements can create many-to-one or one-to-many mappings.
- Data Cleaning: Manual filtering removes repeated or ambiguous samples, relocates offset boxes, and rewrites unmeaningful instructions to produce about 107k high-quality training samples.The process begins with downsampling large datasets to remove repeated prompts before manual inspection.
- Data Cleaning: Automatic cleaning with open-source models and rejection-sampled hard examples failed to work in the reported experiments.These failed attempts nevertheless provided insights for the UI-agent community.
- Reward Function: The grounding reward combines format correctness with point-in-box localization, using weights w1 and w2 to control their relative importance.Valid syntax receives a base reward, while the localization component evaluates whether the predicted box center lies inside the ground-truth box.
3.3 UI Navigation
UI-Venus builds navigation training data from multiple sources, then cleans and augments it through filtering, trace reconstruction, and iterative trajectory generation. Its navigation method aligns historical thought–action context between epochs and enhances sparse actions to improve planning in complex UI scenarios.
- Data Collection: 350k high-quality samples are selected for UI-Venus navigation training from collected data sources.The collection includes open-source navigation datasets, grounding data, and approximately 20k samples from Chinese mobile applications.
- Data Pipeline: The navigation data pipeline filters inconsistent traces, reconstructs information-retrieval traces with final answers, and iteratively generates additional trajectories.Filtering standardizes scroll directions and resamples by apps and subtasks; reconstruction inserts a CallUser answer before Finish.
- Trajectory History Alignment: Historical context consists of prior thought–action pairs, with the context and current screenshot forming the input for each next-step prediction.Thoughts represent reasoning behind actions, while actions represent the executed operations.
- Trajectory History Alignment: The self-evolving alignment mechanism refines historical thoughts between training epochs, producing more coherent context for navigation.Candidate reasoning traces are aligned with the model’s evolving decision patterns and reused in subsequent training epochs.
- Trajectory History Alignment: Inconsistent historical reasoning can reduce decision quality because overly short thoughts omit intermediate observations while verbose thoughts obscure decision milestones.The method targets variability in reasoning style, detail, and abstraction level across navigation data.
- Sparse Action Enhancement: Sparse Action Enhancement addresses imbalanced action frequencies by increasing representation of rare but critical operations such as LongPress.Common actions like Click and Scroll occur frequently, whereas sparse actions can be pivotal to completing complex trajectories.
4 Experiments
UI-Venus is evaluated across grounding and navigation benchmarks using separately trained models and extensive implementation protocols. It achieves state-of-the-art or comparable performance across diverse GUI settings, including multilingual, professional, offline, and online tasks.
- Grounding Benchmarks: 94.1% on ScreenSpot-V2 lets UI-Venus-Ground-7B outperform larger UI-TARS-72B and OpenCUA-32B models scoring 90.3% and 93.4%.The benchmark spans mobile, web, and desktop interfaces with text and icon/widget elements.
- Grounding Benchmarks: UI-Venus-Ground-72B reaches 10.5%, 4.5%, and 9.5% improvements over GTA1-72B for icon grounding in Creative, Scientific, and Office categories.ScreenSpot-Pro uses high-resolution professional software interfaces where small, diverse icons are especially difficult to locate.
- Grounding Benchmarks: 70.4% average on OSWorld-G text matching shows exceptional fine-grained performance relative to UI-TARS-1.5-7B and the GTA1 series.OSWorld-G also evaluates element recognition, layout understanding, fine-grained manipulation, and refusal handling.
- Grounding Benchmarks: 95.3% on ScreenSpot-V2, 61.9% on ScreenSpot-Pro, and 85.0% on CA-GUI establish UI-Venus-Ground-72B as state of the art across these benchmarks.It surpasses GTA1-72B at 94.8% and 58.4% on the first two benchmarks, and UI-TARS-1.5 at 95.2% and 61.6%.
- Grounding Benchmarks: 83.2% average accuracy on CA-GUI demonstrates robust multilingual grounding, with UI-Venus exceeding AgentCPM-GUI by 5% on Fun2Point and 9.4% on Text2Point.The evaluation uses realistic Chinese mobile applications and non-English GUI environments.
- Navigation Benchmarks: 65.9% success rate on AndroidWorld makes UI-Venus-Navi-72B outperform UI-TARS-1.5, while the 7B model reaches 49.1% and exceeds UI-TARS-72B.The online benchmark requires screenshot-only, end-to-end interaction with live mobile applications and real-time feedback.
5 Future Work
Future work centers on reducing hallucination-driven mismatches between reasoning and actions, while improving adaptability to unfamiliar applications through broader trajectory pretraining.
- Reducing mismatches between internal reasoning and final navigation actions remains an open problem.
- Broader pretraining on diverse UI-agent trajectories may improve adaptability to unfamiliar applications.
A Prompt Templates
The prompt templates specify screenshot-based task reasoning, action selection, history use, and explicit completion behavior for UI navigation and grounding.
- A.1 Grounding: Grounding prompts require outputting only the bounding-box coordinates for the instructed screen position.
- A.2 Navigation: Navigation prompts ask the GUI agent to analyze the task, current screenshot, and previous actions before choosing the next action.
- A.2 Navigation: Previous actions are provided as historical context for navigation decisions.
- A.2 Navigation: Additional actions support user calls, long presses, back, home, enter, and recent-app navigation.
- A.2 Navigation: The instructions emphasize understanding the goal and carefully examining the screenshot because summarized history may overclaim effects.
- A.2 Navigation: Question or chat requests require an explicit CallUser reply before the agent finishes the task.
- A.2 Navigation: Text transfer uses selection or long-press interactions to expose copy and paste controls, while reasoning, action, and conclusion use dedicated tags.
B.1 Grounding
Grounding examples cover desktop software, operating systems, websites, shopping interfaces, and mobile applications, with target regions shown as dotted red boxes.
- Grounding examples span desktop professional software, desktop systems, websites, and mobile interfaces.
- Blender grounding targets the control for increasing the Z axis, marked by a dotted red bounding box.
- Excel grounding targets the Redo control, marked by a dotted red bounding box.
- Linux grounding targets the setting for changing terminal encoding to legacy GBK, marked by a dotted red bounding box.
- GitLab grounding targets the control for adding a new item, marked by a dotted red bounding box.
- A shopping-site example targets the customers view, marked by a dotted red bounding box.
- A mobile Chinese-instruction example displays grounding results as dotted red boxes.
B.2 Navigation
Navigation case studies show UI-Venus operating across Chinese-language interfaces and AndroidWorld tasks, including successful goal completion and one documented think–action conflict.
- A Chinese-language navigation trace demonstrates UI-Venus operating with Chinese interface elements and a Chinese goal description.
- In ContactsNewContactDraft, UI-Venus entered the requested contact details without pressing Save.
- In MarkorDeleteAllNotes, UI-Venus achieved the goal and exhibited reflection during Step 3.
- The MarkorDeleteAllNotes trace also contains a conflict between thought and action in Step 5.