Source-linked AI summary
UI-Venus-2 Technical Report
Venus Team, Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, Weizhi Chen, Changlong Gao, Zhangxuan Gu, Yuan Guo, Yusong Hu, Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, Changhua Meng, Han Peng, Xinyu Qiu, Shuheng Shen, Zhongyi Shui, Weiqiang Wang, Ming Wen, Zhuoer Xu, Hang Yan, Kaiwen Yang, Ruilin Yao, Nanjun Yu, Zhengwen Zeng, Lianrui Zhang, Yunzhu Zhang, Zhe Zhao, Beitong Zhou
TL;DR
UI-Venus-2 targets the gap between benchmark-oriented GUI agents and dependable real-world automation, where environment coverage, task construction, and reward verification remain limited. It presents a unified closed-loop agent and jointly scales multilingual mobile, web, desktop, task-generation, and verification capabilities. The resulting open-source system reports strong performance across GUI benchmarks, including 59.7% on KnowUBench and 80.2% on REAL for the 27B model.
Problem
GUI agents remain brittle for real-world use because benchmarks provide limited environment coverage, task construction can be unreliable, and coarse verification can produce exploitable reward signals.
Method
UI-Venus-2 combines a closed-loop reasoning–action framework with jointly scaled environments, function-grounded task generation, visual-keypoint verification, and multi-model voting.
Results
59.7% on KnowUBench and 80.2% on REAL are achieved by UI-Venus-2-27B, surpassing the strongest reported baselines on both benchmarks.
Takeaways & Limitations
UI-Venus-2 provides an open-source, computationally efficient foundation for more generalizable, verifiable GUI-agent research across mobile, web, and desktop environments.
Abstract
from arXiv · showhide
Multimodal GUI agents have emerged as a promising paradigm for digital task automation, yet transitioning from benchmark-oriented models to dependable real-world applications remains challenging due to limited environment coverage, brittle task construction, and unreliable reward verification. In this work, we present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework. To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training. Furthermore, we integrate safety-aware mechanisms to ensure controlled execution of consequential actions. By offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the field toward more generalizable, verifiable, and self-reflective agents for real-world applications.
1 Introduction
UI-Venus-2 addresses the gap between benchmark-oriented GUI agents and dependable real-world use by jointly scaling environments, executable task construction, and reliable verification. It unifies mobile, web, and desktop operation within a closed-loop reasoning–action agent and emphasizes controlled, reproducible deployment.
- Motivation: Real-world GUI agents require broader environment coverage, executable function-grounded tasks, and reward verification that distinguishes true completion from superficial progress.The paper identifies multilingual mobile ecosystems, dynamic websites, desktop operating systems, scalable task construction, and robust verification as central challenges.
- Approach: UI-Venus-2 operates across mobile applications, web platforms, and desktop operating systems through a unified closed-loop reasoning–action paradigm.The agent observes the interface, reasons about task state, executes an action, and uses environmental feedback for its next decision.
- Environment and task scaling: 100+ Chinese and 70+ English applications expand the multilingual mobile environment, while deep-research task generation grounds instructions in application functionality.The strategy targets accuracy, validity, and executability of generated tasks.
- Verification: Visual keypoints, trace-level and sample-level verification, and heterogeneous-model voting provide finer-grained and more robust reinforcement-learning signals.These mechanisms reduce single-judge bias and make rewards less susceptible to reward hacking.
- Open source and evaluation: UI-Venus-2 is released with full-parameter weights and evaluation infrastructure, achieving almost state-of-the-art performance among comparable-scale models across multiple GUI benchmarks.The release is intended to support reproducible research and lower the barrier to GUI-agent development.
2 Methodology
UI-Venus-2 combines visual reasoning with reinforcement-learning-driven interaction in a unified pipeline spanning trajectory mid-training, domain-specific offline RL, and multi-teacher on-policy distillation. Its distillation stage concentrates supervision on structured executable actions, whose errors directly affect environment transitions.
- System overview: UI-Venus-2 observes rendered interfaces, translates natural-language intent into GUI actions, and adapts decisions continuously using environmental feedback.This closed-loop capability spans mobile applications, desktop operating systems, and web platforms.
- Stage I: Multimodal Mid-Training: Mobile, Web, and OS navigation dominate a heterogeneous mid-training corpus collected from simulated interactive environments and executable trajectories.The data spans application interfaces, webpages, and operating-system contexts.
- Training pipeline: The three-stage training pipeline uses trajectory-based mid-training, independent step-level Offline-RL across five domains, and multi-teacher on-policy distillation.The domains are Grounding, CAPTCHA, Mobile, Web, and Computer.
- Stage II: Offline RL: Step-level offline reinforcement learning supervises state-aware action selection, multi-step navigation, transition consistency, and execution reliability across Mobile, OS, and Web tasks.CAPTCHA and Grounding use a different programmatic synthesis framework.
- Stage III: Multi-teacher On-policy Distillation: Because actions alone change the environment, structured action-aware distillation increases supervision for incorrect action spans while respecting dependencies between action types and parameters.Correct complete actions need no correction; incorrect types receive emphasis while downstream parameters are masked.
- Stage III: Multi-teacher On-policy Distillation: The teacher-side hint specifies the correct action type only during scoring, and the distillation advantage compares teacher and student token probabilities with gradients stopped.The student does not receive the hint at inference time.
3 Data Generation and Verification
UI-Venus-2 builds data through a closed-loop system that catalogs capabilities, synthesizes executable contracts, collects trajectories, and feeds observed outcomes back into task generation. Verification combines semantic trajectory inspection, visual completion keypoints, step-level checks, and rules or model voting for ambiguous cases.
- Capability Catalog Construction: Capability catalogs aggregate application evidence and dynamically incorporate observed page states, UI constraints, entity validations, and failure cases.Runtime observations update later task-sampling distributions toward the executable state space.
- Task Construction: Task construction samples catalog capabilities to create single-capability, composite, query-based, batch, and scenario-based tasks formalized as executable contracts.Contracts bind instructions to domains, capability tags, initial states, resources, and expected outcomes.
- Trajectory Collection: Trajectory collection executes contracted tasks through screenshot–action loops and writes both successful and failed execution feedback back into the capability catalog.Feedback includes observed functions, invalid entities, unmet preconditions, and specific failure causes.
- Web Navigation: Web data covers over 4,000 domains across 19 categories and uses real Chrome sessions with a 15-action Playwright interface.The collection combines benchmark and ranked-web sources with accessibility checks and model-based quality scoring.
- Computer Use: Computer-use tasks serialize controlled desktop snapshots, setup operations, resources, provenance, and outcome evaluators into TaskSpec records.Materialization plans provision workspace and warmup state, while fixture fingerprints support deduplication and leakage detection.
- Verification: Semantic Guided Verification extracts completion keypoints, accumulates windowed trajectory evidence, applies hard rules to clear cases, and routes ambiguous cases to multimodal judgment.Sample-level verification adds pre-execution action correctness checks and fine-grained supervision.
4 Experiments
UI-Venus-2 is evaluated across mobile, desktop, web, and GUI-grounding benchmarks, where it generally achieves leading or highly competitive performance across diverse interaction settings. Results also show strong performance at smaller scale, while some comparisons depend on differing scaffolds or reveal persistent long-horizon challenges.
- Mobile Use: 60.5% on MobileGym: UI-Venus-2-27B surpasses Seed2.0 Pro at 52.0% and prior GUI-specific models’ best result of 21.5%.UI-Venus-2-9B reaches 52.7%.
- Mobile Use: 48.7% on VenusBench-Mobile: UI-Venus-2-27B exceeds Opus 4.6 at 36.5% in realistic, user-centric mobile evaluation.The 9B variant reaches 46.5%.
- Mobile Use: 84.0% on AndroidWorld: UI-Venus-2-27B improves 6.4 points over the previous best, despite strong existing baseline performance.The 9B variant achieves 80.2%.
- Mobile Use: 70.3% on MemGUI: UI-Venus-2-27B leads Seed2.0 Pro at 65.6% on memory-intensive mobile GUI scenarios.The 9B variant reaches 62.6%.
- Computer Use: 80.5% on OSWorld-Verified and 55.5% on DeskCraft: UI-Venus-2-27B is competitive across desktop benchmarks, leading DeskCraft but trailing Claude-Opus-4.8 on OSWorld-Verified.DeskCraft performance exceeds Kimi-K2.6 by 14.1 points; OSWorld-Verified comparisons use model-specific action scaffolds.
- Web Navigation: 78.3% on Online-Mind2Web and 80.2% on REAL: UI-Venus-2-27B achieves the best reported performance on both live-web and deterministic realistic-web benchmarks.The 9B variant reaches 74.0% on Online-Mind2Web and 76.9% on REAL.
5 Related Works
Related work frames GUI research as moving toward general-purpose interaction across mobile, web, and desktop environments, with increasing demands for perception, reasoning, grounding, and adaptation.
- Cross-domain GUI agents: GUI research is moving from isolated operations toward general-purpose interaction across mobile devices, web browsers, and desktop operating systems.Practical agents must integrate perception, grounding, reasoning, planning, and interaction across domains.
- Mobile Use: Mobile-use evaluations increasingly emphasize real applications and dynamic device environments rather than only static screenshots, recorded trajectories, or fixed benchmarks.Interface states, application versions, languages, accounts, and interruptions can affect execution.
- Computer Use: Desktop tasks involve dense interfaces, rich action spaces, and longer sequences requiring state understanding, planning, keyboard shortcuts, file manipulation, and window management.Professional software may require coordination across multiple applications.
- Web Navigation: Live-web navigation must continuously reassess and adapt because network failures, real-time updates, authentication, pop-ups, advertisements, and regional differences can change interaction paths.Cross-site tasks add uncertainty because websites evolve independently.
- GUI Grounding: GUI grounding is challenging on high-resolution screens and professional software because controls may be dense, visually similar, ambiguous, or absent.Agents must combine visual, functional, and contextual evidence while avoiding incorrect actions when no valid target exists.
- CAPTCHA Solving: Modern CAPTCHA solving combines fine-grained perception, spatial reasoning, precise grounding, and action generation across clicking, rotation, sliders, dragging, and spatial reasoning.Benchmark performance may not fully reflect deployed interaction patterns.
- GUI Agent Safety: Real-world computer-use agents are safety-critical because seemingly benign instructions can embed latent environmental risks and produce harmful outcomes.Earlier evaluations primarily focused on explicitly risky or adversarial settings.
6 Conclusion
UI-Venus-2 combines multilingual environments, desktop computer-use capabilities, and fine-grained verification to achieve state-of-the-art performance across diverse platforms. The authors position it as a reproducible baseline and collaborative platform for safer, more reliable, general-purpose GUI agents.
- 6 Conclusion: UI-Venus-2 achieves state-of-the-art performance through joint scaling of multilingual environments, desktop computer-use capabilities, and fine-grained verification mechanisms.The evaluation emphasizes generalization across mobile, web, and desktop operating systems.
- 6 Conclusion: The authors present UI-Venus-2 as a reproducible baseline and collaborative platform for advancing safe, reliable, and general-purpose GUI agents.
- 6 Conclusion: The stated deployment scope targets realistic human everyday use across diverse digital platforms.
7 Contributions
The listed contributors are presented in alphabetical order by last name.
- Contributors: Zhuohan Cai, Haoxing Chen, Jiaxuan Chen, and Weizhi Chen are listed among the contributors.
- Contributors: Jianrong Jiang, Jianguo Li, Runze Li, Jinzhen Lin, Zhenyu Ma, and Changhua Meng are listed among the contributors.
- Contributors: Changlong Gao, Zhangxuan Gu, Yuan Guo, and Yusong Hu are listed among the contributors.
A Action Space and Prompt Templates
The appendix defines a unified action space and prompt templates for mobile, web, desktop OS, and CAPTCHA interaction. These templates specify available actions, output formats, execution constraints, and guidance for inspecting interface state.
- A.1 Action Space: Table 7 unifies the action space and maps actions from existing open-source datasets into it.
- A.2 Grounding: Grounding prompts require outputting a target center point as [x,y], or [-1,-1] when the task is infeasible.
- A.3 Mobile: Mobile prompts instruct the agent to analyze the task and screenshot, then execute precise actions using a defined function set.
- A.3 Mobile: Mobile outputs use think and action fields, and the agent marks the task completed with a status message.
- A.3 Mobile: Mobile guidance emphasizes understanding the goal, checking screenshots, requesting user input when needed, and exploring hidden content with swipes.
- A.4 Web: Web prompts require reviewing the task, current screenshot, and previous actions before determining the next action.
- A.4 Web: Web instructions prioritize filters, support scrolling and note-taking, and prohibit repeating unchanged actions or continuously waiting.
- A.5 OS: Desktop OS templates define mouse, keyboard, drag, scroll, typing, and modifier-key actions, while recommending atomic actions and screenshot-based verification.
B Grounding Synthesized Example
The synthetic grounding dataset includes interface screenshots spanning desktop, mobile, and web platforms. It covers both professional software and consumer applications.
- The dataset features screenshots from macOS, Windows, iOS, and web platforms.
- Its interfaces represent both professional software and consumer applications.
- The examples are presented as images from a synthetic grounding dataset.
C.1 VenusBench-CAPTCHA
VenusBench-CAPTCHA evaluates GUI agents on practical CAPTCHA scenarios requiring visual interpretation, reasoning, and executable actions. Its diverse tasks support both overall comparison and category-level diagnosis.
- Coverage and task design: VenusBench-CAPTCHA covers eight interaction types, including OCR entry, ordered clicks, rotation, dragging, slider puzzles, visual reasoning, and verification.The benchmark contains 219 screenshots, with 30 examples in each of the first seven categories and 9 one-click verification examples.
- Coverage and task design: The benchmark preserves mobile or web context across 70 image-size configurations and portrait and landscape layouts.
- Coverage and task design: Tasks are mapped to Click, Click–Type, or Drag actions under a common CAPTCHA-specific prompt.
- Diagnostic value: Evaluation checks executable actions, including action type, count, target region, order, entered text, and drag geometry.
- Diagnostic value: The benchmark separately diagnoses recognition, grounding, sequence following, spatial transformation, visual reasoning, and drag-distance estimation.
C.2 Public-Benchmark Task Coverage
The public-benchmark evaluation spans multiple CAPTCHA datasets and task formulations, with explicit exclusions where released annotations are unreliable or task types fall outside scope.
- Benchmark selection: The evaluation includes Spatial-CAPTCHA-Bench, MCA-Bench, NextGen-CAPTCHAs, and Open CaptchaWorld.
- Spatial-CAPTCHA-Bench: Spatial-CAPTCHA-Bench contains 1,050 image–text instances covering seven formulations across easy, medium, and hard difficulties.
- MCA-Bench: MCA-Bench has 4,000 test examples across 20 categories, with 50 examples per category sampled for evaluation.
- NextGen-CAPTCHAs: NextGen-CAPTCHAs evaluation lists 15 retained task types and excludes Red_Dot and Static_Jigsaw because their annotations contain label errors.
- Open CaptchaWorld: Open CaptchaWorld evaluation covers 16 retained task types.
- Benchmark exclusions: Additional excluded task types include Click_Order, Misleading_Click, Pick_Area, and Place_Dot because released annotations contain label errors.
C.3 Evaluation Protocol
The evaluation fixes datasets before model runs and scores complete CAPTCHA action sequences under a single-turn interface. Strict matching requires correct action structure and target execution.
- Evaluation sets: Evaluation sets are fixed before model execution, with identical data used for every model.
- Evaluation sets: Invalid annotations are excluded without post hoc correction, and tasks are not selected according to model performance.
- Model interface and action parsing: Models receive one screenshot, an instruction, and a CAPTCHA-specific prompt, then emit the complete solution in one response.
- Model interface and action parsing: Supported outputs include Click, LongPress, Click–Type pairs, and Drag, while only the action sequence is scored.
- Action-level correctness: Scoring is strict: action types and counts must exactly match, and clicks must fall inside the required annotated bounding boxes.
- Action-level correctness: Ordered tasks additionally require the annotated action order.