UI-Venus-2 Technical Report

• UI-Venus-2 jointly scales environments, function-grounded tasks, and verification in a closed-loop agent that observes interfaces, reasons about task state, acts, and uses environmental feedback.The introduction identifies limited coverage, brittle task construction, and unreliable reward verification as coupled barriers to dependable real-world GUI use.Source: §1 · p. 2
• CAPTCHA handling prevents verification gates from stalling trajectory collection and downstream progress, while safety-aware mechanisms control potentially consequential actions.Targeted CAPTCHA data also strengthens the capability, supporting further data scaling.Source: §1 · pp. 2–3
• 100+ Chinese and 70+ English mobile applications are covered, with deep-research task generation grounding executable instructions in application functionality.The paper also expands executable environments across mobile, web, and desktop scenarios.Source: §1 · pp. 2–3
• UI-Venus-2 builds desktop operating-system capabilities from scratch through computer-use data collection and task-specific training, unifying mobile, web, and OS interaction.The evaluation overview includes navigation, grounding, and CAPTCHA-solving tasks across representative mobile, web, and computer-use settings.Source: §1 · p. 3
• Task-completion verification uses task-relevant visual keypoints and voting across heterogeneous models to improve robustness and reduce single-judge bias and reward hacking.The system evaluates both trajectories and samples rather than relying only on a coarse holistic judgment of the final interface.Source: §1 · pp. 2–3
• The authors publicly release UI-Venus-2's full-parameter weights and evaluation infrastructure; it achieves almost state-of-the-art performance among comparable-scale models across multiple GUI benchmarks.The release is intended to support reproducibility and lower barriers to GUI-agent research.Source: §1 · p. 3

2.1 System overview

• The agent interprets rendered interfaces, converts natural-language intent into GUI actions, and adapts from feedback, combining complementary task families initialized from Qwen3.5-9B and Qwen3.6-27B.Grounding, CAPTCHA, Mobile, Web, and Computer tasks combine spatial perception, verifiable interaction supervision, and navigation experience; the framework also uses a unified mid-training, offline RL, and OPD pipeline.Source: §2.1 · pp. 3–4

2.2 Stage I: Multimodal Mid-Training

• Mid-training uses heterogeneous synthetic and interaction-oriented data, with Mobile, Web, and OS navigation dominant, and validates collected trajectories through human inspection and automated assessment.Simulated environments span app interfaces, webpages, and operating-system contexts; query seeds help balance realism, linguistic diversity, and task coverage, while verification filters invalid, ambiguous, or low-quality interactions.Source: §2.2 · pp. 4–5, Figure 3 · p. 5

2.3 Stage II: Offline Reinforcement Learning

• Offline RL trains Mobile, OS, and Web navigation with step-level trajectories, while synthesized CAPTCHA and Grounding tasks provide verified, difficulty-controlled action-level supervision.The combined supervision targets state-aware action selection, multi-step navigation, transition consistency, execution reliability, target identification, and spatial localization under realistic visual clutter.Source: §2.3 · p. 5

2.4 Stage III: Multi-teacher On-policy Distillation

• MOPD consolidates foundation-model reasoning, Grounding and CAPTCHA skills, and Mobile, Web, and Computer navigation into a unified policy, using student-sampled trajectories to avoid directly merging independent experts.Because executable actions alone change the environment and occupy a small response span, the method weights distillation according to action correctness: it suppresses correct actions, strengthens supervision for incorrect parameters when the type is correct, and emphasizes type tokens while masking dependent parameters when the type is wrong.Source: §2.4 · pp. 5–6
• A correct-action-type hint is supplied only to the routed teacher scoring the student’s trajectory, not to the student or at inference, concentrating distillation on executable behavior.The teacher evaluates rather than generates a separate response; the report says these GUI-adaptive designs yield more robust fusion across diverse GUI domains.Source: §2.4 · p. 6

3.1 General Generation Pipeline

• An iterative pipeline links capability-catalog construction, executable task generation, and real-environment rollout, using observed outcomes to adapt future task sampling.The feedback loop shifts sampling toward under-explored capabilities and previously failing scenarios.Source: §3.1 · p. 7
• The capability catalog combines application evidence with runtime observations to represent executable functions and update sampling as rollouts reveal states, constraints, validations, and failures.During cold start, Deep Research aggregates official documentation, help pages, user discussions, common workflows, and historical tasks; the catalog records function signatures, required objects and preconditions, compatible functions, and task coverage.Source: §3.1 · p. 7
• Tasks span single-capability, composite, query-based, batch, and scenario-based forms, and a validity gate rejects unsupported, ambiguous, dependency-incomplete, or unverifiable tasks.Each task contract binds its instruction to the application domain, capability tags, initial state, required resources, and expected outcome; successful and failed rollouts feed observed functions, invalid entities, unmet preconditions, and failure causes back into catalog refinement and coverage-aware sampling.Source: §3.1 · p. 7

3.2 Domain-Specific Process

• Web and desktop collection adapt the pipeline to their environments, using curated interactive sites and serialized, preflight-checked desktop tasks with segmentation for long-horizon work.Web curation yields over 4,000 domains across 19 categories, seeds the catalog with 45,000 InSTA-150k-v3 tasks, and uses real Chrome sessions with a 15-action Playwright interface; desktop TaskSpecs specify snapshots, setup, resources, provenance, and evaluators, while verified segment exits seed subsequent segments and accepted prefixes replay to preserve context. The web pool combines browser-agent benchmarks with the Tranco ranking (Pochat et al., 2018), and trajectories are sanitized to remove redundant waits, opposite-direction scrolling, and cyclic action patterns.Source: §3.2 · pp. 7–8
• Synthetic GUI grounding validates pixel-aligned targets from executable interfaces, while synthetic CAPTCHA records derive machine-verifiable answers and action traces from latent puzzle states.Grounding uses DOM semantics and accessibility attributes, nine-point hit tests, and text-node refinement; infeasible instructions become hard negatives. CAPTCHA rule engines cover 70 types, verify solvability before rendering, and export observations, normalized geometry, reasoning supervision, and executable actions.Source: §3.2 · p. 8

3.3 Trajectory Verification

• Trajectory and sample-level verification jointly provide semantic quality stratification and a priori action judgments, supplying multi-resolution reward signals and feedback to task generation.Trace-level verification evaluates semantic alignment of the execution trace with the objective, while sample-level verification checks action correctness before execution using the current screenshot, action, reasoning, and task goal.Source: §3.3 · pp. 8–9

3.3.1 Trace-level Verification

• Semantic Guided Verification (SGV) uses completion keypoints and trajectory evidence to classify runs as completed, partial, infeasible, or failed, including runs that end in failure, timeout, or escalation.It segments trajectories into windows, marks click actions, judges keypoint satisfaction, applies hard rules to unambiguous cases, and routes ambiguous cases to multimodal judgment. SGV stratifies data for curation rather than supplying direct training labels or benchmark scores; its categories guide candidate selection, review or repair, feasibility-rule refinement, failure diagnosis, and catalog updates.Source: §3.3.1 · p. 9

3.3.2 Sample-level Verification

• Sample-level verification classifies pre-execution actions as correct, exploratory, ineffective, or incorrect, then aggregates them into trajectory assessments that distinguish task-grounded progress from generic operations.A two-stage decision checks reasoning-action consistency before task alignment. Infeasible tasks are assessed by whether the agent identifies infeasibility; feasible tasks receive partial credit only if they include at least one valid task-specific operation, not merely app launching, tab switching, or scrolling.Source: §3.3.2 · pp. 9–10

4.1.1 Implementation details

• Inference uses temperature 1.0 and default visual resolution; reasoning is enabled for general tasks, disabled for GUI grounding, and CAPTCHA supports multi-action parsing.General tasks retain and reuse the complete reasoning history; GUI grounding uses temperature 0 for direct localization. Individual benchmark implementation details are provided in their respective experimental sections.Source: §4.1.1 · p. 10

4.1.2 Benchmarks

• The evaluation spans six mobile benchmarks covering everyday use, realistic Android tasks, long-horizon cross-app scenarios, personalization, and memory.MobileGym (Wu et al., 2026a) provides a controllable simulation; VenusBench-Mobile (Gong et al., 2026) tests dynamic everyday use; AndroidWorld (Rawles et al., 2025) uses parameterized real-world Android tasks; MobileWorld (Kong et al., 2026) focuses here on GUI interaction; KnowU-Bench (Chen et al., 2026) tests personalized assistance; and MemGUI-Bench (Liu et al., 2026a) tests memory across sessions and apps.Source: §4.1.2 · p. 10
• Computer-use evaluation covers open-ended desktop tasks, professional workflows with user collaboration, and long-horizon real-world workflows.OSWorld-Verified (Xie et al., 2024) includes desktop, web, operating-system, and multi-app tasks; DeskCraft (Wang et al., 2026) tests clarification, interruptions, and feedback; OSWorld 2.0 (Yuan et al., 2026) covers 108 workflows requiring coordination across apps and information sources.Source: §4.1.2 · p. 11
• Web-navigation evaluation combines live-site interaction, deterministic website simulations, and long-horizon cross-site workflows.WebVoyager (He et al., 2024) evaluates multimodal interaction with popular websites, Online-Mind2Web (Xue et al., 2025) tests realistic live-site tasks, REAL (Garg et al., 2026) provides reproducible simulations, and Odysseys (Jang et al., 2026) tests multi-step navigation across live websites.Source: §4.1.2 · p. 11
• GUI-grounding benchmarks assess precise localization, functional interaction, and spatial understanding across diverse interfaces.VenusBench-GD (Zhou et al., 2025a) and ScreenSpot-Pro emphasize localization; OSWorld-G-R (Xie et al., 2025) tests grounding in functional desktop tasks; and UI-Vision (Nayak et al., 2025) tests visually challenging layouts.Source: §4.1.2 · p. 11
• CAPTCHA evaluation uses five complementary benchmarks spanning spatial reasoning, recognition, logic, target grounding, and executable GUI interaction.VenusBench-CAPTCHA contains 219 screenshots across eight interaction types; the suite also includes 1,050 Spatial-CAPTCHA-Bench examples, a balanced 1,000-example MCA-Bench subset, 319 examples from 15 NextGen-CAPTCHAs task types, and 16 Open CaptchaWorld task types.Source: §4.1.2 · p. 11
• Desktop safety evaluation measures harmful-action attack success on benchmarks for deliberate misuse, prompt injection, model misbehavior, and benign-looking harmful instructions.OSHarm (Kuntz et al., 2026) covers three risk categories, while OSBlind tests safety blind spots; both report Attack Success Rate (ASR), the fraction of tasks where harmful behavior occurs.Source: §4.1.2 · p. 11

4.2.1 Mobile Use

• UI-Venus-2-27B achieves top results on MobileGym (60.5%), VenusBench-Mobile (48.7%), AndroidWorld (84.0%), KnowUBench (59.7%), and MemGUI (70.3%), while trailing specialized Qwen-UI-Agent-27B on MobileWorld at 50 steps.On MobileWorld, UI-Venus-2-27B scores 76.1% at 50 steps and 82.9% at 100 steps; Qwen-UI-Agent-27B scores 82.1% at 50 steps.Source: §4.2.1 · pp. 12–13

4.2.2 Computer Use

• UI-Venus-2-27B leads DeskCraft but places behind Claude-Opus-4.8 on OSWorld-Verified and posts low Binary Accuracy on OSWorld 2.0.On DeskCraft, it scores 55.5%, 14.1 points above Kimi-K2.6 (41.4%). On OSWorld-Verified, it scores 80.5% versus Claude-Opus-4.8 at 83.4%; on OSWorld 2.0, its Binary Accuracy is 2.8% and Partial Score is 13.2%. OSWorld-Verified comparisons reflect model-specific action scaffolds and are not strictly controlled.Source: §4.2.2 · pp. 13–14, Table 2 · p. 13

4.2.3 Web Navigation

• UI-Venus-2-27B leads the reported live-web benchmarks, with strong results on WebVoyager, Online-Mind2Web, REAL, and Odysseys.It scores 93.4% on WebVoyager, 78.3% on Online-Mind2Web, and 80.2% on REAL; on Odysseys it achieves 80.4 averaged and 66.3 perfect rubric scores. WebVoyager uses a refreshed task set and automatic GPT-4o judging; Odysseys uses rubric-based evaluation.Source: §4.2.3 · pp. 14–15, Table 3 · p. 14

4.2.4 GUI Grounding

• UI-Venus-2-27B performs strongly across GUI-grounding benchmarks, leading VenusBench-GD and OSWorld-G-R while ranking second on ScreenSpot-Pro and UI-Vision.It scores 80.1% on VenusBench-GD, 79.1% on OSWorld-G-R, 74.1% on ScreenSpot-Pro, and 66.9% on UI-Vision.Source: §4.2.4 · p. 15, Table 4 · p. 16

4.2.5 CAPTCHA Solving

• On VenusBench-CAPTCHA, MCA-Bench, and Spatial-CAPTCHA-Bench, UI-Venus-2 exceeds the strongest stated general-purpose or task baseline, though Spatial-CAPTCHA scores remain below 50%.UI-Venus-2-27B scores 79.9% Pass@1 on VenusBench-CAPTCHA versus Qwen3.6-27B at 53.0%, and 79.6% on MCA-Bench versus 51.7%. On Spatial-CAPTCHA-Bench, it scores 48.6% versus Seed-2.0-Pro at 43.8%.Source: §4.2.5 · p. 16, Table 5 · p. 17
• UI-Venus-2-27B leads NextGen-CAPTCHAs and Open CaptchaWorld, with a much larger margin over the stated baseline on NextGen-CAPTCHAs.On NextGen-CAPTCHAs, the 27B and 9B models score 54.5% and 47.6% Pass@1, respectively, versus Seed2.0-Pro at 20.4%. On Open CaptchaWorld, the 27B model scores 56.3% versus Seed-2.0-Pro at 55.6%.Source: §4.2.5 · pp. 16–17, Table 5 · p. 17

4.2.6 GUI Agent Safety

• UI-Venus-2 lowers attack success rates on both explicit-threat and benign-context safety evaluations, outperforming the strongest stated baseline on OSHarm.On OSHarm, ASR is 11.3% for the 9B model and 15.3% for the 27B model, versus 18.0% for Qwen3.5-27B. On OSBlind, their ASRs are 48.8% and 47.9%, respectively, compared with 79.4% and 89.3% for their base counterparts.Source: Table 6 · p. 17, §4.2.6 · p. 18

Mobile Use

• Mobile-use research has shifted from static screenshots and fixed benchmarks toward real apps and dynamic devices, raising demands for state tracking, error recovery, and long-horizon decisions.Interface states, app versions, languages, accounts, and interruptions can all affect execution.Source: §5 · p. 18

Computer Use

• Desktop tasks involve denser interfaces and richer action spaces, requiring agents to manage complex actions and longer sequences through planning, memory, recovery, and safe execution.Actions can include keyboard shortcuts, file manipulation, text selection, dragging, window management, and coordination across applications.Source: §5 · p. 18

Web Navigation

• Web navigation must adapt to network failures, changing content, authentication, pop-ups, ads, regional differences, and independently evolving websites.Success requires continuous reassessment of changes that may not have been observed during training.Source: §5 · p. 18

GUI Grounding

• GUI grounding on high-resolution screens and professional software requires precise localization and combining visual, functional, and contextual evidence to handle ambiguous, mismatched, or absent targets.Agents must avoid incorrect actions when no valid target is present.Source: §5 · p. 18

CAPTCHA Solving

• CAPTCHA solving spans ordered clicking, rotation, sliders, dragging, and spatial reasoning, requiring fine-grained perception, precise grounding, and suitable action generation.Benchmark performance may not reflect deployed interaction patterns.Source: §5 · pp. 18–19

GUI Agent Safety

• GUI-agent safety must address costly real-world consequences and risks embedded in benign-seeming instructions, not only explicitly harmful or adversarial requests.Risks may arise from erroneous user instructions, environmental hazards, or agent misalignment.Source: §5 · p. 19

6 Conclusion

• UI-Venus-2 jointly scales multilingual environments, desktop computer-use, and fine-grained verification for state-of-the-art performance and cross-platform generalization, and is intended as a reproducible community baseline for safe, reliable, general-purpose GUI agents.The targeted platforms include mobile, web, and desktop operating systems.Source: §6 · p. 19

A.1 Action Space

• UI-Venus-2 maps actions from existing open-source datasets into a unified action space.Table 7 presents the actions and their definitions.Source: Table 7 · p. 25

A.2 Grounding

• Grounding prompts require the center-point coordinates for the instructed image location, or [-1,-1] when the task is infeasible.The required output format is [x,y].Source: §A.2 · p. 25

A.3 Mobile

• Mobile prompts define coordinate-based actions, app and system controls, text entry, and task completion, with responses formatted as a thought followed by the next action.Guidance emphasizes checking the current screenshot, clarifying with the user when needed, and exploring with Swipe; the prompt also gives specific copy-and-paste steps.Source: §A.3 · pp. 26–27

A.4 Web

• Web prompts add browser-specific controls and require agents to record important screenshot information with TakeNote before scrolling.They also advise using filters for precise searches and avoiding repeated actions when the page is unchanged.Source: §A.4 · pp. 27–29

A.5 OS

• Desktop prompts support mouse and keyboard controls, persistent held inputs, and open-loop Sequences when no intermediate screenshot is needed.The guidance says to release held inputs explicitly, use Wait after slow operations, and use Finished only after successful completion.Source: §A.5 · pp. 29–31

A.6 CAPTCHA

• CAPTCHA prompts restrict effective actions to Click, Drag, Type, and LongPress, and require the complete action sequence at once without an unrequired final submit click.Agents should identify the CAPTCHA type and instruction, ignore unrelated interface elements, and use actions appropriate to selection, input, manipulation, or press-and-hold challenges.Source: §A.6 · pp. 32–33

• Figure 4 shows synthetic grounding dataset images featuring operating systems, office software, entertainment websites, and more.Source: B Grounding Synthesized Example · p. 33

C.1 VenusBench-CAPTCHA

• VenusBench-CAPTCHA tests the full perception–reasoning–action pipeline, since correct interpretation can still fail through inaccurate target selection, ordering, clicking, dragging, or text entry.A verification-gate failure can block an otherwise successful workflow.Source: §C.1 · p. 33
• The compact benchmark contains 219 screenshots across eight CAPTCHA interaction types, with 30 examples in each of seven categories and 9 one-click verification examples.The types are OCR-based text entry, ordered text click, ordered icon click, image rotation, drag-to-end, slider puzzle, visual reasoning, and one-click verification.Source: §C.1 · p. 35
• Its screenshots retain mobile or web context and span 70 image-size configurations in portrait and landscape layouts rather than one visual template.Source: §C.1 · p. 35
• Tasks range from recognition and transcription to semantic selection, order-sensitive grounding, visual reasoning, and continuous geometric manipulation, mapped to Click, Click–Type, or Drag actions.The same CAPTCHA-specific prompt is used across the task types.Source: §C.1 · p. 35
• Evaluation checks executable actions—not just answer text—including action type and count, target region, order, entered text, and drag geometry as applicable.This can expose failures hidden by answer-only or single-point grounding metrics.Source: §C.1 · p. 35
• Fixed screenshots and a deterministic evaluator enable reproducible comparison and routine regression testing, while category-level results diagnose distinct capabilities and the broad performance spread indicates discrimination across evaluated model families.The benchmark probes text recognition and entry, fine-grained grounding, sequence following, spatial transformation, visual reasoning, and drag-distance estimation.Source: §C.1 · p. 35

Spatial-CAPTCHA-Bench

• Spatial-CAPTCHA-Bench coverage uses all 1,050 image–text instances across seven formulations, three difficulty levels, and four spatial abilities.The abilities are reference-system reasoning, orientation and perspective taking, mental rotation, and multi-step spatial visualization; no task type or example is excluded.Source: §C.2 · p. 35

MCA-Bench

• MCA-Bench evaluation uses a fixed, category-balanced 1,000-example subset sampled from its 4,000-example test set.It selects 50 of 200 examples from each of 20 categories, spanning visual recognition, localization, manipulation, and textual logic; the subset is shared across models.Source: §C.2 · p. 35

NextGen-CAPTCHAs

• NextGen-CAPTCHAs results cover only 15 listed task types, excluding Red_Dot and Static_Jigsaw because label errors prevent reliable automatic scoring.The evaluated types are Dice_Roll_Path, Color_Counting, Hole_Counting, Rotation_Match, Backmost_Layer, Layered_Stack, Illusory_Ribbons, Multi_Script, Box_Folding, 3D_Viewpoint, Shadow_Direction, Subway_Paths, Occluded_Pattern_Counting, Shadow_Plausible, and Mirror.Source: §C.2 · pp. 35–36

Open CaptchaWorld

• Open CaptchaWorld’s fixed evaluation set covers 16 listed task types, while four others are omitted because their released annotations contain label errors.The covered types are Bingo, Connect_icon, Coordinates, Dart_Count, Dice_Count, Geometry_Click, Hold_Button, Image_Matching, Image_Recognition, Object_Match, Patch_Select, Path_Finder, Select_Animal, Slide_Puzzle, Unusual_Detection, and Rotation_Match; Click_Order, Misleading_Click, Pick_Area, and Place_Dot are omitted.Source: §C.2 · p. 36

C.3 Evaluation Protocol

• Evaluation uses 219 VenusBench-CAPTCHA examples, 1,050 Spatial-CAPTCHA-Bench examples, 1,000 MCA-Bench examples, 319 NextGen-CAPTCHAs examples, and 16 retained Open CaptchaWorld task types.All sets are fixed before model runs; invalid annotations are excluded, and selection is not based on model performance.Source: §C.3 · pp. 36–37
• In a single-turn setting, models receive one CAPTCHA screenshot, instruction, and CAPTCHA-specific prompt, then emit a complete solution using supported click, long-press, click–type, or drag actions.Only the action sequence is scored; normalized coordinates in [0, 999] or [0, 1], and pixel coordinates, are mapped to native image resolution.Source: §C.3 · p. 37
• A response is incorrect if action types or counts differ from the reference or any action is missing, extra, malformed, or unsupported; target clicks and long-presses must fall inside distinct annotated boxes.Ordered tasks enforce annotated order; other tasks match actions one-to-one to target boxes, while text entry requires an in-box click and an exact answer after trimming surrounding whitespace.Source: §C.3 · p. 37
• Drag tasks require exactly one drag with a start inside the handle box, correct signed horizontal direction and configured relative-error tolerance, and endpoint vertical error below five pixels.For multi-answer bingo, exactly two clicks must match any annotated valid pair, irrespective of order.Source: §C.3 · p. 37