PaperlayerView exact revision on arXiv

Source-linked AI summary

VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics

Yichen Gong, Zhuohan Cai, Sunhao Dai, Yuqi Zhou, Zhangxuan Gu, Changhua Meng, Shuheng Shen

arXiv:2604.06182v1cs.HCcs.AI

TL;DR

Existing mobile GUI benchmarks are app-centric and task-homogeneous, limiting their representation of diverse, unstable real-world usage. VenusBench-Mobile addresses this with user-intent-driven tasks and capability diagnostics, finding roughly 50-point average success-rate drops versus AndroidWorld and near-zero success under environment variations.

  • Problem

    Existing mobile GUI benchmarks are app-centric and task-homogeneous, underrepresenting compositional, cross-app, and intent-driven behaviors relevant to real-world mobile assistance.

  • Method

    VenusBench-Mobile uses top-down, user-intent-driven tasks and capability-oriented annotations to evaluate general-purpose mobile GUI agents under realistic conditions.

  • Results

    SOTA agents show average success-rate drops of around 50 points versus AndroidWorld, with failures dominated by perception and memory deficiencies and near-zero success under environment variations.

  • Takeaways & Limitations

    VenusBench-Mobile reveals that current mobile GUI agents remain far from reliable real-world deployment, particularly because of perception, memory, and robustness limitations.

  • Takeaways & Limitations

    Online learning is evaluated only preliminarily through HTML tasks with randomized button functionalities, remaining a rudimentary benchmark dimension.

Abstract

from arXiv · show

Existing online benchmarks for mobile GUI agents remain largely app-centric and task-homogeneous, failing to reflect the diversity and instability of real-world mobile usage. To this end, we introduce VenusBench-Mobile, a challenging online benchmark for evaluating general-purpose mobile GUI agents under realistic, user-centric conditions. VenusBench-Mobile builds two core evaluation pillars: defining what to evaluate via user-intent-driven task design that reflects real mobile usage, and how to evaluate through a capability-oriented annotation scheme for fine-grained agent behavior analysis. Extensive evaluation of state-of-the-art mobile GUI agents reveals large performance gaps relative to prior benchmarks, indicating that VenusBench-Mobile poses substantially more challenging and realistic tasks and that current agents remain far from reliable real-world deployment. Diagnostic analysis further shows that failures are dominated by deficiencies in perception and memory, which are largely obscured by coarse-grained evaluations. Moreover, even the strongest agents exhibit near-zero success under environment variations, highlighting their brittleness in realistic settings. Based on these insights, we believe VenusBench-Mobile provides an important stepping stone toward robust real-world deployment of mobile GUI agents. Code and data are available at https://github.com/inclusionAI/UI-Venus/tree/VenusBench-Mobile.

1 Introduction

VenusBench-Mobile introduces a challenging, user-centric online benchmark for mobile GUI agents, combining realistic user-intent-driven tasks with capability-oriented diagnostics. Evaluations show substantial performance drops and persistent bottlenecks in perception, memory, and robustness under environment variations.

  • Benchmark design: VenusBench-Mobile evaluates general-purpose mobile GUI agents through user-intent-driven task design that reflects realistic mobile usage beyond app-centric, homogeneous settings.The benchmark adopts a top-down view of how mobile assistants are used in practice rather than testing isolated app functionality.
  • Evaluation findings: Around 50 points: SOTA agents’ average success rates decrease when evaluated on VenusBench-Mobile versus AndroidWorld.The performance gap indicates that VenusBench-Mobile contains substantially more challenging and realistic tasks.
  • Evaluation findings: Perception, memory, and robustness under realistic environment variations remain major limitations of current mobile GUI agents.These bottlenecks contribute to agents remaining far from reliable real-world deployment.
  • Diagnostic evaluation: Its capability-oriented annotation scheme enables fine-grained failure analysis by distinguishing underlying deficiencies obscured by coarse-grained success metrics.Failures can be attributed to specific capability deficiencies rather than reported only as aggregate success rates.

2 Related Work

Related work spans mobile GUI agents that have evolved from rule-based scripts to foundation-model-powered autonomous systems, alongside offline and online benchmarks for evaluating them. These benchmarks differ in whether they assess static action prediction or interactive task execution.

  • Mobile GUI Agents: Mobile GUI agents have evolved from early rule-based scripts to autonomous systems powered by foundation models.They represent an emerging paradigm in human-computer interaction.
  • Mobile GUI Agents: Agent input modalities range from text-only instructions to multimodal text-and-screenshot inputs and pure-visual perception for cross-app automation.The passage categorizes agents by their input modalities.
  • Benchmarks for Mobile GUI Agents: Offline benchmarks such as AndroidControl and GUIOdyssey use static datasets to evaluate action prediction, but lack closed-loop feedback for live system assessment.The passage contrasts offline evaluation with interactive online evaluation.
  • Benchmarks for Mobile GUI Agents: Online benchmarks such as AndroidWorld provide interactive environments for verifiable task execution, while SPA-Bench and MobileWorld scale these environments or introduce hybrid MCP-based tool invocation.These works extend online evaluation through larger environments or hybrid paradigms.

3 VenusBench-Mobile Benchmark

VenusBench-Mobile is designed around user-intent-driven tasks that reflect diverse, imperfect real-world mobile usage rather than app-centric functionality. It combines broad task categories with capability-oriented annotations to measure success and diagnose specific behavioral limitations.

  • Benchmark Design: VenusBench-Mobile defines evaluation around two questions: which mobile-agent tasks to evaluate and how to assess agent performance.The benchmark’s design and construction explicitly address both task selection and performance assessment.
  • Task Taxonomy: User-intent-driven tasks cover diverse and imperfect mobile usage, including functional assistance, conflict resolution, vague instructions, and multi-round interaction.The design treats agents as general-purpose assistants deployed in real-world scenarios rather than as app-functionality executors.
  • Capability Evaluation: Capability annotations enable fine-grained diagnosis across Perception, Understanding, Decision, and Action dimensions.The framework analyzes information extraction, instruction comprehension, strategic adaptation, and GUI interaction execution.
  • Task Taxonomy: The benchmark includes dynamic-state awareness, visual manipulation, hard GUI browsing, noise resistance, Browsecomp-like corner cases, and stability evaluation.These categories test evolving interfaces, precise visual actions, multi-screen information processing, environmental disruptions, multi-constraint tasks, and consistency across varied conditions.
  • Stability Evaluation: Stability Evaluation generates systematic task variants to test robustness across paraphrased instructions and different GUI settings.The dedicated subset assesses whether agents solve the same task consistently under linguistic and visual perturbations.

4 Evaluation Results and Analysis

Evaluation on VenusBench-Mobile uses screenshot-only, stepwise GUI interaction and examines success, capability levels, perturbation stability, and inference cost. Results show substantially lower real-world performance, severe perception and memory deficits, extreme brittleness to variations, and substantial token overhead from agentic reasoning.

  • Evaluation Setup: Screenshot-only evaluation requires agents to receive screenshots and output executable GUI actions step by step.The evaluation examines mainstream mobile GUI agents in a purely vision-based setting.
  • Overall Performance: 36.9% total success is achieved by Gemini-3-Pro, while most open-source models remain below 15% on VenusBench-Mobile.These rates are substantially lower than those reported on traditional benchmarks, indicating greater difficulty in realistic environments.
  • Overall Performance: 2.5% and 4.2% average success rates occur in GSA and GUIM, respectively, exposing blind spots in prior benchmarks.VenusBench-Mobile includes these real-world user-need categories, which were absent from the prior benchmarks summarized in Table 1.
  • Capability Diagnostics: 17.5% to 10.3% is the average Perception score drop from Basic (L1-2) to Advanced (L3-4) levels.Most open-source models also fall below 5% in L3-4 Decision and Perception, while Memory reaches near-zero success for smaller models.
  • Robustness Analysis: 0% Stability Pass Rate is achieved by most models, while the best-performing model reaches only 15% across five task variants.Tablet-mode layout changes cause the largest drops, and language variations reveal inconsistent robustness.
  • Inference Cost: 1.64M total tokens are consumed by MA3, compared with 146.5 tokens per step for standalone GUI-Owl-7B.The agentic framework's per-step cost reaches 438.7, creating a deployment barrier for real-time edge-device use.

5 Conclusion

VenusBench-Mobile evaluates mobile GUI agents under realistic, user-centric conditions through user-intent-driven tasks and capability-oriented diagnostics. It moves evaluation beyond app-centric success rates by supporting fine-grained analysis of agent behaviors.

  • VenusBench-Mobile is a challenging online benchmark for evaluating mobile GUI agents under realistic, user-centric conditions.
  • User-intent-driven task design reflects realistic usage beyond app-centric evaluation.
  • Capability-oriented diagnostic annotation enables fine-grained analysis of agent behaviors.

6 Future Work … A.4 Action (A): Physical GUI Interaction

The paper proposes future benchmark extensions for online learning, lifelong assistance, and scalable user simulation, while defining four proficiency levels across five GUI-agent capability dimensions. The proficiency framework spans perception, instruction understanding, decision-making, and physical interaction, with some levels or dimensions lacking benchmark test cases.

  • 6 Future Work: Future work targets richer online-learning evaluation, long-term personalized assistance, and scalable scenario generation through LLM-based simulated users.Online learning should assess intra-task adaptation and inter-task self-evolution; lifelong evaluation should measure long-term memory and personalization; simulated users could generate diverse instructions, dialogues, and environmental noise.
  • A Four Proficiency Levels Across Five Dimensions: The framework defines four proficiency levels across five capability dimensions, progressing from basic functionality toward general embodied intelligence.Some dimensions are not applicable at lower levels, and representative tasks illustrate each proficiency level within every dimension.
  • A.1 Perception (P): Understanding GUI Environment: Perception levels progress from static visual recognition through semantic layout understanding and precise localization to dynamic temporal perception.The levels cover recognizing labels and controls, understanding multimodal spatial relationships, locating tiny visual elements, and tracking changing interface states.
  • A.2 Understanding (U): Comprehending User Instructions: Instruction understanding ranges from deterministic task execution to complex constraint parsing and disambiguation of vague or environmentally unachievable requests.The framework also defines atomic instruction understanding, for which VenusBench-Mobile has no test cases.
  • A.3 Decision (D): Strategic Reasoning During Execution: Decision proficiency advances from deterministic execution to dynamic strategy adaptation and reflective planning with error correction, alternative paths, and risk interception.The D1 level is not applicable to basic script-following agents, and VenusBench-Mobile has no D1 test cases.
  • A.4 Action (A): Physical GUI Interaction: Action proficiency covers basic touch interactions, controlled scrolling and dragging, precision manipulation, and real-time closed-loop control with visual feedback.VenusBench-Mobile has no test cases for A1 or A4, while A2 and A3 represent trajectory and spatial-precision operations.
  • A.4 Action (A): Physical GUI Interaction: VenusBench-Mobile has no test cases for the A1 basic-operations level.A1 consists of fundamental touch interactions, including tapping and long pressing clearly defined interactive elements.
  • A.4 Action (A): Physical GUI Interaction: The action hierarchy culminates in millisecond-level hand-eye coordination and continuous trajectory correction based on visual feedback.This A4 level concerns dynamic operations such as drawing while adjusting the path during execution.

A.5 Memory (M): Task-Relevant Information Memorization

The memory capability taxonomy spans task-path maintenance, long-term state tracking, and cross-task memory, while basic stateless operation is not tested. VenusBench-Mobile emphasizes increasingly persistent task-relevant information demands across workflows, events, and completed tasks.

  • Memory (M): Task-Relevant Information Memorization: M1 is not applicable because basic-level agents may operate statelessly, and VenusBench-Mobile has no M1 test cases.This level concerns operation without explicit memory mechanisms.
  • Memory (M): Task-Relevant Information Memorization: M2 requires maintaining task objectives and trajectory history across application and page transitions without losing the overall goal.The Tomato timer task tests preserving a pause-at-a-specific-time objective during a dynamic countdown workflow.
  • Memory (M): Task-Relevant Information Memorization: M3 requires tracking extended state evolution by preserving task-relevant information from historical interactions in a dynamic information pool.The SMS task requires filtering eight messages by keywords, preserving qualifying content, and recording it in arrival order.
  • Memory (M): Task-Relevant Information Memorization: M4 requires retaining and retrieving entities, actions, and temporal ordering across completed tasks for retrospective operations.In the Markor example, the agent creates four notes across Rounds 1–4, then must delete the first two in Round 5.

B Benchmark Setting Details

This section details the benchmark setting, covering task curation, action space, app selection, and the configuration of MLLM-as-a-Judge.

  • Benchmark Setting Details: The benchmark setting includes task curation, action-space design, app selection, and MLLM-as-a-Judge configuration.The section provides additional details across these four aspects.

B.1 Details of Primary Task Curation … B.4 MLLM-as-a-Judge Setting

VenusBench-Mobile curates a 149-task primary pool across nine categories and 27 applications, with quality assurance, broad capability coverage, common GUI actions, and MLLM-based verification for visually or semantically evaluated tasks.

  • B.1 Details of Primary Task Curation: The primary task pool contains 149 manually constructed tasks spanning nine categories: FA, CF, VA, MR, GSA, GUIM, HGB, NR, and BC.Tasks run on Android emulators at standard 1080×2400 phone resolution in light mode.
  • B.1 Details of Primary Task Curation: Two-stage quality assurance audits every task and cross-validates PUDAM annotations with independent reviewers.VenusBench-Mobile covers Levels 2–4 more broadly than AndroidWorld, including advanced capabilities.
  • B.2 App List: The benchmark uses 27 applications: 20 from AndroidWorld and 7 newly added to improve GUI diversity and real-world usage coverage.The task distribution reports primary-pool tasks alongside 80 Stability Evaluation variant tasks.
  • B.2 App List: The app-level task counts distinguish 149 primary-pool tasks from 80 additional Stability Evaluation subset variants.This distinction is represented by “# Tasks (Primary + Subset)”.
  • B.3 Action Space: The benchmark’s action space supports common operations used by mobile GUI agents.The supported actions are detailed in table 8.
  • B.4 MLLM-as-a-Judge Setting: Qwen3-VL-30B-A3B-Instruct evaluates 90 of the 149 tasks whose verification requires visual recognition or semantic understanding.These include all tasks in the FA category, using carefully crafted prompts.
  • B.4 MLLM-as-a-Judge Setting: The MLLM-based evaluation framework employs 7 verification interfaces with representative test cases for each interface type.The interfaces are described in the MLLM-as-a-Judge setting.

B.4.1 JUDGE 1: Fact Matching for BC and HGB Tasks

The JUDGE 1 interface evaluates BC and HGB tasks by strictly fact-matching predicted textual answers against deterministic ground truths. It accepts equivalent phrasing but rejects empty, factually different, or extraneous responses.

  • Task Description & Evaluation Logic: BC and HGB tasks use relatively fixed, deterministic ground-truth answers, while predictions may vary in wording.For example, “Expense” and “The answer is Pro Expense” can express the ground-truth answer “Pro Expense.”
  • Prompt Design: The evaluation prompt supplies the task goal, predicted answer, and ground-truth answer, then asks whether the prediction matches.This structure implements the strict fact-matching logic through a mobile GUI task verifier.
  • Fact-Matching Rules: The judge ignores capitalization, punctuation, whitespace, and minor wording differences when determining whether answers express the same fact.The predicted answer must be non-empty before comparison.
  • Fact-Matching Rules: Matching requires identical key factual values, such as numbers, dates, names, filenames, and paths.The core assertion must also be factually equivalent, even when phrased differently.
  • Fact-Matching Rules: Any content beyond the ground-truth answer makes the prediction incorrect.Thus, “Pro Expense, Markor and Calendar” is rejected even though it contains the correct answer.

B.4.2 Judge 2: Semantic Verification for FA Tasks · B.4.3 Judge 3: Conflict Resolution and CF Tasks

Judge 2 semantically verifies Function Assistance answers by checking coverage of ground-truth facts, while Judge 3 evaluates conflict-task refusals for correct decisions and explanations. Both judges tolerate surface-form variation but reject omissions, contradictions, or unsupported successful execution.

  • B.4.2 Judge 2: Semantic Verification for FA Tasks: Judge 2 evaluates Function Assistance answers by requiring coverage of most key steps or core functionalities in the ground truth.The objective is successful task completion rather than exact textual matching.
  • B.4.2 Judge 2: Semantic Verification for FA Tasks: The FA verifier accepts paraphrases, synonyms, and logically equivalent formulations while ignoring capitalization, punctuation, and minor formatting.It treats the ground truth as a set of facts.
  • B.4.2 Judge 2: Semantic Verification for FA Tasks: Predictions covering less than half of ground-truth functionalities are incorrect, as are answers introducing features absent from the ground truth.The judge also rejects answers that miss critical facts or conflict with the ground truth.
  • B.4.2 Judge 2: Semantic Verification for FA Tasks: A ZipXtract example illustrates that semantically equivalent instructions covering all extraction steps pass, whereas automatic extraction claims fail by missing manual steps and contradicting the ground truth.The positive example preserves the sequence of selecting extraction, file, destination, and completion actions.
  • B.4.3 Judge 3: Conflict Resolution and CF Tasks: Judge 3 handles Conflict tasks where the agent must recognize an unachievable request, refuse it, and provide a semantically accurate reason.The evaluation targets both refusal behavior and reasoning quality.
  • B.4.3 Judge 3: Conflict Resolution and CF Tasks: The conflict verifier focuses on semantic agreement and values such as numbers, dates, and names while ignoring capitalization, punctuation, and spacing.The prediction must also be nonempty.
  • B.4.3 Judge 3: Conflict Resolution and CF Tasks: Judge 3 answers no when predictions omit important facts, contradict the ground truth, or claim successful execution where the task should be refused.For Tomoto’s unavailable black theme, a refusal explaining the missing option passes, while a success claim fails.

B.4.4 Judge 4: Visual Page Locating Tasks … B.4.7 Judge 7: GUIM-Editing Verification

Judges 4–7 use MLLM-based visual verification tailored to locating, video, drawing, and image-editing tasks. Their criteria compare screenshots or canvases with task goals while accommodating dynamic video frames and diverse visual outputs.

  • B.4.4 Judge 4: Visual Page Locating Tasks: Locating tasks compare the final screenshot with a ground-truth reference, emphasizing core layout, visible widgets, and functional structure over minor dynamic differences.System timestamps, battery levels, and temporary pop-ups are ignored, while structural equivalence determines whether the target page was reached.
  • B.4.4 Judge 4: Visual Page Locating Tasks: The locating verifier distinguishes correct navigation from opening the right app on the wrong page by checking whether the screenshot matches the reference interface.A positive case preserves the core layout despite a different system time; a negative case shows a different layout.
  • B.4.5 Judge 5 Video Content Verification: Video tasks use direct goal-based screenshot assessment because temporal continuity allows multiple valid frames instead of one fixed ground-truth image.The judge interprets the user query and final screenshot, checking whether the main intent and required elements are fulfilled.
  • B.4.5 Judge 5 Video Content Verification: A video task is successful when the final screenshot shows content satisfying the query, such as an active player displaying water sports rather than a listing or unrelated content.The task example also requires remaining on-screen after completion.
  • B.4.6 Judge 6: GUIM-Drawing Verification: Drawing verification evaluates the final canvas against the task description without a static ground-truth image, checking shape and content, color and detail, and completeness and quantity.The criteria support diverse outputs while assessing geometric and compositional constraints.
  • B.4.6 Judge 6: GUIM-Drawing Verification: The drawing example requires a circle and an internal rectangle whose four vertices lie exactly on the circle’s circumference, with the canvas left on-screen.The positive case satisfies these spatial relationships, whereas the negative case does not.
  • B.4.7 Judge 7: GUIM-Editing Verification: Image-editing verification checks whether requested operations are accurately executed, using operation-specific criteria such as pure-white erasure and clearly visible target-only circling.Extra marked objects, missing targets, or remaining erased content are treated as errors.
  • B.4.7 Judge 7: GUIM-Editing Verification: In the editing example, success means opening fruit.png in Draw, circling only the banana with a red pen, and remaining on-screen after completion.The successful scenario marks only the banana, while the failed attempt marks all fruits.

C Detailed Information of Fine-grained Capabilities · C.1 Implications · D Agent Framework

The appendix validates the fine-grained capability taxonomy by quantifying proficiency-level degradation and identifying perception and memory as major bottlenecks. It also describes a closed-source-model agent framework that combines planning, step summarization, and dedicated UI grounding.

  • C Detailed Information of Fine-grained Capabilities: Perception and Memory show the largest basic-to-advanced degradation, with average drops of 7.2 and 7.5 percentage points respectively.Table 9 reports success rates across Perception, Understanding, Decision, Action, and Memory for basic (L1-2) and advanced (L3-4) tasks.
  • C Detailed Information of Fine-grained Capabilities: Open-source models experience catastrophic collapse at L3-4, with most models falling below 10% in Decision and Memory.The breakdown contrasts basic (L1-2) and advanced (L3-4) proficiency levels across the PUDAM dimensions.
  • C Detailed Information of Fine-grained Capabilities: Even Gemini-3-Pro drops substantially from 43% to 24% in Perception and from 41% to 31% in Memory.These values provide detailed numerical support for the capability bottleneck diagnosis.
  • C Detailed Information of Fine-grained Capabilities: The detailed capability results provide empirical support for validating the capability taxonomy and diagnosing agent bottlenecks.The taxonomy organizes diagnosis around Perception, Understanding, Decision, Action, and Memory.
  • C.1 Implications: Memory improvement requires architectural innovations beyond simply increasing context windows, given catastrophic L3-4 drops even in large models.Perception improvements should target fine-grained spatial reasoning and dynamic state tracking for complex layouts and multi-screen information flow.
  • D Agent Framework: For closed-source evaluation, the framework instantiates both planner and summary models with the evaluated closed-source VLM and uses a dedicated grounding model for UI localization.The agent runs a three-stage pipeline at each step, beginning with action selection from the current screenshot and action-history summaries.
  • D Agent Framework: The agent receives the current user goal, action history, and screenshot, then outputs a JSON-formatted action with a reason.The action space includes status, answer, click, long press, text input, keyboard enter, navigation home, app opening, and waiting.
  • D Agent Framework: After each action, a step-summary prompt compares before-and-after screenshots and records intended effects, failures, and next-step information for future action selection.The resulting summary is added to action history and can preserve information or share it between apps.
Loading 2604.06182v1…