PaperlayerView exact revision on arXiv

Source-linked AI summary

GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent

Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, Jianwei Yin

arXiv:2412.18426v1cs.AI

TL;DR

GUI-agent research lacks comprehensive, standardized evaluation of autonomous GUI testing beyond isolated task automation. The paper introduces GTArena, a formalized benchmark spanning test-case generation, task execution, and GUI defect detection across real, injected, and synthetic defects. Results show notable gaps between current multimodal agents and practical GUI-testing requirements, while the proposed analysis links task-specific performance with general capabilities.

  • Problem

    Existing GUI-testing research lacks standardized, transparent, reproducible evaluation across the complete testing process.

    Source: §1 · pp. 1–2

  • Method

    GTArena formalizes GUI testing with a POMDP framework, Transition Tuple defect model, unified benchmark, and three defect-data sources.

    Source: §1 · p. 2, §2.1 · p. 3, §2.1.1 · p. 3

  • Results

    Current multimodal agents show notable performance gaps across test-case generation, task execution, and GUI defect detection relative to practical application needs.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The benchmark supports targeted model improvement by comparing task-specific performance with broader general capabilities.

    Source: §1 · p. 2, §7 · p. 8

    GUI testing remains constrained by partial observability and by dependencies among perception, navigation, and reasoning capabilities that are difficult to disentangle.

    Source: §1 · p. 2, §4 · p. 6

Abstract

from arXiv · show

Nowadays, research on GUI agents is a hot topic in the AI community. However, current research focuses on GUI task automation, limiting the scope of applications in various GUI scenarios. In this paper, we propose a formalized and comprehensive environment to evaluate the entire process of automated GUI Testing (GTArena), offering a fair, standardized environment for consistent operation of diverse multimodal large language models. We divide the testing process into three key subtasks: test intention generation, test task execution, and GUI defect detection, and construct a benchmark dataset based on these to conduct a comprehensive evaluation. It evaluates the performance of different models using three data types: real mobile applications, mobile applications with artificially injected defects, and synthetic data, thoroughly assessing their capabilities in this relevant task. Additionally, we propose a method that helps researchers explore the correlation between the performance of multimodal language large models in specific scenarios and their general capabilities in standard benchmark tests. Experimental results indicate that even the most advanced models struggle to perform well across all sub-tasks of automated GUI Testing, highlighting a significant gap between the current capabilities of Autonomous GUI Testing and its practical, real-world applicability. This gap provides guidance for the future direction of GUI Agent development. Our code is available at https://github.com/ZJU-ACES-ISE/ChatUITest.

1. Introduction

GUI-agent research has largely emphasized isolated task automation, while autonomous GUI testing requires broader, standardized evaluation. GTArena formalizes the end-to-end process, benchmarks its components, and links task-specific performance with general model capabilities.

  • Existing GUI agents mainly perform isolated tasks, while automated GUI testing poses broader and more complex challenges.Source: §1 · p. 1
  • Prior GUI-testing frameworks vary in metrics, use small or inaccessible datasets, and often rely on complex, costly GPT-based designs that hinder reproducibility and privacy.Source: §1 · pp. 1–2
  • GTArena formalizes the complete testing process and provides a standardized, reproducible environment for fair evaluation of multimodal agents.Source: §1 · p. 2
  • The benchmark evaluates end-to-end performance and individual testing components, revealing gaps between current agents and practical GUI-testing needs.Source: §1 · p. 2
  • The proposed correlation analysis compares models fine-tuned for specific tasks with their general benchmark capabilities to identify targeted improvement needs.Source: §1 · p. 2

2. Autonomous GUI Testing Agent Workflow

The workflow decomposes autonomous GUI testing into test-case generation, task execution, and defect detection, using a POMDP formulation and transition-based defect representation. It records agent interactions and categorizes both display and interaction defects.

2. Autonomous GUI Testing Agent Workflow

  • The workflow comprises Test Case Generation, Test Task Execution, and GUI Defect Detection, enabling structured end-to-end testing.Source: §2 · p. 3

2.1.1 Partially Observable Markov Decision Process

  • The framework models GUI testing as a POMDP because agents observe only partial application state, limiting defect detection from incomplete feedback.Source: §2.1.1 · p. 3

2.1.2 GUI Defect Data Model

  • The Transition Tuple (stateb, action, statea) records GUI states before and after an action, supporting delayed defect analysis across transition paths.Source: §2.1.2 · p. 3

2.2. Test Case Generation

  • Test Case Generation creates test intentions and corresponding action steps so future testing can target high-priority areas across app updates.Source: §2.2 · p. 4

2.3. Test Task Execution

  • During execution, the agent performs Click, Scroll, Type, Stop, and Enter actions while recording screenshots and steps until completion or a problem.Source: §2.3 · p. 4

2.4. GUI Defect Detection

  • Defect detection distinguishes Display Defects from Interaction Defects and classifies issues such as layout errors, nonresponsive operations, and navigation failures.Source: §2.4 · p. 4

3. How To Benchmark Autonomous GUI Testing Agent

The benchmark integrates previously segmented GUI-testing stages into one end-to-end evaluation system. Its data combines real-world defects, controlled injected defects, and synthetic defects to balance authenticity, repeatability, and coverage.

3. How To Benchmark Autonomous GUI Testing Agent

  • The benchmark integrates task execution and defect detection into an end-to-end evaluation system rather than isolating individual components.Source: §3 · pp. 4–5

3.1. Real-world Applications with GUI Defects

  • Real-world defect data is mined from GitHub upgrade logs and issue trackers, then filtered and reproduced to ensure relevant, authentic GUI failures.Source: §3.1 · pp. 5–6

3.2. Applications with injected defects

  • Source-level injection and MutAPK introduce controlled, predefined defects that support repeatable experiments and diverse fault scenarios.Source: §3.2 · p. 6

3.3. Synthetic defect datasets

  • Synthetic datasets transform screenshots from stable applications to simulate visual and interaction defects, extending coverage to complex commercial apps.Source: Figure 3 · p. 5, §3.3 · p. 6

4. Correlation Analysis Between General Capabilities and GUI Autonomous Test Performance

The paper compares task-specific fine-tuning with broad benchmark performance to identify capabilities relevant to GUI Testing. This bridges general capability evaluation and task-specific outcomes while acknowledging that perception and reasoning skills may be interdependent.

  • GUI Testing subtasks require interconnected perception and reasoning abilities, making it difficult to isolate which capabilities drive success.Navigation, visual-element recognition, overlapping-element detection, and handling unresponsive buttons may depend on multiple competencies.Source: §4 · p. 6
  • Fine-tuning on GUI Testing datasets followed by broad benchmark evaluation reveals capabilities relevant to specific testing stages.The comparison uses pre- and post-fine-tuning performance to connect general benchmark changes with task-specific outcomes.Source: §4 · p. 6
  • The framework supports identifying key capabilities, improving visual-agent reliability, and enabling continuous model improvement.Source: §4 · p. 6

5. Experiment

The experiments evaluate multimodal models in a unified framework across test-intention generation, task execution, and GUI defect detection. Results show that leading closed-source models generally outperform open-source models, while explicit test steps improve some models’ execution performance.

5.1. Experiment Setup and Evaluation Criteria

  • The unified setup evaluates test-intention coverage, execution with TM, EM, and SR, and defect detection with accuracy, Recall-D, and Recall-N.Coverage matches generated intentions to human-annotated ground truth, while execution success requires every action in a task to be correct.Source: §5.1 · pp. 6–7

5.2. Baseline Autonomous GUI Testing

  • GPT-4o and Claude perform comparably across most metrics and outperform open-source models, although overall performance remains poor.Table 2 compares models across intention generation, execution, and defect-detection metrics.Source: §5.2 · p. 7, Table 2 · p. 7
  • LLaVA struggles with exact-match execution tasks, while Qwen2-VL performs slightly better, consistent with its prior GUI-data training.Source: §5.2 · p. 7
  • Qwen2-VL’s frequent “no defect” responses yield high recall on normal data but poor performance on defective data, while Claude tends to identify defects.Evaluating Recall-D and Recall-N together is necessary because models can favor one response pattern.Source: §5.2 · p. 7, Table 2 · p. 7

5.3. Ablation Study on Test Task Execution

  • Both LLaVA and Qwen2-VL improve when test steps replace test intentions as instructions, whereas GPT-4o changes minimally and can perform worse than Qwen2-VL.The comparison tests task completion under test-intention versus test-step instructions.Source: §5.3 · p. 7, Table 3 · p. 7

6. Related Work

Prior GUI-agent research emphasizes navigation and task automation, while related benchmarks and defect-detection methods address evaluation and GUI quality from different perspectives. The paper situates GTArena within this broader progression toward multimodal, end-to-end GUI testing.

  • Earlier GUI agents primarily pursued task completion through exploration, while newer methods use multimodal models, prompting, in-context learning, and GUI-specific fine-tuning.Source: §6 · p. 8
  • GUI-navigation benchmarks collect realistic environments or human-operation data, but reported concerns include performance gaps to humans and data-quality limitations.WebArena studies complex web tasks, while AitW and AitZ provide large-scale or annotated navigation data.Source: §6 · p. 8
  • GUI defect-detection research spans scripted testing, UI-component modeling, test repair, semantic input generation, UI-hierarchy extraction, and LLM-based non-crash bug detection.Source: §6 · p. 8

7. Conclusion

The paper presents GTArena as a fair, unified, end-to-end environment for reproducible multimodal GUI-Testing evaluation. Its benchmark covers core testing components and exposes performance gaps that motivate targeted model improvement.

  • The formalized framework decouples GUI defect detection from task execution and introduces a Transition Tuple structure for large-scale defect datasets.Source: §7 · p. 8
  • The unified benchmark evaluates test-case generation, test-task execution, and GUI defect detection for multimodal agents.Source: §7 · p. 8
  • The environment enables convenient and reproducible evaluation of multimodal models as autonomous GUI-Testing agents.Source: §7 · p. 8

A. GUI Defect Types

The GUI defect taxonomy is documented through dedicated tables that specify the defect types used in the benchmark.

B. GUI Defect Dataset Examples

The benchmark illustrates GUI defects through display-defect and interaction-defect tables, real-world defect descriptions, and example artificial-injection episodes.

Related papers

Loading 2412.18426v1…