AUITestAgent: Automatic Requirements Oriented GUI Function Testing

1 INTRODUCTION

• Manual testing and scripts require substantial development and maintenance effort across numerous GUI pages and rapidly iterating apps.The paper proposes AUITestAgent to reduce this effort.Source: §1 · pp. 1–2
• Natural-language GUI testing is underexplored, and existing natural-language interaction methods are goal-oriented rather than checking response correctness throughout each interaction.The paper says trial-and-error strategies in those methods can produce inefficient traces and insufficient success rates for testing.Source: §1 · p. 1
• Step-oriented test requirements let AUITestAgent use step descriptions in prompts to support UI interaction and verification.The authors describe this structure as advantageous for reaching industry-level usability.Source: §1 · p. 2
• AUITestAgent separates GUI interaction from function verification, organizes agents according to command complexity, and extracts requirement-relevant information from GUI elements, pages, and interaction traces.Verification follows interaction because verification results do not affect the interaction process.Source: §1 · p. 2
• AUITestAgent’s experiments report the best GUI interaction performance against AppAgent [35] and MobileAgent [28], with 90% bug recall and a false positive rate below 5%.The introduction also reports deployment across an unspecified number of business lines and an unspecified number of new bugs.Source: §1 · p. 2

2 PRACTICES OF GUI FUNCTION TESTING

• Academic testing tools typically target nonfunctional bugs or require substantial manual intervention, so they do not substantially reduce effort for GUI functional testing.The passage cites prior work as.Source: §2 · p. 2
• In Meituan practice, engineers use manual checks or hard-coded scripts, selecting UI elements from hierarchy files and developing interaction and verification rules.Test requirements are typically written in natural language and checked manually on real devices after GUI development.Source: §2 · p. 2
• Commercial apps’ many business lines, functions, and interfaces make GUI functional testing workloads substantial.GUI test scripts reduce some manual effort but can become outdated as apps change, creating maintenance costs.Source: §2 · p. 2

3.1 Overview

• AUITestAgent extracts interaction commands and verification oracles from requirements, chooses an interaction-agent pattern based on command complexity, then verifies logs and outputs results with reason analysis.The logs include encountered pages, screenshots, LLM outputs, and performed UI actions.Source: §3.1 · p. 3

3.2 Natural Language Driven GUI Interaction

• Concise commands need intent interpretation and state reasoning, while multiple agents add unnecessary cost and may amplify hallucinations for detailed commands.AUITestAgent recognizes command type and uses a specific-steps or concise-expression pattern accordingly.Source: §3.2 · p. 3

3.2.1 Specific steps pattern.

• The specific-steps pattern uses an Observer to identify interactive elements and infer their functions, then a Selector chooses actions and an Executor performs them sequentially.The Observer combines screenshot and UI-hierarchy information; screenshot recognition supplements hierarchy data, including when hierarchy access fails or pages contain WebView components.Source: §3.2.1 · pp. 3–4
• The Observer’s multimodal analysis addresses GUI element-recognition and OCR hallucination challenges, while a function knowledge base supplies descriptions for error-prone elements, especially text-free icons.The knowledge base matches current-page elements using CLIP [26].Source: §3.2.1 · pp. 3–4

3.2.2 Concise expression pattern.

• For concise commands, the Planner turns the request into specific interaction steps, and each step is executed using the specific-steps pattern.The concise-expression pattern adds Planner and Monitor agents to that pattern.Source: §3.2.2 · p. 4
• The Monitor checks completion from the current screenshot and executed-operation record; if incomplete, it describes the state and needed actions for the Planner’s next plan.When commands are complete, the module outputs the interaction log and stops.Source: §3.2.2 · pp. 4–5

3.3 Function Verification

• AUITestAgent formulates GUI function verification as semantic analysis of interaction logs, conducted separately from GUI interaction.The paper identifies this as the first work to investigate GUI function verification.Source: §3.3 · p. 5

3.3.1 Simplifying Test Oracles.

• It converts complex test oracles into simple verification points, each framed as one question about the logged interaction.Different points can concern different pages and UI components in the interaction log.Source: §3.3.1 · p. 5

3.3.2 Multi-dimensional Data Extraction.

• Interaction-log data spans GUI elements, GUI pages, and page-state transitions, such as moving from an order page to payment by tapping Confirm.Element data includes functions and properties; page data includes types and categories.Source: §3.3.2 · p. 5, Table 1 · p. 5
• Agents extract data relevant to each verification point, then an MLLM judges the points against that data and returns structured judgments and explanations.Previously analyzed element functions and page types are reused to speed processing.Source: §3.3.3 · p. 5, §3.3.2 · p. 5

4.1.1 Dataset.

• The dataset used 30 interaction tasks across three difficulty levels and 20 verification oracles, each paired with correct and anomalous traces.The interaction tasks came from eight popular apps; the 40 RQ2 verification tasks included anomalies in half of the traces.Source: §4.1.1 · p. 6

4.1.2 Baselines.

• AUITestAgent was compared with AppAgent and MobileAgent for interactions, and with a two-turn multi-turn-dialogue prompt baseline for verification.GPT-4o was used for AUITestAgent and the baselines in RQ1 and RQ2; interaction trials were capped at Step_ideal + 5.Source: §4.1.2 · pp. 6–7, §4.1.3 · p. 7

4.2 RQ1: Quality of GUI Interactions

• 77% overall task completion, with the highest scores across four metrics, put AUITestAgent ahead of both interaction baselines.As task difficulty rose, all four metrics declined; MobileAgent struggled with tasks requiring many commands or concise execution.Source: Table 4 · p. 7, §4.2 · pp. 7–8

4.3 RQ2: Effectiveness of Function Verification

• AUITestAgent reached 0.90 Oracle Acc.; anomaly detection had 0.85 Oracle Acc., 90% recall, and a 4.5% false positive rate.It identified 18 of 20 anomalous points, with 2 false positives; the baseline reached 0.35 Oracle Acc. in anomaly detection.Source: Table 5 · p. 8, §4.3 · p. 8

4.4 RQ3: Performance in Practical Implementations

• Ten Meituan video-scenario requirements covering 18 verification points saved 1.5 Person-Days per regression round; four new bugs were detected over 10 rounds.The requirements were integrated into weekly regression automation and covered video playback, live-show, merchant, and product-card checkpoints.Source: §4.4 · p. 8

• Requirements that require interaction during verification, such as repeatedly advancing videos until a hotel-related stream appears, are not currently supported.The authors say this limitation is rare and has little impact on industry applicability.Source: §5 · p. 8
• General-purpose LLM training data lacks domain knowledge about app function designs and GUI components, limiting AUITestAgent’s current capabilities.The authors plan a GUI-function dataset and retrieval-augmented generation to improve UI-testing proficiency.Source: §5 · pp. 8–9

• Unlike script-based GUI testing approaches, AUITestAgent neither relies on nor generates hard-coded scripts, which the authors say offers better generalizability and maintainability.Prior work includes AppFlow [12], CraftDroid [19], and CoSer [9].Source: §6 · p. 9
• QTypist [21] generates semantic form inputs, while GPT-Droid [22] extracts GUI information to generate human-like interactions.Source: §6 · p. 9
• AutoDroid uses LLMs to process natural-language descriptions and GUI pages, translating commands into GUI actions; unlike these tools, AUITestAgent can directly perform GUI testing on mobile apps.Experiments demonstrate that AUITestAgent’s techniques, including dynamically organizing agents, lead to the highest quality of GUI interaction generation.Source: §6 · p. 9
• VisionDroid detects non-crash GUI bugs using LLMs to identify unexpected behaviors, whereas AUITestAgent focuses on practical industry testing requirements.Source: §6 · p. 9

• AUITestAgent significantly outperforms existing methods in GUI interaction generation and recalls 90% of injected bugs with a 4.5% FPr.Source: §7 · p. 9
• Dynamically organized agents extract GUI interactions from test requirements, and multi-dimensional data extraction retrieves relevant interaction-trace data for verification.Source: §7 · p. 9
• Unseen bugs detected at Meituan demonstrate practical benefits for testing complex commercial apps and the potential to automate mobile GUI testing.Source: §7 · p. 9