Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps

• 61.3% of over 100,000 GitHub issue reports describe non-crash functional bugs, and 47.7% of these are tagged “important”.Source: §I · p. 1
• Visual-cue bugs can require understanding page semantics and operational logic across transitions, such as incorrect sorting or missing wishlist updates.Source: §I · p. 1
• Trident combines Explorer, Monitor, and Detector agents to navigate functionality, supervise testing, and identify logical-transition bugs through MLLMs.Source: §I · pp. 1–2
• Its design addresses visual-text alignment, functionality-oriented exploration, and test-oracle inference for non-crash bugs.Source: §I · p. 2
• Trident achieves 50%-72% precision and 42%-52% recall, with more than 14%-112% precision and 108%-147% recall boosts over the best baseline on 590 bugs.Source: §I · p. 2
• In 187 Google Play apps, Trident identifies 43 new bugs, including 31 fixed and 12 confirmed by developers.Source: §I · p. 2

• Trident uses Explorer, Monitor, and Detector agents to navigate apps, record and supervise exploration, and analyze logical transitions for functional bugs.Source: §II · pp. 2–3
• The approach must make MLLM-based detection effective despite difficulties in visual-text alignment, functionality-oriented exploration, and oracle inference.Source: §II · p. 3

Challenge 1: Aligning visual and text for MLLM input

• Screenshot annotation and text-visual alignment help the MLLM understand actionable GUI elements and guide subsequent exploration.Source: §II · p. 3

Challenge 2: Functionality-oriented exploration

• Functionality abstraction converts detailed action and page-transition sequences into histories that reduce token use and focus exploration on functional behavior.Source: §II · p. 3

Challenge 3: Inferring test oracle

• The Monitor triggers the Detector at the end of functionality exploration, while functionality-aware Chain-of-Thought first infers oracles before detecting bugs.Source: §II · p. 3

A. Vision-driven Explorer Agent

• The Explorer visually inspects GUI pages and conducts exploration, while aligning visual and textual information to strengthen page-semantic understanding.Source: A. Vision-driven Explorer Agent · p. 3

1) Alignment between Text and Image:

• Text inputs include app details, page context, and widget attributes such as text, bounds, clickability, and scrolling properties.Source: 1) Alignment between Text and Image · p. 3
• Screenshot annotations mark actionable widgets with color-coded boxes for click, input, long-click, check, and scroll actions, plus numbered labels and a legend.Source: 1) Alignment between Text and Image · pp. 3–4
• Each annotated widget is associated with extracted GUI text, and mismatches between requested widget IDs and text trigger retries capped at three attempts before fallback.Source: 1) Alignment between Text and Image · p. 4

2) Explorer Prompt:

• The Explorer Prompt queries the MLLM about interactive widgets and required actions using GUI information, screenshots, testing history, and detector feedback.Source: 2) Explorer Prompt · p. 4

B. Process-aware Monitor Agent

• The Monitor Agent oversees testing, abstracts functionality history, and triggers the Detector when a new functionality begins.Source: B. Process-aware Monitor Agent · p. 4
• The Monitor continuously records GUI pages, view hierarchies, screenshots, and related actions while testing progresses.Source: B. Process-aware Monitor Agent · p. 4
• The function-based history is illustrated as an example of how testing progress is represented for follow-up exploration and detection.Source: Figure 5 · p. 4

1) Function-based Testing History Construction:

• Function-based history organizes testing progress by functionality, visit count, and status to reduce redundant exploration and expose untouched functionality.Source: 1) Function-based Testing History… · p. 4
• The image history view links annotated screenshots and page descriptions to preserve granular alignment between visual and textual testing records.Source: 1) Function-based Testing History… · p. 4

2) Monitor Prompt:

• The monitor prompt combines current-function records and whole-history context in text and image form to query page descriptions, tested functions, and status.Source: 2) Monitor Prompt · pp. 4–5
• The monitor prompt is illustrated as an example input for querying the MLLM about the current GUI testing state.Source: Figure 6 · p. 5

C. Function-aware Detector Agent

• The Detector examines GUI transitions and interactions, identifies logical inconsistencies and unexplored bug-triggering operations, and feeds findings back to exploration.Source: C. Function-aware Detector Agent · p. 5

1) Functionality-driven Chain of Thought (COT):

• Functionality-driven COT guides sequential analysis of the tested functionality, reducing hallucination risk during bug detection.Source: 1) Functionality-driven Chain of… · p. 5
• The MLLM compares the identified functionality, expected path, and actual path to evaluate whether execution follows expected behavior.Source: 1) Functionality-driven Chain of… · p. 5
• Annotated screenshots label actions such as clicks and inputs, helping the model correlate sequential text descriptions with GUI elements.Source: 1) Functionality-driven Chain of… · p. 5

2) Enriching Detector Prompt with Examples:

• The detector prompt uses in-context examples of non-crash logical bugs to clarify the domain-specific detection task.Source: 2) Enriching Detector Prompt with… · p. 5
• The example dataset records bug descriptions, screenshots, and natural-language reproduction paths, with 200 high-quality instances selected initially.Source: 2) Enriching Detector Prompt with… · pp. 5–6
• Retrieval selects the top-K most similar examples using activity-name Word2Vec embeddings and cosine similarity, with K set to 5.Source: 2) Enriching Detector Prompt with… · p. 6

3) Detector Prompt:

• The detector prompt combines bug examples, functionality-driven COT, and real exploration screenshots.Source: 3) Detector Prompt · p. 6
• It queries the MLLM about bug detection, possible bug paths, and exception paths from the integrated exploration context.Source: 3) Detector Prompt · p. 6

D. Implementation

• GPT-4 Vision processes textual and visual information during testing.Source: D. Implementation · p. 6
• UIAutomator supplies view-hierarchy information and extracts text from input widgets for the model’s GUI-page context.Source: D. Implementation · p. 6
• Multiple testing screenshots are combined into one large image so the API can process sequence context cohesively.Source: D. Implementation · p. 6
• Testing histories use concise functionality names and summaries, and earlier records are deleted when token limits are exceeded.Source: D. Implementation · p. 6
• Explorer and Monitor prompts are merged into one request to query exploration state and tested functionalities simultaneously.Source: D. Implementation · p. 6

III. EXPERIMENT DESIGN

• RQ1 compares Trident’s coverage and bug detection with commonly used and state-of-the-art baselines.Source: §III · p. 6
• RQ2 uses ablation experiments to measure how individual Trident modules affect coverage and bug detection performance.Source: §III · p. 6
• RQ3 evaluates usefulness by detecting unseen non-crash functional bugs in real-world apps and reporting them to development teams.Source: §III · p. 6, A. Datasets · p. 7

A. Datasets

• The evaluation uses baseline, GitHub, and injection datasets covering 131, 387, and 72 non-crash bugs, respectively.Source: A. Datasets · pp. 6–7
• The GitHub dataset targets recent apps and broader bug categories, retaining 83 apps with 387 bugs after manual filtering.Source: A. Datasets · p. 6
• The injection dataset reduces potential MLLM data leakage by modifying source code, retaining 33 apps with 72 injected bugs.Source: A. Datasets · pp. 6–7
• Table I reports coverage and bug-detection performance across the baseline, GitHub, and injection datasets.Source: Table I · p. 7
• RQ3 evaluates 187 popular apps selected by app type and download number using the same configurations as the preceding experiments.Source: A. Datasets · p. 7

B. Baselines

• The study compares Trident with 12 GUI-testing techniques, including display-issue, logical-bug, static-analysis, and MLLM-based methods.The baselines are paired with automated exploration tools where required and include AppAgent and GPT-4.Source: B. Baselines · pp. 7–8
• The baseline suite covers specialized display and logical-bug detectors as well as general-purpose automated GUI-testing approaches.This combination supports comparison across visual, functional, static-analysis, and multimodal testing strategies.Source: B. Baselines · p. 7

C. Experimental Setup

• The evaluation runs each tool three times for 50 minutes per app on Google Android emulators, averaging results across runs.Separate accounts and reset account data are used for bugs requiring login.Source: C. Experimental Setup · p. 8

D. Evaluation Metrics

• Coverage is measured by activity and code coverage, while bug detection reports detected bugs, true bugs, precision, and recall.Activities are defined from AndroidManifest.xml, and the evaluation uses the known complete bug set to calculate precision and recall.Source: D. Evaluation Metrics · p. 8

A. Coverage and Bug Detection Performance (RQ1)

• Trident achieves 21%-27% higher activity coverage and 28%-32% higher code coverage than the best baseline, GPTDroid, across three datasets.The reported coverage ranges are 57%-65% for activities and 55%-62% for code.Source: A. Coverage and Bug Detection… · p. 8
• Trident reaches 50%-72% average precision and 42%-65% average recall, exceeding the best baseline by 14%-112% and 108%-147%, respectively.Although some baselines obtain higher precision, Trident detects more bugs overall.Source: A. Coverage and Bug Detection… · p. 8
• Trident’s advantages combine screenshot-aware GUI understanding, function-aware exploration using page semantics and history, and sequence-level bug reasoning.The approach therefore detects functional abnormalities that baselines may miss because they target specific bug types or lack business-logic understanding.Source: A. Coverage and Bug Detection… · p. 8
• Examples include duplicate income records and incorrect navigation from Notifications back to the main page, which Trident detects through test-path semantics.These bugs were not detected by the baseline methods shown in the examples.Source: A. Coverage and Bug Detection… · p. 8, Figure 8 · p. 9

B. Ablation Study (RQ2)

• Removing any sub-module causes a noticeable performance decline, while removing bug examples produces the largest detection loss.Without bug examples, precision falls 54% (0.33 vs. 0.72) and recall falls 92% (0.04 vs. 0.52).Source: B. Ablation Study (RQ2) · p. 9, 1) Contribution of Sub-modules · p. 9

1) Contribution of Sub-modules:

• Removing screenshot annotation, testing history, or the image legend reduces activity coverage by 37% (0.41 vs. 0.65), 35% (0.42 vs. 0.65), or 40% (0.39 vs. 0.65).The results associate visual-text alignment with GUI understanding and testing history with reaching uncovered areas.Source: 1) Contribution of Sub-modules · p. 9
• Removing Function COT decreases precision by 56% (0.37 vs. 0.72) and recall by 88% (0.06 vs. 0.52), because functionality-path inference supports bug detection.Function COT helps the MLLM infer possible paths from functionality descriptions and judge the tested path.Source: 1) Contribution of Sub-modules · p. 9

2) Influence of Different Number of Examples:

• Bug-detection performance peaks with four examples and gradually declines when additional examples introduce excessive information or noise.Both too few and too many examples can damage performance.Source: 2) Influence of Different Number of… · p. 9, Figure 9 · p. 9

C. Usefulness Evaluation (RQ3)

• Among 187 Google Play apps, Trident detects 43 previously undiscovered bugs in 42 apps, with 31 fixed and 12 confirmed by developers.Only nine of these new bugs are also detected by Genie or Odin, and none are rejected by developers.Source: C. Usefulness Evaluation (RQ3) · pp. 9–10

V. DISCUSSION

• For platforms where text extraction is unreliable, Trident can remove textual GUI information while retaining its other exploration and bug-detection modules.The paper presents this adaptation as supporting use beyond Android.Source: §V · p. 10
• In small experiments on 10 iOS and 10 Harmony OS apps, Trident obtains 0.60 bug-detection precision on both platforms and code coverage of 0.47 and 0.57, respectively.The authors describe these results as comparable to Android performance and note that broader experiments remain future work.Source: §V · p. 10, Table IV · p. 10

A. Automated GUI Testing

• Conventional automated GUI testing can lose visual information when converting screenshots to text or relying on incomplete view hierarchies, whereas Trident aligns screenshots with textual GUI information.Dynamic rendering can also cause widgets and image attributes to be missed, reducing coverage and missed operations.Source: A. Automated GUI Testing · p. 10

B. Non-crash Functional Bug Detection

• Existing methods use tester-provided oracles, differential comparisons, category-specific oracle generation, or labeled UI-bug data to detect non-crash bugs.These approaches include comparing views across devices or versions, targeting language or system-setting issues, and training models for display bugs.Source: B. Non-crash Functional Bug Detection · p. 10
• Their main limitations are single-category coverage, reliance on manually labeled oracle data, and difficulty handling complex business scenarios and app functionality.Source: B. Non-crash Functional Bug Detection · p. 10

VII. CONCLUSION

• Trident uses a vision-driven, multi-agent collaborative approach comprising Explorer, Monitor, and Detector agents for non-crash functional bug detection.The paper presents this approach as an application of MLLM capabilities to software engineering.Source: §VII · p. 10
• Trident is positioned as a stepping stone for exploring MLLM applications in software engineering, offering insights and guidance for future work.Source: §VII · p. 10
• Future work will improve performance and extend the approach beyond Android apps to web, desktop, and automotive operating-system testing.Source: §VII · p. 10