Large Language Models for Mobile GUI Text Input Generation: An Empirical Study

Motivation

• 73% of 6,950 surveyed apps require text inputs to reach subsequent pages, making context-appropriate generation central to automated GUI exploration.Random strings such as “hello world” may fail to trigger transitions when pages require specific values.Source: §1 · pp. 2–3

Study Scope

• The study evaluates nine LLMs on 115 UI pages with 183 text-input components and 37 real-world text-input bugs.Source: §1 · p. 3

Main Findings

• Extracted-context and UI-hierarchy-XML prompting achieve similar average PPTRs of 71.4% and 71.0%, while screenshot-vision prompting reaches 65.1%.XML exposes complete structural information but costs substantially more tokens, whereas vision provides visual information without improving effectiveness.Source: §1 · p. 4
• Feedback-enhanced generation improves average PPTR from 69.18% to 73.78% and average bug-detection rate from 50.95% to 64.46%.Source: §1 · p. 4

Human Involvement

• Tester-modified inputs raise bug-detection rate from 36.10% for LLM-generated inputs to 100.00%, while tester-generated inputs reach 93.33%.Source: §1 · p. 4

System Integration

• Integrating LLM-generated inputs into DroidBot increases UI-page exploration by 36.06% with DFS and 26.41% with BFS.Source: §1 · p. 4

2 Background

• DFS explores the deepest available UI path before backtracking, whereas BFS covers current-page components before moving deeper.Source: §2 · p. 4

2.1 Automated Android Testing

• Automated Android testing installs apps, analyzes expected behavior, executes test cases, evaluates results, and resets the environment.Source: §2.1 · p. 5
• Android testing depends on contextual text inputs such as account names and passwords, which are difficult to generate automatically from UI-page information.Hint text in input components often describes the expected content.Source: §2.1 · p. 5

2.2 Large Language Models

• LLMs and VLMs can process natural-language or image-text inputs and generate text for software-testing tasks.Screenshots can preserve layout, widget appearance, and rendered-text cues that textual UI metadata may not fully capture.Source: §2.2 · p. 5

3 Text-Input Generation and Evaluation Workflow

• The workflow comprises objective identification, text-input generation, test-script execution, and test-result evaluation.Source: §3 · p. 5, Figure 3 · p. 6

3.1 Objective Identification

• The study uses black-box GUI-level testing, constructing prompts from displayed UI information and execution outcomes rather than source code or formal specifications.Deeper business-logic constraints therefore require complementary specification-aware techniques.Source: §3.1 · p. 6, §3.2.1 · p. 6

3.2.1 Context Extraction.

• Context extraction collects screenshots, UI-hierarchy XML, component and adjacent text, plus global page information for prompt construction.Component context includes type, hint, and resource ID; adjacent context covers labels above, below, left, and right.Source: §3.2.1 · pp. 6–7

3.2.2 Prompt Construction.

• The three prompts combine shared global, restrictive, and guiding elements with either an image, XML hierarchy, or filtered component-and-adjacent context.Filtering removes irrelevant XML elements while retaining information useful for inferring expected input content.Source: §3.2.2 · pp. 7–8

3.2.3 LLM Communication. •

• LLM communication validates responses against a JSON schema, maps extracted strings to resource IDs, executes them, and resubmits failure details and recent history as feedback.Source: §3.2.3 · pp. 8–9, Figure 4 · p. 9

3.3 Test-Script Execution

• UIAutomator scripts install and launch apps, perform GUI actions, locate components, insert generated values by resource ID, and assert page-state changes.Source: §3.3.1 · p. 9, §3.3.2 · p. 10

3.4 Test-Result Evaluation

• Evaluation waits for expected UI conditions, treats assertion success as test success, and feeds failed objectives back to the LLM for another generation attempt.Repeated execution mitigates randomness from network and device-connection instability.Source: §3.4 · p. 10

4.1 Research Questions

• The study asks how UI-context prompting, feedback, human intervention, and LLM integration affect Android text-input generation and GUI testing.The research questions cover page pass-through, bug detection, tester assessment and modification, DroidBot improvement, and practical testing insights.Source: §4.1 · p. 11

4.2 LLM Selection

• Nine LLMs were selected from leaderboard-based candidates after provider, recency, availability, and multimodal-support screening.The final set contains six proprietary and three open-weight models, while the analysis focuses on prompting, feedback, and observable overheads rather than unverifiable internal model properties.Source: §4.2 · pp. 11–12

4.3.1 Settings for the UI-Context Prompting Effectiveness Evaluation (RQ1).

• The core evaluation covers 115 apps and 115 UI pages containing 184 text-input components, comparing extracted textual context, XML hierarchy, and screenshot-based vision prompts.Each page uses one randomly selected text-input-containing UI page, and the inputs span identity, numeric, comment, query, and geography types.Source: §4.3 · p. 12, §4.3.1 · p. 13

4.3.2 Settings for the Feedback-Protocol Evaluation (RQ2).

• The feedback protocol reuses the same LLM session and incorporates extraction failures, pass-through failures, error details, expected resource IDs, schemas, and generation history into later attempts.Page pass-through stops after success or three attempts, whereas bug detection continues until a bug is revealed or 30 invalid-input attempts are reached.Source: §4.3.2 · pp. 13–14

4.3.3 Settings for Human Assessment and Modification (RQ3).

• Human evaluation uses 20 testers for input-quality assessment and three professional testers for reconstructing and modifying inputs across page-pass-through and bug-detection tasks.The controlled comparison includes LLM-generated, tester-generated, and tester-modified inputs.Source: §4.3.3 · p. 14

4.4 Evaluation Metrics

• The study defines PPTR by successful transition to a new page, BDR by bugs revealed through invalid inputs, and reports cumulative attempt-level metrics plus per-attempt token and latency overhead.It also uses Likert ratings, Kendall’s W for tester agreement, DroidBot activity and page counts, Mann-Whitney U tests, and Vargha–Delaney effect sizes.Source: §4.4.1 · pp. 14–15, §4.4.2 · p. 15, §4.4.3 · p. 15

5.1.1 Page Pass-Through (RQ1.1).

• 71.4% PPTR@3 for extracted context and 71.0% for XML exceeded screenshot vision’s 65.1%, while extracted context required only 1272.7 tokens per attempt.The settings had PPTR@1 values of 67.5%, 67.5%, and 61.4%, respectively; no particular LLM was recommended because most were statistically indistinguishable within settings.Source: §5.1.1 · pp. 16–17
• Removing global, component, adjacent, and guiding sub-prompts reduced average PPTR by 1.4, 2.3, 0.5, and 1.3 percentage points, respectively.The component prompt had the largest average effect, but no single sub-prompt dominated; multi-component dependencies caused some pages to require nearly all fields before passing.Source: §5.1.1 · p. 17

5.1.2 Bug Detection (RQ1.2).

• 51.1% BDR@30 for both extracted context and XML versus 50.8% for screenshot vision, while extracted context cost 1893.7 total tokens per attempt.All settings rose from roughly one-third of bugs at BDR@1 to approximately one-half at BDR@30, and differences across LLMs were mostly not statistically distinguishable.Source: §5.1.2 · pp. 18–19

5.2.1 Feedback for Page Pass-Through (RQ2.1).

• 73.78% average PPTR@3 with feedback versus 69.18% without feedback, a 4.61 pp gain that was statistically significant but moderate and model-dependent.Feedback mainly repaired failed attempts by encoding syntax, identifier, schema, validation, and previously failed-value constraints; 290 of 2291 successful combinations were recovered after an initial failure.Source: §5.2.1 · pp. 20–21, Table 5 · p. 20, Table 6 · p. 20

5.2.2 Feedback for Bug Detection (RQ2.2).

• 64.46% BDR@30 with feedback versus 50.95% without feedback, a statistically significant 13.51 pp gain, although token and latency costs increased substantially.Extracted-context prompting reached 68.5% after feedback, compared with 63.7% for both XML and screenshot vision; feedback used failed inputs as negative evidence for subsequent generation.Source: §5.2.2 · pp. 22–23, Figure 11 · p. 22

5.3.1 Human Assessment (RQ3.1).

• 20 testers showed substantial agreement, with Kendall’s W values from 0.76 to 0.94 across 82,080 assessment records.Valid-input means were 4.14 for extracted context, 4.19 for XML, and 3.82 for screenshot vision, whereas invalid-input means were 3.22, 3.47, and 3.22.Source: §5.3.1 · pp. 23–24
• Testers rated valid inputs highly, with 83.1%, 83.8%, and 73.8% receiving scores 4–5 for extracted context, XML, and screenshot vision, respectively.Invalid-input high-rating proportions were lower at 43.7%, 51.6%, and 47.2%, because bug detection requires diverse, field-specific values reaching relevant validation or execution paths.Source: §5.3.1 · p. 24
• Testers substantially agreed in assessment, rating valid page-pass-through inputs higher than invalid bug-detection inputs because invalid inputs required diversity and bug-triggering potential.Valid inputs mainly needed contextual plausibility, whereas invalid inputs needed to trigger specific bugs.Source: §5.3.1 · p. 25

5.3.2 Human Modification (RQ3.2).

• 93.33% bug-detection effectiveness for tester-generated inputs and 100.00% for tester-modified inputs exceeded the LLM baseline of 36.10% in sampled cases.For page pass-through, both tester-generated and tester-modified inputs reached 90.00%, compared with 59.75% for the LLM baseline.Source: §5.3.2 · pp. 26–27
• Tester improvements came from using task objectives, bug descriptions, execution outcomes, and source candidates to select, normalize, or construct inputs matching page semantics and bug mechanisms.Examples included short keywords, preserved identifiers, compatible multi-field values, exact trigger strings, locale-sensitive formats, and strict empty or whitespace-only fields.Source: §5.3.2 · pp. 26–27

5.4.1 Tool Improvement (RQ4.1).

• 47.10 UI pages under DFS and 32.52 under BFS exceeded original DroidBot averages of 34.62 and 25.72 after integrating LLM-generated text inputs.Activity exploration also increased from 4.34 to 5.86 under DFS and from 4.00 to 4.83 under BFS, because context-aware inputs better satisfy transition requirements.Source: §5.4.1 · p. 28

5.4.2 Testing Insights (RQ4.2).

• The six testing insights recommend effectiveness-cost context selection, targeted feedback, cautious model comparison, hybrid refinement, component-level diagnostics, and metrics matched to testing objectives.Feedback improves effectiveness but adds token and latency costs; PPTR and BDR do not measure full correctness, security, or business-logic coverage.Source: §5.4.2 · pp. 28–29

5.5 Threats to Validity

• The findings are empirical rather than universal because app, bug, model, baseline, metric, and human-evaluation samples have limited representativeness and practical privacy and security constraints.PPTR and BDR capture bounded transition or bug-detection outcomes, while deployment should protect personal data, authentication fields, and proprietary logic.Source: §5.5 · pp. 29–30

6.1 Automated GUI Testing for Mobile Apps

• Android GUI testing methods often struggle with contextual text inputs, leaving UI exploration incomplete despite advances in automated action and string generation.Prior work includes random action testing, context-aware string generation, GPT-3-based input generation, and LLM-driven GUI testing systems.Source: §6.1 · p. 30
• Recent LLM-based GUI systems extend beyond input generation to test-case migration, end-to-end task automation, accessibility testing, and multimodal non-crash bug detection.These approaches translate high-level intents or migrated test cases into executable GUI actions and report over 70% end-to-end task completion for AutoDroid.Source: §6.1 · pp. 30–31

6.2 LLMs for Software Testing

• LLMs have been applied across software-testing tasks including fault localization, unit-test generation, test-oracle generation, test-suite minimization, bug analysis, and mutation testing.The related work positions text-input generation within a broader expansion of LLM use in software testing.Source: §6.2 · p. 31
• Existing studies show promise for LLM-guided web-app input generation and fuzzing, but Android GUI text-input effectiveness across different LLMs remains empirically unstudied.The literature also lacks practical advice on how testers should best use LLMs for this task.Source: §6.2 · p. 31

7 Conclusions and Future Work

• The study evaluated nine recent LLMs across 115 Google Play UI pages, 183 text-input components, and 37 real-world text-input-related bugs.It compared UI-context prompting, feedback protocols, and human-refinement settings.Source: §7 · p. 31
• Extracted-context and UI-hierarchy-XML prompting had similar page-pass-through effectiveness, while extracted context was substantially cheaper and screenshot vision was not automatically better.The conclusion distinguishes effectiveness from prompting cost and visual information.Source: §7 · p. 31
• Feedback improved page pass-through and especially bug-detection effectiveness, but introduced substantial token and latency overheads.The benefit therefore comes with an explicit efficiency trade-off.Source: §7 · p. 31
• Human assessment and modification enabled testers to identify and repair practical weaknesses in selected LLM-generated inputs.This supports a human-refinement role rather than treating generated inputs as final.Source: §7 · p. 31
• Future work targets task-relevant context extraction, multimodal signals for difficult interfaces, broader LLM and component coverage, and stronger app-specific invalid-input generation.The proposed directions include custom views, WebViews, image-only labels, dynamically rendered interfaces, Spinners, and ListViews.Source: §7 · pp. 31–32

A An Example of Context Extraction

• Figure A.1 illustrates extraction of component and adjacent context from an ImgurViewer interface.The example provides a concrete view of the appendix’s context-extraction procedure.Source: A An Example of Context Extraction · p. 37, Figure A.1 · p. 37
• Component context includes the text-input type, displayed hint, and resource ID used to match and locate the component on the UI page.UIAutomator extracts these attributes directly from the text-input component.Source: A An Example of Context Extraction · p. 37
• Adjacent context consists of nearby text labels classified into top, bottom, left, and right positions around the text-input component.Relative positions are determined from candidate TextView boundaries and centers; the example has two top labels, one bottom label, and none on the left or right.Source: A An Example of Context Extraction · p. 37
• The procedure selects the nearest TextView for each spatial position, then extracts its text label as adjacent context.Nearest-neighbor selection uses shortest distance together with the stated positional constraints.Source: A An Example of Context Extraction · pp. 37–38

B Details of UI Pages

• Table B.1 covers 115 experimental UI pages and records each app’s identifier, download count, text-input component count, and transition-triggering UI action.The table’s total is 115 pages, with 183 text-input components across the listed applications.Source: B Details of UI Pages · pp. 38, 40, Table B.1 · p. 38
• Applications with d≥2 were used for the multiple-text-input-component ablation study.The appendix also marks 20 randomly selected applications that testers manually modified.Source: B Details of UI Pages · p. 38
• The listed transition actions include clicking labeled buttons, clicking coordinates, and typing an Enter key.These actions represent the UI events used to trigger movement to subsequent pages.Source: B Details of UI Pages · pp. 39–40

• 37 buggy UI pages are cataloged with app names, descriptions, categories, and issue-report links.Source: C Details of Bugs · p. 40, Table C.1 · p. 40
• Special characters and malformed strings produce failures across fields such as theme colors, markup, selectors, and search queries.Source: C Details of Bugs · p. 41
• The catalog includes crashes triggered by empty or missing required inputs, whitespace, and empty fields.Source: C Details of Bugs · p. 41
• Malformed or constrained inputs cause crashes, unresponsiveness, validation problems, or other incorrect behavior.Source: C Details of Bugs · p. 41
• Specific-input bugs include numbered-list content, long text, multiline editing, and malformed departure or arrival fields.Source: C Details of Bugs · p. 41

D Details of Apps for Tool Improvement Experiment

• 30 apps used in tool-improvement experiments are listed with application identifiers and download counts collected on July 22, 2025.Source: D Details of Apps for Tool Improvement… · p. 42, Table D.1 · p. 42

E Supplementary Ablation Analysis

• The supplementary ablation analysis examines prompt sub-components and multiple text-input components to interpret RQ1 page-pass-through results.Source: E Supplementary Ablation Analysis · p. 42

E.1 Sub-Prompt Ablation Settings

• Prompt ablations remove one global, component, adjacent, or guiding sub-prompt at a time while retaining the restrictive sub-prompt.Preliminary attempts without the restrictive sub-prompt prevented reliable extraction and parsing of LLM outputs.Source: §E.1 · p. 42

E.2 Multi-Component Ablation Settings

• 584 test scenarios cover 32 of 115 UI pages containing multiple text-input components, varying filled-component combinations from τ=0 to τ=d.For each page with d components, all combinations of actively filled components were generated while the others remained empty.Source: §E.2 · p. 42, Table E.2 · p. 43
• Criticality Ψ(c_i) measures a component’s average contribution to PPTR across combinations containing that component.The comparison subtracts pass-through with the component removed from pass-through with all components in the combination filled, weighted by combination size.Source: §E.2 · p. 43
• Positive Ψ(c_i) indicates that including a component improves PPTR, whereas negative values indicate harm.The weighting factor ω(τ_s)=τ_s/d gives greater emphasis to combinations in which most components are filled.Source: §E.2 · p. 43
• Criticality scores are comparable within a page but not across pages, so min-max normalization maps them to [0.0, 1.0].The normalized score identifies components with greater relative criticality among components on the same page.Source: §E.2 · p. 43

E.3 Prompt Ablation Results

• Average PPTR decreases by 1.4, 2.3, 0.5, and 1.3 percentage points when global, component, adjacent, and guiding sub-prompts are removed.Removing the component sub-prompt has the largest average impact, while removing the adjacent sub-prompt has the smallest.Source: §E.3 · p. 44
• Most ablations cause small individual-LLM changes, and statistical effect sizes are generally small.The reported comparisons show most A12 values between 0.49 and 0.52.Source: §E.3 · p. 44
• The component prompt is the most sensitive prompt element because it directly describes the target text-input component.Its removal produced the largest average decrease and a statistically significant decrease for Grok.Source: §E.3 · p. 44
• No single sub-prompt dominates effectiveness; the complete prompt operates through complementary contextual and guiding information.Source: §E.3 · p. 44

Criticality patterns

• 64.52% of the 31 analyzed pages assign equal relative criticality to all components, while 35.48% contain two criticality levels.The normalized criticality vectors distinguish pages whose components contribute uniformly from pages with unequal influence.Source: §E.4 · p. 44

PPTR trends

• PPTR patterns include stable, gradual-increase, and delayed-increase pages as more components are filled.Some pages remain at 0% or 100%, some improve progressively, and others improve only after nearly all components are filled.Source: §E.4 · p. 44

Criticality patterns

• Pages with two criticality levels can produce different PPTRs at the same filling count because higher-criticality components matter more.For example, some two-component pages reach 50.0% when one component is filled and 100.0% when both are filled.Source: §E.4 · p. 45

Multi-component ablation

• Prioritizing more critical text-input components increases PPTR, whereas filling less critical components first reduces it.The ablation summary explicitly links higher PPTR to entering more critical values earlier.Source: §E.4 · p. 45

Feedback-enhanced page pass-through

• The detailed results compare page-pass-through trends and overhead across extracted context, UI-hierarchy XML, and screenshot vision.Tables F.5 and F.6 and Figures F.2 and F.3 cover feedback-enhanced page pass-through, while Tables F.9 and F.10 provide the corresponding tabulated feedback results.Source: Table F.5 · p. 46, Figure F.2 · p. 46, Table F.6 · p. 47

Bug revealing

• The detailed results separately report bug-reveal trends and token/time overhead for the three prompt settings under feedback-enhanced and no-feedback conditions.Tables F.7, F.8, F.11, and F.12 and Figures F.4, F.5, F.7, and F.7 organize these comparisons.Source: Table F.7 · p. 48, Figure F.4 · p. 48, Figure F.5 · p. 48

Prompt-setting overhead

• For PPTR, Table F.13 reports average effectiveness and token/time overhead per attempt across prompt settings, enabling comparison of these measures across the evaluated settings.Source: Table F.13 · p. 53

Cross-setting comparison

• Average effectiveness and token/time overhead per attempt are summarized across prompt settings for both PPTR and BDR.Tables F.13 and F.14 provide cross-setting summaries, while Figures F.6 and F.7 compare no-feedback with feedback-enhanced overhead.Source: Table F.13 · p. 53, Table F.14 · p. 53, Figure F.6 · p. 54