PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation

Kristian Kolthoff, Felix Kretzer, Lennart Fiebig, Christian Bartelt, Alexander Maedche, Simone Paolo Ponzetto

arXiv:2412.11328v1cs.SE

TL;DR

GUI prototyping is valuable but high-fidelity creation is costly, motivating less resource-intensive generation methods. The paper evaluates zero-shot RAGG, PDGG, and SCGG approaches, finding that SCGG is generally the most effective while also identifying retrieval, token, functionality, and validity boundaries.

  • Problem

    High-fidelity GUI prototypes are costly to create, while prior automatic-generation research largely relies on resource-intensive training or fine-tuning and focuses on low-fidelity outputs.

    Source: §I · p. 1

  • Method

    The paper evaluates zero-shot high-fidelity GUI generation using RAGG with LLM-based retrieval re-ranking and filtering, PDGG, and SCGG.

    Source: 2) LLM-Based GUI Re-Ranking · p. 4

  • Results

    SCGG mostly significantly outperforms the other methods, while LLM-based GUI re-ranking significantly outperforms state-of-the-art GUI re-ranking models.

    Source: A. RQ1: Effectiveness of LLM-Based GUI… · p. 7, §IX · p. 11

  • Takeaways & Limitations

    Zero-shot prompting can enhance GUI prototypes, with SCGG showing the highest effectiveness and content generation significantly improving prototype effectiveness.

    Source: §IX · p. 11

    RAGG can retain false positives and depends on relevant retrieval results, while SCGG consumes many tokens and some generated components remain non-interactive.

    Source: §VI · pp. 9–10

Abstract

from arXiv · show

Graphical user interface (GUI) prototyping represents an essential activity in the development of interactive systems, which are omnipresent today. GUI prototypes facilitate elicitation of requirements and help to test, evaluate, and validate ideas with users and the development team. However, creating GUI prototypes is a time-consuming process and often requires extensive resources. While existing research for automatic GUI generation focused largely on resource-intensive training and fine-tuning of LLMs, mainly for low-fidelity GUIs, we investigate the potential and effectiveness of Zero-Shot (ZS) prompting for high-fidelity GUI generation. We propose a Retrieval-Augmented GUI Generation (RAGG) approach, integrated with an LLM-based GUI retrieval re-ranking and filtering mechanism based on a large-scale GUI repository. In addition, we adapt Prompt Decomposition (PDGG) and Self-Critique (SCGG) for GUI generation. To evaluate the effectiveness of the proposed ZS prompting approaches for GUI generation, we extensively evaluated the accuracy and subjective satisfaction of the generated GUI prototypes. Our evaluation, which encompasses over 3,000 GUI annotations from over 100 crowd-workers with UI/UX experience, shows that SCGG, in contrast to PDGG and RAGG, can lead to more effective GUI generation, and provides valuable insights into the defects that are produced by the LLMs in the generated GUI prototypes.

I. INTRODUCTION

GUI prototypes support requirements elicitation and early design evaluation, but high-fidelity prototyping is costly and time-consuming. This paper investigates zero-shot prompting for high-fidelity GUI generation and evaluates RAGG, PDGG, and SCGG.

I. INTRODUCTION

  • High-fidelity GUI prototyping supports stakeholder communication, specific feedback, and evaluation of alternative designs, but requires substantial time and resources.Frequent requirement changes further necessitate repeated prototype redesign.Source: §I · p. 1
  • Existing automatic GUI-generation methods rely on resource-intensive training or fine-tuning and generally produce low-fidelity DSL layouts, while high-fidelity alternatives have input and functionality limitations.Recent zero-shot work focused on verifying requirements implementation rather than generating complete GUI prototypes from descriptions.Source: §I · p. 1
  • The paper explores zero-shot generation of high-fidelity HTML/CSS GUI prototypes from short natural-language requirements because training and fine-tuning are costly and few-shot prompting faces large context requirements.The proposed approaches are RAGG, PDGG, and SCGG.Source: §I · pp. 1–2
  • The evaluation collected over 3,000 GUI annotations from over 100 crowd-workers with UI/UX experience to assess generated-prototype accuracy and subjective satisfaction.The study also provides insights into defects produced by LLM-generated GUIs.Source: §I · p. 2
  • The paper contributes an LLM-based GUI re-ranking mechanism, RAGG, and adaptations of Prompt Decomposition and Self-Critique for zero-shot GUI generation.The re-ranking mechanism is reported to significantly outperform state-of-the-art approaches.Source: §I · p. 2

II. BACKGROUND

  • The work builds on research concerning natural-language GUI retrieval and large language models, where prompting approaches had not yet been investigated for GUI generation.The background motivates applying established prompting ideas to this generation task.Source: §I · p. 1, §II · p. 2

A. NL-based GUI Retrieval

The methodology uses zero-shot instruction and chain-of-thought baselines alongside retrieval-oriented GUI generation. Natural-language retrieval is based on rapid semantic matching over GUI descriptions and screenshots.

A. NL-based GUI Retrieval

  • The retrieval approach uses GUI screenshots and hierarchy data from the Rico repository, matching natural-language requirements to high-level Screen2Words descriptions with embedding models.BM25 or SentenceBERT cosine similarity can provide rapid retrieval over the repository.Source: A. NL-based GUI Retrieval · p. 2

B. Large Language Models (LLMs)

  • The paper frames GUI generation as an application of prompting approaches such as zero-shot instruction, chain-of-thought, prompt decomposition, self-critique, and retrieval-augmented generation.These approaches had shown promise but had not been investigated for GUI generation in the cited background.Source: B. Large Language Models (LLMs) · p. 2

III. METHODOLOGY

  • The methodology section benchmarks more comprehensive approaches against zero-shot instruction and zero-shot chain-of-thought baselines.The overall method overview includes PDGG, RAGG, and SCGG.Source: §III · p. 2, Figure 1 · p. 3

A. Baselines: ZS-Instruction and ZS-CoT

  • The baseline prompts instruct an LLM to generate mobile GUI pages directly in HTML/CSS from brief textual descriptions, with a second baseline requesting step-by-step reasoning.HTML/CSS was selected because it supports GUI interaction and is commonly represented in LLM pretraining data.Source: A. Baselines: ZS-Instruction and ZS-CoT · p. 2

B. Prompt Decomposition for GUI Generation (PDGG)

PDGG decomposes GUI generation into staged reasoning, while RAGG retrieves and filters relevant GUI examples before generation. Both approaches use intermediate representations or repository knowledge to support HTML/CSS synthesis.

B. Prompt Decomposition for GUI Generation (PDGG)

  • PDGG expands high-level requirements into feature descriptions, implementation ideas, and an overall layout before translating the specification into HTML/CSS.Each stage makes the GUI-generation problem more explicit for the final generator.Source: B. Prompt Decomposition for GUI… · p. 3
  • PDGG also includes a single-prompt variant that combines predefined domain-specific reasoning steps into one zero-shot chain-of-thought prompt.This variant complements the separated multi-prompt decomposition pipeline.Source: B. Prompt Decomposition for GUI… · p. 3

C. Retrieval-Augmented GUI Generation (RAGG)

  • RAGG integrates LLM generation with a large GUI repository to provide prototyping knowledge that may not be implicitly stored in the pretrained model.Retrieved GUI screens are incorporated into prompting for generation.Source: C. Retrieval-Augmented GUI Generation… · p. 3

1) GUI Repository:

  • Rico supplies over 72k mobile-app GUIs across 27 domains, while Screen2Words provides over 112k descriptions for over 22k Rico GUIs.The repository filtering additionally removes GUIs with overlaying menus using a CNN classifier with precision .98 and recall .70.Source: 1) GUI Repository · p. 3

2) LLM-Based GUI Re-Ranking:

  • RAGG retrieves top-n GUIs with SentenceBERT, filters them using zero-shot multimodal relevance judgments, and then assigns finer-grained relevance scores from 1 to 10.Critical filtering favors non-relevant decisions under ambiguity to reduce false positives in later generation.Source: 2) LLM-Based GUI Re-Ranking · p. 4

3) Prompting Approaches:

  • One RAGG generation path extracts and aggregates features, designs, and layouts from top-k GUIs before HTML/CSS generation.Semantically similar features are aggregated and ranked by frequency.Source: 3) Prompting Approaches · p. 4
  • A second RAGG path directly provides top-k GUI screenshots as inspirations for features, design, and layout during HTML/CSS generation.The two paths differ in whether repository information is first converted into aggregated textual collections.Source: 3) Prompting Approaches · p. 4

D. Self-Critique Looping for GUI Generation (SCGG)

SCGG iteratively critiques and revises GUI prototypes, while a separate content-generation pipeline adds realistic data and generated images. These components target both prototype quality and visual realism.

D. Self-Critique Looping for GUI Generation (SCGG)

  • SCGG uses a feedback LLM to critique a prototype against the requirements and a prototyping LLM to produce a revised HTML/CSS prototype.The critique addresses missing features, feature implementation, design, and layout while avoiding code.Source: D. Self-Critique Looping for GUI… · p. 4
  • The SCGG loop integrates each critique into the next prototype, using the natural-language requirements, prior prototype, and feedback as inputs.The revised model response is instructed to contain HTML/CSS without explanation.Source: D. Self-Critique Looping for GUI… · p. 4
  • SCGG requires an initial HTML/CSS prototype and therefore starts from the zero-shot instruction prototype for further refinement over k iterations.The initial prototype is needed to provide the first critique.Source: D. Self-Critique Looping for GUI… · p. 4

E. LLM-based GUI Content Generation

  • The content-generation pipeline addresses generic or insufficient data by extending base HTML/CSS with realistic content and adding identifiers to image elements.Examples include additional list items and related products.Source: E. LLM-based GUI Content Generation · p. 4
  • Fig. 2 presents the overview of the LLM-based content-generation pipeline.The figure is associated with the realism-enhancement process described in the surrounding text.Source: Figure 2 · p. 5
  • The pipeline extracts image descriptions from the HTML, generates images with DALL-E-3, uploads them, and inserts the resulting URLs by matching image identifiers.An LLM is used for image-description extraction to exploit contextual information.Source: E. LLM-based GUI Content Generation · p. 5

IV. EVALUATION

The evaluation investigates retrieval, prompting, and GUI-generation effectiveness through five research questions and a staged dataset-creation procedure. It constructs a curated collection of Rico GUIs and high-quality natural-language descriptions for evaluation.

IV. EVALUATION

  • Five research questions examine GUI re-ranking and filtering, prompting approaches, retrieval-example counts, self-critique loops, and content generation.Source: §IV · p. 5

A. GUI Descriptions Dataset

1) Rico GUI Sampling:

  • 146 Rico GUIs were admitted after filtering out screens with fewer than seven component types and obtaining 1,050 ratings across six quality categories.Source: 1) Rico GUI Sampling · p. 5

2) Collecting GUI Prototype Descriptions:

3) Evaluating GUI Prototype Descriptions:

The description-evaluation stage filtered and selected natural-language GUI descriptions, then defined retrieval datasets and metrics for assessing ranking approaches. GPT-4o was evaluated at three temperature settings on the gold standard.

3) Evaluating GUI Prototype Descriptions:

  • Crowd-workers with UI/UX experience rated descriptions for understandability, app purpose, and related quality criteria using 9-point Likert scales.Source: 3) Evaluating GUI Prototype Descriptions · p. 5
  • 574 descriptions for 146 GUIs were rated, and the highest-rated description per GUI was selected; ten formed validation data and 50 formed the final test set.The test set was restricted to 50 examples because generated-GUI annotation was expensive and labor-intensive.Source: 3) Evaluating GUI Prototype Descriptions · p. 5

1) Datasets:

  • Retrieval evaluation used a public gold standard and a 50-description test set, with three annotators providing majority-vote binary relevance labels for retrieved GUIs.The relevance annotations achieved Fleiss Kappa = .403.Source: 1) Datasets · p. 6

2) Evaluation Metrics:

  • Ranking was evaluated with AP, MRR, Precision@k, HITS@k, and NDCG@k, while the second dataset used Precision, Recall, and F1-Measure.Source: 2) Evaluation Metrics · p. 6
  • GPT-4o, a multimodal model with a 128k-token context length, was evaluated on the gold standard at temperatures .25, .50, and .75.The model was accessed in July 2024.Source: 3) Model Setup · p. 6

3) Model Setup:

Generated HTML/CSS GUI prototypes were evaluated by screened Prolific participants with UI/UX experience using subjective satisfaction measures. The experiments compared baselines, prompt decomposition, RAGG, and SCGG under a consistent GPT-4o configuration.

1) Annotation of Generated GUI Prototypes:

  • After screening 126 submissions, 101 participants remained, with reported experience in creating and evaluating visual designs.Source: 1) Annotation of Generated GUI… · p. 6
  • The study produced interactive HTML/CSS GUIs for 50 descriptions, with three annotations per GUI, yielding 3,000 annotations.Participants saw descriptions, eight Likert items, and interactive prototypes, without receiving multiple GUIs from the same description.Source: 1) Annotation of Generated GUI… · p. 6

2) Evaluation Metrics:

3) Model Setup:

  • At temperature t = .50, experiments compared baselines, PDGG, RAGG, and SCGG, plus prompt and encoding variants.Source: 3) Model Setup · p. 6

D. RQ3: Effectiveness of Example Number on RAGG

The evaluation varied the number of retrieved examples, self-critique loops, and GUI content-generation settings, while reporting results and discussion in a subsequent section. The supplied passages specify the experimental designs but not their numerical outcomes.

D. RQ3: Effectiveness of Example Number on RAGG

  • RAGG was evaluated with k = 1, 3, 5, and 7 retrieved GUIs using direct encoding and a consistent GPT-4o configuration.The maximum was limited because only 30% of top-20 retrieval sets contained at least seven relevant screens.Source: D. RQ3: Effectiveness of Example… · p. 6

E. RQ4: Effectiveness of Number of Loops in SCGG

F. RQ5: Effectiveness of GUI Content Generation

V. RESULTS & DISCUSSION

  • The results section presents findings and a more detailed discussion for the previously posed research questions.Source: §V · p. 7

A. RQ1: Effectiveness of LLM-Based GUI Re-ranking

LLM-based re-ranking significantly outperforms prior GUI re-ranking models, while SCGG is the strongest prompting approach across most GUI-generation aspects and larger RAGG example sets improve effectiveness.

A. RQ1: Effectiveness of LLM-Based GUI Re-ranking

  • Average precision reached .818 versus .501 for the best BERT-LTR model, while LLM-based re-ranking significantly outperformed prior models across all metrics.On the actual test dataset, binary relevance filtering achieved P = .757, R = .814 and F1 = .784.Source: A. RQ1: Effectiveness of LLM-Based GUI… · p. 7

B. RQ2: Effectiveness of ZS Prompting for GUI Generation

  • The evaluation compared baseline, PDGG, RAGG, and SCGG using crowd-worker ratings across feature, organization, visual, error, satisfaction, and app-completeness dimensions.Fig. 3 distinguishes models with GUI content generation using hatched bars and marks means with triangles.Source: B. RQ2: Effectiveness of ZS Prompting… · p. 7, Figure 3 · p. 8
  • SCGG significantly outperformed ZS, ZS-CoT and PDGG across most evaluated GUI-generation aspects, including function fulfillment, visual appeal, errors, and overall satisfaction.RAGG with more than five examples also improved Feature Width and Complete App ratings over baseline approaches.Source: B. RQ2: Effectiveness of ZS Prompting… · pp. 7–8

C. RQ3: Effectiveness of Example Number on RAGG

  • RAGG with seven examples significantly outperformed smaller settings k=1 and k=3 for GUI-generation effectiveness.Improvements from k=1 to k=5 were concentrated on feature-related items, while k=1 to k=7 improved all evaluated items.Source: C. RQ3: Effectiveness of Example… · p. 8

D. RQ4: Effectiveness of Number of Loops in SCGG

Additional SCGG loops mainly improve feature-related and app-completeness aspects, while automatic GUI content generation improves visual and overall prototype quality across most prompting approaches.

D. RQ4: Effectiveness of Number of Loops in SCGG

  • After the initial SCGG iteration, later loops mainly added new features and extended prototypes toward app completeness, including headers, navigation, and footers.The authors suggest dynamically focusing feedback on underdeveloped GUI parts as a potential improvement.Source: D. RQ4: Effectiveness of Number of… · p. 9
  • Increasing SCGG rounds mainly improved feature-related and app-completeness aspects rather than most other evaluated items.Compared with one loop, loops two through four significantly improved Feature Width and Complete App in the reported comparisons, except for Complete App at k=4.Source: D. RQ4: Effectiveness of Number of… · pp. 8–9

E. RQ5: Effectiveness of GUI Content Generation

  • Table IV compares ZS-CoT, PDGG, RAGGk=3, and SCGGk=3 with and without content generation using pairwise Wilcoxon tests on N=150.The table evaluates whether adding generated content changes GUI-prototype outcomes within each prompting approach.Source: Table IV · p. 9, E. RQ5: Effectiveness of GUI Content… · p. 9
  • LLM-based GUI content generation significantly improved Visual Appeal, Overall Satisfaction, and Complete App aspects across most prompting approaches.Feature Completeness also improved for most approaches, except ZS-CoT.Source: E. RQ5: Effectiveness of GUI Content… · p. 9

VI. LIMITATIONS

The proposed GUI-generation approaches have limitations involving retrieval dependence, evaluation validity, and the generalizability of Rico-based descriptions.

VI. LIMITATIONS

  • RAGG depends heavily on retrieval quality: false positives can reduce effectiveness, and missing relevant repository examples trigger fallback to zero-shot instructions.Its effectiveness may improve when external repository knowledge complements the LLM’s internal knowledge.Source: §VI · p. 9
  • SCGG achieves the highest effectiveness but consumes many tokens for critique and prototype regeneration.The authors suggest updating only affected GUI sections as a more efficient regeneration strategy.Source: §VI · p. 10

1) Internal Validity:

  • More complex generated prototypes can increase organization, implementation-error, and visual-design difficulty, while unnormalized metrics may favor simpler baselines.This relationship between feature count and measured items is identified as a threat to internal validity.Source: 1) Internal Validity · p. 10

2) External Validity:

  • Rico-based descriptions may restrict generalizability because rare-domain GUIs can be absent or ambiguously described, and retrieval fails for over 20% of descriptions.The evaluation filtered for at least seven GUI component types and excluded functionally simple GUIs by majority voting.Source: 2) External Validity · p. 10

VIII. RELATED WORK

Prior GUI-generation methods mainly produce low-fidelity, non-interactive, or image-based outputs and often require substantial training, while retrieval methods commonly return non-adaptable images. The paper combines retrieval with LLM generation and concludes that SCGG is strongest, although its token cost remains high.

1) GUI Generation:

  • Earlier GUI-generation systems produce coarse layouts, image-based prototypes, or custom representations that lack the interactive and design detail of the proposed prototypes.Several approaches also require extensive training resources or generate outputs difficult to integrate into practical prototyping tools.Source: 1) GUI Generation · p. 10

2) GUI Retrieval:

  • Natural-language retrieval methods rapidly find relevant GUIs, but their outputs are usually non-interactive and non-adaptable images.The paper integrates retrieval with LLMs to generate interactive prototypes and proposes LLM-based re-ranking.Source: 2) GUI Retrieval · p. 10
  • Other retrieval systems use sketches, wireframes, screenshots, GUI components, stylistic keywords, or templates as alternative input types.These methods broaden retrieval beyond natural-language descriptions.Source: 2) GUI Retrieval · p. 11

IX. CONCLUSION & FUTURE WORK

  • SCGG mostly significantly outperforms the other zero-shot methods, while larger RAGG example sets improve several GUI-generation metrics.RAGG is most effective when LLM knowledge and GUI-dataset knowledge are complementary.Source: §IX · p. 11
  • SCGG’s effectiveness is limited by large token consumption, motivating dynamic feedback and updates to only affected GUI sections.The proposed future direction decomposes regeneration to improve efficiency.Source: §IX · p. 11

Related papers

Loading 2412.11328v1…