PaperlayerView exact revision on arXiv

Source-linked AI summary

From Natural Language Requirements to Graphical User Interfaces: Automated Prototyping and Verification with Pretrained Language Models

Kristian Kolthoff

arXiv:2608.24749v1cs.SE

TL;DR

GUI prototyping and verification from natural-language requirements remain labor-intensive and limited by ambiguity. This paper develops retrieval, reranking, generation, and LLM-based verification approaches, reporting improved GUI ranking, prototyping support, and high-fidelity generation over baselines. The findings indicate that these methods can reduce manual effort in GUI prototyping and requirements verification.

  • Problem

    Creating GUI prototypes from natural-language requirements remains labor-intensive and costly, while GUI descriptions can limit retrieval effectiveness.

    Source: §1.2 · p. 33, §4.8 · p. 120

  • Method

    The paper combines data-driven GUI retrieval and reranking with LLM-based generation, editable representations, and multimodal verification of GUI requirements.

    Source: §3.8 · p. 106, §4.8 · p. 120, §7.9 · p. 193

  • Results

    Across its approaches, the paper reports improved GUI ranking, effective prototyping support, and high-fidelity generation that mostly outperformed zero-shot baselines.

    Source: §3.8 · p. 106, §5.8 · p. 142, §6.8 · p. 164

  • Takeaways & Limitations

    The results support using retrieval, reranking, and prompting methods to reduce manual effort in GUI prototyping and requirements-related activities.

    Source: §3.8 · p. 106, §6.8 · p. 164, §7.9 · p. 193

    The findings may not generalize to rare or specialized GUI domains absent from the Rico-based datasets, and broader user evaluations are needed.

    Source: §3.5 · p. 103, §6.5 · p. 162

Abstract

from arXiv · show

Requirements elicitation is essential for developing interactive software systems, as it helps ensure that the resulting product meets stakeholder needs. Since elicitation typically relies on natural language (NL), misunderstandings can arise from its inherent ambiguity. Formal specifications can reduce ambiguity but require technical expertise. GUI prototyping therefore provides a valuable alternative by turning requirements into tangible visual artifacts that support communication, elicitation, and validation. However, creating high-fidelity prototypes remains time-consuming and costly. Similarly, requirements verification, which ensures that implementations conform to specified requirements, is still largely manual, while existing automated approaches are often limited to static, rule-based techniques. This work addresses two challenges: (C1) reducing the effort required to transform NL requirements into GUI prototypes, and (C2) reducing the effort required for requirements verification in GUI applications and prototypes. For C1, we introduce novel NL-based GUI retrieval and reranking methods, new benchmarks, and techniques for efficiently adapting LLMs to GUI generation, including proprietary GUI representations. Their effectiveness is demonstrated on a large benchmark with human annotations. For C2, we propose LLM-based methods for verifying semantically complex NL requirements on static GUI prototypes and introduce a multimodal LLM-based agent for verifying complex functional and non-functional requirements in dynamic GUI applications through automatically generated and evaluated interaction trajectories. Overall, the proposed methods substantially reduce manual effort in GUI prototyping and requirements verification.

Zusammenfassung

Die Arbeit reduziert den manuellen Aufwand für GUI-Prototyping und Anforderungsverifikation durch NL- und LLM-basierte Verfahren. Sie entwickelt Methoden, Benchmarks und einen multimodalen Agenten für die Erzeugung und Verifikation von GUI-Prototypen und -Anwendungen.

Zusammenfassung

  • GUI-Prototyping unterstützt Anforderungserhebung und -validierung durch greifbare visuelle Artefakte, verursacht bei hochreifen Prototypen jedoch erheblichen manuellen, zeitaufwendigen und kostspieligen Aufwand.Anforderungsverifikation bleibt ebenfalls häufig manuell und auf regelbasierte statische Techniken beschränkt.Source: Zusammenfassung · p. 6
  • Die Arbeit adressiert C1 durch NL-basierte GUI-Suche und -Neubewertung, effiziente LLM-Anpassung sowie die Generierung proprietärer GUI-Repräsentationen.Diese Verfahren übersetzen NL-Anforderungen schnell in anpassbare GUI-Prototypen und zeigen hohe Effektivität auf Benchmarks mit menschlichen Annotationen.Source: Zusammenfassung · pp. 6–7
  • Für C2 entwickelt die Arbeit LLM-basierte Methoden zur Verifikation semantisch komplexer Anforderungen auf statischen GUI-Prototypen und einen multimodalen Agenten für dynamische Anwendungen.Der Agent erzeugt und evaluiert automatisch sinnvolle Interaktionspfade zur Verifikation komplexer funktionaler und nicht-funktionaler Anforderungen.Source: Zusammenfassung · pp. 6–7

List of Publications

  • Die Arbeit basiert auf Veröffentlichungen aus führenden Software-Engineering-, HCI- und Requirements-Engineering-Konferenzen sowie einer Fachzeitschrift.Die Publikationen umfassen Texte, Tabellen und Abbildungen und sind chronologisch aufgelistet; einige Beiträge befinden sich noch im Peer-Review.Source: List of Publications · p. 14

Introduction

The introduction presents an approach to automating GUI prototyping and verification from ambiguous natural-language requirements. It covers retrieval, generation, elicitation, editable prototypes, and staged verification of static and dynamic GUI applications using LLM- and MLLM-based methods.

1.1 Motivation

  • Ambiguous, incomplete, heterogeneous NLR and multidimensional GUIs make automatic alignment difficult, motivating pretrained language models for GUI prototyping and verification.NLR lacks fixed syntax and semantics, while GUIs combine structure, functionality, appearance, and dynamic behavior.Source: §1.1 · p. 33

1.2 Problem Statement

  • The work targets two major challenges: automating high-fidelity GUI prototype creation from NLR and automating GUI-application verification against NLR.Both tasks remain labor-intensive and require expert knowledge or technical expertise.Source: §1.1 · p. 32, §1.2 · p. 33

1.3 Solution Overview

  • The proposed solutions span NLR-based GUI retrieval, NLR-to-GUI generation, and LLM- or MLLM-agent-based verification for mobile and web GUIs.The introduction divides the work into three parts addressing the two major challenges.Source: §1.2 · p. 34, §1.3 · pp. 36–37
  • The retrieval approach ranks GUIs from a repository exceeding 72,000 examples using extracted text, information retrieval, pretrained-language-model fine-tuning, multimodal reranking, and dialogue-based guidance.It supports rapid mapping across a broad spectrum of real-world GUIs but remains restricted to the repository.Source: §1.3 · p. 36
  • The generation approach efficiently adapts pretrained LLMs through zero-shot prompting, repository retrieval, prompting strategies, and proprietary component-library retrieval to produce editable GUI prototypes.The editable prototypes are integrated into a GUI prototyping editor with full editing functionality.Source: §1.2 · p. 35, §1.3 · p. 37
  • An MLLM-based agent verifies complex functional and non-functional requirements in dynamic web GUIs by predicting, executing, and evaluating interaction steps iteratively.The approach receives updated GUI states and repeats action prediction to assess requirement fulfillment automatically.Source: §1.3 · p. 37

1.4 Contributions

  • The contributions include new datasets for GUI prototyping and verification, evaluated through novel benchmarks and analyses demonstrating the effectiveness of NLR-based automation.The datasets support training, comprehensive evaluation, and future research involving PLMs, LLMs, and MLLMs.Source: §1.4 · p. 38
  • A GUI retrieval benchmark contains 450 NLR–GUI sets with 20 GUIs each and over 40,500 three-level relevance annotations for training and evaluation.The benchmark supports text-based retrieval, ranking, and reduction of the semantic gap between NLR and multidimensional GUI prototypes.Source: §1.4 · p. 38
  • Additional datasets provide multidimensional Rico annotations with 3,072-dimensional embeddings, high-quality GUI descriptions and prototypes, and interlinked user stories with GUI components.These resources support retrieval, generation, semantic understanding, traceability, and static verification research.Source: §1.4 · pp. 38–39
  • For dynamic verification, a gold-standard dataset covers five web applications and labels user stories as met, unmet, or partially met against detailed acceptance criteria.It is presented as the first dataset combining requirements with application mappings for agent-based GUI verification.Source: §1.4 · p. 39
  • The prototyping contributions include GUI retrieval and reranking, self-elicitation through dialogue, prompting methods such as PD, RAGG, and SC, and editable GUI generation through a proprietary DSL.These methods are integrated into tools including RaWi, GUI-ReRank, SERGUI, a Figma plugin, and related prototyping editors.Source: §1.4 · pp. 39–40
  • For static verification, an LLM matches user stories to GUI components and tests zero-shot, few-shot, and chain-of-thought prompting paradigms.This feasibility approach verifies requirements using fixed GUI hierarchy data while excluding dynamic behavior.Source: §1.3 · p. 37, §1.4 · p. 40

1.5 Outline

  • The thesis is organized into background, three technical parts covering retrieval, generation, and verification, and a final discussion and conclusion.Chapters report motivation, methods, evaluation, results, limitations, and related work.Source: §1.5 · p. 41

Theoretical Background

This section establishes the theoretical foundations for requirements engineering, GUI prototyping, and the computational methods underlying the paper. It covers iterative, human-centered prototyping and GUI representations, along with machine learning, information retrieval, pretrained language models, and techniques for adapting them to retrieval and reasoning tasks.

Theoretical Background

  • The chapter’s theoretical foundation also covers machine learning, information retrieval, pretrained language models, in-context learning, prompt engineering, and context engineering.These computational methods are presented alongside requirements engineering, software engineering, and GUI prototyping as foundational topics.Source: Theoretical Background · p. 43

2.1 Requirements and Software Engineering (RE/SE)

  • RE systematically elicits, specifies, validates, verifies, and manages requirements, whose ambiguity can cause failures and whose late correction becomes substantially more expensive.About 60% of software failures originate in early project stages, while correction costs rise from 3–8-fold during design to 29–1500-fold during operation.Source: §2.1 · pp. 43–44

2.1.1 Requirements Elicitation (REl)

  • Requirements elicitation explores domains and sources, selects context-dependent techniques, and iteratively refines specifications; prototyping is one approach among interviews, task analysis, JAD, and related methods.No universal elicitation method exists, so techniques are commonly combined according to requirements, project phase, analyst experience, and budget or time constraints.Source: §2.1.1 · pp. 48–49

2.1.2 Requirements Validation (RVal)

  • Requirements validation checks alignment with stakeholder needs, whereas verification checks specification correctness, including consistency and absence of internal errors; prototyping supports both elicitation and validation.Validation techniques include commenting, inspections, walk-throughs, prototyping, and model-based simulation, with differing formality, effort, collaboration, and artifact characteristics.Source: §2.1.2 · pp. 49–50

2.1.3 Software Validation (SVal) and Verification (SVer)

  • Software validation assesses alignment with stakeholder needs, while software verification assesses conformity to specifications; the V-model links these activities to testing across abstraction levels.Acceptance testing is validation because it involves users, whereas unit, integration, and system testing are verification activities performed against corresponding specifications.Source: §2.1.3 · pp. 50–52

2.2 GUI Prototyping

  • GUI prototyping creates simplified, abstracted software versions that reduce development costs and risks while providing tangible visual artifacts for requirements elicitation, communication, and validation.Prototypes range from pen-and-paper sketches and mockups to high-fidelity versions resembling the final product; visual representations can reduce ambiguity and enable earlier stakeholder feedback.Source: §2.2 · p. 52

2.2.1 History of GUI Prototyping Tools

  • GUI prototyping tools evolved over roughly forty years from UIMS-inspired and non-programming approaches toward direct manipulation, sketch recognition, widget libraries, and increasingly detailed prototypes.Early systems emphasized WYSIWYG interaction and sketch translation, while widget libraries supported low- and high-fidelity representations.Source: §2.2.1 · pp. 53–54
  • GUI prototyping tools evolved from UIMS-based behavior specification and widget libraries toward annotations, code generation, scenario management, usability testing, collaboration, and commercial editors.Early tools also introduced preview modes, reuse mechanisms, and predefined behaviors to accelerate prototyping.Source: §2.2.1 · pp. 54–55

2.2.2 GUI Prototyping Processes and Approaches

  • GUI prototyping processes are inherently iterative: requirements and context are analyzed, GUI specifications and prototypes are created, and artifacts are evaluated, validated, and refined through user, task, and domain feedback.These models closely resemble requirements-engineering processes while emphasizing design creation and evaluation.Source: §2.2.2 · pp. 55–56, Figure 2.9 · p. 55, Figure 2.10 · p. 56

2.2.3 GUI Prototype Characteristics

  • GUI prototypes vary by relation to the final product, scope of implementation, execution timing, fidelity, and internal representation, with fidelity best understood as a spectrum across visual, functional, interactive, and data-model dimensions.Low-fidelity prototypes are abstract and rapidly created, whereas high-fidelity prototypes provide greater detail, realism, and feedback quality; representations range from web standards to proprietary hierarchical or binary structures.Source: §2.2.2 · p. 56, §2.2.3 · pp. 57–58

2.3 Computational Methods and Techniques

  • The computational foundations comprise machine learning, information retrieval, pretrained language models, and efficient language-model adaptation methods such as parameter-efficient and computationally efficient approaches.These topics establish the methodological background for the paper’s subsequent techniques.Source: §2.3 · p. 58

2.3.1 Machine Learning (ML)

  • Machine learning learns models from data to predict, decide, or represent unseen examples, using unsupervised, supervised, semi-supervised, self-supervised, or reinforcement-learning paradigms.Self-supervised learning is especially important for PLMs because it exploits inherent data correlations, such as contextual or next-token prediction, instead of human annotations.Source: §2.3.1 · pp. 58–59

2.3.2 Information Retrieval (IR)

  • Information retrieval ranks documents for queries using relevance-scoring functions, progressing from exact Boolean matching and sparse BoW or TF-IDF representations to BM25 and semantic word embeddings such as word2vec.TF-IDF reduces the influence of frequent, less informative terms, while word2vec captures semantic relationships through contextual prediction.Source: §2.3.2 · pp. 60–62

2.3.3 Pretrained Language Models (PLMs)

  • Pretrained language models learn contextualized representations from large corpora through token-prediction objectives and typically use Transformer encoder-only or decoder-only architectures.Transformers use self-attention and multi-head self-attention to model long-range dependencies and diverse relationships across representation spaces; BERT is an encoder-only bidirectional example pretrained with masked-language modeling and next-sentence prediction.Source: §2.3.3 · pp. 62–63
  • BERT represents sequences with [CLS] and [SEP] tokens and can be fine-tuned end-to-end or through a task-specific prediction head, while SBERT uses shared BERT encoders and cosine similarity for efficient text comparison.BERTBASE has 12 layers, 768 hidden dimensions, 12 attention heads, and 110M parameters; SBERT pools sentence representations and trains them with a siamese architecture.Source: §2.3.3 · pp. 64–65
  • Pretrained language models span BERT, SBERT, large language models, and multimodal language models, supporting increasingly broad language and image-processing capabilities.BERT uses Transformer encoding and pretraining objectives; SBERT enables efficient sentence-embedding comparison; LLMs scale decoder-based pretrained models; MLLMs integrate visual encoders with LLMs.Source: §2.3.3 · pp. 64, 65, 67
  • LLMs are massive pretrained language models whose instruction-following ability can be improved through supervised fine-tuning, reward modeling, and reinforcement learning.They are typically decoder-only models trained autoregressively on large token sequences, and scaling improves performance while not necessarily producing instruction-following ability without adaptation.Source: §2.3.3 · pp. 65–66

2.3.4 Prompt and Context Engineering (PE/CE)

  • Prompt and context engineering adapts LLM behavior by designing instructions and constructing inputs with task-specific information, examples, external knowledge, and tools.In-context learning adapts at inference time without parameter updates; prompt engineering refines instructions, whereas context engineering systematically assembles relevant context and tools.Source: §2.3.4 · pp. 67–68
  • Key prompting strategies include zero-shot and few-shot learning, Chain-of-Thought reasoning, prompt decomposition, self-critique, and retrieval-augmented generation.Chain-of-Thought elicits intermediate reasoning, decomposition delegates subtasks to specialized prompts, self-critique iteratively revises outputs, and RAG injects retrieved external documents into context for knowledge outside pretraining data.Source: §2.3.4 · pp. 68–69
  • Other reasoning methods extend these approaches by sampling and aggregating diverse paths or organizing intermediate thoughts into tree and graph structures.Self-consistency aggregates multiple reasoning paths, Tree-of-Thoughts supports self-evaluation and backtracking, and Graph-of-Thoughts represents arbitrary dependencies among reasoning steps.Source: §2.3.4 · p. 69

Natural-Language-Based GUI Retrieval

This section presents RaWi and related methods for retrieving reusable GUI prototypes from natural-language requirements, deriving partly editable screens, and supporting rapid prototyping. It also describes the web-editor integration, crowdsourced retrieval benchmark, and evaluation methodology.

3.1 Motivation

  • RaWi maps natural-language requirements to reusable, partly editable GUI screens from a large-scale repository, reducing manual effort in rapid interactive prototyping.The approach targets high-fidelity functionality prototyping for requirements elicitation, particularly when analysts have limited time or experience.Source: §3.1 · pp. 74–76

Natural-Language-Based GUI Retrieval

  • The work contributes a crowdsourced gold standard, a BERT-LTR ranking model, an in-depth comparison of traditional IR, AQE, and BERT-based methods, and a productivity-focused user-study methodology.The benchmark and supporting materials are made publicly available to enable systematic evaluation and future research.Source: Natural-Language-Based GUI Retrieval · p. 73, §3.1 · p. 75

3.2 Approach: RaWi

  • RaWi combines the Rico GUI repository, shared preprocessing for GUI text and NLR queries, multi-stage retrieval and reranking, automatic screen derivation, and a web-based prototyping editor.Initial candidate retrieval is followed by ranking with established IR or BERT-based learning-to-rank models, while screenshots and hierarchy data support partly editable screen generation.Source: §3.2 · pp. 76–77, Figure 3.2 · p. 77

3.2.1 GUI Repository and Text Extraction

  • After filtering, the Rico-based repository contains 57,764 unique GUIs, while excluding entertainment, advertisement or full-screen web-view mismatches, and non-English interfaces.Rico was selected for its scale, domain diversity, and detailed component text; residual completely mismatched hierarchy-screenshot pairs are considered infrequent and difficult to identify.Source: §3.2.1 · pp. 78–79
  • GUI representations extract visible text, hints, activity names, resource identifiers, and icon labels, using tokenization and domain-specific stopword filtering to support whole-GUI and partial-GUI retrieval.Non-visible text is omitted because it can mislead retrieval, while activity names and resource identifiers are tokenized across punctuation, camel case, and snake case.Source: §3.2.1 · p. 79, Figure 3.3 · p. 79

3.2.2 GUI Retrieval and Text Preprocessing

  • The retrieval pipeline normalizes text, evaluates TF-IDF, BM25, dense-embedding nBoW, and PRF-KLD query expansion, and fine-tunes BERT-LTR to address the semantic gap between abstract NLR and fragmented GUI text.PRF-KLD expands queries from top-k retrieved GUIs and adapts expansion across GUI text segments, whereas BERT receives the NLR query and GUI text as a structured query-document pair.Source: §3.2.2 · pp. 79–82
  • RaWi represents each NLR-GUI pair with BERT and ranks it using an FFNN score optimized with pointwise, pairwise, or listwise learning-to-rank losses.A pretrained SBERT model is included as an additional semantic baseline for comparison.Source: §3.2.2 · pp. 82–83

3.2.3 Deriving Editable GUI Prototypes

  • The prototype-derivation method transforms Rico screenshots into partly editable GUI screens by cropping hierarchy-identified components, extracting style properties for labels, buttons, and text inputs, and restoring their original positions.For labels, it additionally refines bounding boxes with Tesseract-OCR and estimates font colors and sizes using histogram- and geometry-based heuristics.Source: §3.2.3 · pp. 83–84

3.2.4 Prototype Implementation

  • The RaWi web-based editor combines GUI search, graphical editing, and a dynamically updated interactive preview for data-driven rapid prototyping.Users can modify retrieved components, create or reuse components, remove or copy them, bind them to other screens, and add simple click-event transitions.Source: §3.2.4 · pp. 84–85

3.3 Experimental Evaluation

  • The experimental evaluation measures GUI-retrieval performance, prototyping productivity against MockPlus, and perceived usefulness using retrieval metrics, a within-subjects study, and a five-point questionnaire with SUS.RQ1 compares TF-IDF, BM25, nBoW, PRF-KLD-based AQE methods, SBERT, and the three BERT-LTR variants.Source: §3.3 · p. 86

3.3.1 RQ1: GUI Retrieval Performance

  • The RQ1 gold standard contains 100 queries with 20 GUIs each, constructed from crowdsourced NLR queries and relevance annotations after filtering for annotation uncertainty and query quality.The source collection retained 931 GUI-NLR-query pairs, while 450 queries received annotations for pooled top-30 GUIs before the final filtering process.Source: §3.3.1 · pp. 86–90

3.4.1 RQ1: GUI Retrieval Performance

  • BERT-LTR models consistently outperform BM25 across mean metrics, with BERT-LTR (2) significantly improving AP, HITS@5,10, P@7,10, and NDCG@3,5,10,15.BERT-LTR models also outperform SBERT across mean metrics, with a significant BERT-LTR (2) improvement on HITS@5.Source: Table 3.2 · p. 95, Table 3.3 · p. 96, §3.4.1 · pp. 97, 99

3.4.2 RQ2: Productivity of Rapid Prototyping

  • RaWi significantly outperforms Mockplus in correct components, diverse data components, and adjusted component counts, with large effects across time steps.RaWi’s advantage is strongest for diverse realistic data, although it tends to retain more unrequested components than traditional prototypes.Source: §3.4.2 · pp. 100–101, Figure 3.15 · p. 100

3.4.3 RQ3: Perceived Usefulness

  • RaWi achieved SUS 81.57 and high ratings for prototyping support, retrieval relevance, retrieval support, and visualization assistance.The four mean ratings were 3.73, 3.63, 4.21, and 4.47, respectively; participants requested more sophisticated editing functionality.Source: §3.4.3 · pp. 101–102, Figure 3.16 · p. 102

3.5 Threats to Validity

  • The evaluation reduced bias through crowdsourced query and relevance annotation procedures, randomized study conditions, and matched prototyping procedures.Potential selection bias remains, and the controlled experiment used two application domains.Source: §3.5 · pp. 102–103

3.6 Limitations

  • RaWi retrieval is restricted to Rico, prototypes may lack cohesive design, and the early implementation has limited editing functionality.The authors suggest integrating datasets such as ReDraw and extending the prototype into a fully fledged graphical editor.Source: §3.6 · p. 103

3.7 Related Work

  • Unlike visual-input retrieval systems, RaWi supports NLR-based GUI search with BERT-based ranking and directly converts retrieved GUIs into partly editable prototypes.Direct retrieval comparison with Guigle is unavailable because its implementation and dataset differ; the work provides a public NLR-based GUI retrieval gold standard.Source: §3.7 · p. 104
  • RaWi retrieves matching GUIs from a large repository using NLR queries and provides partly editable screens for interactive prototyping, facilitating requirements elicitation and validation.This contrasts with traditional tools such as Balsamiq, Sketch (Sketch), Figma (Figma), and Mockplus (Mockplus), which rely primarily on graphical editors, templates, and basic components.Source: §3.7 · pp. 105–106

3.8 Conclusion

  • BERT-LTR models substantially improved NLR-based GUI ranking over the strong text-based BM25 baseline, while RaWi also improved prototyping productivity over a traditional approach.The work introduced a first GUI retrieval gold standard to support systematic evaluation of NLR-based GUI search and addressed challenge C1-1.Source: §3.7 · p. 105, §3.8 · p. 106

GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking

This section presents GUI-ReRank, a multimodal LLM-based approach that combines customizable multi-dimensional retrieval and reranking to connect natural-language requirements with image-rich GUI prototypes. It examines text- and image-based model options, effectiveness–cost trade-offs, and limitations affecting evaluation and use.

4.1 Motivation

  • GUI-ReRank addresses the representation gap by combining customizable multi-dimensional constrained retrieval with MLLM reranking over GUI images or rich textual annotations.The framework supports complex positive and negative requirements across domain, functionality, design, and GUI-component dimensions.Source: §4.1 · pp. 108–109

4.2.1 Automated GUI Repository Annotation

  • The annotation pipeline uses a zero-shot MLLM to create customizable search-dimension text annotations and embeddings, with Rico provided as a pre-annotated, searchable dataset.Default dimensions include domain, functionality, design, GUI components, and displayed text; users can add dimensions such as platform or accessibility.Source: §4.2.1 · pp. 109–110, §4.1 · p. 109, Figure 4.2 · p. 110

4.2.2 Embedding-based Constrained GUI Retrieval

  • The constrained retrieval stage decomposes requirements into search-dimension-specific positive and negative constraints, embeds them, and combines weighted similarities into a final ranking score.This enables fine-grained multi-dimensional searches, including negation, before reranking the top-k GUIs.Source: §4.2.2 · pp. 110–111, Figure 4.3 · p. 111

4.2.3 Multi-Modal (M)LLM-based GUI Reranking

  • The reranker assigns 0–100 dimension-specific relevance scores using either screenshots or annotations, enabling an effectiveness–cost trade-off before normalized score aggregation.Image input is more information-dense but more expensive, whereas text input is cheaper but remains an abstraction of the GUI.Source: §4.2.3 · pp. 111–112, Figure 4.4 · p. 112

4.4.1 RQ1: Effectiveness of MLLM-based GUI Reranking

  • 67.66% higher AP and 87% top-ranked relevance were achieved by GPT-4.1 (Image) over pairwise BERT-LTR (2), including an 89.13% HITS@1 improvement.All MLLM variants substantially outperformed BERT-LTR in mean metrics, with statistically significant differences across all metrics except HITS@10,15.Source: Table 4.1 · p. 113, Table 4.2 · p. 114, §4.4.1 · p. 115

4.3 Experimental Evaluation

  • Evaluation used 100 natural-language queries paired with 20 Rico GUIs, compared MLLM variants against BERT-LTR using ranking metrics and tracked tokens, runtime, and monetary cost.The study tested GPT-4.1, Gemini-2.5, and Claude Sonnet variants, with Wilcoxon tests and Holm correction for significance analysis.Source: §4.3.1 · pp. 113–114, §4.3 · p. 113

4.4.1 RQ1: Effectiveness of MLLM-based GUI Reranking

  • Image-based GPT-4.1 consistently outperformed its text-based counterpart, while larger GPT-4.1 capacities significantly outperformed smaller variants, especially on NDCG@k.The image advantage was significant solely for NDCG@k; GPT-4.1 Nano was an exception, with text outperforming image across mean metrics except HITS@5.Source: §4.4.1 · pp. 115–117

4.4.2 RQ2: MLLM-based Reranking Effectiveness-Cost Trade-offs

  • 3.32% higher mean AP from GPT-4.1 (Image) required 543.9% higher API cost than GPT-4.1 (Text), making text variants substantially more cost-efficient.All text-based GPT-4.1 and Gemini-2.5 models, plus image-based GPT-4.1, lay on the API-cost Pareto frontier.Source: Figure 4.7 · p. 118, §4.4.2 · p. 118
  • Text-based GPT-4.1 and Gemini-2.5 models provide Pareto-efficient MLLM-based GUI reranking solutions, while text-based reranking substantially reduces costs.The trade-off considers effectiveness and reranking time for k=100; text-based models offer high effectiveness at lower monetary, token, and latency costs.Source: §4.4.2 · p. 118, Figure 4.8 · p. 119, §4.6 · p. 120

4.5 Threats to Validity

  • The evaluation’s internal and external validity mainly depend on biases, subjectivity, generalizability, repository diversity, and worker population in the NLR-based GUI retrieval gold standard.Potentially relevant GUI pooling and relevance judgments create internal-validity concerns, while query coverage, Rico diversity, and crowdsourcing workers affect external validity.Source: §4.5 · p. 119

4.6 Limitations

  • MLLM reranking can vary slightly across runs, incurs monetary, token, and latency costs, and requires expertise to tune multi-dimensional retrieval and reranking weights.Manual checks across runs found few deviations in GUI rankings, while text-based reranking reduces costs; leveraging customizable search capabilities still requires appropriate weight tuning.Source: §4.6 · pp. 119–120

4.7 Related Work

  • Earlier work addressed sketch-, wireframe-, component-, text-, or image-based retrieval and reranking, whereas GUI-ReRank introduced multi-dimensional MLLM-based GUI reranking.The chapter positions its approach as the first to propose and explore multi-dimensional MLLM-based GUI reranking.Source: §4.7 · p. 120

4.8 Conclusion

  • GUI-ReRank reduces the semantic gap between natural-language requirements, including non-functional requirements, and multi-dimensional GUI prototypes through annotated representations, constrained retrieval, and MLLM-based rank scoring.Its annotation pipeline supplies multi-dimensional textual GUI representations, while MLLMs use language and image understanding to capture information neglected by hierarchy-text-only approaches.Source: §4.8 · p. 120

Self-Elicitation of Requirements with Automated GUI Prototyping

This section presents SERGUI, a stakeholder-led approach to self-eliciting requirements by combining GUI retrieval and reranking, feature recommendations, visualization, and web-based prototyping support. It examines how these capabilities assist users in creating GUI specifications from natural-language requirements, including their perceived usability and identified limitations.

5.1 Motivation

  • SERGUI addresses RaWi’s lack of guidance for stakeholder-driven elicitation and validation by providing a structured, dialogue-based prototyping process for users with little prior GUI-prototyping experience.Earlier approaches either required experienced analysts, offered no automatic guidance, or focused only on textual requirements.Source: §5.1 · pp. 122–123

Self-Elicitation of Requirements with Automated GUI Prototyping

  • SERGUI enables stakeholders to conduct self-elicitation through automated NLR-based GUI prototyping assistance, reducing dependence on analysts during initial requirements elicitation (Kolthoff et al., 2024a).It combines GUI and feature retrieval with the Rico repository and produces selected GUIs plus aspect-GUIs that highlight particular features.Source: §5.1 · pp. 122–123

5.2 Approach: SERGUI

  • The approach combines an interaction model, NLR-based GUI retrieval, NLR-based feature retrieval, and LLM-based feature recommendation to automate initial requirements elicitation.The LLM component suggests potentially relevant features and illustrates them by matching them to top-ranked GUIs.Source: §5.2 · p. 124

5.2.1 Interaction Model

  • Users specify GUI-level requirements to obtain ranked screens, then specify feature-level requirements to retrieve aspect-GUIs whose selected features can refine the prototype and recompute the ranking.The resulting specification can include a selected GUI, aspect-GUIs, and textual requirements within a simple linear application prototype.Source: §5.2.1 · p. 125

5.2.2 NLR-Based GUI Retrieval

  • The retrieval component filters the Rico repository and ranks GUIs using SBERT embeddings over extracted GUI-hierarchy text, combined with high-level Screen2Words descriptions for a more flexible score.The filtering pipeline removed 23,817 GUIs, leaving 48,402; its CNN classifier achieved Precision=.9818 / Recall=.7012 on a separate test set.Source: §5.2.2 · pp. 126–127

5.2.3 NLR-based GUI Feature Retrieval

  • Feature retrieval extracts textual representations of individual components from the top-k GUIs and matches them to feature requirements using SBERT cosine similarity.Retrieved components are represented as aspect-GUIs that highlight the corresponding feature for visualization and validation.Source: §5.2.3 · p. 127, Figure 5.4 · p. 127

5.2.4 LLM-based GUI Feature Recommendation

  • Given GUI requirements, specified features, and a selected GUI, the LLM recommends up to 30 context-specific features, ranks them against candidate GUI components, and generates short explanations for dialogue-based elicitation.The GUI hierarchy is abstracted into ordered layout groups and components before prompting, while matched aspect-GUIs visualize recommended features.Source: §5.2.4 · pp. 128–129

5.2.5 Feature-based GUI Reranking

  • Selected features are incorporated into a feature-based ensemble reranking score so users can surface more relevant GUIs beyond the initially explored top-20 or top-30 results.The reranking can either promote more relevant GUIs or preserve the user’s original choice.Source: §5.2.5 · p. 130

5.2.6 GUI Prototype Output

  • SERGUI produces an application summary combining an overall wireframe, a selected main GUI, aspect-GUIs, and textual requirements for features lacking relevant visual matches.The web implementation provides a chat section, a workbench for top-k GUIs or aspect-GUIs, and a GUI prototype preview.Source: §5.2.6 · pp. 130–131, §5.2.7 · p. 131

5.3 Experimental Evaluation

  • The experimental evaluation assessed feature recommendation relevance, aspect-GUI retrieval, feature-based reranking, and perceived usefulness through four research questions and corresponding IR, ranking, and usability measures.RQ1 used AP, MRR, and P@k; RQ2 used MRR and HITS@k; RQ3 compared selected-GUI ranks before and after reranking; and RQ4 used Likert ratings and SUS.Source: §5.3 · pp. 131–132

5.3.1 User Study

  • The evaluation used 12 technically experienced participants, 75% of whom had never used a GUI prototyping tool, who created 72 GUIs across 24 applications.The study covered shopping, news, and social applications, with each participant completing two of three prototyping tasks.Source: §5.3.1 · pp. 132–133

5.4.1 RQ1: Feature Recommendation Relevance

  • SERGUI achieved mean AP = 74.1 and MRR = 81.6 for LLM-based feature recommendations, with P@1 = 69.1 and P@10 = 63.8.Approximately three out of four recommended features were marked relevant on average, and the first relevant feature appeared at rank 1.22 on average.Source: §5.4.1 · pp. 134–136

5.4.2 RQ2: GUI Feature Retrieval Relevance

  • SERGUI retrieved relevant aspect-GUIs with MRR = 39.0, HITS@1 = 27.0, HITS@5 = 52.7, and HITS@15 = 69.1.For actively searched features, performance was MRR = 26.2 and HITS@15 = 78.6; matching became harder with over-specified feature text and heterogeneous initial GUI rankings.Source: §5.4.2 · pp. 136–137

5.4.3 RQ3: GUI Reranking Performance

  • 68.05% of cases yielded a better-matching GUI after feature feedback, with an average reranking improvement of +61.89 ranks.The ensemble model achieved MRR = 42.9, compared with 17.2 for SBERT and 21.0 for S2W.Source: §5.4.3 · pp. 137–138

5.4.4 RQ4: Perceived Usefulness

  • SERGUI received SUS = 84.16 versus the average SUS of 68, while participants rated prototyping support and adding initially overlooked features at M = 4.25.Feature visualizations received lower ratings, and participants reported that they sometimes failed to match the recommended features.Source: §5.4.4 · pp. 138–139

5.5 Threats to Validity

  • SERGUI is constrained by Rico’s data-driven domain coverage and a rigid interaction flow with binary feature feedback, motivating broader evaluation, greater flexibility, and fine-grained natural-language feedback.The evaluation used nine GUIs from three domains, and the authors note that subjective relevance annotations and limited participant diversity affect validity.Source: §5.6 · p. 140, §5.5 · p. 140

5.6 Limitations

  • SERGUI remains limited by inconsistent feature granularity and matching, omitted standard features, and views restricted to the initially selected GUI and current prototype.Potential improvements include grouping hierarchical or semantically related features, optimizing feature matching, adding standard actions such as undoing errors, and displaying all selected aspect-GUIs and text features.Source: §5.6 · p. 140

5.7 Related Work

  • SERGUI extends prior GUI retrieval work with automatic feature recommendation, feature visualization and search, feature-based GUI reranking, and an integrated prototyping artifact.The user-friendly, chat-guided tool requires no prior prototyping knowledge or experience and integrates selected GUIs and features for analysts.Source: §5.7 · p. 141

5.8 Conclusion

  • SERGUI effectively supports GUI prototyping by retrieving relevant GUIs, recommending and visualizing GUI features, and achieving high perceived usability.Users can rapidly and independently create GUI specifications through multiple interaction mechanisms while requiring little prototyping experience.Source: §5.8 · p. 142

Retrieval-Augmentation and Self-Critique Prompting

This chapter presents zero-shot prompting methods for generating executable, high-fidelity HTML/CSS GUI prototypes from natural-language requirements, combining prompt decomposition, retrieval augmentation, and self-critique. It evaluates these methods through human and expert annotations, examining their effects on prototype quality and the influence of retrieval examples, self-critique iterations, and content generation.

6.1 Motivation

  • The chapter’s zero-shot approaches target high-fidelity HTML/CSS generation while avoiding the resource costs of training or fine-tuning.The methods address retrieval’s limited customizability and prior approaches’ dependence on resource-intensive training or low-fidelity DSL outputs.Source: §6.1 · pp. 146–147

6.3 Experimental Evaluation

  • RAGG and SC substantially enhance GUI prototypes across multiple evaluation metrics compared with multiple zero-shot baselines.The evaluation used over 20,400 GUI annotations from 101 crowd-workers with UI/UX experience.Source: §6.1 · p. 147

6.2.2 Prompt Decomposition (PD) for GUI Generation

  • Prompt Decomposition progressively extracts GUI features, implementation ideas, and layout structure before translating the refined specification into HTML/CSS.A single-prompt variant combines these domain-specific reasoning steps, reducing propagation of inaccurate information into the final prototype.Source: §6.2.2 · pp. 148–149

6.2.3 Retrieval-Augmented GUI Generation (RAGG)

  • RAGG retrieves candidate GUIs, filters and reranks them with zero-shot multimodal relevance judgments, and conditions generation on their aggregated feature, design, and layout knowledge.It uses SBERT retrieval, binary relevance filtering, finer-grained 1-to-10 relevance scoring, and either extracted GUI knowledge or directly supplied screenshots.Source: §6.2.3 · pp. 150–151

6.2.4 Self-Critique (SC) Looping for GUI Generation

  • SC iteratively improves an initial HTML/CSS prototype by having a feedback MLLM critique missing features, implementation, and design before a prototyping LLM generates revised code.The loop starts from a zero-shot instruction prototype and repeats for k iterations.Source: §6.2.4 · p. 151

6.2.5 LLM-based GUI Content Generation

  • The content-generation pipeline adds realistic data, generates image descriptions, creates corresponding images with DALL-E-3, and inserts their URLs into the HTML/CSS.It first augments the base prototype with content and image identifiers, then matches generated image URLs to those identifiers.Source: §6.2.5 · pp. 151–152, Figure 6.3 · p. 152

6.3 Experimental Evaluation

  • The evaluation compares prompting strategies, RAGG example counts, SC iteration counts, and GUI content generation using human ratings of prototype quality.The requirements dataset was built from filtered Rico GUIs, 574 collected GUI requirements, and subsequent quality ratings by UI/UX-experienced crowd-workers.Source: §6.3 · pp. 152–153, §6.3.1 · p. 154

6.3.1 GUI Requirements Dataset

  • The evaluation used 17 prompting models to generate interactive HTML/CSS GUIs for 50 requirements, producing 20,400 annotations from three ratings across eight 9-point Likert metrics.The requirements came from 146 high-quality GUI natural-language requirements, with 10 used for validation and 50 for final testing; GPT-4o was used at temperature .50.Source: §6.3.1 · p. 154, §6.3.2 · p. 156

6.4.1 RQ1: Effectiveness of Prompting for GUI Generation

  • Self-critique significantly outperformed baselines and prompt decomposition across most metrics, while RAGG with k = 7 improved over at least one baseline in most metrics and differed from SC only on Complete App.SC achieved significant gains on Feature Completion, Feature Extensiveness, Feature Implementation, Visual Appeal, Overall Satisfaction, and Complete App; RAGG improved significantly on Feature Completion, Feature Extensiveness, Visual Appeal, Overall Satisfaction, and Complete App.Source: §6.4.1 · pp. 157–159, Figure 6.5 · p. 158

6.4.2 RQ2: Effectiveness of Example Number on RAGG

  • Increasing RAGG examples raised mean ratings across metrics except Information Organization, with significant k = 1 versus k = 7 gains for Feature Completion, Feature Implementation, Visual Appeal, and Complete App.A significant improvement also occurred between k = 1 and k = 5 for Feature Completion, while most other increases were not significant.Source: §6.4.2 · p. 160

6.4.3 RQ3: Effectiveness of Number of Loops in SC

  • The first self-critique iteration significantly improved most quality metrics, whereas later iterations mainly improved Feature Extensiveness, Visual Appeal, and Complete App.Because later gains diminished while token consumption increased, the study recommends k = 1 as the default.Source: §6.4.3 · pp. 160–161

6.4.4 RQ4: Effectiveness of GUI Content Generation

  • Content generation significantly improved ZS-CoT and PD-ZS across most metrics, while RAGG and SC had consistently higher means without significant differences.Exceptions for ZS-CoT and PD-ZS included Feature Implementation and Information Organization; PD-ZS additionally showed no significant Feature Extensiveness improvement.Source: §6.4.4 · p. 161

6.5 Threats to Validity

  • Internal validity may be affected because feature count and complexity influence organization, implementation errors, and visual appeal, although experienced, screened crowd-workers were used.External validity is limited by Rico-derived descriptions, potentially rare GUI domains, and evaluation with a single high-performing LLM.Source: §6.5 · pp. 161–162

6.6 Limitations

  • RAGG can fall back to baseline prompting when retrieval fails, while false positives and repository gaps limit its effectiveness; SC improves quality but consumes more tokens than baselines.More than 20% of descriptions lacked relevant retrieved GUIs, and SC’s extra token cost could potentially be reduced by regenerating only affected GUI segments.Source: §6.6 · pp. 162–163, §6.5 · p. 162

6.7 Related Work

  • Earlier GUI-generation methods generally produced coarse layouts, image-based prototypes, or DSLs, whereas this approach generated interactive prototypes with greater functional and design detail.The related approaches included transformer and GAN methods, Instigator, Rico-based fine-tuning, MAxPrototyper, and UIDiffuser.Source: §6.7 · p. 163

6.8 Conclusion

  • SC and RAGG mostly outperformed zero-shot baselines for high-fidelity GUI generation on 20,400 expert annotations.Both methods incurred moderate token costs.Source: §6.8 · p. 164
  • RAGG benefited from more retrieved examples and worked best when LLM knowledge and GUI data were complementary.Source: §6.8 · p. 164
  • Self-critique delivered most of its gains after a single iteration, while LLM content generation improved perceived quality, completeness, and satisfaction.These findings support zero-shot prompting as a practical alternative to training or fine-tuning for interactive GUI prototype generation.Source: §6.8 · p. 164

7.1 Motivation

This chapter addresses C1-5 by adapting LLMs to generate editable GUI prototypes from natural-language requirements through two-stage retrieval-augmented generation and Figma integration. It presents an evaluation of an LLM-based prototyping assistant, including component and user-story generation, prototype completion, design quality, user experience, and related limitations and validity threats.

Chapter 7

  • The chapter adapts LLMs to generate fully editable GUI prototypes in proprietary representations that the models were not pretrained on (Kretzer et al., 2025).The work continues challenge C1 after prior retrieval, reranking, and zero-shot prompting approaches for GUI generation.Source: Chapter 7 · p. 165

7.1 Motivation

  • Existing LLM-generated GUI representations are difficult to edit visually and poorly integrated with workflows involving changing requirements, multiple roles, and partially connected tools.The resulting synchronization burden can cause implemented versions to become the de facto latest requirements specification.Source: §7.1 · p. 166
  • The proposed two-stage RAG approach integrates proprietary GUI component libraries, exemplified by Material Design in Figma, for generating editable prototype representations.The method retrieves a compact component summary before supplying full specifications for selected components, icons, and shared attributes, reducing the need to provide the entire library for every generation.Source: §7.1 · p. 167, §7.2.1 · pp. 170–171

7.2.1 LLM-Based Approaches

  • The LLM framework transforms Figma prototypes into compressed representations and supports user-story detection, component matching, user-story generation, contextualized component generation, and rendering back into Figma.It uses zero-shot prompting and automatic specification checks, while generated components can be positioned and integrated into existing prototypes.Source: §7.2.1 · pp. 168, 169, 171

7.2.3 GUI Prototyping Plugin (Treatment)

  • The treatment configuration extends the control assistant with advanced LLM-based automation, whereas the control configuration provides the shared baseline functionality.The comparison is organized around distinct control and treatment features for the subsequent evaluation.Source: §7.2.2 · p. 171, §7.2.3 · p. 173, Table 7.1 · p. 174

7.2.2 GUI Prototyping Plugin (Control)

  • The Figma plugin integrates requirements and prototype work by listing user stories, showing their statuses, synchronizing changes, and supporting requirement-to-component workflows within the prototyping environment.Its design targets UI/UX designers, product owners, and developers, while minimizing the plugin preserves workspace for visual refinement.Source: §7.2.2 · pp. 172–173, Figure 7.2 · p. 172, Figure 7.3 · p. 173

7.2.3 GUI Prototyping Plugin (Treatment)

  • The treatment plugin detects user-story implementation status, matches stories to Figma components, and generates contextually positioned components while preserving user review and control.Users can scan individual stories or all stories in a frame, inspect AI-detected states, manage component associations, preview recommendations, and draw accepted suggestions into Figma.Source: §7.2.3 · pp. 174–175

7.3 Experimental Evaluation

  • The evaluation addressed component-generation effectiveness, assistant effects on prototype quality and story completion, perceived user experience, and generation of user stories from GUI components.The studies used RQ1 through RQ4, including quality annotation, a between-subjects control-treatment study, SUS/NASA-TLX/CSI measures, and crowdworker evaluation of generated stories.Source: §7.3 · p. 176

7.3.2 RQ2, RQ3: GUI Prototyping Assistant Evaluation

  • RQ2 and RQ3 used a controlled between-subjects lab study in which participants created prototypes with control or treatment assistants, followed by Prolific evaluation and self-reported usability and workload measures.The 75-minute lab task involved up to three prototypes, eight stories per prototype, and requirement updates after 30 minutes; prototype evaluation covered functional, visual, interaction, and satisfaction criteria.Source: §7.3.2 · pp. 178, 180

7.3.3 RQ4: User Stories Generation

  • For RQ4, the assistant generated an average of 9.2 unedited user stories for each of 23 recreated Rico GUI prototypes, which Prolific crowdworkers evaluated for accuracy, clarity, specificity, and professional detail.Six crowdworkers with UI/UX experience rated ten prototypes and their corresponding stories each.Source: §7.3.3 · p. 180

7.4 Results

  • The results section reports component-generation findings, lab-study and crowdworker findings for the assistant, and evaluation of automatically generated user stories.The reported analyses cover generated-component fulfillment, prototype quality and story completion, participant self-assessments, assistant usage, and story quality.Source: §7.4 · p. 181

7.3.1 RQ1: GUI Component Generation

  • For RQ1, 102 user stories from 27 GUIs were used to generate components, which six Prolific crowdworkers rated on requirement fulfillment, component completeness and correctness, and textual clarity.Ratings used nine-point Likert scales, with an average of 2.35 ratings per story-component pair.Source: §7.4.1 · p. 181

7.4.1 RQ1: GUI Component Generation

  • 60.82% lower mean total token consumption and 64.35% lower mean input-token consumption were achieved by two-stage RAG while generated GUI components received high ratings.Mean ratings were 7.5 for functional-requirement fulfillment, 7.51 for user-story fulfillment, 7.04 for component correctness, and 6.86 for textual clarity.Source: §7.4.1 · pp. 181–182

7.4.2 RQ2, RQ3: GUI Prototyping Assistant Evaluation

  • In the first app, the treatment group produced higher user-story completion for six stories, with three differences significant, while participants created more apps on average with treatment than control.Treatment participants created 1.75 apps on average versus 1.35 in control; the supplied passage reports the first-app comparison only partially.Source: §7.4.2 · pp. 182–183
  • Participants using the assistant completed more prototypes on average (1.75 vs. 1.35), with higher US-completion ratings and better perceived GUI quality, including significant gains for Consistent Design and Overall Satisfaction.Treatment prototypes also received higher ratings for component and description selection in most cases, although some control prototypes were judged more complete.Source: §7.4.2 · pp. 184–185
  • The assistant improved participants’ perceptions across most evaluation items, significantly increasing confidence in creating professional GUIs, consideration for future use, and perceived design-quality impact.Treatment participants also reported better NASA-TLX outcomes, with significantly higher Own Performance; SUS was 55.50 versus 52.13 and CSI was 56.13 versus 47.63.Source: §7.4.2 · pp. 185–186

7.4.3 RQ4: User Stories Generation

  • Generated user stories received positive ratings for identifying GUI functions, clarity and precision, specificity, and professionalism, with mean scores from 6.02 to 6.70 but substantial variation across source GUI prototypes.The evaluation created 212 user stories from 23 Rico prototypes, averaging 9.22 stories per prototype.Source: §7.4.3 · p. 186

7.5 Discussion

  • The discussion concludes that LLM-generated components effectively supported functional requirements, US fulfillment, and component correctness, while textual clarity remained a principal area for improvement.Suggested refinements include domain-specific prompting or fine-tuning and human-feedback loops; the two-stage RAG approach indicated feasibility for larger workflows where cost is a bottleneck.Source: §7.5.1 · p. 187, §7.5 · p. 187

7.5.2 RQ2, RQ3: LLM-Based Assistant Impact on GUI Quality

  • The assistant improved prototype quality and task progression, but recommendations sometimes favored implicit icon buttons over explicit text labels and lacked sufficient variation.Participants appreciated the reduced work and generally good advice, while requesting adjustable instructions and more diverse recommendations; lower mental and temporal demands were not significant.Source: §7.5.2 · pp. 188–189

7.5.3 RQ4: Effectiveness of Generating US from Components

  • User stories derived from GUI prototypes generally aligned with GUI functionality and purpose, but professionalism was rated slightly lower and quality varied with GUI complexity, ambiguity, and task characteristics.The discussion identifies iterative design workflows as a way to reduce manual effort while leaving higher-level challenges to designers and stakeholders.Source: §7.5.3 · p. 189

7.5.4 Generating GUI Prototypes from USs or Vice-Versa?

  • The assistant can mediate collaboration between product owners and UI/UX designers by generating GUI components from user stories and deriving user stories from design drafts in an iterative refinement process.The authors present automatic GUI creation and user-story creation as complementary collaboration features rather than a strict chicken-and-egg sequence.Source: §7.5.4 · pp. 189–190

7.6 Threats to Validity; 7.7 Limitations; 7.8 Related Work

  • The evaluation is limited by an artificial lab setting with proxy undergraduate and graduate participants, functional user stories tied to single GUI prototypes, static non-interactive prototypes, and incomplete collaboration requirements.Source: §7.6 · pp. 190–191, §7.7 · pp. 191–192

7.1 Motivation

  • A two-stage RAG approach efficiently generates proprietary, editable GUI representations and integrates them with established prototyping tools.The chapter also incorporates LLM-based user-story detection, requirement-to-implementation matching, and user-story generation from prototypes.Source: §7.9 · p. 193
  • Participants using the Figma assistant achieved higher user-story completion, while experts reported significantly higher satisfaction with the resulting GUI prototype designs.The assistant also produced lower task load, higher confidence in creating professionally appearing prototypes, strong interest in future use, and greater perceived impact on design quality.Source: §7.9 · p. 193
  • User stories derived directly from GUI prototypes were often rated specific, explicit, and grounded in identifiable GUI components.The approach was inspired by real-world product-owner requirements and allows those users stories to be refined directly by product owners.Source: §7.9 · p. 193

GUIDE: LLM-Driven GUI Generation Decomposition

GUIDE adapts LLMs to transform high-level natural-language requirements into editable GUI prototypes within Figma by decomposing generation and using retrieval-augmented Material Design components. In a user study, GUIDE significantly improved prototype quality across all nine evaluated metrics, although evaluation was limited to Figma, Material Design, and four prototypes.

8.2.2 GUI Generation Decomposition

  • GUIDE uses a staged pipeline that generates GUI features with zero-shot prompting, retrieves relevant Material Design component types, and then generates validated JSON implementations.A simplified library view reduces consumed tokens by approximately 60%; the selected components’ full specifications are then used for implementation generation and schema validation.Source: §8.2 · p. 197, §8.2.2 · pp. 198–199

8.2.1 Figma Plugin User Interface

  • The Figma plugin lets users review, edit, delete, or add generated features before rendering their implementations as editable GUI components.Users provide short, high-level requirements directly in Figma, while GUIDE displays the derived feature collection for user control.Source: §8.2.1 · pp. 197–198

8.2.3 Prototype Implementation

  • The prototype combines a TypeScript Figma frontend, a proprietary JSON representation shared by the LLM and plugin, and a Python-based Quart REST backend using GPT-4o.The proprietary format supports rendering generated prototypes in Figma, while the backend runs through a scalable gunicorn/nginx server.Source: §8.2.3 · p. 199

8.3 Experimental Evaluation

  • The evaluation used a small between-subjects study in which 11 participants created prototypes with or without GUIDE and 28 Prolific crowdworkers with UI/UX experience rated them.Participants created four prototypes during a 45-minute phase, and crowdworkers assessed requirements fit, component use, text fit, design, organization, interactions, errors, and satisfaction.Source: §8.3 · p. 199, §8.3.1 · pp. 200–201

8.4.1 RQ1: Effectiveness of GUI Prototype Generation

  • GUIDE enabled participants to start more prototypes and achieve higher functional ratings, including requirement satisfaction of 8 (7.393) versus 5 (4.746) and correct-component use of 7 (7.107) versus 5 (4.769).Treatment participants commenced 3.2 prototypes per participant compared with 2.33 for controls, while GUIDE also improved error and design-related metrics.Source: §8.4.1 · pp. 201–202
  • GUIDE significantly outperformed the control group across all nine GUI quality metrics, with higher treatment means and medians for every measure.The study compared prototypes created with and without GUIDE, using crowdworker ratings on 9-point Likert scales; all two-sided Wilcoxon tests were significant.Source: §8.4.1 · pp. 201–202, Figure 8.2 · p. 202

8.5 Threats to Validity

  • Generalizability is limited because the implementation focuses on Figma, Material Design, and individual prototypes rather than broader prototyping environments or entire applications.Future extensions could integrate additional tools and component libraries and support interactions and dependencies across multiple GUIs.Source: §8.5 · p. 202, §8.6 · p. 203

8.1 Motivation

  • GUIDE extends the previous RAG approach by decomposing high-level NLR into low-level GUI features and generating editable prototypes through a Figma plugin using a GUI component library.The approach targets early elicitation stages where fine-grained requirements are often unavailable and supports editable prototype representations in Figma.Source: §8.1 · pp. 196–197, §8.8 · p. 203

Interlinking GUIs and User Stories: A Semi-Automatic LLM-Based Approach

This section studies LLM-based verification of user stories in simplified static GUIs, focusing on detecting implementation and linking requirements to GUI components. It presents a textual GUI abstraction, prompting-based methods, and a human-annotated GUI–user-story dataset to reduce manual verification effort.

9.1 Motivation

  • LLM-based verification addresses whether user stories are implemented in static GUIs and which GUI components realize them, as a feasibility study for more complex verification tasks.The study uses Rico’s structural GUI hierarchy and evaluates zero-shot, few-shot, and chain-of-thought prompting.Source: §9.1 · p. 208
  • The study contributes a novel dataset linking user stories to Rico GUIs and their relevant GUI components through lab-based human annotation.The dataset fills the absence of an existing resource combining GUIs with user stories and component-level matches.Source: §9.1 · p. 209, §9.3.1 · p. 212

9.2 Approach

  • The approach converts Rico GUI hierarchies into textual representations, classifies user-story implementation, and matches each story to its corresponding GUI components.GUI components are organized into layout groups and represented with displayed text, component type, and internal name; a two-tier bullet structure preserves their organization.Source: §9.2 · pp. 209–210, §9.2.1 · p. 210

9.2.2 User Story Implementation Detection

  • Zero-shot, few-shot, and chain-of-thought prompting support implementation detection, while component matching extends the abstraction with identifiers and requires parsable component-ID collections.Zero-shot outputs label probabilities for ranking and certainty, few-shot uses input-output examples, and chain-of-thought generates interpretable reasoning.Source: §9.2.2 · pp. 210–211, §9.2.3 · p. 211

9.3 Experimental Evaluation

  • The evaluation measures implementation detection and component matching with precision, recall, and F1, using a gold standard of 210 user-story–GUI pairs after reserving 21 pairs for few-shot examples.RQ1 forms balanced Implemented and Not-Implemented classes by removing the annotated GUI components from negative instances; RQ2 compares predicted and reference component-ID sets using macro and micro metrics.Source: §9.3 · pp. 211–212, §9.3.2 · pp. 213–214, §9.3.3 · p. 215

9.3.1 Data Collection

  • The final cleaned dataset contains 231 user stories from 59 GUIs across ten domains, with user stories averaging 26.08 words and annotation agreement of Cohen’s κ = .548.Eight participants initially generated 327 user stories; duplicate and invalid entries were removed before the final dataset was formed.Source: §9.3.1 · pp. 212–213

9.4.1 RQ1: User Story Implementation Detection

  • Accuracy reached 85.2 for zero-shot, 84.8 for five-shot, and 82.9 for chain-of-thought prompting at temperature 1.0 in implementation detection.Zero-shot performed best according to mean values, but Holm-corrected pairwise McNemar tests found no statistically significant differences among prompting methods.Source: Table 9.2 · p. 215, §9.4.1 · p. 216
  • False positives arose from semantic misinterpretations, whereas false negatives mainly reflected missing, simplified, or ambiguous component descriptions in the GUI representation.Examples include confusing a city-name button with a complete store address, mistaking a toolbar save icon for marking days complete, and overlooking image-based donation or download controls.Source: §9.4.1 · pp. 216–217, §9.4.2 · p. 218
  • LLM prompting detects user-story implementation in static GUIs with 83.0 precision, 88.6 recall, and 85.7 F1 for zero-shot prompting.No significant pairwise differences were observed among the prompting approaches.Source: §9.4.1 · p. 217

9.4.2 RQ2: User Story GUI Component Matching

  • LLM prompting matches user stories to GUI components with 78.4 Macro precision, 81.9 recall, and 75.5 F1 for ZSA.Micro-F1 scores were 68.1 for ZSA, 65.9 for FS5, and 63.2 for CoTt=0.0; ZS and FS methods did not differ significantly, whereas most CoT methods underperformed them.Source: §9.4.2 · pp. 218–219, Table 9.3 · p. 218

9.5 Threats to Validity

  • Validity is limited by Rico’s simplified, platform-specific GUI representation, lab-created user stories, possible annotation subjectivity, fixed story coverage, and evaluation using a single LLM.Results may vary for other GUI datasets, software platforms, and language models.Source: §9.5 · pp. 219–220, §9.6 · p. 220

9.8 Conclusion

  • The feasibility study confirms that LLM natural-language understanding can identify implemented user stories and match relevant GUI components, supporting automated verification of NLR in static GUIs.The study addresses challenge C2-1 as a foundation for more complex GUI verification tasks.Source: §9.6 · p. 220, §9.8 · p. 221

9.7 Related Work

  • Prior approaches require manually crafted ontologies, brittle GUI-identifier mappings, or effort-demanding modeling, whereas this work uses prompting-based LLM methods without manual fine-tuning.Related work includes Rocha Silva et al. (2019), Mukasa and Kaindl (2008), and Moslehi et al. (2020), whose focus is respectively web-GUI testing, coupled requirements specifications, and source-code traceability.Source: §9.7 · pp. 220–221

GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Applications

GUISpector extends prior feasibility work on static, simplified GUI representations by proposing an MLLM-based computer-use agent for automated natural-language-requirement verification in GUI applications.

GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Applications

  • GUISpector proposes an MLLM-based CUA to automate verification of natural-language requirements in GUI applications, beyond the previous static-representation feasibility study.The source code, interactive prototype, evaluation datasets, and demonstration video are publicly available.Source: GUISpector: An MLLM Agent Framework… · p. 223

C. Agentic Implementation-Verification Loop · Discussion and Conclusion

This section presents GUISpector, an MLLM-based approach for verifying complex natural-language requirements in dynamic GUI applications and supporting autonomous implementation-verification loops for LLM-based programming agents. It discusses its verification effectiveness alongside costs and generalizability limitations.

10.2 Approach: GUISpector

  • The system combines a user-facing verification tool, an MLLM GUI-agent loop, and an MCP interface that enables LLM programming agents to invoke automated verification.Users provide applications, natural-language requirements, and optional test data; the tool organizes statuses, trajectories, evidence, and actionable summaries.Source: Figure 10.1 · p. 224, §10.1 · p. 225, §10.2.1 · p. 226

10.3.1 RQ1: Effectiveness of MLLM CUA for GUI Verification

  • The evaluation used five domain-diverse web applications with 150 requirements and 450 acceptance criteria, whose gold-standard labels showed high inter-annotator agreement.Human annotators agreed on 435 of 450 acceptance criteria, with Cohen’s κ=0.921 and accuracy 96.7%; requirement-level Krippendorff’s α was 0.929.Source: §10.1 · p. 225, §10.3.1 · p. 230

10.2.4 Prototype Implementation

  • GUISpector’s prototype uses a web interface with structured requirement processing, distributed verification tasks, dedicated virtual GUI environments, and simulated parametrized interactions.The implementation combines Django, MySQL, Celery, Redis, Xfce/Xvfb environments, and xdotool, while supporting multiple LLM providers for natural-language processing.Source: §10.2.1 · p. 226, §10.2.4 · p. 229

10.2.2 MLLM GUI Agent Verification Loop

  • GUISpector evaluates requirements by iteratively reasoning over GUI screenshots and trajectories, predicting parametrized actions, executing them in dedicated environments, and collecting evidence for each acceptance criterion.It outputs JSON summaries with acceptance-criterion statuses and evidence, while cleaning environments before and after runs to avoid contamination.Source: §10.2.2 · pp. 227–228

10.2.3 Agentic Implementation-Verification Loop

  • The MCP-based agentic loop lets programming agents retrieve requirements, patch violated implementations, rerun selected verifications, and continue iterating until requirements are met or the next requirement is addressed.A zero-shot prompt supports autonomous implementation-verification cycles inside LLM-powered IDEs or general programming agents.Source: §10.2.3 · pp. 228–229

10.4.1 RQ1: Effectiveness of MLLM CUA GUI Verification

  • AC-level verification achieved F1=94.0 for met and F1=87.0 for unmet classes, while requirements-level F1 scores were 90.5, 87.8, and 63.1 for met, unmet, and partially met.The partially met class was most error-prone because semantic boundary confusions made it harder to distinguish mixed fulfillment.Source: §10.4.1 · pp. 231–234
  • Performance varied across applications: Budget Tracker was best, with met F1=97.6 and unmet F1=93.9, whereas Recipe Generator was worst, with met F1=91.5 and unmet F1=82.1.These results indicate application-dependent variation despite high overall effectiveness.Source: §10.4.1 · p. 232
  • Trajectory analysis found click and scroll were the most frequent actions, while drag, double click, and move were reserved for special interaction cases.Wait, type, and keypress actions also occurred, reflecting rendering delays and custom test-data entry.Source: §10.4.1 · pp. 233–234, Figure 10.8 · p. 234

10.4.1 RQ1: Effectiveness of MLLM CUA for GUI Verification

  • The study evaluated MLLM CUA effectiveness for GUI verification using trajectories containing screenshots, optional reasoning, and parametrized GUI actions.The example trajectory verifies a requirement and three acceptance criteria for marking recipe ingredients as “Use Soon.”Source: Figure 10.9 · p. 235

10.4.2 RQ2: Cost for MLLM-based CUA GUI Verification

10.5 Threats to Validity

  • The evaluation used GUI applications spanning five domains, but Codex-generated requirements and implementations and the limited application set may restrict generalizability.Authors independently reviewed requirements and annotations, reporting high annotation quality and inter-annotator agreement.Source: §10.5 · p. 237

10.6 Limitations

  • MLLM-based CUA verification can take several minutes and incur substantial API costs, although parallelization and merging shared sub-trajectories could improve efficiency.The study focused on effectiveness rather than optimization; external MLLM memory was also identified as a possible efficiency improvement.Source: §10.6 · p. 237

10.7 Related Work

  • Unlike prior GUI testing methods focused mainly on runtime failures or textual GUI abstractions, GUISpector verifies natural-language requirements including visual and dynamic aspects.The approach is presented as the first fully autonomous GUI implementation-verification loop for LLM-based programming agents using an MCP server.Source: §10.7 · pp. 237–238

10.8 Conclusion

  • GUISpector achieved high verification accuracy for acceptance-criterion and requirement-level tasks while enabling autonomous implementation-verification loops through an MCP server.The approach processes complex natural-language requirements in fully-fledged, highly dynamic GUI applications.Source: §10.8 · p. 238

Discussion … Appendix for Chapter 3

The discussion weighs retrieval, generation, human collaboration, cost, evaluation subjectivity, and hallucination mitigation, while the conclusion summarizes contributions to GUI prototyping and verification and the appendices document published resources.

11.1 GUI Retrieval and GUI Generation Trade-offs

  • Retrieval offers fast, low-cost comparison within covered GUI repositories, whereas LLM generation better adapts to uncommon or customized requirements; hybrid strategies can combine their strengths.RAGG incorporates external GUI knowledge, and retrieval can identify prototypes that an LLM subsequently adapts.Source: §11.1 · pp. 241–242

11.2 LLM-based Approaches Augment Human Capabilities

  • LLM-based prototyping is designed to reduce manual labor while preserving human control because generated outputs can still contain errors.The approaches integrate LLMs into existing workflows and support human editing of generated prototypes.Source: §11.2 · p. 242

11.3 Cost-Effectiveness of LLM-based Approaches

  • LLM approaches achieve substantial effectiveness gains but incur high runtime and monetary costs, motivating research into task-specific Small Language Models.Proprietary paid APIs are a major source of cost, especially for verification.Source: §11.3 · p. 242

11.4 Evaluation Challenges for GUI Prototyping Approaches

  • Because GUI relevance and quality are partly subjective, the work combines relevance metrics, user studies, Likert ratings, and many annotators to evaluate prototypes.Objective measures, such as component counts, are complemented by subjective judgments gathered through high-quality crowdsourcing.Source: §11.4 · p. 242

11.5 Hallucinations in LLM-based Approaches

  • Hallucination risks are reduced through artifact grounding, automatic validation, and human-in-the-loop verification, while remaining errors often reflect ambiguity or underspecified representations.For static GUI verification, higher temperatures caused some CoT matching outputs to reach F1=0, whereas other evaluation cases showed no hallucinations.Source: §11.5 · pp. 243–244

Conclusion

  • The thesis addresses automated NLR-based GUI prototyping and verification through BERT retrieval, MLLM reranking, LLM generation, and MLLM-based CUA verification.The conclusion identifies seven sub-challenges spanning retrieval, self-elicitation, editable prototype generation, and static and dynamic GUI verification.Source: Conclusion · pp. 245, 248
  • The proposed methods improve retrieval, self-elicitation, GUI generation, editable prototyping, and verification, including evaluation on over 20,400 GUI annotations from 101 UI/UX experts.Reported results include improved reranking over BERT-LTR, high perceived SERGUI usability, substantial prototyping improvements, and high verification scores at requirements and AC levels.Source: Conclusion · pp. 246–248

Published Resources; A. PUBLISHED RESOURCES

  • Published resources include code, datasets, experimental results, reusable prototypes, and accompanying approach-demo videos.The listed artifacts include Interlinking Code+ Dataset, GUISpector Code+ Dataset, their repositories, and videos documenting the research tools.Source: Published Resources · p. 275

B.1 Experimental Details · Appendix C · Appendix for Chapter 4

The appendices document statistical procedures for evaluating NLR-based GUI retrieval and comparing RaWi with Mockplus on productivity metrics. They use Holm-corrected one-sided Wilcoxon tests and linear mixed-effects models with repeated-measures controls.

B.1 Experimental Details

  • The retrieval evaluation reports Holm-corrected Wilcoxon signed-rank tests for AP, MRR, and HITS@k comparisons.The tests assess whether Model 1 outperforms Model 2 at α = 0.05.Source: Table B.1 · p. 277
  • Additional retrieval analyses apply the same Holm-corrected Wilcoxon framework to precision at rank k (P@k).The comparisons test whether Model 1 is greater than Model 2, with significance defined at α = 0.05.Source: Table B.2 · p. 278
  • NDCG@k retrieval comparisons are likewise evaluated with Holm-corrected Wilcoxon signed-rank tests.The tables assess directional Model 1-versus-Model 2 differences at α = 0.05.Source: Table B.3 · p. 278
  • RaWi and Mockplus are compared on paired per-minute #GUI-comps task data using one-sided Wilcoxon tests with Holm correction.The analyses report means, medians, their differences, and Cliff’s Delta at α = 0.05.Source: Table B.4 · p. 279
  • The paired RaWi–Mockplus analysis is extended to #GUI-comps-div and to #GUI-comps adjusted by #GUI-comps-neg.These tables report Holm-corrected comparisons, while the adjusted-metric table also includes raw p-values.Source: Table B.5 · p. 279, Table B.6 · p. 279
  • Linear mixed-effects models analyze #GUI-comps productivity with approach, time, task, GUI, condition, and an approach-by-time interaction as fixed effects.Participants are modeled with random intercepts for repeated measurements; RaWi, task 1, GUI 1, and condition 1 are reference levels, and time is centered at minute 3.Source: Table B.7 · p. 280, §B.1 · p. 280
  • Parallel mixed-effects models use the same fixed-effect structure for #GUI-comps-div and #GUI-comps-neg-div productivity metrics.They report coefficients, standard errors, Wald z statistics, p-values, and 95% confidence intervals, with Mockplus represented as Approach B.Source: Table B.8 · p. 280, §B.1 · pp. 280–281, Table B.9 · p. 281

C.1 Experimental Details … Appendix for Chapter 6

The appendices report Holm-corrected Wilcoxon signed-rank tests comparing models across retrieval metrics, with significance assessed after correction. They cover AP, MRR, HITS@k, P@k, and NDCG@k, using standardized model abbreviations.

C.1 Experimental Details

  • The C.1 appendix tests whether one model outperforms another using Holm-corrected Wilcoxon signed-rank tests on AP, MRR, and HITS@k.Significance is marked at α = 0.05 after Holm correction.Source: Table C.1 · p. 282

C. APPENDIX FOR CHAPTER 4

D.1 Experimental Details

This section reports Friedman tests comparing zero-shot prompting models across evaluation metrics on both the full and cleaned GUI datasets.

Full dataset

  • Friedman tests assess overall metric differences among zero-shot prompting baselines PD and SC on the full dataset of 50 GUIs.The reported p-values indicate whether at least one model differs significantly from the others for each metric.Source: Table D.1 · p. 284

Cleaned dataset

  • A second Friedman test compares zero-shot prompting models, including RAGG, on the cleaned dataset of 15 GUIs.The p-values likewise indicate whether at least one model differs significantly from the others across evaluation metrics.Source: Table D.2 · p. 284

D. APPENDIX FOR CHAPTER 6 · Appendix E · Appendix for Chapter 9

The appendices report Holm-corrected or paired Wilcoxon signed-rank tests assessing GUI-generation methods across datasets, retrieval-example counts, loop counts, and content generation.

D. APPENDIX FOR CHAPTER 6

  • Table D.3 reports Holm-corrected paired two-sided Wilcoxon tests across eight GUI-quality metrics on the full 50-GUI dataset.The compared methods include Feature Completion, Feature Extensiveness, Feature Implementation, Information Organization, Visual Appeal, Minimal Errors, Overall Satisfaction, and Complete App; significance uses α = 0.05 after Holm correction.Source: Table D.3 · p. 285

Appendix E · Appendix for Chapter 9

  • Table D.3 reports two-sided Holm-corrected p-values from paired post-hoc Wilcoxon signed-rank tests conducted on the full dataset comprising 50 GUIs.Source: Table D.3 · p. 285

D. APPENDIX FOR CHAPTER 6

  • The cleaned 15-GUI dataset is evaluated with Holm-corrected paired two-sided Wilcoxon post-hoc tests across the same eight metrics.The appendix identifies PD1, PD2, SC (k = 4), and RAGG (k = 7) among the compared approaches, with significance at α = 0.05 after Holm correction.Source: D. APPENDIX FOR CHAPTER 6 · p. 285, Table D.4 · p. 285
  • Table D.5 compares RAGG across retrieved-example settings k = 1, 3, 5, and 7 using Holm-corrected paired two-sided Wilcoxon tests.The tests cover eight GUI-quality metrics, and bold asterisks denote significance at α = 0.05 after Holm correction.Source: D. APPENDIX FOR CHAPTER 6 · p. 285, Table D.5 · p. 285
  • Table D.6 compares SC loop counts k = 0, 1, 2, 3, and 4 with paired two-sided Wilcoxon tests across eight GUI-quality metrics.Here, k = 0 is the underlying zero-shot baseline used as SC’s base prototype input, and Holm correction is applied within each metric across pairwise comparisons.Source: Table D.6 · p. 286
  • Prompting techniques are compared with and without content generation using paired Wilcoxon signed-rank tests across eight GUI-quality metrics.The techniques are ZS-CoT, PD-ZS, RAGG, and SC; RAGG and SC both use k = 3, and significance is assessed at α = 0.05.Source: D. APPENDIX FOR CHAPTER 6 · p. 286, Table D.7 · p. 286

E.1 Experimental Details · E. APPENDIX FOR CHAPTER 9

The appendix reports statistical procedures for comparing models on US-detection correctness and per-US F1 scores. It includes Holm-corrected McNemar and one-sided Wilcoxon signed-rank tests, with zero-shot, few-shot, and chain-of-thought abbreviations defined.

E.1 Experimental Details

  • Table E.1 reports Holm-corrected p-values from McNemar tests with continuity correction.Source: Table E.1 · p. 287
  • The McNemar analysis tests paired correctness for US detection, particularly whether Model 1 differs from Model 2.Source: §E.1 · p. 287
  • The comparison records n10, n01, and their sum, Disc., for discordant model predictions.Source: §E.1 · p. 287
  • The appendix defines ZS, FS, and CoT as zero-shot, few-shot, and chain-of-thought settings.Source: §E.1 · p. 287

E. APPENDIX FOR CHAPTER 9

  • Table E.2 reports Holm-corrected p-values from pairwise one-sided Wilcoxon signed-rank tests on paired per-US F1 scores.Source: Table E.2 · p. 288
  • The Wilcoxon tests assess whether Model 1 has greater F1 than Model 2 using ΔF1 = F1_1 − F1_2.Source: Table E.2 · p. 288
  • Bold values with an asterisk indicate significance at α = 0.05 after Holm correction.Source: Table E.2 · p. 288

Related papers

Loading 2608.24749v1…