Towards Human-AI Synergy in UI Design: Supporting Iterative Generation with LLMs

1 Introduction

• Existing generative UI methods often produce unstructured, non-editable outputs, requiring manual reconstruction and limiting customization.Small elements such as icons are also difficult for these models to handle well.Source: §1 · p. 2
• AI support for continuous designer collaboration remains underexplored, including human control, transparency, and fine-grained automation.Source: §1 · p. 2
• Interviews with 10 professional UI/UX designers identified five needs spanning workflow knowledge, input control, prompt-based intent expression, generation control, and thematic consistency.Source: §1 · p. 2
• PrototypeFlow accepts textual requirements and wireframe layouts, then uses a theme module, prompt enhancement, and coordinated specialized modules to generate prototypes.Its modules cover text, images, and retrieval-based icons while integrating component details through a cache pool.Source: §1 · p. 2
• Global theme edits regenerate the prototype, whereas component-level edits trigger localized regeneration for precise iterative refinement.The system balances automated generation with human customization and supports SVG or JSON export.Source: §1 · p. 3
• Automated evaluations, ablations, and three user studies with 16 participants supported PrototypeFlow’s effectiveness and positive reception.The reported findings indicate potential to address the five identified challenges and align GenAI systems with designers’ workflows.Source: §1 · p. 3

2.1 GUI Design Tools and Techniques

• GUI design tools provide component libraries, templates, and high-fidelity prototyping, while retrieval methods connect sketches, wireframes, screenshots, and language to existing designs.Source: §2.1 · pp. 3–4
• Retrieval-based approaches depend on existing designs and leave substantial manual refinement, limiting creative flexibility and customization.Source: §2.1 · p. 4
• Generative methods expand design possibilities but often produce unstructured, non-editable outputs or low-fidelity layouts with limited detailed refinement.Source: §2.1 · p. 4
• Most AI design systems provide either language or image inputs, whereas PrototypeFlow combines wireframes and prompts for precise layout and functional-intent specification.It produces editable SVG prototypes that designers can directly manipulate and iterate.Source: §2.1 · p. 4

2.2 Human-AI Interaction in GUI

• Human-AI GUI research has explored natural-language interaction, mutual disambiguation, and automated tasks, but interactive design generation remains unreliable, unclear, and underexplored.Source: §2.2 · p. 4

2.3 Large Language Models for GUI Generation

• Prior GUI-generation work emphasizes static prompts or one-off variants, with limited coherence, iterative refinement, and transparent workflows.PrototypeFlow uses decoupled generation with editable checkpoints and LLM-based control for refinement and clarification.Source: §2.2 · p. 4, §2.3 · p. 5
• PrototypeFlow enhances prompts and generates editable themes to provide theme-level control over color, style, usage, and layout consistency.Source: §2.3 · p. 5

3 Formative Study and Findings

• The study addressed a gap in actionable guidance for collaborative, fine-grained iterative refinement in real-world professional contexts.Source: §3 · p. 5
• The findings aim to balance accuracy, creative flexibility, and seamless integration with existing workflows.Source: §3 · p. 5
• The study interviewed 10 UI/UX designers from five companies using recorded online interviews and thematic analysis.Participants varied in company size, experience, and role context; coding produced 96 in-vivo codings.Source: §3 · p. 5

3.1 Findings

• The study produced five primary actionable requirements for generative design tools.Source: §3 · p. 5, §3.1 · p. 6
• Participants used established design platforms alongside generative tools, but beta status, instability, and subscription costs limited routine adoption of newer tools.Source: §3.1 · p. 6
• The analysis found that current design and generative tools offer creative value while retaining important input and output limitations.Source: §3.1 · p. 6
• The requirements motivate prompt enhancement, editable checkpoints, and editable theme generation with downstream control.Source: §3.1 · p. 6

F1. Need for Streamlined Design Workflow Supported by Trend- and Brand-Aligned Knowledge

• Designers combine online searches, Midjourney inspiration, and internal guidelines, creating manual coordination between trendiness, compliance, and diversity.Source: F1. Need for Streamlined Design… · pp. 6–7
• Participants wanted scalable tools that reflect current trends and organizational styles while integrating discovered designs into their workflows.Source: F1. Need for Streamlined Design… · p. 7
• PrototypeFlow responds with retrieval-augmented generation whose updateable knowledge base can encode evolving design trends and company-specific requirements.Source: F1. Need for Streamlined Design… · p. 7

Input Control/Modality

• Designers reported that existing tools provide limited input control and often fail to capture precise layout or functional interactivity.Text-only prompts can ignore layout requirements, while image-only input may not communicate component functionality.Source: F2. Need for More Input Control and… · p. 7

Output Editability/Modality

• Editable SVG prototypes let designers directly manipulate visual properties and iterate more efficiently than code or static-image outputs.Code-based and static-image outputs are difficult to refine, especially for designers without programming experience.Source: F2. Need for More Input Control and… · p. 7

Input Control/Modality

• Combining wireframes with natural-language prompts gives designers finer control over pixel-level layout and component functionality.Participants found text-only or image-only input insufficient for fully expressing design intent.Source: F2. Need for More Input Control and… · p. 7

Output Editability/Modality

• PrototypeFlow supports multimodal inputs and editable SVG outputs to keep design transformation accurate, adaptable, and iterative.The system combines wireframes with natural-language prompts and generates prototypes that designers can directly edit.Source: F2. Need for More Input Control and… · p. 7

Need for Supporting Designers in Expressing Intent Through Prompts

• Designers frequently provide high-level prompts lacking theme colors, categories, and other visual details needed for effective design.Participants attributed this to unfamiliarity with prompt structures, uncertainty about AI requirements, and the time required to write detailed prompts.Source: F3. Need for Supporting Designers in… · p. 7
• Participants wanted AI to actively clarify and refine initial prompts so they could express design goals more quickly and accurately.This reflects a need for collaborative intent clarification between designers and AI.Source: F3. Need for Supporting Designers in… · pp. 7–8
• PrototypeFlow enhances prompts by suggesting theme settings, colors, and other design properties that help designers specify detailed requirements.The prompt-enhancement step addresses a capability identified as lacking in existing AI generation tools.Source: F3. Need for Supporting Designers in… · pp. 7–8
• The prompt-enhancement step makes the design process faster and more accurate.The system helps designers quickly specify detailed requirements.Source: F3. Need for Supporting Designers in… · p. 8

Need for Precise Control in Generation Processes

• Existing end-to-end tools provide limited transparency, making designers rely on trial and error when outputs do not match their intent.Designers lack insight into AI reasoning and step-by-step generation processes.Source: F4. Need for Precise Control in… · p. 8
• Designers called for transparent mechanisms that expose and allow refinement of generation steps and intermediate results.Participants specifically wanted to review, edit, and tailor the reasoning process rather than rely only on final outputs.Source: F4. Need for Precise Control in… · p. 8
• PrototypeFlow introduces transparent, editable checkpoints throughout generation so designers can review and refine intermediate results.This directly addresses the lack of editable reasoning and intermediate-generation control in existing tools.Source: F4. Need for Precise Control in… · p. 8
• Natural language functions as a domain-specific language for global and local specifications, enabling precise control over style and components.The approach supports collaborative refinement and more accurate outcomes.Source: F4. Need for Precise Control in… · p. 8

Need for Maintaining Thematic Consistency and Coherence Across Generated Components

• AI-generated pages may contain components that deviate from the intended theme, such as a green component within an otherwise blue design.The problem becomes more pronounced for complex or lengthy pages because of model hallucinations or forgetting.Source: F5. Need for Maintaining Thematic… · p. 8
• Designers must manually correct inconsistent components because existing tools lack editable theme generation with downstream control.Participants described maintaining a unified look as repetitive and time-consuming.Source: F5. Need for Maintaining Thematic… · p. 8
• PrototypeFlow employs an LLM-based controller for editable theme generation with downstream control across generated components.The controller presents natural-language descriptions governing design coherence for designer review and refinement.Source: F5. Need for Maintaining Thematic… · p. 8
• The controller maintains consistency in color, style, and layout while allowing designers to refine coherence throughout the prototype.This targets the repetitive manual effort required to edit downstream components.Source: F5. Need for Maintaining Thematic… · p. 8

4.1 Overview

• PrototypeFlow decouples UI generation into centrally coordinated modules, enabling explainable prototypes and iterative Human-AI collaboration aligned with design intent.The Theme Design Module coordinates text, image, and icon generation before assembling the complete prototype; designers can refine generated outputs.Source: §4.1 · pp. 8–9

4.2 Knowledge Base Construction

• PrototypeFlow constructs UI and icon knowledge bases to ground generation in domain-specific design practices.The UI knowledge base pairs layout and theme descriptions with local component descriptions, while the icon knowledge base pairs SVG code with semantic descriptions [71].Source: §4.2 · p. 9

4.2.1 UI Knowledge Base.

• The UI knowledge base combines Rico, Screen2Words, and Blip2 to represent component layouts, textual semantics, high-level descriptions, and theme attributes.Rico provides about 22k UIs and composition metadata; Screen2Words adds 112k descriptions for 22k screenshots, while Blip2 generates theme descriptions through visual question answering.Source: §4.2.1 · p. 9

4.2.2 Icon Knowledge Base.

• The icon knowledge base uses over 900 Google Material Design Icons represented as editable SVG code paired with semantic descriptions.SVG representation preserves clarity when scaled and allows designers to modify icon colour, style, and shape.Source: §4.2.2 · p. 10

4.3 Theme Design Module

• The Theme Design Module clarifies implicit intent through retrieval, theme-description generation, theme-image generation, and coordinated sub-module execution.It uses domain knowledge and designer collaboration to establish global style, maintain cross-element coherence, and refine component details through a cache pool.Source: §4.3 · p. 10

4.3.1 Knowledge Retrieval.

• Knowledge retrieval embeds the concatenated design prompt and bounding-box layout, compares them with UI knowledge using cosine distance, and retrieves top-k references, with k=2.Preliminary experiments and Wang et al. suggest the first few-shot example is most influential, while later examples provide diminishing returns.Source: §4.3.1 · p. 11

4.3.2 Theme Description Generation.

• Theme Description Generation uses natural-language descriptions as a domain-specific language to clarify implicit intent and bridge designer intent with generated results.Given a design prompt and UI wireframe, the system forms a structured theme-description prompt for coherent design narratives.Source: §4.3.2 · p. 11
• The theme description generator concatenates the input prompt, wireframe layout, and retrieved knowledge items to produce a theme description for downstream design generation.Figure 5 illustrates these inputs and the resulting theme-description prompt, while the theme image subsequently guides overall design and intent alignment.Source: §4.3.2 · pp. 11–12, §4.3.3 · p. 12

4.3.3 Theme Image Generation.

• Theme image generation combines Stable Diffusion with ControlNet to use UI layouts as spatial conditions for coherent theme images.The module takes a text prompt, Gaussian-noise latent image, and UI layout; the model was fine-tuned on UI screenshots and high-level descriptions because general-image training provides insufficient UI-domain knowledge.Source: §4.3.3 · p. 12

4.3.4 Sub-module Execution.

• Sub-module execution selects among 13 Rico component types and uses a central cache to preserve prior results and accumulated knowledge across multi-round interactions.The cache informs prompts with previous outputs and historical context, supporting consistency, intent alignment, and coherent multi-conversation interaction.Source: §4.3.4 · pp. 12–13

4.4 Text Content Module

• The Text Content Module uses GPT-4 to generate position-specific textual recommendations from the theme description and cached context.Its execution prompt combines the module instruction with the central module’s cache to keep generated text consistent with the GUI theme.Source: §4.4 · p. 13

4.5 Image Content Module

• The Image Content Module generates local image-associated content with the fine-tuned Stable Diffusion model, combining theme-image regions with textual descriptions.It disables ControlNet, uses the relevant theme-image area instead of Gaussian noise, and uses the generated theme description as the prompt to align content with the primary theme intent.Source: §4.5 · p. 13

4.6 Icon Module

• The Icon Module uses cached context and theme descriptions to derive semantic phrases, then retrieves matching icon SVG code from a knowledge base.This process accounts for icon positioning and is intended to ensure semantic and visual matches with the GUI design.Source: §4.6 · p. 13

5 Implementation

• Designers enter prompts, select or configure UI elements, adjust layout size and positioning, and preview layouts and generated designs in the editor.The interface includes design-description input, component selection, drag-and-drop resizing or repositioning, layout creation or loading, and real-time preview.Source: §5 · p. 13
• PrototypeFlow displays generated theme descriptions and theme images before prototype generation, then presents component explanations alongside the detailed prototype.These intermediate displays are intended to improve transparency and explainability during Human-AI collaboration.Source: §5 · p. 14
• Generated prototypes can be saved as JSON files for further use after designers review and refine them.The implementation provides a dedicated “Save Prototype” action for exporting the prototype.Source: §5 · p. 14
• Designers can edit every part of the system, refine intermediate and final outputs, visualize changes in real time, and apply grouped text updates.The editor updates the design through an “Update Edit Result” action and supports ongoing refinement after multi-module generation.Source: §5 · p. 14
• The implementation uses gpt-4 for textual generation, text-embedding-ada-002 for embeddings, and a fine-tuned Stable Diffusion checkpoint for image generation.The Stable Diffusion model is trained using Rico UI data divided into 15,743 training, 2,364 validation, and 4,310 testing UIs.Source: §5 · p. 14

6.1 Evaluation Data

• The evaluation uses 3,738 UI descriptions with corresponding wireframes and high-fidelity screenshots, treating screenshots as ground truth on the test set.Images are resized to 512 × 512.Source: §6.1 · p. 14

6.3.2 Results.

• PrototypeFlow outperforms both baselines in quality and diversity, achieving FID 23.76 and GD 13.98 with ControlNet versus FID 33.08 and GD 15.95 without it.Lower FID indicates greater realism, while higher GD indicates greater diversity; both PrototypeFlow variants surpass the baselines on GD.Source: Table 3 · p. 15, §6.3.2 · p. 16

6.2 Metrics

• FID measures similarity between generated and real-image distributions, whereas GD measures low-level visual diversity; lower FID and higher GD are preferred.Both metrics use Inception-v3 features, while GD computes pairwise distances within generated UI designs.Source: §6.2 · p. 15

6.3.1 Baselines.

• RQ1 compares stable-diffusion-1-5 and stable-diffusion-2-1 with and without ControlNet, plus PrototypeFlow without ControlNet.The baselines do not originally incorporate ControlNet, so ControlNet variants are evaluated separately.Source: §6.3.1 · p. 15

6.3.2 Results.

• ControlNet improves PrototypeFlow’s FID by 10.68, or 32%, but reduces GD from 15.95 to 13.98 as layout constraints limit diversity.This trade-off reflects improved realism alongside slightly less generative variety.Source: §6.3.2 · p. 16

6.4.1 Baselines and Ablation Strategy.

• The ablation study sequentially removes six modules across knowledge retrieval, theme description generation, theme image generation, and three content modules.This design measures individual module contributions and robustness across scenarios.Source: §6.4.1 · p. 16, §6.4.2 · p. 16

6.4.2 Results.

• Removing knowledge retrieval raises FID from 23.76 to 42.56, the largest quality decline, while increasing theme-color and component-selection randomness.That randomness can help early ideation but conflicts with established themes and functional requirements, increasing manual editing.Source: §6.4.2 · p. 16
• Removing theme description generation yields FID 28.43 and GD 11.77, while removing theme image generation raises FID to 33.08 and causes visual inconsistencies.Theme description removal preserves the primary palette but introduces localized component color disharmonies; theme image generation connects textual themes to coherent visuals.Source: §6.4.2 · pp. 16–18

6.4.2 Results

• Theme Image Generation is important for consistent, accessible, and visually compelling interfaces because textual cues alone can preserve global color while causing local contrast and visibility problems.Without the image module, a white icon became invisible against a white search bar, and other elements lacked sufficient contrast.Source: §6.4.2 · p. 18
• Removing individual sub-modules produced FID scores of 24.06(+0.3), 24.71(+0.95), and 24.32(+0.56), demonstrating their collective contribution to peak performance.The Text, Image, and Icon modules specialize in different component types, supporting the fine-grained generation approach.Source: §6.4.2 · p. 18
• Ablating the Text, Image, or Icon Content Modules caused the corresponding text, image, or icon components to disappear in Fig. 8.Green boxes 1–24 identify missing components relative to full PrototypeFlow output.Source: §6.4.2 · p. 18
• Every PrototypeFlow module contributes to design quality and diversity, with Knowledge Retrieval and Theme Image Generation most influential while other modules jointly refine the final output.The findings confirm a decoupling strategy in which specialized modules handle distinct components and collaborate to enrich the design.Source: §6.4.2 · p. 18

7.1.1 Procedure.

• The performance study rated mobile GUI designs on functional semantics, design aesthetics, and color harmony using a 5-point Likert scale across 20 Screen2Word tasks.Three prototypes per task were generated with PrototypeFlow and stable diffusion, and participants evaluated designs with the source model concealed.Source: §7.1 · p. 19, §7.1.1 · p. 19

7.1.2 Results & Discussion.

• PrototypeFlow outperformed other methods in design-prototype consistency (Mean=4.13), design aesthetics (Mean=3.83), and colour harmony (Mean=3.62).Its semantic-alignment score was 4.32, significantly higher than stable-diffusion-1-5 (1.58) and stable-diffusion-2-1 (1.7).Source: §7.1.2 · p. 19
• Performance analysis linked high scores to clean layouts, moderately rich content, compatible images, and accurate semantic alignment from decoupled generation.Low-scoring designs showed incomplete structures, basic content, abrupt or small images, and excessive components.Source: §7.1.2 · p. 19
• The performance study also found minor typography errors from over-generated text, which participants suggested addressing through better alignment with typography and design guidelines.PrototypeFlow’s outputs were judged to align closely with real-world screenshots and sometimes exceed poorly designed real prototypes aesthetically.Source: §7.1.2 · p. 19

7.2.1 Procedure.

• The usage study compared PrototypeFlow with Vercel’s V0 and Uizard through prototype-creation tasks, five Likert-scale questions, and expert feedback on five design goals.Participants created two prototypes per tool from provided purposes and wireframes, could modify input descriptions, and had manual modifications recorded.Source: §7.2 · pp. 19–20, §7.2.1 · pp. 20–21

7.2.2 Results & Discussion.

• PrototypeFlow significantly outperformed Vercel’s V0 across all five survey dimensions (p < 0.001) and surpassed Uizard on F2, F3, and F4 (all p < 0.001).It performed similarly to Uizard on thematic consistency (F5; p = 0.071) and slightly lower on retrieval relevance (F1: 3.5 versus 3.6, p = 0.4).Source: §7.2.2 · p. 22
• Usage feedback indicated strengths in detailed element generation, input flexibility, and design-library integration, while Uizard was viewed by some participants as more visually appealing.Nine of sixteen participants praised detailed elements, seven preferred Uizard’s overall style, and four valued importing and integrating personal data.Source: §7.2.2 · p. 22
• PrototypeFlow led Uizard and Vercel V0 in reflecting design intent (4.1 vs. 3.6 and 2.5), prompt enhancement (4.0 vs. 3.2 and 1.6), and human-in-the-loop interaction over Uizard (3.8 vs. 3.5).Designers using PrototypeFlow made 2.6 revisions on average, compared with 4.3 for Uizard and 6.1 for Vercel’s V0. 14/16 participants found its clarification ability particularly helpful.Source: §7.2.2 · p. 22
• PrototypeFlow more effectively supported UI prototyping by communicating design intent, exposing explainable intermediate results, and balancing automation with customization.Participants praised its presentation of components and vital generation information for precision, convenience, and workflow support.Source: §7.2.2 · p. 23

7.2.3 Weaknesses and improvements.

• Participants proposed automated wireframe generation, simpler vector graphics for small areas, and stronger adherence to UI design and accessibility guidelines.These improvements target reduced manual effort, better image functionality, and prototypes that are both visually appealing and interactive.Source: §7.2.3 · p. 23

7.3 User Study 3: Case Study

• The case study compared Input-NoLayout, Input-NoKnowledge, Output-MultiLayout, and Output-MultiTheme using idealized Google self-help search-page variants evaluated by the same 16 designers.Researchers conducted interviews, open-coded responses, and independently coded a subset to analyze perceived usefulness, strengths, limitations, and trade-offs.Source: §7.3 · p. 23, §7.3.1 · p. 23

7.3.1 Experimental Setup.

• During ideation, 13/15 designers preferred Input–NoKnowledge for broader suggestions, while designers generally preferred wireframes for precise layout control and workflow compatibility.Output–MultiLayout and Output–MultiTheme mainly helped less-experienced designers learn through comparison, but were less applicable to real-world team workflows.Source: §7.3.2 · p. 24
• Most experienced designers (11 out of 15) said teams typically align on functionality before generation, making additional alternatives potentially distracting and inefficient.They preferred front-loading diversity during early ideation and viewed post-generation variation as disruptive to streamlined workflows and delivery timelines.Source: §7.3.2 · p. 24
• Input-level variants like Input–NoKnowledge suited experienced designers during early ideation, whereas output-level variants helped novices compare alternatives and learn from variation.Output–MultiLayout and Output–MultiTheme were less useful in team settings where decisions were finalized early.Source: §7.3.2 · p. 24
• The findings suggest systems should dynamically balance control and flexibility across users’ experience levels and design stages.The preferred form and timing of diversity varied with workflow context, including early ideation and team-based decision finalization.Source: §7.3.2 · p. 24

8.1 Tailoring Interaction Modalities to User Expertise

• Senior designers preferred wireframes and visual layouts as a precise, native design language, while future systems should adapt textual, visual, conversational, or guided interaction to novices’ skills.Adaptive interfaces could begin with Q&A or templates for novices and introduce wireframing as proficiency grows, lowering barriers to design participation.Source: §8.1 · pp. 24, 26

8.2 Balancing Fine-Grained Control and Output Diversity in Generative Design

• Designers wanted diversity during early ideation but precision during later creation, indicating that GenAI tools should support structured transitions between divergent exploration and convergent refinement.Thirteen of 15 designers preferred disabling the internal knowledge base during early ideation, whereas unconstrained layouts produced outputs that were too difficult to implement during creation.Source: §8.2 · p. 26

8.3 The Limits of Static Knowledge Bases in Rapidly Evolving Design Contexts

• Static or narrow knowledge bases can produce outdated, repetitive, and domain-limited suggestions, so future systems need automated updating, broader coverage, and domain adaptation.Knowledge retrieval supports realistic, brand-consistent outputs, but overreliance can restrict early exploration; sustained usefulness also depends on learning from ongoing user feedback.Source: §8.3 · pp. 26–27

8.4 Integrating GUI and Web Design Practices for Broader Design Knowledge

• Extending GenAI design tools from mobile and desktop GUI screens to web design requires accounting for fluid layouts, device and browser variation, network conditions, and accessibility needs.Web interfaces provide less predictable control than native applications, exposing limitations in systems trained mainly on mobile screens or static interfaces.Source: §8.4 · p. 27
• GenAI design systems should recognize the intended medium and apply context-suited patterns, such as responsive grids for websites and native components for mobile apps.Website layouts should adapt from mobile to desktop, whereas mobile applications may prioritize platform-specific navigation.Source: §8.4 · p. 27
• Domain-specific design knowledge remains important, as illustrated by DENIM, which addressed early-stage web design through site maps and page flows unavailable in conventional GUI prototyping tools.Nevertheless, visual hierarchy, consistency, and affordances are presented as principles shared across GUI and web design.Source: §8.4 · p. 27
• Cross-platform capability will require diverse training data, support for web-specific building blocks and navigation, and user controls for tailoring generation to platform requirements.The proposed direction is to combine broad design knowledge with dynamically adapted recommendations and outputs across an expanding range of digital interfaces.Source: §8.4 · p. 27
• The current system focuses on GUI prototyping and may not generalize well to web design, which requires responsive reflow, hyperlink navigation, and web-standard compliance.Web design also differs from GUI tools that rely on platform-specific conventions such as iOS Human Interface Guidelines or Android Material Design.Source: §8.4 · p. 27

• PrototypeFlow balances automation with customization by iteratively refining human-provided descriptions and wireframe layouts into high-fidelity prototypes.Source: §9 · p. 27
• Editable intermediate results let designers enhance design intent and rapidly regenerate outputs during the generation process.Source: §9 · pp. 27–28
• Quantitative and qualitative evaluations corroborate PrototypeFlow’s potential to improve the UI / UX design process.Source: §9 · p. 28
• Future work includes automated wireframe generation and dynamic component integration to support a more universal, user-friendly, efficient, and creative process.Source: §9 · p. 28