PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Design and Development with Automated Declarative UI Code Generation

Ting Zhou, Yanjie Zhao, Xinyi Hou, Xiaoyu Sun, Kai Chen, Haoyu Wang

arXiv:2409.11667v1cs.SE

TL;DR

DeclarUI addresses limitations in MLLM-based UI code generation, particularly incomplete component recognition and missing interaction logic. It combines computer vision, MLLMs, Page Transition Graphs, and iterative refinement, outperforming state-of-the-art methods across React Native evaluations and generalizing to Flutter and ArkUI.

  • Problem

    MLLM-based generation from UI designs struggles to accurately recognize components and capture complex interaction logic.

    Source: §2 · p. 3

  • Method

    DeclarUI combines computer vision, MLLMs, Page Transition Graphs for navigation logic, and iterative compilation- and navigation-based refinement.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    DeclarUI outperforms state-of-the-art methods across multiple metrics, including PTG coverage, visual similarity, compilation success, and compilation iterations, with improvements of up to 123%, 55%, and 29% over MLLMs.

    Source: §1 · p. 2, §7 · p. 12

  • Takeaways & Limitations

    DeclarUI generates declarative UI code across React Native, Flutter, and ArkUI, while professional developers report stronger code availability, readability, and maintainability than the baseline.

    Source: §1 · p. 2, §7 · p. 12

    CLIP, SSIM, and PTG Coverage may not fully capture human-perceived visual similarity or the fluidity and intuitiveness of navigation.

    Source: §5.2 · p. 12

Abstract

from arXiv · show

Declarative UI frameworks have gained widespread adoption in mobile app development, offering benefits such as improved code readability and easier maintenance. Despite these advantages, the process of translating UI designs into functional code remains challenging and time-consuming. Recent advancements in multimodal large language models (MLLMs) have shown promise in directly generating mobile app code from user interface (UI) designs. However, the direct application of MLLMs to this task is limited by challenges in accurately recognizing UI components and comprehensively capturing interaction logic. To address these challenges, we propose DeclarUI, an automated approach that synergizes computer vision (CV), MLLMs, and iterative compiler-driven optimization to generate and refine declarative UI code from designs. DeclarUI enhances visual fidelity, functional completeness, and code quality through precise component segmentation, Page Transition Graphs (PTGs) for modeling complex inter-page relationships, and iterative optimization. In our evaluation, DeclarUI outperforms baselines on React Native, a widely adopted declarative UI framework, achieving a 96.8% PTG coverage rate and a 98% compilation success rate. Notably, DeclarUI demonstrates significant improvements over state-of-the-art MLLMs, with a 123% increase in PTG coverage rate, up to 55% enhancement in visual similarity scores, and a 29% boost in compilation success rate. We further demonstrate DeclarUI's generalizability through successful applications to Flutter and ArkUI frameworks.

1 INTRODUCTION

Declarative UI frameworks improve maintainability, but translating visual designs into functional code remains manual, error-prone, and difficult for existing automated approaches. DeclarUI combines computer vision, MLLMs, Page Transition Graphs, and iterative refinement to improve declarative UI generation.

Motivation

  • Declarative UI frameworks improve maintainability, testability, and separation of concerns, yet design-to-code translation remains complex and time-consuming.Developers must manually map visual elements to implementation details, creating a need for automated generation directly from UI designs.Source: §1 · p. 1

Research gap

  • Existing methods struggle with complex designs, interactive logic, visual interpretation, and generalization across declarative frameworks.Heuristic and machine-learning methods extract components imperfectly, program-synthesis methods require detailed specifications, and prior cross-framework work targets imperative frameworks.Source: §1 · p. 1

Approach

  • DeclarUI combines computer vision, MLLMs, Page Transition Graphs, and iterative refinement to generate visually accurate and functionally robust declarative UI code.It structures UI components, represents navigation logic, and performs navigational-integrity and compilation checks after initial generation.Source: §1 · p. 2

Results

  • DeclarUI improved PTG coverage by 123%, visual similarity by up to 55%, and compilation success by 29% over state-of-the-art MLLMs on React Native.The comparison includes Claude-3.5 Sonnet and GPT-4o.Source: §1 · p. 2
  • Professional developers rated DeclarUI higher than the baseline for code availability, modification time, readability, and maintainability.Reported scores were 4.97 vs. 4.15, 4.03 vs. 2.45, 4.62 vs. 3.75, and 4.55 vs. 3.37, respectively.Source: §1 · p. 2

2 BACKGROUND AND MOTIVATIONS

Direct MLLM generation from screenshots can omit components, miss navigation, behave inconsistently across pages, and produce unreliable code. DeclarUI addresses these problems with component segmentation, PTGs, and iterative compilation-driven refinement.

Imprecise Component Recognition

  • MLLM-generated UI code can omit crucial components, such as the LazFlash button missing from the generated Account page.The omission can also remove a navigation element present in the original design.Source: §2 · p. 3

Interactive Logic

  • MLLMs may fail to infer expected transitions and maintain consistent navigation behavior across multiple screenshots.The Account and Home pages should both navigate to the LazFlash page, but the generated code does not implement those transitions consistently.Source: §2 · p. 3

Code Reliability and Compilation Issues

  • Generated MLLM code can contain compilation errors or runtime issues, requiring human intervention that offsets potential time savings.These reliability problems are not directly visible in the figure but are reflected in discrepancies between source screenshots and rendered output.Source: §2 · p. 3

Our Solutions

  • DeclarUI uses Grounding DINO and Segment Anything to isolate UI components before MLLM generation.The preprocessing step is intended to improve recognition and make generated code better reflect design intent.Source: §2 · p. 3
  • DeclarUI represents page navigation with a Page Transition Graph and iteratively checks navigational integrity and compilation.The combined process targets interactive consistency, code quality, and reduced manual intervention.Source: §2 · p. 3

3 APPROACH

DeclarUI converts visual UI designs into declarative code by combining structured interaction and component representations with MLLM generation and iterative validation. Its workflow captures navigation logic, extracts UI components, synthesizes prompts, and refines code through navigation and compilation checks.

3 APPROACH

  • DeclarUI processes Figma mockups and app screenshots as image inputs, then combines PTG construction, component extraction, prompt synthesis, and iterative code refinement.The workflow aims to generate complete UI code while preserving visual structure and interactive logic.Source: §3 · p. 3

3.1 PTG Construction

  • PTGs model app pages as nodes and page transitions as directed edges with triggering conditions and actions, giving MLLMs structured navigation logic.Generated PTGs are reviewed by designers or developers for accuracy and completeness.Source: §3.1 · pp. 3–4

3.2 UI Component Extraction and Representation

  • DeclarUI detects components with Grounding DINO, segments them with SAM, and uses an MLLM to classify types and infer functionality.The resulting representation contains spatial information, segmentation masks, component classifications, and functional descriptions.Source: §3.2 · pp. 4–5

3.3 Prompt Synthesis

  • Prompt synthesis combines the PTG, component analysis, and complete UI images to provide the MLLM with both visual context and structured interaction information.The synthesized prompt supports generation of visually accurate and functionally complete UI code.Source: §3.3 · p. 5

3.4 Iterative Code Refinement

  • DeclarUI iteratively checks generated code for missing PTG transitions and compiler errors, feeding targeted reports and corrections back to the MLLM.Navigation checks allow up to three iterations per page, while project-level compilation correction is limited to three iterations.Source: §3.4 · pp. 5–6

4 EVALUATION

The evaluation investigates DeclarUI’s autonomous effectiveness, the contributions of its modules, and professional developers’ assessments of generated code. It also examines the manual effort required to repair compilation failures.

4 EVALUATION

  • The evaluation is organized around effectiveness against baselines, ablation of key modules, and user study or manual-repair questions.These questions assess autonomous performance and practical developer impact.Source: §4 · p. 6

4.1 Dataset

The dataset contains 50 mobile app UI designs spanning mockups and screenshots from 10 functional categories. Each app contributes five connected pages for evaluating visual and interaction flows.

4.1 Dataset

  • The dataset includes 50 apps and 250 pages, equally divided between Figma mockups and Google Play screenshots across 10 functional categories.The balanced sources combine idealized design artifacts with interfaces from real-world apps.Source: §4.1 · pp. 6–7
  • Each selected app contributes five logically connected pages, enabling analysis of navigation and interaction between related interfaces.Google Play apps were drawn from highly ranked applications, while Figma designs were selected using popularity and mobile-specific criteria.Source: §4.1 · p. 7

4.2 Experimental Setup

The experiments evaluate visual fidelity, navigation completeness, compilation behavior, code quality, and practical usability against GPT-4o and Claude-3.5 in React Native. DeclarUI is also tested across React Native, Flutter, and ArkUI.

4.2.1 Evaluation Metrics.

  • Visual fidelity is measured with CLIP for semantic similarity and SSIM for layout and structural similarity between generated and original UIs.These metrics assess complementary aspects of visual resemblance.Source: §4.2.1 · p. 7
  • Navigation completeness is measured by PCR, comparing PTG edges with transitions observed in generated apps’ UI Transition Graphs obtained using DroidBot.Human evaluators manually calculate alignment between intended and implemented app structures.Source: §4.2.1 · p. 7
  • Compilation outcomes use CSR for successful automated compilation and ACIC for the average number of compilation attempts among successfully compiled apps.Apps that remain unsuccessful after three iterations are assessed with AMCT, the average human correction time.Source: §4.2.1 · p. 7
  • Developer-oriented evaluation measures code availability, modification time, readability, and maintainability using predefined scales and intervals.These metrics are intended to capture practical usability and code quality from a developer perspective.Source: §4.2.1 · p. 7

4.2.2 Baselines.

  • GPT-4o and Claude-3.5 serve as React Native baselines using unprocessed UI images, while prompts follow established image-to-code and LLM-use practices.The comparison is designed to avoid understating baseline MLLM capabilities.Source: §4.2.2 · p. 7
  • DeclarUI is evaluated on React Native, Flutter, and ArkUI using automated experiments designed to minimize human intervention and bias.The experiments run on a dual-RTX 3090 server with 64-core AMD EPYC processors and 512GB of RAM.Source: §4.2.3 · p. 8

4.3 Effectiveness (RQ1)

DeclarUI generates functional UI code across React Native, Flutter, and ArkUI, outperforming direct MLLM baselines in navigation coverage, compilation success, and visual fidelity. Claude-3.5 was selected as its preferred base model after stronger React Native results.

React Native effectiveness

  • 98% CSR for DeclarUI (Claude-3.5) corresponds to only one failed app, whose compilation failure resulted from non-existent file resources.DeclarUI (GPT-4o) had six apps that failed to compile automatically.Source: §4.3 · p. 8
  • 17.2% higher SSIM for DeclarUI (Claude-3.5) than DeclarUI (GPT-4o) indicates stronger reproduction of structural UI elements.The SSIM scores were 0.68 versus 0.58.Source: §4.3 · p. 8
  • DeclarUI (Claude-3.5) was selected for subsequent experiments because it performed strongly in both baseline comparisons and DeclarUI configurations.Its slightly higher ACIC was treated as a small trade-off against stronger visual fidelity and navigation coverage.Source: §4.3 · p. 8
  • 96.8% PTG Coverage Rate and 98% Compilation Success Rate make DeclarUI (Claude-3.5) substantially stronger than the corresponding baselines on React Native.DeclarUI (GPT-4o) also exceeded its baseline, reaching 88% CSR and 88.6% PCR versus 76% and 1.4%, respectively.Source: §4.3 · pp. 8–9, Table 1 · p. 9

Generalization capabilities

  • 99.4% PCR on Flutter, 96.8% on React Native, and 95.9% on ArkUI demonstrate DeclarUI’s navigation coverage across frameworks.Cross-framework CSR ranged from 86% to 98%, with ArkUI showing the lower result and remaining room for improvement.Source: §4.3 · p. 8

4.4 Ablation Study (RQ2)

Ablations show that DeclarUI’s CV module primarily supports visual accuracy, while PTG construction and iterative refinement support navigation correctness and compilable code. Their combination forms the core of the approach.

Impact of CV techniques on UI Generation

  • Removing UI component extraction reduced SSIM from 0.72 to 0.57, CLIP from 0.90 to 0.88, and PCR from 96.8% to 95.1%.The reductions indicate that CV-based component recognition improves structural and semantic similarity as well as navigation coverage.Source: §4.4 · p. 9

Impact of PTG and Refinement Process

  • Removing PTG and refinement reduced PCR to 23.1% and CSR to 80.0%, demonstrating their importance for navigation logic and code compilability.Visual similarity remained relatively high in this ablation.Source: §4.4 · p. 9

Overall ablation findings

  • DeclarUI’s CV techniques contribute to visual accuracy, whereas PTG and refinement support functional correctness and code quality.The ablation results identify the three components as synergistic rather than interchangeable.Source: §4.4 · p. 9

4.5 User Study and Manual Repair (RQ3)

Professional developers rated DeclarUI-generated code higher than the baseline across usability and maintainability measures, while case studies showed stronger component fidelity and navigation consistency. Manual correction times remained limited across frameworks, though they varied substantially.

4.5.1 User Study.

  • 4.97 versus 4.15 availability, 4.03 versus 2.45 modification-time scores, 4.62 versus 3.75 readability, and 4.55 versus 3.37 maintainability favored DeclarUI over the baseline.All reported improvements were statistically significant at p < 0.05.Source: §4.5.1 · p. 10, Table 3 · p. 10
  • 73.3% of DeclarUI code modifications were completed within 10 minutes, whereas 80% of baseline code required more than 10 minutes.Within the baseline group, 27.1% of that 80% was not completed within 20 minutes.Source: §4.5.1 · p. 10
  • DeclarUI improved component fidelity, navigation completeness, and interaction consistency in the shopping-app case study.It reproduced the bottom navigation bar omitted by the baseline and preserved consistent navigation to Orders from Profile and Account.Source: §4.5.1 · p. 10, Figure 8 · p. 11, Figure 9 · p. 11

4.5.2 Manual Repair.

  • 48s AMCT for React Native with Claude-3.5, 43.2s with GPT-4o, 62s for Flutter, and 183s for ArkUI quantify the manual repair effort.The authors characterize these results as requiring minimal manual intervention across frameworks.Source: §4.5.2 · p. 11

5 DISCUSSION

DeclarUI’s discussion highlights promising but uneven framework performance, especially for ArkUI, while identifying metric, prompt, participant, and sample limitations that constrain evaluation.

5.1 Retrieval-Augmented Generation (RAG): Impact and Implications

  • DeclarUI’s ArkUI performance was relatively poor, likely because the MLLM has less knowledge of this newer, less mainstream framework.The authors explored retrieval augmentation to compensate for this knowledge gap.Source: §5.1 · p. 11
  • RAG improved ArkUI results on 100 app pages, reducing ACIC by 0.52, increasing CSR by 4%, and reducing AMCT by 139.5s.The improvements were limited because time constraints restricted the sophistication of the knowledge graph implementation.Source: §5.1 · p. 11
  • Future work should refine the knowledge graph and its integration with MLLMs for ArkUI and other emerging UI frameworks.The proposed directions include richer framework syntax and design-pattern coverage and dynamic updates with new features.Source: §5.1 · p. 11

5.2 Threat to Validity

  • CLIP Score, SSIM, and PTG Coverage may not fully capture human-perceived visual similarity or navigation fluidity and intuitiveness.The user study nevertheless supported the validity of the reported evaluation by finding that DeclarUI outperformed baselines.Source: §5.2 · p. 12
  • Evaluation generalizability is limited by prompt sensitivity, possible participant bias, and a sample of only 50 UI design sets that may not represent all mobile app UIs.The authors recommend expanding the size and diversity of the UI sample.Source: §5.2 · p. 12

6 RELATED WORK

Related work spans mobile UI-to-code generation and broader image-to-code methods, while DeclarUI targets declarative, multi-page applications through component recognition and inter-page logic capture.

Mobile App UI-to-Code Generation

  • DeclarUI is presented as the first approach focused on declarative UI frameworks with automated multi-page mobile UI code generation.It combines advanced CV for component recognition with PTG-based capture of inter-page logic.Source: §6 · p. 12
  • Earlier mobile UI-to-code methods were limited by traditional frameworks, lacked multi-page support, struggled with complex designs, or required substantial manual effort.The cited approaches include framework-specific systems, natural-language generation, and intermediate representations.Source: §6 · p. 12

Image-to-Code Generation

  • Image-to-code research includes deep learning, CV-based, and multimodal learning approaches for converting screenshots, sketches, or other visual inputs into code.Multimodal methods address limitations of single-modal systems by combining data types for robustness and contextual understanding.Source: §6 · p. 12

7 CONCLUSION

DeclarUI combines CV, MLLMs, PTGs, and compiler-driven refinement to generate declarative UI code from designs. It outperforms state-of-the-art methods across multiple metrics and generalizes across three frameworks.

7 CONCLUSION

  • DeclarUI combines CV techniques, MLLMs, and iterative compiler-driven optimization to generate high-quality declarative UI code from mockups and screenshots.Its pipeline addresses component recognition, interactive-logic understanding, and code reliability.Source: §7 · p. 12
  • DeclarUI outperforms state-of-the-art methods across PTG coverage, visual similarity, compilation iterations, and compilation success rates.The conclusion reports improvements across multiple evaluation dimensions rather than a single metric.Source: §7 · p. 12
  • DeclarUI generalizes across React Native, Flutter, and ArkUI, while professional developers provide positive feedback on its practical value.The conclusion links multiframework compatibility with user-study evidence from professional developers.Source: §7 · p. 12

Related papers

Loading 2409.11667v1…