PaperlayerView exact revision on arXiv

Source-linked AI summary

MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs

Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, Michael R. Lyu

arXiv:2412.15310v1cs.SEcs.AIcs.IR

TL;DR

Existing design-to-code methods focus on self-contained pages, leaving key multi-page resource and evaluation needs insufficiently addressed. The paper defines MRWeb, builds a dataset and evaluation framework, and studies MLLMs; it reports that pixel-based metrics align better with human judgments in low-to-medium similarity cases and resource lists improve visual and functional metrics.

  • Problem

    Existing design-to-code targets self-contained pages, while MRWeb lacks integrated resource structures, datasets, and standardized metrics for evaluating navigation and visual correspondence.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces MRWeb and resource lists, builds a 500-website dataset, and evaluates MLLMs on the task.

    Source: §1 · p. 2

  • Results

    Pixel-based metrics align better with human judgments in low-to-medium similarity cases, and resource lists improve visual and functional metrics.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors release the tool, dataset, and evaluation framework to facilitate further research.

    Source: §8 · p. 8

    Non-link-based functionalities remain unsupported and require distinct formulations and evaluation metrics.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Multi-page websites dominate modern web development. However, existing design-to-code methods rely on simplified assumptions, limiting to single-page, self-contained webpages without external resource connection. To address this gap, we introduce the Multi-Page Resource-Aware Webpage (MRWeb) generation task, which transforms UI designs into multi-page, functional web UIs with internal/external navigation, image loading, and backend routing. We propose a novel resource list data structure to track resources, links, and design components. Our study applies existing methods to the MRWeb problem using a newly curated dataset of 500 websites (300 synthetic, 200 real-world). Specifically, we identify the best metric to evaluate the similarity of the web UI, assess the impact of the resource list on MRWeb generation, analyze MLLM limitations, and evaluate the effectiveness of the MRWeb tool in real-world workflows. The results show that resource lists boost navigation functionality from 0% to 66%-80% while facilitating visual similarity. Our proposed metrics and evaluation framework provide new insights into MLLM performance on MRWeb tasks. We release the MRWeb tool, dataset, and evaluation framework to promote further research.

1 Introduction

Existing design-to-code methods target self-contained pages, leaving multi-page navigation, external resources, and backend routing under-addressed. MRWeb introduces resource-aware generation, a benchmark, evaluation metrics, and a tool to study these needs.

1 Introduction

  • 90.3% (271/300) of sampled top-visited websites were multi-page.The sample was drawn from the 300 most-visited websites ranked by the Tranco list.Source: §1 · p. 1
  • MRWeb addresses the gap between self-contained design-to-code and websites with internal navigation, external resources, and backend routing.The paper identifies those capabilities as central complexities of multi-page, resource-aware web UI development.Source: §1 · pp. 1–2
  • The paper proposes a resource list and a 500-website dataset with associated resource lists, screenshots, and ground-truth code.The dataset contains synthetic and real-world websites.Source: §1 · p. 2
  • The study proposes web-UI evaluation metrics, conducts an image-quality assessment, and collects an annotated IQA dataset.It also evaluates state-of-the-art MLLMs on MRWeb and highlights challenges they face.Source: §1 · p. 2
  • The authors release a user-friendly MRWeb tool, along with the code and data, to support further research and development.The tool is presented as part of the study’s contributions.Source: §1 · p. 2

2 Related Work

The paper characterizes existing UI code-generation methods as relying on assumptions that exclude important features of real-world multi-page websites.

2 Related Work

  • Existing UI code-generation methods assume websites have a single or limited number of pages.The paper contrasts this assumption with the prevalence of multi-page websites in its Appendix A study.Source: §2 · p. 2
  • Existing methods assume webpages lack external links.The paper identifies this as one of three assumptions it considers unrealistic.Source: §2 · p. 2
  • Existing methods assume webpages do not interact with backends.The paper says its Appendix A study highlights the mismatch between these assumptions and real-world web UI needs.Source: §2 · p. 2

3 Task Formulation

MRWeb generation uses an MLLM to produce webpage code from a screenshot and resource list, then assesses both functional and visual alignment.

3 Task Formulation

  • A resource list represents a webpage’s navigational and visual elements, including hyperlinks, images, and backend routing.Entries include attributes such as position, type, and URL; the paper says this structure helps replicate navigation and image sources accurately.Source: §3 · p. 2
  • The task maps ground-truth webpage code, its screenshot, and its resource list to generated HTML+CSS code using an MLLM.The generated code is intended to approximate the ground-truth code.Source: §3 · p. 2
  • Evaluation compares the generated resource list with the original and the rendered screenshot with the original screenshot.These comparisons assess functional and visual alignment, respectively.Source: §3 · p. 3

4 Dataset Collection

The dataset combines synthetic and real-world webpages, adapting source data and processing live pages before extracting resources and sampling the benchmark.

4 Dataset Collection

  • The benchmark combines synthetic and real-world data to balance controlled, diverse examples with authentic webpage variety.The paper says the combination provides a comprehensive benchmark and supports robustness across HTML structures and styles.Source: §4 · p. 3

4.1 Synthetic Data Collection

  • The synthetic dataset adapts WebSight by adding links and replacing randomly loaded images with static, unique images.WebSight’s original pages lacked valid internal or external links, and random Unsplash loading could make visuals inconsistent across identical code.Source: §4.1 · p. 3

4.2 Real-world Data Collection

  • The real-world pipeline collects live-site HTML and screenshots, then filters and simplifies pages to remove non-visible noise and non-functional scripts.The source HTML can be excessively lengthy and exceed model token limits.Source: §4.2 · pp. 3–4

4.3 Resource List Extraction

  • Resource lists record link positions, types, and targets, plus image and background-image positions and source URLs.The lists are constructed automatically by iterating through webpage elements and checking their visibility and functionality.Source: §4.3 · p. 4

4.4 Automation & Dataset Statistics

  • The automated pipeline could in principle scale to WebSight’s two million examples and internet-accessible sites, but this benchmark samples 300 synthetic and 200 real-world instances.The paper also notes that companies’ or organizations’ websites and blogs are the most prominent genres in the sample.Source: §4.4 · p. 4

5 Study Setup

The study evaluates three state-of-the-art MLLMs using four prompting strategies. It assesses visual and functional similarity with high-level metrics and examines resource-level alignment with fine-grained metrics.

5.1 Evaluated Models

  • Three MLLMs—Gemini 1.5, GPT-4o, and Claude-3.5—are evaluated, with temperature 0, seed 42, and max_tokens 4096 for each model.Gemini-1.5-Pro was accessed during November 2024; the specified model versions were GPT-4o 20240806 and Claude-3.5-Sonnet 20240620.Source: §5.1 · p. 4

5.2 Prompting Strategies

  • Four prompting strategies are compared: self-contained without resource lists, zero-shot with resource lists, chain-of-thought, and self-refine.Self-contained prompting serves as the baseline for methods adopting resource lists.Source: §5.2 · p. 4

5.3.1 High-level Metrics

  • Visual similarity is evaluated at pixel, structural, and semantic levels, alongside functional similarity.Pixel metrics include MAE, PSNR, and NEMD; higher PSNR and NEMD indicate greater similarity.Source: §5.3.1 · pp. 4–5

Visual: Structure-level metrics

Visual: Semantic-level metrics

  • Semantic-level measures include CLIP Score and LPIPS; Resource Existence Ratio measures the share of reference resources matched in the generated list.Resource matching considers attributes such as whether navigation points to the same link or images share the same source.Source: Visual: Semantic-level metrics · p. 5

5.3.2 Fine-Grained Metrics

  • Fine-grained evaluation measures element position, actionable-element area, color, and text differences for matched resources.Position is normalized to the original webpage size, area difference to the reference element area, color uses CIEDE2000, and text difference is one minus normalized character-match similarity.Source: §5.3.2 · p. 5

6 Experiment Results

The experiments compare web UI similarity metrics, test resource lists and prompting strategies, characterize MLLM weaknesses, and assess the MRWeb tool. Resource lists improve functional similarity, while visual grounding and performance on complex inputs remain challenges.

6.1 The Best Web UI Similarity Metric

  • MAE (0.542) and NEMD (0.508) show the best consistency with human rankings among the evaluated metrics.NEMD achieves the highest CC (0.532) and lowest MAE (0.628) and RMS (0.752) in nonlinear regression.Source: Table 2 · p. 6, §6.1 · pp. 6–7
  • Pixel-based metrics, especially MAE and NEMD, align best overall with human rankings, while SSIM, CLIP, and LPIPS minimize outliers.Pixel-based metrics perform better in low- and medium-similarity cases; structural and semantic metrics perform better in high-similarity cases.Source: §6.1 · pp. 6–7

6.2 Effectiveness of the Resource List

  • Resource lists improve visual similarity across MLLMs and metrics; self-contained prompting consistently yields the lowest visual scores.Resource lists let MLLMs use the exact webpage images, whereas without them they can use only placeholder images.Source: Table 3 · p. 6, §6.2 · p. 7
  • RER rises from 0% to 66%-80% with resource lists, which supply guidance for generating valid resources and links.Under self-contained prompting, functional similarity is near zero; some links are inferred through common sense.Source: Table 4 · p. 6, §6.2 · p. 7
  • Self-refine scores highest across models on visual and functional metrics, while chain-of-thought prompting slightly decreases performance.Gemini-Pro is the top-performing MLLM for reproducing functionality.Source: Table 3 · p. 6, Table 4 · p. 6, §6.2 · p. 7

6.3 Limitations of MLLMs in MRWeb

  • MLLMs’ main challenge is visual grounding: elements average 21.5% positional shift and 35.6% size difference from the originals.Claude-3.5 has the lowest positional shift (15.5%) and area difference (27.9%) under zero-shot prompting; GPT-4o achieves Text Similarity (0.004).Source: §6.3 · pp. 7–8, Table 5 · p. 7
  • GPT-4o’s performance declines as input image size and resource-list length increase, with similar patterns observed in other MLLMs.GPT-4o performs well on simpler websites but its accuracy declines significantly on intricate websites.Source: §6.3 · p. 8, Figure 3 · p. 8

6.4 MRWeb ’s Practical Capabilities.

  • In a three-page personal-website case study, the MRWeb tool achieved a 100% success rate across tested navigation, image, and backend-routing challenges.The generated pages included internal and external links, embedded images, and backend routing.Source: §6.4 · p. 8, Table 6 · p. 8

7 Discussions

The discussion highlights pixel-based metrics, resource lists, and visual grounding as key considerations for MRWeb generation. It also notes that non-link-based functionality remains underexplored.

Visual metrics for UI quality (RQ1)

  • Pixel-based metrics align better with human judgment, especially at low-to-medium similarity; the authors suggest hybrid metrics and targeted fine-tuning for learning-based methods.MAE and NEMD perform well across similarity levels, while learning-based methods are limited in low-to-medium cases.Source: §7 · p. 8

Enhancing website generation with resource lists (RQ2)

  • Resource lists significantly boost both visual and functional metrics in website generation.Source: §7 · p. 8

Improving MLLM visual grounding (RQ3)

  • Metrics on positional shift and area difference expose MLLM limitations in precise positioning and sizing.The authors suggest improving visual grounding or using layout-aware prompts to improve layout reproduction.Source: §7 · p. 8

Advancing MRWeb generation (RQ4)

  • MRWeb supports links, images, and routing, but non-link-based functionality remains underexplored.Source: §7 · p. 8

8 Conclusion

The paper introduces MRWeb generation to address limitations of single-page design-to-code methods. It presents a benchmark, web UI evaluation and MLLM analysis, and a dedicated tool, releasing the tool, dataset, and framework for future research.

8 Conclusion

  • The paper introduces MRWeb generation to address the limitations of single-page design-to-code methods.Source: §8 · p. 8
  • Its contributions include a benchmark dataset, comprehensive web UI image-quality assessment, MLLM performance analysis, and a dedicated generation tool.Source: §8 · p. 8
  • The authors release the MRWeb tool, dataset, and evaluation framework to facilitate future research.Source: §8 · p. 8

Limitations

The paper identifies boundaries in MRWeb's functional coverage, model context handling, and tested backbones.

Limitations

  • MRWeb handles links, images, and routing, but does not support non-link-based functionality, which requires distinct formulations and evaluation metrics.Source: Limitations · p. 9
  • Although GPT-4o has a 128K-token context window, all tested prompts remain within the limit, supporting feasibility for most practical scenarios.Source: Limitations · p. 9
  • The study validates MRWeb using three popular multimodal LLMs; smaller models struggle with complex prompts.Future work will explore emerging models and strategies for more complex input scenarios.Source: Limitations · p. 9

A Motivating study

The motivating study finds that sampled websites commonly exceed single-page and resource-light assumptions. The section also reviews related methods, documents prompt variants, and describes visual similarity metrics.

A Motivating study

B Related Works

  • Prior design-to-code work spans deep learning, computer vision, and MLLMs, but multi-page website generation remains underexplored.Source: B Related Works · pp. 11–12

C Prompts

  • The prompt appendix includes self-contained, action-list zero-shot, chain-of-thought, and self-refinement prompts for webpage generation.Source: C Prompts · p. 12

D Visual Similarity Metrics

  • Visual similarity metrics cover pixel-level differences, structural coherence, and semantic or perceptual similarity.The section describes MAE and PSNR at the pixel level, SSIM for structure, and CLIP and LPIPS for semantic or perceptual comparisons.Source: D Visual Similarity Metrics · pp. 12–13
  • NEMD normalizes EMD to a 0–1 range using the original screenshot as reference; higher scores indicate greater similarity.The normalization makes the measure size-independent, but it is not symmetric because its maximum is calculated relative to the reference.Source: D Visual Similarity Metrics · p. 13

E Image Quality Analysis Details

The section describes how human similarity judgments are processed and how objective image-similarity metrics are compared with them. It uses rank correlation and regression-based measures to assess alignment and prediction accuracy.

E Image Quality Analysis Details

  • Human scores are z-normalized per annotator, outliers are removed, and the remaining ratings are averaged into a mean opinion score for each image pair.The authors state that the image database and subjective scores will be publicly available.Source: E Image Quality Analysis Details · p. 13
  • Variance-weighted linear regression downweights noisy scores, while logistic nonlinear regression tests monotonic relationships that are not strictly linear.Both regression approaches report CC, OR, MAE, and RMSE measures of alignment and prediction error.Source: E Image Quality Analysis Details · pp. 13–14
  • Inter-rater reliability averages pairwise SROCC values, while objective metrics are assessed through rank-based alignment and regression-based prediction measures.The direct SROCC analysis considers absolute correlation; regression measures include correlation, outlier ratio, weighted MAE, and weighted RMSE.Source: E Image Quality Analysis Details · pp. 13–14

F Analysis of Similarity Metrics Across Different Similarity Levels

Similarity metrics behave differently across low-, medium-, and high-similarity image groups, while MLLM output quality varies with prompting and input complexity. The MRWeb tool supports functional multi-page code, and its outputs include real images and links unlike self-contained webpages.

F Analysis of Similarity Metrics Across Different Similarity Levels

  • In the medium-similarity group, all metrics generally decline, with CLIP reaching CC-N: 0.262 under nonlinear mapping while NEMD and MAE remain competitive.SSIM and LPIPS improve in the high-similarity group; LPIPS has SROCC: 0.367.Source: F Analysis of Similarity Metrics… · p. 14
  • Pixel-based MAE and NEMD distinguish dissimilar pairs while retaining reasonable performance on highly similar pairs; learning- and structure-based metrics perform poorly in low- and medium-similarity groups.The table reports NEMD as best in the low-similarity group (CC-V: 0.422, CC-N: 0.430, SROCC: 0.408); in the high-similarity group PSNR leads CC-V (0.582) and CC-N (0.583), while SSIM leads SROCC (0.379).Source: F Analysis of Similarity Metrics… · pp. 14–15

G Why CoT Decrease Performance

  • CoT prompts can compound omissions because MLLMs generate code part by part and often leave out content at the end of each part.Manual inspection found outputs ending with placeholder comments; the omissions lower action existence rates and overall performance.Source: G Why CoT Decrease Performance · p. 15

H MLLMs’ Performance and Input Complexity

  • As input images grow, MAE rises for all models, and action existence ratios fall as action lists lengthen; GPT-4o and Claude-3.5 are more stable than Gemini-Pro on image size.Self-refine consistently performs best for Gemini-Pro and GPT-4o, but offers minimal or occasional negative improvement for Claude-3.5; CoT and self-refine give slight action-ratio advantages for GPT-4o and Claude-3.5.Source: H MLLMs’ Performance and Input… · p. 15, Figure 6 · p. 16, Figure 7 · p. 16

I Developing Website with the MRWeb Tool

  • The MRWeb tool maps design assets and actions to visual elements through action lists, then uses those lists and screenshots as MLLM inputs to generate visually consistent multi-page code.The case study builds home, project, and contact pages for a personal website.Source: I Developing Website with the MRWeb Tool · p. 15

J Visual Comparison of Self-Contained Web and MRWeb

Loading 2412.15310v1…