Source-linked AI summary
Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, Diyi Yang
TL;DR
Existing front-end code generation from visual designs lacks a realistic benchmark for measuring whether generated code faithfully reproduces webpages. Design2Code builds such a benchmark from 484 real-world webpages, adds visual evaluation metrics and prompting studies, and finds that models still struggle especially with visual-element recall and complex examples.
Problem
Visual-design-to-code generation is difficult to evaluate realistically because existing efforts use limited or synthetic webpages, while the task requires understanding diverse visual elements and layouts.
Method
The authors curate 484 real-world webpages, develop fine-grained screenshot metrics, evaluate multimodal models with prompting methods, and add human evaluation plus an 80-example hard set.
Results
Models generate well-formed websites on some easy examples but struggle on complex cases; fine-grained metrics show lag in recalling visual elements and producing correct layouts.
Takeaways & Limitations
Design2Code provides a realistic benchmark and diagnostic evaluation framework for measuring progress in multimodal front-end code generation.
Takeaways & Limitations
Direct training on real webpages is difficult because the data are too long and noisy for stable training.
Abstract
from arXiv · showhide
Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development in which multimodal large language models (MLLMs) directly convert visual designs into code implementations. In this work, we construct Design2Code - the first real-world benchmark for this task. Specifically, we manually curate 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations to validate the performance ranking. To rigorously benchmark MLLMs, we test various multimodal prompting methods on frontier models such as GPT-4o, GPT-4V, Gemini, and Claude. Our fine-grained break-down metrics indicate that models mostly lag in recalling visual elements from the input webpages and generating correct layout designs.
1 Introduction
Design2Code frames screenshot-to-code generation as a difficult but consequential front-end engineering task, then introduces a real-world benchmark and prompting study to measure current MLLMs.
- Visual-to-code generation requires recognizing webpage elements and layouts, then translating them into structured functional code.The task is difficult because it combines visual understanding with code generation.
- 484 manually curated real-world webpages provide challenging, diverse test cases spanning practical use cases and complexity levels.The benchmark uses webpages scraped from the C4 validation set rather than synthetic examples.
- Design2Code evaluates rendered visual similarity with metrics covering element bounding boxes, text, position, and color, which correlate highly with human judgment.These metrics assess the generated screenshot against the reference screenshot rather than relying only on code similarity.
- Text-augmented prompting improves most tested models, whereas few models can improve their outputs through self-revision.The study tests prompting methods that supplement visual input with extracted text or ask models to compare generated and reference webpages.
- Design2Code-HARD adds 80 challenging examples for comparing leading commercial models on difficult webpage-generation cases.The hard set complements the broader benchmark with a focused stress test.
2 The Design2Code Benchmark
The Design2Code benchmark is built from filtered real webpages and evaluates screenshot-to-code systems with fine-grained visual metrics that diagnose both coverage and fidelity.
- 2.1 Test Set Curation: 127.9k scraped webpages are consolidated into single HTML files with embedded CSS before filtering and curation.The initial collection comes from website links in the C4 validation set.
- 2.1 Test Set Curation: Automatic filtering, deduplication, dependency removal, placeholder replacement, and manual review reduce the collection to 484 high-quality webpages.Review checks external-file independence, sensitive content, and proper formatting.
- 2.2 Dataset Analysis: Benchmark files average 31,216 GPT-2 tokens and 158 HTML tags, with examples spanning 84 standard HTML5 tag types.The token range is 784–98,637 and the tag range is 12–528, indicating substantial variation in size and complexity.
- 2.1 Test Set Curation: The benchmark contains webpages from varied real-world domains, with companies or organizations, personal blogs, and personal homepages among the most frequent genres.Domain labels are based on manual annotation of 120 randomly sampled examples.
- 2.3 Automatic Metrics: Text-based code metrics cannot directly determine whether a generated implementation reproduces the reference screenshot, because different code can yield similar webpages.Minor code differences can also create major visual differences in the rendered output.
- 2.3 Automatic Metrics: The metric suite supplements CLIP with element-level matching of visual recall, text content, position, and color.This provides fine-grained diagnosis beyond overall image similarity.
- 2.3 Automatic Metrics: Block-Match compares matched reference and generated visual blocks while penalizing missed and hallucinated elements through area-based scoring.Blocks are detected from both screenshots and optimally matched using text similarity before scoring.
- 2.3 Automatic Metrics: Color similarity uses the CIEDE2000 perceptual color-difference formula across matched text blocks, alongside diagnostic low-level scores.The low-level scores are intended to complement CLIP and assess multiple visual dimensions.
3 Benchmarking: Prompting and Finetuning
The benchmark compares commercial, open-source, and fine-tuned multimodal models under direct, text-augmented, and self-revision prompting, including a hard-example evaluation setting.
- 3 Benchmarking: Prompting and Finetuning: The study benchmarks commercial API, open-source, and fine-tuned models on the Design2Code task.
- 3.1 Prompting Methods: The prompting study assumes models accept an image and text prompt and generate code as output.
- 3.1 Prompting Methods: Direct prompting supplies the reference screenshot and asks the model to generate HTML and CSS code.
- 3.1 Prompting Methods: Text-augmented prompting complements the screenshot with extracted webpage text to reduce the burden of recognizing content and layout simultaneously.
- 3.1 Prompting Methods: Self-revision provides the reference screenshot, an initial rendered webpage, and its generated code, then asks the model to improve visual alignment.
- 3.2 Model Setup: The model set also includes WebSight VLM-8B and Design2Code-18B, which are fine-tuned models based on open architectures and task-specific data.
4 Results and Analysis
Automatic and human evaluations show that prompting effects depend on the model and metric: text augmentation often improves fine-grained visual similarity, while human judgments can diverge from automatic scores. Design2Code-HARD exposes substantial remaining difficulty, especially for long and non-English webpages.
- Automatic evaluation: GPT-4o is the best-performing model across all reported automatic evaluation dimensions.
- Automatic evaluation: Text-augmented prompting increases block-match and text similarity for most tested models, especially models with weaker text recognition.
- Automatic evaluation: Self-revision yields minor gains for GPT-4V and Claude 3 but no improvement for Gemini Pro Vision or the other open-source models.
- Human evaluation: Human evaluation ranks GPT-4o substantially above other baselines, while prompting additions provide limited benefit because direct prompting already captures much of its text and layout.
- Automatic evaluation vs human evaluation: Automatic metrics and human preferences diverge: humans prefer GPT-4V self-revision over text augmentation, and rank WebSight VLM-8B above Design2Code-18B despite worse block-match and text similarity.
- Automatic evaluation vs human evaluation: A logistic-regression model predicts human Win/Lose judgments with 79.9% test accuracy, with high-level visual measures positively associated and text similarity negatively associated.
- Qualitative analysis: In qualitative examples, text augmentation raises GPT-4V block-match from 0.25 to 0.84, while self-revision raises another block-match score from 0.48 to 1.00.
- Design2Code-HARD: Design2Code-HARD contains 80 difficult examples, including long and non-English webpages; state-of-the-art models fail to generate 30%-40% of block elements.
5 Related Work
Design2Code advances UI-to-code research from earlier limited systems by benchmarking multimodal LLMs on diverse real-world webpages. It also connects the task to programming-support tools for non-coding front-end designers.
- Self-revision prompting can add missing text or correct layout errors compared with text-augmented prompting.
- Earlier UI-to-code systems used OCR, computer vision, or CNN-RNN pipelines but struggled with complex visual encoding and long text decoding.
- Design2Code introduces a UI automation benchmark using real-world webpages and evaluates state-of-the-art multimodal LLMs.
- The benchmark complements code-language-model research by providing realistic evaluation for programming-support applications.
6 Conclusion
The paper introduces Design2Code and Design2Code-HARD for evaluating multimodal LLMs on diverse real-world webpages. Results show strong performance on easier examples but persistent difficulty with complex webpages, especially for open-source models.
- Design2Code(-HARD) benchmarks multimodal LLMs using diverse and challenging real-world webpages, automatic metrics, and human evaluations.
- State-of-the-art models generate well-formed websites on some easy examples but still struggle with more complex examples.
- Open-source models remain far behind commercial models, leaving substantial room for future improvement.
- Future work could extend the benchmark to UIs supporting dynamic user interactions.
Limitations
The paper identifies limitations in prompting, training data, input modalities, and webpage dynamics. Addressing them requires improved prompting, data cleaning, redesigned evaluation, and support for interactive webpages.
- Better prompting techniques are needed for multimodal LLMs handling complex webpages, including incremental generation of webpage parts.
- Direct training on real webpages is difficult because the data are too long and noisy, motivating data-cleaning pipelines for stable training.
- The screenshot-only input setting could be extended with Figma frames or sketches, but this requires careful redesign of the evaluation paradigm.
- Extending from static to dynamic webpages requires evaluation of interactive functions beyond visual similarity.
Ethical Considerations
The dataset is released under the ODC-By license and excludes webpages containing private or sensitive information. The authors also recognize dual-use risks, including malicious or unauthorized website generation.
- The dataset uses the ODC-By license, which permits sharing, modification, and use subject to attribution requirements.
- Manual filtering removed webpages containing private or sensitive information, such as dating website profiles.
- Design2Code technologies could be misused to generate malicious websites or code for licensed websites.
- The authors intend the work for research and plan ethical-use guidelines for released data, code, and models.
A Additional Dataset Statistics
The benchmark combines carefully curated real-world webpages with screenshot-based evaluation, model prompting protocols, and training details. Its examples and processing choices are designed to support diverse, challenging test cases while controlling implementation and evaluation conditions.
- Dataset curation: 484 webpages were manually curated from 7k sampled examples after automatic filtering and deduplication.The curation process took approximately one week and aimed to preserve high-quality, diverse real-world webpages.
- Automatic evaluation: Detected reference and generated blocks are optimally matched with the Jonker-Volgenant algorithm using text similarity.Matched pairs below a text-similarity threshold are ignored, and differing block granularity is handled by enumerating merges.
- Dataset statistics: Design2Code examples are more complex than WebSight examples across dataset statistics and span a wider range of difficulty levels.The comparison includes dataset complexity measures and standard deviations.
- Prompting: The study tests direct and self-revision prompts, including prompts that provide extracted webpage text alongside visual inputs.The self-revision prompt asks models to compare a current rendering with the reference and revise the HTML implementation.
F Human Annotation Details
Human evaluations compare webpage candidates by prioritizing content, then layout, then style. The study uses structured pairwise and direct-assessment tasks with multiple annotators and reports inter-annotator agreement.
- Annotators: The human evaluation uses 60 U.S. participants who had completed 2,500 surveys with a pass rate of at least 98%.The instructions ask annotators to follow the content > layout > style priority order.
- Evaluation criteria: Annotators compare candidate webpages against a reference using content, layout, and style in that priority order.Content checks include text and image content; layout checks examine arrangement, order, and hierarchy; style is considered only when earlier dimensions are comparable.
- Pairwise comparison: The pairwise task rates which candidate is closer to the reference, with ties allowed when both candidates are similarly accurate or distant.The instructions emphasize cumulative assessment after checking content, layout, and style.
- Agreement: Fleiss’ kappa for pairwise model comparison is 0.46 across five annotators.Direct assessment reports Fleiss’ kappa values of 0.32 and 0.26 for its two questions.
- Direct assessment: The direct-assessment tasks ask whether an AI-generated webpage is exchangeable with the reference and which webpage is better.Exchangeability considers rough visual similarity, similar content, and serving the same functions; the better-page task considers readability, layout, and style.
G Simulated Win Rate
The paper converts automatic-metric predictions into simulated win rates and compares them with human “win + tie” rates. The strong agreement supports using aggregated automatic metrics for model comparison, while training affects evaluation dimensions differently.
- Win-rate construction: The learned linear model aggregates automatic metrics into simulated win rates for comparing models across the full benchmark.The paper removes text similarity when rerunning the regression and reports the resulting simulated rates in Table 8.
- Win-rate validation: Simulated and human “win + tie” rates are strongly correlated across models, with Pearson r = 0.975 and Kendall τ = 0.931.Simulated rates use a learned linear model over all 484 examples, whereas human rates come from annotations on 100 examples.
- Coverage: The benchmark provides simulated win rates for models without corresponding human annotations.These estimates are reported in an appendix using the learned linear model over all 484 examples.
- Learning dynamics: Block-match, text, and position performance saturates after about 2000 training steps, while color and CLIP similarity continue increasing through 4000–5000 steps.The authors associate the earlier saturation with transferable base-model capabilities and suggest color coding benefits more from HTML training data.
I More Case Study Examples
A representative comparison between WebSight VLM-8B and Design2Code-18B shows a trade-off between visual color and layout performance and accurate text matching.
- Figure 10 comparison: WebSight VLM-8B exceeds Design2Code-18B on color, position, and CLIP similarity, but Design2Code-18B achieves better block-match and text similarity.The reported scores are color 0.99 vs 0.66, position 0.91 vs 0.63, CLIP 0.90 vs 0.83, block-match 0.85 vs 0.99, and text similarity 0.98 vs 1.0.
- Figure 10 comparison: WebSight VLM-8B’s stronger visual scores coexist with hallucinated text and lower text-matching precision and recall.The example illustrates that better color and layout scores do not guarantee more accurate textual content.