PaperlayerView exact revision on arXiv

Source-linked AI summary

DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation

Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, Michael R. Lyu

arXiv:2506.06251v3cs.SEcs.AI

TL;DR

Existing benchmarks leave MLLMs’ capabilities in mainstream framework-based website development unexplored. DesignBench evaluates MLLMs across front-end frameworks and generation, editing, and repair, finding substantially lower performance in framework-based development than in vanilla HTML/CSS.

  • Problem

    Existing benchmarks omit mainstream front-end frameworks, leaving MLLMs’ capabilities in practical framework-based website development unexplored.

    Source: §1 · p. 2

  • Method

    DesignBench evaluates MLLMs across React, Vue, and Angular on design generation, editing, and repair, with 900 webpage samples spanning topics, edit types, and issue categories.

    Source: §1 · pp. 2–3

  • Results

    MLLMs perform substantially worse in framework-based development than in vanilla HTML/CSS and exhibit distinct bottlenecks across generation, editing, and repair.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The benchmark identifies limitations in current MLLMs’ framework-based development and offers actionable advice for researchers and developers.

    Source: §9 · p. 19

    Framework coverage is limited to React, Vue, and Angular.

    Source: §7 · p. 18

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in automated front-end engineering, e.g., generating UI code from visual designs. However, existing front-end UI code generation benchmarks have the following limitations: (1) While framework-based development becomes predominant in modern front-end programming, current benchmarks fail to incorporate mainstream development frameworks. (2) Existing evaluations focus solely on the UI code generation task, whereas practical UI development involves several iterations, including refining editing, and repairing issues. (3) Current benchmarks employ unidimensional evaluation, lacking investigation into influencing factors like task difficulty, input context variations, and in-depth code-level analysis. To bridge these gaps, we introduce DesignBench, a multi-framework, multi-task evaluation benchmark for assessing MLLMs' capabilities in automated front-end engineering. DesignBench encompasses three widely-used UI frameworks (React, Vue, and Angular) alongside vanilla HTML/CSS, and evaluates on three essential front-end tasks (generation, edit, and repair) in real-world development workflows. DesignBench contains 900 webpage samples spanning over 11 topics, 9 edit types, and 6 issue categories, enabling detailed analysis of MLLM performance across multiple dimensions. Our systematic evaluation reveals critical insights into MLLMs' framework-specific limitations, task-related bottlenecks, and performance variations under different conditions, providing guidance for future research in automated front-end development. Our code and data are available at https://github.com/WebPAI/DesignBench.

1 Introduction

DesignBench addresses gaps in front-end code-generation benchmarks by evaluating MLLMs across frameworks, development tasks, and performance dimensions. Its findings expose framework- and task-specific weaknesses as well as sensitivity to task difficulty and input context.

1 Introduction

  • Design-to-code conversion is a labor-intensive web-development step, and MLLMs’ visual code-generation capabilities create opportunities to automate it.The target includes reproducing webpage elements, layouts, text, and colors.Source: §1 · p. 2
  • Existing benchmarks omit mainstream front-end frameworks, focus on generation rather than editing and repair, and lack multidimensional analysis of difficulty, context, and code attributes.Prior benchmarks commonly synthesize webpage code or curate real-world webpages, but do not adequately represent complex development scenarios.Source: §1 · p. 2
  • DesignBench introduces a multi-framework, multi-task benchmark spanning HTML/CSS, React, Vue, and Angular, with 900 samples across more than 11 topics, 9 edit types, and 6 issue categories.It evaluates nine leading MLLMs across generation, editing, and repair, analyzing task difficulty, input modalities, correctness, and reusability.Source: §1 · pp. 2–3
  • MLLMs underperform in framework-based development versus vanilla HTML/CSS, with distinct bottlenecks across generation, editing, and repair.They struggle with framework syntax and component-based reuse; generation is hindered by rendering inaccuracies and compilation errors, while editing and repair are constrained by code localization and issue identification. Performance also degrades with large UI images, complex instructions, or severe issues; code-only inputs consistently outperform image-only inputs for editing and repair, with multimodal combinations adding little. The evaluation identifies 22 failure types across the tasks.Source: §1 · p. 3

2 Background

The background frames front-end development as a progression from design mockups to code generation, editing, and repair, and situates this work within reusable front-end frameworks.

2.1 Web Development Process

  • Front-end development moves from high-fidelity mockups to code generation, then iterative editing and repair when fidelity or deployment issues remain.The example workflow edits the generated code to change the background color and add buttons, then repairs a display collision between an avatar and a name.Source: §2.1 · pp. 3–4

2.2 Front-end Web Development Framework

  • React, Vue, and Angular were predominant in web development in 2025, with popularities of 39.5%, 15.4%, and 17.1%, respectively.Front-end frameworks provide reusable code, standardized components, and templates for building website user interfaces.Source: §2.2 · p. 4

3 Related Work

Prior UI-to-code research spans deep-learning, computer-vision, and MLLM approaches, while existing UI benchmarks vary in data and scope. DesignBench addresses gaps in framework coverage, task coverage, and evaluation dimensions.

3.1 UI Code Generation

  • UI code generation spans deep-learning, computer-vision, and MLLM methods, but prior work omits framework-based development, design editing, and repair.Methods include CNN/LSTM or attention-based GUI-to-code generation, sketch and OCR pipelines, and MLLM approaches such as divide-and-conquer, segmentation, and token compression.Source: §3.1 · p. 4

3.2 Benchmarks for UI Code Generation

  • Benchmarks progressed from DSL-based Pix2code [11] and synthetic-data WebSight and Web2Code [56] to Design2Code’s 484 authentic pages [41] and WebCode2M’s 20,000 samples [20]; interaction-focused benchmarks [51] [45] are out of scope.Interaction2Code and MRWeb focus on interactive and multi-page web application generation.Source: §3.2 · pp. 4–5
  • DesignBench distinguishes itself from existing benchmarks by incorporating varied front-end frameworks, multiple tasks, and diverse evaluation dimensions.Table 1 compares benchmark size, collection method, framework, target tasks, and evaluation dimension.Source: §3.2 · p. 5

4 DesignBench

DesignBench defines generation, editing, and repair tasks and builds a benchmark from framework-based and vanilla webpages, with annotations and statistics spanning diverse topics, modifications, and display issues.

4.1 Task Definition

  • Generation maps a UI mockup to code, editing uses the original image and code plus a natural-language instruction, and repair uses problematic code and its rendered image.The respective outputs are visually matching UI code, updated code incorporating requested modifications, and repaired code that resolves visual design issues.Source: §4.1 · p. 5

4.2 Data Collection

  • The generation dataset combines framework webpages from GitHub and highly visited sites with vanilla webpages, while edit samples originate in React and Vue project histories.The authors selected 152 popular GitHub projects and 158 framework webpages among the top 500 globally visited sites; they also sampled 120 vanilla webpages, yielding 430 generation samples. They curated 359 high-quality React/Vue edit samples and translated 146 into Angular and vanilla HTML/CSS, with student verification.Source: §4.2 · pp. 5–6
  • Edit-sample quality control filtered for compilable projects with multiple iterations and used expert ratings to retain clear instructions and excellent modifications.Five PhD students with three years of front-end development experience rated instruction clarity and modification effectiveness, resolving classifications by majority vote.Source: §4.2 · p. 6

4.3 Data Annotation

  • Repair samples were manually fixed after screening for display problems, and the benchmark classifies issues into six types: occlusion, crowding, text overlap, alignment, color and contrast, and overflow.Five PhD students identified issues, and each repaired an assigned share of the 111 webpages; issue categories were refined through consensus and updated when new types appeared.Source: §4.2 · pp. 6–7, §4.3 · pp. 7–8
  • The annotators classified edit operations as Add, Change, or Delete and assessed difficulty by adjustment count, interdependencies, and cascading changes.They also annotated six adjusted UI attribute types: text, color, position, size, shape, and component-level.Source: §4.3 · p. 7

4.4 Benchmark Statistics

  • DesignBench covers more than 11 webpage topics, varied edit operations and attributes, and six UI issue types, with alignment the most prevalent issue at 42.2%.Edit operations are Change (55.8%), Add (38.1%), and Delete (6.1%); attributes include component-level (27.9%), text (17.1%), position (16.0%), color (15.2%), shape (13.4%), and size (10.4%). Other issue shares are crowding (18.7%), occlusion (18.1%), and overflow (11.4%).Source: §4.4 · p. 8, Figure 4 · p. 8

5 Experiment Setup

Section 5 describes the prompts, models, contamination checks, evaluation metrics, and research questions used to assess MLLM performance across front-end tasks and frameworks.

5.1 Prompt Design

  • Prompts specify task roles and requirements, framework-specific coding conventions, and strict output formats for generation, editing, and repair.Editing prompts instruct models to change only specified elements; repair prompts address occlusion, crowding, overlap, alignment, color or contrast, and overflow. Detailed prompts are available in the code.Source: §5.1 · p. 9

5.2 Models

  • The evaluation studies six MLLMs: Pixtral [6], Qwen [10], LLama, Gemini, GPT, and Claude.Temperature was set to 0, output-token limits to each model’s maximum, and other parameters left at defaults; average API cost was $52 per model, with per-sample processing times of 49 seconds for generation, 29 for editing, and 25 for repair.Source: §5.2 · p. 9

5.3 Data Contamination Check

  • The authors assess contamination risk and use BLEU scores between generated and original webpage code to check whether models copy existing code.They report that open-source models’ reviewed training data did not include web HTML code, edit training data lacked instruction-image-framework-code triples, and public ground-truth repair labels were unavailable; the reported low BLEU scores indicate generation from design images rather than direct copying.Source: §5.3 · p. 9, Table 3 · p. 10

5.4 Metric

  • Evaluation uses code metrics—compilation success rate (CSR), code modification location similarity (CMLS), and code modification content similarity (CMCS)—plus GPT-4o judgments of whether outputs meet design-edit requirements or resolve design-repair issues.Human validation of MLLM scores achieved average accuracy of 95.54% on 359 design-edit samples and 91.89% on 111 design-repair samples; inter-annotator Kappa scores were 0.8648 and 0.8428, respectively.Source: §5.4 · p. 10

5.5 Research Questions

  • The six research questions examine performance across tasks and frameworks, difficulty and context effects, limitations in framework-based webpage development, and task failures.The questions span generation, edit, and repair; compare development frameworks; and investigate both the influence of task difficulty and input context.Source: §5.5 · pp. 10–11

6 Experiment Results · 6.1 RQ1: How do MLLMs perform across distinct front-end tasks? · 6.2 RQ2: What is the comparative performance of MLLMs when applied to different development frameworks?

Claude-3.7, GPT-4o, Gemini-2.0, and Pixtral-124B lead across tasks, while larger variants generally outperform smaller ones. Generation is limited by compilation and visual accuracy; editing and repair by code localization and modification, with vanilla HTML/CSS outperforming framework implementations, especially Angular.

6.1 RQ1: How do MLLMs perform across distinct front-end tasks?

  • Larger variants consistently outperform smaller counterparts, with the advantage especially pronounced in complex tasks requiring code localization and visual understanding.The paper illustrates this pattern with Llama-90B versus Llama-11B, Pixtral-124B versus Pixtral-12B, and Qwen-72B versus Qwen-7B.Source: §6.1 · p. 11
  • Angular generation compilation rates are 0.6867–0.7590 for top models, versus above 0.83 for React and Vue; even the best CLIP scores are only 0.5963–0.6324.The results identify compilation errors and visual inaccuracies as the two primary Design Generation bottlenecks.Source: §6.1 · pp. 11–13
  • Claude-3.7, GPT-4o, Gemini-2.0, and Pixtral-124B are the top-performing MLLMs across the three tasks, with Claude-3.7 highest on most metrics.Claude-3.7 records Design Generation CLIP scores of 0.6024–0.8319 and Design Edit MLLM scores of 8.01–9.15; GPT-4o, Gemini-2.0, and Pixtral-124B also show strong results across tasks.Source: §6.1 · p. 11, Table 5 · p. 12
  • Claude-3.7’s Design Edit CMLS scores are 0.5555–0.7696 and CMCS scores are 0.4551–0.7080, reflecting difficulty locating and modifying target code regions.The paper identifies code localization and modification as the primary bottleneck in Design Edit, despite successful compilation.Source: §6.1 · pp. 12–13

6.2 RQ2: What is the comparative performance of MLLMs when applied to different development frameworks?

  • Vanilla HTML/CSS achieves CLIP scores above 0.72 and perfect compilation, while Angular scores 0.45–0.55 in CLIP and 0.6–0.7 in compilation; React and Vue are intermediate.The results rank vanilla implementations highest and identify framework syntactic complexity as exacerbating compilation and visual-rendering challenges.Source: §6.1 · p. 12, Figure 6 · p. 13, §6.2 · p. 13

6.1 RQ1: How do MLLMs perform across distinct front-end tasks?

  • Claude-3.7’s Design Repair CMLS scores are 0.5089–0.6759 and CMCS scores are 0.3924–0.5859, indicating continued difficulty localizing and semantically modifying target code.These limitations persist even when generated code compiles successfully.Source: §6.1 · p. 13

6.3 RQ3: How does varying task difficulty impact MLLM performance? · 6.4 RQ4: To what extent do different input contexts affect MLLMs’ performance?

Across DesignBench’s three tasks, performance declines as task difficulty increases, with the steepest barriers in repair; for edit and repair, code-only input outperforms image-only input, while combining them adds little improvement.

6.3 RQ3: How does varying task difficulty impact MLLM performance?

  • Generation difficulty is scored as the equal-weight mean of image size, UI-element count, color variety, and layout complexity; scores above 80 are complex, below 30 simple, and 30-80 medium.Each component is normalized to 0-100 before calculating the score.Source: §6.3 · pp. 13–14
  • CLIP generation scores fall from 0.79-0.83 (Easy) to 0.69-0.73 (Hard); MLLM edit scores drop from 8.64-9.05 to 7.51-8.19, and repair scores from 6.93-7.21 to 4.19-6.93.Visual complexity moderately affects generation, operational complexity more strongly affects editing, and repair produces the steepest degradation; smaller models can fail catastrophically.Source: Table 6 · p. 14, §6.3 · p. 14
  • Edit difficulty follows the complexity and scope of modification instructions, using annotator-provided labels from Section 4.3 as ground truth.Source: §6.3 · p. 14
  • Repair difficulty is based on issue severity and required code changes: more than 30 lines is hard, fewer than 10 is easy, and intermediate cases are medium.Source: §6.3 · p. 14

6.4 RQ4: To what extent do different input contexts affect MLLMs’ performance?

  • For edit and repair, top models score higher with code-only than image-only input: edit 8.40-8.43 versus 7.37-7.67, and repair 6.53-6.70 versus 5.47-5.81.The study compares image and code contexts for both tasks; code-only superiority holds across models and tasks.Source: §6.4 · pp. 14–15, Table 7 · p. 15
  • Combining code and image inputs yields no significant gains and sometimes slightly degrades performance, reflecting difficulty localizing edit points and identifying UI issues visually.The authors find that code supplies more semantic information than visual data for edit and repair.Source: §6.4 · p. 15

6.5 RQ5: The limitations of MLLMs when developing framework-based webpages?

MLLMs show framework-specific syntax weaknesses, low compilation-repair and component-based implementation rates, and poor UI-issue detection. These findings indicate limitations in both generating reusable framework code and evaluating rendered interfaces.

6.5.1 Compilation Errors.

  • Compilation errors indicate framework-specific syntax limitations: MLLMs struggle with JSX parsing and React-specific expressions in React, template structure and attributes in Vue, and TypeScript modules and component architecture in Angular.Qwen-7B had 102 Vue errors and 73 Angular errors; Claude-3.7 had 1 Vue error, and Pixtral-124B performed best on Angular. GPT-4o had only 3 total errors.Source: §6.5.1 · pp. 15–16
  • The average compilation-error repair rate across models and frameworks was 0.53, indicating that MLLMs still struggled to fix front-end errors.The evaluation sampled 30 webpages with diverse compilation errors and prompted MLLMs to repair them.Source: §6.5.1 · p. 16, Table 8 · p. 16

6.5.2 Component-based Implementation Limitation.

  • Component-based implementation rates averaged 0.24% on React, 5% on Vue, and 19% on Angular; generated Vue code repeated item structures instead of using v-for.The BBC news-section case study illustrates the lack of reusable, framework-specific implementation.Source: §6.5.2 · p. 16, Table 9 · p. 16

6.5.3 Issue Detection Limitation.

  • 0.2714 was the overall UI issue-detection accuracy across all models and frameworks; average accuracy was 0.2972 on React, 0.2205 on Vue, 0.2275 on Angular, and 0.3403 on Vanilla.The reported framework averages show that issue detection was poor across all four settings.Source: §6.5.3 · p. 17, Table 10 · p. 17

6.6 RQ6: What mistakes do MLLMs make on the three tasks?

MLLMs exhibit distinct task-specific failure patterns: generation is dominated by spatial and structural errors, editing by scope-control errors, and repair by failures to identify defects and formulate corrections.

6.6 RQ6: What mistakes do MLLMs make on the three tasks?

  • Failure analysis used a random 25% sample, with annotators iteratively refining categories and reaching consensus.Continuous communication during annotation helped identify emerging failure types.Source: §6.6 · p. 17
  • The analysis classifies generation failures as compile errors, layout disorder, missing elements, or incorrect element position, color, text, or size; edit and repair failures include compile errors, no action, wrong targets or changes, partial changes, and unnecessary changes.The study examined a random 25% of results and refined failure categories through iterative discussion and consensus annotation.Source: §6.6 · pp. 17–18
  • Generation errors center on wrong element size, missing elements, and wrong position; editing on unnecessary or partial changes; repair on no, wrong, or partial repairs.These patterns point to challenges in layout comprehension, precise instruction following and scope control, and defect identification and corrective action formulation, respectively.Source: §6.6 · p. 18, Figure 9 · p. 18

7 Threats to Validity

The authors address internal-validity concerns about judge reliability and data leakage, and delimit external validity to three dominant frameworks. They report human-evaluation support and explain the benchmark’s framework coverage.

Internal validity

  • Human evaluation found MLLM scoring accuracy above 90%; closed-application webpages, manually written repair ground truth, and low contamination risk address data-leakage concerns.The authors also point to weak framework-based generation, including syntax errors and limited framework-specific features, as evidence against results arising from memorization.Source: §7 · p. 18

External validity

  • The benchmark covers React, Vue, and Angular, selected for their dominance and diverse paradigms, from JSX to templates and TypeScript.The authors note that only a limited set of frameworks is included, while these three represent the majority of contemporary applications.Source: §7 · p. 18

8 Discussion

The discussion recommends improving framework-specific training and visual-code fusion, while helping developers reduce localization, diagnosis, and task-complexity burdens.

Discussion

  • Researchers should enrich MLLM training with modern framework patterns and best practices, and improve visual-code alignment because visual inputs are underused in edit and repair tasks.The supplied findings indicate single-code and multimodal inputs perform equally well on those tasks.Source: §8 · p. 19
  • Developers should specify edit locations and repair issues, and decompose complex instructions and large designs into smaller tasks and components.These practices target weak code localization and issue identification, as well as performance degradation with complex instructions and large UI mockups.Source: §8 · p. 19

9 Conclusion

DesignBench evaluates MLLMs in framework-based front-end development across multiple frameworks and tasks, using experiments to identify limitations and inform future work.

9 Conclusion

  • DesignBench introduces a multi-framework benchmark covering React, Vue, and Angular, and pioneers design editing and repair alongside traditional design generation.It also examines vanilla HTML/CSS.Source: §9 · p. 19
  • Experiments across task complexity, framework compatibility, difficulty, contextual factors, and code-level analysis reveal MLLM limitations and actionable advice for researchers and developers.Source: §9 · p. 19
Loading 2506.06251v3…