Source-linked AI summary
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
Haoyu Sun, Huichen Will Wang, Jiawei Gu, Linjie Li, Yu Cheng
TL;DR
Existing webpage benchmarks often separate perception and code generation and omit conceptual design. FullFront evaluates design, perception QA, and code generation using reconstructed real-world webpage data; tests reveal persistent perception and code-generation weaknesses and a substantial gap from human experts.
Problem
Existing webpage benchmarks often address perception or code generation separately and omit the initial conceptual design phase, leaving MLLM evaluation across the full front-end workflow absent.
Source: §1 · p. 2Method
FullFront benchmarks design, webpage perception QA, and code generation, using a two-stage process to turn real-world webpages into clean, standardized HTML.
Source: §1 · p. 2, §3.1 · p. 4Results
Across tasks, MLLMs show weak fine-grained webpage perception and code-generation difficulties; Claude 3.7 Sonnet's average perception-QA accuracy is below 55%, versus human performance above 95%.
Source: §1 · p. 3Takeaways & Limitations
Current MLLM reliability and stability in front-end engineering remain uncertain given perceptual deficiencies and potential critical code-generation errors.
Source: §5.3 · p. 9Current perceptual deficiencies and potential critical code-generation errors leave MLLM reliability and stability uncertain.
Source: §5.3 · p. 9
Abstract
from arXiv · showhide
Front-end engineering involves a complex workflow where engineers conceptualize designs, translate them into code, and iteratively refine the implementation. While recent benchmarks primarily focus on converting visual designs to code, we present FullFront, a benchmark designed to evaluate Multimodal Large Language Models (MLLMs) \textbf{across the full front-end development pipeline}. FullFront assesses three fundamental tasks that map directly to the front-end engineering pipeline: Webpage Design (conceptualization phase), Webpage Perception QA (comprehension of visual organization and elements), and Webpage Code Generation (implementation phase). Unlike existing benchmarks that use either scraped websites with bloated code or oversimplified LLM-generated HTML, FullFront employs a novel, two-stage process to transform real-world webpages into clean, standardized HTML while maintaining diverse visual designs and avoiding copyright issues. Extensive testing of state-of-the-art MLLMs reveals significant limitations in page perception, code generation (particularly for image handling and layout), and interaction implementation. Our results quantitatively demonstrate performance disparities across models and tasks, and highlight a substantial gap between current MLLM capabilities and human expert performance in front-end engineering. The FullFront benchmark and code are available in https://github.com/Mikivishy/FullFront.
1 Introduction
FullFront evaluates MLLMs across the front-end workflow, addressing prior benchmarks’ omission of conceptual design and limited coverage of perception and implementation. Its results show persistent gaps in webpage design, perception, and code generation, including performance below human experts.
1 Introduction
- Earlier benchmarks commonly separate visual QA from code generation, often omit design conceptualization, and may miss fine-grained layout perception or interactive implementation.The introduction cites IW-Bench [1] and WebCode2M for code generation, and WebQuest [3] and Webqa for webpage understanding.Source: §1 · p. 2
- FullFront evaluates Webpage Design, Webpage Perception QA, and Webpage Code Generation across the front-end workflow using reconstructed real-world webpages and multifaceted visual and code-level metrics.The tasks cover design conceptualization, perception of visual organization and spatial relationships, and functional code generation, including interactions and code refinement.Source: §1 · pp. 2–3
- Claude 3.7 Sonnet averages below 55% accuracy on Webpage Perception QA, while human performance exceeds 95%; code-generation models also struggle with images, layout, and interactions.For Webpage Design, text-to-image MLLMs can produce general layout concepts but lack precision for high-fidelity designs matching detailed descriptions.Source: §1 · pp. 2–3
- The benchmark combines fine-grained visual similarity and code-level metrics, and its evaluation highlights deficiencies in visual perception and code generation, especially for complex layouts, images, and interactive functions.The reported performance disparity between proprietary and open-source models is also identified as a key finding.Source: §1 · pp. 2–3
2 Related Work
Prior MLLM web research spans website interaction, webpage QA, and screenshot-to-code generation. These efforts use varied tasks and data-construction approaches.
Applications of MLLMs in Web
Webpage Benchmarks and Datasets
- Webpage benchmarks cover element grounding and content-reasoning QA [15] [3] [16] [17] [18], screen summarization, and broader QA [20].Screenshot-to-code datasets use scraped and simplified code [21] [1], LLM-generated code [22] [23], or stochastic UI generation; MRWeb [25] addresses external resources and Interaction2Code [26] dynamic pages.Source: §2 · p. 3
3 Benchmark
FullFront curates design, perception-QA, and code-generation data from textual descriptions and real-world webpages, then evaluates outputs with visual and code-level metrics.
3.1 Data Curation
- Webpage Design uses 50 textual descriptions of synthetic webpages, asking text-to-image MLLMs to produce designs and make reasonable choices where specifications are incomplete.The descriptions are sampled from the Text to Code dataset.Source: §3.1 · pp. 3–4
- Webpage Perception QA tests element position, style, spatial relations, and layout through real-world, synthetic, and multi-window QA sets.Real-world QA uses 625 screenshots and 1,250 pairs; synthetic QA has 400 pairs; multi-window QA has 150 pairs. Questions are manually reviewed, and screenshots are checked to remove personal data and harmful content.Source: §3.1 · p. 4
- The code-generation dataset transforms real webpage screenshots into HTML-v1 with GPT-4o, then Claude 3.7 Sonnet refines it into HTML-v2 ground truth, using category-based image handling.The authors say this pipeline yields greater webpage complexity and diversity than other benchmarks.Source: §3.1 · p. 4, Figure 3 · p. 4
- Four code-generation subtasks test image-to-code, text-to-code, interaction authoring, and code refinement using 200, 50, 100, and 50 samples, respectively.Interaction authoring starts from HTML-v1 and adds specified behaviors; code refinement improves provided HTML-v1 to match an HTML-v2 rendering.Source: §3.1 · pp. 4–5
3.2 Evaluation Metrics
- Visual evaluation compares outputs with ground-truth images using CLIP Score and a ten-criterion Gemini Visual Score; code evaluation uses the proposed DOM- and CSS-based Code Score.Code Score combines DOM-tag structural similarity with content-type similarity and implementation rates; design uses visual metrics, perception QA uses accuracy, and code generation uses visual metrics plus Code Score.Source: §3.2 · p. 5
4 Experiments
The experiments evaluate webpage design, perception QA, and code generation, using state-of-the-art MLLMs and a mini dataset for rapid evaluation. Across tasks, results show limits in design fidelity and perception, alongside strong differences between open- and closed-source code-generation performance.
4.1 Evaluation Settings
- Evaluation covers 10 state-of-the-art MLLMs on perception QA and code generation, while webpage design tests GPT-4o and gemini-2.0-flash-exp-image-generation.The results for o1 and Gemini 2.5 Pro are reported only on FullFront-mini, which was constructed for rapid iterative evaluation.Source: §4.1 · p. 5
4.2 Main Results
- GPT-4o outperforms gemini-2.0-flash-exp-image-generation on both Gemini Score and Human Score, though text-to-image models struggle with high-fidelity designs.Qualitative examples also show GPT-4o performing better on page structure, typography, and element implementation fidelity.Source: §4.2 · pp. 5–6, Table 1 · p. 5
- On FullFront-mini, Claude 3.7 Sonnet and Gemini 2.5 Pro average barely above 50% across perception subtasks, while LLaVA-Onevision-72B stays below 35% on each.Human accuracy exceeds MLLMs by 44.5%, 38%, and 38% on the three subtasks; the table covers three perception QA tasks.Source: §4.2 · p. 6, Table 2 · p. 6
- InternVL3-78B leads on full-benchmark Synthetic QA at 53.75% and Multi-window QA at 46.00%, with performance declining on the more complex multi-window task.Models perform nearly identically on single-page Real-world and Synthetic QA, and no substantial open- versus closed-source disparity is reported for perception QA.Source: Table 2 · p. 6, §4.2 · p. 7
- Closed-source models outperform open-source models across all code-generation subtasks and metrics, with no open-source model in any category’s top two.Interaction authoring and text-to-code are especially challenging; the latter produces the lowest overall model performance.Source: Table 3 · p. 6, §4.2 · p. 7
- Code refinement performs better than image-to-code, while interaction authoring and autonomous text-to-code are harder; closed-source models exceed 65% interaction success versus 16% for LLaVA-Onevision-72B.The code-generation results cover four tasks, and blind human evaluation on FullFront-mini frequently gives leading proprietary models above 8/10 for reproduction quality.Source: Table 3 · p. 6, §4.2 · p. 7, Table 5 · p. 7
5 Discussion
The discussion identifies frequent errors in fine-grained perception and code generation, and shows that perception QA performance does not always align with implementation. Human evaluations find generated pages can surpass real-world counterparts, but errors and weak text and form implementation leave reliability uncertain.
5.1 Where do MLLMs struggle most in perceiving webpages?
- Among 200 questions all models except o1 and Gemini 2.5 Pro missed, alignment errors account for 21.5%, size 19.5%, positioning 18.5%, and spacing 15.5%.Illustrations show errors in locating a tag relative to headings and comparing button sizes.Source: §5.1 · p. 7, Figure 4 · p. 8
5.2 What is the relationship between perceptual ability and code performance?
- Perception QA and code-generation performance do not always align: models can misidentify an element’s position in QA yet place it correctly in generated pages.The authors suggest the two processes may have different sensitivities or internal representations and strategies, warranting future investigation.Source: §5.2 · p. 8
5.3 Can MLLMs be an excellent front-end engineer?
- In blind evaluation of 100 pages, leading models were judged superior to real-world counterparts in most cases, but code generation also produced abnormal image sizes, blank images, and isolated interactive elements.Open-source models show these errors markedly more often than closed-source models, reducing reliability and stability; the figure illustrates the three error types.Source: §5.3 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 9
- Text and form implementation similarity scores do not exceed 0.6, and the authors describe overall reliability and stability as uncertain despite generally coherent pages.They propose integrating specialized tools to compensate for perceptual limitations and identify and rectify generation anomalies.Source: §5.3 · p. 9, Table 7 · p. 9
6 Summary
FullFront evaluates the front-end development pipeline and uses diverse, categorized image representations to preserve visual information. FullFront-mini provides a smaller subset for rapid evaluation and exploration.
6 Summary
- FullFront evaluates design, perception, and code generation with synthetic data and a multi-layered system, highlighting challenges in fine visual details and interaction implementation.The authors identify future directions including more advanced metrics, a larger dataset, and new task types.Source: §6 · p. 9
A FullFront-mini
- FullFront-mini includes 200 real-world, 100 synthetic, and 50 multi-window QA samples, plus 20 image-to-code, 10 text-to-code, 20 code-authoring, 10 refinement, and 10 design samples.The code-authoring subset has two samples for each interaction type.Source: A FullFront-mini · p. 12
B.1 Category-based utitization strategy for images
- The image strategy assigns images to 15 predefined categories, each linked to a fixed, non-copyrighted URL, and requires models to set image dimensions and position.Categorized images represent screenshot-derived designs while retaining visual diversity and supporting straightforward dataset expansion.Source: §B.1 · p. 12
B.2 Subtask Specifications
The subtask specifications cover screenshot-to-code, text-to-code, interactive authoring, and code refinement. They define task inputs and implementation requirements for each.
B.2 Subtask Specifications
- Image to Code evaluates HTML replication from 200 screenshots rendered from HTML-v2.It is described as the most straightforward screenshot-to-code task.Source: §B.2 · p. 12
- Text to Code tests webpage generation from 50 textual descriptions of HTML-v2 pages, which Claude 3.7 Sonnet generated and people manually reviewed.Source: §B.2 · p. 12
- Interaction Authoring uses 100 static-page-derived samples with paired behavior screenshots; models return one HTML/CSS/JavaScript file and identify the primary element as #InteractionPart.Claude 3.7 Sonnet adds interaction code, followed by manual verification; models must reproduce the static page and depicted behavior.Source: §B.2 · pp. 12–14
- The six click interaction specifications cover dropdowns, checkboxes, background-color changes, modals, tooltips, and text inputs.Source: §B.2 · p. 13
- The four hover interaction specifications cover dropdowns, bold text, underlining, and tooltips.Source: §B.2 · p. 13
- Code Refinement tests whether models can use an HTML-v1 codebase and an HTML-v2 screenshot to refine the page toward HTML-v2 quality.The task contains 50 samples.Source: §B.2 · p. 14
C Evaluation Metrics Specifications
The evaluation pairs a ten-dimension visual rubric with a code score that measures structural similarity and the accuracy and completeness of matched content. The visual scores use a 0–10 scale, while the code score aggregates structural similarity and implementation-rate-adjusted content similarities.
C.1 Gemini Visual Score: Criteria and Rubric
- Gemini 2.5 Flash compares an MLLM-generated webpage image with its ground-truth image, assigning scores across ten visual dimensions on a 0-to-10 scale; 10 means perfect identity and 0 no discernible similarity.Source: §C.1 · p. 14
C.2 Code Score: Formulation and Components
- The Code Score compares generated and reference HTML by parsing both into DOM trees, extracting CSS, and aggregating multiple similarity measures.Source: §C.2 · p. 16
1. Structural Similarity
- Structural similarity compares generated and reference sequences of HTML tags using thresholded longest-common-subsequence length relative to the reference sequence.If the reference sequence is empty, the score is 1.0.Source: §1 · pp. 16–17
2. Content-Type Similarity
- Content-type scoring matches text, image, and form elements, calculates implementation rates from matched reference elements, and compares matched elements using type-specific similarity measures.Text comparisons include content and CSS style; image comparisons include URL, style, and alt text; form comparisons use attributes, style, and, where applicable, text.Source: §2 · pp. 17–18
3. Final Code Score Aggregation
- The final Code Score is a weighted sum of structural similarity and implementation-rate-adjusted similarities for text content, text style, images, and forms.Adjusting similarities by implementation rates penalizes incompleteness.Source: §3 · p. 18, §2 · p. 18
D.1 Webpage Design
In a representative Webpage Design example, GPT-4o’s output more closely resembles the target webpage than gemini-2.0-flash-exp-image-generation’s output. GPT-4o better reproduces major layout and elements, but fine-grained visual discrepancies remain.
D.1 Webpage Design
- GPT-4o better matches the target’s major page structure, component placement, element completeness, and typography.The comparison notes its header, hero section, category grid, and footer arrangement.Source: §D.1 · pp. 18–19
- GPT-4o’s design more closely resembles the target than gemini-2.0-flash-exp-image-generation’s design in the representative comparison.Source: §D.1 · p. 18, Figure 9 · p. 19
- Despite the closer overall match, GPT-4o’s design retains fine-detail differences, including a light footer where the target has a dark one.The example concludes that precise adherence to detailed visual attributes remains an area for improvement.Source: §D.1 · p. 19
D.2 Webpage Code Generation
Closed-source models generally reproduce page layout and elements better than open-source counterparts, but even leading models miss fine-grained design details. The examples also show that a perceptual error need not recur in generated code.
D.2 Webpage Code Generation
- Closed-source models generally outperform open-source counterparts in page layout and element reproduction, with Claude 3.7 Sonnet achieving high visual similarity in component placement and stylistic consistency.The cited qualitative case study describes Claude 3.7 Sonnet's similarity to the label image as remarkably high.Source: §D.2 · p. 19
- Leading proprietary models still miss fine detail: examples include incorrect headline alignment, an unreproduced circular search field and banner gradient, and omitted header icons.The example also notes that the headline was centered instead of left-aligned.Source: §D.2 · pp. 19–20
- Despite failing a perception question about the “Human Rights Advocate” tag’s position, the depicted models correctly place it above the “NAVI PILLAY” title in generated code.Source: Figure 10 · p. 20
D.3 Correct Code Implementation Despite Perceptual Errors
For the “NAVI PILLAY” example, all evaluated MLLMs misidentified the tag’s position in perceptual QA, yet the depicted models placed it correctly in generated code.
D.3 Correct Code Implementation Despite Perceptual Errors
- All evaluated MLLMs misidentified the “Human Rights Advocate” tag’s position relative to the title and subtitle in perception QA, but the depicted outputs placed it correctly above the title in generated code.The rendered outputs come from webpage code-generation tasks.Source: §D.3 · p. 20