PaperlayerView exact revision on arXiv

Source-linked AI summary

Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs

Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, Zhiqiang Shen

arXiv:2406.20098v2cs.CVcs.AIcs.CL

TL;DR

Existing MLLMs are poor at understanding webpage screenshots and generating faithful HTML, despite strong performance in other modalities. Web2Code combines large-scale instruction data with WUB and WCGB evaluation, and experiments show improved webpage-specific and general visual abilities. Its coverage may remain incomplete because the dataset can be biased, omit HTML scenarios, and include privacy-sensitive human webpages.

  • Problem

    Existing MLLMs and benchmarks insufficiently support webpage screenshot understanding and webpage-to-code generation, which require OCR, spatial reasoning, and long-text generation together.

    Source: §1 · pp. 1–2

  • Method

    Web2Code combines 1179.7k webpage instruction-response pairs with WUB question answering and WCGB rendered-image HTML evaluation, using GPT models to generate and refine data.

    Source: §1 · p. 2, §3 · pp. 3, 5

  • Results

    Finetuning on Web2Code improves image-to-HTML translation, webpage understanding, and general visual proficiency, while the dataset outperforms existing datasets in the reported experiments.

    Source: §1 · p. 2, §4.2 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    Web2Code provides a large webpage-focused resource and evaluation suite for developing MLLMs aimed at web content generation and task automation.

    Source: §1 · p. 2, §6 · p. 10

    The dataset may be biased, may not cover all HTML coding scenarios, and includes some privacy-sensitive webpages containing humans.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio in a variety of understanding and generation tasks. However, current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. To address this problem, we propose $\texttt{Web2Code}$, a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning and an evaluation framework for the webpage understanding and HTML code translation abilities of MLLMs. For dataset construction, we leverage pretrained LLMs to enhance existing webpage-to-code datasets as well as generate a diverse pool of new webpages rendered into images. Specifically, the inputs are webpage images and instructions, while the responses are the webpage's HTML code. We further include diverse natural language QA pairs about the webpage content in the responses to enable a more comprehensive understanding of the web content. To evaluate model performance in these tasks, we develop an evaluation framework for testing MLLMs' abilities in webpage understanding and web-to-code generation. Extensive experiments show that our proposed dataset is beneficial not only to our proposed tasks but also in the general visual domain. We hope our work will contribute to the development of general MLLMs suitable for web-based content generation and task automation. Our data and code are available at https://github.com/MBZUAI-LLM/web2code.

1 Introduction

Existing MLLMs handle many multimodal tasks but struggle to understand webpage screenshots and generate faithful HTML. Web2Code addresses this gap with a large instruction dataset and evaluation suite, improving webpage-specific abilities while preserving broader multimodal performance.

Motivation

  • Web2Code targets a gap in existing MLLMs, which underperform on the combined OCR, spatial reasoning, and long-text generation demands of webpage understanding and webpage-to-code translation.Previous datasets lack instruction information, while existing benchmarks often test these capabilities separately rather than in combination.Source: §1 · pp. 1–2

Contribution

  • Web2Code provides 1179.7k instruction-response pairs containing HTML code plus structured webpage questions and answers.GPT-3.5 and GPT-4 clean existing data and generate new webpages for the dataset.Source: §1 · p. 2
  • The evaluation suite introduces WUB for webpage question answering and WCGB for HTML generation, with WCGB judging rendered webpage fidelity through GPT-4V rather than text similarity.This design evaluates webpage understanding and visual code-generation quality directly.Source: §1 · p. 2

Results

  • Finetuning LLaVA-style MLLMs on Web2Code improves image-to-HTML translation and webpage screenshot perception and reasoning.The authors report that these gains occur without compromising existing general multimodal abilities.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work advanced general multimodal instruction tuning and MLLM architectures, while code-generation studies addressed robustness, security, detection, and efficiency. These lines of research leave webpage-specific multimodal code generation as the focus of Web2Code.

MLLM Dataset

  • Existing MLLM instruction datasets use GPT-generated image-text and dialogue data to improve perception, reasoning, planning, and related capabilities.Representative resources include SVIT, LRV-Instruction, ShareGPT4V, LLaVAR, and LVIS-Instruct4V.Source: §2 · pp. 2–3

MLLM

  • MLLM research explores attention and non-attention vision-language mappings, stronger language backbones, higher image resolution, image crops, supplementary vision encoders, and task-specific vision models.Examples include BLIP2, InstructBLIP, Qwen-VL, LLaVA, LLaVA-NeXT, and SpatialVLM.Source: §2 · p. 3

Code Study

  • Code-generation research studies syntactic robustness, code leakage, synthetic-code detection, and computational efficiency, including the Mercury benchmark and Beyond metric.These studies address general code assistance concerns rather than webpage screenshot-to-HTML translation.Source: §2 · p. 3

3 Dataset Construction

Web2Code constructs instruction data by generating and refining webpage code-image pairs, adding webpage question-answer data, and cleaning WebSRC examples. Its statistics cover dataset composition, structural content, benchmark design, and scale.

3 Dataset Construction

  • The dataset construction pipeline has four components: new HTML image-code pairs, refined code-generation data, new webpage question-answer pairs, and refined WebSRC understanding data.These components are designed to support instruction-following training for both webpage generation and understanding.Source: §3 · p. 3
  • GPT-3.5 generates 60K HTML pages, which Selenium renders into screenshots and converts into LLaVA-style instruction data; prompting adds Modern and Bootstrap styles.The synthetic generation process uses ten quality and diversity criteria.Source: §3 · pp. 3–4
  • GPT-4 refines Pix2Code and WebSight by replacing random letters with meaningful text, rendering screenshots, and preserving Modern or Bootstrap webpage styles.The refinement covers product landing pages, portfolios, blogs, and other categories.Source: §3 · p. 4
  • DWU contains 243.5K GPT-4-generated question-answer pairs from 24.35K webpages, with ten questions per webpage and image-plus-question inputs used for training.The questions target visual and content properties reflected in generated webpage images.Source: §3 · pp. 4–5
  • DWUR removes duplicate WebSRC questions and uses GPT-4 to improve answers, producing 51.5K higher-quality instruction examples.This refinement targets relevance, answer quality, and complex web-based instruction following.Source: Figure 3 · p. 4, §3 · p. 5

3.1 Statistics and Analysis

  • The dataset emphasizes webpage structure and design, with frequent answer terms including section, color, button, and website, and common HTML tags such as <div>, <p>, <meta>, <img>, and <a>.The presence of headings, inputs, and document-level tags is described as reinforcing structural completeness.Source: §3.1 · p. 5
  • WCGB evaluates rendered HTML images with GPT-4V, whereas WUB evaluates answers to webpage question-answer pairs.The two benchmarks therefore assess visual webpage generation fidelity and webpage understanding through distinct evaluation procedures.Source: Figure 6 · p. 6

3.2 Distribution

  • Web2Code contains 1,179.7K instruction points: 884.7K website image-code pairs and 295K question-answer pairs.The QA portion comprises 243.5K DWU pairs and 51.5K WebSRC-derived pairs; the evaluation set contains 1,198 webpage screenshots and 5,990 yes/no WUB pairs.Source: §3.2 · p. 6, Table 2 · p. 6

4 A New Evaluation Framework for Webpage

Web2Code evaluates webpage understanding and HTML generation with complementary question-answer and image-level code-generation benchmarks. Experiments show incremental dataset components improve both capabilities, while visualizations indicate better rendered fidelity with code-enhanced backbones and web data.

4 A New Evaluation Framework for Webpage

  • Web2Code introduces WUB for yes/no webpage understanding and WCGB for HTML generation evaluated by comparing rendered predictions with ground-truth images.WUB contains 5,990 QA pairs from 1,198 screenshots; WCGB scores ten visual criteria grouped into four categories on a 0–10 scale.Source: §4.1 · p. 7

4.1 Evaluation Metric for HTML Code Generation

  • Image-level evaluation prioritizes rendered webpage fidelity over code similarity because different HTML implementations can produce identical webpages.Predicted HTML is rendered with Selenium WebDriver and assessed through visual comparison using GPT-4 Vision.Source: §4.1 · pp. 6–7

4.2 Quantitative Evaluation for HTML Code Generation of MLLMs

  • 8.530 WCGB overall rises stepwise from 1.79 as LLaMA-3 adds Web2Code components to general-domain training.The reported progression is 1.79→6.402→6.716→7.806→8.530; dataset components contribute improvements on both WCGB and WUB.Source: §4.2 · p. 7, Table 3 · p. 8, Table 4 · p. 8
  • LLaMA3-8B achieves the best combined webpage code-generation and webpage-understanding performance, with DWU, DWCGR, and DWUR providing complementary gains.DWCG improves code generation but requires more webpage understanding; DWU recovers and enhances both, DWCGR primarily boosts WCGB, and DWUR improves all metrics.Source: §4.2 · pp. 7–8, Table 4 · p. 8

4.3 Visualizations for Qualitative Evaluation

  • CrystalChat-7B generates higher-quality webpages than Vicuna1.5-7B, and web-trained outputs closely resemble ground-truth layouts and styles.The visualizations also test adaptation to hand-drawn webpage inputs.Source: §4.3 · p. 8, Figure 7 · p. 9, Figure 8 · p. 9

5 General Evaluation of MLLMs Using Web2Code

The study evaluates Web2Code-trained LLaVA-style MLLMs across general visual-language benchmarks and multiple backbones. Web2Code preserves general-domain performance and can improve perception, reasoning, and anti-hallucination metrics.

5 General Evaluation of MLLMs Using Web2Code

  • Web2Code can be combined with LLaVA instruction data without harming general visual-language performance across tested configurations.The evaluation uses MME, ScienceQA, TextVQA, and POPE alongside varied LLM backbones.Source: §5 · p. 9
  • DWU improves CrystalChat on POPE from 86.86 to 87.10, ScienceQA from 67.77 to 68.27, and TextVQA from 57.84 to 58.15.These results show comparable or better performance after adding webpage-understanding data.Source: §5 · p. 10
  • DWCG raises CrystalChat’s MME-P and MME-C results by 40.31 and 5.00 points, respectively, despite targeting webpage code generation.The reported gains indicate that code-generation data can also improve visual-language understanding.Source: §5 · p. 10
  • Refined DWUR and DWCGR preserve comparable overall benchmark performance and improve ScienceQA by 0.4 points.The refined datasets remain effective in the general visual-language domain.Source: §5 · p. 10

6 Conclusion

Web2Code provides a 1.18M-entry webpage-to-code instruction dataset and an evaluation suite that improve webpage and general visual capabilities. The authors position it as a resource for web-content generation and task automation while noting coverage, privacy, and evaluation limitations.

6 Conclusion

  • Web2Code combines a 1.18M-entry webpage-to-code instruction dataset with an evaluation suite for webpage understanding and HTML translation.The dataset uses balanced synthetic generation and diverse existing data to mitigate potential biases.Source: §6 · p. 10
  • Experiments show Web2Code improves webpage capabilities and general visual proficiency, whereas existing datasets produce inferior performance.The authors frame the benchmark as supporting progress toward foundation models for content generation and task automation.Source: §6 · p. 10
  • Web2Code may omit HTML scenarios, include privacy-sensitive webpages, and fail to cover complex real-world code cases comprehensively.The evaluation framework also may not capture code efficiency, readability, or adherence to best practices.Source: §6 · p. 10

A Training Details and Hyperparameters

The study follows a LLaVA-1.5 training setup with frozen vision and language components during projector pretraining, then joint projector and LLM instruction tuning. General-domain evaluation covers multiple benchmarks and backbones, with CrystalChat showing strong text-image performance.

A Training Details and Hyperparameters

  • During pretraining, caption data optimizes the projector while the vision encoder and LLM remain frozen; instruction tuning optimizes both the projector and LLM.Source: A Training Details and Hyperparameters · p. 15
  • Training runs for one epoch in each phase with batch sizes 256 for pretraining and 128 for instruction tuning.The learning rates are 1e−3 and 2e−5 with cosine decay, warmup, AdamW, and no weight decay.Source: A Training Details and Hyperparameters · p. 15

B More Effects of Web2Code on General Domain

  • General-domain evaluation uses MME, ScienceQA, TextVQA, and POPE to assess perception, reasoning, text-oriented VQA, and anti-hallucination.The evaluation is conducted after instruction tuning with Web2Code across different backbones and data configurations.Source: B More Effects of Web2Code on General… · p. 15
  • CrystalChat, Vicuna1.5, and LLaMA2 outperform CrystalCoder and CodeLlama, while CrystalChat leads TextVQA and exceeds CodeLlama by 132.89 MME-P points.All compared models use the LLaVA + DWU + DWCG configuration.Source: B More Effects of Web2Code on General… · p. 15, Table 6 · p. 15

C Data collection of Web2Code

Web2Code emphasizes broad webpage diversity and privacy-aware data curation. The dataset combines varied webpage sources with systematic removal, anonymization, and human auditing of sensitive information.

Diversity

Privacy

  • Privacy protection removes or anonymizes personal data, sensitive fields, and potentially identifying information from the collected webpages.The curation primarily uses publicly available datasets that require no authentication.Source: C Data collection of Web2Code · pp. 15–16
  • A human audit of 1,000 sampled data points found privacy-sensitive issues in only 2.7% of the data, including 16 email addresses and 11 phone numbers.No credit card numbers were found in the sample.Source: C Data collection of Web2Code · p. 16

Evaluation

  • Table 8 reports LLaMA3-8B performance using low-level element-matching metrics and CLIP-based visual-text alignment scores.The table provides the evaluation framework used for assessing generated webpage fidelity.Source: Table 8 · p. 16

D More Analysis on Other Metrics for HTML Code Generation of MLLMs

The analysis broadens HTML-generation evaluation beyond a single metric by combining low-level element matching with CLIP-based visual-text alignment. DWU improves block-match, position, and color attributes, while refined datasets improve all evaluated aspects.

More Analysis on Other Metrics for HTML Code Generation of MLLMs

  • DWU improves block-match, position, and color attributes, while DWCGR and DWUR improve performance across all evaluated aspects.The evaluation combines Design2Code metrics with CLIP-based visual-text alignment assessments.Source: D More Analysis on Other Metrics for… · p. 17

E More Comparisons with Existing Datasets

Comparisons with WebSight show that Web2Code data is more effective across webpage-specific, visual-fidelity, and general-domain evaluations. A smaller generated dataset can outperform a much larger WebSight baseline.

More Comparisons with Existing Datasets

F Qualitative Data Examples in WUB Benchmark

The WUB benchmark uses diverse yes/no questions to probe webpage understanding, while the accompanying materials document GPT-based generation, evaluation prompts, and representative instruction-tuning examples. The data samples span code generation, webpage understanding, refined datasets, and multiple webpage styles.

F Qualitative Data Examples in WUB Benchmark

G.1 Prompt Used to Question and Answer Generation for DWU Data

  • The DWU question-answer generation process uses a GPT4 prompt, as documented in Figure 12.The prompt is specifically associated with generating question-answer pairs for DWU data.Source: Figure 12 · p. 19

G.2 Prompts Used For Instruction generation of DWCG and DWCGR

  • Figures 13 and 14 document GPT4 prompts for generating DWCG and DWCGR instructions, including webpage-style instructions.The generated instructions are shown as examples in Figure 17.Source: Figure 13 · p. 20, Figure 14 · p. 20

G.3 Prompt Used For GPT4-Vision Evaluation in WCGB benchmark

  • Figure 15 documents the GPT4-Vision prompt used for evaluation in the WCGB benchmark.The figure identifies the prompt’s evaluation role but does not report evaluation outcomes.Source: Figure 15 · p. 21

G.4 Prompt Used For QA Generation for WUB benchmark

  • Figure 16 documents the GPT4-Vision prompt used to generate yes/no question-answer pairs for WUB.This prompt complements the qualitative WUB examples.Source: Figure 16 · p. 22

H Data samples

Loading 2406.20098v2…