IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Web

1 Introduction

• Image-to-Web combines fine-grained component recognition, relative-position assessment, and web-code generation, yet large multimodal models have almost no domain-specific evaluation benchmark.Web pages also combine HTML, CSS, and JavaScript, making their functional and visual properties more complex than conventional programming challenges.Source: §1 · p. 1
• Conventional similarity metrics such as BLEU can misjudge web pages because different code can produce visually indistinguishable results and invisible elements vary substantially.These metrics therefore do not precisely assess element completeness or layout intricacies.Source: §1 · p. 2
• Element Accuracy assesses web-element completeness, while Layout Accuracy evaluates layout information; the benchmark describes both as metrics for web elements and layout.The introduction specifies that Element Accuracy uses DOM-tree parsing and examines six dimensions: tag, text content, attribute, style, JavaScript, and children; Layout Accuracy compares a common subsequence of traversed elements.Source: §1 · p. 2
• IW-BENCH contains 1,200 image–web-code pairs across three difficulty levels, with data curated to assess large multimodal models on Image-to-Web.The benchmark construction contribution describes the pairs as spanning three challenging levels.Source: §1 · p. 2
• The proposed five-hop multimodal Chain-of-Thought method is reported to enhance Image-to-Web performance, alongside extensive model evaluation and ablation.The five hops are SoM prompt injection, inferring elements, inferring layout, inferring web code, and reflection.Source: §1 · p. 2

2 Related work

• Prior multimodal benchmarks assess instruction-following and reasoning, whereas IW-BENCH targets large multimodal models’ generation capabilities for Image-to-Web across diverse visual contexts.The related-work discussion identifies WebSight as the only relevant benchmark to the authors’ knowledge, but says it is more like a fine-tuning dataset than an evaluation benchmark.Source: §2 · p. 2
• The related-work discussion points to a distinction between evaluation benchmarks and WebSight’s role as a fine-tuning dataset.Table 1 is captioned as a comparison between IW-BENCH and WebSight; the supplied caption does not provide the comparison’s individual values.Source: §2 · p. 2, Table 1 · p. 2
• IW-BENCH’s wordcloud contains web- and internet-related keywords, including “html” and “header.”Source: Figure 2 · p. 3

3 IW-BENCH

• IW-BENCH’s pipeline crawls publicly available pages and de-identifies and simplifies them for complex-level data, prompts GPT-4 to generate simple and medium data, then filters low-quality image-code pairs.Source: §3 · p. 3

3.1 Data Collection

• Real-world websites were too complex, yielding uniformly poor evaluation with minimal model differentiation, so GPT-4 was prompted to create simpler data; 18% of IW-BENCH data came from real web pages.Source: §3.1 · p. 3

3.2 Data Processing

• Real-page processing de-identifies and simplifies code, then checks whether changes preserve the original page’s visual presentation.The validation process compares screenshots rendered before and after simplification.Source: §3.2 · p. 3

3.3 Expert Review

• Three engineers cross-validate each data point, resolving disagreements through guidelines, training, and majority consensus.Low-quality examples are re-evaluated until agreement is reached.Source: §3.3 · p. 4

3.4 Metric Design

• The metrics build DOM trees for generated and reference pages, then compare visible and invisible elements and their attributes and relationships.The traversal collects tags, attributes, styles, JavaScript bindings, and hierarchical relationships.Source: §3.4 · p. 4

3.4.1 Element Accuracy (EA)

• Element Accuracy averages each test element’s scores across six attributes and counts elements whose score exceeds a threshold.The six attributes are tag, text content, attributes, style, JavaScript, and children.Source: §3.4.1 · p. 4
• EA compares tag names, text similarity, mapped attributes, selected style properties, JavaScript event bindings, and child structure.Child structure is assessed with a tree-edit distance algorithm.Source: §3.4.1 · pp. 4–5

3.4.2 Layout Accuracy (LA)

• Layout Accuracy compares linearized label and generated element lists using the longest common subsequence and an element-similarity function.The definition also uses the label-list length to normalize the comparison.Source: §3.4.2 · p. 5

4 Five-hop multimodal Chain-of-Thought

• The five hops are SoM prompt injection, element inference, layout inference, web-code inference, and reflection.Source: §4 · p. 5

4.1 SoM Prompt Injection

• SoM cues—text labels, arrows, and highlights—direct models' attention to key image details to improve comprehension and precision.Source: §4.1 · p. 5

4.2 Inferring Elements

• Element inference categorizes elements by type and function, including visible and invisible elements and their contextual contributions.Source: §4.2 · p. 5

Content

• The prompts ask models to identify numbered, bounding-box-marked elements, describe their spatial relationships, and generate code from those descriptions without assigning boxes anew.Source: Content · p. 5

4.3 Inferring Layout

• Layout inference analyzes elements' spatial arrangement, alignment, proximity, and hierarchy to understand UI structure and interaction flow.Source: §4.3 · p. 5

4.4 Inferring Web Code

• Web-code inference generates HTML reflecting the analyzed interface's layout, style, and interactive features.Source: §4.4 · p. 5

4.5 Reflection

• Reflection re-renders generated HTML, compares screenshots for element completeness and layout, and feeds analysis back into code generation for up to N iterations.The process is intended to improve code accuracy and visual consistency with the original design.Source: §4.5 · p. 6

5.1 Experiment setting

• The evaluation tests direct model-generated code and five-hop results for element identification and layout arrangement at simple, medium, and complex levels.The experiments use recent multimodal models as baselines; N defaults to 3, and Five-hop MCoT is not run on WebSight because it cannot accommodate simultaneous image and text input.Source: §5.2 · p. 6, §5.1 · p. 6

5.2.1 Overall Performance Comparison

• 48.9% Element Accuracy and 47.9% Layout Accuracy make WebSight the overall leader; GPT4V with Five-hop MCoT scores 45.8% and 44.5%, versus 30.4% and 29.4% without enhancement, respectively.Source: §5.2.1 · p. 6

5.2.2 Performance by Complexity Level

• Across complexity levels, all models' performance decreases as tasks grow harder; WebSight leads both metrics, while Gemini Pro and Qwen-VL-Max drop more as complexity rises.LLaVA-LLaMA-2-13B exhibits a more substantial decrease, which the authors suggest may indicate greater suitability for simple tasks.Source: §5.2.2 · p. 7

5.2.3 Element vs. Layout Accuracy

• Element Accuracy tends to exceed Layout Accuracy, and the gap widens as task complexity increases.WebSight is reported as the top performer in balancing the two metrics.Source: §5.2.3 · p. 7

5.2.4 Impact of Enhancements

• Every model improves on both metrics with Five-hop MCoT, with the largest reported impact at simple and medium complexity levels.Source: §5.2.4 · p. 7

5.2.5 Human-in-the-loop Evaluation

• Metric-based and human-evaluation rankings have a Pearson correlation of 0.8 (P-value 0.104), based on ratings of 100 pages by 20 front-end professionals.Source: §5.2.5 · p. 7

6 Ablation

• The ablation experiments evaluate the five-hop multimodal Chain-of-Thought method using GPT4V.Source: §6 · p. 7

6.1 Ablation on SoM prompt injection

• Removing SoM lowers GPT4V’s average element accuracy by 4.2% and average layout accuracy by 5.1%.Accuracy also declines as task complexity increases; performance is better on simple and medium levels.Source: §6.1 · p. 7, Table 7 · p. 8

6.2 Ablation on Reflection

• Element accuracy rises substantially from N=0 to N=2 reflections, then stabilizes; layout accuracy varies minimally across reflection counts.Element accuracy is higher than layout accuracy for most of the tested range.Source: Figure 6 · p. 7, §6.2 · p. 7

• Across three reflections, text fidelity and navigation layout improve, with the third pass tightening page layout and bringing color and contrast closer to the original.The first pass captures the navigation structure but not all text; the second improves text fidelity and makes the navigation more compact.Source: §7 · p. 8, Figure 7 · p. 8

• IW-BENCH pairs Element Accuracy for element completeness and Layout Accuracy for positional relationships with a five-hop multimodal Chain-of-Thought method for image-to-web conversion.Source: §8 · p. 8

• IW-BENCH currently covers only two languages, and adding languages would broaden its applicability to diverse audiences.Source: Limitations · p. 9
• The dataset needs more samples and broader scenario and context coverage to support robustness, reliability, and validity of benchmark results.Source: Limitations · p. 9

A Data De-identification

• Real-page images are replaced and brand-associated content is modified to reduce direct identification and address copyright and trademark concerns.The authors describe these measures as protecting copyright holders and keeping the project legal and ethical.Source: A Data De-identification · p. 12

B Crowdsourcing

• Participants are web engineers; the study addresses privacy and psychological risks through anonymization, secure storage, debriefing, and voluntary withdrawal.Participants receive $20 per hour, and the study monitors distress and offers support.Source: B Crowdsourcing · p. 12

C.1 Image-to-Web

• Prior image-to-web work includes mock-up-to-HTML generation and screenshot-to-code approaches, while WebSight is described as lacking public performance evaluations.The authors call for a benchmark to evaluate large multimodal models in this domain.Source: §C.1 · p. 12

C.2 Multimodal Large Language Model

• The related-work discussion situates the benchmark among text and vision-language foundation models and existing instruction-following and reasoning benchmarks.The authors plan to evaluate image-to-web capabilities across diverse visual contexts.Source: §C.2 · p. 12

C.3 Chain-of-Thought

• The authors propose five-hop multimodal Chain-of-Thought prompting to evaluate multimodal large models on image-to-web tasks and compare them with other multimodal baselines.Source: §C.3 · p. 13

Content

• The appendix provides domain prompts, element sets for simple and medium difficulty, and lists of events and CSS properties used in the web-generation materials.The domain prompts cover varied site types, including portfolios, news pages, enterprise portals, stores, blogs, and social platforms.Source: Content · pp. 13–14

G Web Page Quality and User Experience Questionnaire

• The questionnaire uses a 1-to-5 scale to assess content clarity, layout, functionality, and overall user experience.Its items ask about comprehensibility, organization, interactive behavior, responsiveness, and satisfaction.Source: G Web Page Quality and User Experience… · pp. 15–16

I More examples in IW-BENCH

• The appendix labels example images as simple, medium, and complex levels.Source: Figure 8 · p. 16, Figure 10 · p. 17, Figure 9 · p. 17

Content

• The attribute lists mark selected HTML attributes as strong or weak comparison features across elements such as links, images, buttons, and form controls.Source: Content · pp. 17–18