Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

1 Introduction

• The paper identifies the lack of large, high-quality screenshot–HTML pairs—not task difficulty itself—as the primary obstacle to VLM proficiency.An open dataset could let researchers inspect its data and limitations and improve it.Source: §1 · pp. 1–2
• The authors favor LLM-synthesized HTML over rendering existing webpages because web code can be noisy, excessively long, and dependent on external files.They say these properties make it harder for models to learn the relationship between screenshots and HTML.Source: §1 · p. 2
• LLM prompts can constrain topic, text length, and image placement, producing cleaner and more structured code for VLM training.The study adopts this synthesis approach.Source: §1 · p. 2
• 2 million synthetic HTML–screenshot pairs make up WebSight, which the authors use to fine-tune an 8-billion-parameter VLM into Sightseer.The paper reports functional HTML generation and adaptation to untrained cases such as handwritten sketches.Source: §1 · p. 2

2 Related work

• Nguyen and Csallner (2015) detected interface elements and applied heuristics to generate mobile UI code, while Beltramelli (2017) used end-to-end deep learning for multiple platforms.Lee et al. (2023) pretrained a VLM to convert masked webpage screenshots into simplified HTML; its model accepts varying image resolutions, like Sightseer.Source: §2 · pp. 2–3
• WebSight-v0.2 grows the dataset 2.5× and adds real content-matched images, Tailwind CSS, higher-resolution screenshots, and richer metadata.The preceding v0.1 release contained 823K synthetic screenshot–HTML pairs using traditional CSS.Source: §2 · p. 3
• Design2Code used WebSight-v0.1 to fine-tune an 18B-parameter VLM after finding synthetic training examples outperformed longer, more complex real-world code data.This work also introduced a benchmark for screenshot-to-HTML generation.Source: §2 · p. 3

3 Construction of the dataset

• A smaller language model generates website themes and designs, which a larger code-focused LLM expands into final HTML to provide varied styles and high-quality code.The concept-generation stage uses Mistral-7B-Instruct, prompted to produce layout ideas across companies and design elements.Source: §3 · pp. 3–4
• Tailwind CSS provides in-document utility classes, supporting visually varied designs without separate style files and keeping code concise for VLM training.The authors prefer standalone code to simplify the learning task.Source: §3 · p. 4
• The final HTML is generated with Deepseek-Coder-33b-instruct, prompted to write complete websites in HTML and Tailwind CSS about each concept.The prompt asks for substantive business text and permits topic-matched images from Unsplash.Source: §3 · p. 4
• Unsplash image URLs let generated pages include dynamically selected images relevant to their topics and sized for the layout.The authors describe this as a solution to the challenge of integrating contextually relevant images into generated HTML.Source: §3 · p. 5
• After filtering pages with insufficient text, generic content, or off-topic images, the authors retain 2 million pages and capture full-page screenshots with Playwright.Screenshots span varied resolutions to support model robustness; Figure 2 presents example pages.Source: Figure 2 · p. 3, §3 · p. 5

4 Fine-tuning a foundation vision-language model on WebSight

• Screenshot-to-HTML conversion requires OCR, spatial understanding, and object recognition; the base model combines Mistral-7B and SigLIP-SO400M with aspect-ratio-preserving image processing.The base model uses Patch n’ Pack and supports input images up to 980 pixels per side.Source: §4 · p. 5
• The authors fine-tune with rank-64 DoRA at a 10^-4 learning rate for 1,100 iterations, seeing 2,016 examples per iteration—slightly less than one epoch.They use parameter-efficient tuning rather than unfreezing all weights.Source: §4 · p. 5
• Validation loss did not reliably indicate real-world code quality or generalization, so checkpoints were selected by manually inspecting generated samples.Figure 3 compares an input webpage with the rendering of Sightseer’s generated code.Source: §4 · p. 5, Figure 3 · p. 6

5.1 Results on different screenshots

• When input designs contain little text, Sightseer tends to preserve that text accurately in its output.Source: §5.1 · p. 7
• Sightseer can generalize beyond its training data to convert a handwritten website sketch into functional HTML.Source: §5.1 · p. 7

5.2 Failure cases

• Sightseer struggles with complex layouts, excessive text, or designs that diverge substantially from its training data.Source: §5.2 · p. 7
• Generated images, text, or buttons may not appear after rendering because of same-color text and syntax errors, reflecting incomplete mastery of HTML and Tailwind CSS.Source: §5.2 · p. 7
• Although Sightseer produces more visually attractive websites, it sometimes makes errors absent from the traditional-CSS WebSight-v0.1 model; the authors hypothesize Tailwind CSS is harder for the base LLM to master.Source: §5.2 · p. 8

• WebSight contains 2 million synthetic pairs of HTML code and corresponding renderings.Source: §6 · p. 8
• Sightseer is an OCR-capable vision-language model fine-tuned on WebSight for webpage screenshot-to-HTML conversion.Source: §6 · p. 8
• The authors present synthetic data and fine-tuning a high-capacity base VLM as a viable path toward UI-development automation and no-code tools.Source: §6 · p. 8
• WebSight is open-sourced to foster further innovation and research in the area.Source: §6 · p. 8