WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch

Introduction

• WebGen-Bench is introduced as the first benchmark for agents building websites from scratch, unlike prior benchmarks that test bug fixes or patches to existing codebases.It assesses planning, development, and management of complex multi-file projects while targeting users with little or no programming background.Source: §1 · pp. 2–3
• The benchmark instructions were derived from 20 requirement categories through brainstorming by 40 computer-science Ph.D. students, then GPT-4o drafted instructions and test cases that authors manually reviewed.Test cases specify website operations and expected outcomes and target both functionality and appearance requirements.Source: §1 · p. 2
• WebVoyager executes website operations to assess outcomes, while GPT-4o grades design harmony and aesthetics on a 1-to-5 scale.The framework is intended to provide a comprehensive assessment of overall website-generation quality.Source: §1 · pp. 2–3, Figure 1 · p. 3
• 27.8% accuracy: DeepSeek-R1 was the strongest evaluated engine on Bolt.diy, while Claude-3.5-Sonnet had the highest average appearance score, 3.0.Bolt.diy, OpenHands, and Aider were evaluated, and the results indicate substantial room for improvement on the benchmark.Source: §1 · p. 3
• 38.2% accuracy on WebGen-Bench: fine-tuned Qwen2.5-Coder-32B-Instruct surpassed its original 9.5% accuracy and DeepSeek-R1.Fine-tuning used Bolt.diy trajectories generated from a subset of WebGen-Instruct by DeepSeek-V3 with rejection sampling.Source: §1 · p. 3
• WebGen-Instruct contains 6,667 website-generation instructions; deduplication removed instructions similar to benchmark items, and fine-tuned 7B, 14B, and 32B Qwen2.5-Coder models form WebGen-LM.The filtering used Jaccard similarity and Sentence-Transformers-based deduplication.Source: §1 · p. 3

Software Engineering Benchmarks

• Unlike single-file coding benchmarks and repository-based repair benchmarks such as SWE-bench [16] [39] and SWE-Lancer [23], WebGen-Bench tests creating multi-file web applications from natural-language instructions.Generated applications must implement multiple functionality and appearance requirements and require independent technical design decisions.Source: §2 · p. 4

LLM-based Code Agents and Pipelines

• The study adapts OpenHands [33] and Aider and uses web-generation-specialized Bolt.diy, while also fine-tuning open-source models on web-instruction trajectories.Prior work has used post-training to improve code-agent performance with open-source models.Source: §2 · p. 4

Automatic Software User-testing

• Prior website-testing work includes UXAgent [19], which simulates user experiences with personas; this work instead uses atomic tests of functionality and appearance requirements.A web-navigation UI agent performs operations and checks whether observed website behavior matches expectations.Source: §2 · p. 4

3.1 Instruction Curation

• The test set contains 101 instructions selected from 10,152 manually created project descriptions, and WebGen-Instruct adds 6,667 decontaminated training instructions.Forty computer science Ph.D. students brainstormed applications and requirements across 20 prevalent web-application categories; GPT-4o generated instructions from the descriptions, with two to eight examples sampled per category.Source: §3.1 · p. 5
• The 101 test instructions are grouped into Content Presentation, User Interaction, and Data Management to support higher-level analysis.Instructions omit technical design specifications so agents can choose implementation approaches for requests framed as coming from non-expert users.Source: §3.1 · p. 5, Table 2 · p. 6

3.2 Test Case Construction and Evaluation

• The researchers created 647 manually validated test cases, with 4–11 per instruction, to cover every requirement while preserving implementation flexibility.Each case pairs an operation testing a functionality or appearance requirement with its expected outcome; two Ph.D. students reviewed and refined GPT-4o drafts.Source: §3.2 · pp. 5–6
• WebVoyager executes test operations and assesses requirement fulfillment as YES, NO, or PARTIAL using interaction trajectories and screenshots.Qwen2.5-VL-32B-Instruct is the navigation agent’s engine; when it reaches its interaction limit, a decision prompt requests a final assessment.Source: §3.2 · p. 6

3.3 Evaluation of Website Appearance

• GPT-4o rates website appearance from 1 to 5 using metrics for rendering success, content relevance, layout harmony, and design modernness.Higher scores indicate better appearance.Source: §3.3 · p. 6, Figure 2 · p. 7

4.1 Experimental Setup

• The study evaluated Bolt.diy, OpenHands with CodeActAgent, and Aider, then tested several proprietary and open-source models on Bolt.diy; models smaller than Qwen2.5-Coder-32B were excluded for output-format failures.The initial framework comparison used DeepSeek-V3, Claude-3.5-Sonnet, and DeepSeek-R1; the Bolt.diy model evaluation also included GPT-4o, O3-mini, Qwen2.5-Coder-32B, and Qwen2.5-72B-Instruct.Source: §4.1 · p. 7
• From 2K WebGen-Instruct instructions, rejection sampling retained 600 trajectories with appearance scores of at least 3 to fine-tune 7B, 14B, and 32B WebGen-LM models.The models were fine-tuned for 2 epochs with a 4e-5 learning rate and batch size 32.Source: §4.1 · p. 7

4.2 Experimental Results

• Bolt.diy with DeepSeek-R1 achieved 27.8% accuracy, versus 26.4% for Claude-3.5-Sonnet; Claude-3.5-Sonnet had the highest appearance score, 3.0.The authors say these leading general-model results remain far from saturating the benchmark.Source: Table 3 · p. 8, §4.2 · p. 8
• Design Validation Testing had the highest accuracy in most cases, while Functional Testing was generally lower; Content Presentation led instruction categories.The authors interpret this pattern as superficial aspects being easier to implement than deeper internal functionality.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 Ablation Studies

• Three human testers independently checked UI-agent results for Claude-3.5-Sonnet, DeepSeek-R1, and DeepSeek-V3, with a fourth adjudicating inconsistent annotations.These manual annotations served as ground truth for assessing alignment with the UI agent.Source: §4.3 · pp. 8–9, Table 5 · p. 9

4.2 Experimental Results

• 38.2% accuracy for WebGen-LM-32B exceeded the best proprietary model, DeepSeek-R1, by 10.4%.The paper attributes the gain to the training set and rejection-sampling process.Source: §4.2 · pp. 7–8, Table 4 · p. 9

4.3 Ablation Studies

• Accuracy consistently increased as Qwen2.5-Coder-32B-Instruct training samples rose from 150 to 300 to 600, though API budget constraints prevented sampling more trajectories.The authors leave further gains from additional data or data augmentation to future work.Source: §4.3 · p. 9, Table 6 · p. 10

5 Conclusion

• 27.8% accuracy is achieved by Bolt.diy with DeepSeek-R1, while fine-tuned Qwen2.5-Coder-32B reaches 38.2%, surpassing the best proprietary model.Fine-tuning uses 600 Bolt.diy trajectories generated by DeepSeek-V3 from a 6,667-instruction training set.Source: §5 · p. 10
• WebGen-Bench tests whether agents can build multi-file website codebases that satisfy functional and visual constraints; the evaluation covers three code-agent frameworks and proprietary and open-source LLMs.Source: §5 · p. 10

A Ethics Statement

• The non-commercial dataset consists entirely of synthetic instructions and test cases, curated manually and synthesized with artificial intelligence.The authors state that its construction maintains distance from potential ethical or legal concerns, particularly intellectual property.Source: A Ethics Statement · p. 14
• The authors cite three copyright safeguards: original base descriptions, abstraction of 20 fundamental categories, and no copying from existing websites or platforms.Source: A Ethics Statement · p. 14
• The dataset and reproduction code are publicly available for researchers evaluating website-generation agents, and both are released under the MIT license.Source: A Ethics Statement · p. 14
• The authors take full responsibility in the event of any rights violations.Source: A Ethics Statement · p. 14

B Limitations and Future Work

• Website generation used TypeScript, JavaScript, CSS, and HTML, but not Python, Java, or Go.The authors attribute the exclusion to the complexity of integrating those languages into the agent framework.Source: B Limitations and Future Work · p. 14
• Expanding support for languages and tools in code-agent website generation is identified as a future research direction.Source: B Limitations and Future Work · p. 14
• The study used supervised fine-tuning alone to improve open-source LLM performance on website generation.Source: B Limitations and Future Work · p. 14
• Reinforcement learning and direct preference optimization [27] were not used and are proposed as opportunities for future exploration.Source: B Limitations and Future Work · p. 14

C Prompt for Deriving Instructions from Website Development Project Descriptions

• The instruction-derivation prompt excludes requirements about technical implementation details.It is used to derive website-generation instructions from project descriptions created by human annotators.Source: C Prompt for Deriving Instructions… · p. 14
• Excluding implementation requirements is intended to evaluate code agents’ ability to make those decisions independently.Source: C Prompt for Deriving Instructions… · p. 14

D Details of the Decontamination Process

• To decontaminate training from testing, authors removed training instructions with 5-gram Jaccard similarity above 0.6 to any test instruction, then removed those with sentence-embedding cosine similarity above 0.55 using Sentence-Transformer all-MiniLM-L6-v2.Source: D Details of the Decontamination Process · p. 14
• Manual inspection of the top three training matches for each test instruction found no semantic duplicates, supporting the authors’ conclusion that the final training set is not contaminated.Examples of top matches are shown for the first three test instructions in Figures 5–7.Source: D Details of the Decontamination Process · p. 14, Figure 5 · p. 16, Figure 6 · p. 17

E Application Categories of WebGen-Instruct and WebGen-Bench.

• The authors summarized 20 application categories from web-development platforms as seed ideas for annotators brainstorming scenarios for WebGen-Instruct and WebGen-Bench.The categories cover varied website and application types, including portfolios, social platforms, e-commerce, business systems, games, learning, travel, jobs, and productivity tools.Source: E Application Categories of… · p. 16, Table 7 · p. 18

F Prompt for Creating Website Test Cases

• The test-case prompt emphasizes covering every functionality and appearance requirement while ensuring each generated test case directly reflects the website-generation instruction.The stated aim is thorough evaluation using valid test cases.Source: F Prompt for Creating Website Test Cases · p. 18

G Prompt for Automatic Evaluation of Test Cases Using an UI Agent

• The UI-agent prompt asks the agent to perform a test-case operation and answer YES, NO, or PARTIAL; a separate prompt elicits a final decision when interaction limits are reached.Source: G Prompt for Automatic Evaluation of… · p. 18

H Prompt for Grading Website Appearance

• The appearance-grading prompt directs GPT-4o to assess rendering, content relevance, layout harmony, modernity, and visual appeal on a 1–5 scale, with higher grades better.Source: H Prompt for Grading Website Appearance · p. 20

I Prompt for Adapting OpenHands Paired with CodeActAgent for WebGen-Bench Evaluation

• Figure 12 presents the prompt for evaluating OpenHands paired with CodeActAgent on WebGen-Bench.Source: I Prompt for Adapting OpenHands Paired… · p. 20

J Prompt for Aider to Generate Websites for WebGen-Bench Evaluation

• Figure 13 shows Aider’s prompt for generating websites for the WebGen-Bench evaluation.Source: J Prompt for Aider to Generate… · p. 20

K Test Case Categories

• Functional testing covers forms, authentication, payments, search and filtering, generation and file operations, e-commerce, and communication features.Examples include registration and login, checkout or donations, report or file creation, downloading or uploading, and sending messages or emails.Source: K Test Case Categories · p. 20
• Data display testing checks dynamic content updates, navigation and refresh, visualization accuracy, and the display of requested details.Visualizations include charts, graphs, and maps.Source: K Test Case Categories · p. 20
• Nearly half of test cases are Functional Testing, around 30% Data Display Testing, and approximately 20% Design Validation Testing.Figure 14 shows the main category distribution; Table 8 presents subcategories and their frequencies.Source: K Test Case Categories · p. 20, Figure 14 · p. 22
• Design validation testing checks UI consistency, color, typography, spacing, responsive behavior across devices, and component styling against design standards.Components include buttons, icons, and cards.Source: K Test Case Categories · p. 22

L Analysis of Reliability of Appearance Scores

• The reliability assessment compares o3 and Claude-3.5-Sonnet grading with GPT-4o and manually grades screenshots for human preference.It evaluates Bolt.diy results produced with Claude-3.5-Sonnet, DeepSeek-R1, and DeepSeek-V3.Source: L Analysis of Reliability of… · p. 22
• Claude-3.5-Sonnet > DeepSeek-R1 > DeepSeek-V3 remains the ranking across grading methods, although o3 and Claude-3.5-Sonnet assign slightly lower scores than GPT-4o.The comparison uses scores for Bolt.diy results from three agent–engine LLMs graded by different methods.Source: L Analysis of Reliability of… · pp. 22–23, Table 9 · p. 23
• For every screenshot set, the graders’ scores descend in the same order: GPT-4o > o3 > Claude-3.5-Sonnet.This consistent pattern is reported across all sets of screenshots.Source: L Analysis of Reliability of… · p. 23, Table 9 · p. 23
• The ensemble results of the three grading models and the human scores both match GPT-4o’s ranking of appearance quality.The analysis manually grades website screenshots to capture human preference.Source: L Analysis of Reliability of… · pp. 22–23, Table 9 · p. 23
• GPT-4o is considered a suitable appearance grader under a limited budget because its rankings align with both ensemble grading and human preference.The authors state that they will add this reliability discussion to the revised paper.Source: L Analysis of Reliability of… · p. 23

M Examples of Websites with Different Appearance Scores

• As appearance scores rise, website quality improves: score-one examples have major rendering errors or irrelevant content, while score-five examples have highly harmonious designs.Fig. 15 presents examples at varying appearance scores.Source: M Examples of Websites with Different… · p. 23

• Examples (a)–(c) show three types of website-loading failures.Source: N Examples of Websites with Different… · p. 23
• Incomplete websites may show only a background, omit UI components such as buttons, or display an image incorrectly.Source: N Examples of Websites with Different… · p. 23
• Other flaws include an uncustomized template, misplaced webpage content such as misaligned text, and an inappropriate background color.Source: N Examples of Websites with Different… · p. 23

YES outcomes

• Five examples output YES when the website’s behavior matches the expected result.The examples cover tasks such as updating a prize-winning record, submitting questions, and sending email.Source: O Examples of UI Agent Testing Processes · p. 24, Figure 17 · p. 26, Figure 18 · p. 27

PARTIAL outcomes

• Three examples output PARTIAL when the expected result is only partly achieved.They test skill-based user search, infrastructure information browsing, and stock-form input validation.Source: O Examples of UI Agent Testing Processes · p. 24, Figure 22 · p. 31, Figure 23 · p. 32

NO outcomes

• Three examples output NO when the website’s behavior does not match the expected outcome.They test player rankings and adding a link to a user’s link tree.Source: O Examples of UI Agent Testing Processes · p. 24, Figure 25 · p. 34, Figure 26 · p. 35

NeurIPS Paper Checklist

• Checklist answers must be Yes, No, or NA with a short justification, and reviewers may consider them in evaluation; a justified No or NA is acceptable.The checklist is an integral part of the submission and is included with the published final paper.Source: NeurIPS Paper Checklist · p. 37
• The authors answer Yes that the abstract and introduction reflect the paper’s contributions and scope, and that the paper discusses its limitations.They state that the discussion is in “Limitations and Future Work.”Source: NeurIPS Paper Checklist · p. 37

3. Theory assumptions and proofs

• The paper reports no theoretical results, and the authors state that experimental reproducibility is detailed in Experiments and open-source code and data are available.They also say the experimental setting is described in Experiments.Source: NeurIPS Paper Checklist · pp. 38–40

4. Experimental result reproducibility; 5. Open access to data and code; 6. Experimental setting/details

• The authors answer Yes that the paper provides open access to code and data and discloses information needed to reproduce the main experiments.The checklist also says training and test details are specified in Experiments.Source: NeurIPS Paper Checklist · pp. 38–40

7. Experiment statistical significance

• The authors state that error bars are omitted because they would be too computationally expensive, while answering Yes that the paper discusses limitations.The checklist’s guidance recommends discussing assumptions, scope, performance factors, efficiency, and applicable privacy or fairness limitations.Source: NeurIPS Paper Checklist · pp. 37, 38, 40

NeurIPS Paper Checklist

• The authors answer Yes that the paper conforms to the NeurIPS Code of Ethics and discusses positive and negative societal impacts, with safeguards for risky releases.They also answer Yes that existing assets are credited and licensed appropriately and that new assets are documented.Source: NeurIPS Paper Checklist · pp. 41–42

14. Crowdsourcing and research with human subjects; 15. Institutional review board approvals; 16. Declaration of LLM usage

• The authors state that they did not use crowdsourcing, that the study poses no participant risks, and that LLM usage is described in Method.They report using authors and student volunteers rather than crowdsourcing.Source: NeurIPS Paper Checklist · pp. 42–43