Source-linked AI summary
WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch
Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, Hongsheng Li
TL;DR
LLM-based agents still lack systematic evaluation for generating customized, functional, and visually constrained websites from scratch. WebGen-Bench addresses this gap with curated instructions, automated functional and appearance testing, and evaluations of multiple code-agent frameworks and models. Bolt.diy with DeepSeek-R1 reaches only 27.8% accuracy, while fine-tuning Qwen2.5-Coder-32B raises accuracy to 38.2%.
Problem
Existing software-engineering evaluations do not systematically measure agents’ ability to build customized websites from scratch for non-expert users.
Method
WebGen-Bench combines curated website-generation instructions, standardized test cases, WebVoyager-based functional evaluation, GPT-4o appearance grading, and multi-framework model evaluations.
Results
27.8% accuracy was achieved by Bolt.diy with DeepSeek-R1, while fine-tuned Qwen2.5-Coder-32B achieved 38.2% accuracy on WebGen-Bench.
Takeaways & Limitations
The benchmark remains challenging for current agents, while training on WebGen-Instruct trajectories improves website-generation accuracy beyond the reported DeepSeek-R1 result.
Takeaways & Limitations
Website generation is primarily limited to TypeScript, JavaScript, CSS, and HTML, and model enhancement uses supervised fine-tuning without reinforcement learning or direct preference optimization.
Abstract
from arXiv · showhide
LLM-based agents have demonstrated great potential in generating and managing code within complex codebases. In this paper, we introduce WebGen-Bench, a novel benchmark designed to measure an LLM-based agent's ability to create multi-file website codebases from scratch. It contains diverse instructions for website generation, created through the combined efforts of human annotators and GPT-4o. These instructions span three major categories and thirteen minor categories, encompassing nearly all important types of web applications. To assess the quality of the generated websites, we use GPT-4o to generate test cases targeting each functionality described in the instructions, and then manually filter, adjust, and organize them to ensure accuracy, resulting in 647 test cases. Each test case specifies an operation to be performed on the website and the expected result after the operation. To automate testing and improve reproducibility, we employ a powerful web-navigation agent to execute tests on the generated websites and determine whether the observed responses align with the expected results. We evaluate three high-performance code-agent frameworks, Bolt.diy, OpenHands, and Aider, using multiple proprietary and open-source LLMs as engines. The best-performing combination, Bolt.diy powered by DeepSeek-R1, achieves only 27.8\% accuracy on the test cases, highlighting the challenging nature of our benchmark. Additionally, we construct WebGen-Instruct, a training set consisting of 6,667 website-generation instructions. Training Qwen2.5-Coder-32B-Instruct on Bolt.diy trajectories generated from a subset of this training set achieves an accuracy of 38.2\%, surpassing the performance of the best proprietary model.
1 Introduction
WebGen-Bench addresses the lack of systematic evaluation for agents generating functional, visually constrained websites from scratch. It combines curated instructions, automated testing, broad framework and model evaluations, and a training set whose fine-tuned model reaches 38.2% accuracy.
- Motivation and benchmark: The benchmark targets a growing need for code agents that help non-experts build customized applications from natural-language requests.Building such applications requires high-level planning, multi-file organization, and nuanced implementation of functionality and design.
- Motivation and benchmark: WebGen-Bench evaluates whether LLM-based agents can construct websites from scratch while satisfying functional and appearance requirements.Unlike repository-level benchmarks focused on modifying existing codebases, it tests planning and management of complex multi-file projects.
- Benchmark construction: The dataset and evaluation process address instruction diversity and reliable website assessment through systematic curation and testing.The curation process draws on website-development requirements and GPT-4o assistance, while testing evaluates functionality and appearance.
- Evaluation results: 27.8% accuracy was achieved by Bolt.diy with DeepSeek-R1 on test cases, while Claude-3.5-Sonnet obtained a 3.0 average appearance score.These results indicate that current agents remain far from saturated on the benchmark.
- Training and results: 38.2% accuracy was achieved by fine-tuned Qwen2.5-Coder-32B on WebGen-Bench, exceeding its original 9.5% accuracy and DeepSeek-R1’s performance.The model was trained on Bolt.diy trajectories generated from WebGen-Instruct.
- Evaluation results: The study evaluates Bolt.diy, OpenHands, and Aider with proprietary and open-source language models as engines.The framework comparison demonstrates the benchmark’s difficulty across multiple agent systems.
2 Related Work
Prior software-engineering research evaluates code agents on coding tasks, repository modifications, and issue resolution, while related work also uses agents for automated user testing. WebGen-Bench differs by evaluating website generation from scratch with atomic tests for functionality and appearance.
- Software engineering benchmarks: Earlier coding benchmarks primarily assess single-file or function-level code generation from curated user queries, contests, model synthesis, or expert designs.These benchmarks evaluate LLM coding ability through task collections from varied sources.
- Software engineering benchmarks: Repository-level benchmarks such as SWE-bench evaluate advanced modifications to existing codebases, including bug fixes and feature patches.These tasks differ from constructing a website codebase from scratch.
- Code agents and pipelines: Agent-based code frameworks interact with executable environments to obtain feedback, while OpenHands and Aider are general-purpose and Bolt.diy specializes in web applications.The paper selects these three open-source frameworks for benchmark evaluation.
- Automatic software user-testing: Automated user-testing research applies agents to websites, graphical interfaces, and games because human testing is costly and difficult to manage.UXAgent uses predefined personas to simulate website user experiences.
- Automatic software user-testing: WebGen-Bench uses atomic test cases targeting functionality and appearance, enabling a web-navigation agent to perform operations and check intended behavior.This distinguishes its evaluation setup from prior agent-based user-testing work.
3 WebGen-Bench
WebGen-Bench evaluates from-scratch website generation using diverse instructions, validated requirement-level test cases, and automated agents that assess functionality and appearance.
- Instruction construction: The benchmark curates project descriptions from website-development listings, author discussions, and brainstorming by forty computer-science Ph.D. students.The authors manually created 10,152 project descriptions before instruction curation.
- Instruction construction: GPT-4o generates website-generation instructions, which are sampled and deduplicated against testing instructions to produce WebGen-Instruct.The resulting training set contains 6,667 instructions after lexical and semantic decontamination.
- Dataset organization: WebGen-Bench reorganizes 101 testing instructions into three broad technical categories with multiple subcategories.Samples may belong to one main category and multiple subcategories.
- Test case construction: Each test case pairs an operation with its expected outcome to verify a specific functionality or appearance requirement.Two Ph.D. students independently reviewed and refined GPT-4o drafts before discrepancies were resolved.
- Test case construction: The final benchmark contains 647 manually validated test cases, with 4–11 cases per instruction, covering requirements while preserving implementation flexibility.The validation process aimed to ensure every instruction requirement was covered and every test case matched a requirement.
- Evaluation: WebVoyager automates test execution and outcome verification, while GPT-4o rates website appearance from 1 to 5 using rendering, relevance, layout, and design criteria.Automated evaluation reduces the burden of human testing, which would take approximately 10.8 hours and cost an estimated $377.8 for all 647 cases.
4 Experiments
The experiments evaluate website generation across agent frameworks, language models, categories, and training settings using weighted test-case accuracy and appearance scores. Results show that website generation remains difficult, with performance varying substantially across models and task categories.
- Experimental Setup: Three code-agent frameworks—Bolt.diy, OpenHands, and Aider—are evaluated with multiple proprietary and open-source language models.The study first compares frameworks and then evaluates several models on the best-performing framework, Bolt.diy.
- Evaluation Metrics: Accuracy is computed from YES and PARTIAL test-case outcomes, weighted by 1 and 0.5 respectively, then divided by the total number of test cases.This weighted scoring procedure is used for the benchmark results.
- Main Results: 27.8% accuracy is achieved by Bolt.diy with DeepSeek-R1, while Claude-3.5-Sonnet reaches 26.4% and obtains the best appearance score of 3.0.The results describe these as the strongest general-model and appearance outcomes, respectively.
- Categorical Results: Functional Testing generally has lower accuracy than Design Validation Testing, while Content Presentation consistently achieves the highest instruction-category accuracies.The results indicate that superficial visual aspects are easier to implement than deeper internal functionalities.
- Ablation Studies: Accuracy consistently increases with the number of training samples, although additional trajectories were not sampled because of API budget constraints.The analysis compares 150, 300, and 600 training samples for Qwen2.5-Coder-32B-Instruct.
5 Conclusion
WebGen-Bench evaluates website generation from scratch and reports low benchmark accuracy, while the accompanying dataset and code are released for research use under the MIT license.
- 27.8% accuracy is achieved by Bolt.diy with DeepSeek-R1 on WebGen-Bench, underscoring the benchmark’s difficulty.
- 38.2% accuracy is achieved after fine-tuning Qwen2.5-Coder-32B on 600 Bolt.diy trajectories generated by DeepSeek-V3, exceeding the best proprietary model.
- The authors state that the benchmark construction avoids copying content from existing websites or platforms and addresses copyright concerns through abstracted categories and author-created descriptions.
- The benchmark dataset and reproduction code are publicly available under the MIT license for evaluating agents that generate websites from scratch.
B Limitations and Future Work
The work is scoped mainly to web technologies and supervised fine-tuning, while its instruction-generation prompt excludes technical implementation requirements so agents make those decisions independently.
- Limitations: Website generation primarily uses TypeScript, JavaScript, CSS, and HTML, excluding Python, Java, and Go because integration into the agent framework is complex.
- Future Work: The study uses supervised fine-tuning but does not apply reinforcement learning or direct preference optimization, leaving those post-training strategies for future work.
- Methodological Scope: The instruction-generation prompt excludes technical implementation details so code agents are evaluated on making those decisions independently.
D Details of the Decontamination Process
The decontamination process removes lexically and semantically similar training instructions, then manually checks nearest remaining matches to assess duplication.
- Lexical Filtering: Instructions with 5-gram Jaccard similarity above 0.6 to a test instruction are removed from the training set.
- Semantic Filtering: Sentence embeddings from all-MiniLM-L6-v2 are used to identify training instructions that are semantically similar to test instructions.
- Manual Inspection: For each test instruction, the three training instructions with highest cosine similarity are manually inspected for semantic duplication.
- Result: The inspected nearest-neighbor training samples are reported as completely different from the test samples, supporting the claim that the final training set is uncontaminated.
E Application Categories of WebGen-Instruct and WebGen-Bench.
WebGen-Instruct and WebGen-Bench cover a broad range of application scenarios, while their test-generation and UI-evaluation prompts operationalize functional and visual assessment.
- Category Construction: Twenty application categories are manually summarized from web-development project listings and used as seed ideas for brainstorming application scenarios.
- Application Categories: The categories include portfolio, brochure, blog, social media, forum, e-commerce, email, project-management, streaming, CRM, ERP, internal, news, publishing, and analytics applications.
- Example Instructions: Example instructions cover neighborhood mapping with demographic, economic, and crime comparisons, interactive dashboards, and specified ivory and forest-green styling.
- Application Categories: The remaining categories include browser-based games, learning platforms, travel booking portals, job-search platforms, and productivity applications.
- Example Instructions: Another example requests a multi-company financial dashboard supporting company-level data, consolidated reports, cross-company comparisons, and mint-cream styling.
- Test Construction: Generated test cases must be atomic, include validation criteria, align with the original instruction, and collectively cover functional and appearance requirements.
- Automated Evaluation: The UI agent performs each test operation and returns YES, NO, or PARTIAL against the expected result, with a final decision prompt used after the interaction limit is reached.
- Visual Evaluation: GPT-4o grades webpage aesthetics from 1 to 5 using rendering, content relevance, layout harmony, modernity, and visual appeal.
K Test Case Categories
The benchmark organizes test cases into Functional Testing, Data Display Testing, and Design Validation Testing, covering application behavior, data presentation, and interface quality.
- Nearly half of test cases assess functionality, around 30% assess data display, and approximately 20% assess design validation.Functional testing is the largest category because web evaluations typically emphasize whether application features work as intended.
- Functional Testing: Functional testing covers forms, authentication, payments, search, filtering, and other application features.
- Data Display Testing: Data display testing checks dynamic rendering, updates, navigation, refresh behavior, visualizations, maps, and detailed information views.
- Design Validation Testing: Design validation testing evaluates interface consistency, color, typography, spacing, responsiveness, and component styling.
L Analysis of Reliability of Appearance Scores
The paper compares appearance scores from multiple automated graders and human evaluation, finding consistent relative rankings across grading methods and examples spanning low to high visual quality.
- The relative ranking of Claude-3.5-Sonnet, DeepSeek-R1, and DeepSeek-V3 remains unchanged when appearance scores are assigned by different graders.The ordering is Claude-3.5-Sonnet > DeepSeek-R1 > DeepSeek-V3.
- GPT-4o, o3, and Claude-3.5-Sonnet produce the same descending score order across every screenshot set.The reported grading order is GPT-4o > o3 > Claude-3.5-Sonnet.
- Ensemble grader results and human scores match GPT-4o’s ranking, supporting its use for relative appearance evaluation.The paper describes GPT-4o as suitable under a limited budget because its rankings align with human preference.
- Websites with appearance scores from one to five range from major rendering errors or irrelevant content to highly harmonious designs.
N Examples of Websites with Different errors or flaws
The paper illustrates common flaws in generated websites, including loading failures, incomplete interfaces, missing or misplaced content, uncustomized templates, and unsuitable colors.
- Generated websites may fail to load, display only a background, omit UI components, or render images incorrectly.
- Other examples include uncustomized templates, misaligned webpage content, and inappropriate background colors.
- Appearance examples show that higher scores correspond to more harmonious visual designs, while low scores include major rendering errors or irrelevant content.
O Examples of UI Agent Testing Processes
The paper presents UI-agent testing trajectories classified as YES, PARTIAL, or NO according to whether website behavior fully matches, partly matches, or fails to match the expected result.
- YES outcomes occur when the website operation produces the expected result, as illustrated across five testing figures.
- PARTIAL outcomes occur when the website achieves only part of the expected result, as illustrated across three testing figures.
- NO outcomes occur when website behavior does not match the expected outcome, as illustrated across three testing figures.
- The test cases cover operations including form validation, ranking display, link addition, record updates, question submission, customer management, calculations, email, ideas, search, and infrastructure browsing.Each operation is paired with an expected website response, such as error guidance, updated records, successful submissions, or filtered results.
NeurIPS Paper Checklist
The checklist covers claim accuracy, limitations, theoretical assumptions, reproducibility, open access, experimental details, statistical reporting, and societal impact. It requires concise justifications and preserves the checklist structure for review and publication.
- General instructions: Checklist answers must use Yes, No, or NA and include a short justification for each question.The checklist remains part of the submission and is visible to reviewers and area chairs.
- Claims and limitations: Authors should ensure abstract and introduction claims match the paper’s contributions, results, assumptions, limitations, and supported scope.Aspirational goals are acceptable when clearly identified as motivation rather than attained results.
- Claims and limitations: The checklist asks authors to discuss limitations, strong assumptions, robustness to violations, practical scope, and performance-influencing factors.It specifically encourages a separate Limitations section and distinguishes undisclosed limitations from papers with no limitations.
- Theory and societal impact: The checklist separately addresses theoretical results, societal impact, and risks from malicious or unintended uses, privacy, fairness, and security concerns.For papers without theoretical results or societal impact, NA is an available answer when properly justified.
- Reproducibility and access: Experimental reproducibility requires sufficient disclosure of training, testing, optimization, and implementation details, with open access to code and data when applicable.The checklist states that the paper open-sources all code and data and points to the Experiments section for reproducibility details.
- Experimental reporting: Statistical reporting should provide appropriate error bars, confidence intervals, or significance tests and explain variability, calculation methods, and assumptions.The paper justifies omitting error bars because reporting them would be too computationally expensive.