PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Development

Yuxuan Wan, Tingshuo Liang, Jiakai Xu, Jingyu Xiao, Yintong Huo, Michael R. Lyu

arXiv:2509.25297v2cs.SEcs.AI

TL;DR

Full-stack web generation remains underexplored because existing methods focus mainly on front-end webpages, while requirements, multi-file dependencies, and functional and visual correctness must be handled together. TDDev uses specialized agents for requirement and test generation, development, and testing within an iterative TDD workflow. It reports a 14.4% improvement in overall accuracy over state-of-the-art baselines, with manual inspection supporting the reliability of UI-agent evaluation.

  • Problem

    Existing web-generation methods are largely limited to front-end tasks, leaving automated full-stack development from user requirements underexplored.

    Source: §1 · pp. 2–3

  • Method

    TDDev orchestrates requirement extraction, executable test generation, full-stack code development, user-interaction simulation, and iterative refinement in a TDD workflow.

    Source: §1 · pp. 3–4

  • Results

    14.4% overall-accuracy improvement over state-of-the-art baselines is reported across diverse application scenarios.

    Source: §1 · p. 4

  • Takeaways & Limitations

    TDDev demonstrates end-to-end generation of web applications addressing both functional correctness and visual fidelity without manual intervention.

    Source: §1 · p. 4, §8 · p. 19

    The evaluation relies on UI-agent testing, so the authors supplement automated results with manual inspection to assess alignment with human judgments.

    Source: §7 · p. 19

Abstract

from arXiv · show

Developing full-stack web applications is complex and time-intensive, demanding proficiency across diverse technologies and frameworks. Although recent advances in multimodal large language models (MLLMs) enable automated webpage generation from visual inputs, current solutions remain limited to front-end tasks and fail to deliver fully functional applications. In this work, we introduce TDDev, the first test-driven development (TDD)-enabled LLM-agent framework for end-to-end full-stack web application generation. Given a natural language description or design image, TDDev automatically derives executable test cases, generates front-end and back-end code, simulates user interactions, and iteratively refines the implementation until all requirements are satisfied. Our framework addresses key challenges in full-stack automation, including underspecified user requirements, complex interdependencies among multiple files, and the need for both functional correctness and visual fidelity. Through extensive experiments on diverse application scenarios, TDDev achieves a 14.4% improvement on overall accuracy compared to state-of-the-art baselines, demonstrating its effectiveness in producing reliable, high-quality web applications without requiring manual intervention.

1 Introduction

Full-stack Req-to-App automation remains difficult because existing approaches largely target front-end generation, while web applications require integrated functionality, testing, and visual quality. TDDev addresses this gap with a multi-agent TDD workflow and reports a 14.4% overall-accuracy improvement over state-of-the-art baselines.

Motivation

  • Full-stack development spans front-end interfaces and back-end logic, including APIs, data storage, and request handling.Source: §1 · p. 2
  • Full-stack development is complex and time-consuming because it demands proficiency across technologies, frameworks, databases, and deployment environments.These demands motivate automatically translating text descriptions or design sketches into functional applications.Source: §1 · p. 2

Research Gap

  • Existing design-to-code methods mainly generate front-end webpages and do not address backend logic, APIs, databases, servers, or multi-component integration.Commercial tools can generate complete websites, but applications produced with state-of-the-art LLMs fail required functionality or compilation in over 70% of reported cases.Source: §1 · p. 2

Contribution

  • TDDev orchestrates requirement-derived tests, full-stack code generation, simulated user interactions, and iterative refinement in an end-to-end TDD workflow.The framework accepts natural-language descriptions or design images and operates without further user intervention.Source: §1 · pp. 3–4

Evaluation

  • 14.4% overall-accuracy improvement over state-of-the-art baselines demonstrates TDDev’s reported effectiveness in generating reliable full-stack web applications.The evaluation uses the multimodal Req-to-App-MM benchmark and requires no manual intervention.Source: §1 · p. 4

Contribution

  • TDDev uses three specialized agents for test generation, development, and testing, addressing underspecified requirements and feedback needs for functionality and visual presentation.The contributions also include extensive experiments, reduced human development effort in a user study, and release of the implementation.Source: §1 · pp. 3–4

2 Background

Req-to-App asks an agent to generate an application whose functionality matches text requirements and whose interface resembles an optional design image. Prior work largely addresses front-end generation or general agentic testing, whereas this paper frames web development as a broader TDD problem.

2.1 Task Formulation: Req-to-App

  • Req-to-App generates a web application from text and optional visual inputs, targeting functional alignment with the description and GUI resemblance to the image.Source: §2.1 · p. 4

2.2.1 UI Code Generation.

  • Front-end code-generation methods have advanced from CNN- and CV-based GUI prototyping to multimodal, layout-aware, interaction-aware, and resource-aware systems, but remain limited in functionality.WebGenBench reports that industry-generated applications fail required functionality or compilation in over 60–70% of cases.Source: §2.2.1 · pp. 4–5

2.2.2 LLM Agents.

  • LLM-agent tools support code generation, repair, and testing, yet commercial website generators still have limited performance and require extensive user intervention.Source: §2.2.2 · p. 5
  • GUI-testing approaches trade off maintenance, coverage, structure, training requirements, or semantic understanding, while MLLM methods still struggle with systematic coverage of targeted web functionality.Web applications additionally require both overall layout checks and fine-grained component behavior.Source: §2.2.3 · p. 5

2.2.3 GUI Testing.

  • Existing test-enhanced code-generation and TDD methods focus on traditional code tasks or program repair and do not apply directly to full-stack web development.Source: §2.2.4 · p. 5

3 Methodology

TDDev coordinates requirement and test generation, template-based full-stack development, and browser-based testing within an iterative feedback loop. Its agents convert high-level inputs into executable specifications, modify selected project files, simulate users, and return actionable refinement feedback.

3 Methodology

  • TDDev orchestrates test-generation, development, and testing agents in a TDD workflow that iteratively refines applications from natural-language or visual specifications.Source: §3 · p. 6

3.1 Test Generation Agent

  • The test-generation workflow decomposes high-level instructions into discrete requirements, including inferred implicit requirements needed for completeness and feasibility.Source: §3.1 · p. 7
  • Requirement elaboration adds functional, static-design, dynamic-interaction, and data-source specifications in deterministic JSON for subsequent implementation.Source: §3.1 · p. 7
  • Soap-opera-inspired cases assign user personas, stepwise actions, and expected outcomes to requirements, producing realistic tests that reduce ambiguity and testing errors.Source: §3.1 · p. 7

3.2 Development Agent

  • The development agent classifies requirements to select among 13 GitHub starter templates, then uses filtered and relevance-selected context to update the project iteratively.It maintains a focused codebase view through file filtering, context-buffer extraction, and an LLM-based relevance-selection step.Source: §3.2 · p. 8, Table 3 · p. 8
  • The development agent applies structured file edits, preferring full-content replacement while supporting explicit unified diffs, formatting cleanup, and protected-file locking.Its prompts include selected file contents, chat history, and current instructions to guide modifications.Source: §3.2 · p. 9

3.3 Testing Agent

  • The testing agent launches the application, checks startup screenshots, executes browser-based interaction tests in parallel, and reports failures with expected-versus-actual outcomes and recommendations.Bounded retries and autonomous registration or authentication handling address transient failures and login barriers.Source: §3.3 · pp. 9–10

4 Experiment

The experiment defines research questions about TDDev’s effectiveness, design components, testing feedback, and developer advantages, then compares it with Bolt.diy and Cursor.

4.1 Research Questions

  • The experiment is organized around four questions covering overall effectiveness, multi-step test generation, testing feedback, and developer-facing advantages.These questions connect system-level performance with specific TDDev design choices.Source: §4.1 · p. 10
  • The evaluation asks whether TDDev automates full-stack development effectively and whether its test-generation and testing-feedback components improve application quality.It also examines advantages for developers over existing open-source tools.Source: §4.1 · p. 10

4.2 Baselines

  • The study compares TDDev against Bolt.diy and Cursor, representing open-source and proprietary industry-level code-agent frameworks.Bolt.diy generates and previews applications through a browser-based WebContainer, while Cursor provides code generation and agent-mode task execution.Source: §4.2 · p. 10
  • The baseline comparison uses Cursor’s agent mode with WebGen-Bench requirements and a subset of 10 test cases, allowing autonomous or default responses to follow-up questions.The process continues until the agent raises no further queries.Source: §4.2 · p. 10

4.3 Backbone Models

The backbone-model evaluation uses GPT-4.1 and Claude-4-Sonnet as proprietary state-of-the-art models, with additional discussion of open models.

4.3 Backbone Models

  • GPT-4.1 and Claude-4-Sonnet serve as the two state-of-the-art proprietary backbone models in the evaluation.The frameworks are evaluated on both models.Source: §4.3 · p. 11
  • The paper also discusses TDDev’s performance with Qwen-2.5-VL and DeepSeek-V3.1 open-source models.Those open-model results are discussed in Section 6.Source: §4.3 · p. 11
  • The backbone-model setup spans proprietary and open-source language models rather than restricting discussion to a single model family.The proprietary models are used for the stated evaluation, while open-source results are discussed separately.Source: §4.3 · p. 11

4.4 Experiment Setup

Experiments run on a 10-core Intel Core i9 iMac with 32GB RAM, using official APIs, zero temperature, and each model’s maximum token allowance.

4.4 Experiment Setup

  • All experiments run on an iMac with a 10-core Intel Core i9 processor and 32GB of RAM.The hardware configuration is fixed across experiments.Source: §4.4 · p. 11
  • All language models are accessed through their official API services.The setup therefore uses API-based model access throughout the experiments.Source: §4.4 · p. 11
  • Model temperatures are set to 0 and maximum tokens are set to each model’s maximum allowable value.These settings are applied to all models.Source: §4.4 · p. 11

4.5 Cost & Efficiency

TDDev automates testing and refinement at a higher per-round token cost than the baselines, while reducing manual effort and maintaining similar development time.

4.5 Cost & Efficiency

  • 0.36 USD per round is TDDev’s token cost, compared with 0.18 USD for Bolt.diy or a 20 USD/month subscription for Cursor.A round denotes a full cycle of code generation and feedback.Source: §4.5 · p. 11
  • All methods require roughly four minutes to develop, but TDDev differs in how testing and refinement are performed.The cost comparison concerns development and testing per round.Source: §4.5 · p. 11
  • TDDev fully automates testing and refinement, whereas Bolt.diy and Cursor rely on manual testing and constant user attention.The paper characterizes TDDev as more efficient despite its slightly higher computational cost.Source: §4.5 · p. 11

4.6 Evaluation

Req-to-App-MM extends WebGen-Bench with design images and evaluates generated applications for functional correctness and visual quality. Functionality is assessed through simulated browser interactions, while visual quality is scored against design images.

4.6 Evaluation

  • Req-to-App-MM augments WebGen-Bench with generated design images, enabling multimodal requirements and visual-consistency evaluation.The evaluation randomly selects 10 websites while preserving the test-set category distribution.Source: §4.6.1 · p. 11

4.6.1 Dataset Construction.

  • BrowserUse simulates each test case from standardized operation-and-result prompts and classifies requirement fulfillment as YES, NO, or PARTIAL.The agent analyzes interaction trajectories and screenshots, with a final judgment when the interaction limit is reached.Source: §4.6.2 · p. 11

4.6.2 Functionality Metrics.

  • The evaluation agent uses Claude-4-Sonnet as its engine, with instructions adopted from [34].Source: §4.6.2 · p. 12

4.6.3 Visual Metrics.

  • Visual quality covers rendering success, content relevance, layout harmony, design modernness, and similarity, scored from 1 to 5 by Claude-4-Sonnet.Visual similarity compares layout, components, color scheme, size and proportion, and overall visual fidelity with the design image.Source: §4.6.3 · pp. 12–13

5 Result & Analysis

TDDev outperforms baseline frameworks in accuracy, reliability, and visual quality, while its multi-step test generation and iterative feedback improve functional outcomes. The system also removes manual intervention during development, though total development time is slightly longer than with Bolt.diy.

5.1.1 Overall Effectiveness.

  • 78.2% accuracy with GPT-4.1 exceeds Cursor’s 60.2% and Bolt.diy’s 25.6%, while Claude-4-Sonnet reaches 70.2% versus 63.8% and 44.5%.TDDev also has the lowest No rate in both configurations and consistently outperforms baselines across most metrics.Source: §5.1.1 · p. 13, Table 6 · p. 14
  • TDDev has no startup failures in either configuration, whereas baselines show launch failures as high as 80.0% for Bolt.diy and 40.0% for Cursor with GPT-4.1.Its appearance score is 4.0 with GPT-4.1, while visual similarity reaches 4.3 with Claude-4-Sonnet but 2.3 with GPT-4.1.Source: §5.1.1 · p. 13

5.1.2 Detailed Result.

  • TDDev achieves strongest or near-strongest category performance, including 91.7% for user interaction, 87.5% for data display, and 100% for design validation with Claude-4-Sonnet.The category-wise evaluation covers both instruction and test-case categories.Source: §5.1.2 · p. 13, Table 7 · p. 14

5.1.3 RQ2: Ablation Study on Test Case Generation Agent.

  • The multi-step test-generation agent raises functionality accuracy to 61.1% from 33.3% for straightforward generation, although straightforward generation reaches 75.0% versus 68.8% in data display.Both approaches achieve 100.0% in design validation.Source: Table 8 · p. 14, §5.1.3 · p. 15
  • Structured requirements guide deeper planning, while refined test cases provide targeted feedback that helps the development agent improve implementations.The multi-step design therefore improves functional reliability and balanced category coverage while maintaining high design fidelity.Source: §5.1.3 · p. 15

5.2 RQ3: Ablation Study on Testing Agent Feedback

  • Three feedback rounds produce 61.1% functionality, 68.8% data display, and 70.2% test accuracy, outperforming the no-feedback baseline’s 58.3%.One or two rounds can reduce accuracy to 25.3%, suggesting insufficient refinement or newly introduced bugs.Source: §5.2 · pp. 15–16, Table 9 · p. 16
  • Three feedback rounds increase visual similarity from 2.3 without feedback to 4.3, while design validation remains 100.0% across settings.The reported gains primarily affect deeper functional and data-related aspects rather than surface styling.Source: §5.2 · p. 16

5.3 RQ4: Advantages for Developers Over Existing Open-Source Tools

  • TDDev requires no manual intervention, whereas Bolt.diy requires 4.7 minutes of intervention across a 15.2-minute session and three interaction rounds.TDDev’s total development time is 18.7 minutes, but users remain free to focus on other work.Source: §5.3 · pp. 16–17, Table 10 · p. 17

6 Discussion

The discussion validates UI-agent testing against manual inspection, then identifies instability from excessive feedback rounds and limitations in current open-source models.

6.1 Accuracy of UI Agent Testing

  • 82.8% of 28 test cases aligned perfectly between BrowserUse and manual testing, with Yes/No rates and Accuracy differing by only 0.8–6.2%.The reported alignment was 23 out of 28 cases.Source: §6.1 · p. 17, Table 11 · p. 18
  • All agent judgment errors arose from partially correct cases, while restricting evaluation to strict Yes/No cases yielded 100% alignment.The strict alignment covered 23 out of 23 cases.Source: §6.1 · p. 17

6.2 Failure Analysis and Future Works

  • Beyond five feedback rounds, overall accuracy showed greater variance because additional refinement opportunities also increased susceptibility to newly introduced errors.The authors therefore caution that excessive feedback cycles may not consistently improve performance.Source: §6.2.1 · p. 18

6.2.1 Scaling Feedback Rounds.

  • Executing multiple rounds and selecting the best output can mitigate feedback-round variation, while branching and rollback are proposed for scalable future refinement.Ten rounds are given as an example selection strategy.Source: §6.2.1 · p. 18

6.2.2 Open-Source LLMs.

  • Current open-source models frequently failed TDDev’s strict JSON/XML formatting requirements, causing parsing errors, truncated outputs, or workflow failures.The limitation concerns both the Test Generation Agent’s JSON and the Development Agent’s XML outputs.Source: §6.2.2 · p. 18
  • Qwen-2.5-VL-72B produced many fail-to-start cases from limited coding ability, whereas DeepSeek-V3.1 lacked visual reasoning and remained limited when paired with Claude-4-Sonnet.The observations indicate difficulty balancing structured output, coding, and multimodal reasoning.Source: §6.2.2 · p. 18

7 Threat to Validity

The validity discussion considers dependence on the multi-agent design and on UI-agent testing reliability, supporting both through ablation evidence and manual inspection.

Effectiveness of the Agentic System Design

  • TDDev’s performance may be constrained by the effectiveness of individual components in its intricate agentic workflow.The authors address this concern with an extensive ablation study.Source: §7 · p. 18
  • The ablation results provide evidence that the test-generation module, testing module, and overall TDD pipeline are robust and effective.This evidence is presented as mitigation for the agentic-design validity concern.Source: §7 · pp. 18–19

Reliability of UI Agent Testing

  • Because evaluation relies on UI-agent testing, the authors supplemented automated testing with manual inspection to assess reliability.The discussion identifies UI-agent reliability and accuracy as a validity concern.Source: §7 · p. 19
  • UI-agent performance closely aligned with human judgments, with an overall alignment rate considered satisfactory.This supports the trustworthiness of the reported automated testing results within the evaluated setting.Source: §7 · p. 19

8 Conclusion

TDDev is presented as a TDD-enabled agent framework for generating end-to-end full-stack web applications from natural-language or visual specifications. Its integrated automation targets functional correctness and visual fidelity while reducing failure rates and manual effort.

Conclusion

  • TDDev generates end-to-end full-stack web applications from natural-language descriptions or visual specifications.The framework is described as the first TDD-enabled LLM-agent framework with this capability.Source: §8 · p. 19
  • By integrating requirement extraction, test-case generation, iterative code refinement, and user-interaction simulation, TDDev addresses approaches limited to front-end generation.The conclusion presents these integrated components as the basis for its full-stack automation.Source: §8 · p. 19
  • Experiments demonstrate functional correctness and visual fidelity alongside significantly reduced failure rates compared with state-of-the-art baselines.The conclusion frames these outcomes as evidence of the potential of TDD-driven LLM agents for web-application generation.Source: §8 · p. 19

Related papers

Loading 2509.25297v2…