Source-linked AI summary
From Runnable to Shippable: Multi-Agent Test-Driven Development for Generating Full-Stack Web Applications from Requirements
Yuxuan Wan, Tingshuo Liang, Jiakai Xu, Jingyu Xiao, Yintong Huo, Michael R Lyu
TL;DR
Generated web applications often remain runnable but fail functional requirements, while humans must manually deploy, test, and diagnose them. TDDev automates this closed TDD loop and improves generation quality by 34–48 percentage points, with the best protocol depending on model generation style.
Problem
State-of-the-art coding agents produce applications that fail functional requirements in over 70% of cases, while automating deployment, browser testing, and failure translation remains limited.
Method
TDDev converts requirements into acceptance tests, deploys and validates applications through simulated browser interactions, and translates failures into structured repair reports.
Results
Generation quality improves by 34–48 percentage points over no TDD across agent–model combinations on WebGen-Bench, with benefits also observed on ArtifactsBench.
Takeaways & Limitations
TDD enforcement should match model generation style: holistic models favor agentic TDD, whereas conservative models favor incremental TDD.
Takeaways & Limitations
Automated testing-agent verdicts achieve 87.5% accuracy and may misclassify outcomes, introducing noise into accuracy measurements.
Abstract
from arXiv · showhide
Coding agents can generate web applications from natural-language descriptions, yet a recent benchmark study shows that generated applications fail to meet functional requirements in over 70% of cases. The core difficulty is that web correctness cannot be assessed from source files or terminal output: the application must be deployed, exercised through simulated browser interactions, and failures must be translated into actionable repair signals -- steps that current agents cannot perform without human mediation. We present TDDev, a framework that automates this closed loop through three stages: (1) converting high-level requirements into structured acceptance tests before any code is written, (2) deploying the application and validating it through browser-based interaction simulation, and (3) translating browser-observed failures into structured repair reports for the coding agent. Enabled by TDDev, we conduct the first controlled empirical study of Test-driven development (TDD) strategies for web application generation, comparing four development protocols across two coding agents, two backbone models, and two benchmarks. TDD infrastructure consistently improves generation quality by 34--48 percentage points over a no-TDD baseline. The central finding is that the optimal protocol depends on the model's generation style: models that build applications holistically benefit most from agentic enforcement, while models that extend code conservatively benefit from incremental enforcement. Mismatching protocol to generation style eliminates the TDD benefit entirely while multiplying token cost up to 25-fold. A user study confirms that TDDev reduces manual developer intervention to zero, shifting the workload from continuous prompt engineering to autonomous, feedback-driven refinement.
1 Introduction
TDDev closes the runnable-to-shippable gap by converting natural-language requirements into acceptance tests, validating deployed applications through browser interactions, and producing structured failure reports for autonomous repair. A controlled study finds that TDD infrastructure improves generation quality, but the best enforcement protocol depends on the model’s generation style.
- Motivation: Over 70% of applications generated by state-of-the-art agents fail to meet functional requirements, exposing the gap between runnable prototypes and shippable web applications.Web correctness requires deployment and simulated browser interaction rather than source-file or terminal inspection alone.
- Challenges: Web TDD requires concretizing vague requirements, validating interactive behavior, and translating observed failures into actionable feedback.These challenges involve executable browser scripts, deployed-application testing, and structured repair instructions.
- TDDev: TDDev automates the closed loop by generating structured acceptance tests, deploying and exercising applications through browser simulation, and producing structured failure reports.The framework reduces reliance on manual deployment, interaction, observation, and prompt-based repair guidance.
- Empirical study: 34–48 percentage points of generation-quality improvement are obtained from TDD infrastructure over a no-TDD baseline across the controlled evaluation.The study compares four development protocols across two coding agents, two backbone models, and two benchmarks.
- Protocol fit: Holistic-generation models benefit most from agentic TDD, whereas conservative-generation models benefit from incremental TDD.Mismatching protocol and generation style eliminates the TDD benefit and can multiply token cost up to 25-fold.
2 Background
Full-stack web application correctness requires deployment, browser rendering, and satisfaction of interaction-based acceptance tests derived from textual requirements. Prior work addresses visual fidelity, software-engineering tasks, GUI testing, and test-guided generation, but leaves a gap in validating full-stack functionality in deployed applications.
- Application Correctness: Correctness requires a deployable application that renders in a browser and satisfies every interaction-and-outcome pair in an acceptance suite derived from the requirements.The application passes only when all acceptance-suite elements are satisfied in the deployed environment.
- Related Work: UI code-generation research improves front-end visual fidelity from screenshots or design images but largely targets appearance rather than full-stack functionality.WebGenBench reports that state-of-the-art systems frequently fail to satisfy functional requirements.
- Related Work: Coding-agent research spans repository-level issue resolution, program repair, build automation, and multi-agent workflows, enabled by directly accessible execution environments.Trajectory analysis also identifies behavioral patterns distinguishing successful and failed executions.
- Related Work: GUI testing methods trade off usability, coverage, maintenance, structure, adaptability, and semantic understanding across record-and-replay, random, model-based, learning-based, and MLLM-based approaches.MLLM-based methods incorporate visual semantics and functional structure, while earlier paradigms face fragility, limited coverage, model dependence, training-data demands, or poor adaptation.
- Related Work: Test feedback improves code generation and repair through training signals, fault localization, patch validation, prompt-provided tests, and interactive clarifying tests.Prior work therefore demonstrates multiple ways to use test outcomes during software development.
3 Methodology
TDDev establishes a closed test-driven loop for web application generation: it derives acceptance tests from requirements, validates deployed applications through browser interactions, and converts failures into repair-ready feedback. The methodology also varies how strictly this loop is enforced, from incremental feature-by-feature development to agent-controlled execution and a no-TDD baseline.
- Acceptance test generation: TDDev converts natural-language requirements into executable acceptance tests before coding, creating an unambiguous target and stable evaluation criteria.Each test specifies a feature, ordered interaction steps, and an observable expected outcome; tests are exposed for review before development.
- Browser validation: TDDev deploys each generated application and uses an LLM-backed browser agent to execute acceptance tests from accessibility-tree observations and test context.The agent operates through Playwright while adapting to application-specific element structures, labels, and navigation flows.
- Acceptance test generation: Requirement derivation uses concrete user personas and goals to produce valid, diverse tests rather than generic, overlapping interpretations.The approach reframes testing around who uses the application and what they want to accomplish, drawing on scenario-based testing.
- Failure translation: When a test fails, TDDev translates the interaction trajectory and observed deviations into structured repair reports that give the coding agent a concrete repair starting point.Reports retain the feature, failed step, observations, and completed steps instead of providing only a vague failure description.
- Enforcement protocols: TDDev evaluates enforcement protocols spanning Incremental feature-by-feature repair, Whole-project repair, Agentic self-directed tool use, and a Non-TDD baseline without TDD infrastructure.Incremental enforces the strictest deploy–test–repair loop, Whole-project repairs the full application after a single implementation pass, and Agentic leaves workflow decisions to the agent.
4 Experiment
The experiment evaluates TDDev’s module reliability and TDD effects across coding agents, backbone models, benchmarks, enforcement conditions, and feedback budgets. It measures test coverage, verdict accuracy, accuracy across attempts, and token consumption under browser-based testing.
- Evaluation metrics: Module reliability is assessed through generated-test coverage against ground-truth features and testing-agent agreement with predetermined verdicts on correct and broken fixtures.Coverage uses LLM-based semantic matching to handle paraphrase, while fixture accuracy is reported separately for correct and broken variants.
- Experimental design: RQ2–RQ4 span four experimental combinations that vary the coding agent, backbone model, and benchmark to assess generalizability.Each combination runs Whole-Project, Incremental, Agentic-TDD, and Non-TDD conditions.
- Benchmarks: WebGen-Bench supplies 101 tasks with human-validated requirements, of which 50 cases are randomly sampled with seed=42 for the main experiments.ArtifactsBench is also used for automated multimodal evaluation of dynamic web UI code generation.
- Implementation: Experiments use ClaudeSDK and OpenCode with Claude Sonnet 4.6 and Qwen-3.5-397B-A17B, while browser testing runs through Playwright with Chromium.ClaudeSDK is the primary agent; OpenCode receives TDDev tools through MCP under Agentic-TDD.
- Experimental design: The four conditions vary only in TDD-loop enforcement, with Whole-Project and Incremental allowing K=5 attempts and Agentic-TDD and Non-TDD invoked once.Every Whole-Project and Incremental attempt is logged for post-hoc feedback-budget analysis.
- Evaluation metrics: RQ2 reports acc@K, RQ3 compares acc@K profiles across enforcement conditions, and RQ4 analyzes the full acc@k curve as feedback attempts increase.Token consumption is recorded as a secondary cost metric.
5 Results
TDDev reliably generates acceptance tests and detects real application defects, while TDD substantially improves web-generation quality when enforcement matches the model’s generation style and task complexity. Additional feedback rounds help some protocols and models, but autonomous enforcement can outperform repeated repair.
- Test generation and defect detection: 91.9% mean test coverage matched 57 of 62 reference cases, with seven of ten applications reaching 100% coverage.The module generated 12.4 versus 6.2 reference tests per application on average, decomposing vague requirements into finer-grained criteria.
- Test generation and defect detection: Misses were confined to structural features requiring operational knowledge absent from high-level requirements, while core domain functionality was fully covered.Examples included site-wide navigation and volunteer information elements apparent only from detailed UI walkthroughs.
- Test generation and defect detection: 87.5% overall testing accuracy comprised 100% broken-variant detection and 75% correct-variant accuracy, with zero false positives.The five false negatives arose from selector-generation errors or conservative rejection of minor wording differences, making the failure mode safe but occasionally costly.
- TDD quality gains: 34–48 percentage-point gains over Non-TDD occurred across WebGen-Bench combinations, compared with a 7.6 percentage-point gain on ArtifactsBench.The smaller ArtifactsBench improvement reflects narrower, simpler tasks often satisfiable in one attempt.
- Protocol-model interaction: The optimal enforcement level depended on generation style: holistic models favored Agentic-TDD, whereas conservative models benefited from structured incremental enforcement.Incremental enforcement matched Qwen’s conservative behavior but did not improve Sonnet over Non-TDD; ArtifactsBench showed little differentiation across enforcement levels.
- Feedback-round dynamics: Whole-Project accuracy roughly doubled by the fifth attempt, while Incremental stagnated for Sonnet but improved from 59.7% to 71.4% for Qwen.Most Whole-Project gains occurred in the first two rounds, and Sonnet’s single Agentic-TDD attempt at 65.8% exceeded five rounds of enforced repair.
6 Discussion
TDDev’s benefits depend on matching TDD enforcement to the coding agent’s generation style, with matched protocols improving accuracy and cost efficiency while mismatches waste retries. User-study evidence shows TDDev removes manual intervention by shifting developers from continuous prompting to autonomous feedback-driven refinement, though automated test verdicts remain imperfect.
- Model–protocol alignment: Matched protocols align with agents’ generation styles: holistic Sonnet benefits from agentic TDD, while conservative read-then-extend Qwen benefits from incremental TDD.Sonnet rewrites complete implementations, whereas Qwen extends existing code incrementally; protocol structure therefore determines whether each model’s strengths are amplified or exposed.
- Model–protocol alignment: 65.8% vs. 31.3% and 71.4% vs. 23.3%: matched Sonnet-agentic and Qwen-incremental configurations more than doubled accuracy over the no-TDD baseline.The finding also identifies enforcement level as consequential: mismatched protocols can waste the entire retry budget.
- Cost trade-offs: 5.9M tokens and 108.7M tokens: the most accurate Sonnet-agentic and Qwen-incremental conditions had sharply different cost profiles, with Qwen’s additional accuracy costing 25 times less efficiency.Sonnet’s condition was 3.4M tokens above baseline at approximately 91K tokens per additional percentage point; Qwen’s was 106.8M above baseline at 2,327K per point.
- Cost trade-offs: 0 accuracy gain and 9.9M tokens: mismatched Sonnet-incremental and Qwen-agentic conditions simultaneously reduced accuracy and wasted token budgets.Qwen-agentic achieved only a 16.7pp gain, whereas whole-project TDD achieved +26.7pp at 4.5M tokens.
- Developer workload: 0 prompts and 0 redirections: TDDev eliminated manual intervention, unlike the baseline, which required repeated diagnosis and reformulation throughout development sessions.Bolt.diy required 4.7 minutes of manual input, three prompting rounds, and 74 additional guidance words during a 15.2-minute session; TDDev took 18.7 minutes, with the extra 3.5 minutes running autonomously.
- Limitations: 87.5% accuracy: the testing agent’s conservative oracle bias produced false negatives but no false positives, with residual errors affecting all conditions equally.These oracle errors are unlikely to change relative comparisons, but they introduce noise into accuracy measurements.
7 Conclusion
TDDev closes the runnable-to-shippable gap by automating acceptance-test generation, browser-based validation, and structured failure reporting. The study finds that TDD enforcement benefits depend on the model’s generation style, while future work targets broader workflows, model families, and adaptive protocols.
- TDDev framework: TDDev automates executable acceptance-test creation, application deployment with simulated browser interactions, and translation of observed failures into structured repair signals.These steps address web-validation tasks that otherwise require human mediation.
- Empirical study: The controlled study compares four development protocols across two coding agents and two backbone models.
- Empirical study: TDD benefits appear across all three agent–model combinations on WebGen-Bench and also hold on ArtifactsBench, although with a smaller margin.The smaller ArtifactsBench margin is attributed to its narrower task distribution.
- Protocol selection: The optimal enforcement strategy interacts systematically with the model’s intrinsic generation style.This interaction extends beyond the aggregate improvement and is identified as a central study finding.
- Future work: Future work will extend validation to authenticated multi-user workflows, test generalization across larger model families, and explore runtime-adaptive enforcement.