Source-linked AI summary
Generation of Web Apps with Agentic IDEs: An Empirical Assessment
Manuel Marceca, Maria Teresa Rossi, Leonardo Mariani
TL;DR
Agentic IDEs remain under-evaluated in end-to-end generation of complete applications. This paper comparatively studies Copilot, Cursor, and Windsurf across five full-stack Web applications, finding maturity on established patterns but persistent difficulty with testing and distributed architectures. The results position developers as supervisors and orchestrators rather than replacements for agentic IDEs.
Problem
End-to-end evidence on agentic IDE performance in complete, multi-layer full-stack application construction remains limited.
Method
The study conducts a controlled comparison of Copilot, Cursor, and Windsurf across five full-stack Web applications, analyzing iterative correction, multiple architectural layers, and human validation effort.
Results
Agentic IDEs perform strongly on established patterns such as CRUD and authentication, but automated testing and distributed, coordination-intensive architectures remain major sources of errors.
Takeaways & Limitations
Agentic IDEs accelerate initial implementations and standard components while shifting developer effort toward prompting, validation, testing, and architectural reasoning.
Takeaways & Limitations
Generalizability is limited because five Web applications and one technology stack do not represent large-scale industrial systems or several production characteristics.
Abstract
from arXiv · showhide
Agentic IDEs are among the most significant innovations in software engineering, aiming to accelerate application development through LLM-based agents that can assist developers during development. However, their evaluation in end-to-end development tasks involving the generation of complete applications remains limited. To fill this gap, we propose a rigorous comparative analysis of three popular agentic IDEs (Copilot, Cursor, and Windsurf) in the generation of five full-stack Web applications from scratch. Results show high maturity in the generation of established patterns, such as CRUD operations and authentication features. In contrast, the generation of less common distributed architectures, such as a task queue architecture, produces significantly more errors. Overall, results show that Agentic IDEs cannot replace developers but shift their role toward building software by orchestrating LLM-based agents through natural-language instructions and iterative refinement. Yet, each agentic IDE shows its peculiarities, although differences are narrow.
1 Introduction
Agentic IDEs extend LLM assistance into supervised, project-wide software development, but their end-to-end capabilities remain insufficiently characterized. This study compares three IDEs across five applications and finds strong performance on established patterns but persistent challenges with complex architectures and testing.
- Motivation: Agentic IDEs let assistants plan changes, modify multiple files, execute commands, and iteratively refine solutions while developers supervise and validate results.Engineering effort consequently shifts toward verification, debugging, integration, and auditing.
- Research gap: Existing benchmarks often emphasize isolated programming tasks or issue resolution rather than interactive, multi-component, full-stack application construction.The paper identifies this as a limitation in current evaluation coverage.
- Study focus: The study evaluates Copilot, Cursor, and Windsurf across five applications spanning CRUD, authentication, file hosting, real-time chat, and distributed task-queue architectures.It analyzes effectiveness, iterative improvement, efficiency and human effort, and model evolution using 48 generated implementations.
- Main findings: Established features such as CRUD and authentication achieve high performance with low error rates and few corrective iterations, whereas complex architectures produce higher error rates.Automated test-suite generation accounts for 49.6% of observed errors, while IDE differences are narrow and involve speed-versus-correctness trade-offs.
- Implication: The paper argues that developers are not replaced but increasingly orchestrate agentic IDEs through natural-language instructions and iterative refinement.Its contributions combine empirical comparison, multiple application complexities, automatic measurements, human judgment, and reproducible experimental material.
2 Related Work
Prior research shows strong localized code-generation performance but weaker coordination across modules, stateful interactions, and architectural constraints. The paper addresses the limited evaluation of agentic IDEs in end-to-end generation of complete full-stack applications.
- LLM-based tools: LLM-based tools can generate code and assist across software-development activities, but generated code may appear plausible while failing during execution.Long-range dependencies, interacting components, repository-level reasoning, security, and quality remain concerns.
- Testing: Automated test generation remains difficult because syntactically valid tests may still miss coverage, edge cases, asynchronous behavior, and I/O-intensive scenarios.The paper connects these known challenges with its own findings on agentic IDEs.
- Assessment approaches: Existing benchmarks such as HumanEval, MBPP, and SWE-bench provide standardized evaluations but focus mainly on function-level synthesis or issue resolution.Repository-level approaches move closer to realistic software engineering by evaluating cross-file context, planning, executability, and correctness.
- Known limitations: Research indicates that effectiveness decreases when systems coordinate modules, manage stateful interactions, or preserve architectural consistency across larger codebases.Generated-code quality also varies with prompts, model configuration, contextual information, and task structure.
- Research gap and contribution: The paper fills an under-explored gap by evaluating agentic IDEs in end-to-end generation of complete full-stack Web applications across layers, complexity levels, iterative correction, and human validation effort.This controlled empirical framing examines both capabilities and limitations together.
3 Methodology
The methodology is a controlled comparative study of three agentic IDEs generating five from-scratch full-stack Web applications of increasing complexity. It combines repeated trials, shared iterative procedures, automatic and manual inspection, and metrics spanning effectiveness, correction, efficiency, and human effort.
- Research questions: The study asks whether agentic IDEs generate effective applications, improve code through iterations, generate applications efficiently, and vary with underlying-model evolution.The evolution comparison repeats Note Manager experiments with Claude Sonnet 3.5 and 4.5 across updated IDE versions.
- Compared systems: Copilot, Cursor, and Windsurf represent plugin-based and native agentic IDE integration patterns, with the initial comparison using the same Claude Sonnet 3.5 model.Claude Sonnet 4.5 is additionally used to address model evolution.
- Generation setup: Applications are generated from scratch using high-level specifications that leave implementation choices to the IDEs while targeting new systems to avoid data leakage.The protocol models developers seeking a rapidly working system under supervised generation.
- Application scenarios: Five applications progress from CRUD and authentication through file handling and real-time communication to asynchronous background processing with queues.The scenarios include a Note Manager, Authentication Module, File Hosting Application, Real-time Chat, and Task Queue.
- Repetitions and inspection: Each application is generated three times, producing 48 project instances overall, and every project is scrutinized through both automatic and manual analysis.The additional Note Manager runs use an updated model.
- Experimental design: The controlled design fixes task specification, technology stack, validation protocol, and measurement procedures while varying the agentic IDE and functionality type.Functionality ranges from well-known domains to specific distributed domains such as real-time chat and Task Queue.
- Validation and metrics: The protocol includes a shared four-stage iterative generation procedure, while repository-structure inspection is the only step requiring qualitative human judgment.Manual verification time serves as a proxy for residual human effort during validation.
4 Results
Agentic IDEs perform strongly on established application patterns but struggle more with complex domains, distributed architectures, and automated test generation. Iterative refinement usually requires few corrections, while tool differences remain narrow and involve trade-offs between speed and correctness.
- Comparative performance: Different IDEs excel in different areas: Copilot averages 1,67 intervention-requiring errors, Cursor reaches 77,7% statement coverage, and Windsurf produces stronger backend code.These complementarities motivate exploring multi-agent solutions, although per-project superiority is not consistently confirmed.
- Quality by layer: Frontend quality is generally high, but recurring issues include insufficient color contrast, missing form labels, console errors, and missing metadata.The agents also tend to reuse similar interface templates across applications.
- Quality by layer: Backend and maintainability are generally good for less challenging domains, with detected issues concentrated in reliability and maintainability rather than critical security vulnerabilities.Test coverage is generally good but remains weaker for complex I/O, asynchronous behavior, and edge cases.
- Error patterns: 49.6% of observed errors come from automated test-suite generation, especially for edge cases, asynchronous behavior, and I/O operations.Generated tests may be incomplete, require manual adjustment, or fail after unsuccessful generation or execution attempts.
- Effectiveness: Agentic IDEs successfully generate full-stack applications, with domain and architectural complexity strongly affecting effectiveness.They perform better on established domains and standard architectural patterns, while Real-time Chat and especially Task Queue produce more issues.
- Self-correction: 0.33–3.33 corrective iterations per project were typically sufficient to obtain an application passing validation, and iterative improvement rarely introduced regressions.Copilot required slightly fewer iterations than Cursor and Windsurf; regressions were seldom introduced while fixing issues.
- Efficiency: 6 minutes and 33 seconds was Cursor’s average execution time, compared with 9 minutes and 43 seconds for Copilot and 14 minutes and 7 seconds for Windsurf.Across projects, execution ranged from 2 minutes and 20 seconds to 20 minutes and 31 seconds without manual coding.
5 Actionable Results
The paper recommends using fully generated full-stack applications as starting points while prioritizing human-effort reduction and recognizing model evolution and IDE choice as relevant factors.
- Teams may consider starting development from full-stack Web applications generated by Agentic IDEs.The generated code can satisfy quality aspects such as maintainability and reliability, making it potentially suitable for professional development.
- Automated test generation remains an open challenge, especially for complex applications such as the Task Queue.Generated tests often fail to achieve high coverage, limiting confidence in automatically generated code.
- Research should prioritize reducing and optimizing human effort rather than only increasing AI efficiency.
- Model evolution remains a key factor in successful software generation from requirements.
- Differences among the three Agentic IDEs are observed across dimensions but remain small, making IDE choice relatively important rather than decisive.
6 Takeaways
Agentic IDEs perform best on established Web-development patterns and are useful for rapid prototyping, but complex asynchronous architectures and generated tests still require careful developer oversight.
- Takeaways for Practitioners: Agentic IDEs perform best on established patterns such as CRUD operations, authentication flows, and request–response APIs.These applications are close to expected behavior and usually require limited corrective interaction.
- Takeaways for Practitioners: Tasks involving real-time communication, background workers, retries, cancellation, asynchronous execution, or complex state transitions remain challenging.Agents may still produce plausible and executable code in these settings.
- Takeaways for Practitioners: AI-generated tests can be incomplete or imprecise, so developers should inspect them and complement them with independent assessment.Suggested checks include user-implemented tests, static and dynamic analysis, and manual code inspection.
- Takeaways for Practitioners: Agentic IDEs can generate similar code artifacts and front-end structures unless prompts provide sufficient precision and personalization.
- Takeaways for Practitioners: Complete applications can be generated quickly, but developer effort shifts toward verifying requirements, assessing tests, debugging, and supervising architecture.
- Takeaways for Researchers: Improving Agentic IDEs requires more reliable test-oracle construction, edge-case identification, and coverage assessment.Generated tests should adequately and correctly validate intended requirements.
- Takeaways for Researchers: Validation tools should guide systematic checking of generated artifacts because validation protocols currently depend entirely on development teams.Understanding validation burden also requires studying its interaction with generation, prompting, debugging, and architectural supervision.
7 Conclusions
Across 48 generated projects, agentic IDEs perform well on established application patterns but remain limited on automated testing and distributed architectures. They offer complementary trade-offs rather than universal superiority, accelerating initial implementation while preserving the need for developer supervision and leaving broader generalization for future work.
- Evaluation scope: 48 generated projects across five application scenarios and three agentic IDEs were evaluated for effectiveness, efficiency, and self-correction.The assessment combined functional validation, automated analysis, and manual inspection.
- Main findings: Established patterns such as CRUD and authentication yield nearly complete, correct applications with few errors, rare regressions, low generation time, and minimal corrective effort.These results indicate that agentic workflows can significantly accelerate development for consolidated architectural patterns.
- Main findings: Automated test generation remains a major error source, while distributed and coordination-intensive architectures involving concurrency, state, and asynchronous workflows show higher error rates and occasional regressions.Test-generation problems include difficulty with edge cases, asynchronous behaviors, and I/O-intensive scenarios.
- Comparison: No IDE is universally superior; instead, the tools exhibit complementary trade-offs involving speed, stability, and code quality.The observed improvements are largely driven by advances in the underlying LLMs.
- Implications: Agentic IDEs accelerate initial implementations and standard components but shift developer effort toward prompt engineering, validation, testing, and architectural reasoning as complexity increases.Developer supervision remains necessary rather than being eliminated by the tools.
- Limitations and future work: Future work should test additional domains and scenarios, including the ability of agentic IDEs to follow fine-grained specifications.The stated scope boundary is full-stack Web application generation.