Source-linked AI summary
Generation of Web Apps with Agentic IDEs: An Empirical Assessment
Manuel Marceca, Maria Teresa Rossi, Leonardo Mariani
TL;DR
Evidence about agentic IDEs in end-to-end application development remains limited. We compare three IDEs across five full-stack Web application scenarios. They perform well on established patterns but remain more error-prone on distributed architectures, while developers retain a supervisory role.
Problem
Method
The paper conducts a controlled comparative study of three agentic IDEs across five full-stack Web application scenarios.
Source: §1 · p. 1, §3.4 · p. 4Results
Agentic IDEs perform well on established CRUD and authentication patterns, but distributed architectures and automated test generation remain challenging sources of errors.
Source: §7 · p. 11Takeaways & Limitations
Agentic IDEs can rapidly generate initial implementations but shift developer effort toward supervision, validation, testing, and architectural reasoning as complexity increases.
Source: §7 · p. 11Generalizability is limited by the five application scenarios and chosen technology stack, which do not represent other system classes such as large-scale industrial software.
Source: §4.5 · p. 10
Abstract
from arXiv · showhide
Agentic IDEs are among the most significant innovations in software engineering, aiming to accelerate application development through LLM-based agents that can assist developers during development. However, their evaluation in end-to-end development tasks involving the generation of complete applications remains limited. To fill this gap, we propose a rigorous comparative analysis of three popular agentic IDEs (Copilot, Cursor, and Windsurf) in the generation of five full-stack Web applications from scratch. Results show high maturity in the generation of established patterns, such as CRUD operations and authentication features. In contrast, the generation of less common distributed architectures, such as a task queue architecture, produces significantly more errors. Overall, results show that Agentic IDEs cannot replace developers but shift their role toward building software by orchestrating LLM-based agents through natural-language instructions and iterative refinement. Yet, each agentic IDE shows its peculiarities, although differences are narrow.
1 Introduction
The paper assesses whether IDE-integrated agents can generate complete full-stack applications, addressing limited evidence from end-to-end development tasks. It finds that common features are handled more reliably than complex architectures, while automated testing remains a major challenge.
1 Introduction
- Agentic IDEs let assistants act across project context while developers supervise, correct, and validate; engineering effort consequently shifts toward verification, debugging, integration, and auditing.The study’s findings concern IDE-integrated workflows, where coding, navigation, command and test execution, error inspection, and corrective prompting share a workspace.Source: §1 · p. 1
- Evidence remains limited on whether agentic IDEs can reliably generate complete full-stack applications, beyond isolated coding tasks, repair benchmarks, and studies of user perceptions.The target applications involve backend logic, persistence, security constraints, interactive frontend behavior, and automated testing.Source: §1 · p. 1
- The study compares Copilot, Cursor, and Windsurf across five applications and analyzes 48 generated implementations against four research questions.The scenarios range from a CRUD note manager and authentication module to secure file hosting, real-time chat, and a distributed task queue.Source: §1 · pp. 1–2
- 49.6% of observed errors came from automated test-suite generation; common CRUD and authentication features had low error rates, while complex architectures were more challenging.Differences between IDEs were narrow, with speed and correctness trade-offs; developers’ work shifts toward orchestrating agents through natural-language instructions and iterative refinement.Source: §1 · p. 2
- The study combines automatic measurements with human-judgment checks, evaluates quality across generated code layers, and provides experimental material for reproduction and extension.Its contributions also include actionable findings intended to inform future research.Source: §1 · p. 2
2 Related Work
Prior work studies AI-assisted coding, repository-level generation, and LLM-based agents, but systematic evaluation of agentic IDEs generating complete applications remains limited; this study evaluates that end-to-end task across application layers and iterative correction.
LLM-Based Tools
- LLM tools generate and explain code, but can produce plausible incorrect code, struggle with long-range dependencies and repository reasoning, and introduce security or quality risks.Test generation also remains challenging for high coverage, edge cases, asynchronous behavior, and I/O-intensive scenarios.Source: §2 · p. 2
Modern IDEs and Their Assessment
- AI-assisted IDEs support productivity and collaborative workflows, yet generated-code quality varies with prompts, model configuration, context, and task structure.HumanEval, MBPP, and SWE-bench assess code correctness, synthesis, or issue resolution, but not generation of entire applications at scale.Source: §2 · pp. 2–3
- HumanEval, MBPP, and SWE-bench standardize function-level or issue-resolution evaluation, while repository-level work approaches realistic tasks but not full applications generated from scratch.Existing studies include cross-file planning and repository-level executability and correctness, but focus on code completion or repository modification.Source: §2 · p. 2
LLM-Based Agents
- LLM-based agents combine code generation with planning, tool interaction, and iterative refinement, but the impact of these loops on complex multi-layer systems remains under-explored.This study examines iterative refinement by letting agents improve generated implementations.Source: §2 · p. 3
Generation of Full Stack Applications with Agentic IDEs
- Systematic evaluation of agentic IDEs on end-to-end generation of complete applications remains under-explored, motivating this study’s controlled assessment across architectural layers and iterative correction.The assessment also considers multiple complexity levels and human validation effort.Source: §2 · p. 3
3 Methodology
The methodology compares agentic IDEs on generating full-stack applications from scratch, evaluating functional quality, iterative repair, efficiency, and model evolution through controlled tasks and validation.
3.1 Research Questions
- The study asks whether agentic IDEs generate high-quality, requirement-compliant applications, correct errors without regressions, and reduce generation time and human effort.These questions frame effectiveness, iterative behavior, and efficiency as distinct evaluation goals.Source: §3.1 · p. 3
- Model evolution is examined by repeating Note Manager experiments with Claude Sonnet 3.5 and 4.5 eight months apart.The later runs use updated models and IDE versions to distinguish model-driven improvements from IDE-specific engineering updates.Source: §3.1 · p. 3
3.2 Agentic IDEs
- The comparison includes Copilot’s plugin-based system and the native agentic IDEs Cursor and Windsurf, with Claude Sonnet 3.5 held constant across tools.Claude Sonnet 4.5 is also used for the model-evolution experiments.Source: §3.2 · pp. 3–4
3.3 Experimental Subjects
- Five applications span CRUD, authentication, file handling, real-time chat, and asynchronous task processing, increasing in architectural and operational complexity.The tasks range from established request-response domains to distributed applications involving network interaction or queues.Source: §3.3 · p. 4
- Each application is generated three times, yielding 48 project instances overall, including three additional Note Manager runs with an updated model.Projects undergo systematic automated and manual scrutiny.Source: §3.3 · p. 4
3.4 Experimental Design and Metrics
- The controlled comparison holds task specifications, technology stack, validation, and measurement procedures constant across IDEs.All systems generate multi-file applications using FastAPI, plain HTML/CSS/JavaScript, pytest, and a consistent repository layout; prompts set API, schema, and testing constraints while leaving implementation choices open.Source: §3.4 · pp. 4–5, §3.3 · p. 4
- Generated projects undergo repeated validation and correction until compliant or a 10-iteration cap; all experiments converged, averaging 1.13–1.73 corrective iterations across IDEs.Validation combines executable checks and manual inspection, including installation, frontend behavior, API behavior, persistence, tests, and project structure; Lighthouse and SonarQube assess frontend and backend quality.Source: §3.4 · pp. 5–6
- Evaluation tracks requirement errors by functional area, frontend and backend audit measures, repair regressions, agent execution time, and manual verification time.A regression is a previously satisfied requirement violated in the next iteration; agent time excludes idle time, while verification time captures residual human effort.Source: §3.4 · p. 6
4 Results
Across the five applications, the Agentic IDEs generated working full-stack applications, but their strengths varied and developers still needed to validate and refine the output. Newer models improved results on established patterns, while complex architectures and some quality attributes remained challenging.
4.1 RQ1 - Effectiveness
- Copilot averaged 1,67 errors and led frontend accessibility, Cursor achieved 77,7% backend statement coverage, and Windsurf produced fewer backend reliability and maintainability issues.The tools showed complementary strengths, but no one IDE was consistently superior across projects.Source: §4.1 · p. 7
- Performance varied by project: Copilot favored established patterns, while Cursor and Windsurf alternated on real-time chat and task queue; all struggled more with specialized domains and distributed architectures.Test errors were concentrated in complex I/O, asynchronous behavior, and edge cases, which generated suites did not consistently handle.Source: §4.1 · p. 7
- Persistence and project structure were consistently correct, but API behavior, generated tests, and frontend accessibility were recurring weak points; agents also tended to reuse similar interface styles.Test generation was the most challenging aspect, with generated suites often incomplete or requiring manual adjustment.Source: §4.1 · p. 7
4.2 RQ2 - Iterations
- Validation usually required few corrections (0.33–3.33 iterations per project), Copilot needed slightly fewer iterations, and regressions were rare.Issues that did not produce explicit error messages demanded more developer effort to diagnose; regressions could arise when fixes affected multiple components.Source: §4.2 · pp. 7–8, Table 3 · p. 8
4.3 RQ3 - Efficiency
- Cursor was fastest (6 minutes and 33 seconds), while Windsurf was slowest (14 minutes and 7 seconds); manual verification took roughly 12 minutes across tools.Careful inspection and corrective prompting remained substantial work even when the application was generated in under 20 minutes; command-line assumptions could also slow agents in mismatched environments.Source: §4.3 · pp. 8–9, Table 4 · p. 8
4.4 RQ4 - Evolution
- Moving from Claude Sonnet 3.5 to Claude Sonnet 4.5 improved test coverage from 48–63% to 95–99% and nearly eliminated corrective iterations and regressions on Note Manager.Backend reliability and maintainability trends were mixed, accessibility scores declined slightly, and complex architectural reasoning remained a persistent limitation.Source: §4.4 · p. 9
4.5 Threats to Validity
- Findings are limited by the five application scenarios and FastAPI/plain HTML-JavaScript stack, while measurement and model stochasticity also constrain certainty.The study mitigated these threats with uniform checklists and automated measures, standardized prompts and procedures, and three repetitions per configuration; its findings do not establish readiness for industrial systems.Source: §4.5 · pp. 9–10
5 Actionable Results
Agentic IDEs can generate useful application starting points, but their effective use still involves human inspection, validation, and iterative correction. The actionable findings call for stronger testing and self-correction, attention to human effort and complex domains, and continued research on models and implementation biases.
Actionable results
- Generated full-stack applications may provide professional-development starting points, though developers still need to inspect the code and prompt corrective actions.The study found generated code already satisfied quality aspects such as maintainability and reliability.Source: §5 · p. 10
- Agentic IDEs consistently struggled to generate high-coverage tests, especially for Task Queue, making stronger self-validation and thorough test suites an open research challenge.Thorough tests are needed to validate generated code and increase developers’ confidence in its correctness.Source: §5 · p. 10
- Real-time chat and especially Task Queue performed worse across issue detection and fixing, maintainability, reliability, and testing, indicating a need for approaches suited to less common domains and architectures.The paper identifies domain-specific and architecturally complex applications as a challenge for agentic IDEs.Source: §5 · p. 10
- Multiple iterations were almost always needed for satisfactory implementations, while the few required iterations often converged, motivating more sophisticated assessment and correction mechanisms.The authors argue generation should not be a sequential process without assessment and iteration.Source: §5 · p. 10
- Human validation effort was as much as, and often higher than, AI task time, so the authors prioritize reducing human effort over increasing AI efficiency.They speculate that more specific requirements would raise human effort, making their measured effort an underapproximation.Source: §5 · pp. 10–11
- Comparisons between two generations of Claude Sonnet models support continued research into better AI models.Source: §5 · p. 11
- The three Agentic IDEs differed across all considered dimensions, but the differences were small, making IDE choice a relatively important factor.Source: §5 · p. 11
- Without careful instructions, Agentic IDEs showed implementation biases, including shared frontend-style tendencies and recurrent Linux command-line instructions.Source: §5 · p. 11
6 Takeaways
Agentic IDEs work best on established Web patterns but remain challenging to validate on complex asynchronous tasks. They shift effort from boilerplate toward checking requirements, tests, debugging, and architecture supervision.
Takeaways for Practitioners
- Agentic IDEs perform best on established Web patterns such as CRUD, authentication, and conventional request–response APIs, while asynchronous and stateful tasks require careful correctness validation.Established-pattern applications are close to expected behavior and typically need limited corrective interaction; challenging tasks may still produce plausible, executable code.Source: §6 · p. 11
- Generated test suites may omit requirements or use incorrect oracles, so developers should inspect them and add independent tests, analysis, and code review.Suggested checks include user-implemented tests, automatic static and dynamic analysis, and manual code inspection.Source: §6 · p. 11
- Insufficiently precise prompts can produce overly similar application designs, making explicit personalization important for more diverse, context-appropriate outputs.The homogenization may lead different organizations to unintentionally produce applications with comparable look-and-feel.Source: §6 · p. 11
- Even when agentic IDEs quickly generate complete applications, developers must validate requirements, tests, and architecture because the tools shift effort rather than eliminate it.The work shifts from writing boilerplate toward verifying requirements, assessing tests, debugging, and supervising architecture.Source: §6 · p. 11
Takeaways for Researchers
- Researchers should improve agentic IDEs’ reasoning across asynchronous processes and coordinated components while maintaining consistency across changes.These capabilities may expand the applicability of the tools.Source: §6 · p. 11
- Researchers should improve generated-test oracles, edge-case identification, and coverage assessment so tests adequately and correctly validate intended requirements.Source: §6 · p. 11
- Agentic IDEs should guide systematic artifact checking, since validation protocols currently depend entirely on development teams; research should also quantify validation effort and its interaction with generation, prompting, debugging, and architectural supervision.Source: §6 · p. 11
7 Conclusions
Across 48 generated projects spanning five application scenarios and three agentic IDEs, the study finds that these tools accelerate standard web-application development but still require developer supervision, especially as complexity increases.
7 Conclusions
- The evaluation assessed effectiveness, efficiency, and self-correction across 48 projects, five application scenarios, and three IDEs using functional validation, automated analysis, and manual inspection.Source: §7 · p. 11
- Agentic IDEs generate nearly complete, mostly correct applications for established patterns such as CRUD and authentication, with few errors, rare regressions, and low generation time.These settings require minimal corrective effort and can significantly accelerate development.Source: §7 · p. 11
- Two limitations persist: automated test generation causes nearly half of observed issues, while distributed, coordination-intensive tasks show higher error rates and occasional regressions.Test-generation errors include difficulties with edge cases, asynchronous behavior, and I/O-intensive scenarios; distributed tasks stress concurrency, state management, and asynchronous workflows.Source: §7 · p. 11
- No IDE is universally superior: tools trade off speed, stability, and code quality, while improvements from model evolution are largely driven by advances in underlying LLMs.Source: §7 · p. 11
- Agentic IDEs speed initial implementations and standard components but shift developer effort toward prompting, validation, testing, and architectural reasoning rather than eliminating supervision.The need for supervision becomes especially pronounced as application complexity increases.Source: §7 · p. 11
- Future work should test other domains and scenarios, including whether IDEs can follow fine-grained specifications stated at different levels of detail.Source: §7 · p. 11