PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies

Siddhant Saxena, Nilesh Trivedi, Vinayaka Jyothi

arXiv:2605.04637v1cs.MAcs.SE

TL;DR

Whether natural-language AI app builders produce production-ready software remains largely unanswered, and existing evaluation frameworks do not cover the full scope of their claims. This paper introduces SWE-WebDev Bench to evaluate these platforms and finds recurring requirement-specification and production-readiness gaps, with no platform above 60% on engineering quality.

  • Problem

    Whether generated applications are production-ready remains largely unanswered, and existing evaluation frameworks do not address the full scope of AI app-building platforms’ claims.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces SWE-WebDev Bench, an evaluation framework designed to assess AI app-building platforms and surface their specific failure modes.

    Source: §1 · p. 2

  • Results

    Across the sample, platforms showed recurring requirement-specification and production-readiness gaps; no platform scored above 60% on engineering quality.

    Source: §8 · p. 31

  • Takeaways & Limitations

    The reported patterns are descriptive of this sample; larger-scale replication is needed to establish their generality.

    Source: §8 · p. 31

    The benchmark’s emphasis may favor platforms with dedicated PM agents and those optimized for complex T4/T5 applications, while different design choices could change platform rankings.

    Source: §7.6 · p. 31

Abstract

from arXiv · show

The emergence of "vibe coding" platforms, where users describe applications in natural language and AI agents autonomously generate full-stack software, has created a need for rigorous evaluation beyond code-level benchmarks. In order to assess them as virtual software development agencies on understanding business requirements, making architectural decisions, writing production code, handling iterative modifications, and maintaining business readiness, we introduce SWE-WebDev Bench, a 68-metric evaluation framework spanning 25 primary and 43 diagnostic metrics across seven groups, organized along three dimensions: Interaction Mode (App Creation Request (ACR) vs. App Modification Request (AMR)), Agency Angle (Product Manager (PM), Engineering, Ops), and Complexity Tier (T4 multi-role SaaS, T5 AI-native). Our evaluation (six platforms, three domains, 18 evaluation cells) reveals four recurring shortcomings in the current generation of AI app builders: (1) A specification bottleneck, where platforms compress rich business requirements into oversimplified technical plans, (2) A pervasive frontend-backend decoupling, where visually polished UIs mask absent or broken backend infrastructure, (3) A steep production-readiness cliff, where no platform scores above 60% on engineering quality and post-generation human effort varies substantially across platforms and (4) Widespread security and infrastructure failures, with no platform exceeding 65% Security Score against a 90% target and concurrency handling as low as 6%. These observations are descriptive of our sample and require larger-scale replication to establish generality. We release SWE-WebDev Bench as a community benchmark to enable such replication and help platform builders identify and address these gaps. Code and benchmark resources are available at: https://github.com/snowmountainAi/webdevbench and https://webdevbench.com/.

1 Introduction

SWE-WebDev Bench evaluates whether AI app-building platforms can function as complete software agencies, diagnosing their ability to turn natural-language business intent into production-ready applications. It introduces a multidimensional benchmark, an initial platform evaluation, and resources for independent replication.

1 Introduction

  • Existing evaluation frameworks do not cover the full agency pipeline, including requirement elicitation, iterative modification, security, infrastructure, and business readiness.Code-level benchmarks require code-level specifications; issue-solving benchmarks focus on existing codebases, while application-level benchmarks omit parts of the full pipeline.Source: §1 · pp. 1–2
  • No evaluated platform exceeds 60% on engineering scores, and Replit’s scores vary by 13 points between domains P1 and P3.The figure reports engineering scores across six platforms and three business domains.Source: Figure 1 · p. 2
  • The proposed framework has 68 metrics organized across Mode × Angle × Tier, with a four-tier judging taxonomy to diagnose production-readiness gaps.The framework is designed to assess AI app builders across distinct interaction modes, agency angles, and complexity tiers.Source: §1 · p. 2
  • SWE-WebDev Bench separately evaluates App Creation Requests and App Modification Requests, treating modification handling as a distinct and harder competency.The paper presents ACR/AMR as the first benchmark distinction of its kind.Source: §1 · p. 2
  • The Canary Requirement methodology uses 80 culturally specific, domain-embedded requirements across four types to distinguish comprehension from template matching.The types are Original, New, Surviving, and Contradiction.Source: §1 · p. 2
  • An initial evaluation covers six platforms across three business domains and reports four recurring shortcomings across the evaluated platforms.The paper describes these observations as an initial evaluation.Source: §1 · p. 3
  • The benchmark is released with prompts, rubrics, and evaluation protocols to support independent replication and benchmarking.The passage provides the project’s GitHub repository and website.Source: §1 · p. 3

2 Related Work

Prior work spans code-task and whole-application benchmarks, visual and agent evaluation, LLM judging, and benchmark governance; SWE-WebDev Bench positions itself as complementary by evaluating requirements, iterative modification, and the pipeline from business intent to deployment readiness.

2.1 Code Generation Benchmarks

  • Code-generation benchmarks progress from function- to class-level tasks, but rely on code-level specifications and assess patches rather than complete business-ready applications.SWE-bench, SWE-bench-Live [6], and FeatBench [7] move toward realistic issue and feature work; FeatBench finds 73.6% of failures stem from regressive implementation, motivating AMR evaluation. These benchmarks generally omit PM, deployment, and business-readiness dimensions.Source: §2.1 · p. 3

2.2 Web Application Generation Benchmarks

  • Existing web-application benchmarks broaden evaluation, but none covers requirement elicitation, iterative modification, and the full pipeline from business intent through deployment readiness.Vibe Code Bench reports 61.8% accuracy for its best model, while WebGen-Bench reports 27.8% accuracy for its best agent; SUSVIBES finds 61% of solutions functionally correct and only 10.5% secure. SWE-WebDev Bench complements these efforts by examining why applications fail and what to fix.Source: §2.2 · pp. 3–4

2.3 Visual and Multimodal Evaluation

  • AI app builders can compress ambiguous business requirements into oversimplified technical plans, a specification loss measured by the PM Agent evaluation dimension.The paper describes this as a specification bottleneck that loses critical domain context.Source: §2.3 · p. 4

2.4 Agent Evaluation Methodology

  • Cognition AI’s evaluation of a single agent on developer tasks informs SWE-WebDev Bench’s Ops/Maintenance angle, while SWE-WebDev evaluates six platforms on business-user tasks across multiple quality dimensions.The related work highlights Cognition’s simulated users, evaluator agents, and interactive self-reflection as relevant precedents.Source: §2.4 · p. 4

2.5 LLM-as-Judge Evaluation

  • LLM judges can approximate human preferences but exhibit systematic and platform-specific biases, motivating structured rubrics and tiered assignment of evaluation tasks.MT-Bench [18] reports > 80% agreement while identifying position, verbosity, and self-enhancement biases; Kim et al. [19] support detailed rubrics, and Li et al. [20] document platform-specific code-evaluation bias. SWE-WebDev assigns high-stakes subjective metrics to Tier 3 expert panels and uses LLM judges for factual verification, with agreement reported in §3 and calibration limitations discussed in §7.6.Source: §2.5 · p. 4

2.6 Benchmark Governance and Maintenance

  • HELM [21], Dynabench [22], and Chatbot Arena [18] provide precedents for living-benchmark maintenance, adversarial evaluation, and community-driven pairwise assessment that inform SWE-WebDev Bench’s governance plan.Source: §2.6 · p. 4

3 The SWE-WebDev Bench Framework

SWE-WebDev Bench evaluates AI app-building platforms as end-to-end software agencies through 68 metrics—25 primary and 43 diagnostic—across interaction mode, agency role, and complexity tier. Its framework is designed to surface invisible failures, diagnose their causes, and assess production readiness.

3.1 Design Principles

  • The framework evaluates the full delivery pipeline, prioritizes failures users cannot verify, pairs scores with diagnostic metrics, and resists benchmark gaming through varied requirements, prompts, and tasks.Its anti-gaming mechanisms include culturally embedded canary requirements, varied prompt styles, and distinguishing creation from modification requests.Source: §3.1 · p. 5

3.2 Evaluation Cube: Three Orthogonal Dimensions

  • The evaluation cube crosses creation versus modification, Product Manager, Engineering, and Operations roles, and T4 SaaS versus T5 AI-native application complexity.AMR tests modifying existing code while preserving functionality; T4 includes features such as RBAC and scheduled jobs, while T5 adds LLM pipelines and trust/safety constraints.Source: §3.2 · pp. 5–6, Figure 2 · p. 6
  • PM capability varied 3.5× across platforms on Inference Quality Score, while Engineering scores had a 6-point spread on Frontend Engineering, making PM the primary differentiator.The authors present these results as support for tagging metrics by agency angle to help diagnose where platform quality breaks down.Source: §3.2 · p. 6

3.3.1 Metric Group Rationale

  • Primary metric groups assess specification fidelity, code quality, integrations, and security and scale, including requirement comprehension, software architecture, real-world services, and concurrent-user performance.Code Quality separates five dimensions; platforms can exceed 70% on Frontend Engineering Score while falling below 10% on Schema Design Score, which composite scores could mask. Security Score, Scalability Architecture Score, and Concurrency & Load Score have targets of ≥90%, ≥70%, and ≥70%, respectively; no platform exceeded 65% on Security Score.Source: §3.3.1 · pp. 7–8
  • The remaining primary groups measure changeability, business readiness, and production readiness, covering regressions and human effort, market-facing features, and costs or gaps between claims and working software.Their metrics include Code Change Impact Score, Effort-to-Fix, Post-PRD Human Effort, SEO & Web Standards Score, Lead & Growth Score, Multilingual/Localisation Score, AI Feature Quality, Total Cost of Correctness, Feature Gap Delta, and Claim Drift Index.Source: §3.3.1 · p. 8

3.3.2 Diagnostic Metrics

  • The 43 diagnostic metrics trace outcomes to planning, build and code, deployment and maintenance, or operator-experience causes; 42% of metric pairs had weak or no correlation.For example, low Feature Completeness can reflect missed stated requirements, hallucinated features, or failure to implement a sound plan; the authors note the seven groups are not fully independent latent factors.Source: §3.3.2 · p. 8

3.4 Judging Taxonomy

  • Judging ranges from automated checks to LLM judges, LLM-plus-human review, and expert panels; observed agreement was κ = 0.82, 0.71, and 0.64 for Tiers 1, 2, and 3.Tier 0 requires no human judgment; highest-stakes metrics BIF, ETF, and FGD receive Tier 3 expert scrutiny. The authors note the calibration samples were small and release prompts and raw scores for independent calibration.Source: §3.4 · p. 9

3.4.1 Target Threshold Calibration

  • Aspirational targets are diagnostic thresholds, not pass/fail gates; Feature Completeness Score targets ≥85% and Canary Retention Rate targets ≥80%.Security Score and Concurrency & Load targets are grounded in external guidance, while some targets reflect author experience and may benefit from broader practitioner calibration.Source: §3.4.1 · p. 9

4 Experimental Design

The experimental design probes six AI app-building platforms with standardized creation and modification prompts across three distinct business domains, using canary requirements and a seven-phase evaluation protocol.

4.1 Platform Selection

  • The study evaluates six platforms spanning different AI app-building approaches.The authors note that two are affiliated with one evaluated platform and refer to §7.6 for bias mitigation.Source: §4.1 · p. 9, §4.2 · p. 10

4.2 Prompt Suite and Snorkel Alignment

  • Six standardized prompts—three App Creation Requests (ACR) and three App Modification Requests (AMR)—align with Snorkel dimensions of environment complexity, autonomy horizon, and output complexity.Each prompt stresses at least two dimensions; ACR prompts also vary authoring style to test robustness to input variation.Source: §4.2 · pp. 9–10

4.2.1 Domain Selection Rationale

  • The three business domains were designed as orthogonal probes so platforms could not score well through a single architectural strength.EdTech tests inference from culturally specific pain points, Field Service tests precise implementation of complex stated logic, and FinTech-AI tests safe, auditable AI behavior.Source: §4.2.1 · p. 10
  • No platform dominates across EdTech, Field Service, and FinTech-AI, with domain-specific score swings of up to 13 percentage points.The domains probe inference depth, faithful execution of detailed business logic, and trustworthy AI features, respectively.Source: §4.2.1 · p. 10

4.2.2 Prompt Style Variation

  • ACR prompt styles range from an informal founder account to an enterprise RFP and a maximum-complexity AI-native specification.The EdTech and Field Service prompts include deliberately contradictory visibility requirements to test whether agents flag conflicts; the FinTech-AI prompt imposes trust and safety requirements such as confidence scoring and mandatory escalation.Source: §4.2.2 · pp. 10–11

4.3 AMR Prompts: Escalating Mutations

  • Three AMR prompts vary complexity, specificity, and change type to diagnose modification handling.P4 retrofits multi-tenancy, P5 requires inferring features from a vague request, and P6 tests an Anthropic-to-Gemini provider fallback and PDF reports without breaking the credit system.Source: §4.3 · p. 11

4.4 Canary Requirements

  • The canary methodology uses 80 culturally specific, domain-embedded requirements to distinguish genuine requirement retention from template matching.Its Surviving type tests whether requirements persist or evolve correctly from ACR to AMR, while Contradiction canaries test whether inconsistent requirements are flagged rather than silently resolved.Source: §4.4 · pp. 11–12

4.5 Evaluation Protocol

  • Each platform-prompt evaluation follows a seven-phase, 14-day protocol grounded in observable evidence, with Engineering Score computed as a weighted mean of applicable G2–G6 primary metrics.The pipeline covers build execution, code and automated audits, feature and canary checks, AMR change injection, expert review, and diagnostic scoring; G1 specification fidelity and G7 cost are reported separately.Source: §4.5 · p. 12, Figure 3 · p. 13

5 Results · 5.1 Cross-Platform Engineering Scores · 5.2 Finding 1: The Specification Bottleneck

Across six platforms, engineering scores remain below production-readiness targets, while requirements elicitation and build-time feedback vary substantially. These observations are descriptive of a small sample, not statistically significant platform rankings.

5 Results

  • The study reports observed scores for 18 ACR evaluation cells, without significance tests or confidence intervals, and calls for larger-scale replication before robust rankings.The authors estimate 15–20 prompts across 5–7 domains would be needed to detect medium effects with paired permutation tests.Source: §5 · p. 13

5.1 Cross-Platform Engineering Scores

  • No platform exceeds 60% engineering score, and every platform scores below 15% on at least one metric, leaving production readiness unresolved in this sample.The evaluation covers six platforms and three ACR prompts; the reported scores are descriptive.Source: §5.1 · p. 13, Table 5 · p. 14

5.1.1 Comprehensive Per-Metric Breakdown

  • No platform passes all metrics; even the highest-scoring platform fails 17 of 22 targets, while Frontend Engineering scores cluster at 68–74% and CBS ranges from 0% to 49%.Table 6 reports averages across applicable ACR primary metrics; the backend and infrastructure gaps include background jobs, scheduled tasks, and complex integrations.Source: Table 6 · p. 14, §5.1.1 · p. 15

5.1.2 Per-Prompt Results

  • On P3, Lovable scores 72% on Schema Design, Replit scores 78% on Frontend Engineering, and v0-Max scores 0% on Schema Design but 72% on Frontend Engineering.Replit’s overall score swings from 54.5% on P3 VettAI to 41.5% on P1 ExamEdge, while no platform dominates all domains.Source: §5.1.2 · p. 15

5.2 Finding 1: The Specification Bottleneck

  • Most platforms compress rich business requirements into oversimplified plans; Canary Retention Rate spans 17.7%–97.7% and Inference Quality Score spans 20–70.Requirement elicitation accounts for approximately 10–15% of the observed quality gap, while whether platforms maintain a build-time feedback loop appears to account for a larger share.Source: §5.2 · p. 16

5.2.1 PM Agent Interaction Traces

  • On the same P1 prompt, QwikBuild asks 15 business-workflow questions across three rounds, Emergent asks five infrastructure questions, and Replit and v0-Max ask none.The excerpts compare PM-agent behavior across four platforms responding to P1 ExamEdge.Source: Figure 6 · p. 18, §5.2.1 · p. 18

5.2.2 Contradiction Handling: The P2 Trap

  • All six platforms detect the explicitly marked P2 contradiction, but responses range from integrating a resolution to flagging it or deferring core features.At the weaker end, one platform flags the contradiction but defers the SLA engine, audit trail, invoicing, time tracking, and customer signatures to future scope.Source: §5.2.2 · pp. 18–19, Table 9 · p. 19

5.2.3 Per-Prompt PM Scores

  • On P3, Under-Clarification Rate reaches 85% for v0-Max and Replit, while only one platform proactively detects the unmarked P1 contradiction.Replit and Base44 have zero conversational turns on P1 and P3; Base44’s plans have 60–72% Plan Structural Validity despite zero questions, but assumptions remain unvalidated.Source: §5.2.3 · pp. 19–20

5.3 Finding 2: The Frontend-Backend Decoupling Problem … 5.6 Preliminary AMR Analysis: Single-Platform Methodology Demonstration

The evaluation finds that strong frontend quality does not ensure backend capability or production readiness, and that security and concurrency remain far below targets. Preliminary AMR results from QwikBuild show both modification-related degradation and cases where modification improves functionality, but require independent cross-platform replication.

5.3 Finding 2: The Frontend-Backend Decoupling Problem

  • At similar frontend scores, platforms diverge sharply in backend capability, forming infrastructure-integrated, ecosystem-leveraged, and frontend-prioritized archetypes.QwikBuild’s FES 68% accompanies the highest CBS (49.3%), while v0-Max’s FES 68% accompanies zero database schema on P3 and zero background jobs across prompts; closing gaps in jobs, integrations, and data pipelines remains a key challenge.Source: §5.3 · pp. 20–21, Figure 7 · p. 21

5.4 Finding 3: The Production Readiness Cliff

  • Every platform needs substantial post-generation human effort, varying fivefold, and even the least effort-intensive remains short of production targets on most metrics.The range is 12 developer-hours and 0 re-prompts to 60 developer-hours and 8 re-prompts. CDI of 4–12% is associated with trustworthy observability artifacts, while CDI of 28–35% forces users to discover failures before fixing them.Source: §5.4 · pp. 21, 23, Figure 8 · p. 22, Figure 9 · p. 23

5.5 Finding 4: Widespread Security and Infrastructure Failures

  • No platform exceeds 65% Security Score against a 90% target, and concurrency handling ranges from 6% to 42% against a 70% target.Common failures include hard-coded frontend API keys, missing CSRF protection and rate limiting, public enumeration endpoints, and inconsistent JWT expiry policies.Source: §5.5 · p. 23, Table 13 · p. 23, Figure 10 · p. 24
  • v0-Max has the highest SWS (40.3%) but the lowest engineering score (25.1%), illustrating how frontend-only measures can mislead about overall application quality.The paper attributes the pattern to well-structured HTML and proper meta tags even when backend infrastructure is absent.Source: §5.5 · pp. 23–24

5.6 Preliminary AMR Analysis: Single-Platform Methodology Demonstration

  • In the author-affiliated QwikBuild-only AMR evaluation, modification scores were 1–14 percentage points below creation scores, with the largest drops in CBS (−24.3pp) and CRR (−14.0pp).ESR increased by +5.4pp, and P6 achieved the evaluation’s first AIA PASS (75%); the authors caution that these results demonstrate methodology and require independent cross-platform replication, not comparative ranking.Source: §5.6 · p. 24, Figure 12 · p. 27

5.6.1 AMR-Specific Diagnostic Metrics

  • QwikBuild’s AMR-specific diagnostics show P6 at 90% ACS, while all prompts had zero regressions and 100% CAR and PUR.New features were not all fully functional (P4: 12/15, P5: 7/9, P6: 8/9), suggesting the edit loop prioritizes preserving existing features; results were scored by affiliated authors and need independent replication.Source: §5.6.1 · pp. 24, 26, Table 15 · p. 25

5.6.2 Canary Survival Under Modification

  • All AMR canaries survive PRD and Plan stages, but 3 of 10 SURVIVING canaries show partial degradation versus 1 of 14 NEW canaries.Degradation occurs at Code and Deploy stages, indicating that evolving existing constraints through structural modification is harder than implementing fresh requirements.Source: §5.6.2 · pp. 26–27, Figure 11 · p. 26

6 Case Studies

The case studies show that requirement elicitation can surface or miss critical business needs, while polished interfaces and strong code do not ensure working, production-ready full-stack applications. Across platforms, the examples reveal distinct architectural blind spots rather than general competence.

Case 1: When Requirement Inference Succeeds and Fails (P1 ExamEdge)

  • A platform’s 15 PM questions elicit proactive domain needs, whereas v0-Max collapses four specified roles into one admin role and reaches FCS of only 28%.Missed requirements upstream can become expensive downstream refactoring tasks.Source: §6 · p. 28

Case 2: The Frontend-Backend Decoupling in Practice (P3 VettAI, Emergent)

  • Emergent’s polished onboarding and SEBI validation (FES: 68%) mask a <Select.Item /> crash that makes case management non-functional across its generated applications.The case contrasts the first 30 seconds of polished interaction with the first 3 minutes of broken use.Source: §6 · p. 28

Case 3: Universal Failure Modes (P3 VettAI, QwikBuild)

  • QwikBuild scores SWS 9.3% versus v0-Max’s 40.3% and Concurrency & Load 42%, with missing web standards and backend connection-pooling, race-condition, and load-testing gaps.Its External Service Reliability is 28.3% versus an 80% target, and its Code Hygiene is 52.8%, the lowest among the top four platforms.Source: §6 · p. 28

Case 4: No Platform Solves the Full Stack (Cross-Platform Blind Spots)

  • Replit achieves the highest FES (74.3%) and CIS (70%) but asks zero business-flow questions, while the cases show platform-specific blind spots rather than general competence.The study presents these patterns as priorities for each platform to address.Source: §6 · p. 28

7 Discussion

The discussion argues that evaluating vibe-coding platforms requires assessing the full software-agency workflow, not just code quality, while treating the benchmark’s findings as provisional. It highlights domain sensitivity and planning-execution feedback, and calls for broader, more independent evaluations.

7.1 Why Existing Benchmarks Cannot Capture This

  • Code-level benchmarks assess code patch quality, but miss requirement understanding, architecture, deployment configuration, and business readiness in generated applications.A platform could pass SWE-bench issues yet fail to build a coherent multi-role application from a business description.Source: §7.1 · p. 28

7.2 Closed-Loop Code Generation as an Architectural Pattern

  • Closed-loop planning and execution may resolve conflicts mid-build; PM elicitation alone accounts for approximately 10–15% of the quality gap.Preliminary AMR data from QwikBuild shows 100% Change Acknowledgment Rate and Plan Update Rate across three modification prompts, but independent cross-platform validation is needed before causal claims.Source: §7.2 · pp. 28–29

7.3 Why These Metrics Matter for the Vibe Coding Era

  • Canary Retention Rate, Claim Drift Index, Code Change Impact Score, and Adaptive Coherence Score matter because vibe-coding users trust outputs without inspecting them.SURVIVING canaries degrade at 3× the rate of NEW canaries; FeatBench found existing features broke in 73.6% of cases [7], and P4 scored 84% on ACS against an 85% target.Source: §7.3 · p. 29

7.5 Benchmark Governance and Maintenance Plan

  • The benchmark proposes versioned, community-contributed evaluations, quarterly re-evaluation, expanded prompts, and conflict-of-interest safeguards.Contributed prompts should include canary requirements and domain-expert validation; future evaluators should be unaffiliated with platforms or use blinded scoring for Tier 2 and Tier 3 metrics.Source: §7.5 · p. 30

7.6 Limitations and Bias Mitigation

  • Two authors’ QwikBuild affiliation and QwikBuild-only AMR evaluation create conflicts and asymmetry; the authors release protocols, report its failures, and recommend blinded evaluation.The non-affiliated author designed the canary requirements and AMR prompts; G1 uses a three-person panel including an external evaluator.Source: §7.6 · p. 30
  • The evaluation is limited by six prompts, one snapshot from February–March 2026, single evaluators for some metrics, and prompt sensitivity; AMR data covers only QwikBuild.The authors report no significance tests or confidence intervals and call for larger prompt suites, repeated evaluations, multiple independent evaluators, and extension of AMR evaluation across all six platforms.Source: §7.6 · p. 30
  • The benchmark may favor dedicated PM agents and T4/T5 applications, so its rankings are diagnostic rather than definitive.The authors recommend evaluating T1–T3 complexity levels and note that different benchmark design choices could produce different platform rankings.Source: §7.6 · pp. 30–31

8 Conclusion

SWE-WebDev Bench is a 68-metric framework for evaluating AI app-building platforms as virtual software agencies. An initial evaluation of six platforms across three domains identified recurring shortcomings in specification, engineering, production readiness, security, and infrastructure.

8 Conclusion

  • Across the evaluated platforms, four recurring shortcomings were specification bottlenecks, frontend-backend decoupling, a production-readiness cliff, and security and infrastructure failures.The evaluation covered six platforms across three business domains.Source: §8 · p. 31
  • IQS ranged from 20–70, and most platforms skipped requirement elicitation; polished UIs could mask absent backend infrastructure and poorly indicate engineering quality.Silently embedded, unverified assumptions were another consequence of skipping elicitation.Source: §8 · p. 31
  • No platform exceeded 60% engineering quality, and post-generation human effort ranged from 14.7–65.7 developer-hours.The paper concludes that generated applications require substantial engineering to ship.Source: §8 · p. 31
  • No platform exceeded 65% Security Score against a 90% target, and concurrency handling was critically weak across the field.Source: §8 · p. 31
  • In a limited sample (3 prompts per domain; 6 platforms), preliminary AMR results showed quality declines in 16 of 19 metrics.Surviving canary requirements had 3× the loss rate of new requirements; targeted modifications sometimes improved on creation quality, so the ACR/AMR boundary was not simply a quality cliff. The authors call for larger-scale replication.Source: §8 · p. 31
  • The authors release SWE-WebDev Bench to support independent evaluation and help builders address gaps in specification, backend infrastructure, security, maintainability, external-service reliability, and web-standards compliance.The released materials include the evaluation framework, scoring protocols, and prompt suite.Source: §8 · p. 31

Appendix · A Complete Per-Metric Scores: P1 ExamEdge

The appendix provides a complete per-metric score table for P1 ExamEdge Academy across five platforms.

A Complete Per-Metric Scores: P1 ExamEdge

  • Table 16 covers all primary metrics for P1 ExamEdge Academy across five platforms.The supplied passage identifies the table’s scope but does not provide its score values.Source: Table 16 · p. 34

B PM Agent Behavioral Analysis

This section summarizes PM Agent behavior by platform and defines CTC as turns to convergence.

B PM Agent Behavioral Analysis

  • Table 17 summarizes PM Agent behavior per platform and defines CTC as turns to convergence.Source: Table 17 · p. 35

C Metric Independence Analysis

Across 15 primary metrics and 18 platform-prompt data points, correlation analysis found both substantial overlap and near-independent quality axes, while metric groups were only modestly separable.

Metric Independence Analysis

  • 0.48 within-group versus 0.44 between-group average |τ| (1.10× ratio) suggests the metric groups organize results but do not capture fully independent latent factors.Table 18 summarizes pairwise correlations across 15 primary metrics and 18 data points.Source: Table 18 · p. 35, C Metric Independence Analysis · p. 35
  • Twelve pairs were strongly correlated (|τ| > 0.70, all p < 0.001), led by CIS–ESR (τ = 0.88), BLS–CLS (τ = 0.86), and CLS–FCS (τ = 0.83).These overlaps are expected because strong backend performance raises integration and concurrency scores; both metrics are retained for distinct diagnostic purposes.Source: C Metric Independence Analysis · p. 35
  • CHS was nearly uncorrelated (|τ| < 0.15) with BLS, CLS, CRR, CIS, and ESR, identifying code maintainability as an independent quality axis.Source: C Metric Independence Analysis · p. 35
  • FES showed weak correlation with backend metrics (|τ| < 0.30 with FCS, CBS, and CRR), supporting the frontend-backend decoupling finding.Source: C Metric Independence Analysis · p. 35
  • SWS inversely correlated with FCS (τ = −0.60), CLS (τ = −0.56), and SS (τ = −0.54), indicating frontend-prioritized platforms score higher on web standards despite lower engineering quality.Source: C Metric Independence Analysis · p. 35

Related papers

Loading 2605.04637v1…