PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies

Siddhant Saxena, Nilesh Trivedi, Vinayaka Jyothi

arXiv:2605.04637v1cs.MAcs.SE

TL;DR

AI app-building platforms claim to deliver production-ready applications, but existing evaluations do not assess that full capability. SWE-WebDev Bench evaluates them across business, engineering, and operational dimensions, finding recurring specification, production-readiness, security, and modification weaknesses.

  • Problem

    Whether AI-generated software is production-ready remains largely unanswered, while existing evaluations do not assess complete application creation from scratch for non-technical users.

  • Method

    SWE-WebDev Bench evaluates AI app-building platforms as virtual software agencies using broad engineering metrics and complex business scenarios testing requirement inference and implementation fidelity.

  • Results

    Across the evaluated platforms, recurring shortcomings include specification bottlenecks, production-readiness failures, security weaknesses, and modification-related quality degradation; no platform exceeds 60% engineering quality.

  • Takeaways & Limitations

    The benchmark provides a diagnostic basis for identifying weaknesses in AI app-building platforms and supports larger-scale replication of these findings.

  • Takeaways & Limitations

    Benchmark design may favor platforms with dedicated PM agents and complex-application specializations, while omitting evaluation at T1–T3 complexity levels.

Abstract

from arXiv · show

The emergence of "vibe coding" platforms, where users describe applications in natural language and AI agents autonomously generate full-stack software, has created a need for rigorous evaluation beyond code-level benchmarks. In order to assess them as virtual software development agencies on understanding business requirements, making architectural decisions, writing production code, handling iterative modifications, and maintaining business readiness, we introduce SWE-WebDev Bench, a 68-metric evaluation framework spanning 25 primary and 43 diagnostic metrics across seven groups, organized along three dimensions: Interaction Mode (App Creation Request (ACR) vs. App Modification Request (AMR)), Agency Angle (Product Manager (PM), Engineering, Ops), and Complexity Tier (T4 multi-role SaaS, T5 AI-native). Our evaluation (six platforms, three domains, 18 evaluation cells) reveals four recurring shortcomings in the current generation of AI app builders: (1) A specification bottleneck, where platforms compress rich business requirements into oversimplified technical plans, (2) A pervasive frontend-backend decoupling, where visually polished UIs mask absent or broken backend infrastructure, (3) A steep production-readiness cliff, where no platform scores above 60% on engineering quality and post-generation human effort varies substantially across platforms and (4) Widespread security and infrastructure failures, with no platform exceeding 65% Security Score against a 90% target and concurrency handling as low as 6%. These observations are descriptive of our sample and require larger-scale replication to establish generality. We release SWE-WebDev Bench as a community benchmark to enable such replication and help platform builders identify and address these gaps. Code and benchmark resources are available at: https://github.com/snowmountainAi/webdevbench and https://webdevbench.com/.

1 Introduction

AI application-building platforms generate deployed full-stack applications from natural-language descriptions, but existing evaluations do not determine whether they function as complete, production-ready software agencies. SWE-WebDev Bench addresses this gap with a multidimensional framework, separate creation and modification testing, and resources for replication.

  • Motivation: Natural-language AI application-building platforms promise deployed full-stack software, yet the production readiness of their output remains largely unanswered.These platforms claim to compress months of development into minutes and make software creation accessible to non-developers.
  • Evaluation gap: Existing code-, issue-, and application-level benchmarks do not assess the full agency pipeline from business requirements through iterative modification, security, infrastructure, and business readiness.They also do not test whether an AI system can build a complete application from scratch for a non-technical user.
  • Benchmark design: SWE-WebDev Bench introduces a 68-metric framework organized across Mode × Angle × Tier with a four-tier judging taxonomy to diagnose production-readiness failures.Its dimensions cover interaction mode, agency angle, and complexity tier.
  • Benchmark design: The benchmark separately evaluates App Creation Requests and App Modification Requests, treating modification handling as a fundamentally different and harder competency.This ACR/AMR distinction is presented as a first for benchmarking these capabilities separately.
  • Benchmark design: SWE-WebDev Bench includes 80 culturally specific, domain-embedded Canary Requirements spanning Original, New, Surviving, and Contradiction types to distinguish comprehension from template matching.The methodology is designed to test whether systems understand requirements embedded in business contexts.
  • Contributions: The authors evaluate six platforms across three business domains, identify four recurring shortcomings, and release prompts, rubrics, and protocols for independent replication.Benchmark resources are available through the project’s GitHub repository and website.

2 Related Work

Existing coding benchmarks progress from code-level tasks toward whole-application evaluation, but generally omit business requirements, iterative modification, deployment, and production readiness. SWE-WebDev Bench addresses these gaps by evaluating the full pipeline from business intent through deployment readiness and diagnosing why applications fail.

  • Code-level benchmarks: HumanEval, MBPP, and ClassEval broaden code-generation evaluation from function-level to class-level tasks but require code-level specifications incompatible with natural-language-only vibe coding.These benchmarks assume inputs such as function signatures or docstrings rather than user intent expressed in natural language.
  • Repository-level benchmarks: SWE-bench, SWE-bench-Live, and FeatBench evaluate realistic issue resolution or feature implementation, but focus on patch quality in existing codebases rather than complete application delivery.FeatBench found that 73.6% of failures stem from regressive implementation, motivating SWE-WebDev Bench’s AMR dimension.
  • Whole-application benchmarks: Whole-application benchmarks improve realism but remain heterogeneous: Vibe Code Bench’s best model reached 61.8% accuracy, while WebGen-Bench’s best agent reached only 27.8%.Other efforts evaluate user preferences, human-centered comparisons, isolated full-stack problems, or security, rather than one coherent business-to-deployment pipeline.
  • SWE-WebDev Bench: SWE-WebDev Bench complements prior whole-application evaluations by measuring requirement elicitation, iterative modification handling, and the pipeline from business intent through deployment readiness.Where Vibe Code Bench measures whether an application works, SWE-WebDev Bench measures why it fails and what to fix.
  • SWE-WebDev Bench: Its PM Agent dimension measures specification bottleneck losses as platforms compress ambiguous business requirements into oversimplified technical plans.This evaluation targets lost domain context during translation from business requirements to technical plans.

3 The SWE-WebDev Bench Framework

SWE-WebDev Bench evaluates AI app-building platforms as full software agencies rather than code generators, measuring delivery across interaction mode, agency role, and application complexity. Its 68-metric framework combines primary scores with diagnostics to expose specification, engineering, operational, and iterative-change failures.

  • Evaluation principles: The benchmark evaluates the full delivery pipeline across product management, engineering, and operations, including requirement interpretation, secure implementation, deployment, and maintenance.This extends beyond existing benchmarks that focus primarily on code patches or function implementations.
  • Evaluation cube: Three orthogonal dimensions define the evaluation cube: Interaction Mode (ACR vs. AMR), Agency Angle (PM × Engineering × Ops), and Complexity Tier (T4 vs. T5).ACR covers new application creation, AMR covers modification with functionality preservation; T4 targets multi-role SaaS and T5 AI-native multitenant applications.
  • Agency-angle diagnosis: 3.5× widest variance appears in PM Inference Quality Score, while Frontend Engineering shows only a 6-point platform spread, identifying PM capability as the stronger differentiator in these results.The decomposition enables failures to be attributed to requirement understanding, engineering, or operations.
  • Metric architecture: 68 metrics comprise 25 primary metrics across 7 groups and 43 diagnostic metrics across 4 categories, covering software delivery from specification through production maintenance.Diagnostic metrics are paired with primary metrics to trace causes such as poor requirement capture, hallucinated features, or implementation failures.
  • Metric groups: G1 Specification Fidelity measures inferred business intent, feature coverage, and culturally specific requirements, while G5 Changeability measures regression impact and developer effort during iterative modifications.Canary Retention Rate uses requirements such as DD/MM/YYYY dates, INR currency, and JEE/NEET conventions; Code Change Impact Score addresses modification regressions affecting 73.6% of coding-agent modifications.

4 Experimental Design

The evaluation tests six AI app-building platforms across six standardized prompts, three business domains, and both app creation and modification requests. Its design combines varied prompt styles, domain-specific probes, canary requirements, and a seven-phase audit to measure requirement comprehension, implementation fidelity, modification handling, and diagnostic performance.

  • Benchmark scope: Six platforms are evaluated using six standardized prompts: three App Creation Requests and three App Modification Requests.The prompts align with Environment Complexity, Autonomy Horizon, and Output Complexity, with each prompt stressing at least two dimensions.
  • Domain coverage: Three orthogonal domains probe inference depth, execution precision, and AI trustworthiness rather than a single architectural strength.EdTech uses vague culturally specific requirements, Field Service uses detailed enterprise logic, and FinTech-AI tests safe, auditable AI features; domain-specific performance swings reach 13 percentage points.
  • Modification handling: The three AMR prompts vary complexity, prompt style, and change type to test additive changes, inferred modifications, and provider-swap refactoring.Examples include retrofitting multi-tenancy, inferring features from a frustrated-user request, and replacing Anthropic with Gemini fallback while adding PDF reports.
  • Requirement comprehension: 80 culturally specific, domain-embedded canary requirements test whether platforms retain detailed user intent beyond generic SaaS template matching.The Surviving canary type directly tests ACR-to-AMR transitions, including evolution of a cross-branch leaderboard into an institute-scoped leaderboard.
  • Evaluation protocol: Each platform×prompt evaluation follows a seven-phase, 14-day protocol covering build execution, code and security audits, feature and canary tests, AMR injection, human scoring, cost aggregation, and diagnostics.The protocol records interaction and build traces, applies automated and manual tests, scores expert-panel metrics, aggregates billing and human effort, and analyzes transcripts and logs.

5 Results · 5.1 Cross-Platform Engineering Scores · 5.2 Finding 1: The Specification Bottleneck

Across six platforms and 18 ACR evaluation cells, SWE-WebDev Bench finds that production readiness remains unresolved and identifies a specification bottleneck in which inadequate requirement elicitation and validation produce downstream quality gaps. The observed results are descriptive of this sample and do not support statistical significance claims.

  • 5.1 Cross-Platform Engineering Scores: No platform exceeds 60% engineering score, and every platform has at least one metric below 15%.The evaluation comprises n = 3 prompts per domain across 6 platforms, yielding 18 ACR evaluation cells; no significance tests or confidence intervals were computed.
  • 5.1.1 Comprehensive Per-Metric Breakdown: The highest-scoring platform still fails 17 of 22 metrics against their respective targets.No platform passes all metrics, leaving production-grade AI app building an open problem.
  • 5.1.1 Comprehensive Per-Metric Breakdown: Frontend Engineering ranges from 68–74% across four platforms, while CBS ranges from 0% to 49%, exposing a frontend-backend infrastructure gap.The frontend scores fall within 6 percentage points, whereas backend and infrastructure metrics show 5–10× spreads.
  • 5.1.1 Comprehensive Per-Metric Breakdown: Canary Retention Rate ranges from 17.7% to 97.7%, a 5.5× spread indicating substantial variation in specification fidelity.Culturally specific, domain-embedded requirements such as date formats, currency conventions, and localization details are silently dropped by most platforms.
  • 5.1.2 Per-Prompt Results: No platform dominates across all domains, and Replit swings from 54.5% on P3 VettAI to 41.5% on P1 ExamEdge.This 13-point swing illustrates domain-sensitive performance and platform-specific strengths and weaknesses.
  • 5.2.1 PM Agent Interaction Traces: QwikBuild asks 15 business workflow questions, Emergent asks 5 infrastructure questions, and Replit and v0-Max ask zero business-flow questions on P1 ExamEdge.These excerpts come from four platforms receiving the same P1 ExamEdge prompt.
  • 5.2.2 Contradiction Handling: The P2 Trap: All six platforms detect the marked P2 contradiction, but weaker handling can flag it without resolution and defer core features to “future scope.”Deferred features include the SLA engine, audit trail, invoicing, time tracking, and customer signatures, producing an incomplete Field Service application.

5.3 Finding 2: The Frontend-Backend Decoupling Problem … 5.6 Preliminary AMR Analysis: Single-Platform Methodology Demonstration

The findings show that polished frontend output often masks weak backend infrastructure, substantial post-generation work, and serious security and concurrency gaps. Preliminary AMR results from QwikBuild indicate that modifications preserve planning-stage requirements but degrade during coding and deployment, while cross-platform AMR conclusions remain unavailable.

  • 5.3 Finding 2: The Frontend-Backend Decoupling Problem: Frontend quality is a poor predictor of backend quality, making background jobs, external integrations, and complex data pipelines major open challenges.The data suggest three architectural strategies with different production-readiness implications.
  • 5.3 Finding 2: The Frontend-Backend Decoupling Problem: At similar frontend quality, platforms diverge sharply in backend capability, with FES ≈68% spanning CBS 0–49%.QwikBuild exemplifies the infrastructure-integrated strategy, reaching FES 68%, CBS 49.3%, and CIS 67.3%, yet still falling short of backend targets.
  • 5.4 Finding 3: The Production Readiness Cliff: 12 developer-hours and 0 re-prompts is the lowest observed post-generation effort, versus 60 developer-hours and 8 re-prompts for the most effort-intensive platform.Even the least effort-intensive platform remains below production targets on most metrics, and effort varies 5× across platforms.
  • 5.4 Finding 3: The Production Readiness Cliff: Higher resource consumption consistently correlates with higher quality, while achieving both high quality and low post-generation effort remains unsolved.The paper suggests that upstream requirement elicitation and build-time feedback loops may be more cost-effective than post-generation debugging.
  • 5.5 Finding 4: Widespread Security and Infrastructure Failures: No platform exceeds 65% Security Score against a 90% target, and concurrency handling ranges from 6% to 42% against a 70% target.Common failures include exposed frontend API keys, missing CSRF protection, absent rate limiting, public enumeration endpoints, and inconsistent JWT expiry policies.
  • 5.5 Finding 4: Widespread Security and Infrastructure Failures: v0-Max achieves the highest SWS at 40.3% while recording the lowest engineering score at 25.1%, because frontend structure can persist without backend infrastructure.Frontend-only metrics can therefore mislead assessments of overall application quality.
  • 5.6.1 AMR-Specific Diagnostic Metrics: QwikBuild’s preliminary AMR evaluation reports ACS 84% for P4 and ACS 90% for P6, with zero regressions across all AMR prompts.P4 loses change correctness through uncalibrated AI insight confidence intervals and incomplete royalty calculations, whereas P6 benefits from an isolated provider abstraction.
  • 5.6.2 Canary Survival Under Modification: 100% of canaries survive PRD and Plan stages, but degradation occurs at Code and Deploy stages; 3 of 10 SURVIVING canaries partially degrade versus 1 of 14 NEW canaries.The SURVIVING canary type therefore shows a 3× higher partial-loss rate, indicating greater difficulty evolving existing constraints than implementing fresh requirements.

6 Case Studies

Case 1 shows that requirement inference can either uncover domain-specific needs and enable proactive features or collapse a specified multi-role hierarchy into a single role, illustrating the specification bottleneck.

  • Case 1: When Requirement Inference Succeeds and Fails: 15 PM questions elicited domain-specific needs absent from the prompt, such as conflict detection for batch scheduling across branches, producing applications with proactive features.
  • Case 1: When Requirement Inference Succeeds and Fails: FCS of only 2 resulted when v0-Max collapsed the specified superadmin/teacher/student/parent hierarchy into a single “admin” role.

7 Discussion

The discussion argues that evaluating vibe coding platforms requires multidimensional, agency-oriented benchmarks rather than code-patch measures alone. It highlights planning–execution feedback loops, domain-specific failure modes, community governance, and important conflicts and architectural biases in the benchmark design.

  • Benchmark scope: Code-level benchmarks suffer dimensional collapse because vibe coding platforms must handle requirements, architecture, code, deployment, and business readiness.A platform may solve code patches yet fail to build a coherent multi-role application.
  • Closed-loop development: Planning–execution feedback loops distinguish stronger platforms by surfacing and resolving conflicts during builds rather than embedding misunderstandings in one-pass specifications and code.PM elicitation alone accounts for approximately 10–15% of the observed gap; the broader advantage comes from interaction between planning and execution.
  • Trust model: Because users delegate entire builds and trust the output, agency-oriented metrics such as Canary Retention Rate become essential for detecting subtle unmet requirements.Unlike users of Copilot or Cursor, vibe coding users may not review every generated line or understand the resulting codebase.
  • Domain-specific evaluation: The three-domain design targets distinct failure modes: inference depth in EdTech, execution precision in Field Service, and AI trustworthiness in FinTech-AI.The domains respectively test Indian coaching conventions, lifecycle and GST/SLA execution, and safe, auditable AI that acknowledges uncertainty.
  • Governance: SWE-WebDev Bench is intended as a living, community-contributed benchmark with versioned evaluations, and results from different periods are not directly comparable.Evaluations record platform versions and dates, while researchers can use the released prompts, rubrics, and scoring protocol.
  • Limitations: The authors disclose conflicts of interest and design biases that may favor dedicated PM agents and platforms optimized for complex T4/T5 applications.Two of three authors are affiliated with QwikBuild, and the AMR evaluation is QwikBuild-only; simpler-use-case strengths may be disadvantaged.

8 Conclusion

SWE-WebDev Bench evaluates AI app-building platforms as virtual software agencies and identifies recurring specification, engineering, production-readiness, security, and infrastructure shortcomings. The authors frame these findings as sample-descriptive, report preliminary modification degradation, and release the framework as a community benchmark.

  • Benchmark contribution: 68 metrics evaluate AI app-building platforms as virtual software agencies, with an initial study spanning six platforms across three business domains.The framework and evaluation are introduced as a basis for assessing these platforms beyond code generation alone.
  • Recurring shortcomings: IQS ranged from 20–70, while most platforms skipped requirement elicitation and embedded unverified assumptions into generated applications.This specification bottleneck indicates substantial variation in inference quality within the evaluated sample.
  • Recurring shortcomings: Polished UIs masked absent backend infrastructure, making visual quality a poor proxy for engineering quality and creating false confidence for non-technical users.The conclusion identifies frontend-backend decoupling as a recurring pattern across the evaluated platforms.
  • Production and infrastructure readiness: No platform exceeded 60% on engineering quality, while ETF ranged from 14.7–65.7 developer-hours, showing that generated applications require substantial engineering to ship.No platform exceeded 65% Security Score against a 90% target, and concurrency handling was critically weak across the field.
  • Limitations and release: The observations are descriptive of a sample with n = 3 prompts per domain and n = 6 platforms, requiring larger-scale replication to establish generality.The authors release the framework, scoring protocols, prompt suite, and supporting resources as a community benchmark for independent evaluation and platform improvement.
  • Modification performance: 16 of 19 metrics declined from ACR to AMR, and SURVIVING canary requirements showed 3× the loss rate of new requirements.These preliminary AMR results indicate that modification systematically degrades quality, although targeted modifications can sometimes improve creation quality.

Appendix · A Complete Per-Metric Scores: P1 ExamEdge

Table 16 reports all primary metrics for P1 ExamEdge Academy across five platforms.

  • A Complete Per-Metric Scores: P1 ExamEdge: Table 16 covers all primary metrics for P1 ExamEdge Academy.The supplied passage does not provide the individual metric names or scores.
  • A Complete Per-Metric Scores: P1 ExamEdge: The comparison spans five platforms.No platform identities or metric values are included in the supplied passage.

B PM Agent Behavioral Analysis

This section summarizes PM Agent behavior for each platform and defines CTC as Turns to Convergence.

  • B PM Agent Behavioral Analysis: Table 17 summarizes PM Agent behavior per platform.
  • B PM Agent Behavioral Analysis: CTC denotes Turns to Convergence.

C Metric Independence Analysis

The analysis tests whether 15 primary metrics capture distinct constructs using Kendall’s τ across 18 platform–prompt data points. Most metrics are not fully independent: 12 pairs are strongly correlated, while Code Hygiene is nearly uncorrelated with several metrics.

  • Method: Kendall’s τ rank correlations assess metric independence across 15 primary metrics and 18 data points from six platforms and three prompts.The analysis examines 105 metric pairs.
  • Correlation Findings: 12 metric pairs show strong correlation (|τ| > 0.70, all p < 0.001).The strongest pairs are CIS–ESR (τ = 0.88), BLS–CLS (τ = 0.86), and CLS–FCS (τ = 0.83).
  • Correlation Findings: CHS is nearly uncorrelated with BLS, CLS, CRR, CIS, and ESR (|τ| < 0.15), supporting its distinct measurement behavior.The analysis retains strongly correlated metrics because they serve distinct diagnostic purposes.
Loading 2605.04637v1…