PaperlayerView exact revision on arXiv

Source-linked AI summary

Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Development

Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, Alex Gu

arXiv:2603.04601v3cs.SEcs.AIcs.CL

TL;DR

Existing code-generation benchmarks largely measure constrained tasks rather than building a complete web application from scratch. Vibe Code Bench evaluates that process across application specifications and browser workflows; across 16 frontier models, the best reaches 61.8% accuracy, while evaluator choice remains a reporting concern.

  • Problem

    Existing benchmarks assess constrained code-generation tasks, but no widely used benchmark measures complete, deployable applications from natural-language specifications.

    Source: §1 · p. 1

  • Method

    Vibe Code Bench evaluates applications built from 100 natural-language specifications using 964 browser workflows scored by an autonomous browser agent.

    Source: §1 · p. 1

  • Results

    Across 16 frontier models, end-to-end application development remains unreliable, with the best model reaching 61.8% accuracy.

    Source: §1 · p. 1, §8 · p. 9

  • Takeaways & Limitations

    Evaluator choice is a first-class reporting concern because automated evaluation is reliable when the right evaluator is used.

    Source: §8 · p. 9

    The evaluation focuses on React/Vite applications with a Supabase backend, and results may not generalize to other frameworks or backend technologies.

    Source: §7 · p. 9

Abstract

from arXiv · show

Code generation has emerged as one of AI's highest-impact use cases, yet existing benchmarks measure isolated tasks rather than the complete "zero-to-one" process of building a working application from scratch. We introduce Vibe Code Bench, a benchmark of 100 web application specifications (50 private validation, 50 held-out test) with 964 browser-based workflows comprising 10,131 substeps, evaluated against deployed applications by an autonomous browser agent. Across 16 frontier models, the best achieves 61.8% accuracy on the test split, revealing that reliable end-to-end application development remains a frontier challenge. We identify self-testing during generation as a strong performance predictor (Pearson r=0.72), and show through a completed human alignment study that evaluator selection materially affects outcomes (31.8-93.6% pairwise step-level agreement). Our contributions include (1) a novel benchmark dataset and browser-based evaluation pipeline for end-to-end web application development, (2) a comprehensive evaluation of 16 frontier models with cost, latency, and error analysis, and (3) an evaluator alignment protocol with both cross-model and human annotation results.

1 Introduction

Existing benchmarks do not widely measure whether models can build complete, deployable web applications from natural-language specifications. Vibe Code Bench evaluates this zero-to-one task and finds that end-to-end application development remains unreliable.

  • The paper frames complete application creation as a potential way for non-technical users to build software and shorten the path from specification to prototype.Source: §1 · p. 1
  • No widely used benchmark measures generating a complete, deployable web application from a natural-language specification, including code, configuration, deployment, and user-facing workflows.Source: §1 · p. 1
  • Vibe Code Bench contains 100 application specifications and evaluates generated applications with 964 browser workflows comprising 10,131 substeps.Tasks are split into private validation and test sets, with 50 tasks in each; a browser agent scores workflow success by substep completion.Source: §1 · p. 1
  • Across 16 frontier models, the top model, GPT-5.3-Codex, passes 61.8% of workflows, and end-to-end application development remains unreliable.Source: §1 · p. 1
  • The benchmark evaluates end-to-end workflow success with an autonomous browser agent, producing fine-grained pass-fail signals.Source: §1 · p. 2
  • The paper evaluates 16 frontier models across providers, including cost, latency, error analyses, and evaluator alignment with cross-model and human studies.Source: §1 · p. 2

2 Related Work

Existing code-generation benchmarks cover isolated functions, repository issue resolution, and scoped tasks, but not the full process of creating a deployable application from scratch. Vibe Code Bench addresses that zero-to-one gap.

  • Function-generation benchmarks test isolated algorithmic reasoning, while SWE-Bench variants assess issue resolution in existing codebases and SWE-Lancer evaluates scoped freelance modifications.Source: §2 · p. 2
  • Existing benchmarks range from function synthesis to repository-level work, but none evaluate the full zero-to-one process of building complete, deployable applications from natural-language specifications.Source: §2 · p. 2
  • WebArena and VisualWebArena test agents navigating existing applications; Vibe Code Bench instead applies browser automation to applications generated by models.Source: §2 · pp. 2–3
  • Agentic coding systems support extended coding sessions, while commercial systems demonstrate that multi-hour development is feasible; systematic evaluation of end-to-end application development has been limited.Source: §2 · p. 3

3 Benchmark Design

Vibe Code Bench is designed to evaluate realistic, reproducible, implementation-agnostic development of complete applications. Its data, generation harness, browser-agent evaluation, and metrics are structured to support that goal.

3.1 Data Construction

  • The benchmark contains 100 disjoint application specifications—50 validation and 50 test—spanning Individual, Solo Founder, and Enterprise Tool domains.Specifications are concise and written from a non-technical perspective; tasks and workflows were drafted, reviewed, and audited before freeze.Source: §3.1 · p. 3
  • 28% of applications require email and/or Stripe integrations, adding workflows involving asynchronous events, delivery verification, and cross-service state consistency.All services run in sandbox/test mode as isolated containers.Source: §3.1 · p. 3

3.2 Generation Harness

  • Models build applications in isolated OpenHands-based environments with terminal, browser, and service access, under a shared five-hour-per-application wall-clock budget.The harness mandates a consistent stack and Docker Compose to support consistent evaluation and portable submissions.Source: §3.2 · p. 4

3.3 Automated Evaluation Pipeline

  • A browser agent executes pre-authored natural-language workflows in fresh sessions and produces structured substep judgments for workflow scoring.Apps first undergo deployment verification; failures at deployment or missing structured output mark workflows failed.Source: §3.3 · pp. 4–5

3.4 Metrics

  • Overall test-split metrics average per-application results; reported measures include workflow accuracy, generation cost, and end-to-end generation latency.Latency excludes evaluation time.Source: §3.4 · p. 5
  • A workflow passes when at least 90% of its substeps succeed, and workflow-level scoring prevents successes in unrelated workflows from masking a broken workflow.Evaluation uses user-visible browser interactions rather than implementation-specific assertions.Source: §3.3 · p. 4, §3.4 · p. 5

4 Experiments and Results

Across the 16 evaluated models, GPT-5.3-Codex ranks first, while leading closed-weight models dominate the top of the test-set rankings. Accuracy generally trades off against cost and latency, and performance varies with task difficulty, integrations, and reasoning or vision settings.

4.2 Cost and Latency

  • Higher cost and longer runtime are generally associated with higher accuracy, but Claude Opus 4.6 reaches near-top accuracy at lower cost and latency than GPT-5.3-Codex.GLM-5 is an outlier because its long trajectories raise both cost and latency; both accuracy frontiers show diminishing returns.Source: Figure 3 · p. 6, §4.2 · p. 6

4.1 Overall Benchmark Performance

  • GPT-5.3-Codex ranks first on the test set, with Anthropic and OpenAI models dominating the top rankings; open-weight models named in the results trail leading closed-weight counterparts.The table reports test-split model performance and marks open-weight models.Source: §4.1 · p. 5, Table 4 · p. 6

4.3 Difficulty, Integrations, and Domains

  • Performance drops sharply on hard applications; five open-weight models score at or below 2.50%, and only the very top models solve the hardest problems.The difficulty table reports accuracy by tier.Source: §4.3 · p. 6, Table 5 · p. 6
  • For apps requiring both email and Stripe, GPT-5.3-Codex falls from 71.25% to 29.58%, the all-model average from 34.18% to 13.49%, and all five open-weight models remain below 8%.Performance also generally declines from Individual to Solo Founder to Enterprise Tool apps; five open-weight models fall below 13% on Enterprise Tool apps.Source: §4.3 · p. 6, Table 6 · p. 6, Table 7 · p. 7
  • GPT and Gemini 3.1 Pro improve as reasoning effort increases, whereas Opus 4.6 is unaffected because adaptive thinking can modify the suggested effort level.The ablation covers low, medium, and maximum available reasoning effort across 20 applications.Source: §4.3 · pp. 6–7, Figure 4 · p. 7
  • Disabling vision reduces accuracy for GPT-5.3-Codex and Gemini 3.1 Pro, has little effect on Opus 4.6 Thinking in this run, and leaves all three ahead of open-weight counterparts.The comparison is a vision ablation on a 20-app subset.Source: §4.3 · p. 7, Figure 5 · p. 7

5 Analysis

Across tool-use, error, and variance analyses, browser-based self-testing is associated with stronger outcomes, while failures commonly reflect missing product capabilities. Evaluator reruns are more consistent than generation outcomes in the tested panel.

5.1 Tool Use and Model Behavior

  • About 95% of tool calls go to bash, file editing, browser use, SQL, and migrations, which map to the benchmark’s core build-and-verify loop.Models generally prefer raw SQL to migration helpers; Tavily search is used by only 18.8% of models.Source: §5.1 · p. 7
  • Browser calls per application correlate with accuracy at Pearson r=0.72, and the association remains r=0.72 after controlling for generation latency.Lower-performing models also tend to terminate earlier, spend less time testing, and show narrower action-phase coverage.Source: §5.1 · p. 7
  • GPT-5.2 makes 26.4 browser calls per app versus 9.4 for GPT-5.2-Codex, while its zero-scoring apps are fewer (9 vs. 14) and perfect apps more numerous (6 vs. 1).Edit calls correlate weakly with accuracy (r=0.09), whereas median browser calls correlate more strongly (r=0.67).Source: §5.1 · p. 7

5.2 Error Analysis

  • Model improvement is driven primarily by reducing zero-scoring apps rather than incremental gains across all apps, and score distributions are often strongly bimodal.Figure 6 shows aggregate trajectory action composition across tasks.Source: §5.2 · p. 7, Figure 6 · p. 8
  • 46.7% of behavioral failures are missing features; authorization issues account for 32.3% for Grok 4.1 Fast Reasoning and 30.3% for GPT-5.2-Codex.The error breakdown indicates that most failures are capability gaps in implemented product surfaces rather than startup failures.Source: §5.2 · p. 8

5.3 Variance Analysis

  • The variance analysis compares repeated generations and evaluator reruns on a six-specification panel using three source models.It used five independent generations per model and app, then five evaluator runs per generated app.Source: §5.3 · p. 8
  • 92.5% of Gemini 3.1 Pro variance, 92.1% of GPT-5.2 variance, and 73.5% of Claude Sonnet 4.6 variance comes from generation rather than rescoring.The authors conclude that major ranking patterns are robust, while near-tie differences warrant caution.Source: §5.3 · p. 8

6 Human Alignment

The study compares browser-based human and model evaluations using pairwise pass/fail agreement on shared workflow substeps. Agreement varies substantially across evaluator pairs, so evaluator choice affects reported alignment outcomes.

6.1 Methodology

  • Each of 18 generated apps was evaluated by two human reviewers and four model evaluators, with scores compared as pass/fail matches on shared substeps.The apps came from six tasks and three source models; no adjudication was performed between evaluators.Source: §6.1 · pp. 8–9, §6.2 · p. 9

6.2 Results

  • Human-human agreement is 88.6–93.6%, indicating relatively consistent manual judgments in this study.The authors treat human labels as a stable external reference for evaluator quality.Source: §6.2 · p. 9
  • Claude Sonnet 4.6 and Claude Sonnet 4.5 align at 87.8%, while GPT-5.2 aligns only 33.1–38.8% with the other model evaluators.The pairwise agreement matrix covers model and human evaluators.Source: §6.2 · p. 9, Table 10 · p. 9
  • Pooled model-human alignment is 86.4% for Claude Sonnet 4.5, 86.3% for Claude Sonnet 4.6, 84.7% for Gemini 3.1 Pro, and 36.1% for GPT-5.2.In this panel, Claude Sonnet 4.5 has the strongest human alignment, with the other two higher-aligning models close behind.Source: §6.2 · p. 9, Table 10 · p. 9
  • GPT-5.2 costs $0.093 per workflow versus $1.48 for Sonnet 4.6, but the study prioritized alignment quality and chose Sonnet 4.5.The authors conclude evaluator choice materially affects alignment outcomes.Source: §6.2 · p. 9

7 Limitations

The benchmark has four stated limitations: it does not measure code quality, uses one generation harness and technology stack, and evaluates web applications only.

7 Limitations

  • Functional-test passes do not establish maintainability, security, or documentation quality, which the study did not evaluate.Code readability, visual design quality, and security vulnerabilities were also not evaluated.Source: §7 · p. 9
  • All results use one OpenHands-based harness, and performance may increase with a different harness.The authors identify the single harness as a limitation.Source: §7 · p. 9
  • The benchmark focuses on React/Vite applications with Supabase, so results may not generalize to other frameworks or backend technologies.Source: §7 · p. 9
  • Only browser-based applications are evaluated; zero-to-one development of command-line, desktop, and mobile software remains unexplored.Source: §7 · p. 9

8 Conclusion

Vibe Code Bench evaluates end-to-end web application development, where the best of 16 frontier models reaches 61.8% accuracy. The conclusion highlights complete failures, self-testing, and evaluator choice as important findings.

8 Conclusion

  • 61.8% accuracy is the best result across 16 frontier models, and reliable end-to-end application development remains unsolved.The benchmark contains 100 web application specifications evaluated through 964 browser-based workflows.Source: §8 · p. 9
  • Model improvement is driven primarily by reducing complete failures rather than incremental gains, while self-testing is strongly associated with performance (r=0.72).Source: §8 · p. 9
  • Evaluator choice matters: automated evaluation is reliable when the right evaluator is used, making evaluator choice a reporting concern.Source: §8 · p. 9
  • Vibe Code Bench tracks progress toward whether AI can build software, rather than merely write code.Source: §8 · p. 9

A Artifacts

The appendices point readers to benchmark artifacts, summarize test-split pass-rate distributions, and show a chronological trajectory for one application.

A Artifacts

  • The modified OpenHands harness, model trajectories, generated applications, and example specification, UI tests, and prompt are available as artifacts.The specification, UI tests, and prompt are in Appendices H, I, and G, respectively.Source: A Artifacts · p. 11

B Pass-Rate Histograms for All Models

  • Pass-rate distributions are often bimodal, especially for top models, and improvement primarily reflects fewer applications scoring zero.Source: Figure 7 · p. 11

C Illustrative Single-App Trajectories

D All-Model Breakdown Results

The appendix provides all-model breakdown tables for accuracy by difficulty, integration, and domain.

D All-Model Breakdown Results

E Tools Provided

The environment provides agents with shell, browser, file-editing, Python, and planning tools, alongside Supabase and Tavily integrations.

OpenHands tools

  • OpenHands supplies persistent shell execution, browser interaction, file editing, Python execution, reasoning logs, task tracking, and task completion.The browser tool supports 15 actions, including navigation, clicking, form entry, and uploads.Source: E Tools Provided · p. 14

Supabase Studio MCP tools

  • Supabase Studio tools cover SQL and migrations, schema and project inspection, logs and advisories, generated TypeScript types, and documentation search.Source: E Tools Provided · p. 14

Tavily MCP tools

  • Tavily tools provide web search, URL extraction, site crawling, and structured site mapping with configurable filters or depth.Source: E Tools Provided · p. 14

F Generation and Evaluation Hyper-Parameters

This section describes the generation and evaluation configurations, including the system prompt’s requirements for building, testing, and deploying applications. The prompt specifies technical, backend, and UI/UX constraints, requires testing the running app through its UI, and prohibits mocked implementations of required external services.

F Generation and Evaluation Hyper-Parameters

  • Generation and fixed-evaluator hyper-parameters are documented separately; all generation runs shared a retry policy, while provider settings were chosen per model to maximize performance.The generation policy allowed up to 20 retries with 15–120-second backoff; Table 15 covers all 16 models, and Table 16 covers the fixed evaluator used in reported runs.Source: F Generation and Evaluation… · p. 15, Table 15 · p. 15, Table 16 · p. 15

G Generation System Prompt

  • The generation prompt required a tested, working desktop-browser app, prescribed React with Vite, and directed models to use Supabase for core backend services.It prohibited relying on user actions during or after generation and required testing the application, including building and starting the Docker Compose app.Source: G Generation System Prompt · pp. 16–17
  • The prompt emphasized a consistent, responsive design system with reusable components, semantic color tokens, interaction feedback, and loading states.It instructed models to avoid direct color styles, use design-system tokens, and provide clear feedback for interactions and loading.Source: G Generation System Prompt · pp. 16–17
  • Deployment instructions required a Docker Compose-runnable app using the provided environment and public URL/port settings, with a backend optional when unnecessary.The prompt also prohibited overwriting the supplied .env file and required building and previewing the app rather than using a development server.Source: G Generation System Prompt · pp. 18–19
  • Deployment instructions require environment-based public ports, Vite listening on all interfaces, and internal-only backend exposure rather than publishing its port.The prompt also requires Docker ignore files for build outputs and dependency folders.Source: G Generation System Prompt · p. 19
  • The workflow requires building and running the app with Docker, checking logs, browsing the UI, saving screenshots, and fixing issues before submission.Agents must also report testing outcomes and deployment problems.Source: G Generation System Prompt · p. 19
  • The prompt forbids mocks for required integrations, requiring real Stripe test-mode calls and SMTP email when payment or email features are specified.It rejects submissions that fake payments or bypass required external calls.Source: G Generation System Prompt · p. 20
  • The prompt requires separate technical and user-facing READMEs and confines application files to the designated generated-app workspace.The user guide should explain how to navigate and test the application, including any default authentication credentials.Source: G Generation System Prompt · p. 20

H Spec Example

The example specification describes a text-only social app with guest and member roles, while the accompanying UI tests check key account, posting, discovery, and engagement workflows.

H Spec Example

  • Members can create accounts, manage profiles and posts, follow others, like and comment, while guests browse public content but cannot perform member actions.The example also specifies username/email sign-in, no required email verification, and 280-character posts.Source: H Spec Example · pp. 20–21
  • The app specification requires a feed with newest/trending sorting and keyword/hashtag search, plus notifications for relevant follow, like, and comment events.Posts are text-only, and key workflows are intended to run in one browser session.Source: H Spec Example · p. 21

I UI Test Example

  • The UI-test examples check onboarding, profile updates, post creation and ownership controls, and guest browsing restrictions.The listed workflows specify 6 substeps for onboarding, profile updates, and post creation; 8 for editing/deleting; and 9 for guest restrictions.Source: I UI Test Example · pp. 21–22
  • Further UI-test examples verify follow-driven feed updates and notifications, like/comment notifications, feed sorting, keyword/hashtag search, and the post character limit.The listed workflows specify 11 substeps each for following and like/comment notifications, 10 for sorting, 7 for search, and 6 for character-limit validation.Source: I UI Test Example · pp. 22–23

Related papers

Loading 2603.04601v3…