PaperlayerView exact revision on arXiv

Source-linked AI summary

WebCraftBench: Evaluating Web Application Generation from a Software Testing Perspective

Chenxu Liu, Zilu Zou, Peizhong Gao, Jiawen Tao, Zhexin Zhang, Guang Chen, Haowei Lin, Ying Zhou, Tianyi Bai, Dolly Deng, Suncong Zheng, Maxm Pan

arXiv:2609.15387v3cs.SEcs.AI

TL;DR

Automated evaluation struggles to reflect human judgments of generated web applications when static checks miss unreachable functionality and interactive checks conflate application and agent failures. WebCraftBench guides runtime exploration with code coverage, organizes evidence in a state-transition graph, and scores applications across three dimensions. It achieves 85.3% agreement with human preferences, while evaluations reveal complementary model strengths.

  • Problem

    Automated benchmarks for generated web applications can miss unreachable functionality or confuse application defects with agent execution failures, limiting their ability to align with human preferences.

    Source: §1 · p. 2

  • Method

    WebCraftBench instruments applications, guides user-simulated exploration with code coverage, abstracts traces into state-transition graphs, and scores aesthetics, usability, and requirement alignment separately from exploration.

    Source: §1 · pp. 2–3

  • Results

    Across 17 frontier LLMs, strengths are complementary across aesthetics, usability, and requirement alignment; agreement with human preferences is 85.3%.

    Source: §1 · pp. 2–3, §7 · p. 11

  • Takeaways & Limitations

    Coverage guidance improves exploration coverage, and model rankings remain stable when the judge model is replaced.

    Source: §1 · p. 2, §7 · p. 11

    The dataset covers single-turn, text-only, front-end-only tasks from one platform and excludes iterative repair, multimodal input, and back-end integration.

    Source: §6 · p. 11

Abstract

from arXiv · show

Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evaluation remains challenging. Static benchmarks can credit functionality that exists in source code but is unreachable at runtime. Interactive benchmarks exercise the application, yet incomplete exploration can cause them to miss implemented functionality and confound application defects with agent execution failures. To address these limitations, we propose WebCraftBench, an interactive benchmark for evaluating web application generation from a software testing perspective. WebCraftBench instruments each generated application and uses code coverage to guide an agent in exploring its functionality through user-simulated interactions. It then abstracts the interaction trace into a state-transition graph and evaluates the application along three dimensions: visual aesthetics, usability, and requirement alignment. By separating exploration from scoring, WebCraftBench collects runtime evidence without constraining exploration to predefined acceptance criteria. WebCraftBench comprises 369 real-world user requirements and 5,088 acceptance criteria. Evaluation of 17 frontier LLMs reveals distinct strengths across the three dimensions, with no model leading on every dimension. On 197 validated sessions sampled from an internal arena, WebCraftBench achieves 85.3\% agreement with human preferences, with agreement generally increasing as the score difference between paired applications grows. Further experiments show that coverage guidance improves exploration coverage and the model rankings remain stable when the judge model is replaced.

1. Introduction

WebCraftBench addresses gaps in automated evaluation by separating runtime exploration from scoring and organizing runtime evidence for evaluation. It evaluates generated web applications on aesthetics, usability, and requirement alignment, with results validated against human preferences.

1. Introduction

  • Static benchmarks can credit source-code functionality that users cannot reach, while interactive benchmarks can confuse application defects with agent execution failures.GUI interaction and evidence-interpretation errors make it difficult to distinguish these causes.Source: §1 · p. 2
  • WebCraftBench uses code coverage to guide exploration toward unexecuted code and quantify exercised implementation, while keeping exploration separate from scoring.This prevents acceptance criteria from constraining the exploration process.Source: §1 · p. 2
  • 85.3% pairwise agreement on 197 validated samples; rankings correlate with Code Arena at ρ=0.890 across 17 shared models.The two evaluations show broadly consistent capability tiers among the shared models.Source: §1 · pp. 2–3
  • The benchmark comprises 369 real-world user requirements and 5,088 acceptance criteria.The criteria are used for scoring, while the benchmark evaluates visual aesthetics, usability, and requirement alignment.Source: §1 · p. 3
  • WebCraftBench evaluates 17 frontier LLMs across visual aesthetics, usability, and requirement alignment, finding complementary strengths across dimensions.Additional experiments report that coverage guidance improves exploration coverage and rankings remain stable across judge models.Source: §1 · pp. 2–3

2. Related Work

Prior web-generation benchmarks either do not exercise applications or constrain interactive verification to predefined targets. WebCraftBench applies coverage-guided exploration and state abstraction to collect and organize runtime evidence.

2.1. Benchmarks for Web Application Generation

  • Static benchmarks may credit functionality that is present in source code but unreachable at runtime, while interactive benchmarks can miss behavior outside predefined targets or confuse agent failures with application defects.Static benchmarks commonly compare outputs with reference designs; interactive approaches use tests, action sequences, screenshots, or agent-checked specifications.Source: §2.1 · p. 3
  • WebCraftBench measures code coverage during exploration to quantify exercised functionality and guide further testing, while withholding acceptance criteria from the exploration agent.Cookie-Bench removes checklists but does not explicitly measure exploration adequacy; I-WebGenBench applies standardized actions to enumerated components.Source: §2.1 · p. 3

2.2. Automated GUI Testing

  • WebCraftBench adapts automated GUI testing approaches: coverage provides exploration feedback, and state abstraction groups near-duplicate pages into a compact state-transition graph for scoring evidence.Prior GUI-testing work studies exploration strategies and state abstraction at different granularities.Source: §2.2 · pp. 3–4

3. Dataset

WebCraftBench collects 369 text-only requirements from production traffic and pairs them with validated checklists of observable acceptance criteria. The dataset includes varied request lengths, informal language, and incomplete specifications.

3. Dataset

  • WebCraftBench contains 369 real-world user requirements, each paired with a checklist used exclusively for scoring.The requirements were collected from an enterprise-internal replica of Code Arena, anonymized, reviewed, and filtered for scope, comprehensibility, and duplicates.Source: §3 · p. 4, §3.1 · p. 4

3.1. Data Source

  • The 369 text-only requirements come from production traffic and range from 7 to 37,903 characters, with a mean of 601 and median of 105.The distribution captures both brief requests and detailed specifications; requirements were screened by annotators and deduplicated lexically and semantically.Source: §3.1 · p. 4

3.2. Requirement Characterization

  • 63.4% of requirements are not fully specified and 60.2% use colloquial language; entertainment and scientific demonstrations are the largest application categories.Requirements are characterized by artifact complexity, expression style, clarity, and application category.Source: §3.2 · pp. 4–5

3.3. Acceptance Criteria

  • The dataset’s 5,088 acceptance criteria cover functional, content, and visual properties, with 2,706, 1,447, and 935 criteria in those categories, respectively.Criteria describe concise, high-level observable properties rather than prescribing implementation details; each task has an average of 13.8 criteria, ranging from 2 to 77.Source: §3.3 · p. 5

4. Evaluation Process

WebCraftBench collects runtime evidence through instrumentation, coverage-guided exploration, and state abstraction, then scores applications using that evidence. Its scoring protocol separately assesses aesthetics, usability, and requirement alignment.

4. Evaluation Process

  • The evaluation workflow has four stages: instrumentation, exploration, state abstraction, and scoring; only scoring uses acceptance criteria.The first three stages collect and organize runtime evidence without access to the criteria.Source: §4 · p. 5

4.1. Instrumentation

  • Instrumentation automatically supports static HTML, Vite, Next, Create React App, and Astro, and records runtime execution with statement counters.Applications that cannot be instrumented remain in the evaluation and undergo exploration and scoring in black-box mode.Source: §4.1 · pp. 5–6

4.2. Coverage-Guided Exploration

  • The exploration agent acts as a software tester without source-code access, and a separate LLM suggests next steps after three consecutive interactions without coverage gains.Exploration ends when the agent declares testing complete or reaches a 100-step budget.Source: §4.2 · p. 6

4.3. State Abstraction

  • State abstraction normalizes snapshots and merges pages with identical representations into states, while user actions define transitions in a compact graph.The graph is used as part of the scoring evidence.Source: §4.3 · pp. 6–7

4.4. Scoring

  • Aesthetics is judged from representative screenshots, usability from the exploration trace and state graph, and requirement alignment from criterion-by-criterion runtime evidence.Aesthetics and usability use deliberative judging with five final scores aggregated by a trimmed mean and scaled to 0–100; requirement alignment scores the fraction of criteria judged implemented. The overall score combines standardized dimension scores, with scores expressing relative performance within the evaluated model pool.Source: §4.4 · pp. 7–8

5. Evaluation Results

Across five research questions, the evaluation finds complementary model strengths, 85.3% agreement with human preferences, stable rankings across judges, and higher exploration coverage with guidance.

5.1. Evaluation Setup

  • The experiment evaluates 17 frontier LLMs on all 369 requirements, producing 6,273 applications under the same four-stage pipeline.GPT-family models use Codex, remaining models use Claude Code, and Claude-Opus-4.8 performs exploration and scoring unless otherwise stated.Source: §5.1 · p. 8

5.2. RQ1: Main Results

  • Claude-Opus-5 ranks first overall and leads usability (+1.757); GPT-5.6-Sol leads aesthetics (+2.239), while Kimi-K3 leads requirement alignment (+1.283).No model leads on every dimension.Source: §5.2 · p. 8
  • Aesthetics and usability correlate at r=0.85 across models but only 0.36 across applications; newer generations score higher within represented families.The results distinguish visual polish from reliable interaction.Source: §5.2 · p. 9

5.3. RQ2: Human Preference Alignment

  • 85.3% of 197 validated pairs match human preferences, with cumulative agreement exceeding 90% when Δ𝑧≥0.25.Disagreements cluster among pairs with small score differences.Source: §5.3 · p. 9, Table 4 · p. 9

5.4. RQ3: Judge Robustness

  • Across judges, overall scores correlate at r=0.983 and rankings at ρ=0.980; 130 of 136 model pairs retain their order (95.6%).Raw-score levels and spreads change substantially despite stable aggregate rankings.Source: §5.4 · p. 10

5.5. RQ4: Instrumentation Reliability

  • Instrumentation succeeds on 6,266 of 6,273 applications (99.89%); all 360 tested Claude-Opus-5 applications remain deployable after instrumentation.The seven failures use unsupported JSX input forms, npm workspaces, or exceed the deployment-size limit; they are evaluated in black-box mode.Source: §5.5 · p. 10

5.6. RQ5: Coverage Improvement

  • 94.3% median function coverage with guidance, versus 91.9% without; 261 tasks reach at least 90% coverage, versus 221 without guidance.The low-coverage tail changes little: 15 guided tasks versus 17 unguided tasks fall below 70%.Source: Figure 3 · p. 10, §5.6 · p. 11
  • Below 70% coverage, applications typically have failures that block further exploration; the tested generated applications remain simple.The authors expect coverage guidance to bring more benefits as models generate more complex applications.Source: §5.6 · p. 11

6. Limitations

The authors identify limitations in judge dependence, model-pool-dependent standardized scores, task scope, and public availability.

6. Limitations

  • Because evaluation relies largely on LLM judges, judge preferences may affect results, although the authors validate scores against human preferences and test replacement-judge stability.The validation reports 85.3% agreement with human preferences and stable rankings across judges.Source: §6 · p. 11
  • Standardized z-scores depend on the participating LLM pool and are unsuitable for direct comparison across pools.The authors suggest fixed reference means and standard deviations for future deployments; raw scores aid interpretation.Source: §6 · p. 11
  • The dataset covers single-turn, text-only, front-end-only tasks from one platform, excluding iterative repair, multimodal input, and back-end integration.Extending the process to other frameworks and back-end webpages is left for future work because of engineering costs.Source: §6 · p. 11
  • The dataset and source code are not publicly released to prevent data leakage and comply with business confidentiality policy.Source: §6 · p. 11

7. Conclusion

WebCraftBench combines coverage-guided exploration, state abstraction, and scoring grounded in runtime evidence to evaluate generated web applications.

7. Conclusion

  • WebCraftBench achieves 85.3% agreement with human preferences and stable rankings across judges, while coverage guidance increases exploration coverage.Across 17 frontier LLMs, results show complementary strengths in aesthetics, usability, and requirement alignment.Source: §7 · p. 11

Appendices

The appendices describe how applications are instrumented and deployed, then specify evidence-based scoring procedures for visual aesthetics, usability, and requirement alignment.

A.1. Instrumentation and Deployment

  • Instrumentation handles source-level and compiled projects, while applications with instrumentation failures remain eligible for black-box exploration and scoring without coverage feedback.Framework detection selects the procedure, and deployment occurs in an isolated HTTP sandbox; instrumentation may affect deployment or runtime behavior.Source: §A.1 · p. 16

A.2. Scoring Procedures

  • For visual aesthetics and usability, three-round deliberation verifies advocate and critic claims before a final judge assigns and aggregates scores.The judge selects a 0–5 base score, adjusts seven aspects, clips the sum to [0, 5], and aggregates five runs by discarding the highest and lowest before scaling to 0–100.Source: §A.2 · p. 16
  • Visual aesthetics uses seven screenshot-grounded aspects, distinguishing deliberate simplicity from unfinished content and avoiding inferences about unobserved dynamic behavior.The aspects include first impression, layout, visual detail, information hierarchy, stylistic consistency, and emotion and trust.Source: §A.2 · p. 16
  • Usability scoring assesses seven interaction-oriented aspects using observed trace steps or state-graph nodes and edges, without treating untested paths as strengths or weaknesses.Observed failures that block core operations warrant substantial penalties.Source: §A.2 · p. 16
  • Requirement alignment checks functional, content, and visual criteria separately, requiring evidence from the running application rather than unexecuted source code.Each criterion receives a Boolean decision, supporting evidence, brief analysis, and confidence estimate; no partial-credit category is used.Source: §A.2 · pp. 16–17

B. Detailed Judge-Robustness Results

Model rankings remain highly consistent when the judge changes from Claude-Opus-4.8 to Gemini-3.7-Flash, although a small number of pairwise orders shift.

B. Detailed Judge-Robustness Results

  • Three of the six changed pairwise orders involve models whose main-judge overall-score differences are below 0.04.All models move by at most two ranks; the top four ranks and ranks 8, 9, and 12 remain unchanged.Source: B. Detailed Judge-Robustness Results · p. 18
  • Raw-score changes vary by dimension, while rankings are compared after standardizing scores within each judge’s 17-model pool.The table compares the main and Gemini judges’ standardized scores and ranks; its rank difference is Gemini rank minus main-judge rank.Source: B. Detailed Judge-Robustness Results · p. 18, Table 5 · p. 18
  • 130 of 136 model pairs (95.6%) retain their relative order across Claude-Opus-4.8 and Gemini-3.7-Flash judges.Overall scores have Pearson correlation r=0.983 and Spearman rank correlation ρ=0.980.Source: Table 5 · p. 18, B. Detailed Judge-Robustness Results · p. 18

C. The High-Margin Human–WebCraftBench Disagreement Case

Among 50 human-preference pairs with Δz≥0.75, WebCraftBench and the human rater disagree once, on a trade-off between visual polish and working functionality.

C. The High-Margin Human–WebCraftBench Disagreement Case

  • The only disagreement among 50 pairs with Δz≥0.75 favors a polished but crashing artifact for the human rater and a working artifact for WebCraftBench.The benchmark’s preference reflects advantages in usability and requirement alignment outweighing the other artifact’s visual advantage.Source: C. The High-Margin Human–WebCraftBench… · p. 19, Figure 4 · p. 19
  • The visually polished artifact’s primary gameplay path crashes after entry, while the sparser alternative’s core Sudoku interaction works.The human rater prefers the visually polished artifact; WebCraftBench prefers the working alternative.Source: C. The High-Margin Human–WebCraftBench… · p. 19
  • This case illustrates different trade-offs between aesthetics and functional reliability, but the vote alone does not identify the rater’s weighting.Source: C. The High-Margin Human–WebCraftBench… · p. 19

D. Three-Round versus Single-Round Scoring

Three-round deliberation produces lower average aesthetics and usability scores and differentiates artifacts that single-round scoring rates similarly; the raw-score appendix reports additional dimension-level relationships and scaling context.

D. Three-Round versus Single-Round Scoring

  • Deliberation yields a wider range of scores for artifacts that single-round scoring rates identically or similarly, reducing endpoint concentration and increasing differentiation.The authors suggest advocacy, criticism, and evidence verification lead to more conservative judgments.Source: D. Three-Round versus Single-Round… · p. 20
  • 15.5 points lower for visual aesthetics and 8.5 points lower for usability are the mean scores under three-round deliberation versus single-round scoring.The comparison uses paired application-level scores; the single-round baseline uses the final-round prompt of the three-round protocol.Source: Figure 5 · p. 20, D. Three-Round versus Single-Round… · p. 20

E. Raw Scores

  • At the artifact level, visual aesthetics and requirement alignment are weakly correlated (r=0.32), while usability and requirement alignment are strongly associated at the model level (r=0.88).Source: E. Raw Scores · p. 21
  • Aesthetics spans 26.8 points across models, versus 7.1–10.4 points for alignment sub-dimensions, so raw averaging would give aesthetics greater influence on between-model differences.The appendix reports raw 0–100 scores; visual aesthetics and usability scores aggregate five final-judge ratings on a 0–5 scale using a trimmed mean and multiply by 20.Source: E. Raw Scores · p. 21, Table 6 · p. 21
Loading 2609.15387v3…