PaperlayerView exact revision on arXiv

Source-linked AI summary

WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, Tao Xie

arXiv:2601.02430v3cs.SEcs.AI

TL;DR

Benchmarking LLM-generated web apps requires authentic user requirements, general evaluation that does not depend on fixed implementations or tests, and interpretable results. WebCoderBench addresses these needs with a real-world dataset, 24 metrics across 9 perspectives, automated rule-based and LLM-as-a-judge evaluation, and human-preference weighting. Across 12 LLMs and 2 agents, no single model achieves dominant effectiveness across all metrics, supporting targeted optimization.

  • Problem

    Benchmarking LLM-generated web apps lacks authentic requirements, general evaluation independent of ground-truth implementations or tests, and interpretable results.

  • Method

    WebCoderBench combines 1,572 authentic user requirements with 24 metrics across 9 perspectives, automated rule-based and LLM-as-a-judge evaluation, and human-preference-based weighting.

  • Results

    No single model achieves dominant effectiveness across all metrics when WebCoderBench evaluates 12 representative LLMs and 2 LLM-based agents.

  • Takeaways & Limitations

    Interpretable, human-aligned results enable developers to optimize models in a targeted manner.

  • Takeaways & Limitations

    The benchmark currently evaluates only front-end web applications implemented in native HTML, with backend tasks and common frameworks deferred to future versions.

Abstract

from arXiv · show

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.

1 Introduction

WebCoderBench addresses the need for authentic, general, and interpretable evaluation of LLM-generated web apps. It combines real user requirements with automated fine-grained metrics and human-preference weighting, revealing varied model strengths and no universally dominant model.

  • Motivation: Existing web-app benchmarks struggle to represent authentic user requirements, evaluate open-ended implementations generally, and provide fine-grained automated analysis.Real requirements vary in style and complexity, while multiple valid implementations make fixed ground-truth code or tests impractical.
  • Benchmark: WebCoderBench contains 1,572 real user requirements collected to reflect diverse real-world expressions and expected web-app complexities.The dataset includes varied modalities and expression styles, from precise to ambiguous and technical to colloquial.
  • Evaluation framework: WebCoderBench provides 24 fine-grained metrics across 9 perspectives using rule-based and LLM-as-a-judge evaluation without ground-truth implementations or test cases.This design aims to support fully automated, objective, and general evaluation of open-ended web-app generation.
  • Interpretability: Human-preference-based metric weights produce overall scores aligned with real-world user priorities while preserving individual metric scores for interpretation.The benchmark therefore reports both quantitative metric-level results and human-aligned aggregate insights.
  • Findings: Experiments on 12 representative LLMs and 2 LLM-based agents show varying strengths across web-app quality perspectives, with no model leading on all 24 metrics.These results provide interpretable information for targeted model improvement.

2 Related Work

Prior web-app generation benchmarks use foundation models and agents to generate applications, but their evaluation settings remain limited. Existing approaches commonly rely on synthetic requirements, reference artifacts, predefined tests, or manual labeling rather than a fully general and automated evaluation process.

  • LLM-driven systems: LLM-driven web-app generation systems include foundation models and conversational agents that translate natural-language requirements into web applications.These agents may use cloud-based sandboxes for rendering and deployment without local environment setup.
  • Reference-based generation: Earlier web-app generation research commonly generates applications from screenshots or sketches and compares code or rendered outputs with corresponding references.Such reference-based evaluation presumes an original code artifact or screenshot for comparison.
  • Benchmark limitations: Existing benchmarks often use LLM-generated or expert-curated requirements, predefined test cases or ground-truth code, and manual labeling.These choices limit authenticity, generality, or automation across open-ended user requirements.

3 Dataset

WebCoderBench constructs a diverse dataset from anonymized real-world user requirements, filtering and deduplicating them before creating validated checklists for objective evaluation. Its 1,572 requirements span varied application categories, clarity levels, expression styles, modalities, lengths, and complexities.

  • Dataset diversity: The dataset includes 20 application categories, with Utility Websites (404), Entertainment (435), and Online Education (131) among the largest represented groups.AI-powered, Fintech, and Scientific Demo applications are also included.
  • Data collection and processing: WebCoderBench retains 1,572 requirements after collecting, merging, reviewing, deduplicating, and filtering real-world user requests.The original collection randomly sampled 5,000 anonymized requirements, and the final filtering retained 1,572.
  • Ground-truth checklists: Ground-truth checklists cover functionality, visual design, and content, combining outputs from three LLMs with human merging and validation.The checklists contain only high-level and minimal requirement points that generated artifacts should satisfy.
  • Dataset diversity: 764 requirements are clear, 730 intermediate, and 78 vague, preserving mostly executable tasks alongside more challenging ambiguous requests.This distribution emphasizes interpretable tasks while retaining difficult cases for LLM evaluation.
  • Dataset diversity: Technical (683) and colloquial (724) descriptions dominate, with role-playing (60) and analogy-based (105) requirements adding linguistic diversity.The dataset therefore includes multiple levels of formality in user expression.
  • Dataset diversity: Requirements range from simple to complex and include 1,413 text-only, 123 image-containing, and 36 URL-containing inputs.Most samples are medium (658) or simple (433) in artifact complexity, while requirement lengths range from 3 to 16,198 characters.
  • Summary: Overall, the dataset covers varied web app types, expression styles, modalities, and complexities for evaluating LLM web app generation.The dataset statistics characterize it as diverse across the dimensions reported in this section.

4 Evaluation Metrics

WebCoderBench evaluates generated web apps through 24 metrics across nine perspectives, combining rule-based and LLM-as-a-judge methods for general and requirement-aligned quality. It aggregates standardized metric scores using weights derived from human rankings so the overall score reflects user preferences.

  • Metric design: WebCoderBench defines 24 quantitative metrics across nine perspectives covering general quality and alignment quality.Each metric produces a score from 0 to 100, with higher values indicating higher quality.
  • General quality: General quality assesses code, visual, content, performance, accessibility, and maintainability through rule-based and LLM-as-a-judge metrics.These perspectives address implementation, presentation, resources, runtime behavior, compatibility, and long-term code maintenance.
  • Alignment quality: Alignment quality uses human-labeled ground-truth checklists and an LLM judge to measure functional, visual, and content consistency with user requirements.The three alignment metrics compare generated artifacts with requirements across functionality, appearance, and textual or multimedia content.
  • Weight assignment: Participants rank the nine perspectives and their internal metrics, producing preference-aligned weights from 141 valid survey responses.The survey received 1,076 views and 899 responses, with responses filtered by completion time before the final valid set was obtained.
  • Weight assignment: Borda Count converts participant rankings into normalized perspective and metric weights, with each metric weight combining its own weight and its perspective weight.The resulting metric weights sum to 1 and are intended to capture trade-offs among quality perspectives.
  • Overall scoring: Overall scores average sample-level weighted z-scores across all 1,572 samples, standardizing metrics to comparable scales before aggregation.For metric j, z_i,j = (x_i,j−µ_j)/σ_j, where µ_j and σ_j are the metric mean and standard deviation over all samples.

5 Evaluation Results and Analysis

WebCoderBench evaluates 12 LLMs and 2 agents using weighted, interpretable metrics across diverse requirements. Results show no universally dominant model, with performance varying across perspectives, model types, requirement characteristics, and user personas.

  • 5.1 Settings: 12 LLMs and 2 agents were evaluated using identical prompts for LLMs, while agents were assessed on 165 requirements.The models span multiple families, versions, modalities, and capability types.
  • 5.2 RQ1: Main Results: GPT-5-High achieved the highest overall score, while GLM-4.5 ranked second overall among all evaluated models.GPT-4o had the weakest effectiveness, with below-average scores across all perspectives and six of nine perspectives ranking last.
  • 5.2 RQ1: Main Results: No single model dominated all 24 metrics; GLM-4.5 exceeded average effectiveness across all nine perspectives but not every fine-grained metric.The results indicate that current models remain specialized rather than universally strong.
  • 5.2 RQ1: Main Results: Open-source models narrowed the effectiveness gap to less than 6% from GPT-5-High, while newer models generally outperformed earlier models such as GPT-4o.The comparison reflects rapid evolution in LLM effectiveness.
  • 5.2 RQ1: Main Results: Generalist models outperformed coder models in effectiveness, indicating that web app generation requires requirement understanding and real-world knowledge in addition to coding ability.LLM-based agents generally aligned well with requirements but showed lower performance and accessibility as external resources and generated interfaces became more complex.
  • 5.3 Additional Analysis: Models sometimes performed better on vague requirements because they allow greater freedom and reduce penalties from fine-grained mismatches.User personas also weighted perspectives differently, with designers emphasizing visual quality and code-related personas emphasizing code quality and maintainability.

6 Conclusion

The paper introduces WebCoderBench as an automated, comprehensive, and interpretable benchmark for web app generation. Its results show narrowing effectiveness differences between open- and closed-source models, rapid capability evolution, and no model dominating every perspective.

  • WebCoderBench combines 1,572 authentic user requirements with 24 evaluation metrics for automated, comprehensive, and interpretable web app assessment.
  • Human-preference weights enable developers to optimize models in a targeted manner using interpretable evaluation results.

Limitations

The benchmark has scope, data-access, dataset-distribution, annotation, metric-design, and metric-stability limitations. The authors describe mitigation steps and planned extensions for several of these constraints.

  • The benchmark evaluates only front-end applications implemented in native HTML, with backend tasks and common frameworks reserved for future versions.
  • The dataset and evaluation-metric implementation cannot be publicly released because the real user requirements are subject to internal legal constraints.
  • Dataset distribution can influence evaluation results despite the use of a standardized collection pipeline.
  • Manual annotation reliability can affect results, although experienced annotators, LLM-assisted labeling, human verification, triple annotation, and majority voting provide mitigation.
  • Evaluation outcomes depend partly on metric design, and the authors plan to enrich the 24-metric suite further.
  • Metric correctness and stability can influence evaluations, so rule-based metrics receive inspections and LLM-as-a-judge metrics are repeated and statistically tested.

A Evaluation Metrics

The evaluation suite documents each metric’s purpose, implementation, and scoring procedure. Scores are scaled for uniformity and discriminability, while the benchmark reports weighted and unweighted views of model differences.

  • The metric documentation specifies each metric’s purpose, implementation details, and score calculation, with final scores ranging from 0 to 100.Scaling is applied to support a uniform distribution and sufficient discriminability.
  • Tables 7–9 provide the detailed purpose, implementation, and score-calculation formulas for the evaluation metrics.

B Unweighted Z-score Results

The paper rescales metric scores into comparable unweighted z-scores before applying derived weights. These z-scores preserve model differences and support rankings by individual metric.

  • Z-scores measure each metric’s deviation from its mean in standard deviations, making raw scores comparable before weighting.The authors present detailed unweighted z-scores in Table 10 and then apply derived metric weights.
  • Unweighted values are substantially larger than the weighted values in Table 4, while still faithfully reflecting differences between models.
  • The evaluation produces 24 sub-leaderboards by ranking models on each individual metric.These rankings expose metric-specific capabilities and indicate where model developers can target optimization.

C Analysis of Unscorable Cases

The evaluation framework organizes web-application quality across user-prioritized perspectives and fine-grained dimensions. Table 11 reports the number of unscorable cases for each metric and model.

  • Table 11 reports the number of unscorable cases for every evaluation metric and model.
  • Human-preference questionnaires rank the importance of visual, code, content, accessibility, and maintainability dimensions.The questionnaire asks real end users to rank evaluation perspectives and their constituent dimensions by importance.
  • Visual quality: The framework measures visual quality through overall experience, component and icon consistency, layout consistency and sparsity, and visual harmony.
  • Code quality: Code-quality metrics cover functionality correctness, best practices, error handling, runtime console errors, and static syntax checking.
  • Content and accessibility: Content and accessibility are evaluated through media and resource validity, copywriting, placeholder use, accessibility support, and browser or mobile compatibility.
  • Requirement processing: Dataset construction merges and validates multi-turn requirements, retaining substantive additions and giving later contradictory turns precedence.

E.2 Dataset Filtering

Dataset filtering removes requirements that are ambiguous, incomplete, non-native to standard web development, or difficult for experts to interpret. Human annotators also validate checklists and classify the retained requirements by modality, clarity, expression style, and artifact complexity.

  • Filtering criteria: Requirements are marked unusable when they are ambiguous or incoherent, lack essential supplementary data, target non-native web scenarios, or are difficult to understand.Examples include an unexplained request, missing reference images, and an ESP32 display implementation.
  • Filtering criteria: Concrete web-app requests such as implementing Tetris or generating a Bomberman-style mini-game are treated as suitable examples.
  • Checklist construction: Human annotators validate LLM-generated functional, visual, and content checklists by adding omitted user requirements and deleting unsupported or unreasonable items.
  • Checklist construction: Checklist items are retained when at least two of three models consider them essential, with semantically equivalent requirements merged.
  • Requirement classification: The dataset classifies retained requirements by input modality, clarity level, expression style, and expected artifact complexity.
  • Artifact complexity: Complexity levels range from stateless single-function tools with minimalist interfaces to collaborative, platform-scale applications requiring real-time synchronization or advanced processing.
Loading 2601.02430v3…