Source-linked AI summary
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
Xinping Lei, Xinyu Che, Junqi Xiong, Chenchen Zhang, Yukai Huang, Chenyu Zhou, Haoyang Huang, Minghao Liu, Letian Zhu, Hongyi Ye, Jinhua Hao, Ken Deng, Zizheng Zhan, Han Li, Dailin Li, Yifan Yao, Ming Sun, Zhaoxiang Zhang, Jiaheng Liu
TL;DR
Existing web-coding benchmarks undermeasure visual fidelity, interaction quality, and repository-level reasoning. WebCompass unifies multimodal generation, editing, and repair with task-aware judging, and finds that closed-source models lead substantially while visual quality remains the persistent bottleneck.
Problem
Existing benchmarks focus on narrow, often text-conditioned and static-correctness settings, leaving important web qualities such as visual fidelity and interaction behavior insufficiently measured.
Method
WebCompass covers seven task categories across text, image, and video, using checklist-guided LLM judging for editing and repair and browser-interacting Agent-as-a-Judge evaluation for generation.
Results
Closed-source models lead by ∼25 points over the best open-source alternatives, while visual quality remains the most persistent bottleneck and no single model dominates generation, editing, and repair.
Takeaways & Limitations
Advancing web-coding agents requires stronger functional reasoning, deeper visual design understanding, and greater output consistency.
Takeaways & Limitations
The repair defect taxonomy is grounded in over 200 real-world community submissions and corresponding GitHub Issues, constraining its scope to observed high-frequency front-end failure modes.
Abstract
from arXiv · showhide
Large language models are rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability, typically text-conditioned generation with static-correctness metrics, leaving visual fidelity, interaction quality, and codebase-level reasoning largely unmeasured. We introduce WebCompass, a multimodal benchmark that provides unified lifecycle evaluation of web engineering capability. Recognizing that real-world web coding is an iterative cycle of generation, editing, and repair, WebCompass spans three input modalities (text, image, video) and three task types (generation, editing, repair), yielding seven task categories that mirror professional workflows. Through a multi-stage, human-in-the-loop pipeline, we curate instances covering 15 generation domains, 16 editing operation types, and 11 repair defect types, each annotated at Easy/Medium/Hard levels. For evaluation, we adopt a checklist-guided LLM-as-a-Judge protocol for editing and repair, and propose a novel Agent-as-a-Judge paradigm for generation that autonomously executes generated websites in a real browser, explores interactive behaviors via the Model Context Protocol (MCP), and iteratively synthesizes targeted test cases, closely approximating human acceptance testing. We evaluate representative closed-source and open-source models and observe that: (1) closed-source models remain substantially stronger and more balanced; (2) editing and repair exhibit distinct difficulty profiles, with repair preserving interactivity better but remaining execution-challenging; (3) aesthetics is the most persistent bottleneck, especially for open-source models; and (4) framework choice materially affects outcomes, with Vue consistently challenging while React and Vanilla/HTML perform more strongly depending on task type.
1 Introduction
WebCompass addresses the limits of traditional web-coding evaluation by unifying multimodal lifecycle tasks with task-aware, execution-based assessment of visual and interactive behavior.
- Motivation: Traditional code metrics miss visual fidelity, interaction behavior, responsiveness, accessibility, and user experience in interactive front-end applications.They primarily measure algorithmic correctness or repository-level bug fixing.
- Evaluation: Editing and repair use checklist-guided LLM judging, while generation uses an autonomous browser agent that explores websites through MCP and synthesizes targeted tests.The protocols reflect constrained patch-based tasks versus open-ended runtime behavior.
- Evaluation: WebCompass combines multimodal coverage with execution-based evaluation to approximate human acceptance testing more realistically and scalably.Visual inspection and interactive probing address complementary aspects of web quality.
- Benchmark scope: WebCompass evaluates generation, editing, and repair across text, image, and video inputs within one framework.This enables cross-task and cross-modality comparisons.
2 WebCompass
WebCompass organizes realistic web-engineering tasks across seven modality-task categories and constructs deterministic, diverse instances through human-curated prototype expansion and task-specific transformations.
- 2.1 Overview: The benchmark combines three input modalities and three task types into seven categories covering generation, editing, and repair workflows.Generation uses text, screenshots, or video; editing and repair produce repository patches from textual or visual inputs.
- 2.1 Overview: WebCompass evaluates nuanced intent understanding, cross-modal reasoning, and repository-level context awareness in realistic web-development scenarios.These dimensions extend beyond basic code generation.
- 2.2 Data construction: A multi-stage human-in-the-loop pipeline collects, filters, manually curates, and expands executable prototypes into downstream editing and repair tasks.Prototype construction includes length filtering, automatic quality scoring, human selection, and single-page or multi-page expansion.
- 2.2 Data construction: Generation data is strengthened with structured requirements, augmented multi-page screenshots, dynamic keyframes, and interaction videos to represent complex behaviors.These procedures address underspecified queries, limited page coverage, and the inability of static screenshots to express animations and state changes.
- 2.2.4 Editing & Repair Task Data Collection Pipeline: Repair instances use vague natural-language hints and exact search/replace annotations, yielding uniquely correct, runnable, reproducible solutions without leaking implementation details.The inverse annotations support automated verification and error localization.
3 Evaluation Methodology
WebCompass uses task-specific evaluation pipelines that combine static evidence with browser execution, interaction testing, and visual inspection. Its scoring aggregates checklist performance while explicitly handling build, rendering, and runtime failures.
- LLM-as-a-Judge for Editing and Repair: Editing and repair evaluation applies model patches, captures before-and-after screenshots and logs, and scores checklist items independently.Repair additionally supplies the defect description, ground-truth modifications, and reference fixed screenshot to the judge.
- Evaluation Design: Traditional tests verify functional behavior, whereas screenshot comparison verifies appearance; the Agent-as-a-Judge pipeline combines both through interactive browser evaluation.This design targets multi-step behaviors, state transitions, dynamic content, visual fidelity, and layout quality.
- Agent-as-a-Judge for Generation: Generation evaluation uses Agent-as-a-Judge to launch websites in a browser, execute interactions, synthesize tests, and score auditable runtime evidence.The agent uses MCP, DOM snapshots, console logs, screenshots, and executable JavaScript tests while preserving checklist assertions.
- Scoring: The task score is the harmonic mean of normalized checklist-item scores, with zero scores smoothed by ϵ = 1 and calculated separately for three evaluation dimensions.The harmonic mean penalizes imbalanced performance across criteria rather than allowing strong dimensions to offset complete failures.
- Failure Handling: Fallback rules distinguish complete build failure, partial rendering failure, and runtime crashes so cascading failures degrade dimensions according to what remains executable and testable.Complete build failure sets functional and visual dimensions to 0, while later-stage failures score rendered or reachable subsets.
4 Experiments
WebCompass experiments show substantial closed-source advantages, persistent visual-quality weaknesses, and task-dependent differences in repair, editing, frameworks, patching, and output stability.
- Main Results: Claude-Opus-4.5 and Gemini-3-Pro-Preview lead overall, while Qwen3-VL-235B-A22B-Instruct trails the top closed-source model by over 26 points.Claude and Gemini achieve Overall scores of 67.40 and 66.68, respectively; Qwen3-VL-235B-A22B-Instruct reaches 41.14.
- Task-Type Breakdown: Generation and Editing generally rank executability above functional and visual dimensions, whereas Repair ranks Interaction Integrity above Reference Fidelity and Root-Cause Targeting.For example, Claude’s Generation scores are RUN 77.18 > SPI 68.95 > DSQ 62.26, while Gemini’s Repair scores are ITI 87.30 > RFF 72.00 > RCT 54.16.
- Main Results: Visual quality is the lowest-scoring axis in Generation and Editing, with Gemini-3-Pro-Preview reaching only 64.07 on Generation Design Quality.Gemini-3-Pro-Preview and Gemini-3-Flash-Preview can outperform GPT-5.2 visually despite comparable executability scores.
- Framework Evaluation: Vanilla leads Generation and Editing across evaluated models, Vue usually underperforms, and React can be strongest for Repair.The framework pattern is attributed to differences in build-toolchain complexity, syntax, component structure, and defect localization.
- Task-Type Breakdown: Advanced Animation editing is hardest, while semantic repair defects are the main repair bottleneck.Editing difficulty increases from Business Scenario to Real-time & Async, Interactive Components, and Advanced Animation; Semantic Error scores lowest in Repair.
- Consistency and Stability: Harmonic-mean aggregation favors consistency, and worst-case evaluation exposes sharper degradation for Qwen3-VL than Gemini.Gemini retains approximately 80% from Pass@1 to W@4, compared with approximately 69.5% for Qwen3-VL; GPT-5.2’s editing category reversal reflects higher cross-subtask variance.
5 Related Work
WebCompass builds on work in code agents and web-development benchmarks while addressing fragmented task and modality coverage. Its task-aware evaluation combines checklist-guided judging for editing and repair with agent-based browser interaction for generation.
- Web-coding benchmarks: Prior web-coding benchmarks commonly organize evaluation around task type and input modality, but many focus on only one of each.Existing categories include generation, editing, and repair, with text or visual inputs.
- WebCompass: WebCompass unifies generation, editing, and repair across text, image, and video inputs for cross-task and cross-modality comparison.The benchmark covers seven task categories spanning these three modalities and three task types.
- Evaluation paradigms: Interactive web artifacts require judging visual quality and interaction behavior alongside code correctness.Common evaluation paradigms include deterministic tests, agent-based interaction, and LLM/MLLM-as-a-Judge, each with distinct coverage or reliability trade-offs.
- Evaluation paradigms: Checklist-guided LLM judging evaluates editing and repair, while Agent-as-a-Judge uses browser interaction and iterative test synthesis for open-ended generation.The protocols are matched to task structure: constrained patch outputs for editing and repair, and diverse long-horizon runtime behavior for generation.
6 Conclusion
WebCompass unifies multimodal evaluation of web generation, editing, and repair with task-aware judging of execution, functionality, and visual quality. Results show a substantial closed-source advantage, persistent visual limitations, and different capability demands across tasks.
- Conclusion: Closed-source models lead by ∼25 points over the best open-source alternatives, while no single model dominates generation, editing, and repair.The conclusion reports that the three task types stress fundamentally different capabilities.
- Conclusion: Visual quality remains the most persistent bottleneck, including for frontier models.The paper argues that progress requires stronger visual design understanding in addition to functional reasoning.
- Conclusion: WebCompass evaluates executability, functional behavior, and visual quality across generation, editing, and repair using text, image, and video modalities.Its task-aware framework combines LLM-as-a-Judge and Agent-as-a-Judge protocols.
A.1 Limitations
The benchmark has important scope and evaluation boundaries: it focuses on front-end development, structured rather than creative intent, dynamic interactions that are difficult to reproduce in real time, static-task contamination risk, and high evaluation cost.
- Scope: WebCompass evaluates front-end web development but excludes back-end design, server-side logic, APIs, and deployment workflows.Extending the benchmark to back-end tasks would provide a more complete assessment of web engineering.
- Scope: Structured generation queries prioritize reproducible instruction-following evaluation over interpreting vague, creative intent.The authors identify this as a trade-off and suggest complementary human-preference evaluation for creative divergence.
- Evaluation boundaries: Evaluation may not fully reflect functionality or user experience on highly dynamic pages with timing-sensitive state transitions.Examples include browser-based games, continuous animations, and rapidly changing application states.
- Evaluation boundaries: As a static benchmark, WebCompass remains susceptible to future data contamination despite diverse sources and original task synthesis.Periodic updates or dynamic task generation may be needed to maintain contamination-free evaluation over time.
- Evaluation cost: Agent-as-a-Judge substantially increases evaluation time and cost by launching browsers, executing interactions, and synthesizing iterative tests.This may limit accessibility for resource-constrained research groups.
A.2 Disclosure of LLM Assistance
The authors state that they independently conceived and executed the scientific work and used LLMs only for language editing and manuscript clarity.
- Disclosure: LLMs were used exclusively for language editing and improving manuscript clarity, with experimental data and reported results subject to rigorous human verification.The authors state that the scientific ideas, implementations, and analyses were independently conducted.
A.3 Per-Dimension Framework Evaluation
Table 6 provides the per-dimension breakdown of the framework-subset evaluation summarized in Section 4.3.2, using the dimension abbreviations defined in Table 3.
- Table 6 reports the per-dimension results for the framework-subset evaluation.
- The table covers the framework-subset evaluation summarized in Section 4.3.2.
- Dimension abbreviations in Table 6 follow the definitions provided in Table 3.
A.4 Model Card
This section identifies the model variants used in the experiments and situates the framework comparison in a per-dimension evaluation across React, Vue, and Vanilla HTML/JS.
- Table 7 lists all model variants referenced in the experiments, including an auxiliary comparison model.The auxiliary model is used in Section 4.3.7.
- Table 6 evaluates different front-end frameworks across per-dimension results.
- Each model is tested on 60 randomly sampled tasks per category using React, Vue, and Vanilla HTML/JS.
- Table 6 marks the best result per framework in green bold and the second-best result in blue underline.
- Table 7 also provides the list of model variants referenced in the experiments.
A.5 Detailed Worst-of-n Stability Results
Table 8 extends the main-text Worst-of-n stability analysis with scores across all nine evaluation dimensions, grouped by task category.
- Table 8 reports the extended Worst-of-n stability analysis.
- The analysis includes Pass@1, W@2, and W@4 scores.
- Scores are reported across all nine evaluation dimensions.
- The results are grouped by task category.
A.6 Prompt Templates
The prompt templates define generation, editing, repair, checklist, and judge-evaluation instructions, with multimodal generation prompts and structured checklist requirements for web evaluation.
- A.6 Prompt Templates: The appendix presents task prompts for generation, editing, and repair, plus evaluation prompts for LLM-as-a-Judge and Agent-as-a-Judge.It also includes auxiliary prompts for checklist generation and error analysis.
- A.6.2 Checklist Generation Prompt: Checklist generation organizes evaluation into Runnability, Spec Implementation, and Design Quality dimensions.
- A.6.2 Checklist Generation Prompt: Runnability uses a fixed 10-point item checking page loading, console errors, and failed network requests.
- A.6.2 Checklist Generation Prompt: Spec Implementation assigns 60-70 points to core functionality, interactions, user journeys, secondary features, and robustness.Core functionality and main interactions receive the most stringent checks.
- A.6.2 Checklist Generation Prompt: Design Quality evaluates universal visual properties and query-specific visuals such as chart readability or animation smoothness.
- A.6.2 Checklist Generation Prompt: The checklist requires 10-16 verifiable items whose maximum scores sum to 100.Items must be checkable through screenshots, console or network panels, and visual alignment checks.
- A.6.3 Generation Prompts: Generation prompts support text, image, and video inputs while requiring pure Markdown with fenced code blocks for complete runnable repositories.The prompts emphasize matching reference designs and including all necessary files, README instructions, and static-server compatibility.
- A.6.3 Generation Prompts: Video-guided generation instructs models to analyze temporal sequences, visual design, layout structure, responsive behavior, and interaction details.The protocol covers animations, state transitions, colors, typography, spacing, shadows, breakpoints, and component hierarchy.
A.6.1 Error Analysis Prompt
The error-analysis prompt standardizes deduction classification by separating execution, functional, visual/style, and non-functional errors, then assigning specific root causes with deterministic scoring and JSON output rules.
- Taxonomy: The taxonomy classifies deductions across four domains using fifteen fine-grained error types and root-cause labels.The domains cover Code Execution, Functional, Visual/Style, and Non-Functional issues; root causes include requirement omission, insufficient reproduction, capability limitation, hallucination, and oversimplification.
- Classification procedure: The decision flow routes each deduction from page crashes to execution errors, nonworking features to functional errors, visual mismatches to style errors, and other defects to non-functional errors.The prompt distinguishes E1.x, E2.x, E3.x, and E4.x according to the observed failure mode.
- Scoring rules: Points are allocated proportionally across distinct issues, split evenly when severity is unclear, and weighted toward critical failures over minor ones.The output rules require deductions to reconcile with max_score - score and represent multiple distinct issues separately.
- Output format: The prompt returns a JSON array containing checklist identifiers, tasks, scores, typed errors, descriptions, deducted points, and root causes.Each error receives exactly one type code, while full-score items contain no errors.