Source-linked AI summary
Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification
Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, Jie Tang
TL;DR
Existing benchmarks provide limited evidence about complex, multimodal, end-to-end website development. Vision2Web addresses this gap with a hierarchical real-world benchmark and workflow-based verification combining GUI agents with VLM judging. Experiments reveal substantial capability gaps across task levels, with limitations emerging in cross-modal reasoning, long-horizon planning, coordination, and full-stack integration.
Problem
Existing benchmarks inadequately cover holistic multimodal website development, hierarchical task complexity, and reliable end-to-end evaluation.
Method
Vision2Web constructs a hierarchical benchmark from real-world websites and evaluates tasks with structured workflows, GUI-agent verification, and VLM-based visual judging.
Results
Experiments reveal notable performance gaps across all three task levels, including limitations in cross-modal reasoning, long-horizon planning, and multi-page coordination.
Takeaways & Limitations
Strong performance on isolated tasks does not reliably transfer to end-to-end system construction, exposing deficiencies in structural complexity, cross-page coordination, and persistent-state reasoning.
Takeaways & Limitations
Evaluation may retain residual errors from automated GUI-agent execution and VLM-based visual judgment, and the benchmark focuses on website development rather than all software engineering tasks.
Abstract
from arXiv · showhide
Recent advances in large language models have improved the capabilities of coding agents, yet systematic evaluation of complex, end-to-end website development remains limited. To address this gap, we introduce Vision2Web, a hierarchical benchmark for visual website development, spanning from static UI-to-code generation, interactive multi-page frontend reproduction, to long-horizon full-stack website development. The benchmark is constructed from real-world websites and comprises a total of 193 tasks across 16 categories, with 918 prototype images and 1,255 test cases. To support flexible, thorough and reliable evaluation, we propose workflow-based agent verification paradigm based on two complementary components: a GUI agent verifier and a VLM-based judge. We evaluate multiple visual language models instantiated under different coding-agent frameworks, revealing substantial performance gaps at all task levels, with state-of-the-art models still struggling on full-stack development.
1. Introduction
Vision2Web addresses limitations in existing coding-agent benchmarks by evaluating visual website development across progressively complex stages with realistic multimodal tasks and workflow-based verification. Its experiments reveal substantial capability gaps across all levels, especially in cross-modal reasoning, long-horizon planning, and multi-page coordination.
- Motivation: Existing benchmarks emphasize localized code edits or static webpage reproduction, leaving holistic multimodal and end-to-end software engineering capabilities insufficiently evaluated.The stated gaps include limited task formulation, misaligned multimodal coverage, and coarse end-to-end evaluation criteria.
- Benchmark: Vision2Web introduces a hierarchical benchmark for visual website development that evaluates autonomous multimodal coding agents through agent verification.Website development combines visual prototypes, textual requirements, and codebases across the software lifecycle.
- Benchmark: Three progressively harder levels—static webpages, interactive frontends, and full-stack websites—enable capability disentanglement and explicit failure attribution across development stages.The levels progress from fine-grained visual understanding to holistic system construction.
- Dataset: 193 website development tasks with 1,255 test cases span four major categories and 16 subcategories, reflecting the diversity of real-world websites.The benchmark is curated from publicly accessible websites through data collection, automated filtering, and agent-assisted annotation.
- Evaluation: Workflow-based verification combines GUI-agent execution of expert-designed workflows with a VLM-based judge to assess functional correctness and visual fidelity reproducibly.The paradigm is implementation-agnostic and preserves flexible agent-based interactions.
- Findings: Experiments reveal notable gaps in state-of-the-art coding agents across all three levels, including limitations in cross-modal reasoning, long-horizon planning, and multi-page coordination.The findings motivate future work on agent reasoning and software development performance.
2. Overview of Vision2Web
Vision2Web organizes visual website development into three increasingly complex task levels and constructs its benchmark from filtered, reviewed real-world websites. The resulting dataset combines multimodal inputs, diverse website categories, and evaluation statistics that scale with task complexity.
- Task Formulation: The benchmark formalizes visual website development as three levels targeting distinct capabilities: static UI reproduction, interactive multi-page frontend development, and full-stack system delivery.The levels progressively add responsive visual fidelity, navigation and state consistency, and integrated debugging with complex application states.
- Task Formulation: Static webpage tasks require responsive reproduction from desktop, tablet, and mobile prototypes, while interactive frontend tasks require coherent navigation across multiple pages.Full-stack tasks add structured requirements, integrated debugging, persistent states, and cohesive system delivery.
- Task Formulation: All tasks include multimedia resources and explicit requirements, enabling systematic assessment across the stages of visual website development.Resources include images, icons, videos, and fonts.
- Dataset Construction: The dataset pipeline combines structural assessment, VLM-based content screening, and manual review to select realistic, diverse, and well-defined website tasks.Structural filtering reduced candidates to 63,515 websites, followed by retention of 7,391 pages after content screening.
- Dataset Statistics: 193 tasks comprise 100 static webpage, 66 interactive frontend, and 27 full-stack website tasks across four major categories and 16 subcategories.The benchmark includes 918 prototype images and 1,255 test cases.
- Dataset Statistics: Task complexity increases from static webpages to interactive frontends and full-stack websites, reflected in prototype images, test cases, and input text tokens.Static tasks emphasize visual fidelity, frontends add navigation interactions, and full-stack tasks combine extensive content with complex functionality.
3. Workflow-Based Agent Verification
Vision2Web converts end-to-end website testing into structured workflows that preserve agent flexibility while constraining execution through dependency-aware verification nodes. GUI agents assess functional behavior and VLM judges assess visual fidelity, with expert-led annotation and human quality control supporting coverage and reproducibility.
- Motivation: Traditional functional and visual tests struggle with diverse website implementations, unconstrained agent behavior, brittle scripts, and poor reproducibility.These challenges motivate a unified verification framework that can handle implementation diversity.
- Framework: Workflow-based verification constrains agent execution through structured test workflows and verification nodes while assessing functional correctness and visual fidelity together.The design aims to preserve agent interaction flexibility and alignment with human visual preferences.
- Workflow Design: Vision2Web represents end-to-end testing as a directed dependency graph whose nodes are verification procedures and whose edges encode sequential dependencies and shared application states.Agent-executable subgraphs instantiate coherent interaction trajectories under shared application contexts.
- Workflow Design: Workflows decouple dependent test nodes to reduce error propagation and integrate related nodes operating within the same application context to improve coverage efficiency.Separating modules prevents earlier failures from obscuring later components.
- Verification Nodes: Functional verification uses GUI agents guided by objectives, actions, and validation criteria, with explicit actions limiting unnecessary exploration and unintended behavior.Functional scores are produced by GUI-agent verification under preserved application state and prior workflow context.
- Verification Nodes: Visual verification uses a VLM judge to compare rendered pages with reference prototypes through component-level rubrics, averaging block-level fidelity scores into a Visual Score.Visual Score is calculated as the average score across prototypes.
- Annotation: Workflow annotation proceeds from expert drafts to Claude Code refinement and human quality control, with validation grounded in observable, implementation-agnostic user behavior.Full-stack workflows cover long-horizon dependencies, cross-module interactions, persistent states, and boundary cases.
- Annotation: Test-case distributions across full-stack tasks highlight diverse functional scenarios and resulting evaluation coverage.The figure is presented as evidence of scenario diversity across website-level tasks.
4. Experiments
Vision2Web evaluates multimodal coding agents across hierarchical website-development tasks, revealing systematic performance gaps in complexity, device adaptation, model capability, framework choice, website category, and system integration. Its workflow-based verification is validated as broadly reliable and its rankings are robust across verifier backbones.
- Main Results: Performance degrades consistently as tasks progress from static webpages to interactive frontends and full-stack websites.Under OpenHands, Gemini-3-Pro-Preview falls on full-stack tasks to VS 11.7, FS 22.6, and DSR 77.8%.
- Main Results: 10–20% lower scores on tablet and mobile layouts show that smaller device form factors reduce visual fidelity.Larger, denser prototype images induce additional performance declines.
- Main Results: Claude-Opus-4.5 achieves the strongest overall performance, while Seed-1.8-VL fails full-stack tasks and Qwen models largely cannot complete multimodal coding tasks.Under OpenHands, Claude-Opus-4.5 reaches VS 38.4 and FS 57.6 on full-stack tasks.
- Main Results: Across most models, OpenHands tends to outperform Claude Code, indicating that framework choice affects agent performance.The comparison excludes Claude models.
- Main Results: Public Service websites yield the strongest full-stack results, whereas SaaS platforms yield the weakest because of greater structural and interaction complexity.Content and Transaction websites show moderate performance.
- Failure Analysis: Agents are most reliable on Navigation & Routing and Authentication & Authorization, but struggle with State Management, CRUD Operations, and File & Media Operations.The harder categories require persistent state tracking, cross-component data flow, or coordination with system-level resources.
- Failure Analysis: Visual errors include misaligned layouts, incorrect sizing, color mismatches, and fragile asset handling, especially for regularly arranged or ambiguously named components.These failures occur even on static webpages.
- Failure Analysis: Visual and functional fidelity degrades across subsequent pages as agents struggle to maintain coherent reasoning across modules and views.Observed failures include missing components, malfunctioning interactions, and broken navigation links.
5. Related Work
Related benchmarks have progressed from localized code generation and issue resolution toward end-to-end development, but remain limited in visual-centric, hierarchical, and rigorous evaluation of multimodal coding agents.
- Visual Website Development: UI-to-code benchmarks map visual layouts to executable code, progressing from synthetic datasets and one-shot screens toward real-world collections and broader settings.Examples include Design2Code, Web2Code, Flame-React, WebSight, and WebCode2M.
- Autonomous Coding Agents: Autonomous coding agents evolved from single-shot generation to interactive systems with planning, tool use, retrieval, and execution feedback.The cited progression includes Self-Planning, CodeChain, CodeAct, ToolCoder, and CodeAgent.
- Evaluation Benchmarks: Code-generation evaluations expanded from file- and function-level correctness to real-world software development and from-scratch project benchmarks.HumanEval, MBPP, APPS, LiveCodeBench, SWE-Bench, WebGen Bench, and VIBE Bench represent this progression.
- Evaluation Benchmarks: Existing benchmarks still lack sufficient visual-centric tasks, structured hierarchical inputs, and fine-grained end-to-end evaluation for cross-modal reasoning.These limitations motivate Vision2Web’s benchmark design.
6. Conclusion
Vision2Web provides a hierarchical benchmark and workflow-based verification paradigm for assessing multimodal coding agents in visual website development. Experiments show that isolated-task strength does not reliably transfer to end-to-end construction, especially under structural, cross-page, and persistent-state demands.
- Conclusion: Vision2Web organizes visual website-development tasks into three hierarchical levels and combines a GUI agent verifier with a VLM-based judge.The design measures functional correctness and visual fidelity under increasing task complexity.
- Conclusion: Strong performance on isolated tasks does not reliably transfer to end-to-end system construction.The benchmark reveals deficiencies in structural complexity, cross-page coordination, and persistent-state reasoning.
- Conclusion: The findings support progressively challenging task designs and reproducible autonomous evaluation for rigorously assessing coding-agent capabilities.This conclusion is stated as a direction for future evaluation practice.
Limitations
Vision2Web’s evaluation has residual automation errors, narrower software-engineering scope, and higher verification costs than code-only benchmarks.
- Automated GUI-agent execution and VLM-based visual judgment may introduce residual errors despite human validation and verifier-sensitivity analyses.
- Vision2Web evaluates end-to-end website development rather than the full spectrum of software engineering tasks.
- Realistic multimodal evaluation costs more than code-only metrics because it verifies deployed behavior, visual fidelity, and user-facing functionality.
Impact Statement
Vision2Web uses publicly accessible resources for academic research in a reproducible, controlled evaluation framework without private, sensitive, or personal information.
- All benchmark data comes from publicly accessible websites and openly available resources and is used solely for academic research.
- The benchmark excludes private, sensitive, and personal information and is intended to support reproducible, controlled evaluation.
A.1. Benchmark Details and Statistics
Vision2Web organizes website diversity into a professional taxonomy and distributes tasks across website categories and hierarchical development levels to balance coverage.
- Website taxonomy: The taxonomy groups websites into Content, Transaction, SaaS Platforms, and Public Services, with subcategories and representative examples.
- Website taxonomy: Table 7 summarizes website categories, subcategory definitions, and representative examples.
- Task distribution: Table 8 distributes tasks across website categories and three hierarchical development levels to balance website-type and software-engineering-complexity coverage.
A.1.2. TASK-LEVEL STATISTICS
Vision2Web characterizes task features across development levels, with static webpage tasks covering broader prototype sizes and greater difficulty than traditional Design2Code benchmarks.
- Static webpage tasks: Static webpage tasks in Vision2Web are more difficult than traditional Design2Code benchmarks and cover a wider range of prototype sizes.
- Prototype images: Figure 5 shows the distribution of prototype image sizes across different device types.
- Task complexity: Figure 6 compares task complexity metrics between Design2Code-Hard and Vision2Web webpage tasks.
- Task distribution: Table 8 presents task distributions across website categories and development levels.
A.1.3. REPRESENTATIVE TASK EXAMPLES
Vision2Web illustrates website-development tasks across static, interactive, and full-stack levels, using representative prototypes, multi-page requirements, and workflow-based verification. The examples pair task objectives and validation criteria with GUI interaction and visual comparison procedures.
- Task levels: The benchmark spans cross-device responsive static webpages, interactive frontend applications, and full-stack websites.These examples correspond to progressively broader website-development requirements.
- Interactive frontend example: The Airbnb Community Center example specifies multiple pages, navigation structures, authenticated and unauthenticated flows, and social interactions.Supported functions include browsing, filtering, posting, replying, liking, and profile access.
- Workflow verification: Workflow instances execute sequential objectives, guided actions, and validation criteria within one shared browser session.The workflow format covers both positive and negative scenarios for form fields and navigation outcomes.
- Workflow verification: The post-creation workflow verifies valid submission, navigation to the new post, and matching title and content.The workflow also checks that empty required content prevents submission and displays an error.
- Agent verification: The GUI verifier operates through predefined actions and screenshots, while the VLM judge compares prototype and actual-page images by segmented UI components.The judge assigns component-level scores for visual alignment, layout, typography, colors, icons, and images.
- Evaluation setup: Evaluation prompts enumerate inputs, specify task-completion expectations, and discourage premature termination or unnecessary over-engineering.The full-stack task prompt frames implementation and deployment as a senior engineering task.
A.4.1. ILLUSTRATIVE FAILURE CASES
Vision2Web’s representative failure cases show progressively more demanding weaknesses across static, interactive, and full-stack development. As complexity and dependencies increase, errors move from visual inaccuracies to cross-page incoherence and system-level functional breakdowns.
- Static webpage tasks: At lower task levels, agents misground fine-grained visual details into layout and styling decisions, producing misalignment and visual inconsistencies.The limitation is directly associated with precise visual reproduction.
- Interactive frontend tasks: As scope expands across components and pages, errors compound through insufficient cross-page state tracking and weak integration of visual and textual cues.These failures reflect increasing cross-module and cross-page dependencies.
- Full-stack website tasks: At the system level, unreliable self-verification and planning lead to accumulated specification deviations, functional-correctness failures, and unstable execution.The failure pattern becomes broader as full-stack requirements demand long-horizon coordination.
- Cross-level pattern: Agent performance systematically degrades as task complexity and dependency structure increase.The analysis traces this degradation from visual reproduction through cross-page coherence to full-system execution.