PaperlayerView exact revision on arXiv

Source-linked AI summary

ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation

Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Changzhi Zhou, Ken Deng, Dengpeng Wu, Guanhua Huang, Kejiao Li, Qi Yi, Ruibin Xiong, Shihui Hu, Yue Zhang, Yuhao Jiang, Zenan Xu, Yuanxing Zhang, Wiggin Zhou, Chayse Zhou, Fengzong Lian

arXiv:2507.04952v2cs.CLcs.SE

TL;DR

Existing benchmarks do not holistically evaluate visual fidelity and interactive behavior in generated artifacts. ArtifactsBench couples a diverse task set with staged execution evidence and checklist-guided MLLM judging. The evaluation aligns with WebDev Arena and human experts, while analyses find that instruction-tuned generalists often outperform specialist models.

  • Problem

    Existing benchmarks emphasize static correctness or non-visual functionality and do not holistically measure visual fidelity and interactive behavior, qualities central to user experience and real-world adoption.

    Source: §1 · p. 1

  • Method

    ArtifactsBench evaluates executable artifacts using staged screenshots and a task-specific, ten-dimension checklist that guides MLLM judging of visual, interactional, and code qualities.

    Source: §1 · p. 1, §3 · p. 3, §3.1 · p. 4

  • Results

    Instruction-tuned generalists often outperform specialist coder/VL models, and ArtifactsBench achieves 94.4% ranking consistency with WebDev Arena.

    Source: §1 · p. 2, §4.2 · p. 5

  • Takeaways & Limitations

    ArtifactsBench offers diagnostics on model strengths and failure modes for evaluating interactive visual artifact generation.

    Source: §6 · p. 9

    Fixed-interval screenshot sampling may not fully capture the fluidity, correctness, and robustness of highly complex, long-horizon, or state-dependent interactions.

    Source: §A.1 · p. 12

Abstract

from arXiv · show

The generative capabilities of Large Language Models (LLMs) are rapidly expanding from static code to dynamic, interactive visual artifacts. This progress is bottlenecked by a critical evaluation gap: established benchmarks focus on algorithmic correctness and are blind to the visual fidelity and interactive integrity that define modern user experiences. To bridge this gap, we introduce ArtifactsBench, a new benchmark and paradigm for the automated, multimodal evaluation of visual code generation. Our framework programmatically renders each generated artifact and captures its dynamic behavior through temporal screenshots. This visual evidence, alongside the source code, is then assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a fine-grained, per-task checklist to ensure holistic and reproducible scoring. We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading LLMs. Our automated evaluation achieves a striking 94.4% ranking consistency with WebDev Arena, the gold-standard for human preference in web development, and over 90% pairwise agreement with human experts. This establishes ArtifactsBench as the first framework to reliably automate the assessment of human-perceived quality at scale. Our analysis provides a high-resolution map of the current SOTA, revealing that generalist models often outperform domain-specific ones. We open-source ArtifactsBench, including the benchmark, evaluation harness, and baseline results at https://artifactsbenchmark.github.io/, to provide the community with a scalable and accurate tool to accelerate the development of user-centric generative models.

1 Introduction

ArtifactsBench addresses the lack of scalable, holistic evaluation for interactive visual artifacts, which require assessing visual fidelity and dynamic behavior as well as static correctness. It combines a diverse benchmark with checklist-guided automated judging, reporting strong alignment with human judgments and WebDev Arena.

1 Introduction

  • Existing benchmarks miss holistic quality across visual presentation and dynamic behavior, leaving evaluation to costly manual inspection or unreliable LLM self-evaluation.The gap matters because these qualities are central to user experience and affect real-world adoption.Source: §1 · p. 1
  • ArtifactsBench provides 1,825 tasks across nine domains, stratified by difficulty, and combines granularity, human consistency, automation, and direct visual evaluation.The benchmark is presented as the first large-scale, hierarchical benchmark for interactive visual artifacts.Source: §1 · pp. 1–2, Table 2 · p. 2
  • Its evaluation pipeline executes artifacts, captures three sequential interaction screenshots, and uses task-specific checklists with an MLLM referee to score visual, interaction, and code qualities.The dual-referee setup uses Gemini-2.5-Pro and Qwen2.5-VL-72B.Source: §1 · p. 2
  • 94.4% ranking consistency with WebDev Arena and up to 90.95% pairwise agreement in a 280-instance expert study support the evaluation’s human alignment.The analysis also finds systematic weaknesses on intensive-interactive tasks and that instruction-tuned generalists often outperform specialist models.Source: §1 · p. 2

2 ArtifactsBench: A Benchmark for Visual Code Generation

ArtifactsBench contains 1,825 executable queries spanning nine topics, with tasks organized by difficulty. Its construction pipeline curates, calibrates, validates, and standardizes tasks and their evaluation checklists.

2.1 Overview

  • The benchmark covers nine topics, including games, SVG generation, web applications, simulations, data science, and management systems, with Easy/Medium/Hard tiers.Difficulty is assigned post hoc from aggregated performance across 30+ LLMs to retain discriminative power.Source: §2.1 · p. 2

2.2 Dataset Construction Pipeline

  • Task sourcing combines expert showcases, open SVG/web-snippet datasets, web case studies, and LLM-generated queries; filters remove incomplete, non-visual, non-interactive, or otherwise unsuitable items.The pipeline prioritizes observable dynamics such as state changes, animations, and responsiveness.Source: §2.2 · pp. 2–3
  • Two-stage deduplication checks prompt, checklist, and normalized code similarity, then screenshot perceptual hashes; flagged items are re-authored or discarded.Experts and LLMs rewrite prompts, with humans verifying them; the realized difficulty split is 559/611/655 and remains stable under leave-family-out resampling.Source: §2.2 · p. 3
  • Each query receives a human-refined, 10-dimension checklist covering visual, interactional, and code qualities, calibrated against a manually curated seed with Cohen’s κ ≥0.8.The ten items are grouped into five vision-oriented and five code-oriented dimensions.Source: §2.2 · p. 3
  • Solvability checks prune underspecified tasks and revise prompts that permit brittle hacks, favoring multiple acceptable but checkable implementations.Tasks are tested using candidate solutions from multiple model families.Source: §2.2 · p. 3
  • A standardized headless Chromium harness captures three staged screenshots at 1024 × 768, while the 1,825-query benchmark retains static items when visual fidelity or structured graphics are the goal.The target Easy/Medium/Hard split of 30/40/30 materializes as 559/611/655.Source: §2.2 · p. 3

3 Evaluation Methodology

ArtifactsBench evaluates artifacts through sandboxed execution, staged screenshots, and checklist-guided judging, first validating the referees against human judgments and then comparing model rankings with WebDev Arena. Its checklist and dual-referee design supports code-aware scoring and reproducibility.

3 Evaluation Methodology

  • The evaluation validates its automatic referee in a 280-instance, six-model human study, then scores all models and checks ranking alignment with WebDev Arena across 1,825 queries.The framework targets stable per-query judgments and model rankings that agree with human preferences.Source: §3 · p. 3, Figure 3 · p. 4

3.1 Fine-Grained Checklists

  • Ten bespoke checklist dimensions score qualities such as functionality, robustness, engineering practices, creativity, aesthetics, and user experience on 10-point scales.Five vision-oriented dimensions assess rendered appearance and interaction affordances; five code-oriented dimensions assess logic, robustness, engineering, scalability, and redundancy.Source: §3.1 · p. 4

3.2 Automated Evaluation and Referees

  • The protocol combines three staged screenshots with the task, full model answer, and checklist so the referee can assess visual evidence and code properties that screenshots cannot reveal.Those code properties include logic correctness, input robustness, modularity, and redundancy.Source: §3.2 · p. 4
  • Qwen2.5-VL-72B and Gemini-2.5-Pro produce highly consistent partial-order constraints, and replacing the deprecated Gemini preview with the stable version leaves conclusions unchanged.The stronger referee provides finer resolution among top systems.Source: §3.2 · p. 4

4 Experiments

The experiments compare more than 30 models under controlled rendering and judging conditions, then assess benchmark performance, diagnostic patterns, and agreement with human preference. ArtifactsBench aligns more closely with WebDev Arena rankings than WebBench does.

4.1 Settings

  • Evaluations covered 30+ models using identical instructions, deterministic seeds, fixed rendering settings, and three screenshots before, during, and after interaction.The dual-referee protocol analyzes performance across interactivity levels and task categories.Source: §4.1 · pp. 4–5

4.2 Main Results and Analysis

  • Gemini-2.5-Pro achieves the highest overall score across open-source and proprietary evaluations, while Claude 4.0-Sonnet approaches state-of-the-art performance.Table 3 reports results across interaction levels and task categories, with an overall average.Source: Table 3 · p. 5, §4.2 · p. 5
  • Performance scales with model size and deliberation; DeepSeek-R1-0528 leads open-source models, while distillation gains 3 points over Qwen-2.5-32B yet trails Qwen3-32B by 5.Source: §4.2 · pp. 5–6
  • All models score lowest on Intensive Interactive and complex Management System tasks, while Qwen-2.5-Instruct outperforms coder-specialized and vision-language counterparts.Source: §4.2 · p. 6

4.3 Fine-grained Analysis

  • Vision- and code-oriented scores are strongly correlated, and model rankings remain stable across difficulty tiers even as the hardest subset challenges top models.The analysis groups ten checklist items into five vision-oriented and five code-oriented criteria.Source: §4.3 · p. 6, Figure 4 · p. 6, Figure 5 · p. 6
  • Execution screenshots and code-aware judging improve agreement; longer responses broadly correlate with scores, but concise, structured outputs remain competitive.Multiple screenshots better capture dynamics; descriptions help but remain inferior to images when visual capability is strong, and removing the answer degrades quality.Source: §4.3 · pp. 6–7, Figure 6 · p. 7, Table 4 · p. 7
  • The two referees induce highly consistent ranking constraints, and replacing the deprecated Gemini preview with the stable version preserves ordering constraints.Residual referee differences concentrate among top systems, where the stronger referee offers finer resolution.Source: §4.3 · p. 7

4.4 Alignment with WebDev Arena

  • 94.4% normalized Footrule consistency with WebDev Arena versus 69.4% for WebBench; ArtifactsBench aligns more closely with human preference.The authors describe WebDev Arena as a human-preference gold standard.Source: Figure 7 · p. 8, Figure 8 · p. 8, §4.4 · p. 8

5 Related Work

Related work spans code correctness, screenshot replication, DOM alignment, development processes, and human-preference evaluation. ArtifactsBench complements these approaches with checklist-guided assessment of live, operable artifacts and safeguards for multimodal judging.

5 Related Work

  • Prior evaluation approaches address algorithmic correctness, screenshot replication, DOM alignment, web-development processes, or human preference, but not holistic live-artifact dynamics.Source: §5 · p. 8
  • ArtifactsBench evaluates operable artifacts, structured graphics, and generated dynamic scenes with checklist-driven snapshot checks for required state transitions and interactive feedback.The protocol aims to provide compact, reviewable evidence for consistent judging across models and tasks.Source: §5 · pp. 8–9
  • DOM or pixel similarity can miss semantics and interaction flow, while MLLM judges risk hallucination and prompt sensitivity; temporal snapshots, tailored checklists, and dual referees are safeguards.Source: §5 · p. 9

6 Conclusion

ArtifactsBench evaluates LLMs on dynamic visual artifacts through a difficulty-calibrated task set and automated checklist-guided scoring of code, visuals, and interaction. Across 30+ models, its evaluations align with expert judgments and WebDev Arena while providing diagnostics on strengths and failure modes.

6 Conclusion

  • ArtifactsBench combines 1,825 difficulty-calibrated tasks with checklist-guided MLLM scoring across code, visuals, and interaction, and reports alignment with expert judgments and WebDev Arena.Source: §6 · p. 9

7 Contributions and Acknowledgements

The acknowledgements identify first authors, algorithm-support contributors, and corresponding authors.

First Autors

Core Contributors — Algorithm Support

Corresponding Authors

A.1 Limitations and Future Work

ArtifactsBench identifies limits in how it evaluates dynamic behavior and in the development setting it covers. Its screenshot sampling may miss complex interactions, and it evaluates final artifacts generated in a single turn rather than iterative agentic development.

Deepening the Evaluation of Interactivity

  • Fixed-interval screenshot sampling may miss the fluidity, correctness, and robustness of long-horizon or state-dependent interactions.Examples include multi-step web workflows and nuanced game physics.Source: §A.1 · p. 12
  • The authors identify richer dynamic analysis, including DOM-based state checks or video-based session analysis, as possible future extensions.Source: §A.1 · p. 12

Exploring Agentic and Iterative Development

  • The benchmark evaluates a final artifact generated in one turn, not iterative refinement, self-debugging, or multi-step development.The paper proposes a multi-turn environment with feedback from a user, linter, or debugger as a future direction.Source: §A.1 · p. 12

A.2 Detailed Analysis of the Results

The analysis finds that generalist models can outperform specialized counterparts, while leading proprietary models retain an overall advantage. Performance also varies with model scale, task difficulty, and interaction demands.

Proprietary multimodal models demonstrate a clear advantage

  • Gemini-2.5-Pro achieves the highest overall score in both open-source and proprietary human evaluations, while Claude 4.0-Sonnet approaches state-of-the-art performance.Figure 9 summarizes overall scores across leading models.Source: §A.2 · p. 12, Figure 9 · p. 13

Performance scales with model size and deliberation time

  • Within Qwen2.5 and Qwen3, performance rises with model capacity, and models using extended inference tend to score higher.Source: §A.2 · p. 12

Analysis of open-source model performance

  • DeepSeek-R1-0528 leads the open-source contenders; DeepSeek-R1-Distill-Qwen-32B scores 3 points above Qwen-2.5-32B but 5 below Qwen3-32B.The authors suggest distillation on limited data may not impart robust, generalizable visual-code synthesis skills.Source: §A.2 · p. 12

Opportunities remain in challenging scenarios

  • Models score lowest on Intensive Interactive tasks and complex project-level visualizations such as Management System cases.Source: §A.2 · p. 12

Generalist skills outperform specialist expertise

  • Qwen-2.5-Instruct surpasses both Qwen-2.5-coder and Qwen2.5-VL-72B, suggesting visual artifact generation draws on more than isolated coding or vision skills.The authors associate this broader capability with reasoning, instruction following, and design aesthetics.Source: §A.2 · p. 12

Visual and Code-Based Analysis

  • Vision-oriented and code-oriented checklist scores are positively correlated, so assessing only one can miss strengths or weaknesses in the other.The analysis reports that model improvements tend to be comprehensive rather than isolated to one capability.Source: §A.2 · p. 13

Difficulty Analysis

  • Even the best-performing models remain below 50 points on the hardest tier, while rankings stay consistent and all three difficulty tiers remain discriminative.Source: §A.2 · p. 13

A.3 Validation with Human Experts

The appendix tests evaluation choices and referee consistency, and documents the benchmark’s query filtering, classification, and scoring prompts. It reports that visual evidence—especially multiple screenshots—improves referee alignment with human judgments.

A.3 Validation with Human Experts

  • The human study sampled 280 queries across six models, used experienced engineers under double-blind randomized scoring, and compared rank orderings through pairwise consistency.Pair ACC is the proportion of response pairs for which the referee and human judges agree on which response ranks higher.Source: §A.3 · pp. 13–14
  • Gemini-2.5-Pro-Preview-0506 and Qwen2.5-VL-72B produce largely consistent ranking constraints, while differences concentrate among top-tier models; stable Gemini preserves the preview’s pairwise orderings.The reported version comparison says replacing the preview does not change leaderboard conclusions.Source: §A.3 · p. 14, Figure 10 · p. 14, Figure 11 · p. 15
  • Including execution screenshots improves pair accuracy, and multiple screenshots further help capture dynamic effects; removing the answer from the input reduces evaluation effectiveness.Text descriptions of screenshots can also improve accuracy, while a referee with strong visual analysis can benefit from direct image input.Source: §A.3 · pp. 14–15

A.4 Reproducibility and future directions

  • The release includes dataset specifications, checklist templates, evaluation scripts, and referee settings, with future work targeting richer dynamics, agentic development, and broader artifact domains.Source: §A.4 · p. 15

A.5 Evaluation Results from Additional Scoring Referees

  • Gemini-2.5-Pro and Qwen2.5-VL-72B yield highly consistent model rankings, though their fine-grained scores can differ on complex interactive tasks.The tables report results across interactivity levels and task categories, with AVG as the global average.Source: §A.5 · p. 15, Table 6 · p. 16, Table 7 · p. 16

A.6 Quality Filtering of Queries

A.7 Classification of Queries

A.8 The Prompt for Model Scoring

  • Checklist generation takes a query as input and receives manual review, while final scoring takes the query, answer, and checklist as input.The appendix presents separate prompts for checklist generation and final scoring.Source: §A.8 · p. 17, Figure 15 · p. 20, Figure 16 · p. 21

Related papers

Loading 2507.04952v2…