PaperlayerView exact revision on arXiv

Source-linked AI summary

WebUIProof: Benchmarking WebUI Code Generators with UI-Agent Execution Harness

Yun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang

arXiv:2610.02617v1cs.SEcs.AIcs.MA

TL;DR

WebUI code-generation evaluation needs to assess rendered interfaces and their behavior, not only compilation or static appearance. WebUIProof combines structured task specifications, executable interaction tests, and a UI-agent harness, and its evaluation finds interaction failures across domains while VisRL improves compact models over SFT and vision-only baselines.

  • Problem

    Prior WebUI benchmarks often use free-form prompts or static checks, leaving interactive functional correctness incompletely evaluated.

    Source: §1 · p. 1

  • Method

    WebUIProof pairs structured specifications with executable interaction tests run by a UI agent, and VisRL trains compact models with composite execution, visual-quality, and test-success rewards.

    Source: §1 · pp. 1–2, §3.4 · p. 5

  • Results

    Interactive WebUI generation remains challenging in general and 3D domains, with substantial headroom on 3D tasks; VisRL improves functional completion and reliability over SFT and vision-only baselines.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The UI-agent harness can provide outcome-level training signals for improving functional completion and reliability beyond static evaluation.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Evaluating WebUI code generation at scale is difficult: outputs may compile and look plausible yet fail under user interaction, and prior benchmarks largely rely on free-form prompts with static checks (build success, screenshots) that miss functional correctness. We introduce WebUIProof, an execution-oriented benchmark that provides structured specifications and dense, executable interaction tests for WebUI generation across two task families: general WebUIs (e.g., dashboards, game, interactive tools) and 3D interactive simulation (e.g., particle/galaxy systems, physics dynamics). WebUIProof includes a UI-agent harness that runs executable interaction tests in a headless browser using an iterative plan--act--observe loop: it locates DOM elements, performs actions, observes resulting UI/DOM changes, and checks the specified assertions. We evaluate across eight commercial LLMs and observe frequent failures on interaction-based requirements even when pages render successfully, especially on 3D simulation interfaces. Finally, we show the UI-agent harness can provide outcome-level training signals. Training compact models (e.g., Qwen2.5 14B and MIMO 7B) with RL rewards derived from executable interaction tests improves functional completion while reducing build failures.

1 Introduction

WebUIProof targets functional WebUI correctness that free-form prompts and static checks do not fully assess, pairing structured specifications with executable interaction tests. Its UI-agent harness evaluates general and 3D interfaces, and the same execution-based approach supplies training signals that improve compact models over SFT.

1 Introduction

  • Prior WebUI benchmarks use free-form prompts, human preference, incomplete test coverage, or static checks that do not verify interactive behavior.The introduction notes that build success, lint/type checks, and screenshots are insufficient for validating interactions.Source: §1 · p. 1
  • WebUIProof contains 219 tasks and 3,000 executable interaction tests across general WebUIs and 3D simulation interfaces.Tasks pair website summaries and UI-element requirements with executable tests; the harness plans actions, observes UI/DOM changes, and verifies assertions.Source: §1 · p. 2
  • VisRL uses the harness to derive outcome-level training signals from rendered UI and interactive tests, alongside build/runtime and visual-quality rewards.The paper also uses multi-temperature sampling and fast-gate filtering to improve rollout efficiency and stabilize learning under costly evaluation.Source: §1 · p. 2
  • An agent-assisted pipeline produces structured specifications and element-aligned executable tests at scale.The pipeline collects webpage visual snapshots and HTML/DOM structure before generating summaries and test recommendations.Source: §1 · p. 2, Figure 1 · p. 2
  • Across eight commercial LLMs, many outputs build and render but fail interaction requirements, especially on 3D simulation interfaces.The benchmark evaluates functional correctness through DOM grounding, interaction execution, and assertions.Source: §1 · p. 2
  • VisRL improves accuracy over SFT by 15.7% on general WebUI tasks and 4.9% on 3D simulation tasks.The approach trains compact models including Qwen2.5 14B Instruct and MIMO 7B.Source: §1 · p. 2

2 Related Works

Prior work spans repository-level software engineering, frontend generation benchmarks, and UI-agent evaluation, but differs in task framing and evaluation signals. WebUIProof pairs structured specifications with executable tests and uses DOM-grounded interaction to verify assertions.

Benchmark for Frontend Code Generation

  • WebUIProof evaluates end-to-end WebUI generation through buildability, visual fidelity, and UI-agent-tested interactivity, unlike repository-level editing and bug-fixing systems.FrontendBench standardized a generation, rendering, and automated-test pipeline with agent feedback loops.Source: §2 · p. 2
  • WebUIProof’s instance format pairs a website summary and required UI elements with executable interaction tests.The construction pipeline collects visual snapshots and HTML/DOM structure from seed webpages, then produces summaries and test recommendations with element-to-test alignment.Source: Figure 2 · p. 3
  • WebDev-Arena relies on human preference for loosely structured prompts, while DesignBench scores screenshot similarity and code-modification similarity without behavioral assertions.WebGen-Bench uses executable web tasks, whereas WebUIProof adds structured specifications and DOM-localized actions to verify test assertions.Source: §2 · p. 3

Web-based Testing with UI Agents

  • The evaluator builds on UI-agent and browser-testing work, implementing an approach based on WebVoyager for general WebUI generation and 3D simulation.The cited context includes headless-browser testing with DOM inspection and event simulation, plus standardized web-agent environments.Source: §2 · p. 3

3 Method

WebUIProof pairs structured WebUI specifications with executable interaction tests across general interfaces and 3D simulations, scored by a fixed UI-agent protocol. Its construction pipeline expands tests through interaction-driven exploration, while VisRL uses execution, visual, and interaction outcomes as training rewards.

3.1 WebUIProof Benchmark Overview

  • WebUIProof covers 219 tasks across 149 general WebUIs and 70 3D simulation WebUIs, including dashboards, games, interactive tools, and simulation scenes.Seed prompts come from WebGen-Bench and WebDev-Arena; 3D tasks draw on Three.js examples and public demo scenes.Source: §3.1 · pp. 3–4
  • Each benchmark instance combines a website summary, required UI elements with stable identifiers and locator hints, and executable test cases with preconditions, procedures, and assertions.Tests cover functionality, data display, and design validation.Source: §3.1 · p. 4

3.2 UI-Agent Evaluation Protocol

  • The fixed UI-agent harness runs each test in a Plan→Action→Observation loop, checks assertions, and aggregates outcomes into Full, Partial, or No completion.The same protocol is applied to every model and task.Source: §3.2 · p. 4

3.3 Benchmark Construction Pipeline

  • The construction pipeline starts from webpage snapshots and DOM to draft summaries and seed tests, then interactively explores dynamic states and updates element requirements and tests.It synthesizes cases from trajectories, checks element coverage, adds tests for uncovered elements, and deduplicates near-identical cases.Source: §3.3 · pp. 4–5

3.4 VisRL: RL Training with Executable Harness

  • VisRL trains compact models with composite rewards for build/runtime execution, rendered visual quality, and UI-agent interaction-test success.Single-signal rewards are described as insufficient: interaction-only optimization can yield minimal functional UIs, while visual-only optimization does not guarantee correct behavior.Source: §3.4 · p. 5
  • VisRL uses multi-temperature sampling to diversify candidates within a fixed best-of-N budget and fast-gate filtering to skip obviously invalid generations before costly execution.The paper reports that removing either strategy increases build failures and reduces accuracy.Source: §3.4 · pp. 5–6

4 Experiment

The evaluation applies a fixed UI-agent harness to commercial and compact models on general and 3D WebUI tasks. Commercial models leave substantial functional headroom, while VisRL improves compact-model accuracy over SFT on both task families.

4.1 Experimental Setting

  • The evaluation uses a frozen Playwright-based UI-agent harness with a Qwen2.5 VL 72B evaluator, headless 1280×800 browser, and 15-step per-task budget.The models include eight state-of-the-art LLMs and two compact training bases, Qwen 2.5 14B Instruct and MIMO 7B.Source: §4.1 · p. 6
  • VisRL training uses Qwen 2.5 14B Instruct as its base, with SFT cold start followed by PPO with LoRA adapters; training data includes 6,000 WebGen-Bench prompts and 1,600+ Three.js prompts.The paper also compares VisRL with pure SFT and PPO with a vision agent.Source: §4.1 · pp. 6–7

4.2 Experimental Results

  • Claude Sonnet 4 leads commercial models at 46.10% accuracy on General WebUI and 40.24% on 3D simulation WebUIs, while the reported top models still show substantial headroom.On General WebUI, Claude Sonnet 4 has a 30.37% build failure rate; on 3D, its build failure rate is 0%.Source: §4.2 · p. 7, Table 3 · p. 7
  • VisRL improves Qwen 2.5 14B Instruct from 26.08% to 41.80% accuracy on General WebUI and from 32.44% to 37.38% on 3D simulation WebUIs.For MIMO 7B, accuracy rises from 23.80% to 33.43% on General WebUI and from 17.12% to 20.10% on 3D simulation WebUIs.Source: §4.2 · p. 7

5 Ablation Studies

The ablations examine evaluator agreement and step-budget effects, benchmark test coverage and instruction complexity, and VisRL learning strategies. They report that agreement improves with more steps, structured specifications provide denser coverage, and multi-temperature sampling and fast-gate filtering support training performance.

5.1 Evaluator Reliability and Failure Modes

  • Human–agent agreement exceeds approximately 80% across three model families on 100 General WebUI samples, while larger evaluator step budgets improve agreement and reduce timeouts.Remaining disagreements are attributed primarily to interaction failures and element-locator failures.Source: §5.1 · p. 8, Table 5 · p. 8

5.2 Analysis on Benchmark Quality

  • As instruction complexity rises across nine sequential steps, accuracy gradually drops for frontier models and follows a similar trend for the self-trained models.ICI combines required element/test size and interaction actions; its score increases to about 68 in the described analysis.Source: §5.2 · p. 8, Figure 4 · p. 9
  • Structured specifications raise test coverage from 30.71% to 97.23% compared with free-form prompts, and the resulting accuracy drop indicates a more challenging benchmark.Coverage counts required UI elements referenced as interaction targets or assertions.Source: §5.2 · p. 8, Figure 5 · p. 9

5.3 Analysis of Learning Strategies in VisRL

  • Removing either multi-temperature sampling or fast-gate filtering increases build failures and reduces overall accuracy; at the same best-of-N count, multi-temperature sampling yields higher accuracy than single-temperature sampling.The authors attribute the sampling advantage to diversity rather than simply more attempts.Source: §5.3 · p. 9, Table 6 · p. 9

5.4 Visualization of Tasks and UI-Agent Execution Traces

  • The execution-trace examples span general WebUIs and 3D simulations, with colored boxes and index labels marking localized task-relevant elements.The visualizations make the agent’s executed interaction targets explicit.Source: §5.4 · p. 9, Figure 6 · p. 9

6 Conclusion

WebUIProof evaluates WebUI generation with structured specifications, executable interaction tests, and a reproducible UI-agent harness across general and 3D tasks. The authors report that interactive generation remains challenging, especially in 3D, and that VisRL improves compact models over SFT and vision-only baselines.

6 Conclusion

  • WebUIProof pairs structured specifications with executable interaction tests across general WebUIs and 3D simulation interfaces.Its reproducible UI-agent harness supports automated evaluation beyond build success or static visual checks.Source: §6 · p. 9
  • Interactive WebUI generation remains challenging in both domains, with substantial headroom on 3D tasks.VisRL improves functional completion and reliability compared with SFT and vision-only baselines; the ablation identifies complex instructions and multi-step requirements as continuing bottlenecks.Source: §6 · p. 10

Appendices

The appendices document the evaluation protocol, benchmark construction, VisRL training, analyses, and extended results. They provide implementation and evaluator details, task-source and category information, and descriptions of additional diagnostics.

A. Reproducibility and Evaluation Protocol

B. WebUIProof Benchmark Construction

  • The benchmark-construction appendix covers task scope, category and test-case distributions, the collection pipeline, and structured-specification examples.Its listed data sources and scope include both general and 3D WebUI tasks.Source: B. WebUIProof Benchmark Construction · p. 12

C. VisRL: Training Details

  • The VisRL appendix lists implementation details, sequence-level rollouts, reward definition, learning strategies, and execution procedures.It also provides algorithm and implementation details for sequence-level PPO, reward computation, candidate selection, and efficiency heuristics.Source: C. VisRL: Training Details · p. 12

D. Reward and Hyperparameter Analyses

E. Extended Results and Diagnostics

  • The extended-results appendix includes per-test-category outcomes, UI-agent traces, additional WebUI visualizations, and build-failure examples.These are presented alongside the reward and hyperparameter analyses in the appendix overview.Source: E. Extended Results and Diagnostics · p. 12

A.1 Execution Environment

  • Evaluation transforms generated React code into a Next.js app, builds a static export, and serves it for browser rendering.The described process extracts and sanitizes code, scaffolds the project, installs dependencies, and runs the build.Source: §A.1 · p. 13

A.2 UI-agent Evaluator Setup

  • The UI-agent evaluator uses WebVoyager with Selenium, combining screenshots and DOM inspection in an iterative action loop.The agent selects predefined actions, observes updated UI state, and assesses completion, with a maximum of 15 iterations; the appendix also lists its action space.Source: §A.2 · p. 13, Table 7 · p. 14

A.3 System Prompt for Code Generation (General + 3D WebUI)

  • Generation prompts standardize distinct output formats for general React/TypeScript interfaces and standalone Three.js simulation pages.The general prompt targets a Next.js pages-router component, while the 3D prompt specifies HTML output and verified CDN imports.Source: §A.3 · p. 14, Figure 8 · p. 15, Figure 9 · p. 16

A.4 Evaluation Metrics and Failure Handling

The evaluation separates executable outcomes into success levels and build failure, while the benchmark draws on varied general WebUI and 3D simulation tasks. Its task scope includes both user-facing workflows and technically demanding interactive simulations.

A.4 Evaluation Metrics and Failure Handling

  • Evaluation outcomes are classified as Full Success, Partial Success, No Success, or Build Failure.Build Failure includes build-time or launch-time problems that prevent UI-agent testing, and receives no credit in the success averages.Source: §A.4 · p. 16

B.1 Data Source and Task Scope

  • General WebUI seed instructions come from WebGen Bench and WebDev-Arena, with generated seed webpages produced using DeepSeek R1.The collection uses 101 WebGenBench test-set instructions and 48 WebDev-Arena-10k instructions.Source: §B.1 · p. 17
  • General tasks span creative media, business data, games and sports, social interaction, and learning and lifestyle applications.The examples include dashboards, data visualizations, chat interfaces, and interactive games; requirements can include real-time interaction, charts, authentication, and responsive design.Source: §B.1 · p. 17
  • The 70 3D simulation challenges draw on Three.js examples and cover astronomy, geometry, physics, and interactive exploration.They involve concepts such as transformations, particle systems, collision detection, and real-time rendering, alongside browser-based controls and scene management.Source: §B.1 · p. 17

B.2 WebUI Categories and Test Case Distribution

WebUIProof includes 149 general WebUI tasks and 70 3D simulations, with category distributions reported for both. The appendix also describes test-case categories, how the UI agent explores pages to create cases, and examples of structured specifications.

B.2 WebUI Categories and Test Case Distribution

  • General WebUI tasks are concentrated in Creative Media (41.6%), while Classical Mechanics leads 3D Simulation (42.9%).Figure 10 reports 149 general tasks and 70 simulations; other general categories include Data Management (20.1%) and Social Interaction (16.1%), while Fluid Waves accounts for 24.3% of simulations.Source: §B.2 · p. 18, Figure 10 · p. 18
  • Test cases are grouped into functionality, data display, and design validation categories.Figure 11 presents their distribution across the two task families; the passage does not state category-specific counts.Source: §B.2 · p. 18, Figure 11 · p. 18

B.3 Data Collection Pipeline and Detail Prompt

  • The collection pipeline starts with webpage screenshots and HTML, then iteratively uses UI observations and discovered elements to recommend test cases.The appendix describes the agent first summarizing functionality and proposing basic cases, then updating recommendations as the UI state changes.Source: §B.3 · p. 19, Figure 12 · p. 19, Figure 13 · p. 19

B.4 Sample of Structured Specification in WebUIProof

  • Structured specifications pair a website description and element requirements with executable test cases and expected results.Examples include checking that a report download starts and that dragging the mouse rotates a 3D galaxy; the benchmark contains 219 such specifications.Source: §B.4 · pp. 20, 22

C VisRL: Training Detail

  • The VisRL appendix provides implementation details for sequence-level PPO, including reward computation, candidate selection, and efficiency heuristics.The appendix overview also lists rollout, reward-definition, and learning-strategy material.Source: C VisRL: Training Detail · p. 23

C.1 Implementation Detail

The appendix details the SFT-to-PPO training setup, sequence-level optimization and reward design, UI-agent execution, and vision rubric; it also reports more stable PPO learning with UI-agent feedback.

C.1 Implementation Detail

  • SFT cold-starts Qwen2.5-14B on 6,000 WebUI samples; PPO then uses one H100 for inference and three for optimization, with LoRA ranks 8 for general and 128 for 3D tasks.The appendix also specifies hardware, training times, and configuration tables.Source: §C.1 · pp. 23–24, Table 8 · p. 23, Table 9 · p. 24

C.2 Sequence-Level Rollout and Probability

  • PPO scores each generated code sequence using one outcome-derived advantage and applies the clipped objective once per sequence.The combined loss includes token entropy to encourage exploration.Source: §C.2 · p. 25

C.3 Reward Definition

  • The outcome reward combines code-metric bonuses, visual quality, UI-agent success, and penalties after code execution and rendering.The visual evaluator scores layout and polish, while the UI agent tests interactions such as form filling, clicking, and navigation.Source: §C.3 · pp. 25–26

C.4 Additional Learning Strategies

  • Multi-temperature sampling explores candidate outputs, while a fast-gate filter skips likely-invalid code before costly build and rendering steps.Candidates are executed and the best is retained subject to a minimum-quality constraint.Source: §C.4 · p. 26

C.5 Pseudo Algorithm for Training Procedure

  • The training algorithm samples candidates, filters and executes them, selects a reward-ranked output, then updates the policy and value model with normalized advantages.The pseudocode allows the selection to be thresholded.Source: §C.5 · p. 26

C.6 UI Agent Task Execution in VisRL

  • The UI agent assigns DOM element IDs, proposes actions from a labeled screenshot and element list, executes them, and scores observable interaction outcomes.Success can be registered through expected text, changed elements, or updated DOM counts; errors or no observable change count as failure.Source: §C.6 · p. 27

C.7 Vision Agent – Evaluation Rubric

  • The vision rubric scores design and interactive-component presence, adds a polish bonus, and assigns zero for build/runtime errors or blank and browser-error pages.Its base score combines design and interactivity.Source: §C.7 · p. 28

D.1 Training Curve for Different PPO Strategies

  • With UI-agent feedback, PPO reward rises early, vision score increases, and invalid actions decline, unlike noisier vision-only PPO.The comparison is shown in average reward, vision score, and invalid action rate.Source: Figure 16 · p. 29, §D.1 · p. 29

D.2 Reward Decomposition Analysis

Reward decomposition compares single-component PPO rewards with the full composite reward across general and 3D WebUI tasks, finding the composite most balanced across quality dimensions.

D.2 Reward Decomposition Analysis

  • The composite reward achieves the most balanced performance across build success, visual quality, and interaction success, while single-component rewards leave dimensions weak.For 3D simulations, the composite maintains build success above 0.9 while improving vision scores and UI-agent test success over single-component baselines.Source: §D.2 · p. 30, Figure 17 · p. 30

D.3 Reward Weight Analysis – α Sensitivity

The weight sweep reveals a visual-quality/functionality trade-off; the selected weights aim to preserve visual quality while maintaining build success and UI-agent correctness.

D.3 Reward Weight Analysis – α Sensitivity

  • The selected weights are α = [0.6, 0.3, 0.1], balancing visual quality with build success and UI-agent correctness gains.The weights correspond to vision, UI-agent, and analyzer rewards.Source: §D.3 · p. 31
  • At (0.8, 0.1, 0.1), average vision score is ∼0.70; shifting weight toward the UI agent raises full success from ∼0.28 to ∼0.37 and build success from ∼0.58 to ∼0.64.The analyzer weight remains fixed at αm = 0.1 during the sweep.Source: Figure 18 · p. 31, §D.3 · p. 31

E.1 Additional Experimental Results (per-Test Category Results)

The appendix reports per-category evaluation results, illustrates UI-agent interaction traces and generated samples, and describes the build-time error categories analyzed across models.

E.1 Additional Experimental Results (per-Test Category Results)

  • Per-test category results report completion and accuracy for general WebUI and 3D tasks across proprietary and self-trained models.The cited tables present general-task results for both model groups and 3D-task results for both groups.Source: §E.1 · p. 32, Table 10 · p. 32, Table 11 · p. 33

E.2 Visualization of UI Agent Traces

E.3 More Visualization

  • Generated samples cover general WebUI categories including creative media, data management, games, social interaction, and e-commerce, alongside multiple 3D simulation examples.The general examples include social media viewers, games, stock reports, and workflow tools; 3D examples include physics and scene simulations.Source: §E.3 · pp. 38, 41, Figure 22 · p. 38, Figure 27 · p. 41

E.4 Analysis of Build Failure Taxonomy

  • Build-time failures are grouped as missing modules, syntax errors, invalid default exports, and runtime reference errors, with counts reported across eight models.The figure describes bars by error type, where higher stacks indicate more frequent build failures.Source: §E.4 · p. 43, Figure 30 · p. 43

Related papers

Loading 2610.02617v1…