PaperlayerView exact revision on arXiv

Source-linked AI summary

WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts

Yuxin Meng, Yuhan Suo, Junjie Wang, Yuhan Sun, Yiyao Yu, Ruixu Zhang, Ruining Hu, Yubin Wang, Shouwei Ruan, Bin Wang, Yuxiang Zhang, Yujiu Yang

arXiv:2606.03220v1cs.CLcs.AI

TL;DR

Existing benchmarks often assess interaction through local evidence rather than enumerating requirement-induced states and transitions, limiting evaluation of whether generated pages actually work. WebRISE compiles explicit and implicit requirements into Interaction Contract Graphs for implementation-agnostic browser execution. Across 442 tasks, five modalities, and 14 models, current systems remain far from solved, while Video is the strongest interaction signal and implicit constraints remain a bottleneck.

  • Problem

    Existing benchmarks rely on local interaction evidence and do not explicitly define the requirement-induced states, transitions, and cross-step constraints needed to assess usable web artifacts.

  • Method

    WebRISE compiles explicit and implicit requirements into Interaction Contract Graphs of observable states, user-intent transitions, and DOM/visual assertions for contract-guided browser evaluation.

  • Results

    Across 442 tasks, five input modalities, and 14 models, current systems remain far from solving interactive web generation; Video provides the strongest interaction signal, while implicit state constraints remain a persistent bottleneck.

  • Takeaways & Limitations

    Generated web artifacts should be evaluated by requirement-level state behavior and conformance, not visual plausibility or isolated action success alone.

  • Takeaways & Limitations

    WebRISE measures frontend interaction conformance rather than deployment readiness because it does not cover back-end services, authentication, external APIs, persistent databases, multi-user concurrency, or long-lived sessions.

Abstract

from arXiv · show

Existing benchmarks for MLLM-generated web artifacts assess interaction through local evidence and miss the requirement-induced states and transitions that determine whether a page works. We introduce WebRISE, which compiles task requirements into Interaction Contract Graphs (ICGs) of observable states, user-intent transitions, and DOM/visual assertions for implementation-agnostic browser execution. WebRISE spans 442 tasks across five input modalities (Text, Markdown, Sketch, Image, Video), with 5,495 transitions and 5,271 requirement checks that separate user-stated functions from implicit product-level constraints. Across 14 MLLMs, even the strongest model reaches only 65.6% transition validity and 66.3% requirement coverage, and visual quality is no proxy for behavior (Qwen3.6-35B-A3B on Markdown: V=80.8 yet T=15.5). Video gives the strongest interaction signal (+10.6 pp implicit coverage over Text), while implicit constraints persist; defect injection shows ICG-based scoring detects state errors at 2-16x the rate of checkpoint-style evaluation.

1 Introduction

WebRISE reframes evaluation of MLLM-generated web artifacts around requirement-induced state-transition conformance rather than appearance or isolated actions. It compiles explicit and implicit requirements into executable interaction contracts and finds substantial gaps in current models’ interaction validity and requirement coverage.

  • Motivation: WebRISE addresses the gap between visually plausible pages and usable interfaces by testing requirement-implied state transitions and state-consistency constraints.Local responses can pass while crosscomponent, cross-view, or cross-step constraints remain violated.
  • Approach: WebRISE represents each task as an Interaction Contract Graph containing observable UI states, user-intent transitions, and DOM/visual assertions.The contract links transition outcomes to explicit requirements and implicit state-consistency constraints.
  • Approach: The benchmark converts explicit and implicit requirements into test items and transitions, then uses a contract-guided agent and dual DOM/visual oracles during browser execution.Transient behaviors are verified as transition-level DOM evidence, while user-visible outcomes are checked through DOM or visual assertions.
  • Findings: 65.6% transition validity and 66.3% requirement coverage are achieved by the strongest model under its best modality, leaving roughly one third of obligations unsatisfied.The strongest model is GPT-5.5, evaluated under Video.
  • Findings: Video improves T, R, and Ri over Text by 8.8, 8.3, and 10.6 percentage points, respectively, while implicit state constraints remain a consistent bottleneck.Defect injection detects 16/25 state-related defects with ICG-based evaluation, versus 8/25 for broad checkpoint-style WebGen and 1/25 for strict checkpoint-style WebGen.
  • Evaluation: 442 tasks span five input modalities and are evaluated across 14 representative models, separating explicit functions from implicit state-consistency constraints.The modalities are Text, Markdown, Sketch, Image, and Video.

2 Related Work

Prior web-generation benchmarks increasingly evaluate interaction, but they commonly rely on scripts, checkpoints, trajectories, visual evidence, or exploration traces. WebRISE instead evaluates implementation-agnostic requirement conformance through linked states, transitions, assertions, and diagnostics.

  • Landscape: Web-generation research has expanded from visual and structural fidelity toward executable artifacts, automated functional testing, visual-interactive evaluation, and agentic verification.This progression includes textual, visual, and video-based specifications.
  • Evaluation target: Static-page benchmarks naturally target visual fidelity, structural similarity, and code executability, whereas interactive artifacts require correct responses to actions and preservation of task-implied state constraints.WebRISE therefore treats generated artifacts as executable, stateful interfaces rather than merely rendered pages or code.
  • Existing evaluation: Existing protocols make interaction observable through scripts, agents, visual judges, demonstrated trajectories, or checkpoints, but often operationalize it through local evidence.Script-based checks may depend on implementation-specific selectors or entry points.
  • WebRISE: WebRISE formulates interaction evaluation as requirement conformance, with an ICG defining requirement-linked states, transitions, and assertions for adaptive-agent execution.This supports diagnosis beyond pass/fail outcomes.

3 WebRISE: Benchmark Design

WebRISE evaluates executable web artifacts against requirement-induced interaction contracts rather than reference implementations or isolated visual evidence. Its benchmark compiles multimodal task requirements into ICGs spanning observable states, user-intent transitions, and DOM/visual predicates.

  • Task Definition: WebRISE evaluates whether browser-executable HTML, CSS, and JavaScript satisfy a task’s interaction behavior under browser execution.Artifacts require no external back-end services or manually prepared runtime state.
  • Task Definition: The shared contract enables comparison of Text, Markdown, Sketch, Image, and Video specifications against the same required interactive behavior.Evaluation tests contract satisfaction rather than reference-DOM matching, fixed selector paths, or a single visual snapshot.
  • Requirement-Induced Interaction Contracts: An Interaction Contract Graph represents stable observable UI states, user-intent transitions, DOM/visual predicates, and requirement-to-test mappings.States are requirement-relevant configurations rather than full DOM snapshots, while transient effects remain transition-level predicates.
  • Requirement-Induced Interaction Contracts: Explicit requirements encode user-stated affordances, whereas implicit requirements encode product constraints such as synchronization, boundary feedback, pagination reset, and stale-state removal.The mapping connects transition evidence to the requirements it verifies.
  • Contract Construction Pipeline: WebRISE compiles expert-informed task materials into test items, ICGs, assertions, visual postconditions, and coverage mappings, then validates each contract with executable ground-truth HTML.The benchmark covers eight domains and 35 scenarios, including Productivity Tools and Social Interaction.

4 Evaluation Protocol

WebRISE executes requirement-linked transitions with an adaptive browser agent and verifies them using process-level DOM evidence and final visual outcomes. It reports reachability, transition correctness, and requirement coverage from transition verdicts, with task-level macro-averaging.

  • Protocol Overview: The contract-guided agent determines how to execute each ICG transition on a generated HTML artifact, while the ICG specifies what must be verified.Transitions are evaluated under browser execution rather than fixed scripts.
  • Protocol Overview: Each transition combines source and target states, an agent goal, preconditions, DOM assertions, and visual postconditions to localize requirement-linked evidence.The evidence bundle records traces, DOM logs, screenshots, snapshots, assertion verdicts, and replay paths.
  • Diagnostic Metrics: A transition is PASS only when its source is reachable, the intended interaction completes, and all required DOM/visual checks hold.Other outcomes are FAIL, BLOCKED, or SKIPPED, and only PASS counts as successful.
  • Protocol Overview: Adaptive execution reserializes the current DOM after every action and replays verified trajectories to restore source states in branching graphs.This avoids fixed selectors or reference DOM paths and separates unreachable states from executable violations.
  • Diagnostic Metrics: Requirement coverage marks a requirement satisfied only when all mapped checks pass, distinguishing explicit functional affordances from implicit state-consistency constraints.Implicit checks include synchronization, boundary feedback, reset behavior, and stale-state removal.
  • Diagnostic Metrics: Metrics are computed per task and macro-averaged, preventing tasks with more transitions or assertions from dominating aggregate scores.The aggregation applies to S%, T%, Re%, Ri%, and R%.

5 Experiments and Findings

WebRISE reveals that interactive web artifact generation remains substantially unsolved: interaction correctness diverges from visual quality, and stateful reasoning remains a major failure source. Multimodal inputs and larger models help unevenly, while contract-based evaluation exposes defects missed by local checkpoints.

  • T = 65.6 and R = 66.3 are GPT-5.5’s best-modality scores, leaving roughly one third of required transitions or requirement checks unsatisfied.
  • Overall scores place proprietary models first, but Kimi-K2.6 achieves the best open-weight Overall score at 63.3 and Qwen3.6-27B reaches 62.5.
  • V = 80.8 for Qwen3.6-35B-A3B on Markdown coexists with T = 15.5 and R = 19.2, showing visual quality is not a proxy for interaction correctness.
  • Video leads interaction-oriented metrics despite Image and Video having similar visual-score distributions, indicating temporal evidence better explains Video’s advantage than static fidelity.
  • Performance is largely flat from 27B to 122B-A10B but improves clearly at 397B-A17B within Qwen3.5, especially under Text and Markdown.
  • ICG-based evaluation detects substantially more injected state defects than checkpoint-style WebGen signals, although the remaining misses show it is not exhaustive.
  • State & Logic dominates direct failures for GPT-5.5 and Kimi-K2.6, while shopping-cart diagnosis exposes stale totals and checkout availability after a checkbox change.

6 Conclusion

WebRISE evaluates generated web artifacts through requirement-induced observable state-transition conformance using Interaction Contract Graphs. Across its benchmark, Video provides the strongest interaction signal, but implicit state constraints remain a persistent bottleneck.

  • WebRISE evaluates MLLM-generated web artifacts through requirement-induced observable state-transition conformance.
  • Interaction Contract Graphs enable implementation-agnostic browser execution and diagnostics at state, transition, and requirement levels.
  • Experiments across 442 tasks, five input modalities, and 14 models show that Video provides the strongest interaction signal while implicit state constraints remain a persistent bottleneck.

Limitations

WebRISE’s controlled self-contained HTML setting supports consistent comparison but does not represent full production web systems. Its conformance evidence is also bounded by the specified contracts, test items, and assertions.

  • WebRISE focuses on self-contained HTML artifacts in a controlled browser environment, excluding production systems with back-end services, authentication, APIs, databases, concurrency, or long-lived sessions.
  • WebRISE measures frontend interaction conformance rather than deployment readiness because its evaluation does not cover full production web systems.
  • Coverage is bounded by specified requirements, generated test items, and DOM/visual assertions, so WebRISE is not an exhaustive characterization of all possible user behaviors.

Ethical Considerations

WebRISE is a diagnostic benchmark rather than a deployable system, and its metrics reflect regionally shaped interaction conventions. Judge scores are useful for ranking but remain sensitive to prompt and API changes.

  • WebRISE is a diagnostic benchmark, not a deployable system.
  • Regional product conventions shape expected interaction because contributors were drawn primarily from a single region.Applications targeting other markets should extend the contract set with locale-specific affordances.
  • LLM-judge scores remain susceptible to prompt sensitivity and API version drift.The paper recommends interpreting reported scores as stable rank-orderings rather than absolute measurements.
  • The benchmark spans 442 tasks across eight domains and 35 scenarios under five input modalities.Its interaction contracts include 5,081 states, 5,495 transitions, and 5,271 requirement checks covering explicit and implicit constraints.

A.3 Human Consistency Validation

The validation studies examine whether WebRISE’s interaction-contract construction, functional evaluator, and modality-specific visual evaluator align with human judgments. The protocol also defines structured browser observations, assertion timing, outcomes, and audit evidence.

  • Validation scope: Human consistency validation checks interaction-contract construction, agent-based functional evaluation, and modality-specific visual evaluation.It serves as a consistency check rather than tuning model outputs or changing the main results.
  • Benchmark basis: The benchmark construction covers 442 tasks and five modalities, with shared task contracts instantiated across different specification signals.The validation references these construction and modality dimensions when assessing evaluator consistency.
  • Browser observations: The indexed DOM observation exposes interaction-relevant elements and state fields without relying on persistent CSS selectors or fixed DOM paths.It records accessibility roles, visible text, attributes, and interaction states for the current page state.
  • Evaluation representation: WebRISE represents each task with requirement-relevant observable states and verifies transient behaviors through transition-level DOM evidence.Persistent outcomes are checked in the final stable DOM state.
  • Scoring rules: Only YES counts as passing when assertion-, transition-, and requirement-level scores are aggregated.Ambiguous element matches return UNCERTAIN rather than selecting a target arbitrarily.
  • Outcomes and auditability: Transitions receive PASS, FAIL, BLOCKED, or SKIPPED outcomes based on reachability, execution, and required DOM/visual checks.Each transition also stores screenshots, action traces, DOM evidence, assertion verdicts, and replay information for auditability.

B.5 Visual Quality Evaluation Details

WebRISE treats visual quality as an auxiliary, modality-specific signal rather than a replacement for executable interaction metrics. Its visual evaluator combines layout, color accessibility, and perceptual aesthetics with reference-dependent weighting.

  • Visual components: The visual evaluator combines layout structure, color accessibility, and perceptual aesthetics.Layout analysis measures alignment, structural clarity, floating-element artifacts, whitespace balance, hierarchy, and polish.
  • Modality-specific aggregation: Visual scoring weights the available reference differently by modality.Aesthetics dominates Text, structural similarity dominates Markdown and Sketch, and layout fidelity and color reproduction dominate Image and Video.
  • Reporting: All visual scores are macro-averaged across tasks and reported for each model-by-modality combination.The scores use a 0–100 scale.

C.1 Evaluation Judge Configuration

WebRISE evaluates interaction, requirement coverage, failure modes, and safety across models and modalities using fixed judge configurations. Results show persistent state-logic bottlenecks, stronger performance from Video on difficult tasks, and complementary coverage from ICG-based checks.

  • Judge configuration: GPT-5-mini scores transition-level DOM assertions and visual postconditions, while Gemini-3-Flash-Preview scores auxiliary visual quality.The same judge configuration is applied across models, tasks, and modalities.
  • Judge robustness: GPT-5-mini remains close to GPT-5.4 on a 100-pair defect-injection control set.The pairs contain GT-validated pages and variants with controlled interaction faults.
  • Modality analysis: Video leads modality-level transition validity, implicit requirement coverage, and overall requirement coverage, while reducing the explicit–implicit gap to 7.7 points.Image has the highest explicit coverage, whereas Sketch has the highest auxiliary visual score but weaker interaction metrics.
  • Difficulty analysis: Hard50 gives Video larger gains over Image than Easy50: 4.9, 6.1, and 4.8 points for T, Ri, and R, respectively.The modality advantage is therefore larger on difficult interaction tasks.
  • Failure attribution: State and logic errors dominate both Hard50 and Easy50, while Hard50 has larger shares of availability and feedback/boundary failures.Easy50 more often exposes a basic interaction path but fails to preserve correct result logic or state consistency.
  • ICG-only errors: ICG detects longer-range constraints that checkpoint-style evaluation misses, including history preservation, cross-feature non-interference, navigation-time retention, and action gating.These cases arise among 13 defect-injected examples where WebGen marks all test items as YES.
  • Safety evaluation: Asynchronous robustness, DOM rendering safety, and request security are the weakest GPT-5.5 rule families.Frequent low-pass checks include missing input constraints, unsafe DOM rendering, repeated-click guards, and sensitive-form protections.

E Code and Data Availability

WebRISE’s release is planned to include the specifications, annotations, graphs, evaluation materials, and aggregated results needed to reproduce the main experiments, while protecting contributor and annotator privacy. The evaluation pipeline derives test contracts and interaction graphs from requirements, executes browser transitions, and judges DOM and visual postconditions.

  • Code and Data Availability: The planned MIT-licensed release includes task specifications, requirement annotations, Interaction Contract Graphs, evaluation scripts, prompt templates, and aggregated experimental results.Information that could identify individual contributors or annotators will be excluded for privacy reasons.
  • Failure Examples: The illustrated failure cases cover missing loading feedback, unchanged waitlist positions, ineffective layer hiding, lost drafts, and reset image-rotation state.These examples show state, logic, feedback, and state-preservation failures across different workflows.
  • Requirement-Derived Testing: Requirement lists are converted into minimal first-load test data contracts and implementation-neutral test items without using a reference implementation.The test items are intended to cover every testable behavior in the requirements.
  • Interaction Contracts: Interaction Contract Graphs represent states and transitions whose specifications include source and target states, agent tasks, mapped test items, DOM assertions, and visual postconditions.This defines the structured contract used to test web interactions.
  • Browser Execution: A browser agent executes one action per iteration from indexed DOM observations, including newly appeared elements marked in the latest observation.The execution prompt instructs the agent to choose exactly one indexed action at a time.
  • Postcondition Evaluation: DOM assertions are judged from structured interaction evidence containing actions, before-and-after DOM snapshots, mutations, attribute changes, node additions or removals, and interactive-element summaries.Visual postconditions are separately judged by comparing screenshots before and after the interaction.
Loading 2606.03220v1…