PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants

Yue Peng, Lanke Xia, Zihan Wang, Jiahao Ye, Ke Ning, Hongyi Wen

arXiv:2608.29387v1cs.AIcs.SE

TL;DR

Generative UI lacks reliable evaluation for maintaining one executable interface as requirements evolve across turns. EvoGenUI-Bench evaluates this setting through browser execution and multi-surface evidence, finding that strong turn-level performance does not ensure sustained episode success, especially for stateful and tool-grounded interfaces.

  • Problem

    Existing evaluations largely separate interface construction from execution, leaving reliability across persistent multi-turn revisions insufficiently assessed.

  • Method

    EvoGenUI-Bench evaluates 150 five-turn tasks across three scenarios by executing updated artifacts in browsers and combining visual, behavioral, source-level, and runtime evidence.

  • Results

    74.9% overall Turn Pass still yielded only 37.3% five-turn episode completion for the best-performing model, with reliability declining especially on later revisions.

  • Takeaways & Limitations

    Generative UI reliability requires testing synchronization among interface behavior, dependent state, external state, and assistant claims as artifacts evolve.

  • Takeaways & Limitations

    The benchmark uses human-authored five-turn episodes in a fixed React/Vite browser environment and excludes natural logs, other frameworks, accessibility, and device conditions.

Abstract

from arXiv · show

Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve. We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state. We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs. Beyond turn-level and episode-level success, we measure cross-turn retention with Adjacent Pass Retention. Across eight models, even the strongest achieves 74.9% Turn Pass while completing only 37.3% of five-turn episodes; APR further falls to 52.4% on tool-grounded tasks. Diagnostic analysis shows that presentation failures center on information architecture, interaction failures on derived-state propagation and affordance binding, and tool-grounded failures additionally involve external-state grounding and requirement decomposition. These results reframe generative UI evaluation from judging isolated outputs to testing whether interface behavior, derived state, external state, and assistant claims remain synchronized as the artifact evolves.

1 Introduction

EVOGENUI frames generative UI as maintaining one executable interface through evolving requirements, rather than generating isolated pages. Its benchmark and evaluation protocol expose a gap between turn-level correctness and sustained multi-turn reliability.

  • Motivation: EVOGENUI treats interface generation as updating one executable artifact while preserving still-valid requirements across evolving user requests.The interface’s behavior, visible state, and accompanying response must remain consistent with conversation history.
  • Motivation: Existing benchmarks largely separate interface construction from task execution, leaving full-sequence consistency insufficiently evaluated.EVOGENUI instead assesses whether behavior, state, and prior requirements remain consistent throughout revision.
  • Benchmark: EVOGENUI-BENCH contains 150 tasks and 750 turns across Presentation, Interaction, and Tool-grounded scenario suites.Each task begins with interface generation and continues through four revisions to the same interface.
  • Evaluation: The protocol builds and executes artifacts in a browser, scoring Presentation, Execution, and Alignment with multi-surface evidence and reliability metrics.It measures Turn Pass, TP@5, and Adjacent Pass Retention.
  • Results: 74.9% overall Turn Pass coexists with only 37.3% five-turn episode completion for the best-performing model.The benchmark therefore reveals substantial divergence between isolated-turn success and sustained episode reliability.
  • Diagnostics: Diagnostic analysis identifies distinct failure mechanisms across presentation, interaction, and tool-grounded scenarios.The contribution includes a human-validated taxonomy of interface-maintenance failures.

2 Related Work

Related benchmarks cover visual fidelity, executable frontend generation, agent interaction, tool use, and open-ended evaluation. EVOGENUI-BENCH extends these directions by evaluating a persistent interface under successive user requests with execution-grounded evidence.

  • Generative UI benchmarks: Existing Generative UI benchmarks primarily evaluate visual fidelity, interactive behavior, executable frontend generation, or self-contained mini-application episodes.Their emphasis is generally on isolated generation rather than persistent interface evolution.
  • Iterative development: Recent iterative-development benchmarks expose forgetting, overwriting of prior features, structural degradation, and broader multi-turn instruction-following failures.EVOGENUI-BENCH extends this direction to evolving executable interfaces.
  • Interactive agents: WebArena, VisualWebArena, OSWorld, and AgentBench evaluate agents operating within environments whose interfaces are supplied by the environment.EVOGENUI-BENCH instead requires the model to generate and continually revise the interface.
  • Tool-grounded agents: Tool-use benchmarks evaluate tool selection, invocation, and persistent state changes, whereas EVOGENUI also evaluates whether those operations are exposed through an evolving interface.The distinction is especially relevant to tool-grounded UI tasks.
  • Evaluation: Executable-interface evaluation combines browser execution, functional testing, temporal screenshots, and multimodal judging to capture behavior beyond reference renderings or static analysis.Execution evidence complements visual and interactive assessment.
  • Evaluation: Judgment-based evaluation must be grounded in execution evidence because an artifact can build successfully while controls, state, prior functionality, or claims remain incorrect.This motivates evidence-grounded evaluation for interface maintenance.

3 EVOGENUI-BENCH

EVOGENUI-BENCH formulates generative UI as five-turn maintenance of one executable artifact, evaluated through browser execution and evidence surfaces. Its controlled construction spans three balanced scenario suites while hiding private validation requirements from generators.

  • Task Formulation: Each task is a five-turn episode in which the same web interface is successively revised instead of independently regenerated.The model receives current context and recent source, then returns both a user-facing response and complete updated source code.
  • Task Formulation: A valid update must implement the new request while preserving all still-valid prior requirements and keeping behavior, visible state, and response mutually consistent.The evaluation unit is the evolving executable artifact.
  • Evaluation Design: Private validation requirements are separated from generator-visible inputs and linked to observable surfaces including UI, DOM, traces, source, responses, logs, and runtime state.Validation targets requested semantics rather than a reference implementation.
  • Task Construction: Human-authored tasks use fixed five-turn structure, balanced scenario coverage, deterministic external state where applicable, and distinct domain-interface-challenge pairings.The suite contains 150 distinct domain labels across 150 tasks.
  • Benchmark Composition: EVOGENUI-BENCH contains 150 tasks and 750 requested turns divided equally among Presentation, Interaction, and Tool-grounded suites.The suites intentionally differ in private validation-contract density.
  • Benchmark Composition: Every task contains five turns, while Req./turn reports the mean hidden private validation requirements per turn.The composition table defines the benchmark’s basic structure and validation-density measure.

4 Evidence-Grounded Evaluation

The benchmark evaluates evolving executable interfaces through browser execution and evidence-grounded scoring. It reports turn-level, episode-level, and cross-turn reliability while preserving failures under requested-slot accounting.

  • Evaluation workflow: Each turn is evaluated through interface generation, browser execution, evidence collection, and model-based scoring.The workflow passes evidence between stages and executes later turns using prior source, snapshots, and restored runtime state when applicable.
  • Accounting and validation: Generation, formatting, build, and execution failures remain countable failures rather than omitted requests or restarted downstream evaluations.Missing artifacts block dependent later turns, and invalidly formatted responses or build failures receive a failing pass indicator.
  • Evaluation dimensions: Presentation, Execution, and Alignment score rendered quality, working behavior, and consistency across the assistant response, artifact, interface, observations, and logs.The evaluator combines screenshots and DOM state with actor traces, tool and resource logs, runtime snapshots, source summaries, build status, and prior-turn context.
  • Reliability metrics: Turn Pass measures individual-turn correctness, while TP@5 and CPT measure sustained episode success and initial uninterrupted passing runs.Because every episode has five turns, TP@5 indicates whether all five turns pass; CPT ranges from zero to five.
  • Reliability metrics: The independence baseline estimates TP@5 from position-specific empirical pass rates under an assumption of independent turn outcomes.Observed outcomes can then be compared with the expected episode-level success under that diagnostic assumption.
  • Reliability metrics: APR measures whether a passing turn remains successful on the next eligible revision when the next generation call returns a response.A failed transition may reflect regression on prior requirements, failure on the new requirement, or both.
  • Accounting and validation: On a blinded 240-case validation sample, the automatic evaluator reaches 86.7% accuracy with Cohen’s κ = 0.73 against human labels.Three non-annotators independently labeled cases, with majority vote defining the reference; inter-annotator agreement was Fleiss’ κ = 0.78.

5 Experiments

Across eight models, turn-level success substantially exceeds sustained five-turn episode reliability, with tool-grounded tasks the most difficult. Reliability also declines after the second turn, while evaluator and regeneration analyses assess robustness.

  • Main Results: 74.9% overall TP is achieved by Claude-Opus-4.7, but only 37.3% of its five-turn episodes pass completely.Across models, TP averages 42.7% versus 11.8% for TP@5.
  • Main Results: Observed TP@5 exceeds the independence estimate for all eight models, indicating that passing turns cluster within episodes.For Claude-Opus-4.7, TP@5 is 37.3% versus an independence estimate of 23.4%.
  • Main Results: Tool-grounded tasks have the lowest model-averaged TP at 25.0% and TP@5 at 5.0%, alongside 11.9 requirements per turn.Presentation and Interaction TP averages are 55.5% and 47.6%, respectively.
  • Main Results: APR is lowest for Tool-grounded tasks at 52.4%, compared with 71.1% for Presentation and 68.7% for Interaction.Among 110 audited failed APR transitions, 52.7% involve regression on prior behavior and 47.3% fail only the new requirement.
  • Main Results: Aggregate pass rates fall to 39.4% at turn 3 and 35.1% at turn 4, with Tool-grounded tasks declining from 39.5% at turn 2 to 18.5% at turn 3.The largest aggregate decline occurs between turns 2 and 3.
  • Robustness and Evaluator Diagnostics: The full-evidence evaluator reaches 87.5% accuracy, while removing actor traces reduces accuracy to 55.0%.Removing the private reference, source code, or screenshot yields 63.8%, 67.5%, and 78.3%, respectively.

6 Analysis and Discussion

Diagnostic analysis identifies scenario-specific mechanisms behind failed interface maintenance and uses executed case studies to show why different evidence surfaces are needed. Presentation, interaction, and tool-grounded failures require distinct checks for synchronization and behavior.

  • Analysis and Discussion: Six diagnostic mechanisms cover information architecture, domain representation, requirement decomposition, affordance binding, derived-state propagation, and external-state grounding.All 2,750 executed non-passing calls receive one primary mechanism label.
  • Analysis and Discussion: The diagnostic judge achieves 90.3% mean exact-label agreement, with multiclass Cohen’s κ of 0.89 and annotator Fleiss’ κ of 0.91.The codebook was validated on a category-balanced sample of 120 failures.
  • Analysis and Discussion: Presentation failures primarily involve information architecture, whereas Interaction failures more often involve derived-state propagation and affordance binding.These interaction failures produce stale dependent views or visible controls that are not wired to behavior.
  • Analysis and Discussion: Tool-grounded failures additionally span external-state grounding and requirement decomposition.Reliable tool-grounded interfaces must coordinate interface state with external runtime state and task requirements.
  • Case Studies: The case studies show that screenshots reveal presentation defects, actor traces and source changes reveal interaction defects, and runtime logs reveal external-state mismatches.Each evidence surface exposes a different failure mechanism.

7 Conclusion

EVOGENUI-BENCH evaluates generative UI as maintenance of an executable artifact across revisions rather than as isolated page generation. Its results show that sustained reliability requires synchronization across interface behavior, dependent state, external state, and assistant claims.

  • Conclusion: EVOGENUI-BENCH combines browser execution with visual, behavioral, source-level, and runtime evidence to measure correctness across revisions.The benchmark evaluates multi-turn maintenance of an executable artifact.
  • Conclusion: Strong single-turn performance does not reliably translate into sustained episode success, particularly for stateful and tool-grounded interfaces.The diagnostic analysis identifies task-dependent mechanisms including interaction wiring, derived-state propagation, requirement decomposition, and external-state grounding.
  • Conclusion: Generative UI reliability requires checking whether interface behavior, dependent state, external state, and assistant claims remain synchronized as the artifact evolves.Neither a final screenshot nor an isolated turn score adequately captures this reliability.

Limitations

The benchmark’s scope is bounded by controlled five-turn episodes, a fixed React/Vite environment, deterministic mock runtimes, and an outcome-level retention metric that does not establish causality.

  • EVOGENUI-BENCH uses human-authored five-turn episodes in a fixed React/Vite browser environment, excluding naturally occurring logs, other frameworks, accessibility requirements, and device conditions.
  • APR measures outcome-level retention rather than causal attribution, so a failed transition may reflect regression, a new-requirement failure, or both.
  • Comprehensive failure attribution would require replaying all prior-turn validation contracts against each later artifact, beyond the post hoc sample audit.
  • Tool-grounded tasks use deterministic mock runtimes, evaluating controlled interface–runtime synchronization without capturing live-service latency, authentication failures, outages, rate limits, or API changes.

Ethical Considerations

The benchmark reduces privacy and leakage risks through synthetic, deterministic tasks and a separation between generator-visible inputs and private validation material, while remaining unsuitable for production-safety certification.

  • EVOGENUI-BENCH uses synthetic tasks, mock external state, and deterministic tool environments instead of private user data or live third-party services.
  • The benchmark manually screens prompts, fixtures, and public resources for personally identifying information and offensive content before release.
  • Intended use: The benchmark is intended for comparing multi-turn generative UI construction and revision, not for certifying safety or production readiness.
  • Generator-visible materials are separated from evaluator-only validation contracts, backend fixtures, runtime state, and evaluator-specific evidence requirements.
  • Task construction: Each task uses a five-turn cumulative revision trajectory with private requirements bound to observable evidence surfaces such as screenshots, DOM, traces, logs, source, and assistant text.
  • Evaluation: The three evaluator-scored dimensions are Presentation, Execution, and Alignment, each using a 1–5 scale and passing at scores of 4 or 5.

C.2 Evaluation Metrics

The evaluation defines official turn-, episode-, and transition-level metrics, supplements them with diagnostic transition audits, and reports uncertainty using task-level bootstrap resampling.

  • Official metrics include Turn Pass, TP@5, APR, CPT, and dimension scores, while countable-only variants remain diagnostic rather than official.
  • Task-level bootstrap 95% confidence intervals resample complete episodes with replacement while preserving within-episode correlations.
  • The APR audit examines eligible transitions where a passing turn is followed by failure, using requests, artifacts, screenshots, traces, and runtime evidence.
  • Failed transitions are categorized as prior-regression-containing or new-requirement-only, and a transition may contain both prior regression and new-requirement failure.
  • Unexecuted downstream slots remain failures for Turn Pass, TP@5, and CPT.

D.2.1 Evaluator Evidence and Component Ablation

Evaluator decisions combine frozen artifact evidence with controlled ablations, diagnostic labeling, and human-reference comparisons to test how evidence sources support turn-level judgments.

  • The evaluator reruns decisions on 240 frozen artifact packets using full evidence or with generated code, screenshots, actor traces, or private references removed.
  • Ablation metrics compare automatic turn-level decisions with fixed reference labels, treating passing turns as the positive class.
  • A separate diagnostic protocol assigns executed non-passing calls to six mechanisms using turn context, visual, source/DOM, actor/tool, build, and output evidence.
  • The benchmark’s local results view links aggregate scores to requests, source, screenshots, actor traces, tool logs, backend state, and evaluator output for manual audit.
  • Generator protocol: The generator receives the current request, public contracts, compact prior-turn context, and latest source files while preserving the accumulated artifact across turns.
  • Evaluator protocol: The evaluator grades strictly from observable evidence and treats source code as runtime evidence insufficient on its own.
Loading 2608.29387v1…