PaperlayerView exact revision on arXiv

Source-linked AI summary

AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfaces

Zhijie Deng, Ling Li, Junhao Ji, Siwei Lyu, Zhipeng Xu, Zulong Chen, Rongyao Fang, Shuai Bai, Xuming Hu, Jiaheng Wei

arXiv:2609.34854v1cs.AI

TL;DR

Existing UI evaluations often assess aesthetic judgment and design action separately, leaving unclear whether success reflects shared aesthetic understanding. AUV-BENCH evaluates four tasks on shared interfaces and aligned diagnosis–repair cases, finding partial aesthetic competence but persistent gaps in precise diagnosis and coherent action.

  • Problem

    Existing evaluations typically assess UI aesthetic understanding and design action in isolation, limiting evidence about whether task successes reflect shared, transferable aesthetic understanding.

    Source: §1 · pp. 1–2

  • Method

    AUV-BENCH evaluates scoring, diagnosis, repair, and text-to-UI generation over 1,395 executable interfaces, with diagnosis and repair paired on 660 controlled-degradation cases.

    Source: §1 · pp. 2–3

  • Results

    Models show partial aesthetic competence, but exact diagnosis peaks at 24.7%, judgment and repair do not consistently align, and generated interfaces achieve only moderate human-calibrated aesthetic quality.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    Current models still lack the fine-grained understanding and judgment–action coherence required for reliable UI design.

    Source: §5 · p. 10

Abstract

from arXiv · show

Multimodal foundation models are increasingly used for evaluating and generating user interfaces (UIs), often producing seemingly reasonable aesthetic judgments and visually plausible pages. However, under professional design scrutiny, their behavior can differ substantially from that of human designers. In professional design practice, designers rely on a systematic set of aesthetic principles that consistently guide judgment, diagnosis, repair, and creation. A coherent aesthetic capability should therefore connect aesthetic judgment with design actions. Existing evaluations, however, typically assess these abilities in isolation, making it difficult to determine whether task-level success reflects a shared aesthetic understanding or merely fragmented task-specific competence. To address this gap, we introduce AUV-Bench, developed in collaboration with professional UI designers around 1,395 executable web interfaces and four tasks: aesthetic scoring, diagnosis, repair, and text-to-UI generation. The tasks share a pool of UIs and aesthetic principles, with diagnosis and repair further aligned on 660 controlled-degradation instances to enable instance-level analysis of judgment and action. Evaluation of 12 models reveals a capability imbalance: models show moderate agreement with professional designers in holistic aesthetic scoring, yet exact diagnosis-chain success peaks at only 24.7%. On the aligned diagnosis-repair cases, correct judgments and successful repairs do not consistently coincide, exposing a Judgment-Action Gap between identifying aesthetic problems and successfully acting on them. In open-ended generation, even leading models achieve only moderate aesthetic quality under human-calibrated evaluation. Overall, current models exhibit partial aesthetic competence, but still lack the fine-grained understanding and judgment-action coherence required for reliable UI design.

1 INTRODUCTION

AUV-BENCH addresses whether UI models share aesthetic understanding across judgment and design actions, rather than succeeding through isolated task-specific competence. Its four tasks and aligned diagnosis–repair cases reveal partial capability but persistent gaps in precise diagnosis and judgment–action coherence.

1 INTRODUCTION

  • Existing evaluations usually treat individual tasks separately, leaving limited support for testing shared aesthetic principles and instance-level links between judgment and corrective action.A successful generation does not directly establish reliable aesthetic judgment, nor does issue identification guarantee an effective repair.Source: §1 · p. 2
  • AUV-BENCH evaluates scoring, diagnosis, repair, and text-to-UI generation over a shared foundation of 1,395 executable web interfaces.Scoring and diagnosis assess aesthetic understanding; repair and text-to-UI generation assess design action.Source: §1 · p. 2
  • Exact fine-grained diagnosis peaks at only 24.7%, while correct judgments and successful repairs do not consistently coincide.Frontier models show moderate agreement with professional designers in holistic scoring, and generation remains only moderately aesthetic under human-calibrated evaluation.Source: §1 · pp. 2–3
  • The benchmark operationalizes professional aesthetic principles and pairs diagnosis with repair on 660 controlled-degradation instances for direct instance-level analysis.Its protocol includes 13 controlled source-level violations with executable verification.Source: §1 · p. 3

2 RELATED WORK

Prior UI evaluation has expanded from reconstruction and functionality toward design quality and action, while aesthetic-evaluation benchmarks have added finer-grained reasoning. AUV-BENCH distinguishes itself by evaluating four capabilities under shared professional principles and linking diagnosis with corrective action.

2 RELATED WORK

  • UI generation and front-end benchmarks increasingly extend evaluation beyond reconstruction and functionality toward design quality and downstream implementation.The cited work includes systems for reconstruction, code generation, refinement, executable correctness, and design reasoning.Source: §2 · p. 3
  • Aesthetic evaluation has moved from holistic preference prediction toward multimodal reasoning, localized critiques, defect localization, and critique actionability.The passage names AVA and NIMA as holistic examples and describes more recent benchmarks for finer-grained analysis.Source: §2 · p. 3
  • Earlier benchmarks typically assess aesthetic tasks separately or connect only a subset, whereas AUV-BENCH places scoring, diagnosis, repair, and generation under shared professional principles.This setup enables systematic analysis of how aesthetic judgment translates into design action.Source: §2 · p. 3

3 AUV-BENCH

AUV-BENCH evaluates four complementary UI aesthetic and design capabilities on shared executable interfaces and professional principles. Its evaluation protocol specifies task-specific metrics and aligns diagnosis with repair for instance-level analysis.

3.1 OVERVIEW

  • AUV-BENCH evaluates aesthetic scoring, diagnosis, repair, and text-to-UI generation using a shared pool of executable reference UIs and professional aesthetic guidelines.Diagnosis and repair share controlled-degradation instances for direct analysis of judgment–action alignment.Source: §3.1 · p. 4

3.3 TASK FORMULATION

  • Scoring predicts eight dimension-level ratings from a reference screenshot, diagnosis identifies degraded candidates and violations, repair patches source code, and generation creates a webpage from requirements and assets.Generation has no reference screenshot; diagnosis and repair share annotations for affected dimensions and regions.Source: §3.1 · p. 4, §3.3 · p. 4

3.2 DATASET STATISTICS

  • The benchmark contains 1,395 executable UIs, with scoring and generation using all pages and diagnosis and repair sharing 660 controlled-degradation cases.The interfaces span 11 industry categories, 12 page types, 386 web domains, and content in 29 languages; the benchmark has 4,110 task instances.Source: §3.2 · p. 4

3.4 BENCHMARK CONSTRUCTION

  • Designers organize principles across seven dimensions, adapt them to task needs, and define five operational diagnosis-and-repair dimensions with 13 executable violation rules.Controlled degradations are retained only when defects are perceptible and localized, match the injected recipe, and avoid unintended changes; generation requirements are reconstructed and reviewed by designers.Source: §3.4 · pp. 5–6

3.5 EVALUATION PROTOCOL

  • Aesthetic scoring compares predicted eight-dimensional ratings with designer mean opinion scores using Spearman rank correlation and mean absolute error, averaged equally across dimensions.Source: §3.5 · p. 6
  • Diagnosis evaluates degraded-candidate detection, dimension attribution, and localization, with Exact Chain Success requiring all three stages to be correct.Localization matches regions only on the correct version at an intersection-over-union threshold of 0.5.Source: §3.5 · p. 6
  • Repair success requires fixing all target violations while avoiding collateral damage, preserving content, and staying within the permitted scope; JAA measures instance-level association with correct diagnosis.JAA uses paired judgment and repair outcomes across 660 instances and is undefined when either outcome has zero variance.Source: §3.5 · pp. 6–7
  • Generation is evaluated with a dimension-calibrated aesthetic score and blind pairwise comparisons of models’ rendered pages.The calibration functions are learned from scoring-task designer ratings and frozen before generation evaluation.Source: §3.5 · pp. 6–7

4 EXPERIMENTS

Across scoring, diagnosis, repair, and generation, current models show partial aesthetic competence but struggle with precise diagnosis, reliable judgment–action alignment, and open-ended design quality. The experiments also probe attribute sensitivity, visual grounding in repair, and whether a coding harness improves generation.

4.1 EXPERIMENTAL SETUP

  • The experiments evaluate frontier, open-weight, and aesthetics-specialized multimodal models under task-specific protocols.The study uses professional designer annotations and a shared foundation of 1,395 executable web interfaces across its four tasks.Source: §4.1 · p. 8

4.2 MAIN RESULTS

  • Exact diagnosis-chain success peaks at only 24.7%, despite frontier models showing moderate agreement with professional designers in holistic aesthetic scoring.Models can distinguish degraded interfaces yet struggle to identify the violated principle and its location.Source: §4.2 · p. 8
  • JAA ranges from 0.07 to 0.44, but correct diagnoses and successful repairs do not consistently coincide.Successful repairs can occur without correct explicit judgment, and correct judgments do not always lead to successful repairs.Source: §4.2 · p. 8
  • Human-calibrated absolute aesthetic scores remain 2.78–3.18 out of 5, even as some models approach 80% Pairwise Win Rates.Relative wins over competing models do not necessarily indicate strong absolute design quality.Source: §4.2 · p. 8

4.3 IN-DEPTH ANALYSIS

  • For 83.5% and 86.4% of samples, Qwen3.7-Plus and Qwen3.8-Max overall scores remain unchanged after degradation, indicating weak principle-level sensitivity.Typography degradation lowers the corresponding score by 0.078 for Qwen3.8-Max and 0.027 for Qwen3.7-Plus.Source: §4.3 · p. 9
  • Removing screenshots reduces repair Pass only from 34.2% to 33.2% for Qwen3.7-Plus and 57.3% to 53.8% for Kimi-K3, but from 54.7% to 25.3% for Qwen3.8-Max.The model-level variation suggests some models can exploit HTML structure rather than fully grounding repair in visual input.Source: §4.3 · p. 9
  • Correct diagnosis improves repair by +8.7, +24.2, and +17.2 percentage points across three models, yet Figure 5 shows judgment and repair outcomes remain misaligned.For Kimi-K3, repair performance drops by 17.5 points when judgment is incorrect.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Claude Code does not consistently improve calibrated aesthetic scores clustered around 3.0, and reshuffles model ranks: Qwen3.7-Plus seventh to third, Qwen3.8-Max second to fourth.The harness enables iterative code generation and editing; additional code manipulation alone does not consistently translate into higher aesthetic quality.Source: §4.3 · pp. 9–10, Table 3 · p. 9

5 CONCLUSION

AUV-BENCH jointly evaluates UI aesthetic judgment and design action across four tasks. The authors conclude that models show partial competence but still struggle with fine-grained diagnosis, open-ended generation, and consistent judgment–action alignment.

5 CONCLUSION

  • AUV-BENCH jointly evaluates UI aesthetic judgment and design action across scoring, diagnosis, repair, and generation.The conclusion identifies fine-grained diagnosis, open-ended generation, and consistent judgment–action alignment as continuing challenges for current models.Source: §5 · p. 10

AI USE STATEMENT

The section describes a professional-designer-developed framework for operationalizing UI aesthetics through design rules and specified quantitative or programmatic checks, including shadow and consistency checks. It also documents generative AI’s role and review or validation of its outputs, alongside designer calibration and quality control for producing benchmark ratings and ground-truth scores.

AI USE STATEMENT

  • Generative AI supported manuscript polishing, code development, benchmark construction, and Text-to-UI aesthetic judging, with author review and validation where specified.GPT-5.4’s scoring was validated and calibrated against professional-designer annotations before evaluating generation results; AI-assisted code was reviewed, verified, and tested by multiple authors.Source: AI USE STATEMENT · p. 10

A AESTHETIC DESIGN PRINCIPLES

  • The guideline organizes professional UI aesthetic principles into seven dimensions and operationalizes applicable rules for evaluation.It draws on HCI and visual-design principles, industry conventions, and explicit standards, emphasizing properties describable from rendered interfaces or design sources.Source: A AESTHETIC DESIGN PRINCIPLES · p. 16

A.1 COLOR

  • Color rules distinguish theme, functional, and neutral roles, prescribe a 60:30:10 allocation, set contrast thresholds, and require functional colors to follow semantic expectations.Normal text requires a 4.5:1 contrast ratio; large text and graphics or interface components require 3:1.Source: §A.1 · pp. 17–18

A.2 SPACING

  • Spacing follows an 8-pixel base grid, with 4- and 12-pixel increments also permitted for denser layouts and smaller gaps.The specified scale includes values such as 4, 8, 12, 16, 24, 32, 40, 48, 64, 96, and 160 pixels.Source: §A.2 · p. 19

A.3 LAYOUT

  • Layout guidance places frequent desktop controls 5%–12% of the viewport from an edge, groups similar blocks in sets of at most five, and follows the upper-left-to-lower-right reading flow.The guideline recommends placing important identifying information upper-left and important operations lower-right.Source: §A.3 · p. 20

A.4 TYPOGRAPHY AND TEXT LAYOUT

  • Typography rules limit typefaces and sizes, prescribe an additive line-height rule and preferred font weights, and specify line-length and alignment guidance.The guideline recommends no more than five type sizes, no more than three weights from {400, 500, 600}, at most 85 characters per line, and left-aligned text with right-aligned numerals in tables and lists.Source: §A.4 · pp. 21–22

A.5 IMAGERY

  • Imagery guidance addresses aspect ratio and recognizability, recommending image-quality assessment algorithms for interface image regions.The guideline identifies commonly used image proportions, but the supplied passage does not enumerate the complete admissible set.Source: §A.5 · pp. 23–24

A.6 SHADOW

  • Shadow guidance assigns interface elements to four semantic elevation levels, matching shadow treatment to each component’s layer and role.Level 0 has no visible shadow; higher levels cover navigation or hover states, associated expansions such as drop-downs and drawers, and prominent prompts such as dialogs.Source: §A.6 · pp. 24–25
  • Shadow levels 1–3 use equal-magnitude offsets in five specified directions with zero spread, and implementation compares extracted component shadow parameters against the semantic level for each component type.The directions are (d, d), (0, −d), (d, 0), (0, d), and (−d, 0).Source: §A.6 · p. 26

A.7 CONSISTENCY

  • Consistency requires equivalent containers to share corner radii, shadows, and inter-container spacing, while instances of each component type follow one shared specification.Implementation identifies equivalent containers or component instances and compares their parameters across instances or within each component class.Source: §A.7 · pp. 26–27

B.1 ANNOTATION PROCEDURE

  • Three designers independently rated each of 1,395 reference UIs across eight aesthetic dimensions, producing 33,480 dimension-level ratings before quality control; all annotators first reviewed shared rubric exemplars and scoring guidelines.Source: §B.1 · pp. 27–28
  • The shared rubric groups fine-grained principles into eight screenshot-judgable dimensions, combining spacing with spatial organization under layout while separately assessing component and visual-style consistency.The dimensions are color, typography, graphics and imagery, layout, component consistency, visual style consistency, copy quality, and Image–Text Fit; shadow is assessed within visual style consistency.Source: §B.1 · p. 29, Table 7 · p. 29

B.2 EVALUATION CRITERIA AND GROUND-TRUTH DISTRIBUTION

  • For each UI and aesthetic dimension, ground truth is the one-decimal mean opinion score across retained expert ratings, with no additional rating-level outlier filtering.The benchmark contains eight dimension-level MOS scores per UI; Figure 24 compares individual annotator and aggregated-score distributions, mapping MOS to the nearest rubric level for visualization only.Source: §B.1 · p. 29, §B.2 · p. 29

B.3 INTER-ANNOTATOR RELIABILITY

The appendix reports professional annotator reliability and documents the model-evaluation prompt, annotation examples, and reference-UI quality checks.

B.3 INTER-ANNOTATOR RELIABILITY

  • Ordinal Krippendorff’s α ranges from 0.328 to 0.431 across dimensions, averaging 0.368; Layout is highest at 0.431 and Color lowest at 0.328.Visual-style Consistency, Copy Quality, and Image–Text Fit have α values of 0.366, 0.334, and 0.402, respectively.Source: §B.3 · pp. 29–30, Table 8 · p. 30
  • ICC(1,3) ranges from 0.596 to 0.694, with a 0.637 macro-average, compared with ICC(1,1) values of 0.329 to 0.431.The authors report that aggregating three professional ratings improves reliability over a single rating, while inter-expert variation remains.Source: §B.3 · p. 30

B.4 MODEL EVALUATION PROMPT

B.5 ANNOTATION EXAMPLES

  • Three representative examples show each UI alongside independent ratings and qualitative comments from three professional designers.The examples illustrate human evaluation and ground-truth aggregation.Source: §B.5 · p. 30, Figure 28 · p. 34, Figure 29 · p. 34

C.1 REFERENCE UI COLLECTION AND QUALITY ASSURANCE

  • Reference-page checks localize dependencies, re-render pages against original screenshots, and filter or inspect substantial visual discrepancies.For the checked subset, the retained pages have a median SSIM of 0.994 and meet the MAE < 10 requirement; designers also inspect rendering quality, asset completeness, and layout integrity.Source: §C.1 · pp. 32–34

C.2 CONTROLLED AESTHETIC VIOLATIONS AND TASK CONSTRUCTION

The benchmark constructs controlled, traceable aesthetic defects only in applicable UI contexts, then aligns diagnosis and repair on the same pages and annotations. Text-to-UI generation uses designer-reviewed requirements that preserve functional intent while leaving visual choices open.

C.2 CONTROLLED AESTHETIC VIOLATIONS AND TASK CONSTRUCTION

  • Thirteen rules cover five aesthetic dimensions and are instantiated only when a UI meets the rule’s structural or semantic context and the degradation is executable and verifiable.Rules with visually insignificant or unreliable attribution effects are excluded; each retained defect has an explicit target, design principle, and traceable modification.Source: §C.2 · p. 35, Table 9 · p. 36
  • The 660 diagnosis and repair instances share 1,209 controlled defects, enabling matched evaluation of identifying issues and taking corrective action on the same pages.The cohort comprises 334 single-defect, 266 multi-defect combo, and 60 stress instances; defect distributions reflect rule applicability rather than forced balance.Source: §C.2 · pp. 36–37, Figure 31 · p. 37
  • The controlled cases progress from single violations to 2–3 co-occurring defects and 4–6 simultaneous defects, testing rule attribution, search completeness, and defect retrieval.Each modified page is rendered and checked for a perceptible intended violation, localization, and agreement between detected violations and the injected recipe.Source: §C.2 · pp. 36–37
  • 56.4% source-identification accuracy was only modestly above chance, suggesting controlled degradations lack readily identifiable synthetic artifacts and resemble natural defects of the same type.The blind comparison covered 65 real–controlled defect pairs assessed by three non-experts.Source: §C.2 · pp. 37–38

C.3 REQUIREMENT RECONSTRUCTION

  • Task 4 requirements describe page purpose, information structure, functional modules, and visual direction, while omitting exact visual parameters and preserving designer-reviewed checklist intent.Models receive the requirement, original textual content, and image assets, but not the reference screenshot; visual implementation choices remain theirs.Source: §C.3 · p. 38

D EXPERIMENTAL DETAILS

The experimental setup standardizes model instructions and preprocessing within each task. Long-page screenshots retain both global structure and local detail, while generation uses filtered local assets and reconstructed requirements without reference screenshots.

D.1 MODEL AND INFERENCE SETTINGS

  • All models within each task use the same instructions, input format, and evaluation protocol; experiments use English prompts at temperature 0, with task-specific output limits and retries.Tasks 1–2 allow 2,000 tokens, Task 3 allows 8,000, and Task 4 has no explicit maximum output length.Source: §D.1 · p. 38

D.2 INPUT PREPROCESSING AND ASSET ORGANIZATION

  • For screenshots taller than 1,800 pixels, preprocessing uses 1,800-pixel slices with 180-pixel overlap and a bottom-aligned final slice when needed.A full-page overview accompanies multi-slice inputs; the model receives it first, followed by slices in top-to-bottom order.Source: §D.2 · p. 38
  • Task 3 receives the degraded UI representation, its HTML source, and the structured diagnosis produced by the same model in Task 2.For diagnosis, clean and degraded screenshots are processed independently and presented in randomized order.Source: §D.2 · p. 39
  • Task 4 supplies requirements, original page text, and at most 32 filtered visual assets, but no reference screenshot or external image sources.Assets are presented through contact sheets and textual descriptions; requirements are in English while visible page text retains its source language.Source: §D.2 · p. 39

E.1 VALIDATION AND CALIBRATION OF GPT-5.4 AS AN AUTOMATIC AESTHETIC JUDGE

The authors validate GPT-5.4 as an automatic aesthetic judge against professional-designer ratings and calibrate its scores using Task 1 annotations. Calibration reduces numerical error and bias, while modestly lowering ranking agreement.

VALIDATION AND CALIBRATION

  • Raw GPT-5.4 ratings had macro-average SRCC 0.570, macro-average MAE 0.950, and mean signed error +0.808, indicating moderate ranking agreement with systematic overestimation.The positive bias was consistent across all eight dimensions.Source: §E.1 · p. 39, Table 10 · p. 40
  • Dimension-specific non-decreasing isotonic mappings convert raw 1–5 ratings to designer MOS and are fitted only on Task 1 annotations before being frozen for Task 4.The resulting judge score averages calibrated ratings across dimensions; invalid or failed generations receive zero.Source: §E.1 · pp. 39–40
  • MAE fell from 0.950 to 0.607, a 36.1% reduction, after five-fold calibration; macro-average signed bias decreased from +0.808 to approximately zero.Table 10 reports pooled out-of-fold results across eight aesthetic dimensions.Source: Table 10 · p. 40, §E.1 · p. 40
  • Calibration reduced macro-average SRCC from 0.570 to 0.508, largely because piecewise-constant isotonic mappings introduced additional ties.The calibration targets numerical agreement with the designer rating scale rather than ranking performance.Source: §E.1 · p. 40, Table 10 · p. 40

E.2 ADDITIONAL FINE-GRAINED ANALYSES

The analyses show that increasingly dense defects leave coarse detection comparatively stable but make fine-grained diagnosis and repair harder. They also reveal distinct dimension-level profiles in scoring, generation, and repair.

Effect of Defect Complexity

  • Claude Opus 5’s ECS fell from 0.374 to 0.139 to 0.017, while Repair Pass fell from 0.859 to 0.680 to 0.517 across single, combo, and stress settings.GPT-5.6 Sol shows the same trend: ECS declines from 0.311 to 0.079 to 0.050 and Repair Pass from 0.850 to 0.466 to 0.400; detection remains comparatively stable.Source: §E.2 · pp. 40–41, Figure 32 · p. 41

Dimension-Wise Aesthetic Capability Profiles

  • Layout is the strongest scoring dimension for leading models (SRCC 0.615–0.656), while copy quality is more difficult (SRCC 0.339–0.449).The authors interpret this as stronger capture of global spatial organization than contextual appropriateness and quality of interface copy.Source: §E.2 · p. 41, Figure 33 · p. 41
  • Generated interfaces score highest on component and visual-style consistency and weakest on graphics and imagery; GPT-5.6 Sol scores 3.349 versus 2.995, and Kimi-K3 scores 3.417 versus 2.996, respectively.The human-calibrated dimension scores distinguish generation profiles from scoring agreement profiles.Source: §E.2 · p. 41, Figure 33 · p. 41

Cross-Industry Analysis

  • Figure 34 compares group-average performance across 11 industries for scoring, diagnosis, repair, and generation; comparisons are intended within each task’s original metric scale.Source: Figure 34 · p. 42

What Limits Successful Repair?

  • Qwen3.7-Plus has 0.995 collateral preservation and perfect content preservation but 0.371 target-fix success and 0.342 Repair Pass; Claude Opus 5 reaches 0.826 target correction and 0.824 visual-scope compliance.The analysis identifies target correction and visual-scope compliance as more restrictive than collateral and content preservation for most frontier models.Source: §E.2 · p. 42

E.3 PERFORMANCE ACROSS INDUSTRIES

Performance varies by industry in aesthetic scoring, diagnosis, and repair, while text-to-UI generation is comparatively stable. Industry effects persist after controlling for model strength and are most pronounced when models must attribute specific problems and repair them.

E.3 PERFORMANCE ACROSS INDUSTRIES

  • The analysis covers industry-level patterns, individual frontier models, model-strength-controlled effects, and variation along the diagnosis–repair pipeline.Table 11 decomposes Aesthetic Repair performance into target fix, collateral preservation, content preservation, and visual-scope compliance.Source: §E.3 · p. 42, Table 11 · p. 42
  • Frontier-model SRCC ranges from 0.293 on E-commerce to 0.673 on Social & Entertainment, while Repair Pass ranges from 0.505 on E-commerce to 0.696 on Local Services.Frontier-model Text-to-UI Generation Pairwise Win Rate remains around 0.69–0.72 across industries; open-weight performance is especially low in Diagnosis and Repair.Source: §E.3 · p. 42
  • Across individual frontier models, E-commerce scoring is generally weaker and Local Services and Social & Entertainment stronger, while diagnosis and repair vary substantially between models.ECS approaches zero across many categories for some models; within-model generation is more stable, with variation driven more by differences between models.Source: §E.3 · p. 42, Figure 35 · p. 43
  • After controlling for model strength, frontier models remain below overall performance on E-commerce in Scoring (−0.256), Diagnosis (−0.043), and Repair (−0.069).Local Services and Social & Entertainment show positive shifts in Scoring and Repair; normalized Text-to-UI Pairwise Win Rate deviations are mostly within about ±0.02.Source: §E.3 · p. 43, Figure 36 · p. 43
  • Frontier-model degradation detection stays around 0.74–0.85 across industries, but greater variation emerges in attribution, exact-chain diagnosis, and downstream repair.Open-weight models retain non-trivial coarse detection ability but drop sharply on attribution, ECS, and Repair, suggesting industry-specific challenges arise at identifying and acting on specific problems.Source: §E.3 · p. 43, Figure 37 · p. 44
Loading 2609.34854v1…