Source-linked AI summary
Seeing Without Understanding: Large Language Model Evaluation of Mobile User Interface Quality, Failure Taxonomy, and Architectural Explanation
Md Rejaul Korim Sadi, Golam Mostofa Naeem, Toufiqur Rahman Tasin, Syed Mostofa Moosa, Mahmudul Hasan Emon, Mahmudur Rashid, Ferdus Ahmed
TL;DR
Mobile UI quality evaluation lacks a scalable, systematically validated alternative to costly heuristics and non-scalable human annotation. The paper evaluates three open-weight language models against a filtered RICO corpus and heuristic baseline, finding chance-level agreement in most evaluation cells, structural-feature decoupling, systematic usability under-rating, and modality-sensitive scores. It interprets these patterns through three transformer-related failure mechanisms while bounding the conclusions to open-weight models and a constructed heuristic baseline.
Problem
Scalable, reliable evidence is lacking on whether language models can substitute for heuristic or human mobile UI quality judgments and why they fail.
Method
The study evaluates three open-weight models on 15,000 filtered RICO screens in JSON and JPEG modes, comparing 270,000 ratings with a heuristic baseline using agreement, Kappa, and correlation analyses.
Results
Agreement with the heuristic baseline is statistically equivalent to chance in 17 of 18 cells, while models underrate usability and fail to track structural features that the heuristic correlates with.
Takeaways & Limitations
The observed divergences align with attention misgrounding, MLE plausibility bias, and autoregressive over-commitment as architectural explanations of model UI-judgment failures.
Takeaways & Limitations
The study is restricted to open-weight model families, and its heuristic baseline is constructed rather than human-rated.
Abstract
from arXiv · showhide
Evaluating mobile user interface quality at scale remains a persistent challenge in software engineering and human-computer interaction. Rule-based heuristic methods offer structural reliability but demand significant engineering effort, while human annotation does not scale to the volume of applications produced annually. Large language models present a promising alternative, yet their reliability for structured UI judgment has not been systematically examined, and the patterns behind their failures remain insufficiently characterized. This paper addresses both gaps. We begin with the complete RICO dataset of 66,261 real-world mobile application screens, from which we derive a refined evaluation corpus of 15,000 screens through a rigorous, literature-guided selection process. Each screen is assessed across seven criteria: structural JSON validity, minimum visible element count, clickable component presence, non-zero layout bounds, image integrity, and perceptual duplicate removal. Against this corpus, we apply a heuristic baseline built from severity-weighted usability signals, normalized layout metrics, and pixel-ratio complexity measures calibrated to real user sentiment. Multiple language models independently rate each screen across usability, layout quality, and visual complexity from structured JSON descriptions and raw screenshots. Dimension-level comparison against the heuristic uses agreement rates, Cohen's Kappa, and confidence calibration. Recurring divergence patterns are organized into a failure taxonomy and interpreted through transformer architectural signatures: MLE plausibility bias, attention misgrounding, and autoregressive over-commitment.
I. INTRODUCTION
The paper examines whether language models can reliably judge mobile UI quality at scale and why their judgments diverge from heuristic evaluation. It constructs a large evaluation corpus, compares model ratings with a structural baseline, and interprets recurring failures through transformer mechanisms.
- Human annotation does not scale, while heuristic methods are reliable but costly and narrow, motivating language models as a third evaluation candidate.
- A 15,000-screen corpus is evaluated with a heuristic baseline and three language models across three dimensions and two input modalities.The study produces 270,000 ratings and compares them using agreement rates, Cohen’s Kappa, and feature-level correlations.
- 17 of 18 evaluation cells show inter-rater agreement with the structural baseline statistically equivalent to chance.Models also systematically underrate usability by approximately one full scale point.
- The heuristic correlates with structural features at coefficients reaching ρ = 0.94 on visual complexity, whereas language models fail to track those features across models and modalities.This pattern persists from a 27B-parameter model to a 235B-parameter vision-capable model.
- The paper links structural feature decoupling, plausibility-driven distributions, and modality-induced inconsistency to attention misgrounding, MLE plausibility bias, and autoregressive commitment.The companion framework supplies architectural explanations for the observed rating signatures rather than merely cataloging errors.
II. RELATED WORK
Prior work provides heuristic UI metrics, scalable evaluation practices, and hallucination taxonomies, but does not fully explain the architectural origins of unreliable mobile UI judgments. This paper positions its contribution at the intersection of these strands while distinguishing quality judgment from UI task completion.
- Heuristic UI evaluation evolved from usability and cognitive-load principles toward continuous structural and aesthetic metrics derived from UI hierarchies.
- Modern language-model evaluation contributes standardized logging, chain-of-thought prompting, and evidence that outputs are sensitive to prompt phrasing.
- Existing hallucination taxonomies distinguish intrinsic from extrinsic errors and faithfulness from factuality, but do not identify the architectural mechanism producing each error.The companion framework addresses this mechanism-level gap.
- Mobile UI research has applied language models to conversational task completion, whereas this paper evaluates direct quality judgment.The studies share a dataset family but ask models to produce different outputs.
III. METHOD
The method builds a filtered RICO-based corpus and a reproducible heuristic scorer designed to preserve continuous variation in UI quality. The resulting baseline combines usability, layout, and visual-complexity signals grounded in prior UI research.
- The study starts from 66,261 RICO screens containing JPEG screenshots and hierarchical JSON UI metadata, then retains 15,000 screens after quality filtering.
- Seven filters enforce valid JSON, at least three visible elements, a clickable component, non-zero bounds, valid images, duplicate removal, and successful category derivation.
- The heuristic baseline scores usability, layout quality, and visual complexity on a 1–5 scale using severity-weighted usability signals and normalized layout metrics.Its inputs include touch-target size, cognitive-load decay, overlap, text presence, whitespace, and alignment measures.
- Normalized weighted scoring replaces fixed deductions to preserve continuous variation rather than collapsing screens into a few stepped values.The preserved variation supports the agreement statistics used later in the analysis.
C. Language Model Evaluation
Three open-weight models rate every selected screen from structured JSON and JPEG inputs using a common UI-quality prompt. Their outputs are compared with heuristic scores and structural features across dimensions, models, and modalities.
- Three open-weight models—gemma3, gemma4, and qwen3-vl—are evaluated in JSON and JPEG modes.The models span 27B, 31B, and 235B parameters, with qwen3-vl being instruction-tuned and vision-capable.
- The prompt requests usability, layout quality, and visual complexity ratings on a 1.0–5.0 scale with exactly two decimal places.
- Figure 1 compares filled heuristic distributions with solid JSON-mode and dashed JPEG-mode model distributions for each dimension and model.The comparison focuses on whether model ratings reproduce, compress, or distort the heuristic distribution.
- 15,000 × 3 × 2 = 90,000 model invocations produce 270,000 individual ratings.Raw responses are archived, and retry logging with incremental writes supports recovery after interruption.
D. Statistical Analysis
The evaluation compares language-model scores with a heuristic baseline using agreement, correlation, difference, and compression metrics. Agreement is essentially chance across the 18 cells, while the heuristic tracks structural features that language models do not.
- Metrics: The analysis computes agreement percentage, Pearson correlation, linearly-weighted Cohen’s Kappa, mean differences, compression ratio, and structural-feature correlations for every evaluation cell.Structural correlations test whether ratings track total_elements and clickable_elements like the heuristic does.
- Agreement: Agreement is essentially chance in 17 of 18 evaluation cells, with only gemma3 (JSON) on visual complexity exceeding κ = 0.15.The strongest cell reaches κ = 0.227; the remaining cells are statistically indistinguishable from chance.
- Structural correlations: Language-model ratings show no comparable relationship to either structural feature despite receiving the same structural input.The comparison isolates whether models use the available JSON features in a way consistent with the heuristic.
B. Systematic Underrating of Usability
Language models systematically underrate usability relative to the heuristic, while failing to track the structural features that inform the baseline. The divergence is consistent across models and modalities.
- Systematic underrating: Mean usability differences range from −1.40 for gemma3 JSON to −0.19 for qwen3-vl JPEG across all six model–modality combinations.Four of six combinations exceed one full scale point in magnitude, and the direction is consistent across model families and modalities.
- Structural correlations: The heuristic correlates with total_elements at ρ = +0.94 on visual complexity and −0.55 on usability.It correlates with clickable_elements at ρ = +0.74 on visual complexity and −0.73 on usability.
- Structural decoupling: Language models given identical structural quantities produce uniformly weak correlations with those features.The models therefore do not reproduce the heuristic’s direct relationships to total_elements and clickable_elements.
- Distributional pattern: Usability differences shift left for every model, whereas visual-complexity differences exhibit model-specific drift.The distributions measure LLM minus heuristic scores, so leftward mass denotes underrating.
- Modality sensitivity: Cross-modality Pearson correlations range from 0.00 to 0.18, indicating that scores change substantially when the same screen is represented as JSON rather than JPEG.The models are sensitive to input modality rather than invariant to it.
D. Distributional Compression Is Model-Dependent
Score compression is model-dependent rather than universal, and changing the input from structured JSON to raw JPEG produces very low cross-modality consistency.
- Distributional compression: Compression ratios are 0.92 for gemma3 and 0.97 for qwen3-vl, while gemma4 expands the distribution with a ratio of 1.25.Ratios below one indicate narrower output distributions than the heuristic’s; gemma4 instead produces a heavier high-magnitude disagreement tail on visual complexity.
- Cross-modality consistency: The same screen receives poorly aligned scores across JSON and JPEG inputs, with Pearson r values between 0.00 and 0.18 across all nine panels.This comparison tests cross-modality consistency for each model and dimension.
- Cross-modality consistency: Models are sensitive to input modality rather than invariant to it.The low correlations apply across all model–dimension pairs.
V. DISCUSSION
The study tests an architectural account of hallucination against large-scale UI ratings and finds evidence for structural misgrounding, model-dependent distributional distortion, and modality-sensitive surface dependence.
- Architectural Mechanisms: The heuristic tracks structural UI features, but language models do not, even when element counts and other quantities appear explicitly in the input.The feature-correlation gap is consistent across all model families and both modalities.
- Architectural Mechanisms: Two of three models compress their output distributions below the heuristic’s range, while gemma4 expands variance without improving agreement.The compression pattern is therefore model-dependent rather than universal.
- Architectural Mechanisms: Cross-modality correlations range from 0.00 to 0.18, showing that identical screens receive substantially different ratings when represented as JSON versus JPEG.The result links rating inconsistency to surface representation rather than only the underlying interface.
- Architectural Mechanisms: The paper identifies three manifestations of compound failure: structural feature decoupling, output distributional shift, and modality-induced rating inconsistency.These operate at structural, distributional, and modality resolutions in the data.
B. Failure Taxonomy
The failure taxonomy organizes model errors into structural blindness, usability collapse, modality surface dependence, and variance distortion, connecting each pattern to architectural behavior.
- Failure Patterns: Structural blindness occurs when models receive element counts, clickable density, nesting depth, and bounds but produce ratings that do not reflect them.This accounts for the feature-correlation gap and is interpreted as the rating-task expression of attention misgrounding.
- Failure Patterns: Usability collapse shifts ratings leftward by approximately one full scale point across all six model–modality cells.The paper interprets this pattern as plausibility bias filtered through training-data priors.
- Failure Patterns: Modality surface dependence makes the same screen receive different ratings as JSON or screenshot, with patterned disagreement rather than random noise.The pattern is presented as a response-level expression of surface-form dependence.
- Failure Patterns: Variance distortion takes opposite forms: two models compress their output ranges, while one expands visual-complexity variance disproportionately.Both directions are inconsistent with the heuristic, indicating that distributional fidelity is not universal.
C. Implications
The results argue that current language models are unreliable substitutes for heuristics or human judgment in structured UI evaluation, while supporting the proposed architectural account within defined scope limits.
- Implications: Language models cannot reliably substitute for rule-based heuristics or human judgment because agreement is near chance, usability is systematically biased, and cross-modality reproducibility is poor.The practical conclusion concerns structured UI quality evaluation in the present model setting.
- Implications: The architectural framework receives applied confirmation: attention misgrounding, MLE plausibility bias, and surface pattern dependence each appear in the rating data.MLE compression is confirmed for two models, while the third shows variance expansion consistent with poor anchoring.
- Implications: The study is limited to open-weight model families on Ollama Cloud, so proprietary models with different training and alignment pipelines may show different signatures.This limitation is especially relevant to MLE compression.
- Implications: The heuristic baseline is constructed rather than human-rated, although the central structural-feature finding does not depend on absolute baseline correctness.The paper treats reproducibility and inspectability as advantages of the heuristic baseline.
- Implications: Comparative architectural studies varying model families, training data, and alignment methods could convert cross-model variance into mechanism-specific evidence.The proposed direction targets the inconsistent strength of MLE-compression signatures.
VII. CONCLUSION
The study finds that current language models process mobile UI representations without reliably understanding UI quality. Their failures include chance-level agreement, modality sensitivity, and architectural patterns consistent with surface-pattern processing.
- Three open-weight language models rated 15,000 screens across three UI quality dimensions and two input modalities.
- Agreement with the heuristic baseline was statistically equivalent to chance in 17 of 18 evaluation cells.
- The same screen received different scores when represented as JSON rather than an image.
- Observed failures aligned with attention substituting co-occurrence for structure, MLE plausibility bias, and transformer reliance on surface patterns.
- The study concludes that language models cannot yet reliably substitute for heuristics or human judgment in structured UI quality evaluation.
- Code, data, summary statistics, figures, and an executable reproduction notebook are publicly available.