PaperlayerView exact revision on arXiv

Source-linked AI summary

MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces

Reuben A. Luera, Ryan Rossi, Franck Dernoncourt, Samyadeep Basu, Sungchul Kim, Subhojyoti Mukherjee, Puneet Mathur, Ruiyi Zhang, Jihyung Kil, Nedim Lipka, Seunghyun Yoon, Jiuxiang Gu, Zichao Wang, Cindy Xiong Bearfield, Branislav Kveton

arXiv:2510.08783v1cs.HCcs.AI

TL;DR

This paper asks whether MLLMs can approximate human judgments of varied user interfaces, beyond behavioral measures such as clicks and conversions. It compares model and human ratings and pairwise preferences, finding moderate alignment on some judgments but weaker performance on specific factors and closer pairwise agreement when screens differ more.

  • Problem

    Prior evaluation work often focuses on behavioral outcomes such as clicks or conversions, while this study examines whether MLLMs can approximate subjective human judgments of varied interfaces.

    Source: §1 · p. 2

  • Method

    The authors compare three MLLMs with crowdsourced human evaluations of 30 interfaces using absolute Likert ratings and pairwise preferences across UI factors.

    Source: §1 · p. 2

  • Results

    MLLMs do moderately well on overall absolute UI scores but struggle with specific factors; pairwise performance is better when the compared screens differ more.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings support using MLLMs to supplement early UI research, particularly for screens that might otherwise remain untested, rather than replacing human evaluation.

    Source: §1 · p. 2

    The study evaluates static interfaces, primarily professionally made under Western design standards, and does not cover the diverse, dynamic interactions of user research.

    Source: §5.5 · p. 10

Abstract

from arXiv · show

In an ideal design pipeline, user interface (UI) design is intertwined with user research to validate decisions, yet studies are often resource-constrained during early exploration. Recent advances in multimodal large language models (MLLMs) offer a promising opportunity to act as early evaluators, helping designers narrow options before formal testing. Unlike prior work that emphasizes user behavior in narrow domains such as e-commerce with metrics like clicks or conversions, we focus on subjective user evaluations across varied interfaces. We investigate whether MLLMs can mimic human preferences when evaluating individual UIs and comparing them. Using data from a crowdsourcing platform, we benchmark GPT-4o, Claude, and Llama across 30 interfaces and examine alignment with human judgments on multiple UI factors. Our results show that MLLMs approximate human preferences on some dimensions but diverge on others, underscoring both their potential and limitations in supplementing early UX research.

1 INTRODUCTION

UI research helps refine usability, trust, productivity, and emotional responses, but limited researcher capacity forces organizations to test only a subset of design ideas. This paper compares MLLM judgments with human evaluations across absolute ratings and pairwise UI comparisons, aiming to identify where they align and diverge.

1 INTRODUCTION

  • The authors frame user research as important because interface choices shape perceived usability and trustworthiness, and can affect productivity and emotional responses.Research-informed design iterations can refine products to be more usable, engaging, and trustworthy.Source: §1 · p. 1
  • Limited research capacity leaves organizations testing only a small subset of ideas rather than conducting rapid, iterative evaluation.The cited Nielsen Norman Group report estimates one researcher for every five designers.Source: §1 · p. 1
  • The study moves beyond behavioral metrics such as clicks and conversions to compare MLLM judgments with human evaluations of subjective UI qualities.The authors examine both Likert-scale evaluations and pairwise comparisons of interfaces.Source: §1 · pp. 1–2
  • MLLMs show promise as UI-judge approximators, but the authors position them as supplements—not replacements—for human evaluation, especially for otherwise untested UIs.For absolute ratings, all three models exceeded 75% accuracy within ±1 score; pairwise performance improved as screen differences increased.Source: §1 · p. 2

2 RELATED WORK

Prior LLM-as-a-judge work spans domains but raises concerns about consistency and bias. UI research has explored design feedback and simulation, while this paper benchmarks multiple MLLMs against human judgments across varied interfaces and evaluation tasks.

2.1 LLM-as-a-Judge

  • Prior work presents LLM judges as flexible, potentially cost-cutting alternatives to human evaluation, while noting concerns about consistency, bias, and reliability.Even strong models can deviate by up to five points on a 10-point scale, according to the cited related work.Source: §2.1 · p. 2
  • The paper uses pairwise MLLM evaluation, in which a model compares two options against a criterion, alongside human pairwise tests and Likert-scale evaluations.The authors describe the pairwise method as closely aligned with A/B testing in user research.Source: §2.1 · p. 2

2.2 MLLMs in User Research and Design

  • Earlier LLM research also used simulated respondents for surveys, with reported agreement of about 75%–85% in larger-scale market research and a stated need for human data to capture user nuance.The figure illustrates sample UIs from landing pages, digital receipts, and catalogs; participants saw high-fidelity, branded, in-product versions.Source: §2.2 · p. 3, Figure 1 · p. 3
  • Prior UI work includes design-heuristic assessment, critique-based feedback, and simulated-user tools, while this study compares several MLLMs with human judgments across absolute and pairwise tasks.The benchmark evaluates nine UI factors across screens from different domains, rather than focusing solely on e-commerce click-through rates.Source: §2.2 · p. 4

3 STUDY METHODOLOGY

The study selected 30 interfaces, defined nine rating factors, and collected human judgments through crowdsourcing. After filtering responses, the authors compared the remaining human ratings with MLLM scores.

3.1 UI Collection

  • The researchers selected 30 interfaces across three categories, balancing design components, styles, complexity, quality, and purpose.The set included 24 professionally made interfaces and six made from scratch; some screens were adjusted to maintain a range of build quality.Source: §3.1 · p. 4

3.2 Factors

  • The study rated interfaces on nine 1–7 Likert-scale factors grouped into cognitive, perceptual, and emotional themes.The factor framework was chosen after consulting literature on reactions to designs and visualizations.Source: §3.2 · p. 4, Table 1 · p. 4

3.2.1 Cognitive Factors:

  • Cognitive factors include ease of use, clarity, and visual hierarchy, which concern apparent usability, uncluttered presentation, and the ordering of interface elements.The text links minimalist design and whitespace to reducing cognitive load.Source: §3.2.1 · p. 4

3.2.2 Perceptual Factors: •

  • Perceptual factors include memorability, trust, and intuitiveness, concerning recall, confidence elicited by the interface, and whether its function is readily understood.The paper notes that headlines, text, and how well an interface is put together contribute to trust.Source: §3.2.2 · p. 4

3.2.3 Emotional Factors: •

  • Emotional factors include aesthetic pleasure, interest, and comfort, concerning visual attractiveness, attention, and feeling at ease while using an interface.The paper associates aesthetic pleasure with choices such as color, font, content density, and overall structure.Source: §3.2.3 · p. 5

3.3 Participants and Procedure

  • Five hundred participants supplied 15,000 responses; each UI received 500 unique responses, and each evaluation was expected to take about 45 seconds to a minute.Participants rated nine factors and then provided a written rationale for their ratings.Source: §3.3 · p. 5

3.4 Result of the Human Data Collection

  • After excluding fast, incomplete, duplicate, and non-professionally made-screen responses, the dataset contained 9,296 human responses for comparison with MLLM ratings.Table 2 reports human means and standard deviations by factor and domain, alongside overall averages.Source: §3.4 · p. 5

4 EVALUATION OF MLLM JUDGES FOR UIS

The evaluation compares GPT-4o, Claude 3.5 Sonnet, and Llama-3.2-11B-Vision with crowdsourced human judgments on absolute UI scores and pairwise preferences. Models align reasonably on some factors, but their performance varies across factors and tasks.

4.1 Task 1: Absolute Score Prediction

  • Absolute-score analyses find models overestimate Ease of Use and underestimate Aesthetic Pleasure, Interesting, and Comfort, yet predict several other factors within 0.50 Likert points.The paper names Clarity, Visual Hierarchy, Memorable, Trustworthy, and Intuitive among those factors.Source: §4.1 · p. 5

4.1.1 Individual Scores.

  • Across all factors and UIs, model means are within 0.35 of the human mean of 6.10; all models are within 0.25 on Trustworthy, while all underestimate Interesting.For Trustworthy, the human score is 6.34; for Interesting, it is 5.82.Source: §4.1.1 · p. 6, Table 2 · p. 7

4.1.3 Ranking Prediction.

  • Interesting has the highest MSE and MAE but the highest correlations for all models, while Comfort has the lowest errors but near-zero correlations; errors and ranking capture different behavior.The paper reports that ranking can reflect relative preferences even when absolute human scores are not reliably predicted.Source: §4.1.3 · p. 7, Table 3 · p. 8

4.2 Task 2: Pairwise Preference Prediction

  • Task 2 directly tests human preference through pairwise UI comparisons, complementing Task 1's prediction of absolute scores.A judge agrees with human preference when it selects the same UI as the one with the higher average human score for that factor.Source: §4.2 · p. 7, §4.2.1 · p. 7

4.1.2 Score Prediction.

  • Across factors, exact accuracy averages 35%-38%, while ±1-accuracy is 72%-77%; Claude is highest on both, and Interesting errors are highest while Comfort's are lowest.The ±1-accuracy is relatively stable across factors, unlike exact accuracy.Source: §4.1.2 · p. 6, Table 3 · p. 8

4.2.1 Preference Prediction.

  • GPT-4o and Claude reach around 75% and 78% on Interesting, but pairwise performance varies by factor; Llama is close to random overall at 51.79%.Overall accuracy for GPT-4o and Claude is close to 59%.Source: §4.2.1 · pp. 7–8, Table 4 · p. 8

4.2.2 Hardness of Preference Prediction.

  • With small human score differences, all judges are only slightly above 50%; at larger differences GPT-4o and Claude rise near 90% and 93%, while Llama reaches about 60%.The absolute difference between average human scores serves as a proxy for how difficult each pair is to judge.Source: Figure 3 · p. 9, §4.2.2 · p. 9

5 DISCUSSION

MLLMs approximate some human UI judgments but struggle with exact and fine-grained evaluations, supporting a supplemental role rather than replacing human research. The discussion also identifies opportunities to improve and broaden the benchmark and models.

5.1 MLLMs as Approximators

  • ±1 accuracy is reasonably high across all three models, but small-gap pairwise judgments approach random and exact accuracy is 48% for Claude and 40% for GPT.MAE ranged from 0.53 to 0.64, and MSE ranged from 0.44 to 0.66.Source: §5.1 · p. 9

5.2 MLLMs’ Role in the Design Process

  • MLLMs can provide early design direction when teams lack resources to test every option, but their imprecision makes them unsuitable for end-stage validation or high-stakes research.The models are described as moderately accurate for designs within ±1 Likert point and for pairwise comparisons with large human-score differences.Source: §5.2 · p. 9

5.3 Benchmark for Fine-Tuned Models

  • Future work could use the benchmark to fine-tune models or prompts for stronger sensitivity to UI factors such as ease of use and clarity.The authors suggest factor-specific data with reinforcement learning from human feedback or chain-of-thought prompting.Source: §5.3 · p. 9

5.4 Ethical Considerations

  • The authors caution against replacing human testing and present supplemental MLLM evaluation as a possible learning aid for junior designers with limited research access.They describe low-cost, rapid evaluations as a way to fill gaps where user research is unavailable and identify trends and larger flaws earlier.Source: §5.4 · p. 10

5.5 Future Work & Limitations

  • The study’s evidence is limited to static interfaces that were primarily professionally made under Western design standards, while future work could broaden device, model, prompt, and design-version coverage.The authors propose studying tablets, TVs, consoles, additional models and prompting strategies, and small changes to the same design.Source: §5.5 · p. 10

6 CONCLUSION

The study compares human and MLLM evaluations of the same interfaces using absolute Likert scoring and pairwise judgments. It finds moderate performance, especially for overall scores and comparisons with larger differences, and recommends supplemental use in early, low-stakes research.

6 CONCLUSION

  • The study compares human and MLLM evaluations of the same interfaces using 1–7 Likert scoring and pairwise evaluation, then compiles the results into a benchmark.The benchmark is intended to support future studies and fine-tuned models for UI judgment.Source: §6 · p. 10
  • MLLMs perform moderately on overall screen scores and pairwise comparisons, but struggle with specific criteria and perform better when compared screens differ more.The conclusion describes the pairwise results as reasonable and notes the stronger performance on comparisons with a higher degree of difference.Source: §6 · p. 10
  • The authors recommend MLLMs for supplementing early, low-stakes user research rather than replacing it.They present the benchmark as groundwork for further study of MLLMs as UI judges.Source: §6 · p. 10

A OPPORTUNITIES (EXTENDED)

The section outlines extensions to the benchmark and reports supplementary evaluation results. Proposed directions include fine-tuning, broader data and tasks, more models and prompts, and comparisons across design iterations.

A OPPORTUNITIES (EXTENDED)

  • The benchmark could support fine-tuning, particularly through reinforcement learning from human feedback, to better supplement user research.The passage also suggests using the benchmark alone or with similar datasets.Source: A OPPORTUNITIES (EXTENDED) · p. 12
  • Figure 4 calculates agreement after taking the higher of the two Task 1 absolute values for each evaluation set, then comparing human and MLLM results.Source: Figure 4 · p. 12
  • Expanding screen diversity beyond mobile and desktop first UIs, and adding tasks such as usability studies or surveys, could make the benchmark more expansive.The passage lists tablets, televisions, and entertainment consoles as additional interface settings.Source: A OPPORTUNITIES (EXTENDED) · p. 12
  • Testing additional models and prompting strategies, including chain-of-thought and few-shot methods, could reveal further outcomes and trends.The passage names Gemini, Qwen, and GPT-4o mini as possible additional models.Source: A OPPORTUNITIES (EXTENDED) · p. 12
  • Mean absolute error was below 1 for every model except on “interesting” UI questions.The passage says models can score within one point of human scores on Likert-scale UI questions.Source: Figure 5 · p. 13
  • Future studies could compare earlier UI iterations by changing a single color, text element, or layout and measuring judgment shifts.This study compared different UI designs with one another.Source: A OPPORTUNITIES (EXTENDED) · pp. 12–13

B ADDITIONAL RESULTS & PROMPTS

Additional figures visualize Task 1 results, while Figures 7 and 8 show the instructions used for Likert-scale evaluation and pairwise UI comparison.

B ADDITIONAL RESULTS & PROMPTS

Loading 2510.08783v1…