Source-linked AI summary
UniAR: A Unified model for predicting human Attention and Responses on visual content
Peizhao Li, Junfeng He, Gang Li, Rachit Bhargava, Shaolei Shen, Nachiappan Valliappan, Youwei Liang, Hongxiang Gu, Venky Ramachandran, Golnaz Farhadi, Yang Li, Kai J Kohlhoff, Vidhya Navalpakkam
TL;DR
Prior work often modeled human attention and subjective preferences separately, with unified prediction across visual content still missing. UniAR uses a multimodal transformer to predict attention, scanpaths, and preferences, and achieves comparable-or-better SOTA performance across diverse benchmarks and zero-shot combinations.
Problem
A unified approach to predicting implicit attention and explicit preferences across diverse visual content is missing.
Source: §1 · p. 2Method
UniAR is a multimodal transformer that conditions predictions of human behavior on images and text prompts describing the content domain, behavior task, and relevant scenario.
Source: §1 · p. 2Results
Across 11 benchmark datasets, UniAR outperforms or performs comparably to task- and dataset-specific SOTA models and generalizes to unseen input/output combinations in zero-shot settings.
Source: §1 · p. 2Takeaways & Limitations
UniAR could augment human decision-making and accelerate UI evaluation by providing preference predictions alongside insights from predicted attention behavior.
Source: §1 · p. 2UniAR cannot directly model the behavior of blind and low-vision users because its attention training data did not include people with visual impairments beyond corrective lenses.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.
1 Introduction
Prior work often treated attention and subjective preferences separately, leaving a unified model across behavior tasks and visual content missing. UniAR brings these predictions together and reports performance comparable to or better than specialized SOTA models across diverse benchmarks.
1 Introduction
- Prior research often studied attention separately from subjective preferences and used dedicated models for particular behavior tasks, content domains, or scenarios.The paper identifies a unified approach spanning implicit attention and explicit preferences as missing.Source: §1 · pp. 1–2
- UniAR is a multimodal transformer that predicts human behavior from attention to likes across diverse visual content.It conditions predictions on images and text prompts describing the content domain, behavior task, and, when relevant, task scenario.Source: §1 · p. 2
- Across 11 benchmark datasets, UniAR outperforms or performs comparably to task- and dataset-specific SOTA models and generalizes zero-shot to unseen input/output combinations.The datasets span natural images, webpages, and graphic designs, with attention and importance heatmaps, scanpaths, and aesthetics or quality scores.Source: §1 · p. 2
- Figure 2 presents UniAR predictions for saliency or importance heatmaps, scanpaths, and ratings alongside ground truth.Examples cover natural images, webpages, graphic designs, and mobile UI.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
Related work covers saliency, scanpath, and subjective-rating prediction, while noting that prior models often specialize in particular tasks and visual domains. UniAR draws on the generalization of unified language and vision models to address that fragmentation.
Saliency prediction
- Saliency prediction estimates which image areas are likely to attract attention; prior approaches use deep backbones and specialized modules, while saliency maps also support UI evaluation, design placement, and image or video compression.Reported approaches include CNNs, Transformers, read-out convolutions, Attentive ConvLSTM, and regression of Gaussian fixation distributions.Source: §2 · pp. 2–3
Scanpath prediction
- Scanpath prediction models the sequence of eye movements, with prior work addressing free viewing, object search, and visual question answering using methods including inverse reinforcement learning and Transformers.Examples include fixation-coordinate prediction conditioned on an object name and fixation distributions conditioned on a question.Source: §2 · pp. 3–4
Subjective rating prediction
- Subjective-rating models predict explicit feedback such as preference or likes, which can be continuous or discrete and reflect technical or aesthetic image quality.Prior methods include statistics-based approaches, CNNs, and Vision Transformers.Source: §2 · p. 4
Limitations of prior work
- Prior models often target a specific task and input domain, whereas unified language and vision models have shown generalization across tasks and domains.The paper draws on that generalization capacity in proposing one model for diverse human visual-behavior predictions.Source: §2 · p. 4
3 Unifying Human Attention and Responses
UniAR unifies attention and response prediction with a multimodal encoder-decoder transformer, using prompts to adapt predictions across visual domains and tasks.
3.1 Model Architecture
- A multimodal encoder-decoder combines image and text representations, then uses separate heatmap, scanpath, and rating predictors; pretraining spans natural images and web/mobile interfaces.Pretraining uses image and screen-region captioning, plus bounding-box prediction for relevant interface items.Source: §3.1 · p. 4
3.2 Heatmap Predictor
- Prompt-selected heatmap tasks share one head trained with pixel-wise ℓ2 loss; saliency aggregates eye fixations, while importance marks critical design elements.Source: §3.2 · p. 5
3.3 Scanpath (Sequence) Predictor
- The decoder autoregressively generates scanpath coordinates as text tokens, a format the authors say can extend to other behavior sequences with minor output-format changes.For N ≥30, generated strings may not match the target format; decoding extracts coordinate pairs and stops at a nonnumeric pair.Source: §3.3 · pp. 5–6
3.4 Rating Predictor
- The rating predictor processes image tokens through convolutional and connected layers and is trained with an ℓ2 loss on rating data.Source: §3.4 · p. 6
3.5 Text Prompt
- Task prompts specify input type, output type, and an optional query, such as an object to search, to adapt predictions to different datasets and scenarios.Source: §3.5 · p. 6
4 Experiment
The experiments evaluate UniAR across established benchmarks, using multiple metrics and training data sampled evenly across datasets. Results cover heatmap, scanpath, and rating prediction, plus transfer to an unseen task-domain combination.
4.1 Protocol
- The protocol reuses recent literature benchmarks and their evaluation metrics, resizing predicted heatmaps to each image's original resolution before evaluation.The benchmark set spans heatmaps, scanpaths, and ratings across datasets and visual domains.Source: §4.1 · pp. 6–7
4.2 Model Training
- UniAR is pretrained on UI-understanding and image-captioning tasks, then fine-tuned; training samples each dataset with equal probability regardless of dataset size.The model has 848 million parameters and was trained for 20k steps on 64 Google Cloud TPU v3 devices.Source: §4.2 · p. 7
4.3 Experiment Results
- 17 of 27 Table 3 metrics rank UniAR best, with 6 more second-best across seven heatmap benchmarks.UniAR beats prior SOTA on every metric for Mobile UI and Imp1k.Source: §4.3 · pp. 7–8, Table 3 · p. 8
- On two scanpath datasets, UniAR is comparable to baselines overall and beats them on every WS-Scanpath metric, ranking best in 4 of 5 reported metrics.The datasets cover object searching in natural images and free viewing on webpages.Source: §4.3 · p. 8, Table 4 · p. 9
- UniAR achieves the best PLCC on both rating datasets and the second-best SRCC on KonIQ-10k.A multi-scale MUSIQ version scores 0.916 versus UniAR's 0.905 on KonIQ-10k SRCC; it is excluded because UniAR uses no multi-scale inputs.Source: Table 2 · p. 7, §4.3 · p. 8
- With scanpath and saliency data from COCO-Freeview plus webpage saliency data, UniAR performs well against prior SOTA on WS-Scanpath without webpage scanpath training; more limited scenarios remain competitive.Each training scenario shares either the test task or image domain, but never both.Source: §4.3 · p. 9, Table 5 · p. 9
5 Limitations and Future Work
UniAR’s use for modeling preferences raises ethical concerns, including reward hacking and the risk of converging on uniform preferences without personalization. The authors also identify limits in dataset representativeness and direct modeling of users with visual impairments.
Ethics and AI principles
- Reward-model use may lead to reward hacking, making predictions less representative of genuine human preferences.The authors suggest considering mitigation techniques and keeping humans in the loop, with predictions serving as a reference rather than a replacement for evolving preferences.Source: §5 · p. 9
Representing diverse human preferences
- Without personalization, UniAR may converge toward a more uniform notion of subjective preference.The authors propose hybrid use to support human decisions or personalized models based on user attributes to promote visual diversity.Source: §5 · pp. 9–10
Adjusting to evolving human preferences
- Human preferences evolve over time, so maintaining accuracy requires adjustment, for example through fine-tuning on more recent data.Future work includes updating training data and exploring continual learning.Source: §5 · p. 10
Dataset limitations
- The work is a proof of concept based on existing publicly available datasets, whose representativeness matters for minimizing potential bias.The reported participant ages skew toward people in their 20s, although crowdsourced Salicon and KonIQ-10k are expected to cover a wider range of ages and other attributes.Source: §5 · p. 10
Improve accessibility
- UniAR does not directly model blind and low-vision users because its attention training data exclude visual impairments beyond corrected vision.The authors describe webpage highlighting for screen readers as a possible accessibility use and suggest incorporating interactions with assistive technologies in future work.Source: §5 · p. 10
6 Conclusion
UniAR predicts implicit and explicit human behaviors on visual content using image and text prompts. Across multiple benchmarks and tasks, it achieves state-of-the-art performance.
6 Conclusion
- UniAR achieved state-of-the-art performance across multiple benchmarks and tasks.The conclusion identifies attention heatmaps, scanpath sequences, and aesthetic or quality scores as the predicted behaviors.Source: §6 · p. 10
- UniAR uses image and text prompts to predict implicit and explicit behavior, from attention to subjective preferences and likes.The model was trained on diverse public datasets covering natural images, graphic designs, webpages, and UIs.Source: §6 · p. 10
- The authors plan to explore additional behavior tasks and domains in future work.Source: §6 · p. 10
A Model Details
UniAR uses a shared image-and-text modeling design with separate predictors for ratings, heatmaps, and scanpaths. Training combines the three predictors’ losses with empirically chosen weights.
A Model Details
- Three predictors produce ratings, attention heatmaps, and scanpaths from the model’s image and text representations.The score predictor outputs a scalar, the heatmap predictor outputs a heatmap, and the scanpath predictor uses a T5 base decoder.Source: A Model Details · p. 16
- The shared components are a ViT B16 image encoder and a T5 base encoder for mixing image and text tokens.The ViT uses 16 × 16 patches; the T5 base encoder has 12 layers, 12 heads, and hidden dimension 768.Source: A Model Details · p. 16
- Training combines sequence cross-entropy, heatmap L2, and score L2 losses using weights [1, 500, 50].The weights were chosen empirically to bring the losses to a similar scale.Source: A Model Details · p. 16
B Dataset Processing
The dataset-processing section reports a resolution adjustment for Imp1k and dataset-specific evaluation or partitioning procedures. Table 6 presents heatmap results across public datasets and metrics.
B Dataset Processing
- The section describes key dataset-processing details.Source: B Dataset Processing · p. 16
- Table 6 reports heatmap prediction results on seven public datasets using eight metrics, spanning digital design and natural images.For Imp1k it covers importance heatmaps; for the other four datasets described in the caption, it covers attention or saliency heatmaps.Source: Table 6 · p. 17
- For Imp1k, the larger of an image and its ground-truth importance map is downsampled to match the lower-resolution item.Source: B Dataset Processing · p. 17
- For Mobile UI, the authors randomly created train and test partitions because the original dataset partition was missing.The number of instances follows Leiva et al. [47].Source: B Dataset Processing · p. 17
C Full Results and More Visualizations
The section reports full heatmap and scanpath performance tables and adds visualizations of UniAR’s heatmap, scanpath, and rating predictions.
C Full Results and More Visualizations
- Additional visualizations show UniAR’s heatmap, scanpath, and rating predictions.Source: C Full Results and More Visualizations · p. 17
- Table 7 covers scanpath prediction on natural-image and digital-design datasets, including object-searching and free-viewing tasks.Source: Table 7 · p. 18
- UniAR provides consistently good predictions across three tasks and multiple datasets, compared with ground truth.The full heatmap and scanpath tables include additional baselines and evaluation metrics.Source: C Full Results and More Visualizations · p. 18