PaperlayerView exact revision on arXiv

Source-linked AI summary

ScreenAI: A Vision-Language Model for UI and Infographics Understanding

Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, Abhanshu Sharma

arXiv:2402.04615v3cs.CVcs.AI

TL;DR

Infographics and UIs pose a unique challenge to building a unified model that can understand, reason, and interact on pictorial pixels. ScreenAI combines PaLI with Pix2struct’s flexible patching and uses UI representations and LLMs to generate training data; its training mixture transfers positively across screen, infographic, and document tasks, and the model achieves state-of-the-art results on a number of public benchmarks.

  • Problem

    Infographics and UIs present a unique challenge to building a unified model that can understand, reason, and interact on pictorial pixels.

    Source: §1 · p. 1

  • Method

    ScreenAI combines PaLI with Pix2struct’s flexible patching and uses textual UI representations with LLMs to generate training data at scale.

    Source: §1 · p. 1

  • Results

    Training on a unified mixture yields positive transfer across screen-related, infographic, and document-related tasks, and ScreenAI achieves state-of-the-art results on a number of public benchmarks.

    Source: §6 · pp. 8–9

  • Takeaways & Limitations

    The released datasets are intended to enable more comprehensive benchmarking of models on screen-related tasks.

    Source: §6 · p. 9

    On some tasks, further research is needed to bridge the gap with models like GPT-4 and Gemini, which are orders of magnitude larger.

    Source: §6 · p. 9

Abstract

from arXiv · show

Screen user interfaces (UIs) and infographics, sharing similar visual language and design principles, play important roles in human communication and human-machine interaction. We introduce ScreenAI, a vision-language model that specializes in UI and infographics understanding. Our model improves upon the PaLI architecture with the flexible patching strategy of pix2struct and is trained on a unique mixture of datasets. At the heart of this mixture is a novel screen annotation task in which the model has to identify the type and location of UI elements. We use these text annotations to describe screens to Large Language Models and automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. We run ablation studies to demonstrate the impact of these design choices. At only 5B parameters, ScreenAI achieves new state-of-the-artresults on UI- and infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and InfographicVQA) compared to models of similar size. Finally, we release three new datasets: one focused on the screen annotation task and two others focused on question answering.

1 Introduction

ScreenAI is a vision-language model for understanding UIs and infographics, combining a flexible vision-language architecture with pixel-derived screen representations and generated training data. The paper reports strong performance across infographic QA benchmarks and introduces datasets for screen annotation and question answering.

1 Introduction

  • ScreenAI targets infographic QA and UI element annotation, summarization, navigation, and QA, combining PaLI [Chen et al., 2023b] with Pix2struct’s [Lee et al., 2023] flexible patching.It recasts vision tasks as text-and-image-to-text problems.Source: §1 · p. 1
  • The model uses a textual representation of UIs for pretraining, inferred from screen or image pixels and applicable beyond web pages and mobile apps to documents and infographics.The paper contrasts this broader scope with prior self-supervised work focused on one domain, including pix2struct [Lee et al., 2023] and HTLM.Source: §1 · p. 1, §1.1 · p. 2
  • The authors use the UI representation and LLMs to generate training data at scale and define pretraining and fine-tuning mixtures spanning UI and infographic understanding tasks.Source: §1 · p. 1
  • The paper releases Screen Annotation, ScreenQA Short, and Complex ScreenQA to support use of its textual representation and broader benchmarking of screen-based question answering.Source: §1 · p. 1
  • At 4.6 billion parameters, ScreenAI achieves state-of-the-art on three public infographics QA benchmarks, surpassing models 10× or larger, and best-in-class or close-to-best performance on other tasks.The paper also reports that performance improves as model size increases.Source: §1 · pp. 1–2

1.1 Related Work

  • The authors report superior downstream performance and hypothesize that joint pretraining on screen, document, and infographic data enables positive transfer across domains.They suggest that abundant data in these domains could enable further gains.Source: §1.1 · p. 2

2 Methodology

ScreenAI adapts a PaLI-style vision-language architecture for varied screen formats and trains models in three sizes through pretraining and task-specific fine-tuning.

2.1 Architecture

  • ScreenAI builds on PaLI’s image-and-text-to-text encoder-decoder and uses Pix2Struct-style arbitrary-grid patching to handle varied screen shapes and aspect ratios without padding or stretching.The model encodes image and text inputs together, then uses an autoregressive decoder to generate text.Source: §2.1 · pp. 2–3

2.2 Model Configurations

  • Models contain 670M, 2B, or 5B parameters; smaller models start from unimodal vision and language checkpoints, while 5B starts from PaLI-3 multimodal pretraining.The sequence budgets are 2024, 2916, and 3364 embeddings respectively; square-image maxima are 720 × 720, 756×756, and 812×812.Source: §2.2 · p. 3

2.3 Stages of Training

  • Pretraining uses large self-supervision- and model-generated datasets with both encoders trainable initially, then freezes ViT; fine-tuning mixes mostly human-labeled tasks, with QA shared then task-specific and other tasks trained individually.Training the vision encoder supports adaptation to the new patching strategy and UI-related images.Source: §2.3 · p. 3

3 Automatic Data Generation

ScreenAI’s pretraining data is generated automatically to scale beyond manual annotation while preserving diversity and complexity. The process turns annotated screenshots into structured screen schemas, which also enable LLM-generated tasks.

3 Automatic Data Generation

  • Automated generation uses specialized smaller models to build a large, diverse pretraining dataset efficiently, avoiding impractical manual annotation.The approach is intended to support data diversity and complexity at scale.Source: §3 · p. 3

3.1 Screen Annotation

  • A DETR-based annotator detects UI elements and predicts their bounding boxes, extending beyond candidate-box classification approaches [Carion et al., 2020] [Li et al., 2022a].Screens crawled from desktop, mobile, and tablet applications and web pages are annotated with UI labels and spatial relationships [Raffel et al., 2020]. Icon classification, image captioning, and OCR add icon names, visual descriptions, and on-screen text.Source: §3.1 · pp. 3–4
  • The resulting screen schema combines UI names, OCR text, descriptions, and normalized bounding boxes, and serves both as a pretraining target and an LLM interface.Parentheses encode a basic hierarchy among elements; providing the schema to LLMs enables more intricate, contextually nuanced task generation.Source: §3.1 · p. 4

3.2 LLMs to Generate Additional Tasks

  • PaLM 2-S generates synthetic QA data in two stages: first the screen schema is produced, then an LLM uses it to generate tasks [Anil et al., 2023b].Prompt engineering is iterative, and human validation checks a subset of generated responses against a predetermined quality threshold.Source: §3.2 · p. 4

4 Data Mixtures

The section separates large-scale pretraining with self-supervised or model-generated labels from human-labeled fine-tuning, then describes the task mixtures and evaluation benchmarks used for ScreenAI. It introduces three new screen-understanding benchmarks alongside established screen, infographic, and document tasks.

4 Data Mixtures

  • Fine-tuning uses labels provided or verified by human raters, whereas pretraining uses self-supervised or model-generated labels and typically larger datasets over more training steps.These are the two primary distinctions between the task sets.Source: §4 · p. 5

4.1 Pretraining Mixture

  • Pretraining combines screen annotation, screen question-answering, navigation, and summarization to cover UI and infographic understanding.Screen annotation includes UI-element detection and identification, OCR, image captioning, and masked text that encourages inference from context and layout.Source: §4.1 · p. 5
  • Screen QA targets identified gaps in arithmetic, counting, and complex infographic understanding, while navigation predicts target-element boxes and summarization captions screen content.For navigation, bounding-box coordinates are bucketized between 0 and 999; summarization produces one or two sentences.Source: §4.1 · p. 5
  • Pretraining tasks span mobile and desktop formats with varied aspect ratios, alongside language, web, image, OCR, and chart-to-table data sources.Dataset weights are proportional to dataset size, subject to a maximum weight per task.Source: §4.1 · p. 5

4.2 Fine-Tuning Tasks and Benchmarks

  • Fine-tuning benchmarks cover screen, infographic, and document understanding, with RefExp and MoTIF recast as object detection tasks.For these tasks, accuracy is reported at IoU=0.13 using one predicted box without candidate boxes; MoTIF Table 4 reports the app-unseen test split.Source: §4.2 · pp. 5–6, Table 3 · p. 7
  • The released Screen Annotation benchmark contains 4.2K human-annotated Rico screenshots, while ScreenQA Short uses alternative short answers and maximum F1 across candidates.Screen Annotation evaluates bounding boxes and UI classes with object-detection metrics, including F1, precision, and recall at IoU=0.1.Source: §4.2 · p. 6
  • Complex ScreenQA complements ScreenQA Short with harder counting, arithmetic, comparison, and non-answerable questions across screens with varied aspect ratios.Source: §4.2 · p. 6
  • Multipage DocVQA training pairs answer-bearing pages with positive examples and subsampled no-answer negatives; ChartQA uses synthetic examples without rationales or loss changes.The paper reports similar ChartQA performance with or without OCR.Source: §4.2 · pp. 6–7

5 Experiments and Results

ScreenAI reports strong benchmark performance on screen and infographic tasks, examines the effects of OCR input and model size, and uses ablations to assess its patching strategy and LLM-generated pre-training data.

5.1 Experiments Setup

  • Fine-tuning freezes the ViT encoder and updates only the language model, uses batch size 512 and input length 128, and generally converges within 30k steps.Output length varies by task, OCR input increases input length, and experiments use the 5B model unless otherwise specified.Source: §5.1 · p. 7

5.2 Results

  • ScreenAI reports new SoTA results on MoTIF, MPDocVQA, and WebSRC; best-in-class results on ChartQA, DocVQA, and InfoVQA; and same or competitive results on three other benchmarks.The comparison also includes Screen Annotations, Referring Expressions, ScreenQA Short, and Complex ScreenQA.Source: §5.2 · p. 7, Table 4 · p. 8
  • OCR input boosts QA performance by up to 4.5% on Complex ScreenQA, MPDocVQA, and InfoVQA, but increases training time and requires OCR at inference.OCR also requires a longer input sequence.Source: §5.2 · p. 8
  • Performance improves with model size from 670M to 5B and has not saturated; gains from 2B to 5B are larger for InfoVQA, ChartQA, and Complex ScreenQA.The larger gains are observed on tasks requiring more complex visual-text and arithmetic reasoning.Source: §5.2 · p. 8

5.3 Ablation Studies

  • For landscape images, pix2struct patching significantly outperforms fixed-grid patching; for portrait images, fixed-grid patching is only marginally better, supporting pix2struct for varied aspect ratios.The ablation compares 670M models fine-tuned across the Table 3 tasks, using an aggregate score based on geometric means.Source: §5.3 · p. 8
  • Adding LLM-generated data to the pre-training mixture improves the aggregate score by 4.6 percentage points.The comparison uses 670M models, with and without LLM-generated pre-training data, fine-tuned on all Table 3 tasks.Source: §5.3 · p. 8

6 Conclusions · Appendix

ScreenAI combines a unified representation for infographics, document images, and UIs with self-supervised training across these domains, achieving competitive results and state of the art on some public benchmarks. The work releases the representation dataset and two screen-task benchmarking datasets, while noting gaps with much larger models on some tasks.

6 Conclusions

  • The unified schema represents complex data and visual information across infographics, document images, and varied UIs, enabling self-supervised tasks that draw on all three domains.The authors report positive transfer from this training mixture to screen-, infographic-, and document-related tasks.Source: §6 · p. 8
  • Fine-tuning metrics improve consistently as model size increases.Source: Figure 8 · p. 9
  • For aspect ratios above 1.0, Pix2Struct patching significantly outperforms fixed-grid patching; below 1.0, fixed-grid patching leads by a smaller margin.The figure reports aggregated scores across all fine-tuned tasks.Source: Figure 9 · p. 9
  • ScreenAI performs competitively and achieves state of the art on a number of public benchmarks, though some tasks still lag models such as GPT-4 and Gemini.The authors note those models are orders of magnitude larger.Source: §6 · p. 9
  • The authors use LLM-based data generation and ablation studies to illustrate the impact of generated data and justify ScreenAI’s design choices.The supplied conclusion does not specify the size or direction of each impact.Source: §6 · p. 9
  • The authors release a dataset using the unified representation and two additional datasets to support more comprehensive benchmarking of screen-related tasks.Source: §6 · p. 9

A Definitions of Metrics … D Screen Navigation Generated Examples

The appendices define evaluation metrics, specify the screen schema, and show LLM prompts and generated examples for screen QA, navigation, and summarization. The schema is supplied to LLMs alongside task-specific instructions for generating content from screens.

A Definitions of Metrics

  • Bounding-box tasks match predictions to ground truth per class at IoU 0.1, then report F1@IoU=0.1 and Acc@IoU=0.1.F1 is the harmonic mean of precision and recall; Acc is top-1 accuracy.Source: A Definitions of Metrics · p. 12
  • Plain-text tasks use CIDEr, SQuAD F1, relaxed accuracy, and ANLS [Mathew et al., 2021].SQuAD F1 applies SQuAD text preprocessing; ANLS is Average Normalized Levenshtein Similarity.Source: A Definitions of Metrics · p. 12
  • Exact Match (EM) is also used for plain-text tasks.The paper directs readers to the Hugging Face datasets Exact Match metric definition.Source: A Definitions of Metrics · p. 12

B Screen Schema Examples

  • The screen schema records UI element names, applicable OCR text, descriptions, and bounding boxes quantized and normalized from 0 to 999.Parentheses group child elements under their parent; the figure overlays schema boxes on the screenshot.Source: B Screen Schema Examples · p. 12

C Prompts For LLM Generated Content

C.1 Screen Question Answering

C.2 Screen Navigation

  • The navigation prompt requests single-step instructions and answers in JSON, with each answer starting “click” and giving quantized element coordinates.It encourages varied, imperative questions that refer to UI elements indirectly.Source: §C.2 · p. 13

C.3 Screen Summarization

  • The summarization prompt requests a two-to-three-sentence, content-focused screen summary rather than a list of UI elements.The paper presents LLM-generated screen-navigation examples in Figure 11 and refers readers to Section 3 for generation details.Source: §C.3 · p. 13, D Screen Navigation Generated Examples · p. 13

E MoTIF Evaluation Results

This section reports ScreenAI metrics across the different splits of the MoTIF task automation dataset.

E MoTIF Evaluation Results

F ScreenQA Short Answers Generation

The section motivates replacing ScreenQA’s single short-answer ground truth with a list of valid answer variants, then describes how those lists were generated and checked.

F ScreenQA Short Answers Generation

  • Because information can have multiple representations, a list of representations for the same factual answer lets a model choose one without penalty; different date formats can denote the same date.Source: F ScreenQA Short Answers Generation · p. 13
  • A few-shot variant of PaLM 2-S [Anil et al., 2023b] generated short-answer lists from each ScreenQA question, UI-element descriptions, and full-sentence answer.The prompts are described in Appendices F.1 and F.2.Source: F ScreenQA Short Answers Generation · p. 13
  • The generated answer lists were checked with simple heuristics and by eyeballing random samples.Examples of ScreenQA questions, answers, and generated short answers appear in Figure 12.Source: F ScreenQA Short Answers Generation · p. 13

F.1 For answers contained in a single UI element · F.2 For answers contained in multiple UI elements

For ScreenQA answers grounded in either one or multiple UI elements, the appendix uses PaLM 2-S prompts to generate concise answer rephrasings from the question, element descriptions, and full-sentence answer [Anil et al., 2023b].

F.1 For answers contained in a single UI element

  • For single-element ScreenQA entries, PaLM 2-S generates a list of short-answer rephrasings using the question, UI element, and full-sentence answer [Anil et al., 2023b].The prompt is applied when the ground truth contains only one UI element.Source: §F.1 · p. 13
  • Single-element rephrasings should be as short as possible, omit extra question wording, use the provided element, and appear in square brackets.Source: §F.1 · p. 13
  • The single-element prompt leaves slots for the question, UI element description, and full-sentence answer before requesting rephrasings.Source: §F.1 · p. 15

F.2 For answers contained in multiple UI elements

  • For ScreenQA entries with multiple ground-truth UI elements, PaLM 2-S generates short-answer rephrasings from the question, element descriptions, and full-sentence answer [Anil et al., 2023b].Source: §F.2 · p. 15
  • For multiple-element answers, rephrasings should be concise, exclude extra question wording, use all provided elements, and be returned in square brackets.Source: §F.2 · p. 15
  • The multiple-element prompt takes the question, descriptions of the first and remaining UI elements, and full-sentence answer as inputs for rephrasing.Source: §F.2 · p. 15

G Complex Question Answering Datasets

The section describes a large-scale, LLM-generated screen QA suite spanning multiple screen formats and question types, with human verification of evaluation data.

G Complex Question Answering Datasets

  • LLMs including PaLM 2-S [Anil et al., 2023b] generate questions targeted by prompt type; datasets cover desktop, mobile, varied mobile aspect ratios, and infographics for pre-training and evaluation, with evaluation data human-verified.The paper also shows examples of LLM-generated QA data verified by human raters.Source: G Complex Question Answering Datasets · p. 15
  • Desktop QA and Long Webpage QA cover desktop screens and long, viewport-height webpages with distinct image sizes and aspect ratios; Non Answerable QA tests whether questions can be answered from a screen.Source: G Complex Question Answering Datasets · p. 15
  • Complex QA targets counting, arithmetic, and comparisons requiring information from multiple screen regions, with mobile, desktop, and long-webpage variants.Source: G Complex Question Answering Datasets · p. 15

H New Benchmarks Repositories

The authors release three evaluation datasets for tasks described in Section 4.2: Screen Annotation, ScreenQA Short, and Complex ScreenQA.

H New Benchmarks Repositories

  • The release includes three evaluation datasets: Screen Annotation (SA), ScreenQA Short (SQA Short), and Complex ScreenQA (Cplx SQA).Each dataset has a corresponding repository.Source: H New Benchmarks Repositories · p. 15
Loading 2402.04615v3…