PaperlayerView exact revision on arXiv

Source-linked AI summary

Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus

Gang Li, Yang Li

arXiv:2209.14927v4cs.CVcs.HCcs.LG

TL;DR

Mobile UI understanding matters for automation and accessibility, but view hierarchies may be inaccurate or unavailable. Spotlight uses screenshots and a focus region in a vision-language model, and reports state-of-the-art results on several UI tasks, outperforming methods that also use hierarchies.

  • Problem

    Mobile UI models often rely on view hierarchies, which can contain missing or inaccurate information and are not always available.

    Source: §1 · pp. 1–2

  • Method

    Spotlight is a vision-only vision-language model that takes a UI screenshot and a focus region as input to generate text for UI tasks.

    Source: §4 · p. 6

  • Results

    Spotlight obtains state-of-the-art results on several representative UI tasks and outperforms prior methods using screenshots and view hierarchies.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Its general architecture supports a range of UI tasks without requiring view hierarchies and is easy to scale.

    Source: §7 · p. 9

    An exploratory attempt to decode serialized view hierarchies during pretraining performed poorly, likely because their structures contain substantial noise.

    Source: §6 · p. 9

Abstract

from arXiv · show

Mobile UI understanding is important for enabling various interaction tasks such as UI automation and accessibility. Previous mobile UI modeling often depends on the view hierarchy information of a screen, which directly provides the structural data of the UI, with the hope to bypass challenging tasks of visual modeling from screen pixels. However, view hierarchies are not always available, and are often corrupted with missing object descriptions or misaligned structure information. As a result, despite the use of view hierarchies could offer short-term gains, it may ultimately hinder the applicability and performance of the model. In this paper, we propose Spotlight, a vision-only approach for mobile UI understanding. Specifically, we enhance a vision-language model that only takes the screenshot of the UI and a region of interest on the screen -- the focus -- as the input. This general architecture of Spotlight is easily scalable and capable of performing a range of UI modeling tasks. Our experiments show that our model establishes SoTA results on several representative UI tasks and outperforms previous methods that use both screenshots and view hierarchies as inputs. Furthermore, we explore multi-task learning and few-shot prompting capacities of the proposed models, demonstrating promising results in the multi-task learning direction.

1 INTRODUCTION

Mobile UI understanding supports interaction tasks, including automation and accessibility, but view hierarchies can be inaccurate or unavailable. Spotlight instead uses screenshots and a focus region, with a vision-language approach reported to perform strongly across UI tasks.

1 INTRODUCTION

  • Although hierarchies provide UI-object types, text, and positions, they can let models bypass visual inference from screenshots.Prior work found hierarchy-using models outperformed vision-only counterparts in UI captioning.Source: §1 · p. 1
  • View hierarchies can contain invalid bounding boxes and missing object labels, and removing text features reduced widget-captioning performance by 17 CiDER points.Reported rates of invalid boxes and missing labels illustrate this data-quality problem.Source: §1 · p. 1
  • View hierarchies are not always available, and retrieving them at runtime imposes additional system constraints on models that depend on them.The paper notes that some mobile UI images crawled from the web lack hierarchies.Source: §1 · p. 2
  • Spotlight uses screenshots and a focus object or region to generate text, with single-task finetuning, multi-task finetuning, and few-shot learning as modeling strategies.The approach is motivated by UI tasks that map between UI objects and text and can use a shared vision-language architecture.Source: §1 · p. 2
  • Spotlight achieves state-of-the-art results on four UI tasks with vision-only inputs, outperforming previous methods that use screenshots and view hierarchies.The tasks are widget captioning, screen summarization, command grounding, and tappability prediction.Source: §1 · p. 2

2 RELATED WORK

Prior mobile UI research established tasks spanning object descriptions, whole-screen summaries, command grounding, and tappability. Related approaches use view hierarchies, region-focused representations, and pretrained vision-language models, each addressing different modeling needs.

2 RELATED WORK

  • Prior benchmarks span object captions, screen summaries, command grounding (Li et al., 2020a; Burns et al., 2022; Bai et al., 2021), and tappability (Schoop et al., 2022).The tasks support accessibility, screen-level understanding, command interaction, and UI design validation, respectively.Source: §2 · p. 2
  • View hierarchies provide object attributes, but prior studies report that multimodal models outperform vision-only variants, while other work repairs noisy hierarchies or detects objects from screenshots.The described approaches include language-model representation learning and multi-task UI modeling.Source: §2 · p. 3
  • Earlier focus methods crop object pixels, bias image features with a focus map, or add a binary-mask channel; cropping can lose context and the extra channel limits use of pretrained RGB models.The paper investigates alternative region representations.Source: §2 · p. 4
  • Pretrained vision-language models encode images and decode text; Spotlight adapts this architecture to explore UI finetuning, multi-task learning, and few-shot capability.Prior examples include Flamingo (Alayrac et al., 2022) and PaLI.Source: §2 · p. 4

3 DATA

Spotlight pretraining combines web-page screenshots and mobile UI data, using object text attributes as targets. Its downstream benchmarks cover four tasks that map between UI objects and language and require both region-level and screen-level attention.

3.1 PRETRAINING DATASETS

  • Pretraining uses 80 million C4 web-page screenshots and 2.69 million mobile screenshots, with the mobile corpus providing in-domain unlabeled data.The mobile screens were collected by crawling apps after random clicks in an Android emulator.Source: §3.1 · p. 4
  • Web-page targets include rendered text and functional descriptions, while mobile targets use leaf-node text, content descriptions, resource IDs, and qualifying OCR text.OCR text must meet an 80% confidence threshold; preprocessing removes URLs, rare strings, and generic phrases.Source: §3.1 · pp. 4–5

3.2 UI TASK DATASETS

  • The four downstream tasks are widget captioning, screen summarization, command grounding, and tappability prediction.They generate descriptions or summaries, find command-matching objects, or predict whether an object is tappable.Source: §3.2 · p. 5
  • The tasks map between UI objects and natural-language descriptions in either direction, requiring focus on a specific region or the full screen.The experiments use the same dataset splits as the benchmarks for comparison.Source: §3.2 · p. 5

4 MODELS

Spotlight combines image encoding and text generation with a bounding-box-conditioned focus-region extractor. Its region representations aim to localize an object while retaining screen context.

4 MODELS

  • ROI Align pools feature-map values from the target region into a fixed-length representation, preserving screen context through ViT self-attention.Unlike hard cropping of raw pixels, the encoded representation can retain context beyond the target area.Source: §4 · pp. 5–6
  • The Region Summarizer projects each bounding-box coordinate into n vectors, adds coordinate-type embeddings, and uses n to control representation capacity.The learned coordinate projections and embeddings determine the number of box vectors and region summaries.Source: §4 · p. 6
  • A Transformer extracts region representations from ViT encodings by cross-attending from box descriptors to the image features and descriptor vectors.The described layer updates the region queries using cross-attention, then a dense layer.Source: §4 · p. 6
  • A Region Summarizer uses bounding-box descriptor vectors to extract region representations from ViT encodings, and it outperforms ROI Align in the reported experiments.Its attention can focus on the boxed target while taking screen context into account.Source: §4 · p. 6
  • All tasks use autoregressive text decoding from the target-region representation, including Yes/No predictions for grounding and tappability.For command grounding, the object with the highest probability of the Yes token is selected.Source: §4 · p. 6

5 EXPERIMENTS

The experiments assess Spotlight through task-specific finetuning, multi-task learning, and few-shot prompting. Task-specific vision-only models outperform prior screenshot-and-view-hierarchy systems across four downstream tasks.

5.1 PRETRAINING

  • Pretraining combines C4 web pages and mobile screens by predicting object text, sampling 9:1 from the larger C4 dataset and mobile data.Examples contain one to five screen-object-text tuples; webpage titles receive a small sampling weight for screenshot summarization.Source: §5.1 · p. 7

5.2.1 FINETUNING

  • Spotlight’s vision-only models achieve state-of-the-art results on all four downstream tasks, significantly outperforming prior models that use screenshots and view hierarchies.Table 3 reports CIDEr for widget captioning and screen summarization, accuracy for command grounding, and F1 for tappability.Source: §5.2.1 · p. 7, Table 3 · p. 8

5.2.2 MULTI-TASK LEARNING

  • Multi-task Spotlight outperforms prior models on all four tasks, matching task-specific models on widget captioning and tappability but trailing them on summarization and grounding.Training samples task batches with weights [3, 2, 15, 1] and uses a prompt tuple alongside a prediction tuple.Source: §5.2.2 · pp. 7–8, Table 4 · p. 8

5.2.3 FEW-SHOT LEARNING

  • Few-shot prompting brings marginal widget-captioning gains over zero-shot for L/16, but not B/16; the models show no meaningful few-shot performance on other tasks.The authors conjecture that the other tasks are farther from the pretraining distribution and suggest larger, more diverse pretraining data and a larger text decoder as possible improvements.Source: §5.2.3 · p. 8, Table 5 · p. 8

6 DISCUSSIONS

The discussion examines how Spotlight attends to target regions and relevant screen context, and identifies noisy view-hierarchy pretraining and relatively small model size as considerations.

6 DISCUSSIONS

  • For widget captioning, attention includes both the target checkbox and distant “Chelsea” text; for screen summarization, it attends to important screen areas.Figure 2 visualizes the last cross-attention layer of the Region Summarizer for L/16 models.Source: §6 · pp. 8–9, Figure 2 · p. 9
  • Directly decoding serialized view hierarchies performed poorly, likely because their structures are noisy; using their structural information for better pretraining remains open.The authors report that pretrained models significantly outperformed models trained from scratch.Source: §6 · p. 9
  • The investigated models are relatively small compared with vision-language efforts using O(10B) parameters, and the authors report a consistent trend of better performance with larger models.Spotlight can adopt larger ViT and T5 checkpoints and is presented as applicable to additional sequence-decoding UI tasks.Source: §6 · p. 9

7 CONCLUSIONS

Spotlight uses a scalable vision-language architecture to understand mobile UIs from screenshots and a focus region, without requiring view hierarchies. It achieves state-of-the-art results on several representative tasks and outperforms prior methods using screenshots and view hierarchies.

7 CONCLUSIONS

  • Spotlight is a vision-only approach that focuses on a screen region and supports a range of mobile UI modeling tasks.Its general vision-language architecture is easy to scale.Source: §7 · p. 9
  • Spotlight achieves state-of-the-art results on several representative UI tasks and outperforms prior methods using screenshots and view hierarchies.Source: §7 · p. 9
  • The paper examines model variants, modeling strategies, finetuning, multitask learning, and few-shot prompting.Source: §7 · p. 9

A HYPERPARAMETERS & MODEL SIZES

The experiments use T5 and ViT models with B/16 and L/16 configurations. The section specifies image processing, training, and inference settings, alongside pretraining, finetuning, and multitask schedules.

A HYPERPARAMETERS & MODEL SIZES

  • The experimental Spotlight models use T5 and ViT, with ViT B/16 and L/16 configurations.The model variants are cataloged in Table 6.Source: A HYPERPARAMETERS & MODEL SIZES · p. 12, Table 6 · p. 13
  • Screens are resized with preserved aspect ratio and padded to 740 × 740 when needed; padding pixels are masked, and bounding boxes are normalized to [0, 1].Training uses batch size 128; inference uses beam size 5 and a maximum decoding length of 64.Source: A HYPERPARAMETERS & MODEL SIZES · p. 12
  • Pretraining runs for 164K steps with ViT B/16 and 156K with L/16, using an initial 10K-step linear warmup.The models are trained on 128 and 256 TPU v3 cores for 54 and 86 hours, respectively.Source: A HYPERPARAMETERS & MODEL SIZES · p. 13
  • Finetuning uses 20K steps for Command Grounding and 10K steps for each of the other three tasks.Learning rates are 1e-3 for Command Grounding and 1e-4 for the other tasks.Source: A HYPERPARAMETERS & MODEL SIZES · p. 13
  • Multitask training runs for 30K steps with task sampling weights [3, 2, 15, 1] for Widget Captioning, Screen Summarization, Command Grounding, and Tappability.Evaluation prompts the model with a screen-object-text tuple from the target task's training set.Source: A HYPERPARAMETERS & MODEL SIZES · p. 13

B ABLATION STUDY

The ablations highlight the importance of pretraining on both mobile and web data and examine alternatives for extracting region information. The accompanying pseudocode maps bounding-box coordinates to queries that cross-attend to ViT features.

B ABLATION STUDY

  • The ablation variants change one option from the B/16 full model and use 100K pretraining steps before downstream finetuning.The Table 3 full model has better results with 164K pretraining steps.Source: B ABLATION STUDY · p. 13
  • Frozen-ViT and from-scratch models perform poorly across all four tasks, while using only one pretraining dataset degrades performance.The authors interpret these results as evidence for mobile-and-web pretraining and combining heterogeneous datasets; Table 7 presents the ablation results.Source: B ABLATION STUDY · p. 13, Table 7 · p. 14
  • Region Summarizer alternatives generally perform reasonably, except on Widget Captioning, where flexible, per-coordinate queries updated through cross-attention help.The bounding-box queries also attend to relevant screen context for caption decoding.Source: B ABLATION STUDY · p. 13
  • Direct ROI Align performs worse than Region Summarizer on Screen Summarization, while using ROI Align vectors as queries helps text decoding but hurts Command Grounding.The authors suggest average pooling may lose information from the full screen.Source: B ABLATION STUDY · p. 13

C PSEUDO CODE

  • The pseudocode maps bounding-box coordinates into query vectors, adds coordinate-type embeddings, then cross-attends to ViT outputs across layers.Each layer updates the queries using cross-attention and a dense transformation.Source: C PSEUDO CODE · p. 14

D PRETRAINING DATASETS

The pretraining data includes mobile UI hierarchies and web HTML attributes. The section describes the hierarchy structure, its possible misalignment with screen visuals, and filters applied to object text and examples.

D PRETRAINING DATASETS

  • A mobile view hierarchy is a tree whose leaf nodes represent screen objects and whose non-terminal nodes group objects.Leaf objects carry attributes such as type, text, visibility, and clickability.Source: D PRETRAINING DATASETS · p. 14
  • Runtime-rendered view hierarchies can contain noise and may not align with the screen's visual representation.Source: D PRETRAINING DATASETS · p. 15
  • Preprocessing removes invisible or uniform-color objects and filters generic, single-character, or URL-only object text.Text is lowercased, and continuous spaces and underscores are normalized to a single space; Table 8 reports HTML-attribute statistics for C4.Source: D PRETRAINING DATASETS · p. 15, Table 8 · p. 15

E DOWNSTREAM TASKS & DATASETS

The section describes datasets for four downstream UI tasks, distinguishing object-level tasks from screen-level summarization and providing task examples and dataset statistics.

E DOWNSTREAM TASKS & DATASETS

  • Widget Captioning, Command Grounding, and Tappability Prediction operate on individual UI objects, while Screen Summarization decodes a summary of the entire screen.The paper lists dataset statistics for all four tasks in Table 9 and shows one example of each in Figure 3.Source: E DOWNSTREAM TASKS & DATASETS · p. 15
  • Table 9 presents dataset statistics for the four downstream tasks.Source: Table 9 · p. 16
  • Figure 3 highlights target objects for illustration in three tasks; the red bounding boxes are not shown to the model.Source: Figure 3 · p. 16
Loading 2209.14927v4…