PaperlayerView exact revision on arXiv

Source-linked AI summary

Tell Me What's Next: Textual Foresight for Generic UI Representations

Andrea Burns, Kate Saenko, Bryan A. Plummer

arXiv:2406.07822v2cs.CVcs.CL

TL;DR

Element-level and global captioning losses do not retain both granularities of UI meaning. Textual Foresight predicts a future screen description from a current UI and localized action, and achieves improved task performance with less pretraining data. OpenApp provides public data and a standardized benchmark for app UI representation learning.

  • Problem

    Element captioning captures local UI meaning but lacks context from the global screen or what an element affords.

    Source: §1 · p. 2

  • Method

    Textual Foresight generates a global description of a future UI from the current screen and a localized action.

    Source: §1 · p. 2

  • Results

    Textual Foresight yields a 5.7% average task performance boost over open-source baselines, using 28x less pretraining data than Spotlight and 2x less than new baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    OpenApp supplies a publicly available app UI dataset and a standardized downstream benchmark for evaluating generic UI representations.

    Source: §7 · p. 9

    The authors report that their methods fall short on prediction-style tasks and that their work is currently most effective for captioning and summarization tasks.

    Source: §8 · p. 10

Abstract

from arXiv · show

Mobile app user interfaces (UIs) are rich with action, text, structure, and image content that can be utilized to learn generic UI representations for tasks like automating user commands, summarizing content, and evaluating the accessibility of user interfaces. Prior work has learned strong visual representations with local or global captioning losses, but fails to retain both granularities. To combat this, we propose Textual Foresight, a novel pretraining objective for learning UI screen representations. Textual Foresight generates global text descriptions of future UI states given a current UI and local action taken. Our approach requires joint reasoning over elements and entire screens, resulting in improved UI features: on generation tasks, UI agents trained with Textual Foresight outperform state-of-the-art by 2% with 28x fewer images. We train with our newly constructed mobile app dataset, OpenApp, which results in the first public dataset for app UI representation learning. OpenApp enables new baselines, and we find Textual Foresight improves average task performance over them by 5.7% while having access to 2x less data.

1 Introduction

Mobile UI representations must account for both the rendered screen and other UI information, while element captioning alone captures local meaning without screen context. Textual Foresight uses a localized action to connect element-level and global screen semantics, and the paper reports improved performance alongside OpenApp, a public dataset for UI representation learning.

1 Introduction

  • AI agents can help people who cannot or prefer not to view or touch app screens, motivating research into which modalities should represent app UIs.The UI may include rendered images, metadata, text, and structural features such as the underlying view hierarchy.Source: §1 · p. 1
  • The paper frames Textual Foresight against element captioning as an objective that additionally models global screen semantics and how an action changes the UI.Figure 1 illustrates the comparison between local object semantics and the global description of the following screen.Source: Figure 1 · p. 1, §1 · p. 1
  • Element captioning can label an icon such as “options,” but that local label lacks context about the screen or what action the element affords.Seeing the screen after clicking the element reveals that it can like, hide, or share a song.Source: §1 · pp. 1–2
  • Textual Foresight generates a global description of a future UI from the current screen and a localized action, joining local element and global screen semantics.The action also provides state-action examples that implicitly teach element affordance.Source: §1 · p. 2
  • OpenApp is presented as the first publicly available dataset for app representation learning, with multiple element- and screen-level caption sets for reproducing Spotlight and training baselines.The paper also provides code and checkpoints publicly.Source: §1 · p. 2
  • 5.7% average task performance boost over open-source baselines, with Textual Foresight using 28x less pretraining data than Spotlight and 2x less than new baselines.Textual Foresight also reaches the best average performance for screen summarization and element captioning.Source: §1 · p. 2

2 Related Work

Prior UI representation methods are difficult to compare because they use proprietary data or evaluate on different tasks. Textual Foresight is presented as a generation-style objective that incorporates UI actions, alongside related approaches to image captioning, visual foresight, and web understanding.

2 Related Work

  • Prior UI representation methods either use proprietary data or evaluate on different tasks, making downstream comparisons challenging.Figure 2 compares methods by predictive versus generative loss and by whether they use UI actions.Source: §2 · p. 2
  • Textual Foresight is the first generation-style loss in this comparison to incorporate action, while Textual Foresight and Spotlight use only the rendered screen to represent the UI.The comparison includes ActionBERT, Screen2Vec, UIBERT, and Spotlight.Source: §2 · pp. 2–3
  • Global image captioning has been used to learn natural-image representations, but image captioning had not been studied for UI representations because screen-caption data was unavailable.Prior work used the approach for vision-only tasks including classification, detection, and segmentation.Source: §2 · p. 3
  • Earlier visual foresight methods in robotics, reinforcement learning, and vision-language navigation predict raw images or intermediate visual features rather than language descriptions of future states.The cited work includes Finn and Levine (2017), Lee et al. (2018), and Wang et al. (2018).Source: §2 · p. 3
  • Related web research includes multimodal web agents, web summarization, and web captioning.The cited examples include Koh et al. (2024), Burns et al. (2023), and Srinivasan et al. (2021).Source: §2 · p. 3

3 UI Representation Learning with Textual Foresight

Textual Foresight trains a vision-language model to predict a global caption of the next UI screen from the current screen and a localized action. This objective is designed to capture both local element and global screen features, then transfer those representations to downstream UI tasks.

3.1 Textual Foresight Definition

  • Textual Foresight predicts a caption of the unseen next screen from the current screen and a question identifying the interacted element.The question includes the element’s normalized bounding-box coordinates, not its text, to encourage use of visual context.Source: §3.1 · p. 4
  • The model minimizes cross-entropy language-modeling loss on the target future-screen caption, using the current screen and foresight question as inputs.At each decoding step, the loss targets the correct vocabulary word; the VLM’s vocabulary probabilities come from its Softmax outputs.Source: §3.1 · p. 4
  • Unlike standard image captioning, Textual Foresight refines current-screen features by requiring them to capture global semantics and the action’s meaning.It can use action sequences involving elements with noisy or absent text, which would otherwise be unusable for element captioning.Source: §3.1 · p. 4

3.2 Pretraining Model

  • The method uses BLIP-2 and replaces its second-stage image-captioning objective with Textual Foresight to refine Q-Former visual query embeddings.The ViT and Q-Former are trainable during downstream finetuning, while the language model remains frozen.Source: §3.2 · p. 4, §3.3 · p. 5

4 OpenApp Dataset

OpenApp combines existing app datasets and adds annotations and processing to support Textual Foresight and other pretraining objectives. Its element- and screen-level captions provide data for local, global, and future-screen captioning.

4 OpenApp Dataset

  • OpenApp combines MoTIF, one snapshot from Fok et al.’s longitudinal study, and Android in the Wild, then annotates and post-processes the merged data.The raw data contains app action sequences with action annotations, screenshots, and view hierarchies.Source: §4 · p. 5

4.1 Element-Level Captions

  • Element captions come from qualifying view-hierarchy text and bounding boxes, filtered for visibility, valid bounds, and non-generic text occurring at least five times.The authors deduplicate app, caption, and bounding-box triplets; they do not add OCR annotations, though AITW annotations were obtained via OCR.Source: §4.1 · p. 5
  • Table 1 reports image and sample counts for Spotlight element captioning, OpenApp element captioning, screen captioning, and Textual Foresight datasets.Spotlight counts are approximate, and the Textual Foresight dataset is the subset of screen captions valid for foresight.Source: Table 1 · p. 5

4.2 Screen-Level Captions

  • OpenApp screen captions are generated with GPT-3.5 Turbo from screen elements, then used as static screen-caption pairs or future-screen captions for valid action triplets.Foresight examples require distinct consecutive screens, a tap action, a non-final sequence state, and a mapped interaction bounding box; this yields almost half as many samples as screen captioning.Source: §4.2 · pp. 5–6

5 Experimental Setup

The experimental setup introduces OpenApp-enabled baselines and describes pretraining, finetuning, and four downstream UI tasks. Evaluation uses generation metrics for captioning and classification or grounding metrics for predictive tasks.

5.1 Baselines

  • OpenApp supports element-list captioning and screen-captioning baselines alongside Textual Foresight, but short element captions caused BLIP-2 optimization issues.The authors therefore compare element captioning against prior published results while releasing the annotations for reuse.Source: §5.1 · p. 6

5.2 Pretraining Settings

  • Pretraining follows BLIP-2 settings without upstream parameter tuning, uses batch size 100 for five epochs, and reports FlanT5 results after model ablations.Images are input at 224x224 resolution, smaller than Spotlight’s 740x740 inputs, due to model-size and GPU-memory constraints.Source: §5.2 · p. 6

5.3 Finetuning Settings

  • Finetuning runs for five epochs with batch size 16, validation-selected learning rates and warmup steps, early stopping, and a single reported run.The learning rate is 1e-5 for screen summarization and element captioning, and 5e-5 for tappability and language grounding; images use 364x364 resolution.Source: §5.3 · pp. 6–7

5.4 Downstream UI Tasks and Metrics

  • The benchmark covers screen summarization, element captioning, tappability prediction, and language grounding, using MUG because Spotlight’s grounding dataset was not open-sourced.About 80% of MUG samples are single-turn; the remaining samples use full multi-turn instructions.Source: §5.4 · pp. 7–8
  • CIDEr evaluates screen and element captioning, while F1 evaluates tappability prediction and accuracy evaluates language grounding.BERTScore and BLEURT-20-D12 are also reported for captioning in the appendix.Source: §5.4 · p. 8

6 Results

The results compare generative tasks—screen summarization and element captioning—with predictive tasks—tappability prediction and language grounding. Textual Foresight combines strong generative-task performance with improvements over the open-source baselines on predictive tasks.

6.1 Generative Tasks

  • BLIP-2 without app-specific pretraining scored 125.1 vs. 106.7 CIDEr points on screen summarization against Spotlight, but trailed it on average, 123.3 vs. 124.3.The authors relate this contrast to global image captioning versus Spotlight's element-caption pretraining and its stronger local-task performance.Source: §6.1 · p. 8
  • Screen-caption pretraining slightly improved screen summarization over BLIP-2 directly but worsened element captioning, scoring 118.9 vs. 121.4.The authors note that GPT captions covered 3.4M unique (app, element list) pairs among 5.7M unique OpenApp images, and suggest incomplete text may conflate differing screens.Source: §6.1 · p. 8
  • 127.9 on screen summarization made element list captioning the strongest screen-summarization pretraining objective, and it was the first baseline to outperform Spotlight on average across the two generative tasks.Its target is the concatenated list of text elements; the authors suggest raw element information may retain useful information for global reasoning.Source: §6.1 · p. 8
  • +6.4 CIDEr points on element captioning over the best open-source baseline, while Textual Foresight also achieves state-of-the-art screen summarization using 28x fewer images than Spotlight.It uses 3M fewer samples than element list captioning, the second-best method.Source: §6.1 · p. 9

6.2 Predictive Tasks

  • Up to 10.3 F1 Score for tappability and 9.7 accuracy points for language grounding distinguish Textual Foresight among the open-source methods, despite using almost half the data of other BLIP-2 variants.These variants remain less performant than prior approaches overall.Source: §6.2 · p. 9
  • Predictive-task finetuning used 14k training samples for tappability and 65k for language grounding, compared with 138k and 78k for element and screen captioning.The authors identify these smaller predictive-task datasets as a consideration when interpreting the results.Source: §6.2 · p. 9

7 Conclusion

The paper introduces Textual Foresight, which uses an action on the current UI to describe a future screen, and contributes OpenApp and a standardized downstream benchmark. On generation tasks, Textual Foresight outperforms prior state of the art while using 28x less data, and uses 2x less data than the open-source baselines it outperforms on average.

7 Conclusion

  • Textual Foresight uses UI actions to connect local element semantics with global screen context, generating a description of the future screen from the current state and action.The paper also contributes OpenApp, with screen- and element-level captions for 5.7M app images, and evaluates on a standardized downstream benchmark.Source: §7 · p. 9
  • On generation tasks, Textual Foresight outperforms prior state of the art using 28x less data, and outperforms the open-source baselines on average using 2x less data.OpenApp provides a public dataset for UI representation learning.Source: §7 · p. 9

8 Limitations

The paper identifies limits in OpenApp’s LLM-generated captions, prediction-style performance, and English-oriented data. It also notes that human annotation or verification could improve dataset quality.

8 Limitations

  • LLM-generated captions may omit image content or lose information useful for representation learning; human annotation or verification could improve OpenApp’s quality.The authors also note that all baselines, including Textual Foresight, fall short on prediction-style tasks; their approach is currently most effective for captioning and summarization.Source: §8 · p. 10
  • All baselines and Textual Foresight fall short on prediction-style tasks, and the authors suggest model framework or data scale may contribute.They did not test every combination of pretraining objectives because of computational and time constraints.Source: §8 · p. 10
  • The datasets were designed and built as English datasets, so models trained for the tasks are currently reliable only for English.The authors call for intentionally curated multilingual UI data and measurement of multilingual content in existing sources.Source: §8 · p. 10

9 Ethics

The paper describes ethical considerations for UI automation and personalization, then details OpenApp’s data-processing pipeline. The pipeline releases code and applies distinct processing choices to element-level and screen-level captions.

9 Ethics

  • UI automation can act on users’ behalf when they cannot double-check outputs, motivating explainable predictions and automation only at high model confidence.The authors say this concern is less applicable to captioning and summarization; they also flag the risk of retaining user-specific information after fine-tuning or personalization, which is outside this work’s scope.Source: §9 · p. 10

A Data Processing Details

  • OpenApp adds annotations to existing open-source datasets rather than collecting new action sequences, and its data-processing code is released for reproduction or modification.The UI data was originally annotated using anonymous login credentials.Source: §9 · p. 10, A Data Processing Details · p. 13

A.1 Element Captioning Data

  • Element captions use eligible view-hierarchy text, content descriptions, or resource IDs, with filters for text quality, repetition, visibility, and valid non-color-block bounding boxes.The rules address potential view-hierarchy noise and inaccurate boxes, which matter for localized tasks.Source: §A.1 · p. 13

A.2 Element List Captioning Data

  • Element-list captions concatenate each screen’s element-caption text with commas, forming a screen-level task whose target is an element-list string rather than natural-language text.The source elements come from the element-captioning dataset.Source: §A.2 · p. 13

A.3 Screen Captioning Data

  • Screen-caption prompts ask GPT-3.5 Turbo for a single sentence about screen functionality and app category, using unique cleaned element lists as inputs.This pipeline omits resource-ID text as noisy metadata and retains uppercase text; identical cleaned lists are queried once, and repeated API calls can vary even at temperature zero.Source: §A.3 · p. 13

A.4 Textual Foresight Data

Textual foresight captions use GPT-3.5 Turbo outputs for screens with taps, while the dataset pipeline maps action coordinates to UI elements and handles unmatched or overlapping boxes. The examples contrast caption sets and show foresight captions for next screens.

A.4 Textual Foresight Data

  • Textual foresight uses only screens with tap actions, since swipes or text-field edits may not change the UI enough to merit a distinct future-state caption.The captions use the same GPT-3.5 Turbo outputs as screen captioning.Source: §A.4 · p. 13
  • When an action misses a view-hierarchy box, the pipeline enlarges nearby bounds and, if needed, creates a 65x65-pixel box centered on the action; overlapping matches are resolved by proximity and area.Back-banner actions are excluded, and some action-to-element mismatches arise from coordinate differences or OCR omissions.Source: §A.4 · p. 14

B Dataset Examples

  • Figure 5 examples distinguish element, element-list, screen, and foresight captions; foresight uses the current screen and an action question, with captions shown beneath the next screen.Some foresight captions act as more descriptive element captions even when they add little beyond the action element’s semantics.Source: B Dataset Examples · p. 14

C Dataset Noise

OpenApp’s noise comes from imperfect action-to-box matching and GPT-generated captions that can miss salient screen details. The appendices also describe downstream-task prompts, text-generation formulations, computational costs, and additional ablations.

C Dataset Noise

  • Action-to-box matching is imperfect: the authors inspected about 100 processed samples per dataset, and note that textual foresight does not always require perfect localization.When no exact match exists, the pipeline may use a nearby box or create one around the action coordinate.Source: C Dataset Noise · p. 14
  • GPT-3.5 Turbo captions can miss salient screen details when distractor elements obscure the screen’s focus, although the authors say unrelated outputs are unlikely.A StubHub gift-card example remained unfocused across stringent, raw, and intermediate element-processing variants; more expensive GPT-4 or raw-hierarchy inputs were discussed as possible alternatives.Source: C Dataset Noise · pp. 14, 17, Figure 6 · p. 16

D.1 Finetuning Set Up

  • For downstream tasks other than screen summarization, models receive a task-specific question during fine-tuning.The supplied widget question asks what describes the functionality of a UI object, and other prompts refer to elements at specified coordinates.Source: §D.1 · p. 17

D.2 Formulating Prediction Tasks as Text Generation

  • Tappability is generated as yes/no interaction captions, while language grounding generates a command for each element and selects the element whose command best matches the ground truth.BLEURT is used for command-caption similarity because it is more correlated with human judgment.Source: §D.2 · p. 17

E Computational Details

  • Textual Foresight pretraining takes about 1.5 days, compared with 3 days for larger element-list and screen-captioning datasets, using four GPUs.Fine-tuning takes 2–6 hours per experiment; models use 188M trainable parameters in pretraining and 1.2B parameters in fine-tuning.Source: E Computational Details · p. 17

F Ablations

  • The additional ablations cover evaluation metrics, replacing FlanT5 with OPT, learning-rate and warm-up settings, and training from a BLIP-2 checkpoint versus from scratch.The authors state they did not test every possible combination of pretraining objectives because of time and computational constraints.Source: F Ablations · p. 17

F.1 Additional Metrics

This appendix adds text-similarity and predictive-task metrics, then reports screen-summarization language-model and learning-rate ablations and pretrained-checkpoint and warmup ablations.

F.1 Additional Metrics

  • BERTScore and BLEURT are less sensitive to caption changes; trends remain consistent for element captioning, but the metrics do not capture screen-summarization differences.Table 4 reports CIDEr, BERTScore, and BLEURT-D12 for generative tasks.Source: §F.1 · p. 18, Table 4 · p. 19
  • Tappability accuracy follows the F1 trend, while the grounding metric changes absolute accuracy but leaves method trends unchanged.Table 5 reports tappability F1 and accuracy and language-grounding accuracy using different text-similarity metrics.Source: §F.1 · p. 18, Table 5 · p. 19

F.2 OPT and Learning Rate Ablations

  • Screen-summarization ablations vary the BLIP-2 language model and initial learning rate, selecting settings by validation performance.The tested language models are FlanT5, OPT2, and OPT6; the table shows a subset of these ablations.Source: §F.2 · p. 18, Table 6 · p. 19

F.3 Pretrained Checkpoint and Warm Up Ablations

  • Initializing from a stage-one BLIP-2 checkpoint consistently performs better than training from scratch, while the best warmup-step count varies by pretrained model.Tables 7–9 cover screen summarization, element captioning, and tappability prediction, tuning warmup steps by validation performance.Source: §F.3 · p. 18, Table 7 · p. 20, Table 8 · p. 21

Related papers

Loading 2406.07822v2…