PaperlayerView exact revision on arXiv

Source-linked AI summary

Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs

Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, Zhe Gan

arXiv:2404.05719v1cs.CVcs.CLcs.HC

TL;DR

General-domain MLLMs struggle with the elongated layouts and small elements of mobile UI screens, limiting precise comprehension and interaction. Ferret-UI addresses this with aspect-ratio-aware any-resolution features, curated elementary and advanced task data, and a comprehensive benchmark; it outperforms GPT-4V on elementary tasks and other compared models on advanced tasks.

  • Problem

    General-domain MLLMs are limited in comprehending and interacting with mobile UI screens containing elongated layouts and small visual elements.

  • Method

    Ferret-UI integrates any-resolution visual processing with Ferret and trains on elementary referring and grounding tasks plus GPT-4-generated advanced reasoning tasks.

  • Results

    Ferret-UI surpasses GPT-4V on elementary UI tasks and surpasses Fuyu and CogAgent on advanced tasks.

  • Takeaways & Limitations

    Ferret-UI provides a UI-centric MLLM with referring, grounding, reasoning, and open-ended instruction capabilities for mobile screens.

  • Takeaways & Limitations

    Ferret-UI depends on UI-element detections and cannot learn undetected screen aspects such as colors, design, usability, or missed elements.

Abstract

from arXiv · show

Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.

1 Introduction

Ferret-UI targets the gap between general-domain MLLMs and the fine-grained perception, grounding, reasoning, and interaction required by mobile UI screens. It combines any-resolution visual processing, task-specific data, and a broad benchmark, outperforming GPT-4V on elementary UI tasks and other evaluated UI models on advanced tasks.

  • Capabilities: The model supports referring with points, boxes, or scribbles and grounding tasks such as finding widgets, icons, text, and listing widgets.These tasks provide visual and spatial knowledge for distinguishing UI types at coarse and fine levels and supporting advanced screen interpretation and action planning.
  • Motivation: Mobile UI automation requires whole-screen comprehension, element-level focus, language-to-action mapping, reasoning, and precise referring and grounding.Referring uses regional image information as input, while grounding identifies precise screen locations in the output.
  • Problem: General-domain referring and grounding models mainly target natural images, while UI screens have elongated layouts and smaller icons and text that can lose detail after resizing.Ferret-UI addresses this limitation with additional sub-image features based on screen aspect ratio.
  • Approach: Ferret-UI integrates any-resolution processing with Ferret and curates elementary semantic-spatial tasks alongside advanced reasoning tasks for UI screens.Elementary tasks include widget classification, icon recognition, OCR, widget finding, text finding, icon finding, and widget listing; advanced tasks include descriptions, conversations, and function inference.
  • Evaluation: Ferret-UI is evaluated on a comprehensive benchmark and significantly surpasses base Ferret, exceeds GPT-4V on elementary UI tasks, and surpasses Fuyu and CogAgent on advanced tasks.The benchmark contains 14 diverse mobile UI tasks involving referring and grounding.

2 Related Work

Related work applies MLLMs to UI understanding, navigation, and referring or grounding, but differs in task scope and dependence on external UI information. Ferret-UI-anyres extends this line with fine-grained image features for mobile screens.

  • UI understanding: Earlier work studies simplified web and mobile screens, while ILuvUI and Spotlight focus on single-screen UI tasks using GPT-generated data.Their tasks include screen summarization and widget interaction.
  • UI navigation: MobileAgent and AppAgent use MLLMs as UI-navigation agents with external detection, overlaid element IDs, or screen XML files for predefined actions.CogAgent instead uses only screen images for complex UI navigation and removes the need for UI-specific modules.
  • Architectural distinction: Ferret-UI-anyres adds fine-grained features from the full screen, aspect-ratio-based sub-images, and regional references before the LLM generates a response.The architecture combines full-image, sub-image, regional, and text representations.
  • Referring and grounding: Ferret-UI focuses on fine-grained mobile UI understanding and aligns with work on referring and grounding in multimodal models.Its architecture is based on Ferret, which supports spatial interaction with regions specified by points, boxes, or free-form shapes.

3 Method

Ferret-UI builds on Ferret’s spatial referring and grounding architecture, adds UI-specific task knowledge, and adapts visual processing to mobile screen aspect ratios. It uses raw screen pixels rather than external detection modules or screen-view files for interaction.

  • Ferret foundation: Ferret provides spatial referring and grounding for regions of varied shapes and detail using points, boxes, or free-form shapes.Its hybrid representation and spatial-aware visual sampler convert region information into continuous features suitable for the language model.
  • UI specialization: Ferret-UI adds UI referring and grounding task construction to instill domain-specific knowledge into Ferret.The task foundation covers OCR, icon recognition, widget classification, finding text, icons, or widgets, and widget listing.
  • Input design: Ferret-UI is self-sufficient because it takes raw screen pixels as input instead of requiring external detection modules or screen-view files.This design supports advanced single-screen interactions.
  • Any-resolution adaptation: Any-resolution processing selects a 1x2 or 2x1 grid according to the screen aspect ratio and partitions the resized screen into sub-images.Portrait screens are divided horizontally and landscape screens vertically, adding sub-image features to global image features.

4 Dataset and Task Formulation

The dataset combines iPhone and Android screens, detection-based elementary tasks, and GPT-4-generated advanced reasoning data. Task-specific samples and platform-diverse evaluation sets support training and benchmarking mobile UI understanding.

  • 4.1 UI Data Collection: The dataset includes mobile screens from both iPhone and Android devices, with Android data drawn from RICO and iPhone data from AMP.The collected splits contain 26,527 Android training images and 3,080 test images, plus 84,685 iPhone training images and 9,410 test images.
  • 4.1 UI Data Collection: UI element annotations are collected with a pretrained pixel-based detector, providing each detected element’s type, text, and bounding box.These detections form the basis for converting screens into task-specific MLLM data.
  • 4.2 Task Formulation: Existing Spotlight tasks are reformatted as conversational question-answer pairs by varying prompts and pairing them with source images and ground-truth answers.The tasks are screen2words, widget captions, and taperception.
  • 4.2 Task Formulation: Seven elementary tasks are generated separately for Android and iPhone: OCR, icon recognition, widget classification, widget listing, and finding text, icons, or widgets.Referring tasks receive bounding boxes in their inputs, whereas grounding tasks produce bounding boxes in their outputs.
  • 4.2 Task Formulation: Elementary training samples use GPT-3.5 Turbo to expand base prompts, while 5K test samples are used across the tasks and training mixtures sample different data ratios.The generation workflow separates icons, text, and non-icon/text widgets and creates referring and grounding samples for each type.
  • 4.2 Task Formulation: Advanced-task data uses GPT-4 with iPhone examples containing more than 2 but fewer than 15 detections, without using the actual images.The four formats are detailed description, conversation perception, conversation interaction, and function inference.
  • 4.2 Task Formulation: Advanced-task descriptions and function-inference responses are paired with prompts, while conversation outputs are parsed into multi-turn training conversations.Bounding-box formats are supported with an in-context example for conversational generation.
  • 4.2 Task Formulation: Evaluation samples cover both platforms, selecting 25 iPhone and 5 Android screens per task, with overlaps yielding 56 unique iPhone and 13 unique Android images.Two QA pairs are randomly selected for conversational tasks to create separate single-question test instances.

5 Experiments

Ferret-UI is evaluated across elementary, advanced, and Spotlight UI tasks against multiple baselines, with analyses examining resolution, task composition, OCR, grounding, and model limitations. It generally performs strongly, but results vary by platform, task, and dependence on detected UI elements.

  • Evaluation setup: The benchmark reports averaged performance for elementary and advanced tasks, alongside CIDEr for screen2words and widget captions and F1 for taperception.Grounding accuracy uses an IoU threshold of 0.5, while widget listing is treated as auxiliary and excluded from averages.
  • Main results: Ferret-UI outperforms Ferret and GPT-4V on most elementary tasks, except iPhone find text.GPT-4V performs particularly worse on Android grounding tasks, where screens contain more numerous and smaller widgets.
  • Resolution analysis: Adding anyres improves iPhone referring and grounding by 2 points, while advanced-task performance rises 20 points on iPhone but drops on Android.The advanced-task effect is therefore platform-dependent rather than uniformly beneficial.
  • Ablation studies: Elementary-task data improves advanced-task performance across platforms, with Android elementary data producing a 9% gain on Android advanced tasks.Using both platforms’ elementary data yields additional 3% and 5% improvements for iPhone and Android advanced tasks, respectively.
  • Elementary-task analysis: OCR analysis shows anyres reduces neighboring-text errors and improves recognition of small or cut-off text, while Ferret-UI can favor valid words over displayed characters.These observations connect enlarged visual inputs with improved handling of fine screen details, while also revealing semantic prediction behavior.
  • Advanced-task analysis: Ferret-UI’s grounded-conversation box accuracy is 91.7% versus GPT-4V’s 93.4%, although GPT-4V’s extra detail is favored by the evaluator.Ferret-UI generates raw coordinates, whereas GPT-4V selects from predefined boxes.
  • Limitations: The model cannot learn screen aspects absent from UI detection, including colors, design, usability, and missed elements such as status-bar components.Its reliance on detected elements limits detailed descriptions compared with GPT-4V’s broader visual commentary.

6 Conclusion

Ferret-UI is a specialized MLLM for mobile UI comprehension and interaction, using any-resolution processing and diverse task data to support referring, grounding, and reasoning.

  • Ferret-UI enhances mobile UI comprehension and interaction through any-resolution processing and training data spanning basic and advanced UI tasks.The model is designed to accommodate varied screen aspect ratios and support referring, grounding, and reasoning.

A Elementary Task Data Generation Details

The elementary-task data generation process applies task-specific formatting, filtering, and test-set sizing rules to produce UI samples for referring and grounding tasks.

  • Checkboxes and toggles merge their original “Checked” and “Unchecked” detection-label classes during data generation.
  • Widget-listing answers use a common phrase and format each element by displayed text and UI type, except text elements receive a dedicated template.
  • Samples containing multiple occurrences of the same UI element are excluded for find-text, find-icon, and find-widget tasks.
  • The test set contains the smaller of 5k and the total number of generated test instances.

B Advanced Task Data Quality Analysis

The advanced-task data analysis reports vocabulary and conversation statistics, with language patterns matching perception-focused and interaction-focused objectives.

  • Advanced-task vocabularies contain 30,866 detailed-description, 15,666 perception, 12,092 interaction, and 14,896 function-inference entries.
  • Conversation interaction contains 33,649 question turns and 29,933 answer turns, with bounding boxes in 15 questions and all answers.
  • Perception questions emphasize visual queries, whereas interaction questions and answers emphasize actions such as tapping.The interaction answers’ predominant action is tap; future work is proposed for scrolling, long-clicking, and text entry.
  • The perception and interaction conversation categories can overlap because their distinction is not always clear-cut.

C Taperception Label Analysis

The taperception label analysis compares human labels with model outputs and dataset labels, finding stronger agreement with GPT-4V and ambiguity in image-only tappability judgments.

  • 30 taperception samples were labeled, including 5 that were difficult to decipher without direct screen interaction.
  • Human labels correlate more strongly with GPT-4V predictions than with taperception dataset labels.The comparison includes filtered unambiguous instances and the full unfiltered set.
  • The disagreement with taperception labels highlights the difficulty of predicting tappability from single images and obtaining clear-cut labels.

D Advanced Task Generation Prompts

The paper presents the prompts used to collect advanced-task data from GPT-4.

  • Prompts were presented for collecting advanced-task data from GPT-4.

E GPT-4V Evaluation Details

The GPT-4V evaluation data were constructed with task-specific screenshot annotations and prompts. Advanced-task generation used dedicated GPT-4 prompts, while several tasks constrained GPT-4V responses to fixed formats or candidate choices.

  • Input Images: GPT-4V inputs were prepared from task-specific screenshot annotations, using original images when tasks required no bounding boxes.For referring tasks, a magenta bounding box marked the specific UI element in the input image.
  • Data Generation: Advanced-task data generation used prompts for GPT-4, while GPT-4V input examples illustrated referring and grounding setups.The referring example concerned one UI element, whereas the grounding example used assigned numeric labels.
  • Input Images: Grounding inputs associated candidate UI elements with numeric labels, enabling GPT-4V to select a label instead of producing coordinates.This candidate-based version addressed GPT-4V’s inability to provide exact bounding-box coordinates.
  • Prompts: Prompts imposed task-specific output constraints, including yes/no answers, brief captions, class-only responses, and numeric-label outputs.These instructions were designed to provide the information and response format needed for each task.
Loading 2404.05719v1…