Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms

1 INTRODUCTION

• Earlier Ferret-UI is constrained by fixed grounding resolutions and a focus on mobile platforms, while platform-specific, high-quality training data is limited.Its text-only GPT-4 prompting represents bounding boxes without visual input or spatial relationships between UI elements.Source: §1 · p. 1
• Ferret-UI 2 extends support to iPhone, Android, iPad, webpages, and AppleTV, beyond Ferret-UI’s mobile focus.The paper describes the model as responding to user intent through single-step interactions across multiple platforms.Source: §1 · pp. 1, 3
• Adaptive gridding supports high-resolution UI perception, while human-collected bounding-box annotations are used to improve referring and grounding precision.The approach aims to preserve perception at the screenshot’s original resolution and better capture UI components and their relationships.Source: §1 · p. 3
• For advanced tasks, Ferret-UI 2 uses GPT-4o with set-of-mark visual prompting to generate multimodal training data for user-centered interactions.For elementary tasks, it converts simple referring and grounding data into conversations; advanced interactions target user intent rather than mechanical click instructions.Source: §1 · p. 3
• Ferret-UI 2 outperforms Ferret-UI across referring, grounding, and user-centric advanced tasks, with strong cross-platform transfer and performance on GUIDE and GUI-World.The advanced tasks span 9 subtasks × 5 platforms; the paper also reports competitive performance compared with GPT-4o.Source: §1 · p. 3

2 RELATED WORK

• Single-platform agents have improved task-specific automation, but their platform focus limits cross-platform flexibility.The cited examples span mobile, web, and computer operating-system interaction.Source: §2 · p. 3
• Multi-platform agents address diverse device ecosystems, including mobile, web, and desktop environments.Examples discussed include systems supporting PC webpages and Android, and agents spanning multiple mobile operating systems.Source: §2 · pp. 3–4
• UI benchmarks cover mobile, desktop, web, and cross-app tasks, with GUI-World identified as a multi-platform benchmark.The paper describes a growing need for unified multi-platform evaluations of adaptability, precision, and efficiency.Source: §2 · p. 4
• Ferret-UI 2 targets universal UI understanding across smartphones, tablets, web interfaces, and smart TVs, focusing on referring, grounding, and reasoning.The stated aim is to create a generalist agent for versatile UI navigation.Source: §2 · p. 4

3.1 DATASET CONSTRUCTION

• The Core-set combines iPhone, Android, iPad, Webpage, and AppleTV screenshots, using human annotations, source-HTML boxes, and OCR-derived text annotations.Webpage widget boxes and non-picture text annotations are parsed from HTML; mobile and AppleTV data use human-collected screenshots and annotated widget boxes.Source: §3.1 · p. 5, Table 1 · p. 5
• The training data is uneven across platforms, so iPad and AppleTV examples receive different loss weights and all three advanced-task types are generated for each example.Other platforms receive one advanced-task type per example.Source: §3.1 · p. 5
• Compared with Ferret-UI's model-detected boxes, Ferret-UI 2 predominantly uses human-collected or HTML-parsed boxes, which the authors report improves annotation quality.The paper says this improvement is evidenced in its Section 4.2 quantitative evaluation.Source: §3.1 · p. 5
• Elementary training data covers three referring and three grounding tasks, while Figure 3 illustrates set-of-mark prompting alongside a generated advanced-task example.The elementary tasks include OCR, widget classification, tappability, widget listing, finding text, and finding a widget.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 MODEL ARCHITECTURE

• The architecture builds on Ferret-UI's Any-Resolution method: CLIP extracts global and local image features, and a Visual Sampler selects instruction-relevant regions for grounded outputs.Adaptive gridding determines the local image crops that are resized and encoded.Source: §3.2 · pp. 7–8
• Adaptive N-gridding selects grid dimensions that minimize aspect-ratio and pixel-number changes within a predefined inference-cost limit.The grid dimensions satisfy Nw + Nh ≤ N, bounding the number of grids while preserving resolution information.Source: §3.2 · p. 8

4.1 EXPERIMENT SETUP

• The evaluation covered 45 constructed tasks—6 elementary and 3 advanced per platform—plus GUIDE and GUI-World; advanced tasks used GPT-4o scoring and multi-IoU.Table 3 reports zero-shot GUI-World performance; Table 4 reports transfer using elementary tasks only.Source: §4.1 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9

4.2 EXPERIMENT RESULTS

• 89.73 GPT-4o score on advanced tasks surpassed Ferret-UI by 43.92 points and GPT-4o by 12.0 points; Llama-3-8B also scored 55.78 IoU on GUIDE.Ferret-UI 2 with Llama-3-8B delivered the best results across most metrics.Source: §4.2 · p. 9
• On GUI-World, Ferret-UI 2 generalized zero-shot and outperformed GUI-Vid on supported iOS, Android, and webpage platforms.The benchmark follows the original GPT-4 evaluation protocol, which does not assess grounding for interaction-related tasks.Source: §4.2 · p. 9, §4.1 · p. 9

4.3 ABLATION STUDY

• iPhone transferred to iPad and Android with at least 68.1 and 71.2 referring scores and 65.2 and 63.1 grounding scores, while iPad and Android transferred to iPhone at around 65.The authors associate this pattern with diverse iPhone screenshots and similar resolution and aspect ratios among the mobile platforms.Source: §4.3 · p. 10
• AppleTV and Web transferred poorly to mobile domains, while models trained on other domains scored around 40 at most on AppleTV, which has a large content-distribution gap.The authors suggest the mostly landscape AppleTV and Web screenshots contrast with mostly portrait mobile screenshots.Source: §4.3 · p. 10
• On iPhone v2 advanced tasks, Ferret-UI 2 scored 89.7 versus Ferret-UI’s 85.97 GPT-4o score and 41.73 versus 39.81 Multi-IoU; gains were smaller on iPhone v1.The authors report that both adaptive N-gridding and the improved dataset contribute to gains, with the dataset playing a larger role; Table 5 compares these improvements against Ferret-UI-anyRes.Source: §4.3 · p. 10, Table 5 · p. 10

5 CONCLUSIONS

• Ferret-UI 2 outperformed Ferret-UI on all tested benchmarks and showed strong zero-shot transferability across platforms.Future work will add platform types and build a generalist agent for universal UI navigation.Source: §5 · p. 10

• The organization’s legal team approved the use of UI-understanding data for this paper.Source: ETHICS STATEMENT · p. 11
• The authors state that they strictly adhered to ethical standards.Source: ETHICS STATEMENT · p. 11
• They report that no private or sensitive information was used or compromised.Source: ETHICS STATEMENT · p. 11

• Ferret-UI 2 is evaluated on 9 subtasks × 5 platforms in the reported performance breakdown.Source: A DETAILED RESULTS ON ELEMENTARY AND… · p. 15, Table 6 · p. 15
• OCR, widget listing, and comprehensive description are relatively difficult compared with other tasks within the same task type.Source: A DETAILED RESULTS ON ELEMENTARY AND… · p. 15
• The breakdown uses a Llama-3-8B backbone and reports GPT-4o evaluation scores, omitting advanced-task multi-IoU scores for simplicity.Source: Table 6 · p. 15

B GROUNDING PERFORMANCE ON SCREENSPOT BENCHMARK

• 54.0% average accuracy on ScreenSpot, outperforming CogAgent and SeeClick while performing well on supported Mobile and Web and fairly on unseen Desktop screenshots.The benchmark data distribution was unseen during Ferret-UI 2 training.Source: B GROUNDING PERFORMANCE ON SCREENSPOT… · p. 15, Table 7 · p. 15

C DETAILED GPT-4O PROMPTS FOR GENERATING ADVANCED TASK DATA

• Comprehensive Description prompts request an overall page-function summary and detailed explanations of areas or groups.Multi-Round Perception QA covers widget content, positions, functionality and status, contextual relationships, and layout grouping and hierarchy.Source: C DETAILED GPT-4O PROMPTS FOR… · pp. 15–16
• Multi-Round Interaction QA asks for user-centered instructions and interaction with controls, while distinguishing interactive widgets from static text and images.Prompts also cover inputs and controls such as fields, dropdowns, checkboxes, and radio buttons, as well as widget states and navigation steps.Source: C DETAILED GPT-4O PROMPTS FOR… · p. 16

Comprehensive Description

• A settings-page description identifies navigation, download, language, news, and notification controls, including their displayed selections and states.The WiFi-only and news checkboxes are checked, the notifications checkbox is unchecked, and the language dropdown is set to English.Source: D ADVANCED TASK EXAMPLES · p. 16

Multi-Round Perception QA

• Product questions identify two item prices, describe the ARV BR ¨OLLOP rating as stars with a numerical 4.9, and explain the shopping cart and Filter controls.The examples also identify a bottom navigation bar and its home and pencil icons.Source: D ADVANCED TASK EXAMPLES · p. 17

Multi-Round Interaction QA

• Interaction examples explain how to refresh content and view recent activities, and describe the General tag as categorizing general-event content.Another example identifies the hosting institution as the California Institute of Technology.Source: D ADVANCED TASK EXAMPLES · p. 17

E DATASET COMPARISON

• Ferret-UI 2 training data consists of 5 platforms, compared with 2 for Ferret-UI; the paper compares its dataset with previous multi-platform datasets in Table 8.Source: E DATASET COMPARISON · p. 17
• Most Ferret-UI 2 bounding boxes, labels, and on-screen text come from source data or human annotation, whereas Ferret-UI annotations are all model-detected.The dataset comparison contrasts the Ferret-UI and Ferret-UI 2 data-generation pipelines.Source: E DATASET COMPARISON · pp. 17–18, Figure 6 · p. 18
• Ferret-UI 2 uses GPT-4o with set-of-mark visual prompting for advanced tasks, while Ferret-UI uses textual prompts with coordinates, box labels, and text annotations.The visual-prompting example shows widgets marked with corner-style boxes and number tags.Source: E DATASET COMPARISON · p. 18, Figure 5 · p. 18
• The authors say Ferret-UI's textual prompting cannot perceive UI elements' visual content, limiting the quality and content diversity of generated advanced tasks.Source: E DATASET COMPARISON · pp. 18–19

F RESOLUTION STATISTICS

• The resolution section summarizes image-resolution statistics by device type; Table 9 is captioned as resolution statistics by device type.Source: F RESOLUTION STATISTICS · p. 19, Table 9 · p. 19

G LABEL STATISTICS AND MAPPING RESULTS OF ORIGINAL DATA ACROSS PLATFORMS

• Original labels from platform data are mapped into a uniform label space using GPT-4 suggestions and additional human review.The post-mapping “Other” label means widget information will be deprecated; the figures compare original-label wedges with converted-label colors across iOS, Android, AppleTV, and webpage data.Source: G LABEL STATISTICS AND MAPPING RESULTS… · p. 19, Figure 7 · p. 20, Figure 8 · p. 20

H ADDITIONAL INFERENCE EXAMPLES

• Additional qualitative examples show Ferret-UI 2 on iPhone, iPad, Android, and AppleTV, alongside a multi-step interaction on real-time webpages following GUIDE-style QAs.Source: H ADDITIONAL INFERENCE EXAMPLES · p. 19, Figure 11 · p. 22, Figure 12 · p. 22