PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniParser for Pure Vision Based GUI Agent

Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah

arXiv:2408.00203v1cs.CVcs.AIcs.CLcs.LG

TL;DR

Vision-language UI agents lack robust screenshot parsing for reliable interactable-element identification and action grounding across platforms and applications. OmniParser combines interactable-region detection, icon descriptions, and OCR to produce structured screenshot representations, improving GPT-4V across ScreenSpot, Mind2Web, and AITW with screenshot-only input. The method still has author-identified failure cases involving repeated elements and coarse bounding boxes.

  • Problem

    Vision-language UI agents lack robust screen parsing for identifying interactable icons, understanding element semantics, and grounding actions across multiple platforms and applications.

  • Method

    OmniParser combines finetuned interactable-region detection and icon-description models with OCR to parse screenshots into structured elements for GPT-4V.

  • Results

    OmniParser improves GPT-4V performance on ScreenSpot, while screenshot-only OmniParser outperforms GPT-4V baselines using additional information on Mind2Web and AITW.

  • Takeaways & Limitations

    The results support OmniParser as a general vision-only tool for parsing user screens and improving grounded action generation across varied UI tasks.

  • Takeaways & Limitations

    GPT-4V often fails when parsed interfaces contain repeated icons or texts, and OmniParser can produce bounding boxes that are too coarse for clickable text.

Abstract

from arXiv · show

The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce \textsc{OmniParser}, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. \textsc{OmniParser} significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, \textsc{OmniParser} with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot.

1 Introduction

OmniParser addresses the gap between promising vision-language UI agents and broadly usable agents by parsing screenshots into structured, actionable elements. It combines specialized screen-parsing models and improves GPT-4V performance across multiple UI benchmarks without requiring information beyond screenshots.

  • Research gap: Current vision-language UI agents remain limited for broadly usable operation across multiple operating systems and applications.Most prior work focuses on restricted applications or platforms.
  • Research gap: Action grounding is difficult because GPT-4V cannot reliably produce exact screen coordinates, while Set-of-Marks prompting grounds actions to numbered bounding boxes.The approach replaces direct coordinate prediction with selection of a specific bounding box.
  • Approach: OmniParser extracts structured bounding boxes and labels from UI screenshots to enhance GPT-4V action prediction across varied user tasks.The method is presented as a general screen-parsing tool for operating systems and applications.
  • Approach: OmniParser combines a curated interactable-region dataset with multiple finetuned models for pure vision-based interface parsing.The contributions include DOM-derived interactable-region data and a parsing method combining specialized models.
  • Evaluation: OmniParser significantly improves the original GPT-4V baseline on ScreenSpot, Mind2Web, and AITW without additional input beyond screenshots.The reported evaluation covers all three benchmarks and compares against the original GPT-4V baseline.

2 Related Works

Prior UI understanding and GUI-agent work often relies on view hierarchies, DOM information, or task-specific settings. OmniParser responds with a current, general web UI dataset built from DOM-derived interactable-region annotations for vision-only parsing.

  • UI screen understanding: Earlier UI-screen models extract semantics effectively but rely on view hierarchy or target visual question answering and screen-summary tasks.These constraints limit their direct use as general screenshot-only screen-understanding tools.
  • Datasets: Existing UI datasets include large screen collections and annotations, but general web-screen datasets do not explicitly target interactable-icon detection on real-world websites.The limitation is stated for web and general operating-system domains.
  • Datasets: OmniParser curates an up-to-date icon-detection dataset from DOM information collected from popular web URLs.DOM-derived bounding boxes provide ground-truth locations for current icons and buttons.
  • GUI agents: Many autonomous GUI agents use DOM or view-hierarchy information to obtain interactable-element locations and overlay bounding boxes for vision-language models.This includes end-to-end action-prediction approaches and Set-of-Marks prompting.
  • Motivation: Because ground-truth interactable information may be unavailable for cross-platform and cross-application tasks, the paper targets systematic extraction of structured elements from general screens.The stated goal is a general screen-parsing approach rather than dependence on environment-specific metadata.

3 Methods

OmniParser separates screen understanding from action prediction by combining interactable-region detection, icon description, and OCR into a structured UI representation. Its outputs provide both a marked screenshot and local textual semantics for GPT-4V.

  • Methods: OmniParser separates UI understanding from next-action prediction so GPT-4V can use extracted screen semantics while focusing on action selection.The method decomposes complex tasks into action steps and moves part of screen analysis into parsing.
  • Methods: The parser integrates finetuned interactable-icon detection, finetuned icon description, and OCR to produce a DOM-like UI representation and marked screenshot.The marked screenshot overlays bounding boxes for potential interactable elements.
  • Interactable Region Detection: Instead of using ground-truth web or mobile locations, OmniParser finetunes a detection model to extract interactable icons and buttons from screenshots.The model follows Set-of-Marks prompting but predicts box IDs from detected regions.
  • Interactable Region Detection: The interactable-region dataset contains 67k unique screenshots with bounding boxes for interactable icons derived from webpage DOM trees.The screenshots were collected from a uniform sample of 100k popular web URLs.
  • Interactable Region Detection: OCR and icon-detection boxes are merged, highly overlapping boxes are removed, and each remaining box receives a unique numeric ID.The overlap-removal threshold is over 90%.
  • Incorporating Local Semantics: Local semantics are added because screenshot boxes and IDs alone can mislead GPT-4V when it must infer icon meaning and choose the next action simultaneously.The parser supplies text and icon descriptions alongside the visual prompt.
  • Incorporating Local Semantics: The icon-description component is trained on 7k GPT-4o-generated icon-description pairs by finetuning BLIP-v2.The resulting model is reported as more reliable for common app-icon descriptions.

4 Experiments and Results

OmniParser is evaluated through controlled label assignment and three UI-agent benchmarks, testing local semantics, interactable-region detection, and screenshot-only operation. Across these evaluations, the method improves GPT-4V grounding and navigation performance relative to relevant baselines.

  • SeeAssign Evaluation: SeeAssign evaluates whether GPT-4V can assign the correct numeric label to a referred bounding box across mobile, desktop, and web screenshots.The 112 tasks are divided into easy, medium, and hard categories according to the number of bounding boxes.
  • SeeAssign Evaluation: 0.705 to 0.938: adding text and icon descriptions as local semantics improves GPT-4V’s correct icon assignment.The evaluation also observes frequent ID mistakes when many bounding boxes overlay the screen.
  • ScreenSpot Evaluation: On ScreenSpot, OmniParser significantly improves the GPT-4V baseline across mobile, desktop, and web platforms, surpassing GUI-finetuned models by a large margin.Local semantics further improves performance, while the finetuned interactable-region detector adds 4.3% accuracy over raw Grounding DINO.
  • Mind2Web Evaluation: On Mind2Web, the finetuned interactable-region detector with local semantics performs better than the raw Grounding DINO variant in all task categories.The categories are Cross-Domain, Cross-Website, and Cross-Tasks, evaluated using parsed screen results and action history as text prompt.
  • Mind2Web Evaluation: Without parsed HTML, OmniParser outperforms GPT-4 performance using HTML in every Mind2Web sub-category and also surpasses GPT-4V with Set-of-Marks prompting.Against GPT-4V with textual choices, it gains 4.1% in Cross-Website and 5.2% in Cross-Domain but falls 0.8% in Cross-Task.
  • AITW Evaluation: 4.7%: OmniParser improves the overall AITW score over the best GPT-4V + history baseline, with gains across most sub-categories.The comparison replaces IconNet with the finetuned interactable-region detector and adds local icon-functionality semantics.

5 Discussions

OmniParser’s common failure cases arise when repeated elements are ambiguous, text boxes are too coarse, or icon meanings depend on page context. The authors identify finer-grained descriptions, integrated OCR and interactable-region detection, and context-aware icon description as potential remedies.

  • Repeated Icons/Texts: Repeated icons or texts can cause GPT-4V to select the wrong element when a task requires distinguishing among visually identical buttons.The proposed remedy is to add finer-grained descriptions so GPT-4V accounts for repeated elements during action prediction.
  • Coarse Prediction of Bounding Boxes: Coarse OCR boxes can place the predicted click outside the target when the box center does not fall within the clickable text region.The authors attribute this to OCR lacking knowledge of which text regions are hyperlinks or clickable.
  • Coarse Prediction of Bounding Boxes: Combining OCR with interactable-region detection is proposed to improve detection of clickable text and hyperlinks.
  • Icon Misinterpretation: Icons with similar shapes may receive incorrect functional descriptions when their meanings depend on the surrounding interface context.The authors propose training an icon-description model that can access the full image context rather than only cropped icons.

6 Conclusion

The conclusion presents OmniParser as a vision-only system that parses UI screenshots into structured elements using finetuned detection and functional-description models. Across ScreenSpot, Mind2Web, and AITW, it improves GPT-4V performance while using screenshot input without extra HTML or Android view-hierarchy information.

  • OmniParser parses UI screenshots into structured elements through finetuned icon detection and functional-description models.
  • OmniParser improves GPT-4V on ScreenSpot and outperforms GPT-4V baselines using HTML information on Mind2Web and a specialized Android icon detector on AITW.
  • The approach is intended to parse screens across PC and mobile platforms without dependence on HTML or Android view hierarchy information.

7 Appendix

The appendix documents the datasets, finetuning procedures, GPT-4V prompts, evaluation setup, baseline details, local-semantics examples, and representative failure cases used to analyze OmniParser.

  • Details of Icon-Description Dataset: Finetuning BLIP-2 for one epoch produced examples comparing the original and finetuned icon-description models.
  • Training Details: The interactable icon detector is a YOLOv8 model trained on 66,990 samples, split into 63,641 training and 3,349 validation examples.Training used 20 epochs, batch size 256, learning rate 1e−3, and four GPUs.
  • Prompt Used for GPT-4V: GPT-4V is prompted either with labeled bounding boxes alone or with bounding boxes plus local icon and text descriptions.
  • Mind2Web Evaluation: Mind2Web baseline comparisons include models using finetuned UI data, HTML information, selected HTML elements, or textual-choice prompts.
  • Local Semantics: Local semantics help GPT-4V make better action predictions by adding OCR text and functional descriptions of icon bounding boxes.
  • Failure Cases: Failure cases include confusion among repeated buttons, missed bold text by OCR, and context-insensitive interpretation of a three-dots icon.
Loading 2408.00203v1…