PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis

Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, Caiming Xiong

arXiv:2505.13227v3cs.AIcs.CLcs.CVcs.HC

TL;DR

GUI-grounding benchmarks often simplify real interactions, although practical tasks require software context, layout understanding, and fine-grained manipulation. This paper introduces a broader benchmark and synthesized dataset, and reports stronger grounding results alongside improved agent performance.

  • Problem

    Existing GUI-grounding benchmarks simplify tasks as short referring expressions, while practical grounding also requires software context, layout understanding, and fine-grained manipulation.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces OSWORLD-G, a 564-sample benchmark, and a 4-million-example grounding dataset, then trains multi-scale models on that dataset.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    The models outperform existing approaches across ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G, while also boosting agent performance on OSWorld and WindowsAgentArena.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Combining specialized data for different interface elements enables compositional generalization to novel interfaces.

    Source: §1 · p. 2

    Refusal modeling remains a significant challenge, with limited improvement attributed to pretraining limitations and VLM hallucination.

    Source: §7 · p. 11

Abstract

from arXiv · show

Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks, improving from 5% to 27% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https://osworld-grounding.github.io.

1 Introduction

Practical GUI grounding must handle software context, layout, fine-grained manipulation, and infeasible instructions, beyond short referring expressions. The paper introduces OSWORLD-G and JEDI and reports stronger grounding and agentic capabilities.

1 Introduction

  • GUI grounding maps natural-language instructions to specific on-screen actions and is a cornerstone for computer-use agents across devices and interaction settings.It can serve as an isolated human-machine interaction component, facilitate multimodal collaboration agents, or enhance end-to-end models.Source: §1 · p. 1
  • Practical grounding requires software commonsense, layout understanding, fine-grained component manipulation, and rejection of infeasible instructions.Examples include interpreting icons and component functions, understanding elements under panels, adjusting sliders or selecting text character by character, and avoiding mistaken application instructions.Source: §1 · p. 1
  • Earlier benchmarks simplify grounding to short referring expressions, while existing data sources trade coarse webpage understanding against costly manual annotation.ScreenSpot-v2 [10] [45] approaches ~90% accuracy with recent methods [32]; webpage correspondences capture coarse element signals but lack fine-grained UI operations, while manual annotation struggles to scale.Source: §1 · pp. 1–2
  • OSWORLD-G covers 564 finely annotated samples across text matching, element recognition, layout understanding, fine-grained manipulation, and infeasibility; JEDI contains 4 million grounding examples.OSWORLD-G annotations include the element types required to solve each task, and JEDI is synthesized for web and desktop domains through multi-perspective task decoupling.Source: §1 · p. 2, Figure 1 · p. 2
  • The approach significantly outperforms existing models on ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G, while improved grounding enhances agentic capabilities on OSWorld and WindowsAgentArena.Ablations identify key factors contributing to grounding performance, and case studies show that combining specialized data for different interface elements enables compositional generalization to novel interfaces.Source: §1 · p. 2

2 Approach

The approach defines GUI grounding as mapping an instruction and visual observation to executable actions without access to interface code or APIs. It introduces OSWORLD-G and constructs JEDI from screenshots and metadata to train visual-language models for grounding.

2.1.1 Benchmark Construction

  • OSWORLD-G contains 564 samples across 32 UI types, with paraphrased instructions, verified bounding boxes, and accuracy evaluated by spatial containment.Annotators familiar with the software tested annotations in the real applications and conducted multiple verification rounds; annotation took about 0.5 human-hours per sample on average.Source: §2.1.1 · p. 3

2.1.2 Data Types

  • OSWORLD-G categorizes grounding into text matching, element recognition, layout understanding, fine-grained manipulation, and infeasible-task refusal.These cover matching explicit text or recognizing icons and their functions, locating elements within GUI modules, precise actions on text or compact controls, and rejecting impossible instructions; the infeasible subset has 54 examples.Source: §2.1.2 · pp. 3–4

2.2 JEDI Data Construction

  • JEDI contains 4 million newly synthesized computer-use grounding examples built from screenshot–metadata pairs transformed into VLM training queries and answers.The metadata can include filenames, rendering code, and element bounding boxes.Source: §2.2 · p. 5

2.2.1 Icon

  • JEDI gathers icon assets from open-source repositories, production websites, and desktop software to train grounding from descriptions of icon appearance and function.Website crawling also captures metadata such as class names and aria labels; software icons are extracted from executables, DLLs, and system libraries across Windows, macOS, and Ubuntu.Source: §2.2.1 · p. 5

2.2.2 Component

  • JEDI synthesizes component screenshots by rendering LLM-generated functional cases in React from production UI-library examples, then supplements them with real-world screenshots and extracted metadata.The real-world augmentation targets interactive behaviors underrepresented in code libraries, including scrolling, spreadsheet-cell clicks, and resizing text boxes.Source: §2.2.2 · p. 5

2.2.3 Layout

  • JEDI builds layout data from Figma prototype designs with structured hierarchy and position metadata, and from OSWorld and WindowsAgentArena screenshots annotated by OmniParser-v2.The prototype exports preserve designer-provided bounding boxes, component types, functional descriptions, and hierarchical relationships.Source: §2.2.3 · pp. 5–6

2.2.4 Data Processing

  • JEDI converts screenshots and metadata into grounding and description formats, using GPT-4o and Claude to enrich appearance and functionality annotations.Grounding examples ask for actions or bounding boxes; description examples ask for descriptions of boxed elements, and multiple query-answer pairs can be compressed into one conversation.Source: §2.2.4 · p. 6

2.2.5 Supplementary Training Data

  • Supplementary training data adds over 2.6 million mismatched instruction–screenshot refusal examples and filters noisy internet-synthesized datasets with UI-TARS-72B.The authors manually inspected a subset of refusal examples and retained matching labeled and predicted examples from sources including SeeClick and OS-Atlas, following Aguvis [48].Source: §2.2.5 · p. 6

3 Experiments

Experiments evaluate JEDI on GUI-grounding benchmarks and online computer-use tasks, finding strong grounding results and gains in agentic performance. They also examine capability differences across grounding task types and refusal behavior.

3.1 Grounding Ability

  • Fine-tuning open-source models on JEDI achieves state-of-the-art performance on GUI-grounding benchmarks, surpassing dedicated computer-use models Operator and UI-TARS despite smaller model size.The evaluations cover ScreenSpot-v2, ScreenSpot-Pro, UI-Vision, and OSWORLD-G.Source: §3.1 · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • On OSWORLD-G, models perform best on text matching, less well on element recognition and layout understanding, and worst on fine-grained manipulation.Models rarely refuse instructions referring to absent screen elements despite refusal data in training; refusal predictions are consistently absent except for Gemini-2.5-Pro, especially among computer-use-specialized models.Source: §3.1 · p. 6

3.2 Agentic Ability

  • The online-task setup uses GPT-4o or o3 as high-level planners and JEDI to convert their next-step natural-language instructions into concrete actions, without specialized agent architecture or model scheduling.The evaluation uses OSWorld and WindowsAgentArena.Source: §3.2 · p. 8
  • A simple agent using JEDI for grounding achieves state-of-the-art performance with general foundation-model planners, surpassing prior approaches using 72B-scale grounding models and matching specialized models.Performance also improves with deployment scale, following a trend similar to Operator.Source: §3.2 · p. 8

4 Analysis

Analysis shows that refined instructions improve grounding, increased mixed-data scale yields continued and more stable gains, and JEDI demonstrates fine-grained GUI understanding across representative cases.

4.1 Effectiveness of Knowledge

  • After instruction refinement, JEDI achieves performance comparable to UI-TARS-72B, while models generally improve with more precise instructions.Instructions were rewritten to rely on universal visual cues such as color and shape, reducing required background knowledge.Source: Figure 5 · p. 8, §4.1 · p. 8

4.2 Performance as Data Scaling

  • As training data scale increases, performance continues to improve without saturation; mixed data types yield more stable gains than scaling a single type.Models were trained for the same number of steps at each data proportion to compare final performance under equal computational resources.Source: Figure 6 · p. 9, §4.2 · p. 9

4.3 Case Study

  • In a detailed OSWORLD-G comparison, selected representative cases where results differed show JEDI-7B’s enhanced fine-grained operation and GUI understanding relative to Qwen2.5-VL-7B-Instruct across four cases.Examples include finding a table cell using timestamp and header information, locating a blank cell from a positional description, and matching a counter-clockwise arrow icon to its “rotate” function.Source: §4.3 · pp. 9–10, Figure 7 · p. 9

5 Related Work

Related work traces digital agents’ shift from mobile and web environments to real-world computer interaction, while identifying grounding precision and oversimplified evaluation as persistent gaps. The paper positions its benchmark and synthetic data as ways to address those gaps.

Digital Agents

  • Digital-agent research spans mobile and web environments, real-world computer interaction, GUI encoding, reinforcement learning, agentic frameworks, and joint visual-language modeling.The literature also distinguishes digital from physical agents.Source: §5 · p. 10
  • Homogeneous synthetic training data limits grounding precision and fine-grained operations, while potentially useful data sources remain underexplored.Examples of unsupported fine-grained operations include slider adjustments and nested menu selections.Source: §5 · p. 10

GUI Grounding

  • GUI grounding approaches have shifted from HTML or accessibility information to visual solutions, but data and evaluation remain oversimplified.They often miss expressed intent, screen-level comprehension such as active-window identification, and operations such as sliders and drag-and-drop; the paper proposes a benchmark and synthetic data to address these gaps.Source: §5 · p. 10

6 Conclusion

The conclusion presents OSWORLD-G and a 4-million-example dataset as responses to overlooked GUI grounding challenges, and reports competitive benchmark results and improved agent performance.

6 Conclusion

  • The models achieve competitive results on ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G, while boosting agent performance in OSWorld and WindowsAgentArena.The passage gives no numerical results for these comparisons.Source: §6 · p. 10
  • OSWORLD-G evaluates overlooked GUI grounding challenges, including fine-grained manipulation and layout understanding, with 564 annotated samples.Source: §6 · p. 10
  • The authors set up multiple pipelines to construct a 4-million-example dataset addressing these GUI grounding challenges.Source: §6 · p. 10

7 Limitations

The work focuses on data synthesis and essential factors, while leaving several scaling and capability directions for future research. These include extracting screenshot data from internet images and videos, improving refusal modeling, and collecting interaction data with human-like traversers.

7 Limitations

  • The authors defer extracting screen captures from internet images and videos with neural networks, which could expand screenshot metadata and grounding data, because of resource restrictions.They leave this expansion to future industrial efforts.Source: §7 · p. 11
  • Refusal modeling remains challenging: models improve only limitedly because of pretraining limitations and hallucinations in vision-language models.Rejecting infeasible actions is important for preventing errors and mitigating risks from incorrect instructions.Source: §7 · p. 11
  • The authors leave human-like digital traversers for future work, proposing they could collect interaction data to improve grounding and model knowledge.Such traversers could interact with or without specific purposes.Source: §7 · p. 11

A Appendix · A.1 OSWORLD-G Statistics · A.1.1 Data Types

OSWORLD-G groups examples into five categories by grounding capability, with each category associated with corresponding interface element types in Table 7.

A.1.1 Data Types

  • OSWORLD-G classifies examples into five grounding-capability categories and maps them to corresponding interface element types in Table 7.Table 7 presents the distribution of benchmark examples across these categories.Source: §A.1.1 · p. 16, Table 7 · p. 16

A.1.2 Comparison with Previous Work

This section compares OSWorld-G with previous benchmarks, presenting the comparison in Table 8.

A.1.2 Comparison with Previous Work

A.1.3 Data Examples

The examples show how OSWorld-G tasks require more than locating controls: models must interpret layout, act at character-level precision, match text or visual patterns, and refuse infeasible actions.

Layout Understanding

  • Closing Calc’s top notification bar requires recognizing that notification bars typically occupy the top of the editing area.The layout-understanding example tests comprehension of the interface’s hierarchical structure.Source: §A.1.3 · p. 16

Fine-grained Manipulation

  • Selecting the position between “person” and “1” requires character-level precision in a small or tightly packed screen region.The fine-grained manipulation example requires an action at character-level granularity.Source: §A.1.3 · p. 16

Text Matching

  • Choosing “As Attachment” requires locating and matching that explicit textual cue in the screenshot.This text-matching example appears in Figure 8a.Source: §A.1.3 · p. 16, Figure 8 · p. 17

Element Recognition

  • Clicking the ellipse icon requires visually recognizing its shape in the interface.This element-recognition example appears in Figure 8b.Source: §A.1.3 · p. 16, Figure 8 · p. 17

Refusal Instruction

  • When “Cindy Williams” is not visible on screen, the expected response is to refrain from clicking an email address.The refusal-instruction example tests recognition that the requested action is infeasible.Source: §A.1.3 · pp. 16–17

A.1.4 Annotation Details … A.6 Agentic Benchmark Results of JEDI

This section documents OSWORLD-G annotation and JEDI data preparation, including instruction filtering and complementary grounding-data pipelines for office software, icons, and interface components. It also presents JEDI’s layout-data formats, qualitative grounding examples and failure modes, and detailed agentic benchmark results.

A.1.4 Annotation Details

  • OSWORLD-G annotations draw on grounding failures, expert-written low-level instructions, precise bounding boxes, and multi-round quality verification.Failure cases from state-of-the-art model trajectories guided instructions designed to map unambiguously to screen actions; strong-model predictions helped resolve annotation inconsistencies.Source: §A.1.4 · p. 17

A.2.1 Overview

  • JEDI improves on AGUVIS by filtering low-quality or irrelevant samples, adding OS-Atlas data, and quality-checking synthetic data with UI-TARS 4.In-house examples are human-recorded computer use, timestamps, and screenshots, later used with models such as GPT to construct instructions.Source: §A.2.1 · p. 17

A.2.5 Cost Analysis

  • Approximately $36,000 was spent on GPT-4o across JEDI data-generation tasks.The estimate covers icon, component, and layout data generation.Source: §A.2.5 · p. 31

A.3.1 Component Collection and Style Augmentation

  • The component pipeline collects examples from four UI libraries, then uses GPT-4o and Claude-3.5-Sonnet to create contextual style variants while avoiding redundancy.Augmented components are rendered in React at randomized positions; Playwright captures screenshots and element-tree information for action generation.Source: §A.3.1 · p. 32, §A.3.2 · p. 32

Step 1: Generate Action Intents

Pipeline 2: Element-Level Action Generation

  • Pipeline 2 filters DOM elements, encodes each with cropped and contextual views plus element metadata, and generates unique or continuous actions before grounding samples.Visibility and atomicity checks help ensure the target is a single visible UI unit.Source: Pipeline 2: Element-Level Action… · pp. 33–34

A.3.3 Comparison of Pipelines

  • Pipeline 2 provides better localization and action diversity than Pipeline 1, and generates most of the dataset.Pipeline 1 is simpler but can have inaccurate bounding boxes, limited action diversity, and vague actions.Source: §A.3.3 · p. 34

A.3.4 Post-Processing and Filtering

  • The visual filter rejects cases showing visible errors, targeting an incorrect GUI element, or having incorrect click localization; GPT-4o-mini filters ambiguous or low-quality instructions from Pipeline 1.Source: §1 · p. 34, §2 · p. 34

2. Instruction Filter (LLM-Based)

  • The LLM-based instruction filter includes non-visual identifiers such as “index 1” and multi-step or compound interactions among its targeted cases.Source: §2 · p. 34

3. Instruction Filter (Rule-Based)

  • Rule-based filtering removes instructions with raw coordinates or structural terms, ambiguous card targets, annotation references, and potentially misaligned directional screen descriptions.The card rule requires spatial qualifiers; references to highlights or markers may mistake annotations for interface elements.Source: §3 · pp. 34–35

A.3.5 Real-world augmentation pipeline

  • Office augmentation pairs web-sourced Word, Excel, and PowerPoint files with common actions, then extracts component coordinates from page structure to synthesize precise interactions.Examples include locating a document scrollbar, clicking the center of cell B3, auto-filling from cell A1, and selecting column D.Source: §A.3.5 · p. 35

Icon description data

  • Icon-description examples map a single icon image to natural-language descriptions of its appearance and function, illustrated with a commuter-rail access icon.The example describes a white train outline and possible uses such as schedules, tickets, or station information.Source: Icon description data · pp. 35–37

Icon grounding data

  • Icon-grounding examples pair an image containing multiple icons with a target-icon description and produce a click-style grounding action in Qwen2.5-VL tool-use format.The example description identifies a density or line-spacing setting.Source: Icon grounding data · p. 37

Component data from Code-and-rendering pipeline

Component data for real-world augmentation

  • Real-world component data uses screenshots from existing websites or applications with instructions to generate grounding actions, illustrated by document, slide, and spreadsheet tasks.Examples include clicking between characters in a document, selecting a title-box handle, and navigating to cell C19.Source: Component data for real-world… · p. 40

Layout description data

  • Layout descriptions combine visual composition, spatial context, interaction, and element type; the example describes a product-image carousel and its navigation.Users can select thumbnails to update the main image or use arrows to scroll through additional images.Source: Layout description data · pp. 44–45

Layout grounding data

  • Layout grounding maps a screenshot and an element description to the element’s bounding box, illustrated with a Microsoft sign-in input for phone, email, or Skype ID.The example describes clicking or tapping the box and entering credentials.Source: Layout grounding data · p. 45

A.5.1 Cases of improvement

  • Across six cases, JEDI handles icon functions, layout cues, subpages, mathematical symbols, and mode-switching better than the compared models or untrained base model.Examples contrast JEDI with Qwen or traditional models, and show that mathematical skill alone does not ensure GUI grounding without fine-tuning on decomposed GUI data.Source: §A.5.1 · p. 46

A.5.2 Cases of failure

  • JEDI’s failures include choosing a functionally correct but unrequested control, confusing active and inactive text, misreading instructions, and missing artistic lettering.The paper identifies instruction wording as a possible source of confusion in the cookie-options case and lack of art and design data as a possible source of difficulty with stylized text.Source: §A.5.2 · p. 48

A.6 Agentic Benchmark Results of JEDI

Related papers

Loading 2505.13227v3…