Source-linked AI summary
Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, Caiming Xiong
TL;DR
GUI grounding benchmarks and datasets often omit the software knowledge, layout reasoning, and precise manipulation required by real computer-use tasks. The paper introduces OSWORLD-G and the 4-million-example JEDI dataset, then trains models and evaluates their grounding and agentic performance. Results show stronger grounding across benchmarks, improved agent performance, and compositional generalization from specialized interface-element data.
Problem
Existing GUI grounding benchmarks oversimplify instructions as short referring expressions and underrepresent software commonsense, layout understanding, and fine-grained manipulation.
Method
The paper creates OSWORLD-G with 564 annotated samples and synthesizes JEDI through multi-perspective task decoupling, then trains multi-scale grounding models.
Results
The models outperform existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G, while improved grounding enhances agentic capabilities on complex computer tasks.
Takeaways & Limitations
Combining specialized data for different interface elements supports compositional generalization to novel interfaces and improves grounding-oriented agent performance.
Takeaways & Limitations
Refusal modeling remains challenging because models show limited improvement amid pretraining limitations and VLM hallucination.
Abstract
from arXiv · showhide
Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks, improving from 5% to 27% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https://osworld-grounding.github.io.
1 Introduction
GUI grounding must map natural-language instructions to executable GUI actions while handling software commonsense, layout, fine-grained manipulation, and infeasible instructions. The paper addresses benchmark and data limitations with OSWORLD-G and JEDI, and reports stronger grounding and agentic performance.
- Motivation: Practical GUI grounding requires software commonsense, layout understanding, fine-grained manipulation, and rejection of infeasible instructions.Existing benchmarks often reduce grounding to short referring expressions that do not capture these requirements.
- Contributions: OSWORLD-G contains 564 finely annotated samples spanning text matching, element recognition, layout understanding, fine-grained manipulation, and infeasibility.Annotations also identify the interface-element types needed for each task.
- Contributions: JEDI is a 4-million-example open grounding dataset created through multi-perspective task decoupling for web and desktop domains.The dataset is designed to address the complexity gaps in existing GUI grounding data.
- Results: Evaluations on ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G show that the approach significantly outperforms existing models in grounding ability.The paper also evaluates whether improved grounding transfers to complex agentic tasks.
- Results: Combining specialized data for different interface elements enables compositional generalization to novel interfaces.Case studies support the paper’s decomposition hypothesis.
2 Approach
The approach defines visual GUI grounding as mapping step-specific instructions and observations to executable coordinate-based actions, then constructs OSWORLD-G and JEDI to cover diverse grounding capabilities. Its data pipeline combines synthetic component rendering with real-world augmentation and evaluates spatially grounded actions across multiple task types.
- Task Definition: A multimodal agent maps visual observations, instructions, and optional state history to actions without access to GUI code or APIs.Actions include types such as click, move_to, and type, with point or bounding-box parameters.
- Task Definition: GUI grounding maps a step-specific instruction and current visual observation to a specific executable action for the agent policy.Accurate grounding at each step is treated as fundamental to overall task success.
- Benchmark Construction: OSWORLD-G samples OSWorld screenshots at 720p and 1080p, annotates instructions with bounding boxes, and scores predictions by spatial containment.The benchmark organizes tasks by text matching, element recognition, layout understanding, fine-grained manipulation, and refusal handling.
- Layout Understanding: Layout understanding requires reasoning over hierarchical screen structure to distinguish visually similar elements across modules.Examples include locating a top notification bar among interface regions.
- Fine-grained Manipulation: Fine-grained manipulation requires high-precision actions in small regions, including character-level cursor placement and interaction with compact controls.The task scope includes text editing, sliders, steppers, and table cells.
- Refusal Handling: OSWORLD-G includes 54 infeasible scenarios that test whether systems reject impossible instructions rather than execute erroneous actions.These cases target safer and more robust interaction under hallucinated or incorrect instructions.
- JEDI Data Construction: JEDI contains 4 million synthesized examples built from screenshot–metadata pairs transformed into VLM queries and answers.Metadata includes filenames, rendering code, and element bounding boxes.
- JEDI Data Construction: The synthesis pipeline selects production UI components, uses LLMs to create functional cases, renders them in React, and extracts structural metadata.Real-world screenshots and structured extraction supplement behaviors underrepresented in code-based libraries.
3 Experiments
The experiments evaluate GUI grounding across four benchmarks and examine whether grounding improvements transfer to agentic computer-use tasks. Fine-tuning open-source models on JEDI achieves state-of-the-art grounding with small models and supports strong performance by general foundation-model agents.
- Grounding Ability: The study evaluates grounding on ScreenSpot-v2, ScreenSpot-Pro, UI-Vision, and OSWORLD-G, which cover distinct grounding settings and capability dimensions.OSWORLD-G specifically evaluates fine-grained and functional components.
- Grounding Ability: OSWORLD-G accuracy is highest for text matching, lower for element recognition and layout understanding, and lowest for fine-grained manipulation.The benchmark exposes uneven performance across grounding capabilities.
- Grounding Ability: Refusal predictions are consistently absent in nearly all evaluated models, including computer-use-specialized models, except Gemini-2.5-Pro.This remains true despite including refusal data during training.
- Agentic Ability: Using JEDI as the grounding component enables non-specialized foundation models to achieve state-of-the-art agent performance, surpassing prior 72B-scale grounding approaches and matching specialized models.The evaluation covers complex tasks in OSWorld and WindowsAgentArena.
4 Analysis
The analysis studies instruction refinement, data scaling, and qualitative grounding behavior. Results indicate that clearer instructions and JEDI training improve grounding, while specialized data supports fine-grained and cross-interface understanding.
- Effectiveness of Knowledge: Instruction refinement generally improves model performance, and JEDI reaches performance comparable to UI-TARS-72B after refinement.The authors interpret this as evidence that precise expressions or relevant interaction experience can enhance grounding.
- Data Scaling: The data-scaling analysis trains on 10%, 20%, 50%, and 100% proportions of icon, component, and layout data for equal numbers of steps.The setup is designed to compare final performance under equal computational resources.
- Case Study: JEDI demonstrates fine-grained grounding by locating targets from timestamps, table headers, paragraph-relative positions, and desktop layouts.These cases include targets without explicit coordinates or location identifiers.
- Case Study: JEDI associates a counter-clockwise arrow icon with the function “rotate,” illustrating icon-function understanding learned from extensive icon data.The comparison contrasts JEDI’s correct grounding with Qwen’s erroneous click.
5 Related Work
Related work situates GUI grounding within digital-agent research and identifies a shift toward visual grounding. It also emphasizes that existing data and evaluation paradigms oversimplify real-world instructions and interactions.
- Digital Agents: Digital-agent research spans mobile, web, and real-world computer interaction, alongside advances in visual encoding and reinforcement-learning approaches.The paper places GUI grounding within the broader development of multimodal digital agents.
- GUI Grounding: Recent GUI-grounding approaches increasingly use pure visual solutions instead of textual HTML or accessibility information.The paper presents this shift as part of the field’s response to real-world interaction requirements.
- GUI Grounding: Existing grounding data and evaluation paradigms oversimplify natural-language instructions through screenshot-text pairings or manual annotations.This limitation motivates evaluating richer grounding capabilities.
6 Conclusion
The paper concludes that addressing overlooked GUI-grounding challenges requires both richer evaluation and large-scale data. Its models improve grounding benchmarks and boost agent performance in online computer-use environments.
- Conclusion: OSWORLD-G introduces 564 annotated samples covering overlooked challenges including fine-grained manipulation and layout understanding.The benchmark is designed for evaluating these capabilities.
- Conclusion: JEDI contains 4 million synthesized examples and supports competitive results on ScreenSpot-v2, ScreenSpot-Pro, and OSWORLD-G.The same approach also boosts agent performance in OSWorld and WindowsAgentArena.
- Conclusion: The results support addressing previously identified gaps in GUI-grounding research through benchmark and dataset design.The conclusion links these design choices to improvements in grounding and agent performance.
7 Limitations
The work primarily focuses on data synthesis methods and identifying factors that affect grounding. It leaves broader screen-capture scaling and deeper refusal modeling for future work.
- Resource restrictions limit further scaling through screen-capture data extracted from internet images and videos.The authors leave this expansion to future industrial efforts.
- Refusal modeling remains difficult because models show limited improvement amid pretraining limitations and VLM hallucination.The authors identify infeasible-action rejection as an important direction for future research.
- Enhanced grounding models may support human-like digital-world traversers that collect interaction data for further improvement.
A.1.1 Data Types
OSWORLD-G organizes examples into five categories requiring different grounding capabilities, with corresponding interface element types documented in Table 7.
- OSWORLD-G examples are categorized into five groups according to the grounding capabilities they require.The categories are linked to corresponding interface element types in Table 7.
A.1.2 Comparison with Previous Work
The paper presents a comparison between OSWORLD-G and previous benchmarks, with the comparison reported in Table 8.
- The comparison between OSWORLD-G and previous work is presented in Table 8.
A.1.3 Data Examples
OSWORLD-G examples cover hierarchical layout understanding, character-level manipulation, text matching, visual element recognition, and infeasible instructions. The examples illustrate distinct grounding demands across these task types.
- Layout Understanding: Layout understanding requires recognizing hierarchical interface structure, such as a notification bar positioned above the editing area.
- Fine-grained Manipulation: Fine-grained manipulation requires precise actions in small regions, including selecting the position between a word and number at character-level granularity.
- Text Matching: Text matching grounds actions by locating explicit instruction text, such as matching “As Attachment” in the screenshot.
- Element Recognition: Element recognition identifies visual patterns such as icons or images, illustrated by clicking an ellipse icon.
- Refusal Instruction: Refusal tasks test whether models avoid impossible actions when the instruction names an element absent from the screen.The example refers to “Cindy Williams,” whose email address is not visible.
A.1.4 Annotation Details
OSWORLD-G uses systematic failure-case collection and expert annotation, while JEDI combines diverse synthesized and in-house data sources for computer-use grounding. The resulting examples cover icons, components, layouts, refusals, and fine-grained operations, with case studies illustrating the targeted capabilities and remaining failures.
- OSWORLD-G annotation: OSWORLD-G annotations begin by collecting grounding failures from state-of-the-art model trajectories and categorizing them by required capability.
- OSWORLD-G annotation: Expert annotators then craft unambiguous low-level instructions that map uniquely to specific screen targets.
- JEDI data composition: The dataset statistics define '# Line' as compressed multiple query-answer pairs used to improve training efficiency.
- JEDI data composition: JEDI combines icon grounding, component manipulation, layout captioning and grounding, refusal data, and in-house annotated or augmented data.
- Data synthesis: The synthesis methodology uses resource and action pools spanning office files and common operations such as spreadsheet cell selection and autofill.
- Case studies: Case studies show JEDI using functional icon knowledge, layout context, clickable-link recognition, and precise text matching, while failures remain for strict action wording, color distinctions, misleading instructions, and artistic fonts.