Source-linked AI summary
ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, Tat-Seng Chua
TL;DR
Professional GUI grounding is under-explored despite the complexity, high resolution, and small targets found in specialized applications. The paper introduces ScreenSpot-Pro and ScreenSeekeR, a planner-guided visual search method, achieving 48.1% versus 18.9% for OS-Atlas-7B without additional training.
Problem
Professional applications remain under-explored, while their complex high-resolution interfaces contain smaller targets that make precise GUI grounding difficult.
Method
The paper introduces ScreenSpot-Pro and ScreenSeekeR, which uses a strong planner to progressively narrow screenshot search areas for grounding.
Results
48.1%: ScreenSeekeR improves OS-Atlas-7B from 18.9% to 48.1% without additional training.
Takeaways & Limitations
The benchmark exposes a gap in GUI grounding for specialized applications, while search-space refinement provides a foundation for improving professional GUI agents.
Takeaways & Limitations
ScreenSpot-Pro evaluates GUI grounding but excludes agent planning and execution tasks because of software licensing restrictions.
Abstract
from arXiv · showhide
Recent advancements in Multi-modal Large Language Models (MLLMs) have led to significant progress in developing GUI agents for general tasks such as web browsing and mobile phone use. However, their application in professional domains remains under-explored. These specialized workflows introduce unique challenges for GUI perception models, including high-resolution displays, smaller target sizes, and complex environments. In this paper, we introduce ScreenSpot-Pro, a new benchmark designed to rigorously evaluate the grounding capabilities of MLLMs in high-resolution professional settings. The benchmark comprises authentic high-resolution images from a variety of professional domains with expert annotations. It spans 23 applications across five industries and three operating systems. Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%. Our experiments reveal that strategically reducing the search area enhances accuracy. Based on this insight, we propose ScreenSeekeR, a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-of-the-art performance with 48.1% without any additional training. We hope that our benchmark and findings will advance the development of GUI agents for professional applications. Code, data and leaderboard can be found at https://gui-agent.github.io/grounding-leaderboard.
1 Introduction
ScreenSpot-Pro targets the under-explored problem of grounding instructions precisely in complex, high-resolution professional interfaces. It introduces an authentic benchmark and ScreenSeekeR, which narrows visual search areas to improve grounding performance.
- Professional GUI grounding remains under-explored because specialized applications are more complex and demand high-resolution interfaces.
- Higher resolutions create smaller relative targets, while current grounding models generally perform worse as target size decreases.
- ScreenSpot-Pro contains authentic high-resolution professional tasks collected across diverse applications and operating systems, with expert annotations.
- ScreenSeekeR uses a strong planner’s GUI knowledge to recursively narrow candidate regions before applying grounding models.
- ScreenSeekeR raises OS-Atlas-7B performance from 18.9% to 48.1% without additional training.
2 Related Works
Prior GUI grounding benchmarks mainly evaluate simplified web, mobile, or cropped-screen tasks, leaving professional high-resolution settings insufficiently represented. Existing high-resolution processing methods also rely on rigid splitting or face model limitations at ultra-high resolutions.
- Previous benchmarks primarily focus on simple web browsing and mobile interaction tasks, oversimplifying GUI grounding.
- ScreenSpot uses cropped regions and VisualWebBench provides candidate targets, while professional productivity tools remain underrepresented.
- Resolution scaling and simple cropping struggle at ultra-high resolutions because of short context lengths and low-resolution training data.
- Predefined image-splitting strategies constrain search flexibility and may remove contextual information in GUI environments.
3 ScreenSpot-Pro: Benchmarking GUI Grounding for Professional High-Resolution Computer Use
ScreenSpot-Pro collects realistic GUI grounding tasks from professional and everyday software using high-resolution screenshots and expert annotation procedures. Its broad application coverage and very small target regions support rigorous evaluation of professional GUI grounding.
- Collection range: ScreenSpot-Pro covers six application genres, including development, creative, CAD, scientific, office, and operating-system software.
- Collection method: Experts with at least five years of relevant application experience recorded realistic tasks during their regular workflows.
- Collection method: The capture tool let experts annotate bounding boxes and instructions in real time, improving annotation consistency and quality.
- Image criteria: Screens prioritized resolutions above 1080p, disabled monitor scaling, and included both displays in dual-monitor setups.
- Quality control: Each instance receives multiple reviews, while target boxes are verified to tightly match interactable GUI regions.
- Dataset statistics: Targets average 0.07% of screenshot area in ScreenSpot-Pro, compared with 2.01% in ScreenSpot.
4 Methods
The methods reduce the search space in high-resolution GUI screenshots through iterative cropping, planner-guided area selection, scoring, and recursive search.
- Planner-free methods: Iterative Zooming repeatedly grounds the full screenshot, splits it into four patches, and continues searching in the selected patch.The splitting strategy uses 2 rows × 2 columns at every step.
- Planner-free methods: Iterative Narrowing follows the same ground-and-zoom procedure but crops patches around the prediction, using half the image width and height.This centers the next search region on the current prediction.
- Planner-free methods: ReGround crops the region surrounding an initial prediction and re-grounds it to make the final determination.The crop size can be configured according to the model’s optimal input size.
- ScreenSeekeR: ScreenSeekeR uses GPT-4o’s GUI knowledge to propose likely search areas, score candidates with a grounder, and recursively narrow the search.It exploits the hierarchical organization of menus, tools, sub-panels, and child windows in application interfaces.
- ScreenSeekeR: Candidate areas are ranked by centrality-weighted voting-box scores, deduplicated with nonmaximum suppression, and searched recursively until verification or maximum depth.Box dilation expands small grounded boxes, while higher-scoring overlapping candidates are retained.
5 Experiments
Experiments show that professional high-resolution grounding is difficult for existing models, while search-space reduction substantially improves performance, especially with ScreenSeekeR.
- End-to-end models: 18.9% accuracy was the best end-to-end result on ScreenSpot-Pro, achieved by OS-Atlas-7B; no other model exceeded 10%.GPT-4o achieved 0.9% direct grounding accuracy despite its broader capabilities.
- End-to-end models: Icon targets were substantially harder to ground than text targets across the benchmarked models.Professional applications compound this difficulty through numerous functions, assumed icon familiarity, and context-specific icon meanings.
- Planner-free methods: 40.2% accuracy made ReGround the best planner-free method with OS-Atlas-7B.Iterative Narrowing slightly outperformed Iterative Focusing, likely because its splitting strategy handles targets near the center axes better.
- Planner-free methods: 1024×1024 crops optimized OS-Atlas-7B, while 768×768 crops optimized UGround; performance declined when resolution moved away from these ranges.Smaller crops lose context, whereas larger images exceed model processing capacity.
- ScreenSeekeR: 48.1% accuracy raised OS-Atlas-7B from 18.9% with ScreenSeekeR, without additional training.GPT-4o supplied the planner despite scoring only 0.8% on direct grounding, and the framework produced explainable search trajectories.
- ScreenSeekeR: 41.9% accuracy followed removal of subsequent searches, 1.7% performance was lost without neighbor inference, and 46.8% resulted from majority-vote scoring.These ablations quantify the contributions of recursive search, neighbor inference, and patch scoring.
6 Conclusion
The paper introduces ScreenSpot-Pro to evaluate GUI grounding in complex professional environments and proposes ScreenSeekeR to improve accuracy by refining the search space without additional training.
- Conclusion: ScreenSpot-Pro exposes poor performance by current models on high-resolution professional GUI grounding and provides a benchmark for future progress.ScreenSeekeR improves accuracy through search-space refinement without additional training.
- Conclusion: The work targets the gap between MLLM-driven GUI agents and specialized professional applications.Its benchmark and method are presented as a foundation for advancing GUI agents in this domain.
Limitations
The benchmark evaluates GUI grounding only and excludes agent planning and execution tasks such as those in OSWorld.
- Scope: ScreenSpot-Pro is limited to GUI grounding, excluding agent planning and execution tasks.The paper attributes this design choice to legal risks associated with software licensing restrictions.
A ScreenSpot-Pro-CN
ScreenSpot-Pro-CN extends the benchmark with Chinese instructions to reflect bilingual professional workflows. Chinese instructions generally reduce grounding performance, though effects vary across models.
- Every benchmark task includes a Chinese instruction translated by GPT-4 and reviewed by bilingual authors.This design reflects professional scenarios in which users switch between native languages and English.
- 16.8% is OS-Atlas-7B’s performance with Chinese instructions, despite being the SOTA model in this comparison.
- 7.7% is UGround-7B’s performance with Chinese instructions, down from 16.4% in the corresponding comparison.
- GPT-4o and QwenVL-7B improved with Chinese instructions, but their overall scores remained low.
B More Details on the Collected Software
The collected software covers professional development, creative, engineering, scientific, analytical, and office workflows, alongside common operating-system tasks across three systems.
- Development and Programming: Development and programming applications cover coding, debugging, and testing, including VSCode, PyCharm, Android Studio, and Quartus.
- Creative Software: Creative applications support visual, audio, and video production, including Photoshop, Premiere, Illustrator, FruitLoops Studio, DaVinci Resolve, Unreal Engine, and Blender.
- Computer-Aided Design (CAD) and Engineering: CAD and engineering tools support precise 2D and 3D design, modeling, and mechanical simulation, including AutoCAD and SolidWorks.
- Scientific and Analytical: Scientific and analytical software supports data analysis, numerical computation, mathematical modeling, and simulation, including MATLAB and Origin.
- Office Software: Office applications support document creation, spreadsheets, communication, and presentations through Word, Excel, and PowerPoint.
- Operation System Commons: Operating System Commons evaluates basic operations such as file management and system utilities on Windows, macOS, and Linux.
D Data Examples
The data examples span professional applications and operating-system tasks, with instructions targeting text, icons, and interface controls at varied screen coordinates.
- Examples include visual-editing, audio-video, game-engine, and operating-system tasks such as “Blur Dissolve,” unlinking audio and video, and restarting from CD.
- Other examples target interface controls, including icons for changing coordinate mode and refreshing a file explorer.
- The examples include both text and icon bounding boxes, with coordinates recorded for each target.
- Figures 5 and 6 provide further examples of tasks in ScreenSpot-Pro.
- The benchmark includes text targets for tasks such as selecting a deb package, changing a model, and executing Python scripts.
- Additional instructions target showing comments and choosing a chord type, illustrating varied application-specific actions.
E Annotator Example
The annotation tool lets experts capture a screenshot, overlay it on the screen, draw a target bounding box, and enter the instruction directly.
- Experts annotate each task by dragging to label the target bounding box and entering the instruction in a popup dialog.When activated, the tool captures a screenshot and overlays it on the screen; the example marks “Open Folder” with a red box.