Source-linked AI summary
Covering Human Action Space for Computer Use: Data Synthesis and Benchmark
Miaosen Zhang, Xiaohan Zhao, Zhihong Tan, Zhou Huoshen, Yijia Fan, Yifan Yang, Kai Qiu, Bei Liu, Justin Wagle, Chenzhong Yin, Mingxi Cheng, Ji Li, Qi Dai, Chong Luo, Xu Yang, Xin Geng, Baining Guo
TL;DR
Computer-use agents remain unreliable on complex interactions, while existing benchmarks and datasets underrepresent these operations. The paper introduces CUActSpot and a scalable synthesis pipeline covering diverse modalities and actions, finding that task and modality diversity improves grounding and that Phi-Ground-Any-4B performs strongly on complex-interaction benchmarks.
Problem
Existing benchmarks focus mainly on single-click GUI-widget grounding, while computer-use agents often operate on tables, documents, charts, and images through dragging and drawing; large-scale datasets for these interactions are also lacking.
Method
The paper introduces CUActSpot across five modalities and diverse mouse actions, alongside a renderer-based pipeline that records screenshots and coordinates and synthesizes instructions and action traces.
Results
Increasing task and modality diversity is more effective than scaling a single modality, and Phi-Ground-Any-4B achieves strong performance on complex interaction benchmarks while remaining competitive among similarly scaled models.
Takeaways & Limitations
CUActSpot broadens computer-use evaluation beyond click-centric settings, while diverse training tasks improve corresponding and cross-modal capabilities and support limited compositional generalization.
Takeaways & Limitations
Benchmark performance can be dominated by software-specific knowledge rather than grounding ability, encouraging coverage of benchmarked applications over generalizable capabilities.
Abstract
from arXiv · showhide
Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git
1 Introduction
GUI-based computer-use agents offer flexible, cross-platform interaction, but their deployment is hindered by failures on complex operations and shortages of suitable benchmarks and training data. CUActSpot broadens evaluation across modalities and actions, while a scalable synthesis pipeline supports diverse training data and motivates variety scaling.
- Motivation: GUI-based CUAs offer near-zero-cost cross-platform generalization and a higher theoretical ceiling, but this paper focuses on their practical deployment bottlenecks.The paper contrasts GUI-based flexibility with CLI-based efficiency while investigating real-world GUI-CUA limitations.
- Failure analysis: Nearly 200 GPT-5.4 tasks across work, web usage, and gaming showed Action Grounding was the most important error source in work settings.The study analyzed failure cases in a Windows VM after excluding system errors.
- Benchmark gap: Existing grounding benchmarks emphasize rare interfaces, software-specific knowledge, single clicks, and GUI widgets rather than common complex interactions.Observed real-world operations include tables, documents, charts, and images, often requiring dragging or drawing.
- CUActSpot: CUActSpot covers GUI, Text, Table, Canvas, and Natural Image modalities with clicking, dragging, and drawing actions.Its tasks include tracing object boundaries in Photoshop for image cutout.
- CUActSpot: CUActSpot performance differs substantially from conventional GUI grounding benchmarks while agreeing more closely with end-to-end results such as OSWorld.The paper presents this agreement as evidence that CUActSpot may better reflect real-world computer-use scenarios.
- Data synthesis: The renderer-based synthesis pipeline generated 50M samples, and ablations found task and modality diversity more effective than scaling a single modality.The paper terms the observed benefit of broader diversity over within-modality scaling “variety scaling.”
2 Related Works
Computer-use agents have evolved from widget- and click-centric systems toward visual coordinate grounding, but both training data and benchmarks retain that narrow interaction prior.
- Computer-use agents: Modular CUAs separate planning from precise grounding, whereas end-to-end CUAs unify perception, reasoning, and action grounding at the cost of massive training data.The modular interface can lose spatial and contextual information.
- GUI action grounding: Early GUI agents represented screens as enumerable widgets selected through accessibility trees, DOMs, or Set-of-Marks identifiers.This design naturally framed action grounding as widget-centric and click-centric.
- GUI action grounding: Although later systems directly predict screen coordinates from visual input, data pipelines and benchmarks continue to produce widget boxes and single-point click labels.The related-work discussion describes this as a persistent widget-centric and click-centric prior.
3 CUActSpot Benchmark
CUActSpot evaluates complex computer-use interactions beyond widget-centric clicking, using diverse modalities, action types, and region-based success rules. Its design reduces ambiguity and specialized-knowledge demands while broadening task coverage.
- Benchmark goals: CUActSpot evaluates complex GUI interactions using a broader range of interaction types while reducing reliance on domain-specific knowledge.The benchmark is designed to measure action capabilities rather than overfitting to specialized software knowledge.
- Evaluation design: Its evaluation represents actions with one, two, or N key points and distinguishes ordered from unordered interactions.This covers clicks, drags, and drawing actions across varied task structures.
- Evaluation rules: Banned regions make any coordinate overlap incorrect, while ordered correct regions require rank-consistent key points.For unordered regions, every correct region must contain at least one key point.
- Evaluation rules: The benchmark reports sample success rate after applying prioritized rules for banned, ordered, and unordered regions.Correctness is determined per sample rather than by a single-point widget-selection rule.
- Benchmark scope: CUActSpot spans GUI, text, table, canvas, and natural-image targets, including insertion, cell manipulation, graphical-object operations, and image-region actions.Examples include dragging cell borders, manipulating PowerPoint objects, and drawing image-cutout boundaries.
- Benchmark distinctions: Compared with traditional benchmarks, CUActSpot offers more diverse task types and reduces ambiguity and dependence on specialized software knowledge.Traditional benchmarks are described as largely click-based and focused on standard GUI elements.
4 General Action Grounding Data Synthetic Pipeline
The synthetic pipeline renders modality-specific scenes, extracts spatial metadata, and uses an LLM to compose complex operation tasks. It supports GUI, table, text, canvas, and natural-image data through specialized rendering procedures.
- General pipeline: Code-based renderers generate screenshots together with bounding boxes and shape control points for visual elements.The resulting data pair screenshots with structured spatial metadata for multiple elements.
- General pipeline: An LLM selects salient elements, combines their metadata, and synthesizes complex operation tasks from rendered scenes.Prompts are designed separately for each modality.
- LLM task generation: OpenAI o3 can use supplied coordinates and intermediate calculations to construct more sophisticated tasks involving relationships among canvas shapes.The paper illustrates inferring positions needed to align an arrow tip with an ellipse control point.
- Modality-specific rendering: GUI data uses CommonCrawl webpages rendered with Playwright, while table data evolves HTML tables through structural and content modifications.The table process introduces multi-column layouts, merged cells, masking, and revised contents.
- Modality-specific rendering: Text data combines sampled fonts and text-background images with Python-based rendering, while canvas data randomly places varied PowerPoint-like shapes.Canvas scenes vary shape type, color, size, background, width, and height.
- Modality-specific rendering: Natural-image data samples regions from SAM images, generates fine-grained descriptions with GPT-4o, and derives polygonal boundaries from segmentation masks.The pipeline applies Suzuki–Abe contour extraction followed by contour sampling.
5 Experiments and Evaluations
The experiments compare CUActSpot with established grounding and agentic benchmarks, then evaluate synthetic-data training and data-composition choices. Results show benchmark sensitivity to software-specific training distributions and favor diverse task and modality coverage over scaling one modality alone.
- Benchmark comparisons: ScreenSpot-Pro and UI-Vision test grounding across software collections, but their tasks often require application-specific knowledge.This design evaluates software knowledge alongside grounding, making benchmark performance sensitive to the applications represented in training data.
- Benchmark comparisons: More than 20 points separate ScreenSpot-Pro and UI-Vision for many recent models, whereas OS-Atlas, UGround, and GPT-5.4 show markedly smaller gaps.The benchmarks differ in software coverage and UI-Vision uses lower screen resolutions, so the gap should not be read as direct evidence of overfitting.
- Benchmark comparisons: OSWorld evaluation controls planning with GPT-5.4-generated single-step instructions, isolating grounding; models stronger on ScreenSpot-Pro do not show a corresponding OSWorld advantage.This result suggests a mismatch between conventional grounding benchmarks and real-world scenarios.
- Synthetic-data training: Phi-Ground-Any-4B, trained on fully synthetic data together with OpenCUA data, outperforms all open-source models smaller than 32B parameters on CUActSpot.Fine-tuning with application-specific data improves ScreenSpot-Pro and UI-Vision performance, supporting the pipeline’s use as a pretraining strategy.
- Empirical studies and ablations: Scaling one modality alone is less effective than increasing task and modality diversity, which also improves capabilities on other modalities.The ablations suggest task diversity may be at least as important as raw data scale for learning knowledge that generalizes across modalities.
- Empirical studies and ablations: The trained model succeeds on more detailed CUActSpot tasks than were explicitly present in training, indicating limited compositional generalization across task combinations.The paper illustrates this with combining textual interaction and visual-region manipulation to operate on text embedded in visual content.
6 Conclusions and Limitations
The paper introduces CUActSpot and a scalable synthesis pipeline for diverse computer-use interactions, while reporting strong performance from Phi-Ground-Any-4B. It also notes that the benchmark is diagnostic rather than exhaustive, especially for long-horizon, stateful, and real-world-distributional coverage.
- CUActSpot broadens computer-use evaluation beyond click-centric settings to diverse interactions and modalities.
- The proposed synthesis pipeline supports scalable coverage, and ablations favor increasing task and modality diversity over scaling a single modality.
- Phi-Ground-Any-4B achieves strong performance on complex interaction benchmarks and remains competitive among similarly scaled models.
- CUActSpot uses manually curated samples and does not exhaustively cover real-world workflows, particularly long-horizon and stateful scenarios.
- Aligning synthetic data more closely with real-world distributions remains an important direction for future work.
A.1 Detailed Tasks Breakdown
The detailed-task appendix documents CUActSpot’s task breakdown and reports modest evaluation variability. Its tables organize the benchmark’s fine-grained tasks, while experiments quantify checkpoint and temperature fluctuations.
- CUActSpot exhibits approximately ±3% fluctuations between adjacent training checkpoints.
- Testing CUActSpot with different temperature settings produces around 2% variation.
- Tables 5 and 6 present the detailed CUActSpot task categories and relate block and row counts to aggregate task totals.
A.2 Benchmark examples
Figures 6 and 7 provide examples of CUActSpot benchmark instances. Together, they illustrate the benchmark’s example-based presentation across two figures.
- Figure 6 presents examples of CUActSpot.
- The benchmark examples are distributed across Figures 6 and 7.
- Figure 7 presents examples of CUActSpot.
B.1 Data sampling
The training setup fixes key optimization and visual-input settings while using a large, mixed data composition. OpenCUA receives increased weighting for quality, but its limited scale makes exclusive use unsatisfactory.
- Training uses 16 visual crops, batch size 5120, and learning rate 8 × 10−5.
- Weight decay of 0.01 and gradient clipping at 0.1 are important for training stability.
- OpenCUA receives increased training proportion because it is manually annotated and expected to have higher quality.
- Using only OpenCUA yields unsatisfactory performance because of its limited scale.
- The overall training budget is approximately 100B tokens and requires about 30 hours on 80 NVIDIA H100 GPUs.
B.2 Data ablation results
This section presents data-ablation results and identifies the evaluation context used for comparing training-data choices.
- The experiments report the best checkpoint saved during training.Checkpoints are saved every 100 training steps.
- Table 8 reports the data ablation results.
- The CommonCrawl data-processing pipeline is documented as part of the data-generation workflow.
C.1 GUI
The GUI data pipeline renders webpages, filters and samples interactive elements, and uses language models to generate diverse screen-operation tasks and executable action traces.
- Rendering: The pipeline renders webpages at varied resolutions and aspect ratios, then filters HTML elements likely to be interactive.It uses Selenium and Chrome Driver, with 1080p, 2K, and 4K pixel areas and aspect ratios from 2:1 to 1:2.
- Rule-based filtering: Filtering and deduplication reduce the preserved webpage collection to 73.5M webpages.
- Element selection and labeling: Element selection samples across canvas regions, prioritizes infrequent icons, and generates 10 samples per screenshot.GPT-4o labels each selected element.
- Task generation: GPT-4o labels selected elements, after which o3 generates screen-operation tasks from annotated screenshots and coordinates.The construction prompt supplies screenshots, elements, and click coordinates.
- Training data format: Generated training examples pair prompts with PyAutoGUI responses, coordinate mappings, and used elements.Coordinates are replaced by symbols such as x1 and y1 in the response code and mapped separately.
- Action types: The action taxonomy includes zero-coordinate, one-coordinate, and multi-coordinate or combined operations such as scrolling, clicking, dragging, and typing.Combined actions include focusing a text box before typing and mouse operations for selection or dragging.
- Action diversity: The generation process emphasizes action diversity, including combinations that may not match the apparent affordances of the selected region.
- The generated tasks include table-like operations such as dragging cells and adjusting cell boundaries, beyond simple clicking.
D Case study
The case study evaluates Phi-Ground-Any on a repeated spreadsheet drag-and-drop task that fills blank cells without altering irrelevant regions.
- The OSWorld LibreOffice Calc task asks the agent to fill blank cells in B1:E30 using the value above each cell.The instruction explicitly prohibits touching irrelevant regions.
- Using GPT-5.4 as planner and Phi-Ground-Any as grounder, the agent repeatedly drags a cell into an empty cell below to copy its content.The repeated operation occurs across steps 13 to 29.
- The repeated drag-and-drop operation is executed correctly across multiple repetitions on tabular data.