Grounding Computer Use Agents on Human Demonstrations

• Grounding is essential because a plan cannot be executed reliably unless the agent identifies the correct on-screen element, while dense layouts, similar controls, and unseen user artifacts make desktop grounding difficult.Source: §1 · p. 1
• GroundCUA provides 3.56M+ human-verified element annotations across 56K screenshots from 87 desktop applications in 12 categories.It includes high-resolution, dense annotations, fine-grained UI categories for half of elements, and related application variants; the data supports a 700K image-instruction pair training set.Source: §1 · pp. 2–3
• GroundNext models at 3B and 7B achieve state-of-the-art results on ScreenSpotPro, OSWorld-G (Xie et al., 2025), and UI-Vision (Nayak et al., 2025) using 700K SFT datapoints versus JEDI’s 9M.A second RL stage further improves grounding without complex reward strategies; GroundNext-3B also performs strongly on OSWorld-Verified agentic tasks, surpassing many larger models and matching JEDI-7B.Source: §1 · pp. 2–3
• GroundNext generalizes beyond desktop despite desktop-only training, with strong results reported across desktop, mobile, and web interface benchmarks.The introduction names MMBench-GUI (L2) and ScreenSpot-v2 among the evaluation benchmarks.Source: §1 · p. 2
• The paper analyzes the roles of SFT and RL, GroundCUA’s cross-domain impact, generalization beyond desktop, and open-source software’s benefits for grounding, and plans to release the dataset and models.Source: §1 · p. 3

Computer-Use Agents

• SFT agents and grounding-focused models have improved cross-platform interaction and language-to-element mapping, while RL approaches use distance-based rewards; reliable grounding remains a bottleneck.The paper focuses on expert-annotated data and data-efficient SFT and RL training to improve grounding.Source: §2 · p. 3

GUI Grounding Datasets

• Mobile datasets use standardized layouts, web datasets often harvest HTML/DOM, and desktop datasets rely on imperfect accessibility trees or simplified synthetic screens.Automated web pipelines may underrepresent small icon-only controls, while accessibility-tree labels can be missing or imprecise.Source: §2 · p. 4

How is ours different?

• 55,568 screenshots across 87 open-source applications contain over 3.56M human-verified UI elements in GROUNDCUA.The dataset averages 64 annotations per screenshot, more than three times OS-ATLAS (Desktop) and above Aguvis-G (9) and UGround (11).Source: §2 · p. 4
• GROUNDCUA captures denser screens, a wider resolution range, smaller elements, and desktop icons, toolbars, and controls that automated tools struggle to capture.Its screenshots range from 0.39 to 7.0M pixels, including very high-resolution images.Source: §2 · p. 4

Collecting demonstrations from human experts

• The dataset combines expert users’ real-world desktop interactions with practical tasks, yielding natural trajectories and screenshots rather than randomly generated interface states.Annotators completed over 10,000 demonstrations across 87 applications.Source: §3 · pp. 4–5

Selecting diverse desktop applications

• GROUNDCUA covers 87 open-source applications across 12 categories, including finance and scientific applications, selected to support broad-domain agents and free release.The applications are intended to mirror functionality in popular closed-source software.Source: §3 · p. 4

Dense annotation of screenshots

• Annotators labeled every visible element in pre-action keyframes with bounding boxes and textual labels, while OCR captured raw text in longer segments.Around 50% of elements also received one of eight high-level categories.Source: §3 · p. 5

Constructing high-quality finetuning instructions

• A multimodal LLM uses annotated boxes, application names, labels, and surrounding context to generate instructions grounded in screenshot content.The pipeline produces Direct, Functional, and Spatial instructions, covering element attributes and context, intended actions, and relative positions.Source: §3 · p. 5

Dataset Statistics

• 56K screenshots contain 3.56 million annotated elements, averaging 64 per screenshot (up to 542), with bounding boxes covering 0.13% of image area on average.The dataset also provides 700K samples for SFT and 10K for RL; screenshot resolution averages 2.03 megapixels.Source: §3 · p. 5

4.1 MODEL TRAINING

• SFT uses Qwen2.5-VL-Instruct at 3B and 7B scales, fine-tuning both the vision encoder and language model on a diverse, coverage-balanced subset of 700K instructions.Training uses a single node with 8 H100 GPUs and a global batch size of 128.Source: §4.1 · pp. 5–6
• RL post-training on 10K elements excluded from the 700K SFT set yielded the strongest generalization in initial experiments and was adopted for the final model.Policy optimization uses RLOO, comparing each rollout’s reward with the average reward of other samples in its group, without a separate critic.Source: §4.1 · p. 6
• RL assigns a discrete reward from normalized distance to the target box, mildly penalizing predictions just outside it, more strongly penalizing distant errors, and encouraging predictions inside it toward the center.The normalized distance is between -1 and 1; the final setup uses groups of 8, batch size 64, and one epoch on 8 H100 GPUs.Source: §4.1 · p. 6

4.2 EVALUATION

• A grounding prediction is correct when its 2D image-coordinate point falls inside the target element’s axis-aligned bounding box; performance is reported as accuracy.Source: §4.2 · p. 6
• Evaluation covers five benchmarks across desktop, mobile, and web, treating UI-Vision as in-domain and the others as out-of-domain; comparisons include SFT-only variants and recent RL-based models.The benchmarks are ScreenspotPro, OSWorld-G, UI-Vision, MMBench-GUI (L2), and Screenspot-v2; the authors note that perfect training/evaluation separation is not always possible.Source: §4.2 · pp. 6–7

5.1 EFFICIENT SUPERVISED FINE-TUNING WITH HIGH-QUALITY DATA

• 68.4 vs. 63.0 Avg (w/o UI-V) puts GROUNDNEXT-3B (SFT) ahead of GUI-Actor-3B, while GROUNDNEXT-7B (RL) achieves the top RL-tuned 7B Avg (w/o UI-V) at 73.0.GROUNDNEXT-3B (RL) also achieves 70.0 Avg (w/o UI-V), the best overall result in that size range.Source: §5.1 · p. 7, Table 3 · p. 8
• 700K instructions, versus JEDI’s 9M, underpin results that outperform larger SFT baselines and remain competitive with RL-tuned systems.In a controlled comparison using the same base model and 100K samples per dataset, GROUNDCUA yields significantly higher SFT averages than Aguvis, UGround, OS-Atlas (Desktop), and JEDI; the paper attributes this to high-quality, densely grounded supervision and targeted instruction design.Source: §5.1 · pp. 7–8

5.2 REINFORCEMENT LEARNING POST-TRAINING RESULTS

• 70.0 Avg (without UI-V) and 68.4 overall for GROUNDNEXT-3B (RL) exceed GROUNDNEXT-3B (SFT)’s 68.4 and 66.4, respectively.For 7B, GROUNDNEXT-7B (RL) scores 70.5, improving on GROUNDNEXT-7B (SFT)’s 69.2 (with UI-V); the authors characterize RL gains as consistent but modest.Source: §5.2 · p. 8, Table 3 · p. 8
• Models with GROUNDCUA SFT show the smallest RL gains, and stronger initial SFT performance correlates with smaller absolute gains.The analysis applies RL using 10K GROUNDCUA examples to SFT models trained on GROUNDCUA or other datasets, suggesting GROUNDCUA SFT leaves fewer errors for RL to correct.Source: Figure 3 · p. 8, §5.2 · p. 9

5.3 FURTHER ANALYSIS

• 50.6 Overall puts GROUNDNEXT-3B ahead of OpenCUA-72B (46.1), Qwen3-VL-Flash (41.6), and Claude-4-Sonnet (41.4) on OSWorld-Verified.Against JEDI-7B, which also uses the o3 planner, its 50.6 overall score is comparable to 51.0 despite its smaller size, and it performs better in OS, Office, and Workflow.Source: §5.3 · p. 9, Table 4 · p. 9
• 3.66% improvement on desktop platforms over InfiGUI-G1 on MMBench-GUI accompanies the best GROUNDNEXT-7B (RL) performance on UI-V, OSW-G, and SSPro.On SSPro icon recognition, GROUNDNEXT outperforms most models by an average of 10.7%.Source: §5.3 · p. 10
• 89.2% on mobile and 81.9% on web for GROUNDNEXT-7B (RL) on MMBench-GUI trail InfiGUI-G1-7B’s 90.9% and 85.3%, respectively.On SSv2, GROUNDNEXT achieves comparable mobile results but falls behind on web.Source: §5.3 · p. 10
• 15.9% in Development and 8.4% in Creative are GROUNDNEXT’s icon-accuracy advantages over InfiGUI-G1-7B on SSPro.GROUNDNEXT achieves the best icon performance in Office Suite, Development, Creative, Scientific, and CAD, and ranks second in OS; the paper notes open-source application diversity as a likely contributor.Source: §5.3 · p. 10

6 CONCLUSION & DISCUSSION

• GroundCUA provides dense desktop annotations across 87 applications, and GroundNext achieves state-of-the-art results on five benchmarks despite using substantially less SFT data than many prior works.The dataset contains 56K screenshots and more than 3.56M elements; the authors emphasize high-quality data over sheer volume and release research artifacts.Source: §6 · p. 10
• Larger fine-tuning sets could scale performance, while dense annotations could support more precise RL reward signals and studies of their effects on grounding accuracy.The paper notes that its models were trained with limited scale and compute and used a simplistic RL reward.Source: §6 · p. 10
• Future work could mix desktop, mobile, and web data for cross-platform generalization, and use platform- and category-level metadata to study continual learning on unseen applications.Balancing domains and addressing transfer bottlenecks will require careful study.Source: §6 · p. 10

• All desktop applications were open-source and permissively licensed, no PII was collected during screenshot annotation, and annotators were fairly compensated under data-protection protocols.The authors also describe the work as focused on responsible development through transparent dataset curation and model training.Source: ETHICS STATEMENT · p. 11
• Although potentially harmful content was filtered, the authors cannot guarantee that GroundCUA-trained models will avoid inappropriate instructions or interactions with sensitive interface elements.They strongly encourage safeguards and human oversight when deploying computer-use agents in production.Source: ETHICS STATEMENT · p. 11
• Human evaluation studies were conducted by collaborating researchers under established ethical guidelines, with no PII collected during evaluation.Source: ETHICS STATEMENT · p. 11
• The authors disclose no conflicts of interest and state that funding sources will be listed in the camera-ready version per ICLR policy.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The authors will publicly release the complete GroundCUA dataset, GroundNext weights, training code, evaluation scripts, and detailed data sheets.They report documenting hyperparameters, training procedures, preprocessing, evaluation metrics, annotation guidelines, quality protocols, instruction-generation prompts, model architectures, and benchmark procedures.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 PLATFORMS

• GroundCUA covers 87 platforms across 12 categories, selected to address under-representation of open-source software and enable release with minimal licensing restrictions.The authors primarily drew platforms from UI-Vision and added four to improve finance and scientific coverage.Source: §A.1 · p. 16
• The authors state that open-source platform selection does not compromise generalization because open- and closed-source counterparts often share UI elements and layouts.They cite LibreOffice and Office Suite as an example and point to Section 5.3 for further evidence.Source: §A.1 · p. 16

A.2 HUMAN ANNOTATION

• The annotation effort used a three-phase collaboration with a professional labeling vendor and a team of around 70 annotators, quality specialists, and project managers.A pilot study refined task instructions through iterative feedback; annotators had technical degrees and prior data-labeling or UI-research experience.Source: §A.2 · p. 16
• Annotators executed tasks in 87 software applications while screen recordings were captured, then labeled keyframes with bounding boxes for visible interface elements.They assigned element names and, for textual elements, text; long text was summarized and supplemented with PaddleOCR when available, and elements were assigned to one of six high-level categories.Source: §A.2 · p. 17
• Annotations were reviewed by quality specialists, cross-checked by the authors, and validated with custom evaluation scripts.Annotators trained on the platforms and guidelines, were compensated hourly, and spent an average of 60–90 minutes on each task, including quality checks.Source: §A.2 · p. 17

A.3 DATASET STATISTICS

• GROUNDCUA’s screenshots range from roughly 0.3 to 7 megapixels, and some contain up to 500 annotated elements.Source: §A.3 · p. 17, Figure 4 · p. 18
• The dataset statistics break down annotations, screenshots, and applications across 12 categories, and show a prevalence of small bounding boxes.Source: §A.3 · p. 17, Figure 4 · p. 18

A.4 COMPARISION WITH PRIOR WORKS

• GROUNDCUA screenshots span 0.5M–7M pixels and average 2.0M, while UGround and OS-Atlas (Desktop) average 1.1M and 1.6M pixels, respectively.Source: §A.4 · p. 17, Figure 5 · p. 19
• GROUNDCUA has a smaller median element size, capturing fine-grained desktop targets such as icons and small controls that other datasets often omit.Source: §A.4 · pp. 17, 19, Figure 5 · p. 19
• GROUNDCUA averages 64 annotations per screenshot—more than three times OS-Atlas (Desktop)—versus 9 for Aguvis and 11 for UGround; JEDI annotations are sparser.Source: §A.4 · p. 19
• GROUNDCUA contrasts with web-focused UGround and Aguvis, automated accessibility-tree-based OS-Atlas, and synthetic-data JEDI through dense, real-world desktop coverage.Source: §A.4 · p. 19

A.5 DATASET EXAMPLES

• Figure 6 presents screenshots from several software platforms with bounding boxes overlaid on the images.Source: §A.5 · p. 19

B INSTRUCTION TUNING DATA

• Filtering reduces over 3.5M annotated elements to roughly 900K unique elements using label matching and crop pHash, then randomizes screenshot selection to limit interface overrepresentation.Source: B INSTRUCTION TUNING DATA · p. 19

B.1 DIRECT INSTRUCTIONS

• Direct instruction generation combines descriptive attributes and context with text-matched labels, visual captions, and heuristic or MLLM templates; Figure 7 shows examples.Descriptive instructions are generated with Qwen2.5-VL-72B using the element box, platform, label, screenshot, and optionally a zoomed crop.Source: §B.1 · pp. 19, 21, §B.4 · p. 22

B.2 FUNCTIONAL INSTRUCTION PROMPT

• Functional instructions identify an element by its purpose, focusing on buttons and menus and prompting Qwen2.5-VL-72B with the screenshot, crop, bounding box, platform, and label.Source: §B.2 · p. 22, Functional Instruction Prompt · p. 22

B.3 SPATIAL INSTRUCTIONS

• Spatial instructions identify a target relative to a nearby, reliably labeled anchor using relations such as left, right, above, below, or between.Source: §B.3 · p. 22

B.5 NEED AND IMPACT OF DIFFERENT INSTRUCTION TYPES

• Functional instructions achieve the highest average performance, while Direct - Text and visual scores 0.64 versus Functional’s 0.60 on OSWorld-G text recognition.The comparison comes from subtype ablations trained on 100K examples per subtype.Source: §B.5 · pp. 22–23, Table 7 · p. 23
• The final 700K mixed SFT set was chosen for performance and prompt robustness; Description-only training drops 4% on ScreenSpot-Pro without its prompt prefix, and final models are evaluated without prefixes.Source: §B.5 · p. 23

B.6 QUALITY OF INSTRUCTION TUNING DATA

• Three independent annotators’ majority vote found a 4% error rate among 100 sampled instructions, which the authors report as evidence of instruction-data reliability.The generation pipeline grounds rephrasing in element labels, bounding boxes, and screenshot context rather than relying on open-ended MLLM knowledge.Source: §B.6 · p. 23

C TRAINING

• The training section outlines SFT and RL design choices, including data selection and filtering, hyperparameters, and experimental observations.Source: C TRAINING · p. 23

C.1 SFT DATA

• The 700K SFT corpus combines direct (50%), functional (35%), and spatial (15%) instructions; training uses two epochs and updates the full model.The reported training setup uses a 3e-6 learning rate, cosine decay, and a 0.05 warmup ratio.Source: §C.1 · p. 23, §C.2 · p. 24

C.3 RL DATA

• RL uses 10K unseen GROUNDCUA samples because SFT-error rejection sampling was limited by noisy or ambiguous labels, while adding 10K GUIAct mobile/web samples gave no consistent gains.The out-of-distribution experiment was preliminary and did not include rejection sampling or other analysis.Source: §C.3 · p. 25

C.4 RL TRAINING DETAILS

• RLOO was selected over GRPO for more stable learning; continuous rewards had sparse, weak gradients, and binary rewards lacked error-magnitude sensitivity, motivating a shaped discrete reward.The final reward distinguishes signed-distance ranges to penalize misses and encourage predictions inside the box to move toward its center, addressing the prevalence of near misses.Source: §C.4 · p. 25, §2 · pp. 25–26

C.5.1 DISCRETE REWARD DESIGN

• The six-level reward design achieved 63.1% on OSWorld-G and the highest average score, 65.2%, across the tested granularity settings.The selected reward values were {−1.0, −0.5, −0.1, 0.1, 0.5, 1.0}.Source: §C.5.1 · p. 27, Table 8 · p. 27

C.5.3 RLOO VS. GRPO

• RLOO achieved performance parity with GRPO, averaging 65.7% versus 65.3%; group size n = 32 performed best, but n = 8 was chosen for efficiency.The authors caution that n = 4 may be unstable for gradient estimation and that n = 32’s marginal gains did not justify its computational cost.Source: §C.5.2 · pp. 27–28, §C.5.3 · p. 28, Table 10 · p. 28

D.1 SCREENSPOTPRO RESULTS

• The appendix reports model results on ScreenSpot-Pro, OSWorld-G, MMBench-GUI, ScreenSpot-v2, and UI-Vision.The supplied passages identify these benchmark result tables but do not provide their score values.Source: §D.1 · p. 28, §D.2 · p. 28, §D.3 · p. 28

D.6.1 HIGH-QUALITY SFT CREATES A “STRONG CEILING”

• For 3–7B models under the employed rewards, strong GROUNDCUA SFT leaves fewer actionable errors, making RL a refinement rather than the primary performance driver.The authors qualify this as a hypothesis, not a general claim that stronger models cannot benefit from RL; related work GTA1 reports a 3.3% average improvement across five benchmarks (Yang et al., 2025).Source: §D.6.1 · pp. 30–32

D.6.2 GROUNDCUA IS A GREAT SOURCE FOR RL FINE-TUNING

• GROUNDCUA-based RL gains 9.8% on average over the Qwen2.5-VL-3B baseline and 1.9% over the next-best baseline (2.5% excluding SSv2).The controlled comparison used 10K samples per dataset and a binary reward; the authors attribute GROUNDCUA’s advantage to less noisy human-annotated labels and diverse platforms.Source: §D.6.2 · p. 32, Table 17 · p. 32

D.7 SFT SCALING BEHAVIOR

• The scaling study trained Qwen2.5-VL-Instruct-3B on 100k, 350k, and 700k GROUNDCUA instruction subsets using identical hyperparameters.Source: §D.7 · p. 32
• Performance improved across almost all benchmarks as data increased, with deduplication adding visual and semantic variety that strengthened grounding.The authors report stable training across scales and suggest larger curated subsets may yield further gains.Source: §D.7 · pp. 32–33
• UI-Vision rose from 29.8 at 100k to 58.2 at 700k instructions, the most pronounced gain as GROUNDCUA SFT data scaled.The scaling ablation compares subsets of 100k, 350k, and 700k instructions.Source: §D.7 · p. 32, Table 18 · p. 33
• OSWorld-G and ScreenSpot-Pro also showed positive trends, indicating consistent benefits in dense desktop scenarios.Source: §D.7 · p. 33

E GROUNDNEXT ERROR ANALYSIS

• Out-of-domain web and mobile errors stem from distribution shifts and limited platform knowledge, with mobile aspect ratios, resolutions, and UI patterns differing from desktop training data.The authors also identify lack of application knowledge as a factor in mobile errors, including on simple queries.Source: E GROUNDNEXT ERROR ANALYSIS · pp. 33–34
• Over 50% of grounding errors have relative distance below 10% from the ground-truth box, indicating many failures are near-misses with correct semantics but imperfect localization.The paper illustrates a near-miss in Figure 11 (left).Source: E GROUNDNEXT ERROR ANALYSIS · p. 34, Figure 11 · p. 34
• On unseen applications, GROUNDNEXT struggles with specialized terminology and icons, such as VMware’s “restart from CD” and “snapshot details,” but remains robust on generic elements.VMware is absent from the training data; “Refresh” and “font size” are cited as generic elements.Source: E GROUNDNEXT ERROR ANALYSIS · p. 34

F LIMITATIONS

• The dataset may underrepresent desktop software diversity, omit dynamic UI states, contain some human-label inconsistencies, and remain costly to scale.The authors also flag benchmark-focused evaluation, robustness to distribution changes, app versions, and UI updates, and the absence of end-to-end agentic task-completion testing as limitations.Source: F LIMITATIONS · p. 34

G LLM USAGE

• The authors used GPT5 to improve paper language, grammar, and LaTeX table code, and to search related work; LLM use in methods and experiments is described elsewhere in the paper.The disclosure specifies these as the paper’s three areas of LLM use.Source: G LLM USAGE · p. 35