DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

1 INTRODUCTION

• The training-data gap is dense supervision for complete desktop scenes that can vary systematically, where similar controls and cross-application distractors compete for attention.Prior resources provide some relevant properties, but none combines them; static corpora cannot vary captured scenes, and executable environments focus on task evaluation rather than dense screen annotation.Source: §1 · p. 1
• DeskForge composes and explores real applications in configurable scenes, capturing interface structure, window layout, and the outcomes of executed actions.Its annotations reconcile screenshots, accessibility information, and window geometry.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Fine-tuning four vision-language models improves accuracy in every held-out condition and raises mean accuracy across five external GUI grounding benchmarks by 1.5 to 24.9 points.For Qwen, mean held-out accuracy rises from 76.26% to 87.54%; under a fixed planner, Qwen solves 50 rather than 31 WebArena-Infinity tasks and 15 rather than 3 OpenApps tasks.Source: §1 · p. 2
• DeskForge-1M contains 1.2M annotated desktop observations, 159.7M element instances, and 917K recorded click transitions.It spans 19 applications, seven appearance presets, and seven display resolutions.Source: §1 · pp. 2–3

2 RELATED WORK

• Structured screen resources retain element attributes and relationships, but annotate individual web pages, mobile screens, or applications rather than composed multi-application desktops.Examples include Rico, WebUI, Pix2Struct, ScreenParse, and GroundCUA (Wu et al., 2023).Source: §2 · p. 3
• DeskForge differs from interface synthesis and automated collection approaches by configuring and exploring multiple real applications within shared scenes.GUIrilla explores applications individually, while other approaches synthesize or vary interfaces (Garkot et al., 2025; Xie et al., 2025; AI2, 2026; Gou et al., 2025).Source: §2 · p. 3
• Unlike executable task environments or static window compositions, DeskForge collects dense annotations and observed click outcomes from an interactive running desktop.WebArena and OSWorld connect actions to task outcomes; WinDeskGround varies captured-window layout but cannot execute further actions.Source: §2 · p. 3
• DeskForge constructs single-step grounding instructions from recorded clicks and before-and-after observations, retaining transitions for possible dynamics learning.This interaction-first approach contrasts with human demonstrations and task-directed automated trajectories (Jian et al., 2026; Mu et al., 2025).Source: §2 · p. 3
• Among the listed resources, only DeskForge combines dense labels, multi-application scenes, a configurable environment, visibility geometry, and recorded transitions.This combination addresses the distinction between annotating individual pages, screens, or applications and annotating composed desktop scenes.Source: §2 · p. 3

3 THE DESKFORGE ENVIRONMENT

• DeskForge composes and explores running desktop applications, relating rendered content to interface structure and recording executed-action outcomes.A scene specification controls applications, initial states, content, window layout and stacking, appearance, and resolution.Source: §3 · pp. 3–4, §3.1 · p. 4

3.1 SCENE COMPOSITION AND VISUAL DIVERSITY

• Scene configuration varies application content and multi-window context, while seven appearance presets vary styling, iconography, window decorations, and desktop layout.The presets share a Linux backend and reproduce selected Windows- and macOS-inspired visual conventions rather than running native applications.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 DENSE STRUCTURED ANNOTATIONS

• Annotations combine screenshots, accessibility observations, and window relationships to represent element attributes, hierarchy, ownership, reading order, and interaction properties.Geometry refinement models clipping and overlap; partially covered controls retain visible support as a union of rectangles, and a human audit found 99.8% of sampled element annotations correct.Source: §3.2 · pp. 4–5

3.3 INTERACTION RECORDING AND INSTRUCTION CONSTRUCTION

• DeskForge explores by clicking sampled actionable elements and recording before-and-after states, target metadata, and aggregate effects, including unchanged outcomes.A vision-language model turns the before and after observations and target context into a single-step instruction whose target is identifiable from the before screenshot.Source: §3.3 · p. 5

4.1 SCALE AND COMPOSITION

• DeskForge-1M contains 1.21M observations, 159.7M element instances, and 917K click transitions across 19 applications, seven appearance presets, and seven display resolutions.Observations contain 132 annotated elements on average; 90.9% record at least two applications, and 97.7% contain occluded elements.Source: §4.1 · p. 6

4.2 ANNOTATION CONTENT AND SUPERVISION VIEWS

• Each observation pairs a screenshot with dense element records and a ScreenTag representation, while transitions store clicks, targets, successor observations, and change summaries.A fixed 200K-example grounding view pairs a before screenshot and synthesized instruction with the recorded target; a human audit judged 97.8% of sampled instructions sound.Source: §4.2 · p. 6
• Qwen3.6-27B synthesizes concise, standard, and more contextual single-step instructions from recorded clicks, target context, and before-and-after screenshots.The successor observation informs instruction construction but is not supplied to the grounding model.Source: §4.2 · p. 6

4.3 EVALUATION SPLITS

• Scene-level partitions keep interaction frames and recaptures of the same scene together, with test conditions for new scenes, held-out applications, a reserved appearance preset, and a reserved resolution.The application split holds out GNOME System Monitor, Pluma, and Xarchiver; the reserved appearance is Quartz Night Nord and the resolution is 2880 × 1800.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

• DeskForge-1M fine-tuning uses 200K single-target grounding examples for four VLMs, with prompts, preprocessing, decoding, and scoring held fixed within each model pair.The models are Qwen3.5-4B, Gemma4-E4B IT, InternVL3.5-8B, and GUI-specialized UI-R1-3B; the first three are general-purpose VLMs.Source: §5 · p. 7

5.1 GROUNDING ACROSS HELD-OUT DESKTOP CONDITIONS

• All four models improve in every held-out condition; Qwen’s mean accuracy rises from 76.26% to 87.54%, while UI-R1 rises from 38.01% to 84.59%.For Qwen, the gap to its base model widens from 9.7 to 14.3 points with more applications on screen, and from 10.9 to 13.7 points as target visible-area loss rises to 35%.Source: §5.1 · p. 8
• Fine-tuning improves grounding for all four models across new scenes, held-out applications, a reserved appearance, and a reserved resolution.The four scene-level conditions are evaluated separately and as a mean, using matched base/fine-tuned pairs.Source: §5.1 · p. 8

5.2 TRANSFER TO EXTERNAL GROUNDING BENCHMARKS

• All four models improve on all five external grounding benchmarks; Qwen gains 11.51 points on ScreenSpot-Pro and 10.11 points on OSWorld-G.The external benchmarks differ from DeskForge-1M in platform coverage, application inventory, and annotation source.Source: §5.2 · p. 8, Table 3 · p. 8
• External gains include GUI-specialized UI-R1, which rises from 14.48% to 27.20% on ScreenSpot-Pro and from 39.87% to 47.14% in five-benchmark mean accuracy.The ScreenSpot-v2 breakdown also reports Qwen and UI-R1 gains on mobile despite DeskForge-1M containing no mobile screens.Source: §5.2 · p. 8

5.3 LONG-HORIZON TASK COMPLETION UNDER A FIXED PLANNER

• With a fixed Qwen3.6-27B planner, Qwen solves 31 to 50 WebArena-Infinity tasks and 3 to 15 OpenApps tasks after fine-tuning.All four fine-tuned action models solve more tasks in both environments; only action-model weights change within each base/fine-tuned comparison.Source: §5.3 · p. 9, Table 4 · p. 9
• Gemma solves 5 to 40 tasks on the WebArena-Infinity panel, the largest increase reported there; InternVL and UI-R1 also solve more tasks in both environments.The task sets contain 119 WebArena-Infinity tasks and 100 longer-horizon OpenApps tasks.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 DENSE ELEMENT DETECTION

• RT-DETRv4-L reaches 62.17% group F1 on GroundCUA, versus 58.55% for OmniParser v2 and 57.59% for ScreenParse YOLO11L.The detector is trained on DeskForge dense annotations and outperforms both comparison detectors on all three localization metrics.Source: §5.4 · p. 9

6 CONCLUSION AND FUTURE WORK

• DeskForge-1M contains 1.2M annotated observations and 159.7M element instances, with click transitions; fine-tuning improves grounding and task completion, and dense annotations also support element detection.The reported task-completion comparisons use a fixed planner.Source: §6 · p. 9
• Current acquisition uses a Linux backend and depends on applications with suitable accessibility support.Broader application integration, native-platform backends, and complementary visual annotation are identified as ways to extend coverage.Source: §6 · pp. 9–10
• VLM training focuses on single-target grounding; joint learning from interface structure and recorded state changes remains future work.The authors propose forward and inverse dynamics objectives, plus goal-conditioned multi-application tasks with state-based verification.Source: §6 · p. 10

A.1 OBSERVATIONS, ANNOTATIONS, AND SUPERVISION VIEWS

• DeskForge-1M contains 1,207,368 observations and 159,723,780 retained visible element instances, averaging 132.29 elements per observation.The observation payload includes screenshots, element annotations, ScreenTag serialization, and capture records; training and evaluation can use stored files without running the environment.Source: §A.1 · p. 15

A.2 SCENE-LEVEL SPLITS

• All captures and recaptures from a scene share one split; a filtered state view contains 1,067,799 observations and 142,537,053 element instances.The split table reports partition counts, while its 200,000 training grounding examples are sampled from 551,651 training-split instructions.Source: §A.2 · pp. 15–16, Table 7 · p. 16
• Seven display resolutions are represented, and Figure 4 reports observation counts and proportions across all 1,207,368 observations, including successive episode frames.Source: Figure 4 · p. 16

A.3 APPLICATION AND DISPLAY COMPOSITION

• Application co-occurrence is computed within captures and averaged at scene level, so longer episodes do not receive extra weight from having more frames.Figure 5 reports application co-occurrence and marginal presence across captures; presence comes from capture metadata rather than visible-window counts.Source: §A.3 · p. 16, Figure 5 · p. 17
• Figure 3 weights observations equally; its occlusion proportion counts elements clipped or removed by the occlusion pass, not visibility of the selected grounding target.This occlusion proportion has a different denominator from retained-element density.Source: §A.3 · p. 16

B.1 EXECUTION AND ACQUISITION

• Acquisition uses isolated Linux desktop sessions, and independent batch jobs scale capture across many sessions.The largest campaign measured 84,807 observations per hour using 1,500 concurrent sessions; these figures describe acquisition, not instruction generation or model training.Source: §B.1 · p. 18

B.2 APPEARANCE PRESETS AND SOURCE ASSETS

• Seven presets vary application themes, icons, window decorations, fonts, and panel or dock layouts while retaining a Linux execution backend.The presets span classic Linux, Ubuntu-like, Windows-inspired, and macOS-inspired styles, including light and dark variants.Source: §B.2 · p. 18

B.3 ANNOTATION VIEWS AND REFINEMENT

• Three annotation views preserve captured hierarchy, support parsing and ScreenTag serialization, or add retained hidden elements and visibility states.Geometry refinement accounts for clipping and occlusion; visible-fragment unions define grounding regions, while audits reject inconsistent or implausibly sparse captures.Source: §B.3 · pp. 18–19

B.4 RECORDED INTERACTIONS AND INSTRUCTION CONSTRUCTION

• Exploration samples actionable elements rather than following predefined task goals, linking each click to before-and-successor observations and an effect summary.Instruction synthesis creates concise, standard, and detailed formulations of a single-step goal, then checks drafts against visual evidence and filters malformed outputs.Source: §B.4 · p. 19

C.1 GROUNDING FINE-TUNING

• Four vision-language models are fine-tuned with LoRA on the 200K single-target grounding view, while their vision encoders and multimodal alignment modules remain frozen.UI-R1 starts from its released GUI-specialized checkpoint and retains its native action-answer convention.Source: §C.1 · p. 19, Table 8 · p. 20

C.2 INFERENCE AND PAIRED COMPARISONS

• Matched base and fine-tuned pairs use fixed evaluation examples, image preparation, prompts, decoding settings, and scoring, while additional public checkpoints retain model-specific protocols.The latter provide model-specific comparisons rather than an equal-inference-budget ranking.Source: §C.2 · pp. 19–20, Table 9 · p. 20

• The held-out DeskForge evaluation contains 105,703 instruction–target examples across four conditions.The reported internal mean weights the four conditions equally.Source: §C.3 · p. 20
• Grounding accuracy counts a prediction as correct when its point falls within the annotated visible region; invalid outputs count as incorrect.External benchmarks retain their respective target-region and scoring conventions.Source: §C.3 · p. 20
• External evaluation uses 1,581 ScreenSpot-Pro, 1,272 ScreenSpot-v2, 564 OSWorld-G, 5,790 UI-Vision, and 3,594 MMBench-GUI examples.Where reported, the equal-benchmark mean averages the five benchmark accuracies rather than pooling their examples.Source: §C.3 · p. 20
• Table 10 separates uniform human-assessment samples from enriched samples selected to cover additional strata.The caption identifies the assessment as covering element annotations and instructions.Source: Table 10 · p. 21

C.4 FIXED-PLANNER TASK EVALUATION

• A fixed Qwen3.6-27B planner selects action types, text or keys, and coordinate-free targets, while the action model supplies screen locations.Models receive screenshots, the task, and execution feedback, without accessibility trees or hidden application state; only action-model weights change within each comparison. WebArena-Infinity uses 119 tasks and OpenApps uses 100 tasks.Source: §C.4 · p. 21

D ANNOTATION AND INSTRUCTION QUALITY

• The human audit samples annotations and instructions, while its supplementary missing-element review is a coverage diagnostic rather than an independent recall estimate.The supplementary review found 30 additional elements on 29 previously inspected screens; the audits do not assess every retained state field.Source: D ANNOTATION AND INSTRUCTION QUALITY · p. 21

E.1 SCREENSPOT-V2 ACROSS INTERFACE DOMAINS

• Both evaluated models gain most on desktop interfaces and also improve on mobile and web interfaces in the ScreenSpot-v2 breakdown.Table 11 reports grounding accuracy by interface domain and gives each model’s full-benchmark score in the Overall column.Source: §E.1 · p. 21, Table 11 · p. 22

E.2 ROBUSTNESS TO SCENE COMPOSITION AND TARGET OCCLUSION

• Fine-tuning widens the gap to the base model as application count and target visible-area loss increase, while heavily occluded targets remain difficult for all models.The figure omits losses above 35%; UI-Venus-2-9B follows the base model’s decline.Source: §E.2 · p. 22

E.3 HELD-OUT GROUNDING FAILURES OF PUBLIC MODELS

• In checked held-out examples, fine-tuned Qwen3.5-4B hits targets that most or all of six public checkpoints miss, often clicking neighboring or similar controls instead.Figure 7 marks the annotated target and model clicks across examples at several display resolutions.Source: §E.3 · p. 22, Figure 7 · p. 23

E.4 DENSE ELEMENT DETECTION

• The GroundCUA detector comparison uses detector-specific confidence thresholds, and Figure 8 shows predictions on Blender, Shotcut, and OBS Studio, which are absent from DeskForge-1M.The displayed predictions recover most annotated controls, including toolbar buttons, menu entries, list items, and dock controls.Source: §E.4 · pp. 22, 24, Figure 8 · p. 24
• The DeskForge-fine-tuned detector scores highest on all three GroundCUA metrics, with group F1 exceeding OmniParser v2 by 3.62 points and ScreenParse by 4.58 points.Higher center-hit F1 and mean best IoU also indicate stronger element coverage and box alignment; Table 12 presents the cross-dataset comparison.Source: §E.4 · p. 22, Table 12 · p. 24

E.5 QUALITATIVE TASK TRAJECTORIES

• In the illustrated tasks, base-model clicks fail while fine-tuned runs succeed; errors include stalled controls, neighboring-item deletion, wrong-field entry, and adjacent Cancel clicks.The runs first differ at a click despite identical screenshots and planner target descriptions.Source: §E.5 · p. 25, Figure 9 · p. 26, Figure 10 · p. 27

F RESOURCE COMPARISON DEFINITIONS

• Table 1 compares specified dataset views and collection frameworks; its terms distinguish dense labels, multi-app scenes, configurable environments, visibility geometry, and recorded transitions.A cross means absence from the specified view, not from every component of the associated project; DeskForge counts element instances across observations.Source: F RESOURCE COMPARISON DEFINITIONS · pp. 25, 28

G PROMPT TEMPLATES

• The prompt appendix preserves instruction-construction templates and matched model-comparison templates, including the separate native UI-R1 format.Each model’s prompt is fixed across its base and fine-tuned checkpoint.Source: G PROMPT TEMPLATES · p. 28, §G.2 · pp. 30–31

G.1 INSTRUCTION SYNTHESIS AND REFINEMENT

• Instruction synthesis proposes coordinate-free semantic goals from before-state, marked-target, target-crop, and after-state images, with visual evidence guiding refinement.Variants share one intent and target, must be achievable in one interaction, and should be uniquely groundable from the unmarked before image.Source: §G.1 · pp. 28–30

G.2 GROUNDING PROMPTS

• Qwen, Gemma, and InternVL use a shared action-program template, while UI-R1 uses its native prompt; the action model returns the next action as code.Coordinates are screen fractions, and the template lists the available actions and termination convention.Source: §G.2 · pp. 30–31

G.3 PLANNER PROMPT

• The fixed planner describes targets without coordinates, leaving a separate grounder to locate them on screen.The planner uses screenshots, execution feedback, and retained interaction context, then checks visible outcomes before finishing.Source: §G.3 · pp. 31–32

H QUALITATIVE EXAMPLES

• The qualitative examples show varied observations, dense element annotations, ScreenTag serialization, and recorded interactions with initial screens, instructions, clicks, and successor screens.The serialization example marks occluded text and represents exposed regions as fragments; coordinates use a normalized 0–500 grid.Source: H QUALITATIVE EXAMPLES · p. 33, Figure 11 · p. 33, Figure 12 · p. 34