CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

Introduction

• Current CUAs remain brittle on professional desktop applications, including 3D modeling software, IDEs, and specialized tools, and struggle with unfamiliar interfaces in popular non-mainstream applications.The passage contrasts these difficulties with stronger performance on simple web tasks.Source: §1 · pp. 1–2
• Human datasets often lack temporal context, while OpenCUA uses sparse screenshots and ScaleCUA (Liu et al., 2026) has 2 million screenshots, under 20 hours at 30 fps.The introduction argues that continuous video, kinematic action traces, and precise UI grounding are needed to capture temporal dynamics and support continuous spatial control.Source: §1 · p. 2
• VideoCUA provides approximately 55 hours and 6 million frames of uncut 30 fps expert demonstrations across over 10,000 tasks and 87 applications, with cursor traces and reasoning annotations.Continuous recordings preserve interaction dynamics beyond final click coordinates and support comparisons of video-based and screenshot-based training.Source: §1 · pp. 2–3
• Unified dense, causal supervision can support foundation action models, visual world models for lookahead planning, and continuous spatial control policies.The paper describes every on-screen element as labeled and every action as logged.Source: §1 · p. 3
• CUA-Suite unifies VideoCUA, GroundCUA’s pixel-precise UI annotations, and UI-Vision’s grounding and planning benchmark into a full-stack ecosystem.The overview describes human GUI trajectories as expert-verified and annotated with keyframes, bounding boxes, and interaction logs.Source: §1 · pp. 2–3, Figure 1 · p. 2
• The authors release all benchmarks, training data, and models as open source to accelerate computer-use-agent research.Source: §1 · p. 3

GUI Visual Grounding Datasets

• Desktop grounding remains difficult: accessibility-tree and synthetic methods can misalign boxes, while existing benchmarks cover narrow slices and provide limited training data.Most grounding datasets target mobile or web interfaces; accessibility trees may be noisy or incomplete and miss desktop pixel-level complexity.Source: §2 · p. 3

Action Prediction and Agent Benchmarks

• Execution benchmarks provide task feedback but often lack dense offline supervision, while current GUI agents remain trained on static screenshot-action pairs.The cited testbeds span web, mobile, and desktop; screenshot-action training limits architectures’ understanding of temporal dynamics.Source: §2 · p. 3

Video-Centric and Trajectory Learning

• Video and trajectory efforts broaden training data, but existing annotations lack frame-level action-to-element grounding, and no dataset combines the identified video, desktop, human-curation, and reasoning properties at scale.Efforts include instructional videos, mined and replayed tutorials, and reverse task synthesis from observed interactions.Source: §2 · pp. 3–4

3.1 CURATION OF CUA-SUITE

• Experts designed over 10,000 real-work tasks across 87 open-source applications, yielding 55 hours and 6 million frames of continuous 30 fps video with synchronized interaction traces.Keyframes immediately before state-changing actions receive manual bounding boxes and text labels; about 50% of elements are assigned one of eight functional categories.Source: §3.1 · p. 4

A Unified Foundation. This robust data engine serves as the single source of truth for the entire CUA-

• Three resources share a foundation of expert human behavior: VIDEOCUA supports complex workflow execution, GROUNDCUA fine-grained grounding, and UI-VISION perception and planning evaluation.The shared data engine is intended to support research beyond the current scope and help advance generalist computer-use agents.Source: A Unified Foundation. This robust data… · p. 5

3.2 UI-VISION

• MAI-UI-32B reaches 47.7% average accuracy on UI-VISION, a 450-demonstration benchmark testing element grounding, layout grounding, and next-action prediction.Top models approach 60% on Basic and Functional categories, while Spatial remains difficult; OpenCUA improves by 7.6 absolute points from 7B to 72B, and an o3 planner raises PhiGround-7B from 27.2% to 36.2%.Source: §3.2 · pp. 5–6, Table 1 · p. 5

3.3 GROUNDCUA

• GROUNDCUA contains 56K annotated screenshots and over 3.6 million UI-element annotations across 87 desktop applications, with high-resolution coverage and human-verified labels.Its annotations produce a 700K instruction-tuning dataset; GROUNDNEXT-3B scores 50.6 on OS-World Verified when paired with the o3 planner.Source: §3.3 · p. 6

3.4 VIDEOCUA

• VIDEOCUA provides approximately 10,000 tasks across 87 professional desktop applications in 55 hours of continuous 30 fps video, more than 2.5× ScaleCUA’s approximately 2 million screenshots.Each trajectory step can be enriched with screen observations, reasoning, natural-language action descriptions, executable code, and reflections; synthesized annotations average 496.7 words per step.Source: §3.4 · p. 6, Table 2 · p. 7, §3.4.1 · p. 7

3.4.2 ACTION PREDICTION EVALUATION

• The action-prediction evaluation tests next actions from the task instruction, current screenshot, and five ground-truth actions, using coordinate-based metrics and human judgments for broader correctness.The automated evaluation covers 1,999 coordinate predictions from 256 tasks; human evaluation samples 49 tasks and 576 steps to assess grounding and action correctness.Source: §3.4.2 · p. 8, §3.4.4 · p. 9

3.4.5 IMPLICATIONS FOR FOUNDATION ACTION MODELS

• 37.7% @50px and 57.6% human-verified stepwise accuracy show that current foundation action models still struggle with professional desktop applications.The 32B model’s human-verified action correctness was 85.9%, versus 52.4% grounding accuracy on coordinate-based steps; performance varied widely across applications.Source: §3.4.3 · p. 8, §3.4.5 · p. 9, §3.4.4 · p. 9

Universality

• CUA-SUITE losslessly converts its continuous videos, cursor trajectories, and expert annotations into formats used by leading agent frameworks without discarding recorded signals.This lets the data remain applicable to new training paradigms without recollection.Source: §4 · p. 10

Generalist Screen Parsing

• Dense, human-verified boxes and functional descriptions support semantics-aware parsing of interactable elements, including canvas-based and custom-drawn widgets missed by DOM-based approaches.MLLMs struggle to parse desktop screenshots reliably (Lu et al., 2024b); ScreenParse has explored GroundCUA data.Source: §4 · p. 10

Continuous Spatial Control

• Intermediate cursor movements and full video context preserve kinematic priors for training feedback-driven imitation-learning or offline-RL navigation policies.GUI-Cursor argues discrete coordinate prediction is insufficient for high-precision tasks; CUA-SUITE captures movement patterns such as Fitts’s Law deceleration.Source: §4 · p. 10

Visual World Models

• Thirty-fps recordings paired with timestamped actions provide dense (s_t, a_t, s_t+1) pairs for action-conditioned video generation and visual lookahead planning.This supports world models based on textual planning (Gu et al., 2024), visual generation (Luo et al., 2025; Rivard et al., 2025; Guan et al., 2026), or code rendering (Zheng et al., 2026).Source: §4 · p. 10

Video-Based Reward Modeling

• Continuous expert videos with task-level annotations provide positive demonstrations for video-based reward models, while step-level annotations enable fine-grained rewards.ExeVRM (Song et al., 2026) demonstrates that assessing completion from execution video can outperform proprietary models.Source: §4 · p. 10

5 CONCLUSION

• Spatial grounding remains the primary bottleneck for current foundation action models, whose performance varies widely across desktop application domains.The conclusion reports this as the central finding of the evaluations.Source: §5 · p. 10
• VideoCUA provides approximately 55 hours of continuous 30 fps recordings across 10,000 tasks and 87 professional applications, with cursor traces and layered reasoning annotations.It is described as the largest open expert video corpus for desktop computer use.Source: §5 · p. 10
• CUA-Suite combines VideoCUA with pixel-precise GroundCUA grounding and rigorous UI-Vision evaluation in one ecosystem.Source: §5 · p. 10
• Its continuous videos, kinematic traces, and dense annotations support research in generalist screen parsing, continuous spatial control, and visual world models.Source: §5 · p. 10
• The authors will release all data, benchmarks, and models to the community and propose CUA-Suite as a foundation for general-purpose computer-use agents.Source: §5 · p. 10

A.1 PLATFORMS

• The platform selection covers 87 platforms across 12 categories, emphasizing open-source software with permissive licenses because such platforms are under-represented in existing datasets and enable less-restricted release.Source: §A.1 · p. 17
• The authors argue that choosing open-source platforms does not compromise generalization because they share UI elements and layouts with closed-source counterparts, citing Feizi et al., 2025.LibreOffice and Microsoft Office Suite are given as an example of shared interface elements, layout, and functionality.Source: §A.1 · p. 17

A.2 HUMAN ANNOTATION

• Around 70 experienced annotators, quality specialists, and project managers worked with a professional data-labeling vendor to curate the dataset.The annotators held at least a bachelor’s degree in technical fields, had prior data-labeling and user-interface research experience, and were mostly based in India and Latin America.Source: §A.2 · p. 17
• After training, annotators created and executed tasks across 87 software applications, with each task taking 60–90 minutes including quality checks.Annotators were compensated hourly.Source: §A.2 · pp. 17–18
• Annotators drew bounding boxes around visible interface elements in keyframes and labeled each with its name, adding text for textual elements.Long text segments received concise summaries, supplemented with PaddleOCR extraction of full text when available; 50% element was assigned one of eight high-level categories.Source: §A.2 · p. 18
• Quality specialists reviewed annotations, authors cross-checked them, and custom evaluation scripts validated them.The passage describes this multi-stage quality assurance process as supporting a dataset of grounded user-interface interactions with high diversity and reliable annotation quality.Source: §A.2 · p. 18

• The appendix presents evaluation preprocessing details and supplementary analyses for the Section 3.4 action prediction experiments.Source: B ADDITIONAL VIDEOCUA EVALUATION… · p. 18

B.1 EVALUATION PREPROCESSING

• Keyframes are captured midway between consecutive actions so the cursor has not yet reached the target, preventing ground-truth position leakage during spatial-grounding evaluation.For an action at timestamp τt, the keyframe is captured at (τt−1 + τt)/2.Source: §B.1 · p. 18
• Preparatory moveTo steps are excluded from evaluation and action history, and a following click uses the moveTo keyframe to avoid capturing the cursor near its target.moveTo→click pairs targeting the same coordinates are treated as a single semantically distinct interaction.Source: §B.1 · p. 18

B.2 APPLICATION-LEVEL ANALYSIS

• Per-application @50px success rates range from under 5% to over 70%, reflecting the diversity of desktop application interfaces.Table 6 reports OpenCUA-32B results for applications with at least 3 coordinate predictions.Source: §B.2 · p. 18, Table 6 · p. 19

Challenging Applications

• Darktable (3.6% @50px), Krita (12.9%), Kodi (14.6%), FreeCAD (22.6%), and QGIS (22.6%) are among the challenging applications.Creative tools have dense toolbars, non-standard layouts, and domain-specific icons; FreeCAD and QGIS have complex multi-panel interfaces, while Kodi has overlay controls and dynamic content.Source: §B.2 · p. 19

Relatively Successful Applications

• OnlyOffice Spreadsheet (73.3%), LibreOffice Calc (66.7%), IntelliJ IDEA (66.7%), DuckDuckGo (64.3%), and Brave (61.9%) are relatively successful.These applications typically have standard toolbars and clear visual hierarchies, which may better align with the models’ training distribution.Source: §B.2 · p. 19

Distance Distribution

• The 32B model produces 1,245 predictions over 50 pixels, compared with 1,624 for the 7B model.These are total high-distance prediction counts.Source: §B.3 · p. 19

Within-Task Variability

• 128 of 256 evaluated tasks contain both predictions within 20 pixels of target and predictions exceeding 500 pixels.This indicates that models can accurately locate some UI elements in a task while predicting coordinates far from the target for others, rather than performing consistently across all steps.Source: §B.3 · pp. 19–20

Distance Distribution

• 20.2% of 32B versus 13.7% of 7B predictions over 50 pixels fall 50–100px from target; 31.0% of 32B versus 33.1% of 7B exceed 500 pixels.The authors interpret the distribution as evidence that scaling shifts errors toward smaller distances rather than solely eliminating errors.Source: §B.3 · p. 19, Table 7 · p. 20

• The appendix expands on Section 4’s four application frontiers with detailed descriptions of how CUA-Suite supports each use case.Source: C APPLICATION POTENTIAL OF CUA-SUITE · p. 20

C.1 ADVANCING GENERALIST SCREEN PARSING

• Current MLLMs struggle to identify interactable regions and understand icon semantics without external modules, as highlighted by OmniParser (Lu et al., 2024b).Source: §C.1 · p. 20
• CUA-Suite provides supervision to train robust, desktop-centric screen parsers.Its annotations support detection and captioning components.Source: §C.1 · p. 20
• Dense human-verified annotations cover interactable elements, while functional captions distinguish visually similar controls for parser training.The annotations cover canvas-based and custom-drawn widgets often missed by DOM-based web datasets.Source: §C.1 · p. 20

C.2 LEARNING CONTINUOUS SPATIAL CONTROL

• GUI-Cursor argues discrete coordinate prediction is insufficient for high-precision GUI tasks and proposes learning continuous cursor movements from visual feedback.Training such policies requires dense kinematic trajectories, which most step-by-step navigation benchmarks lack.Source: §C.2 · p. 20
• Unlike datasets recording only final click coordinates, CUA-Suite logs intermediate MOVE TO cursor movements and complete video context, preserving human kinematic priors such as deceleration near targets.The passage identifies this deceleration pattern with Fitts’s Law.Source: §C.2 · p. 20
• These continuous data support imitation-learning or offline-RL policies that mimic human navigation and enable GUI-Cursor-style iterative, feedback-driven movement to improve robustness to zero-shot VLM spatial hallucinations.The passage presents these training capabilities as enabled by CUA-Suite’s data.Source: §C.2 · p. 21

C.3 ENABLING VISUAL WORLD MODELS

• Sparse screenshots omit immediate visual feedback, leaving datasets without the fine-grained action and high-fidelity future-frame pairs needed for world models (Luo et al., 2025).The passage cites ViMo as an example of recent work facing this limitation.Source: §C.3 · p. 21
• 30 fps recordings provide dense (st, at, st+1) triplets that capture the immediate visual consequences of user actions for video-prediction training.Source: §C.3 · p. 21
• Verified annotations can condition video generation to simulate complex software dynamics, enabling visual lookahead planning before clicks to help prevent errors in mission-critical workflows.Source: §C.3 · p. 21

C.4 VIDEO-BASED REWARD MODELING

• ExeVRM (Song et al., 2026) reports 84.7% accuracy and 87.7% recall on ExeVR-Bench across Ubuntu, macOS, Windows, and Android, outperforming proprietary models.It evaluates task completion directly from execution-video keyframes using vision-language models.Source: §C.4 · p. 21
• Approximately 10,000 expert task demonstrations across 87 professional applications provide continuous 30 fps videos paired with task instructions as positive training examples.These demonstrations capture successful task completions.Source: §C.4 · p. 21
• Per-step observations, thought chains, action descriptions, and reflections enable fine-grained reward signals beyond binary task-level labels.The annotations provide granularity for synthesizing hard negatives through adversarial instruction translation, as used by ExeVRM.Source: §C.4 · p. 21
• Continuous recordings preserve subtle intermediate states, including loading indicators, hover effects, and transient feedback, that sparse-screenshot datasets omit.This temporal richness supports assessment of decisive UI changes in long, high-resolution videos.Source: §C.4 · p. 21

D VIDEOCUA TRAJECTORY EXAMPLES

• Two representative VideoCUA tasks from different application domains are presented as complete CoT-annotated trajectories.Each step includes the screenshot, generated screen-state observation, reasoning chain, executed pyautogui action, and post-action reflection.Source: D VIDEOCUA TRAJECTORY EXAMPLES · p. 22
• Coordinate-based actions are marked on screenshots with colored crosshairs indicating the action type.Red denotes clicks, blue cursor moves, green drags, orange right-clicks, and purple mouse-down events.Source: D VIDEOCUA TRAJECTORY EXAMPLES · p. 22

D.1 KRITA (DIGITAL ART)

• Moving the mouse alone did not select the ellipse tool; clicking it visibly activated the tool and enabled drawing.Source: §D.1 · pp. 23–24
• The ellipse drag produced and finalized a complete circle with a green stroke, but its interior remained unfilled.The task instruction also called for a green fill and an inner shadow using Multiply blending mode; neither is shown as completed here.Source: §D.1 · pp. 22, 27, 28
• Attempts to select the fill bucket at the toolbar’s extreme left edge failed, leaving the circle as a green outline.The reflection attributes the failed click to coordinates outside the tool buttons.Source: §D.1 · pp. 29–30
• The cursor was moved inside the circle to prepare for filling, but this part ends without showing the fill or inner shadow applied.Source: §D.1 · p. 31
• The fill bucket changed the circle from a green outline to a solid green fill, completing the task’s fill step.Source: §D.1 · p. 32
• Right-clicking Paint Layer 1 exposed Layer Style, and choosing it opened the Layer Styles dialog with Inner Shadow among the available effects.Source: §D.1 · pp. 34, 36
• Inner Shadow settings showed Blend Mode set to Multiply, with opacity at 75%, angle at 120°, and distance and size both at 21 px.Source: §D.1 · p. 39
• The Inner Shadow checkbox was checked, enabling the effect with the required Multiply blend mode for the green circle.The task progress also records the circle as drawn and filled green, with the effect selected and enabled.Source: §D.1 · pp. 41–42
• The configured effect had not yet been applied to the layer; confirming with OK was still required to commit the settings and close the dialog.The passages describe clicking OK as the next step, not as an action already completed.Source: §D.1 · pp. 42–43

D.2 GIMP (IMAGE EDITING)

• The initial cursor move targeted the ellipse-selection icon instead of the rectangle tool, but the Rectangle Select tool was later activated successfully.The tool options changed from Move to Rectangle Select, confirming activation.Source: §D.2 · pp. 45–46
• A 125 × 137-pixel rectangular selection was started, but the attempt to enlarge it failed and left an irregular L-shaped selection unsuitable for the logo.The selection was later finalized, but it remained an L shape rather than a clean geometric element.Source: §D.2 · pp. 48–50
• The attempt to open GIMP’s Select menu to clear the irregular selection failed, leaving the selection unchanged and prompting a shift back to canvas-based actions.The menu did not open after the mouse-down action, so the agent proposed dragging on the canvas instead.Source: §D.2 · pp. 52–53
• The initial selection was only 20 × 138 pixels, and the later attempt to select the already-active Rectangle Select tool made no progress toward a usable logo shape.The agent recognized that the narrow selection was unsuitable and that selection tools create selections rather than filled shape objects.Source: §D.2 · pp. 53, 55, 56
• After switching to Bucket Fill with the Warning pattern selected, the agent filled two rectangular shapes with diagonal yellow-and-black stripes.The larger rectangle was filled first; the later canvas state shows both rectangles patterned.Source: §D.2 · pp. 56, 59, 60
• The first attempt to reach the Text tool targeted the wrong toolbox area, but a later action successfully activated the tool.The tool options showed text settings, while the passages still identify adding “new logo” as the remaining task.Source: §D.2 · pp. 60–62
• The Text tool’s I-beam cursor was successfully positioned below the warning-patterned rectangles for text entry.The cursor was placed in the lower canvas area, complementing the shapes above.Source: §D.2 · pp. 62–63
• A click created an empty text box below the shapes and activated GIMP’s text-editing mode.A floating formatting toolbar appeared with Sans-serif font and size 30 settings.Source: §D.2 · p. 63
• The attempted text-box resize produced no visible change; releasing the mouse finalized the drag state, after which two text boxes were visible.The text box remained below the warning-patterned rectangles; the resize attempt was unsuccessful.Source: §D.2 · pp. 64–65
• The recorded typing step issued `typewrite("NEWLOGO")`.The preceding workflow notes described the active text box as ready for input.Source: §D.2 · pp. 65–66