PaperlayerView exact revision on arXiv

Source-linked AI summary

CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents

Xiangru Jian, Shravan Nayak, Kevin Qinghong Lin, Aarash Feizi, Kaixin Li, Patrice Bechard, Spandana Gella, Sai Rajeswar

arXiv:2603.24440v1cs.LGcs.AIcs.CV

TL;DR

Computer-use agents need richer training evidence because existing resources are limited by sparse screenshots, partial supervision, and insufficient continuous human demonstrations. CUA-SUITE addresses this gap by combining expert 30 fps desktop videos with dense grounding annotations and evaluation benchmarks. The resulting ecosystem preserves temporal interaction dynamics across 10,000 tasks and 87 applications while supporting evaluation and further research on computer-use intelligence.

  • Problem

    Computer-use agents lack high-quality human demonstration data with dense planning, grounding, and temporal supervision for complex desktop workflows.

  • Method

    CUA-SUITE combines VIDEOCUA’s continuous expert video and cursor traces with GROUNDCUA’s dense UI annotations and UI-VISION’s grounding and planning evaluation.

  • Results

    VIDEOCUA contains approximately 55 hours and 6 million frames of 30 fps recordings spanning 10,000 tasks and 87 applications, with synchronized interaction logs and reasoning annotations.

  • Takeaways & Limitations

    The multimodal corpus preserves intermediate visual feedback and temporal interaction dynamics that sparse screenshot datasets discard, supporting research on grounded computer-use agents.

Abstract

from arXiv · show

Computer-use agents (CUAs) hold great promise for automating complex desktop workflows, yet progress toward general-purpose agents is bottlenecked by the scarcity of continuous, high-quality human demonstration videos. Recent work emphasizes that continuous video, not sparse screenshots, is the critical missing ingredient for scaling these agents. However, the largest existing open dataset, ScaleCUA, contains only 2 million screenshots, equating to less than 20 hours of video. To address this bottleneck, we introduce CUA-Suite, a large-scale ecosystem of expert video demonstrations and dense annotations for professional desktop computer-use agents. At its core is VideoCUA, which provides approximately 10,000 human-demonstrated tasks across 87 diverse applications with continuous 30 fps screen recordings, kinematic cursor traces, and multi-layerfed reasoning annotations, totaling approximately 55 hours and 6 million frames of expert video. Unlike sparse datasets that capture only final click coordinates, these continuous video streams preserve the full temporal dynamics of human interaction, forming a superset of information that can be losslessly transformed into the formats required by existing agent frameworks. CUA-Suite further provides two complementary resources: UI-Vision, a rigorous benchmark for evaluating grounding and planning capabilities in CUAs, and GroundCUA, a large-scale grounding dataset with 56K annotated screenshots and over 3.6 million UI element annotations. Preliminary evaluation reveals that current foundation action models struggle substantially with professional desktop applications (~60% task failure rate). Beyond evaluation, CUA-Suite's rich multimodal corpus supports emerging research directions including generalist screen parsing, continuous spatial control, video-based reward modeling, and visual world models. All data and models are publicly released.

1 INTRODUCTION

CUA-SUITE addresses the scarcity of rich, human-curated data for computer-use agents by combining continuous expert video, dense annotations, and evaluation resources. Its central corpus, VIDEOCUA, captures professional desktop interaction at high temporal resolution across thousands of tasks and applications.

  • Motivation: Today’s computer-use agents remain brittle in professional desktop applications despite advances in vision-language and foundation models.They perform better on simple web tasks but struggle with tools such as 3D modeling software, IDEs, and specialized applications.
  • Motivation: Existing datasets often provide noisy synthetic data, partial supervision, or sparse screenshots that omit intermediate visual feedback between actions.The paper identifies a gap in training data covering both planning and grounding with rich temporal context.
  • Contributions: VIDEOCUA provides approximately 55 hours and 6 million frames of uncut 30 fps expert demonstrations across 10,000 tasks and 87 applications.The corpus also includes kinematic cursor traces and multi-layered reasoning annotations.
  • Contributions: CUA-SUITE unifies continuous expert video with pixel-precise grounding and rigorous evaluation through VIDEOCUA, GROUNDCUA, and UI-VISION.The unified ecosystem is designed to cover the full stack of computer-use intelligence.
  • Contributions: The authors open-source the benchmarks, training data, and models to accelerate research in computer-use agents.The release makes the ecosystem broadly available for further development and evaluation.

2 RELATED WORK

Prior computer-use datasets and benchmarks cover grounding, execution, or trajectory learning, but existing resources remain fragmented across environments and annotation granularity. In particular, no existing dataset simultaneously combines continuous 30 fps desktop video, human-curated trajectories, and rich multi-layered reasoning annotations at scale.

  • GUI Visual Grounding Datasets: Most visual-grounding datasets target mobile or web environments and rely on accessibility trees or HTML DOM representations.These representations can be noisy or incomplete and do not capture the pixel-level complexity of desktop applications.
  • GUI Visual Grounding Datasets: Desktop grounding benchmarks expose important gaps but cover narrow portions of the desktop ecosystem and often use semi-automated pipelines.Automated grounding methods can also produce misaligned bounding boxes, limiting their suitability as training data.
  • Action Prediction and Agent Benchmarks: Execution-based benchmarks evaluate sequential agent behavior across web, mobile, and desktop environments.Examples include MiniWoB++ and WebArena for web agents, AndroidWorld and AITW for mobile agents, and OSWorld for desktop agents.
  • Video-Centric and Trajectory Learning: Video-centric learning emphasizes continuous observation, while trajectory-synthesis efforts generate training data through annotation, tutorial replay, or reverse task synthesis.These approaches expand trajectory resources but do not by themselves resolve the need for dense frame-level grounding.
  • Video-Centric and Trajectory Learning: No existing dataset simultaneously provides continuous 30 fps video, desktop coverage, human-curated trajectories, and rich multi-layered reasoning annotations at scale.Existing video datasets such as VideoGUI provide high-level task descriptions but lack frame-level links between actions and specific UI elements.

3 CUA-SUITE

CUA-SUITE combines continuous expert desktop-use video, dense UI grounding, and evaluation resources to address gaps in training and assessing computer-use agents. Its preliminary evaluations show that current models remain unreliable on professional desktop applications, especially when precise visual grounding is required.

  • CUA-SUITE: CUA-SUITE unifies VIDEOCUA, GROUNDCUA, and UI-VISION into a platform for execution, fine-grained grounding, and visual perception and planning evaluation.The resources share a foundation of expert human behavior.
  • UI-VISION: 47.7% average accuracy is achieved by MAI-UI-32B on UI-VISION, while the Spatial split remains difficult despite gains in Basic and Functional categories.The cited evaluation reports that overall performance has nearly doubled since UI-VISION was introduced.
  • GROUNDCUA: 56K annotated screenshots and over 3.6 million element annotations provide dense, human-verified grounding coverage across 87 desktop applications.Annotations cover nearly every visible element, including small icons and controls, with fine-grained semantic types for half of the elements.
  • VIDEOCUA: Dense trajectory annotations average 496.7 words per step, combining observation, reasoning, action description, and reflection.The pipeline transforms annotated recordings into richly described state-action sequences for training vision-language agents.
  • Action prediction evaluation: 37.7% @50px is reached by OpenCUA-32B across 256 sampled tasks, while human-verified stepwise accuracy is 57.6%, showing persistent desktop action-prediction difficulty.The 32B model improves over 7B by 21.2 percentage points at @50px, but performance varies from 3.6% to 73.3% across applications.
  • Human evaluation: 52.4% grounding accuracy contrasts with 85.9% action correctness for the 32B model, indicating that intent is often identified while the target UI element is localized incorrectly.Non-coordinate steps reach 67.6% accuracy, whereas coordinate-based grounding remains the weaker component.

4 UNIVERSALITY AND APPLICATION POTENTIAL OF CUA-SUITE

CUA-SUITE is designed as a universal resource because its continuous videos, cursor trajectories, and dense annotations preserve information needed by current and future computer-use paradigms. Its annotations also support generalist screen parsing of desktop interfaces.

  • Continuous video streams, kinematic cursor trajectories, and expert-verified dense annotations form a superset of information for current computer-use paradigms.The data can be transformed into screenshot-action pairs, state-action-next-state triplets, or continuous kinematic traces without discarding recorded signal.
  • CUA-SUITE can be losslessly transformed into specialized formats used by leading agent frameworks.This preserves applicability as new training paradigms emerge without requiring data recollection.
  • Generalist Screen Parsing: Dense human-verified bounding boxes cover interactable desktop elements, including canvas-based and custom-drawn widgets missed by DOM-based approaches.Functional descriptions additionally support semantics-aware element captioning.

5 CONCLUSION

CUA-SUITE is presented as an ecosystem for training and evaluating desktop computer-use agents, centered on VideoCUA and complemented by GroundCUA and UI-Vision. Its corpus supports emerging agent paradigms, while its application selection emphasizes broad open-source desktop coverage.

  • VideoCUA provides approximately 55 hours of continuous 30 fps recordings across 10,000 tasks and 87 professional applications.The corpus includes kinematic cursor traces and multi-layered reasoning annotations.
  • CUA-SUITE combines continuous expert video, pixel-precise grounding, and rigorous evaluation in one ecosystem.The three resources are VideoCUA, GroundCUA, and UI-Vision.
  • CUA-SUITE supports generalist screen parsing, continuous spatial control, and visual world models beyond benchmark and training use.
  • The selected 87 platforms span 12 categories and focus on open-source software with permissive licenses.The selection is motivated by under-representation in existing datasets and release flexibility.

A.2 HUMAN ANNOTATION

Human annotation for CUA-SUITE was organized as a multi-phase professional curation process involving trained annotators, quality assurance specialists, and project managers. Annotators executed tasks, recorded screens, and densely labeled interface elements in extracted keyframes.

  • The annotation effort used a professional data-labeling vendor and spanned three phases beginning with a pilot study.The pilot refined task instructions through iterative feedback.
  • The annotation team included around 70 people across annotators, quality assurance specialists, and project managers.
  • Annotators trained on the platforms and guidelines, with each task requiring 60–90 minutes including quality checks.They were compensated hourly.
  • After screen recordings were collected, keyframes were annotated with bounding boxes and labels for all visible interface elements.Textual elements received their displayed text in addition to the element name.

B ADDITIONAL VIDEOCUA EVALUATION ANALYSIS

This appendix documents evaluation preprocessing and supplementary analyses for the action prediction experiments described in Section 3.4.

  • The appendix presents evaluation preprocessing details for the action prediction experiments.
  • It also includes supplementary analyses associated with those experiments.
  • The referenced action prediction experiments are described in Section 3.4.

B.1 EVALUATION PREPROCESSING

The preprocessing removes cursor-related leakage and redundant movement steps to create fairer, semantically meaningful evaluation trajectories.

  • Keyframes are captured midway between consecutive actions, before the cursor reaches its target, to reduce spatial-grounding information leakage.This timing provides a fairer evaluation of predicted target locations.
  • Redundant moveTo steps are excluded from evaluation and action history because they are preparatory movements rather than distinct interactions.
  • For moveTo→click pairs, the moveTo keyframe replaces the click keyframe because it better avoids capturing the cursor near the target.

B.2 APPLICATION-LEVEL ANALYSIS

Application-level performance varies widely across desktop software: specialized interfaces are hardest, while web-like layouts are relatively easier for the OpenCUA-32B model.

  • Performance ranges from under 5% to over 70% in @50px success rates across desktop applications.Table 6 reports applications with at least 3 coordinate predictions.
  • Challenging Applications: Darktable achieves 3.6% @50px success, while Krita, FreeCAD, QGIS, and Kodi also show low success rates in specialized interfaces.These applications contain dense toolbars, non-standard layouts, domain-specific iconography, or complex panels and overlays.
  • Relatively Successful Applications: OnlyOffice Spreadsheet reaches 73.3% and DuckDuckGo reaches 64.3% @50px success, illustrating stronger performance in web-like or standardized interfaces.Brave, LibreOffice Calc, and IntelliJ IDEA also exhibit relatively high success rates.

B.3 ERROR DISTANCE ANALYSIS

Prediction errors remain substantial, but the 32B model shifts some errors toward smaller distances; performance is also highly variable within individual tasks.

  • 31.0% of 32B and 33.1% of 7B predictions exceeding 50 pixels are more than 500 pixels from the target.Table 7 summarizes the distribution of high-distance predictions.
  • The 32B model places 20.2% of high-distance predictions within 50–100px, compared with 13.7% for the 7B model.This indicates a shift toward smaller errors rather than only eliminating errors.
  • Within-Task Variability: 128 of 256 evaluated tasks contain both predictions within 20 pixels and predictions exceeding 500 pixels.Thus, the model can be accurate for some UI elements and far off-target for others within the same task.

C APPLICATION POTENTIAL OF CUA-SUITE

The appendix describes four application frontiers supported by CUA-SUITE and provides detailed descriptions of each use case.

  • CUA-SUITE’s application potential is organized around four application frontiers introduced earlier in the paper.

C.1 ADVANCING GENERALIST SCREEN PARSING

CUA-SUITE supports generalist screen parsing by providing dense, human-verified supervision for desktop interfaces. Its annotations cover both interactable-element localization and functional semantics.

  • Current multimodal models struggle to identify interactable regions and understand icon semantics from screenshots.
  • CUA-SUITE provides dense supervision for training desktop-centric screen parsers.
  • Human-verified annotations cover all interactable screen elements, including canvas-based and custom-drawn widgets missed by DOM-based datasets.
  • Functional descriptions for icons and widgets add a semantics-aware layer beyond localization.

C.2 LEARNING CONTINUOUS SPATIAL CONTROL

CUA-SUITE enables continuous spatial control by preserving cursor trajectories rather than only final click coordinates. These trajectories support imitation and offline reinforcement learning for feedback-driven navigation.

  • Discrete coordinate prediction is insufficient for high-precision GUI interaction, motivating continuous cursor movement from visual feedback.
  • CUA-SUITE records intermediate cursor movements and complete video context, preserving human kinematic priors such as deceleration near targets.
  • Continuous trajectory data supports imitation learning and offline reinforcement learning policies that mimic human navigation patterns.
  • These policies can perform iterative, feedback-driven navigation and improve robustness to spatial hallucinations in zero-shot vision-language models.

C.3 ENABLING VISUAL WORLD MODELS

CUA-SUITE supports visual world models by pairing fine-grained actions with high-fidelity temporal screen feedback. This enables action-conditioned interface simulation for visual lookahead planning.

  • World-model research is limited by datasets that lack fine-grained actions paired with high-fidelity future frames.
  • The recordings capture immediate visual consequences of user actions, including button presses and menu openings.
  • Complete 30 fps videos provide the (s_t, a_t, s_t+1) triplets needed for video prediction models.
  • Verified annotations can condition generative video models to simulate complex software interface dynamics.
  • Simulated outcomes support visual lookahead planning before actions are executed in mission-critical workflows.

C.4 VIDEO-BASED REWARD MODELING

CUA-SUITE provides continuous expert demonstrations and step-level annotations for video-based reward modeling. Its temporal richness preserves subtle UI feedback needed to assess task completion.

  • ExeVRM evaluates task completion from execution videos and achieves 84.7% accuracy and 87.7% recall on ExeVR-Bench.
  • CUA-SUITE supplies approximately 10,000 successful expert task demonstrations across 87 professional applications for training video-based reward models.
  • Per-step observations, thought chains, action descriptions, and reflections enable fine-grained reward signals beyond binary task labels.
  • Continuous recordings preserve loading indicators, hover effects, and transient feedback that sparse screenshots omit.
  • This temporal detail supports identifying decisive UI changes in long, high-resolution execution videos.

D VIDEOCUA TRAJECTORY EXAMPLES

The VIDEOCUA examples present complete Chain-of-Thought annotated trajectories across application domains, exposing the agent’s observations, reasoning, actions, and post-action reflections at each step.

  • Each trajectory step pairs the observed screenshot with screen-state descriptions, reasoning, executable pyautogui code, and post-action reflection.Coordinate-based actions additionally mark target locations on screenshots with colored crosshairs.

D.1 KRITA (DIGITAL ART)

The Krita trajectory demonstrates a multi-step desktop workflow: selecting an ellipse tool, drawing and filling a circle, then configuring an Inner Shadow with Multiply blending. Reflections distinguish successful state changes from ineffective actions and note when settings remain unapplied.

  • Task and setup: The task requires selecting the ellipse tool, drawing a circle, filling it green, and adding an Inner Shadow with Multiply blending.
  • Tool selection: Hovering over the ellipse tool produced no interface change, so the trajectory identifies clicking as necessary for activation.
  • Shape creation: The completed drag finalized a clean ellipse shape, establishing the first required element before filling and applying effects.
  • Filling: Releasing the mouse finalized the circle and produced a solid green result, allowing the workflow to proceed to effects.The recorded file size increased from 35.7 MiB to 43.3 MiB after the action.
  • Layer effects: Checking Inner Shadow with Multiply enabled the required effect on the green circle layer.
  • Completion state: The trajectory review lists the circle, green fill, opened Layer Styles dialog, enabled Inner Shadow, and Multiply blend mode as configured, but notes the dialog changes were not yet applied.

D.2 GIMP (IMAGE EDITING)

The GIMP demonstration constructs a warning-style logo through selections, fills, and text-tool activation, while exposing several execution errors and corrections. The trajectory shows how visual state changes verify whether each desktop action achieved its intended effect.

  • Task: The task is to create two geometric shapes in a warning style with the text “new logo”.The workflow calls for shape creation, warning-style coloring, and adding the specified text.
  • Shape construction: The Rectangle Select tool is activated as the first required tool for creating the logo’s geometric shapes.Its options panel identifies Rectangle Select and includes settings such as antialiasing, feather edges, and rounded corners.
  • Shape construction: 125 × 137 pixels marks the initial rectangular selection created on the canvas for the first geometric shape.The selection appears at position 818, 363, with its dimensions shown in the status bar.
  • Execution errors: The attempted drag incorrectly caused the rectangular selection to disappear instead of expanding it.The drag coordinates may have been too far from the original selection’s starting point.
  • Execution errors: The second selection became a 20 x 138-pixel strip, which was judged too narrow to serve as a meaningful logo shape.Although a new selection was created, it did not meet the intended geometric-design goal.
  • Filling: The larger rectangle was successfully filled with a yellow-and-black diagonal warning pattern, completing the first geometric shape.The smaller rectangle remained unfilled at that point.
  • Text entry: A mistaken coordinate first failed to activate the Text tool, but a later action successfully activated it and displayed text-specific controls.The failed coordinate targeted the wrong toolbox area; successful activation exposed Font, Size, and formatting options.
Loading 2603.24440v1…