Source-linked AI summary
CUA-Skill: Develop Skills for Computer Using Agent
Tianyi Chen, Yinheng Li, Michael Solodko, Sen Wang, Nan Jiang, Tingyuan Cui, Junheng Hao, Jongwoo Ko, Sara Abdali, Leon Xu, Suzhen Zheng, Hao Fan, Pashmina Cameron, Justin Wagle, Kazuhito Koishida
TL;DR
Computer-using agents lack reusable procedural abstractions for reliable, scalable desktop interaction, especially across long-horizon tasks. CUA-Skill encodes human computer-use knowledge as parameterized skills and execution graphs, and CUA-Skill Agent dynamically retrieves, configures, and executes them. The framework reports 57.5% best-of-three success on WindowsAgentArena and consistent gains across evaluated LLM backbones.
Problem
Existing computer-using agents lack reusable and structured abstractions for human computer-use procedures, limiting scalability, generalization, and robustness on complex tasks.
Method
CUA-Skill encodes human computer-use knowledge as reusable parameterized skills with execution and composition graphs, while CUA-Skill Agent retrieves, configures, and executes skills dynamically.
Results
57.5% best-of-three success rate was achieved by CUA-Skill Agent on WindowsAgentArena, while skill augmentation improved performance across all evaluated LLM backbones.
Takeaways & Limitations
CUA-Skill provides a practical, model-agnostic foundation for scalable desktop agents and directly enables high-success executable trajectory generation.
Abstract
from arXiv · showhide
Computer-Using Agents (CUAs) aim to autonomously operate computer systems to complete real-world tasks. However, existing agentic systems remain difficult to scale and lag behind human performance. A key limitation is the absence of reusable and structured skill abstractions that capture how humans interact with graphical user interfaces and how to leverage these skills. We introduce CUA-Skill, a computer-using agentic skill base that encodes human computer-use knowledge as skills coupled with parameterized execution and composition graphs. CUA-Skill is a large-scale library of carefully engineered skills spanning common Windows applications, serving as a practical infrastructure and tool substrate for scalable, reliable agent development. Built upon this skill base, we construct CUA-Skill Agent, an end-to-end computer-using agent that supports dynamic skill retrieval, argument instantiation, and memory-aware failure recovery. Our results demonstrate that CUA-Skill substantially improves execution success rates and robustness on challenging end-to-end agent benchmarks, establishing a strong foundation for future computer-using agent development. On WindowsAgentArena, CUA-Skill Agent achieves state-of-the-art 57.5% (best of three) successful rate while being significantly more efficient than prior and concurrent approaches. The project page is available at https://microsoft.github.io/cua_skill/.
1. Introduction
CUA-Skill addresses the lack of reusable procedural abstractions in computer-using agents by encoding human computer-use knowledge as parameterized skills and execution graphs. Its associated agent dynamically retrieves and executes these skills, while evaluations report strong success rates.
- Motivation: Existing computer-using agents struggle with long-horizon desktop tasks because low-level action sequences compound grounding, planning, and execution errors.They also lack explicit representations of reusable human computer-use procedures.
- CUA-Skill: CUA-Skill encodes human computer-use knowledge as reusable, parameterized skills coupled with execution and composition graphs.The abstraction forms an intermediate layer between high-level user intent and low-level interaction primitives.
- CUA-Skill Agent: CUA-Skill Agent retrieves skills conditioned on UI state and user goals, re-ranks them using execution context and memory, configures arguments, and executes them through GUI or script paths.The design supports skill expansion, memory-aware failure recovery, and long-horizon task completion.
- CUA-Skill: The library contains hundreds of atomic skills spanning tens of applications, which parameterization and composition can instantiate into millions or more executable task variants.These skills are designed to transfer across tasks and UI states.
- Evaluation: 57.5% best-of-three success rate was achieved by CUA-Skill Agent on the end-to-end WindowsAgentArena CUA benchmark.CUA-Skill also achieved a 76.4% success rate in trajectory generation, outperforming existing approaches by 1.7×–3.6×.
2. Related Works
Related work addresses agent memory, structured planning, code-based composition, and standardized software interfaces. CUA-Skill targets a distinct procedural layer by encoding reusable interaction knowledge with parameterized execution semantics.
- Memory Modules and Knowledge Graph Integration: Memory modules retain execution history and intermediate outcomes to support planning and reuse information across long-horizon interactions.These methods primarily model task state, observations, and historical outcomes.
- Memory Modules and Knowledge Graph Integration: Existing memory and knowledge-graph approaches do not explicitly encode reusable human computer-interaction procedures with parameterized execution semantics.This limits systematic reuse across tasks, applications, and UI contexts.
- Structured Task Planning and MCPs: Structured planning and code-based approaches frame computer use as coordinated long-horizon action composition and modular compilation of high-level intents.These approaches emphasize structured planning rather than isolated tool calls.
- Structured Task Planning and MCPs: MCP standardizes agent access to external software, tools, and data through client–server interfaces, but complex desktop environments can require substantial API engineering.The engineering burden includes exposing application-specific APIs and maintaining underlying codebases.
- Relation to CUA-Skill: CUA-Skill complements planning and MCP-style interfaces by encoding human computer-use knowledge as reusable skills with parameterized execution and composition graphs.The paper states that this abstraction lowers the engineering burden for skill authors and agent developers.
3. Computer-Using Agentic Skills
CUA-Skill represents desktop computer use as reusable, parameterized skills organized into execution and composition graphs. These abstractions support varied realizations, argument validity, and multi-step workflows across applications.
- Skill abstraction: CUA-Skill models computer use as reusable, intent-aligned skills rather than flat sequences of primitive GUI actions.Each skill captures a minimal user intent and can be composed into higher-level workflows.
- Skill abstraction: Each skill combines an application, natural-language intent, argument pool, and parameterized execution graph.The graph realizes the intent through GUI interactions or executable actions conditioned on instantiated arguments.
- Argument domains: Feasible argument domains distinguish enumerable UI choices from open-ended inputs such as file paths, text, and numerical values.These domains constrain arguments to values for which the skill remains well-defined and executable.
- Execution graphs: Execution graphs encode multiple valid, parameterized interaction paths using GUI primitives or scripts, guarded branches, and optional edge preferences.Branches accommodate common UI variants and alternative interaction affordances without redefining the skill.
- Agent operation: The agent algorithm retrieves, reranks, configures, executes, and records outcomes for skills while traversing the library.The workflow uses generated queries, candidate retrieval, skill selection, argument configuration, execution, and memory updates.
- Skill composition: Composition graphs represent valid skill transitions, allowing paths to encode reusable multi-step workflows within and across applications.They capture ordering and compatibility constraints without prescribing a fixed execution plan.
4. CUA-Skill Agent
CUA-Skill Agent incrementally selects, configures, and executes reusable skills using current UI state, history, and memory. Its retrieve-augmented planner narrows candidates, supports fallback actions, and records outcomes for adaptive execution.
- Agent architecture: CUA-Skill Agent incrementally selects, configures, and executes skills conditioned on the current UI state, execution history, and accumulated memory.The architecture includes retrieval, reranking, argument configuration, memory, and execution components.
- Retrieve-augmented planning: The retrieve-augmented planner uses an LLM to select skills and configure arguments without exposing the full library in context.It conditions decisions on the current screen, execution history, and user goal.
- Skill retrieval: Hybrid lexical and semantic retrieval narrows the skill space to a small relevant candidate set, improving scalability and inference efficiency.Skills are indexed by functional descriptions for semantic retrieval and by text for lexical matching.
- Skill selection: The reranker evaluates retrieved skills using UI state, execution history, and argument compatibility, while retaining basic actions as a fallback.Low-level mouse and keyboard actions provide fine-grained control when high-level skills are insufficient.
- Skill execution: Argument instantiation conditions each selected skill on the current UI state, execution history, and user goal within feasible domains.The configured skill is then ready for execution through its parameterized graph.
- Skill execution: Execution traverses the skill graph to invoke GUI or script actions, using a grounding model for spatial UI interactions.The executor selects successor nodes during graph traversal and can use edge weights when preferences are provided.
- Memory and recovery: The memory buffer stores skill intents, effects, successes, and failures so the planner can avoid repeated mistakes and explore alternatives.Recorded outcomes provide a reflective view of progress and past decisions.
5. Numerical Experiment
The evaluation measures skill executability, synthesized trajectory generation, end-to-end agent performance, and ablations across applications and language-model backbones. CUA-Skill achieves reliable execution and improves performance across the evaluated settings.
- 5.1. Evaluation of Skills and Execution Graphs: 76.4% average execution success rate demonstrates reliable skills, with 3.75 GUI primitives per skill on average and at most 20 basic actions.The evaluation reports execution success rate and average primitive count as reliability and complexity measures.
- 5.1. Evaluation of Skills and Execution Graphs: Stable layouts and strong keyboard affordances yield higher success rates, while visually complex or media-heavy applications remain more challenging.Excel, Settings, and Bing Search are cited as easier cases; VLC and PowerPoint are cited as more difficult.
- 5.1. Evaluation of Skills and Execution Graphs: 1.7× higher success than UltraCUA and 3.6× higher than Operator shows that CUA-Skill generates stronger synthesized trajectories.The comparison uses synthesized user tasks and positions skill composition as a direct trajectory-generation application.
- 5.2. End-to-End Performance of the CUA-Skill Agent: The end-to-end evaluation tests autonomous skill retrieval, invocation timing, and argument configuration from natural-language instructions and current UI state.Unlike synthesized compositions, the agent must select and configure skills while failures may arise from retrieval, ranking, instantiation, or execution.
- 5.3. Ablation Study: Ablations examine planner choice, reasoning depth, and skill integration across language-model backbones.The supplied evaluation passages identify LLM planners and skill saliency as studied factors.
- 5.3. Ablation Study: 15.62% is the largest reported skill-integration gain, achieved with GPT-5; gains are also reported for Qwen3-VL-32B-Instruct and GPT-4o.The reported improvements are +5.23% for Qwen3-VL-32B-Instruct, +8.50% for GPT-4o, and +15.62% for GPT-5.
6. Conclusion
The conclusion presents CUA-Skill and CUA-Skill Agent as a skill-centric framework for reusable desktop computer-use knowledge. It reports transferability, high-success trajectory generation, and consistent gains across language-model backbones.
- CUA-Skill encodes human computer-use knowledge as reusable, parameterized skills with execution and composition graphs.
- The framework is described as transferable across tasks and applications and as enabling high-success executable trajectory generation.
- WindowsAgentArena evaluations show consistent performance gains across LLM backbones, supporting CUA-Skill as a model-agnostic foundation for scalable desktop agents.
B. More Related Works
Related work addresses GUI grounding and retrieval-augmented planning as central challenges in desktop-agent operation. The supplied examples illustrate query generation for launching an application.
- GUI Grounding in Desktop Agents: GUI grounding binds each natural-language step to an on-screen target, and grounding must be repeated after desktop state changes.
- GUI Grounding in Desktop Agents: Minor grounding errors can compound into multi-step failures in OS-agent evaluations.
- Retrieval-Augmented Planning: Retrieval-augmented planning revises the next action using environment feedback rather than committing to a complete plan upfront.
- Ensembled Query Generation Example: The Edge example generates multiple equivalent queries for opening the home page, including launching Edge through the icon or Windows menu.
D. Example of Synthesized Tasks by Skill Composition Graph
The synthesized-task examples show how skill composition graphs turn natural-language instructions into parameterized sequences of application-specific skills. Examples cover Excel spreadsheet editing and calculator operations.
- Synthesized User Task Example 1 upon Skill Composition Graph: An Excel task composes workbook opening, sheet renaming, formula insertion, and autofill skills with explicit file, sheet, cell, and formula arguments.
- Synthesized User Task Example 2 upon Skill Composition Graph: A calculator task composes launch, mode switching, number entry, subtraction, multiplication, square-root, and equals skills.
E. Case Study
Three Windows case studies show CUA-Skill completing timer creation, folder creation, and font-color changes through configured actions. Hotkeys reduce reliance on visual grounding, while grounding supplies coordinates when needed.
- ClockCreateTimer: The ClockCreateTimer skill successfully created a 25-minute timer named “Pomodoro Session.”Planner-configured text and grounding-model coordinates instantiate the timer actions.
- Argument Instantiation: Skill arguments are either configured by the planner or predicted by the grounding model, depending on the action.The timer example uses planner-configured text and grounding-based coordinates for clicks.
- FileExplorerCreateNewFolder: The FileExplorerCreateNewFolder skill successfully created and named a Logs folder inside Downloads.Hotkeys handled navigation and folder creation, with grounding needed only when coordinates were required.
- PowerPointSetTextFontColor: The PowerPointSetTextFontColor skill successfully set selected text to Light Blue.Ribbon navigation used hotkeys, while the Light Blue selection used a grounding-model coordinate.
F. CUA Execution Graph
The execution-graph examples depict multi-step computer-use tasks for Excel and Calculator. They illustrate how user instructions are represented as task execution graphs for application-specific actions.
- Excel: The Excel graph example opens betawacc.xlsx, renames Sheet1 to company analysis, and computes the Average column.The figure presents these operations as one CUA task execution graph.
- Calculator: The Calculator graph example represents the instruction “Calculate 398 −174 ×” as a CUA task execution graph.The caption identifies the application and the displayed calculation instruction.