PaperlayerView exact revision on arXiv

Source-linked AI summary

Work2Skill: Capturing Human GUI Experience for Reusable Agent Skill Induction

Yuqi Huang, Hui Li, Yifei Cao, Zikun Cai, Tianci Xie, Yangfan Gao, Pinxue Guo, Chaoyun Zhang, Bingxuan Li, Rui Zheng, Hang Yan, Ning Liao

arXiv:2610.12984v1cs.AIcs.HC

TL;DR

Computer-use agents still struggle with unfamiliar software and workflows that require domain knowledge, while human demonstrations offer procedural experience that must be interpreted in context. Work2Skill aligns and organizes that experience, then uses on-demand evidence inspection to induce reusable skills. Across four execution models, it achieves the highest scores and success rates among evaluated methods, with mean score gains over No Skill on both benchmarks.

  • Problem

    Agents struggle with unfamiliar software and workflows requiring domain knowledge because user requests may leave procedural details unspecified.

    Source: §1 · p. 1

  • Method

    Work2Skill aligns human GUI actions with visual and available accessibility evidence, organizes demonstrations in WorkDemo-350, and interleaves on-demand evidence inspection with skill writing and revision.

    Source: §1 · p. 2

  • Results

    Across four execution models, Work2Skill achieves the highest scores and success rates among evaluated methods, with mean score gains over No Skill of 4.67–10.54 points on MyPCBench and 3.11–7.50 points on the evaluated OSWorld-Verified subset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The resulting skills preserve workflow rules, step dependencies, and completion checks while guiding execution without access to the source recording.

    Source: §1 · p. 2

Abstract

from arXiv · show

Computer-use agents can perform increasingly complex GUI tasks, yet unfamiliar software and workflows requiring human domain knowledge remain challenging. Human demonstrations convey procedural knowledge through the same GUI interaction space, but reuse requires interpreting actions in context and separating workflow rules from task-specific details. We thus introduce Work2Skill, a framework that captures human GUI experience for reusable agent skill induction. WorkRecorder provides the framework's infrastructure for collecting this experience, aligning screen recordings, mouse and keyboard events, and available accessibility-tree information. Using WorkRecorder, we construct WorkDemo-350, the first synchronized multimodal dataset of human desktop GUI demonstrations for reusable agent skill induction and downstream evaluation. Its 350 demonstrations are screened for completeness and task success. We then propose an agentic skill-induction method that inspects evidence on demand to produce reusable skills capturing workflow rules, step dependencies, and completion checks. When the same four execution models are equipped with skills induced by each method under shared execution settings, Work2Skill achieves the highest scores and success rates. Mean score improvements over the corresponding No Skill baseline are 4.67--10.54 points on MyPCBench Bounded Action and 3.11--7.50 points on a 286-task subset of OSWorld-Verified. Our code and data are available at https://github.com/china-qijizhifeng/Work2Skill.

1 INTRODUCTION

Work2Skill addresses the need for procedural knowledge in unfamiliar GUI workflows by capturing human demonstrations and turning them into reusable skills. Its framework combines aligned experience collection, a shared demonstration resource, and on-demand evidence inspection; across four execution models, it achieves the highest scores and success rates among evaluated methods.

1 INTRODUCTION

  • Human demonstrations can convey procedural knowledge through the shared GUI interaction space, but task requests may omit domain-specific procedures such as portal choice, required attachments, or submission checks.The paper frames this as a challenge in unfamiliar software and workflows requiring domain knowledge.Source: §1 · p. 1
  • WorkRecorder aligns screen recordings, mouse and keyboard events, and available accessibility information so agents can interpret actions using visual context and structured UI evidence.Recorded element names and roles provide control semantics while linked video preserves surrounding state changes.Source: §1 · p. 2
  • WorkDemo-350 provides 350 screened demonstrations across 64 MyPCBench Bounded Action tasks and a 286-task OSWorld-Verified subset for comparisons under shared task evaluators and execution conditions.Episodes include synchronized visual and action evidence plus available accessibility-tree information.Source: §1 · p. 2
  • The agentic induction method interleaves on-demand evidence inspection with skill writing and revision, preserving workflow rules, step dependencies, and completion checks while generalizing demonstration-specific inputs.Skills guide execution without access to the source recording.Source: §1 · p. 2
  • Across four execution models, Work2Skill achieves the highest scores and success rates among evaluated methods, improving mean scores over No Skill by 4.67–10.54 points on MyPCBench and 3.11–7.50 points on the evaluated OSWorld-Verified subset.The evaluation uses official evaluators and a shared text-guidance interface.Source: §1 · p. 2

2 RELATED WORK

Prior work uses human demonstrations to support GUI agents and represents procedures in several forms. Related research also studies skill induction, experience retention, and selective evidence inspection.

Learning from Human GUI Demonstrations

  • Human demonstrations support GUI-agent data collection, grounding, training, and efficiency evaluation across desktop, web, and mobile settings.Procedural representations include expert instructions, task graphs, parameterized workflows, semantic traces, tutorials, skill graphs, and SOP skills.Source: §2 · pp. 2–3

Skill Induction and Active Evidence Acquisition

  • Skill-induction research distills procedures from interaction or user traces, while adaptive memories and contextual playbooks retain experience for later use.Experience-based evaluation examines downstream value.Source: §2 · p. 3
  • Agentic video methods iteratively retrieve relevant segments, while GUI systems include trace-and-screenshot diagnosis and selective inspection of procedures, state cards, and keyframes.These approaches address choosing which demonstration evidence to inspect.Source: §2 · p. 3

3 WORK2SKILL FRAMEWORK

Work2Skill connects captured human GUI experience to reusable procedural guidance through WorkRecorder, WorkDemo-350, and an agentic skill-induction process.

3.1 WORKRECORDER

  • WorkRecorder aligns timestamped input events and visual observations, linking actions to keyframes and available target information so an agent can inspect operations in context.Accessibility snapshots provide interface context but need not reflect the exact action-time state.Source: §3.1 · p. 4
  • Each evidence episode packages task metadata, an ordered action trace, and available multimodal evidence, with manifest links enabling targeted inspection rather than sequential processing.Coverage depends on capture and export settings.Source: §3.1 · p. 4

3.2 WORKDEMO-350

  • WorkDemo-350 contains 350 demonstrations spanning 64 MyPCBench Bounded Action tasks and a fixed 286-task OSWorld-Verified subset.The benchmarks cover personal office and everyday workflows as well as general computer-use tasks across desktop and web applications.Source: §3.2 · pp. 4–5, Table 1 · p. 5

3.3 AGENTIC SKILL INDUCTION

  • A NexAU tool-using agent alternates evidence inspection with skill writing and revision, choosing observations in response to task needs and unresolved questions.The agent can return to evidence after drafting; candidate skills are not executed during induction, and downstream results are not revision feedback.Source: §3.3 · p. 5
  • Skills generalize demonstration-specific inputs into task-dependent descriptions while retaining semantic UI targets, dependencies, stop conditions, and visible completion checks.Finalization checks output structure and required tool use, but does not verify skill correctness or task success.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments compare reusable skills under shared execution settings across four models and two benchmark task sets, then test changed requirements, cross-model transfer, induction choices, and a case study. Work2Skill leads the compared methods in Score and success rate, while reuse and efficiency vary by evaluation.

4.2 RQ1: DOES WORK2SKILL IMPROVE DOWNSTREAM GUI EXECUTION?

  • Work2Skill improves mean Score over No Skill by 4.67–10.54 points on MyPCBench and 3.11–7.50 points on the evaluated OSWorld-Verified subset, with the highest Score and SR across both sets and four executors.Execution-step efficiency varies by model: it uses fewer steps than No Skill for Opus-4.8 and both Qwen models, but more for GPT-5.5.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 RQ2: DO SKILLS GENERALIZE TO NEW TASK INSTANCES?

  • On 64 MyPCBench Jitter tasks, Work2Skill improves average Score over No Skill by 3.93–8.64 points and SR by 6.25–9.38 percentage points.Its Score advantage over the stronger demonstration-based baseline is 4.28–6.25 points; it has the highest SR for three executors and ties D2T for Qwen3.5-9B.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 RQ3: DO SKILLS TRANSFER ACROSS MODELS?

  • Figure 4 organizes cross-model transfer results by induction model in rows and executor in columns, with gains measured against each executor’s No Skill baseline.The supplied passages describe the figure’s layout but do not report its transfer outcomes.Source: Figure 4 · p. 8

4.5 RQ4: HOW DO EVIDENCE AND INDUCTION PROMPTS AFFECT SKILL INDUCTION?

  • On 64 MyPCBench tasks, Full Evidence scores 85.27 and fully solves 31 tasks; removing indexed images, UI annotations, or KeyLogs reduces Score by 5.02–10.63 points and SR by 14.06–21.88 percentage points.The largest Score drop is without indexed images, and the largest SR drop is without KeyLogs; fewer steps alone do not establish better completion.Source: Figure 5 · p. 8, §4.5 · p. 8
  • With evidence held fixed, Work2Skill’s prompt improves Score by 3.97 points and SR by 6.25 percentage points over Pipeline Prompt, while reducing average steps from 48.98 to 47.38.It uses 12.7% more skill-induction tokens; Full Evidence yields the best Score and SR among the tested input configurations.Source: Figure 5 · p. 8, §4.5 · p. 9

4.6 CASE STUDY

  • In the Cotogna booking case, Work2Skill guidance satisfies all eight rubric requirements (100/100), while ShowUI-Aloha and D2T each score 55/100.The skill makes intermediate requirements explicit, preserves dependent subgoals across navigation, and checks the reservation and saved favorites before completion.Source: Figure 6 · p. 9, §4.6 · p. 9

5 CONCLUSION

Work2Skill combines aligned human GUI evidence, a curated demonstration dataset, and agentic skill induction. It achieves the highest scores and success rates among compared methods on two evaluated task sets, with further tests supporting reuse across changed requirements and tested models.

5 CONCLUSION

  • Work2Skill combines WorkRecorder’s aligned multimodal evidence, 350 curated WorkDemo-350 demonstrations, and agentic skill induction with on-demand inspection and iterative writing and revision.The resulting procedural guidance can be applied through a shared text interface without access to the original recordings or other demonstration evidence.Source: §5 · pp. 9–10
  • Across four execution models, Work2Skill achieves the highest scores and success rates among compared methods on MyPCBench Bounded Action and the evaluated OSWorld-Verified subset.Further experiments support reuse under changed task requirements within the same application and transfer between the tested models.Source: §5 · p. 10

A.1 ARCHITECTURE AND DEPLOYMENT

WorkRecorder captures desktop activity and available accessibility evidence, then aligns operations with visual context for skill induction. The converted episodes retain task, action, and evidence metadata, but modality availability and exported-image counts vary.

A.1 ARCHITECTURE AND DEPLOYMENT

  • WorkRecorder combines screen and input recording with available accessibility evidence, using target queries and event context to associate UI elements with pointer actions.Remote Linux collection uses AT-SPI; association records can preserve uncertainty and source information.Source: §A.1 · p. 14

A.2 KEYFRAME SELECTION

  • Keyframes are triggered around operations and interface transitions, then filtered by timing, per-action limits, and capture budgets.Before-action context can be unavailable without an explicit pre-action event, and a result frame does not establish that the interface finished updating.Source: §A.2 · pp. 14–15

A.3 TIME ALIGNMENT

  • Evidence is mapped onto a stitched video timeline using first-frame anchors and segment offsets, while remote Linux events are projected from wall-clock time.The launcher rejects host-clock differences above two seconds; this is a deployment check, not an alignment-error bound.Source: §A.3 · p. 15

B.1 EPISODE FILES FOR SKILL INDUCTION

  • Each induction episode has task and recording manifests, with relative paths indexing available files; the authoritative task instruction and inputs belong in task.json.Table 4 identifies the core episode files and notes that remaining artifacts are included when available.Source: §B.1 · p. 15, Table 4 · p. 16

B.2 ACTION RECORDS AND IMAGE ASSOCIATIONS

  • Action records preserve sequence, kind, video-relative time, and available target data, while image indexes associate frames with input events and trajectories.An event frame near an input may not show its effect, so a later observation can be needed to verify the transition.Source: §B.2 · p. 16

B.3 EVIDENCE AVAILABILITY AND COUNTS

  • Episode schemas do not guarantee every modality, and action, keyframe, and exported-image counts differ because capture and export apply separate grouping rules.The induction agent can inspect only evidence exposed in its input directory; exporting all keyframes is a separate option.Source: §B.3 · p. 16

C EVIDENCE-EXPLORATION TOOL DEFINITIONS

The induction setup gives an agent tools to inspect episode files, images, and video, then write and validate reusable skills. A concise prompt leaves exploration and revision flexible, while a pipeline-prompt baseline prescribes an inspection workflow under otherwise shared induction conditions.

C EVIDENCE-EXPLORATION TOOL DEFINITIONS

  • The registered tools support file discovery and reading, ordered image inspection, video sampling or frame extraction, and skill writing and finalization.Finalization checks structural requirements and call ordering; heuristic warnings do not guarantee semantic correctness.Source: C EVIDENCE-EXPLORATION TOOL DEFINITIONS · pp. 17–18, Table 5 · p. 17

C.1 EXAMPLE EXPLORATION TRACE

  • The example trace reads the task, locates and inspects relevant evidence, writes and revises a skill, then reads it back before finalization.It is illustrative rather than a required sequence, and run records provide provenance without sourcing every generated statement.Source: §C.1 · p. 18

D.1 INDUCTION SYSTEM PROMPT

  • The induction system prompt sets evidence boundaries and completion requirements, directing the agent to read the authoritative task and treat recordings as fallible evidence.It requires representative initial, critical, and final-state images when indexed images are available.Source: Induction System Prompt · p. 18

D.2 NEXAU AGENT CONFIGURATION AND COMPLETION

  • The NexAU launcher runs one agent conversation in which evidence inspection and skill authoring occur together, without a separate controller state or numerical information-gain objective.Runtime code enforces structural validation and call ordering, while grounding and generalization remain model instructions.Source: §D.2 · pp. 18–19

D.3 PIPELINE PROMPT

  • The Pipeline Prompt baseline changes the induction system prompt while keeping the model, evidence, tools, authoring policy, and output format aligned with Work2Skill.It prescribes an inspection order and frame-selection rules; the paper offers narrowing evidence exploration as one possible explanation for prompt-ablation results.Source: §D.3 · p. 19, Pipeline Prompt · p. 19

Pipeline Prompt

  • The Pipeline Prompt directs the agent to inspect task and event-aligned evidence, verify action effects, and generalize reusable procedures with semantic targets and task-dependent details.It distinguishes episode-specific values from workflow rules and asks for observable completion checks and supported recovery behavior.Source: Pipeline Prompt · pp. 20–21

D.5 DOWNSTREAM EXECUTION PROMPT

  • Downstream methods share an executor wrapper that makes guidance advisory, preserves the user instruction as authoritative, and requires live-environment verification.No Skill adds no guidance block; method-specific contents are inserted into the same wrapper.Source: §D.5 · p. 23, Table 6 · p. 24

E BENCHMARK TASKS AND EVALUATION

  • Tasks and demonstrations are retained only after infeasible tasks are excluded and recordings are reviewed for completeness, key operations, and correct task completion..Source: E BENCHMARK TASKS AND EVALUATION · p. 24

E.1 MYPCBENCH TASKS AND JITTER PERTURBATIONS

The MyPCBench Jitter set pairs each original task with a variant that changes requirements within the same application context. Existing skill files are reused against revised tasks and rubrics.

E.1 MYPCBENCH TASKS AND JITTER PERTURBATIONS

  • Each of the 64 MyPCBench Bounded Action tasks is paired with a human demonstration and exported multimodal evidence for skill induction.Source: §E.1 · p. 24
  • 64 Jitter variants alter requirements within their original application contexts, covering targets, source materials, parameters, selection rules, constraints, or final states.GPT-5.6-sol assisted in generating revised instructions and evaluation criteria, followed by human verification.Source: §E.1 · p. 24
  • 331 of 366 paired rubric items are updated for perturbed requirements, while 35 generic criteria and all rubric weights and evaluator types remain unchanged.Each variant retains the skill-file mapping of its original task, measuring reuse on a changed task instance.Source: §E.1 · p. 24, Figure 7 · p. 25

E.2 OSWORLD-VERIFIED TASK DISTRIBUTION

The evaluation uses a 286-task OSWorld-Verified subset and official benchmark evaluators. Guidance methods are compared under shared execution conditions, with the same model used for induction and execution by default.

E.2 OSWORLD-VERIFIED TASK DISTRIBUTION

  • The retained OSWorld-Verified subset contains 286 tasks across ten categories, with category percentages calculated over those tasks.Figure 8 summarizes the category distribution.Source: §E.2 · p. 25, Figure 8 · p. 25

E.3 EXECUTION HARNESS AND EVALUATOR CONFIGURATION

  • Official evaluators produce partial or full-credit outcomes; Score averages outputs on a 0–100 scale, while SR counts only tasks receiving full credit.OSWorld’s reporting-script “success rate” is reported as Score in the paper’s tables.Source: §E.3 · p. 26

E.4 BASELINE ADAPTATION FOR FAIR COMPARISON

  • No Skill supplies no guidance, while ShowUI-Aloha action captions and D2T tutorial prose are adapted into the same wrapper used for Work2Skill skills.For each execution model, methods share the base prompt, observations, action interface, step limit, termination policy, and evaluator.Source: §E.4 · p. 26

E.5 MODEL PAIRING AND RUNTIME SETTINGS

  • By default, each execution model uses its own induced skills, while a separate cross-model experiment varies the induction–execution pairing.Induction limits count agent iterations; execution limits count steps.Source: §E.5 · p. 26, Table 8 · p. 26, Table 9 · p. 26

E.6 EVIDENCE-MODALITY ABLATION SETTINGS

The ablations vary which recording evidence the induction agent can inspect while retaining task instructions and common authoring tools. The appendix also documents one induction run for a TableFind reservation task.

E.6 EVIDENCE-MODALITY ABLATION SETTINGS

  • The ablations retain task instructions and common file-reading and skill-authoring tools while varying the recording evidence exposed to the agent.The five conditions share a prompt adapted to their available modalities, along with task-first reading, output validation, and finalization requirements.Source: §E.6 · pp. 27–28, Table 10 · p. 27
  • With video retained, w/o Indexed Images removes indexed images, whereas Video-only supplies task.json and video.mp4 without action traces or a precomputed image index.Video metadata and frame timestamps remain inspectable in Video-only.Source: §E.6 · p. 27
  • w/o UI Annotations removes AX-derived target-element fields and replaces target descriptions with generic action captions while preserving raw-video pixels and click or drag highlights.Image indices are sanitized and their annotation bar is replaced with pixels from the corresponding video timestamp.Source: §E.6 · p. 27
  • w/o KeyLogs jointly removes raw logs and annotated image presentation, but retains action-trace target information and action captions, so it does not isolate the log file’s effect.Keyboard information is not removed altogether because it may remain in the retained actions.Source: §E.6 · p. 27
  • UI-grounded action-text-only induction retains task instructions and ordered captions with available UI target semantics, but removes coordinates, timestamps, raw keylogs, images, and video.The captions can include control names and roles from the corresponding AX tree; this condition supplies target semantics in text rather than a full AX Tree.Source: §E.6 · pp. 27–28

F.1 TASK AND INDUCTION TRACE

  • For the TableFind fallback-reservation example, GPT-5.5 made 30 tool calls, inspected 22 distinct evidence images, sampled a 156.47-second video, and finalized a skill after drafting and revision.The appendix documents the induction output but assigns no downstream execution score to this listing.Source: §F.1 · p. 28

F.2 DETAILED EXECUTION ANALYSIS

In the TableFind case, Work2Skill completes rubric requirements that both baselines leave unfinished. Its generated procedure preserves task values and dependencies across primary and fallback searches, then checks the reservation and favorites.

F.2 DETAILED EXECUTION ANALYSIS

  • The task requires carrying date and party size across searches, distinguishing the primary restaurant from alternatives, and completing fallbacks before confirming the primary reservation.The rubric checks the reservation and these additional requirements, including persistence after refresh.Source: §F.2 · pp. 28–29
  • Both baselines configure and confirm the primary booking but leave alternative coverage, top-choice favorites, and persistence after refresh incomplete.Passing the final-details check does not establish persistence, and the rubric does not establish that all three alternatives and their detail pages were inspected.Source: §F.2 · p. 28
  • Work2Skill passes all eight rubric items and scores 100/100, while both baselines score 55/100.The comparison identifies the difference in this case as completion of secondary subgoals and verification requirements unfinished in the baseline trajectories.Source: §F.2 · pp. 28–29, Table 11 · p. 29

F.3 COMPLETE GENERATED SKILL

  • The generated procedure carries booking values across searches, rechecks fields that may reset, excludes the primary restaurant from fallbacks, and preserves fallback-selection, favoriting, and confirmation order.For “top” choices, it follows the active ranking and verifies visible favorite status rather than toggling an already-saved restaurant off.Source: §F.2 · p. 29, §F.3 · pp. 29–31
  • The procedure delays final confirmation until fallback steps are complete, then checks visible reservation details and saved-favorite states before stopping.It verifies the restaurant, date/time, party size, and note in the confirmation flow, then checks that the confirmed reservation and requested favorites remain visible.Source: §F.2 · p. 29, §F.3 · p. 31

G TOOL USAGE DURING SKILL INDUCTION

Tool-call counts summarize induction activity across both datasets, but the reported counts alone do not establish evidence coverage or skill quality.

G TOOL USAGE DURING SKILL INDUCTION

  • GPT-5.5 invokes video-watching and frame-extraction tools most frequently on both datasets, while Qwen3.5-35B-A3B makes the fewest total calls per task: 14.33 and 12.94.Figure 9 summarizes mean tool calls per task for four models across 64 MyPCBench Bounded Action tasks and the 286-task OSWorld-Verified subset.Source: Figure 9 · p. 32, G TOOL USAGE DURING SKILL INDUCTION · p. 32
  • Invocation counts alone do not establish evidence coverage or skill quality, because calls can return multiple images or varying amounts of text.Repeated finalize-extraction calls also do not by themselves identify failed induction runs; validation diagnostics can support retries before successful completion.Source: G TOOL USAGE DURING SKILL INDUCTION · p. 32

H COMPLETE OSWORLD-VERIFIED TASK LIST

Table 12 lists the 286 OSWorld-Verified tasks used for demonstration collection and evaluation.

H COMPLETE OSWORLD-VERIFIED TASK LIST

Loading 2610.12984v1…