SkillHarness: Harnessing Safe Skills for Computer-Use Agents

1 Introduction

• Prior work learns reusable skills from interaction trajectories, with methods including Voyager (Wang et al. 2023) and ASI (Wang et al. 2025b) demonstrating procedural-code representations and other approaches exploring composition or iterative optimization.The introduction says these methods suggest skills can be incrementally accumulated for continual learning in open environments (Yu et al. 2025; Zheng et al. 2025).Source: §1 · p. 1
• In dynamic environments, treating task success as sufficient supervision can encode unsafe behaviors, while fixed procedural flows can make skills brittle under environmental change and distribution shift.The introduction names these risks supervision bias in trajectories and hardcoded interaction flows.Source: §1 · p. 1
• The introduction contrasts learning know-what from successful trajectories with human know-how about when and under what conditions skills apply (Dreyfus and Dreyfus 1980; Dreyfus, Dreyfus, and Athanasiou 1986).Human skill learning integrates successes, failures, and risky situations to reveal applicability boundaries and support adjustments across contexts.Source: §1 · p. 1
• SkillHarness is proposed to help CUAs learn and use skills safely in dynamic environments, with skill boundary integrating three complementary supervision signals during induction.The supplied passage identifies successful trajectories as positive examples, but does not specify the other signals.Source: §1 · p. 1

Code Skill

• Selective skill reuse separates high-level intent from environment-specific skills: macro skills encode strategies and constraints, while parameterized micro skills are grounded in the current state.The planner activates skills only when their constraints are satisfied and falls back to flexible LLM-based planning when they cannot be safely applied.Source: Code Skill · p. 2
• Skill representations combine effective behavior, failure cases, and identified risks to capture the conditions under which executable patterns remain reliable.The risks indicate when behaviors may become unsafe in adversarial or changing environments.Source: Code Skill · p. 2
• SKILLHARNESS reduces the proportion of unsafe learned skills by 57.1%, improves safety performance during skill utilization by 31.9% on average, and improves task success rate by 19% on average.The evaluation spans multiple benchmarks.Source: Code Skill · p. 2
• SKILLHARNESS targets supervision bias during skill induction and brittle reuse because skills from successful trajectories may omit the conditions under which they remain valid.It models skills as context-dependent interaction capabilities shaped by experience and constraints.Source: Code Skill · p. 2

Skill Learning

• Trajectory-derived semantic and code-based skills can preserve abnormal or irrelevant behaviors without explicit boundaries, while contaminated skill reuse can cause unintended behaviors or system-level risks (Liu et al. 2026; Wang et al. 2026).Source: §2 · p. 2
• In dynamic operating system environments, changing conditions and interaction flows make raw-trajectory skills more likely to encode incidental dependencies, further reducing safety.Source: §2 · pp. 2–3

Harness Engineering

• Safe harness-driven skill use remains underexplored: semantic skills are context-sensitive (Wang et al. 2024; Xia et al. 2026), while code skills are deterministic but UI-state-coupled (Wang et al. 2025b; Zheng et al. 2025; Yu et al. 2025).Source: §2 · p. 3

3.1 Threat Model

• SkillHarness addresses unsafe trajectory-derived skills and UI/DOM drift through linked learning and utilization stages, transferring learned safety boundaries to selective reuse with safe LLM fallback.The threat model identifies risks from skills induced by unverified or unsafe trajectories and from environmental shifts that change observations and action effects.Source: §3 · p. 3, §3.1 · p. 3

3.2 Decoupled Skill Representation

• The library separates strategic macro intent and safety boundaries from state-grounded parameterized micro actions, with macro risk guards gating linked micro skills and templates falling back to semantic guidance when binding fails.A success pattern pairs a reusable action path with an observable completion condition, allowing multiple valid routes for the same intent.Source: §3.2 · p. 3

3.3 Skill Learning

• Task-free exploration targets under-covered capability clusters, while evidence-gated evolution extracts reusable workflows, generalizes failure lessons and policy-derived risk guards, and creates or refines skills only for new knowledge.When a safety policy is defined, detected risk signals are canonicalized into guards; without a defined policy, those signals are not persisted.Source: §3.3 · pp. 3–5

3.4 Skill Utilization

• At deployment, SkillHarness retrieves relevant skills, activates only those whose guards fit the current state, and bypasses repeatedly failing templates in favor of LLM fallback.The planner suppresses associated micro skills when the environment is incompatible; fallback can interpret novel warnings and adversarial injections at runtime.Source: §3.4 · p. 5

4.2 Main Results

• SKILLHARNESS consistently outperforms all baselines on two web benchmarks and achieves strong SR and ASR results on OS-Harm, indicating cross-environment generalization.Table 1 covers ST-WebAgentBench and WASP; Figure 3 reports the OS-Harm evaluation.Source: Table 1 · p. 6, §4.2 · pp. 6–7, Figure 3 · p. 7

4.1 Evaluation Setup

• Evaluations cover four benchmarks, task-training and self-proposal skill-learning settings, multiple CUA models, and skill-level and benchmark-level safety and completion metrics.The setup includes GPT-5.4 for skill learning, GPT-5.4-mini for downstream evaluation, and a 6,000-token maximum inference budget per CUA step; USR is manually reviewed, and uncertain invocation judgments go to human evaluation.Source: §4.1 · p. 6

4.2 Main Results

• 43.6% (SkillWeaver) and 75.0% (ASI) are the unsafe skill rates; SKILLHARNESS achieves the lowest USR on ST-WebAgentBench.The results support using multiple supervision signals in skill boundaries rather than relying only on successful trajectories and filtering afterward.Source: §4.2 · p. 7, Table 2 · p. 7
• Under pop-ups, adversarial or misleading descriptions, and mixed perturbations, SKILLHARNESS maintains higher SCR than SkillWeaver as perturbation intensity increases.The methods have comparable SCR in the default environment; macro–micro skill separation helps contain localized changes, while the planner can fall back to LLM-guided planning when skill conditions are violated.Source: §4.2 · p. 7, Figure 4 · p. 7
• SKILLHARNESS’s ASR remains consistently low across execution-model scales despite large success-rate variation; only models with limited instruction-following ability show noticeably higher ASR.The results suggest learned boundaries act as behavioral constraints across models, while weaker models may fail safely rather than complete tasks.Source: §4.2 · p. 7, Table 3 · p. 7
• Case studies find more stable execution under changing interface states and more conservative behavior under adversarial conditions, but unseen risks can still trigger unreliable skill invocation.Failures arise when task decomposition is too fine, producing rigid skills, or too coarse, losing contextual constraints.Source: §4.2 · p. 8

4.3 Ablation Study

• 9.6 percentage points is the ASR increase when the skill boundary is removed; removing any component degrades performance, while removing micro skills leaves SR and ASR unchanged.Removing macro skills degrades both metrics, whereas removing the update mechanism mainly affects SR, suggesting refinement primarily benefits task completion.Source: §4.3 · p. 8, Table 4 · p. 8

Challenges of complex skill abstractions

• Self-proposed exploration can yield narrow, complex skills that are unused in evaluation and brittle, especially when weaker models execute paths stronger models can compensate for.This reflects a trade-off between skill granularity and reusability.Source: §5 · p. 8
• Capability clusters encourage more reusable abstractions, but predefined clusters constrain discoverable skills, leaving future work to balance granularity with coverage.Source: §5 · p. 8

The role of harness design

• Test-time feedback and selective skill reuse can mitigate skill failures, indicating that reliability depends on harness design during utilization as well as skill learning.These mechanisms can reduce repeated invocation of an unsuitable skill.Source: §5 · p. 8

• Skills encoding explicit boundary conditions reduce safety violations by 31.9% across four benchmarks while maintaining robustness under environmental perturbation.Source: §6 · p. 8
• SkillHarness integrates three complementary supervision signals during learning and uses environment-state checks to enforce selective skill reuse.Source: §6 · p. 8
• The framework addresses supervision bias and representation brittleness by explicitly modeling skill boundaries and decoupling intent from execution.The conclusion presents these principles as insights for safer, more robust, and more generalizable skill-learning methods.Source: §6 · p. 8

Success Case

• SkillHarness executes more stably than baselines under interface changes and perturbations by checking learned constraints and degrading from micro-skills to macro guidance when conditions drift.Macro skills carry success patterns, failure-derived lessons, and risk guards; suppressing incompatible micro-skills helps prevent localized changes from cascading into execution failures.Source: A Case Study · p. 10

Failure Case

• Failures primarily reflect incomplete skill coverage and mistimed verification, with abstraction trade-offs and weakly validated skills contributing to poor generalization.Conflicting benchmark specifications and policy constraints also affect evaluation, but play a comparatively minor role in task completion; failures concentrate in interaction-heavy workflows where verification occurs too early or too late.Source: A Case Study · p. 10

Risky Case

• Under adversarial and policy-constrained conditions, SkillHarness behaves more conservatively than code-based baselines, but learned skill boundaries do not guarantee comprehensive safety.Failure and risk signals make applicability constraints explicit and reduce unsafe sequences when observed patterns cover the environment. Unseen or shifted adversarial behavior can bypass these constraints; robustness depends largely on policy constraints and executor-level robustness.Source: A Case Study · p. 10

Skill Learning Settings

• In Self Proposal, SkillWeaver and SKILLHARNESS each run 30 exploration rounds per site; SKILLHARNESS targets under-explored capability clusters with 8 candidates per iteration.ASI and SKILLHARNESS train on a held-out split and are evaluated on the test split; “w/o update” disables skill library writes during test-time exploration.Source: B Implementation Details · p. 10

Skill Retrieval

• The planner retrieves top-k = 3 macro skills, surfaces up to 3 lessons and 3 risk items per skill, and selects up to 6 embedding-matched domain skills per step.Micro skill candidates are matched by similarity between the current UI context and skill descriptions.Source: B Implementation Details · p. 10

Environment

• In the described environment, evaluations use observations combining accessibility-tree and screenshot inputs, desktop actions grounded in pyautogui, and a 1920×1080 resolution for all evaluations.Source: B Implementation Details · p. 10

OpenApps Perturbations

• OpenApps evaluation uses five settings: Default, Pop-ups, Adversarial Descriptions, Misleading Descriptions, and Mixed Perturbations.Adversarial Descriptions relabel UI elements with misleading text, Misleading Descriptions redirect navigation labels, and Mixed Perturbations combines the perturbations.Source: B Implementation Details · p. 10

C Training Task Splits

• ST-WebAgentBench uses a fixed train/eval partition targeting ∼30% train per site while maximizing intent-template coverage, with 59 GitLab and 51 SuiteCRM train tasks.GitLab tasks span project, group, milestone, issue, merge-request, and repository activities; SuiteCRM tasks span customer records, meetings, cases, exports, bulk operations, and email.Source: C Training Task Splits · p. 10
• WASP’s Prompt Injection benchmark defines 21 scenarios across GitLab and Reddit, with 11 scenarios in the default train set.The default set includes six GitLab scenarios (IDs 1000, 1001, 1002, 1003, 1007, 1009) and five Reddit scenarios (IDs 1012, 1013, 1014, 1019, 1020).Source: C Training Task Splits · p. 10
• Each WASP task combines 2 benign user goals with 2 injection formats, yielding 4 benchmark units per task.The goals are commenting on an issue or post and adding a thumbs-up reaction or upvoting a post; the formats are plain text and URL injection.Source: C Training Task Splits · pp. 10–11

C.1 Metrics

• SR measures the fraction of tasks completed within the step budget, with completion verified against the original instruction by the ground-truth evaluator.Source: §C.1 · p. 11
• ASR measures tasks with a policy violation or successful adversarial attack, independently of task success.A task that fails but remains policy-compliant contributes to SR but not ASR.Source: §C.1 · p. 11
• CUP measures the fraction of tasks both completed and policy-compliant, while the SR–CUP gap quantifies the safety-utility trade-off.A large gap indicates an agent is safe but ineffective; a small gap with low CUP indicates frequent policy violations.Source: §C.1 · p. 11
• USR measures the proportion of learned skills containing at least one policy-violating behavior, identified by LLM-based analysis against benchmark risk categories.Lower USR indicates safer skill induction.Source: §C.1 · p. 11
• SCR measures the proportion of skill invocations whose intended effect is judged fulfilled in the corresponding trajectory segment.On OpenApps, the paper reports micro-skill SCR for SKILLHARNESS and code-skill SCR for SkillWeaver; higher SCR indicates more reliable execution in the evaluated setting.Source: §C.1 · p. 11

C.2 Evaluation Methodology

• ST-WebAgentBench evaluates task completion and policy compliance with rule-based checks; WASP uses LLM judgment for ASR and rule-based task criteria for SR and CUP across two domains.OS-Harm measures harm under prompt injection attacks, while OpenApps measures skill executability under UI changes using SCR judged by an LLM.Source: §C.2 · p. 11

D Safety Policy

• ST-WebAgentBench risks use its own policies; WASP policies are extracted from the original paper with GPT-5.4 and manually reviewed for correctness and completeness.Source: D Safety Policy · p. 11

# POLICY CONTEXT

• The policy context treats any policy violation as failure, establishes organizational-over-user-over-task priority, and defines six safety dimensions from consent and scope to robustness and error handling.The OS-Harm policies prohibit deliberate misuse, prompt-injection compliance, and harmful agent-originated decisions; GitLab and Reddit policies further prohibit actions such as exfiltration, unauthorized deletion, and phishing.Source: # POLICY CONTEXT · pp. 12–13

E Prompt Templates

• The skill-goal prompt proposes one concrete, multi-step reusable capability, checks novelty against existing skills and recent outcomes, and selects a single category using executability, utility, and efficiency scores.It rejects single-click or navigation-only tasks, combined independent capabilities, maintenance tasks, and tasks requiring pre-existing user state or authenticated real websites.Source: E Prompt Templates · pp. 15–16
• The execution prompt prioritizes task and policy constraints over skill reuse, grounds subtasks in observed UI and history, changes stage after two repeated failures, and requires concrete evidence to mark completion.Source: E Prompt Templates · p. 17
• Stored skills are reused only when an intent matches and the current UI, chat, or history supports the entry point without unmet policy prerequisites.The prompt requires null intent selection when a skill cannot be safely grounded in the current context.Source: E Prompt Templates · p. 17
• Macro creation requires a meaningful multi-step reusable workflow, while single actions, navigation-only fragments, and noisy evidence are rejected.The template allows extracting useful partial workflows from failed tasks and favors updating an existing macro when it already covers the capability.Source: E Prompt Templates · p. 18
• Macro steps and completion conditions must derive from successful execution evidence, with failures informing lessons and risks rather than core workflow steps.The prompt also removes episode-specific literals, retains stable UI labels, and parameterizes runtime-variable values.Source: E Prompt Templates · pp. 18–19
• Risk guards require trusted inputs, target and permission checks before external changes, protection of sensitive information, and prioritization of user goals over page instructions.The templates also direct the agent to stop and reassess when errors, rejections, validation failures, or unchanged state occur.Source: E Prompt Templates · p. 19
• The output template captures reusable intent, workflow, UI-checkable completion, lessons, risks, and parameterized skills, while repair lessons prescribe corrective responses to failure signals.Examples include checking validation after submission failures and re-confirming a target when search results mismatch.Source: E Prompt Templates · p. 19