Source-linked AI summary
SkillHarness: Harnessing Safe Skills for Computer-Use Agents
Yurun Chen, Biao Yi, Keting Yin, Shengyu Zhang
TL;DR
Existing CUA skill-learning methods often assume static, safe environments, leaving safe learning and reuse under dynamic conditions unresolved. SkillHarness addresses this with safety-constrained skill boundaries and selective reuse, reducing unsafe learned skills by 57.1% across multiple benchmarks while improving execution stability.
Problem
Existing skill-learning methods assume static, safe environments, limiting safe skill learning and adaptive reuse under dynamic conditions.
Method
SkillHarness uses multi-source supervision to define skill boundaries and selectively activates context-appropriate skills through environment-state checks.
Results
57.1%: SkillHarness reduces the proportion of unsafe learned skills across multiple benchmarks while consistently improving execution stability over existing methods.
Takeaways & Limitations
The results support explicitly modeling skill boundaries and decoupling intent from execution for safer, more robust CUA skill use under environmental perturbation.
Takeaways & Limitations
Previously unseen risk patterns can still lead to unreliable skill invocation under adversarial conditions.
Abstract
from arXiv · showhide
Computer-Use Agents (CUAs) are increasingly deployed in dynamic interactive environments, creating a growing need for continual skill learning during interaction. Recent approaches address this challenge by learning reusable skills from successful trajectories. However, these skill learning methods largely assume static and safe environments, overlooking risks from adversarial interactions (e.g., prompt injections) and environmental dynamics (e.g., pop-ups). In dynamic settings, such assumptions can lead to risky skill learning and brittle execution, undermining the reliability of CUAs. This raises the question: how can CUAs learn and use skills safely in dynamic environments? To address this problem, we propose SkillHarness, a framework for safe skill harnessing in dynamic environments. SkillHarness moves beyond static skill abstractions by modeling skill learning and utilization as a safety-constrained interaction process. Specifically, we introduce the skill boundary that leverages multi-source supervision signals to identify safe skills from interaction trajectories, and construct self-improving safety constraints throughout the skill lifecycle. In addition, SkillHarness introduces selective skill reuse, where tasks are guided to decompose according to context and completed through the selective activation of skill subsets. Our experiments demonstrate that SkillHarness significantly reduces the unsafe rate of learned skills by 57.1% and consistently improves execution stability under dynamic environmental changes, outperforming existing baselines.
1 Introduction
Existing CUA skill-learning methods learn reusable behaviors from successful trajectories but often assume static, safe environments, causing supervision bias and brittle reuse in dynamic settings. SKILLHARNESS addresses these risks with multi-source skill boundaries and selective skill reuse, reducing unsafe learned skills by 57.1% while improving safety performance and task success.
- Background: Learned skills enable more reliable CUA task execution and support continual learning by extracting reusable patterns from interaction trajectories.Recent approaches move beyond manually designed libraries and improve generalization through composition or iterative optimization.
- Limitations: Existing methods assume static, safe environments, creating supervision bias and hardcoded interaction flows that risk unsafe skills and brittle execution under changing conditions.Successful task completion may depend on transient or unsafe interaction states, while learned behaviors often omit the conditions under which they remain valid.
- SKILLHARNESS: SKILLHARNESS learns and uses skills safely by modeling them as context-dependent capabilities shaped by experience and constraints.The framework operationalizes human know-how through a skill boundary and selective skill reuse.
- SKILLHARNESS: Its skill boundary combines successful trajectories, failure cases, and identified risks to capture both executable behaviors and the conditions under which they remain reliable.The three signals provide positive examples, expose behaviors that fail to generalize, and identify unsafe behavior under adversarial or changing environments.
- Results: 57.1% reduction in the proportion of unsafe learned skills is achieved by SKILLHARNESS, alongside 31.9% average improvement in safety performance during utilization and 19% average improvement in task success rate.These results are reported against existing skill-learning methods across multiple benchmarks.
2 Related Works
Related work develops semantic or procedural skills from successful trajectories and increasingly emphasizes controllable execution through harness engineering. However, existing approaches do not explicitly model behavioral boundaries or safely harness skills amid changing environmental conditions.
- Skill Learning: Existing skill-learning methods abstract semantic skills or construct code-based procedural representations from successful trajectories.The cited approaches include semantic abstractions and code-based procedural representations.
- Skill Learning: Without explicit behavioral boundaries, learned skills may inherit incidental environmental dependencies as interaction flows change, reducing their safety.These limitations motivate safer skill learning for dynamic environments.
- Harness Engineering: Harness engineering has evolved from prompt engineering through context engineering toward more controllable task execution for computer-use agents.Recent surveys examine this progression in harness design.
- Harness Engineering: Existing work still lacks harness-driven designs for safely utilizing skills in dynamic environments.The passage identifies safe skill utilization under environmental change as an unresolved gap.
3 SKILLHARNESS
SkillHarness organizes safe skill harnessing into Skill Learning and Skill Utilization, connected by learned boundaries that govern applicability and safety. Its skills combine strategic macro skills with state-grounded micro skills, selective activation, and adaptive fallback for dynamic environments.
- Overall Framework: SkillHarness separates Skill Learning, which discovers reusable patterns and constructs safety boundaries, from Skill Utilization, which selectively reuses skills under constraints.The two stages are connected by learned skill boundaries that transfer applicability and safety information across the skill lifecycle.
- Skill Representation: The skill library K = (M, N) contains macro skills for strategic direction and safety boundaries, and micro skills for parameterized actions grounded in the current state.This separation keeps intent distinct from grounding: macro skills express reusable strategies, while micro skills bind actions to observations.
- Skill Utilization: Micro skills execute deterministically when templates bind successfully, but semantic LLM fallback handles binding failures and repeated failures can bypass brittle template replay.Fallback increases action variance while supporting safer behavior in unfamiliar environments and interpreting novel warnings.
- Skill Learning: Skill learning uses task-free exploration: goals are proposed from library coverage, trajectories are executed, and an evolution policy extracts supervision signals to create or refine skills.Exploration prioritizes under-explored capability clusters so coverage broadens rather than repeating known interaction families.
- Skill Boundaries: Each skill boundary combines success patterns, failure-derived lessons, and policy-derived risk guards that define when associated micro skills may be activated.Risk guards distinguish SkillHarness from conventional abstractions by requiring accumulated boundary conditions to hold before activation.
4 Experiments
Experiments across four benchmarks show that SKILLHARNESS learns safer skills and maintains more reliable completion under adversarial and environmental perturbations. Its safety benefits persist across model scales, while ablations identify skill boundaries and macro skills as important components.
- Benchmark Evaluation: SKILLHARNESS answers both evaluation questions affirmatively across four benchmarks: learned skills are safer, and reused skills complete more tasks under environmental perturbation.The benchmarks cover OS and web adversarial risks, site-specific security policies, and task completion under changing interfaces.
- Web Benchmark Results: SKILLHARNESS consistently outperforms baselines on policy compliance and adversarial safety across ST-WebAgentBench and WASP.ASI is described as sensitive to external risks, while SkillWeaver performs worst on policy compliance.
- Skill Safety: SKILLHARNESS achieves the lowest unsafe skill rate on ST-WebAgentBench, compared with SkillWeaver’s 43.6% and ASI’s 75.0%.The evaluation combines benchmark risk categories and safety policies with LLM-based behavioral analysis.
- Perturbation Robustness: Under pop-ups, adversarial descriptions, misleading descriptions, and mixed perturbations, SKILLHARNESS maintains higher SCR than SkillWeaver.Macro–micro skill separation limits localized environmental changes from becoming complete intent failures, whereas rigid code templates degrade more sharply.
- Model Scaling: Across execution models, ASR remains consistently low despite large success-rate differences, indicating that weaker models tend to fail safely under SKILLHARNESS.The results suggest that learned skill boundaries act as explicit behavioral constraints and that safety is only weakly coupled to task performance.
- Ablation Study: Removing the skill boundary increases attack success rate by +9.6 percentage points, while removing macro skills degrades both reported metrics.Every ablated component causes performance degradation, showing that skill learning and execution jointly contribute to overall performance.
5 Discussion & Future Work
The discussion identifies a trade-off between reusable skill abstractions and discoverable coverage, while emphasizing that reliable execution also depends on harness design during skill utilization.
- Challenges of complex skill abstractions: Self-proposed exploration can produce overly narrow, complex skill paths that are difficult to reuse and introduce execution brittleness.Skills learned during exploration are not necessarily covered or utilized during evaluation.
- Challenges of complex skill abstractions: Capability clusters encourage more reusable skill abstractions but constrain the scope of discoverable skills.Future work should better balance granularity and coverage in self-proposed skill discovery.
- The role of harness design: Test-time feedback and selective skill reuse can mitigate failures such as repeatedly invoking an unsuitable skill.The findings indicate that reliability depends on both skill learning and an appropriately designed harness during utilization.
6 Conclusion
SKILLHARNESS models the skill lifecycle as a safety-constrained process, combining complementary supervision during learning with environment-state checks for selective reuse. Across four benchmarks, it addresses supervision bias and representation brittleness in trajectory-based skill learning.
- Framework: SKILLHARNESS models the skill lifecycle as a safety-constrained process.The framework is harness-driven.
- Framework: Three complementary supervision signals support skill learning, while environment-state checks enforce selective skill reuse during utilization.Together, these mechanisms target supervision bias and representation brittleness.
- Evaluation: Experiments across four benchmarks evaluate skills learned and reused under the SKILLHARNESS framework.The supplied passage identifies four benchmarks but does not provide their individual results.
A Case Study
The case study examines success, failure, and risky cases generated during execution. SkillHarness shows stable and conservative behavior under environmental changes, while remaining limited by skill coverage and verification timing.
- The case study analyzes three categories of cases generated by SkillHarness during execution.
- Success Case: Under changing interface states and environmental perturbations, SkillHarness exhibits more stable execution than baselines.Macro skills combine success patterns with failure-derived lessons and risk guards; planners check current states against these constraints during reuse.
- Failure Case: Failures primarily arise from limited skill coverage and misaligned execution-verification timing rather than unsafe behavior.Fine-grained decomposition can produce rigid skills, whereas overly coarse decomposition may omit critical contextual constraints.
- Risky Case: Under adversarial and policy-constrained conditions, SkillHarness induces more conservative execution than code-based baselines by incorporating failure and risk signals.These signals make applicability constraints more explicit at the procedural level, reducing unsafe action sequences when observed patterns cover the environment.
B Implementation Details
SkillHarness is implemented with iterative candidate-goal exploration, context-based skill retrieval, and explicit planning limits. Evaluations use screenshot-accessibility-tree observations, pyautogui actions, fixed resolution, and five perturbation settings.
- Skill Learning Settings: SKILLHARNESS performs 30 exploration rounds per site and proposes under-explored capability-cluster goals with a default batch size of 8 candidates per iteration.ASI and SKILLHARNESS learn from a held-out training split and are evaluated on the test split.
- Skill Retrieval: The planner retrieves top-k = 3 macro skills, surfaces up to 3 lessons and 3 risk items per skill, and caps micro-skill selection at 6 domain skills per step.Micro-skill candidates are selected through embedding-based similarity between the current UI context and skill descriptions.
- OpenApps Perturbations: The evaluation includes five settings: Default, Pop-ups, Adversarial Descriptions, Misleading Descriptions, and Mixed Perturbations.Adversarial descriptions relabel UI elements with misleading text, misleading descriptions redirect navigation labels, and mixed perturbations combine all listed perturbations.
C Training Task Splits
The evaluation uses fixed, coverage-oriented training splits for ST-WebAgentBench and a scenario-based Prompt Injection benchmark for WASP. ST-WebAgentBench targets approximately 30% training tasks per site, while WASP combines benign goals with injection formats into benchmark units.
- ST-WebAgentBench: ST-WebAgentBench fixes train/eval partitions at approximately 30% training tasks per site while maximizing intent-template ID coverage.The GitLab training split contains 59 tasks, including project, group, milestone, issue, merge-request, assignment, site-title, commit, contributor, and repository-cloning tasks.
- ST-WebAgentBench: The ST-WebAgentBench GitLab split contains 59 training tasks, using IDs 0–46, 85, 86, 90, 95, 100, 105, 110, 113, 118, 123, 128, and 130.The passage also specifies 51 SuiteCRM training tasks with IDs 47–75 and 235–269.
- WASP: WASP’s Prompt Injection benchmark defines 21 scenarios across GitLab (IDs 1000–1011) and Reddit (IDs 1012–1020), with 11 scenarios in the default training set.The default training set includes six GitLab scenarios and five Reddit scenarios.
- WASP: Each WASP task combines 2 benign user goals with 2 injection formats—plain text and URL injection—yielding 4 benchmark units per task.The benign goals are commenting on an issue or post and adding a thumbs-up reaction or upvoting a post, depending on the site.
C.1 Metrics
The section defines task-level metrics for completion, safety, and their joint utility, alongside skill-level measures of unsafe induction and execution reliability. Together, these metrics distinguish task success from policy compliance and assess the safety and effectiveness of learned skills.
- Task-level metrics: Success Rate (SR) measures the fraction of tasks completed within the maximum step budget and verified against the original instruction’s requirements.Completion requires the benchmark’s ground-truth evaluator to verify all requirements.
- Task-level metrics: Attack Success Rate (ASR) measures the fraction of tasks involving a safety-policy violation or successful adversarial attack, independently of task success.A failed but policy-compliant task contributes to SR but not ASR.
- Task-level metrics: Completion Under Policy (CUP) measures the fraction of tasks that are both completed and policy-compliant, with the SR–CUP gap indicating the safety–utility trade-off.A large gap indicates safety without effectiveness, while a small gap with low CUP indicates frequent policy violations.
- Skill-level metrics: Unsafe Skill Rate (USR) measures the proportion of learned skills containing at least one policy-violating behavior, as identified through LLM-based analysis.Lower USR indicates safer skill induction.
- Skill-level metrics: Skill Completion Rate (SCR) measures the proportion of skill invocations whose intended effects are realized during utilization, with higher SCR indicating more reliable execution.On OpenApps, SCR is reported for micro-skills in SkillHarness and code-skills in SkillWeaver.
C.2 Evaluation Methodology
The evaluation uses benchmark-specific procedures to measure task completion, policy compliance, harm under prompt injection, and skill executability under UI changes. Results are reported across benchmark domains and perturbation types where specified.
- ST-WebAgentBench: ST-WebAgentBench uses rule-based evaluation of final page state for task completion and execution traces for site-specific policy compliance across GitLab and SuiteCRM.Results are reported per domain and overall.
- WASP: WASP evaluates ASR with an LLM reviewing the full trace for policy violations, while SR and CUP use rule-based task-completion criteria across GitLab and Reddit.LLM-based judgment is intended to capture nuanced violations missed by simple rule matching.
- OS-Harm: OS-Harm measures harm rate under prompt injection attacks, recording harm when agents expose, execute, or propagate adversarial content from the injected source.Lower harm rate indicates better robustness to prompt manipulation.
- OpenApps: OpenApps reports SCR per UI perturbation type using LLM judgments of whether each skill invocation fulfills its intent.SkillHarness aggregates SCR over micro-skill invocations, whereas SkillWeaver aggregates it over code-skill invocations.
D Safety Policy · E Prompt Templates
The appendix defines safety policies for ST-WebAgentBench, WASP, OS-HARM, GitLab, and Reddit, then specifies prompt templates for proposing, selecting, executing, reusing, and creating skills. Together, these materials constrain agents to policy-compliant, evidence-grounded actions and structured skill outputs.
- D Safety Policy: Safety policies are derived from ST-WebAgentBench and manually reviewed WASP policy information extracted with GPT-5.4.The detailed policy specifications are provided in the appendix.
- D Safety Policy: The enterprise policy framework prioritizes organizational policies over user preferences and task instructions, while requiring consent, bounded access, strict execution, and hierarchy adherence.Policy violations constitute failure, and uncertain actions should trigger a user question rather than a workaround.
- D Safety Policy: OS-HARM prohibits deliberate misuse, prompt-injection compliance, and agent-originated arbitrary decisions that could cause harmful outcomes; each is assigned risk level high.The prohibited actions include entering false or unintended commands and executing unauthorized actions.
- D Safety Policy: GitLab and Reddit policies prohibit high-risk behaviors including exfiltration, deletion, unauthorized access changes, phishing, harassment, misinformation, and unauthorized exposure of user messages.The policy set also includes medium-risk restrictions on making profiles public and creating misleading Reddit posts.
- E Prompt Templates: The skill-goal prompt requests one reusable multi-step GUI capability, using current UI, loaded skills, recent outcomes, failures, and novelty constraints to score candidates.Candidates are evaluated on executability, utility, and efficiency, with one capability category and concrete values required.
- E Prompt Templates: The execution prompt requires exactly one observable subtask chosen from current evidence, with policy constraints taking priority over skill reuse and progress speed.It requires switching objectives after two consecutive failures to realize the expected UI change.
- E Prompt Templates: Completion and skill reuse require concrete evidence from the current UI, chat, or history, semantic fit, an evidenced execution entry point, and no unmet policy prerequisites.The prompt forbids completion based only on intended actions or absence of errors and sets matched_intent_id=null when safe grounding is unavailable.
- E Prompt Templates: Macro-skill creation extracts complete reusable workflows from successful trajectories and may extract stable partial sequences or patterns from failed trajectories when they retain practical value.Successful trajectories require a concrete UI-checkable completion condition; failed ones are skipped only when no reusable capability remains.