Source-linked AI summary
TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation
Zhibo Zhang, Zhen Ouyang, Ling Shi, Kailong Wang
TL;DR
Evaluating generated Agent Skills from artifacts or final outcomes leaves their induced actions and side effects unresolved. TRUSS combines static checks with execution evidence to jointly verify functionality and safety, raising effectiveness to 52.94% with a 100.00% benchmark Security rate.
Problem
Automated Skill authoring requires substantial expertise and validation, while artifact- or outcome-only evaluation leaves induced actions and side effects unresolved.
Method
TRUSS combines static evidence checks with brokered runtime execution in a controllable environment, linking observed failures to Skill content for iterative refinement.
Results
TRUSS improves vulnerability detection, reduces attack success without regression, and raises generation effectiveness to 52.94% with a 100.00% benchmark Security rate.
Takeaways & Limitations
Execution evidence supports jointly verified functional and safety outcomes by exposing failures missed by artifact-level inspection and guiding repairs.
Takeaways & Limitations
Runtime certification remains sensitive to execution-scenario coverage and executor behavior, motivating broader environments, property observers, and agent configurations.
Abstract
from arXiv · showhide
Agent Skills package reusable natural language procedures with executable resources, enabling software agents to acquire task specific capabilities without model adaptation. Automatically generating such Skills can improve task performance, yet evaluating a candidate solely from its artifact or final task outcome leaves unresolved which actions the equipped agent will perform and which side effects those actions will produce. We present TRUSS, an evidence guided framework for generating functionally effective and safety reliable Agent Skills. TRUSS first inspects functional claims against source and domain evidence while evaluating the complete artifact under nine predefined safety properties. Candidates admitted by this static gate are loaded by a shadow agent inside a Controllable Execution Environment, where brokered tools expose requested actions to policy enforcement and record their results as provenance preserving execution traces. Functional failures and property violations are linked back to the responsible Skill content and used to guide iterative refinement. We evaluate TRUSS on 168 SkillInject artifacts, 155 SkillSafetyBench cases, and all 187 tasks in SkillGenBench. TRUSS achieves 100.00\% precision and recall in vulnerability detection. Repair reduces attack success from 38.71\% to 19.35\% with GPT 5.5 and from 46.45\% to 29.68\% with GPT 5.4, with zero attack regression. For Skill generation, TRUSS raises task effectiveness from 17.11\% without Skills to 52.94\%, while increasing the benchmark Security rate from 50.80\% to 100.00\%. These results show that execution evidence can expose behavioral failures missed by artifact inspection and can guide Skill generation toward jointly verified functional and safety outcomes.
1 Introduction
TRUSS addresses the tension between improving agents with reusable Skills and ensuring that the resulting actions remain safe and reliable. It combines static property checks with dynamic execution evidence to verify and refine generated Skills.
- Motivation: Skills package reusable workflows, domain knowledge, and tool integrations into file-based instructions that can improve agent performance without model tuning.
- Requirements: Functional Effectiveness requires verified improvement on the intended task, while Safety Reliability requires induced actions to fulfill functionality without security-relevant behavior.
- Challenge: Detailed procedures and executable helpers can improve task success while expanding the agent’s action space, making static-only generation pipelines struggle to balance effectiveness and safety.
- TRUSS: TRUSS combines predefined static checks with property-level evidence derived from execution traces observed in a Controllable Execution Environment.
- Results: 100% precision and recall are achieved for vulnerability detection on matched clean/injected skill pairs.
- Results: 38.70% to 19.35% harmful rate reduction is reported with GPT-5.5, and 46.45% to 29.68% with GPT-5.4, with no regression.
2 Related Work
Prior work evaluates generated skills through downstream performance or pre-deployment artifact properties, while security research emphasizes their authority and execution risks. TRUSS unifies static and dynamic verification by evaluating functional outcomes and safety properties on execution traces.
- Execution-guided skill generation: Execution-guided methods assess candidate skills through downstream agent performance, including trajectory-based construction and retention of skills that improve held-out validation performance.Skill Set Optimization constructs transferable skills from high-reward trajectories, while EvoSkill analyzes failed executions and retains performance-improving skill folders.
- Pre-deployment artifact evaluation: Artifact-oriented methods evaluate generated skills before deployment for properties such as safety, completeness, executability, maintainability, and cost.SkillNet combines diverse sources to construct skills, uses predefined rubrics for safety assessment, and applies sandbox execution to test bundled code and tool invocations.
- TRUSS verification loop: TRUSS combines static and dynamic verification, checking consistency among declared tasks, capabilities, executable components, data access, and expected side effects before execution.Candidates admitted by static checks are executed by a shadow agent in a Controllable Execution Environment.
- TRUSS verification loop: TRUSS evaluates functional outcomes and safety properties on the same execution trace, distinguishing successful execution from unsafe intermediate behavior.Observed information flows and side effects provide refinement evidence for preserving verified improvements while removing vulnerabilities exposed during environmental interaction.
- Security boundary and safeguards: Agent skills are security-critical because their instructions and resources influence agents with system-command and external-web access, while their authority follows host-agent permissions.Prior analyses and provider guidance identify risks including sensitive-data access, exfiltration, unintended actions, and the need for sandboxing, approval policies, and network controls.
3 Methodology
TRUSS generates complete Agent Skill packages and iteratively verifies them through evidence-based functional inspection, predefined safety checks, and supervised execution. Candidates are refined using provenance-preserving records until required functions are established and applicable safety obligations are resolved.
- Skill formulation: TRUSS models a generated Skill as a standardized artifact containing procedural content and executable resources that a fixed agent loads and executes.The package boundary captures both natural-language guidance and machine-executable content because their interaction determines induced behavior.
- Skill formulation: Generation operates in task conditioned and task agnostic regimes, differing in whether the generator receives a disclosed task specification alongside the source corpus.Task-conditioned generation uses G(c,t), whereas task-agnostic generation uses G(c) and hides downstream tasks until evaluation.
- Iterative checking: The Checker evaluates each candidate through static artifact inspection and runtime behavior inspection, covering both functional obligations and applicable safety properties.Runtime inspection uses a Model Executor, brokered tools, an Intermediate Breaker, and a Disposable Sandbox to supervise requested actions and produce execution traces.
- Evidence-based function inspection: Functional inspection links claimed effects and usage conditions to open-world evidence, admitting claims when evidence supports feasibility or converts residual uncertainty into executable runtime obligations.Each normalized claim is anchored to an identifiable evidence span and an observable runtime criterion when needed.
- Pre-defined safety properties inspection: TRUSS defines nine safety properties in advance, derives their scope from declared behavior and implemented capabilities, and assigns execution-dependent conclusions to runtime obligations.The framework preserves supporting artifact spans or execution events in a Function and Safety Record, which guides refinement until every applicable obligation is resolved.
4 Evaluation
TRUSS is evaluated for vulnerability detection, safety-preserving repair, and verified Skill generation across SkillInject, SkillSafetyBench, and SkillGenBench. Runtime evidence completes static inspection, reduces unsafe behavior without attack regression, and improves both task effectiveness and security.
- RQ1: Vulnerability detection: TRUSS reaches 100.00% precision, recall, and F1 in SkillInject vulnerability detection, resolving errors left by static inspection alone.The evaluation uses 84 matched pairs, yielding 168 blinded artifacts; runtime evidence adds 18.45 percentage points in precision and 5.95 percentage points in recall over the Static Checker.
- RQ2: Safety-preserving repair: 38.71% ASR falls to 19.35% after GPT 5.5 repair, while 46.45% falls to 29.68% with GPT 5.4, with zero attack regression.Repair is evaluated on all 155 SkillSafetyBench cases, eliminating 30 attacks for GPT 5.5 and 26 for GPT 5.4.
- RQ2: Safety-preserving repair: 52.90% safe task completion is achieved with GPT 5.5 after repair, compared with 23.87% for GPT 5.4.GPT 5.5 safely completes 82 tasks, whereas GPT 5.4 safely completes 37; safe task completion credits only completions that also suppress the attack.
- RQ3: Verified Skill generation: 52.94% Effectiveness is achieved by TRUSS on SkillGenBench, versus 17.11% with No Skills and 29.95% with the intermediate LLM generator.TRUSS succeeds on 99 of 187 samples and improves by 35.83 percentage points over No Skills.
- RQ3: Verified Skill generation: 100.00% Security is achieved by TRUSS, improving by 49.20 percentage points over No Skills and 24.60 percentage points over the intermediate generator.The comparison covers all 187 SkillGenBench samples; TRUSS converts the remaining 46 unsafe outcomes into safe executions.
5 Conclusion
TRUSS frames automated Agent Skill generation as joint functional and safety verification, combining static evidence checks with provenance-preserving runtime execution. Across detection, repair, and generation, it improves vulnerability detection, reduces attack success without new attacks, and raises task effectiveness alongside benchmark Security.
- Framework: TRUSS treats automated Agent Skill generation as a joint functional and safety verification problem.Its static stage evaluates functional claims and security-relevant capabilities before agent exposure, while runtime execution produces provenance-preserving evidence from brokered tool interactions.
- Evaluation: 100.00% precision and recall were achieved on 168 SkillInject artifacts.The result demonstrates the framework’s vulnerability-detection performance.
- Evaluation: Attack success was substantially reduced on 155 SkillSafetyBench cases without introducing new attacks.Repair outcomes varied across target models, with residual attack success remaining after repair.
- Evaluation: 52.94% effectiveness and a 100.00% benchmark Security rate were reached across 187 SkillGenBench tasks.These results show gains in both task effectiveness and security during Skill generation.