PaperlayerView exact revision on arXiv

Source-linked AI summary

From Skill Text to Skill Structure: The Scheduling-Structural-Logical Representation for Agent Skills

Qiliang Liang, Hansi Wang, Zhong Liang, Yang Liu

arXiv:2604.24026v4cs.CLcs.AI

TL;DR

Agent skills remain text-heavy, entangling invocation, execution, and action/resource-use evidence. This paper introduces SSL, a structured three-layer representation instantiated with an LLM normalizer, and finds significant gains over text-only baselines in Skill Discovery and Risk Assessment.

  • Problem

    Existing skill artifacts entangle invocation interfaces, execution structure, and action/resource-use evidence in text, leaving open how to expose skills in machine-usable form.

  • Method

    SSL maps SKILL.md artifacts into a typed three-layer representation of scheduling, structural, and logical evidence using an LLM-based normalizer.

  • Results

    MRR@50 rose from 0.649 to 0.729 in Skill Discovery, while macro F1 rose from 0.409 to 0.509 in Risk Assessment versus text-only baselines.

  • Takeaways & Limitations

    SSL exposes useful evidence across distinct skill-centered tasks and supports more inspectable, reusable, and operationally actionable skill representations.

  • Takeaways & Limitations

    Risk Assessment uses model-generated gold labels and a fixed LLM judge, so it measures static risk identification rather than expert security auditing or runtime safety.

Abstract

from arXiv · show

Large language model (LLM) agents increasingly rely on reusable skills: capability packages that combine instructions, control flow, constraints, and tool calls. In current agent systems, however, skills are still represented by text-heavy artifacts, mainly SKILL{.}md-style documents whose machine-usable evidence remains embedded largely in natural-language descriptions. As a result, skill-centered agent systems face a representation problem: both managing skill collections and using skills during agent execution require reasoning over invocation interfaces, execution structure, and concrete side effects, but these signals are often entangled in a single textual surface. An explicit representation of skill knowledge may therefore help make these artifacts easier for machines to acquire and leverage. Drawing on Memory Organization Packets, Script Theory, and Conceptual Dependency from Schank and Abelson's classical work on cognitive linguistic representation, we introduce what is, to our knowledge, the first structured representation for agent skill artifacts that disentangles skill-level scheduling signals, scene-level execution structure, and logic-level action/resource-use evidence: the Scheduling-Structural-Logical (SSL) representation. We instantiate SSL with an LLM-based normalizer and evaluate SSL-derived representations in two tasks, Skill Discovery and Risk Assessment. The experiment shows that SSL significantly outperforms the text-only baselines: in Skill Discovery, MRR@50 improves from 0.649 to 0.729; in Risk Assessment, macro F1 improves from 0.409 to 0.509. These findings suggest that an explicit, source-grounded structure can make agent skills easier to search and review, positioning SSL as a practical step toward more inspectable, reusable, and operationally actionable skill representations, rather than a finished standard or end-to-end skill-management mechanism.

1 Introduction

Agent skills remain text-heavy, obscuring distinct scheduling, execution-structure, and action/resource-use evidence needed for discovery and risk assessment. The paper introduces SSL, a three-layer source-grounded representation instantiated with an LLM-based normalizer, and reports improved performance over text-only representations.

  • Motivation: Skills are typically documented in SKILL.md-style or README-like text, leaving machine-usable evidence embedded in natural-language fields.Natural language is easy for people to author and read but difficult for automated systems to analyze, validate, and reuse reliably.
  • Motivation: SSL addresses a representational bottleneck in which semantically distinct skill properties are collapsed into one textual surface across downstream uses.The proposed representation exposes reusable, source-grounded skill evidence without repeatedly re-parsing the full instruction text.
  • Method: SSL disentangles skill-level scheduling signals, scene-level execution structure, and logic-level action/resource-use evidence in a three-layer representation.An LLM-based normalizer converts existing SKILL.md files into the SSL schema, which remains paired with the original source document.
  • Results: MRR@50 improves from 0.649 to 0.729 in Skill Discovery, while macro F1 improves from 0.409 to 0.509 in Risk Assessment over text-only representations.Skill Discovery uses a rich SSL-derived description view; Risk Assessment uses the combined SKILL.md + SSL view against the complete source document alone.
  • Resources: The released evaluation resources include a 6,184-skill corpus, 431 intent-level Skill Discovery queries, and 252 skills with six-dimensional Risk Assessment labels.These datasets cover public agent skills and support the paper’s two evaluated tasks.

2 Related Work

Prior work has advanced LLM agents through tool use, retrieval, and security analysis, but often leaves individual skills implicitly represented. SSL addresses this gap by organizing invocation interfaces, execution structure, and action/resource-use evidence into an explicit machine-usable form.

  • Agent Systems and Tool Use: LLM agent research has moved beyond standalone prediction and generation toward systems that maintain context, plan across steps, and act through external tools.Early tool-use work commonly modeled external capabilities as atomic APIs or functions selected and incorporated into model reasoning.
  • Skill Representation: Existing efforts largely treat individual skills as repository entries, learned units, or routing targets rather than explicitly structured machine-usable artifacts.The open problem is disentangling invocation interfaces, scene-level execution structure, and action/resource-use evidence.
  • Cognitive Representation Foundations: Schank and Abelson’s Memory Organization Packets, Script Theory, and Conceptual Dependency represent contexts, ordered activities, and primitive action structures.These concepts provide the closest classical analogy for SSL’s three evidence types: invocation-level interfaces, scene-level execution structure, and atomic action/resource-use evidence.
  • Skill Discovery: Skill discovery resembles query–document retrieval, but candidates are executable capabilities, making representation quality central to matching user requests with skills.Tool- and skill-retrieval studies motivate better retrieval models and descriptions while generally assuming candidate representations are already available.
  • Security Analysis: Agent-security research identifies risks where natural-language instructions interact with external capabilities, including indirect prompt injection and multi-step tool-using settings.A remaining need is inspectable pre-invocation evidence covering action types, resource scopes, dependencies, and data-flow cues.

3 The Scheduling-Structural-Logical Representation of Agent Skills

SSL represents each skill artifact through three source-grounded layers that separate invocation signals, execution structure, and atomic operations or resource-use evidence. Its typed, compact, and grounded design supports comparable skill records without inferring behavior absent from the source.

  • Core representation: SSL maps a skill artifact d into a typed representation containing scheduling, structural, and logical components plus containment and entry relations.The scheduling layer captures invocation signals; the structural layer models execution phases and transitions; the logical layer models atomic actions and resource-use evidence.
  • Design goals: SSL is compact, typed, and grounded: it preserves management and use evidence, restricts vocabularies for comparability, and summarizes only source-supported behavior.The representation avoids open-ended or subjective attributes and does not infer hidden behavior.
  • Scheduling layer: The scheduling layer records supported intents, input/output contracts, and coarse dependencies as a comparable skill-level capability interface.It enables repository-level comparison before unfolding the skill’s scene or logic-step structure.
  • Structural layer: The structural layer represents scenes as nodes in a directed graph whose edges encode phase-level transitions across organized execution stages.These stages can include preparation, acquisition, reasoning, action, verification, and recovery.
  • Logical layer: The logical layer represents atomic actions as directed logic-step graphs, recording typed act_type, arguments, effects, and resource boundaries.Logic steps are the smallest operational units in the representation, independent of runtime details.
  • Normalization pipeline: An LLM-based normalizer using DeepSeek-V3.2 converts the full skill document into SSL under schema, vocabulary, and source-grounding constraints.Every populated field must be supported by the source artifact.

4 Evaluation

SSL is evaluated as an intermediate representation in Skill Discovery and Risk Assessment, testing whether structured interface- and operation-level evidence improves retrieval and risk judgment. SSL-augmented inputs outperform text-only alternatives, with the strongest gains when structured evidence complements source context.

  • Evaluation tasks: SSL is evaluated on Skill Discovery, which tests routing requests to the correct skill, and Risk Assessment, which tests recovery of risk signals missed by text-only representations.The tasks assess interface-level and operation-level evidence under controlled comparisons.
  • Skill Discovery: 0.729 MRR@50 is achieved by Desc + SSL-Rich, improving from 0.649 over Desc + Source Outline, the strongest non-SSL baseline.Desc + SSL-Rich is best across all reported retrieval metrics and also exceeds Full SKILL.md at 0.645.
  • Risk Assessment: 0.509 macro F1 is achieved by Full SKILL.md + SSL, improving from 0.409 over Full SKILL.md and also improving macro precision, recall, and accuracy.Full SSL alone is only slightly above the full-document baseline, indicating that the main benefit comes from adding structured evidence to source context.
  • Risk Assessment: The combined view is strongest across all six risk dimensions, with the largest gains on data exfiltration, credential access, and resource abuse.SSL surfaces scattered operational cues, while the original document helps interpret their risk relevance; the macro-F1 improvement is 0.101 with a 95% interval of [0.051, 0.152].

5 Discussion

The discussion positions SSL as a source-grounded evidence layer and shared manifest that makes task-relevant skill signals explicit while preserving the richer context of original documents. Its strongest use is alongside source text, supporting discovery, risk assessment, indexing, inspection, and review.

  • Evidence-layer role: SSL separates source-grounded signals so downstream systems can select task-relevant evidence without losing access to the original skill artifact.It functions as an evidence layer alongside the source document.
  • Evidence-layer role: Skill Discovery benefits mainly from explicit interface and workflow evidence, whereas Risk Assessment depends more on evidence about actions and resources.Both tasks benefit when these evidence types are separated from long instructional prose.
  • Evidence-layer role: SSL is best used with the complete source document because structured evidence improves risk judgments while source text clarifies safeguards, hypothetical status, scope, and human review.Removing incidental prose can improve Skill Discovery matching, but combining the source document with SSL produces the best Risk Assessment result.
  • Shared manifest layer: Skill-centered systems need a shared manifest layer to make reusable facts persistent for registry indexing, phase inspection, and logic-level risk review.SSL supports these functions while retaining access to the original document.
  • Shared manifest layer: SSL could also support skill construction, maintenance, and execution by exposing expected inputs, phase boundaries, dependencies, resource effects, checkpoints, and confirmation needs.The paper’s evaluations focus on pre-execution skill selection and risk assessment, while these additional uses are proposed extensions.

6 Conclusion

The conclusion presents SSL as a structured representation that separates routing, execution, and action/resource-use evidence from raw SKILL.md text. It reports improved evaluation performance while emphasizing that SSL supports skill management but is not sufficient by itself.

  • Conclusion: SSL disentangles routing interfaces, execution structure, and low-level action/resource-use evidence from raw SKILL.md text.It is introduced as a structured representation for agent skills.
  • Conclusion: 0.729 MRR@50 is achieved by Desc + SSL-Rich, improving from 0.649 over the strongest non-SSL baseline in Skill Discovery.The passage reports significant gains over text-only baselines in the main evaluations.
  • Conclusion: SSL is informative but not sufficient by itself, and the paper mainly applies it to skill discovery and pre-execution risk assessment.Future work may refine SSL through repository-level skill graphs, runtime traces, and studies of how agents might use skills.

7 Limitations … C Prompting Protocol of the SSL Normalizer

The paper presents SSL as a compact, source-grounded typed JSON graph that separates skill interfaces, execution phases, and atomic operational actions, while documenting limitations in static extraction, normalization, evaluation scope, and benchmark construction. Its normalizer uses constrained prompting, validation, and bounded retries, producing 6,184 valid records from 6,300 skills.

  • 7 Limitations: SSL remains limited to static artifacts, may miss or coarsen behavior during normalization, and does not directly measure skill use in planning, execution, monitoring, or refinement.It also relies on automatically generated Skill Discovery candidates and model-generated risk labels rather than fully human-authored requests or expert security audits.
  • A.1 Graph-Design Principles: SSL realizes Eq. 1 as a typed JSON graph with linked scheduling, structural, and logical levels, using explicit entry pointers and restricted cross-level relations.The design preserves boundaries among interface evidence, phase structure, and action/resource-use evidence while excluding speculative attributes such as hidden behavior or inferred intent.
  • A.2 Scheduling Layer: The Scheduling Layer records skill-level interface and dependency signals for selecting and invoking skills without re-embedding or re-parsing their source documents.Its fields include goals, intent signatures, tags, patterns, inputs, outputs, dependencies, entry scenes, and coarse control-flow or sensitive-resource indicators.
  • A.3 Structural Layer: The Structural Layer models execution as a scene-level graph whose phases have goals, data contracts, entry and exit conditions, and transition rules.Scenes represent coherent execution phases rather than arbitrary text spans or sentence blocks.
  • A.4 Logical Layer: The Logical Layer decomposes each scene into source-grounded atomic actions and represents their control flow, data flow, resource boundaries, and operational effects.Closed vocabularies classify action primitives such as READ, SELECT, INFER, WRITE, CALL_TOOL, and TERMINATE, while resource_scope records the touched boundary.
  • B A Complete Example of SSL: The normalized Writing Refiner example links scheduling metadata to PREPARE, ACQUIRE, and REASON scenes and to logic steps for validation, guide selection, reading, rule selection, and editing.The instance demonstrates explicit inputs, outputs, conditions, bindings, effects, instruments, and resource scopes across the linked graphs.
  • C Prompting Protocol of the SSL Normalizer: The SSL Normalizer uses constrained NL2JSON prompting with DeepSeek-V3.2, source-grounded output requirements, deterministic parsing, structural validation, semantic checks, and bounded regeneration.Failed outputs are rejected, and unsupported fields remain empty, null, or coarsely categorized rather than being invented.
  • C Prompting Protocol of the SSL Normalizer: 98.16%: 6,184 of 6,300 skill directories produce valid SSL records after bounded validation and retry, while 116 invalid records are excluded.The released corpus accepts only parseable, schema-valid, and source-grounded records.

D Human Audit of Fidelity of SSL Normalizer

A lightweight human audit evaluated whether SSL claims produced by the LLM-based normalizer were supported by original SKILL.md artifacts. Across 100 sampled skills, support accuracy was 83%, indicating mostly grounded fields but weaknesses when workflow steps or resource effects were weakly specified.

  • Audit design: The audit sampled 100 skills and used a binary source-grounding check rather than full manual reconstruction of SSL.Annotators compared original SKILL.md files with selected normalized SSL claims and judged whether each claim was source-supported.
  • Audit rubric: Claims received Yes when explicitly stated or directly and unambiguously paraphrased from the source.The rubric permits concrete implications, such as mapping a report-writing instruction to a WRITE action over LOCAL_FS or CODEBASE.
  • Result: 83% support accuracy was achieved on the 100-skill audit set.Support accuracy is defined as the fraction of audited SSL claims labeled Yes.
  • Limitations: The result indicates that most audited SSL fields were grounded in source artifacts, but over-abstraction remained a weakness for weakly specified workflow steps or resource effects.These cases leave room for improving the normalizer’s fidelity.

E Construction and Quality Control of Skill Discovery Benchmark · F Construction and Rubric of Risk Assessment Benchmark

The benchmarks use controlled construction and source-grounded quality procedures: Skill Discovery evaluates retrieval over generated requests, while Risk Assessment uses stratified sampling, multi-model labeling, review, and auditing. Both protocols emphasize observable evidence from original skill artifacts and define evaluation rules explicitly.

  • E Construction and Quality Control of Skill Discovery Benchmark: Skill Discovery samples 200 source skills from a 6,184-skill corpus and generates candidate requests from normalized evidence and raw SKILL.md views.Both generators use DeepSeek-V3.2 at temperature 0.7.
  • E Construction and Quality Control of Skill Discovery Benchmark: An independent DeepSeek-V3.2 realism pass rates candidate queries for realism, source grounding, and document proximity using only the query and raw source document.Ratings use a 1–5 scale, with higher document proximity indicating excessive closeness to source wording or implementation details.
  • E Construction and Quality Control of Skill Discovery Benchmark: The retrieval benchmark treats each source skill as the sole relevant item, counting near-equivalent neighboring skills as errors under a strict, unambiguous protocol.This may understate retrieval quality; all settings use the same embedding model and FAISS ranking pipeline.
  • E Construction and Quality Control of Skill Discovery Benchmark: A deterministic non-SSL source-outline control extracts headings, short bullets, interface- or resource-looking lines, and short prose spans from SKILL.md.The control tests whether SSL gains can be explained by generic source-text expansion alone; Table 8 reports MRR@50 by query type.
  • F Construction and Rubric of Risk Assessment Benchmark: Risk Assessment samples 252 skills from the same 6,184-skill corpus using high-, medium-, and low-signal strata based on tools, resources, branching, and loops.Stratification ensures sufficient observable risk-relevant evidence for evaluation.
  • F Construction and Rubric of Risk Assessment Benchmark: Three models produce six-dimension risk labels from complete SKILL.md and SSL records, with the SSL record serving as an inspection aid for candidate evidence.Each model outputs risk or no risk plus a short rationale, and outputs must parse into the six-dimension schema.
  • F Construction and Rubric of Risk Assessment Benchmark: Risk labels use first-round majority voting, with disputed dimensions re-labeled by the same three models and finalized by review-round majority vote.Dimensions receiving unanimous first-round votes are accepted directly.
  • F Construction and Rubric of Risk Assessment Benchmark: A sampled manual audit accepts positive labels only when the original SKILL.md visibly supports the claimed risk signal, and risk dimensions may be triggered independently.SSL-only evidence is insufficient; a single skill can receive multiple risk labels.

G Reproducibility, Release, and Responsible Use

The work is designed to be reproducible through inference-based artifact rebuilding and a planned release of schemas, records, labels, rubrics, and reconstruction scripts. Its responsible-use provisions preserve source attribution, limit redistribution, screen released artifacts, and clarify that risk labels are not runtime-safety guarantees or attack instructions.

  • Reproducibility: 98.16% of 6,300 collected skill directories produced valid normalized SSL records, with experiments conducted entirely through inference rather than training or fine-tuning.The final collection contains 6,184 valid normalized SSL records.
  • Reproducibility: Rebuilding the reported artifacts primarily incurs LLM inference costs for normalization, query construction, realism annotation, Risk Assessment gold construction, and fixed-judge evaluation.The reported run used about 6.3k first-pass normalizer calls, roughly 150 second-pass calls, 400 query-generation calls, and 806 realism-annotation calls.
  • Release: The planned release includes the SSL schema, prompting guidelines, normalized records, benchmark queries and labels, Risk Assessment rubrics and labels, and table-rebuilding scripts.The submission version does not provide a public code-and-data package; release is planned upon acceptance.
  • Responsible Use: Released artifacts and derived records retain source attribution and available license metadata, while unavailable or restrictive licenses lead to derived annotations and source pointers instead of raw-text redistribution.This approach respects existing assets while preserving access to structured evidence.
  • Responsible Use: Risk labels describe artifact-level evidence, not executable attack instructions or guarantees of runtime safety, and releases exclude or redact entries containing obvious private credentials or non-public personal data.Manual audits are author quality-control checks over artifacts and model outputs.

H Bootstrap Confidence Intervals for Main Results

This section reports paired non-parametric bootstrap confidence intervals for the two main empirical comparisons in Section 4. It defines query- and skill-level resampling procedures and uses 20,000 replicates with a fixed random seed.

  • Bootstrap procedure: The analysis estimates confidence intervals for Skill Discovery and Risk Assessment by resampling their respective evaluation units.Skill Discovery resamples queries, whereas Risk Assessment resamples skills and recomputes six-dimension binary macro F1.
  • Bootstrap procedure: For Skill Discovery, each bootstrap sample draws 431 queries with replacement and recomputes MRR@50.The query is the resampling unit for this comparison.
  • Reported comparisons: Table 11 reports paired bootstrap confidence intervals over differences between SSL-augmented inputs and corresponding text-only baselines.The intervals cover the main improvements reported in Section 4.

I Case Studies and Qualitative Analysis

The qualitative cases show SSL improving retrieval and risk assessment by exposing structured interface and risk-relevant signals, while also revealing that normalization can understate source-level security context. These examples make aggregate trends concrete through two positive cases and one counterexample.

  • Case selection and label order: The cases were selected from large positive evaluation deltas and one counterexample where adding SSL worsens the risk judgment.Binary risk labels are reported in the order data exfiltration, destructive behavior, privilege escalation, covert execution, resource abuse, and credential access.
  • Case 1: Interface signals recover a missed retrieval target: For an Excel-finance query, Desc + SSL-Rich ranks xlsx-official first versus 2,493 among 6,184 candidates for Desc Only.The full-document variants rank the skill 12th without SSL and 6th with SSL-Rich.
  • Case 2: Risk-relevant evidence changes a risk judgment: With SKILL.md + SSL, DeepSeek-V3.2 matches all six gold risk labels for gws-gmail, versus detecting only data-exfiltration risk from SKILL.md.The gold labels are (risk, risk, no, no, risk, no), covering data exfiltration, destructive behavior, privilege escalation, covert execution, resource abuse, and credential access.
  • Case 3: SSL can understate source-level context: For electric-proxy-auth, SKILL.md correctly detects data-exfiltration and credential-access risk, whereas SKILL.md + SSL predicts no risk on both dimensions.The source discusses server-side proxying, auth token injection, tenant isolation, and secret handling, while the structured record emphasizes validation and request construction.
Loading 2604.24026v4…