PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems

Hongji Pu, Xinyuan Song, Liang Zhao

arXiv:2605.13716v1cs.SEcs.MA

TL;DR

Skill libraries can accumulate persistent defects that task-time retrieval, planning, and repair do not directly maintain. SkillOps addresses this gap with a plug-in framework that represents skills and their relations explicitly, diagnoses library health, and returns a maintained library for existing agents. On ALFWorld, it reports strong standalone and retrieval-heavy gains while using nearly zero library-time LLM calls, though its current evaluation remains limited in scope.

  • Problem

    Skill technical debt consists of persistent library defects that may not break individual skills locally but can reduce future retrieval, composition, and execution reliability, while existing work focuses mainly on task-time use.

  • Method

    SkillOps represents skills as typed Skill Contracts, organizes them in a Hierarchical Skill Ecosystem Graph, diagnoses library health, and applies typed maintenance actions through a drop-in interface.

  • Results

    79.5% task success on ALFWorld made SkillOps the standalone leader, while retrieval-heavy baselines improved by +0.68 to +2.90 percentage points and maintenance used nearly zero library-time LLM cost.

  • Takeaways & Limitations

    Skill libraries can be treated as managed software assets, with maintenance benefits depending on the downstream agent: retrieval-only agents benefit most, while self-repairing agents may conflict with external maintenance.

  • Takeaways & Limitations

    The evaluation is mainly based on a half-synthetic ALFWorld library, and the rule-based maintenance loop may miss semantic redundancy or complex conflicts requiring deeper reasoning.

Abstract

from arXiv · show

Large language model agents increasingly rely on skill libraries for multi-step tasks, yet these libraries can accumulate persistent defects as skills are added, reused, patched, and linked to changing dependencies. We call this failure mode skill technical debt: library-level defects that may not break a single skill locally but can harm future retrieval, composition, and execution. Existing skill-based agents mainly focus on task-time retrieval, planning, and repair, while library-time maintenance remains underexplored. We propose SkillOps, a method-agnostic plug-in framework for maintaining skill libraries. SkillOps represents each skill as a typed Skill Contract (P, O, A, V, F), organizes skills with a Hierarchical Skill Ecosystem Graph, and diagnoses library health across utility, compatibility, risk, and validation dimensions. Given a raw skill library, SkillOps produces a maintained library that can be used by existing retrieval or planning agents without changing their internal code. On ALFWorld, SkillOps achieves 79.5 percent task success as a standalone agent, outperforming the strongest baseline by 8.8 percentage points with no additional task-time large language model calls. As a plug-in layer, it improves retrieval-heavy baselines by 0.68 to 2.90 percentage points. The current rule-based maintenance implementation uses nearly zero library-time large language model calls or tokens, showing that skill-library maintenance can be added as a low-overhead architectural layer.

1 Introduction

SkillOps treats persistent defects in skill libraries as technical debt and adds a library-time maintenance layer alongside task-time retrieval, planning, and repair. It transforms raw libraries into maintained assets that existing agents can use, with strong standalone and retrieval-heavy results at low LLM cost.

  • Motivation: Skill technical debt comprises persistent library defects that can reduce future retrieval, composition, and execution reliability even when individual skills work locally.Examples include redundancy, missing validation, interface drift, and stale implementations.
  • Motivation: Existing skill-based agents emphasize task-time retrieval, composition, validation, and repair rather than directly maintaining the library.Task-time repair can fix an episode without updating the underlying library defect.
  • Approach: SkillOps diagnoses raw libraries and applies typed maintenance actions so existing retrieval-based or task-oriented agents can use a cleaned library without internal code changes.The framework combines typed skill contracts, graph organization, health diagnosis, and feedback-driven maintenance.
  • Results: 79.5% task success on ALFWorld made SkillOps the standalone leader, exceeding the strongest baseline by +8.8 percentage points with zero additional task-time LLM calls.The introduction reports this standalone result as part of the framework’s empirical evaluation.
  • Results: +0.68 to +2.90 percentage points improved retrieval-heavy baselines across the reported settings, while maintenance required nearly zero library-time LLM cost.The reported maintenance implementation was neutral-to-negative in task-time token usage.

2 Problem Setup

SkillOps models each skill as a typed executable contract and organizes the library as a graph of typed inter-skill relations. These representations support typed maintenance actions for redundancy, repair, retirement, validation, and interface compatibility.

  • Skill Contract: A Skill Contract represents each skill as (P, O, A, V, F): preconditions, operation, produced artifacts, validators, and known failure modes.An empty validator set constitutes a validation gap and prevents local correctness checking.
  • Hierarchical Skill Ecosystem Graph: The Hierarchical Skill Ecosystem Graph represents a library as skills and typed directed relations between them.The supplied setup identifies dependency, compatibility, redundancy, and alternative relation types.
  • Hierarchical Skill Ecosystem Graph: Dependency edges connect produced artifacts to downstream preconditions, while compatibility edges connect compatible output and input types.These relations describe how skills can be chained or used together.
  • Hierarchical Skill Ecosystem Graph: Redundancy edges identify equivalent interfaces, whereas alternative edges identify skills pursuing the same goal through different operations.The distinction supports separate handling of duplication and functional alternatives.
  • Maintenance Actions: SkillOps uses typed actions to merge redundant skills, repair operations, retire obsolete or failing skills, add validators, and insert adapters for incompatible interfaces.Instantiation binds task-specific arguments to parameterized skills at task time.

3 SkillOps: A Self-Maintaining Skill Ecosystem

SkillOps combines graph-structured skill planning with library-time diagnosis and typed maintenance. Its task-time loop builds executable plans with compatibility checks and local repair, while its library-time loop updates persistent skill-library health.

  • Task-Time Loop: Skill matching filters skills by relevance and satisfied preconditions before dependency stitching constructs a candidate plan.The relevance score combines BM25 and semantic relevance, and only skills above threshold θ are retained.
  • Task-Time Loop: Plans accept transitions only when dependency and artifact-type compatibility constraints are satisfied, preventing interface mismatches.When compatibility is missing, SkillOps can insert an adapter whose output type satisfies the downstream precondition.
  • Architecture: SkillOps organizes skills as a Hierarchical Skill Ecosystem Graph with contract-level nodes and typed relationships between skills.The graph includes internal contract graphs and external dependency, compatibility, redundancy, and alternative edges.
  • Task-Time Loop: During execution, SkillOps performs local repair and records unrecoverable failures for subsequent library-time maintenance.Repair can substitute an alternative skill or re-invoke repair using the observed error trace as feedback.
  • Library-Time Loop: Library health is diagnosed across utility, redundancy, compatibility, failure risk, and validation gaps, covering retrieval, interface, execution, and validation defects.The health dimensions target low-value skills, near-duplicates, interface mismatches, runtime failures, and missing validators.
  • Library-Time Loop: CGPD propagates upstream risk through dependency edges, while typed actions such as merge, repair, retire, add_validator, and add_adapter update the library.Risk combines local risk with the maximum propagated risk among upstream parents and converges to a unique fixed point.
  • Plug-in Interface: SkillOps is a pure library transformation that lets existing retrieval and planning agents use a maintained library without changing downstream code.The maintained library can be attached to BM25, dense, hybrid, LLM, graph-based, or self-repairing agents.

4 Experiments

The experiments evaluate SkillOps on ALFWorld using controlled comparisons across agent and library configurations. The protocol uses curated skills, degraded variants, common backbones, and strict-order task success.

  • Evaluation Setup: SkillOps is evaluated on ALFWorld, a text-only household-manipulation benchmark with multi-step tasks and structured action sequences.The benchmark supports testing skill retrieval, composition, and maintenance.
  • Evaluation Setup: The skill library starts from 229 curated SkillsBench skills and adds degraded variants for larger library scales.Variants represent redundant clones, stale clones, missing validators, missing artifacts, wrong interfaces, and over-specialized skills.
  • Baselines: The comparison includes flat prompting, LLM-based selection, retrieval-based selection, and dependency-only graph-planning baselines.These baselines cover distinct approaches to skill selection and planning.
  • Baselines: All baselines use the same GPT-4o-mini backbone, skill library, and gold-argument assumption for controlled comparison.The GoS_Style and GraSP_Style systems are clean reproductions because official code was unavailable.
  • Metrics and Protocol: Task Success Rate is the fraction of instances whose action sequence exactly matches the annotated sequence under ALFWorld’s strict-order grader.Evaluation uses three library seeds with 185 task instances per seed and pooled Wilson score 95% confidence intervals.

5 Results

SkillOps improves standalone and plug-in task success while adding little maintenance or task-time cost. Its benefits are strongest for retrieval-heavy agents and remain robust as libraries grow noisier, with ablations showing both planning and maintenance components matter.

  • Standalone performance: 79.5% SR: SkillOps outperforms LLM_Skill_Planner by +8.9pp on ALFWorld with a 200-skill library.Results are averaged across three seeds with zero standard deviation.
  • Plug-in effectiveness: +2.90pp: Hybrid Retrieval gains the most from SkillOps, followed by Dense Only at +1.12pp and BM25 Only at +1.00pp.The plug-in comparison keeps downstream agent code unchanged.
  • Cost: 24 of 35 cells decrease in task-time tokens after maintenance, while only 7 increase.The largest decrease is −3.95% for Dense_Only at lib=1000; pruning reduces redundant or degraded candidates before retrieval.
  • Scale sensitivity: 80.5% SR: SkillOps remains stable at the largest library scale under increasing degradation density and leads the next-best baseline by more than 31pp.Task-time-only baselines are more sensitive to noisy candidate pools.
  • Ablations: Removing the Task-Time Loop drops SR from 79.5% to 15.7%, while removing the Library-Time Loop reduces SR to 71.9%.Graph ablations also reduce SR to 64.6% without external relations and 72.2% without internal skill contracts.

6 Conclusion

SkillOps treats LLM-agent skill libraries as managed software assets rather than static retrieval pools. Its method-conditional results show that maintenance helps retrieval-only agents most, while LLM-planning agents are mostly flat and self-repairing agents may conflict with external maintenance.

  • Conclusion: SkillOps combines typed contracts, a Hierarchical Skill Ecosystem Graph, and observable-rule-driven maintenance through a drop-in interface.The framework is presented as a library-time maintenance layer for LLM agent skill libraries.
  • Conclusion: Maintenance benefits are method-conditional: retrieval-only agents benefit most, LLM-planning agents are mostly flat, and self-repairing agents may conflict with external maintenance.The conclusion reports this pattern alongside improved task success and nearly zero LLM calls in the rule-based maintenance loop.

7 Limitations

The evaluation is constrained by structured inputs, limited benchmark coverage, and the reasoning limits of the rule-based maintenance loop. The supplied procedural passages describe the algorithms and complexity but do not add separate scope limitations.

  • Input assumptions: SkillOps may require structured skill contracts and, in some settings, gold PDDL-style arguments unavailable in real deployments.These requirements constrain applicability beyond the evaluated setup.
  • Evaluation scope: The evaluated library is half-synthetic and based mainly on ALFWorld, so broader benchmarks and real long-running agent logs are needed.This limits how broadly the reported results can be generalized.
  • Maintenance limitations: The rule-based V4 maintenance loop may miss semantic redundancy or complex skill conflicts requiring deeper reasoning.Its nearly zero LLM cost therefore comes with a stated reasoning limitation.
  • Current implementation: CGPD does not improve task success in the current setup because validator fields are not yet consumed during plan-time skill selection.This is a limitation of the current implementation rather than a general claim about CGPD.

B Dataset Details

The evaluation uses ALFWorld and SkillsBench-derived libraries, including synthetically degraded variants that model common skill technical-debt patterns. The work is situated among prior approaches to skill mining, retrieval, validation, and maintenance.

  • ALFWorld: ALFWorld is a text-only household-manipulation benchmark derived from ALFRED, with 185 evaluation tasks from valid_unseen and some valid_seen instances.Task horizons range from 3 to 20 high-level actions, with median 7 and p90 12.
  • SkillsBench Library: SkillsBench provides 229 curated SKILL.md files across 88 tasks and 58 categories as the clean source library.Larger libraries add synthetically degraded variants representing redundant clones, stale clones, missing validators, missing artifacts, wrong interfaces, and over-specialized skills.
  • Library Construction: The synthetic library construction increases scale by injecting common technical-debt patterns into real SkillsBench skills.These variants support controlled evaluation of maintenance under degradation pressure.
  • Related Work: Prior work spans flat textual skill retrieval, one-shot skill mining, failure-driven rewriting, static dependency graphs, and validation or memory structures.SkillOps differs by targeting continuous library-time maintenance with typed inter-skill structure.

D Implementation Details

The experiments use controlled LLM settings, strict-order plan evaluation, multiple seeds, and scaling analyses across library sizes and composition conditions. Implementation details include rule-based maintenance, synthetic degradation, and matched-information probes.

  • Language Model: All LLM-based methods use gpt-4o-mini through the OpenAI Chat Completions API with disk-based SHA-256 caching.Reported API costs are computed from actual evaluation calls.
  • Hyperparameters: SkillOps uses θ = 0.5 for health diagnosis, retrieves k = 10 candidates with BM25, filters five candidates semantically, and allows two local repairs.Adapter insertion is triggered below 0.3 Jaccard similarity between relevant type fields.
  • Baseline Implementations: The baselines include ReAct, LLM_Skill_Planner, and Hybrid_Retrieval with distinct prompting, semantic-ranking, or hybrid lexical retrieval procedures.ReAct receives the full library, while the other baselines select five skills for planning.
  • Plan Evaluation: A task succeeds only when the predicted high-level action sequence exactly matches the annotated high_pddl sequence under strict-order evaluation.This protocol defines Task Success Rate as the fraction of exactly matched task instances.
  • H1 Results: H1 evaluates a 200-skill library across three seeds, with SkillOps_Full achieving 79.5% in every seed and remaining above the baselines.The reported stability indicates the result is not attributable to one favorable initialization.
  • Degradation and Maintenance: The study constructs larger libraries by injecting six programmatic degradation types into real SkillsBench files and implements maintenance through observable-signal action stubs.Actions include repair, add_validator, merge, and other typed operations triggered by signals such as hash collisions, missing validators, and type mismatches.
  • Scaling Protocol: Scaling analyses compare matched-information blind settings, gold-argument settings, and balanced 50/50 healthy/degraded composition across nine library sizes.The balanced setting separates larger candidate pools from increasing degradation density.
  • P0 Probe: In the matched-information probe, SkillOps_blind reaches 77.3% SR versus 70.8% for LLM_Skill_Planner, while gold-argument SkillOps matches LLM_Skill_Planner_pddl at 79.5% SR.With gold structured arguments and contracts, SkillOps also records 77.0 subgoal SR versus 74.4.

I H3 Ablation Study

The H3 ablations identify argument binding, task-time planning, and contract structure as central components, while library-time actions add maintenance capacity for larger or more degraded libraries. Token measurements and k-sensitivity tests further examine the source of SkillOps’s performance and cost profile.

  • Component Ablations: Argument binding, task-time planning, and contract structure are central to SkillOps performance in the component ablations.Library-time actions provide additional maintenance capacity as libraries become larger or more degraded.
  • Token Scaling: SkillOps emits very little LLM tokens per task across the evaluated library scales, whereas LLM-based baselines retain nonzero prompt and completion token costs.The comparison covers five representative scales in the token table and all evaluated scales in the visualization.
  • k-Sensitivity: Increasing k from 6 to 24 improves some retrieval baselines, especially Hybrid and GoS, but does not close the gap to SkillOps.At lib=200, the best k = 24 baseline reaches 51.9% SR versus SkillOps at 79.5%; at lib=2000, the corresponding values are 43.8% and 80.5%.
  • k-Sensitivity: SkillOps remains stronger across all tested library sizes while retrieval baselines respond unevenly to increasing k.The figure summarizes this comparison across representative scales.

J.3 Maintenance Overhead: Task-Time, Library-Time, and Amortization

Maintenance is evaluated for task-time token changes, one-time library-time cost, and amortization across downstream tasks. The results show mostly neutral or reduced task-time tokens and a small maintenance pass, alongside graph-aware risk propagation for preventive actions.

  • Task-Time Cost: Across 35 baseline-scale cells, maintained libraries reduce task-time tokens in 24 cells and are nearly unchanged in 4.Pruning and merging low-quality candidates reduce redundant skill descriptions for most retrieval-based methods.
  • Library-Time Cost: One maintenance pass at N = 2000 uses 9 LLM calls, about 10.8K tokens, and an estimated cost of $0.0026.Most actions remain rule-driven, and the one-time cost is amortized over downstream tasks.
  • Library-Time Cost: The Library-Time Loop adds only a very small maintenance overhead because merge, retire, repair, validator insertion, and adapter insertion use observable signals or sibling inheritance.Limited LLM usage is reserved for compact contract-level edits when needed.
  • CGPD Guarantees: Because α < 1, the CGPD update is a contraction and converges to a unique fixed point under the stated norm.The paper also notes finite-pass computation for acyclic dependency graphs.
  • Graph-Aware Diagnosis: CGPD propagates risk along dependency edges so downstream skills can receive preventive maintenance when upstream artifacts are unreliable.The method identifies skills for actions such as validator or adapter insertion rather than diagnosing each skill independently.
  • Risk Propagation Model: CGPD updates downstream risk using a weighted worst-upstream-risk rule, with α controlling propagated upstream influence.The maximum over parent risks reflects the possibility that one unreliable required artifact makes a downstream skill unsafe.
  • CGPD Algorithm: After convergence, CGPD flags high-risk skills without validators for add_validator, making maintenance decisions graph-aware.A skill may be selected because it inherits upstream risk even when its local risk is insufficient by itself.

K.2 CGPD Results

CGPD is evaluated against basic SkillOps maintenance at two large library sizes, with consistent improvements that support dependency-aware risk propagation.

  • CGPD-augmented maintenance is compared with basic SkillOps at library sizes 1000 and 2000, averaged over 3 seeds.
  • +0.5pp at lib=1000 and +0.6pp at lib=2000 are the reported gains from adding CGPD.
  • The results support dependency-aware risk propagation, which identifies downstream risks from upstream skills and enables preventive maintenance before task failures.
Loading 2605.13716v1…