Source-linked AI summary
Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale
Hao Li, Chunjiang Mu, Jianhao Chen, Siyue Ren, Zhiyao Cui, Yiqun Zhang, Lei Bai, Shuyue Hu
TL;DR
The rapid growth and decentralization of agent skills creates challenges for discovering, managing, and composing capabilities at ecosystem scale. AgentSkillOS addresses these challenges with capability-tree organization, multi-skill DAG orchestration, and a benchmark for artifact-rich task evaluation. Across ecosystems of 200 to 200K skills, it outperforms flat invocation and skill-free baselines, with structured composition identified as key to skill utilization.
Problem
Rapidly expanding and decentralized skill ecosystems make targeted skill discovery, ecosystem management, and reliable multi-skill composition difficult.
Method
AgentSkillOS organizes skills in a recursively constructed capability tree, retrieves and orchestrates multiple skills through DAG-based pipelines, and evaluates them on 30 artifact-rich tasks using pairwise LLM judging with Bradley–Terry aggregation.
Results
Across ecosystem scales from 200 to 200K skills, AgentSkillOS consistently achieves the highest scores and substantially outperforms flat skill invocation and the skill-free baseline.
Takeaways & Limitations
The findings support structured retrieval and composition as important mechanisms for leveraging skills beyond what flat invocation or individual skills provide.
Takeaways & Limitations
The current framework assumes that skills are already collected and available, leaving automated discovery, quality assessment, and continuous integration for future work.
Abstract
from arXiv · showhide
The rapid proliferation of Claude agent skills has raised the central question of how to effectively leverage, manage, and scale the agent skill ecosystem. In this paper, we propose AgentSkillOS, the first principled framework for skill selection, orchestration, and ecosystem-level management. AgentSkillOS comprises two stages: (i) Manage Skills, which organizes skills into a capability tree via node-level recursive categorization for efficient discovery; and (ii) Solve Tasks, which retrieves, orchestrates, and executes multiple skills through DAG-based pipelines. To evaluate the agent's ability to invoke skills, we construct a benchmark of 30 artifact-rich tasks across five categories: data computation, document creation, motion video, visual design, and web interaction. We assess the quality of task outputs using LLM-based pairwise evaluation, and the results are aggregated via a Bradley-Terry model to produce unified quality scores. Experiments across three skill ecosystem scales (200 to 200K skills) show that tree-based retrieval effectively approximates oracle skill selection, and that DAG-based orchestration substantially outperforms native flat invocation even when given the identical skill set. Our findings confirm that structured composition is the key to unlocking skill potential. Our GitHub repository is available at:https://github.com/ynulihao/AgentSkillOS.
1 Introduction
AgentSkillOS addresses the challenges of discovering, managing, and composing skills in a rapidly expanding ecosystem through capability-tree organization and DAG-based orchestration. Across ecosystems of 200 to 200K skills, it improves skill invocation and task-output quality over flat and skill-free baselines.
- Motivation: The decentralized ecosystem’s scale makes it difficult to discover relevant skills, understand capability coverage, and address quality and reliability concerns.Users face fragmented, overlapping, heterogeneously named skills, while platform providers face governance challenges.
- Approach: AgentSkillOS organizes skills into a recursively partitioned capability tree and uses it to retrieve and orchestrate multiple skills for task-specific agents.The framework has Manage Skills and Solve Tasks stages, with the latter using multi-skill pipelines.
- Evaluation: The framework introduces a benchmark of 30 artifact-rich tasks spanning data computation, document creation, motion video, visual design, and web interaction.Tasks are curated by human experts around public skills and real-world deliverable requirements.
- Evaluation: The benchmark evaluates complete end-user-facing artifacts with LLM-based pairwise judgments in both orderings and Bradley–Terry aggregation.This protocol targets fine-grained quality assessment while mitigating position bias.
- Results: Across ecosystem scales from 200 to 200K skills, AgentSkillOS achieves the highest scores and substantially outperforms vanilla flat invocation and the skill-free baseline.The ablation shows that DAG orchestration remains superior even when vanilla invocation receives the identical oracle skill set.
- Results: The results identify structured skill composition, rather than skill availability alone, as central to unlocking the ecosystem’s potential.Tree-based retrieval approximates oracle selection, while DAG orchestration adds substantial performance gains.
2 Method
AgentSkillOS manages large skill ecosystems with a capability tree, then solves tasks by retrieving, orchestrating, and executing multiple skills through DAG-based pipelines.
- Manage Skills: AgentSkillOS organizes skills into a capability tree and uses it to support efficient retrieval before task execution.The tree is constructed offline through recursive partitioning, while retrieval proceeds from broad capability domains to individual skill partitions.
- Manage Skills: Capability Tree Construction: Node-level recursive categorization generates category groups and assigns skills to them, with breadth-first construction from the root.Separating group discovery from skill assignment is intended to reduce missed skills during tree construction.
- Manage Skills: Usage-Frequency Selection: When ecosystems exceed threshold K, the usage-frequency queue selects TopK skills plus user-selected skills for the tree, while remaining skills enter a dormant semantic index.Useful dormant skills can be added to the user-selected set and trigger a tree update.
- Solve Tasks: For each task, AgentSkillOS retrieves candidate skills, constructs alternative DAG orchestration plans, and executes the selected plan.The task-solving stages are retrieval, DAG-based orchestration, and multi-skill execution.
- DAG-based Skill Orchestration: The orchestration strategies trade off quality, efficiency, and simplicity by adding refinement stages, maximizing parallelism, or minimizing essential nodes.The LLM decomposes the task into sub-tasks, specifies dependencies and expected outputs, and organizes them into layered graphs.
- Multi-skill Task Execution: Execution follows DAG dependencies: same-layer nodes run in parallel, whereas nodes across layers run sequentially with prompts specifying each skill and sub-task.The benchmark framework converts diverse artifacts into LLM-evaluable formats before pairwise judging and Bradley–Terry aggregation.
3 Benchmark
The benchmark evaluates skill invocation on 30 human-crafted, artifact-rich tasks spanning five categories and diverse deliverable formats. An LLM judge compares outputs in both orderings, and Bradley–Terry modeling converts outcomes into system rankings.
- Tasks: 30 human-crafted tasks span data computation, document creation, motion video, visual design, and web interaction.The benchmark contains six tasks per category and targets real-world deliverables.
- Tasks: Tasks require polished artifacts in formats including PDF, PPTX, DOCX, HTML pages, videos, and generated images.Layout, design, aesthetics, correctness, and completeness matter beyond code or question-answering performance.
- Tasks: Task complexity is characterized by the number of required skills, expected output files, and distinct output formats.Figure 3 reports these three dimensions across the five benchmark categories.
- Evaluation: Pairwise LLM judging compares systems on correctness, completeness, quality, and aesthetics after converting non-text artifacts into consumable representations.Comparisons are conducted in both orderings to mitigate position bias.
- Evaluation: Conflicting judgments become ties, and all pairwise outcomes are aggregated into a win matrix for ranking N systems.Ties contribute 0.5 to both corresponding matrix entries.
- Evaluation: Bradley–Terry maximum-likelihood fitting produces latent system strengths that are linearly rescaled to [0, 100] as final ranking scores.The model is fit with the MM algorithm and Laplace smoothing with α = 1.
4 Experiments
Experiments across skill ecosystems of 200, 1K, and 200K skills show that AgentSkillOS consistently outperforms native invocation and skill-free generation. Ablations and qualitative cases attribute the gains to structured retrieval and DAG orchestration, whose strategies create distinct quality–efficiency trade-offs.
- Experimental Setup: The benchmark evaluates eight configurations across ecosystems of 200, 1K, and 200K skills, including AgentSkillOS variants, oracle settings, flat invocation, and a skill-free baseline.AgentSkillOS uses capability-tree retrieval and DAG orchestration, while Claude Code variants invoke skills flatly.
- Overall Results: AgentSkillOS variants consistently achieve the highest Bradley–Terry scores across all ecosystem sizes, outperforming w/ Full Pool and the skill-free Vanilla baseline.The advantage persists as the ecosystem scales, while direct provisioning of the full pool remains limited.
- Ablation Results: Tree-based retrieval approximates oracle skill selection, while removing DAG orchestration consistently degrades performance even when the oracle skill set is provided.Quality-First closely approaches Quality-First (Oracle), whereas flat invocation with w/ Oracle Skills performs worse.
- Orchestration Structure: Quality-First produces the largest and deepest DAGs, Efficiency-First favors wider and shallower graphs, and Simplicity-First yields the most compact graphs.These graph structures correspond to preparation and refinement, parallel execution, and minimal-node objectives, respectively.
- Qualitative Results: Across three representative tasks, AgentSkillOS produces richer web pages, polished educational media, and professionally designed presentations than the vanilla baseline.The examples include layered scrolling effects, a Manim-rendered derivative animation with a PDF handout, and quantum-computing slides with Bloch sphere visualizations.
5 Related Work
Prior agent systems extend LLM capabilities through tools and skills, but large decentralized skill ecosystems create discovery and evaluation challenges. Existing benchmarks largely use fixed tool sets and endpoint metrics, leaving open-ended, artifact-rich skill use less examined.
- Tool-Augmented Agents: Tool-augmented agents access APIs, functions, and execution environments to compute, retrieve information, and interact with external software.Skills extend this paradigm by encapsulating composite workflows and domain-specific procedures for modular reuse.
- Multi-Skill Ecosystems: Large decentralized skill repositories make relevant-skill identification difficult for human users, especially when entries contain overlapping capabilities.This creates a central challenge for scalable skill ecosystem management.
- Benchmark Limitations: Existing benchmarks constrain agents to predefined finite tool sets and emphasize endpoint success metrics over qualitative artifact assessment.Consequently, they provide limited insight into open-ended and generative skill use.
- Benchmark Limitations: SkillsBench measures marginal gains from curated skills but focuses predominantly on terminal tasks, leaving multimodal and artifact-rich outputs underexplored.This differs from evaluations designed around diverse deliverables and cross-composed skills.
6 Conclusions and Future Work
AgentSkillOS addresses skill management and task solving through capability-tree organization, DAG-based orchestration, and automated execution. Experiments show that tree-based retrieval approximates oracle selection and DAG orchestration outperforms flat invocation, while future work includes automated skill collection and self-evolution.
- Tree-based retrieval effectively approximates oracle skill selection, enabling agents to discover relevant skills from large-scale ecosystems.
- DAG-based skill orchestration substantially outperforms native flat invocation even with the identical skill set.
- The current framework assumes that skills are already collected and available, leaving automated collection as a future direction.Proposed collection includes discovering skills from open sources, assessing quality, and continuously integrating skills into the ecosystem.
- Skills could support self-evolution by enabling agents to refine instructions, fix failure modes, and generate higher-quality variants from execution feedback.