Source-linked AI summary
OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents
Jiahao Ying, Boxian Ai, Wei Tang, Siyuan Liu, Yixin Cao
TL;DR
OpenSkillEval addresses limited evidence on how models, agent frameworks, and skills interact in real-world downstream tasks. It dynamically evaluates them across evolving task instances and finds that skill availability does not guarantee effective use, with benefits depending strongly on the model and framework.
Problem
Systematic evidence is limited on how agent frameworks and models interact with skills and how to evaluate skill effectiveness for downstream task selection.
Method
OpenSkillEval dynamically generates realistic task instances, curates community skills, and compares models and agent frameworks with and without skills across downstream applications.
Results
Skill availability does not guarantee effective use, while skill benefits depend strongly on the model and agent framework and popular skills yield limited or inconsistent gains.
Takeaways & Limitations
Dynamic, task-grounded evaluation can inform practical selection of agents and skills for downstream applications.
Takeaways & Limitations
The evaluation covers only a subset of available skills and abstracts away aspects of human-agent interaction relevant to deployment.
Abstract
from arXiv · showhide
Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improving agent performance on real-world downstream tasks. However, as the open-source skill ecosystem rapidly expands, it remains unclear how different models and agent frameworks interact with skills, how to evaluate skill quality, and how users should select skills under practical cost-performance trade-offs. In this paper, we present \textsc{OpenSkillEval}, an automatic evaluation framework for both skill-augmented agent systems and the skills themselves. Instead of relying on static benchmarks, \textsc{OpenSkillEval} automatically constructs realistic task instances from evolving real-world artifacts across five categories of downstream applications: presentation generation, front-end web design, poster generation, data visualization, and report generation. It further collects and organizes community-contributed skills for controlled comparison under unified task settings. Using more than 600 dynamically generated task instances and 30 open-source skills, we conduct a systematic evaluation of state-of-the-art models and agent frameworks. Our results show that skill availability does not guarantee effective skill usage, that the benefit of skill augmentation depends strongly on both the underlying model and the agent framework, and that many publicly popular skills do not consistently outperform base agents without skills. These findings highlight the need for dynamic, task-grounded evaluation and provide practical insights into the design, selection, and deployment of skills for LLM agents. Additional cases and benchmark resources are available on the project website: https://yingjiahao14.github.io/OpenSkillEval-Web/.
1 Introduction
The rapid growth of skills for LLM agents creates a need to understand their effectiveness across agent frameworks and downstream tasks. OpenSkillEval addresses this need through dynamic evaluation of skill-augmented agents and the skills themselves.
- Challenges: Open-ended agentic tasks make overall behavior difficult to predict, especially in challenging settings where intrinsic capability may be insufficient.The introduction situates this problem within complex downstream applications such as report generation, document management, and web design.
- Motivation: Community participation has rapidly expanded the ecosystem by creating and integrating many skills for LLM agents.This growth motivates systematic assessment of how skills affect downstream task completion.
- Challenges: Systematic evaluation remains lacking for comparing agent frameworks and their interactions with added skills during execution.This makes actual effectiveness difficult to assess and complicates users’ selection of suitable agents for downstream tasks.
- Proposed Framework: OpenSkillEval automatically evaluates both skill-augmented agent systems and the skills themselves in downstream applications.It dynamically generates test cases rather than relying on static benchmarks, aiming for a more realistic, timely, and comprehensive evaluation setting.
2 OpenSkillEval Framework
OpenSkillEval is a maintainable framework for evaluating LLM-agent skills on realistic downstream tasks. It automatically generates task instances and evaluates both skill use through execution traces and the quality of resulting artifacts.
- Framework Overview: OpenSkillEval decomposes evaluation into automatic test-case generation and complementary analyses of agent trajectories and final artifacts.The framework is designed for sustainable evaluation of a rapidly growing open skill ecosystem in real-world downstream applications.
- Task Categories: Five task categories cover presentation generation, front-end web design, poster generation, data visualization, and report generation.These tasks represent common applications, require multi-step reasoning and tool use, and produce directly evaluable artifacts.
- Task Generation: Task instances are grounded in evolving real-world artifacts or data, then converted into structured specifications and natural-language instructions.Sources include webpages, documents, existing websites, poster materials, visualization taxonomies, and real-world tabular datasets.
- Task Generation: Each generated instance retains a source package, structured task specification, and natural-language instruction, with validation performed without relying on reference outputs.A verifier LLM assesses whether the specification and instruction are information-complete.
- Evaluation Pipeline: Artifact analysis uses task-specific criteria to automatically assess final-output quality across downstream application categories.The metric design draws on PPTEval, GenEval, and WebArena while adapting evaluation to each task’s characteristics.
3 Experimental Results
OpenSkillEval evaluates multiple agent–model systems across five downstream task categories and finds that skill benefits depend strongly on agent behavior, model–framework pairing, and skill design. The strongest systems combine high task performance with efficiency, but skills can be ignored, incompletely followed, or trade completeness and diversity for visual consistency.
- Skill usage: 100 randomly sampled task instances show that skills often remain effectively unused when merely placed in an agent-accessible environment.The experiment prompts agents to complete identical downstream tasks and analyzes execution traces to determine whether injected skills are used.
- Skill usage: Even after explicitly reading skills, agents skip prescribed steps or substantially deviate from intended procedures.Explicitly forcing skill usage largely mitigates the problem of non-use but does not eliminate autonomous behavior.
- Agent and task performance: Claude 4.6 with Claude Code and GPT-5.5 with Codex achieve the strongest overall performance and stability across the five task categories.Presentation and poster generation are especially challenging, while front-end web design is generally stronger but remains limited by weaker responsive design and visual-balance issues.
- Efficiency and cost–performance: Codex consistently uses the fewest tokens, while GPT-5.5 combines this efficiency advantage with strong overall task performance.Poster and presentation generation are generally the most token-intensive, whereas data visualization and report generation are less expensive for most agents.
- Skill effects and design: Skills improve visual-design scores in all 6 presentation-generation settings and roughly half of poster-generation and front-end web-design settings, but richer constraints can reduce completeness and diversity.Stronger procedural constraints tend to produce more consistent but less diverse outputs; frontend-slides can omit content when source material is dense.
4 Human Evaluation
The study supplements its fully automatic evaluation with human assessment of generated task-instance quality and automatic-judgment reliability. Four senior NLP researchers annotate 100 randomly sampled instances balanced across five task categories.
- Four senior natural language processing researchers serve as human annotators.
- The human evaluation randomly samples 100 task instances balanced across five task categories.
- The evaluation assesses both generated task-instance quality and the reliability of automatic judgments.
5 Related Work
Prior work formalizes portable agent skills, studies their construction and evaluation, and develops benchmarks for realistic agent workflows. OpenSkillEval extends dynamic benchmarking to controlled comparisons of community-contributed skills on downstream application tasks.
- Agent skills and procedural augmentation: Agent Skills package procedural expertise as portable, dynamically loadable SKILL.md artifacts, motivating work on automatic skill construction, discovery, refinement, and evaluation.SkillsBench reports substantial but highly variable gains from curated skills, while self-generated skills provide little average benefit.
- Agent skills and procedural augmentation: PinchBench and WildClawBench stress-test agent–skill combinations in realistic workflows, whereas OpenSkillEval compares community-contributed skills within shared downstream application tasks.The comparison is designed to assess skills under the same task settings rather than across unrelated workflows.
- Benchmarks for LLM agents: Existing agent benchmarks cover software engineering, web navigation, and generalist multi-step reasoning, but static task pools can saturate quickly and risk contamination.Dynamic benchmarks address this limitation by releasing fresh problems on a rolling basis.
- Benchmarks for LLM agents: OpenSkillEval extends the dynamic benchmarking philosophy to skill-augmented agents, enabling controlled comparison of competing skills under identical task settings.This approach applies dynamic evaluation to downstream application tasks rather than relying solely on static benchmark pools.
6 Discussion
OpenSkillEval enables large-scale, continuously updated assessment of the rapidly evolving open skill ecosystem, but its current design has practical limitations. Cost constraints restrict skill coverage, while automation abstracts away part of human-agent interaction.
- Discussion: OpenSkillEval makes large-scale and continuously updated evaluation of the rapidly evolving open skill ecosystem feasible.The framework is designed to support scalable assessment over time.
- Discussion: Practical cost constraints prevent the current version from covering a more exhaustive set of available skills.Limited coverage is identified as a primary design limitation.
- Discussion: Automation enables scalability and reproducibility but inevitably abstracts away part of human-agent interaction.The passage presents this abstraction as a limitation of automated evaluation.
7 Conclusion
OpenSkillEval is an automatic framework for evaluating skill-augmented LLM agents and open-source skills on real-world downstream tasks. It supports realistic, scalable analysis, while experiments show that skill availability alone does not ensure effective skill use.
- 7 Conclusion: OpenSkillEval automatically evaluates skill-augmented LLM agents and open-source skills in real-world downstream tasks.The framework is presented as the paper’s central contribution.
- 7 Conclusion: Automatically constructing task instances from evolving artifacts enables more realistic and scalable analysis of skill effectiveness.The framework also collects community-contributed skills and evaluates agent trajectories and final outputs.
- 7 Conclusion: Skill availability does not guarantee effective skill use.This conclusion is supported by the reported experiments.
A Technical Appendices and Supplementary Material · A.1 Experimental Environment
The experimental environment standardizes execution through Harbor, shared containerization, network access, and unified timeouts. Agents receive structured task inputs and category-specific materials, with Claude Opus 4.6 as the primary model and Gemini 3.1 Pro used for ablations.
- A.1 Experimental Environment: Harbor serves as the unified execution framework for all experiments.The authors acknowledge Harbor’s open-source maintainers and continued development.
- A.1 Experimental Environment: All agents run in the same ubuntu:24.04 containerized environment.This standardizes the runtime environment across experiments.
- A.1 Experimental Environment: Runtime evaluation provides network access and unified timeout settings for fairness.The configured values are build_timeout_sec = 1800.0 * 5 and timeout_sec = 900.0 * 5.
- A.1 Experimental Environment: Each task instance supplies a natural-language instruction, structured specification, and required source materials or data files.The instruction is stored as Instruction.md and the task specification as task_input.json.
- A.1 Experimental Environment: Additional inputs vary by task category, including source_brief.md for presentation generation and tabular datasets for report generation.Presentation briefs may contain tables, figures, and other supporting content.
- A.1 Experimental Environment: Claude Opus 4.6 is the primary evaluation model, while Gemini 3.1 Pro is used for ablation analysis.The agent-as-judge setting deploys Claude Opus 4.6 within Claude Code in a Docker-based environment.
A.2 Task-Specific Evaluation Inputs for VLM-Based Judging
The evaluation uses task-specific artifact representations and granularity for VLM-based judging across five downstream tasks. It combines visual inspection with interaction- or agent-based verification where static artifacts cannot capture usability, data correctness, or factual consistency.
- Presentation Generation: Presentation outputs are converted from .pptx to PDF with Microsoft PowerPoint 16.108.1, rasterized into per-slide PNGs, and evaluated for content quality and visual design.The pipeline avoids Linux-friendly converters because they were found to introduce substantial rendering errors.
- Front-end Web Design: Websites are evaluated through vision- and DOM-grounded browsing that generates screenshots and structured reports covering page loading, discovered sections, and missing content.Interaction-based browsing captures multiple pages, navigation paths, and interaction states that static screenshots cannot represent.
- Poster Generation and Data Visualization: Posters and visualizations are supplied directly as single PNG images and judged against task specifications such as required sections and goal-insight satisfaction.These criteria assess whether the artifact fulfills the requested content and communicative goals.
- Poster Generation and Data Visualization: Data visualizations additionally undergo agent-based verification using execution trajectories and source_data.json to check chart-construction steps and data usage correctness.The resulting report focuses on step-level correctness of the intended data usage.
- Report Generation: Reports are converted from HTML to PDF and evaluated with full-page and segmented screenshots, while agent-based checks assess numerical claims and factual consistency.Full-page renderings support global structure assessment, and segmented screenshots enable fine-grained content inspection.
- Evaluation Matrix: Evaluation dimensions are scored by either a VLM judge on a 1–5 scale or an evaluation agent using programmatic verification, with pass rates mapped to 1–5 via 4x + 1.The evaluation matrix spans the five OpenSkillEval task categories.
A.3 Task Input Schemas
Task input schemas convert collected sources into task-specific structured specifications that bridge source materials and executable benchmark instances.
- Task Input Schemas: Collected sources are transformed into task-specific structured specifications during automatic case generation.Each specification is stored as task_input.json and follows the corresponding schema.
- Task Input Schemas: The schemas define the core information required for each downstream task.
- Task Input Schemas: These specifications serve as an intermediate representation from source materials to executable task instances.
1. Data Visualization
The data-visualization task requires the agent to express one insight per case while selecting the chart type autonomously. Optional styling can specify a scientific, publication-ready presentation for researchers and policy makers.
- Task specification: Optional style settings define a scientific theme, a researcher-and-policy-maker audience, and a clean, publication-ready tone.The style block can be omitted to test agent autonomy.
- Task specification: Each case requires one insight, with the agent choosing the chart type.The required goal is to show the accelerating global warming trend over 75 years.
2. Poster Generation … 5. Web Design
The evaluation framework specifies task structures and assesses generated artifacts across poster, presentation, report, web-design, and data-visualization settings. Human evaluation and rubric-based prompts jointly measure instruction quality, content, visual design, completeness, accuracy, responsiveness, and cost-performance trade-offs.
- A.4 Human Evaluation: Human evaluation randomly samples 100 task instances balanced across five categories and uses four senior NLP researchers to assess task-instance and artifact quality.Evaluators receive task inputs and reference data, while artifact assessment adopts the VLM-based automatic-evaluation prompts.
- 2. Poster Generation: Poster-generation evaluation scores visual design and task completeness on 1–5 scales, emphasizing hierarchy, polish, readability, and satisfaction of all requirements.The visual-design rubric covers color, layout, typography, consistency, and professional polish; completeness checks whether all task requirements are met.
- 3. PPT Generation: Presentation evaluation separately measures content quality, visual design, and full-deck completeness, distinguishing lost content from visual flaws and requiring every specified element.Content quality considers message clarity, density, writing, and text–visual balance; design covers styling and layout; completeness checks required content presence, while blank charts count as missing.
- 4. Report Generation: Report evaluation rates writing and analytical depth, visualization quality, and substantive completeness, including whether KPI segmentations and requested content are actually addressed.The rubrics reward data-driven insights, appropriate charts, readable annotations, and fulfillment of every requirement beyond merely including section headings.
- 5. Web Design: Web-design evaluation assesses per-page visual execution and device-specific responsiveness, including typography, layout, overflow, navigation, touch targets, readability, and viewport adaptation.Placeholder media is expected when assets are absent and should be judged for styling rather than penalized for not being real; responsiveness requires touch targets of >=44px.