PaperlayerView exact revision on arXiv

Source-linked AI summary

How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings

Yujian Liu, Jiabao Ji, Li An, Tommi Jaakkola, Yang Zhang, Shiyu Chang

arXiv:2604.04323v1cs.CL

TL;DR

Existing skill benchmarks provide narrowly tailored skills directly to agents, leaving realistic retrieval and imperfect skill matching insufficiently evaluated. This paper studies retrieval and refinement across progressively realistic settings using 34k real-world skills, finding that skill benefits weaken toward no-skill baselines but query-specific refinement can recover performance when relevant skills exist. On Terminal-Bench 2.0, retrieval and refinement raise Claude Opus 4.6’s pass rate from 57.7% to 65.5%.

  • Problem

    Prior skill evaluations are overly idealized because they directly provide hand-crafted, task-specific skills rather than testing retrieval and imperfect task matching.

  • Method

    The paper evaluates retrieval and query-specific or query-agnostic refinement using progressively realistic settings over a collection of 34k real-world skills.

  • Results

    Skill benefits degrade consistently as settings become more realistic, while retrieval and refinement improve Claude Opus 4.6’s Terminal-Bench 2.0 pass rate from 57.7% to 65.5%.

  • Takeaways & Limitations

    Query-specific refinement can substantially recover lost performance when initially retrieved skills are reasonably relevant and high quality, but cannot compensate when relevant skills are absent.

  • Takeaways & Limitations

    The curated + forced-load setting is an upper bound on curated skill utility because it bypasses the practical challenges evaluated elsewhere.

Abstract

from arXiv · show

Agent skills, which are reusable, domain-specific knowledge artifacts, have become a popular mechanism for extending LLM-based agents, yet formally benchmarking skill usage performance remains scarce. Existing skill benchmarking efforts focus on overly idealized conditions, where LLMs are directly provided with hand-crafted, narrowly-tailored task-specific skills for each task, whereas in many realistic settings, the LLM agent may have to search for and select relevant skills on its own, and even the closest matching skills may not be well-tailored for the task. In this paper, we conduct the first comprehensive study of skill utility under progressively challenging realistic settings, where agents must retrieve skills from a large collection of 34k real-world skills and may not have access to any hand-curated skills. Our findings reveal that the benefits of skills are fragile: performance gains degrade consistently as settings become more realistic, with pass rates approaching no-skill baselines in the most challenging scenarios. To narrow this gap, we study skill refinement strategies, including query-specific and query-agnostic approaches, and we show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality. We further demonstrate the generality of retrieval and refinement on Terminal-Bench 2.0, where they improve the pass rate of Claude Opus 4.6 from 57.7% to 65.5%. Our results, consistent across multiple models, highlight both the promise and the current limitations of skills for LLM-based agents. Our code is available at https://github.com/UCSB-NLP-Chang/Skill-Usage.

1 Introduction

This paper argues that prior skill evaluations are overly idealized and studies skill utility as agents retrieve and use skills under progressively realistic conditions. It finds that skill benefits degrade with realism, while retrieval and refinement strategies can recover some performance.

  • Motivation: Prior evaluations often provide hand-crafted, task-specific skills directly in context, bypassing skill discovery and generalization challenges.The flooding-days example shows curated skills supplying the API, data URL, and detection code needed for a near-complete solution.
  • Approach: The study evaluates skill utility using 34k deduplicated real-world skills and progressively more realistic retrieval settings.Settings range from distractor-augmented curated skills to retrieval from the full collection and retrieval after removing curated skills.
  • Approach: Agentic hybrid search, which iteratively formulates queries and evaluates candidates, significantly outperforms the other evaluated retrieval approaches.The retrieval study compares keyword, semantic, hybrid, and agentic search strategies.
  • Findings: Skill benefits degrade consistently as evaluation settings become more realistic, eventually approaching no-skill baselines in the most challenging scenario.This pattern is reported across Claude Opus 4.6, Kimi K2.5, and Qwen3.5-397B-A17B.
  • Findings: Query-specific refinement substantially recovers lost performance when retrieved skills are reasonably relevant and useful, whereas absent relevant skills limit recovery.The paper also studies query-agnostic refinement and analyzes when refinement helps.

2 Related Work

Related work studies reusable knowledge, standardized agentic skills, and agent self-improvement, but prior skill benchmarks largely use directly provided curated skills. This paper extends that literature by evaluating retrieval and refinement under progressively realistic conditions.

  • Reusable knowledge for LLM agents: Research on reusable knowledge for LLM agents includes tools, embodied-environment skill libraries, structured manuals, and reusable workflows.These approaches represent several forms through which agents accumulate and reuse knowledge across tasks.
  • Agentic skills: Agentic skills are filesystem-based knowledge artifacts centered on SKILL.md files with structured metadata and content, optionally accompanied by helper files.Related work spans skill taxonomies, infrastructure, discovery, evolution, routing, persistent memory, and security.
  • Benchmarking gap: Existing skill benchmarks evaluate settings where curated skills are directly provided, whereas this work evaluates skill utility under progressively realistic conditions and studies refinement.This distinction identifies the paper’s position relative to SKILLSBENCH and related real-world software-engineering evaluations.
  • Agent self-improvement and test-time adaptation: Agent self-improvement and test-time adaptation methods let agents learn from feedback through reflection, optimization, reinforcement learning, and inference-time knowledge accumulation.The paper positions skill refinement as connected to this broader line of work.

3 Skill Usage in Realistic Settings

The study evaluates skill utility under progressively realistic conditions, adding skill selection, retrieval, and adaptation challenges. Across these settings, retrieval quality and agent performance decline as skills become less task-specific or harder to access.

  • Skill search engine: Agents retrieve skills from a 34k-skill collection using direct and agentic search over metadata and, for hybrid search, full skill content.The study compares keyword, semantic, hybrid, and agentic retrieval strategies using a skill index built from metadata and SKILL.md content.
  • Skill search engine: Agentic search improves retrieval over direct search, while adding full skill content raises Recall@5 from 63.5% to 65.5% and Recall@10 from 66.7% to 68.3%.With the same semantic tool, agentic search exceeds direct search in Recall@3 by 18.7 points; agentic hybrid search with full content becomes the default.
  • Evaluation settings: The evaluation progresses from forced loading of curated skills to autonomous selection, retrieval with curated skills, retrieval without curated skills, and a no-skills baseline.Each stage introduces selection, retrieval, or adaptation challenges while moving from idealized to realistic conditions.
  • Skill selection: 55.4% pass rate under forced curated loading falls to 51.2% with autonomous loading and 43.5% with distractors for Claude.Only 49% of Claude trajectories load all curated skills in the curated setting, falling to 31% with distractors.
  • Skill retrieval: Retrieval further reduces performance: Claude reaches 40.1% and Kimi 33.5% when curated skills remain in the retrieval pool.Claude’s skill-loading rate also falls from 62% in the curated setting to 44% under retrieval.
  • Skill adaptation: Without curated skills, Claude reaches 38.4%, only 3.0 points above no-skill, while Kimi and Qwen fall below their no-skill baselines.Kimi reaches 19.8% versus a 21.8% baseline, and Qwen reaches 19.7% versus a 20.5% baseline.

4 Narrowing the Gap with Skill Refinement

The paper evaluates query-specific and query-agnostic refinement as ways to improve retrieved skills across SKILLSBENCH and TERMINAL-BENCH 2.0. Query-specific refinement is broadly effective when retrieved skills have sufficient relevance and coverage, while query-agnostic refinement yields smaller and less consistent gains.

  • Refinement Strategies: The study compares query-specific refinement, query-agnostic refinement, and their evaluation on SKILLSBENCH and TERMINAL-BENCH 2.0.Query-specific refinement explores the target task before composing refined skills; query-agnostic refinement improves retrieved skills offline without task knowledge.
  • Results: Query-specific refinement improves performance in 7 out of 9 cases in Table 2.It improves Claude from 40.1% to 48.2% and Qwen from 26.7% to 30.8% on SKILLSBENCH with curated skills in the retrieval pool.
  • Results: +4.1 for Claude, +5.6 for Kimi, and +4.9 for Qwen are the TERMINAL-BENCH 2.0 improvements from query-specific refinement.This benchmark has no curated skills and was not designed with skills in mind.
  • Refinement Strategies: Query-specific refinement can merge relevant information across multiple retrieved skills, unlike query-agnostic refinement.The agent extracts tensor-parallelism knowledge and custom autograd patterns, synthesizing a skill with differentiable collective operations that neither original skill provides alone.
  • Results: Query-agnostic refinement provides smaller, inconsistent gains because it cannot identify task-relevant content or synthesize information across skills without the target task.Claude rises from 40.1% to 42.0% on SKILLSBENCH with curated skills and from 61.4% to 63.3% on TERMINAL-BENCH 2.0.
  • Results: Refinement effectiveness depends on initial skill quality: successful settings have average coverage scores ≥3.83, while the failing setting has scores ≤3.49.The paper characterizes refinement as amplifying existing skill quality rather than generating new knowledge.

5 Conclusion

The conclusion finds that skill benefits decline substantially under realistic retrieval and general-purpose skill conditions. Query-specific refinement can recover much of the lost performance when retrieved skills are reasonably relevant, but cannot compensate when relevant skills are absent.

  • Skill benefits degrade substantially when agents retrieve from large collections and use general-purpose skills not tailored to the task.
  • Query-specific refinement recovers much of the lost performance when retrieved skills have reasonable relevance.
  • Refinement cannot compensate when relevant skills are entirely absent, suggesting that it amplifies existing skill quality rather than generating new knowledge.
  • The findings support better skill retrieval, more effective offline refinement, and skill ecosystems that account for varying model capabilities.

Ethics Statement

The ethics statement describes a study using public, permissively licensed skills and isolated benchmark evaluations without risk to external systems.

  • The skill collection uses public GitHub repositories and skills filtered under MIT and Apache 2.0 licenses.
  • The collection does not contain private or sensitive data.
  • Model evaluations run on established coding benchmarks in isolated Docker containers, posing no risk to external systems.

LLM Usage Disclosure

The authors disclose LLM assistance in research infrastructure, analysis, writing, plotting, and formatting, while attributing research ideas and experimental design to the authors.

  • LLMs assisted with modifying open-source repositories, debugging evaluation infrastructure, and analyzing agent trajectories.
  • LLMs helped revise and smooth drafted text, proofread, write plotting scripts, and format tables and LaTeX elements.
  • The authors state that the research ideas, experimental design, and analysis were their work.

A Skill Search Engine Details

The skill search engine indexes reusable skills and exposes keyword, semantic, and hybrid retrieval, while agentic search iteratively formulates queries and evaluates candidates.

  • Skill index construction: 34,198 skills are indexed using metadata and full SKILL.md content, restricted to permissively licensed MIT and Apache-2.0 repositories.Metadata concatenates each skill’s name and description; full content is the SKILL.md body.
  • Retrieval representations: Dense retrieval prepends an instruction before encoding queries, while content-aware hybrid search combines metadata and content similarities.For content-aware search, semantic similarity is computed as (1 − w) · simmeta + w · simcontent.
  • Search tools: Three HTTP endpoints provide BM25 keyword search, dense semantic search, and hybrid retrieval using Reciprocal Rank Fusion.Hybrid retrieval combines keyword and semantic rankings with configurable weights, defaulting to 0.5 for each method and k = 60.
  • Retrieval configuration: Synthetic-query tuning selects a BM25 content-field weight of 5 and a semantic content weight of w = 0.05 using Recall@5 against curated skills.The tuning process generates one to three short queries per task and evaluates direct, non-agentic search.
  • Agentic search: Agentic search gives the agent a finding-skills skill that supports iterative query formulation and candidate evaluation.The agent can analyze tasks into subtasks, search for each subtask, refine queries, and select skills based on relevance, repository stars, and informative snippets.

B Experiment Details

The experiments evaluate skill retrieval and refinement across two software-task benchmarks, three model–harness pairings, and repeated isolated-container runs with automated verification.

  • Benchmarks: The evaluation covers 84 SKILLSBENCH tasks and all 89 TERMINAL-BENCH 2.0 tasks.Three SKILLSBENCH tasks with known environment or verifier issues are excluded.
  • Models and agent harnesses: Three models are paired with their native harnesses: Claude Opus 4.6 with Claude Code, Kimi K2.5 with Terminus-2, and Qwen3.5 with Qwen-Code.Kimi K2.5 and Qwen3.5 are served locally via SGLang.
  • Evaluation protocol: Each task is run three times in isolated Docker containers and scored with the benchmark’s automated verifiers.SKILLSBENCH uses a 1.5× timeout multiplier; TERMINAL-BENCH 2.0 uses 2× for Kimi K2.5 and Qwen3.5.

C.1 Query-Specific Refinement

Query-specific refinement adapts retrieved skills through a single task-aware exploration cycle, preserving useful guidance while correcting errors and adding missing knowledge.

  • Setting: Query-specific refinement runs inside the task’s Docker environment and gives the agent access to task data, libraries, and tools.The agent cannot access the ground-truth verifier and performs only one refinement iteration.
  • C.1 Query-Specific Refinement: The refinement agent reads the task and all retrieved skills, then attempts the task while actively consulting those skills.It tests suggested guidance, records which parts work or fail, and identifies missing knowledge.
  • Reflection: The protocol requires recording useful, misleading, and missing portions of retrieved skills before creating refined skills.A skill-creator guide provides guidance for writing the refined artifacts.
  • Refinement output: Refined skills retain effective content, remove misleading guidance, add discoveries, and combine related information into task-appropriate guides.The process is task-specific, autonomous, and limited to a single improvement round.

C.2 Query-Agnostic Refinement

Query-agnostic refinement improves each skill offline without a downstream task, using generated test queries and A/B evaluation before one autonomous improvement round.

  • C.2 Query-Agnostic Refinement: Query-agnostic refinement improves each skill independently without knowledge of any target task in a minimal Ubuntu 24.04 container with Python.The refined skill is saved with its SKILL.md and optional supporting files.
  • Refinement procedure: The improvement agent reads the skill and skill-creator guide, generates sample test queries, evaluates the skill with A/B testing, and improves it based on the findings.The process is autonomous and uses one evaluation-and-improvement iteration.
  • Skill coverage evaluation: An LLM judge rates how completely retrieved skills collectively cover each task on a five-point scale.The judge receives the task instruction and retrieved skill contents, then reports a score, covered content, and gaps as JSON.
Loading 2604.04323v1…