PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Retrieval Augmentation for Agentic AI

Weihang Su, Jianming Long, Qingyao Ai, Qiaozhi He, Yichen Tang, Changyue Wang, Yiteng Tu, Yingbo Wang, Yiqun Liu

arXiv:2604.24594v3cs.CLcs.AI

TL;DR

The paper asks how agents can scale beyond explicit in-context skill enumeration when solving tasks that exceed native capabilities. It formulates Skill Retrieval Augmentation, builds a benchmark for decomposed pipeline evaluation, and finds that external skills can substantially improve performance, while current agents struggle with selective, need-aware incorporation and application.

  • Problem

    Explicitly exposing growing skill libraries in context is unscalable, motivating study of how agents can retrieve and use external capabilities on demand.

  • Method

    The paper formulates SRA, constructs a large skill corpus and SRA-Bench, and establishes SR-Agents for systematic evaluation of retrieval, incorporation, and task solving.

  • Results

    External skills can substantially improve downstream performance when correctly accessed and used, but retrieval alone does not solve scalable skill augmentation.

  • Takeaways & Limitations

    SRA is a distinct research problem requiring agents to access capabilities selectively, incorporate them appropriately, and operationalize them robustly.

  • Takeaways & Limitations

    Current agents often fail to recognize when external help is needed, which retrieved skills are worth loading, and how to translate them into reliable gains.

Abstract

from arXiv · show

As large language models (LLMs) evolve into agentic problem solvers, they increasingly rely on external, reusable skills to handle tasks beyond their native parametric capabilities. In existing agent systems, the dominant strategy for incorporating skills is to explicitly enumerate available skills within the context window. However, this strategy fails to scale: as skill corpora expand, context budgets are consumed rapidly, and the agent becomes markedly less accurate in identifying the right skill. To this end, this paper formulates Skill Retrieval Augmentation (SRA), a new paradigm in which agents dynamically retrieve, incorporate, and apply relevant skills from large external skill corpora on demand. To make this problem measurable, we construct a large-scale skill corpus and introduce SRA-Bench, the first benchmark for decomposed evaluation of the full SRA pipeline, covering skill retrieval, skill incorporation, and end-task execution. SRA-Bench contains 5,400 capability-intensive test instances and 636 manually constructed gold skills, which are mixed with web-collected distractor skills to form a large-scale corpus of 26,262 skills. Extensive experiments show that retrieval-based skill augmentation can substantially improve agent performance, validating the promise of the paradigm. At the same time, we uncover a fundamental gap in skill incorporation: current LLM agents tend to load skills at similar rates, regardless of whether a gold skill is retrieved or whether the task actually requires external capabilities. This shows that the bottleneck in skill augmentation lies not only in retrieval but also in the base model's ability to determine which skill to load and when external loading is actually needed. These findings position SRA as a distinct research problem and establish a foundation for the scalable augmentation of capabilities in future agent systems.

1 Introduction

The paper introduces Skill Retrieval Augmentation (SRA) to scale agents’ use of external capabilities beyond explicit in-context skill enumeration, and evaluates retrieval, incorporation, and application as distinct stages. Experiments show substantial promise but identify skill incorporation and need-aware utilization as central bottlenecks.

  • Motivation: Explicitly enumerating skills in context becomes impractical as skill libraries grow, consuming context and impairing relevant-skill identification.The paper motivates retrieval as an alternative to fixed or extensively enumerated skill exposure.
  • SRA paradigm: SRA retrieves, selectively incorporates, and applies reusable external skills from a large capability corpus on demand.Unlike knowledge-centric RAG, SRA retrieves executable capabilities intended to augment functional competence.
  • Benchmark: SRA-Bench decomposes evaluation into skill retrieval, skill incorporation, and end-to-end task performance using queries, answers, and gold-skill annotations.The benchmark is designed to diagnose where an agent succeeds or fails across the full SRA pipeline.
  • Findings: A simple pipeline retrieving and injecting one skill can improve strong LLM agents over skill-free counterparts.This result establishes the practical promise of retrieval-based skill augmentation.
  • Findings: Agents often load skills at similar rates regardless of retrieval success or whether tasks genuinely require external capabilities.The findings indicate bottlenecks in controlled, need-aware incorporation and application, not retrieval quality alone.

2 Problem Formulation

SRA models scalable skill augmentation as a three-stage pipeline: retrieve candidate skills, incorporate selected representations, and apply them during task solving. Skills are reusable capability packages, and successful retrieval or incorporation does not guarantee effective downstream use.

  • 2.1 Agent Skills and Skill Corpus: A skill is a reusable capability package combining a natural-language interface with executable resources for recurring problem classes.It can expose what capability is available, when it applies, and how to use it.
  • 2.1 Agent Skills and Skill Corpus: Each corpus skill is represented by a name, description, main content, and executable payload.These components correspond to semantic identification, intended use, instructions, constraints, procedures, code, tools, or other operational resources.
  • 2.2 Skill Retrieval Augmentation: SRA retrieves relevant external capabilities on demand instead of exposing a fixed skill set directly in the context.The formulation separates scalable augmentation from explicit enumeration of visible skills.
  • 2.2 Skill Retrieval Augmentation: The retrieval stage maps a query and corpus to a ranked list of k candidate skills, with k ≪ N.Ranking reduces a massive capability space to a manageable set of potentially useful candidates.
  • 2.2 Skill Retrieval Augmentation: Skill incorporation determines whether to use external skills and converts retrieved candidates into representations available for downstream solving.The incorporated representation may be selected, rewritten, compressed, restructured, or model-adapted, and may be empty.
  • 2.2 Skill Retrieval Augmentation: Skill application conditions the agent on incorporated skills and produces the final task response.This stage tests whether the agent follows, invokes, and integrates the skills correctly during reasoning.
  • 2.2 Skill Retrieval Augmentation: Successful incorporation does not guarantee successful application because the model may fail to operationalize an available skill during downstream solving.The pipeline therefore evaluates retrieval, incorporation, and application as distinct stages with interacting failure modes.

3 Benchmark Construction

SRA-Bench combines capability-intensive tasks, manually constructed gold skills, and a noisy external corpus to evaluate scalable skill augmentation under realistic retrieval conditions. Its construction transforms source-dataset signals into reusable procedural artifacts and mixes them with web-collected distractors.

  • Benchmark Construction: SRA-Bench contains capability-intensive instances, manually constructed gold skills with annotations, and a noisy corpus containing realistic distractors.These components make decomposed evaluation of skill augmentation empirically tractable.
  • Benchmark Construction: 5,400 test instances are associated with 636 unique gold skills across six benchmarks covering reasoning, logic, tool use, medical calculation, mathematics, and code generation.The benchmark is constructed from TheoremQA, LogicBench, ToolQA, MedCalc-Bench, CHAMP, and BigCodeBench.
  • Benchmark Construction: Source datasets are selected for structured signals that reveal recurring problem-solving patterns and support reliable inference of reusable capabilities.The associated capability must describe how to solve a class of problems rather than directly reveal an answer.
  • Benchmark Construction: Each example pairs a user query and ground-truth answer with one or more manually derived gold skills.The annotations are guided by signals such as theorem names, logic patterns, calculator types, and library names.
  • Benchmark Construction: Gold-skill construction transforms source-side capability signals into standardized reusable skills that specify when and how a capability should be applied.The source annotations identify relevant capabilities but do not themselves provide usage procedures.
  • Benchmark Construction: Gold skills are drafted by an LLM and then revised by experts into standardized Markdown artifacts.Final artifacts include descriptions and procedures, with runnable resources added for capabilities requiring execution.
  • Benchmark Construction: 636 gold skills are mixed with 25,626 web-collected skill documents to form a large corpus of heterogeneous, mostly distractor entries.The documents are cleaned, deduplicated, normalized, and intended to approximate a realistic ecosystem where useful skills are sparse.

4 Study Design and Experimental Setup

The study evaluates whether SRA improves task solving, withstands retrieval noise, and connects retrieval quality to downstream performance, while testing relevance-aware and need-aware skill loading. Experiments standardize benchmarks, models, strategies, metrics, and implementation choices across a systematic pipeline study.

  • 4.1 Study Design: The study asks whether SRA improves performance over skill-free baselines and compares different SR-Agent configurations.It also examines robustness to distractors and variation across agent designs.
  • 4.1 Study Design: Retrieval experiments compare existing methods, including classical lexical matching and dense retrieval, for identifying relevant skills.The study separately asks how retrieval quality affects end-to-end outcomes through incorporation and application.
  • 4.1 Study Design: Skill-loading analyses test whether agents are relevance-aware when gold skills are present and need-aware when tasks exceed native capabilities.These properties determine whether retrieved skills are effectively incorporated and used.
  • 4.1 Study Design: The study covers six research questions spanning SRA utility, retrieval noise, retrieval effectiveness, downstream influence, and incorporation behavior.Together, the questions target uncertainty sources across the full SRA pipeline.
  • 4.2 Experimental Setup: Experiments use six capability-intensive benchmarks paired with annotated gold skills and a shared external corpus.The evaluation spans mathematical reasoning, formal logic, tool use, and code generation.
  • 4.2 Experimental Setup: Retrieval is evaluated with Recall@K and nDCG@K, while end-task performance uses benchmark-specific accuracy, exact match, or pass@1.BigCodeBench uses pass@1 based on unit-test execution.
  • 4.2 Experimental Setup: The model set includes six open-weight LLMs and two proprietary frontier models for skill-loading analyses.All models use a 128K-token context window and temperature 0.7.

5 Systematic Empirical Study

The study finds that SRA can improve agent performance, but success depends on retrieval quality and how skills are exposed, selected, and incorporated. Experiments identify incorporation under noise and need-aware loading as major remaining bottlenecks.

  • RQ1: Overall effectiveness: When the correct skill is provided, Oracle Skill consistently outperforms LLM Direct across nearly all model-benchmark combinations.This indicates that benchmark tasks benefit materially from external procedural knowledge.
  • RQ1: Overall effectiveness: Retrieval-based methods often improve over LLM Direct, but gains vary substantially across models, tasks, and exposure strategies and remain below Oracle Skill.Practical augmentation therefore offers meaningful but non-guaranteed benefits.
  • RQ2: Robustness under distractors: Even with the gold skill present, hard-negative distractors degrade end-task accuracy, shifting the central failure mode from retrieval recall toward skill incorporation.Full Skill Injection is especially vulnerable to prompt overload and procedural confusion, whereas Progressive Disclosure is more stable under heavier noise.
  • RQ2: Robustness under top-k exposure: As BM25 top-k grows, Full Skill Injection typically improves only initially before declining, while Progressive Disclosure remains more stable by selectively loading full content.The decline is attributed to inter-skill interference and downstream reasoning disruption.
  • RQ3: Skill retrieval: Existing retrievers make skill retrieval feasible but not solved: sparse and dense methods are complementary, while LLM reranking provides the strongest overall retrieval performance.Retrieval signals combine lexical overlap, semantic relevance, and procedural intent.
  • RQ4: Retrieval and end-task performance: Stronger retrieval generally improves end-task performance, but the relationship is neither direct nor strictly monotonic because downstream interpretation, incorporation, and execution mediate its effect.Better retrieval increases the likelihood of successful skill use rather than guaranteeing it.

6 Related Work

SRA combines ideas from retrieval-augmented generation, tool use, and modular skills while addressing limitations in how these paradigms expose reusable capabilities. SRA-Bench fills a benchmarking gap by decomposing skill retrieval, incorporation, and downstream application.

  • SRA retrieves executable capability packages, whereas RAG primarily retrieves static knowledge for language-model grounding.
  • Unlike conventional tool-use systems with visible, limited APIs, SRA retrieves modular packages containing instructions, invocation conditions, and procedural guidance from open-ended corpora.
  • Modular-skill research demonstrates reusable executable capabilities, but prior work has primarily focused on skill utility and ecosystems rather than full-pipeline evaluation.
  • SRA-Bench decomposes evaluation into standalone skill retrieval, skill incorporation, and downstream task application, unlike existing tool/function benchmarks.

7 Toward a Research Agenda for Skill Retrieval Augmentation

The paper frames SRA as a broader research agenda involving skill organization, quality control, utility-aware retrieval, and eventual parameter-based augmentation. These directions reflect challenges created by noisy, growing, open-world skill corpora.

  • From unstructured skill collections to structured skill libraries: Unstructured lists of independently indexed skills may become inadequate as skill collections grow, motivating structured skill libraries.
  • Skill quality control, offline refinement, and skill evolution: Open skill ecosystems contain incomplete, outdated, redundant, or incorrect skills whose errors can directly mislead downstream agents.
  • From semantic matching to utility-aware skill retrieval: SRA retrieval should be evaluated by downstream utility, potentially retrieving complementary skill sets or conditioning retrieval on intermediate reasoning states.
  • From semantic matching to utility-aware skill retrieval: Skill retrieval is positioned as a distinct information-retrieval problem rather than a straightforward extension of classical document retrieval.
  • Toward Parametric Skill Augmentation: Parametric Skill Augmentation would transform retrieved skills into plug-in parameters that can be loaded when needed instead of injecting them only through context.

8 Conclusion

The paper concludes that SRA can improve agents through external skills but remains constrained by agents’ ability to access, incorporate, and operationalize those skills selectively. It presents the corpus, benchmark, and analyses as a foundation for a broader research agenda.

  • SRA defines scalable capability augmentation as retrieving, incorporating, and applying reusable skills from large external corpora on demand.
  • External skills can substantially improve downstream performance when agents correctly access and use them.
  • The core bottleneck extends beyond retrieval to selective access, appropriate incorporation, and robust operationalization of retrieved capabilities.
  • The released skill corpus, SRA-Bench, and analyses provide a shared foundation for studying how external skills are organized, maintained, retrieved, and internalized.

A Dataset-Specific Construction Details

The appendix details how SRA-Bench supervision varies across source datasets and documents the construction of standalone gold skills from structured signals. This makes the benchmark construction process inspectable and reproducible.

  • Dataset-specific supervision signals anchor reusable capabilities in theorem names, logic patterns, tool workflows, calculator types, mathematical concepts, and software libraries.
  • Gold skills are manually constructed as standalone capability artifacts rather than directly inherited from source datasets.
  • The appendix presents per-dataset construction procedures to make gold-skill creation easier to inspect, verify, and reproduce.

A.1 TheoremQA

TheoremQA provides text-only theorem-solving instances with structured theorem annotations, which are used to construct reusable theorem-application skills. Because the source data lack solution steps, drafting and revision must supply explicit procedures and address recurring conceptual confusions.

  • TheoremQA retains 747 text-only instances associated with 320 theorems from an original dataset of approximately 800 instances and 334 theorems.
  • The retained theorems span mathematics, EECS, physics, and finance, with each instance linked to exactly one theorem and each theorem mapped to one gold skill.
  • Skill construction gives the LLM theorem metadata, associated questions and answers, and domain-appropriate reference pages, then asks it to write a reusable skill document.
  • The requested skill structure includes a theorem statement, recognition conditions, a step-by-step procedure, common pitfalls, and a new worked example.
  • Expert revision targets procedural under-specification and confusion between closely related concepts, restructuring formula summaries into explicit application procedures with case distinctions and computation templates.

A.1.3 Construction Examples

The construction examples show how expert revision turns formula-correct drafts into reusable skills by adding procedures, recognition guidance, and independently constructed examples. The DDM example emphasizes multi-stage computation, while the Lah-number example emphasizes distinguishing ordered from unordered partitions.

  • Dividend Discount Model: The DDM skill covers zero-growth, constant-growth, and multi-stage valuation, with the constant-growth case requiring D1 rather than D0 in the numerator.
  • Dividend Discount Model: The multi-stage DDM procedure computes high-growth dividends, discounts them, calculates terminal value from Dn+1, discounts that value, and sums the present values.
  • Dividend Discount Model: For D0 = 4, g1 = 18% for 3 years, g2 = 6%, and r = 14%, the worked example obtains P0 ≈71.6.
  • Dividend Discount Model: Expert revision replaced the DDM draft’s unstructured multi-stage explanation with a five-step template, documented the Dn+1 versus Dn pitfall, and introduced a new example.
  • Lah Numbers: Lah numbers count partitions into non-empty ordered subsets, whereas Stirling numbers of the second kind count unordered partitions, making recognition of ordering essential.
  • Lah Numbers: The revised Lah-number skill adds ordering checks, a comparison with Stirling numbers, and a smaller independently constructed example after the draft reused an input instance.

A.3.3 Construction Examples

The flight and agenda examples demonstrate that effective skills must encode domain-specific parsing, tool workflows, and boundaries between lookup and corpus-wide computation. Expert revision adds these operational details when the initial drafts omit them.

  • Flights: The flight lookup skill uses LoadDB, FilterDB, and GetValue for single-record questions over a US domestic flight database.
  • Flights: Flight identifier AA2319 must be split into carrier code AA and flight number 2319 as separate FilterDB conditions before retrieval.
  • Flights: Flight times are stored as numeric HHMM values, so 2143.0 must be interpreted and reported as 21:43.
  • Agenda: The agenda skill handles counting, time-overlap population queries, and free-slot scheduling over a personal-calendar corpus using retrieval plus arithmetic or programmatic processing.
  • Agenda: Expert revision separates single-event lookup from corpus-wide queries and adds Python-based corpus loading, date conversion, filtering, and query-specific workflows.
  • Agenda: RetrieveAgenda returns only the top 3 passages, so it cannot reliably support counting or enumeration across all matching records.

A.4.3 Construction Examples

The medical-calculator examples show that reusable skills must preserve conditional branches, not merely reproduce a central formula. Revisions correct omitted dialysis, threshold, and BMI-dependent weight rules in MELD Na and Cockcroft–Gault skills.

  • MELD Na: The MELD Na skill computes a score from creatinine, bilirubin, INR, sodium, and dialysis status using clamping, logarithmic scoring, sodium adjustment, capping, and rounding.
  • MELD Na: MELD Na bypasses sodium adjustment when the scaled MELD(i) is ≤11 and caps the final score at 40.
  • MELD Na: A worked MELD Na example with creatinine 1.8, bilirubin 3.5, INR 1.4, sodium 130, and no dialysis produces a score of 25.
  • MELD Na: Expert revision restored the dialysis-dependent creatinine override and the MELD(i) ≤11 bypass omitted by the draft and its tool implementation.
  • Cockcroft–Gault: Cockcroft–Gault estimates creatinine clearance for drug dosing and selects adjusted weight through BMI-based branches: actual weight, min(IBW, actual weight), or adjusted body weight.
  • Cockcroft–Gault: Expert revision replaced the draft’s always-overweight adjustment with the complete underweight, normal-weight, and overweight decision tree.

A.5.2 Expert Revision

The expert revision targets gaps in LLM-generated mathematical skills by adding recognition patterns and systematic strategy organization. These additions connect abstract concepts to structural cues and distinct problem-solving procedures.

  • Recognition pattern development: Recognition patterns identify problem types and structural signatures that signal when a concept should be applied.The expert adds cues that generalize across diverse problem types rather than merely restating theorems or reproducing calculations.
  • Solution strategy systematization: Solution strategies are organized by problem type, with each type receiving a systematic procedure.This addresses cases where one concept supports multiple application modes, such as modular arithmetic for impossibility proofs, residue enumeration, or periodicity analysis.

A.5.3 Construction Examples

The construction examples show how expert revisions turn mathematical concepts into skills with recognition cues, procedures, assumptions, and worked examples. AM-GM emphasizes reciprocal and product patterns, while modular arithmetic is organized around four problem types and modulus-selection guidance.

  • AM-GM Inequality (Two Variables): AM-GM application requires non-negative terms and verification that equality conditions are achievable and mutually compatible.For t + 4/t, equality at t = 2 yields the minimum value 4.
  • AM-GM Inequality (Two Variables): AM-GM skills add recognition cues for sum-product, reciprocal, and product-of-factors patterns beyond the inequality formula.The revision replaces a repeated input problem with a new t + 4/t example that exercises reciprocal-expression recognition.
  • Modular Arithmetic: Modular-arithmetic skills classify problems into impossibility, residue enumeration, divisibility, and periodicity, each with a corresponding procedure.The workflow includes classifying the problem, selecting a modulus, enumerating residues, and comparing residue sets or periods.
  • Modular Arithmetic: Expert revisions add modulus-selection heuristics because generic modular identities and procedures do not specify the core strategic choice.The revised skill uses coefficient cues and recommends trying other moduli when the first choice fails.
  • BigCodeBench: The construction process converts library annotations, task solutions, and official documentation into reusable gold skills for BigCodeBench.Each distinct library becomes one gold skill, and canonical solutions expose concrete API usage patterns relevant to passing unit tests.

B Implementation Details

The appendix specifies how retrieved or selected skills are exposed to the model, including full-skill injection, LLM selection, and progressive disclosure. Progressive disclosure lets the model load skills on demand across successive turns.

  • Full-Skill Injection: Retrieved or Oracle-provided skills are prepended in full to the original user prompt.Multiple available skills are concatenated with a horizontal-rule delimiter.
  • LLM Selection: LLM Selection presents a numbered candidate list and asks the model to output only the number of the most relevant skill.Candidates are represented by their names and descriptions, after which the selected skill is injected into the task prompt.
  • Progressive Disclosure: Progressive Disclosure augments the system prompt with a compact skill catalog and an instruction for loading full skill content on demand.The model invokes LOAD_SKILL with a skill index to receive the corresponding methodology.
  • Progressive Disclosure: LOAD_SKILL actions inject the corresponding full skill and prompt the model to continue, with multiple skills loadable across successive turns for up to 10 rounds.This supports iterative skill use rather than requiring all skill content in the initial prompt.
Loading 2604.24594v3…