PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Retrieval Augmentation for Agentic AI

Weihang Su, Jianming Long, Qingyao Ai, Qiaozhi He, Yichen Tang, Changyue Wang, Yiteng Tu, Yingbo Wang, Yiqun Liu

arXiv:2604.24594v3cs.CLcs.AI

TL;DR

Explicitly listing skills in context does not scale as skill libraries grow, making it harder for agents to identify the right capability. This paper formulates Skill Retrieval Augmentation, introduces a benchmark for its full pipeline, and finds that external skills improve performance but retrieval alone does not solve selective skill use.

  • Problem

    Explicit in-context skill injection becomes impractical as skill libraries grow, motivating scalable evaluation of retrieving and using external capabilities.

  • Method

    The paper formulates Skill Retrieval Augmentation and introduces SRA-Bench to evaluate skill retrieval, incorporation, and end-task execution across large external skill corpora.

  • Results

    External skills substantially improve downstream performance when correctly accessed and used, but agents often fail to recognize when help is needed or which retrieved skills to load.

  • Takeaways & Limitations

    SRA is a distinct research problem whose central challenge is selective, appropriate, and robust skill utilization rather than retrieval alone.

  • Takeaways & Limitations

    Current skill augmentation is substantially weakened by agents' limited ability to determine when external loading is needed and which skills to use.

Abstract

from arXiv · show

As large language models (LLMs) evolve into agentic problem solvers, they increasingly rely on external, reusable skills to handle tasks beyond their native parametric capabilities. In existing agent systems, the dominant strategy for incorporating skills is to explicitly enumerate available skills within the context window. However, this strategy fails to scale: as skill corpora expand, context budgets are consumed rapidly, and the agent becomes markedly less accurate in identifying the right skill. To this end, this paper formulates Skill Retrieval Augmentation (SRA), a new paradigm in which agents dynamically retrieve, incorporate, and apply relevant skills from large external skill corpora on demand. To make this problem measurable, we construct a large-scale skill corpus and introduce SRA-Bench, the first benchmark for decomposed evaluation of the full SRA pipeline, covering skill retrieval, skill incorporation, and end-task execution. SRA-Bench contains 5,400 capability-intensive test instances and 636 manually constructed gold skills, which are mixed with web-collected distractor skills to form a large-scale corpus of 26,262 skills. Extensive experiments show that retrieval-based skill augmentation can substantially improve agent performance, validating the promise of the paradigm. At the same time, we uncover a fundamental gap in skill incorporation: current LLM agents tend to load skills at similar rates, regardless of whether a gold skill is retrieved or whether the task actually requires external capabilities. This shows that the bottleneck in skill augmentation lies not only in retrieval but also in the base model's ability to determine which skill to load and when external loading is actually needed. These findings position SRA as a distinct research problem and establish a foundation for the scalable augmentation of capabilities in future agent systems.

1 Introduction

Skill Retrieval Augmentation (SRA) enables agents to retrieve, incorporate, and apply reusable external skills from large corpora instead of enumerating fixed skills in context. SRA-Bench evaluates this pipeline separately and exposes both the performance gains and incorporation bottlenecks of scalable skill use.

  • Skill Retrieval Augmentation: SRA replaces fixed in-context skill enumeration with on-demand retrieval of executable capabilities, whose utility depends on retrieval, incorporation, and downstream application.Unlike knowledge-centric RAG, SRA retrieves capabilities that augment functional competence rather than merely supplying declarative evidence.
  • SRA-Bench: SRA-Bench provides query, answer, and gold-skill annotations for decomposed evaluation of retrieval, incorporation, and end-task execution over large external skill corpora.The benchmark makes SRA a concrete research problem that can be rigorously evaluated and systematically analyzed.
  • Empirical findings: Simple retrieval of one skill can improve strong LLM agents over skill-free counterparts, demonstrating the practical promise of retrieval-based skill augmentation.The experiments use a general-purpose retriever and inject the retrieved skill into context.
  • Empirical findings: Agents load skills at nearly identical rates whether or not the gold skill is retrieved, revealing a disconnect between successful retrieval and effective utilization.Skill-loading behavior also varies dramatically across models without a monotonic relationship to model size.
  • Empirical findings: Agents are no more likely to invoke external skills for tasks requiring them than for tasks solvable natively, indicating absent need-aware skill invocation.These findings identify controlled skill exposure, incorporation, and application—not retrieval alone—as bottlenecks in scalable augmentation.

2 Problem Formulation

SRA formalizes scalable skill augmentation as an on-demand, three-stage pipeline: retrieving relevant skills, incorporating usable representations, and applying them to improve task solving. Its formulation distinguishes reusable capability packages from documents or isolated APIs and recognizes that retrieval, incorporation, and application can fail independently.

  • 2 Problem Formulation: SRA decomposes on-demand augmentation into skill retrieval, skill incorporation, and skill application, separating retrieval, usable integration, and downstream task performance.The formulation evaluates whether an agent can retrieve relevant skills, correctly incorporate them into its active problem-solving state, and translate them into improved task performance.
  • 2.1 Agent Skills and Skill Corpus: A skill is a reusable capability package that specifies not only what an agent can do, but also when and how to apply it.Typical components include a name, description, usage instructions, invocation conditions, procedural guidance, executable code, and auxiliary resources.
  • 2.1 Agent Skills and Skill Corpus: A skill couples a natural-language interface with an executable capability, potentially comprising instructions, constraints, procedures, code, tools, and other operational resources.Practical artifacts may include a title, summary, SKILL.md, scripts, and static assets.
  • 2.2 Skill Retrieval Augmentation: Given query q and corpus C, retrieval maps them to a ranked list L_k of k ≪ N candidate skills, reducing a massive capability space to manageable options.Earlier ranked positions indicate higher estimated relevance to the current query.
  • 2.2 Skill Retrieval Augmentation: Skill incorporation determines whether external skills are needed, which retrieved skills to use, and how to prepare them for downstream solving.The prepared representation e_S may be selected or transformed, including rewritten, compressed, restructured, or model-adapted forms, and may be empty when parametric capability suffices or no candidate is useful.
  • 2.2 Skill Retrieval Augmentation: Skill application uses incorporated capabilities during task solving, requiring the agent to follow, invoke, integrate, and adapt them appropriately.Successful incorporation alone does not guarantee improved performance if the model cannot operationalize the skill downstream.

3 Benchmark Construction

SRA-Bench decomposes skill augmentation into retrieval, incorporation, and downstream execution by pairing capability-intensive instances with manually constructed gold skills in a large distractor-rich corpus.

  • 3 Benchmark Construction: SRA-Bench contains 5,400 capability-intensive test instances, 636 unique gold skills, and a corpus designed to evaluate retrieval, incorporation, and downstream task performance separately.Each instance is associated with one or more relevant skills, while the external corpus mixes gold skills with realistic distractors.
  • 3.1 Source Dataset Selection and Test Instance Curation: The benchmark curates tasks requiring reusable capabilities from six datasets, using structured signals and objective evaluation to support reliable skill annotation and downstream measurement.The selected datasets span theorem reasoning, formal logic, tool-interleaved question answering, medical calculation, competition mathematics, and code generation.
  • 3.2 Gold Skill Construction: Gold skills are created per annotation category through LLM drafting and expert revision, enforcing generality, correctness, leakage control, and an application gap after retrieval.Final artifacts specify applicability, procedures, and pitfalls, with runnable resources attached when execution is inherently required.
  • 3.3 Skill Corpus Collection: The final corpus contains 26,262 skills, including 636 gold skills mixed with 25,626 web-collected documents, making gold capabilities sparse at 2.4%.Documents are collected from public web and community sources, filtered, deduplicated, normalized, and indexed to approximate a heterogeneous ecosystem of mostly irrelevant candidates.

4 Study Design and Experimental Setup

The study frames Skill Retrieval Augmentation as a multi-stage pipeline whose retrieval, incorporation, and application stages create distinct failure modes. It evaluates the paradigm through six research questions and a standardized setup spanning benchmarks, models, skill-use strategies, and metrics.

  • 4.1 Study Design: The study asks whether SRA improves over skill-free baselines and remains effective under noisy retrieval containing distractor, partially relevant, and misleading skills.These questions compare SR-Agent configurations and assess robustness when candidate sets include irrelevant entries.
  • 4.1 Study Design: It further examines retrieval effectiveness, lexical versus dense methods, retrieval quality’s effect on end-task performance, and whether incorporation is relevance-aware and need-aware.RQ3–RQ6 cover retrieval quality, downstream coupling, gold-skill sensitivity, and loading for tasks beyond native capabilities.
  • 4.2 Experimental Setup: Experiments use six SRA-Bench datasets spanning mathematical reasoning, formal logic, tool use, and code generation, with annotated gold skills and a shared external corpus.The benchmarks are TheoremQA [4], LogicBench [22], ToolQA [60], CHAMP [17], MedCalc-Bench [13], and BigCodeBench [61].
  • 4.2 Experimental Setup: The evaluation reports Recall@K and nDCG@K for retrieval, standard accuracy or exact match for reasoning and QA, and pass@1 for BigCodeBench.BigCodeBench pass@1 is based on unit-test execution.
  • 4.2 Experimental Setup: The setup evaluates six open-weight LLMs, adds GLM-5.1 and GPT-5.4 for loading analyses, and compares direct solving, Oracle Skill, Full-Skill Injection, LLM Selection, and Progressive Disclosure.All models use a 128K-token context window and temperature 0.7; BM25 is the default retriever in main experiments.

5 Systematic Empirical Study

Skill Retrieval Augmentation improves agent performance, but robust gains depend on selective skill exposure, retrieval quality, and effective downstream incorporation. Current agents remain unreliable at deciding whether a retrieved skill is relevant or needed.

  • 5.1 RQ1: Does Skill Retrieval Augmentation Improve Agent Performance?: External skills substantially improve performance, while LLM Selection offers the strongest practical trade-off; Progressive Disclosure is less stable and often trails selection-based exposure.Oracle Skill consistently outperforms LLM Direct, but practical methods remain uneven and usually below the oracle upper bound.
  • 5.2.1 Robustness under Controlled Distractors: Hard-negative distractors degrade end-task performance even when the gold skill is present, while Progressive Disclosure is more robust than Full Skill Injection under noise.Full-content injection creates prompt overload and procedural confusion; larger models do not consistently suppress distractors better.
  • 5.2.2 Robustness under Retrieved Top-k Skill Exposure: Increasing top-k exposure can initially help Full Skill Injection but usually causes later decline, whereas Progressive Disclosure remains stable by selectively loading full skill content.The contrast reflects a trade-off between broader coverage and inter-skill interference from injecting additional procedural content.
  • 5.3 RQ3: How Effective Are Existing Retrievers for Skill Retrieval?: Skill retrieval is feasible but task-dependent: sparse and dense methods have complementary strengths, hybrid retrieval improves coverage, and LLM reranking performs best overall.Existing retrievers can achieve high Recall@10, but difficult benchmarks remain, and reranking quality is not strictly monotonic with model scale.
  • 5.4 RQ4: How Does Retrieval Quality Affect End-Task Performance?: Better retrieval generally improves end-task performance, especially through higher top-rank precision, but retrieval quality is necessary rather than sufficient because incorporation and execution mediate gains.Reranking BM25 candidates often yields the most consistent end-to-end gains, while no retrieval paradigm dominates across all tasks.
  • 5.5 RQ5: Are Current SR-Agents Relevance-Aware in Skill Loading?: Most models are not reliably relevance-aware: gold-skill availability only weakly affects loading, while frontier models are more selective than open-source models.Many models load irrelevant skills even when no gold skill appears in BM25 top-50; stronger separation emerges among frontier models and open-source models at 235B scale.
  • 5.6 RQ6: Are Current SR-Agents Need-Aware in Skill Loading?: Current agents are generally not need-aware: they load skills at similar rates whether tasks are natively solvable or exceed their parametric capability.This behavior is not a targeted response to capability gaps, indicating that scalable augmentation requires controlled, selective, and need-aware utilization.

6 Related Work

Prior work spans retrieval-augmented generation, tool use, modular agent skills, and skill-effectiveness benchmarks. SRA distinguishes itself by retrieving executable capabilities and evaluating retrieval, incorporation, and downstream application as a unified pipeline.

  • 6.1 Retrieval Augmented Generation and Tool Use: Unlike RAG and conventional tool use, SRA retrieves executable, modular capabilities from open-ended corpora and evaluates both retrieval relevance and downstream skill application.RAG typically retrieves static knowledge, whereas conventional tool use assumes a visible, limited API set; SRA packages instructions, invocation conditions, and procedural guidance.
  • 6.2 Agent Skills: Prior modular-skill systems emphasize skill acquisition, code synthesis, or ecosystem analysis, whereas SRA targets scalable retrieval-time access to capabilities in massive out-of-context libraries.Voyager [54] maintains an expanding corpus of executable programs for embodied lifelong learning, while other analyses [16] examine agent-skill ecosystems.
  • 6.3 Benchmarking Skill Effectiveness: SRA-Bench decomposes skill augmentation into retrieval, incorporation, and downstream application, diagnosing whether agents can selectively use correct skills from vast corpora.Existing benchmarks mainly assess tool or function invocation, while SkillsBench [15] studies whether curated skills help across tasks without evaluating this full decomposed pipeline.

7 Toward a Research Agenda for Skill Retrieval Augmentation

The paper frames Skill Retrieval Augmentation as a broad agenda for scalable capability augmentation, because retrieval improves downstream performance but does not ensure useful skill incorporation or execution. Future progress therefore requires advances in skill organization, quality, utility-aware retrieval, and parametric augmentation.

  • Research agenda: SRA is broader than retrieval because agents may retrieve relevant skills yet fail to identify which are useful, decide whether external help is needed, or convert skills into better execution.The experiments nevertheless show that retrieving external skills from large corpora can substantially improve downstream performance, establishing the practical promise of scalable capability augmentation.
  • From unstructured skill collections to structured skill libraries: Structured skill libraries could organize related capabilities through graphs, hierarchies, clusters, or dependencies, improving search, reducing interference, and supporting multi-skill reasoning.The corpus and gold-skill annotations provide a testbed for evaluating these structures under decomposed SRA evaluation.
  • Skill quality control, offline refinement, and skill evolution: SRA needs continually maintained skill resources with validation, debugging, offline refinement, and evolution to address incomplete, outdated, redundant, or incorrect actionable capabilities.Refinement may use static analysis, execution-based testing, or agentic self-improvement to repair preconditions, procedures, descriptions, resources, and invocation conditions.
  • From semantic matching to utility-aware skill retrieval: Skill retrieval should optimize expected downstream utility rather than semantic similarity alone, using end-task feedback, capability-gap-aware reranking, complementary skill sets, and intermediate agent state.These challenges motivate studying skill retrieval as a distinct information-retrieval problem rather than a straightforward extension of document retrieval.
  • Toward Parametric Skill Augmentation: Parametric Skill Augmentation could convert frequently used retrieved skills into offline-optimized plug-in parameters, reducing repeated context injection while enabling deeper internalization of procedures and action policies.A hybrid architecture could retain core skills parametrically while retrieving rare, long-tail, or rapidly evolving skills externally; the direction is motivated by Parametric RAG [41].

8 Conclusion

The paper defines Skill Retrieval Augmentation as a scalable paradigm for agents to retrieve, incorporate, and apply external skills on demand, supported by a corpus, SRA-Bench, and SR-Agents. Experiments show that although external skills can improve performance, the central challenge is selectively loading and reliably operationalizing them, motivating a broader research agenda.

  • Contributions: SRA establishes a distinct paradigm and empirical foundation for scalable skill augmentation through an external skill corpus, SRA-Bench, and SR-Agents.The framework evaluates retrieval, incorporation, and application of reusable skills from large corpora on demand.
  • Findings: External skills can substantially improve downstream performance when agents correctly access and use them, but retrieval alone does not solve scalable skill augmentation.Current agents often fail to recognize when help is needed, select which retrieved skills to load, and translate skills into reliable reasoning and action gains.
  • Future agenda: The released corpus, benchmark, and analyses establish a research agenda for organizing, maintaining, retrieving, selecting, refining, internalizing, and accumulating external capabilities.The paper positions SRA as a counterpart to retrieval-augmented generation, emphasizing how agents access capabilities beyond their model weights.

A Dataset-Specific Construction Details

The appendix details how heterogeneous source-dataset supervision is transformed into unified, leakage-controlled gold skills for SRA-Bench. It emphasizes that these artifacts are manually constructed—not directly inherited—to support transparent, verifiable, and reproducible evaluation.

  • SRA-Bench converts dataset-specific signals—such as theorem names, logic patterns, tool workflows, medical calculators, mathematical concepts, and software libraries—into unified gold skills.The appendix traces each dataset from raw annotations through finalized reusable skill artifacts suitable for retrieval, incorporation, and end-to-end evaluation.
  • Each dataset’s construction process combines source materials, dataset-specific inputs for an initial LLM draft, expert revision, leakage control, and concrete annotation examples.The per-dataset presentation makes the construction procedure easier to inspect, verify, and reproduce.
  • Gold skills are manually constructed from structured supervision signals into standalone capability artifacts rather than directly inherited from source datasets.

A.1 TheoremQA · A.1.1 LLM Draft Generation · A.1.2 Expert Revision

TheoremQA skills are built from a text-only theorem dataset, generated through LLM drafts informed by questions, answers, and references, then expert-revised for procedural clarity and conceptual disambiguation. Each theorem maps to one gold skill, supporting reusable theorem-application guidance across STEM domains.

  • A.1 TheoremQA: The text-only TheoremQA subset contains 747 instances associated with 320 theorems after removing image-dependent questions.Each instance includes a question, theorem annotation, ground-truth answer, and answer type.
  • A.1 TheoremQA: The retained theorems cover mathematics, EECS, physics, and finance, with each instance linked to exactly one theorem and each theorem mapped to one gold skill.Domain, field, and subfield tags provide annotation-level context during skill construction.
  • A.1.1 LLM Draft Generation: For each theorem, the LLM drafts a reusable skill from theorem metadata, associated questions and answers, and domain-appropriate reference pages.Because solution steps are absent, the draft infers an application procedure from theorem labels, question distributions, and external references.
  • A.1.1 LLM Draft Generation: The draft format requires a canonical name, a one-sentence description, the theorem statement, recognition criteria, application steps, pitfalls, and a new worked example.Questions and answers characterize application scope, while answer-revealing content, shortcuts, and copied examples are removed during revision.
  • A.1.2 Expert Revision: Expert revision converts encyclopedia-style theorem summaries into step-by-step skills with explicit inputs, case distinctions, and computation templates when drafts under-specify procedures.This failure arises because the source data provide no solution steps.
  • A.1.2 Expert Revision: Expert revision also adds disambiguation cues, comparison rules, and recognition patterns when drafts confuse closely related theorems, formulas, or statistical concepts.Examples include Lah versus Stirling numbers and population versus sample covariance.

A.1.3 Construction Examples … A.3.2 Expert Revision

This section describes the construction of reusable skills across finance, combinatorics, logic, and ToolQA through explicit reasoning examples, domain-specific guidance, and dataset preparation. It also details drafting and expert revision processes that add executable workflows, disambiguation, data normalization, and independently constructed examples.

  • A.1.3 Construction Examples: Expert revisions strengthen Dividend Discount Model and Lah Number skills with structured procedures, distinction from related concepts, and new examples that avoid reusing benchmark instances.For DDM, the expert added a five-step multi-stage computation template and documented the D_n+1 versus D_n pitfall; for Lah numbers, the expert added Stirling-number comparisons and replaced an n=8, k=5 example with n=4, k=2.
  • A.2 LogicBench: LogicBench retains 19 distinct inference patterns across propositional, first-order, and non-monotonic logic, yielding 760 instances mapped one-to-one to gold skills.The benchmark contains binary and four-option questions, and pattern names plus logic categories provide the primary structured signals for skill construction.
  • A.2.1 LLM Draft Generation: LLM drafts combine pattern metadata, questions and answers, and Stanford Encyclopedia of Philosophy pages to synthesize reusable skills with formal rules, recognition cues, procedures, pitfalls, and new examples.Because LogicBench lacks intermediate reasoning or worked explanations, the LLM must derive application procedures from its own knowledge and the supplied materials.
  • A.2.2 Expert Revision: Expert revision adds LogicBench-specific guidance for extracting operative premises from narrative context and distinguishing valid rules from neighboring fallacies.The revisions address hedged or distractor content and contrasts such as modus tollens versus denying the antecedent or affirming the consequent.
  • A.2.3 Construction Examples: Logic construction examples show modus tollens revised with narrative-premise handling, invalid-form contrasts, and a new server-authentication example, while default reasoning targets irrelevant-information filtering.The modus tollens skill formalizes P → Q, ¬Q ⊢ ¬P and adds an explicit four-step application procedure; the default-reasoning example concerns applying defaults while ignoring irrelevant premises.
  • A.2.3 Construction Examples: Construction examples teach default reasoning by identifying the rule and exception, filtering irrelevant premises, and applying the rule to the remaining non-exceptional case.Expert revision expanded a generic draft into a five-step filtering procedure, added explanations of relevance inflation and narrative completion, and replaced an input-reusing example with a newly constructed one.
  • A.3 ToolQA: ToolQA retains seven non-mathematical domains with 1,430 instances, organized into 14 domain-by-difficulty annotation groups, each mapped to one gold skill.Easy instances generally extract one fact, while hard instances require compositional operations such as aggregation, comparison, filtering, or derived computation.
  • A.3.1 LLM Draft Generation: LLM drafts receive domain metadata, tools, schemas, generation templates, and QA examples to produce skills covering schemas, workflows, answer formats, and pitfalls.The templates expose systematic mappings from natural-language question patterns to data operations and answer formats, although inference-time agents do not see them.
  • A.3.2 Expert Revision: Expert revision makes ToolQA skills operational by specifying question-type workflows, tool-call sequences, arguments, and switching decisions.This addresses gaps in drafts that identify relevant fields but rarely explain executable chains such as FilterDB versus SQLInterpreter or retrieval followed by PythonInterpreter.
  • A.3.2 Expert Revision: Revised ToolQA skills also decode domain-specific representations, normalize formats, and construct valid tool inputs before execution.Examples include HHMM-style flight times, concatenated flight identifiers, and differing date formats across questions and corpora.

A.3.3 Construction Examples … A.4.2 Expert Revision

The paper illustrates skill construction across database, calendar, and clinical-calculator tasks, then uses expert revision to turn LLM drafts into complete, executable procedures. The examples emphasize matching tools to query scope, while MedCalc-Bench skills are grounded in worked explanations and canonical calculator specifications.

  • A.3.3 Construction Examples: Expert-revised skills distinguish single-record flight lookups from corpus-wide agenda aggregation, addressing tool limitations and representation mismatches that undermine naive retrieval workflows.Flight skills add identifier parsing, structured tool sequencing, HHMM decoding, and SQL fallback; agenda skills switch from RetrieveAgenda to PythonInterpreter for corpus-wide queries and convert date formats before filtering.
  • A.4 MedCalc-Bench: MedCalc-Bench contains 55 calculator names with 20 test instances per calculator, and each calculator is paired with one gold skill.Each instance includes a patient note, clinical-value question, calculator name, ground-truth answer, and step-by-step explanation.
  • A.4 MedCalc-Bench: Ground-truth explanations provide the primary abstraction material because they explicitly identify clinical variables and calculation procedures, while calculator names link instances to gold skills.The explanations are treated as more information-dense than raw patient notes for constructing reusable calculation skills.
  • A.4.1 LLM Draft Generation: LLM drafts receive each calculator name, 20 worked explanations, and its MDCalc3 reference page, then produce a structured reusable skill.The requested structure includes clinical context, required inputs and unit conversions, symbolic computation steps, a Python implementation, and a new hypothetical worked example.
  • A.4.1 LLM Draft Generation: The draft-generation prompt requires a canonical calculator name, a clinical-use description, input specifications, symbolic computation, executable code, and a worked example on a hypothetical patient.Worked examples supply generalization material, while the MDCalc page supplies canonical formulas and boundary conditions.
  • A.4.2 Expert Revision: Expert revision cross-checks drafts against all annotated instances and the MDCalc reference to restore rare or unobserved conditional branches missing from instance-level explanations.Examples include dialysis-dependent creatinine overrides and BMI-specific handling for underweight patients.
  • A.4.2 Expert Revision: Expert revision also validates implementations by checking outputs and correcting clamping, rounding, and boundary-condition behavior, making skills operationally reliable rather than merely descriptive.The revision process applies these checks to the accompanying executable tool as well as the textual procedure.

A.4.3 Construction Examples · A.5 CHAMP · A.5.1 LLM Draft Generation

The construction examples show that expert revision repairs omitted conditional branches in LLM-generated skills, while CHAMP and its prompting procedure organize concept-linked mathematics problems and worked solutions for reusable skill generation.

  • A.4.3 Construction Examples: Expert revision corrected two MELD Na omissions: the dialysis creatinine override and the MELD(i) ≤11 sodium-adjustment bypass, then reorganized the procedure into six explicit steps.The omitted dialysis branch occurred in 2 of 20 instances, while no instance triggered the omitted threshold branch.
  • A.4.3 Construction Examples: Expert revision replaced Cockcroft-Gault’s always-overweight draft with three BMI branches: actual weight for underweight, min(IBW, actual) for normal weight, and ABW for overweight patients.The revised worked example uses a newly constructed normal-BMI patient to exercise the min(IBW, actual) branch.
  • A.5 CHAMP: CHAMP instances contain competition mathematics questions, ground-truth answers, and official solution steps, with 89 unique concept annotations spanning eight mathematical domains.Each concept is accompanied by a formal concept description.
  • A.5 CHAMP: CHAMP uses multi-label annotations: 49% of instances require at least two concepts, with each concept corresponding to one gold skill.Instances are tagged with all concepts relevant to their solutions.
  • A.5.1 LLM Draft Generation: For each CHAMP concept, the LLM receives definitions, categories, reference pages, associated questions and answers, and official solution steps as generation context.The worked solution steps provide procedural context unavailable in TheoremQA or LogicBench.
  • A.5.1 LLM Draft Generation: The CHAMP prompt asks a mathematics education expert to write a reusable skill document teaching how to apply a concept to new problems.The prompt supplies the concept name, category, definition, reference material, and problems with their solutions.
  • A.5.1 LLM Draft Generation: The generated skill format requires a canonical name, one-sentence description, concept statement, recognition cues, step-by-step procedure, pitfalls, and a new worked example.The worked example must not be taken from the input problems.

A.5.2 Expert Revision

Expert revision for competition-math skills adds recognition patterns and systematizes solution strategies beyond LLM drafts’ theorem restatements and example calculations.

  • Expert Revision: Dataset-specific expert revision extends general revision criteria with recognition-pattern development and solution-strategy systematization.
  • Expert Revision: Experts add problem-type classifications and structural signatures that indicate when a concept applies across diverse competition-math problems.LLM drafts restate theorem definitions and demonstrate calculations without abstracting these recognition patterns.
  • Expert Revision: Experts organize each skill around a typology of problem types, each paired with a systematic procedure rather than a single usage mode.This addresses the multiple strategies that concepts such as modular arithmetic can support, including impossibility proofs, residue enumeration, and periodicity analysis.

A.5.3 Construction Examples … A.6.3 Construction Examples

The construction examples illustrate how expert revision transforms generic descriptions into reusable, task-oriented procedures and API recipes across mathematical, library, and matplotlib tasks. They emphasize strategic guidance, behavioral and return-type contracts, composition cues, and newly constructed examples that support reliable task execution without reproducing input instances.

  • A.5.3 Construction Examples: Expert revisions make AM-GM reusable by adding recognition cues for product-sum, reciprocal, and repeated-factor patterns, then demonstrating a new t + 4/t example instead of reusing an input instance.The revision also emphasizes non-negativity and compatibility of chained equality conditions.
  • A.5.3 Construction Examples: Expert revision transforms modular arithmetic from generic identities into a four-type workflow covering impossibility, residue enumeration, divisibility, and periodicity, with modulus-selection heuristics and a new 8k + 7 impossibility example.The revised skill classifies the problem before selecting a modulus, enumerating residues, and comparing residue sets or periods.
  • A.6 BigCodeBench: BigCodeBench converts each library annotation into a gold skill, allowing instances to have multiple gold skills while providing canonical solutions, tests, and library metadata for execution-based evaluation.The benchmark covers 139 Python libraries and averages 2.8 libraries per task, but lacks annotation-level descriptions, API summaries, and reusable usage guidance.
  • A.6.1 LLM Draft Generation: The LLM draft-generation process combines library documentation with task descriptions and canonical solutions to extract API calls, parameter configuration, return objects, and composition patterns into reusable skill documents.The prescribed document structure includes core API functions, common usage patterns with examples, and common errors or pitfalls.
  • A.6.2 Expert Revision: Expert revision replaces conceptual library summaries with precise API contracts and task-oriented recipes, capturing return types, parameter interactions, edge cases, and function combinations needed for code-generation tasks.For example, re.findall may return captured group contents, plt.hist() returns a tuple, and revised recipes organize operations by intended uses such as URL removal or tokenization.
  • A.6.3 Construction Examples: The re construction example illustrates this revision by documenting match anchoring, capturing-group semantics, substitution ordering, and distinct tokenization recipes rather than merely listing functions.The example highlights that re.findall with one capturing group returns group contents and that substitution order can affect introduced underscores.
  • A.6.3 Construction Examples: Matplotlib appears in 309 annotated data-visualization instances, where the central challenge is returning the exact object type expected by unit tests.Plotting functions may return Axes, tuples, ndarrays, or other specialized objects.
  • A.6.3 Construction Examples: The examples cover bar charts and standardized-matrix histograms, both returning a (DataFrame, Axes) tuple through concrete plotting workflows.The workflows combine pandas or sklearn transformations with seaborn or matplotlib plotting.
  • A.6.3 Construction Examples: The revision addresses omitted return-type specifications and reused input examples by adding a return-type table, OOP-versus-state-machine guidance, correct return patterns, and newly constructed examples.It identifies missing return contracts as the most common source of test failures and notes that omitting set_xticks before set_xticklabels raises a ValueError.
  • A.6.3 Construction Examples: The reference skill specifies plotting return types and usage patterns, including BarContainer for plt.bar, a tuple for plt.hist and ax.pie, and Axes for df.plot or series.plot.It also distinguishes object-oriented usage for returning Axes from state-machine-style plotting and documents numeric bar-chart positioning.

B Implementation Details … B.3 Progressive Disclosure

The implementation appendix defines prompt templates for exposing retrieved or annotated skills during inference. It covers full-skill injection, LLM-based skill selection, and progressive disclosure through on-demand skill loading.

  • B Implementation Details: The appendix’s templates specify how retrieved skills are exposed during inference, with {original user prompt} denoting the corresponding benchmark task prompt unless stated otherwise.
  • B.1 Full-Skill Injection: Full-Skill Injection prepends each provided skill’s complete content to the original user prompt.Skills may come from retrieval or Oracle annotation.
  • B.1 Full-Skill Injection: Multiple skills in Oracle mode are concatenated with a horizontal-rule delimiter, including multi-label CHAMP [17] and BigCodeBench [61] settings.
  • B.2 LLM Selection: LLM Selection presents the task query with numbered candidate skill names and descriptions, requiring the model to output only the most relevant skill number.The selected skill is then injected in full using the Full-Skill Injection format.
  • B.3 Progressive Disclosure: Progressive Disclosure augments the system prompt with a compact skill catalog and an instruction for requesting full skill content on demand.The model invokes loading by writing LOAD_SKILL: <index> on its own line.
  • B.3 Progressive Disclosure: When the model issues a LOAD_SKILL action, the framework injects the corresponding skill’s full content and prompts the model to continue.The model can load multiple skills across successive turns, up to 10 rounds.
Loading 2604.24594v3…