PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents

Seonghyeon Cho, Chanjun Park

arXiv:2609.00549v1cs.CL

TL;DR

Standard evaluations do not cleanly answer whether retrieved skills help on the exact tasks where agents use them, because aggregate comparisons are affected by task-selection bias. The paper formalizes Skill Following and introduces RAE, a paired same-task metric conditioned on retrieval-invoked executions. Across 17 LLMs and coding and mathematical domains, aggregate retrieval lift frequently appears positive while RAE is negative, including on MBPP+ for multiple models.

  • Problem

    Aggregate evaluations compare whole-benchmark or self-selected task subsets, leaving the effect of retrieved skills on the exact retrieval-invoked tasks unresolved.

  • Method

    RAE pairs skill-enabled and skill-disabled outcomes on the same tasks where retrieval was actively invoked and a skill was returned, measuring the retrieval-to-answer Skill Following chain.

  • Results

    Across 17 LLMs spanning coding and mathematical reasoning, aggregate retrieval conclusions frequently reverse under RAE, with positive aggregate lift often accompanying negative actual-use effects.

  • Takeaways & Limitations

    Aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE measures whether retrieval-invoked skill use rescues more outcomes than it harms.

  • Takeaways & Limitations

    The evaluation covers single-pass coding and mathematical tasks under a fixed retrieval interface and fixed procedural skill libraries, so its prevalence and magnitude may not generalize to other settings.

Abstract

from arXiv · show

Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.

1 Introduction

Standard retrieval evaluations can confound skill utility with which tasks retrieve skills. The paper formalizes Skill Following and introduces RAE to measure same-task effects on retrieval-invoked executions, revealing frequent reversals between aggregate lift and actual-use outcomes.

  • Evaluation gap: Aggregate metrics compare whole-benchmark performance or self-selected retrieved and skipped task subsets, leaving same-task skill utility unresolved.Retrieved and skipped tasks may differ in difficulty, prompt complexity, or skill-library relevance.
  • Skill Following: Skill Following denotes the complete retrieval-to-answer process, including receiving, interpreting, grounding, and synthesizing retrieved content into a valid answer.Retrieval is only the preamble of tool use; successful use requires integrating the returned skill into reasoning.
  • Findings: On MBPP+, positive aggregate lift can mask negative same-task effects because selective retrieval on easier tasks may coincide with reasoning disruption after skill integration.The reported pattern is that retrieval appears beneficial system-wide while rescuing fewer outcomes than it harms on retrieval-invoked tasks.
  • RAE: RAE pairs skill-enabled and skill-disabled outcomes on the exact tasks where retrieval was actively invoked and a skill was returned.This protocol conditions evaluation on retrieval-returned tasks to isolate actual-use performance.
  • Findings: 17 LLMs across coding and mathematical reasoning show frequent reversals between aggregate retrieval conclusions and RAE.The paper reports that positive aggregate retrieval lift can coexist with negative actual-use effects.

2 Related Work

Prior agent benchmarks and skill-access datasets evaluate system behavior or paired skill conditions but generally do not isolate the effect of retrieved content on the exact tasks where retrieval occurred. RAE addresses this gap by conditioning paired comparisons on retrieval-invoked executions.

  • Agent evaluation: Agent benchmarks measure end-to-end success or process diagnostics but do not isolate whether retrieved content changed the final outcome of the same task.The cited benchmarks include AgentBench, AgentBoard, WebArena, OSWorld, TheAgentCompany, and τ-bench.
  • Skill benchmarks: SkillsBench, SWE-Skills-Bench, and SkillLearnBench use paired skill conditions, but their effects remain predominantly aggregated over entire evaluation sets.Their paired designs control task identity without restricting analysis to tasks where retrieval was actively invoked.
  • Retrieval and memory: Autonomous retrieval and memory frameworks demonstrate agent-initiated external-artifact use but typically lack paired skill-disabled baselines for each retrieval-invoked execution.The passage names Voyager, Agent KB, ExpeL, Reflexion, and CRITIC as examples.

3 Skill Following

The paper formalizes Skill Following as a complete retrieval-to-answer chain and evaluates it with paired, same-task outcome metrics. RAE isolates tasks where retrieval returned a skill, while OAE measures skill access across the full benchmark.

  • 3 Skill Following: Skill Following is evaluated through strictly paired skill-enabled and skill-disabled executions on identical tasks.This design anchors comparisons to fixed task identities and neutralizes cross-population difficulty confounding.
  • 3 Skill Following: RAE measures the paired outcome difference only on tasks where the skill-enabled execution actively invoked retrieval and successfully returned a skill.The retrieval-invoked subset is denoted Scall, with size ncond = |Scall|.
  • 3 Skill Following: OAE measures the average outcome shift from exposing an agent to skills across the entire evaluation corpus.It includes tasks where the skill-enabled execution bypassed retrieval, so it captures global skill-access utility but dilutes actual-use signal.
  • 3 Skill Following: Aggregate retrieval lift compares self-selected retrieved and non-retrieved tasks, whereas RAE is an unconfounded same-task metric.RAE targets whether retrieval-invoked executions improved outcomes relative to matched skill-disabled baselines.
  • 3 Skill Following: RAE complements rather than replaces OAE by isolating the actual-use component of the broader skill-access effect.The paper reports RAE for each model–skill-pool–retrieval-policy configuration alongside helpful and harmful transition counts.

4 Experiment Setup

The experiments evaluate the measurement framework across diverse models, coding and mathematical benchmarks, and controlled skill environments. Supplementary controls and annotations diagnose how retrieved skill content relates to observed execution behavior.

  • 4 Experiment Setup: The evaluation covers 17 LLMs spanning multiple parameter scales, closed-source APIs, and open-weight models.The panel is intended to support broad generalizability.
  • 4 Experiment Setup: The primary analysis samples 80 MBPP+ tasks, with HumanEval+ testing cross-benchmark consistency and 80-task Math500 partitions providing cross-domain replication.MBPP+ uses seed 42.
  • 4 Experiment Setup: The study uses fixed procedural skill libraries containing 9 coding skills and 8 mathematical skills.The coding pool supports MBPP+ and HumanEval+, while the math pool supports Math500.
  • 4 Experiment Setup: Skills are structured Markdown files indexed with BM25, and search_skills(query) appends the top-3 retrieved skill texts to the agent context.Each skill includes frontmatter and a body with application conditions, code snippets, invariants, and self-tests.
  • 4 Experiment Setup: Each task runs in parallel skill-enabled and skill-disabled conditions sharing the same prompt, generation seed, and decoding configuration.The skill-enabled condition exposes search_skills(query), while the baseline removes the tool definition.
  • 4 Experiment Setup: Diagnostic setups perturb retrieved content through Normal, Schema-Empty, Filler-Dummy, Random-skills, and Corrupted environments.These controls vary whether returned content is usable, informative, relevant, or misleading.
  • 4 Experiment Setup: A blinded GPT-5.5 annotator labels retrieval-invoked traces for adherence, independence, misapplication, interface failure, or unclear behavior.The labels are descriptive diagnostics and remain separate from RAE computation.

5 Results

Across coding and mathematical benchmarks, aggregate retrieval metrics frequently disagree with RAE, often overstating the benefit of retrieved skills. The discrepancy reflects task-selection bias and failures to convert retrieved content into successful final answers.

  • Metric Disagreement: Sign reversals between positive aggregate retrieval lift and negative RAE are common across reportable coding configurations.On MBPP+, repeated reversals appear across model configurations; on HumanEval+, 3 of 13 models show disagreement.
  • Metric Disagreement: RAE compares skill-enabled and skill-disabled outcomes on the same tasks where retrieval was invoked, removing mismatched-task population effects.Aggregate lift instead compares retrieved and non-retrieved subsets selected by the model’s retrieval policy.
  • Cross-Benchmark Consistency: 3 out of 13 HumanEval+ models show disagreement, with aggregate retrieval lift negative but RAE positive for Qwen3-8B, Qwen3.5-9B, and Qwen3-235B.This demonstrates that aggregate comparisons can misestimate retrieval-invoked actual use in either direction.
  • Cross-Domain Results: On Math500, Llama-3.3-70B records +14.2 pp aggregate lift versus −39.4 pp RAE, while Gemini-2.5-Flash-lite records +13.2 pp versus −11.0 pp.The disagreement therefore extends beyond coding into mathematical reasoning.
  • Retrieval Coverage and Integration: Several models with high retrieval coverage still produce negative RAE, showing that invoking retrieval does not reliably yield successful Skill Following.Retrieval is only exposure; the model must interpret, align, integrate, and format the returned content correctly.
  • Mechanistic Failures: Negative RAE persists across non-empty skill-content controls, indicating that structured skill text alone does not determine actual-use outcomes.The measured effect reflects the full chain from invocation and injection through relevance assessment, integration, and final-answer formatting.

6 Conclusion

Across coding and mathematical domains, standard metrics can make skill use appear beneficial while retrieval-invoked execution harms performance on the same tasks. RAE addresses this by pairing skill-enabled and skill-disabled outcomes where retrieval occurs.

  • RAE pairs skill-enabled and skill-disabled executions on the exact tasks where retrieval occurs, measuring the full Skill Following chain.This isolates retrieval-invoked use from whole-benchmark averages and self-selected task comparisons.

Limitations

The evaluation covers a focused setting and RAE has bounded statistical and causal interpretation. Its results should not be generalized beyond the tested protocols and domains without further analysis.

  • The experiments use single-pass coding and mathematical tasks with a fixed retrieval interface and fixed procedural skill libraries.The authors caution against extending prevalence or magnitude claims to long-horizon, iterative, language-centric, or executable-code settings.
  • RAE is a protocol-conditional paired outcome signal, not an unbiased causal effect over pretreatment tasks or a model-level skill-use ability score.Finer causal decomposition would require oracle retrieval, metadata-only retrieval, explicit full-skill fetch, varied skill pools, and adjudicated causal labels.

Ethical Considerations

The paper’s ethical relevance is evaluation reliability: aggregate metrics can hide harmful retrieval-invoked skill use and encourage misleading deployment claims. RAE improves visibility of this failure mode but is not a safety guarantee.

  • Aggregate metrics may present skill-augmented agents as improved systems while hiding cases where retrieval-invoked skill use harms the same tasks.The authors connect this risk to misleading deployment claims and weaker auditability.
  • RAE values remain specific to the evaluated protocol, model, retrieval policy, skill library, and benchmark, so they should accompany complementary diagnostics and reproducibility artifacts.

A Metric Computation and Reporting Filters

The appendix computes paired RAE and aggregate retrieval lift from pooled task-level counts, then applies reporting filters for sparse or undefined comparisons. Sign-disagreement analyses retain only reportable cells and use paired statistical procedures for uncertainty.

  • Metric computation: Cell-level metrics pool raw counts across config-identical reruns before computing the reported rate.Cells fix the model, benchmark, partition seed, skill pool or control condition, and evaluation protocol.
  • Metric computation: RAE aggregates helpful and harmful transitions over tasks where retrieval was invoked and at least one skill was returned.Helpful transitions are SD failures paired with SE successes; harmful transitions reverse that outcome.
  • Metric computation: The paired outcome difference over any subset S is (b − c)/|S|, because concordant pairs contribute zero while helpful and harmful transitions contribute +1 and −1.
  • Metric computation: Aggregate retrieval lift averages paired skill-enabled/skill-disabled differences over the entire task set, including tasks where no skill was retrieved.This distinguishes it from RAE’s retrieval-invoked conditioning.
  • Reporting filters: Aggregate retrieval lift is undefined when either retrieved or non-retrieved task populations is empty.For example, this occurs when the agent invokes retrieval on every task.
  • Reporting filters: Sign-disagreement prevalence uses ncond ≥10 and excludes cells with undefined aggregate retrieval lift from its denominators.Counts remain stable under alternative cutoffs of 5, 15, and 20.
  • Reporting filters: Table 5 reports sign-disagreement cells divided by reportable cells across retrieval-invoked thresholds, while Table 6 reports aggregate retrieval lift and RAE in percentage points.Bold Table 6 entries indicate opposite metric signs; dashes mark non-reportable cells.
  • Reporting filters: A sign disagreement means aggregate retrieval lift and RAE have opposite signs, which is distinct from task-level helpful and harmful transitions.

B Full Coding-Domain Results

The full coding-domain panel reports results across all evaluated models, while the Math500 replication extends the comparison to mathematical reasoning. Reportability depends on the retrieval-invoked subset and denominator availability.

  • B Full Coding-Domain Results: 17 models comprise the full coding-domain panel underlying the main coding results.The appendix retains all evaluated models, including non-reportable cells.
  • B Full Coding-Domain Results: Reportable results exclude cells with empty or insufficient retrieval-invoked subsets or undefined aggregate retrieval-lift denominators.Dashes remain in the full panel for transparency but are excluded from sign-disagreement denominators.
  • B Full Coding-Domain Results: Math500 provides a cross-domain replication reporting aggregate retrieval lift, OAE, RAE, and retrieval-invoked transition counts.The reported transition counts are used to compute RAE.

D Cross-Benchmark Correlation Details

Cross-benchmark analyses show that aggregate retrieval lift and OAE correlate more consistently across MBPP+ and HumanEval+ than RAE. These correlations bound interpretation: RAE is conditional on each configuration’s retrieval-invoked subset, not a benchmark-invariant model ranking.

  • D Cross-Benchmark Correlation Details: 13 models meet the filtered-correlation requirement of ncond ≥10 on both MBPP+ and HumanEval+.Four models are excluded because at least one benchmark has fewer than 10 retrieval-invoked tasks.
  • D Cross-Benchmark Correlation Details: Skill-disabled accuracy is used as the baseline ability control for benchmark-cell comparisons.Partial correlations residualize metric values against the specified skill-disabled accuracy controls before correlating residuals.
  • D Cross-Benchmark Correlation Details: Aggregate retrieval lift has the highest raw cross-benchmark correlation, while OAE has the highest skill-disabled-accuracy-controlled partial correlation.These correlations are reported as diagnostic context rather than evidence that RAE is the most stable metric.
  • D Cross-Benchmark Correlation Details: RAE is a protocol-conditional actual-use signal over each configuration’s retrieval-invoked subset, not a benchmark-invariant ranking of skill-use ability.Stage diagnostics are descriptive proxies and do not define RAE.
  • D Cross-Benchmark Correlation Details: The harmful-transition taxonomy counts skill-disabled-correct/skill-enabled-wrong retrieval-invoked executions and assigns each one primary failure-mode label.The unit is a log-level paired execution, and pooled counts are qualitative diagnostic evidence rather than independent task-frequency estimates.
  • D Cross-Benchmark Correlation Details: GPT-5.5 labels retrieval-invoked answers using task prompts, retrieval queries, retrieved excerpts, and final answers while remaining blinded to disabled answers and correctness labels.The labels diagnose skill uptake behavior and remain separate from RAE computation.

G.1 Human Audit of Annotation Reliability

The human audit evaluates the reliability of GPT-5.5’s post-hoc adherence labels for harmful retrieval transitions. Fine-grained agreement is modest, whereas the binary engaged distinction is more stable, supporting cautious diagnostic interpretation.

  • G.1 Human Audit of Annotation Reliability: Nine non-author annotators independently provide 150 labels for a sample of 50 harmful transitions.The audit tests agreement with GPT-5.5 labels while hiding the model labels and paired outcome information.
  • G.1 Human Audit of Annotation Reliability: Annotators classify how skill-enabled answers handled retrieved skills using five primary labels, including appropriate adherence, ignored or independent use, and format/interface failure.The instructions explicitly prohibit judging task correctness and require an unclear label when evidence is insufficient.
  • G.1 Human Audit of Annotation Reliability: 84% human–human and human-majority–GPT agreement is reported for the binary engaged/not engaged distinction.This binary distinction is more stable than the original five-way taxonomy.
  • G.1 Human Audit of Annotation Reliability: Agreement is modest under the original five-way taxonomy, indicating that fine-grained category boundaries are interpretive.The study therefore treats labels as descriptive post-hoc diagnostics rather than as part of RAE.
  • G.1 Human Audit of Annotation Reliability: The audit conclusion is limited to the low prevalence of appropriate adherence among the audited harmful transitions.This scope follows the reported reliability pattern and does not generalize the labels beyond the audited cases.
Loading 2609.00549v1…