PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does a Skill Add Value? Task-Conditional Gain Prediction for Selective Skill Use

Anjie Xu, Zhiyu Zhang, Ruiqing Ding, Fengli Xu, Leye Wang

arXiv:2609.32274v1cs.AI

TL;DR

Skills may fail to improve an agent’s success, but their incremental benefit is difficult to know before execution. SKILLDELTA transfers paired skill/no-skill gains from historical tasks to new tasks; across five benchmarks and three agents, it beats random activation at matched use rates, with most gains from task-group allocation.

  • Problem

    The paper asks whether a supplied skill improves a fixed agent’s success on its current task, since skill-assisted success alone cannot show incremental benefit.

    Source: §1 · p. 1

  • Method

    SKILLDELTA transfers paired gains from eligible historical tasks to new tasks using a local predictor, without retraining the agent.

    Source: §1 · p. 2

  • Results

    Across five benchmarks and three agents, selective use improves success over random activation at matched expected skill-use rates in all 15 settings, with an average absolute gain of 4.3%.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The success advantage comes mainly from allocating skills across task groups, with the clearest additional within-group selection value on ToolQA.

    Source: §1 · p. 2, §7 · p. 10

    Reported token savings exclude bank construction, target encoding, and retrieval; agent or skill changes require rechecking the evidence.

    Source: §6 · p. 9

Abstract

from arXiv · show

Agent skills are expected to improve task performance. Yet we find that they often provide no benefit, and can even hurt performance while incurring additional token costs. Can we predict whether a skill will help before the agent acts? We introduce SkillDelta, a framework for predicting task-conditional skill gains from paired executions of the same agent with and without the skill. A local predictor transfers these historical gains to new tasks without retraining the agent. Under explicit transfer assumptions, our analysis links support coverage, representation mismatch, and execution noise to prediction error and decision regret. Across five benchmarks and three target agents, paired history improves observed-gain ranking over skill-assisted outcomes alone in 12 of 15 settings. At matched expected skill-use rates, SkillDelta improves success over random activation in all 15 settings, with an average absolute gain of 4.3%. Most of this advantage comes from allocation across task groups. Evidence for additional within-group selection value is strongest on ToolQA and weaker elsewhere. Code is available at https://github.com/TankTechnology/skilldelta.

1 INTRODUCTION

The paper asks whether a supplied skill will improve a fixed agent’s success on its current task, rather than merely whether the agent can succeed with the skill. SKILLDELTA predicts that incremental gain from paired execution history, and selective use improves success over random activation at matched use rates.

1 INTRODUCTION

  • Skills can duplicate existing ability, reduce success, or add tokens without benefit, so skill-assisted success alone does not establish that a skill helped.The paper distinguishes incremental benefit from the outcome achieved when the skill is used.Source: §1 · p. 1
  • SKILLDELTA transfers paired gains from eligible historical tasks to a new task using fixed task embeddings, without retraining the agent.Its analysis links prediction error and decision loss to support coverage, representation mismatch, and execution noise under explicit transfer assumptions.Source: §1 · pp. 1–2
  • At matched expected skill-use rates, selective use improves success over random activation by an average absolute 4.3% across five benchmarks and three agents; paired history improves gain ranking in 12 of 15 settings.The advantage comes mainly from allocation across task groups, with the clearest additional within-group value on ToolQA.Source: §1 · p. 2
  • The authors provide a SKILLDELTA plugin for DeepSeek Harness alongside implementation and data-collection code.The plugin gates skill injection before the first model step.Source: §1 · p. 2

2 PROBLEM DEFINITION: TASK-CONDITIONAL SKILL GAIN

The paper defines task-conditional skill gain as the change in a fixed agent’s success probability between skill and no-skill conditions. Paired executions estimate that gain, which can then guide whether to use the skill.

2.1 EXPECTED GAIN AND PAIRED OBSERVATIONS

  • Skill gain τ(t, s) is the difference between the agent’s success probabilities with the supplied skill and with no skill for the same task.The two conditions use the same agent and execution protocol, differing only in skill injection.Source: §2.1 · pp. 2–3
  • Two tasks with 80% success using the skill can have gains of zero and 40% when their no-skill success probabilities are 80% and 40%, respectively.Predicting skill-assisted success alone treats them alike even though only the second benefits.Source: §2.1 · p. 3
  • Paired observed gain subtracts the no-skill success fraction from the skill-condition fraction, while repetitions reduce execution noise under a fixed execution distribution.A single pair can record harm, no change, or help; pairing matches the task across conditions but not necessarily execution trajectories.Source: §2.1 · p. 3

2.2 SELECTIVE USE AS A DOWNSTREAM APPLICATION

  • The selective-use policy uses the skill when predicted gain exceeds threshold η; at η = 0, it selects positive predicted gains.A larger threshold requires a larger predicted benefit; evaluation compares success and token usage against always-off and always-on policies.Source: §2.2 · p. 3

3 METHOD AND THEORETICAL ANALYSIS

SKILLDELTA predicts a new task’s gain by transferring paired outcomes from eligible historical tasks. The analysis identifies conditions affecting transfer reliability and connects gain-prediction error to the quality and cost of use-or-skip decisions.

3 METHOD AND THEORETICAL ANALYSIS

  • SKILLDELTA transfers observed gains from similar historical tasks to predict whether a supplied skill will help a new task, using support outcomes rather than target execution.It links prediction errors to downstream decision loss.Source: §3 · p. 3

3.1 WHAT CAN TRANSFER?

  • Transfer requires historical and target task gains to be related through the task representation; residual ε_ϕ accounts for gain differences the representation does not explain.Topic similarity alone may miss differences in whether a skill’s procedure helps, as in definition versus unit-conversion questions.Source: §3.1 · p. 3

3.2 WHAT LIMITS GAIN PREDICTION?

  • The prediction bound separates coverage, representation residual, and execution noise: larger support can provide closer neighbors, more repetitions improve precision, and broader neighborhoods can trade lower noise for greater transfer bias.These evidence changes address different sources of error; the paper notes additional conditions for future-task guarantees.Source: §3.2 · p. 4

3.3 LOCAL PREDICTION FROM PAIRED EVIDENCE

  • The local predictor measures paired help or harm, selects eligible neighbors by skill or family scope and question similarity, then weights their gains to score the target before execution.Both helpful and harmful gains contribute; with no eligible support, the fallback is score zero and skip the skill.Source: §3.3 · pp. 4–5

3.4 DECISION ACCURACY AND LOSS

  • Decision accuracy depends on the gain’s margin from the threshold: sufficiently large margins tolerate prediction error, while at η = 0 regret is success probability lost relative to choosing with the true gain.Matched-use comparison against random activation measures recipient selection beyond simply using the skill more often.Source: §3.4 · p. 5

4 EXPERIMENT AND RESULTS

Across five benchmarks and three target-agent stacks, the experiments evaluate leave-one-task-out skill-gain prediction, its decisions, and factors affecting transfer. Paired evidence improves gain ranking in most settings, while selective skill use beats random activation at matched use rates; transfer error responds differently to support, neighborhood size, and repetition.

4.1 BENCHMARKS AND TARGET-AGENT STACKS

  • The evaluation covers four SRA-Bench domains and SpreadsheetBench across Qwen-Turbo, GLM-5.3-Flash, and DeepSeek-V4-Flash, with outcomes kept separate by stack.Tasks are run with and without supplied skills; most main-stack results use one paired execution per task, while LogicBench and all DeepSeek evaluations use three-repeat means.Source: §4.1 · p. 5

4.2 WITHIN-INVENTORY LEAVE-ONE-TASK-OUT EVALUATION

  • Leave-one-task-out prediction excludes both target outcomes and averages signed gains from up to six same-family neighbors, weighted by nonnegative cosine similarity; positive scores trigger skill use.The shared rule uses question-only embeddings and falls back to all available neighbors or Skip skill when support is empty.Source: §4.2 · p. 6

4.3.1 RQ1: SKILL-GAIN STRUCTURE AND PREDICTION

  • Paired history improves observed-gain AUROC over the with-skill-only reference on 12 of 15 panels, but lexical scores outperform it on BigCodeBench across all three stacks.The with-skill-only predictor also exceeds paired prediction on Qwen-Turbo BigCodeBench and SpreadsheetBench, and DeepSeek-V4-Flash ToolQA.Source: §4.3.1 · pp. 6–7, Figure 2 · p. 6

4.3.2 RQ2: DECISION VALUE OF GAIN PREDICTION

  • At the zero-threshold operating point, SKILLDELTA improves success over Always-off and saves tokens versus Always-on in all 15 panels, while success remains below Always-on in 12 panels.Across panels, mean success is 63.12% versus 64.04% for Always-on, with 20.8% mean token savings; direct self-judgment raises total tokens by 2.6–98.2% relative to Always-on.Source: §4.3.2 · p. 7
  • At the same expected skill-use rate, SKILLDELTA achieves higher success than random activation in all 15 panels, with an average absolute gain of 4.3%.Matching each recorded task group’s use rate separates the advantage into 0.61% from within-group selection and 3.72% from between-group allocation.Source: §4.3.2 · p. 7

4.3.3 RQ3: WHAT MAKES TRANSFER WORK?

  • Larger support pools generally help ToolQA and MedCalc-Bench but are non-monotonic across LogicBench agents; larger neighborhoods trade reduced noise for possible transfer bias, while repetitions reduce MAE in all nine panels with diminishing returns.Support helps when it improves the selected neighborhood and preserves relevant skill-effect structure; repetition improves gain precision without changing coverage or representation.Source: §4.3.3 · p. 9

5 RELATED WORK

Related work spans skill utility and retrieval, task-specific performance prediction, heterogeneous-effect estimation, and transfer under selective decisions. SkillDelta focuses on predicting incremental skill benefit for a fixed agent.

Skills, retrieval, and adaptive augmentation.

  • Prior skill research examines utility, retrieval, and content optimization, while tool retrieval and adaptive augmentation study which external resources to use and when.The related work also includes skill testing and coverage.Source: §5 · p. 9

Instance-wise performance and heterogeneous effects.

  • Algorithm selection, performance prediction, and heterogeneous-effect methods estimate task-specific outcomes or treatment effects; shared local weights recover SkillDelta’s signed-gain estimator.Repeated agent tasks permit measurement under both skill-use conditions and study of transfer to new tasks.Source: §5 · p. 9

Transfer and selective decisions.

  • Domain adaptation studies coverage and conditional shift, while selective and conformal prediction formalize uncertainty; SkillDelta connects paired-history coverage and precision to skill-use decisions.Source: §5 · p. 9

6 DISCUSSION AND LIMITATIONS

Prediction is constrained by whether task representations reflect similarity in skill benefit and whether local groups provide enough evidence. Deployment is most practical when paired records can be reused under stable agent and skill conditions.

6 DISCUSSION AND LIMITATIONS

  • TF–IDF raises gain-sign AUROC on three of five Qwen-Turbo benchmarks, including BigCodeBench from 0.527 to 0.582 under the same paired history.Semantic similarity alone need not preserve similarity in skill benefit.Source: §6 · p. 9
  • BigCodeBench’s sparse groups and heterogeneous skill bundles may constrain local evidence, while its natural-shift panel shows success gains over Always-off as task-level ranking weakens.Source: §6 · p. 9
  • Reuse is practical under stable agent and skill conditions, but token savings exclude bank construction, target encoding, and retrieval.Building a bank from scratch requires an investment whose value depends on reuse; agent or skill changes require rechecking its evidence.Source: §6 · p. 9

7 CONCLUSION

SKILLDELTA predicts before execution whether a supplied skill improves a fixed agent’s chance of success. Across five benchmarks and three agents, selection improved success over random activation at matched expected use, with most gains from task-group allocation.

7 CONCLUSION

  • Across five benchmarks and three agents, selection improves success over random activation at the same expected skill-use rate.The gains arise mainly from allocation across task groups.Source: §7 · p. 10
  • The clearest additional within-group selection value appears on ToolQA.Source: §7 · p. 10
  • Paired history can guide which kinds of tasks receive a skill, even where finer task selection is weak.Source: §7 · p. 10

AI USE STATEMENT

The authors led the research and used AI tools to assist with writing, coding, and analysis. They revised and checked the content and take responsibility for the final paper.

AI USE STATEMENT

ETHICS STATEMENT

The study uses existing agent benchmarks and collects no new human-subject data; safety-critical instruction applications require a separate assessment beyond the task-success and token metrics studied.

ETHICS STATEMENT

  • Safety-critical instruction applications require a separate safety assessment beyond the task-success and token metrics studied.The work uses existing agent benchmarks and collects no new human-subject data.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The appendix links historical evidence to gain prediction and then to skill-use decisions, while treating policy evaluation separately. Its guarantees distinguish uncertainty from support outcomes, target-task variation, and support-task sampling.

REPRODUCIBILITY STATEMENT

  • The public release provides code and evaluation utilities; experimental protocols and proofs appear in the appendix, while retained outcomes and analysis scripts are in a separate review supplement.The review supplement contains task-level outcomes, embeddings, and scripts for recomputing main metrics and additional analyses.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 READING GUIDE AND NOTATION

  • The analysis distinguishes uncertainty over support execution outcomes, a fresh target task, and support-task sampling; policy evaluation separately uses recorded executions to assess a chosen policy’s value.Figure 5 presents the chain from historical evidence through prediction to decision guarantees, with policy evaluation as a separate argument.Source: §A.1 · p. 13, Figure 5 · p. 13

A.2 IDENTIFICATION AND OBSERVED GAINS

  • Without a restriction relating support gains to an unseen target, observed support executions cannot identify its gain sign; a single paired label can also differ from expected gain.With independent success probabilities of 1/2 in both conditions, expected gain is zero, while one paired execution can yield an observed gain of +1 or −1.Source: §A.2 · p. 13

A.3 LOCAL GAIN-PREDICTION BOUND

  • The prediction bound separates transfer mismatch from execution noise; more neighbors can reduce noise but enlarge mismatch, while repeated executions improve precision at fixed coverage.Diffuse weights and repetitions reduce the noise term; family averaging is useful when effects vary little within the family.Source: §A.3 · p. 14
  • At δ = 0.05 and k = 6, the minimum noise term is approximately 1.109 for r = 1 and 0.640 for r = 3; the r = 1 case cannot certify a zero-threshold decision.The bound illustrates the trade-off, while numerical certification requires more evidence or tighter uncertainty estimates.Source: §A.3 · p. 14

A.4 FROM GAIN ERROR TO DECISIONS

  • When the error bound holds, disagreement with the threshold oracle implies the true gain lies within the bound of the threshold, and routing regret is at most the bound.The regret bound applies on the event specified by Theorem 1.Source: §A.4.1 · p. 14, §A.4.2 · p. 14

A.4.3 UNCERTAINTY BANDS AND SELECTIVE ABSTENTION

  • With a valid error band, the conservative binary rule never uses the skill when τ(t) ≤ η and has threshold-relative regret at most 2Bδ(t).Source: §A.4.3 · p. 15

A.5 FRESH-TASK GENERALIZATION

Across fresh-task generalization, policy evaluation, empirical controls, and task–skill structure, the paper connects support coverage, representation mismatch, and execution noise to gain prediction and decision loss. Empirically, matched-rate selection value is driven mainly by allocation across task groups, with the clearest within-group evidence on ToolQA.

A.5 FRESH-TASK GENERALIZATION

  • A margin condition bounds decision disagreement by the probability of targets near the decision threshold, while mean gain error also bounds mean regret.When Bδ(T) ≤ b and PrT(|τ(T) − η(T)| ≤ u) ≤ Cu^α, the disagreement probability is at most δ + Cb^α.Source: §A.5 · pp. 15–16
  • Fresh-task gain prediction requires support near target tasks and a representation that captures similarity in skill benefit.The analysis links coverage, representation mismatch, and execution noise to prediction error and decision loss under explicit transfer assumptions.Source: §A.5 · pp. 15–16

A.6 EVALUATING FROZEN AND CROSS-FITTED POLICIES

  • For a fixed finite target inventory, a policy chosen independently of execution outcomes has an unbiased estimate of mean success.Leave-one-task-out actions fitted from other tasks’ outcomes satisfy the stated marginal identity under conditional execution independence; configuration selection still requires separate validation.Source: §A.6 · pp. 16–17

B.2 PREDICTION AND BASELINE CONTROLS

  • The main empirical comparison uses target-excluded, same-family k = 6 cosine-weighted signed-gain scores and enables skills only for positive scores.The common rule harmonizes earlier benchmark-specific analyses retrospectively; configuration selection is outside the evaluation.Source: §B.2 · p. 17

B EMPIRICAL PROTOCOLS AND SUPPLEMENTARY RESULTS

  • Table 3 collects outcome-free relevance and self-judge diagnostics, Table 4 compares Qwen-Turbo estimators given the same paired history, and Figure 6 compares matched-rate selection value across agents and benchmarks.Table 5 gives task-bootstrap intervals for main results as absolute success-rate gains over Always-off, with scores and actions fixed across 10,000 task resamples.Source: §B.2 · p. 17, §B.3 · p. 18, Table 5 · p. 19

B.3 POLICY VALUE AND UNCERTAINTY

  • 3.72% of the 4.33% matched-rate advantage comes from between-group allocation, versus 0.61% from within-group selection.ToolQA provides the strongest within-group evidence, with a three-agent mean of 4.41% and every agent’s interval excluding zero; all remaining within-group intervals include zero.Source: §B.3 · p. 20, Figure 7 · p. 20

B.4 SINGLE- AND DUAL-RIDGE POLICY CONTROLS

  • Ridge policy controls compare single- and dual-head estimators on identical fixed test tasks, with intervals excluding refitting and policy-selection uncertainty.The dual cost head yields additional savings in several panels, with varying test success; an independent offline refit reproduces reported metrics on all 15 panels.Source: §B.4 · p. 20, Table 6 · p. 21, §B.5 · p. 21

C TASK–SKILL STRUCTURE AND SUPPORT COVERAGE

  • Small or singleton task groups limit both reusable within-group evidence and opportunities for within-group selection, while group membership may not fix the injected skill bundle.The comparison does not isolate sparsity from bundle heterogeneity or representation.Source: C TASK–SKILL STRUCTURE AND SUPPORT… · pp. 21–22

D GENERALIZATION STRESS TESTS

Prospective stress tests evaluate frozen SKILLDELTA predictors on unseen tasks and shifted task mixtures, with predictions fixed before target outcomes. Results distinguish gains over skipping from task-level gain ranking, and none of the panels tests unseen skills.

D GENERALIZATION STRESS TESTS

  • ToolQA retains support families and has a positive success gain, while its token-saving interval crosses zero.MedCalc changes the task inventory and calculator-family mixture under the same agent; both provide prospective evidence within supported task mixtures.Source: D GENERALIZATION STRESS TESTS · p. 22
  • Under the natural-distribution extension, SKILLDELTA retains a success gain over Always-off while task-level ranking falls to chance.The BigCode scale panel follows a pilot, so its evidence is sequential; the natural panel extends to the remaining population.Source: D GENERALIZATION STRESS TESTS · p. 22, Table 7 · p. 22
  • The prospective panels use frozen historical configurations and do not evaluate unseen skills.Predictions were serialized before target outcomes were collected; the panels differ in target construction.Source: D GENERALIZATION STRESS TESTS · p. 22, Table 7 · p. 22
Loading 2609.32274v1…