Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement

1 Introduction

• Prior rules often retire on gain alone, but hubs need a reliable per-model verdict because a wrong retirement can withhold a skill users may later need.The cited approaches include curricula, lifecycle methods, and library curation rules.Source: §1 · p. 1
• Gain-only retirement confuses internalized skills with uncovered ones because both can produce near-zero gain, even though uncovered tasks fail with and without the skill.The unaided level distinguishes the two states: it is high for internalized skills and low for uncovered ones.Source: §1 · p. 1
• The paper frames retirement as a certificate of unaided capability and introduces a validity-gated protocol, reliability analysis, and a release of 1,251 skills with tasks and item scores under four models.The protocol trusts a high unaided score only when another model gains from the skill.Source: §1 · p. 2
• With one task per skill, the published band rule removes more than half of uncovered skills under each of four models; on SkillsBench, level-rule retirements cost no detectable reward.The level verdict reproduces across tasks better than the band verdict, but must be read separately for each model.Source: §1 · p. 2

2.1 Dataset construction

• Tasks are synthesized from public skills to state a goal without supplying the skill’s taught convention or answer; cross-family generation and a difficulty gate make the set adversarial to the target.Tasks already passed on more than half their rubric are regenerated and dropped if the retry is also too easy; a critic rejects tasks that cannot be performed as written.Source: §2.1 · p. 2

2.2 Unaided capability score

• Internalization is measured behaviorally by withholding the skill and scoring the model’s unaided level on items testing the capability the skill teaches.The model’s competence is considered internalized when that capability is already available without the skill.Source: §2.2 · p. 2

2.3 Capability items

• Capability items require skill-stated information a competent engineer could not reliably supply otherwise; scoring all task-completion items correlates 0.93 with the selected-item ordering.The audit also reproduces on all task-completion items: the band retires 51.7% of uncovered skills, while the level rule retires 15.7 points more of the internalized state than the band.Source: §2.3 · p. 3

2.4 Validity gate

• A task is admitted only when another model gains from the skill on the same items, indicating that the skill carries usable information rather than merely an easy task.The gate admits 844 of 1,165 tasks (72.4%), but only 52.6% of tasks with unaided score U≥0.8.Source: §2.4 · p. 3

3.1 Three states

• The band retires both low-score uncovered skills and high-score internalized skills because both can have near-zero gain.The uncovered state is 8.6% of the gated adversarial library and 5.9% of Tessl-single; within the band, unaided scores span 0 to 1.Source: §3.1 · p. 3

3.2 Formal analysis

• Any retirement criterion based only on a signal that vanishes when aided and unaided scores are equal gives the same verdict across that diagonal, including uncovered and internalized cases.The difference, log-ratio, mutual information, and single-skill leave-one-out or Shapley attribution satisfy this condition [9].Source: §3.2 · p. 4
• With multiple binary runs per arm, the band’s error persists as runs increase, whereas the level rule’s risk tends to zero if the skill distribution puts no mass at its threshold.The residual error is within a single task, not task noise.Source: §3.2 · p. 4

3.3 Auditing published rules

• On the gated library, the band retires 57.5% of uncovered skills, while the level rule removes 15.9 percentage points more internalized skills than the band.When verdict and state are read on different tasks, the band retires 45% of uncovered skills versus 14% for the level rule.Source: §3.3 · p. 5
• Under every target model, the gain band removes more than half of uncovered skills; on binary outcomes it removes 80% to 99%.The level rule removes 13 to 16 percentage points more internalized skills than the band.Source: §3.3 · p. 5, Table 3 · p. 5

4.1 Reliability

• Parallel-forms reliability is 0.41, while the level rule’s verdict agreement across tasks is κ=0.42 versus 0.23 for the band.Internal consistency is 0.907 and test–retest reliability is 0.90 at the three-run mean; the label’s single-task validation cap is 0.64, rising to 0.84 with four same-batch tasks.Source: §4.1 · pp. 5–6, Figure 3 · p. 6
• At equal three-run budgets, three tasks with one run each misclassify 11.4% of the gated library, versus 16.1% for one task run three times.Adding tasks only within 0.2 of the threshold reaches 11.1% error at three quarters the cost of tripling runs everywhere; one run per task is optimal unless task generation costs more than about three runs.Source: §4.1 · pp. 5–6
• Two judges correlate at 0.89 and agree on retirement verdicts for 91.4% of trajectories, although a third judge found 9% of decidable full marks unsupported.The target model’s judge did not score its own runs systematically higher; disagreements split 20 to 18 between judges.Source: §4.1 · p. 6

4.2 Item validity

• The validity gate establishes that a skill carries usable information for a task, but cannot establish that each item requires that information to pass.Among capability items, 10.0% were satisfied unaided by all four models on every run; these items occur in 29.4% of skills, with a median of 30% of affected-skill items.Source: §4.2 · p. 6

4.3 Correcting for the difficulty gate

• Regenerating 115 skills removed by the difficulty gate yields 69.6% reaching the retirement threshold unaided, versus 40.4% of retained skills.Post-stratification lowers the uncovered base rate from 8.6% to 7.0%, while the band’s share of uncovered skills retired changes little, to 57.2%.Source: §4.3 · pp. 6–7

4.4 Verdicts across deployment models

• Across model pairs, verdicts flip for 14%–23% of skills, and 37% receive contradictory verdicts somewhere among the four models.Resampling runs flips only 5%–9% of verdicts, and 19%–32% of retirements under one model are skills that work for another; re-labelling a new model costs 61% less than starting from scratch.Source: §4.4 · p. 7

4.5 From label to decision

• Because the unaided score averages over items, the procedure uses a high retirement threshold, adds tasks near it, and makes retirement reversible.A lower threshold can retire skills with partly missing content.Source: §4.5 · p. 7

5.1 Executing the rules on SkillsBench

• The level rule retains 91%–97% of retrieval utility across three models, versus 70%–80% for random retirement of the same size.The level rule retires 19%–29% of the library across those models.Source: §5.1 · p. 7
• The benchmark analysis uses 59 admitted probes from 92 probes selected to avoid giving away rubric items, and executes retirements by deleting skills from task environments.The 200 benchmark skills yield 182 scorable probes; two tasks are excluded because their tests reward following a skill’s procedure rather than producing the right output.Source: §5.1 · p. 7
• −0.2 points [−2.7, 2.4] is the level rule’s mean reward change on SkillsBench, with no detectable cost; deleting every skill costs 19.1 points [11.3, 27.0].The benchmark detects no difference between the level rule and the native band: ΔL−ΔbinB is −0.8 points [−3.1, 1.7].Source: Table 4 · p. 8, §5.1 · p. 8
• 28 of the native band’s 31 retirements rest on probes that no trial in either arm ever passed in full.For these probes both binary pass rates are zero, despite the skill raising scores on capability items; the all-items criterion turns partial success into two zeros.Source: §5.1 · p. 8

5.2 Replication on Tessl-single

• On Tessl-single, parallel-forms reliability is 0.364 versus 0.41, and the uncovered state appears with a near-zero unaided score.The library is neither gated nor adversarial, and its tasks depend more strongly on their skills, with mean gain 0.384 versus 0.306.Source: §5.2 · p. 8

5.3 Evidence from automatic skill retrieval

• With automatically retrieved skills, 77% of task pairs fail in both arms and 83% have zero gain.Across four configurations, 6% pass in both arms, 13% are rescued by the skill, and 4% are broken by it.Source: §5.3 · p. 8

Curating Web collections

• Web archives, package registries, and static index pruning curate entries by their own state or retrieval statistics, unlike skills whose value depends on a changing model’s competence.The cited examples include page decay and link rot, package deprecation, and index pruning.Source: §6 · p. 8

Retirement criteria and their guards

• Prior retirement rules often use gain, aided performance, or learned policies, while SLIM uses a level only diagnostically rather than as its retirement decision [28].The closest cited curation method reports 88% directional stability [34], but the reviewed rules had not been audited for the composition of what they remove.Source: §6 · p. 8

Fail-both tasks and the case for the difference

• Earlier work counts tasks that fail with and without a skill [30], but does not tie them to a removal rule; other evaluations either exclude them [7] or do not separate them from both-arm successes [13].Source: §6 · p. 8

Reliability, heterogeneity and evaluation

• Prior reliability reporting ranges from three-seed noise floors [40] and per-seed results [38] to single runs [30], while reported skill effects vary across models, harnesses, and setups [13] [16] [18] [29] [35].The paper situates its gain-score critique in measurement work and Lord’s paradox.Source: §6 · p. 8

7 Conclusion

• The gain cannot distinguish an internalized skill from one where neither the model nor the skill delivers the capability.The conclusion recommends retiring on a reliable unaided label read separately for each model served.Source: §7 · p. 8

• Agents use a single-agent tool-use loop with shell access, at most 40 turns per attempt, and no skill directory or skill text in the unaided arm.Source: Reproducibility statement · p. 9
• A single judge scores runs against task rubrics, while a text adjudicator selects capability items.Source: Reproducibility statement · p. 10
• The authors release the skills, rubrics, item scores, pipeline prompts, and audit scripts on GitHub, with the dataset and datasheet on Hugging Face.Source: Reproducibility statement · p. 10

• Language models synthesize tasks and rubric items, serve as completeness critic and difficulty gate, score runs, and classify rubric items.Source: Use of Large Language Models · p. 10
• The authors report auditing a sample of judge marks and reporting agreement between item classifications and a reference annotation.Source: Use of Large Language Models · p. 10
• The authors used a language model to draft and edit parts of the manuscript, then verified every number, claim, and citation against experimental records.Source: Use of Large Language Models · p. 10

A Dataset counts and the difficulty-gate correction

• The 1,251 released tasks have a median 234-word description and 10-item, 100-point completion rubric; a median of 8 items are retained as capability items.The adjudicator retains 69.2% of rubric items as capability items overall.Source: A Dataset counts and the… · p. 10
• Of 1,251 skills, 844 pass the validity gate and carry the analysis; correcting for dropped skills raises band retirements from 27.8% to 40.3% of the library.The correction reweights dropped skills against retained skills using outcome probabilities within bins of unaided score.Source: A Dataset counts and the… · p. 10, Table 5 · p. 10
• The correction assumes that, within an unaided-score bin, the aided arm behaves alike for dropped and retained skills.Source: A Dataset counts and the… · p. 10

B Witness models for the validity gate

• The gate admits tasks when a witness gains more than 0.1 from the skill; it admits 52.6% of tasks scoring U≥0.8, so witness checks are needed mainly for high-scoring tasks.Those high-scoring tasks represent roughly 36% of a library.Source: B Witness models for the validity gate · p. 10
• Single witnesses recover 60.7%–73.8% of tasks admitted by the three-witness union, while pairs recover 84.8%–89.9%; one witness can share the target’s knowledge and invalidate a useful task.The best and worst single witnesses admit 53.5% and 43.9% of tasks, compared with 72.4% for all three.Source: B Witness models for the validity gate · p. 10
• The skill-text predictor’s AUC falls from 0.798 on ungated tasks to 0.656 after gating, and is 0.575 on the strongest-witness subset.The strongest-witness estimate has interval [0.470, 0.673], indistinguishable from chance.Source: B Witness models for the validity gate · p. 10

C Details of the rule audit

• The audited curation rule retires when the absolute mean gain is below 0.10; it splits skills when task-level gain spread reaches 0.40, a step that never runs with one task per skill.For these single-skill tasks, the gain is the aided-minus-unaided difference on capability items, averaged across three runs per arm.Source: C Details of the rule audit · p. 10
• Across the band’s high–low boundary sweep, enrichment falls as the boundary loosens, but dataset ordering stays stable and no setting removes the effect.The audit defines base rate, enrichment, and retired share of uncovered skills for the threshold comparison.Source: C Details of the rule audit · pp. 10–11, Table 6 · p. 11
• The level rule retires when U≥τ; because uncovered skills require U<0.4, thresholds τ≥0.4 cannot retire any uncovered skills.Its precision is below one only for skills passed unaided but scoring below 0.7 with the skill supplied.Source: C Details of the rule audit · p. 11, Table 7 · p. 11
• The level rule retires 49 internalized skills kept by the restricted band, while the band retires 6 internalized skills kept by the level rule.Among the 49, the skill still lifts performance by 0.1 or more for 39 and lowers it by as much for 10; no rule retires a skill that works.Source: C Details of the rule audit · p. 11, Table 8 · p. 11
• Rules differ chiefly on uncovered skills: the level rule and the band restricted to aided scores A≥0.7 leave them alone.Table 8 counts retirements by skill state on the gated adversarial set with Qwen3.6-27B as target.Source: C Details of the rule audit · p. 11, Table 8 · p. 11

D Proofs of the propositions

• Any gain-only rule gives the same verdict on the A = U diagonal, forcing either retirement of low-level skills or retention of high-level skills.For a single-skill task, leave-one-out and Shapley values equal the aided-minus-unaided success-rate difference; with other skills, attribution is not determined by those two rates alone.Source: D Proofs of the propositions · p. 11
• When the aided arm is at ceiling, gain and unaided score order skills identically up to sign; this holds for 75.9% of skills.Within the gated adversarial library with A ≥ 0.7, unaided scores had standard deviation 0.314 versus 0.077 for aided scores.Source: D Proofs of the propositions · p. 11
• As trial counts grow, the band rule converges to its gain-threshold decision, while the level rule's risk converges to zero except at its threshold.For k ≤ 10 and δ = 0.10, the band retires only when the two arms have equal observed successes.Source: D Proofs of the propositions · p. 11
• The reliability decomposition estimates verdict error under a normal approximation and uses a fixed-budget objective to compare run allocations, not predict error rates.The approximation overstates task-to-task verdict flips at 28% versus 25% observed.Source: D Proofs of the propositions · pp. 11–12

E Computing the reliability facets

• 82% of internalized skills, 72% of skills where the skill works, and 71% of uncovered skills retained their state across run splits.An uncovered skill moved to internalized in 2 of 205 cases; single-run agreement across four classes was 0.70 with κ = 0.58.Source: E Computing the reliability facets · p. 12
• The overall reliability coefficient is 0.41 after removing the mean offset between generation batches, and 0.40 without that adjustment.At three runs, the run facet alone caps a predictor’s correlation with the label at 0.95.Source: E Computing the reliability facets · p. 12

F Why text cannot establish item validity

• Text-based adjudication can check item coherence but not validity, while pass rates cannot detect nondiagnostic items the model already passes.Items whose distinctive values appeared in the task or inputs differed in pass rate by 0.074.Source: F Why text cannot establish item… · p. 12

G SkillsBench protocol and excluded tasks

• Two SkillsBench tasks were excluded because their tests rewarded skill-specific procedures over correct outputs, and keeping them changed no Table 4 estimate by more than 0.02 points.The exclusion was fixed before retirement decisions and applied to every condition.Source: G SkillsBench protocol and excluded… · p. 12