SkillRep: Characterizing and Understanding the Representation Design Space of Agent Skill Retrieval

• SkillRep improves MRR@10 over the strongest same-size retriever by 7.02 points on the official pool and 8.79 on UniPool, and is best on both SkillsBench tiers.Source: §1 · p. 1
• Fine-tuning reverses the ranking of the four training-free modules, changing them from complementary to overlapping.Source: §1 · p. 2
• The three per-request gap quantities are background, unsaid vocabulary—the gold skill content a request leaves unsaid—and dilution from encoding a skill into one vector.Source: §1 · p. 2
• Query expansion supplies gold-document words, document expansion narrows request background, and lexical retrieval recovers shared words diluted by single-vector encoding.About half of document-side expansion’s gain needs nothing specific to a skill.Source: §1 · p. 2
• SkillRep is the first retriever to combine encoder fine-tuning, query- and document-side expansion, background projection, and a lexical channel.Source: §1 · p. 2

• Skill retrievers use encoder fine-tuning, LLM-written expansions, or changes to document representation, but prior methods evaluate only one or two choices on separate encoders and libraries.Source: §2 · pp. 2–3
• Embedding anisotropy concentrates similarities in a few shared directions; post-processing removes dominant directions, while contrastive learning reduces anisotropy during training.Source: §2 · p. 3
• General retrieval methods expand queries or documents and combine dense with lexical retrieval, but combined improvements often fail to add up and gains can shrink over stronger baselines.Source: §2 · p. 3

3 SkillRep

• SkillRep stacks unit-norm request and document embeddings into matrices with 1,024 dimensions.The document and request matrices have N and M rows, respectively.Source: §3 · p. 3

3.1 Encoder fine-tuning

• Contrastive fine-tuning uses 38,308 synthesized request–skill pairs, with each LLM-written request paired to its skill’s body.Training skills come from UniPool’s non-gold documents; test requests and gold skills have no near duplicates in training.Source: §3.1 · p. 3

3.2 Expansion on both sides

• LLM expansion writes text for one side in the style of the other, then moves that side’s embedding toward the generated text.Query-side expansion generates hypothetical skills; document-side expansion generates requests that the skill could answer.Source: §3.2 · p. 3

3.3 Background projection

• Background projection removes shared document directions and rescales the remaining content; projecting requests leaves their ranking unchanged.The background directions are estimated after document-side expansion; two additional generic-word directions are removed without rescaling.Source: §3.3 · pp. 3–4

3.4 Lexical channel

• The lexical channel uses BM25 term overlap independently of vectors, searching with the request and its hypothetical skills.It fuses lexical and dense scores by standardizing each against its candidate-score distribution and adding them linearly.Source: §3.4 · p. 4

Data

• SRA-Bench contributes 5,400 requests from six datasets and an official pool of 26,262 skills; SkillsBench evaluates 75 tasks against easy and hard pools.The SkillsBench pools contain 78,361 and 79,141 skills, respectively; the hard pool adds 780 adversarial distractors.Source: §4 · p. 4

Evaluation

• Table 2 reports macro-averaged SRA-Bench results; ★ marks reruns using the same requests, pools, and metrics.Other rows are reported as published.Source: Table 2 · p. 4

Data

• UniPool contains 111,247 skills from public registries, including SRA-Bench gold skills and excluding specified near-duplicates.Its final pool contains no near-exact duplicates with cosine similarity ≥0.999.Source: §4 · p. 4

Implementation

• Both expansion modules use Qwen3.6-27B at temperature 0.3, with two generated texts per request or skill, and settings are held constant across benchmarks.The dense channel indexes skill bodies, while BM25 indexes names, descriptions, and bodies; fusion uses α=0.3.Source: §4 · p. 4

Metrics

• SRA-Bench reports macro-averaged R@10, MRR@10, and Hit@1 across its six subsets.MRR@10 averages reciprocal rank of the first gold document within rank 10, using zero if none appears in the top 10.Source: §4 · p. 4

5.1 Main results

• SkillRep leads SkillReason-0.6B on the official SRA-Bench pool by +7.02 MRR@10 and +9.15 Hit@1, and SkillSight on UniPool by +8.79 and +8.52.Without query-side expansion, it still leads by 1.93 and 3.05 MRR@10.Source: §5.1 · pp. 4–5
• SkillRep is best on five of six SkillsBench metrics; SkillRet leads easy R@10 by 0.84, while SkillRep leads it by 6.36 and 10.12 MRR@10 across the two tiers.Table 3 reports 75 requests, all using 0.6B encoders.Source: Table 3 · p. 5, §5.1 · p. 5
• On SkillRet, SkillRep without query-side expansion beats every method except SkillRet’s own encoder on every metric, but that encoder trails SkillRep by 16.17 and 20.16 MRR@10 on SRA-Bench.SkillRet’s test requests are synthesized from its skills and stay closer to their wording than SRA-Bench requests do.Source: §5.1 · p. 5, Table 4 · p. 5

5.2 Ablation

• On the fine-tuned encoder, fine-tuning adds +12.94 alone and +8.79 last; the four training-free modules’ gains sum to +15.33 but together yield +13.15.Query-side expansion and the lexical channel are the largest training-free modules there, while background projection is the smallest.Source: Table 5 · p. 5, §5.2 · p. 5
• On the untuned encoder, background projection adds +5.55 alone and the four training-free modules complement one another, totaling +17.30 versus +16.56 summed individual gains.Query-side expansion adds +1.40; the encoder changes which modules contribute most.Source: §5.2 · p. 5

6.1 Background

• Background is request-invariant embedding content that consumes unit length without separating skills; background-only rankings place 89% of documents above gold on the untuned encoder and 68% on the fine-tuned encoder.The background gap measures how much more this component adds to competitors’ scores than to the gold document’s score; a positive gap favors competitors.Source: §6.1 · pp. 5–6

6.2 Unsaid vocabulary

• Unsaid vocabulary is the gold skill’s content words not expressed by the request, measured as lexical, semantic, or soft coverage, with lower coverage meaning more is left unsaid.Semantic coverage counts a gold word when a request word has cosine similarity at least 0.8; soft coverage averages each gold word’s highest cosine.Source: §6.2 · p. 6

6.3 Dilution

• Dilution measures similarity lost when the gold skill’s most request-relevant section is represented by its single document vector; it averages 0.072 on SRA-Bench.Dilution is nearly uncorrelated with document length (r = 0.07) and the three coverage counts (−0.06, −0.04, and +0.01).Source: §6.3 · p. 6

7.1 Background

• Fine-tuning narrows both background gaps, query-side expansion narrows the request background gap by 0.99, and document-side expansion leaves the document gap unchanged.Document-side expansion narrows the request background gap by 0.56; a control attributes 1.67 of its 3.10 MRR@10 gain to a component shared across documents.Source: §7.1 · pp. 6–7, Table 8 · p. 7

7.2 Unsaid vocabulary

• Only query-side expansion’s gain falls with coverage across all three counts (ρ = −0.12, −0.17, and −0.15), consistent with supplying vocabulary requests leave unsaid.Removing the gold-document words absent from the request lowers its gain by 3.12 MRR@10, while removing an IDF-matched set of other words lowers it by 0.08.Source: §7.2 · p. 7

7.3 Dilution

• The one-vector retriever’s RR falls from 77.3 on the least diluted fifth to 56.1 on the most diluted; section-level retrieval and BM25 do not show that decline.Adding unneeded sections costs the one-vector retriever up to 9.68 MRR@10, while section-level retrieval does not change; only the lexical channel’s gain rises with dilution loss (ρ = +0.10).Source: §7.3 · pp. 7–8, Table 9 · p. 8

7.4 Why the modules overlap

• Fine-tuning changes how modules interact: training-free modules complement one another on the untuned encoder but overlap on the fine-tuned one.Modules narrowing a part of the gap already narrowed by another tend to add less; modules with a distinct part to address retain more of their gain.Source: §7.4 · p. 8
• Fine-tuning makes query-side expansion narrow the request background gap, creating overlap with document-side expansion and background projection.On the untuned encoder, query-side expansion does not narrow that gap; after both backgrounds are projected out, its gain rises from +1.40 to +2.41 MRR@10.Source: §7.4 · p. 8

8 Conclusion

• SkillRep combines five representation choices and sets a new state of the art on SRA-Bench and SkillsBench with a 0.6B encoder.The paper concludes that modules narrowing the same part of the request–skill gap overlap, while modules narrowing different parts complement each other.Source: §8 · pp. 8–9

A Encoder fine-tuning

• Training skills overlap with UniPool, but none of SRA-Bench’s gold skills are in the training set.The authors report that SkillRep’s lead over SkillSight is about as large on the official pool as on UniPool: 8.44 against 8.79 MRR@10.Source: A Encoder fine-tuning · p. 10

B Score fusion against rank fusion

• Score fusion adds +4.25 MRR@10 when the lexical channel is added last, compared with +1.34 under rank fusion.Rank fusion can overturn a dense lead of any size; score fusion makes the lexical evidence needed to overturn the ranking grow with the dense margin.Source: B Score fusion against rank fusion · p. 10, Table 11 · p. 10, Table 12 · p. 11
• Score fusion is worth +3.05 and +2.90 MRR@10 on the two SRA-Bench pools, while R@10 barely moves (+0.35, +0.07) and Hit@1 rises by 4.70 and 4.79.The reported pattern is a change in which document ranks first more than a change in which documents are retrieved.Source: B Score fusion against rank fusion · pp. 10–11, Table 12 · p. 11

C The expansion weights and their interactions

• Choosing each expansion weight independently and combining the winners, (0.3, 0.3), scores 0.52 MRR@10 below the jointly chosen (0.3, 0.2).As the query-side weight grows, the best document-side weight moves left and its gain shrinks from 1.67 to 0.76.Source: C The expansion weights and their… · p. 11, Table 13 · p. 11
• On the official SRA-Bench pool, (0.3, 0.2) is best among the fifteen settings; on SkillsBench, it is 0.20 and 0.09 MRR@10 below the best on easy and hard tiers.The authors note that these SkillsBench differences are within the noise of a 75-request benchmark; on SkillRet, the selected setting is 1.56 below the best.Source: C The expansion weights and their… · p. 11, Table 14 · p. 11
• In a stratified split, the full-grid winner scores 75.56 MRR@10 on the held-out half, 0.60 below that half’s best setting.The paper compares this gap with the 7-to-9-point lead over the strongest baselines.Source: C The expansion weights and their… · p. 11

D Background projection: controls and dimensionality

• In the full system, two-level projection is best; single-step projection helps only through three directions, versus six on the fine-tuned encoder alone.Random directions of the same number change nothing; on the untuned encoder, each single step from two to eight directions gains +5.0 to +5.9, versus at most +2.6 on the fine-tuned encoder.Source: Table 15 · p. 11, D Background projection: controls and… · p. 11
• The background-gap analysis is stable across 1–6 top singular directions: 6.07–7.34 on the untuned encoder and 0.16–1.60 on the fine-tuned encoder.Source: D Background projection: controls and… · pp. 11–12
• After the first projection step, the second step’s directions have average squared document length 0.047, so rescaling changes MRR@10 by only −0.02 in the full system and +0.10 with projection alone.Source: D Background projection: controls and… · p. 12

E Separability of the three quantities

• The three quantities are weakly correlated (|r| ≤0.29), except that coverage counts agree (0.83 to 0.93) and background estimates share some variation (0.34).Table 7 controls for the other measures when relating a module’s gain to one quantity.Source: E Separability of the three quantities · p. 12, Table 16 · p. 12

F The three quantities beyond skills

• Background-only ranking puts 89% of the skill library above the gold skill, versus 28 to 41% on BEIR; projection adds 5.65 MRR@10 on skills and lowers it by 0.17 to 0.98 on BEIR.Source: Table 17 · p. 12, F The three quantities beyond skills · p. 12
• Dilution loss is 0.077 on skills, versus −0.001 to 0.026 on BEIR, alongside average skill documents of 504 words versus 191 to 232.Source: Table 17 · p. 12, F The three quantities beyond skills · p. 12
• Skill requests average 21 content words, but only 39% occur in the gold skill, versus 51% on SciFact and 43% on FiQA; gold-side coverage is higher because requests are longer.Gold-side coverage is 0.076 for skills versus 0.007 to 0.049 on BEIR; only NFCorpus has lower request-side overlap than skills.Source: F The three quantities beyond skills · p. 12, Table 17 · p. 12