Source-linked AI summary
Beyond Top-$k$ Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents
Wang Wei, Tiankai Yang, Samyadeep Basu, Hongjie Chen, Yue Zhao, Zhengzhong Tu, Xiyang Hu, Franck Dernoncourt, Ryan A. Rossi, Hoda Eldardiry
TL;DR
Large skill registries make routing difficult because independent relevance ranking can select redundant skills and omit complementary ones needed for multi-step tasks. DSR uses DPP-based reranking with a query-residual diversity kernel to balance relevance and non-redundancy. On SkillRouter, DSR improves recall and full coverage over a strong pointwise baseline, with larger gains on multi-skill queries.
Problem
Large skill registries make independent top-k relevance ranking prone to redundant selections that can omit complementary skills while consuming limited context.
Method
DSR applies DPP-based subset selection after retrieval and quality scoring, using a query-residual diversity kernel to distinguish redundant overlap from shared query relevance.
Results
DSR improves recall and full coverage over a strong pointwise SkillRouter baseline, with larger gains on multi-skill queries and at larger cutoffs.
Takeaways & Limitations
Effective skill routing should account for both individual relevance and redundancy or complementarity among the selected skills.
Takeaways & Limitations
Because DSR operates after retrieval, it cannot recover required skills absent from the initial candidate set, and evaluation metrics do not directly measure downstream execution success.
Abstract
from arXiv · showhide
Large language model (LLM) agents increasingly rely on external skills, but routing user requests over large skill registries is difficult because many skills are functionally redundant while complex tasks often require complementary skill sets. Existing skill routers typically rank candidates independently by query relevance, which can waste context budget on redundant skills. We propose Diverse Skill Routing (DSR), a diversity-aware reranking framework that uses a Determinantal Point Process to balance relevance and non-redundancy. DSR introduces a query-residual diversity kernel that penalizes redundant skill overlap while reducing penalties caused only by shared query relevance. On the SkillRouter benchmark, DSR improves recall and full coverage over a strong pointwise reranking baseline, with larger gains on multi-skill queries. These results suggest that skill routing should be treated not only as relevance ranking, but also as complementary set selection.
1 Introduction
Large skill registries make independent relevance ranking insufficient for compositional tasks because redundant candidates can displace complementary skills. DSR addresses this by selecting skill sets that balance relevance with non-redundancy, improving retrieval outcomes especially for multi-skill queries.
- Motivation: Large skill libraries create a routing bottleneck because limited context windows make exposing all skills infeasible and irrelevant skills may distract execution.SkillRouter evaluates selection over approximately 80K skills, while SkillsBench shows that skill benefits depend on task and skill quality.
- Motivation: Pointwise top-k routing can return redundant skills for compositional requests, omitting complementary capabilities such as parsing, extraction, transformation, and visualization.Independent candidate scoring may waste context budget and weaken the agent’s ability to complete multi-step tasks.
- Method: DSR treats routing as complementary set selection and uses DPP-based selection to balance query-dependent relevance with inter-skill non-redundancy.The framework first retrieves candidates, scores their quality, and then selects a skill set before ordering the selected skills by quality.
- Method: The query-residual diversity kernel removes each skill representation’s query-aligned component before measuring inter-skill redundancy.This reduces penalties for skills jointly relevant to the query while still discouraging near-duplicate candidates.
- Results: DSR improves recall and full coverage over a strong pointwise SkillRouter baseline, with larger gains on multi-skill queries and at larger cutoffs.Ablations identify the query-residual kernel as critical because replacing it with standard inter-skill similarity substantially reduces multi-skill full coverage.
2 Related Work
Related work develops tool- and skill-augmented agents, large-scale skill routing, model routing, and diversity-aware subset selection. These strands provide the context for treating skill routing as a set-selection problem rather than only a relevance-ranking problem.
- Tool use and skill-augmented agents: Tool-use research studies how LLMs call external APIs, control expert models, decompose requests, execute subtasks, and aggregate results.The cited systems include Toolformer and HuggingGPT.
- Skill routing and skill evaluation: Skill-routing research treats skills as reusable resources and examines selection from large registries, including the routing signals available in full skill implementations.SkillsBench separately evaluates whether curated skills improve downstream agent performance and reports dependence on task and skill quality.
- LLM routing: LLM routing selects models according to query-dependent requirements while accounting for heterogeneous capabilities or inference costs.Examples include FrugalGPT’s cascaded routing, EmbedLLM’s model representations, and RouterDC’s routing approach.
- Diversity-aware subset selection: Diversity-aware retrieval and recommendation methods balance relevance with novelty or multiple query aspects, while DPPs model subset selection using item quality and diversity.Maximal Marginal Relevance and DPPs are cited as precedents for diversification.
3 Method
DSR retrieves candidate skills, scores their query relevance, and uses DPP-based greedy selection to construct a compact shortlist balancing quality and non-redundancy. Its query-residual kernel reduces penalties for similarity caused only by shared query relevance while favoring complementary skills.
- Candidate Skill Retrieval: DSR retrieves a small candidate set from the full skill registry before applying diversity-aware reranking within that set.This restricts expensive selection to C_x, making reranking tractable for large registries.
- Candidate Skill Retrieval: An encoder retriever independently embeds the query and skills for efficient candidate retrieval, while a pointwise reranker supplies more expressive query-skill relevance scores.In the main experiments, reranker scores provide the quality signal for DSR.
- Quality-Aware DPP Selection: DSR assigns each candidate a non-negative quality score and constructs a DPP kernel whose determinant favors high-quality subsets with non-redundant skill representations.The quality terms determine how strongly individual skills are favored before diversity is considered.
- Query-Residual Diversity Kernel: The query-residual kernel removes each skill's query-aligned component before measuring inter-skill similarity, reducing penalties caused only by shared query relevance.The residual is blended with the original embedding and normalized, with λ controlling residual-projection strength.
- Query-Residual Diversity Kernel: Raw inter-skill similarity can suppress complementary skills because skills needed for the same multi-step task may share query-aligned embedding directions.The paper illustrates this with parsing, column cleaning, and visualization skills for data analysis.
- Greedy MAP Selection and Ranking: Because exact DPP MAP inference is expensive, DSR greedily adds the candidate with the largest marginal gain until k skills are selected, then sorts them by quality score.Incremental Cholesky updates compute log-determinant gains without recomputing determinants from scratch.
4 Experiments
The experiments evaluate DSR against pointwise and alternative ranking strategies on the SkillRouter benchmark, showing its strongest benefits when queries require multiple complementary skills. Ablations identify query-residual diversity and learned relevance quality as important to coverage gains.
- Experimental Setup: The benchmark contains 75 expert-verified queries over approximately 80K candidate skills, spanning single-skill and multi-skill tasks across EASY and HARD robustness tiers.The EASY tier has 78,361 candidates, while HARD has 79,141 candidates including 780 generated distractors.
- Experimental Setup: DSR keeps SkillRouter's retriever and relevance model fixed, changing only final selection to query-residual DPP selection over retrieved candidates.This isolates the effect of diversity-aware selection from candidate generation and relevance scoring.
- Main Results: At cutoff 20, DSR raises all-query Recall from 0.754 to 0.768 and Full Coverage from 0.560 to 0.573 over SkillRouter.DSR remains close to SkillRouter on MRR, so the coverage gains do not reflect a large loss in early precision.
- Main Results: At cutoff 50, DSR improves all-query Recall from 0.754 to 0.808 and Full Coverage from 0.560 to 0.633.The larger gains at longer cutoffs are consistent with reducing redundant skills in the selected shortlist.
- Main Results: For multi-skill queries, DSR improves Recall@20 from 0.704 to 0.739 and Full Coverage@20 from 0.458 to 0.492, with larger gains at cutoff 50.At cutoff 50, Recall increases from 0.704 to 0.773 and Full Coverage from 0.458 to 0.551.
- Ablation Study: The query-residual kernel produces the largest ablation effect, improving multi-skill Full Coverage@10 from 0.254 to 0.441 over the standard cosine kernel.It penalizes overlap beyond the query direction rather than all inter-skill similarity, preserving complementary skills sharing task relevance.
- Ablation Study: With the query-residual kernel, reranker quality improves Recall@10 from 0.618 to 0.711 and multi-skill Full Coverage@10 from 0.237 to 0.441 over embedding quality.The strongest coverage-oriented results combine learned relevance with query-residual diversity.
- Reranking Strategies: On single-skill queries, Top-k retrieval, SkillRouter, and DSR achieve the same Recall@10 and Full Coverage@10, while multi-skill queries show progressive gains from reranking and diversity-aware selection.For multi-skill queries, DSR reaches Recall@10 of 0.668 and Full Coverage@10 of 0.432, compared with SkillRouter values of 0.659 and 0.424.
5 Conclusion
The paper frames skill routing over large, redundant registries as a joint relevance and complementary-set selection problem. DSR uses DPP-based selection and a query-residual kernel, improving recall and full coverage, especially for multi-skill queries.
- Conclusion: Existing pointwise routers can waste context on redundant skills and miss complementary skills required for multi-step tasks.The problem arises because candidates are ranked independently by query relevance.
- Conclusion: DSR uses DPP-based subset selection to balance skill relevance and non-redundancy.Its query-residual kernel penalizes redundant overlap while reducing penalties caused by shared query relevance.
- Conclusion: Experiments on the SkillRouter benchmark show that DSR improves recall and full coverage over a strong pointwise baseline, with larger gains on multi-skill queries.The query-residual kernel is critical, and benefits are clearest when multiple required skills must be recovered within a limited shortlist.
- Conclusion: The findings indicate that selection remains important even when the retriever and reranker are fixed, because routing must account for redundancy and complementarity among selected skills.This conclusion is stated within the setting of growing skill registries and the evaluated benchmark.
Limitations
The evaluation is limited by its small benchmark, reliance on retrieved candidates, indirect retrieval metrics, and added selection cost. These boundaries motivate broader benchmarks, end-to-end evaluation, joint retrieval and selection, and more efficient approximations.
- Benchmark scope: The benchmark contains only 75 expert-verified queries, limiting coverage of domains, workflows, and skill types.Future evaluation should use broader benchmarks with more domains, more complex workflows, and different skill types.
- Candidate retrieval: DSR cannot recover required skills absent from the initial candidate pool, so performance still depends on retrieval and relevance modeling.The reported results isolate diversity-aware selection given a retrieved candidate set.
- Evaluation metrics: Recall and Full Coverage measure skill retrieval rather than downstream execution success, which may still fail because of tool use, planning, or instruction conflicts.Future evaluations should connect skill-set coverage with end-to-end task completion.
- Computational cost: The post-reranking DPP step adds cost that may matter for latency-sensitive deployments or much larger candidate sets.More efficient approximations and adaptive candidate sizes could reduce this overhead.
A MRR Diagnostic
MRR serves as an early-precision diagnostic, while DSR’s main advantage concerns recovering complete skill sets. DSR remains close to SkillRouter on MRR while improving coverage metrics.
- Metric interpretation: MRR measures the rank of the first retrieved correct skill and is mainly an early-precision diagnostic.It is less aligned with multi-skill routing, which requires recovering the full required skill set.
- All queries: 0.784 MRR@10 on all queries for DSR compares with 0.788 for SkillRouter.The small difference accompanies DSR’s reported improvements in Recall and Full Coverage.
- Multi-skill queries: 0.788 MRR@10 on multi-skill queries for DSR compares with 0.792 for SkillRouter.These small differences suggest improved skill-set coverage without substantially degrading the first relevant skill position.
B LLM Usage
The manuscript used ChatGPT for language editing and drafting support, while experimental results and final scientific decisions were not generated by LLMs. Table 4 uses fixed retrieval and relevance models while reporting MRR alongside coverage metrics.
- LLM assistance: ChatGPT was used during manuscript preparation for language editing and drafting support.
- Scientific decisions: LLMs were not used to generate experimental results or make final scientific decisions.
- Evaluation setup: Table 4 reports both methods with SR-Emb-0.6B retrieval and SR-Rank-0.6B relevance scores.The table includes MRR as an early-precision diagnostic alongside coverage metrics.