Source-linked AI summary
SkillContrast: Difference-Guided Text Selection for Agent Skill Reranking
Jiandong Ding, Honglei Ji, Ming Liu, Tao Duan
TL;DR
Similar agent skills can share query-relevant instructions yet differ in conditions of use, making those distinctions difficult for query-based text selection to preserve. SkillContrast compares retrieved skills and retains differing text with local context for a pretrained reranker. Across four benchmark settings, it produces 54–72 more clean hits than TF–IDF query selection at matched input lengths, while using fewer tokens than full skill bodies.
Problem
Similar skills may share query-relevant instructions while differing in conditions of use, so query-based selection may omit distinctions relevant to a request.
Source: §1 · p. 1Method
SkillContrast compares similar retrieved skills and retains differing instruction text with opening and preceding context for a pretrained reranker.
Source: §1 · p. 1Results
Across four settings, SkillContrast adds 54–72 clean hits over TF–IDF query selection at matched lengths, with gains over prefixes ranging from 66 to 111.
Source: §5 · p. 3Takeaways & Limitations
Candidate-relative differences complement query relevance when selecting compact reranking inputs.
Source: §6 · p. 4Evaluation uses skill text and benchmark annotations without executing skills or testing agents on users; CleanHit excludes the marked risky sibling, not all unsafe candidates.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Similar agent skills can share instructions but differ in their conditions of use. Query-based text selection may retain shared instructions and omit these distinctions. We introduce SkillContrast, a training-free selector that compares retrieved skills and retains their differing text with local context for a pretrained reranker. On 1,235 requests from SameCapRisk-Bench, it yields 54-72 more clean hits (requests that retrieve a helpful skill without its marked risky sibling) than TF-IDF query selection at identical per-candidate input lengths, across 2 retrievers and 2 reranker sizes. Length-matched component replacements identify differing text as the main contributor in the primary setting, with smaller, mixed context effects. Relative to full skill bodies, SkillContrast uses 51.1-58.8% fewer model-input tokens, with 10-18 fewer clean hits at 0.6B and matching or higher observed clean-hit counts at 4B. Candidate-relative differences thus complement query relevance in selecting compact reranking inputs.
1 Introduction
Similar skills can share query-relevant instructions while differing in conditions of use, motivating a selector that exposes those differences to a reranker.
1 Introduction
- Skills with similar purposes may differ in resources, preconditions, or input formats, so both can look relevant even when only one meets a request’s conditions.The introduction gives migration skills targeting different software versions as an example.Source: §1 · p. 1
- Query-based selection may retain the most query-relevant shared instructions while omitting a distinguishing prerequisite elsewhere.Source: §1 · p. 1
- SkillContrast compares similar retrieved skills and retains differing instruction blocks with opening and preceding context for a pretrained reranker, without further training.Skills without a close neighbor use query-based selection; MMR balances reranker scores against redundancy when selecting the final list.Source: §1 · p. 1
- On SameCapRisk-Bench, matched-length comparisons report more clean hits with candidate differences than with TF–IDF query selection.The benchmark checks whether top-three results include a helpful skill without its marked risky sibling.Source: §1 · p. 1
2 Related Work
Prior work addresses skill retrieval, text selection, and list redundancy; SkillContrast focuses on selecting distinguishing text from similar candidates for a frozen reranker.
2 Related Work
- SkillContrast focuses on which instruction text reaches the reranker, comparing similar candidates without changing the retriever or training a new scoring model.Related work distinguishes this aim from systems that retrieve reusable skills or improve query–skill matching at library scale.Source: §2 · p. 1
- Query-relevant selectors such as KeyB and Provence [2] may preserve shared instructions rather than distinguishing conditions for similar skills.Source: §2 · pp. 1–2
- The study keeps ordinary MMR fixed to isolate text selection, treating list redundancy as a separate problem from choosing reranker input.SMMR is cited as a probabilistic batch extension for redundancy handling.Source: §2 · p. 2
3 Method
SkillContrast selects text by comparing similar retrieved skill bodies, adds local context, and passes compact inputs to a frozen reranker before MMR selects the final list.
3 Method
- SkillContrast groups candidates by TF–IDF cosine similarity of at least 0.65, then aligns bodies line by line and maps differences to paragraph and heading blocks.Grouping uses no benchmark family labels, and fenced code remains intact.Source: §3 · p. 2
- For each candidate, retained text includes blocks aligned with differences against other candidates, preceding blocks, and the opening block, with overlaps counted once.The selected blocks remain in source order; if there is no difference, the selection is empty, or it exceeds a 768-token body target, the method uses the full body.Source: §3 · p. 2
- Candidates without comparison partners use query–unit TF–IDF selection, retaining the first unit as context and adding positive-scoring units with predecessors up to a 256-token body target.Short bodies, oversized first units, no fitting positive match, or detected overflow trigger full-body input.Source: §3 · p. 2
- For candidates with usable differences, selection preserves original token IDs and order while leaving the query, instruction, and scoring suffix unchanged.Only the neural reranker input is shortened; extraction and similarity still read full bodies.Source: §3 · p. 2
- A frozen reranker scores each selected input, while MMR balances relevance against similarity to already selected candidates.With no candidate selected, the redundancy penalty is zero, so MMR chooses rank 1 first and cannot remove a top-ranked risky skill.Source: §3 · p. 2
4 Experimental Setup
The experiments evaluate top-three retrieval on SameCapRisk-Bench and compare retrieval pipelines and length-matched text selectors under fixed candidate and reranker settings.
4 Experimental Setup
- The main evaluation uses 1,235 requests from 823 benchmark units across operation-contract, source-role, and caller-interface protocols; 79 development requests are excluded.The dataset release contains 890 units, including 67 development units.Source: §4 · p. 2
- The top-three metrics are Recall@3 for helpful-skill hits, HSR@3 for marked risky-sibling hits, and CleanHit@3 for helpful hits without that sibling.These metrics measure candidate exposure rather than execution failures.Source: §4 · p. 2
- A 2 × 2 design crosses SkillSight and SkillRet candidates with Qwen3-Reranker-0.6B and 4B, holding candidates, full-body similarities, reranker, and MMR fixed within each combination.The primary setting for component and parameter analyses is SkillSight with 0.6B.Source: §4 · p. 2
- Length-matched controls include prefix and query-selected text; component replacements use equally long query-relevant text, with paired bootstrap intervals stratified by protocol.All compact strategies share the no-neighbor branch and full-body fallback without padding; adjusted intervals cover four primary CleanHit comparisons and have limited tail precision.Source: §4 · pp. 2–3
- Figure 1 depicts selection retaining differences, preceding context, and opening context before the frozen reranker and MMR, with full-body similarity feeding MMR separately.The query-based no-neighbor branch is not shown.Source: Figure 1 · p. 3
- Table 1 compares retrieval pipelines using MMR(0.75) on their own top-20 candidates; Table 2 fixes candidates, reranker, and MMR while compact strategies match length.Table 2 reports K = 3 metrics and a token row with rounded per-candidate means and reductions calculated from exact totals.Source: Table 1 · p. 3, Table 2 · p. 3
5 Results and Analysis
SkillContrast improves matched-length text selection across four retriever and reranker settings, with differing text contributing most in the primary component analysis. Compact inputs reduce tokens relative to full skill bodies, while timing and case-study results qualify the benefits.
Full retrieval pipelines
- In the full-pipeline comparison, SkillContrast reaches CH@3 0.837 versus 0.813 for R3-Skill, the strongest reference.The pipelines use different retrieved candidate lists, so subsequent comparisons fix candidates to isolate text selection.Source: §5 · p. 3
RQ1: Do differences improve selection at equal length?
- SkillContrast adds 54 and 57 clean hits over TF-IDF query selection on SkillSight (0.6B/4B), and 69 and 72 on SkillRet, with gains concentrated in operation-contract requests.Gains over prefixes range from 66 to 111 clean hits.Source: §5 · pp. 3–4
RQ2: Do gains come from differences or context?
- Replacing differing-text tokens with equally long query-relevant text reduces clean hits by 78 (1,034 to 956) in SkillSight/0.6B; BM25 loses 90.Retaining differences gives a 6.32-percentage-point CH@3 advantage.Source: Table 3 · p. 4, §5 · p. 4
- Replacing the opening block or preceding context while retaining differences changes clean hits by 0 or +4; query-based context effects vary by candidate pool.Preceding-context replacements change clean hits by +4 and +2 on SkillSight, but −4 and −2 on SkillRet, at 0.6B and 4B respectively.Source: §5 · p. 4
Parameter scans
- Local threshold and MMR scans preserve matched-length gains, while clean hits fall to 896 at λ=0.85 as redundancy receives less weight.The scans vary each parameter separately while keeping text-selection targets fixed.Source: §5 · p. 4
Case study: exposing the requested target
- In the migration case study, query selection omits target versions, while SkillContrast exposes configurations and moves the matching skill to the top with lengths held fixed.The figure’s match labels are offline interpretations.Source: §5 · p. 4, Figure 2 · p. 4
RQ3: How much quality do compact inputs retain?
- Against full bodies, SkillContrast uses 58.8% fewer input tokens on SkillSight and 51.1% fewer on SkillRet at both model sizes, but loses 18 and 10 clean hits at 0.6B.At 4B it gains 10 clean hits on SkillSight and matches SkillRet’s 1,094, with 15 and 8 fewer risky exposures.Source: §5 · p. 4
Online timing
- A single-pass 0.6B timing on 1,229 valid requests found 10.4% lower online time than full bodies, saving 76.8 ms per request.Timing included input construction, encoding, 20 model forwards, and MMR; index and source-feature preparation took 25.1 s and 135.0 s separately.Source: §5 · p. 4
6 Conclusion
The conclusion presents SkillContrast as a compact reranking selector that retains candidate differences with local context. It reports matched-length gains on SameCapRisk-Bench while limiting the evaluation claim to skill text and benchmark annotations.
Conclusion
- SkillContrast retains candidate differences with local context for a pretrained reranker and adds 54–72 clean hits over TF-IDF query selection at matched lengths.Length-matched component replacements identify differing text as the main contributor in the primary setting.Source: §6 · p. 4
- Relative to full skill bodies, SkillContrast uses 51.1–58.8% fewer model-input tokens, with 10–18 fewer clean hits at 0.6B and matching or higher observed counts at 4B.The comparison is based on observed clean-hit counts.Source: §6 · p. 4
- Evaluation uses skill text and benchmark annotations, not skill execution or user-tested agents; CleanHit excludes the marked risky sibling, not all unsafe candidates.The conclusion calls for independent skill libraries with applicability labels to test transfer beyond this benchmark.Source: §6 · p. 4