Source-linked AI summary
Beyond Task-Only Matching: Personalized Skill Routing with Counterfactual Evaluation
Tianle Wang, Yanghe Zou, Xiang Liu, Ziyao Huang, Chenchen Fu, Weiwei Wu
TL;DR
Task-only skill routing can miss user constraints when the same request requires different executable procedures for different profiles. The paper introduces profile-counterfactual evaluation and profile-aware training within SkillFeed, which achieves 0.751 overall Hit@1 and its largest gains on profile-sensitive instances.
Problem
Existing task-only routing and evaluation do not adequately distinguish skills that are equally relevant to a task but differ in compatibility with user constraints.
Method
The paper builds profile-counterfactual benchmark instances and paired supervision, then uses SkillFeed’s progressive retrieval and profile-conditioned reranking to discriminate among candidate skills.
Results
0.751 overall Hit@1 is achieved by SkillFeed, while profile conditioning contributes a 35.1-point gain on profile-counterfactual samples.
Takeaways & Limitations
User profile information is an essential routing signal when profile constraints change which skill is suitable for a fixed task.
Takeaways & Limitations
Performance remains lower on specialized technical and scientific queries, indicating that fine-grained domain and profile constraints remain challenging.
Abstract
from arXiv · showhide
The rapid expansion of reusable skill repositories makes skill routing a critical capability for large language model (LLM) agents. Existing methods treat routing as task-only semantic matching. However, when users with incompatible constraints issue an identical request, this assumption conflates task relevance with skill suitability: a task-only router can select a semantically plausible skill that is unsuitable for the requesting user. To expose this failure mode, we formulate \textit{personalized skill routing} as profile-conditioned retrieval, in which relevance depends jointly on the task and the user profile. We first introduce a profile-counterfactual benchmark, in which the task is held fixed while changes in the user profile induce changes in the reference skill. We further construct paired counterfactual supervision and propose SkillFeed, a progressive retrieve-and-rerank framework that first establishes task--skill alignment and then learns profile-conditioned discrimination. By retrieving body-level evidence and reranking semantically similar but profile-conflicting candidates, SkillFeed identifies skills that satisfy both task requirements and user constraints. On SkillFeed-Bench, SkillFeed attains 75.1\% top-1 retrieval accuracy, a 23.1-point improvement over the corresponding pretrained routing baseline. Adding profile conditioning yields a 35.1-point gain on queries where user profile changes the reference skill. This contrast shows that user profiles are most consequential precisely when they change skill suitability. Our website is publicly available at http://www.aiskillfeed.com .
1 Introduction
Skill routing must account for both the task and user profile because identical requests can require different skills under incompatible constraints. The paper introduces counterfactual evaluation and SkillFeed to learn and test this profile-conditioned routing.
- Motivation: Task-only routing can select a semantically relevant skill that conflicts with a user’s budget, platform, payment, or expertise constraints.The same Shanghai trip request may require a budget planner for a student but a premium planner for a business traveler.
- Motivation: Conventional task-level evaluation cannot determine whether a router responds appropriately when user profiles change while the task remains fixed.Existing benchmarks and retrieve-and-rerank systems primarily evaluate task–skill relevance.
- Problem formulation: The paper formulates personalized skill routing as retrieval jointly conditioned on task and task-relevant user profile.This formulation treats profile-induced changes in the suitable skill as a distinct routing problem.
- Benchmark and supervision: Profile-counterfactual instances hold the task fixed while varying the user profile and reference skill, enabling evaluation and supervision for profile-conditioned routing.The construction targets both response to profile shifts and discrimination among profile-conflicting alternatives.
- Method: SkillFeed-Bench evaluates profile-conditioned routing over a 228K-skill repository, while SkillFeed separates broad candidate recall from profile-specific reranking.The framework first establishes task–skill alignment, then resolves suitability among semantically similar skills.
- Results: 75.1% Hit@1 is achieved by SkillFeed, outperforming the pretrained baseline by 23.1 points on SkillFeed-Bench.The reported contributions also include an 18.6-point gain on profile-sensitive queries and a 35.1-point gain on profile-counterfactual samples.
2 Related Work
Prior work establishes task-level skill retrieval and user-conditioned ranking, but does not explicitly model profile-induced changes in the suitable executable skill under a fixed task. SkillFeed addresses this gap with profile-aware supervision and evaluation.
- Personalized Retrieval and Recommendation: Personalized ranking and search condition relevance on user information, but generally target items, documents, or content preferences rather than executable procedures.Skill routing instead selects a procedure for a concrete task under task-relevant constraints.
- Personalized Retrieval and Recommendation: Skill routing profiles contain task-relevant constraints such as language, platform access, budget, and expertise that can make one workflow suitable and another inapplicable.The paper therefore focuses on profile-induced changes in the suitable skill rather than general personalization.
- Skill Benchmarks: Existing skill benchmarks provide task-level supervision but do not test whether a router revises its selected skill when an unchanged task is paired with a different user profile.This leaves profile-induced target changes outside conventional evaluation.
- Skill Retrieval and Routing: Skill-routing systems use sparse or dense retrieval and retrieve-and-rerank pipelines to generate candidates and perform fine-grained discrimination.These approaches motivate scalable candidate generation and reranking for large skill repositories.
3 Problem Formulation and Benchmark
The paper defines routing as joint task–profile retrieval and builds SkillFeed-Bench to test profile-induced changes in reference skills. The benchmark combines large-scale skill data, structured profiles, controlled counterfactuals, and quality-controlled annotations.
- 3.1 Personalized Skill Routing Formulation: Personalized routing selects the candidate skill that best satisfies a natural-language task and structured user profile jointly.Each skill includes a name, description, and procedural body specifying its workflow and applicability conditions.
- 3.1 Personalized Skill Routing Formulation: The profile includes task-relevant attributes such as region, language, platform access, payment method, expertise, and other constraints.These attributes determine compatibility beyond semantic task relevance.
- 3.1 Personalized Skill Routing Formulation: Top-K retrieval ranks repository candidates by joint task–profile compatibility and returns the ordered highest-scoring skills.Candidate recall succeeds when the annotated reference skill appears in this returned set.
- 3.2 SkillFeed-Benchmark Construction: The repository contains 228,432 deduplicated and format-normalized skills aggregated from public sources.Each repository entry contains a name, description, and procedural body.
- 3.2 SkillFeed-Benchmark Construction: The benchmark collects 272 real-agent tasks and pairs them with structured profiles covering platforms, payment methods, language, region, and additional constraints.Profiles are designed to retain attributes that can affect skill suitability while remaining concise.
- 3.2 SkillFeed-Benchmark Construction: Reference targets are selected by retrieving semantically relevant candidates, then using LLM assessment and pairwise comparison across relevance, compatibility, capability, constraints, specificity, and overfit risk.Each annotated instance is a profile–query–skill triplet.
- 3.2 SkillFeed-Benchmark Construction: 329 evaluation queries comprise 162 profile-sensitive cases, including 77 counterfactual augmentations, and 167 profile-insensitive cases over 228K candidates.Quality controls remove low-confidence or high-overlap samples and add consistency checks plus expert review.
4 Method: SkillFeed
SkillFeed routes skills using both task relevance and user-profile constraints. Its progressive design combines broad retrieval, profile-aware body evidence, and reranking to distinguish semantically similar but incompatible skills.
- Counterfactual Supervision: Counterfactual supervision pairs the same task with different profiles and required rankings, using semantically similar profile-conflicting hard negatives.These paired instances directly supervise ranking reversals when the routing-relevant profile changes.
- Two-Phase Retriever Training: SkillFeed separates task-level candidate coverage from profile-conditioned discrimination during routing.The retriever operates over the full repository, while the reranker focuses on task-relevant alternatives and their profile-specific differences.
- Profile-Aware Skill-Body Retrieval: Profile-aware skill-body retrieval extracts localized conditions from structured skill sections instead of relying only on concise skill descriptions.Relevant conditions may concern inputs, platforms, formats, dependencies, and exceptions distributed across procedural sections.
- Evidence Aggregation: Skill-body evidence is aggregated into a skill-level compatibility score using a maximum term, a top-M mean, and a logarithmic chunk-count term.The maximum preserves decisive conditions, the mean captures repeated evidence, and the logarithmic term mildly rewards multiple supporting chunks without favoring longer files mechanically.
- Integrating Evidence into Routing: Routing combines profile-dense, task-dense, and BM25 rankings with weighted reciprocal rank fusion before final reranking.BM25 preserves exact lexical matches such as tool names, platforms, and file formats, while the reranker receives the task, profile, and complete skill information.
5 Experiment
Experiments evaluate profile-conditioned routing against task-only and general-purpose baselines on SkillFeed-Bench, including profile-sensitive and counterfactual queries. Results show that profile-aware supervision, reranking, multi-source recall, and profile-aware retrieval components contribute complementary gains, while specialized queries remain challenging.
- Experimental Setup: SkillFeed-Bench evaluates routing on 329 samples, including 162 profile-sensitive queries and 77 profile-counterfactual samples.The evaluation reports Hit@K, MRR@20, nDCG@20, and counterfactual-switch accuracy.
- Necessity of Personalized Skill Routing: 0.143 and 0.052 Hit@1 on profile-counterfactual queries are achieved by the pretrained Qwen baseline and task-only Phase 1 retriever, versus 0.611 and 0.623 on profile-insensitive queries.The contrast indicates that task semantics alone are insufficient when user profiles change the suitable skill.
- Profile-Aware Embedding and Reranking: 0.617 Hit@1 is achieved by SkillFeed Emb Phase 2, compared with 0.483 for task-only Phase 1 and 0.471 for unfine-tuned Qwen3-Embedding-8B.The adapted 0.6B model also outperforms the pretrained 8B model, while BM25, E5-large-v2, and BGE-large-en-v1.5 reach 0.410, 0.322, and 0.386.
- Profile-Aware Embedding and Reranking: 0.733 Hit@1 is reached by the SkillFeed reranker with the Phase 2 retriever, outperforming Qwen3-Reranker-8B and SkillRouter pipelines by 10.1 and 11.9 percentage points.This comparison uses the same fixed candidate lists for reranker comparisons, isolating candidate discrimination from recall coverage.
- Multi-Source Recall and Profile Chunk Retrieval: 0.751 Hit@1 and 0.803 MRR@20 are achieved by the full pipeline combining profile-aware dense retrieval, BM25, and SkillFeed reranking.The Profile chunk-based variant lowers Hit@1 to 0.739 but improves Hit@10 to 0.918 and nDCG@20 to 0.838.
- Performance Across User Profile Groups: SkillFeed reaches 0.733 overall Hit@1 but performs lower on specialized technical and scientific queries, while generalizing across heterogeneous user profiles.Fine-grained domain and user profile constraints remain challenging.
- Statistical Significance: 0.751 Hit@1 improves on the pretrained Qwen baseline by 0.231 with p < 0.0001 and a 95% confidence interval of [0.706, 0.794].The SkillFeed reranker gain over embedding-only retrieval is +0.116, while including the profile in reranking adds +0.089.
6 Conclusion
SkillFeed reframes skill routing as profile-conditioned relevance, showing that task semantics alone can fail when user constraints change skill suitability. Personalized retrieval improves performance, with SkillFeed achieving the strongest overall result.
- 0.426 Hit@1 is achieved by task-only retrieval on profile-sensitive queries, compared with 0.630 for personalized retrievers.
- 0.751 Hit@1 overall is achieved by SkillFeed, a two-stage framework combining multi-source candidate recall with profile-conditioned reranking.
- The largest gains are concentrated on profile-sensitive instances, where user profiles change the suitable skill.
- User profile information is established as an essential signal for personalized skill routing in agentic systems.
A.1 Test Sample Format
SkillFeed-Bench represents each evaluation sample with a task, structured user constraints, a reference skill, and hard negatives. The benchmark uses counterfactual annotations and spans diverse skill categories.
- Each test sample records a natural-language task, structured user-context constraints, and one ground-truth skill from a 228K candidate pool.
- 329 test samples include profile-sensitive queries in which profile changes can alter the preferred reference skill.
- Two annotators reviewed instances for task capability, profile-driven preference changes, and meaningful counterfactual shifts preserving the original task.
- The benchmark contains 4 semantically similar but profile-incompatible hard negatives per sample and limits task–ground-truth body token overlap to ≤0.50.
- 10 skill categories provide coverage across diverse application domains, with Development and Business as the largest categories.
B Training and Pipeline
SkillFeed progressively trains and deploys a two-stage router: it first preserves task–skill alignment and candidate coverage, then resolves profile-specific suitability using body-level evidence and reranking.
- Training and supervision: Profile-based and counterfactual supervision teaches the system to distinguish skills that are task-relevant but incompatible with user profiles.
- Training and supervision: Phase 1 learns task-only routing alignment, while Phase 2 appends profile constraints and learns profile-conditioned discrimination.
- Training and supervision: The reranker scores a positive skill against curated hard negatives and uses listwise cross-entropy to train the reference skill to outrank alternatives.
- Inference pipeline: SkillFeed separates broad candidate recall from fine-grained reranking so repository-scale coverage and profile-specific suitability are handled separately.
- Inference pipeline: Recall combines task-oriented dense retrieval, BM25, and persona-conditioned dense retrieval over structured SKILL.md body chunks.
- Inference pipeline: Chunk-level indexing exposes local applicability conditions, constraints, dependencies, and examples that concise skill descriptions may omit.
- Inference pipeline: Chunk-to-skill aggregation favors consistent body-level evidence using the strongest match, a top-M mean, and a logarithmic chunk-count term.
C Additional Experimental Details
The training and reranking records serialize tasks, profiles, skills, and negatives in structured formats. Two training phases use distinct query contents and dataset sizes for progressive routing adaptation.
- Data schema: Each source instance contains a task, an optional user profile, one reference skill, and hard-negative skills.
- Data schema: Skill records serialize a name, description, and SKILL.md body excerpt, exposing procedural details and applicability constraints.
- Training records: Phase 1 removes the user profile and formats queries with the task description and core requirements.
- Training records: 2,712 training instances and 356 validation instances are used in Phase 1.
- Training records: Phase 2 appends country, language, platform, payment, and other user-context constraints to the task query.
- Training records: 7,459 training instances and 608 validation instances comprise Phase 2, including full-body and profile-counterfactual supervision.
- Reranking records: Reranker records pair a task–profile query with one positive and multiple negative skills, using yes/no supervision.
- Reranking records: Embedding records use the user role for documents, whereas generative reranker records use the assistant role, with serialization kept consistent.
C.3 Implementation Details
SkillFeed uses separately initialized dense retrieval and reranking models, trains retrieval in task-centric and profile-conditioned phases, and evaluates configurations across profile groups with uncertainty and significance analyses.
- SkillFeed initializes its dense retriever from Qwen3-Embedding-0.6B and its reranker from Qwen3-Reranker-0.6B.
- The retriever is trained first on 2,712 task-centric instances and then on 7,459 user profile-conditioned instances.
- Training uses 2×A100 GPUs, AdamW, a 1 × 10−5 learning rate, effective batch size 16, maximum sequence length 2048, and three epochs.
- SkillFeed reports per-profile-group performance across country/region, language, domain, and persona sensitivity using all 329 test samples.
- The evaluation includes bootstrap 95% confidence intervals and permutation tests, with significance thresholds of p < 0.05, p < 0.01, and p < 0.001.