PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillQR: Agentic Skill Discovery for Reason-Intensive Query Rewriting

Kenan Jiang, Meixiu Long, Zihao Zhang, Ran Xu, Eugene Agichtein, Fei Liu

arXiv:2610.33768v1cs.IR

TL;DR

Reason-intensive queries can obscure the information needed for retrieval, while existing rewriting approaches often rely on costly models, substantial training, or iterative retrieval and rewriting. SkillQR discovers corpus-grounded strategies and routes queries to them, achieving performance comparable to strong learned baselines with a favorable efficiency–effectiveness trade-off.

  • Problem

    Reason-intensive queries can contain irrelevant details or omit structure needed to connect them to relevant evidence, while rewriting approaches often require costly models, substantial curated data, or iterative retrieval.

    Source: §1 · p. 1

  • Method

    SkillQR discovers compact corpus-grounded rewriting skills without relevance labels or fine-tuning, then routes each query to an appropriate skill at inference time.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    SkillQR achieves performance comparable to strong learned baselines with a favorable efficiency–effectiveness trade-off, and is approximately 45 seconds faster per query than ThinkQE.

    Source: §4.3 · p. 5, §6 · p. 8

  • Takeaways & Limitations

    The findings motivate corpus-aware strategy design as a rewriting signal beyond stronger models or larger training sets.

    Source: §6 · p. 9

    SkillQR relies on LLM proposer, judge, and rewriter agents whose reasoning may be affected by pretraining biases, domain-knowledge limits, and hallucinations from long context.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Dense retrievers provide grounded evidence that mitigates hallucinations in outputs from Large Language Models (LLMs). Real-world queries are often verbose, ambiguously framed, or require multi-step reasoning, introducing irrelevant details that weaken semantic matching for dense retrievers. Query rewriting is a widely adopted pre-retrieval technique that leverages LLMs to reformulate queries and bridge the semantic gap between such queries and the retrieval target. Existing rewriting approaches either rely on expensive proprietary models, substantial training data, or iterative retrieval and rewriting, while largely applying a uniform rewriting strategy regardless of the retrieval corpus. We observe that effective query rewriting instructions are not universal: the optimal strategy depends on the retrieval target, rather than the model alone. We introduce SkillQR, a query rewriting framework that automatically discovers reasoning strategy sets from the retrieval target and dynamically routes a query to the appropriate rewriting strategy at inference time. Without relying on training data, SkillQR achieves competitive performance compared to fine-tuned rewriting models on the BRIGHT and R2MED benchmarks, while also providing lower inference latency than iterative rewriting approaches.

1 Introduction

Real-world and reason-intensive queries can obscure the information needed for retrieval, and existing rewriting approaches often have efficiency and adaptability limits. SkillQR discovers corpus-grounded strategies and routes queries to them, with competitive benchmark performance reported using fewer resources.

1 Introduction

  • Reason-intensive queries may contain irrelevant details or omit critical structure, leaving even strong dense retrievers unable to match the surface query to relevant evidence.Real-world queries can also be indirect and ambiguously framed.Source: §1 · p. 1
  • Existing rewriting approaches can require costly proprietary models, substantial curated training data, or iterative retrieval and rewriting that increases inference cost and latency.These demands can make methods difficult to transfer across corpora or deploy in privacy-sensitive and resource-constrained settings.Source: §1 · p. 1
  • The effective rewriting strategy depends on the retrieval target, yet prior approaches largely keep that strategy fixed across retrieval settings.The paper contrasts searches targeting theoretical computer science papers with those targeting Python Stack Overflow answers.Source: §1 · pp. 1–2
  • SkillQR discovers corpus-grounded rewriting strategies and dynamically routes each query to an appropriate strategy without relevance labels or model fine-tuning.The framework extracts a compact strategy set from the target corpus for inference-time use.Source: §1 · p. 2
  • On BRIGHT and R2MED, SkillQR demonstrates performance comparable to previous rewriting methods while using fewer resources.Source: §1 · p. 2

2 Related Work

Related work covers approaches to reason-intensive retrieval and the trade-offs of query rewriting, then connects reusable agent skills to SkillQR's corpus-guided adaptation.

2.1 Query Rewriting for Reason-Intensive IR

  • BRIGHT evaluates reasoning-intensive retrieval across twelve subjects, while R2MED focuses on medical-domain content.Source: §2.1 · p. 2
  • Reasoning-aware retrievers train to bridge reasoning gaps, while rerankers refine result ordering; query rewriting is presented as complementary to both.Representative rewriting methods use reinforcement learning or specialized fine-tuning.Source: §2.1 · p. 2
  • Existing rewriting methods trade off deployment cost, training resources, transferability, and latency across proprietary models, fine-tuned systems, and iterative pseudo-relevance feedback.Iterative retrieval-and-rewrite can increase prompt length and online latency.Source: §2.1 · p. 2

2.2 Skills in LLM Agents

  • Prior agent-skill work treats skills as reusable instructions and procedures; SkillQR combines offline corpus-guided discovery with query-dependent routing, without model fine-tuning.Related work describes automatic skill discovery through environment interaction and feedback.Source: §2.2 · pp. 2–3

3 SkillQR

SkillQR addresses retrieval queries whose surface form may not match the target by adapting rewriting strategies to the corpus. It discovers reusable strategies offline and routes queries to a selected strategy at inference time.

Rewrite Instruction Template

  • Oracle strategies raise average NDCG@10 from 30.4 to 33.0 over generic rewriting, a +2.6-point gain with no change to the underlying model.The comparison uses human-curated, subject-specific strategies for Coding and Theorem-based tasks and the generic strategy for StackExchange questions.Source: Rewrite Instruction Template · p. 3, §3.1 · p. 3

3.1 Motivation: Rewriting Strategy

  • Human-curated, corpus-specific rewriting strategies outperform a generic strategy, motivating automatic recovery of this costly-to-craft signal.The oracle experiment applies the same rewriting framework to both strategies; manually crafting subject-specific instructions requires holistic corpus understanding and does not scale.Source: Rewrite Instruction Template · p. 3, §3.1 · p. 3

3.2 Problem Formulation

  • The formulation treats relevant documents as connected to a query by a latent reasoning trajectory and seeks a rewritten query that improves retrieval ranking.The trajectory captures implicit reasoning needed to bridge the query and relevant documents; retrieval uses cosine similarity and a ranking metric such as NDCG@10.Source: §3.2 · p. 3

3.3 Agentic Offline Skill Discovery

  • Offline discovery clusters document embeddings, characterizes representative documents by domain and content type, and uses those characterizations to initialize rewriting skills.The pipeline reuses embeddings from the dense retriever to avoid additional encoding cost; each skill is stored in the rewriter’s memory for selective use at inference.Source: §3.3 · pp. 3–4

Skill Initialization Prompt

  • A judge checks whether proposed skills cover sampled documents in each cluster, then recommends refining an existing skill or creating a new one for uncovered content.Refinement broadens an existing skill while preserving its domain focus; novel clusters are grouped for a new skill.Source: Skill Initialization Prompt · p. 4

Judge-Proposer Interaction Example

  • The judge-proposer loop revisits uncovered clusters until all are covered or the iteration limit is reached, yielding a final skill set that approximates reasoning trajectories to retrieval targets.The example distinguishes a climate-dynamics cluster assigned to refinement from a UI/UX-design cluster assigned to a new skill.Source: Judge-Proposer Interaction Example · pp. 4–5

3.4 Inference-Time Rewrite Generation

  • At inference, the rewriter selects a skill by matching the query’s domain, uses the original query’s top-ranked documents as pseudo-relevance feedback, and generates a skill-guided expanded query.The selected skill is instantiated as a prompt for the rewriter.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate SkillQR on BRIGHT and R2MED using two dense retrievers, comparing it with iterative and fine-tuned rewriting baselines. Results show competitive retrieval performance alongside lower inference cost, including when the target corpus shifts to medicine.

4.1 Datasets and Baselines

  • BRIGHT and R2MED evaluations report averaged and per-subject NDCG@10, comparing rewriting baselines including iterative ThinkQE and RL-fine-tuned methods.The study tests generalization with Diver-Retriever and ReasonEmbed.Source: §4.1 · p. 5

4.2 SkillQR Implementation Details

  • Offline discovery clusters each subject corpus, refines at most five skills for up to three rounds, and uses one LLM for routing and rewriting at inference.The cluster count is set to approximately the square root of corpus size; skill discovery is a one-time cost, taking about two hours for BRIGHT on one H100 GPU.Source: §4.2 · p. 5

4.3 BRIGHT Results

  • At 14B scale on BRIGHT, SkillQR matches ThinkQE across both retrievers and is approximately 45 seconds faster per query.SkillQR uses single-pass skill-guided rewriting, compared with ThinkQE’s three iterative answer-generation passes.Source: §4.3 · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • SkillQR reaches 38.3 on Reason-Embed at 7B scale, comparable to RRewriter’s 38.4, while requiring less inference time than the fine-tuned counterparts.TongSearch and RRewriter are fine-tuned on BRIGHT-curated data, and RRewriter also uses human expert skills.Source: §4.3 · pp. 5–6, Table 2 · p. 6

4.4 R2MED for Performance Transferability

  • 46.9 NDCG@10 with Diver significantly exceeds RRewriter’s 44.9 on R2MED at 7B scale, with significant improvements over both fine-tuned baselines on two retrievers.The baselines were trained on BRIGHT-curated data, while SkillQR mines skills from the deployment corpus.Source: §4.4 · p. 6, Table 4 · p. 7
  • On R2MED at 14B scale, SkillQR is statistically comparable to ThinkQE, reaching 48.9 NDCG@10 on Diver and 49.3 on Reason-Embed.The comparison reports less inference cost for SkillQR on both retrievers.Source: §4.4 · p. 6, Table 4 · p. 7
  • In a Skill quality NDCG@10 comparison on BRIGHT using Diver-Retriever and a Qwen2.5-7B-Instruct rewriter, SkillQR does not inherently include PRF and matches human expert skills on average.Source: Table 5 · p. 7

5 Discussions

SkillQR’s discovered skills outperform generic rewriting and match human-designed skills, while its LLM router leads alternative routers on skill-sensitive BRIGHT queries. A qualitative example shows corpus-guided concept unfolding yielding a stronger rewrite than a generic instruction.

5.1 Skill Effectiveness

  • Discovered skills outperform generic rewriting by 1.6 on average, match human-designed skills, and improve Rob by +1.9.SkillQR finds distinct strategies across subjects, while human-designed skills default to generic instructions for StackExchange subjects.Source: §5.1 · p. 7

5.3 Qualitative Analysis

  • For a simple-harmonic-motion query, SkillQR’s concept-unfolding rewrite names relevant concepts and reaches 43.1 NDCG@10, whereas the generic rewrite emphasizes solution articulation.The figure reports the generic rewrite at 0.0 NDCG@10 and SkillQR at 43.1.Source: Figure 2 · p. 8, §5.3 · p. 8

5.2 Router Analysis

  • 42.9 NDCG@10 for SkillQR’s LLM router versus 42.5 for Diver routing; the retrospective best-skill-per-query upper bound is 53.4.On 722 skill-sensitive BRIGHT queries, LLM routing also scores above Laya and random routing without relevance labels or router fine-tuning.Source: Figure 3 · p. 8, §5.2 · p. 8

6 Conclusion

SkillQR discovers rewriting strategies from corpus characteristics and selects among them to align queries with retrieval targets. The authors report performance comparable to strong learned baselines with a favorable efficiency–effectiveness trade-off, motivating corpus-aware strategy design.

6 Conclusion

  • SkillQR selects rewriting strategies from a skill set to align queries with corpus characteristics.The framework is training-free.Source: §6 · p. 8
  • SkillQR achieves performance comparable to strong learned baselines while maintaining a favorable efficiency–effectiveness trade-off.Source: §6 · p. 8
  • The findings motivate corpus-aware strategy design for reason-intensive retrieval.The paper identifies the retrieval corpus as a rewriting signal beyond stronger models or larger training sets.Source: §6 · p. 9

Limitations

The authors identify vulnerabilities in SkillQR’s LLM agents and a gap between the current router and a theoretical upper bound. They suggest richer routing and dynamic skill refresh as future improvements.

Limitations

  • LLM proposer, judge, and rewriter agents may be affected by pretraining biases, limited domain knowledge, and long-context hallucinations.Source: Limitations · p. 9
  • The LLM router falls short of the theoretical upper bound, leaving potential gains from better query-to-skill matching.Source: Limitations · p. 9
  • Future versions may add richer routing strategies and dynamic skill refresh for an evolving corpus.Source: Limitations · p. 9

A Experiment Details

The appendix specifies experimental model settings and points to prompt and human-skill materials used in the experiments. It also identifies the methods, backbones, and implementation sources.

A.1 Models

  • For 7B-scale methods, generation temperature is 0; for 14B-scale methods, the setup uses temperature 0.7 and concatenates two sampled responses with the original query for retrieval.Source: §A.1 · p. 11

A.2 Baseline Query Rewrite Prompts

  • Figure 4 lists the ThinkQE, TongSearch, and Reasoner-Rewriter prompt instructions.Source: §A.2 · p. 11

A.3 Human Skills for BRIGHT and R2MED

  • The appendix presents human skills for all subjects, used by Oracle, RRewriter, and Human Skills in the specified tables.Source: §A.3 · p. 11

A.4 SkillQR Prompts

  • Figures 6–9 provide the proposer, judge, refinement, and rewriter prompts; Table 6 lists methods, backbone models, and implementation sources.Source: §A.4 · p. 11, Table 6 · p. 11

B Time Analysis

Skill mining is reported to finish within 10 minutes for most BRIGHT subjects. Skill discovery is relatively stable to cluster-count changes in the evaluated range, while increasing the skill cap does not consistently improve retrieval.

B Time Analysis

  • Most BRIGHT subjects complete offline skill mining within 10 minutes.The reported timings exclude embedding generation, clustering, and model initialization.Source: B Time Analysis · p. 11, Table 7 · p. 11

C.1 Skill Discovery Design and Sensitivity

  • SkillQR uses MiniBatchKMeans directly because Birch produced one dominant cluster and many small fragments, requiring a further reclustering step.The fragments left the proposer without a useful corpus overview.Source: §C.1 · pp. 11–12
  • Halving or doubling the cluster count reduces averaged NDCG@10 by 0.84 and 1.30 points, respectively, with performance relatively stable across the evaluated range.The tested cluster counts differ fourfold between the smallest and largest settings.Source: §C.1 · p. 12
  • Increasing the skill cap to ten reduces mean NDCG@10 by 2.62 points, although Biology improves, so a larger skill set does not consistently help.The authors suggest proposer-judge refinement may reduce dependence on the initial cluster partition by broadening skill coverage.Source: §C.1 · p. 12

C.2.1 Implementation Details

The router comparison evaluates strategies on queries where candidate skills differ in NDCG@10, and retrospective or calibrated skill selection offers evidence of potential gains. SkillQR’s discovered skills also outperform the query-tailored SkillQR-Syn alternative in the reported comparison.

C.2.1 Implementation Details

  • The router comparison retains 722 queries whose candidate skills yield different NDCG@10 scores and averages results within subjects, then equally across subjects.Compared routers select from the generic instruction and discovered skill set; the embedding router uses cosine similarity, while Laya performs zero-shot multiple-choice selection.Source: §C.2.1 · pp. 12–13

C.2.2 Potential for Router Improvement

  • On 722 skill-sensitive queries, selecting the best single skill per subject achieves 44.79 NDCG@10, compared with 42.90 for LLM routing.Both retrospective reference policies use groundtruth retrieval scores; the best-per-query policy is also considered.Source: §C.2.2 · p. 13
  • With 30 calibration queries per subject, calibrated selection achieves 43.27 NDCG@10 versus 42.80 for LLM routing on held-out queries, a 0.47-point gain.The offline replay averages results over 1,000 random splits and selects a skill using gold relevance judgments.Source: §C.2.2 · p. 13, Table 9 · p. 13
  • The calibration replay assumes gold relevance judgments and complete feedback for all candidate skills, leaving efficient feedback collection and continual online adaptation as future work.These feedback-based extensions add supervision; the current pipeline requires neither task-specific relevance labels nor router fine-tuning.Source: §C.2.2 · p. 14

C.3 Skill Performance

  • SkillQR’s discovered skills work with Reason-Embed, while SkillQR-Syn underperforms SkillQR in the reported BRIGHT comparison.SkillQR-Syn mines skills per cluster, selects the five most query-similar skills, then synthesizes a query-specific skill; the authors suggest this lacks the holistic corpus view from dual-agent refinement.Source: §C.3 · p. 14

D Additional Qualitative Analysis

The qualitative case study contrasts a generic rewrite focused on solving a query with SkillQR’s focus on eliciting the core concept. The AI-use statement says closed-source LLMs were used only to improve manuscript clarity and presentation.

E AI Use Statement

  • Closed-source LLMs were used to improve manuscript clarity and presentation, not research design, experiments, analysis, or conclusions.The authors state they take full responsibility for the research and its conclusions.Source: E AI Use Statement · p. 15
  • Retrospective best-skill selection indicates further improvement potential for routing on skill-sensitive BRIGHT queries.The referenced figure reports NDCG@10 for Diver-Retriever and Qwen2.5-7B-Instruct but does not provide a numeric outcome in its caption.Source: Figure 10 · p. 16

D Additional Qualitative Analysis

  • SkillQR shifts the rewrite from problem-solving toward theorem-explaining, improving NDCG@10 from 0.0 to 43.1 in the illustrated case.The generic skill lists numbers and formulas, whereas SkillQR identifies concept elicitation as the core reasoning need.Source: D Additional Qualitative Analysis · p. 15, Figure 11 · p. 17

Related papers

Loading 2610.33768v1…