Source-linked AI summary
Imagine All The Relevance: Scenario-Profiled Indexing with Knowledge Expansion for Dense Retrieval
Sangam Lee, Ryang Heo, SeongKu Kang, Dongha Lee
TL;DR
Dense retrievers struggle to capture implicit relevance that requires reasoning beyond surface semantics, particularly across differing formats. SPIKE decomposes documents into scenario-based retrieval units, distills teacher-LLM supervision into a smaller generator, and combines scenario- with document-level relevance. Experiments report consistent retrieval improvements across models and query types, while scenarios also support user understanding and RAG context.
Problem
Dense retrieval struggles with reasoning-intensive implicit relevance and with query-document pairs in different formats.
Method
SPIKE generates document scenarios using teacher-LLM supervision, distills them into a smaller generator, and combines scenario-level with document-level relevance during retrieval.
Results
SPIKE consistently improves retrieval performance across dense retrievers, query types, and document types, while also improving usability and providing context for RAG.
Takeaways & Limitations
Scenario-profiled indexing provides retrieval explanations and additional context for LLM-based retrieval-augmented generation.
Takeaways & Limitations
High-quality scenario generation requires powerful LLMs, while applying them across an entire corpus is computationally expensive and impractical at large scale.
Abstract
from arXiv · showhide
Existing dense retrieval models struggle with reasoning-intensive retrieval task as they fail to capture implicit relevance that requires reasoning beyond surface-level semantic information. To address these challenges, we propose Scenario-Profiled Indexing with Knowledge Expansion (SPIKE), a dense retrieval framework that explicitly indexes implicit relevance by decomposing documents into scenario-based retrieval units. SPIKE organizes documents into scenario, which encapsulates the reasoning process necessary to uncover implicit relationships between hypothetical information needs and document content. SPIKE constructs a scenario-augmented dataset using a powerful teacher large language model (LLM), then distills these reasoning capabilities into a smaller, efficient scenario generator. During inference, SPIKE incorporates scenario-level relevance alongside document-level relevance, enabling reasoning-aware retrieval. Extensive experiments demonstrate that SPIKE consistently enhances retrieval performance across various query types and dense retrievers. It also enhances the retrieval experience for users through scenario and offers valuable contextual information for LLMs in retrieval-augmented generation (RAG).
1 Introduction
Dense retrieval struggles with implicit relevance, especially when reasoning or format bridging is required. SPIKE addresses this by indexing document scenarios that model how content satisfies hypothetical information needs.
- Reasoning-intensive queries require implicit relevance that surface-level semantic matching cannot reliably capture.BRIGHT exposes this limitation by requiring reasoning between queries and relevant documents.
- Format mismatches, including natural-language queries and code documents, further challenge dense retrievers trained mainly on natural-language pairs.Recovering relevance requires reasoning over specific code components.
- SPIKE decomposes documents into hypothetical retrieval scenarios that explicitly model reasoning paths connecting information needs with document content.The framework targets implicit relevance while supporting alignment across formats.
- SPIKE uses teacher-generated scenario supervision, distillation into a smaller generator, and joint scenario- and document-level relevance during retrieval.The approach is designed to improve retrieval and provide useful context for users and RAG systems.
- SPIKE is presented as consistently improving performance across retrieval models, query types, and document types while supporting explanations and RAG.These contributions summarize the paper’s intended retrieval and downstream benefits.
2 Related Works
Prior retrieval research addresses reasoning-intensive retrieval and document representation through complementary lines of work. These include reasoning-oriented benchmarks and expansions or replacements of standard document representations.
- Reasoning-intensive retrieval benchmarks target implicit query-document relevance that traditional surface-level matching may miss.BRIGHT is introduced to require reasoning for retrieving relevant documents.
- Document expansion methods append pseudo queries, summaries, or keyphrases, while other methods replace documents with summaries or propositions.These approaches improve retrieval representations and can avoid additional inference-time overhead when performed during indexing.
3 Proposed Method: SPIKE
SPIKE represents documents with structured scenarios that expose implicit relevance, trains a compact scenario generator through distillation, and combines scenario and document scores for retrieval.
- 3.1 Scenario: Reasoning format for modeling implicit relevance: A scenario is a structured reasoning format in which each document yields multiple paths linking hypothetical information needs to its content.Scenario generation is intended to uncover diverse forms of implicit relevance.
- 3.1 Scenario: Reasoning format for modeling implicit relevance: Scenario construction proceeds from a document’s main topic to key aspects, information needs, and explanations connecting each need to the document.The explanation explicitly describes why and how the document satisfies a hypothetical information need.
- 3.2 Scenario generator & Scenario Indexing: SPIKE uses a high-performing teacher LLM to create scenario-augmented supervision, then distills its reasoning capability into a smaller scenario generator.This addresses the cost of applying powerful LLMs across an entire corpus while retaining an efficient generator.
- 3.2 Scenario generator & Scenario Indexing: Generated main-topic and explanation components form scenario representations that are encoded into vectors and indexed alongside standard document representations.The scenario-profiled index supplies additional retrieval units for each document.
- 3.3 Retrieval with scenario: Retrieval computes document and scenario relevance, selects the strongest associated scenario, and combines the two scores using a relevance-weight hyperparameter.An efficient strategy first selects candidate documents with document scores before applying scenario scoring.
4 Experiments
Experiments evaluate SPIKE on BRIGHT for retrieval accuracy, reasoning-augmented queries, user judgments, RAG, and design ablations. Across these settings, SPIKE improves retrieval and makes scenario information useful for users and generators.
- 4.2 SPIKE improves retrieval performance (RQ1): SPIKE consistently improves average BRIGHT retrieval performance across six dense retrievers and 12 datasets, including models below and above 1B parameters.Table 1 reports nDCG@10 and average improvement rates; gains exceed 18% for E5-Mistral and SFR.
- 4.2 SPIKE improves retrieval performance (RQ1): SPIKE’s benefits are strongest for code and TheoT. math documents, while gains are smaller for Aops. and TheoQ. because question-like documents leave less room for scenario enrichment.Scenario profiling bridges natural-language queries with code or LaTeX-formatted content by expressing implicit reasoning in natural language.
- 4.2 SPIKE improves retrieval performance (RQ1): For reasoning-augmented queries, SPIKE with original queries matches or exceeds GPT-4 reasoning-query baselines, and adding SPIKE further improves all evaluated models.The comparison uses average nDCG@10 across original and GPT-4 reasoning queries.
- 4.3 SPIKE enhances retrieval experience for real-world users (RQ2): Human evaluations favor SPIKE over document-only and Document + Main Topic retrieval across comprehensibility, specificity, usefulness, and overall quality.The explanation component particularly improves comprehension and practical decision-making beyond the main topic alone.
- 4.5 Analysis of SPIKE: Among scenario components, explanations perform best individually, combinations with explanations improve further, and retrieval typically peaks at relevance weight α = 0.7.The relevance-weight trend differs by document type: scenarios matter more for natural-language and code documents, whereas math performance improves as α increases.
- 4.4 SPIKE boosts RAG performance with additional context (RQ3): Adding SPIKE scenarios to retrieved documents consistently improves QA accuracy across the evaluated retrievers and generation models in RAG.The paper attributes this improvement to more relevant retrieved documents and the additional scenario context supplied to the generator.
5 Analysis
SPIKE’s analysis shows that scenario components, relevance weighting, efficient candidate selection, and out-of-domain training choices jointly affect reasoning-aware retrieval performance.
- Scenario components: SPIKE’s E component achieves the highest individual retrieval performance, while combining E with M or I yields further improvements across natural-language, code, and math documents.The analysis identifies explicit reasoning modeling as especially important for capturing implicit relevance.
- Relevance weight: α = 0.7 produces peak retrieval performance across nearly all scenario components, although natural-language and code results indicate scenarios can outweigh original document content.For M+E in natural language and code, performance improves as α decreases, suggesting greater reliance on scenario information.
- Zero-shot generalization: SPIKE trained only on BEIR consistently improves retrieval across nearly all BRIGHT datasets despite the domain gap, including substantial gains on code and TheoT.The BEIR-trained generator sometimes outperforms a generator trained on BRIGHT, supporting strong out-of-domain generalization.
- Training-data scale: Using only 4,000 documents for scenario-augmented training already outperforms the baseline, while scaling the dataset produces significantly greater performance gains.This ablation evaluates scenario-generator training-data size using the BEIR corpus.
- Efficient retrieval: SPIKE’s efficient retrieval strategy preserves effectiveness close to exhaustive scenario scoring when k′ is sufficiently large while substantially reducing computational load.The reported comparison shows nearly identical performance to the naive method under sufficiently large candidate sets.
- Case study: On a StackOverflow example, standard dense retrieval misses a relevant code document, whereas SPIKE retrieves it by modeling implicit relevance through scenario-profiled retrieval.The query concerns integrating retrieved documents into a RetrievalQA chain, while the document expresses the relevant information through code examples rather than explicit wording.
6 Conclusion
SPIKE addresses reasoning-intensive retrieval by explicitly modeling implicit relevance through structured document scenarios. The framework improves retrieval across natural language, code, and math, supports usability and RAG, and generalizes beyond its training domain.
- 6 Conclusion: SPIKE reframes documents as structured retrieval scenarios to address implicit relevance that standard dense retrieval models struggle to capture.The conclusion presents scenario-based indexing as the framework’s central approach.
- 6 Conclusion: SPIKE enhances retrieval across natural language, code, and math while improving usability in real-world systems and providing effective context for RAG applications.The conclusion also reports robust out-of-domain generalization.
- 6 Conclusion: SPIKE is evaluated against pseudo-query and summary-based document expansion methods, which provide a comparison with traditional expansion approaches.The paper notes that direct comparison is not entirely fair because prior methods typically use different settings.
A.2 Results
SPIKE outperforms pseudo-query and summary-based document expansion on reasoning-intensive retrieval tasks across dense retrievers, while pseudo-query expansion can degrade performance.
- A.2 Results: SPIKE consistently improves average performance across all dense retrievers, whereas pseudo-query and summary expansion yield smaller gains or performance degradation.The comparison suggests that added context based on surface-level semantic information can introduce noise when reasoning is required.
B.1 Reasoning-augmented query result
SPIKE improves retrieval on GPT-4-generated reasoning queries across all 12 BRIGHT datasets, including queries that already contain explicit reasoning. Table 5 evaluates this comparison using nDCG@10 scores against retrieval models without SPIKE.
- Across all 12 BRIGHT datasets, SPIKE consistently improves retrieval performance on GPT-4-generated reasoning queries.
- SPIKE remains effective when applied to queries that already include explicit reasoning.
- The experiment evaluates standard retrieval models with and without SPIKE using nDCG@10 scores.
B.2 RAG Experimental Result
The RAG experiments provide full results for Claude-3.5-sonnet generation with GPT-4o evaluation, comparing document-only context against document-plus-scenario context.
- A second RAG configuration using Llama3.3-70B-Instruct is documented separately to assess robustness across generation models.
- Table 6 reports full RAG performance results using Claude-3.5-sonnet for generation and GPT-4o for answer evaluation.
- The comparison distinguishes document-only context from context augmented with scenario information.
B.3 Analysis result
The analysis section provides full ablation results and organizes BRIGHT datasets by document type into natural language, code, and mathematics.
- Table 8 presents the full ablation results discussed in Section 5.
- The document-type analysis groups datasets into Natural Language, Code, and Math categories.
C.2.1 Scenario-augmented training dataset
The scenario-augmented dataset is constructed with LLM-generated scenarios in a structured JSON format, using adaptive scenario counts and documented experimental prompts and evaluation procedures.
- GPT-4o generates scenario outputs with greedy decoding, while structured output ensures that scenarios are produced in JSON format.
- The number of generated scenarios varies adaptively with document content, giving longer documents more scenarios and shorter documents fewer.
- The appendix documents prompts for dataset construction, scenario-generator training, RAG answering, and RAG evaluation.