Source-linked AI summary
STAIR (STructure Aware Information Retriever): A novel dataset and LLM based retriever for document structure augmentation
Vineet Kumar, Meghanadh Pulivarthi, vishwajeet kumar, Jaydeep Sen, Riyaz Ahmad Bhat, Sachindra Joshi
TL;DR
Long-context retrievers often discard global corpus structure, making precise retrieval important for RAG. STAIR uses a Table of Contents to store and retrieve information, and achieves 82.6% Recall@1 while reducing hallucinations and outperforming strong baselines.
Problem
Existing long-context retrieval benchmarks and retrievers generally lack or discard structured global views such as Tables of Contents, limiting evaluation of structure-aware retrieval.
Method
STAIR uses an LLM and a corpus Table of Contents to generate the most probable leaf section header for a query, and SearchTome evaluates ToC retrieval across 18 books from 6 domains.
Results
82.6% Recall@1: STAIR outperforms DSI and other strong baselines on SearchTome.
Takeaways & Limitations
ToC-based retrieval supports low-hallucination generative retrieval and generalization when training examples are scarce.
Takeaways & Limitations
The evaluation is limited to corpora where a global structure already exists, so it may not represent the diversity of real-world use cases.
Abstract
from arXiv · showhide
Retrieval Augmented Generation (RAG) is a key component for generating accurate and hallucination free answers using Large Language Models (LLMs). LLMs are improving at handling long context, but still suffer from "lost in the middle" problem. Thus, precise and accurate retrieval is important. Current retrievers chunk long context into length-based manageable chunks - in the process throwing away rich and informative semantic global structure in the corpus. We introduce a novel retrieval system STAIR that empowers an LLM to exploit global structure in a corpus such as a Table of Contents (ToC) to efficiently store and retrieve information from its model parameters. Our thorough and careful ablation studies with a finetuned Differentiable Search Index (DSI) system show that ToC helps build a low hallucination (less than 0.05%) generative Information Retrieval (IR) system and can generalize to examples where very few training samples are available. To further research in this novel direction of ToC based retrieval we release SearchTome - a diverse benchmark created from 18 books across 6 diverse domains to further research in this novel direction. STAIR achieves a high Recall@1 score of 82.6% on SearchTome as compared to DSI (76.9%), where the difference is found to be statistically significant. STAIR easily beats other strong baselines such as BM25 (59.5%), DPR (68.7%) and out-of-the-box Mistral (13.8%).
1 Introduction
STAIR addresses the loss of semantic global structure caused by length-based chunking by using a Table of Contents (ToC) to organize retrieval. It reports improved retrieval, reduced hallucinations, and better generalization with few training examples.
- Motivation: Length-based chunks discard semantic structure and boundaries, causing sub-optimal retrieval quality.The authors link this limitation to chunk competition and reduced semantic coherence.
- Motivation: A ToC provides a global corpus view with semantically coherent, clearly bounded retrieval units.The paper presents ToC entries as topic-organized units with explicit semantic boundaries.
- Approach: STAIR uses an LLM to exploit ToC structure for storing and retrieving corpus information from model parameters.Its design is inspired by human ToC-guided search and model-based corpus storage.
- Results: 82.6 Recall@1 is reported for STAIR, 7.4% better than the closest baseline DSI.The comparison is reported for ToC-based retrieval on the paper’s evaluation setting.
- Resources: SearchTome contains 18 books across 6 domains and provides ToC entries mapped to pertinent text sections.The benchmark includes train, development, and test queries with gold ToC retrieval outputs.
2 Related Work
Related work spans dense, learned sparse, and model-based retrieval, but the paper positions STAIR as adding hierarchical semantic structure to model-based indexing.
- Motivation: Precise retrieval matters for applications including semantic search, open-domain question answering, and RAG.The paper connects retrieval precision with reducing hallucinations in generative models.
- Dense Retrieval: Dense retrieval represents queries and documents as vectors and scores them by vector distance or token-level interaction.Examples include DPR and ColBERT, while RAPTOR builds a hierarchical tree through embedding, clustering, and summarization.
- Learned Sparse Retrieval: Learned sparse retrieval uses sparse high-dimensional representations, extending lexical matching beyond traditional BM25.SPLADE is given as a representative learned sparse system.
- Model-Based Indexing: Model-based indexing embeds an entire corpus in model parameters and directly generates document identifiers.The paper describes STAIR as extending this paradigm with hierarchical and semantic global structure.
3 SearchTome: A new benchmark for Table of Contents based retrieval
SearchTome is introduced as a structured benchmark for evaluating long-context retrieval with a Table of Contents. It uses diverse textbooks and generated queries to test retrieval coverage across domains.
- Why a New Benchmark Is Needed: SearchTome addresses the lack of long-context retrieval benchmarks that provide a structured global view of the input document.The paper contrasts it with benchmarks focused on NLI, summarization, or answer generation without gold retrieval passages.
- SearchTome Construction: The benchmark contains 18 open textbooks spanning six domains, with three books selected per domain.The domains are Education, Finance, Law, Medicine, Natural Sciences, and Social Sciences.
- SearchTome Construction: The authors parse each book’s PDF to extract its Table of Contents and create cleaned structured content.This construction supports evaluation of retrieval using global semantic structure.
- SearchTome Construction: Generated questions are divided into training, development, and test data, with most used as test questions to assess content coverage.The same training setup is used for DSI and STAIR, while the benchmark supplies retrieval targets based on ToC entries.
4 STAIR: Table of Contents Searcher
STAIR retrieves the leaf section best suited to a query by combining book content with the document’s complete Table of Contents. Its training separates corpus knowledge ingestion from learning to select a leaf node, while inference constrains outputs to valid ToC leaves.
- Problem formulation: STAIR formulates retrieval as selecting the correct leaf node from a book’s Table of Contents for a user query.The leaf node’s associated content is intended to answer the query.
- Problem formulation: The ToC is represented as a hierarchy of parent and child section titles, with leaf nodes forming the retrieval candidates.Edges connect a parent section to its subsections, and the leaf-node set is a subset of the ToC.
- Training STAIR: STAIR learns corpus knowledge by mapping book text to corresponding ToC section titles and learns to select the most relevant leaf from the complete global structure.These are described as two correlated but different training tasks.
- Training STAIR: During supervised fine-tuning, the model receives the query together with the complete ToC for that book and generates the corresponding correct leaf node.The ToC remains constant across queries from the same book.
- Training STAIR: Inference uses constrained generation so that outputs are restricted to valid ToC leaf nodes.This pipeline stage limits the output vocabulary to the valid retrieval units.
5 Experiments
The experiments evaluate retrieval systems on book-level test sets using ranking metrics, with STAIR trained and compared against several retrieval baselines. The setup includes corpus-specific fine-tuning for STAIR and DSI alongside out-of-the-box comparison systems.
- Evaluation: The evaluation reports Recall@1, Recall@3, and nDCG@3 on the test set for each book.BeIR is used to compute these metrics.
- Baselines: The comparison includes out-of-the-box Mistral, BM25, DPR, and DSI as retrieval baselines.DPR uses NV-Embed-v2, BM25 uses Elasticsearch, and DSI is fine-tuned on each book’s training split.
- Experimental configuration: DSI is evaluated with a 512-token maximum input length, while STAIR uses 14k tokens and both systems use a maximum output length of 64 tokens.These settings define the reported experimental configuration.
6 Results and Analysis
STAIR outperforms the evaluated retrieval baselines and benefits from Table-of-Contents input, especially for reducing hallucinations and handling leaves with few training examples. Its Recall@1 improvement over DSI is statistically significant.
- STAIR outperforms BM25, DPR, and fine-tuned DSI on SearchTome retrieval.
- The Recall@1 difference between STAIR and DSI is statistically significant with p < 0.05.
- 6.1 Ablations: STAIR maintains hallucination rates near zero across training-example counts, whereas DSI hallucinates more often for sparsely represented leaves.Hallucinations are defined as generated non-leaf nodes.
- 6.2 Error Analysis with Anecdotal Examples: Out-of-the-box Mistral has an 86.20% error rate, including 26.81% non-leaf predictions and 23.86% hallucinations.The analysis describes this as evidence that Mistral needs knowledge ingestion and task fine-tuning.
- 6.1 Ablations: 18.67% is STAIR’s error rate, with only 0.05% of predictions being non-leaf nodes.For comparison, DSI has a 24.31% error rate and 3.25% non-leaf predictions.
- 6.2 Error Analysis with Anecdotal Examples: In a preschool-development example, STAIR identifies the gold leaf, while BM25, DPR, DSI, and Mistral select incorrect or hallucinated alternatives.
- 6.1 Ablations: STAIR retains high Recall@1 for leaves with few training examples, while DSI’s advantage is smaller for those leaves.The ablation attributes this pattern to DSI needing to learn and remember book structure from training pairs without ToC input.
7 Conclusion
STAIR uses Table-of-Contents structure to help an LLM store and retrieve information through generated leaf section headers. It achieves strong retrieval performance while reducing hallucinations, and SearchTome supports further research in ToC-based retrieval.
- STAIR leverages ToC structure to store corpus information in an LLM’s parameters and generate the most probable leaf section header for a query.
- 82.6% Recall@1 gives STAIR an approximately 7.4% gain over the next-best system, outperforming BM25, finetuned DSI, and DPR.
- STAIR’s ToC-based retrieval reduces hallucinations to almost zero and supports retrieval when training examples are scarce.
- SearchTome provides a comprehensive benchmark across 6 domains with 18 books and train, dev, and test splits.
- Future work will dynamically create ToC-like structures for unseen corpora and develop zero-shot iterative retrieval with reasoning over retrieved leaf-node content.
8 Limitations
The current evaluation is limited to corpora that already contain a global structure, leaving broader real-world coverage for future work.
- The evaluation is limited to corpora where a global structure exists, so its coverage of diverse real-world use cases remains incomplete.