Source-linked AI summary
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, Christopher D. Manning
TL;DR
Existing retrieval-augmented language models often retrieve short contiguous chunks, limiting holistic understanding of lengthy documents. RAPTOR recursively embeds, clusters, and summarizes chunks into a multilevel tree for retrieval at different abstractions. Controlled experiments show improvements over traditional retrieval, including new state-of-the-art results across several question-answering tasks.
Problem
Most retrieval-augmented methods retrieve short contiguous chunks, limiting their ability to represent and leverage large-scale discourse structure.
Method
RAPTOR recursively embeds, clusters, and summarizes text chunks to construct a hierarchical tree that supports retrieval at multiple levels of abstraction.
Results
RAPTOR consistently outperforms corresponding retrievers across datasets and sets new benchmarks, including 55.7% F-1 on QASPER and 82.6% accuracy on QuALITY with GPT-4.
Takeaways & Limitations
Recursive summaries enable retrieval-augmented language models to synthesize information across sections of long documents and answer questions at different levels.
Takeaways & Limitations
About 4% of summaries contained minor hallucinations, although these did not propagate to parent nodes or discernibly affect question-answering tasks.
Abstract
from arXiv · showhide
Retrieval-augmented language models can better adapt to changes in world state and incorporate long-tail knowledge. However, most existing methods retrieve only short contiguous chunks from a retrieval corpus, limiting holistic understanding of the overall document context. We introduce the novel approach of recursively embedding, clustering, and summarizing chunks of text, constructing a tree with differing levels of summarization from the bottom up. At inference time, our RAPTOR model retrieves from this tree, integrating information across lengthy documents at different levels of abstraction. Controlled experiments show that retrieval with recursive summaries offers significant improvements over traditional retrieval-augmented LMs on several tasks. On question-answering tasks that involve complex, multi-step reasoning, we show state-of-the-art results; for example, by coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20% in absolute accuracy.
1 INTRODUCTION
Large language models require external retrieval because their domain-specific knowledge is incomplete and world facts change, while conventional retrieval often misses document-wide context. RAPTOR addresses this by recursively clustering and summarizing text into a tree that supports retrieval at multiple levels of abstraction.
- LLMs lack sufficient domain-specific knowledge for particular tasks, and changing world facts can invalidate information encoded in their parameters.
- Most retrieval-augmented methods retrieve a few short, contiguous chunks, limiting their ability to represent large-scale discourse structure.
- RAPTOR recursively clusters text chunks, summarizes each cluster, and constructs a bottom-up tree whose nodes represent different levels of abstraction.
- The hierarchical structure lets an LLM load context chunks at different levels, supporting questions that require either local details or broader synthesis.
- RAPTOR outperforms current retrieval augmentation in controlled experiments and achieves new state-of-the-art results on NarrativeQA, QASPER, and QuALITY.
2 RELATED WORK
Retrieval-augmented systems have advanced through improved retrievers, readers, and end-to-end training, while recursive summarization offers another way to organize context. Standard contiguous chunking remains limited because extracted snippets may omit semantic context or become misleading.
- Retrieval Methods: Retrieval methods have progressed from term-based techniques such as TF-IDF and BM25 to deep learning-based strategies and language-model retrievers.
- Retrieval Methods: End-to-end retrieval systems include Atlas, REALM, and RAG, which jointly or separately integrate retrievers with pretrained language models.
- Retrieval Methods: Standard retrieval pipelines predominantly chunk corpora and encode them with BERT-based retrievers, but contiguous segmentation may miss complete semantic depth.
- Retrieval Methods: Reading extracted snippets from technical or scientific documents can omit important context, making them difficult to read or misleading.
- Recursive summarization: Recursive summarization methods condense smaller text chunks and integrate them into summaries of larger sections, though compression can sometimes be lossy.
3 METHODS
RAPTOR builds a hierarchical tree by clustering embedded text segments, summarizing clusters, and recursively re-embedding the summaries. It supports retrieval across abstraction levels through tree traversal or collapsed-tree search, with soft clustering and model-based summarization forming key components.
- Tree Construction: The corpus is initially segmented into contiguous texts of length 100 and embedded using SBERT.Sentences exceeding the limit are moved wholly to the next chunk to preserve contextual coherence.
- Overview of RAPTOR: RAPTOR recursively embeds, clusters, and summarizes text to construct a multi-layered tree representing documents at different levels of abstraction.Summarized nodes are re-embedded and processed repeatedly until further clustering becomes infeasible.
- Clustering Algorithm: Soft clustering allows text nodes to belong to multiple clusters, accommodating segments that contain information relevant to several topics.The clustering algorithm uses Gaussian Mixture Models, with UMAP reducing embedding dimensionality and varying n_neighbors to create hierarchical structure.
- Clustering Algorithm: BIC selects the number of clusters, after which Expectation-Maximization estimates the GMM means, covariances, and mixture weights.BIC balances model fit against complexity.
- Model-Based Summarization: Clustered nodes are sent to gpt-3.5-turbo, which produces concise summaries that compress retrieved information into a manageable size.If a cluster exceeds the summarization model’s token threshold, clustering is recursively applied within that cluster.
- Querying: RAPTOR queries its tree either by layer-by-layer top-k traversal or by collapsed-tree search across all nodes, with collapsed retrieval consistently performing better in QASPER experiments.Collapsed-tree retrieval searches nodes collectively and can select information at the granularity appropriate to the question, but requires similarity search over all tree nodes.
4 EXPERIMENTS
Experiments evaluate RAPTOR on NarrativeQA, QASPER, and QuALITY using controlled retrieval comparisons, state-of-the-art comparisons, and layer analyses. RAPTOR consistently improves over standard retrieval baselines and benefits from combining original text with higher-level summaries.
- Controlled Baseline Comparisons: RAPTOR consistently outperforms SBERT, BM25, and DPR across NarrativeQA, QASPER, and QuALITY with UnifiedQA-3B.The controlled comparisons evaluate each retriever with and without the RAPTOR tree structure.
- Controlled Baseline Comparisons: 53.1%, 55.7%, and 36.6% are RAPTOR’s QASPER F-1 Match scores with GPT-3, GPT-4, and UnifiedQA, exceeding DPR by 1.8, 2.7, and 4.5 points.The corresponding margins over BM25 are 6.5, 5.5, and 10.2 points.
- Controlled Baseline Comparisons: 62.4% accuracy on QuALITY with RAPTOR improves over DPR and BM25 by 2% and 5.1%, respectively.With UnifiedQA, RAPTOR exceeds DPR and BM25 by 2.7% and 6.7%, respectively.
- Controlled Baseline Comparisons: RAPTOR exceeds BM25 and DPR on NarrativeQA by 7.3 and 2.7 ROUGE-L points, with gains across BLEU-1, BLEU-4, and METEOR.The reported margins range from 1.7 to 5.8 points over BM25 and 0.7 to 2.1 points over DPR.
- Comparison to State-of-the-art Systems: 55.7% F-1 on QASPER with GPT-4 surpasses CoLT5 XL’s 53.9%, setting a new benchmark.The comparison is reported in the state-of-the-art evaluation.
- Comparison to State-of-the-art Systems: 82.6% accuracy on QuALITY with GPT-4 surpasses the previous best 62.3%, while RAPTOR exceeds CoLISA by 21.5% on QuALITY-HARD.QuALITY-HARD contains questions requiring rereading, difficult reasoning, or both.
- Comparison to State-of-the-art Systems: On NarrativeQA, RAPTOR with UnifiedQA sets a new state-of-the-art METEOR score and outperforms recursive summarization on all reported metrics.The paper attributes this advantage to intermediate layers and clustering, which provide general themes and specific details.
- Layer Analysis: The full tree structure supports both thematic and detail-oriented questions by making original text and higher-level summaries available for retrieval.Layer-contribution analyses examine the role of nodes at different levels.
5 CONCLUSION
RAPTOR is a tree-based retrieval system that augments language models with contextual information at multiple abstraction levels. Recursive clustering and summarization enable retrieval that synthesizes information across document sections, outperforming traditional retrieval methods and setting new question-answering benchmarks.
- RAPTOR augments large language models’ parametric knowledge with contextual information at various levels of abstraction.
- Recursive clustering and summarization create a hierarchical tree that synthesizes information across sections of retrieval corpora.
- RAPTOR uses the tree structure during querying to retrieve information more effectively.
- Controlled experiments show that RAPTOR outperforms traditional retrieval methods and sets new performance benchmarks on several question-answering tasks.
6 REPRODUCIBILITY STATEMENT
The experiments use specified language models for question answering and summarization. Evaluation relies on three public datasets, and the source code is stated to be publicly available.
- GPT-3 and GPT-4 are used for question-answering tasks, while GPT-3.5-turbo is used for summarization.
- UnifiedQA is used for question-answering tasks and is publicly available through Hugging Face.
- QuALITY, QASPER, and NarrativeQA are publicly accessible evaluation datasets supporting replication of the retrieval and question-answering tests.
- The RAPTOR source code is stated to be publicly available.
A SCALABILITY AND COMPUTATIONAL EFFICIENCY OF THE TREE-BUILDING PROCESS
RAPTOR’s tree-building process scales linearly with document length in both token expenditure and build time. The section also reports an ablation comparing RAPTOR’s clustering mechanism with a balanced tree built from contiguous chunks.
- Scalability and computational efficiency: RAPTOR tree construction was evaluated on an Apple M1 Mac with 16GB of RAM across context lengths from 12,500 to 78,000 tokens.
- Token expenditure: Figure 5 shows that total token expenditure has a linear correlation with initial document length across QuALITY, QASPER, and NarrativeQA.
- Build time: Figure 6 shows that RAPTOR tree construction time scales linearly with document length for documents up to 80,000 tokens.
- Conclusion: The authors conclude that construction cost scales predictably and linearly in both tokens expended and build time as input complexity and volume grow.
- Clustering ablation: The ablation compares RAPTOR’s clustering and summarization with a balanced tree that recursively summarizes contiguous chunks using a window size of 7 nodes.
B.2 RESULTS & DISCUSSION
The ablation study compares RAPTOR’s clustering mechanism with a recency-based tree approach. Its results indicate that RAPTOR achieves higher accuracy than the alternative.
- RAPTOR’s clustering mechanism improves accuracy over the recency-based tree approach.
- The ablation study compares RAPTOR with a recency-based tree approach.
C DATASET STATISTICS AND COMPRESSION RATIOS
RAPTOR summaries compress child-node content while supporting a dedicated summarization prompt. Across datasets, summaries average 28% of their children’s combined length.
- A 0.28 average summary-to-child length ratio corresponds to a 72% compression rate across all datasets.Summaries average 131 tokens, compared with 86 tokens for child nodes.
- Table 10 reports average summary and child-node lengths across datasets, while Table 11 presents the summarization prompt.
E HALLUCINATION ANALYSIS
The hallucination analysis found that RAPTOR summaries were usually reliable, with minor hallucinations occurring in a small sampled fraction. These errors generally did not propagate upward or measurably affect question answering.
- 4% (6 of 150) sampled nodes contained some form of hallucination.The sample covered 150 nodes across 40 stories, with each node annotated by hand.
- Most hallucinations involved adding minor information from training data or incorrectly extrapolating beyond the summarized text.
- One example incorrectly identified Jr. Ajor and Co-Tan as sisters even though the child text did not explicitly support that relationship.
- Hallucinations did not propagate to higher layers, and they generally did not alter the text’s thematic interpretation.
- The analysis found no discernible impact of hallucinations on QA performance.
G QUALITATIVE ANALYSIS
RAPTOR’s qualitative retrieval analysis compares its hierarchical contexts with DPR on thematic, multi-hop questions. RAPTOR retrieves broader, more relevant information, enabling GPT-4 to produce more complete answers.
- RAPTOR was compared with DPR on thematic, multi-hop questions about a 1500-word Cinderella story.
- RAPTOR’s context enabled GPT-4 to explain Cinderella’s happy ending, while DPR’s context lacked information about the story’s conclusion.
- RAPTOR’s retrieved context included information from across the story, whereas DPR often returned detailed passages from a narrower subset.
- For the theme question, RAPTOR produced a more complete answer covering transformation, adversity, and Cinderella’s eventual happiness and love.
- In another 2600-word story, an upper-level summary nearly directly answered a question about the narrative’s central theme.
H NARRATIVEQA EVALUATION SCRIPT
The supplementary evaluation section documents changes to the NarrativeQA evaluation script and analyzes which RAPTOR tree layers supply retrieved nodes. The analysis reports substantial retrieval from non-leaf layers across datasets and retrievers.
- NARRATIVEQA EVALUATION SCRIPT: The NarrativeQA evaluation script was modified by adding BLEU smoothing and changing BLEU-4 weighting toward an even distribution across n-gram levels.
- LAYER ANALYSIS: The supplementary tables break down RAPTOR performance when querying different tree layers for Stories 2–5.
- LAYER ANALYSIS: Figure 7 and Tables 18–21 report the distribution of retrieved nodes across layers for NarrativeQA, Quality, and Qasper using DPR, SBERT, and BM25.
- LAYER ANALYSIS: 18.5% to 57% of retrieved nodes came from non-leaf nodes across three datasets and three retrievers.