PaperlayerView exact revision on arXiv

Source-linked AI summary

RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, Christopher D. Manning

arXiv:2401.18059v1cs.CLcs.LG

TL;DR

Existing retrieval-augmented language models often retrieve short contiguous chunks, limiting holistic understanding of lengthy documents. RAPTOR recursively embeds, clusters, and summarizes chunks into a multilevel tree for retrieval at different abstractions. Controlled experiments show improvements over traditional retrieval, including new state-of-the-art results across several question-answering tasks.

  • Problem

    Most retrieval-augmented methods retrieve short contiguous chunks, limiting their ability to represent and leverage large-scale discourse structure.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    RAPTOR recursively embeds, clusters, and summarizes text chunks to construct a hierarchical tree that supports retrieval at multiple levels of abstraction.

    Source: §3 · p. 3

  • Results

    RAPTOR consistently outperforms corresponding retrievers across datasets and sets new benchmarks, including 55.7% F-1 on QASPER and 82.6% accuracy on QuALITY with GPT-4.

    Source: §4 · pp. 7–8, §5 · p. 9

  • Takeaways & Limitations

    Recursive summaries enable retrieval-augmented language models to synthesize information across sections of long documents and answer questions at different levels.

    Source: §5 · p. 9

    About 4% of summaries contained minor hallucinations, although these did not propagate to parent nodes or discernibly affect question-answering tasks.

    Source: §3 · p. 4

Abstract

from arXiv · show

Retrieval-augmented language models can better adapt to changes in world state and incorporate long-tail knowledge. However, most existing methods retrieve only short contiguous chunks from a retrieval corpus, limiting holistic understanding of the overall document context. We introduce the novel approach of recursively embedding, clustering, and summarizing chunks of text, constructing a tree with differing levels of summarization from the bottom up. At inference time, our RAPTOR model retrieves from this tree, integrating information across lengthy documents at different levels of abstraction. Controlled experiments show that retrieval with recursive summaries offers significant improvements over traditional retrieval-augmented LMs on several tasks. On question-answering tasks that involve complex, multi-step reasoning, we show state-of-the-art results; for example, by coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20% in absolute accuracy.

1 INTRODUCTION

Large language models require external retrieval because their domain-specific knowledge is incomplete and world facts change, while conventional retrieval often misses document-wide context. RAPTOR addresses this by recursively clustering and summarizing text into a tree that supports retrieval at multiple levels of abstraction.

  • LLMs lack sufficient domain-specific knowledge for particular tasks, and changing world facts can invalidate information encoded in their parameters.Source: §1 · p. 1
  • Most retrieval-augmented methods retrieve a few short, contiguous chunks, limiting their ability to represent large-scale discourse structure.Source: §1 · p. 1
  • RAPTOR recursively clusters text chunks, summarizes each cluster, and constructs a bottom-up tree whose nodes represent different levels of abstraction.Source: §1 · p. 1, Figure 1 · p. 2
  • The hierarchical structure lets an LLM load context chunks at different levels, supporting questions that require either local details or broader synthesis.Source: §1 · p. 1
  • RAPTOR outperforms current retrieval augmentation in controlled experiments and achieves new state-of-the-art results on NarrativeQA, QASPER, and QuALITY.Source: §1 · p. 2

2 RELATED WORK

Retrieval-augmented systems have advanced through improved retrievers, readers, and end-to-end training, while recursive summarization offers another way to organize context. Standard contiguous chunking remains limited because extracted snippets may omit semantic context or become misleading.

Retrieval Methods

  • Retrieval methods have progressed from term-based techniques such as TF-IDF and BM25 to deep learning-based strategies and language-model retrievers.Source: §2 · p. 2
  • End-to-end retrieval systems include Atlas, REALM, and RAG, which jointly or separately integrate retrievers with pretrained language models.Source: §2 · p. 2
  • Standard retrieval pipelines predominantly chunk corpora and encode them with BERT-based retrievers, but contiguous segmentation may miss complete semantic depth.Source: §2 · p. 3
  • Reading extracted snippets from technical or scientific documents can omit important context, making them difficult to read or misleading.Source: §2 · p. 3

Recursive summarization

  • Recursive summarization methods condense smaller text chunks and integrate them into summaries of larger sections, though compression can sometimes be lossy.Source: §2 · p. 3

3 METHODS

RAPTOR builds a hierarchical tree by clustering embedded text segments, summarizing clusters, and recursively re-embedding the summaries. It supports retrieval across abstraction levels through tree traversal or collapsed-tree search, with soft clustering and model-based summarization forming key components.

Tree Construction

  • The corpus is initially segmented into contiguous texts of length 100 and embedded using SBERT.Sentences exceeding the limit are moved wholly to the next chunk to preserve contextual coherence.Source: §3 · p. 3

Overview of RAPTOR

  • RAPTOR recursively embeds, clusters, and summarizes text to construct a multi-layered tree representing documents at different levels of abstraction.Summarized nodes are re-embedded and processed repeatedly until further clustering becomes infeasible.Source: §3 · p. 3

Clustering Algorithm

  • Soft clustering allows text nodes to belong to multiple clusters, accommodating segments that contain information relevant to several topics.The clustering algorithm uses Gaussian Mixture Models, with UMAP reducing embedding dimensionality and varying n_neighbors to create hierarchical structure.Source: §3 · pp. 3–4
  • BIC selects the number of clusters, after which Expectation-Maximization estimates the GMM means, covariances, and mixture weights.BIC balances model fit against complexity.Source: §3 · p. 4

Model-Based Summarization

  • Clustered nodes are sent to gpt-3.5-turbo, which produces concise summaries that compress retrieved information into a manageable size.If a cluster exceeds the summarization model’s token threshold, clustering is recursively applied within that cluster.Source: §3 · p. 4

Querying

  • RAPTOR queries its tree either by layer-by-layer top-k traversal or by collapsed-tree search across all nodes, with collapsed retrieval consistently performing better in QASPER experiments.Collapsed-tree retrieval searches nodes collectively and can select information at the granularity appropriate to the question, but requires similarity search over all tree nodes.Source: §3 · pp. 3–6

4 EXPERIMENTS

Experiments evaluate RAPTOR on NarrativeQA, QASPER, and QuALITY using controlled retrieval comparisons, state-of-the-art comparisons, and layer analyses. RAPTOR consistently improves over standard retrieval baselines and benefits from combining original text with higher-level summaries.

Controlled Baseline Comparisons

  • RAPTOR consistently outperforms SBERT, BM25, and DPR across NarrativeQA, QASPER, and QuALITY with UnifiedQA-3B.The controlled comparisons evaluate each retriever with and without the RAPTOR tree structure.Source: §4 · pp. 6–7
  • 53.1%, 55.7%, and 36.6% are RAPTOR’s QASPER F-1 Match scores with GPT-3, GPT-4, and UnifiedQA, exceeding DPR by 1.8, 2.7, and 4.5 points.The corresponding margins over BM25 are 6.5, 5.5, and 10.2 points.Source: §4 · p. 7
  • 62.4% accuracy on QuALITY with RAPTOR improves over DPR and BM25 by 2% and 5.1%, respectively.With UnifiedQA, RAPTOR exceeds DPR and BM25 by 2.7% and 6.7%, respectively.Source: §4 · p. 7
  • RAPTOR exceeds BM25 and DPR on NarrativeQA by 7.3 and 2.7 ROUGE-L points, with gains across BLEU-1, BLEU-4, and METEOR.The reported margins range from 1.7 to 5.8 points over BM25 and 0.7 to 2.1 points over DPR.Source: §4 · pp. 6–7

Comparison to State-of-the-art Systems

  • 55.7% F-1 on QASPER with GPT-4 surpasses CoLT5 XL’s 53.9%, setting a new benchmark.The comparison is reported in the state-of-the-art evaluation.Source: §4 · p. 8
  • 82.6% accuracy on QuALITY with GPT-4 surpasses the previous best 62.3%, while RAPTOR exceeds CoLISA by 21.5% on QuALITY-HARD.QuALITY-HARD contains questions requiring rereading, difficult reasoning, or both.Source: §4 · p. 8
  • On NarrativeQA, RAPTOR with UnifiedQA sets a new state-of-the-art METEOR score and outperforms recursive summarization on all reported metrics.The paper attributes this advantage to intermediate layers and clustering, which provide general themes and specific details.Source: §4 · p. 8, Table 6 · p. 9

Layer Analysis

  • The full tree structure supports both thematic and detail-oriented questions by making original text and higher-level summaries available for retrieval.Layer-contribution analyses examine the role of nodes at different levels.Source: §4.1 · pp. 8–9, Table 8 · p. 9

5 CONCLUSION

RAPTOR is a tree-based retrieval system that augments language models with contextual information at multiple abstraction levels. Recursive clustering and summarization enable retrieval that synthesizes information across document sections, outperforming traditional retrieval methods and setting new question-answering benchmarks.

  • RAPTOR augments large language models’ parametric knowledge with contextual information at various levels of abstraction.Source: §5 · p. 9
  • Recursive clustering and summarization create a hierarchical tree that synthesizes information across sections of retrieval corpora.Source: §5 · p. 9
  • RAPTOR uses the tree structure during querying to retrieve information more effectively.Source: §5 · p. 9
  • Controlled experiments show that RAPTOR outperforms traditional retrieval methods and sets new performance benchmarks on several question-answering tasks.Source: §5 · p. 9

6 REPRODUCIBILITY STATEMENT

The experiments use specified language models for question answering and summarization. Evaluation relies on three public datasets, and the source code is stated to be publicly available.

  • GPT-3 and GPT-4 are used for question-answering tasks, while GPT-3.5-turbo is used for summarization.Source: §6 · p. 10
  • UnifiedQA is used for question-answering tasks and is publicly available through Hugging Face.Source: §6 · p. 10
  • QuALITY, QASPER, and NarrativeQA are publicly accessible evaluation datasets supporting replication of the retrieval and question-answering tests.Source: §6 · p. 10
  • The RAPTOR source code is stated to be publicly available.Source: §6 · p. 10

A SCALABILITY AND COMPUTATIONAL EFFICIENCY OF THE TREE-BUILDING PROCESS

RAPTOR’s tree-building process scales linearly with document length in both token expenditure and build time. The section also reports an ablation comparing RAPTOR’s clustering mechanism with a balanced tree built from contiguous chunks.

Scalability and computational efficiency

Token expenditure

Build time

Conclusion

Clustering ablation

B.2 RESULTS & DISCUSSION

The ablation study compares RAPTOR’s clustering mechanism with a recency-based tree approach. Its results indicate that RAPTOR achieves higher accuracy than the alternative.

C DATASET STATISTICS AND COMPRESSION RATIOS

RAPTOR summaries compress child-node content while supporting a dedicated summarization prompt. Across datasets, summaries average 28% of their children’s combined length.

E HALLUCINATION ANALYSIS

The hallucination analysis found that RAPTOR summaries were usually reliable, with minor hallucinations occurring in a small sampled fraction. These errors generally did not propagate upward or measurably affect question answering.

  • 4% (6 of 150) sampled nodes contained some form of hallucination.The sample covered 150 nodes across 40 stories, with each node annotated by hand.Source: §E.2 · p. 17, §E.1 · p. 17
  • Most hallucinations involved adding minor information from training data or incorrectly extrapolating beyond the summarized text.Source: §E.2 · p. 17
  • One example incorrectly identified Jr. Ajor and Co-Tan as sisters even though the child text did not explicitly support that relationship.Source: §E.2 · p. 18
  • Hallucinations did not propagate to higher layers, and they generally did not alter the text’s thematic interpretation.Source: §E.2 · p. 18
  • The analysis found no discernible impact of hallucinations on QA performance.Source: §E.3 · p. 18

G QUALITATIVE ANALYSIS

RAPTOR’s qualitative retrieval analysis compares its hierarchical contexts with DPR on thematic, multi-hop questions. RAPTOR retrieves broader, more relevant information, enabling GPT-4 to produce more complete answers.

  • RAPTOR was compared with DPR on thematic, multi-hop questions about a 1500-word Cinderella story.Source: G QUALITATIVE ANALYSIS · p. 18
  • RAPTOR’s context enabled GPT-4 to explain Cinderella’s happy ending, while DPR’s context lacked information about the story’s conclusion.Source: G QUALITATIVE ANALYSIS · p. 20
  • RAPTOR’s retrieved context included information from across the story, whereas DPR often returned detailed passages from a narrower subset.Source: G QUALITATIVE ANALYSIS · p. 20
  • For the theme question, RAPTOR produced a more complete answer covering transformation, adversity, and Cinderella’s eventual happiness and love.Source: G QUALITATIVE ANALYSIS · p. 20
  • In another 2600-word story, an upper-level summary nearly directly answered a question about the narrative’s central theme.Source: G QUALITATIVE ANALYSIS · p. 20

H NARRATIVEQA EVALUATION SCRIPT

The supplementary evaluation section documents changes to the NarrativeQA evaluation script and analyzes which RAPTOR tree layers supply retrieved nodes. The analysis reports substantial retrieval from non-leaf layers across datasets and retrievers.

NARRATIVEQA EVALUATION SCRIPT

LAYER ANALYSIS

Related papers

Loading 2401.18059v1…