PaperlayerView exact revision on arXiv

Source-linked AI summary

HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models

Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, Yu Su

arXiv:2405.14831v3cs.CLcs.AI

TL;DR

LLMs and conventional RAG still struggle to integrate newly acquired knowledge across passage boundaries. HippoRAG combines LLM-based knowledge graphs with Personalized PageRank, inspired by hippocampal memory indexing, and improves multi-hop retrieval while reducing online retrieval cost and latency.

  • Problem

    Current RAG systems encode passages in isolation and therefore struggle with knowledge integration across passage boundaries.

  • Method

    HippoRAG uses an LLM to build a schemaless knowledge graph, retrieval encoders to link concepts, and Personalized PageRank to retrieve from query cues.

  • Results

    HippoRAG outperforms existing RAG methods on challenging multi-hop QA, matches or exceeds IRCoT with 10-30 times lower cost and 6-13 times greater speed, and gains further from integration with IRCoT.

  • Takeaways & Limitations

    HippoRAG provides a long-term-memory framework that retains RAG’s advantages while supporting knowledge integration and continuously updating retrieval.

  • Takeaways & Limitations

    All HippoRAG components are currently used off-the-shelf without extra training, and many errors arise from NER, OpenIE, and graph search.

Abstract

from arXiv · show

In order to thrive in hostile and ever-changing natural environments, mammalian brains evolved to store large amounts of knowledge about the world and continually integrate new information while avoiding catastrophic forgetting. Despite the impressive accomplishments, large language models (LLMs), even with retrieval-augmented generation (RAG), still struggle to efficiently and effectively integrate a large amount of new experiences after pre-training. In this work, we introduce HippoRAG, a novel retrieval framework inspired by the hippocampal indexing theory of human long-term memory to enable deeper and more efficient knowledge integration over new experiences. HippoRAG synergistically orchestrates LLMs, knowledge graphs, and the Personalized PageRank algorithm to mimic the different roles of neocortex and hippocampus in human memory. We compare HippoRAG with existing RAG methods on multi-hop question answering and show that our method outperforms the state-of-the-art methods remarkably, by up to 20%. Single-step retrieval with HippoRAG achieves comparable or better performance than iterative retrieval like IRCoT while being 10-30 times cheaper and 6-13 times faster, and integrating HippoRAG into IRCoT brings further substantial gains. Finally, we show that our method can tackle new types of scenarios that are out of reach of existing methods. Code and data are available at https://github.com/OSU-NLP-Group/HippoRAG.

1 Introduction

Current RAG systems struggle to integrate knowledge across passage boundaries, a capability needed for multi-hop and real-world tasks. HippoRAG addresses this gap by using an associative knowledge graph and Personalized PageRank to support efficient knowledge integration.

  • LLMs still lack continuously updating long-term memory, while RAG mainly presents new knowledge to an otherwise static model.
  • Current RAG encodes passages in isolation, limiting knowledge integration across documents despite its importance for scientific, legal, and medical tasks.
  • Iterative retrieval is used for standard multi-hop QA, but even multiple retrieval-generation steps can fail on path-finding questions.
  • HippoRAG builds a schemaless knowledge graph from a corpus and applies Personalized PageRank from query concepts to integrate information across passages.
  • Around 3 and 20 points of improvement over current RAG methods are reported on MuSiQue and 2WikiMultiHopQA, respectively.

2 HippoRAG

HippoRAG maps components of hippocampal memory indexing onto knowledge-graph construction and retrieval. It separates passage information into graph concepts, links synonyms, and completes query cues through graph propagation.

  • Hippocampal inspiration: Hippocampal memory indexing theory describes pattern separation during encoding and pattern completion during retrieval.
  • Offline Indexing: Offline indexing uses an instruction-tuned LLM to extract OpenIE noun phrases and relations, forming a schemaless knowledge graph over the corpus.
  • Offline Indexing: Retrieval encoders add edges between similar but nonidentical noun phrases, helping connect synonymous concepts for later retrieval.
  • Online Retrieval: Online retrieval extracts query named entities, links them to graph nodes, and uses those nodes as partial cues for pattern completion.
  • The study omits many details of hippocampal memory indexing theory for simplicity.
  • Online Retrieval: Personalized PageRank spreads probability from query nodes through their joint graph neighborhoods before passage scores are computed for ranking.
  • Node Specificity: Node specificity weights query-node probabilities by the inverse number of passages containing each node, providing a local alternative to IDF.

3 Experimental Setup

The experiments evaluate HippoRAG on three multi-hop QA datasets using retrieval and question-answering metrics against conventional, LLM-augmented, and iterative baselines. The setup also examines implementation choices and combined retrieval.

  • Datasets: Evaluation uses 1,000 validation questions each from MuSiQue, 2WikiMultiHopQA, and HotpotQA, with all candidate passages collected for retrieval.
  • Baselines: The comparison includes BM25, Contriever, GTR, ColBERTv2, Propositionizer, RAPTOR, and the iterative method IRCoT.
  • Metrics: Retrieval is measured with R@2 and R@5, while question answering is measured with exact match and F1.
  • Single-step retrieval: Single-step retrieval outperforms baselines on MuSiQue and 2WikiMultiHopQA and is comparable on HotpotQA.
  • Multi-step retrieval: Combining HippoRAG with IRCoT produces complementary improvements on all three datasets.

4 Results

HippoRAG improves retrieval and downstream QA across multi-hop benchmarks, including when paired with iterative IRCoT retrieval. Its single-step retrieval is competitive with IRCoT while substantially reducing online cost and latency.

  • Around 3% and 20% retrieval gains over current RAG methods occur on MuSiQue and 2WikiMultiHopQA, respectively.
  • HippoRAG combined with IRCoT improves R@5 by around 4% on MuSiQue and 18% on 2WikiMultiHopQA.
  • HippoRAG’s QA improvements correlate with retrieval improvements in both single-step and multi-step settings.
  • Up to 3%, 17% and 1% F1 improvements occur on MuSiQue, 2WikiMultiHopQA and HotpotQA using the same QA reader.
  • Single-step HippoRAG is on par with or better than IRCoT while online retrieval is 10-30 times cheaper and 6-13 times faster.

5 Discussions

Ablations and case studies attribute HippoRAG’s gains to flexible graph construction, Personalized PageRank, and graph associations that support complete single-step and path-finding retrieval.

  • 5.1 What Makes HippoRAG Work?: Replacing GPT-3.5 with REBEL for OpenIE causes large performance drops, while Llama-3.1 alternatives remain competitive in several settings.
  • 5.1 What Makes HippoRAG Work?: PPR is more effective than query-node and neighborhood-propagation baselines for including associations in retrieval across all three datasets.
  • 5.1 What Makes HippoRAG Work?: Node specificity helps MuSiQue and HotpotQA, whereas synonymy edges have the largest effect on 2WikiMultiHopQA.
  • 5.2 HippoRAG’s Advantage: Single-Step Multi-Hop Retrieval: The top-5 all-recall gap over ColBERTv2 grows from 3% to 6% on MuSiQue and from 20% to 38% on 2WikiMultiHopQA.
  • 5.3 HippoRAG’s Potential: Path-Finding Multi-Hop Retrieval: HippoRAG retrieves passages through graph associations for path-finding questions that ColBERTv2 and IRCoT fail to solve in the illustrated example.

6 Related Work

Prior work uses parametric memory, RAG, long context, and graph structures to support knowledge retrieval and integration, but HippoRAG differs in how it builds and traverses its graph.

  • Parametric Long-Term Memory: LLM parameters encode substantial world knowledge, but updating that knowledge store remains limited.
  • RAG as Long-Term Memory: RAG offers a simple way to update knowledge, while multi-step methods integrate information across new or updated knowledge elements.
  • RAG as Long-Term Memory: RAPTOR, MemWalker, and GraphRAG summarize knowledge during indexing, whereas HippoRAG adds KG edges to continuously integrate new knowledge.
  • Long Context as Long-Term Memory: The viability of storing long-term memory in massive context windows remains uncertain because of engineering hurdles and apparent long-context limitations.
  • Graph-Based Multi-Hop QA: Graph-based multi-hop QA work divides into graph-augmented reading comprehension and graph-augmented retrieval.
  • Graph-Based Multi-Hop QA: HippoRAG builds a knowledge graph from scratch with LLMs and performs unsupervised multi-hop retrieval, making it more adaptable than prior hyperlink-graph methods.

7 Conclusions & Limitations

HippoRAG shows strong knowledge integration and efficiency across multi-hop retrieval while retaining continuously updating memory. Its main remaining boundary is scalability validation and reliance on off-the-shelf components.

  • Conclusions: HippoRAG demonstrates strong path-following multi-hop QA results, promise on path-finding QA, dramatic efficiency improvements, and continuously updating memory.The authors position it as a middle ground between standard RAG and parametric memory.
  • Limitations: HippoRAG currently uses all components off the shelf without additional training, leaving room for component-specific fine-tuning.Error analysis attributes most system errors to NER and OpenIE.
  • Limitations: Graph-search errors could be reduced by replacing simple PPR with approaches that let relations guide graph traversal directly.The authors identify this as one avenue for future improvement.
  • Limitations: OpenIE consistency requires further improvement for longer documents compared with shorter documents.This limitation is reported from the appendix error analysis.
  • Limitations: HippoRAG’s scalability remains insufficiently validated, including the efficiency and efficacy of its synthetic hippocampal index as its size grows.The authors note that Llama-3.1 can reduce costs, but empirical validation of index scalability remains incomplete.

Appendices

The appendices illustrate HippoRAG’s indexing and retrieval pipeline, examine distractor quality, and provide supplementary implementation and evaluation material. The worked example follows a path through extracted entities, an open knowledge graph, and Personalized PageRank.

  • Appendix overview: The supplementary material covers pipeline examples, dataset comparison, ablations, OpenIE evaluation, path-finding case studies, error analysis, efficiency, implementation, and prompts.These topics are listed across Appendices A–I.
  • Pipeline example: Figure 3 shows a question requiring two supporting passages, while distractors concern the district mentioned in the question.The example is drawn from the path-following MuSiQue setting.
  • Indexing: For the worked example, OpenIE extracts triples such as Alhandra’s birthplace and identity relations from supporting passages.The example includes Alhandra being born in Vila Franca de Xira and Lisbon, alongside an alternate name relation.
  • Indexing: NER and OpenIE are sequentially applied to each corpus passage to form an open knowledge graph for the corpus.The indexing figure displays only the relevant subgraph for the example.
  • Retrieval: Retrieval extracts query entities, selects query nodes with a retrieval encoder, and initializes Personalized PageRank from those nodes.In the example, the named entity Alhandra maps directly to its knowledge-graph node.
  • Distractor analysis: HotpotQA distractors are not substantially more similar to questions than the least similar supporting passages, limiting their effectiveness.The distractor and support similarity distributions are compared using Contriever scores.

C Ablation Statistics

The OpenIE ablation compares GPT-3.5 Turbo, REBEL, and Llama-3.1 models through knowledge-graph statistics. REBEL produces substantially fewer extracted nodes and edges, whereas the Llama models are closer to GPT-3.5 Turbo.

  • OpenIE methods: REBEL generates around half as many nodes and edges as GPT-3.5 Turbo and both Llama-3.1 models.The comparison uses knowledge-graph statistics from the OpenIE ablation.
  • OpenIE methods: The results illustrate REBEL’s lower flexibility in open information extraction compared with open- and closed-source LLMs.Both Llama-3.1 versions generate a similar number of OpenIE triples to GPT-3.5 Turbo.

D Intrinsic OpenIE Evaluation

The intrinsic OpenIE evaluation connects extraction quality with retrieval performance but shows that the relationship is imperfect. The appendix also frames path-finding multi-hop questions as especially challenging because they require integrating information across passages.

  • Evaluation setup: The evaluation extracts 239 gold triples from 20 MuSiQue training documents for intrinsic OpenIE assessment.The results are reported using the CaRB framework.
  • Results: The 70B Llama-3.1-Instruct model outperforms its 8B counterpart in both retrieval and intrinsic metrics.The larger model’s intrinsic gains are specifically concentrated in recall.
  • Results: OpenIE recall appears especially relevant to retrieval, although intrinsic evaluation does not perfectly predict retrieval performance.GPT-3.5 Turbo has stronger intrinsic performance than Llama-3.1-70B-Instruct but only a slightly higher retrieval score.
  • Results: End-to-end extraction systems such as REBEL differ substantially from LLM-based OpenIE systems.This distinction is noted in the Table 9 analysis.
  • Path-finding questions: Path-finding multi-hop questions challenge both single-step and multi-step RAG because they require finding relevant entities among many candidates across passages.The examples include identifying Stanford professors working on Alzheimer’s neuroscience.
  • Dataset construction: The curated path-finding questions use traits of books, movies, authors, directors, professors, or drugs to construct supporting and distractor passages.The construction procedure varies across the first two questions and the third question plus motivating example.

E.2 Qualitative Analysis

HippoRAG demonstrates cross-passage retrieval advantages on path-finding questions and exposes tradeoffs among entity-focused indexing, contextual signals, and PPR. Its uncertainty-aware ensemble can improve results, though performance varies by dataset.

  • Qualitative examples: HippoRAG retrieves answers missed by ColBERTv2 and IRCoT in examples involving books, films, and leukemia treatment.It identifies Black Hawk Down within the first four passages and leverages associations to find Chlorambucil.
  • Error analysis: PPR can fail even when NER and OpenIE work correctly because similar passages create confounding signals.PPR cannot directly leverage query context to distinguish appropriate subgraphs in these cases.
  • Concepts vs. Context Tradeoff: HippoRAG’s entity-centric design can ignore contextual signals, while ColBERTv2 sometimes retrieves contextually related passages instead.The tradeoff helps HippoRAG focus on salient entities but can omit information carried by broader context.
  • Concepts vs. Context Tradeoff: Ignoring contextual cues accounts for around 48% of errors in a small-scale error analysis.The issue is more apparent when concepts are general and context is more important.
  • Uncertainty Ensemble: Uncertainty ensembling further improves MuSiQue and outperforms baselines in R@5 on HotpotQA, but can lower performance on some datasets.The ensemble averages normalized HippoRAG and standard passage-retrieval scores.

F.3 OpenIE Limitations

OpenIE is a critical but imperfect component of HippoRAG’s indexing pipeline. Extraction quality deteriorates for longer passages and can omit entities or temporal information needed for retrieval.

  • Extraction errors: OpenIE can omit crucial long entity names and temporal properties, creating knowledge gaps that impair retrieval and question answering.An example misses the song title “Don’t Let Me Wait Too Long” and the war’s beginning and ending years.
  • Passage length: GPT-3.5-Turbo OpenIE results deteriorate substantially when extracting from longer rather than shorter passages.The paper attributes this likely to greater sentence and paragraph complexity.
  • Efficiency: HippoRAG’s online retrieval is 10 to 30 times cheaper than IRCoT because it extracts query entities instead of processing all retrieved documents.This comparison concerns online retrieval over 1,000 queries using GPT-3.5 Turbo.
  • Efficiency: Offline indexing is around 10 times slower and $15 more expensive for every 10,000 passages than IRCoT.The paper notes that open-source LLMs can substantially reduce these indexing costs.
  • Pipeline prompts: The indexing and query NER prompts and the OpenIE prompt are specified separately for HippoRAG’s pipeline.The method constructs RDF triples from passages and named-entity lists during extraction.
Loading 2405.14831v3…