PaperlayerView exact revision on arXiv

Source-linked AI summary

From RAG to Memory: Non-Parametric Continual Learning for Large Language Models

Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, Yu Su

arXiv:2502.14802v2cs.CLcs.AI

TL;DR

Continual learning with LLMs is limited by difficulties in updating parametric knowledge and by vector RAG’s weak modeling of interconnected memory. HippoRAG 2 combines Personalized PageRank, deeper passage integration, and online LLM filtering, achieving comprehensive gains over standard RAG across factual, sense-making, and associative tasks.

  • Problem

    LLMs face challenges absorbing new knowledge without catastrophic forgetting, while vector-based RAG poorly captures the dynamic and interconnected nature of long-term memory.

  • Method

    HippoRAG 2 combines Personalized PageRank with deeper passage integration, deeper query-to-triple contextualization, and online LLM recognition memory.

  • Results

    HippoRAG 2 consistently improves over standard RAG, including a 7 point average gain in associativity with no deterioration and slight improvements in factual memory and sense-making.

  • Takeaways & Limitations

    The results position HippoRAG 2 as a promising step toward non-parametric continual learning and more human-like long-term memory for LLMs.

  • Takeaways & Limitations

    HippoRAG’s predecessor is hindered by entity-centric processing that causes context loss during indexing and inference and creates semantic-matching difficulties.

Abstract

from arXiv · show

Our ability to continuously acquire, organize, and leverage knowledge is a key feature of human intelligence that AI systems must approximate to unlock their full potential. Given the challenges in continual learning with large language models (LLMs), retrieval-augmented generation (RAG) has become the dominant way to introduce new information. However, its reliance on vector retrieval hinders its ability to mimic the dynamic and interconnected nature of human long-term memory. Recent RAG approaches augment vector embeddings with various structures like knowledge graphs to address some of these gaps, namely sense-making and associativity. However, their performance on more basic factual memory tasks drops considerably below standard RAG. We address this unintended deterioration and propose HippoRAG 2, a framework that outperforms standard RAG comprehensively on factual, sense-making, and associative memory tasks. HippoRAG 2 builds upon the Personalized PageRank algorithm used in HippoRAG and enhances it with deeper passage integration and more effective online use of an LLM. This combination pushes this RAG system closer to the effectiveness of human long-term memory, achieving a 7% improvement in associative memory tasks over the state-of-the-art embedding model while also exhibiting superior factual knowledge and sense-making memory capabilities. This work paves the way for non-parametric continual learning for LLMs. Code and data are available at https://github.com/OSU-NLP-Group/HippoRAG.

1. Introduction

Continual learning remains difficult for LLMs because parametric updates risk incomplete absorption and catastrophic forgetting, while standard RAG lacks interconnected memory capabilities. HippoRAG 2 addresses this robustness gap through deeper contextual retrieval and online LLM use, improving performance across factual, sense-making, and associative tasks.

  • LLMs struggle to absorb evolving knowledge and avoid catastrophic forgetting because of the complex distributional nature of parametric knowledge.
  • Structure-augmented RAG methods target sense-making and associativity by linking disparate passages through summaries or knowledge graphs.
  • Previous structure-augmented methods underperform the strongest embedding-based RAG systems across factual, sense-making, and associative benchmarks.
  • HippoRAG 2 integrates passages into Personalized PageRank graph search, deepens query-to-triple selection, and uses an LLM online to filter irrelevant triples.
  • 7 point average improvement over standard RAG in associativity accompanies no deterioration and slight improvements in factual memory and sense-making.

2. Related Work

Continual learning methods include parameter updates, model editing, and retrieval-augmented generation. Recent RAG systems add graph structures for sense-making and associativity, but HippoRAG 2 uses its knowledge graph to support retrieval while limiting generated noise.

  • Continual Learning: Continual learning approaches generally comprise continual fine-tuning, model editing, and retrieval-augmented generation.Fine-tuning can incorporate new patterns and reasoning skills but risks catastrophic forgetting, while model editing updates are highly localized.
  • Retrieval-Augmented Generation: RAG provides scalable, practical continual learning by retrieving external information at inference time without modifying the language model.
  • Retrieval-Augmented Generation: Improved encoder models, including NV-Embed-v2, enhance RAG retrieval by producing embeddings that better capture semantic relationships.NV-Embed-v2 is the primary comparison model in this paper.
  • Sense-Making: Standard RAG struggles with sense-making because it must integrate information from disparate passages, motivating frameworks such as RAPTOR, GraphRAG, and LightRAG.
  • HippoRAG 2: HippoRAG 2 uses its knowledge graph to aid retrieval rather than expand the retrieval corpus, introducing less LLM-generated noise than GraphRAG and LightRAG.The passage links this design to avoiding deterioration in single- and multi-hop question answering.
  • Associativity: Associativity requires multi-hop connections between disparate facts, which standard RAG cannot emulate through independent vector retrieval.HippoRAG addresses this property using Personalized PageRank over an explicitly constructed open knowledge graph.

3. HippoRAG 2

HippoRAG 2 extends a neurobiologically inspired memory framework with an open knowledge graph that integrates conceptual and contextual information. Its online retrieval links queries to graph nodes, filters candidate triples with recognition memory, and applies Personalized PageRank for context-aware retrieval.

  • Framework: HippoRAG models long-term memory with an LLM neocortex, a knowledge graph and Personalized PageRank hippocampus, and a retrieval encoder.
  • Framework: HippoRAG 2 retains offline indexing and online retrieval while refining the framework to integrate conceptual and contextual information within the knowledge graph.
  • Offline Indexing: Offline indexing uses schema-free OpenIE triples, synonym detection among phrase nodes, and the original passages to form an open knowledge graph.Phrase nodes are the subjects or objects of extracted triples, and relation edges connect them.
  • Online Retrieval: Online retrieval uses embedding scores to select seed nodes, LLM recognition memory to filter triples, and Personalized PageRank to perform context-aware graph retrieval.When no triples are available, the system directly retrieves top-ranked passages using the embedding model.
  • Offline Indexing: The open knowledge graph addresses the concept-context tradeoff by combining sparse phrase representations for concepts with dense representations of source context.Concepts are concise and generalizable, whereas context preserves specific circumstances that shape interpretation.
  • Online Retrieval: HippoRAG 2 reduces HippoRAG’s concept-centric bias by linking queries to both relevant triples and passages rather than relying only on named entities.
  • Online Retrieval: Query-to-triple retrieval first obtains top-k graph triples with an embedding model, then uses an LLM to retain a filtered subset.

4. Experimental Setup

The evaluation compares simple, dense, large embedding, and structure-augmented retrieval baselines across factual, associative, and sense-making memory tasks using QA and retrieval metrics.

  • Baselines: The study compares BM25, Contriever, GTR, three 7B embedding models, and four structure-augmented RAG methods.The structure-augmented methods are RAPTOR, GraphRAG, LightRAG, and HippoRAG.
  • Datasets: The benchmark covers simple QA for factual recall, multi-hop QA for associativity, and discourse understanding for sense-making.Simple QA uses individual-entity questions; multi-hop QA requires connecting multiple passages; discourse understanding uses lengthy narratives.
  • Datasets: Simple QA samples 1,000 queries each from NaturalQuestions and PopQA, while multi-hop QA samples queries from MuSiQue, 2WikiMultihopQA, HotpotQA, and LV-Eval.LV-Eval contributes all 124 queries and is designed to reduce knowledge leakage and overfitting through keyword and phrase replacements.
  • Datasets: Discourse understanding uses 10 lengthy NarrativeQA documents with 293 corresponding queries to test cohesive reasoning over full-length novels.The retrieval corpus is collected similarly to the LV-Eval setup.
  • Evaluation: Evaluation uses passage recall@5 for retrieval and token-based F1 scores for QA.The QA metrics follow the evaluation procedure from MuSiQue.
  • Implementation: HippoRAG 2 uses Llama-3.3-70B-Instruct for extraction and triple filtering and NV-Embed-v2 as its retriever, with the same extractor and retriever used for reproduced structure-augmented baselines.DSPy’s MIPROv2 optimizer tunes the triple-filter prompt.

5. Results

HippoRAG 2 achieves the strongest overall QA and retrieval performance in the reported experiments, improving complex-task results while avoiding deterioration on simpler factual tasks.

  • Evaluation: The reported experiments use Llama-3.3-70B-Instruct as the QA reader, while Table 2 evaluates F1 and Table 3 evaluates passage recall@5.The tables reproduce structure-augmented baselines with the same LLM and retriever for fair comparison.
  • QA Performance: HippoRAG 2 achieves the highest average QA F1 score and remains robust across different benchmark settings.It outperforms NV-Embed-v2 by 9.5% F1 on 2Wiki and 3.1% on LV-Eval.
  • QA Performance: HippoRAG 2 consistently outperforms other methods across nearly all tested QA settings, including with Llama and GPT-4o-mini readers.NV-Embed-v2 generally outperforms structure-augmented methods except for HippoRAG in multi-hop QA.
  • Retrieval Performance: HippoRAG 2 achieves the highest recall scores across most retrieval datasets, improving Recall@5 by 5.0% on MuSiQue and 13.9% on 2Wiki over NV-Embed-v2.The comparison uses the strongest dense retriever as the reference baseline.

6. Discussions

The discussions examine ablations, continual learning under corpus expansion, retriever robustness, and qualitative retrieval examples for HippoRAG 2.

  • Ablation Study: 12.5% average Recall@5 improvement comes from query-to-triple linking over NER-to-node linking.The query-to-triple strategy consistently outperforms the other linking approaches, while query-to-node offers no advantage over NER-to-node.
  • Ablation Study: A 0.05 passage-node reset factor balances phrase and passage nodes during Personalized PageRank.Validation results identify this factor as crucial for PPR performance across scenarios.
  • Robustness to Corpus Expansion: HippoRAG 2’s improvements over NV-Embed-v2 remain consistent in simple NQ and associative MuSiQue continual-learning settings.Both methods retain strong simple-QA performance as knowledge is added, while associative-task performance degrades at similar rates.
  • Retriever Robustness: HippoRAG 2 consistently surpasses direct dense retrieval across different dense retrievers.Table 7 reports passage Recall@5 and shows that the gains remain robust regardless of the retriever used.
  • Qualitative Analysis: Qualitative examples show HippoRAG 2 retrieving passages that support direct answers and multi-hop reasoning more effectively than NV-Embed-v2.The examples include identifying Thrissur directly and retrieving Montebello for a two-step geographic question.

7. Conclusion

The conclusion presents HippoRAG 2 as a framework for approximating the dynamic and interconnected nature of human long-term memory in LLMs.

  • Conclusion: HippoRAG 2 combines Personalized PageRank, deeper passage integration, and online LLM use to improve factual, sense-making, and associative memory tasks.The paper identifies this combination as opening avenues for continual-learning and long-term-memory research.

Impact Statement

The impact statement reports no concerns requiring specific emphasis beyond those generally associated with LLMs and information retrieval systems.

  • Impact Statement: The authors identify no societal concerns requiring specific emphasis beyond those generally associated with LLMs and information retrieval systems.

Appendices

The supplementary material documents prompts, pipeline examples, experiments, graph statistics, error analysis, cost, and implementation details.

  • Appendices: Appendix A provides LLM prompts, including triple-filter instructions, demonstrations, and input format.
  • Appendices: Appendix B provides a HippoRAG 2 online-retrieval pipeline example covering query-to-triple, triple filtering, and seed nodes for PPR.
  • Appendices: Appendices C through E cover detailed experimental results, graph statistics, and error analysis.
  • Appendices: Appendix F discusses cost and efficiency, while Appendix G documents implementation details and hyperparameters.

C. Detailed Experimental Results

Using GPT-4o-mini for indexing and QA, HippoRAG 2 shows competitive or superior QA performance across most datasets and a similar recall@2/recall@5 improvement trend.

  • QA Performance: HippoRAG 2 consistently achieves competitive EM and F1 scores across most datasets when GPT-4o-mini performs indexing and QA reading.It leads on MuSiQue and 2Wiki and shows superior performance on NarrativeQA and LV-Eval.
  • QA Performance: HippoRAG 2 matches or improves F1 relative to NV-Embed-v2, particularly on LV-Eval with reduced knowledge leakage.
  • Retrieval Performance: The recall@2 improvement trend is similar to the recall@5 improvement trend.
  • Knowledge Graph Statistics: Table 10 reports knowledge-graph statistics for OpenIE using Llama-3.3-70B-Instruct or GPT-4o-mini.The statistics use different LLMs for OpenIE.

E. Error Analysis

Error analysis identifies triple filtering and graph search as the main error sources, while the pipeline combines dense retrieval, graph construction, PPR ranking, and downstream QA under substantial computational requirements.

  • Error Analysis: 26%, 41%, and 33% of low-recall samples are classified as 2-hop, 3-hop, and 4-hop questions, respectively.Triple filtering and graph search are the two main error sources.
  • Recognition Memory: 18% of samples retain zero triples after filtering, and 26% have no supporting-document phrase matched after filtering.Filtering can eliminate relevant phrases, although zero triples do not necessarily indicate a filtering error.
  • Recognition Memory: HippoRAG 2 may substitute dense-retrieval results when triple filtering produces no usable triples.
  • Graph Construction: Only 2% of samples lack supporting-passage phrases within one-hop neighbors of linked nodes during graph construction.The authors assume dense-sparse integration generally preserves potentially exploitable information.
  • Personalized PageRank: 50% of samples have at least half of their linked phrase nodes in supporting documents, yet graph search can still fail to return perfect top-5 passages.
  • Efficiency: HippoRAG 2 outperforms the compared RAG methods in QA and retrieval while using fewer tokens than LightRAG and GraphRAG.Its fact embeddings increase memory requirements, creating an explicitly acknowledged tradeoff.
  • PPR Retrieval Pipeline: HippoRAG 2 initializes PPR with phrase and passage nodes, assigns reset probabilities, ranks passage nodes, and sends top passages to QA reading.Phrase nodes come from filtered triples, while passage nodes use embedding similarity scores.

G.2. Comparison Methods

The comparison uses consistent QA evaluation while preserving default indexing settings for GraphRAG and LightRAG and tuning QA hyperparameters on matched samples.

  • Comparison Setup: GraphRAG and LightRAG retain their default hyperparameters and prompts for a consistent comparison.
  • Comparison Setup: The same HippoRAG 2 QA prompt rephrases GraphRAG and LightRAG responses during evaluation.
  • Comparison Setup: QA hyperparameters for GraphRAG and LightRAG are tuned on the same 100 samples used in Appendix G.1.
Loading 2502.14802v2…