PaperlayerView exact revision on arXiv

Source-linked AI summary

MegaMem: A Retrieval Solution for Ultra-Large Context Windows

Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao

arXiv:2608.22137v1cs.AI

TL;DR

MegaMem addresses the challenge of searching hundreds of millions of persistent-memory tokens while passing only bounded source evidence to the answer model. It uses source-resolved dual-view retrieval to separate semantic access from generation evidence, and improves EnterpriseRAG-Bench performance while keeping generation context bounded. The paper also identifies answerability calibration and transfer limitations for its current design.

  • Problem

    Ultra-large memory systems must keep complete codebases, long histories, and heterogeneous records searchable while limiting the detailed evidence passed for each answer.

  • Method

    MegaMem searches distilled records and detailed evidence with multiple query routes, resolves hits to immutable source IDs, fuses and reranks candidates, and generates from highest-ranked detailed evidence under a fixed budget.

  • Results

    MegaMem improves Overall from 68.22 to 82.26 and reaches 86.50 Correctness on EnterpriseRAG-Bench, while keeping generation context bounded as persistent memory grows.

  • Takeaways & Limitations

    Separating searchable memory scale from answer-context size provides a practical path toward retrieval over memories ranging from hundreds of millions to one billion tokens.

  • Takeaways & Limitations

    Source precision remains below the published EnterpriseRAG-Bench leader, transfer is non-uniform across conversational and broad cross-domain stores, and the method does not extend the model's native attention window.

Abstract

from arXiv · show

Modern language models and agents increasingly require persistent memory for complete codebases, long interaction histories, and heterogeneous enterprise records. The key challenge is to keep hundreds of millions of tokens searchable while passing only bounded source evidence to the answer model. We introduce MegaMem, a source-resolved dual-view retrieval system that separates semantic access from generation evidence. Distilled records and detailed evidence are searched with original and transformed queries; every distilled hit resolves to an immutable source ID before reciprocal-rank fusion, deduplication, and cross-encoder reranking; and only the highest-ranked detailed evidence within a fixed budget supports generation. Post-answer attribution then identifies which loaded sources support the fixed answer. We evaluate MegaMem on EnterpriseRAG-Bench, which contains more than 500,000 heterogeneous enterprise documents and approximately 650M tokens. MegaMem improves Overall from 68.22 to 82.26 and reaches 86.50 Correctness. These results show that MegaMem supports ultra-large persistent memory while preserving strong answer accuracy under a bounded generation context. By separating searchable memory scale from answer-context size, MegaMem provides a practical path toward accurate retrieval over memories ranging from hundreds of millions to one billion tokens. Our code is available at https://github.com/ xfab-xinyuansong/MegaMem.git.

1 Introduction

MegaMem addresses ultra-large persistent memory by separating searchable memory from bounded evidence used for generation. Its source-resolved dual-view pipeline improves enterprise retrieval quality while keeping generation context bounded.

  • Large-context systems face missed evidence, distraction, declining performance, and rising attention, latency, and serving costs as repositories grow.
  • Compressed records improve access but may omit dates, exceptions, conditions, or conflicts, while detailed-only retrieval increases distractor exposure and weakens precision.
  • MegaMem separates semantic access from generation evidence by resolving distilled hits to immutable detailed-source identifiers before fusion, reranking, and generation.
  • Overall rises from 68.22 to 82.26 and Correctness reaches 86.50 on EnterpriseRAG-Bench's approximately 650M-token enterprise corpus.
  • Ultra-large memory retrieval must keep 250M to 1B tokens searchable while passing only bounded evidence to the answer model.
  • MegaMem searches both distilled and detailed views with original and transformed queries, then applies reciprocal-rank fusion, deduplication, and cross-encoder reranking.
  • MegaMem achieves the best overall performance while keeping generation context bounded as persistent memory grows.

2 Problem Formulation

MegaMem formalizes persistent context as the full searchable corpus and evidence context as a bounded subset used for one answer. Its source-resolved retrieval ensures compressed representations locate content while detailed evidence supports generation.

  • 2.1 Persistent Context and Evidence Context: D denotes the full persistent corpus with T tokens, while E(q) denotes the evidence set retrieved for question q.
  • 2.1 Persistent Context and Evidence Context: The evidence set satisfies E(q) ⊆ D and tok(E(q)) ≤ B, with B ≪ T.
  • 2.1 Persistent Context and Evidence Context: Persistent context is the full searchable corpus, whereas evidence context is the bounded detailed source evidence consumed for one answer.
  • 2.1 Persistent Context and Evidence Context: The formulation aims to scale the searchable corpus while keeping evidence context bounded despite increasing retrieval difficulty from additional distractors.
  • 2.2 Source-Resolved Retrieval: Compressed or structured representations are used to locate relevant content, not to provide the evidence from which answers are generated.
  • 2.2 Source-Resolved Retrieval: Every compressed hit resolves to its corresponding detailed source chunk before fusion, reranking, and generation.
  • 2.2 Source-Resolved Retrieval: X is the set of detailed source chunks used as the source-resolved evidence space.

3 Method

MegaMem separates semantic retrieval from generation evidence through dual-view memory, source resolution, multi-route ranking, and bounded evidence packing. Post-answer attribution filters reported sources after the answer is fixed without changing that answer.

  • 3.1 Dual-View Memory Construction: MegaMem constructs detailed evidence chunks and up to three compact typed memories per chunk, linked through source pointers.Memory types include fact, procedure, definition, requirement, and decision.
  • 3.1 Dual-View Memory Construction: The detailed index preserves exact wording and provenance, while the distilled index optimizes semantic matching over typed memories.Both views refer to the same underlying evidence.
  • 3.2 Multi-Route Retrieval and Evidence Resolution: Original, canonical, and terminology-diverse queries search both indices, while distilled hits resolve to detailed chunks before candidate aggregation.This improves recall without treating compressed memories as independent evidence items.
  • 3.2 Multi-Route Retrieval and Evidence Resolution: Weighted reciprocal-rank fusion combines retrieval lists, after which a cross-encoder reranks candidates and deduplication merges them by detailed source identity.The highest-ranked detailed chunks are packed until the evidence budget B is reached.
  • 3.3 Bounded Generation and Post-Answer Attribution: Only selected detailed chunks are exposed to the answer model; attribution subsequently identifies supporting loaded sources without revising the generated answer.No distilled memory or unselected source is exposed during generation.
  • 3.3 Bounded Generation and Post-Answer Attribution: Removing attribution leaves answer quality unchanged but increases the average number of reported documents from 2.31 to 5.00.This ablation indicates that attribution primarily improves source precision rather than answer correctness.

4 Experimental Setup

MegaMem is evaluated primarily on EnterpriseRAG-Bench using a frozen validation split, standard retrieval and answer-quality baselines, and additional transfer datasets. The setup measures correctness, completeness, aggregate answer quality, document recall, and unsupported-document reporting.

  • Benchmarks and protocols: MegaMem is additionally evaluated on FinanceBench, HotpotQA, LoCoMo, and UltraDomain for transfer assessment.Results should be compared only within matched settings because protocols, question sets, memory scales, and extractors differ.
  • Baselines and metrics: Baselines include BM25, DPR, Contriever, hierarchical retrieval, graph-based retrieval, and memory-guided retrieval methods.The comparison spans lexical, dense, hierarchical, graph-based, and memory-guided approaches.
  • Baselines and metrics: Correctness measures scoped and condition-preserving conclusions, while Completeness measures coverage of reference-answer facts and Overall aggregates both.Document Recall measures recovered gold sources, and InvDocs measures invalid or unsupported reported documents, with lower values indicating better source precision.
  • Baselines and metrics: Atomic extraction uses gpt-5.4-mini, higher-level abstraction and downstream language tasks use gpt-5.4, and dense indices use text-embedding-3-small.Appendix B provides the full protocol and implementation details.

5 Results

MegaMem remains effective as persistent memory scales to hundreds of millions of tokens under fixed retrieval and evidence budgets. It outperforms direct-retrieval baselines while reducing answer-context requirements and exposing component, transfer, and operational trade-offs.

  • Scaling to Ultra-Large Persistent Memory: 58.02 Overall and 73.50 Correctness at 250M tokens show MegaMem remains useful as persistent memory grows under fixed budgets.Correctness remains above 73% and Document Recall above 66% from 20M to 250M tokens.
  • Comparison with Direct Retrieval Baselines: MegaMem achieves the best Overall, Correctness, Completeness, and Document Recall against direct retrieval and compressed-memory baselines at 10M tokens.Relative to the published benchmark leader, it improves Overall by 20.58%, Correctness by 6.00%, Completeness by 19.38%, and Document Recall by 3.64%.
  • Component Analysis: Post-answer attribution reduces InvDocs from 0.774 to 0.760 without changing answer quality, while removing it increases reported documents from 2.31 to 5.00.The module removes 53.8% of the retrieved source set without changing the generated answer.
  • Component Analysis: Removing the dual index reduces Overall by 17.91% and Document Recall from 81.90% to 70.20%, while removing distillation reduces Overall by 14.15%.Query expansion and reranking add 3.55% and 4.97%, respectively.
  • Efficiency and Transfer: Selective detail reduces answer input by 68.6% at 10M and 68.8% at 20M, with only a 1.5-point Correctness reduction.Distilled-only evidence is shorter but loses much of the information needed for accurate generation.
  • Efficiency and Transfer: Query expansion improves answerable categories by 20.8–100.0% but reduces information-not-found Correctness from 100.0% to 68.8%.A selective oracle restores information-not-found Correctness to 100.0%, identifying calibrated answerability gating as the remaining requirement.

6 Conclusion

MegaMem is a retrieval solution for ultra-large persistent memory that keeps hundreds of millions of tokens searchable while maintaining a bounded generation context. Experiments show improved retrieval and answering quality at scales up to 250M tokens.

  • MegaMem uses distilled memories for semantic access and resolves retrieved content to detailed source evidence before generation.This design separates searchable memory scale from the evidence passed to the answer model.
  • MegaMem keeps hundreds of millions of tokens searchable while maintaining a bounded generation context.
  • MegaMem improves retrieval and answering quality while remaining effective at scales up to 250M tokens.

Limitations

MegaMem’s scope is limited by evaluation variability, source-precision gaps, transfer limitations, and the native attention window of its underlying language model.

  • Most reported configurations use single runs, and detailed-evidence-only packing is estimated from a retained trace rather than a controlled rerun.
  • Source precision remains lower than the published EnterpriseRAG-Bench leader.
  • Performance on LoCoMo and UltraDomain does not transfer uniformly to conversational or broad cross-domain memory stores.
  • MegaMem increases searchable context but does not extend the underlying language model’s native attention window.

B Full Experimental Protocol

The experimental protocol separates development from validation and uses controlled diagnostics to evaluate retrieval degradation, architecture changes, and source-resolution design choices.

  • Implementation: The backend uses separate persistent stores for 60M, 100M, and approximately 250M-token memory scales.
  • Retrieval diagnosis: Gold-evidence Correctness changes by 0.30% with MiniExtractor and 2.33% with FullExtractor from 20M to 250M tokens, versus 12.50% and 19.93% with retrieved evidence.
  • Architecture development: The architecture develops from compressed hierarchical and relational memory toward dual-view retrieval with reranking, query expansion, and attribution.
  • Protocol: The 100-question development split is used for architecture, prompt, and hyperparameter selection, while remaining analyses preserve separate protocols.
  • Retrieval diagnosis: EnterpriseRAG-Bench validation uses retrieved and gold evidence across memory scales to distinguish retrieval failure from generation failure.
  • Architecture development: The key architectural change is the transition from compressed-only memory to source-resolved dual-view retrieval.

E Offline Cost, Online Latency, and Evidence-Context Efficiency

MegaMem separates offline memory construction from online serving and limits answer-model evidence to reduce context while preserving answer quality.

  • Cost and latency: Memory construction becomes more expensive as the corpus grows, but it is performed offline.
  • Cost and latency: Online retrieval remains below 1.1 seconds at 100M tokens, while end-to-end answering remains below 3.3 seconds.
  • Evidence efficiency: MegaMem reduces answer context by approximately 69% relative to full detailed evidence while sacrificing only 1.5 Correctness points.
  • Evidence efficiency: Distilled-only evidence is shorter but performs substantially worse, whereas selective detailed evidence preserves nearly all Correctness of full detailed evidence.
  • Workload: The system retrieves approximately 34 chunks before reranking and evidence packing in the full 500-question diagnostic.

F Cross-Dataset Transfer and Query-Expansion Diagnostics

This section examines transfer across retrieval regimes and query expansion across question types. Query expansion improves answerable cases but requires answerability-aware handling for information-not-found questions.

  • Retrieval and attribution effects: Query expansion recovers 13 of 33 targets missed by the base retriever, a 39.4% recovery rate.
  • Retrieval and attribution effects: Post-answer attribution reduces the average reported source set from 5.00 to 2.31 documents without changing the answer.
  • Query expansion and answerability: Query expansion improves Correctness across all five answerable question categories, with relative gains ranging from 20.8% to 100.0%.The largest gains occur on project-related and high-level questions, where terminology mismatch and indirect references make direct retrieval difficult.
  • Query expansion and answerability: The evaluation uses 500 EnterpriseRAG-Bench questions, with question-type Correctness reported before and after query expansion.Table 14 evaluates all 500 questions at 10M with FullExtractor.
  • Query expansion and answerability: Unconditional expansion reduces information-not-found Correctness from 100.0% to 68.8% by answering five of 16 unanswerable cases.A selective oracle restores information-not-found Correctness to 100.0%, indicating that expansion requires an answerability-aware gate.

G Memory Construction, Inference Order, and Reproducibility

MegaMem constructs linked distilled and detailed memory views, then resolves, fuses, reranks, and packs bounded detailed evidence for answer generation and attribution. The experiments use fixed reproducibility settings and explicit failure fallbacks.

  • Memory construction: Each distilled memory stores the immutable identifier of its detailed source chunk, keeping both indices linked to the same evidence.
  • Memory construction: Algorithm 2 builds a detailed index, a distilled index, and a source map from documents, extracted memories, and embeddings.
  • Inference order: Original and transformed queries retrieve from both indices before distilled hits resolve to detailed chunks for fusion, reranking, and evidence packing.
  • Inference order: The answer is generated from bounded detailed evidence, after which attribution selects only loaded sources supporting the fixed answer.
  • Reproducibility: The evidence context is capped at 4,096 tokens, generated answers at 800 tokens, and reranked candidates at five unique documents.RRF uses γ = 60, and the context-packing study fixes first-stage candidate depth at 20.
  • Failure handling: After all retries fail, extraction returns no memories, query transformation falls back to the original question, attribution returns an empty set, and answering returns a predefined abstention response.

H Visualization of Main Results and Diagnostics

The visual diagnostics compare end-to-end systems, component ablations, query-expansion behavior, and scaling mechanisms. Together, the cited results emphasize complementary retrieval components, answerability-aware expansion, and retrieval as the main scaling bottleneck.

  • Main system comparison: MegaMem achieves the strongest overall result among direct retrieval baselines and compressed-memory variants at the 10M memory scale.It also maintains competitive document recall and a lower invalid-document ratio.
  • Component ablation: Removing any major component reduces Overall, indicating complementary contributions from dual-view memory, reranking, query expansion, and attribution.
  • Query-expansion behavior: Query expansion improves answerable categories, but information-not-found questions require an answerability-aware gate to prevent plausible unsupported evidence.
  • Scaling interpretation: Dual-view retrieval improves evidence recall, while larger-memory performance degradation is driven mainly by retrieval.
  • Scaling interpretation: As memory grows, the changing factor is the probability of selecting sufficient evidence, while Correctness remains stable when required evidence is provided.This supports retrieval, rather than answer generation, as the primary source of observed scaling loss.
  • Prompt templates: The prompt templates specify typed-memory extraction, high-level abstraction, query transformation, candidate selection, evidence-only answering, attribution, and evaluation procedures.
Loading 2608.22137v1…