PaperlayerView exact revision on arXiv

Source-linked AI summary

MEMONDEMAND: A Memory Management System for Large-Scale Enterprise Data

Xinyuan Song, Bowen Zhu, Hasibul Haque, Liang Zhao

arXiv:2608.22141v1cs.AI

TL;DR

Enterprise repositories are massive, heterogeneous, and continuously changing, creating a need for retrieval that combines efficiency, source-faithful evidence, and cross-query adaptation. MEMONDEMAND addresses this with a dynamic collection-specific hierarchy, dual routing-and-evidence memory, and on-demand promotion under a bounded active-state budget. It outperforms the published LB#1 result across EnterpriseRAG-Bench scales and performs strongly across financial, multi-hop, and fact-retrieval benchmarks.

  • Problem

    Enterprise memory must scale to very large, changing repositories while preserving source identity, update consistency, and auditable evidence despite the omissions and irrelevance risks of compressed or stale memories.

  • Method

    MEMONDEMAND combines a dynamic multi-level hierarchy, dual memory separating distilled routing from detailed generation-and-citation evidence, and on-demand promotion under a bounded active-state budget.

  • Results

    MEMONDEMAND consistently outperforms the published LB#1 result across EnterpriseRAG-Bench scales and shows strong performance on FinanceBench, HotpotQA, and FRAMES.

  • Takeaways & Limitations

    The results support MEMONDEMAND as an accurate, efficient, and scalable memory solution for very large enterprise repositories across data scales, domains, and evidence requirements.

  • Takeaways & Limitations

    Main scaling and promotion-floor results use single runs, eager preparation cost is estimated rather than measured end-to-end, and Document Recall decreases at the largest scale.

Abstract

from arXiv · show

Enterprise repositories are large, heteroge- neous, and continuously updated, making re- trieval difficult when efficient access, source- faithful evidence, and cross-query adaptation must be supported together. Enterprise mem- ory extends retrieval beyond the model con- text, but existing systems do not jointly address collection-specific hierarchy construction, low- cost routing, detailed evidence loading, and workload-aware memory updates at this scale. We introduce MEMONDEMAND, short for On- Demand Memory, a memory management sys- tem with three coordinated mechanisms: a dy- namic multi-level hierarchy that determines the abstraction structure and depth for each col- lection, dual memory at every hierarchy level that separates distilled routing from detailed evidence, and on-demand memory promotion that updates node priority under a bounded active-state budget. On EnterpriseRAG-Bench, MEMONDEMAND outperforms the strongest published LB#1 result at every evaluated scale from 10M tokens through the complete 618M- token collection, with gains of 12.23% at 10M and 4.66% at 618M. Results on FinanceBench, HotpotQA, and FRAMES further show strong performance across financial, multi-hop, and fact-retrieval settings. Together, these results establish MEMONDEMAND as an accurate, ef- ficient, and scalable memory solution for very large enterprise repositories across data scales, domains, and evidence requirements. Our code is available at https://github.com/ xfab-xinyuansong/MemOnDemand.git.

1 Introduction

Enterprise retrieval must handle massive, heterogeneous, changing repositories while preserving authoritative evidence and adapting memory use across queries. MEMONDEMAND addresses these needs with collection-specific hierarchy construction, dual routing-and-evidence memory, and workload-aware promotion, achieving strong large-scale benchmark results.

  • Motivation: Enterprise repositories combine heterogeneous sources, changing structures, and potentially billions of items, requiring scalable retrieval with exact records and traceable source identities.Policies, contracts, emails, tables, tickets, code, and records can differ in format, authority, department, source, and version.
  • Motivation: Compressed or stale memories may omit answer-critical details, so enterprise memory must preserve authoritative sources while supporting retrieval and reuse beyond the model context.Retrieved memory should not replace source-faithful evidence or auditable source identities.
  • Method: Dual memory separates compact distilled representations for efficient routing from detailed memory used for generation and citation, with both resolving to the same source ID.This balances search and loading cost against source-specific evidence preservation.
  • Method: On-demand promotion updates node priority from observed use, refreshing frequently accessed nodes and demoting stale or low-value nodes under a bounded active-state budget.The mechanism adapts memory state to changing workloads without retaining unlimited active state.
  • Method: MEMONDEMAND constructs a dynamic multi-level hierarchy whose abstraction structure and depth adapt to each collection.The design preserves direct access to L0 source nodes while adapting across domains and tenants.
  • Results: 12.23% at 10M source tokens and 4.66% on the complete 618M-token collection are MEMONDEMAND’s Combined improvements over EnterpriseRAG-Bench’s published LB#1 reference.It also reduces answer-input tokens by 70.2% and makes the persisted hierarchy query-ready in 1.46 seconds rather than an estimated 593.5 seconds for eager preparation.

2 Problem Formulation

The problem formulation models retrieval over a large enterprise collection and a query sequence, requiring accurate, efficient evidence selection and citation under a bounded answer-input context. It further requires adaptive hierarchy, memory, and cross-query state management as the collection changes.

  • Problem Setup: The formulation represents an enterprise collection D of sources and a query sequence Q, with each source retaining its original content and identity.The system may add representations and states to support retrieval while preserving source identity.
  • Problem Setup: For each query q_t, the system retrieves an evidence set E_t and generates an answer a_t with citations C_t.The retrieved evidence is the basis for answer generation and citation.
  • Constraints: Every citation must refer to evidence supplied to the answer model, and selected evidence must remain within the answer-input budget B.The formulation explicitly constrains answer context rather than allowing unlimited retrieved material.
  • Objective: The central objective is accurate and efficient retrieval as the collection grows and changes, using adaptive hierarchy, compact retrieval memory, detailed answering memory, and bounded cross-query updates.Increasing retrieved context alone does not guarantee that decisive evidence will be identified and used.

3 MEMONDEMAND

MEMONDEMAND combines hierarchical retrieval, dual memory representations, and workload-aware promotion into a bounded, source-linked memory pipeline. It constructs collection-specific abstraction levels, routes with distilled memory, loads detailed evidence only for selected sources, and adapts node priority across queries.

  • System Overview: MEMONDEMAND maintains dynamic hierarchy, dual memory, and cross-query promotion state for each memory node.Nodes record hierarchy level, distilled and detailed representations, token cost, child relations, and promotion status.
  • System Overview: Coarse-to-fine retrieval descends from the highest available level to L0, using child links when selected records lack sufficient information.At L0, source records are ranked and provide the basis for evidence loading.
  • 3.2 Dynamic Multi-Level Hierarchy: The hierarchy infers both abstraction structure and depth separately for each collection through clustering proposals and controller decisions to CREATE, RECLUSTER, or STOP.Accepted parent records retain links to all descendant source records.
  • 3.3 Dual Memory at Each Hierarchy Level: Distilled memory supports low-cost hierarchical routing, while detailed memory is loaded only for highest-ranked L0 sources that fit answer-input budget B.Both representations remain linked to the same source, preserving the path from compact retrieval records to answer evidence and citations.
  • 3.4 On-Demand Memory Promotion: Query-time promotion assigns candidate scores, refreshes repeatedly accessed nodes, and demotes stale nodes under an active-state budget K.Demotions prioritize expired nodes and then remove the lowest-retention nodes; all state changes are logged.
  • 3.4 On-Demand Memory Promotion: The bounded promotion process adapts candidate priority across changing workloads without rebuilding the hierarchy or treating promoted nodes as answer evidence.Decay removes stale or low-value nodes when active capacity is reached.

4 Experimental Setup

The evaluation tests MEMONDEMAND on EnterpriseRAG-Bench across seven collection sizes and supplements it with financial, multi-hop, and fact-retrieval benchmarks. The setup uses a common 500-question protocol, reports answer and evidence metrics, and compares against LB#1.

  • 4 Experimental Setup: EnterpriseRAG-Bench evaluates seven collection sizes from 10M through 618M source tokens, with each system answering the same 500 questions.Systems return both answers and the source IDs used to support them.
  • 4 Experimental Setup: FinanceBench, HotpotQA, and FRAMES extend evaluation to financial-document answering, multi-hop question answering, and multi-source fact retrieval and reasoning.These benchmarks cover complementary evidence and reasoning settings beyond enterprise retrieval at scale.
  • 4 Experimental Setup: Combined summarizes answer and evidence quality, while Combined, ∆Combined, Correct, Complete, and Document Recall are reported as percentages.Evidence F1 and Invalid Document Ratio use the [0, 1] scale; higher Evidence F1 and lower InvDoc are better.
  • 4 Experimental Setup: Implementation details cover model assignment, retrieval settings, evaluation rules, failure handling, indexing, and caching in Appendix B and Table 8.

5 Results

MEMONDEMAND consistently outperforms LB#1 across EnterpriseRAG-Bench scales while reducing answer-input cost through selective detailed-memory loading. Ablations and cross-benchmark evaluations support its quality–cost balance and transfer across retrieval settings.

  • Scaling Results: 12.23% at 10M and 4.66% at 618M are MEMONDEMAND’s gains over LB#1 across all seven evaluated corpus tiers.The system outperforms LB#1 at every tier from 10M through the complete 618M-token collection.
  • Token Consumption: 70.2% fewer answer-input tokens at 20M are achieved by selective detailed-memory loading while reaching 78.66% Combined.At 10M, the reduction is 63.9% with an 80.45% Combined score; distilled-only loading uses fewer tokens but substantially lowers quality.
  • Direct Retrieval Comparisons: MEMONDEMAND achieves the highest Combined and Correct scores across both 10M and 20M direct-retrieval comparisons while using less context than detailed-only alternatives.Detailed-only hierarchy retrieval and direct long-context packing consume several times more context but produce lower Combined scores.
  • Component Ablations: Removing deep navigation, promotion, or focused candidate selection lowers Combined from 80.45% to 64.52%, 64.20%, and 62.64%, respectively.The expanded candidate pool can introduce distractors rather than improve evidence quality.
  • On-Demand Promotion: 1.46 seconds versus an estimated 593.5 seconds yields a 407× reduction in up-front preparation time at 20M through on-demand parent promotion.Promotion is activated for 94.0% of the 500-question evaluation set and averages 2.9 hierarchy-navigation steps per query.
  • External Benchmarks: 80.00% Document Recall on FinanceBench, 78.4% Correct on HotpotQA, and 88.76% Document Recall on FRAMES show strong results across external settings.These benchmarks cover financial-document question answering, multi-hop reasoning, and fact retrieval with additional reasoning.

6 Conclusion

MEMONDEMAND combines dynamic hierarchy, dual memory, and on-demand promotion for large-scale enterprise data. It consistently outperforms LB#1 across EnterpriseRAG-Bench scales and transfers across external settings.

  • Conclusion: MEMONDEMAND combines dynamic hierarchy, dual memory, and on-demand promotion for large-scale enterprise data.The design addresses retrieval, memory representation, and cross-query state together.
  • Conclusion: MEMONDEMAND consistently outperforms LB#1 from 10M tokens through the complete 618M-token collection and transfers across different settings.External benchmark results further support performance beyond the primary enterprise benchmark.

Limitations

The paper identifies evaluation uncertainty, estimated preparation costs, reduced large-scale document recall, and limited benchmark breadth as boundaries on its conclusions.

  • Evaluation Scope: Main scaling results and promotion-floor sweeps use single runs, leaving repeated-run variance and confidence intervals unstudied.This limits assessment of statistical variability in the reported results.
  • Evaluation Scope: The eager-preparation cost is estimated from measured call latency rather than measured through a complete end-to-end run.The reported preparation comparison therefore does not come from a full eager execution.
  • Evaluation Scope: Document Recall decreases at the largest scale, while strong source recovery does not always translate into equally strong answer correctness.The paper notes this pattern across the full-scale and external-benchmark evaluations.
  • Future Work: Future evaluation should cover more domains, longer query streams, and broader model and retrieval configurations.These are stated directions for extending the current evaluation scope.

A.3 Database Physical Design

The paper connects enterprise memory design to database physical design, where stable records remain accessible while derived access structures adapt to changing workloads.

  • Database Physical Design: Database systems preserve stable record access while allowing storage layouts, materialized views, and indexing structures to change.The cited database principles motivate adaptable physical organization without changing the underlying record.

B Implementation Details

MEMONDEMAND uses assigned models and a benchmark protocol to evaluate retrieval, answering, and evidence selection across progressively larger enterprise collections. Its dynamic construction determines realized hierarchy sizes for each corpus tier.

  • Evaluation protocol: The evaluation follows ENTERPRISERAG definitions for answers, supporting evidence, and source IDs, checking answers against canonical answers and returned IDs against benchmark evidence.
  • Model assignment: GPT-5.4 mini constructs L0 memory, while GPT-5.4 handles higher-level construction, retrieval, answering, and evaluation.
  • Hierarchy realization: Dynamic construction determines the number of nodes at each hierarchy level for every corpus tier rather than fixing a single hierarchy size.

C Promotion-Floor Sensitivity

Promotion is controlled by a similarity floor that balances noisy over-promotion against insufficient adaptation. Across 10M, 150M, and 618M collections, the best performance occurs at an intermediate floor of 0.5.

  • Promotion control: A candidate node is promoted only when its promotion score meets similarity floor θ, which controls adaptation under query-time evidence.
  • Sensitivity results: 76.40% Combined at 10M, 71.91% at 150M, and 72.88% on the complete collection are achieved with θ = 0.5.
  • Sensitivity results: At 618M, θ = 0.6 reaches 72.86% Combined with 332 promotions, whereas θ = 0.5 records 641 promotions and 618 demotions at the global optimum.
  • Sensitivity results: The promotion-floor sweep shows an inverted-U pattern: excessive or insufficient promotion lowers performance, while an intermediate level performs best.
  • Retrieval procedure: Dual-memory retrieval descends from the highest hierarchy level, merges route signals, ranks candidates, and loads detailed memories only within answer-input budget B.
  • State updates: On-demand promotion updates active state through thresholded promotion, repeated-use refresh, and demotion when nodes expire or exceed the active-state budget.

E System Properties

The system properties formalize complementary retrieval, distinct-source coverage, answer-input accounting, and deferred preparation. These analyses also identify limits of direct retrieval and distinguish cost guarantees from answer-correctness guarantees.

  • Complementary retrieval: Hierarchical retrieval improves recall only when it recovers a source missed by direct L0 retrieval, so MEMONDEMAND combines rather than replaces the two routes.
  • Distinct-source coverage: Scanning until k distinct sources are collected cannot reduce source coverage relative to deduplicating the first k hits under the same budget.
  • Token accounting: Selective detailed-memory loading saves answer-input tokens because distilled-memory tokens support retrieval without entering the answer-input budget.
  • Deferred preparation: On-demand preparation cannot cost more than preparing every deferred representation in advance over the same query horizon, but this is not an end-to-end latency guarantee.
  • Scaling behavior: Document Recall and Correct generally decrease with collection size for global source-leaf retrieval without promotion.
  • Token use: Answer input is the largest online token component, while retrieval-control and answer-output costs account for smaller shares.

F.3 External visualization Results

Selective evidence loading preserves strong answer quality while reducing answer-input cost, and on-demand promotion reduces preparation before querying. External benchmarks show that source recovery and answer correctness vary with task evidence and reasoning demands.

  • Quality and cost: Selective detailed-memory loading preserves strong Combined performance while using substantially fewer tokens than detailed-only or combined-representation loading.
  • Component sensitivity: The full system performs best when multi-step navigation, on-demand promotion, and controlled candidate selection operate together.
  • Preparation cost: 1.46 seconds is the measured cold-load time for on-demand promotion, whereas eager parent preparation is estimated before query execution.
  • External benchmarks: Source recovery and answer correctness vary across FinanceBench, HotpotQA, and FRAMES because the tasks have different evidence structures and reasoning requirements.
  • Memory construction: Source-leaf distilled records preserve topics, entities, dates, identifiers, and decision-bearing facts while retaining source IDs.
  • Hierarchy construction: Parent abstraction may create, recluster, or stop based on coherence, residual noise, group statistics, representative content, and construction budget.
  • Navigation and answering: Navigation uses distilled routing records to return candidate source IDs, while detailed evidence is reserved for selected sources.
Loading 2608.22141v1…