PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Complex Agentic Memory Always Needed? Naive RAG Is Stronger Than We Thought

Zhengye Bao, Shengtao Zhang, Haokai Zhang, Shiqi Lou, Muning Wen, Minnan Luo

arXiv:2610.30755v1cs.AI

TL;DR

Long-term conversation memory systems often add mutation, structural organization, or learned retrieval controls, raising whether that complexity is necessary. This paper evaluates RAW-RAG, which retrieves and reranks raw history for a reader, and finds it ranks first in most evaluated settings. The findings challenge the need for complex agentic memory in the conversation tasks evaluated.

  • Problem

    Long-term interactions require access to information accumulated across conversations, and it remains an open question whether agentic memory complexity is necessary.

    Source: §1 · p. 2

  • Method

    RAW-RAG applies dense retrieval and cross-encoder reranking to raw conversation history, supplying selected evidence to a reader LLM.

    Source: §1 · p. 3

  • Results

    RAW-RAG ranks first among compared memory systems in six of seven benchmark–model settings across four benchmarks.

    Source: §1 · p. 3, §6.2 · p. 10

  • Takeaways & Limitations

    The findings challenge the need for complex agentic memory in the evaluated conversation tasks and establish naive RAG as an essential baseline.

    Source: §6.2 · p. 10

    Evidence is limited to offline conversation on public benchmarks and does not establish performance in interactive, multimodal, or broader operational settings.

    Source: §6.1 · p. 10

Abstract

from arXiv · show

LLM memory is widely used in long-term conversation understanding. Agentic memory systems often use mutation, structural organization, learned retrieval strategies, or other complex operations. Is this complexity always necessary? We study Raw-RAG, a naive RAG baseline using dense retrieval and reranking over raw history to supply evidence to a reader LLM. Across LoCoMo, LongMemEval, MEME, and MemoryAgentBench, Raw-RAG ranks first among the compared memory systems in six of seven benchmark-model settings, outperforming the strongest compared agentic memory system by 0.13-5.53 points. We investigate why raw-history retrieval remains competitive. Evidence analysis shows that its higher coverage of necessary evidence can offset lower utilization. Useful evidence includes both answer values and the relations and dependency paths needed to derive them. Even complete access to transformed memory stores leaves an accuracy gap relative to raw evidence, highlighting the importance of preserving raw information. These findings challenge the necessity of complex agentic memory for the evaluated conversation tasks and establish naive RAG as an essential baseline.

1. Introduction

Long-term conversation histories can exceed an LLM’s context window, motivating memory systems that store, organize, and retrieve past information. The paper tests whether raw-history RAG can match agentic systems and examines whether explicit state maintenance is needed.

1. Introduction

  • Effective text-changing mutations account for only 1.90% of Mem0’s LoCoMo write decisions and 1.34% of its LongMemEval decisions.UPDATE and DELETE together account for 4.39% and 2.46% of decisions, respectively; many proposed updates are duplicates or malformed.Source: §1 · pp. 2–3
  • Disabling explicit mutation improves LoCoMo accuracy from 67.36% to 69.42% in the tested configuration, without reducing any category score.The ADD/NONE-only policy replaces UPDATE with ADD and DELETE with NONE.Source: Table 1 · p. 3, §1 · p. 3
  • Raw evidence can support difficult dynamic-state reasoning: readers given phase-visible evidence reached 82.32% on cascades and 91.54% on invalidation.The MEME diagnostic supplies evidence in chronological order, eliminating retrieval errors while leaving readers to resolve state changes.Source: §1 · p. 3, Table 2 · p. 3
  • RAW-RAG ranks first among compared memory systems in six of seven benchmark–model settings, making it a strong baseline for evaluating agentic memory.It uses dense retrieval and cross-encoder reranking over raw history.Source: §1 · p. 3
  • The paper identifies evidence access and relational completeness as bottlenecks: retrieved answer values may lack the relations and dependency paths needed to derive answers.Even complete access to transformed memory stores does not close the accuracy gap to raw evidence.Source: §1 · p. 3

2. Related Work

Prior memory systems use varied content processing, structural organization, and retrieval controls, while RAG systems select and rank external evidence. This paper compares those approaches with a fixed dense-retrieval and reranking baseline.

2.1. Agentic memory systems

  • Agentic memory systems use operations such as fact mutation, evolving graphs, hierarchical consolidation, and selective event representations to manage persistent information.Examples include mutation or graph approaches [4] [30] [36] [54], hierarchical or recurrent stores [5] [10] [16] [21] [26] [51], and selective representations [7] [12] [23] [26] [45] [53] [59].Source: §2.1 · p. 3
  • RAW-RAG provides a fixed dense-retrieval and reranking comparison that isolates evidence access from query-time search control.Concurrent ReFind work preserves raw logs and adds iterative lexical search controls [20].Source: §2.1 · p. 4

2.2. Retrieval-augmented generation

  • RAG systems combine external memory with generation, using lexical or dense retrieval and reranking methods such as late-interaction models and cross-encoders.Structure-aware variants add summaries, graphs, or hierarchies, while adaptive variants learn retrieval or evidence-assessment decisions.Source: §2.2 · p. 4

3. The RAW-RAG

RAW-RAG keeps source history untransformed and uses retrieval and reranking to select evidence for a reader. The reader handles temporal order and state reconstruction when answering.

3. The RAW-RAG

  • RAW-RAG indexes raw conversation history without LLM-based extraction, summarization, graph construction, or mutation.Conversations are serialized with timestamps and image captions, divided into overlapping 256-token windows, and embedded in a flat index.Source: §3 · p. 4
  • The evaluation table covers LoCoMo and LongMemEval accuracy by question type, with column maxima excluding Full-context identified in the caption.LongMemEval categories include single-session user, assistant, and preference questions, multi-session questions, knowledge updates, and temporal reasoning.Source: Table 3 · p. 5
  • At query time, cosine similarity retrieves 128 windows and a cross-encoder selects the top 15 to supply as evidence to the reader.The reader uses the selected evidence to resolve temporal order and relevant state changes when generating an answer.Source: §3 · p. 5

4. RAG Is Stronger Than We Thought

The section evaluates RAW-RAG against memory systems on four conversation benchmarks, then tests retrieval design, memory-management additions, and efficiency. RAW-RAG ranks first among compared memory systems in six of seven settings, while its ablations support moderate retrieval depth and show no accuracy gain from added memory management.

4.1. Benchmarks and baselines

  • The evaluation covers LoCoMo, LongMemEval, MEME, and MemoryAgentBench, with comparisons aligned to the benchmark settings described by the authors.The benchmarks span recall, temporal and multi-hop reasoning, preferences and updates, and dynamic memory dependencies.Source: §4.1 · p. 5

4.2. Main results

  • RAW-RAG ranks first among memory systems in six of seven settings, leading LoCoMo by 0.65–0.95 points, LongMemEval by 0.13, GPT-4o-mini MEME by 2.24, and MemoryAgentBench by 4.48–5.53.MemMachine leads DeepSeek-V4-Flash MEME by 1.52 points, while Full-context leads RAW-RAG by 19.98 points in that setting.Source: §4.2 · pp. 5–6, Table 4 · p. 6

4.3. Ablation studies and efficiency

  • Removing reranking lowers MemoryAgentBench Total by 3.92 points, one-item retrieval by 11.73, and Top-120 by 14.57; final k values 5, 10, 30, and 60 stay within 1.3 points.Chunk sizes from 128 to 1,024 do not significantly change Total.Source: §4.3 · p. 6, Figure 4 · p. 7
  • None of the three memory-management additions improves RAW-RAG’s 48.32% MemoryAgentBench Total.Atomic-fact extraction lowers test-time learning from 47.68% to 15.37% and Total by 19.62 points; summary augmentation loses 5.81 points overall, and graph expansion is 5.78 points lower overall.Source: §4.3 · pp. 6–7, Table 5 · p. 7
  • On the evaluated workloads, RAW-RAG combines competitive accuracy with lower ingestion and combined ingestion–retrieval costs.On the 10-episode, 110-question MEME slice, Add-to-ready plus Search takes 305.91 seconds, 5.1× faster than the next system; the studies exclude answer generation and evaluation.Source: Table 6 · p. 7, §4.3 · pp. 7–8

5. Why Is RAW-RAG Enough in Most Cases?

The analyses attribute RAW-RAG’s strength to more frequent access to sufficient evidence, including answer values and the relations needed to derive them. They also find that complete access to transformed stores does not eliminate the performance gap to raw evidence.

5.1. RAW-RAG offsets lower utilization with high coverage

  • Coverage measures whether retrieved evidence is sufficient, while utilization is answer accuracy conditional on sufficient evidence.Coverage uses an LLM judge on MemoryAgentBench and task-specific deterministic checks on MEME.Source: §5.1 · p. 8
  • RAW-RAG achieves the highest coverage on both benchmarks (45.29% and 76.35%), and coverage differences exceed utilization differences, helping offset its lower conditional utilization.Coverage spans 19.07–45.29% on MemoryAgentBench; on MEME, coverage varies by 40.66 points versus 21.04 for utilization.Source: §5.1 · p. 8

5.2. Raw history preserves the paths needed for reasoning

  • On MEME tracking and cascade after-state questions, RAW-RAG has the highest conditional path coverage at 62.34%, versus 56.70% for MemMachine and 9.62% for MemU.Path coverage captures answer values plus chronological order or linked dependency rules and upstream triggers.Source: §5.2 · pp. 8–9, Table 9 · p. 9

5.3. Preserving raw evidence supports downstream recovery

  • Complete-store access improves all seven systems, but RAW-RAG and MemMachine reach 57.54% and 57.46%, respectively, versus 61.60% for raw evidence.The complete-store oracle uses gold-informed selection over exported phase-visible memories; the raw-evidence control selects original annotated support sessions.Source: Figure 5 · p. 9, §5.3 · p. 9
  • Even without the packing ceiling, complete stores remain 7.58–29.38 points below raw evidence, though the residual gap is not a pure measure of ingestion-time information loss.Evidence assembly and positional effects may also contribute.Source: §5.3 · p. 9

6. Limitations and Conclusion

RAW-RAG challenges the need for complex agentic memory in the evaluated conversation tasks, but the evidence is limited to offline public benchmarks. The appendix specifies evaluation settings for the LoCoMo comparisons.

6.1. Limitations and future work

  • The evidence is limited to offline conversation on public benchmarks and does not establish performance in interactive, multimodal, or broader operational settings.Future work should test longer histories, broader model ranges, privacy and deletion constraints, and when structured or compressed memory offers advantages.Source: §6.1 · p. 10

6.2. Conclusion

  • RAW-RAG ranks first among compared memory systems in six of seven benchmark–model settings, challenging the need for complex agentic memory in the evaluated tasks.The conclusion recommends that improved agentic systems demonstrate gains over a well-specified naive RAG baseline and establish what added complexity contributes.Source: §6.2 · p. 10

A. Experimental Settings and Complete Results

  • The appendix supplements main-table comparisons with evaluations aligned to prior studies, retaining published baseline scores and measuring RAW-RAG results in this study.Scores use a 0–100 scale unless otherwise stated; full-context uses no embeddings.Source: A. Experimental Settings and Complete… · p. 16

A.1. LoCoMo results

  • LoCoMo judge accuracy averages binary correctness judgments across questions and judge repetitions, with category and overall results based on 1,540 questions excluding category 5.RAW-RAG uses 256-token windows, 32-token overlap, 128 dense candidates, and 15 reranked chunks expanded to complete turns.Source: §A.1 · p. 16

A.1.1. Text-embedding-3-small setting

  • In the text-embedding-3-small setting, RAW-RAG uses Cohere Rerank-v3.5 and evaluates separate GPT-4o-mini and GPT-4.1-mini reader blocks.Table 10 reports the matched comparison; the supplied passage does not give its score values.Source: §A.1.1 · p. 16, Table 10 · p. 17

A.1.2. Qwen3 retrieval setting

  • The Qwen3 retrieval setting uses Qwen3-Embedding-4B and Qwen3-Reranker-4B with GPT-4o-mini and GPT-4.1-mini readers across all 1,540 LoCoMo questions.RAW-RAG follows the EverMemOS evaluation setting, including history access, answer generation, grading prompts, and score averaging; Table 11 reports results.Source: §A.1.2 · p. 17

A.1.3. Short-answer evaluation

The appendix reports matched short-answer evaluations on LoCoMo and LongMemEval-S, with separate scoring details and benchmark settings. It also flags a discrepancy between Mem0’s published aggregate and the value implied by its category scores.

A.1.3. Short-answer evaluation

  • The short-answer comparison reports LLM-judge accuracy, token F1, and BLEU-1, keeping Mem0’s published results separate from Nemori’s version-1 re-evaluations.RAW-RAG’s minimal rows request only the shortest answer needed; its three metrics evaluate the same generated answers.Source: §A.1.3 · p. 17, Table 12 · p. 19
  • Mem0’s published GPT-4o-mini category scores yield a question-weighted 62.14, rather than its published overall score of 66.88.Question-level judgments are unavailable, so the tables retain both the published category scores and aggregate without recomputing either.Source: §A.1.3 · p. 18

A.2.1. Text-embedding-3-small setting

  • The text-embedding-3-small LongMemEval-S setting evaluates all 500 questions with GPT-4.1-mini and question-weighted overall accuracy.The comparison includes MemoryOS, A-MEM, RecMem, Nemori, MemMachine, Memora, and RAW-RAG; full-context is omitted because it does no embedding-based retrieval.Source: §A.2.1 · p. 18

A.2.2. Qwen3 retrieval setting

  • The Qwen3 LongMemEval-S setting uses Qwen3-Embedding-4B and Qwen3-Reranker-4B over all 500 questions, following the EverMemOS evaluation and repeated-judge procedure.Three GPT-4o-mini judgments per fixed answer are averaged over questions and repetitions; Table 16 reports the corresponding results.Source: §A.2.2 · pp. 18–19

A.2.3. Non-abstention evaluation

The non-abstention comparison removes the 30 annotated abstention items from LongMemEval-S and recalculates question-weighted accuracy using existing answers and judgments.

A.2.3. Non-abstention evaluation

  • The subset removes 30 annotated abstention items from the 500-question set, and the reported accuracy reuses existing answers and judgments.Text-embedding-3-small pairs with Cohere Rerank-v3.5, while Qwen3 pairs Qwen3-Embedding-4B with Qwen3-Reranker-4B.Source: §A.2.3 · pp. 19–20
  • In the matched non-abstention comparison, textemb3-configured RAW-RAG exceeds EMem and EMem-G by 4.80 and 2.90 points, while Qwen3-configured RAW-RAG exceeds them by 7.57 and 5.67 points.The configurations score 87.80 and 90.57; their full-set scores are 87.53 and 89.93.Source: §A.2.3 · p. 20

A.3. PersonaMem-v2 results

The appendix reports RAW-RAG’s PersonaMem-v2 comparison, then tests retrieval choices, evidence budgets, and evidence-access diagnostics. Across the reported analyses, outcomes vary by capability and benchmark, while coverage is identified as the broader cross-system bottleneck.

A.3. PersonaMem-v2 results

  • RAW-RAG achieves 49.62% accuracy on PersonaMem-v2, ranking third among seven systems behind MemOS and EverMemOS and leading no individual scenario.The comparison uses each system’s native profile configuration, and statistical significance was not tested.Source: §A.3 · p. 22, Table 18 · p. 22

A.4.1. Retrieval and memory ablations

  • On MemoryAgentBench, top-60 improves Long-Range Understanding by 2.22 points (Holm-adjusted p = 0.025), although top-10 and top-60 do not significantly change Total.The result illustrates that one global context size need not be optimal for every capability.Source: §A.4.1 · p. 22, Table 19 · p. 23

A.4.2. Evidence-length analysis

  • The evidence-length analysis measures reader-supplied evidence after display-time expansion and formatting, including headers, timestamps, separators, and repeated evidence.It excludes the question, fixed prompt, chat framing, and generated answer; lengths are compared across readers using the same token-counting method.Source: §A.4.2 · p. 23, Table 20 · p. 23

A.4.3. Cross-system budget control

  • RAW-RAG’s mean evidence length is 3,915.8 tokens and stays below the 5,000-token ceiling for every question, while the cap shortens evidence for most EverMemOS and MemMachine questions.Clipping affects 2,011 EverMemOS and 2,030 MemMachine questions, reducing their mean evidence lengths to 4,160.6 and 4,479.2 tokens.Source: §A.4.3 · p. 24, Table 21 · p. 24
  • With a 5,000-token cap on MemoryAgentBench, RAW-RAG needs no clipping and retains 48.32% Overall, versus 44.74% for the strongest baseline in this control.The same cap decreases Overall for all five baselines, though individual sources can improve.Source: §A.4.3 · p. 24, Table 22 · p. 24

A.4.4. Raw-RAG budget sensitivity

  • Under the RAW-RAG budget sensitivity reruns, scores rise on LoCoMo and LongMemEval but fall 1.09 points on MEME as mean evidence length drops 63.5% and 65.2% on the latter two benchmarks.Disabling turn expansion shortens inputs without dropping or clipping selected chunks; the LongMemEval and MEME length reductions are relative to Table 20.Source: §A.4.4 · p. 25, Table 23 · p. 25
  • With the 5,000-token cap, RAW-RAG’s Overall exceeds the strongest non-RAG baseline by 1.30 points on LoCoMo, 1.80 on LongMemEval, and 1.15 on MEME.Category results remain mixed: RAW-RAG trails EMem on LongMemEval multi-session questions and MEME aggregation, deletion, and cascades.Source: §A.4.4 · p. 25, Table 24 · p. 26

A.5.1. Coverage and utilization

  • Across four evidence assessments, all omnibus nulls are rejected, with more significant pairwise contrasts and larger observed ranges for coverage than utilization on MemoryAgentBench and MEME.The uncertainty analysis supports coverage as the broader cross-system bottleneck, while also finding significant variation in utilization.Source: Table 25 · p. 26, §A.5.1 · p. 27, Table 26 · p. 27

A.5.2. Complete-store evaluation

The complete-store diagnostic compares native retrieval, gold-informed access to persisted memory, and a raw-evidence control on MEME. The appendix also describes efficiency measurements for RAW-RAG and its memory-management variants.

A.5.2. Complete-store evaluation

  • The diagnostic’s N, C, and R conditions use native retrieval, gold-informed selection over exported persisted representations, and a shared raw-evidence control, respectively.The conditions use the same 100 MEME episodes and 1,188 question IDs; C and R use a 5,000-token budget and task-aware reader instructions.Source: §A.5.2 · pp. 27, 29
  • MemMachine’s Overall falls from 57.46% to 54.02% without the cap, while the other six stores’ cap-removal intervals include zero.MemMachine’s assembled evidence grows from 4,982.8 to 29,459.7 tokens; its decline remains when restricting analysis to questions with changed evidence.Source: §A.5.2 · pp. 28–29, Table 30 · p. 29
  • On the common subset excluding 12 filtered and paired gating questions, the raw control scores 61.53%, while MemMachine scores 54.85% and RAW-RAG 54.68%.The negative full-set cap-removal interval for MemMachine is sensitive to provider filtering.Source: §A.5.2 · p. 29
  • In MEME’s GPT-4o-mini six-task macro-accuracy diagnostic, every raw-minus-complete-store interval excludes zero, and every raw-minus-uncapped-store interval remains positive, showing cap removal alone does not eliminate the residual raw-evidence gap.Source: §A.5.2 · pp. 29–30, Table 29 · p. 29

A.6. Efficiency and scalability

  • Efficiency measurements separate Add-to-ready and Search on a 10-conversation, 110-question MEME slice from sequential ingestion on one BEAM-1M conversation.Table 32 reports memory-management computation and latency on 11 commonly measured MemoryAgentBench sources; timings exclude answer generation and evaluation.Source: §A.6 · p. 30, Table 32 · p. 30

B. Raw-RAG: Method Details

RAW-RAG preserves source history, retrieves relevant windows with dense search and reranking, and supplies selected evidence to a reader. The appendix also specifies diagnostic measures, memory-management variants, and context-budget interventions.

B.1. Indexing and evidence assembly

  • RAW-RAG preserves each message’s date, speaker, and content in 256-token windows with 32-token overlap, retaining links to source messages for turn reconstruction.Supplied image captions enter as text, and missing dates are marked unknown.Source: §B.1 · p. 31
  • For a question, an embedding encoder selects candidates and a reranker selects reader evidence; cosine similarity ranks windows within the conversation, with a fixed top-k budget.Selected windows can be expanded into complete overlapping source turns; expansion changes reader-visible text, not selection scores.Source: §B.1 · p. 31

B.2. Query construction

  • The original benchmark question is the retrieval query, while question date or evaluation phase determines evidence visibility where available.MemoryAgentBench task instructions are added when constructing reader input, after retrieval.Source: §B.2 · p. 31

B.3. Context-length policies

  • The main RAW-RAG conditions add no evidence-token cap or automatic head-tail clipping after turn expansion; the full assembled input is submitted to the reader.The examined benchmark-model conditions saved identical assembled and reader evidence, with no recorded answer-generation error responses.Source: §B.3 · p. 32

C.1. Coverage, utilization, and path retention

  • Coverage and utilization summarize assessed evidence sufficiency and answer score, while path retention measures whether result-covered contexts also contain the required supporting path.Correct answers outside the sufficient-evidence set may reflect labeling errors, partial evidence, prior knowledge, or guessing.Source: §C.1 · p. 32

C.2. Write-policy and evidence controls

  • The write-policy audit separates proposed mutation decisions from persisted text changes, and its ADD/NONE-only intervention converts proposed updates to additions and deletions to no-ops.An effective update requires persisted old and new text that differ; persisted deletion also counts as effective mutation.Source: §C.2 · p. 33

C.3. Memory-management ablations

  • Memory-management variants compare extracted atomic facts, summaries that augment raw windows, and graph-expanded raw windows that preserve source text for the reader.The atomic-fact condition replaces raw windows in the searchable index; raw summaries retain them, while graph expansion changes candidate recall and disables graph cues in reader context.Source: §C.3 · pp. 33–34

C.4. Context-budget interventions

  • The context-budget interventions alter evidence display and packing under a 5,000-token ceiling, with expansion-aware budgeting first disabling turn expansion before prefix truncation if needed.The outer prompt and question are outside this budget.Source: §C.4 · p. 34

C.5. Complete-store and oracle evaluations

The diagnostic contrasts source-oracle and complete-store evidence conditions, then qualifies what those comparisons can establish. Its statistical procedures report paired uncertainty conditional on observed outputs and builds.

C.5. Complete-store and oracle evaluations

  • The answer-location source oracle supplies all annotated supporting sessions visible before the question cutoff, chronologically, without a token cap or truncation.Its selection does not use the question, gold answer, or entity values; Abs alone receives a value-free dependency-validity instruction.Source: §C.5 · p. 34
  • Complete-store access replaces native retrieval with gold-informed selection over exported persisted representations, paired with a raw-evidence control using the same gold fields.Both conditions use a nominal 5,000-token budget and task policy; the selection, context assembly, budget, and readout differ from the answer-location oracle.Source: §C.5 · pp. 34–35
  • The complete-store selector ranks stored objects using annotated value phrases and question-word overlap, with gold values and answers guiding only this offline selection.Objects without a phrase hit or query-word overlap are not selected; ties follow store enumeration order.Source: §C.5 · p. 35
  • “Complete” means enumerating persisted reader-visible representations, subject to provenance and temporal-cutoff rules; most exports reflect final stores rather than historical snapshots.Overwritten or deleted objects are unavailable, and surviving objects may contain later revisions; EMem is the stated exception with explicit phase snapshots.Source: §C.5 · p. 35, Table 33 · p. 36
  • The raw-to-store gap reflects both stored information and how evidence is presented, so the comparison does not isolate ingestion-time information loss.Native-to-complete gains also combine broader access with changes to task information and readout policy; raw evidence scores below 100%, and rankings vary by task.Source: §C.5 · p. 37

C.6. Statistical inference

  • Reported intervals are paired and conditional on observed reader and judge outputs, store builds, and sufficiency labels; complete-store contrasts use episode-level resampling.Those contrasts recompute the six-task macro-average in each condition and report pointwise intervals without multiplicity correction.Source: §C.6 · pp. 37–38

D. Experimental Implementation and Reproducibility

This section documents implementation and evaluation procedures—including benchmark tasks, scoring, latency, deployment, and system-specific memory and retrieval configurations—to support reproducibility. It also covers supplementary personalization and ingestion-timing workloads and reviews limitations in prior raw-RAG evaluations that motivate stronger retrieval controls.

D.1. Benchmark descriptions

  • LoCoMo uses 1,540 questions across 10 conversations, while LongMemEval-S uses 500 examples with separate histories and an unanswerable condition.LoCoMo questions span single-hop, multi-hop, temporal, and open-domain tasks; LongMemEval-S also tests preferences, knowledge updates, temporal reasoning, and abstention.Source: §D.1.1 · p. 38, §D.1.2 · pp. 38–39
  • MEME evaluates 1,188 phase-specific questions across six tasks, with after-phase credit for Cascade, Absence, and Deletion requiring a correct before-phase answer.Its Overall score is the six-task macro-average; task-specific scoring distinguishes ordered histories, required values, and the correctness conditions for updates, absence, and removal.Source: §D.1.3 · p. 39
  • MemoryAgentBench evaluates 2,071 questions from 14 sources across retrieval, test-time learning, long-range understanding, and conflict resolution.Its aggregate combines native task metrics using nested capability weights, while its embedded LongMemEval source uses settings distinct from the standalone benchmark.Source: §D.1.4 · p. 40
  • PersonaMem-v2 adds a 5,000-question multiple-choice personalization evaluation, while BEAM supplies a 1,710-message conversation for ingestion-throughput, readiness-cost, and persistence timing.PersonaMem-v2 uses exact answer-letter agreement; BEAM is used solely for ingestion measurements.Source: §D.1.5 · pp. 40–41, §D.1.6 · p. 41

D.2. Evaluation workflow

  • The shared workflow separates ingestion, retrieval, answer generation, and scoring, keeps references outside memory construction, and applies question-date filtering where supported.Timestamp filtering keeps unknown-date memories by default and does not by itself establish temporal validity for revised memories or undated semantic features.Source: §D.2 · p. 41

D.3. Baseline configurations

  • RAW-RAG uses a per-conversation local index with default limits of two concurrent indexing conversations, eight searches, and 50 answer requests; agentic baselines use distinct memory-management and retrieval procedures.EverMemOS combines lexical and semantic search with query refinement and evidence-sufficiency checks; MemOS retrieves and reranks up to 20 memories and preferences, while EMem and EMem-G differ in graph-based relevance propagation.Source: §D.3.1 · pp. 41–42, §D.3.2 · p. 42, §D.3.3 · p. 42
  • SimpleMem combines semantic, keyword, and structured retrieval over timestamped dialogue windows, while MemU uses default intention routing and evidence-sufficiency checks within each conversation.SimpleMem passes at most 20 entries to the reader; MemU’s default final result cap is 20, reduced to 10 for MemoryAgentBench.Source: §D.3.5 · p. 43, §D.3.6 · p. 43
  • MemMachine presents episodic and semantic evidence separately, Mem0 supports ADD/UPDATE/DELETE/NONE memory operations, and Memora links indexed cues to factual and episodic content.MemMachine’s undated semantic features bypass date filtering; Mem0’s ADD/NONE-only intervention converts proposed UPDATE to ADD and DELETE to NONE; Memora’s evaluated retrieval combines cue indexing, BM25 hybrid search, and episodic memory.Source: §D.3.7 · pp. 43–44, §D.3.8 · p. 44, §D.3.9 · p. 44

D.3.9. Memora

  • Memora uses isolated Chroma directories per conversation, with bounded and recursively split extraction inputs, three retries, a 600-second timeout, seed 42, and zero temperature.Ordinary evaluation permits 10 ingestion and 10 search workers plus two segmentation workers; MemoryAgentBench instead ingests one prepared source chunk at a time without extra segmentation.Source: §D.3.9 · p. 45

D.3.10. Full-context

  • Full-context gives the reader chronological history directly, without memory transformation or retrieval, applying date filtering when supplied and a bounded-history policy.Direct evaluation uses four history-preparation workers and 10 answer workers; published Full-context results retain their original settings.Source: §D.3.10 · p. 45

D.4. Experimental controls

  • Evaluation controls save each question’s reader context, answer, and score to distinguish retrieval, generation, and judging changes.Comparability requires complete question coverage and unique context-answer-judgment links; MEME evidence must respect question phase, and dense-order fallback omits reranking.Source: §D.4 · p. 45

D.5. Scoring and benchmark-level summaries

  • Benchmark summaries use dataset-specific weighting, including source-level then capability-level aggregation for MemoryAgentBench and averaged ICL source scores before ReDial in Test-Time Learning.Full-benchmark results require complete source coverage; a flat average over questions or TTL sources would assign different weights.Source: §D.5 · p. 45

D.6. Latency measurement

  • Latency measurements isolate ingestion and search stages, excluding reader generation and evaluation, with sequential-ingestion runs using one conversation and outer-ingestion worker.The BEAM comparison uses complete runs over the same 1,710-message conversation; the MEME experiment covers 10 conversations and 110 questions. Timings include local computation, API and network latency, and retries, and depend on readiness rules, internal parallelism, hardware resources, and endpoint load.Source: §D.6 · pp. 45–46

D.7. Computational environment

  • Experiments ran on one local machine while LLM, embedding, and reranking inference used API endpoints; incompatible methods ran in separate software environments.The host had an Intel Core i9-13900HX CPU, 16 GiB RAM, an NVIDIA GeForce RTX 5070 Ti Laptop GPU, and a 1 TB NVMe SSD; the appendix records method-specific packages and versions where available.Source: §D.7 · p. 46

E. Review of Prior RAG Evaluations; E.1. Review scope

  • Prior RAG baselines may have performed poorly because of weak configurations, motivating stronger retrieval controls.The review examines oversized chunks, low top-k, and missing reranking; published reference scores also differ in readers, prompts, question sets, budgets, and scoring.Source: E. Review of Prior RAG Evaluations · p. 46, §E.1 · pp. 46–47

E.2. Comparative analysis

The comparative review identifies configuration weaknesses in prior raw-history RAG baselines and argues that establishing a need for agentic memory requires comparison with RAW-RAG that addresses those constraints. Its table reviews 14 prior evaluations, but scores are not directly comparable across studies.

E.2. Comparative analysis

  • Table 34 reviews raw-history RAG configurations in 14 prior evaluations, but its cross-study scores are not directly comparable and its retrieval diagnoses are not controlled effect estimates.The table’s scores are on a 0–100 scale unless specified otherwise; Memobase review covers only its public configuration.Source: Table 34 · p. 47, §E.2 · p. 47
  • NaiveRAG scored 61.00 on LongMemEval-S with GPT-4o-mini in LightMem [7].Source: §E.2 · p. 48
  • 57.99 GPT4Score was reported for MPNet turn retrieval on author-generated LoCoMo QA by SeCom.The retrieved turns averaged 3288 tokens under a 4k-token budget.Source: §E.2 · p. 48
  • Top-10 turn/session RAG by SGMem [52] scored 54.4/57.6 on LongMemEval and 33.0/40.8 on a 500-question LoCoMo sample.The compared retrieval units were turns and sessions, respectively.Source: §E.2 · p. 48
  • The review identifies different remaining retrieval controls: some evaluations already tune granularity and depth, while broad candidate-pool cross-encoder reranking remains an omission in others.LongMemEval’s reviewed setup includes extensive retrieval tuning but omits reranking over a larger candidate pool; SGMem already tunes chunk size and top-k but lacks broad-pool reranking.Source: §E.3 · p. 48, §E.2 · p. 48

E.3. Synthesis

  • Prior RAG baselines often use overly large chunks or too few retrieval locations, motivating matched comparisons with RAW-RAG that address chunk size, retrieval depth, and ranking.The authors note that quantifying these design choices’ contribution to published score gaps requires controlled reruns.Source: §E.3 · p. 48

F. Extended Discussion and Limitations

This section limits the findings to answer-from-history tasks and outlines operational, reader, representation, and scalability uncertainties. It also documents the evaluated prompts and diagnostic controls for judging evidence sufficiency, dependency validity, answer formats, and complete-store readouts.

F. Extended Discussion and Limitations

  • Raw-history retrieval may not meet operational needs such as exposing authoritative state, triggering actions, or erasing information, which extend beyond answer accuracy.The evaluated tasks ask for answers from recorded histories; operational memory systems may require additional memory-management objectives.Source: F. Extended Discussion and Limitations · p. 49
  • Phase-visible MEME oracle reader differences show state reconstruction depends on reader capability, so retrieval should be evaluated with its intended reader.Whether ingestion-time state construction helps weaker readers or reduces repeated inference remains open; cross-system utilization estimates also depend on which questions have sufficient evidence.Source: F. Extended Discussion and Limitations · p. 49
  • The 47.68% to 15.37% drop under atomic-fact extraction may reflect reduced evidence volume, disrupted demonstrations, reader–format mismatch, and factual loss.Token volume and demonstration format vary together in the ablation, and the separate six-system 5k control leaves these factors coupled; the result shows the tested representation-reader combination performs poorly on TTL.Source: F. Extended Discussion and Limitations · p. 49
  • Raw storage, fixed candidate and reader budgets, and repeated text in expanded windows may constrain scalability; longer workloads or multimodal evidence may favor different trade-offs.Retrieval quality across history lengths and repeated query-time reasoning costs remain to be measured alongside ingestion; hosted services and outputs can also vary.Source: F. Extended Discussion and Limitations · p. 49

G.1. Qwen3 reranker

  • The Qwen3-Reranker-4B setup uses its official model-card chat template with an instruction to judge question–passage relevance.The passage is a candidate retrieval window before display-time expansion to complete turns.Source: §G.1 · p. 49

G.2. Memory-management variants

  • The evaluated memory-management variants extract atomic facts, summarize chunks, or link related chunks when useful for multi-hop retrieval or joint question answering.The prompts specify preserving source-supported details and ordering cues; graph links are based on concrete connections rather than topical or stylistic similarity.Source: §G.2 · pp. 50–51

G.4. LoCoMo reader; G.5. LongMemEval reader

  • The LoCoMo and LongMemEval reader prompts use structured reasoning, while date-annotated LongMemEval questions replace the query with a date-aware version.The shared LoCoMo prompt asks readers to extract, link, and verify details; LongMemEval uses the original question when no question date is annotated.Source: §G.4 · p. 53, §G.5 · p. 55

G.5. LongMemEval reader

  • The LongMemEval reader prompt restricts answers to memories dated at or before the question date.It specifically addresses temporal terms such as “current” or “now.”Source: §G.5 · p. 55

G.6. MEME reader

  • The MEME standard reader prompt supplies phase-appropriate evidence and requires an answer based only on context, returning only the value or stating the information is unavailable.The value-free dependency-validity control and complete-store policy are separate conditions.Source: §G.6 · p. 55

G.7. MemoryAgentBench reader

  • MemoryAgentBench uses source-specific query templates across retrieval, in-context learning, recommendation, long-range understanding, and conflict-resolution tasks.The reader receives the task query and memories, with current time appended; five ICL sources share a numerical-label template, while ReDial uses a recommendation template.Source: §G.7 · pp. 55–57

G.8. PersonaMem-v2 reader

  • PersonaMem-v2 presents the original request with a personalization reminder and ordered answer options, asking the reader to reason before returning a letter.Retrieval uses the original request before the reminder is appended.Source: §G.8 · p. 58

G.9. Semantic sufficiency judge

  • The sufficiency judge marks context answerable only when it supports deriving the gold answer without world knowledge or prior memory.It judges an excerpt selected around literal reference-answer matches, or the first 9,000 characters when none match.Source: §G.9 · pp. 58–59

G.11. Dependency-validity control

  • The Abs source-oracle control instructs readers to treat a dependent fact as stale after its stated upstream dependency changes, unless a new value is given.The added instruction is value-free; Table 2’s Overall combines task-specific conditions.Source: §G.11 · pp. 60–61

G.12. Complete-store task policy

  • For complete-store readouts, the policy distinguishes phase- and task-specific rules for updates, dependencies, deletion, tracking, aggregation, and exact recall.The policy is appended after selected evidence, with its token count reserved within the 5,000-token context budget.Source: §G.12 · p. 61

G.10. Result and path assessment

The task-aware assessment judges result and path coverage jointly, applying task-specific evidence requirements to retrieved context. A nested comparison separates complete answer values from sufficient paths and excludes deletion and invalidation from that comparison.

G.10. Result and path assessment

  • The task-aware assessment jointly judges result and path coverage, reassessing tracking in both phases and deletion, cascade, and invalidation after the update.Other task/phase rows retain their original semantic labels.Source: §G.10 · p. 59
  • Tracking requires all history values in chronological order, deletion requires an explicit targeted instruction or event, and absence of an old value does not establish deletion.Temporal order may be conveyed by dates or explicit language such as then, later, or after.Source: §G.10 · p. 60
  • Cascade and invalidation require the target value, dependency rule, and relevant upstream state to form one linked, executable causal chain.Cascade uses the post-trigger upstream state; invalidation uses the state that invalidates the old target value.Source: §G.10 · p. 60
  • The nested comparison treats value_complete as result coverage and sufficient as path coverage, compares P(path | result) for tracking-after and cascade-after, and excludes deletion and invalidation.Those tasks are excluded because a deleted or invalidated old value is not a valid current result; both labels come from the joint full-context judgment.Source: §G.10 · p. 60

Related papers

Loading 2610.30755v1…