PaperlayerView exact revision on arXiv

Source-linked AI summary

Entity-Memory Graph Retrieval Improves Evidence Coverage in Long-Conversation Question Answering

Shumao Sun

arXiv:2608.27925v1cs.CL

TL;DR

Long-conversation memory systems must retrieve evidence scattered beyond a model's usable context, yet retrieval coverage and generated answer quality can diverge. This paper evaluates an Entity–Memory graph against a matched dense control with fixed representations and reader conditions. On LoCoMo, graph structure improves evidence recall, but the study does not support an overall final-answer F1 gain and remains bounded in scope.

  • Problem

    Long conversations exceed usable context, making it necessary to decide what to store, organize, and retrieve for questions requiring varied reasoning.

  • Method

    The paper builds a conversation-only Entity–Memory graph and compares its retrieval with matched dense retrieval while fixing Memory and query vectors, context budget, answer protocol, and evaluation.

  • Results

    Graph structure improves evidence coverage, while the corresponding overall final-answer F1 difference is not supported; tested requested extractors show limited empirical robustness and embedding robustness is mixed.

  • Takeaways & Limitations

    Separating evidence coverage from answer quality isolates a retrieval-coverage gain without establishing an end-to-end final-answer quality gain.

  • Takeaways & Limitations

    The claim is bounded by ten conversations from one LoCoMo release, two requested extractors, two frozen embedding artifacts, a single reader protocol, and one environment.

Abstract

from arXiv · show

Entity-Memory graph retrieval keeps dialogue turns as verbatim Memory nodes, links repeated mentions through shared Entities, and connects adjacent Memories with directed chronological edges. At query time the retriever moves from Entity gating through semantic fusion and one-hop chronological recovery to dense backfill. The path can keep a neighboring Memory that dense cosine ranking would otherwise omit. A matched dense control shares the Memory and query vectors, context budget, requested answer protocol, and evaluator, isolating graph structure from changes to the reader. On 1,986 questions from ten LoCoMo conversations, graph retrieval raises official evidence recall at top-k 25 from 79.7468% to 84.4842%. The recall advantage is supported from top-k 5 to 50, while no matched cutoff supports an overall final-answer F1 difference. Four paper-eligible requested configurations support empirical robustness across the tested GPT-3.5 and DeepSeek extractors on both outcomes. Embedding robustness is mixed: F1 has no supported contrast, but recall is sensitive to the embedding artifact. The comparison isolates a retrieval-coverage gain from graph structure. It does not establish a final-answer F1 gain, model or embedding equivalence, or cross-dataset generalization.

1 Introduction

Long-conversation memory requires organizing scattered information and retrieving relevant evidence for varied reasoning questions. This paper tests whether conversation-derived Entity–Memory graph structure improves retrieval coverage over a matched dense control while separating retrieval effects from answer quality.

  • Motivation: Long conversations exceed usable context, requiring systems to decide what to store, how to organize it, and what to retrieve.LoCoMo tests single-hop, multi-hop, temporal, commonsense, and adversarial reasoning, while long-range temporal and causal reasoning remain difficult.
  • Motivation: Dense retrieval treats Memory items independently, whereas graph structure represents repeated entities and chronological adjacency.This creates a structural comparison target while preserving the distinction between graph effects and changes to embeddings, context budgets, answer models, or evaluation.
  • Approach: Each dialogue item becomes a Memory node, normalized Entities become shared nodes, and chronological links connect neighboring Memories.At query time, Entity matching is fused with semantic Memory similarity and chronological expansion; questions and evaluation artifacts are excluded from graph construction.
  • Research questions: The study asks whether graph retrieval improves evidence recall over matched dense retrieval and whether that gain translates into higher overall final-answer F1.It also examines retrieval components, context and input profiles, fusion and sequence settings, and requested extractor and embedding configurations.
  • Experimental design: The experiments hold Memory vectors, query vectors, answer protocol, context budget, and evaluation fixed to isolate graph structure.The design combines an all-question matched comparison with component, cutoff, input-profile, and configuration analyses.
  • Headline result: Graph structure improves evidence coverage, while the tested requested extractors support robustness across F1 and recall and embeddings show mixed robustness.The outcome-specific pattern distinguishes supported retrieval effects from unsupported claims of broader answer-quality or equivalence gains.

2 Related Work

Prior work spans dense retrieval, hierarchical and reflective memory, and graph-based organization for long-context question answering. This paper narrows the comparison to whether a conversation-derived graph changes evidence retrieval when representations, reader protocol, and context budget remain fixed.

  • Dense and long-context retrieval: Dense Passage Retrieval provides the flat dense-retrieval control by scoring independently encoded conversational Memory items against a question.Its simplicity and scalability contrast with graph retrieval's use of relations among Memory items.
  • Hierarchical memory: RAPTOR and MemoRAG add structure above individual chunks through retrieval trees, summaries, global memory, and generated clues.The present method retains dialogue items as answer evidence rather than replacing them with generated summaries.
  • Benchmarks: LoCoMo and LongMemEval motivate pipeline-level evaluation across indexing, retrieval, reading, temporal reasoning, updates, and abstention.These benchmarks frame conversational memory as more than a single final-answer score.
  • Agent memory: Generative Agents, MemoryBank, and MemGPT use reflection, persistent retrieval and updating, forgetting, or hierarchical virtual memory to manage long-term interaction history.These systems represent complementary approaches to storing and accessing conversational experience.
  • Structured memory: Mem0 and A-Mem place relational structure in end-to-end memory systems through graph variants or evolving linked memory notes.This study instead tests a narrower graph-retrieval question under a fixed reader protocol and fixed Memory representations.
  • Graph retrieval: GraphRAG, HippoRAG, GRAG, and LightRAG use graphs, subgraphs, summaries, or entity-relation retrieval to expose structure beyond independent text chunks.These systems provide descriptive anchors rather than matched causal comparisons for the present study.
  • Comparison gap: Conversational graph-memory systems motivate graph organization but often change the graph, encoder, reader, prompt, and context budget together.The paper therefore uses an internal matched dense control to isolate graph retrieval, treating external systems as descriptive anchors.

3 Method

The method builds a conversation-only Entity–Memory graph and retrieves context by combining entity matches, semantic similarity, and one-hop chronological recovery, with dense backfill when needed.

  • Task and estimands: The study tests whether graph retrieval changes evidence in an ordered context while Memory representations, query vectors, reader, and evaluator remain fixed.The primary estimand is the paired difference in official evidence recall, with final-answer F1 treated as a separate downstream outcome.
  • Conversation-only graph construction: Each dialogue item becomes a Memory node, normalized Entities become shared nodes, and directed links connect chronologically adjacent Memories.Questions and evaluation artifacts do not shape graph construction.
  • Retrieval signals: The retriever uses question-Entity matches and immutable semantic query vectors as retrieval-only signals.Formal retrieval fails on a cache miss rather than issuing a live query-embedding request.
  • Chronological recovery: One-hop chronological expansion transfers each seed score to its immediately preceding and following Memories.The expansion parameter λ is generally 0.5, with sensitivity tests at 0.25 and 1.0.
  • Semantic fusion and backfill: Within the active candidate set, the primary condition combines Entity scores with signed cosine similarity using α = 0.30 and β = 0.70.If no Entity-scored candidates exist, all Memories enter the candidate set; unused dense-ranked items fill remaining top-k positions.

4 Experimental Setup

The experiments use matched artifacts and a frozen answer-and-evaluation interface to isolate graph structure, while separating formal comparisons from diagnostic conditions and robustness analyses.

  • Matched conditions and comparison boundary: The primary comparison holds dataset order, Memory text and vectors, query vectors, answer interface, model, token budget, evaluation package, and aggregation constant.The treatment activates Entity matching, graph fusion, and chronological expansion; B_embed must reproduce dense-control context identifiers.
  • Matched conditions and comparison boundary: The original LoCoMo DRAGON setup is excluded from matched and inferential comparisons because the pinned upstream reproduction failed the declared tolerance.Primary evidence concerns A, B_embed, and B under shared artifacts.
  • Matched conditions and comparison boundary: Figure 1 contrasts independent cosine-ranked Memory retrieval without graph signals against graph retrieval with Entity matching, chronological expansion, and shared evaluation artifacts.The figure’s neighbor annotation is schematic and does not report a metric.
  • Evaluation protocol and statistical design: The frozen LoCoMo-aligned interface uses gpt-3.5-turbo at temperature 0 with a 32-token completion limit and preserves category-specific evaluation behavior.Overall F1 measures final-answer overlap, while recall_acc is the official evidence-recall definition.
  • Evaluation protocol and statistical design: Every formal condition contains the same 1,986 QA rows from ten conversations and uses paired-question and whole-conversation bootstrap resampling.The design matrix separates structural attribution, operating-condition stress tests, parameter sensitivity, and model substitution.
  • Evaluation protocol and statistical design: The paper-eligible configuration matrix crosses two requested Entity extractors with two embedding aliases and applies Holm correction separately across outcome families.A preregistered sequential stop gate uses an absolute three-point band for cluster intervals.

5 Results

The matched comparison shows a supported evidence-recall advantage for graph retrieval, while overall final-answer F1 is not supported as different. Component, cutoff, input, and configuration analyses qualify where the retrieval advantage holds and how it relates to answer quality.

  • Primary matched comparison: 4.7374 percentage points: complete graph retrieval improves recall_acc over matched dense retrieval, but the 0.4998-point F1 change is unsupported.Recall intervals exclude zero under both estimators; both F1 intervals include zero.
  • Component analysis: Graph operations are associated with recall gains, including sequence expansion, Entity-score fusion, semantic scoring, and deterministic speaker links.Complete B improves recall over no-sequence by 1.6728 points and no-Entity-score-fusion by 4.7626 points; semantic scoring contributes 17.4634 recall points relative to Entity-only retrieval.
  • Cutoff and input sensitivity: Recall advantages persist from top-k 5 through 50, whereas every matched F1 interval includes zero across the tested cutoffs.B-minus-A recall differences range from 3.6159 to 5.5702 points.
  • Cutoff and input sensitivity: With raw text, graph retrieval retains a supported 4.9522-point recall advantage without an F1 advantage; time annotation has no supported within-graph overall effect.The input-profile analysis preserves the distinction between evidence recall and final-answer F1.
  • Robustness across configurations: Across four tested GPT-3.5 and DeepSeek extractor configurations, neither extractor contrast rejects zero for F1 or recall, while embedding effects differ by outcome.Embedding F1 contrasts are unsupported, but frozen-artifact recall reductions of 2.0227 and 2.2864 points remain supported.
  • Illustrative cases: Row examples show that chronological recovery can restore omitted evidence and improve both recall and F1, but individual cases do not isolate a single graph operation as causal.For one temporal question, B moves recall and F1 from 0 to 1 after retrieving the relevant Memory and neighboring turns.

6 Discussion

The discussion separates retrieval evidence coverage from answer quality: graph retrieval improves recall over matched dense retrieval, but no overall F1 gain is supported. Robustness and operational findings are bounded by tested configurations, data, and environments.

  • Evidence coverage versus answer quality: Graph retrieval improves annotated evidence coverage across four context budgets, while the matched overall final-answer F1 difference is not supported.Answer-score-only evaluations can obscure retrieval changes, but the retrieval result should not be described as an end-to-end quality gain.
  • Retrieval components: The complete recall difference is distributed across semantic scoring, Entity fusion, chronological expansion, and speaker links.The semantic channel is the largest isolated contributor; smaller effects are reported for Entity fusion, sequence expansion, and speaker links.
  • Robustness: Across four paper-eligible configurations, no requested-extractor F1 or recall contrast rejects zero after Holm correction.This supports limited empirical robustness across the tested GPT-3.5 and DeepSeek extraction configurations, not extractor equivalence or arbitrary-model invariance.
  • Robustness: Embedding robustness is outcome-specific: F1 has no supported contrast, whereas Doubao frozen artifacts reduce recall relative to TES under fixed fusion.The reported robustness scope is limited to specified outcomes and tested configurations without retuning or validation.
  • Operational behavior: Replaying the same question set with complete caches requires no new provider requests before answer generation and reduces retrieval to milliseconds per question.Unseen questions may require new query-embedding and question-Entity calls, and the measurement does not generalize automatically across machines, providers, or workloads.
  • Limitations: Formal evidence comes from ten conversations in one LoCoMo release, so cross-dataset generalization remains unestablished.Replication on additional conversation-memory benchmarks is needed.
  • Limitations: No matched cutoff supports an overall final-answer F1 difference, so the contribution concerns retrieval coverage rather than general improvement in generated answers.Alternative readers and evidence selectors require separately controlled evaluation.
  • Limitations: The official DRAGON reproduction gate did not pass, leaving the local raw DRAGON experiment diagnostic rather than an official reference comparison.A future comparison requires parity with the pinned upstream implementation.

7 Conclusion

The paper tests whether graph structure can improve evidence retrieval under a matched comparison that fixes representations, generation, and evaluation. It finds a retrieval-coverage advantage without supported overall F1 improvement, while emphasizing the study’s bounded scope.

  • 7 Conclusion: Long-conversation question answering requires preserving semantic relevance and relations among turns scattered across sessions.The matched design holds dense representations, query vectors, answer generation, and evaluation fixed.
  • 7 Conclusion: The Entity–Memory graph improves official evidence recall over dense Memory retrieval on the evaluated LoCoMo set, but the corresponding overall final-answer F1 difference is not supported.The recall advantage persists across top-k 5–50 and the raw-text input profile.
  • 7 Conclusion: Component contrasts associate the recall difference with semantic scoring, Entity fusion, chronological expansion, and speaker links.The recall magnitude is sensitive to the 0.10/0.90 Entity/Semantic fusion setting, while no sequence-scale contrast survives family correction.
  • 7 Conclusion: The study is bounded by ten conversations, two requested extractors, two frozen embedding artifacts, one reader protocol, and one latency environment.The failed official DRAGON reproduction gate further limits the claim.

Reproducibility and Responsible Use

The study records extensive reproducibility artifacts and warns that real-world use of conversation graphs must address personal information and memory governance.

  • Reproducibility: Formal conditions record source commits, resolved configurations, dataset and artifact hashes, isolated outputs, cache usage, telemetry, audits, and prompt budgets.Statistical reports and the final cost report have independent byte-identical reproductions.
  • Responsible Use: Conversation data may contain personal information, including speaker names and image captions, requiring consent, retention, deletion, and access-control policies.The benchmark evaluation does not study privacy attacks or sensitive-memory deletion.

Data and Code Availability

The benchmark data and the study’s code, manifests, validation records, and manuscript evidence snapshots are available from the cited sources.

  • Data and Code Availability: LoCoMo data are available from the benchmark authors, and the study artifacts are available in the referenced GitHub repository.The repository includes code, experiment manifests, validation records, and manuscript evidence snapshots.

A Resolved Primary Configuration

Table 7 presents the resolved settings for the primary B condition.

  • Table 7 reports the resolved settings for the primary B condition.
  • The table concerns the primary B condition.

B Validity and Reproducibility Gates

Table 8 lists checks applied before formal evidence is promoted.

  • Table 8 lists checks applied before promotion of formal evidence.
  • The table addresses pre-promotion validity checks.
  • These checks precede formal evidence promotion.
Loading 2608.27925v1…