PaperlayerView exact revision on arXiv

Source-linked AI summary

EviMem: Evidence-Gap-Driven Iterative Retrieval for Long-Term Conversational Memory

Yuyang Li, Yime He, Zeyu Zhang, Dong Gong

arXiv:2604.27695v1cs.CVcs.CL

TL;DR

Long-term conversational memory struggles with temporal and multi-hop questions because relevant evidence is scattered across sessions and single-pass retrieval cannot adapt to what it finds. EviMem combines evidence-gap-driven iterative retrieval with a coarse-to-fine memory hierarchy, achieving higher Judge Accuracy than MIRIX on LoCoMo temporal and multi-hop questions while running 4.5× faster.

  • Problem

    Single-pass retrieval often misses evidence scattered across sessions for temporal and multi-hop conversational-memory questions.

  • Method

    EviMem combines IRIS, which diagnoses missing evidence through sufficiency evaluation and refines queries, with LaceMem’s coarse-to-fine Index, Edge, and Raw memory layers.

  • Results

    EviMem improves Judge Accuracy over MIRIX on temporal and multi-hop questions, reaching 81.6% and 85.2%, respectively, at 4.5× lower latency.

  • Takeaways & Limitations

    Explicit evidence-insufficiency diagnosis enables targeted gap-filling for complex conversational-memory questions missed by single-pass and implicit-feedback methods.

  • Takeaways & Limitations

    LaceMem is currently constructed from complete conversation snapshots, so extending memory construction to online streaming dialogue remains future work.

Abstract

from arXiv · show

Long-term conversational memory requires retrieving evidence scattered across multiple sessions, yet single-pass retrieval fails on temporal and multi-hop questions. Existing iterative methods refine queries via generated content or document-level signals, but none explicitly diagnoses the evidence gap, namely what is missing from the accumulated retrieval set, leaving query refinement untargeted. We present EviMem, combining IRIS (Iterative Retrieval via Insufficiency Signals), a closed-loop framework that detects evidence gaps through sufficiency evaluation, diagnoses what is missing, and drives targeted query refinement, with LaceMem (Layered Architecture for Conversational Evidence Memory), a coarse-to-fine memory hierarchy supporting fine-grained gap diagnosis. On LoCoMo, EviMem improves Judge Accuracy over MIRIX on temporal (73.3% to 81.6%) and multi-hop (65.9% to 85.2%) questions at 4.5x lower latency. Code: https://github.com/AIGeeksGroup/EviMem.

1 Introduction

EviMem addresses the failure of open-loop, single-pass retrieval on temporal and multi-hop conversational questions by diagnosing missing evidence and targeting subsequent retrieval. It combines IRIS’s sufficiency-driven iteration with LaceMem’s coarse-to-fine memory hierarchy, improving complex-question accuracy while reducing latency.

  • Introduction: IRIS evaluates the accumulated evidence set after each retrieval iteration, diagnoses missing evidence, and uses the insufficiency signal to refine queries.This closes the retrieval loop instead of refining queries blindly from generated content or document-level signals.
  • Introduction: LaceMem organizes conversational memory into semantic tuples, sparse edges, and raw details to support fast search, multi-hop expansion, and grounded generation.Its coarse-to-fine hierarchy enables sufficiency evaluation at atomic-tuple granularity while preserving full conversational detail.
  • Introduction: Temporal reasoning reaches 81.6% Judge Accuracy, versus 73.3% for MIRIX and 58.8% for single-pass retrieval.The results identify temporal reasoning as a complex question type that benefits from evidence-gap-driven iteration.
  • Introduction: 9.54s average latency versus 42.71s for MIRIX represents a 4.5× reduction while EviMem matches or exceeds the competitor’s accuracy.The reduction comes from replacing multi-agent orchestration, according to the supplied passage.

2 Related Work

Related work advances long-term conversational memory through persistent storage and structural priors, while iterative retrieval methods vary in their signals for refinement. Existing evaluation-driven methods assess generated content or individual documents, whereas IRIS evaluates whether accumulated evidence suffices to answer a question.

  • Long-term Conversational Memory: Standard RAG uses flat semantic matching, while MemoryBank and Generative Agents support long-term consistency across sessions.Recent systems add structural priors through agent-controlled memory selection and multi-graph reasoning.
  • Iterative and Adaptive Retrieval: Iterative retrieval methods use generated content as signals, including chain-of-thought steps, draft answers, and generation confidence.IRCoT, Iter-RetGen, and FLARE represent distinct content-driven iteration strategies.
  • Iterative and Adaptive Retrieval: Self-RAG evaluates individual passage relevance, while CRAG classifies individual documents to trigger corrective retrieval.These evaluation-driven methods assess retrieval quality at the passage or document level.
  • Iterative and Adaptive Retrieval: IRIS evaluates whether the accumulated evidence set suffices to answer the question, addressing a gap left by prior iterative methods.Prior methods evaluate generated content or individual documents rather than the sufficiency of accumulated evidence.
  • Structured Memory and Experience Storage: ReasoningBank, HippoRAG, and GraphRAG demonstrate structured experience storage or graph-based organization, but lack an iterative evidence-gap loop.HippoRAG uses single-step PageRank, while GraphRAG pre-indexes community summaries for query-focused summarization.

3 Methodology

EviMem combines LaceMem’s coarse-to-fine evidence hierarchy with IRIS’s closed-loop retrieval, which evaluates sufficiency, diagnoses missing information, and targets subsequent search. The method supports multi-hop expansion, grounded generation, calibrated stopping, and abstention when evidence remains insufficient.

  • LaceMem: LaceMem organizes dialogue into Index, Edge, and Raw layers for search, associative expansion, and verbatim grounding.Retrieval proceeds top-down from semantic tuples to graph-linked records and then full conversational context.
  • LaceMem: Atomic Index tuples enable fine-grained evidence-gap detection, while sparse same-source and semantic-similarity edges support multi-hop expansion across sessions.Each tuple preserves a link to its source turn, and the Edge layer expands initially retrieved seeds without a separate query planner.
  • IRIS evidence-gap diagnosis: IRIS classifies accumulated evidence as EXACT, INFERRABLE, or PARTIAL, assigns confidence c ∈[0, 1], and produces a natural-language description of missing information.The missing-information diagnosis drives query refinement, while per-entity fact buffers expose sparse coverage hidden by aggregate evaluation.
  • IRIS retrieval loop: IRIS iteratively retrieves through an anchor path using the original question and a refinement path using a diagnosis-driven query, then merges, deduplicates, and expands the results.The retrieval budget grows across iterations to widen search scope when earlier attempts are insufficient.
  • Confidence and answer generation: EXACT uses a 0.85 floor, INFERRABLE a 0.75 cap, and PARTIAL a 0.50 cap; IRIS terminates on sufficient evidence or abstains when evidence remains absent.When evidence is INFERRABLE or PARTIAL, an explicit multi-hop reasoning chain guides answer generation; otherwise, persistent absence triggers faithful refusal.

4 Experiments

EviMem outperforms MIRIX overall, especially on evidence-distributed temporal and multi-hop questions, while achieving substantially lower latency. Ablations show that IRIS’s iterative gap-filling and LaceMem’s graph-based memory structure jointly drive these gains.

  • Overall results: EviMem achieves the highest overall scores and leads G-EVAL in four of five categories, while MIRIX retains higher Judge Accuracy on single-hop, open-domain, and adversarial questions.The comparison uses Single-pass as an internal baseline and MIRIX as the primary external baseline.
  • Overall results: +56% relative F1 and ROUGE-L versus MIRIX mark EviMem’s largest gains, confirming improved answer precision and recall through iterative retrieval.Open-domain questions instead favor MIRIX in Judge Accuracy despite EviMem’s superior F1 because MIRIX blends parametric knowledge, whereas IRIS enforces grounding and abstains when evidence is insufficient.
  • Category analysis: Temporal and multi-hop questions benefit most because their evidence is distributed across sessions with little surface overlap, where single-pass retrieval fails.Gains are modest on single-hop questions and largest on multi-hop and temporal categories, matching their greater retrieval difficulty.
  • IRIS ablations: 82.7% temporal Judge Accuracy follows the basic iterative loop from 58.8%, while tiered sufficiency raises multi-hop accuracy from 80.1% to 87.3%.Temporal adaptation increases multi-hop F1 from 0.107 to 0.281, while the final component group raises single-hop F1 from 0.081 to 0.205 and adversarial accuracy from 50.3% to 55.1%.
  • LaceMem ablations: 85.2% multi-hop and 81.6% temporal accuracy result after adding LaceMem’s Edge layer, which also raises multi-hop accuracy from 43.2% to 85.2% and adversarial accuracy from 14.8% to 55.1%.Index-only improves multi-hop Judge Accuracy from 16.7% to 43.2% but causes open-domain and adversarial fragmentation, resolved by graph-based expansion.
  • Efficiency: 4.5× lower latency than MIRIX (9.54s vs. 42.71s) is achieved while maintaining comparable or higher accuracy; latency rises from 8.56s on single-hop to 12.11s on multi-hop questions.Single-pass costs 2.13s, and each IRIS component adds moderate overhead as computation scales with question complexity.

5 Conclusion

EviMem combines LaceMem’s coarse-to-fine memory hierarchy with IRIS’s evidence-gap-driven iterative retrieval. On LoCoMo, it improves Judge Accuracy on temporal and multi-hop questions over the reported baselines.

  • Core approach: EviMem combines LaceMem’s coarse-to-fine memory hierarchy with IRIS’s evidence-gap-driven iterative retrieval.LaceMem organizes dialogue history into Index, Edge, and Raw layers, while IRIS evaluates evidence sufficiency, diagnoses gaps, and refines queries.
  • Memory hierarchy: LaceMem structures dialogue history into Index, Edge, and Raw layers for efficient search, expansion, and grounding.
  • Results: 81.6% Judge Accuracy on temporal questions, versus 73.3% for MIRIX and 58.8% for single-pass, is achieved by EviMem on LoCoMo.
  • Results: 85.2% Judge Accuracy on multi-hop questions, versus 65.9% for MIRIX, is achieved by EviMem on LoCoMo.

Limitations

EviMem’s main limitation is that LaceMem is built from complete conversation snapshots under LoCoMo’s offline protocol, rather than online streaming dialogue. Future work should extend its modular memory construction to ingest and index new turns in real time.

  • Offline memory construction: LaceMem is currently constructed from a complete conversation snapshot under LoCoMo’s standard offline evaluation protocol.This limits evaluation to offline memory construction rather than real-time dialogue ingestion.
  • Online extension: Online streaming construction, with new dialogue turns ingested and indexed in real time, remains future work.The passage identifies this extension as a direction for future exploration.
  • Incremental updates: LaceMem’s separate Index, Edge, and Raw layers are modular and naturally amenable to incremental updates.This modularity provides architectural support for transitioning from snapshot-based to streaming memory construction.

A Detailed IRIS Algorithm · B Case Study

The section specifies the complete IRIS iterative retrieval loop and introduces four pipeline traces illustrating its operation. Algorithm 2 covers the loop’s confidence, entity, termination, and query-refinement details.

  • A Detailed IRIS Algorithm: Algorithm 2 fully specifies the IRIS iterative retrieval loop summarized in Algorithm 1.It provides the complete algorithmic specification rather than only a summary.
  • A Detailed IRIS Algorithm: The specification includes all confidence constants used by IRIS.These constants are part of the loop’s detailed configuration.
  • A Detailed IRIS Algorithm: IRIS incorporates entity-aware adjustments into its iterative retrieval process.Algorithm 2 explicitly includes these adjustments in the full loop specification.
  • A Detailed IRIS Algorithm: The algorithm defines termination thresholds for ending iterative retrieval.These thresholds are included among the loop’s specified controls.
  • A Detailed IRIS Algorithm: Algorithm 2 provides the query-refinement prompt template used by IRIS.The prompt template is part of the mechanism for refining retrieval queries.
  • B Case Study: Four pipeline traces show each step of IRIS.The case study presents these traces to demonstrate the pipeline’s step-by-step operation.

C Per-Iteration Retrieval Metrics

Table 9 presents the full per-iteration retrieval metrics for the LoCoMo benchmark, comparing diagnosis-driven refinement with a generic re-query baseline.

  • Per-Iteration Retrieval Metrics: Table 9 reports the full per-iteration retrieval metrics on the LoCoMo benchmark.These metrics are summarized in §4.3.
  • Per-Iteration Retrieval Metrics: The evaluation compares diagnosis-driven refinement against a generic re-query baseline.The comparison is conducted across the full LoCoMo benchmark.
  • Per-Iteration Retrieval Metrics: The reported metrics characterize retrieval behavior across iterations.Table 9 provides the complete per-iteration view referenced in §4.3.

D Robustness and Judge Validation · E Prompt Templates

The appendix validates EviMem’s iterative retrieval loop through robustness and judge-validation experiments, while documenting the four prompts used by IRIS and the evaluation pipeline. Results show limited sensitivity to backbone and embedding substitutions and preserved category-level behavior.

  • D Robustness and Judge Validation: IRIS iteratively retrieves, tracks entity coverage, evaluates evidence sufficiency, and refines queries using diagnosed missing information.The loop combines anchor and refinement retrieval paths, accumulates evidence, and uses sufficiency status and missing information to guide subsequent queries.
  • D Robustness and Judge Validation: Entity-aware refinement injects under-represented entity context, while strategy selection uses temporal-specific or general rule-based refinement without an LLM call.The generated prompt includes the original question, current query, missing information, iteration, strategy, and optional entity context, and requests only an improved query.
  • D Robustness and Judge Validation: The case studies show refinement recovering missing evidence for multi-hop entity tracking, temporal inference, and causal reasoning through Edge expansion.Examples include discovering Jon’s shared stress-relief method, a missing temporal tuple, and the job-loss causal link after targeted query refinement.
  • D Robustness and Judge Validation: At most 1.2 pp, overall Judge Accuracy degradation occurs under LLM-backbone or embedding substitution with the GPT-4o judge.Table 10 evaluates both substitutions on the full LoCoMo benchmark and reports cross-judge agreement for each cell.
  • D Robustness and Judge Validation: Each category shifts by ≤4 pp under either judge across robustness configurations, while bge-m3 improves Multi-hop by +3.8 pp under the GPT-4o judge.The configurations compare the baseline with DeepSeek-V3.2 backbone and bge-m3 embedding substitutions.
  • E Prompt Templates: The appendix documents four LLM prompts used by IRIS and the evaluation pipeline, including placeholders, system messages, and tier-specific variants.Placeholders are represented in braces, with system messages and tier-specific variants noted inline.

PROMPT TEMPLATE FOR SUFFICIENCY EVALUATION (IRIS) · PROMPT TEMPLATE FOR QUERY REFINEMENT (IRIS) · PROMPT TEMPLATE FOR ANSWER GENERATION (IRIS)

IRIS uses structured sufficiency evaluation to identify whether retrieved facts precisely answer a question, support inference, or remain insufficient, while explicitly recording confidence and missing information. It then refines queries through question-type- and iteration-specific strategies before generating concise, tier-adaptive answers from available evidence.

  • PROMPT TEMPLATE FOR SUFFICIENCY EVALUATION (IRIS): Temporal evaluation distinguishes precise dates or times from inferable temporal clues and vague or merely related information.Examples include “as of February 2023” and “after opening in January”; “recently” is treated as PARTIAL rather than EXACT.
  • PROMPT TEMPLATE FOR SUFFICIENCY EVALUATION (IRIS): Sufficiency evaluation labels evidence as EXACT_MATCH, INFERRABLE, or PARTIAL_MATCH and records confidence plus the specific MISSING information.The evaluator separately determines whether the facts can answer precisely, support reasonable inference, or are related but insufficient.
  • PROMPT TEMPLATE FOR SUFFICIENCY EVALUATION (IRIS): The sufficiency prompt enforces a fixed output containing EXACT, INFERRABLE, PARTIAL, CONFIDENCE, and MISSING fields.Each field must use the specified yes/no, 0.0-1.0, or missing-information format.
  • PROMPT TEMPLATE FOR QUERY REFINEMENT (IRIS): Query refinement receives the original question, current query, missing information, and iteration count, then returns only a concise improved query.The strategy is rule-based on question type and iteration.
  • PROMPT TEMPLATE FOR QUERY REFINEMENT (IRIS): Temporal queries progress from dates and time keywords to date formats, temporal relations, and broader events or milestones across iterations.The listed temporal cues include when, started, launched, opened, as of, after, and before.
  • PROMPT TEMPLATE FOR QUERY REFINEMENT (IRIS): Non-temporal queries progress from specific keywords and entities to related attributes, contextual angles, synonyms, and broader concepts.This rule-based progression changes the search angle as iterations continue.
  • PROMPT TEMPLATE FOR ANSWER GENERATION (IRIS): Answer generation combines tier-adaptive instructions with any built multi-hop reasoning context and directs the model to answer concisely and directly.The prompt is framed as answering questions from provided facts.
  • PROMPT TEMPLATE FOR ANSWER GENERATION (IRIS): Answer style varies by temporal status and confidence: precise temporal evidence requires exact dates, while inference or missing information is stated with calibrated reasoning.Non-temporal exact or high-confidence answers should be direct; lower-confidence cases should rely on reasonable inference or state missing key information.

PROMPT TEMPLATE FOR LLM-AS-JUDGE EVALUATION

The evaluation prompt asks an impartial judge to compare an assistant prediction with a ground-truth answer and determine whether it preserves the same core information. It requires a binary JSON score with a brief reason, while treating equivalent wording and date formats as acceptable and rejecting unsupported, partial, or incorrect responses.

  • Inputs: The prompt supplies the question, ground truth, and prediction for impartial correctness evaluation.These fields are presented as the judge’s comparison inputs.
  • Evaluation criteria: A prediction is correct when it conveys the ground truth’s core information, even with different wording or equivalent date formats.For example, “May 7, 2023” and “7 May 2023” are treated as equivalent.
  • Evaluation criteria: Predictions saying “I don't know” despite an existing ground truth, or omitting the key point, receive a wrong judgment.The criteria explicitly classify both unsupported non-answers and partial answers as wrong.
  • Output format: The judge must return only a JSON object containing a binary score of 1 or 0 and a brief explanation.The required format is {"score": 1 or 0, "reason": "Brief explanation"}.
Loading 2604.27695v1…