PaperlayerView exact revision on arXiv

Source-linked AI summary

Relevance Is Not Sufficiency: What Actually Closes the Evidence Gap in Long-Term Memory QA

Yufeng Li, Shuxin Li, Zhenhua Xu, Junxian Li, Peng Zeng, Sheng Yao, Changting Lin, Gaolei Li, Ran Bi, Meng Han

arXiv:2610.09348v1cs.AI

TL;DR

Long-term memory QA must find evidence that jointly supports an answer, since relevance-ranked records can omit complementary facts. BFR selects complementary evidence and acquires unseen records under a fixed budget; across LoCoMo and LongMemEval-S it improves answer quality and evidence coverage over same-store baselines.

  • Problem

    Relevance-ranked memories can each concern the right topic yet omit a complementary fact needed to answer multi-record questions.

    Source: §1 · p. 2

  • Method

    BFR builds evidence sets over a flat memory store by selecting a compact, requirement-covering subset and acquiring unseen records under a fixed retrieval budget.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    BFR improves answer quality and evidence coverage over same-store baselines on LoCoMo and LongMemEval-S; on LongMemEval-S, judged accuracy rises from 72.4% to 82.2%.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    Long-term memory retrieval can be framed as constructing a set whose members collectively satisfy the information requirements for an answer.

    Source: §6 · p. 10

    Stage I is bounded by its initial candidate pool, so required records outside that pool need further acquisition.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

LLM agents that interact with a user across many sessions accumulate histories that exceed their context window, so they store past interactions in an external memory and answer each question from a small set of retrieved records. Existing memory systems rank records by lexical or embedding relevance, yet the top-ranked memories can each be relevant while jointly omitting a complementary fact that the answer requires, especially for multi-session and temporal questions. Drawing on the distinction between relevance and sufficiency in legal evidence scholarship, we recast memory retrieval as constructing a sufficient memory set. To operationalize this view, we introduce a blinded LLM judgment over the retrieved set, together with Gold Hit and Turn Hit as evidence-coverage proxies. We then propose Budgeted Flat Reconstruction (BFR), which builds sufficient sets over a fixed flat memory store in two stages. Specifically, we first apply Formal Concept Analysis for Memory Selection (FCA-MS) to decompose the question into information requirements and select a compact candidate subset that jointly covers them. Then, we repeatedly acquire unseen records through deeper text search or complementary entity and session views, stopping when the budget is exhausted. Experiments on LoCoMo and LongMemEval-S show that BFR outperforms same-store adaptations of recent agent-memory systems in both answer quality and evidence coverage. Specifically, on LongMemEval-S it raises judged accuracy from 72.4% to 82.2% and Turn Hit to 91.4%.

1 INTRODUCTION

Long-term memory QA must retrieve records that jointly support an answer, not merely records that are individually relevant. The introduction proposes BFR to construct such evidence sets and reports gains in answer quality and evidence coverage.

1 INTRODUCTION

  • Relevant memories can still omit a complementary fact needed for an answer, because relevance applies to individual records rather than the retrieved set.The shutter-release question requires both the February 5 order and February 10 arrival; the arrival falls below the top-k cutoff, leaving the five-day interval unsupported.Source: §1 · p. 2
  • The retrieval challenge has two parts: combine complementary facts within the available candidates and reach necessary facts outside the initial candidate pool.At inference time, no oracle identifies sufficiency, so retrieval must also decide how far to search and when to stop.Source: §1 · p. 2
  • BFR constructs a sufficient memory set over a flat store through requirement-aware selection followed by budgeted acquisition of further evidence.FCA-MS decomposes the question into requirements and selects a compact complementary subset; Stage II acquires unseen records using text or multi-view retrieval.Source: §1 · p. 2
  • On LongMemEval-S, BFR raises judged accuracy from 72.4% to 82.2% and attains a 91.4% Turn Hit.Across LoCoMo and LongMemEval-S, BFR improves judged answer quality and annotated evidence coverage over same-store baselines.Source: §1 · p. 3
  • The paper makes set-level sufficiency measurable with LLM-Suff@Set and uses Gold Hit and Turn Hit as proxies for annotated evidence coverage.These measures distinguish whether a retrieved set supports an answer from whether it reaches annotated answer-bearing evidence.Source: §1 · p. 3
  • The paper evaluates BFR on LoCoMo and LongMemEval-S against same-store baselines.The reported experiments examine answer quality and evidence coverage.Source: §1 · p. 3

2 RELATED WORK

Prior work improves memory representation, retrieval, or organization, while this paper frames the objective as assembling a set that covers answer requirements.

2 RELATED WORK

  • Evidence scholarship distinguishes relevance of an individual item from sufficiency of the complete evidentiary record; this paper applies that distinction constructively to memory retrieval.The paper positions sufficient-context evaluation in RAG as diagnostic, distinguishing missing context from failures to use available context.Source: §2 · p. 3
  • Memory systems use varied structures and access strategies, including salient facts, linked notes, temporal graphs, hierarchical storage, conversation segments, chains, and memory graphs.These approaches transform histories or organize relevant candidates before or during retrieval.Source: §2 · p. 3
  • BFR differs by assessing whether the resulting set covers all answer requirements and acquiring missing support without changing the underlying store.The related systems described improve what can be retrieved or how relevant candidates are organized.Source: §2 · p. 3

3 WHAT COUNTS AS A SUFFICIENT CANDIDATE MEMORY SET

The section defines sufficiency at the level of the retrieved evidence set and distinguishes it from reaching annotated evidence. It illustrates why both dates may be needed to support a temporal answer.

3 WHAT COUNTS AS A SUFFICIENT CANDIDATE MEMORY SET

  • The shutter-release delivery question requires both dates to derive the five-day answer, illustrating why relevant records alone may not form sufficient evidence.The figure contrasts similarity-ranked top-k retrieval with BFR’s selection of complementary records and budgeted completion through text, entity, and session views.Source: §3 · p. 3, Table 1 · p. 4, Figure 2 · p. 4
  • Memory QA treats item-wise relevance and set-level sufficiency as different properties: evidence is sufficient when the assembled records make the answer determinable.A blinded, reference-conditioned LLM adjudication operationalizes this set-level property as LLM-Suff@Set.Source: §3 · p. 4
  • The operational sufficiency assessment is a blinded, reference-conditioned LLM judgment of whether the evidence makes the answer answerable, independently of a particular answerer’s response.The adjudication uses the question, reference answer, and rendered evidence text.Source: §3 · p. 4
  • Gold Hit and Turn Hit measure reaching at least one annotated supporting record or answer-bearing turn, not whether the retrieved set suffices.A hit is necessary for sufficiency but does not establish it; the section reports these proxies alongside the set-level judgment.Source: §3 · p. 4

4 BFR: COMPOSING AND COMPLETING SUFFICIENT EVIDENCE

BFR frames retrieval as constructing a sufficient evidence set over a fixed memory store under a retrieval-call budget. It first composes complementary evidence, then completes the set by acquiring unseen records through scheduled retrieval access.

4.1 PROBLEM: CONSTRUCT A SUFFICIENT SET UNDER A RETRIEVAL BUDGET

  • BFR maximizes expected set sufficiency under a retrieval-call budget while holding the memory store and answerer fixed.Composition covers distinct answer requirements; completion allows a missing exhibit to enter after the first page.Source: §4.1 · pp. 4–5

4.3 STAGE II: COMPLETE MISSING SUPPORT UNDER A BUDGET

  • Stage II retrieves through specified views, excludes records already in Et, and retains novel support until the retrieval budget is exhausted.The budget limits rounds, calls, and newly retained records; exhaustion is used because offline sufficiency measures are unavailable at inference time.Source: §4.1 · p. 5, §4.3 · p. 6

4.2 STAGE I: COMPOSE COMPLEMENTARY EVIDENCE WITH FCA-MS

  • Stage I expands retrieval beyond the usual top-k, decomposes the question into requirements, prefilters candidates with FCA, and selects a compact subset that jointly covers them.The selected notes are grounded to source records to form E0; the initial top-k is a reference cutoff, not a retained base set.Source: §4.2 · pp. 5–6, Figure 3 · p. 5

4.4 ACCESS SCHEDULES FOR COMPLETION

  • BFR-Text repeatedly searches the text index with the original question, while BFR-MV adds entity and session access paths to recover unseen records.BFR-Text advances beyond the initial page; BFR-MV uses the first named person and latest retained record as cues when available.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments compare BFR with same-store memory systems on LOCOMO and LONGMEMEVAL-S, testing answer quality, evidence sufficiency, and retrieval cost. BFR leads overall answer quality on both benchmarks, while additional analyses examine evidence recovery and the cost–quality tradeoff.

RQ1: Does BFR improve answer quality under the same memory store?

  • BFR achieves the best overall answer quality among same-store methods on both benchmarks, leading LOCOMO Stem F1 and Judge across Multi-Hop, Temporal, and Single-Hop, and LONGMEMEVAL-S overall Judge and Turn Hit.Its LONGMEMEVAL-S gain is particularly strong on Temporal questions; the small LOCOMO Open Domain slice has no separate type-level conclusion.Source: Table 2 · p. 7, RQ1: Does BFR improve answer quality… · p. 7, Table 3 · p. 8

5.1 EXPERIMENTAL SETUP

  • Evaluation uses a stratified 200-question LOCOMO subset and all 500 LONGMEMEVAL-S questions, with same-store baselines, a fixed answerer, and a fixed judge within each benchmark.Answer quality uses Porter stem F1 and LLM-Judge on LOCOMO and Judge on LONGMEMEVAL-S; evidence measures complement rather than replace answer scores.Source: §5.1 · p. 7

5 EXPERIMENTS

  • Table 6 reports an ablation of stages and access schedules.The supplied table passage identifies the ablation scope but provides no outcome details.Source: Table 6 · p. 8

RQ2: Do the retrieved sets contain the evidence needed for an answer?

  • On LOCOMO, completion improves complete evidence recovery, but BFR-Text has higher CER while BFR-MV has higher LLM-Suff@Set; A-Mem has higher Turn Hit but lower sufficiency than Mem0 on LONGMEMEVAL-S.The comparison shows annotated-evidence access and set-level sufficiency do not always rank systems alike.Source: RQ2: Do the retrieved sets contain the… · p. 8, Table 5 · p. 9

RQ4: What does additional retrieval cost buy?

  • Most answer-quality gain appears after the first completion call; at three matched calls, BFR-Text exceeds BFR-MV on all four plotted metrics, while BFR-MV peaks at C9 near nine calls.BFR-Text C4 and C6 coincide at four realized calls; the main tables retain C6 as the fixed default rather than a sweep-selected maximum.Source: Figure 5 · p. 9, RQ4: What does additional retrieval… · p. 9

6 CONCLUSION AND DISCUSSION

BFR treats long-term memory retrieval as assembling evidence that jointly supports an answer, rather than ranking individually relevant records. The evidence-law mapping distinguishes composition of complementary facts from completion beyond the initial set, illustrated by a case where the missing arrival date changes the answer.

6 CONCLUSION AND DISCUSSION

  • Across LoCoMo and LongMemEval-S, budgeted completion raises annotated-evidence reach over one-shot ranking, and full BFR further improves coverage and judged answer quality over same-store baselines.Ablations attribute most gains to broader evidence access rather than adaptive state control; gold-free sufficiency estimates remain too unreliable to serve as stopping criteria.Source: §6 · p. 10
  • The authors report using language models for method and evaluation tasks, and say they inspected outputs, reran computations, and made final methodological and editorial decisions.They also report that public benchmark conversations, questions, and reference answers were not generated or altered by these tools.Source: AI USE STATEMENT · p. 11

A EVIDENCE-LAW MAPPING

  • Relevance applies to an individual memory, while sufficiency asks whether the assembled record can support an answer; Gold Hit and Turn Hit proxy annotated-evidence coverage, not sufficiency itself.The mapping is conceptual, not a claim that agent memory should mimic courtroom procedure.Source: A EVIDENCE-LAW MAPPING · p. 15

B CASE TRACE FOR FIGURE 1

  • In the shutter-release case, the order record and other photography memories miss the arrival date, so together they cannot establish the delivery interval.The answer becomes “5 days” when BFR-MV retrieves both required turns; the first two sets have 50% evidence recall.Source: B CASE TRACE FOR FIGURE 1 · p. 15, Table 7 · p. 15

C STRUCTURED MEMORY BOUNDARY

The structured-memory comparison is a boundary rather than a fair same-store baseline because the systems differ in stores, derived facts, and tools. The appendix also specifies Stage I selection and Stage II budgeted completion over flat records.

C STRUCTURED MEMORY BOUNDARY

  • The cross-system slice is a boundary, not a fair same-store comparison, because store, derived facts, and tools change together.The comparison also omits Open Domain questions.Source: C STRUCTURED MEMORY BOUNDARY · p. 16, Table 8 · p. 16
  • Full-MRAgent reaches 88.9% Gold Hit and 95.1% judged accuracy, while MRAgent-Flat reaches 67.9% / 72.8%, BFR-Text 77.8% / 81.5%, and CoM 80.2% / 85.2%.On the overlap slice, Full-MRAgent is judged correct alone on 20 questions versus 2 for MRAgent-Flat; mean evidence-set sizes are 2.4, 14.2, and 34.0 for Full, Flat, and BFR-Text, respectively.Source: C STRUCTURED MEMORY BOUNDARY · p. 16, Table 9 · p. 16

D ALGORITHMIC SPECIFICATION

  • Stage I expands the candidate pool, decomposes the question into requirements, applies an FCA prefilter, selects a compact subset, and grounds it to source records for the initial evidence set.The appendix identifies this stage as Formal Concept Analysis for Memory Selection (FCA-MS).Source: D ALGORITHMIC SPECIFICATION · p. 17
  • Stage II repeatedly retrieves unseen records, merges candidates from the scheduled views, applies a retention rule, and updates the evidence set until a step, call, or new-record budget is exhausted.BFR-Text advances text search; BFR-MV diversifies access through text, entity, and session cues.Source: D ALGORITHMIC SPECIFICATION · p. 17

E IMPLEMENTATION DETAILS

The implementation adapts memory systems to a shared flat conversational-turn store and uses a two-stage initialization and completion process. The reported LoCoMo results measure stability across conversations, while LongMemEval-S was held until the protocol and metrics were frozen.

E IMPLEMENTATION DETAILS

  • Within each benchmark, adapted controls share the flat turn store, extractive answerer, and judge; starred CoM and MRAgent variants are controlled same-store adaptations, not official full systems.Mem0 uses BM25-lite, A-Mem dense top-k without note evolution, CoM uses MiniLM candidates with anchor, chain, and blocking, and MRAgent-Flat omits the graph and derived facts.Source: E IMPLEMENTATION DETAILS · p. 17
  • FCA-MS initialization retrieves 50 candidates, prefilters to at most 20 notes, and selects a set of 1–5 notes before mapping them to 10 source turns for Stage II.The pilot's selected sets average 1.86 notes; source turns are ranked by positive token overlap, and direct source pointers could replace this approximate alignment.Source: E IMPLEMENTATION DETAILS · p. 18
  • The default BFR completion point allows 3 rounds, 6 retrieval calls, and 30 newly retained records; BFR-Text takes unseen text results, while BFR-MV uses a fixed text→entity→session schedule.The schedule excludes current evidence and continues until the observable budget is exhausted.Source: E IMPLEMENTATION DETAILS · p. 18

F DATA USAGE AND SEALING

  • All ten LoCoMo conversations were visible during development, so its results measure stability across conversations rather than generalization to unseen data.LongMemEval-S was not inspected until the LoCoMo protocol, arm definitions, budget grid, and metrics were frozen.Source: F DATA USAGE AND SEALING · p. 18

G DATASET PROVENANCE AND DERIVED EVALUATION SETS

The evaluation uses the released LoCoMo and LongMemEval-S benchmarks, alongside derived LoCoMo subsets and retrieval evidence states. The same-store controls are evaluated on a shared fixed pilot, with incomplete outputs retained in the denominator.

G DATASET PROVENANCE AND DERIVED EVALUATION SETS

  • LoCoMo provides ten long, multi-session conversations with QA annotations and, when available, supporting dialogue-turn identifiers; the study uses the released content without altering reference answers.The benchmark was introduced by Maharana et al. (2024).Source: G DATASET PROVENANCE AND DERIVED… · p. 18
  • LongMemEval-S contains 500 questions across single-session, preference, multi-session, temporal, and knowledge-update categories, and the study evaluates the complete split.Turn Hit uses all 500 questions, counting the 21 without annotated answer turns as misses; 479 have at least one such turn.Source: G DATASET PROVENANCE AND DERIVED… · p. 18
  • The 200-question LoCoMo pilot is a fixed stratified development subset, while the 1,536-question gold subset supports development robustness across conversations rather than held-out evaluation.The conv-30 slice is an 81-question overlap set, not independently sampled, and excludes Open Domain.Source: G DATASET PROVENANCE AND DERIVED… · p. 18
  • The blinded LLM-Suff@Set analysis evaluates seven arms over the same 200 pilot questions, yielding 1,400 method–question evaluations of retrieval outputs rather than additional conversations.Identical evidence states are deduplicated by question and evidence hash.Source: G DATASET PROVENANCE AND DERIVED… · pp. 18–19

H SAME-STORE LOCOMO CONTROLS

  • CoM completes all 200 pilot questions, while MRAgent-Flat completes 199; its empty-evidence question remains in the denominator as an empty prediction.These rows are distinct from the 81-question full-system comparison.Source: H SAME-STORE LOCOMO CONTROLS · p. 19

I BUDGET SWEEP

The budget sweep compares fixed completion schedules across operating points and tests whether evidence-state control adds to budgeted retrieval.

I BUDGET SWEEP

  • BFR-Text −FCA-MS is +4.0 points (p = 0.332), while BFR-MV −FCA-MS is 0.0 points (p = 1.000); no type-specific LLM-Suff@Set difference survives Holm correction.Source: I BUDGET SWEEP · p. 20
  • Across the full grid, BFR-Text occupies the three-call frontier for Judge, F1, LLM-Suff@Set, and Complete Evidence Recall; BFR-MV reaches its highest Judge score at C9.BFR-MV also contributes non-dominated Judge points at C4 and C9.Source: I BUDGET SWEEP · p. 21
  • At Bmid on LOCOMO, BFR-State and BFR-Adaptive reach 73.0% Gold Hit versus 73.5% for fixed BFR-MV, with a clustered 95% CI of [−0.8, 4.5] for State minus MV.The reported ablation does not establish a gain over fixed access.Source: Table 18 · p. 21, I BUDGET SWEEP · pp. 21–22

J DEVELOPMENT ROBUSTNESS

The appendices report development-contact robustness on LOCOMO and a question-type breakdown of LONGMEMEVAL-S Turn Hit.

J DEVELOPMENT ROBUSTNESS

  • On 1,536 development-contact LOCOMO questions, five-fold analysis raises Gold Hit from 54.8% initially to 78.0% with text and 78.1% with multi-view completion.All conversations had development contact.Source: J DEVELOPMENT ROBUSTNESS · p. 22, Table 19 · p. 22

K FULL LONGMEMEVAL-S TYPE BREAKDOWN

  • The LONGMEMEVAL-S type breakdown presents the multi-session and temporal retrieval-coverage pattern underlying the answer-quality results in Table 3.Source: K FULL LONGMEMEVAL-S TYPE BREAKDOWN · p. 22
  • Table 20 reports Turn Hit for six question types and overall, alongside judged accuracy over all questions.Unannotated answer-turn items count as misses in the all-question measure.Source: Table 20 · p. 22

L COMPARISON WITH LEARNED RESIDUAL SET COMPLETION

The section compares learned residual completion with fixed schedules, defines the evidence measures, and reports a matched-budget comparison of text and multi-view retrieval.

L COMPARISON WITH LEARNED RESIDUAL SET COMPLETION

M ANNOTATION PROTOCOL

  • Blinded Suff@Set judging uses the question, reference answer, and anonymized evidence, but not the method, budget, system answer, or gold identifiers.It is a reference-conditioned offline measure, not a human annotation or inference-time stopping signal.Source: M ANNOTATION PROTOCOL · p. 23

N METRIC DEFINITIONS

  • Suff@Set tests whether evidence supports the complete answer; Gold Hit and Turn Hit track any annotated support record or turn, while Complete Evidence Recall requires all annotated records.Source: N METRIC DEFINITIONS · p. 23

P MATCHED THREE-CALL COMPARISON OF BFR-MV AND BFR-TEXT

Loading 2610.09348v1…