PaperlayerView exact revision on arXiv

Source-linked AI summary

When Users Don't Ask: Benchmarking Context-Driven Memory Retrieval in Conversational Agents

Wen-Yu Chang, Yun-Nung Chen

arXiv:2609.03467v1cs.CLcs.AI

TL;DR

Existing memory benchmarks largely test explicit QA rather than natural conversational use. The paper introduces LOCOMO-CONV, evaluates four conversational query styles across retrieval and response quality, and finds retrieval gaps, silent grounding, and limits of abstractive compression. It concludes that reasoning-based memory elaboration is a promising direction while documenting important evaluation and data-scope limitations.

  • Problem

    Existing evaluations primarily probe long-term conversational memory through explicit QA-style queries rather than in-situ conversational interaction.

  • Method

    LOCOMO-CONV rewrites LoCoMo questions into dialog, implicit, counterfactual, and composed queries and evaluates five memory systems on retrieval recall and end-to-end response quality.

  • Results

    Conversational framing reveals retrieval and response gaps, especially for implicit and composed queries; silent grounding shows that memory can improve implicit responses without surfacing the gold fact.

  • Takeaways & Limitations

    The findings suggest reasoning-based memory elaboration may be more promising than lossy compression for preserving details needed for grounded responses.

  • Takeaways & Limitations

    The benchmark uses LLM-generated rewrites and annotations, one primary answer-model and judge family, and only ten LoCoMo conversations.

Abstract

from arXiv · show

Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.

1 Introduction

LOCOMO-CONV reframes long-term memory evaluation around natural conversational use rather than explicit QA probing. It introduces four query styles and evaluates retrieval, response grounding, and supportive context across memory systems.

  • Benchmark motivation and contribution: LOCOMO-CONV recasts LoCoMo’s QA pool into dialog, implicit, counterfactual, and composed conversational query styles.The benchmark evaluates how memory-augmented agents use retrieved memories within natural dialogue.
  • Main findings: Conversational framing exposes retrieval failures that QA-style evaluation can mask, especially for implicit and composed queries.Multi-facet query rewriting narrows this gap for raw-turn memory but not abstractive memory.
  • Main findings: Strong retrieval does not guarantee grounded responses because abstractive compression can discard details needed for response grounding.The findings point toward reasoning-based memory elaboration rather than lossy compression.
  • Main findings: Implicit queries reveal silent grounding, where memory improves responses without explicitly surfacing the gold fact.This exposes a limitation of strict fact-recall metrics.
  • Contributions: The benchmark provides a unified evaluation framework spanning retrieval and end-to-end response generation for extractive and abstractive memory systems.It also releases supportive_memory annotations capturing conversationally useful context beyond original gold evidence.

2 Related Work

Existing long-term memory benchmarks use diverse datasets and measurement axes but predominantly rely on structured probing. LOCOMO-CONV instead evaluates how external memory is retrieved and integrated into free-form conversational responses.

  • Existing benchmarks: LoCoMo and newer benchmarks probe memory through QA, multiple-choice selection, constraint checks, or operation-level traces.These benchmarks target abilities including multi-session reasoning, temporal reasoning, updates, abstention, and personalization.
  • Existing benchmarks: PersonaMem evaluates implicit personalization but tests internalized user traits through multiple-choice selection rather than single-utterance memory surfacing.Its reported frontier-model accuracy is 37–48% on implicit personalization.
  • Existing benchmarks: MADial-Bench, LoCoMo-Plus, and AMemGym examine proactive recall, conversational consistency, or on-policy interaction in settings complementary to LOCOMO-CONV.These approaches differ in whether they require retrieval and expression of a specific verifiable fact from a user utterance.
  • LOCOMO-CONV’s distinction: Nearly all existing benchmarks use structured probing rather than free-form conversational response generation.LOCOMO-CONV departs by testing how an assistant integrates memory into a deployed-style conversational response.
  • Memory architectures: Prior memory architectures include atomic-fact stores, associative event graphs, and condensed representations of past interactions.These approaches motivate comparing different memory-construction strategies under a common conversational evaluation.

3 LoCoMo-Conv

LOCOMO-CONV rewrites LoCoMo questions into conversational forms that preserve their information needs while adding implicit, corrective, and multi-memory demands. Composed queries combine related evidence into requests requiring synthesis across atomic answers.

  • Query styles: Each LoCoMo10 question is rewritten into dialog, implicit, counterfactual, or composed conversational styles while preserving the original gold answers.The styles vary how memory is invoked, from direct first-person requests to situations requiring proactive recall.
  • Query styles: Dialog queries directly reformulate QA in first-person conversational language, whereas implicit queries require proactively inferring that relevant memory should be surfaced.Counterfactual queries add a false premise that the assistant must correct.
  • Query styles: Composed queries combine multiple source QAs into one conversational request requiring multi-memory synthesis.Their gold evidence is the union of source evidence, and their gold answer contains the constituent atomic answers.
  • Composed multi-memory clusters: Candidate composed clusters require overlapping but non-identical evidence and at least two combined evidence turns.These constraints yield 1,069 composed clusters across conversations.
  • Validation and scoring: Three annotators rated generated rewrites 4.6–4.8 of 5 for naturalness, information preservation, and intended-memory invocation on 40 items per style.Composed responses are evaluated with continuous atomic-fact coverage because binary correctness is overly strict for multi-fact synthesis.

4 Evaluation Methodology

The evaluation measures both whether systems retrieve gold dialogue evidence and whether their generated responses use memory appropriately. Response quality is judged with rubrics tailored to each conversational query style.

  • Evaluation dimensions: LOCOMO-CONV evaluates retrieval recall against gold dialogue IDs and end-to-end response quality with style-specific LLM judges.These dimensions separate memory access from the quality of conversational use.
  • Retrieval evaluation: A gold turn counts as retrieved when its verbatim text appears in a returned memory, with metadata matching additionally used for applicable abstractive systems.Recall compares top-K retrieved memories against gold evidence dialogue IDs.
  • Response evaluation: Retrieved memories are supplied to the answer model, whose responses are evaluated by an LLM judge using a style-specific rubric.This creates an end-to-end measure of how retrieved memory supports generated responses.
  • Response evaluation: Dialog and implicit responses receive partial-credit fact_used scores of 1.0, 0.5, or 0.0 based on conveyed substance and specificity.Contradictions, vague allusions, and omissions receive zero credit.
  • Response evaluation: Counterfactual responses are scored as unaware, hedge, or corrected, mapped to 0, 0.5, or 1.Composed responses use atomic-fact coverage to avoid all-or-nothing scoring across multiple facts.

5 Experimental Setup

The study evaluates five representative memory systems under unified retrieval and generation settings, using the same embedding model, backbone LLM, and top-10 memory budget.

  • Five representative systems are evaluated: AnchorMem, A-MEM, mem0, Memora3, and NaiveRAG.
  • All systems use all-MiniLM-L6-v2 for embeddings and gemma-4-31B-it as the backbone LLM.
  • Each system retrieves top-K=10 memories, which are passed to the same answer-generation model.
  • GPT-5.4-mini with reasoning enabled serves as the primary response judge, with agreement checked against Claude-sonnet-4.5 and Qwen3.6-35b-A3B on 1,488 items.

6 Results

Conversational query framing reveals retrieval weaknesses, especially for implicit and composed queries, while retrieval rankings do not consistently predict end-to-end response quality.

  • Retrieval recall: Retrieval performance drops substantially on implicit queries compared with direct dialog queries.
  • Retrieval recall: AnchorMem achieves the highest recall on dialog (0.659) and counterfactual (0.639) queries.
  • Retrieval recall: Abstractive systems outperform AnchorMem on implicit (0.456/0.445 vs. 0.368) and composed (0.374/0.387 vs. 0.279) retrieval.
  • Retrieval recall: Implicit and composed queries remain the hardest retrieval styles across every evaluated system.
  • Response quality: Response-quality rankings differ substantially from retrieval rankings: AnchorMem leads dialog and counterfactual, while Memora leads implicit and composed queries.
  • Response quality: Retrieval improvements do not consistently translate into response gains, reflecting a retrieval-to-response gap.

7 Analysis and Findings

The analyses examine query diversification, silent grounding, reasoning-based memory use, and memory construction, showing that retrieval success depends on both semantic accessibility and preservation of grounding detail.

  • Multi-Facet Query Rewriting: Conversational queries can contain multiple latent retrieval targets, making shallow single-aspect matching insufficient.
  • Multi-Facet Query Rewriting: Multi-facet rewriting improves AnchorMem retrieval by +9.5pt on dialog, +15.6pt on implicit, +9.3pt on counterfactual, and +14.7pt on composed queries.
  • Multi-Facet Query Rewriting: Rewriting degrades mem0 dialog performance, while Memora changes by at most ±0.02 on any style.
  • Silent Grounding: Silent grounding occurs when memory improves implicit-query responses through contextual grounding without explicitly surfacing the gold fact.
  • Silent Grounding: Oracle memory improves faithfulness by +55.1pt versus no memory and engagement by +31.0pt, indicating that strict fact-recall metrics can underestimate retrieval value.
  • Chain-of-Thought Selection: CoT improves dialog, implicit, and composed responses but hurts counterfactual performance because false premises frame subsequent memory use.
  • Chain-of-Thought Selection: CoT beats Oracle by +31 to +40 points on engagement while faithfulness and relevance margins remain within ±7 points.
  • Compression versus Elaboration: The comparison between compression and elaboration is not definitive because the abstractive systems differ in other architectural choices.

8 Conclusion

LOCOMO-CONV evaluates whether memory-augmented agents can invoke memory under realistic conversational framing rather than explicit QA. Its findings reveal hidden retrieval and response gaps, silent grounding, and the need for richer supportive-memory evaluation.

  • LOCOMO-CONV evaluates memory-augmented agents under realistic conversational framing rather than explicit QA-style probing.
  • Conversational framing exposes retrieval and response gaps hidden by QA evaluation, especially for implicit and composed queries.
  • Strict fact-recall metrics miss silent grounding, where memory improves implicit-query responses without explicitly surfacing the gold fact.
  • Abstractive compression can remove details needed for grounded responses, making reasoning-based memory elaboration more promising than lossy compression.
  • The released supportive_memory annotations capture conversationally supportive context beyond the original gold evidence.

Limitations and Future Work

The benchmark is limited by LLM-generated rewrites and annotations, narrow validation and evaluation settings, and a small source-conversation pool. Its source-agnostic construction supports scaling to larger or newer conversation datasets without changing the methodology.

  • LLM-based conversational rewrites and supportive_memory annotations may inherit model-specific biases.Human validation covers 40 items per style rather than the full set.
  • Evaluation relies on one open-weights answer model and one primary judge family, with cross-judge and human validation limited to subsets.
  • The benchmark uses ten LoCoMo conversations, a small pool relative to real-world long-horizon interactions.
  • Because the construction pipeline is source-agnostic, it can scale to larger or newer conversation pools without changing the methodology.
  • Future work should broaden model-family evaluation, expand conversation pools, and improve supportive-memory annotation reliability.

B Data Statistics

LOCOMO-CONV attaches conversational rewrites to LoCoMo10 QA items across four query styles. The styles vary in how memory is invoked, from direct queries to implicit, counterfactual, and multi-memory composed interactions.

  • Data statistics: Dialog and implicit styles share the full 1,986-item QA pool.
  • Data statistics: Counterfactual queries exclude 446 cat-5 adversarial items whose premises have no gold answer.
  • Composed: Composed queries combine two source QAs with overlapping gold evidence, requiring multiple memories to answer well.
  • Query generation: All four styles are generated as first-person utterances from original LoCoMo10 content.
  • Dialog: Dialog queries directly ask the assistant to retrieve a prior memory from the subject speaker’s perspective.
  • Implicit: Implicit queries present a real-life context in which the assistant should independently surface or apply relevant memory without a direct recall request.
  • Counterfactual: Counterfactual queries make the subject speaker assert a plausible but factually wrong version of a previously shared fact.
  • Composed: Composed queries describe a natural situation, plan, decision, or reflection where all listed memories are relevant without naming their gold answers.

E Qualitative Analysis

Qualitative cases show that memory can improve responses through implicit contextual grounding and supportive context, while reasoning prompts can worsen counterfactual correction by accepting false premises.

  • Silent grounding: An implicit-query response can be clearly grounded despite fact_used=0 when it synthesizes related memories without surfacing the gold fact.
  • Supportive memory: Supportive context beyond narrow gold evidence can produce a more emotionally grounded response by linking similar prior emotional states.
  • Prompting variants: CoT prompting combines a reasoning step with explicit citation and memory selection, while +reasoning isolates the reasoning-step effect using the same top-10 retrieval.
  • Speaker attribution: An unanswerable query can trigger hallucination when retrieved memory belongs to a different speaker and is incorrectly attributed to the user.
  • Counterfactual correction: The reasoning step lowers counterfactual correction scores by 0.16–0.19 for every system, while explicit selection recovers only 0.01–0.04.
  • Counterfactual correction: Reasoning prompts increase unaware responses from ∼27% under the plain prompt to 44–53% with reasoning.
  • Why reasoning hurts: A representative case shows reasoning reframing a false premise as established instead of checking it against the retrieved memory.

G Human Annotation: Protocol and Results

Human annotation used blinded, randomized Label Studio protocols to validate query construction, memory identifiability, response judgments, and evaluation reliability. Results support the benchmark’s query validity, judge agreement, and robustness across stochastic index and answer generation.

  • Annotation protocol: Three annotators rated randomized, blinded Label Studio packets, with majority votes for categorical checks and means for other ratings.Annotators did not see gold labels, judge scores, or system identities.
  • Query validation: Query validation found natural rewrites, preserved dialog information needs, valid counterfactual contradictions, and composed queries requiring both member memories.Counterfactual correction accuracy fell from 52.6% on clearly implausible premises to 38.1% on plausible ones.
  • Memory identifiability: Annotators reliably distinguished intended memories from lexically similar distractors without being told which turns were gold.This supports the identifiability of the benchmark’s memory targets.
  • Judge validation: Human labels agreed with the automatic judge at 0.72 for fact_used, 0.76 for counterfactuals, and 0.79 for composed coverage.Pairwise winner reversals were 6.7% for oracle versus no-memory and 7.3% for oracle versus random; disagreements were usually ties.
  • CoT versus oracle: Humans and the judge agreed that chain-of-thought responses were at least comparable to oracle responses, with an overall reversal rate of 2.7%.Faithfulness was essentially tied, while the remaining gap was carried by engagement.
  • Stability checks: Index rebuilds and answer sampling produced stable results, with standard deviations ≤0.011 and main rankings generally exceeding 3× their corresponding standard deviations.Three-seed index means stayed within 1pp of the main table, while five-seed response means stayed within 0.4pp; A-MEM versus mem0 on implicit fact_used was statistically indistinguishable.
Loading 2609.03467v1…