PaperlayerView exact revision on arXiv

Source-linked AI summary

UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

Peijun Qing, Fobo Shi, Soroush Vosoughi

arXiv:2608.30508v1cs.CL

TL;DR

Existing long-term memory benchmarks mainly measure pointwise factual recall, although conversational agents often need to integrate distributed, implicit, and noisy evidence into task-oriented outputs. UTILMEM introduces a 1,717-instance benchmark across five domains targeting four utilization challenges, and finds that factual-memory strength does not reliably transfer to effective utilization, especially under retrieval interference.

  • Problem

    Existing benchmarks primarily measure isolated factual recall, leaving integration of distributed, implicit, and noisy evidence into coherent task-oriented outputs under-evaluated.

  • Method

    UTILMEM constructs evidence-dependent tasks from realistic multi-session trajectories across five domains, filters for evidence sensitivity, and injects semantically similar adversarial distractors.

  • Results

    Factual-memory benchmark strength does not reliably transfer to memory utilization, and relevant retrieval still often fails to produce coherent grounded outputs under retrieval noise.

  • Takeaways & Limitations

    Effective long-term conversational memory requires using distributed evidence and resisting retrieval interference, not merely accessing stored facts.

  • Takeaways & Limitations

    UTILMEM relies on LLM-based construction and evaluation, and its evidence-sensitivity filter introduces a selection effect that limits generalization to all memory use cases.

Abstract

from arXiv · show

Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall: whether a system can recover isolated facts or event-level details from prior interactions. Real-world memory use, however, often requires a more demanding capability: integrating distributed, implicit, and noisy evidence across extended interaction histories into coherent, task-oriented outputs. We call this capability memory utilization. Here, we introduce UtilMem, a diagnostic benchmark comprising 1,717 instances across five domains, designed to evaluate four underexplored aspects of memory utilization: reasoning over dense histories, identifying implicitly relevant memories, synthesizing distributed evidence into summaries, analyses, or plans, and resisting interference from semantically similar distractors. Evaluating a diverse set of retrieval-based and memory-augmented systems, we find that strong performance on conventional factual-memory benchmarks does not reliably translate into effective memory utilization. Moreover, retrieval alone is insufficient: even when relevant evidence is successfully recovered, systems frequently fail to integrate information across sessions or to distinguish useful evidence from plausible distractors. These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational memory will require architectures that explicitly support evidence integration and robustness to retrieval interference. Code is available at https://github.com/peijunallin/UtilMem.

1 Introduction

Existing long-term memory benchmarks emphasize pointwise factual recall, while UTILMEM evaluates memory utilization: integrating distributed, implicit, and noisy evidence into coherent task-oriented outputs. The benchmark targets four challenges and finds that factual-memory strength does not reliably transfer to robust utilization.

  • Motivation: Existing benchmarks primarily test whether systems retrieve relevant spans and answer narrowly scoped factual questions.This leaves real-world memory use underconstrained when evidence is distributed or outputs require synthesis.
  • Benchmark scope: UTILMEM targets dense multi-session reasoning, implicit retrieval, long-form composition, and distractor filtering.These challenges involve temporally separated evidence, latent preferences or states, coherent synthesis, and semantically similar but irrelevant sessions.
  • Construction: UTILMEM uses realistic multi-session trajectories across learning support, finance, mental wellness, fitness coaching, and document analysis.Its synthesis pipeline creates evidence-dependent tasks, filters for answer sensitivity to evidence removal, and injects adversarial distractors differing in intent.
  • Findings: Even when relevant evidence is retrieved, models often fail to compose coherent grounded outputs and may rely on semantically plausible distractors.Such generations can be confident but unsupported.
  • Contributions: A scalable synthesis pipeline produces evidence-sensitive evaluation instances with adversarial retrieval interference.The benchmark is designed to test whether systems separate grounded evidence from plausible interference.
  • Findings: Strong factual-memory benchmark performance does not guarantee robust memory utilization.Systems can perform well on recall while failing to integrate evidence across sessions or resist retrieval noise.

2 Related Work

Prior long-term conversational memory research largely evaluates factual recall over multi-session dialogue, while memory architectures are commonly assessed with recall-oriented signals. UTILMEM addresses the resulting gap by jointly testing dense and implicit evidence, long-form composition, and semantically similar distractor filtering.

  • Existing benchmarks: Long-term memory benchmarks have largely centered on factual question answering over multi-session dialogue histories.Their abilities include information extraction and forms of multi-session, temporal, and adversarial reasoning.
  • Evaluation gap: Most benchmarks emphasize pointwise factual recall or short-answer signals, even when some include multi-session reasoning or summarization.They do not jointly couple dense, implicit evidence with long-form task-oriented composition and similar-distractor filtering.
  • UTILMEM: UTILMEM is the only benchmark in the comparison that jointly targets dense multi-session evidence, implicit retrieval, long-form composition, and distractor filtering.The comparison uses ✓ for fully supported, • for partially supported, and ✗ for unsupported dimensions.
  • Memory architectures: Explicit memory systems organize, compress, and retrieve interaction histories through streams, structured indexes, or heterogeneous memory types.These designs target storage efficiency, update consistency, relational structure, and retrieval precision.
  • Open question: Reported architectural gains are often evaluated with recall-oriented QA signals, leaving evidence-integration performance unresolved.UTILMEM provides a testbed for this distinction between fact lookup and integrated memory use.

3 Benchmark Construction

UTILMEM constructs benchmark instances from realistic multi-session histories, then tests whether questions depend on distributed evidence and whether systems can withstand semantically similar interference. Its pipeline combines evidence synthesis, question generation, counterfactual sensitivity filtering, and adversarial haystack construction.

  • Task formulation: Each benchmark instance contains a complete multi-session history, a question pool, and evidence sessions partitioned from haystack sessions.Questions are open-ended and designed to require synthesis across multiple evidence sessions.
  • Evidence synthesis: The benchmark spans learning support, finance guidance, mental wellness, fitness coaching, and document analysis through realistic conversation-bundle synthesis.Bundles undergo persona-consistency checks, conversational rewriting, and deduplication.
  • Question generation: Candidate questions target multi-session dependence, retrieval-compatible phrasing, deterministic grounded outputs, and non-judgmental organization, integration, listing, or planning.Questions are designed to benefit from aggregating information distributed across sessions while remaining answerable from conversation content.
  • Evidence-sensitivity filtering: Counterfactual degradation filtering retains questions whose answer quality decreases after sequential removal of evidence sessions.Two judges compare answers across progressively reduced contexts, requiring concordant judgments that quality degrades along the sampled removal path.
  • Distractor construction: Adversarial distractors preserve lexical and semantic similarity to evidence while shifting domain or intent, creating retrieval-interfering haystacks.In-domain and cross-domain distractors are merged with evidence sessions to separate evidence dependence from retrieval robustness.

4 Experimental Setup

The experiments compare retrieval-based and structured or agentic memory systems using a rubric-based evaluation of long-form answers. The protocol measures preservation of answer quality relative to a noise-free oracle, with normalized robustness as the headline metric.

  • Systems: The evaluation includes retrieval-only baselines alongside structured and agentic memory architectures.Retrieval baselines use NaiveRAG with multiple embedding models, while the memory systems include A-MEM, Mem0 variants, MemOS, LangMem, and EverMemOS.
  • Evaluation protocol: Because UTILMEM requires multi-session long-form synthesis, evaluation replaces exact match with fidelity to a noise-free upper-bound answer.The rubric assesses completeness preservation, hallucination absence, semantic equivalence, and ungrounded inference.
  • Metrics: Robustness Score averages per-instance scores from 1 to 10, while Normalized Robustness rescales that score to 0–100 and serves as the headline metric.The oracle reference is pinned at 10, and NR represents the percentage of clean-reference score retained.
  • Metrics: Degradation Rate measures the fraction of instances scoring at most 6, the first rubric region containing a clear user-relevant loss.Scores of 7 retain core facts and conclusions, whereas scores at or below 6 indicate bounded errors or more serious failures.
  • Implementation: Benchmark construction and evaluation use fixed temperatures and multiple judge models to support reproducibility and averaged scoring.QA generation and evaluation use temperature 0, and scores are averaged across GPT-5.2 and GPT-5.4 judges.

5 Results

UTILMEM results show that retrieving relevant evidence improves answer quality, but high recall does not ensure effective utilization. Compression-heavy memories and semantically similar distractors substantially impair grounded synthesis even when relevant evidence is available.

  • 5.1 Compression-Heavy Systems on Detail-Sensitive Utilization Tasks: 17.3 NR and 98.3 DR make Mem0 the weakest system overall, while Mem0+Graph reaches only 18.9 NR and 97.8 DR.Graph structure does not recover contextual and reasoning information discarded during fact extraction.
  • 5.1 Compression-Heavy Systems on Detail-Sensitive Utilization Tasks: 56.0 NR and 54.4 DR make A-MEM the strongest structured system, but it still lags behind raw-turn retrieval.MemOS, LangMem, and Ever-MemOS fall between 40.7–48.4 NR and 69.0–80.6 DR.
  • 5.2 Recall Is Necessary but Not Sufficient for Utilization: Mean RS rises monotonically with unit-level recall, confirming that UTILMEM is evidence-sensitive across domains.The analysis defines recall as the fraction of ground-truth evidence sessions retrieved in the top-k.
  • 5.2 Recall Is Necessary but Not Sufficient for Utilization: At recall ≥0.8, mean RS remains 6.4–7.9 versus the clean-reference score of 10, with degradation affecting 53% of Mental Wellness and 29% of Document Analysis instances.These high-recall failures indicate that retrieval alone does not close the utilization gap.
  • 5.3 Sensitivity to Retrieval Interference: At k = 40 under strong noise, NR falls to 47.4, 56.0, and 57.2 for Qwen3-30B, Qwen3-235B, and Qwen3-Max, respectively.DR concurrently rises to 73.6%, 57.6%, and 53.2%, despite recall@k remaining 1.0.
  • 5.3 Sensitivity to Retrieval Interference: Under strong noise, k75% is 1.5, 1.7, and 2.2 for Qwen3-30B, Qwen3-235B, and Qwen3-Max, showing that the first few distractors cause most damage.Marginal NR loss peaks around k ∈[2, 8] and rapidly saturates after approximately k = 10.

6 Conclusion

UTILMEM operationalizes long-term memory utilization as producing grounded, task-oriented outputs from distributed evidence under retrieval interference. Across tested systems, factual-recall strength does not reliably transfer to utilization quality, while turn-level evidence preservation performs better on detail-sensitive tasks, subject to the benchmark’s selection effect.

  • 6 Conclusion: UTILMEM systematically evaluates using distributed evidence to produce grounded, task-oriented outputs under retrieval interference.
  • 6 Conclusion: Factual-recall strength does not reliably transfer to utilization quality across the tested systems and configurations.High recall can coexist with integration failures, and semantically similar distractors can substantially degrade performance.
  • 6 Conclusion: Systems preserving more turn-level evidence outperform the compression-heavy systems tested on these detail-sensitive tasks.The conclusion limits this result to the tested systems and detail-sensitive benchmark setting.
  • 6 Conclusion: The benchmark’s evidence-sensitivity filter contributes a selection effect, so the result should not be generalized to all memory use cases.
  • 6 Conclusion: UTILMEM is intended to support human-grounded validation, causal decomposition of writing, retrieval, and generation failures, and tests of evidence-preserving interventions.

7 Limitations

UTILMEM’s LLM-based construction and evaluation introduce reference, rubric, human-evaluation, and evidence-sensitivity constraints that bound how its results should be interpreted.

  • LLM-based construction and evaluation may inherit stylistic regularities, biases, and reasoning preferences from the models used.This affects rewriting, question generation, distractor synthesis, oracle answers, filtering, and scoring.
  • The oracle-evidence answer is a protocol reference rather than a human optimum, and long-form tasks permit multiple valid organizations and emphases.Absolute scores therefore depend on reference and rubric stability.
  • Large-scale human evaluation remains challenging because long-context, multi-session reasoning tasks require substantial annotation effort.Two GPT-family judges nevertheless show strong agreement in system ranking and instance-level scores.
  • The evidence-sensitivity procedure tests sampled sequential removal paths but does not prove every session is indispensable, monotonic degradation, or minimal retained evidence sets.Conclusions should be interpreted as evidence about UTILMEM-style memory utilization under this procedure.

Ethics Statement

UTILMEM’s Mental Wellness data and intended use impose explicit ethical and clinical boundaries, including limits on diagnostic interpretation and re-identification claims.

  • UTILMEM is intended for research evaluation, not clinical diagnosis, treatment, crisis triage, or other clinical decision-making.Its Mental Wellness domain uses an existing publicly accessible Reddit-derived dataset without treating self-reported conditions as clinically verified diagnoses.
  • Public accessibility of the Reddit-derived posts should not be interpreted as consent for unrestricted reuse.The posts undergo an LLM-mediated transformation pipeline.
  • Transformed conversations may retain contextual details, so UTILMEM does not claim that re-identification is impossible.Users should not attempt to identify or contact the original authors.

A Benchmark Statistics

UTILMEM increases evidence multiplicity and distractor pressure while keeping its overall token budget close to LONGMEMEVAL-S, enabling more direct structural comparison without causal isolation.

  • Evidence session counts: UTILMEM domains require medians of 6, 3, 4, 6, and 5 evidence sessions, versus 2 in LONGMEMEVAL-S.Most samples contain 3–7 designated evidence sessions, increasing the multi-session evidence burden.
  • Haystack composition: Strong and weak distractors raise average histories to roughly 60–70 sessions, while strong distractors dominate every domain by construction.This makes UTILMEM a retrieval-robustness stress test rather than a recall-only benchmark.
  • Haystack composition: 46K evidence tokens per sample make Document Analysis the largest evidence footprint, compared with 4K–10K in the other four domains.The difference reflects the length of 10-K filing items.
  • Total token budget: 120K tokens is UTILMEM’s mean per-sample budget, closely matching LONGMEMEVAL-S at 122K.This reduces raw context length as an obvious explanation for performance differences, while other benchmark and implementation differences remain uncontrolled.

B Models Used Across Benchmark Construction and Evaluation

UTILMEM instantiates a four-step synthesis pipeline across five source datasets, using staged model calls to construct multi-session histories, questions, distractors, and evaluations.

  • Model settings: Temperatures are 0.7 for conversational synthesis and distractor generation, and 0 for QA generation and evaluation.These settings are summarized for the models used at each benchmark stage.
  • Pipeline overview: The pipeline produces a bundle B = {s1, . . . , sK} of multi-turn sessions representing one user’s history.Bundle metadata is passed to downstream question-generation and haystack-construction stages.
  • Learning Support: StudyChat bundles sample one user, K topics, and one chat per topic, enforcing persona consistency through a shared user ID.Previously accepted chat-ID sets are rejected.
  • Finance Guidance: Finance bundles sample categories and QA pairs, reject conflicting demographics or life stages, then expand rewritten pairs into T-turn dialogues.Previously accepted source row-ID sets are rejected before LLM calls.
  • Mental Wellness: Mental Wellness bundles sample subreddit posts, screen persona consistency, and rewrite accepted bodies into conversational sequences.The source subreddits include ADHD, Aspergers, depression, OCD, and ptsd.
  • Fitness Coaching: Fitness sessions are generated longitudinally, with each new session conditioned on the full prior history and assigned a timestamp consistent with its time-gap description.The resulting timeline supplies the temporal anchor for the shared history H.
  • Document Analysis: Document Analysis samples items from a single S&P 500 10-K filing, preserving persona consistency because all sessions concern the same filing.Short items are discarded before sampling.

D Representative Benchmark Instances Across Five Domains

Representative queries span five domains and require combining information across conversations into grounded summaries, analyses, or plans. The examples emphasize longitudinal progression, multi-constraint planning, cross-section synthesis, and explicit grounding constraints.

  • Learning Support: Learning Support queries combine coursework topics, algorithms, datasets, libraries, experiments, and reporting tasks into unified study materials.Examples cover binary trees, kidney-disease data, regression, MLP tuning, RNNs, search algorithms, pandas, and scikit-learn.
  • Finance Guidance: Finance Guidance queries require plans that jointly address debt, liquidity, housing, vehicles, career transitions, deadlines, and children’s savings.The examples connect immediate financial decisions with longer-term goals and administrative follow-up.
  • Mental Wellness: Mental Wellness queries integrate overlapping symptoms, personal experiences, relationship context, environmental stressors, and explicit non-diagnosis constraints.Some examples also request exposure planning or therapist-ready summaries adapted to health and life circumstances.
  • Fitness Coaching: Fitness Coaching queries reconstruct exercise progression and combine training dosage with nutrition, recovery, symptoms, laboratory values, and audience-aware reporting.The requested outputs adapt recommendations or summaries to rehabilitation history, daily routines, and coach or doctor audiences.
  • Document Analysis: Document Analysis queries synthesize accounting estimates, timing assumptions, liabilities, controls, ERP implementation, and auditor opinions across 10-K sections.The examples frame the task as either a cross-sectional synthesis or a structured guide for reading a filing.
  • Evaluation Reference and Validation: The benchmark’s reference score is 10 by definition, while 82.94% of judge pairs differ by at most one point and 93.65% by at most two.The shared oracle reference is a protocol normalization anchor rather than a human optimum, and correlations above 0.84 support agreement in score movement and ranking.

E.4 Qualitative Mem0 Failure Cases

The Mem0 case studies show that non-empty retrieval can still yield severe failures when memories are lossy or semantically misinterpreted. The evaluation protocol uses judge prompts and score capping to expose hallucination, mixed-context, and refusal failures.

  • Failure Cases: Mem0 failure analysis uses non-empty retrieved memories and valid scores to distinguish lossy or distractor-attracted retrieval from empty-store or parsing failures.The authors explicitly state that these cases provide case-level evidence rather than a complete implementation audit.
  • Learning Support: Creative-Writing Leakage: A Learning Support query received score 1.5 when creative-writing memories caused an AI-assignment summary to mix fictional and coursework content.The judges assigned scores [1, 2] and tagged the output for hallucination and mixed context.
  • Finance: Game Context Causes Refusal: A Finance query can be misread as a Monopoly-style game, leading the model to refuse creation of a practical real-life financial plan.The retrieved memories include emergency liquidity, cash tracking, properties, rent income, and cash flow, but the output frames them as game references.
  • Evaluation Protocol: The evaluation judges first classify five severity dimensions, then assign a 1–10 anchor-based score capped by the worst dimension.This severity-then-anchor design prevents unaffected strengths from offsetting a severe hallucination or contradiction.
  • Benchmark Construction: Question generation uses one shared prompt across five domains plus domain templates, while evidence filtering retains candidates only after four concordant pairwise verdicts.Distractor synthesis separately generates adversarial directions and instantiates them into multi-turn conversations across domains.
Loading 2608.30508v1…