PaperlayerView exact revision on arXiv

Source-linked AI summary

MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

Ryuichi Sumida, Koji Inoue, Tatsuya Kawahara

arXiv:2608.24189v1cs.CLcs.HC

TL;DR

Conversational memory is usually evaluated by Direct QA, but it is unclear whether recalling facts on request predicts user satisfaction. This paper introduces MEMUSE, which evaluates natural integration in real user-cued moments, and finds retrieval and conversational use sharply diverge. Natural Integration, rather than Direct QA, is associated with satisfaction in these moments.

  • Problem

    Existing memory benchmarks test elicited retrieval but do not establish whether Direct QA performance translates into user satisfaction.

  • Method

    The paper combines a four-month, seven-condition deployment with MEMUSE, a benchmark of real user-cued moments scored for natural conversational integration.

  • Results

    19.7% to 70.1%: Direct QA accuracy varies across conditions without changing satisfaction, while the same system shows a 71-point retrieval–integration gap.

  • Takeaways & Limitations

    Memory evaluation should assess whether responses naturally use user-referenced context, not only whether systems retrieve facts when asked.

  • Takeaways & Limitations

    MEMUSE detects explicit-cue memory moments, so its approximately 3.5% rate is a lower bound that misses implicit moments.

Abstract

from arXiv · show

Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmark Direct QA varies from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response). To examine this, we introduce MemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation -- a 71-point gap. Within these moments, Natural Integration is associated with satisfaction, whereas Direct QA is not. We release the deployment corpus and MemUse together with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.

1 Introduction

Long-term memory benchmarks typically test elicited retrieval, but the paper argues that conversational usefulness depends on naturally integrating prior context. A four-month deployment motivates MEMUSE, which evaluates real user-cued memory use and reveals a retrieval–integration gap.

  • Motivation: Existing benchmarks ask whether a model can recall a prior fact, assuming elicited retrieval reflects natural conversational use.The paper distinguishes recalling fact X when asked from detecting relevance and weaving prior knowledge into conversation.
  • MEMUSE: MEMUSE evaluates 72 real user-cued memory moments by judging whether natural responses demonstrate memory of the referenced topic.Its items come from deployment conversations rather than externally authored fact-seeking questions.
  • Empirical puzzle: Across seven memory conditions, Direct QA accuracy ranges from 19.7% to 70.1%, yet user satisfaction does not change.The deployment included 40 users and 1,872 sessions over four months.
  • Benchmark design: MEMUSE combines multi-month human-AI deployment, naturally cued evaluation items, and per-session satisfaction ratings.The table contrasts these properties with existing long-term memory benchmarks.
  • Key finding: The same system scores 78.8% on Direct QA but references those facts conversationally only 7.9%, a 71-point gap.This dissociation separates retrieval performance from natural integration.

2 Related Work

Prior memory benchmarks and memory-augmented systems emphasize Direct QA, capacity, retrieval, and long-context prompting. The paper situates MEMUSE within broader concerns that automatic accuracy may not capture human judgment or user experience.

  • Memory Benchmarks: LoCoMo, RealTalk, LUFY, and LongMemEval use synthetic or externally authored Direct QA and do not validate prediction of user experience.These benchmarks evaluate fact retrieval rather than naturally cued conversational memory use.
  • Memory-Augmented Systems: Memory-augmented dialogue systems combine summarization, retrieval, long-context prompting, and explicit memory management.Selective forgetting is used to instantiate the study’s long-context and retrieval capacity range.
  • Memory-Augmented Systems: Long-context prompting can cause models to ignore or degrade on relevant evidence, a risk extended here to natural conversational integration.The paper connects this prior finding to its retrieval–integration gap.
  • Dialogue Evaluation Beyond Accuracy: Dialogue research documents disconnects between automatic metrics and human judgment, alongside work measuring satisfaction and long-term chatbot relationships.This motivates evaluating memory beyond benchmark accuracy.

3 Experimental Setup

The study deployed a GPT-4.1-mini diary companion with seven randomly rotated memory conditions for 40 users over four months. Analyses used balanced exposure and a predefined satisfaction-filtering pipeline.

  • Deployment: Forty users interacted daily with a GPT-4.1-mini AI diary companion from November 2025 to February 2026.The deployment recorded 1,872 sessions through a web interface allowing users to review prior-conversation summaries.
  • Memory Conditions: Sessions were randomly assigned to SUMMARY, LC10/50/100%, or RAG-10/50/100% memory conditions.Long-context conditions prepend important prior turns, while RAG conditions retrieve top turns from the corresponding pool.
  • Memory Conditions: The importance-ranking model exceeded human annotator averages for both important-utterance and not-important identification.It achieved 56.3% versus 41.4% and 93.1% versus 91.3%, respectively, on held-out evaluation.
  • Analysis: Satisfaction analysis excluded 11 users and yielded 1,270 sessions from 29 users for within-user z-scored analysis.Exclusions covered near-constant ratings and flagged template or AI-paste content.

4 Capacity Improves Direct QA but Not Satisfaction

Increasing memory capacity sharply improves existing-benchmark Direct QA, but adding long-context or retrieval capacity beyond a shared summary does not improve user satisfaction. Engagement features predict satisfaction independently, while memory condition does not.

  • Direct QA: 19.7% to 70.1%: average existing-benchmark QA accuracy rises across the seven conditions as capacity increases.The range runs from Summary to LC-100%.
  • Satisfaction: Less than 0.06 within-user SD: every condition’s mean satisfaction differs from Summary-only by less than this amount.LC-100% is not better than Summary; Summary is directionally slightly ahead at +0.02 versus −0.04 SD.
  • Interpretation: The null concerns the marginal value of LC/RAG capacity added beyond a common summary, not memory in general.All seven conditions include the same summary baseline.
  • Satisfaction: Each engagement feature independently predicts higher satisfaction, whereas memory condition does not.The features are response length, response specificity, and cross-session continuity.
  • Robustness: Neither mean, maximum, nor minimum latency, nor their time-to-first-token counterparts, correlates with within-user ratings.On the 29-user subset, all absolute correlations are below 0.06 with p ≥ .07.

5 Detecting and Scoring Memory Moments

The study audits rare, user-cued memory moments in deployment and packages reactive cases into MEMUSE, scored for conversational integration alongside retrieval and reference metrics.

  • Detecting Memory Moments: 147 verified memory events were detected across 1,872 sessions at 95.5% precision.The events comprised 11 probes, 62 re-provisions, 66 proactive recalls, and 8 reactions.
  • Detecting Memory Moments: Memory events occurred in only ∼3.5% of sessions and 1.4% of user turns.This corresponds to approximately one detected memory moment per 73 user turns, versus 15–24% of turns in existing benchmarks.
  • The MEMUSE Benchmark: Reactive cases are benchmarked because users provide a clear per-instance target, whereas proactive recalls lack a canonical next response.Only reactive integration was related to session-level satisfaction in the deployment.
  • The MEMUSE Benchmark: MEMUSE packages 72 reactive moments—62 re-provisions and 10 probes—with 316 fact-seeking questions.Each instance contains 3–5 ground-truth questions, averaging 4.4 questions per instance.
  • The MEMUSE Benchmark: MEMUSE scores Natural Integration as its primary metric, alongside Direct QA and Reference on the same reconstructed context.Reference checks whether facts elicited by Direct QA appear in the natural conversational response.
  • Scoring Validation: LLM-judge validation found substantial human–human agreement on Natural Integration and matching positive rates between the judge and humans.On a stratified subset, Cohen’s κ was 0.57 and positive rates were 51.8% for both LLM and humans.

6 Results

Across deployment and controlled evaluations, retrieval capacity does not translate into natural conversational memory use or satisfaction. User-cued integration is associated with satisfaction, while proactive recalls show timing-related downside without a reliable positive signal.

  • 6.2 Q2: Retrieval and Integration Are Dissociable: Retrieval rose sharply with memory capacity, while integration stayed flat and satisfaction tracked neither across seven conditions.This pattern motivates separating elicited retrieval from natural integration.
  • 6.1 Q1: User-Cued Integration Is Associated With Satisfaction: Direct QA accuracy was uncorrelated with satisfaction (ρ = +0.03), whereas Natural Integration was associated with satisfaction (ρ = +0.29, p = .046).Successful integration carried +0.56 within-user SD higher satisfaction, with a 95% CI of [+0.12, +0.98].
  • 6.1 Q1: User-Cued Integration Is Associated With Satisfaction: LC-100% ranked highest on Direct QA at 78.8% but lowest on Natural Integration at 22.2%.The same model and context therefore produced sharply different rankings across retrieval and integration metrics.
  • 6.2 Q2: Retrieval and Integration Are Dissociable: GPT-4.1-mini achieved 70.1% existing-benchmark QA accuracy with LC-100% context but only 22.2% MEMUSE Natural Integration.Every prior turn was included in LC-100%, yet the model often recovered facts only when explicitly asked.
  • 6.2 Q2: Retrieval and Integration Are Dissociable: Across GPT-5.5 and Gemini 3.1 Pro, Direct QA rose 32–34 points with capacity, while Natural Integration varied by at most 8.2 points.Stronger models raised the integration baseline, but capacity-driven retrieval gains did not produce capacity-driven integration gains.
  • 6.2 Q2: Retrieval and Integration Are Dissociable: Mem0 and Letta raised Natural Integration to 58.3% and 56.9%, respectively, but retained large Direct QA–Reference gaps.The gaps were 34 points for Mem0 and 51 points for Letta; the offline evaluation makes no satisfaction claim.
  • 6.2 Q2: Retrieval and Integration Are Dissociable: Prompt interventions raised Natural Integration levels but left every variant roughly flat across the seven memory conditions.The integration variants spanned at most 21.9 points, compared with Direct QA’s approximately 33-point span.
  • 6.2 Q2: Retrieval and Integration Are Dissociable: The Two-step ablation found conversational generation failed to reference details in 77% of cases even when extraction named them correctly.This localizes the bottleneck downstream of retrieval, in conversational generation.

7 Conclusion

The paper presents a four-month deployment paired with MEMUSE to evaluate memory under integration-aware, user-cued conditions. Direct QA accuracy increased across conditions without changing satisfaction, motivating evaluation aligned with conversational integration.

  • 40 users completed a four-month, seven-condition deployment paired with the MEMUSE benchmark for integration-aware memory evaluation.
  • 19.7% to 70.1%: Direct QA accuracy rose across conditions while user satisfaction did not change.
  • The authors propose realigning memory evaluation with user experience through integration-aware, user-cued benchmarks.

Limitations

The findings are bounded by the summary baseline, explicit-cue construction, small observational evidence, automated binary measurement, participant composition, diary setting, and release constraints.

  • The null result concerns additional long-context or retrieval capacity beyond a shared summary baseline, not memory in general.
  • MEMUSE detects explicit memory cues, so its approximately 3.5% event rate is a lower bound that misses implicit memory-relevant moments.
  • The integration–satisfaction association is based on 48 sessions across 16 users and uses an observational, non-randomized integration variable.
  • Natural Integration is binary, while partial-integration measures were not designed or validated as ordinal metrics.
  • The predominantly female English-speaking sample and diary setting may not generalize to other longitudinal interaction types.
  • Data-release constraints, including summarized rather than raw text in some cases, may limit downstream replicability.

Ethics Statement

The study obtained informed consent, protected participant privacy, and released de-identified research materials under stated withdrawal and data-handling procedures. The deployment used a web-based AI diary with session summaries and documented generation and analysis procedures.

  • Participants gave informed consent for research use and public release, with self-curated disclosures and a one-month withdrawal window.
  • Names, locations, workplaces, contact information, and flagged spans were reviewed and redacted before release.
  • Participants were recruited through a Japanese consumer participant pool and compensated JPY 3,000 per hour.
  • Human annotators were recruited through the same platform and compensated at the same hourly rate.
  • Users accessed the AI Diary through mobile- and desktop-compatible web browsers and could review prior-conversation summaries before sessions.
  • GPT-4.1-mini generated session, daily, and monthly summaries, while only the running session summary was fed into subsequent model context.
  • The deployed and offline systems shared persona, memory context, and response constraints, with GPT-5.4 used additionally as an offline capability probe.
  • The deployment averaged 46.8 sessions per user, maintained balanced condition assignment, and measured continuity and specificity using defined text-based procedures.

B.3 Benchmark Evaluation Pipeline

The benchmark pipeline evaluates existing Direct QA and MEMUSE under reconstructed memory contexts, using standardized generation and judging procedures. MEMUSE emphasizes whether natural responses integrate user-cued memories, with judge validation and documented sensitivity.

  • Existing benchmarks were evaluated across seven memory conditions using GPT-4.1-mini for generation and judging with condition-specific reconstructed contexts.
  • Natural Integration is the primary metric: it judges whether a natural response demonstrates memory of the referenced topic rather than matching a specific token.
  • MEMUSE reconstructs contexts from deployment conversations and evaluates GPT-4.1-mini and GPT-5.4 generations with GPT-5.4-nano binary judging.
  • 1 of 72 MEMUSE instances had at least 50% summary coverage, while 60 had 10–50% and 11 had below 10%.
  • Human validation found substantial agreement for Natural Integration after recalibration, while the original round showed strong sensitivity to rater strictness.
  • The GPT-5.4 comparison is counterfactual because users did not experience those outputs, so it only tests whether judge choice changes the sign.

C.1 Detailed Per-Condition Results

Across users and filtering choices, memory-condition effects on satisfaction remain negligible, while Natural Integration shows a positive but lower-power observational association with satisfaction in memory-moment sessions.

  • Per-condition effects: β = −0.001: condition-level QA accuracy does not predict z-scored satisfaction across 1,270 sessions.Kruskal–Wallis, permutation tests, negligible pairwise effects, and equivalence testing support the same null.
  • Per-condition effects: ρ = +0.008: matched-model Natural Integration is uncorrelated with session-level satisfaction at deployment scale.No significant effects emerge for memory moments, conditions, or their interaction.
  • Sensitivity analyses: All filtering subsets retain the null: condition effects are uniformly negligible, with 95% intervals spanning zero.The analysis covers the full sample, an adversarial-user exclusion, and the original 29-user filter.
  • User heterogeneity: Only 11 of 29 users show a directional preference for memory conditions, with random-slope variance of 0.042.This indicates that the population-level null does not reflect a uniformly shaped subgroup effect.
  • Continuity analysis: Cross-session continuity predicts satisfaction under full-history, fixed-window, and history-size-normalized definitions, despite partial mechanical confounding.All three operationalizations independently predict satisfaction after the mechanical trend is removed.
  • Memory-moment association: β = +0.556: Natural Integration has conditional evidence of higher satisfaction across 48 memory-moment sessions, but the lower-power observational test is not uniformly decisive.The Wald interval straddles zero, while a clustered bootstrap excludes zero; the estimate remains stable across aggregation rules.
  • Interpretation and power: The integration-to-satisfaction association is observational, whereas randomization supports causal claims only for the population-level null direct effect.Engagement-dimension analyses likewise involve non-randomized variables.
  • Interpretation and power: 84.5% power is achieved for d = 0.25, but only 20.9% for d = 0.10; detecting the smaller effect would require approximately 70 users.The study is substantially less sensitive to small effects.

E.3 Prompt and Architectural Ablations

Prompt and architectural interventions produce limited or fragile gains in Natural Integration, with the main bottleneck appearing downstream of retrieval and concentrated in implicit user references.

  • E.3.1 Cross-Variant Overview: The Direct QA-to-Natural Integration continuum declines from 78.8% to 51% for probes and 21% for re-provisions.Re-provisions are substantially harder than semi-explicit memory probes.
  • E.3.1 Cross-Variant Overview: Integration variants span at most 21.9 points across conditions, versus Direct QA’s approximately 33-point span, and remain roughly flat with capacity.Some variants raise Natural Integration levels without making the gains capacity-driven.
  • E.3.4 Single-Call Prompt Variants (V1–V5): V1–V4 reach 28.8%–34.2% Natural Integration on LC-100%, but none differs significantly from V0 after Bonferroni correction.Reference rates increase by only 0.6–1.9 points.
  • E.3.5 Two-Step Extract-then-Integrate Pipeline (V6): The Two-step ablation fails to reference named ground-truth details in 77% of cases even when extraction succeeds.This localizes the bottleneck downstream of retrieval, in conversational generation.
  • E.3.6 Query-Rewrite + Targeted Retrieval (V7): Query rewriting can hurt because users’ personal-memory triggers already contain lexical anchors of the relevant prior turns.Rewritten-query performance remains approximately 48%, identical to V0, regardless of retrieval recall.
  • E.3.8 Holistic Judge Fragility on Recall-Style Prompts: V5 and V2-CoT lift Summary-only Natural Integration by 16.4 points despite no prior facts being available, indicating judge fragility.The interaction with context-bearing conditions is statistically indistinguishable from zero.
  • E.3.8 Holistic Judge Fragility on Recall-Style Prompts: Holistic Natural Integration has lower inter-judge agreement than per-question Reference, with Fleiss κ = .26 versus .65.The reported GPT-5.5 lift is therefore suggestive of a real conditional gain atop judge fragility, not definitive evidence.
  • E.3.9 Event-Type Split (Probe vs. Re-provision): Prompt fixes amplify semi-explicit recall, whereas closing implicit-integration failures requires architectural changes and still yields small absolute improvements.The event-type split explains why interventions behave differently across probes and re-provisions.

F.6 Proactive Recalls: Methods and Extended Analyses

Proactive recalls were evaluated for grounding, appropriateness, and links to satisfaction, with robustness analyses testing whether timing, judge choice, or latency altered the conclusions.

  • Satisfaction association: Reactive events were associated with higher session satisfaction than proactive recalls, with z-scored means of +0.09 versus −0.14.Figure 15 reports the session-level contrast as p = .086.
  • Satisfaction association: The only pronounced proactive satisfaction pattern was low appropriateness; better grounding or appropriateness scores otherwise did not translate into higher satisfaction.Table 27 summarizes 64 events across 24 users and 63 unique sessions.
  • Robustness analyses: Using the same Natural Integration judge and prompt, proactive grounding was uncorrelated with satisfaction (ρ = −0.20, p = .12), unlike reactive integration (ρ = +0.29, p = .046).A second judge reproduced null directional associations, while grounding agreement was substantial and appropriateness agreement was fair.
  • Timing and appropriateness: Well-timed recalls filled conversational openings, whereas mistimed recalls were grounded but disconnected from the user’s immediately preceding turn.The examples contrast an invited movie callback with an unsolicited discussion of caregiving and self-care at session close.
  • Cost and latency: Latency did not explain the capacity satisfaction null: every within-condition correlation had |ρ| ≤ 0.08 and p > .33, including maximum latency.The deployment spanned roughly 1.3 seconds of mean-latency variance, with single-turn maxima reaching 34.3 seconds.

H Data Release Notes

The release provides the deployment corpus, MEMUSE benchmark, judgments, prompts, and analysis code under licenses aligned with non-commercial research use.

  • Released materials: The release includes 1,872 sessions from 40 users across 7 memory conditions, 72 de-identified MEMUSE instances, 316 questions, Natural Integration judgments, scoring prompts, and evaluation code.The deployment corpus includes per-session satisfaction ratings, while the benchmark includes ground-truth facts.
  • Licensing and intended use: The corpus and MEMUSE benchmark use CC BY-NC 4.0, while evaluation and analysis code use the MIT license; conversational content cannot be commercially redistributed.The data artifacts are intended exclusively for non-commercial research use under the participant consent procedures.
Loading 2608.24189v1…