PaperlayerView exact revision on arXiv

Source-linked AI summary

Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents

Yifei Li, Weidong Guo, Lingling Zhang, Rongman Xu, Muye Huang, Hui Liu, Lijiao Xu, Yu Xu, Jun Liu

arXiv:2602.10715v1cs.CLcs.AI

TL;DR

Existing conversational-memory benchmarks largely focus on explicit factual recall, although realistic responses can depend on implicit constraints. LoCoMo-Plus and constraint-consistency evaluation target this setting; the paper reports that cognitive-memory evaluation exposes challenges not captured by surface-form recall.

  • Problem

    Existing benchmarks primarily evaluate explicit factual recall, while realistic interactions can require applying implicit user constraints that are not explicitly queried later.

    Source: §1 · p. 1

  • Method

    LoCoMo-Plus embeds cue–trigger instances in long dialogues and evaluates cognitive memory under semantic disconnect using constraint consistency.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    The paper concludes that surface-form recall misses cognitive-memory behaviors, and that current LLMs and memory-augmented systems struggle under this setting.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Conversational-memory assessment should account for whether responses respect latent constraints, not only whether they match surface forms.

    Source: §7 · p. 8

    LoCoMo-Plus does not cover all human memory or cognitive processes, excluding long-term belief revision, emotional dynamics, and multi-agent memory interactions.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Long-term conversational memory is a core capability for LLM-based dialogue systems, yet existing benchmarks and evaluation protocols primarily focus on surface-level factual recall. In realistic interactions, appropriate responses often depend on implicit constraints such as user state, goals, or values that are not explicitly queried later. To evaluate this setting, we introduce \textbf{LoCoMo-Plus}, a benchmark for assessing cognitive memory under cue--trigger semantic disconnect, where models must retain and apply latent constraints across long conversational contexts. We further show that conventional string-matching metrics and explicit task-type prompting are misaligned with such scenarios, and propose a unified evaluation framework based on constraint consistency. Experiments across diverse backbone models, retrieval-based methods, and memory systems demonstrate that cognitive memory remains challenging and reveals failures not captured by existing benchmarks. Our code and evaluation framework are publicly available at: https://github.com/xjtuleeyf/Locomo-Plus.

1 Introduction

Existing conversational-memory benchmarks primarily test explicit factual recall, while realistic responses can depend on implicit constraints carried across a conversation. LoCoMo-Plus and a constraint-consistency evaluation framework target this gap.

1 Introduction

  • Existing benchmarks largely equate conversational memory with explicit factual recall, leaving implicit constraints in natural interactions insufficiently tested.Relevant information is typically stated earlier and later queried with strong semantic alignment.Source: §1 · pp. 1–2
  • Appropriate responses may depend on retaining earlier goals or constraints even when a later query does not explicitly ask about them.The paper illustrates this with a user’s exam goal and a later question about starting a TV series.Source: §1 · pp. 1–2
  • The paper proposes constraint-consistency evaluation and reports that task-disclosed prompting and string-matching metrics can mislead by conflating memory fidelity with prompt adaptation and generation style.Experiments span diverse backbone models, retrieval-based methods, and memory systems.Source: §1 · p. 2
  • LoCoMo-Plus evaluates whether models retain and apply latent constraints across long conversations under cue–trigger semantic disconnect.The benchmark targets beyond-factual cognitive memory.Source: §1 · p. 2

2 Related Work

Prior long-term memory benchmarks commonly treat memory as retrieval of explicitly stated or semantically matchable information. Evaluation metrics and comparisons also remain limited for implicit, cognitively grounded behavior.

2.1 Benchmarks for Long-Term Memory

  • Long-term memory benchmarks and memory-augmented agents commonly retrieve or use information that is explicit or semantically matchable, limiting coverage of implicit user signals.Examples of such signals include user state, goals, and preferences.Source: §2.1 · p. 2

2.2 Evaluation Metrics and Comparability

  • Generation-based metrics can poorly reflect semantic correctness in open-ended dialogue, where multiple surface-different responses may satisfy the same memory requirement.LLM judges have been explored for comparability, but evaluation still largely focuses on explicit answer correctness.Source: §2.2 · p. 2

3 Problem Definition

The paper distinguishes factual recall from cognitive memory, which applies latent constraints inferred from prior interaction. Its evaluation treats correctness as consistency with those constraints rather than a single reference response.

3 Problem Definition

  • Level-2 Cognitive Memory is the ability to retain and apply latent, behaviorally constraining signals inferred from prior interactions.The paper identifies user state, goals, preferences, values, and causal context as examples; Figure 2 names causal, state, goal, and value components.Source: §3 · p. 3, Figure 2 · p. 3
  • Unlike Level-1 factual memory, cognitive memory does not have one ground-truth response; a response is correct when it is consistent with the latent constraint.The valid response set is defined as Ac = {a | a is consistent with c}.Source: §3 · p. 3
  • The benchmark treats cognitive cases as a higher-level evaluation axis alongside factual memory and prioritizes diagnostic coverage over scale.The paper attributes this choice to the higher generation and verification cost of cognitive cases.Source: §3 · p. 3

4 Benchmark Construction

LoCoMo-Plus constructs long-dialogue examples in which a later trigger depends on an earlier implicit cue despite semantic disconnect and intervening context. Its pipeline generates, filters, validates, and embeds cue–trigger instances.

4 Benchmark Construction

  • LoCoMo-Plus builds cue–trigger instances from scratch and embeds them in long dialogue trajectories to test latent-constraint use under long-context interference.The construction pipeline transforms raw cue dialogues into validated instances through generation, filtering, and validation.Source: §4 · p. 3, Figure 3 · p. 4

4.1 Implicit Cue Dialogue Generation

  • LLM-generated cue dialogues convey memory-relevant state, goals, preferences, constraints, or values implicitly in natural conversation.These snippets form the candidate cue pool.Source: §4.1 · p. 4

4.2 Memory-Worthy Verification

  • Manual verification retains cues that convey persistent or behaviorally constraining information, cannot be inferred trivially from local context, and could plausibly benefit an assistant if remembered.Cues failing these criteria are discarded.Source: §4.2 · p. 4

4.3 Cue–Trigger Query Construction

  • For each verified cue, an LLM generates an underspecified downstream query whose resolution depends on the cue, with a temporal-gap indicator for later embedding.Multiple responses may appear reasonable without the cue, but only cue-consistent responses are valid.Source: §4.3 · p. 4

4.4 Semantic Filtering

  • BM25 and MPNet-based similarity scores filter out cue–query pairs with high lexical or semantic similarity, including repeated, paraphrased, or directly recoverable cues.The remaining pairs form the filtered set.Source: §4.4 · p. 4

4.5 Cue Memory Elicitation Validation

  • Human annotators retain pairs only when a helpful response requires recalling and applying cue information through an implicit constraint rather than surface similarity.This validation yields the final cue–trigger set.Source: §4.5 · p. 4

4.6 Insertion into LoCoMo Dialogues

  • Validated instances are inserted into selected LoCoMo dialogues, with the trigger after a gap, so models must use the cue despite intervening turns and distractors.The insertion preserves long-dialogue structure.Source: §4.6 · p. 4, Figure 4 · p. 5

5 Evaluation Framework

LoCoMo-style evaluations can confound memory capability with responses to task prompts and surface-form overlap. The framework instead uses natural dialogue continuations and constraint-consistency judgments to assess whether responses satisfy implicit constraints.

5.1 Input-Side Assumptions: Task Disclosure

  • Task disclosure can change task-wise answering behavior and undermine comparisons by making performance sensitive to prompts rather than memory capability.Users do not announce that a query requires recalling earlier information.Source: §5.1 · p. 5

5.2 Output-Side Assumptions: Generation-Based Metrics

  • Generation-based metrics can score valid responses differently and bias model comparisons according to verbosity, reasoning style, and response length.They conflate memory fidelity with surface realization.Source: §5.2 · p. 5

5.3 Constraint-Consistency Evaluation

  • The framework presents queries as natural dialogue continuations and evaluates responses against the cue-induced constraint rather than one reference answer.It aims for coherent, comparable assessment across memory tasks and model families, covering factual and cognitive memory.Source: §5.3 · pp. 5–6
  • Correctness is membership in a valid response space, so multiple acceptable response realizations can satisfy the same implicit constraint.Judgment criteria are aligned with each task’s reasoning demands while inputs remain unified across tasks.Source: §5.3 · pp. 5–6

6 Experiments

Across evaluated methods, LoCoMo-Plus remains challenging, and its results expose prompt and output-length biases in conventional evaluation. The experiments also examine judge agreement and how performance changes with dialogue length across memory types.

6.1 Experimental Setup and Model Coverage

  • The evaluation covers context-only open- and closed-source LLMs, top-5 retrieval baselines, and three dedicated memory systems on both benchmarks.Methods use identical dialogue contexts and query formats under the same output-side protocol.Source: §6.1 · p. 6

6.2 Overall Performance and Main Findings

  • All evaluated methods show a substantial LoCoMo-to-LoCoMo-Plus performance gap, while relative differences among methods become less pronounced on LoCoMo-Plus.The pattern holds across backbones, retrieval pipelines, and dedicated memory systems, and the authors associate it with preserving implicit constraints under cue–trigger semantic disconnect.Source: §6.2 · p. 7

6.3 Evaluation Bias Analysis

  • Task disclosure shifts task-wise performance, especially for temporal and adversarial tasks, while traditional metric scores peak near reference length and decline for shorter or longer outputs.Figure 5 compares task-disclosed and unified inputs across task types, assessment methods, and model families; Figure 6 relates metric scores to output length.Source: §6.3 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 7

6.4 Reliability of LLM-as-a-Judge.

  • Human annotators and the LLM judge show strong agreement, and Gemini-2.5-Flash and GPT-4o produce small score differences across response models.Tables 2 and 3 report pairwise agreement and judge-backbone score stability, respectively.Source: §6.4 · p. 8, Table 2 · p. 8, Table 3 · p. 8

6.5 Length Sensitivity Analysis

  • As dialogue length increases, object memory remains robust, episodic memory degrades steadily, and cognitive memory performance collapses rapidly.The analysis evaluates 100 representative cases for each memory type.Source: §6.5 · p. 8, Figure 7 · p. 8

7 Conclusion

The paper concludes that long-term conversational memory evaluation cannot be reduced to surface-form recall. LoCoMo-Plus reframes assessment around constraint consistency and reveals cognitive-memory behaviors that existing benchmarks miss.

7 Conclusion

  • Surface-form recall alone is insufficient for evaluating long-term conversational memory.The conclusion frames the evaluation target as beyond-factual memory involving implicit constraints under cue–trigger semantic disconnect.Source: §7 · p. 8
  • LoCoMo-Plus reframes memory assessment as constraint consistency.The benchmark evaluates implicit constraints under cue–trigger semantic disconnect.Source: §7 · p. 8
  • The benchmark reveals cognitive-memory behaviors that existing benchmarks miss and supports evaluation through evidence-grounded LLM judges.Source: §7 · p. 8

Limitations

LoCoMo-Plus targets implicit-constraint memory in conversational settings, not all human memory or cognitive processes. Its diagnostic focus, judging approach, language, and evaluated systems bound the reported scope.

Limitations

  • LoCoMo-Plus does not model long-term belief revision, emotional dynamics, or multi-agent memory interactions.Source: Limitations · p. 9
  • Its carefully generated and validated instances prioritize diagnostic value over scale, making the dataset smaller than large factual-QA benchmarks and unsuitable for training or fine-tuning LLMs.Source: Limitations · p. 9
  • Judge results may remain sensitive to judge model and prompt, and experiments cover only English conversations and selected backbones, retrieval pipelines, and memory systems.Source: Limitations · p. 9

Ethical Considerations

The section limits the work to research and diagnostic evaluation in a controlled, synthetic or adapted-data setting, then describes data-generation prompts and task-specific judging rules. The judging scheme distinguishes partial factual recall from stricter temporal, adversarial, and cognitive-awareness criteria.

Ethical Considerations

  • The benchmark is for research and diagnosis, not real-world deployment; its dialogues and abstract constraints are not tied to identifiable real users.How persistent memory should be stored, forgotten, or governed in deployed systems is outside the paper’s scope.Source: Ethical Considerations · p. 9

A Prompting and Generation Details

  • Cue prompts introduce a recallable memory anchor, and trigger queries are conditioned on the cue and relation type while varying perspective, temporal gap, and cognitive angle.Prompts use semantically annotated components and are used only for data generation, not evaluation or judgment.Source: A Prompting and Generation Details · p. 12
  • Generation uses five named commercial LLMs, temperature T = 0.7, a 256-token maximum, and 50 instances per relation type, with no additional filtering or post-processing.The prompting and generation procedures are released in the open-source codebase for reproducibility.Source: A Prompting and Generation Details · p. 12

B Judge Design and Evaluation Protocol

  • Factual and commonsense answers receive correct, partial, or wrong labels so coarse recall is distinguished from complete failure or hallucination.For example, “seafood” is partial when the reference specifies “grilled salmon.”Source: B Judge Design and Evaluation Protocol · p. 12
  • Temporal answers and adversarial responses use binary labels because approximate temporal answers are unacceptable and hallucinated answers fail the adversarial criterion.Source: B Judge Design and Evaluation Protocol · p. 12
  • Cognitive-awareness questions use binary labels to assess whether a response acknowledges or adapts to a previously stated memory cue.Source: B Judge Design and Evaluation Protocol · p. 12

B.1 Representative Judge Examples

A representative disagreement between human annotators and an LLM judge arose from an inconsistent reference, not from the judge’s reading of the dialogue evidence.

B.1 Representative Judge Examples

  • Human annotators marked the prediction incorrect because they relied on the flawed reference answer.Source: §B.1 · p. 12
  • The LLM judge’s evidence-grounded decision matched the explicit temporal cue and identified the inconsistent weekday reference.Table 4 presents this representative bias case, where the prediction aligns with dialogue evidence but is penalized by human annotation and surface-form metrics.Source: §B.1 · p. 12, Table 4 · p. 13
  • A disagreement with human annotation does not necessarily indicate an unreliable judge, since the reference itself can be inconsistent.Source: §B.1 · p. 13

C Representative Cognitive Memory Examples

The examples illustrate cognitive memory as retaining and applying implicit personal constraints rather than reproducing an explicit fact. They cover causal, state-based, goal-oriented, and value-based relations.

C Representative Cognitive Memory Examples

Loading 2602.10715v1…