PaperlayerView exact revision on arXiv

Source-linked AI summary

SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents

Wenxuan Wang, Haoyu Sun, Fukuan Hou, Mingyang Song, Weinan Zhang, Yu Cheng, Yang Yang

arXiv:2606.05761v2cs.AIcs.CL

TL;DR

Persistent AI assistants must distinguish relations among similar, diverging, and conflicting memories, but existing evaluations rarely test this capability. SubtleMemory benchmarks relational discrimination and finds current systems weak, especially on contradictory memories even with oracle evidence.

  • Problem

    Persistent assistants accumulate related memories whose subtle relations must be preserved and distinguished for appropriate behavior.

  • Method

    SubtleMemory embeds controlled complementary, nuanced, and contradictory relations in natural histories and diagnoses preservation, retrieval, and reasoning.

  • Results

    Current systems struggle across preservation, retrieval, and reasoning, with contradictory memories remaining dramatically harder even under oracle evidence.

  • Takeaways & Limitations

    SubtleMemory exposes distributed weaknesses in relational memory use and provides a focused testbed for improving long-horizon memory agents.

  • Takeaways & Limitations

    The benchmark covers text-based histories and three controlled relation types, while evaluation depends on selected answer-generation and judging models.

Abstract

from arXiv · show

Persistent AI assistants, such as OpenClaw, accumulate large collections of related memories over long-term interactions. As these memories grow, they may reinforce one another, diverge across contexts, or directly conflict, making correct assistance depend on memory relations rather than isolated recall. Existing long-term memory benchmarks rarely probe how agents preserve and utilize such relations during downstream tasks. To address this gap, we introduce SubtleMemory, a benchmark for fine-grained relational memory discrimination in long-running AI agents. SubtleMemory constructs relation-controlled latent semantic artifacts whose variants instantiate complementary, nuanced, or contradictory relations, and embeds them into realistic user-agent histories, requiring agents to recover distributed relational structures during later queries and instructions. The benchmark contains 1,522 evaluation instances over 10 long histories, grounded in 1,090 relation-controlled memory-variant sets and spanning user-related and non-user-related queries. Evaluating six standalone memory systems, two Claw-style agents with native memory modules, and three Claw-style agents with plugin memory modules, we find that current systems remain weak on fine-grained relational memory discrimination. We further introduce diagnostic protocols that reveal distinct capability profiles across memory preservation, retrieval, and downstream reasoning stages.

1 Introduction

SubtleMemory targets a neglected challenge in persistent AI assistants: preserving and distinguishing subtle relations among similar long-term memories during later task execution. It introduces a relation-controlled benchmark and unified diagnostics, showing that current systems struggle especially with contradictory memories.

  • Motivation: Persistent assistants must preserve and distinguish relations among related memories that may reinforce, diverge across contexts, or conflict, rather than merely recall isolated facts.Correct assistance depends on relations among accumulated memories during continuity, personalization, and informed decision-making.
  • Research gap: Existing long-term memory benchmarks mainly test individual-memory retrieval or manipulation and rarely evaluate relational discrimination among multiple related memories during later tasks.ClawArena evaluates memory evolution over long-horizon interactions but does not systematically probe discrimination among related memory items.
  • Benchmark: SubtleMemory embeds complementary, nuanced, or contradictory relation-controlled semantic variants into natural multi-turn, long-horizon interaction histories.The variants share resolution targets and determine whether downstream reasoning should aggregate, distinguish, or reconcile related memories.
  • Contributions: The paper provides a unified evaluation framework spanning standalone, framework-native, and plugin-based memory agents, plus diagnostics for construction, retrieval, and final response failures.These contributions support consistent evaluation and more precise analysis of where long-term memory systems break down.
  • Key findings: Contradictory-memory instances remain dramatically harder than complementary or nuanced instances, even with oracle evidence, frontier models such as gpt-5.4, and optimized prompting.Across six standalone systems, six native-memory Claw-style agents, and three plugin-memory Claw-style agents, systems struggle to preserve relational information and retrieve sufficient task-relevant evidence.

2 Related Work

Long-term assistant memory has evolved from external support for personalization and experience reuse toward explicit memory-management systems and Claw-style deployment forms. However, existing evaluations rarely test whether agents preserve and use relations among multiple relevant memories during downstream tasks.

  • Early long-term assistant agents used external memory for personalization, reflection, experience reuse, and open-ended task continuation (Park et al., 2023; Zhong et al., 2024; Shinn et al., 2023; Zhao et al., 2024; Wang et al., 2024; Majumder et al., 2023).
  • Recent systems treat memory management as an explicit design problem, while Claw-style agents use either native runtime memory or plugin modules that inject recalled information into context (Xia et al., 2026).
  • Relation-sensitive behavior remains largely implicit in retrieval scores, summaries, links, or routing decisions, leaving unclear whether current memory-aware agents can discriminate among related memories.
  • Long-context and long-term memory benchmarks increasingly evaluate retrieval, reasoning, updating, applying, or abstaining over information in multi-session, agent-facing settings (Bai et al., 2024; An et al., 2024).
  • Existing benchmarks rarely test target-conditioned sets of related memories that must be distinguished, aggregated, or surfaced as conflicts during later task execution.Compatible memories should be aggregated, highly similar memories separated by context or time, and inconsistent memories surfaced as unresolved conflict.

3 Methodology

SubtleMemory models relational memory reasoning by constructing latent semantic artifacts from controlled variants of user-related and non-user facts. It embeds these artifacts in scattered, naturalistic long-horizon conversations and evaluates whether agents preserve, retrieve, discriminate, reconcile, and correctly resolve the underlying relations.

  • Semantic Primitives: Semantic seeds cover user-related facts and non-user knowledge, while controlled variants transform each seed through detail enrichment, selective masking, or semantically neighboring content.The resulting variants are selected for a target and relation before being instantiated into benchmark memories.
  • Semantic Primitives: The benchmark defines each latent semantic artifact as a resolution target, target-conditioned variant set, and compatibility relation governing how memories should be reasoned over.Artifacts encode complementary, nuanced, or contradictory relations that respectively require evidence aggregation, fine-grained discrimination, or conflict handling.
  • User-history: Variants become natural multi-turn task-oriented sessions distributed across separated conversation sessions, simulating relevant memories scattered throughout long-running assistant histories.This design hides related evidence among unrelated interactions rather than exposing facts as isolated memory entries.
  • Memory Injection: For each memory system, the benchmark replays the history according to that system’s memory-formation granularity, producing a final memory state for evaluation.Systems may form memories at different granularities, such as message-level units or message batches.
  • Evaluation Instance: Each query requires agents to recover hidden relations from raw histories, retrieve relevant evidence, generate an answer, and receive an LLM-judged binary correctness label against relation-specific answer guidelines.Agents never observe the latent artifacts directly, so evaluation covers preservation during construction as well as retrieval, discrimination, reconciliation, and downstream reasoning.

4 Experiments

Experiments evaluate standalone memory systems, Claw-style agents, and plugin-augmented agents on SubtleMemory, using calibrated answer-generation models and diagnostic oracle settings. Results show substantial memory-related gaps, with contradictory and temporal discrimination remaining especially difficult.

  • Main results: Under gpt-5.4, A-Mem leads standalone systems at 70.0%, followed by Mem0 at 69.0% and EverMemOS at 68.1%, but remains over 15 points below Oracle at 85.4%.The gap persists across relation types.
  • Relation-specific results: Contradictory relations are hardest: under Oracle access, GPT-5.4 reaches 68.7% and gpt-oss-120b 41.6%, while A-Mem reaches 50.4% and trails Oracle by 18.3 points.These gaps indicate that memory mechanisms and base-model reasoning both remain limiting factors.
  • Answer-generation calibration: Oracle calibration identifies gpt-5.4 with the strong prompt as best at 90.1%, yet contradictory cases remain harder than complementary and nuanced cases.Oracle access bypasses memory extraction and retrieval, isolating answer-generation limitations; this configuration becomes the default for main experiments.
  • Agent-runtime integration: Native OpenClaw reaches 62.5% under gpt-5.4, while plugins raise it to 69.1% with EverMemOS and 71.3% with Mem0; runtime effects remain task- and model-dependent.Standalone systems generally outperform native Claw-style agents, while agent organization improves MemOS on complementary cases by 7.2% but reduces contradictory performance.
  • Temporal discrimination: Under gpt-5.4 and gpt-oss-120b, 10 and 9 of 11 systems, respectively, outperform temporal discrimination on contextual-detail discrimination, unlike Oracle.The contrast identifies temporal reasoning and temporal-aware memory organization as important improvement opportunities.

5 Discussion

The discussion uses a staged waterfall analysis to distinguish memory-preservation and retrieval failures, showing that both jointly shape downstream accuracy. It further links strong preservation to retaining raw interaction sessions and identifies relation-specific retrieval bottlenecks.

  • Diagnostic Protocol: The staged waterfall analysis separates preservation from retrieval failures by comparing oracle, perfect-retrieval, and default-setting correctness.P_preserve measures the share of oracle-correct instances retaining sufficient information, while P_retrieve measures conditional success under default retrieval.
  • Empirical Observations and Insights: MemoBase illustrates the joint impact of both stages, with 39.1% preservation, 75.6% retrieval, and 32.1% overall accuracy.The result shows that relatively strong retrieval cannot compensate for weak memory preservation.
  • Empirical Observations and Insights: A-Mem and OpenClaw achieve the strongest overall preservation rates, at 93.5% and 91.5%, respectively.Both systems preserve original interaction sessions alongside structured memory states, retaining fine-grained cues that compressed abstractions may lose.
  • Empirical Observations and Insights: Relation types expose distinct retrieval bottlenecks: nuanced relations are comparatively easier because they mainly require matching the target memory entry.Complementary and contradictory relations are described as having different retrieval demands, indicating that bottlenecks vary by relation structure.

6 Conclusion

SubtleMemory is a benchmark for fine-grained relational memory discrimination in long-horizon AI agents. It embeds relation-controlled semantic artifacts in user histories and tests recovery of their relations from natural interaction traces.

  • SubtleMemory evaluates fine-grained relational memory discrimination in long-horizon AI agents.
  • The benchmark embeds latent relation-controlled semantic artifacts into user histories.
  • Agents must recover complementary, nuanced, and contradictory relations from natural interaction traces.

Limitations

SubtleMemory measures relational memory discrimination in text-based, long-horizon histories using a controlled relation taxonomy, but its coverage and evaluation depend on constrained settings and models.

  • Scope: SubtleMemory is limited to text-based long-horizon assistant histories and complementary, nuanced, and contradictory memory relations.This controlled design makes fine-grained relational discrimination measurable.
  • Scope: The benchmark does not cover multimodal memories, multilingual interactions, or highly domain-specific workflows.
  • Evaluation: Evaluation depends on the selected answer-generation and judging models, leaving extensions to future work.

Ethical Considerations · Appendix Overview

The paper addresses benchmark-related risks and data-safety practices, while the appendices document SubtleMemory’s formulation, construction pipeline, and experimental protocols. No human participants, crowdworkers, or external annotators were involved.

  • Ethical Considerations: SubtleMemory may invite leaderboard over-optimization that limits generalization, while LLM-judge evaluation can introduce bias or occasional misjudgments.
  • Ethical Considerations: Synthetic histories, generated annotations, and curated seed data were filtered and manually inspected to remove identifying, offensive, or inappropriate content.
  • Ethical Considerations: The study involved no recruited participants, crowdworkers, or external annotators; authors performed manual inspection and annotation, so participant instructions and consent procedures were unnecessary.
  • Appendix Overview: The appendix supplies supporting material on benchmark definition, data construction, and experimental analysis.
  • Appendix Overview: Appendix A compares SubtleMemory with representative long-term memory benchmarks and explains the relation-sensitive capabilities targeted in Table 4.
  • Appendix Overview: Appendix B expands the task formulation through relation taxonomies, evaluation details, and examples of embedding latent semantic variants into user-agent sessions.
  • Appendix Overview: Appendix C documents the five-stage dataset construction pipeline, including seed selection, variant creation, session and instance construction, history assembly, prompts, examples, and statistics.
  • Appendix Overview: Appendix D reports experimental settings, context organization, judge validation, prompts, oracle and perfect-retrieval protocols, model configuration, statistics, case studies, and answer examples.

A Positioning of SubtleMemory

SubtleMemory is a benchmark for fine-grained relational memory discrimination during downstream reasoning. It distinguishes complementary, nuanced, and contradictory memory relations while separating preservation, retrieval, and answer-generation behavior.

  • SubtleMemory evaluates fine-grained relational memory discrimination by organizing instances around resolution targets and target-conditioned semantic variant sets.The benchmark explicitly controls whether relevant memories are complementary, nuanced, or contradictory.
  • Its relation tests assess target-conditioned related memories, complementary aggregation, nuanced context/time discrimination, and contradictory conflict preservation.
  • Its diagnostic dimensions separate memory preservation, retrieval, and answer-generation behavior to characterize relation-sensitive memory use.
  • Table 4 compares SubtleMemory with representative benchmarks spanning multi-session conversation, long-term memory QA, personalization, memory-grounded tool use, and Claw-style agent evaluation.

B Methodology … C.5 Stage 5: User-history Assembly

SubtleMemory builds relation-controlled semantic variant sets from user-related preferences and external knowledge, embeds them into realistic task-oriented histories, and evaluates target-conditioned retrieval and reasoning. Its pipeline spans seed selection, relation-specific variant creation, session and query construction, filtering, and final history assembly.

  • B.1 Preliminary Concepts and Relation Taxonomy: The benchmark distinguishes complementary, nuanced, and contradictory memories, requiring aggregation of compatible facts, target-specific selection, or clarification when memories conflict.Complementary variants jointly support one answer; nuanced variants differ by context; contradictory variants cannot be jointly satisfied under the same target.
  • C.3 Stage 3: Session Construction: Variants are embedded as recoverable information in natural task-oriented dialogues rather than exposed as isolated memory entries, with user and non-user cases distributing evidence across sessions.User sessions vary conversation type, workflow rhythm, message count, persona-signal level, and same-set history, while non-user sessions preserve distributed or redundant evidence according to relation type.
  • C.1 Stage 1: Semantic Seed Selection: User-related seeds retain sanitized persona backgrounds and active preferences from PersonaMem-v2, while non-user seeds are filtered from multi-evidence, repeated-answer, contextual, contradictory, and temporal source records.Ten personas with distinct backgrounds and application domains are selected; external sources include FanOutQA, MuSiQue, QACC, AmbigQA, and HoH under relation-supporting evidence criteria.
  • C.2 Stage 2: Semantic Variant Creation: Relation planning assigns user preferences compatibility types and subtypes before generation, while source-family rules create multi-evidence, any-one, contextual, temporal, or contradictory non-user variant sets.Contradictory AmbigQA sets select conflicting QA entries and remove qualifiers that could reconcile them.
  • User-related session-generation prompt excerpt: Session generation and filtering enforce naturalness, target recoverability, hidden supervision, relation fidelity, and boundaries that prevent labels or conflicts from being revealed prematurely.The assigned user-related variant is expressed through concrete signals, while other same-set variants remain relation context; contradictory non-user sessions embed one target claim without announcing or resolving the conflict.
  • Non-user complementary fact-selection prompt excerpt: Prompted fact-selection and generation interfaces convert source records into benchmark-ready complementary, contradictory, contextual, and temporal bundles while filters validate factual support and relation fidelity.The prompts create canonical conflict questions, ambiguous context questions, complementary fact bundles, and target-conditioned user-related variants.
  • C.4 Stage 4: Evaluation Instance Construction: Evaluation instances use self-contained, realistic, underdetermined queries with relation-specific answer rules, including structured-form or resource-arrangement tasks for user-related cases.Non-user questions preserve hidden dependencies, decisive context or temporal conditions, and unresolved contradictions; each query receives three correct and three incorrect candidates.
  • C.5 Stage 5: User-history Assembly: Final assembly removes non-user facts too close to user memories, assigns remaining sets to personas without reuse, balances relation types, and redistributes timestamps within each persona’s history.When no user-related time span exists, timestamps are placed within a fixed date range.

C.6 Final data composition. … D.4 Main Experiment Case Studies

SubtleMemory’s final data composition and evaluation pipeline combine relation-controlled histories, filtered query-answer instances, native memory-agent execution, and validated semantic judging. Diagnostic protocols separate failures in memory formation, retrieval, and answer generation, while case studies illustrate how distributed or context-dependent facts must be recovered.

  • C.6 Final data composition.: 88.86% of user-related and 73.23% of non-user candidate semantic variant sets pass all construction filters, with Tables 7–8 reporting source, relation, and domain composition.The benchmark artifacts are generated from synthetic histories and evaluation instances seeded from licensed resources, which are not redistributed as standalone data unless permitted.
  • User-related and non-user session excerpts: User-related and non-user cases distribute compatible, nuanced, or contradictory facts across ordinary task-oriented sessions, requiring later queries to recover one fact or combine several sessions.The examples include Amara’s community-program preferences, Korean city symbol flowers, and context-dependent design styles.
  • Query, answer, and instance construction: The generation pipeline produces relation-specific target queries, correct and plausible incorrect answers, and filtered sessions, questions, and answers under explicit naturalness and subtype requirements.Prompt excerpts cover user-related and non-user instance construction, including exactly three correct and three incorrect non-user answer candidates.
  • D.1.1 Evaluation-time Baseline and Model Settings: Evaluations run each system through its native write-and-recall interface with fixed memory settings, while answer generation uses shared GPT-family configurations and Gemini 3.1 Pro Preview Thinking as the judge.Tables 9–10 document implementation sources, memory hyperparameters, answer-generation settings, and provider information.
  • D.1.2 Context Organization with OpenClaw: OpenClaw is treated as an agent-context intervention: standalone systems place retrieved items directly in the answer prompt, whereas OpenClaw organizes instructions and plugin recall as runtime context.The Mem0 example shows a flat numbered # CONTEXT list without OpenClaw versus workspace instructions plus injected plugin context with OpenClaw.
  • D.1.3 LLM-as-judge Validation: The binary semantic judge compares generated answers with accepted correct answers, known incorrect answers, supporting facts, and relation metadata, achieving Cohen’s κ = 0.963 against human labels.Validation uses 45 fixed-seed queries, 15 per relation type, and five candidate answers per query, yielding 225 human-labeled answers.
  • D.1.4–D.2 Answer generation and retrieval protocols: The strong answer prompt is used in the main evaluation because it explicitly targets identification, conflict recognition, and evidence fidelity, especially reducing unsupported conflict resolution for contradictory cases.Oracle evidence bypasses memory formation and retrieval, perfect retrieval reads back provenance-guided stored units, and default evaluation uses each system’s own recall path.
  • D.3 Statistical Analysis and D.4 Main Experiment Case Studies: Main comparisons use stratified 10,000-resample bootstrap intervals and paired exact McNemar tests with Holm correction, while Figures 35–36 decompose representative cases into facts, relations, outputs, and judge decisions.The Oracle-to-Perfect gap measures information survival through memory formation, and the Perfect-to-Default gap measures exposure through retrieval.

D.5 Perfect-retrieval Detailed Results · D.6 Representative Answer Examples · E Use of AI Assistants

The supplementary analyses isolate answer generation under perfect retrieval, illustrate representative correct and incorrect outputs, and document three limited uses of AI assistance. These materials clarify evaluation scope, oracle-versus-non-oracle comparison, and safeguards around benchmark construction, judging, and writing.

  • D.5 Perfect-retrieval Detailed Results: GPT-5.4 perfect-retrieval results provide relation- and subtype-level detail while bypassing each system’s default query-time retrieval path.Answer generation receives stored memory units linked to target evidence sessions; Table 13 reports the corresponding results.
  • D.5 Perfect-retrieval Detailed Results: The perfect-retrieval analysis separates retrieval effects from answer-generation performance by supplying target-linked stored memory units directly.This setting preserves the main results table’s relation and subtype organization while removing default query-time retrieval from the procedure.
  • D.6 Representative Answer Examples: Representative correct and incorrect baseline answers span both user-related and non-user cases and include facts, reference answers, generated answers, and judge decisions.Figures 37 and 38 present these examples, while Figures 35 and 36 show representative main-experiment cases across complementary, nuanced, contradictory, and relation-critical complementary relations.
  • E Use of AI Assistants: LLMs supported benchmark construction by generating and filtering semantic variants, interaction sessions, and evaluation instances, with the full procedure documented in Appendix C.This was one of three limited AI-assistance roles.
  • E Use of AI Assistants: The evaluation protocol used an LLM-based automatic judge with human-agreement validation in Appendix D.1.3, while general-purpose writing tools improved wording and readability rather than originating the research.The passage describes these as limited roles for AI assistance.
  • E Use of AI Assistants: The supplementary evaluation materials report overall accuracy with bootstrap confidence intervals and compare oracle evidence against the best non-oracle system using paired bootstrap intervals and corrected McNemar tests.Table 11 stratifies intervals by relation type; Table 12 defines the oracle-minus-best-non-oracle difference in percentage points and applies Holm correction.
Loading 2606.05761v2…