PaperlayerView exact revision on arXiv

Source-linked AI summary

Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory

Ruizhe Li, Mingxuan Du, Benfeng Xu, Zhendong Mao

arXiv:2607.24368v1cs.CL

TL;DR

Long-term memory retrievers can miss facts whose relevance depends on implicit world-knowledge associations rather than textual similarity. InMind benchmarks this blind spot with diagnostic controls and finds that systems recall stored facts yet apply them poorly to indirect queries, while keeping memory visible recovers most of the gap.

  • Problem

    Memory systems assume needed facts resemble queries, leaving implicit associations—such as allergies affecting ingredient recommendations—insufficiently tested.

  • Method

    InMind evaluates 125 expert-verified tasks with paired controls separating storage, bridging-knowledge, and retrieval failures.

  • Results

    84.0% in-context answering falls to at most 14.4% under retrieval, despite recall reaching up to 100%; an always-in-state probe recovers 68.8%.

  • Takeaways & Limitations

    The benchmark locates the main challenge in query-conditioned relevance judgments and identifies routing—deciding which facts remain visible—as the open problem.

  • Takeaways & Limitations

    The benchmark emphasizes health, wellness, and safety, while GPT-5-mini serves as both answerer and judge, creating a self-preference risk.

Abstract

from arXiv · show

Long-term memory systems store what a user says in an external store and retrieve it when a related query arrives. This interface rests on an assumption so natural that it is rarely stated: a memory that is needed will resemble the query that needs it. World knowledge breaks the assumption. A tree-nut allergy should change the answer to a macaron request through their almond-flour ingredient, yet the two texts share no cue a retriever can see. We call this failure mode the implicit-association blind spot and introduce InMind, a 125-task, expert-verified benchmark spanning ten life domains, with 113 tasks grounded in citable public sources. Its paired controls separate three explanations that existing evaluations conflate: the fact was never stored, the model lacks the bridging knowledge, or the fact was stored and never surfaced. The verdict is clean. With the decisive memory placed in context, the backbone answers 84.0 percent of indirect queries; when the same memory must be retrieved, six vector, graph, and agentic memory systems reach at most 14.4 percent, even though they recall the same facts on demand at up to 100 percent. An embedding with eight times the dimensionality raises answer-blind target recall for every system yet leaves the gap essentially intact. A minimal diagnostic probe that keeps memory visible before the query arrives recovers most of the gap, locating the failure in the query-conditioned interface itself and pointing to routing, deciding which facts must stay visible, as the open problem InMind is built to score.

1 Introduction

Retrieval-based memory can recall a personal fact directly yet fail to apply it when world knowledge creates an indirect connection to a new query. InMind isolates storage, bridging knowledge, and retrieval delivery, showing that the central failure lies in the query-conditioned memory interface.

  • Problem: A system can recall a tree-nut allergy on demand yet recommend almond flour for macarons because the memory was not brought to bear when needed.The allergy and recipe share no retrieval-visible cue; standard macarons use almond flour, a major tree-nut allergen.
  • Problem: Implicit associations violate the retrieval hypothesis that any memory needed to answer a query lies close to it under a computable relevance score.Drug–food interactions, cross-allergies, dietary law, occupational restrictions, and household hazards can require knowledge absent from both the memory and query texts.
  • Benchmark: InMind contains 125 expert-verified tasks, including 113 grounded in citable public sources, with controls separating storage failure, missing bridging knowledge, and retrieval failure.Direct naive queries test storage, in-context controls test backbone knowledge, and target recall tests whether retrieval delivered the fact to context.
  • Results: 84.0% in-context backbone performance and up to 100% on-demand fact recall contrast with at most 14.4% indirect-query application across six vector, graph, and agentic systems.No query-time configuration exceeded 16.0%.
  • Results: A minimal always-in-state probe recovers 68.8% of the gap, locating the open problem in routing decision-critical facts into visible state rather than retrieval alone.An embedding with eight times the dimensionality raises answer-blind target recall for every system without closing the gap, while hybrid systems with profile-style state do not show the same closure.

2 The Blind Spot in Query-Conditioned Memory

Query-conditioned memory systems retrieve before the model can apply world knowledge, assuming necessary memories are recoverable from query-based relevance signals. Implicit associations violate this assumption when a memory is necessary but semantically distant from the query, predicting strong in-context performance and collapsed indirect application under retrieval.

  • Query-conditioned memory: Retrieval selects ˆ M from stored memories using q and θ before the LLM answers from the selected subset.The model reasons only over retrieved memories, making retrieval order decisive.
  • Retrieval Hypothesis: The Retrieval Hypothesis assumes every necessary memory is recoverable through an efficient relevance score computed from the query alone.The authors exclude exhaustive model-based scoring because it forfeits retrieval’s sublinear-cost motivation.
  • Implicit associations: An implicit association requires necessity and semantic distance, so common lexical, embedding, or topical relevance functions provide little retrieval signal.Examples include cat ownership paired with a lily centerpiece query, where veterinary knowledge supplies the unstated bridge.
  • Shared bottleneck: Query-conditioned retrieval creates a shared bottleneck: the model sees memory and query together only after semantic-similarity machinery has chosen what to retrieve.Graphs, agentic search, and hierarchical stores add upstream machinery but cannot substitute for the semantic-similarity link.
  • Testable prediction: The predicted signature is strong indirect-query answers with memory in context, near-ceiling explicit recall under retrieval, and collapsed indirect application.InMind tests this pattern against explanations based on task difficulty, forgetting, or general model weakness.

3 The InMind Benchmark

InMind is a 125-task benchmark designed to isolate implicit-association failures by pairing source-grounded personal facts with naive and indirect queries under realistic memory interference. Its independent content filters, expert verification, and complementary measurements distinguish whether systems stored, retrieved, or correctly applied the relevant fact.

  • Benchmark construction: Each task pairs a naive query with an indirect query, revealing whether a system holds the fact but fails to surface it when needed.The indirect query is designed so the answer should change because of the personal fact, without naming the later object.
  • Benchmark construction: 125 tasks comprise 113 source-grounded and 12 expert-authored cases, with expert-verified bridges connecting personal facts to later decisions.All but twelve tasks trace to citable public sources carrying an expert-verified bridge.
  • Evaluation protocol: 38 further sessions of ordinary interaction separate memory insertion from evaluation within a fixed 47-session conversation trace.This tests whether the injected fact survives realistic interference before it is queried.
  • Evaluation metrics: Target recall measures whether the target fact appears in context, whereas indirect application judges whether the answer follows the source-grounded bridge.The two binary measurements separate failures occurring before the model sees the fact from failures after exposure.

4 Experiments: Locating the Failure

The experiments locate InMind’s failure in query-conditioned access rather than forgetting, reasoning ability, or embedding scale. With the decisive memory visible, GPT-5-mini answers 84.0% of indirect queries, but retrieval systems apply it at no more than 16.0%.

  • Locating the bottleneck: 3.2%–16.0% application across retrieval configurations shows that the decisive fact is rarely surfaced before the answerer must reason over it.The six memory systems themselves topped out at 14.4%, despite naive recall reaching up to 100.0%.
  • In-context control: 84.0% of indirect queries were answered when GPT-5-mini received the target memory directly in context, versus at most 16.0% for retrieval configurations.The backbone succeeded on 105/125 tasks, while the best retrieval configuration had a 95% Wilson interval of [10.6, 23.4].
  • Ruling out forgetting: Up to 100.0% direct recall shows that memories were stored and retrievable on demand, so forgetting does not explain indirect-query failure.A-Mem reached 100.0% direct recall after 38 sessions of interference.
  • Testing representation strength: Stronger embeddings provide limited gains because they encode only associations that leave a distributional trace in training text, while many benchmark bridges are specialized.The passage identifies pharmacological, legal, religious, and developmental bridges as examples.

5 What It Takes to Close the Gap

Retrieval-based memory preserves detailed records at scale but remains insufficient when query-conditioned search cannot expose facts connected only through distant knowledge. An always-visible diagnostic recovers most of the lost performance, identifying routing—what remains visible and when to search—as the open problem.

  • Routing: Routing is the open problem: hybrids must decide which facts earn persistent state and when queries justify searching the fine-grained record.Recency, frequency, and heat-thresholded promotion cannot anticipate distant bridges, while the right criterion varies across medical, financial, and coding assistants.
  • Retrieval limits: Retrieval alone is insufficient because memory returns what resembles its query, even when a bridge-aware question names the relevant danger.A successful probe must use the stored fact’s vocabulary rather than the request’s vocabulary.
  • Always-visible state: An always-in-state memory removes query-time selection, allowing the model to reason jointly over persistent state and the incoming query.The diagnostic uses a whole, maintained 200-line markdown profile prepended to the system prompt, with no embeddings, index, ranking, or query-time retrieval.
  • Always-visible state: The probe recovers most of what retrieval systems lose, demonstrating sufficiency for visible, competently curated memory rather than a controlled ablation of one system.Its diagnostic value comes from eliminating the query-time retriever between the model and memory.
  • Always-visible state: 14.4% with emb3-large is MemoryOS’s strongest application score, yet its 7.2% answer-blind target recall shows that persistent visibility alone does not ensure decisive facts are reached.The probe sits more than fifty points above MemoryOS, sharpening the distinction between visibility and the contents of the visible state.

6 Related Work

Prior memory systems evolved from observation retrieval and external storage toward vector, temporal, graph, and heterogeneous memories, while benchmarks mainly test explicit recall, state aggregation, or entity-based multi-hop reasoning. InMind distinguishes itself by using source-grounded associations rather than connections authored by language models.

  • Memory for language agents: Memory systems progressed from retrieving observations and separating limited context from external storage to vector, temporal, graph, and heterogeneous memory designs.These extensions distill reusable facts, revise records over time, expose relational structure, or separate memory types before aggregation.
  • Memory benchmarks: Existing long-term-memory benchmarks primarily test explicit recall, state aggregation, or entity-based multi-hop reasoning.These settings respectively use direct query-target cues, temporally ordered memories, or an intermediate memory linking query and answer.
  • Construction provenance: LLM-authored benchmark associations preferentially sample connections models already find natural and can re-derive unaided at test time.The passage identifies simulated dialogues and generated tasks as common examples of this construction approach.
  • Construction provenance: InMind uses source-grounded bridges to distinguish its associations from LLM-authored benchmark connections.The supplied passage introduces this contrast after describing how model-authored construction favors associations models already make unaided.

7 Limitations

The benchmark is diagnostic but limited in scope, evaluation independence, and bridge assumptions. Its task mix favors judgeable health, wellness, and safety cases, while probabilistic or disputed bridges and over-warning remain insufficiently measured.

  • Benchmark scope: 125 tasks emphasize health, wellness, and safety, leaving humor, etiquette, long-term goals, and institutional policy uncovered.At n = 125, a few percentage points may reflect sampling noise, whereas the conclusions rely on sixty-to-seventy-point effects.
  • Evaluation: GPT-5-mini serves as both answerer and judge, creating a self-preference risk that could inflate absolute scores.The bias applies to both the backbone control and retrieval systems.
  • Bridge assumptions: The benchmark assumes each bridge is factual and uncontested, although real bridges may be probabilistic, jurisdiction-dependent, or disputed.These assumptions limit how directly benchmark findings transfer to real-world cases.
  • Rubric limitations: Agents optimized for InMind could over-warn because applying the bridge earns credit while over-eagerness costs nothing under the rubric.Measuring this requires negative controls, which the paper leaves to future work.

8 Conclusion

InMind shows that a memory can be essential to a query without resembling it, because world knowledge creates relevance the retriever lacks. This blind spot separates strong answering with memory visible from retrieval-conditioned performance, despite high on-demand recall.

  • Conclusion: 84.0% versus 16.0% captures the performance gap caused by the implicit-association blind spot in leading memory systems.The memory can be essential without resembling the query that needs it.
  • Conclusion: Up to 100% on-demand recall shows that the same systems can retrieve the underlying facts when explicitly asked.The failure is therefore not simply that the facts cannot be recalled at all.
  • Conclusion: Storage, indexing, graph construction, and iterative search improve what an agent finds once it knows what to seek.The conclusion argues that these advances address retrieval after the relevant fact has already been identified, rather than the initial blind spot.

Ethics Statement

InMind uses synthetic user facts and relies mainly on expert-verified public sources, while including sensitive scenarios to test consequential failures. The authors disclose dual-use risks and clarify that always-in-state memory is diagnostic, not a deployment recommendation.

  • Ethics Statement: InMind contains no personal data: all user facts are synthetic, with 113 of 125 tasks grounded in public sources and the remaining 12 expert-authored.Every bridge is expert-verified; sensitive scenarios include intimate partner violence, medical conditions, immigration status, and religious practice.
  • Ethics Statement: The paper documents a reproducible retrieval failure across research and open-source memory systems, including queries that evade retrieval, to support testing without granting adversaries new capabilities.The authors warn that deployed assistants could fail silently, with users unable to detect the problem by simply asking what the agent remembers.
  • Ethics Statement: Always-in-state memory is presented as a diagnostic rather than a deployment recommendation because retaining a full user profile broadens privacy exposure relative to narrow on-demand retrieval.Section 5 explicitly treats full-context memory as a diagnostic, not an endorsement for deployment.

Reproducibility Statement · Appendix · A Content-Filter Details

The paper specifies its construction, evaluation, filtering, and release procedures for reproducibility. Its content filters remove overt retrieval cues, persona conflicts, and unsupported or unnatural tasks before expert verification.

  • Reproducibility Statement: The construction pipeline, source list, extraction prompt, judge prompts, rubric, evaluation procedure, and hyperparameters are documented across the main text and appendices.The evaluation inserts generated memory turns into LME-s conversations, continues the simulated dialogue, and then issues task queries.
  • Reproducibility Statement: The benchmark, per-task outputs, judge verdicts, and evaluation harness will be released publicly.The always-in-state baseline is specified by Algorithm 1 and requires only an LLM and a text file.
  • A Content-Filter Details: All three content filters depend only on task content and the fixed background trace, inspect no system outputs, and were fixed before scoring.The filters reduce 1,000 extracted candidates to the evaluation set.
  • A Content-Filter Details: The similarity filter uses BM25 and all-MiniLM-L6-v2 cosine similarity, discarding candidates whose target memory is an obvious lexical or semantic match to its query.This filter retains 300 of 1,000 candidates and withholds cues that could enable similarity-only retrieval.
  • A Content-Filter Details: The conflict filter rejects injected facts that contradict the persona established by the shared 47-session LME-s background trace.Examples include claiming a cat when the trace says the user has no pets or assigning a pilot’s medical exam to a persona established as a nurse.
  • A Content-Filter Details: Expert reviewers verify factual support, material answer impact, conversational naturalness, and the absence of overt cues such as synonyms or same-topic references.Reviewers assess each surviving candidate against its cited source chunk.

B Memory-Benchmark Taxonomy · C Representative Tasks by Domain

The benchmark distinguishes explicit recall, state aggregation, entity-based multi-hop reasoning, and knowledge-mediated implicit application. Representative tasks test whether stored personal facts alter later answers when the connection depends on an external knowledge bridge across diverse life domains.

  • B Memory-Benchmark Taxonomy: The taxonomy includes explicit recall, state aggregation, entity-based multi-hop reasoning, and InMind’s knowledge-mediated implicit application.These settings define the benchmark’s memory-evaluation coverage.
  • C Representative Tasks by Domain: Each representative task pairs a stored user memory with a later indirect query and a knowledge-mediated bridge that should change the answer.The examples are retained one-per-domain illustrations of benchmark coverage.
  • C Representative Tasks by Domain: A tree-nut allergy should trigger a warning about macarons because standard macarons use almond flour.The assistant should warn that making or eating macarons could trigger the user’s allergy.
  • C Representative Tasks by Domain: Pilot-exam preparation should make an antihistamine question prompt advice about impaired flight safety, possible no-fly periods, and consulting an Aviation Medical Examiner.The query concerns allergies, while the stored professional context changes the relevant safety guidance.
  • C Representative Tasks by Domain: Domestic-violence status should connect a temporary-accommodation request to shelters and safety resources.The stored relationship context changes the answer beyond generic accommodation information.
  • C Representative Tasks by Domain: An employee’s Google offer should make an Apple-stock question address potential competitor conflicts and applicable company policy.The answer depends on the user’s role and should point to the relevant policy.
  • C Representative Tasks by Domain: Other tasks connect cognitive impairment to simpler account setup, Halal rules to capsule ingredients, immigration status to travel warnings, and child-related memories to safety advice.Examples include simple, verifiable username processes; checking gelatin sources; Advance Parole concerns; recall warnings; choking-risk mitigation; and EV route planning.
  • C Representative Tasks by Domain: The remaining examples extend the same pattern to pending I-485 travel, recalled furniture, toddler corn preparation, and Tesla route planning.These bridges require flagging Advance Parole, child-safety instructions, cutting or mashing kernels, and charging-aware routing.

D Representative System Responses · E Task-Generation Prompt

Representative responses show that xMemory recalls a user’s tree-nut allergy directly yet fails to apply it to an almond-flour macaron request, unlike the in-context Backbone control. The task-generation prompt operationalizes this blind spot by requiring a concrete, semantically indirect bridge between a personal fact and a later query.

  • D Representative System Responses: The Backbone control correctly applies the in-context tree-nut allergy to an indirect macaron request and receives judge score: 1.It warns that classic French macarons use almond flour and proposes nut-free alternatives and allergy-safety precautions.
  • D Representative System Responses: The correct Backbone response links the allergy to almond flour and recommends avoiding tree-nut ingredients, cross-contamination, and relevant label warnings.It also identifies safer nut-free approaches while cautioning that seed substitutes may trigger cross-reactions for some people.
  • D Representative System Responses: xMemory answers the naive allergy-recall query correctly but receives judge score: 0 on the indirect query.Its indirect response gives an almond-flour macaron recipe without mentioning the allergy, showing no evidence that the stored constraint was retrieved.
  • E Task-Generation Prompt: The production extractor evaluates one knowledge-base chunk at a time and rejects it unless the chunk itself supports a concrete semantic memory–query bridge.The source prompt was originally written in Chinese and translated faithfully into English with placeholders preserved.
  • E Task-Generation Prompt: The generated task separates a memory-injection turn, where the user states a personal fact, from a later test turn about a seemingly unrelated object or scenario.The generator is designed to test whether an assistant proactively uses the earlier personal fact.
  • E Task-Generation Prompt: The prompt requires entity_1 to be a user-specific fact, entity_2 a later concrete object or scenario, and relation the mechanism connecting them.Examples of entity_1 include pregnancy, chronic conditions, medication use, young children, occupations, certifications, and workplace PPE requirements.
  • E Task-Generation Prompt: The assistant acknowledgement may restate the personal fact but must not mention entity_2, reveal the relation, or provide advice.This preserves the separation between storing the fact and testing whether the system later applies it.
  • E Task-Generation Prompt: The indirect query must contain entity_2 while omitting entity_1 and obvious synonyms, avoiding the answer and sharing only function words with the user message.The prompt also defines naive_query as a direct-recall question whose answer is the personal fact and which does not mention entity_2.

F Automatic Evaluation Prompts … J Public Knowledge Collections

The evaluation framework separates answer quality from retrieval, audits GPT-5-mini judgments against experts, and documents the shared long-horizon, always-in-state, and public-knowledge-task configurations. Its target-recall judge scores only whether the context contains a specific, query-connectable personal fact, while answer-only evaluation ignores retrieval context.

  • F Automatic Evaluation Prompts: The indirect-query judge requires both context_recall and answer_warning, assigning 0 when either the personal fact is absent from context or the required warning is missing.The binary judge receives the source-grounded explanation describing the behavior expected for a correct indirect answer.
  • F Automatic Evaluation Prompts: Target recall is judged independently of generated answers, scoring 1 only when context contains the target fact verbatim or as an unambiguous, query-connectable paraphrase.The judge scores retrieval rather than answer quality and rejects generic relations or facts requiring inference from the query, explanation, or outside knowledge.
  • F.1 Answer-Only Post-Hoc Evaluation: Answer-only post-hoc evaluation withholds retrieved context and scores whether the response applies the personal information through the bridge specified in the explanation.A generic answer that does not make the required consequence clear receives 0, regardless of whether retrieval contained the memory.
  • F.2 Human Audit of GPT-5-mini Judgments: 97.0% accuracy was achieved by Target Recall in the expert audit, compared with 85.0% for the context-aware Application judge and 91.0% for answer-level judgment without context.The audit sampled 100 records across all 16 main-table configurations and ten domains; the Application judge’s 15 errors were all false positives.
  • F.2 Human Audit of GPT-5-mini Judgments: A macaron recipe illustrates false positives: an allergen warning can be substantively correct from query knowledge even when retrieval never surfaces the user’s tree-nut allergy.The answer-only score is 1, whereas a context-aware Application judge may incorrectly credit the warning without verifying that the personal fact was retrieved.
  • G System Hyperparameters: All reported results use unchanged models and configurations on the 125-task benchmark, with stateful systems following the shared middle-injection protocol and Naive RAG using a temporary target chunk.MiniLM denotes all-MiniLM-L6-v2 with 384 dimensions; the MiniLM and emb3-large A-RAG indices are built separately rather than treated as a strict embedding-only swap.
  • H Long-Horizon Injection Protocol: Every memory system processes the same fixed 47-session LME-s trace, inserting each task’s user-fact and assistant-acknowledgement pair at the middle position before replaying the remaining sessions.The protocol processes the first eight sessions, appends the task pair to the ninth session, and then replays the remaining 38 sessions in order.
  • I Always-in-State Baseline: The always-in-state baseline maintains one truncated markdown memory file across sessions and supplies the complete file to the answer model rather than retrieving a subset.The file is capped at 200 lines and 25,000 bytes after each update; both updater and answerer use GPT-5-mini.
Loading 2607.24368v1…