PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents

Saksham Sahai Srivastava

arXiv:2605.17641v1cs.AIcs.CL

TL;DR

Long-horizon agents need to select useful memories rather than assume that relevant-looking context will help, while existing long-context evaluations do not directly measure that selection. CMI tests candidate memories through controlled interventions, and CAUSAL-LOCOMO evaluates selection among useful, irrelevant, and harmful memories. CMI achieves the strongest overall balance between answer quality and memory-selection reliability in the reported evaluation.

  • Problem

    Existing long-context benchmarks test information recovery from dialogue histories but do not directly reveal whether the right memory, a spurious memory, or parametric knowledge produced an answer.

    Source: §1 · pp. 1–2

  • Method

    CMI evaluates candidate memories under controlled conditions and selects those that improve the answer, while CAUSAL-LOCOMO tests selection using useful, irrelevant, and harmful memories.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    CMI achieved the strongest overall balance between answer quality and memory-selection reliability, with a task score of 0.846 across 87 filtered examples.

    Source: §1 · p. 2, §9 · p. 11

  • Takeaways & Limitations

    The results support selecting persistent context according to its estimated causal usefulness rather than semantic relevance alone.

    Source: §7 · p. 9

    The experiments use explicit memory-role annotations, so they do not establish that agents can infer causal roles from raw memories alone.

    Source: §8 · p. 10

Abstract

from arXiv · show

Long-horizon LLM agents rely on persistent memory to support interactions across sessions, yet existing memory systems often retrieve context using semantic similarity or broad history inclusion, treating retrieved memories as uniformly useful. This assumption is fragile because memories may be topically related while remaining irrelevant, stale, or misleading. We propose Causal Memory Intervention (CMI), a causal memory-selection technique that estimates how candidate memories affect the model's answer under controlled interventions, selecting memories that improve task performance while suppressing unstable, irrelevant, or harmful ones. To evaluate this setting, we introduce Causal-LoCoMo, a causally annotated benchmark derived from long conversational data, where each example contains a user request, a structured memory bank, useful memories, irrelevant distractors, and synthetic harmful memories. We compare CMI against vector, graph, reflection, summary, full-history, and no-memory baselines. Results show that CMI achieves a stronger balance between answer quality and robustness to misleading memory, suggesting that reliable long-term memory requires selecting context based on causal usefulness rather than relevance alone. The full framework, benchmark construction code, and experimental pipeline are available at https://github.com/Saksham4796/causal-memory-intervention.

1. Introduction

Long-horizon agents need to distinguish useful memories from context that is merely relevant-looking, stale, or harmful. CMI tests candidate memories through controlled interventions, and CAUSAL-LOCOMO evaluates that selection task with useful, irrelevant, and harmful memories.

1. Introduction

  • Similarity-based retrieval and broad history inclusion can expose agents to memories that are irrelevant, stale, ambiguous, contextually wrong, or adversarially inserted.Existing long-context benchmarks often do not reveal whether an answer came from the right memory, a spurious memory, or parametric knowledge.Source: §1 · pp. 1–2
  • CMI selects candidate memories when controlled no-memory, with-memory, and perturbed-memory comparisons indicate improved task performance that remains stable under perturbation.The method aims to suppress memories that are neutral, unstable, or harmful.Source: §1 · p. 2
  • CAUSAL-LOCOMO evaluates memory selection using useful memories, irrelevant distractors, and synthetic harmful memories, alongside vector, graph, reflection, summary, full-history, and no-memory baselines.The benchmark includes temporal reasoning, multi-evidence, factual, and inferential memory questions.Source: §1 · p. 2
  • CMI achieved the strongest overall balance between answer quality and safe memory selection across 87 filtered examples, with a 0.846 task score and zero poisoned-memory adoption.It also reached a 0.816 success rate, 0.875 useful-memory F1, and 0.990 harmful-memory rejection.Source: §1 · p. 2
  • The results support treating reliable memory use as causal discrimination: determining which memories should influence a response and which should be ignored.The paper frames CMI as an intervention-based alternative to purely similarity-based retrieval (Lewis et al., 2020; Karpukhin et al., 2020).Source: §1 · p. 2

2. Related Work

Prior work studies persistent memory, long-context access, retrieval, security, and causal analysis, but these lines do not directly determine which memories should be selected from a noisy bank. This paper focuses on that selection problem by evaluating whether candidate memories causally improve the current answer.

Long-term memory for agents

  • Prior agent-memory systems support continuity and adaptation, but generally treat memory as useful once retrieved rather than asking which entries should be used for the current task.Examples include Generative Agents, Reflexion, MemoryBank, MemGPT, and Voyager (Park et al., 2023; Shinn et al., 2023; Zhong et al., 2024; Packer et al., 2023; Wang et al., 2023).Source: §2 · p. 3

Long-context evaluation

  • Long-context benchmarks test recovery from supplied histories, while CAUSAL-LOCOMO tests whether an agent chooses useful memories and avoids irrelevant or harmful ones.Earlier evaluation can leave unclear whether a correct answer came from the right memory or whether failure arose from missing, irrelevant, or harmful context.Source: §2 · p. 3

Retrieval and robustness

  • Retrieved evidence can be semantically similar yet incomplete, stale, irrelevant, or misleading, a risk that can accumulate in persistent agent memory.Security work on prompt injection, RAG poisoning, and poisoned experience retrieval motivates actively suppressing misleading memories (Liu et al., 2023; Zou et al., 2025; Srivastava & He, 2025).Source: §2 · p. 3

Causal analysis

  • Unlike causal methods that intervene on neurons, activations, or weights, CMI intervenes on external memory context that can be edited, inspected, and selected at inference time.The paper positions this external-memory intervention as complementary to internal causal analysis (Meng et al., 2022).Source: §2 · p. 3

3. Proposed Framework

CMI frames memory selection as choosing a subset expected to improve the answer, rather than simply retrieving similar context. It proposes candidates broadly, evaluates their effects under interventions, and selects memories only when their benefit is stable and within the selection constraints.

3. Proposed Framework

  • CMI ranks persistent-bank memories by estimated effect on the answer, and the selected subset is supplied to the response model alongside the current request.The bank may contain records from prior interactions, observations, summaries, or adversarial insertions.Source: §3 · p. 4

3.1. Memory Selection as a Causal Decision Problem

  • The ideal selector maximizes task utility over memory subsets subject to a memory budget, but evaluating every subset is combinatorial, so CMI estimates individual candidate contributions.The memory budget is denoted k.Source: §3.1 · p. 4

3.2. Candidate Memory Proposal

  • A broad hybrid retriever proposes candidates without assuming they are useful; in the annotated benchmark, role labels can serve as a risk signal before selection.For label-free deployment, the role signal would need to be replaced by a learned or prompted causal-role predictor.Source: §3.2 · p. 4

3.3. Intervention Scoring

  • CMI compares each candidate under no-memory, with-memory, and perturbed-memory conditions to test both its benefit and whether that benefit survives perturbation.Perturbation is intended to expose brittle, misleading, or superficial cues.Source: §3.3 · p. 4
  • Utility measures improvement over no memory, while stability measures robustness to perturbation; positive utility with negative stability is not considered reliable.The intervention loop uses a deterministic task scorer.Source: §3.3 · pp. 4–5

3.4. CMI Selection Rule

  • The selection rule admits memories only when they improve the task score over the no-memory baseline and do not lose their effect under perturbation.Selection also remains subject to the memory budget and risk-filtering constraints.Source: §3.4 · p. 5

3.5. Comparison to Standard Memory Selection

  • Unlike similarity-based methods, CMI tests whether a memory improves the answer and remains stable, targeting semantically plausible memories that are non-causal or harmful.The evaluation therefore considers both task performance and memory-selection behavior.Source: §3.5 · p. 5

4. Dataset

CAUSAL-LOCOMO turns evidence-grounded long-conversation questions into a shared memory-selection setting. Its structured banks separate useful evidence, irrelevant distractors, and, in selected cases, plausible but misleading synthetic memories.

4. Dataset

  • Each CAUSAL-LOCOMO example uses a LoCoMo question as the request and preserves the original conversation to support full-history baselines.Methods receive the same task and candidate bank but differ in which memories they select or expose to the response model.Source: §4 · p. 5
  • Memory banks contain rewritten answer-supporting evidence, non-supporting conversation memories as distractors, and synthetic harmful entries in selected examples.Harmful entries may contradict the answer, swap entities, introduce incorrect temporal information, or encourage unsupported inference; temporal evidence is resolved using session timestamps.Source: §4 · p. 6
  • Filtering reduced 100 generated candidates to 87 examples, comprising 432 past sessions and 491 memory entries: 89 useful, 348 irrelevant, and 54 harmful.The tasks include 46 temporal, 25 multi-evidence, 14 inferential, and 2 factual memory QA examples.Source: §4 · p. 6

5. Experimental Setup

The experiments compare memory-access strategies under a shared task and scoring setup, varying how context is constructed. They assess both answer quality and memory selection, using CMI interventions alongside retrieval, history, summary, and no-memory baselines.

5. Experimental Setup

  • All methods share the task, memory bank, response model, and scoring pipeline; they differ in context construction, with CMI adding pre-answer memory interventions.The benchmark examples include useful, irrelevant, and harmful memories, and may include past conversational sessions.Source: §5 · p. 6

5.1. Compared Methods

  • No-memory uses only the request, full-history supplies past sessions, and summary memory supplies a synthetic summary derived from the structured memory bank.Full-history and summary-memory are treated as exposing the full bank for selection bookkeeping.Source: §5.1 · p. 6
  • Vector retrieval ranks memories by semantic similarity, graph retrieval uses lexical and scope links, and reflection memory retrieves label-aware reflections by keyword matching.The reflection baseline is not an LLM self-reflection method.Source: §5.1 · pp. 6–7
  • CMI proposes candidate memories and tests them under no-memory, with-memory, and perturbed-memory conditions, selecting those that improve deterministic task scores and remain stable.Source: §5.1 · p. 7

5.2. Evaluation Pipeline

  • Final answers use a shared prompt and scoring procedure, with a hybrid task score combining deterministic checks and a GPT-5 judge; CMI intervention utilities use only the deterministic scorer.The deterministic scorer checks criteria such as required content, word limits, steps, and style.Source: §5.2 · p. 7
  • Task success is the fraction of examples with hybrid task scores of at least 0.7, so it is thresholded hybrid correctness rather than exact-match accuracy.Source: §5.2 · p. 7

5.3. Memory Selection Metrics

  • Memory-selection metrics cover useful-memory recall and precision, their harmonic-mean F1, rejection of irrelevant or harmful memories, poisoned-memory adoption, and selected-memory counts.Full-history and summary-memory selection metrics are bookkeeping measures because those methods are treated as exposing the full bank.Source: §5.3 · p. 7

5.4. Implementation Details

  • The filtered benchmark contains 87 examples across temporal, multi-evidence, inferential, and factual memory QA; experiments use GPT-4.1 for responses and GPT-5 for judging.Decoding uses temperature zero, and the answer-generation prompt is held fixed for memory-based methods.Source: §5.4 · p. 8

6. Results

Across the benchmark, CMI pairs strong answer quality with selective and robust memory use. Its intervention utilities also distinguish useful memories from irrelevant and harmful ones, while task-family results show that it does not lead every category.

6.2. Accuracy–Robustness Tradeoff

  • CMI occupies Figure 2’s high-score, zero-poisoned-adoption region, while vector, graph, and reflection memory adopt poisoned memories more often.The figure plots poisoned-memory adoption rate against task score, with marker size indicating the average number of memories selected or exposed.Source: Figure 2 · p. 8, §6.2 · p. 8

6.3. Performance by Task Family

  • CMI scores best on multi-evidence QA (0.754), matches the best on factual QA (0.985), but trails vector on temporal QA and reflection on inferential QA.Factual QA contains only two examples, so that family’s results warrant caution.Source: §6.3 · p. 8, Table 2 · p. 9

6.1. Main Results

  • 0.846 task score, 0.816 success rate, and 0.875 useful-memory F1 lead the comparison; bad-memory rejection is 0.990, with zero poisoned-memory adoption.CMI selects fewer than one memory on average; reflection and vector memory score 0.845 and 0.839, respectively, while both select three memories on average.Source: §6.1 · p. 8, Table 1 · p. 9

6.4. Causal Utility Diagnostics

  • Useful memories have positive average intervention utility (+0.307), irrelevant memories near-zero utility (−0.009), and harmful memories negative utility (−0.033).Average stability is small across memory types, with useful memories highest at +0.027.Source: Table 3 · p. 9, §6.4 · p. 9, Figure 3 · p. 9

7. Analysis and Discussion

The analysis attributes CMI’s overall tradeoff to testing whether memories improve answers, rather than relying on relevance or broad context. Diagnostics support that distinction, although performance advantages vary across task types.

7. Analysis and Discussion

  • CMI selects a candidate only when it improves deterministic task score over no-memory and remains stable under perturbation, filtering toward positive estimated causal effects.The analysis links this criterion to strong overall task score despite selecting fewer than one memory on average.Source: §7 · p. 9
  • Plausible, topically close harmful memories can attract similarity-based selectors, while full histories may dilute evidence and summaries may remove dates, entity distinctions, or multi-evidence dependencies.The analysis therefore identifies selective, reliable memory use—not memory availability—as the central bottleneck.Source: §7 · pp. 9–10
  • Useful, irrelevant, and harmful memories have positive, near-zero, and negative average utility, respectively, but vector memory slightly outperforms CMI on temporal QA.The analysis frames CMI’s contribution as a stronger overall accuracy–robustness tradeoff, not universal dominance across tasks.Source: §7 · p. 10

8. Limitations and Ethical Considerations

The evaluation is controlled and annotated, with a small benchmark and synthetic harmful memories that limit how broadly its results can be interpreted. Deployment also faces added inference cost and privacy and ethical trade-offs.

8. Limitations and Ethical Considerations

  • Gold memory-role annotations mean the experiments do not establish that agents can infer causal roles from raw memories alone.A deployable CMI would need predicted role estimates evaluated under realistic ambiguity, distribution shift, and adversarial conditions.Source: §8 · p. 10
  • The benchmark uses 87 filtered LoCoMo examples focused mainly on temporal, multi-evidence, and inferential memory QA, and omits several deployed-agent memory failures.Uncovered examples include stale preferences, incorrect summaries, entity-linking errors, conflicting updates, and long-term memory drift.Source: §8 · p. 10
  • Synthetic harmful memories support controlled robustness evaluation but may not represent naturally occurring or human-crafted memory corruptions..Source: §8 · p. 10
  • The final task score depends partly on a GPT-5 judge, while pre-answer intervention calls increase latency and token use relative to simple retrieval.The authors identify judge calibration, prompt sensitivity, and possible biases as concerns; suggested deployment options include caching utilities, risk-gating scoring, or using smaller verifier models.Source: §8 · p. 10
  • Memory reuse raises privacy risks, and filtering trades off suppressing sensitive but relevant context against retaining stale, incorrect, or biased memories.The authors call for consent, access control, retention limits, provenance tracking, and ways for users to inspect, correct, or delete memories.Source: §8 · p. 10

9. Conclusion

The paper introduces CMI, which selects memories based on their causal effects on answers, and evaluates it against multiple memory and prompting baselines. In the reported setting, CMI offers the strongest overall balance of answer quality and memory-selection reliability.

9. Conclusion

  • CMI selects memories based on whether controlled interventions indicate positive and stable causal effects on the model’s answer.Source: §9 · p. 11
  • The CAUSAL-LOCOMO experiments compare CMI with vector, graph, reflection, summary, full-history, and no-memory baselines.Source: §9 · p. 11
  • CMI achieves the strongest overall balance between answer quality and memory-selection reliability in the reported experiments.It obtains the best task score and useful-memory F1 while reducing bad-memory selection and eliminating poisoned-memory adoption in that setting.Source: §9 · p. 11
Loading 2605.17641v1…