PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning

Quan M. Tran, Zhuo Huang, Zhen Fang, Jing Zhang, Mingming Gong, Tongliang Liu

arXiv:2609.39473v1cs.AIcs.LG

TL;DR

Autonomous agents’ memories can carry beliefs that mislead them in unseen environments, and evaluating this false memory is difficult because it concerns internal beliefs and is confounded with generalization failure. FAME uses counterfactual reasoning to measure concept drift in hidden states; experiments report 76.2%–96.7% AUROCs across false-memory settings and show that false memory may not appear in surface answers.

  • Problem

    False memory can arise when agents apply beliefs shaped by prior environments to unseen settings, yet it is difficult to evaluate because it concerns internal beliefs and is often conflated with generalization failure.

    Source: §1 · p. 1

  • Method

    FAME is a training-free framework that evaluates false memory by measuring memory-induced concept drift under counterfactual reasoning using agent hidden states, without answer generation or reward calibration.

    Source: §1 · p. 2

  • Results

    FAME achieves 76.2%–96.7% AUROCs across false-memory settings, while experiments show false memory is not necessarily observable from surface outputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments, taxonomy, and released counterfactual templates establish a basis for studying false memory across spurious correlation, environment shift, and knowledge conflict.

    Source: §1 · p. 2, §7 · p. 9

    The analysis assumes answer-token logit differences are linear in the answer-commit hidden state and that retrieved memory induces a consistent concept with the committed answer.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Autonomous agents increasingly rely on memory to generalize beyond their training environments. However, agents are bounded by what they have seen and believed, and leveraging such memories in unseen environments can introduce biases into their internal beliefs. We formalize this phenomenon as \textit{false memory}, which can arise from spurious correlations, environment shifts, and knowledge conflicts. Despite its importance, false memory is difficult to evaluate because it stems from agent internal beliefs and is easily confounded with ordinary generalization failures. Therefore, we propose FAME, a training-free framework that evaluates false memory through the evolution of agent beliefs under counterfactual reasoning. Specifically, counterfactual scenarios reveal how beliefs change as the latent concept of memory shifts under hypothetical interventions; thus, measuring the resulting concept drift provides a signal for distinguishing faithful versus false memory. Such concepts can be estimated from agent hidden states before answer generation, avoiding the need for reward design or answer sampling. Empirical experiments reveal that simply monitoring answers often fails to detect false memory, while FAME achieves AUROCs of 76.2% - 96.7% across false-memory settings, and outperforms the best baseline by 3.4% - 23.3% across realistic benchmarks, spanning math reasoning (GSM-Symbolic), code generation (GitChameleon), and complex reasoning (BigBench-Hard). We further release corresponding counterfactual templates and facilitate future research on false memory.

1 Introduction

Memories can preserve beliefs tied to past environments, making them misleading under spurious correlations, environment shifts, or knowledge conflicts. FAME evaluates such false memory by tracking counterfactual concept drift in agent hidden states, and experiments report strong AUROC results across these settings.

1 Introduction

  • False memory arises when beliefs entangled with prior environments mislead agents in unseen settings, through spurious correlations, environment shifts, or knowledge conflicts.Examples include interpreting “X” as academic notation rather than a social network and applying US driving assumptions in the UK.Source: §1 · p. 1
  • FAME tracks how an agent’s memory-induced latent concept drifts under counterfactual scenarios, assessing whether that drift aligns with a region associated with faithful or false memory.Concepts can be approximated from hidden states before answer generation.Source: §1 · p. 2
  • FAME evaluates internal beliefs without answer generation or costly reward calibration, addressing evaluation approaches that conflate false memory with generalization failure.Existing approaches may rely on observable answers, reward functions, or costly fine-tuning and substantial human expertise.Source: §1 · p. 2
  • 76.2%–96.7% AUROCs identify false memory across spurious correlation, environment shift, and knowledge conflict, with Pearson r = 0.48–0.98 (p < 0.001).On realistic benchmarks, FAME outperforms the best baseline by 3.4%–23.3% AUROC; surface outputs do not always reveal false memory.Source: §1 · p. 2
  • The paper provides a taxonomy of three false-memory types and theoretical analysis of counterfactual identifiability and hidden-state concept-drift realization.It also releases counterfactual templates to support future research.Source: §1 · p. 2

2 Problem Setup

An agent’s retrieved demonstrations induce a latent concept suited to its memory tasks, but that concept may become inappropriate for a new target query. The setup defines false memory through degraded target-task reward and motivates a training-free, answer-free measure based on counterfactual concept drift.

2 Problem Setup

  • A memory set of query-answer demonstrations induces a latent concept that maximizes expected reward on the memory tasks.The memory set consists of retrieved demonstrations from the agent’s experience.Source: §2 · p. 3
  • False memory occurs when applying the memory-induced concept to a target query yields substantially lower reward than expected on memory tasks.This can happen when the target query falls outside the agent’s past experience.Source: §2 · p. 3
  • Reward-based evaluation requires reward functions and answers for memory and target tasks, while internal beliefs may not be identifiable from observable answers.These requirements make the evaluation costly.Source: §2 · p. 3
  • The proposed evaluation measures whether concept drift under counterfactual queries aligns with the corresponding admissible region distinguishing faithful from false memory.The drift is induced by intervening on a hypothetical query, and its projection into the region identifies false memory.Source: §2 · p. 3

3 Proposed Method

FAME evaluates false memory by intervening counterfactually on queries and measuring changes in memory-induced latent concepts. It estimates these concepts from hidden states and judges drift against scenario-specific expectations.

3.1 Behavioral Change of Concept via Counterfactual

  • The causal model treats memory as inducing a latent concept that a task query updates; counterfactual inference intervenes on the query while holding the memory concept fixed.The procedure abducts the exogenous noise, intervenes on the query, and predicts the updated concept.Source: §3.1 · pp. 3–4

3.2 Geometry of Concept Drift

  • Hidden states approximate predictive concepts under representative demonstrations and sufficient answer-commit states, while leave-one-out comparisons measure counterfactual drift against normal memory-concept variability.The drift anchors the original concept at the memory and measures movement toward the counterfactual region.Source: §3.2 · pp. 4–5
  • Robustness expects drift to remain within the memory-concept region, whereas adaptation expects movement outside that region along the scenario’s adaptation direction.False memory is determined by whether drift crosses the corresponding projected boundary under the relevant expectation.Source: §3.2 · p. 5, §3.3 · p. 5

3.3 False Memory Evaluation

  • Projecting counterfactual concept drift onto the answer-readout direction identifies changes in answer preference without generating an answer, under the stated assumptions.The boundary for false-memory identification is the projection of the scenario’s admissible concept space into the readout space.Source: §3.3 · p. 5

4 Taxonomy of False Memory

The taxonomy frames false memory as belief failures tied to spurious features, changed environments, or conflicting knowledge, with counterfactual queries tailored to each setting.

Spurious correlation

  • Spurious correlation entangles memory with irrelevant features, tested by keeping the feature while changing context or removing it while retaining context.Repetitive words can serve as spurious features.Source: §4 · p. 6

Environment shift

  • Environment shift tests whether skills tied to one environment transfer when a counterfactual query changes the environment.The expected latent concept is robust to environmental changes.Source: §4 · p. 6

Knowledge conflict

  • Knowledge conflict tests whether memory-induced knowledge remains stable when counterfactual queries make prior knowledge more attractive.The setting includes memory knowledge that conflicts with the agent’s prior knowledge.Source: §4 · p. 6

5 Empirical Evaluations

In controlled false-memory settings, FAME outperforms baselines, while answer confidence alone provides little separation between correct and false-memory cases. The analyses examine boundary crossing and whether hidden-state drift can support answer-free evaluation.

5.1 Experimental Setup

  • The controlled evaluation covers spurious correlation, environment shift, and knowledge conflict, using Llama-3B hidden states from the 14th layer and AUROC as its main metric.The settings use 40 spurious-correlation memory sets and 30 each for environment shift and knowledge conflict.Source: §5.1 · p. 7

5.2 Main Results & Further Analysis

  • FAME beats the best baseline by over 14% on spurious keeping, 20% on spurious removal, 8% on environment shift, and 11% on knowledge conflict.Drift magnitude alone is insufficient: crossing the relevant boundary is more informative than a larger drift that stays within the correct region.Source: §5.2 · pp. 7–8
  • Smaller drifts that cross the boundary can signal false memory more strongly than larger drifts that remain within the correct region.Cloud distance and task vectors, which rely on drift magnitude, underperform FAME across settings.Source: §5.2 · pp. 7–8, Figure 6 · p. 7
  • Concept drift correlates with behavioral margin shift, supporting evaluation before answers are generated.The reported correlation is statistically significant across the false-memory categories.Source: §5.2 · p. 8
  • False-memory and correct-answer confidence distributions overlap, making generated answers alone inadequate for identifying false memory.The expected low-confidence versus high-confidence separation is not clearly distinguishable.Source: §5.2 · p. 8

6 Real-World Evaluations

On realistic benchmarks, FAME leads the compared baselines across math, code, and complex-reasoning settings. The section also reports practical case studies and how memory size relates to false-memory or retention rates.

6.1 Benchmarks & Implementation Details

  • The benchmarks assess spurious correlation and environment shift in GSM, knowledge conflict in Git, and spurious correlation in BBH reasoning tasks.The tasks include boolean expression, date understanding, logical deduction, and tracking shuffled objects.Source: §6.1 · p. 8
  • The released templates cover 40 GSM templates, 18 Git templates, and 11 BBH templates, evaluated with five-level grades.Each grade level contains 20 BBH queries or 50 GSM and Git queries.Source: §6.1 · p. 8

6.2 Main Results & Further Analysis

  • FAME scores 0.786 on GSM spurious keeping, 0.790 on GSM spurious removal, 0.860 on GSM environment shift, 0.716 on Git knowledge conflict, and 0.815/0.861 on BBH spurious keeping/removal.On BBH spurious keeping, FAME improves over Input similarity by 58.5%.Source: §6.2 · pp. 8–9
  • Two case studies show FAME identifying false memory before answer generation: an addition association with “younger” and reversion from unary_union to cascaded_union.Responses are generated only for validation in these examples.Source: §6.2 · p. 9
  • As memory size increases, the reported false-memory rate rises in spurious keeping, while the knowledge-conflict retention rate also increases.The authors note that retrieving more can worsen false memory when memory induces a wrong belief in context.Source: §6.2 · p. 9

7 Conclusion

The conclusion presents FAME as a training-free way to evaluate false memory through counterfactual hidden-state concept drift. The related-work discussion distinguishes this focus on internal beliefs from approaches centered on answers or memory degradation during self-evolution.

7 Conclusion

  • FAME identifies false memory through hidden-state concept drift under counterfactuals, achieving 76.2%–96.7% AUROC while revealing cases not observable from surface answers.The conclusion describes false memory as arising from spurious correlations, environment shifts, or knowledge conflicts.Source: §7 · p. 9
  • FAME requires access to agent hidden states and counterfactual queries; automatic counterfactual generation and mitigation are identified as future work.The conclusion also names independent belief representations as a future direction.Source: §7 · p. 9

A Related Work

  • Prior work documents memory degradation during self-evolution, whereas this paper focuses on evaluating memory faithfulness under changing conditions.The related-work section motivates a formal framework for diverse practical scenarios.Source: A Related Work · p. 16
  • Unlike approaches evaluating answer effectiveness, this work tracks representation changes under counterfactual queries to assess whether adaptation reflects faithful belief or false memory.The related-work discussion says answer-effectiveness approaches require sampling answers and incur generation costs.Source: A Related Work · p. 16

B.1 Proof of Proposition 1

Under the stated Markovian independence assumption, the proof identifies the counterfactual updated-concept distribution by reducing an intervention conditional to an observational conditional.

B.1 Proof of Proposition 1

  • Because M is a non-descendant of Q, intervening on Q leaves M unchanged and permits reduction of the counterfactual conditional.The proof invokes the counterfactual reduction rule for pretreatment evidence.Source: §B.1 · p. 16
  • With Q a root variable and the Markovian assumption excluding unobserved common causes, Q is d-separated from Θ′ given M, allowing intervention to be exchanged for observation.This yields P(Θ′ | do(Q = q′), M = m) = P(Θ′ | Q = q′, M = m).Source: §B.1 · p. 16
  • The counterfactual distribution of Θ′ under do(Q = q′) is identifiable from the observational distribution.The proof equates it with P(Θ′ | Q = q′, M = m).Source: §B.1 · p. 16

B.2 Proof of Theorem 1

The section establishes counterfactual identifiability for concept drift, then relates hidden states to predictive concepts and answer preference. It also justifies the selected layer for concept extraction.

B.2 Proof of Theorem 1

  • The twin-network argument establishes counterfactual-factual identifiability under its Markovian assumption, linking the counterfactual distribution to an observational conditional.The network pairs factual and counterfactual worlds with shared exogenous variables; its d-separation result supports the identification.Source: §B.2 · p. 17
  • ETT characterizes concept drift as the difference between γh scores of updated-concept distributions under counterfactual and factual queries.The section writes this as Δ(m, q′, q0) = γh(P(Θ′ | m, q′)) − γh(P(Θ′ | m, q0)).Source: §B.2 · p. 18

B.3 Proof of Theorem 2

  • The answer-commit hidden state approximately recovers the predictive behavior associated with the memory-induced concept, with error from both demonstrations and hidden-state representation.When both approximation errors vanish, the decoded hidden state converges to the concept’s predictive distribution.Source: §B.3 · pp. 18–19

B.4 Proof of Theorem 3

  • The change in answer preference is the projection of concept drift onto the answer direction.Subtracting the leave-one-out preference expressions cancels their fixed bias term before averaging.Source: §B.4 · p. 19

C.1 Layer Choices of Concept Extraction

  • The layer sweep on Tomatoes supports the selected 14th layer as representative enough to extract the concept.The sweep assessed accuracy, AUROCs, and coupling r across layers.Source: §C.1 · p. 19

C.2 Experimental Setup

The setup evaluates false memory across math, code, and complex reasoning with constructed counterfactual templates. Case studies illustrate persistent spurious beliefs, knowledge conflicts, and memory-size effects.

C.2 Experimental Setup

  • The templates cover 40 GSM-Symbolic, 18 GitChameleon 2.0, and 11 BigBench-Hard tasks, with each template built around four memory examples.Counterfactuals vary spurious features or numerical scale in GSM, test legacy library versions in GitChameleon, and probe spurious correlation or environment shift in BigBench-Hard.Source: §C.2 · p. 20

D.1.1 Spurious Correlation

  • In spurious-correlation cases, agents retain memory-linked operations despite changed cues, sometimes calculating correctly before giving wrong, overconfident answers; FAME flags one case before answer generation.Examples include doubling despite a changed operation and multiplying again after correctly finding 10 per dozen.Source: §D.1.1 · pp. 20–21, Figure 11 · p. 21

D.1.2 Knowledge Conflict

  • In knowledge conflict, prior library knowledge can override memory; FAME distinguishes faithful from false cases even when answer confidences are nearly equal (0.964 vs. 0.959).The examples contrast a removed NetworkX function with a pinned version and a memory-specified Gradio function with the agent’s prior.Source: §D.1.2 · pp. 21–22, Figure 12 · p. 21

D.1.3 Effect of Memory Size

  • 1–8 demonstrations yield the correct answer 50, but from 12 onward the agent applies memorized doubling and answers 100; FAME’s signal tracks the switch.The query’s probe rank rises from the 26th to the 85th percentile as memory size increases.Source: §D.1.3 · p. 22, Figure 13 · p. 22

D.2 Ablation Study

The ablation study tests whether false-memory detection depends on matching the drift criterion to the expected behavior and using an informative readout direction. Results support both design choices.

D.2 Ablation Study

  • The ablations remove the regime-specific expectation, swap expectations across regimes, or replace the proposed readout direction with a random one.Experiments use 60% of templates and repeat across 10 trials.Source: §D.2 · pp. 22–23
  • 0.756 AUROC for FAME in spurious removal falls to 0.559 without the robustness expectation, 0.553 with the adaptation expectation, and 0.503 with a random direction.The random-direction result is close to chance.Source: §D.2 · p. 23
  • 0.779 AUROC for FAME in spurious keeping drops to 0.445 without the adaptation expectation and 0.464 with a random direction; the robustness expectation yields 0.674.The no-expectation and random-direction results are near or below chance.Source: §D.2 · p. 23
  • Concept drift alone is insufficient: effective evaluation requires judging drift against expected counterfactual behavior and an informative readout direction.Using a structured expectation from another regime can help, but does not match the regime-aligned criterion; a random direction does not explain the performance.Source: §D.2 · p. 23

D.3 Robustness Study

The section tests FAME’s robustness and cross-model generalization, then details counterfactual templates that vary memory-associated cues while preserving the underlying task. Across three LLMs, FAME outperforms other baselines in all reported false-memory settings.

D.3 Robustness Study

  • On GSM with Llama-3B, replacing the reference cloud with an average point caused only small degradation, while correctness-only filtering slightly improved results.These findings support the robustness of both the reference-cloud construction and the hidden-state readout direction.Source: §D.3 · p. 23

D.4 Generalization Across LLMs

  • For GSM experiments, the selected concept-extraction layers were 16th for Llama-8B, 14th for Mistral-7B, and 36th for Qwen2.5-3B.Layers were selected by sweeping a small subset; Figure 14 shows the layer sweep for the three models.Source: §D.4 · p. 24, Figure 14 · p. 24
  • FAME outperforms the other baselines in all false-memory settings across Llama-8B, Mistral-7B, and Qwen2.5-3B.The reported results are AUROCs on the GSM benchmark.Source: §D.4 · p. 24, Table 5 · p. 24, Table 6 · p. 24

E Counterfactual Templates

  • Each template pairs demonstrations inducing a target concept with counterfactual queries that preserve the task while changing a contextual factor that may be spuriously incorporated.Demonstrations share a task objective and behavioral pattern, with a potentially confounding feature repeatedly associated with the concept.Source: E Counterfactual Templates · pp. 24–25
  • Counterfactual interventions test spurious correlations, environment shifts, and knowledge conflicts, using placeholder entities to vary examples across templates.Spurious keeping changes the task relation while preserving the feature; spurious removal changes the feature while preserving the task relation.Source: E Counterfactual Templates · p. 25
  • Five query levels increase cues that attract interpretation toward the memory-associated feature while keeping the underlying counterfactual task fixed.Cues include explicit feature mentions, repetition, references to previous examples, and prompts encouraging the memory-consistent interpretation.Source: E Counterfactual Templates · p. 25

Related papers

Loading 2609.39473v1…