PaperlayerView exact revision on arXiv

Source-linked AI summary

Honest Lying: Understanding Memory Confabulation in Reflexive Agents

Prakhar Dixit, Sadia Kamal, Tim Oates

arXiv:2605.29463v2cs.LGcs.AI

TL;DR

Reflexion-style agents may store confident but incorrect self-diagnoses as reflective memory, raising whether self-generated lessons remain reliable across trials. The paper operationalizes this failure as memory confabulation, diagnoses repeated reliance with RRR, and finds that programmatic feedback extraction reduces repetition and partially improves outcomes.

  • Problem

    Reflexion agents may write and reuse confident but incorrect accounts of task failures, motivating examination of whether verbal self-diagnosis creates persistent false beliefs.

  • Method

    The paper defines memory confabulation, introduces the log-based Reflection Repetition Rate (RRR), and applies programmatic extraction of trajectory-level failure signals.

  • Results

    Across 16 frozen ALFWorld environments and 121 reflections, the correct target object was mentioned zero times; mitigation reduced RRR from 0.64 to 0.10 and solved 3 of 16 environments.

  • Takeaways & Limitations

    Reflective memory can be actively harmful, so memory write-path validation is as important as retrieval quality.

  • Takeaways & Limitations

    The study examines only the Reflexion architecture, and its target-object mention signal is sufficient but not necessary for detecting confabulation.

Abstract

from arXiv · show

Reflexion-style agents rely on self-generated reflections as memory, implicitly assuming that agents can accurately diagnose their own failures. We show that this assumption can fail systematically: across ALFWorld and HumanEval, agents store confident but incorrect interpretations of the task and continue acting on them across trials, even though the environment resets to the correct task each time. We call this failure mode memory confabulation and introduce the Reflection Repetition Rate (RRR), a log-based metric that detects repeated reliance on incorrect reflective content. Using RRR, we identify 16 frozen environments in ALFWorld, where 0 of 121 reflections mention the correct target object, and 4 analogous cases in HumanEval. Our mitigation replaces open-ended self-diagnosis with programmatic extraction of trajectory-level failure signals, increasing correct object mention from 0% to 86%, reducing RRR from 0.64 to 0.10, and solving 3 of 16 frozen ALFWorld environments, suggesting that reflective memory can reinforce false beliefs rather than correct them.

1. Introduction

Reflexion-style agents use self-generated reflections as reusable memory, but can systematically store and reuse confident false task interpretations. The paper names this failure memory confabulation, measures repeated reflective content with RRR, and reports cross-domain evidence plus a programmatic mitigation.

  • Reflexion converts failed attempts into natural-language reflections retrieved on later trials, assuming self-diagnosis produces useful failure diagnoses.When this assumption holds, failures become reusable lessons.
  • Memory confabulation occurs when agents store confident but incorrect task accounts and reuse them across trials despite the environment re-presenting the correct task.The false beliefs become persistent, self-reinforcing reflective memory rather than one-time generation errors.
  • Memory confabulation differs from hallucination because false content is stored, retrieved, acted upon, and reinforced across multiple trials.The distinction motivates asking whether reflective agents can safely rely on their own self-generated lessons.
  • RRR is introduced as a log-based diagnostic for frozen reflective memory and correlates with trials-to-solve in ALFWorld at r = 0.808.RRR operationalizes repeated reliance on reflective content.
  • 0/121 reflections across 16 frozen ALFWorld environments mention the correct target object, while 4 HumanEval problems repeat near-identical wrong diagnoses.The evidence spans embodied task interaction and code generation.
  • A no-memory ablation shows reflective memory can be actively harmful in some environments the agent can otherwise solve.
  • 86% correct object mention, RRR from 0.64 to 0.10, and 3/16 frozen ALFWorld environments solved followed programmatic extraction of trajectory-level failure signals.The mitigation replaces open-ended self-diagnosis with parsed failure signals.

2. Background

Reflexion mediates learning through retrieved verbal self-critique without gradient updates, while related work frames reflective memory as vulnerable to self-reinforcing error. Memory confabulation is distinguished from single-generation hallucination by its persistent reuse across trials.

  • Reflexion prepends a language model’s natural-language self-critique after failure and performs no gradient updates, making learning entirely context-mediated.It reports 91% pass@1 on HumanEval versus 80% for GPT-4 without reflection when the mechanism works correctly.
  • Reflective memory is described as a write–manage–read loop whose central risk is self-reinforcing error when false conclusions prevent testing an approach again.The paper links this risk to binary feedback that prevents causal diagnosis.
  • ExpeL shares Reflexion’s vulnerability because unconstrained critique can entrench confabulated rules, which are then applied across evaluation environments.Two AGREE votes can make a false rule globally applicable.
  • Hallucination is characterized as a single-generation failure, whereas memory confabulation stores, retrieves, and acts on false content across multiple trials.

3. Problem Formulation

The paper defines memory confabulation from whether reflections mention the task’s correct target object and uses RRR to identify reflective memory that stops evolving. It further separates memory-harmful environments from task-hard environments.

  • 3.1. Operational Definition: After each failure, the agent generates reflection r_t from its trajectory and stores it in memory M_{t+1} = M_t ∪ {r_t}.
  • 3.1. Operational Definition: At trial t + 1, the agent retrieves the updated memory to inform its actions.
  • 3.1. Operational Definition: A reflection is confabulated when it fails to mention the correct target object extracted from the task description presented at every episode start.
  • 3.1. Operational Definition: The confabulation definition can be applied to existing logs using the gamefile directory name and stored reflection text, without new experiments.
  • 3.2. Reflection Repetition Rate (RRR): RRR measures the fraction of reflections after the first that are near-duplicates of earlier reflections, using SequenceMatcher similarity with a 0.85 threshold.RRR = 0 means all reflections are novel; RRR = 1 means all are near-copies.
  • 3.2. Reflection Repetition Rate (RRR): RRR ≥ 0.5 defines frozen reflective memory, meaning at least half of later reflections are near-duplicates and the memory stops evolving across trials.“Frozen environment” refers to reflective memory, not a fixed task or simulator state.
  • 3.3. Two Failure Categories: Frozen environments are divided into memory-harmful and task-hard categories.
  • 3.3. Two Failure Categories: Memory-harmful environments become solvable faster when memory is removed, showing that stored reflections actively mislead the agent.

4. Evidence of Memory Confabulation

Across Reflexion logs, repeated reflective content tracks persistent failure across domains, including false task identities and incorrect object references. The evidence also shows that feedback specificity distinguishes targeted self-correction from frozen or symptom-level confabulation, while memory can either harm solvable tasks or compound capability gaps.

  • 4.1. RRR Analysis: 32% of ALFWorld environments requiring reflection exhibit frozen memory, with RRR ≥0.5.Frozen environments required an average of 7.6 trials to solve versus 1.5 for environments with diverse reflections; RRR correlates with trials-to-solve at r = 0.808.
  • 4.2. Task-Object Confabulation: 0 of 121 reflections across 16 frozen ALFWorld environments mention the correct target object.The target object was extracted from each environment’s gamefile directory name and compared against reflection content.
  • 4.2. Task-Object Confabulation: In env 22, all 14 reflections substituted tomato and microwave for the correct Mug-and-CoffeeMachine task.The false task identity began after trial 0 and persisted across 14 trials despite the correct task being re-presented at every reset.
  • 4.2. Task-Object Confabulation: Frozen environments show both full task substitution and object-only substitution, with destinations sometimes retained correctly.Full substitution replaces object and destination, whereas object-only substitution replaces the object while preserving the destination.
  • 4.3. Cross-Domain Replication: 46% of HotpotQA questions remained unsolved across seven trials, while correction per trial transition was 5.9%, compared with 64% for ALFWorld and 83% for WebShop.Binary correct/wrong feedback does not identify which step in a multi-hop reasoning chain failed, limiting targeted self-correction.
  • 4.3. Cross-Domain Replication: 56% (121/218) of frozen WebShop reflections describe what went wrong without diagnosing which constraint was violated.This symptom confabulation reflects binary feedback that lacks step-level information; HumanEval’s exact assertion feedback supports more targeted reflection and a 17% persistent failure rate.
  • 4.4. Causal Analysis: 2/16 ALFWorld environments are memory-harmful, while 14/16 are task-hard even without memory.The memory-harmful environments solve in one trial without memory but require 7 or 8 with standard Reflexion memory; task-hard environments fail within 10 trials either way.

5. Mitigation and Results

The paper tests structured and programmatic alternatives to open-ended self-diagnosis across frozen ALFWorld and HumanEval cases. Programmatic extraction grounds reflections in trajectory or test failures, improving diagnostic specificity and reducing repeated false-memory patterns, though capability gaps and one regression remain.

  • Programmatic Feedback Extraction: Programmatic feedback extraction parses trajectory-level failure signals instead of asking agents to self-diagnose.For ALFWorld, it extracts “Nothing happens” responses and repeated identical actions; for HumanEval, it extracts failing assertions, error types, and traceback messages.
  • ALFWorld Results: 86% (134/156) of ALFWorld reflections mention the correct target object, up from 0 of 121 under standard Reflexion.The remaining 22 misses are concentrated in env 4, where SoapBar search produces partial regex matches.
  • ALFWorld Results: RRR falls from 0.64 to 0.10, showing that programmatic extraction breaks frozen memory patterns even when some environments remain unsolvable.The mitigation confirms that reducing reflective repetition and solving the underlying task are separable outcomes.
  • ALFWorld Results: Programmatic extraction solves 3/16 frozen ALFWorld environments, uniquely resolving env 35 after surfacing a specific “Nothing happens” response.The agent abandoned a frozen location plan, searched for an alternative desklamp, and solved env 35 in 4 trials.
  • HumanEval Results: 100% (18/18) of HumanEval reflections mention the specific error type, while average RRR decreases from 0.59 to 0.44.This grounds reflections in failing test evidence rather than vague, repeated diagnoses.
  • HumanEval Results: Capability gaps persist despite grounded reflections, and HumanEval/77 regresses from solved to unsolved after the structured prompt disrupts a working strategy.HumanEval/32 and /84 still require algorithmic insight beyond memory-level intervention.

6. Discussion

The paper argues that binary feedback can leave reflective agents vulnerable to confabulation, while stronger models may replace task-identity errors with other action or memory-format failures. It also limits its empirical claims to Reflexion, small causal and HumanEval samples, and a target-object grounding proxy.

  • Cross-domain mechanism: Binary feedback gives agents no information about which step failed, whereas unit tests provide step-level signals that reduce confabulation.The paper reports confabulation falling from 32–82% to 17% with unit-test feedback.
  • Model capability and confabulation: gpt-4o-mini correctly names the target object in 100% of reflections but solves only 2/16 environments, matching the no-memory baseline.This separates preventing task-identity confabulation from solving the underlying task.
  • Model capability and confabulation: Stronger models can trade task-identity confabulation for memory-format or action-space confabulation.The reported examples involve planning text emitted as actions and invalid natural-language commands that loop.
  • Generalization: Memory confabulation is presented as a structural vulnerability arising from binary feedback, self-generated reflection, and persistent retrieval.The paper extends this concern to rule-library agents such as ExpeL, where confabulated rules can be applied globally.
  • Limitations: The study evaluates only Reflexion, uses a causal ablation on 2 of 16 environments and four HumanEval problems, and measures confabulation through target-object mention.The authors caution that other architectures, configurations, and unmeasured error dimensions may differ.

7. Conclusion

The conclusion finds that Reflexion can store and reinforce false task beliefs, making reflective memory harmful on some tasks. It argues that validating what enters memory is as important as improving retrieval.

  • Conclusion: Across 16 frozen environments and 121 reflections, the correct target object was mentioned zero times under standard Reflexion.Two environments required 7–8 trials with memory but solved in 1 trial without it.
  • Conclusion: Grounding interventions prevented confabulation on solvable tasks but did not resolve hard-task failures, identifying feedback granularity as the fundamental bottleneck.The conclusion distinguishes memory-write failures from task-level execution failures.
  • Conclusion: The paper argues that write-path validation should complement retrieval improvements because plausible but false stored beliefs can be worse than no memory.It proposes knowing when not to write or validating causal accuracy before storage.

A. Prompt for Programmatic Extraction

The programmatic-extraction prompt replaces standard self-diagnosis with trajectory-derived failure information before reflection generation. It supplies the task, concrete failure steps, and an action-relevant trajectory context to the language model.

  • Prompt construction: Failure steps are extracted programmatically from the trajectory before the replacement reflection prompt is constructed.This changes the input to reflection generation from open-ended self-diagnosis to parsed evidence.
  • Prompt inputs: The prompt includes the unsuccessful task history and a task line extracted directly from the trajectory.The task line comes from the line containing “Your task is to:”.
  • Failure signals: The failure block records actions receiving “Nothing happens” responses or repeated identical actions indicating a loop.These trajectory-level signals are inserted into the reflection prompt before the LLM generates the reflection.
  • Context filtering: Previous plans are truncated to action-relevant content to filter analysis text that could leak into action generation.The prompt template therefore preserves concrete failure context while restricting irrelevant planning text.
Loading 2605.29463v2…