PaperlayerView exact revision on arXiv

Source-linked AI summary

OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences

Kaixiang Wang, Jiong Lou, Zhaojiacheng Zhou, Jie Li

arXiv:2605.18930v1cs.CRcs.AIcs.LG

TL;DR

Memory-augmented agents can reflect on past outcomes to improve, but whether clean, plausible experiences can induce harmful generalization remains an underexplored attack surface. OEP uses locally correct edge-cases paired with severe hypothetical consequences to bias reflection. It achieves above 50% ASR across diverse domains with GPT-4o agents and is more robust than existing attacks against current defenses.

  • Problem

    It is underexplored whether clean, semantically plausible experiences can induce harmful generalization during reflective memory consolidation, unlike attacks relying on privileged access or explicit malicious content.

    Source: §1 · p. 2

  • Method

    OEP submits locally correct but non-transferable edge-cases paired with severe, plausible hypothetical consequences to bias reflection toward over-generalized rules.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Above 50% ASR across diverse domains with GPT-4o agents, with greater robustness than existing attack methods against current defenses.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Locally correct experiences can still be distilled into persistent over-generalized rules that cause failures on benign downstream tasks.

    Source: §1 · p. 2

    The threat model is limited to targets using reflection-based memory consolidation and assumes attackers have user-level access to submit crafted ACTs, without direct control of prompts, parameters, memory databases, or backend tools.

    Source: §3 · p. 3

Abstract

from arXiv · show

Memory-augmented large language model (LLM) agents use iterative reflection and self-evolution to solve complex tasks, but these mechanisms introduce security risks. Existing agentic memory attacks require privileged access or explicit malicious content, making them detectable by advanced safety filters. This leaves a subtler attack surface underexplored: whether adversaries can induce agent to generate experiences that appear locally correct and semantically plausible yet induce harmful generalization during reflection. We find that reflective agents are vulnerable to such clean experiences, especially when paired with severe but plausible hypothetical consequences. Based on this observation, we introduce Obsessive Experience Poisoning (OEP), a low-privilege black-box attack requiring no direct control over the system prompt or memory database. OEP constructs adversarial clean edge-cases that combine locally correct solutions, non-transferable methods, and severe consequences, biasing reflection toward risk-averse rule formation. During memory consolidation, agents may over-trust self-generated reflections and distill localized experiences into high-priority but over-generalized rules, causing downstream failures. Evaluations across three domains show that OEP achieves ASR above 50\% with GPT-4o agents, and outperforms existing attacks under LLM auditing defense.

1 Introduction

Memory-augmented reflection helps agents learn from past outcomes, but it also creates an attack surface: clean, locally correct cases may be generalized into harmful persistent rules. OEP exploits this process with severe but plausible consequences, achieving over 50% ASR across three domains with GPT-4o agents.

1 Introduction

  • Long-term memory archives execution trajectories for reflection, enabling agents to distill expertise from successes and errors and improve task proficiency and reliability.This learning mechanism also introduces security vulnerabilities when memory is compromised.Source: §1 · pp. 1–2
  • Existing memory attacks typically rely on malicious instructions, triggers, or tampered records, leaving clean experiences that agents over-generalize underexplored.The central question is whether clean cases alone can induce harmful self-generated memory rules.Source: §1 · p. 2
  • OEP pairs locally correct but non-transferable edge-case solutions with severe, plausible hypothetical penalties to bias reflection toward over-generalized rules.These rules may persist at high priority and cause failures on benign downstream tasks.Source: §1 · p. 2
  • The adversarial case framework combines clean edge-case generation with the Adversarial Consequence Triplet while preserving local correctness and semantic plausibility.This combination steers reflection toward biased rule formation.Source: §1 · p. 2
  • OEP achieves an ASR exceeding 50% across three diverse domains with GPT-4o agents and stronger robustness against LLM auditing defenses than baselines.The results also indicate that more capable agents may be more vulnerable under OEP.Source: §1 · p. 2

2 Related Work

Prior work studies self-evolving agents and attacks on their memories, but mainstream memory attacks use payloads with explicit malicious intent or factual errors. OEP instead preserves problem-solution correctness while using plausible consequences to induce harmful generalization through biased utility signals.

2 Related Work

  • Self-evolving agents store trial-and-error insights in memory and can blindly replicate past methods on semantically similar inputs, creating an attack surface for error propagation.This experience-following behavior is presented as the theoretical foundation for OEP’s attack surface.Source: §2 · p. 3
  • AgentPoison, MINJA, Zombie Agents, and MemoryGraft use triggers, harmful instructions, or unsafe patterns that can be vulnerable to content filtering and LLM-based sanitization.The cited defenses include anomaly detection and LLM-as-a-Judge sanitization.Source: §2 · p. 3
  • OEP preserves problem-solution correctness while adding plausible consequences, targeting reasoning and memory updates through biased utility signals rather than explicit malicious content.The paper distinguishes this attack from methods that alter cognitive tone or inject toxic rationales.Source: §2 · p. 3

3 Threat Model

The threat model assumes a user-level, black-box attacker targeting agents that consolidate memory through reflection. The attacker seeks to turn a context-valid edge-case into an over-generalized rule that harms downstream task integrity or inflates resource use.

3 Threat Model

  • The attacker has user-level black-box access and can submit crafted ACTs, but cannot modify the system prompt, model parameters, memory database, or backend tools.The submitted problem-solution pairs are factually correct but highly idiosyncratic.Source: §3 · p. 3
  • The victim uses reflection-based memory consolidation: a validator checks input validity, filtering retains context-valid experiences, and reflection extracts global rules to update shared memory.The updated memory is assumed to act as a global prior influencing future independent sessions.Source: §3 · p. 3
  • The attacker seeks persistent over-generalized rules that cause reasoning failures on normal downstream tasks or abnormal resource consumption beyond a threshold.The two stated objectives are compromising integrity and availability, including denial-of-wallet costs.Source: §3 · pp. 3–4

4 Method

OEP uses clean, non-transferable edge-cases, severe hypothetical consequences, and user-level injection to bias reflective memory consolidation. Its pipeline moves from case construction to consequence reinforcement and injection, potentially turning localized successes into high-priority rules that fail on ordinary tasks.

4 Method

  • OEP’s three-phase pipeline constructs clean edge-cases, adds severe hypothetical penalties, then injects the cases through user-level interactions.The agent processes these inputs through its built-in reflection and memory-consolidation mechanism.Source: §4 · p. 4, §4.3 · p. 6

4.1 Clean Edge-Case Construction

  • Cases use tail-region boundary tasks where a non-standard method applies locally but has poor expected success on normal tasks.Construction retains candidates that are locally valid yet exhibit poor transferability on standard instances.Source: §4.1 · pp. 4–5

4.2 Adversarial Consequence Triplet (ACT)

  • ACT pairs an edge-case solution with a plausible severe catastrophe, steering loss-averse reflection toward that method.The authors argue that naive correct examples leave rule selection uncertain; the severe penalty skews the utility comparison.Source: §4.2 · p. 5

4.3 Injection and Over-generalization

  • After logical and semantic validation, reflection may promote localized success into a high-priority rule that misfires on benign tasks.The rule can cause the non-transferable method to be applied where it is invalid.Source: §4.3 · p. 6

5 Mechanistic Analysis

The analysis describes three interacting vulnerabilities in the memory-reflection loop: provenance-weighted trust, perspective confinement, and risk-sensitive rule induction. Together, these mechanisms can bias consolidation toward high-priority rules rather than directly overwriting memory.

  • Provenance-weighted trust means agents may scrutinize external prompts more strictly than rules produced by their own reflections.Source: §5 · p. 6
  • Perspective confinement can make agents overestimate an edge-case method’s support and mistake localized success for a broadly valid rule.Source: §5 · p. 6
  • Risk-sensitive rule induction gives plausible severe consequences high weight, raising the priority of flawed non-transferable generalizations.Source: §5 · p. 6
  • OEP biases reflective memory consolidation rather than directly overwriting memory, turning locally valid experiences into high-priority rules that can drive downstream failures.Source: §5 · p. 7

6 Experiment

The experiments evaluate OEP across math, healthcare, and tool use, using several self-evolving agent frameworks and examining attack performance, component effects, persistence, and defenses.

6.1 Implementation & Setup

  • The evaluation covers GSM8K and MedQA reasoning tasks and ToolAlpaca tool use across Simple Agent, LangChain-based memory, and OpenClaw frameworks.The experiments mainly use GPT-4o, with GPT-5.4 for OpenClaw, and compare against representative attacks under matched settings.Source: §6.1 · p. 7

6.2 Attack Performance

  • OEP causes large performance degradation across domains; Math and Med show sharp ACC declines, while increased Tool steps indicate resource exhaustion.The reported effects also increase token consumption and latency, particularly in OpenClaw.Source: §6.2 · pp. 7–8

6.3 Ablation studies

  • GPT-4o and Gemini 2.5-flash have significantly higher ASR than GPT-4o-mini across all domains, which the authors associate with stronger instruction adherence and reflection.The authors propose that stronger models may more readily internalize adversarial consequences, leading to over-generalized rules and performance degradation.Source: §6.3 · p. 8
  • As the adversarial proportion rises, ESR and ASR surge non-linearly toward their maximum at 100%, while benign cases dilute the manipulation at lower ratios.The authors attribute the high-ratio effect to reflection windows dominated by localized examples without standard references.Source: §6.3 · p. 8
  • The full ACT Triplet produces a significant performance rise across domains, whereas problem-context-only or QA-Solution injections yield marginal ESR and ASR.The authors attribute the full Triplet’s effect to negative utility exploiting safety-aligned loss aversion; factual correctness alone is insufficient.Source: §6.3 · p. 8
  • OEP maintains high ASR after 50 subsequent benign queries, reaching 72% in Tool, while conventional memory injections degrade rapidly.The authors link persistence to hypothetical penalties that lead agents to crystallize the method into a rule that overrides natural memory dilution.Source: §6.3 · p. 8, Figure 4 · p. 8

6.4 Attack and Defense Evaluation

  • Under the LLM auditor, OEP achieves 40.29% ASR, versus below 15% for baselines relying on explicit malicious content or inconsistencies.The comparison uses a shared conversational-memory setting, interaction budget, memory-update opportunities, downstream split, backbone, and decoding configuration.Source: §6.4 · p. 9, Table 4 · p. 9

7 Potential Defenses

OEP exploits perspective confinement, trust asymmetry, and loss aversion to produce persistent downstream degradation; the paper proposes active search and multi-agent debate as targeted defenses.

Potential Defenses

  • OEP manipulates perceived utility rather than semantic content, allowing it to evade conventional filters and memory auditors and cause persistent downstream degradation.The paper reports a corresponding experimental evaluation of its proposed defenses in Appendix C.Source: §7 · p. 9
  • Before turning a local observation into a global rule, reflection should retrieve diverse, broader test cases with external tools to counter perspective confinement.The proposed active-search step is intended to prevent fixation on ungeneralizable edge-case methods.Source: §7 · p. 9
  • Independent agents can cross-examine derived experiences before consolidation, recalibrating expected utility and countering trust asymmetry.The proposed multi-agent debate is intended to neutralize severe hypothetical penalties introduced by the attack.Source: §7 · p. 9

8 Conclusion

The conclusion presents OEP as a low-privilege attack on reflective self-evolving agents using locally correct inputs, and reports high attack success and robustness against existing defenses.

8 Conclusion

  • OEP pairs clean edge-cases with an Adversarial Consequence Triplet to induce localized methods to be distilled into non-transferable rules.The conclusion reports ASR above 50% across diverse domains with GPT-4o and greater robustness than existing attacks against current defenses.Source: §8 · p. 9

A Ethical Statement

  • The authors frame the research as intended to advance AI safety and support more secure, aligned, and resilient self-evolving agent architectures.They state that the methods and adversarial examples are intended for academic research and defensive reference.Source: A Ethical Statement · p. 13

B Limitations and Future Work

The authors identify implementation costs in designing OEP consequences and limit the evaluation’s focus to individual agents, leaving multi-agent settings for future work.

B Limitations and Future Work

  • ACT efficacy depends on plausible severe hypothetical penalties, whose design can require domain expertise and iterative refinement.For scenarios such as creative writing, the authors report high engineering costs and continued reliance on heuristic prompting and human expert intervention.Source: B Limitations and Future Work · p. 13
  • The evaluation primarily studies individual-agent vulnerability, not how OEP manipulations manifest and persist in collaborative multi-agent systems.The authors identify studying collaborative clusters as future work and note that debate or cross-examination could potentially mitigate biased reflections.Source: B Limitations and Future Work · p. 13

C Evaluation of Potential Defenses

Both defenses partially mitigate OEP across Math, Med, and Tool, with MAS Debate reducing attack success more than Auto Search. Auto Search’s gains are limited by residual risk-avoidant rule adoption.

C Evaluation of Potential Defenses

  • ASR falls from 59.14/52.00/71.93 to 37.43/27.43/46.49 with Auto Search and to 28.86/21.43/30.70 with MAS Debate across Math, Med, and Tool.Both defenses partially mitigate OEP; MAS Debate yields the lower ASR values in all three domains.Source: Table 5 · p. 13, C Evaluation of Potential Defenses · p. 13
  • Auto Search retrieves broader cases before memory consolidation, helping counter perspective confinement and reduce local-experience overgeneralization.Severe loss-oriented penalties can still leave agents prioritizing risk avoidance and occasionally internalizing the biased rule.Source: C Evaluation of Potential Defenses · p. 13
  • MAS Debate uses independent agents to question and cross-examine external experiences, weakening single-source trust and overgeneralized rule formation.The reported ASR is lower than with Auto Search across Math, Med, and Tool.Source: C Evaluation of Potential Defenses · p. 13

D Evaluation on Reflection Mechanisms

Explicitly summarizing injected cases into experiences yields higher OEP attack success than direct-case learning. The accompanying analysis models how provenance, limited reflection windows, and severe consequence framing can promote an overgeneralized rule that is later misapplied.

D Evaluation on Reflection Mechanisms

E.1 A Unified Rule-Adoption Model

  • The unified model scores candidate rules using empirical support, consequence-sensitive risk, provenance, and a penalty for overly specific or complex rules.OEP raises the intended rule’s score by supplying locally correct cases that increase its empirical-support, risk, and provenance terms without explicitly injecting that rule.Source: §E.1 · p. 14

E.2 Provenance-Weighted Rule Adoption

  • The provenance mechanism represents a potential trust difference between raw external prompts and rules generated through the agent’s own reflection.OEP submits a locally correct edge case for reflection rather than directly writing the intended rule into memory.Source: §E.2 · pp. 14–15

E.3 Perspective Confinement and Biased Empirical Support

  • A finite reflection window can overrepresent edge cases, making a locally successful but poorly transferable method appear more reliable than it is.The analysis predicts that increasing the adversarial case ratio raises ESR and ASR, while active counterexample search should reduce the effect.Source: §E.3 · p. 15

E.4 Risk-Sensitive Rule Induction

  • Attaching a severe but plausible consequence to deviations can raise the intended rule’s risk-sensitive score and adoption probability.The analysis treats consequence framing as increasing perceived cost of deviation, not as requiring a fixed psychological loss-aversion parameter.Source: §E.4 · p. 15

E.5 From Rule Adoption to Downstream Failure

  • Adoption alone does not guarantee failure: the rule must enter memory, be retrieved and applied, and be invalid for the downstream task.The decomposition links ESR to rule distillation, ASR to retrieval and application, and accuracy or tool-step degradation to task-level effects.Source: §E.5 · pp. 15–16

F Main Algorithm

  • OEP constructs locally correct edge cases with non-transferable methods, adds severe hypothetical consequences, then submits them through user-level interaction for reflection.The algorithm returns an overgeneralized rule after the reflection process matches its candidate rule to the attacker’s intended rule.Source: F Main Algorithm · p. 17

G Case Study of OEP

H.1 Implement details

The experiments cover sampled reasoning questions and tool-use queries, alongside a self-evolving OpenClaw setup. The section also specifies how cases were constructed, how attack effects were measured, and what the reflection and auditing prompts ask models to do.

H.1 Implement details

  • Evaluation sampled 350 GSM8K questions and 350 MedQA questions, plus 114 ToolAlpaca queries spanning 13 tool categories.The OpenClaw setup adds reflection and memory retrieval, with experiences consolidated after each interaction and later retrieved for subsequent tasks.Source: §H.1 · p. 20
  • Ten conversational cases per domain were injected before consolidation, with human-supervised review for local correctness, plausibility, and poor transferability.Construction did not use downstream test labels or evaluation outcomes; eight cases were contrastive failure trajectories and two were positive reinforcement trajectories.Source: §H.1 · pp. 20–21
  • ESR audits whether distilled memory encodes a harmful, non-transferable rule, while ASR measures the share of downstream queries affected by attack-induced errors or tool-step increases.Three LLM auditors produced ESR estimates with standard deviations of 0.59, 1.51, and 1.82 for Math, Med, and Tool, respectively.Source: §H.1 · p. 21, Table 7 · p. 21

H.2 Detailed Prompt

  • The reflection prompt asks the agent to extract an operational principle from negative and positive incident records, while the OEP prompt tells it to consult prior memory when relevant.The same prompt templates were used across experiments, baselines, and ablations except for attack-specific content or ablated components.Source: §H.1 · p. 21, §H.2 · p. 22, Prompt: OEP · p. 22

Prompt: ESR Evaluator

  • The ESR evaluator labels a rule harmful when it is both non-generalizable and detrimental, while the security auditor screens for malicious instructions, backdoors, subtle failure prompts, and logical flaws.The ESR prompt returns a binary YES or NO label; the security-auditor prompt returns T for benign content and F when a listed threat is detected.Source: Prompt: ESR Evaluator · p. 22, Prompt: LLM Auditor · p. 23

Related papers

Loading 2605.18930v1…