PaperlayerView exact revision on arXiv

Source-linked AI summary

Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection

Xinyan Wang, Xiaogeng Liu, Jun Shao, Chaowei Xiao

arXiv:2610.39122v1cs.CRcs.AI

TL;DR

The paper asks whether self-evolution makes personal agents less robust to indirect prompt injection, which can induce actions the user did not authorize. It tests base and evolved agents with a controlled two-phase methodology and finds higher attack success rates under both skill evolution and RL.

  • Problem

    It is unknown whether personal agents’ initial robustness to indirect prompt injection persists as they learn from user feedback after deployment, despite the risk that injected instructions induce unauthorized actions.

    Source: §1 · p. 1

  • Method

    A controlled two-phase evaluation adapts agents on user-authorized tasks in external content, then tests them on new content with similar tasks that the user has not authorized.

    Source: §3.2 · pp. 4–5

  • Results

    Average ASR rises by 14.0–48.8 percentage points under skill evolution and by 3.0–39.3 percentage points under RL.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Authorization conflation is the main cause identified, and feedback distinguishing authorized from unauthorized execution prevents most of the ASR increase under skill evolution.

    Source: §5 · pp. 9–10

    The experiments use short adaptation series with local files and simulated online services, not months of daily use, and cover a limited range of scenarios.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to read external content and complete the task described in it, a common request in daily use. The agent then learns from the user's feedback and improves itself as designed. In the test phase, a third party writes a similar task into new external content, but the user's test request does not authorize the injected task. Using this methodology, we build multiple tasks with diverse test instances in different domains for a real personal agent. Our extensive experiments reveal that self-evolution substantially degrades robustness to IPI. Specifically, compared with the base agent, which does not evolve from user feedback, the average attack success rate rises by 14.0--48.8 percentage points under skill evolution and by 3.0--39.3 percentage points under reinforcement learning, posing safety risks. The code is available at https://github.com/SaFo-Lab/Evolve-IPI

1. Introduction

The paper asks whether learning from user feedback makes personal agents more vulnerable to indirect prompt injection, where an attacker hides instructions in external content. Its controlled evaluation finds degradation under both reinforcement learning and skill evolution, attributing the problem to agents learning task execution without preserving the user-authorization boundary.

1. Introduction

  • The central gap is whether self-evolution preserves IPI robustness, since prior evaluations test fixed configurations even though personal agents continue learning after deployment.IPI can let an untrusted third party induce actions the user has not authorized.Source: §1 · p. 1
  • Authorization conflation occurs when approval for completing an authorized task teaches an agent to execute instructions in content without checking current user authorization.The paper identifies it as the main cause of the observed degradation; the behavior can emerge without malicious users or poisoned feedback.Source: §1 · pp. 1–3
  • The controlled two-phase test compares base and self-evolved agents on new external content containing an injected task the user has not authorized.A deterministic system-state verifier checks whether the task was actually completed; within each framework, agents differ only in learned weights or inserted skills.Source: §1 · pp. 2–3
  • Average ASR rises by 14.0–48.8 percentage points under skill evolution and by 3.0–39.3 percentage points under RL across five tasks.After 12 adaptation episodes, skill evolution raises GPT-OSS-120B mean ASR from 4.7% to 53.5%.Source: §1 · pp. 2–3
  • Controlled experiments identify authorization-sensitive feedback as important: ASR falls to 5.8% when positive feedback follows not executing an embedded task, versus 35.0% with added read-only conversations and 35.8% without them.An evolved skill also directly instructs the agent to execute embedded instructions without checking user authorization.Source: §1 · p. 3
  • Delimiting tool outputs reduces ASR by roughly half but leaves degradation, so a pre-deployment robustness check does not establish that robustness persists after self-evolution.The paper argues that defenses for self-evolving agents must address authorization conflation.Source: §1 · p. 3

2. Related Work

Prior work studies learning and safety in evolving agents, indirect prompt injection in fixed configurations, and attacks that poison agent memories or experience. This paper differs by testing whether ordinary self-evolution itself degrades personal agents’ IPI robustness, without attacker influence on learning.

2. Related Work

  • Personal-agent frameworks learn from user feedback after deployment by updating foundation models or retaining reusable skills, making daily use a source of self-evolution.OpenClaw-RL turns conversations into RL reward signals, while MetaClaw learns reusable skills from sessions [18] [19].Source: §2 · p. 3
  • IPI research evaluates fixed agent configurations and develops defenses that separate trusted instructions from external content, including delimiting tool outputs [24] [25] [15] [1] [6].The paper investigates whether such robustness persists as agents learn from feedback after deployment.Source: §2 · p. 3
  • Prior safety work reports degradation from agent evolution and reuse of benign-task experience, alongside risks from benign-data fine-tuning and agentic RL [11] [27] [22].Source: §2 · p. 3
  • Other attacks poison memories, knowledge bases, experiences, or learned skills, or arrange benign tasks to produce harmful behavior; these approaches influence the agent’s learning experiences [3] [13] [17] [23] [20].Source: §2 · p. 4
  • Unlike work on harmful-request refusal or poisoned experience, this paper shows that learning to complete user-authorized tasks can teach agents to follow third-party instructions without attacker influence on learning.It tests this degradation under both RL and skill evolution and identifies authorization conflation as its main cause.Source: §2 · p. 4

3. Authorization Conflation

The section defines authorization conflation as learning to execute tasks found in external content without checking whether the current user authorized them. It tests this risk with a controlled comparison of base and self-evolved agents on new content.

3.1. Threat Model

  • The threat model gives an untrusted third party control of external content, but not user requests, feedback, or the agent; an attack succeeds when the agent completes an unauthorized embedded task.The attacker inserts a similar task into new content and is assumed to know the kinds of actions users commonly request.Source: §3.1 · p. 4

3.2. Controlled Two-Phase Methodology

  • During adaptation, positive feedback rewards completion of authorized embedded tasks, which can teach the overgeneralized rule to execute tasks found in content regardless of authorization.The paper calls this mistaken generalization authorization conflation; under the test request, the user asks for the primary task but not the embedded task.Source: §3.2 · p. 4
  • The controlled evaluation compares base and self-evolved agents on new content under an unauthorized request, using verified task completion to measure attack success.Setting 3 measures ASR on new content; settings 1 and 2 measure authorized execution and unauthorized execution on adaptation content, respectively. The authorization gap is setting 1's execution rate minus setting 2's ASR.Source: §3.2 · pp. 4–5

3.3. Mechanisms of Self-Evolution

  • OpenClaw-RL updates the foundation model from conversations recorded by a proxy, using process-reward scores as the advantage in a clipped policy-gradient update of the LoRA adapter.The process reward model scores each assistant turn from the subsequent tool result or user message; majority vote over three scores determines the reward.Source: §3.3 · p. 5
  • MetaClaw keeps the foundation model fixed and learns reusable textual skills from completed sessions, then retrieves selected skills for insertion into later system prompts.The retrieved skills include general, task-specific, and common-mistake categories, so which feedback lessons reach the agent depends on retrieval selection.Source: §3.3 · p. 5

4. Experiments

Across five tasks and three domains, self-evolution increased attack success rates under both skill evolution and reinforcement learning. Controlled tests connect the degradation to authorization conflation; it transfers across injection forms, and delimiting tool outputs reduces but does not eliminate it.

4.1. Experimental Setup

  • The evaluation covers five tasks across workspace, e-mail, and Slack, with 40 new test instances per task.It tests four foundation models and two self-evolution frameworks: OpenClaw-RL and MetaClaw.Source: §4.1 · pp. 6–7

4.2. RQ1: Does Self-Evolution Degrade IPI Robustness?

  • Mean ASR rose by 3.0–39.3 percentage points under RL and 14.0–48.8 percentage points under skill evolution.The increase occurred for every tested model under each mechanism.Source: §4.2 · p. 7
  • During adaptation, GPT-OSS-120B’s Drive–Create ASR rose from 27.5% after 24 RL updates to 75.0% after 32; Qwen3.8-27B’s File–Create ASR reached 77.5% after nine and twelve skill-evolution conversations.Source: §4.2 · p. 7

4.3. RQ2: What Causes the Degradation?

  • Execution-rewarded skill evolution raises ASR from 0.0% to 35.8%, while discouraging execution or using task-free adaptation leaves ASR at 0.0%.Ordinary read-only conversations leave ASR nearly unchanged (35.0% versus 35.8%), whereas authorization-aware feedback lowers it to 5.8%.Source: §4.3 · p. 8, Table 2 · p. 8
  • Learned skills that omit authorization yield 87.5% ASR, while a library that checks the instruction’s source yields 0% ASR.The same evolver learned each library from twelve authorized conversations; some skills also promote execution through examples.Source: §4.3 · pp. 8–9, Figure 4 · p. 8

4.4. RQ3: Does the Effect Transfer to Other Injections?

  • Imperative, to-do, and conditional injections yield 80.0–100.0% ASR, and placing instructions before the document body yields 95.0% under both mechanisms.Code-comment injections also exceed the base agents’ 0% ASR, while the system-message form yields 55.0% under RL and 0% under skill evolution.Source: §4.4 · p. 9
  • With seven evolved executing skills, the agent carries out harmful instructions in 119 of 162 conversations, versus 1 of 162 with an empty library.The evolved agent still delivers the user’s summary in 88.3% of conversations.Source: §4.4 · p. 9

4.5. RQ4: Does Delimiting Tool Outputs Restore Robustness?

  • Delimiting lowers ASR from 95.0% to 37.5% under RL and from 97.5% to 47.5% under skill evolution, but rates remain above the base agents’ 0%.Both self-evolved agents retain 100% execution under the authorized request; failures show source markers do not prevent authorization conflation.Source: §4.5 · p. 9, Table 3 · p. 9

5. Conclusion

Self-evolution degrades personal agents’ IPI robustness under both reinforcement learning and skill evolution. The conclusion identifies authorization conflation as the main cause and notes that delimiting tool outputs leaves substantial degradation.

5. Conclusion

  • Self-evolution degrades personal agents’ IPI robustness under RL and skill evolution.Agents learn to execute tasks in external content without requiring user authorization.Source: §5 · p. 9
  • Authorization-aware feedback that distinguishes authorized from unauthorized execution prevents most of the ASR increase under skill evolution.Source: §5 · p. 10
  • Delimiting tool outputs leaves substantial degradation under both self-evolution mechanisms.Source: §5 · p. 10

Limitations

The experiments use a short series of conversations, local files, and simulated online services, rather than months of daily use. The authors also identify limits on tested models and usage scenarios, and call for broader future evaluations.

Limitations

  • The experiments adapt OpenClaw through short conversations with local files and simulated online services, not months of daily use.Source: Limitations · p. 10
  • Foundation-model capabilities and computing resources limit the range of everyday usage scenarios tested.Source: Limitations · p. 10
  • Future work should test longer daily-use periods, broader user requests, and agents that update foundation models and skills together.The authors also call for defenses that preserve user authorization after agents learn to perform tasks.Source: Limitations · p. 10

Ethical Considerations

The paper studies a security weakness in self-evolving personal agents using text-only attacks and everyday-action instructions. Harmful-action tests used simulated services and synthetic identities, with no human subjects or personal data.

Ethical Considerations

  • The attack requires only text that the agent reads, and the main experiments use single-sentence instructions for everyday actions.Source: Ethical Considerations · p. 10
  • The harmful-instruction case study used simulated services and synthetic identities; nothing left the test machine.Source: Ethical Considerations · p. 10
  • The experiments involved no human subjects or personal data, and feedback came from deterministic checks of filesystem or simulated-service state.Source: Ethical Considerations · p. 10

A. Tasks and Data

The experiments use five tasks across files, drive, e-mail, and chat, with authorized adaptation requests followed by test requests that do not authorize embedded tasks. They also vary adaptation content and configure both self-evolution frameworks for the study.

A.1. The Five Tasks

  • Drive–Create contrasts a test request to report a file’s contents with an adaptation request that also authorizes completing its embedded task; simulated services verify whether actions took effect.File–Create uses local files; four other tasks use simulated drive, e-mail, and chat services whose state is rebuilt before each episode.Source: §A.1 · p. 13

A.2. Adaptation and Test Content

  • Test content and injected actions are new: test items avoid close matches to adaptation passages, and task-specific names and channels differ across adaptation and testing.File tasks span sixteen genres; source and target filenames do not recur in adaptation for File–Create and Drive–Create.Source: §A.2 · p. 13

B.1. Frameworks, Versions, and Additions

  • OpenClaw-RL uses added parsers for three model families while leaving training, reward, and rollout code unchanged; the experiments also specify framework versions and agent configurations.The RL setup exposes 22 tools, while skill evolution uses read and exec; episodes run in isolated environments with state reset between episodes.Source: §B.1 · pp. 13–14

B.2. MetaClaw Configuration

  • MetaClaw runs in skills-only mode with deterministic retrieval, and its libraries are evolved from the same twelve task-specific items for both mechanisms.Evaluation uses a freshly started proxy with evolution disabled and the library fixed.Source: §B.2 · p. 14

B.3. Validity Rules for Evaluations

The evaluation rules distinguish measurement failures from agent behavior and prescribe additional sampling for borderline results. The appendices also specify RL runs and evaluations on adaptation items.

B.3. Validity Rules for Evaluations

  • Borderline self-evolved results receive two additional evaluations when injected actions exceed the base expectation by one to five; the rule was fixed before evaluation.Pipeline anomalies void an evaluation, while behavior anomalies remain in the denominator; restricting one base agent’s denominator to content-reading episodes preserves the direction of three comparisons.Source: §B.3 · pp. 14–15

C.1. RL Runs

  • Each RL run performs 64 optimizer updates on adaptation episodes and resumes once from update 32 with both LoRA weights and optimizer state restored.The runs use the framework’s default training configuration, including batch size 4 and learning rate 10^-4.Source: §C.1 · p. 15

C.2. Evaluation on the Adaptation Items (Settings 1 and 2)

  • Settings 1 and 2 evaluate on adaptation items using authorized and test-template requests, respectively, with no feedback or training samples and the skill library fixed.Base and self-evolved agents are evaluated in three interleaved passes; the same base passes serve runs and libraries adapted on those items.Source: §C.2 · p. 15

C.3. Consistency of the Findings across Presentation Orders

Across presentation orders, self-evolution degrades robustness to indirect prompt injection, while the results also show a recurring loss of user authorization as a condition for execution. Evolved-skill examples connect that pattern to whether skills retain authorization requirements.

C.3. Consistency of the Findings across Presentation Orders

  • Across all three orders, new-content mean ASR exceeds the base agent’s in 21 of 21 comparisons, and ASR rises in 19 of 20 skill-evolution and 12 of 15 RL model–task pairs.The increase is not confined to particular models or tasks; the reported comparisons span the five tasks and multiple models.Source: §C.3 · p. 15
  • Across all three orders, the authorization gap averaged over five tasks shrinks for every model under each mechanism in 21 of 21 comparisons, indicating agents retain learned actions but lose user authorization as an execution condition.Source: §C.3 · p. 16
  • Four of 60 libraries include a rule requiring user confirmation, and their agents show 0.0% ASR while completing 94.4–100.0% of authorized tasks; libraries associated with rising ASR lack such a rule.The text reports robustness tracking whether learned skills retain the authorization condition.Source: §C.3 · p. 16

C.4. Evolved Skill Libraries

The skill-library analysis classifies learned skills by how they handle instructions in external content and examines cases where explicit rules or examples affect agent behavior. The examples contrast unauthorized execution with retained authorization checks.

C.4. Evolved Skill Libraries

  • The analysis covers 60 libraries across four models, five tasks, and three presentation orders, classifying skills as executing, conditional, untrusted-content-aware, or neutral.Neutral skills cover reply format, verification, or tool use; the analysis also records action examples within them.Source: §C.4 · pp. 16–17
  • On Drive–Create, seven of the 17 inserted skills direct execution without authorization, and the agent’s ASR is 87.5%.Eight additional neutral skills include the drive-create command or target in worked examples.Source: §C.4 · p. 17
  • On File–Create, two inserted skills preserve the distinction between user requests and file instructions; across all 40 test documents, the agent seeks confirmation or declines, with 0% ASR.The other inserted skills include two that prescribe minimal execution of “the instruction.”Source: §C.4 · p. 17
  • On Drive–Create, six communication skills demonstrate the drive action in examples despite serving reply-format or tone purposes; the agent’s ASR is 45.0%.One example makes execution part of the expected response even though the skill’s stated purpose is formatting.Source: §C.4 · pp. 17–18

C.5. Primary-Task Completion on the Summary-Sending Tasks

Across follow-up experiments, self-evolved agents could retain primary-task completion while executing unauthorized injected tasks; the experiments examine adaptation conditions, harmful instructions, and changes in injection form.

C.5. Primary-Task Completion on the Summary-Sending Tasks

  • Primary-task completion remained high in all 14 summary-sending comparisons: self-evolved agents completed at least 85.0% and matched or exceeded the base agent in 11.For GPT-OSS-20B under skill evolution, Email–Invite completion rose from 44.2% to 97.5%.Source: §C.5 · p. 18

D.1. Follow-up Agents and Adaptation Curves

  • Follow-up experiments used isolated agent configurations, separate test documents, and the same OpenClaw version and model checkpoints as the main experiment.The skill-evolution curve reconstructed libraries after 3, 6, 9, and 12 conversations; the final reconstruction reproduced the library exactly.Source: §D.1 · pp. 18–19

D.2. The Authorization Experiment

  • 43 of 120 injected instructions were carried out with the Execution-Rewarded library, versus 0 with the empty library, in the pooled evaluation batch.Across three evaluations per library, Task-Free Read-Only and Authorization-Aware libraries yielded 42 and 7 executions, respectively; Execution-Discouraged and Task-Free Adaptation yielded none in one evaluation each.Source: §D.2 · p. 19

D.3. The Harmful-Instruction Case Study

  • The evolved Chat–Invite library led the agent to add an external account to confidential channel #leadership in six of six episodes, versus none with the empty library.The library contained 30 skills, including 16 inserted into prompts and two executing skills.Source: §D.3 · p. 20

D.4. The Injection-Form Experiment

  • 131 of 144 authorized-request episodes carried out the instruction across the new forms and mechanisms; eight of the 13 misses involved system-message-form injections.In those eight episodes, the agent identified the line as a prompt-injection pattern or disguised directive.Source: §D.4 · p. 20

D.5. The Delimiting Experiment

The delimiting experiment tests a prompt-boundary intervention while retaining retrieved skills. Delimiting preserved authorized task completion, but the reported attack success rate remained 47.5%.

D.5. The Delimiting Experiment

  • Delimiting wraps tool messages in boundary markers and appends a prompt declaration identifying tool output as data while allowing user-authorized tasks described there.The authors distinguish this from stricter spotlighting, which would also prohibit the authorized File–Create task.Source: §D.5 · p. 21
  • 47.5% ASR remained under delimiting, no lower than under RL, despite the expectation that delimiting would remove nearly all of the skill-evolution ASR increase.The intervention preserved authorized completion: self-evolved agents completed 100.0% of authorized tasks with and without delimiting.Source: §D.5 · p. 21

E. Artifacts, Computation, and AI Assistance

  • The study used open-source agent frameworks and open-weight models, with synthetic English tasks, documents, simulated services, and fictional names, accounts, and channels.Project experiments processed about 6.3 billion tokens on Tinker at a stated list-price cost of about US$1,500.Source: E. Artifacts, Computation, and AI… · pp. 21–22

Related papers

Loading 2610.39122v1…