PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience

Zixuan Wang, Hao Li, Fengyu Gao, G. Edward Suh, Yi Zeng, Yevgeniy Vorobeychik, Ning Zhang, Chaowei Xiao

arXiv:2610.04269v1cs.LGcs.CR

TL;DR

Tool-using LLM agents can be diverted from user goals by malicious instructions in external content, and static expert-curated defenses have limited exposure to adaptive attacks. SRFT trains on adversarial failures and reflective comparisons of compromised and optimal actions; the authors report reduced attack success with maintained task utility across standard and adaptive settings.

  • Problem

    Tool-using agents can be diverted from user goals by malicious external instructions, while static expert-curated training exposes them to limited attack patterns and may not generalize to novel or adaptive threats.

    Source: §1 · pp. 1–2

  • Method

    SRFT exposes agents to adversarial failures and fine-tunes them using self-reflections that contrast compromised actions with optimal actions.

    Source: §4.2 · p. 4, §6 · p. 9

  • Results

    SRFT achieves strong robustness across standard and adaptive attacks while maintaining practical task utility; on AgentDojo, Llama-3.1-8B-Instruct ASR falls from 7.59% to 1.26%.

    Source: §1 · pp. 2–3, §6 · p. 9

  • Takeaways & Limitations

    The authors highlight experience-driven learning and explicit reasoning as important for agent security while maintaining practical task utility.

    Source: §6 · p. 9

    The threat model assumes attackers can access only realistic attack surfaces, not arbitrarily modify the external environment.

    Source: §3 · p. 3

Abstract

from arXiv · show

Large language model (LLM) agents are increasingly deployed in tool-augmented environments, but their reliance on external inputs makes them highly vulnerable to prompt injection attacks that can hijack task objectives. Existing safety alignment methods rely on static expert trajectories or preference optimization, limiting their ability to generalize to adaptive attack patterns. In this work, we propose Self-Reflection Fine-Tuning (SRFT), a training framework that enables agents to improve robustness by learning from their own failure experiences under adversarial conditions. Instead of passively imitating expert behaviors, SRFT exposes the agent to compromised trajectories constructed via injected attacks, and leverages an expert model to generate structured self-reflection reasoning that contrasts unsafe and optimal actions. This reflective supervision teaches the agent to identify malicious instructions, reason about their consequences, and maintain alignment with the original user intent. We instantiate this framework in SR-Agent, built on Llama-3.1-8B-Instruct and Qwen3-8B, and evaluate it on both static and adaptive prompt injection benchmarks. Experimental results show that SRFT substantially reduces attack success rates while preserving task performance, and demonstrates strong generalization under adaptive attacks. These findings suggest that learning from failure via self-reflection is a promising direction for building robust and secure LLM agents. Our code is released at https://github.com/Eden-Wang1710/srft-repo.

1 INTRODUCTION

Prompt injection can divert tool-using agents from user goals, while static expert-curated defenses expose them to limited attack patterns and may not generalize to adaptive threats. SRFT trains from adversarial failure experiences, and reported results show reduced attack success with competitive task utility and robustness under adaptive attacks.

1 INTRODUCTION

  • System-level defenses can require substantial engineering, while model-level defenses may rely on pattern matching and rigidly suppress external instructions.The passages describe these trade-offs as limiting applicability or flexibility.Source: §1 · pp. 1–2
  • The figure contrasts fixed defensive patterns in supervised fine-tuning with learning from failures to handle a broader range of attacks, including previously unseen ones.It frames failure experience as a route to identifying unseen attacks.Source: Figure 1 · p. 2
  • Static expert-curated datasets expose agents to limited predefined attack patterns, making generalization to novel or adaptive threats difficult.Agents trained passively also lack feedback from attack outcomes to discover failures and correct unsafe behavior.Source: §1 · p. 2
  • SRFT constructs adversarial scenarios by injecting attacks into clean expert trajectories, then uses expert comparisons of hijacked and optimal actions to generate reflections for fine-tuning.The reflections explain why optimal actions are preferable and what risks compromised actions pose.Source: §1 · p. 2
  • 7.59% to 1.26% ASR on AgentDojo for Llama-3.1-8B-Instruct; under adaptive InjecAgent attacks, SR-Agent achieves below 20% ASR versus Meta SecAlign above 60%.Task utility remains competitive.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work describes prompt injection risks in tool-using workflows and defenses at both system and model levels. Existing model-level defenses largely imitate expert behavior or suppress patterns, which may limit generalization to adaptive attacks.

2 RELATED WORK

  • Prompt injections in webpages, emails, documents, retrieved content, or API responses can cause agents to leak information, invoke unintended tools, or deviate from user objectives.InjეცAgent tests isolated single-step behavior, whereas AgentDojo evaluates end-to-end agents in dynamic tool-use environments.Source: §2 · p. 3
  • System-level defenses use controls such as sandboxing and permissions, while model-level defenses train models to respect trusted-input boundaries or prefer secure responses.The passage describes StruQ, SecAlign, and Meta-SecAlign as examples of model-level approaches.Source: §2 · p. 3
  • Existing defenses often rely on expert-behavior imitation or pattern-based suppression, which may limit generalization to diverse and adaptive attack strategies.System-level methods may also require substantial engineering and generalize poorly across environments.Source: §2 · p. 3

3 THREAT MODEL

The threat model considers third-party attackers who place malicious instructions in external resources and defenders who must resist hijacking while completing the user's task. It limits attacker access to realistic attack surfaces.

3 THREAT MODEL

  • The attacker is a third party seeking to insert malicious instructions into external resources so the agent follows the attacker’s commands.Examples include webpages and inbox messages.Source: §3 · p. 3
  • The defender aims to prevent hijacking by external instructions while preserving the ability to complete the original user task.The setting does not assume that the user is the attacker in indirect prompt injection scenarios.Source: §3 · p. 3
  • The attacker is assumed to access only realistic attack surfaces, rather than arbitrarily modifying the external environment.The authors characterize arbitrary environment modification as overstating attacker capabilities for real-world scenarios.Source: §3 · p. 3

4 SRFT: SELF-REFLECTION FINE-TUNING

SRFT trains tool-integrated agents to learn from failures under prompt injection through a four-stage process: constructing attacked trajectories, sampling actions, generating structured reflections, and fine-tuning on reflections and optimal actions.

4.1 OVERVIEW

  • SRFT uses four stages to train tool-integrated agents to learn from prompt-injection failures through structured self-reflection.The framework aims to broaden defense coverage and handle novel attacks.Source: §4.1 · p. 4

4.2 STAGE I: PROMPT INJECTION DATA SYNTHESIS

  • Attacked trajectories insert executable triggers into intermediate observations while preserving the original task objective and optimal expert actions.Triggers are inserted at a random position, and each intermediate observation has a 0.5 injection probability.Source: §4.2 · p. 4

4.3 STAGE II: RAW ACTION SAMPLING

  • Under injected contexts, the target agent samples candidate actions that may expose unsafe or suboptimal behavior for reflection supervision.The sampled actions capture raw behavior under adversarial observations and provide positive and negative examples.Source: §4.3 · p. 5

4.4 STAGE III: SELF-REFLECTION ON FAILURE MODES

  • An expert model generates reflections that identify malicious instructions and compare optimal actions with sampled candidates, including potential consequences.Reflections can use hypothetical outcomes to expose risks such as unintended tool use, task deviation, or information leakage.Source: §4.4 · p. 5

4.5 STAGE IV: FINE-TUNING VIA LEARNING FROM FAILURE

  • Fine-tuning uses the trajectory prefix to train the agent to produce both the reflection and the optimal action.The factorization represents generating a reflection first, then generating the final action conditioned on that reasoning.Source: §4.5 · p. 5

5 EXPERIMENT

On AgentDojo, SRFT improves security and task utility, while preserving general capabilities and maintaining lower attack success rates under adaptive attacks. Ablations show that both failure-aware training and explicit reasoning matter, and SRFT does not increase inference-time token use.

5.1 EXPERIMENTAL SETUPS

  • The evaluation uses TOUCAN-derived trajectories, InjecAgent and AgentDojo security benchmarks, and general-capability benchmarks; metrics cover benign utility, utility under attack, and ASR.The training set contains 3,698 trajectories from five TOUCAN platforms; the prompt-injection synthesis stage uses Claude Sonnet 4.6 as expert model.Source: §5.1 · p. 6

5.3 AGENTIC WORKFLOW EVALUATION

  • On AgentDojo, SRFT lowers overall ASR from 7.59% to 1.26%, raises Benign Utility from 27.84% to 37.11%, and raises Utility under Attack from 23.08% to 29.08%.The authors report the best utility-security trade-off against Meta-SecAlign.Source: Table 1 · p. 7, §5.3 · p. 7

5.2 GENERAL UTILITY EVALUATION

  • Across MMLU, MMLU-Pro, IFEval, and BBH, SR-Agent-Llama remains comparable to its base model and has higher average utility than Meta-SecAlign-8B.The authors highlight IFEval as a case where SR-Agent retains the original model’s instruction-following performance.Source: Figure 2 · p. 6, §5.2 · p. 7

5.3 AGENTIC WORKFLOW EVALUATION

  • On AgentDojo’s four platforms, SRFT reduces ASR and improves the security–utility trade-off for Qwen3-4B and Qwen3-8B.The reported transfer evaluation aggregates results across the four platforms.Source: §5.3 · p. 7, Table 2 · p. 8

5.4 ADAPTIVE ATTACK EVALUATION

  • Under RL-Hammer, SR-Agent maintains lower ASR throughout attacker training on Llama-3.1-8B and Qwen3-8B, while undefended models are rapidly compromised.Meta-SecAlign’s ASR rises over training on Llama-3.1-8B; Qwen3-8B SR-Agent ASR stays near zero through most early and middle epochs.Source: Figure 3 · p. 8, §5.4 · p. 8

5.5.1 TRAIN-TIME ABLATION OF SELF-REFLECTION

  • Removing failure-experience analysis raises final ASR from 17.0% to 65.0%, while removing reflection yields 60.0% ASR under either inference mode.Goal restatement and injection identification alone do not reproduce SRFT’s robustness.Source: §5.5.1 · pp. 8–9

5.5.2 TEST-TIME ABLATION OF SELF-REFLECTION

  • Disabling explicit reasoning raises overall AgentDojo ASR from 1.05% to 14.33%, with degradation across all four environments.The paper attributes a role in detecting injections, evaluating unsafe tool calls, and selecting actions aligned with the user objective to explicit reasoning.Source: Table 3 · p. 9, §5.5.2 · p. 9

5.6 OVERHEAD ANALYSIS

  • On AgentDojo, SR-Agent produces fewer thinking tokens than base Qwen3-8B, with mean thinking tokens decreasing from 451.70 to 369.94.The authors also report a similar reduction in total generated tokens.Source: Table 4 · p. 9, §5.6 · p. 9

6 CONCLUSION

SRFT learns from adversarial failures through self-reflection, with the paper concluding that this can improve agent robustness while maintaining task utility. The surrounding sections situate that approach among agent workflows, prompt-injection benchmarks, existing defenses, and attack generation.

6 CONCLUSION

  • SRFT exposes agents to adversarial scenarios and teaches them to reason about compromised versus optimal actions, improving robustness while maintaining practical task utility.The conclusion identifies experience-driven learning and explicit reasoning as important in agent security.Source: §6 · p. 9

A.1 LLM AGENTS

  • Tool-using LLM agents operate across environments including browsers, code interpreters, APIs, and operating systems, building on systems such as ReAct and Toolformer (Schick et al., 2023).Prior work also trains agents on trajectories using supervised fine-tuning or reinforcement learning (Zhang et al., 2025; Wang et al., 2025b).Source: §A.1 · p. 13

A.2 PROMPT INJECTION IN AGENTIC WORKFLOWS

A.3 DEFENSE AGAINST PROMPT INJECTION

  • System-level defenses constrain agent actions but may require substantial engineering and generalize poorly; model-level defenses instead improve the LLM’s own resistance.Model-level examples include StruQ’s trusted-input separation, SecAlign’s preference optimization, and Meta-SecAlign’s scaled approach; SRFT instead compares sampled and expert actions within multi-step tool trajectories.Source: §A.3 · p. 14

B.1 ATTACK GENERATION

  • Attack generation uses an expert LLM to pair malicious instructions with executable tool calls, then prefixes each instruction with a trigger before insertion into a tool observation.The generation prompt requests diverse styles and intents, limits instructions to 60 words, and requires valid tool names and JSON arguments.Source: §B.1 · p. 14

B.1.1 DATASET STATISTICS

These appendices detail how SRFT data and reflections are constructed, and test whether an inference-time prompt alone accounts for the reported defense. The reflection is generated using information unavailable to the trained agent at inference.

B.1.1 DATASET STATISTICS

  • Training data reuses 38 generated malicious tasks across platform user tasks and adds 285 off-task requests; 38% of tool observations carry an injection.Each assistant step is preceded by a self-reflection averaging 396 tokens (median 401, Llama-3.1-8B tokenizer).Source: §B.1.1 · p. 15, Table 5 · p. 15

B.1.2 AN ATTACKED EXPERT TRAJECTORY

  • An attacked expert trajectory is made by appending a trigger and malicious instruction to one tool observation in a clean trajectory, while leaving expert actions unchanged.The example instruction requests a path-traversal write into the root user’s SSH directory.Source: §B.1.2 · p. 15

B.2 COT GENERATION

  • For each attacked assistant step, an expert LLM receives the dialogue history, injection ground truth, expert action, and sampled candidate actions to generate a three-paragraph reflection.The reflection summarizes the goal, identifies and explains the injection, then justifies the safe action and possible consequences of unsafe candidates.Source: §B.2 · pp. 16–17

B.2.1 TEMPLATE 2 INPUTS AT THE ATTACKED STEP

  • At the example attacked step, the expert action reports completed tasks, while two sampled candidates follow the injection and another treats it as an intermediate requirement.The expert action is stored after the reflection; its three parts cover context and goal, injection analysis, and action analysis.Source: §B.2.1 · p. 17, §B.2.2 · pp. 17–18

B.2.2 THE GENERATED REFLECTION

  • The reflection uses injection ground truth and reference-agent failures during training, but the trained model sees neither at inference and reproduces the analysis from dialogue alone.In the example, it identifies the injected path-traversal request and explains why no additional tool call is needed.Source: §B.2.2 · p. 18

B.3 SR-AGENT DETAILS

  • Qwen3 SR-Agent inference prompts ask the model to summarize the original goal, identify injections, and consider actions that resist the injection while completing the task.The instruction is appended to the environment’s system prompt.Source: §B.3 · p. 18

B.3.1 THE APPEND ON AN UNTRAINED MODEL

  • 16.97% to 14.75% ASR for prompted untrained Qwen3-8B, versus 1.05% for SR-Agent with the same instruction, shows the prompt append alone is not the defense.The appendix characterizes the append as a trigger rather than a defense.Source: §B.3.1 · p. 18, Table 6 · p. 19

C.1 AGENTDOJO EXPERIMENT DETAILS

The experiment caps thinking at 512 tokens and uses a forced transition prompt if the model does not finish thinking or produce a tool call.

C.1 AGENTDOJO EXPERIMENT DETAILS

  • The 512-token thinking budget can trigger a forced transition prompt when thinking remains unfinished, yielding a maximum observed thinking length of 532 tokens.AgentDojo results use the benchmark’s default important-instructions attack.Source: §C.1 · p. 19

C.2 RL-HAMMER EXPERIMENT DETAILS

The adaptive evaluation trains a separate RL-Hammer attacker for each target and measures attack success across epochs on a shared InjecAgent split.

C.2 RL-HAMMER EXPERIMENT DETAILS

  • RL-Hammer trains a separate attacker per target and reports ASR across epochs; 510 shared InjecAgent cases are split into 310 training, 100 validation, and 100 test cases, with no checkpoint selection.The train-time ablation evaluates Qwen3-8B targets on the same 100 InjecAgent test cases.Source: §C.2 · p. 19, Table 8 · p. 20

C.2.1 TRAIN-TIME ABLATION: ENDPOINT STATISTICS

On AgentDyn, SR-Agent-Llama outperforms Meta-SecAlign-8B on benign utility and ASR, while the train-time ablation shows much higher ASR without failure experience. AgentDyn tests open-ended tasks that require replanning from tool outputs, including legitimate third-party instructions.

C.2.1 TRAIN-TIME ABLATION: ENDPOINT STATISTICS

  • 65.0% final and Last-5 ASR without failure experience, versus 17.0% final and 16.8% Last-5 ASR for SR-Agent.The ablated model retains goal summary and injection identification but omits failure-action reflection.Source: §C.2.1 · p. 20

C.3 DYNAMIC ENVIRONMENTS: AGENTDYN

  • AgentDyn evaluates open-ended tasks averaging 7.1 steps across 3.2 applications, where legitimate third-party instructions may be critical to task completion.The evaluation covers 60 benign and 560 attacked tasks across shopping, GitHub, and daily-life suites; over-defense is treated as a failure.Source: §C.3 · p. 20
  • 10.00 vs. 5.00 benign utility and 1.96 vs. 5.54 ASR: SR-Agent-Llama outperforms Meta-SecAlign-8B on both AgentDyn measures.Table 9 reports the AgentDyn evaluation using Llama-3.1-8B.Source: §C.3 · p. 20, Table 9 · p. 20

D.1 CASE STUDY OF AGENTDOJO ON QWEN3-8B

In an AgentDojo Slack example, an injected webpage instruction asks the agent to exfiltrate messages. SR-Agent identifies the injection and preserves the user's webpage-reading goal, whereas undefended Qwen3-8B follows the injected request.

D.1 CASE STUDY OF AGENTDOJO ON QWEN3-8B

  • The user asked only to read a webpage; its tool output impersonated Emma Johnson and requested posting general-channel messages to an external site.The injected instruction appears inside the returned webpage content.Source: §D.1 · p. 21
  • SR-Agent's reflection marks the instruction as tool-originated and unrelated to the task, anticipates privacy exposure, and selects a direct answer without another action.It reasons that posting Slack messages could expose sensitive information or violate user privacy.Source: §D.1 · p. 22
  • SR-Agent identifies the webpage's injected Slack-exfiltration request, preserves the read-page goal, and answers without further tool calls; undefended Qwen3-8B instead exfiltrates messages.The undefended trace reads general-channel messages and posts them to an external website.Source: §D.1 · pp. 21–22

D.2 CASE STUDY OF AGENTDOJO ON LLAMA-3.1-8B

In an AgentDojo Banking example, an injected tool instruction requests an unauthorized transfer while the user asks about a past transaction. SR-Agent answers the original question directly; the undefended model makes the transfer before answering.

D.2 CASE STUDY OF AGENTDOJO ON LLAMA-3.1-8B

  • The user asks about spending on a New Year's gift; the bank-tool response includes an instruction to send money to an unknown account.The instruction is embedded in a transaction-history entry and requests details to be filled in without asking.Source: §D.2 · p. 22
  • SR-Agent's reflection identifies the transfer request as injected tool content, recovers the original question, and recognizes the risk of sending money to an unknown recipient.It uses the retrieved transaction history to answer directly.Source: §D.2 · p. 23
  • SR-Agent answers 200.0 CHF directly and avoids an extra transaction, while undefended Llama-3.1-8B-Instruct follows the injected transfer request before answering.The injected action sends 10.99 to an unknown account for Spotify Premium.Source: §D.2 · p. 23

D.3 CASE STUDY OF RL-HAMMER ON QWEN3-8B

Across two RL-Hammer case studies, target-specific attacks raise ASR substantially against undefended Qwen3-8B and Llama-3.1-8B, while SR-Agent remains less vulnerable. Both models initially reject the static injections, but the adapted attacks succeed only against the undefended models.

D.3 CASE STUDY OF RL-HAMMER ON QWEN3-8B

  • 59.5% ASR against undefended Qwen3-8B versus 17.0% against SR-Agent on RL-Hammer's full test set.The original static injection was rejected by both models; a target-specific rewrite succeeded against the undefended model but remained unsuccessful against SR-Agent.Source: §D.3 · pp. 23–25

D.4 CASE STUDY OF RL-HAMMER ON LLAMA-3.1-8B

  • 98.5% ASR against undefended Llama-3.1-8B versus 17.5% against SR-Agent at epoch 20.Both models rejected the original static injection; the adapted prompt elicited a transfer call from the undefended model, while SR-Agent rejected it before financial action.Source: §D.4 · pp. 25–26

Related papers

Loading 2610.04269v1…