PaperlayerView exact revision on arXiv

Source-linked AI summary

RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents

Mohamed Dhouib, Clement Elliker, Alexi Canesse, Maël Jenny, Lucas-Andrei Thil, Mahammed El-Sharkawy, Sonia Vanier, Elie Bursztein

arXiv:2610.06401v1cs.CRcs.AIcs.CLcs.LG

TL;DR

Tool-using agents must process untrusted content that can carry prompt injections, while still using legitimate tool guidance to complete tasks. RAISED combines model-generated scenarios and self-distillation to preserve clean behavior on injected contexts. Across benchmarks it matches or exceeds SecAlign++ robustness while better preserving benign utility.

  • Problem

    Indirect prompt injection can hijack tool-using agents through untrusted external content, while defenses must still allow them to act on task-relevant tool guidance.

    Source: §1 · p. 1

  • Method

    RAISED has a frozen model generate tool-use scenarios and injections, then trains a student to match clean-context behavior on clean and injected contexts.

    Source: §1 · p. 2, §5.1 · p. 6

  • Results

    Across benchmarks, RAISED matches or exceeds SecAlign++ robustness while better preserving benign utility; on AgentDojo with Gemma, ASR falls from 39.5% to 1.0%.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    Because RAISED preserves the base model’s behavior rather than improving it, it also inherits the base model’s errors.

    Source: §8 · p. 10

    The threat model considers static attacks whose adversarial strings are chosen independently of the defended model.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure mode of these defenses: On benign tool-use tasks, the model refrains from a step needed to finish an authorized task, particularly when that step is indicated by a tool output. To address these limitations, we introduce RAISED (Robust Attack Invariance through Self-Distillation), a training framework that combines self-generation and self-distillation. The model first generates its own tool-use scenarios, with an emphasis on cases where task completion requires acting on legitimate guidance from tool outputs. Then, through self-distillation, the student is trained to match the teacher's clean-context behavior on both clean and injected variants of the same trajectory. RAISED substantially reduces the attack success rate of prompt injections in tool responses while, unlike prior training-based defenses, preserving utility on both agentic and general-purpose benchmarks.

1 INTRODUCTION

Tool-using agents must resist instructions hidden in untrusted tool content while still acting on legitimate guidance there. RAISED combines self-generated scenarios with self-distillation to preserve clean behavior on injected contexts, reducing attacks while retaining utility.

1 INTRODUCTION

  • Indirect prompt injection occurs when instructions hidden in untrusted external content hijack an agent from its assigned task, creating a trade-off between resisting attacks and using task-relevant tool guidance.Agents encounter untrusted content through tools connected to emails, documents, and other services.Source: §1 · p. 1
  • Training-based defenses can leave authorized tasks unfinished when the needed step depends on tool-output guidance, because they provide weaker incentives to act on legitimate guidance than to avoid injected directions.The model may use tool outputs as information while avoiding action on their guidance.Source: §1 · p. 2
  • RAISED pairs self-generation of tool-use scenarios and injections with self-distillation that matches the teacher’s clean-context next-token distribution on clean and injected contexts.Scenarios emphasize tasks requiring action on legitimate guidance from tool outputs.Source: §1 · p. 2
  • On AgentDojo with Gemma, RAISED reduces ASR from 39.5% to 1.0% and utility under attack rises from 64.0% to 80.5%, with a 1-point benign-utility loss.The reported robustness matches SecAlign++ without its 7-point drop.Source: §1 · p. 2
  • The paper reports evaluations across three open-weight models and general-knowledge, agentic-capability, and prompt-injection benchmarks, with reduced attack success and preserved benign utility.It also reports improved utility under attack.Source: §1 · p. 2

2 RELATED WORK

Related work spans prompt-injection threats and defenses, consistency training, self-distillation, and synthetic tool-use data. RAISED combines these lines in multi-turn tool-use trajectories, using clean-context predictive distributions and model-generated scenarios and attacks.

Threats and benchmarks

  • Indirect prompt injection extends direct prompt injection to agents retrieving untrusted content, with attacks and benchmarks targeting instructions planted in tool outputs.The cited threat literature includes Greshake et al. (2023), and tool-output benchmarks include Zhan et al. (2024) and Debenedetti et al. (2024).Source: §2 · p. 2

Defenses

  • Existing defenses include prompt-level prevention, system-level control-flow policies, input detection, and model-level training to resist injected instructions.Examples include spotlighting, prompt sandwiching, CaMeL, PromptGuard, instruction hierarchies, and preference optimization.Source: §2 · pp. 2–3
  • Unlike SecAlign++, which uses base-model responses as sequence-level preference targets, RAISED directly targets the model’s clean-context predictive distribution.RAISED uses that distribution as its training target on injected contexts.Source: §2 · p. 3

Consistency training and self-distillation

  • Prior consistency training and self-distillation address single-turn prompts or privileged contexts, whereas RAISED applies consistency training to multi-turn tool-use trajectories with injections inside tool observations.It distills off-policy from stored top-K teacher distributions along clean continuations and includes cases requiring action on legitimate tool guidance.Source: §2 · p. 3

Self-generated tool-use and adversarial data

  • Prior work synthesizes instruction and tool-use data, while RAISED combines model-based generation with execution-based validation and has the defended model write its own injections after solving each scenario.Earlier injection-training pipelines cited here rely on templated attacks.Source: §2 · p. 3

3 PROBLEM FORMULATION AND THREAT MODEL

The formulation models an agent’s tool-mediated episode, the adversary’s control over tool observations, and the desired invariance between clean and injected contexts. It also defines utility and attack metrics and warns that low attack success alone can mask failed task completion.

3.1 AGENT SETTING

  • Before each turn, context c_j renders the system prompt, tool menu, user request, and the actions and observations so far.An episode begins in state s0 and ends in sJ after tool calls or a final answer.Source: §3.1 · p. 4
  • An episode succeeds when its request-specific check G_u(s_J) holds, meaning the request was fulfilled.Tool observations are the only context component coming from outside the agent’s trust boundary.Source: §3.1 · p. 4

3.2 THREAT MODEL

  • The adversary controls one or more tool outputs and inserts an adversarial string at a chosen position inside an observation, producing an injected context that differs from the clean context only there.The adversary’s goal may be an unintended tool call, disclosure, or harmful action.Source: §3.2 · p. 4
  • The adversary does not control the system prompt, user request, model, or decoding procedure, and cannot access gradients or logits.When targeting a tool call, the adversary is assumed to target a tool available to the agent.Source: §3.2 · p. 4
  • The threat model considers static attacks whose adversarial strings are chosen independently of the defended model.The passage names InjecAgent, AgentDojo, and AgentDyn as examples.Source: §3.2 · p. 4
  • Benign utility U measures successful clean episodes, utility under attack U_atk measures successful injected episodes, and ASR measures injected episodes meeting the adversary’s goal.The paper reports whichever of these metrics each benchmark supports.Source: §3.2 · p. 4
  • The invariance goal equates the defended model’s behavior on injected context with the base model’s behavior on clean context, rather than asking it to ignore tool outputs.Utility and ASR must be considered together: early termination or fewer tool calls can lower ASR while leaving the task unfinished.Source: §3.2 · p. 4

4 THE UTILITY COST OF TRAINING-BASED DEFENSES

Training-based defenses can degrade benign utility through output-distribution drift and reluctance to act on legitimate tool-output guidance. The section measures drift and illustrates how defensive training can leave an authorized tool-use task unfinished.

4 THE UTILITY COST OF TRAINING-BASED DEFENSES

  • The section identifies two utility costs of training-based defenses: shifted output distributions on benign inputs and unfinished benign tool-use tasks when completion depends on tool guidance.The drift analysis shows altered benign behavior but does not establish that the drift directly causes downstream utility loss.Source: §4 · p. 4, §4.1 · p. 5, §4.2 · p. 5

4.1 COST 1: OUTPUT-DISTRIBUTION DRIFT IN BENIGN SETTINGS

  • The drift analysis compares base and trained next-token distributions at each answer position using forward and reverse KL divergence and top-1 agreement.It uses a held-out benign set of 500 standalone questions and teacher-forces the base model’s answers through each checkpoint.Source: §4.1 · pp. 4–5
  • 0.0010 forward and reverse KL, respectively 44× and 55× lower than SFT and 29× and 30× lower than DPO, with 0.993 top-1 agreement, are reported for self-distillation.The controlled comparison uses 5K injection-augmented trajectories, rank-16 LoRA, and 313 training steps; drift increases with further SFT and DPO training while self-distillation remains close to the base distribution.Source: Figure 2 · p. 5, §4.1 · p. 5

4.2 COST 2: BEHAVIORAL DRIFT IN BENIGN TOOL USE

  • A defended model may stop before an authorized password change is complete, asking the user for a verification code that the base model retrieved through tools.The described pattern is using tool outputs as information while avoiding action on their legitimate guidance; RAISED completes corresponding tasks on which SFT and SecAlign++ fail.Source: §4.2 · p. 5

5 METHOD: RAISED

RAISED combines self-generation and self-distillation: generated scenarios target failures to act on legitimate tool guidance, while distillation limits output-distribution drift.

5 METHOD: RAISED

  • RAISED combines self-generation to target behavioral failures with self-distillation to limit output-distribution drift.Together, these stages aim to preserve behavior on clean contexts while resisting injections.Source: §5 · p. 5

5.1 STAGE 1: SELF-GENERATION

  • Stage 1 generates executable tool-use scenarios where task completion depends on legitimate guidance surfaced in tool outputs.The frozen model solves and verifies the scenarios, then generates injections for sampled tool outputs; the attacker is restricted to the untrusted channel.Source: §5.1 · p. 6

5.2 STAGE 2: INJECTION-AUGMENTED SELF-DISTILLATION

  • Stage 2 trains the student to match frozen-model next-token distributions on both clean histories and injected variants.The injected history materializes stored attacks in selected tool outputs, while the teacher target remains unchanged; this encourages injection resistance while preserving legitimate guidance.Source: §5.2 · pp. 6–7
  • For efficiency, the student’s KL-divergence objective uses the teacher’s top-K tokens plus one bucket for the remaining probability mass.The residual bucket generally contains little probability mass, on the order of 10^-5.Source: §5.2 · p. 7

6 EXPERIMENTAL SETUP

The evaluation compares two primary tool-capable models and a range of defense families across prompt-injection, benign agentic-capability, and general-knowledge benchmarks.

6.1 MODELS AND EVALUATED DEFENSES

  • The primary evaluation uses Gemma-4-12B-it and Nemotron-3-Nano-30B-A3B, comparing undefended, prompting, filtering, and alignment-based defenses.Trained defenses are evaluated over three seeds, with mean and standard deviation reported.Source: §6.1 · p. 7

6.2 EVALUATION BENCHMARKS

  • Evaluation covers prompt-injection robustness, benign tool-use capabilities, and general knowledge and reasoning.Prompt-injection benchmarks measure benign utility (U), utility under attack (Uatk), and attack success rate (ASR) where applicable; Table 1 reports these measures for Gemma and Nemotron.Source: §6.2 · pp. 7–8, Table 1 · p. 8

7 RESULTS AND ANALYSIS

Across Gemma and Nemotron, RAISED reduces prompt-injection success while preserving benign utility. Its results also show preserved benign agentic and general capabilities, with ablations favoring self-distillation and self-generated trajectories.

7.1 RAISED IMPROVES THE SECURITY-UTILITY TRADE-OFF

  • Gemma AgentDojo ASR falls from 39.5% to 1.0 ± 0.3%, while Nemotron AgentDojo ASR falls to 0.5 ± 0.1% versus 4.0 ± 0.7% for SecAlign++.On Gemma, clean utility is 84.5% versus 83.5 ± 0.8% for Vanilla; on Nemotron, clean utility remains close to Vanilla.Source: §7.1 · pp. 8–9
  • On Gemma AgentDojo, PromptGuard2 lowers ASR from 39.5% to 16.1% but Uatk from 64.0% to 42.9%, whereas RAISED reaches 80.5 ± 0.5% Uatk.For SecAlign++, the reported checkpoint is the 313-step version, selected to avoid substantial utility degradation; longer training can further reduce ASR at the cost of broader capability degradation.Source: §7.1 · p. 9

7.2 RAISED PRESERVES BENIGN AGENTIC CAPABILITIES

  • On Gemma, RAISED preserves performance across five benign tool-use benchmarks, including a WorkBench increase from 55.9 to 61.5 ± 0.7.Nemotron improves on BFCL from 30.0 to 32.8 ± 0.2, stays close to Vanilla on WorkBench and MMT-Bench, and has moderate decreases on ToolTalk and AsyncTool; the table covers five benchmarks without injection.Source: Table 2 · p. 8, §7.2 · p. 9

7.3 GENERAL CAPABILITIES REMAIN CLOSE TO THE BASE MODEL

  • Across Gemma and Nemotron, RAISED preserves general capabilities on all ten benchmarks, with scores close to Vanilla and no systematic degradation.Table 3 covers non-agentic knowledge, reasoning, multilingual understanding, instruction following, mathematics, and code generation.Source: §7.3 · p. 9, Table 3 · p. 9
  • On Qwen3.8-27B, AgentDojo ASR falls from 5.8% to 0.9 ± 0.4% and AgentDyn ASR from 4.2% to 0.1 ± 0.2%, while benign capabilities are preserved.Qwen3.8-27B already has low prompt-injection ASR before RAISED.Source: §7.3 · p. 9

7.4 ABLATION STUDY

  • At similar robustness, self-distillation preserves more agentic utility than SFT and SecAlign++’s DPO objective, while self-generated trajectories provide a further utility gain.The ablation controls training data and inputs while varying the objective, then fixes the objective while varying the data source.Source: §7.4 · p. 10

8 CONCLUSION

RAISED trains a model to match its clean-context behavior on injected tool-use contexts, and the authors report improved robustness with better-preserved benign utility than SecAlign++. The paper frames this as a defense-in-depth measure, not a replacement for established safeguards.

8 CONCLUSION

  • RAISED uses self-generated tool-use scenarios and injections, then trains the model to match clean-context next-token distributions on injected contexts.The conclusion says this approach matches or exceeds SecAlign++’s robustness while better preserving benign utility and inducing less distribution drift than SFT or DPO.Source: §8 · p. 10
  • RAISED preserves rather than improves the base model’s behavior, so it also inherits the base model’s errors.The authors point to this and other limitations in Section G.Source: §8 · p. 10
  • The authors describe prompt-injection techniques as dual-use and say RAISED is not a substitute for access control, sandboxing, capability restriction, or least-privilege tool design.Their attacks are evaluated in controlled benchmark environments.Source: ETHICS STATEMENT · p. 10

A FORMAL SETTING AND NOTATION

  • A scenario comprises a state space, initial state, tool menu, user request, and success predicate.For self-generated scenarios, the initial state is a JSON object, tools are deterministic Python functions, and the success predicate is a Python predicate.Source: A FORMAL SETTING AND NOTATION · p. 16
  • An episode alternates tool calls and observations or ends with a final answer, and succeeds when its final state satisfies the task predicate.The context before each turn includes the system prompt, tools, request, and prior actions and observations.Source: A FORMAL SETTING AND NOTATION · p. 16
  • The formal setting places injected adversarial text inside tool observations and compares the defended model on injected context with the base model on clean context.The stated metrics are clean utility, utility under attack, and attack success rate.Source: A FORMAL SETTING AND NOTATION · p. 16

B.1 CONTROLLED OUTPUT-DISTRIBUTION DRIFT ANALYSIS

The controlled analysis finds that SFT and DPO drift farther from the base model’s output distribution with longer training, while self-distillation remains close. On benign tool-use tasks, defenses can also fail to act on legitimate guidance from tool outputs, leaving authorized tasks unfinished.

B.1 CONTROLLED OUTPUT-DISTRIBUTION DRIFT ANALYSIS

  • The controlled comparison holds data and fine-tuning setup fixed across SFT, DPO, and self-distillation, using 5K injection-augmented trajectories and rank-16 LoRA for 313 steps.The objectives differ in supervision: fixed targets, preferred-versus-rejected responses, or the frozen base model’s next-token distribution.Source: §B.1 · p. 16
  • SFT maximizes fixed-target likelihood without preserving probabilities assigned to alternative tokens, while DPO constrains relative sequence preference rather than directly matching next-token distributions.For SecAlign-style training, even clean-input preferred responses are evaluated under injected contexts, so the clean-context distribution is not directly constrained.Source: §B.1 · p. 17
  • Increasing DPO’s preference margin does not guarantee that the preferred response’s likelihood is preserved, a mechanism used to explain possible benign-behavior drift.The controlled analysis measures drift in the prompt-injection setting; likelihood displacement was previously studied in broader preference-learning and safety-alignment settings [Razin et al. (2025)].Source: §B.1 · pp. 17–18
  • RAISED matches the base model’s predictive distribution on clean contexts and uses that target on injected contexts to recover the clean distribution despite added injections.The top-K plus residual approximation makes the distributional constraint practical.Source: §B.1 · p. 18
  • With 10K examples, SFT forward and reverse KL reach 0.0609 and 0.0773, DPO reaches 0.0370 and 0.0402, and self-distillation remains at 0.0016 and 0.0017.At 5K examples, SFT KLs are 0.044 and 0.055, while DPO KLs are 0.029 and 0.030.Source: §B.1 · p. 18

B.2 BEHAVIORAL DRIFT IN BENIGN TOOL USE

  • On Gemma’s seven benign tasks, SFT leaves all unfinished, while the base model and RAISED complete every task; SecAlign++ completes some but not all.The examples include benchmark and designed cases involving tool-supplied guidance.Source: §B.2 · p. 18, Table 5 · p. 19
  • On four designed Nemotron tasks, SFT leaves all unfinished, while the base model and RAISED complete all four; SecAlign++ leaves two unfinished.The two SecAlign++ failures recur across three seeds and repeated runs.Source: §B.2 · p. 18
  • Across both models, defended agents obtain relevant tool-output information but may fail to act on it when an authorized follow-up step is required.SecAlign++ successes show this behavior is not universal across all forms of tool-supplied guidance; RAISED preserves the corresponding base-model behavior.Source: §B.2 · p. 18

C METHODS DETAILS

This section details the defenses, RAISED’s self-generated training-data pipeline, evaluation protocol, model coverage, and measures of general and agentic capabilities. It also reports a Qwen stress test in which RAISED further reduces attack success while remaining close to Vanilla on benign capabilities, with internal-state measurements consistent with reduced sensitivity to injected content and smaller changes on clean contexts.

C.1 PROMPT SANDWICHING

  • Prompt sandwiching repeats the user’s task after every tool response, placing the trusted task before the next generation without fine-tuning or an auxiliary model.The implementation follows SecAlign++ (Chen et al., 2026) and Learn Prompting (2024); repeating the task increases context length.Source: §C.1 · pp. 19–20

C.2 SPOTLIGHTING

  • Spotlighting wraps each tool response in « and » and instructs the model not to follow directives inside them, without fine-tuning or an auxiliary model.This is the delimiting variant, following Hines et al. (2024) and its AgentDojo implementation (Debenedetti et al., 2024).Source: §C.2 · p. 20

C.3 PROMPTGUARD2

  • Prompt Guard 2 classifies each tool response independently, replacing flagged responses in full with an omission notice and requiring an auxiliary classifier after each tool call.Following AgentDyn (Li et al., 2026), responses with safety scores below 0.5 are flagged; Prompt Guard 2 is an 86M-parameter classifier (Chennabasappa et al., 2025).Source: §C.3 · p. 21

C.4 SECALIGN++

  • SecAlign++ trains with DPO on preference pairs from Cleaned-Alpaca, adapting its learned untrusted-content boundary to tool responses at inference.Its recipe uses a dedicated untrusted-content role, randomized injection placement, and clean-only undefended responses as preferred targets; the training pairs contain no tool calls or responses. The reported main-comparison checkpoint uses one epoch over 5,008 examples, with three training seeds.Source: §C.4 · pp. 21–22

C.5.1 SELF-GENERATION DETAILS

  • RAISED’s self-generation builds and verifies tool-use scenarios before adding injections, then uses the frozen model’s clean-trajectory outputs as teacher targets.Scenarios emphasize cases where legitimate tool guidance determines a later authorized action, alongside broader tool-use structures. Each model family retains 10,000 verified trajectories; injected contexts preserve the rest of the trajectory, and attacker tools appear in both clean and injected contexts.Source: §C.5.1 · pp. 22, 24

D.1 EVALUATION PROTOCOL

  • SecAlign++ and RAISED are evaluated over three training seeds, while methods without stochastic training runs are evaluated once.Source: §D.1 · p. 25

D.2 ARCHITECTURAL DIVERSITY.

  • The evaluation spans Gemma, Nemotron, and Qwen models with distinct attention, state-space, and mixture-of-experts architectures.The selected configurations include sliding-window and full attention, attention with Mamba-style state-space layers and MoE blocks, and interleaved linear and full attention, respectively.Source: §D.2 · p. 25

D.3 GENERAL-KNOWLEDGE EVALUATION; D.4 AGENTIC-CAPABILITY EVALUATION

  • General-knowledge evaluations measure broad knowledge and problem solving, while agentic evaluations measure tool use and task completion across interactive benchmarks.General evaluations include multiple-choice knowledge and reasoning, instruction following, mathematics, and code generation; agentic evaluation reports micro-averaged success across elementary units, including ToolTalk conversation success and BFCL multi-turn accuracy.Source: §D.3 · p. 25, §D.4 · p. 26

D.4 AGENTIC-CAPABILITY EVALUATION

  • WorkBench measures task-completion accuracy, Multi-Mission Tool Bench parameter-inclusive mission accuracy, and AsyncTool final subtask accuracy for benign tool use.Success is defined by each benchmark’s task or environment-state criteria; tasks range from workplace tool use to evolving missions and asynchronous concurrent tasks.Source: §D.4 · pp. 26–27

D.5 AGENTIC PROMPT-INJECTION EVALUATION

  • The prompt-injection evaluation covers InjecAgent, AgentDojo, and AgentDyn, measuring attack success alongside task utility where defined.AgentDyn includes legitimate third-party instructions, while aggregation follows benchmark rules, including an unweighted mean of AgentDyn’s three suite-level scores.Source: §D.5 · p. 27

D.6 QWEN: ADDITIONAL LOW-ASR STRESS TEST

  • RAISED further reduces attack success on AgentDojo and AgentDyn despite Qwen Vanilla’s already-low ASR.The additional robustness is not restricted to models that begin highly vulnerable to prompt injection.Source: §D.6 · p. 28, Table 8 · p. 28
  • On Qwen, RAISED remains close to Vanilla across five benign agentic benchmarks and ten general-capability benchmarks.This preserves the utility pattern while adding robustness in the low-ASR regime.Source: §D.6 · p. 28, Table 9 · p. 28, Table 10 · p. 28

D.7 LOOKING INSIDE THE MODEL

  • At block 31, injection-induced relative L2 change is 0.45 for the base model versus 0.23 for RAISED; RAISED starts the injected tool call in 12.5% of checked pairs versus 62.5% for the base model.The measurements use 88 clean/injected AgentDojo context pairs, with the behavioral check on 72 pairs.Source: §D.7 · p. 29, Table 11 · p. 29, Figure 7 · p. 29
  • At block 47, RAISED’s clean-context distance from the base model is 0.12 versus 0.19 for SecAlign++, consistent with limiting changes on clean inputs.Across models, injection effects are similar in early blocks and peak around block 31.Source: §D.7 · p. 29, Figure 7 · p. 29

D.8 SECALIGN++ TRAINING-STEP TRADE-OFF

Longer SecAlign++ training improves prompt-injection robustness but can amplify capability degradation, whereas RAISED remains closer to the original checkpoint. The trade-off means attack success alone can conceal losses elsewhere.

D.8 SECALIGN++ TRAINING-STEP TRADE-OFF

  • The main comparison uses the 313-step SecAlign++ checkpoint; longer training’s qualitative trade-off also appears on Gemma and Qwen, with different magnitudes and affected capabilities.Nemotron is presented as a particularly clear illustration.Source: §D.8 · pp. 29–30
  • For Nemotron, longer SecAlign++ training also reduces ToolTalk, BFCL, MMT-Bench, and AsyncTool scores, though WorkBench increases.The agentic effect is uneven rather than a uniform collapse in performance.Source: §D.8 · p. 30
  • On AgentDojo and AgentDyn, extending SecAlign++ from 313 to 625 steps lowers ASR, while general-capability scores fall on several tasks; RAISED remains closer to Vanilla.On AgentDojo, ASR falls from 4.0 ± 0.7% to 0.7 ± 0.0%; MMLU falls from 48.3 ± 0.7 to 42.8 ± 0.3.Source: §D.8 · pp. 29–30, Table 12 · p. 30, Table 13 · p. 30

E ABLATION STUDY

The ablations indicate that both self-distillation and self-generated trajectories contribute to preserving agentic utility while maintaining low attack success. The paper limits its robustness claims to static indirect prompt injections.

E ABLATION STUDY

  • The controlled comparisons isolate objective on Glaive trajectories, then data source under fixed objectives; the main comparison changes both objective and training data.The Glaive setup inserts an instruction from another example into a randomly selected tool output; the injected instruction is unrelated to the original task.Source: E ABLATION STUDY · pp. 30–31
  • On AgentDojo, self-distillation reaches 80.1 utility versus 79.1 for DPO and 73.2 for SFT; on AgentDyn, it reaches 55.5 versus 48.6 and 17.5, respectively.SFT attains slightly lower ASR than self-distillation, but at a much larger utility cost; self-distillation also substantially improves robustness over DPO on AgentDyn.Source: Table 14 · p. 31, E ABLATION STUDY · p. 31
  • With self-generated data, self-distillation and DPO reach the same reported ASR on both tasks, while self-distillation yields higher benign utility.ASR is 1.0 on AgentDojo and 1.6 on AgentDyn for both, within reported variation; utility rises from 82.2 to 83.5 on AgentDojo and from 55.6 to 60.0 on AgentDyn.Source: E ABLATION STUDY · p. 31
  • With DPO fixed, self-generated rather than Glaive trajectories increase utility and reduce ASR on both AgentDojo and AgentDyn.AgentDojo utility rises from 79.1 to 82.2 and ASR falls from 1.8 to 1.0; AgentDyn utility rises from 48.6 to 55.6 and ASR falls from 2.3 to 1.6.Source: E ABLATION STUDY · p. 31
  • With self-distillation fixed, self-generated trajectories raise utility from 80.1 to 83.5 on AgentDojo and from 55.5 to 60.0 on AgentDyn, maintaining low ASR.The reported utility benefit of self-generation therefore persists under either DPO or self-distillation.Source: E ABLATION STUDY · p. 31
  • Disabling the generation patterns lowers utility to 80.8±0.4 on AgentDojo and 56.7±2.1 on AgentDyn, near results with Glaive trajectories.The authors interpret this as evidence that gains come largely from steering generation toward scenarios exercising target behaviors, rather than synthetic generation alone.Source: E ABLATION STUDY · p. 31

F ADAPTIVE ATTACKS

  • Robustness claims concern static injections chosen independently of the defended model; the adaptive-attack runs provided no evidence of adaptive robustness.PISmith's optimization reward remained identically zero for most model–defense configurations, leaving the optimizer without a signal to improve attacks.Source: F ADAPTIVE ATTACKS · p. 32

G LIMITATIONS

The method preserves the clean model's behavior, so its quality is bounded by that model and the method is not a general alignment intervention. Evaluation and computational costs also leave boundaries on the reported scope.

G LIMITATIONS

  • Evaluation covers three model families, but not additional families or frontier models in the trillion-parameter regime.Whether the same security–utility behavior persists at that scale remains an empirical question.Source: G LIMITATIONS · p. 32
  • Clean-context self-distillation preserves the teacher's mistakes as well as its behavior, so defended-model quality is bounded by the clean model.The authors frame the method as a defense against adversarial context influence, not a general alignment or safety intervention.Source: G LIMITATIONS · p. 32
  • For the 10,000-trace corpus, storing top-256 teacher distributions requires approximately 15 GB per model; frontier-scale teacher passes are substantially more expensive.A model such as Kimi K3 activates roughly 104B parameters per token, approximately 4× to 9× the active parameter count of the evaluated dense models.Source: G LIMITATIONS · p. 32

H BROADER IMPACT

  • The authors present the method as defense in depth, complementing rather than replacing access control, sandboxing, and capability restrictions.Source: H BROADER IMPACT · p. 32
Loading 2610.06401v1…