PaperlayerView exact revision on arXiv

Source-linked AI summary

SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation

Yibo Peng, Long Lian, David Wagner, Sizhe Chen

arXiv:2608.21500v1cs.CRcs.AI

TL;DR

Prompt injection lets malicious external data override trusted instructions, while existing defenses remain vulnerable to adaptive attacks. SecOPD adapts on-policy distillation to provide token-level feedback by scoring attacked-input rollouts against clean-input behavior. It achieves 9.0% ASR versus 94.0% for Meta-SecAlign on PISmith and 4.7% versus 5.5% in unseen agentic tool calling.

  • Problem

    Prompt injection threatens AI agents because malicious external data can override trusted instructions, while existing defenses remain vulnerable to adaptive attacks.

  • Method

    SecOPD adapts on-policy distillation by scoring tokens from attacked-input rollouts with the initialization model given corresponding clean inputs.

  • Results

    9.0% ASR versus 94.0% for Meta-SecAlign on PISmith, and 4.7% versus 5.5% in unseen agentic tool calling.

  • Takeaways & Limitations

    Token-level defensive fine-tuning substantially improves adaptive prompt-injection security and generalizes to an unseen agentic tool-calling domain.

  • Takeaways & Limitations

    The method targets indirect prompt injections, assumes trusted versus untrusted input parts are identified, and may face stronger future attacks.

Abstract

from arXiv · show

Prompt injection is listed as the \#1 threat to AI agents. When an agent accesses external data from websites, files, or emails, an attacker may inject a prompt into the data, saying, "Ignore all prior instructions and perform <an attacker's task>." To prevent arbitrary manipulation of agents, defenders try to train secure LLMs, which, however, still suffer from near 100% attack success rates (ASRs) against adaptive prompt injections. We note that this is because existing defensive finetuning recipes rely on sequence-level feedback signals (in DPO or GRPO). Treating an entire output equally prevents the model from learning precisely which output tokens are insecure. In this paper, we propose Secure On-Policy Distillation (SecOPD) that provides token-level feedback to guide defensive fine-tuning. The LLM receives an injected sample and produces a rollout, whose tokens are scored by the initialization model given the corresponding clean input. With more fine-grained training signals, our defended Qwen3.6-27B achieves a 9.0% ASR against the SoTA PISmith adaptive prompt injections, compared to 94.0% for the prior SoTA, Meta-SecAlign. The obtained security generalizes to domains completely unseen in training: in agentic tool calling, SecOPD achieves a 4.7% ASR compared to 5.5% for Meta-SecAlign. Code and the model are available at https://github.com/pppyb/SecOPD and https://huggingface.co/pybbb/Qwen3.6-27B-SecOPD.

1 Introduction

Prompt injection threatens agent deployment because external data can override trusted instructions, while existing defenses remain vulnerable to adaptive attacks. SecOPD addresses this gap with token-level on-policy distillation and substantially lowers attack success rates.

  • Motivation: Prompt injection is a top threat to AI agents because instructions embedded in external data can manipulate execution and cause harmful actions.Examples include vulnerable code patches, malware execution, and private-message exfiltration.
  • Motivation: Existing model-level defenses remain highly vulnerable to adaptive prompt injections, motivating stronger defensive fine-tuning.The paper focuses on building robust LLMs as secure foundations for agents.
  • Approach: SecOPD adapts on-policy distillation by scoring attacked-input rollouts with a teacher model run on the corresponding clean input.The clean-input teacher sees only the trusted instruction and benign data, so it provides a secure training target during training.
  • Approach: Token-level feedback rates each output token according to consistency with the trusted prompt, unlike sequence-level feedback that treats the response as one unit.This addresses mixed responses that partially answer the benign task and partially follow the injection.
  • Results: 9.0% ASR is achieved by SecOPD on PISmith adaptive attacks, compared with 94.0% for Meta-SecAlign on Qwen3.6-27B.The paper describes this as an order-of-magnitude improvement in security.
  • Results: 4.7% ASR is achieved by SecOPD in unseen agentic tool calling, compared with 5.5% for Meta-SecAlign on AgentDojo.The undefended model’s utility remains within a few percentage points.

2 Related Work

Prompt-injection defenses include system-level safeguards and model-level fine-tuning, but SecOPD changes the model-level training signal from response-level feedback to token-level feedback. It adapts on-policy distillation to score attacked-input trajectories against clean-input behavior.

  • Defense categories: System-level defenses add detectors, shields, filters, sanitizers, or action restrictions around a fixed backend LLM.These methods intervene outside the model itself.
  • Defense categories: Model-level defenses encapsulate untrusted data in a distinct message type and fine-tune the LLM to use it as context without following its instructions.Training samples pair simulated injections with secure responses.
  • SecOPD: SecOPD retains the model-level direction but replaces one feedback signal per response with token-level feedback from clean inputs.This distinguishes SecOPD’s training signal from prior model-level defenses.
  • Related distillation: Context distillation reproduces behavior induced by an auxiliary system prompt, whereas prompt-injection defense trains attacked inputs to match corresponding clean-input behavior.The two approaches share a distillation relationship but target different behaviors.
  • SecOPD: On-policy distillation scores every token in student-generated attacked-input trajectories with the initialization model evaluated on the corresponding clean input.The scoring includes reasoning tokens.
  • Adaptive evaluation: Adaptive prompt injections optimize against the defended model and can use search-, gradient-, or reinforcement-learning-based attacks.Static templates are described as weak because they overestimate robustness.

3 Preliminaries

Prompt-injection defenses seek to preserve trusted instructions while ignoring commands embedded in untrusted data, but existing DPO- and GRPO-style methods use sequence-level feedback. OPD instead supplies token-level supervision from a teacher model.

  • 3.1 Prompt Injection Defense Problem: Prompt-injection defense requires following trusted instructions while treating instructions embedded in untrusted data as non-command context.
  • 3.2 Sequence-Level Feedback for Security: DPO trains on preference pairs, but its response-level label does not identify which span follows the injected goal.
  • 3.2 Sequence-Level Feedback for Security: GRPO assigns a scalar reward to each on-policy response, typically using an LLM judge.
  • 3.2 Sequence-Level Feedback for Security: Mixed responses that answer both the benign and injected tasks cannot be properly rated by sequence-level DPO or GRPO feedback.
  • 3.3 Token-Level Feedback for Utility: OPD trains a student on its own rollouts while a teacher provides token-level feedback, yielding fine-grained supervision.

4 Methodology

SecOPD adapts on-policy distillation for prompt-injection defense by pairing attacked and clean inputs. The student generates attacked-input rollouts, while a fixed initialization model scores their tokens under the clean task to guide token-level updates.

  • 4 Methodology: Teacher scoring encourages the model to answer as if the injected instruction were absent.
  • 4 Methodology: SecOPD pairs clean and attacked inputs that share the trusted instruction and benign task context but differ in whether untrusted data contains an injection.
  • 4 Methodology: The student samples a rollout from the attacked input, while the frozen teacher scores the same generated tokens using the corresponding clean input.
  • 4 Methodology: Token-level advantages can preserve trusted-task spans while suppressing spans that follow the injection within the same response.
  • 4 Methodology: Each training iteration generates an attacked-input rollout, scores it under the clean input, and updates the student with token-level advantages.
  • 4 Methodology: SecOPD uses no external security judge or task-specific reward model; supervision comes from the initialization LLM on the benign task.

5 Experiments

Experiments show that SecOPD substantially improves robustness to adaptive prompt injections while preserving utility and transferring security to unseen tool-use tasks. Compared with Meta-SecAlign and GRPO, it offers the strongest overall security–utility balance.

  • Adaptive security: SecOPD reduces PISmith ASR to 9.0%, compared with 94.0% for Meta-SecAlign, while GRPO reaches 61.2%.PISmith is the strongest adaptive attack evaluated.
  • Adaptive security: SecOPD achieves the lowest ASR in both SEP Static and SEP Basic Adaptive settings, reaching 1.3% and 0.2%, respectively.These results show that the security gain is not limited to PISmith.
  • Transfer to tool use: On AgentDojo, SecOPD achieves 4.7% ASR versus 5.5% for Meta-SecAlign, demonstrating transfer to unseen tool-use tasks.AgentDojo tests whether defenses transfer from text completion to tool calling.
  • Utility: GRPO reaches 0.7% ASR but has 82.5% benign utility, compared with 90.7% for SecOPD.This illustrates the security–utility trade-off among defended models.
  • Utility: SecOPD’s average utility is 88.1%, matching the undefended model and exceeding GRPO’s 83.1%, while PISmith ASR remains 9.0%.SecOPD records 88.6% utility on SEP, 90.7% on AgentDojo, and 94.3% on Minerva-Math.
  • Case study: In a PISmith case study, SecOPD answers the trusted biodiversity question without appending the injected sentence, unlike Meta-SecAlign and GRPO.The example illustrates suppression of injection-only tokens while retaining the trusted response.

6 Conclusion

Existing defensive fine-tuning recipes use sequence-level feedback, leaving LLMs vulnerable to adaptive prompt injections. SecOPD assigns fine-grained token-level credit and achieves substantially stronger security than the prior state of the art.

  • Sequence-level feedback in existing defensive fine-tuning recipes leaves LLMs vulnerable to adaptive prompt injections.
  • SecOPD provides token-level training signals to assign fine-grained credit during defensive fine-tuning.
  • SecOPD achieves an order-of-magnitude lower ASR against the strongest adaptive attack than the prior state of the art.
  • SecOPD's results challenge the view that models cannot be relied upon to provide secure behavior.

Limitations

SecOPD addresses indirect prompt injections with benign user prompts and malicious environment data, under an assumption that trusted and untrusted input parts are clearly signaled. The authors do not claim to solve prompt injection broadly, and future attacks may still break the model.

  • The method targets indirect prompt injections, not jailbreaks, direct prompt injections, or attacks where the user is malicious.
  • SecOPD assumes the system receives a clear signal identifying which input parts are trusted and which are untrusted.
  • The authors do not claim to solve prompt injection, and more advanced future attacks may break the model.

Ethical Considerations

The work evaluates prompt-injection defenses in controlled benchmark settings and frames robustness improvements as risk reduction rather than complete protection. Secure deployment still requires defense-in-depth mechanisms.

  • The experiments use controlled benchmark attacks intended to measure model robustness rather than provide instructions for attacking deployed systems.
  • No prompt-injection defense should be treated as complete protection for deployed agents.
  • Deployed agents should additionally use input filtering, action constraints, monitoring, and least-privilege tool access.

A Benchmark Details

The benchmark suite tests whether models separate trusted instructions from untrusted data across static, adaptive, and tool-calling prompt-injection settings. Evaluations vary attack form, injection position, target-model formatting, attacker adaptation, and tool-use outcomes.

  • SEP benchmark: SEP tests whether instruction-tuned LLMs separate trusted instructions from untrusted data.
  • Static attacks: Static SEP evaluation uses six attack variants, with combined ASR counting an example as successful if any variant succeeds.
  • Adaptive attacks: Qwen-adaptive attacks replace literal chat-template delimiters with nearby delimiter-like tokens selected in the model's embedding space.
  • Adaptive attacks: PISmith trains one attacker per target defense using on-policy reinforcement learning from black-box victim feedback, then reports pass@10 ASR on 1,024 SEP examples.
  • Agentic tool calling: AgentDojo evaluates whether tool-calling agents complete attacker-specified tasks across Banking, Slack, Travel, and Workspace.
  • Static attacks: The six static variants differ by attack form, including direct goals, override instructions, delimiter-like completion structures, and combinations thereof.
  • Static attacks: Before variants place injected content before benign data to test whether robustness depends on injection position.
  • Examples: SEP examples keep the trusted instruction fixed while modifying only the untrusted input across attack constructions.

C Utility Benchmarks

The paper evaluates instruction-following, factual, reasoning, mathematical, attack-success, and human-audit behavior across clean and injected settings. These evaluations use benchmark-specific definitions, adaptive attacks, and a multi-call judge protocol.

  • AlpacaEval2 measures general instruction-following quality using length-controlled win rate.
  • SEP Utility measures instruction-following quality on clean examples without injected instructions.
  • MMLU-Pro, GPQA-Diamond, GSM8K, and Minerva-Math evaluate broad factual reasoning, graduate-level science, grade-school mathematics, and competition-style mathematics, respectively.
  • Attack success is judged from the evaluated response span using witness filtering and multiple LLM calls, with success requiring unanimous YES judgments.For thinking-enabled models, evaluation first extracts the final-answer span after the first </think> marker.
  • Human annotation agreed with all 300 audited static SEP labels, confirming all 33 positives and finding no attack success among 267 sampled negatives.The audit was stratified by attack family and included every evaluation-positive output from the audited run.
  • AgentDojo evaluates Utility, AdvUtility, and ASR across Workspace, Banking, Travel, and Slack using 949 user-task–injection-task pairs.Utility measures benign task success, AdvUtility measures user-task success under injection, and ASR measures attacker-task success under injection.

J.1 Utility Error Analysis on Minerva-Math

The Minerva-Math utility analysis examines cases where SecOPD answered incorrectly despite correct answers from the undefended model, finding that most errors reflected output handling rather than systematic reasoning degradation.

  • Only 4/46 cases involved genuine reasoning errors.Most remaining cases involved final-answer organization, extraction, or incomplete completion.
  • Most errors were failures in final-answer organization, extraction, or completion rather than evidence of systematic reasoning degradation.
  • Representative cases included task switching, verifier-mismatched extraction despite deriving 1/16, incomplete derivations, and a cube-triangle reasoning error.One task-switching example output [2, 5) instead of 78, while the reasoning-error example concluded four triangles instead of three.
Loading 2608.21500v1…