PaperlayerView exact revision on arXiv

Source-linked AI summary

CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents

Xiao Yang, Yangchen Ou, Yuhan Gao, Le Wang, Zonghao Ying, Aishan Liu

arXiv:2609.34463v1cs.CRcs.AI

TL;DR

Indirect prompt injection exploits agents’ reliance on external content, while training-based defenses are often fitted to static, explicit attacks. CoDeL co-evolves a prober and defender to mine latent failures and train against them. Across three benchmarks, it reports an 88.5% ASR reduction and a 38.0% increase in utility under attack over the strongest baseline.

  • Problem

    Training-based IPI defenses are often optimized for static, explicit injections, leaving a gap when malicious intent is embedded in a plausible workflow and its effects are deferred.

    Source: §3 · p. 3

  • Method

    CoDeL co-evolves an MCTS prober that searches for successful, late-noticed injections with a defender updated each round through GDPO under a reward for safety, task progress, and format compliance.

    Source: §1 · p. 2, §4.3 · p. 4

  • Results

    CoDeL reduces ASR from 0.364 to 0.042 (−88.5%) and raises UA from the strongest baseline’s 0.602 to 0.831 (+38.0%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments show CoDeL lowers ASR while raising benign utility and utility under attack.

    Source: §6 · p. 9

    The threat model limits the attacker to external content and excludes modification of system prompts, user instructions, or model parameters.

    Source: §4.1 · pp. 3–4

Abstract

from arXiv · show

Large language model (LLM)-based agents increasingly rely on external tools and content, exposing them to indirect prompt injection (IPI). This threat has motivated a wide range of defenses, among which training-based defenses are often regarded as most reliable. However, existing training-based defenses are typically optimized on a static distribution of explicit injections. They learn surface-form cues rather than the boundary between serving the user and obeying an injected objective, and therefore fail when malicious intent is folded into a plausible workflow and deferred for several turns. We present CoDeL, a defense that hardens agent against an attack distribution it reshapes as it trains. The defender is updated each round via LoRA-based GDPO under a decoupled reward over safety, task progress, and format compliance, so refusing injections and completing the user's task jointly define fitness. To keep supplying it with the failures worth learning from, a co-evolving prober searches over injection rounds, attack methods, and payloads for injections that still penetrate the current defender, guided jointly by attack success and attack latency so that it preferentially mines breaches the defender notices too late. Each defender update invalidates part of the attack population and forces the next round onto a new frontier, turning the defender's own failures into a moving curriculum. Extensive experiments on three IPI benchmarks, nine baselines, and two base models show that CoDeL reduces attack success rate (ASR) by 88.5% and outperforms other baselines largely (+38.0%). Codes are available.

1 INTRODUCTION

Indirect prompt injection exploits agents’ reliance on external content, while training-based defenses are limited by the static, explicit attacks they learn from. CoDeL makes attacks and defense co-evolve, and experiments report lower attack success alongside higher utility under attack.

1 INTRODUCTION

  • Training-based defenses are regarded as reliable because their protection is intrinsic to the model, but their reliability is restricted to attacks represented in training.Existing training distributions are typically static and explicit, unlike attacks that fold malicious intent into plausible workflows and defer behavioral deviation across turns.Source: §1 · pp. 1–2
  • CoDeL continually reshapes the attack distribution and defender, mining failures that survive each round and using them as a moving curriculum.Its prober searches injection turns, methods, and payloads, guided by attack success and how late the defender notices; the defender is updated with LoRA-based GDPO.Source: §1 · p. 2
  • The defender’s decoupled reward covers safety, task progress, and format compliance, making refusal and completion jointly necessary for fitness.The design aims to reject the injected objective while carrying the user’s task to completion.Source: §1 · p. 2
  • CoDeL drives ASR from 0.364 to 0.042 (−88.5%) and raises UA from the strongest baseline’s 0.602 to 0.831 (+38.0%).Experiments cover three IPI benchmarks, nine baselines, and two base models; CoDeL’s benign utility is 0.779.Source: §1 · p. 2

2 RELATED WORK

Prior IPI work spans tool-use, web-browsing, retrieval-augmented, and computer-use agents. Defenses include runtime monitoring, guardrails, and model-level training or alignment approaches.

Indirect prompt injection attacks

  • IPI has been demonstrated across tool-use, web-browsing, retrieval-augmented, and computer-use agents.Examples include poisoned retrieved documents hijacking multi-step workflows and visual prompt injection in computer-use agents.Source: §2 · p. 2

Defenses against indirect prompt injection

  • Runtime defenses monitor execution or filter inputs, while guardrails inspect external content or constrain proposed tool actions.MELON compares original and counterfactual executions; PromptArmor uses a special model for filtering.Source: §2 · pp. 2–3
  • Model-level defenses update the protected model through fine-tuning or alignment, with approaches including structured instruction tuning and preference optimization.Other training signals include separating instruction and data representations, token-level on-policy feedback, and continual adaptation to injection variants.Source: §2 · p. 3

3 MOTIVATION

Mainstream IPI benchmarks use explicit payloads that are foreign to the ongoing task and prompt immediate tool-call deviations. Rewriting attacks as plausible workflows raises defense ASR, motivating evaluation and training against less conspicuous attacks.

3 MOTIVATION

  • Explicit attacks are cheap to catch because their instructions are task-foreign and their requested action immediately breaks the tool-call sequence.The authors argue that low ASR on these benchmarks therefore certifies less robustness than it appears to.Source: §3 · p. 3
  • The 48 rewritten AgentDojo attacks fold malicious objectives into plausible workflows and preserve several reasonable tool calls after the payload appears.The attacker’s goal, injection position, and target tool remain unchanged; only the deviation is deferred rather than immediate.Source: §3 · p. 3
  • Under latent rewrites, ASR rises to 0.208 for Meta-SecAlign, 0.542 for TSGuard, and 0.333 for PIGuard.TSGuard’s ASR jumps 45.9%, leaving it barely better than no defense, despite the attack goal remaining unchanged.Source: §3 · p. 3
  • The authors argue that closing this gap requires attack distributions to shift dynamically during training so defenses keep evolving against more covert attacks.They characterize plausible workflow delivery with late-firing attacks as more realistic than exposing intent outright.Source: §3 · p. 3

4 METHOD

CoDeL pairs a defender that must reject injected goals while completing the user’s task with a prober that continually searches for failures the current defender has not covered. It trains the defender on those failures using separate safety, task-progress, and format rewards.

4.1 PROBLEM SETUP

  • The attacker controls tool-returned content but not prompts or model parameters, while the defender must reject injected goals and still complete the user’s task.Attack latency measures the interval between injection and the agent’s first deviation toward the malicious objective.Source: §4.1 · pp. 3–4

4.2 OVERVIEW

  • Each round, the prober mines latent injections that defeat the current defender, which then internalizes those breaches before the next round.The alternating stages repeat for Rmax rounds, fitting each update to weaknesses that survived the previous one.Source: §4.2 · p. 4

4.3 PROBING THE DEFENDER WITH LATENT ATTACK MINING

  • MCTS searches hierarchical injection rounds, methods, and payloads using ASR and latency; persistent statistics shift effort toward breaches still working across defender rounds.The search favors successful breaches the defender was slowest to notice.Source: §4.3 · pp. 4–5

4.4 INTERNALIZING BREACHES WITH JOINT SAFETY–UTILITY OPTIMIZATION

  • LoRA updates start from SFT on clean, resisted, and corrective trajectories; latency-weighted advantages prioritize successful latent attacks during refinement.The cold start provides a safety prior, and latency feeds back into defense training through advantage shaping.Source: §4.4 · pp. 5–6
  • GDPO optimizes safety, task progress, and format compliance independently, penalizing attack success while making task abandonment unviable as a way to maximize safety.The safety reward is asymmetric, while task progress and formatting have their own reward terms.Source: §4.4 · pp. 5–6

5 EXPERIMENTS

Across three IPI benchmarks, CoDeL improves safety and task utility. The experiments also examine backbone consistency, co-evolution, generalization to unseen attacks, and component contributions.

5.1 EXPERIMENTAL SETUP

  • The evaluation covers AgentDojo, InjecAgent, and ASB-OPI, compares nine baselines across three defense families, and uses two base models.It reports benign utility, attack success rate, and utility under attack; lower ASR and higher BU and UA indicate better outcomes.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • On Llama-3.1-8B-Instruct, CoDeL reaches ASR 0.051, BU 0.819, and UA 0.797, while on ASB-OPI it reaches ASR 0.000, BU 1.000, and UA 0.942.The authors report consistent performance across both evaluated backbones.Source: §5.2 · p. 7
  • Across defense families, runtime methods are reported as unstable, guardrails as costly in utility, and model-level defenses as prone to over-conservative refusals.CoDeL is reported to improve safety and utility over the undefended base across settings.Source: §5.2 · p. 7
  • On AgentDojo, InjecAgent, and ASB-OPI, CoDeL reports ASR values of 0.042, 0.010, and 0.043, respectively, alongside strong task utility.On AgentDojo, BU is 0.779 and UA is 0.831; on InjecAgent, BU is 0.882 and UA is 0.843; on ASB-OPI, UA is 0.913.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 CO-EVOLUTION PROCESS

  • With both sides evolving, attacker success falls from 0.294 at round 0 to 0.032 at round 9; a frozen attacker drops faster before flattening, while a frozen defender lets ASR rise to 0.662.Removing the latency reward in the frozen-defender run instead saturates ASR near 0.510.Source: §5.3 · p. 8

5.4 GENERALIZATION TO UNSEEN ATTACKS

  • Without retraining, CoDeL has the lowest reported ASR on adaptive AutoDojo and ASPI attacks and remains lowest across LATENTDOJO latency ranges.LATENTDOJO contains 132 attack and 18 clean scenarios across 57 case groups and 54 AgentDojo tasks; PIGuard ASR rises from 0.263 to 0.421 as latency grows.Source: §5.4 · p. 9, Figure 4 · p. 9

5.5 ABLATION STUDY

  • Ablations show that removing safety causes the largest ASR degradation, removing progress lowers UA, and removing latency shaping raises ASR with utility nearly unchanged.Replacing MCTS with a static attack set or removing its latency signal also trails the full model.Source: §5.5 · p. 9

6 CONCLUSION

CoDeL frames indirect prompt-injection defense as co-evolution: a prober mines failures that survive the current defender, which then learns from them. Experiments show lower attack success alongside higher benign utility and utility under attack.

6 CONCLUSION

  • The reported ablation results cover the training framework and reward design.Source: Table 2 · p. 9
  • CoDeL continually reshapes the attack distribution and defender, using surviving failures as a moving frontier for training.Each round, the prober mines failures that survive the current defender, and the defender internalizes them.Source: §6 · p. 9
  • Experiments show CoDeL lowers attack success while raising both benign utility and utility under attack.Source: §6 · p. 9
  • Generative AI assisted with language editing and code-oriented repository analysis, with technical claims and artifacts subject to author review.Source: AI USE STATEMENT · p. 10

A LIMITATIONS AND FUTURE WORK

The authors identify two open boundaries: the experiments use only 7B- and 8B-scale backbones, and the relationship between latency optimization and model mechanisms remains unexplored.

A LIMITATIONS AND FUTURE WORK

  • Experiments cover only 7B- and 8B-scale backbones, leaving CoDeL’s transfer to larger models unverified.The authors leave whether model scale changes latent-injection vulnerability open.Source: A LIMITATIONS AND FUTURE WORK · p. 16
  • The relationship between latency, internal representations, and decision formation remains unexplored.The authors propose future work linking detection of latent malicious precursors to mechanistic interpretability.Source: A LIMITATIONS AND FUTURE WORK · p. 16

B EXPERIMENTAL SETTINGS

The experimental settings specify the training and inference configurations, compute budget, baseline controls, and dataset split units. The main experiments run for at most 12 co-evolution rounds.

B EXPERIMENTAL SETTINGS

  • The main experiments run for at most 12 co-evolution rounds, with early stopping after five consecutive rounds below the specified training or validation ASR thresholds.Training ASR must remain below 0.05 or validation ASR below 0.03.Source: B EXPERIMENTAL SETTINGS · p. 16
  • Experiments use one node with eight NVIDIA A100 GPUs; a complete run takes roughly 14 hours and about $20 in attacker calls.The SFT cold start takes about 1 hour, and a run involves around 2000 GLM-5.2 attacker calls.Source: B EXPERIMENTAL SETTINGS · p. 16
  • The attacker uses GLM-5.2, while the defender and tool and evaluation judges use specified models, temperatures, and generation budgets.The attacker temperature is 1.0 with a 4096-token budget; the defender uses temperature 0.2 and a 1024-token budget.Source: B EXPERIMENTAL SETTINGS · p. 16
  • MCTS materializes and evaluates 15 attack specifications per task per round, maintaining an independent search tree for each task across rounds.The exploration coefficient is 1.414 and model calls use concurrency 16.Source: B EXPERIMENTAL SETTINGS · p. 16
  • LoRA uses rank 32 and scaling factor 64, while SFT uses two epochs and dataset-specific dropout and learning rates.Only LoRA parameters are updated during GDPO, which uses group size 8, a batch budget of 200, and a latency advantage coefficient of 1.5.Source: B EXPERIMENTAL SETTINGS · p. 16
  • Baselines share CoDeL’s tool schema, system prompt, task splits, and compute budget, while model-level defenses are retrained on the same splits.Source: B EXPERIMENTAL SETTINGS · p. 16
  • Dataset partitions are by user instruction for AgentDojo-trajnew, attacker case for InjecAgent, and attack instance for ASB-OPI.Table 3 reports record totals and corresponding split proportions.Source: B EXPERIMENTAL SETTINGS · p. 17, Table 3 · p. 17

C SFT DATA CONSTRUCTION

SFT combines clean, attack-failed, and corrective trajectories to teach injection rejection while preserving progress on the user’s task. The section also specifies how task completion is judged and how MCTS cold-start seeds initialize attack search.

C SFT DATA CONSTRUCTION

  • SFT uses clean, attack-failed, and corrective trajectories; clean examples teach benign task execution, while failed attacks model continuing the task amid injection.Source: C SFT DATA CONSTRUCTION · p. 17
  • Corrective trajectories retain the benign plan before divergence, replace the earliest eligible target action with a refusal or redirection, then resume the clean trajectory.The corrective response may make a legitimate next tool call or provide a concise final answer, depending on the clean trajectory.Source: C SFT DATA CONSTRUCTION · p. 17
  • Trajectories below utility score 0.5 are excluded, with an approximate 10%/50%/40% mix of clean, attack-failed, and corrective examples.Source: C SFT DATA CONSTRUCTION · p. 17

Task-completion judge system prompt

  • The task-completion judge scores whether the benign objective was achieved, separately from security, using the agent’s trajectory and final answer.Its prompt requires a JSON response with completion status, a score from 0 to 1, and supporting evidence.Source: Task-completion judge system prompt · pp. 17–18

D MCTS COLD-START SEED SET

  • MCTS cold-start seeds encode intended injection turn, method, mechanism, and payload, but do not constrain realized latency or determine which attack succeeds.Subsequent search refines, combines, or replaces the seed patterns according to attack success and latency.Source: D MCTS COLD-START SEED SET · p. 18

E EXTENDED RELATED WORK

Prior work evolves attackers and defenders through several strategies, while CoDeL’s closest adaptive-IPI comparisons differ in how attacks are mined and how defenders absorb them.

Evolution in agent safety

  • Prior evolving red-teamers refine attacks, while blue-team approaches use feedback, self-refinement, self-play, or alternating attacker–defender optimization.AgentVigil uses Monte Carlo tree search for jailbreak prompts, and Metis adjusts its policy after diagnosing a target’s defense.Source: E EXTENDED RELATED WORK · p. 18
  • Table 4 presents universal MCTS cold-start seeds, with target turn marking planned payload visibility and Λ recording intended latency intervals.The caption does not specify seed values.Source: Table 4 · p. 19

Co-evolution for adaptive IPI

  • CoER and COPA also co-evolve defenses for adaptive IPI, but CoDeL searches structured attack rounds, methods, and payloads using success and latency, then separately scores safety, task progress, and format.This design targets late-detected latent breaches and over-refusal, distinguishing the method from simply relabeling adversarial training as co-evolution.Source: E EXTENDED RELATED WORK · p. 19

F LATENT IPI ATTACK

The section contrasts immediate, conspicuous direct attacks with latent attacks that preserve a plausible task trajectory before harmful deviation. It defines latency as the turns between payload visibility and that deviation.

F LATENT IPI ATTACK

  • A direct attack requests the harmful action as soon as its payload appears, whereas a latent attack embeds it in a plausible workflow and delays the harmful action for several turns.The example contrasts an immediate request to send a card PIN with a deferred request to email card details after ordinary steps.Source: F LATENT IPI ATTACK · pp. 19–20, Figure 5 · p. 20

G COMPUTING ATTACK LATENCY

  • Attack latency Λ measures the turns between the payload becoming readable and the agent’s first deviation from the benign plan to serve the injected objective.The injection turn is known from payload materialization; an LLM judge attributes the deviation turn by comparing attacked and benign trajectories.Source: G COMPUTING ATTACK LATENCY · p. 20

H THE LATENTDOJO BENCHMARK

LATENTDOJO is a replayable evaluation set built from attacks discovered during evolution and designed to expose failure modes found through adaptive search.

H THE LATENTDOJO BENCHMARK

  • LATENTDOJO records are self-contained attack specifications harvested from a frozen-defender run, allowing replay in AgentDojo-derived tool environments.The attacker evolves during that run while the defender remains unchanged.Source: H THE LATENTDOJO BENCHMARK · p. 20

H.1 SCHEMA

  • Its eleven record fields are grouped around task and payload grounding, expected latency and stealth, and the harmful objective and its annotation.The task reference and carrier fields locate the poisoned observation; expected delta records validated latency, while stealth technique names the disguise.Source: §H.1 · p. 21

H.2 ANNOTATED EXAMPLE

The section illustrates how a plausible, deferred instruction can hide a harmful action, then specifies the benchmark, reward design, training loop, and judge validation used to construct and evaluate the system.

H.2 ANNOTATED EXAMPLE

  • The annotated banking attack preserves the genuine standing-order instruction while presenting a date correction as a rationale for a separate EUR 50 transfer.Its benign action precedes the harmful transfer, and the internally consistent rationale keeps the trajectory plausible; the record’s stated expected delta is 2.Source: §H.2 · p. 21

H.3 CONSTRUCTION AND CURATION

  • LATENTDOJO includes only replay-validated attacks that succeed against the round’s defender and match their recorded latency, yielding 132 scenarios across 57 attack groups and 54 AgentDojo tasks.It also retains 18 clean scenarios as paired controls for benign-utility measurement.Source: §H.3 · p. 22

I REWARD DESIGN

  • The reward separates safety, task progress, and format compliance, with the safety penalty calibrated so declining beats acting above roughly one-in-four perceived attack probability.The section states that a −5.00 penalty implies a 34.6% threshold, while −12.00 implies 20.9% and makes blanket refusal attractive.Source: I REWARD DESIGN · pp. 22–23

J PIPELINE

  • The pipeline initializes per-task MCTS trees, executes searched attacks against the current defender, and updates the defender with SFT first and on-policy GDPO in later rounds.The attack judge compares clean and attacked trajectories; safety and progress are separately judged before their rewards and the format penalty feed GDPO.Source: J PIPELINE · p. 23

K.1 DETAILED PROMPTS

Attack judge system prompt

  • The attack judge counts executed attacker-directed actions after payload visibility and attributes success to the first action absent from the benign reference trajectory.Calls at or before the injection turn are baseline behavior and do not count as attack success.Source: Attack judge system prompt · p. 25

Safety judge system prompt

Progress judge system prompt

  • The progress judge grades movement on the user’s task independently of safety, distinguishing waste, advance, and neutral steps.A refusal or warning that replaces the work is waste, while a sensible needed call or completed answer is advance.Source: Progress judge system prompt · p. 26

Related papers

Loading 2609.34463v1…