PaperlayerView exact revision on arXiv

Source-linked AI summary

CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement

Boyang Zhang, Qingxin Xiao, Lingwei Dang, Qingyao Wu

arXiv:2609.07529v3cs.LG

TL;DR

Tool agents face malicious instructions embedded in tool outputs, while fixed attacks do not adapt to defender behavior. CoER co-evolves attackers and defenders against current and historical opponents, then refines the defender using safety- and task-verified demonstrations. Across seven domains, it reduces adaptive attack success and raises safe task completion.

  • Problem

    Fixed attack sets cannot evolve with defenders, although tool agents must reject malicious instructions in tool outputs while completing legitimate tasks.

    Source: §1 · pp. 1–2

  • Method

    CoER models within-task adaptive interactions as a general-sum Markov game, trains against current and historical opponents, and refines the defender with verified teacher demonstrations.

    Source: §1 · p. 2, §3 · p. 4, §4.2 · p. 5

  • Results

    Across three seeds, CoER reduces adaptive attack success from Base’s 41.31% to 0.22% and raises safe task completion from 39.60% to 76.24%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across seven domains and external benchmarks, CoER shows improved attack resistance, while controlled experiments find gains from both co-evolved initialization and population-derived refinement data.

    Source: §1 · p. 3, §7 · p. 10

    Generalization to other agent frameworks, nontextual modalities, or larger backbones remains untested; the attacker also shares the defender’s base model.

    Source: §6 · p. 9

Abstract

from arXiv · show

Language-model agents are vulnerable to indirect prompt injection (IPI) during tool use: adversarial instructions hidden in untrusted tool outputs can covertly redirect legitimate task execution. Existing work often trains and evaluates defenses against fixed attacks that do not adapt to the defender's behavior, so the resulting defenses may struggle against adaptive attacks in real-world settings. We argue that a strong defense against adaptive IPI must adapt during training to a continually evolving attacker. Building on this insight, we propose CoER, a verifier-grounded co-evolution and refinement framework that models interleaved tool calls and adaptive injections within a task as a general-sum Markov game: the defender advances the task through successive tool calls, while the attacker can inject multiple times within the same task and adapt subsequent attacks to the defender's responses and prior execution traces. After initializing the attacker from successful trajectories, bilateral adversarial reinforcement learning (BA-RL) retains historical policies from both roles as opponent populations and mixes current and historical opponents, extending training beyond the latest matchup. Attackers from these populations are then reused to challenge teacher agents, and only demonstrations verified for both safety and task completion are used to fine-tune the co-evolved defender. Across seven domains and three evaluation seeds, CoER reduces adaptive attack success from 41.3% to 0.2% and raises safe task completion from 39.6% to 76.2%; external benchmarks also show improved attack resistance. Further experiments validate the effectiveness of bilateral historical-opponent mixing and population-guided refinement. Attacker analyses show that co-evolution strengthens attack capabilities and that the trained attacker uses execution feedback to adapt subsequent injections.

1 INTRODUCTION

Tool agents must use untrusted tool content without following malicious instructions, but fixed attack sets cannot adapt to changing defenses. CoER combines multi-turn co-evolution with verified supervised refinement; across seven domains, it sharply reduces adaptive attack success while improving safe task completion.

1 INTRODUCTION

  • Tool returns can embed malicious instructions that cause data leakage, tool misuse, or unauthorized actions, while fixed attack sets cannot evolve with the defender.A useful defender must reject malicious instructions while preserving legitimate information and completing the task.Source: §1 · p. 1
  • Within-task adaptive IPI couples attacker revisions to visible defender responses and tool-state changes, requiring both attack resistance and legitimate task completion.Unlike dialogue-only safety settings, tool-agent safety concerns the entire execution.Source: §1 · p. 2
  • The paper frames two connected needs: sustain multi-turn co-evolution across tool domains and refine the defender against residual weaknesses without sacrificing task utility.Verified successful demonstrations add concrete safe action sequences to outcome-based training.Source: §1 · p. 2
  • CoER models feedback-dependent interactions as a general-sum Markov game, co-trains against current and historical opponents, then fine-tunes on demonstrations verified for safety and task completion.Attacker SFT initializes exploration, and retained attackers help collect teacher demonstrations for refinement.Source: §1 · p. 2
  • Across three seeds, CoER reduces adaptive attack success from Base’s 41.31% to 0.22% and raises safe task completion from 39.60% to 76.24%.The evaluation covers seven domains; external benchmarks also show improved attack resistance.Source: §1 · p. 3

2 RELATED WORK

Prior work studies IPI benchmarks and defenses, adaptive attack discovery, and attacker–defender co-evolution. CoER places attack discovery and defensive learning within feedback-dependent tool execution and reuses retained attackers for verified teacher demonstrations.

2 RELATED WORK

  • IPI benchmarks span tool-use environments, while defenses include provenance marking, structured inputs, alignment, prompt guards, and control-flow constraints.Adaptive evaluation probes weaknesses beyond fixed attack distributions, motivating joint security–utility assessment.Source: §2 · p. 3
  • Prior adaptive attack work includes red-teaming, strategy reuse, multi-turn jailbreak learning, transferable IPI injections, and attacks generated against frozen baselines.CoER couples attack discovery with defensive learning inside feedback-dependent tool execution.Source: §2 · p. 3
  • Dialogue and tool-agent co-evolution provide evolving safety challenges; CoER exposes both roles to historical opponents and reuses retained attackers to collect verified teacher demonstrations.The paper contrasts this design with ARLAS’s defender-side history and other approaches.Source: §2 · p. 3

3 PROBLEM SETUP: ADAPTIVE IPI ATTACK–DEFENSE AS A MARKOV GAME

The setup treats adaptive IPI as a partially observable, finite-horizon interaction in which defenders act through tools and attackers inject text at configured sites. Independent outcomes distinguish attack compromise from task completion, motivating a general-sum game.

3 PROBLEM SETUP: ADAPTIVE IPI ATTACK–DEFENSE AS A MARKOV GAME

  • The defender controls tool use while the attacker controls text at configured injection sites, with each role acting from its visible history.Attacker observations include the public defender trace, surrounding tool text, and prior attempts; each site triggers at most once.Source: §3 · pp. 3–4
  • The finite-horizon, turn-based, partially observable Markov game state includes tool state, both histories, handled sites, active role, and remaining budget.Neither role observes the full state; the attacker cannot access defender-private instructions or reasoning, and histories reset between executions.Source: §3 · p. 4
  • Attack success and task completion can both equal one, so completing the task does not establish that an execution was safe.For unsuccessful attacks with valid payloads, attacker reward is constant while defender reward varies with task completion; their sum is nonconstant.Source: §3 · p. 4

4 METHOD

CoER initializes attack exploration, trains both roles against changing and retained opponents, and refines the defender with verified teacher demonstrations. The stages use multi-turn interactions and feedback to address residual weaknesses while preserving safe task completion.

4.1 STAGE I: ATTACKER SUPERVISED INITIALIZATION

  • Successful multi-turn attack demonstrations initialize the attacker’s repertoire and bootstrap exploration under sparse terminal rewards.All attacker turns are supervised, retaining both early attempts and later revisions.Source: §4.1 · p. 4

4.2 STAGE II: BILATERAL ADVERSARIAL REINFORCEMENT LEARNING

  • The defender earns positive reward only for safe task completion, so task completion cannot offset compromise within an execution.BA-RL continues the defender’s execution until termination or budget exhaustion for final verification.Source: §4.2 · p. 5
  • BA-RL mixes current opponents with frozen historical policies from both roles, broadening training beyond the latest pairing.Historical attackers maintain exposure to earlier attacks, while historical defenders expose attackers to varied behavior and public feedback; only participating current policies train on their own generated tokens.Source: §4.2 · p. 5

4.3 STAGE III: POPULATION-GUIDED POLICY REFINEMENT

  • Retained attackers adapt injections to teachers’ public feedback; the method keeps normally terminated trajectories verified for both no attack success and task completion.It also includes successful replay from injection-configured tasks whose sites were not triggered, yielding 5,760 demonstrations after deduplication.Source: §4.3 · p. 6

5 EXPERIMENTS

CoER is evaluated across seven AgentDyn and AgentDojo suites using clean, fixed-template, and within-execution adaptive attacks. The experiments assess safety and utility, cross-suite performance, refinement, robustness to further attacks, and attacker learning.

5.1 EXPERIMENTAL SETUP

  • The workload contains 1,512 cases per seed, including 1,187 adaptive cases; baselines range from fixed-template training to bilateral RL and population-guided Defender SFT.U, ASR, and Safe-U measure task completion, attack success, and safe task completion; comparisons use fixed checkpoints across three seeds.Source: §5.1 · p. 6

5.2 Q1: SAFETY–UTILITY UNDER FIXED AND ADAPTIVE ATTACKS

  • 76.24% adaptive Safe-U and 0.22% ASR: CoER has the highest mean Safe-U among same-backbone baselines under fixed and adaptive attacks.Across three evaluation seeds, Defender SFT raises Safe-U from 54.84% to 76.24% and reduces adaptive ASR from 25.78% to 0.22%.Source: §5.2 · pp. 6–7, Table 1 · p. 7
  • Bilateral history raises adaptive Safe-U from 51.73% with NoPop to 54.09%; PPO also exceeds GRPO on adaptive Safe-U, 46.70% versus 36.90%.Defender SFT contributes an 8.13-point fixed Safe-U gain; GRPO takes 7–8× as long per training step as PPO on the same GPUs.Source: §5.2 · p. 7

5.3 Q2: CROSS-SUITE RESULTS AND EXTERNAL EVALUATION

  • CoER improves adaptive Safe-U over BA-RL in all seven suites by 4.2–47.0 points, while external evaluations show gains with protocol-specific utility variation.Shopping has the lowest final Safe-U at 57.2%; under AgentDyn’s broader protocol, Travel’s Safe-U decreases by 15.00 points from BA-RL at unchanged 2.86% ASR.Source: §5.3 · pp. 7–8, Figure 3 · p. 8, Table 2 · p. 8

5.4 Q3–Q4: REFINEMENT ATTRIBUTION AND ROBUSTNESS TO FURTHER ATTACKS

  • Matched experiments attribute refinement gains to both BA-RL initialization and population-derived demonstrations; CoER also retains the lowest union ASR under historical attack budgets.The two matched contributions are 8.42 and 10.36 adaptive Safe-U points; union ASR rises from 0.25% to 4.97% across one-to-eight-attempt budgets.Source: §5.4 · p. 8

5.5 Q5: ATTACKER LEARNING AND FEEDBACK USE

  • Frozen cross-play shows attacker Effective ASR against Base rising from 38.80% to 65.65% after BA-RL, while full execution feedback raises ASR from 29.57% to 38.75%.Across six attackers, Effective ASR is 0.18–0.47% for CoER versus 15.10–31.29% for continued BA-RL without Defender SFT; payload views are qualitative.Source: §5.5 · p. 9, Figure 4 · p. 9

6 LIMITATIONS

CoER’s refinement pipeline incurs data-generation costs and depends on strong teachers; several aspects of its generalization and population design remain untested or incompletely characterized.

6 LIMITATIONS

  • The evaluation covers seven AgentDyn suites; generalization to other agent frameworks, nontextual modalities, and larger backbones remains untested.The attacker shares the defender’s base model, so heterogeneous attackers could expose weaknesses that same-family co-evolution misses.Source: §6 · p. 9
  • Refinement adds teacher dependence and data-generation costs, while sequential stages and population-management hyperparameters constrain the pipeline.The sensitivity of population-management hyperparameters has not been fully characterized.Source: §6 · p. 9

7 CONCLUSION

CoER couples bilateral co-evolution with population-guided refinement to defend tool agents against adaptive indirect prompt injection. Across seven suites, it reports 76.24% mean adaptive Safe-U and 0.22% ASR, with gains also reported on AgentLAB.

7 CONCLUSION

  • CoER models multi-turn interactions as a general-sum Markov game, trains against historical opponents, and reuses retained attackers to collect verified teacher demonstrations.The demonstrations are used to fine-tune the co-evolved defender for safe task execution.Source: §7 · pp. 9–10
  • Across seven suites, CoER achieves 76.24% mean adaptive Safe-U and 0.22% ASR, with safety and utility gains on AgentLAB.With teachers, data size, and SFT updates matched, co-evolved initialization and population-derived data each yield gains, and their combination performs best.Source: §7 · pp. 9–10

AI USE STATEMENT

  • The authors report that generative AI assisted training-data construction, case-study annotation, and research assistance, with authors retaining responsibility for the work.Teacher trajectories were screened for safety, task completion, and quality.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The trained attacker can generate harmful payloads, so the authors restrict its intended use to controlled evaluation, red-teaming, and defense training.They recommend release safeguards for checkpoints, prompts, and generated attack corpora.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The appendices document task splits, verifier validation, training settings, evaluation counts, and manifests intended to accompany the artifact release under stated safeguards.Verifier validation includes human review and per-task reference trajectories.Source: REPRODUCIBILITY STATEMENT · p. 10

Supplementary Material

The supplement specifies how CoER represents, verifies, and trains on adaptive tool-use attacks, then maintains historical opponent populations for continued training.

A ATTACKER EXECUTION AND VERIFICATION

  • Attackers receive public execution traces, benign context, prior attempts, and contract feedback, while defender reasoning and system details remain hidden.Deterministic verifiers separately check attack success and task completion; after a verified attack, further injections stop while the defender continues to termination or its budget.Source: A ATTACKER EXECUTION AND VERIFICATION · pp. 14–15

B TRAINING CONFIGURATION AND IMPLEMENTATION

  • The asynchronous BA-RL workflow advances tool executions, updates only the participating role, refreshes historical populations, then uses retained attackers to collect teacher demonstrations for supervised refinement.The algorithm selects a defender checkpoint, retains attackers, screens safe-success teacher demonstrations, and fine-tunes the selected defender.Source: B TRAINING CONFIGURATION AND… · p. 16

B.1 ATTACKER INITIALIZATION AND ONLINE CONFIGURATION

  • Attacker SFT uses 3,995 verified successful conversations containing 11,655 attacker turns, supervising all retained responses, including earlier attempts not individually responsible for terminal success.The attacker demonstrations are generated against the base defender, then deduplicated before SFT.Source: §B.1 · p. 16

B.2 ROLE-MASKED PPO AND POLICY VERSIONS

  • Role-specific PPO trains only tokens generated by the participating current policy; opponent actions, tool outputs, and repeated payloads are masked from its losses.Terminal payoff is assigned at the last role-generated token, and masked positions preserve the role-specific GAE recurrence.Source: §B.2 · pp. 17–18

B.3 HISTORICAL-POPULATION ADMISSION AND SERVING

  • Each role serves four historical slots, with a probationary checkpoint evaluated across all seven suites before admission; eligible elites are selected using suite-stratified reward histories.A probationary checkpoint needs at least 32 reward-bearing evaluations in each suite; fitness is a population-sampling heuristic based on encountered opponents.Source: §B.3 · p. 18

B.4 DEFENDER-SFT DATA AND SUPERVISION

Defender refinement uses deterministically screened teacher trajectories and untriggered replay, then applies masked causal-language-model supervision to the resulting corpus.

B.4 DEFENDER-SFT DATA AND SUPERVISION

  • Accepted candidates must fail the attack, complete the task, terminate normally, and pass deterministic quality filters before one trajectory per source configuration is selected.Masked cross-entropy applies to teacher assistant turns, with user, tool, and padding positions receiving zero loss; the reported checkpoint is update 360 of a 720-update job.Source: §B.4 · pp. 18–19, Table 4 · p. 19
  • 6,974 of 7,859 safe, task-successful, normally terminated candidates pass quality screening, an 88.74% conditional acceptance rate.The 885 rejected candidates fail the additional quality screen.Source: §B.4 · p. 18
  • 5,760 demonstrations comprise 4,907 attacked trajectories and 853 untriggered replay executions, with replay making up 14.81% of the corpus.The replay examples come from injection-configured tasks whose attack sites were not triggered.Source: §B.4 · p. 19

C MAIN EVALUATION PROTOCOL AND ADDITIONAL COUNTS

Across repeated evaluations and additional stress tests, CoER maintains low adaptive attack success and strong safe completion; official benchmark results also show gains, with utility varying by suite.

C MAIN EVALUATION PROTOCOL AND ADDITIONAL COUNTS

  • The main workload contains 1,512 cases per seed: 157 clean, 168 fixed-template, and 1,187 adaptive cases, with two pre-declared adaptive exclusions.The common adaptive attacker has frozen weights across defenders but adapts injections within each execution; CoER comes from one training run.Source: C MAIN EVALUATION PROTOCOL AND… · p. 19
  • 0.2247% adaptive attack success (8/3,561 cases) and 0.3968% fixed-attack success (2/504 cases) are CoER’s totals across three evaluation seeds.Each seed evaluates 1,187 adaptive and 168 fixed cases.Source: C MAIN EVALUATION PROTOCOL AND… · p. 19
  • Across tested within-execution injection budgets, CoER has the lowest ASR and highest Safe-U, with ASR between 0.00% and 0.17% and Safe-U between 70.94% and 72.28%.Increasing the injection budget raises ASR and reduces Safe-U for Base, PPO, and BA-RL.Source: C MAIN EVALUATION PROTOCOL AND… · p. 22, Table 9 · p. 22
  • Bilateral historical training adds 23 and 17 safe completions over defender-only and attacker-only history, respectively, in a single-run ablation.These are descriptive differences without repeated-training uncertainty; bilateral history also yields the highest observed attacker Effective ASR against both frozen targets.Source: C MAIN EVALUATION PROTOCOL AND… · p. 22, Table 10 · p. 23
  • From Short to Long reference tool-call groups, pooled ASR rises from 13.24% to 37.19% and Safe-U falls from 67.95% to 33.14%; CoER has the best ASR and Safe-U in each group.In Long, CoER has 0.52% ASR and 54.29% Safe-U; the aggregate association does not isolate a causal effect of task length.Source: C MAIN EVALUATION PROTOCOL AND… · p. 23, Table 11 · p. 23

C.1 OFFICIAL AGENTDYN/AGENTDOJO FIXED-ATTACK EVALUATION

  • The official evaluation broadens Workspace coverage for one attack template but establishes neither held-out-template nor domain-OOD generalization.Workspace contributes 560 attack pairs, versus 240 raw pairs in the main adaptive manifest.Source: §C.1 · p. 24
  • CoER’s pooled seven-suite ASR / Safe-U are 2.32% / 77.93%, versus Base’s 20.15% / 62.16%.CoER lowers ASR in all seven suites and improves Safe-U in six; macro Safe-U rises from 52.01% to 70.61%, while utility effects vary by suite.Source: Table 12 · p. 24, §C.1 · p. 24

D EXTERNAL-BENCHMARK PROTOCOLS AND SCOPE

External benchmarks test CoER on AgentLAB’s held-out attack-goal types and InjecAgent payloads, while the section documents protocol boundaries and cautions that published reference results are not controlled comparisons.

D EXTERNAL-BENCHMARK PROTOCOLS AND SCOPE

  • AgentLAB evaluates 949 selected task–goal pairs per defender, allowing one adaptive rewrite when the initial attack fails; task success can coexist with compromise.AgentLAB shares some training environments with CoER, but its attack-goal types are held out from all three training stages.Source: D EXTERNAL-BENCHMARK PROTOCOLS AND SCOPE · p. 25
  • 77.13% Safe-U for CoER versus 37.83% for Base on AgentLAB; CoER’s 14.12% ASR shows residual vulnerability.The evaluation uses GPT-5.4 against all three defenders under the same attack-generation and adaptive-rewrite procedure.Source: D EXTERNAL-BENCHMARK PROTOCOLS AND SCOPE · p. 25, Table 13 · p. 25
  • 0.00% ASR on Base payloads and 1.97% on Enhanced payloads for CoER, versus BA-RL’s 4.34% and 17.35%, respectively.The benchmark reports aggregate ASR separately for Base and Enhanced payload settings.Source: D EXTERNAL-BENCHMARK PROTOCOLS AND SCOPE · p. 25, Table 14 · p. 25
  • InjecAgent and AgentLAB results do not guide checkpoint selection, tuning, or prompt revision; they test external payload transfer and held-out goals in partly shared environments.The two benchmarks therefore assess different transfer settings.Source: D EXTERNAL-BENCHMARK PROTOCOLS AND SCOPE · p. 25

D.1 SOURCES AND PROTOCOLS FOR PUBLISHED REFERENCES

  • Published reference scores use source-specific backbones, prompts, and denominators, so they provide context rather than controlled CoER comparisons.GPT-4o and Qwen3 snapshots are not established as identical across sources, and no verified Meta-SecAlign AgentLAB rate is available.Source: §D.1 · p. 26, Table 15 · p. 26, Table 16 · p. 26

E TRAINING DIAGNOSTICS

Training traces show rising attack pressure and changing relative performance against current and historical opponents. They describe one run with changing tasks and opponents, rather than fixed-opponent strength or uncertainty estimates.

E TRAINING DIAGNOSTICS

  • The diagnostics cover one run: 3,865 of 129,216 adaptive records were excluded, and online records are neither optimizer-update counts nor guaranteed PPO inputs.Tasks and opponents change, so the traces describe training pressure rather than fixed-opponent strength or uncertainty intervals.Source: E TRAINING DIAGNOSTICS · pp. 26–27
  • The attacker’s current–history Effective-ASR contrast shifts from −1.77 to +7.62 points between thirds; the defender contrast is +5.52 and +4.33 points.Figure 6 plots reached-only Effective ASR and contrasts pooled over five trailing blocks; changing tasks and opponents preclude causal inference.Source: E TRAINING DIAGNOSTICS · p. 27, Figure 6 · p. 27
  • Effective ASR rises from 29.32% to 39.92% across the first and last quarters, while reach stays near 64%.U changes from 62.25% to 60.41%, Safe-U from 54.26% to 49.23%, attacker reward from 0.192 to 0.259, and defender reward from 0.302 to 0.186.Source: E TRAINING DIAGNOSTICS · p. 27

F.1 EXECUTION-FEEDBACK ABLATION

The tested attacker is most effective when it can use execution feedback, and qualitative examples show later payloads responding to defender actions. The section also limits what can be inferred from selected traces and visualized payload samples.

F.1 EXECUTION-FEEDBACK ABLATION

  • 40.03% Effective ASR and 38.75% ASR under Full feedback; removing all feedback cuts ASR by 9.18 points and raises Safe-U from 40.69% to 51.56%.The ablation tests attacker a290 against Base Qwen3.5-9B with at most three injections and seed 0.Source: §F.1 · p. 27, Table 17 · p. 28

F.2 BEHAVIOR ACROSS CONTEXTS AND STAGES

  • Figure 7 uses an outcome-selected subset, not the full evaluation attack distribution; matched comparisons do not establish exact full-input identity or isolate feedback effects.The checkpoint match requires the same case, seed, task-text hash, goal contract, and first tool/injection site.Source: §F.2 · pp. 28–29, Figure 7 · p. 28
  • Figure 7’s embeddings show sampled payload behavior, not quantitative strategy diversity, and suite vocabulary and checkpoint or turn overlap remain visible.Its panels cover late-training payloads, matched Travel cases, and three-injection Travel episodes; ellipses show covariance spread, not confidence regions.Source: Figure 7 · p. 28, §F.2 · p. 29

G QUALITATIVE EXAMPLES AND FAILURE MODES

  • In the shopping example, the attacker revises its payload after the defender broadens its search; separately, a resisted attack still fails the legitimate task after a truncated tool call.The latter case illustrates why Safe-U requires both resisting the attack and completing the task.Source: G QUALITATIVE EXAMPLES AND FAILURE MODES · p. 29

H SHARED ATTACKER AND DEFENDER PROMPTS

  • The attacker prompt calls for payloads tied to the current task and gives later injections prior attempts and public defender actions; the general defender prompt adds no IPI-defense instructions.The feedback prompt excludes defender private reasoning.Source: H SHARED ATTACKER AND DEFENDER PROMPTS · p. 30, §6 · p. 31

I EXTENDED RELATED-WORK COMPARISON

  • Unlike SEMA’s open-loop plans, CoER interleaves injections with tool execution and conditions later injections on public consequences while defender actions change future opportunities.The comparison describes feedback-dependent interactions within an ongoing execution.Source: I EXTENDED RELATED-WORK COMPARISON · p. 32
Loading 2609.07529v3…