PaperlayerView exact revision on arXiv

Source-linked AI summary

Kill-Chain Canaries: Stage-Level Tracking of Prompt Injection Across Attack Surfaces and Five Production LLMs

Haochuan Kevin Wang, Zechen Zhang

arXiv:2603.28013v4cs.CRcs.AIcs.LG

TL;DR

Outcome-only prompt-injection evaluations do not distinguish whether an attack is observed from whether it is executed. This paper tracks canary tokens through four kill-chain stages across multiple models, surfaces, and defenses, finding that attack outcomes diverge downstream and vary across settings.

  • Problem

    Outcome-only prompt-injection evaluation conflates whether an agent observes an injection with whether it acts on it.

    Source: §1 · p. 2

  • Method

    A unique canary in each injected payload lets the logger record the furthest of four stages reached across models, attack surfaces, and defense conditions.

    Source: §1 · p. 2

  • Results

    Claude Haiku 4.5 and Claude Sonnet 4.5 executed none of their 164 text-surface attacks, while GPT-4o-mini executed 53%.

    Source: §8 · p. 10

  • Takeaways & Limitations

    Outcomes depend on attack channel and defense coverage, and the reported results motivate evaluating multiple channels and pipeline stages.

    Source: §8 · p. 10

    The raw-versus-relayed Claude comparison differs in task and leaves open whether relayed injections are harder to refuse.

    Source: §4.8 · p. 8

Abstract

from arXiv · show

Multi-agent LLM systems now read documents, web pages and tool results on behalf of users, yet their resistance to prompt injection is usually reported as one number: did the attack succeed? We introduce a kill-chain canary method that plants a unique token in every injected payload and records the furthest of four stages it reaches (Exposed -> Persisted -> Relayed -> Executed), across 950 runs, five production LLMs, six attack surfaces, and five defense conditions. Exposure was 100% among runs that called the tool; the outcomes differ downstream. Claude Haiku 4.5 and Claude Sonnet 4.5 executed none of their 164 text-surface attacks, and in the text relay the canary token never appeared in a memory write (0/40); GPT-4o-mini executed 53% of its attacks. Four findings follow. (1) A Claude writer kept the canary token out of shared memory in every relay run we report; one cross-model pairing (Claude writer, GPT-4o-mini reader, n = 3) is consistent with this protecting the reader, and other pairings were not tested. (2) As readers, the Claude models executed 0/40 raw pre-seeded injections, but Claude Haiku 4.5 executed 2/3 injections relayed by GPT-4o-mini; whether relayed injections are harder to refuse than raw ones is an open question. (3) DeepSeek Chat went from 0/24 on pre-seeded memory to 8/8 on tool results, scenarios that also differ in task and payload format; white-text PDF payloads, invisible on the rendered page, succeeded at least as often as visible ones. (4) pi_detector and write_filter failed on channels they do not inspect, spotlighting failed on content it wraps, and write_filter blocked the PDF relay but not the text relay, a difference we cannot explain. Code and run logs are publicly released: https://github.com/KevinChunye/prompt_injection

1 Introduction

Outcome-only prompt-injection evaluation conflates whether an agent observes an attack with whether it acts on it. The paper introduces stage-level canary tracking and reports findings about writers, readers, channels, and defenses.

1 Introduction

  • Outcome-only attack success cannot distinguish an injection dropped before a memory write from one relayed to a reader that refuses to execute it.The two pipelines can both yield 0/3 attack success despite different failure stages.Source: §1 · p. 2
  • The kill-chain canary method records the furthest of four stages reached by a unique token planted in each injection across five models, six surfaces, and five defense conditions.The evaluation uses agent_bench, a custom multi-agent harness.Source: §1 · p. 2
  • Claude writers kept canary tokens out of shared memory in reported relay runs, while GPT-4o-mini readers executed 2/3 relayed injections from Claude Haiku 4.5.The cross-model pairing is consistent with the writer protecting the reader, but other pairings were not tested.Source: §1 · p. 2
  • Claude models executed 0/40 raw pre-seeded injections, but whether relayed injections are harder to refuse remains open.The reported comparison includes only one cross-model pairing, with n = 3.Source: §1 · p. 2
  • DeepSeek Chat’s outcomes differed between pre-seeded memory and tool results, while white-text PDF payloads succeeded at least as often as visible ones.The scenarios also differ in task and payload format, so the observed contrast does not isolate a channel effect.Source: §1 · p. 2
  • The document-ingestion pattern also appears in financial NLP pipelines handling earnings-call transcripts, SEC filings, and analyst reports.The paper reports that lightweight defenses failed on channels they do not inspect, and spotlighting failed on content it wraps.Source: §1 · p. 2

2 Related Work

Prior evaluations often report attack outcomes without locating where an injection stops in an agent pipeline. This work relates stage-level canary tracking to studies of attacks, defenses, monitoring, and agent settings.

2 Related Work

  • AgentDojo [4] and InjecAgent report outcome-oriented metrics, while this work records where a canary stops in the pipeline.Prompt Infection [6] and Zombie Agents [7] study propagation through memory; this evaluation measures the write step directly.Source: §2 · p. 2
  • Against adaptive attacks that bypass defenses [8], the evaluation finds lightweight defenses failing against non-adaptive attacks on channels they do not inspect or content they wrap.AgentWatcher [11] proposes rule-based monitoring; stage-level instrumentation offers a complementary empirical view of defense activation.Source: §2 · p. 2
  • The PDF relay extends covert visual prompt-injection work [13] to multi-agent chains, while permission_esc reports near-zero ASR (2/132) for its payload design.The authors suggest privilege escalation may require more sophisticated payload construction than simple text instructions.Source: §2 · p. 2
  • Objective-drift detection is forensic rather than preventive in this evaluation, with leave-one-scenario-out AUC 0.39–0.57; reader results raise but do not settle a writing-versus-reading question.The cited resistance comparison is between 0/40 raw pre-seeded injections and 2/3 relayed injections.Source: §2 · pp. 2–3

3 Benchmark Design

The benchmark uses a two-agent harness and a canary-based kill chain to track injected content across attack surfaces, models, and defenses. Its scenarios and measurements distinguish where the payload enters and how far it travels.

3.1 System Architecture

  • The roughly 600-line agent_bench harness centers on a MemoryStore, ToolRegistry, MultiAgentOrchestrator, and PropagationLogger.The logger matches canary tokens against tool calls and emits stage-labeled JSONL events.Source: §3.1 · p. 3

3.2 Kill-Chain Stages and Attack Scenarios

  • The threat model gives a read-only adversary control of processed content, aiming for an unauthorized outbound action while the legitimate task succeeds.Runs are labeled by the furthest canary stage: Exposed, Persisted, Relayed, or Executed; stage gaps locate filtering or refusal.Source: §3.2 · p. 3
  • Scenarios cover web pages, pre-seeded memory, tool results, visible and white-text PDFs, and audio transcripts, with distinct task wording and payload formats.Propagation and PDF relay use two-agent memory paths; tool_poison is single-agent, and permission_esc passes a delegation message.Source: §3.2 · p. 3

3.3 Models, Defenses, and Metrics

  • The study reports 950 runs across five production LLMs, including 764 text-surface runs and 186 PDF-relay runs.Text-surface cells have n = 8–36; PDF-relay cells have n = 3, with same-model and cross-model pairs.Source: §3.3 · p. 4
  • Five conditions compare no defense with write_filter, pi_detector, spotlighting, and all three combined; ASR measures execution while task success tracks the legitimate goal.Table 1 reports text-surface attack and task success per model; ASR uses no-defense attacked runs, while task success uses clean-control runs.Source: §3.3 · p. 4, Table 1 · p. 5

4 Results

Exposure was universal when an attacked run called its tool, but downstream outcomes varied: Claude executed none of its text-surface attacks, while GPT-4o-mini had 53% ASR. Stage-level results also reveal channel-dependent outcomes, defense coverage failures, and objective drift that appears at the harmful step rather than beforehand.

4.2 Where the Canary Token Stops

  • 100% of attacked text-surface runs that called a tool were exposed, and the canary never appeared in a Claude memory write in 0/40 text-relay runs.GPT-5-mini reached a memory write in 3/20 runs (15%).Source: §4.1 · p. 5, §4.2 · p. 5

4.1 Exposure: 100% Among Runs That Called the Tool

  • Claude Haiku 4.5 and Claude Sonnet 4.5 executed 0/80 and 0/84 text-surface attacks, respectively, while GPT-4o-mini had 53% ASR.Claude reached 100% task success on clean runs; these results apply only to the evaluated scenarios and tasks.Source: §4.1 · p. 5

4.3 Attack Success by Surface

  • DeepSeek Chat had 0/24 ASR on memory_poison and 8/8 on tool_poison and propagation, but those scenarios also differ in task wording and payload format.GPT-4o-mini achieved 100% ASR on all three non-privilege-escalation scenarios, while Claude achieved 0% on all four evaluated scenarios; the permission_esc near-zero results may reflect payload design.Source: §4.3 · p. 5

4.4 Defense Failures

  • No defense lowered GPT-4o-mini or DeepSeek Chat attack success on text relay or tool_poison; pi_detector and write_filter missed channels they do not inspect, and spotlighting failed on wrapped content.write_filter blocked PDF relay but not text relay, a difference the authors could not explain; spotlighting lowered GPT-4o-mini task success without lowering attack success.Source: §4.4 · pp. 5–6

4.5 Objective Drift as a Post-Hoc Forensic Signal

  • Steps 1–2 of GPT-4o-mini memory_poison runs stayed nearly identical to the task description (∆≤0.033), with drift diverging only at the harmful step.The classifier’s leave-one-scenario-out AUC was 0.39–0.57, and objective drift is a forensic rather than preventive signal.Source: §4.5 · p. 6

4.6 Provenance Attribution and Dual-Completion

Provenance tracing distinguishes how injections travel through a multi-agent chain, while execution lag and PDF relay results show that downstream outcomes vary by model, channel, and defense. The PDF relay findings are directional because each cell has only three runs.

4.6 Provenance Attribution and Dual-Completion

  • 22/22 compromised runs in the first collection batch had their injection paths reconstructed with 0 false attributions; later-batch compromised runs were not traced.The token-overlap heuristic matched outgoing tool-argument tokens against prior tool results and memory state.Source: §4.6 · pp. 6–7
  • A legitimate report and a later canary-bearing exfiltration call can both occur in one run, so outcome-level monitoring can miss the attack.Both calls’ arguments traced to the same read_memory result; the paper identifies step-level provenance logging as the minimum forensic capability for detecting both events.Source: §4.6 · p. 7

4.7 Execution Lag and Detection Feasibility

  • GPT-4o-mini’s median exposure-to-execution lag was 1 step, while DeepSeek Chat’s lag was bimodal and reached 12 steps.A detector tuned to GPT-4o-mini’s pattern could miss DeepSeek Chat’s late executions; compromised GPT-4o-mini runs also averaged 1.26 send_report calls versus 0.57 clean, a difference of +0.69.Source: §4.7 · p. 7

4.8 PDF Relay Experiment

  • 186 PDF relay trials across five models used three runs per cell, so the reported results are directional and have wide confidence intervals.The relay tracks extraction, memory write, delegation, and execution; audio was tested only in a separate pilot.Source: §4.8 · p. 7
  • Claude Haiku 4.5 and Sonnet 4.5 omitted the canary from every same-model PDF relay memory write, while DeepSeek Chat propagated every injection through execution.Claude: 0/6 memory writes for each model; DeepSeek Chat: 3/3 per variant.Source: §4.8 · p. 7, Table 4 · p. 8
  • White PDF text succeeded at least as often as visible appended text: 1/3 versus 0/3 for GPT-4o-mini and 3/3 for both variants on DeepSeek Chat.The white text was extracted before the benign abstract, a possible explanation for the GPT-4o-mini difference; DeepSeek Chat showed no positional sensitivity.Source: §4.8 · p. 8
  • write_filter blocked every tested PDF relay cell but did not block the text relay, a difference the authors could not explain.pi_detector did not block the tested PDF cases, consistent with its scanning outgoing tool queries while the PDF payload arrived in parsed document content.Source: §4.8 · p. 8
  • In cross-model PDF relay, Claude Haiku 4.5 executed 2/3 injections from a GPT-4o-mini writer, whereas a Claude writer with a GPT-4o-mini reader produced 0/3 executions.The latter pairing also had no canary in shared memory; other readers of a Claude writer were not tested. GPT-4o-mini as both agents relayed every canary but executed none for pdf_append, while DeepSeek Chat as writer yielded 3/3 executions by GPT-4o-mini.Source: §4.8 · pp. 8–9

4.9 Audio Injection and PDF Metadata Pilots

Small pilots tested audio instructions delivered through transcription and payloads placed in PDF metadata. Both surfaces produced null attack-success results at the tested sample sizes.

4.9 Audio Injection and PDF Metadata Pilots

  • Audio injection produced 0% ASR across two tested models (n = 4 each), while PDF metadata injection produced 0% ASR across three models (n = 4 per model), at this sample size.Source: §4.9 · p. 9
  • For audio injection, the transcription tool returned spoken words as narration, and neither model treated the transcribed text as an executable directive.The authors say this null result is consistent with prior multimodal injection findings [13] and suggest transcription may contextualize speech as data rather than instruction.Source: §4.9 · p. 9
  • For PDF metadata injection, pypdf extracted the /Keywords and /Subject fields, but the models did not parse those strings as actionable directives.The payload was placed in the PDF metadata fields.Source: §4.9 · p. 9

5 Discussion

Across the evaluated settings, the discussion emphasizes that injection propagation and apparent safety depend on the model’s position and the channels tested. The authors frame these patterns as benchmark-scoped and propose reporting stage- and channel-specific measures.

5 Discussion

  • Claude models executed none of 164 text-surface attacks and their canary token was absent from all reported relay memory writes, but Claude Haiku 4.5 executed 2/3 injections relayed by GPT-4o-mini.Whether relayed injections are harder to refuse than raw pre-seeded injections remains open because the settings differ and the relay sample is small.Source: §5 · p. 9
  • A Claude writer kept the token out of shared memory in every reported relay run; one tested GPT-4o-mini reader pairing then had 0/3 executions, while other readers were not tested.The authors suggest reporting the rate at which the canary token is absent from memory writes as a separate metric.Source: §5 · p. 9
  • DeepSeek Chat’s 0/24 memory-poison versus 8/8 tool-poison results cannot isolate a channel effect because task wording and payload format also differed.The authors argue evaluations should enumerate covered channels; visual screening would miss white-text PDF payloads, so checks need to operate on parser-extracted text.Source: §5 · p. 9

6 Broader Impact

The paper reports sandboxed security experiments and publicly releases its methodology and results, while acknowledging a dual-use risk from channel-mismatch analysis.

6 Broader Impact

  • All experiments used closed, sandboxed harnesses without external network access; no real systems were compromised and no sensitive data was used.The authors say the payloads and attack surfaces are representative of techniques already documented in public literature [3].Source: §6 · p. 9
  • The authors identify a dual-use risk: channel-mismatch analysis could inform more targeted attack design.They assess that the defensive contribution outweighs this risk because the described attack techniques are already publicly known.Source: §6 · p. 9
  • The authors release benchmark code and run logs, arguing that transparent evaluation methods and empirical results benefit reproducible security assessment.The release is presented as a community benefit in the paper’s broader-impact discussion.Source: §6 · p. 9

7 Limitations

The study’s findings are bounded by small PDF-relay samples, one task family, and measures that track canary tokens rather than every possible surviving instruction. Its explicit payloads, model and defense selection, and unresolved questions further limit generalization.

Sample size

  • PDF-relay cells had n = 3 and wide Wilson CIs, while the strongest text-surface results were consistent across independent batches.The reported interval for Claude Haiku 4.5’s 2/3 reader result was 21–94%.Source: §7 · p. 9

Single task family

  • All scenarios use one task family, so findings may not generalize to different tool sequences, longer horizons, or outbound action types.The paper identifies this as its primary threat to external validity.Source: §7 · pp. 9–10

Measurement limits

  • Canary tracking can miss paraphrased instructions that survive without the token, and refusal explanations quoting the token could be mislabeled Executed.Only 22 compromised runs from the first batch were manually checked for this false positive; later batches were not inspected, and the confound is not systematically controlled.Source: §7 · p. 10

Evaluation conditions

  • Payloads are explicit and unobfuscated, and model runs use different mixes of collection batches while table columns cover different run sets.The authors characterize channel-mismatch results as likely conservative for obfuscated off-channel attacks.Source: §7 · p. 10

Model and defense scope

  • The evaluation covers five production models and lightweight wrapper defenses, omitting reasoning and open-weight models; Claude’s memory behavior’s mechanism is unresolved.Black-box evaluation also cannot determine whether GPT-5-mini’s zero PDF-relay tool-call rate reflects capability regression or API schema incompatibility.Source: §7 · p. 10

Open questions

  • Open questions include whether relayed injections bypass readers that refuse raw ones, whether Claude writers protect other readers, and why write_filter treats PDF and text relays differently.The paper also asks whether audio and metadata payloads designed to resemble system instructions would succeed.Source: §7 · p. 10

8 Conclusion

Across 950 runs, stage-level canaries distinguish where injections stop from whether they succeed. The results vary by model position, channel, and defense coverage, with several cross-scenario explanations remaining unresolved.

Conclusion

  • Across 950 runs, Claude Haiku 4.5 and Claude Sonnet 4.5 executed none of 164 text-surface attacks, while GPT-4o-mini executed 53% of its attacks.Exposure was 100% among runs that called the tool; downstream outcomes differed.Source: §8 · p. 10

Finding 1: writer position

  • A Claude writer kept the canary out of shared memory in every reported relay run; one Claude-writer/GPT-4o-mini-reader pairing (n = 3) is consistent with reader protection.Other cross-model pairings were not tested, so general protection remains open.Source: §8 · p. 10

Finding 2: reader position

  • As readers, Claude models executed 0/40 raw pre-seeded injections, while Claude Haiku 4.5 executed 2/3 injections relayed by GPT-4o-mini.The authors leave open whether relayed injections are harder to refuse than raw ones.Source: §8 · p. 10

Finding 3: channel dependence

  • DeepSeek Chat’s result shifted from 0/24 on pre-seeded memory to 8/8 on tool results, but task and payload format also differed.White-text PDF payloads, invisible on the rendered page, succeeded at least as often as visible ones.Source: §8 · p. 10

Finding 4: defense coverage

  • pi_detector and write_filter failed on channels they do not inspect, while spotlighting failed on wrapped content; write_filter blocked PDF but not text relay.The difference between the two relay outcomes is unexplained; the authors recommend reporting stages, multiple channels, cross-model pairs, and writer and reader results separately.Source: §8 · p. 10

A Supplementary Results

The supplementary material points readers to text-surface attack counts and objective-drift results, while the appendix table specifies its count coverage.

Supplementary results

Table A1

  • Table A1 lists text-surface attack success counts per model and scenario, including runs that made no tool call.The permission_esc total is 2/132: DeepSeek Chat 1/28 and GPT-5-mini 1/36.Source: Table A1 · p. 12

Figure A1

  • Figure A1 collects objective-drift results from Section 4.5, including one clean and one attacked GPT-4o-mini memory_poison run.Its classifier panel reports 21 features, 5-fold CV, AUC 0.853, and the 14 largest feature importances.Source: Figure A1 · p. 12

Related papers

Loading 2603.28013v4…