Source-linked AI summary
Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents
Zhuowen Liu
TL;DR
Benchmark scores do not reliably predict how prompt-injection detectors behave on agent tool outputs, where false alarms can block tasks. The paper replays tool calls from two agent benchmarks and evaluates detectors and judges on those outputs and BIPIA. Detection rankings transfer poorly across benchmarks, while false-positive behavior transfers between the agent benchmarks and training-input form helps explain detector performance.
Problem
It is unclear whether detectors’ public-benchmark scores predict their behavior on serialized tool outputs screened by agents, where false alarms can block tasks.
Source: §1 · p. 1Method
The study replays ground-truth calls from AgentDojo and τ-bench to create benign outputs, labels injected outputs by differential replay, and evaluates fifteen detectors and two judges on these outputs and BIPIA.
Source: §1 · p. 1Results
Detection rankings transfer poorly between benchmarks, while false-positive rates on tool outputs are consistent across the two agent benchmarks (τ = 0.67).
Source: §1 · p. 2Takeaways & Limitations
Evaluations meant to guide deployment should use the agent’s own tool outputs, report detection at a low false-positive rate, and audit detector training data.
Source: §6 · p. 7, §8 · p. 9Real tool outputs such as HTML pages, API responses, and long documents are messier than the evaluated outputs, so absolute deployment rates may differ.
Source: §6 · p. 8
Abstract
from arXiv · showhide
LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including Meta's Prompt Guard 2, and two task-aware LLM judges on these outputs and on the BIPIA benchmark. Detection rankings transfer poorly between benchmarks: the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench. False-positive rates on tool outputs, which range from none to over 90%, do transfer between the two agent benchmarks. Where training data is public, the form of the training inputs explains the results. The BIPIA leader was trained on full BIPIA inputs, but having seen InjecAgent's attack strings as short prompts does not help it find them inside tool outputs; the best detector on both agent benchmarks shares no data with any benchmark and was trained on agent-style inputs. Evaluations meant to inform deployment should use the agent's own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on.
1 Introduction
The paper asks whether public benchmark scores predict detector behavior in agents, where serialized tool outputs—not prompt collections—are screened and false alarms can block tasks. Its replay-based evaluation finds poor detection-ranking transfer across benchmarks and links performance to the form of detector training inputs.
1 Introduction
- False alarms can block an agent’s task, while missed injected text may redirect the agent’s actions.The detector screens tool outputs before the agent sees them, making both errors consequential in deployment.Source: §1 · p. 1
- The study evaluates fifteen detectors and two task-aware LLM judges on AgentDojo, τ-bench, and BIPIA to test whether benchmark scores predict performance inside agents.The authors’ central observation is that detectors work on inputs resembling their training inputs, and benchmark scores mostly measure that resemblance rather than general injection-detection ability.Source: §1 · p. 1
- The evaluation replays ground-truth tool calls without an LLM to produce benign outputs, labels injected outputs by differential replay, and audits detector training data.Substring matching fails when AgentDojo re-wraps injected text during serialization; the audit also compares published training data with benchmark inputs and InjecAgent attacks.Source: §1 · pp. 1–2
- PIGuard catches 95.1% of BIPIA injections at 1% FPR but 2.1% of AgentDojo injections, while Prismor catches 72.2% on AgentDojo and 15.2% on τ-bench.Detection rankings transfer poorly: Kendall’s τ is 0.01 between BIPIA and AgentDojo and 0.28 between the two agent benchmarks; false-positive rates transfer more consistently (τ = 0.67).Source: §1 · p. 2
- PIGuard was trained on full BIPIA inputs, while seeing InjecAgent attacks as short prompts did not improve its detection of them inside τ-bench tool outputs.Horizon-Labs shares no data with any benchmark, was trained on agent-style inputs, and leads both agent benchmarks with almost no false alarms.Source: §1 · p. 2
- The contributions include a replay-based evaluation method, cross-benchmark measurements, training-data audits, and analyses of task awareness, attack goals, and input format.The analyses find that task-irrelevant injections evade every approach outside its training distribution.Source: §1 · p. 2
2 Background and Motivating Example
Detectors screen untrusted tool text because it can redirect an agent, but existing benchmark scores may not reflect behavior on serialized tool outputs. The motivating example shows that keeping false alarms rare can leave injections undetected.
2 Background and Motivating Example
- Detectors are an easy-to-adopt defense because they classify untrusted text and drop what they flag without changing the agent or model.Indirect prompt injection occurs when text returned by a tool redirects the model as instructions.Source: §2 · p. 2
- Figure 1 contrasts a harmless password confirmation blocked by the detector with an injected calendar description that passes once its threshold is set to keep false alarms rare.The example is from an AgentDojo banking task; a flagged benign output can make the task fail.Source: Figure 1 · p. 2, §2 · p. 2
- Existing evaluations use prompt collections and injected documents, but agents screen serialized tool outputs and those benchmarks are not checked for overlap with detector training data.False alarms on tool outputs can block tasks, unlike an error percentage on a balanced test set.Source: §2 · p. 2
- At a 1% false-alarm rate, almost every injected AgentDojo output passes the detector, although the same PIGuard model catches nearly every BIPIA injection at that rate.At default thresholds, the detectors catch many AgentDojo injections but also falsely flag a fifth of benign tool outputs.Source: §2 · p. 2
- Thresholds trade false alarms against missed injections, and the study tests whether either task-aware judges or detectors can achieve useful detection at low false-positive rates.A judge’s verdict depends on the question it is asked; the paper tests whether that question covers different attack goals.Source: §2 · pp. 2–3
3 Deployment Setting and Threat Model
The deployment model places a detector between tools and an agent, with flagged outputs dropped or escalated and any flagged benign output counted as a blocked task. The threat model limits attackers to controlling returned data and evaluates static benchmark attacks.
3 Deployment Setting and Threat Model
- Any task with at least one flagged benign tool output counts as blocked, whether deployment drops the output or stops the task and asks a human.The detector screens every tool output before it enters the agent’s context.Source: §3 · p. 3
- Attackers control part of a tool’s returned data but cannot change the user request, agent, or detector; their goals may be action, task-irrelevant, or task-relevant.Examples include sending money, doing an unrelated task, or changing the answer format.Source: §3 · p. 3
- The detector’s goals are to pass benign tool outputs and flag injected outputs at an operating point that still satisfies the benign-output goal..Source: §3 · p. 3
- The evaluation covers static attacks shipped with public benchmarks; an adaptive attacker targeting a detector would lower detection rates, so reported detection numbers are upper bounds.The false-positive results do not depend on the attacker.Source: §3 · p. 3
4 Methodology
The methodology evaluates detectors and task-aware judges on replayed agent-benchmark outputs and BIPIA, measures detection at low false-positive rates, and audits publicly available training data.
4.1 Detectors and Judges
- Fifteen detectors span several model generations, and two instruction-tuned judges receive the user request and one tool output to assess whether the output asks for an unrequested action.The judges are Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct; the latter’s training data predates AgentDojo.Source: §4.1 · p. 3
4.2 Data
- AgentDojo clean outputs come from replaying ground-truth tool calls without injections or an LLM, yielding 339 benign outputs from 97 tasks.Matched and shifted sets add benchmark attacks and label outputs by differences from clean replay; τ-bench uses the same replay approach and adds InjecAgent instructions to product names.Source: §4.2 · pp. 3–4
- BIPIA samples use clean contexts for email, table, and code-repair tasks, with randomly inserted attack strings; the main results use test-half attacks.The resulting main evaluation set has 1,200 benign and 2,435 injected samples.Source: §4.2 · p. 4
4.3 Metrics
- The evaluation reports default-threshold FPR and TPR, AUROC, and TPR at 1% FPR, with the low-FPR operating point emphasized for deployments that see benign outputs more often.For AgentDojo clean data, it also reports the share of tasks with at least one flagged output; the 1%-FPR threshold is selected on the evaluated benign set.Source: §4.3 · p. 4
- Table 1 lists benign and injected sample counts for the evaluation sets, with the benign side of TB-Matched taken from TB-Clean.AD denotes AgentDojo v1.2 and TB denotes τ-bench.Source: Table 1 · p. 4
4.4 Training-Data Audit
- The training-data audit searches PIGuard’s released split and Horizon-Labs’ training sources for benchmark text and attack markers, including InjecAgent instructions.The searches find only verbatim copies and are lower bounds on overlap; other detectors’ sources are reported as stated in their model cards.Source: §4.4 · p. 4
5 Results
The results compare detector false alarms and injection detection across agent benchmarks, examine how training data and task awareness relate to performance, and test whether input format and metadata shift scores.
5.1 RQ1: False Positives on Benign Tool Outputs
- False-positive rates on benign tool outputs range from 0% to over 90%, and detector rankings transfer between AgentDojo and τ-bench (Kendall τ = 0.67).False-positive rates on ordinary text do not predict this behavior; ProtectAI v2 flags 0.5% of web pages and 1.2% of emails, versus 30.1% of AgentDojo and 66.9% of τ-bench outputs.Source: §5.1 · p. 5
5.2 RQ2: Detection at 1% FPR and Rank Transfer
- At 1% FPR, rankings transfer unreliably: between the agent benchmarks Kendall’s τ is 0.28 (p = 0.17), while Prismor detection falls from 72.2% on AgentDojo to 15.2% on τ-bench.Horizon-Labs and Wolf Defender lead both agent benchmarks.Source: §5.2 · pp. 5–6
5.3 RQ3: Training Data
- Where training data is public, performance tracks input form: PIGuard’s BIPIA training used full inputs, while Horizon-Labs leads both agent benchmarks without audited benchmark overlap.PIGuard detects 52.1% of τ-bench outputs with seen InjecAgent attacks versus 55.8% with unseen attacks; those attacks appeared in short prompts, not tool responses.Source: §5.3 · pp. 6–7
5.4 RQ4: Task Awareness and Attack Goals
- Task-aware 7–8B judges catch 54–78% of agent-tool-output injections at 1% FPR, below the best dedicated detectors; task-irrelevant attacks evade nearly all approaches.The judge prompt was not varied, so the authors cannot determine how much of the judges’ performance gap comes from the prompt.Source: §5.4 · p. 7
5.5 RQ5: Format and Metadata
- Changing serialization markup or input-type metadata can reduce default-threshold false positives by up to about twentyfold, but neither PIGuard nor ProtectAI v2 exceeds 15% detection at 1% FPR on AD-Matched after these changes.For example, declaring Sheltron’s source as tool_result lowers clean-output FPR from 85.6% to 4.4% on τ-bench.Source: §5.5 · p. 7
6 Discussion and Limitations
The discussion recommends evaluating detectors on the tool outputs agents actually screen, at a fixed low false-positive rate, while auditing training data. It cautions that simulated benchmarks, static attacks, limited detector access, and diagnostic-only judges constrain how results should be interpreted.
Choosing a detector for an agent
- Evaluate false positives on the agent’s own tool outputs, report detection at a fixed low FPR, audit training overlap, and prefer detectors trained on agent-like inputs.In this study, Horizon-Labs and Wolf Defender caught more than 70% of injections while keeping false positives near zero on both agent benchmarks; both were built for untrusted content reaching agents.Source: §6 · p. 7
Format effects
- Removing YAML keys lowered PIGuard’s AgentDojo false-positive rate threefold without changing its true-positive rate.The figure compares outputs as-is with outputs after keys are removed.Source: Figure 4 · p. 8
Choosing a detector for an agent
- Even agent-oriented detectors missed most task-irrelevant injections, so detectors should complement rather than replace system-level defenses [6].The paper identifies task-irrelevant injections as a remaining weakness for the leading detectors.Source: §6 · pp. 7–8
Simulated agent environments
- Because AgentDojo and τ-bench use synthetic records and databases, real-output formats may shift absolute false-positive rates.The authors expect relative findings about over-flagging and agent-oriented detectors to carry over, while noting that other injection placements could favor different detectors.Source: §6 · p. 8
Static attacks
Detector coverage
- The evaluation excludes closed commercial API detectors, and only PIGuard and Horizon-Labs publish enough data for training-set audits.For other detectors, the authors can report only what model cards state.Source: §6 · p. 8
The judges are a diagnostic, not a defense
- The task-aware judges are diagnostic rather than defenses: they use one prompt, were not tested against adaptive attacks, and the false-alarm cost of a broader prompt was not measured.The judges call a 7–8B model once per tool output.Source: §6 · p. 8
Verbatim overlap
- The overlap audit detects exact copies only, so paraphrases, shared-template attacks, and unpublished training data can escape it.For PIGuard and BIPIA, the authors state that verbatim overlap alone already explains the score.Source: §6 · p. 8
7 Related Work
The paper places its evaluation among work on detector metrics, ML-security pitfalls, detection defenses, benchmarks, and system-level defenses. It distinguishes its focus on released detectors screening actual agent tool outputs from approaches that alter the model or keep untrusted data out of planning.
Evaluation methodology for injection detectors
- Unlike prior detector evaluations, this study tests released detectors on agent-screened tool outputs, translates false positives into blocked tasks, and audits published training data against praised benchmarks.Related work includes leave-one-dataset-out evaluation [9], low-FPR evaluation [13], and over-defense measurement on benign prompts.Source: §7 · p. 8
Pitfalls in ML and LLM security research
- The study treats data leakage and metric pitfalls as concrete concerns for detectors, measuring overlap directly rather than inferring it; prior work also finds static-attack defenses fall to adaptive attacks [19].The cited surveys catalog pitfalls including data leakage and inappropriate metrics [8].Source: §7 · pp. 8–9
Detection-based defenses
- Other detection defenses include fine-tuned classifiers, adaptive-attack training, and alignment-based methods; this paper’s task-aware judges use a related intuition without training and serve only as diagnostics.The related methods include DataSentinel [17] and AlignSentinel [14].Source: §7 · p. 9
Benchmarks
8 Conclusion
Across fifteen detectors, two agent benchmarks, and BIPIA, detection rankings transfer poorly. Training-input form explains where detectors worked, while false-positive behavior on tool outputs was stable across the agent benchmarks.
Conclusion
- Detection rankings transferred poorly across benchmarks, while tool-output false-positive behavior was stable between agent benchmarks.The conclusion recommends replaying an agent’s own tool outputs, reporting detection at low FPR and task-level cost, and auditing what detectors have seen.Source: §8 · p. 9
Ethics Considerations
The authors use only public benchmarks and benign corpora, reuse published attack strings, and report no experiments against deployed systems or other people’s agents. Detector configurations follow model-card instructions, with specific windowing or truncation handling for long inputs.
Ethics Considerations
- The study uses public benchmarks and benign corpora, creates no new attacks, and runs no experiments against deployed systems or other people’s agents.Every injected sample uses an attack string already published by AgentDojo, BIPIA, or InjecAgent.Source: Ethics Considerations · p. 9
A Detector Configurations
- Long inputs are handled according to model-card configurations, including overlapping windows for Sheltron and 512-token segments for Prompt Guard models.For both windowed approaches, the evaluation takes the maximum score across windows; Prismor’s prompt is truncated at 512 tokens while preserving its template.Source: A Detector Configurations · p. 10
- The detector configuration list spans text-input classifiers, role/source-token and chat-template models, and segmented Prompt Guard variants, with context limits from 512 to 2048 tokens.Source: A Detector Configurations · p. 10
B Detailed Results for Four Methods
This section reports detection metrics for four methods across three matched sets and describes how the task-aware judges assess tool outputs.
B Detailed Results for Four Methods
- Table 4 reports AUROC, TPR at 1% FPR with bootstrap intervals, and default-threshold rates across the three matched sets.BIPIA results use test-half attacks.Source: B Detailed Results for Four Methods · p. 10, Table 4 · p. 11
C Judge Prompt
- The judges receive a user request and tool output, truncated to 6,000 characters, and score the first-token Yes/No distribution.The score is P(Yes)/(P(Yes) + P(No)), summed across case and whitespace variants.Source: C Judge Prompt · p. 10
- The quantized Qwen judge matches its bf16 version to within 0.02 AUROC on AD-Matched.Source: C Judge Prompt · p. 11
- Both judges assess whether tool data asks the assistant to perform unrequested actions, then reply exactly Yes or No.Source: C Judge Prompt · p. 11
D Excluded Detectors
Two detectors were excluded for excessive benign-output flags, and differential replay addresses a labeling failure caused by serialized injections.
D Excluded Detectors
- Both excluded classifiers flagged 100.0% of scored benign README chunks and more than three quarters of AD-Clean tool outputs.The fmops model also flagged two of five plain sentences, so both models were treated as unusable.Source: D Excluded Detectors · p. 11
E Labeling by Differential Replay
- Differential replay labels an output by comparing the same ground-truth call in clean and injected environments, avoiding missed YAML-rewrapped attacks.Verbatim substring matching labeled every output carrying the important_instructions attack as benign; it also silently inflated false-positive rates.Source: E Labeling by Differential Replay · p. 11
- Figure 5's Llama judge has a wide AD-Matched interval because a few high-scoring benign outputs destabilize the 1% threshold.Source: Figure 5 · p. 12