PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt Injection Detection for Email Agents Through Attack Chain Modeling

Ahmad Hashmi, Dhyey Patel, Yunting Yin

arXiv:2609.30657v1cs.CRcs.CL

TL;DR

Email assistants can retrieve untrusted instructions that influence later tool use, while binary malicious-text classification misses the sequence of stages leading to harmful actions. The paper models that chain with text detection, stage verifiers, risk rules, and user-intent consistency checks; its results show stage-dependent robustness and improved mean F1 when action and context signals are added.

  • Problem

    Existing detectors frame prompt injection as binary text classification, although harmful email-agent actions unfold across retrieval, defense bypass, tool use, and unsafe arguments.

    Source: §I · p. 1

  • Method

    The framework combines a text detector, stage-specific verifiers, rule-based risk signals, and user-intent and action-consistency checks in a learned logistic meta-classifier.

    Source: §I · p. 1, B. Proposed Framework · p. 3

  • Results

    Random splits substantially overestimate robustness, attack-chain stages differ in difficulty, and adding action and context signals raises mean F1 from 0.348 to 0.406.

    Source: §I · p. 1, §IX · p. 7

  • Takeaways & Limitations

    The findings support layered defenses that validate later tool calls and arguments and evaluate detectors with benign hard negatives and low-false-positive action-consistency checks.

    Source: §VIII · pp. 6–7

    The framework has yet to be evaluated within a live email-agent pipeline, and future work may expand the diversity and scale of benign hard negatives.

    Source: §IX · p. 7

Abstract

from arXiv · show

Large language model email assistants are particularly vulnerable to indirect prompt injection because untrusted email content can be retrieved into the model context and influence subsequent tool use. Existing prompt injection detectors mainly formulate this problem as binary malicious text classification, which overlooks the important factor that harmful agent behavior often arises through a sequence of stages. We propose a detection framework that models this attack chain by combining a text detector, verifiers specific to each stage, explicit rule-based risk signals, user intent and action consistency analysis, and a logistic decision policy. To support this framework, we derive attack chain labels from prompt injection datasets, evaluate the proposed framework under random splits, temporal phase transfer, conditional stage transfer, cross-dataset transfer, and conduct ablation studies on multiple benchmarks. Results show that random train test splits substantially overestimate robustness under distribution shift, while later tool argument stages are more predictable than earlier stages in the framework. We also show that training on harmless emails that resemble attacks helps reduce false alarms while preserving the ability to detect real attacks. Across five binary benchmarks, our framework achieves a mean F1 score of 0.406 under the strict threshold setting policy, compared with 0.216 for the strongest of five pretrained detectors evaluated without additional training. These results highlight the value of combining attack stage predictions with checks for conflicts between the user's request and instructions in retrieved emails. Our experiments also demonstrate the importance of training with challenging benign examples to balance attack detection and false alarms.

I. INTRODUCTION

Retrieved email content can steer an assistant toward actions that conflict with the user’s intent, while binary text classification obscures the stages involved. The paper proposes modeling those stages alongside text detection and action-consistency checks.

I. INTRODUCTION

  • Harmful email-agent behavior can unfold through retrieval, defense bypass, tool invocation, and unsafe tool arguments, which binary classification does not distinguish.Random train-test splits may also overestimate robustness when future attacks differ from training data.Source: §I · p. 1
  • The framework combines a text detector, stage verifiers, rule-based risk signals, and user-intent and action-consistency analysis in a logistic meta-classifier.It derives labels for retrieval, defense bypass, tool invocation, tool-argument correctness, and end-to-end success.Source: §I · p. 1

A. Indirect Prompt Injection Benchmarks for LLM Agents

  • BIPIA, Tensor Trust [5], and NVIDIA Agentic IPI cover external-content and agent attacks, while these benchmarks primarily evaluate attack success as a binary outcome.Source: A. Indirect Prompt Injection… · p. 2

B. Prompt Injection Detection Models

Prior work includes classifiers for malicious inputs, methods targeting false alarms, and defenses for tool-using agents. The paper frames email-agent security around preventing untrusted content from causing unsafe actions while distinguishing attack stages.

B. Prompt Injection Detection Models

  • Ji et al. combine DeBERTa-v3-base semantic embeddings with heuristic features to classify prompt injections, reporting higher accuracy and lower attack success than several baselines.Source: B. Prompt Injection Detection Models · p. 2
  • InjecGuard [10] addresses over-defense, where benign trigger-like text is flagged as malicious, and introduces NotInject for evaluating false positives.Its model generates benign examples around biased trigger tokens to learn intent rather than react only to suspicious words.Source: B. Prompt Injection Detection Models · p. 2

C. Defenses for Tool-Using Agents

  • AgentDojo covers 97 tasks and 629 security cases, while InjecAgent reports GPT-4 attack success rising from 24% to 47% with an added hacking prompt.Source: C. Defenses for Tool-Using Agents · p. 2
  • Structural defenses such as StruQ and CaMeL, [14] emphasize instruction-data separation and control-flow tracking, inspiring the framework’s intent-consistency and action-risk checks.Source: C. Defenses for Tool-Using Agents · p. 2

III. PROBLEM FORMULATION

  • The threat model treats the user request and retrieved email content as inputs, with the security goal of preventing untrusted content from causing an unsafe email action.The detector is intended to distinguish attack stages, avoid blocking benign security-related text, and generalize across benchmarks.Source: §III · p. 2

A. Attack Stages

  • Binary labels miss whether an assistant retrieved an attack, bypassed a defense, called a tool, or used attacker-selected arguments.Injection language may fail to cause a tool call, while ordinary-looking scheduling text can succeed at the right conversation boundary.Source: A. Attack Stages · p. 2

B. Distribution Shift

Random splits can preserve scenario templates, so the paper uses LLMail-Inject’s two phases to examine transfer from Phase 1 to Phase 2. It also motivates testing benign hard negatives because suspicious-looking text may be harmless.

B. Distribution Shift

  • Random train-test splits may preserve scenario templates and distributions, letting detectors learn surface regularities rather than robust attack risk.LLMail-Inject’s two phases permit testing whether Phase 1 thresholds and rankings generalize to Phase 2.Source: B. Distribution Shift · p. 2

C. Benign Hard Negatives

  • Benign hard negatives can contain trigger words that resemble injection, so evaluation on attacks and ordinary benign text alone may hide overblocking.The paper tests whether explicit hard-negative training reduces this issue when combining email attacks with BIPIA indirect attacks.Source: C. Benign Hard Negatives · p. 3

A. LLMail-Inject

  • LLMail-Inject objective fields are parsed into six labels: retrieval, defense bypass, tool call, destination correctness, content correctness, and full success.Source: A. LLMail-Inject · p. 3
  • Full success was 0.814% in Phase 1 and 0.337% in Phase 2, illustrating why high accuracy is easier to achieve than strong precision and recall.Source: A. LLMail-Inject · p. 3

B. Evaluation Datasets

Evaluation spans attack, benign hard-negative, and multiple detector datasets, with the stated goal of preserving attack recall under distribution shift while limiting false positives. The text baseline is a logistic classifier, with handcrafted features available as explicit risk channels.

B. Evaluation Datasets

  • BIPIA supports cross-dataset transfer, while NotInject [10] supplies benign examples with suspicious or prompt-like wording for measuring overblocking.Source: B. Evaluation Datasets · p. 3
  • Evaluation also uses attack-only NVIDIA Agentic IPI, PromptShield, ShieldLM, and Neuralchemy, which contribute datasets with different row counts and label categories.NVIDIA Agentic IPI contains 1,272 rows across nine domains and 40 tools; the other datasets span binary detector rows and multiple attack or benign categories.Source: B. Evaluation Datasets · p. 3

V. FRAMEWORK AND EXPERIMENTAL DESIGN

  • The experiments test attack recall under distribution shift and false positives on benign hard negatives, comparing baselines and the proposed framework on held-out datasets and thresholds.Source: §V · p. 3

A. Text Detectors

  • The main text detector is a linear logistic classifier trained with SGD and TF-IDF word n-grams; handcrafted security features serve as explicit risk channels in the framework.The classifier uses an L2 penalty, balanced class weights, and validation-tuned thresholds chosen to maximize F1.Source: A. Text Detectors · p. 3

B. Proposed Framework

The framework combines text detection, attack-stage verification, explicit rules, and checks for consistency between user intent and retrieved content. A logistic meta-policy produces a risk score used at review-oriented or strict thresholds.

B. Proposed Framework

  • Six components combine text detection, stage verifiers, selective handling of the sparse full-success verifier, action-context checks, rules, and logistic stacking.The consistency checks compare the user request with retrieved content, including recipient introductions and requests for tool actions not requested by the user.Source: B. Proposed Framework · p. 3
  • The framework outputs one risk score, with validation-F1 and NotInject-based thresholds for review-oriented screening and stricter block-or-confirm decisions.The review configuration excludes the full-success verifier; under the strict policy, all stage verifiers plus action-context features achieve the highest observed mean F1.Source: B. Proposed Framework · p. 4
  • The study compares five pretrained text-classification detectors and a zero-shot gpt-4.1-mini judge against the framework on shared evaluation datasets.The judge is evaluated at its default 0.50 threshold and low-FPR thresholds derived from NotInject.Source: B. Proposed Framework · p. 4

C. Evaluation Protocols and Metrics

The evaluation tests random-split performance, transfer across time phases and datasets, conditional stage transitions, component contributions, and detector baselines. It reports F1, precision, recall, ROC-AUC, and false-positive rate.

C. Evaluation Protocols and Metrics

  • The study evaluates five protocols: stratified random train/validation/test splits; Phase 1-to-Phase 2 transfer; conditional stage modeling; cross-dataset transfer; and proposed-framework comparison against a baseline using word-, character-, and rule-based features.Conditional transitions are evaluated only among examples where the prerequisite stage occurred. The framework comparison uses both the validation-F1 threshold and the stricter NotInject threshold.Source: C. Evaluation Protocols and Metrics · p. 4
  • Framework comparisons test stage- and action-based features against word- and character-level text and rule-based features at validation-F1 and stricter NotInject thresholds.Ablations compare text/rule-only, all-stage, selective-stage, action-context-only, and combined variants.Source: C. Evaluation Protocols and Metrics · p. 4
  • The detector baseline protocol evaluates accessible Hugging Face prompt-injection classifiers, while the LLM-judge protocol uses zero-shot gpt-4.1-mini on full test sets.The judge uses the default 0.50 threshold and low-FPR thresholds derived from NotInject; reported metrics include F1, precision, recall, ROC-AUC, and false-positive rate.Source: C. Evaluation Protocols and Metrics · p. 4

VI. RESULTS

Attack-chain outcomes and transfer results show that performance varies substantially by stage and evaluation split. Conditional tool-argument predictions transfer more reliably than predictions of whether bypassed content will trigger a tool call.

A. Attack Chain Stage Analysis

  • Full-success F1 falls from 0.167 on random splits to 0.000 under Phase 1-to-Phase 2 threshold transfer, while tool-call F1 falls from 0.527 to 0.077.Even with an oracle Phase 2 threshold, full-success F1 is 0.024; defense-bypass prediction is more stable but still degrades.Source: A. Attack Chain Stage Analysis · p. 5, Table III · p. 5
  • Retrieved-but-detected submissions account for 49.66% of rows, while bypassed submissions without a tool call account for 46.80%; only 0.72% are full successes.Among failures after a tool call, wrong content (0.71%) is more common than wrong destination (0.27%).Source: A. Attack Chain Stage Analysis · pp. 4–5
  • Binary attack labeling misses that many messages reach an intermediate stage without completing a harmful action.Stage-specific outcomes expose this distinction between attack-like content and end-to-end success.Source: A. Attack Chain Stage Analysis · p. 5
  • Tool-call-to-destination F1 remains 0.939 and tool-call-to-content F1 remains 0.814 under phase transfer, versus bypass-to-tool-call F1 falling from 0.394 to 0.053.Once a tool call occurs, destination and content correctness are more stable to predict than whether bypassed text triggers a tool call.Source: A. Attack Chain Stage Analysis · p. 5, Table IV · p. 5

B. Reducing False Alarms with Benign Training Examples

Cross-dataset results show weak transfer from LLMail binary proxies to BIPIA, while adding benign hard negatives improves the framework’s strict-policy results in the reported rotation. Threshold adjustment alone reduces attack recall substantially.

B. Reducing False Alarms with Benign Training Examples

  • 0.770 BIPIA test F1 is achieved by combining LLMail with BIPIA, compared with 0.738 for BIPIA in-domain detection and 0.453 for an LLMail-only binary proxy.The LLMail proxy’s benign side contains only 203 false-positive test emails.Source: B. Reducing False Alarms with Benign… · p. 5, Table V · p. 5
  • 36.9% recall on BIPIA attacks remains for the LLMail detector after threshold adjustment reduces NotInject false-positive rate to approximately 5%.The passage attributes the trade-off to substantial overlap between attack and benign hard-negative score distributions.Source: B. Reducing False Alarms with Benign… · p. 5
  • 0.451 mean F1 with zero false positives among 113 held-out NotInject examples is achieved when NotInject examples are included in training, versus 0.038 mean F1 without them.Without NotInject training examples, 6 of 113 benign examples are incorrectly flagged (5.31%).Source: B. Reducing False Alarms with Benign… · p. 5

C. Component Ablation

The framework outperforms five frozen detectors on matched strict mean F1 and BIPIA F1, although some baselines do better on false positives or individual datasets. In the ablation, action and context checks improve BIPIA results, while the strict all-stage variant has the highest mean F1 by a small margin.

C. Component Ablation

  • 0.406 is the highest mean F1 for the strict all-stage action-and-context variant, narrowly above 0.400 for selective stages plus action and context.The difference is small relative to variability.Source: C. Component Ablation · p. 5
  • BIPIA F1 rises from 0.183 for all stages without action and context to 0.733 for selective stages plus action and context.The selective action-and-context policy has 8/339 NotInject false positives (2.36%) and 2.33% NVIDIA recall; the histogram gradient-boosted stacker reaches 0.347 mean F1 overall.Source: C. Component Ablation · pp. 5–6

D. Comparison with Existing Detectors

E. Comparison with a Zero-Shot OpenAI LLM Judge

Against the zero-shot judge, the framework has higher BIPIA F1 but lower mean F1. Padding and wording tests expose vulnerabilities and generalization differences, while error analysis identifies missed boundary attacks and benign text that triggers false alarms.

E. Comparison with a Zero-Shot OpenAI LLM Judge

  • 0.732 versus 0.625 BIPIA F1 favors the framework over the zero-shot judge, but the judge achieves higher mean F1 and performs better on LLMail and NVIDIA.Observed NotInject false positives are similar: 7/339 for the framework versus 6/339 for the judge.Source: E. Comparison with a Zero-Shot OpenAI… · p. 6
  • Locally run sparse features, classifiers, rules, and a logistic meta-policy avoid per-input hosted LLM inference calls.The BIPIA results show useful detection performance without invoking a generative LLM.Source: E. Comparison with a Zero-Shot OpenAI… · p. 6

F. Robustness to Padding and Wording Modifications

  • Repeating benign words with strongly negative learned coefficients can reduce strict-policy recall, unlike random benign padding, which does not reduce recall in these tests.Lowest-risk benign-email padding has only a modest effect.Source: F. Robustness to Padding and Wording… · p. 6
  • 0.928 to 0.916 is the selective stage framework’s recall change after benchmark-specific strings are replaced, compared with a drop from 0.824 to 0.554 for the text-and-rule-only model.The authors suggest relationship checks among the user request, email text, and agent output are less sensitive to those strings.Source: F. Robustness to Padding and Wording… · p. 6

VII. QUALITATIVE ERROR ANALYSIS

  • 50% of sampled Phase 2 full-success errors are high-risk near misses, while 44% are missed conversation-boundary attacks, especially meeting confirmations resembling normal scheduling.Source: §VII · p. 6
  • NotInject false positives most often involve benign technical instructions (30%), security or platform discussion (26%), polysemous trigger words (20%), and creative or roleplay prompts (16%).The analysis describes confusion between text discussing instructions or security and operational attacks.Source: §VII · p. 6

VIII. DISCUSSION

The discussion advocates layered detection, with tool-call and argument validation complementing brittle early-stage detection and benign hard negatives informing training and evaluation. The conclusion reports distribution-shift concerns and argues for moving beyond binary classification toward attack-chain reasoning.

VIII. DISCUSSION

  • Early-stage detection of untrusted instructions remains necessary but brittle; tool-call and argument validation matter because later-stage labels are more learnable after a tool call.At low false-positive thresholds, action-and-context consistency checks are especially useful, and future data collection should prioritize tool action, recipient, and content outcomes.Source: §VIII · pp. 6–7

IX. CONCLUSION

  • Random train-test splits can substantially overestimate detector performance, and benign hard negatives are essential for real-world evaluation.Source: §IX · p. 7
  • Selective stage modeling and action-and-context consistency provide the largest improvements, particularly when false positives must remain low.The framework combines text detection, stage verification, rule-based risk signals, and user-intent/action consistency analysis in a learned meta-classifier.Source: §IX · p. 7
  • The paper identifies live email-agent evaluation and broader, larger benign hard-negative sets as future directions.Source: §IX · p. 7

Related papers

Loading 2609.30657v1…