Source-linked AI summary
AgentWatcher: A Rule-based Prompt Injection Monitor
Yanting Wang, Wei Zou, Runpeng Geng, Jinyuan Jia
TL;DR
Prompt injection defenses struggle with long contexts and lack explicit, interpretable decision rules. AgentWatcher attributes an agent’s output to compact context segments, then applies a rule-based monitor LLM; across agent and long-context evaluations, it detects attacks while maintaining utility.
Problem
Prompt injection threatens LLM applications, while existing detection methods degrade with long contexts and lack explicit rules that make decisions explainable.
Method
AgentWatcher attributes an LLM output action to a small set of influential context segments and uses a monitor LLM to reason over explicit customizable rules.
Results
AgentWatcher reduces ASR to ≤1% on AgentDojo with a 2% utility loss and consistently reduces ASR to nearly zero across evaluated backbone LLMs with utility loss ≤4%.
Takeaways & Limitations
AgentWatcher provides a practical direction for robust, high-utility prompt-injection detection across diverse LLM-agent settings.
Takeaways & Limitations
AgentWatcher requires around 10 seconds per detection, so deployments should invoke it selectively rather than at every agent step.
Abstract
from arXiv · showhide
Large language models (LLMs) and their applications, such as agents, are highly vulnerable to prompt injection attacks. State-of-the-art prompt injection detection methods have the following limitations: (1) their effectiveness degrades significantly as context length increases, and (2) they lack explicit rules that define what constitutes prompt injection, causing detection decisions to be implicit, opaque, and difficult to reason about. In this work, we propose AgentWatcher to address the above two limitations. To address the first limitation, AgentWatcher attributes the LLM's output (e.g., the action of an agent) to a small set of causally influential context segments. By focusing detection on a relatively short text, AgentWatcher can be scalable to long contexts. To address the second limitation, we define a set of rules specifying what does and does not constitute a prompt injection, and use a monitor LLM to reason over these rules based on the attributed text, making the detection decisions more explainable. We conduct a comprehensive evaluation on tool-use agent benchmarks and long-context understanding datasets. The experimental results demonstrate that AgentWatcher can effectively detect prompt injection and maintain utility without attacks. The code is available at https://github.com/wang-yanting/AgentWatcher.
1 Introduction
Prompt injection threatens the security and safety of LLM applications, while existing detectors struggle with long contexts and opaque decisions. AgentWatcher addresses these limitations by attributing actions to compact context spans and applying explicit rule-based monitor reasoning.
- Motivation: Prompt injection embeds malicious instructions in benign-looking content, potentially causing agents to perform unauthorized operations.Such attacks create security and safety risks for deployed LLM applications and agents.
- Motivation: Existing detection methods have limited effectiveness as context length increases and lack explicit rules for defining prompt injection.These limitations reduce scalability and make detection decisions difficult to reason about.
- AgentWatcher: AgentWatcher attributes the contexts responsible for an LLM’s output action, focusing detection on a smaller text span.The method targets contexts behind potentially risky actions such as deleting files, sending sensitive information, or visiting external URLs.
- AgentWatcher: A sliding window locates sink tokens and retains surrounding spans, preventing injected instructions from being fragmented across segment boundaries.This design is intended to improve detection performance over fixed text partitioning.
- AgentWatcher: A monitor LLM evaluates the attributed context against explicit customizable rules using the target task and output action.Its reasoning can identify unauthorized actions and explain which rules classify the context as an injection.
- Evaluation: AgentWatcher is evaluated on four agent benchmarks and six long-context understanding datasets, where it demonstrates strong effectiveness across diverse settings.The supplied introduction identifies broad evaluation coverage but does not provide additional numerical results here.
2 Related Work
Prior defenses divide into prevention-based and detection-based approaches, while general-purpose guardrails can be customized but are not designed specifically for prompt injection. AgentWatcher is presented as complementary to such guardrails.
- Defense categories: Existing defenses are categorized as prevention-based or detection-based, with the two families generally viewed as complementary.Prevention mitigates injection effects, whereas detection identifies and blocks injection attempts.
- Detection defenses: Detection methods assess whether context is contaminated before a backend LLM generates a response.Known-Answer Detection uses an LLM instruction-following test, and DataSentinel extends that approach.
- Guardrails: General-purpose guardrails such as GPT-OSS-Safeguard and Nemotron-Safety-Guard primarily target harmful or unsafe content rather than prompt injection specifically.Their customization support does not make them prompt-injection-specific detectors.
- Guardrails: AgentWatcher can use general-purpose guardrail LLMs as its monitor LLM, making the approaches complementary.The supplied passage states this compatibility without specifying an evaluation result.
3 Problem Formulation
The paper models an LLM agent that completes a target task through multi-step interactions with untrusted external contexts. A detector must identify malicious instructions that may influence generated actions.
- Agent state: An LLM agent receives a target task and completes it through a structured multi-step procedure involving external contexts.The target task can involve actions such as summarizing meeting notes and sending them to a recipient.
- Agent state: The agent state contains retrieved contexts and the sequence of reasoning–action pairs generated by the LLM.Contexts may come from external sources, while each reasoning–action pair records an intermediate step.
- Agent interaction: Contexts can be webpages or emails, and actions can be tool calls, webpage interactions, or final answers depending on the agent type.New contexts may be retrieved after actions such as reading an email or navigating to a webpage.
- Detection goal: Attackers may inject malicious instructions into contexts to influence generated actions.The detector therefore evaluates whether the accumulated context contains malicious instructions at each step.
4 Design of AgentWatcher
AgentWatcher first attributes causally important context spans to an agent action, then uses a rule-conditioned monitor LLM to classify and explain prompt injection. Sliding-window attribution keeps detection focused on compact text.
- Design overview: AgentWatcher computes Detector(S_t) by extracting attributed context C* and passing it with the task, action, and rules to a monitor LLM.The two-phase pipeline separates context attribution from rule-based detection.
- Design overview: Attribution reduces reasoning cost, restricts detection to smaller contexts, and simplifies monitor fine-tuning.These benefits follow from separating attribution from reasoning-based detection.
- Attribution: Attention weights from an attribution LLM measure how strongly context tokens influence tokens in the generated action.The attribution LLM may be the backend model or a separate model when the agent is closed-source.
- Attribution: Because token-level attention can be noisy, AgentWatcher uses sink tokens and surrounding spans to identify important context regions.Sink tokens receive disproportionately high attention and help aggregate segment-level information.
- Attribution: The sliding-window procedure scores candidate windows, selects the highest-scoring window, and expands it with surrounding tokens.The expanded window preserves context around the selected region.
- Attribution: Repeating attribution K times produces K non-overlapping expanded windows whose concatenation forms the final attributed context.A single window is described as suboptimal, motivating multiple windows; the model forward pass is performed once for attention weights.
- Rule-based detection: The monitor LLM reasons over the target task, attributed context, action, and customizable rules, returning reasoning and a binary decision.When positive, it also extracts the suspected malicious instruction.
- Rule-based detection: Rules can jointly consider target tasks and untrusted contexts to distinguish benign task instructions from malicious prompt injections.The paper presents rule customization as a way to improve the utility–robustness trade-off.
5 Evaluation
AgentWatcher is evaluated across agent benchmarks and long-context datasets using utility and attack success rate (ASR). It generally improves robustness while preserving utility, including across backbone LLMs and attribution choices, but incurs higher detection cost.
- Experimental setup: AgentWatcher is evaluated on tool-using agent benchmarks and six long-context understanding datasets against state-of-the-art detection baselines.Utility is measured without attacks, while ASR measures attack success under attack.
- Main results: On AgentDojo, AgentWatcher reduces ASR to ≤1% with only a 2% utility loss and remains the only defense consistently reaching ASR at most 10% across settings.On long-context datasets, it achieves the best trade-off between utility and ASR.
- Main results: Across Claude, Gemini, and GPT backbone families on AgentDojo, AgentWatcher consistently reduces ASR to nearly zero while maintaining utility loss of ≤4%.This result supports effectiveness across different backend LLMs.
- Computation time: AgentWatcher requires 8.2s per detection on one A100 GPU, exceeding defenses without LLM-based reasoning and motivating selective invocation for suspicious tool calls.This is reported as an average single-detection computation time on AgentDojo.
- Ablations: AgentWatcher achieves the best security–utility trade-off across attribution methods and remains effective with different attribution LLMs, including relatively small models.Alternative attribution methods are less consistent across datasets, while attribution-model changes preserve similar clean utility.
- A case study for AgentDyn: On AgentDyn, AgentWatcher is the only defense combining 0.0% ASR with relatively strong utility of 48.3% against the important-instructions attack.The study compares nine representative defenses using GPT-4o and notes that near-zero utility loss is challenging.
- Automatic generation of rules: Automatically generated rules achieve comparable performance to human-crafted rules, although data-driven generation generally lowers ASR at the cost of greater utility loss.The results suggest rule generation can be automated with minimal overall performance impact; GRPO further reduces ASR while maintaining utility, and key attribution parameters have limited sensitivity.
6 Discussion and Limitation
AgentWatcher is effective but incurs non-negligible computational cost, so deployment should invoke detection selectively and may restrict attribution to recent contexts.
- Computational Cost: Around 10 seconds of computational time makes invoking AgentWatcher at every agent step impractical.The authors recommend selective invocation for high-risk tool calls or actions.
- Selective Invocation: Users can trigger detection only for blacklisted high-risk actions, such as shell commands, commit pushes, or credential access.Routine actions such as reading files or running tests may remain outside the blacklist.
- Long Contexts: For long trajectories, restricting attribution to a fixed number of recent external contexts can reduce attribution cost.
7 Conclusion
AgentWatcher combines context attribution with rule-based reasoning to detect prompt injections using compact, influential context regions. The method is presented as improving efficiency, accuracy, interpretability, and robustness while maintaining utility.
- Conclusion: AgentWatcher detects prompt injection through context attribution and rule-based reasoning.
- Benefits: Detection focuses on compact context, which the paper states improves computational efficiency and detection accuracy and simplifies monitor fine-tuning.
- Relation to Prior Defenses: Detection-based defenses complement prevention-based defenses and can therefore contribute to defense-in-depth.
- Context Attribution: The method selects high-scoring, non-overlapping context windows and expands them to retain surrounding tokens contributing to the model’s action.
- Rule-Based Detection: A monitor LLM evaluates the target task, attributed context, and agent action against explicit rules, producing reasoning and a binary decision.
D.1 Training Dataset Construction
The training dataset combines diverse clean and attacked contexts with labels and injected instructions, while GRPO rewards both correct detection and accurate extraction. Training increases explicit rule citation by the monitor.
- Dataset Construction: The training dataset contains 20,000 samples from Cleaned Alpaca, WebArena, and StarCoder, with malicious instructions injected into half the contexts.
- Attack Generation: Generated injections are required to be naturally embedded, contextually relevant, stealthy, and actionable rather than meta-instructions.
- Attack Goals: The dataset covers attacker goals involving code contexts and webpage access trees, including exfiltration, reverse shells, key harvesting, and browsing-data theft.
- GRPO Training: As GRPO training progresses, the monitor increasingly cites rules explicitly, measured by rule citation rate over batches smoothed with a 500-generation running average.
- Reward Design: The reward uses exact classification for clean contexts and BLEU similarity between predicted and ground-truth injected instructions for attacked contexts.
E.1 Benchmarks and Datasets
Evaluation spans agent benchmarks and long-context datasets with realistic attack categories, while an ablation studies monitor fine-tuning. GRPO improves robustness, and BLEU-based rewards best preserve security–utility trade-offs.
- Benchmarks and Metrics: The evaluation uses InjecAgent, AgentDojo, WASP, and long-context datasets with intermediate ASR and utility measurements where specified.
- Attack Categories: Long-context attacks cover phishing injection, content promotion, access denial, and infrastructure failure, distributed uniformly across evaluation samples.
- Attack Construction: The long-context attack prompts preserve the original context while embedding one malicious instruction tied to the target task and benign context.
- Monitor Fine-Tuning: Without fine-tuning, the monitor already achieves low ASR across most datasets, while GRPO further improves robustness.
- Reward Ablation: BLEU-based rewards maintain higher clean utility while achieving similarly low ASR, producing the best overall security–utility trade-off.
G Impact of Hyperparameters
AgentWatcher is robust across a wide range of hyperparameter settings, with robustness improving notably as the number of attribution windows increases.
- AgentWatcher remains largely insensitive to sink detection window size, maintaining stable utility and consistently low attack success rates.
- Increasing left expansion improves utility up to a moderate range, after which additional expansion provides limited gains.
- Right expansion size produces stable performance with only minor variations, indicating low sensitivity to this parameter.
- Increasing the number of attribution windows noticeably improves attack robustness while maintaining stable utility.
H Adaptive Attacks
The evaluation targets attacks that adapt to AgentWatcher’s monitor, including heuristic prompt manipulation and joint optimization attacks. AgentWatcher retains relatively low attack success rates, while explicit delimiters help prevent adaptive instructions from influencing monitor reasoning.
- Adaptive attack design: AgentWatcher is evaluated against heuristic-based and optimization-based adaptive attacks targeting both the backbone and monitor LLMs.Heuristic attacks add adaptive instructions around malicious content, while optimization attacks adapt PAIR and TAP to score both model responses.
- Adaptive attack design: The heuristic strategies include safe-environment framing, fake definitions, utility-loss warnings, and fake delimiters designed to manipulate monitor decisions.
- Adaptive attack design: The optimization attacks use a judge LLM to score monitor and backbone responses, and provide both responses as feedback to the attacker.
- Results: Explicit delimiters marking untrusted content help the monitor distinguish user instructions from embedded malicious instructions, reducing influence from adaptive prompts.
- Results: AgentWatcher achieves relatively low attack success rates against the tested adaptive attacks, but strong adaptive attacks remain highly challenging for robust defense.
I More Discussion on AgentDyn
AgentDyn cases can be intrinsically difficult when benign instructions resemble prompt injection under general-purpose rules. The discussion also presents alternative rule-generation strategies and compares computational time on AgentDojo.
- Classification ambiguity: Some benign instructions are difficult to distinguish from malicious ones, creating false positives in AgentDyn.
- Classification ambiguity: The example combines an unrelated purchase request, credential sharing, another person’s account, and a financial transaction, all of which resemble injection signals.
- Classification ambiguity: Without information about the context source’s trustworthiness, benign instructions may satisfy common prompt-injection definitions and remain intrinsically difficult to classify.
- Rule generation: The paper considers direct, data-driven, and bidirectional strategies for automatically generating rule sets.
- Efficiency: Figure 4 compares AgentWatcher’s computational time with baselines on the AgentDojo benchmark.