Source-linked AI summary
What Guides the Agent? Adjudicating Unauthorized Behavior via Localizing Behavior-Guiding Instructions
Yichao Gao, Yumo Zhang, Yunhao Yao, Haohua Du, Puhan Luo, Ruiqi Li, Zhiqiang Wang
TL;DR
LLM agents can misinterpret untrusted external content as behavior-guiding instructions, leading to unintended tool invocations. AttnLocate localizes these instructions from attention patterns and adjudicates behavior by provider authority, achieving strong localization and adjudication performance while transferring to unseen models and adapting policies without retraining.
Problem
Untrusted external content may be dynamically misinterpreted as behavior-guiding instructions, subverting agent decisions and causing unintended tool invocations.
Method
AttnLocate aggregates multi-head, multi-layer attention features, detects variable-length behavior-guiding spans with a 1-D U-Net and anchor-free head, and adjudicates them using configurable provider authority policies.
Results
Across ten agent configurations from six model families, AttnLocate achieves 0.743 mIoU, 0.956 AUROC, and 0.934 recall, while generalizing to unseen models and adapting policies without retraining.
Takeaways & Limitations
Localizing exact behavior-guiding instructions enables fine-grained, provenance-aware policy enforcement.
Takeaways & Limitations
The trust model treats external context providers as untrustworthy and assumes the LLM agent is honest but vulnerable to malicious injected content.
Abstract
from arXiv · showhide
LLM agents integrated with external resources gain complex task capabilities, yet the unified natural-language context channel makes them vulnerable to injection attacks: untrusted external data may be dynamically parsed as behavior-guiding instructions during LLM inference, thereby subverting the agent's decision. Existing defenses focus on static detection or isolation of malicious content at the input/output level, remains insufficient for detecting such dynamic inducements that arise during model reasoning. We propose Attnlocate, a runtime framework for fine-grained localization of context spans that genuinely influence tool-calling decisions, i.e., behavior-guiding instructions. Attnlocate casts this localization problem as an object detection task, aiming to detect the distinctive activation traces induced by behavior-guiding instructions within the attention matrix. Specifically, we design a multi-head, multi-layer attention aggregation scheme to construct a token-level feature space tailored for object detection. Then, a 1-D U-Net equipped with an anchor-free detection head is deployed to detect these spans. Finally, based on the authority of the provider from which the detected behavior-guiding spans originate, Attnlocate dynamically adjudicates malicious invocation attempts. We evaluate Attnlocate across ten agent configurations from five LLM families, covering scenarios involving indirect prompt injection and tool poisoning. Attnlocate achieves a mean IoU of 0.743, an average AUROC of 0.956, and a 0.934 true-positive rate at 0.067 false-positive rate. It also transfers effectively across unseen models and supports authority policy adaptation without retraining.
1 Introduction
AttnLocate addresses injection attacks by locating context spans that actually guide tool-calling decisions and adjudicating them according to provider authority. Across diverse agent configurations and attacks, it reports strong localization, adjudication, generalization, and policy-adaptation results.
- Motivation: Injection attacks can cause untrusted external content to override benign user requests and trigger unauthorized tool execution.The paper illustrates an injected instruction causing ReadFile to access a sensitive SSH private key instead of the requested file.
- Motivation: Existing defenses do not precisely identify instruction-like spans and their originating providers when malicious content is embedded in benign carriers.The paper argues that fine-grained span and provenance identification is needed beyond overt-content detection and coarse attribution.
- Approach: AttnLocate localizes behavior-guiding instructions during inference and adjudicates unauthorized invocations according to the localized span’s origin.Its key premise is that genuinely decision-influencing tokens produce distinctive attention activation patterns.
- Evaluation: 0.743 average mIoU, 0.956 AUROC, and 0.934 recall were achieved across ten agent configurations from six model families and multiple injection attacks.The system also generalizes to unseen models and adapts authority policies without retraining.
- Approach: AttnLocate combines multi-head, multi-layer attention aggregation, a 1-D U-Net object detector, and an authority-based arbiter.The design targets noisy attention, variable-length span boundaries, and policy-dependent legitimacy.
2 Background and Related Work
The paper frames attention as a fine-grained signal for tracing tool-call decisions to influential context spans. It contrasts this approach with static, architectural, and coarse-grained attribution defenses against injection attacks.
- Attention-based attribution: Attention weights can be aggregated across output tokens, heads, and layers to produce token-level signals of context influence on decisions.These signals motivate attention-based attribution and context-tracing methods.
- Existing defenses: Static scanning detects or filters suspected injections before they reach model inference, while architectural defenses isolate components or separate trusted control from untrusted data.These are two broad categories among existing injection defenses.
- Attribution-based defenses: Attribution methods provide reasoning-time evidence by tracing decisions to influential inputs, using perturbation, attention aggregation, or dependency graphs.Examples include ContextCite, TracLLM, AttnTrace, and MindGuard.
- Illustrative attack: In the illustrated successful attack, attention peaks over the invoked tool, benign user query, and externally injected instruction that guides the malicious decision.The figure treats these spans as behavior-guiding inputs because they determine the final tool call.
- Limitations of prior work: Coarse units such as passages, documents, or metadata fields can entangle short malicious instructions with benign carriers and impede selective intervention.The paper positions fine-grained localization as a response to this limitation.
3 Threat Model
The threat model treats users and the agent as trusted while external providers remain an injection surface. AttnLocate therefore localizes externally supplied instructions and checks their provider authority against configurable execution policy.
- Trust model: External context providers, including databases, repositories, tools, APIs, and services, are treated as untrustworthy injection surfaces.They can incorporate external information into the agent’s reasoning context and enable malicious command injection.
- Trust model: The agent determines and executes tool invocations from the user query q and external context C, which may include provider metadata or prior execution results.The agent is modeled as honest but vulnerable to misinterpreting malicious external content as instructions.
- Attacker model: An attacker may arbitrarily inject, remove, or modify externally supplied content but cannot directly modify trusted workflow components.The attacker’s objective is to make injected span m influence decision y and divert execution from user intent.
- Defense goals: AttnLocate dynamically identifies external spans truly interpreted as behavior-guiding instructions rather than merely checking whether content appears malicious.It then evaluates the identified instruction under configurable security policies.
- Adjudication: The authority arbiter compares the localized span provider’s authority with the minimum authority required for the current decision.The policy assigns provider authority levels and determines whether the invocation is unauthorized.
- Deployment boundary: The defense requires white-box access to inference attention matrices but does not modify the existing agent workflow or invoke external LLMs.The stated deployment scenarios include self-hosted agents and provider-integrated security services.
4 AttnLocate Design
AttnLocate converts attention patterns into token-level features for detecting behavior-guiding instruction spans, then adjudicates whether their providers are authorized to influence a decision. Its design combines attention aggregation, object detection, provenance resolution, and configurable authority policies.
- Attention Aggregation: Attention aggregation combines multi-head and multi-layer signals into token-level features for instruction localization.Head averaging and Gaussian layer weighting reduce variance while preserving complementary contextual signals.
- Attention Aggregation: Decision-conditioned slicing and feature extraction restrict localization to dependencies between the final decision and external context.Each context position is represented using mean, maximum, standard deviation, and learned attention-pooling statistics.
- Instruction Localization: Object detection replaces independent token classification to predict coherent variable-length instruction spans with stable boundaries.A 1-D U-Net restores token resolution, while an anchor-free head predicts objectness, boundary offsets, centerness, and sample-level detection.
- Instruction Localization: Sink-aware regularization emphasizes high-attention background positions so the detector can distinguish attention sinks from genuine behavior-guiding spans.The relative threshold adapts to context length, and sink positions receive greater training weight.
- Unauthorized Behavior Adjudication: Provenance resolution maps localized spans to providers, while a configurable authority policy compares provider authority with the minimum authority required for the decision.The resulting adjudication labels a span malicious when it exists and its provider authority is below the decision requirement.
- Unauthorized Behavior Adjudication: Separating localization from adjudication avoids attack-specific lexical dependence and makes alerts auditable across injection types.Each alert includes the localized span, its provider, and the violated authority requirement.
5 Evaluation
AttnLocate is evaluated for localization and adjudication across multiple LLM agents, attack datasets, baselines, transfer settings, context lengths, and ablations. Results show strong in-distribution performance, cross-model transfer, and sensitivity to context length and design parameters.
- Experimental Setup: AttnLocate is evaluated on MCPTox tool-poisoning and InjecAgent indirect-prompt-injection attacks across ten configurations from six model families.The evaluation uses standard and thinking modes and compares against static scanning, behavior-auditing, and attribution-based monitors.
- Main Results: 0.692–0.858 mIoU spans the reported localization range, while adjudication AUROC ranges from 0.927 to 0.989 across agents.Think-mode performance improves by approximately 5.4% over the baseline on Qwen3 models.
- Main Results: 0.947 TPR and 0.070 FPR are achieved on Phi-4†, the highest TPR and lowest FPR reported there.The comparison is against existing works in Table 2.
- Cross-model Generalization: 0.736 and 0.658 mIoU are obtained on unseen DeepSeek-R1-Qwen-14B† and Phi-4† models, respectively, while Hit@0.5 remains near 0.90.Adjudication AUROCs are 0.893 on average across unseen Qwen3 agents, 0.921 on DeepSeek-R1-Qwen-14B†, and 0.859 on Phi-4†.
- Analysis and Ablations: 0.8105/0.9722 at ≤256 falls to 0.6951/0.8571 at >2048 for mIoU/TPR, while FPR declines from 0.0714 to 0.0159 as context length increases.The reported trend indicates that longer inputs dilute salient signals.
- Analysis and Ablations: µ = 2L/3 yields the highest mIoU of 0.774 with high TPR and markedly lower FPR, while β = 10 reaches peak mIoU of 0.774 with 0.927 TPR.Without sink suppression, TPR is 0.946 but FPR rises to 0.417; increasing β to 12 over-regularizes and degrades both metrics.
- Authority-Policy Analysis: TPR above 0.909 and FPR below 0.086 remain stable as authority policy changes from Principal-only to Tool-authorized to Result-authorized.This supports adaptation to configurable authority policies.
6 conclusion
AttnLocate localizes behavior-guiding instructions from attention patterns and adjudicates unauthorized invocations based on their origin. Evaluations show strong localization and adjudication performance, transfer to unseen models, and policy adaptation without retraining.
- AttnLocate enables fine-grained, provenance-aware policy enforcement by localizing the exact instructions guiding agent behavior.
- Evaluations across ten agent configurations and six model families cover tool poisoning and indirect prompt injection.
- AttnLocate transfers to unseen models and adapts to authority-policy changes without retraining.