PaperlayerView exact revision on arXiv

Source-linked AI summary

AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocations

Yu He, Haozhe Zhu, Yiming Li, Shuo Shao, Hongwei Yao, Zhihao Liu, Zhan Qin

arXiv:2603.10749v2cs.CR

TL;DR

LLM agents remain vulnerable to indirect prompt injection because malicious directives in untrusted observations can be mistaken for legitimate instructions, while input-level defenses may not generalize to unseen payloads. The paper introduces AttriGuard, which uses action-level causal attribution and parallel counterfactual tests to distinguish user-supported tool calls from observation-driven ones. Across four LLMs and two benchmarks, it achieves 0% ASR under static attacks with negligible utility loss and moderate overhead, while remaining resilient to optimization-based attacks.

  • Problem

    Indirect prompt injection causes agents to treat malicious directives in untrusted content as valid instructions, while existing input-level defenses may not generalize to unseen payloads.

  • Method

    AttriGuard attributes and gates tool calls by re-executing the agent under attenuated external observations, using counterfactual tests to assess whether each action survives.

  • Results

    0% ASR under static attacks was achieved across four LLMs and two benchmarks with negligible utility loss and moderate overhead, alongside resilience to optimization-based attacks.

  • Takeaways & Limitations

    Action-level causal attribution provides a defense paradigm that remains resilient across static and optimization-based attack settings within the evaluated scope.

  • Takeaways & Limitations

    AttriGuard may produce false positives on highly dynamic, long-horizon benign workflows, and attribution becomes less reliable when users authorize observations to determine critical actions or arguments.

Abstract

from arXiv · show

LLM agents are highly vulnerable to Indirect Prompt Injection (IPI), where adversaries embed malicious directives in untrusted tool outputs to hijack execution. Most existing defenses treat IPI as an input-level semantic discrimination problem, which often fails to generalize to unseen payloads. We propose a new paradigm, action-level causal attribution, which secures agents by asking why a particular tool call is produced. The central goal is to distinguish tool calls supported by the user's intent from those causally driven by untrusted observations. We instantiate this paradigm with AttriGuard, a runtime defense based on parallel counterfactual tests. For each proposed tool call, AttriGuard verifies its necessity by re-executing the agent under a control-attenuated view of external observations. Technically, AttriGuard combines teacher-forced shadow replay to prevent attribution confounding, hierarchical control attenuation to suppress diverse control channels while preserving task-relevant information, and a fuzzy survival criterion that is robust to LLM stochasticity. Across four LLMs and two agent benchmarks, AttriGuard achieves 0% ASR under static attacks with negligible utility loss and moderate overhead. Importantly, it remains resilient under adaptive optimization-based attacks in settings where leading defenses degrade significantly.

1 Introduction

Indirect prompt injection hijacks LLM agents through malicious directives embedded in untrusted content, exposing a gap in input-level defenses. AttriGuard instead attributes proposed tool calls to user intent or external observations using counterfactual tests, achieving strong static-attack robustness with low utility loss and moderate overhead.

  • IPI attacks embed malicious directives in external content that agents may misinterpret as legitimate instructions, causing unauthorized tool calls, data exfiltration, or financial loss.
  • Existing defenses frame IPI protection as semantic discrimination over inputs, relying on known malicious patterns to generalize to unseen injection strategies.
  • Action-level causal attribution distinguishes legitimate actions caused by user intent from malicious actions induced by untrusted observations.
  • AttriGuard re-executes each proposed action under attenuated external observations and flags tool calls that do not survive the counterfactual test.
  • Teacher-forced replay, hierarchical control attenuation, and fuzzy survival criteria address trajectory divergence, control-channel coverage, and stochastic tool-call variation.
  • 0% ASR across four LLMs and two benchmarks under static attacks accompanies approximately 3% utility degradation and approximately 2× token cost.CAMEL also reaches 0% ASR but incurs approximately 20% utility loss and approximately 5× token cost; other evaluated defenses do not provide complete protection.

2 Preliminaries

This section formalizes LLM agents as closed-loop systems that use tool calls to interact with external environments, then defines IPI attacks, threat models, and explicit non-goals. IPI injects malicious content into observations, potentially redirecting execution toward an attacker’s objective.

  • 2.1 LLM Agent Systems: LLM agents iteratively use policy-driven actions and tool calls to interact with external environments.The agent context includes the user task and interaction history; executing tool calls produces observations for subsequent steps.
  • 2.1 LLM Agent Systems: Tool calls are the primary continuation signal: execution continues when pending calls exist and terminates when none are emitted.A rollout ends when Ct is empty, after which the agent emits a final answer or requests clarification.
  • 2.2 Indirect Prompt Injection Attacks: IPI embeds malicious directives in untrusted observations, causing agents to treat data as instructions and potentially invoke unauthorized tools.Examples include crafted emails that induce exfiltration of sensitive information.
  • 2.2 Indirect Prompt Injection Attacks: IPI succeeds when injected observations alter later actions so execution advances the malicious objective rather than the user task.The formalization allows malicious content to appear in any subset of observations across the trajectory.
  • 2.2 Indirect Prompt Injection Attacks: Data-flow attacks are harder to defend because they manipulate authorized tool-call arguments without requiring extra tools.Restricting the available toolset is ineffective when the malicious and user tasks overlap in tool usage.
  • 2.3 Threat Model: Threat models distinguish realistic attackers, who craft transferable payloads without system knowledge, from adaptive attackers, who optimize payloads using complete system and runtime feedback.Adaptive attackers may use iterative genetic or gradient-based search to evade deployed defenses.

3 Formulating Prompt Injection Defense as Action-level Causal Attribution

This section reframes IPI defense as action-level causal attribution: tool calls are judged by whether they remain supported when untrusted observations lose additional control influence. It defines control effect and control potency to make this distinction measurable without directly inspecting opaque LLM internals.

  • 3.1 Limitations of Model-level Defenses: Existing defenses treat IPI as semantic discrimination over untrusted inputs, making their pattern-based framing brittle under distribution shift.They rely on recognizing known malicious patterns, such as imperative tones and explicit override phrases.
  • 3.2 Action-level Causal Attribution: Action-level causal attribution distinguishes intent-supported calls from observation-driven calls according to their causal dependence on untrusted observations.The framework targets individual tool calls rather than free-form responses because the threat model concerns unintended tool use and real-world side effects.
  • 3.2 Action-level Causal Attribution: Causal attribution asks whether a tool call remains justified when observations provide task-relevant evidence and parameters but no additional steering influence.This avoids relying on attacker-supplied rationales that merely make a targeted call appear task-relevant.
  • 3.3 Control Effect and Control Potency: Control effect measures observation influence on a particular call, whereas control potency measures the observation stream’s aggregate ability to steer the next tool-call distribution.Control effect is call-conditioned, while control potency is distribution-level and can support a shared reference context for candidate calls.
  • 3.3 Control Effect and Control Potency: The ideal control-restricted reference context preserves task-relevant evidence and parameters while removing additional observation-driven steering.The operator I is used as a conceptual baseline rather than computed exactly at runtime.
  • 3.3 Control Effect and Control Potency: A call-specific reference view is circular because identifying the cues that make a call likely already requires knowing whether observations drive that call.Therefore, the reference context must be constructed before and independently of any candidate call.
  • 3.3 Control Effect and Control Potency: Control potency provides a call-agnostic way to weaken the observation history’s steering power and construct one shared proxy for multiple candidate calls.Hierarchical attenuation operators are motivated by the need to weaken control channels generically while preserving task-relevant information.
  • 3.3 Control Effect and Control Potency: AttriGuard approximates call-level control effect through survival under attenuated observations: a call that disappears is treated as strongly observation-driven.Because black-box APIs do not expose call probabilities, the system uses equivalent-call survival as an observable signal and applies fuzzy survival in runtime testing.

4 Our AttriGuard

AttriGuard is a runtime defense that intercepts proposed tool calls, replays the agent under attenuated observations, and gates calls according to fuzzy survival. Its design uses synchronized shadow execution, hierarchical attenuation, and a verification-only branch to preserve execution quality.

  • 4.1 Overview of AttriGuard: AttriGuard implements action-level causal attribution with parallel counterfactual tests that attribute and gate proposed tool calls.The system is presented as a defense designed to maintain low false-positive and false-negative rates.
  • 4.1 Overview of AttriGuard: At each step, AttriGuard intercepts the proposed tool-call set and constructs a shadow context by replacing observations with an incrementally attenuated buffer.The original action history is retained while newly arrived observations are transformed and cached.
  • 4.2 Teacher-forced Shadow Replay: Teacher-forced shadow replay reuses the original action history verbatim, so differences between main and shadow calls primarily reflect the changed observation view rather than benign execution drift.This addresses trajectory divergence caused by small planning or batching differences in a freely running shadow branch.
  • 4.2 Teacher-forced Shadow Replay: The shadow branch is necessary because attenuation may weaken task-relevant parameters, causing missed inputs, lower-fidelity arguments, and accumulating multi-step errors if used for execution.Stronger protection against implicit control channels can require more aggressive attenuation and further amplify utility loss.
  • 4.2 Teacher-forced Shadow Replay: AttriGuard confines attenuation to a verification branch while the main run retains full-fidelity observations for producing tool arguments.This separation allows stronger attenuation during verification without imposing its full utility cost on execution.
  • 4.3 Hierarchical Control Attenuation: Hierarchical attenuation transforms untrusted observations to preserve task-relevant evidence and parameters while suppressing cues that disproportionately steer tool actions.The operator library targets distinct control channels through LLM-based rewriting and provides a configurable spectrum of attenuation levels.
  • 4.3 Hierarchical Control Attenuation: The attenuation hierarchy balances benign utility against suppression of observation-driven steering, but the paper does not claim a formal monotonic guarantee for control potency.Its levels are intended to provide progressively stricter views in ablations, and new operators can be composed into the hierarchy.
  • 4.4 Fuzzy Survival Criterion: Fuzzy survival accepts a proposed call when a shadow call has the same function and either canonicalized matching arguments or a task-conditioned adjudication supports equivalence.Canonicalization ignores superficial formatting differences; an auxiliary judge handles matching function names with nonidentical arguments.

5 Experiments

Across AgentDojo evaluations, AttriGuard largely eliminates static attack success while preserving benign utility, outperforming defenses that trade robustness for substantial utility or efficiency costs. Its ablations and cost analysis show that security depends on carefully calibrated counterfactual verification rather than attenuation during the main execution.

  • Overall defense effectiveness: AttriGuard drives ASR to near 0% across all evaluated AgentDojo settings, including attacks that leave undefended GPT-4.1-mini and Qwen3-32B above 75% ASR.It is especially effective against IMPORTANTMESSAGES and TOOLKNOWLEDGE, which are more difficult for undefended agents than IGNOREPREVIOUS and COMBINED.
  • Overall defense effectiveness: Across nearly all models and task suites, AttriGuard maintains benign utility comparable to the no-defense baseline.The main exception is Llama3.3-70B on Travel, where BU decreases by roughly 20% during long, tool-intensive trajectories.
  • Comparison with other defenses: AttriGuard consistently achieves 0% ASR across four static attacks while retaining better utility than strict-isolation defenses.On Gemini-2.5, CaMeL reaches 0% ASR but averages 38.04% BU versus AttriGuard’s 67.39%.
  • Comparison with other defenses: Model-level defenses often generalize poorly from canonical templates to workflow-framed attacks and across backbone models.The authors attribute this brittleness to treating IPI defense as semantic input discrimination under distribution shift.
  • Efficiency and cost analysis: AttriGuard operates at moderate cost, with roughly 32–41 seconds end-to-end latency and about a 3× increase over no defense.Target-agent token usage is model-dependent: approximately 2× higher on Llama but close to baseline on Gemini.
  • Ablation studies: Ablations show that stronger attenuation restores 0.00% ASR, while weaker attenuation raises ASR to 12.17% despite higher utility.The λ = 1 setting records BU/UA of 76.47%/69.57%, whereas λ = 3 records 67.65%/67.39%.

6 Related Work

Prior work develops LLM agents, characterizes indirect prompt injection, and proposes defenses across multiple control channels. These approaches motivate defense-in-depth because agents can be steered through instructions, memories, and untrusted observations.

  • LLM Agents: LLM agents combine reasoning, planning, and tool use to automate workflows across software engineering, web navigation, and other dynamic environments.
  • IPI Attacks: Indirect prompt injection embeds malicious instructions in untrusted data streams, potentially overriding user intent in LLM-integrated applications.
  • Defenses against IPI Attacks: Existing defenses include input sanitization, cache-feature pruning, and other mitigations, but some reduce utility or remain vulnerable to non-imperative payloads.
  • Other Threats on LLM Agents: Agent threats also target instruction, memory, and protocol channels, not only untrusted observations.
  • Other Threats on LLM Agents: These threat categories motivate defense-in-depth with complementary safeguards across instruction, memory, and protocol channels.

7 Limitations and Future Directions

AttriGuard has applicability and reliability boundaries in long-horizon, delegation-heavy, and non-natural-language settings. Future safeguards must account for false positives, attenuation-model sensitivity, delegated authority, and entangled evidence and control cues.

  • Long-horizon workflows: Long tool-intensive benign workflows may produce false positives because repeated counterfactual checks amplify discrepancies from attenuation, stochasticity, batching, or formatting.
  • Attenuation sensitivity: The prompt-based attenuation component may be sensitive to updates of its auxiliary LLM, although the paradigm only requires reducing observations’ control effect.
  • Delegation boundaries: Delegation-heavy workflows weaken attribution when users authorize external observations to determine actions or arguments constituting the attacker’s objective.
  • Applicability boundaries: Causal attribution cannot verify the authenticity or integrity of heavily delegated content, and it becomes harder when task evidence and control cues share non-natural-language schemas.

8 Conclusion

The paper introduces action-level causal attribution for defending LLM agents against indirect prompt injection and implements it in AttriGuard. The system gates tool calls with parallel counterfactual tests supported by replay, hierarchical attenuation, and fuzzy survival criteria.

  • The paper proposes action-level causal attribution, shifting IPI defense from inspecting untrusted inputs to asking why an agent issues a tool call.
  • AttriGuard operationalizes this paradigm by attributing and gating tool calls through parallel counterfactual tests.
  • AttriGuard combines teacher-forced replay, hierarchical control attenuation, and a fuzzy survival criterion to support reliable attribution.

Ethical Considerations

The study frames indirect prompt injection as a practical threat to autonomous agents and justifies the work by its potential to reduce severe failures. Experiments were conducted in controlled, sandboxed settings, while publication risks were addressed through conservative reporting and withholding optimized payloads.

  • The main stakeholders are end users, data subjects, deploying organizations, and researchers or practitioners who design, evaluate, or audit agent systems.
  • Experiments used controlled benchmark environments and sandboxed or simulated tool interactions rather than real user accounts or newly collected personal data.
  • Publishing defensive techniques may help adversaries refine attacks and may introduce false positives that reduce utility in benign workflows.
  • The authors mitigate publication risks by documenting limitations and failure modes conservatively and withholding successfully optimized injection payloads.
  • The work is ethically justified because reducing unintended data disclosure and unauthorized tool actions may benefit users and system operators.

Open Science

The appendix releases AttriGuard artifacts and documents attack templates, implementation components, and defense baselines spanning prompting, training, system, and detection paradigms.

  • Open Science: The released artifacts include AttriGuard’s implementation, evaluation scripts, and environment and reproduction documentation.
  • Attack Templates: The four static attacks inject attacker-controlled content into observations using override, task-completion, high-priority-user, or tool-disclosure framing.
  • Implementation: AttriGuard’s hierarchical attenuation uses LLM rewriting operators to suppress control signals while preserving task-relevant evidence.
  • Implementation: Its fuzzy survival criterion uses an auxiliary LLM judge for same-function actions with non-identical arguments.
  • Defense Baselines: The evaluated baselines cover prompting, training-based, system-level, and detection-based defenses.
  • Defense Baselines: Detection baselines include classifiers, sanitizers, and masked re-execution methods that filter observations or compare tool-use behavior.

C.1 Results on ASB

On ASB, AttriGuard consistently reduces attack success across models, although residual ASR remains because the benchmark includes trusted user requests for sensitive actions.

  • ASB Results: AttriGuard lowers ASB attack success across models; on GPT-4.1-mini, TOOLKNOWLEDGE ASR falls from 94.25% to 38.50%.
  • ASB Results: ASB residual ASR partly reflects instances resembling direct prompt injection or jailbreaking, which are outside the standard IPI threat model.
  • ASB Results: AttriGuard allows sensitive-tool invocations attributed to trusted user intent, even though ASB counts them as attack successes.

C.2 Results of Sensitivity Test

The sensitivity test finds that auxiliary-model choice affects AttriGuard, while a moderately sized open-source model can still provide competitive performance with low ASR.

  • Auxiliary LLM Sensitivity: Stronger auxiliary LLMs generally perform better, especially on BU and UA, because attenuation and fuzzy survival require instruction-following consistency.
  • Auxiliary LLM Sensitivity: Gemma-3-12B-IT achieves performance comparable to lightweight commercial models with low ASR and competitive utility.
  • Adaptive Evaluation: Static prompt-injection suites can overestimate robustness, motivating adaptive search-based evaluation against defense-aware attackers.

D.1 Threat Model and Attacker Interface

The adaptive evaluation models a defense-aware attacker that searches payloads using observable runtime feedback, bounded query budgets, and evolutionary mutation and scoring components.

  • Threat Model: The attacker injects one candidate payload per end-to-end run while the victim performs a benign user task.
  • Threat Model: The attacker observes the defense description and exposed behavioral feedback but not internal model states.
  • Query Budget: B = 200 evaluations are granted per benchmark instance, with each evaluation corresponding to one full agent run.
  • Adaptive Attack Pipeline: OpenEvolve combines population control, a MAP-Elites archive, LLM-based mutation, and victim-system scoring to search for adaptive payloads.
  • Mutation Interface: Mutations use defense descriptions, task objectives, and prior payload scores and rationales to generate meaningfully different candidates.
  • Scoring Interface: Each candidate receives a scalar fitness score from an end-to-end trace containing attack success, defense state, and messages.
Loading 2603.10749v2…