PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-Sentry: Bounding LLM Agents via Execution Provenance

Rohan Sequeira, Stavros Damianakis, Umar Iqbal, Konstantinos Psounis

arXiv:2603.22868v2cs.CRcs.AI

TL;DR

Agentic systems are difficult to secure because their capabilities and execution flows are not fully known beforehand, complicating validation of whether actions match user intent. Agent-Sentry learns benign execution bounds from provenance traces and applies three layered checks, blocking 94.3% of successful injections while allowing 95.4% of benign executions on combined benchmarks.

  • Problem

    Unknown agent functionalities and probabilistic execution flows make it difficult to validate whether execution fulfills the user’s intended task or includes irrelevant actions.

  • Method

    Agent-Sentry learns execution bounds from provenance graphs and applies structural classification, deterministic allowlisting, and an LLM judge to ambiguous actions.

  • Results

    94.3% of successful injections were blocked while 95.4% of benign executions were allowed on combined AgentDojo and AgentDyn benchmarks.

  • Takeaways & Limitations

    Execution provenance provides an effective security signal for bounding agent behavior while preserving controlled flexibility.

  • Takeaways & Limitations

    The LLM-judge heuristic may not reliably block attacks that embed malicious values inside legitimate-looking content.

Abstract

from arXiv · show

Agentic computing systems, while immensely capable, raise serious security, privacy, and safety concerns. A key issue is that the full set of functionalities offered by these systems, combined with their probabilistic execution flows, is not known beforehand. Given this lack of characterization, it is challenging to validate whether a system has successfully carried out the user's intended task or instead executed irrelevant actions, potentially as a consequence of compromise. We present \emph{Agent Sentry}, a runtime defense that learns a bound on an agent's benign execution from prior legitimate executions and flags any action that falls outside this bound. Agent Sentry layers three complementary checks: a structural classifier over the sequence of actions and the provenance of each function's arguments; a deterministic allowlist check over sensitive argument values; and an LLM judge, invoked only on the residual of actions where the first two checks cannot safely decide between a legitimate new request and a carefully crafted injection. We demonstrate the effectiveness of Agent Sentry in AgentDojo and AgentDyn by blocking 94.3\% of successful injections while allowing 95.1\% of benign executions, without modifying the agent, its tools, or the LLM.

1 Introduction

Agentic systems create security challenges because their capabilities and execution flows are not fully known beforehand. Agent-Sentry bounds behavior using learned provenance patterns and layered runtime checks, while extending evaluation coverage with a larger benchmark.

  • Motivation: Agentic systems’ unknown functionalities and execution flows make it difficult to validate security properties and distinguish intended work from compromised behavior.The paper seeks strong security properties while preserving emergent functionality.
  • Approach: Agent-Sentry learns legitimate structural and provenance patterns from execution graphs rather than relying on manually specified policies.Each graph records where every tool-argument value came from.
  • Approach: Its three-layer pipeline combines a structural classifier, a provenance-aware allowlist, and an LLM judge for residual ambiguous actions.Layer 1 supports both learned and rule-based classifiers; Layer 3 checks alignment with the user’s request.
  • Evaluation: 94.3% of successful injections were blocked while 95.4% of benign executions were allowed on the combined AgentDojo and AgentDyn benchmarks.The paper also reports improved scalability, flexibility, and lower runtime overhead.
  • Evaluation: 30× more scenarios expand the Agent-Sentry Bench beyond prior coverage and include novel use cases.The benchmark extends AgentDojo and AgentDyn with broader scenario coverage.

2 Motivation

Agentic systems are powerful because natural-language instructions can induce new functionality, but their dynamic execution complicates validation and prompt-injection defense. The paper therefore bounds expected execution patterns while retaining controlled flexibility through provenance-aware layered analysis.

  • Motivation: LLM-driven agents can provide complex functionality through natural-language instructions without conventional software changes.Their execution logic is inferred at runtime from prompts and observed information.
  • Motivation: Unknown execution flows make it difficult to determine whether an agent completed the user’s task or was redirected by an injected prompt.The paper uses email summarization as an example of this validation problem.
  • Bounding Agentic Systems: Bounding expected functionality can expose atypical tool-call sequences while avoiding the rigidity of strict predefinition.The paper contrasts an unusual forward() → delete() sequence with the expected read() → forward() flow.
  • Approach: Agent-Sentry records provenance graphs, classifies proposed actions, verifies sensitive values, and uses an LLM judge for unresolved cases.The layered design supports controlled extension when benign functionality was not previously observed.
  • Comparison: The approach learns bounds from prior traces without hand-authored policies and is reported as more scalable and flexible than prior approaches.The paper also reports lower latency overhead than methods relying heavily on LLMs.

3 Threat Model and Assumptions

The threat model covers LLM agents that dynamically invoke retrieval and action tools, including potentially compromised external interfaces and runtime LLM compromise. Agent-Sentry targets unauthorized action tool calls from indirect prompt injection while excluding several behaviors and attack classes.

  • Agent Model: The agents studied combine an LLM, system prompt, natural-language tool descriptions, external services, and programming scaffolding.Their execution logic is inferred at runtime rather than statically defined.
  • Tool Model: Retrieval tools may access trusted first-party data or untrusted third-party data, while action tools perform external operations.The supplied passage establishes retrieval as a major tool category and distinguishes trusted from untrusted sources.
  • Assumptions: The threat model allows compromised tools, external actions, runtime LLM compromise, and attacker payloads injected into agent-accessible data.Retrieved content is assumed to reach the agent without tampering.
  • Defense Objective: The defense aims to block action tool calls unauthorized by the current prompt while preserving legitimate task completion.Actions are labeled against the injection task’s ground-truth signature.
  • Scope: Text-only attacks, direct jailbreaks, and ambiguous prompts fall outside the targeted indirect prompt-injection scope.These cases do not provide the defense with the intended intercepted action-call signal or a sufficiently specified task.

4 Methodology

Agent-Sentry bounds agent execution by learning provenance patterns from legitimate traces and evaluating each action through layered checks. The pipeline combines structural classification, provenance-aware argument verification, and LLM-based intent alignment for unresolved cases.

  • Execution traces and provenance: Agent-Sentry learns execution bounds from provenance graphs that record each tool argument’s sources and derivation metadata.The graph captures how calls contribute values to subsequent calls, while provenance metadata is collected through instrumentation and is not fully trusted alone.
  • Layer 1: Structural classification: Layer 1 classifies action calls as allow, block, or ambiguous using structural features of the provenance graph.The paper evaluates both XGBoost and rule-based classifiers, which inspect provenance structure rather than retrieval text.
  • Design rationale and limitation: The structural layer is robust to changes in injection wording because it relies on provenance fingerprints rather than retrieval text.However, the LLM judge’s descriptive-versus-imperative heuristic may miss attacker values embedded in legitimate-looking content.
  • Layer 2: Grounded argument verification: Layer 2 allows previously observed sensitive argument values from trusted sources and passes novel or unverifiable values onward.Its allowlist covers groundable arguments such as recipients, accounts, credentials, file references, and product identifiers, excluding free-text fields.
  • Layer 3: Intent alignment: Layer 3 uses an LLM judge to resolve calls that remain ambiguous after provenance and allowlist checks by comparing actions with the user’s request.It distinguishes legitimate new values requested by the user from injected values that alter the intended action without changing the call’s structural pattern.

5 Experimental Setup

The evaluation uses seven agents from AgentDojo and AgentDyn, replayed over native traces and a broader benchmark designed to represent varied legitimate behavior. The setup defines trace datasets, classifier and judge configurations, replay methodology, and rate-based metrics.

  • Agents and domains: The evaluation covers seven agents across Banking, Slack, Travel, Workspace, GitHub, DailyLife, and Shopping domains.Four agents come from AgentDojo and three from AgentDyn, with fixed user and injection tasks per domain.
  • Native benchmark limitations: AgentDojo and AgentDyn provide only 5–15 handwritten user and injection tasks per agent, limiting structural-classifier training and realistic behavior coverage.Their prompts also delegate sensitive arguments entirely to untrusted retrieval content, unlike realistic prompts that may anchor some fields directly.
  • Agent-Sentry Bench: Agent-Sentry Bench contains 757 user task specifications and 3,734 paraphrase variants generated and validated across the seven agents.Paraphrases produce structurally distinct traces, allowing the system to learn families of benign executions rather than one canonical trace.
  • Trace datasets: The Agent-Sentry Bench dataset contains 5,380 traces: 3,299 benign and 2,081 successful injections.Native-Trace Bench instead contains 670 traces from 146 user tasks and 63 injection tasks.
  • Evaluation procedure: Experiments replay fixed traces through Agent-Sentry in original order so configurations and ablations are evaluated on identical executions.The replay design avoids variation that would arise from rerunning nondeterministic LLM-driven agents.
  • Metrics: Utility is the fraction of benign traces allowed, while Attack Block Rate is the fraction of malicious traces blocked at or before the injection action.False-positive and false-negative rates are defined as 100 minus Utility and 100 minus ABR, respectively.

6 Experimental Results

Agent-Sentry’s experiments evaluate its three-layer defense across expanded and native execution benchmarks, partial pipelines, latency, generalization, and prior-art comparisons. The results show strong attack blocking and benign utility, with performance and utility varying by benchmark coverage, training data, and Layer 1 design.

  • 6.2 Understanding Per Layer Contribution: 96.1% of actions are decided by XGBoost Layer 1, 0.8% by the Layer 2 allowlist, and 3.1% by the Layer 3 judge.The rule-based configuration routes 54.0%, 15.0%, and 31.0% to Layers 1, 2, and 3, respectively.
  • 6.3 Performance of Partial Pipelines: 93.7% utility and 96.3% ABR are retained by the XGBoost Layers 1–2 partial pipeline, close to the full pipeline’s 96.4% and 96.8%.Under this partial pipeline, 3.1% of traces remain ambiguous rather than receiving a terminal decision.
  • 6.3 Performance of Partial Pipelines: 98.0% utility and 95.4% ABR are obtained by the standalone Layer 3 judge in a tuned setting, but untuned prompts substantially reduce performance.The paper therefore reserves the judge for structurally ambiguous residual cases rather than making it the critical defense layer.
  • 6.5 Generalization to Native-Trace Bench: Under Native-Trace Bench, both Layer 1 variants exceed 84% ABR on every agent, but utility falls to 64.4% and 79.1% compared with 96.4% and 89.3% on Agent-Sentry Bench.The drop is attributed to retrieval-heavy provenance and novel structural patterns; limited benign training data also constrains XGBoost decision boundaries.
  • 6.7 Comparison with Prior Art: 94.3% ABR and 95.1% utility give Agent-Sentry a stronger balance than prior systems, while its 4.30-second latency is roughly 6.2× below CaMeL’s 26.60 seconds.CaMeL reaches 99.3% ABR but only 31.0% utility; Progent has lower Native-Trace utility and is described as rigid under changing tools or benign behavior.
  • 6.8 Stability Across Attack Delivery Mechanisms: 90.8%–92.3% ABR across three attack delivery mechanisms indicates that detection keys on structural provenance signatures rather than injection wording.This comparison is reported on Native-Trace Bench.

7 Related Work

Existing defenses bound agent behavior at different layers, from input interpretation and architectural isolation to semantic intent and execution-policy enforcement. These approaches address complementary aspects of indirect prompt injection and agent safety.

  • Agent security remains difficult because agents dynamically synthesize control flow and tool-mediated dependencies from untrusted inputs rather than executing statically defined programs.
  • Input-Level Constraints and Soft Bounding: Input-level defenses constrain how models interpret context through delimiters, prompt sandwiching, repeated instructions, or data-marking transformations.
  • Architectural and Data-Flow Bounding: Architectural defenses contain execution consequences through isolation or data-flow integrity checks that mediate untrusted content before high-impact actions.
  • Semantic and Intent Bounding: Semantic defenses bound agent goals to the user’s original intent, including activation-delta signals for detecting task drift in white-box models.
  • Policy-Based Execution Enforcement: Policy-based approaches constrain tool-invocation workflows, while automated red teaming primarily discovers vulnerabilities rather than enforcing deployment-time safety.

8 Discussion

The discussion identifies evolving execution coverage, mimicry attacks, and unreliable self-reported provenance as important boundaries on Agent-Sentry’s protection. The system surfaces some unseen cases conservatively, but carefully aligned attacks and provenance errors remain concerns.

  • Evolving execution coverage: New benign behaviors and attack strategies create unseen executions that require periodic updates as agents, tools, and usage patterns evolve.Such cases are surfaced as ambiguous and routed to Layer 3 rather than silently misclassified.
  • Intent ambiguity and mimicry resistance: Mimicry attacks can follow execution paths that are difficult to distinguish from benign behavior, leaving carefully aligned attacks within the system’s susceptibility boundary.The discussion identifies privilege context and external policy signals as possible additional provenance inputs.
  • Reliable provenance source tracking: Structural reasoning depends on per-argument provenance reported by the same LLM driving the agent, so noisy or attacker-influenced source reports can mislead decisions.Current checks reduce but do not eliminate this dependence.

9 Conclusion

Agent-Sentry shows that execution provenance can bound agent behavior and expose injection fingerprints, while a layered design limits reliance on the LLM judge. It operates at the execution layer and can complement other defense classes.

  • Execution provenance and control/data-flow analysis transfer to agentic systems as effective security signals with strong empirical results.
  • A sensitive argument sourced from an untrusted retrieval absent from the user prompt forms an injection fingerprint that attackers cannot directly disguise by changing payload wording.
  • 3.1% of action decisions invoke the LLM judge on Agent-Sentry Bench because Layers 1 and 2 resolve most cases first.
  • Agent-Sentry operates at the execution layer and can be deployed alongside model-centric, input-centric, prompt-injection, and alignment-checking defenses.

A Methodology Section: Supplementary Deta

The supplementary methodology defines a provenance graph, structural features, detectors, allowlists, composite rules, and the LLM judge’s inputs. Together, these components represent argument-level data flow and identify deviations from learned benign execution patterns.

  • Provenance representation: Each execution induces a directed acyclic provenance graph whose vertices are the user prompt and tool calls, with argument-labeled edges encoding data flow.
  • Provenance features: Unknown source identifiers are treated as adversarial downstream, while untrusted hop depth measures consecutive untrusted retrievals feeding an argument.A depth of two or more is defined as the laundered injection signature.
  • Structural detectors: Sensitive-argument detectors capture untrusted sources, laundering, prompt anchoring, provenance novelty, hop-chain anomalies, derivation changes, and missing retrievals.
  • Feature construction: The feature vector contains 17 features grouped into four families, with information-flow features defined over the sensitive argument set.
  • Allowlist scope: The groundable allowlist excludes session-dynamic values and unique free-text payloads because benign history cannot reliably provide reusable values for them.
  • Rule-based classification: Recurring benign motifs abstract retrieval sets while preserving derivation shape and source trust, allowing paraphrase variation without treating trust-level changes as equivalent.
  • Rule composition: Composite rules combine detector subsets because individual detector verdicts are noisy across benign and malicious actions.
  • LLM judge: The Layer 3 judge receives the user prompt, reviewed action, provenance, prior tool-call context, and excerpts from untrusted retrievals.

B.1 Dataset Supplementary information

The supplementary information describes benchmark construction, evaluation protocols, and transfer settings for comparing Agent-Sentry with prior defenses. It emphasizes grouped cross-validation and separation between training and testing traces.

  • Dataset construction: Agent-Sentry Bench is constructed from user-task and injection-task pairs run through multiple LLM models, retaining traces only when the user task or attacker goal succeeds.The primary and supplementary model assignments differ across the benchmark descriptions.
  • Comparative evaluation: Agent-Sentry is compared with Progent and CaMeL, which enforce execution through policies or planner-executor control flows.CaMeL requires live evaluation because it replaces the agent’s tool-calling loop, whereas Progent can be applied to replayed traces.
  • Evaluation protocol: The evaluation uses grouped 5-fold cross-validation so paraphrases of the same user task never cross training and test folds.This prevents performance inflation from testing on task variants related to training traces.
  • Cross-benchmark transfer: Agent-Sentry’s layers are fit on Agent-Sentry Bench and evaluated without fitting on Native-Trace Bench, whose agents, tools, and environment data overlap with the training benchmark.The benchmarks differ in trace volume and user-prompt style.
  • Evaluation protocol: The study also measures how Layer 1 performance changes across 2-, 3-, 4-, and 5-fold cross-validation with corresponding train/test ratios from 50/50 to 80/20.Results are averaged across grouped folds for benign and malicious traces.

C.4 Runtime Latency Overhead: All Configurations

The latency evaluation compares full pipelines and standalone LLM judging under a two-tool-call execution assumption. Provenance-based filtering substantially reduces how often expensive LLM judgment is needed.

  • Configuration: The latency analysis evaluates three configurations involving XGBoost Layer 1 and Layer 3, with rule-based configurations reported under the same deployment assumptions.The assumptions include two action tool calls per trace and 1.2 seconds per LLM judge call.
  • Latency results: 33× lower latency is achieved by the XGBoost pipeline than by standalone Layer 3, at 0.07 s versus 2.40 s per trace.The comparison uses the reported per-trace latency values.
  • Latency results: 3.2× lower latency is achieved by the rule-based pipeline than by standalone Layer 3, at 0.74 s versus 2.40 s per trace.The rule-based pipeline reduces, but does not match, the XGBoost latency reduction.

D.1 Native-Trace Bench K-fold Cross-Validation: Full Results

This section provides the full cross-validation sweep for the XGBoost Layer 1 configuration on Native-Trace Bench. It covers four fold counts and train/test ratios from 50/50 through 80/20.

  • Cross-validation sweep: The XGBoost Layer 1 results are reported for K-fold cross-validation with K ∈ {2, 3, 4, 5}.These settings correspond to train/test ratios of 50/50, 67/33, 75/25, and 80/20.

D.2 CaMeL Per-Agent Results on Native-Trace Bench

The per-agent and sensitivity results show that execution defenses can achieve high pooled blocking rates while differing sharply in utility across agents and model configurations. They also motivate using the LLM judge only as a limited residual layer rather than as the sole defense.

  • CaMeL per-agent results: CaMeL’s pooled Native-Trace Bench result is 31.0% utility and 99.3% ABR, with nearly all retained utility coming from the four AgentDojo agents.CaMeL retains zero utility-passing traces on GitHub, DailyLife, and Shopping because its planner cannot translate those tasks into a complete data flow graph.
  • CaMeL per-agent results: CaMeL retains zero utility-passing traces on the three AgentDyn agents, where its planner cannot produce a complete data flow graph for those tasks.The four AgentDojo agents account for nearly all retained utility.
  • LLM judge sensitivity: The standalone LLM judge’s performance depends on both model choice and prompt design, so the evaluation holds an untuned prompt fixed while comparing two Qwen models.The prompt omits an injection definition, tool-role context, reasoning structure, and few-shot examples.
  • LLM judge sensitivity: 82.0% utility is achieved by the 35B judge versus 70.5% for the 7B judge, while ABR shifts from 75.5% to 81.6%.The smaller model blocks more aggressively, trading additional attack blocking for more benign-action blocking.
  • System design implication: A standalone untuned LLM judge is not suitable as the sole defense, while Agent-Sentry routes only 3.0% of decisions to Layer 3.The three-layer design limits exposure to the judge’s model sensitivity to a small fraction of decisions.
Loading 2603.22868v2…