PaperlayerView exact revision on arXiv

Source-linked AI summary

Securing AI Agents with Information-Flow Control

Manuel Costa, Boris Köpf, Aashish Kolluri, Andrew Paverd, Mark Russinovich, Ahmed Salem, Shruti Tople, Lukas Wutschitz, Santiago Zanella-Béguelin

arXiv:2505.23643v2cs.CRcs.AI

TL;DR

AI agents can be hijacked by indirect prompt injections when processing untrusted data, while existing defenses lack strong assurance. This paper formalizes information-flow security for agent planners and presents FIDES, whose AgentDojo evaluation combines security guarantees with improved task completion.

  • Problem

    Indirect prompt injections can hijack agents through untrusted inputs and exploit delegated tool capabilities, while existing defenses are predominantly probabilistic and provide limited assurance.

  • Method

    The paper models planner information flow, dynamically tracks confidentiality and integrity labels, enforces policies deterministically, and adds selective information-hiding and revealing primitives in FIDES.

  • Results

    FIDES stops all prompt injection attacks with policy checks enabled and completes about 16 % more tasks than a basic planner, rising to 24 % with prompt tuning.

  • Takeaways & Limitations

    FIDES performs a wide range of AgentDojo tasks securely with a modest utility loss compared with systems lacking security guarantees.

  • Takeaways & Limitations

    The restrictive policy rejects some data-dependent actions even when inspected outputs are bounded values such as Booleans, so the evaluation does not endorse more permissive policies.

Abstract

from arXiv · show

As AI agents become increasingly autonomous and capable, ensuring their security against vulnerabilities such as prompt injection becomes critical. This paper explores the use of information-flow control (IFC) to provide security guarantees for AI agents. We present a formal model to reason about the security and expressiveness of agent planners. Using this model, we characterize the class of properties enforceable by dynamic taint-tracking and construct a taxonomy of tasks to evaluate security and utility trade-offs of planner designs. Informed by this exploration, we present Fides, a planner that tracks confidentiality and integrity labels, deterministically enforces security policies, and introduces novel primitives for selectively hiding information. Its evaluation in AgentDojo demonstrates that this approach enables us to complete a broad range of tasks with security guarantees. A tutorial to walk readers through the the concepts introduced in the paper can be found at https://github.com/microsoft/fides

1 Introduction

The paper studies information-flow control for securing AI agents against indirect prompt injection and develops FIDES, a planner that enforces labeled security policies while preserving task utility. Formal analysis, task taxonomy, and AgentDojo evaluation show that FIDES securely completes a broad range of tasks.

  • Indirect prompt injection can hijack agents processing untrusted data and exploit their delegated capabilities for harmful outcomes.Existing defenses are largely probabilistic, while human-in-the-loop prompts can introduce confirmation fatigue and social engineering risks.
  • IFC attaches confidentiality and integrity labels to agent data so consequential actions can be checked deterministically against security policies.The paper uses labels to identify untrusted inputs and restrict actions such as sending external email.
  • The paper formalizes planner security and expressiveness, characterizes dynamic taint-tracking guarantees, and introduces a taxonomy for comparing agentic tasks.The analysis identifies integrity noninterference and explicit secrecy as achievable guarantees under dynamic taint-tracking.
  • FIDES dynamically hides and reveals information, selectively hiding data that would change the planner context’s label and using quarantined LLM inspection with constrained outputs.Type information augments security labels, enabling more fine-grained policies.
  • The task taxonomy shows that FIDES’s selective information primitives expand the class of tasks that can be realized securely.The evaluation compares planner designs on AgentDojo to assess security and utility trade-offs.
  • With policy checks enabled, FIDES stops all prompt injection attacks in AgentDojo.Without policy checks, all planners, including FIDES, succumb to practical prompt injection attacks.
  • With policy checks enabled, FIDES completes about 16 % more tasks than a basic planner using OpenAI reasoning models, rising to 24 % with prompt tuning.The tuned result approaches the performance of a human oracle.
  • With policy checks disabled, FIDES’s selective hiding and revealing does not reduce overall task completion relative to a basic planner when using reasoning models.

2 Background

AI agents interleave LLM queries with external tool calls, creating opportunities for adversaries to influence behavior through tool-returned data. The paper’s threat model assumes trusted configuration but permits adversarial observation and tampering of tool interactions.

  • AI agents augment LLMs with external tools whose calls and arguments are requested by the model and executed by the application.Tool descriptions are included in prompts, and outputs are parsed as either natural-language responses or structured tool calls.
  • The agent loop alternates LLM queries with tool execution, appending each tool result to the conversation history until the LLM produces a final response.Conversation messages identify system, user, assistant, and tool roles.
  • The threat model treats the agent configuration as trusted and fully known to the adversary.The configuration includes the system message, tool descriptions, and LLMs used by the agent.
  • At runtime, adversaries may observe effects of certain tool calls and tamper with data returned by tools.Examples include learning URLs sent to an attacker-controlled server and injecting malicious email or product-review content.

3 Modelling Agent Loops

The paper decomposes agent execution into a planning loop and a planner, formalizing interactions among models, tools, messages, and datastore state. It presents basic and variable-passing planners, including mechanisms for storing and later expanding tool results.

  • Agent and Tool Model: Tools read from and write to a global datastore, allowing the formal model to represent interactions and side effects.Each tool is modeled as a function mapping datastore state and arguments to updated state and a result.
  • Modular Planning Loop: The planning loop delegates decisions to a state-passing planner that can query the model, call a tool, or finish the conversation.The loop handles interaction with the model, tools, and users while remaining parametric in the planner implementation.
  • Basic Planner: The basic planner appends each message to conversation history, forwards model-requested tool calls, and finishes when the model produces a response.This planner plugs into the planning loop to recover the dynamic agent loop.
  • Variable-Passing Planner: The variable-passing planner stores tool results in fresh variables and expands variable names into labeled contents before future tool calls.This separates internal memory from the conversation history and enables later tool arguments to reference stored results.
  • Variable Inspection: Hidden variables can be inspected directly or queried through a quarantined LLM whose output may be constrained to a specific schema.Inspection is needed when the planner cannot determine its next action from visible information.

4 Agents with Information Flow Control

The paper adds confidentiality and integrity labels to agent data and uses lattice-based taint tracking with deterministic policy checks. It distinguishes stronger integrity protection from weaker confidentiality guarantees that prevent explicit flows but permit some implicit leakage.

  • Labels and Lattices: Labels form a lattice whose join operation computes the least upper bound needed to track confidentiality and integrity across data flows.The canonical confidentiality lattice orders public and secret data, while the canonical integrity lattice orders trusted and untrusted data.
  • Taint-Tracking Enforcement: Algorithm 5 propagates labels conservatively through model queries and checks labeled tool calls against a security policy before execution.Tool labels and argument labels are tracked separately, while datastore variables receive labels through taint tracking.
  • Policy Trade-offs: Enforcing both P-T and P-F allows an egressing tool call only when confidentiality and integrity are guaranteed, preventing confidentiality violations from attacks or model mistakes.Enforcing P-F alone bounds illicit data egress but does not prevent prompt injections from triggering egress.
  • Security Guarantees: Non-interference for integrity prevents untrusted data from flowing into trusted sinks, including consequential control-flow decisions.The paper states that this guarantee is sufficient to prevent prompt-injection attacks.
  • Security Guarantees: Explicit secrecy blocks explicit information flows but allows implicit leaks through data-dependent tool-call sequences that an observer may infer.This is weaker than non-interference because the sequence or order of tool calls can still reveal limited information.
  • Security Guarantees: Algorithm 5 guarantees non-interference for integrity and explicit secrecy for confidentiality when policies P-T and P-F are correctly applied.P-T checks the tool label, whereas P-F checks arguments, producing different guarantee strengths.
  • Policy Trade-offs: The selected policy trade-off prioritizes preventing consequential actions over preventing information leaks through tool-call order.Changing policy definitions can yield weaker integrity and stronger confidentiality guarantees, but the chosen design favors practical task completion.

5 FIDES: Advanced IFC for Agents

FIDES extends variable-passing planners with selective hiding, labeled variable expansion, and quarantined inspection using constrained outputs. These mechanisms preserve security while allowing finer-grained use of information, including bounded outputs from untrusted contexts.

  • Selective Introduction of Variables: FIDES selectively hides tool-result nodes only when exposing them would raise the current context’s security label.Hidden nodes are stored in fresh variables with their original labels, while potentially useful lower-risk information remains visible.
  • Selective Introduction of Variables: Hiding restrictive data keeps the conversation label unchanged while allowing later tool calls to reference the stored results through variables.A subsequent Query action can proceed without updating the label of the conversation history.
  • Labeled Variable Expansion: FIDES expands variable names into labeled contents before tool calls, allowing tool-call labels and argument labels to differ.Arguments may originate from earlier tool results retrieved from planner memory, even when the current tool call is trusted.
  • Finer-Grained Policies: Variables enable policies that require trusted tool calls and recipients while permitting message content to depend on untrusted web-search results.This provides finer-grained enforcement than a basic planner.
  • Secure Variable Inspection: Inspecting a variable taints conversation history with its content label, while a quarantined LLM can query variables using constrained output schemas.The quarantined model has no tool access, and structured outputs limit the effect of prompt-injection attacks.
  • Information-Capacity Labels: FIDES combines security labels with an information-capacity type lattice, assigning labels such as ((U, L), bool) or ((U, L), string).Boolean and enumeration outputs carry bounded information, whereas strings can carry unbounded information.
  • Information-Capacity Labels: Low-capacity outputs support more flexible declassification or endorsement policies because they are less useful for prompt-injection payloads or information exfiltration.A one-bit untrusted result may be accepted for consequential actions while an unconstrained string is barred from the same flow.

6 Taxonomy and Expressiveness

The taxonomy distinguishes data-independent tasks, whose tool-call sequence does not depend on returned data, from data-dependent tasks, which require inspecting tool results. Planner designs trade off security and expressiveness: variable passing and quarantined processing preserve policy guarantees in some cases, while constrained queries extend safe handling of dependent tasks with a policy caveat.

  • Task Taxonomy: Data-independent tasks can be completed without the planner viewing tool-call results, whereas data-dependent tasks require returned data to determine subsequent calls.The distinction is based on whether the tool-call sequence depends on data returned by earlier tools.
  • Data-Independent Tasks: The basic planner violates P-T when an action call is generated in a context containing untrusted email contents.This applies to sending a message after reading emails and to creating an event after fetching untrusted emails.
  • Data-Independent Tasks: Variable passing satisfies P-T for data-independent tasks by keeping email contents in planner memory and passing them by reference without affecting action selection.The planner uses the same tool choices while avoiding direct exposure of untrusted data in the action-generating context.
  • Data-Independent Tasks: A quarantined LLM can process untrusted email text outside the planner, preserving P-T for summarization, although its output may still be manipulated.A more restrictive P-T variant could constrain the arguments passed to the consequential action.
  • Data-Dependent Tasks: Variable passing with a quarantined LLM cannot safely realize data-dependent meeting scheduling because the planner must inspect email contents to choose the next call.Inspecting the variable exposes the planner to untrusted content before the consequential set-event call.
  • Data-Dependent Tasks: FIDES uses constrained query-LLM outputs, such as a Boolean meeting indicator and structured event details, to support data-dependent planning while limiting revealed information.The example performs a constrained check, inspects its response, extracts event details, and then calls set event.
  • Data-Dependent Tasks: Under the restrictive P-T policy, the constrained-query example technically fails because the planner context contains the untrusted Boolean output.The paper notes that a more permissive policy could allow the action, but treats such policies cautiously.

7 Experimental Setup

The evaluation uses AgentDojo, a multi-environment benchmark containing benign user tasks and prompt-injection tasks, with labels and policies constructed for the study. Security is assessed by pairing user tasks with injection tasks under a specified attack and averaging results across runs.

  • Benchmark: AgentDojo covers workspace, travel, banking, and Slack environments with real-world-style actions and prompt-injection attack scenarios.Its tasks include reservations, messaging, and financial transactions, and distinguish user tasks from injection tasks.
  • Benchmark: AgentDojo was selected over InjecAgent and ASB because its multi-turn tasks exercise the full planning loop and label-tracking mechanisms.The alternatives were considered less suitable because of single-turn interactions or unlabeled tool calls lacking parameters or return values.
  • Configuration: FIDES requires minimal manual labeling and policy-design effort for AgentDojo, automatically inferring confidentiality labels from task definitions and conservatively labeling relevant fields untrusted.The benchmark does not provide labels or policy definitions, so the evaluation constructs them for the experiments.
  • Evaluation Protocol: Security evaluation pairs each user task with all same-environment injection tasks for a Tool Knowledge attack and reports averages over five runs per task.Predefined functions check whether the user or attacker goals were completed.

2. To evaluate the expressiveness of FIDES in comparison to different planners and models

The experiments measure planner security and utility using attack success rate and task completion rate across FIDES modes, baselines, and frontier models. The setup separates variable passing from the full planner and compares both with dynamic-taint tracking and Tool Filter baselines.

  • Metrics: Attack Success Rate measures the percentage of injection tasks in which the agent completes the attacker’s goal.This metric evaluates security against prompt-injection attacks.
  • Metrics: Task Completion Rate measures the percentage of user tasks in which the user’s goal is successfully completed.This metric evaluates planner utility on benign user tasks.
  • Planners and Baselines: The evaluation includes a variable-passing planner without inspection and full FIDES with query-LLM extraction and variable expansion capabilities.The first targets data-independent tasks, while FIDES is designed for data-independent tasks with query LLM and some policy-compliant data-dependent tasks.
  • Task Completion Evaluation: Figure 3 reports overall task completion rates across all AgentDojo tasks when planners face no attacks and use no policy checks.The figure is intended for comparing planner completion performance under this benign, unchecked condition.
  • Planners and Baselines: The Basic planner with dynamic taint-tracking provides the deterministic-defense baseline, while Tool Filter is the best reported probabilistic defense comparator.Tool Filter asks the LLM to remove tools unnecessary for the task at the beginning of planning.
  • Models: The study evaluates FIDES with five tool-capable frontier models: gpt-4o, o1, o3, o4-mini, and grok-3.gpt-4o and grok-3 are non-reasoning models; o1, o3, and o4-mini are reasoning models.

8 Evaluation Results

The AgentDojo evaluation measures both attack resistance and task completion across planners, policy settings, models, and task categories. FIDES blocks policy-violating attacks while retaining competitive utility, though execution failures and fallback variable expansion limit ideal performance.

  • 8.1 Attack Performance: With policy checks, FIDES blocks all attacks that violate the evaluated policies, while two task types remain outside the enforced tool-call scope.Text-to-text injections can alter user-facing responses, and one calendar-event injection is permitted because it does not exfiltrate data.
  • 8.1 Attack Performance: 133 fewer injections occur for FIDES than Basic with gpt-4o, and policy checks block all attacks that violate P.Without policy enforcement, FIDES still allows injections because low-integrity variables can be expanded and execution continues.
  • 8.2 Expressiveness: Without policy checks, FIDES matches Basic with reasoning models and performs up to 57.52 % better than Variable Passing.Selective inspection primitives help FIDES solve data-independent-with-query-LLM and data-dependent tasks that Variable Passing cannot solve.
  • 8.2 Expressiveness: With policy enforcement, FIDES achieves a higher overall task completion rate than Basic for all models, reaching 16.7 % higher utility with o1.Policy checks lower utility for both planners, and more restrictive policies tend to reduce task completion.
  • 8.2 Expressiveness: Across task categories, FIDES has higher completion rates than Basic and Variable Passing for DI and DIQ tasks under policy checks.The evaluation distinguishes data independent, data independent with query llm, and data dependent tasks; classification uses a human-oracle baseline.
  • 8.2 Expressiveness: FIDES trails ideal utility because of failed query llm executions or planning and fallback variable expansion that causes preventable policy violations.Prompt and interface tuning improves absolute utility by 8.2 % and performance over Basic by 24 % across environments.

9 Related Work

Related work includes probabilistic defenses and deterministic systems inspired by information-flow control. Existing IFC-oriented proposals differ in design focus, while broader guarantees, enforcement mechanisms, and secure task expressiveness remain incompletely understood.

  • Probabilistic Defenses: Probabilistic defenses include system-prompt hardening, structured prompting, input encoding, instruction hierarchies, and training methods for distinguishing instructions from data.These approaches aim to reduce the likelihood of prompt injection attacks in LLM-based systems.
  • Deterministic Defenses: Deterministic defenses track information flow and prevent planners from making decisions based on untrusted data, primarily to prevent indirect prompt injections.Such systems seek deterministic security guarantees rather than relying exclusively on probabilistic behavior.
  • Deterministic Defenses: RTBAS uses taint tracking with attention-based and LLM-as-a-judge label propagation, while other concurrent work uses code-based planning and Dual LLM-like ideas.These systems occupy related points in the design space but differ from FIDES in their planner and propagation mechanisms.

10 Conclusion

The paper formalizes planner execution and information flow, defines task expressiveness categories, and describes FIDES as a taint-tracking planner with selective hiding, revealing, and per-tool policy enforcement. Its design infers labels and uses generic policies to control consequential actions and permitted flows.

  • Formal Model: The formal semantics models planner configurations as command, message, and datastore components evaluated by a deterministic small-step relation.State transformers describe how each evaluation step changes the datastore, with multi-step execution composed inductively.
  • Formal Model: Explicit secrecy means that observations of low variables do not let an adversary rule out any possible initial state.The model captures compatible initial states by comparing low-equivalent states and their observable effects.
  • Task Expressiveness: Tasks map user queries, tool sets, and initial datastores to action traces that solve the task and end with a Finish action.Data-independent tasks admit one tool-call sequence for every datastore, whereas data-dependent tasks require observing tool results to choose calls.
  • FIDES Design: FIDES selectively hides tool results that would change the planner context’s label and securely reveals stored information through a quarantined LLM with constrained decoding.The instrumented variable-passing planner tracks labels while expanding variables for tool calls and selectively hiding labeled values.
  • FIDES Design: The paper’s planner decomposition separates the planning loop from the planner, enabling interchangeable designs and a common interface for policy enforcement.The implementation discussion covers the system message, label assignment and tracking, and security policies.
  • FIDES Design: FIDES infers confidentiality readers from task definitions, labels injection-targeted fields as untrusted, and applies generic permitted-flow and trusted-action policies.The policies authorize readers or receivers, block writes or sends containing untrusted links, and restrict consequential actions to trusted contexts.

D.3 Additional Evaluation Results

The additional evaluation reports utility, task classifications, token usage, and attack results across planner designs, while identifying sources of non-zero performance and evaluation limitations.

  • Sources of non-zero performance: Variable Passing can achieve non-zero performance through incomplete utility evaluation, guesswork, and tasks whose correct behavior is to do nothing.Examples include guessing a highly rated hotel or transaction value, and correctly declining to pay an unspecified bill.
  • Attack results: Successful injections are tabulated across AgentDojo planners using the Tool Knowledge attack, with policy-checking defenses marked.The table notes that two injection tasks are excluded from the attack count in parentheses.
  • Task classification: The proposed taxonomy classifies AgentDojo user tasks to support reproducibility and comparisons of planner expressivity.The classification assigns indices to user tasks.
  • Utility results: Utility results include planner task-completion rates without policy checks and additional comparisons across task categories.Figure 6 reports FIDES utility across categories using DI, DIQ, and DD classifications.
  • Additional measurements: Token-count results compare planners across tasks for o3, while prompt-tuning results report changes in reasoning-model completion rates.The token-count analysis finds similar results across evaluated LLMs.

E Discussion

The discussion examines policy flexibility, label propagation, efficiency, planner awareness, and evaluation scope. It identifies opportunities for higher utility while noting security, efficiency, and benchmarking constraints.

  • Dynamic and fine-grained policies: FIDES permits arbitrary policies based on tools and argument values, but probabilistic detectors or LLM decisions may dilute security guarantees.Dynamically generated policies and labels are proposed as future work that could increase utility.
  • Label propagation: FIDES’s label propagation through LLM responses is sound but may be overly conservative because responses inherit labels from all input messages and tool declarations.The framework can accommodate more permissive analyses that identify which messages influence a response.
  • Efficiency: 2-3× higher token utilization on average makes FIDES less efficient than the Basic planner across the evaluated models.Additional query-LLM turns increase latency and token use, and efficiency optimizations remain future work.
  • Policy-aware planners: Policy-aware planners can seek plans that avoid violations and return data matching user-specified label requirements.A password-reset example illustrates filtering links by trusted email senders, while an AgentDojo task exposed difficulty distinguishing trusted and untrusted sources.
  • Evaluation frameworks: AgentDojo lacks ground-truth labels, a task taxonomy, suitable metrics, and appropriate baselines for evaluating IFC-enhanced planners.The authors extend the benchmark and identify further opportunities to improve task-completion evaluation.
Loading 2505.23643v2…