PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimizing Agent Planning for Security and Autonomy

Aashish Kolluri, Rishi Sharma, Manuel Costa, Boris Köpf, Tobias Nießen, Mark Russinovich, Shruti Tople, Santiago Zanella-Béguelin

arXiv:2602.11416v1cs.CRcs.LG

TL;DR

Indirect prompt injection threatens agents that use external data, while deterministic defenses can enforce security but may reduce task completion. PRUDENTIA combines policy-aware planning with richer human interactions to optimize autonomy and task completion. It achieves higher autonomy and task completion than prior deterministic defenses enforcing the same policies.

  • Problem

    Deterministic information-flow defenses can provide security guarantees but may reduce task completion, while existing evaluations do not quantify their benefits in reduced human oversight.

    Source: §1 · pp. 1–2

  • Method

    PRUDENTIA plans for policy compliance alongside task completion, uses policy and label awareness, handles variable expansion strategically, and lets users endorse untrusted data.

    Source: §1 · p. 2

  • Results

    PRUDENTIA achieves higher autonomy and task completion rate than prior deterministic defenses enforcing the same security policies but oblivious to them.

    Source: §8 · p. 11

  • Takeaways & Limitations

    Autonomy metrics show that deterministic system-level defenses can reduce reliance on human oversight while guaranteeing security.

    Source: §1 · p. 2, §8 · p. 11

    The threat model trusts the user, planner, and tools; direct prompt injection, jailbreaks, tool poisoning, hallucinations, and task misinterpretations are outside scope.

    Source: §6 · p. 10

Abstract

from arXiv · show

Indirect prompt injection attacks threaten AI agents that execute consequential actions, motivating deterministic system-level defenses. Such defenses can provably block unsafe actions by enforcing confidentiality and integrity policies, but currently appear costly: they reduce task completion rates and increase token usage compared to probabilistic defenses. We argue that existing evaluations miss a key benefit of system-level defenses: reduced reliance on human oversight. We introduce autonomy metrics to quantify this benefit: the fraction of consequential actions an agent can execute without human-in-the-loop (HITL) approval while preserving security. To increase autonomy, we design a security-aware agent that (i) introduces richer HITL interactions, and (ii) explicitly plans for both task progress and policy compliance. We implement this agent design atop an existing information-flow control defense against prompt injection and evaluate it on the AgentDojo and WASP benchmarks. Experiments show that this approach yields higher autonomy without sacrificing utility.

1 Introduction

Indirect prompt injection makes agents that use external data vulnerable to hijacking, while deterministic information-flow defenses can enforce security but may reduce task completion. The paper introduces autonomy metrics and PRUDENTIA, a policy-aware planner designed to reduce human oversight while preserving security and utility.

1 Introduction

  • Deterministic IFC defenses can provably prevent prompt injections when data is appropriately labeled and policies are correctly specified, but have reduced AgentDojo task completion by up to 30%.The stated guarantee is that tool calls either satisfy policy or are blocked and can be escalated to human approval.Source: §1 · p. 2
  • Autonomy metrics quantify deterministic defenses’ benefits by measuring reduced reliance on human oversight; IFC can require approval only for actions whose policy compliance cannot be determined.The metrics are HITL load and TCR@k.Source: §1 · p. 2
  • PRUDENTIA makes policy compliance an explicit planning objective alongside task completion, exposing data labels and tool policies, strategically handling untrusted data, and allowing endorsement instead of per-call approval.The design addresses existing IFC agents whose planners are unaware of enforced policies, which can lead to unnecessary violations and reduced autonomy.Source: §1 · p. 2
  • Even basic IFC can reduce AgentDojo HITL load by up to 1.5× without decreasing task completion rate.The paper evaluates PRUDENTIA against secure agent designs on AgentDojo and WASP.Source: §1 · p. 2
  • Up to 9% higher task completion rate at TCR@0 and up to 1.9× lower overall HITL load: PRUDENTIA outperforms FIDES on AgentDojo; on WASP, it achieves full autonomy.WASP consists purely of data-independent tasks.Source: §1 · p. 2

2 Background: Information-flow Control for AI Agents

Information-flow control uses labels to track data security properties and applies policies to tool calls based on those labels. The section describes label propagation, confidentiality and integrity labels, and policies for trusted actions and permitted data flows.

2 Background: Information-flow Control for AI Agents

  • IFC labels describe data security properties throughout a system, and generated data receives the join of the labels of its inputs as a conservative over-approximation.For data z derived from x and y, its label is ℓz = ℓx ⊔ ℓy.Source: §2 · p. 3
  • Integrity labels distinguish trusted from untrusted data, and combining trusted and untrusted inputs yields untrusted data.The integrity lattice is {T, U}, with T ⊑ U and U ⊔ T = U.Source: §2 · p. 3
  • Confidentiality labels can mark data as public or secret; the experiments also use reader-set labels whose join is set intersection.Data derived from public and secret inputs is secret.Source: §2 · p. 3
  • A tool call complies with policy when its context and each argument’s dynamic label are no higher than the levels allowed by that tool’s policy.The two highlighted policies are Trusted action (P-T), for trusted-context decisions, and Permitted flow (P-F), for data recipients authorized to read it.Source: §2 · p. 3
  • Non-consequential calls are always permitted, while consequential and data-egress tools receive P-T or P-F policies to prevent prompt-injection-driven exfiltration.The policies allow robust declassification when a call is made in a trusted context.Source: §2 · p. 3
  • The Dual LLM pattern keeps untrusted tool results hidden from the planner’s LLM, while a quarantined LLM can process them in isolation and return restricted outputs.FIDES also allows inspection of full variable contents, which taints planner context and restricts future actions.Source: §2 · pp. 3–4

3 Autonomy: A New Metric for Secure Agents

The section defines autonomy metrics that account for task completion and human interventions, addressing the human oversight required when policy compliance cannot be determined. It evaluates successful traces under benign conditions and treats policy-violating actions as interventions a human would approve.

3 Autonomy: A New Metric for Secure Agents

  • HITL load counts interventions on successfully completed tasks, while TCR@k is the share of tasks completed with at most k interventions per task.TCR@0 measures full autonomy; TCR@∞ measures task-solving capability with unlimited interventions.Source: §3 · pp. 4–5
  • IFC can reduce interventions because approval is needed only for actions that do not comply with policy, but planners cannot anticipate labels of dynamic tool results.The planner can minimize actions that could violate information-flow policy, using incomplete information.Source: §3 · p. 4
  • Autonomy is evaluated by counting noncompliant calls in planner traces on benign tasks, assuming humans approve such calls in successful completions.Unsuccessful traces are excluded because users may reject calls when the agent is not making progress.Source: §3 · pp. 4–5
  • The formal setup assumes each task has a benign completion trace, and defines a planner’s output as a trace of tool calls that can be assessed for policy violations.The violation count v(τ) counts calls in a trace that fail the information-flow policy.Source: §3 · p. 4
  • Plotting TCR@k across intervention budgets shows the autonomy–utility trade-off; increasing k lets more policy violations be resolved by human intervention rather than task failure.The stated planner goal is to approach the TCR@k curve of an all-knowing agent.Source: §3 · p. 5

4 Planning for Autonomy with PRUDENTIA

PRUDENTIA addresses reduced autonomy from planners that are unaware of enforced policies by explicitly optimizing policy compliance alongside task completion. Its design combines policy awareness, strategic variable expansion, and user endorsement of untrusted data.

4 Planning for Autonomy with PRUDENTIA

  • Existing IFC planners can make unnecessary policy violations because they do not know the enforced policies; PRUDENTIA makes compliance an optimization goal alongside task completion.The section presents the agent’s components for policy-aware planning.Source: §4 · p. 5
  • Tool descriptions expose each tool’s policy, and the agent tracks its context label to predict violations and plan around security constraints.The descriptions identify whether calls are consequential, egress data, or always allowed.Source: §4 · p. 5
  • A dedicated plan tool requires the agent to justify variable expansion and enumerate subsequent calls, discouraging unnecessary expansion that would taint context.Expanding variables exposes untrusted data and permanently taints the context label.Source: §4 · p. 5
  • Endorsing untrusted data relabels it trusted, allowing later P-T calls without individual approval; one endorsement can replace ten per-call interventions in the TODO example.Without endorsement, inspecting the email taints context and requires ten individual interventions.Source: §4 · p. 5
  • When expansion is necessary, PRUDENTIA can choose whether to request endorsement, since endorsement may be unnecessary if no consequential calls follow.The choice is between preserving the context label through endorsement and proceeding without endorsement, which taints context.Source: §4 · pp. 5–6
  • PRUDENTIA omits blanket declassification because whether private information should be declassified depends on the situation and is better handled through approval of individual calls.The design is implemented through context-engineering without modifying the underlying IFC enforcement mechanisms.Source: §4 · p. 6

5 Evaluation

The evaluation compares PRUDENTIA with non-IFC and IFC baselines on AgentDojo and WASP using task completion and human-intervention measures. PRUDENTIA reduces intervention burden while maintaining or improving task completion, and achieves zero attacks and zero HITL on WASP.

5.1 AgentDojo Setup and Results

  • AgentDojo covers banking, messaging, travel, and workspace tasks; baselines include Basic, Basic-IFC, and FIDES, with approval rules aligned to security.The evaluation reports means and standard deviations over five runs. The setup assumes a human reviews and stops consequential tool calls stemming from prompt injections.Source: §5.1 · pp. 6–7
  • 1.5–2.6× fewer HITL interactions for Basic and FIDES than non-IFC Basic, with the same task completion rates, shows IFC can improve autonomy without utility loss.On AgentDojo with o3-mini, Basic-IFC’s HITL load is 32.4 versus Basic’s 48.2 at the same utility; FIDES reduces it further to 18.8.Source: §5.1 · p. 7
  • PRUDENTIA reduces HITL load by up to 2.9× versus Basic with o3-mini and 1.9× versus FIDES; with o4-mini, completion is 73.2% versus FIDES’ 75.7%, at loads 19.2 versus 36.8.Source: §5.1 · pp. 7–8
  • On o3-mini, PRUDENTIA’s TCR@0 is 59.1%, versus FIDES’ 50.1%, Basic-IFC’s 35.5%, and Basic’s 24.3%.TCR@0 measures completion with full autonomy and no policy violations allowed.Source: §5.1 · p. 7, Figure 2 · p. 8

5.2 WASP Setup and Results

  • WASP evaluates prompt-injection security on simulated GitLab and Reddit sites; PRUDENTIA labels user-generated content untrusted and applies the Trusted Action policy to consequential browser tools.The benchmark includes 48 GitLab and 36 Reddit test cases.Source: §5.2 · p. 8, Table 1 · p. 9
  • Across WASP models, PRUDENTIA achieves 0 ASR and 0 HITL load while improving task completion compared with Basic.It uses a similar number of turns to Basic.Source: Table 1 · p. 9, §5.2 · p. 9

6 Discussion

The discussion frames human approval as both a security and compliance gate, while noting that frequent prompts can cause confirmation fatigue. It explains how PRUDENTIA’s planning and HITL design address autonomy, alongside boundaries in the interface, comparison scope, and threat model.

On Deterministic Defenses vs. HITL

  • Frequent HITL approvals defend against attacks and support safety, regulatory, or ethical standards, but can cause confirmation fatigue.Users may become desensitized to security prompts and approve actions without careful consideration.Source: §6 · p. 9
  • IFC can replace HITL for security checks, but not for hallucinations or task misinterpretations; this paper treats a successful policy check as requiring no further intervention.IFC interventions are determined by policy checks, while probabilistic LLMs plan and choose variables to expand or endorse.Source: §6 · p. 10

On HITL Interface Design and Human Error

  • HITL prompts still need interface design that prevents attackers from manipulating what humans see and presents security context and data provenance.The authors leave interface design for future work and anticipate using IFC labels and richer provenance to help people distinguish trusted from untrusted sources.Source: §6 · p. 10

On the Components of Policy-Aware Planning in PRUDENTIA

  • Up to 25% higher TCR@0 on data-independent tasks and up to 2.5× lower HITL load on data-dependent tasks are attributed to distinct planning choices.Strategic variable expansion avoids unnecessary expansion on data-independent tasks; the choice between early endorsement and later tool-call approval contributes on data-dependent tasks.Source: §6 · p. 10

On the Scope of Defenses Considered

  • The evaluation compares deterministic defenses with security guarantees, excluding probabilistic defenses without such guarantees.The threat model trusts the user, planner, and tools, treats some external data as attacker-controlled, and excludes jailbreaks, direct prompt injection, tool poisoning, hallucinations, and task misinterpretations.Source: §6 · p. 10

7 Related Work

Related work covers probabilistic prompt-injection defenses and deterministic systems that control decisions based on untrusted data. The paper argues that deterministic defenses should also be evaluated for autonomy and presents a planner that optimizes autonomy and task completion.

Probabilistic Defenses

Deterministic defenses

  • Deterministic defenses share the aim of keeping agent decisions from relying on untrusted data, using approaches such as isolated planners, label propagation, and Dual LLM patterns.The passage describes f-secure (Wu et al., 2024), RTBAS, CaMeL, FIDES, and Kim et al. (2025).Source: §7 · p. 11
  • Prior deterministic-defense work primarily evaluates TCR@0; this paper instead argues for autonomy and designs a planner that optimizes both autonomy and TCR.The paper contrasts its evaluation focus with prior work while describing its planner’s two objectives.Source: §7 · p. 11

8 Conclusion

The paper introduces autonomy metrics and PRUDENTIA, a secure policy-aware planner evaluated on AgentDojo and WASP. It reports higher autonomy and task completion than prior deterministic defenses enforcing the same policies but unaware of them.

8 Conclusion

  • PRUDENTIA achieves higher autonomy and task completion than prior deterministic defenses enforcing the same security policies but oblivious to them.The paper evaluates the planner on AgentDojo and WASP and introduces autonomy metrics to quantify reliance on human oversight.Source: §8 · p. 11
  • The paper introduces autonomy metrics to quantify reliance on human oversight in AI agents.The conclusion states that deterministic system-level defenses can guarantee security while reducing reliance on human oversight.Source: §8 · p. 11
  • Future work should explore robust HITL interfaces that surface IFC context to support meaningful oversight and reduce human error.The conclusion points to user interfaces that resist adversarial manipulation and expose richer context available in IFC-based defenses.Source: §8 · p. 11

A Case Study: Choice between Endorsement and Approval

The case study traces a combined calendar-and-email task in which PRUDENTIA plans around whether privileged tools are needed. It illustrates a no-endorsement path and shows the resulting appointment information and password-reset link.

A Case Study: Choice between Endorsement and Approval

  • The initial plan avoids endorsement because it needs no privileged tool calls, and the trace reports task completion with 0 user interventions.The plan is to retrieve calendar events and search email; the task combines two workspace tasks.Source: A Case Study: Choice between… · pp. 15–16
  • The calendar step returns three appointments and their descriptions for the May 15, 2024 request.The trace includes summaries of Team Sync, lunch with Sarah, and an introductory meeting.Source: A Case Study: Choice between… · pp. 15–16
  • The trace marks a prompt injection as inserted into an appointment description while showing a plan that avoids endorsement before privileged tool calls.The cited passages describe the inserted injection and the plan’s decision not to request endorsement.Source: A Case Study: Choice between… · p. 15
  • The email step expands the message body to extract a password-reset link, which the trace returns.The displayed result is a TechServices password-reset URL.Source: A Case Study: Choice between… · p. 16

B Additional Results

Across AgentDojo suites, PRUDENTIA generally maintains comparable task completion while reducing human-approval burden, with its strongest autonomy gains on data-dependent tasks. Table 2 summarizes performance across methods and models.

B Additional Results

  • Table 2 provides performance data across agents and models and reports consistent patterns of improvement from Basic through PRUDENTIA.The table summarizes performance across all agents with different models.Source: B Additional Results · p. 17, Table 2 · p. 17
  • With few HITL interactions, PRUDENTIA typically achieves higher utility and significantly lower HITL load than Basic or FIDES on data-dependent tasks.Figure 3 reports TCR@0, TCR@1, TCR@2, and TCR@∞, alongside total HITL load.Source: B Additional Results · p. 17, Figure 3 · p. 18
  • With unlimited HITL load, PRUDENTIA generally achieves utility comparable to other planners, although FIDES is higher in some instances.Figure 4 reports mean task completion rates across AgentDojo’s banking, Slack, travel, and workspace suites.Source: B Additional Results · p. 17, Figure 4 · p. 19
  • PRUDENTIA has the lowest total HITL load in most depicted benchmark instances.Figure 5 shows total HITL load across successfully executed tasks.Source: B Additional Results · p. 17, Figure 5 · p. 19
  • PRUDENTIA’s per-task autonomy improvement is especially large on the Slack suite, which requires more HITL interactions than other suites with the Basic planner.Figure 6 reports HITL interactions per successfully completed task.Source: B Additional Results · p. 17, Figure 6 · p. 19

C Overhead Analysis

The overhead analysis estimates token use and cost from logged AgentDojo traces, while using mean turns as a time-correlated proxy. The authors note that costs were not optimized because the study focused on autonomy gains.

C Overhead Analysis

  • Overhead estimates cover input tokens, output tokens, and USD-cent costs on AgentDojo, but output-token estimates are unavailable for reasoning models.The estimates come from logged traces rather than API responses; cost estimates are limited to GPT-4o because detailed reasoning-output logs were not kept.Source: C Overhead Analysis · p. 20, Table 3 · p. 20, Table 4 · p. 20
  • Costs were not optimized; parallel planning and fine-tuning are proposed as ways to reduce turns and token counts.The authors’ goal was to investigate autonomy gains.Source: C Overhead Analysis · p. 21
  • Wall-clock time was not logged; mean AgentDojo turns, which correlate well with time, are reported instead.Additional turns arise from quarantined LLM calls in FIDES and PRUDENTIA and from PRUDENTIA planning turns.Source: C Overhead Analysis · p. 21, Table 7 · p. 21

D System Prompts

This appendix presents system prompts for the Basic agent, FIDES, and PRUDENTIA. It describes how PRUDENTIA’s prompt quarantines untrusted data and guides careful planning and minimal user intervention during task completion.

D.1 Basic Planner

  • The Basic agent prompt, taken from Debenedetti et al. (2024), tells the agent to avoid guessing tool arguments, disambiguate with tools, retry empty results, and check the current year.The prompt is used with and without IFC.Source: §D.1 · p. 22

D.2 FIDES

  • FIDES, taken from Costa et al. (2025), also prohibits guessing function arguments and instructs the agent to honor formatting requests, retry empty searches, check the year, and not invent email addresses.Source: §D.2 · p. 22
  • FIDES uses variable identifiers to hide tool-output values and specifies identifier formats for basic values, dictionary fields, and list-of-dictionary values.The prompt requires using identifiers exactly as provided in subsequent tool calls.Source: §D.2 · pp. 22–23
  • FIDES directs agents to query the quarantined LLM once to present information, but to query separately for each required argument and pass returned identifiers into a known tool call.Before querying, agents should establish why they need the response and how they will use it.Source: §D.2 · p. 23
  • If quarantined outputs do not suffice, FIDES instructs agents to expand variables for inspection, use exact existing identifiers, ask one-piece-of-information queries, and avoid unnecessary quarantined-LLM use.Agents should not invent answers or variable names, and should use the quarantined model’s outputs.Source: §D.2 · pp. 24–25

D.3 PRUDENTIA

  • PRUDENTIA’s objective is to fulfill requests securely and efficiently with the fewest approvals or endorsements, and it directs agents to create or update an ifc_plan when plans change.Source: §D.3 · p. 25
  • PRUDENTIA treats unendorsed variable expansion as permanently lowering context integrity, so tools restricted to high-integrity contexts run autonomously only before that change and otherwise require user approval.Endorsement certifies variable contents as trusted; each approval or endorsement counts as one user intervention.Source: §D.3 · p. 26
  • Variables hide untrusted data to preserve HIGH INTEGRITY context, and the agent must use only the exact variable identifiers and fields supplied.It must not guess variable names, fields, or indices, or use a broader identifier as a proxy for separately provided values.Source: §D.3 · pp. 26–27
  • The quarantined LLM can process hidden variable contents without exposing them to the agent’s context, but each query should return one piece of information.Use it only when processing variable contents is needed; its outputs remain untrusted and should never be expanded.Source: §D.3 · pp. 27, 28, 30
  • For known tool arguments, the agent should query the quarantined LLM separately for each required argument and pass the returned variables directly to the tool call.Before querying, it should determine why it needs the response, how it will use it, and provide all relevant variables and context.Source: §D.3 · pp. 27–28
  • When expansion is necessary, endorsement expands specified variables and preserves HIGH INTEGRITY context, while approval expands all variables and makes context permanently LOW INTEGRITY.The prompt says to choose based on whether more than one HIGH INTEGRITY tool call is needed after expansion.Source: §D.3 · p. 28
  • The agent should plan tool calls and security requirements before expansion, prioritize HIGH INTEGRITY operations, and defer expansion until no other solution exists.It should expand root variables rather than quarantined-LLM outputs, and avoid expanding variables for as long as possible.Source: §D.3 · p. 29
  • The ifc_plan tool should record the full plan at the start, revise it when new information changes the plan, and document the rationale and expected consequences before each expansion.Before expansion, the plan should explain why quarantined processing is insufficient, what the variable is expected to contain, and whether endorsement is cheaper than approval.Source: §D.3 · p. 29
  • The prompt prioritizes completing the user’s task while minimizing endorsements and approvals, exhausting alternatives before expansion and not giving up without trying it.It also says the user cannot provide clarifications or suggestions and that approval or endorsement must be requested through the designated process.Source: §D.3 · pp. 29–30

E Example Accessibility Tree Observation

The example contrasts an unprocessed accessibility tree with a presentation that hides untrusted content in variables. The page observation includes browser-interface elements and content represented by variables.

E Example Accessibility Tree Observation

  • The unprocessed accessibility tree is shown with the prompt injection highlighted, while the other presentation hides untrusted content in variables, including the injection.The passages introduce the raw tree and describe the hidden-content version.Source: E Example Accessibility Tree Observation · pp. 31–32
  • The example page is a Postmill forum post titled “A note from a gentleman thief 1013,” with no comments shown.The browser view also displays the post’s submission information and an empty-comments message.Source: E Example Accessibility Tree Observation · pp. 31–32
  • The hidden-content observation represents page text and interface elements with variable placeholders while retaining visible trusted links and controls.The example includes navigation links, login and registration links, and voting buttons.Source: E Example Accessibility Tree Observation · pp. 32–33
Loading 2602.11416v1…