PaperlayerView exact revision on arXiv

Source-linked AI summary

SPA: Securing Persistent LLM Agents Across Queries with Plan-First Information-Flow Control

Dylan Girrens, Guangjing Wang

arXiv:2608.27234v1cs.CRcs.SE

TL;DR

Persistent LLM agents can let untrusted data alter plans, reach sensitive tool arguments, or survive into later queries. SPA addresses this with single-shot declarative planning, dual-lattice information-flow control, and label-preserving artifact reuse. On the ‘tool_knowledge’ attack, attack success falls to 0.0% on AgentDojo and 0.2% on AgentDojo-MQ, while strict integrity enforcement reduces utility.

  • Problem

    Persistent agents require protection across planning, tool execution, and cross-query state because attacker-controlled data can alter control flow, enter sensitive tool arguments, or become actionable later.

  • Method

    SPA generates one complete declarative DSL plan per query, verifies dual-lattice confidentiality and integrity flows, and persists labeled artifacts while exposing only semantic metadata to later planners.

  • Results

    Under the ‘tool_knowledge’ attack, attack success is 0.0% on AgentDojo and 0.2% on AgentDojo-MQ.

  • Takeaways & Limitations

    Plan-first execution and label-preserving persistence substantially strengthen persistent agents while preserving most required cross-query reuse opportunities.

  • Takeaways & Limitations

    The evaluation does not exhaustively characterize cross-query attacks because AgentDojo’s tool_knowledge attack only weakly exercises delayed memory poisoning.

Abstract

from arXiv · show

Large language model (LLM) agents increasingly operate over untrusted webpages, documents, tools, and persistent states while exercising authority over security-sensitive resources. Existing defenses typically protect either planning or individual tool interactions, but persistent agents face a broader threat: attacker-controlled data can alter control flow, enter security-sensitive tool arguments, or compromise later queries. We present SPA, a plan-first architecture that secures planning, execution, and cross-query state reuse. SPA invokes the planner once per query to generate a complete executable plan in a declarative domain-specific language, then applies dual-lattice information-flow control to track confidentiality and integrity across explicit data flows and control dependencies. To support persistence without re-exposing untrusted payloads to the planner, SPA stores execution results as labeled artifacts and reveals only semantic metadata during later planning. We evaluate SPA on AgentDojo and AgentDojo-MQ, which is our multi-query extension for measuring secure state reuse and delayed attacks. Under the 'tool_knowledge' attack, SPA with information-flow control reduces attack success to zero on AgentDojo and 0.2% on AgentDojo-MQ. Our results show that plan-first execution combined with label-preserving persistence can substantially strengthen persistent LLM agents, while revealing an important security-utility tradeoff introduced by strict integrity enforcement.

1 Introduction

SPA addresses persistent-agent risks by securing planning, execution, and state reuse together. It combines complete pre-execution plans, information-flow checks, and label-preserving artifacts to measure and reduce delayed attacks.

  • Motivation: Persistent agents can let attacker-controlled content redirect plans, enter security-sensitive tool arguments, or remain dormant until later queries.These risks arise from webpages, documents, tool responses, and reused state.
  • Motivation: Existing defenses leave gaps because repeated planning exposes decisions to tool outputs, while fixed plans do not by themselves constrain arguments or persistent state.The paper identifies a need to secure planning, execution, and cross-query reuse together.
  • SPA: SPA invokes the planner once per query to generate a complete declarative DSL plan whose data and control dependencies can be checked before execution.Tool outputs and stored payloads are not returned to the planner during execution.
  • Evaluation: AgentDojo-MQ extends AgentDojo with multi-query episodes and explicit cross-turn dependencies for measuring persistent-state reuse and delayed attacks.SPA is evaluated on both AgentDojo and AgentDojo-MQ.
  • Evaluation: Under the ‘tool_knowledge’ attack, dual-lattice IFC reduces attack success to 0.0% on AgentDojo and 0.2% on AgentDojo-MQ.The evaluation also quantifies security, utility, and reuse tradeoffs.
  • SPA: SPA preserves cross-query reuse by storing labeled artifacts and exposing only planner-visible metadata, while restoring concrete values and original labels during execution.This design avoids exposing stored payloads to later planners.

2 Background and Related Work

The paper frames prompt injection as both a control-flow and data-flow threat. Prior systems isolate planning, quarantine untrusted processing, or abstract tool metadata, but do not cover the full persistent-agent problem.

  • Prompt Injection: Prompt injection embeds adversarial instructions in direct prompts or indirectly in emails, webpages, files, and tool outputs.The shared natural-language channel makes instructions and data difficult to distinguish.
  • Prompt Injection: Attacks can manipulate tool order or pass attacker-controlled values to security-sensitive tools even when the intended action sequence remains unchanged.These are control-flow and data-flow attack paths.
  • Information-Flow Control: Classical lattice-based IFC restricts information propagation using security classes and ordering policies, with confidentiality and integrity tracking complementary properties.Confidentiality concerns sensitivity, while integrity concerns trustworthiness.
  • Prior Defenses: Interleaved defenses apply IFC dynamically as planners alternate with tool execution, but later planning decisions can remain exposed to untrusted outputs.FIDES is cited as an example maintaining dual labels for planner context.
  • Prior Defenses: Plan-first systems commit to control flow before observing untrusted outputs, while ACE additionally hides installed tool metadata through abstract-tool synthesis and later binding.These approaches address planning exposure but leave distinct coverage questions for arguments and persistence.
  • Prior Defenses: CaMeL separates privileged planning from quarantined processing and mediates tools through capabilities and tool-specific policies.Its metadata can express confidentiality and integrity requirements.

3 Threat Model

The threat model considers persistent agents processing trusted user queries while adversaries control tools or external content. Attacks target planning integrity, execution integrity, confidentiality, and delayed cross-query reuse.

  • System and Adversary: The agent serves a sequence of queries, invokes external tools, and may retain results for later reuse.Adversarial information enters through tools or the external content they access.
  • System and Adversary: An adversary may control third-party tool metadata, tool implementations, and external content returned by other tools.Compromised tools may provide adversarial schemas or attacker-chosen results.
  • Scope: User queries and deployment configuration are trusted and therefore direct user-prompt attacks are outside the stated scope.Lattices, per-tool policies, and tool bindings are configured by the deployer.
  • Security Objectives: The security objectives are planning integrity, execution integrity, and confidentiality for tool calls and user-visible outputs.The objectives distinguish altered plans, attacker-controlled arguments, and unauthorized disclosure.
  • Cross-Query Threats: Persistence creates delayed planning and delayed execution attacks when adversarial information retained during one query becomes relevant during a later query.Persisted artifacts therefore receive the same confidentiality and integrity restrictions as fresh tool outputs.
  • Security Goals: SPA targets planner isolation, execution integrity, confidentiality, and cross-query safety.These goals extend protection beyond a single planning step or tool invocation.

4 System Design

SPA processes each query through a fail-closed, single-shot pipeline: it projects metadata, synthesizes and validates a complete plan, verifies information flows, then executes and commits labeled artifacts.

  • 4.1 System Overview: For each query, SPA generates a complete declarative plan before any tool executes and uses a deterministic interpreter that cannot reorder or modify steps.The planner never observes tool outputs or persisted payloads.
  • 4.2 Persistent Context: The context manager separates persistent state into planner-visible metadata, concrete artifact values, and corresponding security labels.These views are indexed by artifact reference.
  • 4.2 Persistent Context: PlannerVisibleContext exposes only artifact metadata, allowing relevance decisions from schemas and semantic annotations without revealing stored values.Concrete values and labels remain available for execution-time retrieval.
  • 4.1 System Overview: The planner synthesizes a complete DSL plan from the query, tool catalog, and visible context; abstract mode instead plans over synthesized capabilities.Abstract tools are later bound to compatible installed implementations.
  • 4.1 System Overview: SPA validates DSL structure, propagates confidentiality and integrity labels, rejects unsafe flows, and executes only the resulting labeled plan.Execution and context updates occur only after verification succeeds.
  • 4.1 System Overview: The pipeline is fail-closed and single-shot: failures terminate the query, and validation or execution results are never returned for planner revision.This prevents iterative replanning from becoming an adversarial feedback channel.
  • 4.3 Isolated Execution: Tool execution uses per-tool backends that may run in ephemeral, network-isolated containers with read-only filesystems, unprivileged users, and timeouts.Values are also validated against declared JSON Schemas, while untrusted text is processed only by the quarantined LLM.

5 Domain-Specific Planning Language

SPA uses a declarative JSON DSL whose explicit, restricted data dependencies support reliable generation, static checking, and persistent artifact reuse.

  • Plan structure: Plans are ordered lists of DSL steps represented in JSON for unambiguous machine-readable structure and constrained decoding.The language supports per-step annotations and metadata required for persistent artifact reuse.
  • Expressions and predicates: Restricted value expressions use variables, projections, literals, and constructed JSON objects or arrays, excluding templates, arithmetic, and user-defined functions.This makes data dependencies explicit and amenable to static analysis.
  • Step forms: The DSL provides retrieve, tool, quarantined-LLM, display, condition, and loop steps for persistent access, external actions, isolated interpretation, output, branching, and iteration.Tool steps alone can invoke external capabilities and produce external side effects, while quarantined LLM steps interpret unstructured content into structured values.
  • Step forms: Tool and quarantined-LLM steps require planner-authored intent and output-semantics annotations describing why each step runs and what its result denotes.These annotations support later recognition of persisted values.
  • Validation and execution: Validation checks required fields, single assignment, definition-before-use, retrieve-only artifact references, permitted predicates, and local branch or loop scope.Execution is fail-closed and terminates on projection errors, schema violations, or tool failures rather than revising the plan.

6 Information Flow Model

SPA assigns separate confidentiality and integrity labels to values, propagates them through data and control dependencies, and checks flows against endpoint policies before execution.

  • Labels: Each runtime value receives a product label combining confidentiality and integrity, with multiple sources raising confidentiality and lowering integrity conservatively.The confidentiality lattice ranges from public to most sensitive, while the integrity lattice ranges from untrusted to most trusted.
  • Flow rule: A flow is permitted only when the destination confidentiality level is at least the source level and the destination integrity requirement is no higher than the source integrity.These conditions prevent sensitive data from reaching less-protected endpoints and low-integrity data from influencing higher-integrity endpoints.
  • Endpoint policies: Tool policies specify authorized input confidentiality, minimum required input integrity, and an output baseline that supplements labels inferred from arguments.The display channel is modeled as an additional endpoint with its own sink policy.
  • Label propagation: Verification combines persistent artifact labels, query labels, data dependencies, and program-counter context to compute each step’s effective input label.The program counter captures the security context under which a step executes.
  • Implicit flows: Conditional branches and loop bodies inherit labels from their guards or collections, so actions dependent on confidential or low-integrity control data receive corresponding restrictions.Both conditional branches are analyzed under the same updated program-counter label.
  • Enforcement: Unsafe plans never reach execution, while display violations selectively redact offending items and preserve permitted output.Tool-policy violations instead cause plan verification to return a violation.

7 Evaluation

The evaluation measures SPA’s utility, attack resistance, and cross-query artifact reuse on AgentDojo and the multi-query AgentDojo-MQ extension. IFC sharply reduces attack success but lowers concrete utility, while abstract planning is constrained mainly by tool mapping and strict reuse remains strongest in concrete modes.

  • Evaluation design: AgentDojo-MQ extends AgentDojo into multi-turn episodes with explicit artifact dependencies for evaluating persistent reuse and delayed attacks.The conversion preserves the original tools, environments, injection setup, and evaluation checks while annotating produced and reused artifacts.
  • Evaluation design: The evaluation ablates concrete versus abstract planning and enabled versus disabled IFC, with an additional open-memory control exposing persisted values to later planners.Metadata-only configurations expose artifact schemas and semantic metadata, whereas the open-memory control exposes persisted values directly.
  • Single-query results: 0% ASR on all 949 single-query attack runs with IFC reduces attacks from four banking successes, but concrete baseline utility falls from 53.0% to 29.0%.Without IFC, about 81% of concrete trials reach execution; with IFC, about 53% stop after integrity failures.
  • Multi-query results: 0.2% ASR with concrete IFC on AgentDojo-MQ accompanies a concrete baseline-utility decline from 62.5% to 35.3%.About 45% of Concrete IFC trials stop as ifc_failed, while abstract configurations mainly fail at tool mapping and only about 7% reach execution.
  • Multi-query results: Open memory raises baseline utility from 62.5% to 65.9% and attack utility from 60.0% to 66.8%, while ASR increases from 0.3% to 0.6%.The utility changes are heterogeneous across suites, and the authors characterize tool_knowledge as a weak probe of delayed injection.
  • Artifact reuse: 95.4% soft-hit reuse and 80.1% strict reuse are achieved by Concrete No IFC, whereas abstract strict reuse is only 12.5–14.7%.Abstract configurations have 81–87% unscorable reuse because producer turns often fail to store artifacts.
  • Multi-query results: Compared with single-query AgentDojo, multi-query concrete baseline utility is 8.2 points higher without IFC and 5.2 points higher with IFC.These deltas compare task formulations rather than isolating persistence alone.

8 Limitations and Future Work

SPA’s evaluation and deployment scope remain bounded by incomplete coverage of persistence-specific attacks, utility costs from strict integrity enforcement, trusted policies, and synthetic multi-query workloads.

  • Cross-query attack coverage: The current evaluation does not exhaustively characterize cross-query attacks, because AgentDojo’s tool_knowledge attack only weakly exercises delayed memory poisoning.Future attacks should target delayed control-flow injection and attacker-controlled values becoming sensitive tool arguments later.
  • Security–utility tradeoff: Strict integrity enforcement reduces utility when legitimate workflows pass externally controlled data into higher-integrity actions.Under a conservative Biba-style policy, these flows are rejected unless trust is explicitly established.
  • Deployment assumptions: SPA relies on trusted deployment policies and simplifying assumptions, so incorrect policies may reject safe workflows or admit unsafe ones.The confidentiality and integrity lattices, per-tool policies, and tool–implementation bindings are part of the trusted computing base.
  • Evaluation generality: AgentDojo-MQ is derived from 97 AgentDojo tasks with manually specified dependency graphs, limiting generality to naturally occurring long-running interactions.Broader evaluation should include independently collected multi-session workloads and additional planning behaviors.

9 Conclusion

SPA treats prompt injection in persistent agents as a cross-query information-flow problem and makes behavior statically enforceable through complete plans and labeled artifacts. AgentDojo-MQ makes delayed attack paths and security–utility tradeoffs measurable.

  • Conclusion: SPA makes cross-query agent behavior statically enforceable and secures persistent information flow without exposing stored payloads to later planners.Complete executable plans expose future data and control dependencies before tool execution, while label-preserving artifacts carry constraints across query boundaries.
  • Conclusion: AgentDojo-MQ measures delayed attack paths and the security–utility tradeoffs of strong enforcement.

Ethical Considerations

The work evaluates prompt-injection defenses in a controlled benchmark setting without human subjects, private user data, or production-system interaction.

  • Ethical Considerations: Experiments use AgentDojo and do not involve human subjects, private user data, or interaction with production accounts or external systems.The work is designed to reduce unauthorized tool use and disclosure.
  • Ethical Considerations: Implementations should use access controls, least-privilege tool permissions, and storage safeguards.

Open Science

The authors release an anonymized archive containing SPA, benchmark integrations, AgentDojo-MQ, documentation, tests, and reproduction scripts.

  • Released artifacts: The anonymized archive includes the SPA implementation and its plan-first, DSL, IFC, persistence, and planning-mode components.
  • Released artifacts: The release includes the single-query AgentDojo integration with suite policies and benchmark runners.
  • Released artifacts: AgentDojo-MQ artifacts include multi-query specifications, dependency graphs, session adapters, reuse scoring, and runners.
  • Reproducibility: Documentation and tests provide setup instructions, architecture notes, unit tests, and scripts for reproducing reported result summaries.

A Prompts

The appendix specifies SPA’s planner prompt and JSON DSL, including valid step types, references, scoping, predicates, schemas, and execution-oriented planning constraints.

  • A.1 DSL Planner: Planner prompts include the DSL specification, optional context and examples, prior-query records, artifact metadata, tools, and the user query.Concrete mode exposes installed tools; abstract mode supplies a synthesized abstract suite.
  • A.1 DSL Planner: The planner emits one executable JSON DSL plan per user request, following a fixed output contract and step vocabulary.The response contains a single top-level steps array, with steps ordered for execution.
  • A.1 DSL Planner: The DSL supports retrieve, tool, q-llm, display, condition, and loop steps, with bindings introduced only by retrieve, tool, and q-llm.Each binding uses a unique non-empty output name subject to scope rules.
  • A.1 DSL Planner: References distinguish whole bound values from projected fields or indices, requiring projection when downstream consumers need scalar or nested values.Missing keys and projection through JSON null fail at runtime.
  • A.1 DSL Planner: The DSL enforces well-typed, defined-before-use references, unique assignments, lexical branch and loop scope, and predicates inside condition steps.Operands resolve from literals, variables, or projected attributes before predicate evaluation.
  • A.1 DSL Planner: Retrieve loads a persisted artifact for later projection, while tool invokes a catalogued deterministic API using schema-conforming inputs.Retrieve is preferred when an existing artifact already covers the query, avoiding redundant tool calls.
  • A.1 DSL Planner: A q-llm step uses a quarantined memoryless model with supplied inputs, a self-contained prompt, and an object-rooted JSON output schema.The appendix recommends deterministic tools instead when they already provide the needed behavior.

B Additional Figures

The appendix explains label-preserving persistence and constructs AgentDojo-MQ by converting single-query tasks into multi-turn episodes with explicit artifact dependencies.

  • Persistence: SPA separates each persisted artifact into planner-visible metadata, a sealed value, and dual-lattice security labels.Later planning sees schema and semantic annotations, while retrieval restores the value and its original labels during execution.
  • Dataset construction: AgentDojo-MQ converts all 97 AgentDojo tasks into multi-turn episodes while preserving tools, environments, injections, and final checks.Standard AgentDojo does not exercise cross-query persistence or delayed consumption of earlier outputs.
  • Dataset construction: The conversion pipeline classifies dependencies, builds artifact graphs, segments turns, adds produces and depends_on annotations, and emits executable tasks.Dependency classes capture how artifacts, choices, and predicates constrain later actions.
  • Dataset construction: MQ episodes distribute one underlying dependency graph across several queries while evaluating utility and security after the final turn.The workspace example separates finding an event from later retrieving its invitees’ email addresses.
  • Security evaluation: The cross-query design blocks a malicious persisted output when it is later reused at an unauthorized sink.Figures 6–10 jointly illustrate persistence protection and the construction of multi-query evaluation tasks.
  • Persistence: A sample episode produces a labeled artifact after the first query, then allows a later planner to see metadata before execution retrieves the value.The depicted flow is trusted query → planner DSL plan → IFC gate → execution and artifact storage.

C Additional Charts

The additional charts show SPA’s security, utility, persistence, and cross-query comparisons, while documenting execution-coverage and evaluation-scope caveats.

  • Security and utility: Concrete planning achieves substantially higher utility than abstract planning across suites, while aggregate attack success remains below 1%.Abstract tool mapping fails in approximately 75–77% of trials, so its low ASR accompanies low execution coverage.
  • Security and utility: IFC lowers concrete baseline utility by 25.0 percentage points on workspace, 28.6 on slack, 35.0 on travel, and 6.25 on banking.The smaller abstract-mode differences reflect widespread earlier mapping failures rather than weaker IFC enforcement.
  • Security and utility: Concrete No IFC records four attack successes among 949 trials, whereas SPA’s strongest protection must still be interpreted with utility outcomes.All four successes occur on banking task 12, where quarantined-LLM output reaches trusted-input tools.
  • Persistence: Exposing stored values changes utility heterogeneously: banking rises from 56.25% to 81.25%, while travel falls from 70.0% to 50.0%.The comparison is against the Concrete No IFC + Values control.
  • Evaluation caveats: Tool_knowledge injections were not designed for the persistence path, limiting what low measured ASR establishes about persistence-specific attacks.The results support the tested cross-query interactions, not exhaustive attack coverage.
  • Persistence: Concrete No IFC reaches 95.4% episode-macro soft reuse and 80.1% strict reuse on scorable episodes, with 16.8% of opportunities unscorable.Soft reuse is conditional on an available artifact; strict reuse includes producer failures.
  • Evaluation caveats: MQ-to-SQ utility differences are not a persistence ablation because MQ also changes prompt scope, retained conversation, environment state, and scoring.Concrete utility changes by +8.2 / +5.2 baseline, while abstract utility barely moves.
Loading 2608.27234v1…