PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Management

Ruoyao Wen, Hao Li, Chaowei Xiao, Ning Zhang

arXiv:2602.07398v1cs.CRcs.AI

TL;DR

Indirect prompt injection exploits conventional agents’ accumulation of external content and reasoning traces, allowing malicious instructions to persist and degrading decision quality. AgentSys uses hierarchical isolated workers and schema-validated communication to control what enters main-agent memory. It achieves low attack success on AgentDojo and ASB while slightly improving benign utility over undefended agents.

  • Problem

    Conventional agents indiscriminately accumulate tool outputs and reasoning traces, allowing injected instructions to persist and verbose content to degrade decision-making.

  • Method

    AgentSys hierarchically isolates worker-agent execution and allows only schema-validated JSON return values to cross into the main agent’s memory.

  • Results

    AgentSys achieves 0.78% ASR on AgentDojo and 4.25% on ASB, with 64.36% benign utility versus 63.54% for undefended agents.

  • Takeaways & Limitations

    Explicit memory management addresses attack persistence and utility degradation while preserving flexible workflows and robustness across models and adaptive attackers.

  • Takeaways & Limitations

    The LLM-based validator can approve subtly misaligned calls or reject legitimate ones, leaving residual validator failures.

Abstract

from arXiv · show

Indirect prompt injection threatens LLM agents by embedding malicious instructions in external content, enabling unauthorized actions and data theft. LLM agents maintain working memory through their context window, which stores interaction history for decision-making. Conventional agents indiscriminately accumulate all tool outputs and reasoning traces in this memory, creating two critical vulnerabilities: (1) injected instructions persist throughout the workflow, granting attackers multiple opportunities to manipulate behavior, and (2) verbose, non-essential content degrades decision-making capabilities. Existing defenses treat bloated memory as given and focus on remaining resilient, rather than reducing unnecessary accumulation to prevent the attack. We present AgentSys, a framework that defends against indirect prompt injection through explicit memory management. Inspired by process memory isolation in operating systems, AgentSys organizes agents hierarchically: a main agent spawns worker agents for tool calls, each running in an isolated context and able to spawn nested workers for subtasks. External data and subtask traces never enter the main agent's memory; only schema-validated return values can cross boundaries through deterministic JSON parsing. Ablations show isolation alone cuts attack success to 2.19%, and adding a validator/sanitizer further improves defense with event-triggered checks whose overhead scales with operations rather than context length. On AgentDojo and ASB, AgentSys achieves 0.78% and 4.25% attack success while slightly improving benign utility over undefended baselines. It remains robust to adaptive attackers and across multiple foundation models, showing that explicit memory management enables secure, dynamic LLM agent architectures. Our code is available at: https://github.com/ruoyaow/agentsys-memory.

1 Introduction

LLM agents face indirect prompt injection and decision-quality risks because conventional designs retain all external outputs and reasoning traces in memory. AgentSys addresses this through hierarchical isolation and schema-validated communication, achieving low attack success while preserving utility.

  • Indirect prompt injection embeds malicious instructions in third-party content that agents retrieve through tool calls, expanding their attack surface.
  • Conventional agents append all tool outputs, intermediate reasoning, and conversational traces to context, creating persistent and bloated memory.
  • 60.53% ASR occurs when injection enters the first round of four-step tasks, about four times the second-round rate and over ten times the third-round rate.
  • 44.46% utility on short tasks falls to 19.08% on long tasks, a 57% decline associated with verbose accumulated context.
  • Existing defenses either harden or filter bloated memory, or enforce rigid execution constraints that limit adaptive task decomposition.
  • AgentSys isolates worker-agent tool execution and permits only schema-validated JSON return values to enter the main agent’s memory.
  • 0.78% ASR on AgentDojo and 4.25% on ASB are achieved while benign utility reaches 64.36%, versus 63.54% for undefended agents.

2 Background

LLM agents operate through iterative cycles of planning, tool use, observation, and adaptation, with context storing the resulting trace. Indirect prompt injection arises when malicious instructions embedded in external content enter that trace and persist across later decisions.

  • An LLM agent combines planning, tool use, and memory to decompose goals, invoke external tools, and iteratively refine behavior.
  • At each round, the agent selects an action from a policy conditioned on system instructions, tool descriptions, the user query, and the prior trace.
  • The context is updated by appending each new reasoning, action, and observation turn until the agent selects a stop action and produces a final report.
  • Full-history memory supports adaptive multi-step workflows but also exposes later decisions to adversarial external observations.
  • Indirect prompt injection embeds adversarial instructions in seemingly benign external content that enters working memory when retrieved.
  • Control-flow manipulation changes execution paths or tool selection, whereas data-flow manipulation corrupts tool arguments and downstream data.
  • Once injected at round j, an instruction remains in all subsequent contexts and is reprocessed at every later decision point.

3 Existing Defenses and Motivation

Existing defenses span model robustness, detection guardrails, and system-level controls, but they generally accept accumulated memory as given or sacrifice flexibility for security. AgentSys targets the underlying problem by isolating execution and restricting information crossing into working memory.

  • Defense layers: Existing defenses are organized into model-level robustness, detection-based guardrails, and system-level control.
  • Model and detection defenses: Model-level methods separate user instructions from retrieved data through training or inference controls, while detection methods identify and sanitize suspicious content.
  • Detection-based limitations: Detection-based guardrails remain vulnerable to evasion and can reduce utility through false positives.
  • System-level limitations: System-level frameworks improve separation but often rely on predefined workflows, restricted dynamic tool use, or rigid execution constraints.
  • Shared limitation: Existing defense layers operate on accumulated memory, leaving unnecessary content and obsolete observations in the agent’s context.
  • Shared limitation: Detection overhead grows with memory length, while false positives can remove legitimate information and degrade utility.
  • Motivation: AGENTSYS fills this gap with hierarchical memory management, isolated workers, and schema-validated communication intended to preserve adaptive workflows.

4 System and Threat Model

The threat model treats tool-returned observations as attacker-influenceable inputs that can inject instructions into an agent’s persistent working memory. The formal agent model represents execution as policy-driven rounds over accumulated contexts and defines injection by divergence from clean behavior.

  • At each round, the backend LLM selects either termination or a tool call from the current context, while traces are appended until the first stop action.The context includes the system prompt, tool descriptions, user query, and accumulated trace.
  • An indirect prompt injection occurs when attacker-injected instructions enter a tool observation and alter the resulting trace relative to a clean trace.The injected observation may originate from a webpage, file, or API response.
  • The formal divergence measure captures differences in the agent’s actions and/or observations between clean and injected executions.
  • The adversary may influence any tool-returned observation but cannot modify tool execution or the environment transition.Its goal is to steer subsequent policy outputs by embedding instructions that persist across reasoning cycles.

5 AGENTSYS Design

AGENTSYS secures tool-using agents through explicit hierarchical memory management: isolated workers process untrusted outputs, while the main agent receives only schema-conforming structured values. Validator-gated recursion enables multi-step workflows without exposing the main agent to raw tool content.

  • Hierarchical isolation: AGENTSYS separates the trusted main agent from short-lived workers that process untrusted tool outputs in isolated contexts.Raw tool outputs are never appended directly to the main agent’s working memory.
  • Validator-gated recursion: The agent hierarchy supports recursive extraction, with nested worker calls mediated by an alignment validator using the user query, compact call trace, and proposed call triple.The trace contains tool identifiers, arguments, and declared intents, but never raw tool outputs.
  • Schema-constrained interfaces: The main agent pre-declares a minimal typed intent schema that fixes which fields may be returned and their expected types.Schemas use nested dictionaries and lists with primitive-typed leaves, such as strings, numbers, and booleans.
  • Validated returns: Only JSON-parsable return objects conforming to the declared intent are admitted into the main agent’s context; free-form tool text is rejected.This replaces large tool outputs with compact structured objects whose fields are determined by a pre-declared interface.
  • Memory-management benefits: By admitting only compact, schema-validated values, AGENTSYS prevents verbose content from accumulating and stops adversarial instructions from persisting across reasoning cycles.The framework targets both utility degradation from bloated memory and attack persistence from injected instructions.
  • Recovery and safety: When a worker call is denied, AGENTSYS treats the current output as potentially adversarial and initiates a sanitize–restart recovery loop.Recursive calls are therefore checked before execution rather than allowing workers to unilaterally trigger side-effecting actions.

6 Experiments

AGENTSYS combines hierarchical memory isolation with validation and sanitization to reduce attack success while preserving utility, including against adaptive attacks and longer trajectories. Across benchmarks and analyses, it balances security, utility, and computational cost better than relevant baselines.

  • Ablation Study: 0.18% ASR with unconditional sanitization came at the cost of 50.85% benign utility, whereas selective validation balances security and utility.Removing the sanitizer instead raised ASR to 1.54% and lowered benign utility to 57.66%, indicating that sanitization supports recovery from contaminated observations.
  • Ablation Study: 2.19% ASR with hierarchical memory management alone shows that isolating untrusted outputs provides substantial security benefits.The full system reaches 0.78% ASR and 64.36% benign utility by adding validation and sanitization.
  • Overhead Analysis: 63.86 defense quality with 3.25M tokens gave AGENTSYS the highest defense quality at a comparable token cost.Its security-utility position also combined 64.36% benign utility with 99.22% security, corresponding to 0.78% ASR.
  • Adaptive Attackers: 2.06% ASR under PAIR-refined adaptive attacks remained over 93% below the undefended baseline’s 30.66% ASR.ASR increased from 0.78% under the base attack to 1.43% under manual adaptation and 2.06% under PAIR refinement.
  • Trajectory Length: 24.58% utility on long tasks was the highest among compared methods, while ASR remained 0% for trajectories with four or more tool calls.AGENTSYS’s utility drop from short to long tasks was 38.21%, smaller than DRIFT’s 60.96% and Progent’s 52.98% drops.

7 Discussion

AGENTSYS addresses memory-related security and utility problems, but its validator, return channel, and intent schemas retain residual failure modes and design challenges.

  • AGENTSYS uses explicit memory management to address attack persistence and utility degradation in conventional agents.It ensures only essential, task-relevant information enters working memory.
  • Validator Reliability: 2.19% ASR occurs when the validator and sanitizer are removed, while full AGENTSYS reports 0.78% ASR, indicating residual validator failures.The validator can approve subtly misaligned malicious calls or reject legitimate calls.
  • Adaptive Attacks: Schema-validated return channels reduce the attack surface but string-valued fields can still carry adversarial instructions.Adaptive attackers can target fields such as extracted names, though experiments report limited success.
  • Intent Specification Complexity: Precise intent schemas may be difficult to specify for complex or exploratory tasks when the desired information structure is unknown beforehand.Schemas can become too restrictive or too permissive, respectively limiting information flow or expanding the attack surface.

8 Conclusion

AGENTSYS counters indiscriminate memory accumulation by isolating tool processing and restricting returns to compact, schema-validated values. It reports low attack success rates, improved benign utility, multi-model robustness, and practical overhead, supporting architectural memory management as a principled defense for dynamic agents.

  • AGENTSYS uses hierarchical context isolation and schema-bounded communication so only essential, task-relevant information enters working memory.Worker agents process tool outputs in isolated contexts and return compact, schema-validated values to the main agent.
  • 0.78% and 4.25% ASR on AgentDojo and ASB accompany improved benign utility of 64.36% versus 63.54% for undefended baselines.AGENTSYS also achieves 0% ASR on multi-step tasks with at least four tool calls.
  • AGENTSYS maintains robust performance across six foundation models and adaptive attackers with practical computational overhead.The evidence describes this robustness alongside its benchmark security and utility results.
  • Managing working memory through architectural boundaries offers a principled alternative to defenses operating on bloated context.The conclusion contrasts this approach with model-level robustness, detection, and rigid constraints.

Appendix

The appendix reports detailed AgentDojo utility and attack-success results for AGENTSYS across six foundation models, covering benign and attacked settings.

  • Tables 5–7 present AGENTSYS results across six foundation models on AgentDojo.The reported measures include benign utility, attacked utility, and attack success rate.
  • Table 5 reports utility on AgentDojo without attack as percentages.
  • Table 6 reports utility on AgentDojo under attack as percentages.
  • Table 7 reports attack success rate on AgentDojo under attack as percentages.
Loading 2602.07398v1…